#Yjخ4dZddlmZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlmZmZmZmZmZddlmZmZmZmZejeZe jddZ dgZ!hd Z"d Z#d Z$dIdZ%dJdKdZ&dddLdZ'e j(de j)Z*dIdZ+dIdZ,dddMd Z-dNd"Z.d#a/dOd%Z0dPd&Z1dQd)Z2dRd,Z3dSd0Z4dTd3Z5dUd6Z6Gd7d8Z7Gd9d:Z8dVd>Z9dWdAZ:dXdDZ;dYdFZ<GdGdHeZ=dS)ZuvCua-driver backend (macOS, Windows, Linux). Speaks MCP over stdio to `cua-driver`. The Python `mcp` SDK is async, so we run a dedicated asyncio event loop on a background thread and marshal sync calls through it. The same `cua-driver call ` surface (click, type_text, hotkey, drag, scroll, screenshot, launch_app, list_apps, list_windows, get_window_state, move_cursor, wait) works identically across macOS, Windows, and Linux — cua-driver's PARITY matrix marks the action tools VERIFIED on macOS and Windows in the cross-platform Rust port (`cua-driver-rs`). Linux is the most recent runtime (X11 today, Wayland via XWayland; pure- Wayland progress tracked upstream). It is enabled in `check_computer_use_requirements` alongside macOS and Windows. The plumbing in this file is OS-agnostic; per-host gaps (no DISPLAY, missing AT-SPI, etc.) surface as specific blocked checks via `hermes computer-use doctor` rather than failing silently. Install: - **macOS**: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)" - **Windows** (PowerShell): irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex After install, `cua-driver` is on $PATH and supports `cua-driver mcp` (stdio transport) which is what we invoke. The macOS path uses private SkyLight SPIs (SLEventPostToPid, SLPSPostEventRecordTo, _AXObserverAddNotificationAndCheckRemote) that aren't Apple-public and can break on OS updates. The Windows path in cua-driver-rs uses stable Win32 APIs (SendInput + UI Automation) — not subject to the same SPI breakage class. ) annotationsN)AnyDictListOptionalTuple) ActionResult CaptureResultComputerUseBackend UIElementHERMES_CUA_DRIVER_CMDz cua-drivermcp> full screenallscreendesktop fullscreen)progmanworkerwprogram manager shell_traywndtaskbarfinderrdockCUA_DRIVER_RS_TELEMETRY_ENABLEDreturnboolc ddlm}|pi}|dpi}t|dd S#t$rYdSwxYw)uTrue when Hermes should disable cua-driver telemetry for this user. Reads ``computer_use.cua_telemetry`` from config.yaml. Default is False (telemetry off). Any failure to read config fails SAFE — toward the privacy-preserving default of telemetry disabled. r) load_config computer_use cua_telemetryFT)hermes_cli.configrgetr Exception)rcfgcus SC:\Users\server\AppData\Local\hermes\hermes-agent\tools\computer_use\cua_backend.py_cua_telemetry_disabledr(ns 111111kmm!r WW^ $ $ *667777 ttsA A AAbase_envOptional[Dict[str, str]]Dict[str, str]cpt||n tj}tr d|t<|S)aReturn the environment dict for spawning cua-driver. Starts from ``base_env`` (defaults to ``os.environ``) and, when telemetry is disabled (the default), injects ``CUA_DRIVER_RS_TELEMETRY_ENABLED=0``. When the user has opted in, the var is left untouched so cua-driver uses its own default. Used by every cua-driver spawn site (MCP backend, status, doctor, install) so the policy is applied consistently. N0)dictosenvironr(_CUA_TELEMETRY_ENV_VAR)r)envs r'cua_driver_child_envr3s: 8/xxRZ @ @C  *&) "# Jg@timeout driver_cmdstrr6floatTuple[str, List[str]]c  ddlm}tj|dgdd|tj|t }n&#t $r|ttfcYSwxYw|j pd }|j dks|s|ttfS tj |}n-#ttf$r|ttfcYSwxYwt!|t"s|ttfS|d}t!|t"s|ttfS|d}|d }t!|trt'd |Ds|ttfSt!|t(r|s||fS||fS) uReturn ``(command, args)`` that spawn cua-driver's stdio MCP server. Surface 8 of NousResearch/hermes-agent#47072: instead of hardcoding ``["mcp"]`` we ask the driver itself via ``cua-driver manifest`` (trycua/cua#1961). The manifest carries a stable ``mcp_invocation`` pointer with both ``command`` and ``args``, so a future cua-driver that renames or relocates the subcommand keeps working without a Hermes patch. Falls back to ``(driver_cmd, ["mcp"])`` for older drivers that don't expose ``manifest``, or any indeterminate failure — the wrapper must not refuse to start just because the discovery hop failed. r_sanitize_subprocess_envmanifestTcapture_outputtextr6stdinr2mcp_invocationargscommandc3@K|]}t|tVdSN isinstancer8).0as r' z*_resolve_mcp_invocation..s,,N,NAZ3-?-?,N,N,N,N,N,Nr4)tools.environments.localr= subprocessrunDEVNULLr3r$list_CUA_DRIVER_ARGSstdoutstrip returncodejsonloads ValueError TypeErrorrJr.r#rr8) r7r6r=procoutr> invocationrErFs r'_resolve_mcp_invocationr^s$ 2EEEEEE~  $dG$)()=)?)?@@    2224 01111112 ; " # # % %C !34 011112:c??  "2224 01111112 h % %24 01111.//J j$ ' '24 01111 >>& ! !DnnY''G dD ! !2,N,N,N,N,N)N)N24 01111 gs # # 7 4 D=s$AA A'&A')B>>'C('C(z~^\s*(?:-\s+)?\[(\d+)\]\s+(\w+)(?:\s*=\s*"([^"]*)"|\s+"([^"]*)"|\s+\((?!\d+\))([^)]*)\))?(?:\s+(?:\(\d+\)\s+)?id=([^\s\[\]]+))?c"tjdkS)Ndarwinsysplatformr4r' _is_macosres <8 ##r4cNttjtS)zCTrue if `cua-driver` is on $PATH or HERMES_CUA_DRIVER_CMD resolves.)rshutilwhich_CUA_DRIVER_CMDrdr4r'cua_driver_binary_availablerjs  _-- . ..r4g @Optional[Dict[str, Any]]c  ddlm}tjtddgdd|tj|t }n#t$rYdSwxYw|jpd }|sdS tj |}n#ttf$rYdSwxYwt|tr|d rdS|S) aRun ``cua-driver check-update --json`` and return its parsed state. The payload mirrors the ``check_for_update`` MCP tool: ``{current_version, latest_version, update_available, ...}``. Returns ``None`` (callers should stay quiet) when the result is indeterminate: the binary is missing, the driver is too old to support the verb (it predates trycua/cua#1734), the GitHub check failed (an ``error`` field is set), or the output didn't parse. Best-effort; never raises. rr<z check-updatez--jsonTr?NrCerror)rNr=rOrPrirQr3r$rTrUrWrXrYrZrJr.r#)r6r=r[r\datas r'cua_driver_update_checkrosEEEEEE~ nh 7dG$)()=)?)?@@     tt ; " # # % %C tz#  "tt dD ! !TXXg%6%6t Ks$AA AA;BB%$B% Optional[str]ct}|r|dsdS|dpd}|dpd}d|d|dS) z~One-line "an update is available" message, or ``None`` when up to date, indeterminate, or the driver is too old to report.update_availableNlatest_version?current_version cua-driver z is available (you have z7); update with `hermes computer-use install --upgrade`.)ror#)statelatestcurrents r'cua_driver_update_nudgerzs $ % %E  "455t YY' ( ( /CFii)**1cG @f @ @g @ @ @r4FNonecvtrdSdadd}tj|dddS) zEmit an update nudge at most once per process, off-thread so the (cached, ~20h) GitHub poll never blocks the first computer_use action.NTrr{c t}n#t$rYdSwxYw|rtd|dSdS)Nzcomputer_use: %s)rzr$loggerinfo)msgs r'_runz!_maybe_nudge_update.._run3sc )++CC    FF   1 KK*C 0 0 0 0 0 1 1s  zcua-driver-update-check)targetnamedaemonrr{)_update_checked threadingThreadstart)rs r'_maybe_nudge_updater+sXO11113D egggggr4c8tjdkrd}nd}d|dS)Nwin32za irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iexzs /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)"zxcua-driver is not installed. Install with one of: hermes computer-use install Or run the upstream installer directly: zW Or run `hermes tools` and enable the Computer Use toolset to install it automatically.ra) installers r'cua_driver_install_hintr@sG |w 8   3   a  a a ar4markdownList[UIElement]c g}t|D]}|dp@|dp+|dp|dpd}|t t |d|d|d |S) uParse UIElement list from get_window_state AX tree markdown. Last-resort fallback for cua-driver builds that don't carry the canonical ``structuredContent.elements`` array (see ``_parse_elements_from_structured`` — Surface 2 of #47072 prefers that path). Captures the label whichever form cua-driver used: ``= "value"``, ``"quoted"``, ``(parenthesised)``, or ``id=Label``. Bounds always come back ``(0, 0, 0, 0)`` because the markdown surface doesn't carry them — yet another reason to prefer the structured path; element-index clicks don't need them (the driver resolves the index to a frame internally). rCrrrr)indexrolelabelbounds)_ELEMENT_LINE_REfinditergroupappendr int)relementsmrs r'_parse_elements_from_treerUsH  & &x 0 0   JaggajjJAGGAJJJ!''!**J aggajj//        Or4 raw_elementsList[Dict[str, Any]]c g}|D]}t|ts|d}t|tsDt|dtr|dnd}t|dtr|dnd}t|dtr|dnd}d}|r t|dd t|d d t|d d t|d d f}n#t t f$rd}YnwxYw|d }t|tr|r|nd} |t||||| |S)uGSurface 2 of NousResearch/hermes-agent#47072: read the canonical ``structuredContent.elements`` array cua-driver-rs emits on every ``get_window_state`` response (trycua/cua#1961). Each entry has at minimum ``element_index``, ``role``, ``label``; ``frame`` (``{x, y, w, h}``) is included whenever the AT-SPI / AXFrame call returned usable bounds. Older code parsed the same information out of the markdown tree via a regex (lossy: bounds were always ``(0, 0, 0, 0)``) — this path preserves the real frame so downstream consumers (e.g. ``UIElement.center()``) work against pixel coordinates instead of just the index lookup. Unknown / malformed entries are skipped rather than failing the whole walk — the wrapper degrades to "fewer elements" rather than "no elements" on a bad row. element_indexrrCrframeNrxrywh element_token)rrrrr) rJr.r#rr8rZrYrr ) rrrawidxrrrr raw_tokentokens r'_parse_elements_from_structuredrqs"!#H  #t$$  ggo&&#s##  ",SWWV__c"B"BJswwv$.swww/?/?$E$EM   2$.swww/?/?$F$FP   D,8  & & #q))** #q))** #q))** #q))**  z* & & &% & GGO,, ' 377OIO 4          Os B FF,+F,rbytesTuple[int, int]cF|drit|dkrVt|ddd}t|ddd}|dkr |dkr||fS|dr d}t|}|d z|kr||d kr|d z }||d z}|dz }|d vsd |cxkrdkrnn@|dz|krnt|||dzd}|dks ||z|krn{|dvri|dkrbt||dz|dzd}t||dz|dzd}|dkr |dkr||fSn||z }|d z|kdS)zCBest-effort PNG/JPEG dimension sniffing without extra dependencies.sPNG  bigrsr r>> rr)rr) startswithlenr from_bytes)rwidthheightinmarker segment_lens r'_image_dimensions_from_bytesrs ~~*++!CBs2b5z511BrE E22 199!&=  ~~k""  HH!eaii1v~~QQZF FA%%)?)?)?)?4)?)?)?)?)?1uqyy..Qq1uWu==KQ!k/A"5"5!## ^^CAa!e ,fnaltcmdctrlshiftoptionrFcontrolrrr)rFrrcg|]<}||=Srd)rUlower)rKps r' z$_parse_key_combo..s7 N N N1AGGII NQWWYY__   N N Nr4z[+\-]N)rerr#r)rMODIFIER_NAMES KEY_ALIASESparts modifierskeypart normalizeds r'_parse_key_combors [ZZN#HHHK N N4(@(@ N N NEI C __T400  ' '   Z ( ( ( (CC  >r4c4eZdZdZd dZd dZddd Zd d Zd S) _AsyncBridgezNRuns one asyncio loop on a daemon thread; marshals coroutines from the caller.rr{cRd|_d|_tj|_dSrH)_loop_threadrEvent_readyselfs r'__init__z_AsyncBridge.__init__s#:> 37 o'' r4cHjrjrdSjd fd }t j|dd_jjdstd dS) Nrr{ctj_tjjj j jdS#t$rYdSwxYw# jw#t$rYwwxYwxYwrH) asyncionew_event_looprset_event_looprset run_forevercloser$rsr'rz _AsyncBridge.start.._runs /11DJ  "4: . . . KOO     &&(((J$$&&&&& DDJ$$&&&& DsB B'B BBB?B/.B?/ B<9B?;B<<B?Tzcua-driver-loop)rrr@r5z)cua-driver asyncio bridge failed to startr) ris_aliverclearrrrwait RuntimeError)rrs` r'rz_AsyncBridge.starts < DL1133  F       !'tDGXYYY  {,, LJKK K L Lr4>@r6Optional[float]rc:ddlm}|jr |jr|js7t j|r|td|||j}|td| |S)Nr)safe_schedule_threadsafecua-driver bridge not startedr5) agent.async_utilsrrrr r iscoroutiner rresult)rcoror6rfuts r'rPz_AsyncBridge.run s>>>>>>z @ @T\5J5J5L5L @"4((  >?? ?&&tTZ88 ;>?? ?zz'z***r4c|jr=|jr$|j|jj|jr|jdd|_d|_dS)Ng@r5)r is_runningcall_soon_threadsafestoprjoinrs r'rz_AsyncBridge.stopsp : =$*//11 = J + +DJO < < < < + L  c  * * *  r4Nrr)r6rrr)__name__ __module__ __qualname____doc__rrrPrrdr4r'rrssXX(((( LLLL, + + + + +r4rceZdZdZd)dZd*dZd*dZd+d Zd*d Zd*d Z d*dZ d*dZ d*dZ d,dZ d-d.dZd/dZed0dZed1dZed2d!Zed2d"Zd*d#Zd3d&Zd4d3d(ZdS)5_CuaDriverSessionuHolds the mcp ClientSession. Spawned lazily; re-entered on drop. Lifecycle ownership: a single long-running coroutine (`_lifecycle_coro`) opens both the stdio_client and ClientSession contexts, populates capabilities, sets `_ready_event`, and then waits on `_shutdown_event`. When shutdown is signalled the same coroutine closes the contexts — keeping anyio's cancel-scope task-identity invariant intact (the bridge schedules each `bridge.run(coro)` as a NEW task, so opening contexts in one and closing them in another raises "Attempted to exit cancel scope in a different task"). Tool calls run in their own short-lived tasks; they only touch the session object, never the surrounding contexts. bridgerrr{c||_d|_tj|_d|_i|_d|_tj|_ d|_ d|_ d|_ dS)NFrC) _bridge_sessionrLock_lock_started _capabilities_capability_versionr _ready_event_shutdown_event_lifecycle_future _setup_error)rr&s r'rz_CuaDriverSession.__init__3sf  ^%%  .0(* %O--8<!%59r4c2|jstddS)Nzcua-driver session not started)r,rrs r'_require_startedz"_CuaDriverSession._require_startedGs(} A?@@ @ A Ar4c Kddl}ddlm}m}ddlm}ddlm}tj |_ | }d|_ tsttd|_ t!t"\}}| } ||||t%} || 4d{V\} } d |_ || | 4d{V} | d{V| }d |_ || d{V| |_d |_ |jt0d | |z | |z || z |j d{Vdddd{Vn#1d{VswxYwYdddd{Vn#1d{VswxYwYn3#t6$r&}||_|jd}~wwxYwd|_dS#d|_wxYw) u}Long-lived owner of the stdio MCP contexts. Opens, signals ready, blocks on shutdown, then cleans up. enter + exit happen in the SAME asyncio task, so anyio's cancel-scope invariant holds — fixing the "Attempted to exit cancel scope in a different task than it was entered in" warning emitted by the previous _aenter/_aexit split. rN) ClientSessionStdioServerParameters) stdio_clientr<z binary-checkzmanifest-discovery)rFrEr2zmcp-initializezcapability-discoveryreadyzBcua-driver session ready in %.1fs (manifest=%.1fs, mcp_init=%.1fs))timerr6r7mcp.client.stdior8rNr=rrr0 monotonic_startup_phaserjrrr^rir3 initialize_populate_capabilitiesr)r/rr~rr BaseExceptionr2)r_timer6r7r8r=_t0rFrE _t_manifestparamsreadwritesession_t_inites r'_lifecycle_coroz!_CuaDriverSession._lifecycle_coroKs <<<<<<<<111111EEEEEE '}oo-6 !.00 >"#:#<#<=== #7D 3ODDMGT//++K**-,-A-C-CDD F$|F++ 6 6 6 6 6 6 6}e&6#(=u556666666!,,.........#oo//G+AD'55g>>>>>>>>>$+DM*1D'%))+++KK;))C/#c)+- .33555555555+666666666666666666666666666 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 60   !"D    ! ! # # #   !DMMMDDM stBHG?;CG G? G& &G?)G& *G?-H? H  H H  HI  I!H<<II IrGrcK |d{V}t|dgpgD]}t|dd}t|ts)t|dd}|(t|ddpi}|d}t|t rd|D|j|<t|j|<t|dd}|(t|ddpi}|d}t|tr ||_dSdS#t$r&}t d|Yd}~dSd}~wwxYw) uSurface 4: cache per-tool capability sets + capability_version from tools/list. Soft prerequisite — discovery failure leaves the map empty and supports_capability degrades to False.Ntoolsr capabilities model_extrac<h|]}t|t|SrdrI)rKcs r' z;_CuaDriverSession._populate_capabilities..s7555:a+=+=5555r4capability_versionz5cua-driver tools/list capability discovery failed: %s) list_toolsgetattrrJr8r#rRr-rr.r$r~debug) rrG tools_listtool tool_namecapsextracvrIs r'r?z(_CuaDriverSession._populate_capabilitiess U&1133333333J GR88>B : :#D&$77 !)S11t^T::<$D->>D"E 99^44DdD)):55#'555D&y1158EED&y11 %94@@Bz M4@@FBYY344"c"" .+-((( . . U U U LLPRS T T T T T T T T T UsD)D11 E!;EE!c|j5|jr ddddS|j|d|_ddddS#1swxYwYdS)NT)r+r,r(r_start_lifecycle_lockedrs r'rz_CuaDriverSession.starts Z ! !}  ! ! ! ! ! ! ! ! L    ( ( * * * DM  ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !s A4AA#&A#ctj|_d|_d|_|jj}|tdtj | ||_ |j dsI| t|dd}ddlm}td |d |d |jtd |j|jdS) z^Spawn the lifecycle owner and wait for it to reach ready. Caller must hold self._lock.Nrrr5r=unknownr)display_hermes_homezEcua-driver session never reached ready (timeout 30s; stuck in phase: z.). Run `hermes computer-use doctor` and check z&/logs/agent.log for the phase timings.z!cua-driver session setup failed: )rrr/r2r0r(rrrrun_coroutine_threadsaferJr1r_signal_shutdown_lockedrThermes_constantsr`)rloopphaser`s r'r]z)_CuaDriverSession._start_lifecycle_lockeds@&O-- #|! <>?? ?!(!A  " "D" "  %%d%33   ( ( * * *D"2I>>E < < < < < <Q#(QQ'&((QQQ    (GD4EGG$ % ) (r4c|j5|js ddddSd|_|ddddS#1swxYwYdS)NF)r+r,_stop_lifecycle_lockedrs r'rz_CuaDriverSession.stops Z * *=  * * * * * * * *"DM  ' ' ) ) )  * * * * * * * * * * * * * * * * * *s AAA  A c\||j}|dS |dna#tjj$rt dYn1t$r%}t d|Yd}~nd}~wwxYwd|_dS#d|_wxYw)zbSignal shutdown + wait for the lifecycle coroutine to unwind. Caller must hold self._lock.Nr r5z*cua-driver session shutdown timed out (5s)zcua-driver shutdown error: %s) rbr1r concurrentfutures TimeoutErrorr~warningr$)rrrIs r'rgz(_CuaDriverSession._stop_lifecycle_lockeds $$&&&$ ; F * JJsJ # # # #!. I I I NNG H H H H H ? ? ? NN:A > > > > > > > > ? &*D " " "TD " ) ) ) )s88B".B&B"( B1B B"BB"" B+c|jj}|j}|D|D|r2 ||jdS#t $rYdSwxYwdSdSdS)z8Set the asyncio shutdown event from the caller's thread.N)r(rr0rrrr)rrdevents r'rbz)_CuaDriverSession._signal_shutdown_lockeds|!$   1doo6G6G 1 ))%)44444       1 1 1 1sA AArr8rEDict[str, Any]cfK|j||d{V}t|SrH)r) call_tool_extract_tool_result)rrrErs r'_call_tool_asyncz"_CuaDriverSession._call_tool_asyncs=}..tT::::::::#F+++r4N capabilityrWrprc|)|j|tvStfd|jDS)uReturn True when the connected cua-driver advertises the given capability token (trycua/cua#1961 capability vocabulary). When ``tool`` is given, scope the check to that specific tool's advertised capability set. When omitted, return True if ANY tool advertises the capability — useful for "is this feature available anywhere on the driver" probes. Always returns False before the session is started (so consumers on a dead/uninitialised wrapper degrade rather than crash). Nc3 K|]}|vV dSrHrd)rKrYrts r'rMz8_CuaDriverSession.supports_capability..!s(NN$:%NNNNNNr4)r-r#ranyvalues)rrtrWs ` r'supports_capabilityz%_CuaDriverSession.supports_capabilitys_  !3!7!7cee!D!DD DNNNN$2D2K2K2M2MNNNNNNr4c||jvS)umReturn True when ``tools/list`` advertised a tool by this name. Used to route capture(): cua-driver dropped the standalone ``screenshot`` tool and folded full-window PNG capture into ``get_window_state`` (whose own description notes it "Also captures a PNG screenshot of the specified window"). Older drivers that still expose ``screenshot`` keep using it; newer ones fall through to ``get_window_state``. Returns False when discovery hasn't populated the map yet — callers treat that as "unknown" and probe defensively rather than trusting it. )r-)rrs r' _has_toolz_CuaDriverSession._has_tool#st)))r4c*t|jS)zTrue once ``tools/list`` populated the per-tool map. When False, ``_has_tool`` answers are not trustworthy (discovery failed or the session hasn't started) and capture() should probe defensively.)rr-rs r'capabilities_discoveredz)_CuaDriverSession.capabilities_discovered2s D&'''r4c|jS)uDriver-advertised capability vocabulary version (empty string when the driver predates the field — older builds had no version).)r.rs r'rRz$_CuaDriverSession.capability_version9s ''r4excr$c|jj}t|jdd}|dvp4|dod|vpt |t t fS)zHReturn True for MCP/stdio failures that are recoverable by reconnecting.r!rC> EndOfStreamBrokenResourceErrorClosedResourceErroranyioResource) __class__r rTrrJBrokenPipeErrorEOFError)rrmodules r'_is_closed_session_errorz*_CuaDriverSession._is_closed_session_error?sf}% b99 Q Q <!!'**AzT/A <#:;; r4c@t|}d|vp d|vpd|vpd|vS)uReturn True for the cua-driver daemon-proxy EAGAIN congestion error. On macOS the ``cua-driver mcp`` bridge forwards calls to the CuaDriver daemon over a non-blocking unix socket. Heavier ops (notably ``get_window_state``, which walks the AX tree and captures a PNG) can come back as an ``McpError`` carrying ``Resource temporarily unavailable (os error 35)`` — POSIX EAGAIN — when the socket buffer is momentarily full. This is transient by definition: the same call succeeds when retried after a short pause (which is why spaced-out single calls work while rapid/large ones intermittently fail). Detect it by message so we can retry with backoff rather than surfacing an empty 0x0 capture to the model. See the EAGAIN diagnosis in references/catalog-add-troubleshooting (apple-music skill) and the cua-driver daemon-proxy note. z Resource temporarily unavailablez os error 35zdaemon transport errorz daemon proxy)r8)rrs r'_is_transient_daemon_errorz,_CuaDriverSession._is_transient_daemon_errorJsE"#hh .# 5 %# %'3. %$  r4c|jrH |n2#t$r%}td|Yd}~nd}~wwxYwd|_i|_d|_|d|_dS)zRecreate the MCP session after the daemon/stdin transport was closed. Caller must hold self._lock (the reconnect-once retry path holds it).z6cua-driver session cleanup before reconnect failed: %sNFrCT)r,rgr$r~rUr-r.r]rrIs r'_restart_session_lockedz)_CuaDriverSession._restart_session_lockedcs = Z Z++---- Z Z Z UWXYYYYYYYY Z #%  $$&&& s A AA r6r9c ddl}ddl}ddl}ddlm}t |}d} |dkr7d|vr3|dd\} } tj| | |d<td |tj |g} d } d } d}d } t| D]h} | | d d td||t}n(#t $r}t#d|d||d}~wwxYw|jpd }|ddp|jpd dd}t+d|d|dfDd}|dkr; tj||d}n#tj$rd}YnwxYw||}nF|| dz kr;t2d||dz| | || | dz} j|t#d|d| d|g}d}t9|t r|nd}d}t9|t rj|dd up|dd u}|d }|s|d!p| }|rtj|r tA|d"5}tCj"|#$d#}dddn #1swxYwYn3#t $r&}t2%d$||Yd}~nd}~wwxYw|r|&||d%}|)|d&}||d'nd }|r|d(|n|}||||d)| rEtj| r' tj'| S#tP$rYSwxYwSS#| rEtj| r' tj'| w#tP$rYwwxYwwwxYw)*uFallback transport: invoke ``cua-driver call `` as a subprocess instead of going through the stdio MCP bridge. The ``cua-driver mcp`` stdio bridge can persistently fail to forward heavier calls (notably ``get_window_state``) to the daemon with POSIX EAGAIN, while the plain ``cua-driver call`` path — which talks to the daemon over its own socket — keeps working. When the MCP path gives up, we retry over the CLI and remap the JSON into the same dict shape that ``_extract_tool_result`` produces, so callers (capture(), _action(), list_windows parsing) are transport-agnostic. For ``get_window_state`` we route the screenshot to a temp file via ``screenshot_out_file`` so the daemon returns a tiny JSON body (a path) instead of a multi-megabyte base64 blob — the large payload is what congests the daemon socket and triggers EAGAIN in the first place. We read the PNG back from disk and base64-encode it ourselves. The CLI call is itself retried a few times with backoff, since the underlying daemon socket can still be momentarily busy. rNr<get_window_statescreenshot_out_file cua_shot_z.png)prefixsuffixcallrg?rCTg.@)r@rAr6r2zcua-driver CLI fallback for z failed to spawn: c3&K|] }|dk|V dS)Nrd)rKrs r'rMz7_CuaDriverSession._call_tool_via_cli..s&JJ1!r''Q''''JJr4{[r)defaultrzMcua-driver CLI fallback for %s got no JSON (attempt %d/%d); retrying in %.1fsrz returned no JSON after z attempts: FisErroris_errorscreenshot_png_b64screenshot_file_pathrbasciiz.cua-driver CLI fallback: failed reading %s: %s tree_markdown element_countz elementsr)rnimagesstructuredContentr))rOtempfiler:rNr=r.mkstempr/r rirWdumpsrangerPmaxr3r$rrTrUstderrminfindrXJSONDecodeErrorr~rlsleeprJr#pathexistsopenbase64 b64encoderEdecoderUrremoveOSError) rrrEr6 _subprocess _tempfilerAr= call_args shot_filefdrattemptsbackoffparsedlast_errattemptr[rIr\r candidaterrn structuredrshotfpathfhrecrs r'_call_tool_via_cliz$_CuaDriverSession._call_tool_via_clirs( )((($$$$EEEEEEJJ #' % % %*?y*P*P%--[-PPMB HRLLL/8I+ ,dj.C.CDN  ?? ! !j&??DtSwEWEW445I5K5KLL+DD!jjj&'ad'a'a^_'a'abbhiij{(b//11tt9A):DSD(AJJ# >JJJB;;)$(Js566{$;$; /)))$( ) ,!*X\))NN=gk8W KK(((qLG~"747777,477 !#FD3=fd3K3K)UQUJH&$'' E"::i00D8ZFJJz>K)Ee!6!6ee!%eT!2!2Sb'-'7 'B'B'I'I''R'RSSSSSSSSSSSSSSS(eee"LL)Y[`bcdddddddde(MM$'''zz/22#O44B24.....bG3:Dg/////D %/#  RW^^I66 Ii((((D  y RW^^I66 Ii((((D  sP=CP C="C88C==BPFPF1.P0F11DP L/:L# L/#L''L/*L'+L/.P/ M9MPMA P!O66 PP"Q+Q?Q Q  Q Q  Qrc|| |j||||S#t$r}||r8t d||||||cYd}~S| |st d||j 5| dddn #1swxYwY|j||||cYd}~Sd}~wwxYw)Nr5zIcua-driver MCP transport failed on %s (%s); falling back to CLI transportz:cua-driver MCP session closed during %s; reconnecting once) r4r(rPrsr$rr~rlrrr+r)rrrEr6rIs r'rqz_CuaDriverSession.call_tools  X<##D$9$9$$E$Ew#WW W X X X..q11 D4591..tT7CCCCCCCC0033  NNWY] ^ ^ ^ / /,,... / / / / / / / / / / / / / / /<##D$9$9$$E$Ew#WW W W W W W W XsN/A D;AD6D;8D6C6* D66C: :D6=C: >2D60D;6D;)r&rrr{r)rGrrr{rr8rErorrorH)rtr8rWrprr)rr8rrrrrr8)rr$rr)rr8rEror6r9rror)r r!r"r#rr4rJr?rr]rrgrbrsryr{propertyr}rR staticmethodrrrrrqrdr4r'r%r%$s  ::::(AAAAL!L!L!L!\!U!U!U!UF!!!!#%#%#%#%J********&    ,,,, OOOOO * * * *(((X( (((X(    \    \ 0    ssssjXXXXXXXr4r% mcp_resultrrocd}g}g}t|dddu}t|ddpd}g}t|dgpgD]}t|dd}|dkr'|t|dd pd @|d krPt|d d} | r=|| t|d dpd } || |rud d|D} | drt j| n| }n#t j$r| }YnwxYw|||||dS)uConvert an mcp CallToolResult into a plain dict. cua-driver returns a mix of text parts, image parts, and structuredContent. We flatten into: { "data": , "images": [b64, ...], "image_mime_types": [mime, ...], # parallel to `images`, "" when absent "structuredContent": , "isError": bool, } structuredContent is populated from the MCP result's structuredContent field (MCP spec §2024-11-05+) and takes precedence for structured data like list_windows window arrays. `image_mime_types` is the explicit `mimeType` cua-driver emits on every image part as of trycua/cua#1961 (Surface 7 of NousResearch/hermes-agent#47072). Each entry corresponds index-for-index with `images`; an empty string entry signals the part carried no mimeType (older cua-driver build), and the caller should fall back to base64-prefix sniffing. NrFTrcontenttyperArCimagernmimeTyperc3K|]}||V dSrHrd)rKts r'rMz'_extract_tool_result../s'77Q71777777r4)rr)rnrimage_mime_typesrr)rTrrrUrrWrXr) rrnrrrr text_chunksrptypeb64mimejoineds r'rrrrs.DF"$z9e44B ? ? ? ? g  $--C . c"""tZ66<" ''---77k77777 )/)B)B:)N)NZ4:f%%%TZDD#   DDD ,'   s;=D99E  E r\#tuple[Optional[str], Optional[str]]c|dpg}|r=|dr5|dpg}|r|dr|dnd}|d|fS|dpi}|dp|d}|r0|dp|d pd}||fSd S) uPull a (png_b64, mime_type) pair out of a flattened tool result. cua-driver delivers window screenshots in two shapes depending on tool + transport: * As an MCP ``image`` content part — surfaced by ``_extract_tool_result`` in ``out["images"]`` with a parallel ``image_mime_types`` entry. This is what ``get_window_state`` emits over the stdio MCP transport. * As a base64 field inside ``structuredContent`` — ``screenshot_png_b64`` (+ ``screenshot_mime_type``). This is what ``get_window_state`` returns when its structured payload carries the image instead of a content part (newer driver builds; also the shape seen via the ``cua-driver call`` CLI surface). Checking both makes capture() robust to either delivery shape, so the image never silently drops just because the driver moved it between the content list and structuredContent. Returns ``(None, None)`` when neither location carries an image. rrrNrrpng_b64screenshot_mime_type mime_type)NNr#)r\rmimesrrrs r'_image_from_tool_resultr=s(WWX   $"F &)*++1r 7U1X7uQxx4ay$,--3J ..- . . K*..2K2KC  NN1 2 2 ~~k**  Dy :r4value Optional[int]ct|tst|ttfsdS t|}n#t$rYdSwxYw|dkr|ndS)zCReturn a positive integer, rejecting booleans and malformed values.Nr)rJrrr8rY)rrs r' _positive_intrdst%jc &C&CtU ttaZZ66T)sA AA raw_windowsc g}|D]}t|d}t|d}||K|d}t|ttfrt|t s|nd}||dd|||dd  |d d|d |S) u Normalise cua-driver ``list_windows`` entries, dropping unusable ones. Every downstream operation needs both an integer ``pid`` (for get_window_state / action tools) and ``window_id`` (for screenshot / element clicks), so a window missing either is uncapturable. Crucially, on X11 a window's PID comes from the *optional* ``_NET_WM_PID`` property — the desktop root, panels, and override-redirect popups routinely omit it, so the driver reports ``pid: null`` for them. Coercing every entry unconditionally (``int(w["pid"])``) let one such window abort enumeration of the real, targetable windows. We skip the unusable entries instead so capture() and focus_app() still find the windows that matter. ``z_index`` follows CUA Driver semantics: higher = closer to front. Wayland may return ``z_index: null`` (undefined stacking order); we treat null as the lowest priority so real windows still sort above desktop/root windows, and the backmost never ends up selected as the capture target. pid window_idNz_indexrapp_namerC is_on_screenTtitlerrr off_screenrr)rr#rJrr9rr)rwindowsrpid_int window_id_intz_rawrs r'_ingest_windowsros*%'G   e --%aeeK&8&899 ?m3 i  %ec5\::a:eUYCZCZa%%`aj"--&eeND999UU7B''        Nr4ceZdZdZddZddZddZddZdd ZdddZ ddZ ddZ ddZ dddZ dddd d!dd"dd,Zddddd dd-dd3Zd4ddddd5dd8Zdd:Zdd<Zddd>Zdd?Zdd@ZdddCZdddddAdDddIZddJZddKddLZddMZddOZddPZdQdRdSdTdd[Zdd\dd_Zddddd`ddeZdddddfddjZ dd\ddkZ!dAdlddoZ"ddpZ#ddqZ$dAdQdrddvZ%ddwZ&ddxZ'ddyZ(ddzZ)dd~Z*dddddZ+ddZ,ddZ-dS)CuaDriverBackendz@Default computer-use backend. Cross-platform via cua-driver MCP.rr{ct|_t|j|_d|_d|_d|_d|_i|_dtj j dd|_ dS)Nzhermes- ) rr(r%r) _active_pid_active_window_id _last_app _last_target_snapshot_tokensuuiduuid4hex _session_idrs r'rzCuaDriverBackend.__init__sv#~~ )$,77 *.04(,AE13$!B$*,,*:3B3*? A Ar4cVtddlm}|ddddl}||j |jdd|jidS#t$r&}t d|Yd}~dSd}~wwxYw) Nr)ensureztool.computer_useF)prompt start_sessionrGz:cua-driver start_session failed (continuing anonymous): %s) rtools.lazy_depsr importlibinvalidate_cachesr)rrqrr$r~rU)r _lazy_ensurer rIs r'rzCuaDriverBackend.starts ;::::: (7777 ##%%%  Z M # #OiAQ5R S S S S S Z Z Z LLUWX Y Y Y Y Y Y Y Y Y Zs"A88 B(B##B(cl|jjrV |jdd|jin2#t$r%}t d|Yd}~nd}~wwxYw |j|jdS#|jwxYw)N end_sessionrGz7cua-driver end_session failed (continuing teardown): %s) r)r,rqrr$r~rUrr(rs r'rzCuaDriverBackend.stops = ! [ [ '' 4CS7TUUUU [ [ [ VXYZZZZZZZZ [ M   L       DL      s "1 A AA $BB3rc>tjdvrdStS)N)r`rlinuxF)rbrcrjrs r' is_availablezCuaDriverBackend.is_availables# <; ; ;5*,,,r4cLd|_d|_d|_d|_i|_dS)zGForget a capture/focus target so a failed lookup cannot misroute input.N)rrrrrrs r'_clear_active_targetz%CuaDriverBackend._clear_active_targets/!%  "r4rCmoder8messager c X|t|dddgd|dS)zAReturn an empty capture after disarming any prior target context.rNrC)rrrrrapp window_title png_bytes_len)rr )rrrs r'_failed_capturez CuaDriverBackend._failed_capturesC !!###     r4rrErocj |j||}n##t$r|wxYw|ddur[|d}|t d|dt |tr|rd|ndz|S)zKCall a capture-stage tool and disarm state on transport or logical failure.rTrnrvz failedz: rC)r)rqr$rr#rrJr8)rrrEr\rs r'_call_capture_toolz#CuaDriverBackend._call_capture_tools -))$55CC     % % ' ' '   779   % %ggfooG  % % ' ' '+d+++%/%=%=Q'Q>>>>rS  s >rc |dd|jd}|dpidpg}t|}|dd|r|St d |jdd|jdd }n4#t$r'}t d |gcYd }~Sd }~wwxYw|d dur0t d | gS|dpidpg}t|}|dd|S)uLoad normalized visible windows, with the shared CLI recovery path. Windows are sorted by ``z_index`` **descending**: CUA Driver defines higher values as closer to the front, so the frontmost window ends up at index 0 — which is what ``capture()`` and ``focus_app()`` pick as the default target. ``_ingest_windows`` already normalised null ``z_index`` (Wayland) to 0, so those windows sort to the back. list_windowsT)on_screen_onlyrGrrc|dSNrrdrs r'z0CuaDriverBackend._load_windows..3 1Y<r4)rreversezScua-driver list_windows returned no windows over MCP; re-fetching via CLI transportg4@z3cua-driver CLI re-fetch for list_windows failed: %sNrz:cua-driver CLI re-fetch for list_windows returned an errorc|dSr"rdr#s r'r$z0CuaDriverBackend._load_windows..Jr%r4) rrr#rsortr~rlr)rr$rmr)rr\rrcli_outrs r' _load_windowszCuaDriverBackend._load_windows#s%% #0@ A A  ww2339r>>yIIOR !+.. // >>>  N ,    m66#'D4DEEGG     LLNPS T T TIIIIII  ;;y ! !T ) ) LLU V V V  % % ' ' 'I{{#677=2BB9MMSQS !+.. // >>>s$B66 C'C"C'"C'rrc | sgS fd|D}|r|S |}n5#t$r(}td||g}Yd}~nd}~wwxYwt t |D]ttr ddur/ d}t|tst|ttfsv t|}n#t$rYwxYw|dkrfdd D} |vr |t fd |Dr| fd |D} | r| S fd |D} | r| Sfd |D} | r| S fd|DS)aResolve ``app=`` through exact names before convenience substrings. Linux ``list_windows`` can omit an app name while ``list_apps`` retains name/bundle-ID metadata. Exact direct names and exact metadata aliases must win over substring matches: querying ``Code`` must not silently select ``Visual Studio Code`` merely because it is frontmost. c g|]O}t|ddkM|PSrrCr8r#rUrrKr app_lowers r'rz;CuaDriverBackend._match_windows_for_app..[s\   Cj" 5 566<<>>DDFFFF FFFr4z1computer_use list_apps fallback failed for %r: %sNrunningFrrch|]f}t|xt,@gSrd)rJr#r8rUr)rKrraw_apprs r'rQz:CuaDriverBackend._match_windows_for_app..zsr C(8(88u3??EJKKMM ##%%r4) bundle_idbundleIdrr display_namec3 K|]}|vV dSrHrd)rKaliasr0s r'rMz:CuaDriverBackend._match_windows_for_app..s(==EY%'======r4cBg|]}|dv|Srr)rKr exact_pidss r'rz;CuaDriverBackend._match_windows_for_app..s-KKKe 0J0J!0J0J0Jr4c g|];}t|ddv9|.sP   Cj" 5 566<<>>>> >>>r4cBg|]}|dv|Sr:r)rKr partial_pidss r'rz;CuaDriverBackend._match_windows_for_app..s-OOO!!%%,,,2N2NA2N2N2Nr4c g|]p}t|dds9t|ddvn|qS)rrCrr.r/s r'rz;CuaDriverBackend._match_windows_for_app..s|   quuZ,,--3355 Sw!3!344::<<<< <<r3rs @@@@@r'_match_windows_for_appz'CuaDriverBackend._match_windows_for_appMsIIKK%%''  I           >>++LL    LLLcSV W W WLLLLLL   #uu !$ # & &Ggt,,  I0F0F%0O0Okk%((G'4((  7S#J0O0O  'll    axxXG G##s####====W===== &  %%%KKKKWKKK  "! !         "! !OOOOwOOO  $# #         s*A B "BB "D22 D?>D?somNrprrrc | #||_||||dSt|}t|}||||dS|pd||ddddg}nP |}n##t$r|wxYw|s||S|r|p|rn|tvrAd6d ##fd |D}|s||d|dSt|d}n:|8|6|r4| ||} | s||d|dS| }td|D|d} | d|_ | d|_ i|_ | d} |s|js| |_|j |j d|_d} d} g}dx}}d}|dkr|jdp |jj }d}|r;|d|j dd|jd}t+|\} } | sd}||d|j |j |jd}t+|\} } t-|d t.r|d nd}t1|\}}t3jd!|}|r|d"}| st8d#|j  |jd|j |j |jdd$}|d%d&ur|n%|d'r|d'd} d(} n.#t$r&}t8 d)|Yd}~nd}~wwxYwn|d|j |j |jd}d7d+}||rt8d,|j |j  |jd|j |j |jdd$}|d%d&ur|n ||s|}n2#t$r%}t8 d-|Yd}~nd}~wwxYwt-|d t.r|d nd}t1|\}}t3jd.|}|d/pid0}t-|tBr|rtE|}n|rtG|ng}d1|D|_ t+|\} } t3jd!|}|r|d"}d}| rf tIj%| d2} tM| }tO| \}!}"|!r|"r|!}|"}n%#t$rtM| d3zd4z}YnwxYwtQ|||| || ||| 5 S)8uCapture the frontmost on-screen window or an exact known target. Maps hermes `capture(mode, app)` → cua-driver `list_windows` + `get_window_state` (ax/som) or `screenshot` (vision). Nz3z?rCFrrrrorrc|ddd|ddtfdtDS)NrrC rc3 K|]}|vV dSrHrd)rKrhaystacks r'rMzGCuaDriverBackend.capture.._is_desktop_window..s(NN48+NNNNNNr4)r#rrw_DESKTOP_WINDOW_NAMES)rrOs @r'_is_desktop_windowz4CuaDriverBackend.capture.._is_desktop_windowsbeeJ33JJaeeGR6H6HJJPPRRNNNN8MNNNNNNr4c*g|]}| |Srdrd)rKrrQs r'rz,CuaDriverBackend.capture..s*CCCQ-?-?-B-BCqCCCr4z'cBtfddDrdndS)Nc3K|]G}|dddddvVHdS)rrCrMrN)r#r)rKrrs r'rMz=CuaDriverBackend.capture....sr''AEE*b11HHAEE'24F4FHHNNPPP''''''r4)rrrrrrr)rwr#s`r'r$z*CuaDriverBackend.capture..sH3''''['''$$aar4)rz!c3*K|]}|d |VdS)rNrd)rKrs r'rMz+CuaDriverBackend.capture..s+AAQ<AqAAAAAAr4rrrrrvision screenshotjpegU)rformatqualityrGr)rrrGrnzAXWindow\s+"([^"]+)"rzbcua-driver vision capture returned no image over MCP (window_id=%s); re-fetching via CLI transportrrTrz image/pngz8cua-driver CLI re-fetch for vision screenshot failed: %sr\c|drdS|dpi}|ds|drdSt|dtr|dnd}t|pd\}}|o| S)NrFrrrrnrC)r#rJr8rrU)r\sc_txt_trs r' _gws_is_emptyz/CuaDriverBackend.capture.._gws_is_emptys778$$! 5gg1228b77:&&!#''2F*G*G! 5)3CGGFOOS)I)IQcggfooor(332-288::..r4zrcua-driver get_window_state returned an empty result over MCP (pid=%s window_id=%s); re-fetching via CLI transportz7cua-driver CLI re-fetch for get_window_state failed: %sz(\d+)\s+elements?rrc6i|]}|j |j|jSrd)rr)rKrIs r' z,CuaDriverBackend.capture..s6%%%?%%%%r4)validaterr) rrrrrrrrimage_mime_type)rrorr)r\rorr))rrr*r$rrUr_SCREEN_CAPTURE_SENTINELSsortedrInextrrrrrr)r{r}rrrrJr8rrsearchrr~rlrr#rmrRrrr b64decoderrr )$rrrrr target_pidtarget_window_idrrfilteredrrrrfrrrruse_screenshotsc_outgws_outrAr`rwtr)cli_excrbrr sc_elementsrrdetected_widthdetected_heightrQs$ @r'capturezCuaDriverBackend.capturesy, ?i3{i/++O's++J,Y77 !%5%=++[ I2!-# GG ,,..   ))+++  2++D111 ;9,,9J9J9L9LPi9i9i O O O ODCCC'CCCG ++J#JJJ   GG[Y.3.227C@@H ++>*#'#3)-)?'+'7 G{{9--55113333 X..6")("3A"6*5 LLRT[1 <--"+!%!7#/G" / / / /}W%% K$d&< "m>>*#'#3)-)?'+'7 G{{9--55113333*]733*") LLQSZ '1#&F&FN76??BD,T22MGT .88A#;;':;;ArFFzRRK+t,, K K:;GG>BJ4T:::%%!%%%D !(?w'G'G $G_2D99B +!xx{{   6 6&w??? #C 2Ns2S2S/!-o-*E,F 6 6 6 #G q 0A 5  6%'+    sP&A;; B BN N>N99N>"A(R R:R55R:?XX#"X#leftr)elementrrbutton click_countrryrrrzr{rrOptional[List[str]]r c|j}|tdddS|pd}|dvrtddd|dS|d krd nd} ||d } |)|jtd| d S|| d <|j| d<nD|0|.|jtd| dS|| d<|| d<|j| d<ntd| dS|r|| d<|| | S)NFclick*No active window — call capture() first.okactionrrx>rxrightmiddlezunknown button u" — expected left, right, middle.r double_click)rrzz,No active window_id for element_index click.rrz)No active window_id for coordinate click.rrzclick requires element= or x/y.modifier)rr rr_action) rryrrrzr{rr button_normrWrEs r'r~zCuaDriverBackend.clicks ;5(TVVV V'..00 9 9 95(f&(f(f(fhhh h!,!1!1~~w'*kBB  %-#uT,Z\\\\$+D ! $ 6D   ]q}%-#uT,WYYYYDIDI $ 6D  5(IKKK K  )(D ||D$'''r4) from_element to_elementfrom_xyto_xyrzrrrrOptional[Tuple[int, int]]rc|j}|tdddSd|i}|0|.|jtdddS||d<||d<|j|d <n|||jtddd St|d t|d c|d <|d<t|d t|d c|d<|d<|j|d <ntdddS|d|S)NFdragrrrz+No active window_id for element-based drag.rrrz(No active window_id for coordinate drag.rrfrom_xfrom_yto_xto_yzGdrag requires from_element/to_element or from_coordinate/to_coordinate.)rr rrr) rrrrrrzrrrEs r'rzCuaDriverBackend.dragsb ;5(TVVV V %s|  # (>%-#uV,Y[[[[#/D !+D  $ 6D    U%6%-#uV,VXXXX-0__c'!*oo *DNDN),U1XE!H &DL$v, $ 6D  5(qsss s||FD)))r4r)amountryrrr directionrc t|j}|tdddS||tdtd|d}||j||d<|j|d <nM|K|I|jtddd S|jd d r ||d <||d<|j|d <|d|S)NFscrollrrr2)rrrrrz*No active window_id for coordinate scroll.zinput.scroll.coordinatesrWrr)rr rrrr)ryr) rrrryrrrrrEs r'rzCuaDriverBackend.scroll>s ;5(TVVV V"!SV__--    4#9#E$+D ! $ 6D   ]q}%-#uX,XZZZZ}00*1 S S $ 6D ||Hd+++r4rAc~|j}|j}||tdddS|d|||dS)NF type_textrr)rrrArrr r)rrArrs r'rzCuaDriverBackend.type_textgs]* ;)+5(TVVV V||K9VZ)[)[\\\r4rc|j}|j}||tdddSt|\}}|stddd|dS|r|d||||gzdS|d |||d S) NFrrrzCould not parse key from ''.hotkey)rrr press_key)rrr)rrr rr)rrrrkey_namers r'rzCuaDriverBackend.keyos* ;)+5(TVVV V/t44) O5(MT(M(M(MOOO O  @<<#I3<z3I+K+KLL L<< Sy5=.?.?@@ @r4rc|j}|j}||tdddS|tdddS||||d}|d|S)zBSet a value on an element. Handles AXPopUpButton selects natively.NF set_valuerrz,set_value requires element= (element index).)rrrrr)rrryrrrEs r'rzCuaDriverBackend.set_values* ;)+5(TVVV V ?5(VXXX X"$    ||K...r4c (|jdd|ji}|d}t |t r0t |dt r|dS|d}t |t r|St |t r0t |dt r|dSt |trg}|D]v}tj d|}|r]| | d t| dd w|SgS) Nr@rGrappsrnz(.+?)\s+\(pid\s+(\d+)\)rr)rr)r)rqrr#rJr.rRr8 splitlinesrrjrrrUr)rr\rrnrliners r'r@zCuaDriverBackend.list_appssgm%%kIt?O3PQQWW011 j$ ' ' &Jz~~f7M7Mt,T,T &f% %wwv dD ! ! K dD ! ! j&1A1A4&H&H <  dC  D)) V VI8$??VKK)9)9););CPQ OO T TUUUK r4c*|SrH)r*rs r'rzCuaDriverBackend.list_windowss!!###r4F raise_windowc  |}n##t$r|wxYw|||}|r|dnd}|rn|d|_|d|_i|_|d|_|j|jd|_tddd |dd |jd |jd  S|tddd|d S)u`Target an app for subsequent actions without stealing system focus. cua-driver background-automation never needs to bring a window to the front: capture(app=...) already selects the right window via list_windows. We implement focus_app as a pure window-selector — enumerate on-screen windows, find the best match for *app*, and store its pid/window_id so that subsequent click/type calls hit the right process. raise_window=True is intentionally ignored: stealing the user's focus is exactly what this backend is designed to avoid. rNrrrrVT focus_appz Targeted z (pid z , window z) without raising window.rFz#No on-screen window found for app 'r) r*r$rrIrrrrrr )rrrrmatchedrs r'rzCuaDriverBackend.focus_appsf ((**GG     % % ' ' '  --gs;; '0D  %e}D %+K%8D "$&D !#J/DN'!3!!D   TF:$6TTd>NTT"&"8TTT  !!###u[$Q#$Q$Q$QSSS Ss 7)r4rurlsadditional_arguments creates_new_application_instancer4rrrc|s|stdd|ji}|r||d<|r||d<|rt||d<|rt||d<|rd|d<|jd |}|d p d |d iS) uIdempotent launch. Returns ``{pid, bundle_id, name, windows[]}`` so callers can skip an extra ``list_windows`` round-trip before ``get_window_state``. ``creates_new_application_instance=True`` forces a new instance even if the app is already running — use it when concurrent runs may touch the same app so each session gets its own isolated window.z,launch_app requires either bundle_id or namerGr4rrrTr launch_apprrn)rYrrRr)rq)rr4rrrrrEr\s r'rzCuaDriverBackend.launch_apps" M MKLL L )4+;<  * )D   DL  &::DL  F+/0D+E+ED' ( + <7;D3 4m%%lD99&'@FCK+@@r4cL|ddt|iS)zYTerminate by pid. Equivalent to ``kill -9`` on POSIX, ``taskkill /F`` on Windows.kill_apprrr)rrs r'rzCuaDriverBackend.kill_apps#||JC(9:::r4)rcxdt|i}|t||d<|d|S)zActivate a window so subsequent foreground-dispatched input lands on it. cua-driver's docstring notes this is the cheaper path than per-call SetForegroundWindow flashes.rNrbring_to_front)rr)rrrrEs r'rzCuaDriverBackend.bring_to_fronts? !&s3xx0  #ID ||,d333r4ch|dt|t|dS)uHMove the agent-cursor *overlay* to a screen point. This is a visual hint — it does NOT move the real OS pointer (cua-driver explicitly avoids stealing pointer focus). The overlay glides smoothly to the target, so consumers use it before a click to give a visible "where the agent is going" cue. move_cursor)rrr)rrrs r'rzCuaDriverBackend.move_cursors, ||MQc!ff+E+EFFFr4rc|jdd|ji}|dpi}t |ddt |ddfS)zPReturn the *real* OS cursor position in screen points (origin top-left).get_cursor_positionrGrrrr)r)rqrr#r)rr\scs r'rz$CuaDriverBackend.get_cursor_positionstm%% !It/?#@  WW( ) ) /R266#q>>""CsA$7$777r4ct|jdd|ji}|dpiS)zReturn the logical size of the main display in points plus its backing scale factor. Shape: ``{width, height, backing_scale_factor}``.get_screen_sizerGrr)rqrr#rr\s r'rz CuaDriverBackend.get_screen_size$sBm%%  4+;<  ww*++1r1r4g?rYrZ)factorr[r\r9rrrr[r\c  |jdt|t|t|t|t|t||t||jd S)zReturn a JPEG / PNG of a sub-region of a window, optionally scaled. cua-driver supports zoom-to-rect for callers that need a higher-resolution view of a specific element.zoom) rrrrrrr[r\rG)r)rqrr9r) rrrrrrrr[r\s r'rzCuaDriverBackend.zoom-sn }&&vYqauQxxeAhhFmmW' 0 0    r4) cursor_idenabledrc^dt|i}|r||d<|d|S)z:Toggle the agent cursor overlay's visibility for this run.rrset_agent_cursor_enabled)rr)rrrrEs r'rz)CuaDriverBackend.set_agent_cursor_enabledCs;!*4==9  * )D ||6===r4)glide_msdwell_ms idle_hide_msrrrrrci}|t||d<|t||d<|t||d<|r||d<|d|S)uTune the overlay's motion timings — glide duration, post-click dwell, idle-hide delay. Each None means "leave at current value".Nrrrrset_agent_cursor_motion)r9r)rrrrrrEs r'rz(CuaDriverBackend.set_agent_cursor_motionKst "  $XD   $XD   ##(#6#6D  * )D ||5t<<EEww*++1r1r4) record_video output_dirrc|jd|t||jd}|dpiS)a;Enable trajectory recording (per-turn screenshots + action JSON) to ``output_dir``. ``record_video=True`` ALSO captures the main display to ``/recording.mp4`` (H.264). Recording ownership is keyed by this run's session id so concurrent runs don't fight over the recorder.start_recording)rrrGr)r)rqrrr#)rrrr\s r'rz CuaDriverBackend.start_recording}sVm%%&7$ ..': :    ww*++1r1r4ct|jdd|ji}|dpiS)zDisable recording and finalise the mp4 (if video was on). Returns the recorder's final state including ``last_video_path``.stop_recordingrGrrrs r'rzCuaDriverBackend.stop_recordingsEm%%&6 t'9   ww*++1r1r4ct|jdd|ji}|dpiS)zReturn the current recorder state without changing it. Shape: ``{recording, enabled, output_dir, next_turn, last_video_path, last_error, owner, video_active}``.get_recording_staterGrrrs r'rz$CuaDriverBackend.get_recording_statesBm%% !It/?#@  ww*++1r1r4)dry_run speed_factortrajectory_dirrrc|jd|t|t||jdS)zReplay a prior recording's turn stream by re-invoking each turn's tool call in lexical order. ``dry_run=True`` logs without actually firing the tools.replay_trajectory)rrrrG)r)rqrr9r)rrrrs r'rz"CuaDriverBackend.replay_trajectorysH }&&':,G}}!,//' = =    r4cF|jdd|jiS)zBootstrap ffmpeg for ``start_recording(record_video=True)`` on Linux / Windows. macOS records natively via ScreenCaptureKit and doesn't need ffmpeg.install_ffmpegrG)r)rqrrs r'rzCuaDriverBackend.install_ffmpegs+}&& y$*:;   r4ct|jdd|ji}|dpiS)z-Return the current cua-driver runtime config. get_configrGrrrs r'rzCuaDriverBackend.get_configs@m%% 9d&67  ww*++1r1r4c H|dt|S)uSet cua-driver config keys. Common keys include ``max_image_dimension`` (image-output resizing), recording flags, etc. Unknown keys are passed through verbatim — cua-driver validates against its own schema. set_config)rr.)rconfigs r'rzCuaDriverBackend.set_configs ||L$v,,777r4c|jdd|ji}|dp d|diS)a.Return a lightweight snapshot of running regular apps + on-screen visible windows with bounds, z-order, owner pid. Roughly the data ``list_windows`` exposes, in one call. Most callers should prefer ``capture()`` / ``focus_app()`` which already use this shape internally.get_accessibility_treerGrrnrrs r'rz'CuaDriverBackend.get_accessibility_treesK m%% $y$2B&C  ww*++DF /DDr4r page_argsrc t|||jd}|||jd|S)azInteract with a browser page loaded in a running app (Chrome, Safari, Edge, ...). cua-driver routes through CDP / Apple Events / AX tree depending on the target. ``action`` + ``page_args`` shape depends on the requested operation (e.g. ``action="eval"`` takes ``js: str``); see cua-driver's ``page`` tool description for the full grammar.)rrrGpage)rrupdater)rq)rrrrrEs r'rzCuaDriverBackend.pagesNs88'   I}&&vt444r4rr5rkr6c|rt|ni}|d|j|j|||S)uCall any cua-driver MCP tool by name with arbitrary args. ``session`` is injected (preserves the caller's explicit one via setdefault). For tools the wrapper doesn't already type- wrap, this is the supported escape hatch — preferred over reaching for ``self._session.call_tool`` directly because it keeps the session-id contract consistent with everything else.rGr5)r. setdefaultrr)rq)rrrEr6payloads r'rqzCuaDriverBackend.call_toolsN!%,$t***"9d&6777}&&tWg&FFFr4rWc|d}t|tsdS|j|}|sdS|jd|sdS||d<dS)aSurface 6: when the wrapper is about to call a token-capable tool with `element_index`, look up the matching `element_token` from the last snapshot and attach it. cua-driver-rs's contract for combined args is documented in trycua/cua#1961: "element_token takes precedence over element_index when both supplied. Returns an explicit 'stale' error if the snapshot has been superseded." Gated on the per-tool capability claim so we don't send the field to drivers that predate the surface (which would reject the schema with `additionalProperties: false`). rNzaccessibility.element_tokensrr)r#rJrrr)ry)rrWrErrs r'_maybe_attach_element_tokenz,CuaDriverBackend._maybe_attach_element_tokenshh''#s##  F%))#..  F}00 *1    F %_r4c F||||d|j |j||}nG#t $r:}t d|td|d|cYd}~Sd}~wwxYw|d }d}|d}t|tr$t| d d}nt|tr|}t|||t|tr|ni S) NrGzcua-driver %s call failedFzcua-driver error: rrrCrnr)rrrmeta) rrrr)rqr$r~ exceptionr rJr.r8r#)rrrEr\rIrrrns r'rzCuaDriverBackend._action sS ((t444  4#3444 Y-))$55CC Y Y Y   8$ ? ? ?5?WTU?W?WXXX X X X X X X YY 6{ dD ! ! $((9b1122GG c " " Gr$)3D$)?)?!GRIII IsA B/BBBrr)rC)rr8rr8rr r)rr)rrrr8rr)rJNNN) rr8rrprrrrrr )ryrrrrrrzr8r{rrr|rr )rrrrrrrrrzr8rr|rr )rr8rrryrrrrrrr|rr )rAr8rr )rr8rr rH)rr8ryrrr )F)rr8rrrr ) r4rprrprr|rr|rrrro)rrrr )rrrrrr )rrrrrr )rr)rro)rrrr9rr9rr9rr9rr9r[r8r\rrro)rrrrprr ) rrrrrrrrprr ) rr|rrprrprrprr )rrprro)rr8rrrro)rr8rrrr9rro)rr )rrrr8rrrro)rr8rErkr6r9rro)rWr8rErorr{)rr8rErorr ).r r!r"r#rrrrrrrr*rIrwr~rrrrrr@rrrrrrrrrrrrrrrrrrrrrrrqrrrdr4r'rrsJJ#B#B#B#BLZZZZ@    ----####       ((((TM M M M d!!#' I I I I I ^ "&)-1(1(1(1(1(1(l'+$(-1+/)- * * * * * *L!%)-&,&,&,&,&,&,R]]]]@@@@*/////&,$$$$)S)S)S)S)Sh$("$(4816AAAAAAB;;;; 37444444GGGG88882222!       .=A>>>>>>=A<@@D;? ======&GK<@;?:> <<<<<<*;?222222.3 2 2 2 2 2 222222222+003           22228888 E E E E5555$ G&* G G G G G G&&&&4IIIIIIr4rrrH)r)r*rr+)r7r8r6r9rr:)r6r9rrk)rrprr)rr8rr)rrrr)rrrr)rr8rr)rr8rr)rrrro)r\rorr)rrrr)rrrr)>r# __future__rrrconcurrent.futuresrirWloggingr/rrgrOrbrrtypingrrrrrtools.computer_use.backendr r r r getLoggerr r~r0r#rirSrgrPr1r(r3r^compile MULTILINErrerjrorzrrrrrrrrrr%rrrrrrrdr4r'rs!!F#"""""     33333333333333  8 $ $"*..!8,GG7VUU;&     $333333P2:.L   $$$$//// 14&&&&&&R    **82222j$$$$N2////////l^X^X^X^X^X^X^X^XB5555p$$$$N****%%%%XDIDIDIDIDI)DIDIDIDIDIr4