Add background computer-use tool contract
Expose native window listing, indexed window state, and element_action dispatch modes through the Agent Zero connector tool. Update host computer-use prompts and platform skills so agents prefer background structural targeting and only rely on screenshots for foreground or uncertain actions.
Alessandro committed
May 30, 2026 at 21:48 UTC
3c1bdfa5f91e43f80a65f45aef492ea04a7aac4a
6 files changed
+199
-14
plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md
+8
-4
@@ -2,17 +2,17 @@
2
3
Runtime-gated beta desktop control through a connected A0 CLI on the user's host machine. The callable contract is available in the tool prompt. Availability, backend support, and trust mode are checked when the tool runs, together with CLI presence, local enablement, and re-arm state. Computer Use enablement is scoped to the current CLI session, not scoped to a single chat context.
4
5
-Use this for native host desktop UI inspection, screenshots, clicking, scrolling, typing, key presses, and status checks. Do not use it for ordinary web-page navigation or host-browser control; use the browser tool for web pages unless browser automation cannot express the task. For complex desktop workflows, load and follow skill `host-computer-use` before proceeding.
5
+Use this for native host desktop UI inspection, screenshots, background-safe window/element actions when supported, clicking, scrolling, typing, key presses, and status checks. Do not use it for ordinary web-page navigation or host-browser control; use the browser tool for web pages unless browser automation cannot express the task. For complex desktop workflows, load and follow skill `host-computer-use` before proceeding.
6
7
This is the only desktop-control path for the user's connected host/local computer. Do not substitute the `linux-desktop` skill, the Agent Zero Desktop/Xpra surface, `desktopctl.sh`, `code_execution_tool`, or Docker/server shell commands for host screen actions; those target the internal Agent Zero runtime and cannot see or control the user's host screen.
8
9
If the tool reports no CLI, disabled computer use, or `COMPUTER_USE_REARM_REQUIRED`, stop and tell the user to run `/computer-use on` in A0 CLI and approve any host permission prompt.
10
11
-Call `start_session` before screen-driven tasks. Use `status` for state only, `capture` for screenshots without an action, and `stop_session` when the desktop task is complete. Interactive coordinate actions should use normalized global-screen coordinates from the most recent capture.
11
+Call `start_session` before screen-driven tasks. Use `status` for state only, `capture` for screenshots without an action, and `stop_session` when the desktop task is complete. When the backend advertises native window and element-index features, prefer `list_windows` -> `get_window_state` -> `element_action` with `dispatch: "background"` before using global coordinates. Interactive coordinate actions should use normalized global-screen coordinates from the most recent capture.
12
13
Some actions are backend-specific and intentionally documented only in backend skills. If `status` or `start_session` reports backend-specific features or tells you to load a backend skill, load and follow that skill before using those backend-only actions. For structural targeting details, load and follow the backend-specific skill such as `host-computer-use-macos` or `host-computer-use-windows`; do not apply one backend's guidance to another backend.
14
15
-State-changing actions automatically attach a fresh screen after they run. Treat key presses, clicks, scrolling, and typing as attempts, not success: inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. A `type` result only proves keystrokes were sent; it does not prove that text landed in the intended place.
15
+State-changing actions automatically attach a fresh screen after they run unless the backend returns a definitive structural background result. Treat key presses, clicks, scrolling, and typing as attempts, not success; treat foreground fallbacks the same way. Inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. A `type` result only proves keystrokes were sent; it does not prove that text landed in the intended place.
16
17
```json
18
{
@@ -24,10 +24,14 @@ State-changing actions automatically attach a fresh screen after they run. Treat
24
```
25
26
Required argument:
27
-- `action`: one of `start_session`, `status`, `capture`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`; backend skills may document additional backend-only action values
27
+- `action`: one of `start_session`, `status`, `capture`, `list_windows`, `get_window_state`, `element_action`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`; backend skills may document additional backend-only action values
28
29
Optional arguments by action:
30
- `session_id`: session returned by `start_session`
31
+- `pid`, `window_id`: target a native app/window for `get_window_state` and `element_action`
32
+- `element_index`: target an element from the latest `get_window_state`
33
+- `operation`: action such as `invoke`, `press`, `set_value`, `focus`, or backend-specific operations
34
+- `dispatch`: `background`, `auto`, or `foreground`; prefer `background` for `element_action`
35
- `x`, `y`: normalized `[0,1]` global-screen coordinates for `move` and `click`
36
- `button`: `left`, `right`, or `middle` for `click`
37
- `count`: click count for `click`
plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md
+4
@@ -20,6 +20,8 @@ Linux backends can advertise structural AT-SPI features:
20
21
When these features are present, prefer structural targeting over pixel clicks for named buttons, menu items, text fields, dialogs, toolbar items, tab strips, and application windows.
22
23
+If the backend also advertises `native-window-list`, `window-state`, `element-index-targeting`, or `background-dispatch`, prefer the generic background loop from `host-computer-use`: `list_windows` -> `get_window_state` -> `element_action`. If those features are absent, use the AT-SPI snapshot/action flow below.
24
+
25
Use `ax_snapshot` to inspect the Linux AT-SPI tree:
26
27
```json
@@ -68,6 +70,8 @@ Targeting options:
70
71
Use screenshots for proof after every state-changing action. AT-SPI actions and keyboard events are attempts, not proof, and Wayland focus can reject or redirect input when the active window changes.
72
73
+True background dispatch on Linux is compositor, toolkit, and app dependent. Do not claim a Linux action was background-safe unless the tool result explicitly says `actual_dispatch=background`.
74
+
75
On GNOME/Wayland, useful shortcuts include:
76
77
- `Super+H`: hide the active window
plugins/_a0_connector/skills/host-computer-use-macos/SKILL.md
+4
@@ -19,6 +19,8 @@ macOS backends can advertise structural Accessibility features:
19
20
When these features are present, prefer structural targeting over pixel clicks for named controls such as buttons, menu items, text fields, sheets, alerts, toolbar items, and sidebar rows.
21
22
+If the backend also advertises `native-window-list`, `window-state`, `element-index-targeting`, or `background-dispatch`, prefer the generic background loop from `host-computer-use`: `list_windows` -> `get_window_state` -> `element_action`. If those features are absent, use the AX snapshot/action flow below.
23
+
24
Use `ax_snapshot` to inspect the frontmost app's bounded Accessibility tree:
25
26
```json
@@ -65,6 +67,8 @@ Targeting options:
67
68
AX actions are attempts, not proof. They attach a fresh screenshot after state-changing actions; inspect that image before saying the requested outcome happened.
69
70
+Do not assume macOS work happened in the background unless the tool result explicitly says `actual_dispatch=background`. If the result says `background_unavailable`, use foreground dispatch only when foreground control is acceptable.
71
+
72
## macOS Window Actions
73
74
For active-app window tasks, macOS shortcuts are usually:
plugins/_a0_connector/skills/host-computer-use-windows/SKILL.md
+30
-2
@@ -17,10 +17,37 @@ Windows backends can advertise structural UI Automation features:
17
- `uia-structural-targeting`
18
- `uia-element-action`
19
- `uia-window-management`
20
+- `native-window-list`
21
+- `window-state`
22
+- `element-index-targeting`
23
+- `background-dispatch`
24
+- `foreground-dispatch-fallback`
25
21
-When these features are present, prefer structural targeting over pixel clicks for named controls such as buttons, menu items, text fields, dialogs, toolbar items, browser address bars, composer fields, and list rows.
26
+When these features are present, prefer background structural targeting over pixel clicks for named controls such as buttons, menu items, text fields, dialogs, toolbar items, browser address bars, composer fields, and list rows.
27
23
-Use `uia_snapshot` to inspect the bounded Windows UI Automation tree:
28
+Preferred loop:
29
+
30
+1. Use `list_windows` to find the target native window.
31
+2. Use `get_window_state` with the target `pid` and/or `window_id`.
32
+3. Use `element_action` with the returned `element_index` and `dispatch: "background"`.
33
+4. If the backend reports `background_unavailable`, switch to `dispatch: "auto"` or `dispatch: "foreground"` only when foreground control is acceptable.
34
+
35
+Example:
36
+
37
+```json
38
+{
39
+ "tool_name": "computer_use_remote",
40
+ "tool_args": {
41
+ "action": "element_action",
42
+ "window_id": "uia-hwnd:123456",
43
+ "element_index": 7,
44
+ "operation": "invoke",
45
+ "dispatch": "background"
46
+ }
47
+}
48
+```
49
+
50
+Use `uia_snapshot` to inspect the bounded Windows UI Automation tree when the newer window-state loop is not available:
51
52
```json
53
{
@@ -72,6 +99,7 @@ Targeting options:
99
Action selection:
100
101
- Prefer the actions listed on the target node. If a node offers `invoke`, use `invoke`, not `click`.
102
+- Prefer `element_action` over `uia_action` when the backend advertises `element-index-targeting`; it preserves the background-first dispatch contract.
103
- For window focus, hiding, restoring, or maximizing, use `focus_window`, `minimize`, `restore`, or `maximize`; do not click titlebar buttons.
104
- For typing into an app, first structurally focus the app/window if needed, then `set_value` on the target field. A global `type` result only proves keys were sent, not that they landed in the intended control.
105
- After a window operation, navigation, menu open/close, dialog transition, or other layout change, take a fresh `uia_snapshot` before reusing a path.
plugins/_a0_connector/skills/host-computer-use/SKILL.md
+15
-6
@@ -17,7 +17,7 @@ triggers:
17
18
# Host Computer Use
19
20
-This skill unlocks the beta `computer_use_remote` tool for connected local desktop control through A0 CLI.
20
+This skill unlocks the beta `computer_use_remote` tool for connected local desktop control through A0 CLI. Prefer native background-safe computer use when the connected backend advertises window and element-index features.
21
22
## When to Use
23
@@ -55,9 +55,13 @@ Use:
55
56
Arguments:
57
58
-- `action`: `start_session`, `status`, `capture`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`
58
+- `action`: `start_session`, `status`, `capture`, `list_windows`, `get_window_state`, `element_action`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`
59
- `session_id`: optional after `start_session`
60
- backend skills may document additional backend-only action values; use them only when backend metadata advertises matching support and after loading the backend-specific skill
61
+- `list_windows`: returns native top-level window records when the backend supports them
62
+- `get_window_state`: pass `pid` and/or `window_id`; returns a target-window accessibility tree with stable `element_index` values for the current state
63
+- `element_action`: pass `element_index` from the latest `get_window_state`; optional `operation`, `value`/`text`, and `dispatch`
64
+- `dispatch`: `background`, `auto`, or `foreground`; default to `background` for element actions
65
- `move`: `x`, `y` normalized to `[0,1]`
66
- `click`: optional `x`, `y`, optional `button` (`left`, `right`, `middle`), optional `count`
67
- `scroll`: `dx`, `dy`
@@ -72,10 +76,13 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
76
77
1. Call `start_session` first.
78
2. Read the returned `backend_id`, `backend_family`, and `features`; load a backend-specific Computer Use skill when the task needs backend-only affordances.
75
-3. Decide final success from the latest screenshot, not from memory.
76
-4. Interactive actions already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.
77
-5. Use `status` for state without starting a session.
78
-6. Use `capture` only when you need another screenshot without taking an action.
79
+3. If the backend advertises `native-window-list`, call `list_windows` before using coordinates.
80
+4. If the backend advertises `window-state` and `element-index-targeting`, call `get_window_state` for the target `pid`/`window_id`, then use `element_action` with `dispatch: "background"` by default.
81
+5. If `element_action` reports `background_unavailable`, use `dispatch: "auto"` or `dispatch: "foreground"` only when foreground control is acceptable for the user/task.
82
+6. Decide final success from the latest screenshot or a definitive structural result, not from memory.
83
+7. Interactive actions already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.
84
+8. Use `status` for state without starting a session.
85
+9. Use `capture` only when you need another screenshot without taking an action.
86
87
## Backend Skills
88
@@ -88,7 +95,9 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
95
96
- Only the latest screenshot or a definitive tool result counts as evidence.
97
- If a tool result says a screenshot was attached but you cannot actually see the image, stop and report that visual verification is unavailable. Do not continue with another action from an assumed host state.
98
+- Prefer the background-safe native loop when advertised: `list_windows` -> `get_window_state` -> `element_action`.
99
- Outside advertised structural accessibility support, use normalized global screen coordinates; do not assume window ids, element indexes, background-safe input, or semantic click targets unless the runtime explicitly advertises them.
100
+- Treat `dispatch: "foreground"` as intentional control of the user's visible desktop. Use it only after deciding that a background action is unavailable or unsuitable.
101
- On Linux, AT-SPI structural targeting uses backend-specific actions documented in `host-computer-use-linux`; do not apply macOS AX-specific assumptions unless the backend is macOS.
102
- Prefer accessibility and semantic UI paths first: shortcuts, command palettes, menu accelerators, address/search bars, focus traversal, and other keyboard-accessible controls.
103
- Prefer `key` and `type` over pointer actions whenever a reliable keyboard path exists.
plugins/_a0_connector/tools/computer_use_remote.py
+138
-2
@@ -34,6 +34,7 @@ REARM_REQUIRED_DEFAULT_MESSAGE = (
34
)
35
_AUTO_CAPTURE_ACTIONS = {
36
"start_session",
37
+ "element_action",
38
"ax_action",
39
"uia_action",
40
"move",
@@ -58,6 +59,9 @@ _SUPPORTED_ACTIONS = {
59
"start_session",
60
"status",
61
"capture",
62
+ "list_windows",
63
+ "get_window_state",
64
+ "element_action",
65
"ax_snapshot",
66
"ax_action",
67
"uia_snapshot",
@@ -80,7 +84,8 @@ class ComputerUseRemote(Tool):
84
return Response(
85
message=(
86
"action is required and must be one of: "
83
- "start_session, status, capture, ax_snapshot, ax_action, "
87
+ "start_session, status, capture, list_windows, get_window_state, "
88
+ "element_action, ax_snapshot, ax_action, "
89
"uia_snapshot, uia_action, "
90
"move, click, scroll, key, type, stop_session"
91
),
@@ -226,6 +231,11 @@ class ComputerUseRemote(Tool):
231
232
data = result.get("result")
233
result_data = dict(data) if isinstance(data, dict) else {}
234
+ if action == "element_action":
235
+ actual_dispatch = str(result_data.get("actual_dispatch") or "").strip().lower()
236
+ if actual_dispatch in {"background", "none"}:
237
+ return ""
238
+
239
session_id = str(result_data.get("session_id") or self.args.get("session_id") or "").strip()
240
if not session_id:
241
return ""
@@ -324,6 +334,34 @@ class ComputerUseRemote(Tool):
334
payload["text"] = self.args.get("text", "")
335
if self._coerce_bool(self.args.get("submit")):
336
payload["submit"] = True
337
+ elif action == "list_windows":
338
+ for key in ("include_hidden", "include_offscreen", "max_windows"):
339
+ if key in self.args:
340
+ payload[key] = self.args.get(key)
341
+ elif action == "get_window_state":
342
+ for key in ("pid", "window_id", "mode", "max_depth", "max_nodes"):
343
+ if key in self.args:
344
+ payload[key] = self.args.get(key)
345
+ elif action == "element_action":
346
+ for key in (
347
+ "pid",
348
+ "window_id",
349
+ "element_index",
350
+ "path",
351
+ "operation",
352
+ "name",
353
+ "dispatch",
354
+ "value",
355
+ "text",
356
+ "submit",
357
+ ):
358
+ if key in self.args:
359
+ payload[key] = self.args.get(key)
360
+ target = self.args.get("target")
361
+ if isinstance(target, dict):
362
+ payload["target"] = dict(target)
363
+ if "selector" in self.args:
364
+ payload["selector"] = self.args.get("selector")
365
elif action == "ax_snapshot":
366
if "max_depth" in self.args:
367
payload["max_depth"] = self._coerce_int(self.args.get("max_depth"), name="max_depth")
@@ -386,6 +424,12 @@ class ComputerUseRemote(Tool):
424
if action == "capture":
425
summary = self._record_capture(data)
426
return f"Current screen attached: {summary} {CAPTURE_VERIFICATION_NOTE}"
427
+ if action == "list_windows":
428
+ return self._format_window_list(data)
429
+ if action == "get_window_state":
430
+ return self._format_window_state(data)
431
+ if action == "element_action":
432
+ return self._format_element_action(data)
433
if action == "ax_snapshot":
434
return self._format_ax_snapshot(data)
435
if action == "ax_action":
@@ -552,6 +596,94 @@ class ComputerUseRemote(Tool):
596
f"active_contexts={active_text}.{rearm_guidance}"
597
)
598
599
+ def _format_window_list(self, data: dict[str, Any]) -> str:
600
+ windows = data.get("windows") if isinstance(data.get("windows"), list) else []
601
+ count = data.get("count", len(windows))
602
+ if not windows:
603
+ return (
604
+ "No native windows were returned by the computer-use backend. "
605
+ "Use capture or backend-specific snapshots as a fallback."
606
+ )
607
+ lines = [
608
+ (
609
+ f"Computer-use native window list: {count} window(s). "
610
+ "Prefer get_window_state(pid/window_id) before element_action."
611
+ )
612
+ ]
613
+ for item in windows[:40]:
614
+ if not isinstance(item, dict):
615
+ continue
616
+ title = str(item.get("title") or item.get("name") or "").strip()
617
+ app_name = str(item.get("app_name") or item.get("app") or "").strip()
618
+ pid = item.get("pid")
619
+ window_id = str(item.get("window_id") or "").strip()
620
+ role = str(item.get("role") or "window").strip()
621
+ parts = [f"- window_id={window_id or '?'}"]
622
+ if pid is not None:
623
+ parts.append(f"pid={pid}")
624
+ if app_name:
625
+ parts.append(f"app={app_name!r}")
626
+ if title:
627
+ parts.append(f"title={title!r}")
628
+ parts.append(f"role={role}")
629
+ frame = item.get("frame")
630
+ if isinstance(frame, dict):
631
+ parts.append(
632
+ f"frame=({frame.get('x', '?')},{frame.get('y', '?')} "
633
+ f"{frame.get('width', '?')}x{frame.get('height', '?')})"
634
+ )
635
+ flags: list[str] = []
636
+ for flag in ("is_on_screen", "on_current_space", "focused", "visible"):
637
+ if flag in item:
638
+ flags.append(f"{flag}={item.get(flag)}")
639
+ if flags:
640
+ parts.append(" ".join(flags))
641
+ lines.append(" ".join(parts))
642
+ if len(windows) > 40:
643
+ lines.append("... window list truncated; request a narrower app/window target.")
644
+ return "\n".join(lines)
645
+
646
+ def _format_window_state(self, data: dict[str, Any]) -> str:
647
+ tree = data.get("tree") if isinstance(data.get("tree"), dict) else {}
648
+ window = data.get("window") if isinstance(data.get("window"), dict) else {}
649
+ app = data.get("app") if isinstance(data.get("app"), dict) else {}
650
+ title = str(window.get("title") or app.get("name") or "target window").strip()
651
+ window_id = str(window.get("window_id") or data.get("window_id") or "").strip()
652
+ node_count = data.get("node_count", "?")
653
+ truncated = " truncated" if data.get("truncated") else ""
654
+ mode = str(data.get("mode") or "auto").strip()
655
+ return (
656
+ f"Window state for {title!r}"
657
+ f"{f' window_id={window_id}' if window_id else ''}: "
658
+ f"{node_count} element(s){truncated}, mode={mode}. "
659
+ "Use element_action with element_index; dispatch defaults to background."
660
+ f"{self._structural_tree_outline(tree)}"
661
+ )
662
+
663
+ def _format_element_action(self, data: dict[str, Any]) -> str:
664
+ target = data.get("target") if isinstance(data.get("target"), dict) else {}
665
+ operation = str(data.get("operation") or "?")
666
+ requested_dispatch = str(data.get("requested_dispatch") or data.get("dispatch") or "background")
667
+ actual_dispatch = str(data.get("actual_dispatch") or data.get("dispatch") or requested_dispatch)
668
+ fallback = bool(data.get("foreground_fallback_used") or data.get("fallback_used"))
669
+ index = target.get("element_index", data.get("element_index", "?"))
670
+ label = (
671
+ self._uia_target_label(target)
672
+ if target.get("selector") or target.get("automation_id")
673
+ else self._ax_target_label(target)
674
+ )
675
+ if data.get("background_unavailable"):
676
+ reason = str(data.get("reason") or "background dispatch was unavailable")
677
+ return (
678
+ f"Background element action unavailable for element_index={index}: {reason}. "
679
+ "Use dispatch='auto' or dispatch='foreground' only if foreground control is acceptable."
680
+ )
681
+ dispatch_text = (
682
+ f"requested_dispatch={requested_dispatch}, actual_dispatch={actual_dispatch}"
683
+ f"{', foreground_fallback_used=true' if fallback else ''}"
684
+ )
685
+ return f"Performed {operation} on element_index={index} {label}; {dispatch_text}."
686
+
687
def _format_ax_snapshot(self, data: dict[str, Any]) -> str:
688
app = data.get("app") if isinstance(data.get("app"), dict) else {}
689
tree = data.get("tree") if isinstance(data.get("tree"), dict) else {}
@@ -611,7 +743,11 @@ class ComputerUseRemote(Tool):
743
indent = " " * max(0, depth)
744
role = str(node.get("role") or "element")
745
path = node.get("path", [])
614
- parts = [f"{indent}- path={path} role={role}"]
746
+ element_index = node.get("element_index")
747
+ prefix = f"{indent}-"
748
+ if element_index is not None:
749
+ prefix = f"{prefix} element_index={element_index}"
750
+ parts = [f"{prefix} path={path} role={role}"]
751
for key in ("title", "name", "description", "automation_id", "class_name", "selector"):
752
value = node.get(key)
753
if isinstance(value, str) and value.strip():