Harden remote Linux computer use targeting
Require verified window focus before target-bound typing and report remote computer-use receipts from observed evidence.\n\nGuide agents through scoped snapshots and focus verification while refusing window presses and identical failed-action retries.
Alessandro committed
Jul 26, 2026 at 00:24 UTC
4cea5d7197b80604ae6d6c8d40e5f5b1fe942631
7 files changed
+143
-18
plugins/_a0_connector/AGENTS.md
+1
@@ -52,6 +52,7 @@
52
after all chunks for the `op_id` are assembled.
53
- Host browser status metadata may advertise `available_browsers` entries with browser ids, labels, CDP endpoints, status, and enabled state; keep older CLI payloads without those fields compatible.
54
- Model preset definitions exposed through v1 are global; project arguments select scope but never create project-owned definitions. Model switcher state reports the effective main, utility, and embedding models and preserves embedding-change notifications.
55
+- Computer Use receipts describe transport success unless the connector returns explicit effect evidence. Linux target-bound typing requires a verified active/focused `window_id`; window activation uses focus, never a press action on an application or window node. Do not retry an identical failed Computer Use call.
56
57
## Work Guidance
58
plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md
+7
-3
@@ -8,11 +8,15 @@ This is the only desktop-control path for the user's connected host/local comput
8
9
If the tool reports no CLI/Launcher host bridge, disabled computer use, or `COMPUTER_USE_REARM_REQUIRED`, stop and tell the user to run `/computer-use on` in the A0 Launcher chat when using Launcher Host access, or in A0 CLI otherwise, and approve any host permission prompt.
10
11
+If a Computer Use call returns an error, do not repeat the same action with identical arguments. Report the error, or use a materially different safe recovery only when the task still requires it.
12
+
13
Call `start_session` before screen-driven tasks. Use `status` for state only, `capture` for screenshots without an action, and `stop_session` when the desktop task is complete. Read `backend_id`, `backend_family`, `features`, and the structured `capabilities` object in status/session results. When capabilities report native windows, window state, element indexes, and background dispatch, prefer `list_windows` -> `get_window_state` -> `element_action` with `dispatch: "background"` before using global coordinates. Interactive coordinate actions should use normalized global-screen coordinates from the most recent capture.
14
15
Some actions are backend-specific and intentionally documented only in backend skills. If `status` or `start_session` reports backend-specific features or tells you to load a backend skill, load and follow that skill before using those backend-only actions. For structural targeting details, load and follow the backend-specific skill such as `host-computer-use-macos` or `host-computer-use-windows`; do not apply one backend's guidance to another backend.
16
15
-State-changing actions automatically attach a fresh screen after they run unless the backend returns a definitive structural background result. Treat key presses, clicks, scrolling, and typing as attempts, not success; treat foreground fallbacks the same way. Inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. A `type` result only proves keystrokes were sent; it does not prove that text landed in the intended place.
17
+State-changing actions automatically attach a fresh screen after they run unless the backend returns a definitive structural background result. Treat key presses, clicks, scrolling, and typing as attempts, not success; treat foreground fallbacks the same way. Inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. A `type` result proves only that keyboard events were sent unless it explicitly reports both `focus_verified=true` and the target `window_id`.
18
+
19
+When Linux advertises `verified-window-focus` and `target-verified-keyboard-input`, use a real frame/window from `list_windows`, inspect it with `get_window_state`, and focus its window element with foreground `element_action`. Continue to `type` only after the focus result reports `focus_verified=true`, and pass that same `window_id` to `type`. Never press an application/frame/window node to activate it.
20
21
```json
22
{
@@ -28,7 +32,7 @@ Required argument:
32
33
Optional arguments by action:
34
- `session_id`: session returned by `start_session`
31
-- `pid`, `window_id`: target a native app/window for `get_window_state` and `element_action`
35
+- `pid`, `window_id`: target a native window for `get_window_state`, `element_action`, scoped backend snapshots, and guarded keyboard input when supported
36
- `element_index`: target an element from the latest `get_window_state`
37
- `operation`: action such as `invoke`, `press`, `set_value`, `focus`, or backend-specific operations
38
- `dispatch`: `background`, `auto`, or `foreground`; prefer `background` for `element_action`
@@ -37,7 +41,7 @@ Optional arguments by action:
41
- `count`: click count for `click`
42
- `dx`, `dy`: scroll amounts for `scroll`
43
- `key` or `keys`: key press value for `key`
40
-- `text`: text to type for `type`
44
+- `text`: text to type for `type`; Linux target-verified input also requires `window_id`
45
- `submit`: boolean Enter-after-type flag for `type`
46
47
Status/session results may include `contract_version` and `capabilities`. Treat `capabilities.identity.pid`, `capabilities.identity.window_id`, `capabilities.identity.element_index`, and `capabilities.dispatch.background` as the authoritative cross-platform contract for whether the native background loop is available. Use `features` for backend-specific refinements and skill selection.
plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md
+7
-2
@@ -29,13 +29,14 @@ Use `ax_snapshot` to inspect the Linux AT-SPI tree:
29
"tool_name": "computer_use_remote",
30
"tool_args": {
31
"action": "ax_snapshot",
32
+ "window_id": "<window_id from list_windows>",
33
"max_depth": 4,
34
"max_nodes": 200
35
}
36
}
37
```
38
38
-The snapshot returns paths, roles, names/titles, descriptions, frames, states, actions, text previews, values, and child nodes. Use it to choose a target, not as final visual proof.
39
+Pass `window_id` whenever one is known so unrelated applications cannot consume the node budget. The snapshot returns paths, roles, names/titles, descriptions, frames, states, actions, text previews, values, and child nodes. Use it to choose a target, not as final visual proof.
40
41
Use `ax_action` for structural actions:
42
@@ -55,7 +56,7 @@ Use `ax_action` for structural actions:
56
57
Supported operations are:
58
58
-- `press`: activate a button, menu item, tab, checkbox, or similar action-bearing node
59
+- `press`: activate a button, menu item, tab, checkbox, or similar action-bearing node; never use it on an application/frame/window
60
- `focus`: focus a focusable node before typing or keyboard input
61
- `set_value`: set text/value on editable nodes; pass `value` or `text`
62
@@ -72,6 +73,10 @@ Use screenshots for proof after every state-changing action. AT-SPI actions and
73
74
True background dispatch on Linux is compositor, toolkit, and app dependent. Do not claim a Linux action was background-safe unless the tool result explicitly says `actual_dispatch=background`.
75
76
+To bring a Linux window forward, target its frame/window element from `get_window_state` with `element_action`, operation `focus`, and `dispatch: "foreground"` or `"auto"`. Continue only when the result says `focus_verified=true`; an accepted AT-SPI call without active/focused state is not activation proof.
77
+
78
+Linux text injection is target-guarded. Pass the same verified active `window_id` to `type`. If the tool reports `COMPUTER_USE_WINDOW_REQUIRED` or `COMPUTER_USE_TARGET_NOT_FOCUSED`, do not type globally and do not substitute another application target.
79
+
80
On GNOME/Wayland, useful shortcuts include:
81
82
- `Super+H`: hide the active window
plugins/_a0_connector/skills/host-computer-use/SKILL.md
+7
-5
@@ -80,10 +80,12 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
80
4. If the backend advertises native window listing through capabilities or `native-window-list`, call `list_windows` before using coordinates.
81
5. If the backend advertises window state and element-index targeting through capabilities or features, call `get_window_state` for the target `pid`/`window_id`, then use `element_action` with `dispatch: "background"` by default.
82
6. If `element_action` reports `background_unavailable`, use `dispatch: "auto"` or `dispatch: "foreground"` only when foreground control is acceptable for the user/task.
83
-7. Decide final success from the latest screenshot or a definitive structural result, not from memory.
84
-8. Interactive actions already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.
85
-9. Use `status` for state without starting a session.
86
-10. Use `capture` only when you need another screenshot without taking an action.
83
+7. When the backend advertises verified window focus, activate a window only with foreground `element_action` operation `focus`; require `focus_verified=true`. Never use `press` on an application/frame/window node as activation.
84
+8. When the backend advertises target-verified keyboard input, pass the verified active `window_id` to `type`; a missing, inactive, or unverifiable target must fail closed.
85
+9. Decide final success from the latest screenshot or a definitive structural result, not from memory.
86
+10. Interactive actions already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.
87
+11. Use `status` for state without starting a session.
88
+12. Use `capture` only when you need another screenshot without taking an action.
89
90
## Backend Skills
91
@@ -107,7 +109,7 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
109
- If the same approach has already failed twice without visible progress, switch strategy instead of repeating it.
110
- Do not infer focus or task completion from chat logs, sidebars, tool summaries, or status text.
111
- Never claim a state-changing action succeeded until the latest screenshot visibly confirms it.
110
-- A `type` tool result only confirms keystrokes were sent. It is not evidence that the text landed in the intended application.
112
+- A `type` tool result confirms the destination only when it reports `focus_verified=true` with the intended `window_id`; otherwise it confirms only global keyboard events were sent.
113
- For browser-navigation tasks done through this tool, only claim success if the browser content area visibly shows the destination page or result.
114
- If the attached screenshot appears unchanged after a state-changing action, use one explicit `capture` to verify before repeating the same action.
115
- Use `type(..., submit=true)` only for URL or navigation-style entry where Enter should fire immediately after typing.
plugins/_a0_connector/tools/computer_use_remote.py
+29
-7
@@ -332,6 +332,8 @@ class ComputerUseRemote(Tool):
332
payload["key"] = self.args.get("key")
333
elif action == "type":
334
payload["text"] = self.args.get("text", "")
335
+ if "window_id" in self.args:
336
+ payload["window_id"] = self.args.get("window_id")
337
if self._coerce_bool(self.args.get("submit")):
338
payload["submit"] = True
339
elif action == "list_windows":
@@ -363,6 +365,9 @@ class ComputerUseRemote(Tool):
365
if "selector" in self.args:
366
payload["selector"] = self.args.get("selector")
367
elif action == "ax_snapshot":
368
+ for key in ("pid", "window_id"):
369
+ if key in self.args:
370
+ payload[key] = self.args.get(key)
371
if "max_depth" in self.args:
372
payload["max_depth"] = self._coerce_int(self.args.get("max_depth"), name="max_depth")
373
if "max_nodes" in self.args:
@@ -471,9 +476,17 @@ class ComputerUseRemote(Tool):
476
return f"Sent keys: {keys!r}."
477
if action == "type":
478
text = str(data.get("text", "") or "")
474
- if data.get("submitted"):
475
- return f"Typed {len(text)} character(s) and submitted."
476
- return f"Typed {len(text)} character(s)."
479
+ window_id = str(data.get("window_id") or "").strip()
480
+ submitted = " and submitted" if data.get("submitted") else ""
481
+ if data.get("focus_verified") and window_id:
482
+ return (
483
+ f"Sent {len(text)} keyboard character(s){submitted} to verified active "
484
+ f"window_id={window_id}."
485
+ )
486
+ return (
487
+ f"Sent {len(text)} global keyboard character(s){submitted}; destination was not verified. "
488
+ "Inspect the attached screen before claiming where the text landed."
489
+ )
490
return str(data)
491
492
def _format_error(self, result: dict[str, Any]) -> str:
@@ -639,7 +652,7 @@ class ComputerUseRemote(Tool):
652
f"{frame.get('width', '?')}x{frame.get('height', '?')})"
653
)
654
flags: list[str] = []
642
- for flag in ("is_on_screen", "on_current_space", "focused", "visible"):
655
+ for flag in ("is_on_screen", "on_current_space", "active", "focused", "visible"):
656
if flag in item:
657
flags.append(f"{flag}={item.get(flag)}")
658
if flags:
@@ -658,10 +671,16 @@ class ComputerUseRemote(Tool):
671
node_count = data.get("node_count", "?")
672
truncated = " truncated" if data.get("truncated") else ""
673
mode = str(data.get("mode") or "auto").strip()
674
+ state_parts = [
675
+ f"{flag}={window.get(flag)}"
676
+ for flag in ("active", "focused")
677
+ if flag in window
678
+ ]
679
+ state_text = f" {' '.join(state_parts)}." if state_parts else ""
680
return (
681
f"Window state for {title!r}"
682
f"{f' window_id={window_id}' if window_id else ''}: "
664
- f"{node_count} element(s){truncated}, mode={mode}. "
683
+ f"{node_count} element(s){truncated}, mode={mode}.{state_text} "
684
"Use element_action with element_index; dispatch defaults to background."
685
f"{self._structural_tree_outline(tree)}"
686
)
@@ -688,7 +707,8 @@ class ComputerUseRemote(Tool):
707
f"requested_dispatch={requested_dispatch}, actual_dispatch={actual_dispatch}"
708
f"{', foreground_fallback_used=true' if fallback else ''}"
709
)
691
- return f"Performed {operation} on element_index={index} {label}; {dispatch_text}."
710
+ verification = ", focus_verified=true" if data.get("focus_verified") else ""
711
+ return f"Performed {operation} on element_index={index} {label}; {dispatch_text}{verification}."
712
713
def _format_ax_snapshot(self, data: dict[str, Any]) -> str:
714
app = data.get("app") if isinstance(data.get("app"), dict) else {}
@@ -697,8 +717,10 @@ class ComputerUseRemote(Tool):
717
node_count = data.get("node_count", "?")
718
truncated = " truncated" if data.get("truncated") else ""
719
root_label = self._ax_target_label(tree)
720
+ window_id = str(data.get("window_id") or "").strip()
721
+ scope = f" scoped to window_id={window_id}" if data.get("scoped") and window_id else ""
722
return (
701
- f"AX snapshot for {app_name}: {node_count} node(s){truncated}. "
723
+ f"AX snapshot for {app_name}{scope}: {node_count} node(s){truncated}. "
724
f"Root {root_label}. Use path or semantic target fields with ax_action."
725
f"{self._structural_tree_outline(tree)}"
726
)
tests/test_a0_connector_prompt_gating.py
+14
-1
@@ -283,6 +283,14 @@ def test_computer_use_remote_prompt_requires_visual_verification_after_actions()
283
/ "host-computer-use"
284
/ "SKILL.md"
285
).read_text(encoding="utf-8")
286
+ linux_skill = (
287
+ PROJECT_ROOT
288
+ / "plugins"
289
+ / "_a0_connector"
290
+ / "skills"
291
+ / "host-computer-use-linux"
292
+ / "SKILL.md"
293
+ ).read_text(encoding="utf-8")
294
295
assert "Treat key presses, clicks, scrolling, and typing" in prompt
296
assert "attempts, not success" in prompt
@@ -294,8 +302,13 @@ def test_computer_use_remote_prompt_requires_visual_verification_after_actions()
302
assert "minimize" not in prompt.lower()
303
assert "window-manager" not in prompt
304
assert "cannot actually see the image" in skill
297
- assert "A `type` tool result only confirms keystrokes were sent" in skill
305
+ assert "A `type` tool result confirms the destination only when" in skill
306
assert "visibly confirms" in skill
307
+ assert "target-verified-keyboard-input" in prompt
308
+ assert "focus_verified=true" in prompt
309
+ assert "do not repeat the same action with identical arguments" in prompt
310
+ assert "Pass the same verified active `window_id` to `type`" in linux_skill
311
+ assert "never use it on an application/frame/window" in linux_skill
312
assert "hide window" not in skill
313
assert "minimize window" not in skill
314
assert "hide/minimize" not in skill
tests/test_tool_action_contracts.py
+78
@@ -1140,6 +1140,84 @@ def test_computer_use_remote_start_session_reports_backend_features_and_windows_
1140
assert "host-computer-use-windows" in message
1141
1142
1143
+def test_computer_use_remote_forwards_linux_window_scope_and_type_guard(monkeypatch):
1144
+ module = _load_computer_use_remote_tool(monkeypatch)
1145
+ tool = object.__new__(module.ComputerUseRemote)
1146
+ window_id = "atspi-pid:57929:path:20.0"
1147
+
1148
+ tool.args = {
1149
+ "action": "ax_snapshot",
1150
+ "pid": 57929,
1151
+ "window_id": window_id,
1152
+ "max_depth": 4,
1153
+ "max_nodes": 80,
1154
+ }
1155
+ snapshot = tool._build_payload(op_id="op-snapshot", context_id="ctx", action="ax_snapshot")
1156
+ tool.args = {"action": "type", "window_id": window_id, "text": "hello", "submit": True}
1157
+ typed = tool._build_payload(op_id="op-type", context_id="ctx", action="type")
1158
+
1159
+ assert snapshot["pid"] == 57929
1160
+ assert snapshot["window_id"] == window_id
1161
+ assert snapshot["max_depth"] == 4
1162
+ assert snapshot["max_nodes"] == 80
1163
+ assert typed["window_id"] == window_id
1164
+ assert typed["text"] == "hello"
1165
+ assert typed["submit"] is True
1166
+
1167
+
1168
+def test_computer_use_remote_receipts_separate_injection_from_verified_result(monkeypatch):
1169
+ module = _load_computer_use_remote_tool(monkeypatch)
1170
+ tool = object.__new__(module.ComputerUseRemote)
1171
+ window_id = "atspi-pid:57929:path:20.0"
1172
+
1173
+ verified = tool._extract_result(
1174
+ "type",
1175
+ {
1176
+ "ok": True,
1177
+ "result": {
1178
+ "text": "hello",
1179
+ "window_id": window_id,
1180
+ "focus_verified": True,
1181
+ },
1182
+ },
1183
+ )
1184
+ unverified = tool._extract_result(
1185
+ "type",
1186
+ {"ok": True, "result": {"text": "hello"}},
1187
+ )
1188
+ focused = tool._extract_result(
1189
+ "element_action",
1190
+ {
1191
+ "ok": True,
1192
+ "result": {
1193
+ "operation": "focus",
1194
+ "target": {"element_index": 0, "role": "frame", "title": "Discord"},
1195
+ "requested_dispatch": "foreground",
1196
+ "actual_dispatch": "foreground",
1197
+ "focus_verified": True,
1198
+ },
1199
+ },
1200
+ )
1201
+ scoped = tool._extract_result(
1202
+ "ax_snapshot",
1203
+ {
1204
+ "ok": True,
1205
+ "result": {
1206
+ "app": {"name": "Discord"},
1207
+ "tree": {"role": "frame", "title": "Discord"},
1208
+ "node_count": 2,
1209
+ "window_id": window_id,
1210
+ "scoped": True,
1211
+ },
1212
+ },
1213
+ )
1214
+
1215
+ assert f"verified active window_id={window_id}" in verified
1216
+ assert "destination was not verified" in unverified
1217
+ assert "focus_verified=true" in focused
1218
+ assert f"scoped to window_id={window_id}" in scoped
1219
+
1220
+
1221
def test_computer_use_remote_capture_artifact_is_chat_scoped(monkeypatch, tmp_path: Path):
1222
module = _load_computer_use_remote_tool(monkeypatch)
1223