Keep backend computer-use actions out of generic guidance
Move explicit AX action names and argument details out of the always-loaded computer_use_remote prompt and generic host-computer-use skill. The generic guidance now only explains backend discovery and skill loading, while host-computer-use-macos remains the detailed home for macOS structural targeting. Also soften the old Super+H hide-window guidance so window actions are chosen from the reported backend and verified visually.
Alessandro Frau committed
May 23, 2026 at 15:23 UTC
a931759868eb107eec0cfe7cc01ccd2f43e8c31d
4 files changed
+30
-27
plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md
+3
-8
@@ -10,9 +10,9 @@ If the tool reports no CLI, disabled computer use, or `COMPUTER_USE_REARM_REQUIR
10
11
Call `start_session` before screen-driven tasks. Use `status` for state only, `capture` for screenshots without an action, and `stop_session` when the desktop task is complete. Interactive coordinate actions should use normalized global-screen coordinates from the most recent capture.
12
13
-Some actions are backend-specific. Use `ax_snapshot` and `ax_action` only when `status` or `start_session` reports a macOS backend with Accessibility-tree features such as `accessibility-tree-snapshot` or `accessibility-structural-targeting`. For macOS structural targeting details, load and follow skill `host-computer-use-macos`; do not apply AX guidance to non-macOS backends.
13
+Some actions are backend-specific and intentionally documented only in backend skills. If `status` or `start_session` reports backend-specific features or tells you to load a backend skill, load and follow that skill before using those backend-only actions. For macOS structural targeting details, load and follow skill `host-computer-use-macos`; do not apply macOS guidance to non-macOS backends.
14
15
-State-changing actions automatically attach a fresh screen after they run. Treat key presses, clicks, scrolling, typing, and window-manager shortcuts as attempts, not success: inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. For Ubuntu/GNOME/Wayland hide-window tasks, prefer `Super+H` (`{"action":"key","keys":["Super","H"]}`) for the active window; do not use `Alt+F9` as the primary hide/minimize shortcut because it often leaves the window visible. A `type` result only proves keystrokes were sent; it does not prove the window was hidden or that text landed in the intended place.
15
+State-changing actions automatically attach a fresh screen after they run. Treat key presses, clicks, scrolling, and typing as attempts, not success: inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. A `type` result only proves keystrokes were sent; it does not prove that text landed in the intended place.
16
17
```json
18
{
@@ -24,18 +24,13 @@ State-changing actions automatically attach a fresh screen after they run. Treat
24
```
25
26
Required argument:
27
-- `action`: one of `start_session`, `status`, `capture`, `ax_snapshot`, `ax_action`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`
27
+- `action`: one of `start_session`, `status`, `capture`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`; backend skills may document additional backend-only action values
28
29
Optional arguments by action:
30
- `session_id`: session returned by `start_session`
31
- `x`, `y`: normalized `[0,1]` global-screen coordinates for `move` and `click`
32
- `button`: `left`, `right`, or `middle` for `click`
33
- `count`: click count for `click`
34
-- `max_depth`, `max_nodes`: optional bounds for backend-gated `ax_snapshot`
35
-- `path`: element path from `ax_snapshot` for backend-gated `ax_action`
36
-- `target`: semantic element target for backend-gated `ax_action`, for example role/title/description/value/identifier
37
-- `operation` or `ax_action`: backend-gated AX operation such as `press`, `focus`, or `set_value`
38
-- `value` or `text`: value for backend-gated `ax_action` with `set_value`
34
- `dx`, `dy`: scroll amounts for `scroll`
35
- `key` or `keys`: key press value for `key`
36
- `text`: text to type for `type`
plugins/_a0_connector/skills/host-computer-use/SKILL.md
+7
-11
@@ -13,8 +13,6 @@ triggers:
13
- "host screen"
14
- "local screen"
15
- "Ubuntu Wayland desktop"
16
- - "hide window"
17
- - "minimize window"
16
---
17
18
# Host Computer Use
@@ -57,9 +55,9 @@ Use:
55
56
Arguments:
57
60
-- `action`: `start_session`, `status`, `capture`, `ax_snapshot`, `ax_action`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`
58
+- `action`: `start_session`, `status`, `capture`, `move`, `click`, `scroll`, `key`, `type`, `stop_session`
59
- `session_id`: optional after `start_session`
62
-- `ax_snapshot`, `ax_action`: backend-gated structural accessibility actions; use only when backend metadata advertises matching support, and load the backend-specific skill first
60
+- backend skills may document additional backend-only action values; use them only when backend metadata advertises matching support and after loading the backend-specific skill
61
- `move`: `x`, `y` normalized to `[0,1]`
62
- `click`: optional `x`, `y`, optional `button` (`left`, `right`, `middle`), optional `count`
63
- `scroll`: `dx`, `dy`
@@ -75,30 +73,28 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
73
1. Call `start_session` first.
74
2. Read the returned `backend_id`, `backend_family`, and `features`; load a backend-specific Computer Use skill when the task needs backend-only affordances.
75
3. Decide final success from the latest screenshot, not from memory.
78
-4. Interactive actions (`ax_action`, `move`, `click`, `scroll`, `key`, `type`) already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.
76
+4. Interactive actions already attach a fresh screenshot after they run; inspect it before claiming the requested outcome succeeded.
77
5. Use `status` for state without starting a session.
78
6. Use `capture` only when you need another screenshot without taking an action.
79
80
## Backend Skills
81
84
-- If the backend is macOS or features include `accessibility-tree-snapshot` / `accessibility-structural-targeting`, load `host-computer-use-macos` before using `ax_snapshot` or `ax_action`.
82
+- If the backend is macOS or features include `accessibility-tree-snapshot` / `accessibility-structural-targeting`, load `host-computer-use-macos` before using macOS structural Accessibility actions.
83
- Do not use backend-specific actions just because their argument names exist in the generic contract. Treat them as unavailable unless the connected CLI advertises the matching feature.
84
85
## Operating Rules
86
87
- Only the latest screenshot or a definitive tool result counts as evidence.
88
- If a tool result says a screenshot was attached but you cannot actually see the image, stop and report that visual verification is unavailable. Do not continue with another action from an assumed host state.
91
-- Outside advertised AX support, use normalized global screen coordinates; do not assume window ids, element indexes, background-safe input, or semantic click targets unless the runtime explicitly advertises them.
89
+- Outside advertised structural accessibility support, use normalized global screen coordinates; do not assume window ids, element indexes, background-safe input, or semantic click targets unless the runtime explicitly advertises them.
90
- Prefer accessibility and semantic UI paths first: shortcuts, command palettes, menu accelerators, address/search bars, focus traversal, and other keyboard-accessible controls.
91
- Prefer `key` and `type` over pointer actions whenever a reliable keyboard path exists.
92
- When a menu or popup is open, treat it as the active UI and prefer keyboard navigation over clicking small transient rows by coordinate.
93
- If a click dismisses a menu or popup without producing the expected next UI, treat that attempt as failed.
94
- If the same approach has already failed twice without visible progress, switch strategy instead of repeating it.
95
- Do not infer focus or task completion from chat logs, sidebars, tool summaries, or status text.
98
-- Never claim a window was hidden, minimized, moved, text was submitted, or navigation completed until the latest screenshot visibly confirms it.
99
-- On Ubuntu/GNOME/Wayland, use `Super+H` (`{"action":"key","keys":["Super","H"]}`) to hide the active window. Do not use `Alt+F9` as the primary hide/minimize shortcut on this environment; it often leaves the window visible.
100
-- After any hide/minimize shortcut, inspect the fresh screenshot. If the target window or focused composer is still visible, treat the attempt as failed and do not type follow-up text into the active field.
101
-- A `type` tool result only confirms keystrokes were sent. It is not evidence that the text landed in the intended application, nor evidence that a window was hidden first.
96
+- Never claim a state-changing action succeeded until the latest screenshot visibly confirms it.
97
+- A `type` tool result only confirms keystrokes were sent. It is not evidence that the text landed in the intended application.
98
- For browser-navigation tasks done through this tool, only claim success if the browser content area visibly shows the destination page or result.
99
- If the attached screenshot appears unchanged after a state-changing action, use one explicit `capture` to verify before repeating the same action.
100
- Use `type(..., submit=true)` only for URL or navigation-style entry where Enter should fire immediately after typing.
tests/test_a0_connector_prompt_gating.py
+14
-8
@@ -134,19 +134,22 @@ def test_computer_use_remote_prompt_requires_visual_verification_after_actions()
134
/ "SKILL.md"
135
).read_text(encoding="utf-8")
136
137
- assert "Treat key presses, clicks, scrolling, typing" in prompt
137
+ assert "Treat key presses, clicks, scrolling, and typing" in prompt
138
assert "attempts, not success" in prompt
139
assert "visual verification is unavailable" in prompt
140
assert "do not continue by assuming the host state" in prompt
141
- assert "Super+H" in prompt
142
- assert '["Super","H"]' in prompt
143
- assert "Alt+F9" in prompt
141
+ assert "Super+H" not in prompt
142
+ assert "Alt+F9" not in prompt
143
+ assert "hide" not in prompt.lower()
144
+ assert "minimize" not in prompt.lower()
145
+ assert "window-manager" not in prompt
146
assert "cannot actually see the image" in skill
145
- assert "Do not use `Alt+F9` as the primary hide/minimize shortcut" in skill
147
assert "A `type` tool result only confirms keystrokes were sent" in skill
147
- assert "do not type follow-up text into the active field" in skill
148
assert "visibly confirms" in skill
149
- assert "Ubuntu/GNOME/Wayland" in skill
149
+ assert "hide window" not in skill
150
+ assert "minimize window" not in skill
151
+ assert "hide/minimize" not in skill
152
+ assert "window-manager" not in skill
153
154
155
def test_remote_file_and_exec_tools_are_standard_tool_prompts_independent_from_context():
@@ -398,11 +401,14 @@ def test_remote_tool_stubs_are_self_contained_and_reference_per_tool_skills():
401
assert '"tool_name": "computer_use_remote"' in computer_stub
402
assert "load and follow skill `host-computer-use`" in computer_stub
403
assert "host-computer-use-macos" in computer_stub
404
+ assert "ax_snapshot" not in computer_stub
405
+ assert "ax_action" not in computer_stub
406
assert "Do not substitute the `linux-desktop` skill" in computer_stub
407
assert '"tool_name": "computer_use_remote"' in computer_skill
408
assert '"tool_name": "computer_use_remote"' in macos_computer_skill
409
assert "ax_snapshot" in macos_computer_skill
405
- assert "ax_snapshot`/`ax_action` are structural Accessibility targeting" not in computer_skill
410
+ assert "ax_snapshot" not in computer_skill
411
+ assert "ax_action" not in computer_skill
412
assert "Availability, backend support, and trust mode are checked when the tool runs" in computer_stub
413
assert "not `code_execution_tool`" in exec_stub
414
assert "not to" in exec_stub
tests/test_tool_action_contracts.py
+6
@@ -618,8 +618,14 @@ def test_computer_use_remote_is_runtime_checked_standard_tool():
618
assert "checked when the tool runs" in standard_prompt_text
619
assert "visual verification is unavailable" in standard_prompt_text
620
assert "host-computer-use-macos" in standard_prompt_text
621
+ assert "ax_snapshot" not in standard_prompt_text
622
+ assert "ax_action" not in standard_prompt_text
623
assert '"tool_name": "computer_use_remote"' in skill_text
624
+ assert "ax_snapshot" not in skill_text
625
+ assert "ax_action" not in skill_text
626
assert '"tool_name": "computer_use_remote"' in macos_skill_text
627
+ assert "ax_snapshot" in macos_skill_text
628
+ assert "ax_action" in macos_skill_text
629
assert "Backend-specific macOS guidance" in macos_skill_text
630
assert "Beta desktop control" in skill_text
631