@samitouri / QOSami_Agents_Zero-1 / commits / b6705593

Guide Windows computer use through UIA

Add the Windows host computer-use skill and teach computer_use_remote to surface UIA window-management guidance, selector passthrough, and click-last workflow hints. Keep backend-specific actions out of generic guidance while exposing Windows structural operations when the backend advertises them. Tests: uv run --python 3.12 --with-requirements requirements.txt --with-requirements requirements2.txt --with-requirements requirements.dev.txt --with litellm pytest tests\\test_tool_action_contracts.py tests\\test_a0_connector_prompt_gating.py tests\\test_skills_runtime.py -q

Alessandro committed May 23, 2026 at 18:25 UTC b6705593227953a683b0b61eb7596c4cc5da40ef

7 files changed +247 -3

plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md

+1 -1

@@ -10,7 +10,7 @@ If the tool reports no CLI, disabled computer use, or `COMPUTER_USE_REARM_REQUIR

11 Call `start_session` before screen-driven tasks. Use `status` for state only, `capture` for screenshots without an action, and `stop_session` when the desktop task is complete. Interactive coordinate actions should use normalized global-screen coordinates from the most recent capture.

13 -Some actions are backend-specific and intentionally documented only in backend skills. If `status` or `start_session` reports backend-specific features or tells you to load a backend skill, load and follow that skill before using those backend-only actions. For macOS structural targeting details, load and follow skill `host-computer-use-macos`; do not apply macOS guidance to non-macOS backends.

13 +Some actions are backend-specific and intentionally documented only in backend skills. If `status` or `start_session` reports backend-specific features or tells you to load a backend skill, load and follow that skill before using those backend-only actions. For structural targeting details, load and follow the backend-specific skill such as `host-computer-use-macos` or `host-computer-use-windows`; do not apply one backend's guidance to another backend.

15 State-changing actions automatically attach a fresh screen after they run. Treat key presses, clicks, scrolling, and typing as attempts, not success: inspect the latest attached screen, or one explicit `capture` if it is unclear or unchanged, before saying the requested outcome happened. If the tool says a screen was attached but you cannot actually inspect the image, stop and report that visual verification is unavailable; do not continue by assuming the host state. A `type` result only proves keystrokes were sent; it does not prove that text landed in the intended place.

plugins/_a0_connector/skills/host-computer-use-windows/SKILL.md new

+90

@@ -0,0 +1,90 @@

1 +---

2 +name: host-computer-use-windows

3 +description: Backend-specific Windows guidance for `computer_use_remote`. Load after `status` or `start_session` reports backend_family/backend_id `windows` or Windows UI Automation features. Covers UIA structural targeting, Windows session caveats, and screenshot verification.

4 +---

5 +

6 +# Host Computer Use - Windows

7 +

8 +Use this after `host-computer-use` when the connected A0 CLI reports the Windows computer-use backend.

9 +

10 +Do not use this skill for Linux, macOS, Xpra, Docker, or browser-only tasks. If the backend is not Windows or does not advertise UI Automation support, skip UIA actions and follow the generic host computer-use rules.

11 +

12 +## Windows UIA Targeting

13 +

14 +Windows backends can advertise structural UI Automation features:

15 +

16 +- `uia-tree-snapshot`

17 +- `uia-structural-targeting`

18 +- `uia-element-action`

19 +- `uia-window-management`

20 +

21 +When these features are present, prefer structural targeting over pixel clicks for named controls such as buttons, menu items, text fields, dialogs, toolbar items, browser address bars, composer fields, and list rows.

22 +

23 +Use `uia_snapshot` to inspect the bounded Windows UI Automation tree:

24 +

25 +```json

26 +{

27 + "tool_name": "computer_use_remote",

28 + "tool_args": {

29 + "action": "uia_snapshot",

30 + "max_depth": 4,

31 + "max_nodes": 200

32 + }

33 +}

34 +```

35 +

36 +The snapshot returns element paths, roles, names/titles, automation IDs, class names, optional Terminator-style selectors, frames, enabled/focused state, actions, and child nodes. Use it to choose an element, not as final visual proof.

37 +

38 +Use `uia_action` for a structural action:

39 +

40 +```json

41 +{

42 + "tool_name": "computer_use_remote",

43 + "tool_args": {

44 + "action": "uia_action",

45 + "target": {

46 + "role": "Button",

47 + "title": "OK"

48 + },

49 + "operation": "invoke"

50 + }

51 +}

52 +```

53 +

54 +Supported operations are:

55 +

56 +- `invoke`: activate a button, menu item, checkbox, or similar control

57 +- `focus_window`: restore and bring the owning top-level window to the foreground

58 +- `minimize`, `restore`, `maximize`: change the owning top-level window state without clicking titlebar buttons

59 +- `focus`: focus a text field or focusable element after activating its window

60 +- `set_value`: set text/value; pass `value` or `text`

61 +- `click`: click the element through the Windows UIA wrapper only when the snapshot says click is the available action and no structural operation fits

62 +- `close`: close the owning top-level window only when the user explicitly asked to close that app/window

63 +

64 +Targeting options:

65 +

66 +- Prefer a semantic `target` when the control has a stable role plus title/name, automation ID, class name, handle, process ID, framework ID, or selector.

67 +- Use a `path` returned by the latest `uia_snapshot` only while the UI is unchanged.

68 +- You may pass a `selector` or `target.selector` from the snapshot, such as `role:Button && name:OK`.

69 +- If an action reports ambiguity, take a fresh snapshot and narrow the target with role plus title/name/automation ID/class name.

70 +- If an action reports a missing target, take a fresh snapshot before trying coordinates.

71 +

72 +Action selection:

73 +

74 +- Prefer the actions listed on the target node. If a node offers `invoke`, use `invoke`, not `click`.

75 +- For window focus, hiding, restoring, or maximizing, use `focus_window`, `minimize`, `restore`, or `maximize`; do not click titlebar buttons.

76 +- For typing into an app, first structurally focus the app/window if needed, then `set_value` on the target field. A global `type` result only proves keys were sent, not that they landed in the intended control.

77 +- After a window operation, navigation, menu open/close, dialog transition, or other layout change, take a fresh `uia_snapshot` before reusing a path.

78 +- Use pixel `click` only after structural UIA, keyboard, browser, and app-native options do not fit, and only from a fresh screenshot with an unambiguous target.

79 +

80 +UIA actions are attempts, not proof. They attach a fresh screenshot after state-changing actions; inspect that image before saying the requested outcome happened.

81 +

82 +## Windows Session Caveats

83 +

84 +Windows desktop capture and UI Automation depend on the interactive desktop session where A0 CLI is running. Remote Desktop, VM consoles, UAC prompts, elevated apps, locked screens, minimized/disconnected RDP sessions, and services can prevent capture or UIA access.

85 +

86 +If `computer_use_remote` returns `COMPUTER_USE_REARM_REQUIRED`, `COMPUTER_USE_APPROVAL_REQUIRED`, `COMPUTER_USE_CAPTURE_UNAVAILABLE`, `COMPUTER_USE_UIA_UNAVAILABLE`, or `status=rearm required`, stop immediately and ask the user to re-arm or fix the Windows desktop session. Do not bypass a permission/session failure with server screenshots, Docker commands, `linux-desktop`, or browser fallbacks.

87 +

88 +## Geometry

89 +

90 +Windows captures can use a virtual desktop that includes multiple monitors and negative origins. Use the capture/session `origin_x`, `origin_y`, `width`, and `height` as the coordinate space, and use normalized `[0,1]` coordinates only relative to that virtual screen.

plugins/_a0_connector/skills/host-computer-use/SKILL.md

+2 -1

@@ -31,7 +31,7 @@ If the task needs shell execution on the CLI host, load `host-code-execution` se

32 This skill controls the user's connected host/local computer through A0 CLI. It is not the built-in Linux Desktop/Xpra skill.

34 -Never switch to `linux-desktop`, the Agent Zero Desktop/Xpra surface, `desktopctl.sh`, `code_execution_tool`, or Docker/server shell commands as a fallback for host screen actions such as screenshots, clicking, typing, hiding/minimizing windows, or checking visible host UI. Those paths only see the internal Agent Zero runtime. If `computer_use_remote` is unavailable, disabled, or needs re-arming, stop and ask the user to run `/computer-use on` in the A0 CLI and approve the platform permission prompt.

34 +Never switch to `linux-desktop`, the Agent Zero Desktop/Xpra surface, `desktopctl.sh`, `code_execution_tool`, or Docker/server shell commands as a fallback for host screen actions such as screenshots, clicking, typing, desktop state changes, or checking visible host UI. Those paths only see the internal Agent Zero runtime. If `computer_use_remote` is unavailable, disabled, or needs re-arming, stop and ask the user to run `/computer-use on` in the A0 CLI and approve the platform permission prompt.

36 ## Browser Boundary

@@ -80,6 +80,7 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi

80 ## Backend Skills

82 - If the backend is macOS or features include `accessibility-tree-snapshot` / `accessibility-structural-targeting`, load `host-computer-use-macos` before using macOS structural Accessibility actions.

83 +- If the backend is Windows or features include `uia-tree-snapshot` / `uia-structural-targeting`, load `host-computer-use-windows` before using Windows UI Automation structural actions.

84 - Do not use backend-specific actions just because their argument names exist in the generic contract. Treat them as unavailable unless the connected CLI advertises the matching feature.

86 ## Operating Rules

plugins/_a0_connector/tools/computer_use_remote.py

+80

@@ -35,6 +35,7 @@ REARM_REQUIRED_DEFAULT_MESSAGE = (

35 _AUTO_CAPTURE_ACTIONS = {

36 "start_session",

37 "ax_action",

38 + "uia_action",

39 "move",

40 "click",

41 "scroll",

@@ -48,6 +49,7 @@ _SETTLE_DELAY_SCROLL = 0.35

49 _SETTLE_DELAY_KEY = 0.2

50 _SETTLE_DELAY_TYPE = 0.25

51 _SETTLE_DELAY_AX_ACTION = 0.25

52 +_SETTLE_DELAY_UIA_ACTION = 0.25

53 _SETTLE_DELAY_GLOBAL_FOCUS = 0.45

54 _SETTLE_DELAY_PLAIN_ENTER = 0.3

55 _SETTLE_DELAY_SUBMIT = 0.45

@@ -58,6 +60,8 @@ _SUPPORTED_ACTIONS = {

60 "capture",

61 "ax_snapshot",

62 "ax_action",

63 + "uia_snapshot",

64 + "uia_action",

65 "move",

66 "click",

67 "scroll",

@@ -77,6 +81,7 @@ class ComputerUseRemote(Tool):

81 message=(

82 "action is required and must be one of: "

83 "start_session, status, capture, ax_snapshot, ax_action, "

84 + "uia_snapshot, uia_action, "

85 "move, click, scroll, key, type, stop_session"

86 ),

87 break_loop=False,

@@ -262,6 +267,8 @@ class ComputerUseRemote(Tool):

267 return _SETTLE_DELAY_SCROLL

268 if action == "ax_action":

269 return _SETTLE_DELAY_AX_ACTION

270 + if action == "uia_action":

271 + return _SETTLE_DELAY_UIA_ACTION

272 if action == "type" and self._coerce_bool(self.args.get("submit")):

273 return _SETTLE_DELAY_SUBMIT

274 if action == "type":

@@ -335,6 +342,31 @@ class ComputerUseRemote(Tool):

342 payload["value"] = self.args.get("value")

343 if "text" in self.args:

344 payload["text"] = self.args.get("text", "")

345 + elif action == "uia_snapshot":

346 + if "max_depth" in self.args:

347 + payload["max_depth"] = self._coerce_int(self.args.get("max_depth"), name="max_depth")

348 + if "max_nodes" in self.args:

349 + payload["max_nodes"] = self._coerce_int(self.args.get("max_nodes"), name="max_nodes")

350 + elif action == "uia_action":

351 + target = self.args.get("target")

352 + normalized_target: dict[str, Any] = {}

353 + if isinstance(target, dict):

354 + normalized_target.update(target)

355 + if "selector" in self.args:

356 + normalized_target["selector"] = str(self.args.get("selector") or "").strip()

357 + if normalized_target:

358 + payload["target"] = normalized_target

359 + if "path" in self.args:

360 + payload["path"] = self.args.get("path")

361 + operation = self.args.get("operation", self.args.get("uia_action", self.args.get("name")))

362 + if operation is not None:

363 + payload["operation"] = operation

364 + if "value" in self.args:

365 + payload["value"] = self.args.get("value")

366 + if "text" in self.args:

367 + payload["text"] = self.args.get("text", "")

368 + if self._coerce_bool(self.args.get("submit")):

369 + payload["submit"] = True

370

371 return payload

372

@@ -361,6 +393,13 @@ class ComputerUseRemote(Tool):

393 operation = str(data.get("operation") or "?")

394 path = target.get("path", "?")

395 return f"Performed AX {operation} on {self._ax_target_label(target)} path={path}."

396 + if action == "uia_snapshot":

397 + return self._format_uia_snapshot(data)

398 + if action == "uia_action":

399 + target = data.get("target") if isinstance(data.get("target"), dict) else {}

400 + operation = str(data.get("operation") or "?")

401 + path = target.get("path", "?")

402 + return f"Performed Windows UIA {operation} on {self._uia_target_label(target)} path={path}."

403 if action == "status":

404 return self._format_status(data)

405 if action == "start_session":

@@ -453,6 +492,20 @@ class ComputerUseRemote(Tool):

492 " Load skill `host-computer-use-macos` before using macOS AX "

493 "structural actions."

494 )

495 + has_windows_uia = bool(

496 + features

497 + & {

498 + "uia-tree-snapshot",

499 + "uia-structural-targeting",

500 + "uia-element-action",

501 + "uia-window-management",

502 + }

503 + )

504 + if backend_id == "windows" or backend_family == "windows" or has_windows_uia:

505 + return (

506 + " Load skill `host-computer-use-windows` before using Windows UIA "

507 + "structural actions and window-management operations."

508 + )

509 return ""

510

511 def _format_status(self, data: dict[str, Any]) -> str:

@@ -497,6 +550,20 @@ class ComputerUseRemote(Tool):

550 f"Root {root_label}. Use path or semantic target fields with ax_action."

551 )

552

553 + def _format_uia_snapshot(self, data: dict[str, Any]) -> str:

554 + app = data.get("app") if isinstance(data.get("app"), dict) else {}

555 + tree = data.get("tree") if isinstance(data.get("tree"), dict) else {}

556 + app_name = str(app.get("name") or "Windows desktop")

557 + node_count = data.get("node_count", "?")

558 + truncated = " truncated" if data.get("truncated") else ""

559 + root_label = self._uia_target_label(tree)

560 + return (

561 + f"Windows UIA snapshot for {app_name}: {node_count} node(s){truncated}. "

562 + f"Root {root_label}. Prefer node actions with uia_action; use "

563 + f"focus_window/minimize/restore/maximize for windows, and reserve click "

564 + f"for a last resort."

565 + )

566 +

567 def _ax_target_label(self, target: dict[str, Any]) -> str:

568 role = str(target.get("role") or "element")

569 title = str(target.get("title") or target.get("description") or target.get("identifier") or "").strip()

@@ -504,6 +571,19 @@ class ComputerUseRemote(Tool):

571 return f"{role} {title!r}"

572 return role

573

574 + def _uia_target_label(self, target: dict[str, Any]) -> str:

575 + role = str(target.get("role") or "element")

576 + title = str(

577 + target.get("title")

578 + or target.get("name")

579 + or target.get("automation_id")

580 + or target.get("class_name")

581 + or ""

582 + ).strip()

583 + if title:

584 + return f"{role} {title!r}"

585 + return role

586 +

587 def _record_capture(self, data: dict[str, Any]) -> str:

588 display_ref, resolved_capture_id = self._resolve_capture_ref(data)

589 width = data.get("width", "?")

tests/test_a0_connector_prompt_gating.py

+29

@@ -356,6 +356,14 @@ def test_remote_affordance_skills_parse():

356 / "host-computer-use-macos"

357 / "SKILL.md"

358 )

359 + windows_computer_skill = _parse_skill_frontmatter(

360 + PROJECT_ROOT

361 + / "plugins"

362 + / "_a0_connector"

363 + / "skills"

364 + / "host-computer-use-windows"

365 + / "SKILL.md"

366 + )

367

368 assert not legacy_connector_skill.exists()

369 assert text_editor_skill["name"] == "host-file-editing"

@@ -371,6 +379,8 @@ def test_remote_affordance_skills_parse():

379 assert "Ubuntu Wayland desktop" in computer_skill["triggers"]

380 assert macos_computer_skill["name"] == "host-computer-use-macos"

381 assert "macOS guidance" in macos_computer_skill["description"]

382 + assert windows_computer_skill["name"] == "host-computer-use-windows"

383 + assert "Windows guidance" in windows_computer_skill["description"]

384

385

386 def test_remote_tool_stubs_are_self_contained_and_reference_per_tool_skills():

@@ -393,6 +403,14 @@ def test_remote_tool_stubs_are_self_contained_and_reference_per_tool_skills():

403 / "host-computer-use-macos"

404 / "SKILL.md"

405 ).read_text(encoding="utf-8")

406 + windows_computer_skill = (

407 + PROJECT_ROOT

408 + / "plugins"

409 + / "_a0_connector"

410 + / "skills"

411 + / "host-computer-use-windows"

412 + / "SKILL.md"

413 + ).read_text(encoding="utf-8")

414

415 assert "optionally load skill `host-file-editing`" in text_stub

416 assert "optionally load skill `host-code-execution`" in exec_stub

@@ -401,14 +419,25 @@ def test_remote_tool_stubs_are_self_contained_and_reference_per_tool_skills():

419 assert '"tool_name": "computer_use_remote"' in computer_stub

420 assert "load and follow skill `host-computer-use`" in computer_stub

421 assert "host-computer-use-macos" in computer_stub

422 + assert "host-computer-use-windows" in computer_stub

423 assert "ax_snapshot" not in computer_stub

424 assert "ax_action" not in computer_stub

425 + assert "uia_snapshot" not in computer_stub

426 + assert "uia_action" not in computer_stub

427 assert "Do not substitute the `linux-desktop` skill" in computer_stub

428 assert '"tool_name": "computer_use_remote"' in computer_skill

429 assert '"tool_name": "computer_use_remote"' in macos_computer_skill

430 + assert '"tool_name": "computer_use_remote"' in windows_computer_skill

431 assert "ax_snapshot" in macos_computer_skill

432 assert "ax_snapshot" not in computer_skill

433 assert "ax_action" not in computer_skill

434 + assert "uia_snapshot" in windows_computer_skill

435 + assert "uia_action" in windows_computer_skill

436 + assert "focus_window" in windows_computer_skill

437 + assert "minimize" in windows_computer_skill

438 + assert "If a node offers `invoke`, use `invoke`, not `click`" in windows_computer_skill

439 + assert "uia_snapshot" not in computer_skill

440 + assert "uia_action" not in computer_skill

441 assert "Availability, backend support, and trust mode are checked when the tool runs" in computer_stub

442 assert "not `code_execution_tool`" in exec_stub

443 assert "not to" in exec_stub

tests/test_skills_runtime.py

+2 -1

@@ -292,6 +292,7 @@ def test_renamed_skills_use_standard_frontmatter_only():

292 PROJECT_ROOT / "plugins" / "_a0_connector" / "skills" / "host-code-execution" / "SKILL.md",

293 PROJECT_ROOT / "plugins" / "_a0_connector" / "skills" / "host-computer-use" / "SKILL.md",

294 PROJECT_ROOT / "plugins" / "_a0_connector" / "skills" / "host-computer-use-macos" / "SKILL.md",

295 + PROJECT_ROOT / "plugins" / "_a0_connector" / "skills" / "host-computer-use-windows" / "SKILL.md",

296 PROJECT_ROOT / "plugins" / "_a0_connector" / "skills" / "host-file-editing" / "SKILL.md",

297 PROJECT_ROOT / "plugins" / "_a0_connector" / "skills" / "setup-a0-cli" / "SKILL.md",

298 PROJECT_ROOT / "plugins" / "_browser" / "skills" / "browser-automation" / "SKILL.md",

@@ -410,7 +411,7 @@ def test_hidden_skills_filter_agent_visible_skill_catalog(monkeypatch, tmp_path:

411 "get_paths",

412 lambda agent, *parts: [str(skills_root)],

413 )

413 - monkeypatch.setattr(runtime.files, "exists", lambda path: str(path) == str(skills_root))

414 + monkeypatch.setattr(runtime.files, "exists", lambda path: Path(str(path)) == skills_root)

415 monkeypatch.setattr(

416 runtime.plugin_helpers,

417 "get_plugin_config",

tests/test_tool_action_contracts.py

+43

@@ -607,6 +607,10 @@ def test_computer_use_remote_is_runtime_checked_standard_tool():

607 project_root

608 / "plugins/_a0_connector/skills/host-computer-use-macos/SKILL.md"

609 ).read_text(encoding="utf-8")

610 + windows_skill_text = (

611 + project_root

612 + / "plugins/_a0_connector/skills/host-computer-use-windows/SKILL.md"

613 + ).read_text(encoding="utf-8")

614

615 assert standard_prompt_path.exists()

616 assert not (

@@ -618,15 +622,26 @@ def test_computer_use_remote_is_runtime_checked_standard_tool():

622 assert "checked when the tool runs" in standard_prompt_text

623 assert "visual verification is unavailable" in standard_prompt_text

624 assert "host-computer-use-macos" in standard_prompt_text

625 + assert "host-computer-use-windows" in standard_prompt_text

626 assert "ax_snapshot" not in standard_prompt_text

627 assert "ax_action" not in standard_prompt_text

628 + assert "uia_snapshot" not in standard_prompt_text

629 + assert "uia_action" not in standard_prompt_text

630 assert '"tool_name": "computer_use_remote"' in skill_text

631 assert "ax_snapshot" not in skill_text

632 assert "ax_action" not in skill_text

633 + assert "uia_snapshot" not in skill_text

634 + assert "uia_action" not in skill_text

635 assert '"tool_name": "computer_use_remote"' in macos_skill_text

636 assert "ax_snapshot" in macos_skill_text

637 assert "ax_action" in macos_skill_text

638 + assert '"tool_name": "computer_use_remote"' in windows_skill_text

639 + assert "uia_snapshot" in windows_skill_text

640 + assert "uia_action" in windows_skill_text

641 + assert "focus_window" in windows_skill_text

642 + assert "If a node offers `invoke`, use `invoke`, not `click`" in windows_skill_text

643 assert "Backend-specific macOS guidance" in macos_skill_text

644 + assert "Backend-specific Windows guidance" in windows_skill_text

645 assert "Beta desktop control" in skill_text

646

647

@@ -656,3 +671,31 @@ def test_computer_use_remote_start_session_reports_backend_features_and_macos_sk

671 assert "backend=macos/macos" in message

672 assert "features=accessibility-tree-snapshot, accessibility-structural-targeting" in message

673 assert "host-computer-use-macos" in message

674 +

675 +

676 +def test_computer_use_remote_start_session_reports_backend_features_and_windows_skill(monkeypatch):

677 + module = _load_computer_use_remote_tool(monkeypatch)

678 + tool = object.__new__(module.ComputerUseRemote)

679 +

680 + message = tool._extract_result(

681 + "start_session",

682 + {

683 + "ok": True,

684 + "result": {

685 + "session_id": "s1",

686 + "width": 3840,

687 + "height": 2160,

688 + "backend_id": "windows",

689 + "backend_family": "windows",

690 + "features": [

691 + "uia-tree-snapshot",

692 + "uia-structural-targeting",

693 + ],

694 + },

695 + },

696 + )

697 +

698 + assert "session_id=s1" in message

699 + assert "backend=windows/windows" in message

700 + assert "features=uia-tree-snapshot, uia-structural-targeting" in message

701 + assert "host-computer-use-windows" in message