@samitouri / QOSami_Agents_Zero / commits / 97953db4

Guide computer-use remote through Linux AT-SPI

Add a Linux-specific host computer-use skill, route Wayland/AT-SPI backends to it instead of macOS AX guidance, and include compact structural tree outlines in AX/UIA snapshot responses so agents can pick paths and semantic targets from the tool result.

Alessandro committed May 23, 2026 at 19:25 UTC 97953db46ba0ff49bef1d0ea59e48a8a06c91bff
3 files changed +167
plugins/_a0_connector/skills/host-computer-use-linux/SKILL.md new
+86
@@ -0,0 +1,86 @@
1 +---
2 +name: host-computer-use-linux
3 +description: Backend-specific Linux guidance for `computer_use_remote`. Load after `status` or `start_session` reports backend_family `linux`, backend_id `wayland`, or AT-SPI features. Covers AT-SPI structural targeting, Wayland portal caveats, and screenshot verification.
4 +---
5 +
6 +# Host Computer Use - Linux
7 +
8 +Use this after `host-computer-use` when the connected A0 CLI reports the Linux/Wayland computer-use backend.
9 +
10 +Do not use this skill for macOS, Windows, Xpra, Docker, browser-only tasks, or the internal Agent Zero Desktop. If the backend is not Linux or does not advertise AT-SPI support, skip Linux structural actions and follow the generic host computer-use rules.
11 +
12 +## Linux AT-SPI Targeting
13 +
14 +Linux backends can advertise structural AT-SPI features:
15 +
16 +- `atspi-tree-snapshot`
17 +- `atspi-structural-targeting`
18 +- `atspi-element-action`
19 +- `atspi-set-value`
20 +
21 +When these features are present, prefer structural targeting over pixel clicks for named buttons, menu items, text fields, dialogs, toolbar items, tab strips, and application windows.
22 +
23 +Use `ax_snapshot` to inspect the Linux AT-SPI tree:
24 +
25 +```json
26 +{
27 + "tool_name": "computer_use_remote",
28 + "tool_args": {
29 + "action": "ax_snapshot",
30 + "max_depth": 4,
31 + "max_nodes": 200
32 + }
33 +}
34 +```
35 +
36 +The snapshot returns paths, roles, names/titles, descriptions, frames, states, actions, text previews, values, and child nodes. Use it to choose a target, not as final visual proof.
37 +
38 +Use `ax_action` for structural actions:
39 +
40 +```json
41 +{
42 + "tool_name": "computer_use_remote",
43 + "tool_args": {
44 + "action": "ax_action",
45 + "target": {
46 + "role": "push button",
47 + "title": "OK"
48 + },
49 + "operation": "press"
50 + }
51 +}
52 +```
53 +
54 +Supported operations are:
55 +
56 +- `press`: activate a button, menu item, tab, checkbox, or similar action-bearing node
57 +- `focus`: focus a focusable node before typing or keyboard input
58 +- `set_value`: set text/value on editable nodes; pass `value` or `text`
59 +
60 +Targeting options:
61 +
62 +- Prefer a semantic `target` when a node has a stable role plus title/name/description/text/state/action.
63 +- Use a `path` returned by the latest `ax_snapshot` only while the UI is unchanged.
64 +- If an action reports ambiguity, take a fresh snapshot and narrow the target with role plus title/name/description.
65 +- If an action reports a missing target, take a fresh snapshot before trying coordinates.
66 +
67 +## Wayland Notes
68 +
69 +Use screenshots for proof after every state-changing action. AT-SPI actions and keyboard events are attempts, not proof, and Wayland focus can reject or redirect input when the active window changes.
70 +
71 +On GNOME/Wayland, useful shortcuts include:
72 +
73 +- `Super+H`: hide the active window
74 +- `Alt+Tab`: switch applications
75 +- `Ctrl+L`: focus a browser address bar when the browser is already focused
76 +- `Ctrl+T`: open a new browser tab when the browser is already focused
77 +
78 +Treat every shortcut as an attempt. Inspect the fresh screenshot before saying it worked. If text lands in the wrong app, stop and reassess from `capture` or `ax_snapshot`; do not continue typing from assumed focus.
79 +
80 +Some apps expose shallow AT-SPI trees unless their own accessibility support is enabled. If the AT-SPI tree is too shallow for a task, fall back in this order: app-native/browser tooling, reliable keyboard paths, then normalized coordinate clicks from a fresh screenshot.
81 +
82 +## Permissions
83 +
84 +If `computer_use_remote` returns `COMPUTER_USE_AX_UNAVAILABLE`, `COMPUTER_USE_REARM_REQUIRED`, `COMPUTER_USE_APPROVAL_REQUIRED`, or `status=rearm required`, stop immediately and ask the user to re-arm or fix the Linux desktop accessibility/session state.
85 +
86 +Do not bypass a permission or host-visibility failure with server screenshots, Docker commands, the built-in Linux Desktop/Xpra skill, or `code_execution_tool`.
plugins/_a0_connector/skills/host-computer-use/SKILL.md
+2
@@ -79,6 +79,7 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
79
80 ## Backend Skills
81
82 +- If the backend is Linux/Wayland or features include `atspi-tree-snapshot` / `atspi-structural-targeting`, load `host-computer-use-linux` before using Linux AT-SPI structural actions.
83 - If the backend is macOS or features include `accessibility-tree-snapshot` / `accessibility-structural-targeting`, load `host-computer-use-macos` before using macOS structural Accessibility actions.
84 - If the backend is Windows or features include `uia-tree-snapshot` / `uia-structural-targeting`, load `host-computer-use-windows` before using Windows UI Automation structural actions.
85 - Do not use backend-specific actions just because their argument names exist in the generic contract. Treat them as unavailable unless the connected CLI advertises the matching feature.
@@ -88,6 +89,7 @@ If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm requi
89 - Only the latest screenshot or a definitive tool result counts as evidence.
90 - If a tool result says a screenshot was attached but you cannot actually see the image, stop and report that visual verification is unavailable. Do not continue with another action from an assumed host state.
91 - Outside advertised structural accessibility support, use normalized global screen coordinates; do not assume window ids, element indexes, background-safe input, or semantic click targets unless the runtime explicitly advertises them.
92 +- On Linux, AT-SPI structural targeting uses backend-specific actions documented in `host-computer-use-linux`; do not apply macOS AX-specific assumptions unless the backend is macOS.
93 - Prefer accessibility and semantic UI paths first: shortcuts, command palettes, menu accelerators, address/search bars, focus traversal, and other keyboard-accessible controls.
94 - Prefer `key` and `type` over pointer actions whenever a reliable keyboard path exists.
95 - When a menu or popup is open, treat it as the active UI and prefer keyboard navigation over clicking small transient rows by coordinate.
plugins/_a0_connector/tools/computer_use_remote.py
+79
@@ -480,6 +480,20 @@ class ComputerUseRemote(Tool):
480 backend_id = str(data.get("backend_id", "") or "").strip().lower()
481 backend_family = str(data.get("backend_family", "") or "").strip().lower()
482 features = {feature.lower() for feature in self._backend_features(data)}
483 + has_linux_atspi = bool(
484 + features
485 + & {
486 + "atspi-tree-snapshot",
487 + "atspi-structural-targeting",
488 + "atspi-element-action",
489 + "atspi-set-value",
490 + }
491 + )
492 + if backend_id in {"wayland", "x11", "linux"} or backend_family == "linux" or has_linux_atspi:
493 + return (
494 + " Load skill `host-computer-use-linux` before using Linux AT-SPI "
495 + "structural actions."
496 + )
497 has_macos_ax = bool(
498 features
499 & {
@@ -548,6 +562,7 @@ class ComputerUseRemote(Tool):
562 return (
563 f"AX snapshot for {app_name}: {node_count} node(s){truncated}. "
564 f"Root {root_label}. Use path or semantic target fields with ax_action."
565 + f"{self._structural_tree_outline(tree)}"
566 )
567
568 def _format_uia_snapshot(self, data: dict[str, Any]) -> str:
@@ -562,8 +577,72 @@ class ComputerUseRemote(Tool):
577 f"Root {root_label}. Prefer node actions with uia_action; use "
578 f"focus_window/minimize/restore/maximize for windows, and reserve click "
579 f"for a last resort."
580 + f"{self._structural_tree_outline(tree)}"
581 )
582
583 + def _structural_tree_outline(self, tree: dict[str, Any], *, max_lines: int = 80) -> str:
584 + if not tree:
585 + return ""
586 + lines: list[str] = ["", "", "Nodes:"]
587 + truncated = False
588 +
589 + def visit(node: dict[str, Any], depth: int) -> None:
590 + nonlocal truncated
591 + if len(lines) - 3 >= max_lines:
592 + truncated = True
593 + return
594 + lines.append(self._structural_node_line(node, depth=depth))
595 + children = node.get("children")
596 + if not isinstance(children, list):
597 + return
598 + for child in children:
599 + if len(lines) - 3 >= max_lines:
600 + truncated = True
601 + break
602 + if isinstance(child, dict):
603 + visit(child, depth + 1)
604 +
605 + visit(tree, 0)
606 + if truncated:
607 + lines.append("... outline truncated; request a narrower max_depth/max_nodes snapshot if needed.")
608 + return "\n".join(lines)
609 +
610 + def _structural_node_line(self, node: dict[str, Any], *, depth: int) -> str:
611 + indent = " " * max(0, depth)
612 + role = str(node.get("role") or "element")
613 + path = node.get("path", [])
614 + parts = [f"{indent}- path={path} role={role}"]
615 + for key in ("title", "name", "description", "automation_id", "class_name", "selector"):
616 + value = node.get(key)
617 + if isinstance(value, str) and value.strip():
618 + parts.append(f"{key}={value.strip()[:120]!r}")
619 + break
620 + frame = node.get("frame")
621 + if isinstance(frame, dict):
622 + x = frame.get("x", "?")
623 + y = frame.get("y", "?")
624 + width = frame.get("width", "?")
625 + height = frame.get("height", "?")
626 + parts.append(f"frame=({x},{y} {width}x{height})")
627 + actions = node.get("actions")
628 + if isinstance(actions, list) and actions:
629 + names = [
630 + str(item.get("name") or "").strip()
631 + for item in actions
632 + if isinstance(item, dict) and str(item.get("name") or "").strip()
633 + ]
634 + if names:
635 + parts.append(f"actions={','.join(names[:6])}")
636 + states = node.get("states")
637 + if isinstance(states, list) and states:
638 + values = [str(item).strip() for item in states if str(item).strip()]
639 + if values:
640 + parts.append(f"states={','.join(values[:8])}")
641 + text = node.get("text")
642 + if isinstance(text, str) and text.strip():
643 + parts.append(f"text={text.strip()[:120]!r}")
644 + return " ".join(parts)
645 +
646 def _ax_target_label(self, target: dict[str, Any]) -> str:
647 role = str(target.get("role") or "element")
648 title = str(target.get("title") or target.get("description") or target.get("identifier") or "").strip()