Route Computer Use approval through Launcher
Make /computer-use on and off act on the current Launcher Host access lease while preserving A0 CLI guidance in ordinary WebUI sessions. Update connector prompts and regressions so permission recovery consistently accounts for both Launcher and CLI entry points.
Alessandro committed
Jul 16, 2026 at 13:26 UTC
38a594544183c71df5c7dacad8d5e4da4b2e6fc4
12 files changed
+153
-26
plugins/_a0_connector/AGENTS.md
+3
-1
@@ -52,7 +52,9 @@
52
border-radius tokens for its controls and popover. After Disconnect is
53
acknowledged, keep the popover open and turn that same action into Reconnect;
54
use only the narrow Launcher WebContents bridge so this also works in a
55
- detached Instance window. Pausing remains the reversible master switch.
55
+ detached Instance window. `/computer-use on` may use that same bounded bridge
56
+ to request approval for the current lease, while `/computer-use off` disables
57
+ only the gateway scope. Pausing remains the reversible master switch.
58
- File operation results may arrive as chunked JSON/base64
59
`connector_file_op_result` frames; resolve the pending file operation only
60
after all chunks for the `op_id` are assembled.
plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md
+2
-2
@@ -1,12 +1,12 @@
1
### computer_use_remote
2
3
-Shown when a connected A0 CLI advertises enabled Computer Use (`/computer-use on`) and does not need re-arming. Runtime-gated beta desktop control through that CLI on the user's host machine. Availability, backend support, and trust mode are checked when the tool runs, together with CLI presence, local enablement, and re-arm state. Computer Use enablement is scoped to the current CLI session, not scoped to a single chat context.
3
+Shown when a connected A0 CLI or Launcher host gateway advertises enabled Computer Use (`/computer-use on`) and does not need re-arming. Runtime-gated beta desktop control through that host bridge on the user's machine. Availability, backend support, and trust mode are checked when the tool runs, together with host-bridge presence, local enablement, and re-arm state. Computer Use enablement is scoped to the current CLI session or Launcher Host access lease, not scoped to a single chat context.
4
5
Use this for native host desktop UI inspection, screenshots, background-safe window/element actions when supported, clicking, scrolling, typing, key presses, and status checks. Do not use it for ordinary web-page navigation or host-browser control; use the browser tool for web pages unless browser automation cannot express the task. For complex desktop workflows, load and follow skill `host-computer-use` before proceeding.
6
7
This is the only desktop-control path for the user's connected host/local computer. Do not substitute the `linux-desktop` skill, the Agent Zero Desktop/Xpra surface, `desktopctl.sh`, `code_execution_tool`, or Docker/server shell commands for host screen actions; those target the internal Agent Zero runtime and cannot see or control the user's host screen.
8
9
-If the tool reports no CLI, disabled computer use, or `COMPUTER_USE_REARM_REQUIRED`, stop and tell the user to run `/computer-use on` in A0 CLI and approve any host permission prompt.
9
+If the tool reports no CLI/Launcher host bridge, disabled computer use, or `COMPUTER_USE_REARM_REQUIRED`, stop and tell the user to run `/computer-use on` in the A0 Launcher chat when using Launcher Host access, or in A0 CLI otherwise, and approve any host permission prompt.
10
11
Call `start_session` before screen-driven tasks. Use `status` for state only, `capture` for screenshots without an action, and `stop_session` when the desktop task is complete. Read `backend_id`, `backend_family`, `features`, and the structured `capabilities` object in status/session results. When capabilities report native windows, window state, element indexes, and background dispatch, prefer `list_windows` -> `get_window_state` -> `element_action` with `dispatch: "background"` before using global coordinates. Interactive coordinate actions should use normalized global-screen coordinates from the most recent capture.
12
plugins/_a0_connector/skills/host-computer-use-macos/SKILL.md
+1
-1
@@ -82,6 +82,6 @@ When a visible close/minimize/full-screen button or menu item is accessible in t
82
83
## Permissions
84
85
-macOS Screen Recording affects screenshots. Accessibility/Input Monitoring affect structural targeting and input. If `computer_use_remote` returns `COMPUTER_USE_REARM_REQUIRED`, `COMPUTER_USE_APPROVAL_REQUIRED`, or `status=rearm required`, stop immediately and ask the user to run `/computer-use on` in the A0 CLI and approve the macOS prompt if shown.
85
+macOS Screen Recording affects screenshots. Accessibility/Input Monitoring affect structural targeting and input. If `computer_use_remote` returns `COMPUTER_USE_REARM_REQUIRED`, `COMPUTER_USE_APPROVAL_REQUIRED`, or `status=rearm required`, stop immediately and ask the user to run `/computer-use on` in the A0 Launcher chat when using Launcher Host access, or in A0 CLI otherwise, and approve the macOS prompt if shown.
86
87
Do not bypass a permission failure with server screenshots, Docker commands, `linux-desktop`, or browser fallbacks.
plugins/_a0_connector/skills/host-computer-use/SKILL.md
+4
-4
@@ -17,7 +17,7 @@ triggers:
17
18
# Host Computer Use
19
20
-This skill unlocks the beta `computer_use_remote` tool for connected local desktop control through A0 CLI. Prefer native background-safe computer use when the connected backend advertises window and element-index features.
20
+This skill unlocks the beta `computer_use_remote` tool for connected local desktop control through A0 CLI or a Launcher host gateway. Prefer native background-safe computer use when the connected backend advertises window and element-index features.
21
22
## When to Use
23
@@ -31,7 +31,7 @@ If the task needs shell execution on the CLI host, load `host-code-execution` se
31
32
This skill controls the user's connected host/local computer through A0 CLI. It is not the built-in Linux Desktop/Xpra skill.
33
34
-Never switch to `linux-desktop`, the Agent Zero Desktop/Xpra surface, `desktopctl.sh`, `code_execution_tool`, or Docker/server shell commands as a fallback for host screen actions such as screenshots, clicking, typing, desktop state changes, or checking visible host UI. Those paths only see the internal Agent Zero runtime. If `computer_use_remote` is unavailable, disabled, or needs re-arming, stop and ask the user to run `/computer-use on` in the A0 CLI and approve the platform permission prompt.
34
+Never switch to `linux-desktop`, the Agent Zero Desktop/Xpra surface, `desktopctl.sh`, `code_execution_tool`, or Docker/server shell commands as a fallback for host screen actions such as screenshots, clicking, typing, desktop state changes, or checking visible host UI. Those paths only see the internal Agent Zero runtime. If `computer_use_remote` is unavailable, disabled, or needs re-arming, stop and ask the user to run `/computer-use on` in the A0 Launcher chat when using Launcher Host access, or in A0 CLI otherwise, and approve the platform permission prompt.
35
36
## Browser Boundary
37
@@ -68,9 +68,9 @@ Arguments:
68
- `key`: `key` or `keys`
69
- `type`: `text`, optional `submit` boolean
70
71
-Availability, backend support, and trust mode are checked when the tool runs. If no CLI is connected or local computer use is disabled, tell the user what to enable instead of using the server environment.
71
+Availability, backend support, and trust mode are checked when the tool runs. If no A0 CLI or Launcher host gateway is connected, or local computer use is disabled, tell the user what to enable instead of using the server environment.
72
73
-If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm required`, stop the computer-use sequence immediately. Do not retry `start_session`, do not call `capture`, and do not use shell, vision, or screenshot fallbacks to bypass it. Tell the user that the A0 CLI has Computer Use configured but the installed desktop-control backend is not armed; they should run `/computer-use on` in the A0 CLI and approve the platform permission prompt if shown.
73
+If any tool result contains `COMPUTER_USE_REARM_REQUIRED` or `status=rearm required`, stop the computer-use sequence immediately. Do not retry `start_session`, do not call `capture`, and do not use shell, vision, or screenshot fallbacks to bypass it. Tell the user that the connected host has Computer Use configured but the installed desktop-control backend is not armed; they should run `/computer-use on` in the A0 Launcher chat when using Launcher Host access, or in A0 CLI otherwise, and approve the platform permission prompt if shown.
74
75
## Core Loop
76
plugins/_a0_connector/tools/computer_use_remote.py
+3
-2
@@ -487,8 +487,9 @@ class ComputerUseRemote(Tool):
487
return (
488
"COMPUTER_USE_REARM_REQUIRED: "
489
f"{detail} Stop using computer_use_remote for now; ask the user to re-arm "
490
- "Computer Use in the A0 CLI with /computer-use on and approve the platform "
491
- "permission prompt if shown. "
490
+ "Computer Use with /computer-use on in the A0 Launcher chat when using "
491
+ "Launcher Host access, or in A0 CLI, and approve the platform permission "
492
+ "prompt if shown. "
493
"Do not retry or use screenshot fallbacks."
494
)
495
if code:
plugins/_a0_connector/webui/launcher-gateway-store.js
+28
-1
@@ -109,7 +109,34 @@ const model = {
109
};
110
if (!scopes.files) scopes.file_write = false;
111
if (!scopes.file_write) scopes.code_execution = false;
112
- await this.control({ action: "replace_scopes", scopes });
112
+ return this.control({ action: "replace_scopes", scopes });
113
+ },
114
+
115
+ async setComputerUse(enabled) {
116
+ if (!this.gateway) await this.refresh();
117
+ if (!this.gateway) throw new Error("Launcher Host access is not connected.");
118
+ if (!this.gateway.master_enabled) {
119
+ throw new Error("Launcher Host access is paused. Resume it first.");
120
+ }
121
+
122
+ const requested = Boolean(enabled);
123
+ if (requested && typeof window.a0LauncherHost?.rearmComputerUse !== "function") {
124
+ throw new Error(
125
+ "This A0 Launcher cannot request Computer Use permission. Update it or use A0 CLI.",
126
+ );
127
+ }
128
+ if (Boolean(this.gateway.scopes?.computer_use) !== requested) {
129
+ if (!await this.setScope("computer_use", requested)) return false;
130
+ }
131
+ if (!requested) return true;
132
+
133
+ const response = await window.a0LauncherHost.rearmComputerUse();
134
+ if (!response?.ok) {
135
+ throw new Error(
136
+ response?.message || "Computer Use permission could not be requested.",
137
+ );
138
+ }
139
+ return true;
140
},
141
142
async emergencyDisconnect() {
plugins/_commands/AGENTS.md
+3
@@ -30,6 +30,9 @@
30
- On startup, `_commands` copies legacy `usr/plugins/commands` command and skill files into `usr/plugins/_commands` without overwriting existing files, copies scoped legacy command folders to `_commands`, and disables the legacy `commands` plugin roots to prevent duplicate WebUI popovers.
31
- Script commands must expose `run(payload)` and return a string or a dict with `text` and optional `effects`; `show_markdown` effects render as auto-dismissing toast notifications.
32
- Script commands may emit `send_message` with `text` to submit the rendered composer text immediately after command resolution.
33
+- Built-in `/computer-use on|off` emits a bounded `computer_use` effect. A
34
+ Launcher-owned WebUI applies it to that tab's Host access lease; an ordinary
35
+ WebUI directs the user to run the same command in A0 CLI.
36
37
## Work Guidance
38
plugins/_commands/commands/connector_commands.py
+37
-15
@@ -57,7 +57,7 @@ def run(payload: dict[str, Any]) -> dict[str, Any]:
57
if command == "attach":
58
return _effects({"type": "attach_files"})
59
if command == "computer-use":
60
- return _show_markdown("Computer Use", _computer_use_status(context_id, raw_args))
60
+ return _handle_computer_use(context_id, raw_args)
61
if command == "copy":
62
return _effects({"type": "copy_transcript"})
63
if command == "status":
@@ -143,6 +143,30 @@ def _handle_browser(context: AgentContext | None, raw_args: str) -> dict[str, An
143
return _effects(_toast(f"Browser runtime set to {label}."))
144
145
146
+def _handle_computer_use(context_id: str, raw_args: str) -> dict[str, Any]:
147
+ action = (
148
+ "-".join(
149
+ part.strip().lower().replace("_", "-") for part in raw_args.split()
150
+ )
151
+ or "status"
152
+ )
153
+ enabled = action in {"on", "enable", "enabled", "true", "yes", "1"}
154
+ disabled = action in {"off", "disable", "disabled", "false", "no", "0"}
155
+ if enabled or disabled:
156
+ command = "on" if enabled else "off"
157
+ return _effects(
158
+ {
159
+ "type": "computer_use",
160
+ "enabled": enabled,
161
+ "fallback": (
162
+ "Computer Use permissions are controlled on the connected host. "
163
+ f"Run `/computer-use {command}` in the A0 CLI terminal."
164
+ ),
165
+ }
166
+ )
167
+ return _show_markdown("Computer Use", _computer_use_status(context_id))
168
+
169
+
170
def _browser_status(context: AgentContext | None) -> str:
171
project_name = projects.get_context_project_name(context) if context else ""
172
settings = plugins.get_plugin_config("_browser", project_name=project_name or "", agent_profile="") or {}
@@ -267,25 +291,21 @@ def _status(context: AgentContext | None) -> str:
291
)
292
293
270
-def _computer_use_status(context_id: str, raw_args: str) -> str:
294
+def _computer_use_status(context_id: str) -> str:
295
from plugins._a0_connector.helpers import ws_runtime
296
273
- action = "-".join(part.strip().lower().replace("_", "-") for part in raw_args.split()) or "status"
274
- sids = ws_runtime.remote_tool_sids_for_context(context_id) if context_id else sorted(ws_runtime.connected_sids())
297
+ sids = (
298
+ ws_runtime.remote_tool_sids_for_context(context_id)
299
+ if context_id
300
+ else sorted(ws_runtime.connected_sids())
301
+ )
302
if not sids:
303
return (
277
- "No A0 CLI is connected to this WebUI session.\n\n"
278
- "Computer Use requires the CLI because the desktop permission prompt and native backend live on the CLI host. "
279
- "Start A0 CLI, connect it to this Agent Zero instance, then run `/computer-use on` in the CLI."
304
+ "No A0 CLI or Launcher host gateway is connected to this WebUI session.\n\n"
305
+ "Open this Instance in A0 Launcher with Host access, or connect A0 CLI, then run `/computer-use on`."
306
)
307
282
- if action in {"on", "off", "enable", "disable", "enabled", "disabled", "true", "false", "yes", "no", "1", "0"}:
283
- return (
284
- "Computer Use must be armed from the connected A0 CLI because it controls local desktop permissions.\n\n"
285
- "Run `/computer-use on` or `/computer-use off` in the CLI terminal."
286
- )
287
-
288
- lines = ["Connected A0 CLI sessions:"]
308
+ lines = ["Connected host-control sessions:"]
309
for sid in sids:
310
metadata = ws_runtime.computer_use_metadata_for_sid(sid) or {}
311
if not metadata:
@@ -297,7 +317,9 @@ def _computer_use_status(context_id: str, raw_args: str) -> str:
317
detail = str(metadata.get("last_error") or metadata.get("support_reason") or "").strip()
318
suffix = f" ({detail})" if detail else ""
319
lines.append(f"- `{sid}`: {state}, {supported}, status: {status}{suffix}")
300
- lines.append("\nUse `/computer-use on|off|status` in the CLI to change local Computer Use.")
320
+ lines.append(
321
+ "\nUse `/computer-use on|off` in A0 Launcher or A0 CLI to change local Computer Use."
322
+ )
323
return "\n".join(lines)
324
325
plugins/_commands/tests/test_commands_plugin.py
+28
@@ -233,6 +233,34 @@ def test_models_command_always_opens_modal():
233
}
234
235
236
+@pytest.mark.parametrize(("argument", "enabled"), [("on", True), ("off", False)])
237
+def test_computer_use_command_dispatches_to_launcher_or_cli(
238
+ argument: str,
239
+ enabled: bool,
240
+):
241
+ result = connector_commands.run(
242
+ {
243
+ "invocation": {
244
+ "command_name": "computer-use",
245
+ "raw_arguments": argument,
246
+ },
247
+ "context": {"context_id": ""},
248
+ }
249
+ )
250
+
251
+ assert result["text"] == ""
252
+ assert result["effects"] == [
253
+ {
254
+ "type": "computer_use",
255
+ "enabled": enabled,
256
+ "fallback": (
257
+ "Computer Use permissions are controlled on the connected host. "
258
+ f"Run `/computer-use {argument}` in the A0 CLI terminal."
259
+ ),
260
+ }
261
+ ]
262
+
263
+
264
@pytest.mark.asyncio
265
async def test_commands_api_crud_and_resolve_text_and_script(
266
scope_fixture: ScopeFixture,
plugins/_commands/webui/commands-slash-store.js
+29
@@ -419,6 +419,35 @@ const model = {
419
);
420
continue;
421
}
422
+ if (type === "computer_use") {
423
+ hadToast = true;
424
+ if (typeof window.a0LauncherHost?.getState !== "function") {
425
+ notifyInfo(
426
+ "Computer Use",
427
+ String(effect.fallback || "Run this command in A0 CLI."),
428
+ );
429
+ continue;
430
+ }
431
+ try {
432
+ const { store } = await import(
433
+ "/plugins/_a0_connector/webui/launcher-gateway-store.js"
434
+ );
435
+ const enabled = Boolean(effect.enabled);
436
+ if (!await store.setComputerUse(enabled)) {
437
+ hadError = true;
438
+ continue;
439
+ }
440
+ notifySuccess(
441
+ enabled
442
+ ? "Computer Use permission requested. Approve the system dialog."
443
+ : "Computer Use is off for this Launcher host.",
444
+ );
445
+ } catch (error) {
446
+ hadError = true;
447
+ notifyError(error?.message || "Computer Use could not be changed.");
448
+ }
449
+ continue;
450
+ }
451
if (type === "goal_changed") {
452
window.dispatchEvent(new CustomEvent("goal:changed", { detail: effect }));
453
continue;
tests/test_a0_connector_launcher_gateway.py
+14
@@ -100,6 +100,20 @@ def test_launcher_gateway_disconnect_becomes_reconnect_through_the_launcher_brid
100
assert 'state: "connecting", connected: false, gateway: null' in source
101
102
103
+def test_launcher_computer_use_command_uses_the_bounded_approval_bridge() -> None:
104
+ root = Path(__file__).parents[1]
105
+ gateway_source = (
106
+ root / "plugins" / "_a0_connector" / "webui" / "launcher-gateway-store.js"
107
+ ).read_text(encoding="utf-8")
108
+ commands_source = (
109
+ root / "plugins" / "_commands" / "webui" / "commands-slash-store.js"
110
+ ).read_text(encoding="utf-8")
111
+
112
+ assert "async setComputerUse(enabled)" in gateway_source
113
+ assert "window.a0LauncherHost.rearmComputerUse()" in gateway_source
114
+ assert 'type === "computer_use"' in commands_source
115
+
116
+
117
def test_launcher_gateway_is_fallback_after_context_bound_cli() -> None:
118
context_id = f"ctx-{uuid.uuid4()}"
119
cli_sid = _sid("cli")
tests/test_a0_connector_prompt_gating.py
+1
@@ -268,6 +268,7 @@ def test_computer_use_remote_prompt_keeps_runtime_failures_actionable():
268
assert "disabled computer use" in prompt
269
assert "COMPUTER_USE_REARM_REQUIRED" in prompt
270
assert "/computer-use on" in prompt
271
+ assert "A0 Launcher chat" in prompt
272
273
274
def test_computer_use_remote_prompt_requires_visual_verification_after_actions():