improve computer-use screenshot refresh guidance
Add post-action settle/fresh-capture handling for computer_use_remote, include capture ids and coordinate-space summaries in screenshot attachments, and tighten prompt guidance so agents use the latest capture without assuming semantic/window targeting.
Alessandro committed
Apr 24, 2026 at 14:27 UTC
603fc2064b250b4ce391d36a67d27e717e55778d
5 files changed
+50
-18
plugins/_a0_connector/plugin.yaml
+1
-1
@@ -1,7 +1,7 @@
1
name: _a0_connector
2
title: A0 Connector
3
description: Current Agent Zero connector plugin for HTTP plus /ws integration, using session auth and handler activation through auth.handlers.
4
-version: 1.5
4
+version: "1.5"
5
settings_sections:
6
- external
7
- developer
plugins/_a0_connector/prompts/agent.extras.computer_use_remote.md
+5
-1
@@ -7,10 +7,14 @@ Features: `{{features}}`
7
Support note: `{{support_reason}}`
8
9
- Use this for local desktop and native UI tasks on the connected machine.
10
-- If the task is browser-only and the user is flexible, prefer `browser_agent` because it is usually more reliable and token-efficient than screenshot-driven desktop control.
10
+- If the task is browser-only and the user is flexible, prefer the direct `browser` tool because it is usually more reliable and token-efficient than screenshot-driven desktop control.
11
- Use `start_session` before interactive desktop actions. `status` is for inspection; `stop_session` ends the session.
12
- Base every decision on the latest screenshot or a definitive tool result, not memory.
13
+- The current action API uses normalized global screen coordinates. Do not assume window IDs, element indices, background-safe input, or semantic click targets unless the advertised features explicitly say they exist.
14
+- If features include `real-cursor-may-move` or `focus-risk`, expect pointer actions to affect the visible desktop state; prefer keyboard/accessibility routes even more strongly.
15
- Successful `start_session`, `move`, `click`, `scroll`, `key`, and `type` calls already attach a fresh screenshot.
16
+- Each attached screenshot includes a `capture id`; treat the latest attached capture as authoritative and ignore superseded capture references.
17
+- If an attached screenshot looks unchanged after a state-changing action, use one explicit `capture` to verify before repeating the same action.
18
- Use `capture` only when you need a screen refresh without taking another action.
19
- Prefer accessibility and semantic UI paths first: application shortcuts, command palettes, menu accelerators, address/search bars, focus traversal, selection shortcuts, and other keyboard-accessible controls.
20
- Prefer `key` and `type` over pointer actions whenever there is a plausible keyboard or accessibility path. Use `tab`, `shift+tab`, arrow keys, hotkeys, text search, and submit keys before reaching for the mouse.
plugins/_a0_connector/prompts/agent.system.tool.computer_use_remote.md
+3
@@ -21,7 +21,10 @@ Action-specific fields:
21
- `type`: `text`, optional `submit` boolean
22
23
## Runtime Notes
24
+- The current action API uses normalized global screen coordinates; do not assume window IDs, element indices, background-safe input, or semantic click targets unless runtime guidance explicitly advertises them.
25
- Successful `start_session`, `move`, `click`, `scroll`, `key`, and `type` calls automatically attach a fresh screenshot.
26
+- Attached screenshots include a `capture id`; use the latest capture as the coordinate basis.
27
+- If the attached screenshot appears unchanged after a state-changing action, verify once with `capture` before repeating the same action.
28
- `status` reports the current computer-use state without starting a session.
29
- Prefer accessibility, semantic UI controls, hotkeys, focus traversal, and other keyboard paths before pointer actions.
30
- For viewport movement, prefer keyboard scrolling first; use `scroll` when a wheel-style scroll is the most reliable way to move an already-focused viewport or pane.
plugins/_a0_connector/tools/computer_use_remote.py
+40
-15
@@ -30,9 +30,15 @@ _AUTO_CAPTURE_ACTIONS = {
30
"type",
31
}
32
_SETTLE_DELAY_START_SESSION = 0.2
33
+_SETTLE_DELAY_MOVE = 0.1
34
+_SETTLE_DELAY_CLICK = 0.35
35
+_SETTLE_DELAY_SCROLL = 0.35
36
+_SETTLE_DELAY_KEY = 0.2
37
+_SETTLE_DELAY_TYPE = 0.25
38
_SETTLE_DELAY_GLOBAL_FOCUS = 0.45
39
_SETTLE_DELAY_PLAIN_ENTER = 0.3
40
_SETTLE_DELAY_SUBMIT = 0.45
41
+_FRESH_CAPTURE_TIMEOUT = 0.45
42
_SUPPORTED_ACTIONS = {
43
"start_session",
44
"status",
@@ -160,15 +166,15 @@ class ComputerUseRemote(Tool):
166
if settle_seconds > 0:
167
await asyncio.sleep(settle_seconds)
168
163
- capture_result = await self._dispatch_payload(
164
- sid=sid,
165
- payload={
166
- "op_id": str(uuid.uuid4()),
167
- "context_id": context_id,
168
- "action": "capture",
169
- "session_id": session_id,
170
- },
171
- )
169
+ capture_payload = {
170
+ "op_id": str(uuid.uuid4()),
171
+ "context_id": context_id,
172
+ "action": "capture",
173
+ "session_id": session_id,
174
+ "fresh": True,
175
+ "fresh_timeout_seconds": _FRESH_CAPTURE_TIMEOUT,
176
+ }
177
+ capture_result = await self._dispatch_payload(sid=sid, payload=capture_payload)
178
if not bool(capture_result.get("ok")):
179
return f"Automatic screen refresh failed: {self._format_error(capture_result)}"
180
@@ -176,14 +182,22 @@ class ComputerUseRemote(Tool):
182
if not isinstance(capture_data, dict):
183
return "Automatic screen refresh failed: missing capture payload."
184
179
- self._record_capture(capture_data)
180
- return "Latest screen attached."
185
+ summary = self._record_capture(capture_data)
186
+ return f"Latest screen attached: {summary}"
187
188
def _auto_capture_settle_seconds(self, action: str) -> float:
189
if action == "start_session":
190
return _SETTLE_DELAY_START_SESSION
191
+ if action == "move":
192
+ return _SETTLE_DELAY_MOVE
193
+ if action == "click":
194
+ return _SETTLE_DELAY_CLICK
195
+ if action == "scroll":
196
+ return _SETTLE_DELAY_SCROLL
197
if action == "type" and self._coerce_bool(self.args.get("submit")):
198
return _SETTLE_DELAY_SUBMIT
199
+ if action == "type":
200
+ return _SETTLE_DELAY_TYPE
201
if action != "key":
202
return 0.0
203
@@ -192,7 +206,7 @@ class ComputerUseRemote(Tool):
206
return _SETTLE_DELAY_GLOBAL_FOCUS
207
if keyset == {"enter"}:
208
return _SETTLE_DELAY_PLAIN_ENTER
195
- return 0.0
209
+ return _SETTLE_DELAY_KEY
210
211
def _requested_keys(self) -> list[str]:
212
keys_value = self.args.get("keys")
@@ -252,8 +266,8 @@ class ComputerUseRemote(Tool):
266
data = {}
267
268
if action == "capture":
255
- self._record_capture(data)
256
- return "Current screen attached."
269
+ summary = self._record_capture(data)
270
+ return f"Current screen attached: {summary}"
271
if action == "status":
272
return self._format_status(data)
273
if action == "start_session":
@@ -309,7 +323,18 @@ class ComputerUseRemote(Tool):
323
_image_path, display_path = self._resolve_capture_path(data)
324
width = data.get("width", "?")
325
height = data.get("height", "?")
312
- summary = f"Computer-use capture {width}x{height}."
326
+ capture_id = str(data.get("capture_id") or Path(display_path).stem or "?").strip()
327
+ coordinate_space = str(data.get("coordinate_space") or "normalized_global_screen").strip()
328
+ summary = (
329
+ f"Computer-use capture id={capture_id} {width}x{height}, "
330
+ f"coordinates={coordinate_space} [0,1]."
331
+ )
332
+ if data.get("fresh") is True:
333
+ if "fresh_after_satisfied" in data:
334
+ fresh_state = "confirmed" if data.get("fresh_after_satisfied") is not False else "not confirmed"
335
+ summary = f"{summary} Fresh frame {fresh_state}."
336
+ else:
337
+ summary = f"{summary} Fresh capture requested."
338
content = [
339
{"type": "text", "text": summary},
340
{"type": "image_url", "image_url": {"url": display_path}},
skills/computer-use-remote/SKILL.md
+1
-1
@@ -23,7 +23,7 @@ allowed_tools:
23
24
Load this skill before using `computer_use_remote` for local desktop and native UI tasks on the connected machine.
25
26
-For ordinary website browsing, search, form filling, and web downloads, prefer `browser_agent` instead. If the user is flexible and the task is browser-only, guide them toward browser tools because they are usually more reliable and token-efficient than screenshot-driven computer use.
26
+For ordinary website browsing, search, form filling, and web downloads, prefer the direct `browser` tool instead. If the user is flexible and the task is browser-only, guide them toward browser tools because they are usually more reliable and token-efficient than screenshot-driven computer use.
27
28
## Core Loop
29