@samitouri / QOSami_Agents_Zero-2 / commits / a174072e

Generalize Desktop CLI agent guidance

Refine the Linux Desktop skill so agents distinguish shell prompts from target CLI prompts before sending natural-language input. Use a generic nested CLI-agent launch pattern with TARGET_CLI and optional FALLBACK_CMD instead of Codex-specific instructions, and add canvas setup assertions to keep the guidance generic.

Alessandro committed May 5, 2026 at 17:18 UTC a174072e67ded2bc4128f8a0da46d042ae042a7d
2 files changed +59 -28
plugins/_office/skills/linux-desktop/SKILL.md
+53 -28
@@ -38,23 +38,24 @@ Keep these standing rules:
38 2. Treat ODF as first-class for LibreOffice office work: ODT in Writer, ODS in Calc, ODP in Impress. Use DOCX/XLSX/PPTX only for explicit Microsoft compatibility.
39 3. Use the Desktop only when the user asks for the Desktop, a GUI app, binary Office visual work, or visual confirmation.
40 4. Never open the Desktop/canvas automatically from a tool result if the user has not opened it. Offer the explicit Open in canvas action instead.
41 -5. Launch common apps from the Desktop icons, the header buttons, or `scripts/desktopctl.sh`.
41 +5. Launch common apps from the Desktop icons, the header buttons, or `/a0/plugins/_office/skills/linux-desktop/scripts/desktopctl.sh`.
42 6. Use the external Agent Zero Browser for web browsing. Do not launch an operating-system browser in this version.
43 7. Verify GUI work by observing the desktop state, checking window titles, and saving the file before reporting success. If exact terminal text matters, load or inspect the screenshot path returned by the final observation, not a screenshot captured before the text appeared.
44
45 ## Control Flow
46
47 -Use the helper script when the Desktop is already open and you need reliable app launches, clicks, keystrokes, or window checks from the agent shell:
47 +Use the helper script when the Desktop is already open and you need reliable app launches, clicks, keystrokes, or window checks from the agent shell. In the live Agent Zero runtime, prefer the absolute path so the command works from any current directory:
48
49 ```bash
50 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh check
51 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh state --json
52 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh observe --json --screenshot
53 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh launch calc
54 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh wait-window LibreOffice
55 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh windows LibreOffice
56 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh focus LibreOffice
57 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh key ctrl+s
50 +DESKTOP=/a0/plugins/_office/skills/linux-desktop/scripts/desktopctl.sh
51 +$DESKTOP check
52 +$DESKTOP state --json
53 +$DESKTOP observe --json --screenshot
54 +$DESKTOP launch calc
55 +$DESKTOP wait-window LibreOffice
56 +$DESKTOP windows LibreOffice
57 +$DESKTOP focus LibreOffice
58 +$DESKTOP key ctrl+s
59 ```
60
61 The script targets the persistent `agent-zero-desktop` X display, sets `DISPLAY`, `XAUTHORITY`, and `HOME` to the XFCE profile, then uses `xdotool` for input. Startup normally prepares this session. If `check` fails during explicit Desktop work, report that the Desktop runtime is not ready instead of installing packages ad hoc.
@@ -62,21 +63,23 @@ The script targets the persistent `agent-zero-desktop` X display, sets `DISPLAY`
63 For direct app launches without coordinates:
64
65 ```bash
65 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh launch writer
66 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh launch calc
67 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh launch impress
68 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh launch terminal
69 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh launch settings
70 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh open-path /a0/usr/workdir
71 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh focus "LibreOffice"
72 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh paste-text "Text to insert"
73 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh key ctrl+s
66 +DESKTOP=/a0/plugins/_office/skills/linux-desktop/scripts/desktopctl.sh
67 +$DESKTOP launch writer
68 +$DESKTOP launch calc
69 +$DESKTOP launch impress
70 +$DESKTOP launch terminal
71 +$DESKTOP launch settings
72 +$DESKTOP open-path /a0/usr/workdir
73 +$DESKTOP focus "LibreOffice"
74 +$DESKTOP paste-text "Text to insert"
75 +$DESKTOP key ctrl+s
76 ```
77
78 For live spreadsheet coworking, use the Calc helper instead of hand-written UNO snippets:
79
80 ```bash
79 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh calc-set-cell /a0/usr/workdir/example.xlsx Sheet1 B2 "Cowork verified live"
81 +DESKTOP=/a0/plugins/_office/skills/linux-desktop/scripts/desktopctl.sh
82 +$DESKTOP calc-set-cell /a0/usr/workdir/example.xlsx Sheet1 B2 "Cowork verified live"
83 ```
84
85 This opens the workbook in the visible Desktop Calc session if needed, changes the cell through LibreOffice, saves the workbook, and verifies the `.xlsx` on disk. Because the edit happens through the running LibreOffice session, the user can see the sheet update without refreshing the Desktop surface.
@@ -84,14 +87,15 @@ This opens the workbook in the visible Desktop Calc session if needed, changes t
87 For coordinate actions, clicks are explicitly last resort. First try `launch`, `open-path`, `wait-window`, `focus`, `key`, `paste-text`, `save`, or an app-native helper. If a coordinate action is still necessary, base it on a fresh screenshot observation and verify immediately afterward:
88
89 ```bash
87 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh observe --json --screenshot
88 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh click 120 180
89 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh dblclick 120 180
90 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh right-click 120 180
91 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh drag 120 180 400 180
92 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh scroll down 3
93 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh type "Text to enter"
94 -plugins/_office/skills/linux-desktop/scripts/desktopctl.sh observe --json
90 +DESKTOP=/a0/plugins/_office/skills/linux-desktop/scripts/desktopctl.sh
91 +$DESKTOP observe --json --screenshot
92 +$DESKTOP click 120 180
93 +$DESKTOP dblclick 120 180
94 +$DESKTOP right-click 120 180
95 +$DESKTOP drag 120 180 400 180
96 +$DESKTOP scroll down 3
97 +$DESKTOP type "Text to enter"
98 +$DESKTOP observe --json
99 ```
100
101 When browser automation is available, the higher-level QA flow is:
@@ -115,6 +119,27 @@ Terminal apps are visual state, not structured logs. When the task depends on ex
119
120 For nested CLI agents, a successful proof requires both the input prompt and the nested agent's visible response in the final screenshot, or another deterministic saved transcript produced by the CLI itself.
121
122 +Guard the boundary between the shell and the target CLI carefully:
123 +
124 +- A shell prompt such as `root@...#` means the target CLI is not currently receiving chat input. Never paste natural-language text into that shell prompt unless it is deliberately quoted as an argument to a shell command.
125 +- If launching a CLI by name can fail, use a shell-safe fallback command and wait for the CLI's own prompt before sending the user's natural-language message.
126 +- After a launch failure such as `command not found`, do not continue by sending the chat message. Start the fallback CLI command or report the blocker.
127 +- A prompt like `>`, `Implement {feature}`, or a CLI-specific input box inside the terminal is different from a shell prompt. Only then should `paste-text "natural language"` followed by `key Return` be used as chat input.
128 +
129 +Example for a nested CLI-agent smoke test:
130 +
131 +```bash
132 +DESKTOP=/a0/plugins/_office/skills/linux-desktop/scripts/desktopctl.sh
133 +$DESKTOP focus "Terminal"
134 +$DESKTOP paste-text 'TARGET_CLI="example-cli-agent"; FALLBACK_CMD=""; if command -v "$TARGET_CLI" >/dev/null 2>&1; then "$TARGET_CLI"; elif [ -n "$FALLBACK_CMD" ]; then sh -lc "$FALLBACK_CMD"; else echo "CLI agent not found: $TARGET_CLI"; fi'
135 +$DESKTOP key Return
136 +$DESKTOP observe --json --screenshot
137 +# Verify the screenshot shows the target CLI prompt, not a shell prompt, before sending natural language:
138 +$DESKTOP paste-text 'Reply with exactly the requested smoke-test token.'
139 +$DESKTOP key Return
140 +$DESKTOP observe --json --screenshot
141 +```
142 +
143 ## Desktop Locations
144
145 The Desktop exposes stable folders for common user work:
tests/test_office_canvas_setup.py
+6
@@ -365,6 +365,7 @@ def test_official_libreoffice_desktop_route_and_packages_are_declared():
365 assert "/a0/usr/workdir" in linux_desktop_skill
366 assert "/a0/usr/projects" in linux_desktop_skill
367 assert "desktopctl.sh" in linux_desktop_skill
368 + assert "/a0/plugins/_office/skills/linux-desktop/scripts/desktopctl.sh" in linux_desktop_skill
369 assert "calc-set-cell" in linux_desktop_skill
370 assert "Clicks are explicitly last resort" in linux_desktop_skill or "clicks are explicitly last resort" in linux_desktop_skill
371 assert "fresh Desktop observation" in linux_desktop_skill
@@ -372,6 +373,11 @@ def test_official_libreoffice_desktop_route_and_packages_are_declared():
373 assert "Terminal And CLI Agent Verification" in linux_desktop_skill
374 assert "Do not report from an earlier screenshot path" in linux_desktop_skill
375 assert "screenshot path returned by that final observation" in linux_desktop_skill
376 + assert "Never paste natural-language text into that shell prompt" in linux_desktop_skill
377 + assert "command not found" in linux_desktop_skill
378 + assert "TARGET_CLI=\"example-cli-agent\"" in linux_desktop_skill
379 + assert "FALLBACK_CMD" in linux_desktop_skill
380 + assert "@openai/codex" not in linux_desktop_skill
381 assert "xdotool" in linux_desktopctl
382 assert "agent-zero-desktop" in linux_desktopctl
383 assert "launch_app" in linux_desktopctl