Slim browser prompt with progressive skills
Keep the browser tool prompt as a compact callable contract and move detailed workflows into browser-automation, with fragile form guidance chained through browser-form-workflows. Add regression coverage for the skills_tool loading path, host browser setup hint, and prompt token ceiling.
Alessandro committed
Jun 15, 2026 at 03:23 UTC
c56b65803bb0a7a36cecf739266698d3a3167f38
6 files changed
+33
-34
plugins/_browser/AGENTS.md
+3
@@ -22,10 +22,13 @@
22
## Work Guidance
23
24
- Coordinate tool, helper, and panel changes so browser state shown in the UI matches tool behavior.
25
+- Keep `prompts/agent.system.tool.browser.md` as a compact callable contract; move detailed browser workflows into `skills/browser-automation/SKILL.md`.
26
+- Keep fragile form guidance progressively disclosed through `skills/browser-form-workflows/SKILL.md`, linked from the browser prompt through `browser-automation`.
27
28
## Verification
29
30
- Smoke-test browser launch, navigation, DOM capture, and WebUI viewer after runtime changes.
31
+- Run browser prompt/skill regression tests after changing browser prompt or Browser plugin skills.
32
33
## Child DOX Index
34
plugins/_browser/prompts/agent.system.tool.browser.md
+8
-29
@@ -3,36 +3,15 @@ Rendered browser automation for pages that need interaction, JavaScript, forms,
3
4
Prefer `search_engine` or `document_query` for plain text research. The tool must not open a Browser surface automatically. Use the tool headlessly unless the user opens the Browser surface or asks for the optional visible WebUI viewer.
5
6
-The browser may run in Docker container mode or A0 CLI host-browser mode depending on settings. Container-mode paths resolve inside Agent Zero; host-mode paths resolve on the connected A0 CLI host.
6
+When the user asks for "my browser", "host browser", "local browser", local Chrome, or opening a URL in their host browser, use this `browser` tool. Do not substitute `computer_use_remote`, `code_execution_remote`, `xdg-open`, `sensible-browser`, or Python `webbrowser.open`. If setup fails and mentions remote debugging, tell the user to open `chrome://inspect/#remote-debugging`, enable "Allow remote debugging for this browser instance", run `/browser host on`, and retry.
7
8
-When the user asks for "my browser", "host browser", "local browser", local Chrome, or opening a URL in their host browser, use this `browser` tool. Do not substitute `computer_use_remote`, `code_execution_remote`, or host shell launchers such as `xdg-open`, `sensible-browser`, or Python `webbrowser.open`. If host-browser setup fails and mentions remote debugging, stop and tell the user to open `chrome://inspect/#remote-debugging`, enable "Allow remote debugging for this browser instance", run `/browser host on`, and retry.
8
+For rendered browsing workflows, multi-step interaction, screenshots, downloads, uploads, forms, or host/container mode decisions, first load `browser-automation` with `skills_tool:load`, then call this tool using the loaded instructions. For fragile forms, `browser-automation` links to `browser-form-workflows`; load it when selects, checkboxes, radios, uploads, contenteditable fields, validation, or submission state are central.
9
10
-For complex browser workflows, load skill `browser-automation`. For fragile forms, load skill `browser-form-workflows`.
10
+Actions: tabs `open`, `list`, `state`, `set_active`, `navigate`, `back`, `forward`, `reload`, `close`, `close_all`; inspect `content`, `detail`, `screenshot`; interact `click`, `hover`, `double_click`, `right_click`, `drag`, `type`, `submit`, `type_submit`, `scroll`, `select_option`, `set_checked`, `upload_file`; advanced `evaluate`, `key_chord`, `mouse`, `wheel`, `keyboard`, `clipboard`, `set_viewport`, `multi`.
11
12
-Actions: `open`, `list`, `state`, `set_active`, `navigate`, `back`, `forward`, `reload`, `content`, `detail`, `screenshot`, `click`, `hover`, `double_click`, `right_click`, `drag`, `type`, `submit`, `type_submit`, `scroll`, `evaluate`, `key_chord`, `mouse`, `wheel`, `keyboard`, `clipboard`, `set_viewport`, `select_option`, `set_checked`, `upload_file`, `multi`, `close`, `close_all`.
13
-
14
-Common args: `action`, `browser_id`, `url`, `ref`, `target_ref`, `text`, `selector`, `selectors`, `script`, `modifiers`, `keys`, `key`, `include_content`, `focus_popup`, `event_type`, `x`, `y`, `to_x`, `to_y`, `delta_x`, `delta_y`, `button`, `quality`, `full_page`, `path`, `paths`, `value`, `values`, `checked`, `width`, `height`, `calls`.
15
-
16
-Workflow:
17
-- `open` creates a tab and returns id/state. In host-browser mode, if the requested URL is already open, the host may reuse and activate that existing tab.
18
-- If the user asks for an existing tab, page title, or already-open URL, call `list` first, match by `title` or `currentUrl`, then use `set_active` or `navigate` on that `browser_id` instead of opening a new tab.
19
-- `content` returns markdown with refs like `[link 3]`, `[button 6]`, `[input text 8]`.
20
-- Interactions use refs from the latest `content` capture.
21
-- For same-page controls that are easier to identify structurally, `click`, `type`, `submit`, `type_submit`, `scroll`, `select_option`, `set_checked`, and `upload_file` may use `selector` instead of `ref`; the tool resolves the selector through `content` first.
22
-- `click` with `x`/`y` and no `ref` is treated as a coordinate mouse click. `type` with text and no `ref` types into the currently focused element. `key_chord` accepts either `["Control", "A"]` or `"CTRL+A"`.
23
-- `navigate` reuses an existing `browser_id` and is preferred for serial browsing.
24
-- Screenshots are explicit only; the browser does not automatically load screenshots. Call `vision_load` with the returned `vision_load.tool_args.paths` value before reasoning visually. When no `path` is requested, browser screenshots are saved as chat-scoped artifacts; explicit `path` requests remain user-owned files.
12
+Rules:
13
+- If the user asks for an existing tab, page title, or already-open URL, call `list` first, match by `title` or `currentUrl`, then use `set_active` or `navigate` on that `browser_id`.
14
+- Use refs from the latest `content`; same-page controls may use `selector`.
15
+- Screenshots are explicit only; the browser does not automatically load screenshots. Call `vision_load` with the returned `vision_load.tool_args.paths` value before reasoning visually.
16
- Keep the tab set small; close pages after extracting what you need.
26
-
27
-`multi` is only a browser action: use `tool_name: "browser"` with `tool_args.action: "multi"`. Never use `tool_name: "multi"`.
28
-
29
-Example:
30
-~~~json
31
-{
32
- "tool_name": "browser",
33
- "tool_args": {
34
- "action": "open",
35
- "url": "https://example.com"
36
- }
37
-}
38
-~~~
17
+- `multi` is only a browser action: use `tool_name: "browser"` with `tool_args.action: "multi"`. Never use `tool_name: "multi"`.
plugins/_browser/skills/browser-automation/SKILL.md
+5
-2
@@ -5,7 +5,9 @@ description: Use for complex Agent Zero browser automation, including multi-tab
5
6
# Browser Tool
7
8
-Use the `browser` tool for rendered pages, forms, logins, downloads, JavaScript-heavy sites, screenshots, and visual inspection. Prefer `search_engine` or `document_query` for plain text research.
8
+Use this skill after the compact `browser` tool prompt points you here. It is the progressive-disclosure workflow guide for rendered pages, multi-step browser work, logins, downloads, JavaScript-heavy sites, screenshots, host/container browser mode, and visual inspection. Prefer `search_engine` or `document_query` for plain text research.
9
+
10
+For fragile forms, load `browser-form-workflows` with `skills_tool:load` before acting when selects, checkboxes, radios, file uploads, contenteditable fields, validation, or final submission state are central to the task.
11
12
## Core Workflow
13
@@ -14,6 +16,7 @@ Use the `browser` tool for rendered pages, forms, logins, downloads, JavaScript-
16
3. Interact with refs using `click`, `type`, `submit`, `scroll`, etc.
17
4. Use `navigate` on an existing `browser_id` for serial browsing.
18
5. Keep only a small working tab set; close pages when finished.
19
+6. If the user asks for an existing tab, page title, or already-open URL, call `list` first, match by `title` or `currentUrl`, then use `set_active` or `navigate` on that `browser_id` instead of opening a new tab.
20
21
## Modes
22
@@ -39,7 +42,7 @@ Screenshot args include `quality`, `full_page`, and optional `path`. Without `pa
42
- `select_option` works for native selects and detectable ARIA listbox/combobox controls.
43
- `set_checked` works for checkbox, radio, switch, and toggle-like refs.
44
- `upload_file` works for file input refs or associated labels; verify the file exists in the active browser environment.
42
-- For fragile forms, load skill `browser-form-workflows`.
45
+- For fragile forms, call `skills_tool` with `action: "load"` and `skill_name: "browser-form-workflows"`, then follow that form-specific workflow before filling or submitting.
46
47
## Pointer And Keyboard
48
plugins/_browser/skills/browser-form-workflows/SKILL.md
+1
-1
@@ -5,7 +5,7 @@ description: Use for complex Agent Zero Browser form workflows involving selects
5
6
# Browser Forms
7
8
-Use this skill for complex Browser form workflows where the page state may depend on selects, checkboxes, radios, file uploads, contenteditable fields, validation, or visual confirmation.
8
+Use this skill as the form-specific extension of `browser-automation`. Load it after, or alongside, `browser-automation` when the page state may depend on selects, checkboxes, radios, file uploads, contenteditable fields, validation, or visual confirmation.
9
10
Start with `browser:content` to capture current refs, then use `browser:detail` on ambiguous fields before acting. Prefer ref-based form actions before coordinates.
11
plugins/_browser/webui/config.html
+1
-1
@@ -37,7 +37,7 @@
37
class="browser-config-field-help"
38
x-show="$store.browserConfig.config.runtime_backend === 'host_required'"
39
>
40
- Uses Chrome, Edge, or Chromium on the A0 CLI host. Keep A0 CLI connected; the browser opens when the agent first needs it.
40
+ Uses Chrome, Edge, or Chromium on the A0 CLI host. Keep A0 CLI connected; the browser opens when the agent first needs it. For an already-open browser, use chrome://inspect/#remote-debugging and enable "Allow remote debugging for this browser instance".
41
</span>
42
</label>
43
tests/test_browser_agent_regressions.py
+15
-1
@@ -896,6 +896,8 @@ def test_browser_tool_does_not_auto_open_canvas_policy_is_documented():
896
prompt = (
897
PROJECT_ROOT / "plugins" / "_browser" / "prompts" / "agent.system.tool.browser.md"
898
).read_text(encoding="utf-8")
899
+ from helpers import tokens
900
+
901
config = (PROJECT_ROOT / "plugins" / "_browser" / "default_config.yaml").read_text(
902
encoding="utf-8"
903
)
@@ -912,21 +914,33 @@ def test_browser_tool_does_not_auto_open_canvas_policy_is_documented():
914
assert "set_checked" in prompt
915
assert "upload_file" in prompt
916
assert "browser-form-workflows" in prompt
917
+ assert "first load `browser-automation` with `skills_tool:load`" in prompt
918
+ assert "`browser-automation` links to `browser-form-workflows`" in prompt
919
assert "does not automatically load screenshots" in prompt
920
assert "chrome://inspect/#remote-debugging" in prompt
921
+ assert tokens.approximate_tokens(prompt) <= 650
922
assert "already open" in config
923
assert "already-open Browser surface" in config_html
924
assert "chrome://inspect/#remote-debugging" in config_html
925
926
922
-def test_browser_forms_skill_is_plugin_owned_and_discoverable():
927
+def test_browser_skills_are_plugin_owned_and_progressively_linked():
928
+ automation_path = (
929
+ PROJECT_ROOT / "plugins" / "_browser" / "skills" / "browser-automation" / "SKILL.md"
930
+ )
931
skill_path = PROJECT_ROOT / "plugins" / "_browser" / "skills" / "browser-form-workflows" / "SKILL.md"
932
+ assert automation_path.exists()
933
assert skill_path.exists()
934
+ automation = automation_path.read_text(encoding="utf-8")
935
skill = skill_path.read_text(encoding="utf-8")
936
assert skill.startswith("---\n")
937
frontmatter = skill.split("---", 2)[1]
938
assert "name: browser-form-workflows" in frontmatter
939
assert "description:" in frontmatter
940
+ assert "progressive-disclosure workflow guide" in automation
941
+ assert "`browser-form-workflows` with `skills_tool:load`" in automation
942
+ assert 'skill_name: "browser-form-workflows"' in automation
943
+ assert "form-specific extension of `browser-automation`" in skill
944
assert "select_option" in skill
945
assert "set_checked" in skill
946
assert "upload_file" in skill