browser: replace browser-use agent with native browser
Introduce the new built-in Browser plugin for Agent Zero, replacing the legacy browser-use-based browser agent with a direct Playwright-powered browser tool, live WebUI viewer, browser session controls, status APIs, configuration, and extension-management support. Add browser-specific modal behavior so the browser can run as a floating, resizable, no-backdrop window, including modal focus, toggle, and idempotent open helpers for richer WebUI surfaces. Remove the old `_browser_agent` core plugin and the `browser-use` dependency, then clean up stale browser-model wiring and references across agent code, model configuration docs, setup guides, troubleshooting docs, skills, and Agent Zero knowledge. Update regression and WebUI extension-surface coverage for the new browser architecture and modal behavior. The legacy browser-use implementation has been extracted from core so it can continue separately as a community plugin published through the A0 Plugin Index for any user or professional that were relying on it for workflow.
Alessandro committed
Apr 24, 2026 at 15:43 UTC
983d431a5eb785eb9deba9fdfd471fa93f349603
65 files changed
+6934
-1924
README.md
+5
-5
@@ -99,12 +99,12 @@ A detailed setup guide for Windows, macOS, and Linux can be found in the Agent Z
99
100
101
102
-### Browser Agent
102
+### Browser
103
104
-- Browser automation is provided by the built-in `_browser_agent` plugin.
105
-- It uses the effective Main Model resolved by `_model_config`; there is no separate browser model slot.
106
-- Browser vision follows the Main Model's vision setting.
107
-- Playwright Chromium: **Docker** images ship the headless shell preinstalled. **Local development** installs it on first Browser Agent use via `ensure_playwright_binary()` in `plugins/_browser_agent/helpers/playwright.py` (into `tmp/playwright`); you can pre-install manually (see [Development Setup](docs/setup/dev-setup.md)) to skip the wait.
104
+- Browser automation is provided by the built-in `_browser` plugin and the direct `browser` tool.
105
+- The tool uses Playwright operations controlled by the main agent, with typed page refs such as `[link 3]` and `[button 6]`.
106
+- The plugin includes a visible WebUI browser viewer for open sessions.
107
+- Playwright Chromium: **Docker** images ship the headless shell preinstalled. **Local development** installs it on first browser use via `ensure_playwright_binary()` in `plugins/_browser/helpers/playwright.py` (into `tmp/playwright`); you can pre-install manually (see [Development Setup](docs/setup/dev-setup.md)) to skip the wait.
108
109
4. **Completely Customizable and Extensible**
110
agent.py
+1
-5
@@ -740,10 +740,6 @@ class Agent:
740
def get_utility_model(self):
741
return None
742
743
- @extension.extensible
744
- def get_browser_model(self):
745
- return None
746
-
743
@extension.extensible
744
def get_embedding_model(self):
745
return None
@@ -1044,4 +1040,4 @@ class Agent:
1040
message=message,
1041
loop_data=loop_data,
1042
**kwargs,
1047
- )
\ No newline at end of file
1043
+ )
docs/agents/AGENTS.modals.md
+18
@@ -216,6 +216,24 @@ Outcome:
216
- Nested modals don’t “flatten” into each other.
217
- The backdrop always darkens the page behind the active modal without hiding lower modals incorrectly.
218
219
+### Floating no-backdrop modals
220
+
221
+Use `.modal-floating` on the outer `.modal` when a modal should behave like a floating utility panel instead of a blocking dialog. This is for special live surfaces such as the browser panel where the user should keep seeing and interacting with the chat or dashboard behind the panel.
222
+
223
+Working contract:
224
+
225
+- `.modal-floating` suppresses the shared `.modal-backdrop` for that modal.
226
+- `.modal-floating` makes the full-screen `.modal` shell pointer-transparent.
227
+- `.modal-floating .modal-inner` remains pointer-active, so the floating panel itself still receives clicks, keyboard focus, drag handlers, resize handles, and form input.
228
+- Floating modal sizing, dragging, and resizing are still component-owned unless promoted to shared modal CSS later. The modal system only provides the backdrop and pointer-event behavior.
229
+
230
+Good to know:
231
+
232
+- A floating modal does not close by clicking the page behind it, because those clicks pass through to the app. Keep an obvious close button in the modal header.
233
+- If a floating modal opens another normal modal, the normal modal can still use the backdrop; stacking remains governed by the shared z-index logic.
234
+- Use `.modal-no-backdrop` only when a component needs backdrop suppression without click-through floating behavior. Prefer `.modal-floating` for utility panels.
235
+- Do not use `.modal-floating` for destructive confirmations, settings forms, auth, import/export, or workflows that require the user to finish or dismiss the dialog before interacting with the rest of the app.
236
+
237
---
238
239
## Writing a modal component (conventions)
docs/guides/mcp-setup.md
+1
-1
@@ -114,4 +114,4 @@ Community-tested and reliable MCP servers:
114
- **VSCode MCP** - IDE workflows
115
116
> [!TIP]
117
-> For browser automation tasks, the built-in Browser Agent plugin covers the default workflow. MCP-based browser tools are still useful when you need a different browser stack, remote browser control, or an alternative to the built-in Playwright Chromium (preinstalled in Docker; on demand via `ensure_playwright_binary()` in local dev).
117
+> For browser automation tasks, the built-in `_browser` plugin and direct `browser` tool cover the default workflow. MCP-based browser tools are still useful when you need a different browser stack, remote browser control, or an alternative to the built-in Playwright Chromium (preinstalled in Docker; on demand via `ensure_playwright_binary()` in local dev).
docs/guides/projects.md
+1
-1
@@ -221,7 +221,7 @@ SMTP_PASSWORD=email_pwd_here
221
222
### Subagent Configuration
223
224
-Projects can enable or disable specific subagents. This is configured via the UI and stored in `.a0proj/agents.json`. The Browser Agent is not a subagent; it is a built-in plugin.
224
+Projects can enable or disable specific subagents. This is configured via the UI and stored in `.a0proj/agents.json`. The browser tool is not a subagent; it is a built-in plugin.
225
226
### Project LLM Configuration
227
docs/guides/troubleshooting.md
+3
-3
@@ -26,8 +26,8 @@ Refer to the [Choosing your LLMs](../setup/installation.md#installing-and-using-
26
**7. How can I make Agent Zero retain memory between sessions?**
27
Use **Settings → Backup & Restore** and avoid mapping the entire `/a0` directory. See [How to update Agent Zero](../setup/installation.md#how-to-update-agent-zero).
28
29
-**8. My browser agent fails or says Playwright is missing. What now?**
30
-The built-in Browser Agent is a plugin that uses the Main Model from `_model_config`. **Docker:** the Chromium headless shell is shipped preinstalled (typically under `/a0/tmp/playwright`). **Local development:** if the binary is missing, `ensure_playwright_binary()` in `plugins/_browser_agent/helpers/playwright.py` runs `playwright install chromium --only-shell` into `tmp/playwright` on first Browser Agent use (you may see UI notifications). To install ahead of time, run `PLAYWRIGHT_BROWSERS_PATH=tmp/playwright playwright install chromium --only-shell` after `pip install -r requirements.txt`. If you prefer an external browser stack, use MCP alternatives such as Browser OS, Chrome DevTools, or Playwright MCP. See [MCP Setup](mcp-setup.md).
29
+**8. My browser tool fails or says Playwright is missing. What now?**
30
+The built-in browser is provided by the `_browser` plugin and the direct `browser` tool. **Docker:** the Chromium headless shell is shipped preinstalled (typically under `/a0/tmp/playwright`). **Local development:** if the binary is missing, `ensure_playwright_binary()` in `plugins/_browser/helpers/playwright.py` runs `playwright install chromium --only-shell` into `tmp/playwright` on first browser use (you may see UI notifications). To install ahead of time, run `PLAYWRIGHT_BROWSERS_PATH=tmp/playwright playwright install chromium --only-shell` after `pip install -r requirements.txt`. If you prefer an external browser stack, use MCP alternatives such as Browser OS, Chrome DevTools, or Playwright MCP. See [MCP Setup](mcp-setup.md).
31
32
**9. My secrets disappeared after a backup restore.**
33
Secrets are stored in `/a0/usr/secrets.env` and are not always included in backup archives. Copy them manually.
@@ -36,7 +36,7 @@ Secrets are stored in `/a0/usr/secrets.env` and are not always included in backu
36
- Join the Agent Zero [Skool](https://www.skool.com/agent-zero) or [Discord](https://discord.gg/B8KZKNsPpj) community.
37
38
**11. How do I adjust API rate limits?**
39
-Use the model rate limit fields in Settings (Main Model and Utility Model sections) to set request/input/output limits. The Browser Agent inherits the Main Model limits. These map to the model config limits (for example `limit_requests`, `limit_input`, `limit_output`).
39
+Use the model rate limit fields in Settings (Main Model and Utility Model sections) to set request/input/output limits. These map to the model config limits (for example `limit_requests`, `limit_input`, `limit_output`).
40
41
**12. My `code_execution_tool` doesn't work, what's wrong?**
42
- Ensure Docker is installed and running.
docs/guides/usage.md
+2
-2
@@ -126,8 +126,8 @@ Agent Zero's power comes from its ability to use [tools](../developer/architectu
126
127
- **Understand Tools:** Agent Zero includes default tools like knowledge (powered by SearXNG), code execution, and communication. Understand the capabilities of these tools and how to invoke them.
128
129
-### Browser Agent Status & MCP Alternatives
130
-The built-in Browser Agent is provided by the `_browser_agent` plugin. It uses the effective Main Model from `_model_config`, including per-chat overrides and the Main Model vision flag. Playwright Chromium is preinstalled in **Docker**; in **local development** it is installed on demand when needed via `ensure_playwright_binary()` (see [Development Setup](../setup/dev-setup.md) to pre-install).
129
+### Browser Tool Status & MCP Alternatives
130
+The built-in browser is provided by the `_browser` plugin and direct `browser` tool. It uses Playwright operations controlled by the main agent, exposes typed page refs for links, buttons, images, and inputs, and includes a WebUI viewer for open browser sessions. Playwright Chromium is preinstalled in **Docker**; in **local development** it is installed on demand when needed via `ensure_playwright_binary()` (see [Development Setup](../setup/dev-setup.md) to pre-install).
131
132
If you need a different browser stack or want external browser tooling, MCP-based browser tools are still a strong option:
133
docs/setup/dev-setup.md
+1
-1
@@ -69,7 +69,7 @@ Now when you select one of the python files in the project, you should see prope
69
pip install -r requirements.txt
70
PLAYWRIGHT_BROWSERS_PATH=tmp/playwright playwright install chromium --only-shell
71
```
72
-The first command installs Python dependencies. The second installs the Chromium headless shell into `tmp/playwright` ahead of time (same path in Docker: `/a0/tmp/playwright`). If you skip the second command, **local development** still downloads the shell on first Browser Agent use through `ensure_playwright_binary()` in `plugins/_browser_agent/helpers/playwright.py`. Pre-installing avoids that wait. **Docker** images ship the shell preinstalled; runtime install is for local dev when the binary is missing.
72
+The first command installs Python dependencies. The second installs the Chromium headless shell into `tmp/playwright` ahead of time (same path in Docker: `/a0/tmp/playwright`). If you skip the second command, **local development** still downloads the shell on first browser use through `ensure_playwright_binary()` in `plugins/_browser/helpers/playwright.py`. Pre-installing avoids that wait. **Docker** images ship the shell preinstalled; runtime install is for local dev when the binary is missing.
73
Errors in the code editor caused by missing packages should now be gone. If not, try reloading the window.
74
75
docs/setup/installation.md
+2
-2
@@ -405,7 +405,7 @@ The Settings page is the control center for selecting the Large Language Models
405
406
| LLM Role | Description |
407
| --- | --- |
408
-| `chat_llm` | This is the primary LLM used for conversations, agent reasoning, tool use, and the built-in browser agent. Vision support controls browser vision and image understanding. |
408
+| `chat_llm` | This is the primary LLM used for conversations, agent reasoning, and tool use. Vision support controls image understanding. |
409
| `utility_llm` | This LLM handles internal tasks like summarizing messages, managing memory, and processing internal prompts. Using a smaller, less expensive model here can improve efficiency. |
410
| `embedding_llm` | The embedding model shipped with A0 runs on CPU and is responsible for generating embeddings used for memory retrieval and knowledge base lookups. Changing the `embedding_llm` will re-index all of A0's memory. |
411
@@ -416,7 +416,7 @@ The Settings page is the control center for selecting the Large Language Models
416
3. Click "Save" to apply the changes.
417
418
> [!NOTE]
419
-> The Browser Agent does not have a separate model slot. It uses the effective Main Model resolved by `_model_config`, including per-chat overrides and the Main Model vision flag.
419
+> The built-in browser does not have a separate model slot. The main agent decides when to call the direct `browser` tool.
420
421
### Important Considerations
422
knowledge/main/about/capabilities.md
+1
-1
@@ -76,7 +76,7 @@ An external REST API is available for programmatic task submission. Agent-to-Age
76
- **No persistent state between chats** unless explicitly memorized or saved to files.
77
- **Context window**: long conversations are summarized automatically, which can lose detail.
78
- **Memory recall is approximate**: similarity search may miss relevant memories or surface irrelevant ones.
79
-- **No GUI interaction** outside the browser agent (which is separate from the main agent).
79
+- **No GUI interaction** outside built-in browser tooling or configured computer-use integrations.
80
- **Container boundary**: the agent cannot affect systems outside the Docker container unless network access or volume mounts are configured.
81
- **Model capability ceiling**: tool usage quality and reasoning depth are bounded by the underlying LLM. Small models may struggle with complex multi-step tool use.
82
- **No real-time data** beyond web search. The agent's own knowledge cutoff is the underlying model's training cutoff.
knowledge/main/about/configuration.md
+2
-2
@@ -6,11 +6,11 @@ Agent Zero uses three configurable LLM roles:
6
7
| Role | Purpose |
8
|------|---------|
9
-| `chat_llm` | Primary model for all agent reasoning, tool use, and the Browser Agent |
9
+| `chat_llm` | Primary model for all agent reasoning and tool use |
10
| `utility_llm` | Secondary model for internal framework tasks: memory summarization, query generation, history compression, memory recall filtering |
11
| `embedding_llm` | Produces vector embeddings for memory and knowledge indexing |
12
13
-The utility model handles high-volume, lower-stakes operations and can be a cheaper/faster model than the chat model. The Browser Agent uses the effective chat model resolved by `_model_config`, including per-chat overrides and the chat model vision flag. Changing the embedding model invalidates the existing vector index - the entire knowledge base is re-indexed automatically.
13
+The utility model handles high-volume, lower-stakes operations and can be a cheaper/faster model than the chat model. Browser automation is exposed as the direct `browser` tool; the main agent decides when to call it. Changing the embedding model invalidates the existing vector index - the entire knowledge base is re-indexed automatically.
14
15
## Model Providers
16
models.py
+1
-1
@@ -45,7 +45,7 @@ from sentence_transformers import SentenceTransformer
45
from pydantic import ConfigDict
46
47
48
-# disable extra logging, must be done repeatedly, otherwise browser-use will turn it back on for some reason
48
+# keep provider logging quiet in normal operation
49
def turn_off_logging():
50
os.environ["LITELLM_LOG"] = "ERROR" # only errors
51
litellm.suppress_debug_info = True
plugins/_browser/api/extensions.py
new
+27
@@ -0,0 +1,27 @@
1
+from helpers.api import ApiHandler, Request
2
+from plugins._browser.helpers.extension_manager import (
3
+ get_extensions_root,
4
+ install_chrome_web_store_extension,
5
+ list_browser_extensions,
6
+)
7
+
8
+
9
+class Extensions(ApiHandler):
10
+ async def process(self, input: dict, request: Request) -> dict:
11
+ action = input.get("action", "list")
12
+
13
+ if action == "list":
14
+ return {
15
+ "ok": True,
16
+ "root": str(get_extensions_root()),
17
+ "extensions": list_browser_extensions(),
18
+ }
19
+
20
+ if action == "install_web_store":
21
+ try:
22
+ result = install_chrome_web_store_extension(str(input.get("url", "")))
23
+ except ValueError as exc:
24
+ return {"ok": False, "error": str(exc)}
25
+ return result
26
+
27
+ return {"ok": False, "error": f"Unknown action: {action}"}
plugins/_browser/api/status.py
new
+32
@@ -0,0 +1,32 @@
1
+from helpers.api import ApiHandler, Request
2
+from plugins._browser.helpers.config import build_browser_launch_config, get_browser_config
3
+from plugins._browser.helpers.playwright import get_playwright_binary, get_playwright_cache_dir
4
+from plugins._browser.helpers.runtime import known_context_ids
5
+
6
+
7
+class Status(ApiHandler):
8
+ async def process(self, input: dict, request: Request) -> dict:
9
+ browser_config = get_browser_config()
10
+ launch_config = build_browser_launch_config(browser_config)
11
+ runtime_binary = get_playwright_binary(
12
+ full_browser=launch_config["requires_full_browser"]
13
+ )
14
+ shell_binary = get_playwright_binary(full_browser=False)
15
+ chromium_binary = get_playwright_binary(full_browser=True)
16
+ return {
17
+ "plugin": "_browser",
18
+ "playwright": {
19
+ "cache_dir": get_playwright_cache_dir(),
20
+ "binary_found": bool(runtime_binary),
21
+ "binary_path": str(runtime_binary) if runtime_binary else "",
22
+ "headless_shell_binary_path": str(shell_binary) if shell_binary else "",
23
+ "chromium_binary_path": str(chromium_binary) if chromium_binary else "",
24
+ "launch_mode": launch_config["browser_mode"],
25
+ },
26
+ "extensions": {
27
+ **launch_config["extensions"],
28
+ "launch_mode": launch_config["browser_mode"],
29
+ "requires_full_browser": launch_config["requires_full_browser"],
30
+ },
31
+ "contexts": known_context_ids(),
32
+ }
plugins/_browser/api/ws_browser.py
new
+241
@@ -0,0 +1,241 @@
1
+from __future__ import annotations
2
+
3
+import asyncio
4
+from typing import Any, ClassVar
5
+
6
+from agent import AgentContext
7
+from helpers.ws import WsHandler
8
+from helpers.ws_manager import WsResult
9
+from plugins._browser.helpers.runtime import get_runtime
10
+
11
+
12
+class WsBrowser(WsHandler):
13
+ _streams: ClassVar[dict[tuple[str, str], asyncio.Task[None]]] = {}
14
+
15
+ async def on_disconnect(self, sid: str) -> None:
16
+ for key in [key for key in self._streams if key[0] == sid]:
17
+ task = self._streams.pop(key)
18
+ task.cancel()
19
+
20
+ async def process(
21
+ self,
22
+ event: str,
23
+ data: dict[str, Any],
24
+ sid: str,
25
+ ) -> dict[str, Any] | WsResult | None:
26
+ if not event.startswith("browser_"):
27
+ return None
28
+
29
+ if event == "browser_viewer_subscribe":
30
+ return await self._subscribe(data, sid)
31
+ if event == "browser_viewer_unsubscribe":
32
+ return self._unsubscribe(data, sid)
33
+ if event == "browser_viewer_command":
34
+ return await self._command(data, sid)
35
+ if event == "browser_viewer_input":
36
+ return await self._input(data, sid)
37
+
38
+ return WsResult.error(
39
+ code="UNKNOWN_BROWSER_EVENT",
40
+ message=f"Unknown browser event: {event}",
41
+ correlation_id=data.get("correlationId"),
42
+ )
43
+
44
+ async def _subscribe(self, data: dict[str, Any], sid: str) -> dict[str, Any] | WsResult:
45
+ context_id = self._context_id(data)
46
+ if not context_id:
47
+ return self._error("MISSING_CONTEXT", "context_id is required", data)
48
+ if not AgentContext.get(context_id):
49
+ return self._error("CONTEXT_NOT_FOUND", f"Context '{context_id}' was not found", data)
50
+
51
+ runtime = await get_runtime(context_id)
52
+ listing = await runtime.call("list")
53
+ browsers = listing.get("browsers") or []
54
+ if not browsers:
55
+ opened = await runtime.call("open", "about:blank")
56
+ listing = await runtime.call("list")
57
+ browsers = listing.get("browsers") or []
58
+ if opened.get("id"):
59
+ listing["last_interacted_browser_id"] = opened.get("id")
60
+ active_id = data.get("browser_id") or listing.get("last_interacted_browser_id")
61
+ if not active_id and browsers:
62
+ active_id = browsers[0].get("id")
63
+
64
+ stream_key = (sid, context_id)
65
+ existing = self._streams.pop(stream_key, None)
66
+ if existing:
67
+ existing.cancel()
68
+ self._streams[stream_key] = asyncio.create_task(
69
+ self._stream_frames(sid, context_id, active_id)
70
+ )
71
+
72
+ return {
73
+ "context_id": context_id,
74
+ "active_browser_id": active_id,
75
+ "browsers": browsers,
76
+ }
77
+
78
+ def _unsubscribe(self, data: dict[str, Any], sid: str) -> dict[str, Any] | WsResult:
79
+ context_id = self._context_id(data)
80
+ if not context_id:
81
+ return self._error("MISSING_CONTEXT", "context_id is required", data)
82
+ task = self._streams.pop((sid, context_id), None)
83
+ if task:
84
+ task.cancel()
85
+ return {"context_id": context_id, "unsubscribed": True}
86
+
87
+ async def _command(self, data: dict[str, Any], sid: str) -> dict[str, Any] | WsResult:
88
+ context_id = self._context_id(data)
89
+ if not context_id:
90
+ return self._error("MISSING_CONTEXT", "context_id is required", data)
91
+ runtime = await get_runtime(context_id)
92
+ command = str(data.get("command") or "").strip().lower().replace("-", "_")
93
+ browser_id = data.get("browser_id")
94
+
95
+ try:
96
+ if command == "open":
97
+ result = await runtime.call("open", data.get("url") or "about:blank")
98
+ elif command == "navigate":
99
+ result = await runtime.call("navigate", browser_id, data.get("url") or "")
100
+ elif command == "back":
101
+ result = await runtime.call("back", browser_id)
102
+ elif command == "forward":
103
+ result = await runtime.call("forward", browser_id)
104
+ elif command == "reload":
105
+ result = await runtime.call("reload", browser_id)
106
+ elif command == "close":
107
+ result = await runtime.call("close_browser", browser_id)
108
+ elif command == "list":
109
+ result = await runtime.call("list")
110
+ else:
111
+ return self._error("UNKNOWN_COMMAND", f"Unknown browser command: {command}", data)
112
+ except Exception as exc:
113
+ return self._error("COMMAND_FAILED", str(exc), data)
114
+
115
+ listing = await runtime.call("list")
116
+ last_interacted_browser_id = listing.get("last_interacted_browser_id")
117
+ await self.emit_to(
118
+ sid,
119
+ "browser_viewer_state",
120
+ {
121
+ "context_id": context_id,
122
+ "result": result,
123
+ "browsers": listing.get("browsers") or [],
124
+ "last_interacted_browser_id": last_interacted_browser_id,
125
+ },
126
+ correlation_id=data.get("correlationId"),
127
+ )
128
+ return {
129
+ "result": result,
130
+ "browsers": listing.get("browsers") or [],
131
+ "last_interacted_browser_id": last_interacted_browser_id,
132
+ }
133
+
134
+ async def _input(self, data: dict[str, Any], sid: str) -> dict[str, Any] | WsResult:
135
+ context_id = self._context_id(data)
136
+ if not context_id:
137
+ return self._error("MISSING_CONTEXT", "context_id is required", data)
138
+ runtime = await get_runtime(context_id, create=False)
139
+ if not runtime:
140
+ return self._error("NO_BROWSER_RUNTIME", "No browser runtime exists for this context", data)
141
+
142
+ input_type = str(data.get("input_type") or "").strip().lower()
143
+ browser_id = data.get("browser_id")
144
+ try:
145
+ if input_type == "mouse":
146
+ result = await runtime.call(
147
+ "mouse",
148
+ browser_id,
149
+ data.get("event_type") or "click",
150
+ float(data.get("x") or 0),
151
+ float(data.get("y") or 0),
152
+ data.get("button") or "left",
153
+ )
154
+ elif input_type == "keyboard":
155
+ result = await runtime.call(
156
+ "keyboard",
157
+ browser_id,
158
+ key=str(data.get("key") or ""),
159
+ text=str(data.get("text") or ""),
160
+ )
161
+ elif input_type == "viewport":
162
+ result = await runtime.call(
163
+ "set_viewport",
164
+ browser_id,
165
+ int(data.get("width") or 0),
166
+ int(data.get("height") or 0),
167
+ )
168
+ elif input_type == "wheel":
169
+ result = await runtime.call(
170
+ "wheel",
171
+ browser_id,
172
+ float(data.get("x") or 0),
173
+ float(data.get("y") or 0),
174
+ float(data.get("delta_x") or 0),
175
+ float(data.get("delta_y") or 0),
176
+ )
177
+ else:
178
+ return self._error("UNKNOWN_INPUT", f"Unknown browser input: {input_type}", data)
179
+ except Exception as exc:
180
+ return self._error("INPUT_FAILED", str(exc), data)
181
+
182
+ return {"state": result}
183
+
184
+ async def _stream_frames(
185
+ self,
186
+ sid: str,
187
+ context_id: str,
188
+ browser_id: int | str | None,
189
+ ) -> None:
190
+ while True:
191
+ try:
192
+ runtime = await get_runtime(context_id, create=False)
193
+ if runtime:
194
+ listing = await runtime.call("list")
195
+ browsers = listing.get("browsers") or []
196
+ browser_ids = {str(browser.get("id")) for browser in browsers}
197
+ requested_id = str(browser_id or "") if browser_id else ""
198
+ active_id = (
199
+ browser_id
200
+ if requested_id and requested_id in browser_ids
201
+ else listing.get("last_interacted_browser_id")
202
+ )
203
+ if active_id and str(active_id) not in browser_ids:
204
+ active_id = None
205
+ if not active_id and browsers:
206
+ active_id = browsers[0].get("id")
207
+ if active_id:
208
+ frame = await runtime.call("screenshot", active_id)
209
+ frame["context_id"] = context_id
210
+ frame["browsers"] = browsers
211
+ await self.emit_to(sid, "browser_viewer_frame", frame)
212
+ else:
213
+ await self.emit_to(
214
+ sid,
215
+ "browser_viewer_frame",
216
+ {
217
+ "context_id": context_id,
218
+ "browser_id": None,
219
+ "browsers": browsers,
220
+ "image": "",
221
+ "mime": "",
222
+ "state": None,
223
+ },
224
+ )
225
+ await asyncio.sleep(0.75)
226
+ except asyncio.CancelledError:
227
+ raise
228
+ except Exception:
229
+ await asyncio.sleep(1.5)
230
+
231
+ @staticmethod
232
+ def _context_id(data: dict[str, Any]) -> str:
233
+ return str(data.get("context_id") or data.get("context") or "").strip()
234
+
235
+ @staticmethod
236
+ def _error(code: str, message: str, data: dict[str, Any]) -> WsResult:
237
+ return WsResult.error(
238
+ code=code,
239
+ message=message,
240
+ correlation_id=data.get("correlationId"),
241
+ )
plugins/_browser/assets/browser-page-content.js
new
+2891
@@ -0,0 +1,2891 @@
1
+(() => {
2
+ const GLOBAL_KEY = "__spaceBrowserPageContent__";
3
+ const DOM_HELPER_KEY = "__spaceBrowserDomHelper__";
4
+ const VERSION = "6";
5
+ const BLOCK_TAGS = new Set([
6
+ "ADDRESS",
7
+ "ARTICLE",
8
+ "ASIDE",
9
+ "BLOCKQUOTE",
10
+ "BODY",
11
+ "DETAILS",
12
+ "DIV",
13
+ "DL",
14
+ "FIELDSET",
15
+ "FIGCAPTION",
16
+ "FIGURE",
17
+ "FOOTER",
18
+ "FORM",
19
+ "H1",
20
+ "H2",
21
+ "H3",
22
+ "H4",
23
+ "H5",
24
+ "H6",
25
+ "HEADER",
26
+ "HR",
27
+ "HTML",
28
+ "LI",
29
+ "MAIN",
30
+ "NAV",
31
+ "OL",
32
+ "P",
33
+ "PRE",
34
+ "SECTION",
35
+ "TABLE",
36
+ "TBODY",
37
+ "TD",
38
+ "TFOOT",
39
+ "TH",
40
+ "THEAD",
41
+ "TR",
42
+ "UL"
43
+ ]);
44
+ const SKIP_TAGS = new Set([
45
+ "HEAD",
46
+ "LINK",
47
+ "META",
48
+ "NOSCRIPT",
49
+ "SCRIPT",
50
+ "STYLE",
51
+ "TEMPLATE"
52
+ ]);
53
+ const INTERACTIVE_ROLES = new Set([
54
+ "button",
55
+ "checkbox",
56
+ "combobox",
57
+ "link",
58
+ "menuitem",
59
+ "menuitemcheckbox",
60
+ "menuitemradio",
61
+ "option",
62
+ "radio",
63
+ "searchbox",
64
+ "slider",
65
+ "spinbutton",
66
+ "switch",
67
+ "tab",
68
+ "textbox"
69
+ ]);
70
+ const INTERACTIVE_EVENT_NAMES = new Set([
71
+ "auxclick",
72
+ "change",
73
+ "click",
74
+ "contextmenu",
75
+ "dblclick",
76
+ "input",
77
+ "keydown",
78
+ "keypress",
79
+ "keyup",
80
+ "mousedown",
81
+ "mouseup",
82
+ "pointerdown",
83
+ "pointerup",
84
+ "submit",
85
+ "touchend",
86
+ "touchstart"
87
+ ]);
88
+ const INTERACTIVE_EVENT_PROPERTIES = [...INTERACTIVE_EVENT_NAMES]
89
+ .map((eventName) => `on${eventName}`);
90
+
91
+ if (globalThis[GLOBAL_KEY]?.version === VERSION) {
92
+ return;
93
+ }
94
+
95
+ const state = {
96
+ backend: "live",
97
+ captureId: 0,
98
+ capturedAt: 0,
99
+ captureOptions: {
100
+ includeLabelQuotes: false,
101
+ includeLinkUrls: false,
102
+ includeSemanticTags: true,
103
+ includeStateTags: true,
104
+ includeListIndentation: true,
105
+ includeListMarkers: false
106
+ },
107
+ entries: new Map()
108
+ };
109
+
110
+ function isElementNode(value) {
111
+ return Boolean(value && value.nodeType === 1);
112
+ }
113
+
114
+ function isTextNode(value) {
115
+ return Boolean(value && value.nodeType === 3);
116
+ }
117
+
118
+ function normalizeText(value) {
119
+ return String(value ?? "")
120
+ .replace(/\s+/gu, " ")
121
+ .trim();
122
+ }
123
+
124
+ function looksLikeSerializedHtmlText(value) {
125
+ const normalizedValue = normalizeText(value);
126
+ if (!normalizedValue || !normalizedValue.includes("<") || !normalizedValue.includes(">")) {
127
+ return false;
128
+ }
129
+
130
+ if (/<!(?:doctype|--)\b/iu.test(normalizedValue)) {
131
+ return true;
132
+ }
133
+
134
+ if (/<\/?(?:style|script)\b[\s\S]*?>/iu.test(normalizedValue)) {
135
+ return true;
136
+ }
137
+
138
+ const tagMatches = normalizedValue.match(/<\/?[a-z][^>]*>/giu) || [];
139
+ return tagMatches.length >= 3 && normalizedValue.length >= 80;
140
+ }
141
+
142
+ function looksLikeBrowserHelperMarkupText(value) {
143
+ const normalizedValue = normalizeText(value);
144
+ if (!normalizedValue) {
145
+ return false;
146
+ }
147
+
148
+ return /space-browser-(?:frame-document|shadow-root)/iu.test(normalizedValue)
149
+ || /data-space-browser-(?:frame|node|status|frame-url|frame-title|frame-src)/iu.test(normalizedValue);
150
+ }
151
+
152
+ function looksLikeMinifiedScriptText(value) {
153
+ const normalizedValue = normalizeText(value);
154
+ if (!normalizedValue || normalizedValue.length < 400) {
155
+ return false;
156
+ }
157
+
158
+ const jsSignals = [
159
+ /\bfunction\b/u,
160
+ /\breturn\b/u,
161
+ /\bvar\b/u,
162
+ /\bnew\b/u,
163
+ /\bcase\b/u,
164
+ /\bswitch\b/u,
165
+ /\bwhile\b/u,
166
+ /\bfor\b/u,
167
+ /\b(?:localStorage|postMessage|document\.|window\.|parent\.)/u,
168
+ /\bthis\./u,
169
+ /(?:&&|\|\||>>>|!==|===)/u
170
+ ].reduce((count, pattern) => count + (pattern.test(normalizedValue) ? 1 : 0), 0);
171
+
172
+ if (jsSignals < 4) {
173
+ return false;
174
+ }
175
+
176
+ const punctuationCount = (normalizedValue.match(/[{}[\]();=<>\\]/gu) || []).length;
177
+ return punctuationCount / normalizedValue.length >= 0.12;
178
+ }
179
+
180
+ function shouldDropReadableText(value) {
181
+ const normalizedValue = normalizeText(value);
182
+ if (!normalizedValue) {
183
+ return true;
184
+ }
185
+
186
+ return looksLikeBrowserHelperMarkupText(normalizedValue)
187
+ || looksLikeSerializedHtmlText(normalizedValue)
188
+ || looksLikeMinifiedScriptText(normalizedValue);
189
+ }
190
+
191
+ function normalizeAttributeText(value) {
192
+ return normalizeText(value).slice(0, 160);
193
+ }
194
+
195
+ function escapeMarkdownText(value) {
196
+ return String(value ?? "").replace(/([\\`*_{}\[\]()#+\-!|>])/gu, "\\$1");
197
+ }
198
+
199
+ function quoteText(value) {
200
+ return JSON.stringify(String(value ?? ""));
201
+ }
202
+
203
+ function truncateText(value, maxLength = 120) {
204
+ const normalizedValue = normalizeText(value);
205
+ if (normalizedValue.length <= maxLength) {
206
+ return normalizedValue;
207
+ }
208
+
209
+ return `${normalizedValue.slice(0, Math.max(0, maxLength - 1)).trimEnd()}...`;
210
+ }
211
+
212
+ function delayMs(timeoutMs) {
213
+ return new Promise((resolve) => {
214
+ globalThis.setTimeout(resolve, Math.max(0, Number(timeoutMs) || 0));
215
+ });
216
+ }
217
+
218
+ function parseCssColor(value) {
219
+ const normalizedValue = normalizeText(value);
220
+ if (!normalizedValue || normalizedValue === "transparent") {
221
+ return null;
222
+ }
223
+
224
+ const rgbMatch = normalizedValue.match(/^rgba?\(([^)]+)\)$/iu);
225
+ if (rgbMatch) {
226
+ const parts = rgbMatch[1]
227
+ .split(",")
228
+ .map((part) => Number.parseFloat(String(part || "").trim()))
229
+ .filter((part) => Number.isFinite(part));
230
+ if (parts.length >= 3) {
231
+ return {
232
+ r: Math.max(0, Math.min(255, parts[0])),
233
+ g: Math.max(0, Math.min(255, parts[1])),
234
+ b: Math.max(0, Math.min(255, parts[2])),
235
+ a: parts.length >= 4 ? Math.max(0, Math.min(1, parts[3])) : 1
236
+ };
237
+ }
238
+ }
239
+
240
+ const hexMatch = normalizedValue.match(/^#([\da-f]{3,8})$/iu);
241
+ if (!hexMatch) {
242
+ return null;
243
+ }
244
+
245
+ const hex = hexMatch[1];
246
+ if (hex.length === 3 || hex.length === 4) {
247
+ const [r, g, b, a = "f"] = hex.split("");
248
+ return {
249
+ r: Number.parseInt(`${r}${r}`, 16),
250
+ g: Number.parseInt(`${g}${g}`, 16),
251
+ b: Number.parseInt(`${b}${b}`, 16),
252
+ a: Number.parseInt(`${a}${a}`, 16) / 255
253
+ };
254
+ }
255
+
256
+ if (hex.length === 6 || hex.length === 8) {
257
+ return {
258
+ r: Number.parseInt(hex.slice(0, 2), 16),
259
+ g: Number.parseInt(hex.slice(2, 4), 16),
260
+ b: Number.parseInt(hex.slice(4, 6), 16),
261
+ a: hex.length === 8 ? Number.parseInt(hex.slice(6, 8), 16) / 255 : 1
262
+ };
263
+ }
264
+
265
+ return null;
266
+ }
267
+
268
+ function rgbToHsl(color) {
269
+ if (!color) {
270
+ return null;
271
+ }
272
+
273
+ const r = color.r / 255;
274
+ const g = color.g / 255;
275
+ const b = color.b / 255;
276
+ const max = Math.max(r, g, b);
277
+ const min = Math.min(r, g, b);
278
+ const delta = max - min;
279
+ const lightness = (max + min) / 2;
280
+ let hue = 0;
281
+ let saturation = 0;
282
+
283
+ if (delta > 0) {
284
+ saturation = delta / (1 - Math.abs(2 * lightness - 1));
285
+ if (max === r) {
286
+ hue = 60 * (((g - b) / delta) % 6);
287
+ } else if (max === g) {
288
+ hue = 60 * (((b - r) / delta) + 2);
289
+ } else {
290
+ hue = 60 * (((r - g) / delta) + 4);
291
+ }
292
+ }
293
+
294
+ if (hue < 0) {
295
+ hue += 360;
296
+ }
297
+
298
+ return {
299
+ hue,
300
+ lightness,
301
+ saturation
302
+ };
303
+ }
304
+
305
+ function isTrustedHtmlRequirementError(error) {
306
+ return /TrustedHTML/iu.test(String(error?.message || error || ""));
307
+ }
308
+
309
+ function joinBlocks(blocks) {
310
+ return blocks
311
+ .map((block) => String(block || "").trim())
312
+ .filter(Boolean)
313
+ .join("\n\n")
314
+ .trim();
315
+ }
316
+
317
+ function cleanReadableMarkdown(value) {
318
+ const lines = String(value || "")
319
+ .replace(/<style\\?>[\s\S]*?<\/style\\?>/giu, "")
320
+ .replace(/<script\\?>[\s\S]*?<\/script\\?>/giu, "")
321
+ .replace(/<space\\-browser\\-(?:frame\\-document|shadow\\-root)\b[\s\S]*?<\/space\\-browser\\-(?:frame\\-document|shadow\\-root)>/giu, "")
322
+ .split("\n");
323
+
324
+ const filteredLines = [];
325
+ let insideCodeFence = false;
326
+
327
+ lines.forEach((line) => {
328
+ const trimmedLine = String(line || "").trim();
329
+ if (trimmedLine.startsWith("```")) {
330
+ insideCodeFence = !insideCodeFence;
331
+ filteredLines.push(line);
332
+ return;
333
+ }
334
+
335
+ if (!trimmedLine || insideCodeFence) {
336
+ filteredLines.push(line);
337
+ return;
338
+ }
339
+
340
+ if (shouldDropReadableText(trimmedLine)) {
341
+ return;
342
+ }
343
+
344
+ filteredLines.push(line);
345
+ });
346
+
347
+ return filteredLines
348
+ .join("\n")
349
+ .replace(/\n{3,}/gu, "\n\n")
350
+ .trim();
351
+ }
352
+
353
+ function joinInlineParts(parts) {
354
+ return String(parts
355
+ .map((part) => String(part || "").trim())
356
+ .filter(Boolean)
357
+ .join(" "))
358
+ .replace(/\s+([,.;!?])/gu, "$1")
359
+ .replace(/([([{\u201c])\s+/gu, "$1")
360
+ .replace(/\s+([\])}\u201d])/gu, "$1")
361
+ .replace(/\s*\n\s*/gu, "\n")
362
+ .replace(/[ \t]+\n/gu, "\n")
363
+ .replace(/\n{3,}/gu, "\n\n")
364
+ .trim();
365
+ }
366
+
367
+ function indentBlock(text, level = 1) {
368
+ const prefix = " ".repeat(Math.max(0, level));
369
+ return String(text || "")
370
+ .split("\n")
371
+ .map((line) => `${prefix}${line}`)
372
+ .join("\n");
373
+ }
374
+
375
+ function createNamedError(name, message, details = {}) {
376
+ const error = new Error(message);
377
+ error.name = name;
378
+ Object.assign(error, details);
379
+ return error;
380
+ }
381
+
382
+ function coerceSelectorList(payload) {
383
+ if (typeof payload === "string") {
384
+ return [payload];
385
+ }
386
+
387
+ if (Array.isArray(payload?.selectors)) {
388
+ return payload.selectors;
389
+ }
390
+
391
+ if (typeof payload?.selectors === "string") {
392
+ return [payload.selectors];
393
+ }
394
+
395
+ if (Array.isArray(payload?.selector)) {
396
+ return payload.selector;
397
+ }
398
+
399
+ if (typeof payload?.selector === "string") {
400
+ return [payload.selector];
401
+ }
402
+
403
+ if (Array.isArray(payload)) {
404
+ return payload;
405
+ }
406
+
407
+ return [];
408
+ }
409
+
410
+ function normalizeSelectorList(payload) {
411
+ return coerceSelectorList(payload)
412
+ .map((selector) => String(selector || "").trim())
413
+ .filter(Boolean);
414
+ }
415
+
416
+ function normalizeIncludeLinkUrls(payload) {
417
+ return payload?.includeLinkUrls === true;
418
+ }
419
+
420
+ function normalizeIncludeLabelQuotes(payload) {
421
+ return payload?.includeLabelQuotes === true;
422
+ }
423
+
424
+ function normalizeIncludeListIndentation(payload) {
425
+ return payload?.includeListIndentation !== false;
426
+ }
427
+
428
+ function normalizeIncludeListMarkers(payload) {
429
+ return payload?.includeListMarkers === true;
430
+ }
431
+
432
+ function normalizeIncludeStateTags(payload) {
433
+ return payload?.includeStateTags !== false;
434
+ }
435
+
436
+ function normalizeIncludeSemanticTags(payload) {
437
+ return payload?.includeSemanticTags !== false;
438
+ }
439
+
440
+ function formatSummaryValue(value, options = {}) {
441
+ const normalizedValue = normalizeText(value);
442
+ if (!normalizedValue) {
443
+ return "";
444
+ }
445
+
446
+ if (options.includeLabelQuotes === true) {
447
+ return quoteText(normalizedValue);
448
+ }
449
+
450
+ return escapeMarkdownText(normalizedValue);
451
+ }
452
+
453
+ function normalizeFrameChain(value) {
454
+ const rawFrameChain = Array.isArray(value)
455
+ ? value
456
+ : typeof value === "string"
457
+ ? value.split(">")
458
+ : [];
459
+
460
+ return rawFrameChain
461
+ .map((entry) => String(entry || "").trim())
462
+ .filter(Boolean);
463
+ }
464
+
465
+ function getDomHelper() {
466
+ const helper = globalThis[DOM_HELPER_KEY];
467
+ if (
468
+ helper
469
+ && typeof helper.captureDocument === "function"
470
+ && typeof helper.detailNode === "function"
471
+ && typeof helper.clickNode === "function"
472
+ && typeof helper.typeNode === "function"
473
+ && typeof helper.submitNode === "function"
474
+ && typeof helper.typeSubmitNode === "function"
475
+ && typeof helper.scrollNode === "function"
476
+ ) {
477
+ return helper;
478
+ }
479
+
480
+ return null;
481
+ }
482
+
483
+ function requireDomHelper(actionLabel) {
484
+ const helper = getDomHelper();
485
+ if (helper) {
486
+ return helper;
487
+ }
488
+
489
+ throw createNamedError(
490
+ "BrowserPageContentHelperUnavailableError",
491
+ `Browser page content cannot ${actionLabel} without the desktop DOM helper.`,
492
+ {
493
+ code: "browser_page_content_dom_helper_unavailable",
494
+ details: {
495
+ action: String(actionLabel || "resolve")
496
+ }
497
+ }
498
+ );
499
+ }
500
+
501
+ function normalizeReferenceId(value) {
502
+ if (typeof value === "number" && Number.isFinite(value)) {
503
+ return String(Math.trunc(value));
504
+ }
505
+
506
+ if (typeof value === "string") {
507
+ return value.trim();
508
+ }
509
+
510
+ if (value && typeof value === "object") {
511
+ return normalizeReferenceId(value.referenceId ?? value.ref ?? value.id);
512
+ }
513
+
514
+ return "";
515
+ }
516
+
517
+ function getTagName(element) {
518
+ return String(element?.tagName || "").toUpperCase();
519
+ }
520
+
521
+ function getAttributeNamesSafe(element) {
522
+ try {
523
+ if (typeof element?.getAttributeNames === "function") {
524
+ return element.getAttributeNames();
525
+ }
526
+
527
+ return [...(element?.attributes || [])]
528
+ .map((attribute) => String(attribute?.name || "").trim())
529
+ .filter(Boolean);
530
+ } catch {
531
+ return [];
532
+ }
533
+ }
534
+
535
+ function normalizeInteractiveEventName(value) {
536
+ return String(value || "")
537
+ .trim()
538
+ .toLowerCase()
539
+ .split(/[.:]/u, 1)[0];
540
+ }
541
+
542
+ function isInteractiveEventName(value) {
543
+ return INTERACTIVE_EVENT_NAMES.has(normalizeInteractiveEventName(value));
544
+ }
545
+
546
+ function isInteractiveEventAttributeName(attributeName) {
547
+ const normalizedName = String(attributeName || "").trim().toLowerCase();
548
+ if (!normalizedName) {
549
+ return false;
550
+ }
551
+
552
+ if (normalizedName.startsWith("@")) {
553
+ return isInteractiveEventName(normalizedName.slice(1));
554
+ }
555
+
556
+ if (normalizedName.startsWith("x-on:") || normalizedName.startsWith("v-on:")) {
557
+ return isInteractiveEventName(normalizedName.slice(5));
558
+ }
559
+
560
+ if (normalizedName.startsWith("ng-")) {
561
+ return isInteractiveEventName(normalizedName.slice(3));
562
+ }
563
+
564
+ if (normalizedName.startsWith("on") && normalizedName.length > 2) {
565
+ return isInteractiveEventName(normalizedName.slice(2));
566
+ }
567
+
568
+ return false;
569
+ }
570
+
571
+ function hasHelperManagedNodeReference(element) {
572
+ return Boolean(normalizeAttributeText(element?.getAttribute?.("data-space-browser-node-id")));
573
+ }
574
+
575
+ function hasInteractiveEventHandlerAttribute(element) {
576
+ return getAttributeNamesSafe(element).some((attributeName) => {
577
+ return isInteractiveEventAttributeName(attributeName);
578
+ });
579
+ }
580
+
581
+ function hasInteractiveEventHandlerProperty(element) {
582
+ return INTERACTIVE_EVENT_PROPERTIES.some((propertyName) => {
583
+ return typeof element?.[propertyName] === "function";
584
+ });
585
+ }
586
+
587
+ function hasInteractiveEventHandler(element) {
588
+ return hasInteractiveEventHandlerAttribute(element) || hasInteractiveEventHandlerProperty(element);
589
+ }
590
+
591
+ function isStyleDeclarationHidden(styleValue) {
592
+ const normalizedStyleValue = String(styleValue || "")
593
+ .toLowerCase()
594
+ .replace(/\s+/gu, "");
595
+
596
+ if (!normalizedStyleValue) {
597
+ return false;
598
+ }
599
+
600
+ return /(?:^|;)display:none(?:;|$)/u.test(normalizedStyleValue)
601
+ || /(?:^|;)visibility:hidden(?:;|$)/u.test(normalizedStyleValue)
602
+ || /(?:^|;)visibility:collapse(?:;|$)/u.test(normalizedStyleValue)
603
+ || /(?:^|;)content-visibility:hidden(?:;|$)/u.test(normalizedStyleValue)
604
+ || /(?:^|;)opacity:0(?:\.0+)?(?:;|$)/u.test(normalizedStyleValue);
605
+ }
606
+
607
+ function isComputedStyleHidden(computedStyle) {
608
+ if (!computedStyle) {
609
+ return false;
610
+ }
611
+
612
+ const display = normalizeText(computedStyle.display).toLowerCase();
613
+ const visibility = normalizeText(computedStyle.visibility).toLowerCase();
614
+ const contentVisibility = normalizeText(computedStyle.contentVisibility).toLowerCase();
615
+ const opacity = Number(computedStyle.opacity || 1);
616
+
617
+ return display === "none"
618
+ || visibility === "hidden"
619
+ || visibility === "collapse"
620
+ || contentVisibility === "hidden"
621
+ || opacity <= 0;
622
+ }
623
+
624
+ function isEffectivelyHiddenByAncestor(element) {
625
+ let current = element;
626
+
627
+ while (isElementNode(current)) {
628
+ if (current.hidden || current.getAttribute?.("aria-hidden") === "true") {
629
+ return true;
630
+ }
631
+
632
+ if (isStyleDeclarationHidden(current.getAttribute?.("style"))) {
633
+ return true;
634
+ }
635
+
636
+ if (isComputedStyleHidden(getComputedStyleSafe(current))) {
637
+ return true;
638
+ }
639
+
640
+ current = current.parentElement;
641
+ }
642
+
643
+ return false;
644
+ }
645
+
646
+ function isHiddenElement(element) {
647
+ if (!isElementNode(element)) {
648
+ return true;
649
+ }
650
+
651
+ const tagName = getTagName(element);
652
+ if (SKIP_TAGS.has(tagName)) {
653
+ return true;
654
+ }
655
+
656
+ if (element.hidden || element.getAttribute?.("aria-hidden") === "true") {
657
+ return true;
658
+ }
659
+
660
+ if (tagName === "INPUT" && String(element.getAttribute?.("type") || "").toLowerCase() === "hidden") {
661
+ return true;
662
+ }
663
+
664
+ if (isStyleDeclarationHidden(element.getAttribute?.("style"))) {
665
+ return true;
666
+ }
667
+
668
+ const computedStyle = getComputedStyleSafe(element);
669
+ if (isComputedStyleHidden(computedStyle)) {
670
+ return true;
671
+ }
672
+
673
+ return isEffectivelyHiddenByAncestor(element.parentElement);
674
+ }
675
+
676
+ function isBlockElement(element) {
677
+ return BLOCK_TAGS.has(getTagName(element));
678
+ }
679
+
680
+ function isInteractiveElement(element) {
681
+ if (!isElementNode(element) || isHiddenElement(element)) {
682
+ return false;
683
+ }
684
+
685
+ if (hasHelperManagedNodeReference(element)) {
686
+ return true;
687
+ }
688
+
689
+ const tagName = getTagName(element);
690
+ if (tagName === "A" && element.hasAttribute?.("href")) {
691
+ return true;
692
+ }
693
+
694
+ if (tagName === "BUTTON" || tagName === "INPUT" || tagName === "SELECT" || tagName === "TEXTAREA" || tagName === "SUMMARY") {
695
+ return true;
696
+ }
697
+
698
+ if (String(element.getAttribute?.("contenteditable") || "").toLowerCase() === "true") {
699
+ return true;
700
+ }
701
+
702
+ const role = String(element.getAttribute?.("role") || "").trim().toLowerCase();
703
+ return INTERACTIVE_ROLES.has(role) || hasInteractiveEventHandler(element);
704
+ }
705
+
706
+ function getComputedStyleSafe(element) {
707
+ try {
708
+ return globalThis.getComputedStyle?.(element) || null;
709
+ } catch {
710
+ return null;
711
+ }
712
+ }
713
+
714
+ function getElementRectSafe(element) {
715
+ try {
716
+ const rect = element?.getBoundingClientRect?.();
717
+ if (!rect) {
718
+ return null;
719
+ }
720
+
721
+ return {
722
+ height: Number(rect.height) || 0,
723
+ width: Number(rect.width) || 0,
724
+ x: Number(rect.x) || 0,
725
+ y: Number(rect.y) || 0
726
+ };
727
+ } catch {
728
+ return null;
729
+ }
730
+ }
731
+
732
+ function readSerializedTagList(element, attributeName) {
733
+ const rawValue = normalizeText(element?.getAttribute?.(attributeName));
734
+ if (!rawValue) {
735
+ return [];
736
+ }
737
+
738
+ return rawValue
739
+ .split(/\s+/u)
740
+ .map((part) => normalizeText(part))
741
+ .filter(Boolean);
742
+ }
743
+
744
+ function detectSemanticTone(element, computedStyle, metadata = {}) {
745
+ const opacity = Number(computedStyle?.opacity || 1);
746
+ const backgroundColor = parseCssColor(computedStyle?.backgroundColor || "");
747
+ const borderColor = parseCssColor(computedStyle?.borderTopColor || "");
748
+ const foregroundColor = parseCssColor(computedStyle?.color || "");
749
+ const isButtonLike = ["BUTTON", "INPUT", "SUMMARY"].includes(getTagName(element))
750
+ || ["button", "tab", "menuitem"].includes(String(element?.getAttribute?.("role") || "").trim().toLowerCase());
751
+
752
+ if (metadata.disabled || metadata.blocked || opacity <= 0.58) {
753
+ return "muted";
754
+ }
755
+
756
+ const preferredColor = [backgroundColor, borderColor, foregroundColor]
757
+ .filter((color) => color && color.a > 0.15)
758
+ .map((color) => ({
759
+ color,
760
+ hsl: rgbToHsl(color)
761
+ }))
762
+ .find((entry) => entry.hsl && entry.hsl.saturation >= 0.2);
763
+
764
+ if (!preferredColor) {
765
+ return "";
766
+ }
767
+
768
+ const {
769
+ hue,
770
+ lightness,
771
+ saturation
772
+ } = preferredColor.hsl;
773
+ if (saturation < 0.2) {
774
+ return "";
775
+ }
776
+
777
+ if ((hue >= 345 || hue < 20) && lightness >= 0.18 && lightness <= 0.82) {
778
+ return "error";
779
+ }
780
+
781
+ if (hue >= 20 && hue < 65 && lightness >= 0.2 && lightness <= 0.9) {
782
+ return "warning";
783
+ }
784
+
785
+ if (hue >= 65 && hue < 170 && lightness >= 0.16 && lightness <= 0.84) {
786
+ return "success";
787
+ }
788
+
789
+ if (hue >= 170 && hue < 280 && lightness >= 0.14 && lightness <= 0.82) {
790
+ if (isButtonLike && backgroundColor?.a > 0.2) {
791
+ return "primary";
792
+ }
793
+ return "";
794
+ }
795
+
796
+ return "";
797
+ }
798
+
799
+ function collectElementStateMetadata(element, options = {}) {
800
+ if (!isElementNode(element)) {
801
+ return {
802
+ descriptorTags: [],
803
+ semanticTags: [],
804
+ stateTags: []
805
+ };
806
+ }
807
+
808
+ const computedStyle = getComputedStyleSafe(element);
809
+ const rect = getElementRectSafe(element);
810
+ const tagName = getTagName(element);
811
+ const ariaDisabled = String(element.getAttribute?.("aria-disabled") || "").trim().toLowerCase() === "true";
812
+ const ariaBusy = String(element.getAttribute?.("aria-busy") || "").trim().toLowerCase() === "true";
813
+ const ariaChecked = String(element.getAttribute?.("aria-checked") || "").trim().toLowerCase() === "true";
814
+ const ariaCurrent = normalizeText(element.getAttribute?.("aria-current"));
815
+ const ariaInvalid = String(element.getAttribute?.("aria-invalid") || "").trim().toLowerCase() === "true";
816
+ const ariaPressed = String(element.getAttribute?.("aria-pressed") || "").trim().toLowerCase() === "true";
817
+ const ariaReadonly = String(element.getAttribute?.("aria-readonly") || "").trim().toLowerCase() === "true";
818
+ const ariaRequired = String(element.getAttribute?.("aria-required") || "").trim().toLowerCase() === "true";
819
+ const ariaSelected = String(element.getAttribute?.("aria-selected") || "").trim().toLowerCase() === "true";
820
+ const helperStateTags = readSerializedTagList(element, "data-space-browser-state-tags");
821
+ const helperSemanticTags = readSerializedTagList(element, "data-space-browser-semantic-tags");
822
+ const closestInert = typeof element.closest === "function" ? element.closest("[inert]") : null;
823
+ const pointerEventsNone = normalizeText(computedStyle?.pointerEvents || "").toLowerCase() === "none";
824
+ const disabled = Boolean(element.disabled || ariaDisabled || closestInert || helperStateTags.includes("disabled"));
825
+ const blocked = !disabled && (pointerEventsNone || helperStateTags.includes("blocked"));
826
+ const checked = Boolean(element.checked || ariaChecked || helperStateTags.includes("checked"));
827
+ const selected = tagName === "OPTION"
828
+ ? Boolean(element.selected)
829
+ : Boolean(ariaSelected || helperStateTags.includes("selected"));
830
+ const invalid = Boolean(ariaInvalid || helperStateTags.includes("invalid") || element.matches?.(":invalid"));
831
+ const readonly = Boolean(element.readOnly || ariaReadonly);
832
+ const required = Boolean(element.required || ariaRequired);
833
+ const expanded = String(element.getAttribute?.("aria-expanded") || "").trim().toLowerCase() === "true" || helperStateTags.includes("expanded");
834
+ const pressed = ariaPressed || helperStateTags.includes("pressed");
835
+ const busy = ariaBusy || helperStateTags.includes("busy");
836
+ const current = Boolean((ariaCurrent && ariaCurrent !== "false") || helperStateTags.includes("current"));
837
+ const zeroRect = Boolean(
838
+ rect
839
+ && element.ownerDocument === globalThis.document
840
+ && rect.width <= 1
841
+ && rect.height <= 1
842
+ );
843
+ const opacity = Number(computedStyle?.opacity || 1);
844
+ const semanticTone = helperSemanticTags[0] || detectSemanticTone(element, computedStyle, {
845
+ blocked,
846
+ disabled
847
+ });
848
+ const stateTags = helperStateTags.length
849
+ ? helperStateTags.slice()
850
+ : [
851
+ disabled ? "disabled" : "",
852
+ !disabled && (blocked || zeroRect) ? "blocked" : "",
853
+ checked ? "checked" : "",
854
+ selected && tagName !== "SELECT" ? "selected" : "",
855
+ invalid ? "invalid" : "",
856
+ expanded ? "expanded" : "",
857
+ pressed ? "pressed" : ""
858
+ ].filter(Boolean);
859
+
860
+ const semanticTags = helperSemanticTags.length
861
+ ? helperSemanticTags.slice(0, 1)
862
+ : (semanticTone ? [semanticTone] : []);
863
+ const descriptorTags = [
864
+ ...(options.includeStateTags !== false ? stateTags : []),
865
+ ...(options.includeSemanticTags !== false ? semanticTags : [])
866
+ ];
867
+
868
+ return {
869
+ blocked,
870
+ busy,
871
+ checked,
872
+ current,
873
+ cursor: normalizeText(computedStyle?.cursor || "").toLowerCase(),
874
+ descriptorTags,
875
+ disabled,
876
+ expanded,
877
+ invalid,
878
+ opacity,
879
+ pointerEventsNone,
880
+ pressed,
881
+ readonly,
882
+ required,
883
+ selected,
884
+ semanticTags,
885
+ semanticTone,
886
+ stateTags,
887
+ visible: !isHiddenElement(element),
888
+ zeroRect
889
+ };
890
+ }
891
+
892
+ function getReferenceValueMetadata(element) {
893
+ const tagName = getTagName(element);
894
+ const helperLiveValue = normalizeText(element?.getAttribute?.("data-space-browser-live-value"));
895
+ const helperSelectedValue = normalizeText(element?.getAttribute?.("data-space-browser-selected-text"));
896
+ if (tagName === "INPUT") {
897
+ const inputType = String(element.getAttribute?.("type") || element.type || "text").toLowerCase();
898
+ if (inputType === "password") {
899
+ return "";
900
+ }
901
+ return truncateText(helperLiveValue || element.value || element.getAttribute?.("value") || "", 96);
902
+ }
903
+
904
+ if (tagName === "TEXTAREA") {
905
+ return truncateText(helperLiveValue || element.value || "", 96);
906
+ }
907
+
908
+ if (tagName === "SELECT") {
909
+ if (helperSelectedValue) {
910
+ return helperSelectedValue;
911
+ }
912
+ const selectedOptions = [...(element.selectedOptions || [])]
913
+ .map((option) => truncateText(option.textContent || option.label || option.value || "", 48))
914
+ .filter(Boolean);
915
+ return selectedOptions.join(" | ");
916
+ }
917
+
918
+ if (String(element.getAttribute?.("contenteditable") || "").toLowerCase() === "true") {
919
+ return truncateText(element.textContent || "", 96);
920
+ }
921
+
922
+ return "";
923
+ }
924
+
925
+ function collectMetaLines(doc = globalThis.document) {
926
+ const lines = [];
927
+ const title = normalizeAttributeText(doc?.title || "");
928
+ const description = normalizeAttributeText(
929
+ doc?.querySelector?.('meta[name="description"]')?.getAttribute?.("content") || ""
930
+ );
931
+ const url = String(globalThis.location?.href || "");
932
+
933
+ if (!title && !description && !url) {
934
+ return "";
935
+ }
936
+
937
+ lines.push("---");
938
+ if (title) {
939
+ lines.push(`title: ${quoteText(title)}`);
940
+ }
941
+ if (description) {
942
+ lines.push(`description: ${quoteText(description)}`);
943
+ }
944
+ if (url) {
945
+ lines.push(`url: ${quoteText(url)}`);
946
+ }
947
+ lines.push("---");
948
+ return lines.join("\n");
949
+ }
950
+
951
+ function summarizeUrl(value) {
952
+ const normalizedValue = String(value || "").trim();
953
+ if (!normalizedValue) {
954
+ return "";
955
+ }
956
+
957
+ try {
958
+ const url = new URL(normalizedValue, globalThis.location?.href || "http://localhost/");
959
+ if (url.origin === globalThis.location?.origin) {
960
+ const relative = `${url.pathname || "/"}${url.search || ""}${url.hash || ""}`;
961
+ return truncateText(relative || "/", 96);
962
+ }
963
+
964
+ return truncateText(`${url.hostname}${url.pathname || "/"}`, 96);
965
+ } catch {
966
+ return truncateText(normalizedValue, 96);
967
+ }
968
+ }
969
+
970
+ function getElementText(element) {
971
+ return normalizeText(element?.textContent || "");
972
+ }
973
+
974
+ function collectLabelCandidates(element, options = {}) {
975
+ const includeAlt = options.includeAlt !== false;
976
+ const includeDescendantImageAlt = options.includeDescendantImageAlt !== false;
977
+ const includePlaceholder = options.includePlaceholder === true;
978
+ const includeText = options.includeText !== false;
979
+ const collectedLabels = [];
980
+
981
+ try {
982
+ if (Array.isArray(element?.labels) || typeof element?.labels?.forEach === "function") {
983
+ element.labels.forEach((labelElement) => {
984
+ const text = getElementText(labelElement);
985
+ if (text) {
986
+ collectedLabels.push(text);
987
+ }
988
+ });
989
+ }
990
+ } catch {
991
+ // Ignore labels lookup failures from non-form elements.
992
+ }
993
+
994
+ [
995
+ element?.getAttribute?.("aria-label"),
996
+ element?.getAttribute?.("title")
997
+ ].forEach((candidate) => {
998
+ const text = normalizeAttributeText(candidate);
999
+ if (text) {
1000
+ collectedLabels.push(text);
1001
+ }
1002
+ });
1003
+
1004
+ if (includeAlt) {
1005
+ const altText = normalizeAttributeText(element?.getAttribute?.("alt"));
1006
+ if (altText) {
1007
+ collectedLabels.push(altText);
1008
+ }
1009
+ }
1010
+
1011
+ if (includePlaceholder) {
1012
+ const placeholderText = normalizeAttributeText(element?.getAttribute?.("placeholder"));
1013
+ if (placeholderText) {
1014
+ collectedLabels.push(placeholderText);
1015
+ }
1016
+ }
1017
+
1018
+ if (includeDescendantImageAlt) {
1019
+ try {
1020
+ [...(element?.querySelectorAll?.("img[alt], img[title]") || [])]
1021
+ .slice(0, 3)
1022
+ .forEach((mediaElement) => {
1023
+ const text = normalizeAttributeText(
1024
+ mediaElement.getAttribute?.("alt")
1025
+ || mediaElement.getAttribute?.("title")
1026
+ );
1027
+ if (text) {
1028
+ collectedLabels.push(text);
1029
+ }
1030
+ });
1031
+ } catch {
1032
+ // Ignore descendant-media lookup failures.
1033
+ }
1034
+ }
1035
+
1036
+ if (includeText) {
1037
+ const textContent = getElementText(element);
1038
+ if (textContent) {
1039
+ collectedLabels.push(textContent);
1040
+ }
1041
+ }
1042
+
1043
+ return [...new Set(collectedLabels.filter(Boolean))];
1044
+ }
1045
+
1046
+ function getLabelText(element, options = {}) {
1047
+ return collectLabelCandidates(element, options)[0] || "";
1048
+ }
1049
+
1050
+ function serializeElementSnapshot(element) {
1051
+ if (!isElementNode(element)) {
1052
+ return "";
1053
+ }
1054
+
1055
+ try {
1056
+ if (typeof element.outerHTML === "string" && element.outerHTML) {
1057
+ return element.outerHTML;
1058
+ }
1059
+ } catch {
1060
+ // Fall through to XMLSerializer.
1061
+ }
1062
+
1063
+ try {
1064
+ if (typeof globalThis.XMLSerializer === "function") {
1065
+ return new globalThis.XMLSerializer().serializeToString(element);
1066
+ }
1067
+ } catch {
1068
+ // Ignore serialization errors.
1069
+ }
1070
+
1071
+ return "";
1072
+ }
1073
+
1074
+ function getReferenceKind(element) {
1075
+ const tagName = getTagName(element);
1076
+ const role = String(element.getAttribute?.("role") || "").trim().toLowerCase();
1077
+ const inputType = String(element.getAttribute?.("type") || element.type || "text").toLowerCase();
1078
+
1079
+ if (tagName === "A" || role === "link") {
1080
+ return "link";
1081
+ }
1082
+
1083
+ if (tagName === "IMG") {
1084
+ return "image";
1085
+ }
1086
+
1087
+ if (tagName === "BUTTON" || ["button", "menuitem", "tab"].includes(role)) {
1088
+ return "button";
1089
+ }
1090
+
1091
+ if (tagName === "TEXTAREA") {
1092
+ return "textarea";
1093
+ }
1094
+
1095
+ if (tagName === "SELECT" || role === "combobox") {
1096
+ return "select";
1097
+ }
1098
+
1099
+ if (tagName === "SUMMARY") {
1100
+ return "summary";
1101
+ }
1102
+
1103
+ if (tagName === "INPUT") {
1104
+ if (["button", "submit", "reset"].includes(inputType)) {
1105
+ return "button";
1106
+ }
1107
+
1108
+ if (inputType === "checkbox") {
1109
+ return "checkbox";
1110
+ }
1111
+
1112
+ if (inputType === "radio") {
1113
+ return "radio";
1114
+ }
1115
+
1116
+ return `input ${inputType || "text"}`;
1117
+ }
1118
+
1119
+ if (String(element.getAttribute?.("contenteditable") || "").toLowerCase() === "true") {
1120
+ return "editable";
1121
+ }
1122
+
1123
+ if (role === "searchbox") {
1124
+ return "input search";
1125
+ }
1126
+
1127
+ if (role === "textbox") {
1128
+ return "input text";
1129
+ }
1130
+
1131
+ if (hasHelperManagedNodeReference(element) || hasInteractiveEventHandler(element)) {
1132
+ return "button";
1133
+ }
1134
+
1135
+ return role || tagName.toLowerCase();
1136
+ }
1137
+
1138
+ function collectReferenceSummaryData(element, options = {}) {
1139
+ const tagName = getTagName(element);
1140
+ const role = String(element.getAttribute?.("role") || "").trim().toLowerCase();
1141
+ const id = normalizeAttributeText(element.getAttribute?.("id"));
1142
+ const name = normalizeAttributeText(element.getAttribute?.("name"));
1143
+ const kind = getReferenceKind(element);
1144
+ const stateMetadata = collectElementStateMetadata(element, options);
1145
+ const formatValue = (value) => formatSummaryValue(value, options);
1146
+ const includeLinkUrls = options.includeLinkUrls === true;
1147
+ const parts = [];
1148
+ const appendFallbackIdOrName = () => {
1149
+ if (id) {
1150
+ parts.push(`#${id}`);
1151
+ return;
1152
+ }
1153
+
1154
+ if (name) {
1155
+ parts.push(`name=${formatValue(name)}`);
1156
+ }
1157
+ };
1158
+
1159
+ if (tagName === "A" || role === "link") {
1160
+ const hrefSummary = summarizeUrl(element.getAttribute?.("href") || element.href || "");
1161
+ const label = truncateText(getLabelText(element, {
1162
+ includeAlt: false,
1163
+ includeDescendantImageAlt: true,
1164
+ includePlaceholder: false,
1165
+ includeText: true
1166
+ }), 120);
1167
+ const displayLabel = label || hrefSummary;
1168
+
1169
+ if (displayLabel) {
1170
+ parts.push(formatValue(displayLabel));
1171
+ } else {
1172
+ appendFallbackIdOrName();
1173
+ }
1174
+
1175
+ if (includeLinkUrls) {
1176
+ if (hrefSummary && hrefSummary !== displayLabel) {
1177
+ parts.push(`-> ${hrefSummary}`);
1178
+ }
1179
+ }
1180
+ } else if (tagName === "BUTTON" || ["button", "menuitem", "tab"].includes(role)) {
1181
+ const label = truncateText(getLabelText(element, {
1182
+ includeAlt: false,
1183
+ includeDescendantImageAlt: true,
1184
+ includePlaceholder: false,
1185
+ includeText: true
1186
+ }), 120);
1187
+ if (label) {
1188
+ parts.push(formatValue(label));
1189
+ } else {
1190
+ appendFallbackIdOrName();
1191
+ }
1192
+ } else if (tagName === "TEXTAREA" || role === "textbox" || role === "searchbox") {
1193
+ const label = truncateText(getLabelText(element, {
1194
+ includeAlt: false,
1195
+ includeDescendantImageAlt: false,
1196
+ includePlaceholder: false,
1197
+ includeText: true
1198
+ }), 120);
1199
+ if (label) {
1200
+ parts.push(formatValue(label));
1201
+ }
1202
+ const placeholder = normalizeAttributeText(element.getAttribute?.("placeholder"));
1203
+ if (placeholder) {
1204
+ parts.push(`placeholder=${formatValue(placeholder)}`);
1205
+ } else if (!label) {
1206
+ appendFallbackIdOrName();
1207
+ }
1208
+ } else if (tagName === "SELECT" || role === "combobox") {
1209
+ const label = truncateText(getLabelText(element, {
1210
+ includeAlt: false,
1211
+ includeDescendantImageAlt: false,
1212
+ includePlaceholder: false,
1213
+ includeText: true
1214
+ }), 120);
1215
+ if (label) {
1216
+ parts.push(formatValue(label));
1217
+ } else {
1218
+ appendFallbackIdOrName();
1219
+ }
1220
+
1221
+ const selectedValue = getReferenceValueMetadata(element);
1222
+ const selectedOptions = selectedValue
1223
+ ? [selectedValue]
1224
+ : [...(element.selectedOptions || [])]
1225
+ .map((option) => truncateText(option.textContent || "", 48))
1226
+ .filter(Boolean);
1227
+ if (selectedOptions.length) {
1228
+ parts.push(`selected=${formatValue(selectedOptions.join(" | "))}`);
1229
+ }
1230
+ } else if (tagName === "SUMMARY") {
1231
+ const label = truncateText(getLabelText(element, {
1232
+ includeAlt: false,
1233
+ includeDescendantImageAlt: true,
1234
+ includePlaceholder: false,
1235
+ includeText: true
1236
+ }), 120);
1237
+ if (label) {
1238
+ parts.push(formatValue(label));
1239
+ } else {
1240
+ appendFallbackIdOrName();
1241
+ }
1242
+ } else if (tagName === "INPUT") {
1243
+ const inputType = String(element.getAttribute?.("type") || element.type || "text").toLowerCase();
1244
+ if (["button", "submit", "reset"].includes(inputType)) {
1245
+ const label = truncateText(getLabelText(element, {
1246
+ includeAlt: false,
1247
+ includeDescendantImageAlt: false,
1248
+ includePlaceholder: false,
1249
+ includeText: false
1250
+ }) || element.value || "", 120);
1251
+ if (label) {
1252
+ parts.push(formatValue(label));
1253
+ } else {
1254
+ appendFallbackIdOrName();
1255
+ }
1256
+ } else if (["checkbox", "radio"].includes(inputType)) {
1257
+ const label = truncateText(getLabelText(element, {
1258
+ includeAlt: false,
1259
+ includeDescendantImageAlt: false,
1260
+ includePlaceholder: false,
1261
+ includeText: false
1262
+ }), 120);
1263
+ if (label) {
1264
+ parts.push(formatValue(label));
1265
+ } else {
1266
+ appendFallbackIdOrName();
1267
+ }
1268
+ } else if (inputType === "file") {
1269
+ const label = truncateText(getLabelText(element, {
1270
+ includeAlt: false,
1271
+ includeDescendantImageAlt: false,
1272
+ includePlaceholder: false,
1273
+ includeText: false
1274
+ }), 120);
1275
+ if (label) {
1276
+ parts.push(formatValue(label));
1277
+ } else {
1278
+ appendFallbackIdOrName();
1279
+ }
1280
+ } else {
1281
+ const label = truncateText(getLabelText(element, {
1282
+ includeAlt: false,
1283
+ includeDescendantImageAlt: false,
1284
+ includePlaceholder: false,
1285
+ includeText: false
1286
+ }), 120);
1287
+ if (label) {
1288
+ parts.push(formatValue(label));
1289
+ }
1290
+
1291
+ const placeholder = normalizeAttributeText(element.getAttribute?.("placeholder"));
1292
+ const value = inputType === "password"
1293
+ ? ""
1294
+ : getReferenceValueMetadata(element);
1295
+
1296
+ if (placeholder) {
1297
+ parts.push(`placeholder=${formatValue(placeholder)}`);
1298
+ }
1299
+ if (value) {
1300
+ parts.push(`value=${formatValue(value)}`);
1301
+ }
1302
+ if (!label && !placeholder && !value) {
1303
+ appendFallbackIdOrName();
1304
+ }
1305
+ }
1306
+ } else if (String(element.getAttribute?.("contenteditable") || "").toLowerCase() === "true") {
1307
+ const label = truncateText(getLabelText(element, {
1308
+ includeAlt: false,
1309
+ includeDescendantImageAlt: false,
1310
+ includePlaceholder: false,
1311
+ includeText: true
1312
+ }), 120);
1313
+ if (label) {
1314
+ parts.push(formatValue(label));
1315
+ } else {
1316
+ appendFallbackIdOrName();
1317
+ }
1318
+ } else if (tagName === "IMG") {
1319
+ const srcSummary = summarizeUrl(element.currentSrc || element.getAttribute?.("src") || element.src || "");
1320
+ const label = truncateText(getLabelText(element, {
1321
+ includeAlt: true,
1322
+ includeDescendantImageAlt: false,
1323
+ includePlaceholder: false,
1324
+ includeText: false
1325
+ }), 120);
1326
+ const displayLabel = label || srcSummary;
1327
+ if (displayLabel) {
1328
+ parts.push(formatValue(displayLabel));
1329
+ } else {
1330
+ appendFallbackIdOrName();
1331
+ }
1332
+ } else if (role) {
1333
+ const label = truncateText(getLabelText(element, {
1334
+ includeAlt: false,
1335
+ includeDescendantImageAlt: true,
1336
+ includePlaceholder: false,
1337
+ includeText: true
1338
+ }), 120);
1339
+ if (label) {
1340
+ parts.push(formatValue(label));
1341
+ } else {
1342
+ appendFallbackIdOrName();
1343
+ }
1344
+ } else {
1345
+ const label = truncateText(getLabelText(element, {
1346
+ includeAlt: false,
1347
+ includeDescendantImageAlt: true,
1348
+ includePlaceholder: false,
1349
+ includeText: true
1350
+ }), 120);
1351
+ if (label) {
1352
+ parts.push(formatValue(label));
1353
+ } else {
1354
+ appendFallbackIdOrName();
1355
+ }
1356
+ }
1357
+
1358
+ return {
1359
+ descriptorTags: stateMetadata.descriptorTags.slice(),
1360
+ kind,
1361
+ semanticTags: stateMetadata.semanticTags.slice(),
1362
+ state: stateMetadata,
1363
+ summary: parts.filter(Boolean).join(" ")
1364
+ };
1365
+ }
1366
+
1367
+ function createReferenceEntry(element, referenceId, options = {}) {
1368
+ const nodeId = normalizeAttributeText(element.getAttribute?.("data-space-browser-node-id"));
1369
+ const frameId = normalizeAttributeText(element.getAttribute?.("data-space-browser-frame-id"));
1370
+ const frameChain = normalizeFrameChain(element.getAttribute?.("data-space-browser-frame-chain"));
1371
+ const helperBacked = Boolean(nodeId && frameChain.length);
1372
+ const summaryData = collectReferenceSummaryData(element, options);
1373
+
1374
+ return {
1375
+ connected: helperBacked ? true : element.isConnected !== false,
1376
+ dom: serializeElementSnapshot(element),
1377
+ descriptorTags: summaryData.descriptorTags,
1378
+ element: helperBacked ? null : element,
1379
+ frameChain,
1380
+ frameId,
1381
+ helperBacked,
1382
+ id: normalizeAttributeText(element.getAttribute?.("id")),
1383
+ name: normalizeAttributeText(element.getAttribute?.("name")),
1384
+ nodeId,
1385
+ referenceId,
1386
+ kind: summaryData.kind,
1387
+ semanticTags: summaryData.semanticTags,
1388
+ state: summaryData.state,
1389
+ summary: summaryData.summary,
1390
+ tagName: getTagName(element)
1391
+ };
1392
+ }
1393
+
1394
+ function ensureReference(element, context) {
1395
+ if (context.referenceIdsByElement.has(element)) {
1396
+ return context.referenceIdsByElement.get(element);
1397
+ }
1398
+
1399
+ const referenceId = String(context.nextReferenceId++);
1400
+ const entry = createReferenceEntry(element, referenceId, context.options);
1401
+ context.referenceIdsByElement.set(element, referenceId);
1402
+ context.entries.set(referenceId, entry);
1403
+ return referenceId;
1404
+ }
1405
+
1406
+ function renderReference(element, context) {
1407
+ const referenceId = ensureReference(element, context);
1408
+ const entry = context.entries.get(referenceId);
1409
+ const kind = normalizeText(entry?.kind || getTagName(element).toLowerCase());
1410
+ const descriptorTags = Array.isArray(entry?.descriptorTags)
1411
+ ? entry.descriptorTags.map((tag) => normalizeText(tag)).filter(Boolean)
1412
+ : [];
1413
+ const summary = normalizeText(entry?.summary || "");
1414
+ const descriptor = [...descriptorTags, kind, referenceId].filter(Boolean).join(" ");
1415
+ return summary ? `[${descriptor}] ${summary}` : `[${descriptor}]`;
1416
+ }
1417
+
1418
+ function isReferenceableElement(element) {
1419
+ return isInteractiveElement(element) || getTagName(element) === "IMG";
1420
+ }
1421
+
1422
+ function renderInlineNode(node, context) {
1423
+ if (isTextNode(node)) {
1424
+ const textContent = normalizeText(node.textContent || "");
1425
+ if (shouldDropReadableText(textContent)) {
1426
+ return "";
1427
+ }
1428
+
1429
+ return escapeMarkdownText(textContent);
1430
+ }
1431
+
1432
+ if (!isElementNode(node) || isHiddenElement(node)) {
1433
+ return "";
1434
+ }
1435
+
1436
+ if (isReferenceableElement(node)) {
1437
+ return renderReference(node, context);
1438
+ }
1439
+
1440
+ const tagName = getTagName(node);
1441
+
1442
+ if (tagName === "LABEL" && (node.getAttribute?.("for") || node.querySelector?.("input, textarea, select, button"))) {
1443
+ return "";
1444
+ }
1445
+
1446
+ if (tagName === "BR") {
1447
+ return "\n";
1448
+ }
1449
+
1450
+ if (tagName === "STRONG" || tagName === "B") {
1451
+ const content = renderInlineChildren(node, context);
1452
+ return content ? `**${content}**` : "";
1453
+ }
1454
+
1455
+ if (tagName === "EM" || tagName === "I") {
1456
+ const content = renderInlineChildren(node, context);
1457
+ return content ? `*${content}*` : "";
1458
+ }
1459
+
1460
+ if (tagName === "S" || tagName === "STRIKE" || tagName === "DEL") {
1461
+ const content = renderInlineChildren(node, context);
1462
+ return content ? `~~${content}~~` : "";
1463
+ }
1464
+
1465
+ if (tagName === "CODE") {
1466
+ const content = normalizeText(node.textContent || "");
1467
+ return content ? `\`${content.replace(/`/gu, "\\`")}\`` : "";
1468
+ }
1469
+
1470
+ return renderInlineChildren(node, context);
1471
+ }
1472
+
1473
+ function renderInlineChildren(element, context) {
1474
+ const parts = [];
1475
+
1476
+ element.childNodes.forEach((childNode) => {
1477
+ const renderedChild = renderInlineNode(childNode, context);
1478
+ if (renderedChild) {
1479
+ parts.push(renderedChild);
1480
+ }
1481
+ });
1482
+
1483
+ return joinInlineParts(parts);
1484
+ }
1485
+
1486
+ function renderParagraph(element, context) {
1487
+ return renderInlineChildren(element, context);
1488
+ }
1489
+
1490
+ function renderHeading(element, context) {
1491
+ const level = Math.min(6, Math.max(1, Number.parseInt(getTagName(element).slice(1), 10) || 1));
1492
+ const content = renderInlineChildren(element, context);
1493
+ return content ? `${"#".repeat(level)} ${content}` : "";
1494
+ }
1495
+
1496
+ function renderCodeBlock(element) {
1497
+ const content = String(element.textContent || "").trimEnd();
1498
+ if (!content) {
1499
+ return "";
1500
+ }
1501
+
1502
+ return `\`\`\`\n${content.replace(/```/gu, "\\`\\`\\`")}\n\`\`\``;
1503
+ }
1504
+
1505
+ function renderBlockquote(element, context) {
1506
+ const content = renderBlockChildren(element, context);
1507
+ if (!content) {
1508
+ return "";
1509
+ }
1510
+
1511
+ return content
1512
+ .split("\n")
1513
+ .map((line) => `> ${line}`)
1514
+ .join("\n");
1515
+ }
1516
+
1517
+ function renderListItem(element, context, depth, index, ordered) {
1518
+ const includeListMarkers = context.options.includeListMarkers === true;
1519
+ const includeListIndentation = context.options.includeListIndentation !== false;
1520
+ const marker = includeListMarkers ? (ordered ? `${index + 1}.` : "-") : "";
1521
+ const indentation = includeListIndentation ? " ".repeat(Math.max(0, depth)) : "";
1522
+ const inlineParts = [];
1523
+ const nestedBlocks = [];
1524
+
1525
+ element.childNodes.forEach((childNode) => {
1526
+ if (isElementNode(childNode) && (getTagName(childNode) === "UL" || getTagName(childNode) === "OL")) {
1527
+ const nestedList = renderList(childNode, context, depth + 1);
1528
+ if (nestedList) {
1529
+ nestedBlocks.push(nestedList);
1530
+ }
1531
+ return;
1532
+ }
1533
+
1534
+ const renderedChild = renderInlineNode(childNode, context);
1535
+ if (renderedChild) {
1536
+ inlineParts.push(renderedChild);
1537
+ }
1538
+ });
1539
+
1540
+ const head = joinInlineParts(inlineParts);
1541
+ const linePrefix = marker ? `${indentation}${marker} ` : indentation;
1542
+ const lines = [`${linePrefix}${head || "(empty)"}`];
1543
+ nestedBlocks.forEach((nestedBlock) => {
1544
+ lines.push(indentBlock(nestedBlock, includeListIndentation ? 1 : 0));
1545
+ });
1546
+ return lines.join("\n");
1547
+ }
1548
+
1549
+ function renderList(element, context, depth = 0) {
1550
+ const ordered = getTagName(element) === "OL";
1551
+ return [...element.children]
1552
+ .filter((child) => getTagName(child) === "LI" && !isHiddenElement(child))
1553
+ .map((item, index) => renderListItem(item, context, depth, index, ordered))
1554
+ .filter(Boolean)
1555
+ .join("\n");
1556
+ }
1557
+
1558
+ function renderTableCell(element, context) {
1559
+ return renderInlineChildren(element, context);
1560
+ }
1561
+
1562
+ function renderTable(element, context) {
1563
+ const rows = [...element.querySelectorAll?.(":scope > thead > tr, :scope > tbody > tr, :scope > tr, :scope > tfoot > tr") || []]
1564
+ .filter((row) => getTagName(row) === "TR");
1565
+
1566
+ if (!rows.length) {
1567
+ return "";
1568
+ }
1569
+
1570
+ const renderedRows = rows.map((row) => {
1571
+ return [...row.children]
1572
+ .filter((cell) => ["TD", "TH"].includes(getTagName(cell)) && !isHiddenElement(cell))
1573
+ .map((cell) => renderTableCell(cell, context));
1574
+ }).filter((cells) => cells.length);
1575
+
1576
+ if (!renderedRows.length) {
1577
+ return "";
1578
+ }
1579
+
1580
+ const columnCount = Math.max(...renderedRows.map((cells) => cells.length));
1581
+ const normalizedRows = renderedRows.map((cells) => {
1582
+ const nextCells = cells.slice();
1583
+ while (nextCells.length < columnCount) {
1584
+ nextCells.push("");
1585
+ }
1586
+ return nextCells;
1587
+ });
1588
+
1589
+ const headerRow = normalizedRows[0];
1590
+ const separatorRow = headerRow.map(() => "---");
1591
+ const tableLines = [
1592
+ `| ${headerRow.join(" | ")} |`,
1593
+ `| ${separatorRow.join(" | ")} |`
1594
+ ];
1595
+
1596
+ normalizedRows.slice(1).forEach((row) => {
1597
+ tableLines.push(`| ${row.join(" | ")} |`);
1598
+ });
1599
+
1600
+ return tableLines.join("\n");
1601
+ }
1602
+
1603
+ function renderGenericContainer(element, context) {
1604
+ return renderBlockChildren(element, context);
1605
+ }
1606
+
1607
+ function renderElementAsBlock(element, context) {
1608
+ if (!isElementNode(element) || isHiddenElement(element)) {
1609
+ return "";
1610
+ }
1611
+
1612
+ if (isReferenceableElement(element)) {
1613
+ return renderReference(element, context);
1614
+ }
1615
+
1616
+ const tagName = getTagName(element);
1617
+
1618
+ if (tagName === "LABEL" && (element.getAttribute?.("for") || element.querySelector?.("input, textarea, select, button"))) {
1619
+ return "";
1620
+ }
1621
+
1622
+ if (/^H[1-6]$/u.test(tagName)) {
1623
+ return renderHeading(element, context);
1624
+ }
1625
+
1626
+ if (tagName === "P") {
1627
+ return renderParagraph(element, context);
1628
+ }
1629
+
1630
+ if (tagName === "PRE") {
1631
+ return renderCodeBlock(element);
1632
+ }
1633
+
1634
+ if (tagName === "BLOCKQUOTE") {
1635
+ return renderBlockquote(element, context);
1636
+ }
1637
+
1638
+ if (tagName === "UL" || tagName === "OL") {
1639
+ return renderList(element, context);
1640
+ }
1641
+
1642
+ if (tagName === "TABLE") {
1643
+ return renderTable(element, context);
1644
+ }
1645
+
1646
+ if (tagName === "HR") {
1647
+ return "---";
1648
+ }
1649
+
1650
+ return renderGenericContainer(element, context);
1651
+ }
1652
+
1653
+ function renderBlockChildren(element, context) {
1654
+ const blocks = [];
1655
+ const inlineParts = [];
1656
+
1657
+ const flushInlineParts = () => {
1658
+ const inlineText = joinInlineParts(inlineParts.splice(0, inlineParts.length));
1659
+ if (inlineText) {
1660
+ blocks.push(inlineText);
1661
+ }
1662
+ };
1663
+
1664
+ element.childNodes.forEach((childNode) => {
1665
+ if (isTextNode(childNode)) {
1666
+ const rawTextContent = normalizeText(childNode.textContent || "");
1667
+ if (shouldDropReadableText(rawTextContent)) {
1668
+ return;
1669
+ }
1670
+
1671
+ const textContent = escapeMarkdownText(rawTextContent);
1672
+ if (textContent) {
1673
+ inlineParts.push(textContent);
1674
+ }
1675
+ return;
1676
+ }
1677
+
1678
+ if (!isElementNode(childNode) || isHiddenElement(childNode)) {
1679
+ return;
1680
+ }
1681
+
1682
+ const renderedChild = renderElementAsBlock(childNode, context);
1683
+ if (!renderedChild) {
1684
+ return;
1685
+ }
1686
+
1687
+ if (isBlockElement(childNode) || isReferenceableElement(childNode)) {
1688
+ flushInlineParts();
1689
+ blocks.push(renderedChild);
1690
+ return;
1691
+ }
1692
+
1693
+ inlineParts.push(renderedChild);
1694
+ });
1695
+
1696
+ flushInlineParts();
1697
+ return joinBlocks(blocks);
1698
+ }
1699
+
1700
+ function createCaptureContext(payload = null) {
1701
+ return {
1702
+ entries: new Map(),
1703
+ nextReferenceId: 1,
1704
+ options: {
1705
+ includeLabelQuotes: normalizeIncludeLabelQuotes(payload),
1706
+ includeLinkUrls: normalizeIncludeLinkUrls(payload),
1707
+ includeSemanticTags: normalizeIncludeSemanticTags(payload),
1708
+ includeStateTags: normalizeIncludeStateTags(payload),
1709
+ includeListIndentation: normalizeIncludeListIndentation(payload),
1710
+ includeListMarkers: normalizeIncludeListMarkers(payload)
1711
+ },
1712
+ referenceIdsByElement: new WeakMap()
1713
+ };
1714
+ }
1715
+
1716
+ function resolveSelectorTargets(payload, doc = globalThis.document) {
1717
+ const selectors = normalizeSelectorList(payload);
1718
+ if (!selectors.length) {
1719
+ return {
1720
+ includeMetaData: true,
1721
+ items: [
1722
+ {
1723
+ key: "document",
1724
+ targets: [doc?.body || doc?.documentElement].filter(Boolean)
1725
+ }
1726
+ ]
1727
+ };
1728
+ }
1729
+
1730
+ return {
1731
+ includeMetaData: false,
1732
+ items: selectors.map((selector) => {
1733
+ let targets = [];
1734
+ try {
1735
+ targets = [...(doc?.querySelectorAll?.(selector) || [])];
1736
+ } catch (error) {
1737
+ throw createNamedError(
1738
+ "BrowserPageContentSelectorError",
1739
+ `Browser page content could not resolve selector "${selector}".`,
1740
+ {
1741
+ code: "browser_page_content_selector_error",
1742
+ details: {
1743
+ selector
1744
+ },
1745
+ cause: error
1746
+ }
1747
+ );
1748
+ }
1749
+
1750
+ return {
1751
+ key: selector,
1752
+ targets
1753
+ };
1754
+ })
1755
+ };
1756
+ }
1757
+
1758
+ function parseSnapshotFragment(html, parser) {
1759
+ return parser.parseFromString(
1760
+ `<!DOCTYPE html><html><body>${String(html || "")}</body></html>`,
1761
+ "text/html"
1762
+ );
1763
+ }
1764
+
1765
+ function renderSnapshotFragment(html, captureContext, parser) {
1766
+ const parsedDocument = parseSnapshotFragment(html, parser);
1767
+ const blocks = [];
1768
+ const inlineParts = [];
1769
+
1770
+ const flushInlineParts = () => {
1771
+ const inlineText = joinInlineParts(inlineParts.splice(0, inlineParts.length));
1772
+ if (inlineText) {
1773
+ blocks.push(inlineText);
1774
+ }
1775
+ };
1776
+
1777
+ parsedDocument.body.childNodes.forEach((childNode) => {
1778
+ if (isTextNode(childNode)) {
1779
+ const rawTextContent = normalizeText(childNode.textContent || "");
1780
+ if (shouldDropReadableText(rawTextContent)) {
1781
+ return;
1782
+ }
1783
+
1784
+ const textContent = escapeMarkdownText(rawTextContent);
1785
+ if (textContent) {
1786
+ inlineParts.push(textContent);
1787
+ }
1788
+ return;
1789
+ }
1790
+
1791
+ if (!isElementNode(childNode) || isHiddenElement(childNode)) {
1792
+ return;
1793
+ }
1794
+
1795
+ const renderedChild = renderElementAsBlock(childNode, captureContext);
1796
+ if (!renderedChild) {
1797
+ return;
1798
+ }
1799
+
1800
+ if (isBlockElement(childNode) || isReferenceableElement(childNode)) {
1801
+ flushInlineParts();
1802
+ blocks.push(renderedChild);
1803
+ return;
1804
+ }
1805
+
1806
+ inlineParts.push(renderedChild);
1807
+ });
1808
+
1809
+ flushInlineParts();
1810
+ return cleanReadableMarkdown(joinBlocks(blocks));
1811
+ }
1812
+
1813
+ function captureLive(payload = null) {
1814
+ const captureContext = createCaptureContext(payload);
1815
+ const resolvedTargets = resolveSelectorTargets(payload);
1816
+ const snapshot = {};
1817
+
1818
+ resolvedTargets.items.forEach((item) => {
1819
+ const blocks = [];
1820
+ if (resolvedTargets.includeMetaData && item.key === "document") {
1821
+ const meta = collectMetaLines(globalThis.document);
1822
+ if (meta) {
1823
+ blocks.push(meta);
1824
+ }
1825
+ }
1826
+
1827
+ item.targets.forEach((target) => {
1828
+ const renderedTarget = renderElementAsBlock(target, captureContext);
1829
+ if (renderedTarget) {
1830
+ blocks.push(renderedTarget);
1831
+ }
1832
+ });
1833
+
1834
+ snapshot[item.key] = cleanReadableMarkdown(joinBlocks(blocks));
1835
+ });
1836
+
1837
+ state.captureId += 1;
1838
+ state.capturedAt = Date.now();
1839
+ state.backend = "live";
1840
+ state.captureOptions = { ...captureContext.options };
1841
+ state.entries = captureContext.entries;
1842
+ return snapshot;
1843
+ }
1844
+
1845
+ async function captureWithDomHelper(payload = null) {
1846
+ const helper = requireDomHelper("capture content");
1847
+ const selectors = normalizeSelectorList(payload);
1848
+ const helperPayload = {
1849
+ snapshotMode: "content"
1850
+ };
1851
+ if (selectors.length) {
1852
+ helperPayload.selectors = selectors;
1853
+ }
1854
+ const documentSnapshot = await helper.captureDocument({
1855
+ ...helperPayload
1856
+ });
1857
+ const snapshot = {};
1858
+ const parser = new globalThis.DOMParser();
1859
+ const captureContext = createCaptureContext(payload);
1860
+ try {
1861
+ if (selectors.length && documentSnapshot?.targets && typeof documentSnapshot.targets === "object") {
1862
+ selectors.forEach((selector) => {
1863
+ snapshot[selector] = renderSnapshotFragment(documentSnapshot.targets?.[selector] || "", captureContext, parser);
1864
+ });
1865
+
1866
+ state.captureId += 1;
1867
+ state.capturedAt = Date.now();
1868
+ state.backend = "dom_helper";
1869
+ state.captureOptions = { ...captureContext.options };
1870
+ state.entries = captureContext.entries;
1871
+ return snapshot;
1872
+ }
1873
+
1874
+ const parsedDocument = parser.parseFromString(String(documentSnapshot?.html || ""), "text/html");
1875
+ const resolvedTargets = resolveSelectorTargets(payload, parsedDocument);
1876
+
1877
+ resolvedTargets.items.forEach((item) => {
1878
+ const blocks = [];
1879
+ if (resolvedTargets.includeMetaData && item.key === "document") {
1880
+ const meta = collectMetaLines(parsedDocument);
1881
+ if (meta) {
1882
+ blocks.push(meta);
1883
+ }
1884
+ }
1885
+
1886
+ item.targets.forEach((target) => {
1887
+ const renderedTarget = renderElementAsBlock(target, captureContext);
1888
+ if (renderedTarget) {
1889
+ blocks.push(renderedTarget);
1890
+ }
1891
+ });
1892
+
1893
+ snapshot[item.key] = cleanReadableMarkdown(joinBlocks(blocks));
1894
+ });
1895
+
1896
+ state.captureId += 1;
1897
+ state.capturedAt = Date.now();
1898
+ state.backend = "dom_helper";
1899
+ state.captureOptions = { ...captureContext.options };
1900
+ state.entries = captureContext.entries;
1901
+ return snapshot;
1902
+ } catch (error) {
1903
+ if (!isTrustedHtmlRequirementError(error)) {
1904
+ throw error;
1905
+ }
1906
+
1907
+ return captureLive(payload);
1908
+ }
1909
+ }
1910
+
1911
+ async function capture(payload = null) {
1912
+ if (getDomHelper()) {
1913
+ return captureWithDomHelper(payload);
1914
+ }
1915
+
1916
+ return captureLive(payload);
1917
+ }
1918
+
1919
+ function detailLive(entry) {
1920
+ const liveState = entry.connected && entry.element
1921
+ ? collectElementStateMetadata(entry.element, state.captureOptions)
1922
+ : entry.state || collectElementStateMetadata(null);
1923
+ return {
1924
+ captureId: state.captureId,
1925
+ capturedAt: state.capturedAt,
1926
+ connected: entry.connected,
1927
+ descriptorTags: liveState.descriptorTags,
1928
+ dom: entry.connected ? serializeElementSnapshot(entry.element) || entry.dom : entry.dom,
1929
+ referenceId: entry.referenceId,
1930
+ semanticTags: liveState.semanticTags,
1931
+ state: liveState,
1932
+ summary: entry.summary,
1933
+ tagName: entry.tagName
1934
+ };
1935
+ }
1936
+
1937
+ async function detail(referenceId) {
1938
+ const entry = requireReferenceEntry(referenceId, {
1939
+ actionLabel: "detail",
1940
+ requireConnected: false
1941
+ });
1942
+
1943
+ if (entry.helperBacked) {
1944
+ const helper = requireDomHelper("resolve detail");
1945
+ const resolvedDetail = await helper.detailNode(entry.frameChain, entry.nodeId);
1946
+ return {
1947
+ captureId: state.captureId,
1948
+ capturedAt: state.capturedAt,
1949
+ connected: resolvedDetail?.connected !== false,
1950
+ descriptorTags: Array.isArray(resolvedDetail?.descriptorTags) ? resolvedDetail.descriptorTags : (entry.descriptorTags || []),
1951
+ dom: String(resolvedDetail?.dom || entry.dom || ""),
1952
+ frameChain: entry.frameChain.slice(),
1953
+ frameId: entry.frameId,
1954
+ nodeId: entry.nodeId,
1955
+ referenceId: entry.referenceId,
1956
+ semanticTags: Array.isArray(resolvedDetail?.semanticTags) ? resolvedDetail.semanticTags : (entry.semanticTags || []),
1957
+ state: resolvedDetail?.state || entry.state || collectElementStateMetadata(null),
1958
+ summary: entry.summary,
1959
+ tagName: String(resolvedDetail?.tagName || entry.tagName || "")
1960
+ };
1961
+ }
1962
+
1963
+ return detailLive(entry);
1964
+ }
1965
+
1966
+ function requireReferenceEntry(referenceId, options = {}) {
1967
+ const normalizedReferenceId = normalizeReferenceId(referenceId);
1968
+ if (!normalizedReferenceId) {
1969
+ throw createNamedError(
1970
+ "BrowserPageContentReferenceError",
1971
+ "Browser page content requests require a reference id.",
1972
+ {
1973
+ code: "browser_page_content_reference_required",
1974
+ details: {
1975
+ action: String(options.actionLabel || "resolve")
1976
+ }
1977
+ }
1978
+ );
1979
+ }
1980
+
1981
+ if (!state.entries.size) {
1982
+ throw createNamedError(
1983
+ "BrowserPageContentReferenceError",
1984
+ `Browser page content has no reference capture for "${normalizedReferenceId}".`,
1985
+ {
1986
+ code: "browser_page_content_reference_missing_capture",
1987
+ details: {
1988
+ action: String(options.actionLabel || "resolve"),
1989
+ referenceId: normalizedReferenceId
1990
+ }
1991
+ }
1992
+ );
1993
+ }
1994
+
1995
+ const entry = state.entries.get(normalizedReferenceId);
1996
+ if (!entry) {
1997
+ throw createNamedError(
1998
+ "BrowserPageContentReferenceError",
1999
+ `Browser page content could not find reference "${normalizedReferenceId}".`,
2000
+ {
2001
+ code: "browser_page_content_reference_not_found",
2002
+ details: {
2003
+ action: String(options.actionLabel || "resolve"),
2004
+ referenceId: normalizedReferenceId
2005
+ }
2006
+ }
2007
+ );
2008
+ }
2009
+
2010
+ refreshReferenceEntry(entry);
2011
+
2012
+ if (options.requireConnected !== false && !entry.connected) {
2013
+ throw createNamedError(
2014
+ "BrowserPageContentReferenceError",
2015
+ `Browser page content reference "${normalizedReferenceId}" is no longer connected.`,
2016
+ {
2017
+ code: "browser_page_content_reference_disconnected",
2018
+ details: {
2019
+ action: String(options.actionLabel || "resolve"),
2020
+ referenceId: normalizedReferenceId
2021
+ }
2022
+ }
2023
+ );
2024
+ }
2025
+
2026
+ return entry;
2027
+ }
2028
+
2029
+ function refreshReferenceEntry(entry) {
2030
+ if (!entry || entry.helperBacked || !entry.element) {
2031
+ return entry;
2032
+ }
2033
+
2034
+ entry.connected = entry.element.isConnected !== false;
2035
+ if (entry.connected) {
2036
+ entry.dom = serializeElementSnapshot(entry.element) || entry.dom;
2037
+ entry.id = normalizeAttributeText(entry.element.getAttribute?.("id"));
2038
+ entry.name = normalizeAttributeText(entry.element.getAttribute?.("name"));
2039
+ const summaryData = collectReferenceSummaryData(entry.element, state.captureOptions);
2040
+ entry.descriptorTags = summaryData.descriptorTags;
2041
+ entry.kind = summaryData.kind;
2042
+ entry.semanticTags = summaryData.semanticTags;
2043
+ entry.state = summaryData.state;
2044
+ entry.summary = summaryData.summary;
2045
+ entry.tagName = getTagName(entry.element);
2046
+ }
2047
+
2048
+ return entry;
2049
+ }
2050
+
2051
+ function scrollElementIntoView(element) {
2052
+ try {
2053
+ element.scrollIntoView?.({
2054
+ behavior: "auto",
2055
+ block: "center",
2056
+ inline: "center"
2057
+ });
2058
+ return true;
2059
+ } catch {
2060
+ return false;
2061
+ }
2062
+ }
2063
+
2064
+ function focusElement(element) {
2065
+ try {
2066
+ element.focus?.({
2067
+ preventScroll: true
2068
+ });
2069
+ return true;
2070
+ } catch {
2071
+ try {
2072
+ element.focus?.();
2073
+ return true;
2074
+ } catch {
2075
+ return false;
2076
+ }
2077
+ }
2078
+ }
2079
+
2080
+ function describeActiveElement(element) {
2081
+ if (!isElementNode(element)) {
2082
+ return "";
2083
+ }
2084
+
2085
+ const tagName = getTagName(element).toLowerCase();
2086
+ const id = normalizeAttributeText(element.getAttribute?.("id"));
2087
+ const name = normalizeAttributeText(element.getAttribute?.("name"));
2088
+ const label = truncateText(getLabelText(element, {
2089
+ includeAlt: false,
2090
+ includeDescendantImageAlt: true,
2091
+ includePlaceholder: false,
2092
+ includeText: false
2093
+ }), 48);
2094
+ return [tagName, id ? `#${id}` : "", name ? `name=${name}` : "", label].filter(Boolean).join(" ");
2095
+ }
2096
+
2097
+ function getActionObservationRoot(element) {
2098
+ if (!isElementNode(element)) {
2099
+ return globalThis.document?.body || globalThis.document?.documentElement || null;
2100
+ }
2101
+
2102
+ return element.closest?.("form, fieldset, dialog, [role='dialog'], [role='alert'], [role='status'], [aria-live], article, section, main, li, tr, td, th")
2103
+ || element.parentElement
2104
+ || element;
2105
+ }
2106
+
2107
+ function getElementDirectText(element) {
2108
+ if (!isElementNode(element)) {
2109
+ return "";
2110
+ }
2111
+
2112
+ return normalizeText(
2113
+ [...(element.childNodes || [])]
2114
+ .filter((node) => isTextNode(node))
2115
+ .map((node) => node.textContent || "")
2116
+ .join(" ")
2117
+ );
2118
+ }
2119
+
2120
+ function collectNearbyTextEntries(root, limit = 24) {
2121
+ if (!isElementNode(root)) {
2122
+ return [];
2123
+ }
2124
+
2125
+ const entries = [];
2126
+ const seen = new Set();
2127
+ const acceptElement = (element) => {
2128
+ if (!isElementNode(element) || isHiddenElement(element) || entries.length >= limit) {
2129
+ return;
2130
+ }
2131
+
2132
+ const role = normalizeText(element.getAttribute?.("role")).toLowerCase();
2133
+ const directText = getElementDirectText(element);
2134
+ const fallbackText = ["alert", "status"].includes(role) || element.hasAttribute?.("aria-live")
2135
+ ? getElementText(element)
2136
+ : "";
2137
+ const text = truncateText(directText || fallbackText, 220);
2138
+ if (!text) {
2139
+ return;
2140
+ }
2141
+
2142
+ const key = `${role}|${text}`;
2143
+ if (seen.has(key)) {
2144
+ return;
2145
+ }
2146
+ seen.add(key);
2147
+ const state = collectElementStateMetadata(element, {
2148
+ includeSemanticTags: true,
2149
+ includeStateTags: true
2150
+ });
2151
+ entries.push({
2152
+ invalid: state.invalid === true,
2153
+ role,
2154
+ semanticTone: state.semanticTone || "",
2155
+ text
2156
+ });
2157
+ };
2158
+
2159
+ acceptElement(root);
2160
+ const walker = globalThis.document?.createTreeWalker?.(root, globalThis.NodeFilter?.SHOW_ELEMENT ?? 1);
2161
+ if (!walker) {
2162
+ return entries;
2163
+ }
2164
+
2165
+ let currentNode = walker.nextNode();
2166
+ while (currentNode && entries.length < limit) {
2167
+ acceptElement(currentNode);
2168
+ currentNode = walker.nextNode();
2169
+ }
2170
+
2171
+ return entries;
2172
+ }
2173
+
2174
+ function captureActionEffectSnapshot(element) {
2175
+ const observationRoot = getActionObservationRoot(element);
2176
+ return {
2177
+ activeElement: describeActiveElement(globalThis.document?.activeElement),
2178
+ observationRoot,
2179
+ observationText: truncateText(getElementText(observationRoot), 2000),
2180
+ targetDom: truncateText(serializeElementSnapshot(element), 2000),
2181
+ targetState: collectElementStateMetadata(element, {
2182
+ includeSemanticTags: true,
2183
+ includeStateTags: true
2184
+ }),
2185
+ textEntries: collectNearbyTextEntries(observationRoot),
2186
+ value: getReferenceValueMetadata(element)
2187
+ };
2188
+ }
2189
+
2190
+ async function waitForObservedActionWindow(observationRoot, {
2191
+ quietMs = 40,
2192
+ timeoutMs = 180
2193
+ } = {}) {
2194
+ const target = observationRoot?.ownerDocument?.body
2195
+ || observationRoot?.ownerDocument?.documentElement
2196
+ || globalThis.document?.body
2197
+ || globalThis.document?.documentElement;
2198
+ if (!target || typeof globalThis.MutationObserver !== "function") {
2199
+ await delayMs(timeoutMs);
2200
+ return {
2201
+ attributeNames: [],
2202
+ mutationCount: 0
2203
+ };
2204
+ }
2205
+
2206
+ const attributeNames = new Set();
2207
+ let lastMutationAt = 0;
2208
+ let mutationCount = 0;
2209
+ const observer = new globalThis.MutationObserver((mutations) => {
2210
+ mutationCount += mutations.length;
2211
+ lastMutationAt = Date.now();
2212
+ mutations.forEach((mutation) => {
2213
+ if (mutation.type === "attributes" && mutation.attributeName) {
2214
+ attributeNames.add(String(mutation.attributeName));
2215
+ }
2216
+ });
2217
+ });
2218
+
2219
+ try {
2220
+ observer.observe(target, {
2221
+ attributes: true,
2222
+ characterData: true,
2223
+ childList: true,
2224
+ subtree: true
2225
+ });
2226
+ const startedAt = Date.now();
2227
+ while (Date.now() - startedAt < timeoutMs) {
2228
+ await delayMs(20);
2229
+ if (mutationCount > 0 && Date.now() - lastMutationAt >= quietMs) {
2230
+ break;
2231
+ }
2232
+ }
2233
+ } finally {
2234
+ observer.disconnect();
2235
+ }
2236
+
2237
+ return {
2238
+ attributeNames: [...attributeNames],
2239
+ mutationCount
2240
+ };
2241
+ }
2242
+
2243
+ async function withObservedActionWindow(observationRoot, action, options = {}) {
2244
+ const target = observationRoot?.ownerDocument?.body
2245
+ || observationRoot?.ownerDocument?.documentElement
2246
+ || globalThis.document?.body
2247
+ || globalThis.document?.documentElement;
2248
+ if (!target || typeof globalThis.MutationObserver !== "function") {
2249
+ const result = await action();
2250
+ const observedMutations = await waitForObservedActionWindow(observationRoot, options);
2251
+ return {
2252
+ observedMutations,
2253
+ result
2254
+ };
2255
+ }
2256
+
2257
+ const attributeNames = new Set();
2258
+ let lastMutationAt = 0;
2259
+ let mutationCount = 0;
2260
+ const observer = new globalThis.MutationObserver((mutations) => {
2261
+ mutationCount += mutations.length;
2262
+ lastMutationAt = Date.now();
2263
+ mutations.forEach((mutation) => {
2264
+ if (mutation.type === "attributes" && mutation.attributeName) {
2265
+ attributeNames.add(String(mutation.attributeName));
2266
+ }
2267
+ });
2268
+ });
2269
+
2270
+ try {
2271
+ observer.observe(target, {
2272
+ attributes: true,
2273
+ characterData: true,
2274
+ childList: true,
2275
+ subtree: true
2276
+ });
2277
+ const result = await action();
2278
+ const quietMs = Math.max(0, Number(options.quietMs) || 40);
2279
+ const timeoutMs = Math.max(0, Number(options.timeoutMs) || 180);
2280
+ const startedAt = Date.now();
2281
+ while (Date.now() - startedAt < timeoutMs) {
2282
+ await delayMs(20);
2283
+ if (mutationCount > 0 && Date.now() - lastMutationAt >= quietMs) {
2284
+ break;
2285
+ }
2286
+ }
2287
+ return {
2288
+ observedMutations: {
2289
+ attributeNames: [...attributeNames],
2290
+ mutationCount
2291
+ },
2292
+ result
2293
+ };
2294
+ } finally {
2295
+ observer.disconnect();
2296
+ }
2297
+ }
2298
+
2299
+ function compareDescriptorTags(beforeTags = [], afterTags = []) {
2300
+ const beforeValue = beforeTags.filter(Boolean).join("|");
2301
+ const afterValue = afterTags.filter(Boolean).join("|");
2302
+ return beforeValue !== afterValue;
2303
+ }
2304
+
2305
+ function buildActionEffectResult(entry, beforeSnapshot, afterSnapshot, observedMutations, extra = {}) {
2306
+ const newTextEntries = afterSnapshot.textEntries.filter((entryData) => {
2307
+ return !beforeSnapshot.textEntries.some((beforeEntry) => beforeEntry.text === entryData.text);
2308
+ });
2309
+ const validationEntries = newTextEntries.filter((entryData) => {
2310
+ return entryData.invalid
2311
+ || ["alert", "status"].includes(entryData.role)
2312
+ || ["error", "warning"].includes(entryData.semanticTone);
2313
+ });
2314
+ const focusChanged = beforeSnapshot.activeElement !== afterSnapshot.activeElement;
2315
+ const nearbyTextChanged = beforeSnapshot.observationText !== afterSnapshot.observationText;
2316
+ const valueChanged = beforeSnapshot.value !== afterSnapshot.value;
2317
+ const checkedChanged = beforeSnapshot.targetState.checked !== afterSnapshot.targetState.checked;
2318
+ const selectedChanged = beforeSnapshot.targetState.selected !== afterSnapshot.targetState.selected;
2319
+ const expandedChanged = beforeSnapshot.targetState.expanded !== afterSnapshot.targetState.expanded;
2320
+ const pressedChanged = beforeSnapshot.targetState.pressed !== afterSnapshot.targetState.pressed;
2321
+ const descriptorChanged = compareDescriptorTags(beforeSnapshot.targetState.descriptorTags, afterSnapshot.targetState.descriptorTags);
2322
+ const targetDomChanged = beforeSnapshot.targetDom !== afterSnapshot.targetDom;
2323
+ const domChanged = Boolean(observedMutations.mutationCount) || targetDomChanged || nearbyTextChanged;
2324
+ const status = {
2325
+ alertTextAdded: newTextEntries.some((entryData) => ["alert", "status"].includes(entryData.role)),
2326
+ checkedChanged,
2327
+ descriptorChanged,
2328
+ domChanged,
2329
+ expandedChanged,
2330
+ focusChanged,
2331
+ nearbyTextChanged,
2332
+ pressedChanged,
2333
+ reacted: false,
2334
+ selectedChanged,
2335
+ targetChanged: descriptorChanged || targetDomChanged || valueChanged || checkedChanged || selectedChanged || expandedChanged || pressedChanged,
2336
+ targetDomChanged,
2337
+ valueChanged,
2338
+ validationTextAdded: validationEntries.length > 0
2339
+ };
2340
+ status.reacted = Object.entries(status).some(([key, value]) => key !== "reacted" && value === true);
2341
+ status.noObservedEffect = !status.reacted;
2342
+
2343
+ return {
2344
+ ...extra,
2345
+ descriptorTags: afterSnapshot.targetState.descriptorTags.slice(),
2346
+ effect: {
2347
+ mutationAttributes: observedMutations.attributeNames.slice(0, 8),
2348
+ mutationCount: observedMutations.mutationCount,
2349
+ newText: newTextEntries.map((entryData) => entryData.text).slice(0, 3),
2350
+ semanticHints: [...new Set(newTextEntries.map((entryData) => entryData.semanticTone).filter(Boolean))].slice(0, 3),
2351
+ validationText: validationEntries.map((entryData) => entryData.text).slice(0, 3)
2352
+ },
2353
+ semanticTags: afterSnapshot.targetState.semanticTags.slice(),
2354
+ state: afterSnapshot.targetState,
2355
+ status
2356
+ };
2357
+ }
2358
+
2359
+ function buildActionResult(entry, extra = {}) {
2360
+ return {
2361
+ captureId: state.captureId,
2362
+ descriptorTags: Array.isArray(entry?.descriptorTags) ? entry.descriptorTags.slice() : [],
2363
+ referenceId: entry.referenceId,
2364
+ semanticTags: Array.isArray(entry?.semanticTags) ? entry.semanticTags.slice() : [],
2365
+ state: entry.state || collectElementStateMetadata(entry.element, state.captureOptions),
2366
+ summary: entry.summary,
2367
+ tagName: entry.tagName,
2368
+ ...extra
2369
+ };
2370
+ }
2371
+
2372
+ function buildHelperBackedActionResult(entry, helperResult, extra = {}) {
2373
+ return {
2374
+ captureId: state.captureId,
2375
+ descriptorTags: Array.isArray(helperResult?.descriptorTags) ? helperResult.descriptorTags : (entry.descriptorTags || []),
2376
+ frameChain: entry.frameChain.slice(),
2377
+ frameId: entry.frameId,
2378
+ nodeId: entry.nodeId,
2379
+ referenceId: entry.referenceId,
2380
+ semanticTags: Array.isArray(helperResult?.semanticTags) ? helperResult.semanticTags : (entry.semanticTags || []),
2381
+ state: helperResult?.state || entry.state || collectElementStateMetadata(null),
2382
+ summary: entry.summary,
2383
+ tagName: String(helperResult?.tagName || entry.tagName || ""),
2384
+ ...extra
2385
+ };
2386
+ }
2387
+
2388
+ function mergeActionOutcomeResults(...results) {
2389
+ const normalizedResults = results.filter(Boolean);
2390
+ const mergedStatus = {};
2391
+ const mergedEffect = {
2392
+ mutationAttributes: [],
2393
+ mutationCount: 0,
2394
+ newText: [],
2395
+ semanticHints: [],
2396
+ validationText: []
2397
+ };
2398
+
2399
+ normalizedResults.forEach((result) => {
2400
+ Object.entries(result?.status || {}).forEach(([key, value]) => {
2401
+ if (typeof value === "boolean") {
2402
+ mergedStatus[key] = mergedStatus[key] === true || value === true;
2403
+ }
2404
+ });
2405
+ if (Number.isFinite(result?.effect?.mutationCount)) {
2406
+ mergedEffect.mutationCount += Number(result.effect.mutationCount);
2407
+ }
2408
+ ["mutationAttributes", "newText", "semanticHints", "validationText"].forEach((key) => {
2409
+ const values = Array.isArray(result?.effect?.[key]) ? result.effect[key] : [];
2410
+ values.forEach((value) => {
2411
+ if (value && !mergedEffect[key].includes(value)) {
2412
+ mergedEffect[key].push(value);
2413
+ }
2414
+ });
2415
+ });
2416
+ });
2417
+
2418
+ mergedStatus.reacted = Object.entries(mergedStatus).some(([key, value]) => key !== "reacted" && key !== "noObservedEffect" && value === true);
2419
+ mergedStatus.noObservedEffect = !mergedStatus.reacted;
2420
+ return {
2421
+ effect: mergedEffect,
2422
+ status: mergedStatus
2423
+ };
2424
+ }
2425
+
2426
+ function dispatchDomEvent(target, eventName, EventType = "Event", options = {}) {
2427
+ const EventConstructor = typeof globalThis[EventType] === "function"
2428
+ ? globalThis[EventType]
2429
+ : globalThis.Event;
2430
+ const event = new EventConstructor(eventName, {
2431
+ bubbles: true,
2432
+ cancelable: true,
2433
+ composed: true,
2434
+ ...options
2435
+ });
2436
+ target.dispatchEvent(event);
2437
+ return event;
2438
+ }
2439
+
2440
+ function dispatchKeyboardEvent(target, eventName, options = {}) {
2441
+ const KeyboardEventConstructor = typeof globalThis.KeyboardEvent === "function"
2442
+ ? globalThis.KeyboardEvent
2443
+ : globalThis.Event;
2444
+ const event = new KeyboardEventConstructor(eventName, {
2445
+ bubbles: true,
2446
+ cancelable: true,
2447
+ composed: true,
2448
+ code: "Enter",
2449
+ key: "Enter",
2450
+ ...options
2451
+ });
2452
+
2453
+ [
2454
+ ["charCode", Number(options.charCode ?? 0)],
2455
+ ["keyCode", Number(options.keyCode ?? 13)],
2456
+ ["which", Number(options.which ?? 13)]
2457
+ ].forEach(([propertyName, propertyValue]) => {
2458
+ try {
2459
+ if (typeof event[propertyName] !== "number") {
2460
+ Object.defineProperty(event, propertyName, {
2461
+ configurable: true,
2462
+ enumerable: true,
2463
+ value: propertyValue
2464
+ });
2465
+ }
2466
+ } catch {
2467
+ // Ignore read-only KeyboardEvent properties.
2468
+ }
2469
+ });
2470
+
2471
+ target.dispatchEvent(event);
2472
+ return event;
2473
+ }
2474
+
2475
+ function setNativeValue(element, nextValue) {
2476
+ const tagName = getTagName(element);
2477
+ const normalizedValue = String(nextValue ?? "");
2478
+
2479
+ if (tagName === "INPUT") {
2480
+ const descriptor = Object.getOwnPropertyDescriptor(globalThis.HTMLInputElement?.prototype || {}, "value");
2481
+ if (typeof descriptor?.set === "function") {
2482
+ descriptor.set.call(element, normalizedValue);
2483
+ } else {
2484
+ element.value = normalizedValue;
2485
+ }
2486
+ return normalizedValue;
2487
+ }
2488
+
2489
+ if (tagName === "TEXTAREA") {
2490
+ const descriptor = Object.getOwnPropertyDescriptor(globalThis.HTMLTextAreaElement?.prototype || {}, "value");
2491
+ if (typeof descriptor?.set === "function") {
2492
+ descriptor.set.call(element, normalizedValue);
2493
+ } else {
2494
+ element.value = normalizedValue;
2495
+ }
2496
+ return normalizedValue;
2497
+ }
2498
+
2499
+ if (tagName === "SELECT") {
2500
+ const matchedOption = [...(element.options || [])].find((option) => {
2501
+ return option.value === normalizedValue
2502
+ || normalizeText(option.textContent || "") === normalizeText(normalizedValue)
2503
+ || normalizeText(option.label || "") === normalizeText(normalizedValue);
2504
+ });
2505
+
2506
+ const resolvedValue = matchedOption ? matchedOption.value : normalizedValue;
2507
+ const descriptor = Object.getOwnPropertyDescriptor(globalThis.HTMLSelectElement?.prototype || {}, "value");
2508
+ if (typeof descriptor?.set === "function") {
2509
+ descriptor.set.call(element, resolvedValue);
2510
+ } else {
2511
+ element.value = resolvedValue;
2512
+ }
2513
+ return resolvedValue;
2514
+ }
2515
+
2516
+ if (String(element.getAttribute?.("contenteditable") || "").toLowerCase() === "true") {
2517
+ element.textContent = normalizedValue;
2518
+ return normalizedValue;
2519
+ }
2520
+
2521
+ throw createNamedError(
2522
+ "BrowserPageContentActionError",
2523
+ `Browser page content cannot type into <${getTagName(element).toLowerCase()}>.`,
2524
+ {
2525
+ code: "browser_page_content_type_unsupported"
2526
+ }
2527
+ );
2528
+ }
2529
+
2530
+ async function updateElementValue(referenceId, value) {
2531
+ const entry = requireReferenceEntry(referenceId, {
2532
+ actionLabel: "type"
2533
+ });
2534
+
2535
+ if (entry.helperBacked) {
2536
+ const helper = requireDomHelper("type into reference");
2537
+ const typedResult = await helper.typeNode(entry.frameChain, entry.nodeId, value);
2538
+ return buildHelperBackedActionResult(entry, typedResult, {
2539
+ effect: typedResult?.effect || {},
2540
+ status: typedResult?.status || {},
2541
+ value: typedResult?.value ?? String(value ?? "")
2542
+ });
2543
+ }
2544
+
2545
+ const element = entry.element;
2546
+ const beforeSnapshot = captureActionEffectSnapshot(element);
2547
+
2548
+ const {
2549
+ result: appliedValue,
2550
+ observedMutations
2551
+ } = await withObservedActionWindow(beforeSnapshot.observationRoot, async () => {
2552
+ scrollElementIntoView(element);
2553
+ focusElement(element);
2554
+ const nextValue = setNativeValue(element, value);
2555
+
2556
+ if (typeof element.setSelectionRange === "function") {
2557
+ try {
2558
+ element.setSelectionRange(String(nextValue).length, String(nextValue).length);
2559
+ } catch {
2560
+ // Ignore selection errors for unsupported input types.
2561
+ }
2562
+ }
2563
+
2564
+ dispatchDomEvent(element, "beforeinput", "InputEvent", {
2565
+ data: String(value ?? ""),
2566
+ inputType: "insertText"
2567
+ });
2568
+ dispatchDomEvent(element, "input", "InputEvent", {
2569
+ data: String(value ?? ""),
2570
+ inputType: "insertText"
2571
+ });
2572
+ dispatchDomEvent(element, "change");
2573
+ return nextValue;
2574
+ });
2575
+
2576
+ refreshReferenceEntry(entry);
2577
+ return buildActionResult(entry, {
2578
+ ...buildActionEffectResult(entry, beforeSnapshot, captureActionEffectSnapshot(element), observedMutations),
2579
+ value: appliedValue
2580
+ });
2581
+ }
2582
+
2583
+ async function activateElement(referenceId) {
2584
+ const entry = requireReferenceEntry(referenceId, {
2585
+ actionLabel: "click"
2586
+ });
2587
+
2588
+ if (entry.helperBacked) {
2589
+ const helper = requireDomHelper("click reference");
2590
+ const clickedResult = await helper.clickNode(entry.frameChain, entry.nodeId);
2591
+ return buildHelperBackedActionResult(entry, clickedResult, {
2592
+ effect: clickedResult?.effect || {},
2593
+ status: clickedResult?.status || {}
2594
+ });
2595
+ }
2596
+
2597
+ const element = entry.element;
2598
+ const beforeSnapshot = captureActionEffectSnapshot(element);
2599
+
2600
+ scrollElementIntoView(element);
2601
+ focusElement(element);
2602
+
2603
+ if (beforeSnapshot.targetState.disabled) {
2604
+ throw createNamedError(
2605
+ "BrowserPageContentActionError",
2606
+ `Browser page content reference "${entry.referenceId}" is disabled.`,
2607
+ {
2608
+ code: "browser_page_content_click_disabled"
2609
+ }
2610
+ );
2611
+ }
2612
+
2613
+ const {
2614
+ observedMutations
2615
+ } = await withObservedActionWindow(beforeSnapshot.observationRoot, async () => {
2616
+ if (typeof element.click === "function") {
2617
+ element.click();
2618
+ } else {
2619
+ dispatchDomEvent(element, "click", "MouseEvent", {
2620
+ button: 0
2621
+ });
2622
+ }
2623
+ });
2624
+
2625
+ refreshReferenceEntry(entry);
2626
+ return buildActionResult(entry, buildActionEffectResult(
2627
+ entry,
2628
+ beforeSnapshot,
2629
+ captureActionEffectSnapshot(element),
2630
+ observedMutations
2631
+ ));
2632
+ }
2633
+
2634
+ async function submitElement(referenceId) {
2635
+ const entry = requireReferenceEntry(referenceId, {
2636
+ actionLabel: "submit"
2637
+ });
2638
+
2639
+ if (entry.helperBacked) {
2640
+ const helper = requireDomHelper("submit reference");
2641
+ const submittedResult = await helper.submitNode(entry.frameChain, entry.nodeId);
2642
+ return buildHelperBackedActionResult(entry, submittedResult, {
2643
+ effect: submittedResult?.effect || {},
2644
+ status: submittedResult?.status || {}
2645
+ });
2646
+ }
2647
+
2648
+ const element = entry.element;
2649
+ const tagName = getTagName(element);
2650
+ const beforeSnapshot = captureActionEffectSnapshot(element);
2651
+
2652
+ const {
2653
+ observedMutations
2654
+ } = await withObservedActionWindow(beforeSnapshot.observationRoot, async () => {
2655
+ scrollElementIntoView(element);
2656
+ focusElement(element);
2657
+
2658
+ if (tagName === "FORM") {
2659
+ if (typeof element.requestSubmit === "function") {
2660
+ element.requestSubmit();
2661
+ } else {
2662
+ const submitEvent = dispatchDomEvent(element, "submit");
2663
+ if (!submitEvent.defaultPrevented) {
2664
+ element.submit?.();
2665
+ }
2666
+ }
2667
+ } else if (typeof element.form?.requestSubmit === "function") {
2668
+ if (tagName === "BUTTON" || tagName === "INPUT") {
2669
+ element.form.requestSubmit(element);
2670
+ } else {
2671
+ element.form.requestSubmit();
2672
+ }
2673
+ } else if (element.form) {
2674
+ const submitEvent = dispatchDomEvent(element.form, "submit");
2675
+ if (!submitEvent.defaultPrevented) {
2676
+ element.form.submit?.();
2677
+ }
2678
+ } else if (typeof element.click === "function") {
2679
+ element.click();
2680
+ } else {
2681
+ throw createNamedError(
2682
+ "BrowserPageContentActionError",
2683
+ `Browser page content cannot submit reference "${entry.referenceId}".`,
2684
+ {
2685
+ code: "browser_page_content_submit_unsupported"
2686
+ }
2687
+ );
2688
+ }
2689
+ });
2690
+
2691
+ refreshReferenceEntry(entry);
2692
+ return buildActionResult(entry, buildActionEffectResult(
2693
+ entry,
2694
+ beforeSnapshot,
2695
+ captureActionEffectSnapshot(element),
2696
+ observedMutations
2697
+ ));
2698
+ }
2699
+
2700
+ function shouldEnterSubmitForm(element) {
2701
+ const tagName = getTagName(element);
2702
+ if (tagName !== "INPUT") {
2703
+ return false;
2704
+ }
2705
+
2706
+ const inputType = String(element.getAttribute?.("type") || element.type || "text").toLowerCase();
2707
+ return ![
2708
+ "button",
2709
+ "checkbox",
2710
+ "color",
2711
+ "file",
2712
+ "hidden",
2713
+ "image",
2714
+ "radio",
2715
+ "range",
2716
+ "reset",
2717
+ "submit"
2718
+ ].includes(inputType);
2719
+ }
2720
+
2721
+ async function pressEnterElement(referenceId, actionLabel = "type_submit") {
2722
+ const entry = requireReferenceEntry(referenceId, {
2723
+ actionLabel
2724
+ });
2725
+
2726
+ if (entry.helperBacked) {
2727
+ const helper = requireDomHelper("press enter on reference");
2728
+ const submittedResult = await helper.typeSubmitNode(entry.frameChain, entry.nodeId, "");
2729
+ return buildHelperBackedActionResult(entry, submittedResult, {
2730
+ effect: submittedResult?.effect || {},
2731
+ status: submittedResult?.status || {}
2732
+ });
2733
+ }
2734
+
2735
+ const element = entry.element;
2736
+ const beforeSnapshot = captureActionEffectSnapshot(element);
2737
+
2738
+ const {
2739
+ observedMutations
2740
+ } = await withObservedActionWindow(beforeSnapshot.observationRoot, async () => {
2741
+ scrollElementIntoView(element);
2742
+ focusElement(element);
2743
+
2744
+ const keydownEvent = dispatchKeyboardEvent(element, "keydown", {
2745
+ charCode: 0,
2746
+ keyCode: 13,
2747
+ which: 13
2748
+ });
2749
+ const keypressEvent = dispatchKeyboardEvent(element, "keypress", {
2750
+ charCode: 13,
2751
+ keyCode: 13,
2752
+ which: 13
2753
+ });
2754
+ const keyupEvent = dispatchKeyboardEvent(element, "keyup", {
2755
+ charCode: 0,
2756
+ keyCode: 13,
2757
+ which: 13
2758
+ });
2759
+
2760
+ if (
2761
+ !keydownEvent.defaultPrevented
2762
+ && !keypressEvent.defaultPrevented
2763
+ && !keyupEvent.defaultPrevented
2764
+ && shouldEnterSubmitForm(element)
2765
+ ) {
2766
+ if (typeof element.form?.requestSubmit === "function") {
2767
+ element.form.requestSubmit();
2768
+ } else if (element.form) {
2769
+ const submitEvent = dispatchDomEvent(element.form, "submit");
2770
+ if (!submitEvent.defaultPrevented) {
2771
+ element.form.submit?.();
2772
+ }
2773
+ }
2774
+ }
2775
+ });
2776
+
2777
+ refreshReferenceEntry(entry);
2778
+ return buildActionResult(entry, buildActionEffectResult(
2779
+ entry,
2780
+ beforeSnapshot,
2781
+ captureActionEffectSnapshot(element),
2782
+ observedMutations
2783
+ ));
2784
+ }
2785
+
2786
+ async function typeAndSubmit(referenceId, value) {
2787
+ const entry = requireReferenceEntry(referenceId, {
2788
+ actionLabel: "type_submit"
2789
+ });
2790
+
2791
+ if (entry.helperBacked) {
2792
+ const helper = requireDomHelper("type and submit reference");
2793
+ const submittedResult = await helper.typeSubmitNode(entry.frameChain, entry.nodeId, value);
2794
+ return buildHelperBackedActionResult(entry, submittedResult, {
2795
+ effect: submittedResult?.effect || {},
2796
+ status: submittedResult?.status || {},
2797
+ value: submittedResult?.value ?? String(value ?? "")
2798
+ });
2799
+ }
2800
+
2801
+ const typed = await updateElementValue(referenceId, value);
2802
+ const submitted = await pressEnterElement(referenceId);
2803
+ const mergedOutcome = mergeActionOutcomeResults(typed, submitted);
2804
+
2805
+ return {
2806
+ ...submitted,
2807
+ ...mergedOutcome,
2808
+ value: typed.value
2809
+ };
2810
+ }
2811
+
2812
+ async function scrollToReference(referenceId) {
2813
+ const entry = requireReferenceEntry(referenceId, {
2814
+ actionLabel: "scroll"
2815
+ });
2816
+
2817
+ if (entry.helperBacked) {
2818
+ const helper = requireDomHelper("scroll to reference");
2819
+ const scrollResult = await helper.scrollNode(entry.frameChain, entry.nodeId);
2820
+ return buildHelperBackedActionResult(entry, scrollResult, {
2821
+ effect: scrollResult?.effect || {},
2822
+ status: scrollResult?.status || {}
2823
+ });
2824
+ }
2825
+
2826
+ const beforeSnapshot = captureActionEffectSnapshot(entry.element);
2827
+ scrollElementIntoView(entry.element);
2828
+ focusElement(entry.element);
2829
+ refreshReferenceEntry(entry);
2830
+ const afterSnapshot = captureActionEffectSnapshot(entry.element);
2831
+ const scrollEffect = buildActionEffectResult(entry, beforeSnapshot, afterSnapshot, {
2832
+ attributeNames: [],
2833
+ mutationCount: 0
2834
+ });
2835
+ return buildActionResult(entry, {
2836
+ ...scrollEffect,
2837
+ status: {
2838
+ ...scrollEffect.status,
2839
+ reacted: true,
2840
+ noObservedEffect: false
2841
+ }
2842
+ });
2843
+ }
2844
+
2845
+ globalThis[GLOBAL_KEY] = {
2846
+ click(referenceId) {
2847
+ return activateElement(referenceId);
2848
+ },
2849
+ capture,
2850
+ clear() {
2851
+ state.captureId = 0;
2852
+ state.capturedAt = 0;
2853
+ state.captureOptions = {
2854
+ includeLabelQuotes: false,
2855
+ includeLinkUrls: false,
2856
+ includeSemanticTags: true,
2857
+ includeStateTags: true,
2858
+ includeListIndentation: true,
2859
+ includeListMarkers: false
2860
+ };
2861
+ state.entries = new Map();
2862
+ },
2863
+ detail,
2864
+ getState() {
2865
+ return {
2866
+ captureId: state.captureId,
2867
+ capturedAt: state.capturedAt,
2868
+ includeLabelQuotes: state.captureOptions.includeLabelQuotes === true,
2869
+ includeLinkUrls: state.captureOptions.includeLinkUrls === true,
2870
+ includeSemanticTags: state.captureOptions.includeSemanticTags !== false,
2871
+ includeStateTags: state.captureOptions.includeStateTags !== false,
2872
+ includeListIndentation: state.captureOptions.includeListIndentation !== false,
2873
+ includeListMarkers: state.captureOptions.includeListMarkers === true,
2874
+ referenceCount: state.entries.size
2875
+ };
2876
+ },
2877
+ scroll(referenceId) {
2878
+ return scrollToReference(referenceId);
2879
+ },
2880
+ submit(referenceId) {
2881
+ return submitElement(referenceId);
2882
+ },
2883
+ type(referenceId, value) {
2884
+ return updateElementValue(referenceId, value);
2885
+ },
2886
+ typeSubmit(referenceId, value) {
2887
+ return typeAndSubmit(referenceId, value);
2888
+ },
2889
+ version: VERSION
2890
+ };
2891
+})();
plugins/_browser/default_config.yaml
new
+10
@@ -0,0 +1,10 @@
1
+# Load unpacked Chromium extension directories into the Browser tool.
2
+# Paths must be readable from the Agent Zero runtime itself.
3
+extensions_enabled: false
4
+
5
+# One unpacked extension directory per item.
6
+extension_paths: []
7
+
8
+# Optional _model_config preset used by Browser-owned model helpers.
9
+# Empty uses the effective Main Model.
10
+model_preset: ""
plugins/_browser/extensions/python/_functions/agent/AgentContext/remove/start/_10_cleanup_browser_runtime.py
new
+10
@@ -0,0 +1,10 @@
1
+from helpers.extension import Extension
2
+from plugins._browser.helpers.runtime import close_runtime_sync
3
+
4
+
5
+class CleanupBrowserRuntimeOnRemove(Extension):
6
+ def execute(self, data: dict = {}, **kwargs):
7
+ args = data.get("args", ())
8
+ context_id = args[0] if isinstance(args, tuple) and args else ""
9
+ if context_id:
10
+ close_runtime_sync(str(context_id), delete_profile=True)
plugins/_browser/extensions/python/_functions/agent/AgentContext/reset/start/_10_cleanup_browser_runtime.py
new
+11
@@ -0,0 +1,11 @@
1
+from helpers.extension import Extension
2
+from plugins._browser.helpers.runtime import close_runtime_sync
3
+
4
+
5
+class CleanupBrowserRuntimeOnReset(Extension):
6
+ def execute(self, data: dict = {}, **kwargs):
7
+ args = data.get("args", ())
8
+ context = args[0] if isinstance(args, tuple) and args else None
9
+ context_id = getattr(context, "id", "")
10
+ if context_id:
11
+ close_runtime_sync(context_id, delete_profile=True)
plugins/_browser/extensions/python/system_prompt/_20_browser_context.py
new
+59
@@ -0,0 +1,59 @@
1
+from __future__ import annotations
2
+
3
+from typing import Any
4
+
5
+from agent import LoopData
6
+from helpers.extension import Extension
7
+from plugins._browser.helpers.runtime import get_runtime
8
+
9
+
10
+class BrowserContextPrompt(Extension):
11
+ async def execute(
12
+ self,
13
+ system_prompt: list[str] = [],
14
+ loop_data: LoopData = LoopData(),
15
+ **kwargs: Any,
16
+ ):
17
+ if not self.agent:
18
+ return
19
+
20
+ runtime = await get_runtime(self.agent.context.id, create=False)
21
+ if not runtime:
22
+ return
23
+
24
+ try:
25
+ listing = await runtime.call("list")
26
+ except Exception:
27
+ return
28
+
29
+ browsers = listing.get("browsers") or []
30
+ if not browsers:
31
+ return
32
+
33
+ rows = ["browser id|url|title"]
34
+ for browser in browsers:
35
+ rows.append(
36
+ f"{browser.get('id')}|{browser.get('currentUrl', '')}|{browser.get('title', '')}"
37
+ )
38
+
39
+ section = ["currently open web browsers", "\n".join(rows)]
40
+ last_id = listing.get("last_interacted_browser_id")
41
+ if last_id:
42
+ try:
43
+ state = await runtime.call("state", last_id)
44
+ content = await runtime.call("content", last_id, None)
45
+ document = content.get("document") if isinstance(content, dict) else ""
46
+ if document:
47
+ section.extend(
48
+ [
49
+ "",
50
+ "last interacted web browser",
51
+ f"browser id|url|title\n{state.get('id')}|{state.get('currentUrl', '')}|{state.get('title', '')}",
52
+ "page content↓",
53
+ str(document),
54
+ ]
55
+ )
56
+ except Exception:
57
+ pass
58
+
59
+ system_prompt.append("\n".join(section))
plugins/_browser/extensions/python/webui_ws_disconnect/_50_browser.py
new
+24
@@ -0,0 +1,24 @@
1
+from __future__ import annotations
2
+
3
+from typing import Any
4
+
5
+from helpers.extension import Extension
6
+from plugins._browser.api.ws_browser import WsBrowser
7
+
8
+
9
+class BrowserWebuiWsDisconnect(Extension):
10
+ async def execute(
11
+ self,
12
+ instance: Any = None,
13
+ sid: str = "",
14
+ **kwargs: Any,
15
+ ) -> None:
16
+ if instance is None:
17
+ return
18
+ handler = WsBrowser(
19
+ instance.socketio,
20
+ instance.lock,
21
+ manager=instance.manager,
22
+ namespace=instance.namespace,
23
+ )
24
+ await handler.on_disconnect(sid)
plugins/_browser/extensions/python/webui_ws_event/_50_browser.py
new
+47
@@ -0,0 +1,47 @@
1
+from __future__ import annotations
2
+
3
+from typing import Any
4
+
5
+from helpers.extension import Extension
6
+from helpers.ws_manager import WsResult
7
+from plugins._browser.api.ws_browser import WsBrowser
8
+
9
+
10
+class BrowserWebuiWsEvents(Extension):
11
+ async def execute(
12
+ self,
13
+ instance: Any = None,
14
+ sid: str = "",
15
+ event_type: str = "",
16
+ data: dict[str, Any] | None = None,
17
+ response_data: dict[str, Any] | None = None,
18
+ **kwargs: Any,
19
+ ) -> None:
20
+ if not event_type.startswith("browser_") or instance is None or response_data is None:
21
+ return
22
+
23
+ handler = WsBrowser(
24
+ instance.socketio,
25
+ instance.lock,
26
+ manager=instance.manager,
27
+ namespace=instance.namespace,
28
+ )
29
+ result = await handler.process(event_type, data or {}, sid)
30
+ if result is None:
31
+ return
32
+
33
+ if isinstance(result, WsResult):
34
+ payload = result.as_result(
35
+ handler_id=handler.identifier,
36
+ fallback_correlation_id=(data or {}).get("correlationId"),
37
+ )
38
+ if payload.get("ok"):
39
+ response_data.update(payload.get("data") or {})
40
+ else:
41
+ response_data["browser_error"] = payload.get("error") or {
42
+ "code": "BROWSER_ERROR",
43
+ "error": "Browser request failed",
44
+ }
45
+ return
46
+
47
+ response_data.update(result)
plugins/_browser/extensions/webui/chat-input-bottom-actions-start/browser-button.html
new
+17
@@ -0,0 +1,17 @@
1
+<button
2
+ type="button"
3
+ class="text-button browser-chat-action"
4
+ title="Show or hide Browser"
5
+ aria-label="Show or hide Browser"
6
+ data-bs-placement="top"
7
+ data-bs-trigger="hover"
8
+ @click="window.toggleModal ? window.toggleModal('/plugins/_browser/webui/main.html') : window.openModal('/plugins/_browser/webui/main.html')"
9
+>
10
+ <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" width="14" height="14" aria-hidden="true">
11
+ <rect x="3" y="4" width="18" height="16" rx="2"></rect>
12
+ <path d="M3 8h18"></path>
13
+ <path d="M7 6h.01"></path>
14
+ <path d="M10 6h.01"></path>
15
+ </svg>
16
+ <p>Browser</p>
17
+</button>
plugins/_browser/extensions/webui/get_tool_message_handler/browser-tool-handler.js
new
+76
@@ -0,0 +1,76 @@
1
+import {
2
+ createActionButton,
3
+ copyToClipboard,
4
+} from "/components/messages/action-buttons/simple-action-buttons.js";
5
+import { store as stepDetailStore } from "/components/modals/process-step-detail/step-detail-store.js";
6
+import { store as speechStore } from "/components/chat/speech/speech-store.js";
7
+import {
8
+ buildDetailPayload,
9
+ cleanStepTitle,
10
+ drawProcessStep,
11
+} from "/js/messages.js";
12
+
13
+const BROWSER_MODAL = "/plugins/_browser/webui/main.html";
14
+
15
+export default async function registerBrowserToolHandler(extData) {
16
+ if (extData?.tool_name === "browser") {
17
+ extData.handler = drawBrowserTool;
18
+ }
19
+}
20
+
21
+function drawBrowserTool({
22
+ id,
23
+ type,
24
+ heading,
25
+ content,
26
+ kvps,
27
+ timestamp,
28
+ agentno = 0,
29
+ ...additional
30
+}) {
31
+ const title = cleanStepTitle(heading);
32
+ const displayKvps = { ...kvps };
33
+ const headerLabels = [
34
+ kvps?._tool_name && { label: kvps._tool_name, class: "tool-name-badge" },
35
+ ].filter(Boolean);
36
+ const contentText = String(content ?? "");
37
+ const browserButton = createActionButton(
38
+ "visibility",
39
+ "Browser",
40
+ () => {
41
+ if (window.ensureModalOpen) {
42
+ void window.ensureModalOpen(BROWSER_MODAL);
43
+ return;
44
+ }
45
+ void window.openModal?.(BROWSER_MODAL);
46
+ },
47
+ );
48
+ browserButton.setAttribute("title", "Open Browser");
49
+ browserButton.setAttribute("aria-label", "Open Browser");
50
+ browserButton.setAttribute("data-bs-placement", "top");
51
+ browserButton.setAttribute("data-bs-trigger", "hover");
52
+ const actionButtons = [browserButton];
53
+
54
+ if (contentText.trim()) {
55
+ actionButtons.push(
56
+ createActionButton("detail", "", () =>
57
+ stepDetailStore.showStepDetail(
58
+ buildDetailPayload(arguments[0], { headerLabels }),
59
+ ),
60
+ ),
61
+ createActionButton("speak", "", () => speechStore.speak(contentText)),
62
+ createActionButton("copy", "", () => copyToClipboard(contentText)),
63
+ );
64
+ }
65
+
66
+ return drawProcessStep({
67
+ id,
68
+ title,
69
+ code: "WWW",
70
+ classes: undefined,
71
+ kvps: displayKvps,
72
+ content,
73
+ actionButtons: actionButtons.filter(Boolean),
74
+ log: arguments[0],
75
+ });
76
+}
plugins/_browser/helpers/__init__.py
new
+1
@@ -0,0 +1 @@
1
+# Built-in direct browser helpers.
plugins/_browser/helpers/config.py
new
+272
@@ -0,0 +1,272 @@
1
+from __future__ import annotations
2
+
3
+from pathlib import Path
4
+from typing import TYPE_CHECKING, Any
5
+
6
+if TYPE_CHECKING:
7
+ from agent import Agent
8
+
9
+
10
+PLUGIN_NAME = "_browser"
11
+MODEL_PRESET_KEY = "model_preset"
12
+BASE_BROWSER_ARGS = [
13
+ "--no-sandbox",
14
+ "--disable-dev-shm-usage",
15
+ "--disable-gpu",
16
+]
17
+
18
+
19
+def _normalize_extension_paths(value: Any) -> list[str]:
20
+ if isinstance(value, str):
21
+ candidates = value.replace("\r\n", "\n").replace("\r", "\n").split("\n")
22
+ elif isinstance(value, (list, tuple, set)):
23
+ candidates = list(value)
24
+ else:
25
+ candidates = []
26
+
27
+ normalized_paths: list[str] = []
28
+ seen: set[str] = set()
29
+ for entry in candidates:
30
+ raw_path = str(entry or "").strip()
31
+ if not raw_path:
32
+ continue
33
+ normalized = str(Path(raw_path).expanduser())
34
+ if normalized in seen:
35
+ continue
36
+ seen.add(normalized)
37
+ normalized_paths.append(normalized)
38
+ return normalized_paths
39
+
40
+
41
+def _normalize_model_preset(value: Any) -> str:
42
+ return str(value or "").strip()
43
+
44
+
45
+def normalize_browser_config(settings: dict[str, Any] | None) -> dict[str, Any]:
46
+ raw = settings if isinstance(settings, dict) else {}
47
+ return {
48
+ "extensions_enabled": bool(raw.get("extensions_enabled", False)),
49
+ "extension_paths": _normalize_extension_paths(raw.get("extension_paths", [])),
50
+ MODEL_PRESET_KEY: _normalize_model_preset(raw.get(MODEL_PRESET_KEY, "")),
51
+ }
52
+
53
+
54
+def browser_runtime_config(settings: dict[str, Any] | None) -> dict[str, Any]:
55
+ config = normalize_browser_config(settings)
56
+ return {
57
+ "extensions_enabled": config["extensions_enabled"],
58
+ "extension_paths": config["extension_paths"],
59
+ }
60
+
61
+
62
+def get_browser_config(agent: "Agent | None" = None) -> dict[str, Any]:
63
+ from helpers import plugins
64
+
65
+ return normalize_browser_config(plugins.get_plugin_config(PLUGIN_NAME, agent=agent) or {})
66
+
67
+
68
+def get_browser_model_preset_name(
69
+ agent: "Agent | None" = None,
70
+ settings: dict[str, Any] | None = None,
71
+) -> str:
72
+ config = (
73
+ normalize_browser_config(settings)
74
+ if settings is not None
75
+ else get_browser_config(agent=agent)
76
+ )
77
+ return str(config.get(MODEL_PRESET_KEY, "") or "").strip()
78
+
79
+
80
+def get_browser_model_preset_options(
81
+ agent: "Agent | None" = None,
82
+ settings: dict[str, Any] | None = None,
83
+) -> list[dict[str, Any]]:
84
+ from plugins._model_config.helpers import model_config
85
+
86
+ selected_name = get_browser_model_preset_name(agent=agent, settings=settings)
87
+ options: list[dict[str, Any]] = []
88
+ found_selected = False
89
+
90
+ for preset in model_config.get_presets():
91
+ name = str(preset.get("name", "") or "").strip()
92
+ if not name:
93
+ continue
94
+ if name == selected_name:
95
+ found_selected = True
96
+ chat_cfg = preset.get("chat", {}) if isinstance(preset, dict) else {}
97
+ if not isinstance(chat_cfg, dict):
98
+ chat_cfg = {}
99
+ provider = str(chat_cfg.get("provider", "") or "").strip()
100
+ model_name = str(chat_cfg.get("name", "") or "").strip()
101
+ summary = " / ".join(part for part in (provider, model_name) if part)
102
+ options.append(
103
+ {
104
+ "name": name,
105
+ "label": name,
106
+ "missing": False,
107
+ "summary": summary,
108
+ }
109
+ )
110
+
111
+ if selected_name and not found_selected:
112
+ options.append(
113
+ {
114
+ "name": selected_name,
115
+ "label": f"{selected_name} (missing)",
116
+ "missing": True,
117
+ "summary": "",
118
+ }
119
+ )
120
+
121
+ return options
122
+
123
+
124
+def resolve_browser_model_selection(
125
+ agent: "Agent | None" = None,
126
+ settings: dict[str, Any] | None = None,
127
+) -> dict[str, Any]:
128
+ from plugins._model_config.helpers import model_config
129
+
130
+ preset_name = get_browser_model_preset_name(agent=agent, settings=settings)
131
+ if preset_name:
132
+ preset = model_config.get_preset_by_name(preset_name)
133
+ if isinstance(preset, dict):
134
+ chat_cfg = preset.get("chat", {})
135
+ if isinstance(chat_cfg, dict) and (
136
+ str(chat_cfg.get("provider", "") or "").strip()
137
+ or str(chat_cfg.get("name", "") or "").strip()
138
+ ):
139
+ return {
140
+ "config": chat_cfg,
141
+ "source_kind": "preset",
142
+ "source_label": f"Preset '{preset_name}' via _model_config",
143
+ "selected_preset_name": preset_name,
144
+ "preset_status": "active",
145
+ "warning": "",
146
+ }
147
+ return {
148
+ "config": model_config.get_chat_model_config(agent),
149
+ "source_kind": "main",
150
+ "source_label": "Main Model via _model_config",
151
+ "selected_preset_name": preset_name,
152
+ "preset_status": "invalid",
153
+ "warning": (
154
+ f"Configured browser preset '{preset_name}' does not define a chat model. "
155
+ "Falling back to the Main Model."
156
+ ),
157
+ }
158
+
159
+ return {
160
+ "config": model_config.get_chat_model_config(agent),
161
+ "source_kind": "main",
162
+ "source_label": "Main Model via _model_config",
163
+ "selected_preset_name": preset_name,
164
+ "preset_status": "missing",
165
+ "warning": (
166
+ f"Configured browser preset '{preset_name}' was not found. "
167
+ "Falling back to the Main Model."
168
+ ),
169
+ }
170
+
171
+ return {
172
+ "config": model_config.get_chat_model_config(agent),
173
+ "source_kind": "main",
174
+ "source_label": "Main Model via _model_config",
175
+ "selected_preset_name": "",
176
+ "preset_status": "none",
177
+ "warning": "",
178
+ }
179
+
180
+
181
+def resolve_browser_model(agent: "Agent", settings: dict[str, Any] | None = None):
182
+ selection = resolve_browser_model_selection(agent=agent, settings=settings)
183
+ if selection["source_kind"] == "main":
184
+ return agent.get_chat_model()
185
+
186
+ import models
187
+ from plugins._model_config.helpers import model_config
188
+
189
+ model_config_object = model_config.build_model_config(
190
+ selection["config"],
191
+ models.ModelType.CHAT,
192
+ )
193
+ return models.get_chat_model(
194
+ model_config_object.provider,
195
+ model_config_object.name,
196
+ model_config=model_config_object,
197
+ **model_config_object.build_kwargs(),
198
+ )
199
+
200
+
201
+def describe_browser_extensions(settings: dict[str, Any] | None) -> dict[str, Any]:
202
+ config = normalize_browser_config(settings)
203
+ path_details: list[dict[str, Any]] = []
204
+ for extension_path in config["extension_paths"]:
205
+ path = Path(extension_path)
206
+ exists = path.exists()
207
+ is_dir = path.is_dir() if exists else False
208
+ path_details.append(
209
+ {
210
+ "path": extension_path,
211
+ "exists": exists,
212
+ "is_dir": is_dir,
213
+ "loadable": exists and is_dir,
214
+ }
215
+ )
216
+
217
+ active_paths = [item["path"] for item in path_details if item["loadable"]]
218
+ invalid_paths = [item["path"] for item in path_details if not item["loadable"]]
219
+ active = bool(config["extensions_enabled"] and active_paths)
220
+
221
+ warnings: list[str] = []
222
+ if config["extensions_enabled"] and not config["extension_paths"]:
223
+ warnings.append(
224
+ "Extensions are enabled, but no unpacked extension directories are configured."
225
+ )
226
+ elif config["extensions_enabled"] and not active_paths:
227
+ warnings.append(
228
+ "Extensions are enabled, but none of the configured extension directories are readable unpacked folders."
229
+ )
230
+ elif invalid_paths:
231
+ warnings.append(
232
+ "Some configured extension directories are missing or not directories, so they will be skipped."
233
+ )
234
+
235
+ return {
236
+ "enabled": bool(config["extensions_enabled"]),
237
+ "active": active,
238
+ "configured_paths": config["extension_paths"],
239
+ "active_paths": active_paths,
240
+ "invalid_paths": invalid_paths,
241
+ "path_details": path_details,
242
+ "active_path_count": len(active_paths),
243
+ "warnings": warnings,
244
+ }
245
+
246
+
247
+def build_browser_launch_config(settings: dict[str, Any] | None) -> dict[str, Any]:
248
+ extensions = describe_browser_extensions(settings)
249
+ args = list(BASE_BROWSER_ARGS)
250
+ channel: str | None = None
251
+ browser_mode = "headless_shell"
252
+
253
+ if extensions["active"]:
254
+ joined_paths = ",".join(extensions["active_paths"])
255
+ args.extend(
256
+ [
257
+ f"--disable-extensions-except={joined_paths}",
258
+ f"--load-extension={joined_paths}",
259
+ ]
260
+ )
261
+ channel = "chromium"
262
+ browser_mode = "chromium_extensions"
263
+ else:
264
+ args.insert(0, "--headless=new")
265
+
266
+ return {
267
+ "args": args,
268
+ "browser_mode": browser_mode,
269
+ "channel": channel,
270
+ "extensions": extensions,
271
+ "requires_full_browser": bool(extensions["active"]),
272
+ }
plugins/_browser/helpers/extension_manager.py
new
+177
@@ -0,0 +1,177 @@
1
+from __future__ import annotations
2
+
3
+import json
4
+import re
5
+import shutil
6
+import tempfile
7
+import urllib.request
8
+import zipfile
9
+from pathlib import Path
10
+from typing import Any
11
+
12
+from helpers import files, plugins
13
+from plugins._browser.helpers.config import PLUGIN_NAME, get_browser_config
14
+
15
+
16
+EXTENSION_ID_RE = re.compile(r"^[a-p]{32}$")
17
+WEB_STORE_ID_RE = re.compile(r"(?<![a-p])([a-p]{32})(?![a-p])")
18
+WEB_STORE_DOWNLOAD_URL = (
19
+ "https://clients2.google.com/service/update2/crx"
20
+ "?response=redirect"
21
+ "&prodversion=120.0.0.0"
22
+ "&acceptformat=crx2,crx3"
23
+ "&x=id%3D{extension_id}%26installsource%3Dondemand%26uc"
24
+)
25
+
26
+
27
+def get_extensions_root() -> Path:
28
+ root = Path(files.get_abs_path("usr/browser-extensions"))
29
+ root.mkdir(parents=True, exist_ok=True)
30
+ return root
31
+
32
+
33
+def parse_chrome_web_store_extension_id(value: str) -> str:
34
+ source = str(value or "").strip()
35
+ if EXTENSION_ID_RE.fullmatch(source):
36
+ return source
37
+
38
+ match = WEB_STORE_ID_RE.search(source)
39
+ if match:
40
+ return match.group(1)
41
+
42
+ raise ValueError("Enter a Chrome Web Store URL or a 32-character extension id.")
43
+
44
+
45
+def list_browser_extensions() -> list[dict[str, Any]]:
46
+ root = get_extensions_root()
47
+ config = get_browser_config()
48
+ enabled_paths = {str(Path(path).expanduser()) for path in config["extension_paths"]}
49
+ entries: list[dict[str, Any]] = []
50
+
51
+ for manifest_path in sorted(root.glob("**/manifest.json")):
52
+ extension_dir = manifest_path.parent
53
+ try:
54
+ manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
55
+ except Exception:
56
+ manifest = {}
57
+ extension_path = str(extension_dir)
58
+ entries.append(
59
+ {
60
+ "name": manifest.get("name") or extension_dir.name,
61
+ "version": manifest.get("version") or "",
62
+ "path": extension_path,
63
+ "enabled": extension_path in enabled_paths,
64
+ }
65
+ )
66
+
67
+ return entries
68
+
69
+
70
+def install_chrome_web_store_extension(source: str) -> dict[str, Any]:
71
+ extension_id = parse_chrome_web_store_extension_id(source)
72
+ target = get_extensions_root() / "chrome-web-store" / extension_id
73
+
74
+ with tempfile.TemporaryDirectory(prefix="a0-browser-ext-") as tmp:
75
+ archive_path = Path(tmp) / f"{extension_id}.crx"
76
+ _download_crx(extension_id, archive_path)
77
+ payload_path = Path(tmp) / f"{extension_id}.zip"
78
+ payload_path.write_bytes(_crx_zip_payload(archive_path.read_bytes()))
79
+ extracted_path = Path(tmp) / "extracted"
80
+ _safe_extract_zip(payload_path, extracted_path)
81
+
82
+ if not (extracted_path / "manifest.json").is_file():
83
+ raise ValueError("Downloaded extension did not contain a manifest.json file.")
84
+
85
+ if target.exists():
86
+ shutil.rmtree(target)
87
+ target.parent.mkdir(parents=True, exist_ok=True)
88
+ shutil.copytree(extracted_path, target)
89
+
90
+ config = _enable_extension_path(target)
91
+ manifest = _read_manifest(target)
92
+ return {
93
+ "ok": True,
94
+ "id": extension_id,
95
+ "name": manifest.get("name") or extension_id,
96
+ "version": manifest.get("version") or "",
97
+ "path": str(target),
98
+ "extensions_enabled": config["extensions_enabled"],
99
+ "extension_paths": config["extension_paths"],
100
+ }
101
+
102
+
103
+def _download_crx(extension_id: str, archive_path: Path) -> None:
104
+ url = WEB_STORE_DOWNLOAD_URL.format(extension_id=extension_id)
105
+ request = urllib.request.Request(
106
+ url,
107
+ headers={
108
+ "User-Agent": (
109
+ "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
110
+ "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
111
+ )
112
+ },
113
+ )
114
+ with urllib.request.urlopen(request, timeout=30) as response:
115
+ data = response.read()
116
+ if not data:
117
+ raise ValueError("Chrome Web Store returned an empty extension package.")
118
+ archive_path.write_bytes(data)
119
+
120
+
121
+def _crx_zip_payload(data: bytes) -> bytes:
122
+ if data.startswith(b"PK"):
123
+ return data
124
+ if data[:4] != b"Cr24":
125
+ raise ValueError("Downloaded package is not a CRX or ZIP archive.")
126
+
127
+ version = int.from_bytes(data[4:8], "little")
128
+ if version == 2:
129
+ public_key_len = int.from_bytes(data[8:12], "little")
130
+ signature_len = int.from_bytes(data[12:16], "little")
131
+ offset = 16 + public_key_len + signature_len
132
+ elif version == 3:
133
+ header_len = int.from_bytes(data[8:12], "little")
134
+ offset = 12 + header_len
135
+ else:
136
+ raise ValueError(f"Unsupported CRX version: {version}.")
137
+
138
+ payload = data[offset:]
139
+ if not payload.startswith(b"PK"):
140
+ raise ValueError("CRX payload did not contain a ZIP archive.")
141
+ return payload
142
+
143
+
144
+def _safe_extract_zip(archive_path: Path, target_dir: Path) -> None:
145
+ target_dir.mkdir(parents=True, exist_ok=True)
146
+ root = target_dir.resolve()
147
+ with zipfile.ZipFile(archive_path) as archive:
148
+ for member in archive.infolist():
149
+ destination = (target_dir / member.filename).resolve()
150
+ if not destination.is_relative_to(root):
151
+ raise ValueError("Extension archive contains an unsafe path.")
152
+ if member.is_dir():
153
+ destination.mkdir(parents=True, exist_ok=True)
154
+ continue
155
+ destination.parent.mkdir(parents=True, exist_ok=True)
156
+ with archive.open(member) as source, destination.open("wb") as output:
157
+ shutil.copyfileobj(source, output)
158
+
159
+
160
+def _enable_extension_path(extension_path: Path) -> dict[str, Any]:
161
+ config = get_browser_config()
162
+ path = str(extension_path)
163
+ paths = list(config["extension_paths"])
164
+ if path not in paths:
165
+ paths.append(path)
166
+ config["extensions_enabled"] = True
167
+ config["extension_paths"] = paths
168
+ plugins.save_plugin_config(PLUGIN_NAME, "", "", config)
169
+ return config
170
+
171
+
172
+def _read_manifest(extension_path: Path) -> dict[str, Any]:
173
+ manifest_path = extension_path / "manifest.json"
174
+ try:
175
+ return json.loads(manifest_path.read_text(encoding="utf-8"))
176
+ except Exception:
177
+ return {}
plugins/_browser/helpers/playwright.py
new
+57
@@ -0,0 +1,57 @@
1
+import os
2
+import subprocess
3
+from pathlib import Path
4
+
5
+from helpers import files
6
+
7
+HEADLESS_SHELL_PATTERNS = (
8
+ "chromium_headless_shell-*/chrome-*/headless_shell",
9
+ "chromium_headless_shell-*/chrome-*/headless_shell.exe",
10
+)
11
+
12
+FULL_CHROMIUM_PATTERNS = (
13
+ "chromium-*/chrome-linux/chrome",
14
+ "chromium-*/chrome-win/chrome.exe",
15
+)
16
+
17
+
18
+def get_playwright_cache_dir() -> str:
19
+ return files.get_abs_path("tmp/playwright")
20
+
21
+
22
+def configure_playwright_env() -> str:
23
+ cache_dir = get_playwright_cache_dir()
24
+ os.environ["PLAYWRIGHT_BROWSERS_PATH"] = cache_dir
25
+ return cache_dir
26
+
27
+
28
+def get_playwright_binary(*, full_browser: bool = False) -> Path | None:
29
+ cache_dir = Path(get_playwright_cache_dir())
30
+ patterns = FULL_CHROMIUM_PATTERNS if full_browser else (HEADLESS_SHELL_PATTERNS + FULL_CHROMIUM_PATTERNS)
31
+ for pattern in patterns:
32
+ binary = next(cache_dir.glob(pattern), None)
33
+ if binary and binary.exists():
34
+ return binary
35
+ return None
36
+
37
+
38
+def ensure_playwright_binary(*, full_browser: bool = False) -> Path:
39
+ binary = get_playwright_binary(full_browser=full_browser)
40
+ if binary:
41
+ return binary
42
+
43
+ cache_dir = configure_playwright_env()
44
+ env = os.environ.copy()
45
+ env["PLAYWRIGHT_BROWSERS_PATH"] = cache_dir
46
+ install_command = ["playwright", "install", "chromium"]
47
+ if not full_browser:
48
+ install_command.append("--only-shell")
49
+ subprocess.check_call(
50
+ install_command,
51
+ env=env,
52
+ )
53
+
54
+ binary = get_playwright_binary(full_browser=full_browser)
55
+ if not binary:
56
+ raise RuntimeError("Playwright Chromium binary not found after installation")
57
+ return binary
plugins/_browser/helpers/runtime.py
new
+623
@@ -0,0 +1,623 @@
1
+from __future__ import annotations
2
+
3
+import atexit
4
+import asyncio
5
+import base64
6
+import re
7
+import shutil
8
+import threading
9
+from dataclasses import dataclass
10
+from pathlib import Path
11
+from typing import Any
12
+from urllib.parse import urlsplit, urlunsplit
13
+
14
+from helpers import files
15
+from helpers.defer import DeferredTask
16
+from helpers.print_style import PrintStyle
17
+
18
+from plugins._browser.helpers.config import build_browser_launch_config, get_browser_config
19
+from plugins._browser.helpers.playwright import configure_playwright_env, ensure_playwright_binary
20
+
21
+
22
+PLUGIN_DIR = Path(__file__).resolve().parents[1]
23
+CONTENT_HELPER_PATH = PLUGIN_DIR / "assets" / "browser-page-content.js"
24
+RUNTIME_DATA_KEY = "_browser_runtime"
25
+DEFAULT_VIEWPORT = {"width": 1024, "height": 768}
26
+
27
+_SPECIAL_SCHEME_RE = re.compile(r"^(?:about|blob|data|file|mailto|tel):", re.I)
28
+_URL_SCHEME_RE = re.compile(r"^[a-z][a-z\d+\-.]*://", re.I)
29
+_LOCAL_HOST_RE = re.compile(
30
+ r"^(?:localhost|\[[0-9a-f:.]+\]|(?:\d{1,3}\.){3}\d{1,3})(?::\d+)?$",
31
+ re.I,
32
+)
33
+_TYPED_HOST_RE = re.compile(
34
+ r"^(?:localhost|\[[0-9a-f:.]+\]|(?:\d{1,3}\.){3}\d{1,3}|"
35
+ r"(?:[a-z\d](?:[a-z\d-]{0,61}[a-z\d])?\.)+[a-z\d-]{2,63})(?::\d+)?$",
36
+ re.I,
37
+)
38
+_SAFE_CONTEXT_RE = re.compile(r"[^a-zA-Z0-9_.-]+")
39
+
40
+
41
+def normalize_url(value: str) -> str:
42
+ raw = str(value or "").strip()
43
+ if not raw:
44
+ raise ValueError("Browser navigation requires a non-empty URL.")
45
+
46
+ def with_trailing_path(url: str) -> str:
47
+ parts = urlsplit(url)
48
+ if parts.scheme in {"http", "https"} and not parts.path:
49
+ return urlunsplit((parts.scheme, parts.netloc, "/", parts.query, parts.fragment))
50
+ return urlunsplit(parts)
51
+
52
+ try:
53
+ host = re.split(r"[/?#]", raw, 1)[0] or ""
54
+ if (
55
+ not _URL_SCHEME_RE.match(raw)
56
+ and not _SPECIAL_SCHEME_RE.match(raw)
57
+ and not raw.startswith(("/", "?", "#", "."))
58
+ and not re.search(r"\s", raw)
59
+ and _TYPED_HOST_RE.match(host)
60
+ ):
61
+ protocol = "http://" if _LOCAL_HOST_RE.match(host) else "https://"
62
+ return with_trailing_path(protocol + raw)
63
+
64
+ parts = urlsplit(raw)
65
+ if parts.scheme:
66
+ return with_trailing_path(raw)
67
+ except Exception:
68
+ pass
69
+
70
+ return with_trailing_path("https://" + raw)
71
+
72
+
73
+def _safe_context_id(context_id: str) -> str:
74
+ return _SAFE_CONTEXT_RE.sub("_", str(context_id or "default")).strip("._") or "default"
75
+
76
+
77
+@dataclass
78
+class BrowserPage:
79
+ id: int
80
+ page: Any
81
+
82
+
83
+class BrowserRuntime:
84
+ def __init__(self, context_id: str):
85
+ self.context_id = str(context_id)
86
+ self._core = _BrowserRuntimeCore(self.context_id)
87
+ self._worker = DeferredTask(thread_name=f"BrowserRuntime-{self.context_id}")
88
+ self._closed = False
89
+
90
+ async def call(self, method: str, *args: Any, **kwargs: Any) -> Any:
91
+ if self._closed and method != "close":
92
+ raise RuntimeError("Browser runtime is closed.")
93
+
94
+ async def runner():
95
+ fn = getattr(self._core, method)
96
+ return await fn(*args, **kwargs)
97
+
98
+ return await self._worker.execute_inside(runner)
99
+
100
+ async def close(self, delete_profile: bool = False) -> None:
101
+ if self._closed:
102
+ return
103
+ try:
104
+ await self.call("close", delete_profile=delete_profile)
105
+ finally:
106
+ self._closed = True
107
+ self._worker.kill(terminate_thread=True)
108
+
109
+
110
+class _BrowserRuntimeCore:
111
+ def __init__(self, context_id: str):
112
+ self.context_id = context_id
113
+ self.safe_context_id = _safe_context_id(context_id)
114
+ self.playwright = None
115
+ self.context = None
116
+ self.pages: dict[int, BrowserPage] = {}
117
+ self.next_browser_id = 1
118
+ self.last_interacted_browser_id: int | None = None
119
+ self._content_helper_source: str | None = None
120
+
121
+ @property
122
+ def profile_dir(self) -> Path:
123
+ return Path(files.get_abs_path("tmp/browser/sessions", self.safe_context_id))
124
+
125
+ @property
126
+ def downloads_dir(self) -> Path:
127
+ return Path(files.get_abs_path("usr/downloads/browser"))
128
+
129
+ async def ensure_started(self) -> None:
130
+ if self.context:
131
+ return
132
+
133
+ from playwright.async_api import async_playwright
134
+
135
+ self.profile_dir.mkdir(parents=True, exist_ok=True)
136
+ self.downloads_dir.mkdir(parents=True, exist_ok=True)
137
+ browser_config = get_browser_config()
138
+ launch_config = build_browser_launch_config(browser_config)
139
+ configure_playwright_env()
140
+ browser_binary = ensure_playwright_binary(
141
+ full_browser=launch_config["requires_full_browser"]
142
+ )
143
+
144
+ self.playwright = await async_playwright().start()
145
+ launch_kwargs: dict[str, Any] = {
146
+ "user_data_dir": str(self.profile_dir),
147
+ "headless": True,
148
+ "accept_downloads": True,
149
+ "downloads_path": str(self.downloads_dir),
150
+ "viewport": DEFAULT_VIEWPORT,
151
+ "screen": DEFAULT_VIEWPORT,
152
+ "no_viewport": False,
153
+ "args": launch_config["args"],
154
+ }
155
+ if launch_config["channel"]:
156
+ launch_kwargs["channel"] = launch_config["channel"]
157
+ else:
158
+ launch_kwargs["executable_path"] = str(browser_binary)
159
+ self.context = await self.playwright.chromium.launch_persistent_context(
160
+ **launch_kwargs
161
+ )
162
+ self.context.set_default_timeout(30000)
163
+ self.context.set_default_navigation_timeout(30000)
164
+ await self.context.add_init_script(self._shadow_dom_script())
165
+ await self.context.add_init_script(path=str(CONTENT_HELPER_PATH))
166
+
167
+ for page in list(self.context.pages):
168
+ if page.url == "about:blank":
169
+ try:
170
+ await page.close()
171
+ except Exception:
172
+ pass
173
+ continue
174
+ self._register_page(page)
175
+
176
+ async def open(self, url: str = "about:blank") -> dict[str, Any]:
177
+ await self.ensure_started()
178
+ page = await self.context.new_page()
179
+ browser_page = self._register_page(page)
180
+ self.last_interacted_browser_id = browser_page.id
181
+ if url and url != "about:blank":
182
+ await self._goto(page, normalize_url(url))
183
+ else:
184
+ await self._settle(page)
185
+ return {"id": browser_page.id, "state": await self._state(browser_page.id)}
186
+
187
+ async def list(self) -> dict[str, Any]:
188
+ await self.ensure_started()
189
+ return {
190
+ "browsers": [await self._state(browser_id) for browser_id in sorted(self.pages)],
191
+ "last_interacted_browser_id": self.last_interacted_browser_id,
192
+ }
193
+
194
+ async def state(self, browser_id: int | str | None = None) -> dict[str, Any]:
195
+ await self.ensure_started()
196
+ return await self._state(self._resolve_browser_id(browser_id))
197
+
198
+ async def navigate(self, browser_id: int | str | None, url: str) -> dict[str, Any]:
199
+ await self.ensure_started()
200
+ resolved_id = self._resolve_browser_id(browser_id)
201
+ page = self._page(resolved_id)
202
+ await self._goto(page, normalize_url(url))
203
+ self.last_interacted_browser_id = resolved_id
204
+ return await self._state(resolved_id)
205
+
206
+ async def back(self, browser_id: int | str | None = None) -> dict[str, Any]:
207
+ await self.ensure_started()
208
+ resolved_id = self._resolve_browser_id(browser_id)
209
+ page = self._page(resolved_id)
210
+ await page.go_back(wait_until="domcontentloaded", timeout=10000)
211
+ await self._settle(page)
212
+ self.last_interacted_browser_id = resolved_id
213
+ return await self._state(resolved_id)
214
+
215
+ async def forward(self, browser_id: int | str | None = None) -> dict[str, Any]:
216
+ await self.ensure_started()
217
+ resolved_id = self._resolve_browser_id(browser_id)
218
+ page = self._page(resolved_id)
219
+ await page.go_forward(wait_until="domcontentloaded", timeout=10000)
220
+ await self._settle(page)
221
+ self.last_interacted_browser_id = resolved_id
222
+ return await self._state(resolved_id)
223
+
224
+ async def reload(self, browser_id: int | str | None = None) -> dict[str, Any]:
225
+ await self.ensure_started()
226
+ resolved_id = self._resolve_browser_id(browser_id)
227
+ page = self._page(resolved_id)
228
+ await page.reload(wait_until="domcontentloaded", timeout=15000)
229
+ await self._settle(page)
230
+ self.last_interacted_browser_id = resolved_id
231
+ return await self._state(resolved_id)
232
+
233
+ async def content(
234
+ self,
235
+ browser_id: int | str | None = None,
236
+ payload: dict[str, Any] | None = None,
237
+ ) -> dict[str, Any]:
238
+ await self.ensure_started()
239
+ resolved_id = self._resolve_browser_id(browser_id)
240
+ page = self._page(resolved_id)
241
+ await self._ensure_content_helper(page)
242
+ result = await page.evaluate(
243
+ "(payload) => globalThis.__spaceBrowserPageContent__.capture(payload || null)",
244
+ payload or None,
245
+ )
246
+ self.last_interacted_browser_id = resolved_id
247
+ return result or {}
248
+
249
+ async def detail(self, browser_id: int | str | None, reference_id: int | str) -> dict[str, Any]:
250
+ await self.ensure_started()
251
+ resolved_id = self._resolve_browser_id(browser_id)
252
+ page = self._page(resolved_id)
253
+ await self._ensure_content_helper(page)
254
+ result = await page.evaluate(
255
+ "(ref) => globalThis.__spaceBrowserPageContent__.detail(ref)",
256
+ reference_id,
257
+ )
258
+ self.last_interacted_browser_id = resolved_id
259
+ return result or {}
260
+
261
+ async def evaluate(self, browser_id: int | str | None, script: str) -> dict[str, Any]:
262
+ await self.ensure_started()
263
+ resolved_id = self._resolve_browser_id(browser_id)
264
+ page = self._page(resolved_id)
265
+ result = await page.evaluate(str(script or "undefined"))
266
+ self.last_interacted_browser_id = resolved_id
267
+ return {"result": result, "state": await self._state(resolved_id)}
268
+
269
+ async def click(self, browser_id: int | str | None, reference_id: int | str) -> dict[str, Any]:
270
+ return await self._reference_action("click", browser_id, reference_id)
271
+
272
+ async def submit(self, browser_id: int | str | None, reference_id: int | str) -> dict[str, Any]:
273
+ return await self._reference_action("submit", browser_id, reference_id)
274
+
275
+ async def scroll(self, browser_id: int | str | None, reference_id: int | str) -> dict[str, Any]:
276
+ return await self._reference_action("scroll", browser_id, reference_id)
277
+
278
+ async def type(
279
+ self,
280
+ browser_id: int | str | None,
281
+ reference_id: int | str,
282
+ text: str,
283
+ ) -> dict[str, Any]:
284
+ return await self._reference_action("type", browser_id, reference_id, text)
285
+
286
+ async def type_submit(
287
+ self,
288
+ browser_id: int | str | None,
289
+ reference_id: int | str,
290
+ text: str,
291
+ ) -> dict[str, Any]:
292
+ return await self._reference_action("typeSubmit", browser_id, reference_id, text)
293
+
294
+ async def close_browser(self, browser_id: int | str | None = None) -> dict[str, Any]:
295
+ await self.ensure_started()
296
+ resolved_id = self._resolve_browser_id(browser_id)
297
+ page = self._page(resolved_id)
298
+ await page.close()
299
+ self.pages.pop(resolved_id, None)
300
+ if self.last_interacted_browser_id == resolved_id:
301
+ self.last_interacted_browser_id = next(iter(sorted(self.pages)), None)
302
+ return await self.list()
303
+
304
+ async def close_all_browsers(self) -> dict[str, Any]:
305
+ await self.ensure_started()
306
+ for browser_id in list(self.pages):
307
+ try:
308
+ await self.pages[browser_id].page.close()
309
+ except Exception:
310
+ pass
311
+ self.pages.clear()
312
+ self.last_interacted_browser_id = None
313
+ return {"browsers": [], "last_interacted_browser_id": None}
314
+
315
+ async def screenshot(
316
+ self,
317
+ browser_id: int | str | None = None,
318
+ *,
319
+ quality: int = 70,
320
+ ) -> dict[str, Any]:
321
+ await self.ensure_started()
322
+ resolved_id = self._resolve_browser_id(browser_id)
323
+ page = self._page(resolved_id)
324
+ image = await page.screenshot(type="jpeg", quality=max(20, min(95, int(quality))))
325
+ return {
326
+ "browser_id": resolved_id,
327
+ "mime": "image/jpeg",
328
+ "image": base64.b64encode(image).decode("ascii"),
329
+ "state": await self._state(resolved_id),
330
+ }
331
+
332
+ async def set_viewport(
333
+ self,
334
+ browser_id: int | str | None,
335
+ width: int,
336
+ height: int,
337
+ ) -> dict[str, Any]:
338
+ await self.ensure_started()
339
+ resolved_id = self._resolve_browser_id(browser_id)
340
+ page = self._page(resolved_id)
341
+ viewport = {
342
+ "width": max(320, min(4096, int(width or DEFAULT_VIEWPORT["width"]))),
343
+ "height": max(200, min(4096, int(height or DEFAULT_VIEWPORT["height"]))),
344
+ }
345
+ await page.set_viewport_size(viewport)
346
+ self.last_interacted_browser_id = resolved_id
347
+ return {"state": await self._state(resolved_id), "viewport": viewport}
348
+
349
+ async def mouse(
350
+ self,
351
+ browser_id: int | str | None,
352
+ event_type: str,
353
+ x: float,
354
+ y: float,
355
+ button: str = "left",
356
+ ) -> dict[str, Any]:
357
+ await self.ensure_started()
358
+ resolved_id = self._resolve_browser_id(browser_id)
359
+ page = self._page(resolved_id)
360
+ event_type = str(event_type or "click").lower()
361
+ if event_type == "move":
362
+ await page.mouse.move(float(x), float(y))
363
+ elif event_type == "down":
364
+ await page.mouse.down(button=button)
365
+ elif event_type == "up":
366
+ await page.mouse.up(button=button)
367
+ else:
368
+ await page.mouse.click(float(x), float(y), button=button)
369
+ await self._settle(page, short=True)
370
+ self.last_interacted_browser_id = resolved_id
371
+ return await self._state(resolved_id)
372
+
373
+ async def wheel(
374
+ self,
375
+ browser_id: int | str | None,
376
+ x: float,
377
+ y: float,
378
+ delta_x: float = 0,
379
+ delta_y: float = 0,
380
+ ) -> dict[str, Any]:
381
+ await self.ensure_started()
382
+ resolved_id = self._resolve_browser_id(browser_id)
383
+ page = self._page(resolved_id)
384
+ await page.mouse.move(float(x), float(y))
385
+ await page.mouse.wheel(float(delta_x), float(delta_y))
386
+ await self._settle(page, short=True)
387
+ self.last_interacted_browser_id = resolved_id
388
+ return await self._state(resolved_id)
389
+
390
+ async def keyboard(
391
+ self,
392
+ browser_id: int | str | None,
393
+ *,
394
+ key: str = "",
395
+ text: str = "",
396
+ ) -> dict[str, Any]:
397
+ await self.ensure_started()
398
+ resolved_id = self._resolve_browser_id(browser_id)
399
+ page = self._page(resolved_id)
400
+ if text:
401
+ await page.keyboard.type(str(text))
402
+ elif key:
403
+ await page.keyboard.press(str(key))
404
+ await self._settle(page, short=True)
405
+ self.last_interacted_browser_id = resolved_id
406
+ return await self._state(resolved_id)
407
+
408
+ async def close(self, delete_profile: bool = False) -> None:
409
+ for browser_id in list(self.pages):
410
+ try:
411
+ await self.pages[browser_id].page.close()
412
+ except Exception:
413
+ pass
414
+ self.pages.clear()
415
+ if self.context:
416
+ try:
417
+ await self.context.close()
418
+ except Exception as exc:
419
+ PrintStyle.warning(f"Browser context close failed: {exc}")
420
+ self.context = None
421
+ if self.playwright:
422
+ try:
423
+ await self.playwright.stop()
424
+ except Exception as exc:
425
+ PrintStyle.warning(f"Playwright stop failed: {exc}")
426
+ self.playwright = None
427
+ self.last_interacted_browser_id = None
428
+ if delete_profile:
429
+ shutil.rmtree(self.profile_dir, ignore_errors=True)
430
+
431
+ async def _reference_action(
432
+ self,
433
+ helper_method: str,
434
+ browser_id: int | str | None,
435
+ reference_id: int | str,
436
+ text: str | None = None,
437
+ ) -> dict[str, Any]:
438
+ resolved_id = self._resolve_browser_id(browser_id)
439
+ page = self._page(resolved_id)
440
+ await self._ensure_content_helper(page)
441
+ if text is None:
442
+ action = await page.evaluate(
443
+ "(args) => globalThis.__spaceBrowserPageContent__[args.method](args.ref)",
444
+ {"method": helper_method, "ref": reference_id},
445
+ )
446
+ else:
447
+ action = await page.evaluate(
448
+ "(args) => globalThis.__spaceBrowserPageContent__[args.method](args.ref, args.text)",
449
+ {"method": helper_method, "ref": reference_id, "text": text},
450
+ )
451
+ await self._settle(page, short=False)
452
+ self.last_interacted_browser_id = resolved_id
453
+ return {"action": action or {}, "state": await self._state(resolved_id)}
454
+
455
+ async def _goto(self, page: Any, url: str) -> None:
456
+ from playwright.async_api import TimeoutError as PlaywrightTimeoutError
457
+
458
+ try:
459
+ await page.goto(url, wait_until="domcontentloaded", timeout=30000)
460
+ except PlaywrightTimeoutError:
461
+ PrintStyle.warning(f"Browser navigation timed out after DOM handoff: {url}")
462
+ await self._settle(page)
463
+
464
+ async def _settle(self, page: Any, short: bool = False) -> None:
465
+ from playwright.async_api import TimeoutError as PlaywrightTimeoutError
466
+
467
+ try:
468
+ await page.wait_for_load_state(
469
+ "domcontentloaded",
470
+ timeout=1000 if short else 5000,
471
+ )
472
+ except PlaywrightTimeoutError:
473
+ pass
474
+ await asyncio.sleep(0.1 if short else 0.35)
475
+
476
+ async def _state(self, browser_id: int) -> dict[str, Any]:
477
+ browser_page = self.pages.get(int(browser_id))
478
+ if not browser_page:
479
+ raise KeyError(f"Browser {browser_id} is not open.")
480
+ page = browser_page.page
481
+ try:
482
+ title = await page.title()
483
+ except Exception:
484
+ title = ""
485
+ try:
486
+ history_length = await page.evaluate("() => globalThis.history?.length || 0")
487
+ except Exception:
488
+ history_length = 0
489
+ return {
490
+ "id": browser_page.id,
491
+ "currentUrl": page.url,
492
+ "title": title,
493
+ "canGoBack": bool(history_length and int(history_length) > 1),
494
+ "canGoForward": False,
495
+ "loading": False,
496
+ }
497
+
498
+ def _register_page(self, page: Any) -> BrowserPage:
499
+ existing = self._browser_id_for_page(page)
500
+ if existing is not None:
501
+ return self.pages[existing]
502
+ browser_id = self.next_browser_id
503
+ self.next_browser_id += 1
504
+ browser_page = BrowserPage(id=browser_id, page=page)
505
+ self.pages[browser_id] = browser_page
506
+
507
+ def on_close() -> None:
508
+ self.pages.pop(browser_id, None)
509
+
510
+ page.on("close", on_close)
511
+ return browser_page
512
+
513
+ def _browser_id_for_page(self, page: Any) -> int | None:
514
+ for browser_id, browser_page in self.pages.items():
515
+ if browser_page.page == page:
516
+ return browser_id
517
+ return None
518
+
519
+ def _resolve_browser_id(self, browser_id: int | str | None = None) -> int:
520
+ if browser_id is None or str(browser_id).strip() == "":
521
+ if self.last_interacted_browser_id in self.pages:
522
+ return int(self.last_interacted_browser_id)
523
+ if self.pages:
524
+ return sorted(self.pages)[0]
525
+ raise KeyError("No browser is open. Use action=open first.")
526
+ value = str(browser_id).strip()
527
+ if value.startswith("browser-"):
528
+ value = value.split("-", 1)[1]
529
+ resolved = int(value)
530
+ if resolved not in self.pages:
531
+ raise KeyError(f"Browser {resolved} is not open.")
532
+ return resolved
533
+
534
+ def _page(self, browser_id: int) -> Any:
535
+ return self.pages[int(browser_id)].page
536
+
537
+ async def _ensure_content_helper(self, page: Any) -> None:
538
+ has_helper = await page.evaluate(
539
+ "() => Boolean(globalThis.__spaceBrowserPageContent__?.capture)"
540
+ )
541
+ if has_helper:
542
+ return
543
+ if self._content_helper_source is None:
544
+ self._content_helper_source = CONTENT_HELPER_PATH.read_text(encoding="utf-8")
545
+ await page.evaluate(self._content_helper_source)
546
+
547
+ @staticmethod
548
+ def _shadow_dom_script() -> str:
549
+ return """
550
+(() => {
551
+ const original = Element.prototype.attachShadow;
552
+ if (original && !original.__a0BrowserOpenShadowPatch) {
553
+ const patched = function attachShadow(options) {
554
+ return original.call(this, { ...(options || {}), mode: "open" });
555
+ };
556
+ patched.__a0BrowserOpenShadowPatch = true;
557
+ Element.prototype.attachShadow = patched;
558
+ }
559
+})();
560
+"""
561
+
562
+
563
+_runtimes: dict[str, BrowserRuntime] = {}
564
+_runtime_lock = threading.RLock()
565
+
566
+
567
+async def get_runtime(context_id: str, *, create: bool = True) -> BrowserRuntime | None:
568
+ context_id = str(context_id or "").strip()
569
+ if not context_id:
570
+ raise ValueError("context_id is required")
571
+ with _runtime_lock:
572
+ runtime = _runtimes.get(context_id)
573
+ if runtime is None and create:
574
+ runtime = BrowserRuntime(context_id)
575
+ _runtimes[context_id] = runtime
576
+ return runtime
577
+
578
+
579
+async def close_runtime(context_id: str, *, delete_profile: bool = True) -> None:
580
+ context_id = str(context_id or "").strip()
581
+ if not context_id:
582
+ return
583
+ with _runtime_lock:
584
+ runtime = _runtimes.pop(context_id, None)
585
+ if runtime:
586
+ await runtime.close(delete_profile=delete_profile)
587
+
588
+
589
+def close_runtime_sync(context_id: str, *, delete_profile: bool = True) -> None:
590
+ task = DeferredTask(thread_name="BrowserCleanup")
591
+ task.start_task(close_runtime, context_id, delete_profile=delete_profile)
592
+ try:
593
+ task.result_sync(timeout=30)
594
+ finally:
595
+ task.kill(terminate_thread=True)
596
+
597
+
598
+async def close_all_runtimes(*, delete_profiles: bool = False) -> None:
599
+ with _runtime_lock:
600
+ runtimes = list(_runtimes.values())
601
+ _runtimes.clear()
602
+ for runtime in runtimes:
603
+ try:
604
+ await runtime.close(delete_profile=delete_profiles)
605
+ except Exception as exc:
606
+ PrintStyle.warning(f"Browser runtime cleanup failed: {exc}")
607
+
608
+
609
+def close_all_runtimes_sync() -> None:
610
+ task = DeferredTask(thread_name="BrowserCleanupAll")
611
+ task.start_task(close_all_runtimes, delete_profiles=False)
612
+ try:
613
+ task.result_sync(timeout=30)
614
+ finally:
615
+ task.kill(terminate_thread=True)
616
+
617
+
618
+def known_context_ids() -> list[str]:
619
+ with _runtime_lock:
620
+ return sorted(_runtimes)
621
+
622
+
623
+atexit.register(close_all_runtimes_sync)
plugins/_browser/hooks.py
new
+47
@@ -0,0 +1,47 @@
1
+from __future__ import annotations
2
+
3
+from helpers import files, plugins, yaml as yaml_helper
4
+from plugins._browser.helpers.config import (
5
+ PLUGIN_NAME,
6
+ browser_runtime_config,
7
+ normalize_browser_config,
8
+)
9
+from plugins._browser.helpers.runtime import close_all_runtimes_sync
10
+
11
+
12
+def _load_saved_browser_config(project_name: str = "", agent_profile: str = "") -> dict:
13
+ entries = plugins.find_plugin_assets(
14
+ plugins.CONFIG_FILE_NAME,
15
+ plugin_name=PLUGIN_NAME,
16
+ project_name=project_name,
17
+ agent_profile=agent_profile,
18
+ only_first=True,
19
+ )
20
+ path = entries[0].get("path", "") if entries else ""
21
+ if path and files.exists(path):
22
+ return files.read_file_json(path) or {}
23
+
24
+ plugin_dir = plugins.find_plugin_dir(PLUGIN_NAME)
25
+ default_path = (
26
+ files.get_abs_path(plugin_dir, plugins.CONFIG_DEFAULT_FILE_NAME)
27
+ if plugin_dir
28
+ else ""
29
+ )
30
+ if default_path and files.exists(default_path):
31
+ return yaml_helper.loads(files.read_file(default_path)) or {}
32
+
33
+ return {}
34
+
35
+
36
+def get_plugin_config(default=None, **kwargs):
37
+ return normalize_browser_config(default)
38
+
39
+
40
+def save_plugin_config(settings=None, project_name="", agent_profile="", **kwargs):
41
+ normalized = normalize_browser_config(settings)
42
+ current = normalize_browser_config(
43
+ _load_saved_browser_config(project_name=project_name, agent_profile=agent_profile)
44
+ )
45
+ if browser_runtime_config(normalized) != browser_runtime_config(current):
46
+ close_all_runtimes_sync()
47
+ return normalized
plugins/_browser/plugin.yaml
new
+9
@@ -0,0 +1,9 @@
1
+name: _browser
2
+title: Browser
3
+description: Built-in direct Playwright browser tool and WebUI viewer.
4
+version: 1.0.0
5
+always_enabled: false
6
+settings_sections:
7
+ - external
8
+per_project_config: false
9
+per_agent_config: false
plugins/_browser/prompts/agent.system.tool.browser.md
new
+48
@@ -0,0 +1,48 @@
1
+### browser
2
+direct Playwright browser control with visible WebUI viewer
3
+use for web browsing, page inspection, forms, downloads, and browser-only tasks
4
+state stays open per chat context
5
+refs come from content as typed markers: [link 3], [button 6], [image 1], [input text 8]
6
+
7
+actions: open list state navigate back forward reload content detail click type submit type_submit scroll evaluate close close_all
8
+common args: action browser_id url ref text selector selectors script
9
+
10
+workflow:
11
+- open creates a new browser and returns id/state
12
+- content returns readable page markdown with typed refs
13
+- detail inspects one ref, including link/image/input/button metadata
14
+- click/type/type_submit/submit/scroll use refs from latest content capture and return {action,state}
15
+- navigate/back/forward/reload return fresh state
16
+- list shows open browsers
17
+
18
+examples:
19
+~~~json
20
+{
21
+ "tool_name": "browser",
22
+ "tool_args": {
23
+ "action": "open",
24
+ "url": "https://example.com"
25
+ }
26
+}
27
+~~~
28
+
29
+~~~json
30
+{
31
+ "tool_name": "browser",
32
+ "tool_args": {
33
+ "action": "content",
34
+ "browser_id": 1
35
+ }
36
+}
37
+~~~
38
+
39
+~~~json
40
+{
41
+ "tool_name": "browser",
42
+ "tool_args": {
43
+ "action": "click",
44
+ "browser_id": 1,
45
+ "ref": 3
46
+ }
47
+}
48
+~~~
plugins/_browser/tools/browser.py
new
+107
@@ -0,0 +1,107 @@
1
+from __future__ import annotations
2
+
3
+import json
4
+from typing import Any
5
+
6
+from helpers.tool import Response, Tool
7
+from plugins._browser.helpers.runtime import get_runtime
8
+
9
+
10
+class Browser(Tool):
11
+ async def execute(
12
+ self,
13
+ action: str = "",
14
+ browser_id: int | str | None = None,
15
+ url: str = "",
16
+ ref: int | str | None = None,
17
+ text: str = "",
18
+ selector: str = "",
19
+ selectors: list[str] | None = None,
20
+ script: str = "",
21
+ **kwargs: Any,
22
+ ) -> Response:
23
+ action = str(action or self.method or "state").strip().lower().replace("-", "_")
24
+ runtime = await get_runtime(self.agent.context.id)
25
+
26
+ try:
27
+ if action == "open":
28
+ result = await runtime.call("open", url or "about:blank")
29
+ elif action == "list":
30
+ result = await runtime.call("list")
31
+ elif action == "state":
32
+ result = await runtime.call("state", browser_id)
33
+ elif action == "navigate":
34
+ result = await runtime.call("navigate", browser_id, url)
35
+ elif action == "back":
36
+ result = await runtime.call("back", browser_id)
37
+ elif action == "forward":
38
+ result = await runtime.call("forward", browser_id)
39
+ elif action == "reload":
40
+ result = await runtime.call("reload", browser_id)
41
+ elif action == "content":
42
+ payload = self._selector_payload(selector, selectors)
43
+ result = await runtime.call("content", browser_id, payload)
44
+ elif action == "detail":
45
+ result = await runtime.call("detail", browser_id, self._require_ref(ref))
46
+ elif action == "click":
47
+ result = await runtime.call("click", browser_id, self._require_ref(ref))
48
+ elif action == "type":
49
+ result = await runtime.call("type", browser_id, self._require_ref(ref), text)
50
+ elif action == "submit":
51
+ result = await runtime.call("submit", browser_id, self._require_ref(ref))
52
+ elif action in {"type_submit", "typesubmit"}:
53
+ result = await runtime.call(
54
+ "type_submit",
55
+ browser_id,
56
+ self._require_ref(ref),
57
+ text,
58
+ )
59
+ elif action == "scroll":
60
+ result = await runtime.call("scroll", browser_id, self._require_ref(ref))
61
+ elif action == "evaluate":
62
+ result = await runtime.call("evaluate", browser_id, script)
63
+ elif action == "close":
64
+ result = await runtime.call("close_browser", browser_id)
65
+ elif action == "close_all":
66
+ result = await runtime.call("close_all_browsers")
67
+ else:
68
+ return Response(
69
+ message=f"Unknown browser action: {action}",
70
+ break_loop=False,
71
+ )
72
+ except Exception as exc:
73
+ return Response(message=f"Browser {action} failed: {exc}", break_loop=False)
74
+
75
+ return Response(message=self._format_result(action, result), break_loop=False)
76
+
77
+ def get_log_object(self):
78
+ return self.agent.context.log.log(
79
+ type="tool",
80
+ heading=f"icon://captive_portal {self.agent.agent_name}: Using browser",
81
+ content="",
82
+ kvps=self.args,
83
+ _tool_name=self.name,
84
+ )
85
+
86
+ @staticmethod
87
+ def _require_ref(ref: int | str | None) -> int | str:
88
+ if ref is None or str(ref).strip() == "":
89
+ raise ValueError("ref is required for this browser action")
90
+ return ref
91
+
92
+ @staticmethod
93
+ def _selector_payload(selector: str = "", selectors: list[str] | None = None) -> dict | None:
94
+ if selectors:
95
+ return {"selectors": selectors}
96
+ if selector:
97
+ return {"selector": selector}
98
+ return None
99
+
100
+ @staticmethod
101
+ def _format_result(action: str, result: Any) -> str:
102
+ if action == "content" and isinstance(result, dict):
103
+ if set(result.keys()) == {"document"}:
104
+ return str(result.get("document") or "")
105
+ return json.dumps(result, indent=2, ensure_ascii=False)
106
+
107
+ return json.dumps(result, indent=2, ensure_ascii=False, default=str)
plugins/_browser/webui/browser-config-store.js
new
+155
@@ -0,0 +1,155 @@
1
+import { createStore } from "/js/AlpineStore.js";
2
+import { fetchApi } from "/js/api.js";
3
+
4
+const MODEL_CONFIG_API = "/plugins/_model_config/model_presets";
5
+
6
+function normalizePathList(value) {
7
+ const source = Array.isArray(value)
8
+ ? value
9
+ : String(value || "").split(/\r?\n/);
10
+ const seen = new Set();
11
+ const paths = [];
12
+ for (const item of source) {
13
+ const path = String(item || "").trim();
14
+ if (!path || seen.has(path)) continue;
15
+ seen.add(path);
16
+ paths.push(path);
17
+ }
18
+ return paths;
19
+}
20
+
21
+function ensureConfig(config) {
22
+ if (!config || typeof config !== "object") return null;
23
+ if (typeof config.extensions_enabled !== "boolean") {
24
+ config.extensions_enabled = Boolean(config.extensions_enabled);
25
+ }
26
+ config.extension_paths = normalizePathList(config.extension_paths);
27
+ config.model_preset = String(config.model_preset || "").trim();
28
+ delete config.model;
29
+ return config;
30
+}
31
+
32
+export const store = createStore("browserConfig", {
33
+ config: null,
34
+ extensionPathsText: "",
35
+ presets: [],
36
+ presetsLoading: false,
37
+ presetsError: "",
38
+ _presetsLoaded: false,
39
+
40
+ async init(config) {
41
+ this.bindConfig(config);
42
+ await this.loadPresets();
43
+ },
44
+
45
+ cleanup() {
46
+ this.config = null;
47
+ this.extensionPathsText = "";
48
+ this.presetsError = "";
49
+ },
50
+
51
+ bindConfig(config) {
52
+ const safeConfig = ensureConfig(config);
53
+ if (!safeConfig) return;
54
+ if (this.config === safeConfig) return;
55
+ this.config = safeConfig;
56
+ this.extensionPathsText = safeConfig.extension_paths.join("\n");
57
+ },
58
+
59
+ setExtensionPathsText(value) {
60
+ this.extensionPathsText = String(value || "");
61
+ this.syncExtensionPaths();
62
+ },
63
+
64
+ syncExtensionPaths() {
65
+ const safeConfig = ensureConfig(this.config);
66
+ if (!safeConfig) return;
67
+ safeConfig.extension_paths = normalizePathList(this.extensionPathsText);
68
+ },
69
+
70
+ hasPaths() {
71
+ return this.pathCount() > 0;
72
+ },
73
+
74
+ pathCount() {
75
+ return normalizePathList(this.extensionPathsText).length;
76
+ },
77
+
78
+ pathCountLabel() {
79
+ const count = this.pathCount();
80
+ if (!count) return "No extension paths configured";
81
+ return `${count} path${count === 1 ? "" : "s"} configured`;
82
+ },
83
+
84
+ extensionModeReady() {
85
+ const safeConfig = ensureConfig(this.config);
86
+ return Boolean(safeConfig?.extensions_enabled && this.pathCount());
87
+ },
88
+
89
+ async loadPresets() {
90
+ if (this._presetsLoaded || this.presetsLoading) return;
91
+ this.presetsLoading = true;
92
+ this.presetsError = "";
93
+ try {
94
+ const response = await fetchApi(MODEL_CONFIG_API, {
95
+ method: "POST",
96
+ headers: { "Content-Type": "application/json" },
97
+ body: JSON.stringify({ action: "get" }),
98
+ });
99
+ const data = await response.json().catch(() => ({}));
100
+ this.presets = Array.isArray(data?.presets)
101
+ ? data.presets.filter((preset) => String(preset?.name || "").trim())
102
+ : [];
103
+ this._presetsLoaded = true;
104
+ } catch (error) {
105
+ this.presets = [];
106
+ this.presetsError = error instanceof Error ? error.message : String(error);
107
+ } finally {
108
+ this.presetsLoading = false;
109
+ }
110
+ },
111
+
112
+ selectedPreset() {
113
+ const selected = String(this.config?.model_preset || "").trim();
114
+ if (!selected) return null;
115
+ return this.presets.find((preset) => preset?.name === selected) || null;
116
+ },
117
+
118
+ presetOptions() {
119
+ const selected = String(this.config?.model_preset || "").trim();
120
+ const options = this.presets.map((preset) => ({
121
+ ...preset,
122
+ label: preset.name,
123
+ missing: false,
124
+ }));
125
+ if (selected && this._presetsLoaded && !options.some((preset) => preset.name === selected)) {
126
+ options.push({
127
+ name: selected,
128
+ label: `${selected} (missing)`,
129
+ missing: true,
130
+ });
131
+ }
132
+ return options;
133
+ },
134
+
135
+ selectedPresetSummary() {
136
+ const selected = String(this.config?.model_preset || "").trim();
137
+ if (!selected) return "Using the effective Main Model.";
138
+
139
+ const preset = this.selectedPreset();
140
+ if (!preset) return `Preset "${selected}" is not available. Browser will fall back to the Main Model.`;
141
+
142
+ const chat = preset.chat || {};
143
+ const parts = [chat.provider, chat.name].filter((item) => String(item || "").trim());
144
+ return parts.length ? parts.join(" / ") : "This preset has no Main Model; Browser will fall back to the Main Model.";
145
+ },
146
+
147
+ selectedPresetMissing() {
148
+ const selected = String(this.config?.model_preset || "").trim();
149
+ return Boolean(selected && this._presetsLoaded && !this.selectedPreset());
150
+ },
151
+
152
+ openPresets() {
153
+ void globalThis.openModal?.("/plugins/_model_config/webui/main.html");
154
+ },
155
+});
plugins/_browser/webui/browser-store.js
new
+596
@@ -0,0 +1,596 @@
1
+import { createStore } from "/js/AlpineStore.js";
2
+import { callJsonApi } from "/js/api.js";
3
+import { getNamespacedClient } from "/js/websocket.js";
4
+import { store as chatInputStore } from "/components/chat/input/input-store.js";
5
+import { store as fileBrowserStore } from "/components/modals/file-browser/file-browser-store.js";
6
+import { store as pluginSettingsStore } from "/components/plugins/plugin-settings-store.js";
7
+
8
+const websocket = getNamespacedClient("/ws");
9
+websocket.addHandlers(["ws_webui"]);
10
+
11
+const EXTENSIONS_ROOT_FALLBACK = "/a0/usr/browser-extensions";
12
+
13
+function firstOk(response) {
14
+ const result = response?.results?.find((item) => item?.ok);
15
+ if (result) {
16
+ const data = result.data || {};
17
+ if (data.browser_error) {
18
+ throw new Error(data.browser_error.error || data.browser_error.code || "Browser request failed");
19
+ }
20
+ return data;
21
+ }
22
+ const error = response?.results?.find((item) => !item?.ok)?.error;
23
+ if (error) throw new Error(error.error || error.code || "Browser request failed");
24
+ return {};
25
+}
26
+
27
+const model = {
28
+ loading: true,
29
+ error: "",
30
+ status: null,
31
+ contextId: "",
32
+ browsers: [],
33
+ activeBrowserId: null,
34
+ address: "",
35
+ frameSrc: "",
36
+ frameState: null,
37
+ connected: false,
38
+ addressFocused: false,
39
+ _frameOff: null,
40
+ _stateOff: null,
41
+ _lastFrameAt: 0,
42
+ _floatingCleanup: null,
43
+ _stageElement: null,
44
+ _stageResizeObserver: null,
45
+ _viewportSyncTimer: null,
46
+ _lastViewportKey: "",
47
+ extensionMenuOpen: false,
48
+ extensionInstallUrl: "",
49
+ extensionActionLoading: false,
50
+ extensionActionMessage: "",
51
+ extensionActionError: "",
52
+ extensionsRoot: "",
53
+ extensionsList: [],
54
+
55
+ async refreshStatus() {
56
+ this.status = await callJsonApi("/plugins/_browser/status", {});
57
+ },
58
+
59
+ async refreshExtensionsList() {
60
+ const response = await callJsonApi("/plugins/_browser/extensions", { action: "list" });
61
+ if (response?.ok) {
62
+ this.extensionsRoot = response.root || EXTENSIONS_ROOT_FALLBACK;
63
+ this.extensionsList = Array.isArray(response.extensions) ? response.extensions : [];
64
+ }
65
+ },
66
+
67
+ toggleExtensionsMenu() {
68
+ this.extensionMenuOpen = !this.extensionMenuOpen;
69
+ if (this.extensionMenuOpen) {
70
+ this.extensionActionMessage = "";
71
+ this.extensionActionError = "";
72
+ void this.refreshExtensionsList();
73
+ }
74
+ },
75
+
76
+ closeExtensionsMenu() {
77
+ this.extensionMenuOpen = false;
78
+ },
79
+
80
+ resolveContextId() {
81
+ const urlContext = new URLSearchParams(globalThis.location?.search || "").get("ctxid");
82
+ const selectedChat = globalThis.Alpine?.store?.("chats")?.selected;
83
+ return globalThis.getContext?.() || urlContext || selectedChat || "";
84
+ },
85
+
86
+ async openExtensionsSettings() {
87
+ if (!pluginSettingsStore?.openConfig) {
88
+ this.error = "Browser settings are unavailable.";
89
+ return;
90
+ }
91
+ try {
92
+ this.closeExtensionsMenu();
93
+ await pluginSettingsStore.openConfig("_browser");
94
+ await this.refreshAfterSettingsClose();
95
+ } catch (error) {
96
+ this.error = error instanceof Error ? error.message : String(error);
97
+ }
98
+ },
99
+
100
+ async refreshAfterSettingsClose() {
101
+ this.loading = true;
102
+ this.error = "";
103
+ try {
104
+ await this.refreshStatus();
105
+ await this.refreshExtensionsList();
106
+ this.connected = false;
107
+ this.browsers = [];
108
+ this.setActiveBrowserId(null);
109
+ this.address = "";
110
+ this.frameState = null;
111
+ this.frameSrc = "";
112
+ if (this.contextId) {
113
+ await this.connectViewer();
114
+ }
115
+ } finally {
116
+ this.loading = false;
117
+ }
118
+ },
119
+
120
+ async openExtensionsFolder() {
121
+ this.closeExtensionsMenu();
122
+ try {
123
+ if (!this.extensionsRoot) {
124
+ await this.refreshExtensionsList();
125
+ }
126
+ void fileBrowserStore.open(this.extensionsRoot || EXTENSIONS_ROOT_FALLBACK);
127
+ } catch (error) {
128
+ this.extensionActionError = error instanceof Error ? error.message : String(error);
129
+ }
130
+ },
131
+
132
+ createExtensionWithAgent() {
133
+ this._prefillAgentPrompt(
134
+ [
135
+ "Use the a0-browser-ext skill to create a new Chrome extension for Agent Zero's Browser.",
136
+ "Start by asking me for the extension name, purpose, target websites, and required permissions.",
137
+ `Create it under ${this.extensionsRoot || EXTENSIONS_ROOT_FALLBACK}/<extension-slug> and keep permissions minimal.`,
138
+ ].join("\n")
139
+ );
140
+ },
141
+
142
+ askAgentInstallExtension() {
143
+ const url = String(this.extensionInstallUrl || "").trim();
144
+ this._prefillAgentPrompt(
145
+ [
146
+ "Use the a0-browser-ext skill to install and review a Chrome Web Store extension for Agent Zero's Browser.",
147
+ url ? `Chrome Web Store URL or id: ${url}` : "Ask me for the Chrome Web Store URL or extension id first.",
148
+ "Explain the permissions and any sandbox risk before enabling it.",
149
+ ].join("\n")
150
+ );
151
+ },
152
+
153
+ async installExtensionFromUrl() {
154
+ const url = String(this.extensionInstallUrl || "").trim();
155
+ this.extensionActionMessage = "";
156
+ this.extensionActionError = "";
157
+ if (!url) {
158
+ this.extensionActionError = "Paste a Chrome Web Store URL or extension id first.";
159
+ return;
160
+ }
161
+
162
+ this.extensionActionLoading = true;
163
+ try {
164
+ const response = await callJsonApi("/plugins/_browser/extensions", {
165
+ action: "install_web_store",
166
+ url,
167
+ });
168
+ if (!response?.ok) {
169
+ throw new Error(response?.error || "Install failed.");
170
+ }
171
+ this.extensionInstallUrl = "";
172
+ this.extensionActionMessage = `Installed ${response.name || response.id}. Browser sessions restart when extension settings change.`;
173
+ await this.refreshStatus();
174
+ await this.refreshExtensionsList();
175
+ } catch (error) {
176
+ this.extensionActionError = error instanceof Error ? error.message : String(error);
177
+ } finally {
178
+ this.extensionActionLoading = false;
179
+ }
180
+ },
181
+
182
+ _prefillAgentPrompt(prompt) {
183
+ chatInputStore.message = prompt;
184
+ chatInputStore.adjustTextareaHeight?.();
185
+ chatInputStore.focus?.();
186
+ this.closeExtensionsMenu();
187
+ },
188
+
189
+ async onOpen(element = null) {
190
+ this.loading = true;
191
+ this.error = "";
192
+ this.setupFloatingModal(element);
193
+ this.contextId = this.resolveContextId();
194
+ try {
195
+ await this.refreshStatus();
196
+ await this.connectViewer();
197
+ } catch (error) {
198
+ this.error = error instanceof Error ? error.message : String(error);
199
+ } finally {
200
+ this.loading = false;
201
+ }
202
+ },
203
+
204
+ async connectViewer() {
205
+ if (!this.contextId) {
206
+ this.connected = false;
207
+ this.error = "No active chat context is selected.";
208
+ return;
209
+ }
210
+ this.error = "";
211
+ await this._bindSocketEvents();
212
+ const response = await websocket.request(
213
+ "browser_viewer_subscribe",
214
+ {
215
+ context_id: this.contextId,
216
+ browser_id: this.activeBrowserId,
217
+ },
218
+ { timeoutMs: 10000 },
219
+ );
220
+ const data = firstOk(response);
221
+ this.browsers = data.browsers || [];
222
+ this.setActiveBrowserId(data.active_browser_id || this.activeBrowserId || null);
223
+ this.connected = true;
224
+ this.queueViewportSync(true);
225
+ },
226
+
227
+ async _bindSocketEvents() {
228
+ if (!this._frameOff) {
229
+ const frameHandler = ({ data }) => {
230
+ if (data?.context_id !== this.contextId) return;
231
+ this.browsers = data.browsers || this.browsers;
232
+ this.setActiveBrowserId(data.browser_id || data.state?.id || this.activeBrowserId);
233
+ this.frameState = data.state || null;
234
+ if (!this.addressFocused && data.state?.currentUrl) {
235
+ this.address = data.state.currentUrl;
236
+ }
237
+ this.frameSrc = data.image ? `data:${data.mime || "image/jpeg"};base64,${data.image}` : "";
238
+ if (!data.image && !data.state) {
239
+ this.setActiveBrowserId(null);
240
+ this.frameState = null;
241
+ this.frameSrc = "";
242
+ }
243
+ this._lastFrameAt = Date.now();
244
+ };
245
+ await websocket.on("browser_viewer_frame", frameHandler);
246
+ this._frameOff = () => websocket.off("browser_viewer_frame", frameHandler);
247
+ }
248
+ if (!this._stateOff) {
249
+ const stateHandler = ({ data }) => {
250
+ if (data?.context_id !== this.contextId) return;
251
+ this.browsers = data.browsers || [];
252
+ this.setActiveBrowserId(data.last_interacted_browser_id || this.firstBrowserId());
253
+ this.queueViewportSync(true);
254
+ };
255
+ await websocket.on("browser_viewer_state", stateHandler);
256
+ this._stateOff = () => websocket.off("browser_viewer_state", stateHandler);
257
+ }
258
+ },
259
+
260
+ async command(command, extra = {}) {
261
+ this.error = "";
262
+ const previousActiveBrowserId = this.activeBrowserId;
263
+ try {
264
+ const response = await websocket.request(
265
+ "browser_viewer_command",
266
+ {
267
+ context_id: this.contextId,
268
+ browser_id: this.activeBrowserId,
269
+ command,
270
+ ...extra,
271
+ },
272
+ { timeoutMs: 20000 },
273
+ );
274
+ const data = firstOk(response);
275
+ this.browsers = data.browsers || this.browsers;
276
+ const result = data.result || {};
277
+ this.setActiveBrowserId(
278
+ result.id
279
+ || result.state?.id
280
+ || result.last_interacted_browser_id
281
+ || data.last_interacted_browser_id
282
+ || this.firstBrowserId()
283
+ );
284
+ if (!this.activeBrowserId) {
285
+ this.frameState = null;
286
+ this.frameSrc = "";
287
+ }
288
+ if (result.state?.currentUrl || result.currentUrl) {
289
+ this.address = result.state?.currentUrl || result.currentUrl;
290
+ }
291
+ const activeChanged = this.activeBrowserId && this.activeBrowserId !== previousActiveBrowserId;
292
+ if ((command === "open" || command === "close" || activeChanged) && this.contextId && this.activeBrowserId) {
293
+ await this.connectViewer();
294
+ }
295
+ this.queueViewportSync(true);
296
+ } catch (error) {
297
+ this.error = error instanceof Error ? error.message : String(error);
298
+ }
299
+ },
300
+
301
+ async go() {
302
+ const url = String(this.address || "").trim();
303
+ if (!url) return;
304
+ this.addressFocused = false;
305
+ globalThis.document?.activeElement?.blur?.();
306
+ if (this.activeBrowserId) {
307
+ await this.command("navigate", { url });
308
+ } else {
309
+ await this.command("open", { url });
310
+ }
311
+ },
312
+
313
+ onAddressFocus() {
314
+ this.addressFocused = true;
315
+ },
316
+
317
+ onAddressBlur() {
318
+ this.addressFocused = false;
319
+ if (this.frameState?.currentUrl && !String(this.address || "").trim()) {
320
+ this.address = this.frameState.currentUrl;
321
+ }
322
+ },
323
+
324
+ async selectBrowser(id) {
325
+ if (String(id || "").trim() === "") {
326
+ await this.command("open", { url: "about:blank" });
327
+ return;
328
+ }
329
+ this.setActiveBrowserId(id);
330
+ if (this.contextId) {
331
+ await this.connectViewer();
332
+ }
333
+ },
334
+
335
+ firstBrowserId() {
336
+ const first = Array.isArray(this.browsers) ? this.browsers[0] : null;
337
+ return first?.id || null;
338
+ },
339
+
340
+ setActiveBrowserId(id) {
341
+ const previous = this.activeBrowserId;
342
+ const numeric = Number(id) || null;
343
+ const exists = !numeric || !Array.isArray(this.browsers) || this.browsers.some((browser) => Number(browser.id) === numeric);
344
+ this.activeBrowserId = exists ? numeric : null;
345
+ if (this.activeBrowserId !== previous) {
346
+ this._lastViewportKey = "";
347
+ }
348
+ },
349
+
350
+ pointerCoordinatesFor(event, element = null) {
351
+ const target = element || event?.currentTarget;
352
+ if (!target) return null;
353
+ const rect = target.getBoundingClientRect();
354
+ const naturalWidth = target.naturalWidth || rect.width;
355
+ const naturalHeight = target.naturalHeight || rect.height;
356
+ return {
357
+ x: ((event.clientX - rect.left) / Math.max(1, rect.width)) * naturalWidth,
358
+ y: ((event.clientY - rect.top) / Math.max(1, rect.height)) * naturalHeight,
359
+ };
360
+ },
361
+
362
+ currentViewportSize() {
363
+ const stage = this._stageElement;
364
+ if (!stage) return null;
365
+ const width = Math.floor(stage.clientWidth || 0);
366
+ const height = Math.floor(stage.clientHeight || 0);
367
+ if (width < 80 || height < 80) return null;
368
+ return {
369
+ width: Math.max(320, width),
370
+ height: Math.max(200, height),
371
+ };
372
+ },
373
+
374
+ queueViewportSync(force = false) {
375
+ if (this._viewportSyncTimer) {
376
+ globalThis.clearTimeout(this._viewportSyncTimer);
377
+ }
378
+ this._viewportSyncTimer = globalThis.setTimeout(() => {
379
+ this._viewportSyncTimer = null;
380
+ void this.syncViewport(force);
381
+ }, force ? 0 : 80);
382
+ },
383
+
384
+ async syncViewport(force = false) {
385
+ if (!this.contextId || !this.activeBrowserId) return;
386
+ const viewport = this.currentViewportSize();
387
+ if (!viewport) return;
388
+ const key = `${this.activeBrowserId}:${viewport.width}x${viewport.height}`;
389
+ if (!force && this._lastViewportKey === key) return;
390
+ try {
391
+ await websocket.emit("browser_viewer_input", {
392
+ context_id: this.contextId,
393
+ browser_id: this.activeBrowserId,
394
+ input_type: "viewport",
395
+ width: viewport.width,
396
+ height: viewport.height,
397
+ });
398
+ this._lastViewportKey = key;
399
+ } catch (error) {
400
+ this._lastViewportKey = "";
401
+ console.warn("Browser viewport sync failed", error);
402
+ }
403
+ },
404
+
405
+ async sendMouse(eventType, event) {
406
+ if (!this.activeBrowserId || !event?.currentTarget) return;
407
+ const pointer = this.pointerCoordinatesFor(event);
408
+ if (!pointer) return;
409
+ await websocket.emit("browser_viewer_input", {
410
+ context_id: this.contextId,
411
+ browser_id: this.activeBrowserId,
412
+ input_type: "mouse",
413
+ event_type: eventType,
414
+ x: pointer.x,
415
+ y: pointer.y,
416
+ button: "left",
417
+ });
418
+ },
419
+
420
+ async sendWheel(event) {
421
+ if (!this.activeBrowserId || !event) return;
422
+ const image = event.currentTarget?.querySelector?.(".browser-frame") || event.target?.closest?.(".browser-frame");
423
+ const pointer = this.pointerCoordinatesFor(event, image);
424
+ if (!pointer) return;
425
+ await websocket.emit("browser_viewer_input", {
426
+ context_id: this.contextId,
427
+ browser_id: this.activeBrowserId,
428
+ input_type: "wheel",
429
+ x: pointer.x,
430
+ y: pointer.y,
431
+ delta_x: Number(event.deltaX || 0),
432
+ delta_y: Number(event.deltaY || 0),
433
+ });
434
+ },
435
+
436
+ async sendKey(event) {
437
+ if (!this.activeBrowserId) return;
438
+ if (event.ctrlKey || event.metaKey || event.altKey) return;
439
+ const editable = ["INPUT", "TEXTAREA", "SELECT"].includes(event.target?.tagName);
440
+ if (editable) return;
441
+ event.preventDefault();
442
+ const printable = event.key && event.key.length === 1;
443
+ await websocket.emit("browser_viewer_input", {
444
+ context_id: this.contextId,
445
+ browser_id: this.activeBrowserId,
446
+ input_type: "keyboard",
447
+ key: printable ? "" : event.key,
448
+ text: printable ? event.key : "",
449
+ });
450
+ },
451
+
452
+ async cleanup() {
453
+ if (this.contextId) {
454
+ try {
455
+ await websocket.emit("browser_viewer_unsubscribe", { context_id: this.contextId });
456
+ } catch {}
457
+ }
458
+ this._frameOff?.();
459
+ this._stateOff?.();
460
+ this._frameOff = null;
461
+ this._stateOff = null;
462
+ this._floatingCleanup?.();
463
+ this._floatingCleanup = null;
464
+ this._stageResizeObserver?.disconnect?.();
465
+ this._stageResizeObserver = null;
466
+ this._stageElement = null;
467
+ if (this._viewportSyncTimer) {
468
+ globalThis.clearTimeout(this._viewportSyncTimer);
469
+ this._viewportSyncTimer = null;
470
+ }
471
+ this._lastViewportKey = "";
472
+ this.extensionMenuOpen = false;
473
+ this.extensionActionLoading = false;
474
+ this.connected = false;
475
+ },
476
+
477
+ setupFloatingModal(element = null) {
478
+ this._floatingCleanup?.();
479
+ const root = element || globalThis.document?.querySelector(".browser-panel");
480
+ const modal = root?.closest?.(".modal");
481
+ const inner = modal?.querySelector?.(".modal-inner");
482
+ const body = modal?.querySelector?.(".modal-bd");
483
+ const header = modal?.querySelector?.(".modal-header");
484
+ const stage = root?.querySelector?.(".browser-stage");
485
+ if (!modal || !inner || !header) return;
486
+ modal.classList.add("modal-floating");
487
+ inner.classList.add("browser-modal");
488
+ body?.classList?.add("browser-modal-body");
489
+ this._stageElement = stage || null;
490
+
491
+ const rect = inner.getBoundingClientRect();
492
+ inner.style.left = `${Math.max(8, rect.left)}px`;
493
+ inner.style.top = `${Math.max(8, rect.top)}px`;
494
+ inner.style.transform = "none";
495
+
496
+ let drag = null;
497
+ let resizeObserver = null;
498
+ const viewportGap = 8;
499
+ const clampPosition = (left, top) => {
500
+ const bounds = inner.getBoundingClientRect();
501
+ const maxLeft = Math.max(viewportGap, globalThis.innerWidth - bounds.width - viewportGap);
502
+ const maxTop = Math.max(viewportGap, globalThis.innerHeight - bounds.height - viewportGap);
503
+ return {
504
+ left: Math.min(Math.max(viewportGap, left), maxLeft),
505
+ top: Math.min(Math.max(viewportGap, top), maxTop),
506
+ };
507
+ };
508
+ const clampGeometry = () => {
509
+ const bounds = inner.getBoundingClientRect();
510
+ const left = Math.max(viewportGap, bounds.left);
511
+ const top = Math.max(viewportGap, bounds.top);
512
+ const maxWidth = Math.max(320, globalThis.innerWidth - viewportGap * 2);
513
+ const maxHeight = Math.max(300, globalThis.innerHeight - viewportGap * 2);
514
+ if (bounds.width > maxWidth) {
515
+ inner.style.width = `${maxWidth}px`;
516
+ }
517
+ if (bounds.height > maxHeight) {
518
+ inner.style.height = `${maxHeight}px`;
519
+ }
520
+ const next = clampPosition(left, top);
521
+ inner.style.left = `${next.left}px`;
522
+ inner.style.top = `${next.top}px`;
523
+ inner.style.maxWidth = `${Math.max(320, globalThis.innerWidth - next.left - viewportGap)}px`;
524
+ inner.style.maxHeight = `${Math.max(300, globalThis.innerHeight - next.top - viewportGap)}px`;
525
+ this.queueViewportSync();
526
+ };
527
+ clampGeometry();
528
+ globalThis.addEventListener("resize", clampGeometry);
529
+ if (globalThis.ResizeObserver) {
530
+ resizeObserver = new ResizeObserver(clampGeometry);
531
+ resizeObserver.observe(inner);
532
+ if (stage) {
533
+ this._stageResizeObserver?.disconnect?.();
534
+ this._stageResizeObserver = new ResizeObserver(() => this.queueViewportSync());
535
+ this._stageResizeObserver.observe(stage);
536
+ }
537
+ }
538
+ globalThis.requestAnimationFrame(() => this.queueViewportSync(true));
539
+
540
+ const onPointerMove = (event) => {
541
+ if (!drag) return;
542
+ const next = clampPosition(
543
+ drag.left + event.clientX - drag.x,
544
+ drag.top + event.clientY - drag.y,
545
+ );
546
+ inner.style.left = `${next.left}px`;
547
+ inner.style.top = `${next.top}px`;
548
+ clampGeometry();
549
+ };
550
+ const onPointerUp = () => {
551
+ drag = null;
552
+ globalThis.removeEventListener("pointermove", onPointerMove);
553
+ globalThis.removeEventListener("pointerup", onPointerUp);
554
+ try {
555
+ header.releasePointerCapture?.(header.__browserPanelPointerId || 0);
556
+ } catch {}
557
+ };
558
+ const onPointerDown = (event) => {
559
+ if (event.button !== 0) return;
560
+ if (event.target?.closest?.("button, input, select, textarea, a")) return;
561
+ const current = inner.getBoundingClientRect();
562
+ drag = {
563
+ x: event.clientX,
564
+ y: event.clientY,
565
+ left: current.left,
566
+ top: current.top,
567
+ };
568
+ header.__browserPanelPointerId = event.pointerId;
569
+ header.setPointerCapture?.(event.pointerId);
570
+ globalThis.addEventListener("pointermove", onPointerMove);
571
+ globalThis.addEventListener("pointerup", onPointerUp);
572
+ event.preventDefault();
573
+ };
574
+ header.addEventListener("pointerdown", onPointerDown);
575
+
576
+ this._floatingCleanup = () => {
577
+ header.removeEventListener("pointerdown", onPointerDown);
578
+ globalThis.removeEventListener("pointermove", onPointerMove);
579
+ globalThis.removeEventListener("pointerup", onPointerUp);
580
+ globalThis.removeEventListener("resize", clampGeometry);
581
+ resizeObserver?.disconnect?.();
582
+ this._stageResizeObserver?.disconnect?.();
583
+ this._stageResizeObserver = null;
584
+ };
585
+ },
586
+
587
+ get activeTitle() {
588
+ return this.frameState?.title || "Browser";
589
+ },
590
+
591
+ get activeUrl() {
592
+ return this.frameState?.currentUrl || this.address || "about:blank";
593
+ },
594
+};
595
+
596
+export const store = createStore("browserPage", model);
plugins/_browser/webui/config.html
new
+225
@@ -0,0 +1,225 @@
1
+<html>
2
+<head>
3
+ <title>Browser Settings</title>
4
+ <script type="module">
5
+ import { store } from "/plugins/_browser/webui/browser-config-store.js";
6
+ </script>
7
+</head>
8
+
9
+<body>
10
+ <div x-data>
11
+ <template x-if="$store.browserConfig && config">
12
+ <div
13
+ class="browser-config-sections"
14
+ x-init="$store.browserConfig.init(config)"
15
+ x-effect="$store.browserConfig.bindConfig(config)"
16
+ x-destroy="$store.browserConfig.cleanup()"
17
+ >
18
+ <div class="browser-config-card">
19
+ <div class="section-title">Browser Model Preset</div>
20
+ <div class="section-description">
21
+ Choose an optional Model Configuration preset for Browser-owned model helpers. Leave it
22
+ on default to follow the effective Main Model.
23
+ </div>
24
+
25
+ <div class="field">
26
+ <div class="field-label">
27
+ <div class="field-title">Preset</div>
28
+ <div class="field-description" x-text="$store.browserConfig.selectedPresetSummary()"></div>
29
+ </div>
30
+ <div class="field-control">
31
+ <select x-model="config.model_preset" :disabled="$store.browserConfig.presetsLoading">
32
+ <option value="">Default Main Model</option>
33
+ <template x-for="preset in $store.browserConfig.presetOptions()" :key="preset.name">
34
+ <option :value="preset.name" x-text="preset.label"></option>
35
+ </template>
36
+ </select>
37
+ </div>
38
+ </div>
39
+
40
+ <div class="browser-config-note" x-show="$store.browserConfig.presetsLoading">
41
+ <span class="material-symbols-outlined spinning">progress_activity</span>
42
+ <span>Loading model presets...</span>
43
+ </div>
44
+
45
+ <div class="browser-config-warning" x-show="$store.browserConfig.selectedPresetMissing()">
46
+ <span class="material-symbols-outlined">warning</span>
47
+ <span>The saved preset is missing. Browser will use the effective Main Model until you choose another preset.</span>
48
+ </div>
49
+
50
+ <div class="browser-config-note" x-show="$store.browserConfig.presetsError">
51
+ <span class="material-symbols-outlined">error</span>
52
+ <span x-text="$store.browserConfig.presetsError"></span>
53
+ </div>
54
+
55
+ <div class="browser-config-actions">
56
+ <button type="button" class="btn btn-field" @click="$store.browserConfig.openPresets()">
57
+ <span class="material-symbols-outlined">tune</span>
58
+ <span>Edit Presets</span>
59
+ </button>
60
+ </div>
61
+ </div>
62
+
63
+ <div class="browser-config-card">
64
+ <div class="section-title">Chrome Extensions</div>
65
+ <div class="section-description">
66
+ Load unpacked Chromium extensions into the Browser tool. When extensions are active,
67
+ Browser switches from Playwright's lightweight headless shell to bundled Chromium so
68
+ the extensions can actually load.
69
+ </div>
70
+
71
+ <div class="browser-config-warning">
72
+ <span class="material-symbols-outlined">warning</span>
73
+ <span>
74
+ Browser extensions run inside the Docker browser sandbox, but malicious or buggy
75
+ extensions can still damage that sandboxed environment. Install only extensions you
76
+ trust and keep permissions as small as possible.
77
+ </span>
78
+ </div>
79
+
80
+ <div class="field">
81
+ <div class="field-label">
82
+ <div class="field-title">Enable extensions</div>
83
+ <div class="field-description">
84
+ Turn this on only when you have unpacked extension folders ready. Saving changes
85
+ restarts active Browser sessions so the new launch mode applies immediately.
86
+ </div>
87
+ </div>
88
+ <div class="field-control">
89
+ <label class="toggle">
90
+ <input type="checkbox" x-model="config.extensions_enabled" />
91
+ <span class="toggler"></span>
92
+ </label>
93
+ </div>
94
+ </div>
95
+
96
+ <div class="field">
97
+ <div class="field-label">
98
+ <div class="field-title">Extension directories</div>
99
+ <div class="field-description">
100
+ One unpacked extension directory per line. Use paths that are visible inside the
101
+ runtime environment itself, especially when Agent Zero is running in Docker.
102
+ </div>
103
+ </div>
104
+ <div class="field-control">
105
+ <textarea
106
+ :value="$store.browserConfig.extensionPathsText"
107
+ @input="$store.browserConfig.setExtensionPathsText($event.target.value)"
108
+ rows="6"
109
+ placeholder="/a0/usr/browser-extensions/my-extension"
110
+ ></textarea>
111
+ </div>
112
+ </div>
113
+
114
+ <div class="browser-config-note">
115
+ <span class="material-symbols-outlined">info</span>
116
+ <span>
117
+ This first version supports unpacked extension folders only. Chrome Web Store installs
118
+ and `.crx` files are out of scope for now.
119
+ </span>
120
+ </div>
121
+
122
+ <div class="browser-config-note">
123
+ <span class="material-symbols-outlined">deployed_code</span>
124
+ <span>
125
+ Playwright currently requires a persistent Chromium context for extension loading, so
126
+ Browser stays in its faster headless-shell mode until valid extension folders are both
127
+ configured and enabled.
128
+ </span>
129
+ </div>
130
+
131
+ <div class="browser-config-pill-row">
132
+ <span class="browser-config-pill" x-text="$store.browserConfig.pathCountLabel()"></span>
133
+ <span class="browser-config-pill tone-active" x-show="$store.browserConfig.extensionModeReady()">
134
+ Extension mode ready
135
+ </span>
136
+ </div>
137
+ </div>
138
+ </div>
139
+ </template>
140
+ </div>
141
+
142
+ <style>
143
+ .browser-config-sections {
144
+ display: flex;
145
+ flex-direction: column;
146
+ gap: 16px;
147
+ }
148
+
149
+ .browser-config-card {
150
+ display: flex;
151
+ flex-direction: column;
152
+ gap: 14px;
153
+ padding: 16px;
154
+ border: 1px solid var(--color-border);
155
+ border-radius: 8px;
156
+ }
157
+
158
+ .browser-config-card textarea {
159
+ min-height: 132px;
160
+ resize: vertical;
161
+ font-family: var(--font-family-monospace, monospace);
162
+ }
163
+
164
+ .browser-config-actions {
165
+ display: flex;
166
+ flex-wrap: wrap;
167
+ gap: 8px;
168
+ }
169
+
170
+ .browser-config-note {
171
+ display: flex;
172
+ align-items: flex-start;
173
+ gap: 8px;
174
+ padding: 10px 12px;
175
+ border-radius: 8px;
176
+ background: color-mix(in srgb, var(--color-panel) 82%, transparent);
177
+ color: var(--color-text-secondary);
178
+ font-size: var(--font-size-small);
179
+ }
180
+
181
+ .browser-config-warning {
182
+ display: flex;
183
+ align-items: flex-start;
184
+ gap: 9px;
185
+ padding: 11px 12px;
186
+ border: 1px solid color-mix(in srgb, #d97706 44%, var(--color-border));
187
+ border-radius: 8px;
188
+ background: color-mix(in srgb, #d97706 14%, var(--color-background));
189
+ color: color-mix(in srgb, var(--color-text) 86%, #92400e);
190
+ font-size: var(--font-size-small);
191
+ line-height: 1.4;
192
+ }
193
+
194
+ .browser-config-warning .material-symbols-outlined {
195
+ color: #b45309;
196
+ font-size: 20px;
197
+ }
198
+
199
+ .browser-config-pill-row {
200
+ display: flex;
201
+ flex-wrap: wrap;
202
+ gap: 8px;
203
+ }
204
+
205
+ .browser-config-pill {
206
+ display: inline-flex;
207
+ align-items: center;
208
+ gap: 6px;
209
+ min-height: 28px;
210
+ padding: 0 10px;
211
+ border-radius: 999px;
212
+ border: 1px solid color-mix(in srgb, var(--color-border) 70%, transparent);
213
+ background: color-mix(in srgb, var(--color-panel) 88%, transparent);
214
+ font-size: 0.78rem;
215
+ color: var(--color-text-secondary);
216
+ }
217
+
218
+ .browser-config-pill.tone-active {
219
+ color: #1b5e20;
220
+ border-color: rgba(27, 94, 32, 0.18);
221
+ background: rgba(46, 125, 50, 0.12);
222
+ }
223
+ </style>
224
+</body>
225
+</html>
plugins/_browser/webui/main.html
new
+556
@@ -0,0 +1,556 @@
1
+<html class="browser-modal">
2
+<head>
3
+ <title>Browser</title>
4
+ <script type="module">
5
+ import { store } from "/plugins/_browser/webui/browser-store.js";
6
+ </script>
7
+</head>
8
+<body class="browser-modal-body">
9
+ <div x-data>
10
+ <template x-if="$store.browserPage">
11
+ <div
12
+ class="browser-panel"
13
+ x-create="$store.browserPage.onOpen($el)"
14
+ x-destroy="$store.browserPage.cleanup()"
15
+ @keydown.window="$store.browserPage.sendKey($event)"
16
+ >
17
+ <div class="browser-meta">
18
+ <div class="browser-meta-top">
19
+ <div class="browser-titleline">
20
+ <span
21
+ class="browser-live-dot"
22
+ :class="{ active: $store.browserPage.connected && $store.browserPage.frameSrc }"
23
+ ></span>
24
+ <span class="browser-title">Browser</span>
25
+ <span class="browser-id" x-show="$store.browserPage.activeBrowserId" x-text="'#' + $store.browserPage.activeBrowserId"></span>
26
+ </div>
27
+ <div class="browser-session-controls">
28
+ <select class="browser-select" x-model="$store.browserPage.activeBrowserId" @change="$store.browserPage.selectBrowser($event.target.value)">
29
+ <option value="">New Browser</option>
30
+ <template x-for="browser in $store.browserPage.browsers" :key="browser.id">
31
+ <option :value="browser.id" x-text="'#' + browser.id + ' ' + (browser.title || browser.currentUrl || 'about:blank')"></option>
32
+ </template>
33
+ </select>
34
+ <div
35
+ class="browser-extension-menu"
36
+ @click.outside="$store.browserPage.closeExtensionsMenu()"
37
+ @keydown.escape.window="$store.browserPage.closeExtensionsMenu()"
38
+ >
39
+ <button
40
+ type="button"
41
+ class="btn btn-icon-action browser-extensions"
42
+ title="Browser extensions"
43
+ aria-label="Browser extensions"
44
+ @click.stop="$store.browserPage.toggleExtensionsMenu()"
45
+ :aria-expanded="$store.browserPage.extensionMenuOpen.toString()"
46
+ :class="{ 'is-active': $store.browserPage.status?.extensions?.active }"
47
+ >
48
+ <span class="material-symbols-outlined">extension</span>
49
+ </button>
50
+ <div
51
+ class="browser-extension-dropdown"
52
+ x-show="$store.browserPage.extensionMenuOpen"
53
+ x-transition
54
+ style="display: none;"
55
+ >
56
+ <div class="browser-extension-warning">
57
+ <span class="material-symbols-outlined">warning</span>
58
+ <span>
59
+ Extensions run inside the Docker browser sandbox, but malicious or buggy extensions can still damage that environment. Review what you install.
60
+ </span>
61
+ </div>
62
+ <button type="button" class="dropdown-item" @click="$store.browserPage.createExtensionWithAgent()">
63
+ <span class="material-symbols-outlined">add_circle</span>
64
+ <span>+ Create New with A0</span>
65
+ </button>
66
+ <div class="browser-extension-url">
67
+ <label for="browser-extension-url">Chrome Web Store URL</label>
68
+ <input
69
+ id="browser-extension-url"
70
+ type="url"
71
+ x-model="$store.browserPage.extensionInstallUrl"
72
+ @keydown.enter.prevent="$store.browserPage.installExtensionFromUrl()"
73
+ placeholder="https://chromewebstore.google.com/detail/..."
74
+ />
75
+ <div class="browser-extension-url-actions">
76
+ <button
77
+ type="button"
78
+ class="btn btn-ok"
79
+ @click="$store.browserPage.installExtensionFromUrl()"
80
+ :disabled="$store.browserPage.extensionActionLoading"
81
+ >
82
+ <span class="material-symbols-outlined" x-text="$store.browserPage.extensionActionLoading ? 'progress_activity' : 'download'"></span>
83
+ <span>Install URL</span>
84
+ </button>
85
+ <button type="button" class="btn btn-field" @click="$store.browserPage.askAgentInstallExtension()">
86
+ <span class="material-symbols-outlined">psychology_alt</span>
87
+ <span>Ask A0</span>
88
+ </button>
89
+ </div>
90
+ </div>
91
+ <button type="button" class="dropdown-item" @click="$store.browserPage.openExtensionsFolder()">
92
+ <span class="material-symbols-outlined">folder_open</span>
93
+ <span>My Browser Extensions</span>
94
+ </button>
95
+ <button type="button" class="dropdown-item" @click="$store.browserPage.openExtensionsSettings()">
96
+ <span class="material-symbols-outlined">tune</span>
97
+ <span>Browser Extension Settings</span>
98
+ </button>
99
+ <div class="browser-extension-message" x-show="$store.browserPage.extensionActionMessage" x-text="$store.browserPage.extensionActionMessage"></div>
100
+ <div class="browser-extension-error" x-show="$store.browserPage.extensionActionError" x-text="$store.browserPage.extensionActionError"></div>
101
+ </div>
102
+ </div>
103
+ <button class="btn btn-icon-action browser-close" title="Close Browser" @click="$confirmClick($event, () => $store.browserPage.command('close'))" :disabled="!$store.browserPage.activeBrowserId">
104
+ <span class="material-symbols-outlined">close</span>
105
+ </button>
106
+ </div>
107
+ </div>
108
+ </div>
109
+
110
+ <div class="browser-toolbar">
111
+ <div class="browser-navigation">
112
+ <button class="btn btn-icon-action" title="Back" @click="$store.browserPage.command('back')" :disabled="!$store.browserPage.activeBrowserId">
113
+ <span class="material-symbols-outlined">arrow_back</span>
114
+ </button>
115
+ <button class="btn btn-icon-action" title="Forward" @click="$store.browserPage.command('forward')" :disabled="!$store.browserPage.activeBrowserId">
116
+ <span class="material-symbols-outlined">arrow_forward</span>
117
+ </button>
118
+ <button class="btn btn-icon-action" title="Reload" @click="$store.browserPage.command('reload')" :disabled="!$store.browserPage.activeBrowserId">
119
+ <span class="material-symbols-outlined">refresh</span>
120
+ </button>
121
+ </div>
122
+
123
+ <form class="browser-address-form" @submit.prevent="$store.browserPage.go()">
124
+ <span class="material-symbols-outlined browser-address-icon">language</span>
125
+ <input
126
+ class="browser-address"
127
+ x-model="$store.browserPage.address"
128
+ @focus="$store.browserPage.onAddressFocus()"
129
+ @blur="$store.browserPage.onAddressBlur()"
130
+ placeholder="https://example.com"
131
+ autocomplete="off"
132
+ />
133
+ </form>
134
+ </div>
135
+
136
+ <div class="browser-status" x-show="$store.browserPage.loading">
137
+ <span class="material-symbols-outlined spinning">progress_activity</span>
138
+ <span>Connecting browser...</span>
139
+ </div>
140
+ <div class="browser-error" x-show="$store.browserPage.error" x-text="$store.browserPage.error"></div>
141
+
142
+ <div
143
+ class="browser-stage"
144
+ tabindex="0"
145
+ @click="$el.focus()"
146
+ @wheel.prevent="$store.browserPage.sendWheel($event)"
147
+ >
148
+ <template x-if="$store.browserPage.frameSrc">
149
+ <img
150
+ class="browser-frame"
151
+ :src="$store.browserPage.frameSrc"
152
+ @click="$store.browserPage.sendMouse('click', $event)"
153
+ @mousemove.throttle.250ms="$store.browserPage.sendMouse('move', $event)"
154
+ draggable="false"
155
+ />
156
+ </template>
157
+ <template x-if="!$store.browserPage.frameSrc && !$store.browserPage.loading">
158
+ <div class="browser-empty">
159
+ <span class="material-symbols-outlined">captive_portal</span>
160
+ <button class="btn btn-field" @click="$store.browserPage.command('open', { url: 'about:blank' })">Open Browser</button>
161
+ </div>
162
+ </template>
163
+ </div>
164
+ </div>
165
+ </template>
166
+ </div>
167
+
168
+ <style>
169
+ .modal-inner.browser-modal {
170
+ box-sizing: border-box;
171
+ container-type: inline-size;
172
+ width: min(78vw, 1120px);
173
+ height: min(88vh, 900px);
174
+ min-width: min(320px, calc(100vw - 16px));
175
+ min-height: min(480px, calc(100vh - 16px));
176
+ max-width: calc(100vw - 16px);
177
+ max-height: calc(100vh - 16px);
178
+ resize: both;
179
+ border: 1px solid color-mix(in srgb, var(--color-border) 75%, transparent);
180
+ border-radius: 7px;
181
+ box-shadow: 0 18px 48px rgba(0, 0, 0, 0.32);
182
+ background: color-mix(in srgb, var(--color-background) 94%, #000 6%);
183
+ }
184
+
185
+ .modal.modal-floating {
186
+ pointer-events: none;
187
+ }
188
+
189
+ .modal.modal-floating .modal-inner {
190
+ pointer-events: auto;
191
+ }
192
+
193
+ .modal-inner.browser-modal .modal-header {
194
+ min-height: 34px;
195
+ padding: 0.35rem 0.75rem 0.35rem 1rem;
196
+ cursor: move;
197
+ user-select: none;
198
+ background: color-mix(in srgb, var(--color-background) 92%, #000 8%);
199
+ border-bottom: 1px solid color-mix(in srgb, var(--color-border) 70%, transparent);
200
+ }
201
+
202
+ .modal-inner.browser-modal .modal-title {
203
+ font-size: 0.95rem;
204
+ letter-spacing: 0;
205
+ }
206
+
207
+ .modal-inner.browser-modal .modal-close {
208
+ font-size: 1.35rem;
209
+ line-height: 1;
210
+ }
211
+
212
+ .modal-inner.browser-modal .modal-scroll {
213
+ flex: 1 1 auto;
214
+ min-height: 0;
215
+ overflow: hidden;
216
+ padding: 0;
217
+ }
218
+
219
+ .modal-inner.browser-modal .modal-bd.browser-modal-body {
220
+ box-sizing: border-box;
221
+ display: flex;
222
+ flex-direction: column;
223
+ height: 100%;
224
+ padding: 0;
225
+ min-height: 0;
226
+ }
227
+
228
+ .modal-inner.browser-modal .modal-bd.browser-modal-body > div[x-data] {
229
+ display: flex;
230
+ flex: 1 1 auto;
231
+ min-height: 0;
232
+ }
233
+
234
+ .browser-panel {
235
+ box-sizing: border-box;
236
+ display: flex;
237
+ flex: 1 1 auto;
238
+ flex-direction: column;
239
+ gap: 0;
240
+ height: 100%;
241
+ min-height: 0;
242
+ }
243
+
244
+ .browser-toolbar {
245
+ display: grid;
246
+ grid-template-columns: auto minmax(0, 1fr);
247
+ grid-template-areas: "nav address";
248
+ gap: 6px;
249
+ align-items: center;
250
+ padding: 7px 8px;
251
+ border-bottom: 1px solid color-mix(in srgb, var(--color-border) 70%, transparent);
252
+ background: color-mix(in srgb, var(--color-panel) 90%, transparent);
253
+ }
254
+
255
+ .browser-navigation {
256
+ grid-area: nav;
257
+ display: flex;
258
+ gap: 4px;
259
+ }
260
+
261
+ .browser-address-form {
262
+ grid-area: address;
263
+ min-width: 0;
264
+ position: relative;
265
+ margin: 0;
266
+ }
267
+
268
+ .browser-address-icon {
269
+ position: absolute;
270
+ left: 10px;
271
+ top: 50%;
272
+ transform: translateY(-50%);
273
+ font-size: 18px;
274
+ opacity: 0.58;
275
+ pointer-events: none;
276
+ }
277
+
278
+ .browser-address,
279
+ .browser-select {
280
+ width: 100%;
281
+ min-height: 32px;
282
+ padding: 5px 9px;
283
+ border-radius: 6px;
284
+ border: 1px solid color-mix(in srgb, var(--color-border) 72%, transparent);
285
+ background: var(--color-input);
286
+ color: var(--color-text);
287
+ font: inherit;
288
+ }
289
+
290
+ .browser-address {
291
+ padding-left: 34px;
292
+ }
293
+
294
+ .browser-select {
295
+ min-width: 0;
296
+ }
297
+
298
+ .browser-meta {
299
+ display: grid;
300
+ grid-template-columns: minmax(0, 1fr);
301
+ gap: 6px;
302
+ padding: 6px 10px;
303
+ border-bottom: 1px solid color-mix(in srgb, var(--color-border) 65%, transparent);
304
+ background: color-mix(in srgb, var(--color-panel) 82%, transparent);
305
+ }
306
+
307
+ .browser-meta-top {
308
+ display: grid;
309
+ grid-template-columns: minmax(0, 1fr) auto;
310
+ gap: 10px;
311
+ align-items: center;
312
+ }
313
+
314
+ .browser-titleline {
315
+ display: flex;
316
+ align-items: center;
317
+ gap: 8px;
318
+ min-width: 0;
319
+ }
320
+
321
+ .browser-session-controls {
322
+ display: flex;
323
+ align-items: center;
324
+ gap: 6px;
325
+ min-width: 0;
326
+ }
327
+
328
+ .browser-session-controls .browser-select {
329
+ width: min(320px, 52cqw);
330
+ }
331
+
332
+ .browser-session-controls .browser-extensions.is-active {
333
+ color: #2e7d32;
334
+ }
335
+
336
+ .browser-extension-menu {
337
+ position: relative;
338
+ display: flex;
339
+ flex: 0 0 auto;
340
+ }
341
+
342
+ .browser-extension-dropdown {
343
+ position: absolute;
344
+ top: calc(100% + 6px);
345
+ right: 0;
346
+ z-index: 40;
347
+ display: flex;
348
+ flex-direction: column;
349
+ gap: 7px;
350
+ width: min(360px, calc(100vw - 24px));
351
+ padding: 10px;
352
+ border: 1px solid color-mix(in srgb, var(--color-border) 78%, transparent);
353
+ border-radius: 7px;
354
+ background: var(--color-background);
355
+ box-shadow: 0 16px 38px rgba(0, 0, 0, 0.28);
356
+ }
357
+
358
+ .browser-extension-dropdown .dropdown-item {
359
+ display: flex;
360
+ align-items: center;
361
+ gap: 8px;
362
+ width: 100%;
363
+ min-height: 34px;
364
+ padding: 7px 9px;
365
+ border: 0;
366
+ border-radius: 6px;
367
+ background: transparent;
368
+ color: var(--color-text);
369
+ font-weight: 600;
370
+ text-align: left;
371
+ cursor: pointer;
372
+ }
373
+
374
+ .browser-extension-dropdown .dropdown-item:hover {
375
+ background: color-mix(in srgb, var(--color-panel) 82%, transparent);
376
+ }
377
+
378
+ .browser-extension-warning,
379
+ .browser-extension-message,
380
+ .browser-extension-error {
381
+ display: flex;
382
+ align-items: flex-start;
383
+ gap: 8px;
384
+ padding: 9px 10px;
385
+ border-radius: 7px;
386
+ font-size: 0.8rem;
387
+ line-height: 1.35;
388
+ }
389
+
390
+ .browser-extension-warning {
391
+ border: 1px solid color-mix(in srgb, #d97706 42%, var(--color-border));
392
+ background: color-mix(in srgb, #d97706 14%, var(--color-background));
393
+ color: color-mix(in srgb, var(--color-text) 86%, #92400e);
394
+ }
395
+
396
+ .browser-extension-warning .material-symbols-outlined {
397
+ color: #b45309;
398
+ font-size: 19px;
399
+ }
400
+
401
+ .browser-extension-url {
402
+ display: flex;
403
+ flex-direction: column;
404
+ gap: 7px;
405
+ padding: 8px;
406
+ border: 1px solid color-mix(in srgb, var(--color-border) 58%, transparent);
407
+ border-radius: 7px;
408
+ background: var(--color-panel);
409
+ }
410
+
411
+ .browser-extension-url label {
412
+ font-size: 0.76rem;
413
+ color: var(--color-text-secondary);
414
+ }
415
+
416
+ .browser-extension-url input {
417
+ min-width: 0;
418
+ min-height: 32px;
419
+ padding: 6px 8px;
420
+ border: 1px solid color-mix(in srgb, var(--color-border) 72%, transparent);
421
+ border-radius: 6px;
422
+ background: var(--color-input);
423
+ color: var(--color-text);
424
+ }
425
+
426
+ .browser-extension-url-actions {
427
+ display: flex;
428
+ flex-wrap: wrap;
429
+ gap: 7px;
430
+ }
431
+
432
+ .browser-extension-url-actions .btn {
433
+ display: inline-flex;
434
+ align-items: center;
435
+ gap: 6px;
436
+ min-height: 30px;
437
+ }
438
+
439
+ .browser-extension-message {
440
+ background: color-mix(in srgb, #15803d 12%, var(--color-background));
441
+ color: color-mix(in srgb, var(--color-text) 88%, #166534);
442
+ }
443
+
444
+ .browser-extension-error {
445
+ background: color-mix(in srgb, #be123c 12%, var(--color-background));
446
+ color: #9f1239;
447
+ }
448
+
449
+ .browser-live-dot {
450
+ width: 8px;
451
+ height: 8px;
452
+ border-radius: 50%;
453
+ background: #777;
454
+ flex: 0 0 auto;
455
+ }
456
+
457
+ .browser-live-dot.active {
458
+ background: #2e7d32;
459
+ box-shadow: 0 0 0 4px rgba(46, 125, 50, 0.13);
460
+ }
461
+
462
+ .browser-title {
463
+ overflow: hidden;
464
+ text-overflow: ellipsis;
465
+ white-space: nowrap;
466
+ }
467
+
468
+ .browser-title {
469
+ font-size: 0.9rem;
470
+ font-weight: 650;
471
+ }
472
+
473
+ .browser-id {
474
+ font-size: 0.78rem;
475
+ opacity: 0.68;
476
+ }
477
+
478
+ .browser-stage {
479
+ flex: 1 1 auto;
480
+ display: flex;
481
+ flex-direction: column;
482
+ min-height: 0;
483
+ overflow: auto;
484
+ background: #fff;
485
+ outline: none;
486
+ }
487
+
488
+ .browser-frame {
489
+ flex: 0 0 auto;
490
+ display: block;
491
+ width: 100%;
492
+ height: auto;
493
+ user-select: none;
494
+ background: #fff;
495
+ }
496
+
497
+ .browser-status,
498
+ .browser-error,
499
+ .browser-empty {
500
+ display: flex;
501
+ align-items: center;
502
+ gap: 8px;
503
+ min-height: 42px;
504
+ font-size: 0.88rem;
505
+ }
506
+
507
+ .browser-status,
508
+ .browser-error {
509
+ padding: 0 12px;
510
+ }
511
+
512
+ .browser-error {
513
+ color: #9f1239;
514
+ }
515
+
516
+ .browser-empty {
517
+ display: grid;
518
+ flex: 1 1 auto;
519
+ width: 100%;
520
+ min-height: 0;
521
+ justify-items: center;
522
+ align-content: center;
523
+ text-align: center;
524
+ padding: 24px;
525
+ color: var(--color-text);
526
+ background: var(--color-background);
527
+ }
528
+
529
+ @container (max-width: 460px) {
530
+ .browser-meta-top {
531
+ grid-template-columns: minmax(0, 1fr);
532
+ }
533
+
534
+ .browser-session-controls {
535
+ width: 100%;
536
+ }
537
+
538
+ .browser-session-controls .browser-select {
539
+ flex: 1 1 auto;
540
+ width: auto;
541
+ }
542
+
543
+ .browser-extension-dropdown {
544
+ right: 0;
545
+ left: auto;
546
+ width: min(296px, calc(100vw - 72px));
547
+ }
548
+
549
+ .browser-address,
550
+ .browser-select {
551
+ min-height: 34px;
552
+ }
553
+ }
554
+ </style>
555
+</body>
556
+</html>
plugins/_browser_agent/api/model_preset.py
deleted
-35
@@ -1,35 +0,0 @@
1
-from helpers.api import ApiHandler, Request, Response
2
-
3
-from plugins._browser_agent.helpers.model_preset import (
4
- get_browser_model_preset_name,
5
- save_browser_model_preset_name,
6
-)
7
-from plugins._model_config.helpers import model_config
8
-
9
-
10
-class ModelPreset(ApiHandler):
11
- async def process(self, input: dict, request: Request) -> dict | Response:
12
- action = str(input.get("action", "get") or "get").strip().lower()
13
-
14
- if action == "get":
15
- return {
16
- "ok": True,
17
- "preset_name": get_browser_model_preset_name(),
18
- }
19
-
20
- if action not in {"set", "clear"}:
21
- return Response(status=400, response=f"Unknown action: {action}")
22
-
23
- preset_name = ""
24
- if action == "set":
25
- preset_name = str(input.get("preset_name", "") or "").strip()
26
- if not preset_name:
27
- return Response(status=400, response="Missing preset_name")
28
- if not model_config.get_preset_by_name(preset_name):
29
- return Response(status=404, response=f"Preset '{preset_name}' not found")
30
-
31
- save_browser_model_preset_name(preset_name)
32
- return {
33
- "ok": True,
34
- "preset_name": preset_name,
35
- }
plugins/_browser_agent/api/status.py
deleted
-54
@@ -1,54 +0,0 @@
1
-import importlib.metadata
2
-
3
-from helpers.api import ApiHandler, Request, Response
4
-from plugins._browser_agent.helpers.model_preset import (
5
- get_browser_model_preset_options,
6
- resolve_browser_model_selection,
7
-)
8
-from plugins._browser_agent.helpers.playwright import (
9
- get_playwright_binary,
10
- get_playwright_cache_dir,
11
-)
12
-
13
-
14
-class Status(ApiHandler):
15
- async def process(self, input: dict, request: Request) -> dict | Response:
16
- selection = resolve_browser_model_selection()
17
- cfg = selection["config"]
18
- binary = get_playwright_binary()
19
-
20
- browser_use_ok = False
21
- browser_use_error = ""
22
- browser_use_version = ""
23
- try:
24
- import browser_use # noqa: F401
25
-
26
- browser_use_ok = True
27
- browser_use_version = importlib.metadata.version("browser-use")
28
- except Exception as e:
29
- browser_use_error = str(e)
30
-
31
- return {
32
- "plugin": "_browser_agent",
33
- "model_source": selection["source_label"],
34
- "model_source_kind": selection["source_kind"],
35
- "selected_preset_name": selection["selected_preset_name"],
36
- "preset_status": selection["preset_status"],
37
- "preset_warning": selection["warning"],
38
- "available_presets": get_browser_model_preset_options(),
39
- "model": {
40
- "provider": cfg.get("provider", ""),
41
- "name": cfg.get("name", ""),
42
- "vision": bool(cfg.get("vision", False)),
43
- },
44
- "playwright": {
45
- "cache_dir": get_playwright_cache_dir(),
46
- "binary_found": bool(binary),
47
- "binary_path": str(binary) if binary else "",
48
- },
49
- "browser_use": {
50
- "import_ok": browser_use_ok,
51
- "version": browser_use_version,
52
- "error": browser_use_error,
53
- },
54
- }
plugins/_browser_agent/assets/init_override.js
deleted
-246
@@ -1,246 +0,0 @@
1
-// open all shadow doms
2
-(function () {
3
- const originalAttachShadow = Element.prototype.attachShadow;
4
- Element.prototype.attachShadow = function attachShadow(options) {
5
- return originalAttachShadow.call(this, { ...options, mode: "open" });
6
- };
7
-})();
8
-
9
-// // Create a global bridge for iframe communication
10
-// (function() {
11
-// let elementCounter = 0;
12
-// const ignoredTags = [
13
-// "style",
14
-// "script",
15
-// "meta",
16
-// "link",
17
-// "svg",
18
-// "noscript",
19
-// "path",
20
-// ];
21
-
22
-// function isElementVisible(element) {
23
-// // Return true for non-element nodes
24
-// if (element.nodeType !== Node.ELEMENT_NODE) {
25
-// return true;
26
-// }
27
-
28
-// const computedStyle = window.getComputedStyle(element);
29
-
30
-// // Check if element is hidden via CSS
31
-// if (
32
-// computedStyle.display === "none" ||
33
-// computedStyle.visibility === "hidden" ||
34
-// computedStyle.opacity === "0"
35
-// ) {
36
-// return false;
37
-// }
38
-
39
-// // Check for hidden input type
40
-// if (element.tagName === "INPUT" && element.type === "hidden") {
41
-// return false;
42
-// }
43
-
44
-// // Check for hidden attribute
45
-// if (
46
-// element.hasAttribute("hidden") ||
47
-// element.getAttribute("aria-hidden") === "true"
48
-// ) {
49
-// return false;
50
-// }
51
-
52
-// return true;
53
-// }
54
-
55
-// function convertAttribute(tag, attr) {
56
-// let out = {
57
-// name: attr.name,
58
-// value: attr.value,
59
-// };
60
-
61
-// if (["srcset"].includes(out.name)) return null;
62
-// if (out.name.startsWith("data-") && out.name != "data-A0UID" && out.name != "data-a0-frame-id") return null;
63
-
64
-// if (tag === "img" && out.value.startsWith("data:")) out.value = "data...";
65
-
66
-// return out;
67
-// }
68
-
69
-// // This function will be available in all frames
70
-// window.__A0_extractFrameContent = function() {
71
-// // Get the current frame's DOM content
72
-// const extractContent = (node) => {
73
-// if (!node) return "";
74
-
75
-// let content = "";
76
-// const tagName = node.tagName ? node.tagName.toLowerCase() : "";
77
-
78
-// // Skip ignored tags
79
-// if (tagName && ignoredTags.includes(tagName)) {
80
-// return "";
81
-// }
82
-
83
-// if (node.nodeType === Node.ELEMENT_NODE) {
84
-// // Add unique ID to the actual DOM element
85
-// if (tagName) {
86
-// const uid = elementCounter++;
87
-// node.setAttribute("data-A0UID", uid);
88
-// }
89
-
90
-// content += `<${tagName}`;
91
-
92
-// // Add invisible attribute if element is not visible
93
-// if (!isElementVisible(node)) {
94
-// content += " invisible";
95
-// }
96
-
97
-// // Add attributes with conversion
98
-// for (let attr of node.attributes) {
99
-// const out = convertAttribute(tagName, attr);
100
-// if (out) content += ` ${out.name}="${out.value}"`;
101
-// }
102
-
103
-// if (tagName) {
104
-// content += ` selector="${node.getAttribute("data-A0UID")}"`;
105
-// }
106
-
107
-// content += ">";
108
-
109
-// // Handle shadow DOM
110
-// if (node.shadowRoot) {
111
-// content += "<!-- Shadow DOM Start -->";
112
-// for (let shadowChild of node.shadowRoot.childNodes) {
113
-// content += extractContent(shadowChild);
114
-// }
115
-// content += "<!-- Shadow DOM End -->";
116
-// }
117
-
118
-// // Handle child nodes
119
-// for (let child of node.childNodes) {
120
-// content += extractContent(child);
121
-// }
122
-
123
-// content += `</${tagName}>`;
124
-// } else if (node.nodeType === Node.TEXT_NODE) {
125
-// content += node.textContent;
126
-// } else if (node.nodeType === Node.COMMENT_NODE) {
127
-// content += `<!--${node.textContent}-->`;
128
-// }
129
-
130
-// return content;
131
-// };
132
-
133
-// return extractContent(document.documentElement);
134
-// };
135
-
136
-// // Setup message listener in each frame
137
-// window.addEventListener('message', function(event) {
138
-// if (event.data === 'A0_REQUEST_CONTENT') {
139
-// // Extract content and send it back to parent
140
-// const content = window.__A0_extractFrameContent();
141
-// // Use '*' as targetOrigin since we're in a controlled environment
142
-// window.parent.postMessage({
143
-// type: 'A0_FRAME_CONTENT',
144
-// content: content,
145
-// frameId: window.frameElement?.getAttribute('data-a0-frame-id')
146
-// }, '*');
147
-// }
148
-// });
149
-
150
-// // Function to extract content from all frames
151
-// window.__A0_extractAllFramesContent = async function(rootNode = document) {
152
-// let content = "";
153
-
154
-// // Extract content from current document
155
-// content += window.__A0_extractFrameContent();
156
-
157
-// // Find all iframes
158
-// const iframes = rootNode.getElementsByTagName('iframe');
159
-
160
-// // Create a map to store frame contents
161
-// const frameContents = new Map();
162
-
163
-// // Setup promise for each iframe
164
-// const framePromises = Array.from(iframes).map((iframe) => {
165
-// return new Promise((resolve) => {
166
-// const frameId = 'frame_' + Math.random().toString(36).substr(2, 9);
167
-// iframe.setAttribute('data-a0-frame-id', frameId);
168
-
169
-// // Setup one-time message listener for this specific frame
170
-// const listener = function(event) {
171
-// if (event.data?.type === 'A0_FRAME_CONTENT' &&
172
-// event.data?.frameId === frameId) {
173
-// frameContents.set(frameId, event.data.content);
174
-// window.removeEventListener('message', listener);
175
-// resolve();
176
-// }
177
-// };
178
-// window.addEventListener('message', listener);
179
-
180
-// // Request content from frame
181
-// iframe.contentWindow.postMessage('A0_REQUEST_CONTENT', '*');
182
-
183
-// // Timeout after 2 seconds
184
-// setTimeout(resolve, 2000);
185
-// });
186
-// });
187
-
188
-// // Wait for all frames to respond or timeout
189
-// await Promise.all(framePromises);
190
-
191
-// // Add frame contents in order
192
-// for (let iframe of iframes) {
193
-// const frameId = iframe.getAttribute('data-a0-frame-id');
194
-// const frameContent = frameContents.get(frameId);
195
-// if (frameContent) {
196
-// content += `<!-- IFrame ${iframe.src || 'unnamed'} Content Start -->`;
197
-// content += frameContent;
198
-// content += `<!-- IFrame Content End -->`;
199
-// }
200
-// }
201
-
202
-// return content;
203
-// };
204
-// })();
205
-
206
-// // override iframe creation to inject our script into them
207
-// (function() {
208
-// // Store the original createElement to use for iframe creation
209
-// const originalCreateElement = document.createElement;
210
-
211
-// // Override createElement to catch iframe creation
212
-// document.createElement = function(tagName, options) {
213
-// const element = originalCreateElement.call(document, tagName, options);
214
-// if (tagName.toLowerCase() === 'iframe') {
215
-// // Override the src setter
216
-// const originalSrcSetter = Object.getOwnPropertyDescriptor(HTMLIFrameElement.prototype, 'src').set;
217
-// Object.defineProperty(element, 'src', {
218
-// set: function(value) {
219
-// // Call original setter
220
-// originalSrcSetter.call(this, value);
221
-
222
-// // Wait for load and inject our script
223
-// this.addEventListener('load', () => {
224
-// try {
225
-// // Try to inject our script into the iframe
226
-// const iframeDoc = this.contentWindow.document;
227
-// const script = iframeDoc.createElement('script');
228
-// script.textContent = `
229
-// // Make iframe accessible
230
-// document.domain = document.domain;
231
-// // Disable security policies if possible
232
-// if (window.SecurityPolicyViolationEvent) {
233
-// window.SecurityPolicyViolationEvent = undefined;
234
-// }
235
-// `;
236
-// iframeDoc.head.appendChild(script);
237
-// } catch(e) {
238
-// console.warn('Could not inject into iframe:', e);
239
-// }
240
-// }, { once: true });
241
-// }
242
-// });
243
-// }
244
-// return element;
245
-// };
246
-// })();
plugins/_browser_agent/extensions/python/_functions/agent/Agent/get_browser_model/start/_10_browser_agent.py
deleted
-7
@@ -1,7 +0,0 @@
1
-from helpers.extension import Extension
2
-from plugins._browser_agent.helpers.browser_llm import build_browser_model_for_agent
3
-
4
-class BrowserModelProvider(Extension):
5
- def execute(self, data: dict = {}, **kwargs):
6
- if self.agent:
7
- data["result"] = build_browser_model_for_agent(self.agent)
plugins/_browser_agent/extensions/webui/get_message_handler/browser-agent-handler.js
deleted
-54
@@ -1,54 +0,0 @@
1
-import {
2
- createActionButton,
3
- copyToClipboard,
4
-} from "/components/messages/action-buttons/simple-action-buttons.js";
5
-import { store as stepDetailStore } from "/components/modals/process-step-detail/step-detail-store.js";
6
-import { store as speechStore } from "/components/chat/speech/speech-store.js";
7
-import {
8
- buildDetailPayload,
9
- cleanStepTitle,
10
- drawProcessStep,
11
-} from "/js/messages.js";
12
-
13
-export default async function registerBrowserAgentHandler(extData) {
14
- if (extData?.type === "browser") {
15
- extData.handler = drawMessageBrowserAgent;
16
- }
17
-}
18
-
19
-function drawMessageBrowserAgent({
20
- id,
21
- type,
22
- heading,
23
- content,
24
- kvps,
25
- timestamp,
26
- agentno = 0,
27
- ...additional
28
-}) {
29
- const title = cleanStepTitle(heading);
30
- const displayKvps = { ...kvps };
31
- const answerText = String(kvps?.answer ?? "");
32
- const actionButtons = answerText.trim()
33
- ? [
34
- createActionButton("detail", "", () =>
35
- stepDetailStore.showStepDetail(
36
- buildDetailPayload(arguments[0], { headerLabels: [] }),
37
- ),
38
- ),
39
- createActionButton("speak", "", () => speechStore.speak(answerText)),
40
- createActionButton("copy", "", () => copyToClipboard(answerText)),
41
- ].filter(Boolean)
42
- : [];
43
-
44
- return drawProcessStep({
45
- id,
46
- title,
47
- code: "WWW",
48
- classes: undefined,
49
- kvps: displayKvps,
50
- content,
51
- actionButtons,
52
- log: arguments[0],
53
- });
54
-}
plugins/_browser_agent/extensions/webui/get_tool_message_handler/browser-tool-handler.js
deleted
-15
@@ -1,15 +0,0 @@
1
-import { drawMessageToolSimple } from "/js/messages.js";
2
-
3
-/**
4
- * Registers the browser_agent tool message handler to set the custom badge.
5
- * @param {object} extData
6
- */
7
-export default async function registerBrowserToolHandler(extData) {
8
- if (extData?.tool_name === "browser_agent") {
9
- extData.handler = drawBrowserTool;
10
- }
11
-}
12
-
13
-function drawBrowserTool(args) {
14
- return drawMessageToolSimple({ ...args, code: "WWW" });
15
-}
plugins/_browser_agent/helpers/__init__.py
deleted
-1
@@ -1 +0,0 @@
1
-# Built-in browser agent helpers.
plugins/_browser_agent/helpers/browser_llm.py
deleted
-162
@@ -1,162 +0,0 @@
1
-from typing import Any, List, Optional
2
-import litellm
3
-from litellm import acompletion
4
-from langchain_core.callbacks.manager import CallbackManagerForLLMRun
5
-from langchain_core.messages import BaseMessage
6
-
7
-import models
8
-from browser_use.llm import ChatGoogle, ChatOpenRouter
9
-
10
-from plugins._browser_agent.helpers import browser_use_monkeypatch
11
-from plugins._browser_agent.helpers import model_preset
12
-from plugins._browser_agent.helpers import browser_use_openrouter_compat
13
-from plugins._browser_agent.helpers import browser_use_output_sanitize
14
-
15
-
16
-_BROWSER_USE_PATCHED = False
17
-
18
-
19
-def apply_browser_use_patches() -> None:
20
- global _BROWSER_USE_PATCHED
21
- if _BROWSER_USE_PATCHED:
22
- return
23
-
24
- browser_use_monkeypatch.apply()
25
- litellm.modify_params = True
26
- _BROWSER_USE_PATCHED = True
27
-
28
-
29
-class AsyncAIChatReplacement:
30
- class _Completions:
31
- def __init__(self, wrapper):
32
- self._wrapper = wrapper
33
-
34
- async def create(self, *args, **kwargs):
35
- return await self._wrapper._acall(*args, **kwargs)
36
-
37
- class _Chat:
38
- def __init__(self, wrapper):
39
- self.completions = AsyncAIChatReplacement._Completions(wrapper)
40
-
41
- def __init__(self, wrapper, *args, **kwargs):
42
- self._wrapper = wrapper
43
- self.chat = AsyncAIChatReplacement._Chat(wrapper)
44
-
45
-
46
-class BrowserCompatibleChatWrapper(ChatOpenRouter):
47
- """
48
- A wrapper for browser agent that can filter/sanitize messages
49
- before sending them to the LLM.
50
- """
51
-
52
- def __init__(self, *args, **kwargs):
53
- apply_browser_use_patches()
54
- models.turn_off_logging()
55
- self._wrapper = models.LiteLLMChatWrapper(*args, **kwargs)
56
- self.model = self._wrapper.model_name
57
- self.kwargs = self._wrapper.kwargs
58
-
59
- @property
60
- def model_name(self) -> str:
61
- return self._wrapper.model_name
62
-
63
- @property
64
- def provider(self) -> str:
65
- return self._wrapper.provider
66
-
67
- def get_client(self, *args, **kwargs): # type: ignore
68
- return AsyncAIChatReplacement(self, *args, **kwargs)
69
-
70
- async def _acall(
71
- self,
72
- messages: List[BaseMessage],
73
- stop: Optional[List[str]] = None,
74
- run_manager: Optional[CallbackManagerForLLMRun] = None,
75
- **kwargs: Any,
76
- ):
77
- models.apply_rate_limiter_sync(self._wrapper.a0_model_conf, str(messages))
78
-
79
- try:
80
- model = kwargs.pop("model", None)
81
- effective_model = model or self._wrapper.model_name
82
- kwrgs = {**self._wrapper.kwargs, **kwargs}
83
- request_messages = messages
84
-
85
- # hack from browser-use to fix json schema for gemini (additionalProperties, $defs, $ref)
86
- if "response_format" in kwrgs and "json_schema" in kwrgs["response_format"] and effective_model and effective_model.startswith("gemini/"):
87
- kwrgs["response_format"]["json_schema"] = ChatGoogle("")._fix_gemini_schema(kwrgs["response_format"]["json_schema"])
88
-
89
- if browser_use_openrouter_compat.should_use_openrouter_prompt_schema_fallback(
90
- provider=self.provider,
91
- model_name=effective_model,
92
- kwargs=kwrgs,
93
- ):
94
- fallback_request = browser_use_openrouter_compat.build_json_object_fallback_request(
95
- messages=messages,
96
- kwargs=kwrgs,
97
- )
98
- if fallback_request is not None:
99
- request_messages, kwrgs = fallback_request
100
-
101
- resp = await acompletion(
102
- model=self._wrapper.model_name,
103
- messages=request_messages,
104
- stop=stop,
105
- **kwrgs,
106
- )
107
-
108
- # Gemini: strip triple backticks and conform schema
109
- try:
110
- msg = resp.choices[0].message # type: ignore
111
- if self.provider == "gemini" and isinstance(getattr(msg, "content", None), str):
112
- cleaned = browser_use_monkeypatch.gemini_clean_and_conform(msg.content) # type: ignore
113
- if cleaned:
114
- msg.content = cleaned
115
- except Exception:
116
- pass
117
-
118
- except Exception as e:
119
- raise e
120
-
121
- # Structured output: normalize keys/models reject (e.g. "" on action dicts) and repair partial JSON
122
- try:
123
- rf = kwrgs.get("response_format") or {}
124
- if "json_schema" in rf or "json_object" in rf:
125
- msg_obj = resp.choices[0].message
126
- raw_content = getattr(msg_obj, "content", None)
127
- fixed = browser_use_output_sanitize.sanitize_llm_message_content_for_browser_use(raw_content) # type: ignore[arg-type]
128
- if fixed is not None:
129
- msg_obj.content = fixed
130
- except Exception:
131
- pass
132
-
133
- return resp
134
-
135
-
136
-def build_browser_model_from_config(
137
- model_config: models.ModelConfig,
138
-) -> BrowserCompatibleChatWrapper:
139
- apply_browser_use_patches()
140
- original_provider = model_config.provider.lower()
141
- provider_name, kwargs = models._merge_provider_defaults( # type: ignore[attr-defined]
142
- "chat", original_provider, model_config.build_kwargs()
143
- )
144
- return models._get_litellm_chat( # type: ignore[attr-defined]
145
- BrowserCompatibleChatWrapper,
146
- model_config.name,
147
- provider_name,
148
- model_config,
149
- **kwargs,
150
- )
151
-
152
-def build_browser_model_for_agent(agent=None) -> BrowserCompatibleChatWrapper:
153
- """Build and return the browser-use adapter using chat model config."""
154
- from plugins._model_config.helpers.model_config import (
155
- build_model_config,
156
- )
157
- import models
158
-
159
- selection = model_preset.resolve_browser_model_selection(agent)
160
- cfg = selection["config"]
161
- mc = build_model_config(cfg, models.ModelType.CHAT)
162
- return build_browser_model_from_config(mc)
plugins/_browser_agent/helpers/browser_use.py
deleted
-4
@@ -1,4 +0,0 @@
1
-from helpers import dotenv
2
-dotenv.save_dotenv_value("ANONYMIZED_TELEMETRY", "false")
3
-import browser_use
4
-import browser_use.utils
plugins/_browser_agent/helpers/browser_use_monkeypatch.py
deleted
-166
@@ -1,166 +0,0 @@
1
-from typing import Any
2
-from browser_use.llm import ChatGoogle
3
-from helpers import dirty_json
4
-
5
-from plugins._browser_agent.helpers import browser_use_output_sanitize
6
-
7
-
8
-# ------------------------------------------------------------------------------
9
-# Gemini Helper for Output Conformance
10
-# ------------------------------------------------------------------------------
11
-# This function sanitizes and conforms the JSON output from Gemini to match
12
-# the specific schema expectations of the browser-use library. It handles
13
-# markdown fences, aliases actions (like 'complete_task' to 'done'), and
14
-# intelligently constructs a valid 'data' object for the final action.
15
-
16
-def gemini_clean_and_conform(text: str):
17
- obj = None
18
- try:
19
- # dirty_json parser is robust enough to handle markdown fences
20
- obj = dirty_json.parse(text)
21
- except Exception:
22
- return None # return None if parsing fails
23
-
24
- if not isinstance(obj, dict):
25
- return None
26
-
27
- obj = browser_use_output_sanitize.normalize_parsed_browser_use_output(obj)
28
-
29
- # Conform actions to browser-use expectations
30
- if isinstance(obj.get("action"), list):
31
- normalized_actions = []
32
- for item in obj["action"]:
33
- if not isinstance(item, dict):
34
- continue # Skip non-dict items
35
-
36
- action_key, action_value = next(iter(item.items()), (None, None))
37
- if not action_key:
38
- continue
39
-
40
- # Alias 'complete_task' to 'done' to handle inconsistencies
41
- if action_key == "complete_task":
42
- action_key = "done"
43
-
44
- # Create a mutable copy of the value
45
- v = (action_value or {}).copy()
46
-
47
- if action_key in ("scroll_down", "scroll_up", "scroll"):
48
- is_down = action_key != "scroll_up"
49
- v.setdefault("down", is_down)
50
- v.setdefault("num_pages", 1.0)
51
- normalized_actions.append({"scroll": v})
52
- elif action_key == "go_to_url":
53
- v.setdefault("new_tab", False)
54
- normalized_actions.append({action_key: v})
55
- elif action_key == "done":
56
- # If `data` is missing, construct it from other keys
57
- if "data" not in v:
58
- # Pop fields from the top-level `done` object
59
- response_text = v.pop("response", None)
60
- summary_text = v.pop("page_summary", None)
61
- title_text = v.pop("title", "Task Completed")
62
-
63
- final_response = response_text or "Task completed successfully." # browser-use expects string
64
- final_summary = summary_text or "No page summary available." # browser-use expects string
65
-
66
- v["data"] = {
67
- "title": title_text,
68
- "response": final_response,
69
- "page_summary": final_summary,
70
- }
71
-
72
- v.setdefault("success", True)
73
- normalized_actions.append({action_key: v})
74
- else:
75
- normalized_actions.append(item)
76
- obj["action"] = normalized_actions
77
-
78
- return dirty_json.stringify(obj)
79
-
80
-# ------------------------------------------------------------------------------
81
-# Monkey-patch for browser-use Gemini schema issue
82
-# ------------------------------------------------------------------------------
83
-# The original _fix_gemini_schema in browser_use.llm.google.chat.ChatGoogle
84
-# removes the 'title' property but fails to remove it from the 'required' list,
85
-# causing a validation error with the Gemini API. This patch corrects that behavior.
86
-
87
-def _patched_fix_gemini_schema(self, schema: dict[str, Any]) -> dict[str, Any]:
88
- """
89
- Convert a Pydantic model to a Gemini-compatible schema.
90
-
91
- This function removes unsupported properties like 'additionalProperties' and resolves
92
- $ref references that Gemini doesn't support.
93
- """
94
-
95
- # Handle $defs and $ref resolution
96
- if '$defs' in schema:
97
- defs = schema.pop('$defs')
98
-
99
- def resolve_refs(obj: Any) -> Any:
100
- if isinstance(obj, dict):
101
- if '$ref' in obj:
102
- ref = obj.pop('$ref')
103
- ref_name = ref.split('/')[-1]
104
- if ref_name in defs:
105
- # Replace the reference with the actual definition
106
- resolved = defs[ref_name].copy()
107
- # Merge any additional properties from the reference
108
- for key, value in obj.items():
109
- if key != '$ref':
110
- resolved[key] = value
111
- return resolve_refs(resolved)
112
- return obj
113
- else:
114
- # Recursively process all dictionary values
115
- return {k: resolve_refs(v) for k, v in obj.items()}
116
- elif isinstance(obj, list):
117
- return [resolve_refs(item) for item in obj]
118
- return obj
119
-
120
- schema = resolve_refs(schema)
121
-
122
- # Remove unsupported properties
123
- def clean_schema(obj: Any) -> Any:
124
- if isinstance(obj, dict):
125
- # Remove unsupported properties
126
- cleaned = {}
127
- for key, value in obj.items():
128
- if key not in ['additionalProperties', 'title', 'default']:
129
- cleaned_value = clean_schema(value)
130
- # Handle empty object properties - Gemini doesn't allow empty OBJECT types
131
- if (
132
- key == 'properties'
133
- and isinstance(cleaned_value, dict)
134
- and len(cleaned_value) == 0
135
- and isinstance(obj.get('type', ''), str)
136
- and obj.get('type', '').upper() == 'OBJECT'
137
- ):
138
- # Convert empty object to have at least one property
139
- cleaned['properties'] = {'_placeholder': {'type': 'string'}}
140
- else:
141
- cleaned[key] = cleaned_value
142
-
143
- # If this is an object type with empty properties, add a placeholder
144
- if (
145
- isinstance(cleaned.get('type', ''), str)
146
- and cleaned.get('type', '').upper() == 'OBJECT'
147
- and 'properties' in cleaned
148
- and isinstance(cleaned['properties'], dict)
149
- and len(cleaned['properties']) == 0
150
- ):
151
- cleaned['properties'] = {'_placeholder': {'type': 'string'}}
152
-
153
- # PATCH: Also remove 'title' from the required list if it exists
154
- if 'required' in cleaned and isinstance(cleaned.get('required'), list):
155
- cleaned['required'] = [p for p in cleaned['required'] if p != 'title']
156
-
157
- return cleaned
158
- elif isinstance(obj, list):
159
- return [clean_schema(item) for item in obj]
160
- return obj
161
-
162
- return clean_schema(schema)
163
-
164
-def apply():
165
- """Applies the monkey-patch to ChatGoogle."""
166
- ChatGoogle._fix_gemini_schema = _patched_fix_gemini_schema
plugins/_browser_agent/helpers/browser_use_openrouter_compat.py
deleted
-93
@@ -1,93 +0,0 @@
1
-from __future__ import annotations
2
-
3
-import copy
4
-import json
5
-from typing import Any
6
-
7
-def is_openrouter_request(provider: str | None, model_name: str | None) -> bool:
8
- provider_name = (provider or "").lower()
9
- model = (model_name or "").lower()
10
- return provider_name == "openrouter" or model.startswith("openrouter/")
11
-
12
-
13
-def has_json_schema_response_format(kwargs: dict[str, Any]) -> bool:
14
- response_format = kwargs.get("response_format")
15
- return isinstance(response_format, dict) and (
16
- response_format.get("type") == "json_schema" or "json_schema" in response_format
17
- )
18
-
19
-
20
-def should_use_openrouter_prompt_schema_fallback(
21
- provider: str | None, model_name: str | None, kwargs: dict[str, Any]
22
-) -> bool:
23
- """
24
- OpenRouter sometimes routes browser-use structured output through providers
25
- that reject large compiled grammars. Avoid the hard error entirely by
26
- downgrading to `json_object` before the first request.
27
- """
28
- return is_openrouter_request(provider, model_name) and has_json_schema_response_format(kwargs)
29
-
30
-
31
-def relax_strict_tool_schemas(tools: Any) -> Any:
32
- """
33
- Disable strict tool grammar on fallback while keeping tool definitions intact.
34
- """
35
- if not isinstance(tools, list):
36
- return tools
37
-
38
- relaxed = copy.deepcopy(tools)
39
- for tool in relaxed:
40
- if not isinstance(tool, dict):
41
- continue
42
- function_spec = tool.get("function")
43
- if isinstance(function_spec, dict) and function_spec.get("strict") is True:
44
- function_spec["strict"] = False
45
- return relaxed
46
-
47
-
48
-def _schema_hint_text(response_format: dict[str, Any]) -> str | None:
49
- schema_payload = response_format.get("json_schema")
50
- if not isinstance(schema_payload, dict):
51
- return None
52
-
53
- compact_schema = json.dumps(
54
- schema_payload,
55
- ensure_ascii=False,
56
- separators=(",", ":"),
57
- )
58
- return (
59
- "Return only a single JSON object with no markdown fences, prose, or extra text. "
60
- "Follow this schema exactly: "
61
- f"{compact_schema}"
62
- )
63
-
64
-
65
-def prepend_schema_hint_to_messages(
66
- messages: list[Any], response_format: dict[str, Any]
67
-) -> list[Any]:
68
- hint = _schema_hint_text(response_format)
69
- if not hint:
70
- return list(messages)
71
- return [{"role": "system", "content": hint}, *list(messages)]
72
-
73
-
74
-def build_json_object_fallback_request(
75
- messages: list[Any],
76
- kwargs: dict[str, Any],
77
-) -> tuple[list[Any], dict[str, Any]] | None:
78
- """
79
- Replace strict json_schema with json_object and move schema guidance into the prompt.
80
-
81
- This keeps browser-use's local validation path while avoiding provider-side
82
- grammar compilation limits on OpenRouter.
83
- """
84
- response_format = kwargs.get("response_format")
85
- if not isinstance(response_format, dict):
86
- return None
87
-
88
- updated_kwargs = copy.deepcopy(kwargs)
89
- updated_kwargs["response_format"] = {"type": "json_object"}
90
- if "tools" in updated_kwargs:
91
- updated_kwargs["tools"] = relax_strict_tool_schemas(updated_kwargs["tools"])
92
- updated_messages = prepend_schema_hint_to_messages(messages, response_format)
93
- return updated_messages, updated_kwargs
plugins/_browser_agent/helpers/browser_use_output_sanitize.py
deleted
-79
@@ -1,79 +0,0 @@
1
-"""
2
-Utilities to normalize LLM replies before browser-use parses them into AgentOutput.
3
-
4
-Some models (e.g. via OpenRouter) emit extra JSON keys such as "" : "", which
5
-Pydantic rejects as extra_forbidden on strict action union members.
6
-"""
7
-
8
-from __future__ import annotations
9
-
10
-from typing import Any
11
-
12
-from helpers import dirty_json
13
-
14
-
15
-def deep_strip_empty_string_keys(obj: Any) -> Any:
16
- """
17
- Recursively remove dict entries whose key is the empty string.
18
-
19
- Browser-use action objects must be discriminated unions with a single
20
- action key; spurious "" keys break validation for every union variant.
21
- """
22
- if isinstance(obj, dict):
23
- return {
24
- k: deep_strip_empty_string_keys(v)
25
- for k, v in obj.items()
26
- if k != ""
27
- }
28
- if isinstance(obj, list):
29
- return [deep_strip_empty_string_keys(item) for item in obj]
30
- return obj
31
-
32
-
33
-def normalize_parsed_browser_use_output(obj: dict) -> dict:
34
- """Apply all normalizations safe for a parsed AgentOutput-shaped dict."""
35
- out = deep_strip_empty_string_keys(obj)
36
- if not isinstance(out, dict):
37
- return obj
38
- return out
39
-
40
-
41
-def parse_and_sanitize_llm_json(text: str) -> str | None:
42
- """
43
- Parse message content and return JSON text safe for AgentOutput parsing.
44
-
45
- Returns None if the string is not a JSON object.
46
- """
47
- try:
48
- obj = dirty_json.parse(text)
49
- except Exception:
50
- return None
51
- if not isinstance(obj, dict):
52
- return None
53
- return dirty_json.stringify(normalize_parsed_browser_use_output(obj))
54
-
55
-
56
-def sanitize_llm_message_content_for_browser_use(content: str | None) -> str | None:
57
- """
58
- Best-effort sanitize assistant message content in place for browser-use.
59
-
60
- - If content parses as a dict: strip bad keys and re-serialize.
61
- - If content is non-JSON or trailing garbage: try dirty_json parse; if dict, sanitize.
62
- - Otherwise return the original string.
63
- """
64
- if content is None:
65
- return None
66
- stripped = content.strip()
67
- if not stripped:
68
- return content
69
- sanitized = parse_and_sanitize_llm_json(stripped)
70
- if sanitized is not None:
71
- return sanitized
72
- if not stripped.startswith("{"):
73
- try:
74
- obj = dirty_json.parse(stripped)
75
- except Exception:
76
- return content
77
- if isinstance(obj, dict):
78
- return dirty_json.stringify(normalize_parsed_browser_use_output(obj))
79
- return content
plugins/_browser_agent/helpers/model_preset.py
deleted
-122
@@ -1,122 +0,0 @@
1
-from __future__ import annotations
2
-
3
-from typing import Any
4
-
5
-from helpers import plugins as plugin_helpers
6
-from plugins._model_config.helpers import model_config
7
-
8
-
9
-MODEL_PRESET_KEY = "model_preset"
10
-
11
-
12
-def get_browser_model_preset_name(agent=None) -> str:
13
- config = plugin_helpers.get_plugin_config("_browser_agent", agent=agent) or {}
14
- return str(config.get(MODEL_PRESET_KEY, "") or "").strip()
15
-
16
-
17
-def get_browser_model_preset_options(agent=None) -> list[dict[str, Any]]:
18
- selected_name = get_browser_model_preset_name(agent)
19
- options: list[dict[str, Any]] = []
20
- found_selected = False
21
-
22
- for preset in model_config.get_presets():
23
- name = str(preset.get("name", "") or "").strip()
24
- if not name:
25
- continue
26
- if name == selected_name:
27
- found_selected = True
28
- chat_cfg = preset.get("chat", {}) if isinstance(preset, dict) else {}
29
- if not isinstance(chat_cfg, dict):
30
- chat_cfg = {}
31
- provider = str(chat_cfg.get("provider", "") or "").strip()
32
- model_name = str(chat_cfg.get("name", "") or "").strip()
33
- summary = " / ".join(part for part in (provider, model_name) if part)
34
- options.append(
35
- {
36
- "name": name,
37
- "label": name,
38
- "missing": False,
39
- "summary": summary,
40
- }
41
- )
42
-
43
- if selected_name and not found_selected:
44
- options.append(
45
- {
46
- "name": selected_name,
47
- "label": f"{selected_name} (missing)",
48
- "missing": True,
49
- "summary": "",
50
- }
51
- )
52
-
53
- return options
54
-
55
-
56
-def resolve_browser_model_selection(agent=None) -> dict[str, Any]:
57
- preset_name = get_browser_model_preset_name(agent)
58
- if preset_name:
59
- preset = model_config.get_preset_by_name(preset_name)
60
- if isinstance(preset, dict):
61
- chat_cfg = preset.get("chat", {})
62
- if isinstance(chat_cfg, dict) and (
63
- str(chat_cfg.get("provider", "") or "").strip()
64
- or str(chat_cfg.get("name", "") or "").strip()
65
- ):
66
- return {
67
- "config": chat_cfg,
68
- "source_kind": "preset",
69
- "source_label": f"Preset '{preset_name}' via _model_config",
70
- "selected_preset_name": preset_name,
71
- "preset_status": "active",
72
- "warning": "",
73
- }
74
- return {
75
- "config": model_config.get_chat_model_config(agent),
76
- "source_kind": "main",
77
- "source_label": "Main Model via _model_config",
78
- "selected_preset_name": preset_name,
79
- "preset_status": "invalid",
80
- "warning": (
81
- f"Configured browser preset '{preset_name}' does not define a chat model. "
82
- "Falling back to the Main Model."
83
- ),
84
- }
85
-
86
- return {
87
- "config": model_config.get_chat_model_config(agent),
88
- "source_kind": "main",
89
- "source_label": "Main Model via _model_config",
90
- "selected_preset_name": preset_name,
91
- "preset_status": "missing",
92
- "warning": (
93
- f"Configured browser preset '{preset_name}' was not found. "
94
- "Falling back to the Main Model."
95
- ),
96
- }
97
-
98
- return {
99
- "config": model_config.get_chat_model_config(agent),
100
- "source_kind": "main",
101
- "source_label": "Main Model via _model_config",
102
- "selected_preset_name": "",
103
- "preset_status": "none",
104
- "warning": "",
105
- }
106
-
107
-
108
-def save_browser_model_preset_name(preset_name: str) -> None:
109
- normalized = str(preset_name or "").strip()
110
- config = plugin_helpers.get_plugin_config("_browser_agent") or {}
111
-
112
- if normalized:
113
- config[MODEL_PRESET_KEY] = normalized
114
- else:
115
- config.pop(MODEL_PRESET_KEY, None)
116
-
117
- plugin_helpers.save_plugin_config(
118
- "_browser_agent",
119
- project_name="",
120
- agent_profile="",
121
- settings=config,
122
- )
plugins/_browser_agent/helpers/playwright.py
deleted
-38
@@ -1,38 +0,0 @@
1
-import os
2
-import sys
3
-from pathlib import Path
4
-import subprocess
5
-from helpers import files
6
-
7
-
8
-# this helper ensures that playwright is installed in /lib/playwright
9
-# should work for both docker and local installation
10
-
11
-def get_playwright_binary():
12
- pw_cache = Path(get_playwright_cache_dir())
13
- for pattern in (
14
- "chromium_headless_shell-*/chrome-*/headless_shell",
15
- "chromium_headless_shell-*/chrome-*/headless_shell.exe",
16
- ):
17
- binary = next(pw_cache.glob(pattern), None)
18
- if binary:
19
- return binary
20
- return None
21
-
22
-def get_playwright_cache_dir():
23
- return files.get_abs_path("tmp/playwright")
24
-
25
-def ensure_playwright_binary():
26
- bin = get_playwright_binary()
27
- if not bin:
28
- cache = get_playwright_cache_dir()
29
- env = os.environ.copy()
30
- env["PLAYWRIGHT_BROWSERS_PATH"] = cache
31
- subprocess.check_call(
32
- ["playwright", "install", "chromium", "--only-shell"],
33
- env=env
34
- )
35
- bin = get_playwright_binary()
36
- if not bin:
37
- raise Exception("Playwright binary not found after installation")
38
- return bin
plugins/_browser_agent/plugin.yaml
deleted
-8
@@ -1,8 +0,0 @@
1
-name: _browser_agent
2
-title: Browser Agent
3
-description: Built-in browser-use automation tool.
4
-version: 1.0.0
5
-always_enabled: false
6
-settings_sections: []
7
-per_project_config: false
8
-per_agent_config: false
plugins/_browser_agent/prompts/agent.system.tool.browser.md
deleted
-7
@@ -1,7 +0,0 @@
1
-### browser_agent
2
-subordinate browser worker for web tasks
3
-args: `message`, `reset`
4
-- give clear task-oriented instructions, credentials, and a stop condition
5
-- `reset=true` starts a new browser session; `false` continues the current one
6
-- when continuing, refer to open pages instead of restarting
7
-downloads go to `/a0/tmp/downloads`
plugins/_browser_agent/prompts/browser_agent.system.md
deleted
-22
@@ -1,22 +0,0 @@
1
-# Operation instruction
2
-Keep your tasks solution as simple and straight forward as possible
3
-Follow instructions as closely as possible
4
-When told go to website, open the website. If no other instructions: stop there
5
-Do not interact with the website unless told to
6
-Always accept all cookies if prompted on the website, NEVER go to browser cookie settings
7
-If asked specific questions about a website, be as precise and close to the actual page content as possible
8
-If you are waiting for instructions: you should end the task and mark as done
9
-
10
-## Task Completion
11
-When you have completed the assigned task OR are waiting for further instructions:
12
-1. Use the "Complete task" action to mark the task as complete
13
-2. Provide the required parameters: title, response, and page_summary
14
-3. Do NOT continue taking actions after calling "Complete task"
15
-
16
-## Important Notes
17
-- Always call "Complete task" when your objective is achieved
18
-- In page_summary respond with one paragraph of main content plus an overview of page elements
19
-- Response field is used to answer to user's task or ask additional questions
20
-- If you navigate to a website and no further actions are requested, call "Complete task" immediately
21
-- If you complete any requested interaction (clicking, typing, etc.), call "Complete task"
22
-- Never leave a task running indefinitely - always conclude with "Complete task"
plugins/_browser_agent/tools/browser_agent.py
deleted
-440
@@ -1,440 +0,0 @@
1
-import asyncio
2
-import time
3
-from typing import Optional, cast
4
-from agent import Agent, InterventionException
5
-from pathlib import Path
6
-
7
-from helpers.tool import Tool, Response
8
-from helpers import files, defer, persist_chat, strings
9
-from plugins._browser_agent.helpers.browser_use import browser_use # type: ignore[attr-defined]
10
-from helpers.print_style import PrintStyle
11
-from plugins._browser_agent.helpers.playwright import ensure_playwright_binary
12
-from helpers.secrets import get_secrets_manager
13
-from extensions.python.message_loop_start._10_iteration_no import get_iter_no
14
-from pydantic import BaseModel
15
-import uuid
16
-from helpers.dirty_json import DirtyJson
17
-
18
-
19
-PLUGIN_DIR = Path(__file__).resolve().parents[1]
20
-
21
-
22
-class State:
23
- @staticmethod
24
- async def create(agent: Agent):
25
- state = State(agent)
26
- return state
27
-
28
- def __init__(self, agent: Agent):
29
- self.agent = agent
30
- self.browser_session: Optional[browser_use.BrowserSession] = None
31
- self.task: Optional[defer.DeferredTask] = None
32
- self.use_agent: Optional[browser_use.Agent] = None
33
- self.secrets_dict: Optional[dict[str, str]] = None
34
- self.iter_no = 0
35
-
36
- def __del__(self):
37
- self.kill_task()
38
- files.delete_dir(self.get_user_data_dir()) # cleanup user data dir
39
-
40
- def get_user_data_dir(self):
41
- return str(
42
- Path.home()
43
- / ".config"
44
- / "browseruse"
45
- / "profiles"
46
- / f"agent_{self.agent.context.id}"
47
- )
48
-
49
- def _get_browser_http_headers(self):
50
- # ignored for now
51
- return {}
52
-
53
- def _get_browser_vision(self):
54
- from plugins._model_config.helpers.model_config import get_chat_model_config
55
- cfg = get_chat_model_config(self.agent)
56
- return cfg.get("vision", False)
57
-
58
- async def _initialize(self):
59
- if self.browser_session:
60
- return
61
-
62
- # for some reason we need to provide exact path to headless shell, otherwise it looks for headed browser
63
- pw_binary = ensure_playwright_binary()
64
-
65
- self.browser_session = browser_use.BrowserSession(
66
- browser_profile=browser_use.BrowserProfile(
67
- headless=True,
68
- disable_security=True,
69
- chromium_sandbox=False,
70
- accept_downloads=True,
71
- downloads_path=files.get_abs_path("usr/downloads"),
72
- allowed_domains=["*", "http://*", "https://*"],
73
- executable_path=pw_binary,
74
- keep_alive=True,
75
- minimum_wait_page_load_time=1.0,
76
- wait_for_network_idle_page_load_time=2.0,
77
- maximum_wait_page_load_time=10.0,
78
- window_size={"width": 1024, "height": 2048},
79
- screen={"width": 1024, "height": 2048},
80
- viewport={"width": 1024, "height": 2048},
81
- no_viewport=False,
82
- args=["--headless=new", "--no-sandbox"],
83
- # Use a unique user data directory to avoid conflicts
84
- user_data_dir=self.get_user_data_dir(),
85
- extra_http_headers=self._get_browser_http_headers(),
86
- )
87
- )
88
-
89
- await self.browser_session.start() if self.browser_session else None
90
- # self.override_hooks()
91
-
92
- # --------------------------------------------------------------------------
93
- # Patch to enforce vertical viewport size
94
- # --------------------------------------------------------------------------
95
- # Browser-use auto-configuration overrides viewport settings, causing wrong
96
- # aspect ratio. We fix this by directly setting viewport size after startup.
97
- # --------------------------------------------------------------------------
98
-
99
- if self.browser_session:
100
- try:
101
- page = await self.browser_session.get_current_page()
102
- if page:
103
- await page.set_viewport_size({"width": 1024, "height": 2048})
104
- except Exception as e:
105
- PrintStyle().warning(f"Could not force set viewport size: {e}")
106
-
107
- # --------------------------------------------------------------------------
108
-
109
- # Add init script to the browser session
110
- if self.browser_session and self.browser_session.browser_context:
111
- js_override = str(PLUGIN_DIR / "assets" / "init_override.js")
112
- await self.browser_session.browser_context.add_init_script(path=js_override) if self.browser_session else None
113
-
114
- def start_task(self, task: str):
115
- if self.task and self.task.is_alive():
116
- self.kill_task()
117
-
118
- self.task = defer.DeferredTask(
119
- thread_name="BrowserAgent" + self.agent.context.id
120
- )
121
- if self.agent.context.task:
122
- self.agent.context.task.add_child_task(self.task, terminate_thread=True)
123
- self.task.start_task(self._run_task, task) if self.task else None
124
- return self.task
125
-
126
- def kill_task(self):
127
- if self.task:
128
- self.task.kill(terminate_thread=True)
129
- self.task = None
130
- if self.browser_session:
131
- try:
132
- import asyncio
133
-
134
- loop = asyncio.new_event_loop()
135
- asyncio.set_event_loop(loop)
136
- loop.run_until_complete(self.browser_session.close()) if self.browser_session else None
137
- loop.close()
138
- except Exception as e:
139
- PrintStyle().error(f"Error closing browser session: {e}")
140
- finally:
141
- self.browser_session = None
142
- self.use_agent = None
143
- self.iter_no = 0
144
-
145
- async def _run_task(self, task: str):
146
- await self._initialize()
147
-
148
- class DoneResult(BaseModel):
149
- title: str
150
- response: str
151
- page_summary: str
152
-
153
- # Initialize controller
154
- controller = browser_use.Controller(output_model=DoneResult)
155
-
156
- # Register custom completion action with proper ActionResult fields
157
- @controller.registry.action("Complete task", param_model=DoneResult)
158
- async def complete_task(params: DoneResult):
159
- result = browser_use.ActionResult(
160
- is_done=True, success=True, extracted_content=params.model_dump_json()
161
- )
162
- return result
163
-
164
- model = self.agent.get_browser_model()
165
-
166
- try:
167
-
168
- secrets_manager = get_secrets_manager(self.agent.context)
169
- secrets_dict = secrets_manager.load_secrets()
170
-
171
- self.use_agent = browser_use.Agent(
172
- task=task,
173
- browser_session=self.browser_session,
174
- llm=model,
175
- use_vision=self._get_browser_vision(),
176
- extend_system_message=self.agent.read_prompt(
177
- "prompts/browser_agent.system.md"
178
- ),
179
- controller=controller,
180
- enable_memory=False, # Disable memory to avoid state conflicts
181
- llm_timeout=3000, # TODO rem
182
- sensitive_data=cast(dict[str, str | dict[str, str]] | None, secrets_dict or {}), # Pass secrets
183
- )
184
- except Exception as e:
185
- raise Exception(
186
- f"Browser agent initialization failed. This might be due to model compatibility issues. Error: {e}"
187
- ) from e
188
-
189
- self.iter_no = get_iter_no(self.agent)
190
-
191
- async def hook(agent: browser_use.Agent):
192
- await self.agent.wait_if_paused()
193
- if self.iter_no != get_iter_no(self.agent):
194
- raise InterventionException("Task cancelled")
195
-
196
- # try:
197
- result = None
198
- if self.use_agent:
199
- result = await self.use_agent.run(
200
- max_steps=50, on_step_start=hook, on_step_end=hook
201
- )
202
- return result
203
-
204
- async def get_page(self):
205
- if self.use_agent and self.browser_session:
206
- try:
207
- return await self.use_agent.browser_session.get_current_page() if self.use_agent.browser_session else None
208
- except Exception:
209
- # Browser session might be closed or invalid
210
- return None
211
- return None
212
-
213
- async def get_selector_map(self):
214
- """Get the selector map for the current page state."""
215
- if self.use_agent:
216
- await self.use_agent.browser_session.get_state_summary(cache_clickable_elements_hashes=True) if self.use_agent.browser_session else None
217
- return await self.use_agent.browser_session.get_selector_map() if self.use_agent.browser_session else None
218
- await self.use_agent.browser_session.get_state_summary(
219
- cache_clickable_elements_hashes=True
220
- )
221
- return await self.use_agent.browser_session.get_selector_map()
222
- return {}
223
-
224
-
225
-class BrowserAgent(Tool):
226
-
227
- async def execute(self, message="", reset="", **kwargs):
228
- self.guid = self.agent.context.generate_id() # short random id
229
- reset = str(reset).lower().strip() == "true"
230
- await self.prepare_state(reset=reset)
231
- message = get_secrets_manager(self.agent.context).mask_values(message, placeholder="<secret>{key}</secret>") # mask any potential passwords passed from A0 to browser-use to browser-use format
232
- task = self.state.start_task(message) if self.state else None
233
-
234
- # wait for browser agent to finish and update progress with timeout
235
- timeout_seconds = 300 # 5 minute timeout
236
- start_time = time.time()
237
-
238
- fail_counter = 0
239
- while not task.is_ready() if task else False:
240
- # Check for timeout to prevent infinite waiting
241
- if time.time() - start_time > timeout_seconds:
242
- PrintStyle().warning(
243
- self._mask(f"Browser agent task timeout after {timeout_seconds} seconds, forcing completion")
244
- )
245
- break
246
-
247
- await self.agent.handle_intervention()
248
- await asyncio.sleep(1)
249
- try:
250
- if task and task.is_ready(): # otherwise get_update hangs
251
- break
252
- try:
253
- update = await asyncio.wait_for(self.get_update(), timeout=10)
254
- fail_counter = 0 # reset on success
255
- except asyncio.TimeoutError:
256
- fail_counter += 1
257
- PrintStyle().warning(
258
- self._mask(f"browser_agent.get_update timed out ({fail_counter}/3)")
259
- )
260
- if fail_counter >= 3:
261
- PrintStyle().warning(
262
- self._mask("3 consecutive browser_agent.get_update timeouts, breaking loop")
263
- )
264
- break
265
- continue
266
- update_log = update.get("log", get_use_agent_log(None))
267
- self.update_progress("\n".join(update_log))
268
- screenshot = update.get("screenshot", None)
269
- if screenshot:
270
- self.log.update(screenshot=screenshot)
271
- except Exception as e:
272
- PrintStyle().error(self._mask(f"Error getting update: {str(e)}"))
273
-
274
- if task and not task.is_ready():
275
- PrintStyle().warning(self._mask("browser_agent.get_update timed out, killing the task"))
276
- self.state.kill_task() if self.state else None
277
- return Response(
278
- message=self._mask("Browser agent task timed out, not output provided."),
279
- break_loop=False,
280
- )
281
-
282
- # final progress update
283
- if self.state and self.state.use_agent:
284
- log_final = get_use_agent_log(self.state.use_agent)
285
- self.update_progress("\n".join(log_final))
286
-
287
- # collect result with error handling
288
- try:
289
- result = await task.result() if task else None
290
- except Exception as e:
291
- PrintStyle().error(self._mask(f"Error getting browser agent task result: {str(e)}"))
292
- # Return a timeout response if task.result() fails
293
- answer_text = self._mask(f"Browser agent task failed to return result: {str(e)}")
294
- self.log.update(answer=answer_text)
295
- return Response(message=answer_text, break_loop=False)
296
- # finally:
297
- # # Stop any further browser access after task completion
298
- # # self.state.kill_task()
299
- # pass
300
-
301
- # Check if task completed successfully
302
- if result and result.is_done():
303
- answer = result.final_result()
304
- try:
305
- if answer and isinstance(answer, str) and answer.strip():
306
- answer_data = DirtyJson.parse_string(answer)
307
- answer_text = strings.dict_to_text(answer_data) # type: ignore
308
- else:
309
- answer_text = (
310
- str(answer) if answer else "Task completed successfully"
311
- )
312
- except Exception as e:
313
- answer_text = (
314
- str(answer)
315
- if answer
316
- else f"Task completed with parse error: {str(e)}"
317
- )
318
- else:
319
- # Task hit max_steps without calling done()
320
- urls = result.urls() if result else []
321
- current_url = urls[-1] if urls else "unknown"
322
- answer_text = (
323
- f"Task reached step limit without completion. Last page: {current_url}. "
324
- f"The browser agent may need clearer instructions on when to finish."
325
- )
326
-
327
- # Mask answer for logs and response
328
- answer_text = self._mask(answer_text)
329
-
330
- # update the log (without screenshot path here, user can click)
331
- self.log.update(answer=answer_text)
332
-
333
- # add screenshot to the answer if we have it
334
- if (
335
- self.log.kvps
336
- and "screenshot" in self.log.kvps
337
- and self.log.kvps["screenshot"]
338
- ):
339
- path = self.log.kvps["screenshot"].split("//", 1)[-1].split("&", 1)[0]
340
- answer_text += f"\n\nScreenshot: {path}"
341
-
342
- # respond (with screenshot path)
343
- return Response(message=answer_text, break_loop=False)
344
-
345
- def get_log_object(self):
346
- return self.agent.context.log.log(
347
- type="browser",
348
- heading=f"icon://captive_portal {self.agent.agent_name}: Calling Browser Agent",
349
- content="",
350
- kvps=self.args,
351
- )
352
-
353
- async def get_update(self):
354
- await self.prepare_state()
355
-
356
- result = {}
357
- agent = self.agent
358
- ua = self.state.use_agent if self.state else None
359
- page = await self.state.get_page() if self.state else None
360
-
361
- if ua and page:
362
- try:
363
-
364
- async def _get_update():
365
-
366
- # await agent.wait_if_paused() # no need here
367
-
368
- # Build short activity log
369
- result["log"] = get_use_agent_log(ua)
370
-
371
- path = files.get_abs_path(
372
- persist_chat.get_chat_folder_path(agent.context.id),
373
- "browser",
374
- "screenshots",
375
- f"{self.guid}.png",
376
- )
377
- files.make_dirs(path)
378
- await page.screenshot(path=path, full_page=False, timeout=3000)
379
- result["screenshot"] = f"img://{path}&t={str(time.time())}"
380
-
381
- if self.state and self.state.task and not self.state.task.is_ready():
382
- await self.state.task.execute_inside(_get_update)
383
-
384
- except Exception:
385
- pass
386
-
387
- return result
388
-
389
- async def prepare_state(self, reset=False):
390
- self.state = self.agent.get_data("_browser_agent_state")
391
- if reset and self.state:
392
- self.state.kill_task()
393
- if not self.state or reset:
394
- self.state = await State.create(self.agent)
395
- self.agent.set_data("_browser_agent_state", self.state)
396
-
397
- def update_progress(self, text):
398
- text = self._mask(text)
399
- short = text.split("\n")[-1]
400
- if len(short) > 50:
401
- short = short[:50] + "..."
402
- progress = f"Browser: {short}"
403
-
404
- self.log.update(progress=text)
405
- self.agent.context.log.set_progress(progress)
406
-
407
- def _mask(self, text: str) -> str:
408
- try:
409
- return get_secrets_manager(self.agent.context).mask_values(text or "")
410
- except Exception as e:
411
- return text or ""
412
-
413
- # def __del__(self):
414
- # if self.state:
415
- # self.state.kill_task()
416
-
417
-
418
-def get_use_agent_log(use_agent: browser_use.Agent | None):
419
- result = ["🚦 Starting task"]
420
- if use_agent:
421
- action_results = use_agent.history.action_results() or []
422
- short_log = []
423
- for item in action_results:
424
- # final results
425
- if item.is_done:
426
- if item.success:
427
- short_log.append("✅ Done")
428
- else:
429
- short_log.append(
430
- f"❌ Error: {item.error or item.extracted_content or 'Unknown error'}"
431
- )
432
-
433
- # progress messages
434
- else:
435
- text = item.extracted_content
436
- if text:
437
- first_line = text.split("\n", 1)[0][:200]
438
- short_log.append(first_line)
439
- result.extend(short_log)
440
- return result
plugins/_browser_agent/webui/browser-agent-store.js
deleted
-51
@@ -1,51 +0,0 @@
1
-import { createStore } from "/js/AlpineStore.js";
2
-import { callJsonApi } from "/js/api.js";
3
-
4
-const STATUS_API = "/plugins/_browser_agent/status";
5
-const MODEL_PRESET_API = "/plugins/_browser_agent/model_preset";
6
-
7
-const model = {
8
- loading: true,
9
- savingPreset: false,
10
- error: "",
11
- status: null,
12
-
13
- async refreshStatus() {
14
- this.status = await callJsonApi(STATUS_API, {});
15
- },
16
-
17
- async savePreset(presetName) {
18
- this.savingPreset = true;
19
- try {
20
- await callJsonApi(MODEL_PRESET_API, {
21
- action: presetName ? "set" : "clear",
22
- preset_name: presetName || "",
23
- });
24
- this.error = "";
25
- await this.refreshStatus();
26
- } catch (error) {
27
- this.error = error instanceof Error ? error.message : String(error);
28
- await this.refreshStatus();
29
- } finally {
30
- this.savingPreset = false;
31
- }
32
- },
33
-
34
- async onOpen() {
35
- this.loading = true;
36
- this.error = "";
37
-
38
- try {
39
- await this.refreshStatus();
40
- } catch (error) {
41
- this.status = null;
42
- this.error = error instanceof Error ? error.message : String(error);
43
- } finally {
44
- this.loading = false;
45
- }
46
- },
47
-
48
- cleanup() {},
49
-};
50
-
51
-export const store = createStore("browserAgentPage", model);
plugins/_browser_agent/webui/main.html
deleted
-232
@@ -1,232 +0,0 @@
1
-<html>
2
-<head>
3
- <title>Browser Agent</title>
4
- <script type="module">
5
- import { store } from "/plugins/_browser_agent/webui/browser-agent-store.js";
6
- </script>
7
-</head>
8
-<body>
9
- <div x-data>
10
- <template x-if="$store.browserAgentPage">
11
- <div
12
- x-create="$store.browserAgentPage.onOpen()"
13
- x-destroy="$store.browserAgentPage.cleanup()"
14
- class="browser-agent-page"
15
- >
16
- <div class="section-description">
17
- Built-in browser automation plugin backed by `browser-use` and Playwright.
18
- Model selection stays in `_model_config`; the browser agent can follow the effective Main Model or use one saved preset just for browser tasks.
19
- </div>
20
-
21
- <div class="browser-agent-card" x-show="$store.browserAgentPage.loading">
22
- <div class="status-row">
23
- <span class="material-symbols-outlined spinning">progress_activity</span>
24
- <span>Loading browser status...</span>
25
- </div>
26
- </div>
27
-
28
- <div class="browser-agent-card error" x-show="!$store.browserAgentPage.loading && $store.browserAgentPage.error">
29
- <div class="field-title">Status check failed</div>
30
- <div class="field-description" x-text="$store.browserAgentPage.error"></div>
31
- </div>
32
-
33
- <template x-if="!$store.browserAgentPage.loading && $store.browserAgentPage.status">
34
- <div class="browser-agent-grid">
35
- <div class="browser-agent-card">
36
- <div class="field-title">Model Source</div>
37
- <div class="field-description" x-text="$store.browserAgentPage.status.model_source"></div>
38
- <div class="field-description" x-show="$store.browserAgentPage.status.preset_warning" x-text="$store.browserAgentPage.status.preset_warning"></div>
39
- </div>
40
-
41
- <div class="browser-agent-card">
42
- <div class="field-title">Resolved Browser Model</div>
43
- <div class="status-row">
44
- <span class="status-key">Provider</span>
45
- <span class="status-value" x-text="$store.browserAgentPage.status.model.provider || 'Not configured'"></span>
46
- </div>
47
- <div class="status-row">
48
- <span class="status-key">Model</span>
49
- <span class="status-value" x-text="$store.browserAgentPage.status.model.name || 'Not configured'"></span>
50
- </div>
51
- <div class="status-row">
52
- <span class="status-key">Vision</span>
53
- <span class="status-badge" :class="$store.browserAgentPage.status.model.vision ? 'ok' : 'warn'" x-text="$store.browserAgentPage.status.model.vision ? 'Enabled' : 'Disabled'"></span>
54
- </div>
55
- </div>
56
-
57
- <div class="browser-agent-card">
58
- <div class="field-title">Browser Model Preset</div>
59
- <div class="field-description">
60
- Pick an optional `_model_config` preset for browser-only runs. Leave it empty to keep using the effective Main Model.
61
- </div>
62
- <label class="browser-agent-select-label" for="browser-agent-preset-select">Preset</label>
63
- <select
64
- id="browser-agent-preset-select"
65
- class="browser-agent-select"
66
- :disabled="$store.browserAgentPage.savingPreset"
67
- x-model="$store.browserAgentPage.status.selected_preset_name"
68
- @change="$store.browserAgentPage.savePreset($store.browserAgentPage.status.selected_preset_name)"
69
- >
70
- <option value="">Use Main Model</option>
71
- <template x-for="preset in $store.browserAgentPage.status.available_presets" :key="preset.name">
72
- <option :value="preset.name" x-text="preset.label"></option>
73
- </template>
74
- </select>
75
- <div class="field-description" x-show="$store.browserAgentPage.savingPreset">Saving browser preset...</div>
76
- </div>
77
-
78
- <div class="browser-agent-card">
79
- <div class="field-title">Playwright Runtime</div>
80
- <div class="status-row">
81
- <span class="status-key">Binary</span>
82
- <span class="status-badge" :class="$store.browserAgentPage.status.playwright.binary_found ? 'ok' : 'fail'" x-text="$store.browserAgentPage.status.playwright.binary_found ? 'Found' : 'Missing'"></span>
83
- </div>
84
- <div class="status-row">
85
- <span class="status-key">Cache</span>
86
- <span class="status-value mono" x-text="$store.browserAgentPage.status.playwright.cache_dir"></span>
87
- </div>
88
- <div class="status-row" x-show="$store.browserAgentPage.status.playwright.binary_path">
89
- <span class="status-key">Path</span>
90
- <span class="status-value mono" x-text="$store.browserAgentPage.status.playwright.binary_path"></span>
91
- </div>
92
- <div class="field-description" x-show="!$store.browserAgentPage.status.playwright.binary_found">
93
- Docker images ship the Playwright Chromium shell preinstalled. In local development, the first run installs it on demand via <span class="mono">ensure_playwright_binary()</span> if missing.
94
- </div>
95
- </div>
96
-
97
- <div class="browser-agent-card">
98
- <div class="field-title">browser-use</div>
99
- <div class="status-row">
100
- <span class="status-key">Import</span>
101
- <span class="status-badge" :class="$store.browserAgentPage.status.browser_use.import_ok ? 'ok' : 'fail'" x-text="$store.browserAgentPage.status.browser_use.import_ok ? 'Ready' : 'Error'"></span>
102
- </div>
103
- <div class="status-row" x-show="$store.browserAgentPage.status.browser_use.version">
104
- <span class="status-key">Version</span>
105
- <span class="status-value" x-text="$store.browserAgentPage.status.browser_use.version"></span>
106
- </div>
107
- <div class="field-description mono" x-show="$store.browserAgentPage.status.browser_use.error" x-text="$store.browserAgentPage.status.browser_use.error"></div>
108
- </div>
109
- </div>
110
- </template>
111
-
112
- <div class="browser-agent-actions">
113
- <button class="btn btn-field" @click="openModal('/plugins/_model_config/webui/main.html')">
114
- Open Presets
115
- </button>
116
- <button class="btn btn-field" @click="openModal('/plugins/_model_config/webui/api-keys.html')">
117
- Open API Keys
118
- </button>
119
- </div>
120
- </div>
121
- </template>
122
- </div>
123
-
124
- <style>
125
- .browser-agent-page {
126
- display: flex;
127
- flex-direction: column;
128
- gap: 14px;
129
- }
130
-
131
- .browser-agent-grid {
132
- display: grid;
133
- gap: 12px;
134
- grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
135
- }
136
-
137
- .browser-agent-card {
138
- display: flex;
139
- flex-direction: column;
140
- gap: 10px;
141
- padding: 14px;
142
- background: var(--color-input);
143
- border: 1px solid var(--color-border);
144
- border-radius: 10px;
145
- }
146
-
147
- .browser-agent-card.error {
148
- border-color: rgba(214, 40, 40, 0.35);
149
- }
150
-
151
- .browser-agent-actions {
152
- display: flex;
153
- gap: 8px;
154
- flex-wrap: wrap;
155
- }
156
-
157
- .browser-agent-select-label {
158
- font-size: 0.78rem;
159
- opacity: 0.75;
160
- }
161
-
162
- .browser-agent-select {
163
- width: 100%;
164
- min-height: 36px;
165
- padding: 8px 10px;
166
- border-radius: 8px;
167
- border: 1px solid var(--color-border);
168
- background: var(--color-bg);
169
- color: var(--color-text);
170
- }
171
-
172
- .browser-agent-select:disabled {
173
- opacity: 0.7;
174
- cursor: wait;
175
- }
176
-
177
- .status-row {
178
- display: flex;
179
- align-items: flex-start;
180
- justify-content: space-between;
181
- gap: 12px;
182
- font-size: 0.84rem;
183
- }
184
-
185
- .status-key {
186
- opacity: 0.7;
187
- min-width: 64px;
188
- }
189
-
190
- .status-value {
191
- text-align: right;
192
- word-break: break-word;
193
- }
194
-
195
- .status-badge {
196
- padding: 2px 8px;
197
- border-radius: 999px;
198
- font-size: 0.76rem;
199
- font-weight: 600;
200
- border: 1px solid transparent;
201
- }
202
-
203
- .status-badge.ok {
204
- color: #1b5e20;
205
- background: rgba(46, 125, 50, 0.14);
206
- border-color: rgba(46, 125, 50, 0.24);
207
- }
208
-
209
- .status-badge.warn {
210
- color: #8a6100;
211
- background: rgba(191, 144, 0, 0.14);
212
- border-color: rgba(191, 144, 0, 0.24);
213
- }
214
-
215
- .status-badge.fail {
216
- color: #9f1239;
217
- background: rgba(190, 24, 93, 0.12);
218
- border-color: rgba(190, 24, 93, 0.24);
219
- }
220
-
221
- .mono {
222
- font-family: var(--font-mono);
223
- font-size: 0.78rem;
224
- }
225
-
226
- option {
227
- background: var(--color-input);
228
- color: var(--color-text);
229
- }
230
- </style>
231
-</body>
232
-</html>
plugins/_browser_agent/webui/thumbnail.jpg
Binary files a/plugins/_browser_agent/webui/thumbnail.jpg and /dev/null differ
plugins/_model_config/README.md
-1
@@ -23,7 +23,6 @@ This plugin centralizes model selection and model-related settings for the appli
23
- Allows a chat context to store a temporary override or preset reference in context data.
24
- **Model object construction**
25
- Builds `ModelConfig` objects and the runtime chat, utility, and embedding wrappers used elsewhere in the app.
26
- - Note: Browser model wiring now lives in the `_browser_agent` plugin.
26
- **API key validation**
27
- Reports configured providers that still require API keys.
28
requirements.txt
-1
@@ -1,6 +1,5 @@
1
a2wsgi==1.10.8
2
ansio==0.0.1
3
-browser-use==0.5.11
3
docker==7.1.0
4
duckduckgo-search==6.1.12
5
faiss-cpu==1.11.0
skills/a0-browser-ext/SKILL.md
new
+106
@@ -0,0 +1,106 @@
1
+---
2
+name: a0-browser-ext
3
+description: Create, inspect, install, and safely maintain Chrome extensions for Agent Zero's built-in Browser plugin.
4
+tags: ["agent-zero", "browser", "chrome-extension", "playwright", "manifest-v3"]
5
+---
6
+
7
+# Agent Zero Browser Extensions
8
+
9
+Use this skill when the user wants to create a new Browser extension, modify an existing extension, or install a Chrome Web Store extension for Agent Zero's direct `_browser` plugin.
10
+
11
+## Operating Model
12
+
13
+- Agent Zero loads Browser extensions from unpacked directories.
14
+- Create user-owned extensions under `/a0/usr/browser-extensions/<extension-slug>/`.
15
+- Browser extension paths must be visible inside the Docker runtime. Prefer `/a0/usr/browser-extensions/...` paths over host-only paths.
16
+- The Browser puzzle menu can open "My Browser Extensions", seed a "+ Create New with A0" request, and install Chrome Web Store URLs.
17
+- Chrome Web Store installs are converted into unpacked extension folders before Browser can load them.
18
+- Extension setting changes restart active Browser runtimes so Playwright can relaunch Chromium with the extension arguments.
19
+
20
+## Safety First
21
+
22
+Browser extensions run inside the Docker browser sandbox, but malicious or buggy extensions can still damage that sandboxed environment, corrupt browser profiles, exfiltrate page data visible to the Browser, or make browsing unreliable.
23
+
24
+Before creating or installing an extension:
25
+
26
+- State the requested behavior in one sentence.
27
+- List the minimum permissions and host permissions needed.
28
+- Avoid `<all_urls>` unless the user explicitly needs broad page access.
29
+- Avoid remote code, eval-style execution, hidden credential collection, and broad network access.
30
+- Do not store secrets in extension files.
31
+- Prefer content scripts for page-local behavior and service workers for coordination.
32
+- Tell the user when an extension can read or modify page content.
33
+
34
+## Create New Extension
35
+
36
+1. Ask for the extension name, user-visible purpose, target websites, and whether it needs a popup, content script, background service worker, options page, or side panel.
37
+2. Choose a lowercase slug such as `reader-highlighter`.
38
+3. Create `/a0/usr/browser-extensions/<slug>/manifest.json`.
39
+4. Add only the files the extension actually needs.
40
+5. Validate JSON syntax and confirm `manifest_version` is `3`.
41
+6. Keep generated code small, readable, and easy for the user to audit.
42
+7. After creating the folder, tell the user to open Browser's puzzle menu, use "Browser Extension Settings", enable extensions, and include the new folder path if it is not already enabled.
43
+
44
+Minimal Manifest V3 starter:
45
+
46
+```json
47
+{
48
+ "manifest_version": 3,
49
+ "name": "Agent Zero Example Extension",
50
+ "version": "0.1.0",
51
+ "description": "Small, auditable Browser extension created with Agent Zero.",
52
+ "permissions": [],
53
+ "host_permissions": [],
54
+ "action": {
55
+ "default_title": "A0 Extension"
56
+ }
57
+}
58
+```
59
+
60
+Content script starter:
61
+
62
+```json
63
+{
64
+ "manifest_version": 3,
65
+ "name": "Agent Zero Page Helper",
66
+ "version": "0.1.0",
67
+ "description": "Adds a small page helper for specific sites.",
68
+ "permissions": [],
69
+ "host_permissions": ["https://example.com/*"],
70
+ "content_scripts": [
71
+ {
72
+ "matches": ["https://example.com/*"],
73
+ "js": ["content.js"],
74
+ "run_at": "document_idle"
75
+ }
76
+ ]
77
+}
78
+```
79
+
80
+## Install From Chrome Web Store
81
+
82
+If the user gives a Chrome Web Store URL or extension id:
83
+
84
+1. Confirm they understand the sandbox warning.
85
+2. Extract the 32-character extension id from the URL.
86
+3. Prefer the Browser puzzle menu's URL installer for direct installs.
87
+4. If installing manually, download the CRX from Chrome's update service, extract the ZIP payload safely, and place it under `/a0/usr/browser-extensions/chrome-web-store/<extension-id>/`.
88
+5. Inspect `manifest.json` and summarize name, version, permissions, host permissions, and suspicious capabilities.
89
+6. Enable only after the user accepts the risk.
90
+
91
+Common URL shapes:
92
+
93
+```text
94
+https://chromewebstore.google.com/detail/name/<extension-id>
95
+https://chrome.google.com/webstore/detail/name/<extension-id>
96
+<extension-id>
97
+```
98
+
99
+## Review Checklist
100
+
101
+- `manifest.json` parses cleanly.
102
+- Every permission has a reason.
103
+- Host matches are specific.
104
+- No credential scraping, hidden data upload, or remote executable code.
105
+- UI text is concise and tells the truth.
106
+- The extension can be removed by deleting its folder from `/a0/usr/browser-extensions/` and removing the path from Browser settings.
skills/a0-development/SKILL.md
+1
-1
@@ -706,7 +706,7 @@ The framework ships with these core plugins in `/a0/plugins/`:
706
| `_memory` | Persistent vector memory system |
707
| `_text_editor` | File read/write/patch with line numbers |
708
| `_model_config` | LLM model selection and configuration |
709
-| `_browser_agent` | Browser automation and web interaction |
709
+| `_browser` | Direct browser automation and WebUI viewing |
710
| `_infection_check` | Prompt injection safety checks |
711
| `_error_retry` | Retry on critical exceptions |
712
| `_email_integration` | Email communication via IMAP/SMTP |
tests/test_browser_agent_regressions.py
+382
-45
@@ -1,74 +1,404 @@
1
-import asyncio
2
-import importlib
3
-import json
1
import sys
2
+import threading
3
from pathlib import Path
4
from types import SimpleNamespace
5
6
+import pytest
7
+
8
9
PROJECT_ROOT = Path(__file__).resolve().parents[1]
10
if str(PROJECT_ROOT) not in sys.path:
11
sys.path.insert(0, str(PROJECT_ROOT))
12
13
-import plugins._browser_agent.helpers.browser_use_monkeypatch as browser_use_monkeypatch
14
-import plugins._browser_agent.tools.browser_agent as browser_agent_module
13
+from plugins._browser.helpers.config import (
14
+ build_browser_launch_config,
15
+ get_browser_model_preset_options,
16
+ normalize_browser_config,
17
+ resolve_browser_model_selection,
18
+)
19
+from plugins._browser.helpers.extension_manager import (
20
+ _crx_zip_payload,
21
+ parse_chrome_web_store_extension_id,
22
+)
23
+from plugins._browser.helpers.runtime import normalize_url
24
+import plugins._browser.hooks as browser_hooks_module
25
+import plugins._browser.tools.browser as browser_tool_module
26
+import plugins._browser.api.ws_browser as ws_browser_module
27
+
28
+
29
+def test_browser_url_normalization_matches_address_bar_hosts():
30
+ assert normalize_url("localhost:3000") == "http://localhost:3000/"
31
+ assert normalize_url("127.0.0.1:8000/path") == "http://127.0.0.1:8000/path"
32
+ assert normalize_url("novinky.cz") == "https://novinky.cz/"
33
+ assert normalize_url("https://example.com") == "https://example.com/"
34
+ assert normalize_url("about:blank") == "about:blank"
35
+
36
+
37
+def test_browser_config_normalizes_extension_paths(tmp_path):
38
+ extension_dir = tmp_path / "extension"
39
+ extension_dir.mkdir()
40
+
41
+ config = normalize_browser_config(
42
+ {
43
+ "extensions_enabled": 1,
44
+ "extension_paths": [str(extension_dir), "", " ", str(extension_dir)],
45
+ }
46
+ )
47
+
48
+ assert config == {
49
+ "extensions_enabled": True,
50
+ "extension_paths": [str(extension_dir)],
51
+ "model_preset": "",
52
+ }
53
+
54
+
55
+def test_browser_config_normalizes_model_preset():
56
+ assert normalize_browser_config({"model_preset": " Research "})["model_preset"] == "Research"
57
+ assert "model" not in normalize_browser_config({"model": "main"})
58
+
59
+
60
+def test_browser_model_selection_uses_presets(monkeypatch):
61
+ import plugins._browser.helpers.config as browser_config_module
62
+ from plugins._model_config.helpers import model_config
63
64
+ monkeypatch.setattr(
65
+ browser_config_module,
66
+ "get_browser_config",
67
+ lambda agent=None: {"model_preset": "Research", "extensions_enabled": False, "extension_paths": []},
68
+ )
69
+ monkeypatch.setattr(
70
+ model_config,
71
+ "get_preset_by_name",
72
+ lambda name: {
73
+ "name": "Research",
74
+ "chat": {"provider": "openrouter", "name": "example/model"},
75
+ } if name == "Research" else None,
76
+ )
77
+
78
+ selection = resolve_browser_model_selection(SimpleNamespace())
79
+
80
+ assert selection["source_kind"] == "preset"
81
+ assert selection["config"] == {"provider": "openrouter", "name": "example/model"}
82
17
-def test_gemini_clean_and_conform_normalizes_known_single_action_shapes():
18
- raw = (
19
- '{"action":['
20
- '{"complete_task":{"title":"T","response":"R","page_summary":"S"}}'
21
- ']}'
83
+
84
+def test_browser_model_selection_falls_back_to_main_for_missing_preset(monkeypatch):
85
+ from plugins._model_config.helpers import model_config
86
+
87
+ monkeypatch.setattr(model_config, "get_preset_by_name", lambda name: None)
88
+ monkeypatch.setattr(
89
+ model_config,
90
+ "get_chat_model_config",
91
+ lambda agent=None: {"provider": "openrouter", "name": "main/model"},
92
)
93
24
- cleaned = browser_use_monkeypatch.gemini_clean_and_conform(raw)
94
+ selection = resolve_browser_model_selection(SimpleNamespace(), {"model_preset": "Missing"})
95
+
96
+ assert selection["source_kind"] == "main"
97
+ assert selection["preset_status"] == "missing"
98
+ assert selection["config"] == {"provider": "openrouter", "name": "main/model"}
99
+
100
+
101
+def test_browser_model_preset_options_include_missing_selected(monkeypatch):
102
+ from plugins._model_config.helpers import model_config
103
+
104
+ monkeypatch.setattr(
105
+ model_config,
106
+ "get_presets",
107
+ lambda: [{"name": "Balance", "chat": {"provider": "openrouter", "name": "model"}}],
108
+ )
109
+
110
+ options = get_browser_model_preset_options(settings={"model_preset": "Deleted"})
111
+
112
+ assert options[-1]["name"] == "Deleted"
113
+ assert options[-1]["missing"] is True
114
26
- assert cleaned is not None
27
- parsed = json.loads(cleaned)
28
- assert parsed["action"] == [
115
+
116
+def test_browser_launch_config_switches_to_chromium_for_extensions(tmp_path):
117
+ extension_dir = tmp_path / "extension"
118
+ extension_dir.mkdir()
119
+
120
+ launch = build_browser_launch_config(
121
{
30
- "done": {
31
- "success": True,
32
- "data": {
33
- "title": "T",
34
- "response": "R",
35
- "page_summary": "S",
36
- },
37
- }
122
+ "extensions_enabled": True,
123
+ "extension_paths": [str(extension_dir)],
124
+ }
125
+ )
126
+
127
+ assert launch["browser_mode"] == "chromium_extensions"
128
+ assert launch["channel"] == "chromium"
129
+ assert launch["requires_full_browser"] is True
130
+ assert launch["extensions"]["active"] is True
131
+ assert any(arg.startswith("--load-extension=") for arg in launch["args"])
132
+ assert "--headless=new" not in launch["args"]
133
+
134
+
135
+def test_browser_extension_manager_parses_web_store_urls():
136
+ extension_id = "a" * 32
137
+
138
+ assert parse_chrome_web_store_extension_id(extension_id) == extension_id
139
+ assert (
140
+ parse_chrome_web_store_extension_id(
141
+ f"https://chromewebstore.google.com/detail/example/{extension_id}"
142
+ )
143
+ == extension_id
144
+ )
145
+ assert (
146
+ parse_chrome_web_store_extension_id(
147
+ f"https://chrome.google.com/webstore/detail/example/{extension_id}?hl=en"
148
+ )
149
+ == extension_id
150
+ )
151
+
152
+
153
+def test_browser_extension_manager_extracts_crx3_zip_payload():
154
+ payload = b"PK\x03\x04zip-payload"
155
+ header = b"metadata"
156
+ crx = b"Cr24" + (3).to_bytes(4, "little") + len(header).to_bytes(4, "little") + header + payload
157
+
158
+ assert _crx_zip_payload(crx) == payload
159
+
160
+
161
+def test_browser_extension_menu_exposes_agent_and_url_paths():
162
+ html = (PROJECT_ROOT / "plugins" / "_browser" / "webui" / "main.html").read_text(
163
+ encoding="utf-8"
164
+ )
165
+ skill = PROJECT_ROOT / "skills" / "a0-browser-ext" / "SKILL.md"
166
+
167
+ assert "+ Create New with A0" in html
168
+ assert "Chrome Web Store URL" in html
169
+ assert "My Browser Extensions" in html
170
+ assert "malicious or buggy extensions" in html
171
+ assert skill.exists()
172
+
173
+
174
+def test_browser_save_plugin_config_restarts_runtimes_on_change(monkeypatch, tmp_path):
175
+ extension_dir = tmp_path / "extension"
176
+ extension_dir.mkdir()
177
+ restarted = []
178
+
179
+ monkeypatch.setattr(
180
+ browser_hooks_module,
181
+ "_load_saved_browser_config",
182
+ lambda project_name="", agent_profile="": {
183
+ "extensions_enabled": False,
184
+ "extension_paths": [],
185
},
39
- ]
186
+ )
187
+ monkeypatch.setattr(
188
+ browser_hooks_module,
189
+ "close_all_runtimes_sync",
190
+ lambda: restarted.append(True),
191
+ )
192
193
+ result = browser_hooks_module.save_plugin_config(
194
+ {
195
+ "extensions_enabled": True,
196
+ "extension_paths": [str(extension_dir)],
197
+ },
198
+ project_name="",
199
+ agent_profile="",
200
+ )
201
42
-class DummyBrowserSession:
43
- def __init__(self) -> None:
44
- self.kill_called = False
45
- self.close_called = False
202
+ assert result["extensions_enabled"] is True
203
+ assert result["extension_paths"] == [str(extension_dir)]
204
+ assert result["model_preset"] == ""
205
+ assert restarted == [True]
206
47
- async def kill(self) -> None:
48
- self.kill_called = True
207
50
- async def close(self) -> None:
51
- self.close_called = True
208
+def test_browser_save_plugin_config_does_not_restart_runtimes_for_preset_only(monkeypatch):
209
+ restarted = []
210
211
+ monkeypatch.setattr(
212
+ browser_hooks_module,
213
+ "_load_saved_browser_config",
214
+ lambda project_name="", agent_profile="": {
215
+ "extensions_enabled": False,
216
+ "extension_paths": [],
217
+ "model_preset": "",
218
+ },
219
+ )
220
+ monkeypatch.setattr(
221
+ browser_hooks_module,
222
+ "close_all_runtimes_sync",
223
+ lambda: restarted.append(True),
224
+ )
225
+
226
+ result = browser_hooks_module.save_plugin_config(
227
+ {
228
+ "extensions_enabled": False,
229
+ "extension_paths": [],
230
+ "model_preset": "Research",
231
+ },
232
+ project_name="",
233
+ agent_profile="",
234
+ )
235
+
236
+ assert result["model_preset"] == "Research"
237
+ assert restarted == []
238
+
239
+
240
+@pytest.mark.asyncio
241
+async def test_browser_tool_dispatches_direct_actions(monkeypatch):
242
+ calls = []
243
+
244
+ class FakeRuntime:
245
+ async def call(self, method, *args):
246
+ calls.append((method, args))
247
+ if method == "content":
248
+ return {"document": "[link 1] Example"}
249
+ return {"ok": True, "method": method, "args": args}
250
+
251
+ async def fake_get_runtime(context_id, create=True):
252
+ assert context_id == "ctx"
253
+ return FakeRuntime()
254
+
255
+ monkeypatch.setattr(browser_tool_module, "get_runtime", fake_get_runtime)
256
+ agent = SimpleNamespace(context=SimpleNamespace(id="ctx"))
257
+ tool = browser_tool_module.Browser(
258
+ agent=agent,
259
+ name="browser",
260
+ method=None,
261
+ args={},
262
+ message="",
263
+ loop_data=None,
264
+ )
265
+
266
+ response = await tool.execute(action="content", browser_id=1)
267
+
268
+ assert response.message == "[link 1] Example"
269
+ assert calls == [("content", (1, None))]
270
+
271
+
272
+@pytest.mark.asyncio
273
+async def test_browser_viewer_subscribe_unregisters_stream(monkeypatch):
274
+ class FakeRuntime:
275
+ def __init__(self) -> None:
276
+ self.opened = False
277
+
278
+ async def call(self, method, *args):
279
+ if method == "list":
280
+ if self.opened:
281
+ return {
282
+ "browsers": [{"id": 1, "currentUrl": "about:blank", "title": ""}],
283
+ "last_interacted_browser_id": 1,
284
+ }
285
+ return {"browsers": [], "last_interacted_browser_id": None}
286
+ if method == "open":
287
+ self.opened = True
288
+ return {"id": 1, "state": {"id": 1, "currentUrl": "about:blank"}}
289
+ raise AssertionError(method)
290
+
291
+ async def fake_get_runtime(context_id, create=True):
292
+ assert context_id == "ctx"
293
+ return FakeRuntime()
294
+
295
+ monkeypatch.setattr(ws_browser_module, "get_runtime", fake_get_runtime)
296
+ monkeypatch.setattr(
297
+ ws_browser_module.AgentContext,
298
+ "get",
299
+ staticmethod(lambda context_id: SimpleNamespace(id=context_id)),
300
+ )
301
+
302
+ handler = ws_browser_module.WsBrowser(
303
+ SimpleNamespace(),
304
+ threading.RLock(),
305
+ manager=None,
306
+ )
307
+
308
+ result = await handler.process(
309
+ "browser_viewer_subscribe",
310
+ {"context_id": "ctx", "correlationId": "c1"},
311
+ "sid-1",
312
+ )
313
54
-class DummyAgent:
55
- def __init__(self) -> None:
56
- self.context = SimpleNamespace(id="ctx", task=None)
314
+ assert result["context_id"] == "ctx"
315
+ assert ("sid-1", "ctx") in ws_browser_module.WsBrowser._streams
316
317
+ await handler.on_disconnect("sid-1")
318
59
-def test_browser_session_teardown_prefers_kill_for_keep_alive_sessions():
60
- state = browser_agent_module.State(DummyAgent())
61
- session = DummyBrowserSession()
62
- state.browser_session = session
319
+ assert ("sid-1", "ctx") not in ws_browser_module.WsBrowser._streams
320
64
- state.kill_task()
321
66
- assert session.kill_called is True
67
- assert session.close_called is False
322
+@pytest.mark.asyncio
323
+async def test_browser_viewer_viewport_input_dispatches_resize(monkeypatch):
324
+ calls = []
325
326
+ class FakeRuntime:
327
+ async def call(self, method, *args, **kwargs):
328
+ calls.append((method, args, kwargs))
329
+ return {"ok": True, "method": method, "args": args}
330
70
-def test_browser_cleanup_extensions_follow_new_extensible_path_layout():
71
- extension = importlib.import_module("helpers.extension")
331
+ async def fake_get_runtime(context_id, create=True):
332
+ assert context_id == "ctx"
333
+ assert create is False
334
+ return FakeRuntime()
335
+
336
+ monkeypatch.setattr(ws_browser_module, "get_runtime", fake_get_runtime)
337
+
338
+ handler = ws_browser_module.WsBrowser(
339
+ SimpleNamespace(),
340
+ threading.RLock(),
341
+ manager=None,
342
+ )
343
+
344
+ result = await handler.process(
345
+ "browser_viewer_input",
346
+ {
347
+ "context_id": "ctx",
348
+ "browser_id": 7,
349
+ "input_type": "viewport",
350
+ "width": 1280,
351
+ "height": 720,
352
+ },
353
+ "sid-1",
354
+ )
355
+
356
+ assert result == {"state": {"ok": True, "method": "set_viewport", "args": (7, 1280, 720)}}
357
+ assert calls == [("set_viewport", (7, 1280, 720), {})]
358
+
359
+
360
+@pytest.mark.asyncio
361
+async def test_browser_viewer_wheel_input_dispatches_scroll(monkeypatch):
362
+ calls = []
363
+
364
+ class FakeRuntime:
365
+ async def call(self, method, *args, **kwargs):
366
+ calls.append((method, args, kwargs))
367
+ return {"ok": True, "method": method, "args": args}
368
+
369
+ async def fake_get_runtime(context_id, create=True):
370
+ assert context_id == "ctx"
371
+ assert create is False
372
+ return FakeRuntime()
373
+
374
+ monkeypatch.setattr(ws_browser_module, "get_runtime", fake_get_runtime)
375
+
376
+ handler = ws_browser_module.WsBrowser(
377
+ SimpleNamespace(),
378
+ threading.RLock(),
379
+ manager=None,
380
+ )
381
+
382
+ result = await handler.process(
383
+ "browser_viewer_input",
384
+ {
385
+ "context_id": "ctx",
386
+ "browser_id": 3,
387
+ "input_type": "wheel",
388
+ "x": 320,
389
+ "y": 480,
390
+ "delta_x": 0,
391
+ "delta_y": 640,
392
+ },
393
+ "sid-1",
394
+ )
395
+
396
+ assert result == {"state": {"ok": True, "method": "wheel", "args": (3, 320.0, 480.0, 0.0, 640.0)}}
397
+ assert calls == [("wheel", (3, 320.0, 480.0, 0.0, 640.0), {})]
398
+
399
+
400
+def test_browser_cleanup_extensions_follow_extensible_path_layout():
401
+ extension = __import__("helpers.extension", fromlist=["_get_extension_classes"])
402
remove_classes = extension._get_extension_classes( # type: ignore[attr-defined]
403
"_functions/agent/AgentContext/remove/start"
404
)
@@ -76,5 +406,12 @@ def test_browser_cleanup_extensions_follow_new_extensible_path_layout():
406
"_functions/agent/AgentContext/reset/start"
407
)
408
79
- assert any(cls.__name__ == "CleanupBrowserStateOnRemove" for cls in remove_classes)
80
- assert any(cls.__name__ == "CleanupBrowserStateOnReset" for cls in reset_classes)
409
+ assert any(cls.__name__ == "CleanupBrowserRuntimeOnRemove" for cls in remove_classes)
410
+ assert any(cls.__name__ == "CleanupBrowserRuntimeOnReset" for cls in reset_classes)
411
+
412
+
413
+def test_legacy_browser_dependency_is_removed():
414
+ assert not (PROJECT_ROOT / "plugins" / ("_browser" + "_agent")).exists()
415
+ assert ("browser" + "-use") not in (PROJECT_ROOT / "requirements.txt").read_text(
416
+ encoding="utf-8"
417
+ )
tests/test_webui_extension_surfaces.py
+27
-6
@@ -10,7 +10,7 @@ from typing import Iterator
10
import pytest
11
from flask import Flask
12
13
-PROJECT_ROOT = Path(__file__).resolve().parents[2]
13
+PROJECT_ROOT = Path(__file__).resolve().parents[1]
14
if str(PROJECT_ROOT) not in sys.path:
15
sys.path.insert(0, str(PROJECT_ROOT))
16
@@ -75,6 +75,19 @@ def _temporary_probe_plugin(surface: str) -> Iterator[tuple[str, str]]:
75
dir=plugins_root,
76
) as temp_plugin_dir:
77
plugin_id = Path(temp_plugin_dir).name
78
+ (Path(temp_plugin_dir) / "plugin.yaml").write_text(
79
+ (
80
+ f"name: {plugin_id}\n"
81
+ f"title: {plugin_id}\n"
82
+ "description: Temporary WebUI surface probe.\n"
83
+ "version: 0.0.0\n"
84
+ "always_enabled: false\n"
85
+ ),
86
+ encoding="utf-8",
87
+ )
88
+ from helpers import cache
89
+
90
+ cache.clear("*(plugins)*")
91
probe_file = (
92
Path(temp_plugin_dir)
93
/ "extensions"
@@ -91,7 +104,10 @@ def _temporary_probe_plugin(surface: str) -> Iterator[tuple[str, str]]:
104
),
105
encoding="utf-8",
106
)
94
- yield plugin_id, probe_file.name
107
+ try:
108
+ yield plugin_id, probe_file.name
109
+ finally:
110
+ cache.clear("*(plugins)*")
111
112
113
@pytest.mark.asyncio
@@ -117,8 +133,13 @@ async def test_webui_surface_extension_point_end_to_end(
133
f"{plugin_id}/extensions/webui/{surface}/{probe_file_name}"
134
)
135
120
- assert any(
121
- extension.get("plugin_id") == plugin_id
122
- and str(extension.get("path", "")).replace("\\", "/").endswith(expected_suffix)
136
+ extension_paths = [
137
+ str(
138
+ extension.get("path", "")
139
+ if isinstance(extension, dict)
140
+ else extension
141
+ ).replace("\\", "/")
142
for extension in extensions
124
- )
143
+ ]
144
+
145
+ assert any(path.endswith(expected_suffix) for path in extension_paths)
webui/js/modals.js
+62
-10
@@ -5,6 +5,20 @@ import { callJsExtensions } from "/js/extensions.js";
5
// Modal functionality
6
const modalStack = [];
7
8
+function findModalIndexByPath(modalPath) {
9
+ return modalStack.findIndex((modal) => modal.path === modalPath);
10
+}
11
+
12
+function focusModal(modalPath) {
13
+ const modalIndex = findModalIndexByPath(modalPath);
14
+ if (modalIndex === -1) return false;
15
+ if (modalIndex === modalStack.length - 1) return true;
16
+ const [modal] = modalStack.splice(modalIndex, 1);
17
+ modalStack.push(modal);
18
+ updateModalZIndexes();
19
+ return true;
20
+}
21
+
22
function getModalScrollElement(modal) {
23
return modal?.element?.querySelector(".modal-scroll");
24
}
@@ -38,6 +52,15 @@ backdrop.style.display = "none";
52
backdrop.style.backdropFilter = "blur(5px)";
53
document.body.appendChild(backdrop);
54
55
+function modalSuppressesBackdrop(modal) {
56
+ const path = String(modal?.path || "");
57
+ return path === "/plugins/_browser/webui/main.html"
58
+ || path === "plugins/_browser/webui/main.html"
59
+ || modal?.element?.classList?.contains("modal-floating")
60
+ || modal?.element?.classList?.contains("modal-no-backdrop")
61
+ || modal?.inner?.classList?.contains("modal-no-backdrop");
62
+}
63
+
64
// Function to update z-index for all modals and backdrop
65
function updateModalZIndexes() {
66
// Base z-index for modals
@@ -51,20 +74,26 @@ function updateModalZIndexes() {
74
modal.element.style.zIndex = baseZIndex + index * 20;
75
});
76
54
- // Always show backdrop
77
+ const backdropModalStack = modalStack.filter((modal) => !modalSuppressesBackdrop(modal));
78
+
79
+ if (backdropModalStack.length === 0) {
80
+ backdrop.style.display = "none";
81
+ return;
82
+ }
83
+
84
backdrop.style.display = "block";
85
+ backdrop.style.backdropFilter = "blur(5px)";
86
+ backdrop.style.backgroundColor = "";
87
57
- if (modalStack.length > 1) {
58
- // For multiple modals, position backdrop between the top two
88
+ if (backdropModalStack.length === modalStack.length && modalStack.length > 1) {
89
const topModalIndex = modalStack.length - 1;
60
- const previousModalZIndex = baseZIndex + (topModalIndex - 1) * 20;
61
- backdrop.style.zIndex = previousModalZIndex + 10;
62
- } else if (modalStack.length === 1) {
63
- // For single modal, position backdrop below it
64
- backdrop.style.zIndex = baseZIndex - 1;
90
+ backdrop.style.zIndex = baseZIndex + (topModalIndex - 1) * 20 + 10;
91
} else {
66
- // No modals, hide backdrop
67
- backdrop.style.display = "none";
92
+ const topBackdropModal = backdropModalStack[backdropModalStack.length - 1];
93
+ const topBackdropModalIndex = modalStack.indexOf(topBackdropModal);
94
+ backdrop.style.zIndex = topBackdropModalIndex > 0
95
+ ? baseZIndex + (topBackdropModalIndex - 1) * 20 + 10
96
+ : baseZIndex - 1;
97
}
98
}
99
@@ -213,6 +242,26 @@ export async function openModal(modalPath, beforeClose = null) {
242
});
243
}
244
245
+export function isModalOpen(modalPath) {
246
+ return findModalIndexByPath(modalPath) !== -1;
247
+}
248
+
249
+export async function ensureModalOpen(modalPath, beforeClose = null) {
250
+ if (focusModal(modalPath)) return null;
251
+ return openModal(modalPath, beforeClose);
252
+}
253
+
254
+export async function toggleModal(modalPath, beforeClose = null) {
255
+ if (!isModalOpen(modalPath)) {
256
+ return openModal(modalPath, beforeClose);
257
+ }
258
+ while (isModalOpen(modalPath)) {
259
+ const closed = await closeModal(modalPath);
260
+ if (closed === false) return false;
261
+ }
262
+ return true;
263
+}
264
+
265
// Function to close modal
266
export async function closeModal(modalPath = null) {
267
if (modalStack.length === 0) return;
@@ -369,3 +418,6 @@ document.addEventListener("keydown", (e) => {
418
globalThis.openModal = openModal;
419
globalThis.closeModal = closeModal;
420
globalThis.scrollModal = scrollModal;
421
+globalThis.isModalOpen = isModalOpen;
422
+globalThis.ensureModalOpen = ensureModalOpen;
423
+globalThis.toggleModal = toggleModal;