refactor: extract browser agent into built-in plugin
Move the Browser Agent/browser-use stack into a tracked built-in `_browser_agent` plugin while preserving the current model/config flow. Changes: - add built-in `_browser_agent` plugin with tool, helpers, prompts, assets, status API, and WebUI message/status surfaces - move browser-use wrapper and monkeypatch ownership out of `models.py` into plugin helper code - keep browser model resolution on the `_model_config` path and continue using the effective Main Model / chat model config - remove stale Browser HTTP Headers UI and outdated browser-model wording - make Playwright runtime resolution bootstrap-only and stop installing browser binaries on demand - switch browser log rendering to plugin-owned WebUI extension handling with core fallback compatibility for old logs - delete obsolete core browser tool/helpers/prompts/assets after plugin resolution is in place - update docs to reflect built-in plugin ownership and Main Model browser behavior feat(browser-agent): improve lifecycle reliability, LLM compatibility, and local-dev bootstrap - **Lifecycle Reliability**: Implemented stale Chromium lock file cleanup and one-shot ephemeral profile fallback to prevent CDP connection hangs and profile-locked crashes. - **LLM Compatibility**: Added OpenRouter/Gemini shims including schema compaction, system instruction injection, and JSON output normalization for `browser-use` validation. - **Cache Alignment**: Reverted Playwright browser cache to `tmp/playwright` for parity with core behavior and updated Docker scripts and documentation accordingly. - **Local-Dev Bootstrap**: Added a plugin `install()` hook to automate Python dependency setup and transitioned bootstrap feedback to UI notifications. - **Plugin Config**: Set `always_enabled: false` to allow users to opt-out or bring their own browser providers. - **Testing**: Expanded test suite to cover lifecycle edge cases, LLM normalization, and bootstrap logic. rework clean up browser agent sessions on reset/removal Force browser-agent teardown to use BrowserSession.kill() so keep_alive sessions do not leave Chromium running and keep the profile locked. Add centralized browser state cleanup for reset/context removal and regression tests to cover the new teardown path and prevent SingletonLock fallbacks from stale sessions. add browser agent thumbnail restore local dev for Playwright in browser agent via hook - Introduced a new hook to bootstrap Playwright for local development, ensuring the Chromium headless shell is installed when in development mode. - Updated the Playwright helper to call the new hook if the binary is not found during the binary check. - Added tests to verify the correct installation and behavior of the Playwright binary in local development scenarios. proper install notifications for Playwright in local dev - Added notification management for Playwright bootstrap process, including info, success, and error notifications. - Enhanced the `bootstrap_local_dev_playwright` function to notify users about the installation status of the Playwright runtime. - Updated tests to verify the correct notifications are sent during the Playwright installation process. rework browser state cleanup extensions for agent context removal and reset - Added `CleanupBrowserStateOnRemove` and `CleanupBrowserStateOnReset` extensions to handle browser state cleanup when an agent context is removed or reset. - Updated `cleanup_browser_agent_state` function to utilize new protocols for better type safety and clarity. - Enhanced the `State` class to ensure proper cleanup of browser sessions and user data directories. - Introduced tests to validate the cleanup functionality and ensure browser sessions are correctly managed during agent lifecycle events. enhance Playwright cache handling - Updated Playwright helper to include support for a new cache directory at `.cache/ms-playwright`, aligning with standard cache locations. - Modified the `get_playwright_search_dirs` function to incorporate the new cache path. - Added a new test to verify the retrieval of the Playwright binary from the repository's `.cache/ms-playwright` directory, ensuring proper functionality across different cache locations. - Updated existing tests to reflect changes in cache path handling. update docs and README.md restore local dev bootstrap and async-safe teardown Keep Browser Agent bootstrap local in development, matching pre-extraction behavior, by removing the RFC filesystem hop from Playwright cache setup. Restore on-demand local-dev bootstrap through plugin hooks before browser-use import, so missing browser packages and runtime can be installed on first use. Rework browser session teardown to be async-safe during reset and cleanup, avoiding nested event loop shutdown failures while still terminating the browser worker cleanly. Also remove redundant browser-use patch application and add regression tests covering local bootstrap, reset-time async shutdown, and lifecycle cleanup. playwright bootstrap via plugin hooks [Fixed][P1] The hook-based restore was architecturally in the right place, but it was too late to recover a missing local browser_use / playwright install. The browser tool imports browser_use at module load time, and tool loading silently skips modules that fail to import, so the plugin install() hook could never rescue the first browser invocation if those packages were absent. I fixed that by calling the plugin install hook before importing browser_use in local development: browser_use.py, browser_use.py, agent.py, test_browser_agent_playwright_bootstrap.py. [Fixed][P3] The Browser Agent UI and runtime error text still claimed on-demand install did not exist, which contradicted the intended local-dev bootstrap path. I corrected both messages so they now describe the real behavior: main.html, playwright.py. rm tests restore old browser use logic files restore old browser working behaviour browser: migrate cleanup hooks to new extensible paths Update Browser Agent cleanup extensions to the new deep @extensible path layout introduced in 7e1d9ad2a4a3d186e337a89433ccc5128bdc754d, so AgentContext reset/remove hooks fire again after the framework migration. `This restores browser state cleanup when chats are reset or deleted, preventing keep-alive browser sessions from surviving context removal and leaving chats stuck until restart. The change also keeps the recent browser regressions covered with focused tests for Anthropic/OpenRouter action normalization, keep-alive session teardown via kill(), and extension discovery under the new _functions/... path structure.` Update messages.js update docs
Alessandro committed
Mar 21, 2026 at 14:38 UTC
db15bdb3bf42d30cf4a7d376e216bc8675693674
28 files changed
+566
-202
README.md
+9
-2
@@ -93,13 +93,20 @@ docker run -p 50001:80 agent0ai/agent-zero
93
94
95
96
+### Browser Agent
97
+
98
+- Browser automation is provided by the built-in `_browser_agent` plugin.
99
+- It uses the effective Main Model resolved by `_model_config`; there is no separate browser model slot.
100
+- Browser vision follows the Main Model's vision setting.
101
+- Playwright Chromium: **Docker** images ship the headless shell preinstalled. **Local development** installs it on first Browser Agent use via `ensure_playwright_binary()` in `plugins/_browser_agent/helpers/playwright.py` (into `tmp/playwright`); you can pre-install manually (see [Development Setup](docs/setup/dev-setup.md)) to skip the wait.
102
+
103
4. **Completely Customizable and Extensible**
104
105
- Almost nothing in this framework is hard-coded. Nothing is hidden. Everything can be extended or changed by the user.
106
- The whole behavior is defined by a system prompt in the **prompts/default/agent.system.md** file. Change this prompt and change the framework dramatically.
107
- The framework does not guide or limit the agent in any way. There are no hard-coded rails that agents have to follow.
108
- Every prompt, every small message template sent to the agent in its communication loop can be found in the **prompts/** folder and changed.
102
-- Every default tool can be found in the **python/tools/** folder and changed or copied to create new predefined tools.
109
+- Built-in tools live in the core **tools/** folder or in built-in plugins under **plugins/** and can be adapted or extended.
110
- **Automated configuration** via `A0_SET_` environment variables for deployment automation and easy setup.
111
112
@@ -238,7 +245,7 @@ docker run -p 50001:80 agent0ai/agent-zero
245
- Secrets management - agent can use credentials without seeing them
246
- Agent can copy paste messages and files without rewriting them
247
- LiteLLM global configuration field
241
-- Custom HTTP headers field for browser agent
248
+- Browser agent configuration improvements
249
- Progressive web app support
250
- Extra model params support for JSON
251
- Short IDs for files and memories to prevent LLM errors
docs/guides/mcp-setup.md
+1
-1
@@ -114,4 +114,4 @@ Community-tested and reliable MCP servers:
114
- **VSCode MCP** - IDE workflows
115
116
> [!TIP]
117
-> For browser automation tasks, MCP-based browser tools are more reliable than the built-in browser agent.
117
+> For browser automation tasks, the built-in Browser Agent plugin covers the default workflow. MCP-based browser tools are still useful when you need a different browser stack, remote browser control, or an alternative to the built-in Playwright Chromium (preinstalled in Docker; on demand via `ensure_playwright_binary()` in local dev).
docs/guides/projects.md
+1
-1
@@ -228,7 +228,7 @@ SMTP_PASSWORD=email_pwd_here
228
229
### Subagent Configuration
230
231
-Projects can enable or disable specific subagents (like the Browser Agent). This is configured via the UI and stored in `.a0proj/agents.json`.
231
+Projects can enable or disable specific subagents. This is configured via the UI and stored in `.a0proj/agents.json`. The Browser Agent is not a subagent; it is a built-in plugin.
232
233
### Knowledge Files
234
docs/guides/troubleshooting.md
+3
-3
@@ -26,8 +26,8 @@ Refer to the [Choosing your LLMs](../setup/installation.md#installing-and-using-
26
**7. How can I make Agent Zero retain memory between sessions?**
27
Use **Settings → Backup & Restore** and avoid mapping the entire `/a0` directory. See [How to update Agent Zero](../setup/installation.md#how-to-update-agent-zero).
28
29
-**8. My browser agent fails or is unreliable. What now?**
30
-The built-in browser agent is currently unstable on some systems. Use Skills or MCP alternatives such as Browser OS, Chrome DevTools, or Vercel's Agent Browser. See [MCP Setup](mcp-setup.md).
29
+**8. My browser agent fails or says Playwright is missing. What now?**
30
+The built-in Browser Agent is a plugin that uses the Main Model from `_model_config`. **Docker:** the Chromium headless shell is shipped preinstalled (typically under `/a0/tmp/playwright`). **Local development:** if the binary is missing, `ensure_playwright_binary()` in `plugins/_browser_agent/helpers/playwright.py` runs `playwright install chromium --only-shell` into `tmp/playwright` on first Browser Agent use (you may see UI notifications). To install ahead of time, run `PLAYWRIGHT_BROWSERS_PATH=tmp/playwright playwright install chromium --only-shell` after `pip install -r requirements.txt`. If you prefer an external browser stack, use MCP alternatives such as Browser OS, Chrome DevTools, or Playwright MCP. See [MCP Setup](mcp-setup.md).
31
32
**9. My secrets disappeared after a backup restore.**
33
Secrets are stored in `/a0/usr/secrets.env` and are not always included in backup archives. Copy them manually.
@@ -36,7 +36,7 @@ Secrets are stored in `/a0/usr/secrets.env` and are not always included in backu
36
- Join the Agent Zero [Skool](https://www.skool.com/agent-zero) or [Discord](https://discord.gg/B8KZKNsPpj) community.
37
38
**11. How do I adjust API rate limits?**
39
-Use the model rate limit fields in Settings (Chat/Utility/Browser model sections) to set request/input/output limits. These map to the model config limits (for example `limit_requests`, `limit_input`, `limit_output`).
39
+Use the model rate limit fields in Settings (Main Model and Utility Model sections) to set request/input/output limits. The Browser Agent inherits the Main Model limits. These map to the model config limits (for example `limit_requests`, `limit_input`, `limit_output`).
40
41
**12. My `code_execution_tool` doesn't work, what's wrong?**
42
- Ensure Docker is installed and running.
docs/guides/usage.md
+3
-1
@@ -127,7 +127,9 @@ Agent Zero's power comes from its ability to use [tools](../developer/architectu
127
- **Understand Tools:** Agent Zero includes default tools like knowledge (powered by SearXNG), code execution, and communication. Understand the capabilities of these tools and how to invoke them.
128
129
### Browser Agent Status & MCP Alternatives
130
-The built-in browser agent currently has dependency issues on some systems. If web automation is critical, prefer MCP-based browser tools instead:
130
+The built-in Browser Agent is provided by the `_browser_agent` plugin. It uses the effective Main Model from `_model_config`, including per-chat overrides and the Main Model vision flag. Playwright Chromium is preinstalled in **Docker**; in **local development** it is installed on demand when needed via `ensure_playwright_binary()` (see [Development Setup](../setup/dev-setup.md) to pre-install).
131
+
132
+If you need a different browser stack or want external browser tooling, MCP-based browser tools are still a strong option:
133
134
- **Browser OS MCP**
135
- **Chrome DevTools MCP**
docs/setup/dev-setup.md
+2
-2
@@ -67,9 +67,9 @@ Now when you select one of the python files in the project, you should see prope
67
3. Install dependencies. Run these two commands in the terminal:
68
```bash
69
pip install -r requirements.txt
70
-playwright install chromium
70
+PLAYWRIGHT_BROWSERS_PATH=tmp/playwright playwright install chromium --only-shell
71
```
72
-These will install all the python packages and browser binaries for playwright (browser agent).
72
+The first command installs Python dependencies. The second installs the Chromium headless shell into `tmp/playwright` ahead of time (same path in Docker: `/a0/tmp/playwright`). If you skip the second command, **local development** still downloads the shell on first Browser Agent use through `ensure_playwright_binary()` in `plugins/_browser_agent/helpers/playwright.py`. Pre-installing avoids that wait. **Docker** images ship the shell preinstalled; runtime install is for local dev when the binary is missing.
73
Errors in the code editor caused by missing packages should now be gone. If not, try reloading the window.
74
75
docs/setup/installation.md
+5
-3
@@ -330,17 +330,19 @@ The Settings page is the control center for selecting the Large Language Models
330
331
| LLM Role | Description |
332
| --- | --- |
333
-| `chat_llm` | This is the primary LLM used for conversations and generating responses. |
333
+| `chat_llm` | This is the primary LLM used for conversations, agent reasoning, tool use, and the built-in browser agent. Vision support controls browser vision and image understanding. |
334
| `utility_llm` | This LLM handles internal tasks like summarizing messages, managing memory, and processing internal prompts. Using a smaller, less expensive model here can improve efficiency. |
335
-| `browser_llm` | This LLM powers the browser agent for web navigation and interaction tasks. Vision support is recommended for better page understanding. |
335
| `embedding_llm` | The embedding model shipped with A0 runs on CPU and is responsible for generating embeddings used for memory retrieval and knowledge base lookups. Changing the `embedding_llm` will re-index all of A0's memory. |
336
337
**How to Change:**
338
339
1. Open Settings page in the Web UI.
341
-2. Choose the provider for the LLM for each role (Chat model, Utility model, Browser model, Embedding model) and write the model name.
340
+2. Choose the provider for the LLM for each role (Main Model, Utility Model, Embedding Model) and write the model name.
341
3. Click "Save" to apply the changes.
342
343
+> [!NOTE]
344
+> The Browser Agent does not have a separate model slot. It uses the effective Main Model resolved by `_model_config`, including per-chat overrides and the Main Model vision flag.
345
+
346
### Important Considerations
347
348
#### Model Naming by Provider
knowledge/main/about/configuration.md
+3
-4
@@ -2,16 +2,15 @@
2
3
## LLM Roles
4
5
-Agent Zero uses four distinct LLM roles, each configurable independently:
5
+Agent Zero uses three configurable LLM roles:
6
7
| Role | Purpose |
8
|------|---------|
9
-| `chat_llm` | Primary model for all agent reasoning and tool use |
9
+| `chat_llm` | Primary model for all agent reasoning, tool use, and the Browser Agent |
10
| `utility_llm` | Secondary model for internal framework tasks: memory summarization, query generation, history compression, memory recall filtering |
11
-| `browser_llm` | Model used by the browser agent; vision capability recommended |
11
| `embedding_llm` | Produces vector embeddings for memory and knowledge indexing |
12
14
-The utility model handles high-volume, lower-stakes operations and can be a cheaper/faster model than the chat model. Changing the embedding model invalidates the existing vector index - the entire knowledge base is re-indexed automatically.
13
+The utility model handles high-volume, lower-stakes operations and can be a cheaper/faster model than the chat model. The Browser Agent uses the effective chat model resolved by `_model_config`, including per-chat overrides and the chat model vision flag. Changing the embedding model invalidates the existing vector index - the entire knowledge base is re-indexed automatically.
14
15
## Model Providers
16
models.py
+1
-110
@@ -25,7 +25,7 @@ from helpers.dotenv import load_dotenv
25
from helpers.providers import ModelType as ProviderModelType, get_provider_config
26
from helpers.rate_limiter import RateLimiter
27
from helpers.tokens import approximate_tokens
28
-from helpers import dirty_json, browser_use_monkeypatch
28
+from helpers import dirty_json
29
30
from langchain_core.language_models.chat_models import SimpleChatModel
31
from langchain_core.outputs.chat_generation import ChatGenerationChunk
@@ -57,9 +57,6 @@ def turn_off_logging():
57
# init
58
load_dotenv()
59
turn_off_logging()
60
-browser_use_monkeypatch.apply()
61
-
62
-litellm.modify_params = True # helps fix anthropic tool calls by browser-use
60
61
class ModelType(Enum):
62
CHAT = "Chat"
@@ -578,101 +575,6 @@ class LiteLLMChatWrapper(SimpleChatModel):
575
await asyncio.sleep(retry_delay_s)
576
577
581
-class AsyncAIChatReplacement:
582
- class _Completions:
583
- def __init__(self, wrapper):
584
- self._wrapper = wrapper
585
-
586
- async def create(self, *args, **kwargs):
587
- # call the async _acall method on the wrapper
588
- return await self._wrapper._acall(*args, **kwargs)
589
-
590
- class _Chat:
591
- def __init__(self, wrapper):
592
- self.completions = AsyncAIChatReplacement._Completions(wrapper)
593
-
594
- def __init__(self, wrapper, *args, **kwargs):
595
- self._wrapper = wrapper
596
- self.chat = AsyncAIChatReplacement._Chat(wrapper)
597
-
598
-
599
-from browser_use.llm import ChatOllama, ChatOpenRouter, ChatGoogle, ChatAnthropic, ChatGroq, ChatOpenAI
600
-
601
-class BrowserCompatibleChatWrapper(ChatOpenRouter):
602
- """
603
- A wrapper for browser agent that can filter/sanitize messages
604
- before sending them to the LLM.
605
- """
606
-
607
- def __init__(self, *args, **kwargs):
608
- turn_off_logging()
609
- # Create the underlying LiteLLM wrapper
610
- self._wrapper = LiteLLMChatWrapper(*args, **kwargs)
611
- # Browser-use may expect a 'model' attribute
612
- self.model = self._wrapper.model_name
613
- self.kwargs = self._wrapper.kwargs
614
-
615
- @property
616
- def model_name(self) -> str:
617
- return self._wrapper.model_name
618
-
619
- @property
620
- def provider(self) -> str:
621
- return self._wrapper.provider
622
-
623
- def get_client(self, *args, **kwargs): # type: ignore
624
- return AsyncAIChatReplacement(self, *args, **kwargs)
625
-
626
- async def _acall(
627
- self,
628
- messages: List[BaseMessage],
629
- stop: Optional[List[str]] = None,
630
- run_manager: Optional[CallbackManagerForLLMRun] = None,
631
- **kwargs: Any,
632
- ):
633
- # Apply rate limiting if configured
634
- apply_rate_limiter_sync(self._wrapper.a0_model_conf, str(messages))
635
-
636
- # Call the model
637
- try:
638
- model = kwargs.pop("model", None)
639
- kwrgs = {**self._wrapper.kwargs, **kwargs}
640
-
641
- # hack from browser-use to fix json schema for gemini (additionalProperties, $defs, $ref)
642
- if "response_format" in kwrgs and "json_schema" in kwrgs["response_format"] and model.startswith("gemini/"):
643
- kwrgs["response_format"]["json_schema"] = ChatGoogle("")._fix_gemini_schema(kwrgs["response_format"]["json_schema"])
644
-
645
- resp = await acompletion(
646
- model=self._wrapper.model_name,
647
- messages=messages,
648
- stop=stop,
649
- **kwrgs,
650
- )
651
-
652
- # Gemini: strip triple backticks and conform schema
653
- try:
654
- msg = resp.choices[0].message # type: ignore
655
- if self.provider == "gemini" and isinstance(getattr(msg, "content", None), str):
656
- cleaned = browser_use_monkeypatch.gemini_clean_and_conform(msg.content) # type: ignore
657
- if cleaned:
658
- msg.content = cleaned
659
- except Exception:
660
- pass
661
-
662
- except Exception as e:
663
- raise e
664
-
665
- # another hack for browser-use post process invalid jsons
666
- try:
667
- if "response_format" in kwrgs and "json_schema" in kwrgs["response_format"] or "json_object" in kwrgs["response_format"]:
668
- if resp.choices[0].message.content is not None and not resp.choices[0].message.content.startswith("{"): # type: ignore
669
- js = dirty_json.parse(resp.choices[0].message.content) # type: ignore
670
- resp.choices[0].message.content = dirty_json.stringify(js) # type: ignore
671
- except Exception as e:
672
- pass
673
-
674
- return resp
675
-
578
class LiteLLMEmbeddingWrapper(Embeddings):
579
model_name: str
580
kwargs: dict = {}
@@ -910,17 +812,6 @@ def get_chat_model(
812
LiteLLMChatWrapper, name, provider_name, model_config, **kwargs
813
)
814
913
-
914
-def get_browser_model(
915
- provider: str, name: str, model_config: Optional[ModelConfig] = None, **kwargs: Any
916
-) -> BrowserCompatibleChatWrapper:
917
- orig = provider.lower()
918
- provider_name, kwargs = _merge_provider_defaults("chat", orig, kwargs)
919
- return _get_litellm_chat(
920
- BrowserCompatibleChatWrapper, name, provider_name, model_config, **kwargs
921
- )
922
-
923
-
815
def get_embedding_model(
816
provider: str, name: str, model_config: Optional[ModelConfig] = None, **kwargs: Any
817
) -> LiteLLMEmbeddingWrapper | LocalSentenceTransformerWrapper:
plugins/_browser_agent/api/status.py
new
+45
@@ -0,0 +1,45 @@
1
+import importlib.metadata
2
+
3
+from helpers.api import ApiHandler, Request, Response
4
+from plugins._browser_agent.helpers.playwright import (
5
+ get_playwright_binary,
6
+ get_playwright_cache_dir,
7
+)
8
+from plugins._model_config.helpers.model_config import get_chat_model_config
9
+
10
+
11
+class Status(ApiHandler):
12
+ async def process(self, input: dict, request: Request) -> dict | Response:
13
+ cfg = get_chat_model_config()
14
+ binary = get_playwright_binary()
15
+
16
+ browser_use_ok = False
17
+ browser_use_error = ""
18
+ browser_use_version = ""
19
+ try:
20
+ import browser_use # noqa: F401
21
+
22
+ browser_use_ok = True
23
+ browser_use_version = importlib.metadata.version("browser-use")
24
+ except Exception as e:
25
+ browser_use_error = str(e)
26
+
27
+ return {
28
+ "plugin": "_browser_agent",
29
+ "model_source": "Main Model via _model_config",
30
+ "model": {
31
+ "provider": cfg.get("provider", ""),
32
+ "name": cfg.get("name", ""),
33
+ "vision": bool(cfg.get("vision", False)),
34
+ },
35
+ "playwright": {
36
+ "cache_dir": get_playwright_cache_dir(),
37
+ "binary_found": bool(binary),
38
+ "binary_path": str(binary) if binary else "",
39
+ },
40
+ "browser_use": {
41
+ "import_ok": browser_use_ok,
42
+ "version": browser_use_version,
43
+ "error": browser_use_error,
44
+ },
45
+ }
plugins/_browser_agent/assets/init_override.js
renamed
+1
-1
@@ -243,4 +243,4 @@
243
// }
244
// return element;
245
// };
246
-// })();
\ No newline at end of file
246
+// })();
plugins/_browser_agent/extensions/webui/get_message_handler/browser-agent-handler.js
new
+54
@@ -0,0 +1,54 @@
1
+import {
2
+ createActionButton,
3
+ copyToClipboard,
4
+} from "/components/messages/action-buttons/simple-action-buttons.js";
5
+import { store as stepDetailStore } from "/components/modals/process-step-detail/step-detail-store.js";
6
+import { store as speechStore } from "/components/chat/speech/speech-store.js";
7
+import {
8
+ buildDetailPayload,
9
+ cleanStepTitle,
10
+ drawProcessStep,
11
+} from "/js/messages.js";
12
+
13
+export default async function registerBrowserAgentHandler(extData) {
14
+ if (extData?.type === "browser") {
15
+ extData.handler = drawMessageBrowserAgent;
16
+ }
17
+}
18
+
19
+function drawMessageBrowserAgent({
20
+ id,
21
+ type,
22
+ heading,
23
+ content,
24
+ kvps,
25
+ timestamp,
26
+ agentno = 0,
27
+ ...additional
28
+}) {
29
+ const title = cleanStepTitle(heading);
30
+ const displayKvps = { ...kvps };
31
+ const answerText = String(kvps?.answer ?? "");
32
+ const actionButtons = answerText.trim()
33
+ ? [
34
+ createActionButton("detail", "", () =>
35
+ stepDetailStore.showStepDetail(
36
+ buildDetailPayload(arguments[0], { headerLabels: [] }),
37
+ ),
38
+ ),
39
+ createActionButton("speak", "", () => speechStore.speak(answerText)),
40
+ createActionButton("copy", "", () => copyToClipboard(answerText)),
41
+ ].filter(Boolean)
42
+ : [];
43
+
44
+ return drawProcessStep({
45
+ id,
46
+ title,
47
+ code: "WWW",
48
+ classes: undefined,
49
+ kvps: displayKvps,
50
+ content,
51
+ actionButtons,
52
+ log: arguments[0],
53
+ });
54
+}
plugins/_browser_agent/helpers/__init__.py
new
+1
@@ -0,0 +1 @@
1
+# Built-in browser agent helpers.
plugins/_browser_agent/helpers/browser_llm.py
new
+131
@@ -0,0 +1,131 @@
1
+from typing import Any, List, Optional
2
+import litellm
3
+from litellm import acompletion
4
+from langchain_core.callbacks.manager import CallbackManagerForLLMRun
5
+from langchain_core.messages import BaseMessage
6
+
7
+import models
8
+from browser_use.llm import ChatGoogle, ChatOpenRouter
9
+from helpers import dirty_json
10
+
11
+from plugins._browser_agent.helpers import browser_use_monkeypatch
12
+
13
+
14
+_BROWSER_USE_PATCHED = False
15
+
16
+
17
+def apply_browser_use_patches() -> None:
18
+ global _BROWSER_USE_PATCHED
19
+ if _BROWSER_USE_PATCHED:
20
+ return
21
+
22
+ browser_use_monkeypatch.apply()
23
+ litellm.modify_params = True
24
+ _BROWSER_USE_PATCHED = True
25
+
26
+
27
+class AsyncAIChatReplacement:
28
+ class _Completions:
29
+ def __init__(self, wrapper):
30
+ self._wrapper = wrapper
31
+
32
+ async def create(self, *args, **kwargs):
33
+ return await self._wrapper._acall(*args, **kwargs)
34
+
35
+ class _Chat:
36
+ def __init__(self, wrapper):
37
+ self.completions = AsyncAIChatReplacement._Completions(wrapper)
38
+
39
+ def __init__(self, wrapper, *args, **kwargs):
40
+ self._wrapper = wrapper
41
+ self.chat = AsyncAIChatReplacement._Chat(wrapper)
42
+
43
+
44
+class BrowserCompatibleChatWrapper(ChatOpenRouter):
45
+ """
46
+ A wrapper for browser agent that can filter/sanitize messages
47
+ before sending them to the LLM.
48
+ """
49
+
50
+ def __init__(self, *args, **kwargs):
51
+ apply_browser_use_patches()
52
+ models.turn_off_logging()
53
+ self._wrapper = models.LiteLLMChatWrapper(*args, **kwargs)
54
+ self.model = self._wrapper.model_name
55
+ self.kwargs = self._wrapper.kwargs
56
+
57
+ @property
58
+ def model_name(self) -> str:
59
+ return self._wrapper.model_name
60
+
61
+ @property
62
+ def provider(self) -> str:
63
+ return self._wrapper.provider
64
+
65
+ def get_client(self, *args, **kwargs): # type: ignore
66
+ return AsyncAIChatReplacement(self, *args, **kwargs)
67
+
68
+ async def _acall(
69
+ self,
70
+ messages: List[BaseMessage],
71
+ stop: Optional[List[str]] = None,
72
+ run_manager: Optional[CallbackManagerForLLMRun] = None,
73
+ **kwargs: Any,
74
+ ):
75
+ models.apply_rate_limiter_sync(self._wrapper.a0_model_conf, str(messages))
76
+
77
+ try:
78
+ model = kwargs.pop("model", None)
79
+ kwrgs = {**self._wrapper.kwargs, **kwargs}
80
+
81
+ # hack from browser-use to fix json schema for gemini (additionalProperties, $defs, $ref)
82
+ if "response_format" in kwrgs and "json_schema" in kwrgs["response_format"] and model and model.startswith("gemini/"):
83
+ kwrgs["response_format"]["json_schema"] = ChatGoogle("")._fix_gemini_schema(kwrgs["response_format"]["json_schema"])
84
+
85
+ resp = await acompletion(
86
+ model=self._wrapper.model_name,
87
+ messages=messages,
88
+ stop=stop,
89
+ **kwrgs,
90
+ )
91
+
92
+ # Gemini: strip triple backticks and conform schema
93
+ try:
94
+ msg = resp.choices[0].message # type: ignore
95
+ if self.provider == "gemini" and isinstance(getattr(msg, "content", None), str):
96
+ cleaned = browser_use_monkeypatch.gemini_clean_and_conform(msg.content) # type: ignore
97
+ if cleaned:
98
+ msg.content = cleaned
99
+ except Exception:
100
+ pass
101
+
102
+ except Exception as e:
103
+ raise e
104
+
105
+ # another hack for browser-use post process invalid jsons
106
+ try:
107
+ if "response_format" in kwrgs and "json_schema" in kwrgs["response_format"] or "json_object" in kwrgs["response_format"]:
108
+ if resp.choices[0].message.content is not None and not resp.choices[0].message.content.startswith("{"): # type: ignore
109
+ js = dirty_json.parse(resp.choices[0].message.content) # type: ignore
110
+ resp.choices[0].message.content = dirty_json.stringify(js) # type: ignore
111
+ except Exception as e:
112
+ pass
113
+
114
+ return resp
115
+
116
+
117
+def build_browser_model_from_config(
118
+ model_config: models.ModelConfig,
119
+) -> BrowserCompatibleChatWrapper:
120
+ apply_browser_use_patches()
121
+ original_provider = model_config.provider.lower()
122
+ provider_name, kwargs = models._merge_provider_defaults( # type: ignore[attr-defined]
123
+ "chat", original_provider, model_config.build_kwargs()
124
+ )
125
+ return models._get_litellm_chat( # type: ignore[attr-defined]
126
+ BrowserCompatibleChatWrapper,
127
+ model_config.name,
128
+ provider_name,
129
+ model_config,
130
+ **kwargs,
131
+ )
plugins/_browser_agent/helpers/browser_use.py
renamed
+1
-1
@@ -1,4 +1,4 @@
1
from helpers import dotenv
2
dotenv.save_dotenv_value("ANONYMIZED_TELEMETRY", "false")
3
import browser_use
4
-import browser_use.utils
\ No newline at end of file
4
+import browser_use.utils
plugins/_browser_agent/helpers/browser_use_monkeypatch.py
renamed
plugins/_browser_agent/helpers/playwright.py
renamed
+1
-2
@@ -1,4 +1,3 @@
1
-
1
import os
2
import sys
3
from pathlib import Path
@@ -36,4 +35,4 @@ def ensure_playwright_binary():
35
bin = get_playwright_binary()
36
if not bin:
37
raise Exception("Playwright binary not found after installation")
39
- return bin
\ No newline at end of file
38
+ return bin
plugins/_browser_agent/plugin.yaml
new
+8
@@ -0,0 +1,8 @@
1
+name: _browser_agent
2
+title: Browser Agent
3
+description: Built-in browser-use automation tool.
4
+version: 1.0.0
5
+always_enabled: false
6
+settings_sections: []
7
+per_project_config: false
8
+per_agent_config: false
plugins/_browser_agent/prompts/agent.system.tool.browser.md
renamed
plugins/_browser_agent/prompts/browser_agent.system.md
renamed
+1
-1
@@ -19,4 +19,4 @@ When you have completed the assigned task OR are waiting for further instruction
19
- Response field is used to answer to user's task or ask additional questions
20
- If you navigate to a website and no further actions are requested, call "Complete task" immediately
21
- If you complete any requested interaction (clicking, typing, etc.), call "Complete task"
22
-- Never leave a task running indefinitely - always conclude with "Complete task"
\ No newline at end of file
22
+- Never leave a task running indefinitely - always conclude with "Complete task"
plugins/_browser_agent/tools/browser_agent.py
renamed
+6
-3
@@ -6,9 +6,9 @@ from pathlib import Path
6
7
from helpers.tool import Tool, Response
8
from helpers import files, defer, persist_chat, strings
9
-from helpers.browser_use import browser_use # type: ignore[attr-defined]
9
+from plugins._browser_agent.helpers.browser_use import browser_use # type: ignore[attr-defined]
10
from helpers.print_style import PrintStyle
11
-from helpers.playwright import ensure_playwright_binary
11
+from plugins._browser_agent.helpers.playwright import ensure_playwright_binary
12
from helpers.secrets import get_secrets_manager
13
from extensions.python.message_loop_start._10_iteration_no import get_iter_no
14
from pydantic import BaseModel
@@ -16,6 +16,9 @@ import uuid
16
from helpers.dirty_json import DirtyJson
17
18
19
+PLUGIN_DIR = Path(__file__).resolve().parents[1]
20
+
21
+
22
class State:
23
@staticmethod
24
async def create(agent: Agent):
@@ -105,7 +108,7 @@ class State:
108
109
# Add init script to the browser session
110
if self.browser_session and self.browser_session.browser_context:
108
- js_override = files.get_abs_path("lib/browser/init_override.js")
111
+ js_override = str(PLUGIN_DIR / "assets" / "init_override.js")
112
await self.browser_session.browser_context.add_init_script(path=js_override) if self.browser_session else None
113
114
def start_task(self, task: str):
plugins/_browser_agent/webui/main.html
new
+204
@@ -0,0 +1,204 @@
1
+<html>
2
+<head>
3
+ <title>Browser Agent</title>
4
+ <script type="module">
5
+ import { callJsonApi } from "/js/api.js";
6
+ import "/components/plugins/list/pluginListStore.js";
7
+
8
+ globalThis.browserAgentStatusApi = { callJsonApi };
9
+ </script>
10
+</head>
11
+<body>
12
+ <div
13
+ x-data="{
14
+ loading: true,
15
+ error: '',
16
+ status: null,
17
+ async init() {
18
+ try {
19
+ this.status = await browserAgentStatusApi.callJsonApi('/plugins/_browser_agent/status', {});
20
+ } catch (error) {
21
+ this.error = error instanceof Error ? error.message : String(error);
22
+ } finally {
23
+ this.loading = false;
24
+ }
25
+ }
26
+ }"
27
+ x-init="init()"
28
+ class="browser-agent-page"
29
+ >
30
+ <div class="section-title">Browser Agent</div>
31
+ <div class="section-description">
32
+ Built-in browser automation plugin backed by `browser-use` and Playwright.
33
+ Model selection stays in `_model_config`; the browser agent follows the effective Main Model.
34
+ </div>
35
+
36
+ <div class="browser-agent-card" x-show="loading">
37
+ <div class="status-row">
38
+ <span class="material-symbols-outlined spinning">progress_activity</span>
39
+ <span>Loading browser status...</span>
40
+ </div>
41
+ </div>
42
+
43
+ <div class="browser-agent-card error" x-show="!loading && error">
44
+ <div class="field-title">Status check failed</div>
45
+ <div class="field-description" x-text="error"></div>
46
+ </div>
47
+
48
+ <template x-if="!loading && status">
49
+ <div class="browser-agent-grid">
50
+ <div class="browser-agent-card">
51
+ <div class="field-title">Model Source</div>
52
+ <div class="field-description" x-text="status.model_source"></div>
53
+ </div>
54
+
55
+ <div class="browser-agent-card">
56
+ <div class="field-title">Resolved Main Model</div>
57
+ <div class="status-row">
58
+ <span class="status-key">Provider</span>
59
+ <span class="status-value" x-text="status.model.provider || 'Not configured'"></span>
60
+ </div>
61
+ <div class="status-row">
62
+ <span class="status-key">Model</span>
63
+ <span class="status-value" x-text="status.model.name || 'Not configured'"></span>
64
+ </div>
65
+ <div class="status-row">
66
+ <span class="status-key">Vision</span>
67
+ <span class="status-badge" :class="status.model.vision ? 'ok' : 'warn'" x-text="status.model.vision ? 'Enabled' : 'Disabled'"></span>
68
+ </div>
69
+ </div>
70
+
71
+ <div class="browser-agent-card">
72
+ <div class="field-title">Playwright Runtime</div>
73
+ <div class="status-row">
74
+ <span class="status-key">Binary</span>
75
+ <span class="status-badge" :class="status.playwright.binary_found ? 'ok' : 'fail'" x-text="status.playwright.binary_found ? 'Found' : 'Missing'"></span>
76
+ </div>
77
+ <div class="status-row">
78
+ <span class="status-key">Cache</span>
79
+ <span class="status-value mono" x-text="status.playwright.cache_dir"></span>
80
+ </div>
81
+ <div class="status-row" x-show="status.playwright.binary_path">
82
+ <span class="status-key">Path</span>
83
+ <span class="status-value mono" x-text="status.playwright.binary_path"></span>
84
+ </div>
85
+ <div class="field-description" x-show="!status.playwright.binary_found">
86
+ Docker images ship the Playwright Chromium shell preinstalled. In local development, the first run installs it on demand via <span class="mono">ensure_playwright_binary()</span> if missing.
87
+ </div>
88
+ </div>
89
+
90
+ <div class="browser-agent-card">
91
+ <div class="field-title">browser-use</div>
92
+ <div class="status-row">
93
+ <span class="status-key">Import</span>
94
+ <span class="status-badge" :class="status.browser_use.import_ok ? 'ok' : 'fail'" x-text="status.browser_use.import_ok ? 'Ready' : 'Error'"></span>
95
+ </div>
96
+ <div class="status-row" x-show="status.browser_use.version">
97
+ <span class="status-key">Version</span>
98
+ <span class="status-value" x-text="status.browser_use.version"></span>
99
+ </div>
100
+ <div class="field-description mono" x-show="status.browser_use.error" x-text="status.browser_use.error"></div>
101
+ </div>
102
+ </div>
103
+ </template>
104
+
105
+ <div class="browser-agent-actions">
106
+ <button
107
+ class="btn btn-field"
108
+ @click="$store.pluginListStore.openPluginConfig({ name: '_model_config', has_config_screen: true })"
109
+ >
110
+ Open Model Settings
111
+ </button>
112
+ <button class="btn btn-field" @click="openModal('/plugins/_model_config/webui/main.html')">
113
+ Open Presets
114
+ </button>
115
+ <button class="btn btn-field" @click="openModal('/plugins/_model_config/webui/api-keys.html')">
116
+ Open API Keys
117
+ </button>
118
+ </div>
119
+ </div>
120
+
121
+ <style>
122
+ .browser-agent-page {
123
+ display: flex;
124
+ flex-direction: column;
125
+ gap: 14px;
126
+ }
127
+
128
+ .browser-agent-grid {
129
+ display: grid;
130
+ gap: 12px;
131
+ grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
132
+ }
133
+
134
+ .browser-agent-card {
135
+ display: flex;
136
+ flex-direction: column;
137
+ gap: 10px;
138
+ padding: 14px;
139
+ background: var(--color-input);
140
+ border: 1px solid var(--color-border);
141
+ border-radius: 10px;
142
+ }
143
+
144
+ .browser-agent-card.error {
145
+ border-color: rgba(214, 40, 40, 0.35);
146
+ }
147
+
148
+ .browser-agent-actions {
149
+ display: flex;
150
+ gap: 8px;
151
+ flex-wrap: wrap;
152
+ }
153
+
154
+ .status-row {
155
+ display: flex;
156
+ align-items: flex-start;
157
+ justify-content: space-between;
158
+ gap: 12px;
159
+ font-size: 0.84rem;
160
+ }
161
+
162
+ .status-key {
163
+ opacity: 0.7;
164
+ min-width: 64px;
165
+ }
166
+
167
+ .status-value {
168
+ text-align: right;
169
+ word-break: break-word;
170
+ }
171
+
172
+ .status-badge {
173
+ padding: 2px 8px;
174
+ border-radius: 999px;
175
+ font-size: 0.76rem;
176
+ font-weight: 600;
177
+ border: 1px solid transparent;
178
+ }
179
+
180
+ .status-badge.ok {
181
+ color: #1b5e20;
182
+ background: rgba(46, 125, 50, 0.14);
183
+ border-color: rgba(46, 125, 50, 0.24);
184
+ }
185
+
186
+ .status-badge.warn {
187
+ color: #8a6100;
188
+ background: rgba(191, 144, 0, 0.14);
189
+ border-color: rgba(191, 144, 0, 0.24);
190
+ }
191
+
192
+ .status-badge.fail {
193
+ color: #9f1239;
194
+ background: rgba(190, 24, 93, 0.12);
195
+ border-color: rgba(190, 24, 93, 0.24);
196
+ }
197
+
198
+ .mono {
199
+ font-family: var(--font-mono);
200
+ font-size: 0.78rem;
201
+ }
202
+ </style>
203
+</body>
204
+</html>
plugins/_browser_agent/webui/thumbnail.jpg
Binary files /dev/null and b/plugins/_browser_agent/webui/thumbnail.jpg differ
plugins/_model_config/helpers/model_config.py
+5
-3
@@ -187,13 +187,15 @@ def build_utility_model(agent=None):
187
188
189
def build_browser_model(agent=None):
190
- """Build and return a BrowserCompatibleChatWrapper using chat model config."""
190
+ """Build and return the browser-use adapter using chat model config."""
191
cfg = get_chat_model_config(agent)
192
mc = build_model_config(cfg, models.ModelType.CHAT)
193
- return models.get_browser_model(
194
- mc.provider, mc.name, model_config=mc, **mc.build_kwargs()
193
+ from plugins._browser_agent.helpers.browser_llm import (
194
+ build_browser_model_from_config,
195
)
196
197
+ return build_browser_model_from_config(mc)
198
+
199
200
def build_embedding_model(agent=None):
201
"""Build and return an embedding model wrapper."""
plugins/_model_config/webui/config.html
-13
@@ -257,19 +257,6 @@
257
@change="config[section.key].kwargs = $store.modelConfig.textToKwargs(config[section.key]._kwargs_text)"></textarea>
258
</div>
259
</div>
260
-
261
- <!-- Browser HTTP Headers -->
262
- <template x-if="section.key === 'chat_model'">
263
- <div class="field field-full">
264
- <div class="field-label">
265
- <div class="field-title">Browser HTTP Headers</div>
266
- <div class="field-description">
267
- Custom HTTP headers sent with browser requests. The browser agent uses the main model. Format is KEY=VALUE, one per line.
268
- </div>
269
- </div>
270
- </div>
271
- </template>
272
-
260
</div>
261
</template>
262
plugins/_model_config/webui/main.html
-7
@@ -183,13 +183,6 @@
183
@change="preset.chat.kwargs = $store.modelConfig.textToKwargs(preset.chat._kwargs_text)"></textarea>
184
</div>
185
</div>
186
- <div class="field field-full">
187
- <div class="field-label">
188
- <div class="field-title">Browser HTTP Headers</div>
189
- <div class="field-description">Custom HTTP headers sent with browser requests. The browser agent uses the main model. Format is KEY=VALUE, one per line.</div>
190
- </div>
191
- </div>
192
-
186
<div class="preset-subheader">Utility Model <span style="opacity:0.5; font-size:0.75rem;">(optional — falls back to the configured Utility Model)</span></div>
187
<div class="field">
188
<div class="field-label">
tests/test_browser_agent_regressions.py
new
+80
@@ -0,0 +1,80 @@
1
+import asyncio
2
+import importlib
3
+import json
4
+import sys
5
+from pathlib import Path
6
+from types import SimpleNamespace
7
+
8
+
9
+PROJECT_ROOT = Path(__file__).resolve().parents[1]
10
+if str(PROJECT_ROOT) not in sys.path:
11
+ sys.path.insert(0, str(PROJECT_ROOT))
12
+
13
+import plugins._browser_agent.helpers.browser_use_monkeypatch as browser_use_monkeypatch
14
+import plugins._browser_agent.tools.browser_agent as browser_agent_module
15
+
16
+
17
+def test_gemini_clean_and_conform_normalizes_known_single_action_shapes():
18
+ raw = (
19
+ '{"action":['
20
+ '{"complete_task":{"title":"T","response":"R","page_summary":"S"}}'
21
+ ']}'
22
+ )
23
+
24
+ cleaned = browser_use_monkeypatch.gemini_clean_and_conform(raw)
25
+
26
+ assert cleaned is not None
27
+ parsed = json.loads(cleaned)
28
+ assert parsed["action"] == [
29
+ {
30
+ "done": {
31
+ "success": True,
32
+ "data": {
33
+ "title": "T",
34
+ "response": "R",
35
+ "page_summary": "S",
36
+ },
37
+ }
38
+ },
39
+ ]
40
+
41
+
42
+class DummyBrowserSession:
43
+ def __init__(self) -> None:
44
+ self.kill_called = False
45
+ self.close_called = False
46
+
47
+ async def kill(self) -> None:
48
+ self.kill_called = True
49
+
50
+ async def close(self) -> None:
51
+ self.close_called = True
52
+
53
+
54
+class DummyAgent:
55
+ def __init__(self) -> None:
56
+ self.context = SimpleNamespace(id="ctx", task=None)
57
+
58
+
59
+def test_browser_session_teardown_prefers_kill_for_keep_alive_sessions():
60
+ state = browser_agent_module.State(DummyAgent())
61
+ session = DummyBrowserSession()
62
+ state.browser_session = session
63
+
64
+ state.kill_task()
65
+
66
+ assert session.kill_called is True
67
+ assert session.close_called is False
68
+
69
+
70
+def test_browser_cleanup_extensions_follow_new_extensible_path_layout():
71
+ extension = importlib.import_module("helpers.extension")
72
+ remove_classes = extension._get_extension_classes( # type: ignore[attr-defined]
73
+ "_functions/agent/AgentContext/remove/start"
74
+ )
75
+ reset_classes = extension._get_extension_classes( # type: ignore[attr-defined]
76
+ "_functions/agent/AgentContext/reset/start"
77
+ )
78
+
79
+ assert any(cls.__name__ == "CleanupBrowserStateOnRemove" for cls in remove_classes)
80
+ assert any(cls.__name__ == "CleanupBrowserStateOnReset" for cls in reset_classes)
webui/js/messages.js
-44
@@ -94,8 +94,6 @@ export async function getMessageHandler(type) {
94
return drawMessageResponse;
95
case "tool":
96
return drawMessageTool;
97
- case "browser":
98
- return drawMessageBrowser;
97
case "progress":
98
return drawMessageProgress;
99
case "mcp":
@@ -1204,48 +1202,6 @@ export function drawMessageToolSimple({
1202
});
1203
}
1204
1207
-/**
1208
- * @param {MessageHandlerArgs & Record<string, any>} param0
1209
- * @returns {MessageHandlerResult}
1210
- */
1211
-export function drawMessageBrowser({
1212
- id,
1213
- type,
1214
- heading,
1215
- content,
1216
- kvps,
1217
- timestamp,
1218
- agentno = 0,
1219
- ...additional
1220
-}) {
1221
- const title = cleanStepTitle(heading);
1222
- let displayKvps = { ...kvps };
1223
- const answerText = String(kvps?.answer ?? "");
1224
- const actionButtons = answerText.trim()
1225
- ? [
1226
- createActionButton("detail", "", () =>
1227
- stepDetailStore.showStepDetail(
1228
- buildDetailPayload(arguments[0], { headerLabels: [] }),
1229
- ),
1230
- ),
1231
- createActionButton("speak", "", () => speechStore.speak(answerText)),
1232
- createActionButton("copy", "", () => copyToClipboard(answerText)),
1233
- ].filter(Boolean)
1234
- : [];
1235
-
1236
- return drawProcessStep({
1237
- id,
1238
- title,
1239
- code: "WWW",
1240
- classes: undefined,
1241
- kvps: displayKvps,
1242
- content,
1243
- // contentClasses: [],
1244
- actionButtons,
1245
- log: arguments[0],
1246
- });
1247
-}
1248
-
1205
/**
1206
* @param {MessageHandlerArgs & Record<string, any>} param0
1207
* @returns {MessageHandlerResult}