@samitouri / QOSami_Agents_Zero-1 / commits / 59f5ae03

Integrate vision sidecar routing with model presets

Add a strict per-preset sidecar slot, Main-first routing, one-call multi-image analysis, parallel-worker image resolution, and inline model settings with focused regression coverage.

Alessandro committed Aug 25, 2026 at 15:14 UTC 59f5ae03ea618aecb6e9dd3ee4657b6dee35cff1
21 files changed +878 -148
extensions/python/system_prompt/_11_tools_prompt.py
+7 -2
@@ -51,10 +51,15 @@ async def build_prompt(agent: Agent) -> str:
51 prompt = agent.read_prompt("agent.system.tools.md", tools=tools_str)
52
53 # vision support
54 - from plugins._model_config.helpers.model_config import get_chat_model_config
54 + from plugins._model_config.helpers.model_config import (
55 + get_chat_model_config,
56 + use_vision_sidecar,
57 + )
58
59 chat_cfg = get_chat_model_config(agent)
57 - if chat_cfg.get("vision", False):
60 + if use_vision_sidecar(agent):
61 + prompt += "\n\n" + agent.read_prompt("agent.system.tools_vision_sidecar.md")
62 + elif chat_cfg.get("vision", False):
63 prompt += "\n\n" + agent.read_prompt("agent.system.tools_vision.md")
64
65 return prompt
helpers/responses_tools.py
+6 -1
@@ -114,8 +114,13 @@ def _local_tool_prompts(agent: Any) -> list[tuple[str, str]]:
114
115 def _vision_tool_prompt(agent: Any) -> str:
116 try:
117 - from plugins._model_config.helpers.model_config import get_chat_model_config
117 + from plugins._model_config.helpers.model_config import (
118 + get_chat_model_config,
119 + use_vision_sidecar,
120 + )
121
122 + if use_vision_sidecar(agent):
123 + return agent.read_prompt("agent.system.tools_vision_sidecar.md")
124 if not get_chat_model_config(agent).get("vision", False):
125 return ""
126 return agent.read_prompt("agent.system.tools_vision.md")
helpers/responses_tools.py.dox.md
+3 -3
@@ -12,7 +12,7 @@
12
13 ## Local Contracts
14
15 -- Build local function tools from enabled `agent.system.tool.*.md` prompt files and include `vision_load` only when the active chat model enables the matching vision prompt.
15 +- Build local function tools from enabled `agent.system.tool.*.md` prompt files and include `vision_load` when either Main native vision or the effective preset's Vision Model enables the matching prompt.
16 - Discover local prompt files through `helpers.subagents.get_paths`; this module
17 owns the Responses-specific prompt-name compatibility rules.
18 - Local prompt-derived function names use existing bullet declarations that pair a backticked name with `arg` or `args` for multi-tool prompt files, otherwise prefer explicit `"tool_name"` examples, then the first prompt heading, and finally the prompt filename.
@@ -25,8 +25,8 @@
25 - Preserve original Agent Zero tool names through the native Responses name map.
26 - Keep MCP tool schemas merged after local prompt-derived tools.
27 - Apply `helpers.tool_policy` before emitting local or MCP schemas; a blocked
28 - capability is absent from provider-native tool definitions. Vision remains
29 - controlled solely by the active chat model configuration.
28 + capability is absent from provider-native tool definitions. Vision routing
29 + is controlled by the effective model preset rather than Agent Editor.
30 - Connector remote tools are advertised only when `_a0_connector` runtime metadata says the matching connected CLI capability is currently available.
31
32 ## Work Guidance
plugins/_model_config/AGENTS.md
+7 -2
@@ -14,7 +14,7 @@
14
15 ## Local Contracts
16
17 -- `Default` is the first global preset and cannot be deleted or renamed. It owns the complete main, utility, and embedding baseline.
17 +- `Default` is the first global preset and cannot be deleted or renamed. It owns the complete main, utility, and embedding baseline; its Vision Model slot is optional.
18 - Preset definitions are global. Global, project, agent-profile, and project/profile plugin configs persist only `model_preset`; chats may persist a preset reference as their explicit override.
19 - Preserve scoped plugin resolution order and fall back invalid or missing scope/chat references to `Default`.
20 - Project Settings `llm` payloads are owned here through the generic `helpers.projects` project extension-data hooks; keep project helper code agnostic to `_model_config` paths, presets, and inheritance rules.
@@ -22,7 +22,12 @@
22 - Check API-key readiness only for the effective model configuration; unused global presets must not produce Welcome-screen warnings.
23 - Coordinate OAuth-backed providers with `_oauth` instead of hardcoding provider-specific auth here.
24 - `model_config_get` exposes `model_configured` as a derived chat-model readiness flag from provider, model name, and API-key availability.
25 -- Non-default presets may inherit omitted slots or durable tuning from `Default`, but must replace or clear per-slot `kwargs` so provider-specific extra params never leak across model providers.
25 +- Non-default presets may inherit omitted main, utility, or embedding slots and durable tuning from `Default`, but must replace or clear per-slot `kwargs` so provider-specific extra params never leak across model providers.
26 +- The optional `vision` slot is strictly per preset and never inherited from `Default`; an empty slot disables sidecar vision for that preset.
27 +- Main native vision wins by default. A configured Vision Model handles `vision_load` when Main lacks vision, or when that preset explicitly enables `override_main`.
28 +- Keep the optional Vision provider/model selector inside the Main Model card and flush with Main's field alignment, without a nested left inset. Show it only while Main vision is disabled or `override_main` is enabled; do not render a standalone Vision Model card.
29 +- Show `Use separate Vision Model` immediately below `Supports Vision` while Main vision is enabled, not inside Advanced Settings; describe the disabled state as using Main's native vision.
30 +- In model overviews, render the effective Vision Model as a text-only `Vision override / Provider / Model` child aligned with Main's provider column, not as an icon-bearing peer row.
31 - Changing a model provider in the settings UI must clear `api_base` and `kwargs` because both may be provider-specific.
32 - Repair provider-specific model-config aliases at the model-config read/build boundary; keep provider-specific repairs out of provider-agnostic core wrappers such as `models.py`.
33 - `modelConfig.createPresetEditor()` owns local preset drafts, row actions, and stable UI-only row keys so deletion or renaming cannot rebind nested model fields.
plugins/_model_config/extensions/webui/chat-input-progress-start/model-switcher.html
+10
@@ -65,6 +65,16 @@
65 </span>
66 </div>
67 </template>
68 + <template x-if="preset.vision?.name && (!preset.chat?.vision || preset.vision?.override_main)">
69 + <div class="model-switcher-model-row">
70 + <span class="model-switcher-model-label">Vision</span>
71 + <span class="model-switcher-model-value">
72 + <span x-text="preset.vision.provider" style="opacity:0.5;"></span>
73 + <span style="opacity:0.3; margin:0 3px;">/</span>
74 + <span x-text="preset.vision.name"></span>
75 + </span>
76 + </div>
77 + </template>
78 <template x-if="preset.utility?.name">
79 <div class="model-switcher-model-row">
80 <span class="model-switcher-model-label">Utility</span>
plugins/_model_config/helpers/model_config.py
+51 -2
@@ -16,11 +16,21 @@ PRESET_SCOPE_GLOBAL = "global"
16 PRESET_SCOPE_PROJECT = "project"
17 PRESET_SLOT_CONFIG_SECTIONS = {
18 "chat": "chat_model",
19 + "vision": "vision_model",
20 "utility": "utility_model",
21 "embedding": "embedding_model",
22 }
23 MODEL_SLOT_PRESET_REPLACE_FIELDS = {"kwargs"}
24 IMPLICIT_PRESET_SLOT_DEFAULTS = {
25 + "vision": {
26 + "vision": True,
27 + "max_embeds": 10,
28 + "override_main": False,
29 + "rl_requests": 0,
30 + "rl_input": 0,
31 + "rl_output": 0,
32 + "kwargs": {},
33 + },
34 "utility": {
35 "ctx_length": 128000,
36 "ctx_input": 0.7,
@@ -267,6 +277,9 @@ def _clean_preset_for_file(preset: dict) -> dict:
277 slot_config = preset.get(slot)
278 if isinstance(slot_config, dict):
279 slot_clean = _strip_ui_fields(slot_config, strip_api_key=True)
280 + if slot == "vision" and _slot_has_identity(slot_clean):
281 + slot_clean["vision"] = True
282 + slot_clean.setdefault("max_embeds", 10)
283 cleaned[slot] = (
284 slot_clean
285 if name == DEFAULT_PRESET_NAME
@@ -339,7 +352,12 @@ def validate_presets(presets: list, *, require_default: bool = True) -> list:
352 def normalize_config_for_save(config: dict) -> dict:
353 """Remove UI-only fields and inline API keys before storing scoped config."""
354 cleaned = deepcopy(config or {})
342 - for section_name in ("chat_model", "utility_model", "embedding_model"):
355 + for section_name in (
356 + "chat_model",
357 + "vision_model",
358 + "utility_model",
359 + "embedding_model",
360 + ):
361 section = cleaned.get(section_name)
362 if isinstance(section, dict):
363 cleaned[section_name] = _strip_ui_fields(section, strip_api_key=True)
@@ -659,10 +677,12 @@ def build_config_from_preset(
677 continue
678 slot_config = _get_preset_slot_config(preset, slot)
679 if not _should_apply_preset_slot(slot, slot_config):
680 + if slot == "vision":
681 + config[section] = {}
682 continue
683 config[section] = _merge_model_slot(
684 slot,
665 - config.get(section, {}),
685 + {} if slot == "vision" else config.get(section, {}),
686 slot_config,
687 strip_api_key=strip_api_key,
688 )
@@ -738,6 +758,23 @@ def get_chat_model_config(agent=None) -> dict:
758 return get_effective_config(agent).get("chat_model", {})
759
760
761 +def get_vision_model_config(agent=None) -> dict:
762 + """Get the optional, strictly per-preset Vision Model config."""
763 + return get_effective_config(agent).get("vision_model", {})
764 +
765 +
766 +def use_vision_sidecar(agent=None) -> bool:
767 + """Return whether vision_load should delegate to the preset's Vision Model."""
768 + vision_cfg = get_vision_model_config(agent)
769 + if not (
770 + str(vision_cfg.get("provider") or "").strip()
771 + and str(vision_cfg.get("name") or "").strip()
772 + ):
773 + return False
774 + chat_cfg = get_chat_model_config(agent)
775 + return not bool(chat_cfg.get("vision")) or bool(vision_cfg.get("override_main"))
776 +
777 +
778 def get_utility_model_config(agent=None) -> dict:
779 """Get utility model config, with per-chat override if active."""
780 return get_effective_config(agent).get("utility_model", {})
@@ -835,6 +872,16 @@ def build_utility_model(agent=None):
872 )
873
874
875 +def build_vision_model(agent=None):
876 + """Build the optional Vision Model selected by the effective preset."""
877 + cfg = get_vision_model_config(agent)
878 + mc = build_model_config(cfg, models.ModelType.CHAT)
879 + mc.vision = True
880 + return models.get_chat_model(
881 + mc.provider, mc.name, model_config=mc, **mc.build_kwargs()
882 + )
883 +
884 +
885 def build_embedding_model(agent=None):
886 """Build and return an embedding model wrapper."""
887 cfg = get_embedding_model_config(agent)
@@ -881,6 +928,8 @@ def get_missing_api_key_providers(agent=None) -> list[dict]:
928 ("Utility Model", cfg.get("utility_model", {})),
929 ("Embedding Model", get_embedding_model_config(agent)),
930 ]
931 + if use_vision_sidecar(agent):
932 + checks.insert(1, ("Vision Model", cfg.get("vision_model", {})))
933
934 for label, model_cfg in checks:
935 provider = model_cfg.get("provider", "")
plugins/_model_config/webui/main.html
+19 -2
@@ -21,7 +21,7 @@
21 <div class="preset-editor-intro">
22 <div class="field-title">Model Presets</div>
23 <div class="field-description">
24 - Each preset contains the complete main, utility, and embedding model setup. Default is always available.
24 + Each preset contains the main, utility, embedding, and optional vision-sidecar setup. Default is always available.
25 </div>
26 </div>
27
@@ -67,9 +67,21 @@
67 <div class="section-title">Main Model</div>
68 <div class="section-description">Primary model for conversations, reasoning, and tools.</div>
69 </div>
70 - <div x-data="{ get model() { return selectedPreset.chat; }, modelType: 'chat', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }">
70 + <div x-data="{ get model() { return selectedPreset.chat; }, get visionModel() { return selectedPreset.vision; }, modelType: 'chat', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }">
71 <x-component path="/plugins/_model_config/webui/model-field.html"></x-component>
72 </div>
73 + <template x-if="!selectedPreset.chat.vision || selectedPreset.vision.override_main">
74 + <div class="vision-sidecar-selector">
75 + <div class="field-title">Vision sidecar</div>
76 + <div class="field-description"
77 + x-text="selectedPreset.chat.vision
78 + ? 'Overrides Main model vision for vision_load.'
79 + : 'Used by vision_load while Main model vision is disabled. Leave empty to disable image analysis.'"></div>
80 + <div x-data="{ get model() { return selectedPreset.vision; }, modelType: 'vision', providers: $store.modelConfig.chatProviders, searchType: 'chat', apiKeyMode: 'store' }">
81 + <x-component path="/plugins/_model_config/webui/model-field.html"></x-component>
82 + </div>
83 + </div>
84 + </template>
85 </section>
86
87 <section class="preset-model-section">
@@ -166,6 +178,11 @@
178 margin-top: 0;
179 }
180
181 + .vision-sidecar-selector {
182 + margin: 0.75rem 0 0;
183 + padding: 0.25rem 0 0;
184 + }
185 +
186 .preset-editor-secondary-actions {
187 display: flex;
188 flex-wrap: wrap;
plugins/_model_config/webui/model-config-store.js
+40 -6
@@ -7,6 +7,7 @@ import { switcherState, switcherMethods } from "/plugins/_model_config/webui/swi
7
8 export const MODEL_SECTIONS = [
9 { key: 'chat_model', title: 'Main Model', desc: 'Primary model for chat, reasoning, and browser tasks.' },
10 + { key: 'vision_model', title: 'Vision Model', desc: 'Optional model used by vision_load when Main vision is unavailable or overridden.' },
11 { key: 'utility_model', title: 'Utility Model', desc: 'Lightweight model for background tasks: memory management, prompt preparation, summarization.' },
12 { key: 'embedding_model', title: 'Embedding Model', desc: 'Model for generating vector embeddings used in knowledge retrieval.' }
13 ];
@@ -59,6 +60,15 @@ function isBlankPresetValue(value) {
60 }
61
62 const IMPLICIT_PRESET_SLOT_DEFAULTS = {
63 + vision: {
64 + vision: true,
65 + max_embeds: 10,
66 + override_main: false,
67 + rl_requests: 0,
68 + rl_input: 0,
69 + rl_output: 0,
70 + kwargs: {},
71 + },
72 utility: {
73 ctx_length: 128000,
74 ctx_input: 0.7,
@@ -143,15 +153,21 @@ export function configFromPreset(preset, baseConfig, stripApiKey = true) {
153 const config = clonePlain(baseConfig || {});
154 const slots = [
155 ['chat', 'chat_model'],
156 + ['vision', 'vision_model'],
157 ['utility', 'utility_model'],
158 ['embedding', 'embedding_model'],
159 ];
160
161 for (const [slotKey, sectionKey] of slots) {
162 const slot = preset?.[slotKey];
152 - if (!slot || typeof slot !== 'object') continue;
153 - if (!hasModelIdentity(slot)) continue;
154 - config[sectionKey] = mergeModelSlot(config[sectionKey] || {}, slot, stripApiKey, slotKey);
163 + if (slotKey === 'vision') config[sectionKey] = {};
164 + if (!slot || typeof slot !== 'object' || !hasModelIdentity(slot)) continue;
165 + config[sectionKey] = mergeModelSlot(
166 + slotKey === 'vision' ? {} : (config[sectionKey] || {}),
167 + slot,
168 + stripApiKey,
169 + slotKey,
170 + );
171 }
172
173 return config;
@@ -206,8 +222,10 @@ export const store = createStore("modelConfig", {
222 const source = (rawPresets || []).filter(p => p && typeof p === 'object');
223 const rawDefault = source.find(p => String(p.name || '').toLowerCase() === 'default') || {};
224 const slot = value => ({ provider: '', name: '', api_key: '', api_base: '', kwargs: {}, ...(value || {}) });
225 + const visionSlot = value => ({ ...slot(value), vision: true, max_embeds: Number(value?.max_embeds ?? 10), override_main: !!value?.override_main });
226 const defaultConfig = {
227 chat_model: slot(rawDefault.chat),
228 + vision_model: hasModelIdentity(rawDefault.vision) ? visionSlot(rawDefault.vision) : {},
229 utility_model: slot(rawDefault.utility),
230 embedding_model: slot(rawDefault.embedding),
231 };
@@ -219,6 +237,7 @@ export const store = createStore("modelConfig", {
237 return {
238 name: p.name || '',
239 chat: { ...slot(effective.chat_model), _kwargs_text: kwargsToText(effective.chat_model?.kwargs) },
240 + vision: { ...visionSlot(effective.vision_model), _kwargs_text: kwargsToText(effective.vision_model?.kwargs) },
241 utility: { ...slot(effective.utility_model), _kwargs_text: kwargsToText(effective.utility_model?.kwargs) },
242 embedding: { ...slot(effective.embedding_model), _kwargs_text: kwargsToText(effective.embedding_model?.kwargs) },
243 };
@@ -273,6 +292,7 @@ export const store = createStore("modelConfig", {
292 // Config field initialization (converts kwargs dicts to editable text)
293 initConfigFields(config) {
294 if (config?.chat_model) config.chat_model._kwargs_text = kwargsToText(config.chat_model.kwargs);
295 + if (config?.vision_model) config.vision_model._kwargs_text = kwargsToText(config.vision_model.kwargs);
296 if (config?.utility_model) config.utility_model._kwargs_text = kwargsToText(config.utility_model.kwargs);
297 if (config?.embedding_model) config.embedding_model._kwargs_text = kwargsToText(config.embedding_model.kwargs);
298 },
@@ -340,6 +360,7 @@ export const store = createStore("modelConfig", {
360 || this.presets[0]
361 || {
362 chat: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },
363 + vision: { provider: '', name: '', api_base: '', vision: true, max_embeds: 10, override_main: false, kwargs: {}, _kwargs_text: '' },
364 utility: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },
365 embedding: { provider: '', name: '', api_base: '', kwargs: {}, _kwargs_text: '' },
366 }
@@ -420,9 +441,10 @@ export const store = createStore("modelConfig", {
441 const clean = presets.map(p => {
442 const c = { name: p.name };
443 const isDefault = p._originalName === DEFAULT_PRESET_NAME;
423 - for (const slot of ['chat', 'utility']) {
444 + for (const slot of ['chat', 'vision', 'utility']) {
445 if (p[slot]) {
446 const rest = cleanPresetSlot(p[slot], true, slot, isDefault);
447 + if (slot === 'vision' && hasModelIdentity(rest)) rest.vision = true;
448 if (hasModelIdentity(rest)) c[slot] = rest;
449 }
450 }
@@ -608,9 +630,15 @@ export const store = createStore("modelConfig", {
630 const label = (list, id) => (list.find(x => x.value === id) || {}).label || id || '\u2014';
631 return [
632 { icon: 'chat', title: 'Main', cfg: preset?.chat, pList: chatP },
633 + { icon: 'eye', title: 'Vision', cfg: preset?.vision, pList: chatP },
634 { icon: 'manufacturing', title: 'Utility', cfg: preset?.utility, pList: chatP },
635 { icon: 'database', title: 'Embedding', cfg: preset?.embedding, pList: embedP },
613 - ].map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
636 + ]
637 + .filter(s => s.title !== 'Vision' || (
638 + hasModelIdentity(s.cfg)
639 + && (!preset?.chat?.vision || s.cfg?.override_main)
640 + ))
641 + .map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
642 },
643
644 getPreset(name) {
@@ -747,9 +775,15 @@ export const store = createStore("modelConfig", {
775 const label = (list, id) => (list.find(x => x.value === id) || {}).label || id || '\u2014';
776 return [
777 { icon: 'chat', title: 'Main', cfg: cfg.chat_model, pList: chatP },
778 + { icon: 'eye', title: 'Vision', cfg: cfg.vision_model, pList: chatP },
779 { icon: 'manufacturing', title: 'Utility', cfg: cfg.utility_model, pList: chatP },
780 { icon: 'database', title: 'Embedding', cfg: cfg.embedding_model, pList: embedP },
752 - ].map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
781 + ]
782 + .filter(s => s.title !== 'Vision' || (
783 + hasModelIdentity(s.cfg)
784 + && (!cfg.chat_model?.vision || s.cfg?.override_main)
785 + ))
786 + .map(s => ({ icon: s.icon, title: s.title, provider: label(s.pList, s.cfg?.provider), name: s.cfg?.name || '\u2014' }));
787 },
788
789 async refreshModelsSummary(contextId = '') {
plugins/_model_config/webui/model-field.html
+30 -2
@@ -10,13 +10,14 @@
10 Reusable model configuration field set.
11 Parent x-data scope must provide:
12 model — reactive object with provider, name, api_key, api_base, ctx_length, ctx_history, ctx_input, vision, max_embeds, rl_requests, rl_input, rl_output, kwargs, _kwargs_text
13 - modelType — 'chat' | 'utility' | 'embedding'
13 + modelType — 'chat' | 'vision' | 'utility' | 'embedding'
14 providers — array of { value, label }
15 searchType — 'chat' | 'embedding'
16 apiKeyMode — 'store' (key lives in $store.modelConfig.apiKeyValues), 'inline' (key lives in model.api_key), or 'none'
17 Optional:
18 providerFallback — fallback provider string for search/API key status (e.g. preset.chat.provider for utility slot)
19 apiBaseFallback — fallback api_base string for search (e.g. preset.chat.api_base for utility slot)
20 + visionModel — vision slot object, required by the chat model's sidecar switch
21 -->
22 <div x-data="{ get _prov() { return model.provider || (typeof providerFallback !== 'undefined' ? providerFallback : ''); }, get _apiBase() { return model.api_base || (typeof apiBaseFallback !== 'undefined' ? apiBaseFallback : ''); } }">
23 <!-- Provider -->
@@ -146,8 +147,23 @@
147 </div>
148 </template>
149
150 + <template x-if="modelType === 'chat' && model.vision">
151 + <div class="field">
152 + <div class="field-label">
153 + <div class="field-title">Use separate Vision Model</div>
154 + <div class="field-description">When disabled, vision_load uses this model's native vision.</div>
155 + </div>
156 + <div class="field-control">
157 + <label class="toggle">
158 + <input type="checkbox" x-model="visionModel.override_main" />
159 + <span class="toggler"></span>
160 + </label>
161 + </div>
162 + </div>
163 + </template>
164 +
165 <!-- Context window size (main and utility only) -->
150 - <template x-if="modelType !== 'embedding'">
166 + <template x-if="modelType === 'chat' || modelType === 'utility'">
167 <div class="field">
168 <div class="field-label">
169 <div class="field-title">Context window size</div>
@@ -207,6 +223,18 @@
223 </div>
224 </template>
225
226 + <template x-if="modelType === 'vision'">
227 + <div class="field">
228 + <div class="field-label">
229 + <div class="field-title">Max embeds</div>
230 + <div class="field-description">Maximum number of images sent to the Vision Model in one call. Set to 0 for unlimited.</div>
231 + </div>
232 + <div class="field-control">
233 + <input type="number" min="0" x-model.number="model.max_embeds" />
234 + </div>
235 + </div>
236 + </template>
237 +
238 <!-- Utility-specific: ctx_input slider -->
239 <template x-if="modelType === 'utility'">
240 <div class="field">
plugins/_model_config/webui/preset-overview.html
+26 -4
@@ -17,10 +17,22 @@
17
18 <div class="model-preset-rows" aria-live="polite">
19 <template x-for="model in modelRows" :key="model.title">
20 - <div class="model-preset-row">
21 - <x-icon class="model-preset-icon" :name="model.icon"></x-icon>
22 - <span class="model-preset-role" x-text="model.title"></span>
23 - <span class="model-preset-identity">
20 + <div class="model-preset-row"
21 + :class="{ 'model-preset-row-nested': model.title === 'Vision' }">
22 + <template x-if="model.title !== 'Vision'">
23 + <x-icon class="model-preset-icon" :name="model.icon"></x-icon>
24 + </template>
25 + <template x-if="model.title !== 'Vision'">
26 + <span class="model-preset-role" x-text="model.title"></span>
27 + </template>
28 + <span class="model-preset-identity"
29 + :class="{ 'model-preset-identity-vision': model.title === 'Vision' }">
30 + <template x-if="model.title === 'Vision'">
31 + <span>
32 + <span class="model-preset-role">Vision override</span>
33 + <span class="model-preset-separator">/</span>
34 + </span>
35 + </template>
36 <span class="model-preset-provider" x-text="model.provider"></span>
37 <span class="model-preset-separator">/</span>
38 <span x-text="model.name"></span>
@@ -86,6 +98,16 @@
98 border-top: 1px solid var(--color-border);
99 }
100
101 + .model-preset-row-nested {
102 + border-top: 0 !important;
103 + padding-top: var(--spacing-xs);
104 + font-size: 0.76rem;
105 + }
106 +
107 + .model-preset-identity-vision {
108 + grid-column: 3;
109 + }
110 +
111 .model-preset-icon,
112 .model-preset-provider,
113 .model-preset-separator {
prompts/AGENTS.md
+1
@@ -25,6 +25,7 @@
25 - Read the rendering path before changing placeholders or filenames.
26 - Prefer small prompt additions over broad rewrites when fixing a specific behavior.
27 - Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.
28 +- Keep native and sidecar vision prompts synchronized with `vision_load`: related images belong in one call, while a sidecar result is text-only unless Main explicitly requests its native raw route.
29 - Update tests or snapshots when prompt budget, required sections, or generated system content changes.
30
31 ## Verification
prompts/agent.system.tools_vision_sidecar.md new
+48
@@ -0,0 +1,48 @@
1 +## multimodal vision tools
2 +
3 +### vision_load
4 +analyze images with the separate Vision Model and return a text result
5 +args: `paths` list of absolute image paths or ephemeral image refs, `query` optional focused instruction, `raw` optional boolean
6 +Input schema for tool_args:
7 +```json
8 +{
9 + "type": "object",
10 + "properties": {
11 + "paths": {
12 + "type": "array",
13 + "items": {"type": "string"},
14 + "description": "Absolute image paths or ephemeral image refs."
15 + },
16 + "query": {
17 + "type": "string",
18 + "description": "What the Vision Model should inspect, compare, locate, or read."
19 + },
20 + "raw": {
21 + "type": "boolean",
22 + "description": "Use the Main model's native vision instead, when Main supports vision."
23 + }
24 + },
25 + "required": ["paths"],
26 + "additionalProperties": false
27 +}
28 +```
29 +rules:
30 +- put all images needed for one comparison or visual task in the same `paths` array; they are sent in one Vision Model call
31 +- use a focused `query`; if omitted, the Vision Model returns a concise general description
32 +- the result is a text capsule; the Main model does not receive the raw images
33 +- use `raw=true` only when Main supports vision and must inspect the pixels itself
34 +- only bitmaps are supported
35 +example:
36 +```json
37 +{
38 + "thoughts": [
39 + "I need to compare both screenshots before answering."
40 + ],
41 + "headline": "Comparing screenshots",
42 + "tool_name": "vision_load",
43 + "tool_args": {
44 + "paths": ["/path/to/before.png", "/path/to/after.png"],
45 + "query": "Compare the error banners and describe what changed."
46 + }
47 +}
48 +```
tests/test_browser_agent_regressions.py
+13 -7
@@ -36,8 +36,13 @@ class _TestAgentContextType:
36
37
38 class _TestResponse(SimpleNamespace):
39 - def __init__(self, message="", break_loop=False, **kwargs):
40 - super().__init__(message=message, break_loop=break_loop, **kwargs)
39 + def __init__(self, message="", break_loop=False, additional=None, **kwargs):
40 + super().__init__(
41 + message=message,
42 + break_loop=break_loop,
43 + additional=additional,
44 + **kwargs,
45 + )
46
47
48 class _TestTool:
@@ -96,6 +101,9 @@ _model_config_stub = ModuleType("plugins._model_config.helpers.model_config")
101 _model_config_stub.get_presets = lambda: []
102 _model_config_stub.get_preset_by_name = lambda name: None
103 _model_config_stub.get_chat_model_config = lambda agent=None: {}
104 +_model_config_stub.get_vision_model_config = lambda agent=None: {}
105 +_model_config_stub.use_vision_sidecar = lambda agent=None: False
106 +_model_config_stub.build_vision_model = lambda agent=None: None
107 sys.modules.setdefault("plugins._model_config.helpers.model_config", _model_config_stub)
108
109
@@ -4297,11 +4305,9 @@ async def test_vision_load_materializes_ephemeral_browser_refs(monkeypatch, tmp_
4305
4306 monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
4307 monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
4300 - monkeypatch.setattr(
4301 - vision_load_module.plugins,
4302 - "get_plugin_config",
4303 - lambda *args, **kwargs: {"chat_model": {"max_embeds": 10}},
4304 - )
4308 + monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": True, "max_embeds": 10})
4309 + monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
4310 + monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
4311
4312 tool_results = []
4313 messages = []
tests/test_model_config_api_keys.py
+1 -1
@@ -149,7 +149,7 @@ def test_model_config_frontend_tracks_provider_api_key_edits():
149 assert "/plugins/_model_config/missing_api_key_status" not in model_gate_content
150 assert '@input="$store.modelConfig.setApiKeyValue(_prov, $el.value)"' in config_content
151 assert "apiKeyMode: 'none'" not in preset_modal_content
152 - assert preset_modal_content.count("apiKeyMode: 'store'") == 3
152 + assert preset_modal_content.count("apiKeyMode: 'store'") == 4
153 assert "$store.modelConfig.resetApiKeyDrafts();" in preset_modal_content
154 assert "await $store.modelConfig.refreshApiKeyStatus();" in preset_modal_content
155 assert "await store.persistAllDirtyApiKeys();" in store_content
tests/test_model_config_project_presets.py
+38
@@ -1092,6 +1092,44 @@ def test_preset_application_inherits_optional_slots(monkeypatch, tmp_path):
1092 assert config["embedding_model"] == base_config["embedding_model"]
1093
1094
1095 +def test_preset_vision_slot_is_optional_and_never_inherited(monkeypatch, tmp_path):
1096 + _prepare_a0_tree(monkeypatch, tmp_path)
1097 +
1098 + from plugins._model_config.helpers import model_config
1099 +
1100 + base_config = {
1101 + "chat_model": {"provider": "openrouter", "name": "main", "vision": False},
1102 + "vision_model": {
1103 + "provider": "openrouter",
1104 + "name": "default-vision",
1105 + "max_embeds": 2,
1106 + },
1107 + }
1108 +
1109 + without_sidecar = model_config.build_config_from_preset(
1110 + {"name": "Text only", "chat": {"provider": "openrouter", "name": "text"}},
1111 + base_config,
1112 + )
1113 + with_sidecar = model_config.build_config_from_preset(
1114 + {
1115 + "name": "Visual",
1116 + "chat": {"provider": "openrouter", "name": "text"},
1117 + "vision": {
1118 + "provider": "anthropic",
1119 + "name": "visual",
1120 + "max_embeds": 5,
1121 + },
1122 + },
1123 + base_config,
1124 + )
1125 +
1126 + assert without_sidecar["vision_model"] == {}
1127 + assert with_sidecar["vision_model"]["provider"] == "anthropic"
1128 + assert with_sidecar["vision_model"]["name"] == "visual"
1129 + assert with_sidecar["vision_model"]["max_embeds"] == 5
1130 + assert "default-vision" not in str(with_sidecar["vision_model"])
1131 +
1132 +
1133 def test_legacy_utility_preset_defaults_preserve_tuning_but_clear_kwargs(
1134 monkeypatch,
1135 tmp_path,
tests/test_model_config_ui.py
+43
@@ -85,6 +85,49 @@ def test_preset_editor_uses_standard_modal_footer_buttons() -> None:
85 assert "preset-editor-footer" not in preset_modal
86
87
88 +def test_preset_editor_nests_one_conditional_vision_selector_in_main() -> None:
89 + preset_modal = read("plugins", "_model_config", "webui", "main.html")
90 + model_field = read("plugins", "_model_config", "webui", "model-field.html")
91 + preset_overview = read("plugins", "_model_config", "webui", "preset-overview.html")
92 + preset_store = read("plugins", "_model_config", "webui", "model-config-store.js")
93 +
94 + main_start = preset_modal.index('<div class="section-title">Main Model</div>')
95 + utility_start = preset_modal.index('<div class="section-title">Utility Model</div>')
96 + selector_start = preset_modal.index('class="vision-sidecar-selector"')
97 + supports_start = model_field.index('<div class="field-title">Supports Vision</div>')
98 + override_start = model_field.index('<div class="field-title">Use separate Vision Model</div>')
99 + context_start = model_field.index('<div class="field-title">Context window size</div>')
100 + advanced_start = model_field.index('<!-- Advanced Settings (collapsed by default) -->')
101 +
102 + assert '<div class="section-title">Vision Model</div>' not in preset_modal
103 + assert preset_modal.count('class="vision-sidecar-selector"') == 1
104 + assert main_start < selector_start < utility_start
105 + assert '!selectedPreset.chat.vision || selectedPreset.vision.override_main' in preset_modal
106 + assert "get visionModel() { return selectedPreset.vision; }" in preset_modal
107 + assert "modelType: 'vision'" in preset_modal
108 + assert preset_modal.count("apiKeyMode: 'store'") == 4
109 + assert "margin: 0.75rem 0 0;" in preset_modal
110 + assert "padding: 0.25rem 0 0;" in preset_modal
111 + assert "border-left: 2px solid var(--color-border);" not in preset_modal
112 + assert "Use separate Vision Model" in model_field
113 + assert supports_start < override_start < context_start < advanced_start
114 + assert "When disabled, vision_load uses this model's native vision." in model_field
115 + assert "When enabled, vision_load uses the preset's Vision Model" not in model_field
116 + assert 'x-model="visionModel.override_main"' in model_field
117 + assert '<div class="advanced-section" x-data="{ advOpen: false }">' in model_field
118 + assert "model-preset-row-nested" in preset_overview
119 + assert "model.title === 'Vision'" in preset_overview
120 + assert preset_overview.count("model.title !== 'Vision'") == 2
121 + assert '<span class="model-preset-role">Vision override</span>' in preset_overview
122 + assert "'model-preset-identity-vision': model.title === 'Vision'" in preset_overview
123 + assert "grid-column: 3;" in preset_overview
124 + assert "padding-top: var(--spacing-xs);" in preset_overview
125 + assert "margin-left: 2rem;" not in preset_overview
126 + assert "border-left: 1px solid var(--color-border);" not in preset_overview
127 + assert "if (slotKey === 'vision') config[sectionKey] = {};" in preset_store
128 + assert "['chat', 'vision', 'utility']" in preset_store
129 +
130 +
131 def test_plugin_settings_reset_is_explicit_and_does_not_capture_toast_early() -> None:
132 settings_modal = read("webui", "components", "plugins", "plugin-settings.html")
133 settings_store = read("webui", "components", "plugins", "plugin-settings-store.js")
tests/test_parallel_tool.py
+36
@@ -160,6 +160,42 @@ def test_normalize_parallel_tool_calls_accepts_json_string_array() -> None:
160 assert calls[1].tool_args["message"] == "Research nuclear fusion news in Italian."
161
162
163 +def test_parallel_keeps_mixed_tools_and_batched_vision_paths() -> None:
164 + calls = parallel_tools.normalize_parallel_tool_calls(
165 + [
166 + {"tool_name": "search_a", "tool_args": {"query": "one"}},
167 + {"tool_name": "search_b", "tool_args": {"query": "two"}},
168 + {"tool_name": "browser_agent", "tool_args": {"message": "open page"}},
169 + {
170 + "tool_name": "vision_load",
171 + "tool_args": {
172 + "paths": ["/tmp/before.png", "/tmp/after.png"],
173 + "query": "compare",
174 + },
175 + },
176 + ]
177 + )
178 +
179 + assert [call.tool_name for call in calls] == [
180 + "search_a",
181 + "search_b",
182 + "browser_agent",
183 + "vision_load",
184 + ]
185 + assert calls[3].tool_args["paths"] == ["/tmp/before.png", "/tmp/after.png"]
186 +
187 +
188 +def test_parallel_allows_multiple_independent_vision_calls() -> None:
189 + calls = parallel_tools.normalize_parallel_tool_calls(
190 + [
191 + {"tool_name": "vision_load", "tool_args": {"paths": ["/tmp/a.png"]}},
192 + {"tool_name": "vision_load", "tool_args": {"paths": ["/tmp/b.png"]}},
193 + ]
194 + )
195 +
196 + assert [call.tool_name for call in calls] == ["vision_load", "vision_load"]
197 +
198 +
199 def test_subordinate_prompts_share_reusable_tree_contract() -> None:
200 call_prompt = (PROJECT_ROOT / "prompts/agent.system.tool.call_sub.md").read_text(
201 encoding="utf-8"
tests/test_tool_policy.py
+53
@@ -546,6 +546,59 @@ async def test_vision_tool_follows_chat_config_not_profile_policy(
546 assert tool_policy.resolve_tool(agent, "vision_load").source == "runtime-config"
547
548
549 +@pytest.mark.asyncio
550 +async def test_vision_sidecar_prompt_replaces_native_vision_prompt(
551 + monkeypatch, tmp_path: Path
552 +) -> None:
553 + _write_prompt(tmp_path, "agent.system.tools.md", "TOOLS\n{{tools}}")
554 + _write_prompt(
555 + tmp_path,
556 + "agent.system.tools_vision.md",
557 + "### vision_load\nnative pixels\nargs: `paths`",
558 + )
559 + _write_prompt(
560 + tmp_path,
561 + "agent.system.tools_vision_sidecar.md",
562 + "### vision_load\nsidecar capsule\nargs: `paths`, `query`",
563 + )
564 + monkeypatch.setattr(tool_policy.subagents, "get_paths", _prompt_paths(tmp_path))
565 + monkeypatch.setattr(
566 + "plugins._model_config.helpers.model_config.get_chat_model_config",
567 + lambda agent: {"vision": True},
568 + )
569 + monkeypatch.setattr(
570 + "plugins._model_config.helpers.model_config.use_vision_sidecar",
571 + lambda agent: True,
572 + )
573 + monkeypatch.setattr(responses_tools, "_mcp_tools", lambda agent: [])
574 + agent = _Agent(tmp_path)
575 +
576 + prompt = await _11_tools_prompt.build_prompt(agent)
577 + schemas, _name_map = responses_tools.build_responses_function_tools(agent)
578 +
579 + assert "sidecar capsule" in prompt
580 + assert "native pixels" not in prompt
581 + assert schemas[0]["name"] == "vision_load"
582 + assert schemas[0]["description"] == "sidecar capsule"
583 +
584 +
585 +def test_vision_sidecar_prompt_declares_multi_image_native_schema() -> None:
586 + prompt = (
587 + Path(__file__).resolve().parents[1]
588 + / "prompts"
589 + / "agent.system.tools_vision_sidecar.md"
590 + ).read_text(encoding="utf-8")
591 + schema = responses_tools._schema_from_prompt(prompt)
592 +
593 + assert schema["required"] == ["paths"]
594 + assert schema["properties"]["paths"] == {
595 + "type": "array",
596 + "items": {"type": "string"},
597 + "description": "Absolute image paths or ephemeral image refs.",
598 + }
599 + assert {"query", "raw"} <= schema["properties"].keys()
600 +
601 +
602 def test_mcp_prompt_and_native_schema_omit_blocked_tool(
603 monkeypatch, tmp_path: Path
604 ) -> None:
tests/test_vision_load_image_refs.py
+223 -7
@@ -1,3 +1,4 @@
1 +import asyncio
2 import types
3 from types import SimpleNamespace
4 import sys
@@ -13,8 +14,13 @@ from helpers import images
14
15
16 class _TestResponse(SimpleNamespace):
16 - def __init__(self, message="", break_loop=False, **kwargs):
17 - super().__init__(message=message, break_loop=break_loop, **kwargs)
17 + def __init__(self, message="", break_loop=False, additional=None, **kwargs):
18 + super().__init__(
19 + message=message,
20 + break_loop=break_loop,
21 + additional=additional,
22 + **kwargs,
23 + )
24
25
26 class _TestTool:
@@ -74,11 +80,9 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatch
80
81 monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
82 monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
77 - monkeypatch.setattr(
78 - vision_load_module.plugins,
79 - "get_plugin_config",
80 - lambda *args, **kwargs: {"chat_model": {"max_embeds": 10}},
81 - )
83 + monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": True, "max_embeds": 10})
84 + monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
85 + monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
86
87 async def direct_call(func, *args, **kwargs):
88 return func(*args, **kwargs)
@@ -121,3 +125,215 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatch
125 stored_path = tmp_path / stored_ref.removeprefix("/a0/")
126 assert stored_path.read_bytes() == b"png-data"
127 assert updates[-1]["result"] == "1 images loaded, 0 skipped"
128 +
129 +
130 +def test_vision_sidecar_route_matrix_prefers_main_native_vision(monkeypatch):
131 + from plugins._model_config.helpers import model_config
132 +
133 + cases = [
134 + ({"vision": False}, {}, False),
135 + ({"vision": True}, {"provider": "p", "name": "v"}, False),
136 + ({"vision": False}, {"provider": "p", "name": "v"}, True),
137 + (
138 + {"vision": True},
139 + {"provider": "p", "name": "v", "override_main": True},
140 + True,
141 + ),
142 + ]
143 + for chat, vision, expected in cases:
144 + monkeypatch.setattr(
145 + model_config,
146 + "get_effective_config",
147 + lambda _agent=None, chat=chat, vision=vision: {
148 + "chat_model": chat,
149 + "vision_model": vision,
150 + },
151 + )
152 + assert model_config.use_vision_sidecar() is expected
153 +
154 +
155 +@pytest.mark.anyio
156 +async def test_vision_sidecar_sends_multiple_images_once_and_keeps_history_text_only(
157 + monkeypatch,
158 + tmp_path,
159 +):
160 + _install_tool_stub(monkeypatch)
161 + import tools.vision_load as vision_load_module
162 +
163 + async def direct_call(func, *args, **kwargs):
164 + return func(*args, **kwargs)
165 +
166 + calls = []
167 +
168 + class FakeVisionModel:
169 + async def unified_call(self, **kwargs):
170 + calls.append(kwargs)
171 + return "The second screenshot fixes the red login error.", ""
172 +
173 + monkeypatch.setattr(vision_load_module.runtime, "call_development_function", direct_call)
174 + monkeypatch.setattr(vision_load_module, "build_vision_model", lambda _agent: FakeVisionModel())
175 + monkeypatch.setattr(
176 + vision_load_module,
177 + "get_chat_model_config",
178 + lambda _agent: {"vision": True, "max_embeds": 1},
179 + )
180 + monkeypatch.setattr(
181 + vision_load_module,
182 + "get_vision_model_config",
183 + lambda _agent: {"provider": "test", "name": "vision", "max_embeds": 5},
184 + )
185 + monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: True)
186 +
187 + image_paths = [tmp_path / "before.png", tmp_path / "after.png"]
188 + for path in image_paths:
189 + path.write_bytes(b"png-data")
190 +
191 + tool_results = []
192 + raw_messages = []
193 + agent = SimpleNamespace(
194 + context=SimpleNamespace(id=""),
195 + agent_name="Agent 0",
196 + hist_add_tool_result=lambda *args, **kwargs: tool_results.append((args, kwargs)),
197 + hist_add_message=lambda *args, **kwargs: raw_messages.append((args, kwargs)),
198 + )
199 + tool = vision_load_module.VisionLoad(
200 + agent=agent,
201 + name="vision_load",
202 + method=None,
203 + args={"paths": [str(path) for path in image_paths]},
204 + message="",
205 + loop_data=None,
206 + )
207 + tool.log = SimpleNamespace(id="vision-log", update=lambda **kwargs: None)
208 +
209 + response = await tool.execute(
210 + paths=[str(path) for path in image_paths],
211 + query="Compare the login errors.",
212 + )
213 + response.additional = {"_responses_output_item": {"output": response.message}}
214 + await tool.after_execution(response)
215 +
216 + assert len(calls) == 1
217 + content = calls[0]["messages"][1].content
218 + assert content[0] == {"type": "text", "text": "Compare the login errors."}
219 + assert [item["type"] for item in content].count("image_url") == 2
220 + assert "fixes the red login error" in response.message
221 + assert response.message != "dummy"
222 + assert raw_messages == []
223 + assert tool.loaded_paths == [str(path) for path in image_paths]
224 + assert tool_results[0][1]["_responses_output_item"]["output"] == response.message
225 +
226 +
227 +@pytest.mark.anyio
228 +async def test_parallel_worker_consumes_parent_ephemeral_image(monkeypatch, tmp_path):
229 + _install_tool_stub(monkeypatch)
230 + import tools.vision_load as vision_load_module
231 +
232 + def fake_get_abs_path(*parts):
233 + return str(tmp_path.joinpath(*parts))
234 +
235 + def fake_normalize_a0_path(path):
236 + return "/a0/" + str(Path(path).relative_to(tmp_path)).replace("\\", "/")
237 +
238 + monkeypatch.setattr(vision_load_module.chat_media.files, "get_abs_path", fake_get_abs_path)
239 + monkeypatch.setattr(vision_load_module.chat_media.files, "normalize_a0_path", fake_normalize_a0_path)
240 + monkeypatch.setattr(vision_load_module.VisionLoad, "_config_agent", lambda self: self.agent)
241 + monkeypatch.setattr(
242 + vision_load_module,
243 + "get_chat_model_config",
244 + lambda _agent: {"vision": True, "max_embeds": 10},
245 + )
246 + monkeypatch.setattr(vision_load_module, "get_vision_model_config", lambda _agent: {})
247 + monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: False)
248 +
249 + parent_id = "parent-vision"
250 + ref = vision_load_module.ephemeral_images.put_image_bytes(
251 + context_id=parent_id,
252 + mime="image/png",
253 + payload=b"png-data",
254 + name="shot.png",
255 + )
256 + context = SimpleNamespace(
257 + id="parallel-worker",
258 + get_data=lambda key: parent_id
259 + if key == vision_load_module.PARALLEL_WORKER_PARENT_CONTEXT_KEY
260 + else None,
261 + )
262 + agent = SimpleNamespace(context=context, agent_name="Agent 0")
263 + tool = vision_load_module.VisionLoad(
264 + agent=agent,
265 + name="vision_load",
266 + method=None,
267 + args={"paths": [ref]},
268 + message="",
269 + loop_data=None,
270 + )
271 +
272 + await tool.execute(paths=[ref])
273 +
274 + assert tool.loaded_paths == ["shot.png"]
275 + assert vision_load_module.ephemeral_images.get_image(ref, context_id=parent_id) is None
276 + stored_ref = tool.images_dict["shot.png"]
277 + assert stored_ref.startswith("/a0/usr/chats/parent-vision/images/vision-load/shot-")
278 +
279 +
280 +@pytest.mark.anyio
281 +async def test_independent_vision_sidecar_calls_can_run_concurrently(monkeypatch, tmp_path):
282 + _install_tool_stub(monkeypatch)
283 + import tools.vision_load as vision_load_module
284 +
285 + active = 0
286 + max_active = 0
287 + call_count = 0
288 +
289 + class FakeVisionModel:
290 + async def unified_call(self, **kwargs):
291 + nonlocal active, max_active, call_count
292 + active += 1
293 + call_count += 1
294 + max_active = max(max_active, active)
295 + await asyncio.sleep(0.02)
296 + active -= 1
297 + return "done", ""
298 +
299 + async def direct_call(func, *args, **kwargs):
300 + return func(*args, **kwargs)
301 +
302 + monkeypatch.setattr(vision_load_module.runtime, "call_development_function", direct_call)
303 + monkeypatch.setattr(vision_load_module, "build_vision_model", lambda _agent: FakeVisionModel())
304 + monkeypatch.setattr(vision_load_module, "get_chat_model_config", lambda _agent: {"vision": False})
305 + monkeypatch.setattr(
306 + vision_load_module,
307 + "get_vision_model_config",
308 + lambda _agent: {"provider": "test", "name": "vision", "max_embeds": 10},
309 + )
310 + monkeypatch.setattr(vision_load_module, "use_vision_sidecar", lambda _agent: True)
311 +
312 + image_paths = [tmp_path / "one.png", tmp_path / "two.png"]
313 + for path in image_paths:
314 + path.write_bytes(b"png-data")
315 +
316 + def make_tool(index):
317 + agent = SimpleNamespace(context=SimpleNamespace(id=""), agent_name=f"Agent {index}")
318 + return vision_load_module.VisionLoad(
319 + agent=agent,
320 + name="vision_load",
321 + method=None,
322 + args={"paths": [str(path) for path in image_paths]},
323 + message="",
324 + loop_data=None,
325 + )
326 +
327 + responses = await asyncio.gather(
328 + *(
329 + make_tool(index).execute(
330 + paths=[str(path) for path in image_paths],
331 + query=f"inspection {index}",
332 + )
333 + for index in range(4)
334 + )
335 + )
336 +
337 + assert call_count == 4
338 + assert max_active == 4
339 + assert all("done" in response.message for response in responses)
tools/vision_load.py
+214 -106
@@ -1,89 +1,219 @@
1 -from helpers.print_style import PrintStyle
2 -from helpers.tool import Tool, Response
3 -from helpers import runtime, files, plugins, ephemeral_images, images, chat_media
1 +import asyncio
2 +import json
3 from mimetypes import guess_type
5 -from helpers import history
4
7 -# image token estimation for context window
5 +from langchain_core.messages import HumanMessage, SystemMessage
6 +
7 +from helpers import chat_media, ephemeral_images, files, history, images, runtime
8 +from helpers.parallel_tools import PARALLEL_WORKER_PARENT_CONTEXT_KEY, coerce_bool
9 +from helpers.print_style import PrintStyle
10 +from helpers.tool import Response, Tool
11 +from plugins._model_config.helpers.model_config import (
12 + build_vision_model,
13 + get_chat_model_config,
14 + get_vision_model_config,
15 + use_vision_sidecar,
16 +)
17 +
18 TOKENS_ESTIMATE = 1500
19 +VISION_TIMEOUT_SECONDS = 300
20 +VISION_SYSTEM_PROMPT = (
21 + "You are a precise vision analyst. Answer only what was asked about the images. "
22 + "Be concise and factual. Preserve exact visible text when asked to read it."
23 +)
24 +DEFAULT_VISION_QUERY = (
25 + "Describe the images precisely, including the key objects, visible text, and layout."
26 +)
27
28
29 class VisionLoad(Tool):
12 - async def execute(self, paths: list[str] = [], **kwargs) -> Response:
13 -
14 - self.images_dict = {}
30 + async def execute(
31 + self,
32 + paths: list[str] | str | None = None,
33 + query: str = "",
34 + raw: bool = False,
35 + **kwargs,
36 + ) -> Response:
37 + self.images_dict: dict[str, str] = {}
38 self.loaded_paths: list[str] = []
39 self.skipped_paths: list[str] = []
40 + self._config_owner = self._config_agent()
41 + self._main_has_vision = bool(
42 + get_chat_model_config(self._config_owner).get("vision", False)
43 + )
44 + self._delegated = use_vision_sidecar(self._config_owner) and not (
45 + coerce_bool(raw, False) and self._main_has_vision
46 + )
47 + self._max_embeds = self._get_max_embeds()
48 +
49 + normalized = self._normalize_paths(paths)
50 + if isinstance(normalized, str):
51 + self._history_result = normalized
52 + return Response(message=normalized, break_loop=False)
53
18 - max_embeds = self._get_max_embeds()
54 requested = [
20 - (str(path or "").strip(), self._display_input_path(str(path or "").strip(), idx + 1))
21 - for idx, path in enumerate(paths)
55 + (path.strip(), self._display_input_path(path.strip(), index + 1))
56 + for index, path in enumerate(normalized)
57 ]
23 - limited_paths = requested if max_embeds <= 0 else requested[-max_embeds:]
24 - self.skipped_paths = (
25 - [display for _, display in requested[:-max_embeds]]
26 - if max_embeds > 0 and len(requested) > max_embeds
27 - else []
28 - )
58 + limited = requested if self._max_embeds <= 0 else requested[-self._max_embeds :]
59 + if self._max_embeds > 0 and len(requested) > self._max_embeds:
60 + self.skipped_paths = [display for _, display in requested[: -self._max_embeds]]
61
30 - for idx, (path, display_path) in enumerate(limited_paths):
62 + for index, (path, display_path) in enumerate(limited):
63 if not path:
64 continue
65 if ephemeral_images.is_ref(path):
34 - image = ephemeral_images.consume_image(
35 - path,
36 - context_id=self._context_id(),
37 - )
66 + image = ephemeral_images.consume_image(path, context_id=self._context_id())
67 if image is None:
68 continue
40 - display = image.display_name or display_path
69 + display_path = image.display_name or display_path
70 stored_ref = self._store_ephemeral_image(image)
42 - if stored_ref:
43 - self.images_dict[display] = stored_ref
44 - self.loaded_paths.append(display)
45 - continue
46 - if self._is_data_image_url(path):
47 - stored_ref = self._store_data_url(path, preferred_name=f"vision-load-{idx + 1}.png")
48 - if stored_ref:
49 - self.images_dict[display_path] = stored_ref
50 - self.loaded_paths.append(display_path)
51 - continue
52 - if not await runtime.call_development_function(files.exists, str(path)):
71 + elif self._is_data_image_url(path):
72 + stored_ref = self._store_data_url(
73 + path, preferred_name=f"vision-load-{index + 1}.png"
74 + )
75 + elif await runtime.call_development_function(files.exists, path):
76 + mime_type, _ = guess_type(path)
77 + if not mime_type or not mime_type.startswith("image/"):
78 + continue
79 + try:
80 + stored_ref = self._store_local_image(
81 + path, preferred_name=files.basename(path)
82 + )
83 + except (FileNotFoundError, OSError, ValueError):
84 + continue
85 + else:
86 continue
87
55 - if path not in self.images_dict:
56 - mime_type, _ = guess_type(str(path))
57 - if mime_type and mime_type.startswith("image/"):
58 - try:
59 - stored_ref = self._store_local_image(path, preferred_name=files.basename(path))
60 - self.images_dict[display_path] = stored_ref
61 - self.loaded_paths.append(display_path)
62 - except (FileNotFoundError, OSError, ValueError):
63 - continue
88 + if stored_ref:
89 + self.images_dict[display_path] = stored_ref
90 + self.loaded_paths.append(display_path)
91
65 - return Response(message="dummy", break_loop=False)
92 + summary = self._summary()
93 + if self._delegated and self.images_dict:
94 + try:
95 + capsule = await self._call_vision_model(
96 + list(self.images_dict.values()), self._query(query, kwargs)
97 + )
98 + message = (
99 + f"Vision Model analyzed {len(self.images_dict)} image(s)"
100 + f"; {len(self.skipped_paths)} skipped.\n\n{capsule.strip()}"
101 + )
102 + self._history_result = message
103 + return Response(message=message, break_loop=False)
104 + except Exception as exc:
105 + message = f"Vision Model error: {str(exc)[:1000]}"
106 + self._history_result = f"{summary}\n\n{message}"
107 + return Response(message=message, break_loop=False)
108 +
109 + if self.images_dict and not self._main_has_vision:
110 + summary += (
111 + "\n\nImages were not injected because neither Main native vision nor "
112 + "a usable Vision Model is active."
113 + )
114 + self._history_result = (
115 + summary if self.images_dict or self.skipped_paths else "No images processed"
116 + )
117 + message = (
118 + "No images processed"
119 + if not self.images_dict and not self.skipped_paths
120 + else f"{len(self.images_dict)} images loaded, {len(self.skipped_paths)} skipped"
121 + )
122 + return Response(message=message, break_loop=False)
123 +
124 + async def after_execution(self, response: Response, **kwargs):
125 + log_id = str(getattr(getattr(self, "log", None), "id", "") or "")
126 + self.agent.hist_add_tool_result(
127 + self.name,
128 + self._history_result,
129 + id=log_id,
130 + **(response.additional or {}),
131 + )
132 +
133 + if self.images_dict and self._main_has_vision and not self._delegated:
134 + content = [
135 + {"type": "image_url", "image_url": {"url": image_path}}
136 + for image_path in self.images_dict.values()
137 + ]
138 + self.agent.hist_add_message(
139 + False,
140 + content=history.RawMessage(
141 + raw_content=content,
142 + preview="<Image attachments loaded by path>",
143 + ),
144 + tokens=TOKENS_ESTIMATE * len(content),
145 + )
146 +
147 + PrintStyle(
148 + font_color="#1B4F72", background_color="white", padding=True, bold=True
149 + ).print(f"{self.agent.agent_name}: Response from tool '{self.name}'")
150 + PrintStyle(font_color="#85C1E9").print(response.message)
151 + if getattr(self, "log", None):
152 + self.log.update(result=response.message)
153
154 def _get_max_embeds(self) -> int:
68 - cfg = plugins.get_plugin_config("_model_config", agent=self.agent) or {}
69 - chat_cfg = cfg.get("chat_model", {})
70 - max_embeds = chat_cfg.get("max_embeds", 10)
71 - return int(max_embeds or 0)
155 + cfg = (
156 + get_vision_model_config(self._config_owner)
157 + if self._delegated
158 + else get_chat_model_config(self._config_owner)
159 + )
160 + try:
161 + return int(cfg.get("max_embeds", 10) or 0)
162 + except (TypeError, ValueError):
163 + return 10
164
165 def _context_id(self) -> str:
74 - return str(getattr(getattr(self.agent, "context", None), "id", "") or "").strip()
166 + context = getattr(self.agent, "context", None)
167 + if not context:
168 + return ""
169 + get_data = getattr(context, "get_data", None)
170 + parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""
171 + return str(parent_id or getattr(context, "id", "") or "").strip()
172 +
173 + def _config_agent(self):
174 + context = getattr(self.agent, "context", None)
175 + get_data = getattr(context, "get_data", None)
176 + parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""
177 + if parent_id:
178 + from agent import AgentContext
179 +
180 + parent = AgentContext.get(str(parent_id))
181 + if parent:
182 + return parent.agent0
183 + return self.agent
184 +
185 + async def _call_vision_model(self, image_paths: list[str], query: str) -> str:
186 + model = build_vision_model(self._config_owner)
187 + content = [{"type": "text", "text": query or DEFAULT_VISION_QUERY}]
188 + content.extend(
189 + {"type": "image_url", "image_url": {"url": path}}
190 + for path in image_paths
191 + )
192 + response, _ = await asyncio.wait_for(
193 + model.unified_call(
194 + messages=[
195 + SystemMessage(content=VISION_SYSTEM_PROMPT),
196 + HumanMessage(content=content),
197 + ],
198 + explicit_caching=False,
199 + max_tokens=2000,
200 + ),
201 + timeout=VISION_TIMEOUT_SECONDS,
202 + )
203 + if not str(response or "").strip():
204 + raise RuntimeError("Vision Model returned an empty response.")
205 + return str(response)
206
207 def _store_ephemeral_image(self, image: ephemeral_images.EphemeralImage) -> str:
208 context_id = self._context_id()
209 if not context_id:
210 return image.data_url
211 source = chat_media.infer_source(image.ref, image.display_name)
81 - category = chat_media.category_for_source(source)
212 saved = chat_media.save_image_base64(
213 context_id=context_id,
214 data=image.data,
215 mime_type=image.mime,
86 - category=category,
216 + category=chat_media.category_for_source(source),
217 source=source,
218 preferred_name=image.display_name,
219 )
@@ -94,11 +224,10 @@ class VisionLoad(Tool):
224 if not context_id:
225 return data_url
226 source = chat_media.infer_source(data_url, preferred_name)
97 - category = chat_media.category_for_source(source)
227 saved = chat_media.save_image_data_url(
228 context_id=context_id,
229 data_url=data_url,
101 - category=category,
230 + category=chat_media.category_for_source(source),
231 source=source,
232 preferred_name=preferred_name,
233 )
@@ -115,6 +244,37 @@ class VisionLoad(Tool):
244 preferred_name=preferred_name,
245 )
246
247 + def _summary(self) -> str:
248 + loaded = "\n".join(self.loaded_paths) if self.loaded_paths else "none"
249 + skipped = "\n".join(self.skipped_paths) if self.skipped_paths else "none"
250 + return (
251 + f"Loaded images ({len(self.loaded_paths)}):\n{loaded}\n\n"
252 + f"Skipped images ({len(self.skipped_paths)}, max {self._max_embeds}):\n{skipped}"
253 + )
254 +
255 + @staticmethod
256 + def _normalize_paths(paths: list[str] | str | None) -> list[str] | str:
257 + if isinstance(paths, str):
258 + try:
259 + decoded = json.loads(paths)
260 + except json.JSONDecodeError:
261 + decoded = paths
262 + paths = decoded if isinstance(decoded, list) else [paths]
263 + if paths is None:
264 + return []
265 + if not isinstance(paths, (list, tuple)):
266 + return "vision_load error: `paths` must be an array of image paths."
267 + return [str(path or "").strip() for path in paths]
268 +
269 + @staticmethod
270 + def _query(query: str, kwargs: dict) -> str:
271 + if str(query or "").strip():
272 + return str(query).strip()
273 + for key in ("prompt", "question", "instruction", "focus", "request"):
274 + if str(kwargs.get(key) or "").strip():
275 + return str(kwargs[key]).strip()
276 + return DEFAULT_VISION_QUERY
277 +
278 @staticmethod
279 def _is_data_image_url(value: str) -> bool:
280 normalized = str(value or "").strip().lower()
@@ -125,57 +285,5 @@ class VisionLoad(Tool):
285 if ephemeral_images.is_ref(value):
286 return ephemeral_images.display_ref(value)
287 if cls._is_data_image_url(value):
128 - prefix = value.split(",", 1)[0]
129 - return f"{prefix},<ephemeral-image-{index}>"
288 + return f"{value.split(',', 1)[0]},<ephemeral-image-{index}>"
289 return value
131 -
132 - async def after_execution(self, response: Response, **kwargs):
133 -
134 - # build image data messages for LLMs, or error message
135 - content = []
136 - loaded_count = len(self.loaded_paths)
137 - skipped_count = len(self.skipped_paths)
138 - loaded_summary = "\n".join(self.loaded_paths) if self.loaded_paths else "none"
139 - skipped_summary = "\n".join(self.skipped_paths) if self.skipped_paths else "none"
140 - summary = (
141 - f"Loaded images: {loaded_count}\n"
142 - f"Loaded images:\n{loaded_summary}\n\n"
143 - f"Skipped images: {skipped_count}\n"
144 - f"Skipped images (max {self._get_max_embeds()} loaded at a time according to model configuration):\n{skipped_summary}"
145 - )
146 - if self.images_dict:
147 - self.agent.hist_add_tool_result(self.name, summary, id=self.log.id if self.log else "")
148 - for path, image_path in self.images_dict.items():
149 - if image_path:
150 - content.append(
151 - {
152 - "type": "image_url",
153 - "image_url": {"url": image_path},
154 - }
155 - )
156 - else:
157 - content.append(
158 - {
159 - "type": "text",
160 - "text": "Error processing image " + path,
161 - }
162 - )
163 - # append as raw message content for LLMs with vision tokens estimate
164 - msg = history.RawMessage(raw_content=content, preview="<Image attachments loaded by path>")
165 - self.agent.hist_add_message(
166 - False, content=msg, tokens=TOKENS_ESTIMATE * len(content)
167 - )
168 - else:
169 - self.agent.hist_add_tool_result(self.name, summary if self.skipped_paths else "No images processed", id=self.log.id if self.log else "")
170 -
171 - # print and log short version
172 - message = (
173 - "No images processed"
174 - if not self.images_dict and not self.skipped_paths
175 - else f"{loaded_count} images loaded, {skipped_count} skipped"
176 - )
177 - PrintStyle(
178 - font_color="#1B4F72", background_color="white", padding=True, bold=True
179 - ).print(f"{self.agent.agent_name}: Response from tool '{self.name}'")
180 - PrintStyle(font_color="#85C1E9").print(message)
181 - self.log.update(result=message)
tools/vision_load.py.dox.md
+9 -3
@@ -3,7 +3,7 @@
3 ## Purpose
4
5 - Own the `vision_load.py` agent tool.
6 -- This module loads images into model-visible content for vision-capable models.
6 +- This module routes images either into Main model-visible content or through the preset's optional Vision Model.
7 - Keep this file-level DOX profile synchronized with `vision_load.py` because this directory is intentionally flat.
8
9 ## Ownership
@@ -12,13 +12,19 @@
12 - `vision_load.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation.
13 - Classes:
14 - `VisionLoad` (`Tool`)
15 - - `async execute(self, paths: list[str]=..., **kwargs) -> Response`
15 + - `async execute(self, paths, query="", raw=False, **kwargs) -> Response`
16 - `async after_execution(self, response: Response, **kwargs)`
17 - Notable constants/configuration names: `TOKENS_ESTIMATE`.
18
19 ## Runtime Contracts
20
21 - Tool modules must define `helpers.tool.Tool` subclasses and return `helpers.tool.Response` from `execute(...)`.
22 +- One call may contain multiple paths. The delegated route sends every selected path in one Vision Model request and returns one textual capsule.
23 +- Main native vision wins unless the effective preset selects the sidecar route; `raw=true` returns to Main native vision only when Main supports it.
24 +- Delegation completes during `execute(...)` so native Responses function output contains the real capsule before `after_execution(...)` persists it.
25 +- Delegated history contains the text capsule only. Native history contains the tool result followed by one raw message holding all loaded image blocks.
26 +- Direct parallel workers resolve ephemeral refs, model routing, and durable chat-media storage against their recorded parent context; independent vision jobs remain generic parallel jobs.
27 +- `max_embeds` comes from the model that actually receives the images.
28 - Update this file whenever tool arguments, output shape, `break_loop` behavior, intervention handling, prompt instructions, or side effects change.
29 - `VisionLoad` is a `Tool`.
30 - `VisionLoad` defines `execute(...)`.
@@ -27,7 +33,7 @@
33
34 ## Key Concepts
35
30 -- Important called helpers/classes observed in the source: `self._get_max_embeds`, `Response`, `str.strip`, `self._context_id`, `chat_media.infer_source`, `chat_media.category_for_source`, `chat_media.save_image_base64`, `chat_media.save_image_data_url`, `chat_media.materialize_image_ref`, `str.strip.lower`, `ephemeral_images.is_ref`, `cls._is_data_image_url`, `self._is_data_image_url`, `plugins.get_plugin_config`, `images.to_data_url`, `normalized.startswith`, `ephemeral_images.display_ref`, `join`, `self.agent.hist_add_tool_result`, `history.RawMessage`.
36 +- Important called helpers/classes observed in the source: `build_vision_model`, `use_vision_sidecar`, `self._get_max_embeds`, `Response`, `self._context_id`, `chat_media.save_image_base64`, `chat_media.save_image_data_url`, `chat_media.materialize_image_ref`, `ephemeral_images.consume_image`, `images.to_data_url`, `self.agent.hist_add_tool_result`, `history.RawMessage`.
37 - Keep request/response, tool, or helper semantics documented here at the same time as source changes.
38
39 ## Work Guidance