@samitouri / QOSami_Agents_Zero / commits / 540ef8d8

Restore route-agnostic vision tool contract

Reuse the stock public vision_load prompt and remove the sidecar-specific schema and query argument. Keep delegated analysis instructions in a private framework prompt, preserve the parent request in parallel workers, and retain one-call multi-image batching.

Alessandro committed Aug 25, 2026 at 21:45 UTC 540ef8d8154315789ecf02921540ec5b321ab422

10 files changed +66 -61

helpers/parallel_tools.py

@@ -485,6 +485,7 @@ async def _run_direct_tool_job(parent_context_id: str, job: ParallelJob) -> str:

485 _copy_project(parent_context, worker_context)

486

487 worker_agent = worker_context.agent0

488 + worker_agent.last_user_message = parent_context.agent0.last_user_message

489 worker_agent.loop_data = LoopData()

490 return await execute_tool_call(

491 worker_agent,

helpers/parallel_tools.py.dox.md

+1 -1

@@ -32,7 +32,7 @@

32 - Subordinate child chats are tagged with job metadata, remain outside the scheduler task list, and may use normal child-chat tools including `parallel`.

33 - Nested parallel jobs started by a parallel subordinate are registered as child `DeferredTask` instances so stopping the ancestor also stops its descendants.

34 - Direct tool jobs run in isolated background contexts and are blocked from recursively invoking `parallel`.

35 -- Direct tool jobs inherit the parent's active per-chat model override.

35 +- Direct tool jobs inherit the parent's active per-chat model override and current user message.

36 - Direct tool background context cleanup removes both the in-memory context and any transient chat folder left on disk.

37 - Parent-visible child log items are created for each wrapped call so the WebUI can inspect concurrent children separately while the wrapper result remains model-history-only.

38 - Child tool logs mirror normal tool-call visible args; job ids remain available through wrapper results and prompt extras rather than visible process-step args.

prompts/AGENTS.md

+1 -1

@@ -25,7 +25,7 @@

25 - Read the rendering path before changing placeholders or filenames.

26 - Prefer small prompt additions over broad rewrites when fixing a specific behavior.

27 - Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.

28 -- Keep the single `vision_load` prompt accurate for both native and separate Vision Model routing: related images belong in one call, and a Vision Model result is text-only.

28 +- Keep the single `vision_load` tool prompt route-agnostic and preserve one-call loading for related images; internal image-analysis instructions belong in framework prompts, not Python strings.

29 - Update tests or snapshots when prompt budget, required sections, or generated system content changes.

31 ## Verification

prompts/agent.system.tools_vision.md

+6 -23

@@ -1,40 +1,23 @@

1 ## multimodal vision tools

3 ### vision_load

4 -load or analyze images for visual reasoning

5 -args: `paths` list of absolute image paths or ephemeral image refs, `query` optional focused instruction

6 -Input schema for tool_args:

7 -```json

8 -{

9 - "type": "object",

10 - "properties": {

11 - "paths": {

12 - "type": "array",

13 - "items": {"type": "string"}

14 - },

15 - "query": {"type": "string"}

16 - },

17 - "required": ["paths"],

18 - "additionalProperties": false

19 -}

20 -```

4 +load images into the model for visual reasoning

5 +args: `paths` list of absolute image paths or tool-returned ephemeral image refs

6 rules:

22 -- put all images needed for one comparison or visual task in the same `paths` array

7 +- load all relevant images in one call when comparing screenshots or pages

8 - use when the task depends on screenshots, diagrams, scanned documents, charts, or photos

24 -- use a focused `query` when asking the configured Vision Model to inspect, compare, locate, or read something

9 - only bitmaps are supported; convert other formats first if needed

26 -- the tool result reports loaded and skipped image counts

10 +- the tool result includes loaded/skipped image totals and the corresponding path lists

11 example:

12 ```json

13 {

14 "thoughts": [

15 "I need to inspect the screenshot before answering."

16 ],

33 - "headline": "Comparing screenshots",

17 + "headline": "Loading screenshot for visual analysis",

18 "tool_name": "vision_load",

19 "tool_args": {

36 - "paths": ["/path/to/before.png", "/path/to/after.png"],

37 - "query": "Compare the error banners and describe what changed."

20 + "paths": ["/path/to/screenshot.png"]

21 }

22 }

23 ```

prompts/fw.vision_load.md new

@@ -0,0 +1,4 @@

1 +Analyze the attached image(s) for the current request. Return only relevant visible details, text, and layout.

2 +

3 +Current request:

4 +{{request}}

tests/test_parallel_tool.py

@@ -360,10 +360,13 @@ async def test_direct_parallel_worker_inherits_chat_model_override(monkeypatch)

360 )

361 override = {"preset_name": "Text only"}

362 parent.set_data("chat_model_override", override)

363 + current_user_message = object()

364 + parent.agent0.last_user_message = current_user_message

365 observed = {}

366

367 async def fake_execute_tool_call(agent, *_args, **_kwargs):

368 observed["override"] = agent.context.get_data("chat_model_override")

369 + observed["last_user_message"] = agent.last_user_message

370 return "done"

371

372 async def remove_context(context_id):

@@ -383,6 +386,7 @@ async def test_direct_parallel_worker_inherits_chat_model_override(monkeypatch)

386 try:

387 assert await parallel_tools._run_direct_tool_job(parent_id, job) == "done"

388 assert observed["override"] == override

389 + assert observed["last_user_message"] is current_user_message

390 finally:

391 AgentContext.remove(parent_id)

392

tests/test_tool_policy.py

+9 -8

@@ -580,7 +580,7 @@ async def test_active_vision_model_uses_canonical_vision_prompt(

580 assert schemas[0]["description"] == "canonical vision"

581

582

583 -def test_vision_prompt_declares_multi_image_native_schema() -> None:

583 +def test_vision_prompt_stays_route_agnostic_and_batches_paths() -> None:

584 source = (

585 Path(__file__).resolve().parents[1]

586 / "prompts"

@@ -588,14 +588,15 @@ def test_vision_prompt_declares_multi_image_native_schema() -> None:

588 ).read_text(encoding="utf-8")

589 schema = responses_tools._schema_from_prompt(source)

590

591 - assert schema["required"] == ["paths"]

592 - assert schema["properties"]["paths"] == {

593 - "type": "array",

594 - "items": {"type": "string"},

591 + assert schema == {

592 + "type": "object",

593 + "properties": {},

594 + "additionalProperties": True,

595 }

596 - assert "query" in schema["properties"]

597 - assert "raw" not in schema["properties"]

598 - assert "Vision Model" in source

596 + assert "load all relevant images in one call" in source

597 + assert "Input schema for tool_args" not in source

598 + assert "Vision Model" not in source

599 + assert "query" not in source

600

601

602 def test_mcp_prompt_and_native_schema_omit_blocked_tool(

tests/test_vision_load_image_refs.py

+20 -12

@@ -124,9 +124,9 @@ async def test_vision_load_materializes_local_image_to_chat_artifact(monkeypatch

124 tool.log = SimpleNamespace(id="vision-log", update=lambda **kwargs: updates.append(kwargs))

125

126 invalid = await tool.execute(paths=None)

127 - assert invalid.message == "vision_load error: `paths` must be an array."

127 + assert invalid.message == "vision_load error: `paths` must be a string or an array."

128

129 - response = await tool.execute(paths=[str(image_path)])

129 + response = await tool.execute(paths=str(image_path))

130 image_path.unlink()

131 await tool.after_execution(response)

132

@@ -205,6 +205,10 @@ async def test_vision_model_sends_multiple_images_once_and_keeps_history_text_on

205 agent = SimpleNamespace(

206 context=SimpleNamespace(id=""),

207 agent_name="Agent 0",

208 + last_user_message=SimpleNamespace(

209 + output_text=lambda: "Compare the login errors."

210 + ),

211 + read_prompt=lambda _name, request: f"Analyze: {request}",

212 hist_add_tool_result=lambda *args, **kwargs: tool_results.append((args, kwargs)),

213 hist_add_message=lambda *args, **kwargs: raw_messages.append((args, kwargs)),

214 )

@@ -218,16 +222,16 @@ async def test_vision_model_sends_multiple_images_once_and_keeps_history_text_on

222 )

223 tool.log = SimpleNamespace(id="vision-log", update=lambda **kwargs: None)

224

221 - response = await tool.execute(

222 - paths=[str(path) for path in image_paths],

223 - query="Compare the login errors.",

224 - )

225 + response = await tool.execute(paths=[str(path) for path in image_paths])

226 response.additional = {"_responses_output_item": {"output": response.message}}

227 await tool.after_execution(response)

228

229 assert len(calls) == 1

230 content = calls[0]["messages"][0].content

230 - assert content[0] == {"type": "text", "text": "Compare the login errors."}

231 + assert content[0] == {

232 + "type": "text",

233 + "text": "Analyze: Compare the login errors.",

234 + }

235 assert [item["type"] for item in content].count("image_url") == 2

236 assert "max_tokens" not in calls[0]

237 assert "explicit_caching" not in calls[0]

@@ -326,7 +330,14 @@ async def test_independent_vision_model_calls_can_run_concurrently(monkeypatch,

330 path.write_bytes(b"png-data")

331

332 def make_tool(index):

329 - agent = SimpleNamespace(context=SimpleNamespace(id=""), agent_name=f"Agent {index}")

333 + agent = SimpleNamespace(

334 + context=SimpleNamespace(id=""),

335 + agent_name=f"Agent {index}",

336 + last_user_message=SimpleNamespace(

337 + output_text=lambda: f"inspection {index}"

338 + ),

339 + read_prompt=lambda _name, request: request,

340 + )

341 return vision_load_module.VisionLoad(

342 agent=agent,

343 name="vision_load",

@@ -338,10 +349,7 @@ async def test_independent_vision_model_calls_can_run_concurrently(monkeypatch,

349

350 responses = await asyncio.gather(

351 *(

341 - make_tool(index).execute(

342 - paths=[str(path) for path in image_paths],

343 - query=f"inspection {index}",

344 - )

352 + make_tool(index).execute(paths=[str(path) for path in image_paths])

353 for index in range(4)

354 )

355 )

tools/vision_load.py

+17 -14

@@ -13,23 +13,20 @@ from plugins._model_config.helpers.model_config import (

14 # image token estimation for context window

15 TOKENS_ESTIMATE = 1500

16 -DEFAULT_VISION_QUERY = (

17 - "Describe the images precisely, including the key objects, visible text, and layout."

18 -)

18 class VisionLoad(Tool):

22 - async def execute(

23 - self, paths: list[str] = [], query: str = "", **kwargs

24 - ) -> Response:

19 + async def execute(self, paths: list[str] | str = [], **kwargs) -> Response:

21 self.images_dict = {}

22 self.loaded_paths: list[str] = []

23 self.skipped_paths: list[str] = []

24 self.vision_config = get_vision_model_config(self.agent)

25 + if isinstance(paths, str):

26 + paths = [paths]

27 if not isinstance(paths, list):

28 return Response(

32 - message="vision_load error: `paths` must be an array.",

29 + message="vision_load error: `paths` must be a string or an array.",

30 break_loop=False,

31 )

@@ -83,15 +80,13 @@ class VisionLoad(Tool):

80 message = self._summary() if self.images_dict or self.skipped_paths else "No images processed"

81 if self.vision_config and self.images_dict:

82 try:

86 - capsule = await self._call_vision_model(

87 - list(self.images_dict.values()), query

88 - )

83 + capsule = await self._call_vision_model(list(self.images_dict.values()))

84 message = (

90 - f"Vision Model analyzed {len(self.images_dict)} image(s)"

85 + f"Analyzed {len(self.images_dict)} image(s)"

86 f"; {len(self.skipped_paths)} skipped.\n\n{capsule.strip()}"

87 )

88 except Exception as exc:

94 - message = f"Vision Model error: {str(exc)[:1000]}"

89 + message = f"Image analysis error: {str(exc)[:1000]}"

90 return Response(message=message, break_loop=False)

92 def _get_max_embeds(self) -> int:

@@ -104,8 +99,16 @@ class VisionLoad(Tool):

99 parent_id = get_data(PARALLEL_WORKER_PARENT_CONTEXT_KEY) if get_data else ""

100 return str(parent_id or getattr(context, "id", "") or "").strip()

101

107 - async def _call_vision_model(self, image_paths: list[str], query: str) -> str:

108 - content = [{"type": "text", "text": str(query or "").strip() or DEFAULT_VISION_QUERY}]

102 + async def _call_vision_model(self, image_paths: list[str]) -> str:

103 + user_message = getattr(self.agent, "last_user_message", None)

104 + output_text = getattr(user_message, "output_text", None)

105 + request = str(output_text() if callable(output_text) else "").strip()

106 + content = [

107 + {

108 + "type": "text",

109 + "text": self.agent.read_prompt("fw.vision_load.md", request=request),

110 + }

111 + ]

112 content.extend(

113 {"type": "image_url", "image_url": {"url": path}}

114 for path in image_paths

tools/vision_load.py.dox.md

+3 -2

@@ -12,15 +12,16 @@

12 - `vision_load.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation.

13 - Classes:

14 - `VisionLoad` (`Tool`)

15 - - `async execute(self, paths, query="", **kwargs) -> Response`

15 + - `async execute(self, paths, **kwargs) -> Response`

16 - `async after_execution(self, response: Response, **kwargs)`

17 - Notable constants/configuration names: `TOKENS_ESTIMATE`.

19 ## Runtime Contracts

21 - Tool modules must define `helpers.tool.Tool` subclasses and return `helpers.tool.Response` from `execute(...)`.

22 -- One call may contain multiple paths. The Vision Model route sends every selected path in one request and returns one textual capsule.

22 +- One call may contain multiple paths; a bare string is treated as one path. The Vision Model route sends every selected path in one request and returns one textual capsule.

23 - Model configuration exposes a Vision Model only when the effective preset selects that route; otherwise this tool follows Main's native vision path.

24 +- The public tool contract is route-agnostic. A Vision Model receives the current user request through `fw.vision_load.md`; direct parallel workers inherit that request from their parent.

25 - Delegation completes during `execute(...)` so native Responses function output contains the real capsule before `after_execution(...)` persists it.

26 - Delegated history contains the text capsule only. Native history contains the tool result followed by one raw message holding all loaded image blocks.

27 - Direct parallel workers inherit the parent's model override generically. This tool uses their recorded parent context only to resolve ephemeral refs and durable chat media.