| 1 | # vision_load.py DOX |
| 2 | |
| 3 | ## Purpose |
| 4 | |
| 5 | - Own the `vision_load.py` agent tool. |
| 6 | - This module routes images either into Main model-visible content or through the preset's optional Vision Model. |
| 7 | - Keep this file-level DOX profile synchronized with `vision_load.py` because this directory is intentionally flat. |
| 8 | |
| 9 | ## Ownership |
| 10 | |
| 11 | - `vision_load.py` owns the runtime implementation. |
| 12 | - `vision_load.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation. |
| 13 | - Classes: |
| 14 | - `VisionLoad` (`Tool`) |
| 15 | - `async execute(self, paths, query="", **kwargs) -> Response` |
| 16 | - `async after_execution(self, response: Response, **kwargs)` |
| 17 | - Notable constants/configuration names: `TOKENS_ESTIMATE`. |
| 18 | |
| 19 | ## Runtime Contracts |
| 20 | |
| 21 | - Tool modules must define `helpers.tool.Tool` subclasses and return `helpers.tool.Response` from `execute(...)`. |
| 22 | - One call may contain multiple paths; a bare string is treated as one path. The Vision Model route sends every selected path in one request and returns one textual capsule. |
| 23 | - Model configuration exposes a Vision Model only when the effective preset selects that route; otherwise this tool follows Main's native vision path. |
| 24 | - The public tool contract is route-agnostic and accepts an optional focused `query`. A Vision Model receives both that query and the current user request through `fw.vision_load.md`; direct parallel workers inherit the request from their parent. |
| 25 | - Native Main vision already retains the query in its authored tool-call transcript, so native raw history remains image-only and does not repeat model-authored instructions as user content. |
| 26 | - Delegation completes during `execute(...)` so native Responses function output contains the real capsule before `after_execution(...)` persists it. |
| 27 | - Delegated history contains the text capsule only. Native history contains the tool result followed by one raw message holding all loaded image blocks. |
| 28 | - In a direct parallel worker, native image content is queued for the parent and promoted immediately after the outer `parallel` result; the disposable worker never owns the only copy of model-visible pixels. |
| 29 | - Direct parallel workers inherit the parent's model override generically. This tool uses their recorded parent context only to resolve ephemeral refs and durable chat media. |
| 30 | - `max_embeds` comes from the model that actually receives the images. |
| 31 | - Native tool-result summaries omit the skipped-images section when no images were skipped; when the limit drops inputs, the section reports the skipped paths and active maximum. |
| 32 | - Vision Model calls use the selected model's Advanced `kwargs`; this tool does not impose a separate timeout or output-token limit. |
| 33 | - An empty Vision Model response is reported as an image-analysis error instead of a successful empty capsule. |
| 34 | - Update this file whenever tool arguments, output shape, `break_loop` behavior, intervention handling, prompt instructions, or side effects change. |
| 35 | - `VisionLoad` is a `Tool`. |
| 36 | - `VisionLoad` defines `execute(...)`. |
| 37 | - Observed side-effect areas: filesystem writes, model calls, plugin state, settings/state persistence, secret handling. |
| 38 | - Imported dependency areas include: `helpers`, `helpers.tool`, `langchain_core.messages`, `mimetypes`, and `_model_config`. |
| 39 | |
| 40 | ## Key Concepts |
| 41 | |
| 42 | - Important called helpers/classes observed in the source: `build_vision_model`, `get_vision_model_config`, `self._get_max_embeds`, `Response`, `self._context_id`, `chat_media.save_image_base64`, `chat_media.save_image_data_url`, `chat_media.materialize_image_ref`, `ephemeral_images.consume_image`, `images.to_data_url`, `history.RawMessage`, `super().after_execution`, `model.unified_call`. |
| 43 | - Keep request/response, tool, or helper semantics documented here at the same time as source changes. |
| 44 | |
| 45 | ## Work Guidance |
| 46 | |
| 47 | - Keep tool output concise, model-readable, and safe for history persistence. |
| 48 | - Coordinate argument or behavior changes with prompt tool instructions and skill guidance. |
| 49 | - Respect intervention flow for long-running, external, or user-visible operations. |
| 50 | |
| 51 | ## Verification |
| 52 | |
| 53 | - Run targeted tool and prompt-contract tests for changed behavior; smoke-test agent execution when no focused test exists. |
| 54 | - Related tests observed by source search: |
| 55 | - `tests/test_browser_agent_regressions.py` |
| 56 | - `tests/test_host_browser_connector.py` |
| 57 | - `tests/test_office_desktop_state.py` |
| 58 | - `tests/test_vision_load_image_refs.py` |
| 59 | |
| 60 | ## Child DOX Index |
| 61 | |
| 62 | No child DOX files. |