Refine document query image routing
Prefer vision_load for images, screenshots, scans, charts, photos, and diagrams when vision tools are available. Keep document_query focused on PDFs, documents, large text-heavy files, code-file Q&A, and fallback OCR when vision cannot read the needed text. Update bundled prompt, skill guidance, DOX notes, and regression coverage for the routing contract.
Alessandro committed
Jun 19, 2026 at 01:49 UTC
77552b939451b710be803752cfb0b08fa038e5a0
6 files changed
+27
-15
plugins/_document_query/AGENTS.md
+1
@@ -20,6 +20,7 @@
20
## Work Guidance
21
22
- Coordinate parser changes with timeout handling and fallback behavior.
23
+- Keep prompt and skill guidance clear that image files, screenshots, scans, charts, photos, and diagrams should go to vision tools first when available; `document_query` image OCR is a text-focused fallback when vision is unavailable or cannot read the needed text.
24
25
## Verification
26
plugins/_document_query/prompts/agent.system.tool.document_query.md
+4
-2
@@ -1,7 +1,9 @@
1
### document_query
2
-read, extract, summarize, compare, OCR, or answer questions over local/remote documents, code files, and text-heavy document images/scans.
2
+read, extract, summarize, compare, or answer questions over local/remote documents, PDFs, Office files, HTML/text/code files, large text-heavy files, and fallback OCR when vision tools cannot read a document image/scan.
3
4
-For document Q&A, document/code analysis, multi-document comparison, or OCR/text extraction from images/scans, first load the `document-query` skill with `skills_tool:load`, then call this tool using the loaded instructions.
4
+For document Q&A, document/code analysis, multi-document comparison, PDF or large-file extraction, or fallback OCR after vision tools are unavailable or insufficient, first load the `document-query` skill with `skills_tool:load`, then call this tool using the loaded instructions.
5
+
6
+Use vision tools first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when available. Call `document_query` for images only as document-style OCR fallback when vision cannot read the needed text.
7
8
Minimal args after loading the skill:
9
- `document`: one local path/URL or a list of paths/URLs
plugins/_document_query/skills/document-query/SKILL.md
+11
-9
@@ -1,6 +1,6 @@
1
---
2
name: document-query
3
-description: Use when reading, extracting, summarizing, comparing, OCRing, or answering questions over local or remote documents, code files, PDFs, Office files, HTML/text files, and text-heavy document images or scans with the document_query tool.
3
+description: Use when reading, extracting, summarizing, comparing, or answering questions over local or remote documents, code files, PDFs, Office files, HTML/text files, large text-heavy files, and fallback OCR for document images or scans when vision tools are unavailable or insufficient.
4
version: 1.0.0
5
author: Agent Zero Team
6
tags: ["documents", "ocr", "qa", "pdf", "code", "analysis"]
@@ -10,8 +10,8 @@ trigger_patterns:
10
- ask questions about a document
11
- summarize document
12
- compare documents
13
- - extract text from image
14
- - OCR document
13
+ - fallback image OCR
14
+ - OCR document fallback
15
- analyze code file
16
---
17
@@ -24,13 +24,15 @@ Use the `document_query` tool to read, extract, summarize, compare, or answer qu
24
Use `document_query` for:
25
26
- Local files and URLs containing document text: PDF, HTML, Office files, plain text, Markdown, CSV/TSV, XML/JSON, logs, code files, and similar content.
27
+- Large text-heavy files where ordinary reading/search would be too broad and the user needs Q&A, summarization, comparison, or extraction.
28
- Q&A over one or more documents.
29
- Summaries, comparisons, entity extraction, key-point extraction, and table/text extraction.
30
- Code-file Q&A when the user points to specific files or URLs and wants answers from their contents.
30
-- Text-heavy images, scans, screenshots, and document images when the task is OCR/text extraction or Q&A over visible text.
31
-- Image OCR when vision tools are unavailable, when the main chat model is not multimodal, or when the user wants document text rather than visual scene understanding.
31
+- Fallback OCR over text-heavy document images, scans, or screenshots only when vision tools are unavailable, insufficient, or unable to read the needed text.
32
33
-Do not use `document_query` for purely visual questions that require spatial/visual reasoning beyond document text; use vision tools when available for those cases.
33
+Use vision tools first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when available. Do not route images to `document_query` as a first-pass reader.
34
+
35
+Do not use `document_query` for purely visual questions, first-pass screenshot inspection, or tasks that require spatial/visual reasoning beyond document text; use vision tools when available for those cases.
36
37
## Inputs
38
@@ -111,14 +113,14 @@ For directories or codebases, first identify the relevant files with file/search
113
}
114
```
115
114
-### OCR Or Q&A Over A Document Image
116
+### Fallback OCR After Vision Cannot Read A Document Image
117
118
```json
119
{
120
"thoughts": [
119
- "The user wants text from a scanned document image."
121
+ "Vision tools are unavailable or could not read the scanned document text, so document_query is the fallback OCR path."
122
],
121
- "headline": "Reading text from the scanned document",
123
+ "headline": "Reading document text with fallback OCR",
124
"tool_name": "document_query",
125
"tool_args": {
126
"document": "/a0/usr/workdir/scan.png",
prompts/AGENTS.md
+1
@@ -23,6 +23,7 @@
23
24
- Read the rendering path before changing placeholders or filenames.
25
- Prefer small prompt additions over broad rewrites when fixing a specific behavior.
26
+- Keep document/OCR routing explicit: image files, screenshots, scans, charts, photos, and diagrams should prefer vision tools when available, while `document_query` is for documents, large text-heavy files, and fallback OCR.
27
- Update tests or snapshots when prompt budget, required sections, or generated system content changes.
28
29
## Verification
prompts/agent.system.main.tips.md
+3
-3
@@ -24,8 +24,8 @@ always use specialized subordinate agents for specialized tasks matching their p
24
25
## Documents and OCR
26
27
-use document_query to read, extract, summarize, compare, or answer questions about documents from local paths or URLs
27
+use document_query to read, extract, summarize, compare, or answer questions about documents from local paths or URLs, especially PDFs, Office files, HTML/text files, logs, code files, and large files that need Q&A
28
use document_query for Q&A, summaries, comparisons, or extraction over specific code files when the user asks about file contents rather than asking to edit or search the codebase
29
-use document_query for document images, screenshots, scans, and other image files when the task is text extraction/OCR or Q&A over document content
30
-when vision tools are unavailable or the main chat model is not multimodal, use document_query for image OCR instead of asking the user to switch models
29
+use vision_load first for image files, screenshots, scans, charts, photos, diagrams, and other visual inputs when vision tools are available
30
+use document_query for image OCR only when vision tools cannot read the image, vision tools are unavailable, or the user specifically needs document-style fallback OCR over visible text
31
keep parser/runtime details internal; users only need the document answer
tests/test_document_query_plugin.py
+7
-1
@@ -559,9 +559,15 @@ def test_document_query_prompt_uses_progressive_skill_disclosure():
559
assert skill is not None
560
assert "document_query for Q&A" in main_prompt
561
assert "specific code files" in main_prompt
562
+ assert "use vision_load first for image files" in main_prompt
563
+ assert "document_query for image OCR only when vision tools cannot read" in main_prompt
564
assert "skills_tool:load" in prompt
565
assert "document-query" in prompt
566
assert "document_query" in prompt
567
+ assert "Use vision tools first" in prompt
568
+ assert "fallback OCR" in prompt
569
assert "answering questions over local or remote documents" in skill.description
570
+ assert "fallback OCR" in skill.description
571
assert "### Answer Questions Over A Document" in skill.content
567
- assert "### OCR Or Q&A Over A Document Image" in skill.content
572
+ assert "Use vision tools first" in skill.content
573
+ assert "### Fallback OCR After Vision Cannot Read A Document Image" in skill.content