| 1 | # Document Query Plugin |
| 2 | |
| 3 | Load, parse, index, and Q&A over local and remote documents with configurable |
| 4 | timeouts and thread-safe parsers. |
| 5 | |
| 6 | ## Features |
| 7 | |
| 8 | - **Strategy-pattern parsers** - MIME-type routing to dedicated parser classes |
| 9 | - **Centralized fetching** - local and HTTP(S) resources are fetched once, size-checked, then passed to parsers |
| 10 | - **LiteParse first path** - fast local parsing for PDFs and supported document/image formats, with legacy fallbacks |
| 11 | - **Adaptive OCR** - long PDFs skip OCR automatically to avoid pathological parse times |
| 12 | - **Adaptive indexing** - very large extracted documents increase chunk size to keep embedding work bounded |
| 13 | - **Bounded parser execution** - sync parsers are offloaded to asyncio.to_thread and globally capped across chats |
| 14 | - **Configurable timeouts** - per-document and gather-level timeouts |
| 15 | - **Expanded format support** - PDF, HTML, text, YAML, XML, TOML, JS, TS, images, and catch-all Unstructured |
| 16 | |
| 17 | ## Configuration |
| 18 | |
| 19 | See default_config.yaml for all options. Key settings: |
| 20 | |
| 21 | | Setting | Default | Description | |
| 22 | |---------|---------|-------------| |
| 23 | | fetch_timeout | 30 | HTTP fetch timeout (seconds) | |
| 24 | | fetch_retries | 3 | HTTP retry attempts | |
| 25 | | max_remote_bytes | 52428800 | Max remote document size | |
| 26 | | per_document_timeout | 60 | Max time for a single document parse | |
| 27 | | gather_timeout | 120 | Max time for all documents combined | |
| 28 | | parser_concurrency | 1 | Max parser jobs running across all chats in one process | |
| 29 | | context_intro_chunks | 2 | Leading chunks included per document for title/abstract grounding | |
| 30 | | chunk_size | 1000 | Text splitter chunk size | |
| 31 | | chunk_overlap | 100 | Text splitter overlap | |
| 32 | | max_index_chunks | 1200 | Maximum indexed chunks before adaptive chunk sizing, or 0 for no cap | |
| 33 | | search_threshold | 0.5 | Similarity search threshold | |
| 34 | | liteparse_enabled | true | Prefer LiteParse before legacy parser fallbacks | |
| 35 | | liteparse_num_workers | 2 | Max LiteParse OCR workers per parser job | |
| 36 | | liteparse_ocr_auto_disable_pages | 30 | Disable OCR for PDFs at or above this effective page count | |
| 37 | | thread_offload | true | Offload sync parsers to thread pool | |
| 38 | |
| 39 | LiteParse is installed into the Agent Zero framework runtime from hooks.py during |
| 40 | plugin install/startup. If installation fails, the plugin logs the error and |
| 41 | continues with the legacy parser fallbacks. |
| 42 | |
| 43 | LiteParse always runs in a child process so native parser and OCR failures stay |
| 44 | isolated from the Web UI process. |
| 45 | |
| 46 | ## Parsers |
| 47 | |
| 48 | | Parser | MIME Types | Backend | |
| 49 | |--------|-----------|---------| |
| 50 | | LiteParseParser | PDF, Office/OpenDocument, images | LiteParse | |
| 51 | | PdfParser | application/pdf | PyMuPDF + Tesseract OCR fallback | |
| 52 | | HtmlParser | text/html | Markdownify transformer | |
| 53 | | TextParser | text/*, application/json, YAML, XML, TOML, JS, TS, shell | Direct read | |
| 54 | | ImageParser | image/* | UnstructuredLoader | |
| 55 | | UnstructuredParser | * (catch-all) | UnstructuredLoader hi-res | |
| 56 | |
| 57 | ## Adding a new parser |
| 58 | |
| 59 | 1. Create helpers/parsers/<format>.py extending BaseParser |
| 60 | 2. Set mimetypes class attribute |
| 61 | 3. Implement _parse_sync(document, config) |
| 62 | 4. Register in helpers/parsers/__init__.py |