| 1 | # tokens.py DOX |
| 2 | |
| 3 | ## Purpose |
| 4 | |
| 5 | - Own the `tokens.py` helper module. |
| 6 | - This module counts and approximates tokens and trims text to budgets. |
| 7 | - Keep this file-level DOX profile synchronized with `tokens.py` because this directory is intentionally flat. |
| 8 | |
| 9 | ## Ownership |
| 10 | |
| 11 | - `tokens.py` owns the runtime implementation. |
| 12 | - `tokens.py.dox.md` owns durable notes about responsibilities, contracts, side effects, and verification for that implementation. |
| 13 | - Top-level functions: |
| 14 | - `count_tokens(text: str, encoding_name=...) -> int` |
| 15 | - `approximate_tokens(text: str) -> int` |
| 16 | - `sanitize_embedded_image_data_urls(text: str) -> str` |
| 17 | - `approximate_prompt_tokens(text: str) -> int` |
| 18 | - `trim_to_tokens(text: str, max_tokens: int, direction: Literal['start', 'end'], ellipsis: str=...) -> str` |
| 19 | - Notable constants/configuration names: `APPROX_BUFFER`, `TRIM_BUFFER`, `EMBEDDED_IMAGE_DATA_PLACEHOLDER`, `_EMBEDDED_IMAGE_DATA_URL_PATTERN`. |
| 20 | |
| 21 | ## Runtime Contracts |
| 22 | |
| 23 | - Helper modules own reusable framework APIs and must preserve public callers unless all callers, tests, and docs are updated together. |
| 24 | - Update this file whenever public functions, classes, persistence behavior, path/security assumptions, side effects, or cross-module contracts change. |
| 25 | - Observed side-effect areas: secret handling. |
| 26 | - Imported dependency areas include: `re`, `tiktoken`, `typing`. |
| 27 | |
| 28 | ## Key Concepts |
| 29 | |
| 30 | - Important called helpers/classes observed in the source: `re.compile`, `tiktoken.get_encoding`, `encoding.encode`, `_EMBEDDED_IMAGE_DATA_URL_PATTERN.sub`, `approximate_tokens`, `count_tokens`, `sanitize_embedded_image_data_urls`. |
| 31 | - Keep request/response, tool, or helper semantics documented here at the same time as source changes. |
| 32 | |
| 33 | ## Work Guidance |
| 34 | |
| 35 | - Preserve public helper APIs used by core code and plugins unless every caller is updated. |
| 36 | - Keep path, auth, secret, persistence, network, and subprocess behavior explicit and bounded. |
| 37 | - Prefer adding cohesive helper functions here only when behavior is reused across modules. |
| 38 | |
| 39 | ## Verification |
| 40 | |
| 41 | - Run targeted tests for changed helper behavior; run security regressions for auth, filesystem, WebSocket, tunnel, upload, or secret-handling helpers. |
| 42 | - Related tests observed by source search: |
| 43 | - `tests/test_chat_compaction.py` |
| 44 | - `tests/test_default_prompt_budget.py` |
| 45 | - `tests/test_history_compression_wait.py` |
| 46 | - `tests/test_mcp_handler_multimodal.py` |
| 47 | - `tests/test_oauth_codex.py` |
| 48 | - `tests/test_oauth_gemini_api.py` |
| 49 | - `tests/test_oauth_xai_grok.py` |
| 50 | - `tests/test_ws_security.py` |
| 51 | |
| 52 | ## Child DOX Index |
| 53 | |
| 54 | No child DOX files. |