@devries-cameron20 / composio-1 / commits / 611a18726
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

fix(cli): stop tiktoken special-token literals from failing execute (#4464)

## Summary `composio execute` exits 1 whenever a tool response contains the literal text `<|endoftext|>` or `<|endofprompt|>`. A README about tokenizers is enough. A 66-byte payload reproduces it. The tool has already run by then. Side effects happen, stdout stays empty, no session history entry is written. Someone who sent an email this way sees a failure and has no response to inspect. Cause: js-tiktoken's `encode(text, allowedSpecial = [], disallowedSpecial = "all")`. The CLI passed only the text, so every special token was disallowed and `encode` threw. Stacked on #4463. GitHub retargets this to `next` once that merges. ## Changes `countOutputTokens`, the only `encode` call, now passes `'all'`: ```ts const countOutputTokens = (json: string): number => getExecuteOutputEncoder().encode(json, 'all').length; ``` The encoder is a length gauge for the inline-vs-file decision, so allowing the literals is the right reading. Each one counts as the single special token it encodes to, not as the seven tokens its characters would be. Counts for text without the literals do not change. The byte pre-filter in #4463 hides this below 10KB by skipping the encoder. That is cover, not a fix, which is why it stayed out of that PR. ## Type of change - [x] Bug fix - [ ] New feature - [ ] Refactor/Chore - [ ] Documentation - [ ] Breaking change ## How Has This Been Tested? Bun 1.4.1+4661e494f, Node 24.17.0, pnpm 11.8.0, linux-x64. 1. New test drives `composio execute` through the `cli([...])` harness with a response holding both literals, sized past the inline threshold so the count is computed. It reads the stored file back and asserts both literals survived. 2. Against the unfixed code it fails with `Error: The text contains a special token that is not allowed: <|endoftext|>`. With the fix it passes. Suite count goes 90 to 91. 3. `pnpm run typecheck && pnpm run validate:boundaries && pnpm exec vitest run test/src/commands/tools/tools.execute.cmd.test.ts test/src/commands/run.cmd.test.ts` To see it by hand: point any tool at content holding one of the literals and make the response larger than 10KB. ## Screenshots (if applicable) Not applicable. ## Checklist - [x] I have read the Code of Conduct and this PR adheres to it - [x] I ran linters/tests locally and they passed - [ ] I updated documentation as needed - [x] I added tests or explain why not applicable - [ ] I added a changeset if this change affects published packages No docs cover the threshold. `@composio/cli` is private, so no changeset. ## Additional context o200k defines exactly two special tokens. `encode` builds a regex from the disallowed set and throws on the first match, before tokenizing. That guard exists to stop callers from smuggling control tokens into a model prompt. This code is measuring a string. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01EzaE7oGVgziJ5nRvBhcci2

Daksh committed Sep 14, 2026 at 13:52 UTC 611a18726f1361091e5c98fce4ec0c4234998028

No changes.