Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
fix(cli): stop tiktoken special-token literals from failing execute
`Tiktoken.encode` signs as `encode(text, allowedSpecial = [], disallowedSpecial = "all")`. The CLI passed only the text, so every special token was disallowed and the call threw on any response containing the literal `<|endoftext|>` or `<|endofprompt|>`. A 66-byte payload is enough. Reading a README that documents a tokenizer hits it. The throw landed in `prepareExecuteOutput`, after `spinner.stop('Execution successful')` had already printed. So the tool had run, its side effects had happened, and the CLI still exited 1 with nothing on stdout and no session history entry. Here the encoder is only a length gauge for the inline-versus-file decision, so those literals are ordinary characters. Passing `allowedSpecial: 'all'` counts them instead of rejecting the payload. Token counts are unchanged on text that contains no special tokens. The preceding commit's byte-length pre-filter hid this below 10KB by skipping the encoder. Larger responses still reached it, and both call sites were affected: the threshold check and the `tokenCount` reported for a stored file. Both now go through one `countOutputTokens` helper. The regression test drives the real command with a response holding both literals, sized past the inline threshold so the count is actually computed. Verified it fails without the fix, with the original error: Error: The text contains a special token that is not allowed: <|endoftext|> Verified with typecheck (src and test), oxlint, validate:boundaries, and the tools.execute and run command suites, now 90 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EzaE7oGVgziJ5nRvBhcci2