@devries-cameron20 / composio Mirror · read-only / commits / 72fbd0055
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 09:45 UTC

fix(anthropic): cap prompt-cache breakpoints at one per request (#4724)

With `cacheTools: true`, the provider put a `cache_control` breakpoint on every wrapped tool and on every `tool_result`. Anthropic allows at most 4 breakpoints per request, shared across the system prompt, tools, and messages. A caller with 5+ tools had the whole request rejected, and a multi-turn loop that appends each turn's tool results would reach the limit again after a few turns. A breakpoint caches everything up to and including it, so one breakpoint on the last tool covers the whole tool list. Fixes #4723 ## Changes - `wrapTools` keeps `cache_control` only on the last tool when `cacheTools` is enabled. The key is deleted from the others, not set to `undefined`, so it is absent from the tool objects. - `handleToolCalls` no longer marks `tool_result` blocks. They are appended to the message history every turn, so a breakpoint on them would add one more to each later request. - `wrapTool` (singular) is unchanged: it still attaches `cache_control` to the single tool it wraps. The fix is scoped to the batch entry point used to build a request's `tools` array. - The README, the constructor JSDoc, and the changeset describe this, including that the cap is per request: lists merged from several `wrapTools` calls each keep a breakpoint on their last tool. - Tests in `test/anthropic.test.ts`: cache only on the last tool of a 3-tool batch (checked with `not.toHaveProperty`), a single-tool batch is still cached, no breakpoint when caching is disabled, and no `tool_result` carries a breakpoint with caching enabled. ## Overlap with #4746 #4746 also fixes #4723 and gives the same result on the tool list and on tool results. This PR additionally removes the key instead of setting it to `undefined`, and updates the README and changeset. ## Type of change - [x] Bug fix - [ ] New feature - [ ] Refactor/Chore - [ ] Documentation - [ ] Breaking change ## How Has This Been Tested? `pnpm vitest run test/anthropic.test.ts`: 22 passed. The new tool_result test fails on the previous code. Run on Node 22.22, the repo declares >=24.17 and prints a warning. `pnpm exec tsc --noEmit` and `pnpm exec oxlint ts/packages/providers/anthropic`: clean. `pnpm exec prettier --check`: clean. ## Checklist - [x] I have read the Code of Conduct and this PR adheres to it - [x] I have read and understand every line of this diff, including any LLM-generated code - [x] Any third-party docs links or promotional content follow the third-party links policy — n/a - [x] I ran linters/tests locally and they passed - [x] I updated documentation as needed (README and JSDoc for `cacheTools`) - [x] I added tests or explain why not applicable - [x] I added a changeset if this change affects published packages --------- Co-authored-by: jkomyno <alberto@composio.dev>

Avanish Garg committed Oct 12, 2026 at 16:20 UTC 72fbd00551093890acec1ff5aaacbccfd7540940
5 files changed +98 -6
.changeset/fix-anthropic-cache-breakpoint-overflow.md new
+5
@@ -0,0 +1,5 @@
1 +---
2 +'@composio/anthropic': patch
3 +---
4 +
5 +Fix `cacheTools: true` exceeding Anthropic's 4-breakpoint-per-request limit (shared with the system prompt and messages), which causes the whole request to be rejected. `wrapTools` now places a single breakpoint on the last tool, which caches the entire tool list since a breakpoint covers everything up to and including it, instead of one on every tool. `handleToolCalls` no longer marks `tool_result` blocks, because they accumulate in the message history and would exceed the limit after a few turns.
docs/content/docs/providers/anthropic.mdx
+1 -1
@@ -276,7 +276,7 @@ The provider registers each tool's complete object schema. Object arguments that
276
277 A few things are specific to the Anthropic provider:
278
279 -- **Tool caching.** Pass `cacheTools: true` to the constructor (`new AnthropicProvider({ cacheTools: true })`) to attach Anthropic's ephemeral `cache_control` to every tool definition and tool-result block. This lets Claude reuse cached tool schemas across requests and can cut prompt cost when you send the same large tool set on every turn.
279 +- **Tool caching.** Pass `cacheTools: true` to the constructor (`new AnthropicProvider({ cacheTools: true })`) to attach Anthropic's ephemeral `cache_control` to the last tool definition, which caches the whole tool list. This lets Claude reuse cached tool schemas across requests and can cut prompt cost when you send the same large tool set on every turn. Anthropic allows at most 4 `cache_control` breakpoints per request, so tool-result blocks are not marked, and lists merged from several `wrapTools` calls each keep a breakpoint on their last tool.
280 - **`handleToolCalls` returns Messages API content, not raw strings.** TypeScript returns a ready-to-append `user` message of `tool_result` blocks. Python returns raw results in model-call order, which the sample wraps into `tool_result` blocks.
281 - **String-encoded tool inputs are handled for you.** Claude occasionally emits a tool's `input` as a JSON string instead of an object. The provider normalizes it before both direct and session execution.
282 - **`cacheTools` only.** The constructor takes no other options. There is no agentic execution in this provider; you run the loop (Messages API) or hand the loop to the Claude Agent SDK (Agent SDK tab).
ts/packages/providers/anthropic/README.md
+3 -1
@@ -68,7 +68,9 @@ Building on the Claude Agent SDK instead? Use [`@composio/claude-agent-sdk`](../
68
69 ## Provider options
70
71 -- `cacheTools`: pass `new AnthropicProvider({ cacheTools: true })` to attach Anthropic's ephemeral `cache_control` to every tool definition and tool-result block. This lets Claude reuse cached tool schemas across requests and can cut prompt cost when you send the same large tool set on every turn. It is the only constructor option.
71 +- `cacheTools`: pass `new AnthropicProvider({ cacheTools: true })` to attach Anthropic's ephemeral `cache_control` to your tool definitions, letting Claude reuse the cached tool schemas across requests. It is the only constructor option.
72 + - Anthropic allows at most 4 `cache_control` breakpoints per request, shared across the system prompt, tools, and messages. A breakpoint caches everything up to and including it, so `wrapTools` places one breakpoint on the last tool instead of one per tool. Tool-result blocks from `handleToolCalls` are never marked, because you append them to the message list every turn and their breakpoints would pile up.
73 + - The cap applies to the whole request, not to a single call. If you merge the results of several `wrapTools` calls (for example from several `tools.get()` calls), each list's last tool carries a breakpoint, so keep the total within the limit.
74 - `handleToolCalls(session, message)` returns `Anthropic.Messages.MessageParam[]`, not raw strings, so you append the result directly to your message list. Pass a user ID instead of a session for tools fetched with `tools.get()`. For finer control, `executeToolCall(session, toolUseBlock)` runs a single `tool_use` block and returns the result as a JSON string.
75 - Claude occasionally emits a tool's `input` as a JSON string instead of an object; the provider normalizes this before execution.
76
ts/packages/providers/anthropic/src/index.ts
+22 -4
@@ -98,7 +98,12 @@ export class AnthropicProvider extends BaseNonAgenticProvider<
98 * Creates a new instance of the AnthropicProvider.
99 *
100 * @param {Object} [options] - Configuration options for the provider
101 - * @param {boolean} [options.cacheTools=false] - Whether to cache tools using Anthropic's ephemeral cache
101 + * @param {boolean} [options.cacheTools=false] - Whether to cache tools using Anthropic's ephemeral
102 + * cache. Anthropic allows at most 4 cache_control breakpoints per request, shared across the
103 + * system prompt, tools, and messages, and a breakpoint caches everything up to and including
104 + * it — so {@link wrapTools} places a single breakpoint on the last tool, covering the whole
105 + * tool list with one breakpoint. Tool results are never marked, since they accumulate in the
106 + * message history across turns.
107 *
108 * @example
109 * ```typescript
@@ -194,7 +199,7 @@ export class AnthropicProvider extends BaseNonAgenticProvider<
199 name: tool.slug,
200 description: tool.description || '',
201 input_schema: schema,
197 - cache_control: this.cacheTools ? { type: 'ephemeral' } : undefined,
202 + ...(this.cacheTools ? { cache_control: { type: 'ephemeral' } } : {}),
203 };
204 }
205
@@ -237,7 +242,18 @@ export class AnthropicProvider extends BaseNonAgenticProvider<
242 * ```
243 */
244 override wrapTools(tools: ComposioTool[]): AnthropicToolCollection {
240 - return tools.map(tool => this.wrapTool(tool));
245 + const wrapped = tools.map(tool => this.wrapTool(tool));
246 + // A cache_control breakpoint caches every block up to and including it,
247 + // so one breakpoint on the last tool covers the entire tool list.
248 + // Anthropic caps breakpoints at 4 per request (shared with the system
249 + // prompt and messages); a breakpoint on every tool would exceed that
250 + // limit as soon as a caller passes 5+ tools.
251 + if (this.cacheTools) {
252 + for (const tool of wrapped.slice(0, -1)) {
253 + delete tool.cache_control;
254 + }
255 + }
256 + return wrapped;
257 }
258
259 /**
@@ -402,10 +418,12 @@ export class AnthropicProvider extends BaseNonAgenticProvider<
418 type: 'tool_result',
419 tool_use_id: toolUse.id,
420 content: toolResult,
405 - cache_control: this.cacheTools ? { type: 'ephemeral' } : undefined,
421 });
422 }
423
424 + // Tool results are not marked as cache breakpoints. The caller appends
425 + // every turn's results to the message history, so a breakpoint here would
426 + // add one more to each later request until the 4-breakpoint limit is hit.
427 return outputs.length > 0 ? [{ role: 'user', content: outputs }] : [];
428 }
429
ts/packages/providers/anthropic/test/anthropic.test.ts
+67
@@ -176,6 +176,42 @@ describe('AnthropicProvider', () => {
176 const wrapped = provider.wrapTools([]);
177 expect(wrapped).toEqual([]);
178 });
179 +
180 + it('should place a single cache_control breakpoint on the last tool when caching is enabled', () => {
181 + const cachingProvider = new AnthropicProvider({ cacheTools: true });
182 + const tools: Tool[] = [
183 + { ...mockTool, slug: 'tool-a' },
184 + { ...mockTool, slug: 'tool-b' },
185 + { ...mockTool, slug: 'tool-c' },
186 + ];
187 +
188 + const wrapped = cachingProvider.wrapTools(tools);
189 +
190 + expect(wrapped[0]).not.toHaveProperty('cache_control');
191 + expect(wrapped[1]).not.toHaveProperty('cache_control');
192 + expect(wrapped[2].cache_control).toEqual({ type: 'ephemeral' });
193 + });
194 +
195 + it('should still cache a single-tool batch', () => {
196 + const cachingProvider = new AnthropicProvider({ cacheTools: true });
197 +
198 + const wrapped = cachingProvider.wrapTools([mockTool]);
199 +
200 + expect(wrapped[0].cache_control).toEqual({ type: 'ephemeral' });
201 + });
202 +
203 + it('should omit cache_control from every tool when caching is disabled', () => {
204 + const tools: Tool[] = [
205 + { ...mockTool, slug: 'tool-a' },
206 + { ...mockTool, slug: 'tool-b' },
207 + ];
208 +
209 + const wrapped = provider.wrapTools(tools);
210 +
211 + for (const tool of wrapped) {
212 + expect(tool).not.toHaveProperty('cache_control');
213 + }
214 + });
215 });
216
217 describe('executeToolCall', () => {
@@ -422,6 +458,37 @@ describe('AnthropicProvider', () => {
458 ]);
459 });
460
461 + it('should not place cache_control on any tool_result when caching is enabled', async () => {
462 + const cachingProvider = new AnthropicProvider({ cacheTools: true });
463 + cachingProvider._setExecuteToolFn(mockExecuteToolFn);
464 + const userId = 'test-user';
465 + const message = {
466 + id: 'msg_123',
467 + content: [
468 + { type: 'text', text: 'Hello' },
469 + { type: 'tool_use', id: 'tu_123', name: 'test-tool-1', input: {} },
470 + { type: 'tool_use', id: 'tu_456', name: 'test-tool-2', input: {} },
471 + { type: 'tool_use', id: 'tu_789', name: 'test-tool-3', input: {} },
472 + ],
473 + } as Anthropic.Message;
474 +
475 + const executeToolCallSpy = vi.spyOn(cachingProvider, 'executeToolCall');
476 + executeToolCallSpy
477 + .mockResolvedValueOnce(JSON.stringify({ result: 'success-1' }))
478 + .mockResolvedValueOnce(JSON.stringify({ result: 'success-2' }))
479 + .mockResolvedValueOnce(JSON.stringify({ result: 'success-3' }));
480 +
481 + const results = await cachingProvider.handleToolCalls(userId, message);
482 +
483 + // Results are appended to the message history every turn, so a breakpoint on them would
484 + // pile up across turns and exceed Anthropic's 4-breakpoint limit.
485 + const outputs = results[0].content as Anthropic.Messages.ToolResultBlockParam[];
486 + expect(outputs).toHaveLength(3);
487 + for (const output of outputs) {
488 + expect(output).not.toHaveProperty('cache_control');
489 + }
490 + });
491 +
492 it('should handle messages without tool calls', async () => {
493 const userId = 'test-user';
494 const message = {