@devries-cameron20 / composio Mirror · read-only / commits / aac2c2cf7
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 03:37 UTC

fix(ts/e2e): keep Tool Router arguments open for Mastra agents on OpenAI (#5461)

## Changes - Patch `@mastra/schema-compat@1.3.4` (pulled in by the pinned `@mastra/core@1.53.0`) so its OpenAI layer keeps open maps open. Upstream sets `additionalProperties: false` on every object, so the Tool Router `COMPOSIO_MULTI_EXECUTE_TOOL` `tools[].arguments` map (`{type: object, properties: {}, additionalProperties: true}`) reached the model as an object that allows no keys. Models that follow the schema then ran `HACKERNEWS_GET_USER` with `{}` until the step limit. The latest Mastra (core 1.85.0, schema-compat 1.4.0) still does this. - Copy `patches/` into the Node and CLI E2E images before `pnpm install`, and trigger the E2E workflow on `patches/**`. - Move the Mastra Zod 3/4 and Cloudflare agent tests from `gpt-5.1` to `gpt-5.5`. On 2026-11-18, 16 `gpt-5.1` Responses API replies in CI came back HTTP 200 with `status: incomplete`, `reason: max_output_tokens`, empty output, and 0/0 tokens, with no `max_output_tokens` in the request. That fault caused the earlier undefined-object failures. - Require the Mastra tests to get the karma from a real `HACKERNEWS_GET_USER` execution with `username: "pg"`. Before, a karma the model recalled from memory passed. - Log agent diagnostics (tool calls, search and execution arguments, per-step finish reason, usage, response IDs) only on failure. ## Verification - With the patch, an offline Mastra agent run against a stub MCP server sends `arguments` with `additionalProperties: true` for both the live and the documented schema shape, on Zod 3 and Zod 4. Without it, both send `false`. - 4 E2E runs on 1dab9229c (1 PR and 3 dispatched) passed every job, including all Mastra tests on Node 22, 24, and 26 and the Cloudflare agent test. - Before the patch, `gpt-5.5` failed 4 of 18 Mastra tests in 3 runs with `arguments: {}` on every execution. - Typecheck reports only the existing Mastra/OpenAI model type mismatch that the base also has.

Alberto Schiabel committed Nov 19, 2026 at 14:01 UTC aac2c2cf786c50db61671a6734f016663abef6a0
9 files changed +307 -63
.github/workflows/ts.test-e2e.yml
+2
@@ -10,6 +10,7 @@ on:
10 # 0.1.0-alpha.76 -> 2.0.0-rc.5 skipped this suite entirely that way.
11 - 'pnpm-workspace.yaml'
12 - 'pnpm-lock.yaml'
13 + - 'patches/**'
14 - '.github/actions/setup-node-pnpm-bun/action.yml'
15 - '.github/actions/build-e2e-image/action.yml'
16 - '.github/workflows/ts.test-e2e.yml'
@@ -24,6 +25,7 @@ on:
25 # 0.1.0-alpha.76 -> 2.0.0-rc.5 skipped this suite entirely that way.
26 - 'pnpm-workspace.yaml'
27 - 'pnpm-lock.yaml'
28 + - 'patches/**'
29 - '.github/actions/setup-node-pnpm-bun/action.yml'
30 - '.github/actions/build-e2e-image/action.yml'
31 - '.github/workflows/ts.test-e2e.yml'
patches/@mastra__schema-compat@1.3.4.patch new
+28
@@ -0,0 +1,28 @@
1 +diff --git a/dist/index.js b/dist/index.js
2 +index b8c857b..2c2726a 100644
3 +--- a/dist/index.js
4 ++++ b/dist/index.js
5 +@@ -5444,7 +5444,8 @@ var OpenAISchemaCompatLayer = class extends SchemaCompatLayer3 {
6 + ];
7 + }
8 + if (isObjectSchema(schema)) {
9 +- schema.additionalProperties = false;
10 ++ const isOpenMap = Object.keys(schema.properties ?? {}).length === 0 && (schema.additionalProperties === true || (typeof schema.additionalProperties === "object" && schema.additionalProperties !== null));
11 ++ if (!isOpenMap) schema.additionalProperties = false;
12 + if (schema.properties) {
13 + for (const key of Object.keys(schema.properties)) {
14 + const prop = schema.properties[key];
15 +diff --git a/dist/index.cjs b/dist/index.cjs
16 +index c9d2329..9de7be8 100644
17 +--- a/dist/index.cjs
18 ++++ b/dist/index.cjs
19 +@@ -5444,7 +5444,8 @@ var OpenAISchemaCompatLayer = class extends SchemaCompatLayer3 {
20 + ];
21 + }
22 + if (isObjectSchema(schema)) {
23 +- schema.additionalProperties = false;
24 ++ const isOpenMap = Object.keys(schema.properties ?? {}).length === 0 && (schema.additionalProperties === true || (typeof schema.additionalProperties === "object" && schema.additionalProperties !== null));
25 ++ if (!isOpenMap) schema.additionalProperties = false;
26 + if (schema.properties) {
27 + for (const key of Object.keys(schema.properties)) {
28 + const prop = schema.properties[key];
pnpm-lock.yaml
+7 -4
@@ -143,6 +143,9 @@ overrides:
143
144 pnpmfileChecksum: sha256-X6eCJXmBNe7VV40iD0uZlsr4IMg+ZL/Qodvx5bNk4wI=
145
146 +patchedDependencies:
147 + '@mastra/schema-compat@1.3.4': 1d32e0a15ad79a9d35393cd910d52caaf920adb79cf3b1e3cd41d7fbb200f42b
148 +
149 importers:
150
151 .:
@@ -8136,7 +8139,7 @@ snapshots:
8139 '@ai-sdk/ui-utils-v5': '@ai-sdk/ui-utils@1.2.11(zod@3.25.76)'
8140 '@isaacs/ttlcache': 2.1.5
8141 '@lukeed/uuid': 2.0.1
8139 - '@mastra/schema-compat': 1.3.4(zod@3.25.76)
8142 + '@mastra/schema-compat': 1.3.4(patch_hash=1d32e0a15ad79a9d35393cd910d52caaf920adb79cf3b1e3cd41d7fbb200f42b)(zod@3.25.76)
8143 '@modelcontextprotocol/sdk': 1.32.0(@cfworker/json-schema@4.1.6)(zod@3.25.76)
8144 '@sindresorhus/slugify': 2.2.1
8145 '@standard-schema/spec': 1.1.0
@@ -8182,7 +8185,7 @@ snapshots:
8185 '@ai-sdk/ui-utils-v5': '@ai-sdk/ui-utils@1.2.11(zod@4.6.7)'
8186 '@isaacs/ttlcache': 2.1.5
8187 '@lukeed/uuid': 2.0.1
8185 - '@mastra/schema-compat': 1.3.4(zod@4.6.7)
8188 + '@mastra/schema-compat': 1.3.4(patch_hash=1d32e0a15ad79a9d35393cd910d52caaf920adb79cf3b1e3cd41d7fbb200f42b)(zod@4.6.7)
8189 '@modelcontextprotocol/sdk': 1.32.0(@cfworker/json-schema@4.1.6)(zod@4.6.7)
8190 '@sindresorhus/slugify': 2.2.1
8191 '@standard-schema/spec': 1.1.0
@@ -8262,7 +8265,7 @@ snapshots:
8265 zod: 4.6.7
8266 zod-from-json-schema: 0.5.2
8267
8265 - '@mastra/schema-compat@1.3.4(zod@3.25.76)':
8268 + '@mastra/schema-compat@1.3.4(patch_hash=1d32e0a15ad79a9d35393cd910d52caaf920adb79cf3b1e3cd41d7fbb200f42b)(zod@3.25.76)':
8269 dependencies:
8270 json-schema-to-zod: 2.8.1
8271 zod: 3.25.76
@@ -8270,7 +8273,7 @@ snapshots:
8273 zod-from-json-schema-v3: zod-from-json-schema@0.0.5
8274 zod-to-json-schema: 3.25.2(zod@3.25.76)
8275
8273 - '@mastra/schema-compat@1.3.4(zod@4.6.7)':
8276 + '@mastra/schema-compat@1.3.4(patch_hash=1d32e0a15ad79a9d35393cd910d52caaf920adb79cf3b1e3cd41d7fbb200f42b)(zod@4.6.7)':
8277 dependencies:
8278 json-schema-to-zod: 2.8.1
8279 zod: 4.6.7
pnpm-workspace.yaml
+8
@@ -121,6 +121,14 @@ peerDependencyRules:
121 undici: ^7.29.0
122 zod: ^4.0.0
123
124 +patchedDependencies:
125 + # temporary: @mastra/schema-compat 1.3.4 (via the pinned @mastra/core) closes
126 + # every object schema it sends to OpenAI models, open maps included. The Tool
127 + # Router multi-execute `arguments` map then reaches the model as an object
128 + # that allows no keys, and models that follow the schema execute tools with
129 + # `{}`. Drop once @mastra/core can move to a release that keeps open maps.
130 + '@mastra/schema-compat@1.3.4': patches/@mastra__schema-compat@1.3.4.patch
131 +
132 overrides:
133 # temporary: @mastra/mcp 1.17.3 raised its @mastra/core peer floor to 1.64,
134 # above the Workers-compatible ceiling pinned in the catalog above. Drop this
ts/e2e-tests/_utils/Dockerfile.cli
+3
@@ -53,6 +53,9 @@ COPY package.json \
53 toolchain-versions.json \
54 ./
55
56 +# pnpm applies these dependency patches during install
57 +COPY patches/ ./patches/
58 +
59 # Copy all ts/packages (pre-built)
60 COPY ts/packages/ ./ts/packages/
61
ts/e2e-tests/_utils/Dockerfile.node
+3
@@ -60,6 +60,9 @@ COPY package.json \
60 toolchain-versions.json \
61 ./
62
63 +# pnpm applies these dependency patches during install
64 +COPY patches/ ./patches/
65 +
66 # Copy all ts/packages (pre-built)
67 COPY ts/packages/ ./ts/packages/
68
ts/e2e-tests/runtimes/cloudflare/cf-workers-tool-router-ai/src/index.ts
+18 -1
@@ -138,7 +138,7 @@ app.get('/test/agent', async c => {
138 const openai = createOpenAI({ apiKey: c.env.OPENAI_API_KEY });
139
140 const result = await generateText({
141 - model: openai('gpt-5.1'),
141 + model: openai('gpt-5.5'),
142 prompt: `Look up the HackerNews user "pg" with HACKERNEWS_GET_USER, then return the exact karma value from that tool result.`,
143 output: Output.object({
144 schema: z.object({
@@ -149,6 +149,23 @@ app.get('/test/agent', async c => {
149 tools,
150 });
151
152 + if (result.finishReason !== 'stop') {
153 + const responseBodySchema = z.object({ incomplete_details: z.unknown() });
154 + console.error(
155 + 'Agent step diagnostics',
156 + JSON.stringify(
157 + result.steps.map(step => ({
158 + finishReason: step.finishReason,
159 + responseId: step.response.id,
160 + incompleteDetails: responseBodySchema.safeParse(step.response.body).data
161 + ?.incomplete_details,
162 + usage: step.usage,
163 + toolCalls: step.toolCalls.map(({ toolName, input }) => ({ toolName, input })),
164 + }))
165 + )
166 + );
167 + }
168 +
169 const toolCalls = result.steps.flatMap(step =>
170 step.toolCalls.map(toolCall => ({ toolName: toolCall.toolName }))
171 );
ts/e2e-tests/runtimes/node/mastra-tool-router-zod-v3/e2e.test.ts
+119 -29
@@ -8,7 +8,7 @@
8
9 import { Composio } from '@composio/core';
10 import { MastraProvider } from '@composio/mastra';
11 -import { MCPClient } from '@mastra/mcp';
11 +import { getMcpCallToolMeta, MCPClient } from '@mastra/mcp';
12 import { Agent } from '@mastra/core/agent';
13 import { e2e } from '@e2e-tests/utils';
14 import { TIMEOUTS } from '@e2e-tests/utils/const';
@@ -18,6 +18,64 @@ import { stepCountIs } from 'ai';
18 import { z } from 'zod';
19
20 const MCP_CONNECT_TIMEOUT_MS = 15_000;
21 +const KARMA_SCHEMA = z.object({ karma: z.number() });
22 +const SEARCH_TOOL = 'myHttpClient_COMPOSIO_SEARCH_TOOLS';
23 +const EXECUTE_TOOL = 'myHttpClient_COMPOSIO_MULTI_EXECUTE_TOOL';
24 +
25 +// Allowlist for failure logs: Zod strips every other field.
26 +const idsSchema = z.object({ log_id: z.string().optional(), request_id: z.string().optional() });
27 +const searchResponseSchema = idsSchema.extend({
28 + data: z
29 + .object({
30 + results: z.array(z.object({ primary_tool_slugs: z.array(z.string()).optional() })).optional(),
31 + tool_schemas: z
32 + .object({
33 + HACKERNEWS_GET_USER: z
34 + .object({
35 + input_schema: z.object({ required: z.array(z.string()).optional() }).nullish(),
36 + })
37 + .optional(),
38 + })
39 + .optional(),
40 + })
41 + .optional(),
42 +});
43 +const executeArgsSchema = z.object({
44 + tools: z.array(z.object({ tool_slug: z.string(), arguments: z.record(z.string(), z.unknown()) })),
45 +});
46 +const mcpTextResultSchema = z.object({
47 + _meta: idsSchema.optional(),
48 + content: z.array(z.object({ type: z.literal('text'), text: z.string() })),
49 +});
50 +
51 +/** Flat log record for one search result, given as a structured value or an MCP text envelope. */
52 +function searchRecord(payload: { toolCallId: string; isError?: boolean; result: unknown }) {
53 + const envelope = mcpTextResultSchema.safeParse(payload.result).data;
54 + let value: unknown = payload.result;
55 + if (envelope) {
56 + try {
57 + value = JSON.parse(envelope.content.map(part => part.text).join(''));
58 + } catch {
59 + value = undefined;
60 + }
61 + }
62 + const response = searchResponseSchema.safeParse(value).data;
63 + const meta = idsSchema.safeParse(getMcpCallToolMeta(payload.result) ?? envelope?._meta).data;
64 + return {
65 + toolCallId: payload.toolCallId,
66 + isError: payload.isError,
67 + log_id: meta?.log_id ?? response?.log_id,
68 + request_id: meta?.request_id ?? response?.request_id,
69 + requiredInput: response?.data?.tool_schemas?.HACKERNEWS_GET_USER?.input_schema?.required,
70 + matchedSlugs: response?.data?.results?.flatMap(r => r.primary_tool_slugs ?? []),
71 + };
72 +}
73 +
74 +/** Text of an MCP tool result, or its JSON when the result is structured. */
75 +function toolResultText(result: unknown) {
76 + const envelope = mcpTextResultSchema.safeParse(result).data;
77 + return envelope ? envelope.content.map(part => part.text).join('') : JSON.stringify(result);
78 +}
79
80 declare module 'bun' {
81 interface Env {
@@ -70,39 +128,71 @@ e2e(import.meta.url, {
128 },
129 });
130
73 - const tools = await mcpClient.listTools();
74 - const openai = createOpenAI({ apiKey: Bun.env.OPENAI_API_KEY });
131 + try {
132 + const tools = await mcpClient.listTools();
133 + expect(Object.keys(tools).length).toBeGreaterThan(0);
134 + const openai = createOpenAI({ apiKey: Bun.env.OPENAI_API_KEY });
135
76 - const hackernewsAgent = new Agent({
77 - id: 'test',
78 - name: 'test',
79 - instructions: `You're a helpful Hackernews agent, able to finding information about users.`,
80 - model: openai('gpt-5.1'),
81 - tools,
82 - });
136 + const hackernewsAgent = new Agent({
137 + id: 'test',
138 + name: 'test',
139 + instructions: `You're a helpful Hackernews agent, able to finding information about users.`,
140 + model: openai('gpt-5.5'),
141 + tools,
142 + });
143
84 - const result = await hackernewsAgent.generate(
85 - 'Look up user "pg", and tell me their karma score.',
86 - {
87 - structuredOutput: {
88 - schema: z.object({
89 - karma: z.number(),
90 - }),
91 - },
92 - stopWhen: stepCountIs(10),
93 - }
94 - );
144 + const result = await hackernewsAgent.generate(
145 + 'Look up the HackerNews user "pg" with HACKERNEWS_GET_USER, then return the exact karma value from that tool result.',
146 + { structuredOutput: { schema: KARMA_SCHEMA }, stopWhen: stepCountIs(10) }
147 + );
148
96 - const toolCalls = result.toolCalls.flatMap(toolCall => toolCall.payload.toolName);
97 - const toolCount = Object.keys(tools).length;
149 + // The karma must come from a HACKERNEWS_GET_USER execution, not from the model's memory.
150 + const lookupCallIds = new Set(
151 + result.toolCalls
152 + .filter(({ payload }) => payload.toolName === EXECUTE_TOOL)
153 + .filter(({ payload }) =>
154 + executeArgsSchema
155 + .safeParse(payload.args)
156 + .data?.tools.some(
157 + ({ tool_slug, arguments: args }) =>
158 + tool_slug === 'HACKERNEWS_GET_USER' && args.username === 'pg'
159 + )
160 + )
161 + .map(({ payload }) => payload.toolCallId)
162 + );
163 + const lookupResults = result.toolResults
164 + .filter(({ payload }) => lookupCallIds.has(payload.toolCallId))
165 + .map(({ payload }) => toolResultText(payload.result));
166
99 - expect(toolCount).toBeGreaterThan(0);
100 - expect(toolCalls).toBeDefined();
101 - expect(result.error).toBeUndefined();
102 - expect(result.object).toBeDefined();
103 - expect(result.object.karma).toBeGreaterThanOrEqual(0);
167 + const parsed = KARMA_SCHEMA.safeParse(result.object);
168 + if (!parsed.success || lookupResults.length === 0) {
169 + const diagnostics = {
170 + finishReason: result.finishReason,
171 + toolCalls: result.toolCalls.map(({ payload }) => payload.toolName),
172 + searches: result.toolResults
173 + .filter(({ payload }) => payload.toolName === SEARCH_TOOL)
174 + .map(({ payload }) => searchRecord(payload)),
175 + executions: result.toolCalls
176 + .filter(({ payload }) => payload.toolName === EXECUTE_TOOL)
177 + .map(({ payload }) => executeArgsSchema.safeParse(payload.args).data?.tools),
178 + lookupResults: lookupResults.map(text => text.slice(0, 500)),
179 + steps: result.steps.map(step => ({
180 + finishReason: step.finishReason,
181 + usage: step.usage,
182 + providerMetadata: step.providerMetadata,
183 + })),
184 + warnings: result.warnings,
185 + };
186 + console.error('Agent diagnostics', JSON.stringify(diagnostics, null, 2));
187 + }
188
105 - await mcpClient.disconnect();
189 + expect(result.error).toBeUndefined();
190 + const { karma } = KARMA_SCHEMA.parse(result.object);
191 + expect(karma).toBeGreaterThan(150_000);
192 + expect(lookupResults.join('\n')).toContain(String(karma));
193 + } finally {
194 + await mcpClient.disconnect();
195 + }
196 },
197 {
198 timeout: TIMEOUTS.LLM_LONG,
ts/e2e-tests/runtimes/node/mastra-tool-router-zod-v4/e2e.test.ts
+119 -29
@@ -8,7 +8,7 @@
8
9 import { Composio } from '@composio/core';
10 import { MastraProvider } from '@composio/mastra';
11 -import { MCPClient } from '@mastra/mcp';
11 +import { getMcpCallToolMeta, MCPClient } from '@mastra/mcp';
12 import { Agent } from '@mastra/core/agent';
13 import { e2e } from '@e2e-tests/utils';
14 import { TIMEOUTS } from '@e2e-tests/utils/const';
@@ -18,6 +18,64 @@ import { stepCountIs } from 'ai';
18 import { z } from 'zod';
19
20 const MCP_CONNECT_TIMEOUT_MS = 15_000;
21 +const KARMA_SCHEMA = z.object({ karma: z.number() });
22 +const SEARCH_TOOL = 'myHttpClient_COMPOSIO_SEARCH_TOOLS';
23 +const EXECUTE_TOOL = 'myHttpClient_COMPOSIO_MULTI_EXECUTE_TOOL';
24 +
25 +// Allowlist for failure logs: Zod strips every other field.
26 +const idsSchema = z.object({ log_id: z.string().optional(), request_id: z.string().optional() });
27 +const searchResponseSchema = idsSchema.extend({
28 + data: z
29 + .object({
30 + results: z.array(z.object({ primary_tool_slugs: z.array(z.string()).optional() })).optional(),
31 + tool_schemas: z
32 + .object({
33 + HACKERNEWS_GET_USER: z
34 + .object({
35 + input_schema: z.object({ required: z.array(z.string()).optional() }).nullish(),
36 + })
37 + .optional(),
38 + })
39 + .optional(),
40 + })
41 + .optional(),
42 +});
43 +const executeArgsSchema = z.object({
44 + tools: z.array(z.object({ tool_slug: z.string(), arguments: z.record(z.string(), z.unknown()) })),
45 +});
46 +const mcpTextResultSchema = z.object({
47 + _meta: idsSchema.optional(),
48 + content: z.array(z.object({ type: z.literal('text'), text: z.string() })),
49 +});
50 +
51 +/** Flat log record for one search result, given as a structured value or an MCP text envelope. */
52 +function searchRecord(payload: { toolCallId: string; isError?: boolean; result: unknown }) {
53 + const envelope = mcpTextResultSchema.safeParse(payload.result).data;
54 + let value: unknown = payload.result;
55 + if (envelope) {
56 + try {
57 + value = JSON.parse(envelope.content.map(part => part.text).join(''));
58 + } catch {
59 + value = undefined;
60 + }
61 + }
62 + const response = searchResponseSchema.safeParse(value).data;
63 + const meta = idsSchema.safeParse(getMcpCallToolMeta(payload.result) ?? envelope?._meta).data;
64 + return {
65 + toolCallId: payload.toolCallId,
66 + isError: payload.isError,
67 + log_id: meta?.log_id ?? response?.log_id,
68 + request_id: meta?.request_id ?? response?.request_id,
69 + requiredInput: response?.data?.tool_schemas?.HACKERNEWS_GET_USER?.input_schema?.required,
70 + matchedSlugs: response?.data?.results?.flatMap(r => r.primary_tool_slugs ?? []),
71 + };
72 +}
73 +
74 +/** Text of an MCP tool result, or its JSON when the result is structured. */
75 +function toolResultText(result: unknown) {
76 + const envelope = mcpTextResultSchema.safeParse(result).data;
77 + return envelope ? envelope.content.map(part => part.text).join('') : JSON.stringify(result);
78 +}
79
80 declare module 'bun' {
81 interface Env {
@@ -70,39 +128,71 @@ e2e(import.meta.url, {
128 },
129 });
130
73 - const tools = await mcpClient.listTools();
74 - const openai = createOpenAI({ apiKey: Bun.env.OPENAI_API_KEY });
131 + try {
132 + const tools = await mcpClient.listTools();
133 + expect(Object.keys(tools).length).toBeGreaterThan(0);
134 + const openai = createOpenAI({ apiKey: Bun.env.OPENAI_API_KEY });
135
76 - const hackernewsAgent = new Agent({
77 - id: 'test',
78 - name: 'test',
79 - instructions: `You're a helpful Hackernews agent, able to finding information about users.`,
80 - model: openai('gpt-5.1'),
81 - tools,
82 - });
136 + const hackernewsAgent = new Agent({
137 + id: 'test',
138 + name: 'test',
139 + instructions: `You're a helpful Hackernews agent, able to finding information about users.`,
140 + model: openai('gpt-5.5'),
141 + tools,
142 + });
143
84 - const result = await hackernewsAgent.generate(
85 - 'Look up user "pg", and tell me their karma score.',
86 - {
87 - structuredOutput: {
88 - schema: z.object({
89 - karma: z.number(),
90 - }),
91 - },
92 - stopWhen: stepCountIs(10),
93 - }
94 - );
144 + const result = await hackernewsAgent.generate(
145 + 'Look up the HackerNews user "pg" with HACKERNEWS_GET_USER, then return the exact karma value from that tool result.',
146 + { structuredOutput: { schema: KARMA_SCHEMA }, stopWhen: stepCountIs(10) }
147 + );
148
96 - const toolCalls = result.toolCalls.flatMap(toolCall => toolCall.payload.toolName);
97 - const toolCount = Object.keys(tools).length;
149 + // The karma must come from a HACKERNEWS_GET_USER execution, not from the model's memory.
150 + const lookupCallIds = new Set(
151 + result.toolCalls
152 + .filter(({ payload }) => payload.toolName === EXECUTE_TOOL)
153 + .filter(({ payload }) =>
154 + executeArgsSchema
155 + .safeParse(payload.args)
156 + .data?.tools.some(
157 + ({ tool_slug, arguments: args }) =>
158 + tool_slug === 'HACKERNEWS_GET_USER' && args.username === 'pg'
159 + )
160 + )
161 + .map(({ payload }) => payload.toolCallId)
162 + );
163 + const lookupResults = result.toolResults
164 + .filter(({ payload }) => lookupCallIds.has(payload.toolCallId))
165 + .map(({ payload }) => toolResultText(payload.result));
166
99 - expect(toolCount).toBeGreaterThan(0);
100 - expect(toolCalls).toBeDefined();
101 - expect(result.error).toBeUndefined();
102 - expect(result.object).toBeDefined();
103 - expect(result.object.karma).toBeGreaterThanOrEqual(0);
167 + const parsed = KARMA_SCHEMA.safeParse(result.object);
168 + if (!parsed.success || lookupResults.length === 0) {
169 + const diagnostics = {
170 + finishReason: result.finishReason,
171 + toolCalls: result.toolCalls.map(({ payload }) => payload.toolName),
172 + searches: result.toolResults
173 + .filter(({ payload }) => payload.toolName === SEARCH_TOOL)
174 + .map(({ payload }) => searchRecord(payload)),
175 + executions: result.toolCalls
176 + .filter(({ payload }) => payload.toolName === EXECUTE_TOOL)
177 + .map(({ payload }) => executeArgsSchema.safeParse(payload.args).data?.tools),
178 + lookupResults: lookupResults.map(text => text.slice(0, 500)),
179 + steps: result.steps.map(step => ({
180 + finishReason: step.finishReason,
181 + usage: step.usage,
182 + providerMetadata: step.providerMetadata,
183 + })),
184 + warnings: result.warnings,
185 + };
186 + console.error('Agent diagnostics', JSON.stringify(diagnostics, null, 2));
187 + }
188
105 - await mcpClient.disconnect();
189 + expect(result.error).toBeUndefined();
190 + const { karma } = KARMA_SCHEMA.parse(result.object);
191 + expect(karma).toBeGreaterThan(150_000);
192 + expect(lookupResults.join('\n')).toContain(String(karma));
193 + } finally {
194 + await mcpClient.disconnect();
195 + }
196 },
197 {
198 timeout: TIMEOUTS.LLM_LONG,