@devries-cameron20 / composio Mirror · read-only / commits / 71ec523d2
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 16:00 UTC

feat(examples): live-sweep harness, manifest, and scheduled workflow (#4117)

This PR: - builds on merged PR #4107 - adds `examples-manifest.json`: 93 runnable entrypoints across both SDKs with tiers (unattended / provisioned / bounded / excluded), per-entry timeouts, env contracts, and labeled readiness markers for OAuth flows - adds `harness/run.mjs`: sweeps entries against staging with per-entry Composio traces, negative controls (garbage credentials must fail), a `selftest`, and a baseline/candidate client swap for parity runs - rejects every non-staging `COMPOSIO_BASE_URL` before the harness or provisioning script can send credentials - adds `--llm mock`: LLM examples run against a pinned `@copilotkit/aimock` server with scripted tool-call fixtures (`harness/llm-mock/`) — full sweeps spend zero model tokens while Composio tool execution stays live on staging; 46/48 LLM entries pass mocked, and the two `@openai/agents` entries are marked `llmMock:false` - adds `examples-live.yml`: nightly gc-provision → mock sweep → four-entry live LLM canary; dispatchable per language, client, entry subset, and LLM mode, and bound to the `staging` GitHub environment - accepts separate TypeScript tarball and Python wheel inputs for candidate sweeps, restores exact pre-run workspace files on failure, and rejects empty or partial parity comparisons The configured `COMPOSIO_API_KEY` must belong to a dedicated, disposable staging project. The workflow becomes dispatchable after it exists on the default branch. ## Context Examples historically went stale because nothing executed them. PR #4107 makes them runnable and provisionable; this PR adds the machinery that runs them on a schedule and validates the Stainless→self-managed client swap via baseline/candidate parity sweeps. ```mermaid graph TD M[examples-manifest.json] --> R[harness/run.mjs sweep] P[examples-provision.mjs --gc] -->|COMPOSIO_EXAMPLES_* ids| R R -->|tier 1-3 entries| E[TS + Python examples] A[aimock fixtures] -->|--llm mock| E E -->|live tool execution + traces| C[Composio staging] R --> N[nightly live canary: 4 entries] ``` ## Verification - `env -u COMPOSIO_BASE_URL node harness/run.mjs selftest` - `pnpm test:examples` against a clean tracked snapshot - manifest integrity: 93 unique entry IDs and 93 existing files - JavaScript, Python, JSON, YAML, and embedded Bash syntax checks - `pnpm validate:changesets`

Alberto Schiabel committed Aug 14, 2026 at 15:47 UTC 71ec523d258bf415c6cfeb2bd9854c3ef4e9348e
15 files changed +2360 -22
.github/workflows/examples-live.yml new
+167
@@ -0,0 +1,167 @@
1 +name: Examples Manifest Sweep (staging)
2 +
3 +# Runs the example entrypoints from examples-manifest.json against the
4 +# STAGING backend using a dedicated, disposable examples project — never
5 +# the org's main key. Non-blocking by design: scheduled + on demand, never a
6 +# required PR check. Local runs and CI share the same code path
7 +# (harness/run.mjs sweep).
8 +on:
9 + schedule:
10 + - cron: '0 7 * * *' # 07:00 UTC nightly
11 + workflow_dispatch:
12 + inputs:
13 + lang:
14 + description: 'Language to sweep'
15 + type: choice
16 + options: [all, ts, py]
17 + default: all
18 + client:
19 + description: 'Client under test'
20 + type: choice
21 + options: [baseline, candidate]
22 + default: baseline
23 + ids:
24 + description: 'Comma-separated entry ids (empty = full manifest)'
25 + type: string
26 + default: ''
27 + client_tarball_url:
28 + description: 'Candidate TypeScript client tarball URL (required for TS candidate entries)'
29 + type: string
30 + default: ''
31 + client_wheel_url:
32 + description: 'Candidate Python client wheel URL (required for Python candidate entries)'
33 + type: string
34 + default: ''
35 + llm:
36 + description: 'LLM traffic: mock (aimock, zero model spend) or live'
37 + type: choice
38 + options: [mock, live]
39 + default: mock
40 +
41 +concurrency:
42 + group: ${{ github.workflow }}
43 + cancel-in-progress: false
44 +
45 +permissions:
46 + contents: read
47 +
48 +env:
49 + # Staging URL is not a secret; the keys are. The key must belong to a
50 + # project that holds no valuable connections or data — never the org's
51 + # main project. Example runs create their own resources and only ever
52 + # delete what they created.
53 + COMPOSIO_BASE_URL: https://staging-backend.composio.dev
54 + COMPOSIO_API_KEY: ${{ secrets.COMPOSIO_API_KEY }}
55 + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
56 + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
57 + # mock unless explicitly dispatched live; the nightly adds a small live canary.
58 + LLM_MODE: ${{ inputs.llm || 'mock' }}
59 + # Entries mocks cannot cover (llmMock:false) plus one representative
60 + # tool-calling example per language: the drift signal mocks can't see.
61 + CANARY_IDS: ts/openai/agents-api,ts/openai/agents-api-tool-router,ts/openai/chat-completions,py/tool_router/langchain_agent
62 +
63 +jobs:
64 + examples-live:
65 + name: ${{ inputs.client || 'baseline' }} sweep (${{ inputs.lang || 'all' }})
66 + environment: staging
67 + runs-on: ubuntu-latest
68 + timeout-minutes: 60
69 +
70 + steps:
71 + - name: Checkout Code
72 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
73 +
74 + - name: Setup Node.js, pnpm, Bun
75 + uses: ./.github/actions/setup-node-pnpm-bun
76 + with:
77 + enable-turbo-cache: 'true'
78 +
79 + - name: Setup Python and uv
80 + uses: ./.github/actions/setup-python-uv
81 +
82 + - name: Require credentials
83 + run: |
84 + if [ -z "$COMPOSIO_API_KEY" ]; then
85 + echo "::error::COMPOSIO_API_KEY is required."
86 + exit 1
87 + fi
88 + # LLM keys are only needed when live model traffic will run:
89 + # an explicit live dispatch, or the nightly's live canary step.
90 + if [ "$LLM_MODE" = "live" ] || [ "${{ github.event_name }}" = "schedule" ]; then
91 + if [ -z "$OPENAI_API_KEY" ] || [ -z "$ANTHROPIC_API_KEY" ]; then
92 + echo "::error::OPENAI_API_KEY and ANTHROPIC_API_KEY are required for live LLM runs."
93 + exit 1
94 + fi
95 + fi
96 +
97 + - name: Install Dependencies
98 + run: pnpm install --frozen-lockfile
99 +
100 + - name: Build SDK packages
101 + run: pnpm run build:packages
102 +
103 + - name: Fetch candidate client artifacts
104 + if: ${{ inputs.client == 'candidate' }}
105 + env:
106 + CLIENT_TARBALL_URL: ${{ inputs.client_tarball_url }}
107 + CLIENT_WHEEL_URL: ${{ inputs.client_wheel_url }}
108 + LANG_INPUT: ${{ inputs.lang || 'all' }}
109 + run: |
110 + set -euo pipefail
111 + mkdir -p .artifacts/candidate
112 + if [ "$LANG_INPUT" = "all" ] || [ "$LANG_INPUT" = "ts" ]; then
113 + if [[ "$CLIENT_TARBALL_URL" != https://* ]]; then
114 + echo "::error::TS candidate entries require an HTTPS client_tarball_url"
115 + exit 1
116 + fi
117 + curl -fsSL -o .artifacts/candidate/composio-client.tgz "$CLIENT_TARBALL_URL"
118 + echo "COMPOSIO_CLIENT_TARBALL=$PWD/.artifacts/candidate/composio-client.tgz" >> "$GITHUB_ENV"
119 + fi
120 + if [ "$LANG_INPUT" = "all" ] || [ "$LANG_INPUT" = "py" ]; then
121 + if [[ "$CLIENT_WHEEL_URL" != https://* ]]; then
122 + echo "::error::Python candidate entries require an HTTPS client_wheel_url"
123 + exit 1
124 + fi
125 + curl -fsSL -o .artifacts/candidate/composio-client.whl "$CLIENT_WHEEL_URL"
126 + echo "COMPOSIO_CLIENT_WHEEL=$PWD/.artifacts/candidate/composio-client.whl" >> "$GITHUB_ENV"
127 + fi
128 +
129 + - name: Harness selftest
130 + run: node harness/run.mjs selftest
131 +
132 + - name: Verify provisioned project state
133 + run: node scripts/examples-provision.mjs --gc > /tmp/examples-provision.env
134 +
135 + - name: Sweep
136 + env:
137 + CLIENT_INPUT: ${{ inputs.client || 'baseline' }}
138 + IDS_INPUT: ${{ inputs.ids }}
139 + LANG_INPUT: ${{ inputs.lang || 'all' }}
140 + run: |
141 + set -euo pipefail
142 + # Load the provisioned COMPOSIO_EXAMPLES_* ids emitted above.
143 + source /tmp/examples-provision.env
144 + ARGS=(--client "$CLIENT_INPUT" --llm "$LLM_MODE")
145 + if [ "$LANG_INPUT" != "all" ]; then
146 + ARGS+=(--lang "$LANG_INPUT")
147 + fi
148 + if [ -n "$IDS_INPUT" ]; then
149 + ARGS+=(--ids "$IDS_INPUT")
150 + fi
151 + node harness/run.mjs sweep "${ARGS[@]}"
152 +
153 + - name: Live LLM canary
154 + if: ${{ github.event_name == 'schedule' }}
155 + run: |
156 + set -euo pipefail
157 + source /tmp/examples-provision.env
158 + node harness/run.mjs sweep --client baseline --llm live --ids "$CANARY_IDS"
159 +
160 + - name: Upload sweep results and traces
161 + if: ${{ always() }}
162 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
163 + with:
164 + name: examples-live-results
165 + path: .artifacts/examples-parity/
166 + retention-days: 14
167 + if-no-files-found: ignore
examples-manifest.json new
+1294
@@ -0,0 +1,1294 @@
1 +{
2 + "$comment": "Inventory of runnable example entrypoints, consumed by harness/run.mjs. Tiers: 1=unattended, 2=needs provisioned state (scripts/examples-provision.mjs), 3=bounded (pass at readiness match, then terminate), X=excluded. Readiness regexes must anchor on a labeled output line the example prints on success (e.g. 'Visit this URL to authorize: https?://'), never a bare 'https?://' — a URL inside an error message must not count as ready. llmMock:false marks entries whose LLM traffic cannot run against the mock server (live canary only).",
3 + "entries": [
4 + {
5 + "id": "ts/anthropic/index",
6 + "lang": "ts",
7 + "pkg": "anthropic-example",
8 + "file": "ts/examples/anthropic/src/index.ts",
9 + "tier": "1",
10 + "env": [
11 + "ANTHROPIC_API_KEY"
12 + ],
13 + "toolkits": [
14 + "hackernews"
15 + ],
16 + "timeoutSec": 180
17 + },
18 + {
19 + "id": "ts/anthropic/streaming",
20 + "lang": "ts",
21 + "pkg": "anthropic-example",
22 + "file": "ts/examples/anthropic/src/streaming.ts",
23 + "tier": "1",
24 + "env": [
25 + "ANTHROPIC_API_KEY"
26 + ],
27 + "toolkits": [
28 + "hackernews"
29 + ],
30 + "timeoutSec": 180
31 + },
32 + {
33 + "id": "ts/anthropic/mcp",
34 + "lang": "ts",
35 + "pkg": "anthropic-example",
36 + "file": "ts/examples/anthropic/src/mcp.ts",
37 + "tier": "2",
38 + "env": [
39 + "ANTHROPIC_API_KEY"
40 + ],
41 + "ids": [
42 + "COMPOSIO_EXAMPLES_USER_ID"
43 + ],
44 + "toolkits": [
45 + "gmail"
46 + ],
47 + "timeoutSec": 180
48 + },
49 + {
50 + "id": "ts/anthropic/experimental.mcp",
51 + "lang": "ts",
52 + "pkg": "anthropic-example",
53 + "file": "ts/examples/anthropic/src/experimental.mcp.ts",
54 + "tier": "2",
55 + "env": [
56 + "ANTHROPIC_API_KEY"
57 + ],
58 + "ids": [
59 + "COMPOSIO_EXAMPLES_USER_ID"
60 + ],
61 + "toolkits": [
62 + "gmail"
63 + ],
64 + "timeoutSec": 180
65 + },
66 + {
67 + "id": "ts/anthropic/claude-agents-sdk",
68 + "lang": "ts",
69 + "pkg": "anthropic-example",
70 + "file": "ts/examples/anthropic/src/claude-agents-sdk.ts",
71 + "tier": "2",
72 + "env": [
73 + "ANTHROPIC_API_KEY"
74 + ],
75 + "toolkits": [
76 + "gmail"
77 + ],
78 + "timeoutSec": 300,
79 + "notes": "needs @anthropic-ai/claude-code CLI on PATH"
80 + },
81 + {
82 + "id": "ts/cloudflare-wrangler/dev",
83 + "lang": "ts",
84 + "pkg": "cloudflare-wrangler-example",
85 + "file": "ts/examples/cloudflare-wrangler/src/index.ts",
86 + "tier": "3",
87 + "script": "dev",
88 + "readiness": "Ready on|Listening",
89 + "toolkits": [
90 + "googledrive"
91 + ],
92 + "timeoutSec": 120,
93 + "notes": "wrangler dev; env must reach .dev.vars"
94 + },
95 + {
96 + "id": "ts/connected-accounts/index",
97 + "lang": "ts",
98 + "pkg": "connected-accounts-example",
99 + "file": "ts/examples/connected-accounts/src/index.ts",
100 + "tier": "3",
101 + "readiness": "authorize the user: https?://",
102 + "toolkits": [
103 + "github"
104 + ],
105 + "timeoutSec": 90
106 + },
107 + {
108 + "id": "ts/connected-accounts/api-key",
109 + "lang": "ts",
110 + "pkg": "connected-accounts-example",
111 + "file": "ts/examples/connected-accounts/src/api-key.ts",
112 + "tier": "2",
113 + "ids": [
114 + "COMPOSIO_EXAMPLES_APIKEY_AUTH_CONFIG_ID",
115 + "COMPOSIO_EXAMPLES_APIKEY_PLACEHOLDER",
116 + "COMPOSIO_EXAMPLES_USER_ID"
117 + ],
118 + "toolkits": [],
119 + "timeoutSec": 90
120 + },
121 + {
122 + "id": "ts/connected-accounts/magic-flow-demo",
123 + "lang": "ts",
124 + "pkg": "connected-accounts-example",
125 + "file": "ts/examples/connected-accounts/src/magic-flow-demo.ts",
126 + "tier": "3",
127 + "env": [
128 + "OPENAI_API_KEY"
129 + ],
130 + "readiness": "Redirect the user to https?://",
131 + "timeoutSec": 120
132 + },
133 + {
134 + "id": "ts/connected-accounts/multiple-connected-accounts",
135 + "lang": "ts",
136 + "pkg": "connected-accounts-example",
137 + "file": "ts/examples/connected-accounts/src/multiple-connected-accounts.ts",
138 + "tier": "2",
139 + "ids": [
140 + "COMPOSIO_EXAMPLES_GMAIL_AUTH_CONFIG_ID"
141 + ],
142 + "toolkits": [
143 + "gmail"
144 + ],
145 + "timeoutSec": 90
146 + },
147 + {
148 + "id": "ts/connected-accounts/scopes",
149 + "lang": "ts",
150 + "pkg": "connected-accounts-example",
151 + "file": "ts/examples/connected-accounts/src/scopes.ts",
152 + "tier": "2",
153 + "ids": [
154 + "COMPOSIO_EXAMPLES_GMAIL_AUTH_CONFIG_ID"
155 + ],
156 + "toolkits": [
157 + "gmail"
158 + ],
159 + "timeoutSec": 90
160 + },
161 + {
162 + "id": "ts/connected-accounts/toolkit-authorize",
163 + "lang": "ts",
164 + "pkg": "connected-accounts-example",
165 + "file": "ts/examples/connected-accounts/src/toolkit-authorize.ts",
166 + "tier": "3",
167 + "readiness": "authorize the toolkit: https?://",
168 + "toolkits": [
169 + "github"
170 + ],
171 + "timeoutSec": 90
172 + },
173 + {
174 + "id": "ts/error-handling-demo/index",
175 + "lang": "ts",
176 + "pkg": "error-handling-example",
177 + "file": "ts/examples/error-handling-demo/src/index.ts",
178 + "tier": "1",
179 + "backend": false,
180 + "timeoutSec": 60
181 + },
182 + {
183 + "id": "ts/file-handling/index",
184 + "lang": "ts",
185 + "pkg": "file-handling-example",
186 + "file": "ts/examples/file-handling/src/index.ts",
187 + "tier": "X",
188 + "toolkits": [
189 + "gmail"
190 + ],
191 + "notes": "sends real email by design; excluded until a non-send variant is decided by the human",
192 + "reason": "outbound-email"
193 + },
194 + {
195 + "id": "ts/file-handling/url",
196 + "lang": "ts",
197 + "pkg": "file-handling-example",
198 + "file": "ts/examples/file-handling/src/url.ts",
199 + "tier": "X",
200 + "toolkits": [
201 + "gmail"
202 + ],
203 + "reason": "outbound-email",
204 + "notes": "sends real email by design; excluded until a non-send variant is decided by the human"
205 + },
206 + {
207 + "id": "ts/google/index",
208 + "lang": "ts",
209 + "pkg": "google-example",
210 + "file": "ts/examples/google/src/index.ts",
211 + "tier": "X",
212 + "env": [
213 + "GEMINI_API_KEY"
214 + ],
215 + "toolkits": [
216 + "hackernews"
217 + ],
218 + "reason": "dropped-provider",
219 + "notes": "gemini/notion are out of scope for this goal by decision (2026-08-10)"
220 + },
221 + {
222 + "id": "ts/google/experimental.mcp",
223 + "lang": "ts",
224 + "pkg": "google-example",
225 + "file": "ts/examples/google/src/experimental.mcp.ts",
226 + "tier": "X",
227 + "env": [
228 + "GEMINI_API_KEY"
229 + ],
230 + "ids": [
231 + "COMPOSIO_EXAMPLES_GMAIL_AUTH_CONFIG_ID",
232 + "COMPOSIO_EXAMPLES_USER_ID"
233 + ],
234 + "toolkits": [
235 + "gmail"
236 + ],
237 + "reason": "dropped-provider",
238 + "notes": "gemini/notion are out of scope for this goal by decision (2026-08-10)"
239 + },
240 + {
241 + "id": "ts/json-schema-to-zod/index",
242 + "lang": "ts",
243 + "pkg": "json-schema-to-zod-example",
244 + "file": "ts/examples/json-schema-to-zod/src/index.ts",
245 + "tier": "1",
246 + "timeoutSec": 90
247 + },
248 + {
249 + "id": "ts/langchain/index",
250 + "lang": "ts",
251 + "pkg": "langchain-example",
252 + "file": "ts/examples/langchain/src/index.ts",
253 + "tier": "1",
254 + "env": [
255 + "OPENAI_API_KEY"
256 + ],
257 + "toolkits": [
258 + "hackernews"
259 + ],
260 + "timeoutSec": 180
261 + },
262 + {
263 + "id": "ts/langchain/openai",
264 + "lang": "ts",
265 + "pkg": "langchain-example",
266 + "file": "ts/examples/langchain/src/openai.ts",
267 + "tier": "2",
268 + "env": [
269 + "OPENAI_API_KEY"
270 + ],
271 + "ids": [
272 + "COMPOSIO_EXAMPLES_USER_ID"
273 + ],
274 + "toolkits": [
275 + "gmail"
276 + ],
277 + "timeoutSec": 300
278 + },
279 + {
280 + "id": "ts/llamaindex/index",
281 + "lang": "ts",
282 + "pkg": "llamaindex-example",
283 + "file": "ts/examples/llamaindex/src/index.ts",
284 + "tier": "1",
285 + "env": [
286 + "OPENAI_API_KEY"
287 + ],
288 + "toolkits": [
289 + "hackernews"
290 + ],
291 + "timeoutSec": 180
292 + },
293 + {
294 + "id": "ts/mastra/index",
295 + "lang": "ts",
296 + "pkg": "mastra-example",
297 + "file": "ts/examples/mastra/src/index.ts",
298 + "tier": "1",
299 + "env": [
300 + "OPENAI_API_KEY"
301 + ],
302 + "toolkits": [
303 + "hackernews"
304 + ],
305 + "timeoutSec": 180
306 + },
307 + {
308 + "id": "ts/mastra/tool-router",
309 + "lang": "ts",
310 + "pkg": "mastra-example",
311 + "file": "ts/examples/mastra/src/tool-router.ts",
312 + "tier": "1",
313 + "env": [
314 + "OPENAI_API_KEY"
315 + ],
316 + "toolkits": [
317 + "hackernews"
318 + ],
319 + "timeoutSec": 180
320 + },
321 + {
322 + "id": "ts/mastra/smoke",
323 + "lang": "ts",
324 + "pkg": "mastra-example",
325 + "file": "ts/examples/mastra/src/smoke.ts",
326 + "tier": "1",
327 + "toolkits": [
328 + "hackernews"
329 + ],
330 + "timeoutSec": 90
331 + },
332 + {
333 + "id": "ts/mastra/cloudflare",
334 + "lang": "ts",
335 + "pkg": "mastra-example",
336 + "file": "ts/examples/mastra/src/cloudflare.ts",
337 + "tier": "X",
338 + "reason": "worker-module",
339 + "notes": "covered by cf:dry-run in ts.examples.yml"
340 + },
341 + {
342 + "id": "ts/mcp/index",
343 + "lang": "ts",
344 + "pkg": "mcp-example",
345 + "file": "ts/examples/mcp/src/index.ts",
346 + "tier": "2",
347 + "env": [
348 + "OPENAI_API_KEY"
349 + ],
350 + "ids": [
351 + "COMPOSIO_EXAMPLES_GMAIL_AUTH_CONFIG_ID",
352 + "COMPOSIO_EXAMPLES_USER_ID"
353 + ],
354 + "toolkits": [
355 + "gmail"
356 + ],
357 + "timeoutSec": 180
358 + },
359 + {
360 + "id": "ts/modifiers/index",
361 + "lang": "ts",
362 + "pkg": "modifiers-example",
363 + "file": "ts/examples/modifiers/src/index.ts",
364 + "tier": "1",
365 + "env": [
366 + "OPENAI_API_KEY"
367 + ],
368 + "toolkits": [
369 + "hackernews"
370 + ],
371 + "timeoutSec": 180
372 + },
373 + {
374 + "id": "ts/openai/index",
375 + "lang": "ts",
376 + "pkg": "openai-example",
377 + "file": "ts/examples/openai/src/index.ts",
378 + "tier": "1",
379 + "env": [
380 + "OPENAI_API_KEY"
381 + ],
382 + "toolkits": [
383 + "hackernews"
384 + ],
385 + "timeoutSec": 180
386 + },
387 + {
388 + "id": "ts/openai/tool-router",
389 + "lang": "ts",
390 + "pkg": "openai-example",
391 + "file": "ts/examples/openai/src/tool-router.ts",
392 + "tier": "1",
393 + "env": [
394 + "OPENAI_API_KEY"
395 + ],
396 + "toolkits": [
397 + "hackernews"
398 + ],
399 + "timeoutSec": 180
400 + },
401 + {
402 + "id": "ts/openai/smoke",
403 + "lang": "ts",
404 + "pkg": "openai-example",
405 + "file": "ts/examples/openai/src/smoke.ts",
406 + "tier": "1",
407 + "toolkits": [
408 + "hackernews"
409 + ],
410 + "timeoutSec": 90
411 + },
412 + {
413 + "id": "ts/openai/chat-completions",
414 + "lang": "ts",
415 + "pkg": "openai-example",
416 + "file": "ts/examples/openai/src/chat-completions.ts",
417 + "tier": "1",
418 + "env": [
419 + "OPENAI_API_KEY"
420 + ],
421 + "toolkits": [
422 + "hackernews"
423 + ],
424 + "timeoutSec": 180
425 + },
426 + {
427 + "id": "ts/openai/tools",
428 + "lang": "ts",
429 + "pkg": "openai-example",
430 + "file": "ts/examples/openai/src/tools.ts",
431 + "tier": "2",
432 + "env": [
433 + "OPENAI_API_KEY"
434 + ],
435 + "toolkits": [
436 + "googledrive"
437 + ],
438 + "timeoutSec": 180
439 + },
440 + {
441 + "id": "ts/openai/cloudflare",
442 + "lang": "ts",
443 + "pkg": "openai-example",
444 + "file": "ts/examples/openai/src/cloudflare.ts",
445 + "tier": "X",
446 + "reason": "worker-module",
447 + "notes": "covered by cf:dry-run in ts.examples.yml"
448 + },
449 + {
450 + "id": "ts/openai/agents-api",
451 + "lang": "ts",
452 + "pkg": "openai-example",
453 + "file": "ts/examples/openai/src/agents-api/index.ts",
454 + "tier": "1",
455 + "env": [
456 + "OPENAI_API_KEY"
457 + ],
458 + "toolkits": [
459 + "hackernews"
460 + ],
461 + "timeoutSec": 180,
462 + "llmMock": false,
463 + "notes": "@openai/agents rejects aimock responses-api output; live canary only."
464 + },
465 + {
466 + "id": "ts/openai/agents-api-mcp",
467 + "lang": "ts",
468 + "pkg": "openai-example",
469 + "file": "ts/examples/openai/src/agents-api/experiment.mcp.ts",
470 + "tier": "2",
471 + "env": [
472 + "OPENAI_API_KEY"
473 + ],
474 + "ids": [
475 + "COMPOSIO_EXAMPLES_GMAIL_AUTH_CONFIG_ID",
476 + "COMPOSIO_EXAMPLES_USER_ID"
477 + ],
478 + "toolkits": [
479 + "gmail"
480 + ],
481 + "timeoutSec": 180
482 + },
483 + {
484 + "id": "ts/openai/agents-api-tool-router",
485 + "lang": "ts",
486 + "pkg": "openai-example",
487 + "file": "ts/examples/openai/src/agents-api/experiment.tool-router.ts",
488 + "tier": "1",
489 + "env": [
490 + "OPENAI_API_KEY"
491 + ],
492 + "toolkits": [
493 + "hackernews"
494 + ],
495 + "timeoutSec": 180,
496 + "llmMock": false,
497 + "notes": "@openai/agents rejects aimock responses-api output; live canary only."
498 + },
499 + {
500 + "id": "ts/openai/responses-api",
501 + "lang": "ts",
502 + "pkg": "openai-example",
503 + "file": "ts/examples/openai/src/responses-api/index.ts",
504 + "tier": "1",
505 + "env": [
506 + "OPENAI_API_KEY"
507 + ],
508 + "toolkits": [
509 + "hackernews"
510 + ],
511 + "timeoutSec": 180
512 + },
513 + {
514 + "id": "ts/openai/responses-api-mcp",
515 + "lang": "ts",
516 + "pkg": "openai-example",
517 + "file": "ts/examples/openai/src/responses-api/experiment.mcp.ts",
518 + "tier": "2",
519 + "env": [
520 + "OPENAI_API_KEY"
521 + ],
522 + "ids": [
523 + "COMPOSIO_EXAMPLES_GMAIL_AUTH_CONFIG_ID",
524 + "COMPOSIO_EXAMPLES_USER_ID"
525 + ],
526 + "toolkits": [
527 + "gmail"
528 + ],
529 + "timeoutSec": 180
530 + },
531 + {
532 + "id": "ts/session-management/index",
533 + "lang": "ts",
534 + "pkg": "session-management-example",
535 + "file": "ts/examples/session-management/src/index.ts",
536 + "tier": "1",
537 + "toolkits": [
538 + "hackernews"
539 + ],
540 + "timeoutSec": 120
541 + },
542 + {
543 + "id": "ts/tool-router/index",
544 + "lang": "ts",
545 + "pkg": "tool-router-example",
546 + "file": "ts/examples/tool-router/src/index.ts",
547 + "tier": "2",
548 + "env": [
549 + "OPENAI_API_KEY"
550 + ],
551 + "toolkits": [
552 + "gmail"
553 + ],
554 + "timeoutSec": 180
555 + },
556 + {
557 + "id": "ts/tool-router/authorize",
558 + "lang": "ts",
559 + "pkg": "tool-router-example",
560 + "file": "ts/examples/tool-router/src/authorize.ts",
561 + "tier": "3",
562 + "readiness": "Visit this URL to authorize: https?://",
563 + "toolkits": [
564 + "gmail"
565 + ],
566 + "timeoutSec": 90
567 + },
568 + {
569 + "id": "ts/tool-router/claude-agent-sdk",
570 + "lang": "ts",
571 + "pkg": "tool-router-example",
572 + "file": "ts/examples/tool-router/src/claude-agent-sdk.ts",
573 + "tier": "2",
574 + "env": [
575 + "ANTHROPIC_API_KEY"
576 + ],
577 + "toolkits": [
578 + "gmail"
579 + ],
580 + "timeoutSec": 300,
581 + "notes": "needs @anthropic-ai/claude-code CLI on PATH"
582 + },
583 + {
584 + "id": "ts/tool-router/custom-tools",
585 + "lang": "ts",
586 + "pkg": "tool-router-example",
587 + "file": "ts/examples/tool-router/src/custom-tools.ts",
588 + "tier": "2",
589 + "toolkits": [
590 + "gmail"
591 + ],
592 + "timeoutSec": 120
593 + },
594 + {
595 + "id": "ts/tool-router/custom-tools-agent",
596 + "lang": "ts",
597 + "pkg": "tool-router-example",
598 + "file": "ts/examples/tool-router/src/custom-tools-agent.ts",
599 + "tier": "2",
600 + "env": [
601 + "OPENAI_API_KEY"
602 + ],
603 + "toolkits": [
604 + "gmail"
605 + ],
606 + "timeoutSec": 180
607 + },
608 + {
609 + "id": "ts/tool-router/custom-tools-e2e",
610 + "lang": "ts",
611 + "pkg": "tool-router-example",
612 + "file": "ts/examples/tool-router/src/custom-tools-e2e.ts",
613 + "tier": "2",
614 + "env": [
615 + "OPENAI_API_KEY"
616 + ],
617 + "toolkits": [
618 + "weathermap",
619 + "gmail"
620 + ],
621 + "timeoutSec": 240
622 + },
623 + {
624 + "id": "ts/tool-router/direct-tools-preset",
625 + "lang": "ts",
626 + "pkg": "tool-router-example",
627 + "file": "ts/examples/tool-router/src/direct-tools-preset.ts",
628 + "tier": "1",
629 + "env": [
630 + "OPENAI_API_KEY"
631 + ],
632 + "toolkits": [
633 + "hackernews"
634 + ],
635 + "timeoutSec": 180
636 + },
637 + {
638 + "id": "ts/tool-router/langchain",
639 + "lang": "ts",
640 + "pkg": "tool-router-example",
641 + "file": "ts/examples/tool-router/src/langchain.ts",
642 + "tier": "2",
643 + "env": [
644 + "OPENAI_API_KEY"
645 + ],
646 + "toolkits": [
647 + "gmail"
648 + ],
649 + "timeoutSec": 180,
650 + "ids": [
651 + "COMPOSIO_EXAMPLES_USER_ID"
652 + ]
653 + },
654 + {
655 + "id": "ts/tool-router/mcp",
656 + "lang": "ts",
657 + "pkg": "tool-router-example",
658 + "file": "ts/examples/tool-router/src/mcp.ts",
659 + "tier": "2",
660 + "env": [
661 + "OPENAI_API_KEY"
662 + ],
663 + "toolkits": [
664 + "gmail"
665 + ],
666 + "timeoutSec": 180
667 + },
668 + {
669 + "id": "ts/tool-router/multi-account",
670 + "lang": "ts",
671 + "pkg": "tool-router-example",
672 + "file": "ts/examples/tool-router/src/multi-account.ts",
673 + "tier": "3",
674 + "env": [
675 + "OPENAI_API_KEY"
676 + ],
677 + "readiness": "Visit this URL to authorize: https?://",
678 + "toolkits": [
679 + "gmail"
680 + ],
681 + "timeoutSec": 120,
682 + "ids": [
683 + "COMPOSIO_EXAMPLES_USER_ID"
684 + ]
685 + },
686 + {
687 + "id": "ts/tool-router/openai-agents",
688 + "lang": "ts",
689 + "pkg": "tool-router-example",
690 + "file": "ts/examples/tool-router/src/openai-agents.ts",
691 + "tier": "X",
692 + "reason": "stdin-repl"
693 + },
694 + {
695 + "id": "ts/tool-router/preload",
696 + "lang": "ts",
697 + "pkg": "tool-router-example",
698 + "file": "ts/examples/tool-router/src/preload.ts",
699 + "tier": "1",
700 + "env": [
701 + "OPENAI_API_KEY"
702 + ],
703 + "toolkits": [
704 + "hackernews"
705 + ],
706 + "timeoutSec": 180
707 + },
708 + {
709 + "id": "ts/tool-router/session-update",
710 + "lang": "ts",
711 + "pkg": "tool-router-example",
712 + "file": "ts/examples/tool-router/src/session-update.ts",
713 + "tier": "1",
714 + "toolkits": [
715 + "hackernews"
716 + ],
717 + "timeoutSec": 120
718 + },
719 + {
720 + "id": "ts/tool-router/toolkits",
721 + "lang": "ts",
722 + "pkg": "tool-router-example",
723 + "file": "ts/examples/tool-router/src/toolkits.ts",
724 + "tier": "1",
725 + "toolkits": [],
726 + "timeoutSec": 120
727 + },
728 + {
729 + "id": "ts/toolkits/index",
730 + "lang": "ts",
731 + "pkg": "toolkits-example",
732 + "file": "ts/examples/toolkits/src/index.ts",
733 + "tier": "3",
734 + "env": [
735 + "OPENAI_API_KEY"
736 + ],
737 + "readiness": "authorize the toolkit: https?://",
738 + "toolkits": [
739 + "github"
740 + ],
741 + "timeoutSec": 120
742 + },
743 + {
744 + "id": "ts/toolkits/toolkits",
745 + "lang": "ts",
746 + "pkg": "toolkits-example",
747 + "file": "ts/examples/toolkits/src/toolkits.ts",
748 + "tier": "1",
749 + "toolkits": [],
750 + "timeoutSec": 120
751 + },
752 + {
753 + "id": "ts/toolkits/auth-fields",
754 + "lang": "ts",
755 + "pkg": "toolkits-example",
756 + "file": "ts/examples/toolkits/src/auth-fields.ts",
757 + "tier": "1",
758 + "toolkits": [],
759 + "timeoutSec": 120
760 + },
761 + {
762 + "id": "ts/tools/index",
763 + "lang": "ts",
764 + "pkg": "tools-example",
765 + "file": "ts/examples/tools/src/index.ts",
766 + "tier": "2",
767 + "toolkits": [
768 + "googledrive"
769 + ],
770 + "timeoutSec": 180,
771 + "ids": [
772 + "COMPOSIO_EXAMPLES_USER_ID"
773 + ]
774 + },
775 + {
776 + "id": "ts/tools/ai",
777 + "lang": "ts",
778 + "pkg": "tools-example",
779 + "file": "ts/examples/tools/src/ai.ts",
780 + "tier": "2",
781 + "env": [
782 + "OPENAI_API_KEY"
783 + ],
784 + "toolkits": [
785 + "googledrive"
786 + ],
787 + "timeoutSec": 180,
788 + "ids": [
789 + "COMPOSIO_EXAMPLES_USER_ID"
790 + ]
791 + },
792 + {
793 + "id": "ts/tools/tool-conversion",
794 + "lang": "ts",
795 + "pkg": "tools-example",
796 + "file": "ts/examples/tools/src/tool-conversion.ts",
797 + "tier": "1",
798 + "toolkits": [],
799 + "timeoutSec": 120
800 + },
801 + {
802 + "id": "ts/triggers/index",
803 + "lang": "ts",
804 + "pkg": "triggers-example",
805 + "file": "ts/examples/triggers/src/index.ts",
806 + "tier": "3",
807 + "readiness": "Subscribed to trigger events",
808 + "toolkits": [
809 + "gmail"
810 + ],
811 + "timeoutSec": 120
812 + },
813 + {
814 + "id": "ts/triggers/create",
815 + "lang": "ts",
816 + "pkg": "triggers-example",
817 + "file": "ts/examples/triggers/src/create.ts",
818 + "tier": "2",
819 + "ids": [
820 + "COMPOSIO_EXAMPLES_GMAIL_CONNECTED_ACCOUNT_ID",
821 + "COMPOSIO_EXAMPLES_USER_ID"
822 + ],
823 + "toolkits": [
824 + "gmail"
825 + ],
826 + "serial": true,
827 + "timeoutSec": 120
828 + },
829 + {
830 + "id": "ts/triggers/subscribe",
831 + "lang": "ts",
832 + "pkg": "triggers-example",
833 + "file": "ts/examples/triggers/src/subscribe.ts",
834 + "tier": "3",
835 + "ids": [
836 + "COMPOSIO_EXAMPLES_GMAIL_CONNECTED_ACCOUNT_ID",
837 + "COMPOSIO_EXAMPLES_USER_ID"
838 + ],
839 + "readiness": "Subscribed to triggers",
840 + "toolkits": [
841 + "gmail"
842 + ],
843 + "timeoutSec": 120
844 + },
845 + {
846 + "id": "ts/triggers/test",
847 + "lang": "ts",
848 + "pkg": "triggers-example",
849 + "file": "ts/examples/triggers/src/test.ts",
850 + "tier": "2",
851 + "toolkits": [
852 + "gmail"
853 + ],
854 + "timeoutSec": 120,
855 + "notes": "triage: entry not yet characterized"
856 + },
857 + {
858 + "id": "ts/triggers/trigger-types",
859 + "lang": "ts",
860 + "pkg": "triggers-example",
861 + "file": "ts/examples/triggers/src/trigger-types.ts",
862 + "tier": "1",
863 + "toolkits": [],
864 + "timeoutSec": 120
865 + },
866 + {
867 + "id": "ts/triggers/webhook-server",
868 + "lang": "ts",
869 + "pkg": "triggers-example",
870 + "file": "ts/examples/triggers/src/webhook-server.ts",
871 + "tier": "X",
872 + "reason": "external-tunnel"
873 + },
874 + {
875 + "id": "ts/vercel/ai",
876 + "lang": "ts",
877 + "pkg": "vercel-example",
878 + "file": "ts/examples/vercel/src/ai.ts",
879 + "tier": "1",
880 + "env": [
881 + "OPENAI_API_KEY"
882 + ],
883 + "toolkits": [
884 + "hackernews"
885 + ],
886 + "timeoutSec": 180
887 + },
888 + {
889 + "id": "ts/vercel/stream",
890 + "lang": "ts",
891 + "pkg": "vercel-example",
892 + "file": "ts/examples/vercel/src/stream.ts",
893 + "tier": "1",
894 + "env": [
895 + "OPENAI_API_KEY"
896 + ],
897 + "toolkits": [
898 + "hackernews"
899 + ],
900 + "timeoutSec": 180
901 + },
902 + {
903 + "id": "ts/vercel/tool-loop-agent",
904 + "lang": "ts",
905 + "pkg": "vercel-example",
906 + "file": "ts/examples/vercel/src/tool-loop-agent.ts",
907 + "tier": "1",
908 + "env": [
909 + "OPENAI_API_KEY"
910 + ],
911 + "toolkits": [
912 + "hackernews"
913 + ],
914 + "timeoutSec": 240
915 + },
916 + {
917 + "id": "ts/vercel/tool-router-ai",
918 + "lang": "ts",
919 + "pkg": "vercel-example",
920 + "file": "ts/examples/vercel/src/tool-router-ai.ts",
921 + "tier": "2",
922 + "env": [
923 + "OPENAI_API_KEY"
924 + ],
925 + "toolkits": [
926 + "gmail"
927 + ],
928 + "timeoutSec": 180
929 + },
930 + {
931 + "id": "ts/versioning/index",
932 + "lang": "ts",
933 + "pkg": "versioning-example",
934 + "file": "ts/examples/versioning/src/index.ts",
935 + "tier": "1",
936 + "toolkits": [
937 + "gmail",
938 + "googlecalendar"
939 + ],
940 + "timeoutSec": 120
941 + },
942 + {
943 + "id": "py/tools",
944 + "lang": "py",
945 + "file": "python/examples/tools.py",
946 + "tier": "2",
947 + "ids": [
948 + "COMPOSIO_EXAMPLES_GITHUB_CONNECTED_ACCOUNT_ID"
949 + ],
950 + "toolkits": [
951 + "github"
952 + ],
953 + "timeoutSec": 120
954 + },
955 + {
956 + "id": "py/toolkits",
957 + "lang": "py",
958 + "file": "python/examples/toolkits.py",
959 + "tier": "2",
960 + "toolkits": [
961 + "github"
962 + ],
963 + "serial": true,
964 + "timeoutSec": 120,
965 + "notes": "authorize() initiates OAuth state"
966 + },
967 + {
968 + "id": "py/auth_configs",
969 + "lang": "py",
970 + "file": "python/examples/auth_configs.py",
971 + "tier": "2",
972 + "toolkits": [],
973 + "serial": true,
974 + "timeoutSec": 120,
975 + "notes": "creates/updates/deletes org auth configs"
976 + },
977 + {
978 + "id": "py/connected_accounts",
979 + "lang": "py",
980 + "file": "python/examples/connected_accounts.py",
981 + "tier": "3",
982 + "readiness": "Visit this URL to authorize: https?://",
983 + "ids": [
984 + "COMPOSIO_EXAMPLES_GMAIL_AUTH_CONFIG_ID",
985 + "COMPOSIO_EXAMPLES_APIKEY_AUTH_CONFIG_ID",
986 + "COMPOSIO_EXAMPLES_APIKEY_PLACEHOLDER",
987 + "COMPOSIO_EXAMPLES_USER_ID"
988 + ],
989 + "toolkits": [
990 + "gmail"
991 + ],
992 + "timeoutSec": 90
993 + },
994 + {
995 + "id": "py/triggers",
996 + "lang": "py",
997 + "file": "python/examples/triggers.py",
998 + "tier": "3",
999 + "ids": [
1000 + "COMPOSIO_EXAMPLES_GMAIL_CONNECTED_ACCOUNT_ID",
1001 + "COMPOSIO_EXAMPLES_USER_ID"
1002 + ],
1003 + "readiness": "Subscribed|[Ll]istening for events",
1004 + "toolkits": [
1005 + "gmail"
1006 + ],
1007 + "timeoutSec": 300
1008 + },
1009 + {
1010 + "id": "py/modifiers",
1011 + "lang": "py",
1012 + "file": "python/examples/modifiers.py",
1013 + "tier": "X",
1014 + "env": [
1015 + "NOTION_API_KEY"
1016 + ],
1017 + "toolkits": [
1018 + "hackernews",
1019 + "notion"
1020 + ],
1021 + "reason": "dropped-provider",
1022 + "notes": "gemini/notion are out of scope for this goal by decision (2026-08-10)"
1023 + },
1024 + {
1025 + "id": "py/mcp",
1026 + "lang": "py",
1027 + "file": "python/examples/mcp_example.py",
1028 + "tier": "2",
1029 + "env": [
1030 + "OPENAI_API_KEY"
1031 + ],
1032 + "ids": [
1033 + "COMPOSIO_EXAMPLES_SLACK_AUTH_CONFIG_ID",
1034 + "COMPOSIO_EXAMPLES_USER_ID"
1035 + ],
1036 + "toolkits": [
1037 + "slack"
1038 + ],
1039 + "timeoutSec": 180,
1040 + "pyWith": [
1041 + "langchain-mcp-adapters",
1042 + "langgraph",
1043 + "langchain",
1044 + "langchain-openai"
1045 + ]
1046 + },
1047 + {
1048 + "id": "py/fastapi_app",
1049 + "lang": "py",
1050 + "file": "python/examples/fastapi_app.py",
1051 + "tier": "3",
1052 + "readiness": "Uvicorn running|Application startup",
1053 + "ids": [
1054 + "COMPOSIO_EXAMPLES_GMAIL_AUTH_CONFIG_ID",
1055 + "COMPOSIO_EXAMPLES_USER_ID"
1056 + ],
1057 + "pyWith": [
1058 + "fastapi",
1059 + "uvicorn"
1060 + ],
1061 + "timeoutSec": 120
1062 + },
1063 + {
1064 + "id": "py/custom_tools_agent_test",
1065 + "lang": "py",
1066 + "file": "python/examples/custom_tools_agent_test.py",
1067 + "tier": "2",
1068 + "env": [
1069 + "OPENAI_API_KEY"
1070 + ],
1071 + "toolkits": [
1072 + "gmail"
1073 + ],
1074 + "timeoutSec": 240,
1075 + "pyWith": [
1076 + "openai-agents>=0.19"
1077 + ]
1078 + },
1079 + {
1080 + "id": "py/experimental_tool_router",
1081 + "lang": "py",
1082 + "file": "python/examples/experimental_tool_router_example.py",
1083 + "tier": "3",
1084 + "readiness": "Redirect URL: https?://",
1085 + "toolkits": [
1086 + "github"
1087 + ],
1088 + "timeoutSec": 120
1089 + },
1090 + {
1091 + "id": "py/experimental_tool_router_advanced",
1092 + "lang": "py",
1093 + "file": "python/examples/experimental_tool_router_advanced_example.py",
1094 + "tier": "1",
1095 + "toolkits": [
1096 + "github",
1097 + "slack",
1098 + "linear",
1099 + "jira"
1100 + ],
1101 + "timeoutSec": 120,
1102 + "ids": [
1103 + "COMPOSIO_EXAMPLES_USER_ID",
1104 + "COMPOSIO_EXAMPLES_GITHUB_AUTH_CONFIG_ID",
1105 + "COMPOSIO_EXAMPLES_SLACK_AUTH_CONFIG_ID",
1106 + "COMPOSIO_EXAMPLES_GITHUB_CONNECTED_ACCOUNT_ID",
1107 + "COMPOSIO_EXAMPLES_SLACK_CONNECTED_ACCOUNT_ID"
1108 + ]
1109 + },
1110 + {
1111 + "id": "py/tool_router/authorize",
1112 + "lang": "py",
1113 + "file": "python/examples/tool_router/authorize.py",
1114 + "tier": "3",
1115 + "readiness": "Redirect URL: https?://",
1116 + "toolkits": [
1117 + "gmail"
1118 + ],
1119 + "timeoutSec": 90
1120 + },
1121 + {
1122 + "id": "py/tool_router/claude_agent",
1123 + "lang": "py",
1124 + "file": "python/examples/tool_router/claude_agent.py",
1125 + "tier": "2",
1126 + "env": [
1127 + "ANTHROPIC_API_KEY"
1128 + ],
1129 + "toolkits": [
1130 + "gmail"
1131 + ],
1132 + "pyWith": [
1133 + "claude-agent-sdk",
1134 + "composio-claude-agent-sdk",
1135 + "./python"
1136 + ],
1137 + "timeoutSec": 300,
1138 + "ids": [
1139 + "COMPOSIO_EXAMPLES_USER_ID"
1140 + ]
1141 + },
1142 + {
1143 + "id": "py/tool_router/crewai_agent",
1144 + "lang": "py",
1145 + "file": "python/examples/tool_router/crewai_agent.py",
1146 + "tier": "2",
1147 + "env": [
1148 + "OPENAI_API_KEY"
1149 + ],
1150 + "toolkits": [
1151 + "gmail"
1152 + ],
1153 + "pyWith": [
1154 + "crewai>=1.15"
1155 + ],
1156 + "timeoutSec": 300,
1157 + "ids": [
1158 + "COMPOSIO_EXAMPLES_USER_ID"
1159 + ]
1160 + },
1161 + {
1162 + "id": "py/tool_router/direct_tools_preset",
1163 + "lang": "py",
1164 + "file": "python/examples/tool_router/direct_tools_preset.py",
1165 + "tier": "1",
1166 + "env": [
1167 + "OPENAI_API_KEY"
1168 + ],
1169 + "toolkits": [
1170 + "hackernews"
1171 + ],
1172 + "timeoutSec": 180,
1173 + "pyWith": [
1174 + "openai-agents>=0.19"
1175 + ]
1176 + },
1177 + {
1178 + "id": "py/tool_router/preload",
1179 + "lang": "py",
1180 + "file": "python/examples/tool_router/preload.py",
1181 + "tier": "1",
1182 + "env": [
1183 + "OPENAI_API_KEY"
1184 + ],
1185 + "toolkits": [
1186 + "hackernews"
1187 + ],
1188 + "timeoutSec": 180,
1189 + "pyWith": [
1190 + "openai-agents>=0.19"
1191 + ]
1192 + },
1193 + {
1194 + "id": "py/tool_router/files",
1195 + "lang": "py",
1196 + "file": "python/examples/tool_router/files.py",
1197 + "tier": "1",
1198 + "env": [
1199 + "OPENAI_API_KEY"
1200 + ],
1201 + "toolkits": [],
1202 + "timeoutSec": 180
1203 + },
1204 + {
1205 + "id": "py/tool_router/langchain_agent",
1206 + "lang": "py",
1207 + "file": "python/examples/tool_router/langchain_agent.py",
1208 + "tier": "2",
1209 + "env": [
1210 + "OPENAI_API_KEY"
1211 + ],
1212 + "toolkits": [
1213 + "gmail"
1214 + ],
1215 + "pyWith": [
1216 + "langchain",
1217 + "langchain-openai",
1218 + "langchain-mcp-adapters"
1219 + ],
1220 + "timeoutSec": 240,
1221 + "ids": [
1222 + "COMPOSIO_EXAMPLES_USER_ID"
1223 + ]
1224 + },
1225 + {
1226 + "id": "py/tool_router/openai_agents",
1227 + "lang": "py",
1228 + "file": "python/examples/tool_router/openai_agents.py",
1229 + "tier": "2",
1230 + "env": [
1231 + "OPENAI_API_KEY"
1232 + ],
1233 + "toolkits": [
1234 + "gmail"
1235 + ],
1236 + "pyWith": [
1237 + "openai-agents>=0.19"
1238 + ],
1239 + "timeoutSec": 240
1240 + },
1241 + {
1242 + "id": "py/tool_router/tool_router_mcp",
1243 + "lang": "py",
1244 + "file": "python/examples/tool_router/tool_router_mcp.py",
1245 + "tier": "2",
1246 + "env": [
1247 + "OPENAI_API_KEY"
1248 + ],
1249 + "toolkits": [
1250 + "github",
1251 + "gmail"
1252 + ],
1253 + "timeoutSec": 240,
1254 + "pyWith": [
1255 + "openai-agents>=0.19"
1256 + ]
1257 + },
1258 + {
1259 + "id": "py/tool_router/session_update",
1260 + "lang": "py",
1261 + "file": "python/examples/tool_router/session_update.py",
1262 + "tier": "1",
1263 + "toolkits": [
1264 + "gmail",
1265 + "github"
1266 + ],
1267 + "timeoutSec": 120
1268 + },
1269 + {
1270 + "id": "py/tool_router/toolkits",
1271 + "lang": "py",
1272 + "file": "python/examples/tool_router/toolkits.py",
1273 + "tier": "1",
1274 + "toolkits": [],
1275 + "timeoutSec": 120
1276 + },
1277 + {
1278 + "id": "py/tool_router/tools",
1279 + "lang": "py",
1280 + "file": "python/examples/tool_router/tools.py",
1281 + "tier": "2",
1282 + "env": [
1283 + "OPENAI_API_KEY"
1284 + ],
1285 + "toolkits": [
1286 + "gmail"
1287 + ],
1288 + "timeoutSec": 180,
1289 + "pyWith": [
1290 + "openai-agents>=0.19"
1291 + ]
1292 + }
1293 + ]
1294 +}
harness/fixtures/always-fails.ts new
+3
@@ -0,0 +1,3 @@
1 +// Known-bad selftest fixture: must exit non-zero.
2 +console.error('fixture: failing on purpose');
3 +process.exit(1);
harness/fixtures/always_fails.py new
+6
@@ -0,0 +1,6 @@
1 +"""Known-bad selftest fixture: must exit non-zero."""
2 +
3 +import sys
4 +
5 +print("fixture: failing on purpose", file=sys.stderr)
6 +sys.exit(1)
harness/fixtures/trace-check.ts new
+7
@@ -0,0 +1,7 @@
1 +// Selftest fixture: makes one unauthenticated request against the backend so
2 +// the fetch shim must record a non-2xx composio line in COMPOSIO_TRACE_FILE.
3 +const base = process.env.COMPOSIO_BASE_URL ?? 'https://staging-backend.composio.dev';
4 +const res = await fetch(`${base}/api/v3/toolkits`, {
5 + headers: { 'x-api-key': 'selftest-invalid-key' },
6 +});
7 +console.log(`trace-check: status ${res.status}`);
harness/fixtures/trace_check.py new
+11
@@ -0,0 +1,11 @@
1 +"""Selftest fixture: one unauthenticated httpx request against the backend so
2 +the sitecustomize shim must record a non-2xx composio line."""
3 +
4 +import os
5 +
6 +import httpx
7 +
8 +base = os.environ.get("COMPOSIO_BASE_URL", "https://staging-backend.composio.dev")
9 +with httpx.Client() as client:
10 + response = client.get(f"{base}/api/v3/toolkits", headers={"x-api-key": "selftest-invalid-key"})
11 +print(f"trace-check: status {response.status_code}")
harness/llm-mock/aimock.json new
+5
@@ -0,0 +1,5 @@
1 +{
2 + "llm": {
3 + "fixtures": "./fixtures"
4 + }
5 +}
harness/llm-mock/fixtures/default.json new
+29
@@ -0,0 +1,29 @@
1 +{
2 + "$comment": "Deterministic LLM fixtures for --llm mock sweeps. One file so fixture order is explicit: specific matches first, catch-alls last (first match wins). Tool-call fixtures guard with hasToolResult:false so the follow-up turn falls through to the wrap-up fixture. Composio tool execution stays live; only the model is mocked.",
3 + "fixtures": [
4 + {
5 + "match": { "userMessage": "HackerNews user", "hasToolResult": false },
6 + "response": {
7 + "toolCalls": [
8 + { "name": "HACKERNEWS_GET_USER", "arguments": { "username": "pg" }, "id": "call_hn_user" }
9 + ]
10 + }
11 + },
12 + {
13 + "match": { "userMessage": "HackerNews", "hasToolResult": false },
14 + "response": {
15 + "toolCalls": [
16 + { "name": "HACKERNEWS_GET_TOP_STORIES", "arguments": { "size": 3 }, "id": "call_hn_top" }
17 + ]
18 + }
19 + },
20 + {
21 + "match": { "hasToolResult": true },
22 + "response": { "content": "Mock model wrap-up: summarized the tool result deterministically." }
23 + },
24 + {
25 + "match": {},
26 + "response": { "content": "Mock model response: no tool needed for this prompt." }
27 + }
28 + ]
29 +}
harness/parity.mjs new
+73
@@ -0,0 +1,73 @@
1 +#!/usr/bin/env node
2 +// Trace-parity comparator: node harness/parity.mjs <baseline-run-dir> <candidate-run-dir>
3 +// Per entry green in BOTH runs: the sets of distinct (method, path-template)
4 +// pairs must be equal, ignoring pairs allowlisted in parity-variance.json.
5 +// Prints a JSON report; exit 0 = all parities hold, 1 = mismatches.
6 +
7 +import { readFileSync, readdirSync, existsSync } from 'node:fs';
8 +import { join, resolve, dirname } from 'node:path';
9 +import { fileURLToPath } from 'node:url';
10 +
11 +const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
12 +const [baseDir, candDir] = process.argv.slice(2);
13 +if (!baseDir || !candDir) {
14 + console.error('usage: parity.mjs <baseline-run-dir> <candidate-run-dir>');
15 + process.exit(2);
16 +}
17 +
18 +const variance = JSON.parse(readFileSync(join(ROOT, 'parity-variance.json'), 'utf8'));
19 +const ignored = new Set((variance.entries ?? []).map((e) => e.pair));
20 +
21 +const loadRun = (dir) => {
22 + const results = readFileSync(join(dir, 'results.jsonl'), 'utf8')
23 + .trim()
24 + .split('\n')
25 + .filter(Boolean)
26 + .map((l) => JSON.parse(l));
27 + const byId = new Map();
28 + for (const r of results) byId.set(r.id, r);
29 + return byId;
30 +};
31 +
32 +const tracePairs = (dir, id) => {
33 + const file = join(dir, 'traces', `${id.replace(/[/]/g, '__')}.jsonl`);
34 + const pairs = new Set();
35 + if (!existsSync(file)) return pairs;
36 + for (const line of readFileSync(file, 'utf8').trim().split('\n').filter(Boolean)) {
37 + const rec = JSON.parse(line);
38 + if (rec.m && rec.p) pairs.add(`${rec.m} ${rec.p}`);
39 + }
40 + return pairs;
41 +};
42 +
43 +const baseline = loadRun(baseDir);
44 +const candidate = loadRun(candDir);
45 +
46 +const report = [];
47 +const ids = new Set([...baseline.keys(), ...candidate.keys()]);
48 +for (const id of [...ids].sort()) {
49 + const b = baseline.get(id);
50 + const c = candidate.get(id);
51 + if (!b || !c) {
52 + report.push({
53 + id,
54 + parity: false,
55 + reason: `missing ${b ? 'candidate' : 'baseline'} result`,
56 + });
57 + continue;
58 + }
59 + if (b.status !== 'green' || c.status !== 'green') {
60 + report.push({ id, parity: false, reason: `status baseline=${b.status} candidate=${c.status}` });
61 + continue;
62 + }
63 + const bp = tracePairs(baseDir, id);
64 + const cp = tracePairs(candDir, id);
65 + const onlyBaseline = [...bp].filter((p) => !cp.has(p) && !ignored.has(p));
66 + const onlyCandidate = [...cp].filter((p) => !bp.has(p) && !ignored.has(p));
67 + const match = onlyBaseline.length === 0 && onlyCandidate.length === 0;
68 + report.push({ id, parity: match, ...(match ? {} : { onlyBaseline, onlyCandidate }) });
69 +}
70 +
71 +const green = report.filter((r) => r.parity).length;
72 +console.log(JSON.stringify({ compared: report.length, parityGreen: green, ignoredPairs: ignored.size, entries: report }, null, 2));
73 +process.exit(report.length > 0 && report.every((r) => r.parity) ? 0 : 1);
harness/run.mjs new
+545
@@ -0,0 +1,545 @@
1 +#!/usr/bin/env node
2 +// Examples runner: sweeps the entrypoints in examples-manifest.json against
3 +// the live backend, recording per-entry results and Composio traces.
4 +//
5 +// node harness/run.mjs sweep --client baseline|candidate [--lang ts|py] [--ids a,b] [--tiers 1,2,3] [--llm live|mock]
6 +// node harness/run.mjs neg [--lang ts|py] [--ids a,b] [--sample N] [--seed S]
7 +// node harness/run.mjs selftest
8 +//
9 +// Exit codes: 0 ok · 1 findings (red entries / failed expectations) · 2 usage/env error.
10 +
11 +import { spawn } from 'node:child_process';
12 +import { mkdirSync, readFileSync, writeFileSync, appendFileSync, existsSync, rmSync } from 'node:fs';
13 +import { dirname, join, relative, resolve } from 'node:path';
14 +import { fileURLToPath } from 'node:url';
15 +import { requireStagingBaseUrl, STAGING_BASE_URL } from './staging-backend.mjs';
16 +
17 +const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
18 +const ARTIFACTS = join(ROOT, '.artifacts', 'examples-parity');
19 +const STAINLESS_TS_VERSION = '0.1.0-alpha.76';
20 +const AIMOCK_VERSION = '1.38.0';
21 +const AIMOCK_PORT = Number(process.env.AIMOCK_PORT ?? 4010);
22 +
23 +// Set by cmdSweep when --llm mock: LLM traffic goes to a local aimock server
24 +// and provider keys are overridden so a mock sweep can never spend tokens.
25 +let LLM_MOCK = false;
26 +
27 +const args = process.argv.slice(2);
28 +const cmd = args[0];
29 +const opt = (name, fallback) => {
30 + const i = args.indexOf(`--${name}`);
31 + return i >= 0 && args[i + 1] !== undefined ? args[i + 1] : fallback;
32 +};
33 +
34 +class HarnessError extends Error {
35 + constructor(message, exitCode = 2) {
36 + super(message);
37 + this.exitCode = exitCode;
38 + }
39 +}
40 +
41 +const fail = (msg, code = 2) => {
42 + throw new HarnessError(msg, code);
43 +};
44 +
45 +const loadManifest = () => {
46 + const manifest = JSON.parse(readFileSync(join(ROOT, 'examples-manifest.json'), 'utf8'));
47 + for (const e of manifest.entries) {
48 + if (!e.id || !e.lang || !e.file || !e.tier) fail(`manifest entry missing required fields: ${JSON.stringify(e)}`);
49 + if (e.lang === 'ts' && e.tier !== 'X' && !e.pkg) fail(`ts entry ${e.id} missing pkg`);
50 + if (e.tier === '3' && !e.readiness) fail(`tier-3 entry ${e.id} missing readiness regex`);
51 + }
52 + return manifest.entries;
53 +};
54 +
55 +const selectEntries = () => {
56 + const lang = opt('lang');
57 + const ids = opt('ids') ? opt('ids').split(',') : null;
58 + const tiers = (opt('tiers', '1,2,3')).split(',');
59 + return loadManifest().filter(
60 + (e) =>
61 + e.tier !== 'X' &&
62 + tiers.includes(e.tier) &&
63 + (!lang || e.lang === lang) &&
64 + (!ids || ids.includes(e.id))
65 + );
66 +};
67 +
68 +const baseUrl = () => {
69 + try {
70 + return requireStagingBaseUrl();
71 + } catch (error) {
72 + fail(error instanceof Error ? error.message : String(error));
73 + }
74 +};
75 +
76 +// Deterministic PRNG so --seed reproduces a negative-control sample.
77 +const mulberry32 = (a) => () => {
78 + a |= 0; a = (a + 0x6d2b79f5) | 0;
79 + let t = Math.imul(a ^ (a >>> 15), 1 | a);
80 + t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
81 + return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
82 +};
83 +
84 +const runId = () => new Date().toISOString().replace(/[-:T]/g, '').slice(0, 14) + '-' + Math.random().toString(36).slice(2, 8);
85 +
86 +/** Spawn one entry; resolve {exit, readinessMatched, timedOut, durationMs}. */
87 +const runEntry = (entry, env, { garbage = false } = {}) =>
88 + new Promise((resolveRun) => {
89 + const isTs = entry.lang === 'ts';
90 + let cmdline;
91 + if (isTs && entry.script) {
92 + cmdline = ['pnpm', '--filter', entry.pkg, 'run', entry.script];
93 + } else if (isTs) {
94 + // pnpm --filter runs with the example package as cwd; derive the package
95 + // dir from the entry file path (ts/examples/<dir>/src/...).
96 + const pkgDir = entry.file.split('/').slice(0, 3).join('/');
97 + const srcRel = relative(join(ROOT, pkgDir), join(ROOT, entry.file));
98 + cmdline = ['pnpm', '--filter', entry.pkg, 'exec', 'tsx', srcRel];
99 + } else {
100 + cmdline = ['uv', 'run', '--project', 'python'];
101 + for (const dep of entry.pyWith ?? []) cmdline.push('--with', dep);
102 + cmdline.push('python', join(ROOT, entry.file));
103 + }
104 +
105 + const child = spawn(cmdline[0], cmdline.slice(1), {
106 + cwd: ROOT,
107 + env,
108 + detached: true,
109 + stdio: ['ignore', 'pipe', 'pipe'],
110 + });
111 +
112 + const started = Date.now();
113 + const timeoutMs = (entry.timeoutSec ?? 120) * 1000;
114 + let readinessMatched = false;
115 + let timedOut = false;
116 + let settled = false;
117 + let outputTail = '';
118 + const readiness = entry.tier === '3' && entry.readiness ? new RegExp(entry.readiness) : null;
119 +
120 + const killTree = () => {
121 + try { process.kill(-child.pid, 'SIGTERM'); } catch { /* gone */ }
122 + setTimeout(() => { try { process.kill(-child.pid, 'SIGKILL'); } catch { /* gone */ } }, 10_000).unref();
123 + };
124 +
125 + const onChunk = (chunk) => {
126 + const text = chunk.toString();
127 + outputTail = (outputTail + text).slice(-4000);
128 + if (readiness && !readinessMatched && readiness.test(outputTail)) {
129 + readinessMatched = true;
130 + // Grace period so the in-flight backend call lands in the trace, then stop the listener.
131 + setTimeout(killTree, 3000).unref();
132 + }
133 + };
134 + child.stdout.on('data', onChunk);
135 + child.stderr.on('data', onChunk);
136 +
137 + const timer = setTimeout(() => {
138 + timedOut = true;
139 + killTree();
140 + }, timeoutMs);
141 +
142 + child.on('exit', (code, signal) => {
143 + if (settled) return;
144 + settled = true;
145 + clearTimeout(timer);
146 + resolveRun({
147 + exit: code ?? (signal ? 128 : 1),
148 + readinessMatched,
149 + timedOut,
150 + durationMs: Date.now() - started,
151 + outputTail: garbage ? undefined : outputTail.slice(-1500),
152 + });
153 + });
154 + child.on('error', () => {
155 + if (settled) return;
156 + settled = true;
157 + clearTimeout(timer);
158 + resolveRun({ exit: 127, readinessMatched: false, timedOut: false, durationMs: Date.now() - started });
159 + });
160 + });
161 +
162 +const isGreen = (entry, r) => {
163 + if (entry.tier === '3') return r.readinessMatched || (!r.timedOut && r.exit === 0);
164 + return !r.timedOut && r.exit === 0;
165 +};
166 +
167 +const buildEnv = (entry, runDir, { garbage = false } = {}) => {
168 + const traceFile = join(runDir, 'traces', `${entry.id.replace(/[/]/g, '__')}.jsonl`);
169 + mkdirSync(dirname(traceFile), { recursive: true });
170 + const env = { ...process.env };
171 + env.COMPOSIO_BASE_URL = baseUrl();
172 + env.COMPOSIO_TRACE_FILE = traceFile;
173 + env.COMPOSIO_LOG_LEVEL ??= 'error';
174 + // The outbound-email denylist is fixed by the shims; never overridable per run.
175 + delete env.COMPOSIO_TOOL_DENYLIST;
176 + if (entry.lang === 'ts') {
177 + env.NODE_OPTIONS = `${env.NODE_OPTIONS ?? ''} --import=file://${join(ROOT, 'harness', 'trace', 'register.mjs')}`.trim();
178 + } else {
179 + env.PYTHONPATH = `${join(ROOT, 'harness', 'trace-py')}${env.PYTHONPATH ? `:${env.PYTHONPATH}` : ''}`;
180 + }
181 + if (garbage) {
182 + env.COMPOSIO_API_KEY = 'nc-invalid-0000000000000000';
183 + env.OPENAI_API_KEY = 'nc-invalid';
184 + env.ANTHROPIC_API_KEY = 'nc-invalid';
185 + env.GEMINI_API_KEY = 'nc-invalid';
186 + env.NOTION_API_KEY = 'nc-invalid';
187 + } else if (LLM_MOCK) {
188 + const mock = `http://127.0.0.1:${AIMOCK_PORT}`;
189 + env.OPENAI_BASE_URL = `${mock}/v1`;
190 + // langchain/litellm read the older variable name.
191 + env.OPENAI_API_BASE = `${mock}/v1`;
192 + env.ANTHROPIC_BASE_URL = mock;
193 + env.OPENAI_API_KEY = 'aimock';
194 + env.ANTHROPIC_API_KEY = 'aimock';
195 + }
196 + return { env, traceFile };
197 +};
198 +
199 +const LLM_KEYS = new Set(['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GEMINI_API_KEY']);
200 +
201 +const missingEnv = (entry) => {
202 + const wanted = [...(entry.env ?? []), ...(entry.ids ?? [])].filter(
203 + (k) => !(LLM_MOCK && LLM_KEYS.has(k))
204 + );
205 + if (entry.backend !== false) wanted.unshift('COMPOSIO_API_KEY');
206 + return wanted.filter((k) => !process.env[k]);
207 +};
208 +
209 +/** Run entries with bounded concurrency; serial entries run last, alone. */
210 +const runPool = async (entries, runDir, { garbage = false, concurrency = 4 } = {}) => {
211 + const parallel = entries.filter((e) => !e.serial);
212 + const serial = entries.filter((e) => e.serial);
213 + const results = [];
214 + const worker = async (queue) => {
215 + for (;;) {
216 + const entry = queue.shift();
217 + if (!entry) return;
218 + if (LLM_MOCK && entry.llmMock === false) {
219 + results.push({ id: entry.id, tier: entry.tier, status: 'skipped', reason: 'llm-mock-unsupported' });
220 + console.log(` ~ ${entry.id} skipped (llm-mock-unsupported)`);
221 + continue;
222 + }
223 + const missing = garbage ? [] : missingEnv(entry);
224 + if (missing.length > 0) {
225 + results.push({ id: entry.id, tier: entry.tier, status: 'skipped', reason: `missing-env:${missing.join(',')}` });
226 + console.log(` ~ ${entry.id} skipped (${missing.join(',')})`);
227 + continue;
228 + }
229 + const { env, traceFile } = buildEnv(entry, runDir, { garbage });
230 + const r = await runEntry(entry, env, { garbage });
231 + // An entry that tripped the outbound-email guard is red no matter how it
232 + // exited: catching the refusal and continuing is not coverage.
233 + const blocked =
234 + existsSync(traceFile) && readFileSync(traceFile, 'utf8').includes('"s":"BLOCKED"');
235 + const green = !blocked && isGreen(entry, r);
236 + results.push({
237 + id: entry.id,
238 + tier: entry.tier,
239 + status: green ? 'green' : 'red',
240 + ...(blocked ? { reason: 'blocked-tool' } : {}),
241 + exit: r.exit,
242 + timedOut: r.timedOut,
243 + readinessMatched: r.readinessMatched,
244 + durationMs: r.durationMs,
245 + traceFile: relative(ROOT, traceFile),
246 + outputTail: green ? undefined : r.outputTail,
247 + });
248 + console.log(` ${green ? '✓' : '✗'} ${entry.id} (${(r.durationMs / 1000).toFixed(1)}s${r.timedOut ? ', timeout' : ''})`);
249 + }
250 + };
251 + const queue = [...parallel];
252 + await Promise.all(Array.from({ length: Math.min(concurrency, Math.max(queue.length, 1)) }, () => worker(queue)));
253 + for (const entry of serial) await worker([entry]);
254 + return results;
255 +};
256 +
257 +// --- candidate swap (TS) --------------------------------------------------
258 +
259 +const WORKSPACE_YAML = join(ROOT, 'pnpm-workspace.yaml');
260 +const LOCKFILE = join(ROOT, 'pnpm-lock.yaml');
261 +
262 +const sh = (cmdline, opts = {}) =>
263 + new Promise((resolveSh, rejectSh) => {
264 + const child = spawn(cmdline[0], cmdline.slice(1), { cwd: ROOT, stdio: ['ignore', 'pipe', 'pipe'], ...opts });
265 + let out = '';
266 + child.stdout.on('data', (c) => (out += c));
267 + child.stderr.on('data', (c) => (out += c));
268 + child.on('exit', (code) => (code === 0 ? resolveSh(out) : rejectSh(new Error(`${cmdline.join(' ')} → exit ${code}\n${out.slice(-2000)}`))));
269 + });
270 +
271 +const resolvedTsClientVersion = async () => {
272 + const out = await sh(['node', '-e', "console.log(JSON.parse(require('fs').readFileSync(require.resolve('@composio/client/package.json', {paths:[require('path').join(process.cwd(),'ts/packages/core')]}), 'utf8')).version)"]);
273 + return out.trim().split('\n').pop();
274 +};
275 +
276 +const applyTsCandidate = async (tarball) => {
277 + if (!existsSync(tarball)) fail(`COMPOSIO_CLIENT_TARBALL not found: ${tarball}`);
278 + const yaml = readFileSync(WORKSPACE_YAML, 'utf8');
279 + if (/^overrides:$/m.test(yaml)) {
280 + // Merge into the repo's existing overrides block (e.g. security pins);
281 + // restoreTsBaseline() writes back the exact pre-run snapshot either way.
282 + writeFileSync(WORKSPACE_YAML, yaml.replace(/^overrides:$/m, `overrides:\n '@composio/client': file:${tarball}`));
283 + } else {
284 + writeFileSync(WORKSPACE_YAML, `${yaml}\noverrides:\n '@composio/client': file:${tarball}\n`);
285 + }
286 + await sh(['pnpm', 'install', '--no-frozen-lockfile']);
287 + const version = await resolvedTsClientVersion();
288 + if (version === STAINLESS_TS_VERSION) {
289 + fail(`override did not take effect: @composio/client still resolves to ${version}`);
290 + }
291 + console.log(`candidate @composio/client resolved: ${version}`);
292 + await sh(['pnpm', 'run', 'build:packages']);
293 +};
294 +
295 +const snapshotFiles = (paths) => new Map(paths.map((path) => [path, readFileSync(path)]));
296 +
297 +const restoreFiles = (snapshot) => {
298 + for (const [path, contents] of snapshot) writeFileSync(path, contents);
299 +};
300 +
301 +const snapshotTsBaseline = () => snapshotFiles([WORKSPACE_YAML, LOCKFILE]);
302 +
303 +const restoreTsBaseline = async (snapshot) => {
304 + restoreFiles(snapshot);
305 + await sh(['pnpm', 'install', '--frozen-lockfile']);
306 +};
307 +
308 +const verifyPyCandidate = async (wheel) => {
309 + if (!existsSync(wheel)) fail(`COMPOSIO_CLIENT_WHEEL not found: ${wheel}`);
310 + const out = await sh(['uv', 'run', '--project', 'python', '--with', wheel, 'python', '-c', 'import composio_client, importlib.metadata; print(importlib.metadata.version("composio-client"))']);
311 + const version = out.trim().split('\n').pop();
312 + if (!version.startsWith('2.')) {
313 + fail(`python candidate did not take effect (resolved ${version}); Stage B (SDK migration) must land first`);
314 + }
315 + return version;
316 +};
317 +
318 +// --- llm mock ---------------------------------------------------------------
319 +
320 +/** Start aimock serving harness/llm-mock/fixtures; resolve once it answers. */
321 +const startLlmMock = async (runDir) => {
322 + const mockDir = join(ROOT, 'harness', 'llm-mock');
323 + const logFile = join(runDir, 'aimock.log');
324 + const child = spawn(
325 + 'npx',
326 + ['-y', `@copilotkit/aimock@${AIMOCK_VERSION}`, '-c', 'aimock.json', '-p', String(AIMOCK_PORT), '-h', '127.0.0.1'],
327 + { cwd: mockDir, detached: true, stdio: ['ignore', 'pipe', 'pipe'] }
328 + );
329 + child.stdout.on('data', (c) => appendFileSync(logFile, c));
330 + child.stderr.on('data', (c) => appendFileSync(logFile, c));
331 +
332 + // First run may download the package; allow a generous readiness window.
333 + const deadline = Date.now() + 120_000;
334 + for (;;) {
335 + try {
336 + const res = await fetch(`http://127.0.0.1:${AIMOCK_PORT}/v1/models`);
337 + if (res.ok) break;
338 + } catch { /* not up yet */ }
339 + if (Date.now() > deadline) {
340 + try { process.kill(-child.pid, 'SIGKILL'); } catch { /* gone */ }
341 + fail(`aimock did not become ready on :${AIMOCK_PORT} (see ${relative(ROOT, logFile)})`);
342 + }
343 + await new Promise((r) => setTimeout(r, 500));
344 + }
345 + console.log(`aimock ${AIMOCK_VERSION} ready on :${AIMOCK_PORT}`);
346 + return () => {
347 + try { process.kill(-child.pid, 'SIGTERM'); } catch { /* gone */ }
348 + };
349 +};
350 +
351 +// --- commands ---------------------------------------------------------------
352 +
353 +const cmdSweep = async () => {
354 + const client = opt('client', 'baseline');
355 + if (!['baseline', 'candidate'].includes(client)) fail(`--client must be baseline|candidate`);
356 + const llm = opt('llm', 'live');
357 + if (!['live', 'mock'].includes(llm)) fail(`--llm must be live|mock`);
358 + LLM_MOCK = llm === 'mock';
359 + const entries = selectEntries();
360 + if (entries.length === 0) fail('no entries selected');
361 + if (!process.env.COMPOSIO_API_KEY) fail('COMPOSIO_API_KEY is required for a sweep (disposable examples-project key)', 2);
362 +
363 + const id = `${runId()}-${client}${LLM_MOCK ? '-mock' : ''}`;
364 + const runDir = join(ARTIFACTS, id);
365 + mkdirSync(join(runDir, 'traces'), { recursive: true });
366 +
367 + let tsBaselineSnapshot;
368 + let stopLlmMock;
369 + const pyWheel = process.env.COMPOSIO_CLIENT_WHEEL;
370 + try {
371 + if (LLM_MOCK) stopLlmMock = await startLlmMock(runDir);
372 + if (client === 'candidate') {
373 + const wantTs = entries.some((e) => e.lang === 'ts');
374 + const wantPy = entries.some((e) => e.lang === 'py');
375 + if (wantTs) {
376 + if (!process.env.COMPOSIO_CLIENT_TARBALL) fail('candidate sweep with ts entries needs COMPOSIO_CLIENT_TARBALL');
377 + tsBaselineSnapshot = snapshotTsBaseline();
378 + await applyTsCandidate(process.env.COMPOSIO_CLIENT_TARBALL);
379 + }
380 + if (wantPy) {
381 + if (!pyWheel) fail('candidate sweep with py entries needs COMPOSIO_CLIENT_WHEEL');
382 + const v = await verifyPyCandidate(pyWheel);
383 + console.log(`candidate composio-client (py) resolved: ${v}`);
384 + for (const e of entries) if (e.lang === 'py') e.pyWith = [...(e.pyWith ?? []), pyWheel];
385 + }
386 + }
387 +
388 + console.log(`sweep ${id}: ${entries.length} entries → ${runDir}`);
389 + const results = await runPool(entries, runDir, { concurrency: Number(opt('concurrency', '4')) });
390 + for (const r of results) appendFileSync(join(runDir, 'results.jsonl'), `${JSON.stringify(r)}\n`);
391 + const counts = {
392 + green: results.filter((r) => r.status === 'green').length,
393 + red: results.filter((r) => r.status === 'red').length,
394 + skipped: results.filter((r) => r.status === 'skipped').length,
395 + };
396 + writeFileSync(
397 + join(runDir, 'summary.json'),
398 + JSON.stringify({ runId: id, client, llm, baseUrl: baseUrl(), counts, startedAt: new Date().toISOString() }, null, 2)
399 + );
400 + console.log(`sweep ${id}: ${counts.green} green / ${counts.red} red / ${counts.skipped} skipped`);
401 + process.exitCode = counts.red > 0 ? 1 : 0;
402 + } finally {
403 + if (stopLlmMock) stopLlmMock();
404 + if (tsBaselineSnapshot) await restoreTsBaseline(tsBaselineSnapshot);
405 + }
406 +};
407 +
408 +const cmdNeg = async () => {
409 + let entries = selectEntries().filter((e) => e.backend !== false);
410 + const sample = opt('sample');
411 + if (sample) {
412 + const rand = mulberry32(Number(opt('seed', '42')));
413 + entries = [...entries].sort(() => rand() - 0.5).slice(0, Number(sample));
414 + }
415 + const id = `${runId()}-neg`;
416 + const runDir = join(ARTIFACTS, id);
417 + mkdirSync(join(runDir, 'traces'), { recursive: true });
418 + console.log(`negative controls ${id}: ${entries.length} entries (garbage credentials; expectation: red)`);
419 + const results = await runPool(entries, runDir, { garbage: true, concurrency: Number(opt('concurrency', '6')) });
420 + const swallowers = results.filter((r) => r.status === 'green').map((r) => r.id);
421 + for (const r of results) appendFileSync(join(runDir, 'results.jsonl'), `${JSON.stringify(r)}\n`);
422 + writeFileSync(join(runDir, 'summary.json'), JSON.stringify({ runId: id, mode: 'neg', total: results.length, swallowers }, null, 2));
423 + if (swallowers.length > 0) {
424 + console.log(`NEGATIVE-CONTROL FAILURES (green under garbage credentials — these swallow errors):`);
425 + for (const s of swallowers) console.log(` ✗ ${s}`);
426 + process.exitCode = 1;
427 + } else {
428 + console.log('all negative controls red, as expected');
429 + }
430 +};
431 +
432 +const cmdSelftest = async () => {
433 + const runDir = join(ARTIFACTS, `${runId()}-selftest`);
434 + mkdirSync(join(runDir, 'traces'), { recursive: true });
435 + let failures = 0;
436 + const check = (name, ok) => {
437 + console.log(` ${ok ? '✓' : '✗'} ${name}`);
438 + if (!ok) failures += 1;
439 + };
440 +
441 + // 0. backend safety: the harness accepts only the canonical staging root.
442 + check('staging backend is accepted', requireStagingBaseUrl() === STAGING_BASE_URL);
443 + let productionRejected = false;
444 + try {
445 + requireStagingBaseUrl('https://backend.composio.dev');
446 + } catch {
447 + productionRejected = true;
448 + }
449 + check('production backend is rejected', productionRejected);
450 +
451 + let cleanupRan = false;
452 + let cleanupError;
453 + try {
454 + try {
455 + fail('expected selftest failure');
456 + } finally {
457 + cleanupRan = true;
458 + }
459 + } catch (error) {
460 + cleanupError = error;
461 + }
462 + check(
463 + 'usage failures unwind through cleanup',
464 + cleanupRan && cleanupError instanceof HarnessError
465 + );
466 +
467 + const cleanupFixture = join(runDir, 'cleanup-fixture.txt');
468 + writeFileSync(cleanupFixture, 'user contents');
469 + const cleanupSnapshot = snapshotFiles([cleanupFixture]);
470 + writeFileSync(cleanupFixture, 'candidate contents');
471 + restoreFiles(cleanupSnapshot);
472 + check('cleanup restores pre-run file contents', readFileSync(cleanupFixture, 'utf8') === 'user contents');
473 +
474 + // 1. known-good: error-handling-demo (no backend, no keys).
475 + const good = loadManifest().find((e) => e.id === 'ts/error-handling-demo/index');
476 + const { env: goodEnv } = buildEnv(good, runDir);
477 + delete goodEnv.COMPOSIO_API_KEY;
478 + const goodRes = await runEntry(good, goodEnv);
479 + check('known-good (error-handling-demo) exits 0', goodRes.exit === 0);
480 +
481 + // 2. known-bad fixtures: must exit non-zero in both languages.
482 + const badTsOut = await sh(['pnpm', 'exec', 'tsx', 'harness/fixtures/always-fails.ts']).then(() => 0).catch(() => 1);
483 + check('known-bad ts fixture exits non-zero', badTsOut === 1);
484 + const badPy = await sh(['uv', 'run', '--project', 'python', 'python', 'harness/fixtures/always_fails.py']).then(() => 0).catch(() => 1);
485 + check('known-bad py fixture exits non-zero', badPy === 1);
486 +
487 + // 3. trace shims: an unauthenticated backend request must appear in the trace.
488 + const tsTrace = join(runDir, 'traces', 'shim-ts.jsonl');
489 + await sh(['pnpm', 'exec', 'tsx', 'harness/fixtures/trace-check.ts'], {
490 + env: { ...process.env, COMPOSIO_TRACE_FILE: tsTrace, COMPOSIO_BASE_URL: baseUrl(), NODE_OPTIONS: `--import=file://${join(ROOT, 'harness', 'trace', 'register.mjs')}` },
491 + }).catch(() => {});
492 + const tsLines = existsSync(tsTrace) ? readFileSync(tsTrace, 'utf8').trim().split('\n').filter(Boolean) : [];
493 + check('ts fetch shim recorded a composio call', tsLines.some((l) => JSON.parse(l).p?.includes('/api/')));
494 +
495 + const pyTrace = join(runDir, 'traces', 'shim-py.jsonl');
496 + await sh(['uv', 'run', '--project', 'python', 'python', 'harness/fixtures/trace_check.py'], {
497 + env: { ...process.env, COMPOSIO_TRACE_FILE: pyTrace, COMPOSIO_BASE_URL: baseUrl(), PYTHONPATH: join(ROOT, 'harness', 'trace-py') },
498 + }).catch(() => {});
499 + const pyLines = existsSync(pyTrace) ? readFileSync(pyTrace, 'utf8').trim().split('\n').filter(Boolean) : [];
500 + check('py httpx shim recorded a composio call', pyLines.some((l) => JSON.parse(l).p?.includes('/api/')));
501 +
502 + // 4. parity must compare at least one entry and reject missing candidates.
503 + const parityBase = join(runDir, 'parity-base');
504 + const parityCandidate = join(runDir, 'parity-candidate');
505 + for (const dir of [parityBase, parityCandidate]) {
506 + mkdirSync(join(dir, 'traces'), { recursive: true });
507 + writeFileSync(join(dir, 'results.jsonl'), `${JSON.stringify({ id: 'fixture', status: 'green' })}\n`);
508 + writeFileSync(
509 + join(dir, 'traces', 'fixture.jsonl'),
510 + `${JSON.stringify({ m: 'GET', p: '/api/v3/toolkits', s: '2xx' })}\n`
511 + );
512 + }
513 + const parityMatches = await sh(['node', 'harness/parity.mjs', parityBase, parityCandidate])
514 + .then(() => true)
515 + .catch(() => false);
516 + check('parity accepts matching runs', parityMatches);
517 + writeFileSync(join(parityCandidate, 'results.jsonl'), '');
518 + const missingCandidateRejected = await sh(['node', 'harness/parity.mjs', parityBase, parityCandidate])
519 + .then(() => false)
520 + .catch(() => true);
521 + check('parity rejects missing candidate entries', missingCandidateRejected);
522 +
523 + rmSync(runDir, { recursive: true, force: true });
524 + console.log(failures === 0 ? 'selftest: PASS' : `selftest: FAIL (${failures})`);
525 + process.exitCode = failures === 0 ? 0 : 1;
526 +};
527 +
528 +try {
529 + switch (cmd) {
530 + case 'sweep':
531 + await cmdSweep();
532 + break;
533 + case 'neg':
534 + await cmdNeg();
535 + break;
536 + case 'selftest':
537 + await cmdSelftest();
538 + break;
539 + default:
540 + fail('usage: run.mjs <sweep|neg|selftest> [options]');
541 + }
542 +} catch (error) {
543 + console.error(`run.mjs: ${error instanceof Error ? error.message : String(error)}`);
544 + process.exitCode = error instanceof HarnessError ? error.exitCode : 1;
545 +}
harness/staging-backend.mjs new
+25
@@ -0,0 +1,25 @@
1 +export const STAGING_BASE_URL = 'https://staging-backend.composio.dev';
2 +
3 +export const requireStagingBaseUrl = (
4 + value = process.env.COMPOSIO_BASE_URL ?? STAGING_BASE_URL
5 +) => {
6 + let url;
7 + try {
8 + url = new URL(value);
9 + } catch {
10 + throw new Error(`refusing invalid COMPOSIO_BASE_URL: ${value}`);
11 + }
12 +
13 + const isStagingRoot =
14 + url.origin === STAGING_BASE_URL &&
15 + url.pathname === '/' &&
16 + !url.search &&
17 + !url.hash &&
18 + !url.username &&
19 + !url.password;
20 + if (!isStagingRoot) {
21 + throw new Error(`refusing non-staging COMPOSIO_BASE_URL: ${value}`);
22 + }
23 +
24 + return STAGING_BASE_URL;
25 +};
harness/trace-py/sitecustomize.py new
+108
@@ -0,0 +1,108 @@
1 +"""Harness-owned httpx tracer, loaded via PYTHONPATH (sitecustomize hook).
2 +
3 +Examples must never reference COMPOSIO_TRACE_FILE themselves (lint-enforced).
4 +"""
5 +
6 +import json
7 +import os
8 +import re
9 +import sys
10 +from urllib.parse import urlsplit
11 +
12 +_TRACE = os.environ.get("COMPOSIO_TRACE_FILE")
13 +_STAGING_BASE_URL = "https://staging-backend.composio.dev"
14 +
15 +if _TRACE:
16 + _base_url = os.environ.get("COMPOSIO_BASE_URL", _STAGING_BASE_URL)
17 + _parsed_base_url = urlsplit(_base_url)
18 + if not (
19 + _parsed_base_url.scheme == "https"
20 + and _parsed_base_url.hostname == "staging-backend.composio.dev"
21 + and _parsed_base_url.path in {"", "/"}
22 + and not _parsed_base_url.query
23 + and not _parsed_base_url.fragment
24 + and not _parsed_base_url.username
25 + and not _parsed_base_url.password
26 + ):
27 + print(
28 + f"sitecustomize: refusing non-staging COMPOSIO_BASE_URL: {_base_url}",
29 + file=sys.stderr,
30 + )
31 + os._exit(78)
32 +
33 + try:
34 + import httpx
35 +
36 + _LLM_HOSTS = {
37 + "api.openai.com",
38 + "api.anthropic.com",
39 + "generativelanguage.googleapis.com",
40 + }
41 + _BACKEND_HOST = _parsed_base_url.hostname
42 + _ID_SEG = re.compile(
43 + r"^(ca_|ac_|ti_|tr_|trs_|sess_|auth_|req_|proj_|org_)[\w-]+$"
44 + r"|^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"
45 + r"|^\d+$",
46 + re.IGNORECASE,
47 + )
48 +
49 + def _template(path: str) -> str:
50 + return "/".join("{id}" if _ID_SEG.match(seg) else seg for seg in path.split("/"))
51 +
52 + def _record(obj) -> None:
53 + try:
54 + with open(_TRACE, "a") as f:
55 + f.write(json.dumps(obj) + "\n")
56 + except OSError:
57 + pass
58 +
59 + def _log(request, response) -> None:
60 + host = request.url.host
61 + if host == _BACKEND_HOST:
62 + status = f"{response.status_code // 100}xx" if response is not None else "ERR"
63 + _record({"m": request.method.upper(), "p": _template(request.url.path), "s": status})
64 + elif host in _LLM_HOSTS:
65 + _record({"llm": host})
66 +
67 + # Outbound-email guard: tool executions matching the denylist are
68 + # refused at the transport, never forwarded to the backend.
69 + _DENY = re.compile(
70 + os.environ.get(
71 + "COMPOSIO_TOOL_DENYLIST",
72 + r"GMAIL_SEND|GMAIL_REPLY|SEND_EMAIL|SEND_DRAFT|OUTLOOK[A-Z_]*SEND",
73 + ),
74 + re.IGNORECASE,
75 + )
76 +
77 + def _guard(request) -> None:
78 + if request.url.host == _BACKEND_HOST and _DENY.search(request.url.path):
79 + _record({"m": request.method.upper(), "p": _template(request.url.path), "s": "BLOCKED"})
80 + raise RuntimeError(f"harness: outbound-email tool execution blocked ({request.url.path})")
81 +
82 + _orig_send = httpx.Client.send
83 +
84 + def _send(self, request, *args, **kwargs):
85 + _guard(request)
86 + response = None
87 + try:
88 + response = _orig_send(self, request, *args, **kwargs)
89 + return response
90 + finally:
91 + _log(request, response)
92 +
93 + httpx.Client.send = _send
94 +
95 + _orig_send_async = httpx.AsyncClient.send
96 +
97 + async def _send_async(self, request, *args, **kwargs):
98 + _guard(request)
99 + response = None
100 + try:
101 + response = await _orig_send_async(self, request, *args, **kwargs)
102 + return response
103 + finally:
104 + _log(request, response)
105 +
106 + httpx.AsyncClient.send = _send_async
107 + except Exception: # noqa: BLE001 - tracing must never break the example
108 + pass
harness/trace/register.mjs new
+73
@@ -0,0 +1,73 @@
1 +// Harness-owned fetch tracer. Injected via NODE_OPTIONS=--import; examples
2 +// must never reference COMPOSIO_TRACE_FILE themselves (lint-enforced).
3 +import { requireStagingBaseUrl } from '../staging-backend.mjs';
4 +
5 +const traceFile = process.env.COMPOSIO_TRACE_FILE;
6 +
7 +if (traceFile) {
8 + const { appendFileSync } = await import('node:fs');
9 +
10 + const LLM_HOSTS = new Set([
11 + 'api.openai.com',
12 + 'api.anthropic.com',
13 + 'generativelanguage.googleapis.com',
14 + ]);
15 +
16 + const backendHost = new URL(requireStagingBaseUrl()).hostname;
17 +
18 + const ID_SEG =
19 + /^(ca_|ac_|ti_|tr_|trs_|sess_|auth_|req_|proj_|org_)[\w-]+$|^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$|^\d+$/i;
20 + const template = (path) =>
21 + path
22 + .split('/')
23 + .map((seg) => (ID_SEG.test(seg) ? '{id}' : seg))
24 + .join('/');
25 +
26 + const record = (obj) => {
27 + try {
28 + appendFileSync(traceFile, `${JSON.stringify(obj)}\n`);
29 + } catch {
30 + // never break the example over tracing
31 + }
32 + };
33 +
34 + // Outbound-email guard: tool executions matching the denylist are refused at
35 + // the transport, never forwarded to the backend.
36 + const DENY = new RegExp(process.env.COMPOSIO_TOOL_DENYLIST ?? 'GMAIL_SEND|GMAIL_REPLY|SEND_EMAIL|SEND_DRAFT|OUTLOOK[A-Z_]*SEND', 'i');
37 +
38 + const origFetch = globalThis.fetch;
39 + globalThis.fetch = async function tracedFetch(input, init) {
40 + let url;
41 + try {
42 + url = new URL(typeof input === 'string' || input instanceof URL ? input : input.url);
43 + } catch {
44 + return origFetch.call(this, input, init);
45 + }
46 + const method = (
47 + init?.method ??
48 + (typeof input === 'object' && input !== null && 'method' in input ? input.method : 'GET') ??
49 + 'GET'
50 + ).toUpperCase();
51 +
52 + if (url.hostname === backendHost && DENY.test(url.pathname)) {
53 + record({ m: method, p: template(url.pathname), s: 'BLOCKED' });
54 + throw new Error(`harness: outbound-email tool execution blocked (${url.pathname})`);
55 + }
56 +
57 + let res;
58 + try {
59 + res = await origFetch.call(this, input, init);
60 + return res;
61 + } finally {
62 + if (url.hostname === backendHost) {
63 + record({
64 + m: method,
65 + p: template(url.pathname),
66 + s: res ? `${Math.floor(res.status / 100)}xx` : 'ERR',
67 + });
68 + } else if (LLM_HOSTS.has(url.hostname)) {
69 + record({ llm: url.hostname });
70 + }
71 + }
72 + };
73 +}
parity-variance.json new
+3
@@ -0,0 +1,3 @@
1 +{
2 + "entries": []
3 +}
scripts/examples-provision.mjs
+11 -22
@@ -30,34 +30,23 @@
30 // values are ever printed. The API-key demo value stored for serpapi is a
31 // deliberately fake placeholder, not a real key.
32
33 -const BASE_URL = process.env.COMPOSIO_BASE_URL ?? 'https://backend.composio.dev';
33 +import { requireStagingBaseUrl } from '../harness/staging-backend.mjs';
34 +
35 +let BASE_URL;
36 +try {
37 + BASE_URL = requireStagingBaseUrl();
38 +} catch (error) {
39 + console.error(error instanceof Error ? error.message : String(error));
40 + process.exit(1);
41 +}
42 const API_KEY = process.env.COMPOSIO_API_KEY;
43 const USER_ID = process.env.COMPOSIO_EXAMPLES_USER_ID ?? 'examples';
44 const INITIATE_MISSING = process.argv.includes('--initiate-missing');
45 const GC = process.argv.includes('--gc');
46 const DRY_RUN = process.argv.includes('--dry-run');
47
40 -// The examples only ever run against the production backend with the
41 -// disposable project key. Allowlist the Composio hosts rather than denylisting
42 -// loopback spellings: the API key travels in every request, so anything that is
43 -// not a Composio backend must be refused, not just localhost.
44 -const { baseHost, baseProtocol } = (() => {
45 - try {
46 - const url = new URL(BASE_URL);
47 - return { baseHost: url.hostname, baseProtocol: url.protocol };
48 - } catch {
49 - return { baseHost: null, baseProtocol: null };
50 - }
51 -})();
52 -if (!baseHost || !(baseHost === 'composio.dev' || baseHost.endsWith('.composio.dev'))) {
53 - console.error(`refusing non-Composio COMPOSIO_BASE_URL: ${BASE_URL}`);
54 - process.exit(1);
55 -}
56 -if (baseProtocol !== 'https:') {
57 - // The API key is sent on every request. Plain http would put it on the wire.
58 - console.error(`refusing non-https COMPOSIO_BASE_URL: ${BASE_URL}`);
59 - process.exit(1);
60 -}
48 +// The examples only ever run against staging with a disposable project key.
49 +// requireStagingBaseUrl refuses every other host before the key is read or sent.
50 if (!API_KEY) {
51 console.error('COMPOSIO_API_KEY is required (dedicated examples-project key)');
52 process.exit(1);