@devries-cameron20 / composio Mirror · read-only / commits / 5b898e795
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 14:45 UTC

fix(harness): unblock python examples parity runs (#4228)

This PR: - unpins the examples harness from the staging host: `COMPOSIO_BASE_URL` now selects the backend and still defaults to staging, with the structural checks kept (https, bare root, no path/query/fragment/credentials) — `examples-live.yml` sets staging explicitly, so CI is unchanged - renames `harness/staging-backend.mjs` to `harness/backend-url.mjs` and `requireStagingBaseUrl` to `resolveBackendBaseUrl`, and relaxes the matching hard-exit in `harness/trace-py/sitecustomize.py` - fixes the Python candidate swap: `relaxPyClientPin()` drops the project's exact `composio-client==` pin for the duration of a candidate sweep, so `COMPOSIO_CLIENT_WHEEL` can win over it, and `python/pyproject.toml` + `uv.lock` are restored from a pre-run snapshot afterwards - adds `harness/README.md` covering backend selection, the candidate-swap flow for both languages, and how to tell a real parity pass from a vacuous one - ignores `.artifacts/`, where every sweep writes its results, traces, and downloaded client - extends `harness/run.mjs selftest` with eleven checks over the new backend-URL behaviour and the pin relaxation; the suite passes locally ## Context Found while producing a Python examples parity run for the `composio-client` 2.x bump, which lives on a separate branch. Both problems blocked that run and neither is specific to it. The staging pin made a comparison against any other project impossible without editing the harness in place. The Python candidate swap was worse than blocking, because it failed quietly. A candidate sweep layers the wheel onto every entry with `uv run --with`; seven of the twenty-two Python entries also install the local `./python` project through `pyWith`, and uv cannot satisfy that project's exact pin and the candidate wheel together: ``` No solution found when resolving `--with` dependencies: Because only composio==0.20.0 is available and composio==0.20.0 depends on composio-client==<pinned>, [...] your requirements are unsatisfiable. ``` Those entries went red in under a tenth of a second for a packaging reason rather than a client one, and `parity.mjs` only compares entries green on both sides — so the comparison silently shrank to fifteen entries and still reported a clean pass. With the pin relaxed, the same sweep runs 22/22 green and compares all twenty-two. https://claude.ai/code/session_015Bx2BaSwuq2cxgJSRtAxag

Alberto Schiabel committed Aug 24, 2026 at 18:55 UTC 5b898e7957c338c508409e26271546e2bf71fbac
8 files changed +209 -44
.gitignore
+3
@@ -75,3 +75,6 @@ docs/.workflow-data/*
75 # Local agent / tooling state (regenerated per machine)
76 .pi/
77 .playwright-mcp/
78 +
79 +# Examples harness sweep output (results, traces, candidate client artifacts)
80 +.artifacts/
harness/README.md new
+82
@@ -0,0 +1,82 @@
1 +# Examples harness
2 +
3 +Runs the entrypoints inventoried in `examples-manifest.json` against a live
4 +Composio backend, records every backend call each entry makes, and compares two
5 +runs call-for-call.
6 +
7 +- `harness/run.mjs` — sweeps entries, writes `results.jsonl` + one trace file per
8 + entry under `.artifacts/examples-parity/<run-id>/`.
9 +- `harness/parity.mjs` — compares two run directories.
10 +- `harness/trace/register.mjs`, `harness/trace-py/sitecustomize.py` — the fetch
11 + and httpx tracers, injected by the runner. Examples never reference
12 + `COMPOSIO_TRACE_FILE` themselves.
13 +- `scripts/examples-provision.mjs` — provisions the auth configs and connected
14 + accounts the tier-2/3 entries need, and prints them as `COMPOSIO_EXAMPLES_*`
15 + exports.
16 +
17 +## Backend selection
18 +
19 +`COMPOSIO_BASE_URL` picks the backend and defaults to staging. Any bare `https`
20 +root is accepted; a URL carrying a path, query, fragment, or embedded
21 +credentials is refused, because the tracers pin the backend host and the
22 +provisioner appends its own paths. `.github/workflows/examples-live.yml` sets
23 +staging explicitly, so CI is unaffected by the default.
24 +
25 +Point this at a project whose data you are willing to have the examples touch.
26 +`scripts/examples-provision.mjs --gc` deletes examples-owned resources older
27 +than 24h across the whole project — never run it against a project you care
28 +about. The outbound-email denylist is enforced in the tracers and is not
29 +overridable per run; `--llm mock` additionally keeps model traffic on a local
30 +`aimock` server so no agent can decide to write something.
31 +
32 +## Comparing a client bump
33 +
34 +A parity run is two sweeps over the same entry ids, one on each side of the
35 +change, compared by traced `(method, path-template)` pairs.
36 +
37 +```bash
38 +# Provisioned ids for the project you are sweeping. Capture, then eval — a
39 +# failed provisioning run must not be swallowed.
40 +out=$(node scripts/examples-provision.mjs) && eval "$out"
41 +
42 +# Baseline: the pinned client, from a checkout of the base branch.
43 +node harness/run.mjs sweep --client baseline --lang py --llm mock --ids "$IDS"
44 +
45 +# Candidate: the same entries with the candidate client swapped in.
46 +COMPOSIO_CLIENT_WHEEL=/abs/path/composio_client-<version>-py3-none-any.whl \
47 + node harness/run.mjs sweep --client candidate --lang py --llm mock --ids "$IDS"
48 +
49 +node harness/parity.mjs <baseline-run-dir> <candidate-run-dir>
50 +```
51 +
52 +Pass `--ids` explicitly rather than relying on the default selection, so both
53 +sides run the same set even when the two checkouts disagree about the manifest.
54 +
55 +The candidate client comes from a **local artifact**, not a version spec:
56 +`COMPOSIO_CLIENT_TARBALL` for TypeScript, `COMPOSIO_CLIENT_WHEEL` for Python.
57 +Fetch the Python one with `pip download composio-client==<version> --no-deps`.
58 +Both swaps abort the sweep if the client the project resolves does not actually
59 +change, and both restore the files they touched when the sweep ends.
60 +
61 +For Python the runner also drops the project's exact `composio-client==` pin for
62 +the duration of a candidate sweep. Several entries install the local `./python`
63 +project through `pyWith`, and uv cannot satisfy that pin and the candidate wheel
64 +at once — without this, those entries fail to resolve and go red for a packaging
65 +reason, quietly shrinking the comparison.
66 +
67 +`parity.mjs` only compares entries green in **both** runs; anything red, skipped,
68 +or missing on either side becomes `parity: false` with a reason rather than
69 +failing the comparator. Read `compared` alongside `parityGreen`, and check that
70 +the traces are non-empty — parity over two empty traces holds vacuously.
71 +
72 +## Verifying the harness itself
73 +
74 +```bash
75 +node harness/run.mjs selftest
76 +```
77 +
78 +Covers backend-URL handling, the candidate-swap guards, the known-good and
79 +known-bad fixtures, both tracers, and the comparator's own accept/reject
80 +behaviour. `node harness/run.mjs neg` is the complementary check on the examples:
81 +every entry must go red under garbage credentials, so an entry that swallows its
82 +errors cannot pass as coverage.
harness/backend-url.mjs new
+30
@@ -0,0 +1,30 @@
1 +export const STAGING_BASE_URL = 'https://staging-backend.composio.dev';
2 +
3 +// Resolves the backend the examples run against: whatever COMPOSIO_BASE_URL
4 +// says, defaulting to staging. The structural checks are not about which
5 +// backend it is — a base URL carrying a path, query, fragment, or embedded
6 +// credentials means the caller meant something else, and the origin
7 +// normalisation below only holds for a bare root.
8 +export const resolveBackendBaseUrl = (
9 + value = process.env.COMPOSIO_BASE_URL ?? STAGING_BASE_URL
10 +) => {
11 + let url;
12 + try {
13 + url = new URL(value);
14 + } catch {
15 + throw new Error(`refusing invalid COMPOSIO_BASE_URL: ${value}`);
16 + }
17 +
18 + const isBareHttpsRoot =
19 + url.protocol === 'https:' &&
20 + url.pathname === '/' &&
21 + !url.search &&
22 + !url.hash &&
23 + !url.username &&
24 + !url.password;
25 + if (!isBareHttpsRoot) {
26 + throw new Error(`refusing malformed COMPOSIO_BASE_URL: ${value}`);
27 + }
28 +
29 + return url.origin;
30 +};
harness/run.mjs
+80 -11
@@ -12,7 +12,7 @@ import { spawn } from 'node:child_process';
12 import { mkdirSync, readFileSync, writeFileSync, appendFileSync, existsSync, rmSync } from 'node:fs';
13 import { dirname, join, relative, resolve } from 'node:path';
14 import { fileURLToPath } from 'node:url';
15 -import { requireStagingBaseUrl, STAGING_BASE_URL } from './staging-backend.mjs';
15 +import { resolveBackendBaseUrl, STAGING_BASE_URL } from './backend-url.mjs';
16
17 const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
18 const ARTIFACTS = join(ROOT, '.artifacts', 'examples-parity');
@@ -67,7 +67,7 @@ const selectEntries = () => {
67
68 const baseUrl = () => {
69 try {
70 - return requireStagingBaseUrl();
70 + return resolveBackendBaseUrl();
71 } catch (error) {
72 fail(error instanceof Error ? error.message : String(error));
73 }
@@ -315,6 +315,35 @@ const verifyPyCandidate = async (wheel) => {
315 return version;
316 };
317
318 +const PY_PROJECT = join(ROOT, 'python', 'pyproject.toml');
319 +const UV_LOCK = join(ROOT, 'uv.lock');
320 +const PY_CLIENT_PIN = /^(\s*)"composio-client==[^"]+",$/m;
321 +
322 +/**
323 + * Drop the exact `composio-client==` pin from the python project.
324 + *
325 + * A candidate sweep layers the wheel onto every entry with `uv run --with`, but
326 + * several entries also install the local `./python` project, whose exact pin the
327 + * wheel contradicts — uv then refuses the whole overlay ("no solution found")
328 + * and those entries go red for a packaging reason rather than a client one.
329 + * Without the pin the wheel is the only `composio-client` in play. Baseline
330 + * sweeps never call this, and restorePyBaseline() writes the pre-run files back
331 + * either way.
332 + */
333 +const relaxPyClientPin = (toml) => {
334 + if (!PY_CLIENT_PIN.test(toml)) {
335 + fail(`no exact composio-client pin found in ${relative(ROOT, PY_PROJECT)}`);
336 + }
337 + return toml.replace(PY_CLIENT_PIN, '$1"composio-client",');
338 +};
339 +
340 +const snapshotPyBaseline = () => snapshotFiles([PY_PROJECT, UV_LOCK]);
341 +
342 +const restorePyBaseline = async (snapshot) => {
343 + restoreFiles(snapshot);
344 + await sh(['uv', 'sync', '--project', 'python', '--frozen']);
345 +};
346 +
347 // --- llm mock ---------------------------------------------------------------
348
349 /** Start aimock serving harness/llm-mock/fixtures; resolve once it answers. */
@@ -365,6 +394,7 @@ const cmdSweep = async () => {
394 mkdirSync(join(runDir, 'traces'), { recursive: true });
395
396 let tsBaselineSnapshot;
397 + let pyBaselineSnapshot;
398 let stopLlmMock;
399 const pyWheel = process.env.COMPOSIO_CLIENT_WHEEL;
400 try {
@@ -381,6 +411,8 @@ const cmdSweep = async () => {
411 if (!pyWheel) fail('candidate sweep with py entries needs COMPOSIO_CLIENT_WHEEL');
412 const v = await verifyPyCandidate(pyWheel);
413 console.log(`candidate composio-client (py) resolved: ${v}`);
414 + pyBaselineSnapshot = snapshotPyBaseline();
415 + writeFileSync(PY_PROJECT, relaxPyClientPin(readFileSync(PY_PROJECT, 'utf8')));
416 for (const e of entries) if (e.lang === 'py') e.pyWith = [...(e.pyWith ?? []), pyWheel];
417 }
418 }
@@ -402,6 +434,7 @@ const cmdSweep = async () => {
434 } finally {
435 if (stopLlmMock) stopLlmMock();
436 if (tsBaselineSnapshot) await restoreTsBaseline(tsBaselineSnapshot);
437 + if (pyBaselineSnapshot) await restorePyBaseline(pyBaselineSnapshot);
438 }
439 };
440
@@ -438,15 +471,29 @@ const cmdSelftest = async () => {
471 if (!ok) failures += 1;
472 };
473
441 - // 0. backend safety: the harness accepts only the canonical staging root.
442 - check('staging backend is accepted', requireStagingBaseUrl() === STAGING_BASE_URL);
443 - let productionRejected = false;
444 - try {
445 - requireStagingBaseUrl('https://backend.composio.dev');
446 - } catch {
447 - productionRejected = true;
448 - }
449 - check('production backend is rejected', productionRejected);
474 + // 0. backend URL: staging is the default, any other bare https root is
475 + // honoured as given, and anything structurally unusable is refused.
476 + check('staging backend is the default', resolveBackendBaseUrl(undefined) === STAGING_BASE_URL);
477 + check(
478 + 'a non-staging bare https root is honoured',
479 + resolveBackendBaseUrl('https://backend.composio.dev') === 'https://backend.composio.dev'
480 + );
481 + const refuses = (value) => {
482 + try {
483 + resolveBackendBaseUrl(value);
484 + return false;
485 + } catch {
486 + return true;
487 + }
488 + };
489 + check('a base URL carrying a path is refused', refuses('https://backend.composio.dev/api/v3'));
490 + check('a base URL carrying a query is refused', refuses('https://backend.composio.dev/?x=1'));
491 + check(
492 + 'a base URL carrying credentials is refused',
493 + refuses('https://user:pass@backend.composio.dev')
494 + );
495 + check('a plaintext base URL is refused', refuses('http://backend.composio.dev'));
496 + check('an unparseable base URL is refused', refuses('not-a-url'));
497
498 let cleanupRan = false;
499 let cleanupError;
@@ -471,6 +518,28 @@ const cmdSelftest = async () => {
518 restoreFiles(cleanupSnapshot);
519 check('cleanup restores pre-run file contents', readFileSync(cleanupFixture, 'utf8') === 'user contents');
520
521 + // The python candidate swap only resolves once the project's exact
522 + // composio-client pin is out of the way; see relaxPyClientPin.
523 + const pinnedProject = readFileSync(PY_PROJECT, 'utf8');
524 + const relaxedProject = relaxPyClientPin(pinnedProject);
525 + check('the python project pins composio-client exactly', /"composio-client==/.test(pinnedProject));
526 + check(
527 + 'relaxing the pin keeps composio-client a dependency',
528 + /^\s*"composio-client",$/m.test(relaxedProject) && !/"composio-client==/.test(relaxedProject)
529 + );
530 + check(
531 + 'relaxing the pin touches nothing else',
532 + relaxedProject.replace(/^\s*"composio-client",$/m, '') ===
533 + pinnedProject.replace(/^\s*"composio-client==[^"]+",$/m, '')
534 + );
535 + let unpinnedProjectRejected = false;
536 + try {
537 + relaxPyClientPin(relaxedProject);
538 + } catch {
539 + unpinnedProjectRejected = true;
540 + }
541 + check('relaxing an already-unpinned project is refused', unpinnedProjectRejected);
542 +
543 // 1. known-good: error-handling-demo (no backend, no keys).
544 const good = loadManifest().find((e) => e.id === 'ts/error-handling-demo/index');
545 const { env: goodEnv } = buildEnv(good, runDir);
harness/staging-backend.mjs deleted
-25
@@ -1,25 +0,0 @@
1 -export const STAGING_BASE_URL = 'https://staging-backend.composio.dev';
2 -
3 -export const requireStagingBaseUrl = (
4 - value = process.env.COMPOSIO_BASE_URL ?? STAGING_BASE_URL
5 -) => {
6 - let url;
7 - try {
8 - url = new URL(value);
9 - } catch {
10 - throw new Error(`refusing invalid COMPOSIO_BASE_URL: ${value}`);
11 - }
12 -
13 - const isStagingRoot =
14 - url.origin === STAGING_BASE_URL &&
15 - url.pathname === '/' &&
16 - !url.search &&
17 - !url.hash &&
18 - !url.username &&
19 - !url.password;
20 - if (!isStagingRoot) {
21 - throw new Error(`refusing non-staging COMPOSIO_BASE_URL: ${value}`);
22 - }
23 -
24 - return STAGING_BASE_URL;
25 -};
harness/trace-py/sitecustomize.py
+6 -2
@@ -13,11 +13,15 @@ _TRACE = os.environ.get("COMPOSIO_TRACE_FILE")
13 _STAGING_BASE_URL = "https://staging-backend.composio.dev"
14
15 if _TRACE:
16 + # Mirrors harness/backend-url.mjs: COMPOSIO_BASE_URL selects the backend and
17 + # defaults to staging. The structural checks stay — a base URL carrying a
18 + # path, query, fragment, or embedded credentials means the caller meant
19 + # something else, and the host pin below needs a bare root to be meaningful.
20 _base_url = os.environ.get("COMPOSIO_BASE_URL", _STAGING_BASE_URL)
21 _parsed_base_url = urlsplit(_base_url)
22 if not (
23 _parsed_base_url.scheme == "https"
20 - and _parsed_base_url.hostname == "staging-backend.composio.dev"
24 + and _parsed_base_url.hostname
25 and _parsed_base_url.path in {"", "/"}
26 and not _parsed_base_url.query
27 and not _parsed_base_url.fragment
@@ -25,7 +29,7 @@ if _TRACE:
29 and not _parsed_base_url.password
30 ):
31 print(
28 - f"sitecustomize: refusing non-staging COMPOSIO_BASE_URL: {_base_url}",
32 + f"sitecustomize: refusing malformed COMPOSIO_BASE_URL: {_base_url}",
33 file=sys.stderr,
34 )
35 os._exit(78)
harness/trace/register.mjs
+2 -2
@@ -1,6 +1,6 @@
1 // Harness-owned fetch tracer. Injected via NODE_OPTIONS=--import; examples
2 // must never reference COMPOSIO_TRACE_FILE themselves (lint-enforced).
3 -import { requireStagingBaseUrl } from '../staging-backend.mjs';
3 +import { resolveBackendBaseUrl } from '../backend-url.mjs';
4
5 const traceFile = process.env.COMPOSIO_TRACE_FILE;
6
@@ -13,7 +13,7 @@ if (traceFile) {
13 'generativelanguage.googleapis.com',
14 ]);
15
16 - const backendHost = new URL(requireStagingBaseUrl()).hostname;
16 + const backendHost = new URL(resolveBackendBaseUrl()).hostname;
17
18 const ID_SEG =
19 /^(ca_|ac_|ti_|tr_|trs_|sess_|auth_|req_|proj_|org_)[\w-]+$|^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$|^\d+$/i;
scripts/examples-provision.mjs
+6 -4
@@ -30,11 +30,11 @@
30 // values are ever printed. The API-key demo value stored for serpapi is a
31 // deliberately fake placeholder, not a real key.
32
33 -import { requireStagingBaseUrl } from '../harness/staging-backend.mjs';
33 +import { resolveBackendBaseUrl } from '../harness/backend-url.mjs';
34
35 let BASE_URL;
36 try {
37 - BASE_URL = requireStagingBaseUrl();
37 + BASE_URL = resolveBackendBaseUrl();
38 } catch (error) {
39 console.error(error instanceof Error ? error.message : String(error));
40 process.exit(1);
@@ -45,8 +45,10 @@ const INITIATE_MISSING = process.argv.includes('--initiate-missing');
45 const GC = process.argv.includes('--gc');
46 const DRY_RUN = process.argv.includes('--dry-run');
47
48 -// The examples only ever run against staging with a disposable project key.
49 -// requireStagingBaseUrl refuses every other host before the key is read or sent.
48 +// COMPOSIO_BASE_URL selects the backend and defaults to staging;
49 +// resolveBackendBaseUrl refuses a structurally unusable one before the key is
50 +// read or sent. Point this at a project whose resources you are willing to have
51 +// this script create — and, under --gc, delete.
52 if (!API_KEY) {
53 console.error('COMPOSIO_API_KEY is required (dedicated examples-project key)');
54 process.exit(1);