Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

next
md 82 lines 3.92 KB
Rendered Raw
1 # Examples harness
2
3 Runs the entrypoints inventoried in `examples-manifest.json` against a live
4 Composio backend, records every backend call each entry makes, and compares two
5 runs call-for-call.
6
7 - `harness/run.mjs` — sweeps entries, writes `results.jsonl` + one trace file per
8 entry under `.artifacts/examples-parity/<run-id>/`.
9 - `harness/parity.mjs` — compares two run directories.
10 - `harness/trace/register.mjs`, `harness/trace-py/sitecustomize.py` — the fetch
11 and httpx tracers, injected by the runner. Examples never reference
12 `COMPOSIO_TRACE_FILE` themselves.
13 - `scripts/examples-provision.mjs` — provisions the auth configs and connected
14 accounts the tier-2/3 entries need, and prints them as `COMPOSIO_EXAMPLES_*`
15 exports.
16
17 ## Backend selection
18
19 `COMPOSIO_BASE_URL` picks the backend and defaults to staging. Any bare `https`
20 root is accepted; a URL carrying a path, query, fragment, or embedded
21 credentials is refused, because the tracers pin the backend host and the
22 provisioner appends its own paths. `.github/workflows/examples-live.yml` sets
23 staging explicitly, so CI is unaffected by the default.
24
25 Point this at a project whose data you are willing to have the examples touch.
26 `scripts/examples-provision.mjs --gc` deletes examples-owned resources older
27 than 24h across the whole project — never run it against a project you care
28 about. The outbound-email denylist is enforced in the tracers and is not
29 overridable per run; `--llm mock` additionally keeps model traffic on a local
30 `aimock` server so no agent can decide to write something.
31
32 ## Comparing a client bump
33
34 A parity run is two sweeps over the same entry ids, one on each side of the
35 change, compared by traced `(method, path-template)` pairs.
36
37 ```bash
38 # Provisioned ids for the project you are sweeping. Capture, then eval — a
39 # failed provisioning run must not be swallowed.
40 out=$(node scripts/examples-provision.mjs) && eval "$out"
41
42 # Baseline: the pinned client, from a checkout of the base branch.
43 node harness/run.mjs sweep --client baseline --lang py --llm mock --ids "$IDS"
44
45 # Candidate: the same entries with the candidate client swapped in.
46 COMPOSIO_CLIENT_WHEEL=/abs/path/composio_client-<version>-py3-none-any.whl \
47 node harness/run.mjs sweep --client candidate --lang py --llm mock --ids "$IDS"
48
49 node harness/parity.mjs <baseline-run-dir> <candidate-run-dir>
50 ```
51
52 Pass `--ids` explicitly rather than relying on the default selection, so both
53 sides run the same set even when the two checkouts disagree about the manifest.
54
55 The candidate client comes from a **local artifact**, not a version spec:
56 `COMPOSIO_CLIENT_TARBALL` for TypeScript, `COMPOSIO_CLIENT_WHEEL` for Python.
57 Fetch the Python one with `pip download composio-client==<version> --no-deps`.
58 Both swaps abort the sweep if the client the project resolves does not actually
59 change, and both restore the files they touched when the sweep ends.
60
61 For Python the runner also drops the project's exact `composio-client==` pin for
62 the duration of a candidate sweep. Several entries install the local `./python`
63 project through `pyWith`, and uv cannot satisfy that pin and the candidate wheel
64 at once — without this, those entries fail to resolve and go red for a packaging
65 reason, quietly shrinking the comparison.
66
67 `parity.mjs` only compares entries green in **both** runs; anything red, skipped,
68 or missing on either side becomes `parity: false` with a reason rather than
69 failing the comparator. Read `compared` alongside `parityGreen`, and check that
70 the traces are non-empty — parity over two empty traces holds vacuously.
71
72 ## Verifying the harness itself
73
74 ```bash
75 node harness/run.mjs selftest
76 ```
77
78 Covers backend-URL handling, the candidate-swap guards, the known-good and
79 known-bad fixtures, both tracers, and the comparator's own accept/reject
80 behaviour. `node harness/run.mjs neg` is the complementary check on the examples:
81 every entry must go red under garbage credentials, so an entry that swallows its
82 errors cannot pass as coverage.