Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
| 1 | # Examples harness |
| 2 | |
| 3 | Runs the entrypoints inventoried in `examples-manifest.json` against a live |
| 4 | Composio backend, records every backend call each entry makes, and compares two |
| 5 | runs call-for-call. |
| 6 | |
| 7 | - `harness/run.mjs` — sweeps entries, writes `results.jsonl` + one trace file per |
| 8 | entry under `.artifacts/examples-parity/<run-id>/`. |
| 9 | - `harness/parity.mjs` — compares two run directories. |
| 10 | - `harness/trace/register.mjs`, `harness/trace-py/sitecustomize.py` — the fetch |
| 11 | and httpx tracers, injected by the runner. Examples never reference |
| 12 | `COMPOSIO_TRACE_FILE` themselves. |
| 13 | - `scripts/examples-provision.mjs` — provisions the auth configs and connected |
| 14 | accounts the tier-2/3 entries need, and prints them as `COMPOSIO_EXAMPLES_*` |
| 15 | exports. |
| 16 | |
| 17 | ## Backend selection |
| 18 | |
| 19 | `COMPOSIO_BASE_URL` picks the backend and defaults to staging. Any bare `https` |
| 20 | root is accepted; a URL carrying a path, query, fragment, or embedded |
| 21 | credentials is refused, because the tracers pin the backend host and the |
| 22 | provisioner appends its own paths. `.github/workflows/examples-live.yml` sets |
| 23 | staging explicitly, so CI is unaffected by the default. |
| 24 | |
| 25 | Point this at a project whose data you are willing to have the examples touch. |
| 26 | `scripts/examples-provision.mjs --gc` deletes examples-owned resources older |
| 27 | than 24h across the whole project — never run it against a project you care |
| 28 | about. The outbound-email denylist is enforced in the tracers and is not |
| 29 | overridable per run; `--llm mock` additionally keeps model traffic on a local |
| 30 | `aimock` server so no agent can decide to write something. |
| 31 | |
| 32 | ## Comparing a client bump |
| 33 | |
| 34 | A parity run is two sweeps over the same entry ids, one on each side of the |
| 35 | change, compared by traced `(method, path-template)` pairs. |
| 36 | |
| 37 | ```bash |
| 38 | # Provisioned ids for the project you are sweeping. Capture, then eval — a |
| 39 | # failed provisioning run must not be swallowed. |
| 40 | out=$(node scripts/examples-provision.mjs) && eval "$out" |
| 41 | |
| 42 | # Baseline: the pinned client, from a checkout of the base branch. |
| 43 | node harness/run.mjs sweep --client baseline --lang py --llm mock --ids "$IDS" |
| 44 | |
| 45 | # Candidate: the same entries with the candidate client swapped in. |
| 46 | COMPOSIO_CLIENT_WHEEL=/abs/path/composio_client-<version>-py3-none-any.whl \ |
| 47 | node harness/run.mjs sweep --client candidate --lang py --llm mock --ids "$IDS" |
| 48 | |
| 49 | node harness/parity.mjs <baseline-run-dir> <candidate-run-dir> |
| 50 | ``` |
| 51 | |
| 52 | Pass `--ids` explicitly rather than relying on the default selection, so both |
| 53 | sides run the same set even when the two checkouts disagree about the manifest. |
| 54 | |
| 55 | The candidate client comes from a **local artifact**, not a version spec: |
| 56 | `COMPOSIO_CLIENT_TARBALL` for TypeScript, `COMPOSIO_CLIENT_WHEEL` for Python. |
| 57 | Fetch the Python one with `pip download composio-client==<version> --no-deps`. |
| 58 | Both swaps abort the sweep if the client the project resolves does not actually |
| 59 | change, and both restore the files they touched when the sweep ends. |
| 60 | |
| 61 | For Python the runner also drops the project's exact `composio-client==` pin for |
| 62 | the duration of a candidate sweep. Several entries install the local `./python` |
| 63 | project through `pyWith`, and uv cannot satisfy that pin and the candidate wheel |
| 64 | at once — without this, those entries fail to resolve and go red for a packaging |
| 65 | reason, quietly shrinking the comparison. |
| 66 | |
| 67 | `parity.mjs` only compares entries green in **both** runs; anything red, skipped, |
| 68 | or missing on either side becomes `parity: false` with a reason rather than |
| 69 | failing the comparator. Read `compared` alongside `parityGreen`, and check that |
| 70 | the traces are non-empty — parity over two empty traces holds vacuously. |
| 71 | |
| 72 | ## Verifying the harness itself |
| 73 | |
| 74 | ```bash |
| 75 | node harness/run.mjs selftest |
| 76 | ``` |
| 77 | |
| 78 | Covers backend-URL handling, the candidate-swap guards, the known-good and |
| 79 | known-bad fixtures, both tracers, and the comparator's own accept/reject |
| 80 | behaviour. `node harness/run.mjs neg` is the complementary check on the examples: |
| 81 | every entry must go red under garbage credentials, so an entry that swallows its |
| 82 | errors cannot pass as coverage. |