Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
Mirrored from https://github.com/Camo67/composio.git
Synced Nov 21, 06:00 UTC
| 1 | # Hill-climbing the composio skills |
| 2 | |
| 3 | This is how to tell whether a change to the `composio-cli` skill (or to the CLI's help and error text) makes agents better at using the CLI, and how to iterate on it. It is written for a person or for a coding agent doing the iterating. |
| 4 | |
| 5 | The replay suite (`e2e.test.ts`) is not the measure. It replays fixed model answers, so a skill change cannot move it. The measure is `eval.ts`: live, repeated runs of the journeys with the real agent, graded by the same checks. |
| 6 | |
| 7 | ## What a run is |
| 8 | |
| 9 | One journey, one arm, one attempt: |
| 10 | |
| 11 | - **Agent.** Claude Code 2.1.310, headless, in the journey container (`--agent claude-code`, the default). The skill is installed in `~/.claude/skills/composio-cli/`, among Claude Code's own skills, so the agent has to find it from its description, as a user's agent does. `--agent loop` instead gives the skill to the minimal loop as its system prompt: cheaper to reason about, but it measures only what the skill says, not whether it is found. |
| 12 | - **Arm.** `skill` installs the skill built from the working tree; `no-skill` installs nothing. The difference is what the skill is worth. |
| 13 | - **Model.** `claude-haiku-5-5` or `claude-opus-5-5`; the model proxy refuses any other. |
| 14 | - **Grade.** The journey's checks, which read what the fake backend recorded, never the agent's prose. A run passes when every check holds. Detours (a command on the wrong tree, a request to the wrong project) are counted apart and never fail a run. |
| 15 | |
| 16 | ## The numbers |
| 17 | |
| 18 | For each journey and arm, over its graded runs: |
| 19 | |
| 20 | | number | meaning | what to watch | |
| 21 | | ---------- | ----------------------------------------- | -------------------------------------------------------------- | |
| 22 | | pass@1 | share of runs that passed | the plain success rate, with a 95% interval | |
| 23 | | pass@k | chance that at least one of k runs passes | whether the agent can do it at all | |
| 24 | | pass^k | chance that all k runs pass | whether a user can rely on it; the number to raise | |
| 25 | | wrong tree | share of runs with at least one detour | the root-tree / `composio dev` confusion this suite exists for | |
| 26 | |
| 27 | Splits are averaged per journey, so a journey with more runs does not weigh more. `train` journeys are the ones changes are tuned against; `holdout` journeys only confirm. |
| 28 | |
| 29 | ## The loop |
| 30 | |
| 31 | The key is loaded only into the command's environment: |
| 32 | |
| 33 | ```sh |
| 34 | cd ts/e2e-tests/cli/agent-journeys |
| 35 | set -a; . ~/.config/composio-e2e/llm.env; set +a |
| 36 | ``` |
| 37 | |
| 38 | 1. **Check the setup.** `bun run eval.ts run --model claude-haiku-5-5 --dry` prints the plan and spends nothing; `--smoke` runs one journey once. |
| 39 | 2. **Measure the baseline.** Both arms, every journey, three runs each: |
| 40 | `bun run eval.ts run --model claude-haiku-5-5 --runs 3 --arms skill,no-skill --label baseline` |
| 41 | 3. **Read the failures.** `results/<batch>/summary.md` lists which checks failed how often. `results/<batch>/logs/<journey>.<arm>.<attempt>.json` has the commands, what they printed, what the agent wrote, the requests the fake received, and every Claude Code event. Find what the agent misunderstood before changing anything. |
| 42 | 4. **Make one change.** Edit the skill's source in `ts/packages/cli/skills-src/composio-cli/`, or the CLI text the agent misread. Keep it to one idea, so a result can be attributed to it. |
| 43 | 5. **Measure the train split.** `bun run eval.ts run --model claude-haiku-5-5 --runs 3 --split train --label <change>`, then |
| 44 | `bun run eval.ts compare <baseline batch> <change batch>`. |
| 45 | 6. **Decide.** Keep the change when the train split's mean pass^k rises and no train journey's pass@1 falls by more than one run. A difference of one run out of three is noise: when it decides the outcome, run both batches again with `--runs 5` (earlier runs with the same fingerprint are reused, so only the extra runs are paid for). |
| 46 | 7. **Confirm.** Run the holdout split on the kept change and on the baseline, and the train split on `claude-opus-5-5`. A change that helps train and hurts holdout fits the journeys, not the agent: drop it. |
| 47 | 8. **Record.** Results are git-ignored. Keep the batches a decision rests on with `bun run eval.ts keep <batch> <decision>`, which copies their records (not the logs) into `evals/<decision>/`, and write the decision's `README.md` there: what the runs showed, the change, the tables, the commands to repeat it. See [evals/](./evals/README.md); the pull request links to it. |
| 48 | |
| 49 | ## Rules |
| 50 | |
| 51 | - **Do not change the harness in the same step as the skill.** The journeys, the checks, the person and the fake backend (`agent-journeys/src/`, `fake-backend/src/`) are what grade a run; their hash is part of every batch, and `compare` refuses two batches with different hashes. Fix a grader in its own change, then measure a new baseline. |
| 52 | - **Do not tune the checks to the agent.** A check that fails because the agent took a valid path the check did not foresee is a grader bug; fix it with a test in `journeys.test.ts`. A check that fails because the agent did the wrong thing is the signal. |
| 53 | - **Know what a journey's checks notice before trusting its pass^k.** `bun run mutate.ts --journeys <id>` plays the hand-authored path once per command with that command left out (see the [README](./README.md#what-a-mutation-run-proves)). A step it calls `check-too-weak` is a way to end somewhere else that the journey still passes; strengthen the check, in its own change, before reading a rate off it. `--strict` exits 1 on one, so a run with it is the check to make before trusting a rate, or to gate on in CI when the containers are worth it. A step it calls `unused` is a read or a command that fails on purpose, and no check should depend on it. |
| 54 | - **Repeated runs are comparable only if they start alike.** `determinism.test.ts` pins the seeded world, and the replay suite checks that every replayed attempt of a journey started from the same one and that a rig left idle after login stays quiet. `eval.ts` does not record the starting world of its runs; it relies on those tests. When one fails, fix the seed or the leak before comparing batches. |
| 55 | - **Do not add the journeys' wording to the skill.** The skill is for every user's task; a sentence that only matches a journey's prompt raises train and nothing else. |
| 56 | - **Spend little.** A Haiku run costs about $0.002 to $0.01; Opus about 40 times that. `--max-cost-usd` (default $2) stops a batch from starting runs past it, and `--run-budget-usd` caps one run. |
| 57 | |
| 58 | ## Adding a journey |
| 59 | |
| 60 | A journey that no agent fails cannot separate two skills, and one that every agent fails cannot either. Add one when a live run shows a failure no journey catches: the prompt a user would type, the state it starts from, checks on what the fake recorded, and a `split`. `journeys.test.ts` then checks that an agent doing nothing fails it. |
| 61 | |
| 62 | ## Limits |
| 63 | |
| 64 | See "Measuring a change" in the [README](./README.md#measuring-a-change): outbound traffic is not blocked, requests of overlapping or background commands are attributed together, and five journeys are a thin train and holdout set. Two more: |
| 65 | |
| 66 | - `composio login` installs the skill for real users, so a `no-skill` run that logs in may end with the skill installed. That is the product's behaviour, and it means `no-skill` measures "not installed at the start". |
| 67 | - Claude Code's own skills are present in both arms, as they are for a user. |