@devries-cameron20 / composio Mirror · read-only
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 06:00 UTC

next
md 67 lines 7.32 KB
Rendered Raw
1 # Hill-climbing the composio skills
2
3 This is how to tell whether a change to the `composio-cli` skill (or to the CLI's help and error text) makes agents better at using the CLI, and how to iterate on it. It is written for a person or for a coding agent doing the iterating.
4
5 The replay suite (`e2e.test.ts`) is not the measure. It replays fixed model answers, so a skill change cannot move it. The measure is `eval.ts`: live, repeated runs of the journeys with the real agent, graded by the same checks.
6
7 ## What a run is
8
9 One journey, one arm, one attempt:
10
11 - **Agent.** Claude Code 2.1.310, headless, in the journey container (`--agent claude-code`, the default). The skill is installed in `~/.claude/skills/composio-cli/`, among Claude Code's own skills, so the agent has to find it from its description, as a user's agent does. `--agent loop` instead gives the skill to the minimal loop as its system prompt: cheaper to reason about, but it measures only what the skill says, not whether it is found.
12 - **Arm.** `skill` installs the skill built from the working tree; `no-skill` installs nothing. The difference is what the skill is worth.
13 - **Model.** `claude-haiku-5-5` or `claude-opus-5-5`; the model proxy refuses any other.
14 - **Grade.** The journey's checks, which read what the fake backend recorded, never the agent's prose. A run passes when every check holds. Detours (a command on the wrong tree, a request to the wrong project) are counted apart and never fail a run.
15
16 ## The numbers
17
18 For each journey and arm, over its graded runs:
19
20 | number | meaning | what to watch |
21 | ---------- | ----------------------------------------- | -------------------------------------------------------------- |
22 | pass@1 | share of runs that passed | the plain success rate, with a 95% interval |
23 | pass@k | chance that at least one of k runs passes | whether the agent can do it at all |
24 | pass^k | chance that all k runs pass | whether a user can rely on it; the number to raise |
25 | wrong tree | share of runs with at least one detour | the root-tree / `composio dev` confusion this suite exists for |
26
27 Splits are averaged per journey, so a journey with more runs does not weigh more. `train` journeys are the ones changes are tuned against; `holdout` journeys only confirm.
28
29 ## The loop
30
31 The key is loaded only into the command's environment:
32
33 ```sh
34 cd ts/e2e-tests/cli/agent-journeys
35 set -a; . ~/.config/composio-e2e/llm.env; set +a
36 ```
37
38 1. **Check the setup.** `bun run eval.ts run --model claude-haiku-5-5 --dry` prints the plan and spends nothing; `--smoke` runs one journey once.
39 2. **Measure the baseline.** Both arms, every journey, three runs each:
40 `bun run eval.ts run --model claude-haiku-5-5 --runs 3 --arms skill,no-skill --label baseline`
41 3. **Read the failures.** `results/<batch>/summary.md` lists which checks failed how often. `results/<batch>/logs/<journey>.<arm>.<attempt>.json` has the commands, what they printed, what the agent wrote, the requests the fake received, and every Claude Code event. Find what the agent misunderstood before changing anything.
42 4. **Make one change.** Edit the skill's source in `ts/packages/cli/skills-src/composio-cli/`, or the CLI text the agent misread. Keep it to one idea, so a result can be attributed to it.
43 5. **Measure the train split.** `bun run eval.ts run --model claude-haiku-5-5 --runs 3 --split train --label <change>`, then
44 `bun run eval.ts compare <baseline batch> <change batch>`.
45 6. **Decide.** Keep the change when the train split's mean pass^k rises and no train journey's pass@1 falls by more than one run. A difference of one run out of three is noise: when it decides the outcome, run both batches again with `--runs 5` (earlier runs with the same fingerprint are reused, so only the extra runs are paid for).
46 7. **Confirm.** Run the holdout split on the kept change and on the baseline, and the train split on `claude-opus-5-5`. A change that helps train and hurts holdout fits the journeys, not the agent: drop it.
47 8. **Record.** Results are git-ignored. Keep the batches a decision rests on with `bun run eval.ts keep <batch> <decision>`, which copies their records (not the logs) into `evals/<decision>/`, and write the decision's `README.md` there: what the runs showed, the change, the tables, the commands to repeat it. See [evals/](./evals/README.md); the pull request links to it.
48
49 ## Rules
50
51 - **Do not change the harness in the same step as the skill.** The journeys, the checks, the person and the fake backend (`agent-journeys/src/`, `fake-backend/src/`) are what grade a run; their hash is part of every batch, and `compare` refuses two batches with different hashes. Fix a grader in its own change, then measure a new baseline.
52 - **Do not tune the checks to the agent.** A check that fails because the agent took a valid path the check did not foresee is a grader bug; fix it with a test in `journeys.test.ts`. A check that fails because the agent did the wrong thing is the signal.
53 - **Know what a journey's checks notice before trusting its pass^k.** `bun run mutate.ts --journeys <id>` plays the hand-authored path once per command with that command left out (see the [README](./README.md#what-a-mutation-run-proves)). A step it calls `check-too-weak` is a way to end somewhere else that the journey still passes; strengthen the check, in its own change, before reading a rate off it. `--strict` exits 1 on one, so a run with it is the check to make before trusting a rate, or to gate on in CI when the containers are worth it. A step it calls `unused` is a read or a command that fails on purpose, and no check should depend on it.
54 - **Repeated runs are comparable only if they start alike.** `determinism.test.ts` pins the seeded world, and the replay suite checks that every replayed attempt of a journey started from the same one and that a rig left idle after login stays quiet. `eval.ts` does not record the starting world of its runs; it relies on those tests. When one fails, fix the seed or the leak before comparing batches.
55 - **Do not add the journeys' wording to the skill.** The skill is for every user's task; a sentence that only matches a journey's prompt raises train and nothing else.
56 - **Spend little.** A Haiku run costs about $0.002 to $0.01; Opus about 40 times that. `--max-cost-usd` (default $2) stops a batch from starting runs past it, and `--run-budget-usd` caps one run.
57
58 ## Adding a journey
59
60 A journey that no agent fails cannot separate two skills, and one that every agent fails cannot either. Add one when a live run shows a failure no journey catches: the prompt a user would type, the state it starts from, checks on what the fake recorded, and a `split`. `journeys.test.ts` then checks that an agent doing nothing fails it.
61
62 ## Limits
63
64 See "Measuring a change" in the [README](./README.md#measuring-a-change): outbound traffic is not blocked, requests of overlapping or background commands are attributed together, and five journeys are a thin train and holdout set. Two more:
65
66 - `composio login` installs the skill for real users, so a `no-skill` run that logs in may end with the skill installed. That is the product's behaviour, and it means `no-skill` measures "not installed at the start".
67 - Claude Code's own skills are present in both arms, as they are for a user.