Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
Mirrored from https://github.com/Camo67/composio.git
Synced Nov 21, 04:45 UTC
Hill-climbing the composio skills
This is how to tell whether a change to the composio-cli skill (or to the CLI's help and error text) makes agents better at using the CLI, and how to iterate on it. It is written for a person or for a coding agent doing the iterating.
The replay suite (e2e.test.ts) is not the measure. It replays fixed model answers, so a skill change cannot move it. The measure is eval.ts: live, repeated runs of the journeys with the real agent, graded by the same checks.
What a run is
One journey, one arm, one attempt:
- Agent. Claude Code 2.1.310, headless, in the journey container (
--agent claude-code, the default). The skill is installed in~/.claude/skills/composio-cli/, among Claude Code's own skills, so the agent has to find it from its description, as a user's agent does.--agent loopinstead gives the skill to the minimal loop as its system prompt: cheaper to reason about, but it measures only what the skill says, not whether it is found. - Arm.
skillinstalls the skill built from the working tree;no-skillinstalls nothing. The difference is what the skill is worth. - Model.
claude-haiku-5-5orclaude-opus-5-5; the model proxy refuses any other. - Grade. The journey's checks, which read what the fake backend recorded, never the agent's prose. A run passes when every check holds. Detours (a command on the wrong tree, a request to the wrong project) are counted apart and never fail a run.
The numbers
For each journey and arm, over its graded runs:
| number | meaning | what to watch |
|---|---|---|
| pass@1 | share of runs that passed | the plain success rate, with a 95% interval |
| pass@k | chance that at least one of k runs passes | whether the agent can do it at all |
| passk | chance that all k runs pass | whether a user can rely on it; the number to raise |
| wrong tree | share of runs with at least one detour | the root-tree / composio dev confusion this suite exists for |
Splits are averaged per journey, so a journey with more runs does not weigh more. train journeys are the ones changes are tuned against; holdout journeys only confirm.
The loop
The key is loaded only into the command's environment:
cd ts/e2e-tests/cli/agent-journeys
set -a; . ~/.config/composio-e2e/llm.env; set +a
- Check the setup.
bun run eval.ts run --model claude-haiku-5-5 --dryprints the plan and spends nothing;--smokeruns one journey once. - Measure the baseline. Both arms, every journey, three runs each:
bun run eval.ts run --model claude-haiku-5-5 --runs 3 --arms skill,no-skill --label baseline - Read the failures.
results/<batch>/summary.mdlists which checks failed how often.results/<batch>/logs/<journey>.<arm>.<attempt>.jsonhas the commands, what they printed, what the agent wrote, the requests the fake received, and every Claude Code event. Find what the agent misunderstood before changing anything. - Make one change. Edit the skill's source in
ts/packages/cli/skills-src/composio-cli/, or the CLI text the agent misread. Keep it to one idea, so a result can be attributed to it. - Measure the train split.
bun run eval.ts run --model claude-haiku-5-5 --runs 3 --split train --label <change>, thenbun run eval.ts compare <baseline batch> <change batch>. - Decide. Keep the change when the train split's mean passk rises and no train journey's pass@1 falls by more than one run. A difference of one run out of three is noise: when it decides the outcome, run both batches again with
--runs 5(earlier runs with the same fingerprint are reused, so only the extra runs are paid for). - Confirm. Run the holdout split on the kept change and on the baseline, and the train split on
claude-opus-5-5. A change that helps train and hurts holdout fits the journeys, not the agent: drop it. - Record. Results are git-ignored. Keep the batches a decision rests on with
bun run eval.ts keep <batch> <decision>, which copies their records (not the logs) intoevals/<decision>/, and write the decision'sREADME.mdthere: what the runs showed, the change, the tables, the commands to repeat it. See evals/; the pull request links to it.
Rules
- Do not change the harness in the same step as the skill. The journeys, the checks, the person and the fake backend (
agent-journeys/src/,fake-backend/src/) are what grade a run; their hash is part of every batch, andcomparerefuses two batches with different hashes. Fix a grader in its own change, then measure a new baseline. - Do not tune the checks to the agent. A check that fails because the agent took a valid path the check did not foresee is a grader bug; fix it with a test in
journeys.test.ts. A check that fails because the agent did the wrong thing is the signal. - Know what a journey's checks notice before trusting its passk.
bun run mutate.ts --journeys <id>plays the hand-authored path once per command with that command left out (see the README). A step it callscheck-too-weakis a way to end somewhere else that the journey still passes; strengthen the check, in its own change, before reading a rate off it.--strictexits 1 on one, so a run with it is the check to make before trusting a rate, or to gate on in CI when the containers are worth it. A step it callsunusedis a read or a command that fails on purpose, and no check should depend on it. - Repeated runs are comparable only if they start alike.
determinism.test.tspins the seeded world, and the replay suite checks that every replayed attempt of a journey started from the same one and that a rig left idle after login stays quiet.eval.tsdoes not record the starting world of its runs; it relies on those tests. When one fails, fix the seed or the leak before comparing batches. - Do not add the journeys' wording to the skill. The skill is for every user's task; a sentence that only matches a journey's prompt raises train and nothing else.
- Spend little. A Haiku run costs about $0.002 to $0.01; Opus about 40 times that.
--max-cost-usd(default $2) stops a batch from starting runs past it, and--run-budget-usdcaps one run.
Adding a journey
A journey that no agent fails cannot separate two skills, and one that every agent fails cannot either. Add one when a live run shows a failure no journey catches: the prompt a user would type, the state it starts from, checks on what the fake recorded, and a split. journeys.test.ts then checks that an agent doing nothing fails it.
Limits
See "Measuring a change" in the README: outbound traffic is not blocked, requests of overlapping or background commands are attributed together, and five journeys are a thin train and holdout set. Two more:
composio logininstalls the skill for real users, so ano-skillrun that logs in may end with the skill installed. That is the product's behaviour, and it meansno-skillmeasures "not installed at the start".- Claude Code's own skills are present in both arms, as they are for a user.