Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
| 1 | # Baseline after the grader guards |
| 2 | |
| 3 | The reference batch for every later decision about the `composio-cli` skill, measured on the harness that adds mutation and determinism guards ([#5418](https://github.com/ComposioHQ/composio/pull/5418)). |
| 4 | |
| 5 | The journeys, checks, person and fake backend changed, and their hash is part of every batch, so no batch kept in `evals/2026-11-11-*` describes a run of the current harness: `compare` refuses to put them beside a new one. This directory holds the batch to compare against instead. |
| 6 | |
| 7 | ## What was run |
| 8 | |
| 9 | ```sh |
| 10 | cd ts/e2e-tests/cli/agent-journeys |
| 11 | set -a; . ~/.config/composio-e2e/llm.env; set +a |
| 12 | bun run eval.ts run --model claude-haiku-5-5 --runs 5 --arms skill,no-skill --force --label baseline |
| 13 | bun run eval.ts keep <batch> baseline-grader-guards |
| 14 | ``` |
| 15 | |
| 16 | Claude Code 2.1.310, `claude-haiku-5-5`, 5 runs per journey and arm, 50 runs, $0.096. Harness `edbcd56c3084`, skill `25e724edfae2`, CLI `6710a5ed`; the batch records `dirty: false`. `--force` reused no earlier run. The batch is `2026-11-16T18-57-32-claude-code-claude-haiku-5-5-baseline`. |
| 17 | |
| 18 | `2026-11-16T17-52-05-…` is kept beside it and is **not** the baseline: it is the same harness, skill and CLI over three runs, and it is here only because the last section below compares the two. Read a rate from the five-run batch. |
| 19 | |
| 20 | | split | journey | arm | passed | pass@1 | pass^k | wrong tree | |
| 21 | | ------- | ---------------------------------- | -------- | ------ | --------------- | ------ | ---------- | |
| 22 | | train | consumer-slack-message | no-skill | 0/5 | 0% (0%–43%) | 0% | 0% | |
| 23 | | train | consumer-slack-message | skill | 5/5 | 100% (57%–100%) | 100% | 0% | |
| 24 | | train | developer-execute-unconnected | no-skill | 1/5 | 20% (4%–62%) | 0% | 0% | |
| 25 | | train | developer-execute-unconnected | skill | 5/5 | 100% (57%–100%) | 100% | 0% | |
| 26 | | train | first-run-login | no-skill | 3/5 | 60% (23%–88%) | 0% | 0% | |
| 27 | | train | first-run-login | skill | 2/5 | 40% (12%–77%) | 0% | 0% | |
| 28 | | holdout | developer-auth-configs-and-execute | no-skill | 5/5 | 100% (57%–100%) | 100% | 20% | |
| 29 | | holdout | developer-auth-configs-and-execute | skill | 5/5 | 100% (57%–100%) | 100% | 0% | |
| 30 | | holdout | run-inline-script | no-skill | 5/5 | 100% (57%–100%) | 100% | 0% | |
| 31 | | holdout | run-inline-script | skill | 5/5 | 100% (57%–100%) | 100% | 0% | |
| 32 | |
| 33 | | split | arm | mean pass@1 | mean pass^k | mean wrong tree | |
| 34 | | ------- | -------- | ----------- | ----------- | --------------- | |
| 35 | | train | skill | 80% | 67% | 0% | |
| 36 | | train | no-skill | 27% | 0% | 0% | |
| 37 | | holdout | skill | 100% | 100% | 0% | |
| 38 | | holdout | no-skill | 100% | 100% | 10% | |
| 39 | |
| 40 | ## What this batch says |
| 41 | |
| 42 | - **The train split separates the arms.** 80% pass@1 and 67% pass^k with the skill, against 27% and 0% without it. That is the first batch in `evals/` where a train arm reaches 5/5 on a journey the other arm cannot do at all. |
| 43 | - **`consumer-slack-message` shows the mechanism, not just the gap.** With the skill, 5/5 runs ran `composio` and sent the message, in 5 to 10 model turns. Without it, 0/5: every run spent its one or two commands on `env | grep -i slack` or `which slack`, concluded there was no Slack tool here, and stopped. None of the five ran `composio` at all. The skill's worth on this journey is the sentence that names the CLI. |
| 44 | - **`first-run-login` is the exception and is partly a timeout rate.** 2/5 with the skill against 3/5 without it. Three of the five skill runs and two of the five no-skill runs ended at the six-minute wall-clock deadline before reporting a turn (`stop: deadline`, no model turns), which the checks record as four failures: no login session, no key, no stored login, and the model not finishing by itself. Only the other runs graded how the agent logged in. |
| 45 | - **The holdout split cannot separate the arms.** Both arms are 5/5 on both journeys. The one wrong-tree count is a single no-skill run of `developer-auth-configs-and-execute` that took two detours on its way and still passed every check, so the rate is a count of one. |
| 46 | - **The no-skill arm did not behave the same way in the batch before this one.** That batch (`2026-11-16T17-52-05-…`, kept here, 3 runs) records the same `agentVersion`, `model`, `skillSha256`, `harnessSha256` and `cliSha256`, and `dirty: false`, yet `consumer-slack-message` no-skill passed 3/3 there: each of its runs ran `composio --help` on its own and sent the message. Here it is 0/5. Nothing the batch records differs, so the difference is model behaviour the batch does not capture. Treat a rate from fewer than five runs as provisional, and re-run the baseline next to a change rather than comparing against one batch. |
| 47 | |
| 48 | ## What it does not say |
| 49 | |
| 50 | Nothing here is a decision about the skill. It measures one model: `claude-haiku-5-5` at Claude Code 2.1.310, on the train and holdout journeys as they are now. `claude-opus-5-5` is not measured here; `HILLCLIMB.md` uses it to confirm a change that the train split already supports, which needs a baseline of its own first. The mutation run in [#5418](https://github.com/ComposioHQ/composio/pull/5418) covers the other half of the same question, which steps of each hand-authored path the checks notice: 10 of 17 are `needed`, 7 are `unused` reads or commands that fail on purpose, and none is `check-too-weak`. |
| 51 | |
| 52 | ## To use it |
| 53 | |
| 54 | ```sh |
| 55 | bun run eval.ts run --model claude-haiku-5-5 --runs 5 --arms skill,no-skill --split train --label <change> |
| 56 | bun run eval.ts compare evals/baseline-grader-guards/2026-11-16T18-57-32-claude-code-claude-haiku-5-5-baseline <new batch> |
| 57 | ``` |
| 58 | |
| 59 | `compare` refuses the two when a harness file changed since this batch. Measure a new baseline in that case, in the change that moves the harness, and keep it here. |