@devries-cameron20 / composio Mirror · read-only / commits / 812bcf42a
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 04:45 UTC

test(ts/e2e): run live agent journey evals with pass@k (#5354)

This PR: - part of [PRDE-2212](https://linear.app/composio/issue/PRDE-2212/cli-e2e-tests-fake-backend-operation-suites-and-agent-journeys-with) - builds on top of https://github.com/ComposioHQ/composio/pull/5353; split from https://github.com/ComposioHQ/composio/pull/5312 - adds manual repeated `eval.ts` runs through Claude Code 2.1.310 or the minimal loop, with and without the skill, reporting `pass@1`, `pass@k`, `pass^k`, and wrong-tree rates per journey and train/holdout split - fingerprints batches by skill, grading code, CLI binary, agent version, and model; reuses matching runs and refuses comparisons across different graders - keeps the Anthropic key in a host proxy that forwards only messages and token-count requests to fixed upstream URLs for allowed models - bounds live runs by time, command count, and cost, excludes infrastructure failures from rates, and documents the measurement procedure in `HILLCLIMB.md` - adds token pricing, batch and metric tests, local `/results/` ignores, and the `eval` script; keeps the shared replay driver contract from the preceding PR ## Context Frozen install, CLI dependency builds, package typechecks, Prettier, all 102 keyless journey tests, and the inherited contract, operation, and upgrade suites passed. `eval.ts run --dry --model claude-haiku-5-5` also passed. Only keyless tests and the dry run were used here; no paid model run or recording was started. The historical baseline in #5312 had developer auth execution at 3/5 with the skill versus 5/5 without it; subsequent skill changes reached 5/5, and the link URL fix raised unconnected developer execution from 0/5 to 5/5. Those kept batches and their measured costs follow in the decision-record PR. Complete CLI workflow validation after the base merges and this PR is retargeted to `next`.

Alberto Schiabel committed Nov 16, 2026 at 12:51 UTC 812bcf42a2a7a02b9e56c50bac3f2ead4546a60e
17 files changed +1766 -2
ts/e2e-tests/_utils/Dockerfile.cli.dockerignore
+2
@@ -8,3 +8,5 @@
8 # node_modules.
9 **/node_modules
10 .git
11 +# Live eval batches: local, and they change on every run
12 +**/agent-journeys/results
ts/e2e-tests/cli/agent-journeys/.gitignore new
+2
@@ -0,0 +1,2 @@
1 +# Live eval batches: local measurements, see HILLCLIMB.md
2 +/results/
ts/e2e-tests/cli/agent-journeys/HILLCLIMB.md new
+65
@@ -0,0 +1,65 @@
1 +# Hill-climbing the composio skills
2 +
3 +This is how to tell whether a change to the `composio-cli` skill (or to the CLI's help and error text) makes agents better at using the CLI, and how to iterate on it. It is written for a person or for a coding agent doing the iterating.
4 +
5 +The replay suite (`e2e.test.ts`) is not the measure. It replays fixed model answers, so a skill change cannot move it. The measure is `eval.ts`: live, repeated runs of the journeys with the real agent, graded by the same checks.
6 +
7 +## What a run is
8 +
9 +One journey, one arm, one attempt:
10 +
11 +- **Agent.** Claude Code 2.1.310, headless, in the journey container (`--agent claude-code`, the default). The skill is installed in `~/.claude/skills/composio-cli/`, among Claude Code's own skills, so the agent has to find it from its description, as a user's agent does. `--agent loop` instead gives the skill to the minimal loop as its system prompt: cheaper to reason about, but it measures only what the skill says, not whether it is found.
12 +- **Arm.** `skill` installs the skill built from the working tree; `no-skill` installs nothing. The difference is what the skill is worth.
13 +- **Model.** `claude-haiku-5-5` or `claude-opus-5-5`; the model proxy refuses any other.
14 +- **Grade.** The journey's checks, which read what the fake backend recorded, never the agent's prose. A run passes when every check holds. Detours (a command on the wrong tree, a request to the wrong project) are counted apart and never fail a run.
15 +
16 +## The numbers
17 +
18 +For each journey and arm, over its graded runs:
19 +
20 +| number | meaning | what to watch |
21 +| ---------- | ----------------------------------------- | -------------------------------------------------------------- |
22 +| pass@1 | share of runs that passed | the plain success rate, with a 95% interval |
23 +| pass@k | chance that at least one of k runs passes | whether the agent can do it at all |
24 +| pass^k | chance that all k runs pass | whether a user can rely on it; the number to raise |
25 +| wrong tree | share of runs with at least one detour | the root-tree / `composio dev` confusion this suite exists for |
26 +
27 +Splits are averaged per journey, so a journey with more runs does not weigh more. `train` journeys are the ones changes are tuned against; `holdout` journeys only confirm.
28 +
29 +## The loop
30 +
31 +The key is loaded only into the command's environment:
32 +
33 +```sh
34 +cd ts/e2e-tests/cli/agent-journeys
35 +set -a; . ~/.config/composio-e2e/llm.env; set +a
36 +```
37 +
38 +1. **Check the setup.** `bun run eval.ts run --model claude-haiku-5-5 --dry` prints the plan and spends nothing; `--smoke` runs one journey once.
39 +2. **Measure the baseline.** Both arms, every journey, three runs each:
40 + `bun run eval.ts run --model claude-haiku-5-5 --runs 3 --arms skill,no-skill --label baseline`
41 +3. **Read the failures.** `results/<batch>/summary.md` lists which checks failed how often. `results/<batch>/logs/<journey>.<arm>.<attempt>.json` has the commands, what they printed, what the agent wrote, the requests the fake received, and every Claude Code event. Find what the agent misunderstood before changing anything.
42 +4. **Make one change.** Edit the skill's source in `ts/packages/cli/skills-src/composio-cli/`, or the CLI text the agent misread. Keep it to one idea, so a result can be attributed to it.
43 +5. **Measure the train split.** `bun run eval.ts run --model claude-haiku-5-5 --runs 3 --split train --label <change>`, then
44 + `bun run eval.ts compare <baseline batch> <change batch>`.
45 +6. **Decide.** Keep the change when the train split's mean pass^k rises and no train journey's pass@1 falls by more than one run. A difference of one run out of three is noise: when it decides the outcome, run both batches again with `--runs 5` (earlier runs with the same fingerprint are reused, so only the extra runs are paid for).
46 +7. **Confirm.** Run the holdout split on the kept change and on the baseline, and the train split on `claude-opus-5-5`. A change that helps train and hurts holdout fits the journeys, not the agent: drop it.
47 +8. **Record.** Results are git-ignored. Keep the batches a decision rests on with `bun run eval.ts keep <batch> <decision>`, which copies their records (not the logs) into `evals/<decision>/`, and write the decision's `README.md` there: what the runs showed, the change, the tables, the commands to repeat it.
48 +
49 +## Rules
50 +
51 +- **Do not change the harness in the same step as the skill.** The journeys, the checks, the person and the fake backend (`agent-journeys/src/`, `fake-backend/src/`) are what grade a run; their hash is part of every batch, and `compare` refuses two batches with different hashes. Fix a grader in its own change, then measure a new baseline.
52 +- **Do not tune the checks to the agent.** A check that fails because the agent took a valid path the check did not foresee is a grader bug; fix it with a test in `journeys.test.ts`. A check that fails because the agent did the wrong thing is the signal.
53 +- **Do not add the journeys' wording to the skill.** The skill is for every user's task; a sentence that only matches a journey's prompt raises train and nothing else.
54 +- **Spend little.** A Haiku run costs about $0.002 to $0.01; Opus about 40 times that. `--max-cost-usd` (default $2) stops a batch from starting runs past it, and `--run-budget-usd` caps one run.
55 +
56 +## Adding a journey
57 +
58 +A journey that no agent fails cannot separate two skills, and one that every agent fails cannot either. Add one when a live run shows a failure no journey catches: the prompt a user would type, the state it starts from, checks on what the fake recorded, and a `split`. `journeys.test.ts` then checks that an agent doing nothing fails it.
59 +
60 +## Limits
61 +
62 +See "Measuring a change" in the [README](./README.md#measuring-a-change): outbound traffic is not blocked, requests of overlapping or background commands are attributed together, and five journeys are a thin train and holdout set. Two more:
63 +
64 +- `composio login` installs the skill for real users, so a `no-skill` run that logs in may end with the skill installed. That is the product's behaviour, and it means `no-skill` measures "not installed at the start".
65 +- Claude Code's own skills are present in both arms, as they are for a user.
ts/e2e-tests/cli/agent-journeys/README.md
+22
@@ -159,6 +159,28 @@ How to read it:
159 - Set 01's `no` for `claude-opus-5-5` on `developer-execute-unconnected` was a handoff nobody answered: it gave the person the URL and stopped. With a person who answers, it finishes (02, 03).
160 - In sets 02 and 03, a run that reached the dev tree and then failed did so at the not-connected error; a run that failed on the root tree did so at `composio link slack --no-browser`.
161
162 +## Measuring a change
163 +
164 +The replay suite answers "does the CLI still behave on paths models have taken". It cannot answer "does this change make models do better", because a replay does not consult the model. Recording does, and the pieces here are meant to be the base of that measurement:
165 +
166 +- **Environment**: `startCliRig()` gives a fresh container with the real binary, a seeded backend that every host the CLI knows is pointed at, and a journal of which project and user every request acted on. It needs no account and changes nothing outside the fake as long as the agent stays on `composio`; the container's other outbound traffic is not blocked. Each run has its own container and backend, so runs are repeatable and can run in parallel.
167 +- **Task**: a journey's `prompt`, `setup`, and starting directory.
168 +- **Grader**: a journey's `checks` for success and its `detours` for how the run got there; see [How a run is graded](#how-a-run-is-graded). They read exit codes, command output, and the journal, never the model's prose, so a grade is a fact and not a judgement. Each is reported by name, which says where a failed run went wrong (wrong project, never linked, wrong text) and not only that it failed. Compute the wrong-tree rate from the detours of every run, the failed ones included.
169 +- **Subject**: the skill text, rendered from source on every run and identified by `skillSha256`; and the CLI, identified by `revision`.
170 +- **Person**: fixed, so two runs differ only in the subject and the model.
171 +
172 +To compare two versions of the skill or the CLI, use `eval.ts`, not `record.ts`. It runs every journey k times per arm with Claude Code (or the minimal loop) against the real API, grades each run with the same checks, and reports pass@1, pass@k, pass^k and the wrong-tree rate per journey and split. [HILLCLIMB.md](./HILLCLIMB.md) is the procedure. A batch records what it measured: the skill's hash, a hash of the harness (journeys, graders, person, fake backend), the CLI image id, the agent and its version, and the model. Runs with the same fingerprint are reused instead of paid for again, and `compare` refuses batches graded by different harnesses.
173 +
174 +What the live runner bounds: each run's agent has six minutes, forty shell commands, and a dollar cap Claude Code enforces itself; a batch stops starting runs at `--max-cost-usd`. A run that fails for reasons outside the agent (the harness threw, the proxy refused a request) is recorded as `infra` and left out of every rate.
175 +
176 +What it still does not have:
177 +
178 +- **Enforced isolation.** The container reaches the fake because every host the CLI knows is redirected to it, not because other traffic is blocked. An agent without the skill has tried to call the production backend directly with the fake project's keys. It failed only because the image has no `curl`; those keys work nowhere else, but nothing would stop the request.
179 +- **Exact attribution in Claude Code runs.** Requests are attributed to a command by the time between its call and its result. Commands that overlap, or that the agent leaves running in the background, share requests.
180 +- **Breadth.** Every journey starts with the binary installed and uses the org's default developer project, where a request with no project scope lands on the right project by the fake's fallback. There is no journey for installing or upgrading, for a non-default project, for a script read from a file or stdin, and the person never declines, is never slow, and always has a browser. Five journeys split three and two are a thin train and held-out set.
181 +
182 +Add a journey when a real agent failure is found that no journey would have caught: write the prompt a user would type, the state it starts from, and checks on the journal. The cases that separate versions best are the ones where the two trees are easy to confuse, a connection is missing, or the person has to act.
183 +
184 ## The skill's commands
185
186 `skill-commands.test.ts` needs no model: it takes every `composio …` line from the code blocks of the built skill and its references and looks its command path up in the binary.
ts/e2e-tests/cli/agent-journeys/batch.test.ts new
+119
@@ -0,0 +1,119 @@
1 +import { describe, expect, it } from 'bun:test';
2 +import {
3 + fingerprint,
4 + renderComparison,
5 + summarize,
6 + type BatchInfo,
7 + type RunRecord,
8 +} from './src/batch';
9 +
10 +const run = (fields: Partial<RunRecord>): RunRecord => ({
11 + journey: 'consumer-slack-message',
12 + split: 'train',
13 + arm: 'skill',
14 + attempt: 1,
15 + fingerprint: 'f',
16 + outcome: 'passed',
17 + checks: [],
18 + detours: [],
19 + stop: 'finished',
20 + modelTurns: 3,
21 + commands: [],
22 + costUsd: 0.01,
23 + durationMs: 1000,
24 + ...fields,
25 +});
26 +
27 +const info = (fields: Partial<BatchInfo> = {}): BatchInfo => ({
28 + id: 'batch',
29 + startedAt: '2026-11-11T00:00:00.000Z',
30 + agent: 'claude-code',
31 + agentVersion: '2.1.310',
32 + model: 'claude-haiku-5-5',
33 + runs: 3,
34 + arms: ['skill'],
35 + journeys: ['consumer-slack-message'],
36 + revision: 'abc',
37 + dirty: false,
38 + skillSha256: 'skill-a',
39 + harnessSha256: 'harness-a',
40 + cliSha256: 'cli-a',
41 + maxCostUsd: 1,
42 + ...fields,
43 +});
44 +
45 +describe('summarize', () => {
46 + it('leaves infra failures out of every rate', () => {
47 + const [cell] = summarize([
48 + run({ attempt: 1 }),
49 + run({ attempt: 2, outcome: 'failed', checks: [{ name: 'it went to #general', ok: false }] }),
50 + run({ attempt: 3, outcome: 'infra' }),
51 + ]);
52 + expect(cell?.n).toBe(2);
53 + expect(cell?.infra).toBe(1);
54 + expect(cell?.passAt1).toBe(0.5);
55 + expect(cell?.passAtK).toBe(1);
56 + expect(cell?.passHatK).toBe(0);
57 + expect(cell?.failedChecks).toEqual({ 'it went to #general': 1 });
58 + });
59 +
60 + it('counts a run with a detour toward the wrong-tree rate, passed or not', () => {
61 + const detour = [{ name: 'no `composio dev` command ran', ok: false }];
62 + const [cell] = summarize([run({ detours: detour }), run({ attempt: 2 })]);
63 + expect(cell?.wrongTreeRate).toBe(0.5);
64 + expect(cell?.passAt1).toBe(1);
65 + });
66 +});
67 +
68 +describe('fingerprint', () => {
69 + const parts = {
70 + journey: 'first-run-login',
71 + agent: 'claude-code' as const,
72 + agentVersion: '2.1.310',
73 + model: 'claude-haiku-5-5',
74 + harnessSha256: 'h',
75 + cliSha256: 'i',
76 + };
77 +
78 + it('changes with the skill on the skill arm only', () => {
79 + expect(fingerprint({ ...parts, arm: 'skill', skillSha256: 'a' })).not.toBe(
80 + fingerprint({ ...parts, arm: 'skill', skillSha256: 'b' })
81 + );
82 + expect(fingerprint({ ...parts, arm: 'no-skill', skillSha256: 'a' })).toBe(
83 + fingerprint({ ...parts, arm: 'no-skill', skillSha256: 'b' })
84 + );
85 + });
86 +
87 + it('changes with the harness, the CLI and the model', () => {
88 + const base = fingerprint({ ...parts, arm: 'skill', skillSha256: 'a' });
89 + expect(fingerprint({ ...parts, arm: 'skill', skillSha256: 'a', harnessSha256: 'x' })).not.toBe(
90 + base
91 + );
92 + expect(fingerprint({ ...parts, arm: 'skill', skillSha256: 'a', cliSha256: 'x' })).not.toBe(
93 + base
94 + );
95 + expect(
96 + fingerprint({ ...parts, arm: 'skill', skillSha256: 'a', model: 'claude-opus-5-5' })
97 + ).not.toBe(base);
98 + });
99 +});
100 +
101 +describe('renderComparison', () => {
102 + const baseline = { info: info(), runs: [run({ outcome: 'failed' }), run({ attempt: 2 })] };
103 +
104 + it('reports the change in pass@1 per journey and split', () => {
105 + const candidate = {
106 + info: info({ id: 'after', skillSha256: 'skill-b' }),
107 + runs: [run({}), run({ attempt: 2 })],
108 + };
109 + const text = renderComparison(baseline, candidate, { arm: 'skill', allowHarnessChange: false });
110 + expect(text).toContain('| train | consumer-slack-message | 50% → 100% | +50 |');
111 + });
112 +
113 + it('refuses batches graded by different harnesses', () => {
114 + const candidate = { info: info({ harnessSha256: 'harness-b' }), runs: [run({})] };
115 + expect(() =>
116 + renderComparison(baseline, candidate, { arm: 'skill', allowHarnessChange: false })
117 + ).toThrow(/different harnesses/);
118 + });
119 +});
ts/e2e-tests/cli/agent-journeys/eval.ts new
+449
@@ -0,0 +1,449 @@
1 +/**
2 + * Live, repeated journey runs: the measurement a skill change is judged by.
3 + *
4 + * bun run eval.ts run --model claude-haiku-5-5 [options]
5 + * bun run eval.ts report <batch>
6 + * bun run eval.ts compare <baseline batch> <candidate batch> [--arm skill]
7 + * bun run eval.ts keep <batch> <decision-name>
8 + *
9 + * A batch is named by its id under `results/`, or by the path of a batch
10 + * directory kept in the repository (see `evals/`).
11 + *
12 + * `run` options:
13 + * --agent claude-code|loop Claude Code with the skill installed (default), or
14 + * the minimal loop with the skill as system prompt
15 + * --runs <k> runs per journey and arm (default 3)
16 + * --arms skill,no-skill which arms to run (default skill)
17 + * --split train|holdout|all which journeys (default all)
18 + * --journeys <id,id> exactly these journeys
19 + * --max-cost-usd <usd> stop starting runs once this much is spent (default 2)
20 + * --run-budget-usd <usd> Claude Code stops a single run at this cost (default 0.5)
21 + * --concurrency <n> runs at once (default 2)
22 + * --label <name> appended to the batch id
23 + * --force run everything, reusing no earlier run
24 + * --dry print the plan; start nothing, spend nothing
25 + * --smoke one journey, one run, to check the setup
26 + *
27 + * The API key is read from ANTHROPIC_API_KEY and stays in this process: the
28 + * agent reaches the model through `src/model-proxy.ts`. Results go to
29 + * `results/<batch>/` (git-ignored). See HILLCLIMB.md for how to use them.
30 + */
31 +
32 +import { execFileSync } from 'node:child_process';
33 +import { createHash } from 'node:crypto';
34 +import { appendFileSync, existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
35 +import { join, relative, resolve } from 'node:path';
36 +import { parseArgs } from 'node:util';
37 +import { startJourneyContainer } from '@e2e-tests/utils';
38 +import { z } from 'zod';
39 +import { anthropicAdapter } from './src/anthropic-adapter';
40 +import {
41 + AGENTS,
42 + ARMS,
43 + fingerprint,
44 + previousRuns,
45 + readBatch,
46 + renderComparison,
47 + renderSummary,
48 + treeHash,
49 + type AgentName,
50 + type Arm,
51 + type BatchInfo,
52 + type RunRecord,
53 +} from './src/batch';
54 +import { CLAUDE_CODE_VERSION, claudeCodeDriver, ensureClaudeCodeBinary } from './src/claude-code';
55 +import { MAX_OUTPUT_TOKENS } from './src/adapters';
56 +import { JOURNEYS, type Journey, type JourneyRun } from './src/journeys';
57 +import { ALLOWED_MODELS, startModelProxy, type ModelProxy } from './src/model-proxy';
58 +import { loopDriver, runJourneyWith, type Driver } from './src/run-journey';
59 +import { buildSkill } from './src/skill';
60 +
61 +const HERE = import.meta.dir;
62 +const RESULTS = join(HERE, 'results');
63 +const REPO_TS = resolve(HERE, '..', '..', '..');
64 +/**
65 + * What drives and grades a run: the journeys and their checks, the person,
66 + * the agents, and the fake backend. A change here makes earlier results
67 + * incomparable. The reporting code (`batch.ts`, `metrics.ts`) is left out.
68 + */
69 +const HARNESS_PATHS = [
70 + 'e2e-tests/cli/fake-backend/src',
71 + ...[
72 + 'agent.ts',
73 + 'anthropic-adapter.ts',
74 + 'claude-code.ts',
75 + 'journeys.ts',
76 + 'person.ts',
77 + 'run-journey.ts',
78 + 'scrub.ts',
79 + 'skill.ts',
80 + ].map(file => `e2e-tests/cli/agent-journeys/src/${file}`),
81 +];
82 +/** Wall-clock limit for one run's agent. */
83 +const RUN_DEADLINE_MS = 6 * 60_000;
84 +/** Shell commands one run may start. The longest recorded run used eleven. */
85 +const MAX_COMMANDS = 40;
86 +
87 +const fail = (message: string): never => {
88 + console.error(message);
89 + process.exit(2);
90 +};
91 +
92 +const git = (...args: string[]): string =>
93 + execFileSync('git', args, { cwd: HERE, encoding: 'utf8' }).trim();
94 +
95 +/** A batch id under `results/`, or the path of a batch directory kept elsewhere (`evals/`). */
96 +const batchDirectory = (name: string): string =>
97 + existsSync(join(name, 'batch.json')) ? resolve(name) : resolve(RESULTS, name);
98 +
99 +const [command, ...rest] = process.argv.slice(2);
100 +
101 +if (command === 'report') {
102 + const [batch] = rest;
103 + if (!batch) fail('Usage: bun run eval.ts report <batch>');
104 + const { info, runs } = readBatch(batchDirectory(batch!));
105 + process.stdout.write(renderSummary(info, runs));
106 + process.exit(0);
107 +}
108 +
109 +if (command === 'keep') {
110 + const [batch, decision] = rest;
111 + if (!batch || !decision || !/^[a-z0-9][a-z0-9-]*$/.test(decision)) {
112 + fail('Usage: bun run eval.ts keep <batch> <decision-name>');
113 + }
114 + const source = batchDirectory(batch!);
115 + const { info } = readBatch(source);
116 + const target = join(HERE, 'evals', decision!, info.id);
117 + mkdirSync(target, { recursive: true });
118 + // The records and the summary, not the logs: they are large and hold every command's output.
119 + for (const name of ['batch.json', 'runs.jsonl', 'summary.md']) {
120 + const text = readFileSync(join(source, name), 'utf8');
121 + if (/sk-ant-|sk-journey-/.test(text)) fail(`${name} holds a key or a proxy token; not kept.`);
122 + writeFileSync(join(target, name), text);
123 + }
124 + console.log(`kept ${relative(process.cwd(), target)}`);
125 + process.exit(0);
126 +}
127 +
128 +if (command === 'compare') {
129 + const { values, positionals } = parseArgs({
130 + args: rest,
131 + allowPositionals: true,
132 + options: {
133 + arm: { type: 'string', default: 'skill' },
134 + 'allow-harness-change': { type: 'boolean', default: false },
135 + },
136 + });
137 + const [baseline, candidate] = positionals;
138 + const arm = z.enum(ARMS).safeParse(values.arm);
139 + if (!baseline || !candidate || !arm.success) {
140 + fail(
141 + 'Usage: bun run eval.ts compare <baseline batch> <candidate batch> [--arm skill|no-skill]'
142 + );
143 + }
144 + process.stdout.write(
145 + renderComparison(readBatch(batchDirectory(baseline!)), readBatch(batchDirectory(candidate!)), {
146 + arm: arm.data!,
147 + allowHarnessChange: values['allow-harness-change'] ?? false,
148 + })
149 + );
150 + process.exit(0);
151 +}
152 +
153 +if (command !== 'run') {
154 + fail('Usage: bun run eval.ts run|report|compare|keep … (see the header of eval.ts)');
155 +}
156 +
157 +const { values } = parseArgs({
158 + args: rest,
159 + options: {
160 + model: { type: 'string' },
161 + agent: { type: 'string', default: 'claude-code' },
162 + runs: { type: 'string', default: '3' },
163 + arms: { type: 'string', default: 'skill' },
164 + split: { type: 'string', default: 'all' },
165 + journeys: { type: 'string' },
166 + 'max-cost-usd': { type: 'string', default: '2' },
167 + 'run-budget-usd': { type: 'string', default: '0.5' },
168 + concurrency: { type: 'string', default: '2' },
169 + label: { type: 'string' },
170 + force: { type: 'boolean', default: false },
171 + dry: { type: 'boolean', default: false },
172 + smoke: { type: 'boolean', default: false },
173 + },
174 +});
175 +
176 +const Options = z.object({
177 + model: z.enum(ALLOWED_MODELS),
178 + agent: z.enum(AGENTS),
179 + runs: z.coerce.number().int().min(1).max(20),
180 + arms: z.array(z.enum(ARMS)).min(1),
181 + split: z.enum(['train', 'holdout', 'all']),
182 + maxCostUsd: z.coerce.number().positive(),
183 + runBudgetUsd: z.coerce.number().positive(),
184 + concurrency: z.coerce.number().int().min(1).max(8),
185 + label: z
186 + .string()
187 + .regex(/^[a-z0-9][a-z0-9-]*$/)
188 + .optional(),
189 +});
190 +const parsed = Options.safeParse({
191 + model: values.model,
192 + agent: values.agent,
193 + runs: values.smoke ? '1' : values.runs,
194 + arms: values.arms?.split(',').map(arm => arm.trim()),
195 + split: values.split,
196 + maxCostUsd: values['max-cost-usd'],
197 + runBudgetUsd: values['run-budget-usd'],
198 + concurrency: values.concurrency,
199 + label: values.label,
200 +});
201 +if (!parsed.success) {
202 + fail(`${z.prettifyError(parsed.error)}\n\nModels: ${ALLOWED_MODELS.join(', ')}`);
203 +}
204 +const options = parsed.data!;
205 +
206 +const named = values.journeys?.split(',').map(id => id.trim());
207 +const unknown = named?.filter(id => !JOURNEYS.some(journey => journey.id === id)) ?? [];
208 +if (unknown.length > 0) fail(`Unknown journeys: ${unknown.join(', ')}`);
209 +let journeys: ReadonlyArray<Journey> = JOURNEYS.filter(
210 + journey =>
211 + (named === undefined || named.includes(journey.id)) &&
212 + (options.split === 'all' || journey.split === options.split)
213 +);
214 +if (values.smoke) journeys = journeys.slice(0, 1);
215 +if (journeys.length === 0) fail('No journey matches.');
216 +
217 +const apiKey = process.env.ANTHROPIC_API_KEY;
218 +if (!apiKey && !values.dry) fail('ANTHROPIC_API_KEY is not set; nothing was started.');
219 +
220 +// What the batch measures. The CLI is the binary a journey container would
221 +// run now: starting one builds the image when the checkout changed. The
222 +// binary is hashed, not the image, which also holds the test sources.
223 +const probe = await startJourneyContainer({ label: 'eval-probe' });
224 +const binaries = await probe.exec('cd /usr/local/bin && sha256sum composio *.mjs | sha256sum', {
225 + cwd: '/',
226 +});
227 +await probe.stop();
228 +if (binaries.exitCode !== 0) fail(`Could not hash the CLI binary: ${binaries.stderr}`);
229 +const cliSha256 = binaries.stdout.split(' ')[0]!.trim();
230 +
231 +const agentVersion = options.agent === 'claude-code' ? CLAUDE_CODE_VERSION : 'loop';
232 +const skillSha256 = createHash('sha256').update(JSON.stringify(buildSkill())).digest('hex');
233 +const harnessSha256 = treeHash(REPO_TS, HARNESS_PATHS);
234 +const startedAt = new Date();
235 +const batchId = `${startedAt.toISOString().replace(/[:.]/g, '-').slice(0, 19)}-${options.agent}-${options.model}${options.label ? `-${options.label}` : ''}`;
236 +const info: BatchInfo = {
237 + id: batchId,
238 + startedAt: startedAt.toISOString(),
239 + agent: options.agent,
240 + agentVersion,
241 + model: options.model,
242 + runs: options.runs,
243 + arms: options.arms,
244 + journeys: journeys.map(journey => journey.id),
245 + revision: git('rev-parse', '--short', 'HEAD'),
246 + dirty: git('status', '--porcelain', '--', ':/ts/packages/cli', ':/ts/e2e-tests/cli') !== '',
247 + skillSha256,
248 + harnessSha256,
249 + cliSha256,
250 + maxCostUsd: options.maxCostUsd,
251 +};
252 +
253 +interface Planned {
254 + readonly journey: Journey;
255 + readonly arm: Arm;
256 + readonly attempt: number;
257 + readonly fingerprint: string;
258 +}
259 +
260 +const earlier = values.force ? new Map<string, RunRecord[]>() : previousRuns(RESULTS);
261 +const reused: RunRecord[] = [];
262 +const planned: Planned[] = [];
263 +for (const journey of journeys) {
264 + for (const arm of options.arms) {
265 + const print = fingerprint({
266 + journey: journey.id,
267 + arm,
268 + agent: options.agent,
269 + agentVersion,
270 + model: options.model,
271 + skillSha256,
272 + harnessSha256,
273 + cliSha256,
274 + });
275 + const available = (earlier.get(print) ?? []).slice(-options.runs);
276 + reused.push(...available.map((run, index) => ({ ...run, attempt: index + 1 })));
277 + for (let attempt = available.length + 1; attempt <= options.runs; attempt += 1) {
278 + planned.push({ journey, arm, attempt, fingerprint: print });
279 + }
280 + }
281 +}
282 +
283 +console.log(`batch ${batchId}`);
284 +console.log(`agent ${options.agent} ${agentVersion}, ${options.model}`);
285 +console.log(`journeys ${journeys.map(journey => `${journey.id} (${journey.split})`).join(', ')}`);
286 +console.log(`arms ${options.arms.join(', ')}; ${options.runs} run(s) each`);
287 +console.log(
288 + `runs ${planned.length} new, ${reused.length} reused; budget $${options.maxCostUsd}`
289 +);
290 +if (values.dry) {
291 + for (const run of planned)
292 + console.log(` would run ${run.journey.id} ${run.arm} #${run.attempt}`);
293 + process.exit(0);
294 +}
295 +
296 +const directory = join(RESULTS, batchId);
297 +mkdirSync(join(directory, 'logs'), { recursive: true });
298 +writeFileSync(join(directory, 'batch.json'), `${JSON.stringify(info, null, 2)}\n`);
299 +const record = (run: RunRecord) =>
300 + appendFileSync(join(directory, 'runs.jsonl'), `${JSON.stringify(run)}\n`);
301 +for (const run of reused) record(run);
302 +
303 +const proxy: ModelProxy = await startModelProxy({ apiKey: apiKey! });
304 +const binary = options.agent === 'claude-code' ? ensureClaudeCodeBinary() : undefined;
305 +let spent = 0;
306 +let interrupted = false;
307 +process.on('SIGINT', () => {
308 + interrupted = true;
309 + console.log('\ninterrupted: finishing the runs in flight, starting no more');
310 +});
311 +
312 +/** Nothing that could carry a credential is written. */
313 +const redact = (text: string): string => {
314 + const cleaned = text.replaceAll(proxy.token, '<proxy-token>');
315 + if (cleaned.includes(apiKey!) || /sk-ant-/.test(cleaned)) {
316 + throw new Error('Refusing to write a log that contains an API key.');
317 + }
318 + return cleaned;
319 +};
320 +
321 +const driverFor = (arm: Arm, budgetUsd: number, onEvent: (event: unknown) => void): Driver =>
322 + options.agent === 'claude-code'
323 + ? claudeCodeDriver({
324 + model: options.model,
325 + proxy,
326 + binary: binary!,
327 + withSkill: arm === 'skill',
328 + budgetUsd,
329 + deadlineMs: RUN_DEADLINE_MS,
330 + maxCommands: MAX_COMMANDS,
331 + onEvent,
332 + })
333 + : loopDriver(
334 + inputs =>
335 + anthropicAdapter({
336 + baseURL: 'https://api.anthropic.com',
337 + apiKey: apiKey!,
338 + model: options.model,
339 + maxTokens: MAX_OUTPUT_TOKENS,
340 + ...inputs,
341 + }),
342 + { skill: arm === 'skill' ? 'system-prompt' : 'none', scrubbed: false }
343 + );
344 +
345 +const runOne = async (plan: Planned): Promise<void> => {
346 + const name = `${plan.journey.id}.${plan.arm}.${plan.attempt}`;
347 + const remaining = options.maxCostUsd - spent;
348 + if (interrupted || remaining <= 0) {
349 + console.log(` skipped ${name}: ${interrupted ? 'interrupted' : 'budget spent'}`);
350 + return;
351 + }
352 + const events: unknown[] = [];
353 + const started = Date.now();
354 + const refusedBefore = proxy.refused().length;
355 + let run: JourneyRun | undefined;
356 + try {
357 + run = await runJourneyWith(
358 + plan.journey,
359 + driverFor(plan.arm, Math.min(options.runBudgetUsd, remaining), event => events.push(event))
360 + );
361 + const checks = plan.journey.checks(run);
362 + const detours = (plan.journey.detours?.(run) ?? []).filter(detour => !detour.ok);
363 + const stop =
364 + typeof run.transcript.stop === 'string' ? run.transcript.stop : run.transcript.stop.provider;
365 + // A harness or provider failure says nothing about the agent.
366 + const refused = proxy.refused().slice(refusedBefore);
367 + const infra = stop.startsWith('exited ') || refused.length > 0;
368 + const costUsd = run.usage?.costUsd ?? 0;
369 + spent += costUsd;
370 + const result: RunRecord = {
371 + journey: plan.journey.id,
372 + split: plan.journey.split,
373 + arm: plan.arm,
374 + attempt: plan.attempt,
375 + fingerprint: plan.fingerprint,
376 + outcome: infra ? 'infra' : checks.every(check => check.ok) ? 'passed' : 'failed',
377 + checks: checks.map(check => ({
378 + name: check.name,
379 + ok: check.ok,
380 + ...(check.ok ? {} : { seen: JSON.stringify(check.seen ?? null).slice(0, 500) }),
381 + })),
382 + detours: detours.map(detour => ({
383 + name: detour.name,
384 + ok: false,
385 + seen: JSON.stringify(detour.seen ?? null).slice(0, 500),
386 + })),
387 + stop,
388 + modelTurns: run.transcript.modelTurns,
389 + commands: run.transcript.commands.map(item => ({
390 + command: item.command,
391 + exitCode: item.exitCode,
392 + })),
393 + costUsd,
394 + durationMs: Date.now() - started,
395 + ...(infra ? { error: `stop: ${stop}; proxy refused: ${refused.join('; ')}` } : {}),
396 + };
397 + writeFileSync(
398 + join(directory, 'logs', `${name}.json`),
399 + redact(
400 + JSON.stringify(
401 + { ...result, transcript: run.transcript, requests: run.requests, events },
402 + null,
403 + 2
404 + )
405 + )
406 + );
407 + record(result);
408 + console.log(
409 + ` ${result.outcome.padEnd(6)} ${name} $${costUsd.toFixed(3)} ${result.commands.length} cmd ${detours.length} detour(s)`
410 + );
411 + } catch (error) {
412 + const message = error instanceof Error ? error.message : String(error);
413 + record({
414 + journey: plan.journey.id,
415 + split: plan.journey.split,
416 + arm: plan.arm,
417 + attempt: plan.attempt,
418 + fingerprint: plan.fingerprint,
419 + outcome: 'infra',
420 + checks: [],
421 + detours: [],
422 + stop: 'error',
423 + modelTurns: 0,
424 + commands: [],
425 + costUsd: 0,
426 + durationMs: Date.now() - started,
427 + error: redact(message).slice(0, 1000),
428 + });
429 + console.log(` infra ${name} ${message.slice(0, 200)}`);
430 + } finally {
431 + await run?.rig.stop();
432 + }
433 +};
434 +
435 +try {
436 + const queue = [...planned];
437 + await Promise.all(
438 + Array.from({ length: Math.min(options.concurrency, queue.length) }, async () => {
439 + for (let next = queue.shift(); next !== undefined; next = queue.shift()) await runOne(next);
440 + })
441 + );
442 +} finally {
443 + await proxy.close();
444 +}
445 +
446 +const summary = renderSummary(info, readBatch(directory).runs);
447 +writeFileSync(join(directory, 'summary.md'), summary);
448 +process.stdout.write(`\n${summary}`);
449 +console.log(`results ${directory}`);
ts/e2e-tests/cli/agent-journeys/metrics.test.ts new
+49
@@ -0,0 +1,49 @@
1 +import { describe, expect, it } from 'bun:test';
2 +import { passAtK, passHatK, wilson } from './src/metrics';
3 +
4 +describe('pass@k and pass^k', () => {
5 + it('agree with pass@1 when k is 1', () => {
6 + expect(passAtK(5, 3, 1)).toBeCloseTo(0.6);
7 + expect(passHatK(5, 3, 1)).toBeCloseTo(0.6);
8 + });
9 +
10 + it('are 1 or 0 when every run passed or none did', () => {
11 + expect(passAtK(3, 3, 3)).toBe(1);
12 + expect(passHatK(3, 3, 3)).toBe(1);
13 + expect(passAtK(3, 0, 3)).toBe(0);
14 + expect(passHatK(3, 0, 3)).toBe(0);
15 + });
16 +
17 + it('pass@k rises and pass^k falls as k grows', () => {
18 + // 2 of 3 runs passed.
19 + expect(passAtK(3, 2, 2)).toBe(1);
20 + expect(passHatK(3, 2, 2)).toBeCloseTo(1 / 3);
21 + expect(passHatK(3, 2, 3)).toBe(0);
22 + });
23 +
24 + it('match the closed forms for n=10, c=7, k=3', () => {
25 + // 1 - C(3,3)/C(10,3) and C(7,3)/C(10,3).
26 + expect(passAtK(10, 7, 3)).toBeCloseTo(1 - 1 / 120);
27 + expect(passHatK(10, 7, 3)).toBeCloseTo(35 / 120);
28 + });
29 +
30 + it('refuse counts that cannot happen', () => {
31 + expect(() => passAtK(3, 4, 1)).toThrow();
32 + expect(() => passHatK(3, 1, 4)).toThrow();
33 + expect(() => passHatK(3, 1, 0)).toThrow();
34 + });
35 +});
36 +
37 +describe('wilson', () => {
38 + it('brackets the observed rate', () => {
39 + const interval = wilson(2, 3);
40 + expect(interval.low).toBeLessThan(2 / 3);
41 + expect(interval.high).toBeGreaterThan(2 / 3);
42 + });
43 +
44 + it('stays inside [0, 1] at the edges', () => {
45 + expect(wilson(0, 3).low).toBe(0);
46 + expect(wilson(3, 3).high).toBe(1);
47 + expect(wilson(0, 0)).toEqual({ low: 0, high: 1 });
48 + });
49 +});
ts/e2e-tests/cli/agent-journeys/model-proxy.test.ts new
+84
@@ -0,0 +1,84 @@
1 +/**
2 + * The model proxy's refusals. Each request here is refused before anything is
3 + * forwarded, so the test needs no network and no key.
4 + */
5 +
6 +import { afterAll, beforeAll, describe, expect, it } from 'bun:test';
7 +import { request } from 'node:http';
8 +import { startModelProxy, type ModelProxy } from './src/model-proxy';
9 +
10 +let proxy: ModelProxy;
11 +let port: number;
12 +
13 +beforeAll(async () => {
14 + proxy = await startModelProxy({ apiKey: 'sk-not-a-real-key' });
15 + port = Number(new URL(proxy.dockerUrl).port);
16 +});
17 +
18 +afterAll(async () => {
19 + await proxy.close();
20 +});
21 +
22 +/**
23 + * Sends the path exactly as written. `fetch` would resolve `..` segments
24 + * before sending, so the proxy would never see them.
25 + */
26 +const send = (
27 + path: string,
28 + token: string,
29 + body: unknown = { model: 'claude-haiku-5-5' },
30 + method = 'POST'
31 +) =>
32 + new Promise<number>((resolve, reject) => {
33 + const outgoing = request(
34 + {
35 + host: '127.0.0.1',
36 + port,
37 + path,
38 + method,
39 + headers: { 'x-api-key': token, 'content-type': 'application/json' },
40 + },
41 + response => {
42 + response.resume();
43 + resolve(response.statusCode ?? 0);
44 + }
45 + );
46 + outgoing.on('error', reject);
47 + outgoing.end(method === 'GET' ? undefined : JSON.stringify(body));
48 + });
49 +
50 +describe('the model proxy', () => {
51 + it('refuses a request without its token', async () => {
52 + expect(await send('/v1/messages', 'sk-guess')).toBe(401);
53 + });
54 +
55 + it('refuses a path outside the API, however it is spelled', async () => {
56 + for (const path of [
57 + '/v2/messages',
58 + '/v1/../admin',
59 + '/v1/%2e%2e/admin',
60 + '/v1/messages/../models',
61 + ]) {
62 + expect(await send(path, proxy.token)).toBe(404);
63 + }
64 + });
65 +
66 + it('refuses API endpoints other than messages, where a model could hide', async () => {
67 + const batch = { requests: [{ custom_id: 'a', params: { model: 'claude-fable-5-2' } }] };
68 + expect(await send('/v1/messages/batches', proxy.token, batch)).toBe(404);
69 + expect(await send('/v1/models', proxy.token, undefined, 'GET')).toBe(404);
70 + expect(await send('/v1/messages', proxy.token, undefined, 'GET')).toBe(404);
71 + });
72 +
73 + it('refuses a model other than Haiku 5.5 and Opus 5.5', async () => {
74 + expect(await send('/v1/messages', proxy.token, { model: 'claude-fable-5-2' })).toBe(400);
75 + expect(proxy.refused().at(-1)).toBe('model claude-fable-5-2');
76 + });
77 +
78 + it('refuses a request that names no model', async () => {
79 + expect(await send('/v1/messages?beta=true', proxy.token, { messages: [] })).toBe(400);
80 + expect(await send('/v1/messages/count_tokens', proxy.token, 'not json')).toBe(400);
81 + expect(proxy.refused().at(-1)).toBe('model (none)');
82 + expect(proxy.forwarded()).toEqual({});
83 + });
84 +});
ts/e2e-tests/cli/agent-journeys/package.json
+2 -1
@@ -7,7 +7,8 @@
7 "typecheck": "tsc --noEmit",
8 "test:e2e": "bun test",
9 "test:e2e:cli": "bun test",
10 - "record": "bun run record.ts"
10 + "record": "bun run record.ts",
11 + "eval": "bun run eval.ts"
12 },
13 "devDependencies": {
14 "@anthropic-ai/sdk": "^0.135.0",
ts/e2e-tests/cli/agent-journeys/src/anthropic-adapter.ts
+10
@@ -5,6 +5,7 @@
5 import Anthropic from '@anthropic-ai/sdk';
6 import { z } from 'zod';
7 import { SHELL_TOOL, type ModelAdapter, type ModelTurn } from './agent';
8 +import { addTokens, ZERO_USAGE, type Usage } from './usage';
9
10 const ShellInput = z.object({ command: z.string() });
11
@@ -19,6 +20,7 @@ export const anthropicAdapter = (options: {
20 // No retries: a miss in strict replay, or a live failure while recording, must surface once.
21 const client = new Anthropic({ baseURL: options.baseURL, apiKey: options.apiKey, maxRetries: 0 });
22 const messages: Anthropic.MessageParam[] = [{ role: 'user', content: options.prompt }];
23 + let usage: Usage = ZERO_USAGE;
24
25 return {
26 provider: 'anthropic',
@@ -38,6 +40,13 @@ export const anthropicAdapter = (options: {
40 ],
41 messages,
42 });
43 + usage = addTokens(usage, options.model, {
44 + // A replayed answer may carry no usage at all.
45 + input: response.usage?.input_tokens ?? 0,
46 + output: response.usage?.output_tokens ?? 0,
47 + cacheRead: response.usage?.cache_read_input_tokens ?? 0,
48 + cacheWrite: response.usage?.cache_creation_input_tokens ?? 0,
49 + });
50 // The whole content goes back unchanged, thinking blocks included.
51 messages.push({ role: 'assistant', content: response.content });
52
@@ -65,5 +74,6 @@ export const anthropicAdapter = (options: {
74 say: message => {
75 messages.push({ role: 'user', content: message });
76 },
77 + usage: () => usage,
78 };
79 };
ts/e2e-tests/cli/agent-journeys/src/batch.ts new
+339
@@ -0,0 +1,339 @@
1 +/**
2 + * Batches of live runs: what each run recorded, how runs are reused, and the
3 + * summaries `eval.ts` prints.
4 + *
5 + * A batch is a directory under `results/`: `batch.json` says what was
6 + * measured, `runs.jsonl` has one line per run, `logs/` the full transcripts.
7 + */
8 +
9 +import { createHash } from 'node:crypto';
10 +import { existsSync, readdirSync, readFileSync, statSync } from 'node:fs';
11 +import { join, relative } from 'node:path';
12 +import { z } from 'zod';
13 +import { passAtK, passHatK, wilson } from './metrics';
14 +
15 +export const ARMS = ['skill', 'no-skill'] as const;
16 +export type Arm = (typeof ARMS)[number];
17 +
18 +export const AGENTS = ['claude-code', 'loop'] as const;
19 +export type AgentName = (typeof AGENTS)[number];
20 +
21 +const CheckRecord = z.object({ name: z.string(), ok: z.boolean(), seen: z.string().optional() });
22 +
23 +export const RunRecord = z.object({
24 + journey: z.string(),
25 + split: z.enum(['train', 'holdout']),
26 + arm: z.enum(ARMS),
27 + attempt: z.number(),
28 + fingerprint: z.string(),
29 + /**
30 + * `passed` and `failed` are graded. `infra` means the run says nothing about
31 + * the agent (the harness or the provider failed) and is left out of every rate.
32 + */
33 + outcome: z.enum(['passed', 'failed', 'infra']),
34 + checks: z.array(CheckRecord),
35 + /** Detours taken; never a failure, see `Journey.detours`. */
36 + detours: z.array(CheckRecord),
37 + stop: z.string(),
38 + modelTurns: z.number(),
39 + commands: z.array(z.object({ command: z.string(), exitCode: z.number() })),
40 + costUsd: z.number(),
41 + durationMs: z.number(),
42 + error: z.string().optional(),
43 + /** Set when the run was copied from an earlier batch with the same fingerprint. */
44 + reusedFrom: z.string().optional(),
45 +});
46 +export type RunRecord = z.infer<typeof RunRecord>;
47 +
48 +export const BatchInfo = z.object({
49 + id: z.string(),
50 + startedAt: z.string(),
51 + agent: z.enum(AGENTS),
52 + agentVersion: z.string(),
53 + model: z.string(),
54 + runs: z.number(),
55 + arms: z.array(z.enum(ARMS)),
56 + journeys: z.array(z.string()),
57 + revision: z.string(),
58 + /** Uncommitted changes under `ts/packages/cli` or the harness. */
59 + dirty: z.boolean(),
60 + skillSha256: z.string(),
61 + harnessSha256: z.string(),
62 + /** sha256 of the CLI binary and its companion modules, as the journey container has them. */
63 + cliSha256: z.string(),
64 + maxCostUsd: z.number(),
65 +});
66 +export type BatchInfo = z.infer<typeof BatchInfo>;
67 +
68 +/** sha256 of the given files and of every file under the given directories, by relative path. */
69 +export const treeHash = (root: string, paths: ReadonlyArray<string>): string => {
70 + const hash = createHash('sha256');
71 + const walk = (path: string) => {
72 + if (statSync(path).isDirectory()) {
73 + for (const name of readdirSync(path).sort()) {
74 + if (name !== 'node_modules') walk(join(path, name));
75 + }
76 + return;
77 + }
78 + hash
79 + .update(`${relative(root, path)}\0`)
80 + .update(readFileSync(path))
81 + .update('\0');
82 + };
83 + for (const path of paths) walk(join(root, path));
84 + return hash.digest('hex');
85 +};
86 +
87 +/**
88 + * What makes two runs interchangeable: the same journey, under the same
89 + * graders, with the same CLI, agent, model, and (for the skill arm) skill.
90 + */
91 +export const fingerprint = (parts: {
92 + journey: string;
93 + arm: Arm;
94 + agent: AgentName;
95 + agentVersion: string;
96 + model: string;
97 + skillSha256: string;
98 + harnessSha256: string;
99 + cliSha256: string;
100 +}): string =>
101 + createHash('sha256')
102 + .update(
103 + JSON.stringify({
104 + ...parts,
105 + skillSha256: parts.arm === 'skill' ? parts.skillSha256 : 'none',
106 + })
107 + )
108 + .digest('hex')
109 + .slice(0, 16);
110 +
111 +export const readRuns = (batchDirectory: string): RunRecord[] => {
112 + const path = join(batchDirectory, 'runs.jsonl');
113 + if (!existsSync(path)) return [];
114 + return readFileSync(path, 'utf8')
115 + .split('\n')
116 + .filter(line => line.trim() !== '')
117 + .map(line => RunRecord.parse(JSON.parse(line)));
118 +};
119 +
120 +export const readBatch = (batchDirectory: string): { info: BatchInfo; runs: RunRecord[] } => ({
121 + info: BatchInfo.parse(JSON.parse(readFileSync(join(batchDirectory, 'batch.json'), 'utf8'))),
122 + runs: readRuns(batchDirectory),
123 +});
124 +
125 +/** Graded runs from earlier batches, by fingerprint, oldest first. */
126 +export const previousRuns = (resultsDirectory: string): Map<string, RunRecord[]> => {
127 + const byFingerprint = new Map<string, RunRecord[]>();
128 + if (!existsSync(resultsDirectory)) return byFingerprint;
129 + for (const name of readdirSync(resultsDirectory).sort()) {
130 + const directory = join(resultsDirectory, name);
131 + if (!statSync(directory).isDirectory()) continue;
132 + for (const run of readRuns(directory)) {
133 + if (run.outcome === 'infra' || run.reusedFrom !== undefined) continue;
134 + const list = byFingerprint.get(run.fingerprint) ?? [];
135 + list.push({ ...run, reusedFrom: name });
136 + byFingerprint.set(run.fingerprint, list);
137 + }
138 + }
139 + return byFingerprint;
140 +};
141 +
142 +export interface CellSummary {
143 + readonly journey: string;
144 + readonly split: 'train' | 'holdout';
145 + readonly arm: Arm;
146 + /** Graded runs. */
147 + readonly n: number;
148 + readonly passed: number;
149 + readonly infra: number;
150 + readonly passAt1: number;
151 + readonly interval: { low: number; high: number };
152 + /** At k = the number of graded runs. */
153 + readonly passAtK: number;
154 + readonly passHatK: number;
155 + /** Share of graded runs that took at least one detour. */
156 + readonly wrongTreeRate: number;
157 + readonly meanCostUsd: number;
158 + readonly meanTurns: number;
159 + /** How often each check failed, by name. */
160 + readonly failedChecks: Readonly<Record<string, number>>;
161 +}
162 +
163 +export const summarize = (runs: ReadonlyArray<RunRecord>): CellSummary[] => {
164 + const cells = new Map<string, RunRecord[]>();
165 + for (const run of runs) {
166 + const key = `${run.journey}\0${run.arm}`;
167 + cells.set(key, [...(cells.get(key) ?? []), run]);
168 + }
169 + return [...cells.values()]
170 + .map(cell => {
171 + const [first] = cell;
172 + const graded = cell.filter(run => run.outcome !== 'infra');
173 + const n = graded.length;
174 + const passed = graded.filter(run => run.outcome === 'passed').length;
175 + const failedChecks: Record<string, number> = {};
176 + for (const run of graded) {
177 + for (const check of run.checks.filter(item => !item.ok)) {
178 + failedChecks[check.name] = (failedChecks[check.name] ?? 0) + 1;
179 + }
180 + }
181 + const mean = (values: number[]) =>
182 + values.length === 0 ? 0 : values.reduce((sum, value) => sum + value, 0) / values.length;
183 + return {
184 + journey: first!.journey,
185 + split: first!.split,
186 + arm: first!.arm,
187 + n,
188 + passed,
189 + infra: cell.length - n,
190 + passAt1: n === 0 ? 0 : passed / n,
191 + interval: wilson(passed, n),
192 + passAtK: n === 0 ? 0 : passAtK(n, passed, n),
193 + passHatK: n === 0 ? 0 : passHatK(n, passed, n),
194 + wrongTreeRate: n === 0 ? 0 : graded.filter(run => run.detours.length > 0).length / n,
195 + meanCostUsd: mean(cell.map(run => run.costUsd)),
196 + meanTurns: mean(graded.map(run => run.modelTurns)),
197 + failedChecks,
198 + };
199 + })
200 + .sort((left, right) =>
201 + left.split === right.split
202 + ? left.journey.localeCompare(right.journey) || left.arm.localeCompare(right.arm)
203 + : left.split === 'train'
204 + ? -1
205 + : 1
206 + );
207 +};
208 +
209 +const percent = (value: number) => `${Math.round(value * 100)}%`;
210 +const money = (value: number) => `$${value.toFixed(value < 1 ? 3 : 2)}`;
211 +
212 +/** Mean over journeys, so a journey with more runs does not count for more. */
213 +const splitMean = (
214 + cells: ReadonlyArray<CellSummary>,
215 + split: 'train' | 'holdout',
216 + arm: Arm,
217 + pick: (cell: CellSummary) => number
218 +): number | undefined => {
219 + const chosen = cells.filter(cell => cell.split === split && cell.arm === arm && cell.n > 0);
220 + return chosen.length === 0
221 + ? undefined
222 + : chosen.reduce((sum, cell) => sum + pick(cell), 0) / chosen.length;
223 +};
224 +
225 +export const renderSummary = (info: BatchInfo, runs: ReadonlyArray<RunRecord>): string => {
226 + const cells = summarize(runs);
227 + const totalCost = runs
228 + .filter(run => run.reusedFrom === undefined)
229 + .reduce((sum, run) => sum + run.costUsd, 0);
230 + const lines = [
231 + `# Batch ${info.id}`,
232 + '',
233 + `${info.agent} ${info.agentVersion}, ${info.model}, ${info.runs} run(s) per journey and arm.`,
234 + `Revision ${info.revision}${info.dirty ? ' with uncommitted changes' : ''}; skill ${info.skillSha256.slice(0, 12)}; harness ${info.harnessSha256.slice(0, 12)}.`,
235 + `Spent ${money(totalCost)} on new runs; ${runs.filter(run => run.reusedFrom !== undefined).length} run(s) reused.`,
236 + '',
237 + '| split | journey | arm | passed | pass@1 (95% CI) | pass@k | pass^k | wrong tree | mean cost | infra |',
238 + '|---|---|---|---|---|---|---|---|---|---|',
239 + ...cells.map(
240 + cell =>
241 + `| ${cell.split} | ${cell.journey} | ${cell.arm} | ${cell.passed}/${cell.n} | ${percent(cell.passAt1)} (${percent(cell.interval.low)}–${percent(cell.interval.high)}) | ${percent(cell.passAtK)} | ${percent(cell.passHatK)} | ${percent(cell.wrongTreeRate)} | ${money(cell.meanCostUsd)} | ${cell.infra} |`
242 + ),
243 + '',
244 + '| split | arm | mean pass@1 | mean pass^k | mean wrong tree |',
245 + '|---|---|---|---|---|',
246 + ];
247 + for (const split of ['train', 'holdout'] as const) {
248 + for (const arm of info.arms) {
249 + const passAt1 = splitMean(cells, split, arm, cell => cell.passAt1);
250 + if (passAt1 === undefined) continue;
251 + lines.push(
252 + `| ${split} | ${arm} | ${percent(passAt1)} | ${percent(splitMean(cells, split, arm, cell => cell.passHatK) ?? 0)} | ${percent(splitMean(cells, split, arm, cell => cell.wrongTreeRate) ?? 0)} |`
253 + );
254 + }
255 + }
256 + const failures = cells.filter(cell => Object.keys(cell.failedChecks).length > 0);
257 + if (failures.length > 0) {
258 + lines.push('', '## Failed checks', '');
259 + for (const cell of failures) {
260 + lines.push(`- ${cell.journey} (${cell.arm}):`);
261 + for (const [name, count] of Object.entries(cell.failedChecks)) {
262 + lines.push(` - ${count}/${cell.n} ${name}`);
263 + }
264 + }
265 + }
266 + return `${lines.join('\n')}\n`;
267 +};
268 +
269 +/**
270 + * Baseline against candidate, per journey and per split, on the skill arm.
271 + * Refuses batches graded by different harnesses: their numbers do not compare.
272 + */
273 +export const renderComparison = (
274 + baseline: { info: BatchInfo; runs: ReadonlyArray<RunRecord> },
275 + candidate: { info: BatchInfo; runs: ReadonlyArray<RunRecord> },
276 + options: { arm: Arm; allowHarnessChange: boolean }
277 +): string => {
278 + const problems: string[] = [];
279 + if (baseline.info.harnessSha256 !== candidate.info.harnessSha256) {
280 + problems.push(
281 + 'The batches were graded by different harnesses (journeys, graders, person or fake backend changed).'
282 + );
283 + }
284 + if (baseline.info.model !== candidate.info.model) {
285 + problems.push(`Different models: ${baseline.info.model} and ${candidate.info.model}.`);
286 + }
287 + if (baseline.info.agent !== candidate.info.agent) {
288 + problems.push(`Different agents: ${baseline.info.agent} and ${candidate.info.agent}.`);
289 + }
290 + if (problems.length > 0 && !options.allowHarnessChange) {
291 + throw new Error(`${problems.join('\n')}\nPass --allow-harness-change to compare anyway.`);
292 + }
293 +
294 + const before = new Map(
295 + summarize(baseline.runs)
296 + .filter(cell => cell.arm === options.arm)
297 + .map(cell => [cell.journey, cell])
298 + );
299 + const after = summarize(candidate.runs).filter(cell => cell.arm === options.arm);
300 + const signed = (value: number) => `${value > 0 ? '+' : ''}${Math.round(value * 100)}`;
301 + const lines = [
302 + `# ${baseline.info.id} → ${candidate.info.id} (${options.arm} arm)`,
303 + '',
304 + ...problems.map(problem => `> ${problem}`),
305 + ...(problems.length > 0 ? [''] : []),
306 + `Skill ${baseline.info.skillSha256.slice(0, 12)} → ${candidate.info.skillSha256.slice(0, 12)}.`,
307 + '',
308 + '| split | journey | pass@1 | Δ pass@1 | pass^k | Δ pass^k | wrong tree | Δ wrong tree |',
309 + '|---|---|---|---|---|---|---|---|',
310 + ];
311 + const deltas = { train: [] as number[][], holdout: [] as number[][] };
312 + for (const cell of after) {
313 + const old = before.get(cell.journey);
314 + if (!old || old.n === 0 || cell.n === 0) continue;
315 + const delta = [
316 + cell.passAt1 - old.passAt1,
317 + cell.passHatK - old.passHatK,
318 + cell.wrongTreeRate - old.wrongTreeRate,
319 + ];
320 + deltas[cell.split].push(delta);
321 + lines.push(
322 + `| ${cell.split} | ${cell.journey} | ${percent(old.passAt1)} → ${percent(cell.passAt1)} | ${signed(delta[0]!)} | ${percent(old.passHatK)} → ${percent(cell.passHatK)} | ${signed(delta[1]!)} | ${percent(old.wrongTreeRate)} → ${percent(cell.wrongTreeRate)} | ${signed(delta[2]!)} |`
323 + );
324 + }
325 + lines.push(
326 + '',
327 + '| split | journeys | mean Δ pass@1 | mean Δ pass^k | mean Δ wrong tree |',
328 + '|---|---|---|---|---|'
329 + );
330 + for (const split of ['train', 'holdout'] as const) {
331 + const rows = deltas[split];
332 + if (rows.length === 0) continue;
333 + const mean = (index: number) => rows.reduce((sum, row) => sum + row[index]!, 0) / rows.length;
334 + lines.push(
335 + `| ${split} | ${rows.length} | ${signed(mean(0))} | ${signed(mean(1))} | ${signed(mean(2))} |`
336 + );
337 + }
338 + return `${lines.join('\n')}\n`;
339 +};
ts/e2e-tests/cli/agent-journeys/src/claude-code.ts new
+337
@@ -0,0 +1,337 @@
1 +/**
2 + * Claude Code as the agent of a journey.
3 + *
4 + * The minimal loop in `agent.ts` puts the skill in its system prompt, so it
5 + * measures whether the skill's text steers a model that has already read it.
6 + * A user's agent finds the skill on its own, among others, from its
7 + * description. This driver runs the real harness inside the journey container,
8 + * headless, with the skill installed where Claude Code looks for it, so a run
9 + * also measures whether the skill is found and used.
10 + *
11 + * The binary is the Linux build Anthropic publishes to npm, fetched once per
12 + * version and mounted read-only. The container reaches the model only through
13 + * `model-proxy.ts`: it holds a one-off token, never the API key.
14 + */
15 +
16 +import type { JournalEntry } from '@e2e-tests/cli-fake-backend';
17 +import { execFileSync } from 'node:child_process';
18 +import { chmodSync, existsSync, mkdirSync, mkdtempSync, renameSync, rmSync } from 'node:fs';
19 +import { homedir, tmpdir } from 'node:os';
20 +import { join } from 'node:path';
21 +import { z } from 'zod';
22 +import type { ExecutedCommand, StopReason } from './agent';
23 +import type { ModelProxy } from './model-proxy';
24 +import type { Driver } from './run-journey';
25 +import { addTokens, ZERO_USAGE, type Usage } from './usage';
26 +
27 +/** Pinned so two runs of the same skill measure the same harness. */
28 +export const CLAUDE_CODE_VERSION = '2.1.310';
29 +
30 +const CONTAINER_BINARY = '/opt/claude-code/claude';
31 +
32 +/** How long Claude Code may take to exit once the run is over. */
33 +const END_GRACE_MS = 10_000;
34 +
35 +/** The tools the agent gets: a shell, reading files, and skills. No web, no subagents. */
36 +const TOOLS = 'Bash,Read,Glob,Grep,Skill';
37 +
38 +/** The Linux architecture of the Docker engine, in npm's naming. */
39 +const dockerArchitecture = (): 'arm64' | 'x64' => {
40 + const reported = execFileSync('docker', ['info', '--format', '{{.Architecture}}'], {
41 + encoding: 'utf8',
42 + }).trim();
43 + if (reported === 'aarch64' || reported === 'arm64') return 'arm64';
44 + if (reported === 'x86_64' || reported === 'amd64') return 'x64';
45 + throw new Error(`Unsupported Docker architecture: ${reported}`);
46 +};
47 +
48 +/**
49 + * The host path of the Linux `claude` binary, downloaded on first use into
50 + * `~/.cache/composio-e2e/claude-code/`. `npm pack` checks the package against
51 + * the registry's integrity hash.
52 + */
53 +export const ensureClaudeCodeBinary = (): string => {
54 + const architecture = dockerArchitecture();
55 + const directory = join(
56 + homedir(),
57 + '.cache',
58 + 'composio-e2e',
59 + 'claude-code',
60 + `${CLAUDE_CODE_VERSION}-linux-${architecture}`
61 + );
62 + const binary = join(directory, 'claude');
63 + if (existsSync(binary)) return binary;
64 +
65 + const scratch = mkdtempSync(join(tmpdir(), 'composio-e2e-claude-code-'));
66 + try {
67 + const packageName = `@anthropic-ai/claude-code-linux-${architecture}@${CLAUDE_CODE_VERSION}`;
68 + const tarball = execFileSync(
69 + 'npm',
70 + ['pack', packageName, '--pack-destination', scratch, '--silent'],
71 + { encoding: 'utf8' }
72 + )
73 + .trim()
74 + .split('\n')
75 + .at(-1);
76 + if (!tarball) throw new Error(`npm pack ${packageName} printed no file name`);
77 + execFileSync('tar', ['xzf', join(scratch, tarball), '-C', scratch]);
78 + mkdirSync(directory, { recursive: true });
79 + renameSync(join(scratch, 'package', 'claude'), binary);
80 + chmodSync(binary, 0o755);
81 + return binary;
82 + } finally {
83 + rmSync(scratch, { recursive: true, force: true });
84 + }
85 +};
86 +
87 +const ContentBlock = z.looseObject({
88 + type: z.string(),
89 + id: z.string().optional(),
90 + name: z.string().optional(),
91 + text: z.string().optional(),
92 + input: z.unknown().optional(),
93 + tool_use_id: z.string().optional(),
94 + content: z.unknown().optional(),
95 + is_error: z.boolean().optional(),
96 +});
97 +
98 +const StreamEvent = z.looseObject({
99 + type: z.string(),
100 + subtype: z.string().optional(),
101 + message: z.looseObject({ content: z.union([z.string(), z.array(ContentBlock)]) }).optional(),
102 + tool_use_result: z.unknown().optional(),
103 + result: z.string().optional(),
104 + is_error: z.boolean().optional(),
105 + num_turns: z.number().optional(),
106 + total_cost_usd: z.number().optional(),
107 + usage: z
108 + .looseObject({
109 + input_tokens: z.number().optional(),
110 + output_tokens: z.number().optional(),
111 + cache_read_input_tokens: z.number().optional(),
112 + cache_creation_input_tokens: z.number().optional(),
113 + })
114 + .optional(),
115 +});
116 +
117 +const BashInput = z.looseObject({ command: z.string() });
118 +const BashResult = z.looseObject({ stdout: z.string(), stderr: z.string() });
119 +const TextPart = z.looseObject({ type: z.literal('text'), text: z.string() });
120 +
121 +/** A tool result's text, whether Claude Code sent a string or text blocks. */
122 +const resultText = (content: unknown): string => {
123 + if (typeof content === 'string') return content;
124 + if (!Array.isArray(content)) return '';
125 + return content
126 + .flatMap(part => {
127 + const parsed = TextPart.safeParse(part);
128 + return parsed.success ? [parsed.data.text] : [];
129 + })
130 + .join('\n');
131 +};
132 +
133 +const userMessage = (text: string): string =>
134 + `${JSON.stringify({ type: 'user', message: { role: 'user', content: text } })}\n`;
135 +
136 +export interface ClaudeCodeOptions {
137 + readonly model: string;
138 + readonly proxy: ModelProxy;
139 + readonly binary: string;
140 + /** Install the skill (`true`), or run without it. */
141 + readonly withSkill: boolean;
142 + /** Claude Code stops itself once a run has cost this much. */
143 + readonly budgetUsd: number;
144 + /** Wall-clock limit for the agent; the run is cut off after it. */
145 + readonly deadlineMs: number;
146 + /** Cut-off for shell commands, so a looping agent cannot run forever. */
147 + readonly maxCommands: number;
148 + /** Every stream event, for the run's log. */
149 + readonly onEvent?: (event: unknown) => void;
150 +}
151 +
152 +export const claudeCodeDriver = (options: ClaudeCodeOptions): Driver => ({
153 + skill: options.withSkill ? 'claude-code-skills' : 'none',
154 + scrubbed: false,
155 + readOnlyMounts: [{ source: options.binary, target: CONTAINER_BINARY }],
156 + drive: async ({ rig, journey, person }) => {
157 + const environment: Record<string, string> = {
158 + ANTHROPIC_BASE_URL: options.proxy.dockerUrl,
159 + ANTHROPIC_API_KEY: options.proxy.token,
160 + // Lets permissions be skipped as root; the container is the sandbox.
161 + IS_SANDBOX: '1',
162 + CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: '1',
163 + DISABLE_AUTOUPDATER: '1',
164 + // Every model Claude Code might pick for a side task is the run's model,
165 + // or Haiku, so the proxy's allow-list holds.
166 + ANTHROPIC_SMALL_FAST_MODEL: 'claude-haiku-5-5',
167 + ANTHROPIC_DEFAULT_HAIKU_MODEL: 'claude-haiku-5-5',
168 + ANTHROPIC_DEFAULT_SONNET_MODEL: options.model,
169 + ANTHROPIC_DEFAULT_OPUS_MODEL: options.model,
170 + CLAUDE_CODE_SUBAGENT_MODEL: options.model,
171 + };
172 + const child = Bun.spawn(
173 + [
174 + 'docker',
175 + 'exec',
176 + '-i',
177 + '--workdir',
178 + journey.cwd ?? rig.journey.home,
179 + ...Object.entries(environment).flatMap(([name, value]) => ['-e', `${name}=${value}`]),
180 + rig.journey.name,
181 + CONTAINER_BINARY,
182 + '-p',
183 + '--input-format',
184 + 'stream-json',
185 + '--output-format',
186 + 'stream-json',
187 + '--verbose',
188 + '--model',
189 + options.model,
190 + '--permission-mode',
191 + 'bypassPermissions',
192 + '--tools',
193 + TOOLS,
194 + '--no-session-persistence',
195 + '--max-budget-usd',
196 + String(options.budgetUsd),
197 + ],
198 + { stdin: 'pipe', stdout: 'pipe', stderr: 'pipe' }
199 + );
200 + const send = (text: string) => {
201 + child.stdin.write(userMessage(text));
202 + void child.stdin.flush();
203 + };
204 +
205 + const commands: ExecutedCommand[] = [];
206 + const commandRequests: Array<ReadonlyArray<JournalEntry>> = [];
207 + const messages: string[] = [];
208 + const replies: Array<{ afterModelTurn: number; text: string }> = [];
209 + /** Bash calls that have started, by tool-use id, with the journal position at their start. */
210 + const running = new Map<string, { command: string; mark: number }>();
211 + let usage: Usage = ZERO_USAGE;
212 + let modelTurns = 0;
213 + let stop: StopReason | undefined;
214 + let finished = false;
215 +
216 + const cutOff = (reason: string) => {
217 + if (stop === undefined) stop = { provider: reason };
218 + child.kill();
219 + };
220 + const deadline = setTimeout(() => cutOff('deadline'), options.deadlineMs);
221 + let ending: ReturnType<typeof setTimeout> | undefined;
222 + /**
223 + * Closing stdin ends the session, but a command the agent left running in
224 + * the background keeps Claude Code alive. The run is over once the agent
225 + * stopped and the person had nothing to say, so it gets a moment, then goes.
226 + */
227 + const end = () => {
228 + child.stdin.end();
229 + ending ??= setTimeout(() => child.kill(), END_GRACE_MS);
230 + };
231 +
232 + const handle = (event: z.infer<typeof StreamEvent>) => {
233 + const content = event.message?.content;
234 + const blocks = Array.isArray(content) ? content : [];
235 + if (event.type === 'assistant') {
236 + const text = blocks
237 + .filter(block => block.type === 'text' && block.text !== undefined)
238 + .map(block => block.text)
239 + .join('\n');
240 + if (text !== '') {
241 + messages.push(text);
242 + // The person reads what the agent writes as it writes it.
243 + person.hear(text, false);
244 + }
245 + for (const block of blocks) {
246 + if (block.type !== 'tool_use' || block.name !== 'Bash' || block.id === undefined)
247 + continue;
248 + const input = BashInput.safeParse(block.input);
249 + if (!input.success) continue;
250 + running.set(block.id, { command: input.data.command, mark: rig.fake.journal.mark() });
251 + if (running.size + commands.length > options.maxCommands) cutOff('command-limit');
252 + }
253 + }
254 + if (event.type === 'user') {
255 + for (const block of blocks) {
256 + if (block.type !== 'tool_result' || block.tool_use_id === undefined) continue;
257 + const started = running.get(block.tool_use_id);
258 + if (!started) continue;
259 + running.delete(block.tool_use_id);
260 + const payload = resultText(block.content);
261 + const exitCode = block.is_error ? Number(/^Exit code (\d+)/.exec(payload)?.[1] ?? 1) : 0;
262 + const output = BashResult.safeParse(event.tool_use_result);
263 + commands.push({
264 + command: started.command,
265 + exitCode,
266 + stdout: output.success ? output.data.stdout : payload.replace(/^Exit code \d+\n?/, ''),
267 + stderr: output.success ? output.data.stderr : '',
268 + payload,
269 + });
270 + // Requests between the call and its result. Calls that overlap share requests.
271 + commandRequests.push(rig.fake.journal.since(started.mark));
272 + }
273 + }
274 + if (event.type === 'result') {
275 + modelTurns += event.num_turns ?? 0;
276 + usage = {
277 + ...addTokens(usage, '', {
278 + input: event.usage?.input_tokens ?? 0,
279 + output: event.usage?.output_tokens ?? 0,
280 + cacheRead: event.usage?.cache_read_input_tokens ?? 0,
281 + cacheWrite: event.usage?.cache_creation_input_tokens ?? 0,
282 + }),
283 + // Claude Code reports the session's running total.
284 + costUsd: event.total_cost_usd ?? usage.costUsd,
285 + };
286 + if (event.subtype !== 'success') {
287 + stop = { provider: event.subtype ?? 'error' };
288 + end();
289 + return;
290 + }
291 + const reply = person.hear(event.result ?? '', true);
292 + if (reply === undefined) {
293 + finished = true;
294 + end();
295 + return;
296 + }
297 + replies.push({ afterModelTurn: modelTurns, text: reply });
298 + send(reply);
299 + }
300 + };
301 +
302 + send(journey.prompt);
303 + const decoder = new TextDecoder();
304 + let buffer = '';
305 + for await (const chunk of child.stdout) {
306 + buffer += decoder.decode(chunk, { stream: true });
307 + for (let newline = buffer.indexOf('\n'); newline >= 0; newline = buffer.indexOf('\n')) {
308 + const line = buffer.slice(0, newline).trim();
309 + buffer = buffer.slice(newline + 1);
310 + if (line === '') continue;
311 + const raw: unknown = JSON.parse(line);
312 + options.onEvent?.(raw);
313 + const event = StreamEvent.safeParse(raw);
314 + if (event.success) handle(event.data);
315 + }
316 + }
317 + const stderr = await new Response(child.stderr).text();
318 + await child.exited;
319 + clearTimeout(deadline);
320 + clearTimeout(ending);
321 + if (stop === undefined && !finished) {
322 + stop = { provider: `exited ${child.exitCode}: ${stderr.trim().slice(-300)}` };
323 + }
324 +
325 + return {
326 + transcript: {
327 + commands,
328 + messages,
329 + replies,
330 + stop: stop ?? 'finished',
331 + modelTurns,
332 + },
333 + commandRequests,
334 + usage,
335 + };
336 + },
337 +});
ts/e2e-tests/cli/agent-journeys/src/journeys.ts
+11
@@ -57,6 +57,12 @@ export interface Journey {
57 readonly id: string;
58 readonly title: string;
59 readonly prompt: string;
60 + /**
61 + * `train` journeys are the ones a skill change is tuned against; `holdout`
62 + * journeys only measure, so a change that fits the train set and nothing
63 + * else shows up there. See HILLCLIMB.md.
64 + */
65 + readonly split: 'train' | 'holdout';
66 /** Directory the agent's shell starts in; defaults to the home directory. */
67 readonly cwd?: string;
68 readonly setup: (rig: CliRig) => Promise<void>;
@@ -293,6 +299,7 @@ export const JOURNEYS: ReadonlyArray<Journey> = [
299 id: 'consumer-slack-message',
300 title: 'consumer task: send a Slack message from an unconnected account',
301 prompt: 'Send a Slack message to the #general channel that says: deploy finished',
302 + split: 'train',
303 setup: loginRig,
304 checks: run => [
305 stoppedByItself(run),
@@ -320,6 +327,7 @@ export const JOURNEYS: ReadonlyArray<Journey> = [
327 title: 'developer task: list auth configs and run a tool as the test user',
328 prompt:
329 'I am building an app on Composio. In my developer project, list the auth configs, then execute SLACK_SEND_MESSAGE as my test user to post "hello from dev" in #general.',
330 + split: 'holdout',
331 cwd: DEV_DIRECTORY,
332 setup: async rig => {
333 await developerRig(rig);
@@ -360,6 +368,7 @@ export const JOURNEYS: ReadonlyArray<Journey> = [
368 title: 'tree confusion: a developer task that needs a connection first',
369 prompt:
370 'In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post "hello from dev" in #general.',
371 + split: 'train',
372 cwd: DEV_DIRECTORY,
373 setup: developerRig,
374 checks: run => [
@@ -421,6 +430,7 @@ export const JOURNEYS: ReadonlyArray<Journey> = [
430 id: 'first-run-login',
431 title: 'first run: find out the CLI is not logged in, and log in',
432 prompt: 'Check whether the Composio CLI on this machine is logged in. If it is not, log in.',
433 + split: 'train',
434 setup: async () => {},
435 checks: run => {
436 const whoami = run.transcript.commands.filter(command =>
@@ -473,6 +483,7 @@ export const JOURNEYS: ReadonlyArray<Journey> = [
483 title: '`composio run`: one inline script that executes a tool',
484 prompt:
485 'Use a single `composio run` inline script to send "build passed" to #general on Slack and print the timestamp of the sent message.',
486 + split: 'holdout',
487 setup: async rig => {
488 await rig.login();
489 rig.fake.world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
ts/e2e-tests/cli/agent-journeys/src/metrics.ts new
+51
@@ -0,0 +1,51 @@
1 +/**
2 + * Pass rates over repeated runs of one journey.
3 + *
4 + * With n graded runs of which c passed:
5 + *
6 + * - pass@k: the chance that at least one of k runs passes. It answers "can
7 + * the agent do this at all", and rises with k.
8 + * - pass^k: the chance that all k runs pass. It answers "can a user rely on
9 + * it", and falls with k. A skill change that makes the agent dependable
10 + * moves pass^k even when pass@k is already 1.
11 + *
12 + * Both are the unbiased estimates over the n runs (Chen et al. 2021 for
13 + * pass@k; Yao et al. 2024, tau-bench, for pass^k), so k may be smaller than n.
14 + */
15 +
16 +const choose = (n: number, k: number): number => {
17 + if (k < 0 || k > n) return 0;
18 + let result = 1;
19 + for (let i = 1; i <= k; i += 1) result = (result * (n - k + i)) / i;
20 + return result;
21 +};
22 +
23 +const assertCounts = (n: number, c: number, k: number) => {
24 + if (!Number.isInteger(n) || !Number.isInteger(c) || !Number.isInteger(k)) {
25 + throw new Error(`counts must be integers (n=${n}, c=${c}, k=${k})`);
26 + }
27 + if (c < 0 || c > n || k < 1 || k > n) {
28 + throw new Error(`need 0 <= c <= n and 1 <= k <= n (n=${n}, c=${c}, k=${k})`);
29 + }
30 +};
31 +
32 +export const passAtK = (n: number, c: number, k: number): number => {
33 + assertCounts(n, c, k);
34 + return 1 - choose(n - c, k) / choose(n, k);
35 +};
36 +
37 +export const passHatK = (n: number, c: number, k: number): number => {
38 + assertCounts(n, c, k);
39 + return choose(c, k) / choose(n, k);
40 +};
41 +
42 +/** The 95% Wilson score interval for c successes in n runs. */
43 +export const wilson = (c: number, n: number): { low: number; high: number } => {
44 + if (n === 0) return { low: 0, high: 1 };
45 + const z = 1.96;
46 + const p = c / n;
47 + const denominator = 1 + (z * z) / n;
48 + const centre = (p + (z * z) / (2 * n)) / denominator;
49 + const half = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / denominator;
50 + return { low: Math.max(0, centre - half), high: Math.min(1, centre + half) };
51 +};
ts/e2e-tests/cli/agent-journeys/src/model-proxy.ts new
+170
@@ -0,0 +1,170 @@
1 +/**
2 + * The only way a journey container reaches a model during a live run.
3 + *
4 + * The agent inside the container gets a one-off token, never the API key: its
5 + * shell commands inherit its environment, so a key there could end up in a
6 + * transcript. This proxy checks the token, checks the model, swaps the token
7 + * for the key, and streams the answer back unchanged.
8 + *
9 + * It listens on every interface, because that is how a container reaches the
10 + * host, so the token is what keeps others on the network from using the key.
11 + */
12 +
13 +import { randomBytes, timingSafeEqual } from 'node:crypto';
14 +import { createServer, type IncomingMessage, type ServerResponse } from 'node:http';
15 +import { z } from 'zod';
16 +
17 +/** The models a live run may call. Anything else is refused before it costs anything. */
18 +export const ALLOWED_MODELS = ['claude-haiku-5-5', 'claude-opus-5-5'] as const;
19 +export type AllowedModel = (typeof ALLOWED_MODELS)[number];
20 +
21 +const isAllowedModel = (model: string | undefined): model is AllowedModel =>
22 + ALLOWED_MODELS.some(allowed => allowed === model);
23 +
24 +/** Request headers that describe the hop, or carry a credential, and are never forwarded. */
25 +const DROPPED_REQUEST_HEADERS = new Set([
26 + 'host',
27 + 'connection',
28 + 'content-length',
29 + 'x-api-key',
30 + 'authorization',
31 + 'accept-encoding',
32 +]);
33 +
34 +/**
35 + * The requests an agent needs, by the path it sends, and the fixed upstream URL each goes to.
36 + * Anything else (batches, files, admin) is refused, because a request there may name no model,
37 + * or one the proxy cannot see. Nothing the agent sends becomes part of an upstream URL.
38 + */
39 +const FORWARDED: ReadonlyMap<string, string> = new Map(
40 + ['/v1/messages', '/v1/messages/count_tokens'].flatMap(path =>
41 + ['', '?beta=true'].map(query => [`${path}${query}`, `https://api.anthropic.com${path}${query}`])
42 + )
43 +);
44 +
45 +const ModelField = z.looseObject({ model: z.string().optional() });
46 +
47 +const parseJson = (body: Buffer): unknown => {
48 + try {
49 + return JSON.parse(body.toString('utf8'));
50 + } catch {
51 + return undefined;
52 + }
53 +};
54 +
55 +export interface ModelProxy {
56 + /** Base URL for `ANTHROPIC_BASE_URL` inside a container. */
57 + readonly dockerUrl: string;
58 + /** What the agent sends as its API key. Valid only for this proxy's lifetime. */
59 + readonly token: string;
60 + /** Requests forwarded so far, per model. */
61 + readonly forwarded: () => Readonly<Record<string, number>>;
62 + /** Requests refused so far, with the reason. */
63 + readonly refused: () => ReadonlyArray<string>;
64 + close(): Promise<void>;
65 +}
66 +
67 +const readBody = async (request: IncomingMessage): Promise<Buffer> => {
68 + const chunks: Buffer[] = [];
69 + for await (const chunk of request) chunks.push(Buffer.from(chunk));
70 + return Buffer.concat(chunks);
71 +};
72 +
73 +const refuse = (response: ServerResponse, status: number, message: string) => {
74 + response.writeHead(status, { 'content-type': 'application/json' });
75 + response.end(JSON.stringify({ type: 'error', error: { type: 'proxy_refused', message } }));
76 +};
77 +
78 +export const startModelProxy = async (options: { apiKey: string }): Promise<ModelProxy> => {
79 + const token = `sk-journey-${randomBytes(24).toString('hex')}`;
80 + const expected = Buffer.from(token);
81 + const forwarded: Record<string, number> = {};
82 + const refused: string[] = [];
83 +
84 + const tokenMatches = (request: IncomingMessage): boolean => {
85 + const header = request.headers['x-api-key'] ?? request.headers.authorization ?? '';
86 + const sent = Buffer.from(String(header).replace(/^Bearer\s+/i, ''));
87 + return sent.length === expected.length && timingSafeEqual(sent, expected);
88 + };
89 +
90 + const server = createServer((request, response) => {
91 + void (async () => {
92 + const path = request.url ?? '/';
93 + if (!tokenMatches(request)) {
94 + refused.push(`bad token for ${path}`);
95 + return refuse(response, 401, 'unknown token');
96 + }
97 + const target = request.method === 'POST' ? FORWARDED.get(path) : undefined;
98 + if (target === undefined) {
99 + refused.push(`${request.method} ${path}`);
100 + return refuse(
101 + response,
102 + 404,
103 + `only POST ${[...FORWARDED.keys()].join(', ')} is forwarded, not ${request.method} ${path}`
104 + );
105 + }
106 + // Every forwarded request names its model at the top level, so no other model is reachable.
107 + const body = await readBody(request);
108 + const parsed = ModelField.safeParse(parseJson(body));
109 + const model = parsed.success ? parsed.data.model : undefined;
110 + if (!isAllowedModel(model)) {
111 + refused.push(`model ${model ?? '(none)'}`);
112 + return refuse(response, 400, `model ${model ?? '(none)'} is not allowed in journey runs`);
113 + }
114 + forwarded[model] = (forwarded[model] ?? 0) + 1;
115 +
116 + const headers = new Headers();
117 + for (const [name, value] of Object.entries(request.headers)) {
118 + if (value === undefined || DROPPED_REQUEST_HEADERS.has(name)) continue;
119 + headers.set(name, Array.isArray(value) ? value.join(', ') : value);
120 + }
121 + headers.set('x-api-key', options.apiKey);
122 +
123 + const upstream = await fetch(target, {
124 + method: request.method,
125 + headers,
126 + body: body.length > 0 ? new Uint8Array(body) : undefined,
127 + });
128 + const responseHeaders: Record<string, string> = {};
129 + upstream.headers.forEach((value, name) => {
130 + // fetch already decoded the body, so its length and encoding no longer hold.
131 + if (name !== 'content-encoding' && name !== 'content-length') responseHeaders[name] = value;
132 + });
133 + response.writeHead(upstream.status, responseHeaders);
134 + if (upstream.body) {
135 + for await (const chunk of upstream.body) response.write(chunk);
136 + }
137 + response.end();
138 + })().catch(error => {
139 + if (!response.headersSent) {
140 + refuse(response, 502, error instanceof Error ? error.message : String(error));
141 + } else {
142 + response.destroy();
143 + }
144 + });
145 + });
146 +
147 + const port = await new Promise<number>((resolve, reject) => {
148 + server.once('error', reject);
149 + server.listen({ host: '0.0.0.0', port: 0 }, () => {
150 + const address = server.address();
151 + if (address === null || typeof address === 'string') {
152 + reject(new Error('the model proxy has no port'));
153 + return;
154 + }
155 + resolve(address.port);
156 + });
157 + });
158 +
159 + return {
160 + dockerUrl: `http://host.docker.internal:${port}`,
161 + token,
162 + forwarded: () => ({ ...forwarded }),
163 + refused: () => [...refused],
164 + close: () =>
165 + new Promise<void>(resolve => {
166 + server.closeAllConnections();
167 + server.close(() => resolve());
168 + }),
169 + };
170 +};
ts/e2e-tests/cli/agent-journeys/src/run-journey.ts
+2 -1
@@ -3,7 +3,8 @@
3 * journey's starting state, then an agent, and the evidence it left.
4 *
5 * The agent is a driver. The replay tests and `record.ts` use the minimal
6 - * loop in `agent.ts`; other agents can use the same driver contract and evidence checks.
6 + * loop in `agent.ts`; a live run can also drive Claude Code, see
7 + * `claude-code.ts`. Either way the checks read the same evidence.
8 */
9
10 import { startCliRig, type CliRig, type RigOptions } from '@e2e-tests/cli-fake-backend/rig';
ts/e2e-tests/cli/agent-journeys/src/usage.ts
+52
@@ -1,3 +1,10 @@
1 +/**
2 + * What a live run cost.
3 + *
4 + * Claude Code reports its own cost. The minimal loop does not, so its cost is
5 + * worked out from the token counts and the list prices below.
6 + */
7 +
8 export interface Usage {
9 readonly costUsd: number;
10 readonly inputTokens: number;
@@ -5,3 +12,48 @@ export interface Usage {
12 readonly cacheReadTokens: number;
13 readonly cacheWriteTokens: number;
14 }
15 +
16 +/**
17 + * USD per million tokens: Anthropic list prices as of 2026-11, with cache
18 + * reads at 0.1x and 5-minute cache writes at 1.25x the input price. Only the
19 + * models a live run may call (see `model-proxy.ts`) are listed.
20 + */
21 +const PRICES: Readonly<
22 + Record<string, { input: number; output: number; cacheRead: number; cacheWrite: number }>
23 +> = {
24 + // Prompts up to 100K tokens; a journey's prompt stays well under that.
25 + 'claude-haiku-5-5': { input: 0.1, output: 0.5, cacheRead: 0.01, cacheWrite: 0.125 },
26 + 'claude-opus-5-5': { input: 4, output: 20, cacheRead: 0.2, cacheWrite: 5 },
27 +};
28 +
29 +export const ZERO_USAGE: Usage = {
30 + costUsd: 0,
31 + inputTokens: 0,
32 + outputTokens: 0,
33 + cacheReadTokens: 0,
34 + cacheWriteTokens: 0,
35 +};
36 +
37 +export const addTokens = (
38 + usage: Usage,
39 + model: string,
40 + tokens: { input: number; output: number; cacheRead?: number; cacheWrite?: number }
41 +): Usage => {
42 + // A replayed model (a fixture's `hand-authored`, a rehearsal stand-in) costs nothing.
43 + const price = PRICES[model] ?? { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 };
44 + const cacheRead = tokens.cacheRead ?? 0;
45 + const cacheWrite = tokens.cacheWrite ?? 0;
46 + return {
47 + costUsd:
48 + usage.costUsd +
49 + (tokens.input * price.input +
50 + tokens.output * price.output +
51 + cacheRead * price.cacheRead +
52 + cacheWrite * price.cacheWrite) /
53 + 1e6,
54 + inputTokens: usage.inputTokens + tokens.input,
55 + outputTokens: usage.outputTokens + tokens.output,
56 + cacheReadTokens: usage.cacheReadTokens + cacheRead,
57 + cacheWriteTokens: usage.cacheWriteTokens + cacheWrite,
58 + };
59 +};