@devries-cameron20 / composio-1 / commits / b8e5258d4
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

test(ts/e2e): add grader mutation and determinism guards (#5418)

This PR: - adds `mutate.ts`, which plays each journey's hand-authored path with no model, first whole and then once per command with that command left out, and grades every run with the journey's checks - gives each dropped step a verdict — `needed` (a check failed), `unused` (every check passed and the run left the same outcome), or `check-too-weak` (every check passed but the run ended somewhere else) - adds `ModelAdapter.hasMore` so a scripted turn whose only command was dropped keeps the loop going; an adapter without it behaves exactly as before - adds `World.snapshot()` and `worldHash()` to the fake backend, and pins the seeded world in `determinism.test.ts` - records `JourneyRun.startingWorldHash` after a journey's setup, and has the replay suite check that every attempt of a journey started from the same world - has the replay suite leave a logged-in rig idle for five seconds, then check that it sent no request, changed nothing, and still runs commands - records the first mutation run on all five journeys: 17 steps, 10 `needed`, 7 `unused`, 0 `check-too-weak`; the `unused` steps are listed in the agent-journeys README - breaking for eval batches: `agent.ts`, `run-journey.ts` and `fake-backend/src` are all inside `eval.ts`'s `HARNESS_PATHS`, so `eval.ts compare` now refuses to compare a batch from before this commit with one from after it — measure a new baseline after merge ## Context A journey's checks are the only thing that grades a run, and nothing showed they would notice a missing step. The mutation run answers that per journey: a step marked `check-too-weak` is a way to end somewhere else that the journey still passes, so that check has to be strengthened before a `pass^k` number off the journey can be trusted. The determinism guards answer the other half — `pass^k` compares attempts of one journey, which says nothing if the fake handed those attempts different starting worlds, or if something besides the agent kept writing to the journal. Both are adapted from the `check --mutate`, `reset` and `idle` stages of [miroir](https://github.com/raphaelDkhn/miroir), an RL-environment harness. No code is taken from it; the diff is test-only and confined to `ts/e2e-tests/cli/`. ## Verification - `bun run typecheck` in `agent-journeys` and `fake-backend`: clean - `oxlint ts/e2e-tests/cli`, antislop, and prettier on the changed files: clean - `bun test journeys.test.ts person.test.ts metrics.test.ts mutation.test.ts determinism.test.ts`: 74 pass, 0 fail - `bun test e2e.test.ts`: 46 pass, 0 fail, 42s — includes the new idle guard (12s) and the attempt-isolation test - `bun run mutate.ts --journeys first-run-login`: 3 `needed`, 1 `unused`, 0 `check-too-weak`; all five journeys: 10 `needed`, 7 `unused`, 0 `check-too-weak` No changeset: `ts/e2e-tests/*` packages are private and never appear in `.changeset/`.

Alberto Schiabel committed Nov 18, 2026 at 18:40 UTC b8e5258d4e3d4c3acd62df70564febb527b70d04
15 files changed +976 -3
ts/e2e-tests/cli/agent-journeys/HILLCLIMB.md
+2
@@ -50,6 +50,8 @@ set -a; . ~/.config/composio-e2e/llm.env; set +a
50
51 - **Do not change the harness in the same step as the skill.** The journeys, the checks, the person and the fake backend (`agent-journeys/src/`, `fake-backend/src/`) are what grade a run; their hash is part of every batch, and `compare` refuses two batches with different hashes. Fix a grader in its own change, then measure a new baseline.
52 - **Do not tune the checks to the agent.** A check that fails because the agent took a valid path the check did not foresee is a grader bug; fix it with a test in `journeys.test.ts`. A check that fails because the agent did the wrong thing is the signal.
53 +- **Know what a journey's checks notice before trusting its pass^k.** `bun run mutate.ts --journeys <id>` plays the hand-authored path once per command with that command left out (see the [README](./README.md#what-a-mutation-run-proves)). A step it calls `check-too-weak` is a way to end somewhere else that the journey still passes; strengthen the check, in its own change, before reading a rate off it.
54 +- **Repeated runs are comparable only if they start alike.** `determinism.test.ts` pins the seeded world, and the replay suite checks that every replayed attempt of a journey started from the same one and that a rig left idle after login stays quiet. `eval.ts` does not record the starting world of its runs; it relies on those tests. When one fails, fix the seed or the leak before comparing batches.
55 - **Do not add the journeys' wording to the skill.** The skill is for every user's task; a sentence that only matches a journey's prompt raises train and nothing else.
56 - **Spend little.** A Haiku run costs about $0.002 to $0.01; Opus about 40 times that. `--max-cost-usd` (default $2) stops a batch from starting runs past it, and `--run-budget-usd` caps one run.
57
ts/e2e-tests/cli/agent-journeys/README.md
+32 -1
@@ -8,6 +8,13 @@ pnpm --filter @e2e-tests/cli-agent-journeys test:e2e:cli
8
9 No API key is read and no model is called when the tests run.
10
11 +To see which commands of each hand-authored path the journey's checks depend on, play the path once per command with that command left out (Docker, no key, one container per run):
12 +
13 +```bash
14 +cd ts/e2e-tests/cli/agent-journeys
15 +bun run mutate.ts [--journeys first-run-login] [--json] [--strict]
16 +```
17 +
18 ## What a passing journey proves
19
20 Read this before drawing a conclusion from a green run.
@@ -19,6 +26,30 @@ Read this before drawing a conclusion from a green run.
26
27 A replay on its own proves nothing: aimock answers the same whether a command succeeded or failed. So every journey is judged by its checks (`src/journeys.ts`), which read the commands that ran, their exit codes and output, and the fake backend's journal: which tool ran, on which project, as which user. None of them reads the model's prose. See [How a run is graded](#how-a-run-is-graded).
28
29 +### What a mutation run proves
30 +
31 +`mutate.ts` plays a journey's hand-authored path with no model (`src/scripted.ts`), first whole and then once per command with that command left out, and grades each run with the journey's checks. Each dropped command gets a verdict (`src/mutation.ts`):
32 +
33 +- `needed`: some check failed without it.
34 +- `unused`: every check passed, and the run left the same outcome as the whole path: the same tools ran with the same arguments, as the same user on the same project; the same connections and logins were made; the next `composio whoami` found the same login. A read the checks do not depend on lands here.
35 +- `check-too-weak`: every check passed, but the outcome differs. The run ended somewhere else and no check noticed.
36 +
37 +When the whole path fails a check, the tool says so, drops nothing, and exits 1; `--strict` also exits 1 on any `unused` or `check-too-weak` step, so with the `unused` steps listed below it fails on four of the five journeys today. Results go to `results/<batch>/`.
38 +
39 +What it proves: whether the checks notice that one step of the taught path is missing. What it does not prove: that they catch a wrong path the hand-authored file never takes (another command in place of a dropped one, two steps missing, the right step on the wrong project), or anything about a real model. The outcome compares only what is listed above; a difference elsewhere in the fake's state is not seen.
40 +
41 +Last run, on every journey (2026-11-16, at the commit that added the tool): 17 steps, 10 `needed`, 7 `unused`, none `check-too-weak`, every unmutated path passing.
42 +
43 +| Journey | `unused` steps |
44 +| ------------------------------------ | --------------------------------------------------------------------------------------------------- |
45 +| `consumer-slack-message` | `composio search`; the first `composio execute`, which fails because Slack is not connected |
46 +| `developer-auth-configs-and-execute` | `cat` of the `composio-dev.md` reference; `composio dev --help` |
47 +| `developer-execute-unconnected` | the wrong-tree `composio dev execute`; the first `playground-execute`, which fails as not connected |
48 +| `first-run-login` | the last `composio whoami` |
49 +| `run-inline-script` | none |
50 +
51 +Each is a read or a command that fails on purpose. The checks grade what a run changed, so `unused` is the expected verdict for both kinds. The failing commands are on the path for the error text an agent recovers from: in `developer-execute-unconnected` the journey's `scriptedChecks` pin that text when the hand-authored file is replayed; in `consumer-slack-message` nothing does. A mutation run does not grade `scriptedChecks`.
52 +
53 ## How a run works
54
55 1. `src/skill.ts` renders the skill from `ts/packages/cli/skills-src/composio-cli`, so a change to the skill changes the system prompt here. The system prompt is the built `SKILL.md` preceded by one line naming the directory where the skill's files are in the container, as an agent host does; the references are written there for the model to `cat`.
@@ -163,7 +194,7 @@ How to read it:
194
195 The replay suite answers "does the CLI still behave on paths models have taken". It cannot answer "does this change make models do better", because a replay does not consult the model. Recording does, and the pieces here are meant to be the base of that measurement:
196
166 -- **Environment**: `startCliRig()` gives a fresh container with the real binary, a seeded backend that every host the CLI knows is pointed at, and a journal of which project and user every request acted on. It needs no account and changes nothing outside the fake as long as the agent stays on `composio`; the container's other outbound traffic is not blocked. Each run has its own container and backend, so runs are repeatable and can run in parallel.
197 +- **Environment**: `startCliRig()` gives a fresh container with the real binary, a seeded backend that every host the CLI knows is pointed at, and a journal of which project and user every request acted on. It needs no account and changes nothing outside the fake as long as the agent stays on `composio`; the container's other outbound traffic is not blocked. Each run has its own container and backend, so runs are repeatable and can run in parallel. Two tests hold that up: `determinism.test.ts` pins the seeded world by its hash (`worldHash` in the fake backend), and the replay suite checks that every replayed attempt of a journey started from the same world after its setup, and that a rig left alone after a login sends no request and changes nothing.
198 - **Task**: a journey's `prompt`, `setup`, and starting directory.
199 - **Grader**: a journey's `checks` for success and its `detours` for how the run got there; see [How a run is graded](#how-a-run-is-graded). They read exit codes, command output, and the journal, never the model's prose, so a grade is a fact and not a judgement. Each is reported by name, which says where a failed run went wrong (wrong project, never linked, wrong text) and not only that it failed. Compute the wrong-tree rate from the detours of every run, the failed ones included.
200 - **Subject**: the skill text, rendered from source on every run and identified by `skillSha256`; and the CLI, identified by `revision`.
ts/e2e-tests/cli/agent-journeys/determinism.test.ts new
+92
@@ -0,0 +1,92 @@
1 +/**
2 + * The seeded world, without a container: every rig starts from the same
3 + * state, and that state is pinned, so a change to the seed is a decision and
4 + * not a drift. `pass^k` compares attempts of one journey; it means nothing if
5 + * they did not start alike.
6 + */
7 +
8 +import { describe, expect, it } from 'bun:test';
9 +import {
10 + CONSUMER_PROJECT,
11 + CONSUMER_USER_ID,
12 + DEVELOPER_PROJECT,
13 + World,
14 + worldHash,
15 + type Connection,
16 +} from '@e2e-tests/cli-fake-backend';
17 +
18 +/**
19 + * `worldHash(new World())`. Update it in the change that alters the seed, and
20 + * say in that change why: every earlier eval batch was run against the old one.
21 + */
22 +const SEEDED_WORLD_HASH = '7621fa4ed357c514';
23 +
24 +describe('the seeded world', () => {
25 + it('hashes the same in every new instance', () => {
26 + expect(worldHash(new World())).toBe(worldHash(new World()));
27 + });
28 +
29 + it('is the pinned seed', () => {
30 + expect(worldHash(new World())).toBe(SEEDED_WORLD_HASH);
31 + });
32 +
33 + it('ignores the deployment’s user API key, which no request changes', () => {
34 + // A digest of a credential is a password hash to any reader, and the key is
35 + // fixed per rig, so it is not state two runs are compared by.
36 + expect(worldHash(new World({ userApiKey: 'uk_e2e_another_deployment' }))).toBe(
37 + worldHash(new World())
38 + );
39 + });
40 +});
41 +
42 +describe('worldHash', () => {
43 + it('does not depend on the order of an object’s keys', () => {
44 + const world = new World();
45 + world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
46 + const before = worldHash(world);
47 + const [connection] = world.connections;
48 + world.connections[0] = Object.fromEntries(Object.entries(connection!).reverse()) as Connection;
49 + expect(worldHash(world)).toBe(before);
50 + });
51 +
52 + // One change per kind of state a request can leave behind.
53 + const changes: ReadonlyArray<[string, (world: World) => void]> = [
54 + [
55 + 'a connection',
56 + world => void world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack'),
57 + ],
58 + ['an auth config', world => void world.authConfigForToolkit(CONSUMER_PROJECT.label, 'slack')],
59 + ['a login session', world => void world.createLoginSession()],
60 + [
61 + 'a tool-router session',
62 + world => void world.createToolRouterSession(DEVELOPER_PROJECT.label, CONSUMER_USER_ID),
63 + ],
64 + ['a minted project key', world => void world.mintProjectKey(DEVELOPER_PROJECT.label, 'e2e')],
65 + [
66 + 'a release',
67 + world => void world.releases.push({ tag: 'v0.0.1', prerelease: false, assets: [] }),
68 + ],
69 + ['how links complete', world => void (world.linkCompletesAfterPolls = null)],
70 + ['a tool execution', world => void world.nextLogId()],
71 + ];
72 +
73 + for (const [name, change] of changes) {
74 + it(`changes with ${name}`, () => {
75 + const world = new World();
76 + change(world);
77 + expect(worldHash(world)).not.toBe(worldHash(new World()));
78 + });
79 + }
80 +
81 + it('changes when a connection’s status does', () => {
82 + const world = new World();
83 + const pending = world.beginConnection({
84 + projectLabel: CONSUMER_PROJECT.label,
85 + userId: CONSUMER_USER_ID,
86 + toolkit: 'slack',
87 + });
88 + const before = worldHash(world);
89 + world.authorizeConnection(pending.id);
90 + expect(worldHash(world)).not.toBe(before);
91 + });
92 +});
ts/e2e-tests/cli/agent-journeys/e2e.test.ts
+57
@@ -9,6 +9,8 @@
9
10 import { afterAll, beforeAll, describe, expect, it } from 'bun:test';
11 import { LLMock } from '@copilotkit/aimock';
12 +import { summarize, worldHash } from '@e2e-tests/cli-fake-backend';
13 +import { startCliRig } from '@e2e-tests/cli-fake-backend/rig';
14 import { join } from 'node:path';
15 import { adapterFor } from './src/adapters';
16 import { HAND_AUTHORED, listFixtures } from './src/fixtures';
@@ -22,6 +24,41 @@ process.env.AIMOCK_STRICT_TURN_INDEX = '1';
24
25 const fixtures = listFixtures();
26
27 +/**
28 + * How long the idle rig is left alone. Only a request sent on its own within
29 + * this window is seen; a slower timer would get past the guard.
30 + */
31 +const IDLE_MS = 5_000;
32 +
33 +/** The world each attempt of a journey started from, by journey id. */
34 +const startingWorlds = new Map<string, Array<{ fixture: string; hash: string }>>();
35 +
36 +describe('the rig', () => {
37 + // A journey's journal is read as what the agent did. If the binary, or what a
38 + // setup started, kept sending requests once the setup was over, those would
39 + // land in the run's journal and be graded as the agent's.
40 + it(
41 + 'left alone after a login, records no request, changes nothing, and still runs commands',
42 + async () => {
43 + const rig = await startCliRig({ label: 'idle' });
44 + try {
45 + await rig.login();
46 + const mark = rig.fake.journal.mark();
47 + const before = worldHash(rig.fake.world);
48 + await Bun.sleep(IDLE_MS);
49 + expect(summarize(rig.fake.journal.since(mark))).toEqual([]);
50 + expect(worldHash(rig.fake.world)).toBe(before);
51 + const alive = await rig.run('echo alive');
52 + expect(alive.exitCode).toBe(0);
53 + expect(alive.stdout.trim()).toBe('alive');
54 + } finally {
55 + await rig.stop();
56 + }
57 + },
58 + JOURNEY_TIMEOUT_MS
59 + );
60 +});
61 +
62 describe('the fixture set', () => {
63 it('has no fixture for a journey that does not exist', () => {
64 const known = new Set(JOURNEYS.map(journey => journey.id));
@@ -61,6 +98,12 @@ for (const journey of JOURNEYS) {
98 mock.loadFixtureFile(fixture.path);
99 const baseURL = await mock.start();
100 run = await runJourney(journey, adapterFor(fixture.provider, fixture.model, baseURL));
101 + const attempts = startingWorlds.get(journey.id) ?? [];
102 + attempts.push({
103 + fixture: `${fixture.provider} ${fixture.model}`,
104 + hash: run.startingWorldHash,
105 + });
106 + startingWorlds.set(journey.id, attempts);
107 }, JOURNEY_TIMEOUT_MS);
108
109 afterAll(async () => {
@@ -115,3 +158,17 @@ for (const journey of JOURNEYS) {
158 }
159 });
160 }
161 +
162 +// Runs after every journey above: bun runs a file's tests in the order they are
163 +// defined. pass^k compares attempts of one journey, which says nothing about
164 +// the agent if the fake handed them different starting worlds.
165 +describe('attempt isolation', () => {
166 + it('started every attempt of a journey from the same world', () => {
167 + // An empty map would pass the comparison below without comparing anything.
168 + expect(startingWorlds.size).toBeGreaterThan(0);
169 + const differing = [...startingWorlds].filter(
170 + ([, attempts]) => new Set(attempts.map(attempt => attempt.hash)).size > 1
171 + );
172 + expect(Object.fromEntries(differing)).toEqual({});
173 + });
174 +});
ts/e2e-tests/cli/agent-journeys/mutate.ts new
+102
@@ -0,0 +1,102 @@
1 +/**
2 + * Grader mutation: plays each journey's hand-authored path with one command
3 + * left out, and reports whether the journey's checks noticed.
4 + *
5 + * bun run mutate.ts [--journeys <id,id>] [--json] [--strict]
6 + *
7 + * The path is played with no model (`src/scripted.ts`) against a fresh rig,
8 + * first whole, then once per dropped command, and graded by the same checks as
9 + * a replay. Verdicts are explained in `src/mutation.ts`.
10 + *
11 + * What a report shows: which commands of the taught path the checks depend
12 + * on, and which could go without any check failing. What it does not show:
13 + * that the checks catch a wrong path the hand-authored file never takes, or
14 + * anything about a real model.
15 + *
16 + * Exits 1 when an unmutated path fails a check, and with `--strict` also when
17 + * a step is `unused` or `check-too-weak`. Results go to `results/<batch>/`
18 + * (git-ignored). Needs Docker; spends nothing.
19 + */
20 +
21 +import { mkdirSync, writeFileSync } from 'node:fs';
22 +import { join, relative } from 'node:path';
23 +import { parseArgs } from 'node:util';
24 +import { HAND_AUTHORED, listFixtures } from './src/fixtures';
25 +import { journeyById, JOURNEYS, type Journey } from './src/journeys';
26 +import {
27 + exitCodeOf,
28 + grade,
29 + renderReports,
30 + reportJourney,
31 + type Graded,
32 + type JourneyReport,
33 +} from './src/mutation';
34 +import { runJourneyWith } from './src/run-journey';
35 +import { readScriptedPath, scriptedCommands, scriptedDriver } from './src/scripted';
36 +
37 +const HERE = import.meta.dir;
38 +
39 +const { values } = parseArgs({
40 + args: process.argv.slice(2),
41 + options: {
42 + journeys: { type: 'string' },
43 + json: { type: 'boolean', default: false },
44 + strict: { type: 'boolean', default: false },
45 + },
46 +});
47 +
48 +const selected = (): ReadonlyArray<Journey> => {
49 + try {
50 + return values.journeys ? values.journeys.split(',').map(journeyById) : JOURNEYS;
51 + } catch (error) {
52 + console.error(error instanceof Error ? error.message : String(error));
53 + console.error('Usage: bun run mutate.ts [--journeys <id,id>] [--json] [--strict]');
54 + process.exit(2);
55 + }
56 +};
57 +const journeys = selected();
58 +
59 +const scripted = new Map(
60 + listFixtures()
61 + .filter(fixture => fixture.provider === 'openai' && fixture.model === HAND_AUTHORED)
62 + .map(fixture => [fixture.journeyId, fixture.path])
63 +);
64 +
65 +const play = async (journey: Journey, path: string, drop?: number): Promise<Graded> => {
66 + const run = await runJourneyWith(journey, scriptedDriver(path, { drop }));
67 + try {
68 + return grade(journey, run);
69 + } finally {
70 + await run.rig.stop();
71 + }
72 +};
73 +
74 +const reports: JourneyReport[] = [];
75 +for (const journey of journeys) {
76 + const path = scripted.get(journey.id);
77 + // e2e.test.ts already fails when a journey has no hand-authored path.
78 + if (!path) throw new Error(`${journey.id} has no hand-authored fixture`);
79 + const fixture = relative(HERE, path);
80 + const commands = scriptedCommands(readScriptedPath(path));
81 + console.error(`${journey.id}: unmutated path`);
82 + const unmutated = await play(journey, path);
83 + const mutated: Array<{ command: string; graded: Graded }> = [];
84 + if (unmutated.checks.every(check => check.ok)) {
85 + for (const [index, command] of commands.entries()) {
86 + console.error(`${journey.id}: without step ${index + 1}, ${command.slice(0, 80)}`);
87 + mutated.push({ command, graded: await play(journey, path, index) });
88 + }
89 + }
90 + reports.push(reportJourney({ journey: journey.id, fixture, unmutated, mutated }));
91 +}
92 +
93 +const table = renderReports(reports);
94 +const batch = `${new Date().toISOString().replace(/[:.]/g, '-').slice(0, 19)}-mutate`;
95 +const directory = join(HERE, 'results', batch);
96 +mkdirSync(directory, { recursive: true });
97 +writeFileSync(join(directory, 'mutations.json'), `${JSON.stringify(reports, null, 2)}\n`);
98 +writeFileSync(join(directory, 'summary.txt'), table);
99 +
100 +process.stdout.write(values.json ? `${JSON.stringify(reports, null, 2)}\n` : table);
101 +console.error(`written ${relative(process.cwd(), directory)}`);
102 +process.exit(exitCodeOf(reports, values.strict ?? false));
ts/e2e-tests/cli/agent-journeys/mutation.test.ts new
+373
@@ -0,0 +1,373 @@
1 +/**
2 + * Grader mutation, without a container: the scripted path plays what the
3 + * hand-authored file says, a dropped command keeps the rest of the path, and
4 + * each verdict follows from made-up evidence.
5 + */
6 +
7 +import { describe, expect, it } from 'bun:test';
8 +import {
9 + CONSUMER_PROJECT,
10 + CONSUMER_USER_ID,
11 + DEVELOPER_PROJECT,
12 + World,
13 + type JournalEntry,
14 +} from '@e2e-tests/cli-fake-backend';
15 +import { runAgent, type ModelAdapter, type ModelTurn, type ShellOutcome } from './src/agent';
16 +import { fixturePath, HAND_AUTHORED } from './src/fixtures';
17 +import type { Check, Evidence, Journey } from './src/journeys';
18 +import {
19 + exitCodeOf,
20 + grade,
21 + outcomeOf,
22 + renderReports,
23 + reportJourney,
24 + verdictOf,
25 +} from './src/mutation';
26 +import { readScriptedPath, scriptedAdapter, scriptedCommands } from './src/scripted';
27 +
28 +const OK: ShellOutcome = { exitCode: 0, stdout: '', stderr: '' };
29 +
30 +/** Plays an adapter against a shell that answers every command with `OK`. */
31 +const play = (
32 + adapter: ModelAdapter,
33 + hear?: (message: string, waiting: boolean) => string | undefined
34 +) => {
35 + const ran: string[] = [];
36 + return runAgent({
37 + adapter,
38 + maxTurns: 12,
39 + hear,
40 + execute: async command => {
41 + ran.push(command);
42 + return OK;
43 + },
44 + }).then(transcript => ({ transcript, ran }));
45 +};
46 +
47 +const turns: ReadonlyArray<ModelTurn> = [
48 + { text: '', calls: [{ id: 'a', command: 'composio whoami' }] },
49 + { text: 'Open the page, I will wait.', calls: [{ id: 'b', command: 'composio login --poll' }] },
50 + { text: '', calls: [{ id: 'c', command: 'composio whoami' }] },
51 + { text: 'Logged in.', calls: [] },
52 +];
53 +
54 +describe('the scripted path', () => {
55 + it('reads the hand-authored file turn by turn, shell calls only', () => {
56 + const path = readScriptedPath(fixturePath('first-run-login', 'openai', HAND_AUTHORED));
57 + expect(scriptedCommands(path)).toEqual([
58 + 'composio whoami',
59 + 'composio login',
60 + 'composio login --poll',
61 + 'composio whoami',
62 + ]);
63 + expect(path.at(-1)).toEqual({
64 + text: 'The CLI is now logged in as dev@acme-e2e.test.',
65 + calls: [],
66 + });
67 + });
68 +
69 + it('plays every command, then finishes on the last turn', async () => {
70 + const { transcript, ran } = await play(scriptedAdapter(turns));
71 + expect(ran).toEqual(['composio whoami', 'composio login --poll', 'composio whoami']);
72 + expect(transcript.stop).toBe('finished');
73 + expect(transcript.modelTurns).toBe(4);
74 + });
75 +
76 + it('leaves out the dropped command and plays the turns after it', async () => {
77 + const { transcript, ran } = await play(scriptedAdapter(turns, { drop: 1 }));
78 + expect(ran).toEqual(['composio whoami', 'composio whoami']);
79 + expect(transcript.messages).toEqual(['Open the page, I will wait.', 'Logged in.']);
80 + expect(transcript.stop).toBe('finished');
81 + expect(transcript.modelTurns).toBe(4);
82 + });
83 +
84 + it('hands the person the text of a turn whose command was dropped', async () => {
85 + const heard: Array<[string, boolean]> = [];
86 + await play(scriptedAdapter(turns, { drop: 1 }), (message, waiting) => {
87 + heard.push([message, waiting]);
88 + return undefined;
89 + });
90 + expect(heard[1]).toEqual(['Open the page, I will wait.', true]);
91 + });
92 +
93 + it('stops as the script ending when the person answers its last turn', async () => {
94 + const { transcript, ran } = await play(scriptedAdapter(turns), () => 'Thanks.');
95 + expect(ran).toEqual(['composio whoami', 'composio login --poll', 'composio whoami']);
96 + expect(transcript.stop).toEqual({ provider: 'script-ended' });
97 + });
98 +});
99 +
100 +describe('runAgent with an adapter that does not say whether more turns follow', () => {
101 + // A model's turn without a command is its last, as before `hasMore` existed.
102 + it('stops at the first turn that asks for no command', async () => {
103 + let asked = 0;
104 + const adapter: ModelAdapter = {
105 + provider: 'openai',
106 + model: 'test',
107 + next: async () => {
108 + asked += 1;
109 + return asked === 1 ? { text: 'Nothing to do.', calls: [] } : turns[0]!;
110 + },
111 + record: () => {},
112 + say: () => {},
113 + };
114 + const { transcript, ran } = await play(adapter);
115 + expect(asked).toBe(1);
116 + expect(ran).toEqual([]);
117 + expect(transcript.stop).toBe('finished');
118 + });
119 +});
120 +
121 +/** A made-up journal entry: a request that changed nothing, unless `fields` say otherwise. */
122 +const entry = (fields: Partial<JournalEntry>): JournalEntry => ({
123 + seq: 1,
124 + role: 'backend',
125 + method: 'POST',
126 + path: '/',
127 + query: {},
128 + route: null,
129 + keyClass: 'user',
130 + keyKnown: true,
131 + orgId: null,
132 + project: CONSUMER_PROJECT.label,
133 + requestedProject: null,
134 + projectFrom: null,
135 + userId: null,
136 + body: null,
137 + status: 200,
138 + effect: null,
139 + tool: null,
140 + fault: null,
141 + ...fields,
142 +});
143 +
144 +// Made-up evidence: what the verdicts read is the checks and the outcome.
145 +const evidenceWith = (world: World, requests: ReadonlyArray<JournalEntry> = []): Evidence => ({
146 + transcript: { commands: [], messages: [], replies: [], stop: 'finished', modelTurns: 1 },
147 + world,
148 + requests,
149 + commandRequests: [],
150 + startingConnections: [],
151 + whoamiAfter: OK,
152 +});
153 +
154 +const journeyChecking = (checks: (run: Evidence) => ReadonlyArray<Check>): Journey => ({
155 + id: 'made-up',
156 + title: 'made up',
157 + prompt: '',
158 + split: 'train',
159 + setup: async () => {},
160 + checks,
161 +});
162 +
163 +/** Passes when a login session exists and is linked; says nothing about anything else. */
164 +const loggedIn = journeyChecking(run => [
165 + { name: 'a login session was linked', ok: run.world.loginSessions[0]?.status === 'linked' },
166 +]);
167 +
168 +const linkedWorld = () => {
169 + const world = new World();
170 + world.linkLoginSession(world.createLoginSession().id);
171 + return world;
172 +};
173 +
174 +describe('verdicts', () => {
175 + const unmutated = grade(loggedIn, evidenceWith(linkedWorld()));
176 +
177 + it('calls a step needed when a check fails without it', () => {
178 + const mutated = grade(loggedIn, evidenceWith(new World()));
179 + expect(verdictOf(unmutated, mutated)).toBe('needed');
180 + });
181 +
182 + it('calls a step unused when every check passes and the outcome is the same', () => {
183 + const mutated = grade(loggedIn, evidenceWith(linkedWorld()));
184 + expect(verdictOf(unmutated, mutated)).toBe('unused');
185 + });
186 +
187 + it('calls a check too weak when every check passes but the run ended elsewhere', () => {
188 + // The same login, plus a Slack account connected that the unmutated run never connected.
189 + const world = linkedWorld();
190 + world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
191 + const mutated = grade(loggedIn, evidenceWith(world));
192 + expect(mutated.checks.every(check => check.ok)).toBe(true);
193 + expect(verdictOf(unmutated, mutated)).toBe('check-too-weak');
194 + });
195 +
196 + it('leaves the connections the setup made out of the outcome', () => {
197 + const world = new World();
198 + const seeded = world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
199 + expect(outcomeOf({ ...evidenceWith(world), startingConnections: [seeded.id] })).toBe(
200 + outcomeOf(evidenceWith(new World()))
201 + );
202 + });
203 +
204 + it('counts a tool that ran in the outcome, and a request that changed nothing not', () => {
205 + const quiet = outcomeOf(evidenceWith(new World()));
206 + expect(outcomeOf(evidenceWith(new World(), [entry({ route: 'tools.list' })]))).toBe(quiet);
207 + expect(
208 + outcomeOf(
209 + evidenceWith(new World(), [
210 + entry({ effect: 'tool-executed', tool: { slug: 'SLACK_SEND_MESSAGE', arguments: {} } }),
211 + ])
212 + )
213 + ).not.toBe(quiet);
214 + });
215 + // Every field the outcome compares, one at a time: if `outcomeOf` stopped
216 + // reading one, a run that ended elsewhere in that way would be called `unused`.
217 + describe('an outcome that differs in one thing', () => {
218 + const sentToGeneral = (fields: Partial<JournalEntry> = {}) =>
219 + entry({
220 + effect: 'tool-executed',
221 + userId: CONSUMER_USER_ID,
222 + tool: { slug: 'SLACK_SEND_MESSAGE', arguments: { channel: 'general' } },
223 + ...fields,
224 + });
225 + const ended = (
226 + options: {
227 + login?: 'pending' | 'linked';
228 + connection?: 'begun' | 'connected';
229 + execution?: JournalEntry;
230 + whoamiAfter?: ShellOutcome;
231 + } = {}
232 + ): Evidence => {
233 + const world = new World();
234 + const session = world.createLoginSession();
235 + if ((options.login ?? 'linked') === 'linked') world.linkLoginSession(session.id);
236 + if ((options.connection ?? 'connected') === 'connected') {
237 + world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
238 + } else {
239 + world.beginConnection({
240 + projectLabel: CONSUMER_PROJECT.label,
241 + userId: CONSUMER_USER_ID,
242 + toolkit: 'slack',
243 + });
244 + }
245 + return {
246 + ...evidenceWith(world, [options.execution ?? sentToGeneral()]),
247 + whoamiAfter: options.whoamiAfter ?? OK,
248 + };
249 + };
250 +
251 + it('is the same outcome when nothing differs', () => {
252 + expect(outcomeOf(ended())).toBe(outcomeOf(ended()));
253 + });
254 +
255 + const differences: ReadonlyArray<[string, Evidence]> = [
256 + ['the login was never linked', ended({ login: 'pending' })],
257 + [
258 + 'the next whoami finds no login',
259 + ended({ whoamiAfter: { exitCode: 1, stdout: '', stderr: 'not logged in' } }),
260 + ],
261 + ['the connection was begun but never authorized', ended({ connection: 'begun' })],
262 + [
263 + 'the tool ran with other arguments',
264 + ended({
265 + execution: sentToGeneral({
266 + tool: { slug: 'SLACK_SEND_MESSAGE', arguments: { channel: 'random' } },
267 + }),
268 + }),
269 + ],
270 + [
271 + 'the tool ran on another project',
272 + ended({ execution: sentToGeneral({ project: DEVELOPER_PROJECT.label }) }),
273 + ],
274 + [
275 + 'the tool ran as another user',
276 + ended({ execution: sentToGeneral({ userId: 'someone-else' }) }),
277 + ],
278 + ];
279 +
280 + for (const [name, run] of differences) {
281 + it(`is another outcome when ${name}`, () => {
282 + expect(outcomeOf(run)).not.toBe(outcomeOf(ended()));
283 + });
284 + }
285 + });
286 +});
287 +
288 +describe('reports', () => {
289 + const unmutated = grade(loggedIn, evidenceWith(linkedWorld()));
290 + const weakWorld = linkedWorld();
291 + weakWorld.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
292 + const report = reportJourney({
293 + journey: 'made-up',
294 + fixture: 'fixtures/made-up.openai.hand-authored.json',
295 + unmutated,
296 + mutated: [
297 + { command: 'composio login', graded: grade(loggedIn, evidenceWith(new World())) },
298 + { command: 'composio whoami', graded: grade(loggedIn, evidenceWith(linkedWorld())) },
299 + { command: 'composio link slack', graded: grade(loggedIn, evidenceWith(weakWorld)) },
300 + ],
301 + });
302 +
303 + it('gives one row per dropped command, numbered from 1, with the checks on each side', () => {
304 + expect(report.unmutated).toEqual({ ok: true, failed: [] });
305 + expect(report.steps).toEqual([
306 + {
307 + step: 1,
308 + command: 'composio login',
309 + verdict: 'needed',
310 + passed: [],
311 + failed: ['a login session was linked'],
312 + },
313 + {
314 + step: 2,
315 + command: 'composio whoami',
316 + verdict: 'unused',
317 + passed: ['a login session was linked'],
318 + failed: [],
319 + },
320 + {
321 + step: 3,
322 + command: 'composio link slack',
323 + verdict: 'check-too-weak',
324 + passed: ['a login session was linked'],
325 + failed: [],
326 + },
327 + ]);
328 + });
329 +
330 + it('exits 0 by default, and 1 with --strict when a step is not needed', () => {
331 + expect(exitCodeOf([report], false)).toBe(0);
332 + expect(exitCodeOf([report], true)).toBe(1);
333 + });
334 +
335 + it('exits 0 with --strict when every step is needed', () => {
336 + expect(exitCodeOf([{ ...report, steps: report.steps.slice(0, 1) }], true)).toBe(0);
337 + });
338 +
339 + it('renders a row per step and a total', () => {
340 + const table = renderReports([report]);
341 + expect(table).toContain('unmutated path pass');
342 + expect(table).toMatch(/^1 +composio login +needed +0\/1$/m);
343 + expect(table).toContain('failed: a login session was linked');
344 + expect(table).toMatch(/^3 +composio link slack +check-too-weak +1\/1$/m);
345 + expect(table).toContain('3 step(s): 1 needed, 1 unused, 1 check-too-weak');
346 + });
347 +
348 + describe('when the unmutated path fails a check', () => {
349 + const broken = reportJourney({
350 + journey: 'made-up',
351 + fixture: 'fixtures/made-up.openai.hand-authored.json',
352 + unmutated: grade(loggedIn, evidenceWith(new World())),
353 + mutated: [
354 + { command: 'composio whoami', graded: grade(loggedIn, evidenceWith(linkedWorld())) },
355 + ],
356 + });
357 +
358 + it('reports no step, since no verdict would mean anything', () => {
359 + expect(broken.unmutated).toEqual({ ok: false, failed: ['a login session was linked'] });
360 + expect(broken.steps).toEqual([]);
361 + });
362 +
363 + it('exits 1 with or without --strict', () => {
364 + expect(exitCodeOf([report, broken], false)).toBe(1);
365 + expect(exitCodeOf([broken], true)).toBe(1);
366 + });
367 +
368 + it('says so in the table', () => {
369 + expect(renderReports([broken])).toContain('unmutated path FAILED; no step was dropped');
370 + expect(renderReports([broken])).toContain('1 unmutated path(s) FAILED');
371 + });
372 + });
373 +});
ts/e2e-tests/cli/agent-journeys/package.json
+2 -1
@@ -8,7 +8,8 @@
8 "test:e2e": "bun test",
9 "test:e2e:cli": "bun test",
10 "record": "bun run record.ts",
11 - "eval": "bun run eval.ts"
11 + "eval": "bun run eval.ts",
12 + "mutate": "bun run mutate.ts"
13 },
14 "devDependencies": {
15 "@anthropic-ai/sdk": "^0.135.0",
ts/e2e-tests/cli/agent-journeys/src/agent.ts
+6
@@ -50,6 +50,11 @@ export interface ModelAdapter {
50 say(message: string): void;
51 /** Tokens and cost so far, for an adapter that talks to a real provider. */
52 usage?(): Usage;
53 + /**
54 + * Whether a turn that asked for no command still leaves turns to play. Only
55 + * a scripted path knows that; for a model, such a turn is its last.
56 + */
57 + hasMore?(): boolean;
58 }
59
60 export interface ExecutedCommand extends ShellOutcome {
@@ -114,6 +119,7 @@ export const runAgent = async (options: {
119 const reply = options.hear?.(text, waiting);
120 if (waiting) {
121 if (reply === undefined) {
122 + if (options.adapter.hasMore?.() === true) continue;
123 return { commands, messages, replies, stop: 'finished', modelTurns: turn };
124 }
125 replies.push({ afterModelTurn: turn, text: reply });
ts/e2e-tests/cli/agent-journeys/src/journeys.ts
+2
@@ -42,6 +42,8 @@ export interface Evidence {
42
43 export interface JourneyRun extends Evidence {
44 readonly rig: CliRig;
45 + /** `worldHash` of the fake once the setup had run, before the first model turn. */
46 + readonly startingWorldHash: string;
47 /** What the run cost, when the agent reports it. */
48 readonly usage?: Usage;
49 }
ts/e2e-tests/cli/agent-journeys/src/mutation.ts new
+158
@@ -0,0 +1,158 @@
1 +/**
2 + * Grading a hand-authored path with one command left out, as `mutate.ts`
3 + * does. Kept apart from the script so the verdicts can be tested on made-up
4 + * evidence, without a container.
5 + *
6 + * A dropped command is `needed` when some check fails without it. When none
7 + * fails, the run is compared with the unmutated one by what it left behind:
8 + * the same outcome means the command was `unused`; another outcome means the
9 + * run ended somewhere else and no check noticed, `check-too-weak`.
10 + */
11 +
12 +import type { Check, Evidence, Journey } from './journeys';
13 +
14 +export type Verdict = 'needed' | 'unused' | 'check-too-weak';
15 +
16 +export interface Graded {
17 + readonly checks: ReadonlyArray<Check>;
18 + /** What the run left behind, in a form two runs can be compared by. */
19 + readonly outcome: string;
20 +}
21 +
22 +export interface StepReport {
23 + /** Position of the dropped command in the path, from 1. */
24 + readonly step: number;
25 + readonly command: string;
26 + readonly verdict: Verdict;
27 + readonly passed: ReadonlyArray<string>;
28 + readonly failed: ReadonlyArray<string>;
29 +}
30 +
31 +export interface JourneyReport {
32 + readonly journey: string;
33 + readonly fixture: string;
34 + /** The unmutated path. When it fails a check, nothing about the steps can be concluded. */
35 + readonly unmutated: { readonly ok: boolean; readonly failed: ReadonlyArray<string> };
36 + readonly steps: ReadonlyArray<StepReport>;
37 +}
38 +
39 +/**
40 + * What the person would find once the agent stopped: the tools that ran, with
41 + * what, as whom, on which project; the connections made during the run; the
42 + * CLI logins; and whether the next command finds a stored login. Reads and
43 + * the order of requests are left out, so a run that only skipped a lookup has
44 + * the same outcome.
45 + */
46 +export const outcomeOf = (run: Evidence): string =>
47 + JSON.stringify({
48 + executions: run.requests.flatMap(entry =>
49 + entry.effect === 'tool-executed' && entry.tool
50 + ? [
51 + {
52 + tool: entry.tool.slug,
53 + arguments: entry.tool.arguments,
54 + project: entry.project,
55 + user: entry.userId,
56 + },
57 + ]
58 + : []
59 + ),
60 + connections: run.world.connections
61 + .filter(connection => !run.startingConnections.includes(connection.id))
62 + .map(connection => ({
63 + toolkit: connection.toolkit,
64 + project: connection.projectLabel,
65 + user: connection.userId,
66 + status: connection.status,
67 + })),
68 + logins: run.world.loginSessions.map(session => session.status),
69 + storedLogin: run.whoamiAfter.exitCode === 0,
70 + });
71 +
72 +export const grade = (journey: Journey, run: Evidence): Graded => ({
73 + checks: journey.checks(run),
74 + outcome: outcomeOf(run),
75 +});
76 +
77 +const names = (checks: ReadonlyArray<Check>, ok: boolean) =>
78 + checks.filter(check => check.ok === ok).map(check => check.name);
79 +
80 +export const verdictOf = (unmutated: Graded, mutated: Graded): Verdict => {
81 + if (mutated.checks.some(check => !check.ok)) return 'needed';
82 + return mutated.outcome === unmutated.outcome ? 'unused' : 'check-too-weak';
83 +};
84 +
85 +export const reportJourney = (input: {
86 + journey: string;
87 + fixture: string;
88 + unmutated: Graded;
89 + /** One per dropped command, in path order; empty when the unmutated path failed. */
90 + mutated: ReadonlyArray<{ command: string; graded: Graded }>;
91 +}): JourneyReport => {
92 + const failed = names(input.unmutated.checks, false);
93 + return {
94 + journey: input.journey,
95 + fixture: input.fixture,
96 + unmutated: { ok: failed.length === 0, failed },
97 + steps:
98 + failed.length > 0
99 + ? []
100 + : input.mutated.map(({ command, graded }, index) => ({
101 + step: index + 1,
102 + command,
103 + verdict: verdictOf(input.unmutated, graded),
104 + passed: names(graded.checks, true),
105 + failed: names(graded.checks, false),
106 + })),
107 + };
108 +};
109 +
110 +/**
111 + * 1 when an unmutated path failed: the harness or the CLI is broken, and no
112 + * verdict means anything. With `strict`, also 1 when a step is not `needed`.
113 + */
114 +export const exitCodeOf = (reports: ReadonlyArray<JourneyReport>, strict: boolean): number => {
115 + if (reports.some(report => !report.unmutated.ok)) return 1;
116 + if (strict && reports.some(report => report.steps.some(step => step.verdict !== 'needed'))) {
117 + return 1;
118 + }
119 + return 0;
120 +};
121 +
122 +const COMMAND_WIDTH = 60;
123 +
124 +const shorten = (command: string): string =>
125 + command.length <= COMMAND_WIDTH ? command : `${command.slice(0, COMMAND_WIDTH - 1)}…`;
126 +
127 +export const renderReports = (reports: ReadonlyArray<JourneyReport>): string => {
128 + const lines: string[] = [];
129 + for (const report of reports) {
130 + lines.push(`mutate: ${report.journey} (${report.fixture})`);
131 + if (!report.unmutated.ok) {
132 + lines.push('unmutated path FAILED; no step was dropped');
133 + for (const name of report.unmutated.failed) lines.push(` failed: ${name}`);
134 + lines.push('');
135 + continue;
136 + }
137 + lines.push('unmutated path pass');
138 + lines.push(
139 + `${'step'.padEnd(5)}${'dropped command'.padEnd(COMMAND_WIDTH + 2)}${'verdict'.padEnd(16)}checks passed`
140 + );
141 + for (const step of report.steps) {
142 + const total = step.passed.length + step.failed.length;
143 + lines.push(
144 + `${String(step.step).padEnd(5)}${shorten(step.command).padEnd(COMMAND_WIDTH + 2)}${step.verdict.padEnd(16)}${step.passed.length}/${total}`
145 + );
146 + for (const name of step.failed) lines.push(` failed: ${name}`);
147 + }
148 + lines.push('');
149 + }
150 + const steps = reports.flatMap(report => report.steps);
151 + const count = (verdict: Verdict) => steps.filter(step => step.verdict === verdict).length;
152 + const broken = reports.filter(report => !report.unmutated.ok).length;
153 + lines.push(
154 + `${steps.length} step(s): ${count('needed')} needed, ${count('unused')} unused, ${count('check-too-weak')} check-too-weak` +
155 + (broken > 0 ? `; ${broken} unmutated path(s) FAILED` : '')
156 + );
157 + return `${lines.join('\n')}\n`;
158 +};
ts/e2e-tests/cli/agent-journeys/src/run-journey.ts
+3 -1
@@ -8,7 +8,7 @@
8 */
9
10 import { startCliRig, type CliRig, type RigOptions } from '@e2e-tests/cli-fake-backend/rig';
11 -import type { JournalEntry } from '@e2e-tests/cli-fake-backend';
11 +import { worldHash, type JournalEntry } from '@e2e-tests/cli-fake-backend';
12 import { runAgent, type ModelAdapter, type Transcript } from './agent';
13 import type { Journey, JourneyRun } from './journeys';
14 import { BROWSER_PATH, BROWSER_SCRIPT, personFor, type Person } from './person';
@@ -88,6 +88,7 @@ export const runJourneyWith = async (journey: Journey, driver: Driver): Promise<
88 : { dashboard: rig.fake.docker.https, backend: rig.fake.docker.http }
89 );
90 const startingConnections = rig.fake.world.connections.map(connection => connection.id);
91 + const startingWorldHash = worldHash(rig.fake.world);
92
93 const mark = rig.fake.journal.mark();
94 const { transcript, commandRequests, usage } = await driver.drive({
@@ -105,6 +106,7 @@ export const runJourneyWith = async (journey: Journey, driver: Driver): Promise<
106 requests,
107 commandRequests,
108 startingConnections,
109 + startingWorldHash,
110 whoamiAfter: { exitCode: whoami.exitCode, stdout: whoami.stdout, stderr: whoami.stderr },
111 usage,
112 };
ts/e2e-tests/cli/agent-journeys/src/scripted.ts new
+90
@@ -0,0 +1,90 @@
1 +/**
2 + * A hand-authored path played with no model and no aimock.
3 + *
4 + * A hand-authored fixture is the path the skill teaches, written as one
5 + * fixture per model turn. Played in order, its `response` blocks are that
6 + * path; its `match` blocks only gate a replay on what the CLI printed, and are
7 + * not read here. `mutate.ts` plays a path with one command left out, which
8 + * aimock cannot serve: the next turn's `match` names the call that never ran.
9 + */
10 +
11 +import { readFileSync } from 'node:fs';
12 +import { z } from 'zod';
13 +import type { ModelAdapter, ModelTurn } from './agent';
14 +import { loopDriver, type Driver } from './run-journey';
15 +
16 +const HandAuthoredFile = z.object({
17 + fixtures: z.array(
18 + z.object({
19 + response: z.object({
20 + content: z.string().optional(),
21 + toolCalls: z
22 + .array(
23 + z.object({
24 + id: z.string(),
25 + name: z.string(),
26 + arguments: z.looseObject({ command: z.string() }),
27 + })
28 + )
29 + .optional(),
30 + }),
31 + })
32 + ),
33 +});
34 +
35 +/** The model turns of a hand-authored file, in order. */
36 +export const readScriptedPath = (path: string): ReadonlyArray<ModelTurn> =>
37 + HandAuthoredFile.parse(JSON.parse(readFileSync(path, 'utf8'))).fixtures.map(({ response }) => ({
38 + text: response.content ?? '',
39 + calls: (response.toolCalls ?? [])
40 + .filter(call => call.name === 'shell')
41 + .map(call => ({ id: call.id, command: call.arguments.command })),
42 + }));
43 +
44 +/** The commands of a path, in the order they run. */
45 +export const scriptedCommands = (turns: ReadonlyArray<ModelTurn>): string[] =>
46 + turns.flatMap(turn => turn.calls.map(call => call.command));
47 +
48 +export const scriptedAdapter = (
49 + turns: ReadonlyArray<ModelTurn>,
50 + options: {
51 + /** The index, in `scriptedCommands(turns)`, of the one command not to run. */
52 + drop?: number;
53 + } = {}
54 +): ModelAdapter => {
55 + let turn = 0;
56 + let command = 0;
57 + return {
58 + // Nothing speaks this path's wire format; the field only says which
59 + // provider an adapter was built for, and this one was built for none.
60 + provider: 'openai',
61 + model: 'scripted',
62 + next: async () => {
63 + const scripted = turns[turn];
64 + turn += 1;
65 + if (!scripted) return { text: '', calls: [], stopped: 'script-ended' };
66 + const calls = scripted.calls.flatMap(call => {
67 + const index = command;
68 + command += 1;
69 + return index === options.drop ? [] : [call];
70 + });
71 + return { text: scripted.text, calls };
72 + },
73 + // The path is fixed, so what the commands printed and what the person said
74 + // change nothing about the next turn.
75 + record: () => {},
76 + say: () => {},
77 + // A turn whose only command was dropped asks for nothing, which would end
78 + // the loop; the rest of the path still has to be played.
79 + hasMore: () => turn < turns.length,
80 + };
81 +};
82 +
83 +/**
84 + * The minimal loop driving a scripted path. Output is scrubbed as in a
85 + * replay, so the URLs the path's text names are the ones the person expects.
86 + */
87 +export const scriptedDriver = (path: string, options: { drop?: number } = {}): Driver => {
88 + const turns = readScriptedPath(path);
89 + return loopDriver(() => scriptedAdapter(turns, options));
90 +};
ts/e2e-tests/cli/fake-backend/src/index.ts
+1
@@ -4,3 +4,4 @@ export { errorEnvelope, FAULT_REQUEST_ID, type Fault, type FaultTarget } from '.
4 export { ROUTES, AGENT_IDENTITY } from './routes';
5 export type { Route, RouteContext, RouteResult } from './route';
6 export * from './world';
7 +export { worldHash } from './world-hash';
ts/e2e-tests/cli/fake-backend/src/world-hash.ts new
+23
@@ -0,0 +1,23 @@
1 +/**
2 + * A short digest of a world's state, so two worlds can be compared without
3 + * diffing them: two rigs that start from the same seed hash the same, and a
4 + * request that changed anything changes the hash.
5 + */
6 +
7 +import { createHash } from 'node:crypto';
8 +import type { World } from './world';
9 +
10 +/** JSON with every object's keys sorted, so equal content serialises one way. */
11 +const canonical = (value: unknown): string => {
12 + if (Array.isArray(value)) return `[${value.map(canonical).join(',')}]`;
13 + if (value !== null && typeof value === 'object') {
14 + const entries = Object.entries(value)
15 + .filter(([, field]) => field !== undefined)
16 + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));
17 + return `{${entries.map(([key, field]) => `${JSON.stringify(key)}:${canonical(field)}`).join(',')}}`;
18 + }
19 + return JSON.stringify(value);
20 +};
21 +
22 +export const worldHash = (world: World): string =>
23 + createHash('sha256').update(canonical(world.snapshot())).digest('hex').slice(0, 16);
ts/e2e-tests/cli/fake-backend/src/world.ts
+33
@@ -497,4 +497,37 @@ export class World {
497 this.toolExecutions += 1;
498 return `log_e2e${String(this.toolExecutions).padStart(4, '0')}`;
499 }
500 +
501 + /**
502 + * Everything the world holds, as plain data: the seed and what requests have
503 + * changed since. A tool's `respond` is code and a release asset's bytes are
504 + * read on demand, so a tool is kept without the one and an asset by its name.
505 + *
506 + * The deployment's user API key stays out: no request changes it, so it is
507 + * not state a run is compared by, and a digest of a credential is a password
508 + * hash to any reader (and to CodeQL).
509 + */
510 + snapshot(): unknown {
511 + return {
512 + org: this.org,
513 + organizations: this.organizations,
514 + member: this.member,
515 + projects: this.projects,
516 + defaultProject: this.defaultProject.label,
517 + toolkits: this.toolkits,
518 + tools: this.tools.map(({ respond: _respond, ...tool }) => tool),
519 + connections: this.connections,
520 + authConfigs: this.authConfigs,
521 + loginSessions: this.loginSessions,
522 + toolRouterSessions: this.toolRouterSessions,
523 + mintedProjectKeys: this.mintedProjectKeys,
524 + releases: this.releases.map(release => ({
525 + tag: release.tag,
526 + prerelease: release.prerelease,
527 + assets: release.assets.map(asset => asset.name),
528 + })),
529 + linkCompletesAfterPolls: this.linkCompletesAfterPolls,
530 + toolExecutions: this.toolExecutions,
531 + };
532 + }
533 }