@devries-cameron20 / composio Mirror · read-only / commits / ab262b214
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 03:37 UTC

test(cli-e2e): gate mutate --strict on check-too-weak steps only (#5419)

This PR: - builds on https://github.com/ComposioHQ/composio/pull/5418 — base is `test/agent-journey-grader-guards` - makes `mutate.ts --strict` fail on `check-too-weak` steps alone, instead of failing on any step that is not `needed` - moves the exit reason into `failuresOf()` in `src/mutation.ts`, so `--strict` names the journey, step and command it rejects on stderr instead of exiting silently - keeps `unused` in the report and the README table, where the seven listed steps are expected: reads, a `--help`, and commands that fail on purpose - updates `HILLCLIMB.md` and the agent-journeys README to say what the gate now means ## Context `--strict` was defined as "every step is load-bearing", which no journey can satisfy. A read or a `cat` of a skill reference changes no state, so no outcome check can depend on it and it is always `unused`; dropping those steps to make the flag green would delete the exploration the hand-authored paths exist to document. The defect signal is the other verdict: `check-too-weak` means a run ended somewhere else and every check still passed. With that split, `--strict` is green on the current tree and means something: | veredict | what it says | `--strict` | | --- | --- | --- | | `needed` | a check noticed the step is gone | pass | | `unused` | no check noticed, and the run ended the same way | pass | | `check-too-weak` | no check noticed, but the run ended somewhere else | fail | Wiring it into CI is left out on purpose: it is 22 containers, 1m41s locally, and it only changes meaning when the journeys, the checks or the fake backend change — a path-filtered job is the follow-up, not a step on every PR. ## Verification - `bun run mutate.ts --strict` on all five journeys: exit 0, `17 step(s): 10 needed, 7 unused, 0 check-too-weak`, 1m41s - `bun test ./mutation.test.ts`: 28 pass, 0 fail — including `--strict` passing with `unused` steps and failing with `check-too-weak`, and the failure message it prints - `bun run typecheck`: clean; `oxlint`, antislop and prettier on the changed files: clean - `bun test e2e.test.ts` is unchanged by this PR (no harness file is touched), so it is not re-run here

Alberto Schiabel committed Nov 18, 2026 at 18:45 UTC ab262b214d5ed2f7f39a50685eaddfc7ca3d6c46
5 files changed +59 -14
ts/e2e-tests/cli/agent-journeys/HILLCLIMB.md
+1 -1
@@ -50,7 +50,7 @@ set -a; . ~/.config/composio-e2e/llm.env; set +a
50
51 - **Do not change the harness in the same step as the skill.** The journeys, the checks, the person and the fake backend (`agent-journeys/src/`, `fake-backend/src/`) are what grade a run; their hash is part of every batch, and `compare` refuses two batches with different hashes. Fix a grader in its own change, then measure a new baseline.
52 - **Do not tune the checks to the agent.** A check that fails because the agent took a valid path the check did not foresee is a grader bug; fix it with a test in `journeys.test.ts`. A check that fails because the agent did the wrong thing is the signal.
53 -- **Know what a journey's checks notice before trusting its pass^k.** `bun run mutate.ts --journeys <id>` plays the hand-authored path once per command with that command left out (see the [README](./README.md#what-a-mutation-run-proves)). A step it calls `check-too-weak` is a way to end somewhere else that the journey still passes; strengthen the check, in its own change, before reading a rate off it.
53 +- **Know what a journey's checks notice before trusting its pass^k.** `bun run mutate.ts --journeys <id>` plays the hand-authored path once per command with that command left out (see the [README](./README.md#what-a-mutation-run-proves)). A step it calls `check-too-weak` is a way to end somewhere else that the journey still passes; strengthen the check, in its own change, before reading a rate off it. `--strict` exits 1 on one, so a run with it is the check to make before trusting a rate, or to gate on in CI when the containers are worth it. A step it calls `unused` is a read or a command that fails on purpose, and no check should depend on it.
54 - **Repeated runs are comparable only if they start alike.** `determinism.test.ts` pins the seeded world, and the replay suite checks that every replayed attempt of a journey started from the same one and that a rig left idle after login stays quiet. `eval.ts` does not record the starting world of its runs; it relies on those tests. When one fails, fix the seed or the leak before comparing batches.
55 - **Do not add the journeys' wording to the skill.** The skill is for every user's task; a sentence that only matches a journey's prompt raises train and nothing else.
56 - **Spend little.** A Haiku run costs about $0.002 to $0.01; Opus about 40 times that. `--max-cost-usd` (default $2) stops a batch from starting runs past it, and `--run-budget-usd` caps one run.
ts/e2e-tests/cli/agent-journeys/README.md
+1 -1
@@ -34,7 +34,7 @@ A replay on its own proves nothing: aimock answers the same whether a command su
34 - `unused`: every check passed, and the run left the same outcome as the whole path: the same tools ran with the same arguments, as the same user on the same project; the same connections and logins were made; the next `composio whoami` found the same login. A read the checks do not depend on lands here.
35 - `check-too-weak`: every check passed, but the outcome differs. The run ended somewhere else and no check noticed.
36
37 -When the whole path fails a check, the tool says so, drops nothing, and exits 1; `--strict` also exits 1 on any `unused` or `check-too-weak` step, so with the `unused` steps listed below it fails on four of the five journeys today. Results go to `results/<batch>/`.
37 +When the whole path fails a check, the tool says so, drops nothing, and exits 1. `--strict` also exits 1 when a step is `check-too-weak` and names it, which is the gate worth having: it fails only when a way to end somewhere else went unnoticed. It passes on every journey today. An `unused` step never fails, because a read or a command that fails on purpose changes nothing for an outcome check to notice — see the steps listed below. Results go to `results/<batch>/`.
38
39 What it proves: whether the checks notice that one step of the taught path is missing. What it does not prove: that they catch a wrong path the hand-authored file never takes (another command in place of a dropped one, two steps missing, the right step on the wrong project), or anything about a real model. The outcome compares only what is listed above; a difference elsewhere in the fake's state is not seen.
40
ts/e2e-tests/cli/agent-journeys/mutate.ts
+7 -2
@@ -14,8 +14,10 @@
14 * anything about a real model.
15 *
16 * Exits 1 when an unmutated path fails a check, and with `--strict` also when
17 - * a step is `unused` or `check-too-weak`. Results go to `results/<batch>/`
18 - * (git-ignored). Needs Docker; spends nothing.
17 + * a step is `check-too-weak`: a way to end somewhere else that no check
18 + * noticed. `unused` is reported and never fails, because a read or a command
19 + * that fails on purpose changes nothing for an outcome check to notice.
20 + * Results go to `results/<batch>/` (git-ignored). Needs Docker; spends nothing.
21 */
22
23 import { mkdirSync, writeFileSync } from 'node:fs';
@@ -25,6 +27,7 @@ import { HAND_AUTHORED, listFixtures } from './src/fixtures';
27 import { journeyById, JOURNEYS, type Journey } from './src/journeys';
28 import {
29 exitCodeOf,
30 + failuresOf,
31 grade,
32 renderReports,
33 reportJourney,
@@ -99,4 +102,6 @@ writeFileSync(join(directory, 'summary.txt'), table);
102
103 process.stdout.write(values.json ? `${JSON.stringify(reports, null, 2)}\n` : table);
104 console.error(`written ${relative(process.cwd(), directory)}`);
105 +const failures = failuresOf(reports, values.strict ?? false);
106 +for (const failure of failures) console.error(`failed ${failure}`);
107 process.exit(exitCodeOf(reports, values.strict ?? false));
ts/e2e-tests/cli/agent-journeys/mutation.test.ts
+23 -1
@@ -17,6 +17,7 @@ import { fixturePath, HAND_AUTHORED } from './src/fixtures';
17 import type { Check, Evidence, Journey } from './src/journeys';
18 import {
19 exitCodeOf,
20 + failuresOf,
21 grade,
22 outcomeOf,
23 renderReports,
@@ -327,9 +328,25 @@ describe('reports', () => {
328 ]);
329 });
330
330 - it('exits 0 by default, and 1 with --strict when a step is not needed', () => {
331 + // `report` has one step of each verdict: needed, unused, check-too-weak.
332 + it('exits 0 by default, whatever the verdicts are', () => {
333 expect(exitCodeOf([report], false)).toBe(0);
334 + expect(failuresOf([report], false)).toEqual([]);
335 + });
336 +
337 + it('exits 1 with --strict on a check-too-weak step, and says which', () => {
338 expect(exitCodeOf([report], true)).toBe(1);
339 + expect(failuresOf([report], true)).toEqual([
340 + 'made-up: step 3 (composio link slack) is check-too-weak',
341 + ]);
342 + });
343 +
344 + // A read, a `--help`, or a command that fails on purpose changes nothing for
345 + // an outcome check to notice, so `unused` is not a grader defect.
346 + it('exits 0 with --strict when the only steps that are not needed are unused', () => {
347 + const withoutWeak = { ...report, steps: report.steps.slice(0, 2) };
348 + expect(exitCodeOf([withoutWeak], true)).toBe(0);
349 + expect(failuresOf([withoutWeak], true)).toEqual([]);
350 });
351
352 it('exits 0 with --strict when every step is needed', () => {
@@ -365,6 +382,11 @@ describe('reports', () => {
382 expect(exitCodeOf([broken], true)).toBe(1);
383 });
384
385 + it('names the journey whose path broke, in both modes', () => {
386 + expect(failuresOf([broken], false)).toEqual(['made-up: the unmutated path failed a check']);
387 + expect(failuresOf([broken], true)).toEqual(['made-up: the unmutated path failed a check']);
388 + });
389 +
390 it('says so in the table', () => {
391 expect(renderReports([broken])).toContain('unmutated path FAILED; no step was dropped');
392 expect(renderReports([broken])).toContain('1 unmutated path(s) FAILED');
ts/e2e-tests/cli/agent-journeys/src/mutation.ts
+27 -9
@@ -108,16 +108,34 @@ export const reportJourney = (input: {
108 };
109
110 /**
111 - * 1 when an unmutated path failed: the harness or the CLI is broken, and no
112 - * verdict means anything. With `strict`, also 1 when a step is not `needed`.
111 + * Why a run exits non-zero, in the order a reader would fix them. An
112 + * unmutated path that fails means the harness or the CLI is broken, and no
113 + * verdict means anything. With `strict`, a `check-too-weak` step also fails:
114 + * the run ended somewhere else and no check noticed.
115 + *
116 + * `unused` never fails. A read, a `--help`, or a command that fails on purpose
117 + * changes nothing for a check to notice, and the checks grade the outcome, so
118 + * a journey made only of `needed` steps is not a state worth aiming at.
119 */
114 -export const exitCodeOf = (reports: ReadonlyArray<JourneyReport>, strict: boolean): number => {
115 - if (reports.some(report => !report.unmutated.ok)) return 1;
116 - if (strict && reports.some(report => report.steps.some(step => step.verdict !== 'needed'))) {
117 - return 1;
118 - }
119 - return 0;
120 -};
120 +export const failuresOf = (
121 + reports: ReadonlyArray<JourneyReport>,
122 + strict: boolean
123 +): ReadonlyArray<string> => [
124 + ...reports
125 + .filter(report => !report.unmutated.ok)
126 + .map(report => `${report.journey}: the unmutated path failed a check`),
127 + ...(strict
128 + ? reports.flatMap(report =>
129 + report.steps
130 + .filter(step => step.verdict === 'check-too-weak')
131 + .map(step => `${report.journey}: step ${step.step} (${step.command}) is check-too-weak`)
132 + )
133 + : []),
134 +];
135 +
136 +/** 1 when `failuresOf` is not empty, else 0. */
137 +export const exitCodeOf = (reports: ReadonlyArray<JourneyReport>, strict: boolean): number =>
138 + failuresOf(reports, strict).length > 0 ? 1 : 0;
139
140 const COMMAND_WIDTH = 60;
141