@devries-cameron20 / composio-1 / commits / 2a4d95ac2
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

test(ts/e2e): replay agent journeys against the fake backend (#5353)

This PR: - part of [PRDE-2212](https://linear.app/composio/issue/PRDE-2212/cli-e2e-tests-fake-backend-operation-suites-and-agent-journeys-with) - builds on top of https://github.com/ComposioHQ/composio/pull/5352; split from https://github.com/ComposioHQ/composio/pull/5312 - replays five `agent-journeys` through `@copilotkit/aimock` in strict mode, using the built `composio-cli` skill and a minimal agent with one `shell` tool - grades executed commands and the backend journal for the requested send, project, user, connection, and stored login; reports wrong-tree `detours` separately from success - models a `person` who approves only authorization pages that reach them and can answer when the agent waits - keeps hand-authored paths, recorded fixtures, and every historical transcript with its revision and skill hash; a recording is one sample and replay cannot measure a changed skill - adds keyless replay CI and retains the reusable driver contract and shared `Usage` type here so replay code typechecks without the live runner ## Context Frozen install, CLI dependency builds, package typechecks, authored-file Prettier, and all 84 keyless replay tests passed; the inherited 31 contract, 133 operation, and five upgrade tests passed too. The generated lockfile retains pnpm formatting, matching the base convention. The README describes replay and recording only. Live drivers, costs, metrics, train/holdout fields, and repeated batches arrive separately. No model API was called during validation. Complete CLI workflow validation after the base merges and this PR is retargeted to `next`.

Alberto Schiabel committed Nov 16, 2026 at 12:49 UTC 2a4d95ac29548eb059624a32d1e896dd4908c97e
51 files changed +5283 -11
.github/workflows/ts.test-e2e.yml
+4 -1
@@ -219,7 +219,8 @@ jobs:
219 composio.cli_version=${{ steps.cli-version.outputs.value }}
220 org.opencontainers.image.revision=${{ github.sha }}
221
222 - # These suites run the CLI against a fake Composio deployment, so they need no key and run on every PR, fork and
222 + # These suites run the CLI against a fake Composio deployment and replayed
223 + # model turns, so they need no key and run on every PR, fork and
224 # Dependabot PRs included. The keys are blanked to keep it that way.
225 - name: Run CLI E2E Tests (no API keys)
226 env:
@@ -231,6 +232,7 @@ jobs:
232 pnpm exec turbo test:e2e:cli
233 --filter=@e2e-tests/cli-fake-backend
234 --filter=@e2e-tests/cli-operations
235 + --filter=@e2e-tests/cli-agent-journeys
236
237 # After the keyless step, so a missing key cannot keep those suites from running.
238 - name: Check API keys
@@ -266,6 +268,7 @@ jobs:
268 --filter='@e2e-tests/cli-*'
269 --filter='!@e2e-tests/cli-fake-backend'
270 --filter='!@e2e-tests/cli-operations'
271 + --filter='!@e2e-tests/cli-agent-journeys'
272
273 # Cloudflare Workerd E2E tests
274 test-e2e-cloudflare:
pnpm-lock.yaml
+64
@@ -30,6 +30,9 @@ catalogs:
30 '@composio/client':
31 specifier: 2.1.32
32 version: 2.1.32
33 + '@copilotkit/aimock':
34 + specifier: 1.47.0
35 + version: 1.47.0
36 '@effect/language-service':
37 specifier: ^0.87.5
38 version: 0.87.5
@@ -240,6 +243,33 @@ importers:
243 specifier: 'catalog:'
244 version: 7.0.2
245
246 + ts/e2e-tests/cli/agent-journeys:
247 + devDependencies:
248 + '@anthropic-ai/sdk':
249 + specifier: ^0.135.0
250 + version: 0.135.0(zod@4.6.7)
251 + '@copilotkit/aimock':
252 + specifier: 'catalog:'
253 + version: 1.47.0(vitest@5.1.0)
254 + '@e2e-tests/cli-fake-backend':
255 + specifier: workspace:*
256 + version: link:../fake-backend
257 + '@e2e-tests/utils':
258 + specifier: workspace:*
259 + version: link:../../_utils
260 + '@types/bun':
261 + specifier: 'catalog:'
262 + version: 1.4.4
263 + openai:
264 + specifier: 'catalog:'
265 + version: 7.55.0(@aws-sdk/credential-provider-node@3.972.87)(@smithy/signature-v4@5.7.4)(google-auth-library@10.9.1)(undici@6.29.1)(ws@8.22.2)(zod@4.6.7)
266 + typescript:
267 + specifier: 'catalog:'
268 + version: 7.0.2
269 + zod:
270 + specifier: 'catalog:'
271 + version: 4.6.7
272 +
273 ts/e2e-tests/cli/agent-signin:
274 devDependencies:
275 '@e2e-tests/utils':
@@ -2076,6 +2106,15 @@ packages:
2106 zod:
2107 optional: true
2108
2109 + '@anthropic-ai/sdk@0.135.0':
2110 + resolution: {integrity: sha512-WWrdL57JhU5KoynpTjt9GXK6k7ud9nPw2uw/Trn+oBXh6s0KpjhAccWZm2Qxw08ARYG1UDQj5iC+JO42j1rt/g==}
2111 + hasBin: true
2112 + peerDependencies:
2113 + zod: ^3.25.0 || ^4.0.0
2114 + peerDependenciesMeta:
2115 + zod:
2116 + optional: true
2117 +
2118 '@anthropic-ai/sdk@1.7.0':
2119 resolution: {integrity: sha512-Z+0c1iHQozlt4ymMHapPXEXfFnSbPEWGnSKbkavt+qeyaMQ3Sd6QNxfQj7vyhHYqiqYadC9shrXub8ISKf7xjQ==}
2120 hasBin: true
@@ -2435,6 +2474,19 @@ packages:
2474 effect:
2475 optional: true
2476
2477 + '@copilotkit/aimock@1.47.0':
2478 + resolution: {integrity: sha512-wFPQ8olsToCnsEfJricsCizm9YFrrv/GJ77avB0YRsGP1r5hxvQ4SOl0ATUBa6aowSXkr3ENJaxWsndVXSZfkw==}
2479 + engines: {node: '>=20.15.0'}
2480 + hasBin: true
2481 + peerDependencies:
2482 + jest: '>=29'
2483 + vitest: '>=3'
2484 + peerDependenciesMeta:
2485 + jest:
2486 + optional: true
2487 + vitest:
2488 + optional: true
2489 +
2490 '@cspotcode/source-map-support@0.8.1':
2491 resolution: {integrity: sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw==}
2492 engines: {node: '>=12'}
@@ -7128,6 +7180,14 @@ snapshots:
7180 optionalDependencies:
7181 zod: 4.6.7
7182
7183 + '@anthropic-ai/sdk@0.135.0(zod@4.6.7)':
7184 + dependencies:
7185 + '@opentelemetry/api': 1.10.0
7186 + json-schema-to-ts: 3.1.1
7187 + standardwebhooks: 1.1.1
7188 + optionalDependencies:
7189 + zod: 4.6.7
7190 +
7191 '@anthropic-ai/sdk@1.7.0(zod@4.6.7)':
7192 dependencies:
7193 '@opentelemetry/api': 1.10.0
@@ -7598,6 +7658,10 @@ snapshots:
7658 optionalDependencies:
7659 effect: 4.1.3
7660
7661 + '@copilotkit/aimock@1.47.0(vitest@5.1.0)':
7662 + optionalDependencies:
7663 + vitest: 5.1.0(@opentelemetry/api@1.10.0)(@types/node@26.6.6)(@vitest/ui@5.1.0)(vite@7.3.5(@types/node@26.6.6)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
7664 +
7665 '@cspotcode/source-map-support@0.8.1':
7666 dependencies:
7667 '@jridgewell/trace-mapping': 0.3.9
pnpm-workspace.yaml
+2
@@ -37,6 +37,8 @@ catalog:
37 '@cloudflare/vitest-pool-workers': 0.23.0
38 '@cloudflare/workers-types': ^5.20261110.1
39 '@composio/client': 2.1.32
40 + # Same version as harness/run.mjs (AIMOCK_VERSION); move both together.
41 + '@copilotkit/aimock': 1.47.0
42 '@effect/language-service': ^0.87.5
43 '@effect/platform-bun': ^4.1.3
44 '@effect/vitest': ^4.1.3
ts/e2e-tests/cli/README.md
+10 -9
@@ -10,13 +10,14 @@ Tests use `runCmd` to execute shell commands in the container and assert on exit
10
11 ## Test Suites
12
13 -| Suite | Description | Env Vars |
14 -| ------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------- |
15 -| [upgrade](./upgrade/) | `composio upgrade` replaces a running Linux executable | None |
16 -| [operations](./operations/) | Login, session, search, execute, link, run, upgrade and their `dev` counterparts under a fault catalogue | None |
17 -| [fake-backend](./fake-backend/) | Test support: the fake Composio deployment the operation suites talk to, and its contract check | None |
18 -| [version](./version/) | `composio version` output and exit code | None |
19 -| [whoami](./whoami/) | `composio whoami` prints the API key | `COMPOSIO_USER_API_KEY` |
13 +| Suite | Description | Env Vars |
14 +| ----------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------- |
15 +| [upgrade](./upgrade/) | `composio upgrade` replaces a running Linux executable | None |
16 +| [operations](./operations/) | Login, session, search, execute, link, run, upgrade and their `dev` counterparts under a fault catalogue | None |
17 +| [agent-journeys](./agent-journeys/) | Tasks driven by a model replayed through aimock, with the `composio-cli` skill as its guidance | None |
18 +| [fake-backend](./fake-backend/) | Test support: the fake Composio deployment the two suites above talk to, and its contract check | None |
19 +| [version](./version/) | `composio version` output and exit code | None |
20 +| [whoami](./whoami/) | `composio whoami` prints the API key | `COMPOSIO_USER_API_KEY` |
21
22 ## Journeys
23
@@ -31,9 +32,9 @@ const result = await journey.exec('composio whoami');
32 await journey.stop();
33 ```
34
34 -A journey container mounts nothing from the host and passes through no API key. The `operations` suites use it through `startCliRig` from `@e2e-tests/cli-fake-backend/rig`, which also starts the fake backend and points every host the CLI knows at it. Other outbound traffic is not blocked.
35 +A journey container mounts nothing from the host and passes through no API key. The `operations` and `agent-journeys` suites use it through `startCliRig` from `@e2e-tests/cli-fake-backend/rig`, which also starts the fake backend and points every host the CLI knows at it. Other outbound traffic is not blocked.
36
36 -The `operations` suites and `fake-backend` need no secret. CI runs them in their own step with the API keys blanked, on every pull request, fork and Dependabot ones included; the suites that talk to staging run in a second step that is skipped when the keys are missing.
37 +Those two suites and `fake-backend` need no secret. CI runs them in their own step with the API keys blanked, on every pull request, fork and Dependabot ones included; the suites that talk to staging run in a second step that is skipped when the keys are missing.
38
39 ## Isolation Tool
40
ts/e2e-tests/cli/agent-journeys/README.md new
+164
@@ -0,0 +1,164 @@
1 +# CLI agent journeys
2 +
3 +A journey is a task given to a model that has one tool, a shell, and one piece of guidance, the `composio-cli` skill. The model's commands run against the real `composio` binary in a journey container; the binary talks to the fake backend in [`../fake-backend`](../fake-backend/README.md); the model itself is replayed by [aimock](https://github.com/CopilotKit/aimock) from a fixture file.
4 +
5 +```bash
6 +pnpm --filter @e2e-tests/cli-agent-journeys test:e2e:cli
7 +```
8 +
9 +No API key is read and no model is called when the tests run.
10 +
11 +## What a passing journey proves
12 +
13 +Read this before drawing a conclusion from a green run.
14 +
15 +- **Hand-authored fixtures** (`*.openai.hand-authored.json`) prove that _the path the skill teaches works against the binary_. Nobody's model chose those commands. They do not show that a real model follows the skill.
16 +- **Recorded fixtures** (`*.anthropic.<model>.json`) prove that _this model, given the skill and the prompt, once took this path, and the path still works against the binary_. One recording is one sample, not a rate.
17 +- Neither kind proves anything about the real backend. See the fake's fidelity section.
18 +- **A replay does not notice a changed skill or a changed tool schema.** A fixture is matched on the turn's position, the previous call, and that command's exit code (hand-authored: a substring of its output). The system prompt and the schema text a command printed are not part of the match, so after a skill edit the replays stay green and say nothing about the new skill. Only a new recording does.
19 +
20 +A replay on its own proves nothing: aimock answers the same whether a command succeeded or failed. So every journey is judged by its checks (`src/journeys.ts`), which read the commands that ran, their exit codes and output, and the fake backend's journal: which tool ran, on which project, as which user. None of them reads the model's prose. See [How a run is graded](#how-a-run-is-graded).
21 +
22 +## How a run works
23 +
24 +1. `src/skill.ts` renders the skill from `ts/packages/cli/skills-src/composio-cli`, so a change to the skill changes the system prompt here. The system prompt is the built `SKILL.md` preceded by one line naming the directory where the skill's files are in the container, as an agent host does; the references are written there for the model to `cat`.
25 +2. `src/agent.ts` is the loop: ask the model, run each `shell` call in the container, hand back `{"exit_code", "stdout", "stderr"}`, repeat, for at most 12 model turns. It adds no prompt of its own. When the model stops to wait for the person, the person may answer, and the loop goes on.
26 +3. `src/scrub.ts` makes output repeatable before the model sees it: the fake's per-run URLs become `https://backend.composio.test` and `https://dashboard.composio.test`, temporary directories and wall-clock timestamps are replaced, long output keeps its head and tail.
27 +4. `src/person.ts` is the person the agent works for. See [The person](#the-person).
28 +5. Two adapters speak the wire formats through the official SDKs: `src/anthropic-adapter.ts` (Messages) and `src/openai-adapter.ts` (Chat Completions). Both send a placeholder key to the local aimock.
29 +
30 +aimock runs in strict mode with exactly one fixture file loaded: one file per journey, provider, and model. A request the file does not answer gets a 503 and fails the test, and the suite also asserts that every fixture in the file was served exactly once.
31 +
32 +## The person
33 +
34 +A login or an account authorization needs the account's owner in a browser. No agent can approve it for them, so a journey that needs one has to model what reaches the person. If the model approved everything, an agent that never showed anyone the URL would pass.
35 +
36 +Two things reach this person:
37 +
38 +- a page the CLI opens in their browser (the container has an `xdg-open` that loads the page, and on the fake backend loading an authorization page approves it);
39 +- what the agent writes to them.
40 +
41 +They do not read the output of the agent's commands. The CLI assumes the same: off a terminal `composio login` prints "For agents: Show the URL above to the user".
42 +
43 +They approve an authorization when its page reaches them: the CLI opened it, or the agent wrote its complete URL (the fake's origin, the path, the id). An id on its own, or the same path on another host, is not a page anyone can open, and approves nothing. Markdown around the URL is fine; a URL written without `https://` is not recognised, which is a limit of this model.
44 +
45 +When the agent has stopped to wait, they answer `Done, I approved it in my browser. Please continue.`, and the agent gets another turn. They say it once per approval, and only while no command of the agent has been told of it: after `composio login --poll` returned the linked session there is nothing to wait for, and the person stays quiet. "Told" means the fake answered a request with the approved state. Whether that answer reached the agent (a response lost on the way, output sent to `/dev/null`) is not modelled. So handing the person the URL and stopping is a correct move, not a failure: the journey goes on, and its checks are the same as for an agent that waited.
46 +
47 +`person.test.ts` pins these rules without a container.
48 +
49 +What follows from this model:
50 +
51 +- `composio link <toolkit>` opens the browser and waits: approved while it waits.
52 +- `composio link <toolkit> --no-wait` prints the URL and returns: approved once the agent shows the URL to the person.
53 +- `composio link <toolkit> --no-browser` waits without opening anything, and off a terminal prints nothing until the account is active. Nothing reaches the person, so it times out. The real CLI behaves the same; `../operations` pins it as an expected failure.
54 +
55 +Not modelled: a machine with no browser, a person who reads the agent's command output, a person who declines or is slow, and a person who approves the wrong account.
56 +
57 +## Journeys
58 +
59 +| Id | Task | Starts from |
60 +| ------------------------------------ | ---------------------------------------------- | --------------------------------------------------- |
61 +| `consumer-slack-message` | Send a Slack message | Logged in, Slack not connected |
62 +| `developer-auth-configs-and-execute` | List auth configs, run a tool as the test user | Logged in, developer directory, test user connected |
63 +| `developer-execute-unconnected` | Run a tool as the test user | Logged in, developer directory, nothing connected |
64 +| `first-run-login` | Find out whether the CLI is logged in; log in | Empty home |
65 +| `run-inline-script` | Send a message from one `composio run` script | Logged in, Slack connected |
66 +
67 +`developer-execute-unconnected` is the tree-confusion case. Its hand-authored fixture scripts a model that first uses the root tree's word on the dev tree (`composio dev execute`), recovers from the parse error, then follows the CLI's not-connected hint, which names the dev-tree link. Its `scriptedChecks` pin how the CLI answered each of those commands.
68 +
69 +## How a run is graded
70 +
71 +A journey has two lists, and they answer different questions.
72 +
73 +**Checks** decide whether the run reached the goal. They grade what the run changed, not what it tried:
74 +
75 +- `SLACK_SEND_MESSAGE` completed once in the whole run, on any project and through any of the three execute routes, and it was the message asked for: the right project and user, `#general` (or `general`), and the text verbatim. A second send, a send on the other project, another channel, and other words each fail a check of their own, so a text mismatch is not mistaken for tree confusion. A developer execution must also name its project in the request; a session that happened to resolve to the right one is not enough. A lookup beside the send is let through; another tool that changes something is not.
76 +- No account became connected on a project the task does not involve. A person who approved `composio link slack` during a developer task gave access nobody asked for, and it stays.
77 +- `run-inline-script`: the send happened during a shell command that is one `composio run`, which exited 0 and printed the timestamp, not during a `composio execute` followed by a script that prints what it was told. Requests are attributed per shell command and the command is recognised by its text, so this holds for the ways a shell line names `composio`; an alias or a wrapper script would get past it.
78 +- `first-run-login`: a new `composio whoami` after the model stopped exits 0 as the person, in their org. The requests of the process that logged in do not show that anything was stored.
79 +- `developer-auth-configs-and-execute`: the auth configs were listed on the developer project. The CLI sends `dev auth-configs list` without a project scope today, and the fake answers an unscoped request with the developer project, so this check cannot tell which directory the agent ran it in. `operations/identity.test.ts` holds that defect as an expected failure; once it is fixed, the check should also require the scope headers.
80 +
81 +**Detours** are what a confused agent does on the way: a `composio dev` command during a consumer task, a root `composio execute` or `composio link` during a developer task, a request to the other project that changed nothing. A detour never fails a run. A run that tried the wrong tree, read the error, and finished on the right one reached the goal, and its detours say how it got there. Reading the two lists separately gives a success rate and a wrong-tree rate; `composio dev --help` during a consumer task counts toward the second and not against the first.
82 +
83 +The hand-authored path of each journey must take no detour: it is the path the skill teaches.
84 +
85 +`journeys.test.ts` runs the graders on made-up evidence, without a container: for each journey a run that reached the goal, and each way of looking as if it did.
86 +
87 +## Fixtures
88 +
89 +| Journey | OpenAI | Anthropic `claude-haiku-5-5` | Anthropic `claude-opus-5-5` |
90 +| ------------------------------------ | ------------- | ---------------------------- | --------------------------- |
91 +| `consumer-slack-message` | hand-authored | recorded | none |
92 +| `developer-auth-configs-and-execute` | hand-authored | none | recorded |
93 +| `developer-execute-unconnected` | hand-authored | none | recorded |
94 +| `first-run-login` | hand-authored | recorded | recorded |
95 +| `run-inline-script` | hand-authored | recorded | recorded |
96 +
97 +"none" means the model's latest recorded run did not reach the journey's goal. Such a run is not turned into a passing test; its transcript is in [`transcripts/`](#transcripts). A journey with no Anthropic fixture at all is reported by the suite as a `todo`.
98 +
99 +Recordings are optional; the hand-authored path is not. The suite fails when a journey has no hand-authored fixture, when a file in `fixtures/` is not named `<journey>.<provider>.<model>.json`, and when a fixture names a journey that does not exist, so a deleted or misnamed file cannot remove a journey's coverage unnoticed.
100 +
101 +How a fixture is matched:
102 +
103 +- Hand-authored: the first turn matches the prompt; each later turn matches the id of the previous `shell` call and a substring of what that command printed. If the CLI prints something else, the next turn has no match.
104 +- Recorded: aimock's recorder keys a turn on the latest message from the person, the model, and the turn's position. `record.ts` adds the id of the previous call and that command's exit code, so a CLI that now exits differently leaves the replay without a match. The output text of a recorded turn is not part of the match; the journey's checks cover it.
105 +
106 +## Recording
107 +
108 +One command records one journey with one model, once, with no retries:
109 +
110 +```bash
111 +cd ts/e2e-tests/cli/agent-journeys
112 +ANTHROPIC_API_KEY=… bun run record.ts --set <name> --journey consumer-slack-message --provider anthropic --model claude-haiku-5-5
113 +OPENAI_API_KEY=… bun run record.ts --set <name> --journey consumer-slack-message --provider openai --model <model>
114 +```
115 +
116 +`--set` names what the run measures (`skill-names-playground-execute`, say). The transcript goes to `transcripts/<set>/`, with the run's checks and its detours.
117 +
118 +aimock runs in record mode between the loop and the provider. The loop still sends the placeholder key; aimock swaps it for the key from the environment on the way out, so the key is in no request the loop builds. The script prints the commands with their exit codes and each check, and writes the transcript whether or not the run reached the goal.
119 +
120 +- A run that passes every check is written to `fixtures/<journey>.<provider>.<model>.json` and is picked up by the suite from its file name.
121 +- A run that fails a check writes no fixture, and leaves an older fixture of the same journey and model in place. Delete that fixture if the suite should stop replaying the older run. Read the transcript: a model that took the wrong tree is a finding about the skill or the CLI, not a recording to redo until it passes.
122 +- Before anything is written the script refuses output containing the key, a provider key prefix, or a recorded credential header.
123 +
124 +A recording costs a few thousand input tokens per turn (the skill is resent each turn) and at most 2,048 output tokens per turn.
125 +
126 +A recording made with an OpenAI model is added beside the hand-authored fixture, which stays: it is the baseline every journey must have.
127 +
128 +## Transcripts
129 +
130 +`transcripts/<set>/<journey>.<provider>.<model>.json` is one run as it happened, kept whether it passed or not: the prompt, every command with its exit code and output, the model's messages, the person's replies, each check, and what was measured (`revision`, `cliPackageModified`, `skillSha256`, `recordedOn`). Nothing replays them; they are the record of how models behaved.
131 +
132 +A transcript keeps the checks as they were when it was recorded. Sets 01 to 03 predate the stricter send, connection, and stored-login checks described under [How a run is graded](#how-a-run-is-graded) and have no `detours`; their commands and output are complete, so they can be read against today's rules by hand, but not re-graded, because a transcript does not hold the fake's journal.
133 +
134 +| Set | CLI and skill | Person | Runs |
135 +| -------------------------- | ------------------------------------------------ | --------------------------------------------------------- | ----------------------------------------------- |
136 +| `01-approve-everything` | `82e851aad` | Approved every login and authorization after each command | All five journeys, both models |
137 +| `02-approve-what-is-shown` | `82e851aad` | As described above | The three runs the stricter person could change |
138 +| `03-dev-link-hint` | `82e851aad` plus the dev-tree not-connected hint | As described above | `developer-execute-unconnected`, both models |
139 +
140 +Reached the goal, per set (`-` means not run in that set):
141 +
142 +| Journey | Model | 01 | 02 | 03 |
143 +| ------------------------------------ | ------------------ | --- | --- | --- |
144 +| `consumer-slack-message` | `claude-haiku-5-5` | yes | - | - |
145 +| `consumer-slack-message` | `claude-opus-5-5` | yes | no | - |
146 +| `developer-auth-configs-and-execute` | `claude-haiku-5-5` | no | - | - |
147 +| `developer-auth-configs-and-execute` | `claude-opus-5-5` | yes | - | - |
148 +| `developer-execute-unconnected` | `claude-haiku-5-5` | no | no | no |
149 +| `developer-execute-unconnected` | `claude-opus-5-5` | no | yes | yes |
150 +| `first-run-login` | `claude-haiku-5-5` | yes | - | - |
151 +| `first-run-login` | `claude-opus-5-5` | yes | - | - |
152 +| `run-inline-script` | `claude-haiku-5-5` | yes | - | - |
153 +| `run-inline-script` | `claude-opus-5-5` | yes | - | - |
154 +
155 +How to read it:
156 +
157 +- Every cell is one run. `claude-haiku-5-5` took the root tree, the dev tree, and the root tree again on the same developer prompt across the three sets, so one cell says little about a rate.
158 +- Set 01's `yes` for `claude-opus-5-5` on `consumer-slack-message` was the lenient person's doing: it ran `composio link slack --no-browser`, which reaches nobody. Set 02 is the honest result.
159 +- Set 01's `no` for `claude-opus-5-5` on `developer-execute-unconnected` was a handoff nobody answered: it gave the person the URL and stopped. With a person who answers, it finishes (02, 03).
160 +- In sets 02 and 03, a run that reached the dev tree and then failed did so at the not-connected error; a run that failed on the root tree did so at `composio link slack --no-browser`.
161 +
162 +## The skill's commands
163 +
164 +`skill-commands.test.ts` needs no model: it takes every `composio …` line from the code blocks of the built skill and its references and looks its command path up in the binary.
ts/e2e-tests/cli/agent-journeys/e2e.test.ts new
+117
@@ -0,0 +1,117 @@
1 +/**
2 + * Agent journeys, replayed.
3 + *
4 + * Each fixture file is one journey as one model played it. aimock serves that
5 + * file in strict mode, so a request the file does not answer fails the test;
6 + * the journey's checks then look at what the commands did, because a replay
7 + * alone passes even when every command failed.
8 + */
9 +
10 +import { afterAll, beforeAll, describe, expect, it } from 'bun:test';
11 +import { LLMock } from '@copilotkit/aimock';
12 +import { join } from 'node:path';
13 +import { adapterFor } from './src/adapters';
14 +import { HAND_AUTHORED, listFixtures } from './src/fixtures';
15 +import { JOURNEYS, type JourneyRun } from './src/journeys';
16 +import { runJourney } from './src/run-journey';
17 +
18 +const JOURNEY_TIMEOUT_MS = 1_200_000;
19 +
20 +// A fixture's `turnIndex` must hold exactly; aimock otherwise treats it as a hint.
21 +process.env.AIMOCK_STRICT_TURN_INDEX = '1';
22 +
23 +const fixtures = listFixtures();
24 +
25 +describe('the fixture set', () => {
26 + it('has no fixture for a journey that does not exist', () => {
27 + const known = new Set(JOURNEYS.map(journey => journey.id));
28 + expect(fixtures.filter(fixture => !known.has(fixture.journeyId))).toEqual([]);
29 + });
30 +
31 + // The baseline: it needs no key to produce, and it is the only thing that
32 + // exercises the OpenAI wire format, so no journey may be without it.
33 + it('has the hand-authored OpenAI path of every journey', () => {
34 + const scripted = new Set(
35 + fixtures
36 + .filter(fixture => fixture.provider === 'openai' && fixture.model === HAND_AUTHORED)
37 + .map(fixture => fixture.journeyId)
38 + );
39 + expect(JOURNEYS.map(journey => journey.id).filter(id => !scripted.has(id))).toEqual([]);
40 + });
41 +});
42 +
43 +for (const journey of JOURNEYS) {
44 + describe(`journey: ${journey.title}`, () => {
45 + const played = fixtures.filter(fixture => fixture.journeyId === journey.id);
46 +
47 + if (!played.some(fixture => fixture.provider === 'anthropic')) {
48 + // A recording is kept only when the run reached the goal; see the README.
49 + it.todo(`anthropic: no recorded run of ${journey.id} reached its goal`);
50 + }
51 +
52 + for (const fixture of played) {
53 + const origin = fixture.model === HAND_AUTHORED ? 'hand-authored' : 'recorded';
54 +
55 + describe(`${fixture.provider} ${fixture.model} (${origin})`, () => {
56 + let mock: LLMock;
57 + let run: JourneyRun;
58 +
59 + beforeAll(async () => {
60 + mock = new LLMock({ port: 0, strict: true, logLevel: 'silent' });
61 + mock.loadFixtureFile(fixture.path);
62 + const baseURL = await mock.start();
63 + run = await runJourney(journey, adapterFor(fixture.provider, fixture.model, baseURL));
64 + }, JOURNEY_TIMEOUT_MS);
65 +
66 + afterAll(async () => {
67 + // The container and the servers go away even when the log cannot be written.
68 + try {
69 + await run?.rig.journey.writeDebugLog(
70 + join(import.meta.dir, `DEBUG.${journey.id}.${fixture.provider}.${fixture.model}.log`)
71 + );
72 + } finally {
73 + await run?.rig.stop();
74 + await mock?.stop();
75 + }
76 + });
77 +
78 + it('replays every recorded turn, and nothing else', () => {
79 + const requests = mock.getRequests();
80 + expect(requests.map(request => request.response.status)).toEqual(requests.map(() => 200));
81 + // Each fixture answered exactly one request: none skipped, none reused.
82 + const served = requests.map(request => request.response.fixture);
83 + expect(new Set(served).size).toBe(served.length);
84 + expect(served).toHaveLength(mock.getFixtures().length);
85 + });
86 +
87 + it('ran at least one command, and every command reached the container', () => {
88 + expect(run.transcript.commands.length).toBeGreaterThan(0);
89 + expect(run.rig.journey.steps().map(step => step.command)).toEqual(
90 + expect.arrayContaining(run.transcript.commands.map(command => command.command))
91 + );
92 + });
93 +
94 + it('reached the goal', () => {
95 + const failed = journey.checks(run).filter(check => !check.ok);
96 + expect(failed).toEqual([]);
97 + });
98 +
99 + if (origin === 'hand-authored') {
100 + // The scripted path is the one the skill teaches; it never leaves its tree.
101 + it('took no detour', () => {
102 + const taken = (journey.detours?.(run) ?? []).filter(check => !check.ok);
103 + expect(taken).toEqual([]);
104 + });
105 + }
106 +
107 + if (origin === 'hand-authored' && journey.scriptedChecks) {
108 + const { scriptedChecks } = journey;
109 + it('the CLI answered each scripted command as described', () => {
110 + const failed = scriptedChecks(run).filter(check => !check.ok);
111 + expect(failed).toEqual([]);
112 + });
113 + }
114 + });
115 + }
116 + });
117 +}
ts/e2e-tests/cli/agent-journeys/fixtures/consumer-slack-message.anthropic.claude-haiku-5-5.json new
+163
@@ -0,0 +1,163 @@
1 +{
2 + "$comment": "Recorded from anthropic claude-haiku-5-5 by record.ts. Do not edit by hand; record again.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "turnIndex": 0,
7 + "userMessage": "Send a Slack message to the #general channel that says: deploy finished",
8 + "model": "claude-haiku-5-5",
9 + "hasToolResult": false
10 + },
11 + "response": {
12 + "toolCalls": [
13 + {
14 + "id": "toolu_01DT8ERFKDiGexrcVsgcHwr1",
15 + "name": "shell",
16 + "arguments": "{\"command\":\"composio execute SLACK_SEND_MESSAGE --get-schema\"}"
17 + }
18 + ],
19 + "usage": {
20 + "input_tokens": 2652,
21 + "cache_creation_input_tokens": 0,
22 + "cache_read_input_tokens": 0,
23 + "cache_creation": {
24 + "ephemeral_5m_input_tokens": 0,
25 + "ephemeral_1h_input_tokens": 0
26 + },
27 + "output_tokens": 168
28 + }
29 + },
30 + "metadata": {
31 + "systemHash": "6ee9d189",
32 + "toolsHash": "95a7fdfe"
33 + }
34 + },
35 + {
36 + "match": {
37 + "turnIndex": 1,
38 + "userMessage": "Send a Slack message to the #general channel that says: deploy finished",
39 + "model": "claude-haiku-5-5",
40 + "hasToolResult": true,
41 + "toolCallId": "toolu_01DT8ERFKDiGexrcVsgcHwr1",
42 + "toolResultContains": "\"exit_code\":0,"
43 + },
44 + "response": {
45 + "toolCalls": [
46 + {
47 + "id": "toolu_01GZCNChP9qHLoARjymEeCZs",
48 + "name": "shell",
49 + "arguments": "{\"command\":\"composio execute SLACK_SEND_MESSAGE -d '{ channel: \\\"#general\\\", text: \\\"deploy finished\\\" }'\"}"
50 + }
51 + ],
52 + "usage": {
53 + "input_tokens": 3114,
54 + "cache_creation_input_tokens": 0,
55 + "cache_read_input_tokens": 0,
56 + "cache_creation": {
57 + "ephemeral_5m_input_tokens": 0,
58 + "ephemeral_1h_input_tokens": 0
59 + },
60 + "output_tokens": 87
61 + }
62 + },
63 + "metadata": {
64 + "systemHash": "6ee9d189",
65 + "toolsHash": "95a7fdfe"
66 + }
67 + },
68 + {
69 + "match": {
70 + "turnIndex": 2,
71 + "userMessage": "Send a Slack message to the #general channel that says: deploy finished",
72 + "model": "claude-haiku-5-5",
73 + "hasToolResult": true,
74 + "toolCallId": "toolu_01GZCNChP9qHLoARjymEeCZs",
75 + "toolResultContains": "\"exit_code\":1,"
76 + },
77 + "response": {
78 + "toolCalls": [
79 + {
80 + "id": "toolu_0127uqjnBQzn8fmy7kHupVeN",
81 + "name": "shell",
82 + "arguments": "{\"command\":\"composio link slack\"}"
83 + }
84 + ],
85 + "usage": {
86 + "input_tokens": 3321,
87 + "cache_creation_input_tokens": 0,
88 + "cache_read_input_tokens": 0,
89 + "cache_creation": {
90 + "ephemeral_5m_input_tokens": 0,
91 + "ephemeral_1h_input_tokens": 0
92 + },
93 + "output_tokens": 72
94 + },
95 + "content": "Slack isn't connected yet, so I'll start the link flow."
96 + },
97 + "metadata": {
98 + "systemHash": "6ee9d189",
99 + "toolsHash": "95a7fdfe"
100 + }
101 + },
102 + {
103 + "match": {
104 + "turnIndex": 3,
105 + "userMessage": "Send a Slack message to the #general channel that says: deploy finished",
106 + "model": "claude-haiku-5-5",
107 + "hasToolResult": true,
108 + "toolCallId": "toolu_0127uqjnBQzn8fmy7kHupVeN",
109 + "toolResultContains": "\"exit_code\":0,"
110 + },
111 + "response": {
112 + "toolCalls": [
113 + {
114 + "id": "toolu_01DRoezXpTnynHs7KEncA9qL",
115 + "name": "shell",
116 + "arguments": "{\"command\":\"composio execute SLACK_SEND_MESSAGE -d '{ channel: \\\"#general\\\", text: \\\"deploy finished\\\" }'\"}"
117 + }
118 + ],
119 + "usage": {
120 + "input_tokens": 3571,
121 + "cache_creation_input_tokens": 0,
122 + "cache_read_input_tokens": 0,
123 + "cache_creation": {
124 + "ephemeral_5m_input_tokens": 0,
125 + "ephemeral_1h_input_tokens": 0
126 + },
127 + "output_tokens": 87
128 + }
129 + },
130 + "metadata": {
131 + "systemHash": "6ee9d189",
132 + "toolsHash": "95a7fdfe"
133 + }
134 + },
135 + {
136 + "match": {
137 + "turnIndex": 4,
138 + "userMessage": "Send a Slack message to the #general channel that says: deploy finished",
139 + "model": "claude-haiku-5-5",
140 + "hasToolResult": true,
141 + "toolCallId": "toolu_01DRoezXpTnynHs7KEncA9qL",
142 + "toolResultContains": "\"exit_code\":0,"
143 + },
144 + "response": {
145 + "usage": {
146 + "input_tokens": 3821,
147 + "cache_creation_input_tokens": 0,
148 + "cache_read_input_tokens": 0,
149 + "cache_creation": {
150 + "ephemeral_5m_input_tokens": 0,
151 + "ephemeral_1h_input_tokens": 0
152 + },
153 + "output_tokens": 84
154 + },
155 + "content": "The message \"deploy finished\" was posted to #general.\n\nSlack wasn't connected at first, so I ran `composio link slack` to authorize it, then retried the send. The link completed on the first try, and the send succeeded (log ID `log_e2e0001`)."
156 + },
157 + "metadata": {
158 + "systemHash": "6ee9d189",
159 + "toolsHash": "95a7fdfe"
160 + }
161 + }
162 + ]
163 +}
ts/e2e-tests/cli/agent-journeys/fixtures/consumer-slack-message.openai.hand-authored.json new
+83
@@ -0,0 +1,83 @@
1 +{
2 + "$comment": "Hand-authored: no model produced this file. It scripts the path the composio-cli skill teaches, one fixture per model turn. After the first turn each fixture is gated on the id of the previous shell call and on a substring of what that command printed, so a CLI that answers differently leaves the next turn without a match and the strict replay fails.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "userMessage": "Send a Slack message to the #general channel",
7 + "hasToolResult": false,
8 + "turnIndex": 0
9 + },
10 + "response": {
11 + "toolCalls": [
12 + {
13 + "id": "call_search",
14 + "name": "shell",
15 + "arguments": {
16 + "command": "composio search \"send a slack message\""
17 + }
18 + }
19 + ]
20 + }
21 + },
22 + {
23 + "match": {
24 + "toolCallId": "call_search",
25 + "toolResultContains": "SLACK_SEND_MESSAGE"
26 + },
27 + "response": {
28 + "toolCalls": [
29 + {
30 + "id": "call_execute_1",
31 + "name": "shell",
32 + "arguments": {
33 + "command": "composio execute SLACK_SEND_MESSAGE -d '{\"channel\":\"general\",\"text\":\"deploy finished\"}'"
34 + }
35 + }
36 + ]
37 + }
38 + },
39 + {
40 + "match": {
41 + "toolCallId": "call_execute_1",
42 + "toolResultContains": "Run `composio link slack`, then retry."
43 + },
44 + "response": {
45 + "toolCalls": [
46 + {
47 + "id": "call_link",
48 + "name": "shell",
49 + "arguments": {
50 + "command": "composio link slack"
51 + }
52 + }
53 + ]
54 + }
55 + },
56 + {
57 + "match": {
58 + "toolCallId": "call_link",
59 + "toolResultContains": "is now ACTIVE (toolkit: slack)"
60 + },
61 + "response": {
62 + "toolCalls": [
63 + {
64 + "id": "call_execute_2",
65 + "name": "shell",
66 + "arguments": {
67 + "command": "composio execute SLACK_SEND_MESSAGE -d '{\"channel\":\"general\",\"text\":\"deploy finished\"}'"
68 + }
69 + }
70 + ]
71 + }
72 + },
73 + {
74 + "match": {
75 + "toolCallId": "call_execute_2",
76 + "toolResultContains": "log_e2e0001"
77 + },
78 + "response": {
79 + "content": "Sent \"deploy finished\" to #general."
80 + }
81 + }
82 + ]
83 +}
ts/e2e-tests/cli/agent-journeys/fixtures/developer-auth-configs-and-execute.anthropic.claude-opus-5-5.json new
+162
@@ -0,0 +1,162 @@
1 +{
2 + "$comment": "Recorded from anthropic claude-opus-5-5 by record.ts. Do not edit by hand; record again.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "turnIndex": 0,
7 + "userMessage": "I am building an app on Composio. In my developer project, list the auth configs, then execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
8 + "model": "claude-opus-5-5",
9 + "hasToolResult": false
10 + },
11 + "response": {
12 + "toolCalls": [
13 + {
14 + "id": "toolu_01EeMhtrZbPm6vQsK7eQygXq",
15 + "name": "shell",
16 + "arguments": "{\"command\":\"cat /skills/composio-cli/references/composio-dev.md\"}"
17 + }
18 + ],
19 + "usage": {
20 + "input_tokens": 2689,
21 + "cache_creation_input_tokens": 0,
22 + "cache_read_input_tokens": 0,
23 + "cache_creation": {
24 + "ephemeral_5m_input_tokens": 0,
25 + "ephemeral_1h_input_tokens": 0
26 + },
27 + "output_tokens": 94
28 + }
29 + },
30 + "metadata": {
31 + "systemHash": "6ee9d189",
32 + "toolsHash": "95a7fdfe"
33 + }
34 + },
35 + {
36 + "match": {
37 + "turnIndex": 1,
38 + "userMessage": "I am building an app on Composio. In my developer project, list the auth configs, then execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
39 + "model": "claude-opus-5-5",
40 + "hasToolResult": true,
41 + "toolCallId": "toolu_01EeMhtrZbPm6vQsK7eQygXq",
42 + "toolResultContains": "\"exit_code\":0,"
43 + },
44 + "response": {
45 + "toolCalls": [
46 + {
47 + "id": "toolu_01CziAJ2Rd7jxguoAjDdhAb8",
48 + "name": "shell",
49 + "arguments": "{\"command\":\"composio dev auth-configs list; echo ---; composio dev connected-accounts list; echo ---; composio dev --help\"}"
50 + }
51 + ],
52 + "usage": {
53 + "input_tokens": 3591,
54 + "cache_creation_input_tokens": 0,
55 + "cache_read_input_tokens": 0,
56 + "cache_creation": {
57 + "ephemeral_5m_input_tokens": 0,
58 + "ephemeral_1h_input_tokens": 0
59 + },
60 + "output_tokens": 84
61 + }
62 + },
63 + "metadata": {
64 + "systemHash": "6ee9d189",
65 + "toolsHash": "95a7fdfe"
66 + }
67 + },
68 + {
69 + "match": {
70 + "turnIndex": 2,
71 + "userMessage": "I am building an app on Composio. In my developer project, list the auth configs, then execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
72 + "model": "claude-opus-5-5",
73 + "hasToolResult": true,
74 + "toolCallId": "toolu_01CziAJ2Rd7jxguoAjDdhAb8",
75 + "toolResultContains": "\"exit_code\":0,"
76 + },
77 + "response": {
78 + "toolCalls": [
79 + {
80 + "id": "toolu_01YD8N3UU1CDVH3xSXpGNTAL",
81 + "name": "shell",
82 + "arguments": "{\"command\":\"composio dev playground-execute --help; composio dev playground-execute SLACK_SEND_MESSAGE --get-schema\"}"
83 + }
84 + ],
85 + "usage": {
86 + "input_tokens": 4935,
87 + "cache_creation_input_tokens": 0,
88 + "cache_read_input_tokens": 0,
89 + "cache_creation": {
90 + "ephemeral_5m_input_tokens": 0,
91 + "ephemeral_1h_input_tokens": 0
92 + },
93 + "output_tokens": 86
94 + }
95 + },
96 + "metadata": {
97 + "systemHash": "6ee9d189",
98 + "toolsHash": "95a7fdfe"
99 + }
100 + },
101 + {
102 + "match": {
103 + "turnIndex": 3,
104 + "userMessage": "I am building an app on Composio. In my developer project, list the auth configs, then execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
105 + "model": "claude-opus-5-5",
106 + "hasToolResult": true,
107 + "toolCallId": "toolu_01YD8N3UU1CDVH3xSXpGNTAL",
108 + "toolResultContains": "\"exit_code\":0,"
109 + },
110 + "response": {
111 + "toolCalls": [
112 + {
113 + "id": "toolu_013GTDHe6gGd2mGJ56sgpun9",
114 + "name": "shell",
115 + "arguments": "{\"command\":\"composio dev playground-execute SLACK_SEND_MESSAGE -d '{ \\\"channel\\\": \\\"general\\\", \\\"text\\\": \\\"hello from dev\\\" }'\"}"
116 + }
117 + ],
118 + "usage": {
119 + "input_tokens": 6451,
120 + "cache_creation_input_tokens": 0,
121 + "cache_read_input_tokens": 0,
122 + "cache_creation": {
123 + "ephemeral_5m_input_tokens": 0,
124 + "ephemeral_1h_input_tokens": 0
125 + },
126 + "output_tokens": 93
127 + }
128 + },
129 + "metadata": {
130 + "systemHash": "6ee9d189",
131 + "toolsHash": "95a7fdfe"
132 + }
133 + },
134 + {
135 + "match": {
136 + "turnIndex": 4,
137 + "userMessage": "I am building an app on Composio. In my developer project, list the auth configs, then execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
138 + "model": "claude-opus-5-5",
139 + "hasToolResult": true,
140 + "toolCallId": "toolu_013GTDHe6gGd2mGJ56sgpun9",
141 + "toolResultContains": "\"exit_code\":0,"
142 + },
143 + "response": {
144 + "usage": {
145 + "input_tokens": 6706,
146 + "cache_creation_input_tokens": 0,
147 + "cache_read_input_tokens": 0,
148 + "cache_creation": {
149 + "ephemeral_5m_input_tokens": 0,
150 + "ephemeral_1h_input_tokens": 0
151 + },
152 + "output_tokens": 576
153 + },
154 + "content": "I listed your project's auth configs and posted \"hello from dev\" to #general as your test user.\n\n**Auth configs** (`composio dev auth-configs list`): all three are Composio-managed, use OAuth2 and are enabled.\n\n| ID | Name | Toolkit | Connections |\n|---|---|---|---|\n| `ac_e2e0001` | slack-developer | slack | 1 |\n| `ac_e2e0002` | github-developer | github | 0 |\n| `ac_e2e0003` | gmail-developer | gmail | 0 |\n\n**Sending the message:** Your project has one connected account, an active Slack account (`ca_e2e0001`, word_id `slack_e2e_0001`) for test user `pg-test-00000000-0000-4000-8000-00000000a001`, using `slack-developer`. I checked the tool's inputs (`channel` and `text`), then ran:\n\n```bash\ncomposio dev playground-execute SLACK_SEND_MESSAGE -d '{ \"channel\": \"general\", \"text\": \"hello from dev\" }'\n```\n\nSlack returned `ok: true`. The message timestamp is `1767225600.000100` and the log ID is `log_e2e0001`.\n\nI didn't pass `--user-id`, so the command used the default test user. The output didn't say which user that was, but the project's only Slack connection belongs to the test user above, so that's the one that posted. To pick the user explicitly, add `--user-id <id>`. You can see the run with `composio dev logs tools`.\n\nGitHub and Gmail have no connected accounts yet. Before you test those tools, connect an account with `composio dev connected-accounts link`."
155 + },
156 + "metadata": {
157 + "systemHash": "6ee9d189",
158 + "toolsHash": "95a7fdfe"
159 + }
160 + }
161 + ]
162 +}
ts/e2e-tests/cli/agent-journeys/fixtures/developer-auth-configs-and-execute.openai.hand-authored.json new
+83
@@ -0,0 +1,83 @@
1 +{
2 + "$comment": "Hand-authored: no model produced this file. It scripts the path the composio-cli skill teaches, one fixture per model turn. After the first turn each fixture is gated on the id of the previous shell call and on a substring of what that command printed, so a CLI that answers differently leaves the next turn without a match and the strict replay fails.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "userMessage": "In my developer project, list the auth configs",
7 + "hasToolResult": false,
8 + "turnIndex": 0
9 + },
10 + "response": {
11 + "toolCalls": [
12 + {
13 + "id": "call_reference",
14 + "name": "shell",
15 + "arguments": {
16 + "command": "cat /skills/composio-cli/references/composio-dev.md"
17 + }
18 + }
19 + ]
20 + }
21 + },
22 + {
23 + "match": {
24 + "toolCallId": "call_reference",
25 + "toolResultContains": "composio dev auth-configs list"
26 + },
27 + "response": {
28 + "toolCalls": [
29 + {
30 + "id": "call_auth_configs",
31 + "name": "shell",
32 + "arguments": {
33 + "command": "composio dev auth-configs list"
34 + }
35 + }
36 + ]
37 + }
38 + },
39 + {
40 + "match": {
41 + "toolCallId": "call_auth_configs",
42 + "toolResultContains": "ac_e2e0001"
43 + },
44 + "response": {
45 + "toolCalls": [
46 + {
47 + "id": "call_dev_help",
48 + "name": "shell",
49 + "arguments": {
50 + "command": "composio dev --help"
51 + }
52 + }
53 + ]
54 + }
55 + },
56 + {
57 + "match": {
58 + "toolCallId": "call_dev_help",
59 + "toolResultContains": "playground-execute"
60 + },
61 + "response": {
62 + "toolCalls": [
63 + {
64 + "id": "call_playground",
65 + "name": "shell",
66 + "arguments": {
67 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE -d '{\"channel\":\"general\",\"text\":\"hello from dev\"}'"
68 + }
69 + }
70 + ]
71 + }
72 + },
73 + {
74 + "match": {
75 + "toolCallId": "call_playground",
76 + "toolResultContains": "log_e2e0001"
77 + },
78 + "response": {
79 + "content": "Listed the auth configs and posted \"hello from dev\" to #general as the test user."
80 + }
81 + }
82 + ]
83 +}
ts/e2e-tests/cli/agent-journeys/fixtures/developer-execute-unconnected.anthropic.claude-opus-5-5.json new
+386
@@ -0,0 +1,386 @@
1 +{
2 + "$comment": "Recorded from anthropic claude-opus-5-5 by record.ts. Do not edit by hand; record again.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "turnIndex": 0,
7 + "userMessage": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
8 + "model": "claude-opus-5-5",
9 + "hasToolResult": false
10 + },
11 + "response": {
12 + "toolCalls": [
13 + {
14 + "id": "toolu_01KsMZbam8hfHbvz6Dpy4gG2",
15 + "name": "shell",
16 + "arguments": "{\"command\":\"cat /skills/composio-cli/references/composio-dev.md\"}"
17 + }
18 + ],
19 + "usage": {
20 + "input_tokens": 2674,
21 + "cache_creation_input_tokens": 0,
22 + "cache_read_input_tokens": 0,
23 + "cache_creation": {
24 + "ephemeral_5m_input_tokens": 0,
25 + "ephemeral_1h_input_tokens": 0
26 + },
27 + "output_tokens": 95
28 + }
29 + },
30 + "metadata": {
31 + "systemHash": "6ee9d189",
32 + "toolsHash": "95a7fdfe"
33 + }
34 + },
35 + {
36 + "match": {
37 + "turnIndex": 1,
38 + "userMessage": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
39 + "model": "claude-opus-5-5",
40 + "hasToolResult": true,
41 + "toolCallId": "toolu_01KsMZbam8hfHbvz6Dpy4gG2",
42 + "toolResultContains": "\"exit_code\":0,"
43 + },
44 + "response": {
45 + "toolCalls": [
46 + {
47 + "id": "toolu_017s4oFeENE22UfrhxsRXHUz",
48 + "name": "shell",
49 + "arguments": "{\"command\":\"composio whoami; composio dev --help; composio dev connected-accounts list --toolkits slack 2>&1 || composio dev connected-accounts list\"}"
50 + }
51 + ],
52 + "usage": {
53 + "input_tokens": 3575,
54 + "cache_creation_input_tokens": 0,
55 + "cache_read_input_tokens": 0,
56 + "cache_creation": {
57 + "ephemeral_5m_input_tokens": 0,
58 + "ephemeral_1h_input_tokens": 0
59 + },
60 + "output_tokens": 94
61 + }
62 + },
63 + "metadata": {
64 + "systemHash": "6ee9d189",
65 + "toolsHash": "95a7fdfe"
66 + }
67 + },
68 + {
69 + "match": {
70 + "turnIndex": 2,
71 + "userMessage": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
72 + "model": "claude-opus-5-5",
73 + "hasToolResult": true,
74 + "toolCallId": "toolu_017s4oFeENE22UfrhxsRXHUz",
75 + "toolResultContains": "\"exit_code\":0,"
76 + },
77 + "response": {
78 + "toolCalls": [
79 + {
80 + "id": "toolu_01FKqJVkvrp5qjUcEYV8EN8Y",
81 + "name": "shell",
82 + "arguments": "{\"command\":\"composio dev playground-execute --help; composio dev connected-accounts list --help\"}"
83 + }
84 + ],
85 + "usage": {
86 + "input_tokens": 4320,
87 + "cache_creation_input_tokens": 0,
88 + "cache_read_input_tokens": 0,
89 + "cache_creation": {
90 + "ephemeral_5m_input_tokens": 0,
91 + "ephemeral_1h_input_tokens": 0
92 + },
93 + "output_tokens": 125
94 + }
95 + },
96 + "metadata": {
97 + "systemHash": "6ee9d189",
98 + "toolsHash": "95a7fdfe"
99 + }
100 + },
101 + {
102 + "match": {
103 + "turnIndex": 3,
104 + "userMessage": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
105 + "model": "claude-opus-5-5",
106 + "hasToolResult": true,
107 + "toolCallId": "toolu_01FKqJVkvrp5qjUcEYV8EN8Y",
108 + "toolResultContains": "\"exit_code\":0,"
109 + },
110 + "response": {
111 + "toolCalls": [
112 + {
113 + "id": "toolu_01RapFsMY1ecVdXkAtBEv9br",
114 + "name": "shell",
115 + "arguments": "{\"command\":\"composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \\\"general\\\", markdown_text: \\\"hello from dev\\\" }'\"}"
116 + }
117 + ],
118 + "usage": {
119 + "input_tokens": 5984,
120 + "cache_creation_input_tokens": 0,
121 + "cache_read_input_tokens": 0,
122 + "cache_creation": {
123 + "ephemeral_5m_input_tokens": 0,
124 + "ephemeral_1h_input_tokens": 0
125 + },
126 + "output_tokens": 100
127 + }
128 + },
129 + "metadata": {
130 + "systemHash": "6ee9d189",
131 + "toolsHash": "95a7fdfe"
132 + }
133 + },
134 + {
135 + "match": {
136 + "turnIndex": 4,
137 + "userMessage": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
138 + "model": "claude-opus-5-5",
139 + "hasToolResult": true,
140 + "toolCallId": "toolu_01RapFsMY1ecVdXkAtBEv9br",
141 + "toolResultContains": "\"exit_code\":1,"
142 + },
143 + "response": {
144 + "toolCalls": [
145 + {
146 + "id": "toolu_011mLodqSvm92aRHufRf8nsK",
147 + "name": "shell",
148 + "arguments": "{\"command\":\"composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \\\"general\\\", text: \\\"hello from dev\\\" }'\"}"
149 + }
150 + ],
151 + "usage": {
152 + "input_tokens": 6384,
153 + "cache_creation_input_tokens": 0,
154 + "cache_read_input_tokens": 0,
155 + "cache_creation": {
156 + "ephemeral_5m_input_tokens": 0,
157 + "ephemeral_1h_input_tokens": 0
158 + },
159 + "output_tokens": 96
160 + }
161 + },
162 + "metadata": {
163 + "systemHash": "6ee9d189",
164 + "toolsHash": "95a7fdfe"
165 + }
166 + },
167 + {
168 + "match": {
169 + "turnIndex": 5,
170 + "userMessage": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
171 + "model": "claude-opus-5-5",
172 + "hasToolResult": true,
173 + "toolCallId": "toolu_011mLodqSvm92aRHufRf8nsK",
174 + "toolResultContains": "\"exit_code\":0,"
175 + },
176 + "response": {
177 + "toolCalls": [
178 + {
179 + "id": "toolu_01YHBEZaYAouaD9DLN6iHaJB",
180 + "name": "shell",
181 + "arguments": "{\"command\":\"composio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \\\"general\\\", text: \\\"hello from dev\\\" }'\"}"
182 + }
183 + ],
184 + "usage": {
185 + "input_tokens": 6719,
186 + "cache_creation_input_tokens": 0,
187 + "cache_read_input_tokens": 0,
188 + "cache_creation": {
189 + "ephemeral_5m_input_tokens": 0,
190 + "ephemeral_1h_input_tokens": 0
191 + },
192 + "output_tokens": 110
193 + },
194 + "content": "Dry run looks good, using the test user. Executing now."
195 + },
196 + "metadata": {
197 + "systemHash": "6ee9d189",
198 + "toolsHash": "95a7fdfe"
199 + }
200 + },
201 + {
202 + "match": {
203 + "turnIndex": 6,
204 + "userMessage": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
205 + "model": "claude-opus-5-5",
206 + "hasToolResult": true,
207 + "toolCallId": "toolu_01YHBEZaYAouaD9DLN6iHaJB",
208 + "toolResultContains": "\"exit_code\":1,"
209 + },
210 + "response": {
211 + "toolCalls": [
212 + {
213 + "id": "toolu_01GKyYWhhD8YQUBQxtHoCThL",
214 + "name": "shell",
215 + "arguments": "{\"command\":\"composio dev connected-accounts link --help\"}"
216 + }
217 + ],
218 + "usage": {
219 + "input_tokens": 6973,
220 + "cache_creation_input_tokens": 0,
221 + "cache_read_input_tokens": 0,
222 + "cache_creation": {
223 + "ephemeral_5m_input_tokens": 0,
224 + "ephemeral_1h_input_tokens": 0
225 + },
226 + "output_tokens": 97
227 + }
228 + },
229 + "metadata": {
230 + "systemHash": "6ee9d189",
231 + "toolsHash": "95a7fdfe"
232 + }
233 + },
234 + {
235 + "match": {
236 + "turnIndex": 7,
237 + "userMessage": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
238 + "model": "claude-opus-5-5",
239 + "hasToolResult": true,
240 + "toolCallId": "toolu_01GKyYWhhD8YQUBQxtHoCThL",
241 + "toolResultContains": "\"exit_code\":0,"
242 + },
243 + "response": {
244 + "toolCalls": [
245 + {
246 + "id": "toolu_01DMa86cCSAwEnAqCHuDr7xt",
247 + "name": "shell",
248 + "arguments": "{\"command\":\"composio dev connected-accounts link slack --list; composio dev connected-accounts link slack --no-browser --no-wait\"}"
249 + }
250 + ],
251 + "usage": {
252 + "input_tokens": 7656,
253 + "cache_creation_input_tokens": 0,
254 + "cache_read_input_tokens": 0,
255 + "cache_creation": {
256 + "ephemeral_5m_input_tokens": 0,
257 + "ephemeral_1h_input_tokens": 0
258 + },
259 + "output_tokens": 84
260 + }
261 + },
262 + "metadata": {
263 + "systemHash": "6ee9d189",
264 + "toolsHash": "95a7fdfe"
265 + }
266 + },
267 + {
268 + "match": {
269 + "turnIndex": 8,
270 + "userMessage": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
271 + "model": "claude-opus-5-5",
272 + "hasToolResult": true,
273 + "toolCallId": "toolu_01DMa86cCSAwEnAqCHuDr7xt",
274 + "toolResultContains": "\"exit_code\":0,"
275 + },
276 + "response": {
277 + "usage": {
278 + "input_tokens": 7958,
279 + "cache_creation_input_tokens": 0,
280 + "cache_read_input_tokens": 0,
281 + "cache_creation": {
282 + "ephemeral_5m_input_tokens": 0,
283 + "ephemeral_1h_input_tokens": 0
284 + },
285 + "output_tokens": 399
286 + },
287 + "content": "I couldn't post the message yet. Your test user doesn't have Slack connected in the developer project.\n\n**What I did:**\n1. Used `composio dev playground-execute`, which runs tools as your project's test user (`pg-test-00000000-0000-4000-8000-00000000a001`).\n2. Did a dry run first. The tool takes `text` for the message, not `markdown_text`, so I corrected that. The dry run then passed with `{ channel: \"general\", text: \"hello from dev\" }`.\n3. The real run failed with: `No active connection found for toolkit \"slack\"`. The test user has no connected Slack accounts.\n4. I started connecting Slack for the test user. It's waiting for you to approve access.\n\n**What you need to do:** open this link and approve Slack access:\n\n👉 https://backend.composio.test/api/v3/s/ca_e2e0001\n\n(The pending connection's ID is `ca_e2e0001`.)\n\nOnce you've approved it, tell me and I'll send the message, or run it yourself:\n\n```bash\ncomposio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }'\n```"
288 + },
289 + "metadata": {
290 + "systemHash": "6ee9d189",
291 + "toolsHash": "95a7fdfe"
292 + }
293 + },
294 + {
295 + "match": {
296 + "turnIndex": 9,
297 + "userMessage": "Done, I approved it in my browser. Please continue.",
298 + "model": "claude-opus-5-5",
299 + "hasToolResult": false
300 + },
301 + "response": {
302 + "toolCalls": [
303 + {
304 + "id": "toolu_017esCgNW5HY7rZYCFjzib28",
305 + "name": "shell",
306 + "arguments": "{\"command\":\"composio dev connected-accounts link slack --list\"}"
307 + }
308 + ],
309 + "usage": {
310 + "input_tokens": 8378,
311 + "cache_creation_input_tokens": 0,
312 + "cache_read_input_tokens": 0,
313 + "cache_creation": {
314 + "ephemeral_5m_input_tokens": 0,
315 + "ephemeral_1h_input_tokens": 0
316 + },
317 + "output_tokens": 60
318 + }
319 + },
320 + "metadata": {
321 + "systemHash": "6ee9d189",
322 + "toolsHash": "95a7fdfe"
323 + }
324 + },
325 + {
326 + "match": {
327 + "turnIndex": 10,
328 + "userMessage": "Done, I approved it in my browser. Please continue.",
329 + "model": "claude-opus-5-5",
330 + "hasToolResult": true,
331 + "toolCallId": "toolu_017esCgNW5HY7rZYCFjzib28",
332 + "toolResultContains": "\"exit_code\":0,"
333 + },
334 + "response": {
335 + "toolCalls": [
336 + {
337 + "id": "toolu_017DxkMfTbN8bk8tGotFZpfo",
338 + "name": "shell",
339 + "arguments": "{\"command\":\"composio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \\\"general\\\", text: \\\"hello from dev\\\" }'\"}"
340 + }
341 + ],
342 + "usage": {
343 + "input_tokens": 8856,
344 + "cache_creation_input_tokens": 0,
345 + "cache_read_input_tokens": 0,
346 + "cache_creation": {
347 + "ephemeral_5m_input_tokens": 0,
348 + "ephemeral_1h_input_tokens": 0
349 + },
350 + "output_tokens": 91
351 + }
352 + },
353 + "metadata": {
354 + "systemHash": "6ee9d189",
355 + "toolsHash": "95a7fdfe"
356 + }
357 + },
358 + {
359 + "match": {
360 + "turnIndex": 11,
361 + "userMessage": "Done, I approved it in my browser. Please continue.",
362 + "model": "claude-opus-5-5",
363 + "hasToolResult": true,
364 + "toolCallId": "toolu_017DxkMfTbN8bk8tGotFZpfo",
365 + "toolResultContains": "\"exit_code\":0,"
366 + },
367 + "response": {
368 + "usage": {
369 + "input_tokens": 9109,
370 + "cache_creation_input_tokens": 0,
371 + "cache_read_input_tokens": 0,
372 + "cache_creation": {
373 + "ephemeral_5m_input_tokens": 0,
374 + "ephemeral_1h_input_tokens": 0
375 + },
376 + "output_tokens": 230
377 + },
378 + "content": "\"hello from dev\" is now posted in #general as your test user.\n\n- **Slack connection:** `ca_e2e0001` is active for your test user (`pg-test-00000000-0000-4000-8000-00000000a001`).\n- **Message:** Slack accepted it (`ok: true`). Its timestamp is `1767225600.000100`.\n- **Log ID:** `log_e2e0001`. You can look it up with `composio dev logs tools`.\n\nTo send this again:\n\n```bash\ncomposio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }'\n```"
379 + },
380 + "metadata": {
381 + "systemHash": "6ee9d189",
382 + "toolsHash": "95a7fdfe"
383 + }
384 + }
385 + ]
386 +}
ts/e2e-tests/cli/agent-journeys/fixtures/developer-execute-unconnected.openai.hand-authored.json new
+83
@@ -0,0 +1,83 @@
1 +{
2 + "$comment": "Hand-authored: no model produced this file. It scripts the path the composio-cli skill teaches, one fixture per model turn. After the first turn each fixture is gated on the id of the previous shell call and on a substring of what that command printed, so a CLI that answers differently leaves the next turn without a match and the strict replay fails.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "userMessage": "In my Composio developer project, execute SLACK_SEND_MESSAGE",
7 + "hasToolResult": false,
8 + "turnIndex": 0
9 + },
10 + "response": {
11 + "toolCalls": [
12 + {
13 + "id": "call_wrong_tree",
14 + "name": "shell",
15 + "arguments": {
16 + "command": "composio dev execute SLACK_SEND_MESSAGE -d '{\"channel\":\"general\",\"text\":\"hello from dev\"}'"
17 + }
18 + }
19 + ]
20 + }
21 + },
22 + {
23 + "match": {
24 + "toolCallId": "call_wrong_tree",
25 + "toolResultContains": "Unknown subcommand"
26 + },
27 + "response": {
28 + "toolCalls": [
29 + {
30 + "id": "call_playground_1",
31 + "name": "shell",
32 + "arguments": {
33 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE -d '{\"channel\":\"general\",\"text\":\"hello from dev\"}'"
34 + }
35 + }
36 + ]
37 + }
38 + },
39 + {
40 + "match": {
41 + "toolCallId": "call_playground_1",
42 + "toolResultContains": "Run `composio dev connected-accounts link slack`"
43 + },
44 + "response": {
45 + "toolCalls": [
46 + {
47 + "id": "call_dev_link",
48 + "name": "shell",
49 + "arguments": {
50 + "command": "composio dev connected-accounts link slack"
51 + }
52 + }
53 + ]
54 + }
55 + },
56 + {
57 + "match": {
58 + "toolCallId": "call_dev_link",
59 + "toolResultContains": "is now ACTIVE (toolkit: slack)"
60 + },
61 + "response": {
62 + "toolCalls": [
63 + {
64 + "id": "call_playground_2",
65 + "name": "shell",
66 + "arguments": {
67 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE -d '{\"channel\":\"general\",\"text\":\"hello from dev\"}'"
68 + }
69 + }
70 + ]
71 + }
72 + },
73 + {
74 + "match": {
75 + "toolCallId": "call_playground_2",
76 + "toolResultContains": "log_e2e0001"
77 + },
78 + "response": {
79 + "content": "Posted \"hello from dev\" to #general as the test user."
80 + }
81 + }
82 + ]
83 +}
ts/e2e-tests/cli/agent-journeys/fixtures/first-run-login.anthropic.claude-haiku-5-5.json new
+164
@@ -0,0 +1,164 @@
1 +{
2 + "$comment": "Recorded from anthropic claude-haiku-5-5 by record.ts. Do not edit by hand; record again.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "turnIndex": 0,
7 + "userMessage": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
8 + "model": "claude-haiku-5-5",
9 + "hasToolResult": false
10 + },
11 + "response": {
12 + "toolCalls": [
13 + {
14 + "id": "toolu_014y1sFUoRBc1ebfonU8EqGS",
15 + "name": "shell",
16 + "arguments": "{\"command\":\"composio whoami\"}"
17 + }
18 + ],
19 + "usage": {
20 + "input_tokens": 2659,
21 + "cache_creation_input_tokens": 0,
22 + "cache_read_input_tokens": 0,
23 + "cache_creation": {
24 + "ephemeral_5m_input_tokens": 0,
25 + "ephemeral_1h_input_tokens": 0
26 + },
27 + "output_tokens": 50
28 + }
29 + },
30 + "metadata": {
31 + "systemHash": "6ee9d189",
32 + "toolsHash": "95a7fdfe"
33 + }
34 + },
35 + {
36 + "match": {
37 + "turnIndex": 1,
38 + "userMessage": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
39 + "model": "claude-haiku-5-5",
40 + "hasToolResult": true,
41 + "toolCallId": "toolu_014y1sFUoRBc1ebfonU8EqGS",
42 + "toolResultContains": "\"exit_code\":1,"
43 + },
44 + "response": {
45 + "toolCalls": [
46 + {
47 + "id": "toolu_01FyBnAtspFmeLAURYkLwXiu",
48 + "name": "shell",
49 + "arguments": "{\"command\":\"composio login\"}"
50 + }
51 + ],
52 + "usage": {
53 + "input_tokens": 2757,
54 + "cache_creation_input_tokens": 0,
55 + "cache_read_input_tokens": 0,
56 + "cache_creation": {
57 + "ephemeral_5m_input_tokens": 0,
58 + "ephemeral_1h_input_tokens": 0
59 + },
60 + "output_tokens": 60
61 + },
62 + "content": "Not logged in. Running login now."
63 + },
64 + "metadata": {
65 + "systemHash": "6ee9d189",
66 + "toolsHash": "95a7fdfe"
67 + }
68 + },
69 + {
70 + "match": {
71 + "turnIndex": 2,
72 + "userMessage": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
73 + "model": "claude-haiku-5-5",
74 + "hasToolResult": true,
75 + "toolCallId": "toolu_01FyBnAtspFmeLAURYkLwXiu",
76 + "toolResultContains": "\"exit_code\":0,"
77 + },
78 + "response": {
79 + "toolCalls": [
80 + {
81 + "id": "toolu_01MkR6XPCsvsEHA24TwcwCgv",
82 + "name": "shell",
83 + "arguments": "{\"command\":\"composio login --poll\"}"
84 + }
85 + ],
86 + "usage": {
87 + "input_tokens": 3087,
88 + "cache_creation_input_tokens": 0,
89 + "cache_read_input_tokens": 0,
90 + "cache_creation": {
91 + "ephemeral_5m_input_tokens": 0,
92 + "ephemeral_1h_input_tokens": 0
93 + },
94 + "output_tokens": 230
95 + },
96 + "content": "The login needs a browser step. Please open this URL and finish signing in: https://dashboard.composio.test?cliKey=00000000-0000-4000-8000-000000050001\n\nI'll wait for the login to complete."
97 + },
98 + "metadata": {
99 + "systemHash": "6ee9d189",
100 + "toolsHash": "95a7fdfe"
101 + }
102 + },
103 + {
104 + "match": {
105 + "turnIndex": 3,
106 + "userMessage": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
107 + "model": "claude-haiku-5-5",
108 + "hasToolResult": true,
109 + "toolCallId": "toolu_01MkR6XPCsvsEHA24TwcwCgv",
110 + "toolResultContains": "\"exit_code\":0,"
111 + },
112 + "response": {
113 + "toolCalls": [
114 + {
115 + "id": "toolu_01W7hx1TNLtdZhg9hVb5JSWt",
116 + "name": "shell",
117 + "arguments": "{\"command\":\"composio whoami\"}"
118 + }
119 + ],
120 + "usage": {
121 + "input_tokens": 3591,
122 + "cache_creation_input_tokens": 0,
123 + "cache_read_input_tokens": 0,
124 + "cache_creation": {
125 + "ephemeral_5m_input_tokens": 0,
126 + "ephemeral_1h_input_tokens": 0
127 + },
128 + "output_tokens": 50
129 + }
130 + },
131 + "metadata": {
132 + "systemHash": "6ee9d189",
133 + "toolsHash": "95a7fdfe"
134 + }
135 + },
136 + {
137 + "match": {
138 + "turnIndex": 4,
139 + "userMessage": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
140 + "model": "claude-haiku-5-5",
141 + "hasToolResult": true,
142 + "toolCallId": "toolu_01W7hx1TNLtdZhg9hVb5JSWt",
143 + "toolResultContains": "\"exit_code\":0,"
144 + },
145 + "response": {
146 + "usage": {
147 + "input_tokens": 3739,
148 + "cache_creation_input_tokens": 0,
149 + "cache_read_input_tokens": 0,
150 + "cache_creation": {
151 + "ephemeral_5m_input_tokens": 0,
152 + "ephemeral_1h_input_tokens": 0
153 + },
154 + "output_tokens": 144
155 + },
156 + "content": "The Composio CLI was not logged in. I ran `composio login`, you completed the browser step, and `composio login --poll` saved the credentials. `composio whoami` now confirms the session:\n\n- **Account:** dev@acme-e2e.test (human)\n- **Current org:** Acme E2E\n\nAnother org, Globex E2E, is also available. To switch to it, run `composio orgs switch --org-id ok_e2eglobex001`."
157 + },
158 + "metadata": {
159 + "systemHash": "6ee9d189",
160 + "toolsHash": "95a7fdfe"
161 + }
162 + }
163 + ]
164 +}
ts/e2e-tests/cli/agent-journeys/fixtures/first-run-login.anthropic.claude-opus-5-5.json new
+196
@@ -0,0 +1,196 @@
1 +{
2 + "$comment": "Recorded from anthropic claude-opus-5-5 by record.ts. Do not edit by hand; record again.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "turnIndex": 0,
7 + "userMessage": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
8 + "model": "claude-opus-5-5",
9 + "hasToolResult": false
10 + },
11 + "response": {
12 + "toolCalls": [
13 + {
14 + "id": "toolu_01CgRnGM11DuSrQ9ict337eQ",
15 + "name": "shell",
16 + "arguments": "{\"command\":\"composio whoami 2>&1\"}"
17 + }
18 + ],
19 + "usage": {
20 + "input_tokens": 2659,
21 + "cache_creation_input_tokens": 0,
22 + "cache_read_input_tokens": 0,
23 + "cache_creation": {
24 + "ephemeral_5m_input_tokens": 0,
25 + "ephemeral_1h_input_tokens": 0
26 + },
27 + "output_tokens": 55
28 + }
29 + },
30 + "metadata": {
31 + "systemHash": "6ee9d189",
32 + "toolsHash": "95a7fdfe"
33 + }
34 + },
35 + {
36 + "match": {
37 + "turnIndex": 1,
38 + "userMessage": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
39 + "model": "claude-opus-5-5",
40 + "hasToolResult": true,
41 + "toolCallId": "toolu_01CgRnGM11DuSrQ9ict337eQ",
42 + "toolResultContains": "\"exit_code\":1,"
43 + },
44 + "response": {
45 + "toolCalls": [
46 + {
47 + "id": "toolu_01MyLdQ5L6dR4Q7Deb9npFid",
48 + "name": "shell",
49 + "arguments": "{\"command\":\"composio login --help 2>&1 | head -40\"}"
50 + }
51 + ],
52 + "usage": {
53 + "input_tokens": 2761,
54 + "cache_creation_input_tokens": 0,
55 + "cache_read_input_tokens": 0,
56 + "cache_creation": {
57 + "ephemeral_5m_input_tokens": 0,
58 + "ephemeral_1h_input_tokens": 0
59 + },
60 + "output_tokens": 60
61 + }
62 + },
63 + "metadata": {
64 + "systemHash": "6ee9d189",
65 + "toolsHash": "95a7fdfe"
66 + }
67 + },
68 + {
69 + "match": {
70 + "turnIndex": 2,
71 + "userMessage": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
72 + "model": "claude-opus-5-5",
73 + "hasToolResult": true,
74 + "toolCallId": "toolu_01MyLdQ5L6dR4Q7Deb9npFid",
75 + "toolResultContains": "\"exit_code\":0,"
76 + },
77 + "response": {
78 + "toolCalls": [
79 + {
80 + "id": "toolu_01PfGPKZQa4E1WiXuNTMxTku",
81 + "name": "shell",
82 + "arguments": "{\"command\":\"composio login --no-wait 2>&1\"}"
83 + }
84 + ],
85 + "usage": {
86 + "input_tokens": 3215,
87 + "cache_creation_input_tokens": 0,
88 + "cache_read_input_tokens": 0,
89 + "cache_creation": {
90 + "ephemeral_5m_input_tokens": 0,
91 + "ephemeral_1h_input_tokens": 0
92 + },
93 + "output_tokens": 58
94 + }
95 + },
96 + "metadata": {
97 + "systemHash": "6ee9d189",
98 + "toolsHash": "95a7fdfe"
99 + }
100 + },
101 + {
102 + "match": {
103 + "turnIndex": 3,
104 + "userMessage": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
105 + "model": "claude-opus-5-5",
106 + "hasToolResult": true,
107 + "toolCallId": "toolu_01PfGPKZQa4E1WiXuNTMxTku",
108 + "toolResultContains": "\"exit_code\":0,"
109 + },
110 + "response": {
111 + "toolCalls": [
112 + {
113 + "id": "toolu_01AVc8JXiG4QuyxCQi39iN5t",
114 + "name": "shell",
115 + "arguments": "{\"command\":\"composio login --poll 2>&1\"}"
116 + }
117 + ],
118 + "usage": {
119 + "input_tokens": 3543,
120 + "cache_creation_input_tokens": 0,
121 + "cache_read_input_tokens": 0,
122 + "cache_creation": {
123 + "ephemeral_5m_input_tokens": 0,
124 + "ephemeral_1h_input_tokens": 0
125 + },
126 + "output_tokens": 120
127 + },
128 + "content": "Open this URL in your browser to log in:\n\n**https://dashboard.composio.test?cliKey=00000000-0000-4000-8000-000000050001**\n\nI'll wait for the login to finish."
129 + },
130 + "metadata": {
131 + "systemHash": "6ee9d189",
132 + "toolsHash": "95a7fdfe"
133 + }
134 + },
135 + {
136 + "match": {
137 + "turnIndex": 4,
138 + "userMessage": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
139 + "model": "claude-opus-5-5",
140 + "hasToolResult": true,
141 + "toolCallId": "toolu_01AVc8JXiG4QuyxCQi39iN5t",
142 + "toolResultContains": "\"exit_code\":0,"
143 + },
144 + "response": {
145 + "toolCalls": [
146 + {
147 + "id": "toolu_01HWpEgV3G3kShr9B9ZZNYHT",
148 + "name": "shell",
149 + "arguments": "{\"command\":\"composio whoami 2>&1\"}"
150 + }
151 + ],
152 + "usage": {
153 + "input_tokens": 3937,
154 + "cache_creation_input_tokens": 0,
155 + "cache_read_input_tokens": 0,
156 + "cache_creation": {
157 + "ephemeral_5m_input_tokens": 0,
158 + "ephemeral_1h_input_tokens": 0
159 + },
160 + "output_tokens": 55
161 + }
162 + },
163 + "metadata": {
164 + "systemHash": "6ee9d189",
165 + "toolsHash": "95a7fdfe"
166 + }
167 + },
168 + {
169 + "match": {
170 + "turnIndex": 5,
171 + "userMessage": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
172 + "model": "claude-opus-5-5",
173 + "hasToolResult": true,
174 + "toolCallId": "toolu_01HWpEgV3G3kShr9B9ZZNYHT",
175 + "toolResultContains": "\"exit_code\":0,"
176 + },
177 + "response": {
178 + "usage": {
179 + "input_tokens": 4090,
180 + "cache_creation_input_tokens": 0,
181 + "cache_read_input_tokens": 0,
182 + "cache_creation": {
183 + "ephemeral_5m_input_tokens": 0,
184 + "ephemeral_1h_input_tokens": 0
185 + },
186 + "output_tokens": 153
187 + },
188 + "content": "The Composio CLI wasn't logged in. It is now: you approved the login in your browser, and `composio whoami` confirms it.\n\n- **Account:** dev@acme-e2e.test\n- **Current organization:** Acme E2E (`ok_e2eacme00001`)\n\nYour account also belongs to **Globex E2E** (`ok_e2eglobex001`). To switch to it, run `composio orgs switch --org-id ok_e2eglobex001`."
189 + },
190 + "metadata": {
191 + "systemHash": "6ee9d189",
192 + "toolsHash": "95a7fdfe"
193 + }
194 + }
195 + ]
196 +}
ts/e2e-tests/cli/agent-journeys/fixtures/first-run-login.openai.hand-authored.json new
+84
@@ -0,0 +1,84 @@
1 +{
2 + "$comment": "Hand-authored: no model produced this file. It scripts the path the composio-cli skill teaches, one fixture per model turn. After the first turn each fixture is gated on the id of the previous shell call and on a substring of what that command printed, so a CLI that answers differently leaves the next turn without a match and the strict replay fails.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "userMessage": "Check whether the Composio CLI on this machine is logged in",
7 + "hasToolResult": false,
8 + "turnIndex": 0
9 + },
10 + "response": {
11 + "toolCalls": [
12 + {
13 + "id": "call_whoami_1",
14 + "name": "shell",
15 + "arguments": {
16 + "command": "composio whoami"
17 + }
18 + }
19 + ]
20 + }
21 + },
22 + {
23 + "match": {
24 + "toolCallId": "call_whoami_1",
25 + "toolResultContains": "You are not logged in yet. Please run `composio login`."
26 + },
27 + "response": {
28 + "toolCalls": [
29 + {
30 + "id": "call_login",
31 + "name": "shell",
32 + "arguments": {
33 + "command": "composio login"
34 + }
35 + }
36 + ]
37 + }
38 + },
39 + {
40 + "match": {
41 + "toolCallId": "call_login",
42 + "toolResultContains": "cliKey=00000000-0000-4000-8000-000000050001"
43 + },
44 + "response": {
45 + "content": "Open this URL in your browser to log in, and I will wait for it: https://dashboard.composio.test?cliKey=00000000-0000-4000-8000-000000050001",
46 + "toolCalls": [
47 + {
48 + "id": "call_poll",
49 + "name": "shell",
50 + "arguments": {
51 + "command": "composio login --poll"
52 + }
53 + }
54 + ]
55 + }
56 + },
57 + {
58 + "match": {
59 + "toolCallId": "call_poll",
60 + "toolResultContains": "current_org"
61 + },
62 + "response": {
63 + "toolCalls": [
64 + {
65 + "id": "call_whoami_2",
66 + "name": "shell",
67 + "arguments": {
68 + "command": "composio whoami"
69 + }
70 + }
71 + ]
72 + }
73 + },
74 + {
75 + "match": {
76 + "toolCallId": "call_whoami_2",
77 + "toolResultContains": "dev@acme-e2e.test"
78 + },
79 + "response": {
80 + "content": "The CLI is now logged in as dev@acme-e2e.test."
81 + }
82 + }
83 + ]
84 +}
ts/e2e-tests/cli/agent-journeys/fixtures/run-inline-script.anthropic.claude-haiku-5-5.json new
+136
@@ -0,0 +1,136 @@
1 +{
2 + "$comment": "Recorded from anthropic claude-haiku-5-5 by record.ts. Do not edit by hand; record again.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "turnIndex": 0,
7 + "userMessage": "Use a single `composio run` inline script to send \"build passed\" to #general on Slack and print the timestamp of the sent message.",
8 + "model": "claude-haiku-5-5",
9 + "hasToolResult": false
10 + },
11 + "response": {
12 + "toolCalls": [
13 + {
14 + "id": "toolu_01G92FrEpnpkB12CqnnRdTo3",
15 + "name": "shell",
16 + "arguments": "{\"command\":\"composio search \\\"send a message to a slack channel\\\" --toolkits slack\"}"
17 + },
18 + {
19 + "id": "toolu_01KewDTHMcgWnSXxMeXeZSeH",
20 + "name": "shell",
21 + "arguments": "{\"command\":\"composio whoami\"}"
22 + }
23 + ],
24 + "usage": {
25 + "input_tokens": 2670,
26 + "cache_creation_input_tokens": 0,
27 + "cache_read_input_tokens": 0,
28 + "cache_creation": {
29 + "ephemeral_5m_input_tokens": 0,
30 + "ephemeral_1h_input_tokens": 0
31 + },
32 + "output_tokens": 315
33 + },
34 + "content": "I'll confirm the Slack slug and its input schema before writing the script."
35 + },
36 + "metadata": {
37 + "systemHash": "6ee9d189",
38 + "toolsHash": "95a7fdfe"
39 + }
40 + },
41 + {
42 + "match": {
43 + "turnIndex": 1,
44 + "userMessage": "Use a single `composio run` inline script to send \"build passed\" to #general on Slack and print the timestamp of the sent message.",
45 + "model": "claude-haiku-5-5",
46 + "hasToolResult": true,
47 + "toolCallId": "toolu_01KewDTHMcgWnSXxMeXeZSeH",
48 + "toolResultContains": "\"exit_code\":0,"
49 + },
50 + "response": {
51 + "toolCalls": [
52 + {
53 + "id": "toolu_01LL5tsDxfRkRMG1KpzG2zUx",
54 + "name": "shell",
55 + "arguments": "{\"command\":\"composio execute SLACK_SEND_MESSAGE --get-schema\"}"
56 + }
57 + ],
58 + "usage": {
59 + "input_tokens": 3729,
60 + "cache_creation_input_tokens": 0,
61 + "cache_read_input_tokens": 0,
62 + "cache_creation": {
63 + "ephemeral_5m_input_tokens": 0,
64 + "ephemeral_1h_input_tokens": 0
65 + },
66 + "output_tokens": 69
67 + }
68 + },
69 + "metadata": {
70 + "systemHash": "6ee9d189",
71 + "toolsHash": "95a7fdfe"
72 + }
73 + },
74 + {
75 + "match": {
76 + "turnIndex": 2,
77 + "userMessage": "Use a single `composio run` inline script to send \"build passed\" to #general on Slack and print the timestamp of the sent message.",
78 + "model": "claude-haiku-5-5",
79 + "hasToolResult": true,
80 + "toolCallId": "toolu_01LL5tsDxfRkRMG1KpzG2zUx",
81 + "toolResultContains": "\"exit_code\":0,"
82 + },
83 + "response": {
84 + "toolCalls": [
85 + {
86 + "id": "toolu_01Hfd7ZbcJRqiRfRHwnTVct1",
87 + "name": "shell",
88 + "arguments": "{\"command\":\"composio run '\\n const res = await execute(\\\"SLACK_SEND_MESSAGE\\\", { channel: \\\"#general\\\", text: \\\"build passed\\\" });\\n const ts = res.data?.ts ?? res.data?.message?.ts ?? res.data?.data?.ts;\\n console.log(ts ? `Sent at ts=${ts}` : JSON.stringify(res.data, null, 2));\\n'\"}"
89 + }
90 + ],
91 + "usage": {
92 + "input_tokens": 4092,
93 + "cache_creation_input_tokens": 0,
94 + "cache_read_input_tokens": 0,
95 + "cache_creation": {
96 + "ephemeral_5m_input_tokens": 0,
97 + "ephemeral_1h_input_tokens": 0
98 + },
99 + "output_tokens": 295
100 + },
101 + "content": "The schema takes `channel` and `text`. Now I'll send the message with a single `composio run` script."
102 + },
103 + "metadata": {
104 + "systemHash": "6ee9d189",
105 + "toolsHash": "95a7fdfe"
106 + }
107 + },
108 + {
109 + "match": {
110 + "turnIndex": 3,
111 + "userMessage": "Use a single `composio run` inline script to send \"build passed\" to #general on Slack and print the timestamp of the sent message.",
112 + "model": "claude-haiku-5-5",
113 + "hasToolResult": true,
114 + "toolCallId": "toolu_01Hfd7ZbcJRqiRfRHwnTVct1",
115 + "toolResultContains": "\"exit_code\":0,"
116 + },
117 + "response": {
118 + "usage": {
119 + "input_tokens": 4471,
120 + "cache_creation_input_tokens": 0,
121 + "cache_read_input_tokens": 0,
122 + "cache_creation": {
123 + "ephemeral_5m_input_tokens": 0,
124 + "ephemeral_1h_input_tokens": 0
125 + },
126 + "output_tokens": 144
127 + },
128 + "content": "I sent \"build passed\" to #general with one `composio run` script, and the message timestamp came back as `1767225600.000100`.\n\nThe script called `SLACK_SEND_MESSAGE` with `channel: \"#general\"` and `text: \"build passed\"`, then printed the `ts` field from the response. I checked the tool's input schema first, and the `whoami` check showed I'm logged in as dev@acme-e2e.test with Slack connected."
129 + },
130 + "metadata": {
131 + "systemHash": "6ee9d189",
132 + "toolsHash": "95a7fdfe"
133 + }
134 + }
135 + ]
136 +}
ts/e2e-tests/cli/agent-journeys/fixtures/run-inline-script.anthropic.claude-opus-5-5.json new
+97
@@ -0,0 +1,97 @@
1 +{
2 + "$comment": "Recorded from anthropic claude-opus-5-5 by record.ts. Do not edit by hand; record again.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "turnIndex": 0,
7 + "userMessage": "Use a single `composio run` inline script to send \"build passed\" to #general on Slack and print the timestamp of the sent message.",
8 + "model": "claude-opus-5-5",
9 + "hasToolResult": false
10 + },
11 + "response": {
12 + "toolCalls": [
13 + {
14 + "id": "toolu_01XyE4ywTvLamSYfRXUJdJuq",
15 + "name": "shell",
16 + "arguments": "{\"command\":\"composio execute SLACK_SEND_MESSAGE --get-schema 2>&1 | head -50\"}"
17 + }
18 + ],
19 + "usage": {
20 + "input_tokens": 2670,
21 + "cache_creation_input_tokens": 0,
22 + "cache_read_input_tokens": 0,
23 + "cache_creation": {
24 + "ephemeral_5m_input_tokens": 0,
25 + "ephemeral_1h_input_tokens": 0
26 + },
27 + "output_tokens": 195
28 + },
29 + "content": "I'll check the Slack tool's schema first so the script uses the right arguments."
30 + },
31 + "metadata": {
32 + "systemHash": "6ee9d189",
33 + "toolsHash": "95a7fdfe"
34 + }
35 + },
36 + {
37 + "match": {
38 + "turnIndex": 1,
39 + "userMessage": "Use a single `composio run` inline script to send \"build passed\" to #general on Slack and print the timestamp of the sent message.",
40 + "model": "claude-opus-5-5",
41 + "hasToolResult": true,
42 + "toolCallId": "toolu_01XyE4ywTvLamSYfRXUJdJuq",
43 + "toolResultContains": "\"exit_code\":0,"
44 + },
45 + "response": {
46 + "toolCalls": [
47 + {
48 + "id": "toolu_01Aif6hvKXTrN3oVJXyXcvKJ",
49 + "name": "shell",
50 + "arguments": "{\"command\":\"composio run '\\n const res = await execute(\\\"SLACK_SEND_MESSAGE\\\", { channel: \\\"#general\\\", text: \\\"build passed\\\" });\\n const d = res.data ?? {};\\n const ts = d.ts ?? d.message?.ts ?? d.response_data?.ts;\\n if (!ts) { console.log(\\\"No ts found; full response:\\\", JSON.stringify(res, null, 2)); }\\n else console.log(\\\"Message timestamp:\\\", ts);\\n' 2>&1\"}"
51 + }
52 + ],
53 + "usage": {
54 + "input_tokens": 3159,
55 + "cache_creation_input_tokens": 0,
56 + "cache_read_input_tokens": 0,
57 + "cache_creation": {
58 + "ephemeral_5m_input_tokens": 0,
59 + "ephemeral_1h_input_tokens": 0
60 + },
61 + "output_tokens": 200
62 + }
63 + },
64 + "metadata": {
65 + "systemHash": "6ee9d189",
66 + "toolsHash": "95a7fdfe"
67 + }
68 + },
69 + {
70 + "match": {
71 + "turnIndex": 2,
72 + "userMessage": "Use a single `composio run` inline script to send \"build passed\" to #general on Slack and print the timestamp of the sent message.",
73 + "model": "claude-opus-5-5",
74 + "hasToolResult": true,
75 + "toolCallId": "toolu_01Aif6hvKXTrN3oVJXyXcvKJ",
76 + "toolResultContains": "\"exit_code\":0,"
77 + },
78 + "response": {
79 + "usage": {
80 + "input_tokens": 3444,
81 + "cache_creation_input_tokens": 0,
82 + "cache_read_input_tokens": 0,
83 + "cache_creation": {
84 + "ephemeral_5m_input_tokens": 0,
85 + "ephemeral_1h_input_tokens": 0
86 + },
87 + "output_tokens": 322
88 + },
89 + "content": "I sent \"build passed\" to #general on Slack with one `composio run` script.\n\n**Timestamp of the sent message:** `1767225600.000100`\n\nThe command I ran:\n```bash\ncomposio run '\n const res = await execute(\"SLACK_SEND_MESSAGE\", { channel: \"#general\", text: \"build passed\" });\n const d = res.data ?? {};\n const ts = d.ts ?? d.message?.ts ?? d.response_data?.ts;\n if (!ts) { console.log(\"No ts found; full response:\", JSON.stringify(res, null, 2)); }\n else console.log(\"Message timestamp:\", ts);\n'\n```\n\nBefore running it, I checked the tool's inputs with `composio execute SLACK_SEND_MESSAGE --get-schema`. The only required fields are `channel` and `text`.\n\nThe script looks for the timestamp in a few places in the response, because the schema doesn't say where it is. If it doesn't find one, it prints the full response instead."
90 + },
91 + "metadata": {
92 + "systemHash": "6ee9d189",
93 + "toolsHash": "95a7fdfe"
94 + }
95 + }
96 + ]
97 +}
ts/e2e-tests/cli/agent-journeys/fixtures/run-inline-script.openai.hand-authored.json new
+32
@@ -0,0 +1,32 @@
1 +{
2 + "$comment": "Hand-authored: no model produced this file. It scripts the path the composio-cli skill teaches, one fixture per model turn. After the first turn each fixture is gated on the id of the previous shell call and on a substring of what that command printed, so a CLI that answers differently leaves the next turn without a match and the strict replay fails.",
3 + "fixtures": [
4 + {
5 + "match": {
6 + "userMessage": "Use a single `composio run` inline script",
7 + "hasToolResult": false,
8 + "turnIndex": 0
9 + },
10 + "response": {
11 + "toolCalls": [
12 + {
13 + "id": "call_run",
14 + "name": "shell",
15 + "arguments": {
16 + "command": "composio run 'const sent = await execute(\"SLACK_SEND_MESSAGE\", { channel: \"general\", text: \"build passed\" }); console.log(sent.data.ts);'"
17 + }
18 + }
19 + ]
20 + }
21 + },
22 + {
23 + "match": {
24 + "toolCallId": "call_run",
25 + "toolResultContains": "1767225600.000100"
26 + },
27 + "response": {
28 + "content": "Sent \"build passed\" to #general; timestamp 1767225600.000100."
29 + }
30 + }
31 + ]
32 +}
ts/e2e-tests/cli/agent-journeys/journeys.test.ts new
+403
@@ -0,0 +1,403 @@
1 +/**
2 + * The graders, without a container: a run that reached the goal passes, and
3 + * each way of looking as if it did fails the check that names it.
4 + */
5 +
6 +import { describe, expect, it } from 'bun:test';
7 +import {
8 + CONSUMER_PROJECT,
9 + CONSUMER_USER_ID,
10 + DEVELOPER_PROJECT,
11 + GLOBAL_TEST_USER_ID,
12 + MEMBER,
13 + ORG,
14 + World,
15 + type JournalEntry,
16 +} from '@e2e-tests/cli-fake-backend';
17 +import { toolPayload, type ExecutedCommand, type ShellOutcome } from './src/agent';
18 +import { journeyById, JOURNEYS, type Evidence } from './src/journeys';
19 +
20 +const TIMESTAMP = '1767225600.000100';
21 +const LOGGED_IN: ShellOutcome = {
22 + exitCode: 0,
23 + stdout: `{"account_type":"human","email":"${MEMBER.email}","current_org_name":"${ORG.name}"}\n`,
24 + stderr: '',
25 +};
26 +
27 +let seq = 0;
28 +const request = (fields: Partial<JournalEntry>): JournalEntry => ({
29 + seq: (seq += 1),
30 + role: 'backend',
31 + method: 'POST',
32 + path: '/',
33 + query: {},
34 + route: null,
35 + keyClass: 'user',
36 + keyKnown: true,
37 + orgId: ORG.id,
38 + project: null,
39 + requestedProject: null,
40 + projectFrom: null,
41 + userId: null,
42 + body: null,
43 + status: 200,
44 + effect: null,
45 + tool: null,
46 + fault: null,
47 + ...fields,
48 +});
49 +
50 +const send = (
51 + tree: 'consumer' | 'developer',
52 + args: Record<string, unknown>,
53 + fields: Partial<JournalEntry> = {}
54 +): JournalEntry =>
55 + request({
56 + role: tree === 'consumer' ? 'dashboard' : 'backend',
57 + route: tree === 'consumer' ? 'dashboard.execute' : 'toolRouter.session.execute',
58 + project: tree === 'consumer' ? CONSUMER_PROJECT.label : DEVELOPER_PROJECT.label,
59 + // A developer command names its project; the Dashboard resolves the consumer one.
60 + requestedProject: tree === 'consumer' ? null : DEVELOPER_PROJECT.label,
61 + userId: tree === 'consumer' ? CONSUMER_USER_ID : GLOBAL_TEST_USER_ID,
62 + effect: 'tool-executed',
63 + tool: { slug: 'SLACK_SEND_MESSAGE', arguments: args },
64 + ...fields,
65 + });
66 +
67 +/** One shell command and the requests it caused. */
68 +const step = (
69 + command: string,
70 + requests: ReadonlyArray<JournalEntry> = [],
71 + outcome: Partial<ShellOutcome> = {}
72 +) => {
73 + const result: ShellOutcome = { exitCode: 0, stdout: '', stderr: '', ...outcome };
74 + const executed: ExecutedCommand = { command, ...result, payload: toolPayload(result) };
75 + return { executed, requests };
76 +};
77 +
78 +const evidence = (
79 + world: World,
80 + steps: ReadonlyArray<ReturnType<typeof step>>,
81 + overrides: Partial<Evidence> = {}
82 +): Evidence => ({
83 + transcript: {
84 + commands: steps.map(taken => taken.executed),
85 + messages: [],
86 + replies: [],
87 + stop: 'finished',
88 + modelTurns: steps.length + 1,
89 + },
90 + world,
91 + requests: steps.flatMap(taken => taken.requests),
92 + commandRequests: steps.map(taken => taken.requests),
93 + startingConnections: [],
94 + whoamiAfter: LOGGED_IN,
95 + ...overrides,
96 +});
97 +
98 +const failed = (id: string, run: Evidence): string[] =>
99 + journeyById(id)
100 + .checks(run)
101 + .filter(check => !check.ok)
102 + .map(check => check.name);
103 +
104 +const detoursTaken = (id: string, run: Evidence): string[] =>
105 + (journeyById(id).detours?.(run) ?? []).filter(check => !check.ok).map(check => check.name);
106 +
107 +const DEV_SEND = `composio dev playground-execute SLACK_SEND_MESSAGE -d '{}'`;
108 +const ROOT_SEND = `composio execute SLACK_SEND_MESSAGE -d '{}'`;
109 +const hello = { channel: 'general', text: 'hello from dev' };
110 +
111 +describe('developer-execute-unconnected', () => {
112 + const id = 'developer-execute-unconnected';
113 + const connected = () => {
114 + const world = new World();
115 + world.connect(DEVELOPER_PROJECT.label, GLOBAL_TEST_USER_ID, 'slack');
116 + return world;
117 + };
118 +
119 + it('passes when the message was sent once, on the developer project', () => {
120 + expect(failed(id, evidence(connected(), [step(DEV_SEND, [send('developer', hello)])]))).toEqual(
121 + []
122 + );
123 + });
124 +
125 + it('fails when the message was also sent on the consumer project', () => {
126 + const world = connected();
127 + world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
128 + const run = evidence(world, [
129 + step('composio link slack'),
130 + step(ROOT_SEND, [send('consumer', hello)]),
131 + step(DEV_SEND, [send('developer', hello)]),
132 + ]);
133 + expect(failed(id, run)).toEqual([
134 + 'SLACK_SEND_MESSAGE completed once in the whole run',
135 + `it ran on the ${DEVELOPER_PROJECT.label} project, as ${GLOBAL_TEST_USER_ID}`,
136 + `no account was connected outside the ${DEVELOPER_PROJECT.label} project`,
137 + ]);
138 + });
139 +
140 + it('fails when the person approved a consumer connection, even with nothing sent there', () => {
141 + const world = connected();
142 + world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
143 + const run = evidence(world, [
144 + step('composio link slack'),
145 + step(DEV_SEND, [send('developer', hello)]),
146 + ]);
147 + expect(failed(id, run)).toEqual([
148 + `no account was connected outside the ${DEVELOPER_PROJECT.label} project`,
149 + ]);
150 + });
151 +
152 + it('passes after a wrong-tree attempt that changed nothing, and reports the detour', () => {
153 + const attempt = request({
154 + role: 'dashboard',
155 + route: 'dashboard.execute',
156 + project: CONSUMER_PROJECT.label,
157 + userId: CONSUMER_USER_ID,
158 + });
159 + const run = evidence(connected(), [
160 + step(ROOT_SEND, [attempt], { exitCode: 1 }),
161 + step(DEV_SEND, [send('developer', hello)]),
162 + ]);
163 + expect(failed(id, run)).toEqual([]);
164 + expect(detoursTaken(id, run)).toEqual([
165 + 'no root `composio execute` tried to run the tool',
166 + 'nothing was executed or linked on the consumer project, completed or not',
167 + ]);
168 + });
169 +
170 + it('does not count describing the tool on the root tree as trying to run it', () => {
171 + const run = evidence(connected(), [
172 + step('composio execute SLACK_SEND_MESSAGE --get-schema'),
173 + step(DEV_SEND, [send('developer', hello)]),
174 + ]);
175 + expect(detoursTaken(id, run)).toEqual([]);
176 + });
177 +
178 + it('fails when the request named another project, or none, and the session decided', () => {
179 + const scopeCheck = `it ran on the ${DEVELOPER_PROJECT.label} project, as ${GLOBAL_TEST_USER_ID}`;
180 + const crossed = send('developer', hello, { requestedProject: CONSUMER_PROJECT.label });
181 + const unscoped = send('developer', hello, { requestedProject: null });
182 + expect(failed(id, evidence(connected(), [step(DEV_SEND, [crossed])]))).toEqual([scopeCheck]);
183 + expect(failed(id, evidence(connected(), [step(DEV_SEND, [unscoped])]))).toEqual([scopeCheck]);
184 + });
185 +});
186 +
187 +describe('consumer-slack-message', () => {
188 + const id = 'consumer-slack-message';
189 + const connected = () => {
190 + const world = new World();
191 + world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
192 + return world;
193 + };
194 + const sent = (args: Record<string, unknown>) =>
195 + evidence(connected(), [step(ROOT_SEND, [send('consumer', args)])]);
196 +
197 + it('passes for #general or general, with the text as asked', () => {
198 + expect(failed(id, sent({ channel: '#general', text: 'deploy finished' }))).toEqual([]);
199 + expect(failed(id, sent({ channel: 'general', text: 'deploy finished' }))).toEqual([]);
200 + });
201 +
202 + it('fails on another channel', () => {
203 + expect(failed(id, sent({ channel: 'private-leadership', text: 'deploy finished' }))).toEqual([
204 + 'it went to #general',
205 + ]);
206 + });
207 +
208 + it('fails on other words around the text', () => {
209 + expect(failed(id, sent({ channel: '#general', text: 'deploy finished WRONG TEXT' }))).toEqual([
210 + 'its text was exactly "deploy finished"',
211 + ]);
212 + });
213 +
214 + it('fails on a second send, whatever it said', () => {
215 + const run = evidence(connected(), [
216 + step(ROOT_SEND, [send('consumer', { channel: '#general', text: 'deploy finished' })]),
217 + step(ROOT_SEND, [send('consumer', { channel: '#general', text: 'deploy finished' })]),
218 + ]);
219 + expect(failed(id, run)).toEqual(['SLACK_SEND_MESSAGE completed once in the whole run']);
220 + });
221 +
222 + it('sees a send through the direct execute route', () => {
223 + const direct = send(
224 + 'consumer',
225 + { channel: '#general', text: 'deploy finished' },
226 + { role: 'backend', route: 'tools.execute' }
227 + );
228 + const run = evidence(connected(), [
229 + step(ROOT_SEND, [send('consumer', { channel: '#general', text: 'deploy finished' })]),
230 + step('curl …', [direct]),
231 + ]);
232 + expect(failed(id, run)).toContain('SLACK_SEND_MESSAGE completed once in the whole run');
233 + });
234 +
235 + it('lets a lookup through, and fails on another tool that changes something', () => {
236 + const other = (slug: string) => send('consumer', {}, { tool: { slug, arguments: {} } });
237 + const withTool = (slug: string) =>
238 + evidence(connected(), [
239 + step('composio execute …', [other(slug)]),
240 + step(ROOT_SEND, [send('consumer', { channel: '#general', text: 'deploy finished' })]),
241 + ]);
242 + expect(failed(id, withTool('HACKERNEWS_GET_USER'))).toEqual([]);
243 + expect(failed(id, withTool('GMAIL_SEND_EMAIL'))).toEqual([
244 + 'no other tool that changes something ran',
245 + ]);
246 + });
247 +
248 + it('reports a look at the dev tree as a detour, not a failure', () => {
249 + const run = evidence(connected(), [
250 + step('composio dev --help'),
251 + step(ROOT_SEND, [send('consumer', { channel: '#general', text: 'deploy finished' })]),
252 + ]);
253 + expect(failed(id, run)).toEqual([]);
254 + expect(detoursTaken(id, run)).toEqual(['no `composio dev` command ran']);
255 + });
256 +});
257 +
258 +describe('run-inline-script', () => {
259 + const id = 'run-inline-script';
260 + const built = { channel: '#general', text: 'build passed' };
261 + const world = () => {
262 + const seeded = new World();
263 + const connection = seeded.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
264 + return { seeded, startingConnections: [connection.id] };
265 + };
266 + const SCRIPT = `composio run 'const r = await execute("SLACK_SEND_MESSAGE", {}); console.log(r.data.ts)'`;
267 +
268 + it('passes when the script sent the message and printed its timestamp', () => {
269 + const { seeded, startingConnections } = world();
270 + const run = evidence(
271 + seeded,
272 + [step(SCRIPT, [send('consumer', built)], { stdout: `${TIMESTAMP}\n` })],
273 + { startingConnections }
274 + );
275 + expect(failed(id, run)).toEqual([]);
276 + });
277 +
278 + it("takes no detour for the CLI's own read of the org's settings", () => {
279 + const { seeded, startingConnections } = world();
280 + const orgRead = request({
281 + route: '_internal.org.consumer.retrieveConfig',
282 + project: DEVELOPER_PROJECT.label,
283 + method: 'GET',
284 + });
285 + const run = evidence(
286 + seeded,
287 + [step(SCRIPT, [orgRead, send('consumer', built)], { stdout: `${TIMESTAMP}\n` })],
288 + { startingConnections }
289 + );
290 + expect(detoursTaken(id, run)).toEqual([]);
291 + });
292 +
293 + it('fails when `composio execute` sent it and a script only printed the timestamp', () => {
294 + const { seeded, startingConnections } = world();
295 + const run = evidence(
296 + seeded,
297 + [
298 + step(`composio execute SLACK_SEND_MESSAGE -d '{}'`, [send('consumer', built)]),
299 + step(`composio run 'console.log("${TIMESTAMP}")'`, [], { stdout: `${TIMESTAMP}\n` }),
300 + ],
301 + { startingConnections }
302 + );
303 + expect(failed(id, run)).toEqual([
304 + 'the message was sent by a command that is one `composio run`, which exited 0 and printed its timestamp',
305 + ]);
306 + });
307 +
308 + it('fails when the binary is named by its path, or the line runs two scripts', () => {
309 + const { seeded, startingConnections } = world();
310 + const one = (command: string) =>
311 + evidence(seeded, [step(command, [send('consumer', built)], { stdout: `${TIMESTAMP}\n` })], {
312 + startingConnections,
313 + });
314 + const failure =
315 + 'the message was sent by a command that is one `composio run`, which exited 0 and printed its timestamp';
316 + expect(
317 + failed(
318 + id,
319 + one(
320 + `/usr/local/bin/composio execute SLACK_SEND_MESSAGE -d '{}'; composio run 'console.log("${TIMESTAMP}")'`
321 + )
322 + )
323 + ).toEqual([failure]);
324 + expect(failed(id, one(`${SCRIPT}; composio run 'console.log(1)'`))).toEqual([failure]);
325 + });
326 +
327 + it('fails when both happened in one shell command', () => {
328 + const { seeded, startingConnections } = world();
329 + const run = evidence(
330 + seeded,
331 + [
332 + step(
333 + `composio execute SLACK_SEND_MESSAGE -d '{}'; composio run 'console.log("${TIMESTAMP}")'`,
334 + [send('consumer', built)],
335 + { stdout: `${TIMESTAMP}\n` }
336 + ),
337 + ],
338 + { startingConnections }
339 + );
340 + expect(failed(id, run)).toEqual([
341 + 'the message was sent by a command that is one `composio run`, which exited 0 and printed its timestamp',
342 + ]);
343 + });
344 +});
345 +
346 +describe('first-run-login', () => {
347 + const id = 'first-run-login';
348 + const loggedIn = () => {
349 + const world = new World();
350 + world.createLoginSession();
351 + world.linkLoginSession();
352 + return [
353 + world,
354 + [
355 + step('composio whoami', [], { exitCode: 1 }),
356 + step('composio login --poll', [
357 + request({ method: 'GET', route: '_internal.cli.getSession' }),
358 + request({ method: 'GET', route: 'auth.session.retrieveInfo' }),
359 + ]),
360 + ],
361 + ] as const;
362 + };
363 +
364 + it('passes when the next process finds the login', () => {
365 + const [world, steps] = loggedIn();
366 + expect(failed(id, evidence(world, steps))).toEqual([]);
367 + });
368 +
369 + it('reads the login from the JSON, however it is laid out', () => {
370 + const [world, steps] = loggedIn();
371 + const pretty = JSON.stringify(JSON.parse(LOGGED_IN.stdout), null, 2);
372 + expect(
373 + failed(id, evidence(world, steps, { whoamiAfter: { ...LOGGED_IN, stdout: pretty } }))
374 + ).toEqual([]);
375 + });
376 +
377 + it('fails when the key was fetched and nothing was stored', () => {
378 + const [world, steps] = loggedIn();
379 + const run = evidence(world, steps, {
380 + whoamiAfter: { exitCode: 1, stdout: '', stderr: 'You are not logged in yet.' },
381 + });
382 + expect(failed(id, run)).toEqual([
383 + 'the login was stored: a new `composio whoami` exits 0 as the person, in their org',
384 + ]);
385 + });
386 +});
387 +
388 +// Harbor's "nop" agent: a grader that an agent passes by doing nothing measures nothing.
389 +describe('an agent that does nothing', () => {
390 + for (const journey of JOURNEYS) {
391 + it(`fails ${journey.id}`, () => {
392 + const world = new World();
393 + // The account the journey's setup may have connected is already there.
394 + world.connect(DEVELOPER_PROJECT.label, GLOBAL_TEST_USER_ID, 'slack');
395 + world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
396 + const idle = evidence(world, [], {
397 + startingConnections: world.connections.map(connection => connection.id),
398 + whoamiAfter: { exitCode: 1, stdout: '', stderr: 'You are not logged in yet.' },
399 + });
400 + expect(failed(journey.id, idle)).not.toEqual([]);
401 + });
402 + }
403 +});
ts/e2e-tests/cli/agent-journeys/package.json new
+22
@@ -0,0 +1,22 @@
1 +{
2 + "name": "@e2e-tests/cli-agent-journeys",
3 + "version": "0.0.0",
4 + "private": true,
5 + "type": "module",
6 + "scripts": {
7 + "typecheck": "tsc --noEmit",
8 + "test:e2e": "bun test",
9 + "test:e2e:cli": "bun test",
10 + "record": "bun run record.ts"
11 + },
12 + "devDependencies": {
13 + "@anthropic-ai/sdk": "^0.135.0",
14 + "@copilotkit/aimock": "catalog:",
15 + "@e2e-tests/cli-fake-backend": "workspace:*",
16 + "@e2e-tests/utils": "workspace:*",
17 + "@types/bun": "catalog:",
18 + "openai": "catalog:",
19 + "typescript": "catalog:",
20 + "zod": "catalog:"
21 + }
22 +}
ts/e2e-tests/cli/agent-journeys/person.test.ts new
+135
@@ -0,0 +1,135 @@
1 +/**
2 + * The person model, without a container: what makes them approve, and when
3 + * they say so.
4 + */
5 +
6 +import { describe, expect, it } from 'bun:test';
7 +import {
8 + CONSUMER_PROJECT,
9 + CONSUMER_USER_ID,
10 + DEVELOPER_PROJECT,
11 + GLOBAL_TEST_USER_ID,
12 + World,
13 +} from '@e2e-tests/cli-fake-backend';
14 +import { PERSON_REPLY, personFor } from './src/person';
15 +import { SCRUBBED_BACKEND_URL, SCRUBBED_DASHBOARD_URL } from './src/scrub';
16 +
17 +const start = () => {
18 + const world = new World();
19 + // Nothing completes by waiting, as in a journey.
20 + world.linkCompletesAfterPolls = null;
21 + const person = personFor(world, {
22 + dashboard: SCRUBBED_DASHBOARD_URL,
23 + backend: SCRUBBED_BACKEND_URL,
24 + });
25 + return { world, person };
26 +};
27 +
28 +const loginUrl = (id: string) => `${SCRUBBED_DASHBOARD_URL}?cliKey=${id}`;
29 +const authorizationUrl = (id: string) => `${SCRUBBED_BACKEND_URL}/api/v3/s/${id}`;
30 +
31 +describe('what the person approves', () => {
32 + it('approves a login whose complete URL the agent wrote', () => {
33 + const { world, person } = start();
34 + const session = world.createLoginSession();
35 + person.hear(`Open ${loginUrl(session.id)} to log in.`, false);
36 + expect(session.status).toBe('linked');
37 + });
38 +
39 + it('reads a URL through markdown and trailing punctuation', () => {
40 + const { world, person } = start();
41 + const first = world.createLoginSession();
42 + const second = world.createLoginSession();
43 + person.hear(`[Log in](${loginUrl(first.id)}), or: **${loginUrl(second.id)}**.`, false);
44 + expect([first.status, second.status]).toEqual(['linked', 'linked']);
45 + });
46 +
47 + it('does not approve a login from its id alone', () => {
48 + const { world, person } = start();
49 + const session = world.createLoginSession();
50 + person.hear(`Debug session id: ${session.id}`, true);
51 + expect(session.status).toBe('pending');
52 + });
53 +
54 + it('does not approve a login whose id is followed by more characters', () => {
55 + const { world, person } = start();
56 + const session = world.createLoginSession();
57 + person.hear(`Open ${loginUrl(session.id)}0 to log in.`, true);
58 + expect(session.status).toBe('pending');
59 + });
60 +
61 + it('does not approve an authorization page on another host', () => {
62 + const { world, person } = start();
63 + const connection = world.beginConnection({
64 + projectLabel: DEVELOPER_PROJECT.label,
65 + userId: GLOBAL_TEST_USER_ID,
66 + toolkit: 'slack',
67 + });
68 + person.hear(`Authorize here: https://wrong.invalid/api/v3/s/${connection.id}`, true);
69 + expect(connection.status).toBe('INITIATED');
70 + });
71 +
72 + it('does not recognise a URL written without its scheme', () => {
73 + const { world, person } = start();
74 + const session = world.createLoginSession();
75 + person.hear(`Open ${loginUrl(session.id).replace('https://', '')}`, true);
76 + expect(session.status).toBe('pending');
77 + });
78 +});
79 +
80 +describe('when the person says they approved', () => {
81 + it('answers an agent that stopped to wait, once', () => {
82 + const { world, person } = start();
83 + const connection = world.beginConnection({
84 + projectLabel: CONSUMER_PROJECT.label,
85 + userId: CONSUMER_USER_ID,
86 + toolkit: 'slack',
87 + });
88 + const message = `Please authorize Slack: ${authorizationUrl(connection.id)}`;
89 + expect(person.hear(message, true)).toBe(PERSON_REPLY);
90 + expect(connection.status).toBe('ACTIVE');
91 + // The agent thanks them and stops again without checking: nothing new to say.
92 + expect(person.hear('Thanks, Slack is connected.', true)).toBeUndefined();
93 + });
94 +
95 + it('answers later when the URL came beside a command and the agent stops afterwards', () => {
96 + const { world, person } = start();
97 + const connection = world.beginConnection({
98 + projectLabel: CONSUMER_PROJECT.label,
99 + userId: CONSUMER_USER_ID,
100 + toolkit: 'slack',
101 + });
102 + expect(person.hear(`Authorize: ${authorizationUrl(connection.id)}`, false)).toBeUndefined();
103 + // A turn of unrelated commands, with nothing written.
104 + expect(person.hear('', false)).toBeUndefined();
105 + expect(person.hear('Tell me once you have authorized Slack.', true)).toBe(PERSON_REPLY);
106 + });
107 +
108 + it('stays quiet once a command of the agent saw the approval', () => {
109 + const { world, person } = start();
110 + const session = world.createLoginSession();
111 + expect(person.hear(`Log in at ${loginUrl(session.id)}`, false)).toBeUndefined();
112 + // `composio login --poll` in the same turn reads the linked session.
113 + session.reportedLinked = true;
114 + expect(person.hear('You are logged in.', true)).toBeUndefined();
115 + });
116 +
117 + it('says so after approving a page the CLI opened in their browser', () => {
118 + const { world, person } = start();
119 + const connection = world.beginConnection({
120 + projectLabel: CONSUMER_PROJECT.label,
121 + userId: CONSUMER_USER_ID,
122 + toolkit: 'slack',
123 + });
124 + world.authorizeConnection(connection.id);
125 + expect(person.hear('I opened the page in your browser. Tell me when done.', true)).toBe(
126 + PERSON_REPLY
127 + );
128 + });
129 +
130 + it('has nothing to say about a connection that existed before the agent started', () => {
131 + const { world, person } = start();
132 + world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
133 + expect(person.hear('All done.', true)).toBeUndefined();
134 + });
135 +});
ts/e2e-tests/cli/agent-journeys/record.ts new
+243
@@ -0,0 +1,243 @@
1 +/**
2 + * Records one journey with a real model.
3 + *
4 + * ANTHROPIC_API_KEY=… bun run record.ts --set <name> --journey <id> --provider anthropic --model <model>
5 + * OPENAI_API_KEY=… bun run record.ts --set <name> --journey <id> --provider openai --model <model>
6 + *
7 + * aimock sits between the agent loop and the provider, forwards each request,
8 + * and records the answer. The run is judged by the journey's own checks: a run
9 + * that reached the goal becomes `fixtures/<journey>.<provider>.<model>.json`;
10 + * a run that did not leaves the fixtures alone. Either way the transcript is
11 + * kept in `transcripts/<set>/`, where a set names what was being measured.
12 + *
13 + * One command records one journey with one model, once. There are no retries.
14 + */
15 +
16 +import { LLMock, type RecordProviderKey } from '@copilotkit/aimock';
17 +import { execFileSync } from 'node:child_process';
18 +import { createHash } from 'node:crypto';
19 +import { mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
20 +import { tmpdir } from 'node:os';
21 +import { join } from 'node:path';
22 +import { parseArgs } from 'node:util';
23 +import { z } from 'zod';
24 +import { adapterFor } from './src/adapters';
25 +import { fixturePath, HAND_AUTHORED, type Provider } from './src/fixtures';
26 +import { journeyById, JOURNEYS } from './src/journeys';
27 +import { runJourney } from './src/run-journey';
28 +import { buildSkill } from './src/skill';
29 +
30 +const UPSTREAMS: Record<Provider, { url: string; keyVariable: string }> = {
31 + anthropic: { url: 'https://api.anthropic.com', keyVariable: 'ANTHROPIC_API_KEY' },
32 + openai: { url: 'https://api.openai.com', keyVariable: 'OPENAI_API_KEY' },
33 +};
34 +
35 +const usage = () =>
36 + [
37 + 'Usage: bun run record.ts --set <name> --journey <id> --provider <anthropic|openai> --model <model>',
38 + `Journeys: ${JOURNEYS.map(journey => journey.id).join(', ')}`,
39 + ].join('\n');
40 +
41 +const { values } = parseArgs({
42 + options: {
43 + journey: { type: 'string' },
44 + provider: { type: 'string' },
45 + model: { type: 'string' },
46 + set: { type: 'string' },
47 + // For rehearsing the recorder against a local stand-in for the provider.
48 + upstream: { type: 'string' },
49 + },
50 +});
51 +
52 +const provider = z.enum(['anthropic', 'openai']).safeParse(values.provider);
53 +const set = z
54 + .string()
55 + .regex(/^[a-z0-9][a-z0-9-]*$/)
56 + .safeParse(values.set);
57 +if (
58 + !set.success ||
59 + !values.journey ||
60 + !provider.success ||
61 + !values.model ||
62 + values.model === HAND_AUTHORED
63 +) {
64 + console.error(usage());
65 + process.exit(2);
66 +}
67 +const journey = journeyById(values.journey);
68 +const model = values.model;
69 +const upstream = UPSTREAMS[provider.data];
70 +const apiKey = process.env[upstream.keyVariable];
71 +if (!apiKey && !values.upstream) {
72 + console.error(`${upstream.keyVariable} is not set; nothing was sent.`);
73 + process.exit(2);
74 +}
75 +
76 +/** What aimock's recorder writes: one file per answered request. */
77 +const RecordedFile = z.object({
78 + fixtures: z
79 + .array(
80 + z.looseObject({
81 + match: z.looseObject({ turnIndex: z.number().optional() }),
82 + response: z.looseObject({
83 + toolCalls: z.array(z.looseObject({ id: z.string().optional() })).optional(),
84 + }),
85 + })
86 + )
87 + .min(1),
88 +});
89 +
90 +const outDirectory = join(import.meta.dir, 'transcripts', set.data);
91 +mkdirSync(outDirectory, { recursive: true });
92 +
93 +const git = (...args: string[]): string =>
94 + execFileSync('git', args, { cwd: import.meta.dir, encoding: 'utf8' }).trim();
95 +const CLI_PACKAGE = ':/ts/packages/cli';
96 +/** What the run measured: the checkout, and the skill text the model was given. */
97 +const provenance = {
98 + recordedOn: new Date().toISOString().slice(0, 10),
99 + revision: git('rev-parse', '--short', 'HEAD'),
100 + // Uncommitted changes to the CLI or the skill mean the revision alone does not name what ran.
101 + cliPackageModified: git('status', '--porcelain', '--', CLI_PACKAGE) !== '',
102 + // The system prompt and the reference files the model can open.
103 + skillSha256: createHash('sha256').update(JSON.stringify(buildSkill())).digest('hex'),
104 +};
105 +const rawDirectory = mkdtempSync(join(tmpdir(), 'composio-agent-journey-recording-'));
106 +
107 +const providerKey: RecordProviderKey = provider.data;
108 +const mock = new LLMock({
109 + port: 0,
110 + logLevel: 'silent',
111 + record: {
112 + providers: { [providerKey]: values.upstream ?? upstream.url },
113 + // The agent loop only ever sends a placeholder; aimock holds the real key.
114 + providerKeys: apiKey ? { [providerKey]: apiKey } : {},
115 + fixturePath: rawDirectory,
116 + },
117 +});
118 +
119 +const baseURL = await mock.start();
120 +let exitCode = 0;
121 +try {
122 + const run = await runJourney(journey, adapterFor(provider.data, model, baseURL));
123 + try {
124 + const checks = journey.checks(run);
125 + const reachedGoal = checks.every(check => check.ok);
126 + // Reported, never a failure: what the model tried on the way.
127 + const detours = journey.detours?.(run) ?? [];
128 +
129 + const recorded = readdirSync(rawDirectory, { recursive: true, encoding: 'utf8' })
130 + .filter(name => name.endsWith('.json'))
131 + .flatMap(
132 + name =>
133 + RecordedFile.parse(JSON.parse(readFileSync(join(rawDirectory, name), 'utf8'))).fixtures
134 + )
135 + .sort((left, right) => (left.match.turnIndex ?? 0) - (right.match.turnIndex ?? 0));
136 +
137 + // The recorder keys a turn on its position only. Each turn after the first
138 + // is also tied to the call it answers and to that command's exit code, so a
139 + // CLI that now exits differently leaves the replay without a match.
140 + let commandsSeen = 0;
141 + const fixtures = recorded.map((fixture, index) => {
142 + const previous = index === 0 ? undefined : recorded[index - 1];
143 + const previousCalls = previous?.response.toolCalls ?? [];
144 + commandsSeen += previousCalls.length;
145 + const answered = run.transcript.commands[commandsSeen - 1];
146 + const lastCallId = previousCalls.at(-1)?.id;
147 + return previous && lastCallId && answered
148 + ? {
149 + ...fixture,
150 + match: {
151 + ...fixture.match,
152 + toolCallId: lastCallId,
153 + toolResultContains: `"exit_code":${answered.exitCode},`,
154 + },
155 + }
156 + : fixture;
157 + });
158 +
159 + const fixtureFile = JSON.stringify(
160 + {
161 + $comment: `Recorded from ${provider.data} ${model} by record.ts. Do not edit by hand; record again.`,
162 + fixtures,
163 + },
164 + null,
165 + 2
166 + );
167 + const transcriptFile = JSON.stringify(
168 + {
169 + journey: journey.id,
170 + provider: provider.data,
171 + model,
172 + ...provenance,
173 + prompt: journey.prompt,
174 + reachedGoal,
175 + stop: run.transcript.stop,
176 + modelTurns: run.transcript.modelTurns,
177 + checks,
178 + detours,
179 + commands: run.transcript.commands.map(({ command, exitCode: code, stdout, stderr }) => ({
180 + command,
181 + exitCode: code,
182 + stdout,
183 + stderr,
184 + })),
185 + messages: run.transcript.messages,
186 + replies: run.transcript.replies,
187 + },
188 + null,
189 + 2
190 + );
191 +
192 + // Nothing that could carry a credential is kept, whatever the recorder did.
193 + const leaks = [fixtureFile, transcriptFile].flatMap(text => [
194 + ...(apiKey && text.includes(apiKey) ? ['the API key'] : []),
195 + // A provider key by its prefix, or a credential header recorded as a JSON key.
196 + ...(/sk-ant-|sk-proj-|"(?:x-api-key|authorization)"\s*:/i.test(text)
197 + ? ['a credential marker']
198 + : []),
199 + ]);
200 + if (leaks.length > 0) {
201 + throw new Error(
202 + `Refusing to write: the recording contains ${[...new Set(leaks)].join(' and ')}.`
203 + );
204 + }
205 +
206 + const stem = `${journey.id}.${provider.data}.${model}`;
207 + const transcriptPath = join(outDirectory, `${stem}.json`);
208 + writeFileSync(transcriptPath, `${transcriptFile}\n`);
209 +
210 + console.log(`journey ${journey.id}`);
211 + console.log(`model ${provider.data} ${model}`);
212 + console.log(
213 + `stop ${JSON.stringify(run.transcript.stop)} after ${run.transcript.modelTurns} model turn(s)`
214 + );
215 + for (const command of run.transcript.commands) {
216 + console.log(` [${command.exitCode}] ${command.command}`);
217 + }
218 + for (const check of checks) {
219 + console.log(
220 + ` ${check.ok ? 'ok ' : 'FAIL'} ${check.name}${check.ok ? '' : ` (saw ${check.seen ?? 'nothing'})`}`
221 + );
222 + }
223 + for (const detour of detours.filter(taken => !taken.ok)) {
224 + console.log(` detour ${detour.name} (saw ${detour.seen ?? 'it'})`);
225 + }
226 + console.log(`transcript ${transcriptPath}`);
227 +
228 + if (reachedGoal) {
229 + const target = fixturePath(journey.id, provider.data, model);
230 + writeFileSync(target, `${fixtureFile}\n`);
231 + console.log(`fixture ${target} (${fixtures.length} turn(s))`);
232 + } else {
233 + console.log('fixture not written: the run did not reach the goal');
234 + exitCode = 1;
235 + }
236 + } finally {
237 + await run.rig.stop();
238 + }
239 +} finally {
240 + await mock.stop();
241 + rmSync(rawDirectory, { recursive: true, force: true });
242 +}
243 +process.exit(exitCode);
ts/e2e-tests/cli/agent-journeys/skill-commands.test.ts new
+86
@@ -0,0 +1,86 @@
1 +/**
2 + * The commands the skill teaches, checked against the binary.
3 + *
4 + * Every `composio …` line in a code block of the built skill is reduced to its
5 + * command path and looked up with `composio help <path>`. A path the binary
6 + * does not know is a line that sends an agent into a parse error on its first
7 + * try.
8 + */
9 +
10 +import { afterAll, beforeAll, describe, expect, it } from 'bun:test';
11 +import { startCliRig, type CliRig } from '@e2e-tests/cli-fake-backend/rig';
12 +import { buildSkill } from './src/skill';
13 +
14 +const RIG_TIMEOUT_MS = 1_200_000;
15 +
16 +/** `composio dev toolkits info github` -> `dev toolkits info github`: the words before any flag, quote, or slug. */
17 +const leadingWords = (line: string): string[] => {
18 + const words: string[] = [];
19 + for (const token of line.trim().split(/\s+/).slice(1)) {
20 + if (!/^[a-z][a-z-]*$/.test(token)) break;
21 + words.push(token);
22 + }
23 + return words;
24 +};
25 +
26 +const taughtCommands = (): Array<{ file: string; line: string; words: string[] }> => {
27 + const skill = buildSkill();
28 + const documents = { 'SKILL.md': skill.skillMarkdown, ...skill.references };
29 + return Object.entries(documents).flatMap(([file, markdown]) =>
30 + [...markdown.matchAll(/```bash\n([\s\S]*?)```/g)].flatMap(block =>
31 + (block[1] ?? '')
32 + .split('\n')
33 + .filter(line => line.startsWith('composio '))
34 + .map(line => ({ file, line, words: leadingWords(line) }))
35 + )
36 + );
37 +};
38 +
39 +describe('skill: every command it teaches exists', () => {
40 + let rig: CliRig;
41 + const unknown: Array<{ file: string; line: string; error: string }> = [];
42 + const commands = taughtCommands();
43 +
44 + beforeAll(async () => {
45 + rig = await startCliRig({ label: 'skill-commands' });
46 + const paths = new Map(commands.map(command => [command.words.join(' '), command]));
47 + for (const [, command] of paths) {
48 + // `composio help` rejects an argument and an unknown subcommand differently.
49 + // Trailing words that are arguments (`link gmail`) are dropped one at a
50 + // time; a word that is not a subcommand stops the search. `<path> --help`
51 + // cannot be used: it prints the parent's help and exits 0 for both.
52 + for (let length = command.words.length; length > 0; length -= 1) {
53 + const result = await rig.run(`composio help ${command.words.slice(0, length).join(' ')}`);
54 + if (result.exitCode === 0) break;
55 + if (!result.stderr.includes('Unexpected positional argument')) {
56 + const error = /Unknown subcommand[^\n]*/.exec(result.stderr)?.[0] ?? result.stderr.trim();
57 + unknown.push({ file: command.file, line: command.line, error });
58 + break;
59 + }
60 + }
61 + }
62 + }, RIG_TIMEOUT_MS);
63 +
64 + afterAll(async () => {
65 + await rig.stop();
66 + });
67 +
68 + it('finds command lines in the skill and its references', () => {
69 + expect(commands.length).toBeGreaterThan(20);
70 + expect(new Set(commands.map(command => command.file))).toEqual(
71 + new Set(['SKILL.md', 'composio-dev.md', 'power-user-examples.md', 'troubleshooting.md'])
72 + );
73 + });
74 +
75 + it('SKILL.md teaches only commands the binary has', () => {
76 + expect(unknown.filter(command => command.file === 'SKILL.md')).toEqual([]);
77 + });
78 +
79 + it('the references teach only commands the binary has', () => {
80 + expect(
81 + unknown
82 + .filter(command => command.file !== 'SKILL.md')
83 + .map(command => `${command.file}: ${command.line}`)
84 + ).toEqual([]);
85 + });
86 +});
ts/e2e-tests/cli/agent-journeys/src/adapters.ts new
+31
@@ -0,0 +1,31 @@
1 +import type { ModelAdapter } from './agent';
2 +import { anthropicAdapter } from './anthropic-adapter';
3 +import type { Provider } from './fixtures';
4 +import { openaiAdapter } from './openai-adapter';
5 +import type { AdapterInputs } from './run-journey';
6 +
7 +/**
8 + * Per-turn output cap. Enough for a command and a sentence around it, with
9 + * room for the thinking current models cannot switch off.
10 + */
11 +export const MAX_OUTPUT_TOKENS = 2048;
12 +
13 +/**
14 + * The key the adapters send. It only ever reaches the local aimock: in replay
15 + * nothing is forwarded, and in record mode aimock swaps a key with this prefix
16 + * for the real one on the way out.
17 + */
18 +export const PLACEHOLDER_API_KEY = 'sk-aimock-composio-e2e';
19 +
20 +export const adapterFor =
21 + (provider: Provider, model: string, baseURL: string) =>
22 + (inputs: AdapterInputs): ModelAdapter => {
23 + const options = {
24 + baseURL,
25 + apiKey: PLACEHOLDER_API_KEY,
26 + model,
27 + maxTokens: MAX_OUTPUT_TOKENS,
28 + ...inputs,
29 + };
30 + return provider === 'anthropic' ? anthropicAdapter(options) : openaiAdapter(options);
31 + };
ts/e2e-tests/cli/agent-journeys/src/agent.ts new
+136
@@ -0,0 +1,136 @@
1 +/**
2 + * A minimal agent: one model, one tool (`shell`), a loop that runs each
3 + * command the model asks for and hands back what it printed.
4 + *
5 + * It exists to drive the CLI the way a coding agent does, so it adds nothing
6 + * of its own: no planning prompt, no retries, no command rewriting. The one
7 + * other voice is the person the agent works for, who may answer when the model
8 + * stops to wait for them.
9 + */
10 +
11 +import type { Usage } from './usage';
12 +
13 +export interface ShellCall {
14 + /** The provider's id for the tool call. */
15 + readonly id: string;
16 + readonly command: string;
17 +}
18 +
19 +export interface ShellOutcome {
20 + readonly exitCode: number;
21 + readonly stdout: string;
22 + readonly stderr: string;
23 +}
24 +
25 +/** Why the loop ended. */
26 +export type StopReason =
27 + /** The model answered without asking for another command. */
28 + | 'finished'
29 + /** The turn limit was reached while the model still wanted to run commands. */
30 + | 'turn-limit'
31 + /** The model stopped for a reason of its own (`max_tokens`, `refusal`, …). */
32 + | { readonly provider: string };
33 +
34 +export interface ModelTurn {
35 + readonly text: string;
36 + readonly calls: ReadonlyArray<ShellCall>;
37 + /** Set when the provider reports a stop the loop cannot continue from. */
38 + readonly stopped?: string;
39 +}
40 +
41 +/** One provider's wire format. It owns the conversation it sends. */
42 +export interface ModelAdapter {
43 + readonly provider: 'anthropic' | 'openai';
44 + readonly model: string;
45 + /** Asks the model for its next turn, given everything recorded so far. */
46 + next(): Promise<ModelTurn>;
47 + /** Records the outcomes of the calls of the turn `next()` just returned. */
48 + record(results: ReadonlyArray<{ call: ShellCall; payload: string }>): void;
49 + /** Adds a message from the person, after a turn that asked for no command. */
50 + say(message: string): void;
51 + /** Tokens and cost so far, for an adapter that talks to a real provider. */
52 + usage?(): Usage;
53 +}
54 +
55 +export interface ExecutedCommand extends ShellOutcome {
56 + readonly command: string;
57 + /** The exact text the model received for this command. */
58 + readonly payload: string;
59 +}
60 +
61 +export interface Transcript {
62 + readonly commands: ReadonlyArray<ExecutedCommand>;
63 + /** The model's text, per turn, in order. */
64 + readonly messages: ReadonlyArray<string>;
65 + /** What the person answered, and after which model turn. */
66 + readonly replies: ReadonlyArray<{ readonly afterModelTurn: number; readonly text: string }>;
67 + readonly stop: StopReason;
68 + readonly modelTurns: number;
69 +}
70 +
71 +export const SHELL_TOOL = {
72 + name: 'shell',
73 + description:
74 + "Run a command line with sh in the user's terminal. Returns the exit code, stdout, and stderr.",
75 + parameters: {
76 + type: 'object' as const,
77 + properties: {
78 + command: { type: 'string', description: 'The command line to run.' },
79 + },
80 + required: ['command'],
81 + additionalProperties: false,
82 + },
83 +};
84 +
85 +/** The tool result as the model reads it. `exit_code` leads so a fixture can gate on it. */
86 +export const toolPayload = (outcome: ShellOutcome): string =>
87 + JSON.stringify({ exit_code: outcome.exitCode, stdout: outcome.stdout, stderr: outcome.stderr });
88 +
89 +export const runAgent = async (options: {
90 + adapter: ModelAdapter;
91 + /** Runs a command and returns its output, already scrubbed for the model. */
92 + execute: (command: string) => Promise<ShellOutcome>;
93 + /**
94 + * Hands the person what the model wrote in each turn, which may be nothing.
95 + * `waiting` is true when the model asked for no command; a returned string
96 + * is their reply, and the loop goes on with it.
97 + */
98 + hear?: (message: string, waiting: boolean) => string | undefined;
99 + /** Hard cap on model turns, so a looping model cannot run forever. */
100 + maxTurns: number;
101 +}): Promise<Transcript> => {
102 + const commands: ExecutedCommand[] = [];
103 + const messages: string[] = [];
104 + const replies: Array<{ afterModelTurn: number; text: string }> = [];
105 +
106 + for (let turn = 1; turn <= options.maxTurns; turn += 1) {
107 + const { text, calls, stopped } = await options.adapter.next();
108 + if (text !== '') messages.push(text);
109 + if (stopped !== undefined) {
110 + return { commands, messages, replies, stop: { provider: stopped }, modelTurns: turn };
111 + }
112 + const waiting = calls.length === 0;
113 + // The person reads the text before the turn's commands run, as they would in a chat.
114 + const reply = options.hear?.(text, waiting);
115 + if (waiting) {
116 + if (reply === undefined) {
117 + return { commands, messages, replies, stop: 'finished', modelTurns: turn };
118 + }
119 + replies.push({ afterModelTurn: turn, text: reply });
120 + options.adapter.say(reply);
121 + continue;
122 + }
123 +
124 + const results: Array<{ call: ShellCall; payload: string }> = [];
125 + // One at a time: the commands share a home directory and a backend.
126 + for (const call of calls) {
127 + const outcome = await options.execute(call.command);
128 + const payload = toolPayload(outcome);
129 + commands.push({ command: call.command, ...outcome, payload });
130 + results.push({ call, payload });
131 + }
132 + options.adapter.record(results);
133 + }
134 +
135 + return { commands, messages, replies, stop: 'turn-limit', modelTurns: options.maxTurns };
136 +};
ts/e2e-tests/cli/agent-journeys/src/anthropic-adapter.ts new
+69
@@ -0,0 +1,69 @@
1 +/**
2 + * Anthropic Messages API adapter.
3 + */
4 +
5 +import Anthropic from '@anthropic-ai/sdk';
6 +import { z } from 'zod';
7 +import { SHELL_TOOL, type ModelAdapter, type ModelTurn } from './agent';
8 +
9 +const ShellInput = z.object({ command: z.string() });
10 +
11 +export const anthropicAdapter = (options: {
12 + baseURL: string;
13 + apiKey: string;
14 + model: string;
15 + system: string;
16 + prompt: string;
17 + maxTokens: number;
18 +}): ModelAdapter => {
19 + // No retries: a miss in strict replay, or a live failure while recording, must surface once.
20 + const client = new Anthropic({ baseURL: options.baseURL, apiKey: options.apiKey, maxRetries: 0 });
21 + const messages: Anthropic.MessageParam[] = [{ role: 'user', content: options.prompt }];
22 +
23 + return {
24 + provider: 'anthropic',
25 + model: options.model,
26 + next: async (): Promise<ModelTurn> => {
27 + const response = await client.messages.create({
28 + model: options.model,
29 + max_tokens: options.maxTokens,
30 + // The run without the skill has no system prompt at all.
31 + ...(options.system === '' ? {} : { system: options.system }),
32 + tools: [
33 + {
34 + name: SHELL_TOOL.name,
35 + description: SHELL_TOOL.description,
36 + input_schema: SHELL_TOOL.parameters,
37 + },
38 + ],
39 + messages,
40 + });
41 + // The whole content goes back unchanged, thinking blocks included.
42 + messages.push({ role: 'assistant', content: response.content });
43 +
44 + const text = response.content
45 + .filter(block => block.type === 'text')
46 + .map(block => block.text)
47 + .join('\n');
48 + const calls = response.content
49 + .filter(block => block.type === 'tool_use')
50 + .map(block => ({ id: block.id, command: ShellInput.parse(block.input).command }));
51 +
52 + const continues = response.stop_reason === 'tool_use' || response.stop_reason === 'end_turn';
53 + return { text, calls, stopped: continues ? undefined : (response.stop_reason ?? 'unknown') };
54 + },
55 + record: results => {
56 + messages.push({
57 + role: 'user',
58 + content: results.map(({ call, payload }) => ({
59 + type: 'tool_result' as const,
60 + tool_use_id: call.id,
61 + content: payload,
62 + })),
63 + });
64 + },
65 + say: message => {
66 + messages.push({ role: 'user', content: message });
67 + },
68 + };
69 +};
ts/e2e-tests/cli/agent-journeys/src/fixtures.ts new
+49
@@ -0,0 +1,49 @@
1 +/**
2 + * Fixture files: one per journey, provider, and model.
3 + *
4 + * fixtures/<journey>.<provider>.<model>.json
5 + *
6 + * The model segment of a hand-authored file is `hand-authored`; no real model
7 + * produced it.
8 + */
9 +
10 +import { readdirSync } from 'node:fs';
11 +import { join } from 'node:path';
12 +
13 +export const FIXTURE_DIRECTORY = join(import.meta.dir, '..', 'fixtures');
14 +export const HAND_AUTHORED = 'hand-authored';
15 +
16 +export type Provider = 'anthropic' | 'openai';
17 +
18 +export interface FixtureFile {
19 + readonly journeyId: string;
20 + readonly provider: Provider;
21 + readonly model: string;
22 + readonly path: string;
23 +}
24 +
25 +export const fixturePath = (journeyId: string, provider: Provider, model: string): string =>
26 + join(FIXTURE_DIRECTORY, `${journeyId}.${provider}.${model}.json`);
27 +
28 +const FILE_NAME =
29 + /^(?<journeyId>[a-z0-9-]+)\.(?<provider>anthropic|openai)\.(?<model>[a-z0-9.-]+)\.json$/;
30 +
31 +/** Every file in the fixture directory. A file the name pattern does not explain is an error. */
32 +export const listFixtures = (): FixtureFile[] =>
33 + readdirSync(FIXTURE_DIRECTORY)
34 + .sort()
35 + .map(name => {
36 + const groups = FILE_NAME.exec(name)?.groups;
37 + if (!groups?.journeyId || !groups.model) {
38 + // Skipping it would drop a journey's coverage without a failing test.
39 + throw new Error(
40 + `fixtures/${name} is not named <journey>.<anthropic|openai>.<model>.json; rename or remove it.`
41 + );
42 + }
43 + return {
44 + journeyId: groups.journeyId,
45 + provider: groups.provider === 'anthropic' ? ('anthropic' as const) : ('openai' as const),
46 + model: groups.model,
47 + path: join(FIXTURE_DIRECTORY, name),
48 + };
49 + });
ts/e2e-tests/cli/agent-journeys/src/journeys.ts new
+526
@@ -0,0 +1,526 @@
1 +/**
2 + * The agent journeys: a user prompt, the state the CLI starts in, and what
3 + * must be true afterwards.
4 + *
5 + * A check reads the commands that ran, what they printed, and what the fake
6 + * backend received. None of them reads the model's prose except to see that it
7 + * stopped: a model can say "done" after every command failed.
8 + */
9 +
10 +import {
11 + CONSUMER_PROJECT,
12 + CONSUMER_USER_ID,
13 + DEVELOPER_PROJECT,
14 + GLOBAL_TEST_USER_ID,
15 + MEMBER,
16 + ORG,
17 + type JournalEntry,
18 + type World,
19 +} from '@e2e-tests/cli-fake-backend';
20 +import type { CliRig } from '@e2e-tests/cli-fake-backend/rig';
21 +import { z } from 'zod';
22 +import type { ExecutedCommand, ShellOutcome, Transcript } from './agent';
23 +import type { Usage } from './usage';
24 +
25 +/** What a journey is graded on: what ran, what it printed, and what it changed. */
26 +export interface Evidence {
27 + readonly transcript: Transcript;
28 + /** The fake's state once the model had stopped. */
29 + readonly world: World;
30 + /** What the fake received between the first and the last model turn. */
31 + readonly requests: ReadonlyArray<JournalEntry>;
32 + /** What the fake received while each of `transcript.commands` ran, in the same order. */
33 + readonly commandRequests: ReadonlyArray<ReadonlyArray<JournalEntry>>;
34 + /** Ids of the connections the journey's setup left behind, before the first model turn. */
35 + readonly startingConnections: ReadonlyArray<string>;
36 + /**
37 + * `composio whoami` in a new process, once the model had stopped: what the
38 + * next command on this machine would find. Its requests are not in `requests`.
39 + */
40 + readonly whoamiAfter: ShellOutcome;
41 +}
42 +
43 +export interface JourneyRun extends Evidence {
44 + readonly rig: CliRig;
45 + /** What the run cost, when the agent reports it. */
46 + readonly usage?: Usage;
47 +}
48 +
49 +export interface Check {
50 + readonly name: string;
51 + readonly ok: boolean;
52 + /** What was seen instead, for the failure message. */
53 + readonly seen?: string;
54 +}
55 +
56 +export interface Journey {
57 + readonly id: string;
58 + readonly title: string;
59 + readonly prompt: string;
60 + /** Directory the agent's shell starts in; defaults to the home directory. */
61 + readonly cwd?: string;
62 + readonly setup: (rig: CliRig) => Promise<void>;
63 + /**
64 + * What must hold however the model got there. These grade what the run
65 + * changed: what was executed, connected, and stored. A run that reached the
66 + * goal after trying the wrong thing still passes.
67 + */
68 + readonly checks: (run: Evidence) => ReadonlyArray<Check>;
69 + /**
70 + * What a confused agent does on the way: a command on the other tree, a
71 + * request to the other project. `ok: false` means the run took the detour.
72 + * Reported next to the checks and never a failure, so a wrong-tree rate can
73 + * be read separately from the success rate.
74 + */
75 + readonly detours?: (run: Evidence) => ReadonlyArray<Check>;
76 + /**
77 + * What must hold on the hand-authored path only: how the CLI answered the
78 + * specific commands that path scripts.
79 + */
80 + readonly scriptedChecks?: (run: Evidence) => ReadonlyArray<Check>;
81 +}
82 +
83 +export const DEV_DIRECTORY = '/work/app';
84 +
85 +const check = (name: string, ok: boolean, seen?: unknown): Check => ({
86 + name,
87 + ok,
88 + seen:
89 + ok || seen === undefined ? undefined : typeof seen === 'string' ? seen : JSON.stringify(seen),
90 +});
91 +
92 +const isComposio = (command: ExecutedCommand, pattern: RegExp) => pattern.test(command.command);
93 +// Where a `composio` invocation can start in a command line: at its start,
94 +// after a separator, or after the directory of a binary named by its path.
95 +const DEV_TREE = /(^|[;&|(\s/])composio\s+dev(\s|$)/;
96 +const ROOT_EXECUTE = /(^|[;&|(\s/])composio\s+execute\s/;
97 +/** A root `composio execute` that runs the tool, not one that only describes it. */
98 +const ROOT_EXECUTE_RUNS =
99 + /(^|[;&|(\s/])composio\s+execute\s(?![^;&|]*--(?:get-schema|dry-run|help)\b)/;
100 +const ROOT_LINK = /(^|[;&|(\s/])composio\s+link(\s|$)/;
101 +const RUN = /(^|[;&|(\s/])composio\s+run\s/g;
102 +const PLAYGROUND_EXECUTE = /composio\s+dev\s+(?:--\S+\s+\S+\s+)*playground-execute\s/;
103 +
104 +const SEND = 'SLACK_SEND_MESSAGE';
105 +
106 +/** Every tool execution that completed, whichever route ran it. */
107 +const executions = (requests: ReadonlyArray<JournalEntry>) =>
108 + requests.flatMap(entry =>
109 + // A Dashboard execution answers 200 even when the tool was refused; the effect says it ran.
110 + entry.effect === 'tool-executed' && entry.tool ? [{ entry, tool: entry.tool }] : []
111 + );
112 +
113 +const describeExecutions = (requests: ReadonlyArray<JournalEntry>) =>
114 + executions(requests).map(({ entry, tool }) => ({
115 + tool: tool.slug,
116 + project: entry.project,
117 + user: entry.userId,
118 + arguments: tool.arguments,
119 + }));
120 +
121 +/** `general` and `#general` name the same channel. */
122 +const channelName = (value: unknown): string =>
123 + typeof value === 'string' ? value.trim().replace(/^#/, '') : '';
124 +
125 +/**
126 + * The person asked for one message. So one `SLACK_SEND_MESSAGE` completed in
127 + * the whole run, on any project and through any route, and it is that
128 + * message: a second send, a send on the other project, another channel, and
129 + * other words each fail a check of their own. A lookup beside it changes
130 + * nothing and is let through; any other tool that changes something is not.
131 + */
132 +const sentOnce = (
133 + run: Evidence,
134 + expected: {
135 + text: string;
136 + project: string;
137 + userId: string;
138 + /**
139 + * Whether the request itself must name the project. A developer command
140 + * sends it; a consumer execution names none and the Dashboard resolves it.
141 + */
142 + scope: 'named' | 'resolved';
143 + }
144 +): ReadonlyArray<Check> => {
145 + const all = executions(run.requests);
146 + const sends = all.filter(({ tool }) => tool.slug === SEND);
147 + const otherChanges = all.filter(
148 + ({ tool }) => tool.slug !== SEND && run.world.findTool(tool.slug)?.readOnly !== true
149 + );
150 + const seen = describeExecutions(run.requests);
151 + const everySend = (holds: (execution: (typeof sends)[number]) => boolean) =>
152 + sends.length > 0 && sends.every(holds);
153 + return [
154 + check(`${SEND} completed once in the whole run`, sends.length === 1, seen),
155 + check(
156 + `it ran on the ${expected.project} project, as ${expected.userId}`,
157 + everySend(
158 + ({ entry }) =>
159 + entry.project === expected.project &&
160 + entry.userId === expected.userId &&
161 + (entry.requestedProject === expected.project ||
162 + (expected.scope === 'resolved' && entry.requestedProject === null))
163 + ),
164 + seen
165 + ),
166 + check(
167 + 'it went to #general',
168 + everySend(({ tool }) => channelName(tool.arguments.channel) === 'general'),
169 + seen
170 + ),
171 + check(
172 + `its text was exactly "${expected.text}"`,
173 + everySend(
174 + ({ tool }) =>
175 + typeof tool.arguments.text === 'string' && tool.arguments.text.trim() === expected.text
176 + ),
177 + seen
178 + ),
179 + check('no other tool that changes something ran', otherChanges.length === 0, seen),
180 + ];
181 +};
182 +
183 +/** A connection made during the run on a project the task never involved. */
184 +const connectedOnlyOn = (run: Evidence, project: string): Check => {
185 + const stray = run.world.connections.filter(
186 + connection =>
187 + connection.status === 'ACTIVE' &&
188 + connection.projectLabel !== project &&
189 + !run.startingConnections.includes(connection.id)
190 + );
191 + return check(
192 + `no account was connected outside the ${project} project`,
193 + stray.length === 0,
194 + stray.map(connection => `${connection.toolkit} on ${connection.projectLabel}`)
195 + );
196 +};
197 +
198 +const ranNo = (run: Evidence, name: string, pattern: RegExp): Check => {
199 + const matching = run.transcript.commands.filter(command => isComposio(command, pattern));
200 + return check(
201 + name,
202 + matching.length === 0,
203 + matching.map(command => command.command)
204 + );
205 +};
206 +
207 +/** What `composio whoami` prints on stdout off a terminal, as far as a login is concerned. */
208 +const Whoami = z.looseObject({
209 + account_type: z.string(),
210 + email: z.string(),
211 + current_org_name: z.string(),
212 +});
213 +
214 +/** The login a `composio whoami` reported, or `undefined` when it reported none. */
215 +const loginReportedBy = (outcome: ShellOutcome) => {
216 + if (outcome.exitCode !== 0) return undefined;
217 + try {
218 + return Whoami.safeParse(JSON.parse(outcome.stdout)).data;
219 + } catch {
220 + return undefined;
221 + }
222 +};
223 +
224 +const lastCommandSucceeded = (run: Evidence, pattern: RegExp): boolean => {
225 + const matching = run.transcript.commands.filter(command => isComposio(command, pattern));
226 + return matching.at(-1)?.exitCode === 0;
227 +};
228 +
229 +const stoppedByItself = (run: Evidence): Check =>
230 + check(
231 + 'the model finished by itself, within the turn limit',
232 + run.transcript.stop === 'finished',
233 + run.transcript.stop
234 + );
235 +
236 +const loginRig = async (rig: CliRig): Promise<void> => {
237 + await rig.login();
238 +};
239 +
240 +const developerRig = async (rig: CliRig): Promise<void> => {
241 + await rig.login();
242 + const result = await rig.run(
243 + `mkdir -p ${DEV_DIRECTORY} && cd ${DEV_DIRECTORY} && composio dev init --yes --no-browser`
244 + );
245 + if (result.exitCode !== 0) throw new Error(`composio dev init failed: ${result.stderr}`);
246 +};
247 +
248 +/**
249 + * Requests that acted on a project other than the one the task is about.
250 + * Organization routes (`_internal.org.*`) are left out: the CLI reads the
251 + * org's settings on its own, under Claude Code on every command, and the
252 + * project the fake journals for them is the key's default, not a choice.
253 + */
254 +const requestsOutside = (run: Evidence, project: string): Check => {
255 + const outside = run.requests.filter(
256 + entry =>
257 + (entry.role === 'backend' || entry.role === 'dashboard') &&
258 + !entry.route?.startsWith('_internal.org.') &&
259 + entry.project !== null &&
260 + entry.project !== project
261 + );
262 + return check(
263 + `every request that named a project named the ${project} project`,
264 + outside.length === 0,
265 + [...new Set(outside.map(entry => `${entry.route} on ${entry.project}`))]
266 + );
267 +};
268 +
269 +/** What a developer task looks like when the agent reaches for the root tree. */
270 +const rootTreeDetours = (run: Evidence): ReadonlyArray<Check> => [
271 + ranNo(run, 'no root `composio execute` tried to run the tool', ROOT_EXECUTE_RUNS),
272 + ranNo(run, 'no root `composio link` ran', ROOT_LINK),
273 + check(
274 + 'nothing was executed or linked on the consumer project, completed or not',
275 + !run.requests.some(
276 + entry =>
277 + entry.project === CONSUMER_PROJECT.label &&
278 + (entry.route === 'dashboard.execute' ||
279 + entry.route === 'toolRouter.session.execute' ||
280 + entry.route === 'toolRouter.session.link')
281 + )
282 + ),
283 +];
284 +
285 +/** What a consumer task looks like when the agent reaches for the dev tree. */
286 +const devTreeDetours = (run: Evidence): ReadonlyArray<Check> => [
287 + ranNo(run, 'no `composio dev` command ran', DEV_TREE),
288 + requestsOutside(run, CONSUMER_PROJECT.label),
289 +];
290 +
291 +export const JOURNEYS: ReadonlyArray<Journey> = [
292 + {
293 + id: 'consumer-slack-message',
294 + title: 'consumer task: send a Slack message from an unconnected account',
295 + prompt: 'Send a Slack message to the #general channel that says: deploy finished',
296 + setup: loginRig,
297 + checks: run => [
298 + stoppedByItself(run),
299 + check(
300 + 'the last `composio execute` exited 0',
301 + lastCommandSucceeded(run, ROOT_EXECUTE),
302 + run.transcript.commands.map(command => `${command.exitCode} ${command.command}`)
303 + ),
304 + ...sentOnce(run, {
305 + text: 'deploy finished',
306 + project: CONSUMER_PROJECT.label,
307 + userId: CONSUMER_USER_ID,
308 + scope: 'resolved',
309 + }),
310 + check(
311 + 'the account was linked on the consumer project',
312 + run.world.activeConnection(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack') !== undefined
313 + ),
314 + connectedOnlyOn(run, CONSUMER_PROJECT.label),
315 + ],
316 + detours: devTreeDetours,
317 + },
318 + {
319 + id: 'developer-auth-configs-and-execute',
320 + title: 'developer task: list auth configs and run a tool as the test user',
321 + prompt:
322 + 'I am building an app on Composio. In my developer project, list the auth configs, then execute SLACK_SEND_MESSAGE as my test user to post "hello from dev" in #general.',
323 + cwd: DEV_DIRECTORY,
324 + setup: async rig => {
325 + await developerRig(rig);
326 + rig.fake.world.connect(DEVELOPER_PROJECT.label, GLOBAL_TEST_USER_ID, 'slack');
327 + },
328 + checks: run => [
329 + stoppedByItself(run),
330 + check(
331 + '`composio dev auth-configs list` ran and exited 0',
332 + lastCommandSucceeded(run, /composio\s+dev\s+auth-configs\s+list/),
333 + run.transcript.commands.map(command => `${command.exitCode} ${command.command}`)
334 + ),
335 + check(
336 + 'the auth configs were read from the developer project',
337 + run.requests.some(
338 + entry =>
339 + entry.route === 'authConfigs.list' &&
340 + entry.status === 200 &&
341 + entry.project === DEVELOPER_PROJECT.label
342 + )
343 + ),
344 + ...sentOnce(run, {
345 + text: 'hello from dev',
346 + project: DEVELOPER_PROJECT.label,
347 + userId: GLOBAL_TEST_USER_ID,
348 + scope: 'named',
349 + }),
350 + check(
351 + 'it ran through `composio dev playground-execute`',
352 + lastCommandSucceeded(run, PLAYGROUND_EXECUTE)
353 + ),
354 + connectedOnlyOn(run, DEVELOPER_PROJECT.label),
355 + ],
356 + detours: rootTreeDetours,
357 + },
358 + {
359 + id: 'developer-execute-unconnected',
360 + title: 'tree confusion: a developer task that needs a connection first',
361 + prompt:
362 + 'In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post "hello from dev" in #general.',
363 + cwd: DEV_DIRECTORY,
364 + setup: developerRig,
365 + checks: run => [
366 + stoppedByItself(run),
367 + ...sentOnce(run, {
368 + text: 'hello from dev',
369 + project: DEVELOPER_PROJECT.label,
370 + userId: GLOBAL_TEST_USER_ID,
371 + scope: 'named',
372 + }),
373 + check(
374 + 'the account was linked on the developer project for the test user',
375 + run.world.activeConnection(DEVELOPER_PROJECT.label, GLOBAL_TEST_USER_ID, 'slack') !==
376 + undefined
377 + ),
378 + // The confusion this journey exists for: `composio link` connects the
379 + // consumer user, the person approves it, and the developer task is no nearer.
380 + connectedOnlyOn(run, DEVELOPER_PROJECT.label),
381 + check(
382 + 'the last `composio dev playground-execute` exited 0',
383 + lastCommandSucceeded(run, PLAYGROUND_EXECUTE),
384 + run.transcript.commands.map(command => `${command.exitCode} ${command.command}`)
385 + ),
386 + ],
387 + detours: rootTreeDetours,
388 + scriptedChecks: run => {
389 + const [wrongTree, firstAttempt] = run.transcript.commands;
390 + return [
391 + // A command from the root tree's vocabulary on the dev tree: recoverable.
392 + check(
393 + '`composio dev execute` exits 1 with nothing on stdout',
394 + wrongTree?.exitCode === 1 && wrongTree.stdout === '',
395 + wrongTree
396 + ),
397 + check(
398 + 'its stderr names the unknown subcommand',
399 + wrongTree?.stderr.includes('Unknown subcommand "execute" for "composio dev"') === true,
400 + wrongTree?.stderr.slice(-300)
401 + ),
402 + check(
403 + 'the help printed with it lists the real command, and both survive truncation',
404 + wrongTree?.payload.includes('playground-execute') === true &&
405 + wrongTree.payload.includes('Unknown subcommand'),
406 + wrongTree?.payload.length
407 + ),
408 + // The not-connected message names the link of the tree it was raised
409 + // on. A root-tree link would connect the consumer user and fix nothing.
410 + check(
411 + 'the unconnected `playground-execute` names the dev-tree link',
412 + firstAttempt?.exitCode === 1 &&
413 + firstAttempt.stdout.includes('Run `composio dev connected-accounts link slack`') &&
414 + !firstAttempt.stdout.includes('`composio link slack`'),
415 + firstAttempt?.stdout
416 + ),
417 + ];
418 + },
419 + },
420 + {
421 + id: 'first-run-login',
422 + title: 'first run: find out the CLI is not logged in, and log in',
423 + prompt: 'Check whether the Composio CLI on this machine is logged in. If it is not, log in.',
424 + setup: async () => {},
425 + checks: run => {
426 + const whoami = run.transcript.commands.filter(command =>
427 + isComposio(command, /composio\s+whoami/)
428 + );
429 + const stored = loginReportedBy(run.whoamiAfter);
430 + return [
431 + stoppedByItself(run),
432 + // By what it printed as well as by its exit code: an agent that pipes
433 + // `whoami` into `head` sees the message with exit code 0.
434 + check(
435 + 'the first `composio whoami` reported no login',
436 + whoami[0] !== undefined &&
437 + (whoami[0].exitCode === 1 ||
438 + /not logged in/i.test(`${whoami[0].stdout}\n${whoami[0].stderr}`)),
439 + whoami.map(command => `${command.exitCode} ${command.stdout.slice(0, 120)}`)
440 + ),
441 + check(
442 + 'exactly one CLI login session was created, and it was linked',
443 + run.world.loginSessions.length === 1 && run.world.loginSessions[0]?.status === 'linked',
444 + run.world.loginSessions
445 + ),
446 + check(
447 + 'the login completed: the user API key was fetched for the linked session',
448 + run.requests.some(
449 + entry => entry.route === '_internal.cli.getSession' && entry.status === 200
450 + ) &&
451 + run.requests.some(
452 + entry => entry.route === 'auth.session.retrieveInfo' && entry.keyKnown
453 + )
454 + ),
455 + // The requests above happen in the process that logged in. This is the
456 + // next process: it finds the login only if it was stored.
457 + check(
458 + 'the login was stored: a new `composio whoami` exits 0 as the person, in their org',
459 + stored?.account_type === 'human' &&
460 + stored.email === MEMBER.email &&
461 + stored.current_org_name === ORG.name,
462 + run.whoamiAfter
463 + ),
464 + check(
465 + 'no agent account was signed up in place of the person',
466 + !run.requests.some(entry => entry.route === 'agents.signup')
467 + ),
468 + ];
469 + },
470 + },
471 + {
472 + id: 'run-inline-script',
473 + title: '`composio run`: one inline script that executes a tool',
474 + prompt:
475 + 'Use a single `composio run` inline script to send "build passed" to #general on Slack and print the timestamp of the sent message.',
476 + setup: async rig => {
477 + await rig.login();
478 + rig.fake.world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
479 + },
480 + checks: run => {
481 + // The commands during which a tool execution completed.
482 + const sending = run.transcript.commands.filter(
483 + (_, index) => executions(run.commandRequests[index] ?? []).length > 0
484 + );
485 + return [
486 + stoppedByItself(run),
487 + ...sentOnce(run, {
488 + text: 'build passed',
489 + project: CONSUMER_PROJECT.label,
490 + userId: CONSUMER_USER_ID,
491 + scope: 'resolved',
492 + }),
493 + // The send and the timestamp belong to one command, and that command
494 + // is the script: not `composio execute` followed by a script that
495 + // prints what it was told. This reads the command's text, so it holds
496 + // for the ways a shell line names `composio`, not for an alias or a
497 + // wrapper script.
498 + check(
499 + 'the message was sent by a command that is one `composio run`, which exited 0 and printed its timestamp',
500 + sending.length === 1 &&
501 + sending.every(
502 + command =>
503 + (command.command.match(RUN) ?? []).length === 1 &&
504 + !isComposio(command, ROOT_EXECUTE_RUNS) &&
505 + !isComposio(command, PLAYGROUND_EXECUTE) &&
506 + command.exitCode === 0 &&
507 + command.stdout.includes('1767225600.000100')
508 + ),
509 + sending.map(command => `${command.exitCode} ${command.command.slice(0, 160)}`)
510 + ),
511 + connectedOnlyOn(run, CONSUMER_PROJECT.label),
512 + ];
513 + },
514 + detours: devTreeDetours,
515 + },
516 +];
517 +
518 +export const journeyById = (id: string): Journey => {
519 + const journey = JOURNEYS.find(candidate => candidate.id === id);
520 + if (!journey) {
521 + throw new Error(
522 + `Unknown journey "${id}". Known: ${JOURNEYS.map(candidate => candidate.id).join(', ')}`
523 + );
524 + }
525 + return journey;
526 +};
ts/e2e-tests/cli/agent-journeys/src/openai-adapter.ts new
+65
@@ -0,0 +1,65 @@
1 +/**
2 + * OpenAI Chat Completions adapter.
3 + */
4 +
5 +import OpenAI from 'openai';
6 +import { z } from 'zod';
7 +import { SHELL_TOOL, type ModelAdapter, type ModelTurn } from './agent';
8 +
9 +const ShellInput = z.object({ command: z.string() });
10 +
11 +export const openaiAdapter = (options: {
12 + baseURL: string;
13 + apiKey: string;
14 + model: string;
15 + system: string;
16 + prompt: string;
17 + maxTokens: number;
18 +}): ModelAdapter => {
19 + const client = new OpenAI({
20 + baseURL: `${options.baseURL}/v1`,
21 + apiKey: options.apiKey,
22 + maxRetries: 0,
23 + });
24 + const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
25 + { role: 'system', content: options.system },
26 + { role: 'user', content: options.prompt },
27 + ];
28 +
29 + return {
30 + provider: 'openai',
31 + model: options.model,
32 + next: async (): Promise<ModelTurn> => {
33 + const response = await client.chat.completions.create({
34 + model: options.model,
35 + max_completion_tokens: options.maxTokens,
36 + tools: [{ type: 'function', function: SHELL_TOOL }],
37 + messages,
38 + });
39 + const choice = response.choices[0];
40 + if (!choice) return { text: '', calls: [], stopped: 'no-choice' };
41 + messages.push(choice.message);
42 +
43 + const calls = (choice.message.tool_calls ?? [])
44 + .filter(call => call.type === 'function')
45 + .map(call => ({
46 + id: call.id,
47 + command: ShellInput.parse(JSON.parse(call.function.arguments)).command,
48 + }));
49 + const continues = choice.finish_reason === 'tool_calls' || choice.finish_reason === 'stop';
50 + return {
51 + text: choice.message.content ?? '',
52 + calls,
53 + stopped: continues ? undefined : choice.finish_reason,
54 + };
55 + },
56 + record: results => {
57 + for (const { call, payload } of results) {
58 + messages.push({ role: 'tool', tool_call_id: call.id, content: payload });
59 + }
60 + },
61 + say: message => {
62 + messages.push({ role: 'user', content: message });
63 + },
64 + };
65 +};
ts/e2e-tests/cli/agent-journeys/src/person.ts new
+106
@@ -0,0 +1,106 @@
1 +/**
2 + * The person the agent works for.
3 + *
4 + * A login or an account authorization needs them: only the owner of an account
5 + * can approve access to it, in a browser. So a journey has to model what
6 + * reaches them, or an agent that never showed them the URL would pass.
7 + *
8 + * Two things reach this person:
9 + *
10 + * - a page the CLI opens in their browser, and
11 + * - what the agent writes to them.
12 + *
13 + * They do not read the output of the agent's commands. The CLI assumes the
14 + * same: off a terminal `composio login` prints "For agents: Show the URL above
15 + * to the user".
16 + *
17 + * They approve an authorization when the page for it reaches them: the CLI
18 + * opened it, or the agent wrote its complete URL. An id on its own, or the
19 + * path under some other host, is not a page they can open. A URL written
20 + * without its scheme is not recognised either; that is a limit of this model,
21 + * not a claim about people.
22 + *
23 + * When the agent has stopped to wait, they say that they approved, once per
24 + * approval, and only while the CLI has not been told: an agent whose own
25 + * command already saw the login or the connection has nothing to wait for.
26 + * "Told" is the fake having answered a request with the approved state;
27 + * whether the answer reached the agent is not modelled.
28 + */
29 +
30 +import type { World } from '@e2e-tests/cli-fake-backend';
31 +
32 +export const PERSON_REPLY = 'Done, I approved it in my browser. Please continue.';
33 +
34 +/** Where the CLI looks for a browser on Linux. */
35 +export const BROWSER_PATH = '/usr/local/bin/xdg-open';
36 +
37 +/**
38 + * The person's browser. It loads the page it is asked to open, and on the fake
39 + * backend loading an authorization page approves it. A page behind TLS is left
40 + * alone: bash has no TLS, and the CLI opens no such page off a terminal.
41 + */
42 +export const BROWSER_SCRIPT = `#!/bin/bash
43 +url="$1"
44 +case "$url" in http://*) ;; *) exit 0 ;; esac
45 +rest="\${url#http://}"
46 +authority="\${rest%%/*}"
47 +exec 3<>"/dev/tcp/\${authority%%:*}/\${authority##*:}" || exit 0
48 +printf 'GET /%s HTTP/1.0\\r\\nHost: %s\\r\\n\\r\\n' "\${rest#*/}" "$authority" >&3
49 +cat <&3 >/dev/null
50 +`;
51 +
52 +export interface Person {
53 + /**
54 + * Hands the person a message from the agent. `waiting` is true when the
55 + * agent has stopped and expects them to act; their reply, if they have one,
56 + * is returned.
57 + */
58 + hear(message: string, waiting: boolean): string | undefined;
59 +}
60 +
61 +/** The origins of the fake, as the agent sees them in command output. */
62 +export interface PersonUrls {
63 + /** Where the login page lives: `<dashboard>?cliKey=<login session id>`. */
64 + readonly dashboard: string;
65 + /** Where an authorization page lives: `<backend>/api/v3/s/<connection id>`. */
66 + readonly backend: string;
67 +}
68 +
69 +const AUTHORIZATION_PATH = /^\/api\/v\d+(?:\.\d+)?\/s\/([^/]+)$/;
70 +
71 +/** The complete URLs in a message, without the punctuation or markup around them. */
72 +const urlsIn = (message: string): URL[] =>
73 + (message.match(/https?:\/\/[^\s<>"'`)\]]+/g) ?? []).flatMap(candidate => {
74 + const text = candidate.replace(/[.,;:!?*_]+$/, '');
75 + return URL.canParse(text) ? [new URL(text)] : [];
76 + });
77 +
78 +export const personFor = (world: World, urls: PersonUrls): Person => {
79 + const dashboard = new URL(urls.dashboard).origin;
80 + const backend = new URL(urls.backend).origin;
81 + const acknowledged = new Set<string>();
82 +
83 + return {
84 + hear: (message, waiting) => {
85 + for (const url of urlsIn(message)) {
86 + const loginId =
87 + url.origin === dashboard && url.pathname === '/' ? url.searchParams.get('cliKey') : null;
88 + if (
89 + loginId !== null &&
90 + world.loginSessions.some(
91 + session => session.id === loginId && session.status === 'pending'
92 + )
93 + ) {
94 + world.linkLoginSession(loginId);
95 + }
96 + const connectionId =
97 + url.origin === backend ? AUTHORIZATION_PATH.exec(url.pathname)?.[1] : undefined;
98 + if (connectionId !== undefined) world.authorizeConnection(connectionId);
99 + }
100 + if (!waiting) return undefined;
101 + const news = world.unreportedApprovals().filter(id => !acknowledged.has(id));
102 + for (const id of news) acknowledged.add(id);
103 + return news.length > 0 ? PERSON_REPLY : undefined;
104 + },
105 + };
106 +};
ts/e2e-tests/cli/agent-journeys/src/run-journey.ts new
+157
@@ -0,0 +1,157 @@
1 +/**
2 + * Runs one journey: a fresh rig, the skill where the agent finds it, the
3 + * journey's starting state, then an agent, and the evidence it left.
4 + *
5 + * The agent is a driver. The replay tests and `record.ts` use the minimal
6 + * loop in `agent.ts`; other agents can use the same driver contract and evidence checks.
7 + */
8 +
9 +import { startCliRig, type CliRig, type RigOptions } from '@e2e-tests/cli-fake-backend/rig';
10 +import type { JournalEntry } from '@e2e-tests/cli-fake-backend';
11 +import { runAgent, type ModelAdapter, type Transcript } from './agent';
12 +import type { Journey, JourneyRun } from './journeys';
13 +import { BROWSER_PATH, BROWSER_SCRIPT, personFor, type Person } from './person';
14 +import { scrub, SCRUBBED_BACKEND_URL, SCRUBBED_DASHBOARD_URL } from './scrub';
15 +import { buildSkill, SKILL_DIRECTORY, systemPrompt, type BuiltSkill } from './skill';
16 +import type { Usage } from './usage';
17 +
18 +/** Upper bound on model turns. The longest scripted journey takes eight. */
19 +export const MAX_TURNS = 12;
20 +
21 +/** Upper bound on one command, so a command that waits for a person cannot stall a run. */
22 +export const COMMAND_TIMEOUT_SECONDS = 90;
23 +
24 +export interface AdapterInputs {
25 + readonly system: string;
26 + readonly prompt: string;
27 +}
28 +
29 +export interface DriverContext {
30 + readonly rig: CliRig;
31 + readonly journey: Journey;
32 + readonly person: Person;
33 + /** The skill, when the driver places it. */
34 + readonly skill: BuiltSkill | undefined;
35 +}
36 +
37 +export interface DriverResult {
38 + readonly transcript: Transcript;
39 + /** The requests each command caused, in the order of `transcript.commands`. */
40 + readonly commandRequests: ReadonlyArray<ReadonlyArray<JournalEntry>>;
41 + readonly usage?: Usage;
42 +}
43 +
44 +export interface Driver {
45 + /**
46 + * Where the skill goes: the system prompt of the minimal loop, the skills
47 + * directory Claude Code reads, or nowhere (the run without the skill).
48 + */
49 + readonly skill: 'system-prompt' | 'claude-code-skills' | 'none';
50 + /**
51 + * Whether the agent reads command output with the per-run URLs and clocks
52 + * replaced. Replay needs it; a live agent reads what the CLI printed.
53 + */
54 + readonly scrubbed: boolean;
55 + readonly readOnlyMounts?: RigOptions['readOnlyMounts'];
56 + drive(context: DriverContext): Promise<DriverResult>;
57 +}
58 +
59 +/** Where Claude Code looks for user skills, under the journey's home. */
60 +export const claudeSkillDirectory = (home: string): string => `${home}/.claude/skills/composio-cli`;
61 +
62 +const placeSkill = async (rig: CliRig, directory: string, skill: BuiltSkill): Promise<void> => {
63 + await rig.journey.writeFile(`${directory}/SKILL.md`, skill.skillMarkdown);
64 + for (const [name, markdown] of Object.entries(skill.references)) {
65 + await rig.journey.writeFile(`${directory}/references/${name}`, markdown);
66 + }
67 +};
68 +
69 +export const runJourneyWith = async (journey: Journey, driver: Driver): Promise<JourneyRun> => {
70 + const rig = await startCliRig({ label: journey.id, readOnlyMounts: driver.readOnlyMounts });
71 + try {
72 + const skill = driver.skill === 'none' ? undefined : buildSkill();
73 + if (skill && driver.skill === 'system-prompt') await placeSkill(rig, SKILL_DIRECTORY, skill);
74 + if (skill && driver.skill === 'claude-code-skills') {
75 + await placeSkill(rig, claudeSkillDirectory(rig.journey.home), skill);
76 + }
77 + await rig.journey.writeFile(BROWSER_PATH, BROWSER_SCRIPT);
78 + await rig.run(`chmod +x ${BROWSER_PATH}`);
79 + // Nothing is approved by waiting: only the person approves, see `person.ts`.
80 + rig.fake.world.linkCompletesAfterPolls = null;
81 + await journey.setup(rig);
82 + // The person reads what the agent writes, so they see the URLs the agent saw.
83 + const person = personFor(
84 + rig.fake.world,
85 + driver.scrubbed
86 + ? { dashboard: SCRUBBED_DASHBOARD_URL, backend: SCRUBBED_BACKEND_URL }
87 + : { dashboard: rig.fake.docker.https, backend: rig.fake.docker.http }
88 + );
89 + const startingConnections = rig.fake.world.connections.map(connection => connection.id);
90 +
91 + const mark = rig.fake.journal.mark();
92 + const { transcript, commandRequests, usage } = await driver.drive({
93 + rig,
94 + journey,
95 + person,
96 + skill,
97 + });
98 + const requests = rig.fake.journal.since(mark);
99 + const whoami = await rig.run('composio whoami', { timeoutSeconds: COMMAND_TIMEOUT_SECONDS });
100 + return {
101 + transcript,
102 + rig,
103 + world: rig.fake.world,
104 + requests,
105 + commandRequests,
106 + startingConnections,
107 + whoamiAfter: { exitCode: whoami.exitCode, stdout: whoami.stdout, stderr: whoami.stderr },
108 + usage,
109 + };
110 + } catch (error) {
111 + await rig.stop();
112 + throw error;
113 + }
114 +};
115 +
116 +/**
117 + * The minimal loop of `agent.ts`. With `skill: 'none'` the model gets no
118 + * system prompt at all.
119 + */
120 +export const loopDriver = (
121 + makeAdapter: (inputs: AdapterInputs) => ModelAdapter,
122 + options: { skill?: 'system-prompt' | 'none'; scrubbed?: boolean } = {}
123 +): Driver => ({
124 + skill: options.skill ?? 'system-prompt',
125 + scrubbed: options.scrubbed ?? true,
126 + drive: async ({ rig, journey, person, skill }) => {
127 + const commandRequests: Array<ReadonlyArray<JournalEntry>> = [];
128 + const adapter = makeAdapter({
129 + system: skill ? systemPrompt(skill) : '',
130 + prompt: journey.prompt,
131 + });
132 + const transcript = await runAgent({
133 + adapter,
134 + maxTurns: MAX_TURNS,
135 + hear: person.hear,
136 + execute: async command => {
137 + const result = await rig.run(command, {
138 + cwd: journey.cwd,
139 + timeoutSeconds: COMMAND_TIMEOUT_SECONDS,
140 + });
141 + commandRequests.push(result.requests);
142 + const shown = (text: string) => (options.scrubbed === false ? text : scrub(text, rig.fake));
143 + return {
144 + exitCode: result.exitCode,
145 + stdout: shown(result.stdout),
146 + stderr: shown(result.stderr),
147 + };
148 + },
149 + });
150 + return { transcript, commandRequests, usage: adapter.usage?.() };
151 + },
152 +});
153 +
154 +export const runJourney = (
155 + journey: Journey,
156 + makeAdapter: (inputs: AdapterInputs) => ModelAdapter
157 +): Promise<JourneyRun> => runJourneyWith(journey, loopDriver(makeAdapter));
ts/e2e-tests/cli/agent-journeys/src/scrub.ts new
+38
@@ -0,0 +1,38 @@
1 +/**
2 + * Makes command output repeatable before a model sees it.
3 + *
4 + * A fixture is matched against the conversation, so anything that differs
5 + * between two runs of the same journey (a port, a temporary directory, a clock
6 + * reading) would make a replay miss. The fake backend already fixes ids and
7 + * timestamps; this removes what the machine adds.
8 + */
9 +
10 +import { SEEDED_AT, SEEDED_EXPIRY, type FakeComposio } from '@e2e-tests/cli-fake-backend';
11 +
12 +/** Stable stand-ins for the fake's per-run URLs. */
13 +export const SCRUBBED_BACKEND_URL = 'https://backend.composio.test';
14 +export const SCRUBBED_DASHBOARD_URL = 'https://dashboard.composio.test';
15 +
16 +const HEAD_CHARACTERS = 2500;
17 +const TAIL_CHARACTERS = 1500;
18 +const SEEDED_TIMESTAMPS = new Set([SEEDED_AT, SEEDED_EXPIRY]);
19 +
20 +/** Keeps both ends: a parse error sits under a page of help, at the very end. */
21 +const truncate = (text: string): string =>
22 + text.length <= HEAD_CHARACTERS + TAIL_CHARACTERS
23 + ? text
24 + : `${text.slice(0, HEAD_CHARACTERS)}\n[... ${text.length - HEAD_CHARACTERS - TAIL_CHARACTERS} characters omitted ...]\n${text.slice(-TAIL_CHARACTERS)}`;
25 +
26 +export const scrub = (text: string, fake: FakeComposio): string =>
27 + truncate(
28 + text
29 + .replace(/\x1b\[[0-9;]*m/g, '')
30 + .replaceAll(fake.docker.https, SCRUBBED_DASHBOARD_URL)
31 + .replaceAll(fake.docker.http, SCRUBBED_BACKEND_URL)
32 + .replace(/composio-run-artifacts-[A-Za-z0-9]+/g, 'composio-run-artifacts-SCRUBBED')
33 + .replace(/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z/g, timestamp =>
34 + SEEDED_TIMESTAMPS.has(timestamp) ? timestamp : '<timestamp>'
35 + )
36 + .replace(/\[\d{2}:\d{2}:\d{2}\.\d{3}\]/g, '[<time>]')
37 + .replace(/composio-cli-\d{4}-\d{2}-\d{2}/g, 'composio-cli-<date>')
38 + );
ts/e2e-tests/cli/agent-journeys/src/skill.ts new
+34
@@ -0,0 +1,34 @@
1 +/**
2 + * The system prompt of every journey: the `composio-cli` skill, built from its
3 + * source the way `pnpm --filter @composio/cli run build:skills` builds it.
4 + *
5 + * Rendering it here, not reading a checked-in copy, is what makes a change to
6 + * the skill reach these tests.
7 + */
8 +
9 +import {
10 + renderComposioCliSkill,
11 + renderReferenceFiles,
12 +} from '../../../../packages/cli/skills-src/composio-cli/index';
13 +
14 +/** Where the skill's files sit inside the journey container. */
15 +export const SKILL_DIRECTORY = '/skills/composio-cli';
16 +
17 +export interface BuiltSkill {
18 + /** `SKILL.md`, as an agent host loads it. */
19 + readonly skillMarkdown: string;
20 + /** `references/<name>.md`, which `SKILL.md` tells the agent to open when needed. */
21 + readonly references: Readonly<Record<string, string>>;
22 +}
23 +
24 +export const buildSkill = (): BuiltSkill => ({
25 + skillMarkdown: renderComposioCliSkill('stable'),
26 + references: renderReferenceFiles('stable'),
27 +});
28 +
29 +/**
30 + * The skill text with the one line an agent host adds when it loads a skill:
31 + * where its files are, so the relative links in `SKILL.md` can be followed.
32 + */
33 +export const systemPrompt = (skill: BuiltSkill): string =>
34 + `Base directory for this skill: ${SKILL_DIRECTORY}\n\n${skill.skillMarkdown}`;
ts/e2e-tests/cli/agent-journeys/src/usage.ts new
+7
@@ -0,0 +1,7 @@
1 +export interface Usage {
2 + readonly costUsd: number;
3 + readonly inputTokens: number;
4 + readonly outputTokens: number;
5 + readonly cacheReadTokens: number;
6 + readonly cacheWriteTokens: number;
7 +}
ts/e2e-tests/cli/agent-journeys/transcripts/01-approve-everything/consumer-slack-message.anthropic.claude-haiku-5-5.json new
+70
@@ -0,0 +1,70 @@
1 +{
2 + "journey": "consumer-slack-message",
3 + "provider": "anthropic",
4 + "model": "claude-haiku-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "Send a Slack message to the #general channel that says: deploy finished",
10 + "reachedGoal": true,
11 + "stop": "finished",
12 + "modelTurns": 5,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "the last `composio execute` exited 0",
20 + "ok": true
21 + },
22 + {
23 + "name": "the message was sent once, on the consumer project, as the consumer user",
24 + "ok": true
25 + },
26 + {
27 + "name": "the account was linked on the consumer project",
28 + "ok": true
29 + },
30 + {
31 + "name": "no `composio dev` command ran",
32 + "ok": true
33 + },
34 + {
35 + "name": "every request that named a project named the consumer project",
36 + "ok": true
37 + }
38 + ],
39 + "commands": [
40 + {
41 + "command": "composio execute SLACK_SEND_MESSAGE --get-schema",
42 + "exitCode": 0,
43 + "stdout": "{\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"version\": \"20260101_00\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"channel\": {\n \"type\": \"string\",\n \"description\": \"Channel name or id to post to.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Message text.\"\n }\n },\n \"required\": [\n \"channel\",\n \"text\"\n ]\n }\n}\n",
44 + "stderr": ""
45 + },
46 + {
47 + "command": "composio execute SLACK_SEND_MESSAGE -d '{ channel: \"#general\", text: \"deploy finished\" }'",
48 + "exitCode": 1,
49 + "stdout": "{\n \"successful\": false,\n \"error\": \"No active connection found for toolkit \\\"slack\\\". Run `composio link slack`, then retry.\",\n \"slug\": \"ToolRouterV2_NoActiveConnection\"\n}\n",
50 + "stderr": ""
51 + },
52 + {
53 + "command": "composio link slack",
54 + "exitCode": 0,
55 + "stdout": "{\n \"status\": \"success\",\n \"message\": \"Connected account \\\"ca_e2e0001\\\" is now ACTIVE (toolkit: slack).\",\n \"connected_account_id\": \"ca_e2e0001\",\n \"toolkit\": \"slack\",\n \"redirect_url\": \"https://backend.composio.test/api/v3/s/ca_e2e0001\"\n}\n",
56 + "stderr": ""
57 + },
58 + {
59 + "command": "composio execute SLACK_SEND_MESSAGE -d '{ channel: \"#general\", text: \"deploy finished\" }'",
60 + "exitCode": 0,
61 + "stdout": "{\n \"successful\": true,\n \"data\": {\n \"ok\": true,\n \"channel\": \"#general\",\n \"ts\": \"1767225600.000100\",\n \"message\": {\n \"text\": \"deploy finished\"\n }\n },\n \"error\": null,\n \"logId\": \"log_e2e0001\"\n}\n",
62 + "stderr": ""
63 + }
64 + ],
65 + "messages": [
66 + "Slack isn't connected yet, so I'll start the link flow.",
67 + "The message \"deploy finished\" was posted to #general.\n\nSlack wasn't connected at first, so I ran `composio link slack` to authorize it, then retried the send. The link completed on the first try, and the send succeeded (log ID `log_e2e0001`)."
68 + ],
69 + "replies": []
70 +}
ts/e2e-tests/cli/agent-journeys/transcripts/01-approve-everything/consumer-slack-message.anthropic.claude-opus-5-5.json new
+69
@@ -0,0 +1,69 @@
1 +{
2 + "journey": "consumer-slack-message",
3 + "provider": "anthropic",
4 + "model": "claude-opus-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "Send a Slack message to the #general channel that says: deploy finished",
10 + "reachedGoal": true,
11 + "stop": "finished",
12 + "modelTurns": 5,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "the last `composio execute` exited 0",
20 + "ok": true
21 + },
22 + {
23 + "name": "the message was sent once, on the consumer project, as the consumer user",
24 + "ok": true
25 + },
26 + {
27 + "name": "the account was linked on the consumer project",
28 + "ok": true
29 + },
30 + {
31 + "name": "no `composio dev` command ran",
32 + "ok": true
33 + },
34 + {
35 + "name": "every request that named a project named the consumer project",
36 + "ok": true
37 + }
38 + ],
39 + "commands": [
40 + {
41 + "command": "composio execute SLACK_SEND_MESSAGE --get-schema 2>&1 | head -50",
42 + "exitCode": 0,
43 + "stdout": "{\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"version\": \"20260101_00\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"channel\": {\n \"type\": \"string\",\n \"description\": \"Channel name or id to post to.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Message text.\"\n }\n },\n \"required\": [\n \"channel\",\n \"text\"\n ]\n }\n}\n",
44 + "stderr": ""
45 + },
46 + {
47 + "command": "composio execute SLACK_SEND_MESSAGE -d '{ \"channel\": \"#general\", \"text\": \"deploy finished\" }' 2>&1",
48 + "exitCode": 1,
49 + "stdout": "{\n \"successful\": false,\n \"error\": \"No active connection found for toolkit \\\"slack\\\". Run `composio link slack`, then retry.\",\n \"slug\": \"ToolRouterV2_NoActiveConnection\"\n}\n",
50 + "stderr": ""
51 + },
52 + {
53 + "command": "composio link slack --no-browser 2>&1",
54 + "exitCode": 0,
55 + "stdout": "{\n \"status\": \"success\",\n \"message\": \"Connected account \\\"ca_e2e0001\\\" is now ACTIVE (toolkit: slack).\",\n \"connected_account_id\": \"ca_e2e0001\",\n \"toolkit\": \"slack\",\n \"redirect_url\": \"https://backend.composio.test/api/v3/s/ca_e2e0001\"\n}\n",
56 + "stderr": ""
57 + },
58 + {
59 + "command": "composio execute SLACK_SEND_MESSAGE -d '{ \"channel\": \"#general\", \"text\": \"deploy finished\" }' 2>&1",
60 + "exitCode": 0,
61 + "stdout": "{\n \"successful\": true,\n \"data\": {\n \"ok\": true,\n \"channel\": \"#general\",\n \"ts\": \"1767225600.000100\",\n \"message\": {\n \"text\": \"deploy finished\"\n }\n },\n \"error\": null,\n \"logId\": \"log_e2e0001\"\n}\n",
62 + "stderr": ""
63 + }
64 + ],
65 + "messages": [
66 + "I posted \"deploy finished\" to the #general channel in Slack, and Slack confirmed it went through (message timestamp `1767225600.000100`).\n\nThe first try failed because no Slack account was connected. I ran `composio link slack` to connect one, and Slack is now connected (account `ca_e2e0001`), so you won't need to set it up again next time."
67 + ],
68 + "replies": []
69 +}
ts/e2e-tests/cli/agent-journeys/transcripts/01-approve-everything/developer-auth-configs-and-execute.anthropic.claude-haiku-5-5.json new
+95
@@ -0,0 +1,95 @@
1 +{
2 + "journey": "developer-auth-configs-and-execute",
3 + "provider": "anthropic",
4 + "model": "claude-haiku-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "I am building an app on Composio. In my developer project, list the auth configs, then execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
10 + "reachedGoal": false,
11 + "stop": "finished",
12 + "modelTurns": 7,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "`composio dev auth-configs list` ran and exited 0",
20 + "ok": true
21 + },
22 + {
23 + "name": "the auth configs were read from the developer project",
24 + "ok": true
25 + },
26 + {
27 + "name": "the tool ran once on the developer project as the global test user",
28 + "ok": false,
29 + "seen": "[]"
30 + },
31 + {
32 + "name": "it ran through `composio dev playground-execute`",
33 + "ok": false
34 + },
35 + {
36 + "name": "nothing was executed or linked on the consumer project",
37 + "ok": false
38 + }
39 + ],
40 + "commands": [
41 + {
42 + "command": "cat /skills/composio-cli/references/composio-dev.md 2>/dev/null || echo \"NOT_FOUND\"",
43 + "exitCode": 0,
44 + "stdout": "# Composio Dev\n\nLoad this file only when the user explicitly needs developer-project workflows. Do not lead with `composio dev` for ordinary top-level CLI usage.\n\n## Use `dev init` To Bind A Directory\n\nUse `composio dev init` when the user wants to attach the current directory to a developer project.\n\nIf later commands complain about missing developer project context, come back to `dev init`.\n\n```bash\ncomposio dev init -y --no-browser\n```\n\n## Inspect Toolkits And Versions\n\nUse toolkit commands to inspect developer-scoped capabilities:\n\n```bash\ncomposio dev toolkits list\ncomposio dev toolkits info github\ncomposio dev toolkits search \"email\"\ncomposio dev toolkits version github\n```\n\n## Manage Auth Configs\n\nUse auth-config commands when the user is configuring developer-project authentication behavior:\n\n```bash\ncomposio dev auth-configs list\n```\n\n## Manage Connected Accounts\n\nUse developer connected-account commands when the user is working with developer-project users or auth-config-specific flows.\n\nTop-level `composio link` is consumer-only. Use `dev connected-accounts link` for developer-project flows.\n\n```bash\ncomposio dev connected-accounts list\n```\n\n## Work With Triggers\n\nUse trigger commands when the user is creating, inspecting, or listening to developer-project triggers.\n\n```bash\ncomposio dev triggers list\ncomposio dev triggers info AGENT_MAIL_NEW_EMAIL_TRIGGER\n```\n\n## Inspect Logs\n\nUse logs when the user is debugging tool executions or trigger deliveries inside a developer project.\n\n```bash\ncomposio dev logs tools\ncomposio dev logs triggers\n```\n\n## Switch Or Inspect Org And Project Context\n\nUse these commands when the user needs to confirm or change the active developer scope:\n\nIf the user only wants to connect and execute tools as an end user, return to the top-level workflow instead of staying in `composio dev`.\n\n```bash\ncomposio dev orgs list\ncomposio dev projects list\n```\n",
45 + "stderr": ""
46 + },
47 + {
48 + "command": "composio whoami",
49 + "exitCode": 0,
50 + "stdout": "{\"account_type\":\"human\",\"email\":\"dev@acme-e2e.test\",\"current_org_name\":\"Acme E2E\",\"enhanced_controls_enabled\":false}\n",
51 + "stderr": ""
52 + },
53 + {
54 + "command": "composio dev auth-configs list",
55 + "exitCode": 0,
56 + "stdout": "[\n {\n \"id\": \"ac_e2e0001\",\n \"name\": \"slack-developer\",\n \"toolkit_slug\": \"slack\",\n \"auth_scheme\": \"OAUTH2\",\n \"type\": \"default\",\n \"status\": \"ENABLED\",\n \"no_of_connections\": 1,\n \"is_composio_managed\": true\n },\n {\n \"id\": \"ac_e2e0002\",\n \"name\": \"github-developer\",\n \"toolkit_slug\": \"github\",\n \"auth_scheme\": \"OAUTH2\",\n \"type\": \"default\",\n \"status\": \"ENABLED\",\n \"no_of_connections\": 0,\n \"is_composio_managed\": true\n },\n {\n \"id\": \"ac_e2e0003\",\n \"name\": \"gmail-developer\",\n \"toolkit_slug\": \"gmail\",\n \"auth_scheme\": \"OAUTH2\",\n \"type\": \"default\",\n \"status\": \"ENABLED\",\n \"no_of_connections\": 0,\n \"is_composio_managed\": true\n }\n]\n",
57 + "stderr": ""
58 + },
59 + {
60 + "command": "composio execute --help 2>&1 | head -80",
61 + "exitCode": 0,
62 + "stdout": "DESCRIPTION\n Execute a tool by slug. Validates inputs against cached schemas and checks connections\nautomatically — just try it and it will tell you what to fix.\n\nUSAGE\n composio execute [flags] <slug> [<additional-slugs...>]\n\nARGUMENTS\n slug string Tool slug (e.g. \"GITHUB_CREATE_ISSUE\")\n additional-slugs... string Additional tool slugs for --parallel (optional)\n\nFLAGS\n --parallel, -p Execute repeated tool slug/data groups concurrently\n --data, -d string JSON arguments, @file, or - for stdin\n --file string Inject a local file path into the single file_uploadable input\n --account string Connected account selector for the inferred toolkit. Matches alias, word_id, or connected account id.\n --get-schema Fetch and print the CLI-facing input schema without executing\n --dry-run Validate and preview the tool call without executing\n --skip-connection-check Skip the connected-account check\n --skip-tool-params-check Skip input validation against cached schema\n --skip-checks Skip both connection and input validation checks\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nEXAMPLES\n # Send an email\n composio execute GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\", subject: \"Hello\", body: \"World\" }'\n\n # Create a GitHub issue with a named account\n composio execute GITHUB_CREATE_ISSUE --account work -d '{ owner: \"acme\", repo: \"app\", title: \"Bug report\", body: \"Steps to reproduce...\" }'\n\n # Preview what a tool call would send without executing\n composio execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", markdown_text: \"Hello team\" }'\n\n # Print the tool input schema as JSON (--help shows command options)\n composio execute GMAIL_SEND_EMAIL --get-schema\n\n # Read arguments from a file\n composio execute GITHUB_CREATE_ISSUE -d @issue.json\n\n # Execute multiple tools concurrently\n composio execute -p GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\" }' GITHUB_CREATE_AN_ISSUE -d '{ owner: \"acme\", repo: \"app\", title: \"Bug\" }'\n",
63 + "stderr": ""
64 + },
65 + {
66 + "command": "composio execute SLACK_SEND_MESSAGE --get-schema",
67 + "exitCode": 0,
68 + "stdout": "{\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"version\": \"20260101_00\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"channel\": {\n \"type\": \"string\",\n \"description\": \"Channel name or id to post to.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Message text.\"\n }\n },\n \"required\": [\n \"channel\",\n \"text\"\n ]\n }\n}\n",
69 + "stderr": ""
70 + },
71 + {
72 + "command": "composio dev connected-accounts list",
73 + "exitCode": 0,
74 + "stdout": "[\n {\n \"id\": \"ca_e2e0001\",\n \"alias\": null,\n \"word_id\": \"slack_e2e_0001\",\n \"user_id\": \"pg-test-00000000-0000-4000-8000-00000000a001\",\n \"toolkit_slug\": \"slack\",\n \"auth_config_id\": \"ac_e2e0001\",\n \"auth_scheme\": \"OAUTH2\",\n \"status\": \"ACTIVE\",\n \"created_at\": \"2026-01-01T00:00:00.000Z\"\n }\n]\n",
75 + "stderr": ""
76 + },
77 + {
78 + "command": "composio execute SLACK_SEND_MESSAGE --account slack_e2e_0001 -d '{ channel: \"general\", text: \"hello from dev\" }'",
79 + "exitCode": 1,
80 + "stdout": "",
81 + "stderr": "No connected account matched \"slack_e2e_0001\" for toolkit \"slack\". No active connected accounts were found for that toolkit.\n"
82 + },
83 + {
84 + "command": "composio execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }'",
85 + "exitCode": 1,
86 + "stdout": "{\n \"successful\": false,\n \"error\": \"No active connection found for toolkit \\\"slack\\\". Run `composio link slack`, then retry.\",\n \"slug\": \"ToolRouterV2_NoActiveConnection\"\n}\n",
87 + "stderr": ""
88 + }
89 + ],
90 + "messages": [
91 + "The `--account` selector didn't match. I'll retry without it and let the CLI choose the default Slack connection.",
92 + "I listed the auth configs, but I did not post the Slack message. The CLI can't use the test user's Slack connection.\n\n**Auth configs in your developer project (Acme E2E):**\n\n| Name | ID | Toolkit | Auth | Status | Connections |\n|---|---|---|---|---|---|\n| slack-developer | ac_e2e0001 | slack | OAUTH2 | ENABLED | 1 |\n| github-developer | ac_e2e0002 | github | OAUTH2 | ENABLED | 0 |\n| gmail-developer | ac_e2e0003 | gmail | OAUTH2 | ENABLED | 0 |\n\n**Sending the message:**\n\n- `SLACK_SEND_MESSAGE` takes `channel` and `text`, so I sent `{ channel: \"general\", text: \"hello from dev\" }`.\n- The developer project has one active Slack connected account (`ca_e2e0001`, user `pg-test-00000000-0000-4000-8000-00000000a001`). That is probably your test user.\n- `composio execute --account slack_e2e_0001` failed: no active Slack connected account matched.\n- Running without `--account` failed with `ToolRouterV2_NoActiveConnection`: no active Slack connection for the CLI's own user, `dev@acme-e2e.test`.\n\nThe top-level `execute` command only sees connections owned by the CLI's signed-in user. It cannot act as an arbitrary end user from your developer project.\n\nTo post as the test user, you can:\n1. Run `composio link slack` to connect Slack for `dev@acme-e2e.test`, then retry. This posts as that account, not the test user. It needs a browser OAuth step, so I haven't started it without your go-ahead.\n2. Call the Slack send through your app's own Composio SDK, using the test user's ID (`pg-test-00000000-0000-4000-8000-00000000a001`) so it resolves to `ca_e2e0001`.\n\nWhich would you like?"
93 + ],
94 + "replies": []
95 +}
ts/e2e-tests/cli/agent-journeys/transcripts/01-approve-everything/developer-auth-configs-and-execute.anthropic.claude-opus-5-5.json new
+69
@@ -0,0 +1,69 @@
1 +{
2 + "journey": "developer-auth-configs-and-execute",
3 + "provider": "anthropic",
4 + "model": "claude-opus-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "I am building an app on Composio. In my developer project, list the auth configs, then execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
10 + "reachedGoal": true,
11 + "stop": "finished",
12 + "modelTurns": 5,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "`composio dev auth-configs list` ran and exited 0",
20 + "ok": true
21 + },
22 + {
23 + "name": "the auth configs were read from the developer project",
24 + "ok": true
25 + },
26 + {
27 + "name": "the tool ran once on the developer project as the global test user",
28 + "ok": true
29 + },
30 + {
31 + "name": "it ran through `composio dev playground-execute`",
32 + "ok": true
33 + },
34 + {
35 + "name": "nothing was executed or linked on the consumer project",
36 + "ok": true
37 + }
38 + ],
39 + "commands": [
40 + {
41 + "command": "cat /skills/composio-cli/references/composio-dev.md",
42 + "exitCode": 0,
43 + "stdout": "# Composio Dev\n\nLoad this file only when the user explicitly needs developer-project workflows. Do not lead with `composio dev` for ordinary top-level CLI usage.\n\n## Use `dev init` To Bind A Directory\n\nUse `composio dev init` when the user wants to attach the current directory to a developer project.\n\nIf later commands complain about missing developer project context, come back to `dev init`.\n\n```bash\ncomposio dev init -y --no-browser\n```\n\n## Inspect Toolkits And Versions\n\nUse toolkit commands to inspect developer-scoped capabilities:\n\n```bash\ncomposio dev toolkits list\ncomposio dev toolkits info github\ncomposio dev toolkits search \"email\"\ncomposio dev toolkits version github\n```\n\n## Manage Auth Configs\n\nUse auth-config commands when the user is configuring developer-project authentication behavior:\n\n```bash\ncomposio dev auth-configs list\n```\n\n## Manage Connected Accounts\n\nUse developer connected-account commands when the user is working with developer-project users or auth-config-specific flows.\n\nTop-level `composio link` is consumer-only. Use `dev connected-accounts link` for developer-project flows.\n\n```bash\ncomposio dev connected-accounts list\n```\n\n## Work With Triggers\n\nUse trigger commands when the user is creating, inspecting, or listening to developer-project triggers.\n\n```bash\ncomposio dev triggers list\ncomposio dev triggers info AGENT_MAIL_NEW_EMAIL_TRIGGER\n```\n\n## Inspect Logs\n\nUse logs when the user is debugging tool executions or trigger deliveries inside a developer project.\n\n```bash\ncomposio dev logs tools\ncomposio dev logs triggers\n```\n\n## Switch Or Inspect Org And Project Context\n\nUse these commands when the user needs to confirm or change the active developer scope:\n\nIf the user only wants to connect and execute tools as an end user, return to the top-level workflow instead of staying in `composio dev`.\n\n```bash\ncomposio dev orgs list\ncomposio dev projects list\n```\n",
44 + "stderr": ""
45 + },
46 + {
47 + "command": "composio dev auth-configs list; echo ---; composio dev connected-accounts list; echo ---; composio dev --help",
48 + "exitCode": 0,
49 + "stdout": "[\n {\n \"id\": \"ac_e2e0001\",\n \"name\": \"slack-developer\",\n \"toolkit_slug\": \"slack\",\n \"auth_scheme\": \"OAUTH2\",\n \"type\": \"default\",\n \"status\": \"ENABLED\",\n \"no_of_connections\": 1,\n \"is_composio_managed\": true\n },\n {\n \"id\": \"ac_e2e0002\",\n \"name\": \"github-developer\",\n \"toolkit_slug\": \"github\",\n \"auth_scheme\": \"OAUTH2\",\n \"type\": \"default\",\n \"status\": \"ENABLED\",\n \"no_of_connections\": 0,\n \"is_composio_managed\": true\n },\n {\n \"id\": \"ac_e2e0003\",\n \"name\": \"gmail-developer\",\n \"toolkit_slug\": \"gmail\",\n \"auth_scheme\": \"OAUTH2\",\n \"type\": \"default\",\n \"status\": \"ENABLED\",\n \"no_of_connections\": 0,\n \"is_composio_managed\": true\n }\n]\n---\n[\n {\n \"id\": \"ca_e2e0001\",\n \"alias\": null,\n \"word_id\": \"slack_e2e_0001\",\n \"user_id\": \"pg-test-00000000-0000-4000-8000-00000000a001\",\n \"toolkit_slug\": \"slack\",\n \"auth_config_id\": \"ac_e2e0001\",\n \"auth_scheme\": \"OAUTH2\",\n \"status\": \"ACTIVE\",\n \"created_at\": \"2026-01-01T00:00:00.000Z\"\n }\n]\n---\nDESCRIPTION\n Developer workflows: init, playground execution, logs, projects, toolkits, accounts, and triggers.\n\nUSAGE\n composio dev <subcommand> [flags]\n\nFLAGS\n --mode choice Set developer mode on or off. (choices: on, off)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nSUBCOMMANDS\n init Initialize this directory with a developer project.\n playground-execute Test tool executions against playground users using your developer project auth configs.\n listen Listen to realtime trigger events for your developer project and optionally forward them into your local dev environment.\n logs Inspect trigger and tool execution logs.\n toolkits Discover and inspect Composio toolkits.\n auth-configs View and manage Composio auth configs.\n connected-accounts View and manage Composio connected accounts.\n triggers Inspect and subscribe to trigger events.\n projects Inspect developer projects and project context helpers.\nEXAMPLES\n composio dev --mode on\n composio dev --mode off\n",
50 + "stderr": ""
51 + },
52 + {
53 + "command": "composio dev playground-execute --help; composio dev playground-execute SLACK_SEND_MESSAGE --get-schema",
54 + "exitCode": 0,
55 + "stdout": "DESCRIPTION\n Test tool executions against playground users using your developer project auth configs.\nUses --user-id when provided, otherwise falls back to your local or global playground test user id.\nArguments are validated against cached tool schemas in `~/.composio/tool_definitions/` when available.\n\nUSAGE\n composio dev playground-execute [flags] <slug> [<additional-slugs...>]\n\nARGUMENTS\n slug string Tool slug (e.g. \"GITHUB_CREATE_ISSUE\")\n additional-slugs... string Additional tool slugs for --parallel (optional)\n\nFLAGS\n --parallel, -p Execute repeated tool slug/data groups concurrently\n --data, -d string JSON arguments, @file, or - for stdin\n --file string Inject a local file path into the single file_uploadable input\n --account string Connected account selector for the inferred toolkit. Matches alias, word_id, or connected account id.\n --get-schema Fetch and print the CLI-facing input schema without executing\n --dry-run Validate and preview the tool call without executing\n --skip-connection-check Skip the connected-account check\n --skip-tool-params-check Skip input validation against cached schema\n --skip-checks Skip both connection and input validation checks\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nEXAMPLES\n # Send an email\n composio dev playground-execute GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\", subject: \"Hello\", body: \"World\" }'\n\n # Create a GitHub issue with a named account\n composio dev playground-execute GITHUB_CREATE_ISSUE --account work -d '{ owner: \"acme\", repo: \"app\", title: \"Bug report\", body: \"Steps to reproduce...\" }'\n\n # Preview what a tool call would send without executing\n composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", markdown_text: \"Hello team\" }'\n\n # Print the tool input schema as JSON (--help shows command options)\n composio dev playground-execute GMAIL_SEND_EMAIL --get-schema\n\n # Read arguments from a file\n composio dev playground-execute GITHUB_CREATE_ISSUE -d @issue.json\n\n # Execute multiple tools concurrently\n composio dev playground-execute -p GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\" }' GITHUB_CREATE_AN_ISSUE -d '{ owner: \"acme\", repo: \"app\", title: \"Bug\" }'\n{\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"version\": \"20260101_00\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"channel\": {\n \"type\": \"string\",\n \"description\": \"Channel name or id to post to.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Message text.\"\n }\n },\n \"required\": [\n \"channel\",\n \"text\"\n ]\n }\n}\n",
56 + "stderr": ""
57 + },
58 + {
59 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE -d '{ \"channel\": \"general\", \"text\": \"hello from dev\" }'",
60 + "exitCode": 0,
61 + "stdout": "{\n \"successful\": true,\n \"data\": {\n \"ok\": true,\n \"channel\": \"general\",\n \"ts\": \"1767225600.000100\",\n \"message\": {\n \"text\": \"hello from dev\"\n }\n },\n \"error\": null,\n \"logId\": \"log_e2e0001\"\n}\n",
62 + "stderr": ""
63 + }
64 + ],
65 + "messages": [
66 + "I listed your project's auth configs and posted \"hello from dev\" to #general as your test user.\n\n**Auth configs** (`composio dev auth-configs list`): all three are Composio-managed, use OAuth2 and are enabled.\n\n| ID | Name | Toolkit | Connections |\n|---|---|---|---|\n| `ac_e2e0001` | slack-developer | slack | 1 |\n| `ac_e2e0002` | github-developer | github | 0 |\n| `ac_e2e0003` | gmail-developer | gmail | 0 |\n\n**Sending the message:** Your project has one connected account, an active Slack account (`ca_e2e0001`, word_id `slack_e2e_0001`) for test user `pg-test-00000000-0000-4000-8000-00000000a001`, using `slack-developer`. I checked the tool's inputs (`channel` and `text`), then ran:\n\n```bash\ncomposio dev playground-execute SLACK_SEND_MESSAGE -d '{ \"channel\": \"general\", \"text\": \"hello from dev\" }'\n```\n\nSlack returned `ok: true`. The message timestamp is `1767225600.000100` and the log ID is `log_e2e0001`.\n\nI didn't pass `--user-id`, so the command used the default test user. The output didn't say which user that was, but the project's only Slack connection belongs to the test user above, so that's the one that posted. To pick the user explicitly, add `--user-id <id>`. You can see the run with `composio dev logs tools`.\n\nGitHub and Gmail have no connected accounts yet. Before you test those tools, connect an account with `composio dev connected-accounts link`."
67 + ],
68 + "replies": []
69 +}
ts/e2e-tests/cli/agent-journeys/transcripts/01-approve-everything/developer-execute-unconnected.anthropic.claude-haiku-5-5.json new
+76
@@ -0,0 +1,76 @@
1 +{
2 + "journey": "developer-execute-unconnected",
3 + "provider": "anthropic",
4 + "model": "claude-haiku-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
10 + "reachedGoal": false,
11 + "stop": "finished",
12 + "modelTurns": 7,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "the tool ran once on the developer project as the global test user",
20 + "ok": false,
21 + "seen": "[{\"project\":\"consumer\",\"user\":\"consumer-user-e2e-acme-0001\"}]"
22 + },
23 + {
24 + "name": "the account was linked on the developer project for the test user",
25 + "ok": false
26 + },
27 + {
28 + "name": "the last `composio dev playground-execute` exited 0",
29 + "ok": false,
30 + "seen": "[\"0 composio execute SLACK_SEND_MESSAGE --get-schema\",\"0 composio execute --help 2>&1; echo \\\"---\\\"; composio whoami 2>&1\",\"0 composio execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \\\"general\\\", text: \\\"hello from dev\\\" }' 2>&1\",\"1 composio execute SLACK_SEND_MESSAGE -d '{ channel: \\\"general\\\", text: \\\"hello from dev\\\" }' 2>&1\",\"0 timeout 60 composio link slack --no-browser 2>&1\",\"0 composio execute SLACK_SEND_MESSAGE -d '{ channel: \\\"general\\\", text: \\\"hello from dev\\\" }' 2>&1\"]"
31 + }
32 + ],
33 + "commands": [
34 + {
35 + "command": "composio execute SLACK_SEND_MESSAGE --get-schema",
36 + "exitCode": 0,
37 + "stdout": "{\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"version\": \"20260101_00\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"channel\": {\n \"type\": \"string\",\n \"description\": \"Channel name or id to post to.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Message text.\"\n }\n },\n \"required\": [\n \"channel\",\n \"text\"\n ]\n }\n}\n",
38 + "stderr": ""
39 + },
40 + {
41 + "command": "composio execute --help 2>&1; echo \"---\"; composio whoami 2>&1",
42 + "exitCode": 0,
43 + "stdout": "DESCRIPTION\n Execute a tool by slug. Validates inputs against cached schemas and checks connections\nautomatically — just try it and it will tell you what to fix.\n\nUSAGE\n composio execute [flags] <slug> [<additional-slugs...>]\n\nARGUMENTS\n slug string Tool slug (e.g. \"GITHUB_CREATE_ISSUE\")\n additional-slugs... string Additional tool slugs for --parallel (optional)\n\nFLAGS\n --parallel, -p Execute repeated tool slug/data groups concurrently\n --data, -d string JSON arguments, @file, or - for stdin\n --file string Inject a local file path into the single file_uploadable input\n --account string Connected account selector for the inferred toolkit. Matches alias, word_id, or connected account id.\n --get-schema Fetch and print the CLI-facing input schema without executing\n --dry-run Validate and preview the tool call without executing\n --skip-connection-check Skip the connected-account check\n --skip-tool-params-check Skip input validation against cached schema\n --skip-checks Skip both connection and input validation checks\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nEXAMPLES\n # Send an email\n composio execute GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\", subject: \"Hello\", body: \"World\" }'\n\n # Create a GitHub issue with a named account\n composio execute GITHUB_CREATE_ISSUE --account work -d '{ owner: \"acme\", repo: \"app\", title: \"Bug report\", body: \"Steps to reproduce...\" }'\n\n # Preview what a tool call would send without executing\n composio execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", markdown_text: \"Hello team\" }'\n\n # Print the tool input schema as JSON (--help shows command options)\n composio execute GMAIL_SEND_EMAIL --get-schema\n\n # Read arguments from a file\n composio execute GITHUB_CREATE_ISSUE -d @issue.json\n\n # Execute multiple tools concurrently\n composio execute -p GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\" }' GITHUB_CREATE_AN_ISSUE -d '{ owner: \"acme\", repo: \"app\", title: \"Bug\" }'\n---\n{\"account_type\":\"human\",\"email\":\"dev@acme-e2e.test\",\"current_org_name\":\"Acme E2E\",\"enhanced_controls_enabled\":false}\n",
44 + "stderr": ""
45 + },
46 + {
47 + "command": "composio execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", text: \"hello from dev\" }' 2>&1",
48 + "exitCode": 0,
49 + "stdout": "{\n \"successful\": true,\n \"dryRun\": true,\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"arguments\": {\n \"channel\": \"general\",\n \"text\": \"hello from dev\"\n },\n \"userId\": \"consumer-user-e2e-acme-0001\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"schemaVersion\": \"20260101_00\"\n}\n",
50 + "stderr": ""
51 + },
52 + {
53 + "command": "composio execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }' 2>&1",
54 + "exitCode": 1,
55 + "stdout": "{\n \"successful\": false,\n \"error\": \"No active connection found for toolkit \\\"slack\\\". Run `composio link slack`, then retry.\",\n \"slug\": \"ToolRouterV2_NoActiveConnection\"\n}\n",
56 + "stderr": ""
57 + },
58 + {
59 + "command": "timeout 60 composio link slack --no-browser 2>&1",
60 + "exitCode": 0,
61 + "stdout": "{\n \"status\": \"success\",\n \"message\": \"Connected account \\\"ca_e2e0001\\\" is now ACTIVE (toolkit: slack).\",\n \"connected_account_id\": \"ca_e2e0001\",\n \"toolkit\": \"slack\",\n \"redirect_url\": \"https://backend.composio.test/api/v3/s/ca_e2e0001\"\n}\n",
62 + "stderr": ""
63 + },
64 + {
65 + "command": "composio execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }' 2>&1",
66 + "exitCode": 0,
67 + "stdout": "{\n \"successful\": true,\n \"data\": {\n \"ok\": true,\n \"channel\": \"general\",\n \"ts\": \"1767225600.000100\",\n \"message\": {\n \"text\": \"hello from dev\"\n }\n },\n \"error\": null,\n \"logId\": \"log_e2e0001\"\n}\n",
68 + "stderr": ""
69 + }
70 + ],
71 + "messages": [
72 + "The Slack toolkit isn't connected for this user, so I'll start the Slack link flow.",
73 + "The message \"hello from dev\" was posted to #general, and Slack returned `ok: true` (message ts `1767225600.000100`, log ID `log_e2e0001`).\n\nI wasn't able to confirm that this is your test user:\n\n- Slack wasn't connected for the default user, so I ran `composio link slack --no-browser`. It created connected account `ca_e2e0001` and marked it ACTIVE. That link is what the post went through.\n- The dry run reported the request under userId `consumer-user-e2e-acme-0001`. I didn't pass `--account` or a user selector, so the post used whichever user the CLI resolves by default.\n\nIf your test user is a different identity, tell me its alias or account ID and I'll re-run with `--account`. Checking which user is which would take the `composio dev` commands for connected accounts, which I haven't used here."
74 + ],
75 + "replies": []
76 +}
ts/e2e-tests/cli/agent-journeys/transcripts/01-approve-everything/developer-execute-unconnected.anthropic.claude-opus-5-5.json new
+90
@@ -0,0 +1,90 @@
1 +{
2 + "journey": "developer-execute-unconnected",
3 + "provider": "anthropic",
4 + "model": "claude-opus-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
10 + "reachedGoal": false,
11 + "stop": "finished",
12 + "modelTurns": 9,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "the tool ran once on the developer project as the global test user",
20 + "ok": false,
21 + "seen": "[]"
22 + },
23 + {
24 + "name": "the account was linked on the developer project for the test user",
25 + "ok": true
26 + },
27 + {
28 + "name": "the last `composio dev playground-execute` exited 0",
29 + "ok": false,
30 + "seen": "[\"0 cat /skills/composio-cli/references/composio-dev.md\",\"0 composio whoami; composio dev --help; composio execute --help 2>&1 | head -60\",\"0 composio dev playground-execute --help; composio dev projects list 2>&1 | head -30\",\"1 composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \\\"general\\\", markdown_text: \\\"hello from dev\\\" }'\",\"0 composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \\\"general\\\", text: \\\"hello from dev\\\" }'\",\"1 composio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \\\"general\\\", text: \\\"hello from dev\\\" }'\",\"0 composio dev connected-accounts list 2>&1 | head -40; composio dev connected-accounts link --help 2>&1 | head -40\",\"0 composio dev connected-accounts link slack --no-browser --no-wait\"]"
31 + }
32 + ],
33 + "commands": [
34 + {
35 + "command": "cat /skills/composio-cli/references/composio-dev.md",
36 + "exitCode": 0,
37 + "stdout": "# Composio Dev\n\nLoad this file only when the user explicitly needs developer-project workflows. Do not lead with `composio dev` for ordinary top-level CLI usage.\n\n## Use `dev init` To Bind A Directory\n\nUse `composio dev init` when the user wants to attach the current directory to a developer project.\n\nIf later commands complain about missing developer project context, come back to `dev init`.\n\n```bash\ncomposio dev init -y --no-browser\n```\n\n## Inspect Toolkits And Versions\n\nUse toolkit commands to inspect developer-scoped capabilities:\n\n```bash\ncomposio dev toolkits list\ncomposio dev toolkits info github\ncomposio dev toolkits search \"email\"\ncomposio dev toolkits version github\n```\n\n## Manage Auth Configs\n\nUse auth-config commands when the user is configuring developer-project authentication behavior:\n\n```bash\ncomposio dev auth-configs list\n```\n\n## Manage Connected Accounts\n\nUse developer connected-account commands when the user is working with developer-project users or auth-config-specific flows.\n\nTop-level `composio link` is consumer-only. Use `dev connected-accounts link` for developer-project flows.\n\n```bash\ncomposio dev connected-accounts list\n```\n\n## Work With Triggers\n\nUse trigger commands when the user is creating, inspecting, or listening to developer-project triggers.\n\n```bash\ncomposio dev triggers list\ncomposio dev triggers info AGENT_MAIL_NEW_EMAIL_TRIGGER\n```\n\n## Inspect Logs\n\nUse logs when the user is debugging tool executions or trigger deliveries inside a developer project.\n\n```bash\ncomposio dev logs tools\ncomposio dev logs triggers\n```\n\n## Switch Or Inspect Org And Project Context\n\nUse these commands when the user needs to confirm or change the active developer scope:\n\nIf the user only wants to connect and execute tools as an end user, return to the top-level workflow instead of staying in `composio dev`.\n\n```bash\ncomposio dev orgs list\ncomposio dev projects list\n```\n",
38 + "stderr": ""
39 + },
40 + {
41 + "command": "composio whoami; composio dev --help; composio execute --help 2>&1 | head -60",
42 + "exitCode": 0,
43 + "stdout": "{\"account_type\":\"human\",\"email\":\"dev@acme-e2e.test\",\"current_org_name\":\"Acme E2E\",\"enhanced_controls_enabled\":false}\nDESCRIPTION\n Developer workflows: init, playground execution, logs, projects, toolkits, accounts, and triggers.\n\nUSAGE\n composio dev <subcommand> [flags]\n\nFLAGS\n --mode choice Set developer mode on or off. (choices: on, off)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nSUBCOMMANDS\n init Initialize this directory with a developer project.\n playground-execute Test tool executions against playground users using your developer project auth configs.\n listen Listen to realtime trigger events for your developer project and optionally forward them into your local dev environment.\n logs Inspect trigger and tool execution logs.\n toolkits Discover and inspect Composio toolkits.\n auth-configs View and manage Composio auth configs.\n connected-accounts View and manage Composio connected accounts.\n triggers Inspect and subscribe to trigger events.\n projects Inspect developer projects and project context helpers.\nEXAMPLES\n composio dev --mode on\n composio dev --mode off\nDESCRIPTION\n Execute a tool by slug. Validates inputs against cached schemas and checks connections\nautomatically — just try it and it will tell you what to fix.\n\nUSAGE\n composio execute [flags] <slug> [<additional-slugs...>]\n\nARGUMENTS\n slug string Tool slug (e.g. \"GITHUB_CREATE_ISSUE\")\n additional-slugs... string Additional tool slugs for --parallel (optional)\n\nFLAGS\n --parallel, -p Execute repeated tool slug/data groups concurrently\n --data, -d string JSON arguments, @file, or - for stdin\n --file string Inject a local file path into the single file_uploadable input\n --account string Connected account selector for the inferred toolkit. Matches alias, word_id, or connected account id.\n --get-schema Fetch and print the CLI-facing input schema without executing\n --dry-run Validate and preview the tool call without executing\n --skip-connection-check Skip the connected-account check\n --skip-tool-params-check Skip input validation against cached schema\n --skip-checks Skip both connection and input validation checks\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nEXAMPLES\n # Send an email\n composio execute GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\", subject: \"Hello\", body: \"World\" }'\n\n # Create a GitHub issue with a named account\n composio execute GITHUB_CREATE_ISSUE --account work -d '{ owner: \"acme\", repo: \"app\", title: \"Bug report\", body: \"Steps to reproduce...\" }'\n\n # Preview what a tool call would send without executing\n composio execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", markdown_text: \"Hello team\" }'\n\n # Print the tool input schema as JSON (--help shows command options)\n composio execute GMAIL_SEND_EMAIL --get-schema\n\n # Read arguments from a file\n composio execute GITHUB_CREATE_ISSUE -d @issue.json\n\n # Execute multiple tools concurrently\n composio execute -p GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\" }' GITHUB_CREATE_AN_ISSUE -d '{ owner: \"acme\", repo: \"app\", title: \"Bug\" }'\n",
44 + "stderr": ""
45 + },
46 + {
47 + "command": "composio dev playground-execute --help; composio dev projects list 2>&1 | head -30",
48 + "exitCode": 0,
49 + "stdout": "DESCRIPTION\n Test tool executions against playground users using your developer project auth configs.\nUses --user-id when provided, otherwise falls back to your local or global playground test user id.\nArguments are validated against cached tool schemas in `~/.composio/tool_definitions/` when available.\n\nUSAGE\n composio dev playground-execute [flags] <slug> [<additional-slugs...>]\n\nARGUMENTS\n slug string Tool slug (e.g. \"GITHUB_CREATE_ISSUE\")\n additional-slugs... string Additional tool slugs for --parallel (optional)\n\nFLAGS\n --parallel, -p Execute repeated tool slug/data groups concurrently\n --data, -d string JSON arguments, @file, or - for stdin\n --file string Inject a local file path into the single file_uploadable input\n --account string Connected account selector for the inferred toolkit. Matches alias, word_id, or connected account id.\n --get-schema Fetch and print the CLI-facing input schema without executing\n --dry-run Validate and preview the tool call without executing\n --skip-connection-check Skip the connected-account check\n --skip-tool-params-check Skip input validation against cached schema\n --skip-checks Skip both connection and input validation checks\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nEXAMPLES\n # Send an email\n composio dev playground-execute GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\", subject: \"Hello\", body: \"World\" }'\n\n # Create a GitHub issue with a named account\n composio dev playground-execute GITHUB_CREATE_ISSUE --account work -d '{ owner: \"acme\", repo: \"app\", title: \"Bug report\", body: \"Steps to reproduce...\" }'\n\n # Preview what a tool call would send without executing\n composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", markdown_text: \"Hello team\" }'\n\n # Print the tool input schema as JSON (--help shows command options)\n composio dev playground-execute GMAIL_SEND_EMAIL --get-schema\n\n # Read arguments from a file\n composio dev playground-execute GITHUB_CREATE_ISSUE -d @issue.json\n\n # Execute multiple tools concurrently\n composio dev playground-execute -p GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\" }' GITHUB_CREATE_AN_ISSUE -d '{ owner: \"acme\", repo: \"app\", title: \"Bug\" }'\n[{\"id\":\"pr_e2edeveloper1\",\"name\":\"acme-e2e-dev\",\"is_selected_global_project\":false},{\"id\":\"pr_e2edeveloper2\",\"name\":\"acme-e2e-staging\",\"is_selected_global_project\":false}]\n",
50 + "stderr": ""
51 + },
52 + {
53 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", markdown_text: \"hello from dev\" }'",
54 + "exitCode": 1,
55 + "stdout": "",
56 + "stderr": "\r\n💥 ToolInputValidationError • Input validation failed for SLACK_SEND_MESSAGE.\nSchema: /home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\n- <root>: Unknown key \"markdown_text\". Use \"text\" instead. Allowed top-level keys: channel, text\n- text: Instance does not have required property \"text\".\nCaused by: <root>: Unknown key \"markdown_text\". Use \"text\" instead. Allowed top-level keys: channel, text\ntext: Instance does not have required property \"text\". \r\n\r\n◯\r\n╰─ composio-cli\r\n at cZe (/$bunfs/root/composio:1193:9290) \r\n \r\n\n"
57 + },
58 + {
59 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", text: \"hello from dev\" }'",
60 + "exitCode": 0,
61 + "stdout": "{\n \"successful\": true,\n \"dryRun\": true,\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"arguments\": {\n \"channel\": \"general\",\n \"text\": \"hello from dev\"\n },\n \"userId\": \"pg-test-00000000-0000-4000-8000-00000000a001\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"schemaVersion\": \"20260101_00\"\n}\n",
62 + "stderr": ""
63 + },
64 + {
65 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }'",
66 + "exitCode": 1,
67 + "stdout": "{\n \"successful\": false,\n \"error\": \"No active connection found for toolkit \\\"slack\\\". Run `composio link slack`, then retry.\",\n \"slug\": \"ToolRouterV2_NoActiveConnection\"\n}\n",
68 + "stderr": ""
69 + },
70 + {
71 + "command": "composio dev connected-accounts list 2>&1 | head -40; composio dev connected-accounts link --help 2>&1 | head -40",
72 + "exitCode": 0,
73 + "stdout": "DESCRIPTION\n Connect an external account (GitHub, Gmail, Slack, etc.) so tools can act on your behalf.\nOpens a browser for OAuth authorization and waits for confirmation.\n\nUSAGE\n composio dev connected-accounts link [flags] [<toolkit>]\n\nARGUMENTS\n toolkit string Toolkit slug to link (e.g. \"github\", \"gmail\") (optional)\n\nFLAGS\n --auth-config string Auth config ID (e.g. \"ac_...\"). Uses legacy flow (no Tool Router).\n --no-wait Do not wait for authorization; only print link info\n --no-browser Do not open the browser automatically; print the URL to open manually\n --alias string Alias to assign to the connected account. Required when creating an additional account for the same toolkit/auth config.\n --list List existing connected accounts for the toolkit instead of creating a new link\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nEXAMPLES\n composio link github\n composio link gmail --alias work\n composio link github --list\n",
74 + "stderr": ""
75 + },
76 + {
77 + "command": "composio dev connected-accounts link slack --no-browser --no-wait",
78 + "exitCode": 0,
79 + "stdout": "{\n \"status\": \"pending\",\n \"message\": \"Complete authorization by opening the URL\",\n \"connected_account_id\": \"ca_e2e0001\",\n \"redirect_url\": \"https://backend.composio.test/api/v3/s/ca_e2e0001\",\n \"toolkit\": \"slack\",\n \"project_type\": \"DEVELOPER\"\n}\n",
80 + "stderr": ""
81 + }
82 + ],
83 + "messages": [
84 + "Let me check the schema and do a dry run first.",
85 + "Dry run looks good, resolving to your playground test user. Executing for real:",
86 + "Your test user has no Slack connection in this project. I'll make an authorization link for it:",
87 + "The message wasn't posted yet because your test user hasn't connected a Slack account in this developer project.\n\n**What happened:**\n- A dry run with `composio dev playground-execute` passed. It used your test user `pg-test-00000000-0000-4000-8000-00000000a001` with `{ channel: \"general\", text: \"hello from dev\" }`. This tool takes `text`, not `markdown_text`.\n- The real run failed with \"No active connection found for toolkit slack.\" The test user has no connected accounts.\n- I started a Slack connection for that user in the developer project (`ca_e2e0001`, status: pending).\n\n**What you need to do:** open this link and approve Slack access:\nhttps://backend.composio.test/api/v3/s/ca_e2e0001\n\nOnce you've approved it, tell me and I'll send the message with:\n```bash\ncomposio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }'\n```\n\nThe error message suggests `composio link slack`, but that would connect your personal account, not the test user. That's why I used `composio dev connected-accounts link` instead.\n\nIf `general` isn't accepted as the channel name, I can use `#general` or the channel's ID instead."
88 + ],
89 + "replies": []
90 +}
ts/e2e-tests/cli/agent-journeys/transcripts/01-approve-everything/first-run-login.anthropic.claude-haiku-5-5.json new
+67
@@ -0,0 +1,67 @@
1 +{
2 + "journey": "first-run-login",
3 + "provider": "anthropic",
4 + "model": "claude-haiku-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
10 + "reachedGoal": true,
11 + "stop": "finished",
12 + "modelTurns": 5,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "the first `composio whoami` reported no login (exit 1)",
20 + "ok": true
21 + },
22 + {
23 + "name": "exactly one CLI login session was created, and it was linked",
24 + "ok": true
25 + },
26 + {
27 + "name": "the login completed: the user API key was fetched for the linked session",
28 + "ok": true
29 + },
30 + {
31 + "name": "no agent account was signed up in place of the person",
32 + "ok": true
33 + }
34 + ],
35 + "commands": [
36 + {
37 + "command": "composio whoami",
38 + "exitCode": 1,
39 + "stdout": "",
40 + "stderr": "You are not logged in yet. Please run `composio login`.\n"
41 + },
42 + {
43 + "command": "composio login",
44 + "exitCode": 0,
45 + "stdout": "Open this URL in your browser to log in:\n\n https://dashboard.composio.test?cliKey=00000000-0000-4000-8000-000000050001\n\nThen run this command to complete login:\n\n composio login --poll\n\nhint: For agents: Show the URL above to the user to click, then run the command above. The command uses the cached login key, polls for up to 10 minutes, and exits once credentials are saved. Do not ask the user whether to poll — they already requested login.\n\nhint: For unattended agents: If no human is available to open the URL, run `composio login --agent` instead — it signs the CLI in with a Composio agent account (creating one if needed) without a browser. Never use it when a human is present to log in with their own account.\n",
46 + "stderr": ""
47 + },
48 + {
49 + "command": "composio login --poll",
50 + "exitCode": 0,
51 + "stdout": "{\n \"email\": \"dev@acme-e2e.test\",\n \"current_org\": {\n \"id\": \"ok_e2eacme00001\",\n \"name\": \"Acme E2E\"\n },\n \"organizations\": [\n {\n \"id\": \"ok_e2eacme00001\",\n \"name\": \"Acme E2E\",\n \"selected\": true\n },\n {\n \"id\": \"ok_e2eglobex001\",\n \"name\": \"Globex E2E\",\n \"selected\": false\n }\n ],\n \"switch_command\": \"composio orgs switch --org-id <org_id>\"\n}\n",
52 + "stderr": ""
53 + },
54 + {
55 + "command": "composio whoami",
56 + "exitCode": 0,
57 + "stdout": "{\"account_type\":\"human\",\"email\":\"dev@acme-e2e.test\",\"current_org_name\":\"Acme E2E\",\"enhanced_controls_enabled\":false}\n",
58 + "stderr": ""
59 + }
60 + ],
61 + "messages": [
62 + "Not logged in. Running login now.",
63 + "The login needs a browser step. Please open this URL and finish signing in: https://dashboard.composio.test?cliKey=00000000-0000-4000-8000-000000050001\n\nI'll wait for the login to complete.",
64 + "The Composio CLI was not logged in. I ran `composio login`, you completed the browser step, and `composio login --poll` saved the credentials. `composio whoami` now confirms the session:\n\n- **Account:** dev@acme-e2e.test (human)\n- **Current org:** Acme E2E\n\nAnother org, Globex E2E, is also available. To switch to it, run `composio orgs switch --org-id ok_e2eglobex001`."
65 + ],
66 + "replies": []
67 +}
ts/e2e-tests/cli/agent-journeys/transcripts/01-approve-everything/first-run-login.anthropic.claude-opus-5-5.json new
+72
@@ -0,0 +1,72 @@
1 +{
2 + "journey": "first-run-login",
3 + "provider": "anthropic",
4 + "model": "claude-opus-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "Check whether the Composio CLI on this machine is logged in. If it is not, log in.",
10 + "reachedGoal": true,
11 + "stop": "finished",
12 + "modelTurns": 6,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "the first `composio whoami` reported no login (exit 1)",
20 + "ok": true
21 + },
22 + {
23 + "name": "exactly one CLI login session was created, and it was linked",
24 + "ok": true
25 + },
26 + {
27 + "name": "the login completed: the user API key was fetched for the linked session",
28 + "ok": true
29 + },
30 + {
31 + "name": "no agent account was signed up in place of the person",
32 + "ok": true
33 + }
34 + ],
35 + "commands": [
36 + {
37 + "command": "composio whoami 2>&1",
38 + "exitCode": 1,
39 + "stdout": "You are not logged in yet. Please run `composio login`.\n",
40 + "stderr": ""
41 + },
42 + {
43 + "command": "composio login --help 2>&1 | head -40",
44 + "exitCode": 0,
45 + "stdout": "DESCRIPTION\n Log in to the Composio SDK.\n\nUSAGE\n composio login [flags]\n\nFLAGS\n --no-browser Login without browser interaction\n --poll Poll the most recent pending browser login and complete it\n --no-wait Print login URL and session info, then exit without opening browser or waiting\n --key string Poll and complete login using the session key from composio login\n --user-api-key string Log in directly with a Composio user API key\n --org string Current organization ID or name to store for CLI commands\n --yes, -y Skip org picker; use current org\n --no-skill-install Skip installing the composio-cli skill for Claude Code\n --agent Sign up or log in using a Composio agent identity\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n",
46 + "stderr": ""
47 + },
48 + {
49 + "command": "composio login --no-wait 2>&1",
50 + "exitCode": 0,
51 + "stdout": "Open this URL in your browser to log in:\n\n https://dashboard.composio.test?cliKey=00000000-0000-4000-8000-000000050001\n\nThen run this command to complete login:\n\n composio login --poll\n\nhint: For agents: Show the URL above to the user to click, then run the command above. The command uses the cached login key, polls for up to 10 minutes, and exits once credentials are saved. Do not ask the user whether to poll — they already requested login.\n\nhint: For unattended agents: If no human is available to open the URL, run `composio login --agent` instead — it signs the CLI in with a Composio agent account (creating one if needed) without a browser. Never use it when a human is present to log in with their own account.\n",
52 + "stderr": ""
53 + },
54 + {
55 + "command": "composio login --poll 2>&1",
56 + "exitCode": 0,
57 + "stdout": "{\n \"email\": \"dev@acme-e2e.test\",\n \"current_org\": {\n \"id\": \"ok_e2eacme00001\",\n \"name\": \"Acme E2E\"\n },\n \"organizations\": [\n {\n \"id\": \"ok_e2eacme00001\",\n \"name\": \"Acme E2E\",\n \"selected\": true\n },\n {\n \"id\": \"ok_e2eglobex001\",\n \"name\": \"Globex E2E\",\n \"selected\": false\n }\n ],\n \"switch_command\": \"composio orgs switch --org-id <org_id>\"\n}\n",
58 + "stderr": ""
59 + },
60 + {
61 + "command": "composio whoami 2>&1",
62 + "exitCode": 0,
63 + "stdout": "{\"account_type\":\"human\",\"email\":\"dev@acme-e2e.test\",\"current_org_name\":\"Acme E2E\",\"enhanced_controls_enabled\":false}\n",
64 + "stderr": ""
65 + }
66 + ],
67 + "messages": [
68 + "Open this URL in your browser to log in:\n\n**https://dashboard.composio.test?cliKey=00000000-0000-4000-8000-000000050001**\n\nI'll wait for the login to finish.",
69 + "The Composio CLI wasn't logged in. It is now: you approved the login in your browser, and `composio whoami` confirms it.\n\n- **Account:** dev@acme-e2e.test\n- **Current organization:** Acme E2E (`ok_e2eacme00001`)\n\nYour account also belongs to **Globex E2E** (`ok_e2eglobex001`). To switch to it, run `composio orgs switch --org-id ok_e2eglobex001`."
70 + ],
71 + "replies": []
72 +}
ts/e2e-tests/cli/agent-journeys/transcripts/01-approve-everything/run-inline-script.anthropic.claude-haiku-5-5.json new
+59
@@ -0,0 +1,59 @@
1 +{
2 + "journey": "run-inline-script",
3 + "provider": "anthropic",
4 + "model": "claude-haiku-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "Use a single `composio run` inline script to send \"build passed\" to #general on Slack and print the timestamp of the sent message.",
10 + "reachedGoal": true,
11 + "stop": "finished",
12 + "modelTurns": 4,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "a `composio run` command exited 0 and printed the message timestamp",
20 + "ok": true
21 + },
22 + {
23 + "name": "the message was sent once, on the consumer project, as the consumer user",
24 + "ok": true
25 + }
26 + ],
27 + "commands": [
28 + {
29 + "command": "composio search \"send a message to a slack channel\" --toolkits slack",
30 + "exitCode": 0,
31 + "stdout": "{\n \"results\": [\n {\n \"index\": 1,\n \"use_case\": \"send a message to a slack channel\",\n \"execution_guidance\": \"Execute the primary tool with the documented arguments.\",\n \"primary_tool_slugs\": [\n \"SLACK_SEND_MESSAGE\"\n ],\n \"related_tool_slugs\": [],\n \"toolkits\": [\n \"slack\"\n ]\n }\n ],\n \"tool_schemas\": {\n \"primary\": {\n \"SLACK_SEND_MESSAGE\": \"~/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\"\n },\n \"related_tools_path_format\": \"~/.composio/tool_definitions/<TOOL_SLUG>.json\"\n },\n \"connected_toolkits\": [\n \"slack\"\n ],\n \"next_steps\": {\n \"guidance\": \"You can directly proceed with these steps without waiting for the user to ask. Link accounts first if needed, then execute tools.\",\n \"steps\": [\n {\n \"action\": \"Link a user account\",\n \"command\": \"composio link slack\"\n },\n {\n \"action\": \"Execute a tool\",\n \"command\": \"composio execute \\\"SLACK_SEND_MESSAGE\\\" -d '{\\\"channel\\\":\\\"\\\",\\\"text\\\":\\\"\\\"}'\"\n }\n ]\n }\n}\n",
32 + "stderr": ""
33 + },
34 + {
35 + "command": "composio whoami",
36 + "exitCode": 0,
37 + "stdout": "{\"account_type\":\"human\",\"email\":\"dev@acme-e2e.test\",\"current_org_name\":\"Acme E2E\",\"enhanced_controls_enabled\":false}\n",
38 + "stderr": ""
39 + },
40 + {
41 + "command": "composio execute SLACK_SEND_MESSAGE --get-schema",
42 + "exitCode": 0,
43 + "stdout": "{\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"version\": \"20260101_00\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"channel\": {\n \"type\": \"string\",\n \"description\": \"Channel name or id to post to.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Message text.\"\n }\n },\n \"required\": [\n \"channel\",\n \"text\"\n ]\n }\n}\n",
44 + "stderr": ""
45 + },
46 + {
47 + "command": "composio run '\n const res = await execute(\"SLACK_SEND_MESSAGE\", { channel: \"#general\", text: \"build passed\" });\n const ts = res.data?.ts ?? res.data?.message?.ts ?? res.data?.data?.ts;\n console.log(ts ? `Sent at ts=${ts}` : JSON.stringify(res.data, null, 2));\n'",
48 + "exitCode": 0,
49 + "stdout": "Sent at ts=1767225600.000100\n",
50 + "stderr": "RUN_LOG_FILE=/tmp/composio-run-artifacts-SCRUBBED/run.log\n"
51 + }
52 + ],
53 + "messages": [
54 + "I'll confirm the Slack slug and its input schema before writing the script.",
55 + "The schema takes `channel` and `text`. Now I'll send the message with a single `composio run` script.",
56 + "I sent \"build passed\" to #general with one `composio run` script, and the message timestamp came back as `1767225600.000100`.\n\nThe script called `SLACK_SEND_MESSAGE` with `channel: \"#general\"` and `text: \"build passed\"`, then printed the `ts` field from the response. I checked the tool's input schema first, and the `whoami` check showed I'm logged in as dev@acme-e2e.test with Slack connected."
57 + ],
58 + "replies": []
59 +}
ts/e2e-tests/cli/agent-journeys/transcripts/01-approve-everything/run-inline-script.anthropic.claude-opus-5-5.json new
+46
@@ -0,0 +1,46 @@
1 +{
2 + "journey": "run-inline-script",
3 + "provider": "anthropic",
4 + "model": "claude-opus-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "Use a single `composio run` inline script to send \"build passed\" to #general on Slack and print the timestamp of the sent message.",
10 + "reachedGoal": true,
11 + "stop": "finished",
12 + "modelTurns": 3,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "a `composio run` command exited 0 and printed the message timestamp",
20 + "ok": true
21 + },
22 + {
23 + "name": "the message was sent once, on the consumer project, as the consumer user",
24 + "ok": true
25 + }
26 + ],
27 + "commands": [
28 + {
29 + "command": "composio execute SLACK_SEND_MESSAGE --get-schema 2>&1 | head -50",
30 + "exitCode": 0,
31 + "stdout": "{\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"version\": \"20260101_00\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"channel\": {\n \"type\": \"string\",\n \"description\": \"Channel name or id to post to.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Message text.\"\n }\n },\n \"required\": [\n \"channel\",\n \"text\"\n ]\n }\n}\n",
32 + "stderr": ""
33 + },
34 + {
35 + "command": "composio run '\n const res = await execute(\"SLACK_SEND_MESSAGE\", { channel: \"#general\", text: \"build passed\" });\n const d = res.data ?? {};\n const ts = d.ts ?? d.message?.ts ?? d.response_data?.ts;\n if (!ts) { console.log(\"No ts found; full response:\", JSON.stringify(res, null, 2)); }\n else console.log(\"Message timestamp:\", ts);\n' 2>&1",
36 + "exitCode": 0,
37 + "stdout": "RUN_LOG_FILE=/tmp/composio-run-artifacts-SCRUBBED/run.log\nMessage timestamp: 1767225600.000100\n",
38 + "stderr": ""
39 + }
40 + ],
41 + "messages": [
42 + "I'll check the Slack tool's schema first so the script uses the right arguments.",
43 + "I sent \"build passed\" to #general on Slack with one `composio run` script.\n\n**Timestamp of the sent message:** `1767225600.000100`\n\nThe command I ran:\n```bash\ncomposio run '\n const res = await execute(\"SLACK_SEND_MESSAGE\", { channel: \"#general\", text: \"build passed\" });\n const d = res.data ?? {};\n const ts = d.ts ?? d.message?.ts ?? d.response_data?.ts;\n if (!ts) { console.log(\"No ts found; full response:\", JSON.stringify(res, null, 2)); }\n else console.log(\"Message timestamp:\", ts);\n'\n```\n\nBefore running it, I checked the tool's inputs with `composio execute SLACK_SEND_MESSAGE --get-schema`. The only required fields are `channel` and `text`.\n\nThe script looks for the timestamp in a few places in the response, because the schema doesn't say where it is. If it doesn't find one, it prints the full response instead."
44 + ],
45 + "replies": []
46 +}
ts/e2e-tests/cli/agent-journeys/transcripts/02-approve-what-is-shown/consumer-slack-message.anthropic.claude-opus-5-5.json new
+65
@@ -0,0 +1,65 @@
1 +{
2 + "journey": "consumer-slack-message",
3 + "provider": "anthropic",
4 + "model": "claude-opus-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "Send a Slack message to the #general channel that says: deploy finished",
10 + "reachedGoal": false,
11 + "stop": "finished",
12 + "modelTurns": 4,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "the last `composio execute` exited 0",
20 + "ok": false,
21 + "seen": "[\"0 composio execute SLACK_SEND_MESSAGE --get-schema 2>&1 | head -50\",\"1 composio execute SLACK_SEND_MESSAGE -d '{ \\\"channel\\\": \\\"#general\\\", \\\"text\\\": \\\"deploy finished\\\" }' 2>&1\",\"124 composio link slack --no-browser 2>&1\"]"
22 + },
23 + {
24 + "name": "the message was sent once, on the consumer project, as the consumer user",
25 + "ok": false,
26 + "seen": "[]"
27 + },
28 + {
29 + "name": "the account was linked on the consumer project",
30 + "ok": true
31 + },
32 + {
33 + "name": "no `composio dev` command ran",
34 + "ok": true
35 + },
36 + {
37 + "name": "every request that named a project named the consumer project",
38 + "ok": true
39 + }
40 + ],
41 + "commands": [
42 + {
43 + "command": "composio execute SLACK_SEND_MESSAGE --get-schema 2>&1 | head -50",
44 + "exitCode": 0,
45 + "stdout": "{\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"version\": \"20260101_00\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"channel\": {\n \"type\": \"string\",\n \"description\": \"Channel name or id to post to.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Message text.\"\n }\n },\n \"required\": [\n \"channel\",\n \"text\"\n ]\n }\n}\n",
46 + "stderr": ""
47 + },
48 + {
49 + "command": "composio execute SLACK_SEND_MESSAGE -d '{ \"channel\": \"#general\", \"text\": \"deploy finished\" }' 2>&1",
50 + "exitCode": 1,
51 + "stdout": "{\n \"successful\": false,\n \"error\": \"No active connection found for toolkit \\\"slack\\\". Run `composio link slack`, then retry.\",\n \"slug\": \"ToolRouterV2_NoActiveConnection\"\n}\n",
52 + "stderr": ""
53 + },
54 + {
55 + "command": "composio link slack --no-browser 2>&1",
56 + "exitCode": 124,
57 + "stdout": "",
58 + "stderr": ""
59 + }
60 + ],
61 + "messages": [
62 + "I couldn't send the message because your Slack account isn't connected to Composio yet.\n\n- **What I tried:** I sent \"deploy finished\" to `#general`, but Composio said there's no active Slack connection.\n- **Linking attempt:** `composio link slack --no-browser` timed out without showing an authorization link, so the connection still isn't set up.\n\n**What you need to do:** Run this in your own terminal and approve Slack access in the browser:\n\n```bash\ncomposio link slack\n```\n\nOnce that works, tell me and I'll send the message. If you want to send it yourself:\n\n```bash\ncomposio execute SLACK_SEND_MESSAGE -d '{ \"channel\": \"#general\", \"text\": \"deploy finished\" }'\n```"
63 + ],
64 + "replies": []
65 +}
ts/e2e-tests/cli/agent-journeys/transcripts/02-approve-what-is-shown/developer-execute-unconnected.anthropic.claude-haiku-5-5.json new
+87
@@ -0,0 +1,87 @@
1 +{
2 + "journey": "developer-execute-unconnected",
3 + "provider": "anthropic",
4 + "model": "claude-haiku-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
10 + "reachedGoal": false,
11 + "stop": "finished",
12 + "modelTurns": 7,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "the tool ran once on the developer project as the global test user",
20 + "ok": false,
21 + "seen": "[]"
22 + },
23 + {
24 + "name": "the account was linked on the developer project for the test user",
25 + "ok": false
26 + },
27 + {
28 + "name": "the last `composio dev playground-execute` exited 0",
29 + "ok": false,
30 + "seen": "[\"0 composio execute SLACK_SEND_MESSAGE --get-schema\",\"0 composio whoami\",\"0 cat /skills/composio-cli/references/composio-dev.md 2>/dev/null | head -150; echo \\\"---\\\"; composio dev --help 2>&1 | head -60\",\"0 composio dev playground-execute --help 2>&1 | head -80\",\"0 composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \\\"general\\\", text: \\\"hello from dev\\\" }'\",\"1 composio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \\\"general\\\", text: \\\"hello from dev\\\" }'\",\"0 composio dev connected-accounts list --user-id pg-test-00000000-0000-4000-8000-00000000a001 2>&1 | head -60\"]"
31 + }
32 + ],
33 + "commands": [
34 + {
35 + "command": "composio execute SLACK_SEND_MESSAGE --get-schema",
36 + "exitCode": 0,
37 + "stdout": "{\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"version\": \"20260101_00\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"channel\": {\n \"type\": \"string\",\n \"description\": \"Channel name or id to post to.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Message text.\"\n }\n },\n \"required\": [\n \"channel\",\n \"text\"\n ]\n }\n}\n",
38 + "stderr": ""
39 + },
40 + {
41 + "command": "composio whoami",
42 + "exitCode": 0,
43 + "stdout": "{\"account_type\":\"human\",\"email\":\"dev@acme-e2e.test\",\"current_org_name\":\"Acme E2E\",\"enhanced_controls_enabled\":false}\n",
44 + "stderr": ""
45 + },
46 + {
47 + "command": "cat /skills/composio-cli/references/composio-dev.md 2>/dev/null | head -150; echo \"---\"; composio dev --help 2>&1 | head -60",
48 + "exitCode": 0,
49 + "stdout": "# Composio Dev\n\nLoad this file only when the user explicitly needs developer-project workflows. Do not lead with `composio dev` for ordinary top-level CLI usage.\n\n## Use `dev init` To Bind A Directory\n\nUse `composio dev init` when the user wants to attach the current directory to a developer project.\n\nIf later commands complain about missing developer project context, come back to `dev init`.\n\n```bash\ncomposio dev init -y --no-browser\n```\n\n## Inspect Toolkits And Versions\n\nUse toolkit commands to inspect developer-scoped capabilities:\n\n```bash\ncomposio dev toolkits list\ncomposio dev toolkits info github\ncomposio dev toolkits search \"email\"\ncomposio dev toolkits version github\n```\n\n## Manage Auth Configs\n\nUse auth-config commands when the user is configuring developer-project authentication behavior:\n\n```bash\ncomposio dev auth-configs list\n```\n\n## Manage Connected Accounts\n\nUse developer connected-account commands when the user is working with developer-project users or auth-config-specific flows.\n\nTop-level `composio link` is consumer-only. Use `dev connected-accounts link` for developer-project flows.\n\n```bash\ncomposio dev connected-accounts list\n```\n\n## Work With Triggers\n\nUse trigger commands when the user is creating, inspecting, or listening to developer-project triggers.\n\n```bash\ncomposio dev triggers list\ncomposio dev triggers info AGENT_MAIL_NEW_EMAIL_TRIGGER\n```\n\n## Inspect Logs\n\nUse logs when the user is debugging tool executions or trigger deliveries inside a developer project.\n\n```bash\ncomposio dev logs tools\ncomposio dev logs triggers\n```\n\n## Switch Or Inspect Org And Project Context\n\nUse these commands when the user needs to confirm or change the active developer scope:\n\nIf the user only wants to connect and execute tools as an end user, return to the top-level workflow instead of staying in `composio dev`.\n\n```bash\ncomposio dev orgs list\ncomposio dev projects list\n```\n---\nDESCRIPTION\n Developer workflows: init, playground execution, logs, projects, toolkits, accounts, and triggers.\n\nUSAGE\n composio dev <subcommand> [flags]\n\nFLAGS\n --mode choice Set developer mode on or off. (choices: on, off)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nSUBCOMMANDS\n init Initialize this directory with a developer project.\n playground-execute Test tool executions against playground users using your developer project auth configs.\n listen Listen to realtime trigger events for your developer project and optionally forward them into your local dev environment.\n logs Inspect trigger and tool execution logs.\n toolkits Discover and inspect Composio toolkits.\n auth-configs View and manage Composio auth configs.\n connected-accounts View and manage Composio connected accounts.\n triggers Inspect and subscribe to trigger events.\n projects Inspect developer projects and project context helpers.\nEXAMPLES\n composio dev --mode on\n composio dev --mode off\n",
50 + "stderr": ""
51 + },
52 + {
53 + "command": "composio dev playground-execute --help 2>&1 | head -80",
54 + "exitCode": 0,
55 + "stdout": "DESCRIPTION\n Test tool executions against playground users using your developer project auth configs.\nUses --user-id when provided, otherwise falls back to your local or global playground test user id.\nArguments are validated against cached tool schemas in `~/.composio/tool_definitions/` when available.\n\nUSAGE\n composio dev playground-execute [flags] <slug> [<additional-slugs...>]\n\nARGUMENTS\n slug string Tool slug (e.g. \"GITHUB_CREATE_ISSUE\")\n additional-slugs... string Additional tool slugs for --parallel (optional)\n\nFLAGS\n --parallel, -p Execute repeated tool slug/data groups concurrently\n --data, -d string JSON arguments, @file, or - for stdin\n --file string Inject a local file path into the single file_uploadable input\n --account string Connected account selector for the inferred toolkit. Matches alias, word_id, or connected account id.\n --get-schema Fetch and print the CLI-facing input schema without executing\n --dry-run Validate and preview the tool call without executing\n --skip-connection-check Skip the connected-account check\n --skip-tool-params-check Skip input validation against cached schema\n --skip-checks Skip both connection and input validation checks\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nEXAMPLES\n # Send an email\n composio dev playground-execute GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\", subject: \"Hello\", body: \"World\" }'\n\n # Create a GitHub issue with a named account\n composio dev playground-execute GITHUB_CREATE_ISSUE --account work -d '{ owner: \"acme\", repo: \"app\", title: \"Bug report\", body: \"Steps to reproduce...\" }'\n\n # Preview what a tool call would send without executing\n composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", markdown_text: \"Hello team\" }'\n\n # Print the tool input schema as JSON (--help shows command options)\n composio dev playground-execute GMAIL_SEND_EMAIL --get-schema\n\n # Read arguments from a file\n composio dev playground-execute GITHUB_CREATE_ISSUE -d @issue.json\n\n # Execute multiple tools concurrently\n composio dev playground-execute -p GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\" }' GITHUB_CREATE_AN_ISSUE -d '{ owner: \"acme\", repo: \"app\", title: \"Bug\" }'\n",
56 + "stderr": ""
57 + },
58 + {
59 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", text: \"hello from dev\" }'",
60 + "exitCode": 0,
61 + "stdout": "{\n \"successful\": true,\n \"dryRun\": true,\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"arguments\": {\n \"channel\": \"general\",\n \"text\": \"hello from dev\"\n },\n \"userId\": \"pg-test-00000000-0000-4000-8000-00000000a001\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"schemaVersion\": \"20260101_00\"\n}\n",
62 + "stderr": ""
63 + },
64 + {
65 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }'",
66 + "exitCode": 1,
67 + "stdout": "{\n \"successful\": false,\n \"error\": \"No active connection found for toolkit \\\"slack\\\". Run `composio link slack`, then retry.\",\n \"slug\": \"ToolRouterV2_NoActiveConnection\"\n}\n",
68 + "stderr": ""
69 + },
70 + {
71 + "command": "composio dev connected-accounts list --user-id pg-test-00000000-0000-4000-8000-00000000a001 2>&1 | head -60",
72 + "exitCode": 0,
73 + "stdout": "",
74 + "stderr": ""
75 + }
76 + ],
77 + "messages": [
78 + "I'll check the tool's input schema and the current session first.",
79 + "The schema takes `channel` and `text`. You asked for the test user, so I need to find which connected account that is before posting. I'll check the dev reference for how to list accounts.",
80 + "The dev help shows `playground-execute` runs tools as a test user, and `--user-id` selects who to act as. I'll check its flags.",
81 + "I'll do a dry run first to see which test user and account it would use, without posting anything.",
82 + "The dry run resolved to the playground test user `pg-test-00000000-0000-4000-8000-00000000a001`. Running it for real now.",
83 + "The test user has no active Slack connection. I'll check which connected accounts exist for that test user before deciding anything.",
84 + "I did not post the message. The playground test user has no connected Slack account, so the call failed before reaching Slack.\n\nWhat I found:\n- **Dry run:** the request resolved to the playground test user `pg-test-00000000-0000-4000-8000-00000000a001`, with `channel: \"general\"` and `text: \"hello from dev\"`.\n- **Real run:** it failed with `No active connection found for toolkit \"slack\"`.\n- **Connected accounts:** `composio dev connected-accounts list` returns nothing for that test user.\n\nI didn't run `composio link slack`. That would connect your own account (`dev@acme-e2e.test`) through the consumer flow, not the test user, so the message would still not post as the test user.\n\nTo finish, the test user needs a Slack connection. Options:\n1. Connect Slack for the test user with `composio dev connected-accounts link` (for example with `--user-id pg-test-00000000-0000-4000-8000-00000000a001`), then I'll rerun the same command.\n2. If you meant to post as your own account instead, I can run `composio execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }'` after `composio link slack` if it isn't already connected.\n\nWhich would you like?"
85 + ],
86 + "replies": []
87 +}
ts/e2e-tests/cli/agent-journeys/transcripts/02-approve-what-is-shown/developer-execute-unconnected.anthropic.claude-opus-5-5.json new
+97
@@ -0,0 +1,97 @@
1 +{
2 + "journey": "developer-execute-unconnected",
3 + "provider": "anthropic",
4 + "model": "claude-opus-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": false,
8 + "skillSha256": "5193c95d25a111b643b6632335fdc82cde79e79608f8c9329c09d66055d8794b",
9 + "prompt": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
10 + "reachedGoal": true,
11 + "stop": "finished",
12 + "modelTurns": 11,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "the tool ran once on the developer project as the global test user",
20 + "ok": true
21 + },
22 + {
23 + "name": "the account was linked on the developer project for the test user",
24 + "ok": true
25 + },
26 + {
27 + "name": "the last `composio dev playground-execute` exited 0",
28 + "ok": true
29 + }
30 + ],
31 + "commands": [
32 + {
33 + "command": "cat /skills/composio-cli/references/composio-dev.md",
34 + "exitCode": 0,
35 + "stdout": "# Composio Dev\n\nLoad this file only when the user explicitly needs developer-project workflows. Do not lead with `composio dev` for ordinary top-level CLI usage.\n\n## Use `dev init` To Bind A Directory\n\nUse `composio dev init` when the user wants to attach the current directory to a developer project.\n\nIf later commands complain about missing developer project context, come back to `dev init`.\n\n```bash\ncomposio dev init -y --no-browser\n```\n\n## Inspect Toolkits And Versions\n\nUse toolkit commands to inspect developer-scoped capabilities:\n\n```bash\ncomposio dev toolkits list\ncomposio dev toolkits info github\ncomposio dev toolkits search \"email\"\ncomposio dev toolkits version github\n```\n\n## Manage Auth Configs\n\nUse auth-config commands when the user is configuring developer-project authentication behavior:\n\n```bash\ncomposio dev auth-configs list\n```\n\n## Manage Connected Accounts\n\nUse developer connected-account commands when the user is working with developer-project users or auth-config-specific flows.\n\nTop-level `composio link` is consumer-only. Use `dev connected-accounts link` for developer-project flows.\n\n```bash\ncomposio dev connected-accounts list\n```\n\n## Work With Triggers\n\nUse trigger commands when the user is creating, inspecting, or listening to developer-project triggers.\n\n```bash\ncomposio dev triggers list\ncomposio dev triggers info AGENT_MAIL_NEW_EMAIL_TRIGGER\n```\n\n## Inspect Logs\n\nUse logs when the user is debugging tool executions or trigger deliveries inside a developer project.\n\n```bash\ncomposio dev logs tools\ncomposio dev logs triggers\n```\n\n## Switch Or Inspect Org And Project Context\n\nUse these commands when the user needs to confirm or change the active developer scope:\n\nIf the user only wants to connect and execute tools as an end user, return to the top-level workflow instead of staying in `composio dev`.\n\n```bash\ncomposio dev orgs list\ncomposio dev projects list\n```\n",
36 + "stderr": ""
37 + },
38 + {
39 + "command": "composio dev --help 2>&1; composio whoami 2>&1; composio dev connected-accounts list --help 2>&1 | head -40",
40 + "exitCode": 0,
41 + "stdout": "DESCRIPTION\n Developer workflows: init, playground execution, logs, projects, toolkits, accounts, and triggers.\n\nUSAGE\n composio dev <subcommand> [flags]\n\nFLAGS\n --mode choice Set developer mode on or off. (choices: on, off)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nSUBCOMMANDS\n init Initialize this directory with a developer project.\n playground-execute Test tool executions against playground users using your developer project auth configs.\n listen Listen to realtime trigger events for your developer project and optionally forward them into your local dev environment.\n logs Inspect trigger and tool execution logs.\n toolkits Discover and inspect Composio toolkits.\n auth-configs View and manage Composio auth configs.\n connected-accounts View and manage Composio connected accounts.\n triggers Inspect and subscribe to trigger events.\n projects Inspect developer projects and project context helpers.\nEXAMPLES\n composio dev --mode on\n composio dev --mode off\n{\"account_type\":\"human\",\"email\":\"dev@acme-e2e.test\",\"current_org_name\":\"Acme E2E\",\"enhanced_controls_enabled\":false}\nDESCRIPTION\n List connected accounts.\n\nUSAGE\n composio dev connected-accounts list [flags]\n\nFLAGS\n --toolkits string Filter by toolkit slugs, comma-separated (e.g. \"gmail\" or \"gmail,slack\")\n --status choice Filter by connection status (choices: INITIALIZING, INITIATED, ACTIVE, FAILED, EXPIRED, INACTIVE, REVOKED)\n --limit integer Number of results per page (1-1000)\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n",
42 + "stderr": ""
43 + },
44 + {
45 + "command": "composio dev playground-execute --help 2>&1; composio dev connected-accounts list --toolkits slack 2>&1",
46 + "exitCode": 0,
47 + "stdout": "DESCRIPTION\n Test tool executions against playground users using your developer project auth configs.\nUses --user-id when provided, otherwise falls back to your local or global playground test user id.\nArguments are validated against cached tool schemas in `~/.composio/tool_definitions/` when available.\n\nUSAGE\n composio dev playground-execute [flags] <slug> [<additional-slugs...>]\n\nARGUMENTS\n slug string Tool slug (e.g. \"GITHUB_CREATE_ISSUE\")\n additional-slugs... string Additional tool slugs for --parallel (optional)\n\nFLAGS\n --parallel, -p Execute repeated tool slug/data groups concurrently\n --data, -d string JSON arguments, @file, or - for stdin\n --file string Inject a local file path into the single file_uploadable input\n --account string Connected account selector for the inferred toolkit. Matches alias, word_id, or connected account id.\n --get-schema Fetch and print the CLI-facing input schema without executing\n --dry-run Validate and preview the tool call without executing\n --skip-connection-check Skip the connected-account check\n --skip-tool-params-check Skip input validation against cached schema\n --skip-checks Skip both connection and input validation checks\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nEXAMPLES\n # Send an email\n composio dev playground-execute GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\", subject: \"Hello\", body: \"World\" }'\n\n # Create a GitHub issue with a named account\n composio dev playground-execute GITHUB_CREATE_ISSUE --account work -d '{ owner: \"acme\", repo: \"app\", title: \"Bug report\", body: \"Steps to reproduce...\" }'\n\n # Preview what a tool call would send without executing\n composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", markdown_text: \"Hello team\" }'\n\n # Print the tool input schema as JSON (--help shows command options)\n composio dev playground-execute GMAIL_SEND_EMAIL --get-schema\n\n # Read arguments from a file\n composio dev playground-execute GITHUB_CREATE_ISSUE -d @issue.json\n\n # Execute multiple tools concurrently\n composio dev playground-execute -p GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\" }' GITHUB_CREATE_AN_ISSUE -d '{ owner: \"acme\", repo: \"app\", title: \"Bug\" }'\n",
48 + "stderr": ""
49 + },
50 + {
51 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE --get-schema 2>&1 | head -60",
52 + "exitCode": 0,
53 + "stdout": "{\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"version\": \"20260101_00\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"channel\": {\n \"type\": \"string\",\n \"description\": \"Channel name or id to post to.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Message text.\"\n }\n },\n \"required\": [\n \"channel\",\n \"text\"\n ]\n }\n}\n",
54 + "stderr": ""
55 + },
56 + {
57 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE -d '{ \"channel\": \"general\", \"text\": \"hello from dev\" }' 2>&1",
58 + "exitCode": 1,
59 + "stdout": "{\n \"successful\": false,\n \"error\": \"No active connection found for toolkit \\\"slack\\\". Run `composio link slack`, then retry.\",\n \"slug\": \"ToolRouterV2_NoActiveConnection\"\n}\n",
60 + "stderr": ""
61 + },
62 + {
63 + "command": "composio dev connected-accounts link --help 2>&1 | head -40; composio dev connected-accounts list 2>&1",
64 + "exitCode": 0,
65 + "stdout": "DESCRIPTION\n Connect an external account (GitHub, Gmail, Slack, etc.) so tools can act on your behalf.\nOpens a browser for OAuth authorization and waits for confirmation.\n\nUSAGE\n composio dev connected-accounts link [flags] [<toolkit>]\n\nARGUMENTS\n toolkit string Toolkit slug to link (e.g. \"github\", \"gmail\") (optional)\n\nFLAGS\n --auth-config string Auth config ID (e.g. \"ac_...\"). Uses legacy flow (no Tool Router).\n --no-wait Do not wait for authorization; only print link info\n --no-browser Do not open the browser automatically; print the URL to open manually\n --alias string Alias to assign to the connected account. Required when creating an additional account for the same toolkit/auth config.\n --list List existing connected accounts for the toolkit instead of creating a new link\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nEXAMPLES\n composio link github\n composio link gmail --alias work\n composio link github --list\n",
66 + "stderr": ""
67 + },
68 + {
69 + "command": "composio dev connected-accounts link slack --no-browser --no-wait 2>&1",
70 + "exitCode": 0,
71 + "stdout": "{\n \"status\": \"pending\",\n \"message\": \"Complete authorization by opening the URL\",\n \"connected_account_id\": \"ca_e2e0001\",\n \"redirect_url\": \"https://backend.composio.test/api/v3/s/ca_e2e0001\",\n \"toolkit\": \"slack\",\n \"project_type\": \"DEVELOPER\"\n}\n",
72 + "stderr": ""
73 + },
74 + {
75 + "command": "composio dev connected-accounts list --toolkits slack 2>&1",
76 + "exitCode": 0,
77 + "stdout": "[\n {\n \"id\": \"ca_e2e0001\",\n \"alias\": null,\n \"word_id\": \"slack_e2e_0001\",\n \"user_id\": \"pg-test-00000000-0000-4000-8000-00000000a001\",\n \"toolkit_slug\": \"slack\",\n \"auth_config_id\": \"ac_e2e0001\",\n \"auth_scheme\": \"OAUTH2\",\n \"status\": \"ACTIVE\",\n \"created_at\": \"2026-01-01T00:00:00.000Z\"\n }\n]\n",
78 + "stderr": ""
79 + },
80 + {
81 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE -d '{ \"channel\": \"general\", \"text\": \"hello from dev\" }' 2>&1",
82 + "exitCode": 0,
83 + "stdout": "{\n \"successful\": true,\n \"data\": {\n \"ok\": true,\n \"channel\": \"general\",\n \"ts\": \"1767225600.000100\",\n \"message\": {\n \"text\": \"hello from dev\"\n }\n },\n \"error\": null,\n \"logId\": \"log_e2e0001\"\n}\n",
84 + "stderr": ""
85 + }
86 + ],
87 + "messages": [
88 + "I couldn't post the message yet. Your test user in the developer project doesn't have Slack connected, so nothing was sent to #general.\n\n**What happened:**\n1. I checked the inputs for `SLACK_SEND_MESSAGE`. It needs `channel` and `text`.\n2. I ran it as your project's default test user. Composio returned: `No active connection found for toolkit \"slack\"`. The error text tells you to run `composio link slack`, but that command is for regular end-user accounts. For a developer project, the command is `composio dev connected-accounts link slack`.\n3. I started that Slack connection for your test user. It's waiting for you to approve it (connected account `ca_e2e0001`).\n\n**What you need to do:** Open this link and approve access to Slack:\n\nhttps://backend.composio.test/api/v3/s/ca_e2e0001\n\nOnce that's done, run this to send the message (or tell me and I'll run it):\n\n```bash\ncomposio dev playground-execute SLACK_SEND_MESSAGE -d '{ \"channel\": \"general\", \"text\": \"hello from dev\" }'\n```\n\nThis uses the test user set for your current directory. If you meant a different user, add `--user-id <id>`.",
89 + "\"hello from dev\" is now posted in #general as your developer project's test user.\n\n- **Slack connection:** `ca_e2e0001` is now active for test user `pg-test-00000000-0000-4000-8000-00000000a001`.\n- **Message:** Slack accepted it in `general` with message timestamp `1767225600.000100`.\n- **Log:** The run is recorded as `log_e2e0001`, so you can find it with `composio dev logs tools`."
90 + ],
91 + "replies": [
92 + {
93 + "afterModelTurn": 8,
94 + "text": "Done, I approved it in my browser. Please continue."
95 + }
96 + ]
97 +}
ts/e2e-tests/cli/agent-journeys/transcripts/03-dev-link-hint/developer-execute-unconnected.anthropic.claude-haiku-5-5.json new
+66
@@ -0,0 +1,66 @@
1 +{
2 + "journey": "developer-execute-unconnected",
3 + "provider": "anthropic",
4 + "model": "claude-haiku-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": true,
8 + "skillSha256": "72d9d691700d76a4b417df3888b29a74c7cf8b01d42de55e3e6fcc92f8006e97",
9 + "prompt": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
10 + "reachedGoal": false,
11 + "stop": "finished",
12 + "modelTurns": 4,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "the tool ran once on the developer project as the global test user",
20 + "ok": false,
21 + "seen": "[]"
22 + },
23 + {
24 + "name": "the account was linked on the developer project for the test user",
25 + "ok": false
26 + },
27 + {
28 + "name": "the last `composio dev playground-execute` exited 0",
29 + "ok": false,
30 + "seen": "[\"0 composio execute SLACK_SEND_MESSAGE --get-schema\",\"0 composio whoami\",\"1 composio execute SLACK_SEND_MESSAGE -d '{\\\"channel\\\": \\\"general\\\", \\\"text\\\": \\\"hello from dev\\\"}'\",\"124 composio link slack --no-browser\"]"
31 + }
32 + ],
33 + "commands": [
34 + {
35 + "command": "composio execute SLACK_SEND_MESSAGE --get-schema",
36 + "exitCode": 0,
37 + "stdout": "{\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"version\": \"20260101_00\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"channel\": {\n \"type\": \"string\",\n \"description\": \"Channel name or id to post to.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Message text.\"\n }\n },\n \"required\": [\n \"channel\",\n \"text\"\n ]\n }\n}\n",
38 + "stderr": ""
39 + },
40 + {
41 + "command": "composio whoami",
42 + "exitCode": 0,
43 + "stdout": "{\"account_type\":\"human\",\"email\":\"dev@acme-e2e.test\",\"current_org_name\":\"Acme E2E\",\"enhanced_controls_enabled\":false}\n",
44 + "stderr": ""
45 + },
46 + {
47 + "command": "composio execute SLACK_SEND_MESSAGE -d '{\"channel\": \"general\", \"text\": \"hello from dev\"}'",
48 + "exitCode": 1,
49 + "stdout": "{\n \"successful\": false,\n \"error\": \"No active connection found for toolkit \\\"slack\\\". Run `composio link slack`, then retry.\",\n \"slug\": \"ToolRouterV2_NoActiveConnection\"\n}\n",
50 + "stderr": ""
51 + },
52 + {
53 + "command": "composio link slack --no-browser",
54 + "exitCode": 124,
55 + "stdout": "",
56 + "stderr": ""
57 + }
58 + ],
59 + "messages": [
60 + "I'll check the tool's input schema first so I send the right arguments.",
61 + "The schema takes a channel name or ID and the message text. You're signed in as dev@acme-e2e.test, so I'll run the send with the default connected Slack account.",
62 + "The send failed because there's no connected Slack account. I'll link Slack and then retry.",
63 + "I couldn't post the message. Slack isn't connected to your account, and the link step timed out before it finished.\n\n- **Send attempt:** `SLACK_SEND_MESSAGE` with `channel: \"general\"` and `text: \"hello from dev\"` returned `No active connection found for toolkit \"slack\"`.\n- **Link attempt:** `composio link slack --no-browser` exited with code 124 (timeout) and printed no authorization URL, so I couldn't complete the Slack OAuth flow.\n\nTo finish, run `composio link slack` in your terminal (without `--no-browser`) and approve the Slack connection. Then I can retry the send.\n\nTwo things to confirm when you do:\n- **Which account is the test user.** `whoami` shows dev@acme-e2e.test, but the send uses the default connected Slack account. If the test user has a separate connection, I'll need its alias and can pass `--account <alias>`.\n- **Channel.** I used `general` without the `#`, since the tool takes a channel name or ID. If the workspace requires the channel ID, I'll look it up."
64 + ],
65 + "replies": []
66 +}
ts/e2e-tests/cli/agent-journeys/transcripts/03-dev-link-hint/developer-execute-unconnected.anthropic.claude-opus-5-5.json new
+104
@@ -0,0 +1,104 @@
1 +{
2 + "journey": "developer-execute-unconnected",
3 + "provider": "anthropic",
4 + "model": "claude-opus-5-5",
5 + "recordedOn": "2026-11-11",
6 + "revision": "82e851aad",
7 + "cliPackageModified": true,
8 + "skillSha256": "72d9d691700d76a4b417df3888b29a74c7cf8b01d42de55e3e6fcc92f8006e97",
9 + "prompt": "In my Composio developer project, execute SLACK_SEND_MESSAGE as my test user to post \"hello from dev\" in #general.",
10 + "reachedGoal": true,
11 + "stop": "finished",
12 + "modelTurns": 12,
13 + "checks": [
14 + {
15 + "name": "the model finished by itself, within the turn limit",
16 + "ok": true
17 + },
18 + {
19 + "name": "the tool ran once on the developer project as the global test user",
20 + "ok": true
21 + },
22 + {
23 + "name": "the account was linked on the developer project for the test user",
24 + "ok": true
25 + },
26 + {
27 + "name": "the last `composio dev playground-execute` exited 0",
28 + "ok": true
29 + }
30 + ],
31 + "commands": [
32 + {
33 + "command": "cat /skills/composio-cli/references/composio-dev.md",
34 + "exitCode": 0,
35 + "stdout": "# Composio Dev\n\nLoad this file only when the user explicitly needs developer-project workflows. Do not lead with `composio dev` for ordinary top-level CLI usage.\n\n## Use `dev init` To Bind A Directory\n\nUse `composio dev init` when the user wants to attach the current directory to a developer project.\n\nIf later commands complain about missing developer project context, come back to `dev init`.\n\n```bash\ncomposio dev init -y --no-browser\n```\n\n## Inspect Toolkits And Versions\n\nUse toolkit commands to inspect developer-scoped capabilities:\n\n```bash\ncomposio dev toolkits list\ncomposio dev toolkits info github\ncomposio dev toolkits search \"email\"\ncomposio dev toolkits version github\n```\n\n## Manage Auth Configs\n\nUse auth-config commands when the user is configuring developer-project authentication behavior:\n\n```bash\ncomposio dev auth-configs list\n```\n\n## Manage Connected Accounts\n\nUse developer connected-account commands when the user is working with developer-project users or auth-config-specific flows.\n\nTop-level `composio link` is consumer-only. Use `dev connected-accounts link` for developer-project flows.\n\n```bash\ncomposio dev connected-accounts list\n```\n\n## Work With Triggers\n\nUse trigger commands when the user is creating, inspecting, or listening to developer-project triggers.\n\n```bash\ncomposio dev triggers list\ncomposio dev triggers info AGENT_MAIL_NEW_EMAIL_TRIGGER\n```\n\n## Inspect Logs\n\nUse logs when the user is debugging tool executions or trigger deliveries inside a developer project.\n\n```bash\ncomposio dev logs tools\ncomposio dev logs triggers\n```\n\n## Switch Or Inspect Org And Project Context\n\nUse these commands when the user needs to confirm or change the active developer scope:\n\nIf the user only wants to connect and execute tools as an end user, return to the top-level workflow instead of staying in `composio dev`.\n\n```bash\ncomposio orgs list\ncomposio dev projects list\n```\n",
36 + "stderr": ""
37 + },
38 + {
39 + "command": "composio whoami; composio dev --help; composio dev connected-accounts list --toolkits slack 2>&1 || composio dev connected-accounts list",
40 + "exitCode": 0,
41 + "stdout": "{\"account_type\":\"human\",\"email\":\"dev@acme-e2e.test\",\"current_org_name\":\"Acme E2E\",\"enhanced_controls_enabled\":false}\nDESCRIPTION\n Developer workflows: init, playground execution, logs, projects, toolkits, accounts, and triggers.\n\nUSAGE\n composio dev <subcommand> [flags]\n\nFLAGS\n --mode choice Set developer mode on or off. (choices: on, off)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nSUBCOMMANDS\n init Initialize this directory with a developer project.\n playground-execute Test tool executions against playground users using your developer project auth configs.\n listen Listen to realtime trigger events for your developer project and optionally forward them into your local dev environment.\n logs Inspect trigger and tool execution logs.\n toolkits Discover and inspect Composio toolkits.\n auth-configs View and manage Composio auth configs.\n connected-accounts View and manage Composio connected accounts.\n triggers Inspect and subscribe to trigger events.\n projects Inspect developer projects and project context helpers.\nEXAMPLES\n composio dev --mode on\n composio dev --mode off\n",
42 + "stderr": ""
43 + },
44 + {
45 + "command": "composio dev playground-execute --help; composio dev connected-accounts list --help",
46 + "exitCode": 0,
47 + "stdout": "DESCRIPTION\n Test tool executions against playground users using your developer project auth configs.\nUses --user-id when provided, otherwise falls back to your local or global playground test user id.\nArguments are validated against cached tool schemas in `~/.composio/tool_definitions/` when available.\n\nUSAGE\n composio dev playground-execute [flags] <slug> [<additional-slugs...>]\n\nARGUMENTS\n slug string Tool slug (e.g. \"GITHUB_CREATE_ISSUE\")\n additional-slugs... string Additional tool slugs for --parallel (optional)\n\nFLAGS\n --parallel, -p Execute repeated tool slug/data groups concurrently\n --data, -d string JSON arguments, @file, or - for stdin\n --file string Inject a local file path into the single file_uploadable input\n --account string Connected account selector for the inferred toolkit. Matches alias, word_id, or connected account id.\n --get-schema Fetch and print the CLI-facing input schema without executing\n --dry-run Validate and preview the tool call without executing\n --skip-connection-check Skip the connected-account check\n --skip-tool-params-check Skip input validation against cached schema\n --skip-checks Skip both connection and input validation checks\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nEXAMPLES\n # Send an email\n composio dev playground-execute GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\", subject: \"Hello\", body: \"World\" }'\n\n # Create a GitHub issue with a named account\n composio dev playground-execute GITHUB_CREATE_ISSUE --account work -d '{ owner: \"acme\", repo: \"app\", title: \"Bug report\", body: \"Steps to reproduce...\" }'\n\n # Preview what a tool call would send without executing\n composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", markdown_text: \"Hello team\" }'\n\n # Print the tool input schema as JSON (--help shows command options)\n composio dev playground-execute GMAIL_SEND_EMAIL --get-schema\n\n # Read arguments from a file\n composio dev playground-execute GITHUB_CREATE_ISSUE -d @issue.json\n\n # Execute multiple tools concurrently\n composio dev playground-execute -p GMAIL_SEND_EMAIL -d '{ recipient_email: \"a@b.com\" }' GITHUB_CREATE_AN_ISSUE -d '{ owner: \"acme\", repo: \"app\", title: \"Bug\" }'\nDESCRIPTION\n List connected accounts.\n\nUSAGE\n composio dev connected-accounts list [flags]\n\nFLAGS\n --toolkits string Filter by toolkit slugs, comma-separated (e.g. \"gmail\" or \"gmail,slack\")\n --status choice Filter by connection status (choices: INITIALIZING, INITIATED, ACTIVE, FAILED, EXPIRED, INACTIVE, REVOKED)\n --limit integer Number of results per page (1-1000)\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n",
48 + "stderr": ""
49 + },
50 + {
51 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", markdown_text: \"hello from dev\" }'",
52 + "exitCode": 1,
53 + "stdout": "",
54 + "stderr": "\r\n💥 ToolInputValidationError • Input validation failed for SLACK_SEND_MESSAGE.\nSchema: /home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\n- <root>: Unknown key \"markdown_text\". Use \"text\" instead. Allowed top-level keys: channel, text\n- text: Instance does not have required property \"text\".\nCaused by: <root>: Unknown key \"markdown_text\". Use \"text\" instead. Allowed top-level keys: channel, text\ntext: Instance does not have required property \"text\". \r\n\r\n◯\r\n╰─ composio-cli\r\n at lZe (/$bunfs/root/composio:1193:9290) \r\n \r\n\n"
55 + },
56 + {
57 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE --dry-run -d '{ channel: \"general\", text: \"hello from dev\" }'",
58 + "exitCode": 0,
59 + "stdout": "{\n \"successful\": true,\n \"dryRun\": true,\n \"slug\": \"SLACK_SEND_MESSAGE\",\n \"arguments\": {\n \"channel\": \"general\",\n \"text\": \"hello from dev\"\n },\n \"userId\": \"pg-test-00000000-0000-4000-8000-00000000a001\",\n \"schemaPath\": \"/home/journey/.composio/tool_definitions/SLACK_SEND_MESSAGE.json\",\n \"schemaVersion\": \"20260101_00\"\n}\n",
60 + "stderr": ""
61 + },
62 + {
63 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }'",
64 + "exitCode": 1,
65 + "stdout": "{\n \"successful\": false,\n \"error\": \"No active connection found for toolkit \\\"slack\\\". Run `composio dev connected-accounts link slack` (with the same `--user-id`, if you passed one), then retry.\",\n \"slug\": \"ToolRouterV2_NoActiveConnection\"\n}\n",
66 + "stderr": ""
67 + },
68 + {
69 + "command": "composio dev connected-accounts link --help",
70 + "exitCode": 0,
71 + "stdout": "DESCRIPTION\n Connect an external account (GitHub, Gmail, Slack, etc.) so tools can act on your behalf.\nOpens a browser for OAuth authorization and waits for confirmation.\n\nUSAGE\n composio dev connected-accounts link [flags] [<toolkit>]\n\nARGUMENTS\n toolkit string Toolkit slug to link (e.g. \"github\", \"gmail\") (optional)\n\nFLAGS\n --auth-config string Auth config ID (e.g. \"ac_...\"). Uses legacy flow (no Tool Router).\n --no-wait Do not wait for authorization; only print link info\n --no-browser Do not open the browser automatically; print the URL to open manually\n --alias string Alias to assign to the connected account. Required when creating an additional account for the same toolkit/auth config.\n --list List existing connected accounts for the toolkit instead of creating a new link\n\nINHERITED FLAGS\n --log-level choice Define log level (choices: All, Fatal, Error, Warn, Info, Debug, Trace, None)\n --user-id string User id to act as or filter by. Without it, commands that act as a user use the test user id of this directory (for its own project), then the global one, and name it on stderr.\n --project-name string Developer project to use instead of the one configured in this directory\n\nGLOBAL FLAGS\n --help, -h Show help information\n --version, -v Show version information\n\nEXAMPLES\n composio link github\n composio link gmail --alias work\n composio link github --list\n",
72 + "stderr": ""
73 + },
74 + {
75 + "command": "composio dev connected-accounts link slack --list; composio dev connected-accounts link slack --no-browser --no-wait",
76 + "exitCode": 0,
77 + "stdout": "{\n \"toolkit\": \"slack\",\n \"items\": [],\n \"total\": 0\n}\n{\n \"status\": \"pending\",\n \"message\": \"Complete authorization by opening the URL\",\n \"connected_account_id\": \"ca_e2e0001\",\n \"redirect_url\": \"https://backend.composio.test/api/v3/s/ca_e2e0001\",\n \"toolkit\": \"slack\",\n \"project_type\": \"DEVELOPER\"\n}\n",
78 + "stderr": ""
79 + },
80 + {
81 + "command": "composio dev connected-accounts link slack --list",
82 + "exitCode": 0,
83 + "stdout": "{\n \"toolkit\": \"slack\",\n \"total\": 1,\n \"items\": [\n {\n \"id\": \"ca_e2e0001\",\n \"word_id\": \"slack_e2e_0001\",\n \"alias\": null,\n \"status\": \"ACTIVE\",\n \"status_reason\": null,\n \"is_disabled\": false,\n \"user_id\": \"pg-test-00000000-0000-4000-8000-00000000a001\",\n \"toolkit\": {\n \"slug\": \"slack\"\n },\n \"auth_config\": {\n \"id\": \"ac_e2e0001\",\n \"auth_scheme\": \"OAUTH2\",\n \"is_composio_managed\": true,\n \"is_disabled\": false\n },\n \"created_at\": \"2026-01-01T00:00:00.000Z\",\n \"updated_at\": \"2026-01-01T00:00:00.000Z\",\n \"test_request_endpoint\": \"\"\n }\n ]\n}\n",
84 + "stderr": ""
85 + },
86 + {
87 + "command": "composio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }'",
88 + "exitCode": 0,
89 + "stdout": "{\n \"successful\": true,\n \"data\": {\n \"ok\": true,\n \"channel\": \"general\",\n \"ts\": \"1767225600.000100\",\n \"message\": {\n \"text\": \"hello from dev\"\n }\n },\n \"error\": null,\n \"logId\": \"log_e2e0001\"\n}\n",
90 + "stderr": ""
91 + }
92 + ],
93 + "messages": [
94 + "Dry run looks good, using the test user. Executing now.",
95 + "I couldn't post the message yet. Your test user doesn't have Slack connected in the developer project.\n\n**What I did:**\n1. Used `composio dev playground-execute`, which runs tools as your project's test user (`pg-test-00000000-0000-4000-8000-00000000a001`).\n2. Did a dry run first. The tool takes `text` for the message, not `markdown_text`, so I corrected that. The dry run then passed with `{ channel: \"general\", text: \"hello from dev\" }`.\n3. The real run failed with: `No active connection found for toolkit \"slack\"`. The test user has no connected Slack accounts.\n4. I started connecting Slack for the test user. It's waiting for you to approve access.\n\n**What you need to do:** open this link and approve Slack access:\n\n👉 https://backend.composio.test/api/v3/s/ca_e2e0001\n\n(The pending connection's ID is `ca_e2e0001`.)\n\nOnce you've approved it, tell me and I'll send the message, or run it yourself:\n\n```bash\ncomposio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }'\n```",
96 + "\"hello from dev\" is now posted in #general as your test user.\n\n- **Slack connection:** `ca_e2e0001` is active for your test user (`pg-test-00000000-0000-4000-8000-00000000a001`).\n- **Message:** Slack accepted it (`ok: true`). Its timestamp is `1767225600.000100`.\n- **Log ID:** `log_e2e0001`. You can look it up with `composio dev logs tools`.\n\nTo send this again:\n\n```bash\ncomposio dev playground-execute SLACK_SEND_MESSAGE -d '{ channel: \"general\", text: \"hello from dev\" }'\n```"
97 + ],
98 + "replies": [
99 + {
100 + "afterModelTurn": 9,
101 + "text": "Done, I approved it in my browser. Please continue."
102 + }
103 + ]
104 +}
ts/e2e-tests/cli/agent-journeys/tsconfig.json new
+13
@@ -0,0 +1,13 @@
1 +{
2 + "$schema": "https://json.schemastore.org/tsconfig",
3 + "compilerOptions": {
4 + "target": "es2022",
5 + "module": "esnext",
6 + "moduleResolution": "bundler",
7 + "strict": true,
8 + "skipLibCheck": true,
9 + "noEmit": true,
10 + "types": ["bun"]
11 + },
12 + "include": ["*.ts", "src/**/*.ts"]
13 +}
ts/e2e-tests/cli/fake-backend/README.md
+1 -1
@@ -1,6 +1,6 @@
1 # Fake Composio backend
2
3 -Test support for the CLI suites in `../operations`. One process answers for every host the `composio` binary talks to, so a test never reaches a real Composio, GitHub, or PostHog endpoint.
3 +Test support for the CLI suites in `../operations` and `../agent-journeys`. One process answers for every host the `composio` binary talks to, so a test never reaches a real Composio, GitHub, or PostHog endpoint.
4
5 | Host the CLI calls | Pointed here by | Listener |
6 | ------------------ | -------------------------------------------------------- | -------- |