@devries-cameron20 / composio Mirror · read-only / commits / 5fbf6ba97
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 04:45 UTC

test(ts/e2e): add CLI operation suites and the fault catalogue (#5352)

This PR: - part of [PRDE-2212](https://linear.app/composio/issue/PRDE-2212/cli-e2e-tests-fake-backend-operation-suites-and-agent-journeys-with) - builds on top of https://github.com/ComposioHQ/composio/pull/5351; split from https://github.com/ComposioHQ/composio/pull/5312 - adds 133 operation tests for identity across root and `dev` commands, login, sessions, tool execution, and network, auth, filesystem, process, stream, and upgrade faults - documents the fault catalogue and fake-backend limitations in `operations/README.md` - pins CLI defects with `it.failing`: rejected-key `whoami`, unscoped auth-config reads, malformed search responses and rejected-key errors, empty piped `search --human`, corrupt schema caches, rejected login keys, and `EPIPE` crashes - pairs expected failures with passing `today:` checks where a crash or timeout could otherwise hide a different failure, preserving coverage for the open fixes in #5333 and #5334 - adds `operations` to the keyless CI step and covers the link URL, wait, and help fixes from https://github.com/ComposioHQ/composio/pull/5326 and https://github.com/ComposioHQ/composio/pull/5327 ## Context Frozen install, CLI dependency builds, package typechecks, authored-file Prettier, and all 133 keyless operation tests passed; the inherited 31 contract and five upgrade tests also passed. The generated lockfile retains pnpm formatting, which already differs from Prettier on the base. Two known gaps stay pinned as they behave today: `composio dev link`, which does not exist, never names `composio dev connected-accounts link`, and `composio dev link --help` exits 0. The tests use the real CLI binary against the fake services, with no API keys. Complete CLI workflow validation after the base merges and this PR is retargeted to `next`.

Alberto Schiabel committed Nov 16, 2026 at 12:41 UTC 5fbf6ba972178e4dec62e66eb03842e3ddcc5f95
17 files changed +2607 -9
.agents/skills/cli-e2e/references/docker-cli-e2e.md
+2
@@ -13,6 +13,8 @@ Key properties:
13 - `HOME=/tmp`; auth and API state must come from env vars or command setup.
14 - stdout is not a TTY, so piped-mode output rules apply.
15
16 +A test that needs several commands to share state (log in, then link, then execute) uses a journey container instead: `startJourneyContainer` from `@e2e-tests/utils`, or `startCliRig` from `@e2e-tests/cli-fake-backend/rig` when the commands need a backend. See `ts/e2e-tests/cli/operations/README.md` for the fault catalogue those suites implement and `ts/e2e-tests/cli/fake-backend/README.md` for what the fake backend does and does not verify.
17 +
18 ## Package Template
19
20 ```json
.github/workflows/ts.test-e2e.yml
+2
@@ -230,6 +230,7 @@ jobs:
230 run: >-
231 pnpm exec turbo test:e2e:cli
232 --filter=@e2e-tests/cli-fake-backend
233 + --filter=@e2e-tests/cli-operations
234
235 # After the keyless step, so a missing key cannot keep those suites from running.
236 - name: Check API keys
@@ -264,6 +265,7 @@ jobs:
265 pnpm exec turbo test:e2e:cli
266 --filter='@e2e-tests/cli-*'
267 --filter='!@e2e-tests/cli-fake-backend'
268 + --filter='!@e2e-tests/cli-operations'
269
270 # Cloudflare Workerd E2E tests
271 test-e2e-cloudflare:
pnpm-lock.yaml
+18
@@ -276,6 +276,24 @@ importers:
276 specifier: 'catalog:'
277 version: 1.4.4
278
279 + ts/e2e-tests/cli/operations:
280 + devDependencies:
281 + '@e2e-tests/cli-fake-backend':
282 + specifier: workspace:*
283 + version: link:../fake-backend
284 + '@e2e-tests/utils':
285 + specifier: workspace:*
286 + version: link:../../_utils
287 + '@types/bun':
288 + specifier: 'catalog:'
289 + version: 1.4.4
290 + typescript:
291 + specifier: 'catalog:'
292 + version: 7.0.2
293 + zod:
294 + specifier: 'catalog:'
295 + version: 4.6.7
296 +
297 ts/e2e-tests/cli/run:
298 devDependencies:
299 '@e2e-tests/utils':
ts/e2e-tests/cli/README.md
+9 -8
@@ -10,12 +10,13 @@ Tests use `runCmd` to execute shell commands in the container and assert on exit
10
11 ## Test Suites
12
13 -| Suite | Description | Env Vars |
14 -| ------------------------------- | ------------------------------------------------------------------------------------ | ----------------------- |
15 -| [upgrade](./upgrade/) | `composio upgrade` replaces a running Linux executable | None |
16 -| [fake-backend](./fake-backend/) | Test support: the fake Composio deployment CLI tests talk to, and its contract check | None |
17 -| [version](./version/) | `composio version` output and exit code | None |
18 -| [whoami](./whoami/) | `composio whoami` prints the API key | `COMPOSIO_USER_API_KEY` |
13 +| Suite | Description | Env Vars |
14 +| ------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------- |
15 +| [upgrade](./upgrade/) | `composio upgrade` replaces a running Linux executable | None |
16 +| [operations](./operations/) | Login, session, search, execute, link, run, upgrade and their `dev` counterparts under a fault catalogue | None |
17 +| [fake-backend](./fake-backend/) | Test support: the fake Composio deployment the operation suites talk to, and its contract check | None |
18 +| [version](./version/) | `composio version` output and exit code | None |
19 +| [whoami](./whoami/) | `composio whoami` prints the API key | `COMPOSIO_USER_API_KEY` |
20
21 ## Journeys
22
@@ -30,9 +31,9 @@ const result = await journey.exec('composio whoami');
31 await journey.stop();
32 ```
33
33 -A journey container mounts nothing from the host and passes through no API key. Tests can use it through `startCliRig` from `@e2e-tests/cli-fake-backend/rig`, which also starts the fake backend and points every host the CLI knows at it. Other outbound traffic is not blocked.
34 +A journey container mounts nothing from the host and passes through no API key. The `operations` suites use it through `startCliRig` from `@e2e-tests/cli-fake-backend/rig`, which also starts the fake backend and points every host the CLI knows at it. Other outbound traffic is not blocked.
35
35 -`fake-backend` needs no secret. CI runs it in their own step with the API keys blanked, on every pull request, fork and Dependabot ones included; the suites that talk to staging run in a second step that is skipped when the keys are missing.
36 +The `operations` suites and `fake-backend` need no secret. CI runs them in their own step with the API keys blanked, on every pull request, fork and Dependabot ones included; the suites that talk to staging run in a second step that is skipped when the keys are missing.
37
38 ## Isolation Tool
39
ts/e2e-tests/cli/fake-backend/README.md
+1 -1
@@ -1,6 +1,6 @@
1 # Fake Composio backend
2
3 -Test support for CLI end-to-end suites. One process answers for every host the `composio` binary talks to, so a test never reaches a real Composio, GitHub, or PostHog endpoint.
3 +Test support for the CLI suites in `../operations`. One process answers for every host the `composio` binary talks to, so a test never reaches a real Composio, GitHub, or PostHog endpoint.
4
5 | Host the CLI calls | Pointed here by | Listener |
6 | ------------------ | -------------------------------------------------------- | -------- |
ts/e2e-tests/cli/operations/README.md new
+153
@@ -0,0 +1,153 @@
1 +# CLI operation suites and fault catalogue
2 +
3 +Deterministic end-to-end tests of the operations an agent drives the `composio` binary through: login and session, `whoami`/`logout`, `search`, `execute`, `link`, `connections`, `run`, `upgrade`, and their `composio dev` counterparts. No model is involved. The binary runs in a journey container (`startJourneyContainer` from `@e2e-tests/utils`) and talks only to the fake backend in [`../fake-backend`](../fake-backend/README.md).
4 +
5 +Every test asserts three things where they apply: what the command returned (exit code, stdout, stderr), what the fake backend received (route, key class, project, user), and what was left on disk.
6 +
7 +```bash
8 +pnpm --filter @e2e-tests/cli-operations test:e2e:cli # all files
9 +cd ts/e2e-tests/cli/operations && bun test identity.test.ts # one file
10 +```
11 +
12 +A failing run leaves `DEBUG.<name>.log` files here with each command and its output.
13 +
14 +## Reading the catalogue
15 +
16 +The catalogue starts from one question: what can go wrong for a CLI that an agent runs unattended, where nobody reads stderr unless the exit code says so, and a wrong success is worse than a failure? Each row names a fault, where it is injected, and what must hold.
17 +
18 +- **exit**: the CLI's exit code. **stdout**: `data` means pure machine-readable output and nothing else; `empty` means zero bytes.
19 +- **Status**: `ok` is implemented and passing. `expected failure` is implemented with `it.failing` because the CLI does not do what the row says today; the test turns red when the CLI is fixed, and the marker is then removed. The `it.failing` test asserts only the defect; where any other failure (a crash, a timeout) could also make it fail, a passing `today:` test pins what the CLI does instead, so the expected failure cannot hide a different one. `known gap` is behaviour that hinders an agent but that nothing in the repo declares wrong; a passing test pins what the CLI does today, so a change is noticed, and the row says what would be better.
20 +
21 +### Identity: which project, which user (`identity.test.ts`)
22 +
23 +| Fault or question | Injected at | Must hold | Status |
24 +| ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
25 +| Root `execute` run from an initialized developer directory | cwd | One Dashboard execution, consumer project, consumer user, user key; no request names the developer project | ok |
26 +| Dev `playground-execute` | cwd | One backend session execution, developer project, global test user; nothing goes to the Dashboard or the consumer project | ok |
27 +| Dev `--user-id` | flag | The session acts as that user | ok |
28 +| Dev `--project-name` | flag | The session is on the named developer project | ok |
29 +| Project API key minted by `dev init` | journal | Never sent by any command; every request carries the user key | ok |
30 +| Account linked on one tree, used on the other | world | Not connected on the other tree, both directions | ok |
31 +| Root `execute` without a connection | world | exit 1; the error names `composio link <toolkit>` | ok |
32 +| Dev `playground-execute` without a connection | world | exit 1; the error names `composio dev connected-accounts link <toolkit>`, not the root-tree `composio link` | ok |
33 +| `dev auth-configs list` in a directory set up for another project | `.composio/project.json` | The request is scoped to that project | expected failure: the request carries no org or project scope |
34 +| Command from the other tree (`dev execute`, `dev link`, root `playground-execute`, root `auth-configs`, root `--user-id`) | argv | exit 1; stdout empty; stderr names the unknown subcommand or flag and lists the real commands; no request | ok |
35 +| `composio dev link` | argv | stderr names `composio dev connected-accounts link` | known gap: the `dev` help lists `connected-accounts`, but no equivalent command is named |
36 +| `composio dev link --help` | argv | exit 1 | known gap: prints the `dev` help and exits 0 |
37 +| `composio link --help`, `composio dev connected-accounts link --help` | argv | exit 0; every example names the page's own command; no request | ok |
38 +| Missing acting user, root tree | resolve returns an empty consumer user id | exit 1; `Missing consumer user id. Run composio login and try again.`; no execution | ok |
39 +| Missing acting user, dev tree, no project for the directory | cwd | exit 1; `No developer project configured for this directory. Run composio dev init.`; no request | ok |
40 +| Missing acting user, dev tree, `--project-name`, no test user | stored login | exit 1; `Missing user id. Provide --user-id, or run composio login …`; no execution | ok |
41 +
42 +### Login and session (`login-session.test.ts`)
43 +
44 +"Session" here is the CLI login session (created, polled, linked) and the session artifact directory.
45 +
46 +| Fault or question | Injected at | Must hold | Status |
47 +| ------------------------------------ | ----------- | ----------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
48 +| First run, `whoami` | empty home | exit 1; stdout empty; names `composio login`; no request | ok |
49 +| Headless `login` | no terminal | exit 0; one session created; URL with the session key and `composio login --poll` on stdout; pending session cached; nothing stored | ok |
50 +| `login --poll`, session never linked | world | Keeps polling (at least two reads in 8 s); nothing stored | ok |
51 +| `login --poll`, session linked | world | exit 0; JSON summary; key, backend URLs, org and global test user stored; pending file removed | ok |
52 +| `whoami` after login | | exit 0; JSON on stdout; stderr empty; only user-key requests | ok |
53 +| `login` when already logged in | | exit 0; no request; stored login unchanged | ok |
54 +| `logout` | | Key removed; `whoami` then exits 1 without a request | ok |
55 +| `login --key`, session not linked | world | exit 1; `Session not yet linked`; nothing stored | ok |
56 +| `login --key`, linked | world | exit 0; login stored | ok |
57 +| `login --key`, unknown session | 404 | exit 1; names `composio login --no-wait`; nothing stored | ok |
58 +| `login --user-api-key`, rejected key | 401 | exit 1; stdout empty; key not written anywhere | ok |
59 +| Same, as an expected failure | 401 | No defect report | expected failure: printed as a defect report with an empty title |
60 +| `artifacts cwd` | | exit 0; one absolute path; no login or request needed; stable per directory; follows `COMPOSIO_SESSION_DIR` | ok |
61 +
62 +### Network (`network-faults.test.ts`)
63 +
64 +| Fault | Injected at | Must hold | Status |
65 +| ------------------------------------------------------------------------------ | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
66 +| Connection refused | `COMPOSIO_BASE_URL` to a closed port | exit 1; stdout empty; `ECONNREFUSED` on stderr | ok |
67 +| HTTP 500, 429, HTML 502 page, reset before response, on every attempt | `search` | exit 1; stdout empty; cause on stderr; three attempts | ok |
68 +| Reset in the middle of the body | `search` | exit 1; stdout empty; one attempt | ok |
69 +| Body that is not JSON | `search` | exit 1; stdout empty; says the response shape is unexpected and names `composio upgrade` | ok |
70 +| One 500, one 429 with Retry-After, one reset, then healthy | `search` | exit 0; stderr empty; normal result; two attempts | ok |
71 +| Slow response (4 s) | `search` | exit 0; one attempt | ok |
72 +| Valid JSON of another shape; empty 200 body | `search` | exit 1 with an unexpected-response message | expected failure: crashes with a `TypeError` naming a minified variable |
73 +| HTTP 500, 429, HTML page, non-JSON body, reset before response, reset mid-body | tool execution, Dashboard transport and backend transport | exit 1; **exactly one execution request**; stdout is JSON with `successful: false` and an error; no `data` | ok |
74 +
75 +Observed and left as is: on the backend transport an HTML error page is copied whole into the `error` field of the result.
76 +
77 +### Auth (`auth-faults.test.ts`)
78 +
79 +| Fault | Injected at | Must hold | Status |
80 +| -------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
81 +| No key | empty home, seven root and dev commands | exit 1; stdout empty; names `composio login`; no request | ok |
82 +| Key only in `COMPOSIO_USER_API_KEY` | env | exit 1; names `composio login` or `composio orgs switch`; no request | ok |
83 +| Stored key rejected (401 everywhere) | all backend routes | exit 1; rejection visible; nothing but 401s; stored login untouched | ok |
84 +| Same, recovery command | | Names `composio login` | expected failure: `search` prints only `Invalid API key` |
85 +| Same, `whoami` | | exit 1; stdout empty | expected failure: exit 0 and a JSON object of nulls |
86 +| Key sent to a backend that did not issue it | `COMPOSIO_BASE_URL` to a second fake | Warning on stderr naming both backends and `composio logout && composio login`; stdout free of it; the other backend rejects the key; nothing reaches the issuer; stored login untouched | ok |
87 +| Project key (`ak_`) where a user key belongs | `COMPOSIO_USER_API_KEY` | Arrives classified as a project key and is rejected | ok |
88 +
89 +### Filesystem (`filesystem-faults.test.ts`)
90 +
91 +| Fault | Injected at | Must hold | Status |
92 +| ------------------------------------------------------------------------- | ------------------------------ | ----------------------------------------------------------------------------- | ------------------------------------------- |
93 +| Stored login truncated, wrong JSON type, empty, missing | `user_data.json` | exit 1; names `composio login`; no request; a new login repairs it | ok |
94 +| Corrupt `update-check.json`, `config.json`, `tool-permissions-cache.json` | `~/.composio` | Command still succeeds | ok |
95 +| Corrupt cached tool schema | `tool_definitions/<SLUG>.json` | Fetched again | expected failure: fatal `is not valid JSON` |
96 +| Same | | At least names the file; deleting it recovers | ok |
97 +| Read-only home (ordinary user) | `chmod -R a-w ~/.composio` | `whoami`, `search`, `execute` succeed with data on stdout; the tool runs once | ok |
98 +| Read-only home, `logout` | | exit 1; names the file; login untouched | ok |
99 +| Eight concurrent `search` on a cold cache | background jobs | All exit 0; no stderr; cache file valid JSON; login intact | ok |
100 +| Four concurrent `execute` | | All exit 0; four executions, one each | ok |
101 +| Four concurrent `login` | | All exit 0; one complete stored login; no temporary files left | ok |
102 +
103 +### Process and streams (`process-streams.test.ts`)
104 +
105 +| Fault | Injected at | Must hold | Status |
106 +| -------------------------------------------------------- | ------------------------------ | -------------------------------------------------------------------------------- | ----------------------------------------------------- |
107 +| SIGINT / SIGTERM while `link` waits | `kill` once the URL is printed | exit 130 / 143; stdout empty; the link was requested once | ok |
108 +| SIGINT while `run` executes a script | `kill` once the script started | exit 130; stdout empty, also after the script would have finished | ok |
109 +| `link` off a terminal, nobody authorizes | no terminal | The authorization URL is printed before the wait | ok: on stderr, with a hint that names `--no-wait` |
110 +| `link --no-wait` | | exit 0; JSON with the URL; no status polling | ok |
111 +| Reader closes stdout before the CLI writes (`… \| true`) | pipe | CLI exits 0; no stderr | expected failure: exits 1 with an `EPIPE` stack trace |
112 +| `-d -` with JSON on stdin | pipe | Arguments read from stdin | ok |
113 +| `-d -` with nothing on stdin | `/dev/null` | exit 1 at once; stdout empty; no execution | ok |
114 +| No terminal on any stream | | stdout is the JSON; stderr empty | ok |
115 +| stdout piped, stderr a terminal | `script` | JSON in the pipe; human rendering on the terminal, not a second copy of the data | ok |
116 +| stdout a terminal, stderr redirected | `script` | No data on the terminal | ok |
117 +| Update notice pending | seeded `update-check.json` | Never on stdout; shown on a terminal stderr | ok |
118 +| Plugin hint under an agent host | `CLAUDECODE=1` | On stderr; never on stdout | ok |
119 +| `search --human` off a terminal | no terminal | Results are printed | expected failure: prints nothing, exits 0 |
120 +
121 +### Upgrade (`upgrade.test.ts`, and `../upgrade`)
122 +
123 +Every row: a failed upgrade leaves the executable's inode, size and mtime unchanged, and `composio version` still runs.
124 +
125 +| Fault | Injected at | Must hold | Status |
126 +| -------------------------------------------- | ---------------------------------- | ---------------------------------------------------------------------------------- | ----------------- |
127 +| No archive for this platform | release assets | exit 1; says so; no download | ok |
128 +| Checksum mismatch | `checksums.txt` | exit 1; expected and actual digests; no extraction | ok |
129 +| Corrupt archive with matching checksum | archive bytes | exit 1; `Failed to extract zip archive` | ok |
130 +| Download cut off mid-body | download route | exit 1; one download attempt | ok |
131 +| Pinned version never released | release-by-tag 404 | exit 1; names the tag; no download | ok |
132 +| Pinned version | two releases | Downloads the pinned release's archive, not the latest | ok |
133 +| Already on the latest release | `release-tag.txt` | exit 0; no download; executable untouched | ok |
134 +| No write permission on the install directory | read-only directory, ordinary user | exit 1; names the file; executable untouched and runnable; no temporary files left | ok (`../upgrade`) |
135 +
136 +Install faults live in [`../install`](../install/README.md), which runs in its own workflow against a packaged release: checksum mismatch, corrupt archive, missing platform archive, and a download cut off partway.
137 +
138 +### Tool loop (`tool-loop.test.ts`)
139 +
140 +Not faults: the search, execute, link, execute loop and the commands around it (`connections list`, `--get-schema`, `--dry-run`, input validation, unknown slug, `run` with `execute()` and `search()`), each checked against the journal.
141 +
142 +## Not modelled
143 +
144 +Listed so nobody reads their absence as coverage.
145 +
146 +- **Network**: packet loss, duplicated or reordered responses, partitions that heal mid-request, DNS failure, TLS errors other than an untrusted certificate, proxies (`HTTP_PROXY`), IPv6, redirects, timeouts the CLI gives up on (the Dashboard deadline is 15 minutes), responses larger than memory.
147 +- **Backend state**: an account that expires between the connection check and the execution, a tool that ran although its response was lost (the fake cannot act without answering), rate limits that depend on history, pagination past one page, several accounts per toolkit and `--account` selection, enhanced-controls permission prompts.
148 +- **Auth**: key rotation mid-command, OS keyring storage (the container has no Secret Service, so the key is stored in `user_data.json`), agent keys beyond what `../agent-signin` covers, org switching.
149 +- **Filesystem**: full disk, a home on a network or case-insensitive filesystem, symlinked `~/.composio`, files written by a newer or older CLI version, a stale cache that is valid JSON but out of date, two CLIs of different versions sharing a home.
150 +- **Process**: SIGHUP, SIGKILL, a second signal during shutdown, being killed between writing a temporary file and renaming it, stdin as a terminal with a person typing, terminal resize, non-UTF-8 locales.
151 +- **Upgrade and install**: a successful upgrade through the download path (the archive here is never a real build), upgrading while another invocation runs, a partially replaced companion module, macOS and Windows.
152 +- **Platforms**: everything here runs the Linux binary of the host's architecture as root in Debian; macOS, x64-on-arm, musl, and non-root homes (apart from the read-only rows) are not exercised.
153 +- **Commands**: `proxy`, `listen`, triggers, logs, `generate`, `setup`, `orgs`, and `dev` commands beyond `init`, `auth-configs list`, `playground-execute`, and `connected-accounts link`/`list`.
ts/e2e-tests/cli/operations/auth-faults.test.ts new
+224
@@ -0,0 +1,224 @@
1 +/**
2 + * Auth faults: no key, a key the backend rejects, a key sent to a backend that
3 + * did not issue it, and a key of the wrong class.
4 + */
5 +
6 +import { afterAll, afterEach, beforeAll, describe, expect, it } from 'bun:test';
7 +import {
8 + DEVELOPER_PROJECT,
9 + FOREIGN_USER_API_KEY,
10 + startFakeComposio,
11 + USER_API_KEY,
12 + type FakeComposio,
13 +} from '@e2e-tests/cli-fake-backend';
14 +import { startCliRig, type CliRig } from '@e2e-tests/cli-fake-backend/rig';
15 +import {
16 + clean,
17 + commandRequests,
18 + EXECUTE_SLACK,
19 + PLAYGROUND_EXECUTE_SLACK,
20 + readStoredLogin,
21 + RIG_TIMEOUT_MS,
22 + SEARCH_SLACK,
23 + TEST_TIMEOUT_MS,
24 + writeDebugLog,
25 +} from './support';
26 +
27 +const NOT_LOGGED_IN = 'You are not logged in yet. Please run `composio login`.';
28 +
29 +describe('auth faults: no key', () => {
30 + let rig: CliRig;
31 +
32 + beforeAll(async () => {
33 + rig = await startCliRig({ label: 'auth-none' });
34 + }, RIG_TIMEOUT_MS);
35 +
36 + afterAll(async () => {
37 + await writeDebugLog(rig, import.meta.dir, 'auth-none');
38 + await rig.stop();
39 + });
40 +
41 + for (const command of [
42 + SEARCH_SLACK,
43 + EXECUTE_SLACK,
44 + 'composio link slack',
45 + 'composio connections list',
46 + 'composio dev auth-configs list',
47 + PLAYGROUND_EXECUTE_SLACK,
48 + 'composio dev connected-accounts link slack',
49 + ]) {
50 + it(
51 + `\`${command.split(' -d ')[0]}\` names \`composio login\` and sends nothing`,
52 + async () => {
53 + const result = await rig.run(command);
54 + expect(result.exitCode).toBe(1);
55 + expect(result.stdout).toBe('');
56 + expect(clean(result.stderr)).toBe(NOT_LOGGED_IN);
57 + expect(commandRequests(result)).toEqual([]);
58 + },
59 + TEST_TIMEOUT_MS
60 + );
61 + }
62 +
63 + it(
64 + 'a key from COMPOSIO_USER_API_KEY alone has no org: the command says how to get one',
65 + async () => {
66 + const result = await rig.run(`COMPOSIO_USER_API_KEY=${USER_API_KEY} ${SEARCH_SLACK}`);
67 + expect(result.exitCode).toBe(1);
68 + expect(result.stdout).toBe('');
69 + expect(clean(result.stderr)).toBe(
70 + 'No current org configured. Run `composio login` or `composio orgs switch`.'
71 + );
72 + expect(commandRequests(result)).toEqual([]);
73 + },
74 + TEST_TIMEOUT_MS
75 + );
76 +});
77 +
78 +describe('auth faults: a stored key the backend stops accepting', () => {
79 + let rig: CliRig;
80 +
81 + beforeAll(async () => {
82 + rig = await startCliRig({ label: 'auth-rejected' });
83 + await rig.login();
84 + }, RIG_TIMEOUT_MS);
85 +
86 + afterEach(() => rig.fake.faults.clear());
87 +
88 + afterAll(async () => {
89 + await writeDebugLog(rig, import.meta.dir, 'auth-rejected');
90 + await rig.stop();
91 + });
92 +
93 + const reject = () =>
94 + rig.fake.faults.inject({
95 + route: /^(GET|POST) \/api\//,
96 + fault: {
97 + kind: 'status',
98 + status: 401,
99 + slug: 'Auth_InvalidApiKey',
100 + message: 'Invalid API key',
101 + },
102 + });
103 +
104 + for (const command of [SEARCH_SLACK, EXECUTE_SLACK, 'composio connections list']) {
105 + it(
106 + `\`${command.split(' -d ')[0]}\`: exit 1, the rejection on stderr, the login left in place`,
107 + async () => {
108 + reject();
109 + const result = await rig.run(command);
110 + expect(result.exitCode).toBe(1);
111 + expect(`${clean(result.stdout)}\n${clean(result.stderr)}`).toContain('Invalid API key');
112 + expect(commandRequests(result).length).toBeGreaterThan(0);
113 + expect(commandRequests(result).every(entry => entry.status === 401)).toBe(true);
114 + expect((await readStoredLogin(rig)).api_key).toBe(USER_API_KEY);
115 + },
116 + TEST_TIMEOUT_MS
117 + );
118 + }
119 +
120 + // CLI gap: `search` prints only the backend's "Invalid API key", with no command to recover.
121 + it.failing(
122 + 'a rejected stored key names `composio login` as the way out',
123 + async () => {
124 + reject();
125 + const result = await rig.run(SEARCH_SLACK);
126 + expect(clean(result.stderr)).toContain('composio login');
127 + },
128 + TEST_TIMEOUT_MS
129 + );
130 +
131 + // CLI defect: `whoami` answers a 401 with exit 0 and a JSON object of nulls, so "check the login with whoami" passes on a dead key.
132 + it.failing(
133 + '`whoami` fails when the backend rejects the stored key',
134 + async () => {
135 + reject();
136 + const result = await rig.run('composio whoami');
137 + expect(result.exitCode).toBe(1);
138 + expect(result.stdout).toBe('');
139 + },
140 + TEST_TIMEOUT_MS
141 + );
142 +});
143 +
144 +describe('auth faults: a key sent to another backend', () => {
145 + let rig: CliRig;
146 + let elsewhere: FakeComposio;
147 +
148 + beforeAll(async () => {
149 + rig = await startCliRig({ label: 'auth-mismatch' });
150 + // A deployment that issued its own key and has never seen this rig's.
151 + elsewhere = await startFakeComposio({ userApiKey: FOREIGN_USER_API_KEY });
152 + await rig.login();
153 + }, RIG_TIMEOUT_MS);
154 +
155 + afterAll(async () => {
156 + await writeDebugLog(rig, import.meta.dir, 'auth-mismatch');
157 + await elsewhere.close();
158 + await rig.stop();
159 + });
160 +
161 + it(
162 + 'COMPOSIO_BASE_URL pointing away from the issuer warns on stderr before sending the key',
163 + async () => {
164 + const result = await rig.run(`COMPOSIO_BASE_URL=${elsewhere.docker.http} composio whoami`);
165 + const stderr = clean(result.stderr);
166 + expect(stderr).toContain(
167 + `COMPOSIO_BASE_URL sends your stored API key to ${elsewhere.docker.http}, but it was issued by ${rig.fake.docker.http}.`
168 + );
169 + expect(stderr).toContain('composio logout && composio login');
170 + expect(result.stdout).not.toContain('COMPOSIO_BASE_URL');
171 + // The stored key did go to the other backend, which rejected it; nothing went to the issuer.
172 + const received = elsewhere.journal.entries().filter(entry => entry.role === 'backend');
173 + expect(received.length).toBeGreaterThan(0);
174 + expect(received.every(entry => entry.keyClass === 'user' && !entry.keyKnown)).toBe(true);
175 + expect(received.every(entry => entry.status === 401)).toBe(true);
176 + expect(commandRequests(result)).toEqual([]);
177 + expect((await readStoredLogin(rig)).api_key).toBe(USER_API_KEY);
178 + },
179 + TEST_TIMEOUT_MS
180 + );
181 +
182 + it(
183 + 'a command that needs the backend fails there with the rejection on stderr',
184 + async () => {
185 + const result = await rig.run(`COMPOSIO_BASE_URL=${elsewhere.docker.http} ${SEARCH_SLACK}`);
186 + expect(result.exitCode).toBe(1);
187 + expect(result.stdout).toBe('');
188 + const stderr = clean(result.stderr);
189 + expect(stderr).toContain('COMPOSIO_BASE_URL sends your stored API key');
190 + expect(stderr).toContain('Invalid API key');
191 + expect(commandRequests(result)).toEqual([]);
192 + },
193 + TEST_TIMEOUT_MS
194 + );
195 +});
196 +
197 +describe('auth faults: a key of the wrong class', () => {
198 + let rig: CliRig;
199 +
200 + beforeAll(async () => {
201 + rig = await startCliRig({ label: 'auth-class' });
202 + await rig.login();
203 + }, RIG_TIMEOUT_MS);
204 +
205 + afterAll(async () => {
206 + await writeDebugLog(rig, import.meta.dir, 'auth-class');
207 + await rig.stop();
208 + });
209 +
210 + it(
211 + 'a project API key in the user-key slot is sent as a user key and rejected',
212 + async () => {
213 + const result = await rig.run(
214 + `COMPOSIO_USER_API_KEY=${DEVELOPER_PROJECT.apiKey} composio whoami`
215 + );
216 + const sent = commandRequests(result);
217 + expect(sent.length).toBeGreaterThan(0);
218 + // The journal classifies by prefix: an `ak_` key arrived where a `uk_` key belongs.
219 + expect(sent.every(entry => entry.keyClass === 'project')).toBe(true);
220 + expect(sent.every(entry => entry.status === 401)).toBe(true);
221 + },
222 + TEST_TIMEOUT_MS
223 + );
224 +});
ts/e2e-tests/cli/operations/filesystem-faults.test.ts new
+267
@@ -0,0 +1,267 @@
1 +/**
2 + * Filesystem faults: the state under `~/.composio` missing, corrupt,
3 + * read-only, or written by several invocations at once.
4 + */
5 +
6 +import { afterAll, beforeAll, describe, expect, it } from 'bun:test';
7 +import {
8 + CONSUMER_PROJECT,
9 + CONSUMER_USER_ID,
10 + MEMBER,
11 + USER_API_KEY,
12 +} from '@e2e-tests/cli-fake-backend';
13 +import { z } from 'zod';
14 +import { startCliRig, type CliRig } from '@e2e-tests/cli-fake-backend/rig';
15 +import {
16 + clean,
17 + commandRequests,
18 + EXECUTE_SLACK,
19 + ExecutionResult,
20 + readStoredLogin,
21 + RIG_TIMEOUT_MS,
22 + SEARCH_SLACK,
23 + SearchResult,
24 + stdoutJson,
25 + TEST_TIMEOUT_MS,
26 + writeDebugLog,
27 +} from './support';
28 +
29 +const Whoami = z.looseObject({ email: z.string() });
30 +
31 +const NOT_LOGGED_IN = 'You are not logged in yet. Please run `composio login`.';
32 +const STORED_LOGIN = '~/.composio/user_data.json';
33 +
34 +describe('filesystem faults: the stored login', () => {
35 + let rig: CliRig;
36 +
37 + beforeAll(async () => {
38 + rig = await startCliRig({ label: 'fs-login' });
39 + }, RIG_TIMEOUT_MS);
40 +
41 + afterAll(async () => {
42 + await writeDebugLog(rig, import.meta.dir, 'fs-login');
43 + await rig.stop();
44 + });
45 +
46 + for (const { name, damage } of [
47 + { name: 'truncated JSON', damage: `printf '{"api_key":"uk_' > ${STORED_LOGIN}` },
48 + { name: 'JSON of the wrong type', damage: `echo '[]' > ${STORED_LOGIN}` },
49 + { name: 'an empty file', damage: `: > ${STORED_LOGIN}` },
50 + { name: 'a missing file', damage: `rm -f ${STORED_LOGIN}` },
51 + ]) {
52 + it(
53 + `${name}: commands report no login, and a new login repairs it`,
54 + async () => {
55 + await rig.login();
56 + await rig.run(damage);
57 +
58 + const result = await rig.run(SEARCH_SLACK);
59 + expect(result.exitCode).toBe(1);
60 + expect(result.stdout).toBe('');
61 + expect(clean(result.stderr)).toBe(NOT_LOGGED_IN);
62 + expect(commandRequests(result)).toEqual([]);
63 +
64 + await rig.login();
65 + expect((await readStoredLogin(rig)).api_key).toBe(USER_API_KEY);
66 + expect((await rig.run(SEARCH_SLACK)).exitCode).toBe(0);
67 + },
68 + TEST_TIMEOUT_MS
69 + );
70 + }
71 +});
72 +
73 +describe('filesystem faults: cache files', () => {
74 + let rig: CliRig;
75 +
76 + beforeAll(async () => {
77 + rig = await startCliRig({ label: 'fs-cache' });
78 + await rig.login();
79 + }, RIG_TIMEOUT_MS);
80 +
81 + afterAll(async () => {
82 + await writeDebugLog(rig, import.meta.dir, 'fs-cache');
83 + await rig.stop();
84 + });
85 +
86 + it(
87 + 'corrupt update-check, config, and permission caches do not fail a command',
88 + async () => {
89 + await rig.run(SEARCH_SLACK);
90 + await rig.run(
91 + `for f in update-check.json config.json tool-permissions-cache.json; do echo '{corrupt' > ~/.composio/$f; done`
92 + );
93 + const result = await rig.run(SEARCH_SLACK);
94 + expect(result.exitCode).toBe(0);
95 + expect(stdoutJson(result, SearchResult).results).toHaveLength(1);
96 + // The tests below corrupt one file each; leave nothing else broken for them.
97 + await rig.run(
98 + 'cd ~/.composio && rm -f update-check.json config.json tool-permissions-cache.json'
99 + );
100 + },
101 + TEST_TIMEOUT_MS
102 + );
103 +
104 + // CLI defect: a corrupt cached tool schema is fatal ("is not valid JSON") instead of being fetched again.
105 + it.failing(
106 + 'a corrupt cached tool schema is fetched again',
107 + async () => {
108 + await rig.run(SEARCH_SLACK);
109 + await rig.run(`echo '{corrupt' > ~/.composio/tool_definitions/SLACK_SEND_MESSAGE.json`);
110 + const result = await rig.run(SEARCH_SLACK);
111 + expect(result.exitCode).toBe(0);
112 + },
113 + TEST_TIMEOUT_MS
114 + );
115 +
116 + it(
117 + 'a corrupt cached tool schema is at least named, so it can be deleted',
118 + async () => {
119 + await rig.run(`mkdir -p ~/.composio/tool_definitions`);
120 + await rig.run(`echo '{corrupt' > ~/.composio/tool_definitions/SLACK_SEND_MESSAGE.json`);
121 + const result = await rig.run(SEARCH_SLACK);
122 + expect(result.exitCode).toBe(1);
123 + expect(result.stdout).toBe('');
124 + expect(clean(result.stderr)).toContain('is not valid JSON');
125 + expect(clean(result.stderr)).toContain(
126 + `${rig.journey.home}/.composio/tool_definitions/SLACK_SEND_MESSAGE.json`
127 + );
128 + await rig.run(`rm -f ~/.composio/tool_definitions/SLACK_SEND_MESSAGE.json`);
129 + expect((await rig.run(SEARCH_SLACK)).exitCode).toBe(0);
130 + },
131 + TEST_TIMEOUT_MS
132 + );
133 +});
134 +
135 +describe('filesystem faults: a read-only home', () => {
136 + let rig: CliRig;
137 + // Root ignores file modes, so these commands run as an ordinary user.
138 + const asUser = { user: '1000:1000' };
139 +
140 + beforeAll(async () => {
141 + rig = await startCliRig({ label: 'fs-readonly' });
142 + await rig.login();
143 + rig.fake.world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
144 + await rig.run(SEARCH_SLACK);
145 + await rig.run('chown -R 1000:1000 ~ && chmod -R a-w ~/.composio');
146 + }, RIG_TIMEOUT_MS);
147 +
148 + afterAll(async () => {
149 + await writeDebugLog(rig, import.meta.dir, 'fs-readonly');
150 + await rig.stop();
151 + });
152 +
153 + it(
154 + 'reads keep working: `whoami` and `search` succeed with data on stdout',
155 + async () => {
156 + const whoami = await rig.run('composio whoami', asUser);
157 + expect(whoami.exitCode).toBe(0);
158 + expect(stdoutJson(whoami, Whoami).email).toBe(MEMBER.email);
159 + const search = await rig.run(SEARCH_SLACK, asUser);
160 + expect(search.exitCode).toBe(0);
161 + expect(stdoutJson(search, SearchResult).results).toHaveLength(1);
162 + },
163 + TEST_TIMEOUT_MS
164 + );
165 +
166 + it(
167 + '`execute` still runs the tool; the failed cache write stays off stdout',
168 + async () => {
169 + const result = await rig.run(EXECUTE_SLACK, asUser);
170 + expect(result.exitCode).toBe(0);
171 + expect(stdoutJson(result, ExecutionResult).successful).toBe(true);
172 + expect(result.stdout).not.toMatch(/EACCES|EROFS|permission denied/i);
173 + expect(result.requests.filter(entry => entry.route === 'dashboard.execute')).toHaveLength(1);
174 + },
175 + TEST_TIMEOUT_MS
176 + );
177 +
178 + it(
179 + '`logout` fails, names the file it could not write, and leaves the login intact',
180 + async () => {
181 + const result = await rig.run('composio logout', asUser);
182 + expect(result.exitCode).toBe(1);
183 + expect(result.stdout).toBe('');
184 + const stderr = clean(result.stderr);
185 + expect(stderr).toContain('PermissionDenied');
186 + expect(stderr).toContain(`${rig.journey.home}/.composio/user_data.json`);
187 + expect((await readStoredLogin(rig)).api_key).toBe(USER_API_KEY);
188 + },
189 + TEST_TIMEOUT_MS
190 + );
191 +});
192 +
193 +describe('filesystem faults: concurrent invocations', () => {
194 + let rig: CliRig;
195 +
196 + beforeAll(async () => {
197 + rig = await startCliRig({ label: 'fs-concurrent' });
198 + await rig.login();
199 + rig.fake.world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
200 + }, RIG_TIMEOUT_MS);
201 +
202 + afterAll(async () => {
203 + await writeDebugLog(rig, import.meta.dir, 'fs-concurrent');
204 + await rig.stop();
205 + });
206 +
207 + it(
208 + 'eight searches on a cold cache all succeed and leave valid state',
209 + async () => {
210 + const result = await rig.run(
211 + [
212 + 'rm -rf ~/.composio/tool_definitions',
213 + `for i in 1 2 3 4 5 6 7 8; do { ${SEARCH_SLACK} > /tmp/c$i.out 2> /tmp/c$i.err; echo $? > /tmp/c$i.rc; } & done`,
214 + 'wait',
215 + 'cat /tmp/c*.rc | tr -d "\\n"; echo',
216 + 'cat /tmp/c*.err',
217 + 'cat ~/.composio/tool_definitions/SLACK_SEND_MESSAGE.json > /dev/null',
218 + ].join('\n')
219 + );
220 + expect(result.exitCode).toBe(0);
221 + expect(result.stdout.trim()).toBe('00000000');
222 + expect(result.stderr).toBe('');
223 + const cached: unknown = JSON.parse(
224 + await rig.journey.readFile(
225 + `${rig.journey.home}/.composio/tool_definitions/SLACK_SEND_MESSAGE.json`
226 + )
227 + );
228 + expect(cached).toBeObject();
229 + expect((await readStoredLogin(rig)).api_key).toBe(USER_API_KEY);
230 + },
231 + TEST_TIMEOUT_MS
232 + );
233 +
234 + it(
235 + 'four executions at once each reach the Dashboard exactly once',
236 + async () => {
237 + const result = await rig.run(
238 + [
239 + `for i in 1 2 3 4; do { ${EXECUTE_SLACK} > /tmp/e$i.out 2> /tmp/e$i.err; echo $? > /tmp/e$i.rc; } & done`,
240 + 'wait',
241 + 'cat /tmp/e*.rc | tr -d "\\n"; echo',
242 + ].join('\n')
243 + );
244 + expect(result.stdout.trim()).toBe('0000');
245 + expect(result.requests.filter(entry => entry.route === 'dashboard.execute')).toHaveLength(4);
246 + },
247 + TEST_TIMEOUT_MS
248 + );
249 +
250 + it(
251 + 'a login racing other logins leaves one complete stored login',
252 + async () => {
253 + const result = await rig.run(
254 + [
255 + `for i in 1 2 3 4; do { composio login --user-api-key ${USER_API_KEY} --no-skill-install > /dev/null 2> /tmp/l$i.err; echo $? > /tmp/l$i.rc; } & done`,
256 + 'wait',
257 + 'cat /tmp/l*.rc | tr -d "\\n"; echo',
258 + ].join('\n')
259 + );
260 + expect(result.stdout.trim()).toBe('0000');
261 + expect((await readStoredLogin(rig)).api_key).toBe(USER_API_KEY);
262 + const leftovers = await rig.run('ls -a ~/.composio | grep -c "composio-tmp" || true');
263 + expect(leftovers.stdout.trim()).toBe('0');
264 + },
265 + TEST_TIMEOUT_MS
266 + );
267 +});
ts/e2e-tests/cli/operations/identity.test.ts new
+438
@@ -0,0 +1,438 @@
1 +/**
2 + * Identity: which project and which user a command acts on.
3 + *
4 + * The root tree acts on the consumer project as the consumer user; the dev
5 + * tree acts on a developer project as a test user. Each claim below is checked
6 + * against what the fake backend received, not against what the CLI printed.
7 + */
8 +
9 +import { afterAll, beforeAll, describe, expect, it } from 'bun:test';
10 +import {
11 + CONSUMER_PROJECT,
12 + CONSUMER_USER_ID,
13 + DEVELOPER_PROJECT,
14 + GLOBAL_TEST_USER_ID,
15 + OTHER_DEVELOPER_PROJECT,
16 + ORG,
17 +} from '@e2e-tests/cli-fake-backend';
18 +import { startCliRig, type CliRig } from '@e2e-tests/cli-fake-backend/rig';
19 +import {
20 + clean,
21 + commandRequests,
22 + DEV_DIRECTORY,
23 + EXECUTE_SLACK,
24 + ExecutionResult,
25 + initDeveloperDirectory,
26 + PLAYGROUND_EXECUTE_SLACK,
27 + RIG_TIMEOUT_MS,
28 + routesOf,
29 + SLACK_ARGUMENTS,
30 + stdoutJson,
31 + TEST_TIMEOUT_MS,
32 + writeDebugLog,
33 +} from './support';
34 +
35 +const inDevDirectory = (command: string) => `cd ${DEV_DIRECTORY} && ${command}`;
36 +
37 +/** A row of the help printed with the error: the name indented, then its description or type. */
38 +const helpRow = (name: string) =>
39 + new RegExp(`^\\s+${name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}\\s`, 'm');
40 +
41 +describe('identity: the two command trees', () => {
42 + let rig: CliRig;
43 +
44 + beforeAll(async () => {
45 + rig = await startCliRig({ label: 'identity' });
46 + await rig.login();
47 + await initDeveloperDirectory(rig);
48 + }, RIG_TIMEOUT_MS);
49 +
50 + afterAll(async () => {
51 + await writeDebugLog(rig, import.meta.dir, 'identity');
52 + await rig.stop();
53 + });
54 +
55 + describe('the same tool on each tree', () => {
56 + it(
57 + 'root `execute` acts on the consumer project as the consumer user',
58 + async () => {
59 + rig.fake.world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
60 + const result = await rig.run(inDevDirectory(EXECUTE_SLACK));
61 +
62 + expect(result.exitCode).toBe(0);
63 + expect(stdoutJson(result, ExecutionResult).successful).toBe(true);
64 +
65 + const execution = result.requests.filter(entry => entry.route === 'dashboard.execute');
66 + expect(execution).toHaveLength(1);
67 + expect(execution[0]).toMatchObject({
68 + role: 'dashboard',
69 + keyClass: 'user',
70 + orgId: ORG.id,
71 + project: CONSUMER_PROJECT.label,
72 + userId: CONSUMER_USER_ID,
73 + });
74 + // Run from an initialized developer directory on purpose: the
75 + // directory's project must not leak into a root-tree command.
76 + const projects = commandRequests(result).map(entry => entry.project);
77 + expect(projects).not.toContain(DEVELOPER_PROJECT.label);
78 + expect(routesOf(result)).not.toContain('toolRouter.session.execute');
79 + },
80 + TEST_TIMEOUT_MS
81 + );
82 +
83 + it(
84 + 'dev `playground-execute` acts on the developer project as the global test user',
85 + async () => {
86 + rig.fake.world.connect(DEVELOPER_PROJECT.label, GLOBAL_TEST_USER_ID, 'slack');
87 + const result = await rig.run(inDevDirectory(PLAYGROUND_EXECUTE_SLACK));
88 +
89 + expect(result.exitCode).toBe(0);
90 + expect(stdoutJson(result, ExecutionResult).successful).toBe(true);
91 +
92 + const execution = result.requests.filter(
93 + entry => entry.route === 'toolRouter.session.execute'
94 + );
95 + expect(execution).toHaveLength(1);
96 + expect(execution[0]).toMatchObject({
97 + role: 'backend',
98 + keyClass: 'user',
99 + project: DEVELOPER_PROJECT.label,
100 + userId: GLOBAL_TEST_USER_ID,
101 + });
102 + expect(routesOf(result)).not.toContain('dashboard.execute');
103 + expect(commandRequests(result).map(entry => entry.project)).not.toContain(
104 + CONSUMER_PROJECT.label
105 + );
106 + },
107 + TEST_TIMEOUT_MS
108 + );
109 +
110 + it(
111 + 'dev `--user-id` replaces the test user',
112 + async () => {
113 + rig.fake.world.connect(DEVELOPER_PROJECT.label, 'alice', 'slack');
114 + const result = await rig.run(inDevDirectory(`${PLAYGROUND_EXECUTE_SLACK} --user-id alice`));
115 +
116 + expect(result.exitCode).toBe(0);
117 + const execution = result.requests.find(
118 + entry => entry.route === 'toolRouter.session.execute'
119 + );
120 + expect(execution).toMatchObject({ project: DEVELOPER_PROJECT.label, userId: 'alice' });
121 + },
122 + TEST_TIMEOUT_MS
123 + );
124 +
125 + it(
126 + 'dev `--project-name` targets the named developer project',
127 + async () => {
128 + rig.fake.world.connect(OTHER_DEVELOPER_PROJECT.label, GLOBAL_TEST_USER_ID, 'slack');
129 + const result = await rig.run(
130 + inDevDirectory(
131 + `${PLAYGROUND_EXECUTE_SLACK} --project-name ${OTHER_DEVELOPER_PROJECT.name}`
132 + )
133 + );
134 +
135 + expect(result.exitCode).toBe(0);
136 + const execution = result.requests.find(
137 + entry => entry.route === 'toolRouter.session.execute'
138 + );
139 + expect(execution).toMatchObject({
140 + project: OTHER_DEVELOPER_PROJECT.label,
141 + userId: GLOBAL_TEST_USER_ID,
142 + });
143 + },
144 + TEST_TIMEOUT_MS
145 + );
146 + });
147 +
148 + describe('connections do not cross trees', () => {
149 + it(
150 + 'an account linked on the root tree is not connected on the dev tree',
151 + async () => {
152 + const link = await rig.run(inDevDirectory('composio link github'));
153 + expect(link.exitCode).toBe(0);
154 + expect(
155 + link.requests.find(entry => entry.route === 'toolRouter.session.link')
156 + ).toMatchObject({ project: CONSUMER_PROJECT.label, userId: CONSUMER_USER_ID });
157 +
158 + const execution = await rig.run(
159 + inDevDirectory(
160 + `composio dev playground-execute GITHUB_GET_THE_AUTHENTICATED_USER -d '{}'`
161 + )
162 + );
163 + expect(execution.exitCode).toBe(1);
164 + expect(stdoutJson(execution, ExecutionResult)).toMatchObject({
165 + successful: false,
166 + slug: 'ToolRouterV2_NoActiveConnection',
167 + });
168 + },
169 + TEST_TIMEOUT_MS
170 + );
171 +
172 + it(
173 + 'an account linked on the dev tree is not connected on the root tree',
174 + async () => {
175 + const link = await rig.run(inDevDirectory('composio dev connected-accounts link gmail'));
176 + expect(link.exitCode).toBe(0);
177 + expect(
178 + link.requests.find(entry => entry.route === 'toolRouter.session.link')
179 + ).toMatchObject({ project: DEVELOPER_PROJECT.label, userId: GLOBAL_TEST_USER_ID });
180 +
181 + const execution = await rig.run(
182 + `composio execute GMAIL_SEND_EMAIL -d '{"recipient_email":"a@b.test"}'`
183 + );
184 + expect(execution.exitCode).toBe(1);
185 + expect(stdoutJson(execution, ExecutionResult)).toMatchObject({
186 + successful: false,
187 + slug: 'ToolRouterV2_NoActiveConnection',
188 + });
189 + },
190 + TEST_TIMEOUT_MS
191 + );
192 +
193 + it(
194 + 'root `execute` without a connection names the root command that fixes it',
195 + async () => {
196 + const result = await rig.run(
197 + `composio execute GMAIL_SEND_EMAIL -d '{"recipient_email":"a@b.test"}'`
198 + );
199 + expect(result.exitCode).toBe(1);
200 + expect(stdoutJson(result, ExecutionResult).error).toContain('composio link gmail');
201 + },
202 + TEST_TIMEOUT_MS
203 + );
204 +
205 + it(
206 + 'dev `playground-execute` without a connection names a dev-tree command that fixes it',
207 + async () => {
208 + const result = await rig.run(
209 + inDevDirectory(
210 + `composio dev playground-execute GITHUB_GET_THE_AUTHENTICATED_USER -d '{}'`
211 + )
212 + );
213 + const { error } = stdoutJson(result, ExecutionResult);
214 + expect(error).toContain('composio dev connected-accounts link github');
215 + expect(error).not.toContain('composio link github');
216 + },
217 + TEST_TIMEOUT_MS
218 + );
219 + });
220 +
221 + describe('the developer project of the directory', () => {
222 + const OTHER_DIRECTORY = '/work/other-app';
223 + const listInOtherDirectory = async () => {
224 + await rig.journey.writeFile(
225 + `${OTHER_DIRECTORY}/.composio/project.json`,
226 + JSON.stringify({
227 + org_id: ORG.id,
228 + project_id: OTHER_DEVELOPER_PROJECT.nanoId,
229 + project_name: OTHER_DEVELOPER_PROJECT.name,
230 + })
231 + );
232 + const result = await rig.run(`cd ${OTHER_DIRECTORY} && composio dev auth-configs list`);
233 + return { result, listing: result.requests.find(entry => entry.route === 'authConfigs.list') };
234 + };
235 +
236 + it(
237 + "today: `dev auth-configs list` in another project's directory sends no scope and gets the key's default project",
238 + async () => {
239 + const { result, listing } = await listInOtherDirectory();
240 + expect(result.exitCode).toBe(0);
241 + expect(listing).toMatchObject({
242 + project: DEVELOPER_PROJECT.label,
243 + projectFrom: 'key-default',
244 + });
245 + },
246 + TEST_TIMEOUT_MS
247 + );
248 +
249 + // CLI defect: commands on the shared backend client send the user key with no org or project scope, so the backend picks the project.
250 + it.failing(
251 + '`dev auth-configs list` is scoped to the project the directory is set up for',
252 + async () => {
253 + const { listing } = await listInOtherDirectory();
254 + expect(listing?.projectFrom).toBe('scope-headers');
255 + },
256 + TEST_TIMEOUT_MS
257 + );
258 + });
259 +
260 + describe('a command from the wrong tree', () => {
261 + const cases = [
262 + {
263 + command: `composio dev execute SLACK_SEND_MESSAGE -d ${SLACK_ARGUMENTS}`,
264 + message: 'Unknown subcommand "execute" for "composio dev"',
265 + listed: 'playground-execute',
266 + },
267 + {
268 + command: 'composio dev link slack',
269 + message: 'Unknown subcommand "link" for "composio dev"',
270 + listed: 'connected-accounts',
271 + },
272 + {
273 + command: `composio playground-execute SLACK_SEND_MESSAGE -d ${SLACK_ARGUMENTS}`,
274 + message: 'Unknown subcommand "playground-execute" for "composio"',
275 + listed: 'execute',
276 + },
277 + {
278 + command: 'composio auth-configs list',
279 + message: 'Unknown subcommand "auth-configs" for "composio"',
280 + listed: 'dev',
281 + },
282 + {
283 + command: `${EXECUTE_SLACK} --user-id alice`,
284 + message: 'Unrecognized flag: --user-id in command composio execute',
285 + listed: '--account',
286 + },
287 + ] as const;
288 +
289 + for (const { command, message, listed } of cases) {
290 + it(
291 + `\`${command.split(' -d ')[0]}\` fails before any request, naming what was wrong`,
292 + async () => {
293 + const result = await rig.run(command);
294 + expect(result.exitCode).toBe(1);
295 + expect(result.stdout).toBe('');
296 + const stderr = clean(result.stderr);
297 + expect(stderr).toContain(message);
298 + // The help printed with the error is what lets an agent find the real command.
299 + expect(stderr).toMatch(helpRow(listed));
300 + expect(commandRequests(result)).toEqual([]);
301 + },
302 + TEST_TIMEOUT_MS
303 + );
304 + }
305 +
306 + // The two tests below pin what the CLI does today, not what it should do.
307 + // Nothing in the repo says what a wrong-tree command should answer, so they
308 + // are listed as known gaps in the README instead of as expected failures.
309 + it(
310 + 'today: `composio dev link --help` prints the `dev` help and exits 0, as if the command existed',
311 + async () => {
312 + const result = await rig.run('composio dev link --help');
313 + expect(result.exitCode).toBe(0);
314 + expect(`${result.stdout}${clean(result.stderr)}`).toContain('connected-accounts');
315 + },
316 + TEST_TIMEOUT_MS
317 + );
318 +
319 + it(
320 + 'today: `composio dev link` lists `connected-accounts` but never names `connected-accounts link`',
321 + async () => {
322 + const result = await rig.run('composio dev link slack');
323 + expect(clean(result.stderr)).not.toContain('composio dev connected-accounts link');
324 + },
325 + TEST_TIMEOUT_MS
326 + );
327 + });
328 +
329 + describe('a help page teaches its own tree', () => {
330 + // One `link` definition serves both trees, so its examples must follow the page they are on.
331 + const examplesOf = (help: string) =>
332 + (help.split('EXAMPLES')[1] ?? '')
333 + .split('\n')
334 + .map(line => line.trim())
335 + .filter(line => line.startsWith('composio '));
336 +
337 + for (const page of ['composio link', 'composio dev connected-accounts link']) {
338 + it(
339 + `\`${page} --help\` shows examples of \`${page}\` only`,
340 + async () => {
341 + const result = await rig.run(`${page} --help`);
342 + expect(result.exitCode).toBe(0);
343 + const examples = examplesOf(clean(`${result.stdout}${result.stderr}`));
344 + expect(examples.length).toBeGreaterThan(0);
345 + expect(examples.filter(example => !example.startsWith(`${page} `))).toEqual([]);
346 + expect(commandRequests(result)).toEqual([]);
347 + },
348 + TEST_TIMEOUT_MS
349 + );
350 + }
351 + });
352 +
353 + // Last, so it covers every command above: links, auth configs, both executions.
354 + it(
355 + 'never sends the project API key `dev init` minted',
356 + async () => {
357 + expect(rig.fake.world.mintedProjectKeys).toHaveLength(1);
358 + const classes = new Set(
359 + rig.fake.journal
360 + .entries()
361 + .filter(entry => entry.role === 'backend' || entry.role === 'dashboard')
362 + .map(entry => entry.keyClass)
363 + );
364 + expect([...classes]).toEqual(['user']);
365 + },
366 + TEST_TIMEOUT_MS
367 + );
368 +});
369 +
370 +describe('identity: a missing acting user', () => {
371 + let rig: CliRig;
372 +
373 + beforeAll(async () => {
374 + rig = await startCliRig({ label: 'acting-user' });
375 + await rig.login();
376 + }, RIG_TIMEOUT_MS);
377 +
378 + afterAll(async () => {
379 + await writeDebugLog(rig, import.meta.dir, 'acting-user');
380 + await rig.stop();
381 + });
382 +
383 + it(
384 + 'root tree: an empty consumer user id fails with the consumer message',
385 + async () => {
386 + const disarm = rig.fake.faults.inject({
387 + route: '_internal.org.consumer.project.resolve',
388 + fault: {
389 + kind: 'schema-drift',
390 + transform: body => ({ ...(body as Record<string, unknown>), consumer_user_id: '' }),
391 + },
392 + });
393 + const result = await rig.run(EXECUTE_SLACK);
394 + disarm();
395 +
396 + expect(result.exitCode).toBe(1);
397 + expect(result.stdout).toBe('');
398 + expect(clean(result.stderr)).toBe(
399 + 'Missing consumer user id. Run `composio login` and try again.'
400 + );
401 + expect(routesOf(result)).not.toContain('dashboard.execute');
402 + },
403 + TEST_TIMEOUT_MS
404 + );
405 +
406 + it(
407 + 'dev tree: no developer project for the directory fails with the dev message',
408 + async () => {
409 + const result = await rig.run(PLAYGROUND_EXECUTE_SLACK);
410 + expect(result.exitCode).toBe(1);
411 + expect(result.stdout).toBe('');
412 + expect(clean(result.stderr)).toBe(
413 + 'No developer project configured for this directory. Run `composio dev init`.'
414 + );
415 + expect(commandRequests(result)).toEqual([]);
416 + },
417 + TEST_TIMEOUT_MS
418 + );
419 +
420 + it(
421 + 'dev tree: `--project-name` without any test user fails with the dev message',
422 + async () => {
423 + await rig.run(
424 + `sed -i 's/"test_user_id":"[^"]*"/"test_user_id":null/' ~/.composio/user_data.json`
425 + );
426 + const result = await rig.run(
427 + `${PLAYGROUND_EXECUTE_SLACK} --project-name ${DEVELOPER_PROJECT.name}`
428 + );
429 + expect(result.exitCode).toBe(1);
430 + expect(result.stdout).toBe('');
431 + expect(clean(result.stderr)).toBe(
432 + 'Missing user id. Provide --user-id, or run `composio login` to store a global test user id.'
433 + );
434 + expect(routesOf(result)).not.toContain('toolRouter.session.execute');
435 + },
436 + TEST_TIMEOUT_MS
437 + );
438 +});
ts/e2e-tests/cli/operations/login-session.test.ts new
+330
@@ -0,0 +1,330 @@
1 +/**
2 + * Login and session: the CLI login session (created, polled, linked), the
3 + * stored login it leaves on disk, `whoami`, `logout`, and the session
4 + * artifact directory.
5 + */
6 +
7 +import { afterAll, beforeAll, describe, expect, it } from 'bun:test';
8 +import {
9 + FOREIGN_USER_API_KEY,
10 + GLOBAL_TEST_USER_ID,
11 + MEMBER,
12 + ORG,
13 + USER_API_KEY,
14 +} from '@e2e-tests/cli-fake-backend';
15 +import { startCliRig, type CliRig } from '@e2e-tests/cli-fake-backend/rig';
16 +import { z } from 'zod';
17 +import {
18 + clean,
19 + commandRequests,
20 + readStoredLogin,
21 + RIG_TIMEOUT_MS,
22 + routesOf,
23 + stdoutJson,
24 + TEST_TIMEOUT_MS,
25 + writeDebugLog,
26 +} from './support';
27 +
28 +const NOT_LOGGED_IN = 'You are not logged in yet. Please run `composio login`.';
29 +
30 +const Whoami = z.looseObject({
31 + account_type: z.string(),
32 + email: z.string(),
33 + current_org_name: z.string(),
34 +});
35 +
36 +const PendingLogin = z.looseObject({ key: z.string(), loginUrl: z.string() });
37 +
38 +const PollLogin = z.looseObject({
39 + email: z.string(),
40 + current_org: z.object({ id: z.string(), name: z.string() }),
41 + organizations: z.array(z.looseObject({ id: z.string(), selected: z.boolean() })),
42 +});
43 +
44 +describe('login session: browser login completed by polling', () => {
45 + let rig: CliRig;
46 + const home = () => `${rig.journey.home}/.composio`;
47 +
48 + beforeAll(async () => {
49 + rig = await startCliRig({ label: 'login-poll' });
50 + }, RIG_TIMEOUT_MS);
51 +
52 + afterAll(async () => {
53 + await writeDebugLog(rig, import.meta.dir, 'login-poll');
54 + await rig.stop();
55 + });
56 +
57 + it(
58 + 'first run: `whoami` reports no login without calling the backend',
59 + async () => {
60 + const result = await rig.run('composio whoami');
61 + expect(result.exitCode).toBe(1);
62 + expect(result.stdout).toBe('');
63 + expect(clean(result.stderr)).toBe(NOT_LOGGED_IN);
64 + expect(commandRequests(result)).toEqual([]);
65 + },
66 + TEST_TIMEOUT_MS
67 + );
68 +
69 + it(
70 + 'headless `login` creates a pending session and prints how to finish it',
71 + async () => {
72 + const result = await rig.run('composio login');
73 + expect(result.exitCode).toBe(0);
74 + expect(routesOf(result)).toEqual(['_internal.cli.createSession']);
75 +
76 + const session = rig.fake.world.loginSessions.at(-1);
77 + expect(session?.status).toBe('pending');
78 + const stdout = clean(result.stdout);
79 + expect(stdout).toContain(`${rig.fake.docker.https}?cliKey=${session?.id}`);
80 + expect(stdout).toContain('composio login --poll');
81 +
82 + const pending = PendingLogin.parse(
83 + JSON.parse(await rig.journey.readFile(`${home()}/pending-login-session.json`))
84 + );
85 + expect(pending.key).toBe(session?.id ?? '');
86 + expect((await readStoredLogin(rig)).api_key).toBeNull();
87 + },
88 + TEST_TIMEOUT_MS
89 + );
90 +
91 + it(
92 + '`login --poll` keeps polling while nobody approved the session',
93 + async () => {
94 + const result = await rig.run('composio login --poll --no-skill-install', {
95 + timeoutSeconds: 8,
96 + });
97 + // 124: still waiting when the test's own time limit ended it.
98 + expect(result.exitCode).toBe(124);
99 + expect(result.stdout).toBe('');
100 + const polls = result.requests.filter(entry => entry.route === '_internal.cli.getSession');
101 + expect(polls.length).toBeGreaterThanOrEqual(2);
102 + expect((await readStoredLogin(rig)).api_key).toBeNull();
103 + },
104 + TEST_TIMEOUT_MS
105 + );
106 +
107 + it(
108 + '`login --poll` stores the login once the session is linked',
109 + async () => {
110 + rig.fake.world.linkLoginSession();
111 + const result = await rig.run('composio login --poll --no-skill-install');
112 + expect(result.exitCode).toBe(0);
113 +
114 + const summary = stdoutJson(result, PollLogin);
115 + expect(summary.email).toBe(MEMBER.email);
116 + expect(summary.current_org).toEqual({ id: ORG.id, name: ORG.name });
117 + expect(summary.organizations.filter(org => org.selected).map(org => org.id)).toEqual([
118 + ORG.id,
119 + ]);
120 +
121 + expect(await readStoredLogin(rig)).toMatchObject({
122 + api_key: USER_API_KEY,
123 + base_url: rig.fake.docker.http,
124 + web_url: rig.fake.docker.https,
125 + org_id: ORG.id,
126 + test_user_id: GLOBAL_TEST_USER_ID,
127 + });
128 + const leftovers = await rig.run(`ls ${home()}`);
129 + expect(leftovers.stdout).not.toContain('pending-login-session.json');
130 + },
131 + TEST_TIMEOUT_MS
132 + );
133 +
134 + it(
135 + '`whoami` prints the account as JSON on stdout and nothing on stderr',
136 + async () => {
137 + const result = await rig.run('composio whoami');
138 + expect(result.exitCode).toBe(0);
139 + expect(result.stderr).toBe('');
140 + expect(stdoutJson(result, Whoami)).toMatchObject({
141 + account_type: 'human',
142 + email: MEMBER.email,
143 + current_org_name: ORG.name,
144 + });
145 + expect(commandRequests(result).every(entry => entry.keyClass === 'user')).toBe(true);
146 + },
147 + TEST_TIMEOUT_MS
148 + );
149 +
150 + it(
151 + '`login` when already logged in changes nothing and creates no session',
152 + async () => {
153 + const before = await readStoredLogin(rig);
154 + const result = await rig.run('composio login');
155 + expect(result.exitCode).toBe(0);
156 + expect(commandRequests(result)).toEqual([]);
157 + expect(await readStoredLogin(rig)).toEqual(before);
158 + },
159 + TEST_TIMEOUT_MS
160 + );
161 +
162 + it(
163 + '`logout` forgets the key, and `whoami` then reports no login',
164 + async () => {
165 + const logout = await rig.run('composio logout');
166 + expect(logout.exitCode).toBe(0);
167 + expect(logout.stdout).toBe('');
168 + expect((await readStoredLogin(rig)).api_key).toBeNull();
169 +
170 + const whoami = await rig.run('composio whoami');
171 + expect(whoami.exitCode).toBe(1);
172 + expect(clean(whoami.stderr)).toBe(NOT_LOGGED_IN);
173 + expect(commandRequests(whoami)).toEqual([]);
174 + },
175 + TEST_TIMEOUT_MS
176 + );
177 +});
178 +
179 +describe('login session: completed with an explicit session key', () => {
180 + let rig: CliRig;
181 +
182 + beforeAll(async () => {
183 + rig = await startCliRig({ label: 'login-key' });
184 + }, RIG_TIMEOUT_MS);
185 +
186 + afterAll(async () => {
187 + await writeDebugLog(rig, import.meta.dir, 'login-key');
188 + await rig.stop();
189 + });
190 +
191 + it(
192 + '`login --key` on a session nobody approved fails without storing a login',
193 + async () => {
194 + await rig.run('composio login --no-wait');
195 + const session = rig.fake.world.loginSessions.at(-1);
196 + const result = await rig.run(`composio login --key ${session?.id} --no-wait`);
197 + expect(result.exitCode).toBe(1);
198 + expect(result.stdout).toBe('');
199 + expect(clean(result.stderr)).toContain('Session not yet linked');
200 + expect((await readStoredLogin(rig)).api_key).toBeNull();
201 + },
202 + TEST_TIMEOUT_MS
203 + );
204 +
205 + it(
206 + '`login --key` on a linked session stores the login',
207 + async () => {
208 + const session = rig.fake.world.linkLoginSession();
209 + const result = await rig.run(
210 + `composio login --key ${session.id} --no-wait --no-skill-install`
211 + );
212 + expect(result.exitCode).toBe(0);
213 + expect(stdoutJson(result, z.looseObject({ email: z.string(), org_id: z.string() }))).toEqual(
214 + expect.objectContaining({ email: MEMBER.email, org_id: ORG.id })
215 + );
216 + expect((await readStoredLogin(rig)).api_key).toBe(USER_API_KEY);
217 + },
218 + TEST_TIMEOUT_MS
219 + );
220 +
221 + it(
222 + '`login --key` with an unknown session names the command that gets a new one',
223 + async () => {
224 + await rig.run('composio logout');
225 + const result = await rig.run('composio login --key no-such-session --no-wait');
226 + expect(result.exitCode).toBe(1);
227 + expect(result.stdout).toBe('');
228 + expect(clean(result.stderr)).toBe(
229 + 'Session not found or expired. Run `composio login --no-wait` to get a new session.'
230 + );
231 + expect((await readStoredLogin(rig)).api_key).toBeNull();
232 + },
233 + TEST_TIMEOUT_MS
234 + );
235 +});
236 +
237 +describe('login session: a user API key given directly', () => {
238 + let rig: CliRig;
239 +
240 + beforeAll(async () => {
241 + rig = await startCliRig({ label: 'login-direct' });
242 + }, RIG_TIMEOUT_MS);
243 +
244 + afterAll(async () => {
245 + await writeDebugLog(rig, import.meta.dir, 'login-direct');
246 + await rig.stop();
247 + });
248 +
249 + it(
250 + 'a key the backend rejects is not stored',
251 + async () => {
252 + const result = await rig.run(
253 + `composio login --user-api-key ${FOREIGN_USER_API_KEY} --no-skill-install`
254 + );
255 + expect(result.exitCode).toBe(1);
256 + expect(result.stdout).toBe('');
257 + expect(clean(result.stderr)).toContain('Invalid API key');
258 + const stored = await rig.run('cat ~/.composio/user_data.json');
259 + expect(stored.stdout).not.toContain(FOREIGN_USER_API_KEY);
260 + },
261 + TEST_TIMEOUT_MS
262 + );
263 +
264 + // CLI defect: a rejected key at login is printed as a defect report with an empty title, not as an expected failure.
265 + it.failing(
266 + 'a key the backend rejects is reported as an expected failure, without a defect report',
267 + async () => {
268 + const result = await rig.run(
269 + `composio login --user-api-key ${FOREIGN_USER_API_KEY} --no-skill-install`
270 + );
271 + expect(clean(result.stderr)).not.toContain('💥');
272 + },
273 + TEST_TIMEOUT_MS
274 + );
275 +
276 + it(
277 + 'an accepted key is stored with the org and the global test user',
278 + async () => {
279 + const result = await rig.login();
280 + expect(result.exitCode).toBe(0);
281 + expect(await readStoredLogin(rig)).toMatchObject({
282 + api_key: USER_API_KEY,
283 + org_id: ORG.id,
284 + test_user_id: GLOBAL_TEST_USER_ID,
285 + });
286 + },
287 + TEST_TIMEOUT_MS
288 + );
289 +});
290 +
291 +describe('session artifacts', () => {
292 + let rig: CliRig;
293 +
294 + beforeAll(async () => {
295 + rig = await startCliRig({ label: 'artifacts' });
296 + }, RIG_TIMEOUT_MS);
297 +
298 + afterAll(async () => {
299 + await writeDebugLog(rig, import.meta.dir, 'artifacts');
300 + await rig.stop();
301 + });
302 +
303 + it(
304 + '`artifacts cwd` prints one absolute directory, without a login or a request',
305 + async () => {
306 + const result = await rig.run('composio artifacts cwd');
307 + expect(result.exitCode).toBe(0);
308 + expect(result.stderr).toBe('');
309 + expect(result.stdout.trim()).toMatch(/^\/\S+$/);
310 + expect(commandRequests(result)).toEqual([]);
311 + },
312 + TEST_TIMEOUT_MS
313 + );
314 +
315 + it(
316 + '`artifacts cwd` is stable for a directory and follows COMPOSIO_SESSION_DIR',
317 + async () => {
318 + const first = await rig.run('composio artifacts cwd');
319 + const second = await rig.run('composio artifacts cwd');
320 + expect(second.stdout).toBe(first.stdout);
321 +
322 + const overridden = await rig.run(
323 + 'COMPOSIO_SESSION_DIR=/tmp/e2e-sessions composio artifacts cwd'
324 + );
325 + expect(overridden.exitCode).toBe(0);
326 + expect(overridden.stdout.trim().startsWith('/tmp/e2e-sessions')).toBe(true);
327 + },
328 + TEST_TIMEOUT_MS
329 + );
330 +});
ts/e2e-tests/cli/operations/network-faults.test.ts new
+257
@@ -0,0 +1,257 @@
1 +/**
2 + * Network faults: what the CLI does when a host it depends on misbehaves.
3 + *
4 + * Two invariants carry most of the weight. A read may be retried, and either
5 + * heals or fails with exit 1, empty stdout, and the cause on stderr. A tool
6 + * execution is sent exactly once whatever happens, because a second attempt
7 + * could act twice.
8 + */
9 +
10 +import { afterAll, afterEach, beforeAll, describe, expect, it } from 'bun:test';
11 +import { CONSUMER_PROJECT, CONSUMER_USER_ID, type Fault } from '@e2e-tests/cli-fake-backend';
12 +import { startCliRig, type CliRig } from '@e2e-tests/cli-fake-backend/rig';
13 +import {
14 + clean,
15 + EXECUTE_SLACK,
16 + ExecutionResult,
17 + RIG_TIMEOUT_MS,
18 + SEARCH_SLACK,
19 + SearchResult,
20 + stdoutJson,
21 + TEST_TIMEOUT_MS,
22 + writeDebugLog,
23 +} from './support';
24 +
25 +const SEARCH = 'toolRouter.session.search';
26 +
27 +describe('network faults on a read (`search`)', () => {
28 + let rig: CliRig;
29 +
30 + beforeAll(async () => {
31 + rig = await startCliRig({ label: 'net-read' });
32 + await rig.login();
33 + }, RIG_TIMEOUT_MS);
34 +
35 + afterEach(() => rig.fake.faults.clear());
36 +
37 + afterAll(async () => {
38 + await writeDebugLog(rig, import.meta.dir, 'net-read');
39 + await rig.stop();
40 + });
41 +
42 + const searchRequests = (result: Awaited<ReturnType<CliRig['run']>>) =>
43 + result.requests.filter(entry => entry.route === SEARCH);
44 +
45 + it(
46 + 'connection refused: fails with the cause and an empty stdout',
47 + async () => {
48 + // Port 9 (discard) has no listener on the host.
49 + const result = await rig.run(
50 + `COMPOSIO_BASE_URL=http://host.docker.internal:9 ${SEARCH_SLACK}`
51 + );
52 + expect(result.exitCode).toBe(1);
53 + expect(result.stdout).toBe('');
54 + expect(clean(result.stderr)).toContain('ECONNREFUSED');
55 + expect(result.requests).toEqual([]);
56 + },
57 + TEST_TIMEOUT_MS
58 + );
59 +
60 + const persistent: ReadonlyArray<{
61 + name: string;
62 + fault: Fault;
63 + stderr: ReadonlyArray<string>;
64 + attempts: number;
65 + }> = [
66 + {
67 + name: 'HTTP 500 on every attempt',
68 + fault: { kind: 'status', status: 500 },
69 + stderr: ['Injected HTTP 500'],
70 + attempts: 3,
71 + },
72 + {
73 + name: 'HTTP 429 on every attempt',
74 + fault: { kind: 'status', status: 429, headers: { 'retry-after': '1' } },
75 + stderr: ['Injected HTTP 429'],
76 + attempts: 3,
77 + },
78 + {
79 + name: 'an HTML error page from a proxy',
80 + fault: { kind: 'html', status: 502 },
81 + stderr: ['502'],
82 + attempts: 3,
83 + },
84 + {
85 + name: 'connection reset before any response',
86 + fault: { kind: 'reset' },
87 + stderr: ['ECONNRESET'],
88 + attempts: 3,
89 + },
90 + {
91 + name: 'connection reset in the middle of the body',
92 + fault: { kind: 'reset-mid-body' },
93 + stderr: ['ECONNRESET'],
94 + attempts: 1,
95 + },
96 + {
97 + name: 'a body that is not JSON',
98 + fault: { kind: 'malformed-json' },
99 + stderr: ['is not in the shape this version of the CLI expects', 'composio upgrade'],
100 + attempts: 1,
101 + },
102 + ];
103 +
104 + for (const { name, fault, stderr, attempts } of persistent) {
105 + it(
106 + `${name}: exit 1, empty stdout, cause on stderr`,
107 + async () => {
108 + rig.fake.faults.inject({ route: SEARCH, fault });
109 + const result = await rig.run(SEARCH_SLACK);
110 +
111 + expect(result.exitCode).toBe(1);
112 + expect(result.stdout).toBe('');
113 + for (const text of stderr) expect(clean(result.stderr)).toContain(text);
114 + expect(searchRequests(result)).toHaveLength(attempts);
115 + },
116 + TEST_TIMEOUT_MS
117 + );
118 + }
119 +
120 + const transient: ReadonlyArray<{ name: string; fault: Fault }> = [
121 + { name: 'one HTTP 500', fault: { kind: 'status', status: 500 } },
122 + {
123 + name: 'one HTTP 429 with Retry-After',
124 + fault: { kind: 'status', status: 429, headers: { 'retry-after': '1' } },
125 + },
126 + { name: 'one connection reset', fault: { kind: 'reset' } },
127 + ];
128 +
129 + for (const { name, fault } of transient) {
130 + it(
131 + `${name}: the retry heals it and the caller sees a normal result`,
132 + async () => {
133 + rig.fake.faults.inject({ route: SEARCH, fault, times: 1 });
134 + const result = await rig.run(SEARCH_SLACK);
135 +
136 + expect(result.exitCode).toBe(0);
137 + expect(result.stderr).toBe('');
138 + expect(stdoutJson(result, SearchResult).results[0]?.primary_tool_slugs).toEqual([
139 + 'SLACK_SEND_MESSAGE',
140 + ]);
141 + expect(searchRequests(result).map(entry => entry.status)).toEqual([
142 + fault.kind === 'status' ? fault.status : 0,
143 + 200,
144 + ]);
145 + },
146 + TEST_TIMEOUT_MS
147 + );
148 + }
149 +
150 + it(
151 + 'a slow response: waits for it instead of giving up',
152 + async () => {
153 + rig.fake.faults.inject({ route: SEARCH, fault: { kind: 'delay', ms: 4000 } });
154 + const result = await rig.run(SEARCH_SLACK);
155 + expect(result.exitCode).toBe(0);
156 + expect(searchRequests(result)).toHaveLength(1);
157 + expect(stdoutJson(result, SearchResult).results).toHaveLength(1);
158 + },
159 + TEST_TIMEOUT_MS
160 + );
161 +
162 + for (const { name, fault } of [
163 + {
164 + name: 'valid JSON of an unexpected shape',
165 + fault: { kind: 'schema-drift', transform: () => ({ unexpected: true }) },
166 + },
167 + { name: 'an empty 200 body', fault: { kind: 'empty-body' } },
168 + ] satisfies ReadonlyArray<{ name: string; fault: Fault }>) {
169 + it(
170 + `today: ${name}: exit 1 with a raw TypeError`,
171 + async () => {
172 + rig.fake.faults.inject({ route: SEARCH, fault });
173 + const result = await rig.run(SEARCH_SLACK);
174 + expect(result.exitCode).toBe(1);
175 + expect(result.stdout).toBe('');
176 + expect(clean(result.stderr)).toContain('TypeError');
177 + },
178 + TEST_TIMEOUT_MS
179 + );
180 +
181 + // CLI defect: a 200 whose JSON lacks the fields `search` reads crashes with a raw TypeError naming a minified variable.
182 + it.failing(
183 + `${name}: reported as an unexpected response`,
184 + async () => {
185 + rig.fake.faults.inject({ route: SEARCH, fault });
186 + const result = await rig.run(SEARCH_SLACK);
187 + expect(clean(result.stderr)).toContain(
188 + 'is not in the shape this version of the CLI expects'
189 + );
190 + },
191 + TEST_TIMEOUT_MS
192 + );
193 + }
194 +});
195 +
196 +const EXECUTION_FAULTS: ReadonlyArray<{ name: string; fault: Fault }> = [
197 + { name: 'HTTP 500', fault: { kind: 'status', status: 500 } },
198 + { name: 'HTTP 429', fault: { kind: 'status', status: 429, headers: { 'retry-after': '1' } } },
199 + { name: 'an HTML error page', fault: { kind: 'html', status: 502 } },
200 + { name: 'a body that is not JSON', fault: { kind: 'malformed-json' } },
201 + { name: 'connection reset before any response', fault: { kind: 'reset' } },
202 + { name: 'connection reset in the middle of the body', fault: { kind: 'reset-mid-body' } },
203 +];
204 +
205 +for (const transport of [
206 + { name: 'Dashboard', dashboard: true, route: 'dashboard.execute' },
207 + { name: 'backend', dashboard: false, route: 'toolRouter.session.execute' },
208 +] as const) {
209 + describe(`network faults on a tool execution (${transport.name} transport)`, () => {
210 + let rig: CliRig;
211 +
212 + beforeAll(async () => {
213 + rig = await startCliRig({
214 + label: `net-exec-${transport.name.toLowerCase()}`,
215 + dashboard: transport.dashboard,
216 + });
217 + await rig.login();
218 + rig.fake.world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
219 + }, RIG_TIMEOUT_MS);
220 +
221 + afterEach(() => rig.fake.faults.clear());
222 +
223 + afterAll(async () => {
224 + await writeDebugLog(rig, import.meta.dir, `net-exec-${transport.name.toLowerCase()}`);
225 + await rig.stop();
226 + });
227 +
228 + it(
229 + 'without a fault the execution goes through this transport once',
230 + async () => {
231 + const result = await rig.run(EXECUTE_SLACK);
232 + expect(result.exitCode).toBe(0);
233 + expect(stdoutJson(result, ExecutionResult).successful).toBe(true);
234 + expect(result.requests.filter(entry => entry.route === transport.route)).toHaveLength(1);
235 + },
236 + TEST_TIMEOUT_MS
237 + );
238 +
239 + for (const { name, fault } of EXECUTION_FAULTS) {
240 + it(
241 + `${name}: sent exactly once, reported as a failed result on stdout`,
242 + async () => {
243 + rig.fake.faults.inject({ route: transport.route, fault });
244 + const result = await rig.run(EXECUTE_SLACK);
245 +
246 + expect(result.exitCode).toBe(1);
247 + expect(result.requests.filter(entry => entry.route === transport.route)).toHaveLength(1);
248 + const outcome = stdoutJson(result, ExecutionResult);
249 + expect(outcome.successful).toBe(false);
250 + expect(outcome.error ?? '').not.toBe('');
251 + expect(outcome.data).toBeUndefined();
252 + },
253 + TEST_TIMEOUT_MS
254 + );
255 + }
256 + });
257 +}
ts/e2e-tests/cli/operations/package.json new
+18
@@ -0,0 +1,18 @@
1 +{
2 + "name": "@e2e-tests/cli-operations",
3 + "version": "0.0.0",
4 + "private": true,
5 + "type": "module",
6 + "scripts": {
7 + "typecheck": "tsc --noEmit",
8 + "test:e2e": "bun test",
9 + "test:e2e:cli": "bun test"
10 + },
11 + "devDependencies": {
12 + "@e2e-tests/cli-fake-backend": "workspace:*",
13 + "@e2e-tests/utils": "workspace:*",
14 + "@types/bun": "catalog:",
15 + "typescript": "catalog:",
16 + "zod": "catalog:"
17 + }
18 +}
ts/e2e-tests/cli/operations/process-streams.test.ts new
+288
@@ -0,0 +1,288 @@
1 +/**
2 + * Process and streams: signals, a closed stdout pipe, and the three stream
3 + * contracts (stdout is data only; decoration, notices, and hints stay off it).
4 + */
5 +
6 +import { afterAll, beforeAll, describe, expect, it } from 'bun:test';
7 +import { CONSUMER_PROJECT, CONSUMER_USER_ID, MEMBER } from '@e2e-tests/cli-fake-backend';
8 +import { startCliRig, type CliRig } from '@e2e-tests/cli-fake-backend/rig';
9 +import { z } from 'zod';
10 +import {
11 + clean,
12 + ExecutionResult,
13 + RIG_TIMEOUT_MS,
14 + runWithTerminal,
15 + SEARCH_SLACK,
16 + SearchResult,
17 + stdoutJson,
18 + TEST_TIMEOUT_MS,
19 + writeDebugLog,
20 +} from './support';
21 +
22 +const Whoami = z.looseObject({ email: z.string() });
23 +const UPDATE_NOTICE = 'Update available';
24 +const PLUGIN_HINT = 'Tip: running under Claude Code';
25 +
26 +describe('process: signals', () => {
27 + let rig: CliRig;
28 +
29 + beforeAll(async () => {
30 + rig = await startCliRig({ label: 'signals' });
31 + await rig.login();
32 + // Nobody finishes the authorization, so `link` is still waiting when the signal arrives.
33 + rig.fake.world.linkCompletesAfterPolls = null;
34 + }, RIG_TIMEOUT_MS);
35 +
36 + afterAll(async () => {
37 + await writeDebugLog(rig, import.meta.dir, 'signals');
38 + await rig.stop();
39 + });
40 +
41 + /**
42 + * Sends the signal once `ready` is on stderr, so a slow start cannot make the
43 + * signal arrive before the command is waiting. `settle` seconds after the
44 + * command ended, stdout is measured, which catches a child left running.
45 + */
46 + const interrupted = (command: string, signal: 'INT' | 'TERM', ready: string, settle = 0) =>
47 + [
48 + `${command} > /tmp/sig.out 2> /tmp/sig.err &`,
49 + 'pid=$!',
50 + `for i in $(seq 1 150); do grep -q '${ready}' /tmp/sig.err && break; sleep 0.1; done`,
51 + `grep -q '${ready}' /tmp/sig.err || echo "never ready"`,
52 + `kill -${signal} $pid`,
53 + 'wait $pid',
54 + 'echo "status=$?"',
55 + `sleep ${settle}`,
56 + 'echo "stdout-bytes=$(wc -c < /tmp/sig.out)"',
57 + ].join('\n');
58 +
59 + for (const { signal, status, toolkit } of [
60 + { signal: 'INT', status: 130, toolkit: 'slack' },
61 + { signal: 'TERM', status: 143, toolkit: 'github' },
62 + ] as const) {
63 + it(
64 + `SIG${signal} while \`link\` waits: exits ${status} with nothing on stdout`,
65 + async () => {
66 + const result = await rig.run(interrupted(`composio link ${toolkit}`, signal, '/api/v3/s/'));
67 + expect(result.stdout).not.toContain('never ready');
68 + expect(result.stdout).toContain(`status=${status}`);
69 + expect(result.stdout).toContain('stdout-bytes=0');
70 + // The link was requested once, and the interrupted wait did not request another.
71 + expect(
72 + result.requests.filter(entry => entry.route === 'toolRouter.session.link')
73 + ).toHaveLength(1);
74 + },
75 + TEST_TIMEOUT_MS
76 + );
77 + }
78 +
79 + it(
80 + 'SIGINT while `run` executes a script: exits 130 and the script stops',
81 + async () => {
82 + const result = await rig.run(
83 + interrupted(
84 + `composio run 'console.error("started"); await new Promise(resolve => setTimeout(resolve, 4000)); console.log("finished")'`,
85 + 'INT',
86 + 'started',
87 + // Longer than the script's 4 seconds: a script left running would print by then.
88 + 5
89 + )
90 + );
91 + expect(result.stdout).not.toContain('never ready');
92 + expect(result.stdout).toContain('status=130');
93 + expect(result.stdout).toContain('stdout-bytes=0');
94 + },
95 + TEST_TIMEOUT_MS
96 + );
97 +
98 + it(
99 + '`link --no-browser` off a terminal prints the authorization URL on stderr before it waits',
100 + async () => {
101 + // Killed while it waits: whatever it printed, it printed before the authorization.
102 + const result = await rig.run('composio link gmail --no-browser', { timeoutSeconds: 6 });
103 + const connection = rig.fake.world.connections.at(-1);
104 + expect(result.exitCode).toBe(124);
105 + expect(result.stderr).toContain(`${rig.fake.docker.http}/api/v3/s/${connection?.id}`);
106 + expect(result.stderr).toContain('For agents: Show the URL above to the user');
107 + expect(result.stderr).toContain('--no-wait');
108 + // stdout is for the connected account's JSON, which only comes once it is active.
109 + expect(result.stdout).toBe('');
110 + },
111 + TEST_TIMEOUT_MS
112 + );
113 +
114 + it(
115 + '`link --no-wait` prints the authorization URL and returns at once',
116 + async () => {
117 + const result = await rig.run('composio link gmail --no-wait');
118 + expect(result.exitCode).toBe(0);
119 + const connection = rig.fake.world.connections.at(-1);
120 + expect(
121 + stdoutJson(result, z.looseObject({ status: z.string(), redirect_url: z.string() }))
122 + ).toMatchObject({
123 + status: 'pending',
124 + redirect_url: `${rig.fake.docker.http}/api/v3/s/${connection?.id}`,
125 + });
126 + expect(result.requests.map(entry => entry.route)).not.toContain('connectedAccounts.retrieve');
127 + },
128 + TEST_TIMEOUT_MS
129 + );
130 +});
131 +
132 +describe('streams', () => {
133 + let rig: CliRig;
134 +
135 + beforeAll(async () => {
136 + rig = await startCliRig({ label: 'streams' });
137 + await rig.login();
138 + rig.fake.world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
139 + }, RIG_TIMEOUT_MS);
140 +
141 + afterAll(async () => {
142 + await writeDebugLog(rig, import.meta.dir, 'streams');
143 + await rig.stop();
144 + });
145 +
146 + const seedUpdateAvailable = () =>
147 + rig.run(
148 + `printf '{"lastChecked":"%s","latestVersion":"99.0.0"}' "$(date -u +%Y-%m-%dT%H:%M:%S.000Z)" > ~/.composio/update-check.json`
149 + );
150 +
151 + // `true` exits without reading, so the pipe is closed before the CLI writes; a
152 + // reader like `head -1` would let a small result fit in the pipe's buffer.
153 + const CLOSED_EARLY = `{ ${SEARCH_SLACK}; echo $? > /tmp/pipe.rc; } | true; cat /tmp/pipe.rc`;
154 +
155 + it(
156 + 'today: a reader that closes stdout before the CLI writes makes it exit 1 with an EPIPE trace',
157 + async () => {
158 + const result = await rig.run(CLOSED_EARLY);
159 + expect(result.stdout).toBe('1\n');
160 + expect(result.stderr).toContain('EPIPE');
161 + },
162 + TEST_TIMEOUT_MS
163 + );
164 +
165 + // CLI defect: a closed stdout is reported as an EPIPE stack trace, and the command fails.
166 + it.failing(
167 + 'a reader that closes stdout early does not fail the command',
168 + async () => {
169 + const result = await rig.run(CLOSED_EARLY);
170 + expect(result.stderr).not.toContain('EPIPE');
171 + },
172 + TEST_TIMEOUT_MS
173 + );
174 +
175 + it(
176 + 'stdin: `-d -` reads the arguments from a pipe',
177 + async () => {
178 + const result = await rig.run('composio execute SLACK_SEND_MESSAGE -d -', {
179 + stdin: '{"channel":"general","text":"from stdin"}',
180 + });
181 + expect(result.exitCode).toBe(0);
182 + expect(stdoutJson(result, ExecutionResult).data).toMatchObject({
183 + message: { text: 'from stdin' },
184 + });
185 + },
186 + TEST_TIMEOUT_MS
187 + );
188 +
189 + it(
190 + 'stdin: `-d -` with nothing on stdin fails without waiting for input',
191 + async () => {
192 + const result = await rig.run('composio execute SLACK_SEND_MESSAGE -d -', {
193 + timeoutSeconds: 10,
194 + });
195 + expect(result.exitCode).toBe(1);
196 + expect(result.stdout).toBe('');
197 + expect(clean(result.stderr)).toContain('Invalid JSON input');
198 + expect(result.requests.map(entry => entry.route)).not.toContain('dashboard.execute');
199 + },
200 + TEST_TIMEOUT_MS
201 + );
202 +
203 + it(
204 + 'no terminal at all: stdout is the JSON and stderr is empty',
205 + async () => {
206 + const result = await rig.run('composio whoami');
207 + expect(result.exitCode).toBe(0);
208 + expect(stdoutJson(result, Whoami).email).toBe(MEMBER.email);
209 + expect(result.stderr).toBe('');
210 + },
211 + TEST_TIMEOUT_MS
212 + );
213 +
214 + it(
215 + 'stdout piped, stderr a terminal: the JSON goes to the pipe, decoration to the terminal',
216 + async () => {
217 + const result = await runWithTerminal(rig, 'composio whoami > /tmp/tty.out');
218 + expect(result.exitCode).toBe(0);
219 + const piped = await rig.journey.readFile('/tmp/tty.out');
220 + expect(Whoami.parse(JSON.parse(piped)).email).toBe(MEMBER.email);
221 + expect(result.terminal).toContain(MEMBER.email);
222 + // The terminal shows the human rendering, not a second copy of the data.
223 + expect(result.terminal).not.toContain('"account_type"');
224 + },
225 + TEST_TIMEOUT_MS
226 + );
227 +
228 + it(
229 + 'stdout a terminal, stderr redirected: no data leaks onto the terminal',
230 + async () => {
231 + const result = await runWithTerminal(rig, 'composio whoami 2> /tmp/tty.err');
232 + expect(result.exitCode).toBe(0);
233 + expect(result.terminal).not.toContain('"account_type"');
234 + expect(await rig.journey.readFile('/tmp/tty.err')).toBe('');
235 + },
236 + TEST_TIMEOUT_MS
237 + );
238 +
239 + it(
240 + 'the update notice is never written to stdout',
241 + async () => {
242 + await seedUpdateAvailable();
243 + const piped = await rig.run('composio whoami');
244 + expect(stdoutJson(piped, Whoami).email).toBe(MEMBER.email);
245 + expect(piped.stdout).not.toContain(UPDATE_NOTICE);
246 +
247 + const withTerminal = await runWithTerminal(rig, 'composio whoami > /tmp/notice.out');
248 + expect(withTerminal.terminal).toContain(`${UPDATE_NOTICE}:`);
249 + expect(withTerminal.terminal).toContain('99.0.0');
250 + const data = await rig.journey.readFile('/tmp/notice.out');
251 + expect(Whoami.parse(JSON.parse(data)).email).toBe(MEMBER.email);
252 + },
253 + TEST_TIMEOUT_MS
254 + );
255 +
256 + it(
257 + 'the plugin hint under an agent host is written to stderr, never stdout',
258 + async () => {
259 + const result = await rig.run(`CLAUDECODE=1 ${SEARCH_SLACK}`);
260 + expect(result.exitCode).toBe(0);
261 + expect(stdoutJson(result, SearchResult).results).toHaveLength(1);
262 + expect(result.stdout).not.toContain(PLUGIN_HINT);
263 + expect(clean(result.stderr)).toContain(PLUGIN_HINT);
264 + },
265 + TEST_TIMEOUT_MS
266 + );
267 +
268 + it(
269 + 'today: `search --human` off a terminal prints nothing and exits 0',
270 + async () => {
271 + const result = await rig.run(`${SEARCH_SLACK} --human`);
272 + expect(result.exitCode).toBe(0);
273 + expect(result.stdout).toBe('');
274 + expect(result.stderr).toBe('');
275 + },
276 + TEST_TIMEOUT_MS
277 + );
278 +
279 + // CLI defect: `--human` renders through terminal decoration only, so off a terminal it prints nothing and exits 0.
280 + it.failing(
281 + '`search --human` off a terminal still prints the results',
282 + async () => {
283 + const result = await rig.run(`${SEARCH_SLACK} --human`);
284 + expect(`${result.stdout}${result.stderr}`).toContain('SLACK_SEND_MESSAGE');
285 + },
286 + TEST_TIMEOUT_MS
287 + );
288 +});
ts/e2e-tests/cli/operations/support.ts new
+110
@@ -0,0 +1,110 @@
1 +/**
2 + * Helpers shared by the operation suites.
3 + */
4 +
5 +import { join } from 'node:path';
6 +import { sanitizeOutput } from '@e2e-tests/utils';
7 +import type { JournalEntry } from '@e2e-tests/cli-fake-backend';
8 +import type { CliRig, CommandResult } from '@e2e-tests/cli-fake-backend/rig';
9 +import { z } from 'zod';
10 +
11 +/** Per-test budget: a command is a few hundred milliseconds, a retried one a few seconds. */
12 +export const TEST_TIMEOUT_MS = 90_000;
13 +
14 +/** Budget for starting a rig, which may build the CLI image on a cold machine. */
15 +export const RIG_TIMEOUT_MS = 1_200_000;
16 +
17 +export const SLACK_ARGUMENTS = `'{"channel":"general","text":"hello"}'`;
18 +export const EXECUTE_SLACK = `composio execute SLACK_SEND_MESSAGE -d ${SLACK_ARGUMENTS}`;
19 +export const PLAYGROUND_EXECUTE_SLACK = `composio dev playground-execute SLACK_SEND_MESSAGE -d ${SLACK_ARGUMENTS}`;
20 +export const SEARCH_SLACK = 'composio search "send a slack message"';
21 +
22 +/** A directory outside the journey's home, initialized by `composio dev init`. */
23 +export const DEV_DIRECTORY = '/work/app';
24 +
25 +export const clean = sanitizeOutput;
26 +
27 +/**
28 + * The requests a command made to do its job. The update check and telemetry
29 + * run beside every command and are asserted on their own.
30 + */
31 +export const commandRequests = (result: CommandResult): ReadonlyArray<JournalEntry> =>
32 + result.requests.filter(entry => entry.role === 'backend' || entry.role === 'dashboard');
33 +
34 +export const routesOf = (result: CommandResult): string[] =>
35 + commandRequests(result).map(entry => entry.route ?? `unrouted ${entry.method} ${entry.path}`);
36 +
37 +/** Parses stdout as JSON with `schema`; a stdout that is not pure JSON fails the test. */
38 +export const stdoutJson = <T>(result: CommandResult, schema: z.ZodType<T>): T => {
39 + const raw = clean(result.stdout);
40 + let parsed: unknown;
41 + try {
42 + parsed = JSON.parse(raw);
43 + } catch (cause) {
44 + throw new Error(
45 + `stdout is not pure JSON (exit ${result.exitCode}).\nstdout: ${raw.slice(0, 400)}\nstderr: ${clean(result.stderr).slice(0, 400)}`,
46 + { cause }
47 + );
48 + }
49 + return schema.parse(parsed);
50 +};
51 +
52 +export const ExecutionResult = z.looseObject({
53 + successful: z.boolean(),
54 + error: z.string().nullable().optional(),
55 + slug: z.string().optional(),
56 + data: z.record(z.string(), z.unknown()).optional(),
57 +});
58 +
59 +export const SearchResult = z.looseObject({
60 + results: z.array(
61 + z.looseObject({ use_case: z.string(), primary_tool_slugs: z.array(z.string()) })
62 + ),
63 + next_steps: z.looseObject({
64 + steps: z.array(z.looseObject({ action: z.string(), command: z.string() })),
65 + }),
66 +});
67 +
68 +/** The stored login, as `~/.composio/user_data.json` holds it. */
69 +export const StoredLogin = z.looseObject({
70 + api_key: z.string().nullable(),
71 + base_url: z.string().nullable(),
72 + web_url: z.string().nullable(),
73 + org_id: z.string().nullable(),
74 + test_user_id: z.string().nullable(),
75 +});
76 +
77 +export const readStoredLogin = async (rig: CliRig) =>
78 + StoredLogin.parse(
79 + JSON.parse(await rig.journey.readFile(`${rig.journey.home}/.composio/user_data.json`))
80 + );
81 +
82 +/** Initializes `DEV_DIRECTORY` for the org's default developer project. */
83 +export const initDeveloperDirectory = async (rig: CliRig): Promise<void> => {
84 + const result = await rig.run(
85 + `mkdir -p ${DEV_DIRECTORY} && cd ${DEV_DIRECTORY} && composio dev init --yes --no-browser`
86 + );
87 + if (result.exitCode !== 0) {
88 + throw new Error(`composio dev init failed: ${clean(result.stderr) || clean(result.stdout)}`);
89 + }
90 +};
91 +
92 +/**
93 + * Runs `command` with a pseudo-terminal on all three streams, then applies the
94 + * redirections written into the command itself. `script` allocates the
95 + * terminal; what the terminal showed lands in the returned `terminal` text.
96 + */
97 +export const runWithTerminal = async (rig: CliRig, command: string) => {
98 + const transcript = '/tmp/e2e-terminal.txt';
99 + const quoted = `'${`stty cols 200; ${command}`.replace(/'/g, `'\\''`)}'`;
100 + const result = await rig.run(`script -qec ${quoted} ${transcript} > /dev/null`);
101 + const terminal = clean(await rig.journey.readFile(transcript))
102 + .split('\n')
103 + .filter(line => !line.startsWith('Script started on') && !line.startsWith('Script done on'))
104 + .join('\n');
105 + return { ...result, terminal };
106 +};
107 +
108 +/** Writes the journey's steps next to the suite, as the other CLI suites do. */
109 +export const writeDebugLog = (rig: CliRig, importMetaDir: string, name: string): Promise<void> =>
110 + rig.journey.writeDebugLog(join(importMetaDir, `DEBUG.${name}.log`));
ts/e2e-tests/cli/operations/tool-loop.test.ts new
+292
@@ -0,0 +1,292 @@
1 +/**
2 + * The loop agents live in, without a model: search, execute, link, execute
3 + * again, then the commands around it (`connections`, `--get-schema`,
4 + * `--dry-run`, `run`). Each step checks what the fake backend received, so a
5 + * step that printed success without acting fails here.
6 + */
7 +
8 +import { afterAll, beforeAll, describe, expect, it } from 'bun:test';
9 +import { CONSUMER_PROJECT, CONSUMER_USER_ID } from '@e2e-tests/cli-fake-backend';
10 +import { startCliRig, type CliRig } from '@e2e-tests/cli-fake-backend/rig';
11 +import { z } from 'zod';
12 +import {
13 + clean,
14 + commandRequests,
15 + EXECUTE_SLACK,
16 + ExecutionResult,
17 + RIG_TIMEOUT_MS,
18 + routesOf,
19 + SEARCH_SLACK,
20 + SearchResult,
21 + stdoutJson,
22 + TEST_TIMEOUT_MS,
23 + writeDebugLog,
24 +} from './support';
25 +
26 +const Connections = z.record(
27 + z.string(),
28 + z.array(z.looseObject({ id: z.string(), status: z.string() }))
29 +);
30 +
31 +describe('tool loop: search, execute, link, execute', () => {
32 + let rig: CliRig;
33 +
34 + beforeAll(async () => {
35 + rig = await startCliRig({ label: 'tool-loop' });
36 + await rig.login();
37 + }, RIG_TIMEOUT_MS);
38 +
39 + afterAll(async () => {
40 + await writeDebugLog(rig, import.meta.dir, 'tool-loop');
41 + await rig.stop();
42 + });
43 +
44 + it(
45 + '`search` finds the slug and names the next commands',
46 + async () => {
47 + const result = await rig.run(SEARCH_SLACK);
48 + expect(result.exitCode).toBe(0);
49 + expect(result.stderr).toBe('');
50 + const found = stdoutJson(result, SearchResult);
51 + expect(found.results[0]?.primary_tool_slugs).toEqual(['SLACK_SEND_MESSAGE']);
52 + // Not connected yet, so the first suggested step is the link.
53 + expect(found.next_steps.steps.map(step => step.command)[0]).toBe('composio link slack');
54 +
55 + const search = result.requests.find(entry => entry.route === 'toolRouter.session.search');
56 + expect(search).toMatchObject({ project: CONSUMER_PROJECT.label, userId: CONSUMER_USER_ID });
57 + expect(search?.body).toEqual({ queries: [{ use_case: 'send a slack message' }] });
58 + },
59 + TEST_TIMEOUT_MS
60 + );
61 +
62 + it(
63 + '`execute` before linking fails with the command that fixes it',
64 + async () => {
65 + const result = await rig.run(EXECUTE_SLACK);
66 + expect(result.exitCode).toBe(1);
67 + expect(result.stderr).toBe('');
68 + expect(stdoutJson(result, ExecutionResult)).toMatchObject({
69 + successful: false,
70 + error:
71 + 'No active connection found for toolkit "slack". Run `composio link slack`, then retry.',
72 + slug: 'ToolRouterV2_NoActiveConnection',
73 + });
74 + },
75 + TEST_TIMEOUT_MS
76 + );
77 +
78 + it(
79 + '`connections list` shows nothing connected yet',
80 + async () => {
81 + const result = await rig.run('composio connections list');
82 + expect(result.exitCode).toBe(0);
83 + expect(stdoutJson(result, Connections)).toEqual({});
84 + },
85 + TEST_TIMEOUT_MS
86 + );
87 +
88 + it(
89 + '`link` creates the connection for the consumer user and waits until it is active',
90 + async () => {
91 + const result = await rig.run('composio link slack');
92 + expect(result.exitCode).toBe(0);
93 + expect(
94 + stdoutJson(result, z.looseObject({ status: z.string(), toolkit: z.string() }))
95 + ).toMatchObject({ status: 'success', toolkit: 'slack' });
96 +
97 + const link = result.requests.filter(entry => entry.route === 'toolRouter.session.link');
98 + expect(link).toHaveLength(1);
99 + expect(link[0]).toMatchObject({
100 + project: CONSUMER_PROJECT.label,
101 + userId: CONSUMER_USER_ID,
102 + body: { toolkit: 'slack' },
103 + });
104 + expect(routesOf(result)).toContain('connectedAccounts.retrieve');
105 + expect(
106 + rig.fake.world.activeConnection(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack')
107 + ).toBeDefined();
108 + },
109 + TEST_TIMEOUT_MS
110 + );
111 +
112 + it(
113 + '`execute` after linking runs the tool with the given arguments',
114 + async () => {
115 + const result = await rig.run(EXECUTE_SLACK);
116 + expect(result.exitCode).toBe(0);
117 + expect(stdoutJson(result, ExecutionResult)).toMatchObject({
118 + successful: true,
119 + data: { ok: true, channel: 'general', message: { text: 'hello' } },
120 + });
121 +
122 + const execution = result.requests.filter(entry => entry.route === 'dashboard.execute');
123 + expect(execution).toHaveLength(1);
124 + expect(execution[0]?.body).toMatchObject({
125 + json: { tool_slug: 'SLACK_SEND_MESSAGE', arguments: { channel: 'general', text: 'hello' } },
126 + });
127 + },
128 + TEST_TIMEOUT_MS
129 + );
130 +
131 + it(
132 + '`connections list` shows the linked account as active',
133 + async () => {
134 + const result = await rig.run('composio connections list');
135 + expect(result.exitCode).toBe(0);
136 + const connections = stdoutJson(result, Connections);
137 + expect(Object.keys(connections)).toEqual(['slack']);
138 + expect(connections.slack?.map(account => account.status)).toEqual(['ACTIVE']);
139 + },
140 + TEST_TIMEOUT_MS
141 + );
142 +});
143 +
144 +describe('tool loop: looking before acting', () => {
145 + let rig: CliRig;
146 +
147 + beforeAll(async () => {
148 + rig = await startCliRig({ label: 'tool-inspect' });
149 + await rig.login();
150 + rig.fake.world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
151 + }, RIG_TIMEOUT_MS);
152 +
153 + afterAll(async () => {
154 + await writeDebugLog(rig, import.meta.dir, 'tool-inspect');
155 + await rig.stop();
156 + });
157 +
158 + const executions = (result: Awaited<ReturnType<CliRig['run']>>) =>
159 + result.requests.filter(entry => entry.route === 'dashboard.execute');
160 +
161 + it(
162 + '`--get-schema` prints the input schema and executes nothing',
163 + async () => {
164 + const result = await rig.run('composio execute SLACK_SEND_MESSAGE --get-schema');
165 + expect(result.exitCode).toBe(0);
166 + const schema = stdoutJson(
167 + result,
168 + z.looseObject({
169 + slug: z.string(),
170 + inputSchema: z.looseObject({ required: z.array(z.string()) }),
171 + })
172 + );
173 + expect(schema.slug).toBe('SLACK_SEND_MESSAGE');
174 + expect(schema.inputSchema.required).toEqual(['channel', 'text']);
175 + expect(executions(result)).toEqual([]);
176 + },
177 + TEST_TIMEOUT_MS
178 + );
179 +
180 + it(
181 + '`--dry-run` validates and reports the acting user without executing',
182 + async () => {
183 + const result = await rig.run(`${EXECUTE_SLACK} --dry-run`);
184 + expect(result.exitCode).toBe(0);
185 + expect(
186 + stdoutJson(result, z.looseObject({ dryRun: z.boolean(), userId: z.string() }))
187 + ).toMatchObject({ dryRun: true, userId: CONSUMER_USER_ID });
188 + expect(executions(result)).toEqual([]);
189 + },
190 + TEST_TIMEOUT_MS
191 + );
192 +
193 + it(
194 + 'arguments that miss a required field are rejected before the tool runs',
195 + async () => {
196 + const result = await rig.run(
197 + `composio execute SLACK_SEND_MESSAGE -d '{"channel":"general"}'`
198 + );
199 + expect(result.exitCode).toBe(1);
200 + const outcome = stdoutJson(result, ExecutionResult);
201 + expect(outcome.successful).toBe(false);
202 + expect(outcome.error).toContain('text');
203 + expect(executions(result)).toEqual([]);
204 + },
205 + TEST_TIMEOUT_MS
206 + );
207 +
208 + it(
209 + 'an unknown tool slug fails with the backend message',
210 + async () => {
211 + const result = await rig.run(`composio execute SLACK_NO_SUCH_TOOL -d '{}'`);
212 + expect(result.exitCode).toBe(1);
213 + expect(stdoutJson(result, ExecutionResult)).toMatchObject({
214 + successful: false,
215 + error: 'Tool SLACK_NO_SUCH_TOOL not found',
216 + });
217 + },
218 + TEST_TIMEOUT_MS
219 + );
220 +});
221 +
222 +describe('tool loop: `run` with an inline script', () => {
223 + let rig: CliRig;
224 +
225 + beforeAll(async () => {
226 + rig = await startCliRig({ label: 'run' });
227 + await rig.login();
228 + rig.fake.world.connect(CONSUMER_PROJECT.label, CONSUMER_USER_ID, 'slack');
229 + }, RIG_TIMEOUT_MS);
230 +
231 + afterAll(async () => {
232 + await writeDebugLog(rig, import.meta.dir, 'run');
233 + await rig.stop();
234 + });
235 +
236 + it(
237 + 'the injected `execute()` runs the tool on the consumer project and returns its result',
238 + async () => {
239 + const result = await rig.run(
240 + `composio run 'const sent = await execute("SLACK_SEND_MESSAGE", { channel: "general", text: "from run" }); console.log(JSON.stringify({ ok: sent.data.ok, text: sent.data.message.text }));'`
241 + );
242 + expect(result.exitCode).toBe(0);
243 + expect(result.stdout.trim()).toBe('{"ok":true,"text":"from run"}');
244 +
245 + const execution = result.requests.filter(entry => entry.route === 'dashboard.execute');
246 + expect(execution).toHaveLength(1);
247 + expect(execution[0]).toMatchObject({
248 + project: CONSUMER_PROJECT.label,
249 + userId: CONSUMER_USER_ID,
250 + keyClass: 'user',
251 + });
252 + },
253 + TEST_TIMEOUT_MS
254 + );
255 +
256 + it(
257 + 'the injected `search()` returns what `composio search` prints',
258 + async () => {
259 + const result = await rig.run(
260 + `composio run 'const found = await search("send a slack message"); console.log(found.results[0].primary_tool_slugs.join(","));'`
261 + );
262 + expect(result.exitCode).toBe(0);
263 + expect(result.stdout.trim()).toBe('SLACK_SEND_MESSAGE');
264 + expect(routesOf(result)).toContain('toolRouter.session.search');
265 + },
266 + TEST_TIMEOUT_MS
267 + );
268 +
269 + it(
270 + 'an `execute()` that is not connected fails the script with the link command on stderr',
271 + async () => {
272 + const result = await rig.run(
273 + `composio run 'await execute("GITHUB_GET_THE_AUTHENTICATED_USER"); console.log("unreachable");'`
274 + );
275 + expect(result.exitCode).toBe(1);
276 + expect(result.stdout).toBe('');
277 + expect(clean(result.stderr)).toContain('Run `composio link github`, then retry.');
278 + },
279 + TEST_TIMEOUT_MS
280 + );
281 +
282 + it(
283 + 'a script that never calls a helper makes no tool request',
284 + async () => {
285 + const result = await rig.run(`composio run 'console.log(1 + 1)'`);
286 + expect(result.exitCode).toBe(0);
287 + expect(result.stdout.trim()).toBe('2');
288 + expect(commandRequests(result).filter(entry => entry.route?.includes('execute'))).toEqual([]);
289 + },
290 + TEST_TIMEOUT_MS
291 + );
292 +});
ts/e2e-tests/cli/operations/tsconfig.json new
+13
@@ -0,0 +1,13 @@
1 +{
2 + "$schema": "https://json.schemastore.org/tsconfig",
3 + "compilerOptions": {
4 + "target": "es2022",
5 + "module": "esnext",
6 + "moduleResolution": "bundler",
7 + "strict": true,
8 + "skipLibCheck": true,
9 + "noEmit": true,
10 + "types": ["bun"]
11 + },
12 + "include": ["*.ts", "src/**/*.ts"]
13 +}
ts/e2e-tests/cli/operations/upgrade.test.ts new
+185
@@ -0,0 +1,185 @@
1 +/**
2 + * Upgrade faults: `composio upgrade` against a fake GitHub that publishes
3 + * releases with something wrong. The invariant in every row is that a failed
4 + * upgrade leaves the installed executable untouched and runnable.
5 + *
6 + * The successful replacement itself is covered by `../upgrade`, which hands
7 + * the CLI a local bundle; here the release metadata and the download are real
8 + * HTTP, and only the archive bytes are not a real build.
9 + */
10 +
11 +import { afterAll, afterEach, beforeAll, describe, expect, it } from 'bun:test';
12 +import type { Release } from '@e2e-tests/cli-fake-backend';
13 +import { startCliRig, type CliRig, type CommandResult } from '@e2e-tests/cli-fake-backend/rig';
14 +import { clean, RIG_TIMEOUT_MS, TEST_TIMEOUT_MS, writeDebugLog } from './support';
15 +
16 +const EXECUTABLE = '/usr/local/bin/composio';
17 +const TAG = '@composio/cli@2.0.0';
18 +const NOT_A_ZIP = new TextEncoder().encode('not a zip archive\n');
19 +const sha256 = (bytes: Uint8Array) => new Bun.CryptoHasher('sha256').update(bytes).digest('hex');
20 +const text = (value: string) => new TextEncoder().encode(value);
21 +
22 +describe('upgrade faults', () => {
23 + let rig: CliRig;
24 + let archive: string;
25 + let fingerprintBefore: string;
26 +
27 + /** Inode, size, and mtime: any replacement or rewrite of the executable changes one of them. */
28 + const fingerprint = async () => (await rig.run(`stat -c '%i %s %Y' ${EXECUTABLE}`)).stdout.trim();
29 +
30 + const publish = (...releases: Release[]) => {
31 + rig.fake.world.releases.length = 0;
32 + rig.fake.world.releases.push(...releases);
33 + };
34 +
35 + const expectUntouched = async (result: CommandResult) => {
36 + expect(result.stdout).toBe('');
37 + expect(await fingerprint()).toBe(fingerprintBefore);
38 + const version = await rig.run('composio version');
39 + expect(version.exitCode).toBe(0);
40 + expect(version.stdout.trim()).toMatch(/^\d+\.\d+\.\d+/);
41 + };
42 +
43 + const downloads = (result: CommandResult) =>
44 + result.requests.filter(entry => entry.route === 'github.releases.download');
45 +
46 + beforeAll(async () => {
47 + rig = await startCliRig({ label: 'upgrade' });
48 + const machine = (await rig.run('uname -m')).stdout.trim();
49 + archive = `composio-linux-${machine === 'aarch64' ? 'aarch64' : 'x64'}.zip`;
50 + fingerprintBefore = await fingerprint();
51 + }, RIG_TIMEOUT_MS);
52 +
53 + afterEach(() => rig.fake.faults.clear());
54 +
55 + afterAll(async () => {
56 + await writeDebugLog(rig, import.meta.dir, 'upgrade');
57 + await rig.stop();
58 + });
59 +
60 + it(
61 + 'no asset for this platform: fails before downloading anything',
62 + async () => {
63 + publish({
64 + tag: TAG,
65 + prerelease: false,
66 + assets: [{ name: 'composio-darwin-aarch64.zip', read: () => NOT_A_ZIP }],
67 + });
68 + const result = await rig.run('composio upgrade');
69 + expect(result.exitCode).toBe(1);
70 + expect(clean(result.stderr)).toContain(`No stable CLI releases found with ${archive}`);
71 + expect(downloads(result)).toEqual([]);
72 + await expectUntouched(result);
73 + },
74 + TEST_TIMEOUT_MS
75 + );
76 +
77 + it(
78 + 'checksum mismatch: the archive is rejected before extraction',
79 + async () => {
80 + publish({
81 + tag: TAG,
82 + prerelease: false,
83 + assets: [
84 + { name: archive, read: () => NOT_A_ZIP },
85 + { name: 'checksums.txt', read: () => text(`${'0'.repeat(64)} ${archive}\n`) },
86 + ],
87 + });
88 + const result = await rig.run('composio upgrade');
89 + expect(result.exitCode).toBe(1);
90 + const stderr = clean(result.stderr);
91 + expect(stderr).toContain(`Checksum mismatch for ${archive}`);
92 + expect(stderr).toContain(sha256(NOT_A_ZIP));
93 + expect(stderr).not.toContain('Failed to extract');
94 + await expectUntouched(result);
95 + },
96 + TEST_TIMEOUT_MS
97 + );
98 +
99 + it(
100 + 'a corrupt archive with a matching checksum: fails at extraction',
101 + async () => {
102 + publish({
103 + tag: TAG,
104 + prerelease: false,
105 + assets: [
106 + { name: archive, read: () => NOT_A_ZIP },
107 + { name: 'checksums.txt', read: () => text(`${sha256(NOT_A_ZIP)} ${archive}\n`) },
108 + ],
109 + });
110 + const result = await rig.run('composio upgrade');
111 + expect(result.exitCode).toBe(1);
112 + expect(clean(result.stderr)).toContain('Failed to extract zip archive');
113 + await expectUntouched(result);
114 + },
115 + TEST_TIMEOUT_MS
116 + );
117 +
118 + it(
119 + 'a download cut off mid-body: fails without a second attempt',
120 + async () => {
121 + publish({ tag: TAG, prerelease: false, assets: [{ name: archive, read: () => NOT_A_ZIP }] });
122 + rig.fake.faults.inject({
123 + route: 'github.releases.download',
124 + fault: { kind: 'reset-mid-body' },
125 + });
126 + const result = await rig.run('composio upgrade');
127 + expect(result.exitCode).toBe(1);
128 + expect(clean(result.stderr)).toContain('Failed to read downloaded binary');
129 + expect(downloads(result)).toHaveLength(1);
130 + await expectUntouched(result);
131 + },
132 + TEST_TIMEOUT_MS
133 + );
134 +
135 + it(
136 + 'a pinned version that was never released: names the tag it looked for',
137 + async () => {
138 + publish({ tag: TAG, prerelease: false, assets: [{ name: archive, read: () => NOT_A_ZIP }] });
139 + const result = await rig.run('composio upgrade 9.9.9');
140 + expect(result.exitCode).toBe(1);
141 + expect(clean(result.stderr)).toContain(
142 + 'Failed to fetch tags/@composio/cli@9.9.9 release from GitHub'
143 + );
144 + expect(downloads(result)).toEqual([]);
145 + await expectUntouched(result);
146 + },
147 + TEST_TIMEOUT_MS
148 + );
149 +
150 + it(
151 + 'a pinned version: downloads that release, not the latest one',
152 + async () => {
153 + const older = '@composio/cli@1.5.0';
154 + publish(
155 + { tag: TAG, prerelease: false, assets: [{ name: archive, read: () => NOT_A_ZIP }] },
156 + { tag: older, prerelease: false, assets: [{ name: archive, read: () => NOT_A_ZIP }] }
157 + );
158 + const result = await rig.run('composio upgrade 1.5.0');
159 + // The archive is not a real build, so the upgrade still fails; which release it fetched is the point.
160 + expect(result.exitCode).toBe(1);
161 + expect(downloads(result).map(entry => decodeURIComponent(entry.path))).toEqual([
162 + `/ComposioHQ/composio/releases/download/${older}/${archive}`,
163 + ]);
164 + await expectUntouched(result);
165 + },
166 + TEST_TIMEOUT_MS
167 + );
168 +
169 + it(
170 + 'already on the latest release: succeeds without downloading',
171 + async () => {
172 + publish({ tag: TAG, prerelease: false, assets: [{ name: archive, read: () => NOT_A_ZIP }] });
173 + // A packaged install records its release next to the executable.
174 + await rig.journey.writeFile('/usr/local/bin/release-tag.txt', `${TAG}\n`);
175 + const result = await rig.run('composio upgrade');
176 + await rig.run('rm -f /usr/local/bin/release-tag.txt');
177 +
178 + expect(result.exitCode).toBe(0);
179 + expect(result.stdout).toBe('');
180 + expect(downloads(result)).toEqual([]);
181 + expect(await fingerprint()).toBe(fingerprintBefore);
182 + },
183 + TEST_TIMEOUT_MS
184 + );
185 +});