@devries-cameron20 / composio Mirror · read-only
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 21, 04:45 UTC

next
md 153 lines 28.2 KB
Rendered Raw
1 # CLI operation suites and fault catalogue
2
3 Deterministic end-to-end tests of the operations an agent drives the `composio` binary through: login and session, `whoami`/`logout`, `search`, `execute`, `link`, `connections`, `run`, `upgrade`, and their `composio dev` counterparts. No model is involved. The binary runs in a journey container (`startJourneyContainer` from `@e2e-tests/utils`) and talks only to the fake backend in [`../fake-backend`](../fake-backend/README.md).
4
5 Every test asserts three things where they apply: what the command returned (exit code, stdout, stderr), what the fake backend received (route, key class, project, user), and what was left on disk.
6
7 ```bash
8 pnpm --filter @e2e-tests/cli-operations test:e2e:cli # all files
9 cd ts/e2e-tests/cli/operations && bun test identity.test.ts # one file
10 ```
11
12 A failing run leaves `DEBUG.<name>.log` files here with each command and its output.
13
14 ## Reading the catalogue
15
16 The catalogue starts from one question: what can go wrong for a CLI that an agent runs unattended, where nobody reads stderr unless the exit code says so, and a wrong success is worse than a failure? Each row names a fault, where it is injected, and what must hold.
17
18 - **exit**: the CLI's exit code. **stdout**: `data` means pure machine-readable output and nothing else; `empty` means zero bytes.
19 - **Status**: `ok` is implemented and passing. `expected failure` is implemented with `it.failing` because the CLI does not do what the row says today; the test turns red when the CLI is fixed, and the marker is then removed. The `it.failing` test asserts only the defect; where any other failure (a crash, a timeout) could also make it fail, a passing `today:` test pins what the CLI does instead, so the expected failure cannot hide a different one. `known gap` is behaviour that hinders an agent but that nothing in the repo declares wrong; a passing test pins what the CLI does today, so a change is noticed, and the row says what would be better.
20
21 ### Identity: which project, which user (`identity.test.ts`)
22
23 | Fault or question | Injected at | Must hold | Status |
24 | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
25 | Root `execute` run from an initialized developer directory | cwd | One Dashboard execution, consumer project, consumer user, user key; no request names the developer project | ok |
26 | Dev `playground-execute` | cwd | One backend session execution, developer project, global test user; nothing goes to the Dashboard or the consumer project | ok |
27 | Dev `--user-id` | flag | The session acts as that user | ok |
28 | Dev `--project-name` | flag | The session is on the named developer project | ok |
29 | Project API key minted by `dev init` | journal | Never sent by any command; every request carries the user key | ok |
30 | Account linked on one tree, used on the other | world | Not connected on the other tree, both directions | ok |
31 | Root `execute` without a connection | world | exit 1; the error names `composio link <toolkit>` | ok |
32 | Dev `playground-execute` without a connection | world | exit 1; the error names `composio dev connected-accounts link <toolkit>`, not the root-tree `composio link` | ok |
33 | `dev auth-configs list` in a directory set up for another project | `.composio/project.json` | The request is scoped to that project | ok |
34 | Command from the other tree (`dev execute`, `dev link`, root `playground-execute`, root `auth-configs`, root `--user-id`) | argv | exit 1; stdout empty; stderr names the unknown subcommand or flag and lists the real commands; no request | ok |
35 | `composio dev link` | argv | stderr names `composio dev connected-accounts link` | known gap: the `dev` help lists `connected-accounts`, but no equivalent command is named |
36 | `composio dev link --help` | argv | exit 1 | known gap: prints the `dev` help and exits 0 |
37 | `composio link --help`, `composio dev connected-accounts link --help` | argv | exit 0; every example names the page's own command; no request | ok |
38 | Missing acting user, root tree | resolve returns an empty consumer user id | exit 1; `Missing consumer user id. Run composio login and try again.`; no execution | ok |
39 | Missing acting user, dev tree, no project for the directory | cwd | exit 1; `No developer project configured for this directory. Run composio dev init.`; no request | ok |
40 | Missing acting user, dev tree, `--project-name`, no test user | stored login | exit 1; `Missing user id. Provide --user-id, or run composio login …`; no execution | ok |
41
42 ### Login and session (`login-session.test.ts`)
43
44 "Session" here is the CLI login session (created, polled, linked) and the session artifact directory.
45
46 | Fault or question | Injected at | Must hold | Status |
47 | ------------------------------------ | ----------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------ |
48 | First run, `whoami` | empty home | exit 1; stdout empty; names `composio login`; no request | ok |
49 | Headless `login` | no terminal | exit 0; one session created; URL with the session key and `composio login --poll` on stdout; pending session cached; nothing stored | ok |
50 | `login --poll`, session never linked | world | Keeps polling (at least two reads in 8 s); nothing stored | ok |
51 | `login --poll`, session linked | world | exit 0; JSON summary; key, backend URLs, org and global test user stored; pending file removed | ok |
52 | `whoami` after login | | exit 0; JSON on stdout; stderr empty; only user-key requests | ok |
53 | `login` when already logged in | | exit 0; no request; stored login unchanged | ok |
54 | `logout` | | Key removed; `whoami` then exits 1 without a request | ok |
55 | `login --key`, session not linked | world | exit 1; `Session not yet linked`; nothing stored | ok |
56 | `login --key`, linked | world | exit 0; login stored | ok |
57 | `login --key`, unknown session | 404 | exit 1; names `composio login --no-wait`; nothing stored | ok |
58 | `login --user-api-key`, rejected key | 401 | exit 1; stdout empty; key not written anywhere | ok |
59 | Same, as an expected failure | 401 | No defect report | ok |
60 | `artifacts cwd` | | exit 0; one absolute path; no login or request needed; stable per directory; follows `COMPOSIO_SESSION_DIR` | ok |
61
62 ### Network (`network-faults.test.ts`)
63
64 | Fault | Injected at | Must hold | Status |
65 | ------------------------------------------------------------------------------ | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ |
66 | Connection refused | `COMPOSIO_BASE_URL` to a closed port | exit 1; stdout empty; `ECONNREFUSED` on stderr | ok |
67 | HTTP 500, 429, HTML 502 page, reset before response, on every attempt | `search` | exit 1 (75 for the 429, with how to retry); stdout empty; cause on stderr; three attempts | ok |
68 | Reset in the middle of the body | `search` | exit 1; stdout empty; one attempt | ok |
69 | Body that is not JSON | `search` | exit 1; stdout empty; says the response shape is unexpected and names `composio upgrade` | ok |
70 | One 500, one 429 with Retry-After, one reset, then healthy | `search` | exit 0; stderr empty; normal result; two attempts | ok |
71 | Slow response (4 s) | `search` | exit 0; one attempt | ok |
72 | Valid JSON of another shape; empty 200 body | `search` | exit 1 with an unexpected-response message | ok |
73 | HTTP 500, 429, HTML page, non-JSON body, reset before response, reset mid-body | tool execution, Dashboard transport and backend transport | exit 1 (75 for the 429); **exactly one execution request**; stdout is JSON with `successful: false`, an error and `retry` (`safe` for the 429, else `check-first`); no `data` | ok |
74
75 Observed and left as is: on the backend transport an HTML error page is copied whole into the `error` field of the result.
76
77 ### Auth (`auth-faults.test.ts`)
78
79 | Fault | Injected at | Must hold | Status |
80 | -------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ |
81 | No key | empty home, seven root and dev commands | exit 1; stdout empty; names `composio login`; no request | ok |
82 | Key only in `COMPOSIO_USER_API_KEY` | env | exit 1; names `composio login` or `composio orgs switch`; no request | ok |
83 | Stored key rejected (401 everywhere) | all backend routes | exit 1; rejection visible; nothing but 401s; stored login untouched | ok |
84 | Same, recovery command | | Names `composio login` | ok |
85 | Same, `whoami` | | exit 1; stdout empty | ok |
86 | Key sent to a backend that did not issue it | `COMPOSIO_BASE_URL` to a second fake | Warning on stderr naming both backends and `composio logout && composio login`; stdout free of it; the other backend rejects the key; nothing reaches the issuer; stored login untouched | ok |
87 | Project key (`ak_`) where a user key belongs | `COMPOSIO_USER_API_KEY` | Arrives classified as a project key and is rejected | ok |
88
89 ### Filesystem (`filesystem-faults.test.ts`)
90
91 | Fault | Injected at | Must hold | Status |
92 | ------------------------------------------------------------------------- | ------------------------------ | ----------------------------------------------------------------------------- | ------------------------------------------- |
93 | Stored login truncated, wrong JSON type, empty, missing | `user_data.json` | exit 1; names `composio login`; no request; a new login repairs it | ok |
94 | Corrupt `update-check.json`, `config.json`, `tool-permissions-cache.json` | `~/.composio` | Command still succeeds | ok |
95 | Corrupt cached tool schema | `tool_definitions/<SLUG>.json` | Fetched again | expected failure: fatal `is not valid JSON` |
96 | Same | | At least names the file; deleting it recovers | ok |
97 | Read-only home (ordinary user) | `chmod -R a-w ~/.composio` | `whoami`, `search`, `execute` succeed with data on stdout; the tool runs once | ok |
98 | Read-only home, `logout` | | exit 1; names the file; login untouched | ok |
99 | Eight concurrent `search` on a cold cache | background jobs | All exit 0; no stderr; cache file valid JSON; login intact | ok |
100 | Four concurrent `execute` | | All exit 0; four executions, one each | ok |
101 | Four concurrent `login` | | All exit 0; one complete stored login; no temporary files left | ok |
102
103 ### Process and streams (`process-streams.test.ts`)
104
105 | Fault | Injected at | Must hold | Status |
106 | -------------------------------------------------------- | ------------------------------ | -------------------------------------------------------------------------------- | ------------------------------------------------- |
107 | SIGINT / SIGTERM while `link` waits | `kill` once the URL is printed | exit 130 / 143; stdout empty; the link was requested once | ok |
108 | SIGINT while `run` executes a script | `kill` once the script started | exit 130; stdout empty, also after the script would have finished | ok |
109 | `link` off a terminal, nobody authorizes | no terminal | The authorization URL is printed before the wait | ok: on stderr, with a hint that names `--no-wait` |
110 | `link --no-wait` | | exit 0; JSON with the URL; no status polling | ok |
111 | Reader closes stdout before the CLI writes (`… \| true`) | pipe | CLI exits 0; no stderr | ok |
112 | `-d -` with JSON on stdin | pipe | Arguments read from stdin | ok |
113 | `-d -` with nothing on stdin | `/dev/null` | exit 1 at once; stdout empty; no execution | ok |
114 | No terminal on any stream | | stdout is the JSON; stderr empty | ok |
115 | stdout piped, stderr a terminal | `script` | JSON in the pipe; human rendering on the terminal, not a second copy of the data | ok |
116 | stdout a terminal, stderr redirected | `script` | No data on the terminal | ok |
117 | Update notice pending | seeded `update-check.json` | Never on stdout; shown on a terminal stderr | ok |
118 | Plugin hint under an agent host | `CLAUDECODE=1` | On stderr; never on stdout | ok |
119 | `search --human` off a terminal | no terminal | Results are printed | expected failure: prints nothing, exits 0 |
120
121 ### Upgrade (`upgrade.test.ts`, and `../upgrade`)
122
123 Every row: a failed upgrade leaves the executable's inode, size and mtime unchanged, and `composio version` still runs.
124
125 | Fault | Injected at | Must hold | Status |
126 | -------------------------------------------- | ---------------------------------- | ---------------------------------------------------------------------------------- | ----------------- |
127 | No archive for this platform | release assets | exit 1; says so; no download | ok |
128 | Checksum mismatch | `checksums.txt` | exit 1; expected and actual digests; no extraction | ok |
129 | Corrupt archive with matching checksum | archive bytes | exit 1; `Failed to extract zip archive` | ok |
130 | Download cut off mid-body | download route | exit 1; one download attempt | ok |
131 | Pinned version never released | release-by-tag 404 | exit 1; names the tag; no download | ok |
132 | Pinned version | two releases | Downloads the pinned release's archive, not the latest | ok |
133 | Already on the latest release | `release-tag.txt` | exit 0; no download; executable untouched | ok |
134 | No write permission on the install directory | read-only directory, ordinary user | exit 1; names the file; executable untouched and runnable; no temporary files left | ok (`../upgrade`) |
135
136 Install faults live in [`../install`](../install/README.md), which runs in its own workflow against a packaged release: checksum mismatch, corrupt archive, missing platform archive, and a download cut off partway.
137
138 ### Tool loop (`tool-loop.test.ts`)
139
140 Not faults: the search, execute, link, execute loop and the commands around it (`connections list`, `--get-schema`, `--dry-run`, input validation, unknown slug, `run` with `execute()` and `search()`), each checked against the journal.
141
142 ## Not modelled
143
144 Listed so nobody reads their absence as coverage.
145
146 - **Network**: packet loss, duplicated or reordered responses, partitions that heal mid-request, DNS failure, TLS errors other than an untrusted certificate, proxies (`HTTP_PROXY`), IPv6, redirects, timeouts the CLI gives up on (the Dashboard deadline is 15 minutes), responses larger than memory.
147 - **Backend state**: an account that expires between the connection check and the execution, a tool that ran although its response was lost (the fake cannot act without answering), rate limits that depend on history, pagination past one page, several accounts per toolkit and `--account` selection, enhanced-controls permission prompts.
148 - **Auth**: key rotation mid-command, OS keyring storage (the container has no Secret Service, so the key is stored in `user_data.json`), agent keys beyond what `../agent-signin` covers, org switching.
149 - **Filesystem**: full disk, a home on a network or case-insensitive filesystem, symlinked `~/.composio`, files written by a newer or older CLI version, a stale cache that is valid JSON but out of date, two CLIs of different versions sharing a home.
150 - **Process**: SIGHUP, SIGKILL, a second signal during shutdown, being killed between writing a temporary file and renaming it, stdin as a terminal with a person typing, terminal resize, non-UTF-8 locales.
151 - **Upgrade and install**: a successful upgrade through the download path (the archive here is never a real build), upgrading while another invocation runs, a partially replaced companion module, macOS and Windows.
152 - **Platforms**: everything here runs the Linux binary of the host's architecture as root in Debian; macOS, x64-on-arm, musl, and non-root homes (apart from the read-only rows) are not exercised.
153 - **Commands**: `proxy`, `listen`, triggers, logs, `generate`, `setup`, `orgs`, and `dev` commands beyond `init`, `auth-configs list`, `playground-execute`, and `connected-accounts link`/`list`.