1
+---
2
+description: Write end-to-end tests for CLI commands using the Docker-based test framework in ts/e2e-tests/cli/.
3
+---
4
+
5
# CLI E2E Test Development
6
7
Write, expand, and maintain end-to-end tests for CLI commands in `ts/e2e-tests/cli/`.
13
- Debugging a failing CLI e2e test
14
- Reviewing whether a CLI command's output contract is properly tested
15
16
+For CLI **design** (arguments, flags, help text, UX), see the `create-cli` skill.
17
+For CLI **implementation** (Effect patterns, services, command registration), see the `implement-cli-command` skill.
18
+
19
## Architecture
20
21
Each CLI e2e test runs the compiled `composio` binary inside a **scratch Debian Docker container**. The binary is self-contained (built via `bun build --compile`) — no Node, Bun, or pnpm exists in the runtime image.
32
33
### What "piped mode" means for tests
34
28
-Inside Docker `sh -c`, the composio binary's stdout is never a TTY. This means `ui.output()` writes to stdout and all Clack decoration is suppressed. Both test groups verify piped-mode behavior:
29
-
30
-| Test Group | What It Verifies |
31
-|---|---|
32
-| **Command execution** (`composio version`) | stdout contains the data, stderr is empty |
33
-| **Stdout redirection** (`composio version > out.txt`) | Shell-level redirect captures data into a file; Docker stdout and stderr are both empty |
35
+Inside Docker, the composio binary's stdout is never a TTY. This triggers the CLI's piped-mode behavior (see `ts/packages/cli/AGENTS.md` § "Output Conventions"):
36
+- `ui.output()` writes to stdout
37
+- All Clack decoration is suppressed
38
+- stderr is empty for successful commands
39
40
## File Structure
41
73
}
74
```
75
71
-## Test Patterns
72
-
73
-Every test file starts with the same structure:
76
+## Choosing a Test Pattern
77
75
-```typescript
76
-import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
77
-import { TIMEOUTS } from '@e2e-tests/utils/const';
78
-import { describe, it, expect, beforeAll } from 'bun:test';
78
+Use this decision tree to select the right pattern:
79
80
-e2e(import.meta.url, {
81
- versions: { cli: ['current'] },
82
- // env: { ... }, // Only if needed
83
- defineTests: ({ runCmd }) => {
84
- // Tests here
85
- },
86
-});
80
```
81
+Does the command need env vars (e.g., COMPOSIO_API_KEY)?
82
+├─ No → Is the output deterministic (exact value known at test time)?
83
+│ ├─ Yes → Pattern A (simple command, exact assertions)
84
+│ └─ No → Pattern D (fuzzy assertions: toContain, toBeGreaterThan)
85
+└─ Yes → Is the output deterministic given the env?
86
+ ├─ Yes → Pattern B (env vars + exact assertions)
87
+ └─ No → Pattern D (env vars + fuzzy assertions)
88
+
89
+Are you testing an error case (missing args, invalid input)?
90
+└─ Yes → Pattern C (non-zero exit code, stderr non-empty)
91
+
92
+Does the command perform an action with no machine-readable output?
93
+└─ Yes → Pattern E (exitCode=0, stdout empty, no redirect test)
94
+```
95
+
96
+## Test Patterns
97
89
-### Pattern A: Simple Command, No Env Vars
98
+### Pattern A: Simple Command, No Env Vars (Canonical)
99
91
-For commands that produce deterministic output without needing authentication.
100
+For commands that produce deterministic output without needing authentication. This is the base pattern — all other patterns are variations of this.
101
102
**Reference**: `ts/e2e-tests/cli/version/e2e.test.ts`
103
165
});
166
```
167
168
+**Structure summary:**
169
+- Two test groups: **command execution** (stdout has data, stderr empty) + **stdout redirection** (file has data, Docker stdout/stderr both empty)
170
+- Use `sanitizeOutput()` on stdout and file contents before assertions
171
+- Use `TIMEOUTS.FIXTURE` for the `beforeAll` that runs Docker commands
172
+
173
### Pattern B: Command Requiring Env Vars
174
161
-For commands that need authentication or configuration from the host environment.
175
+**Same as Pattern A**, with three additions:
176
177
**Reference**: `ts/e2e-tests/cli/whoami/e2e.test.ts`
178
165
-Three additions compared to Pattern A:
166
-
179
1. **Type augmentation** for compile-time safety on `Bun.env`:
180
```typescript
181
declare module 'bun' {
199
200
The rest follows the same two-group structure (command execution + stdout redirection).
201
190
-**Full example:**
191
-
192
-```typescript
193
-/**
194
- * CLI whoami command e2e test
195
- *
196
- * Verifies that the compiled composio CLI prints the API key in a scratch container.
197
- */
198
-
199
-import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
200
-import { TIMEOUTS } from '@e2e-tests/utils/const';
201
-import { describe, it, expect, beforeAll } from 'bun:test';
202
-
203
-declare module 'bun' {
204
- interface Env {
205
- COMPOSIO_API_KEY: string;
206
- }
207
-}
208
-
209
-e2e(import.meta.url, {
210
- versions: {
211
- cli: ['current'],
212
- },
213
- env: {
214
- COMPOSIO_API_KEY: Bun.env.COMPOSIO_API_KEY,
215
- },
216
- defineTests: ({ runCmd }) => {
217
- const expectedApiKey = Bun.env.COMPOSIO_API_KEY.trim();
218
- let whoamiResult: E2ETestResult;
219
- let redirectedResult: E2ETestResultWithFiles<'out.txt'>;
220
-
221
- beforeAll(async () => {
222
- whoamiResult = await runCmd('composio whoami');
223
- redirectedResult = await runCmd({
224
- command: 'composio whoami > out.txt',
225
- files: ['out.txt'],
226
- });
227
- }, TIMEOUTS.FIXTURE);
228
-
229
- describe('composio whoami', () => {
230
- it('exits successfully', () => {
231
- expect(whoamiResult.exitCode).toBe(0);
232
- });
233
-
234
- it('stdout contains the API key', () => {
235
- expect(sanitizeOutput(whoamiResult.stdout)).toBe(expectedApiKey);
236
- });
237
-
238
- it('stderr is empty', () => {
239
- expect(whoamiResult.stderr).toBe('');
240
- });
241
- });
242
-
243
- describe('stdout redirection to out.txt', () => {
244
- it('exits successfully', () => {
245
- expect(redirectedResult.exitCode).toBe(0);
246
- });
247
-
248
- it('stdout is empty', () => {
249
- expect(redirectedResult.stdout).toBe('');
250
- });
251
-
252
- it('stderr is empty', () => {
253
- expect(redirectedResult.stderr).toBe('');
254
- });
255
-
256
- it('out.txt contains the API key', () => {
257
- expect(sanitizeOutput(redirectedResult.files['out.txt'])).toBe(expectedApiKey);
258
- });
259
- });
260
- },
261
-});
262
-```
263
-
202
### Pattern C: Error Case Testing
203
266
-For verifying that the CLI fails gracefully with correct exit codes and error messages.
204
+**Differences from Pattern A:**
205
+- No stdout redirection test needed — error cases don't produce data output
206
+- Assert non-zero exit code with `not.toBe(0)` (don't hardcode a specific error code)
207
+- Assert stderr is non-empty; optionally use `toContain()` for specific error message fragments
208
+- Import only `E2ETestResult`, not `E2ETestResultWithFiles`
209
210
```typescript
269
-/**
270
- * CLI tools-info error case e2e test
271
- *
272
- * Verifies that the CLI fails gracefully when required arguments are missing.
273
- */
274
-
211
import { e2e, sanitizeOutput, type E2ETestResult } from '@e2e-tests/utils';
212
import { TIMEOUTS } from '@e2e-tests/utils/const';
213
import { describe, it, expect, beforeAll } from 'bun:test';
214
215
e2e(import.meta.url, {
280
- versions: {
281
- cli: ['current'],
282
- },
216
+ versions: { cli: ['current'] },
217
defineTests: ({ runCmd }) => {
218
let missingArgResult: E2ETestResult;
219
234
});
235
```
236
303
-Notes:
304
-- No stdout redirection test needed — error cases don't produce data output.
305
-- Use `not.toBe(0)` for exit code rather than a specific number (the CLI may change error codes).
306
-- Assert stderr is non-empty; optionally use `toContain()` for specific error message fragments.
237
+Pattern C can be combined with A or B in the same test file to test both success and error paths.
238
239
### Pattern D: Multi-line / API-dependent Output
240
310
-For commands that call the Composio API and return dynamic, multi-line data.
311
-
312
-```typescript
313
-/**
314
- * CLI toolkits-list command e2e test
315
- *
316
- * Verifies that the CLI lists available toolkits from the API.
317
- */
318
-
319
-import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
320
-import { TIMEOUTS } from '@e2e-tests/utils/const';
321
-import { describe, it, expect, beforeAll } from 'bun:test';
322
-
323
-declare module 'bun' {
324
- interface Env {
325
- COMPOSIO_API_KEY: string;
326
- }
327
-}
328
-
329
-e2e(import.meta.url, {
330
- versions: {
331
- cli: ['current'],
332
- },
333
- env: {
334
- COMPOSIO_API_KEY: Bun.env.COMPOSIO_API_KEY,
335
- },
336
- defineTests: ({ runCmd }) => {
337
- let listResult: E2ETestResult;
338
- let redirectedResult: E2ETestResultWithFiles<'out.txt'>;
339
-
340
- beforeAll(async () => {
341
- listResult = await runCmd('composio toolkits list');
342
- redirectedResult = await runCmd({
343
- command: 'composio toolkits list > out.txt',
344
- files: ['out.txt'],
345
- });
346
- }, TIMEOUTS.FIXTURE);
347
-
348
- describe('composio toolkits list', () => {
349
- it('exits successfully', () => {
350
- expect(listResult.exitCode).toBe(0);
351
- });
352
-
353
- it('stdout is non-empty', () => {
354
- expect(sanitizeOutput(listResult.stdout).length).toBeGreaterThan(0);
355
- });
356
-
357
- it('stdout contains known toolkits', () => {
358
- const output = sanitizeOutput(listResult.stdout);
359
- // Use well-known toolkits that are always present
360
- expect(output).toContain('github');
361
- });
362
-
363
- it('stdout has multiple lines', () => {
364
- const lines = sanitizeOutput(listResult.stdout).split('\n').filter(Boolean);
365
- expect(lines.length).toBeGreaterThan(1);
366
- });
367
-
368
- it('stderr is empty', () => {
369
- expect(listResult.stderr).toBe('');
370
- });
371
- });
372
-
373
- describe('stdout redirection to out.txt', () => {
374
- it('exits successfully', () => {
375
- expect(redirectedResult.exitCode).toBe(0);
376
- });
241
+**Same structure as Pattern A or B** (depending on whether env vars are needed), with different assertion strategies:
242
378
- it('stdout is empty', () => {
379
- expect(redirectedResult.stdout).toBe('');
380
- });
381
-
382
- it('stderr is empty', () => {
383
- expect(redirectedResult.stderr).toBe('');
384
- });
385
-
386
- it('out.txt contains data', () => {
387
- expect(sanitizeOutput(redirectedResult.files['out.txt']).length).toBeGreaterThan(0);
388
- });
389
- });
390
- },
391
-});
392
-```
393
-
394
-Key rules for API-dependent output:
243
- **Never use exact-match assertions** (`toBe`) on API data — the data changes over time.
244
- Use `toContain()` for known, stable items (e.g., `github`, `gmail` are always present).
245
- Use `toBeGreaterThan()` for structural checks (line count, length).
246
- Use `toMatch()` with regex for format validation.
247
248
+```typescript
249
+// Instead of:
250
+expect(sanitizeOutput(listResult.stdout)).toBe(exactValue);
251
+
252
+// Use fuzzy assertions:
253
+expect(sanitizeOutput(listResult.stdout).length).toBeGreaterThan(0);
254
+expect(sanitizeOutput(listResult.stdout)).toContain('github');
255
+const lines = sanitizeOutput(listResult.stdout).split('\n').filter(Boolean);
256
+expect(lines.length).toBeGreaterThan(1);
257
+```
258
+
259
+The redirect test group is identical to Pattern A — assert the file contains data, Docker stdout/stderr are empty.
260
+
261
### Pattern E: Action Command, No Stdout Data
262
263
For commands that perform an action but produce no machine-readable output (e.g., `logout`, `upgrade`).
264
404
-```typescript
405
-/**
406
- * CLI logout command e2e test
407
- *
408
- * Verifies that the CLI logout command completes without error.
409
- */
410
-
411
-import { e2e, type E2ETestResult } from '@e2e-tests/utils';
412
-import { TIMEOUTS } from '@e2e-tests/utils/const';
413
-import { describe, it, expect, beforeAll } from 'bun:test';
265
+**Differences from Pattern A:**
266
+- No redirection test — there's no data to redirect
267
+- stdout should be literally empty (no `sanitizeOutput()` needed)
268
+- Import only `E2ETestResult`, not `E2ETestResultWithFiles`
269
270
+```typescript
271
e2e(import.meta.url, {
416
- versions: {
417
- cli: ['current'],
418
- },
272
+ versions: { cli: ['current'] },
273
defineTests: ({ runCmd }) => {
274
let logoutResult: E2ETestResult;
275
294
});
295
```
296
443
-Notes:
444
-- No redirection test needed — there's no data to redirect.
445
-- No `sanitizeOutput()` needed — stdout should be literally empty.
446
-- Import only `E2ETestResult`, not `E2ETestResultWithFiles`.
447
-
448
-## Output Contract Rules
449
-
450
-Enforce these in every CLI e2e test:
451
-
452
-1. **Data commands** produce machine-readable output on stdout (via `ui.output()`).
453
-2. **Action commands** produce no stdout data — stdout is always empty.
454
-3. **stderr is always empty** for successful commands (non-TTY suppresses all Clack decoration).
455
-4. **Redirected output** (`> out.txt`) captures the same data that direct stdout contains; Docker stdout and stderr both become empty.
456
-
297
## Commands NOT Testable with This Framework
298
299
Do not attempt to write e2e tests for:
397
398
## API Reference
399
560
-### `e2e(importMetaUrl, config)`
561
-
562
-Entry point. Pass `import.meta.url` as the first argument — the framework infers `cwd` and `suiteName` from the file path.
563
-
564
-### `E2EConfig`
565
-
566
-```typescript
567
-{
568
- versions: { cli: ['current'] }, // Always use ['current'] for CLI tests
569
- env?: Record<string, string | undefined>, // Env vars to pass to Docker
570
- defineTests: (ctx: DefineTestsContext) => void,
571
-}
572
-```
573
-
400
### `runCmd` (two overloads)
401
402
```typescript
411
// Access: result.files['out.txt']
412
```
413
588
-### `sanitizeOutput(output)`
589
-
590
-Strips ANSI escape codes, normalizes `\r\n` to `\n`, trims whitespace. Use on stdout and file contents before assertions.
591
-
414
### `TIMEOUTS`
415
416
+Defined in `ts/e2e-tests/_utils/src/const.ts`:
417
+
418
```typescript
419
TIMEOUTS.DEFAULT // 5_000ms — individual test timeout
420
TIMEOUTS.FIXTURE // 120_000ms — beforeAll timeout (Docker startup + command execution)
431
| `ts/e2e-tests/cli/version/e2e.test.ts` | Pattern A reference |
432
| `ts/e2e-tests/cli/whoami/e2e.test.ts` | Pattern B reference |
433
| `ts/e2e-tests/_utils/src/types.ts` | `E2EConfig`, `E2ETestResult`, `DefineTestsContext` types |
610
-| `ts/e2e-tests/_utils/src/e2e.ts` | `e2e()` entry point |
434
| `ts/e2e-tests/_utils/src/const.ts` | `TIMEOUTS`, `WELL_KNOWN_ENV_VARS` |
435
| `ts/e2e-tests/_utils/src/sanitize.ts` | `sanitizeOutput()` |
436
| `ts/e2e-tests/_utils/Dockerfile.cli` | Docker image definition |
437
| `ts/e2e-tests/cli/README.md` | Test suites table |
438
| `CLI.md` | Planned CLI commands |
616
-| `ts/packages/cli/CLAUDE.md` | CLI architecture and output conventions |
439
+| `ts/packages/cli/AGENTS.md` | CLI architecture and output conventions |