Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
feat(ai): add "create-cli-e2e" skill
jkomyno committed
Feb 18, 2026 at 16:33 UTC
6c4e81405933f07671e94cf11ff41344f2550db8
1 file changed
+616
.claude/skills/create-cli-e2e/SKILL.md
new
+616
@@ -0,0 +1,616 @@
1
+# CLI E2E Test Development
2
+
3
+Write, expand, and maintain end-to-end tests for CLI commands in `ts/e2e-tests/cli/`.
4
+
5
+## When to Use
6
+
7
+- Adding a new e2e test suite for a CLI command
8
+- Modifying or extending an existing CLI e2e test
9
+- Debugging a failing CLI e2e test
10
+- Reviewing whether a CLI command's output contract is properly tested
11
+
12
+## Architecture
13
+
14
+Each CLI e2e test runs the compiled `composio` binary inside a **scratch Debian Docker container**. The binary is self-contained (built via `bun build --compile`) — no Node, Bun, or pnpm exists in the runtime image.
15
+
16
+Key properties:
17
+
18
+- Each test suite = a directory under `ts/e2e-tests/cli/<suite-name>/`
19
+- Use `runCmd` only. Never use `runFixture` (throws an error for CLI tests). Never set `usesFixtures`.
20
+- **Each `runCmd` call creates a fresh container.** No state persists between calls.
21
+- Commands run inside `sh -c '...'` — POSIX shell only, no bash-isms.
22
+- Containers have network access — API-calling commands work.
23
+- `HOME=/tmp`, cache dir is `/tmp/.composio/` — auth passes via env vars only.
24
+- `process.stdout.isTTY` is always `false` inside Docker — the CLI always runs in piped mode.
25
+
26
+### What "piped mode" means for tests
27
+
28
+Inside Docker `sh -c`, the composio binary's stdout is never a TTY. This means `ui.output()` writes to stdout and all Clack decoration is suppressed. Both test groups verify piped-mode behavior:
29
+
30
+| Test Group | What It Verifies |
31
+|---|---|
32
+| **Command execution** (`composio version`) | stdout contains the data, stderr is empty |
33
+| **Stdout redirection** (`composio version > out.txt`) | Shell-level redirect captures data into a file; Docker stdout and stderr are both empty |
34
+
35
+## File Structure
36
+
37
+For a new test suite `<suite-name>`, create 2 files:
38
+
39
+```
40
+ts/e2e-tests/cli/<suite-name>/
41
+├── e2e.test.ts # Test file
42
+└── package.json # Package manifest
43
+```
44
+
45
+### Naming Conventions
46
+
47
+- **Directory**: hyphen-separated lowercase matching the command structure
48
+ - `version`, `whoami`, `toolkits-list`, `tools-info`, `auth-configs-list`, `connected-accounts-link`
49
+- **Package name**: `@e2e-tests/cli-<suite-name>`
50
+ - `@e2e-tests/cli-version`, `@e2e-tests/cli-toolkits-list`
51
+
52
+### package.json Template
53
+
54
+```json
55
+{
56
+ "name": "@e2e-tests/cli-<suite-name>",
57
+ "version": "0.0.0",
58
+ "private": true,
59
+ "type": "module",
60
+ "scripts": {
61
+ "typecheck": "tsc --noEmit",
62
+ "test:e2e": "bun test e2e.test.ts",
63
+ "test:e2e:cli": "bun test e2e.test.ts"
64
+ },
65
+ "devDependencies": {
66
+ "@e2e-tests/utils": "workspace:*"
67
+ }
68
+}
69
+```
70
+
71
+## Test Patterns
72
+
73
+Every test file starts with the same structure:
74
+
75
+```typescript
76
+import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
77
+import { TIMEOUTS } from '@e2e-tests/utils/const';
78
+import { describe, it, expect, beforeAll } from 'bun:test';
79
+
80
+e2e(import.meta.url, {
81
+ versions: { cli: ['current'] },
82
+ // env: { ... }, // Only if needed
83
+ defineTests: ({ runCmd }) => {
84
+ // Tests here
85
+ },
86
+});
87
+```
88
+
89
+### Pattern A: Simple Command, No Env Vars
90
+
91
+For commands that produce deterministic output without needing authentication.
92
+
93
+**Reference**: `ts/e2e-tests/cli/version/e2e.test.ts`
94
+
95
+```typescript
96
+/**
97
+ * CLI version command e2e test
98
+ *
99
+ * Verifies that the compiled composio CLI behaves correctly in a scratch container.
100
+ */
101
+
102
+import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
103
+import { TIMEOUTS } from '@e2e-tests/utils/const';
104
+import { describe, it, expect, beforeAll } from 'bun:test';
105
+import cliPkg from '../../../packages/cli/package.json' with { type: 'json' };
106
+
107
+e2e(import.meta.url, {
108
+ versions: {
109
+ cli: ['current'],
110
+ },
111
+ defineTests: ({ runCmd }) => {
112
+ const expectedVersion = String(cliPkg.version ?? '').trim();
113
+ let versionResult: E2ETestResult;
114
+ let redirectedResult: E2ETestResultWithFiles<'out.txt'>;
115
+
116
+ beforeAll(async () => {
117
+ versionResult = await runCmd('composio version');
118
+ redirectedResult = await runCmd({
119
+ command: 'composio version > out.txt',
120
+ files: ['out.txt'],
121
+ });
122
+ }, TIMEOUTS.FIXTURE);
123
+
124
+ describe('composio version', () => {
125
+ it('exits successfully', () => {
126
+ expect(versionResult.exitCode).toBe(0);
127
+ });
128
+
129
+ it('stdout matches snapshot', () => {
130
+ expect(sanitizeOutput(versionResult.stdout)).toBe(expectedVersion);
131
+ });
132
+
133
+ it('stderr matches snapshot', () => {
134
+ expect(versionResult.stderr).toBe('');
135
+ });
136
+ });
137
+
138
+ describe('stdout redirection to out.txt', () => {
139
+ it('exits successfully', () => {
140
+ expect(redirectedResult.exitCode).toBe(0);
141
+ });
142
+
143
+ it('stdout is empty', () => {
144
+ expect(redirectedResult.stdout).toBe('');
145
+ });
146
+
147
+ it('stderr is empty', () => {
148
+ expect(redirectedResult.stderr).toBe('');
149
+ });
150
+
151
+ it('out.txt matches snapshot', () => {
152
+ expect(sanitizeOutput(redirectedResult.files['out.txt'])).toBe(expectedVersion);
153
+ });
154
+ });
155
+ },
156
+});
157
+```
158
+
159
+### Pattern B: Command Requiring Env Vars
160
+
161
+For commands that need authentication or configuration from the host environment.
162
+
163
+**Reference**: `ts/e2e-tests/cli/whoami/e2e.test.ts`
164
+
165
+Three additions compared to Pattern A:
166
+
167
+1. **Type augmentation** for compile-time safety on `Bun.env`:
168
+ ```typescript
169
+ declare module 'bun' {
170
+ interface Env {
171
+ COMPOSIO_API_KEY: string;
172
+ }
173
+ }
174
+ ```
175
+
176
+2. **Pass env vars** in the config:
177
+ ```typescript
178
+ env: {
179
+ COMPOSIO_API_KEY: Bun.env.COMPOSIO_API_KEY,
180
+ },
181
+ ```
182
+
183
+3. **Derive expected values** from the env:
184
+ ```typescript
185
+ const expectedApiKey = Bun.env.COMPOSIO_API_KEY.trim();
186
+ ```
187
+
188
+The rest follows the same two-group structure (command execution + stdout redirection).
189
+
190
+**Full example:**
191
+
192
+```typescript
193
+/**
194
+ * CLI whoami command e2e test
195
+ *
196
+ * Verifies that the compiled composio CLI prints the API key in a scratch container.
197
+ */
198
+
199
+import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
200
+import { TIMEOUTS } from '@e2e-tests/utils/const';
201
+import { describe, it, expect, beforeAll } from 'bun:test';
202
+
203
+declare module 'bun' {
204
+ interface Env {
205
+ COMPOSIO_API_KEY: string;
206
+ }
207
+}
208
+
209
+e2e(import.meta.url, {
210
+ versions: {
211
+ cli: ['current'],
212
+ },
213
+ env: {
214
+ COMPOSIO_API_KEY: Bun.env.COMPOSIO_API_KEY,
215
+ },
216
+ defineTests: ({ runCmd }) => {
217
+ const expectedApiKey = Bun.env.COMPOSIO_API_KEY.trim();
218
+ let whoamiResult: E2ETestResult;
219
+ let redirectedResult: E2ETestResultWithFiles<'out.txt'>;
220
+
221
+ beforeAll(async () => {
222
+ whoamiResult = await runCmd('composio whoami');
223
+ redirectedResult = await runCmd({
224
+ command: 'composio whoami > out.txt',
225
+ files: ['out.txt'],
226
+ });
227
+ }, TIMEOUTS.FIXTURE);
228
+
229
+ describe('composio whoami', () => {
230
+ it('exits successfully', () => {
231
+ expect(whoamiResult.exitCode).toBe(0);
232
+ });
233
+
234
+ it('stdout contains the API key', () => {
235
+ expect(sanitizeOutput(whoamiResult.stdout)).toBe(expectedApiKey);
236
+ });
237
+
238
+ it('stderr is empty', () => {
239
+ expect(whoamiResult.stderr).toBe('');
240
+ });
241
+ });
242
+
243
+ describe('stdout redirection to out.txt', () => {
244
+ it('exits successfully', () => {
245
+ expect(redirectedResult.exitCode).toBe(0);
246
+ });
247
+
248
+ it('stdout is empty', () => {
249
+ expect(redirectedResult.stdout).toBe('');
250
+ });
251
+
252
+ it('stderr is empty', () => {
253
+ expect(redirectedResult.stderr).toBe('');
254
+ });
255
+
256
+ it('out.txt contains the API key', () => {
257
+ expect(sanitizeOutput(redirectedResult.files['out.txt'])).toBe(expectedApiKey);
258
+ });
259
+ });
260
+ },
261
+});
262
+```
263
+
264
+### Pattern C: Error Case Testing
265
+
266
+For verifying that the CLI fails gracefully with correct exit codes and error messages.
267
+
268
+```typescript
269
+/**
270
+ * CLI tools-info error case e2e test
271
+ *
272
+ * Verifies that the CLI fails gracefully when required arguments are missing.
273
+ */
274
+
275
+import { e2e, sanitizeOutput, type E2ETestResult } from '@e2e-tests/utils';
276
+import { TIMEOUTS } from '@e2e-tests/utils/const';
277
+import { describe, it, expect, beforeAll } from 'bun:test';
278
+
279
+e2e(import.meta.url, {
280
+ versions: {
281
+ cli: ['current'],
282
+ },
283
+ defineTests: ({ runCmd }) => {
284
+ let missingArgResult: E2ETestResult;
285
+
286
+ beforeAll(async () => {
287
+ missingArgResult = await runCmd('composio tools info');
288
+ }, TIMEOUTS.FIXTURE);
289
+
290
+ describe('composio tools info (missing argument)', () => {
291
+ it('exits with non-zero code', () => {
292
+ expect(missingArgResult.exitCode).not.toBe(0);
293
+ });
294
+
295
+ it('stderr contains an error message', () => {
296
+ expect(missingArgResult.stderr).not.toBe('');
297
+ });
298
+ });
299
+ },
300
+});
301
+```
302
+
303
+Notes:
304
+- No stdout redirection test needed — error cases don't produce data output.
305
+- Use `not.toBe(0)` for exit code rather than a specific number (the CLI may change error codes).
306
+- Assert stderr is non-empty; optionally use `toContain()` for specific error message fragments.
307
+
308
+### Pattern D: Multi-line / API-dependent Output
309
+
310
+For commands that call the Composio API and return dynamic, multi-line data.
311
+
312
+```typescript
313
+/**
314
+ * CLI toolkits-list command e2e test
315
+ *
316
+ * Verifies that the CLI lists available toolkits from the API.
317
+ */
318
+
319
+import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
320
+import { TIMEOUTS } from '@e2e-tests/utils/const';
321
+import { describe, it, expect, beforeAll } from 'bun:test';
322
+
323
+declare module 'bun' {
324
+ interface Env {
325
+ COMPOSIO_API_KEY: string;
326
+ }
327
+}
328
+
329
+e2e(import.meta.url, {
330
+ versions: {
331
+ cli: ['current'],
332
+ },
333
+ env: {
334
+ COMPOSIO_API_KEY: Bun.env.COMPOSIO_API_KEY,
335
+ },
336
+ defineTests: ({ runCmd }) => {
337
+ let listResult: E2ETestResult;
338
+ let redirectedResult: E2ETestResultWithFiles<'out.txt'>;
339
+
340
+ beforeAll(async () => {
341
+ listResult = await runCmd('composio toolkits list');
342
+ redirectedResult = await runCmd({
343
+ command: 'composio toolkits list > out.txt',
344
+ files: ['out.txt'],
345
+ });
346
+ }, TIMEOUTS.FIXTURE);
347
+
348
+ describe('composio toolkits list', () => {
349
+ it('exits successfully', () => {
350
+ expect(listResult.exitCode).toBe(0);
351
+ });
352
+
353
+ it('stdout is non-empty', () => {
354
+ expect(sanitizeOutput(listResult.stdout).length).toBeGreaterThan(0);
355
+ });
356
+
357
+ it('stdout contains known toolkits', () => {
358
+ const output = sanitizeOutput(listResult.stdout);
359
+ // Use well-known toolkits that are always present
360
+ expect(output).toContain('github');
361
+ });
362
+
363
+ it('stdout has multiple lines', () => {
364
+ const lines = sanitizeOutput(listResult.stdout).split('\n').filter(Boolean);
365
+ expect(lines.length).toBeGreaterThan(1);
366
+ });
367
+
368
+ it('stderr is empty', () => {
369
+ expect(listResult.stderr).toBe('');
370
+ });
371
+ });
372
+
373
+ describe('stdout redirection to out.txt', () => {
374
+ it('exits successfully', () => {
375
+ expect(redirectedResult.exitCode).toBe(0);
376
+ });
377
+
378
+ it('stdout is empty', () => {
379
+ expect(redirectedResult.stdout).toBe('');
380
+ });
381
+
382
+ it('stderr is empty', () => {
383
+ expect(redirectedResult.stderr).toBe('');
384
+ });
385
+
386
+ it('out.txt contains data', () => {
387
+ expect(sanitizeOutput(redirectedResult.files['out.txt']).length).toBeGreaterThan(0);
388
+ });
389
+ });
390
+ },
391
+});
392
+```
393
+
394
+Key rules for API-dependent output:
395
+- **Never use exact-match assertions** (`toBe`) on API data — the data changes over time.
396
+- Use `toContain()` for known, stable items (e.g., `github`, `gmail` are always present).
397
+- Use `toBeGreaterThan()` for structural checks (line count, length).
398
+- Use `toMatch()` with regex for format validation.
399
+
400
+### Pattern E: Action Command, No Stdout Data
401
+
402
+For commands that perform an action but produce no machine-readable output (e.g., `logout`, `upgrade`).
403
+
404
+```typescript
405
+/**
406
+ * CLI logout command e2e test
407
+ *
408
+ * Verifies that the CLI logout command completes without error.
409
+ */
410
+
411
+import { e2e, type E2ETestResult } from '@e2e-tests/utils';
412
+import { TIMEOUTS } from '@e2e-tests/utils/const';
413
+import { describe, it, expect, beforeAll } from 'bun:test';
414
+
415
+e2e(import.meta.url, {
416
+ versions: {
417
+ cli: ['current'],
418
+ },
419
+ defineTests: ({ runCmd }) => {
420
+ let logoutResult: E2ETestResult;
421
+
422
+ beforeAll(async () => {
423
+ logoutResult = await runCmd('composio logout');
424
+ }, TIMEOUTS.FIXTURE);
425
+
426
+ describe('composio logout', () => {
427
+ it('exits successfully', () => {
428
+ expect(logoutResult.exitCode).toBe(0);
429
+ });
430
+
431
+ it('stdout is empty', () => {
432
+ expect(logoutResult.stdout).toBe('');
433
+ });
434
+
435
+ it('stderr is empty', () => {
436
+ expect(logoutResult.stderr).toBe('');
437
+ });
438
+ });
439
+ },
440
+});
441
+```
442
+
443
+Notes:
444
+- No redirection test needed — there's no data to redirect.
445
+- No `sanitizeOutput()` needed — stdout should be literally empty.
446
+- Import only `E2ETestResult`, not `E2ETestResultWithFiles`.
447
+
448
+## Output Contract Rules
449
+
450
+Enforce these in every CLI e2e test:
451
+
452
+1. **Data commands** produce machine-readable output on stdout (via `ui.output()`).
453
+2. **Action commands** produce no stdout data — stdout is always empty.
454
+3. **stderr is always empty** for successful commands (non-TTY suppresses all Clack decoration).
455
+4. **Redirected output** (`> out.txt`) captures the same data that direct stdout contains; Docker stdout and stderr both become empty.
456
+
457
+## Commands NOT Testable with This Framework
458
+
459
+Do not attempt to write e2e tests for:
460
+
461
+| Category | Examples | Why |
462
+|---|---|---|
463
+| **Interactive commands** | `login`, `init` (with prompts) | No TTY in Docker — Clack prompts hang |
464
+| **Long-running commands** | `triggers listen` | No mechanism to stop the process |
465
+| **Browser-dependent commands** | `login` (without `--no-browser`) | No browser in Docker |
466
+| **Host filesystem writers** | `generate` (writes to project dir) | Docker container is isolated |
467
+
468
+## Environment Variables
469
+
470
+### Auto-passed vars
471
+
472
+`COMPOSIO_API_KEY` and `OPENAI_API_KEY` are automatically forwarded to Docker containers if present on the host (defined in `WELL_KNOWN_ENV_VARS`).
473
+
474
+### Declaring required vars
475
+
476
+For tests that need env vars:
477
+
478
+1. Add the `declare module 'bun'` augmentation (compile-time safety).
479
+2. Pass via `E2EConfig.env` (runtime delivery to Docker).
480
+3. If any `E2EConfig.env` value is `undefined`, the test fails fast at startup — before any Docker container runs.
481
+
482
+### Multiple env vars
483
+
484
+```typescript
485
+declare module 'bun' {
486
+ interface Env {
487
+ COMPOSIO_API_KEY: string;
488
+ COMPOSIO_BASE_URL: string;
489
+ }
490
+}
491
+
492
+e2e(import.meta.url, {
493
+ env: {
494
+ COMPOSIO_API_KEY: Bun.env.COMPOSIO_API_KEY,
495
+ COMPOSIO_BASE_URL: Bun.env.COMPOSIO_BASE_URL,
496
+ },
497
+ // ...
498
+});
499
+```
500
+
501
+## Shell Quoting
502
+
503
+Commands run inside `sh -c '...'` in a POSIX shell. Rules:
504
+
505
+- Use double quotes for flag values with spaces or special chars:
506
+ ```typescript
507
+ runCmd('composio tools info "GMAIL_SEND_EMAIL"')
508
+ ```
509
+- Use single quotes inside double-quoted contexts for literal strings.
510
+- **No bash-isms**: no `[[`, no `$()`, no arrays.
511
+- `PATH` is `/usr/local/bin:/bin` — set in the Dockerfile.
512
+
513
+## Checklist for Adding a New Test Suite
514
+
515
+1. **Create directory**: `ts/e2e-tests/cli/<suite-name>/`
516
+2. **Create `package.json`**: Use the template above with name `@e2e-tests/cli-<suite-name>`
517
+3. **Create `e2e.test.ts`**: Follow the appropriate pattern (A through E)
518
+4. **Run `pnpm install`** from the monorepo root to resolve the workspace link
519
+5. **Run `pnpm test:e2e:cli`** to verify the test passes
520
+6. **Update `ts/e2e-tests/cli/README.md`**: Add a new row to the test suites table
521
+
522
+### README table format
523
+
524
+```markdown
525
+| [suite-name](./suite-name/) | `composio <command>` description | `ENV_VAR_1`, `ENV_VAR_2` (or None) |
526
+```
527
+
528
+## Troubleshooting
529
+
530
+### Reading DEBUG.log
531
+
532
+Each test suite generates a `DEBUG.log` in its directory with Docker execution details: container name, command, duration, exit code, stdout, stderr. Read this first when a test fails.
533
+
534
+### Stale Docker image
535
+
536
+If CLI code changed but tests show old behavior, the Docker image is cached. Rebuild:
537
+
538
+```bash
539
+docker rmi composio-e2e-cli:$(jq -r .version ts/packages/cli/package.json)
540
+```
541
+
542
+### Shell quoting issues
543
+
544
+Check the command string in `DEBUG.log`. Look for unescaped special characters that break `sh -c` parsing.
545
+
546
+### Missing env vars
547
+
548
+The error appears at test startup (before any Docker container runs), not during execution. Verify `Bun.env.*` values are set in your shell.
549
+
550
+### Container statelessness
551
+
552
+Each `runCmd` call runs in a fresh container. If you need to test a multi-step workflow (e.g., "set config then run command"), you must chain commands in a single `runCmd` call:
553
+
554
+```typescript
555
+runCmd('composio logout && composio whoami')
556
+```
557
+
558
+## API Reference
559
+
560
+### `e2e(importMetaUrl, config)`
561
+
562
+Entry point. Pass `import.meta.url` as the first argument — the framework infers `cwd` and `suiteName` from the file path.
563
+
564
+### `E2EConfig`
565
+
566
+```typescript
567
+{
568
+ versions: { cli: ['current'] }, // Always use ['current'] for CLI tests
569
+ env?: Record<string, string | undefined>, // Env vars to pass to Docker
570
+ defineTests: (ctx: DefineTestsContext) => void,
571
+}
572
+```
573
+
574
+### `runCmd` (two overloads)
575
+
576
+```typescript
577
+// Simple: returns { exitCode, stdout, stderr }
578
+const result: E2ETestResult = await runCmd('composio version');
579
+
580
+// With file capture: returns { exitCode, stdout, stderr, files }
581
+const result: E2ETestResultWithFiles<'out.txt'> = await runCmd({
582
+ command: 'composio version > out.txt',
583
+ files: ['out.txt'],
584
+});
585
+// Access: result.files['out.txt']
586
+```
587
+
588
+### `sanitizeOutput(output)`
589
+
590
+Strips ANSI escape codes, normalizes `\r\n` to `\n`, trims whitespace. Use on stdout and file contents before assertions.
591
+
592
+### `TIMEOUTS`
593
+
594
+```typescript
595
+TIMEOUTS.DEFAULT // 5_000ms — individual test timeout
596
+TIMEOUTS.FIXTURE // 120_000ms — beforeAll timeout (Docker startup + command execution)
597
+TIMEOUTS.LLM_SHORT // 30_000ms — commands involving LLM calls
598
+TIMEOUTS.LLM_LONG // 60_000ms — commands involving longer LLM calls
599
+```
600
+
601
+Use `TIMEOUTS.FIXTURE` for the `beforeAll` that runs Docker commands. Use `TIMEOUTS.DEFAULT` for individual `it()` blocks (the default).
602
+
603
+## Reference Files
604
+
605
+| File | Purpose |
606
+|---|---|
607
+| `ts/e2e-tests/cli/version/e2e.test.ts` | Pattern A reference |
608
+| `ts/e2e-tests/cli/whoami/e2e.test.ts` | Pattern B reference |
609
+| `ts/e2e-tests/_utils/src/types.ts` | `E2EConfig`, `E2ETestResult`, `DefineTestsContext` types |
610
+| `ts/e2e-tests/_utils/src/e2e.ts` | `e2e()` entry point |
611
+| `ts/e2e-tests/_utils/src/const.ts` | `TIMEOUTS`, `WELL_KNOWN_ENV_VARS` |
612
+| `ts/e2e-tests/_utils/src/sanitize.ts` | `sanitizeOutput()` |
613
+| `ts/e2e-tests/_utils/Dockerfile.cli` | Docker image definition |
614
+| `ts/e2e-tests/cli/README.md` | Test suites table |
615
+| `CLI.md` | Planned CLI commands |
616
+| `ts/packages/cli/CLAUDE.md` | CLI architecture and output conventions |