@devries-cameron20 / composio Mirror · read-only / commits / 6c4e81405
Fork 0

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

Mirrored from https://github.com/Camo67/composio.git

Synced Nov 22, 08:15 UTC

feat(ai): add "create-cli-e2e" skill

jkomyno committed Feb 18, 2026 at 16:33 UTC 6c4e81405933f07671e94cf11ff41344f2550db8
1 file changed +616
.claude/skills/create-cli-e2e/SKILL.md new
+616
@@ -0,0 +1,616 @@
1 +# CLI E2E Test Development
2 +
3 +Write, expand, and maintain end-to-end tests for CLI commands in `ts/e2e-tests/cli/`.
4 +
5 +## When to Use
6 +
7 +- Adding a new e2e test suite for a CLI command
8 +- Modifying or extending an existing CLI e2e test
9 +- Debugging a failing CLI e2e test
10 +- Reviewing whether a CLI command's output contract is properly tested
11 +
12 +## Architecture
13 +
14 +Each CLI e2e test runs the compiled `composio` binary inside a **scratch Debian Docker container**. The binary is self-contained (built via `bun build --compile`) — no Node, Bun, or pnpm exists in the runtime image.
15 +
16 +Key properties:
17 +
18 +- Each test suite = a directory under `ts/e2e-tests/cli/<suite-name>/`
19 +- Use `runCmd` only. Never use `runFixture` (throws an error for CLI tests). Never set `usesFixtures`.
20 +- **Each `runCmd` call creates a fresh container.** No state persists between calls.
21 +- Commands run inside `sh -c '...'` — POSIX shell only, no bash-isms.
22 +- Containers have network access — API-calling commands work.
23 +- `HOME=/tmp`, cache dir is `/tmp/.composio/` — auth passes via env vars only.
24 +- `process.stdout.isTTY` is always `false` inside Docker — the CLI always runs in piped mode.
25 +
26 +### What "piped mode" means for tests
27 +
28 +Inside Docker `sh -c`, the composio binary's stdout is never a TTY. This means `ui.output()` writes to stdout and all Clack decoration is suppressed. Both test groups verify piped-mode behavior:
29 +
30 +| Test Group | What It Verifies |
31 +|---|---|
32 +| **Command execution** (`composio version`) | stdout contains the data, stderr is empty |
33 +| **Stdout redirection** (`composio version > out.txt`) | Shell-level redirect captures data into a file; Docker stdout and stderr are both empty |
34 +
35 +## File Structure
36 +
37 +For a new test suite `<suite-name>`, create 2 files:
38 +
39 +```
40 +ts/e2e-tests/cli/<suite-name>/
41 +├── e2e.test.ts # Test file
42 +└── package.json # Package manifest
43 +```
44 +
45 +### Naming Conventions
46 +
47 +- **Directory**: hyphen-separated lowercase matching the command structure
48 + - `version`, `whoami`, `toolkits-list`, `tools-info`, `auth-configs-list`, `connected-accounts-link`
49 +- **Package name**: `@e2e-tests/cli-<suite-name>`
50 + - `@e2e-tests/cli-version`, `@e2e-tests/cli-toolkits-list`
51 +
52 +### package.json Template
53 +
54 +```json
55 +{
56 + "name": "@e2e-tests/cli-<suite-name>",
57 + "version": "0.0.0",
58 + "private": true,
59 + "type": "module",
60 + "scripts": {
61 + "typecheck": "tsc --noEmit",
62 + "test:e2e": "bun test e2e.test.ts",
63 + "test:e2e:cli": "bun test e2e.test.ts"
64 + },
65 + "devDependencies": {
66 + "@e2e-tests/utils": "workspace:*"
67 + }
68 +}
69 +```
70 +
71 +## Test Patterns
72 +
73 +Every test file starts with the same structure:
74 +
75 +```typescript
76 +import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
77 +import { TIMEOUTS } from '@e2e-tests/utils/const';
78 +import { describe, it, expect, beforeAll } from 'bun:test';
79 +
80 +e2e(import.meta.url, {
81 + versions: { cli: ['current'] },
82 + // env: { ... }, // Only if needed
83 + defineTests: ({ runCmd }) => {
84 + // Tests here
85 + },
86 +});
87 +```
88 +
89 +### Pattern A: Simple Command, No Env Vars
90 +
91 +For commands that produce deterministic output without needing authentication.
92 +
93 +**Reference**: `ts/e2e-tests/cli/version/e2e.test.ts`
94 +
95 +```typescript
96 +/**
97 + * CLI version command e2e test
98 + *
99 + * Verifies that the compiled composio CLI behaves correctly in a scratch container.
100 + */
101 +
102 +import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
103 +import { TIMEOUTS } from '@e2e-tests/utils/const';
104 +import { describe, it, expect, beforeAll } from 'bun:test';
105 +import cliPkg from '../../../packages/cli/package.json' with { type: 'json' };
106 +
107 +e2e(import.meta.url, {
108 + versions: {
109 + cli: ['current'],
110 + },
111 + defineTests: ({ runCmd }) => {
112 + const expectedVersion = String(cliPkg.version ?? '').trim();
113 + let versionResult: E2ETestResult;
114 + let redirectedResult: E2ETestResultWithFiles<'out.txt'>;
115 +
116 + beforeAll(async () => {
117 + versionResult = await runCmd('composio version');
118 + redirectedResult = await runCmd({
119 + command: 'composio version > out.txt',
120 + files: ['out.txt'],
121 + });
122 + }, TIMEOUTS.FIXTURE);
123 +
124 + describe('composio version', () => {
125 + it('exits successfully', () => {
126 + expect(versionResult.exitCode).toBe(0);
127 + });
128 +
129 + it('stdout matches snapshot', () => {
130 + expect(sanitizeOutput(versionResult.stdout)).toBe(expectedVersion);
131 + });
132 +
133 + it('stderr matches snapshot', () => {
134 + expect(versionResult.stderr).toBe('');
135 + });
136 + });
137 +
138 + describe('stdout redirection to out.txt', () => {
139 + it('exits successfully', () => {
140 + expect(redirectedResult.exitCode).toBe(0);
141 + });
142 +
143 + it('stdout is empty', () => {
144 + expect(redirectedResult.stdout).toBe('');
145 + });
146 +
147 + it('stderr is empty', () => {
148 + expect(redirectedResult.stderr).toBe('');
149 + });
150 +
151 + it('out.txt matches snapshot', () => {
152 + expect(sanitizeOutput(redirectedResult.files['out.txt'])).toBe(expectedVersion);
153 + });
154 + });
155 + },
156 +});
157 +```
158 +
159 +### Pattern B: Command Requiring Env Vars
160 +
161 +For commands that need authentication or configuration from the host environment.
162 +
163 +**Reference**: `ts/e2e-tests/cli/whoami/e2e.test.ts`
164 +
165 +Three additions compared to Pattern A:
166 +
167 +1. **Type augmentation** for compile-time safety on `Bun.env`:
168 + ```typescript
169 + declare module 'bun' {
170 + interface Env {
171 + COMPOSIO_API_KEY: string;
172 + }
173 + }
174 + ```
175 +
176 +2. **Pass env vars** in the config:
177 + ```typescript
178 + env: {
179 + COMPOSIO_API_KEY: Bun.env.COMPOSIO_API_KEY,
180 + },
181 + ```
182 +
183 +3. **Derive expected values** from the env:
184 + ```typescript
185 + const expectedApiKey = Bun.env.COMPOSIO_API_KEY.trim();
186 + ```
187 +
188 +The rest follows the same two-group structure (command execution + stdout redirection).
189 +
190 +**Full example:**
191 +
192 +```typescript
193 +/**
194 + * CLI whoami command e2e test
195 + *
196 + * Verifies that the compiled composio CLI prints the API key in a scratch container.
197 + */
198 +
199 +import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
200 +import { TIMEOUTS } from '@e2e-tests/utils/const';
201 +import { describe, it, expect, beforeAll } from 'bun:test';
202 +
203 +declare module 'bun' {
204 + interface Env {
205 + COMPOSIO_API_KEY: string;
206 + }
207 +}
208 +
209 +e2e(import.meta.url, {
210 + versions: {
211 + cli: ['current'],
212 + },
213 + env: {
214 + COMPOSIO_API_KEY: Bun.env.COMPOSIO_API_KEY,
215 + },
216 + defineTests: ({ runCmd }) => {
217 + const expectedApiKey = Bun.env.COMPOSIO_API_KEY.trim();
218 + let whoamiResult: E2ETestResult;
219 + let redirectedResult: E2ETestResultWithFiles<'out.txt'>;
220 +
221 + beforeAll(async () => {
222 + whoamiResult = await runCmd('composio whoami');
223 + redirectedResult = await runCmd({
224 + command: 'composio whoami > out.txt',
225 + files: ['out.txt'],
226 + });
227 + }, TIMEOUTS.FIXTURE);
228 +
229 + describe('composio whoami', () => {
230 + it('exits successfully', () => {
231 + expect(whoamiResult.exitCode).toBe(0);
232 + });
233 +
234 + it('stdout contains the API key', () => {
235 + expect(sanitizeOutput(whoamiResult.stdout)).toBe(expectedApiKey);
236 + });
237 +
238 + it('stderr is empty', () => {
239 + expect(whoamiResult.stderr).toBe('');
240 + });
241 + });
242 +
243 + describe('stdout redirection to out.txt', () => {
244 + it('exits successfully', () => {
245 + expect(redirectedResult.exitCode).toBe(0);
246 + });
247 +
248 + it('stdout is empty', () => {
249 + expect(redirectedResult.stdout).toBe('');
250 + });
251 +
252 + it('stderr is empty', () => {
253 + expect(redirectedResult.stderr).toBe('');
254 + });
255 +
256 + it('out.txt contains the API key', () => {
257 + expect(sanitizeOutput(redirectedResult.files['out.txt'])).toBe(expectedApiKey);
258 + });
259 + });
260 + },
261 +});
262 +```
263 +
264 +### Pattern C: Error Case Testing
265 +
266 +For verifying that the CLI fails gracefully with correct exit codes and error messages.
267 +
268 +```typescript
269 +/**
270 + * CLI tools-info error case e2e test
271 + *
272 + * Verifies that the CLI fails gracefully when required arguments are missing.
273 + */
274 +
275 +import { e2e, sanitizeOutput, type E2ETestResult } from '@e2e-tests/utils';
276 +import { TIMEOUTS } from '@e2e-tests/utils/const';
277 +import { describe, it, expect, beforeAll } from 'bun:test';
278 +
279 +e2e(import.meta.url, {
280 + versions: {
281 + cli: ['current'],
282 + },
283 + defineTests: ({ runCmd }) => {
284 + let missingArgResult: E2ETestResult;
285 +
286 + beforeAll(async () => {
287 + missingArgResult = await runCmd('composio tools info');
288 + }, TIMEOUTS.FIXTURE);
289 +
290 + describe('composio tools info (missing argument)', () => {
291 + it('exits with non-zero code', () => {
292 + expect(missingArgResult.exitCode).not.toBe(0);
293 + });
294 +
295 + it('stderr contains an error message', () => {
296 + expect(missingArgResult.stderr).not.toBe('');
297 + });
298 + });
299 + },
300 +});
301 +```
302 +
303 +Notes:
304 +- No stdout redirection test needed — error cases don't produce data output.
305 +- Use `not.toBe(0)` for exit code rather than a specific number (the CLI may change error codes).
306 +- Assert stderr is non-empty; optionally use `toContain()` for specific error message fragments.
307 +
308 +### Pattern D: Multi-line / API-dependent Output
309 +
310 +For commands that call the Composio API and return dynamic, multi-line data.
311 +
312 +```typescript
313 +/**
314 + * CLI toolkits-list command e2e test
315 + *
316 + * Verifies that the CLI lists available toolkits from the API.
317 + */
318 +
319 +import { e2e, sanitizeOutput, type E2ETestResult, type E2ETestResultWithFiles } from '@e2e-tests/utils';
320 +import { TIMEOUTS } from '@e2e-tests/utils/const';
321 +import { describe, it, expect, beforeAll } from 'bun:test';
322 +
323 +declare module 'bun' {
324 + interface Env {
325 + COMPOSIO_API_KEY: string;
326 + }
327 +}
328 +
329 +e2e(import.meta.url, {
330 + versions: {
331 + cli: ['current'],
332 + },
333 + env: {
334 + COMPOSIO_API_KEY: Bun.env.COMPOSIO_API_KEY,
335 + },
336 + defineTests: ({ runCmd }) => {
337 + let listResult: E2ETestResult;
338 + let redirectedResult: E2ETestResultWithFiles<'out.txt'>;
339 +
340 + beforeAll(async () => {
341 + listResult = await runCmd('composio toolkits list');
342 + redirectedResult = await runCmd({
343 + command: 'composio toolkits list > out.txt',
344 + files: ['out.txt'],
345 + });
346 + }, TIMEOUTS.FIXTURE);
347 +
348 + describe('composio toolkits list', () => {
349 + it('exits successfully', () => {
350 + expect(listResult.exitCode).toBe(0);
351 + });
352 +
353 + it('stdout is non-empty', () => {
354 + expect(sanitizeOutput(listResult.stdout).length).toBeGreaterThan(0);
355 + });
356 +
357 + it('stdout contains known toolkits', () => {
358 + const output = sanitizeOutput(listResult.stdout);
359 + // Use well-known toolkits that are always present
360 + expect(output).toContain('github');
361 + });
362 +
363 + it('stdout has multiple lines', () => {
364 + const lines = sanitizeOutput(listResult.stdout).split('\n').filter(Boolean);
365 + expect(lines.length).toBeGreaterThan(1);
366 + });
367 +
368 + it('stderr is empty', () => {
369 + expect(listResult.stderr).toBe('');
370 + });
371 + });
372 +
373 + describe('stdout redirection to out.txt', () => {
374 + it('exits successfully', () => {
375 + expect(redirectedResult.exitCode).toBe(0);
376 + });
377 +
378 + it('stdout is empty', () => {
379 + expect(redirectedResult.stdout).toBe('');
380 + });
381 +
382 + it('stderr is empty', () => {
383 + expect(redirectedResult.stderr).toBe('');
384 + });
385 +
386 + it('out.txt contains data', () => {
387 + expect(sanitizeOutput(redirectedResult.files['out.txt']).length).toBeGreaterThan(0);
388 + });
389 + });
390 + },
391 +});
392 +```
393 +
394 +Key rules for API-dependent output:
395 +- **Never use exact-match assertions** (`toBe`) on API data — the data changes over time.
396 +- Use `toContain()` for known, stable items (e.g., `github`, `gmail` are always present).
397 +- Use `toBeGreaterThan()` for structural checks (line count, length).
398 +- Use `toMatch()` with regex for format validation.
399 +
400 +### Pattern E: Action Command, No Stdout Data
401 +
402 +For commands that perform an action but produce no machine-readable output (e.g., `logout`, `upgrade`).
403 +
404 +```typescript
405 +/**
406 + * CLI logout command e2e test
407 + *
408 + * Verifies that the CLI logout command completes without error.
409 + */
410 +
411 +import { e2e, type E2ETestResult } from '@e2e-tests/utils';
412 +import { TIMEOUTS } from '@e2e-tests/utils/const';
413 +import { describe, it, expect, beforeAll } from 'bun:test';
414 +
415 +e2e(import.meta.url, {
416 + versions: {
417 + cli: ['current'],
418 + },
419 + defineTests: ({ runCmd }) => {
420 + let logoutResult: E2ETestResult;
421 +
422 + beforeAll(async () => {
423 + logoutResult = await runCmd('composio logout');
424 + }, TIMEOUTS.FIXTURE);
425 +
426 + describe('composio logout', () => {
427 + it('exits successfully', () => {
428 + expect(logoutResult.exitCode).toBe(0);
429 + });
430 +
431 + it('stdout is empty', () => {
432 + expect(logoutResult.stdout).toBe('');
433 + });
434 +
435 + it('stderr is empty', () => {
436 + expect(logoutResult.stderr).toBe('');
437 + });
438 + });
439 + },
440 +});
441 +```
442 +
443 +Notes:
444 +- No redirection test needed — there's no data to redirect.
445 +- No `sanitizeOutput()` needed — stdout should be literally empty.
446 +- Import only `E2ETestResult`, not `E2ETestResultWithFiles`.
447 +
448 +## Output Contract Rules
449 +
450 +Enforce these in every CLI e2e test:
451 +
452 +1. **Data commands** produce machine-readable output on stdout (via `ui.output()`).
453 +2. **Action commands** produce no stdout data — stdout is always empty.
454 +3. **stderr is always empty** for successful commands (non-TTY suppresses all Clack decoration).
455 +4. **Redirected output** (`> out.txt`) captures the same data that direct stdout contains; Docker stdout and stderr both become empty.
456 +
457 +## Commands NOT Testable with This Framework
458 +
459 +Do not attempt to write e2e tests for:
460 +
461 +| Category | Examples | Why |
462 +|---|---|---|
463 +| **Interactive commands** | `login`, `init` (with prompts) | No TTY in Docker — Clack prompts hang |
464 +| **Long-running commands** | `triggers listen` | No mechanism to stop the process |
465 +| **Browser-dependent commands** | `login` (without `--no-browser`) | No browser in Docker |
466 +| **Host filesystem writers** | `generate` (writes to project dir) | Docker container is isolated |
467 +
468 +## Environment Variables
469 +
470 +### Auto-passed vars
471 +
472 +`COMPOSIO_API_KEY` and `OPENAI_API_KEY` are automatically forwarded to Docker containers if present on the host (defined in `WELL_KNOWN_ENV_VARS`).
473 +
474 +### Declaring required vars
475 +
476 +For tests that need env vars:
477 +
478 +1. Add the `declare module 'bun'` augmentation (compile-time safety).
479 +2. Pass via `E2EConfig.env` (runtime delivery to Docker).
480 +3. If any `E2EConfig.env` value is `undefined`, the test fails fast at startup — before any Docker container runs.
481 +
482 +### Multiple env vars
483 +
484 +```typescript
485 +declare module 'bun' {
486 + interface Env {
487 + COMPOSIO_API_KEY: string;
488 + COMPOSIO_BASE_URL: string;
489 + }
490 +}
491 +
492 +e2e(import.meta.url, {
493 + env: {
494 + COMPOSIO_API_KEY: Bun.env.COMPOSIO_API_KEY,
495 + COMPOSIO_BASE_URL: Bun.env.COMPOSIO_BASE_URL,
496 + },
497 + // ...
498 +});
499 +```
500 +
501 +## Shell Quoting
502 +
503 +Commands run inside `sh -c '...'` in a POSIX shell. Rules:
504 +
505 +- Use double quotes for flag values with spaces or special chars:
506 + ```typescript
507 + runCmd('composio tools info "GMAIL_SEND_EMAIL"')
508 + ```
509 +- Use single quotes inside double-quoted contexts for literal strings.
510 +- **No bash-isms**: no `[[`, no `$()`, no arrays.
511 +- `PATH` is `/usr/local/bin:/bin` — set in the Dockerfile.
512 +
513 +## Checklist for Adding a New Test Suite
514 +
515 +1. **Create directory**: `ts/e2e-tests/cli/<suite-name>/`
516 +2. **Create `package.json`**: Use the template above with name `@e2e-tests/cli-<suite-name>`
517 +3. **Create `e2e.test.ts`**: Follow the appropriate pattern (A through E)
518 +4. **Run `pnpm install`** from the monorepo root to resolve the workspace link
519 +5. **Run `pnpm test:e2e:cli`** to verify the test passes
520 +6. **Update `ts/e2e-tests/cli/README.md`**: Add a new row to the test suites table
521 +
522 +### README table format
523 +
524 +```markdown
525 +| [suite-name](./suite-name/) | `composio <command>` description | `ENV_VAR_1`, `ENV_VAR_2` (or None) |
526 +```
527 +
528 +## Troubleshooting
529 +
530 +### Reading DEBUG.log
531 +
532 +Each test suite generates a `DEBUG.log` in its directory with Docker execution details: container name, command, duration, exit code, stdout, stderr. Read this first when a test fails.
533 +
534 +### Stale Docker image
535 +
536 +If CLI code changed but tests show old behavior, the Docker image is cached. Rebuild:
537 +
538 +```bash
539 +docker rmi composio-e2e-cli:$(jq -r .version ts/packages/cli/package.json)
540 +```
541 +
542 +### Shell quoting issues
543 +
544 +Check the command string in `DEBUG.log`. Look for unescaped special characters that break `sh -c` parsing.
545 +
546 +### Missing env vars
547 +
548 +The error appears at test startup (before any Docker container runs), not during execution. Verify `Bun.env.*` values are set in your shell.
549 +
550 +### Container statelessness
551 +
552 +Each `runCmd` call runs in a fresh container. If you need to test a multi-step workflow (e.g., "set config then run command"), you must chain commands in a single `runCmd` call:
553 +
554 +```typescript
555 +runCmd('composio logout && composio whoami')
556 +```
557 +
558 +## API Reference
559 +
560 +### `e2e(importMetaUrl, config)`
561 +
562 +Entry point. Pass `import.meta.url` as the first argument — the framework infers `cwd` and `suiteName` from the file path.
563 +
564 +### `E2EConfig`
565 +
566 +```typescript
567 +{
568 + versions: { cli: ['current'] }, // Always use ['current'] for CLI tests
569 + env?: Record<string, string | undefined>, // Env vars to pass to Docker
570 + defineTests: (ctx: DefineTestsContext) => void,
571 +}
572 +```
573 +
574 +### `runCmd` (two overloads)
575 +
576 +```typescript
577 +// Simple: returns { exitCode, stdout, stderr }
578 +const result: E2ETestResult = await runCmd('composio version');
579 +
580 +// With file capture: returns { exitCode, stdout, stderr, files }
581 +const result: E2ETestResultWithFiles<'out.txt'> = await runCmd({
582 + command: 'composio version > out.txt',
583 + files: ['out.txt'],
584 +});
585 +// Access: result.files['out.txt']
586 +```
587 +
588 +### `sanitizeOutput(output)`
589 +
590 +Strips ANSI escape codes, normalizes `\r\n` to `\n`, trims whitespace. Use on stdout and file contents before assertions.
591 +
592 +### `TIMEOUTS`
593 +
594 +```typescript
595 +TIMEOUTS.DEFAULT // 5_000ms — individual test timeout
596 +TIMEOUTS.FIXTURE // 120_000ms — beforeAll timeout (Docker startup + command execution)
597 +TIMEOUTS.LLM_SHORT // 30_000ms — commands involving LLM calls
598 +TIMEOUTS.LLM_LONG // 60_000ms — commands involving longer LLM calls
599 +```
600 +
601 +Use `TIMEOUTS.FIXTURE` for the `beforeAll` that runs Docker commands. Use `TIMEOUTS.DEFAULT` for individual `it()` blocks (the default).
602 +
603 +## Reference Files
604 +
605 +| File | Purpose |
606 +|---|---|
607 +| `ts/e2e-tests/cli/version/e2e.test.ts` | Pattern A reference |
608 +| `ts/e2e-tests/cli/whoami/e2e.test.ts` | Pattern B reference |
609 +| `ts/e2e-tests/_utils/src/types.ts` | `E2EConfig`, `E2ETestResult`, `DefineTestsContext` types |
610 +| `ts/e2e-tests/_utils/src/e2e.ts` | `e2e()` entry point |
611 +| `ts/e2e-tests/_utils/src/const.ts` | `TIMEOUTS`, `WELL_KNOWN_ENV_VARS` |
612 +| `ts/e2e-tests/_utils/src/sanitize.ts` | `sanitizeOutput()` |
613 +| `ts/e2e-tests/_utils/Dockerfile.cli` | Docker image definition |
614 +| `ts/e2e-tests/cli/README.md` | Test suites table |
615 +| `CLI.md` | Planned CLI commands |
616 +| `ts/packages/cli/CLAUDE.md` | CLI architecture and output conventions |