Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

next
md 353 lines 13.5 KB
Rendered Raw
1 # E2E Test Utilities
2
3 Shared infrastructure for running `@composio/core` and CLI end-to-end tests in isolated Docker environments.
4
5 ## What's Here
6
7 | File/Directory | Purpose |
8 | ----------------- | ---------------------------------------------------- |
9 | `src/` | TypeScript utilities (e2e runner, config, types) |
10 | `scripts/` | Docker build and cleanup scripts |
11 | `Dockerfile.node` | Multi-stage Dockerfile for Node.js test environments |
12 | `Dockerfile.cli` | Scratch Dockerfile for CLI test environments |
13
14 ## API
15
16 ### `e2e`
17
18 The main entry point for e2e tests. Automatically infers the working directory and suite name from the caller's location. Uses `bun:test` for the test framework.
19
20 ```typescript
21 import { e2e, type E2ETestResult } from '@e2e-tests/utils';
22 import { TIMEOUTS } from '@e2e-tests/utils/const';
23 import { describe, it, expect, beforeAll } from 'bun:test';
24
25 e2e(import.meta.url, {
26 versions: {
27 node: ['22.23.5', '24.22.3', '26.13.1'], // optional, defaults to mise.toml
28 cli: ['current'], // optional, defaults to CLI package.json version
29 },
30 env: { MY_VAR: 'value' }, // optional env vars
31 defineTests: ({ runtime, runCmd, runFixture }) => {
32 let result: E2ETestResult;
33
34 beforeAll(async () => {
35 result = await runFixture({ filename: 'fixtures/test.mjs' });
36 }, TIMEOUTS.FIXTURE);
37
38 describe('output', () => {
39 it('exits successfully', () => {
40 expect(result.exitCode).toBe(0);
41 });
42 });
43 },
44 });
45 ```
46
47 ### `E2EConfig`
48
49 Configuration object passed to `e2e()`:
50
51 | Property | Type | Description |
52 | -------------- | ------------------------------------- | ------------------------------------------------------------- |
53 | `versions` | `RuntimeVersions` | Runtime versions to test. See RuntimeVersions below |
54 | `env` | `Record<string, string \| undefined>` | Environment variables for Docker. Validated at startup |
55 | `usesFixtures` | `boolean` | When true, sets cwd to `{testDir}/fixtures`. Default: `false` |
56 | `defineTests` | `(ctx: DefineTestsContext) => void` | Callback to define tests using bun:test primitives |
57
58 ### `RuntimeVersions`
59
60 | Property | Type | Description |
61 | -------- | -------------------------------- | ------------------------------------------ |
62 | `node` | `readonly NodeVersionFromUser[]` | Node.js versions. Defaults to `mise.toml` |
63 | `cli` | `readonly CliVersionFromUser[]` | CLI versions. Defaults to CLI package.json |
64
65 ### `DefineTestsContext`
66
67 The context passed to the `defineTests` callback:
68
69 | Property | Type/Signature | Description |
70 | ------------ | ---------------------------------------------------------------------------------- | ----------------------------------------- |
71 | `runtime` | `'node' \| 'cli'` | Current runtime being tested |
72 | `runCmd` | `(command: string) => Promise<E2ETestResult>` | Run arbitrary command in Docker container |
73 | `runFixture` | `(options: RunFixtureOptions) => Promise<E2ETestResult \| E2ETestResultWithSetup>` | Run fixture with optional setup phase |
74
75 ### `RunFixtureOptions`
76
77 Options for `runFixture()`:
78
79 | Property | Type | Description |
80 | ---------- | -------- | ------------------------------------------------------------------- |
81 | `filename` | `string` | Fixture file path relative to cwd (e.g., `'index.mjs'`) |
82 | `setup` | `string` | Optional setup command (e.g., `'npm install'`). Enables volume mode |
83
84 **Behavior:**
85
86 - **Without `setup`**: Runs `node <filename>` directly. Returns `E2ETestResult`.
87 - **With `setup`**: Creates a Docker volume, runs setup with volume mounted read-write, then runs fixture with volume mounted read-only. Returns `E2ETestResultWithSetup`.
88
89 ```typescript
90 // Simple fixture (no dependencies to install)
91 const result = await runFixture({ filename: 'test.mjs' });
92
93 // Fixture with setup phase (uses Docker volumes)
94 const result = await runFixture({
95 filename: 'index.mjs',
96 setup: 'npm install --legacy-peer-deps',
97 });
98 expect(result.setup.exitCode).toBe(0); // Check setup phase
99 expect(result.exitCode).toBe(0); // Check fixture phase
100 ```
101
102 ### `E2ETestResult`
103
104 Result returned by `runCmd` and `runFixture`:
105
106 ```typescript
107 interface E2ETestResult {
108 exitCode: number; // Exit code from the command (0 = success)
109 stdout: string; // Captured stdout
110 stderr: string; // Captured stderr
111 }
112 ```
113
114 ### `runCmd` with File Capture
115
116 When you need to assert on files created inside the container, pass a `files` array:
117
118 ```typescript
119 const result = await runCmd({
120 command: 'composio version > out.txt',
121 files: ['out.txt'],
122 });
123
124 expect(result.files['out.txt']).toBe('0.1.24');
125 ```
126
127 The files are copied out of the container after execution and returned as a map in the result.
128
129 ### `startJourneyContainer`
130
131 One CLI container that stays up while a test runs several commands in it, so a login or a cache file written by one command is there for the next. Unlike `runCmd`, it mounts no host `~/.composio` and passes through no API key.
132
133 ```typescript
134 import { startJourneyContainer } from '@e2e-tests/utils';
135
136 const journey = await startJourneyContainer({ env: { COMPOSIO_BASE_URL: mockUrl } });
137 const first = await journey.exec('composio login --user-api-key uk_test');
138 const second = await journey.exec('composio whoami', { timeoutSeconds: 30 });
139 await journey.writeFile('/work/app/.composio/project.json', '{}');
140 await journey.stop();
141 ```
142
143 | Member | Description |
144 | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
145 | `exec(command, options?)` | Runs a POSIX `sh` command line. Options: `env`, `stdin`, `cwd` (defaults to the home), `timeoutSeconds` (default 60; exit code 124 on expiry), `user` (`uid:gid`, to run without root) |
146 | `writeFile(path, content)` / `readFile(path)` | Write or read a file inside the container |
147 | `steps()` / `writeDebugLog(path)` | The commands run so far with their results |
148 | `home` | The home directory (default `/home/journey`), created empty when the journey starts |
149 | `stop()` | Removes the container |
150
151 ### `E2ETestResultWithSetup`
152
153 Extended result when `runFixture` is called with a `setup` option:
154
155 ```typescript
156 interface E2ETestResultWithSetup extends E2ETestResult {
157 setup: {
158 exitCode: number;
159 stdout: string;
160 stderr: string;
161 };
162 }
163 ```
164
165 Top-level fields (`exitCode`, `stdout`, `stderr`) reflect the fixture result. The `setup` object contains the setup command result.
166
167 ### `sanitizeOutput`
168
169 Utility for stable test comparisons. Removes ANSI escape codes, normalizes line endings, and trims whitespace.
170
171 ```typescript
172 import { sanitizeOutput } from '@e2e-tests/utils';
173
174 const clean = sanitizeOutput(result.stdout);
175 ```
176
177 ### `TIMEOUTS`
178
179 Predefined timeout constants for tests (in milliseconds):
180
181 ```typescript
182 import { TIMEOUTS } from '@e2e-tests/utils/const';
183
184 it(
185 'calls LLM',
186 async () => {
187 // test code
188 },
189 { timeout: TIMEOUTS.LLM_SHORT }
190 );
191 ```
192
193 | Constant | Value | Use Case |
194 | ----------- | --------- | ------------------------------------------ |
195 | `DEFAULT` | `5_000` | Standard test operations |
196 | `FIXTURE` | `120_000` | `beforeAll` hooks that call `runFixture()` |
197 | `LLM_SHORT` | `30_000` | Quick LLM calls |
198 | `LLM_LONG` | `60_000` | Complex LLM operations |
199
200 ## Version Resolution
201
202 ### Node.js Version Resolution
203
204 Node.js versions to test are resolved in this order:
205
206 1. **`COMPOSIO_E2E_NODE_VERSION` env var** (highest priority): Use `[env_value]`
207 2. **`config.versions.node`**: Use the provided array
208 3. **Default**: Use version from `mise.toml` file
209
210 ### Well-Known Node Versions
211
212 The following versions are pre-defined in `const.ts`:
213
214 - `22.23.5`
215 - `24.22.3`
216 - `26.13.1`
217 - `current` (resolves to `mise.toml` version)
218
219 ### CLI image freshness
220
221 Local CLI suites run `docker build` before using an image. Docker checks the current build inputs against its content cache, including uncommitted source and local build outputs. The CLI version tag alone does not establish freshness.
222
223 GitHub Actions can reuse its prebuilt image when the `org.opencontainers.image.revision` label matches the checkout's commit. A missing or different label triggers a rebuild. A failed rebuild fails the test setup instead of accepting an existing tag.
224
225 ### CLI Version Resolution
226
227 CLI versions to test are resolved in this order:
228
229 1. **`COMPOSIO_E2E_CLI_VERSION` env var** (highest priority): Use `[env_value]`
230 2. **`config.versions.cli`**: Use the provided array
231 3. **Default**: Use version from `ts/packages/cli/package.json`
232
233 ### Well-Known CLI Versions
234
235 - `current` (resolves to CLI package.json version)
236
237 ## Environment Variable Validation
238
239 Environment variables passed to `E2EConfig.env` are validated at test startup. If any variable has an `undefined` value, the test fails fast with a clear error message:
240
241 ```
242 [my-test] Missing required environment variables: COMPOSIO_API_KEY, OPENAI_API_KEY
243 Set these variables before running the tests, or remove them from E2EConfig.env if not required.
244 ```
245
246 This prevents silent failures from missing credentials.
247
248 ## The `usesFixtures` Option
249
250 When `usesFixtures: true` is set:
251
252 - Working directory changes to `{testDir}/fixtures/`
253 - Docker volume mounts at `fixtures/node_modules`
254 - Fixture paths in `runFixture({ filename })` are relative to `fixtures/`
255
256 Use this for tests that have their own `package.json` and need to run `npm install`:
257
258 ```typescript
259 import { TIMEOUTS } from '@e2e-tests/utils/const';
260
261 e2e(import.meta.url, {
262 usesFixtures: true,
263 defineTests: ({ runFixture }) => {
264 beforeAll(async () => {
265 // Both commands run in fixtures/ directory
266 result = await runFixture({
267 filename: 'index.mjs', // Resolves to fixtures/index.mjs
268 setup: 'npm install', // Runs in fixtures/
269 });
270 }, TIMEOUTS.FIXTURE);
271 },
272 });
273 ```
274
275 ## Scripts
276
277 ```bash
278 # Pre-build Docker images for all well-known Node and CLI versions
279 pnpm docker:build
280
281 # Remove all e2e Docker images (Node.js and CLI)
282 pnpm docker:clean
283 ```
284
285 ## DEBUG.log Output
286
287 Each test suite generates a `DEBUG.log` file with structured output grouped by runtime version:
288
289 ```
290 ================================================================================
291 E2E Test: openai-zod4-compat
292 Started: 2026-01-30T12:18:42.000Z
293 Test file: ts/e2e-tests/runtimes/node/openai-zod4-compat/e2e.test.ts
294 Runtime versions: Node.js 22.23.5, Node.js 24.22.3, Node.js 26.13.1
295 ================================================================================
296
297 ################################################################################
298 ### Node.js 22.23.5
299 ################################################################################
300 Image: composio-e2e-node:22.23.5
301
302 --- Phase 1/2: setup ---
303 Container: e2e-openai-zod4-compat-22-22-3-1769775520382-setup
304 Command: npm install --legacy-peer-deps
305 Duration: 2.55s
306 Exit Code: 0 (success)
307
308 [stdout]
309 added 3 packages, and audited 5 packages in 2s
310
311 [stderr]
312 (empty)
313
314 --- Phase 2/2: fixture ---
315 Container: e2e-openai-zod4-compat-22-22-3-1769775520382-fixture
316 Command: node index.mjs
317 Duration: 0.56s
318 Exit Code: 0 (success)
319
320 [stdout]
321 zod@4 works
322 openai@5 works
323 All packages work together!
324
325 [stderr]
326 (empty)
327
328 ================================================================================
329 Summary
330 ================================================================================
331 Node.js 22.23.5: PASS (2 phases, 3.11s total)
332 Node.js 24.22.3: PASS (2 phases, 3.09s total)
333 Node.js 26.13.1: PASS (2 phases, 3.08s total)
334
335 Finished: 2026-01-30T12:18:46.500Z
336 Total duration: 4.50s
337 ================================================================================
338 ```
339
340 **Features:**
341
342 - File cleared at start of each test run (no stale data)
343 - Phases grouped by Node version for easy scanning
344 - Visual hierarchy: `===` for file boundaries, `###` for versions, `---` for phases
345 - Empty stdout/stderr shown as `(empty)`
346 - Summary with pass/fail/skip status and timing
347
348 ## Behavior
349
350 - Builds an isolated Docker container and runs the test command inside it
351 - Docker is required
352 - Tests run sequentially per Node version
353 - Volume cleanup is best-effort (doesn't fail tests on cleanup errors)