Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.
Mirrored from https://github.com/Camo67/composio.git
Synced Nov 21, 04:45 UTC
| 1 | # E2E Test Utilities |
| 2 | |
| 3 | Shared infrastructure for running `@composio/core` and CLI end-to-end tests in isolated Docker environments. |
| 4 | |
| 5 | ## What's Here |
| 6 | |
| 7 | | File/Directory | Purpose | |
| 8 | | ----------------- | ---------------------------------------------------- | |
| 9 | | `src/` | TypeScript utilities (e2e runner, config, types) | |
| 10 | | `scripts/` | Docker build and cleanup scripts | |
| 11 | | `Dockerfile.node` | Multi-stage Dockerfile for Node.js test environments | |
| 12 | | `Dockerfile.cli` | Scratch Dockerfile for CLI test environments | |
| 13 | |
| 14 | ## API |
| 15 | |
| 16 | ### `e2e` |
| 17 | |
| 18 | The main entry point for e2e tests. Automatically infers the working directory and suite name from the caller's location. Uses `bun:test` for the test framework. |
| 19 | |
| 20 | ```typescript |
| 21 | import { e2e, type E2ETestResult } from '@e2e-tests/utils'; |
| 22 | import { TIMEOUTS } from '@e2e-tests/utils/const'; |
| 23 | import { describe, it, expect, beforeAll } from 'bun:test'; |
| 24 | |
| 25 | e2e(import.meta.url, { |
| 26 | versions: { |
| 27 | node: ['22.23.5', '24.22.3', '26.13.1'], // optional, defaults to mise.toml |
| 28 | cli: ['current'], // optional, defaults to CLI package.json version |
| 29 | }, |
| 30 | env: { MY_VAR: 'value' }, // optional env vars |
| 31 | defineTests: ({ runtime, runCmd, runFixture }) => { |
| 32 | let result: E2ETestResult; |
| 33 | |
| 34 | beforeAll(async () => { |
| 35 | result = await runFixture({ filename: 'fixtures/test.mjs' }); |
| 36 | }, TIMEOUTS.FIXTURE); |
| 37 | |
| 38 | describe('output', () => { |
| 39 | it('exits successfully', () => { |
| 40 | expect(result.exitCode).toBe(0); |
| 41 | }); |
| 42 | }); |
| 43 | }, |
| 44 | }); |
| 45 | ``` |
| 46 | |
| 47 | ### `E2EConfig` |
| 48 | |
| 49 | Configuration object passed to `e2e()`: |
| 50 | |
| 51 | | Property | Type | Description | |
| 52 | | -------------- | ------------------------------------- | ------------------------------------------------------------- | |
| 53 | | `versions` | `RuntimeVersions` | Runtime versions to test. See RuntimeVersions below | |
| 54 | | `env` | `Record<string, string \| undefined>` | Environment variables for Docker. Validated at startup | |
| 55 | | `usesFixtures` | `boolean` | When true, sets cwd to `{testDir}/fixtures`. Default: `false` | |
| 56 | | `defineTests` | `(ctx: DefineTestsContext) => void` | Callback to define tests using bun:test primitives | |
| 57 | |
| 58 | ### `RuntimeVersions` |
| 59 | |
| 60 | | Property | Type | Description | |
| 61 | | -------- | -------------------------------- | ------------------------------------------ | |
| 62 | | `node` | `readonly NodeVersionFromUser[]` | Node.js versions. Defaults to `mise.toml` | |
| 63 | | `cli` | `readonly CliVersionFromUser[]` | CLI versions. Defaults to CLI package.json | |
| 64 | |
| 65 | ### `DefineTestsContext` |
| 66 | |
| 67 | The context passed to the `defineTests` callback: |
| 68 | |
| 69 | | Property | Type/Signature | Description | |
| 70 | | ------------ | ---------------------------------------------------------------------------------- | ----------------------------------------- | |
| 71 | | `runtime` | `'node' \| 'cli'` | Current runtime being tested | |
| 72 | | `runCmd` | `(command: string) => Promise<E2ETestResult>` | Run arbitrary command in Docker container | |
| 73 | | `runFixture` | `(options: RunFixtureOptions) => Promise<E2ETestResult \| E2ETestResultWithSetup>` | Run fixture with optional setup phase | |
| 74 | |
| 75 | ### `RunFixtureOptions` |
| 76 | |
| 77 | Options for `runFixture()`: |
| 78 | |
| 79 | | Property | Type | Description | |
| 80 | | ---------- | -------- | ------------------------------------------------------------------- | |
| 81 | | `filename` | `string` | Fixture file path relative to cwd (e.g., `'index.mjs'`) | |
| 82 | | `setup` | `string` | Optional setup command (e.g., `'npm install'`). Enables volume mode | |
| 83 | |
| 84 | **Behavior:** |
| 85 | |
| 86 | - **Without `setup`**: Runs `node <filename>` directly. Returns `E2ETestResult`. |
| 87 | - **With `setup`**: Creates a Docker volume, runs setup with volume mounted read-write, then runs fixture with volume mounted read-only. Returns `E2ETestResultWithSetup`. |
| 88 | |
| 89 | ```typescript |
| 90 | // Simple fixture (no dependencies to install) |
| 91 | const result = await runFixture({ filename: 'test.mjs' }); |
| 92 | |
| 93 | // Fixture with setup phase (uses Docker volumes) |
| 94 | const result = await runFixture({ |
| 95 | filename: 'index.mjs', |
| 96 | setup: 'npm install --legacy-peer-deps', |
| 97 | }); |
| 98 | expect(result.setup.exitCode).toBe(0); // Check setup phase |
| 99 | expect(result.exitCode).toBe(0); // Check fixture phase |
| 100 | ``` |
| 101 | |
| 102 | ### `E2ETestResult` |
| 103 | |
| 104 | Result returned by `runCmd` and `runFixture`: |
| 105 | |
| 106 | ```typescript |
| 107 | interface E2ETestResult { |
| 108 | exitCode: number; // Exit code from the command (0 = success) |
| 109 | stdout: string; // Captured stdout |
| 110 | stderr: string; // Captured stderr |
| 111 | } |
| 112 | ``` |
| 113 | |
| 114 | ### `runCmd` with File Capture |
| 115 | |
| 116 | When you need to assert on files created inside the container, pass a `files` array: |
| 117 | |
| 118 | ```typescript |
| 119 | const result = await runCmd({ |
| 120 | command: 'composio version > out.txt', |
| 121 | files: ['out.txt'], |
| 122 | }); |
| 123 | |
| 124 | expect(result.files['out.txt']).toBe('0.1.24'); |
| 125 | ``` |
| 126 | |
| 127 | The files are copied out of the container after execution and returned as a map in the result. |
| 128 | |
| 129 | ### `startJourneyContainer` |
| 130 | |
| 131 | One CLI container that stays up while a test runs several commands in it, so a login or a cache file written by one command is there for the next. Unlike `runCmd`, it mounts no host `~/.composio` and passes through no API key. |
| 132 | |
| 133 | ```typescript |
| 134 | import { startJourneyContainer } from '@e2e-tests/utils'; |
| 135 | |
| 136 | const journey = await startJourneyContainer({ env: { COMPOSIO_BASE_URL: mockUrl } }); |
| 137 | const first = await journey.exec('composio login --user-api-key uk_test'); |
| 138 | const second = await journey.exec('composio whoami', { timeoutSeconds: 30 }); |
| 139 | await journey.writeFile('/work/app/.composio/project.json', '{}'); |
| 140 | await journey.stop(); |
| 141 | ``` |
| 142 | |
| 143 | | Member | Description | |
| 144 | | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |
| 145 | | `exec(command, options?)` | Runs a POSIX `sh` command line. Options: `env`, `stdin`, `cwd` (defaults to the home), `timeoutSeconds` (default 60; exit code 124 on expiry), `user` (`uid:gid`, to run without root) | |
| 146 | | `writeFile(path, content)` / `readFile(path)` | Write or read a file inside the container | |
| 147 | | `steps()` / `writeDebugLog(path)` | The commands run so far with their results | |
| 148 | | `home` | The home directory (default `/home/journey`), created empty when the journey starts | |
| 149 | | `stop()` | Removes the container | |
| 150 | |
| 151 | ### `E2ETestResultWithSetup` |
| 152 | |
| 153 | Extended result when `runFixture` is called with a `setup` option: |
| 154 | |
| 155 | ```typescript |
| 156 | interface E2ETestResultWithSetup extends E2ETestResult { |
| 157 | setup: { |
| 158 | exitCode: number; |
| 159 | stdout: string; |
| 160 | stderr: string; |
| 161 | }; |
| 162 | } |
| 163 | ``` |
| 164 | |
| 165 | Top-level fields (`exitCode`, `stdout`, `stderr`) reflect the fixture result. The `setup` object contains the setup command result. |
| 166 | |
| 167 | ### `sanitizeOutput` |
| 168 | |
| 169 | Utility for stable test comparisons. Removes ANSI escape codes, normalizes line endings, and trims whitespace. |
| 170 | |
| 171 | ```typescript |
| 172 | import { sanitizeOutput } from '@e2e-tests/utils'; |
| 173 | |
| 174 | const clean = sanitizeOutput(result.stdout); |
| 175 | ``` |
| 176 | |
| 177 | ### `TIMEOUTS` |
| 178 | |
| 179 | Predefined timeout constants for tests (in milliseconds): |
| 180 | |
| 181 | ```typescript |
| 182 | import { TIMEOUTS } from '@e2e-tests/utils/const'; |
| 183 | |
| 184 | it( |
| 185 | 'calls LLM', |
| 186 | async () => { |
| 187 | // test code |
| 188 | }, |
| 189 | { timeout: TIMEOUTS.LLM_SHORT } |
| 190 | ); |
| 191 | ``` |
| 192 | |
| 193 | | Constant | Value | Use Case | |
| 194 | | ----------- | --------- | ------------------------------------------ | |
| 195 | | `DEFAULT` | `5_000` | Standard test operations | |
| 196 | | `FIXTURE` | `120_000` | `beforeAll` hooks that call `runFixture()` | |
| 197 | | `LLM_SHORT` | `30_000` | Quick LLM calls | |
| 198 | | `LLM_LONG` | `60_000` | Complex LLM operations | |
| 199 | |
| 200 | ## Version Resolution |
| 201 | |
| 202 | ### Node.js Version Resolution |
| 203 | |
| 204 | Node.js versions to test are resolved in this order: |
| 205 | |
| 206 | 1. **`COMPOSIO_E2E_NODE_VERSION` env var** (highest priority): Use `[env_value]` |
| 207 | 2. **`config.versions.node`**: Use the provided array |
| 208 | 3. **Default**: Use version from `mise.toml` file |
| 209 | |
| 210 | ### Well-Known Node Versions |
| 211 | |
| 212 | The following versions are pre-defined in `const.ts`: |
| 213 | |
| 214 | - `22.23.5` |
| 215 | - `24.22.3` |
| 216 | - `26.13.1` |
| 217 | - `current` (resolves to `mise.toml` version) |
| 218 | |
| 219 | ### CLI image freshness |
| 220 | |
| 221 | Local CLI suites run `docker build` before using an image. Docker checks the current build inputs against its content cache, including uncommitted source and local build outputs. The CLI version tag alone does not establish freshness. |
| 222 | |
| 223 | GitHub Actions can reuse its prebuilt image when the `org.opencontainers.image.revision` label matches the checkout's commit. A missing or different label triggers a rebuild. A failed rebuild fails the test setup instead of accepting an existing tag. |
| 224 | |
| 225 | ### CLI Version Resolution |
| 226 | |
| 227 | CLI versions to test are resolved in this order: |
| 228 | |
| 229 | 1. **`COMPOSIO_E2E_CLI_VERSION` env var** (highest priority): Use `[env_value]` |
| 230 | 2. **`config.versions.cli`**: Use the provided array |
| 231 | 3. **Default**: Use version from `ts/packages/cli/package.json` |
| 232 | |
| 233 | ### Well-Known CLI Versions |
| 234 | |
| 235 | - `current` (resolves to CLI package.json version) |
| 236 | |
| 237 | ## Environment Variable Validation |
| 238 | |
| 239 | Environment variables passed to `E2EConfig.env` are validated at test startup. If any variable has an `undefined` value, the test fails fast with a clear error message: |
| 240 | |
| 241 | ``` |
| 242 | [my-test] Missing required environment variables: COMPOSIO_API_KEY, OPENAI_API_KEY |
| 243 | Set these variables before running the tests, or remove them from E2EConfig.env if not required. |
| 244 | ``` |
| 245 | |
| 246 | This prevents silent failures from missing credentials. |
| 247 | |
| 248 | ## The `usesFixtures` Option |
| 249 | |
| 250 | When `usesFixtures: true` is set: |
| 251 | |
| 252 | - Working directory changes to `{testDir}/fixtures/` |
| 253 | - Docker volume mounts at `fixtures/node_modules` |
| 254 | - Fixture paths in `runFixture({ filename })` are relative to `fixtures/` |
| 255 | |
| 256 | Use this for tests that have their own `package.json` and need to run `npm install`: |
| 257 | |
| 258 | ```typescript |
| 259 | import { TIMEOUTS } from '@e2e-tests/utils/const'; |
| 260 | |
| 261 | e2e(import.meta.url, { |
| 262 | usesFixtures: true, |
| 263 | defineTests: ({ runFixture }) => { |
| 264 | beforeAll(async () => { |
| 265 | // Both commands run in fixtures/ directory |
| 266 | result = await runFixture({ |
| 267 | filename: 'index.mjs', // Resolves to fixtures/index.mjs |
| 268 | setup: 'npm install', // Runs in fixtures/ |
| 269 | }); |
| 270 | }, TIMEOUTS.FIXTURE); |
| 271 | }, |
| 272 | }); |
| 273 | ``` |
| 274 | |
| 275 | ## Scripts |
| 276 | |
| 277 | ```bash |
| 278 | # Pre-build Docker images for all well-known Node and CLI versions |
| 279 | pnpm docker:build |
| 280 | |
| 281 | # Remove all e2e Docker images (Node.js and CLI) |
| 282 | pnpm docker:clean |
| 283 | ``` |
| 284 | |
| 285 | ## DEBUG.log Output |
| 286 | |
| 287 | Each test suite generates a `DEBUG.log` file with structured output grouped by runtime version: |
| 288 | |
| 289 | ``` |
| 290 | ================================================================================ |
| 291 | E2E Test: openai-zod4-compat |
| 292 | Started: 2026-01-30T12:18:42.000Z |
| 293 | Test file: ts/e2e-tests/runtimes/node/openai-zod4-compat/e2e.test.ts |
| 294 | Runtime versions: Node.js 22.23.5, Node.js 24.22.3, Node.js 26.13.1 |
| 295 | ================================================================================ |
| 296 | |
| 297 | ################################################################################ |
| 298 | ### Node.js 22.23.5 |
| 299 | ################################################################################ |
| 300 | Image: composio-e2e-node:22.23.5 |
| 301 | |
| 302 | --- Phase 1/2: setup --- |
| 303 | Container: e2e-openai-zod4-compat-22-22-3-1769775520382-setup |
| 304 | Command: npm install --legacy-peer-deps |
| 305 | Duration: 2.55s |
| 306 | Exit Code: 0 (success) |
| 307 | |
| 308 | [stdout] |
| 309 | added 3 packages, and audited 5 packages in 2s |
| 310 | |
| 311 | [stderr] |
| 312 | (empty) |
| 313 | |
| 314 | --- Phase 2/2: fixture --- |
| 315 | Container: e2e-openai-zod4-compat-22-22-3-1769775520382-fixture |
| 316 | Command: node index.mjs |
| 317 | Duration: 0.56s |
| 318 | Exit Code: 0 (success) |
| 319 | |
| 320 | [stdout] |
| 321 | zod@4 works |
| 322 | openai@5 works |
| 323 | All packages work together! |
| 324 | |
| 325 | [stderr] |
| 326 | (empty) |
| 327 | |
| 328 | ================================================================================ |
| 329 | Summary |
| 330 | ================================================================================ |
| 331 | Node.js 22.23.5: PASS (2 phases, 3.11s total) |
| 332 | Node.js 24.22.3: PASS (2 phases, 3.09s total) |
| 333 | Node.js 26.13.1: PASS (2 phases, 3.08s total) |
| 334 | |
| 335 | Finished: 2026-01-30T12:18:46.500Z |
| 336 | Total duration: 4.50s |
| 337 | ================================================================================ |
| 338 | ``` |
| 339 | |
| 340 | **Features:** |
| 341 | |
| 342 | - File cleared at start of each test run (no stale data) |
| 343 | - Phases grouped by Node version for easy scanning |
| 344 | - Visual hierarchy: `===` for file boundaries, `###` for versions, `---` for phases |
| 345 | - Empty stdout/stderr shown as `(empty)` |
| 346 | - Summary with pass/fail/skip status and timing |
| 347 | |
| 348 | ## Behavior |
| 349 | |
| 350 | - Builds an isolated Docker container and runs the test command inside it |
| 351 | - Docker is required |
| 352 | - Tests run sequentially per Node version |
| 353 | - Volume cleanup is best-effort (doesn't fail tests on cleanup errors) |