The best way to control Claude Code, Codex and more from any computer or phone
| 1 | # Scenarios |
| 2 | |
| 3 | A scenario is an [e2e](https://e2e.com/docs) test that shows one behavior in the real app: a real |
| 4 | server on its own T3 home, real HTTP and WebSocket, and a real browser on the web client or a |
| 5 | pooled iPhone simulator running the mobile app. The |
| 6 | dev replay agent stands in for the model, so a run is free and repeatable. Use one as the proof |
| 7 | for a bug fix: it fails on the base commit and passes on the fix. |
| 8 | |
| 9 | Scenarios live in [`scripts/scenarios`](../../scripts/scenarios) as `*.e2e.ts`. e2e is the |
| 10 | runner; its docs ship in `scripts/node_modules/e2e/docs` and `e2e guide` prints them. |
| 11 | |
| 12 | ## Write one |
| 13 | |
| 14 | ```ts |
| 15 | import { expect } from "e2e"; |
| 16 | |
| 17 | import { test, WEB } from "./t3.ts"; |
| 18 | |
| 19 | test( |
| 20 | "Tab moves from the add-project Git URL field to its submit button", |
| 21 | WEB, |
| 22 | async ({ t3, app, screen, browser }) => { |
| 23 | await t3.project("demo"); |
| 24 | await t3.open(); |
| 25 | |
| 26 | await screen.getByRole("button", "Add project").tap(); |
| 27 | await screen.getByText("Git URL").tap(); |
| 28 | await screen.getByRole("combobox").fill("https://github.com/pingdotgg/t3code.git"); |
| 29 | const submit = screen.getByRole("button", "Continue (Enter)"); |
| 30 | await expect(submit).toBeEnabled(); |
| 31 | |
| 32 | await browser.keyboard.press("Tab"); |
| 33 | await app.screenshot("after-tab"); |
| 34 | await expect(submit).toBeFocused(); |
| 35 | }, |
| 36 | ); |
| 37 | ``` |
| 38 | |
| 39 | `screen`, `browser`, `app`, and `expect` are e2e's own. [`t3.ts`](../../scripts/scenarios/t3.ts) |
| 40 | adds one fixture for what e2e cannot know about T3 Code: |
| 41 | |
| 42 | - `t3.project(title, files)` creates a git repository and adds it as a project. |
| 43 | - `t3.thread({ project, title, messages })` starts a thread and plays one replayed turn per |
| 44 | message before returning, so the thread has real history. `model` picks the recording. It |
| 45 | returns the thread, which `t3.open` takes. |
| 46 | - `t3.open(path)` opens the app, already signed in, at a path. `t3.open(thread)` opens it on a |
| 47 | thread. |
| 48 | |
| 49 | State is created through the server's own CLI and HTTP API, so it is real events and works on a |
| 50 | base commit whose database schema differs. Projects go through the API (`/api/projects/mutate`), |
| 51 | since each CLI call starts a process that takes about 0.7 s; a base older than that endpoint |
| 52 | cannot run a scenario. New threads default to the replay agent's `showcase` recording: type a |
| 53 | message and send it. To play something else, add a recording to the |
| 54 | [replay agent](development.md#replay-provider). |
| 55 | |
| 56 | Write one test per file: a proof runs the file and needs exactly one test in it. |
| 57 | |
| 58 | Take an `app.screenshot("label")` where the bug shows. The proof pairs base and head captures by |
| 59 | label. e2e adds a capture at the point of failure itself. |
| 60 | |
| 61 | Write only `expect` assertions for the thing the bug breaks. A proof counts a failed assertion as |
| 62 | the bug. A control that cannot be found, or an app that does not start, counts as a scenario that |
| 63 | could not run. |
| 64 | |
| 65 | Keep `agent.act` and `agent.assert` out of scenarios: they call a model, and no model is |
| 66 | configured. |
| 67 | |
| 68 | ### Getting to the screen |
| 69 | |
| 70 | Most scenarios that could not run stopped on the way to the screen the bug is on, not on the |
| 71 | bug. Three rules avoid that: |
| 72 | |
| 73 | - **Open by link, not by sidebar.** `await t3.open(thread)` lands on the thread |
| 74 | ([`thread-opens-by-link.e2e.ts`](../../scripts/scenarios/thread-opens-by-link.e2e.ts)), and |
| 75 | `await t3.open("/settings/general")` on a settings page. A sidebar row is a button that covers |
| 76 | its own title, so a tap on `getByText("Earlier work")` is intercepted. When the sidebar is what |
| 77 | the scenario is about, target the row's button: `screen.getByRole("button", "Earlier work")`. |
| 78 | - **Let `t3.open` do the navigating.** It waits until the pairing page has signed in before it |
| 79 | goes anywhere. A second `app.open` straight after the pairing link cancels the sign-in and |
| 80 | shows the pairing page again. |
| 81 | - **Seed before opening.** Create projects and threads first, then call `t3.open`. With no |
| 82 | project the app opens its first-run page instead of the one the scenario expects. |
| 83 | |
| 84 | The app starts with only `PATH`, `HOME` and the temp directory in its environment, whatever the |
| 85 | shell that ran the scenario has set. Name what it needs in `T3_SCENARIO_INHERIT_ENV`: exact |
| 86 | names or prefixes ending in `*`, separated by commas. |
| 87 | |
| 88 | ```sh |
| 89 | T3_SCENARIO_INHERIT_ENV='SHELL,MY_FIXTURE_*' vp run scenarios:prove scripts/scenarios/name.e2e.ts --base main |
| 90 | ``` |
| 91 | |
| 92 | ## Run |
| 93 | |
| 94 | ```sh |
| 95 | vp run scenarios # all of them |
| 96 | cd scripts && pnpm exec e2e run scenarios/replay-turn-streams.e2e.ts --headed |
| 97 | ``` |
| 98 | |
| 99 | e2e starts the dev stack through [`start-app.ts`](../../scripts/scenarios/start-app.ts) on a free |
| 100 | port, runs the tests, and stops it. Results land in `scripts/.e2e`: `report.json`, `summary.md`, |
| 101 | and per test a `trace.md`, its screenshots, and a video. `vp run scenarios --target web` runs one |
| 102 | target; flags go straight after the script name, without `--`. |
| 103 | |
| 104 | The first run downloads e2e's browser (about 120 MB). To fetch it ahead of time, such as in a |
| 105 | machine image, run this from the repository root after `vp i`: |
| 106 | |
| 107 | ```sh |
| 108 | node scripts/node_modules/playwright-core/cli.js install --only-shell chromium ffmpeg |
| 109 | ``` |
| 110 | |
| 111 | e2e opens the web app at `127.0.0.1`, and the start script tells the web dev server to bind that |
| 112 | address. It does not use `localhost`, which some machines resolve to `::1` first. |
| 113 | |
| 114 | `e2e run --help` lists the rest: `--grep`, `--watch`, `--last-failed`, `--changed-since`. |
| 115 | |
| 116 | Run one at a time per checkout: a run recreates the checkout's scenario T3 home and its `.e2e` |
| 117 | output. Use a second worktree for a second run. |
| 118 | |
| 119 | ## Prove a fix |
| 120 | |
| 121 | ```sh |
| 122 | vp run scenarios:prove scripts/scenarios/add-project-tab-reaches-submit.e2e.ts \ |
| 123 | --base e509db3a2f^ --head e509db3a2f --upload-command "ghdrop upload" |
| 124 | ``` |
| 125 | |
| 126 | [`prove-scenario.ts`](../../scripts/prove-scenario.ts) runs `e2e run` for that scenario on both |
| 127 | commits and compares the two reports. It prints, and writes to `scripts/.e2e/prove/proof.json`, |
| 128 | `failsOnBase`, `passesOnHead`, `baseSha`, `headSha`, `verdict`, each side's `steps`, `captures`, |
| 129 | `videos`, and `markdown`: a table with base and head captures side by side, each labelled with |
| 130 | its commit, ready for a pull request description. `--upload-command` names a command that takes |
| 131 | the capture paths and prints one URL per line. `--head` defaults to `HEAD`. |
| 132 | |
| 133 | It exits 0 only when the scenario fails on base and passes on head, and 1 when it ran and did not |
| 134 | prove the fix. A scenario that passes on base does not reproduce the bug, and the verdict says so. |
| 135 | |
| 136 | Two more codes mean there is no verdict on the behavior, and `couldNotRun` in the proof says |
| 137 | which: |
| 138 | |
| 139 | - **3, `scenario`:** the app ran and the scenario stopped before its assertion, usually on a |
| 140 | control it did not find. Read that side's `message` and `steps`, fix the scenario, and run it |
| 141 | again. |
| 142 | - **2, `app`:** the app did not start or no test ran: a bad revision, a failed install, a port in |
| 143 | use. `scripts/.e2e/app.log` has the app's output. |
| 144 | |
| 145 | Each side's result is also written as it ends, to `scripts/.e2e/prove/base.json` and |
| 146 | `head.json`, so a run that is cut short keeps the side it finished. |
| 147 | |
| 148 | It runs the scenario on one target: iOS when the file imports `t3-mobile.ts`, web otherwise. |
| 149 | `--target web` or `--target ios` overrides that. |
| 150 | |
| 151 | The scenario and e2e always come from your checkout. Only the app comes from the commit under |
| 152 | test: a temporary worktree beside your checkout, installed with `vp i` and removed afterwards (one |
| 153 | left by a run that was killed is replaced). A |
| 154 | commit that is your clean `HEAD` runs in place. So a new scenario can prove a fix against a base |
| 155 | that predates it. |
| 156 | |
| 157 | ## iOS |
| 158 | |
| 159 | An iOS scenario imports from [`t3-mobile.ts`](../../scripts/scenarios/t3-mobile.ts) and passes |
| 160 | `IOS` where a web one passes `WEB`: |
| 161 | |
| 162 | ```ts |
| 163 | import { expect } from "e2e"; |
| 164 | |
| 165 | import { IOS, test } from "./t3-mobile.ts"; |
| 166 | |
| 167 | test( |
| 168 | "The iOS app lists a seeded thread and opens its history", |
| 169 | IOS, |
| 170 | async ({ t3, app, screen }) => { |
| 171 | const project = await t3.project("demo"); |
| 172 | await t3.thread({ project, title: "Earlier work", messages: ["first", "second"] }); |
| 173 | await t3.open(); |
| 174 | |
| 175 | const thread = screen.getByText("Earlier work", { exact: false }).first(); |
| 176 | await expect(thread).toBeVisible({ timeout: 120_000 }); |
| 177 | await thread.tap(); |
| 178 | await expect(screen.getByText("second fixture turn complete", { exact: false })).toBeVisible(); |
| 179 | }, |
| 180 | ); |
| 181 | ``` |
| 182 | |
| 183 | `t3.project` and `t3.thread` are the same. The run fetches or builds the dev client for the |
| 184 | checkout's native fingerprint before it leases a device; `t3.open()` installs it, loads it from |
| 185 | Metro, and pairs it with the scenario's server. `device` is |
| 186 | e2e's [device fixture](https://e2e.com/docs/mobile#the-device-fixture) for system actions. |
| 187 | |
| 188 | The run leases an iPhone 18 Pro from the Simlock pool, so it needs `SIMLOCK_AGENT_DEVICE_URL` |
| 189 | and `AGENT_DEVICE_DAEMON_AUTH_TOKEN`, as `mobile:e2e` does, and a gateway on Simlock 0.7.0 or |
| 190 | later. A web-only run leases nothing. The device reaches this machine over the tailnet on ports |
| 191 | 13990 (server) and 8190 (Metro); set `T3_SCENARIO_SERVER_PORT` and `T3_SCENARIO_METRO_PORT` when |
| 192 | another run on the machine holds them. An iOS run takes two to three minutes, most of it the |
| 193 | device and the first bundle. `e2e run --target web` or `--target ios` runs one side. |
| 194 | |
| 195 | `scenarios:prove` works the same for an iOS scenario. Each commit gets the dev client built for |
| 196 | its own native code. |
| 197 | |
| 198 | ## Other tiers |
| 199 | |
| 200 | - **Android.** Add a target with `mobile({ platform: "android", device: simlock("Pixel 9") })` |
| 201 | and open the dev client with the Android form of the Expo link. Not built. |
| 202 | - **Live provider.** Point the default model in `start-app.ts` at a real provider's cheapest |
| 203 | model. Assertions on exact reply text would not hold. |
| 204 | |
| 205 | ## In CI |
| 206 | |
| 207 | No job runs scenarios, and none should run all of them on every pull request. e2e can select the |
| 208 | ones a branch touches with `e2e run --changed-since origin/main` once |
| 209 | [test filters](https://e2e.com/docs/filters) are configured, and `@e2e-dev/github` posts the |
| 210 | result as a pull request comment. |
| 211 | |
| 212 | ## What a scenario cannot cover |
| 213 | |
| 214 | - Desktop shell behavior (menus, windows, IPC), and Android. |
| 215 | - Provider-specific behavior. The replay agent speaks generic ACP. |
| 216 | - State no command can create, such as a run stopped by a usage limit. |
| 217 | - Timing. Replay streaming is simulated, so a scenario cannot show a performance regression. |