@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 03:37 UTC

main
md 217 lines 10 KB
Rendered Raw
1 # Scenarios
2
3 A scenario is an [e2e](https://e2e.com/docs) test that shows one behavior in the real app: a real
4 server on its own T3 home, real HTTP and WebSocket, and a real browser on the web client or a
5 pooled iPhone simulator running the mobile app. The
6 dev replay agent stands in for the model, so a run is free and repeatable. Use one as the proof
7 for a bug fix: it fails on the base commit and passes on the fix.
8
9 Scenarios live in [`scripts/scenarios`](../../scripts/scenarios) as `*.e2e.ts`. e2e is the
10 runner; its docs ship in `scripts/node_modules/e2e/docs` and `e2e guide` prints them.
11
12 ## Write one
13
14 ```ts
15 import { expect } from "e2e";
16
17 import { test, WEB } from "./t3.ts";
18
19 test(
20 "Tab moves from the add-project Git URL field to its submit button",
21 WEB,
22 async ({ t3, app, screen, browser }) => {
23 await t3.project("demo");
24 await t3.open();
25
26 await screen.getByRole("button", "Add project").tap();
27 await screen.getByText("Git URL").tap();
28 await screen.getByRole("combobox").fill("https://github.com/pingdotgg/t3code.git");
29 const submit = screen.getByRole("button", "Continue (Enter)");
30 await expect(submit).toBeEnabled();
31
32 await browser.keyboard.press("Tab");
33 await app.screenshot("after-tab");
34 await expect(submit).toBeFocused();
35 },
36 );
37 ```
38
39 `screen`, `browser`, `app`, and `expect` are e2e's own. [`t3.ts`](../../scripts/scenarios/t3.ts)
40 adds one fixture for what e2e cannot know about T3 Code:
41
42 - `t3.project(title, files)` creates a git repository and adds it as a project.
43 - `t3.thread({ project, title, messages })` starts a thread and plays one replayed turn per
44 message before returning, so the thread has real history. `model` picks the recording. It
45 returns the thread, which `t3.open` takes.
46 - `t3.open(path)` opens the app, already signed in, at a path. `t3.open(thread)` opens it on a
47 thread.
48
49 State is created through the server's own CLI and HTTP API, so it is real events and works on a
50 base commit whose database schema differs. Projects go through the API (`/api/projects/mutate`),
51 since each CLI call starts a process that takes about 0.7 s; a base older than that endpoint
52 cannot run a scenario. New threads default to the replay agent's `showcase` recording: type a
53 message and send it. To play something else, add a recording to the
54 [replay agent](development.md#replay-provider).
55
56 Write one test per file: a proof runs the file and needs exactly one test in it.
57
58 Take an `app.screenshot("label")` where the bug shows. The proof pairs base and head captures by
59 label. e2e adds a capture at the point of failure itself.
60
61 Write only `expect` assertions for the thing the bug breaks. A proof counts a failed assertion as
62 the bug. A control that cannot be found, or an app that does not start, counts as a scenario that
63 could not run.
64
65 Keep `agent.act` and `agent.assert` out of scenarios: they call a model, and no model is
66 configured.
67
68 ### Getting to the screen
69
70 Most scenarios that could not run stopped on the way to the screen the bug is on, not on the
71 bug. Three rules avoid that:
72
73 - **Open by link, not by sidebar.** `await t3.open(thread)` lands on the thread
74 ([`thread-opens-by-link.e2e.ts`](../../scripts/scenarios/thread-opens-by-link.e2e.ts)), and
75 `await t3.open("/settings/general")` on a settings page. A sidebar row is a button that covers
76 its own title, so a tap on `getByText("Earlier work")` is intercepted. When the sidebar is what
77 the scenario is about, target the row's button: `screen.getByRole("button", "Earlier work")`.
78 - **Let `t3.open` do the navigating.** It waits until the pairing page has signed in before it
79 goes anywhere. A second `app.open` straight after the pairing link cancels the sign-in and
80 shows the pairing page again.
81 - **Seed before opening.** Create projects and threads first, then call `t3.open`. With no
82 project the app opens its first-run page instead of the one the scenario expects.
83
84 The app starts with only `PATH`, `HOME` and the temp directory in its environment, whatever the
85 shell that ran the scenario has set. Name what it needs in `T3_SCENARIO_INHERIT_ENV`: exact
86 names or prefixes ending in `*`, separated by commas.
87
88 ```sh
89 T3_SCENARIO_INHERIT_ENV='SHELL,MY_FIXTURE_*' vp run scenarios:prove scripts/scenarios/name.e2e.ts --base main
90 ```
91
92 ## Run
93
94 ```sh
95 vp run scenarios # all of them
96 cd scripts && pnpm exec e2e run scenarios/replay-turn-streams.e2e.ts --headed
97 ```
98
99 e2e starts the dev stack through [`start-app.ts`](../../scripts/scenarios/start-app.ts) on a free
100 port, runs the tests, and stops it. Results land in `scripts/.e2e`: `report.json`, `summary.md`,
101 and per test a `trace.md`, its screenshots, and a video. `vp run scenarios --target web` runs one
102 target; flags go straight after the script name, without `--`.
103
104 The first run downloads e2e's browser (about 120 MB). To fetch it ahead of time, such as in a
105 machine image, run this from the repository root after `vp i`:
106
107 ```sh
108 node scripts/node_modules/playwright-core/cli.js install --only-shell chromium ffmpeg
109 ```
110
111 e2e opens the web app at `127.0.0.1`, and the start script tells the web dev server to bind that
112 address. It does not use `localhost`, which some machines resolve to `::1` first.
113
114 `e2e run --help` lists the rest: `--grep`, `--watch`, `--last-failed`, `--changed-since`.
115
116 Run one at a time per checkout: a run recreates the checkout's scenario T3 home and its `.e2e`
117 output. Use a second worktree for a second run.
118
119 ## Prove a fix
120
121 ```sh
122 vp run scenarios:prove scripts/scenarios/add-project-tab-reaches-submit.e2e.ts \
123 --base e509db3a2f^ --head e509db3a2f --upload-command "ghdrop upload"
124 ```
125
126 [`prove-scenario.ts`](../../scripts/prove-scenario.ts) runs `e2e run` for that scenario on both
127 commits and compares the two reports. It prints, and writes to `scripts/.e2e/prove/proof.json`,
128 `failsOnBase`, `passesOnHead`, `baseSha`, `headSha`, `verdict`, each side's `steps`, `captures`,
129 `videos`, and `markdown`: a table with base and head captures side by side, each labelled with
130 its commit, ready for a pull request description. `--upload-command` names a command that takes
131 the capture paths and prints one URL per line. `--head` defaults to `HEAD`.
132
133 It exits 0 only when the scenario fails on base and passes on head, and 1 when it ran and did not
134 prove the fix. A scenario that passes on base does not reproduce the bug, and the verdict says so.
135
136 Two more codes mean there is no verdict on the behavior, and `couldNotRun` in the proof says
137 which:
138
139 - **3, `scenario`:** the app ran and the scenario stopped before its assertion, usually on a
140 control it did not find. Read that side's `message` and `steps`, fix the scenario, and run it
141 again.
142 - **2, `app`:** the app did not start or no test ran: a bad revision, a failed install, a port in
143 use. `scripts/.e2e/app.log` has the app's output.
144
145 Each side's result is also written as it ends, to `scripts/.e2e/prove/base.json` and
146 `head.json`, so a run that is cut short keeps the side it finished.
147
148 It runs the scenario on one target: iOS when the file imports `t3-mobile.ts`, web otherwise.
149 `--target web` or `--target ios` overrides that.
150
151 The scenario and e2e always come from your checkout. Only the app comes from the commit under
152 test: a temporary worktree beside your checkout, installed with `vp i` and removed afterwards (one
153 left by a run that was killed is replaced). A
154 commit that is your clean `HEAD` runs in place. So a new scenario can prove a fix against a base
155 that predates it.
156
157 ## iOS
158
159 An iOS scenario imports from [`t3-mobile.ts`](../../scripts/scenarios/t3-mobile.ts) and passes
160 `IOS` where a web one passes `WEB`:
161
162 ```ts
163 import { expect } from "e2e";
164
165 import { IOS, test } from "./t3-mobile.ts";
166
167 test(
168 "The iOS app lists a seeded thread and opens its history",
169 IOS,
170 async ({ t3, app, screen }) => {
171 const project = await t3.project("demo");
172 await t3.thread({ project, title: "Earlier work", messages: ["first", "second"] });
173 await t3.open();
174
175 const thread = screen.getByText("Earlier work", { exact: false }).first();
176 await expect(thread).toBeVisible({ timeout: 120_000 });
177 await thread.tap();
178 await expect(screen.getByText("second fixture turn complete", { exact: false })).toBeVisible();
179 },
180 );
181 ```
182
183 `t3.project` and `t3.thread` are the same. The run fetches or builds the dev client for the
184 checkout's native fingerprint before it leases a device; `t3.open()` installs it, loads it from
185 Metro, and pairs it with the scenario's server. `device` is
186 e2e's [device fixture](https://e2e.com/docs/mobile#the-device-fixture) for system actions.
187
188 The run leases an iPhone 18 Pro from the Simlock pool, so it needs `SIMLOCK_AGENT_DEVICE_URL`
189 and `AGENT_DEVICE_DAEMON_AUTH_TOKEN`, as `mobile:e2e` does, and a gateway on Simlock 0.7.0 or
190 later. A web-only run leases nothing. The device reaches this machine over the tailnet on ports
191 13990 (server) and 8190 (Metro); set `T3_SCENARIO_SERVER_PORT` and `T3_SCENARIO_METRO_PORT` when
192 another run on the machine holds them. An iOS run takes two to three minutes, most of it the
193 device and the first bundle. `e2e run --target web` or `--target ios` runs one side.
194
195 `scenarios:prove` works the same for an iOS scenario. Each commit gets the dev client built for
196 its own native code.
197
198 ## Other tiers
199
200 - **Android.** Add a target with `mobile({ platform: "android", device: simlock("Pixel 9") })`
201 and open the dev client with the Android form of the Expo link. Not built.
202 - **Live provider.** Point the default model in `start-app.ts` at a real provider's cheapest
203 model. Assertions on exact reply text would not hold.
204
205 ## In CI
206
207 No job runs scenarios, and none should run all of them on every pull request. e2e can select the
208 ones a branch touches with `e2e run --changed-since origin/main` once
209 [test filters](https://e2e.com/docs/filters) are configured, and `@e2e-dev/github` posts the
210 result as a pull request comment.
211
212 ## What a scenario cannot cover
213
214 - Desktop shell behavior (menus, windows, IPC), and Android.
215 - Provider-specific behavior. The replay agent speaks generic ACP.
216 - State no command can create, such as a run stopped by a usage limit.
217 - Timing. Replay streaming is simulated, so a scenario cannot show a performance regression.