The best way to control Claude Code, Codex and more from any computer or phone

main
md 195 lines 9.92 KB
Rendered Raw
1 # cross-version-test
2
3 Deploys a tiny app — a Cloudflare **state store** + one **Worker** — across a
4 range of alchemy versions, **upgrading the same app in place**, to catch
5 regressions in the version-to-version upgrade path.
6
7 `run.ts` runs two tests **back to back** (sequentially — they share one
8 account-wide state store and one fixed worker name, so they can never overlap):
9
10 1. **`upgrade` — sequential 1-by-1 upgrade.** Deploy the oldest version, then
11 upgrade the same app through every version in order
12 (v4 → v5 → v6 → v7 → current), walking the state store up one version at a
13 time.
14 2. **`jump` — direct-to-latest.** For each older example, deploy it and then
15 upgrade the same app **directly to latest** (the current branch), skipping
16 the intermediate versions — a big-jump upgrade (e.g. state store v4 → v7 in
17 one step).
18
19 Both assert, after every deploy, that the live worker serves the `marker` baked
20 into the version just deployed — proving the running code was actually replaced
21 in place. Each also exercises the **state-store upgrade** path: alchemy's
22 Cloudflare state store is an account-wide, version-stamped singleton that
23 `alchemy provider cloudflare bootstrap` migrates between versions.
24
25 **Fresh state store per scenario.** Each scenario ("unit" — the whole `upgrade`
26 chain is one unit; each `jump` is its own unit) owns a brand-new state store:
27 the runner tears the store (worker + secrets) down before and after every unit.
28 So the `upgrade` chain runs on its own store, that store is destroyed, then each
29 `jump` deploys a fresh store. Pass `--reuse-store` to skip the teardowns and
30 share one store across units (faster, less isolated).
31
32 ## Layout
33
34 ```
35 cross-version-test/
36 run.ts # orchestrator — runs the stages in order
37 test/
38 01-beta.39/ # alchemy@2.0.0-beta.39 (last npm release with state store v4)
39 02-beta.44/ # alchemy@2.0.0-beta.44 (last npm release with state store v5)
40 03-beta.45/ # alchemy@2.0.0-beta.45 (last npm release with state store v6)
41 04-beta.59/ # alchemy@2.0.0-beta.59 (latest v2 on npm, state store v7)
42 05-current/ # current branch (workspace source, state store v7)
43 ```
44
45 Each stage folder is a standalone alchemy app: `alchemy.run.ts` (identical
46 across stages) + `src/worker.ts` (differs only by a `marker` and, for older
47 versions, `main: import.meta.filename` vs HEAD's `import.meta.url`) +
48 `test/integ.test.ts` (an example-style integration test).
49
50 - **npm stages** (`01`, `02`) pin an exact `alchemy` version **and a matching
51 `effect` version** in `package.json`, and get their own isolated
52 `node_modules` via `bun install`. The `effect` pin matters: alchemy floats on
53 the `effect` 4.0 beta line, so an old alchemy paired with a too-new `effect`
54 crashes (e.g. beta.45 breaks on `effect` ≥ beta.84 via a `SchemaAST` change).
55 Each stage therefore pins the `effect` version contemporaneous with its
56 alchemy — its peer-dependency floor (beta.39 & beta.44 → `effect@4.0.0-beta.66`;
57 beta.45 → `beta.74`; beta.59 → `beta.84`).
58 - **workspace stage** (`05-current`) has no dependencies — it resolves
59 `alchemy`/`effect` from the monorepo (repo-root `node_modules`) so it runs the
60 **current branch source**.
61
62 All stages share one identity — `Stack("CrossVersionApp")`, a fixed worker
63 `name`, the same account — which is what makes each deploy an in-place upgrade
64 rather than a new app. (TEST 2 uses the stage `<stage>-jump` to keep its state
65 separate from TEST 1's.)
66
67 ## Running
68
69 ```sh
70 # from the repo root — runs BOTH tests back to back
71 bun run test/cross-version-test/run.ts --profile <cloudflare-profile>
72 # or
73 ALCHEMY_PROFILE=<cloudflare-profile> bun run test/cross-version-test/run.ts
74 # run just one test
75 bun run test/cross-version-test/run.ts --profile <p> --test upgrade
76 bun run test/cross-version-test/run.ts --profile <p> --test jump
77 ```
78
79 ### Flags
80
81 | Flag | Default | Meaning |
82 | ---------------- | ------------------ | ------------------------------------------------------------- |
83 | `--profile <p>` | `$ALCHEMY_PROFILE` | Cloudflare auth profile (`~/.alchemy/profiles.json`). Required. |
84 | `--test <names>` | `upgrade,jump` | Comma-separated tests to run: `upgrade`, `jump`. |
85 | `--stage <s>` | `xver` | Base alchemy stage name (TEST 2 uses `<stage>-jump`). |
86 | `--only <dirs>` | all | Comma-separated stage dirs (e.g. `01-beta.39,02-beta.44`). |
87 | `--no-install` | off | Skip `bun install` in npm stages (reuse existing installs). |
88 | `--keep` | off | Leave the final app + state store deployed (skip the last unit's teardown). |
89 | `--reuse-store` | off | Reuse one state store across units (faster, less isolated) instead of a fresh one per unit. |
90
91 ## Per-version integration tests
92
93 Each stage folder also has a standalone `test/integ.test.ts` (mirrors
94 `examples/*/test/integ.test.ts`): it deploys the stack in `beforeAll`, makes an
95 HTTP request to the worker and asserts the live `marker`, and destroys in
96 `afterAll`. Run one from its folder:
97
98 ```sh
99 cd test/cross-version-test/test/05-current
100 ALCHEMY_PROFILE=<profile> bun test # bun test buffers — no live output until it finishes
101 ```
102
103 These run on a dedicated stage (`integ`) so they don't touch the orchestrator's
104 `xver` state. They deploy through the **account-wide state store**, so it must
105 already be at the version matching that folder's alchemy (**v4** beta.39, **v5**
106 beta.44, **v6** beta.45, **v7** beta.59/current). Either run `run.ts` (which
107 bootstraps per stage), or bootstrap first:
108
109 ```sh
110 cd test/cross-version-test/test/01-beta.39
111 bun run alc -- cloudflare bootstrap --profile <profile> # brings the store to v4
112 bun test
113 ```
114
115 ## Tearing down the state store
116
117 `run.ts` tears the state store down automatically between units (unless
118 `--reuse-store`), and the last unit's store is removed at the end too (unless
119 `--keep`). So a normal run leaves the account clean.
120
121 To remove the state store manually — the `alchemy-state-store` worker, the
122 bearer-token + encryption-key secrets, and the now-empty Secrets Store (e.g.
123 after a `--keep` run or an aborted run):
124
125 ```sh
126 cd test/cross-version-test/test/05-current # uses the current-branch CLI
127 bun run alc -- provider cloudflare teardown --profile <profile>
128 ```
129
130 `provider cloudflare teardown` is the inverse of
131 `provider cloudflare bootstrap`. It's idempotent and only removes resources
132 Alchemy created. It leaves a Secrets Store in place if it still contains
133 foreign secrets.
134
135 ## ⚠️ Use a dedicated Cloudflare account
136
137 The Cloudflare state store is **account-wide** and shared by every stack on the
138 account/profile. Stage `01` bootstraps state store **v4**, which **downgrades**
139 the store if the account already had a higher version — that can break other stacks using
140 the same account. **Point this at a dedicated/throwaway Cloudflare account.**
141
142 The runner tears the state store down between units and at the end, so a normal
143 run leaves the account clean.
144
145 ## Known-bad upgrade paths
146
147 Two upgrade paths are **known broken** and are therefore **not tested**
148 (commented out of `EDGES` in `run.ts`). Both are deterministic and reproduced
149 across runs:
150
151 | Path | Status | Why |
152 | --- | --- | --- |
153 | v4 → v5 | ✅ works | |
154 | **v5 → v6** | ❌ **known bad** | **v6 (beta.45) can't read pre-v6 state** — `500 GET DecodeError` |
155 | v6 → v7 | ✅ works | |
156 | v5/v6/v7 → worktree | ✅ works | |
157 | **v4 → worktree** | ❌ **known bad** | **a v4 store can't be upgraded to current** — `415` on write |
158
159 Details:
160
161 - **`v5 → v6` (beta.44 → beta.45):** after the store is upgraded to v6, reading
162 a record written under ≤v5 fails with `500 GET … DecodeError`. beta.45's
163 legacy-record read (createdAt/updatedAt reshape, PR #427) is broken; **fixed
164 in v7** — `v5 → worktree` (skipping beta.45) works. Don't step pre-v6 state
165 through beta.45.
166 - **`v4 → worktree` (beta.39 → current):** the current state-store client can't
167 write to a v4-format store — the upgrade fails with `415 Unsupported Media
168 Type` on `PUT …/StateStoreEncryptionKey`. v4 (beta.37–39) predates the RPC
169 state-store rewrite at v5, so its store HTTP API is wire-incompatible with
170 current. Step a v4 store up to ≥v5 first (`v5/v6/v7 → worktree` all work).
171
172 Everything else passes. Transient `404/500` while bootstrapping a freshly
173 (re)deployed state-store worker is expected (the worker isn't serving yet when
174 bootstrap hoists into it) and is absorbed by the per-step retry.
175
176 ## Adding versions / scenarios
177
178 Add a new `test/NN-<label>/` folder (copy an existing one), pin its `alchemy`
179 version in `package.json`, set the `marker` + `main` form appropriately in
180 `src/worker.ts`, then add an entry to the `STAGES` array in `run.ts`. Keep
181 `Stack("CrossVersionApp")` and the worker `name` identical so it upgrades the
182 same app.
183
184 ### Version → state-store-version reference
185
186 | alchemy | state store version | pinned `effect` | notes |
187 | ---------------- | ------------------- | --------------- | ---------------------------------- |
188 | beta.29 … 30 | 1 | — | version-gating introduced (#156) |
189 | beta.31 … 32 | 2 | — | |
190 | beta.33 … 36 | 3 | — | |
191 | **beta.37 … 39** | **4** | beta.66 | last release with v4 = beta.39 |
192 | **beta.40 … 44** | **5** | beta.66 | last release with v5 = beta.44 |
193 | **beta.45** | **6** | beta.74 | only release with v6 |
194 | **beta.46 … 59** | **7** | beta.84 | bumped to v7 in beta.46 (PR #477); last v7 on npm = beta.59 (= latest) |
195 | current branch | 7 | workspace | |