The best way to control Claude Code, Codex and more from any computer or phone

main
md 380 lines 19 KB
Rendered Raw
1 ---
2 name: prepare-release
3 description: Prepare an alchemy release by converging the live test suite — nuke + clear:state the testing account, run every test through a tag-driven `--plan` (unit → per-provider live in parallel → local), triage failures as they stream in, fan out one fix agent per failing/leaking service, census leaks, and loop until two consecutive rounds are green and clean. Fixes land as one alchemy PR plus at most one distilled PR and one floci PR. Use for "prepare a release", "/prepare-release", "release hardening run", "run the full suite before release", optionally scoped ("/prepare-release aws cloudflare"). To harden one provider's suite on its own, use the harden skill.
4 metadata:
5 internal: true
6 ---
7
8 # Prepare a release
9
10 This session is the **coordinator** of the convergence loop in AGENTS.md
11 ("The convergence loop: nuke → test → census → fix-fleet"). You own the clean
12 slate, the test runs, triage, type-checking, the census, and the PRs. Fix
13 agents own one service each. The user merges; you never do.
14
15 Each provider in scope converges with the **harden** skill
16 (`.agents/skills/harden/SKILL.md`): its loop, root-cause order, isolation
17 rules and the provider-bug patterns it lists all apply here. This skill adds
18 the release-wide pieces around it: the clean slate, the plan across
19 providers, the fix fleet, and the PRs. When the user only asks to harden a
20 provider, use the harden skill instead of this one.
21
22 The arguments name the scope: providers (`aws cloudflare gcp …`) or `all`. With
23 no arguments, propose a scope (default: unit + local + every `provider:*` live
24 tag) and confirm it together with the nuke in a single question. **Nuking the
25 testing account is destructive — always get an explicit go-ahead first.**
26
27 ## 0. Preflight
28
29 Start from a clean, up-to-date `main` in this worktree, on a fresh branch:
30
31 ```sh
32 git status # must be clean; never git stash
33 git fetch origin main && git checkout -b release/prep-$(date +%Y-%m-%d) origin/main
34 git submodule update --init -- submodules/distilled
35 pnpm install
36 bun --version # must be 1.3.13 — 1.4 fails every file at collection
37 # if not: mkdir -p /tmp/bun1313 && (cd /tmp/bun1313 && npm i --no-save bun@1.3.13)
38 # export PATH=/tmp/bun1313/node_modules/.bin:$PATH (prefix every pnpm test with this)
39 node -v # alchemy CLI needs >= 24.11.1 and pnpm wants 24.x — node 25
40 # makes pnpm print a [WARN] that breaks scripts/ensure-built.ts. If not 24.11+:
41 # mkdir -p /tmp/node24 && (cd /tmp/node24 && npm i --no-save node@24)
42 # export PATH=/tmp/node24/node_modules/.bin:$PATH
43 # Put these PATH exports + `set -a; source .env; set +a` in /tmp/release/env.sh and
44 # source it before every command — env doesn't persist between shell calls.
45 set -a; source .env; set +a # Cloudflare creds (pnpm download:env if missing)
46 aws sso login # AWS live + nuke ride the SSO session
47 mkdir -p /tmp/release
48 pnpm exec tsrs -b # REQUIRED before live runs: emits distilled lib/
49 ```
50
51 Runtime bundles (Lambda, Fly, Hetzner, Neon Functions) resolve distilled
52 packages through their `lib/` build output, not `src/`. A fresh worktree has
53 no `lib/` for most providers, so every deployed runtime would crash at boot
54 with `Cannot find package '@distilled.cloud/…'`.
55
56 Never set or unset `CI` for tests (a profile-store migration can wipe
57 `~/.alchemy`). If an AWS run fails in ~100ms during setup, the SSO session has
58 expired — re-run `aws sso login`.
59
60 ## 1. Clean slate
61
62 ```sh
63 pnpm nuke --yes --profile testing # scripts/nuke.sh requires --profile; no --yes hangs
64 pnpm clear:state --profile testing --yes --no-input # prompts without --yes, fails in a non-TTY
65 pnpm nuke --dry-run --profile testing > /tmp/release/census-0.txt # baseline; should be residue only
66 ```
67
68 Never run nuke while any suite is running. Holdouts that survive two nuke
69 passes are stuck deletes — put that service on the worklist; its delete path is
70 the bug. Documented-undeletable residue (excluded in `scripts/nuke.sh`, e.g.
71 `AWS.BackupSearch.SearchJob`, zone singletons) is not a finding.
72
73 ## 2. Build the plan and dry-run it
74
75 Tests carry tags (`unit`, `live`, `local`, `provider:<cloud>`,
76 `provider:<cloud>:<service>`). `--tags` takes expressions with `&&`, `||`, `!`,
77 parentheses and `*`; `--plan` takes a JSON array of sequential phases, where a
78 nested array runs branches in parallel and **the first matching branch wins**.
79 Tags listed under `optInTags` only run when named explicitly (wildcards and
80 negations don't count) — never name one without asking; they gate paid add-ons
81 and entitlements.
82
83 Default plan (drop providers outside the scope):
84
85 ```json
86 [
87 { "tags": ["unit && !live"] },
88 [
89 { "tags": ["live && provider:aws"], "concurrency": 32 },
90 { "tags": ["live && provider:cloudflare"], "concurrency": 16 },
91 { "tags": ["live && provider:gcp"], "concurrency": 16 },
92 { "tags": ["live && !provider:aws && !provider:cloudflare && !provider:gcp"], "concurrency": 8 }
93 ],
94 { "tags": ["local"], "concurrency": 8 },
95 { "tags": [] }
96 ]
97 ```
98
99 - Phase 1 runs the unit tests first: a broken unit layer makes every live
100 failure noise, so stop and fix it before spending cloud time.
101 - The final `[]` branch is the catch-all for untagged tests. Everything it
102 picks up is a tagging gap — add the missing tags in the release PR.
103 - Never use `"unbounded"` and never exceed 32 per branch. Saturation shows up
104 as hundreds of 0ms `beforeAll` TimeoutErrors; real failures fail slow.
105
106 Write the plan to `/tmp/release/plan.json` and preview it:
107
108 ```sh
109 pnpm test --profile testing --plan "$(cat /tmp/release/plan.json)" --dry-run
110 ```
111
112 For reference, on 2026-10-09 the default plan selected 11,614 tests in 2,880
113 files (unit 3,089 · aws 3,361 · cloudflare 1,577 · gcp 2,085 · other live
114 811 · local 592 · untagged 99 in 19 files). Collection takes ~10s. Check
115 the branch counts against expectations. A branch with 0 tests is an
116 invocation bug (bad tag, quoting), not a pass. Show the user the preview
117 before the real run.
118
119 ## 3. Run it
120
121 A full round can outlast a 2-hour background-task limit, so detach it and
122 watch the output file with a monitor that filters to failed results
123 (`grep -E "\] ✗ |failed to run|^Tests:"` — a bare `✗` also matches the
124 `[n/N ✓ ✗]` progress counter on every line):
125
126 ```sh
127 nohup pnpm test --profile testing --plan "$(cat /tmp/release/plan.json)" \
128 > /tmp/release/round-1.out 2>&1 & disown
129 ```
130
131 Keep running notes in `/tmp/release/STATE.md` (round, blockers, worklist,
132 residue) so the loop survives context compaction.
133
134 Note the `Full log: .alchemy/log/test/…` path at the end. Failures print
135 inline with their error as soon as they happen, so start triage at the first
136 failure rather than waiting for the run to finish. If nothing finishes for 10s
137 the runner lists running tests — a test stuck far past its timeout is a hang
138 to fix, not something to wait out. Check the "queued for an exclusive lock"
139 entries in that list before calling it a hang: those tests haven't started.
140
141 Test time is the release's biggest cost, so one slow or exclusive test must
142 never hold up the rest of a run. If it does, fix the runner or narrow the
143 lock; don't lower concurrency and don't wait it out. Tests that only contend
144 for a shared cloud resource take a label (`{ exclusive: "aws:ses-account" }`)
145 instead of `exclusive: true`, which only process-global state needs. On
146 2026-10-31 a writer-preferring `exclusive: true` gate froze the AWS suite
147 twice behind a 25-minute ElastiCache test.
148
149 ## 4. Triage
150
151 Group failures by their most specific `provider:<cloud>:<service>` tag (the
152 service is the unit of ownership). Classify each:
153
154 | Bucket | Signal | Fix lands in |
155 | --- | --- | --- |
156 | Provider bug | wrong reconcile/diff/delete, leak, crash | `packages/alchemy/src/…` (alchemy PR) |
157 | Untyped error | `Unknown*Error`, out-of-union status error, wrong schema | distilled patch (distilled PR) |
158 | Emulator bug | only fails under floci | `submodules/floci` (floci PR) |
159 | Test bug | non-deterministic or hard-coded name, unbounded wait, missing propagation retry | the test (alchemy PR) |
160 | Platform/entitlement | plan-gated, slow async provisioning, beta API | skipIf-gate on the typed tag + exact error, keep an ungated probe |
161 | Flake | passes alone, fails under load | re-run alone and on main first |
162
163 Root-cause priority is **provider bug > distilled patch > test fix**, and a
164 longer timeout comes last, only for work that is slow by nature (image
165 builds, cold starts). Never paper over a provider leak or untyped error in a
166 test. A failure that only shows up in some runs is usually two runs touching
167 the same cloud object — a shared name or stage, a missing lock label, or a
168 retry that leaks — not a flake; see the isolation rules in the harden skill.
169 Skip a test only for a permanent account blocker (quota, entitlement), with
170 `test.provider.skip` and a comment quoting the exact error. For suspected flakes,
171 build a verdict table: `Failure | Re-run alone | On main | Verdict`. A failure
172 that also fails on main is still in scope for the release — it just isn't a
173 regression.
174
175 ## 5. Census
176
177 After the run finishes (and never while one is running):
178
179 ```sh
180 pnpm nuke --dry-run --profile testing > /tmp/release/census-1.txt
181 diff /tmp/release/census-0.txt /tmp/release/census-1.txt
182 ```
183
184 Every new resource is a leak, and a green service can still leak. **Leave the
185 leaked resources live** as evidence for the fix agents. The worklist is
186 failing services ∪ leaking services.
187
188 ## 6. Fix fleet
189
190 Launch one agent per service on the worklist, in parallel. In T3 Code each
191 fixer is a detached top-level thread from `t3_thread_launch` with
192 `workspaceStrategy: { type: "existing_worktree", worktreePath: <this worktree> }`,
193 never a `delegate_task` subagent thread. Collect each result with
194 `t3_thread_wait` + `t3_thread_read`. Run ~12 at once (
195 account-singleton services like CloudTrail, Config, SecurityHub, GuardDuty,
196 ControlTower and IdentityCenter run as one sequential chain). Agents work in
197 this worktree and **commit but don't push**; they touch only their service's
198 files, plus single minimal insertions in shared files. The prompt is a
199 contract — include every item:
200
201 1. The service it owns, its directory, and its tag (e.g. `provider:aws:ec2`).
202 2. Its exact failures (error text + test names) and its leak inventory from the
203 census diff.
204 3. Assess-first: partial work may exist from an interrupted run; finish it
205 rather than rewriting.
206 4. Reading list: AGENTS.md Reconciler doctrine + Typed Error Doctrine + speed
207 doctrine, the resource source, the test, and the distilled service +
208 existing patches.
209 5. Root-cause priority (provider > distilled > test) and the rule that it
210 never hides a leak in the test.
211 6. **No `tsc`, no `pnpm build`, no `pnpm nuke`** — the coordinator owns these.
212 7. Speed doctrine: every run is
213 `pnpm test --tags '<its tag>' --profile testing`; per-test
214 timeouts ≤ 90–120s; every retry bounded (`times ≤ 8–10`, < 60s total);
215 three-iteration budget, then gate and report.
216 8. Distilled patches for its service only, regenerated in place:
217 - Cloudflare: `cd submodules/distilled/packages/cloudflare && bun scripts/generate.ts --resource <svc> && pnpm exec oxfmt src/services/<svc>.ts`
218 - AWS: `cd submodules/distilled/packages/aws && bun scripts/generate.ts --sdk <svc>`
219 Tests resolve distilled from `src/`, so there's nothing to rebuild or wait for.
220 9. Test rules: start and end with `stack.destroy()`, out-of-band verification
221 through distilled, no mocks. Names: let the resource generate the physical
222 name. Only a test about explicitly provided names passes one, and it is
223 derived from `stack.stage` so it is isolated to the user running the test.
224 Never `Date.now()`, UUIDs or random values in a name: every run is
225 deterministic and leaves no orphans. A resource the API can never delete
226 keeps one shared constant name that every run adopts. Fix any hard-coded
227 name you find in the suites you touch.
228 10. Done means: its **full** tag passes, and distilled list/describe calls show
229 zero orphans from its service.
230 11. Structured result: `{ service, tag, testsPassed, testCommand, rootCause,
231 files, patches (with reasons), skippedTests (with exact errors), leaksFixed,
232 notes }`.
233
234 While agents run, watch for stalls: a transcript that hasn't been written for
235 ~5 minutes with no child test process is stalled — `t3_thread_interrupt` it
236 and re-dispatch with a new `t3_thread_launch`.
237
238 When results come in, review each diff yourself before accepting it. Reject
239 any that catch a catch-all error, widen a cast, add `Effect.orDie` to a
240 lifecycle op, mock a response, or skip a test without a typed reason.
241
242 ## 7. Gate
243
244 ```sh
245 pnpm exec tsrs -b # coordinator only, once per round
246 pnpm docs:check-jsdoc # if any props/attrs changed
247 ```
248
249 Fix cross-cutting fallout yourself (`Providers.ts` nested `Layer.mergeAll`
250 groups, barrel conflicts). Commit the round — one commit per service fix,
251 conventional-commit scoped (`fix(aws/ec2): …`) — and **push immediately**.
252
253 ## 8. Loop
254
255 **Converge with targeted runs, then confirm with full ones.** Live tests are
256 slow, so after the first full plan every fixing run is targeted:
257
258 1. **Track passes cumulatively.** Every run (yours and the agents') writes
259 `PASS <file> > <title>` lines to `packages/alchemy/.alchemy/log/test/*.log`.
260 Rebuild the passed set from all logs, not just the last round:
261 ```sh
262 cat packages/alchemy/.alchemy/log/test/*.log | grep -E "^PASS " \
263 | sed -E 's/^PASS //; s/ \([0-9.]+m?s\).*$//' | sort -u > /tmp/release/passed.txt
264 ```
265 2. **Compute the residual:** tests with a `FAIL` line and no `PASS` line, plus
266 tests that never finished (a killed or stalled round) — find those by
267 diffing the runner's "still running" lists against the result lines.
268 3. **Fix agents get each failing area passing individually** (their own
269 suites, `-t` for single tests in long sequential files).
270 4. **One residual run** of only what's still unpassed — file paths, narrowed
271 with `-t` where a file is long and sequential (e.g. `Fly/BlueGreen.test.ts`
272 has ~180 tests at ~100s each; never run it unfiltered).
273 5. **Confirm stability per provider** with the harden skill's loop: full
274 runs of each provider's suite, a census after each, until two consecutive
275 runs are green and clean. One green run proves little — retry leaks, lagging
276 listings and shared names only show up across runs.
277 6. **Finish with nuke + clear:state + census.** The goal is consecutive green
278 runs and a census that shows `Nothing to delete`.
279
280 If a census leftover reappears after nuke, its `list` or `delete` is the bug
281 (e.g. `list` returning archived or Cloudflare-managed objects), not the test.
282
283 Report each round to the user, briefly:
284
285 ```md
286 ### Round N: 1834 passed · 12 failed · 3 leaking services
287
288 | Service | Tag | Failure | Bucket | Status |
289 | --- | --- | --- | --- | --- |
290 | EC2 | provider:aws:ec2 | InternetGateway delete timeout | provider bug | fixed, 63/63 |
291 | R2 | provider:cloudflare:r2 | UnknownCloudflareError 10058 | distilled | patched, 30/30 |
292
293 Next: round N+1 (full plan).
294 ```
295
296 ## 9. PRs
297
298 At most three, one per repo. Open them as soon as there is a first fix, and
299 push every later fix promptly.
300
301 **distilled** (only if there are patches) — work in a standalone clone, never
302 a `git worktree add` from inside `submodules/distilled` (its `core.worktree`
303 belongs to another checkout):
304
305 ```sh
306 git clone --reference submodules/distilled https://github.com/alchemy-run/distilled /tmp/release/distilled
307 cd /tmp/release/distilled && git checkout -b fix/release-prep-$(date +%Y-%m-%d)
308 # copy the patches + regenerated services from submodules/distilled, commit per service
309 ```
310
311 Title: `fix: typed errors and schema fixes from release prep`. The body lists
312 each patch with the error it types, e.g. `R2 putBucketLifecycle: code 10058 →
313 LifecycleRuleInvalid`.
314
315 **floci** (only if emulator fixes are needed) — floci is skipped by default
316 (`git submodule update --init --checkout -- submodules/floci`); use the same
317 standalone-clone pattern against `https://github.com/alchemy-run/floci`. If alchemy needs the new emulator image, it needs a release
318 tag.
319
320 **alchemy** — branch `release/prep-<date>`, title
321 `fix: release hardening (<date>)`. It contains the provider, test, gating and
322 tagging fixes, plus submodule bumps pointing at the distilled/floci PR heads.
323 The merge order is distilled → floci → alchemy. After each upstream PR merges,
324 repoint the submodule at its **`main` merge commit** (never a side branch),
325 rerun the affected tags, and push.
326
327 Descriptions follow AGENTS.md's PR conventions: one plain sentence, then
328 user-facing snippets or a per-service table of what was fixed. No `#`/`##`
329 headings and no test plan. Write the body to a file and pass `--body-file`.
330 Link every PR to this thread with the T3 `link_pull_request` tool when it's
331 available. Leave PRs as drafts until the gate passes, then `gh pr ready`.
332
333 ## 10. Final report
334
335 End with the merge ask:
336
337 ```md
338 ### Release prep converged after N rounds. I recommend merging, in this order:
339
340 1. distilled — <what it types>
341 https://github.com/alchemy-run/distilled/pull/X
342 2. alchemy — <K> service fixes, <G> gated tests, <T> tagging fixes
343 https://github.com/alchemy-run/alchemy/pull/Y
344
345 | Phase | Tests | Result |
346 | --- | --- | --- |
347 | unit | 856 | ✅ |
348 | live provider:aws | 1662 | ✅ 1650 · 12 skipped (gated) |
349 | … | | |
350
351 Gated (needs entitlement / platform): <test → exact error>.
352 Fails on main too (pre-existing, now fixed): <list>.
353 Census: clean except <documented residue>.
354 ```
355
356 Say plainly what was **not** run (out-of-scope providers, opt-in tags). Don't
357 report CI status in place of the test results.
358
359 ## Pitfalls seen in practice
360
361 - **Commit with the pinned env sourced.** The pre-commit hook runs pnpm; with
362 the wrong bun/node or a stray temp workspace dir under `examples/` (left by
363 a killed test) it re-installs and rewrites `pnpm-lock.yaml`. Never commit
364 the lockfile or Prisma `generated/contract.*` churn.
365 - **Distilled regeneration can `pnpm install` inside `submodules/distilled`**,
366 creating a nested `node_modules` with a second `effect` copy ("Unable to get
367 redacted value" everywhere). Move `submodules/distilled/node_modules` aside
368 and check `packages/*/node_modules/effect` links point at the root store.
369 - **Other sessions share the account.** Another worktree's live resources
370 show up in your census; check `ps -eo etime,command | grep alchemy-test`
371 before deleting a leftover you can't trace to your own run. A linked
372 worktree defaults to its own test stage (`test_$USER_<worktree>`), so two
373 worktrees don't collide; suites that set their own stage must prefix it
374 with `Test.defaultStage()`.
375 - **Provider `console.log` doesn't reach the run log.** Debug with
376 `Effect.logInfo`, and probe API behavior with a throwaway test file rather
377 than guessing.
378 - **Account blockers are not code.** Billing (GCP), token permission groups
379 (e.g. Cloudflare K2) and retired services (AWS FinSpace) get reported to the
380 user with the exact error; never enable billing or edit shared tokens.