The best way to control Claude Code, Codex and more from any computer or phone
| 1 | --- |
| 2 | name: prepare-release |
| 3 | description: Prepare an alchemy release by converging the live test suite — nuke + clear:state the testing account, run every test through a tag-driven `--plan` (unit → per-provider live in parallel → local), triage failures as they stream in, fan out one fix agent per failing/leaking service, census leaks, and loop until two consecutive rounds are green and clean. Fixes land as one alchemy PR plus at most one distilled PR and one floci PR. Use for "prepare a release", "/prepare-release", "release hardening run", "run the full suite before release", optionally scoped ("/prepare-release aws cloudflare"). To harden one provider's suite on its own, use the harden skill. |
| 4 | metadata: |
| 5 | internal: true |
| 6 | --- |
| 7 | |
| 8 | # Prepare a release |
| 9 | |
| 10 | This session is the **coordinator** of the convergence loop in AGENTS.md |
| 11 | ("The convergence loop: nuke → test → census → fix-fleet"). You own the clean |
| 12 | slate, the test runs, triage, type-checking, the census, and the PRs. Fix |
| 13 | agents own one service each. The user merges; you never do. |
| 14 | |
| 15 | Each provider in scope converges with the **harden** skill |
| 16 | (`.agents/skills/harden/SKILL.md`): its loop, root-cause order, isolation |
| 17 | rules and the provider-bug patterns it lists all apply here. This skill adds |
| 18 | the release-wide pieces around it: the clean slate, the plan across |
| 19 | providers, the fix fleet, and the PRs. When the user only asks to harden a |
| 20 | provider, use the harden skill instead of this one. |
| 21 | |
| 22 | The arguments name the scope: providers (`aws cloudflare gcp …`) or `all`. With |
| 23 | no arguments, propose a scope (default: unit + local + every `provider:*` live |
| 24 | tag) and confirm it together with the nuke in a single question. **Nuking the |
| 25 | testing account is destructive — always get an explicit go-ahead first.** |
| 26 | |
| 27 | ## 0. Preflight |
| 28 | |
| 29 | Start from a clean, up-to-date `main` in this worktree, on a fresh branch: |
| 30 | |
| 31 | ```sh |
| 32 | git status # must be clean; never git stash |
| 33 | git fetch origin main && git checkout -b release/prep-$(date +%Y-%m-%d) origin/main |
| 34 | git submodule update --init -- submodules/distilled |
| 35 | pnpm install |
| 36 | bun --version # must be 1.3.13 — 1.4 fails every file at collection |
| 37 | # if not: mkdir -p /tmp/bun1313 && (cd /tmp/bun1313 && npm i --no-save bun@1.3.13) |
| 38 | # export PATH=/tmp/bun1313/node_modules/.bin:$PATH (prefix every pnpm test with this) |
| 39 | node -v # alchemy CLI needs >= 24.11.1 and pnpm wants 24.x — node 25 |
| 40 | # makes pnpm print a [WARN] that breaks scripts/ensure-built.ts. If not 24.11+: |
| 41 | # mkdir -p /tmp/node24 && (cd /tmp/node24 && npm i --no-save node@24) |
| 42 | # export PATH=/tmp/node24/node_modules/.bin:$PATH |
| 43 | # Put these PATH exports + `set -a; source .env; set +a` in /tmp/release/env.sh and |
| 44 | # source it before every command — env doesn't persist between shell calls. |
| 45 | set -a; source .env; set +a # Cloudflare creds (pnpm download:env if missing) |
| 46 | aws sso login # AWS live + nuke ride the SSO session |
| 47 | mkdir -p /tmp/release |
| 48 | pnpm exec tsrs -b # REQUIRED before live runs: emits distilled lib/ |
| 49 | ``` |
| 50 | |
| 51 | Runtime bundles (Lambda, Fly, Hetzner, Neon Functions) resolve distilled |
| 52 | packages through their `lib/` build output, not `src/`. A fresh worktree has |
| 53 | no `lib/` for most providers, so every deployed runtime would crash at boot |
| 54 | with `Cannot find package '@distilled.cloud/…'`. |
| 55 | |
| 56 | Never set or unset `CI` for tests (a profile-store migration can wipe |
| 57 | `~/.alchemy`). If an AWS run fails in ~100ms during setup, the SSO session has |
| 58 | expired — re-run `aws sso login`. |
| 59 | |
| 60 | ## 1. Clean slate |
| 61 | |
| 62 | ```sh |
| 63 | pnpm nuke --yes --profile testing # scripts/nuke.sh requires --profile; no --yes hangs |
| 64 | pnpm clear:state --profile testing --yes --no-input # prompts without --yes, fails in a non-TTY |
| 65 | pnpm nuke --dry-run --profile testing > /tmp/release/census-0.txt # baseline; should be residue only |
| 66 | ``` |
| 67 | |
| 68 | Never run nuke while any suite is running. Holdouts that survive two nuke |
| 69 | passes are stuck deletes — put that service on the worklist; its delete path is |
| 70 | the bug. Documented-undeletable residue (excluded in `scripts/nuke.sh`, e.g. |
| 71 | `AWS.BackupSearch.SearchJob`, zone singletons) is not a finding. |
| 72 | |
| 73 | ## 2. Build the plan and dry-run it |
| 74 | |
| 75 | Tests carry tags (`unit`, `live`, `local`, `provider:<cloud>`, |
| 76 | `provider:<cloud>:<service>`). `--tags` takes expressions with `&&`, `||`, `!`, |
| 77 | parentheses and `*`; `--plan` takes a JSON array of sequential phases, where a |
| 78 | nested array runs branches in parallel and **the first matching branch wins**. |
| 79 | Tags listed under `optInTags` only run when named explicitly (wildcards and |
| 80 | negations don't count) — never name one without asking; they gate paid add-ons |
| 81 | and entitlements. |
| 82 | |
| 83 | Default plan (drop providers outside the scope): |
| 84 | |
| 85 | ```json |
| 86 | [ |
| 87 | { "tags": ["unit && !live"] }, |
| 88 | [ |
| 89 | { "tags": ["live && provider:aws"], "concurrency": 32 }, |
| 90 | { "tags": ["live && provider:cloudflare"], "concurrency": 16 }, |
| 91 | { "tags": ["live && provider:gcp"], "concurrency": 16 }, |
| 92 | { "tags": ["live && !provider:aws && !provider:cloudflare && !provider:gcp"], "concurrency": 8 } |
| 93 | ], |
| 94 | { "tags": ["local"], "concurrency": 8 }, |
| 95 | { "tags": [] } |
| 96 | ] |
| 97 | ``` |
| 98 | |
| 99 | - Phase 1 runs the unit tests first: a broken unit layer makes every live |
| 100 | failure noise, so stop and fix it before spending cloud time. |
| 101 | - The final `[]` branch is the catch-all for untagged tests. Everything it |
| 102 | picks up is a tagging gap — add the missing tags in the release PR. |
| 103 | - Never use `"unbounded"` and never exceed 32 per branch. Saturation shows up |
| 104 | as hundreds of 0ms `beforeAll` TimeoutErrors; real failures fail slow. |
| 105 | |
| 106 | Write the plan to `/tmp/release/plan.json` and preview it: |
| 107 | |
| 108 | ```sh |
| 109 | pnpm test --profile testing --plan "$(cat /tmp/release/plan.json)" --dry-run |
| 110 | ``` |
| 111 | |
| 112 | For reference, on 2026-10-09 the default plan selected 11,614 tests in 2,880 |
| 113 | files (unit 3,089 · aws 3,361 · cloudflare 1,577 · gcp 2,085 · other live |
| 114 | 811 · local 592 · untagged 99 in 19 files). Collection takes ~10s. Check |
| 115 | the branch counts against expectations. A branch with 0 tests is an |
| 116 | invocation bug (bad tag, quoting), not a pass. Show the user the preview |
| 117 | before the real run. |
| 118 | |
| 119 | ## 3. Run it |
| 120 | |
| 121 | A full round can outlast a 2-hour background-task limit, so detach it and |
| 122 | watch the output file with a monitor that filters to failed results |
| 123 | (`grep -E "\] ✗ |failed to run|^Tests:"` — a bare `✗` also matches the |
| 124 | `[n/N ✓ ✗]` progress counter on every line): |
| 125 | |
| 126 | ```sh |
| 127 | nohup pnpm test --profile testing --plan "$(cat /tmp/release/plan.json)" \ |
| 128 | > /tmp/release/round-1.out 2>&1 & disown |
| 129 | ``` |
| 130 | |
| 131 | Keep running notes in `/tmp/release/STATE.md` (round, blockers, worklist, |
| 132 | residue) so the loop survives context compaction. |
| 133 | |
| 134 | Note the `Full log: .alchemy/log/test/…` path at the end. Failures print |
| 135 | inline with their error as soon as they happen, so start triage at the first |
| 136 | failure rather than waiting for the run to finish. If nothing finishes for 10s |
| 137 | the runner lists running tests — a test stuck far past its timeout is a hang |
| 138 | to fix, not something to wait out. Check the "queued for an exclusive lock" |
| 139 | entries in that list before calling it a hang: those tests haven't started. |
| 140 | |
| 141 | Test time is the release's biggest cost, so one slow or exclusive test must |
| 142 | never hold up the rest of a run. If it does, fix the runner or narrow the |
| 143 | lock; don't lower concurrency and don't wait it out. Tests that only contend |
| 144 | for a shared cloud resource take a label (`{ exclusive: "aws:ses-account" }`) |
| 145 | instead of `exclusive: true`, which only process-global state needs. On |
| 146 | 2026-10-31 a writer-preferring `exclusive: true` gate froze the AWS suite |
| 147 | twice behind a 25-minute ElastiCache test. |
| 148 | |
| 149 | ## 4. Triage |
| 150 | |
| 151 | Group failures by their most specific `provider:<cloud>:<service>` tag (the |
| 152 | service is the unit of ownership). Classify each: |
| 153 | |
| 154 | | Bucket | Signal | Fix lands in | |
| 155 | | --- | --- | --- | |
| 156 | | Provider bug | wrong reconcile/diff/delete, leak, crash | `packages/alchemy/src/…` (alchemy PR) | |
| 157 | | Untyped error | `Unknown*Error`, out-of-union status error, wrong schema | distilled patch (distilled PR) | |
| 158 | | Emulator bug | only fails under floci | `submodules/floci` (floci PR) | |
| 159 | | Test bug | non-deterministic or hard-coded name, unbounded wait, missing propagation retry | the test (alchemy PR) | |
| 160 | | Platform/entitlement | plan-gated, slow async provisioning, beta API | skipIf-gate on the typed tag + exact error, keep an ungated probe | |
| 161 | | Flake | passes alone, fails under load | re-run alone and on main first | |
| 162 | |
| 163 | Root-cause priority is **provider bug > distilled patch > test fix**, and a |
| 164 | longer timeout comes last, only for work that is slow by nature (image |
| 165 | builds, cold starts). Never paper over a provider leak or untyped error in a |
| 166 | test. A failure that only shows up in some runs is usually two runs touching |
| 167 | the same cloud object — a shared name or stage, a missing lock label, or a |
| 168 | retry that leaks — not a flake; see the isolation rules in the harden skill. |
| 169 | Skip a test only for a permanent account blocker (quota, entitlement), with |
| 170 | `test.provider.skip` and a comment quoting the exact error. For suspected flakes, |
| 171 | build a verdict table: `Failure | Re-run alone | On main | Verdict`. A failure |
| 172 | that also fails on main is still in scope for the release — it just isn't a |
| 173 | regression. |
| 174 | |
| 175 | ## 5. Census |
| 176 | |
| 177 | After the run finishes (and never while one is running): |
| 178 | |
| 179 | ```sh |
| 180 | pnpm nuke --dry-run --profile testing > /tmp/release/census-1.txt |
| 181 | diff /tmp/release/census-0.txt /tmp/release/census-1.txt |
| 182 | ``` |
| 183 | |
| 184 | Every new resource is a leak, and a green service can still leak. **Leave the |
| 185 | leaked resources live** as evidence for the fix agents. The worklist is |
| 186 | failing services ∪ leaking services. |
| 187 | |
| 188 | ## 6. Fix fleet |
| 189 | |
| 190 | Launch one agent per service on the worklist, in parallel. In T3 Code each |
| 191 | fixer is a detached top-level thread from `t3_thread_launch` with |
| 192 | `workspaceStrategy: { type: "existing_worktree", worktreePath: <this worktree> }`, |
| 193 | never a `delegate_task` subagent thread. Collect each result with |
| 194 | `t3_thread_wait` + `t3_thread_read`. Run ~12 at once ( |
| 195 | account-singleton services like CloudTrail, Config, SecurityHub, GuardDuty, |
| 196 | ControlTower and IdentityCenter run as one sequential chain). Agents work in |
| 197 | this worktree and **commit but don't push**; they touch only their service's |
| 198 | files, plus single minimal insertions in shared files. The prompt is a |
| 199 | contract — include every item: |
| 200 | |
| 201 | 1. The service it owns, its directory, and its tag (e.g. `provider:aws:ec2`). |
| 202 | 2. Its exact failures (error text + test names) and its leak inventory from the |
| 203 | census diff. |
| 204 | 3. Assess-first: partial work may exist from an interrupted run; finish it |
| 205 | rather than rewriting. |
| 206 | 4. Reading list: AGENTS.md Reconciler doctrine + Typed Error Doctrine + speed |
| 207 | doctrine, the resource source, the test, and the distilled service + |
| 208 | existing patches. |
| 209 | 5. Root-cause priority (provider > distilled > test) and the rule that it |
| 210 | never hides a leak in the test. |
| 211 | 6. **No `tsc`, no `pnpm build`, no `pnpm nuke`** — the coordinator owns these. |
| 212 | 7. Speed doctrine: every run is |
| 213 | `pnpm test --tags '<its tag>' --profile testing`; per-test |
| 214 | timeouts ≤ 90–120s; every retry bounded (`times ≤ 8–10`, < 60s total); |
| 215 | three-iteration budget, then gate and report. |
| 216 | 8. Distilled patches for its service only, regenerated in place: |
| 217 | - Cloudflare: `cd submodules/distilled/packages/cloudflare && bun scripts/generate.ts --resource <svc> && pnpm exec oxfmt src/services/<svc>.ts` |
| 218 | - AWS: `cd submodules/distilled/packages/aws && bun scripts/generate.ts --sdk <svc>` |
| 219 | Tests resolve distilled from `src/`, so there's nothing to rebuild or wait for. |
| 220 | 9. Test rules: start and end with `stack.destroy()`, out-of-band verification |
| 221 | through distilled, no mocks. Names: let the resource generate the physical |
| 222 | name. Only a test about explicitly provided names passes one, and it is |
| 223 | derived from `stack.stage` so it is isolated to the user running the test. |
| 224 | Never `Date.now()`, UUIDs or random values in a name: every run is |
| 225 | deterministic and leaves no orphans. A resource the API can never delete |
| 226 | keeps one shared constant name that every run adopts. Fix any hard-coded |
| 227 | name you find in the suites you touch. |
| 228 | 10. Done means: its **full** tag passes, and distilled list/describe calls show |
| 229 | zero orphans from its service. |
| 230 | 11. Structured result: `{ service, tag, testsPassed, testCommand, rootCause, |
| 231 | files, patches (with reasons), skippedTests (with exact errors), leaksFixed, |
| 232 | notes }`. |
| 233 | |
| 234 | While agents run, watch for stalls: a transcript that hasn't been written for |
| 235 | ~5 minutes with no child test process is stalled — `t3_thread_interrupt` it |
| 236 | and re-dispatch with a new `t3_thread_launch`. |
| 237 | |
| 238 | When results come in, review each diff yourself before accepting it. Reject |
| 239 | any that catch a catch-all error, widen a cast, add `Effect.orDie` to a |
| 240 | lifecycle op, mock a response, or skip a test without a typed reason. |
| 241 | |
| 242 | ## 7. Gate |
| 243 | |
| 244 | ```sh |
| 245 | pnpm exec tsrs -b # coordinator only, once per round |
| 246 | pnpm docs:check-jsdoc # if any props/attrs changed |
| 247 | ``` |
| 248 | |
| 249 | Fix cross-cutting fallout yourself (`Providers.ts` nested `Layer.mergeAll` |
| 250 | groups, barrel conflicts). Commit the round — one commit per service fix, |
| 251 | conventional-commit scoped (`fix(aws/ec2): …`) — and **push immediately**. |
| 252 | |
| 253 | ## 8. Loop |
| 254 | |
| 255 | **Converge with targeted runs, then confirm with full ones.** Live tests are |
| 256 | slow, so after the first full plan every fixing run is targeted: |
| 257 | |
| 258 | 1. **Track passes cumulatively.** Every run (yours and the agents') writes |
| 259 | `PASS <file> > <title>` lines to `packages/alchemy/.alchemy/log/test/*.log`. |
| 260 | Rebuild the passed set from all logs, not just the last round: |
| 261 | ```sh |
| 262 | cat packages/alchemy/.alchemy/log/test/*.log | grep -E "^PASS " \ |
| 263 | | sed -E 's/^PASS //; s/ \([0-9.]+m?s\).*$//' | sort -u > /tmp/release/passed.txt |
| 264 | ``` |
| 265 | 2. **Compute the residual:** tests with a `FAIL` line and no `PASS` line, plus |
| 266 | tests that never finished (a killed or stalled round) — find those by |
| 267 | diffing the runner's "still running" lists against the result lines. |
| 268 | 3. **Fix agents get each failing area passing individually** (their own |
| 269 | suites, `-t` for single tests in long sequential files). |
| 270 | 4. **One residual run** of only what's still unpassed — file paths, narrowed |
| 271 | with `-t` where a file is long and sequential (e.g. `Fly/BlueGreen.test.ts` |
| 272 | has ~180 tests at ~100s each; never run it unfiltered). |
| 273 | 5. **Confirm stability per provider** with the harden skill's loop: full |
| 274 | runs of each provider's suite, a census after each, until two consecutive |
| 275 | runs are green and clean. One green run proves little — retry leaks, lagging |
| 276 | listings and shared names only show up across runs. |
| 277 | 6. **Finish with nuke + clear:state + census.** The goal is consecutive green |
| 278 | runs and a census that shows `Nothing to delete`. |
| 279 | |
| 280 | If a census leftover reappears after nuke, its `list` or `delete` is the bug |
| 281 | (e.g. `list` returning archived or Cloudflare-managed objects), not the test. |
| 282 | |
| 283 | Report each round to the user, briefly: |
| 284 | |
| 285 | ```md |
| 286 | ### Round N: 1834 passed · 12 failed · 3 leaking services |
| 287 | |
| 288 | | Service | Tag | Failure | Bucket | Status | |
| 289 | | --- | --- | --- | --- | --- | |
| 290 | | EC2 | provider:aws:ec2 | InternetGateway delete timeout | provider bug | fixed, 63/63 | |
| 291 | | R2 | provider:cloudflare:r2 | UnknownCloudflareError 10058 | distilled | patched, 30/30 | |
| 292 | |
| 293 | Next: round N+1 (full plan). |
| 294 | ``` |
| 295 | |
| 296 | ## 9. PRs |
| 297 | |
| 298 | At most three, one per repo. Open them as soon as there is a first fix, and |
| 299 | push every later fix promptly. |
| 300 | |
| 301 | **distilled** (only if there are patches) — work in a standalone clone, never |
| 302 | a `git worktree add` from inside `submodules/distilled` (its `core.worktree` |
| 303 | belongs to another checkout): |
| 304 | |
| 305 | ```sh |
| 306 | git clone --reference submodules/distilled https://github.com/alchemy-run/distilled /tmp/release/distilled |
| 307 | cd /tmp/release/distilled && git checkout -b fix/release-prep-$(date +%Y-%m-%d) |
| 308 | # copy the patches + regenerated services from submodules/distilled, commit per service |
| 309 | ``` |
| 310 | |
| 311 | Title: `fix: typed errors and schema fixes from release prep`. The body lists |
| 312 | each patch with the error it types, e.g. `R2 putBucketLifecycle: code 10058 → |
| 313 | LifecycleRuleInvalid`. |
| 314 | |
| 315 | **floci** (only if emulator fixes are needed) — floci is skipped by default |
| 316 | (`git submodule update --init --checkout -- submodules/floci`); use the same |
| 317 | standalone-clone pattern against `https://github.com/alchemy-run/floci`. If alchemy needs the new emulator image, it needs a release |
| 318 | tag. |
| 319 | |
| 320 | **alchemy** — branch `release/prep-<date>`, title |
| 321 | `fix: release hardening (<date>)`. It contains the provider, test, gating and |
| 322 | tagging fixes, plus submodule bumps pointing at the distilled/floci PR heads. |
| 323 | The merge order is distilled → floci → alchemy. After each upstream PR merges, |
| 324 | repoint the submodule at its **`main` merge commit** (never a side branch), |
| 325 | rerun the affected tags, and push. |
| 326 | |
| 327 | Descriptions follow AGENTS.md's PR conventions: one plain sentence, then |
| 328 | user-facing snippets or a per-service table of what was fixed. No `#`/`##` |
| 329 | headings and no test plan. Write the body to a file and pass `--body-file`. |
| 330 | Link every PR to this thread with the T3 `link_pull_request` tool when it's |
| 331 | available. Leave PRs as drafts until the gate passes, then `gh pr ready`. |
| 332 | |
| 333 | ## 10. Final report |
| 334 | |
| 335 | End with the merge ask: |
| 336 | |
| 337 | ```md |
| 338 | ### Release prep converged after N rounds. I recommend merging, in this order: |
| 339 | |
| 340 | 1. distilled — <what it types> |
| 341 | https://github.com/alchemy-run/distilled/pull/X |
| 342 | 2. alchemy — <K> service fixes, <G> gated tests, <T> tagging fixes |
| 343 | https://github.com/alchemy-run/alchemy/pull/Y |
| 344 | |
| 345 | | Phase | Tests | Result | |
| 346 | | --- | --- | --- | |
| 347 | | unit | 856 | ✅ | |
| 348 | | live provider:aws | 1662 | ✅ 1650 · 12 skipped (gated) | |
| 349 | | … | | | |
| 350 | |
| 351 | Gated (needs entitlement / platform): <test → exact error>. |
| 352 | Fails on main too (pre-existing, now fixed): <list>. |
| 353 | Census: clean except <documented residue>. |
| 354 | ``` |
| 355 | |
| 356 | Say plainly what was **not** run (out-of-scope providers, opt-in tags). Don't |
| 357 | report CI status in place of the test results. |
| 358 | |
| 359 | ## Pitfalls seen in practice |
| 360 | |
| 361 | - **Commit with the pinned env sourced.** The pre-commit hook runs pnpm; with |
| 362 | the wrong bun/node or a stray temp workspace dir under `examples/` (left by |
| 363 | a killed test) it re-installs and rewrites `pnpm-lock.yaml`. Never commit |
| 364 | the lockfile or Prisma `generated/contract.*` churn. |
| 365 | - **Distilled regeneration can `pnpm install` inside `submodules/distilled`**, |
| 366 | creating a nested `node_modules` with a second `effect` copy ("Unable to get |
| 367 | redacted value" everywhere). Move `submodules/distilled/node_modules` aside |
| 368 | and check `packages/*/node_modules/effect` links point at the root store. |
| 369 | - **Other sessions share the account.** Another worktree's live resources |
| 370 | show up in your census; check `ps -eo etime,command | grep alchemy-test` |
| 371 | before deleting a leftover you can't trace to your own run. A linked |
| 372 | worktree defaults to its own test stage (`test_$USER_<worktree>`), so two |
| 373 | worktrees don't collide; suites that set their own stage must prefix it |
| 374 | with `Test.defaultStage()`. |
| 375 | - **Provider `console.log` doesn't reach the run log.** Debug with |
| 376 | `Effect.logInfo`, and probe API behavior with a throwaway test file rather |
| 377 | than guessing. |
| 378 | - **Account blockers are not code.** Billing (GCP), token permission groups |
| 379 | (e.g. Cloudflare K2) and retired services (AWS FinSpace) get reported to the |
| 380 | user with the exact error; never enable billing or edit shared tokens. |