The best way to control Claude Code, Codex and more from any computer or phone
| 1 | --- |
| 2 | name: release-overnight |
| 3 | description: Prepare an alchemy release unattended overnight, in series — after one go-ahead, nuke the testing account, then walk every provider's live suite one at a time (Cloudflare → core unit → core integration → small providers → Fly → GCP → AWS), fixing each until its whole suite passes in one run AND the account census is clean, then run `pnpm test:examples`, and finish with one alchemy PR plus at most one distilled PR and one floci PR, all linked, that merge into a release passing every test. Use for "prepare the release overnight", "/release-overnight", "run the release suite in series while I sleep"; for the parallel, interactive version use prepare-release. |
| 4 | metadata: |
| 5 | internal: true |
| 6 | --- |
| 7 | |
| 8 | # Release overnight |
| 9 | |
| 10 | This thread runs the whole release unattended, **one suite at a time**. It owns |
| 11 | every step: the clean slate, each run, the fixes, type-checking, the census |
| 12 | and the PRs. It launches no subagents or threads, and it runs nothing in |
| 13 | parallel with a live suite. Going slowly in series is the point. A live suite |
| 14 | under load produces fake failures, and parallel fixers on one account leak |
| 15 | into each other's census. |
| 16 | |
| 17 | A provider is **done** when its whole suite passes in one run at the current |
| 18 | head and the census shows nothing new. The release is done when every |
| 19 | provider is done at the final head, `pnpm test:examples` passes, and the |
| 20 | account is clean. The user merges; you never do. |
| 21 | |
| 22 | ## 0. One question, then unattended |
| 23 | |
| 24 | Ask once, up front, in a single question. Confirm three things: the scope |
| 25 | (default: everything in §2), that you'll nuke the testing account before each |
| 26 | step for the whole run, and any opt-in tags or paid add-ons to include |
| 27 | (default: none). **Nuking is destructive, so get an explicit yes.** After |
| 28 | that, never ask again. Take every decision yourself, record it in |
| 29 | `STATE.md`, and keep going. An answer you would have asked for goes in |
| 30 | the final report. |
| 31 | |
| 32 | Don't end the turn until §6 is done. Ending the turn ends the night. Wait on |
| 33 | long runs with `t3_background_wait`, never with `sleep` or a polling loop. |
| 34 | |
| 35 | ## 1. Preflight and durable state |
| 36 | |
| 37 | Start from the latest `origin/main`, on a fresh branch: |
| 38 | |
| 39 | ```sh |
| 40 | git status # must be clean; never git stash |
| 41 | git fetch origin main |
| 42 | git checkout -b release/overnight-$(date +%Y-%m-%d) origin/main |
| 43 | git log -1 --format='%H %s' origin/main # record as "Base" in STATE.md |
| 44 | git submodule update --init -- submodules/distilled |
| 45 | ``` |
| 46 | |
| 47 | Then run the rest of |
| 48 | [prepare-release §0](../prepare-release/SKILL.md#0-preflight): `pnpm install`, |
| 49 | pinned bun/node in `/tmp/release/env.sh`, `.env`, `aws sso login`, and |
| 50 | `pnpm exec tsrs -b`. |
| 51 | |
| 52 | Don't merge `main` into the branch mid-run. A moving base would make every |
| 53 | `Passed at` meaningless. §6 picks up whatever landed overnight. |
| 54 | |
| 55 | Keep the run's state in `/tmp/release/STATE.md` and update it after every |
| 56 | event. Context will be compacted many times overnight, and this file is how |
| 57 | you resume: |
| 58 | |
| 59 | ```md |
| 60 | # Release overnight 2026-10-31 — branch release/overnight-2026-10-31 |
| 61 | Base: origin/main <sha> · Scope: all · opt-in: none · nuke authorized: yes |
| 62 | PRs: alchemy #— · distilled #— · floci #— |
| 63 | |
| 64 | | # | Step | Command | Status | Passed at | Notes | |
| 65 | | - | ---- | ------- | ------ | --------- | ----- | |
| 66 | | 1 | Cloudflare | pnpm test:integration test/Cloudflare | done | a1b2c3d | | |
| 67 | | 2 | core unit | pnpm test | fixing (iter 2) | | Plan diff regression | |
| 68 | |
| 69 | Gated: <test → typed tag + exact error> |
| 70 | Blocked (account, not code): <step → exact error> |
| 71 | ``` |
| 72 | |
| 73 | **On every start, read `STATE.md` first.** If it exists, resume at the first |
| 74 | step that isn't `done`. Don't restart the night. |
| 75 | |
| 76 | ## 2. The steps, in order |
| 77 | |
| 78 | Cloudflare goes first. State and most other suites deploy through |
| 79 | `packages/cloudflare`, so getting it green first keeps its failures out of |
| 80 | every later step. After that, cheap and narrow first, so a broken shared |
| 81 | layer shows up in minutes, not after AWS. Paths are relative to |
| 82 | `packages/integration-tests`: |
| 83 | |
| 84 | | # | Step | Paths | `--concurrency` | |
| 85 | | - | ---- | ----- | --------------- | |
| 86 | | 1 | Cloudflare | `test/Cloudflare` | 16 | |
| 87 | | 2 | core unit | `pnpm test` (packages/alchemy) | default | |
| 88 | | 3 | core integration | `test/*.test.ts test/Bundle test/Cli test/Local test/Runtime test/Util test/Website` | 8 | |
| 89 | | 4 | building blocks | `test/Docker test/Redis test/Kubernetes test/SQL test/Drizzle test/Domain test/Auth test/DNS test/ACME` | 8 | |
| 90 | | 5 | Git, GitHub, Axiom, Stripe | one step each | 8 | |
| 91 | | 6 | Turso, Neon, Planetscale, Prisma | one step each | 8 | |
| 92 | | 7 | Hetzner, Railway, Fly | one step each | 8 | |
| 93 | | 8 | GCP | `test/GCP` | 16 | |
| 94 | | 9 | AWS | `test/AWS` | 32 | |
| 95 | | 10 | examples | `pnpm test:examples --profile testing` | — | |
| 96 | |
| 97 | Every new top-level directory under `packages/integration-tests/test` gets a |
| 98 | step. If one is missing from this table, add it to steps 5–7 by size, and fix |
| 99 | this skill in the release PR. Never pass `--concurrency unbounded` or go |
| 100 | over 32. Never run a sequential monster file unfiltered in a fix iteration |
| 101 | (e.g. `Fly/BlueGreen.test.ts`). Its confirming run is the step's run. |
| 102 | |
| 103 | ## 3. One step |
| 104 | |
| 105 | 1. **Clean slate.** Run `pnpm nuke --yes --profile testing` and wait for it |
| 106 | to finish. Then save `pnpm nuke --dry-run --profile testing` to |
| 107 | `/tmp/release/census-<step>-before.txt`. What's left is residue |
| 108 | (`scripts/nuke.sh` excludes). A holdout that survives two nukes is a stuck |
| 109 | delete in that provider. Fix it in this step. |
| 110 | 2. **Run** the step in the background, with the output written to a file: |
| 111 | ```sh |
| 112 | source /tmp/release/env.sh |
| 113 | pnpm test:integration <paths> --profile testing --concurrency <n> \ |
| 114 | > /tmp/release/<step>-run-<k>.out 2>&1 |
| 115 | ``` |
| 116 | Wait on it with `t3_background_wait`. Start triage as soon as failures |
| 117 | stream in (`grep -E "\] ✗ |failed to run|^Tests:"`). A test running far |
| 118 | past its timeout is a hang. Read its log under `.alchemy/log/test/` and fix |
| 119 | the cause. |
| 120 | 3. **Census.** Once the run has finished, save `pnpm nuke --dry-run --profile |
| 121 | testing` to `census-<step>-after.txt` and diff it against `-before`. Every |
| 122 | new line is a leak, even from a green test. Leave leaks live while you fix |
| 123 | them, since they're your evidence. |
| 124 | 4. **Fix**, using the buckets and root-cause priority in |
| 125 | [prepare-release §4](../prepare-release/SKILL.md#4-triage): |
| 126 | provider bug > distilled patch > test fix. Never paper over a leak or an |
| 127 | untyped error in a test. Re-run only the failing files, with `-t` when a |
| 128 | file is long, until each one passes. A failure that also fails on `main` |
| 129 | is still yours to fix. |
| 130 | 5. **Budget.** Give a failure three fix iterations. If the blocker is |
| 131 | platform behavior (entitlement, slow async provisioning, beta API), gate it |
| 132 | on the typed tag with the exact error, keep an ungated probe, and move on. |
| 133 | If the blocker is the account (billing, token permissions, a retired |
| 134 | service), mark the step `blocked` with the exact error and move on. Never |
| 135 | enable billing or edit shared tokens. |
| 136 | 6. **Confirm.** Nuke, then run the **whole step once more** and take another |
| 137 | census. The step is `done` only when that single run is green and the |
| 138 | census diff is empty. Record the HEAD sha in `Passed at`. |
| 139 | 7. **Gate and push.** Run `pnpm exec tsrs -b` and `pnpm docs:check-jsdoc` if |
| 140 | any props changed. Commit one commit per fix, scoped (`fix(aws/ec2): …`), |
| 141 | then push the alchemy branch and any distilled/floci branch (§5). |
| 142 | |
| 143 | ## 4. Keep earlier steps valid |
| 144 | |
| 145 | A later fix can break an earlier step. The release needs every step to pass |
| 146 | at the **final** head, not just at the head where it once passed. After each |
| 147 | step, check what has changed since each `done` step passed: |
| 148 | |
| 149 | ```sh |
| 150 | git diff --name-only <passed-at>..HEAD |
| 151 | ``` |
| 152 | |
| 153 | Mark a step `stale` when that list touches `packages/alchemy/`, |
| 154 | `packages/alchemy-test/`, its own provider packages, a package its tests |
| 155 | deploy through (most suites deploy through `packages/cloudflare` for state), |
| 156 | or the distilled/floci pins. Before step 10, re-run every stale step (§3.1–6) |
| 157 | in order until none is left. When a fix only touches one provider's own |
| 158 | package, the other steps stay valid. |
| 159 | |
| 160 | ## 5. PRs, linked |
| 161 | |
| 162 | There are at most three PRs: alchemy, distilled and floci. Open the alchemy PR as a **draft** at |
| 163 | the first fix commit, and push after every step, so a crash at 4am loses |
| 164 | nothing. Use |
| 165 | [prepare-release §9](../prepare-release/SKILL.md#9-prs) for the standalone |
| 166 | distilled/floci clones, titles and bodies. On top of that: |
| 167 | |
| 168 | - alchemy's branch pins `submodules/distilled` / `submodules/floci` at the |
| 169 | current heads of their PR branches, and pushes a bump each time those |
| 170 | branches move. |
| 171 | - Link them both ways. Put `Requires alchemy-run/distilled#X` (and floci) in |
| 172 | the alchemy body, and `Required by alchemy-run/alchemy#Y` in each upstream |
| 173 | body. Write the merge order distilled → floci → alchemy into the alchemy |
| 174 | body. |
| 175 | - `link_pull_request` every PR to this thread. |
| 176 | - Mark the PRs ready (`gh pr ready`) only at §6, once everything is `done`. |
| 177 | |
| 178 | ## 6. Finish |
| 179 | |
| 180 | After step 10 is done and no step is stale, `git fetch origin main`. If |
| 181 | `main` has moved past `Base`, merge it, resolve any conflicts, run |
| 182 | `pnpm exec tsrs -b`, and apply §4 to the merged changes. Re-run every step it |
| 183 | marks stale, and step 10 if any step re-ran. Repeat until a fetch finds |
| 184 | nothing new. The PR must be mergeable at a head that already contains the |
| 185 | latest `main`. Then nuke one last time and take a final census. It must show residue only. Then, and only then, end the |
| 186 | turn with the report: |
| 187 | |
| 188 | ```md |
| 189 | ### Release ready at <sha> after one overnight pass. I recommend merging, in this order: |
| 190 | |
| 191 | 1. distilled — <what it types> — https://github.com/alchemy-run/distilled/pull/X |
| 192 | 2. floci — <what it fixes> — https://github.com/alchemy-run/floci/pull/Z |
| 193 | 3. alchemy — <K> fixes, <G> gated tests — https://github.com/alchemy-run/alchemy/pull/Y |
| 194 | |
| 195 | | Step | Tests | Result | Passed at | |
| 196 | | ---- | ----- | ------ | --------- | |
| 197 | | Cloudflare | 1577 | ✅ | a1b2c3d | |
| 198 | | AWS | 3361 | ✅ 3349 · 12 gated | f00ba47 | |
| 199 | | examples | 38 | ✅ | f00ba47 | |
| 200 | |
| 201 | Gated: <test → typed tag + exact error>. |
| 202 | Blocked (account): <step → exact error>. |
| 203 | Decisions taken overnight: <ones you'd have asked about>. |
| 204 | Census: clean except <documented residue>. |
| 205 | ``` |
| 206 | |
| 207 | After each upstream PR merges, the submodule has to be repointed at its |
| 208 | `main` merge commit and the steps that commit invalidates have to be re-run. |
| 209 | Say so in the report rather than claiming the release is final. Say plainly |
| 210 | what didn't run (out-of-scope steps, opt-in tags, blocked steps). Don't report |
| 211 | CI status in place of these results. |
| 212 | |
| 213 | ## Pitfalls |
| 214 | |
| 215 | Everything in |
| 216 | [prepare-release's pitfalls](../prepare-release/SKILL.md#pitfalls-seen-in-practice) |
| 217 | applies here. In addition: |
| 218 | |
| 219 | - **The SSO session expires overnight.** An AWS step failing in ~100ms at |
| 220 | setup means the token is gone. Run `aws sso login` again. If that needs a |
| 221 | human, mark the step `blocked` and go on to the non-AWS steps, then come |
| 222 | back to it. |
| 223 | - **Never overlap nuke with a run**, and never start a step's run before its |
| 224 | nuke has finished. |
| 225 | - **Steps share one account.** A leak from step 4 shows up in step 5's census |
| 226 | unless step 4's confirming census was clean. Don't move on with a dirty |
| 227 | census. |