@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 07:15 UTC

main
md 227 lines 11 KB
Rendered Raw
1 ---
2 name: release-overnight
3 description: Prepare an alchemy release unattended overnight, in series — after one go-ahead, nuke the testing account, then walk every provider's live suite one at a time (Cloudflare → core unit → core integration → small providers → Fly → GCP → AWS), fixing each until its whole suite passes in one run AND the account census is clean, then run `pnpm test:examples`, and finish with one alchemy PR plus at most one distilled PR and one floci PR, all linked, that merge into a release passing every test. Use for "prepare the release overnight", "/release-overnight", "run the release suite in series while I sleep"; for the parallel, interactive version use prepare-release.
4 metadata:
5 internal: true
6 ---
7
8 # Release overnight
9
10 This thread runs the whole release unattended, **one suite at a time**. It owns
11 every step: the clean slate, each run, the fixes, type-checking, the census
12 and the PRs. It launches no subagents or threads, and it runs nothing in
13 parallel with a live suite. Going slowly in series is the point. A live suite
14 under load produces fake failures, and parallel fixers on one account leak
15 into each other's census.
16
17 A provider is **done** when its whole suite passes in one run at the current
18 head and the census shows nothing new. The release is done when every
19 provider is done at the final head, `pnpm test:examples` passes, and the
20 account is clean. The user merges; you never do.
21
22 ## 0. One question, then unattended
23
24 Ask once, up front, in a single question. Confirm three things: the scope
25 (default: everything in §2), that you'll nuke the testing account before each
26 step for the whole run, and any opt-in tags or paid add-ons to include
27 (default: none). **Nuking is destructive, so get an explicit yes.** After
28 that, never ask again. Take every decision yourself, record it in
29 `STATE.md`, and keep going. An answer you would have asked for goes in
30 the final report.
31
32 Don't end the turn until §6 is done. Ending the turn ends the night. Wait on
33 long runs with `t3_background_wait`, never with `sleep` or a polling loop.
34
35 ## 1. Preflight and durable state
36
37 Start from the latest `origin/main`, on a fresh branch:
38
39 ```sh
40 git status # must be clean; never git stash
41 git fetch origin main
42 git checkout -b release/overnight-$(date +%Y-%m-%d) origin/main
43 git log -1 --format='%H %s' origin/main # record as "Base" in STATE.md
44 git submodule update --init -- submodules/distilled
45 ```
46
47 Then run the rest of
48 [prepare-release §0](../prepare-release/SKILL.md#0-preflight): `pnpm install`,
49 pinned bun/node in `/tmp/release/env.sh`, `.env`, `aws sso login`, and
50 `pnpm exec tsrs -b`.
51
52 Don't merge `main` into the branch mid-run. A moving base would make every
53 `Passed at` meaningless. §6 picks up whatever landed overnight.
54
55 Keep the run's state in `/tmp/release/STATE.md` and update it after every
56 event. Context will be compacted many times overnight, and this file is how
57 you resume:
58
59 ```md
60 # Release overnight 2026-10-31 — branch release/overnight-2026-10-31
61 Base: origin/main <sha> · Scope: all · opt-in: none · nuke authorized: yes
62 PRs: alchemy #— · distilled #— · floci #—
63
64 | # | Step | Command | Status | Passed at | Notes |
65 | - | ---- | ------- | ------ | --------- | ----- |
66 | 1 | Cloudflare | pnpm test:integration test/Cloudflare | done | a1b2c3d | |
67 | 2 | core unit | pnpm test | fixing (iter 2) | | Plan diff regression |
68
69 Gated: <test → typed tag + exact error>
70 Blocked (account, not code): <step → exact error>
71 ```
72
73 **On every start, read `STATE.md` first.** If it exists, resume at the first
74 step that isn't `done`. Don't restart the night.
75
76 ## 2. The steps, in order
77
78 Cloudflare goes first. State and most other suites deploy through
79 `packages/cloudflare`, so getting it green first keeps its failures out of
80 every later step. After that, cheap and narrow first, so a broken shared
81 layer shows up in minutes, not after AWS. Paths are relative to
82 `packages/integration-tests`:
83
84 | # | Step | Paths | `--concurrency` |
85 | - | ---- | ----- | --------------- |
86 | 1 | Cloudflare | `test/Cloudflare` | 16 |
87 | 2 | core unit | `pnpm test` (packages/alchemy) | default |
88 | 3 | core integration | `test/*.test.ts test/Bundle test/Cli test/Local test/Runtime test/Util test/Website` | 8 |
89 | 4 | building blocks | `test/Docker test/Redis test/Kubernetes test/SQL test/Drizzle test/Domain test/Auth test/DNS test/ACME` | 8 |
90 | 5 | Git, GitHub, Axiom, Stripe | one step each | 8 |
91 | 6 | Turso, Neon, Planetscale, Prisma | one step each | 8 |
92 | 7 | Hetzner, Railway, Fly | one step each | 8 |
93 | 8 | GCP | `test/GCP` | 16 |
94 | 9 | AWS | `test/AWS` | 32 |
95 | 10 | examples | `pnpm test:examples --profile testing` | — |
96
97 Every new top-level directory under `packages/integration-tests/test` gets a
98 step. If one is missing from this table, add it to steps 5–7 by size, and fix
99 this skill in the release PR. Never pass `--concurrency unbounded` or go
100 over 32. Never run a sequential monster file unfiltered in a fix iteration
101 (e.g. `Fly/BlueGreen.test.ts`). Its confirming run is the step's run.
102
103 ## 3. One step
104
105 1. **Clean slate.** Run `pnpm nuke --yes --profile testing` and wait for it
106 to finish. Then save `pnpm nuke --dry-run --profile testing` to
107 `/tmp/release/census-<step>-before.txt`. What's left is residue
108 (`scripts/nuke.sh` excludes). A holdout that survives two nukes is a stuck
109 delete in that provider. Fix it in this step.
110 2. **Run** the step in the background, with the output written to a file:
111 ```sh
112 source /tmp/release/env.sh
113 pnpm test:integration <paths> --profile testing --concurrency <n> \
114 > /tmp/release/<step>-run-<k>.out 2>&1
115 ```
116 Wait on it with `t3_background_wait`. Start triage as soon as failures
117 stream in (`grep -E "\] ✗ |failed to run|^Tests:"`). A test running far
118 past its timeout is a hang. Read its log under `.alchemy/log/test/` and fix
119 the cause.
120 3. **Census.** Once the run has finished, save `pnpm nuke --dry-run --profile
121 testing` to `census-<step>-after.txt` and diff it against `-before`. Every
122 new line is a leak, even from a green test. Leave leaks live while you fix
123 them, since they're your evidence.
124 4. **Fix**, using the buckets and root-cause priority in
125 [prepare-release §4](../prepare-release/SKILL.md#4-triage):
126 provider bug > distilled patch > test fix. Never paper over a leak or an
127 untyped error in a test. Re-run only the failing files, with `-t` when a
128 file is long, until each one passes. A failure that also fails on `main`
129 is still yours to fix.
130 5. **Budget.** Give a failure three fix iterations. If the blocker is
131 platform behavior (entitlement, slow async provisioning, beta API), gate it
132 on the typed tag with the exact error, keep an ungated probe, and move on.
133 If the blocker is the account (billing, token permissions, a retired
134 service), mark the step `blocked` with the exact error and move on. Never
135 enable billing or edit shared tokens.
136 6. **Confirm.** Nuke, then run the **whole step once more** and take another
137 census. The step is `done` only when that single run is green and the
138 census diff is empty. Record the HEAD sha in `Passed at`.
139 7. **Gate and push.** Run `pnpm exec tsrs -b` and `pnpm docs:check-jsdoc` if
140 any props changed. Commit one commit per fix, scoped (`fix(aws/ec2): …`),
141 then push the alchemy branch and any distilled/floci branch (§5).
142
143 ## 4. Keep earlier steps valid
144
145 A later fix can break an earlier step. The release needs every step to pass
146 at the **final** head, not just at the head where it once passed. After each
147 step, check what has changed since each `done` step passed:
148
149 ```sh
150 git diff --name-only <passed-at>..HEAD
151 ```
152
153 Mark a step `stale` when that list touches `packages/alchemy/`,
154 `packages/alchemy-test/`, its own provider packages, a package its tests
155 deploy through (most suites deploy through `packages/cloudflare` for state),
156 or the distilled/floci pins. Before step 10, re-run every stale step (§3.1–6)
157 in order until none is left. When a fix only touches one provider's own
158 package, the other steps stay valid.
159
160 ## 5. PRs, linked
161
162 There are at most three PRs: alchemy, distilled and floci. Open the alchemy PR as a **draft** at
163 the first fix commit, and push after every step, so a crash at 4am loses
164 nothing. Use
165 [prepare-release §9](../prepare-release/SKILL.md#9-prs) for the standalone
166 distilled/floci clones, titles and bodies. On top of that:
167
168 - alchemy's branch pins `submodules/distilled` / `submodules/floci` at the
169 current heads of their PR branches, and pushes a bump each time those
170 branches move.
171 - Link them both ways. Put `Requires alchemy-run/distilled#X` (and floci) in
172 the alchemy body, and `Required by alchemy-run/alchemy#Y` in each upstream
173 body. Write the merge order distilled → floci → alchemy into the alchemy
174 body.
175 - `link_pull_request` every PR to this thread.
176 - Mark the PRs ready (`gh pr ready`) only at §6, once everything is `done`.
177
178 ## 6. Finish
179
180 After step 10 is done and no step is stale, `git fetch origin main`. If
181 `main` has moved past `Base`, merge it, resolve any conflicts, run
182 `pnpm exec tsrs -b`, and apply §4 to the merged changes. Re-run every step it
183 marks stale, and step 10 if any step re-ran. Repeat until a fetch finds
184 nothing new. The PR must be mergeable at a head that already contains the
185 latest `main`. Then nuke one last time and take a final census. It must show residue only. Then, and only then, end the
186 turn with the report:
187
188 ```md
189 ### Release ready at <sha> after one overnight pass. I recommend merging, in this order:
190
191 1. distilled — <what it types> — https://github.com/alchemy-run/distilled/pull/X
192 2. floci — <what it fixes> — https://github.com/alchemy-run/floci/pull/Z
193 3. alchemy — <K> fixes, <G> gated tests — https://github.com/alchemy-run/alchemy/pull/Y
194
195 | Step | Tests | Result | Passed at |
196 | ---- | ----- | ------ | --------- |
197 | Cloudflare | 1577 | ✅ | a1b2c3d |
198 | AWS | 3361 | ✅ 3349 · 12 gated | f00ba47 |
199 | examples | 38 | ✅ | f00ba47 |
200
201 Gated: <test → typed tag + exact error>.
202 Blocked (account): <step → exact error>.
203 Decisions taken overnight: <ones you'd have asked about>.
204 Census: clean except <documented residue>.
205 ```
206
207 After each upstream PR merges, the submodule has to be repointed at its
208 `main` merge commit and the steps that commit invalidates have to be re-run.
209 Say so in the report rather than claiming the release is final. Say plainly
210 what didn't run (out-of-scope steps, opt-in tags, blocked steps). Don't report
211 CI status in place of these results.
212
213 ## Pitfalls
214
215 Everything in
216 [prepare-release's pitfalls](../prepare-release/SKILL.md#pitfalls-seen-in-practice)
217 applies here. In addition:
218
219 - **The SSO session expires overnight.** An AWS step failing in ~100ms at
220 setup means the token is gone. Run `aws sso login` again. If that needs a
221 human, mark the step `blocked` and go on to the non-AWS steps, then come
222 back to it.
223 - **Never overlap nuke with a run**, and never start a step's run before its
224 nuke has finished.
225 - **Steps share one account.** A leak from step 4 shows up in step 5's census
226 unless step 4's confirming census was clean. Don't move on with a dirty
227 census.