The best way to control Claude Code, Codex and more from any computer or phone

main
md 311 lines 15.7 KB
Rendered Raw
1 # Per-PR audit procedure
2
3 Shared by the `audit` and `pr-queue` skills. Each skill builds its own list of
4 PRs in §1 (`audit`: the PRs the user names; `pr-queue`: a searched,
5 ranked shortlist) and then works through that list here, one PR at a time:
6 fix it, push it, and ask the user to merge or close it. The user merges; you
7 never do. "The list" below means whichever list the skill built.
8
9 If the audit finds a problem shared across the list (e.g. an existing suite
10 full of fakes), propose fixing that first in its own PR off main, then work
11 through the list. When the user asks, sweep main into every listed PR before
12 the one-by-one pass.
13
14 ## 2. Check out and merge main
15
16 Start from a clean tree (`git status`). Never `git stash` — the stash stack is
17 shared with other worktrees and sessions.
18
19 ```sh
20 gh pr view <n> --json headRepositoryOwner,headRepository,headRefName,baseRefName,maintainerCanModify,body,files,commits
21 gh pr checkout <n>
22 git fetch origin main && git merge origin/main
23 git submodule update --init -- submodules/distilled # checkout moved it to the PR's old pin
24 ```
25
26 After every checkout/merge, check `git status` for hook fallout: a
27 rewritten `pnpm-lock.yaml` (restore main's unless the PR changes deps), a
28 stray untracked `distilled/` git worktree at the root (`git worktree remove`),
29 regenerated example files (e.g. the Prisma fixture's `generated/`). Never
30 commit any of it.
31
32 - Fork head repos can be named differently from upstream; read
33 `headRepository`/`headRepositoryOwner` before pushing.
34 - Resolve conflicts by intent. Main wins on code it has since refactored; port
35 the PR's change onto the new shape. Many conflicts are formatting-only (main
36 moved to oxfmt) — check whether main made a real change, keep the PR's logic,
37 reformat.
38 - **A clean textual merge can still be wrong.** Read main's intervening
39 changes to the files the PR rewrites and check them semantically (a
40 dropped argument can silently disable behavior). Port the PR's tests into
41 main's restructured test file and keep main's timeouts.
42 - **A PR far behind main may no longer test anything** — main may have fixed
43 half the bug or changed the code path. Re-check after merging.
44 - **Stacks:** merge main into the base PR first and push, then merge the base
45 branch into the child. After the base squash-merges, retarget the child to
46 `main`; resolve its conflicts by confirming main's file equals the base's
47 final version.
48 - After each merge, other listed PRs touching the same files (`Providers.ts`,
49 barrel `index.ts`) may conflict — re-merge main when you pick them up.
50 - A fresh worktree needs `git submodule update --init` and `pnpm install`
51 before anything runs.
52
53 ## 3. Review the code
54
55 Read the full diff against `origin/main` and apply AGENTS.md. The standards:
56
57 - **There is a regression test.** No test changes → add one, preferably by
58 extending an existing test.
59 - **Tests live in the existing suite** for the resource (e.g.
60 `Worker.test.ts`), or extend an existing deployed fixture with a new route.
61 A new one-off test file — especially one with mocks — fails the audit. Move
62 pure-helper cases into the canonical unit file and delete the one-off.
63 - **Tests are end-to-end.** Only `test.provider` with
64 `stack.deploy`/`stack.destroy`, or `beforeAll(deploy(Stack))` fixtures. No
65 mocked HTTP/SDK/CLI responses, fake layers, direct lifecycle calls, or
66 fake-cloud engine tests. Cause drift for real through the cloud API (e.g.
67 a point-in-time restore swapped into place), then redeploy and verify. Use
68 real local infrastructure where it helps (a `registry:2` container, a kind
69 cluster, Docker Postgres + `cloudflared`). Assume Docker is installed:
70 never `skipIf` on Docker availability; a missing daemon fails the test.
71 A spy that wraps the real service is fine; a fake is not.
72 - **Also audit the existing tests the PR touches.** If a suite already fakes
73 what it claims to test, say so plainly — don't mention it in passing.
74 - **Pure utilities are the exception:** one shared implementation in
75 `src/` with a broad unit suite over in-memory fixtures is welcome. Where a
76 real tool defines the behavior (git, docker), check the cases against it.
77 - **The test covers what a reviewer would worry about** (e.g. a worker with
78 the database bound, when the change is about replacing that database).
79 - **Tests never modify checked-in files.** Copy fixtures into the gitignored
80 `packages/alchemy/.tmp` and edit the copy (import `alchemy/...` there; `@/`
81 doesn't resolve from `.tmp`).
82 - **Per-user isolation:** no hard-coded account- or zone-unique physical
83 names. Use engine-generated names, or derive them from `stack.stage`.
84 - Reconciler doctrine, Typed Error Doctrine (tagged errors, never
85 `Effect.fail(new Error(...))` or catching catch-alls by status), no
86 `Effect.orDie` in lifecycle ops, no raw Promise/`node:fs`.
87 - **Environment through `effect/Config`.** Inside Effect code, read env vars
88 with `Config.String(...)` (`Config.withDefault`/`Config.option` when
89 optional), never `process.env` — wrapping `process.env` in `Effect.sync`
90 is not enough. `Effect.sync` is for facts `Config` can't express
91 (`process.platform`, `process.cwd()`). Module-scope test gates
92 (`skipIf(!!process.env.FAST)`) are exempt. Fix violations in the PR's
93 diff, including ones the contributor wrote.
94
95 **Design review — answer these in the report before the user has to ask:**
96 does a change replace a resource, and can that lose data (a database bound to
97 a worker)? Are names spelled out rather than abbreviated
98 (`CertificateTransparencyAlerting`, not `CtAlerting`)? What are the defaults?
99 Does it slow the happy path? Are values assumed resolved that could be
100 `Output`s? Does it put things in the right namespace? Does it follow the
101 library's design rules, or is it really user error? Surface product decisions
102 (e.g. a change in how resources are named per stage) with their migration cost
103 rather than making them silently.
104
105 Fix what's fixable yourself — rewrite a contributor's mocked test into the
106 right shape, and fix the problems you flag rather than only listing them. Keep
107 the PR to its scope; out-of-scope gaps become follow-ups.
108
109 **Distilled:** before adding a patch, find the root cause upstream — the
110 vendor may have changed the schema on purpose. A new typed error goes in its
111 own distilled PR; once merged, point the submodule at the distilled `main`
112 commit, never a side branch. A distilled bump with breaking fallout gets its
113 own alchemy PR (`fix(...)` if it repairs something); trim the contributor's PR
114 back to its own files. Unexplained cross-provider diffs in a PR confuse the
115 user — remove them or explain them.
116
117 **Bugs on main** found while testing get their own PR, merged first.
118
119 **Close or park:**
120 - Recommend closing when superseded on main, a duplicate, or the wrong
121 approach. Cite what supersedes it. For overlapping PRs, run the weaker PR's
122 tests against the stronger PR's code, and credit the closed author.
123 - If a closed PR has one good unrelated part, suggest the author narrow it.
124 - If a bug can't be reproduced, ask the author for a repro, park it, move on.
125 Treat upvotes as a reason to ask rather than close.
126 - Parked PRs wait for an external update; don't keep reporting "nothing
127 changed" on them.
128
129 ## 4. Update JSDoc
130
131 Every changed or new prop/attribute has field-level JSDoc (`@default` where
132 relevant), and the resource-level JSDoc has `###` sections with
133 `**Example:**` snippets for the new behavior, metadata tags last. Run
134 `pnpm docs:check-jsdoc`, and `pnpm docs:gen` to check generation. The
135 generated pages under `website/src/content/docs/providers/` are gitignored —
136 remove any a contributor committed.
137
138 ## 5. Run the tests
139
140 - Run **every touched test file in full**, plus the suites covering the
141 changed source — not just the new case via `-t`. Wide-impact changes get
142 the broad suites (e.g. all of `test/Cloudflare/Workers`, plus
143 `test:examples`). Dev-mode changes also need the `*.local.test.ts` suites.
144 - **Prove it fails on main.** Commit first, then put the `src/` files back to
145 main's version (`git show origin/main:<path> > <path>`), run the new case,
146 confirm it fails *for the right reason*, and restore from git (never from a
147 `/tmp` backup — a crash loses it, and `git checkout <file>` discards
148 uncommitted edits). A test that passes on main isn't testing the bug: an
149 identical redeploy plans a no-op and never reaches `reconcile`, an unchanged
150 fixture reuses a cached image, dev mode skips the code path. Check the logs
151 or timing to confirm the path actually ran. `packages/frontend-frameworks`
152 is consumed from `dist/`, so rebuild with `npx tsdown` around this check.
153 - Run with `pnpm test <file> --profile testing`. "0 files"
154 or "0 tests" is an invocation bug (bad path, zsh word-splitting — use
155 arrays or `xargs`), never a pass.
156 - Credentials: for Cloudflare live, `set -a; source .env; set +a`
157 (`pnpm download:env` if missing). AWS live uses SSO; a ~100ms setup failure
158 means an expired session — ask the user to run `aws sso login`. Never set or
159 unset `CI` for tests: a profile-store migration can wipe `~/.alchemy`.
160 - Flaky failures: re-run the file alone, run it on main, then alternate. Report
161 a verdict table: `Failure | Re-run alone | On main | Verdict`. Lower
162 `--concurrency` (e.g. 8) for a clean signal on big suites. A flake you fix
163 gets its own PR.
164 - Don't overlap two runs of the same stack: the second run's opening
165 `stack.destroy()` tears down the first's resources.
166 - Runs against main and interrupted runs leak resources. Clean up (delete
167 out-of-band resources before `stack.destroy()`) and verify out-of-band that
168 nothing is left. `OwnedBySomeoneElse` on a named resource is a leftover
169 from an earlier run — ask before deleting anything, offering a full
170 `pnpm nuke` or deleting only that stack/stage's resources.
171 - After a crash: check for surviving test processes, files still swapped to
172 main, leaked resources, and background agents' progress before rerunning.
173 - Long suites block the worktree. Advance the next PR in a separate worktree
174 meanwhile. A large refactor can go to a background agent in its own
175 worktree with "commit, don't push"; review its result before pushing.
176 - Before pushing, run the CI Check steps your change can break:
177 `pnpm exec tsrs -b`, `pnpm lint`, `pnpm format:check`,
178 `pnpm docs:check-jsdoc` and `pnpm exports:check`. They're in
179 `.github/workflows/check.yml`. A new test file or fixture is type-checked
180 too.
181
182 ## 6. Push and update the PR
183
184 - **Push every change as soon as it's made** — the user reviews GitHub, not
185 your local branch.
186 - Before each push: `git fetch` and merge any new commits from the author;
187 check `git diff --stat origin/main...HEAD` for unrelated files the
188 repo-wide pre-commit formatter swept in, and revert them. Run
189 `pnpm exec oxfmt --check` on every file the PR changes and read the whole
190 output.
191 - **Leave every PR with CI passing.** After each push, wait for every check
192 on the new head to finish (`gh pr checks <n>`). When one fails, read its log
193 (`gh run view <id> --log-failed`), fix the cause, push, and wait again.
194 Don't end your turn while a local suite is still running: wait on it with
195 `t3_background_wait`, since ending the turn is your report. To wait on CI,
196 `watch_pull_request` the PR after the push and end the turn; you're woken
197 when its checks finish. Ask to merge only once they pass, then
198 `unwatch_pull_request`.
199 - Make sure CI actually runs. First-time contributors' runs sit at
200 `action_required` and pushes don't start them. Read the whole diff first:
201 approve the runs (`gh run approve <id>`) only when it doesn't touch
202 `.github/`, scripts CI executes, or anything that reads secrets or sends
203 data out. Otherwise ask the user. Don't approve runs on parked PRs. If no
204 checks appear at all, find out why (a draft, a push that didn't trigger
205 `pull_request`) and get them started.
206 - "Ready" means: tests pass locally, every CI check on the final head has
207 passed, not a draft, mergeable. A pending, failing or empty check list is
208 not ready. Then `gh pr ready <n>`.
209
210 **Rewrite the description to show the developer experience** — every PR you
211 touch, even when the author's body looks fine. Explain the change through the
212 code a user writes, not through prose or bullet points about the
213 implementation. Contributor bodies usually arrive as Change/Tests/Checks
214 bullet lists; replace them rather than appending to them.
215
216 - one plain sentence on what changes and why, no heading above it
217 - then `ts` snippets of the user's code that is new or now works — the props,
218 values and variants they can now write. For a fix or a changed API, a
219 before/after `diff` of the user's code
220 - check every API signature in a snippet against the source
221 - prose only for the "why" a snippet can't show, one or two sentences
222 - no `#`/`##` headings, bullet dumps, Change/Tests/Checks sections or test
223 plans
224 - end with the attribution line your harness asks for, if any
225
226 Bad — describes the implementation:
227
228 ```md
229 ## Changes
230 - Added `retention` prop to `LogGroup`
231 - Updated reconcile to call `putRetentionPolicy` when retention changes
232 - Added tests
233 ```
234
235 Good — shows the user's code:
236
237 ````md
238 `LogGroup` now takes a retention period, so logs stop growing forever.
239
240 ```ts
241 const logs = yield* AWS.Logs.LogGroup("Logs", {
242 retention: "30 days", // or "forever"
243 });
244 ```
245
246 Changing `retention` updates the group in place; it no longer replaces it.
247 ````
248
249 Write it to a file:
250
251 ```sh
252 gh pr edit <n> --body-file /tmp/pr-<n>.md
253 # gh pr edit can fail on the deprecated Projects (classic) API — use REST:
254 gh api -X PATCH repos/alchemy-run/alchemy/pulls/<n> -F body=@/tmp/pr-<n>.md
255 gh api -X PATCH repos/alchemy-run/alchemy/pulls/<n> -f base=main # retarget
256 ```
257
258 **Comments posted on GitHub must stand alone** — no references to this
259 conversation. State what's allowed, what was fixed, and why the rule exists,
260 with a code example.
261
262 If the T3 `link_pull_request` tool is available, link every PR you touch or
263 open to this thread.
264
265 ## 7. Report and ask
266
267 Keep it short — longer explanations only when asked. End the turn after each
268 PR with:
269
270 ```md
271 ### #N: <behavior>. I recommend merging.
272
273 <Why: the bug or gap, with the error text if there is one.>
274 <The fix, often a short diff.>
275 <What I changed on top of the author's work.>
276
277 | Test | Code | Result |
278 | --- | --- | --- |
279 | test/AWS/RDS/DBInstance.test.ts (full, live) | PR | ✅ 6/6 |
280 | … "refresh after restore" | main (expected to fail) | ❌ fails as expected: <error> |
281
282 Not run: <suites and why>.
283 <Design-review answers, decisions for you, follow-ups.>
284 Next in the list: #M (omit when it was the only one).
285
286 https://github.com/alchemy-run/alchemy/pull/N
287 ```
288
289 - Say which tests ran and whether they passed. Every CI check must have
290 passed on the head you report before you call it "ready", but CI status
291 doesn't replace the test report.
292 - Label deliberate runs against main as expected failures.
293 - Say plainly what was **not** run.
294 - When fixes span several PRs, say which PR holds each fix and give the merge
295 order as a numbered list with one URL per line, closes included.
296 - When a decision is the user's, give numbered options with a recommendation.
297 - **The last line of every message about a PR is its full URL** — progress
298 notes and questions included. Several PRs → all URLs at the end.
299
300 ## 8. Keep the list moving
301
302 - Don't wait idle on a merge: start the next PR and keep the pending link in
303 each message.
304 - On "merged, next": check it really merged (`gh pr view <n> --json state` —
305 the user sometimes names the wrong number), re-merge the new main into the
306 next PR, then continue.
307 - On "merge", "close", "park", "skip": update the table and act. Close only on
308 explicit instruction, with a standalone comment crediting the author and
309 naming what supersedes it.
310 - Answer "why" or "justify" questions directly with the error text and a diff
311 snippet. "Refresh my memory" gets the status table plus open items.