The best way to control Claude Code, Codex and more from any computer or phone
Mirrored from https://github.com/Camo67/t3code.git
Synced Nov 21, 07:15 UTC
| 1 | # Per-PR audit procedure |
| 2 | |
| 3 | Shared by the `audit` and `pr-queue` skills. Each skill builds its own list of |
| 4 | PRs in §1 (`audit`: the PRs the user names; `pr-queue`: a searched, |
| 5 | ranked shortlist) and then works through that list here, one PR at a time: |
| 6 | fix it, push it, and ask the user to merge or close it. The user merges; you |
| 7 | never do. "The list" below means whichever list the skill built. |
| 8 | |
| 9 | If the audit finds a problem shared across the list (e.g. an existing suite |
| 10 | full of fakes), propose fixing that first in its own PR off main, then work |
| 11 | through the list. When the user asks, sweep main into every listed PR before |
| 12 | the one-by-one pass. |
| 13 | |
| 14 | ## 2. Check out and merge main |
| 15 | |
| 16 | Start from a clean tree (`git status`). Never `git stash` — the stash stack is |
| 17 | shared with other worktrees and sessions. |
| 18 | |
| 19 | ```sh |
| 20 | gh pr view <n> --json headRepositoryOwner,headRepository,headRefName,baseRefName,maintainerCanModify,body,files,commits |
| 21 | gh pr checkout <n> |
| 22 | git fetch origin main && git merge origin/main |
| 23 | git submodule update --init -- submodules/distilled # checkout moved it to the PR's old pin |
| 24 | ``` |
| 25 | |
| 26 | After every checkout/merge, check `git status` for hook fallout: a |
| 27 | rewritten `pnpm-lock.yaml` (restore main's unless the PR changes deps), a |
| 28 | stray untracked `distilled/` git worktree at the root (`git worktree remove`), |
| 29 | regenerated example files (e.g. the Prisma fixture's `generated/`). Never |
| 30 | commit any of it. |
| 31 | |
| 32 | - Fork head repos can be named differently from upstream; read |
| 33 | `headRepository`/`headRepositoryOwner` before pushing. |
| 34 | - Resolve conflicts by intent. Main wins on code it has since refactored; port |
| 35 | the PR's change onto the new shape. Many conflicts are formatting-only (main |
| 36 | moved to oxfmt) — check whether main made a real change, keep the PR's logic, |
| 37 | reformat. |
| 38 | - **A clean textual merge can still be wrong.** Read main's intervening |
| 39 | changes to the files the PR rewrites and check them semantically (a |
| 40 | dropped argument can silently disable behavior). Port the PR's tests into |
| 41 | main's restructured test file and keep main's timeouts. |
| 42 | - **A PR far behind main may no longer test anything** — main may have fixed |
| 43 | half the bug or changed the code path. Re-check after merging. |
| 44 | - **Stacks:** merge main into the base PR first and push, then merge the base |
| 45 | branch into the child. After the base squash-merges, retarget the child to |
| 46 | `main`; resolve its conflicts by confirming main's file equals the base's |
| 47 | final version. |
| 48 | - After each merge, other listed PRs touching the same files (`Providers.ts`, |
| 49 | barrel `index.ts`) may conflict — re-merge main when you pick them up. |
| 50 | - A fresh worktree needs `git submodule update --init` and `pnpm install` |
| 51 | before anything runs. |
| 52 | |
| 53 | ## 3. Review the code |
| 54 | |
| 55 | Read the full diff against `origin/main` and apply AGENTS.md. The standards: |
| 56 | |
| 57 | - **There is a regression test.** No test changes → add one, preferably by |
| 58 | extending an existing test. |
| 59 | - **Tests live in the existing suite** for the resource (e.g. |
| 60 | `Worker.test.ts`), or extend an existing deployed fixture with a new route. |
| 61 | A new one-off test file — especially one with mocks — fails the audit. Move |
| 62 | pure-helper cases into the canonical unit file and delete the one-off. |
| 63 | - **Tests are end-to-end.** Only `test.provider` with |
| 64 | `stack.deploy`/`stack.destroy`, or `beforeAll(deploy(Stack))` fixtures. No |
| 65 | mocked HTTP/SDK/CLI responses, fake layers, direct lifecycle calls, or |
| 66 | fake-cloud engine tests. Cause drift for real through the cloud API (e.g. |
| 67 | a point-in-time restore swapped into place), then redeploy and verify. Use |
| 68 | real local infrastructure where it helps (a `registry:2` container, a kind |
| 69 | cluster, Docker Postgres + `cloudflared`). Assume Docker is installed: |
| 70 | never `skipIf` on Docker availability; a missing daemon fails the test. |
| 71 | A spy that wraps the real service is fine; a fake is not. |
| 72 | - **Also audit the existing tests the PR touches.** If a suite already fakes |
| 73 | what it claims to test, say so plainly — don't mention it in passing. |
| 74 | - **Pure utilities are the exception:** one shared implementation in |
| 75 | `src/` with a broad unit suite over in-memory fixtures is welcome. Where a |
| 76 | real tool defines the behavior (git, docker), check the cases against it. |
| 77 | - **The test covers what a reviewer would worry about** (e.g. a worker with |
| 78 | the database bound, when the change is about replacing that database). |
| 79 | - **Tests never modify checked-in files.** Copy fixtures into the gitignored |
| 80 | `packages/alchemy/.tmp` and edit the copy (import `alchemy/...` there; `@/` |
| 81 | doesn't resolve from `.tmp`). |
| 82 | - **Per-user isolation:** no hard-coded account- or zone-unique physical |
| 83 | names. Use engine-generated names, or derive them from `stack.stage`. |
| 84 | - Reconciler doctrine, Typed Error Doctrine (tagged errors, never |
| 85 | `Effect.fail(new Error(...))` or catching catch-alls by status), no |
| 86 | `Effect.orDie` in lifecycle ops, no raw Promise/`node:fs`. |
| 87 | - **Environment through `effect/Config`.** Inside Effect code, read env vars |
| 88 | with `Config.String(...)` (`Config.withDefault`/`Config.option` when |
| 89 | optional), never `process.env` — wrapping `process.env` in `Effect.sync` |
| 90 | is not enough. `Effect.sync` is for facts `Config` can't express |
| 91 | (`process.platform`, `process.cwd()`). Module-scope test gates |
| 92 | (`skipIf(!!process.env.FAST)`) are exempt. Fix violations in the PR's |
| 93 | diff, including ones the contributor wrote. |
| 94 | |
| 95 | **Design review — answer these in the report before the user has to ask:** |
| 96 | does a change replace a resource, and can that lose data (a database bound to |
| 97 | a worker)? Are names spelled out rather than abbreviated |
| 98 | (`CertificateTransparencyAlerting`, not `CtAlerting`)? What are the defaults? |
| 99 | Does it slow the happy path? Are values assumed resolved that could be |
| 100 | `Output`s? Does it put things in the right namespace? Does it follow the |
| 101 | library's design rules, or is it really user error? Surface product decisions |
| 102 | (e.g. a change in how resources are named per stage) with their migration cost |
| 103 | rather than making them silently. |
| 104 | |
| 105 | Fix what's fixable yourself — rewrite a contributor's mocked test into the |
| 106 | right shape, and fix the problems you flag rather than only listing them. Keep |
| 107 | the PR to its scope; out-of-scope gaps become follow-ups. |
| 108 | |
| 109 | **Distilled:** before adding a patch, find the root cause upstream — the |
| 110 | vendor may have changed the schema on purpose. A new typed error goes in its |
| 111 | own distilled PR; once merged, point the submodule at the distilled `main` |
| 112 | commit, never a side branch. A distilled bump with breaking fallout gets its |
| 113 | own alchemy PR (`fix(...)` if it repairs something); trim the contributor's PR |
| 114 | back to its own files. Unexplained cross-provider diffs in a PR confuse the |
| 115 | user — remove them or explain them. |
| 116 | |
| 117 | **Bugs on main** found while testing get their own PR, merged first. |
| 118 | |
| 119 | **Close or park:** |
| 120 | - Recommend closing when superseded on main, a duplicate, or the wrong |
| 121 | approach. Cite what supersedes it. For overlapping PRs, run the weaker PR's |
| 122 | tests against the stronger PR's code, and credit the closed author. |
| 123 | - If a closed PR has one good unrelated part, suggest the author narrow it. |
| 124 | - If a bug can't be reproduced, ask the author for a repro, park it, move on. |
| 125 | Treat upvotes as a reason to ask rather than close. |
| 126 | - Parked PRs wait for an external update; don't keep reporting "nothing |
| 127 | changed" on them. |
| 128 | |
| 129 | ## 4. Update JSDoc |
| 130 | |
| 131 | Every changed or new prop/attribute has field-level JSDoc (`@default` where |
| 132 | relevant), and the resource-level JSDoc has `###` sections with |
| 133 | `**Example:**` snippets for the new behavior, metadata tags last. Run |
| 134 | `pnpm docs:check-jsdoc`, and `pnpm docs:gen` to check generation. The |
| 135 | generated pages under `website/src/content/docs/providers/` are gitignored — |
| 136 | remove any a contributor committed. |
| 137 | |
| 138 | ## 5. Run the tests |
| 139 | |
| 140 | - Run **every touched test file in full**, plus the suites covering the |
| 141 | changed source — not just the new case via `-t`. Wide-impact changes get |
| 142 | the broad suites (e.g. all of `test/Cloudflare/Workers`, plus |
| 143 | `test:examples`). Dev-mode changes also need the `*.local.test.ts` suites. |
| 144 | - **Prove it fails on main.** Commit first, then put the `src/` files back to |
| 145 | main's version (`git show origin/main:<path> > <path>`), run the new case, |
| 146 | confirm it fails *for the right reason*, and restore from git (never from a |
| 147 | `/tmp` backup — a crash loses it, and `git checkout <file>` discards |
| 148 | uncommitted edits). A test that passes on main isn't testing the bug: an |
| 149 | identical redeploy plans a no-op and never reaches `reconcile`, an unchanged |
| 150 | fixture reuses a cached image, dev mode skips the code path. Check the logs |
| 151 | or timing to confirm the path actually ran. `packages/frontend-frameworks` |
| 152 | is consumed from `dist/`, so rebuild with `npx tsdown` around this check. |
| 153 | - Run with `pnpm test <file> --profile testing`. "0 files" |
| 154 | or "0 tests" is an invocation bug (bad path, zsh word-splitting — use |
| 155 | arrays or `xargs`), never a pass. |
| 156 | - Credentials: for Cloudflare live, `set -a; source .env; set +a` |
| 157 | (`pnpm download:env` if missing). AWS live uses SSO; a ~100ms setup failure |
| 158 | means an expired session — ask the user to run `aws sso login`. Never set or |
| 159 | unset `CI` for tests: a profile-store migration can wipe `~/.alchemy`. |
| 160 | - Flaky failures: re-run the file alone, run it on main, then alternate. Report |
| 161 | a verdict table: `Failure | Re-run alone | On main | Verdict`. Lower |
| 162 | `--concurrency` (e.g. 8) for a clean signal on big suites. A flake you fix |
| 163 | gets its own PR. |
| 164 | - Don't overlap two runs of the same stack: the second run's opening |
| 165 | `stack.destroy()` tears down the first's resources. |
| 166 | - Runs against main and interrupted runs leak resources. Clean up (delete |
| 167 | out-of-band resources before `stack.destroy()`) and verify out-of-band that |
| 168 | nothing is left. `OwnedBySomeoneElse` on a named resource is a leftover |
| 169 | from an earlier run — ask before deleting anything, offering a full |
| 170 | `pnpm nuke` or deleting only that stack/stage's resources. |
| 171 | - After a crash: check for surviving test processes, files still swapped to |
| 172 | main, leaked resources, and background agents' progress before rerunning. |
| 173 | - Long suites block the worktree. Advance the next PR in a separate worktree |
| 174 | meanwhile. A large refactor can go to a background agent in its own |
| 175 | worktree with "commit, don't push"; review its result before pushing. |
| 176 | - Before pushing, run the CI Check steps your change can break: |
| 177 | `pnpm exec tsrs -b`, `pnpm lint`, `pnpm format:check`, |
| 178 | `pnpm docs:check-jsdoc` and `pnpm exports:check`. They're in |
| 179 | `.github/workflows/check.yml`. A new test file or fixture is type-checked |
| 180 | too. |
| 181 | |
| 182 | ## 6. Push and update the PR |
| 183 | |
| 184 | - **Push every change as soon as it's made** — the user reviews GitHub, not |
| 185 | your local branch. |
| 186 | - Before each push: `git fetch` and merge any new commits from the author; |
| 187 | check `git diff --stat origin/main...HEAD` for unrelated files the |
| 188 | repo-wide pre-commit formatter swept in, and revert them. Run |
| 189 | `pnpm exec oxfmt --check` on every file the PR changes and read the whole |
| 190 | output. |
| 191 | - **Leave every PR with CI passing.** After each push, wait for every check |
| 192 | on the new head to finish (`gh pr checks <n>`). When one fails, read its log |
| 193 | (`gh run view <id> --log-failed`), fix the cause, push, and wait again. |
| 194 | Don't end your turn while a local suite is still running: wait on it with |
| 195 | `t3_background_wait`, since ending the turn is your report. To wait on CI, |
| 196 | `watch_pull_request` the PR after the push and end the turn; you're woken |
| 197 | when its checks finish. Ask to merge only once they pass, then |
| 198 | `unwatch_pull_request`. |
| 199 | - Make sure CI actually runs. First-time contributors' runs sit at |
| 200 | `action_required` and pushes don't start them. Read the whole diff first: |
| 201 | approve the runs (`gh run approve <id>`) only when it doesn't touch |
| 202 | `.github/`, scripts CI executes, or anything that reads secrets or sends |
| 203 | data out. Otherwise ask the user. Don't approve runs on parked PRs. If no |
| 204 | checks appear at all, find out why (a draft, a push that didn't trigger |
| 205 | `pull_request`) and get them started. |
| 206 | - "Ready" means: tests pass locally, every CI check on the final head has |
| 207 | passed, not a draft, mergeable. A pending, failing or empty check list is |
| 208 | not ready. Then `gh pr ready <n>`. |
| 209 | |
| 210 | **Rewrite the description to show the developer experience** — every PR you |
| 211 | touch, even when the author's body looks fine. Explain the change through the |
| 212 | code a user writes, not through prose or bullet points about the |
| 213 | implementation. Contributor bodies usually arrive as Change/Tests/Checks |
| 214 | bullet lists; replace them rather than appending to them. |
| 215 | |
| 216 | - one plain sentence on what changes and why, no heading above it |
| 217 | - then `ts` snippets of the user's code that is new or now works — the props, |
| 218 | values and variants they can now write. For a fix or a changed API, a |
| 219 | before/after `diff` of the user's code |
| 220 | - check every API signature in a snippet against the source |
| 221 | - prose only for the "why" a snippet can't show, one or two sentences |
| 222 | - no `#`/`##` headings, bullet dumps, Change/Tests/Checks sections or test |
| 223 | plans |
| 224 | - end with the attribution line your harness asks for, if any |
| 225 | |
| 226 | Bad — describes the implementation: |
| 227 | |
| 228 | ```md |
| 229 | ## Changes |
| 230 | - Added `retention` prop to `LogGroup` |
| 231 | - Updated reconcile to call `putRetentionPolicy` when retention changes |
| 232 | - Added tests |
| 233 | ``` |
| 234 | |
| 235 | Good — shows the user's code: |
| 236 | |
| 237 | ````md |
| 238 | `LogGroup` now takes a retention period, so logs stop growing forever. |
| 239 | |
| 240 | ```ts |
| 241 | const logs = yield* AWS.Logs.LogGroup("Logs", { |
| 242 | retention: "30 days", // or "forever" |
| 243 | }); |
| 244 | ``` |
| 245 | |
| 246 | Changing `retention` updates the group in place; it no longer replaces it. |
| 247 | ```` |
| 248 | |
| 249 | Write it to a file: |
| 250 | |
| 251 | ```sh |
| 252 | gh pr edit <n> --body-file /tmp/pr-<n>.md |
| 253 | # gh pr edit can fail on the deprecated Projects (classic) API — use REST: |
| 254 | gh api -X PATCH repos/alchemy-run/alchemy/pulls/<n> -F body=@/tmp/pr-<n>.md |
| 255 | gh api -X PATCH repos/alchemy-run/alchemy/pulls/<n> -f base=main # retarget |
| 256 | ``` |
| 257 | |
| 258 | **Comments posted on GitHub must stand alone** — no references to this |
| 259 | conversation. State what's allowed, what was fixed, and why the rule exists, |
| 260 | with a code example. |
| 261 | |
| 262 | If the T3 `link_pull_request` tool is available, link every PR you touch or |
| 263 | open to this thread. |
| 264 | |
| 265 | ## 7. Report and ask |
| 266 | |
| 267 | Keep it short — longer explanations only when asked. End the turn after each |
| 268 | PR with: |
| 269 | |
| 270 | ```md |
| 271 | ### #N: <behavior>. I recommend merging. |
| 272 | |
| 273 | <Why: the bug or gap, with the error text if there is one.> |
| 274 | <The fix, often a short diff.> |
| 275 | <What I changed on top of the author's work.> |
| 276 | |
| 277 | | Test | Code | Result | |
| 278 | | --- | --- | --- | |
| 279 | | test/AWS/RDS/DBInstance.test.ts (full, live) | PR | ✅ 6/6 | |
| 280 | | … "refresh after restore" | main (expected to fail) | ❌ fails as expected: <error> | |
| 281 | |
| 282 | Not run: <suites and why>. |
| 283 | <Design-review answers, decisions for you, follow-ups.> |
| 284 | Next in the list: #M (omit when it was the only one). |
| 285 | |
| 286 | https://github.com/alchemy-run/alchemy/pull/N |
| 287 | ``` |
| 288 | |
| 289 | - Say which tests ran and whether they passed. Every CI check must have |
| 290 | passed on the head you report before you call it "ready", but CI status |
| 291 | doesn't replace the test report. |
| 292 | - Label deliberate runs against main as expected failures. |
| 293 | - Say plainly what was **not** run. |
| 294 | - When fixes span several PRs, say which PR holds each fix and give the merge |
| 295 | order as a numbered list with one URL per line, closes included. |
| 296 | - When a decision is the user's, give numbered options with a recommendation. |
| 297 | - **The last line of every message about a PR is its full URL** — progress |
| 298 | notes and questions included. Several PRs → all URLs at the end. |
| 299 | |
| 300 | ## 8. Keep the list moving |
| 301 | |
| 302 | - Don't wait idle on a merge: start the next PR and keep the pending link in |
| 303 | each message. |
| 304 | - On "merged, next": check it really merged (`gh pr view <n> --json state` — |
| 305 | the user sometimes names the wrong number), re-merge the new main into the |
| 306 | next PR, then continue. |
| 307 | - On "merge", "close", "park", "skip": update the table and act. Close only on |
| 308 | explicit instruction, with a standalone comment crediting the author and |
| 309 | naming what supersedes it. |
| 310 | - Answer "why" or "justify" questions directly with the error text and a diff |
| 311 | snippet. "Refresh my memory" gets the status table plus open items. |