The best way to control Claude Code, Codex and more from any computer or phone
docs(testing): new tests extend existing cases and stay at one layer (#26166)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Theo Browne committed
Oct 27, 2026 at 17:41 UTC
68495b641b3e5834421b6549751eb65ed66fc6f4
13 files changed
+692
-38
.coderabbit.config.ts
+1
-1
@@ -27,7 +27,7 @@ export default defineConfig((ctx) => ({
27
instructions: "Hold changed code to the rules in docs/internals/web-ui.md.",
28
},
29
{
30
- path: "**/*.test.{ts,tsx}",
30
+ path: "**/*.test.*",
31
instructions:
32
"Hold changed tests to the rules in docs/internals/testing.md. Request tests only as those rules allow.",
33
},
.macroscope/check-run-agents/test-signal.md
new
+27
@@ -0,0 +1,27 @@
1
+---
2
+title: Test Signal
3
+model: gpt-6-1-terra
4
+reasoning: low
5
+input: incremental
6
+tools:
7
+ - browse_code
8
+ - modify_pr
9
+# An explicit include overrides ignore.md, which skips test files for every other check.
10
+include:
11
+ - "**/*.test.*"
12
+exclude:
13
+ - ".repos/**"
14
+requires:
15
+ - Check
16
+maxBudgetPerRun: 1
17
+maxBudgetPerPR: 5
18
+conclusion: neutral
19
+---
20
+
21
+# Test signal review
22
+
23
+Review only the tests this pull request adds or grows. Tests it moves, renames, or deletes are not findings. The rules are in `docs/internals/testing.md` under `## What to test`; read that section first and hold new tests to it.
24
+
25
+Post each finding as a brief inline comment on the new test with one fix. Ask only for fewer or smaller tests, never for more coverage.
26
+
27
+When there are no findings, make the entire final response exactly `All clear` on one line with nothing else.
AGENTS.md
+4
-2
@@ -102,11 +102,13 @@ An empty database is a bad test. Seed your worktree's `.t3` with a copy of real
102
- While you iterate, run `vp test run <files>`. For big files, add `-t "<name>"`.
103
- Use `vp run typecheck`, not `npx tsc`; it caches per package. Both run tsc-rs (`typescript` is the `@tsc-rs/typescript` drop-in).
104
- Run `node -e` snippets from `apps/server` or the owning package. `effect` does not resolve from the repo root.
105
-- Test meaningful logic or observable behavior. Do not render components to static markup to assert props or attributes, or add tests that merely assert callback wiring or mirror the implementation.
105
+- Test behavior that can break, such as orchestration and event-sourcing, provider protocol translation, auth and permission guards, persistence and migrations, external-format parsing, race fixes, and the logic a bug fix changed. Leave out tests for what types already guarantee, trivial logic, copy text, smoke runs, mock-call mirrors, callback wiring, and props rendered to static markup.
106
+- Extend before you add. When an existing test or table covers the behavior, add a row or an assertion there. Write a new `it` only when no case fits.
107
+- Test at one layer: the lowest one that fails on `main`. Do not repeat the case in a wrapper test, at each call site, or in web and mobile copies of the same logic.
108
- [Testing](docs/internals/testing.md) carries these same test rules, plus review guidance, for review bots. Change both together.
109
- **Do not run repo-wide checks.** No `vp check`, no `vp run -r test`, no `vp run -r typecheck` unless I ask. CI owns the full suite.
110
- Backend behavior changes ship with focused tests for that behavior.
109
-- Proof for a bug fix depends on the bug. Logic, data, state, and protocol bugs get a failing test first (the fewest that fail on `main` for that reason, in the module's existing test file), then the fix. Purely visual bugs (spacing, colour, alignment, a layout shift) get before/after captures as their proof and no test: do not add UI tests for nitpicks, and do not ask for one in review when a capture already shows it. A server-only fix with no visible surface ships the test plus a short runtime transcript.
111
+- Proof for a bug fix depends on the bug. Logic, data, state, and protocol bugs get a failing test first (the fewest that fail on `main` for that reason, in an existing case of the module's test file where one fits), then the fix. Purely visual bugs (spacing, colour, alignment, a layout shift) get before/after captures as their proof and no test: do not add UI tests for nitpicks, and do not ask for one in review when a capture already shows it. A server-only fix with no visible surface ships the test plus a short runtime transcript.
112
- The server is event-sourced, and side effects run after the command commits. In tests, drain the effect worker (`OrchestrationEffectWorkerV2.drain`) or await the specific persisted event or `Deferred` that marks the milestone. Never wait on sleeps or polling. A test that needs a timeout to pass is wrong. Concretely:
113
- `it.effect` runs on a `TestClock`; `Effect.sleep` only advances through `TestClock.adjust`. `it.live` is wall-clock and only for subprocess or network fixtures.
114
- No `yieldNow` or `sleep` polling loops. Await the `Deferred`, take from the `Queue`, or drain the worker.
docs/internals/testing.md
+5
-3
@@ -4,10 +4,12 @@ Rules for writing tests, and for asking for them in review. AGENTS.md carries th
4
5
## What to test
6
7
-- Test meaningful logic or observable behavior. Do not render components to static markup to assert props or attributes, or add tests that merely assert callback wiring or mirror the implementation.
8
-- In review, ask for a missing test only when it would catch a real regression. Do not ask for one assertion per constant, label, or `switch` case.
7
+- Test behavior that can break, such as orchestration and event-sourcing, provider protocol translation, auth and permission guards, persistence and migrations, external-format parsing, race fixes, and the logic a bug fix changed. Leave out tests for what types already guarantee, trivial logic, copy text, smoke runs, mock-call mirrors, callback wiring, and props rendered to static markup.
8
+- Extend before you add. When an existing test or table covers the behavior, add a row or an assertion there. Write a new `it` only when no case fits.
9
+- Test at one layer: the lowest one that fails on `main`. Do not repeat the case in a wrapper test, at each call site, or in web and mobile copies of the same logic.
10
+- In review, ask for a missing test only when it would catch a real regression. Do not ask for one assertion per constant, label, or `switch` case. Flag a new test that breaks the rules above, and name the fix: fold it into the existing case or table, delete it, or keep one layer.
11
- Backend behavior changes ship with focused tests for that behavior.
10
-- Proof for a bug fix depends on the bug. Logic, data, state, and protocol bugs get a failing test first (the fewest that fail on `main` for that reason, in the module's existing test file), then the fix. Purely visual bugs (spacing, colour, alignment, a layout shift) get before/after captures as their proof and no test: do not add UI tests for nitpicks, and do not ask for one in review when a capture already shows it. A server-only fix with no visible surface ships the test plus a short runtime transcript.
12
+- Proof for a bug fix depends on the bug. Logic, data, state, and protocol bugs get a failing test first (the fewest that fail on `main` for that reason, in an existing case of the module's test file where one fits), then the fix. Purely visual bugs (spacing, colour, alignment, a layout shift) get before/after captures as their proof and no test: do not add UI tests for nitpicks, and do not ask for one in review when a capture already shows it. A server-only fix with no visible surface ships the test plus a short runtime transcript.
13
14
## Waiting for work
15
evals/README.md
+5
-4
@@ -4,10 +4,11 @@ An eval runs a real model against T3 Code's real text or code and scores what th
4
5
Write or run an eval when a change affects what a model sees or does: agent instructions, MCP tool descriptions and schemas, tool results, generated titles, or the guidance an agent gets for a feature such as Device Manager.
6
7
-| Eval | Measures |
8
-| ------------------------------- | ------------------------------------------------------------ |
9
-| [mcp-tool-use](mcp-tool-use/) | Whether agents use T3's MCP tools correctly in fake threads. |
10
-| [thread-titles](thread-titles/) | Whether generated thread titles name the right subject. |
7
+| Eval | Measures |
8
+| ------------------------------- | --------------------------------------------------------------------------- |
9
+| [mcp-tool-use](mcp-tool-use/) | Whether agents use T3's MCP tools correctly in fake threads. |
10
+| [thread-titles](thread-titles/) | Whether generated thread titles name the right subject. |
11
+| [test-signal](test-signal/) | Whether a cheap model running the test-signal check flags low-signal tests. |
12
13
## Run an eval
14
evals/mcp-tool-use/run.ts
+1
-28
@@ -50,6 +50,7 @@ import {
50
CLAUDE_PROVIDER,
51
makeClaudeQueryOptions,
52
} from "../../apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts";
53
+import { proxyConfig } from "../shared/proxy.ts";
54
import { changeRequestsCreatedBy } from "../../apps/server/src/orchestration-v2/PullRequestSyncReactor.ts";
55
/** Agent models by id, with the display name a harness gives them. */
56
const MODELS = {
@@ -1342,34 +1343,6 @@ const callCodex = Effect.fn("evaluateMcpToolUse.callCodex")(function* (
1343
};
1344
});
1345
1345
-// Read credentials into memory only. Python's TOML parser avoids another runner dependency.
1346
-const decodeProxyConfig = Schema.decodeUnknownSync(
1347
- Schema.Struct({ url: Schema.String, headers: Schema.Record(Schema.String, Schema.String) }),
1348
-);
1349
-const proxyConfig = () =>
1350
- decodeProxyConfig(
1351
- JSON.parse(
1352
- NodeChildProcess.execFileSync(
1353
- "python3",
1354
- [
1355
- "-c",
1356
- `
1357
-import json, os, pathlib, tomllib
1358
-home = pathlib.Path.home() / '.codex'
1359
-p = tomllib.loads((home / 'config.toml').read_text())['model_providers']['vibeproxy']
1360
-h = p.get('http_headers', {}).copy()
1361
-h.update({k: os.environ[v] for k,v in p.get('env_http_headers', {}).items()})
1362
-key = os.environ[p['env_key']] if p.get('env_key') else p.get('experimental_bearer_token')
1363
-if not key and p.get('requires_openai_auth'): key = json.loads((home / 'auth.json').read_text()).get('OPENAI_API_KEY')
1364
-if key: h['Authorization'] = 'Bearer ' + key
1365
-print(json.dumps({'url': p['base_url'].rstrip('/'), 'headers': h}))
1366
-`,
1367
- ],
1368
- { encoding: "utf8" },
1369
- ),
1370
- ),
1371
- );
1372
-
1346
const postModel = Effect.fn("evaluateMcpToolUse.postModel")(function* (
1347
apiKey: string,
1348
body: unknown,
evals/package.json
+1
@@ -6,6 +6,7 @@
6
"mcp-tool-use": "node --env-file-if-exists=../.env mcp-tool-use/run.ts",
7
"mcp-tool-use:trim-capture": "node mcp-tool-use/trim-capture.ts",
8
"thread-titles": "node --env-file-if-exists=../.env thread-titles/run.ts",
9
+ "test-signal": "node --env-file-if-exists=../.env test-signal/run.ts",
10
"typecheck": "tsc --noEmit",
11
"test": "vp test run --config ../vite.config.ts --dir ."
12
},
evals/shared/proxy.ts
new
+33
@@ -0,0 +1,33 @@
1
+// Eval-only helper: it reads the local Codex config through python3 at startup.
2
+// @effect-diagnostics nodeBuiltinImport:off
3
+import * as NodeChildProcess from "node:child_process";
4
+import * as Schema from "effect/Schema";
5
+
6
+// Read credentials into memory only. Python's TOML parser avoids another runner dependency.
7
+const decodeProxyConfig = Schema.decodeUnknownSync(
8
+ Schema.Struct({ url: Schema.String, headers: Schema.Record(Schema.String, Schema.String) }),
9
+);
10
+/** Reads the `vibeproxy` CLIProxyAPI provider from `~/.codex/config.toml`. */
11
+export const proxyConfig = () =>
12
+ decodeProxyConfig(
13
+ JSON.parse(
14
+ NodeChildProcess.execFileSync(
15
+ "python3",
16
+ [
17
+ "-c",
18
+ `
19
+import json, os, pathlib, tomllib
20
+home = pathlib.Path.home() / '.codex'
21
+p = tomllib.loads((home / 'config.toml').read_text())['model_providers']['vibeproxy']
22
+h = p.get('http_headers', {}).copy()
23
+h.update({k: os.environ[v] for k,v in p.get('env_http_headers', {}).items()})
24
+key = os.environ[p['env_key']] if p.get('env_key') else p.get('experimental_bearer_token')
25
+if not key and p.get('requires_openai_auth'): key = json.loads((home / 'auth.json').read_text()).get('OPENAI_API_KEY')
26
+if key: h['Authorization'] = 'Bearer ' + key
27
+print(json.dumps({'url': p['base_url'].rstrip('/'), 'headers': h}))
28
+`,
29
+ ],
30
+ { encoding: "utf8" },
31
+ ),
32
+ ),
33
+ );
evals/test-signal/README.md
new
+23
@@ -0,0 +1,23 @@
1
+# Test signal
2
+
3
+Runs the Macroscope `test-signal` check ([`.macroscope/check-run-agents/test-signal.md`](../../.macroscope/check-run-agents/test-signal.md)) on real and synthetic test diffs, and checks that a cheap model flags low-signal tests and leaves good ones alone. Use it to cut the check to the fewest words that still work.
4
+
5
+The model gets the check body, the PR's test diff, `read_file` and `grep` at the PR head, and `post_comment`. The wrapper prompt around the body is a stand-in for Macroscope's, which is not public.
6
+
7
+## Run
8
+
9
+```sh
10
+cd evals
11
+pnpm test-signal --out runs/baseline
12
+pnpm test-signal --out runs/cut --prompt /path/to/variant.md --case new-table-row
13
+```
14
+
15
+- Models go through the `vibeproxy` CLIProxyAPI provider in `~/.codex/config.toml`. Default models: `gpt-6.1-terra` and `gpt-6-luna`. Pass `--model` once per model to change them.
16
+- Flags: `--prompt <file>` (a variant of the check, default the real one), `--case <id>` (repeatable), `--samples <n>` (default 2, and a 3rd when the first 2 disagree), `--concurrency <n>` (default 6), `--reasoning <level>` (default `low`, as the check runs).
17
+- A bad case passes when a comment lands inside a target test. A guard case passes when the check posts nothing and replies exactly `All clear`.
18
+- `summary.md` gives the prompt hash and word count, then per model: bad cases caught, guards clean, errors, and tokens.
19
+- Cost: proxy quota only, about 0.5M tokens for a full run on both models.
20
+
21
+## Add a case
22
+
23
+Add an entry to `cases.ts`. A `real` case names a merged commit and its test paths. A `synthetic` case inserts one test after an anchor string in a real file. Bad cases list the titles of the tests to flag.
evals/test-signal/cases.ts
new
+177
@@ -0,0 +1,177 @@
1
+/**
2
+ * Pull request test diffs for the test-signal check. A `real` case replays test files from a
3
+ * merged PR. A `synthetic` case inserts one test into a real file at `rev`, for a pattern with no
4
+ * clean real example.
5
+ */
6
+export type Case = {
7
+ readonly id: string;
8
+ readonly kind: "bad" | "guard";
9
+ readonly source: string;
10
+ readonly rev: string;
11
+ /** Titles of the tests the check must flag. Guard cases expect no comment and `All clear`. */
12
+ readonly targets: ReadonlyArray<{ readonly path: string; readonly title: string }>;
13
+} & (
14
+ | { readonly type: "real"; readonly paths: ReadonlyArray<string> }
15
+ | {
16
+ readonly type: "synthetic";
17
+ readonly path: string;
18
+ readonly after: string;
19
+ readonly insert: string;
20
+ }
21
+);
22
+
23
+const base = "cb7e3814d71c8fcdc5b803c660c29b7d7e39be96";
24
+
25
+export const cases: ReadonlyArray<Case> = [
26
+ {
27
+ id: "copy-text-at-every-call-site",
28
+ kind: "bad",
29
+ source: "#23999 checks the PR or MR label in 11 files",
30
+ type: "real",
31
+ rev: "218e5c09df",
32
+ paths: [
33
+ "packages/client-runtime/src/state/gitActions.test.ts",
34
+ "apps/web/src/lib/composerContextRecords.test.ts",
35
+ "apps/mobile/src/lib/composerContext.test.ts",
36
+ ],
37
+ targets: [
38
+ {
39
+ path: "packages/client-runtime/src/state/gitActions.test.ts",
40
+ title: "says PR for a GitHub ref",
41
+ },
42
+ {
43
+ path: "packages/client-runtime/src/state/gitActions.test.ts",
44
+ title: "says MR for a GitLab ref",
45
+ },
46
+ {
47
+ path: "packages/client-runtime/src/state/gitActions.test.ts",
48
+ title: "names the host's change request in the default-branch dialog",
49
+ },
50
+ {
51
+ path: "apps/web/src/lib/composerContextRecords.test.ts",
52
+ title: "names the change request the way its host does",
53
+ },
54
+ {
55
+ path: "apps/mobile/src/lib/composerContext.test.ts",
56
+ title: "describes the change request with its host's term",
57
+ },
58
+ ],
59
+ },
60
+ {
61
+ id: "same-case-in-wrapper-test",
62
+ kind: "bad",
63
+ source: "#22452 added the same two cases to threadAccess and OrchestratorMcpService",
64
+ type: "real",
65
+ rev: "b260a7cf69",
66
+ paths: [
67
+ "apps/server/src/mcp/threadAccess.test.ts",
68
+ "apps/server/src/mcp/OrchestratorMcpService.test.ts",
69
+ ],
70
+ // Either copy counts: the check must notice the duplicate. Which copy it keeps is secondary.
71
+ targets: [
72
+ {
73
+ path: "apps/server/src/mcp/threadAccess.test.ts",
74
+ title: "points a bare uuid at the mcp: thread it belongs to",
75
+ },
76
+ {
77
+ path: "apps/server/src/mcp/threadAccess.test.ts",
78
+ title: "keeps the plain message for an id that matches nothing",
79
+ },
80
+ {
81
+ path: "apps/server/src/mcp/OrchestratorMcpService.test.ts",
82
+ title: "points a bare uuid at the mcp: thread it belongs to",
83
+ },
84
+ {
85
+ path: "apps/server/src/mcp/OrchestratorMcpService.test.ts",
86
+ title: "keeps the plain message for an id that matches nothing",
87
+ },
88
+ ],
89
+ },
90
+ {
91
+ id: "new-it-for-a-table-row",
92
+ kind: "bad",
93
+ source: "Synthetic: a new `it` next to the table that already covers it",
94
+ type: "synthetic",
95
+ rev: base,
96
+ path: "packages/shared/src/markdownLinks.test.ts",
97
+ after: " expect(parseMarkdownFileLink(href)?.path).toBe(path);\n });\n",
98
+ insert:
99
+ '\n it("recognizes a Windows path with forward slashes as a file", () => {\n expect(parseMarkdownFileLink("C:/repo/src/app.ts")?.path).toBe("C:/repo/src/app.ts");\n });\n',
100
+ targets: [
101
+ {
102
+ path: "packages/shared/src/markdownLinks.test.ts",
103
+ title: "recognizes a Windows path with forward slashes as a file",
104
+ },
105
+ ],
106
+ },
107
+ {
108
+ id: "mock-call-mirror",
109
+ kind: "bad",
110
+ source: "Synthetic: asserts the mock call the code makes, already covered above it",
111
+ type: "synthetic",
112
+ rev: base,
113
+ path: "packages/client-runtime/src/state/threadRequests.test.ts",
114
+ after: " expect(send).toHaveBeenCalledOnce();\n });\n",
115
+ insert:
116
+ '\n it("calls respond with decline", async () => {\n const respond = vi.fn<ApprovalResponder>(async () => AsyncResult.success(undefined));\n await declinePendingApprovals([approval], respond);\n expect(respond).toHaveBeenCalledWith(requestId, "decline");\n expect(respond).toHaveBeenCalledTimes(1);\n });\n',
117
+ targets: [
118
+ {
119
+ path: "packages/client-runtime/src/state/threadRequests.test.ts",
120
+ title: "calls respond with decline",
121
+ },
122
+ ],
123
+ },
124
+ {
125
+ id: "type-guaranteed",
126
+ kind: "bad",
127
+ source: "Synthetic: checks a return type the signature already declares",
128
+ type: "synthetic",
129
+ rev: base,
130
+ path: "packages/shared/src/markdownLinks.test.ts",
131
+ after: " expect(isMarkdownFileLinkLabel(label, href)).toBe(expected);\n });\n",
132
+ insert:
133
+ '\n it("returns a boolean", () => {\n expect(typeof isMarkdownFileLinkLabel("a.ts", "/repo/a.ts")).toBe("boolean");\n });\n',
134
+ targets: [{ path: "packages/shared/src/markdownLinks.test.ts", title: "returns a boolean" }],
135
+ },
136
+ {
137
+ id: "deadlock-race-fix",
138
+ kind: "guard",
139
+ source: "#22094 run metrics read lineage inside the transaction so a writer cannot deadlock",
140
+ type: "real",
141
+ rev: "f8e4834861",
142
+ paths: ["apps/server/src/orchestration-v2/RunMetrics.test.ts"],
143
+ targets: [],
144
+ },
145
+ {
146
+ id: "clone-retry-fix",
147
+ kind: "guard",
148
+ source: "#23170 clones retry over HTTPS when the SSH key is refused",
149
+ type: "real",
150
+ rev: "a49d18cf3b",
151
+ paths: ["apps/server/src/sourceControl/SourceControlRepositoryService.test.ts"],
152
+ targets: [],
153
+ },
154
+ {
155
+ id: "deleted-tests",
156
+ kind: "guard",
157
+ source: "#22592 pruned low-value tests",
158
+ type: "real",
159
+ rev: "a2f4df0412",
160
+ paths: [
161
+ "apps/server/src/workspace/WorkspaceEntries.test.ts",
162
+ "apps/server/src/workspace/WorkspacePaths.test.ts",
163
+ ],
164
+ targets: [],
165
+ },
166
+ {
167
+ id: "new-table-row",
168
+ kind: "guard",
169
+ source: "Synthetic: the right way to add the Windows path case",
170
+ type: "synthetic",
171
+ rev: base,
172
+ path: "packages/shared/src/markdownLinks.test.ts",
173
+ after: ' ["file://localhost/home/me/notes.md", "/home/me/notes.md"],\n',
174
+ insert: ' ["C:/repo/src/app.ts", "C:/repo/src/app.ts"],\n',
175
+ targets: [],
176
+ },
177
+];
evals/test-signal/grade.ts
new
+51
@@ -0,0 +1,51 @@
1
+export type Comment = { readonly path: string; readonly line: number; readonly body: string };
2
+
3
+const testStart = /^\s*(it|test|describe)(\.\w+)*\(/;
4
+
5
+/** 1-based line range of the test whose title is `title`, up to the next test or describe. */
6
+export const testRange = (source: string, title: string) => {
7
+ const lines = source.split("\n");
8
+ const start = lines.findIndex(
9
+ (line) => line.includes(`"${title}"`) || line.includes(`'${title}'`),
10
+ );
11
+ if (start < 0) return undefined;
12
+ const next = lines.findIndex((line, index) => index > start && testStart.test(line));
13
+ return { start: start + 1, end: next < 0 ? lines.length : next };
14
+};
15
+
16
+// Models often anchor a comment on the blank line or brace just above the test.
17
+const slack = 2;
18
+
19
+/**
20
+ * A bad case passes when a comment lands inside a target test. A guard case passes when the
21
+ * check posts nothing and ends with exactly `All clear`, as Macroscope requires.
22
+ */
23
+export const grade = (input: {
24
+ readonly kind: "bad" | "guard";
25
+ readonly targets: ReadonlyArray<{
26
+ readonly path: string;
27
+ readonly start: number;
28
+ readonly end: number;
29
+ }>;
30
+ readonly comments: ReadonlyArray<Comment>;
31
+ readonly final: string;
32
+}) => {
33
+ if (input.kind === "guard") {
34
+ if (input.comments.length > 0)
35
+ return { pass: false, reason: `${input.comments.length} comment(s)` };
36
+ if (input.final.trim() !== "All clear")
37
+ return { pass: false, reason: "final reply is not `All clear`" };
38
+ return { pass: true, reason: "all clear" };
39
+ }
40
+ const hit = input.comments.some((comment) =>
41
+ input.targets.some(
42
+ (target) =>
43
+ comment.path === target.path &&
44
+ comment.line >= target.start - slack &&
45
+ comment.line <= target.end,
46
+ ),
47
+ );
48
+ return hit
49
+ ? { pass: true, reason: "flagged a target" }
50
+ : { pass: false, reason: `missed (${input.comments.length} comment(s) elsewhere)` };
51
+};
evals/test-signal/run.test.ts
new
+27
@@ -0,0 +1,27 @@
1
+import { describe, expect, it } from "vite-plus/test";
2
+import { grade, testRange } from "./grade.ts";
3
+
4
+const source = [
5
+ 'describe("x", () => {',
6
+ ' it("first", () => {',
7
+ " expect(1).toBe(1);",
8
+ " });",
9
+ "",
10
+ ' it("second", () => {});',
11
+ "});",
12
+].join("\n");
13
+
14
+describe("grade", () => {
15
+ const target = { path: "a.test.ts", ...testRange(source, "first")! };
16
+ const comment = (line: number) => ({ path: "a.test.ts", line, body: "fold it" });
17
+
18
+ it.each([
19
+ ["bad, comment inside the target", "bad", [comment(3)], "", true],
20
+ ["bad, comment on the next test", "bad", [comment(6)], "", false],
21
+ ["guard, nothing posted", "guard", [], "All clear", true],
22
+ ["guard, nothing posted but extra text", "guard", [], "All clear. Nice tests.", false],
23
+ ["guard, a comment", "guard", [comment(3)], "All clear", false],
24
+ ] as const)("%s", (_name, kind, comments, final, pass) => {
25
+ expect(grade({ kind, targets: [target], comments, final }).pass).toBe(pass);
26
+ });
27
+});
evals/test-signal/run.ts
new
+337
@@ -0,0 +1,337 @@
1
+#!/usr/bin/env node
2
+// An eval script, not server code: it shells out to git and reads files directly.
3
+// @effect-diagnostics nodeBuiltinImport:off
4
+import * as NodeChildProcess from "node:child_process";
5
+import * as NodeCrypto from "node:crypto";
6
+import * as NodeFS from "node:fs";
7
+import * as NodeOS from "node:os";
8
+import * as NodePath from "node:path";
9
+import * as NodeUtil from "node:util";
10
+import * as Effect from "effect/Effect";
11
+import * as Schedule from "effect/Schedule";
12
+import * as Schema from "effect/Schema";
13
+import { FetchHttpClient, HttpClient, HttpClientRequest } from "effect/http";
14
+import { proxyConfig } from "../shared/proxy.ts";
15
+import { cases, type Case } from "./cases.ts";
16
+import { grade, testRange, type Comment } from "./grade.ts";
17
+
18
+const { values } = NodeUtil.parseArgs({
19
+ options: {
20
+ out: { type: "string" },
21
+ model: { type: "string", multiple: true },
22
+ prompt: { type: "string" },
23
+ case: { type: "string", multiple: true },
24
+ samples: { type: "string", default: "2" },
25
+ concurrency: { type: "string", default: "6" },
26
+ reasoning: { type: "string", default: "low" },
27
+ },
28
+});
29
+if (!values.out) throw new Error("Use --out <directory> [--model <id>] [--prompt <file>].");
30
+const outDirectory = values.out;
31
+const models = values.model ?? ["gpt-6.1-terra", "gpt-6-luna"];
32
+const repo = NodePath.resolve(import.meta.dirname, "../..");
33
+const checkFile =
34
+ values.prompt ?? NodePath.join(repo, ".macroscope/check-run-agents/test-signal.md");
35
+// Macroscope gives the model the body; the front matter only configures the run.
36
+const instructions = NodeFS.readFileSync(checkFile, "utf8")
37
+ .replace(/^---\n[\s\S]*?\n---\n/, "")
38
+ .trim();
39
+const words = instructions.split(/\s+/).length;
40
+const promptHash = NodeCrypto.createHash("sha256").update(instructions).digest("hex").slice(0, 12);
41
+const selected = values.case ? cases.filter((entry) => values.case!.includes(entry.id)) : cases;
42
+const proxy = proxyConfig();
43
+
44
+const git = (args: ReadonlyArray<string>) => {
45
+ const result = NodeChildProcess.spawnSync("git", args, {
46
+ cwd: repo,
47
+ encoding: "utf8",
48
+ maxBuffer: 1 << 26,
49
+ });
50
+ return result.status === 0 ? result.stdout : undefined;
51
+};
52
+
53
+/** The case's PR head as files the model can read, plus the diff it reviews. */
54
+const world = (entry: Case) => {
55
+ const files = new Map<string, string>();
56
+ if (entry.type === "real") {
57
+ const diff = git(["diff", `${entry.rev}~1`, entry.rev, "--", ...entry.paths]);
58
+ // An empty diff would grade a guard case as a clean pass.
59
+ if (!diff) throw new Error(`${entry.id}: no diff at ${entry.rev}.`);
60
+ return { diff, files };
61
+ }
62
+ const before = git(["show", `${entry.rev}:${entry.path}`]);
63
+ if (!before?.includes(entry.after)) throw new Error(`${entry.id}: anchor not found.`);
64
+ const after = before.replace(entry.after, entry.after + entry.insert);
65
+ files.set(entry.path, after);
66
+ const directory = NodeFS.mkdtempSync(NodePath.join(NodeOS.tmpdir(), "t3-test-signal-"));
67
+ NodeFS.writeFileSync(NodePath.join(directory, "before"), before);
68
+ NodeFS.writeFileSync(NodePath.join(directory, "after"), after);
69
+ const raw = NodeChildProcess.spawnSync("git", ["diff", "--no-index", "before", "after"], {
70
+ cwd: directory,
71
+ encoding: "utf8",
72
+ }).stdout;
73
+ NodeFS.rmSync(directory, { recursive: true });
74
+ const diff = raw
75
+ .replace("diff --git a/before b/after", `diff --git a/${entry.path} b/${entry.path}`)
76
+ .replace("--- a/before", `--- a/${entry.path}`)
77
+ .replace("+++ b/after", `+++ b/${entry.path}`);
78
+ return { diff, files };
79
+};
80
+
81
+// The check reads the rules at the PR head, where they always exist. Old case commits predate
82
+// them, so serve the rules from this checkout.
83
+const ruleFiles = new Set(["docs/internals/testing.md", "AGENTS.md"]);
84
+const readAt = (entry: Case, files: Map<string, string>, path: string) =>
85
+ ruleFiles.has(path)
86
+ ? NodeFS.readFileSync(NodePath.join(repo, path), "utf8")
87
+ : (files.get(path) ?? git(["show", `${entry.rev}:${path}`]));
88
+
89
+/** `git grep` at `rev`, with overlay and rule files searched in their served form instead. */
90
+const grepAt = (entry: Case, files: Map<string, string>, pattern: string, scope: string) => {
91
+ // Status 2 is a bad pattern. Show grep's message, as a real tool would, so the model can fix it.
92
+ const check = NodeChildProcess.spawnSync("grep", ["-E", "--", pattern], {
93
+ input: "",
94
+ encoding: "utf8",
95
+ });
96
+ if (check.status !== 0 && check.status !== 1) return check.stderr.trim() || "grep failed.";
97
+ const served = new Map(files);
98
+ for (const path of ruleFiles) served.set(path, readAt(entry, files, path) ?? "");
99
+ // Served files also exist in this checkout, so git applies its own pathspec rules to the scope.
100
+ const matched = new Set((git(["ls-files", "--", scope || "."]) ?? "").split("\n"));
101
+ const inScope = (path: string) => matched.has(path);
102
+ const fromRev = (git(["grep", "-n", "-E", "-e", pattern, entry.rev, "--", scope || "."]) ?? "")
103
+ .split("\n")
104
+ .filter((line) => line !== "")
105
+ .map((line) => line.replace(`${entry.rev}:`, ""))
106
+ .filter((line) => !served.has(line.slice(0, line.indexOf(":"))));
107
+ const fromServed = [...served]
108
+ .filter(([path]) => inScope(path))
109
+ .flatMap(([path, text]) =>
110
+ NodeChildProcess.spawnSync("grep", ["-n", "-E", "--", pattern], {
111
+ input: text,
112
+ encoding: "utf8",
113
+ })
114
+ .stdout.split("\n")
115
+ .filter((line) => line !== "")
116
+ .map((line) => `${path}:${line}`),
117
+ );
118
+ return [...fromServed, ...fromRev].slice(0, 80).join("\n") || "No matches.";
119
+};
120
+
121
+const tools = [
122
+ {
123
+ type: "function",
124
+ name: "read_file",
125
+ description: "Read a file at the pull request head, with line numbers.",
126
+ parameters: {
127
+ type: "object",
128
+ properties: {
129
+ path: { type: "string" },
130
+ start_line: { type: "integer" },
131
+ end_line: { type: "integer" },
132
+ },
133
+ required: ["path"],
134
+ },
135
+ },
136
+ {
137
+ type: "function",
138
+ name: "grep",
139
+ description:
140
+ "Search the repository at the pull request head with an extended regular expression.",
141
+ parameters: {
142
+ type: "object",
143
+ properties: { pattern: { type: "string" }, path: { type: "string" } },
144
+ required: ["pattern"],
145
+ },
146
+ },
147
+ {
148
+ type: "function",
149
+ name: "post_comment",
150
+ description: "Post an inline review comment on a line of the pull request head.",
151
+ parameters: {
152
+ type: "object",
153
+ properties: { path: { type: "string" }, line: { type: "integer" }, body: { type: "string" } },
154
+ required: ["path", "line", "body"],
155
+ },
156
+ },
157
+];
158
+
159
+const system = (body: string) =>
160
+ [
161
+ "You are a Macroscope Check Run Agent reviewing a GitHub pull request.",
162
+ "Follow the check instructions below. Read the repository at the pull request head with read_file and grep.",
163
+ "Leave inline review comments with post_comment. Then give your final response.",
164
+ "",
165
+ "<check_instructions>",
166
+ body,
167
+ "</check_instructions>",
168
+ ].join("\n");
169
+
170
+const decodeResponse = Schema.decodeUnknownSync(
171
+ Schema.fromJsonString(
172
+ Schema.Struct({
173
+ output: Schema.Array(Schema.Record(Schema.String, Schema.Unknown)),
174
+ usage: Schema.optional(Schema.Struct({ total_tokens: Schema.optional(Schema.Number) })),
175
+ }),
176
+ ),
177
+);
178
+const decodeArguments = Schema.decodeUnknownSync(
179
+ Schema.fromJsonString(Schema.Record(Schema.String, Schema.Unknown)),
180
+);
181
+const encodeRow = Schema.encodeSync(Schema.fromJsonString(Schema.Unknown));
182
+
183
+const post = Effect.fn("evaluateTestSignal.post")(function* (body: unknown) {
184
+ const client = (yield* HttpClient.HttpClient).pipe(
185
+ HttpClient.retryTransient({
186
+ retryOn: "errors-and-responses",
187
+ schedule: Schedule.exponential("2 seconds"),
188
+ times: 4,
189
+ }),
190
+ );
191
+ const response = yield* HttpClientRequest.post(`${proxy.url}/responses`).pipe(
192
+ HttpClientRequest.setHeaders(proxy.headers),
193
+ HttpClientRequest.bodyJson(body),
194
+ Effect.flatMap(client.execute),
195
+ );
196
+ const text = yield* response.text;
197
+ if (response.status !== 200)
198
+ return yield* Effect.die(new Error(`Model API ${response.status}: ${text}`));
199
+ return decodeResponse(text);
200
+});
201
+
202
+const runSample = Effect.fn("evaluateTestSignal.sample")(function* (entry: Case, model: string) {
203
+ const { diff, files } = world(entry);
204
+ const comments: Array<Comment> = [];
205
+ const input: Array<unknown> = [
206
+ {
207
+ role: "user",
208
+ content: `Pull request diff for the files this check includes:\n\n\`\`\`diff\n${diff}\`\`\``,
209
+ },
210
+ ];
211
+ let final = "";
212
+ let tokens = 0;
213
+ for (let step = 0; step < 16; step++) {
214
+ const response = yield* post({
215
+ model,
216
+ instructions: system(instructions),
217
+ input,
218
+ tools,
219
+ reasoning: { effort: values.reasoning },
220
+ store: false,
221
+ include: ["reasoning.encrypted_content"],
222
+ });
223
+ tokens += response.usage?.total_tokens ?? 0;
224
+ input.push(...response.output);
225
+ const calls = response.output.filter((item) => item.type === "function_call");
226
+ final = response.output
227
+ .filter((item) => item.type === "message")
228
+ .flatMap((item) => (item.content as Array<{ text?: string }>).map((part) => part.text ?? ""))
229
+ .join("");
230
+ if (calls.length === 0) break;
231
+ for (const call of calls) {
232
+ const args = decodeArguments(String(call.arguments));
233
+ let output: string;
234
+ if (call.name === "post_comment") {
235
+ comments.push({
236
+ path: String(args.path),
237
+ line: Number(args.line),
238
+ body: String(args.body),
239
+ });
240
+ output = "Posted.";
241
+ } else if (call.name === "read_file") {
242
+ const text = readAt(entry, files, String(args.path));
243
+ const lines = text?.split("\n") ?? [];
244
+ const start = Math.max(1, Number(args.start_line ?? 1));
245
+ const end = Math.min(lines.length, Number(args.end_line ?? start + 399));
246
+ output =
247
+ text === undefined
248
+ ? "File not found."
249
+ : lines
250
+ .slice(start - 1, end)
251
+ .map((line, index) => `${start + index}: ${line}`)
252
+ .join("\n");
253
+ } else {
254
+ output = grepAt(entry, files, String(args.pattern), String(args.path ?? ""));
255
+ }
256
+ input.push({ type: "function_call_output", call_id: call.call_id, output });
257
+ }
258
+ }
259
+ const targets = entry.targets.flatMap((target) => {
260
+ const range = testRange(readAt(entry, files, target.path) ?? "", target.title);
261
+ if (!range) throw new Error(`${entry.id}: target "${target.title}" not found.`);
262
+ return [{ path: target.path, ...range }];
263
+ });
264
+ return { ...grade({ kind: entry.kind, targets, comments, final }), comments, final, tokens };
265
+});
266
+
267
+const sampleSafely = (entry: Case, model: string, sample: number) =>
268
+ runSample(entry, model).pipe(
269
+ Effect.map((result) => ({ case: entry.id, kind: entry.kind, model, sample, ...result })),
270
+ Effect.catchCause((cause) =>
271
+ Effect.succeed({ case: entry.id, kind: entry.kind, model, sample, error: String(cause) }),
272
+ ),
273
+ );
274
+
275
+type Row = Effect.Success<ReturnType<typeof sampleSafely>>;
276
+const isGraded = (row: Row) => "pass" in row;
277
+const passed = (row: Row) => "pass" in row && row.pass;
278
+
279
+const summary = await Effect.runPromise(
280
+ Effect.gen(function* () {
281
+ NodeFS.mkdirSync(outDirectory, { recursive: true });
282
+ const samples = Number(values.samples);
283
+ const pairs = models.flatMap((model) => selected.map((entry) => ({ model, entry })));
284
+ const rows: Array<Row> = (yield* Effect.forEach(
285
+ pairs.flatMap(({ model, entry }) =>
286
+ Array.from({ length: samples }, (_, n) => ({ model, entry, n })),
287
+ ),
288
+ ({ model, entry, n }) => sampleSafely(entry, model, n),
289
+ { concurrency: Number(values.concurrency) },
290
+ )).slice();
291
+ // A third sample settles a split.
292
+ const splits = pairs.filter(({ model, entry }) => {
293
+ const own = rows.filter((row) => row.model === model && row.case === entry.id);
294
+ return own.length === 2 && own.some(passed) && !own.every(passed);
295
+ });
296
+ rows.push(
297
+ ...(yield* Effect.forEach(splits, ({ model, entry }) => sampleSafely(entry, model, samples), {
298
+ concurrency: Number(values.concurrency),
299
+ })),
300
+ );
301
+ NodeFS.writeFileSync(
302
+ NodePath.join(outDirectory, "results.ndjson"),
303
+ rows.map((row) => encodeRow(row)).join("\n") + "\n",
304
+ );
305
+
306
+ const lines = [`prompt ${promptHash}, ${words} words, ${checkFile}`, ""];
307
+ lines.push("| model | bad caught | guards clean | errors | tokens |", "|---|---|---|---|---|");
308
+ for (const model of models) {
309
+ const verdicts = selected.map((entry) => {
310
+ const own = rows.filter((row) => row.model === model && row.case === entry.id);
311
+ // A run error counts as a failed sample, so one pass and one error is not a majority.
312
+ const graded = own.filter(isGraded);
313
+ const passes = own.filter(passed).length;
314
+ return {
315
+ entry,
316
+ pass: passes > own.length / 2,
317
+ errors: own.length - graded.length,
318
+ };
319
+ });
320
+ const count = (kind: Case["kind"]) => {
321
+ const own = verdicts.filter((verdict) => verdict.entry.kind === kind);
322
+ return `${own.filter((verdict) => verdict.pass).length}/${own.length}`;
323
+ };
324
+ const tokens = rows
325
+ .filter((row) => row.model === model && "tokens" in row)
326
+ .reduce((sum, row) => sum + ("tokens" in row ? row.tokens : 0), 0);
327
+ const errors = verdicts.reduce((sum, verdict) => sum + verdict.errors, 0);
328
+ lines.push(`| ${model} | ${count("bad")} | ${count("guard")} | ${errors} | ${tokens} |`);
329
+ const failed = verdicts.filter((verdict) => !verdict.pass).map((verdict) => verdict.entry.id);
330
+ if (failed.length > 0) lines.push(`| failed: ${failed.join(", ")} | | | | |`);
331
+ }
332
+ const report = lines.join("\n") + "\n";
333
+ NodeFS.writeFileSync(NodePath.join(outDirectory, "summary.md"), report);
334
+ return report;
335
+ }).pipe(Effect.provide(FetchHttpClient.layer)),
336
+);
337
+process.stdout.write(summary);