The best way to control Claude Code, Codex and more from any computer or phone
test(evals): measure T3's PR tools against raw forge CLIs (#39922)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Julius Marminge committed
Nov 20, 2026 at 16:42 UTC
634ee43deb6d5fc0feae77153d33df1e97ffb303
7 files changed
+749
-30
evals/mcp-tool-use/README.md
+58
-1
@@ -28,7 +28,7 @@ Cases are JSON lines in `cases/`, one file per area. The case schema in `run.ts`
28
- `experience`: the real mistake, in a sentence. `source`: where it happened.
29
- `setup`: the fake thread, its `world` (repo, branch, files, PRs, pages), `priorTurns`, and `mocks` for tool results. With `world.backgroundTasks`, `t3_background_wait` answers from T3's own handler and ThreadManagementService, so its result and errors are the checkout's; a prior wait with `"result": "default"` gets that result too.
30
- `setup.priorTurns[].toolCalls[].resultFile`: load a repository-relative file as the tool result instead of copying its contents into the case. An empty `prompt` resumes a top-level thread after its last prior tool result without adding a user message. Side-chat and subagent setups require a nonempty prompt.
31
-- `expect`: the checks, such as `calls`, `callsMustNot`, `maxCalls`, `order`, `shellMust`, `textMust`, `finalTextMust` (the last text, once the turn ended), and `linked`. A call's `args` match partially: `~text` is a case-insensitive substring, `"<=N"` / `">=N"` bound a number, `null` matches a missing key, and keys may be dotted paths. For `calls` and `callsMustNot`, `argsAnyOf` accepts any one of its partial argument objects, in addition to `args`. Only `calls` supports `success: true`, which requires a recorded successful result. `distinctCalls` supports only `tool` and `args`. Device cases also use `device` to check every child brief, forbid raw UI commands, bound failed device actions per user turn, and require artifact paths in completed final answers. Secrets cases use `secret` (see [Secrets cases](#secrets-cases)).
31
+- `expect`: the checks, such as `calls`, `callsMustNot`, `maxCalls`, `order`, `shellMust`, `textMust`, `finalTextMust` (the last text, once the turn ended), and `linked`. A call's `args` match partially: `~text` is a case-insensitive substring, `re:` a case-sensitive regex, `"<=N"` / `">=N"` bound a number, `null` matches a missing key, and keys may be dotted paths. For `calls` and `callsMustNot`, `argsAnyOf` accepts any one of its partial argument objects, in addition to `args`. Only `calls` supports `success: true`, which requires a recorded successful result. `distinctCalls` supports only `tool` and `args`. Device cases also use `device` to check every child brief, forbid raw UI commands, bound failed device actions per user turn, and require artifact paths in completed final answers. Secrets cases use `secret` (see [Secrets cases](#secrets-cases)).
32
33
Check that the case builds with `--dry-run --case <id>`. Then run it on the text before your fix, where it must fail, and on the text after, where it must pass.
34
@@ -290,6 +290,63 @@ Passes per case, main's runtime text (3 samples) → this branch's "Link every o
290
291
After the change, the first reference to a file is an absolute link. The remaining failures come from two other kinds of span. Some repeat the file later as relative inline code (Sonnet's other-checkout failures). Others quote values rather than cite files, such as `main` set to `src/index.ts` or the import `./ws`. Chat still turns those spans into pills.
292
293
+## PR tools vs raw CLIs
294
+
295
+`cases/pr-tools-vs-cli.ndjson` asks whether T3's PR, CI and issue tools pay for the context they add. Each case is a forge task from a real failure in 2026-09/11 Claude and Codex sessions. `--arm raw` removes those tools and the `<pull_request_linking>` instructions, so the agent has only `gh`, `cafe` and the shell. `--arm tools` (the default) is T3 as shipped. Run both arms on the same model and compare:
296
+
297
+```sh
298
+pnpm mcp-tool-use --arm tools --cases mcp-tool-use/cases/pr-tools-vs-cli.ndjson --samples 3 --out runs/tools
299
+pnpm mcp-tool-use --arm raw --cases mcp-tool-use/cases/pr-tools-vs-cli.ndjson --samples 3 --out runs/raw
300
+pnpm mcp-tool-use:compare runs/tools/results.jsonl runs/raw/results.jsonl
301
+```
302
+
303
+`compare` regrades every row with this checkout's cases and prints, per case and arm: pass rate, model calls, forge calls (T3 forge tools plus `gh`/`cafe`/`glab`/`src` commands), failed forge calls, forge output in KB, requested sleep, `--help` calls, tokens and cost. Cost charges cache writes as cache reads, so the order the runs went in doesn't decide which arm looks cheaper.
304
+
305
+Both arms see the same world. Shell mocks apply to either arm's commands, so an agent in the tools arm that reaches for `gh` hits the same rate limit. Where the real tools refuse, the tools arm is refused too: `world.unpushed` makes `pull_request_create` fail until a `git push`, and `pull_request_merge` compares `expectedHeadSha` with the full head, as `PullRequestService` does. A `{"$error": "..."}` mock fails any T3 tool with that text. `expectRaw` grades the raw arm when its right outcome differs: only T3 can watch a PR, so the raw arm passes `cli-babysit-green` by waiting on the checks.
306
+
307
+Some cases are capability cases rather than mistake cases:
308
+
309
+- `cli-rate-limited-read` and `cli-auth-missing` assume T3 is signed in through the T3 Code for GitHub app, whose quota and token are separate from `gh`'s.
310
+- The git.cafe cases come in two forms. The plain ones say nothing about the forge. The `-instructed` ones put "use cafe" in the project instructions, as a maintained AGENTS.md would.
311
+
312
+### Results
313
+
314
+3 samples per case and arm, 2026-11-20, on T3 as of `6d6ca60` ("tools"). Codex models ran through CLIProxyAPI with `gpt-6.1-terra` as the shell. "Both can finish" leaves out the capability cases: rate limit, gh signed out, waiting on checks, and the uninstructed git.cafe cases. Means per sample; Sonnet cost is at warm-cache prices.
315
+
316
+| Model | Arm | Pass | Model calls | Forge calls | Failed forge calls | Ktokens | Both can finish: pass, ktokens |
317
+| ----------- | ----- | ----- | ----------- | ----------- | ------------------ | ------- | ------------------------------ |
318
+| Sonnet 5.5 | raw | 34/42 | 3.5 | 2.0 | 0.4 | 81 | 27/27, 79 |
319
+| Sonnet 5.5 | tools | 42/42 | 3.8 | 1.6 | 0.0 | 90 | 27/27, 89 |
320
+| GPT-6 Luna | raw | 31/40 | 7.3 | 4.1 | 1.0 | 147 | 24/26, 109 |
321
+| GPT-6 Luna | tools | 41/42 | 4.6 | 1.7 | 0.0 | 91 | 26/27, 97 |
322
+| GPT-6.1 Sol | raw | 29/41 | 6.2 | 4.4 | 1.0 | 122 | 26/27, 92 |
323
+| GPT-6.1 Sol | tools | 42/42 | 4.5 | 2.4 | 0.0 | 89 | 27/27, 99 |
324
+
325
+Errored samples are left out.
326
+
327
+**Where the tools win:**
328
+
329
+- **They never failed a forge call.** The raw arm failed one call per sample on Codex and 0.4 on Sonnet.
330
+- **Capability cases.**
331
+ - The raw arm cannot read checks while `gh` is rate-limited: 0–1 of 3 passed per model.
332
+ - It never found `cafe` on a git.cafe repo whose instructions don't name it: 0/3 on review comments for all three models.
333
+ - GPT-6.1 Sol's raw arm said "I'll check every minute" with nothing running, so it never reported green checks: 0/2.
334
+ - GPT-6 Luna's raw arm polled for 18 model calls on average.
335
+- **On Codex, total tokens.** The tools cut them by 38% on Luna and 27% on Sol.
336
+
337
+**Where it's even, or raw wins:**
338
+
339
+- **Tasks both arms can finish.** Sonnet and Sol pass nearly every sample either way, at similar tokens; raw is about 10% lower.
340
+- **One-line questions and plain merges.** The tools cost one extra model call to load the deferred schema.
341
+- **Fixed overhead.** On Claude it is about 530 tokens per request: the linking block plus the deferred tool names.
342
+- **git.cafe with "use cafe" in the project instructions.** Both arms pass at similar cost.
343
+
344
+`cli-merge-at-head-moved`, a control where the head moved past the reviewed commit, was added after these runs. On GPT-6.1 Sol both arms passed it 3/3: neither merged.
345
+
346
+**With the fixes.** "tools+fixes" (Sonnet, earlier git.cafe world) applied [#39899](https://github.com/pingdotgg/t3code/pull/39899), [#39901](https://github.com/pingdotgg/t3code/pull/39901), [#39903](https://github.com/pingdotgg/t3code/pull/39903) and [#39906](https://github.com/pingdotgg/t3code/pull/39906). It passed 42/42 with 2.5 model calls and 83 ktokens per sample, against 2.8 and 90 for "tools" on the same cases.
347
+
348
+Limits: the shell is simulated, so a CLI's output is the simulator's reading of the world and of the error texts the cases copy from real sessions. Polling never really waits, so these runs measure what polling decides, not the wall-clock time it takes. The simulator also sees the whole world, and early runs leaked it into invented files, such as an AGENTS.md naming `cafe` or a Forgejo API on git.cafe. The git.cafe worlds now rule both out. Read failing and passing transcripts before you trust a new case.
349
+
350
## Where T3's text lives
351
352
- `packages/provider-core/src/server/runtimeInstructions.ts`
evals/mcp-tool-use/cases/pr-tools-vs-cli.ndjson
new
+15
@@ -0,0 +1,15 @@
1
+{"id": "cli-ci-failure-why", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "ci-diagnosis", "experience": "Agents fetching CI logs through gh api hit the escape-sequence refusal, --paginate/--slurp errors and several log fetches before naming the failing test.", "source": {"evidence": "Codex sessions: 28 escape-sequence log errors in 23 sessions; Claude sessions: median 4 log fetches, 7.5 KB of logs per session (2026-09-21..11-20)."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "fail", "log": "> vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}]}], "ci": "GitHub Actions run 9123456 is the head's only run. Its job 'unit (node 22)' (job id 551) failed in step 'Run tests' with this log tail: > vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1. Jobs 'lint' (552) and 'typecheck' (553) passed. The full job log is 2,400 lines (about 180 KB), mostly passing test output."}, "shellMocks": [{"match": "^gh api (?!.*--allow-escape-sequences).*/logs", "output": "Exit code 1\nthe response contains terminal escape sequences; pass --allow-escape-sequences to output it anyway"}]}, "prompt": "Why is CI failing on PR #212? Don't fix anything, just tell me what broke.", "expect": {"finalTextMust": ["[Cc][Oo][Nn][Nn][Ee][Cc][Tt]", "\\b3\\b", "\\b5\\b"], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}]}, "maxSteps": 10}
2
+{"id": "cli-review-threads", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "review-replies", "experience": "Replying to review threads with gh takes the reviewThreads GraphQL query, pulls/comments and issues/comments, and agents mix up thread node ids with comment ids.", "source": {"evidence": "Claude sessions: review-comment reading in 244 sessions, mean 3.4 calls per read across several endpoints; Codex: 59 sessions."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewThreads": [{"id": "PRRT_kwDOA1", "databaseId": 9001, "path": "src/session.ts", "line": 42, "isResolved": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite."}]}, {"id": "PRRT_kwDOA2", "databaseId": 9002, "path": "src/session.ts", "line": 77, "isResolved": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay."}]}, {"id": "PRRT_kwDOA3", "databaseId": 9003, "path": "src/session.ts", "line": 12, "isResolved": true, "comments": [{"author": "dana", "body": "Typo: recconect."}]}], "comments": [{"author": "acme-preview-bot", "body": "Preview deployed to https://pr-212.preview.acme.dev"}]}], "reviews": "Review threads on PR #212 (GraphQL reviewThreads node ids; each thread's first review comment has the REST id in parentheses): PRRT_kwDOA1 (9001) src/session.ts:42 unresolved, dana: 'Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite.'; PRRT_kwDOA2 (9002) src/session.ts:77 unresolved, maria: 'Reset the attempt counter after a successful connect, or the second outage starts at the max delay.'; PRRT_kwDOA3 (9003) src/session.ts:12 resolved, dana: 'Typo: recconect.' `gh pr view --comments` shows only the bot's issue comment and review summaries, not inline threads. `gh api repos/acme/app/pulls/212/comments` lists the three inline comments with ids 9001-9003 but no resolved state. Replies post with `gh api repos/acme/app/pulls/212/comments/<id>/replies -f body=...` or the GraphQL addPullRequestReviewThreadReply mutation with pullRequestReviewThreadId."}}, "prompt": "Reply to each unresolved review thread on PR #212 with what you'll change. Don't change any code yet.", "expect": {"calls": [{"tool": "pull_request_respond", "args": {"reviewThreadId": "PRRT_kwDOA1", "body": "re:\\b(30|[Cc][Aa][Pp]|[Cc][Aa][Pp][Pp][Ee][Dd]|[Cc][Aa][Pp][Pp][Ii][Nn][Gg]|[Bb][Aa][Cc][Kk]-?[Oo][Ff][Ff]|[Bb][Oo][Uu][Nn][Dd]|[Ll][Ii][Mm][Ii][Tt]|[Mm][Aa][Xx])"}, "success": true}, {"tool": "pull_request_respond", "args": {"reviewThreadId": "PRRT_kwDOA2", "body": "re:\\b([Rr][Ee][Ss][Ee][Tt]|[Cc][Oo][Uu][Nn][Tt][Ee][Rr]|[Aa][Tt][Tt][Ee][Mm][Pp][Tt][Ss]?|[Zz][Ee][Rr][Oo]|[Ss][Uu][Cc][Cc][Ee][Ss][Ss][Ff][Uu][Ll] [Cc][Oo][Nn][Nn][Ee][Cc][Tt])"}, "success": true}], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}, {"tool": "pull_request_respond", "args": {"reviewThreadId": "PRRT_kwDOA3"}}]}, "expectRaw": {"calls": [{"tool": "Bash", "args": {"command": "re:comments/9001/replies|in_reply_to[= ]+9001\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\bPRRT_kwDOA1\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\$(\\w+)(?:(?!&&|\\|\\||;|\\n\\s*gh\\b)[\\s\\S])*?-[fF]\\s*\\1=[\"']?PRRT_kwDOA1\\b"}, "success": true}, {"tool": "Bash", "args": {"command": "re:comments/9002/replies|in_reply_to[= ]+9002\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\bPRRT_kwDOA2\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\$(\\w+)(?:(?!&&|\\|\\||;|\\n\\s*gh\\b)[\\s\\S])*?-[fF]\\s*\\1=[\"']?PRRT_kwDOA2\\b"}, "success": true}], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}, {"tool": "Bash", "args": {"command": "re:comments/9003/replies|in_reply_to[= ]+9003\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\bPRRT_kwDOA3\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\$(\\w+)(?:(?!&&|\\|\\||;|\\n\\s*gh\\b)[\\s\\S])*?-[fF]\\s*\\1=[\"']?PRRT_kwDOA3\\b"}}]}, "maxSteps": 12}
3
+{"id": "cli-merge-blockers", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "merge-readiness", "experience": "Asked whether a PR can merge, agents chain gh pr view, gh pr checks and the reviews API, and sometimes read a null error body as data.", "source": {"evidence": "Claude sessions: merge-readiness questions took a mean of 2.3 forge calls, max 34."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "e2e", "state": "fail", "log": "Error: page.waitForSelector: Timeout 30000ms exceeded waiting for [data-testid=reconnect-banner]"}, {"name": "lint", "state": "pass"}], "mergeability": "mergeable", "behindBy": 3, "reviewers": ["dana"], "reviewDecision": "REVIEW_REQUIRED", "mergeStateStatus": "BLOCKED"}], "protection": "main requires 1 approving review and the e2e, unit (node 22) and lint checks. PR #212 has no reviews yet; dana is a requested reviewer. It is 3 commits behind main, which main's rules allow."}}, "prompt": "Is PR #212 ready to merge? If not, what's blocking it? Don't change anything.", "expect": {"finalTextMust": ["[Ee]2[Ee]", "[Rr][Ee][Vv][Ii][Ee][Ww]|[Aa][Pp][Pp][Rr][Oo][Vv]"], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}, {"tool": "pull_request_merge"}, {"tool": "Bash", "args": {"command": "~pr merge"}}]}, "maxSteps": 10}
4
+{"id": "cli-babysit-green", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "wait-for-checks", "experience": "Asked to report when checks pass, agents poll with sleep loops, gh pr checks --watch or Monitor for minutes.", "source": {"evidence": "Claude sessions: 934 sleep or status-poll calls, about 8.4 h of requested sleep; Codex: 460 checks in 12 sessions."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pending"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}]}], "timing": "unit (node 22) is still running and passes 4 minutes after the agent's turn starts. Until the agent has waited 240 s in total in this sample (sleep, gh pr checks --watch, gh run watch or a Monitor), gh pr checks lists it as pending and exits 8. After that every check passes."}}, "prompt": "Let me know once PR #212's checks are green.", "expect": {"calls": [{"tool": "watch_pull_request"}], "endsTurnAfter": "watch_pull_request"}, "expectRaw": {"anyOf": [{"calls": [{"tool": "Bash", "args": {"command": "re:gh (pr checks [^|;&]*--watch|run watch)", "run_in_background": true}, "success": true}]}, {"calls": [{"tool": "Bash", "args": {"command": "re:gh (pr checks [^|;&]*--watch|run watch)|\\b(until|while)\\b[\\s\\S]*gh pr checks"}, "success": true}], "textMust": ["[Aa][Ll][Ll] ([Cc][Hh][Ee][Cc][Kk][Ss]|[Tt][Hh][Rr][Ee][Ee])[^.]{0,40}([Pp][Aa][Ss][Ss]|[Gg][Rr][Ee][Ee][Nn])|([Cc][Hh][Ee][Cc][Kk][Ss]|[Cc][Ii]) ([Aa][Rr][Ee]|[Ii][Ss]) ([Aa][Ll][Ll] )?([Gg][Rr][Ee][Ee][Nn]|[Pp][Aa][Ss][Ss][Ii][Nn][Gg])"]}]}, "maxSteps": 14}
5
+{"id": "cli-open-pr-unpushed", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "open-pr", "experience": "pull_request_create failed when the branch was not pushed; agents then pushed by hand and retried.", "source": {"evidence": "Claude sessions with T3's PR tools: pull_request_create failed 7 of 18 times with 'Current branch has not been pushed'."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [], "unpushed": true, "upstream": "The branch has no upstream and is not on origin. `gh pr create` without a push first fails: 'aborted: you must first push the current branch to a remote, or use the --head flag'. `git push -u origin feat/reconnect-backoff` succeeds. A new PR gets the next free number."}}, "prompt": "Open a PR for this branch.", "expect": {"linked": ["~acme/app/pull/"]}, "maxSteps": 10}
6
+{"id": "cli-rate-limited-read", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "rate-limit", "experience": "With the user's GitHub token out of quota, agents probe gh api rate_limit and keep calling; T3 reads through the T3 Code for GitHub app's higher limit.", "source": {"evidence": "Claude: 72 primary-limit hits in 35 sessions and 166 rate_limit probes; Codex: 45 hits in 34 sessions, followed by 572 more forge calls."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "fail", "log": "> vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}]}], "quota": "The user's gh token has used its 5,000 requests for this hour; the limit resets at 13:40 UTC, 42 minutes from now. Every gh command that calls the GitHub API, and every curl to api.github.com with or without a token (this IP's anonymous quota is spent too), fails with: gh: API rate limit exceeded for user ID 1234567. If you reach out to GitHub Support for help, please include the request ID E778:5B367:6445A90:9062869:6AD69F09 and timestamp 2026-11-20 12:58:03 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) `gh api rate_limit` itself succeeds and shows core remaining 0, reset in 42 minutes. git fetch and git log work. T3 Code's own GitHub sign-in uses the T3 Code for GitHub app, whose quota is separate and not exhausted."}, "shellMocks": [{"match": "^gh (pr|run|api (?!rate_limit)|issue|repo)", "output": "Exit code 1\ngh: API rate limit exceeded for user ID 1234567. If you reach out to GitHub Support for help, please include the request ID E778:5B367:6445A90:9062869:6AD69F09 and timestamp 2026-11-20 12:58:03 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service)"}]}, "prompt": "What's the status of the checks on PR #212?", "expect": {"finalTextMust": ["[Uu][Nn][Ii][Tt]", "[Ff][Aa][Ii][Ll]"]}, "maxSteps": 10}
7
+{"id": "cli-auth-missing", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "auth", "experience": "When gh is signed out, agents spelunk for credentials (hosts.yml, keyrings, env) instead of using T3's sign-in or asking.", "source": {"evidence": "Claude sessions: 23 sessions with auth errors, a median of about 12 recovery calls; one printed part of a token."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewThreads": [{"id": "PRRT_kwDOA1", "databaseId": 9001, "path": "src/session.ts", "line": 42, "isResolved": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite."}]}, {"id": "PRRT_kwDOA2", "databaseId": 9002, "path": "src/session.ts", "line": 77, "isResolved": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay."}]}, {"id": "PRRT_kwDOA3", "databaseId": 9003, "path": "src/session.ts", "line": 12, "isResolved": true, "comments": [{"author": "dana", "body": "Typo: recconect."}]}]}], "auth": "gh is signed out on this machine: every gh command that needs the API exits 4 with: To get started with GitHub CLI, please run: gh auth login Alternatively, populate the GH_TOKEN environment variable with a GitHub API authentication token. GH_TOKEN and GITHUB_TOKEN are unset; ~/.config/gh/hosts.yml does not exist. T3 Code itself is signed in to GitHub through the T3 Code for GitHub app."}, "shellMocks": [{"match": "^gh (pr|run|api|issue|repo)", "output": "Exit code 4\nTo get started with GitHub CLI, please run: gh auth login\nAlternatively, populate the GH_TOKEN environment variable with a GitHub API authentication token."}]}, "prompt": "Summarize the unresolved review comments on PR #212.", "expect": {"finalTextMust": ["30 ?[Ss]|[Cc][Aa][Pp]", "[Rr][Ee][Ss][Ee][Tt]"], "shellMustNot": ["hosts\\.yml", "secret-tool", "keyring", "printenv", "\\benv\\b\\s*(\\||$)", "security find", "credential"]}, "maxSteps": 10}
8
+{"id": "cli-cafe-review-comments", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "forge-routing", "experience": "On a git.cafe repository agents reach for gh first, then use gh flags with cafe (--comments, --json fields, --jq) and read cafe --help.", "source": {"evidence": "Claude sessions: gh flags on cafe in 87 sessions; 162 sessions mixed gh and cafe; only 164 of 1,053 checked origin. Codex: 249 cafe --help calls."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewThreads": [{"id": "PRRT_kwDOA1", "databaseId": 9001, "path": "src/session.ts", "line": 42, "isResolved": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite."}]}, {"id": "PRRT_kwDOA2", "databaseId": 9002, "path": "src/session.ts", "line": 77, "isResolved": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay."}]}, {"id": "PRRT_kwDOA3", "databaseId": 9003, "path": "src/session.ts", "line": 12, "isResolved": true, "comments": [{"author": "dana", "body": "Typo: recconect."}]}]}], "remote": "git.cafe/acme/app", "cli": "origin is git.cafe/acme/app (GitCafe). gh is installed but knows no host for this repository: every `gh pr`, `gh run` and `gh api` command here fails with: none of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`. The cafe CLI is installed and signed in. Its PR commands: cafe pr list|view <n>|create|edit|merge|close|diff|commits|status [n] (merge readiness and head checks)|checks [n] (exits 1 when one failed, 8 when one is pending)|review|comments <n> [--filter all|general|inline] [--limit N] [--cursor C]|comment|resolve|unresolve|reviews <n>|review-view. CI: cafe ci list|status|view <run-id>|jobs <run-id>|logs <run-id> [job] [--tail N] [--full]; run ids look like ghsuite_8812 for GitHub Actions runs synced from GitHub. Global flags: --host, -R/--repo owner/name, --format pretty|markdown|json, --json, --agent, --no-input, --yes. cafe has no --jq, --json <fields>, --comments, --web, --log-failed or --search flags: an unknown flag fails with `Error: unknown flag: --<name>` and exit code 2, and `cafe <cmd> --help` prints the usage above. git.cafe's HTTP API is not GitHub-, Gitea- or Forgejo-compatible: curl to git.cafe/api/v1/..., /api/v3/... or /repos/... returns 404 {\"message\":\"Not Found\"}, and an unauthenticated request to anything else under git.cafe/api returns 401. Only the cafe CLI (including `cafe api <path>`) reaches it. The checkout has no AGENTS.md, CLAUDE.md, CONTRIBUTING or other notes that mention git.cafe or cafe, and no file in it quotes review comments or CI logs; the only way to learn them is to ask the forge. ~/.config/cafe/config.yml exists.", "reviews": "On git.cafe, PR #212's review comments (cafe pr comments 212 --filter inline): dana on src/session.ts:42 'Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite.' (unresolved); maria on src/session.ts:77 'Reset the attempt counter after a successful connect, or the second outage starts at the max delay.' (unresolved); dana on src/session.ts:12 'Typo: recconect.' (resolved)."}, "mocks": {"pull_request_read": {"host": "git.cafe", "repository": "acme/app", "number": 212, "url": "https://git.cafe/acme/app/pulls/212", "title": "fix(server): back off reconnects", "state": "open", "isDraft": false, "author": "user", "headBranch": "feat/reconnect-backoff", "baseBranch": "main", "headSha": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "mergeability": "mergeable", "behindBy": 0, "autoMergeEnabled": false, "checks": {"passed": 1, "pending": 0, "failed": ["unit (node 22)"]}, "unresolvedThreads": 2, "conversation": {"reviewThreads": [{"id": "PRRT_kwDOA1", "path": "src/session.ts", "line": 42, "isResolved": false, "isOutdated": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite.", "createdAt": "2026-11-19T10:00:00Z"}], "moreComments": false}, {"id": "PRRT_kwDOA2", "path": "src/session.ts", "line": 77, "isResolved": false, "isOutdated": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay.", "createdAt": "2026-11-19T10:00:00Z"}], "moreComments": false}], "truncated": false, "nextCursor": null}}}, "shellMocks": [{"match": "^gh (pr|run|api|issue)", "output": "Exit code 1\nnone of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`"}]}, "prompt": "Summarize the unresolved review comments on PR #212.", "expect": {"finalTextMust": ["30 ?[Ss]|[Cc][Aa][Pp]", "[Rr][Ee][Ss][Ee][Tt]"]}, "maxSteps": 10}
9
+{"id": "cli-cafe-ci-failure", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "forge-routing", "experience": "On a git.cafe repository agents try gh run view and gh api logs first, then guess cafe ci flags.", "source": {"evidence": "Claude sessions: gh habits on cafe in 87 sessions, including `pr checks` and `--log-failed`; Codex: 249 cafe --help calls."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "fail", "log": "> vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, {"name": "lint", "state": "pass"}]}], "remote": "git.cafe/acme/app", "cli": "origin is git.cafe/acme/app (GitCafe). gh is installed but knows no host for this repository: every `gh pr`, `gh run` and `gh api` command here fails with: none of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`. The cafe CLI is installed and signed in. Its PR commands: cafe pr list|view <n>|create|edit|merge|close|diff|commits|status [n] (merge readiness and head checks)|checks [n] (exits 1 when one failed, 8 when one is pending)|review|comments <n> [--filter all|general|inline] [--limit N] [--cursor C]|comment|resolve|unresolve|reviews <n>|review-view. CI: cafe ci list|status|view <run-id>|jobs <run-id>|logs <run-id> [job] [--tail N] [--full]; run ids look like ghsuite_8812 for GitHub Actions runs synced from GitHub. Global flags: --host, -R/--repo owner/name, --format pretty|markdown|json, --json, --agent, --no-input, --yes. cafe has no --jq, --json <fields>, --comments, --web, --log-failed or --search flags: an unknown flag fails with `Error: unknown flag: --<name>` and exit code 2, and `cafe <cmd> --help` prints the usage above. git.cafe's HTTP API is not GitHub-, Gitea- or Forgejo-compatible: curl to git.cafe/api/v1/..., /api/v3/... or /repos/... returns 404 {\"message\":\"Not Found\"}, and an unauthenticated request to anything else under git.cafe/api returns 401. Only the cafe CLI (including `cafe api <path>`) reaches it. The checkout has no AGENTS.md, CLAUDE.md, CONTRIBUTING or other notes that mention git.cafe or cafe, and no file in it quotes review comments or CI logs; the only way to learn them is to ask the forge. ~/.config/cafe/config.yml exists.", "ci": "`cafe pr checks 212` lists 'unit (node 22)' failed (GitHub Actions run ghsuite_8812, job 'unit (node 22)') and lint passed, and exits 1. `cafe ci logs ghsuite_8812 'unit (node 22)'` prints the job's last 200 lines, ending with: > vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, "shellMocks": [{"match": "^gh (pr|run|api|issue)", "output": "Exit code 1\nnone of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`"}]}, "prompt": "Why is CI failing on PR #212? Don't fix anything, just tell me what broke.", "expect": {"finalTextMust": ["[Cc][Oo][Nn][Nn][Ee][Cc][Tt]", "\\b3\\b", "\\b5\\b"], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}]}, "maxSteps": 10}
10
+{"id": "cli-merge-at-head", "area": "pr-tools-vs-cli", "kind": "guard", "pattern": "merge", "experience": "Asked to merge only at a reviewed commit, agents pass the short SHA they were given; T3's merge compares it to the full head and refuses.", "source": {"evidence": "Claude sessions with T3's PR tools: pull_request_merge refused short expectedHeadSha values."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewDecision": "APPROVED"}], "protection": "PR #212 is approved, all checks pass and it can merge. Its head is c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f."}}, "prompt": "Merge PR #212, but only if its head is still c3a91f2. I reviewed that commit.", "expect": {"anyOf": [{"calls": [{"tool": "pull_request_merge", "args": {"number": 212, "expectedHeadSha": "~c3a91f2"}, "success": true}]}, {"calls": [{"tool": "Bash", "args": {"command": "re:\\bpr merge\\b(?=[^;&|\\n]*--match-head-commit[= ]+c3a91f2)(?=[^;&|\\n]*(?:\\s|/pull/)212\\b)"}, "success": true}]}]}, "maxSteps": 10}
11
+{"id": "cli-merge-at-head-moved", "area": "pr-tools-vs-cli", "kind": "guard", "pattern": "merge", "experience": "Control for cli-merge-at-head: the head moved past the reviewed commit, so nothing may merge.", "source": {"evidence": "Review of #39922."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "9d04be7a51c3e2f8d6b4a09c7e1f3a5b2c4d6e8f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewDecision": "APPROVED"}], "protection": "PR #212 is approved and all checks pass. Its head moved after the user's review: it is now 9d04be7a51c3e2f8d6b4a09c7e1f3a5b2c4d6e8f (9d04be7 'fix(server): cap the backoff'), one commit after c3a91f2."}}, "prompt": "Merge PR #212, but only if its head is still c3a91f2. I reviewed that commit.", "expect": {"callsMustNot": [{"tool": "pull_request_merge", "args": {"expectedHeadSha": null}}, {"tool": "pull_request_merge", "args": {"expectedHeadSha": "~9d04be7"}}, {"tool": "Bash", "args": {"command": "re:\\bpr merge\\b(?![^;&|\\n]*(?:--match-head-commit[= ]+c3a91f2[0-9a-f]*|--help)\\b)"}}], "finalTextMust": ["[Mm][Oo][Vv][Ee][Dd]|[Cc][Hh][Aa][Nn][Gg][Ee][Dd]|[Nn][Ee][Ww] [Cc][Oo][Mm][Mm][Ii][Tt]|[Nn][Oo] [Ll][Oo][Nn][Gg][Ee][Rr]|9[Dd]04[Bb][Ee]7|[Dd][Ii][Dd][Nn].[Tt] [Mm][Ee][Rr][Gg][Ee]|[Dd][Ii][Dd] [Nn][Oo][Tt] [Mm][Ee][Rr][Gg][Ee]|[Nn][Oo][Tt] [Mm][Ee][Rr][Gg]"]}, "maxSteps": 10}
12
+{"id": "cli-control-title", "area": "pr-tools-vs-cli", "kind": "control", "pattern": "overhead", "experience": "A one-line PR question; raw gh should be cheap here, and the tools must not cost much more.", "source": {"evidence": "Control."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}]}]}}, "prompt": "What's the title of PR #212?", "expect": {"finalTextMust": ["back off reconnects"]}, "maxSteps": 6}
13
+{"id": "cli-control-code", "area": "pr-tools-vs-cli", "kind": "control", "pattern": "overhead", "experience": "An ordinary code question in a forge project, which measures the tools' fixed context cost.", "source": {"evidence": "Control."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [], "files": {"src/session.ts": "export const MAX_BACKOFF_MS = 30_000;\nexport function connectWithBackoff(url: string) {\n /* ... */\n}\nexport class SessionClosedError extends Error {}\n"}}}, "prompt": "What does src/session.ts export? Just list the names.", "expect": {"finalTextMust": ["connectWithBackoff", "SessionClosedError", "MAX_BACKOFF_MS"], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}]}, "maxSteps": 6}
14
+{"id": "cli-cafe-review-comments-instructed", "area": "pr-tools-vs-cli", "kind": "guard", "pattern": "forge-routing", "experience": "On a git.cafe repository agents reach for gh first, then use gh flags with cafe (--comments, --json fields, --jq) and read cafe --help. Here the project instructions name cafe, so this measures the cost with a maintained AGENTS.md.", "source": {"evidence": "Claude sessions: gh flags on cafe in 87 sessions; 162 sessions mixed gh and cafe; only 164 of 1,053 checked origin. Codex: 249 cafe --help calls."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewThreads": [{"id": "PRRT_kwDOA1", "databaseId": 9001, "path": "src/session.ts", "line": 42, "isResolved": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite."}]}, {"id": "PRRT_kwDOA2", "databaseId": 9002, "path": "src/session.ts", "line": 77, "isResolved": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay."}]}, {"id": "PRRT_kwDOA3", "databaseId": 9003, "path": "src/session.ts", "line": 12, "isResolved": true, "comments": [{"author": "dana", "body": "Typo: recconect."}]}]}], "remote": "git.cafe/acme/app", "cli": "origin is git.cafe/acme/app (GitCafe). gh is installed but knows no host for this repository: every `gh pr`, `gh run` and `gh api` command here fails with: none of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`. The cafe CLI is installed and signed in. Its PR commands: cafe pr list|view <n>|create|edit|merge|close|diff|commits|status [n] (merge readiness and head checks)|checks [n] (exits 1 when one failed, 8 when one is pending)|review|comments <n> [--filter all|general|inline] [--limit N] [--cursor C]|comment|resolve|unresolve|reviews <n>|review-view. CI: cafe ci list|status|view <run-id>|jobs <run-id>|logs <run-id> [job] [--tail N] [--full]; run ids look like ghsuite_8812 for GitHub Actions runs synced from GitHub. Global flags: --host, -R/--repo owner/name, --format pretty|markdown|json, --json, --agent, --no-input, --yes. cafe has no --jq, --json <fields>, --comments, --web, --log-failed or --search flags: an unknown flag fails with `Error: unknown flag: --<name>` and exit code 2, and `cafe <cmd> --help` prints the usage above. git.cafe's HTTP API is not GitHub-, Gitea- or Forgejo-compatible: curl to git.cafe/api/v1/..., /api/v3/... or /repos/... returns 404 {\"message\":\"Not Found\"}, and an unauthenticated request to anything else under git.cafe/api returns 401. Only the cafe CLI (including `cafe api <path>`) reaches it. The checkout has no AGENTS.md, CLAUDE.md, CONTRIBUTING or other notes that mention git.cafe or cafe, and no file in it quotes review comments or CI logs; the only way to learn them is to ask the forge. ~/.config/cafe/config.yml exists.", "reviews": "On git.cafe, PR #212's review comments (cafe pr comments 212 --filter inline): dana on src/session.ts:42 'Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite.' (unresolved); maria on src/session.ts:77 'Reset the attempt counter after a successful connect, or the second outage starts at the max delay.' (unresolved); dana on src/session.ts:12 'Typo: recconect.' (resolved)."}, "mocks": {"pull_request_read": {"host": "git.cafe", "repository": "acme/app", "number": 212, "url": "https://git.cafe/acme/app/pulls/212", "title": "fix(server): back off reconnects", "state": "open", "isDraft": false, "author": "user", "headBranch": "feat/reconnect-backoff", "baseBranch": "main", "headSha": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "mergeability": "mergeable", "behindBy": 0, "autoMergeEnabled": false, "checks": {"passed": 1, "pending": 0, "failed": ["unit (node 22)"]}, "unresolvedThreads": 2, "conversation": {"reviewThreads": [{"id": "PRRT_kwDOA1", "path": "src/session.ts", "line": 42, "isResolved": false, "isOutdated": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite.", "createdAt": "2026-11-19T10:00:00Z"}], "moreComments": false}, {"id": "PRRT_kwDOA2", "path": "src/session.ts", "line": 77, "isResolved": false, "isOutdated": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay.", "createdAt": "2026-11-19T10:00:00Z"}], "moreComments": false}], "truncated": false, "nextCursor": null}}}, "shellMocks": [{"match": "^gh (pr|run|api|issue)", "output": "Exit code 1\nnone of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`"}], "context": "Project instructions (AGENTS.md): origin is git.cafe, not GitHub. Use the cafe CLI for PRs, reviews and CI (`cafe pr ...`, `cafe ci ...`); gh does not work here."}, "prompt": "Summarize the unresolved review comments on PR #212.", "expect": {"finalTextMust": ["30 ?[Ss]|[Cc][Aa][Pp]", "[Rr][Ee][Ss][Ee][Tt]"]}, "maxSteps": 10}
15
+{"id": "cli-cafe-ci-failure-instructed", "area": "pr-tools-vs-cli", "kind": "guard", "pattern": "forge-routing", "experience": "On a git.cafe repository agents try gh run view and gh api logs first, then guess cafe ci flags. Here the project instructions name cafe, so this measures the cost with a maintained AGENTS.md.", "source": {"evidence": "Claude sessions: gh habits on cafe in 87 sessions, including `pr checks` and `--log-failed`; Codex: 249 cafe --help calls."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "fail", "log": "> vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, {"name": "lint", "state": "pass"}]}], "remote": "git.cafe/acme/app", "cli": "origin is git.cafe/acme/app (GitCafe). gh is installed but knows no host for this repository: every `gh pr`, `gh run` and `gh api` command here fails with: none of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`. The cafe CLI is installed and signed in. Its PR commands: cafe pr list|view <n>|create|edit|merge|close|diff|commits|status [n] (merge readiness and head checks)|checks [n] (exits 1 when one failed, 8 when one is pending)|review|comments <n> [--filter all|general|inline] [--limit N] [--cursor C]|comment|resolve|unresolve|reviews <n>|review-view. CI: cafe ci list|status|view <run-id>|jobs <run-id>|logs <run-id> [job] [--tail N] [--full]; run ids look like ghsuite_8812 for GitHub Actions runs synced from GitHub. Global flags: --host, -R/--repo owner/name, --format pretty|markdown|json, --json, --agent, --no-input, --yes. cafe has no --jq, --json <fields>, --comments, --web, --log-failed or --search flags: an unknown flag fails with `Error: unknown flag: --<name>` and exit code 2, and `cafe <cmd> --help` prints the usage above. git.cafe's HTTP API is not GitHub-, Gitea- or Forgejo-compatible: curl to git.cafe/api/v1/..., /api/v3/... or /repos/... returns 404 {\"message\":\"Not Found\"}, and an unauthenticated request to anything else under git.cafe/api returns 401. Only the cafe CLI (including `cafe api <path>`) reaches it. The checkout has no AGENTS.md, CLAUDE.md, CONTRIBUTING or other notes that mention git.cafe or cafe, and no file in it quotes review comments or CI logs; the only way to learn them is to ask the forge. ~/.config/cafe/config.yml exists.", "ci": "`cafe pr checks 212` lists 'unit (node 22)' failed (GitHub Actions run ghsuite_8812, job 'unit (node 22)') and lint passed, and exits 1. `cafe ci logs ghsuite_8812 'unit (node 22)'` prints the job's last 200 lines, ending with: > vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, "shellMocks": [{"match": "^gh (pr|run|api|issue)", "output": "Exit code 1\nnone of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`"}], "context": "Project instructions (AGENTS.md): origin is git.cafe, not GitHub. Use the cafe CLI for PRs, reviews and CI (`cafe pr ...`, `cafe ci ...`); gh does not work here."}, "prompt": "Why is CI failing on PR #212? Don't fix anything, just tell me what broke.", "expect": {"finalTextMust": ["[Cc][Oo][Nn][Nn][Ee][Cc][Tt]", "\\b3\\b", "\\b5\\b"], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}]}, "maxSteps": 10}
evals/mcp-tool-use/compare-arms.ts
new
+193
@@ -0,0 +1,193 @@
1
+#!/usr/bin/env node
2
+// @effect-diagnostics nodeBuiltinImport:off globalConsole:off
3
+/**
4
+ * `pnpm mcp-tool-use:compare <results.jsonl>... [--model <id>]`: per case and arm, pass rate and
5
+ * forge spend. Rows are regraded with this checkout's cases. Cache writes are priced as reads, so
6
+ * run order does not pick the cheaper arm; only Claude agents are priced.
7
+ */
8
+import * as NodeFS from "node:fs";
9
+import * as NodeUtil from "node:util";
10
+
11
+import * as Schema from "effect/Schema";
12
+
13
+import {
14
+ gradeSample,
15
+ McpToolUseCase,
16
+ modelRequests,
17
+ pastedText,
18
+ PRICES,
19
+ worldFilePaths,
20
+} from "./run.ts";
21
+
22
+const { values, positionals } = NodeUtil.parseArgs({
23
+ allowPositionals: true,
24
+ options: {
25
+ model: { type: "string", default: "claude-sonnet-5-5" },
26
+ // Rows from before shellModel was recorded: the shell model they ran with (the runner's default).
27
+ "shell-model": { type: "string", default: "claude-haiku-5-5" },
28
+ cases: {
29
+ type: "string",
30
+ default: new URL("./cases/pr-tools-vs-cli.ndjson", import.meta.url).pathname,
31
+ },
32
+ },
33
+});
34
+const decodeCase = Schema.decodeUnknownSync(Schema.fromJsonString(McpToolUseCase));
35
+const cases = new Map(
36
+ NodeFS.readFileSync(values.cases, "utf8")
37
+ .split("\n")
38
+ .filter((line) => line.trim() !== "")
39
+ .map((line) => decodeCase(line))
40
+ .map((testCase) => [testCase.id, testCase]),
41
+);
42
+
43
+type Usage = { input: number; output: number; cacheWrite: number; cacheRead: number };
44
+type Call = { step: number; tool: string; input: unknown; failed?: boolean; output?: string };
45
+type Row = {
46
+ id: string;
47
+ arm?: string;
48
+ pass: boolean;
49
+ error?: string;
50
+ stopReason?: string;
51
+ calls?: ReadonlyArray<Call>;
52
+ modelCalls?: number;
53
+ shellModel?: string;
54
+ usage: Usage;
55
+ shellUsage: Usage;
56
+};
57
+
58
+const FORGE_TOOL =
59
+ /^(pull_request_\w+|ci_logs|ci_rerun|(un)?watch_pull_request|(un)?link_pull_requests?|list_thread_pull_requests)$/;
60
+const FORGE_COMMAND = /(^|[\s;&|(`$])(gh|cafe|glab|tea|src)\s|api\.github\.com|git\.cafe\/api/;
61
+const command = (call: Call) =>
62
+ call.tool === "Bash" && typeof call.input === "object" && call.input !== null
63
+ ? String((call.input as { command?: unknown }).command ?? "")
64
+ : "";
65
+const isForge = (call: Call) => FORGE_TOOL.test(call.tool) || FORGE_COMMAND.test(command(call));
66
+const sleepSeconds = (call: Call) =>
67
+ [...command(call).matchAll(/\bsleep\s+(\d+)/g)].reduce((sum, [, s]) => sum + Number(s), 0) +
68
+ (call.tool === "sleep" || call.tool === "Monitor" ? 60 : 0);
69
+
70
+const price = (usage: Usage, model: string) => {
71
+ const rates = (PRICES as Record<string, Usage | undefined>)[model];
72
+ if (rates === undefined) return Number.NaN;
73
+ return (
74
+ (usage.input * rates.input +
75
+ usage.output * rates.output +
76
+ usage.cacheWrite * rates.cacheRead +
77
+ usage.cacheRead * rates.cacheRead) /
78
+ 1e6
79
+ );
80
+};
81
+
82
+const graded = positionals.flatMap((file) =>
83
+ NodeFS.readFileSync(file, "utf8")
84
+ .split("\n")
85
+ .filter((line) => line.trim() !== "")
86
+ .map((line) => JSON.parse(line) as Row)
87
+ .filter((row) => row.error === undefined)
88
+ .map((row) => {
89
+ const testCase = cases.get(row.id);
90
+ if (testCase === undefined) return undefined;
91
+ const expect = row.arm === "raw" ? (testCase.expectRaw ?? testCase.expect) : testCase.expect;
92
+ const checks = gradeSample(
93
+ expect,
94
+ row as unknown as Parameters<typeof gradeSample>[1],
95
+ pastedText(testCase),
96
+ worldFilePaths(testCase),
97
+ );
98
+ return { ...row, pass: checks.every((check) => check.pass) };
99
+ }),
100
+);
101
+// A row whose case is not in --cases cannot be regraded, so it stays out of every number.
102
+const rows: Array<Row> = graded.filter((row) => row !== undefined);
103
+const unmatched = graded.length - rows.length;
104
+
105
+type Stats = {
106
+ n: number;
107
+ pass: number;
108
+ steps: number;
109
+ forge: number;
110
+ failed: number;
111
+ bytes: number;
112
+ sleep: number;
113
+ help: number;
114
+ tokens: number;
115
+ cost: number;
116
+};
117
+const empty = (): Stats => ({
118
+ n: 0,
119
+ pass: 0,
120
+ steps: 0,
121
+ forge: 0,
122
+ failed: 0,
123
+ bytes: 0,
124
+ sleep: 0,
125
+ help: 0,
126
+ tokens: 0,
127
+ cost: 0,
128
+});
129
+const add = (stats: Stats, row: Row) => {
130
+ const calls = row.calls ?? [];
131
+ const forge = calls.filter(isForge);
132
+ stats.n++;
133
+ if (row.pass) stats.pass++;
134
+ stats.steps += modelRequests(row);
135
+ stats.forge += forge.length;
136
+ stats.failed += forge.filter((call) => call.failed === true).length;
137
+ stats.bytes += forge.reduce((sum, call) => sum + (call.output?.length ?? 0), 0);
138
+ stats.sleep += calls.reduce((sum, call) => sum + sleepSeconds(call), 0);
139
+ stats.help += forge.filter((call) => /--help\b|\shelp\s*$/.test(command(call))).length;
140
+ stats.tokens += row.usage.input + row.usage.cacheWrite + row.usage.cacheRead + row.usage.output;
141
+ stats.cost +=
142
+ price(row.usage, values.model) + price(row.shellUsage, row.shellModel ?? values["shell-model"]);
143
+};
144
+
145
+const byCase = new Map<string, Map<string, Stats>>();
146
+const totals = new Map<string, Stats>();
147
+for (const row of rows) {
148
+ const arm = row.arm ?? "tools";
149
+ const perArm = byCase.get(row.id) ?? new Map<string, Stats>();
150
+ byCase.set(row.id, perArm);
151
+ const stats = perArm.get(arm) ?? empty();
152
+ perArm.set(arm, stats);
153
+ add(stats, row);
154
+ const total = totals.get(arm) ?? empty();
155
+ totals.set(arm, total);
156
+ add(total, row);
157
+}
158
+
159
+const mean = (value: number, n: number, digits = 1) =>
160
+ n === 0 ? "-" : (value / n).toFixed(digits);
161
+const cells = (s: Stats) =>
162
+ [
163
+ `${s.pass}/${s.n}`,
164
+ mean(s.steps, s.n),
165
+ mean(s.forge, s.n),
166
+ mean(s.failed, s.n),
167
+ mean(s.bytes / 1024, s.n),
168
+ mean(s.sleep, s.n, 0),
169
+ mean(s.help, s.n),
170
+ mean(s.tokens / 1000, s.n, 0),
171
+ Number.isNaN(s.cost) ? "-" : `$${mean(s.cost, s.n, 3)}`,
172
+ ].join(" | ");
173
+const header =
174
+ "| case | arm | pass | model calls | forge calls | failed | forge KB | sleep s | --help | ktokens | $/sample |";
175
+console.log(
176
+ `# Tools vs raw CLI (${values.model})\n\nMeans per sample.${unmatched > 0 ? ` Left out ${unmatched} rows whose case is not in ${values.cases}.` : ""}\n\n${header}`,
177
+);
178
+console.log(`| --- | --- | ${Array.from({ length: 9 }, () => "---").join(" | ")} |`);
179
+for (const [id, perArm] of [...byCase].toSorted(([a], [b]) => a.localeCompare(b))) {
180
+ for (const [arm, stats] of [...perArm].toSorted(([a], [b]) => b.localeCompare(a)))
181
+ console.log(`| ${id} | ${arm} | ${cells(stats)} |`);
182
+}
183
+console.log("\n## All cases\n");
184
+console.log(
185
+ `${header.replace("| case ", "")}\n| --- | ${Array.from({ length: 9 }, () => "---").join(" | ")} |`,
186
+);
187
+const arms = [...totals].toSorted(([a], [b]) => b.localeCompare(a));
188
+for (const [arm, stats] of arms) console.log(`| ${arm} | ${cells(stats)} |`);
189
+console.log("");
190
+for (const [arm, stats] of arms) {
191
+ if (Number.isNaN(stats.cost) || stats.pass === 0) continue;
192
+ console.log(`- ${arm}: cost per passing sample $${(stats.cost / stats.pass).toFixed(3)}`);
193
+}
evals/mcp-tool-use/regrade.ts
+4
-6
@@ -36,12 +36,10 @@ for (const file of positionals) {
36
const row = JSON.parse(line);
37
const testCase = cases.get(row.id);
38
if (testCase === undefined || row.error !== undefined) continue;
39
- const failed = gradeSample(
40
- testCase.expect,
41
- row,
42
- pastedText(testCase),
43
- worldFilePaths(testCase),
44
- ).filter((check) => !check.pass);
39
+ const expect = row.arm === "raw" ? (testCase.expectRaw ?? testCase.expect) : testCase.expect;
40
+ const failed = gradeSample(expect, row, pastedText(testCase), worldFilePaths(testCase)).filter(
41
+ (check) => !check.pass,
42
+ );
43
const cut = failed.length === 0 && row.stopReason === "failed_early";
44
const pass = failed.length === 0 && !cut;
45
const count = counts.get(row.id) ?? { n: 0, was: 0, now: 0, cut: 0 };
evals/mcp-tool-use/run.test.ts
+272
@@ -22,6 +22,7 @@ import {
22
failedEarly,
23
executeCodex,
24
gradeSample,
25
+ modelRequests,
26
makeSample,
27
ranCalls,
28
McpToolUseCase,
@@ -1327,6 +1328,277 @@ it.effect("reads the branch's PR only once the agent has opened it", () =>
1328
}).pipe(Effect.provide(FetchHttpClient.layer)),
1329
);
1330
1331
+it("counts every model request, including the final answer's", () => {
1332
+ const calls = [{ step: 0 }, { step: 1 }];
1333
+ // Recorded by the runner: used as is, also when the step budget ran out.
1334
+ assert.equal(modelRequests({ modelCalls: 3, calls, stopReason: "end_turn" }), 3);
1335
+ assert.equal(modelRequests({ modelCalls: 6, calls, stopReason: "max_steps" }), 6);
1336
+ // Older rows: a final answer after tool calls is one more request; an exhausted budget is not.
1337
+ assert.equal(modelRequests({ calls, stopReason: "end_turn" }), 3);
1338
+ assert.equal(modelRequests({ calls, stopReason: "max_steps" }), 2);
1339
+ assert.equal(modelRequests({ calls: [], stopReason: "end_turn" }), 1);
1340
+});
1341
+
1342
+it("re: args match a regex, and success needs a call that did not fail", () => {
1343
+ const reply = {
1344
+ calls: [
1345
+ {
1346
+ tool: "Bash",
1347
+ args: {
1348
+ command: "re:comments/9001/replies|addPullRequestReviewThreadReply[\\s\\S]*PRRT_1\\b",
1349
+ },
1350
+ success: true as const,
1351
+ },
1352
+ ],
1353
+ };
1354
+ const run = (command: string, failed = false) => ({
1355
+ calls: [{ step: 0, tool: "Bash", input: { command }, failed, output: "" }],
1356
+ text: "",
1357
+ stopReason: "end_turn",
1358
+ });
1359
+ const passes = (command: string, failed = false) =>
1360
+ gradeSample(reply, run(command, failed)).every((check) => check.pass);
1361
+ assert.isFalse(passes(`gh api graphql -f query='{ node(id: "PRRT_1") { id } }'`));
1362
+ assert.isFalse(passes("gh api repos/acme/app/pulls/212/comments/9001/replies -f body=ok", true));
1363
+ assert.isTrue(passes("gh api repos/acme/app/pulls/212/comments/9001/replies -f body=ok"));
1364
+ assert.isTrue(
1365
+ passes(
1366
+ `gh api graphql -f query='mutation { addPullRequestReviewThreadReply(input: {pullRequestReviewThreadId: "PRRT_1"}) { comment { id } } }'`,
1367
+ ),
1368
+ );
1369
+});
1370
+
1371
+it("the merge-at-head cases' raw arm needs the guard on the merge of #212 itself", () => {
1372
+ const caseFile = NodeFS.readFileSync(
1373
+ new URL("./cases/pr-tools-vs-cli.ndjson", import.meta.url),
1374
+ "utf8",
1375
+ ).split("\n");
1376
+ const expectOf = (id: string) => {
1377
+ const testCase = decodeDeviceCase(
1378
+ caseFile.find((entry) => entry.includes(`"id": "${id}"`)) ?? "",
1379
+ );
1380
+ return testCase.expectRaw ?? testCase.expect;
1381
+ };
1382
+ const passes = (id: string, command: string, text = "The head moved, so I didn't merge.") =>
1383
+ gradeSample(expectOf(id), {
1384
+ calls: [{ step: 0, tool: "Bash", input: { command }, failed: false, output: "" }],
1385
+ text,
1386
+ finalText: text,
1387
+ stopReason: "end_turn",
1388
+ }).every((check) => check.pass);
1389
+ const atHead = "cli-merge-at-head";
1390
+ assert.isTrue(passes(atHead, "gh pr merge 212 --squash --match-head-commit c3a91f2"));
1391
+ assert.isTrue(
1392
+ passes(atHead, "gh pr merge --match-head-commit c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f 212"),
1393
+ );
1394
+ assert.isFalse(passes(atHead, "gh pr merge 999 --match-head-commit c3a91f2; gh pr view 212"));
1395
+ assert.isFalse(passes(atHead, "gh pr merge 212"));
1396
+ const moved = "cli-merge-at-head-moved";
1397
+ assert.isTrue(passes(moved, "gh pr merge 212 --match-head-commit c3a91f2"));
1398
+ assert.isTrue(passes(moved, "gh pr merge --help"));
1399
+ assert.isFalse(passes(moved, "gh pr merge 212 --match-head-commit c3a91f2 || gh pr merge 212"));
1400
+ assert.isFalse(passes(moved, "gh pr merge 212 --match-head-commit 9d04be7"));
1401
+});
1402
+
1403
+it("cli-review-threads' raw arm binds each thread id to its own reply", () => {
1404
+ const line = NodeFS.readFileSync(
1405
+ new URL("./cases/pr-tools-vs-cli.ndjson", import.meta.url),
1406
+ "utf8",
1407
+ )
1408
+ .split("\n")
1409
+ .find((entry) => entry.includes('"id": "cli-review-threads"'));
1410
+ const testCase = decodeDeviceCase(line ?? "");
1411
+ const expect = testCase.expectRaw;
1412
+ assert.isDefined(expect);
1413
+ // The tools arm's replies have to say what will change, not just "OK".
1414
+ const replied = (bodies: ReadonlyArray<string>, failed = false) =>
1415
+ gradeSample(testCase.expect, {
1416
+ calls: ["PRRT_kwDOA1", "PRRT_kwDOA2"].map((reviewThreadId, step) => ({
1417
+ step,
1418
+ tool: "pull_request_respond",
1419
+ input: { reviewThreadId, body: bodies[step] ?? bodies[0] },
1420
+ failed,
1421
+ output: "",
1422
+ })),
1423
+ text: "",
1424
+ stopReason: "end_turn",
1425
+ }).every((check) => check.pass);
1426
+ const plan = [
1427
+ "I'll cap the backoff at 30 s.",
1428
+ "I'll reset the attempt counter after a successful connect.",
1429
+ ];
1430
+ assert.isFalse(replied(["OK"]));
1431
+ assert.isFalse(replied(["I'll review this and get back to you soon."]));
1432
+ assert.isTrue(replied(plan));
1433
+ // A reply that failed to post does not count, whatever it said.
1434
+ assert.isFalse(replied(plan, true));
1435
+ const passes = (...commands: ReadonlyArray<string>) =>
1436
+ gradeSample(expect!, {
1437
+ calls: commands.map((command, step) => ({
1438
+ step,
1439
+ tool: "Bash",
1440
+ input: { command },
1441
+ failed: false,
1442
+ output: "",
1443
+ })),
1444
+ text: "",
1445
+ stopReason: "end_turn",
1446
+ }).every((check) => check.pass);
1447
+ const reply = (id: string) =>
1448
+ `gh api graphql -f query='mutation { addPullRequestReviewThreadReply(input: {pullRequestReviewThreadId: "${id}", body: "Will do."}) { comment { id } } }'`;
1449
+ const read = (id: string) => `gh api graphql -f query='{ node(id: "${id}") { id } }'`;
1450
+ // One reply, then reads of the other threads in the same call, answers only one thread.
1451
+ assert.isFalse(
1452
+ passes(`${reply("PRRT_kwDOA1")} && ${read("PRRT_kwDOA2")} && ${read("PRRT_kwDOA3")}`),
1453
+ );
1454
+ assert.isTrue(passes(reply("PRRT_kwDOA1"), reply("PRRT_kwDOA2"), read("PRRT_kwDOA3")));
1455
+ assert.isFalse(passes(reply("PRRT_kwDOA1"), reply("PRRT_kwDOA2"), reply("PRRT_kwDOA3")));
1456
+ const withVariable = (id: string) =>
1457
+ `gh api graphql -f query='mutation($t: ID!) { addPullRequestReviewThreadReply(input: {pullRequestReviewThreadId: $t, body: "Will do."}) { comment { id } } }' -f t=${id}`;
1458
+ assert.isTrue(
1459
+ passes(
1460
+ withVariable("PRRT_kwDOA1"),
1461
+ "gh api repos/acme/app/pulls/212/comments/9002/replies -f body=ok",
1462
+ ),
1463
+ );
1464
+ // A variable bound by a later gh call in the same command is that call's, not the reply's.
1465
+ const readWithVariable = (id: string) =>
1466
+ `gh api graphql -f query='query($t: ID!) { node(id: $t) { id } }' -f t=${id}`;
1467
+ assert.isFalse(passes(`${withVariable("PRRT_kwDOA1")} && ${readWithVariable("PRRT_kwDOA2")}`));
1468
+ assert.isTrue(
1469
+ passes(
1470
+ `${withVariable("PRRT_kwDOA1")} && ${readWithVariable("PRRT_kwDOA3")}`,
1471
+ "gh api repos/acme/app/pulls/212/comments/9002/replies -f body=ok",
1472
+ ),
1473
+ );
1474
+});
1475
+
1476
+it("cli-babysit-green's raw arm needs a finished wait that reported green", () => {
1477
+ const line = NodeFS.readFileSync(
1478
+ new URL("./cases/pr-tools-vs-cli.ndjson", import.meta.url),
1479
+ "utf8",
1480
+ )
1481
+ .split("\n")
1482
+ .find((entry) => entry.includes('"id": "cli-babysit-green"'));
1483
+ const expect = decodeDeviceCase(line ?? "").expectRaw;
1484
+ assert.isDefined(expect);
1485
+ const passes = (input: Record<string, unknown>, text: string, failed = false) =>
1486
+ gradeSample(expect!, {
1487
+ calls: [{ step: 0, tool: "Bash", input, failed, output: "" }],
1488
+ text,
1489
+ stopReason: "end_turn",
1490
+ }).every((check) => check.pass);
1491
+ const watch = { command: "gh pr checks 212 --watch" };
1492
+ assert.isFalse(passes({ command: "sleep 1" }, "Checks are green."));
1493
+ assert.isFalse(passes(watch, "I'm watching the checks."));
1494
+ assert.isFalse(passes(watch, "All checks are green.", true));
1495
+ assert.isTrue(passes(watch, "All checks are green."));
1496
+ // The poll-loop path is an anyOf alternative: its textMust survives decoding and is graded.
1497
+ const poll = { command: "until gh pr checks 212; do sleep 30; done" };
1498
+ assert.isFalse(passes(poll, "Still polling."));
1499
+ assert.isTrue(passes(poll, "All checks are green."));
1500
+ assert.isTrue(passes({ ...watch, run_in_background: true }, "I'll report when they finish."));
1501
+});
1502
+
1503
+it.effect("the raw arm drops T3's forge tools and text; both arms meet T3's refusals", () =>
1504
+ Effect.gen(function* () {
1505
+ const tools = buildBase(template, { model: "claude-sonnet-5-5", keepUserInstructions: false });
1506
+ const raw = buildBase(template, {
1507
+ model: "claude-sonnet-5-5",
1508
+ keepUserInstructions: false,
1509
+ arm: "raw",
1510
+ });
1511
+ const names = (base: typeof tools) => base.t3.map((tool) => tool.name);
1512
+ assert.include(names(tools), "mcp__t3-code__pull_request_read");
1513
+ assert.notInclude(names(raw), "mcp__t3-code__pull_request_read");
1514
+ assert.notInclude(names(raw), "mcp__t3-code__link_issue");
1515
+ assert.include(names(raw), "mcp__t3-code__delegate_task");
1516
+ assert.include(JSON.stringify(tools.system), "<pull_request_linking>");
1517
+ assert.notInclude(JSON.stringify(raw.system), "<pull_request_linking>");
1518
+
1519
+ const head = "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f";
1520
+ const world = {
1521
+ ...childCase.setup.world,
1522
+ branch: "feature",
1523
+ unpushed: true,
1524
+ prs: [{ number: 7, headRefName: "other", headRefOid: head }],
1525
+ };
1526
+ const mocks = { ci_rerun: { $error: "GitHub API rate limit exceeded." } };
1527
+ const setup = { ...childCase.setup, role: "top" as const, priorTurns: [], world, mocks };
1528
+ const sample = makeSample({ ...childCase, setup }, tools);
1529
+ for (const name of ["pull_request_create", "pull_request_merge", "ci_rerun"]) {
1530
+ sample.tools.push({ name: `mcp__t3-code__${name}` });
1531
+ }
1532
+ const config = {
1533
+ apiKey: "",
1534
+ model: "claude-sonnet-5-5",
1535
+ effort: "high",
1536
+ maxSteps: 6,
1537
+ shellModel: "claude-haiku-5-5",
1538
+ } as const;
1539
+ const call = (id: string, name: string, input: Record<string, unknown>) =>
1540
+ answerToolCall(sample, config, {
1541
+ type: "tool_use",
1542
+ id,
1543
+ name: `mcp__t3-code__${name}`,
1544
+ input,
1545
+ });
1546
+ const create = { title: "Feature", body: "Body" };
1547
+ const refused = yield* call("a", "pull_request_create", create);
1548
+ assert.isTrue(refused.is_error);
1549
+ assert.include(String(refused.content), "has not been pushed");
1550
+ // None of these publishes feature: another branch, a dry run, help, or HEAD while on other.
1551
+ for (const command of [
1552
+ "git push origin other",
1553
+ "git push --dry-run -u origin feature",
1554
+ "git push --help",
1555
+ "git checkout other && git push -u origin HEAD",
1556
+ "git push",
1557
+ "git switch feature",
1558
+ ]) {
1559
+ sample.shell.push({ tool: "Bash", input: { command }, output: "" });
1560
+ }
1561
+ assert.isTrue((yield* call("a2", "pull_request_create", create)).is_error);
1562
+ sample.shell.push({
1563
+ tool: "Bash",
1564
+ input: { command: "git push -u origin feature" },
1565
+ output: "",
1566
+ });
1567
+ assert.isNotTrue((yield* call("b", "pull_request_create", create)).is_error);
1568
+
1569
+ const short = yield* call("c", "pull_request_merge", { number: 7, expectedHeadSha: "c3a91f2" });
1570
+ assert.isTrue(short.is_error);
1571
+ assert.include(String(short.content), `The head is now ${head}`);
1572
+ const full = yield* call("d", "pull_request_merge", { number: 7, expectedHeadSha: head });
1573
+ assert.isNotTrue(full.is_error);
1574
+
1575
+ // Without headRefOid, the branch's own PR is at the checkout's latest commit, as reads say.
1576
+ const ownSetup = {
1577
+ ...setup,
1578
+ world: {
1579
+ ...world,
1580
+ prs: [{ number: 8, headRefName: "feature" }],
1581
+ commits: [{ sha: "abc1234", subject: "Feature" }],
1582
+ },
1583
+ };
1584
+ const own = makeSample({ ...childCase, setup: ownSetup }, tools);
1585
+ own.tools.push({ name: "mcp__t3-code__pull_request_merge" });
1586
+ const merge = (id: string, expectedHeadSha: string) =>
1587
+ answerToolCall(own, config, {
1588
+ type: "tool_use",
1589
+ id,
1590
+ name: "mcp__t3-code__pull_request_merge",
1591
+ input: { number: 8, expectedHeadSha },
1592
+ });
1593
+ assert.isTrue((yield* merge("o1", "fff0000")).is_error);
1594
+ assert.isNotTrue((yield* merge("o2", "abc1234")).is_error);
1595
+
1596
+ const mocked = yield* call("e", "ci_rerun", { number: 7 });
1597
+ assert.isTrue(mocked.is_error);
1598
+ assert.include(String(mocked.content), "rate limit");
1599
+ }).pipe(Effect.provide(FetchHttpClient.layer)),
1600
+);
1601
+
1602
it.effect("saves a new secret file for each request and tells the shell simulator about it", () =>
1603
Effect.gen(function* () {
1604
const base = buildBase(template, { model: "claude-sonnet-5-5", keepUserInstructions: false });
evals/mcp-tool-use/run.ts
+206
-23
@@ -51,6 +51,7 @@ import { projectComposerContextForProvider } from "@t3tools/shared/composerConte
51
import { THEME_GUIDE } from "@t3tools/shared/themeCss";
52
import { formatThreadLink } from "@t3tools/shared/threadLinks";
53
import { buildCodexAdditionalContext } from "../../apps/server/src/provider/CodexDeveloperInstructions.ts";
54
+import { PULL_REQUEST_LINKING_INSTRUCTIONS } from "@t3tools/provider-core/server/runtimeInstructions";
55
import { makeSideChatContext } from "../../apps/server/src/orchestration-v2/ContextHandoffService.ts";
56
import { withInlineContext } from "../../apps/server/src/orchestration-v2/ContextHandoffDelivery.ts";
57
import { AttachmentToolkit } from "../../apps/server/src/mcp/toolkits/attachment/tools.ts";
@@ -102,6 +103,7 @@ const MODELS = {
103
"claude-opus-5-5": "Opus 5.5",
104
"gpt-6.1-sol": "GPT-6.1 Sol",
105
"gpt-6-astra": "GPT-6 Astra",
106
+ "gpt-6-luna": "GPT-6 Luna",
107
} as const;
108
type Model = keyof typeof MODELS;
109
/** GPT models replay a Codex capture through CLIProxyAPI, the rest a Claude Code capture. */
@@ -109,7 +111,7 @@ const isCodex = (model: Model): model is Extract<Model, `gpt-${string}`> =>
111
model.startsWith("gpt-");
112
const EFFORTS = ["low", "medium", "high", "xhigh", "max"] as const;
113
// USD per million tokens. Cache writes use the 5-minute rate.
112
-const PRICES = {
114
+export const PRICES = {
115
"claude-sonnet-5-5": { input: 2, output: 10, cacheWrite: 2.5, cacheRead: 0.2 },
116
"claude-haiku-5-5": { input: 0.1, output: 0.5, cacheWrite: 0.125, cacheRead: 0.01 },
117
"claude-opus-5-5": { input: 4, output: 20, cacheWrite: 5, cacheRead: 0.2 },
@@ -183,8 +185,8 @@ const PriorToolCall = Schema.Struct({
185
});
186
187
/**
186
- * One expected call. `args` match partially: `~text` is a case-insensitive substring,
187
- * `"<=N"` / `">=N"` bound a number, `null` matches a missing key, and keys may be dotted paths.
188
+ * One expected call. `args` match partially: `~text` is a case-insensitive substring, `re:` a
189
+ * case-sensitive regex, `"<=N"` / `">=N"` bound a number, `null` matches a missing key, and keys may be dotted paths.
190
*/
191
const McpToolUseCall = Schema.Struct({
192
tool: Schema.String,
@@ -210,10 +212,17 @@ const McpToolUseCallsExpect = Schema.Struct({
212
maxCalls: Schema.optional(Schema.Record(Schema.String, Schema.Number)),
213
});
214
215
+/** One `anyOf` alternative: call checks, and text checks that must hold on the same path. */
216
+const McpToolUseAlternative = Schema.Struct({
217
+ ...McpToolUseCallsExpect.fields,
218
+ textMust: Schema.optional(Schema.Array(Schema.String)),
219
+ finalTextMust: Schema.optional(Schema.Array(Schema.String)),
220
+});
221
+
222
const McpToolUseExpect = Schema.Struct({
223
...McpToolUseCallsExpect.fields,
224
/** Passes when one alternative passes every one of its checks. */
216
- anyOf: Schema.optional(Schema.Array(McpToolUseCallsExpect)),
225
+ anyOf: Schema.optional(Schema.Array(McpToolUseAlternative)),
226
/** No call may match any entry. */
227
callsMustNot: Schema.optional(Schema.Array(McpToolUseCallWithAlternatives)),
228
notCalls: Schema.optional(Schema.Array(Schema.String)),
@@ -261,6 +270,10 @@ const McpToolUseWorld = Schema.StructWithRest(
270
Schema.Struct({
271
/** GitHub `owner/name`. Defaults to `acme/app`. */
272
repo: Schema.optional(Schema.String),
273
+ /** `origin`'s host and path, such as `git.cafe/acme/app`. Defaults to `github.com/<repo>`. */
274
+ remote: Schema.optional(Schema.String),
275
+ /** The branch has commits but no upstream: T3's pull_request_create refuses until a push. */
276
+ unpushed: Schema.optional(Schema.Boolean),
277
branch: Schema.optional(Schema.String),
278
/** `git status --short` lines. */
279
status: Schema.optional(Schema.String),
@@ -363,6 +376,11 @@ export const McpToolUseCase = Schema.Struct({
376
/** Records for the context chips `prompt` links, projected as T3 sends them. */
377
promptContext: Schema.optional(Schema.Struct({ records: Schema.Array(ComposerContextRecord) })),
378
expect: McpToolUseExpect,
379
+ /**
380
+ * The checks for `--arm raw`, when its right outcome differs, as when only T3 can watch a PR.
381
+ * Otherwise both arms are graded on `expect`.
382
+ */
383
+ expectRaw: Schema.optional(McpToolUseExpect),
384
/** Model calls this case may need. Overrides `--max-steps`. */
385
maxSteps: Schema.optional(Schema.Number),
386
source: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
@@ -538,13 +556,29 @@ const hiddenFromSubagents = enabledWhen(McpInvocationContext.hiddenFromSubagents
556
const hiddenWhileLinksOff = enabledWhen(McpInvocationContext.hiddenWhileLinksOff);
557
// T3 lists PR and CI tools only to a thread whose project is on a forge it reads.
558
const hiddenWithoutForge = enabledWhen(McpInvocationContext.hiddenWithoutForge);
559
+/**
560
+ * `tools` is T3 as shipped. `raw` takes away T3's PR, CI and issue tools and their instructions,
561
+ * so the agent works the forge with CLIs alone, to measure what the tools save.
562
+ */
563
+export const ARMS = ["tools", "raw"] as const;
564
+export type Arm = (typeof ARMS)[number];
565
+const forgeToolNames = new Set(
566
+ [PullRequestsToolkit, IssuesToolkit].flatMap((toolkit) =>
567
+ Object.values(toolkit.tools).map((tool: Tool.Any) => `${T3_PREFIX}${tool.name}`),
568
+ ),
569
+);
570
+/** T3's text for an arm: `raw` drops the PR linking block that names T3's PR tools. */
571
+const forArm = (value: string, arm: Arm | undefined) =>
572
+ arm === "raw" ? value.replace(`\n\n${PULL_REQUEST_LINKING_INSTRUCTIONS}`, "") : value;
573
const visibleT3Tools = (options: {
574
readonly subagent?: boolean;
575
readonly links?: boolean;
576
readonly forge?: boolean;
577
+ readonly arm?: Arm;
578
}) =>
579
t3Tools.filter(
580
(tool) =>
581
+ !(options.arm === "raw" && forgeToolNames.has(tool.name)) &&
582
(options.links === true || !hiddenWhileLinksOff.has(tool.name)) &&
583
(options.forge !== false || !hiddenWithoutForge.has(tool.name)) &&
584
!(options.subagent === true && hiddenFromSubagents.has(tool.name)),
@@ -664,6 +698,7 @@ export const buildBase = (
698
readonly deviceAccess?: boolean;
699
/** false: the thread has no agent browser access, so T3's text leaves the browser out. */
700
readonly browserAccess?: boolean;
701
+ readonly arm?: Arm;
702
},
703
) => {
704
const t3 = visibleT3Tools(options);
@@ -689,7 +724,7 @@ export const buildBase = (
724
const { model } = options;
725
const append = options.nativeSubagent
726
? ""
692
- : t3SystemAppend(model, options.deviceAccess, options.browserAccess);
727
+ : forArm(t3SystemAppend(model, options.deviceAccess, options.browserAccess), options.arm);
728
const modelLine = `You are powered by the model named ${MODELS[model]}. The exact model ID is ${model}.`;
729
const joined = systemTexts.join("\n");
730
// The capture's home directory names its user. Cases use `/home/user` paths instead.
@@ -759,6 +794,7 @@ const buildCodexBase = (
794
readonly deviceAccess?: boolean;
795
/** false: the thread has no agent browser access, so T3's text leaves the browser out. */
796
readonly browserAccess?: boolean;
797
+ readonly arm?: Arm;
798
},
799
): Base => {
800
const t3 = visibleT3Tools(options);
@@ -797,7 +833,9 @@ const buildCodexBase = (
833
{ browser: options.browserAccess ?? true, device: options.deviceAccess ?? true },
834
);
835
input.push(
800
- ...Object.entries(context).map(([key, entry]) => message(`<${key}>${entry.value}</${key}>`)),
836
+ ...Object.entries(context).map(([key, entry]) =>
837
+ message(`<${key}>${forArm(entry.value, options.arm)}</${key}>`),
838
+ ),
839
);
840
const nativeTools = input
841
.flatMap((item) =>
@@ -895,18 +933,23 @@ const branchPullRequestCreatedBy = (entry: ShellEntry, branch: string | undefine
933
const checkedOutBranch = (sample: Sample) => {
934
let branch = sample.testCase.setup.world?.branch ?? "";
935
for (const entry of sample.shell) {
898
- const command = isRecord(entry.input) ? valueText(entry.input.command) : "";
899
- for (const part of command.replace(/\s\d?>&\d/g, "").split(/[;&|\n]+/)) {
900
- // `git checkout <ref> -- <paths>` restores files and stays on the branch.
901
- if (/\s--(\s|$)/.test(part)) continue;
902
- const match = /\bgit (?:checkout|switch)\s+(?:-[bcBC]\s+)?([A-Za-z0-9][\w./-]*)\s*$/.exec(
903
- part.trim(),
904
- );
905
- if (match !== null) branch = match[1]!;
906
- }
936
+ for (const part of shellParts(entry)) branch = switchedTo(part) ?? branch;
937
}
938
return branch;
939
};
940
+/** A shell entry's commands, split at separators, with fd redirects dropped. */
941
+const shellParts = (entry: ShellEntry) =>
942
+ (isRecord(entry.input) ? valueText(entry.input.command) : "")
943
+ .replace(/\s\d?>&\d/g, "")
944
+ .split(/[;&|\n]+/);
945
+/** The branch one command checks out or switches to, if it does. */
946
+const switchedTo = (part: string) => {
947
+ // `git checkout <ref> -- <paths>` restores files and stays on the branch.
948
+ if (/\s--(\s|$)/.test(part)) return undefined;
949
+ return /\bgit (?:checkout|switch)\s+(?:-[bcBC]\s+)?([A-Za-z0-9][\w./-]*)\s*$/.exec(
950
+ part.trim(),
951
+ )?.[1];
952
+};
953
954
/**
955
* One sample's state. ToolSearch adds loaded schemas to `tools`, like Claude Code. `shell` is the
@@ -1264,6 +1307,10 @@ const worldPullRequests = (sample: Sample) => {
1307
return Array.isArray(prs) ? prs.filter(isRecord) : [];
1308
};
1309
1310
+/** A world PR's `reviewThreads`: id, path, line, isResolved, comments of author and body. */
1311
+const worldReviewThreads = (pr: Readonly<Record<string, unknown>> | undefined) =>
1312
+ Array.isArray(pr?.reviewThreads) ? pr.reviewThreads.filter(isRecord) : [];
1313
+
1314
/** World fixtures spell states as GitHub does (`MERGED`) or as T3 does (`merged`). */
1315
const worldState = (pr: Readonly<Record<string, unknown>> | undefined) => {
1316
const state = valueText(pr?.state ?? "open").toLowerCase();
@@ -1468,8 +1515,8 @@ export const t3Default = (
1515
: number === own.number
1516
? (sample.testCase.setup.world?.commits?.[0]?.sha ?? null)
1517
: null,
1471
- mergeability: "mergeable",
1472
- behindBy: 0,
1518
+ mergeability: valueText(world?.mergeability ?? "mergeable"),
1519
+ behindBy: typeof world?.behindBy === "number" ? world.behindBy : 0,
1520
autoMergeEnabled: false,
1521
checks: {
1522
passed: checkRuns.filter((check) => check.status === "success").length,
@@ -1480,7 +1527,8 @@ export const t3Default = (
1527
.filter((check) => check.status === "failure" || check.status === "cancelled")
1528
.map((check) => check.name),
1529
},
1483
- unresolvedThreads: 0,
1530
+ unresolvedThreads: worldReviewThreads(world).filter((thread) => thread.isResolved !== true)
1531
+ .length,
1532
...(include.includes("body") ? { body: valueText(world?.body ?? "") } : {}),
1533
...(include.includes("checks") ? { checkRuns } : {}),
1534
...(include.includes("diff")
@@ -1489,7 +1537,29 @@ export const t3Default = (
1537
...(include.includes("threads") || include.includes("comments")
1538
? {
1539
conversation: {
1492
- ...(include.includes("threads") ? { reviewThreads: [] } : {}),
1540
+ ...(include.includes("threads")
1541
+ ? {
1542
+ reviewThreads: worldReviewThreads(world)
1543
+ .filter(
1544
+ (thread) => input.includeResolved === true || thread.isResolved !== true,
1545
+ )
1546
+ .map((thread) => ({
1547
+ id: valueText(thread.id),
1548
+ path: valueText(thread.path),
1549
+ line: typeof thread.line === "number" ? thread.line : null,
1550
+ isResolved: thread.isResolved === true,
1551
+ isOutdated: false,
1552
+ comments: (Array.isArray(thread.comments) ? thread.comments : [])
1553
+ .filter(isRecord)
1554
+ .map((comment) => ({
1555
+ author: valueText(comment.author),
1556
+ body: valueText(comment.body),
1557
+ createdAt: "2026-10-27T10:00:00Z",
1558
+ })),
1559
+ moreComments: false,
1560
+ })),
1561
+ }
1562
+ : {}),
1563
...(include.includes("comments")
1564
? {
1565
comments: comments.map((comment, index) => ({
@@ -1874,7 +1944,7 @@ export const shellRequest = (sample: Sample, call: Omit<ShellEntry, "output">, m
1944
const command = call.tool === "Bash" && isRecord(call.input) ? call.input.command : undefined;
1945
const mocks = shellMocksFor(sample, typeof command === "string" ? command : undefined);
1946
const machine = [
1877
- `Working directory: ${cwd} on ${platform}, a git checkout of github.com/${repo}.${date ? ` Today is ${date}.` : ""}`,
1947
+ `Working directory: ${cwd} on ${platform}, a git checkout of ${world?.remote ?? `github.com/${repo}`}.${date ? ` Today is ${date}.` : ""}`,
1948
gitStatus(testCase),
1949
`World: ${valueText(world ?? {})}`,
1950
...(mocks.length > 0
@@ -2052,6 +2122,64 @@ const recordForgeChange = (
2122
}
2123
};
2124
2125
+/**
2126
+ * A successful `git push` in this sample published `branch`: a bare push or a push to a remote
2127
+ * alone pushes the current branch; otherwise a refspec has to name it or HEAD. Dry runs don't count.
2128
+ */
2129
+const pushed = (sample: Sample, branch: string) => {
2130
+ // HEAD and a bare push publish whatever is checked out when that push runs.
2131
+ let current = sample.testCase.setup.world?.branch ?? "";
2132
+ for (const entry of sample.shell) {
2133
+ const succeeded = entry.tool === "Bash" && !/^Exit code [1-9]/.test(entry.output);
2134
+ for (const part of shellParts(entry)) {
2135
+ current = switchedTo(part) ?? current;
2136
+ const words = /\bgit\s+push\b(.*)/.exec(part)?.[1]?.trim().split(/\s+/).filter(Boolean);
2137
+ if (!succeeded || words === undefined) continue;
2138
+ if (["--dry-run", "-n", "--help", "-h"].some((flag) => words.includes(flag))) continue;
2139
+ const refs = words.filter((word) => !word.startsWith("-")).slice(1);
2140
+ const sources = refs.map((ref) => ref.split(":")[0]!.replace(/^\+/, ""));
2141
+ if (
2142
+ refs.length === 0
2143
+ ? current === branch
2144
+ : sources.some((source) => source === branch || (source === "HEAD" && current === branch))
2145
+ )
2146
+ return true;
2147
+ }
2148
+ }
2149
+ return false;
2150
+};
2151
+
2152
+/**
2153
+ * Refusals the real tools give for this world, so the tools arm meets the same walls as the CLI.
2154
+ * Messages are PullRequestService's and GitManager's.
2155
+ */
2156
+const productRefusal = (sample: Sample, name: string, input: Readonly<Record<string, unknown>>) => {
2157
+ const world = sample.testCase.setup.world;
2158
+ if (
2159
+ name === "pull_request_create" &&
2160
+ world?.unpushed === true &&
2161
+ !pushed(sample, checkedOutBranch(sample))
2162
+ ) {
2163
+ return "Current branch has not been pushed. Push before creating a PR.";
2164
+ }
2165
+ if (name === "pull_request_merge" && typeof input.expectedHeadSha === "string") {
2166
+ const own = branchPullRequest(sample);
2167
+ const number = input.number ?? own.number;
2168
+ const pr = worldPullRequests(sample).find((candidate) => Number(candidate.number) === number);
2169
+ // The same head pull_request_read reports: the world's, else the checkout's for its own PR.
2170
+ const head =
2171
+ typeof pr?.headRefOid === "string"
2172
+ ? pr.headRefOid
2173
+ : number === own.number
2174
+ ? world?.commits?.[0]?.sha
2175
+ : undefined;
2176
+ if (head !== undefined && head !== input.expectedHeadSha.trim()) {
2177
+ return `The head is now ${head}, not ${input.expectedHeadSha}. Review the new commits first.`;
2178
+ }
2179
+ }
2180
+ return undefined;
2181
+};
2182
+
2183
export const answerToolCall = Effect.fn("evaluateMcpToolUse.answerToolCall")(function* (
2184
sample: Sample,
2185
config: Config,
@@ -2123,6 +2251,12 @@ export const answerToolCall = Effect.fn("evaluateMcpToolUse.answerToolCall")(fun
2251
const result = yield* sample.backgroundWait(input);
2252
return result.isError ? failed(result.text) : toolResult(call.id, result.text);
2253
}
2254
+ // `{"$error": "..."}` fails the call with that text, as T3 reports a forge error.
2255
+ if (isRecord(mock) && typeof mock.$error === "string") return failed(mock.$error);
2256
+ if (isT3Name(call.name)) {
2257
+ const refused = productRefusal(sample, name, input);
2258
+ if (refused !== undefined) return failed(refused);
2259
+ }
2260
// A pasted secret is saved without a card, so a mocked card outcome does not apply.
2261
if (mock !== undefined && mock !== "default" && !pastedSecret(name, input)) {
2262
if (name === "request_secret") savedSecretFile(sample, input, mock);
@@ -2192,6 +2326,9 @@ const matches = (expected: unknown, actual: unknown): boolean => {
2326
if (typeof expected === "string" && expected.startsWith("~")) {
2327
return valueText(actual).toLowerCase().includes(expected.slice(1).toLowerCase());
2328
}
2329
+ if (typeof expected === "string" && expected.startsWith("re:")) {
2330
+ return new RegExp(expected.slice(3)).test(valueText(actual));
2331
+ }
2332
if (Array.isArray(expected)) {
2333
return (
2334
Array.isArray(actual) && expected.every((item) => actual.some((at) => matches(item, at)))
@@ -2690,7 +2827,9 @@ const runSample = Effect.fn("evaluateMcpToolUse.runSample")(function* (
2827
// Codex's code-mode cells take more model calls, so a case's own budget grows for it too.
2828
const extra = isCodex(config.model) ? 4 : 0;
2829
const maxSteps = testCase.maxSteps === undefined ? config.maxSteps : testCase.maxSteps + extra;
2830
+ let modelCalls = 0;
2831
for (let step = 0; step < maxSteps; step++) {
2832
+ modelCalls++;
2833
const response = input
2834
? yield* callCodex(config, codexRequest(base, input))
2835
: yield* callModel(config.apiKey, requestBody(sample, config.effort, messages));
@@ -2801,6 +2940,7 @@ const runSample = Effect.fn("evaluateMcpToolUse.runSample")(function* (
2940
text: said.join("\n\n"),
2941
stopReason,
2942
finalText,
2943
+ modelCalls,
2944
cwd: base.environment.cwd,
2945
linked: [...sample.linked],
2946
...(sample.devices ? { launchers: [...sample.devices.launchers] } : {}),
@@ -2810,6 +2950,21 @@ const runSample = Effect.fn("evaluateMcpToolUse.runSample")(function* (
2950
};
2951
}, Effect.scoped);
2952
2953
+/**
2954
+ * Model requests a sample made. Rows from before `modelCalls` was recorded count up to the last
2955
+ * step with tool calls, plus the final answer's request when the turn ended.
2956
+ */
2957
+export const modelRequests = (row: {
2958
+ readonly modelCalls?: number | undefined;
2959
+ readonly calls?: ReadonlyArray<{ readonly step: number }> | undefined;
2960
+ readonly stopReason?: string | undefined;
2961
+}) => {
2962
+ if (row.modelCalls !== undefined) return row.modelCalls;
2963
+ const calls = row.calls ?? [];
2964
+ if (calls.length === 0) return 1;
2965
+ return Math.max(...calls.map((call) => call.step)) + 1 + (row.stopReason === "end_turn" ? 1 : 0);
2966
+};
2967
+
2968
type ResultRow = Pick<McpToolUseCase, "id" | "kind" | "pattern"> & {
2969
readonly role: string;
2970
readonly pass: boolean;
@@ -2925,6 +3080,7 @@ const Flags = Schema.Struct({
3080
...PROXY_SHELL_MODELS,
3081
]),
3082
"user-instructions": Schema.Literals(["keep", "drop"]),
3083
+ arm: Schema.Literals(ARMS),
3084
"dry-run": Schema.Boolean,
3085
"stop-on-regression": Schema.Boolean,
3086
"regression-budget": Schema.optional(PositiveIntOrZero),
@@ -2965,6 +3121,8 @@ const main = Effect.gen(function* () {
3121
"shell-model": { ...string, default: "claude-haiku-5-5" },
3122
// T3's text must work for every user, so the captured CLAUDE.md is dropped by default.
3123
"user-instructions": { ...string, default: "drop" },
3124
+ // `raw` runs without T3's PR, CI and issue tools, to compare against the CLIs.
3125
+ arm: { ...string, default: "tools" },
3126
"dry-run": { type: "boolean", default: false },
3127
"stop-on-regression": { type: "boolean", default: false },
3128
"regression-budget": string,
@@ -3004,7 +3162,22 @@ const main = Effect.gen(function* () {
3162
(setup.harness === undefined || setup.harness === harness),
3163
);
3164
if (selectedCases.length === 0) return yield* Effect.die(new Error("No case matches --case."));
3007
- const selected = yield* Effect.forEach(selectedCases, (testCase) =>
3165
+ // The raw arm has no PR or issue tools, so a case whose prior turns call one cannot be replayed.
3166
+ const replaysForgeTool = (testCase: McpToolUseCase) =>
3167
+ (testCase.setup.priorTurns ?? []).some((turn) =>
3168
+ (turn.toolCalls ?? []).some((call) => forgeToolNames.has(`${T3_PREFIX}${call.tool}`)),
3169
+ );
3170
+ const unreplayable =
3171
+ flags.arm === "raw" ? selectedCases.filter(replaysForgeTool).map(({ id }) => id) : [];
3172
+ if (unreplayable.length > 0) {
3173
+ yield* Effect.logWarning(
3174
+ `--arm raw skips ${unreplayable.length} cases whose prior turns call PR or issue tools: ${unreplayable.join(", ")}`,
3175
+ );
3176
+ }
3177
+ const armCases = selectedCases.filter(({ id }) => !unreplayable.includes(id));
3178
+ if (armCases.length === 0)
3179
+ return yield* Effect.die(new Error("No selected case can run with --arm raw."));
3180
+ const selected = yield* Effect.forEach(armCases, (testCase) =>
3181
Effect.gen(function* () {
3182
const priorTurns = yield* Effect.forEach(testCase.setup.priorTurns ?? [], (turn) =>
3183
Effect.gen(function* () {
@@ -3020,6 +3193,7 @@ const main = Effect.gen(function* () {
3193
);
3194
return {
3195
...testCase,
3196
+ expect: flags.arm === "raw" ? (testCase.expectRaw ?? testCase.expect) : testCase.expect,
3197
setup: {
3198
...testCase.setup,
3199
priorTurns,
@@ -3056,6 +3230,7 @@ const main = Effect.gen(function* () {
3230
forge,
3231
deviceAccess,
3232
browserAccess,
3233
+ arm: flags.arm,
3234
}),
3235
),
3236
)
@@ -3070,6 +3245,7 @@ const main = Effect.gen(function* () {
3245
forge,
3246
deviceAccess,
3247
browserAccess,
3248
+ arm: flags.arm,
3249
});
3250
return role === "native-subagent"
3251
? Effect.promise(() =>
@@ -3157,7 +3333,14 @@ const main = Effect.gen(function* () {
3333
const inFlight = yield* Semaphore.make(flags.concurrency);
3334
const runOne = (testCase: McpToolUseCase) => {
3335
const { id, kind, pattern } = testCase;
3160
- const identity = { id, kind, pattern, role: testCase.setup.role };
3336
+ const identity = {
3337
+ id,
3338
+ kind,
3339
+ pattern,
3340
+ role: testCase.setup.role,
3341
+ arm: flags.arm,
3342
+ shellModel: config.shellModel,
3343
+ };
3344
const usage = { usage: emptyUsage(), shellUsage: emptyUsage() };
3345
const privateValues = [config.apiKey, ...Object.values(config.proxy?.headers ?? {})]
3346
.flatMap((value) => [value, value.replace(/^Bearer /i, "")])
@@ -3232,7 +3415,7 @@ const main = Effect.gen(function* () {
3415
{ concurrency: flags.concurrency, discard: true },
3416
).pipe(Effect.ignore);
3417
const samples = flags.samples === undefined ? "2-3 (adaptive)" : flags.samples;
3235
- const header = `Commit \`${commit}\`, ${flags.model} at ${effort} effort, user instructions ${flags["user-instructions"]}, ${samples} samples per case, template \`${path.basename(templatePath)}\``;
3418
+ const header = `Commit \`${commit}\`, ${flags.model} at ${effort} effort, user instructions ${flags["user-instructions"]}, arm ${flags.arm}, ${samples} samples per case, template \`${path.basename(templatePath)}\``;
3419
const finished = new Set(rows.map((row) => row.id)).size;
3420
const stopped =
3421
regressions.length === 0
evals/package.json
+1
@@ -6,6 +6,7 @@
6
"mcp-tool-use": "node --env-file-if-exists=../.env mcp-tool-use/run.ts",
7
"mcp-tool-use:trim-capture": "node mcp-tool-use/trim-capture.ts",
8
"mcp-tool-use:regrade": "node mcp-tool-use/regrade.ts",
9
+ "mcp-tool-use:compare": "node mcp-tool-use/compare-arms.ts",
10
"thread-titles": "node --env-file-if-exists=../.env thread-titles/run.ts",
11
"pr-content": "node --env-file-if-exists=../.env pr-content/run.ts",
12
"voice-cleanup": "node --env-file-if-exists=../.env voice-cleanup/run.ts",