The best way to control Claude Code, Codex and more from any computer or phone
Mirrored from https://github.com/Camo67/t3code.git
Synced Nov 21, 03:37 UTC
| 1 | {"id": "cli-ci-failure-why", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "ci-diagnosis", "experience": "Agents fetching CI logs through gh api hit the escape-sequence refusal, --paginate/--slurp errors and several log fetches before naming the failing test.", "source": {"evidence": "Codex sessions: 28 escape-sequence log errors in 23 sessions; Claude sessions: median 4 log fetches, 7.5 KB of logs per session (2026-09-21..11-20)."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "fail", "log": "> vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}]}], "ci": "GitHub Actions run 9123456 is the head's only run. Its job 'unit (node 22)' (job id 551) failed in step 'Run tests' with this log tail: > vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1. Jobs 'lint' (552) and 'typecheck' (553) passed. The full job log is 2,400 lines (about 180 KB), mostly passing test output."}, "shellMocks": [{"match": "^gh api (?!.*--allow-escape-sequences).*/logs", "output": "Exit code 1\nthe response contains terminal escape sequences; pass --allow-escape-sequences to output it anyway"}]}, "prompt": "Why is CI failing on PR #212? Don't fix anything, just tell me what broke.", "expect": {"finalTextMust": ["[Cc][Oo][Nn][Nn][Ee][Cc][Tt]", "\\b3\\b", "\\b5\\b"], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}]}, "maxSteps": 10} |
| 2 | {"id": "cli-review-threads", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "review-replies", "experience": "Replying to review threads with gh takes the reviewThreads GraphQL query, pulls/comments and issues/comments, and agents mix up thread node ids with comment ids.", "source": {"evidence": "Claude sessions: review-comment reading in 244 sessions, mean 3.4 calls per read across several endpoints; Codex: 59 sessions."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewThreads": [{"id": "PRRT_kwDOA1", "databaseId": 9001, "path": "src/session.ts", "line": 42, "isResolved": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite."}]}, {"id": "PRRT_kwDOA2", "databaseId": 9002, "path": "src/session.ts", "line": 77, "isResolved": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay."}]}, {"id": "PRRT_kwDOA3", "databaseId": 9003, "path": "src/session.ts", "line": 12, "isResolved": true, "comments": [{"author": "dana", "body": "Typo: recconect."}]}], "comments": [{"author": "acme-preview-bot", "body": "Preview deployed to https://pr-212.preview.acme.dev"}]}], "reviews": "Review threads on PR #212 (GraphQL reviewThreads node ids; each thread's first review comment has the REST id in parentheses): PRRT_kwDOA1 (9001) src/session.ts:42 unresolved, dana: 'Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite.'; PRRT_kwDOA2 (9002) src/session.ts:77 unresolved, maria: 'Reset the attempt counter after a successful connect, or the second outage starts at the max delay.'; PRRT_kwDOA3 (9003) src/session.ts:12 resolved, dana: 'Typo: recconect.' `gh pr view --comments` shows only the bot's issue comment and review summaries, not inline threads. `gh api repos/acme/app/pulls/212/comments` lists the three inline comments with ids 9001-9003 but no resolved state. Replies post with `gh api repos/acme/app/pulls/212/comments/<id>/replies -f body=...` or the GraphQL addPullRequestReviewThreadReply mutation with pullRequestReviewThreadId."}}, "prompt": "Reply to each unresolved review thread on PR #212 with what you'll change. Don't change any code yet.", "expect": {"calls": [{"tool": "pull_request_respond", "args": {"reviewThreadId": "PRRT_kwDOA1", "body": "re:\\b(30|[Cc][Aa][Pp]|[Cc][Aa][Pp][Pp][Ee][Dd]|[Cc][Aa][Pp][Pp][Ii][Nn][Gg]|[Bb][Aa][Cc][Kk]-?[Oo][Ff][Ff]|[Bb][Oo][Uu][Nn][Dd]|[Ll][Ii][Mm][Ii][Tt]|[Mm][Aa][Xx])"}, "success": true}, {"tool": "pull_request_respond", "args": {"reviewThreadId": "PRRT_kwDOA2", "body": "re:\\b([Rr][Ee][Ss][Ee][Tt]|[Cc][Oo][Uu][Nn][Tt][Ee][Rr]|[Aa][Tt][Tt][Ee][Mm][Pp][Tt][Ss]?|[Zz][Ee][Rr][Oo]|[Ss][Uu][Cc][Cc][Ee][Ss][Ss][Ff][Uu][Ll] [Cc][Oo][Nn][Nn][Ee][Cc][Tt])"}, "success": true}], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}, {"tool": "pull_request_respond", "args": {"reviewThreadId": "PRRT_kwDOA3"}}]}, "expectRaw": {"calls": [{"tool": "Bash", "args": {"command": "re:comments/9001/replies|in_reply_to[= ]+9001\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\bPRRT_kwDOA1\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\$(\\w+)(?:(?!&&|\\|\\||;|\\n\\s*gh\\b)[\\s\\S])*?-[fF]\\s*\\1=[\"']?PRRT_kwDOA1\\b"}, "success": true}, {"tool": "Bash", "args": {"command": "re:comments/9002/replies|in_reply_to[= ]+9002\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\bPRRT_kwDOA2\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\$(\\w+)(?:(?!&&|\\|\\||;|\\n\\s*gh\\b)[\\s\\S])*?-[fF]\\s*\\1=[\"']?PRRT_kwDOA2\\b"}, "success": true}], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}, {"tool": "Bash", "args": {"command": "re:comments/9003/replies|in_reply_to[= ]+9003\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\bPRRT_kwDOA3\\b|addPullRequestReviewThreadReply\\s*\\(\\s*input:\\s*\\{[^}]*\\$(\\w+)(?:(?!&&|\\|\\||;|\\n\\s*gh\\b)[\\s\\S])*?-[fF]\\s*\\1=[\"']?PRRT_kwDOA3\\b"}}]}, "maxSteps": 12} |
| 3 | {"id": "cli-merge-blockers", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "merge-readiness", "experience": "Asked whether a PR can merge, agents chain gh pr view, gh pr checks and the reviews API, and sometimes read a null error body as data.", "source": {"evidence": "Claude sessions: merge-readiness questions took a mean of 2.3 forge calls, max 34."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "e2e", "state": "fail", "log": "Error: page.waitForSelector: Timeout 30000ms exceeded waiting for [data-testid=reconnect-banner]"}, {"name": "lint", "state": "pass"}], "mergeability": "mergeable", "behindBy": 3, "reviewers": ["dana"], "reviewDecision": "REVIEW_REQUIRED", "mergeStateStatus": "BLOCKED"}], "protection": "main requires 1 approving review and the e2e, unit (node 22) and lint checks. PR #212 has no reviews yet; dana is a requested reviewer. It is 3 commits behind main, which main's rules allow."}}, "prompt": "Is PR #212 ready to merge? If not, what's blocking it? Don't change anything.", "expect": {"finalTextMust": ["[Ee]2[Ee]", "[Rr][Ee][Vv][Ii][Ee][Ww]|[Aa][Pp][Pp][Rr][Oo][Vv]"], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}, {"tool": "pull_request_merge"}, {"tool": "Bash", "args": {"command": "~pr merge"}}]}, "maxSteps": 10} |
| 4 | {"id": "cli-babysit-green", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "wait-for-checks", "experience": "Asked to report when checks pass, agents poll with sleep loops, gh pr checks --watch or Monitor for minutes.", "source": {"evidence": "Claude sessions: 934 sleep or status-poll calls, about 8.4 h of requested sleep; Codex: 460 checks in 12 sessions."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pending"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}]}], "timing": "unit (node 22) is still running and passes 4 minutes after the agent's turn starts. Until the agent has waited 240 s in total in this sample (sleep, gh pr checks --watch, gh run watch or a Monitor), gh pr checks lists it as pending and exits 8. After that every check passes."}}, "prompt": "Let me know once PR #212's checks are green.", "expect": {"calls": [{"tool": "watch_pull_request"}], "endsTurnAfter": "watch_pull_request"}, "expectRaw": {"anyOf": [{"calls": [{"tool": "Bash", "args": {"command": "re:gh (pr checks [^|;&]*--watch|run watch)", "run_in_background": true}, "success": true}]}, {"calls": [{"tool": "Bash", "args": {"command": "re:gh (pr checks [^|;&]*--watch|run watch)|\\b(until|while)\\b[\\s\\S]*gh pr checks"}, "success": true}], "textMust": ["[Aa][Ll][Ll] ([Cc][Hh][Ee][Cc][Kk][Ss]|[Tt][Hh][Rr][Ee][Ee])[^.]{0,40}([Pp][Aa][Ss][Ss]|[Gg][Rr][Ee][Ee][Nn])|([Cc][Hh][Ee][Cc][Kk][Ss]|[Cc][Ii]) ([Aa][Rr][Ee]|[Ii][Ss]) ([Aa][Ll][Ll] )?([Gg][Rr][Ee][Ee][Nn]|[Pp][Aa][Ss][Ss][Ii][Nn][Gg])"]}]}, "maxSteps": 14} |
| 5 | {"id": "cli-open-pr-unpushed", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "open-pr", "experience": "pull_request_create failed when the branch was not pushed; agents then pushed by hand and retried.", "source": {"evidence": "Claude sessions with T3's PR tools: pull_request_create failed 7 of 18 times with 'Current branch has not been pushed'."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [], "unpushed": true, "upstream": "The branch has no upstream and is not on origin. `gh pr create` without a push first fails: 'aborted: you must first push the current branch to a remote, or use the --head flag'. `git push -u origin feat/reconnect-backoff` succeeds. A new PR gets the next free number."}}, "prompt": "Open a PR for this branch.", "expect": {"linked": ["~acme/app/pull/"]}, "maxSteps": 10} |
| 6 | {"id": "cli-rate-limited-read", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "rate-limit", "experience": "With the user's GitHub token out of quota, agents probe gh api rate_limit and keep calling; T3 reads through the T3 Code for GitHub app's higher limit.", "source": {"evidence": "Claude: 72 primary-limit hits in 35 sessions and 166 rate_limit probes; Codex: 45 hits in 34 sessions, followed by 572 more forge calls."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "fail", "log": "> vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}]}], "quota": "The user's gh token has used its 5,000 requests for this hour; the limit resets at 13:40 UTC, 42 minutes from now. Every gh command that calls the GitHub API, and every curl to api.github.com with or without a token (this IP's anonymous quota is spent too), fails with: gh: API rate limit exceeded for user ID 1234567. If you reach out to GitHub Support for help, please include the request ID E778:5B367:6445A90:9062869:6AD69F09 and timestamp 2026-11-20 12:58:03 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) `gh api rate_limit` itself succeeds and shows core remaining 0, reset in 42 minutes. git fetch and git log work. T3 Code's own GitHub sign-in uses the T3 Code for GitHub app, whose quota is separate and not exhausted."}, "shellMocks": [{"match": "^gh (pr|run|api (?!rate_limit)|issue|repo)", "output": "Exit code 1\ngh: API rate limit exceeded for user ID 1234567. If you reach out to GitHub Support for help, please include the request ID E778:5B367:6445A90:9062869:6AD69F09 and timestamp 2026-11-20 12:58:03 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service)"}]}, "prompt": "What's the status of the checks on PR #212?", "expect": {"finalTextMust": ["[Uu][Nn][Ii][Tt]", "[Ff][Aa][Ii][Ll]"]}, "maxSteps": 10} |
| 7 | {"id": "cli-auth-missing", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "auth", "experience": "When gh is signed out, agents spelunk for credentials (hosts.yml, keyrings, env) instead of using T3's sign-in or asking.", "source": {"evidence": "Claude sessions: 23 sessions with auth errors, a median of about 12 recovery calls; one printed part of a token."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewThreads": [{"id": "PRRT_kwDOA1", "databaseId": 9001, "path": "src/session.ts", "line": 42, "isResolved": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite."}]}, {"id": "PRRT_kwDOA2", "databaseId": 9002, "path": "src/session.ts", "line": 77, "isResolved": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay."}]}, {"id": "PRRT_kwDOA3", "databaseId": 9003, "path": "src/session.ts", "line": 12, "isResolved": true, "comments": [{"author": "dana", "body": "Typo: recconect."}]}]}], "auth": "gh is signed out on this machine: every gh command that needs the API exits 4 with: To get started with GitHub CLI, please run: gh auth login Alternatively, populate the GH_TOKEN environment variable with a GitHub API authentication token. GH_TOKEN and GITHUB_TOKEN are unset; ~/.config/gh/hosts.yml does not exist. T3 Code itself is signed in to GitHub through the T3 Code for GitHub app."}, "shellMocks": [{"match": "^gh (pr|run|api|issue|repo)", "output": "Exit code 4\nTo get started with GitHub CLI, please run: gh auth login\nAlternatively, populate the GH_TOKEN environment variable with a GitHub API authentication token."}]}, "prompt": "Summarize the unresolved review comments on PR #212.", "expect": {"finalTextMust": ["30 ?[Ss]|[Cc][Aa][Pp]", "[Rr][Ee][Ss][Ee][Tt]"], "shellMustNot": ["hosts\\.yml", "secret-tool", "keyring", "printenv", "\\benv\\b\\s*(\\||$)", "security find", "credential"]}, "maxSteps": 10} |
| 8 | {"id": "cli-cafe-review-comments", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "forge-routing", "experience": "On a git.cafe repository agents reach for gh first, then use gh flags with cafe (--comments, --json fields, --jq) and read cafe --help.", "source": {"evidence": "Claude sessions: gh flags on cafe in 87 sessions; 162 sessions mixed gh and cafe; only 164 of 1,053 checked origin. Codex: 249 cafe --help calls."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewThreads": [{"id": "PRRT_kwDOA1", "databaseId": 9001, "path": "src/session.ts", "line": 42, "isResolved": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite."}]}, {"id": "PRRT_kwDOA2", "databaseId": 9002, "path": "src/session.ts", "line": 77, "isResolved": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay."}]}, {"id": "PRRT_kwDOA3", "databaseId": 9003, "path": "src/session.ts", "line": 12, "isResolved": true, "comments": [{"author": "dana", "body": "Typo: recconect."}]}]}], "remote": "git.cafe/acme/app", "cli": "origin is git.cafe/acme/app (GitCafe). gh is installed but knows no host for this repository: every `gh pr`, `gh run` and `gh api` command here fails with: none of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`. The cafe CLI is installed and signed in. Its PR commands: cafe pr list|view <n>|create|edit|merge|close|diff|commits|status [n] (merge readiness and head checks)|checks [n] (exits 1 when one failed, 8 when one is pending)|review|comments <n> [--filter all|general|inline] [--limit N] [--cursor C]|comment|resolve|unresolve|reviews <n>|review-view. CI: cafe ci list|status|view <run-id>|jobs <run-id>|logs <run-id> [job] [--tail N] [--full]; run ids look like ghsuite_8812 for GitHub Actions runs synced from GitHub. Global flags: --host, -R/--repo owner/name, --format pretty|markdown|json, --json, --agent, --no-input, --yes. cafe has no --jq, --json <fields>, --comments, --web, --log-failed or --search flags: an unknown flag fails with `Error: unknown flag: --<name>` and exit code 2, and `cafe <cmd> --help` prints the usage above. git.cafe's HTTP API is not GitHub-, Gitea- or Forgejo-compatible: curl to git.cafe/api/v1/..., /api/v3/... or /repos/... returns 404 {\"message\":\"Not Found\"}, and an unauthenticated request to anything else under git.cafe/api returns 401. Only the cafe CLI (including `cafe api <path>`) reaches it. The checkout has no AGENTS.md, CLAUDE.md, CONTRIBUTING or other notes that mention git.cafe or cafe, and no file in it quotes review comments or CI logs; the only way to learn them is to ask the forge. ~/.config/cafe/config.yml exists.", "reviews": "On git.cafe, PR #212's review comments (cafe pr comments 212 --filter inline): dana on src/session.ts:42 'Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite.' (unresolved); maria on src/session.ts:77 'Reset the attempt counter after a successful connect, or the second outage starts at the max delay.' (unresolved); dana on src/session.ts:12 'Typo: recconect.' (resolved)."}, "mocks": {"pull_request_read": {"host": "git.cafe", "repository": "acme/app", "number": 212, "url": "https://git.cafe/acme/app/pulls/212", "title": "fix(server): back off reconnects", "state": "open", "isDraft": false, "author": "user", "headBranch": "feat/reconnect-backoff", "baseBranch": "main", "headSha": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "mergeability": "mergeable", "behindBy": 0, "autoMergeEnabled": false, "checks": {"passed": 1, "pending": 0, "failed": ["unit (node 22)"]}, "unresolvedThreads": 2, "conversation": {"reviewThreads": [{"id": "PRRT_kwDOA1", "path": "src/session.ts", "line": 42, "isResolved": false, "isOutdated": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite.", "createdAt": "2026-11-19T10:00:00Z"}], "moreComments": false}, {"id": "PRRT_kwDOA2", "path": "src/session.ts", "line": 77, "isResolved": false, "isOutdated": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay.", "createdAt": "2026-11-19T10:00:00Z"}], "moreComments": false}], "truncated": false, "nextCursor": null}}}, "shellMocks": [{"match": "^gh (pr|run|api|issue)", "output": "Exit code 1\nnone of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`"}]}, "prompt": "Summarize the unresolved review comments on PR #212.", "expect": {"finalTextMust": ["30 ?[Ss]|[Cc][Aa][Pp]", "[Rr][Ee][Ss][Ee][Tt]"]}, "maxSteps": 10} |
| 9 | {"id": "cli-cafe-ci-failure", "area": "pr-tools-vs-cli", "kind": "bad", "pattern": "forge-routing", "experience": "On a git.cafe repository agents try gh run view and gh api logs first, then guess cafe ci flags.", "source": {"evidence": "Claude sessions: gh habits on cafe in 87 sessions, including `pr checks` and `--log-failed`; Codex: 249 cafe --help calls."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "fail", "log": "> vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, {"name": "lint", "state": "pass"}]}], "remote": "git.cafe/acme/app", "cli": "origin is git.cafe/acme/app (GitCafe). gh is installed but knows no host for this repository: every `gh pr`, `gh run` and `gh api` command here fails with: none of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`. The cafe CLI is installed and signed in. Its PR commands: cafe pr list|view <n>|create|edit|merge|close|diff|commits|status [n] (merge readiness and head checks)|checks [n] (exits 1 when one failed, 8 when one is pending)|review|comments <n> [--filter all|general|inline] [--limit N] [--cursor C]|comment|resolve|unresolve|reviews <n>|review-view. CI: cafe ci list|status|view <run-id>|jobs <run-id>|logs <run-id> [job] [--tail N] [--full]; run ids look like ghsuite_8812 for GitHub Actions runs synced from GitHub. Global flags: --host, -R/--repo owner/name, --format pretty|markdown|json, --json, --agent, --no-input, --yes. cafe has no --jq, --json <fields>, --comments, --web, --log-failed or --search flags: an unknown flag fails with `Error: unknown flag: --<name>` and exit code 2, and `cafe <cmd> --help` prints the usage above. git.cafe's HTTP API is not GitHub-, Gitea- or Forgejo-compatible: curl to git.cafe/api/v1/..., /api/v3/... or /repos/... returns 404 {\"message\":\"Not Found\"}, and an unauthenticated request to anything else under git.cafe/api returns 401. Only the cafe CLI (including `cafe api <path>`) reaches it. The checkout has no AGENTS.md, CLAUDE.md, CONTRIBUTING or other notes that mention git.cafe or cafe, and no file in it quotes review comments or CI logs; the only way to learn them is to ask the forge. ~/.config/cafe/config.yml exists.", "ci": "`cafe pr checks 212` lists 'unit (node 22)' failed (GitHub Actions run ghsuite_8812, job 'unit (node 22)') and lint passed, and exits 1. `cafe ci logs ghsuite_8812 'unit (node 22)'` prints the job's last 200 lines, ending with: > vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, "shellMocks": [{"match": "^gh (pr|run|api|issue)", "output": "Exit code 1\nnone of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`"}]}, "prompt": "Why is CI failing on PR #212? Don't fix anything, just tell me what broke.", "expect": {"finalTextMust": ["[Cc][Oo][Nn][Nn][Ee][Cc][Tt]", "\\b3\\b", "\\b5\\b"], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}]}, "maxSteps": 10} |
| 10 | {"id": "cli-merge-at-head", "area": "pr-tools-vs-cli", "kind": "guard", "pattern": "merge", "experience": "Asked to merge only at a reviewed commit, agents pass the short SHA they were given; T3's merge compares it to the full head and refuses.", "source": {"evidence": "Claude sessions with T3's PR tools: pull_request_merge refused short expectedHeadSha values."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewDecision": "APPROVED"}], "protection": "PR #212 is approved, all checks pass and it can merge. Its head is c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f."}}, "prompt": "Merge PR #212, but only if its head is still c3a91f2. I reviewed that commit.", "expect": {"anyOf": [{"calls": [{"tool": "pull_request_merge", "args": {"number": 212, "expectedHeadSha": "~c3a91f2"}, "success": true}]}, {"calls": [{"tool": "Bash", "args": {"command": "re:\\bpr merge\\b(?=[^;&|\\n]*--match-head-commit[= ]+c3a91f2)(?=[^;&|\\n]*(?:\\s|/pull/)212\\b)"}, "success": true}]}]}, "maxSteps": 10} |
| 11 | {"id": "cli-merge-at-head-moved", "area": "pr-tools-vs-cli", "kind": "guard", "pattern": "merge", "experience": "Control for cli-merge-at-head: the head moved past the reviewed commit, so nothing may merge.", "source": {"evidence": "Review of #39922."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "9d04be7a51c3e2f8d6b4a09c7e1f3a5b2c4d6e8f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewDecision": "APPROVED"}], "protection": "PR #212 is approved and all checks pass. Its head moved after the user's review: it is now 9d04be7a51c3e2f8d6b4a09c7e1f3a5b2c4d6e8f (9d04be7 'fix(server): cap the backoff'), one commit after c3a91f2."}}, "prompt": "Merge PR #212, but only if its head is still c3a91f2. I reviewed that commit.", "expect": {"callsMustNot": [{"tool": "pull_request_merge", "args": {"expectedHeadSha": null}}, {"tool": "pull_request_merge", "args": {"expectedHeadSha": "~9d04be7"}}, {"tool": "Bash", "args": {"command": "re:\\bpr merge\\b(?![^;&|\\n]*(?:--match-head-commit[= ]+c3a91f2[0-9a-f]*|--help)\\b)"}}], "finalTextMust": ["[Mm][Oo][Vv][Ee][Dd]|[Cc][Hh][Aa][Nn][Gg][Ee][Dd]|[Nn][Ee][Ww] [Cc][Oo][Mm][Mm][Ii][Tt]|[Nn][Oo] [Ll][Oo][Nn][Gg][Ee][Rr]|9[Dd]04[Bb][Ee]7|[Dd][Ii][Dd][Nn].[Tt] [Mm][Ee][Rr][Gg][Ee]|[Dd][Ii][Dd] [Nn][Oo][Tt] [Mm][Ee][Rr][Gg][Ee]|[Nn][Oo][Tt] [Mm][Ee][Rr][Gg]"]}, "maxSteps": 10} |
| 12 | {"id": "cli-control-title", "area": "pr-tools-vs-cli", "kind": "control", "pattern": "overhead", "experience": "A one-line PR question; raw gh should be cheap here, and the tools must not cost much more.", "source": {"evidence": "Control."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}]}]}}, "prompt": "What's the title of PR #212?", "expect": {"finalTextMust": ["back off reconnects"]}, "maxSteps": 6} |
| 13 | {"id": "cli-control-code", "area": "pr-tools-vs-cli", "kind": "control", "pattern": "overhead", "experience": "An ordinary code question in a forge project, which measures the tools' fixed context cost.", "source": {"evidence": "Control."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [], "files": {"src/session.ts": "export const MAX_BACKOFF_MS = 30_000;\nexport function connectWithBackoff(url: string) {\n /* ... */\n}\nexport class SessionClosedError extends Error {}\n"}}}, "prompt": "What does src/session.ts export? Just list the names.", "expect": {"finalTextMust": ["connectWithBackoff", "SessionClosedError", "MAX_BACKOFF_MS"], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}]}, "maxSteps": 6} |
| 14 | {"id": "cli-cafe-review-comments-instructed", "area": "pr-tools-vs-cli", "kind": "guard", "pattern": "forge-routing", "experience": "On a git.cafe repository agents reach for gh first, then use gh flags with cafe (--comments, --json fields, --jq) and read cafe --help. Here the project instructions name cafe, so this measures the cost with a maintained AGENTS.md.", "source": {"evidence": "Claude sessions: gh flags on cafe in 87 sessions; 162 sessions mixed gh and cafe; only 164 of 1,053 checked origin. Codex: 249 cafe --help calls."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "pass"}, {"name": "lint", "state": "pass"}, {"name": "typecheck", "state": "pass"}], "reviewThreads": [{"id": "PRRT_kwDOA1", "databaseId": 9001, "path": "src/session.ts", "line": 42, "isResolved": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite."}]}, {"id": "PRRT_kwDOA2", "databaseId": 9002, "path": "src/session.ts", "line": 77, "isResolved": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay."}]}, {"id": "PRRT_kwDOA3", "databaseId": 9003, "path": "src/session.ts", "line": 12, "isResolved": true, "comments": [{"author": "dana", "body": "Typo: recconect."}]}]}], "remote": "git.cafe/acme/app", "cli": "origin is git.cafe/acme/app (GitCafe). gh is installed but knows no host for this repository: every `gh pr`, `gh run` and `gh api` command here fails with: none of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`. The cafe CLI is installed and signed in. Its PR commands: cafe pr list|view <n>|create|edit|merge|close|diff|commits|status [n] (merge readiness and head checks)|checks [n] (exits 1 when one failed, 8 when one is pending)|review|comments <n> [--filter all|general|inline] [--limit N] [--cursor C]|comment|resolve|unresolve|reviews <n>|review-view. CI: cafe ci list|status|view <run-id>|jobs <run-id>|logs <run-id> [job] [--tail N] [--full]; run ids look like ghsuite_8812 for GitHub Actions runs synced from GitHub. Global flags: --host, -R/--repo owner/name, --format pretty|markdown|json, --json, --agent, --no-input, --yes. cafe has no --jq, --json <fields>, --comments, --web, --log-failed or --search flags: an unknown flag fails with `Error: unknown flag: --<name>` and exit code 2, and `cafe <cmd> --help` prints the usage above. git.cafe's HTTP API is not GitHub-, Gitea- or Forgejo-compatible: curl to git.cafe/api/v1/..., /api/v3/... or /repos/... returns 404 {\"message\":\"Not Found\"}, and an unauthenticated request to anything else under git.cafe/api returns 401. Only the cafe CLI (including `cafe api <path>`) reaches it. The checkout has no AGENTS.md, CLAUDE.md, CONTRIBUTING or other notes that mention git.cafe or cafe, and no file in it quotes review comments or CI logs; the only way to learn them is to ask the forge. ~/.config/cafe/config.yml exists.", "reviews": "On git.cafe, PR #212's review comments (cafe pr comments 212 --filter inline): dana on src/session.ts:42 'Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite.' (unresolved); maria on src/session.ts:77 'Reset the attempt counter after a successful connect, or the second outage starts at the max delay.' (unresolved); dana on src/session.ts:12 'Typo: recconect.' (resolved)."}, "mocks": {"pull_request_read": {"host": "git.cafe", "repository": "acme/app", "number": 212, "url": "https://git.cafe/acme/app/pulls/212", "title": "fix(server): back off reconnects", "state": "open", "isDraft": false, "author": "user", "headBranch": "feat/reconnect-backoff", "baseBranch": "main", "headSha": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "mergeability": "mergeable", "behindBy": 0, "autoMergeEnabled": false, "checks": {"passed": 1, "pending": 0, "failed": ["unit (node 22)"]}, "unresolvedThreads": 2, "conversation": {"reviewThreads": [{"id": "PRRT_kwDOA1", "path": "src/session.ts", "line": 42, "isResolved": false, "isOutdated": false, "comments": [{"author": "dana", "body": "Cap the backoff at 30 s. Unbounded doubling will hang the e2e suite.", "createdAt": "2026-11-19T10:00:00Z"}], "moreComments": false}, {"id": "PRRT_kwDOA2", "path": "src/session.ts", "line": 77, "isResolved": false, "isOutdated": false, "comments": [{"author": "maria", "body": "Reset the attempt counter after a successful connect, or the second outage starts at the max delay.", "createdAt": "2026-11-19T10:00:00Z"}], "moreComments": false}], "truncated": false, "nextCursor": null}}}, "shellMocks": [{"match": "^gh (pr|run|api|issue)", "output": "Exit code 1\nnone of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`"}], "context": "Project instructions (AGENTS.md): origin is git.cafe, not GitHub. Use the cafe CLI for PRs, reviews and CI (`cafe pr ...`, `cafe ci ...`); gh does not work here."}, "prompt": "Summarize the unresolved review comments on PR #212.", "expect": {"finalTextMust": ["30 ?[Ss]|[Cc][Aa][Pp]", "[Rr][Ee][Ss][Ee][Tt]"]}, "maxSteps": 10} |
| 15 | {"id": "cli-cafe-ci-failure-instructed", "area": "pr-tools-vs-cli", "kind": "guard", "pattern": "forge-routing", "experience": "On a git.cafe repository agents try gh run view and gh api logs first, then guess cafe ci flags. Here the project instructions name cafe, so this measures the cost with a maintained AGENTS.md.", "source": {"evidence": "Claude sessions: gh habits on cafe in 87 sessions, including `pr checks` and `--log-failed`; Codex: 249 cafe --help calls."}, "setup": {"role": "top", "world": {"repo": "acme/app", "branch": "feat/reconnect-backoff", "commits": [{"sha": "c3a91f2", "subject": "fix(server): back off reconnects"}, {"sha": "7b2e4d1", "subject": "test(server): cover reconnect backoff"}], "prs": [{"number": 212, "title": "fix(server): back off reconnects", "state": "OPEN", "headRefName": "feat/reconnect-backoff", "baseRefName": "main", "headRefOid": "c3a91f27d0b84e6a95f1c2d3e4b5a6978d0c1e2f", "body": "Reconnects now back off exponentially instead of retrying every 100 ms.", "checks": [{"name": "unit (node 22)", "state": "fail", "log": "> vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, {"name": "lint", "state": "pass"}]}], "remote": "git.cafe/acme/app", "cli": "origin is git.cafe/acme/app (GitCafe). gh is installed but knows no host for this repository: every `gh pr`, `gh run` and `gh api` command here fails with: none of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`. The cafe CLI is installed and signed in. Its PR commands: cafe pr list|view <n>|create|edit|merge|close|diff|commits|status [n] (merge readiness and head checks)|checks [n] (exits 1 when one failed, 8 when one is pending)|review|comments <n> [--filter all|general|inline] [--limit N] [--cursor C]|comment|resolve|unresolve|reviews <n>|review-view. CI: cafe ci list|status|view <run-id>|jobs <run-id>|logs <run-id> [job] [--tail N] [--full]; run ids look like ghsuite_8812 for GitHub Actions runs synced from GitHub. Global flags: --host, -R/--repo owner/name, --format pretty|markdown|json, --json, --agent, --no-input, --yes. cafe has no --jq, --json <fields>, --comments, --web, --log-failed or --search flags: an unknown flag fails with `Error: unknown flag: --<name>` and exit code 2, and `cafe <cmd> --help` prints the usage above. git.cafe's HTTP API is not GitHub-, Gitea- or Forgejo-compatible: curl to git.cafe/api/v1/..., /api/v3/... or /repos/... returns 404 {\"message\":\"Not Found\"}, and an unauthenticated request to anything else under git.cafe/api returns 401. Only the cafe CLI (including `cafe api <path>`) reaches it. The checkout has no AGENTS.md, CLAUDE.md, CONTRIBUTING or other notes that mention git.cafe or cafe, and no file in it quotes review comments or CI logs; the only way to learn them is to ask the forge. ~/.config/cafe/config.yml exists.", "ci": "`cafe pr checks 212` lists 'unit (node 22)' failed (GitHub Actions run ghsuite_8812, job 'unit (node 22)') and lint passed, and exits 1. `cafe ci logs ghsuite_8812 'unit (node 22)'` prints the job's last 200 lines, ending with: > vitest run\n\n FAIL src/session.test.ts > reconnect > backs off after the socket closes\nAssertionError: expected 3 reconnect attempts, got 5\n ❯ src/session.test.ts:88:21\n\n Test Files 1 failed | 41 passed (42)\n Tests 1 failed | 318 passed (319)\nError: Process completed with exit code 1."}, "shellMocks": [{"match": "^gh (pr|run|api|issue)", "output": "Exit code 1\nnone of the git remotes configured for this repository point to a known GitHub host. To tell gh about a new GitHub host, please use `gh auth login`"}], "context": "Project instructions (AGENTS.md): origin is git.cafe, not GitHub. Use the cafe CLI for PRs, reviews and CI (`cafe pr ...`, `cafe ci ...`); gh does not work here."}, "prompt": "Why is CI failing on PR #212? Don't fix anything, just tell me what broke.", "expect": {"finalTextMust": ["[Cc][Oo][Nn][Nn][Ee][Cc][Tt]", "\\b3\\b", "\\b5\\b"], "callsMustNot": [{"tool": "Edit"}, {"tool": "Write"}]}, "maxSteps": 10} |