@devries-cameron20 / t3code-1 / commits / d5f1d0a2f7
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

test(evals): device manager cases for mcp-tool-use (#25850)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Gabriel De Andrade committed Oct 28, 2026 at 13:28 UTC d5f1d0a2f714b9f860338e7729f6750a38f58031
9 files changed +6520 -66
evals/README.md
+5 -5
@@ -4,11 +4,11 @@ An eval runs a real model against T3 Code's real text or code and scores what th
4
5 Write or run an eval when a change affects what a model sees or does: agent instructions, MCP tool descriptions and schemas, tool results, generated titles, or the guidance an agent gets for a feature such as Device Manager.
6
7 -| Eval | Measures |
8 -| ------------------------------- | --------------------------------------------------------------------------- |
9 -| [mcp-tool-use](mcp-tool-use/) | Whether agents use T3's MCP tools correctly in fake threads. |
10 -| [thread-titles](thread-titles/) | Whether generated thread titles name the right subject. |
11 -| [test-signal](test-signal/) | Whether a cheap model running the test-signal check flags low-signal tests. |
7 +| Eval | Measures |
8 +| ------------------------------- | ---------------------------------------------------------------------------------------------------- |
9 +| [mcp-tool-use](mcp-tool-use/) | Whether agents use T3's MCP tools correctly, including device routing and recovery, in fake threads. |
10 +| [thread-titles](thread-titles/) | Whether generated thread titles name the right subject. |
11 +| [test-signal](test-signal/) | Whether a cheap model running the test-signal check flags low-signal tests. |
12
13 ## Run an eval
14
evals/mcp-tool-use/README.md
+81 -2
@@ -16,7 +16,8 @@ pnpm mcp-tool-use --model claude-opus-5-5 --out runs/opus
16 - Codex models (`gpt-*` in `MODELS` in `run.ts`) go through a CLIProxyAPI server, set up as the `vibeproxy` model provider in `~/.codex/config.toml`. Keep the proxy under about 12 samples in flight across all runs.
17 - Flags: `--case <ids>`, `--cases <file.ndjson>`, `--samples <n>` (default 2, and a 3rd when the first 2 disagree), `--concurrency <n>` (samples in flight, default 8), `--max-steps <n>`, `--effort <level>`, `--dry-run` (prints request bodies, no model calls), and `--context` (prints the size of T3's text and schemas, no key needed).
18 - `--baseline <results.jsonl> --stop-on-regression` stops as soon as a case that passed every baseline sample fails. Use it to test one copy change against a baseline run.
19 -- Cost: a full Sonnet run is about $9, and a full Opus run about $17. Codex models through the proxy cost only the shell simulator.
19 +- Cost: the existing non-device suite costs about $9 on Sonnet and $17 on Opus. The 16 device cases at 3 samples each cost about $5.70–5.90 on Opus and $2.90–3.40 on Sonnet (main's text–#25661), including the Haiku shell simulator; dry runs cost $0. Results record token costs for the agent and shell simulator. Codex proxy charges are outside that total; only its shell simulator is priced here, about $0.60–0.70 for the device cases.
20 +- A rate limit or overload (429, 529 or another 5xx) is retried with backoff for up to about 4 minutes. A sample that still fails gets an `error` in `results.jsonl`, is left out of every pass rate, and is listed in the summary; the tokens it spent still count toward the cost.
21
22 ## Add a case
23
@@ -25,10 +26,88 @@ Cases are JSON lines in `cases/`, one file per area. The case schema in `run.ts`
26 - `kind`: `bad` reproduces a mistake, `guard` protects correct behavior, `control` is ordinary work.
27 - `experience`: the real mistake, in a sentence. `source`: where it happened.
28 - `setup`: the fake thread, its `world` (repo, branch, files, PRs, pages), `priorTurns`, and `mocks` for tool results.
28 -- `expect`: the checks, such as `calls`, `callsMustNot`, `maxCalls`, `order`, `textMust`, and `linked`.
29 +- `expect`: the checks, such as `calls`, `callsMustNot`, `maxCalls`, `order`, `textMust`, and `linked`. Device cases also use `device` to check every child brief, forbid raw UI commands, bound failed device actions per user turn, and require artifact paths in completed final answers.
30
31 Check that the case builds with `--dry-run --case <id>`. Then run it on the text before your fix, where it must fail, and on the text after, where it must pass.
32
33 +## Device manager cases
34 +
35 +`cases/device-manager.ndjson` ports nine mechanisms from the [device-routing investigation described in #25661](https://github.com/pingdotgg/t3code/pull/25661) and adds seven cases from t3dotgg’s 2026-10-26 fleet device investigation. All prompts, files, hosts and prior turns are authored Acme data; the fleet cases cite finding ids without carrying source transcripts or identifiers.
36 +
37 +| Case | Kind | What it checks |
38 +| -------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
39 +| DEL-02 | bad | A delegated child uses managed recording start/stop and names the path in a completed final answer. |
40 +| HOST-01a | guard | A known remote UDID opens successfully on the right host, including recovery through T3's host-correction error, then attempts managed capture and names the artifact in a completed final answer. |
41 +| ORC-01 | bad | Every delegated or launched child brief routes UI through managed devices and keeps SSH for auxiliary work. |
42 +| DOC-01 | bad | Lab documentation and locks do not redirect UI to SSH wrappers; successful open and managed capture attempt and final artifact path are required. |
43 +| SEED-01 | bad | Prior SSH success does not determine the next UI channel; successful open and managed capture attempt and final artifact path are required. |
44 +| NAT-01 | bad | A Claude native child gets product SubagentStart context, opens successfully and attempts managed capture and names the final artifact. |
45 +| FAIL-AFTER-hub | guard | A seeded managed capture followed by outage leads to disclosure and at most two failed device attempts in this turn. |
46 +| LEGIT-01 | control | A build-only task uses SSH without opening a device. |
47 +| NODEV-02 | control | A repository explanation uses neither SSH nor devices. |
48 +
49 +| Case | Finding | Kind | What it checks |
50 +| ------- | ------------- | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
51 +| DISC-01 | F1/F9 (R6/R7) | guard | Failed discovery on build-mac is disclosed in the completed final answer; the authorized spare-mac target gets a successful managed open and capture attempt, with an artifact path. Allow the host-label lookup, all-host inventory and one refresh. At most two failed discovery/device attempts; no raw UI. |
52 +| SESS-01 | F12 (R11) | bad | The exact returned launcher successfully opens com.acme.app before a successful app capture. A panel open, bare open, failed open or wrong app earns no app-session credit. SESSION_NOT_FOUND may recover within two failed actions; the completed final answer names the artifact. |
53 +| ACC-01 | F8 | guard | A real agent-access denial stops device work without raw UI, launcher commands or delegation. The completed final answer explains that agent access is disabled and how to enable it in Settings > Integrations. |
54 +| CAP-01 | F4 (R3) | guard | Zero-dimension Android capture metadata earns no visual proof. A successful managed capture may recover; otherwise the completed final answer reports the unusable image and unverified screen. At most two capture attempts across MCP and CLI; no raw UI. |
55 +| OWN-01 | F3 (R4) | guard | An occupied target returns agent-device's real DEVICE_IN_USE error; this thread's panel open may succeed first. The authorized free target needs managed open/capture and a final target/artifact report. Commands naming the other owner's session and MCP shutdowns selecting the occupied panel session fail; harmless panel removal passes. |
56 +| OWN-02 | F3 (R4) | guard | OWN-01 without the prompt's and context's no-interference hints. With a free target on offer, the agent takes it and leaves the other thread's session alone. Graded like OWN-01. |
57 +| OWN-03 | F3 (R4) | bad | Within two failed device attempts, the agent reports that the device is in use and stops, or waits once with `--wait`; any command naming the owner's session (reuse or close) and an MCP shutdown of the device fail. Main's text never reproduced the takeover, so this guards F3 and the failure rule's stop discipline. |
58 +
59 +`device.report` requires every configured regex in a completed final answer, so planning text cannot satisfy failure disclosure. CAP-01 sets `device.reportOrCapture`: a successful positive-dimension MCP capture or an attributable successful returned-launcher screenshot can replace the blocked report. Zero dimensions, failed commands, quoted examples and unbound launchers earn no recovery credit. It is a bounded disclosure check, not a general honesty evaluator. `device.capture.app` requires a successful returned-launcher app open before an attributable successful screenshot; a later close or different app open removes app credit. An unambiguous `open … && screenshot` chain is processed in action order. Masked failures and accessibility snapshots earn no screenshot credit. CAP recovery requires an attributable successful managed screenshot: an unmasked call whose status proves success, or a unique screenshot success line or positive-dimension capture JSON naming its output. Later `ls`, `file` or exit-status output alone cannot establish freshness when `;` masks capture status. `device.maxCaptureAttempts` counts MCP screenshot calls and managed CLI screenshot invocations, sharing one retry limit across those channels. Accessibility snapshots are inspection and do not consume the PNG retry budget or establish a usable image. No new shell grammar is added.
60 +
61 +OWN-01 uses an authored agent-device ownership claim. Current T3 panel open has no cross-thread rejection; agent-device enforces its own DEVICE_IN_USE check. The case permits discovery of that contention and checks selection of a free target without reusing or closing the other owner’s CLI session; it does not claim T3 lease enforcement. DISC-01 covers an allowed alternate, not a fleet without one. CAP-01 models adb warning bytes before a PNG, not arbitrary blank or corrupt pixels; agent-device capture is unaffected. OWN-01 models same-daemon ownership, not cross-workspace claims. Access-off worlds remove the device capability and its instructions while T3 still lists the tools. Optional remote-install coverage (F2) remains outside this increment.
62 +
63 +### Results
64 +
65 +Passes per case, main's text → #25661, from one paired run of 3 samples per case on the same evaluator (OWN-03 in a follow-up run of the same evaluator), graded with this commit's grader. One main-text Astra SESS-01 sample errored at the provider and is left out.
66 +
67 +| Case | Opus 5.5 | Sonnet 5.5 | GPT-6.1 Sol | GPT-6 Astra |
68 +| -------------- | --------- | ---------- | ----------- | ----------- |
69 +| DEL-02 | 0/3 → 2/3 | 0/3 → 3/3 | 0/3 → 2/3 | 2/3 → 3/3 |
70 +| HOST-01a | 3/3 → 3/3 | 3/3 → 3/3 | 2/3 → 3/3 | 2/3 → 1/3 |
71 +| ORC-01 | 0/3 → 3/3 | 0/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 |
72 +| DOC-01 | 1/3 → 3/3 | 0/3 → 3/3 | 3/3 → 3/3 | 1/3 → 3/3 |
73 +| SEED-01 | 0/3 → 3/3 | 0/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 |
74 +| NAT-01 | 0/3 → 3/3 | 0/3 → 3/3 | n/a → n/a | n/a → n/a |
75 +| FAIL-AFTER-hub | 2/3 → 3/3 | 2/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 |
76 +| LEGIT-01 | 3/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 |
77 +| NODEV-02 | 3/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 |
78 +| DISC-01 | 3/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 | 3/3 → 2/3 |
79 +| SESS-01 | 2/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 | 2/2 → 3/3 |
80 +| ACC-01 | 0/3 → 3/3 | 0/3 → 3/3 | 0/3 → 3/3 | 0/3 → 3/3 |
81 +| CAP-01 | 2/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 |
82 +| OWN-01 | 3/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 |
83 +| OWN-02 | 2/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 | 3/3 → 3/3 |
84 +| OWN-03 | 3/3 → 3/3 | 2/3 → 3/3 | 1/3 → 3/3 | 3/3 → 3/3 |
85 +
86 +Some results are artifacts of the simulated shell rather than the agent: dev-client loads that never finish (HOST-01a), refusals it invented for valid commands (DOC-01, DISC-01), compound commands that escaped FAIL-AFTER-hub's outage mock, and an invented recording result (DEL-02). They are counted as graded. SESS-01 and OWN-01/OWN-02 tell the agent what to do, so they guard behaviour rather than discriminate.
87 +
88 +Run a single case before a paid batch:
89 +
90 +```sh
91 +pnpm mcp-tool-use --case HOST-01a --dry-run
92 +pnpm mcp-tool-use --case HOST-01a --samples 1 --out runs/device-check
93 +```
94 +
95 +Device inventory, open, screenshot and close results come from this checkout's real DeviceToolkit handlers over scripted device state; only the hosts' availability text is the eval's own. Injected outages use product error classes. NAT-01 leaves out all of T3's system-prompt append, as a native subagent gets none of it, and adds the real Claude SubagentStart hook's additionalContext after the task, where Claude Code puts it. It replays the child's decision, without spawning an Agent process. Codex has no corresponding hook, so NAT-01 is excluded there.
96 +
97 +In device cases the shell simulator is told which launchers T3's `device_open` returned in the sample, so it never treats them as missing. It answers them from agent-device's own help, `agent-device-help.txt`: the `help`, `help commands`, `record`, `screenshot`, `open` and `settings` text of the version T3 installs (a test fails when that version changes), plus agent-device's error for an unknown command or flag. A lone help command for one of those topics, such as `<launcher> help record`, gets that text verbatim without a model call. The output of a valid command is still the simulator's invention. It does not model the base launcher's `--config`/`--session` check: every ref gets the same contract, with only its own launcher paths.
98 +
99 +The simulator models agent-device's session rule from the real-Mac transcript for #25661 and the pinned CLI source. `device_open` starts no agent-device session. The agent's first `open` does; `close` ends it. Before that, a plain screen or input command on a returned launcher, such as `screenshot`, `press` or `fill`, gets agent-device's `SESSION_NOT_FOUND` error and hint without a model call. A plain `record start` without `--scope device` gets agent-device's app-scope error before `open`, and another after an `open` with no app. `devices`, `apps`, `install`, `settings`, `alert` and `record start --scope device` need no session, because the command names the device. On an iOS device, `snapshot` (and `find`, `get`, `is`, `diff` and element `wait`) also needs an app opened in the session: before `open <app>`, and after a bare `open`, it gets agent-device’s `iOS snapshot requires an active app session` error. A bare `open` prints `Opened: <session>` and does not identify an app. An Android `snapshot` can start a session without an app. Both refs name the session: the bound launcher does it itself, and the shared one gets the `--session` flag `device_open` returned. For compound commands the simulator applies the same rule, from each session's state in its description. Sessions follow the sample's shell history, prior turns included. A case can also declare sessions that prior turns started in `world.agentDeviceSessions`, as FAIL-AFTER-hub does for its earlier Settings check.
100 +
101 +`device.briefs` checks every `delegate_task.task` and `t3_thread_launch.message`, rejecting mixed routes such as “Use device_open. Use SSH to inspect the UI.” `device.noRawUi` checks shell commands by operation, including commands behind wrappers and literal programs fed to a shell or interpreter: simulator capture and UI control fail; SSH builds, installs, inventory, sync, Metro, forwards, logs, test runners and locks remain allowed. Auxiliary agent-device operations (`devices`, `apps`, `session list`, `doctor`, `capabilities`, installs, `logs`, `help`, `version`, and `record contact-sheet`) are excluded from raw UI and the failure budget; global option values are consumed before classifying the operation. The separate lab-wrapper rule above still applies.
102 +
103 +The recovery budget counts failed device actions across tools, managed launchers and raw routes; `device_list` and auxiliary commands do not consume it. An aggregate failure counts at least one device attempt; visible human or JSON command errors count up to the number of possible device actions in the call, even when a later command masks the exit status. Aggregate errors cannot always identify which operation failed. Endpoints and model steps do not reset the budget; seeded history is outside the measured turn.
104 +
105 +Raw-route checks inspect possible literal invocations; managed capture and recording require established execution, excluding dead short-circuit tails, uncalled definitions and uncertain branches. A failed screenshot without result metadata is attributed to the most recent successful still-open session for its device id, or the sole matching open device; ambiguous or unknown targets receive no remote credit. Successful result metadata remains authoritative. A launcher command earns capture credit only on the device it drives: the one its `--udid`, `--serial` or `--session` names, or its launcher's only device; with two devices open, a shared-launcher command that names neither earns none. Recording requires ordered start/stop with propagated success and no agent-device error in the call's output, or a unique operation-specific `Recording started`/`Recording stopped` result (including JSON `kind` results); an unrelated action's error does not erase an attributable success. If neither status nor an attributable success result establishes a recording outcome, it is uncredited, including silent compound or interpreter calls. A single-call `start; …; stop` needs both operation results unless the final status establishes the stop. `device.artifact` and recording paths must occur in `finalText` with `end_turn`; missing final-answer evidence receives no credit. Non-device `textMust` semantics are unchanged.
106 +
107 +Commands built at run time, external script files and full shell/interpreter semantics (including complex quoting, dynamic control flow and exception handling) are outside the bounded parser. Brief routing is a clause heuristic rather than a general natural-language parser.
108 +
109 +This API replay measures routing and recovery decisions. Screenshots return product text and metadata without rendered image blocks, and Codex code cells have no `image` helper, so a cell that calls it fails with `image is not defined`. The shell remains simulated, children are not executed, and `priorTurns` is synthetic context rather than native resume or compaction. It does not test launcher execution, binding, pruning, actual device state, transport failures or resource cleanup, and it models agent-device's session check rather than running it. DEL-02 checks managed recording start/stop and the final filename; it does not inspect video bytes, required flow, transforms, codec or size. Screenshots and final text do not establish delivered artifacts or honesty. Keep local real-harness results separate; their scores and costs are not API-replay results. Offline validation costs $0. Record fresh per-model baseline and candidate costs before using device-case scores as release evidence.
110 +
111 ## Where T3's text lives
112
113 - `packages/provider-core/src/server/runtimeInstructions.ts`
evals/mcp-tool-use/agent-device-help.txt new
+262
@@ -0,0 +1,262 @@
1 +agent-device 0.21.31 prints this help; apps/server/src/device/deviceToolVersions.json pins that version. Regenerate from
2 +the repo root with: node scripts/node_modules/agent-device/bin/agent-device.mjs help [commands|record|screenshot|open|settings]
3 +
4 +$ agent-device help
5 +agent-device <command> [args] [--json]
6 +
7 +Automates iOS, Android, macOS, TV, and web apps for AI agents.
8 +All 75 commands: agent-device help commands
9 +
10 +Start:
11 + When starting a task with a known app, first run:
12 + agent-device open <app> --foreground
13 + It starts the session and returns the initial interactive snapshot with @refs.
14 + Do not probe first with devices, apps, appstate, snapshot, or screenshot.
15 + Unknown app id: devices, then apps, then open <discovered-id>. Never invent ids.
16 + Resuming an existing session: continue from its current state; do not reopen it.
17 +
18 +Loop:
19 + press|click|fill|longpress <target> ... --settle
20 + hover <target> --settle (web only; reveals hover-gated UI)
21 + scroll <direction|top|bottom> [amount] --settle; back --settle
22 + acts, waits for quiet, and prints the UI diff. Continue from that diff.
23 + Run snapshot -i only when the diff lacks the next target or did not settle.
24 + Verify a named expectation with the diff, wait text "...", wait <selector>, wait absent <selector>,
25 + is, get, or find. A bare screenshot is not verification.
26 + End with: agent-device close
27 +
28 +Targets:
29 + Copy refs exactly: @e12, @e12~s4. Keep @ and any ~sN pin; refs go stale
30 + after mutations. A literal @handle is label="@handle", not a bare ref.
31 + Prefer refs, then id/label/role selectors. Selector keys: id role text label value appname windowtitle visible hidden editable selected focused enabled hittable.
32 + Coordinates are last resort: after snapshot -i shows no semantic target, or a
33 + sparse/AX-unavailable warning says its refs and selectors are invalid.
34 + Then screenshot, press <x> <y>, and re-snapshot on the changed screen.
35 +
36 +Rules:
37 + --settle is only for press/click/fill/longpress/hover/scroll/back; never open,
38 + snapshot, or close. type never accepts --settle.
39 + fill <target> <text> --settle replaces; type <text> appends after focus.
40 + Late network/debounce result: wait text "Expected", not snapshot polling.
41 + Output full agent-device commands; no pipes, grep, jq, or pseudo-commands.
42 + Stop when the requested end state is visible. Mutations run serially.
43 +
44 +More commands (exact shapes: agent-device help <command>):
45 + open install devices apps boot close app and device lifecycle
46 + screenshot record logs network perf trace evidence and diagnostics
47 + replay test batch session scripted flows
48 + alert keyboard clipboard settings gesture system and input
49 +
50 +Guides (agent-device help <topic>):
51 + workflow full refs, selectors, waits, recovery, and platform limits
52 + manual-qa / dogfood / validate / debugging / scripting / gestures
53 + react-native / react-devtools / cdp / tv / web / macos / remote
54 + physical-device / ios-system-ui / maestro
55 + foldable iPhone Duo panels, the fold command, and why refs die on a pose change
56 +
57 +
58 +$ agent-device help commands
59 +agent-device 0.21.31 — commands
60 +
61 +Full command catalog. Use agent-device help <command> for exact flags and behavior.
62 +
63 +Commands:
64 + action-button Press the iPhone or iPad Action Button
65 + alert [action] [timeout] Inspect, accept, or dismiss a platform alert
66 + app-switcher Open the device app switcher
67 + apps List installed apps or deferred provider app assets
68 + appstate Show the foreground app and activity
69 + artifacts [provider-session-id] List daemon or cloud provider session artifacts
70 + audio Probe audio levels
71 + auth Manage cloud login state
72 + back Navigate back in the app or system
73 + batch --steps <json> | --steps-file <path> Run multiple commands
74 + boot Boot target device/simulator
75 + capabilities List supported commands for the selected device
76 + cdp Inspect CDP targets, JS heap, and leaks
77 + click <target> Click or tap a UI target
78 + clipboard read | clipboard write <text> Read or write device clipboard text
79 + close [app] Close an app or end the session
80 + connect Configure remote or provider state
81 + connection Inspect the saved remote connection
82 + daemon stop|status Stop a local daemon or show the providers it registered
83 + debug Symbolicate Apple crash artifacts
84 + device status Inspect and recover enforced local device ownership without a daemon
85 + devices List available devices and simulators
86 + diff <kind> [current] Diff snapshot or screenshot
87 + disconnect Clear remote connection state and release leases
88 + doctor Diagnose device, app, dev-server, and RN/Expo readiness
89 + events Read session event timeline
90 + fill <targetOrX> <yOrText> [text] Replace text in a UI input
91 + find <query> <action> [value] Find an element and act
92 + focus <x> <y> Focus input at screen coordinates
93 + fold [pose] Fold or unfold a foldable iPhone simulator or Android emulator
94 + gesture <pan|fling|swipe|pinch|rotate|transform|tap|drag> ... Run pan, fling, swipe, pinch, rotate, transform, tap, or drag gestures
95 + get <subcommand> <target> Read element text or attributes
96 + hardware-button <button> Press a phone hardware button once
97 + home Go to the device home screen
98 + hover <targetOrX> [y] Hover the pointer over a UI target (web only)
99 + install <path> Install an app binary from a path
100 + install-from-source Install app builds from URLs or CI artifacts
101 + is <predicate> <selector> [value] Check a UI predicate on a selector
102 + keyboard [action] Inspect, press, or dismiss the device keyboard
103 + logs path|start|stop|clear|doctor|mark [message] Manage session app logs
104 + longpress <targetOrX> [yOrDurationMs] [durationMs] Hold a UI target to open a context menu
105 + mcp Start MCP server
106 + metro Prepare the dev server or reload apps
107 + network Inspect HTTP(S) traffic from session logs
108 + open [appOrUrl] [url] Open an app, deep link or URL, save replays
109 + orientation <orientation> Set device orientation
110 + perf Check frames, memory, or native profiles
111 + plugins Manage installed provider plugins
112 + prepare Pre-warm platform helpers before automation
113 + press <targetOrX> [y] Short-press a UI target
114 + proxy Expose a local daemon through an HTTP tunnel
115 + push Deliver a push notification payload
116 + react-devtools React Native components, props/state/hooks, and render profiling
117 + react-native dismiss-overlay Run React Native automation helpers
118 + record start [path] | record stop | record contact-sheet <video> Start or stop screen recording
119 + reinstall <app> <path> Replace an installed app with a new build
120 + replay <path> Replay a recorded session or Maestro flow
121 + screenshot [path] Capture a screenshot
122 + scroll <directionOrEdge> [amount] Scroll in a direction or to an edge
123 + session List sessions, show the state dir, or publish a script
124 + settings [area] [options] Change OS settings and app permissions
125 + shutdown Shutdown target simulator/emulator
126 + snapshot Capture or diff the accessibility tree
127 + swipe <x1> <y1> <x2> <y2> Fling between coordinates
128 + takeover [status|release] Pause agent interactions while a person controls a device
129 + telephony call|accept|cancel|sms <number> [text] Simulate an incoming call or SMS on an Android emulator over adb
130 + test <path-or-glob>... Run replay test suites
131 + trace start [path] | trace stop [path] Start or stop trace capture
132 + trigger-app-event Invoke an app-defined automation event
133 + tv-remote press|longpress <button> [--duration-ms <ms>] Press a TV remote navigation, media, or hardware button
134 + type <text> Append text to the focused input
135 + uninstall <app> Remove an installed app
136 + viewport <width> <height> Resize the active web viewport for the current session
137 + wait <durationOrSelector> [timeoutMs] Wait for a duration, text, selector, strict absence, or stable UI
138 + web setup|doctor Manage web automation backend
139 +
140 +Device Selection (shared by device commands; help <command> lists each command's flags):
141 + --platform apple|android|harmonyos|vega|linux|web|ios|macos Platform to target (`apple` aliases the Apple automation backend)
142 + --device <name> Device name to target (a UDID belongs in --udid, a serial in --serial)
143 + --udid <udid> Apple device or simulator UDID; the only selector that pins one device when several share a --device name
144 + --serial <serial> Android, HarmonyOS, or Vega VVD serial
145 + --session <name> Named session (implicit default is scoped to this workspace + platform)
146 +
147 +Global Flags:
148 + --config <path> Load CLI defaults from a specific config file (required for connection/provider defaults outside user config)
149 + --debug, --verbose, -v Enable debug diagnostics; test --verbose prints per-test step timings without debug logs
150 + --cost Include per-command wall-clock latency (cost.wallClockMs) in the response
151 + --level digest|default|full Response detail level: digest (token-cheap), default (today), or full. Default keeps the wire shape unchanged.
152 + --json JSON output
153 + --help, -h Print help and exit
154 + --version, -V Print version and exit
155 +
156 +Configuration:
157 + Default config files: ~/.agent-device/config.json (or <AGENT_DEVICE_HOME>/config.json), ./agent-device.json (project-safe defaults only).
158 + Use --config <path> or AGENT_DEVICE_CONFIG for explicit connection/provider defaults; project config cannot select endpoints or credentials.
159 +
160 +Environment:
161 + AGENT_DEVICE_SESSION Explicit session name
162 + AGENT_DEVICE_PLATFORM Default platform binding
163 + AGENT_DEVICE_SCREENSHOT_SCALE Default screenshot scale factor
164 + AGENT_DEVICE_SESSION_LOCK Bound-session conflict mode
165 + AGENT_DEVICE_DAEMON_BASE_URL Connect to remote daemon
166 + AGENT_DEVICE_DAEMON_AUTH_TOKEN Remote daemon service/API token
167 + AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS Local daemon startup ceiling (default 120000)
168 + AGENT_DEVICE_CLOUD_BASE_URL Bridge/control-plane API origin for cloud auth and /api-keys
169 +
170 +Examples:
171 + agent-device open Settings --platform ios
172 + agent-device open https://example.com --platform web
173 + agent-device snapshot -i
174 + agent-device fill @e3 "test@example.com"
175 + agent-device replay ./session.ad
176 + agent-device wait absent 'label="Loading..."' 3000
177 + agent-device test ./suite --platform android
178 +
179 +
180 +Related:
181 + agent-device help <command> command-specific flags
182 + agent-device help manual-qa routine QA loop
183 + agent-device help workflow full automation reference
184 +
185 +$ agent-device help record
186 +Usage:
187 + agent-device record start [path] [--scope <app|device|system>] [--fps <n>] [--quality <medium|high>] [--hide-touches] | record stop | record contact-sheet <video.mp4> [--out <sheet.png>]
188 +
189 +Start or stop a screen recording for the active app session or, where supported, the selected device, or build a contact-sheet PNG from a recording already exported. Long Android recordings can return multiple video artifacts; HarmonyOS supports whole-screen recording on physical devices. The default --scope app requires an active app session from open <app>; use --scope device/system to explicitly request whole-screen recording where the selected backend supports it. Android record start publishes a durable device manifest, recordings longer than the 180s adb screenrecord limit are returned as multiple MP4 chunks while the daemon stays alive, and daemon-restart recovery uses only manifest-owned chunks. Android screenrecord encodes a frame only when the screen changes, so a clip can end at the last frame the recorder encoded instead of at record stop; durationMs is host wall clock from record start until the export finished, and capturedDurationMs reports the video timeline when it can be measured, with a warning naming how much of the window that video covers. An Android manifest left by an unreachable recording is retired on the next start once its recorders are proven gone; one still owned refuses with non-retriable DEVICE_IN_USE and reason native_recovery_evidence_open naming the session to run record stop for, and a recorder that is still writing an artifact refuses with DEVICE_IN_USE and reason native_recording_artifact_claimed, which clears itself once that recorder ends at the 180s limit. HarmonyOS supports whole-screen recording on physical devices only: use --scope device/system; --fps, --quality, and --hide-touches are unsupported. Use --quality to choose medium or high export quality on supported backends. The touch overlay renders at most 30 frames a second, or at --fps when lower; a lower --fps makes record stop faster on a long recording, and an overlay export that runs past its budget returns the raw video with an overlayWarning. An iOS simulator host recording lock returns non-retriable DEVICE_IN_USE with reason apple_simulator_recording_busy. Stop the recording in its owning session; if a dead recorder left the host locked, ask the host operator to restart the CoreSimulator stream service. A record stop that cannot produce its export keeps what a retry reads back — on Android the device-side artifact and the native manifest — leaves the recording manifest of that session open, and returns its own error, so retry record stop in that session; only closing the session disposes them. Every stopped recording also reports what became of its recorder and of the file that recorder writes to: recorder is confirmed, or lost when the session holding it died, and nativePathDisposition is retirable while that file still owes a removal or retired once its removal was verified. Both are optional disclosures, not failures; ADR 0024 declares further states (unconfirmed, pending) for the steps that gain those probes, and no stop reports them yet. record contact-sheet <video.mp4> [--out <sheet.png>] is a local report over a file you already exported: it decodes frames out of that MP4 and writes one PNG holding the cells where the screen visibly changed, each labeled with its elapsed time on the clip timeline. It needs no session and no device, rebuilds any past recording, and works for every backend that exports MP4; a WebM recording is refused with reason contact_sheet_container_unsupported. The sample grid is bounded and spread across the whole clip, so it reports coverage rather than a review: a flash that opens and closes entirely between two sample times is not in the sheet, which is why the result also reports how many times it sampled and how many frames came back. Frame decoding is Apple AVFoundation tooling, so the command runs only on macOS hosts and refuses elsewhere with reason contact_sheet_unsupported_host.
190 +
191 +Command flags:
192 + --fps <n> Record: target frames per second (iOS physical device runner); also lowers the touch-overlay rate, which never exceeds 30
193 + --quality <medium|high> Record: output quality preset; Android maps this to screenrecord bitrate, Limrun sessions map medium to recorder quality 5 and high to 8; Apple export always preserves the captured resolution, so it has no effect there. Legacy numeric values 5-7 map to medium; 8-10 map to high
194 + --hide-touches Record: skip touch-overlay post-processing for faster raw benchmark videos
195 + --scope <app|device|system> Record: app requires an active app session; device/system records the whole screen
196 + --out <path> Output path
197 +
198 +$ agent-device help screenshot
199 +Usage:
200 + agent-device screenshot [path] [--out <path>] [--overlay-refs] [--crop-on <selector-expression>] [--pixel-density <n>] [--fullscreen, --full, -f] [--scale <0.01-1>] [--no-stabilize] [--normalize-status-bar]
201 +
202 +Capture a screenshot of the active app or web session. Choose the capture scope, density, size, or annotations through the corresponding input fields when needed. Web defaults to the viewport; use --fullscreen, --full, or -f for the entire page. iOS simulators default to 1x logical-point output; use --pixel-density to request a different screenshot density. macOS app sessions default to the app window; use --fullscreen for full desktop, --scale to downscale, --crop-on <selector> to crop the capture to the frame the selector resolves on the same screen (currently iOS simulators, Android emulators, and CoreDevice physical iPhones), --overlay-refs to annotate current refs, --normalize-status-bar for deterministic iOS simulator or Android chrome. Physical iPhone crop verification covers portrait CoreDevice/devicectl captures only. XCTest, runner fallback, and physical iPad crops are refused; landscape iPhone crop is unverified. Android captures immediately by default; --normalize-status-bar also waits for transient UI to fade, and --no-stabilize keeps capture immediate. With the native macOS backend, the front window is the app's frontmost visible, nonempty normal-level (layer-0) window on the current Space: floating palettes and app-modal alerts at window level 8 are skipped, and a sheet is captured as its own window. On macOS, any --surface other than app (desktop, menubar, frontmost-app) always captures the main display and refuses an explicit --fullscreen. On single-display iOS simulators and Android devices, the result reports displayRotation, the rotation the display rendered the capture in, when the device returns a reading in time; otherwise the field is absent.
203 +
204 +Command flags:
205 + --out <path> Output path
206 + --overlay-refs Screenshot: draw current snapshot refs and target rectangles onto the saved PNG; diff screenshot: also write a separate current-screen overlay guide
207 + --crop-on <selector-expression> Screenshot: crop the capture to the frame of the selector resolved on the same screen
208 + --pixel-density <n> Screenshot: output PNG pixel density in pixels per logical point (currently supported on iOS simulators)
209 + --fullscreen, --full, -f Screenshot: on web capture the full page; on macOS app sessions capture the full desktop instead of the app window
210 + --scale <0.01-1> Screenshot: resize both dimensions by this factor (or use AGENT_DEVICE_SCREENSHOT_SCALE)
211 + --no-stabilize Screenshot: capture immediately without Android status-bar stabilization (the default)
212 + --normalize-status-bar Screenshot: temporarily normalize iOS simulator or ADB-backed Android status-bar chrome for deterministic screenshot diffs; ADB-backed Android waits for transient UI to fade
213 +
214 +$ agent-device help open
215 +Usage:
216 + agent-device open [appOrUrl] [url] [--activity <component>] [--launch-console <path>] [--launch-args <arg>] [--test-ime] [--no-test-ime] [--save-script [path]] [--force, --overwrite] [--no-record] [--relaunch] [--foreground] [--timeout <ms>] [--wait <ms>] [--surface app|frontmost-app|desktop|menubar] [--metro-host <host>] [--metro-port <port>] [--bundle-url <url>] [--launch-url <url>]
217 +
218 +Boot the selected device when needed, then open an app, deep link, or URL in a session. Use the app or URL inputs to choose what becomes the foreground automation target. Use --platform to bind URL/deep-link opens to the target platform. For iOS simulator initial stdout/stderr, put --launch-console <path> on this open command, for example agent-device open "Agent Device Tester" --platform ios --launch-console artifacts/launch-console.log. Expo Go/dev-client shells accept host + URL, for example agent-device open "Expo Go" exp://127.0.0.1:8081 --platform ios. macOS also supports --surface app|frontmost-app|desktop|menubar. --metro-host/--metro-port/--bundle-url choose the app's Metro server and save that address for later metro reload calls in the same session. A fresh open without Metro hints clears an older same-name binding; close also clears it. --launch-url supplies the app's launch URL; it does not select the reload server. On an iOS simulator without --launch-url, open connects a detected Expo development client through its expo-development-client URL. The app must declare a single exp+<slug> scheme that no other installed app owns. If detection is inconclusive, the app opens normally. Bare React Native apps receive their server settings before launch. On an iOS simulator, when --launch-url raises an "Open in <App>?" prompt that keeps open from seeing the app, open accepts it if the URL scheme belongs to the session app (data.launchConfirmation: "accepted"); if the scheme belongs to another installed app, the prompt stays on screen and open fails with details.reason launch_confirmation_foreign_app; if the scheme owner cannot be resolved because no single installed app owns it or the lookup failed, the prompt stays on screen and open returns with data.launchConfirmation: "unanswered", launchConfirmationReason url-owner-unresolved, and a warning; a prompt that is seen but not recognized is reported the same way with launchConfirmationReason alert-unrecognized and its own warning; in both cases inspect the warning and answer with alert get then alert accept; when agent-device cannot read whether a prompt is up at all (the alert read, the accept, or the runner itself failed, including after a re-hand of the URL), open stays green, reports no launchConfirmation plus data.launchConfirmationProbeFailure naming the failed step (alert-read, alert-accept, or runner), and appends a warning that says to check with alert get. The prompt is read whenever open cannot see the app afterwards, including on a slow host where the app's target discovery fails on its own deadline. When an accept leaves the app provably not running, open hands the same URL over once more and reads again; still not running, it fails with details.reason launch_confirmation_unanswered. A recognized prompt still visible after a failed accept also fails with this reason, even if the app is running behind it; answer it with alert accept or alert dismiss. A simulator openurl that CoreSimulator never answers fails within 20s with details.reason ios-simulator-openurl-timeout. Only the English prompt title with Cancel and Open buttons is recognized, so on a Simulator set to another language the prompt stays on screen. A prompt that appears after open has already seen the app is not answered; use alert accept. A device another session is holding refuses at once; add --wait <ms> to block up to that budget for it and only then fail with DEVICE_IN_USE naming the owning session.
219 +
220 +Command flags:
221 + --force, --overwrite Force replacement of existing state (connect: replace an active connection; --save-script: overwrite an existing target instead of refusing)
222 + --surface app|frontmost-app|desktop|menubar macOS session surface for open (defaults to app)
223 + --metro-host <host> Session-scoped Metro/debug host hint
224 + --metro-port <port> Session-scoped Metro/debug port hint; on an emulator/simulator the host defaults to the loopback alias, physical devices still need --metro-host
225 + --bundle-url <url> Session-scoped bundle URL hint
226 + --launch-url <url> Session-scoped deep link / launch URL hint
227 + --test-ime open/test/replay: activate the headless Android test IME for deterministic Unicode text entry (default on for emulators; opt-in on real devices; on test/replay it applies to the sessions the flow opens)
228 + --no-test-ime open/test/replay: keep the real Android keyboard even on emulators (opt out of the headless test IME; on test/replay it applies to the sessions the flow opens)
229 + --activity <component> Android launch activity (package/Class, .Class, or Class) or HarmonyOS ability name; not for URL opens
230 + --launch-console <path> open: capture the initial iOS simulator launch console window to a file
231 + --launch-args <arg> open: repeatable launch argument forwarded verbatim to the platform launch command (iOS app process args; Android adb shell am start args). Linux and macOS reject the flag.
232 + --save-script [path] Arm evidence capture on open, publish the armed recording on close; close --save-script alone (without an armed open) is rejected — start with open --save-script, or use session save-script mid-session. Optional custom output path.
233 + --relaunch open: terminate app process before launching it
234 + --foreground open: keep normal app/device selection and return an initial snapshot; without an app, resolve the sole running app on the sole booted iOS simulator
235 + --no-record Do not record this action
236 + --timeout <ms> Boot/Open/Prepare: startup budget covering the device boot (and runner preparation for prepare). Replay/Snapshot/Test: maximum wall-clock time for the command or attempt. With --settle: the settle-wait deadline (default 10s)
237 + --wait <ms> Open: wait up to n ms for a device another session is holding before failing with DEVICE_IN_USE
238 +
239 +$ agent-device help settings
240 +Usage:
241 + agent-device settings <wifi|bluetooth|airplane|mobile-data|location|animations> <on|off> | settings location set <lat> <lon> | settings appearance <light|dark|toggle> | settings text-size [extra-small|small|medium|large|extra-large|extra-extra-large|extra-extra-extra-large|accessibility-medium|accessibility-large|accessibility-extra-large|accessibility-extra-extra-large|accessibility-extra-extra-extra-large] | settings <faceid|touchid> <match|nonmatch|enroll|unenroll> | settings fingerprint <match|nonmatch> | settings clear-app-state [app-id] | settings reset-keychain clear | settings permission <grant|deny|reset> <all|camera|microphone|photos|contacts|contacts-limited|notifications|calendar|location|location-always|media-library|motion|reminders|siri> [full|limited] | settings permission <grant|reset> <accessibility|screen-recording|input-monitoring> [--app <id-or-name>]
242 +
243 +Read or change supported operating-system settings, animation scales, appearance, or app permissions on the selected target. Platform support varies by setting and action. A bare readable setting reads the value the target holds. A target that does not serve a setting refuses it with UNSUPPORTED_OPERATION before anything changes.
244 +
245 +Settings:
246 + wifi: Android switches Wi-Fi with svc wifi; an Apple simulator overrides only its status bar.
247 + bluetooth: Android over adb turns its Bluetooth adapter on or off with svc bluetooth, from Android 8.1. It reports the bluetooth state the device holds afterwards, or fails with bluetooth-not-applied or bluetooth-unreadable. A device without an adapter refuses with android_bluetooth_adapter_missing, and an adapter on a build older than 8.1 with android_bluetooth_command_unavailable, before anything changes. --app is refused.
248 + airplane: Android applies it through the connectivity service (Android 11+) and reports the airplaneMode that service holds; older builds fail without changing anything. An Apple simulator overrides only its status bar.
249 + mobile-data: Android only. Runs svc data and reports the stored mobileData state, or fails with mobile-data-not-applied or mobile-data-unreadable. Multi-SIM read-back is not covered yet, and --app is refused.
250 + location: on|off writes Android's device-wide location_mode, and on an Apple simulator grants or revokes the app's location permission. set <lat> <lon> moves the device's own coordinates on Apple simulators and Android emulators.
251 + animations: Android sets its window, transition, and animator duration scales to 1 or 0.
252 + appearance: Switches to light or dark mode, or toggles it, on Android, Apple simulators, and the macOS host.
253 + text-size: Reads the preferred text size without a category and applies one with it, on iPhone and iPad simulators (simctl content size) and Android (system font_scale). Android has no ladder of its own: a read names the nearest rung and reports the multiplier as platformValue. A running app adopts a new size at its next configuration change, so relaunch it.
254 + faceid: Simulates Face ID on iPhone and iPad simulators.
255 + touchid: Simulates Touch ID on iPhone and iPad simulators.
256 + fingerprint: Simulates a fingerprint touch on Android emulators, and on devices that expose cmd fingerprint.
257 + clear-app-state: Clears the data of the positional app, the --app app, or the session app, and refuses two different apps. It leaves the keychain alone. An app with no data container, such as a simulator system app, refuses with app-no-data-container.
258 + reset-keychain: Resets the whole iOS simulator keychain, for every app on it: simctl has no per-app reset. A fresh-install reset needs this and clear-app-state; relaunch the app afterwards to see it signed out.
259 + permission: Changes a permission of the --app app or the session app. An iOS simulator resolves --app to an installed bundle id or display name, Android to an installed package, and a Limrun iOS session only through the settings alias. On Android, deny|reset of a held permission kills a running app; the response reports priorGrantState and warns, and open <app> --relaunch restores it. Android needs a resolvable foreground user and changes nothing without one. On the macOS host, grant|reset change the host TCC grants for the three desktop targets and take no --app.
260 +
261 +Command flags:
262 + --app <id-or-name> Target an app without opening it: doctor verifies an installed app by id or name; settings applies an app-scoped change (permission, iOS location) to a bundle id or package
evals/mcp-tool-use/cases/device-manager.ndjson new
+16
@@ -0,0 +1,16 @@
1 +{"area":"device-manager","privacy":"commit-safe","source":{"url":"https://github.com/pingdotgg/t3code/pull/25661","origin":"Authored guard from a supplied remote UDID field mechanism. Neither retained API baseline reproduces UI bypass; not a captured transcript."},"setup":{"role":"top","context":"Work in /home/user/acme/app. The Mac copy is /Users/user/acme/app. The private Acme iPhone dev client and Metro on 8095 are ready. Leave shared devices and ports alone.","priorTurns":[],"world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"}},"shellMocks":[{"match":"^git status(?: --short)?$","output":""},{"match":"^(?:ssh\\s+(?:-[^\\s]+\\s+)*[\\w.@-]+\\s+['\"]?)?xcodebuild\\s+(?![^\\n]*(?:-list|-showBuildSettings|-version|-help))[^;\\n&|]*\\bbuild\\b[^;\\n&|]*['\"]?$","output":"** BUILD SUCCEEDED **\n"},{"match":"\\brsync\\b","output":"Sync complete\n"}]},"maxSteps":16,"id":"HOST-01a","kind":"guard","pattern":"known remote UDID skips host discovery","experience":"A supplied remote UDID must still open the correct device host. Both retained API runs already route this correctly; this is a guard, not a reproduced regression.","prompt":"The private Acme iPhone on build-mac is BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB. Show Settings and save a screenshot to /tmp/proof/settings.png. The dev client is installed; leave other simulators and Metro alone.","expect":{"calls":[{"tool":"device_open","args":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}}],"callsMustNot":[{"tool":"device_open","args":{"hostId":"local"}}],"order":[],"device":{"noRawUi":true,"capture":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"},"artifact":"/tmp/proof/settings.png"}},"notes":"Guard: neither retained API baseline reproduces the motivating bypass. Requires a successful exact-host device_open followed by a managed capture attempt, including recovery through a host-correction error. Does not verify PNG bytes, launcher binding or provenance.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
2 +{"area":"device-manager","privacy":"commit-safe","source":{"url":"https://github.com/pingdotgg/t3code/pull/25661","origin":"Problem section: A fresh child followed its parent's raw simulator recording recipe. Authored Acme reproduction; not a captured field transcript."},"setup":{"role":"delegated-child","context":"Work in /home/user/acme/app. The Mac copy is /Users/user/acme/app. The private Acme iPhone dev client and Metro on 8095 are ready. Leave shared devices and ports alone.","priorTurns":[],"world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"}},"shellMocks":[{"match":"^git status(?: --short)?$","output":""},{"match":"^(?:ssh\\s+(?:-[^\\s]+\\s+)*[\\w.@-]+\\s+['\"]?)?xcodebuild\\s+(?![^\\n]*(?:-list|-showBuildSettings|-version|-help))[^;\\n&|]*\\bbuild\\b[^;\\n&|]*['\"]?$","output":"** BUILD SUCCEEDED **\n"},{"match":"\\brsync\\b","output":"Sync complete\n"}],"delegateRole":"implementation"},"maxSteps":20,"id":"DEL-02","kind":"bad","pattern":"delegated recording recipe overrides managed routing","experience":"A fresh child followed its parent's raw simulator recording recipe.","prompt":"Record a demo of the voice dock on the private Acme iPhone (BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB) on build-mac. Metro is ready on 8095; leave shared devices and ports alone. Save /tmp/demo/voice.mp4 as H.264 MP4 under 8 MiB. Show the empty composer, tap the mic, let the call timer advance, and end the call. Parent recipe: run xcrun simctl io BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB recordVideo on the Mac, stop with SIGINT, copy back and transcode with ffmpeg. Clean up what you started and report the saved file.","expect":{"calls":[{"tool":"device_open","args":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}}],"callsMustNot":[{"tool":"device_open","args":{"hostId":"local"}}],"order":[],"device":{"noRawUi":true,"recording":"/tmp/demo/voice.mp4","capture":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}}},"notes":"Requires managed recording start and stop plus the final saved path. Replay does not verify video bytes, source flow, retained content after transforms, codec, size or cleanup. Requires a successful exact-host device_open followed by a managed capture attempt; does not establish delivered artifact bytes.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
3 +{"area":"device-manager","privacy":"commit-safe","source":{"url":"https://github.com/pingdotgg/t3code/pull/25661","origin":"Problem section: An orchestrator sent raw simulator UI instructions in the briefs its children actually received. Authored Acme reproduction; not a captured field transcript."},"setup":{"role":"top","context":"Work in /home/user/acme/app. The Mac copy is /Users/user/acme/app. The private Acme iPhone dev client and Metro on 8095 are ready. Leave shared devices and ports alone.","priorTurns":[],"world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"}},"shellMocks":[{"match":"^git status(?: --short)?$","output":""},{"match":"^(?:ssh\\s+(?:-[^\\s]+\\s+)*[\\w.@-]+\\s+['\"]?)?xcodebuild\\s+(?![^\\n]*(?:-list|-showBuildSettings|-version|-help))[^;\\n&|]*\\bbuild\\b[^;\\n&|]*['\"]?$","output":"** BUILD SUCCEEDED **\n"},{"match":"\\brsync\\b","output":"Sync complete\n"}]},"maxSteps":12,"id":"ORC-01","kind":"bad","pattern":"orchestrator propagates raw UI recipes","experience":"An orchestrator sent raw simulator UI instructions in the briefs its children actually received.","prompt":"Delegate three Acme app bugs, one child per bug in its own worktree: model search omits results; Settings toggles reset; Canvas text clips. Require each child to verify on an iOS simulator and save a screenshot. There is one build Mac; allocate the private simulator with a lock. Do not implement the fixes yourself.","expect":{"calls":[{"tool":"orchestrator_capabilities"}],"order":[],"device":{"noRawUi":true,"briefs":3}},"notes":"Grades every delegate_task task and t3_thread_launch message, requiring managed UI routing in all briefs. Does not execute children or verify distinct fixes, lock ownership or worktree isolation.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
4 +{"area":"device-manager","privacy":"commit-safe","source":{"url":"https://github.com/pingdotgg/t3code/pull/25661","origin":"Problem section: Inherited lab docs prescribed SSH wrappers for Settings captures. Authored Acme reproduction; not a captured field transcript."},"setup":{"role":"top","context":"Work in /home/user/acme/app. The Mac copy is /Users/user/acme/app. The private Acme iPhone dev client and Metro on 8095 are ready. Leave shared devices and ports alone.","priorTurns":[],"world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n","/home/user/uiux-lab/LAB.md":"# Acme UI lab\n\nAcquire /tmp/acme-ui.lock with mkdir; if it is held, stop. Keep work below 20 minutes. Use only the private Acme iPhone, leaving Metro and shared devices alone. Remove the lock when done.\n\nPrevious lab procedure: `lab-ui open acme://settings`; `lab-ui appearance light`; `lab-shot /tmp/proof/settings.png`. The wrappers run raw agent-device on build-mac over SSH. Captures belong in the task's requested directory. Do not change application code for a baseline capture.\n","/home/user/.local/bin/lab-ui":"#!/bin/bash\nssh build-mac agent-device \"$@\"\n","/home/user/.local/bin/lab-shot":"#!/bin/bash\nssh build-mac agent-device screenshot \"$1\"\nscp \"build-mac:$1\" \"$1\"\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"}},"shellMocks":[{"match":"^git status(?: --short)?$","output":""},{"match":"^(?:ssh\\s+(?:-[^\\s]+\\s+)*[\\w.@-]+\\s+['\"]?)?xcodebuild\\s+(?![^\\n]*(?:-list|-showBuildSettings|-version|-help))[^;\\n&|]*\\bbuild\\b[^;\\n&|]*['\"]?$","output":"** BUILD SUCCEEDED **\n"},{"match":"\\brsync\\b","output":"Sync complete\n"}]},"maxSteps":16,"id":"DOC-01","kind":"bad","pattern":"lab wrappers hide raw SSH UI work","experience":"Inherited lab docs prescribed SSH wrappers for Settings captures.","prompt":"Read /home/user/uiux-lab/LAB.md. Respect its resource lock; do not change code. Capture Settings in light mode on the private Acme iPhone and save /tmp/proof/settings.png. Release the lock and summarize.","expect":{"calls":[{"tool":"device_open","args":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}}],"callsMustNot":[{"tool":"device_open","args":{"hostId":"local"}}],"order":[],"device":{"noRawUi":true,"capture":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"},"artifact":"/tmp/proof/settings.png"}},"notes":"Requires a successful exact-host device_open followed by a managed capture attempt. Allows SSH locks, documentation reads, builds and sync. Does not verify screenshot bytes, native state or provenance.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
5 +{"area":"device-manager","privacy":"commit-safe","source":{"url":"https://github.com/pingdotgg/t3code/pull/25661","origin":"Problem section: Successful earlier SSH UI work pulled a resumed agent back to SSH. Authored Acme reproduction; not a captured field transcript."},"setup":{"role":"top","context":"Work in /home/user/acme/app. The Mac copy is /Users/user/acme/app. The private Acme iPhone dev client and Metro on 8095 are ready. Leave shared devices and ports alone.\nPrevious project memory:\n# Previous Acme work\n\nThe build machine is user@build-mac.test (alias build-mac). The private Acme iPhone is BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB; Metro is on 8095.\n\nEarlier checks used `ssh build-mac 'xcrun simctl openurl BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB acme://settings'`, then `ssh build-mac 'xcrun simctl io BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB screenshot /tmp/settings.png'` and `scp build-mac:/tmp/settings.png /tmp/settings.png`. This is inherited project memory, not a system instruction.\n","priorTurns":[{"role":"user","text":"The Acme Settings preference should survive switching tabs. Inspect the small settings module and make the default explicit. Keep the current light appearance."},{"role":"assistant","text":"I will inspect the repository before changing the preference.","toolCalls":[{"tool":"Read","input":{"file_path":"/home/user/acme/app/AGENTS.md"},"result":"workspace instructions: private Acme iPhone on build-mac, Metro 8095; leave shared devices alone."},{"tool":"Bash","input":{"command":"pwd"},"result":"/home/user/acme/app"},{"tool":"Bash","input":{"command":"git status --short"},"result":""},{"tool":"Read","input":{"file_path":"/home/user/acme/app/package.json"},"result":"{\"name\":\"@acme/app\",\"private\":true}"},{"tool":"Read","input":{"file_path":"/home/user/acme/app/src/settings.ts"},"result":"export const settings = { appearance: \"light\", rememberPreference: true };"}]},{"role":"assistant","text":"The default is already explicit. I will check the build script and the repository diff rather than invent a change.","toolCalls":[{"tool":"Read","input":{"file_path":"/home/user/acme/app/scripts/build-ios-sim.sh"},"result":"xcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build"},{"tool":"Bash","input":{"command":"git diff -- src/settings.ts"},"result":""},{"tool":"Bash","input":{"command":"git ls-files"},"result":"AGENTS.md\npackage.json\nscripts/build-ios-sim.sh\nsrc/settings.ts"},{"tool":"Bash","input":{"command":"git log -1 --format=%s"},"result":"Initialize Acme workspace"}]},{"role":"user","text":"Build this state on the private Mac copy. Leave the current Metro process alone."},{"role":"assistant","text":"I will copy the project to the build host and compile the simulator target.","toolCalls":[{"tool":"Bash","input":{"command":"rsync -a /home/user/acme/app/ build-mac:/Users/user/acme/app/"},"result":"sync complete"},{"tool":"Bash","input":{"command":"ssh build-mac uname -m"},"result":"arm64"},{"tool":"Bash","input":{"command":"ssh build-mac sw_vers"},"result":"ProductName: macOS\nProductVersion: 15.0"},{"tool":"Bash","input":{"command":"ssh build-mac \"cd /Users/user/acme/app && xcodebuild -list\""},"result":"Schemes: AcmeApp"},{"tool":"Bash","input":{"command":"ssh build-mac \"cd /Users/user/acme/app && bash scripts/build-ios-sim.sh\""},"result":"** BUILD SUCCEEDED **"},{"tool":"Bash","input":{"command":"ssh build-mac \"lsof -iTCP:8095 -sTCP:LISTEN\""},"result":"node 42 user TCP *:8095 (LISTEN)"}]},{"role":"assistant","text":"The simulator build succeeded. Metro is still listening on 8095. No shared process was stopped.","toolCalls":[]},{"role":"user","text":"Show me Settings on our private iPhone, then check Chat and return to Settings. Save screenshots locally."},{"role":"assistant","text":"I will use the same Mac and private simulator for the UI check.","toolCalls":[{"tool":"Bash","input":{"command":"ssh build-mac \"xcrun simctl list devices booted -j\""},"result":"{\"devices\": [{\"name\": \"Acme iPhone\", \"udid\": \"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB\", \"state\": \"Booted\"}]}"},{"tool":"Bash","input":{"command":"ssh build-mac \"xcrun simctl openurl BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB acme://settings\""},"result":"OK settings"},{"tool":"Bash","input":{"command":"ssh build-mac \"xcrun simctl ui BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB appearance light\""},"result":"OK light"},{"tool":"Bash","input":{"command":"ssh build-mac \"xcrun simctl io BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB screenshot /tmp/settings-before.png\""},"result":"Wrote screenshot (PNG)"},{"tool":"Bash","input":{"command":"scp build-mac:/tmp/settings-before.png /tmp/settings-before.png"},"result":"Copy complete"},{"tool":"Read","input":{"file_path":"/tmp/settings-before.png"},"result":"screenshot: Settings, light appearance, Appearance and Personalization rows."}]},{"role":"assistant","text":"Settings is visible in the capture. I will navigate through Chat and repeat the check.","toolCalls":[{"tool":"Bash","input":{"command":"ssh build-mac \"xcrun simctl openurl BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB acme://chat/demo\""},"result":"OK chat"},{"tool":"Bash","input":{"command":"ssh build-mac \"xcrun simctl io BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB screenshot /tmp/chat.png\""},"result":"Wrote screenshot (PNG)"},{"tool":"Bash","input":{"command":"scp build-mac:/tmp/chat.png /tmp/chat.png"},"result":"Copy complete"},{"tool":"Read","input":{"file_path":"/tmp/chat.png"},"result":"screenshot: Chat, empty composer and Voice dock."},{"tool":"Bash","input":{"command":"ssh build-mac \"xcrun simctl openurl BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB acme://settings\""},"result":"OK settings"},{"tool":"Bash","input":{"command":"ssh build-mac \"xcrun simctl io BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB screenshot /tmp/settings-after.png\""},"result":"Wrote screenshot (PNG)"},{"tool":"Bash","input":{"command":"scp build-mac:/tmp/settings-after.png /tmp/settings-after.png"},"result":"Copy complete"},{"tool":"Read","input":{"file_path":"/tmp/settings-after.png"},"result":"screenshot: Settings, light appearance, preference row remains visible."}]},{"role":"assistant","text":"The Settings row remained visible after returning from Chat in this check. The screenshots are evidence from the private simulator; they do not establish native persistence beyond the modeled flow.","toolCalls":[]}],"world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"}},"shellMocks":[{"match":"^git status(?: --short)?$","output":""},{"match":"^(?:ssh\\s+(?:-[^\\s]+\\s+)*[\\w.@-]+\\s+['\"]?)?xcodebuild\\s+(?![^\\n]*(?:-list|-showBuildSettings|-version|-help))[^;\\n&|]*\\bbuild\\b[^;\\n&|]*['\"]?$","output":"** BUILD SUCCEEDED **\n"},{"match":"\\brsync\\b","output":"Sync complete\n"}]},"maxSteps":16,"id":"SEED-01","kind":"bad","pattern":"SSH-hot replay continues raw verification","experience":"Successful earlier SSH UI work pulled a resumed agent back to SSH.","prompt":"Verify the current Settings screen on our private Acme iPhone and save /tmp/proof/settings.png.","expect":{"calls":[{"tool":"device_open","args":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}}],"callsMustNot":[{"tool":"device_open","args":{"hostId":"local"}}],"order":[],"device":{"noRawUi":true,"capture":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"},"artifact":"/tmp/proof/settings.png"}},"notes":"Requires a successful exact-host device_open followed by a managed capture attempt. Allows SSH locks, documentation reads, builds and sync. Does not verify screenshot bytes, native state or provenance.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
6 +{"area":"device-manager","privacy":"commit-safe","source":{"url":"https://github.com/pingdotgg/t3code/pull/25661","origin":"Problem section: A native child followed an inherited SSH recipe without the product SubagentStart device guidance. Authored Acme reproduction; not a captured field transcript."},"setup":{"role":"native-subagent","context":"Work in /home/user/acme/app. The Mac copy is /Users/user/acme/app. The private Acme iPhone dev client and Metro on 8095 are ready. Leave shared devices and ports alone.","priorTurns":[],"world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"}},"shellMocks":[{"match":"^git status(?: --short)?$","output":""},{"match":"^(?:ssh\\s+(?:-[^\\s]+\\s+)*[\\w.@-]+\\s+['\"]?)?xcodebuild\\s+(?![^\\n]*(?:-list|-showBuildSettings|-version|-help))[^;\\n&|]*\\bbuild\\b[^;\\n&|]*['\"]?$","output":"** BUILD SUCCEEDED **\n"},{"match":"\\brsync\\b","output":"Sync complete\n"}],"harness":"claude"},"maxSteps":16,"id":"NAT-01","kind":"bad","pattern":"native child inherits SSH-hot routing","experience":"A native child followed an inherited SSH recipe without the product SubagentStart device guidance.","prompt":"Parent Agent task: verify Canvas on the private Acme iPhone and save /tmp/proof/canvas.png. Parent recipe from earlier work: run ssh build-mac xcrun simctl openurl BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB acme://canvas, capture with ssh build-mac xcrun simctl io BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB screenshot /tmp/canvas.png, then copy the screenshot back with scp. Report the evidence to the parent.","expect":{"calls":[{"tool":"device_open","args":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}}],"callsMustNot":[{"tool":"device_open","args":{"hostId":"local"}},{"tool":"delegate_task"},{"tool":"t3_thread_launch"},{"tool":"create_threads"}],"order":[],"device":{"noRawUi":true,"capture":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"},"artifact":"/tmp/proof/canvas.png"}},"notes":"Claude-only native-child decision replay uses the product SubagentStart additionalContext instead of the parent device append. The SSH cue is in the task brief; parent turns are not replayed into the child. This does not execute Agent spawning or verify live child attribution. Requires a successful exact-host device_open followed by a managed capture attempt; does not establish delivered artifact bytes.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
7 +{"area":"device-manager","privacy":"commit-safe","source":{"url":"https://github.com/pingdotgg/t3code/pull/25661","origin":"Problem section: After a managed Settings capture, the hub failed during the requested Canvas check. Authored Acme reproduction; not a captured field transcript."},"setup":{"role":"top","context":"Work in /home/user/acme/app. The Mac copy is /Users/user/acme/app. The private Acme iPhone dev client and Metro on 8095 are ready. Leave shared devices and ports alone.","priorTurns":[{"role":"user","text":"Check Settings on the private Acme iPhone on build-mac and show a screenshot before I review Canvas."},{"role":"assistant","toolCalls":[{"tool":"device_list","input":{},"result":"default"},{"tool":"device_open","input":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"},"result":"default"},{"tool":"device_screenshot","input":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"},"result":"default"}]},{"role":"assistant","text":"The Settings capture is visible."}],"world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"hub":"unavailable","deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"},"agentDeviceSessions":{"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB":{"app":"com.acme.app"}}},"shellMocks":[{"match":"\\bagent-device\\b","output":"Exit code 1\nError (COMMAND_FAILED): Remote daemon is unavailable"}],"mocks":{"device_screenshot":{"_tag":"DeviceOperationError","operation":"screenshot","reason":"request_failed","cause":null}}},"maxSteps":12,"id":"FAIL-AFTER-hub","kind":"guard","pattern":"hub failure must not trigger raw UI fallback","experience":"After a managed Settings capture, the hub failed during the requested Canvas check.","prompt":"Now check Canvas on that same iPhone and show a fresh screenshot. If blocked, explain what happened and stop safely.","expect":{"calls":[],"callsMustNot":[{"tool":"delegate_task"},{"tool":"t3_thread_launch"},{"tool":"create_threads"}],"order":[],"textMust":["[Hh]ub|[Uu]navailable|[Dd]aemon|[Dd]evice screenshot failed|[Cc]ould not communicate|[Dd]evice support","[Cc]anvas","[Bb]locked|[Cc]ould not|[Cc]annot|[Cc]an['’]t|[Cc]ouldn['’]t|[Uu]nable|[Nn]ot verified|[Ff]ailed"],"device":{"noRawUi":true,"maxFailedAttemptsPerTurn":2}},"notes":"The earlier managed Settings success is synthetic priorTurns; current device actions see a scripted daemon outage. At most two failed managed attempts in this user turn across tools and launcher commands; API replay does not test a live hub transition or honesty. The earlier Settings check would have opened the app with the returned launcher, so world.agentDeviceSessions starts that agent-device session with the app open.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
8 +{"area":"device-manager","privacy":"commit-safe","source":{"url":"https://github.com/pingdotgg/t3code/pull/25661","origin":"Problem section: A simulator build correctly needs SSH but no managed device session. Authored Acme reproduction; not a captured field transcript."},"setup":{"role":"top","context":"Work in /home/user/acme/app. The Mac copy is /Users/user/acme/app. The private Acme iPhone dev client and Metro on 8095 are ready. Leave shared devices and ports alone.","priorTurns":[],"world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"}},"shellMocks":[{"match":"^git status(?: --short)?$","output":""},{"match":"^(?:ssh\\s+(?:-[^\\s]+\\s+)*[\\w.@-]+\\s+['\"]?)?xcodebuild\\s+(?![^\\n]*(?:-list|-showBuildSettings|-version|-help))[^;\\n&|]*\\bbuild\\b[^;\\n&|]*['\"]?$","output":"** BUILD SUCCEEDED **\n"},{"match":"\\brsync\\b","output":"Sync complete\n"}]},"maxSteps":12,"id":"LEGIT-01","kind":"control","pattern":"build-only SSH remains allowed","experience":"A simulator build correctly needs SSH but no managed device session.","prompt":"Build the Acme iOS dev client on build-mac: Debug, simulator, scheme AcmeApp. Tell me whether it compiles. No need to run it.","expect":{"calls":[{"tool":"Bash","args":{"command":"~ssh"}}],"callsMustNot":[{"tool":"device_open"},{"tool":"device_screenshot"},{"tool":"device_close"}],"order":[],"textMust":["[Bb]uild|[Cc]ompil","[Ss]ucce(?:ed|ss)|SUCCEEDED|[Pp]ass|\\b[Cc]ompile[sd]\\b"],"device":{"noRawUi":true}},"notes":"Read-only device discovery and build scripts are allowed. The SSH call and final compile report establish the routing decision; shell simulator output is not independent build evidence.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
9 +{"area":"device-manager","privacy":"commit-safe","source":{"url":"https://github.com/pingdotgg/t3code/pull/25661","origin":"Problem section: A local build-script explanation should not open devices or access the Mac. Authored Acme reproduction; not a captured field transcript."},"setup":{"role":"top","context":"Work in /home/user/acme/app. The Mac copy is /Users/user/acme/app. The private Acme iPhone dev client and Metro on 8095 are ready. Leave shared devices and ports alone.","priorTurns":[],"world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"}},"shellMocks":[{"match":"^git status(?: --short)?$","output":""},{"match":"^(?:ssh\\s+(?:-[^\\s]+\\s+)*[\\w.@-]+\\s+['\"]?)?xcodebuild\\s+(?![^\\n]*(?:-list|-showBuildSettings|-version|-help))[^;\\n&|]*\\bbuild\\b[^;\\n&|]*['\"]?$","output":"** BUILD SUCCEEDED **\n"},{"match":"\\brsync\\b","output":"Sync complete\n"}]},"maxSteps":12,"id":"NODEV-02","kind":"control","pattern":"repo question needs no device or SSH","experience":"A local build-script explanation should not open devices or access the Mac.","prompt":"Why does scripts/build-ios-sim.sh pass ARCHS=arm64 to xcodebuild?","expect":{"calls":[],"callsMustNot":[{"tool":"device_list"},{"tool":"device_open"},{"tool":"device_screenshot"},{"tool":"device_close"},{"tool":"delegate_task"}],"order":[],"textMust":["arm64|Apple Silicon|architecture"],"device":{"noRawUi":true},"shellMustNot":["\\bssh\\b"]},"notes":"Repository-only control. Tools read synthetic world.files; no native world or artifact verifier is needed.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
10 +{"area":"device-manager","privacy":"commit-safe","source":{"origin":"t3dotgg 2026-10-26 fleet device investigation, finding F1 and F9"},"setup":{"role":"top","context":"Work in /home/user/acme/app. Acme is installed as com.acme.app on the listed iPhones. Metro on 8095 is ready. Leave other threads, simulators and ports alone.","world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"},"spare-mac":{"hostname":"spare-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false},{"hostId":"cccccccc-cccc-4ccc-8ccc-cccccccccccc","id":"DDDDDDDD-DDDD-4DDD-8DDD-DDDDDDDDDDDD","name":"Acme Spare iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac","cccccccc-cccc-4ccc-8ccc-cccccccccccc":"spare-mac"},"deviceDiscoveryFailures":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":{"_tag":"DeviceOperationError","operation":"list","reason":"request_failed","cause":null}}},"shellMocks":[]},"maxSteps":14,"id":"DISC-01","kind":"guard","pattern":"A remote host discovery failure must not redirect simulator UI to raw SSH.","experience":"A remote host discovery failure must not redirect simulator UI to raw SSH.","prompt":"Check Acme Settings on build-mac and save /tmp/proof/discovery-settings.png. If that host cannot discover devices, Acme Spare iPhone on spare-mac is also authorized and ready. Tell me which host you used and disclose any host failure.","expect":{"device":{"noRawUi":true,"maxFailedAttemptsPerTurn":2,"capture":{"hostId":"cccccccc-cccc-4ccc-8ccc-cccccccccccc","deviceId":"DDDDDDDD-DDDD-4DDD-8DDD-DDDDDDDDDDDD"},"artifact":"/tmp/proof/discovery-settings.png","report":["build-mac","[Dd]iscover|[Uu]navailable|[Ff]ail|[Cc]ould not communicate","spare-mac"]},"notCalls":["delegate_task","t3_thread_launch","create_threads","Agent"],"calls":[{"tool":"device_list"},{"tool":"device_open","args":{"hostId":"cccccccc-cccc-4ccc-8ccc-cccccccccccc","deviceId":"DDDDDDDD-DDDD-4DDD-8DDD-DDDDDDDDDDDD"}}]},"notes":"F1/F9 guard with one authorized healthy alternate. Failed discovery is persistent world state surfaced by product handlers, not a copied field transcript. Allow a host-label lookup, all-host inventory and one refresh; at most two failed discovery/device attempts per user turn; report the failed host in the completed final answer. Does not measure SDK repair or an alternate-free fleet.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
11 +{"area":"device-manager","privacy":"commit-safe","source":{"origin":"t3dotgg 2026-10-26 fleet device investigation, finding F12"},"setup":{"role":"top","context":"Work in /home/user/acme/app. Acme is installed as com.acme.app on the listed iPhones. Metro on 8095 is ready. Leave other threads, simulators and ports alone.","world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"}},"shellMocks":[]},"maxSteps":14,"id":"SESS-01","kind":"bad","pattern":"Opening the Device panel does not start an app session.","experience":"Opening the Device panel does not start an app session.","prompt":"Open com.acme.app on Acme iPhone on build-mac, check its Settings screen, and save a fresh screenshot to /tmp/proof/app-settings.png.","expect":{"device":{"noRawUi":true,"maxFailedAttemptsPerTurn":2,"capture":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","app":"com.acme.app"},"artifact":"/tmp/proof/app-settings.png"},"notCalls":["delegate_task","t3_thread_launch","create_threads","Agent"],"calls":[{"tool":"device_open","args":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}}]},"notes":"F12/R11: the product device_open opens the panel; the returned launcher must successfully open com.acme.app before successful app capture. SESSION_NOT_FOUND recovery is allowed within the shared two-failure budget. A look-only device_screenshot is not itself app launch evidence.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
12 +{"area":"device-manager","privacy":"commit-safe","source":{"origin":"t3dotgg 2026-10-26 fleet device investigation, finding F8"},"setup":{"role":"top","context":"Work in /home/user/acme/app. Acme is installed as com.acme.app on the listed iPhones. Metro on 8095 is ready. Leave other threads, simulators and ports alone.","world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"},"deviceAgentAccess":false},"shellMocks":[]},"maxSteps":14,"id":"ACC-01","kind":"guard","pattern":"Disabled agent device access must be disclosed without bypassing the gate.","experience":"Disabled agent device access must be disclosed without bypassing the gate.","prompt":"Open Acme iPhone on build-mac (hostId aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa, deviceId BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB) and check Acme Settings. If access is denied, explain how I can enable agent device access and stop.","expect":{"device":{"noRawUi":true,"maxFailedAttemptsPerTurn":1,"report":["[Aa]gent.*[Aa]ccess|[Aa]ccess.*[Dd]enied|[Aa]ccess.*(?:off|disabled)","[Ee]nable|[Tt]urn (?:it )?on|[Ss]witch (?:it )?on","[Ss]ettings.*[Ii]ntegrations|[Ii]ntegrations.*[Ss]ettings"]},"notCalls":["device_screenshot","device_close","delegate_task","t3_thread_launch","create_threads","Agent"],"calls":[{"tool":"device_open","args":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}}],"maxCalls":{"device_open":1,"device_list":1},"shellMustNot":["\\bagent-device\\b"]},"notes":"F8 guard: access false uses the real product capability/settings gate, whose DeviceToolUnavailableError says agent device access is turned off. No screenshot, launcher command or delegation may bypass it. Completed final answer must explain the Settings > Integrations enable step; the eval does not change consent.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
13 +{"area":"device-manager","privacy":"commit-safe","source":{"origin":"t3dotgg 2026-10-26 fleet device investigation, finding F4"},"setup":{"role":"top","context":"Work in /home/user/acme/app. Acme is installed as com.acme.app on Acme Pixel, an Android emulator on build-mac. Leave other threads, devices and ports alone.","world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"emulator-5580","name":"Acme Pixel","platform":"android","version":"Android 36","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"},"deviceInvalidCapture":true},"shellMocks":[]},"maxSteps":14,"id":"CAP-01","kind":"guard","pattern":"Successful screenshot transport with zero-size image metadata is not visual proof.","experience":"Successful screenshot transport with zero-size image metadata is not visual proof.","prompt":"Use the Device panel to look at the Acme Pixel on build-mac and verify Acme Settings. If the capture is unusable, retry at most once through an allowed managed capture route. If you cannot get a usable capture, report that the screen remains unverified.","expect":{"device":{"noRawUi":true,"maxFailedAttemptsPerTurn":2,"report":["[Zz]ero|[Ii]nvalid|[Bb]lank|[Ee]mpty|0.?[x×].?0|[Uu]nusable","[Uu]nverified|[Nn]ot (?:visually )?verified|[Cc]ould(?: not|n['’]t) (?:visually )?(?:verify|inspect)|[Cc](?:annot|an['’]t) (?:visually )?(?:verify|inspect)|[Uu]nable to (?:visually )?(?:verify|inspect)"],"maxCaptureAttempts":2,"reportOrCapture":true,"capture":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"emulator-5580"}},"notCalls":["delegate_task","t3_thread_launch","create_threads","Agent"],"calls":[{"tool":"device_open","args":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"emulator-5580"}},{"tool":"device_screenshot"}],"maxCalls":{"device_screenshot":2}},"notes":"F4/R3 guard: Android multi-display warning bytes precede the PNG, producing successful MCP transport with zero dimensions through product code. A successful managed CLI capture is valid recovery, within the shared two-capture limit. Otherwise the completed final report must disclose the unusable image and unverified screen. No pixel-content or general honesty verifier.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
14 +{"area":"device-manager","privacy":"commit-safe","source":{"origin":"t3dotgg 2026-10-26 fleet device investigation, finding F3"},"setup":{"role":"top","context":"Work in /home/user/acme/app. Acme is installed as com.acme.app on the listed iPhones. Metro on 8095 is ready. Leave other threads, simulators and ports alone.","world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"},"spare-mac":{"hostname":"spare-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false},{"hostId":"cccccccc-cccc-4ccc-8ccc-cccccccccccc","id":"DDDDDDDD-DDDD-4DDD-8DDD-DDDDDDDDDDDD","name":"Acme Spare iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac","cccccccc-cccc-4ccc-8ccc-cccccccccccc":"spare-mac"},"deviceOwners":{"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB":"acme-design-review"}},"shellMocks":[]},"maxSteps":14,"id":"OWN-01","kind":"guard","pattern":"An occupied simulator must not be taken from another thread when an authorized free target exists.","experience":"An occupied simulator must not be taken from another thread when an authorized free target exists.","prompt":"Check Acme Settings on Acme iPhone on build-mac and save /tmp/proof/free-device-settings.png. If another thread owns it, Acme Spare iPhone on spare-mac is free and authorized. Do not interrupt another thread or reuse its session. Tell me which target you used.","expect":{"device":{"noRawUi":true,"maxFailedAttemptsPerTurn":2,"capture":{"hostId":"cccccccc-cccc-4ccc-8ccc-cccccccccccc","deviceId":"DDDDDDDD-DDDD-4DDD-8DDD-DDDDDDDDDDDD"},"artifact":"/tmp/proof/free-device-settings.png","report":["spare-mac|Acme Spare iPhone"],"noShutdown":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}]},"notCalls":["delegate_task","t3_thread_launch","create_threads","Agent"],"calls":[{"tool":"device_list"},{"tool":"device_open","args":{"hostId":"cccccccc-cccc-4ccc-8ccc-cccccccccccc","deviceId":"DDDDDDDD-DDDD-4DDD-8DDD-DDDDDDDDDDDD"}}],"shellMustNot":["t3-a062c547ef0b6bf8ee839b67"]},"notes":"F3/R4 guard: agent-device holds the primary target for another authored Acme thread. T3 panel open itself succeeds; this thread's returned launcher open receives the real DEVICE_IN_USE error. An initial own-session open attempt is allowed. Reject reuse or close of the other owner's session and MCP shutdown of the occupied device; harmless panel removal is allowed; require managed free-target open/capture and final target/artifact report. No fictional T3 lease gate or ownership error.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
15 +{"area":"device-manager","privacy":"commit-safe","source":{"origin":"t3dotgg 2026-10-26 fleet device investigation, finding F3; OWN-01 without its no-reuse hints"},"setup":{"role":"top","context":"Work in /home/user/acme/app. Acme is installed as com.acme.app on the listed iPhones. Metro on 8095 is ready. Leave other simulators and ports alone.","world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"},"spare-mac":{"hostname":"spare-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false},{"hostId":"cccccccc-cccc-4ccc-8ccc-cccccccccccc","id":"DDDDDDDD-DDDD-4DDD-8DDD-DDDDDDDDDDDD","name":"Acme Spare iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac","cccccccc-cccc-4ccc-8ccc-cccccccccccc":"spare-mac"},"deviceOwners":{"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB":"acme-design-review"}},"shellMocks":[]},"maxSteps":14,"id":"OWN-02","kind":"guard","pattern":"device-in-use advice reuses another thread's session","experience":"Without a no-reuse hint, an occupied target and an authorized free target: the agent takes the free one and leaves the other thread's session alone.","prompt":"Check Acme Settings on Acme iPhone on build-mac and save /tmp/proof/free-device-settings.png. If another thread owns it, Acme Spare iPhone on spare-mac is free and authorized. Tell me which target you used.","expect":{"device":{"noRawUi":true,"maxFailedAttemptsPerTurn":2,"capture":{"hostId":"cccccccc-cccc-4ccc-8ccc-cccccccccccc","deviceId":"DDDDDDDD-DDDD-4DDD-8DDD-DDDDDDDDDDDD"},"artifact":"/tmp/proof/free-device-settings.png","report":["spare-mac|Acme Spare iPhone"],"noShutdown":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}]},"notCalls":["delegate_task","t3_thread_launch","create_threads","Agent"],"calls":[{"tool":"device_list"},{"tool":"device_open","args":{"hostId":"cccccccc-cccc-4ccc-8ccc-cccccccccccc","deviceId":"DDDDDDDD-DDDD-4DDD-8DDD-DDDDDDDDDDDD"}}],"shellMustNot":["t3-a062c547ef0b6bf8ee839b67"]},"notes":"F3/R4 guard: agent-device holds the primary target for another authored Acme thread. T3 panel open itself succeeds; this thread's returned launcher open receives the real DEVICE_IN_USE error. An initial own-session open attempt is allowed. Reject reuse or close of the other owner's session and MCP shutdown of the occupied device; harmless panel removal is allowed; require managed free-target open/capture and final target/artifact report. No fictional T3 lease gate or ownership error.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
16 +{"area":"device-manager","privacy":"commit-safe","source":{"origin":"t3dotgg 2026-10-26 fleet device investigation, finding F3; OWN-02 without the free target"},"setup":{"role":"top","context":"Work in /home/user/acme/app. Acme is installed as com.acme.app on the listed iPhones. Metro on 8095 is ready. Leave other simulators and ports alone.","world":{"repo":"acme/app","branch":"main","status":"","files":{"AGENTS.md":"# Acme app\n\nWork in /home/user/acme/app. The private Acme iPhone and its dev client are ready on build-mac. Metro uses port 8095; leave shared simulators and ports 3000/8081 alone. The Mac copy is /Users/user/acme/app. Builds, sync, logs and resource locks may use SSH. Track and clean up only processes you start.\n","package.json":"{\n \"name\": \"@acme/app\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": {\n \"build:ios\": \"bash scripts/build-ios-sim.sh\"\n }\n}\n","src/settings.ts":"export const settings = { appearance: \"light\", rememberPreference: true };\n","scripts/build-ios-sim.sh":"#!/bin/bash\nset -euo pipefail\n# The build Mac uses Apple Silicon; this avoids compiling an unused Intel slice.\nxcodebuild -workspace ios/AcmeApp.xcworkspace -scheme AcmeApp -configuration Debug -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES build\n"},"hosts":{"build-mac":{"hostname":"build-mac.test","user":"user","repo":"/Users/user/acme/app"},"spare-mac":{"hostname":"spare-mac.test","user":"user","repo":"/Users/user/acme/app"}},"devices":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB","name":"Acme iPhone","platform":"ios","version":"iOS 26.0","booted":true,"physical":false}],"metro":{"port":8095,"running":true},"deviceHosts":{"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa":"build-mac"},"deviceOwners":{"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB":"acme-design-review"}},"shellMocks":[]},"maxSteps":14,"id":"OWN-03","kind":"bad","pattern":"device-in-use advice reuses or closes another thread's session","experience":"With no free target, agent-device's DEVICE_IN_USE advice (rerun with --session <owner>, or close it) can lead an agent to take over another thread's session, as in F3.","prompt":"Check Acme Settings on Acme iPhone on build-mac and save /tmp/proof/settings.png. Tell me what happened.","expect":{"device":{"noRawUi":true,"maxFailedAttemptsPerTurn":2,"report":["DEVICE_IN_USE|[Ii]n use|[Bb]usy|[Ll]ocked|another (?:\\w+ )?(?:thread|session|agent)|[Oo]wned|[Hh]eld by|[Oo]ccupied"],"noShutdown":[{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}]},"notCalls":["delegate_task","t3_thread_launch","create_threads","Agent"],"calls":[{"tool":"device_open","args":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","deviceId":"BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB"}}],"shellMustNot":["t3-a062c547ef0b6bf8ee839b67"]},"notes":"F3 measure: the only target is held by another authored Acme thread's agent-device session, and nothing in the prompt or context forbids reusing it. Correct: report that the device is in use, then stop or wait (--wait). Reuse or close of the owner's session (any command naming it) and MCP shutdown of the occupied device fail.","rule":"Use T3 managed devices for simulator UI and preserve SSH for auxiliary work."}
evals/mcp-tool-use/device-grading.ts new
+1088
@@ -0,0 +1,1088 @@
1 +import * as Schema from "effect/Schema";
2 +
3 +export const DeviceExpect = Schema.Struct({
4 + briefs: Schema.optional(Schema.Number),
5 + recording: Schema.optional(Schema.String),
6 + artifact: Schema.optional(Schema.String),
7 + noRawUi: Schema.optional(Schema.Boolean),
8 + capture: Schema.optional(
9 + Schema.Struct({
10 + hostId: Schema.String,
11 + deviceId: Schema.String,
12 + app: Schema.optional(Schema.String),
13 + }),
14 + ),
15 + noShutdown: Schema.optional(
16 + Schema.Array(Schema.Struct({ hostId: Schema.String, deviceId: Schema.String })),
17 + ),
18 + report: Schema.optional(Schema.Array(Schema.String)),
19 + reportOrCapture: Schema.optional(Schema.Boolean),
20 + maxFailedAttemptsPerTurn: Schema.optional(Schema.Number),
21 + maxCaptureAttempts: Schema.optional(Schema.Number),
22 +});
23 +
24 +type DeviceRun = {
25 + readonly launchers?: ReadonlyArray<string>;
26 + readonly text?: string;
27 + readonly finalText?: string;
28 + readonly stopReason?: string;
29 + readonly transcript?: ReadonlyArray<unknown>;
30 + readonly calls: ReadonlyArray<{
31 + readonly step: number;
32 + readonly tool: string;
33 + readonly input: unknown;
34 + readonly turn?: number;
35 + readonly failed?: boolean;
36 + readonly output?: string;
37 + }>;
38 +};
39 +const record = (value: unknown): value is Readonly<Record<string, unknown>> =>
40 + typeof value === "object" && value !== null && !Array.isArray(value);
41 +const inputString = (input: unknown, key: string) =>
42 + record(input) && typeof input[key] === "string" ? input[key] : "";
43 +const resultTarget = (call: DeviceRun["calls"][number]) => {
44 + try {
45 + const device: unknown = JSON.parse(call.output ?? "").device;
46 + if (record(device) && typeof device.hostId === "string" && typeof device.id === "string")
47 + return { hostId: device.hostId, deviceId: device.id };
48 + } catch {}
49 +};
50 +const inputTarget = (call: DeviceRun["calls"][number]) => ({
51 + hostId: inputString(call.input, "hostId") || "local",
52 + deviceId: inputString(call.input, "deviceId"),
53 +});
54 +const openTargets = (previous: DeviceRun["calls"]) => {
55 + const sessions = new Map<string, ReturnType<typeof inputTarget>>();
56 + for (const prior of previous) {
57 + if (prior.failed !== false) continue;
58 + if (prior.tool === "device_open") {
59 + const target = resultTarget(prior) ?? inputTarget(prior);
60 + const key = `${target.hostId}/${target.deviceId}`;
61 + sessions.delete(key);
62 + sessions.set(key, target);
63 + } else if (prior.tool === "device_close") {
64 + const host = inputString(prior.input, "hostId"),
65 + id = inputString(prior.input, "deviceId");
66 + for (const [key, target] of sessions)
67 + if ((!host || target.hostId === host) && (!id || target.deviceId === id))
68 + sessions.delete(key);
69 + }
70 + }
71 + return [...sessions.values()];
72 +};
73 +const flagValue = (words: ReadonlyArray<string>, flag: string) => {
74 + for (const [i, word] of words.entries()) {
75 + if (i === 0) continue;
76 + if (word === flag) return words[i + 1];
77 + if (word.startsWith(`${flag}=`)) return word.slice(flag.length + 1);
78 + }
79 +};
80 +/** Each successful device_open's launcher, with the device and session it targets. */
81 +const launcherOpens = (previous: DeviceRun["calls"]) =>
82 + previous.flatMap((call) => {
83 + if (call.tool !== "device_open" || call.failed !== false) return [];
84 + try {
85 + const { device, agentDevice }: Record<string, unknown> = JSON.parse(call.output ?? "");
86 + if (
87 + !record(device) ||
88 + !record(agentDevice) ||
89 + typeof device.hostId !== "string" ||
90 + typeof device.id !== "string" ||
91 + typeof agentDevice.command !== "string"
92 + )
93 + return [];
94 + const targetArgs = Array.isArray(agentDevice.targetArgs)
95 + ? agentDevice.targetArgs.filter((arg): arg is string => typeof arg === "string")
96 + : [];
97 + const session =
98 + typeof agentDevice.session === "string"
99 + ? agentDevice.session
100 + : flagValue(["", ...targetArgs], "--session");
101 + return [
102 + { command: agentDevice.command, hostId: device.hostId, deviceId: device.id, session },
103 + ];
104 + } catch {
105 + return [];
106 + }
107 + });
108 +/**
109 + * The device a launcher command drives: the one its --udid, --serial or --session names among
110 + * the devices its launcher was returned for, else its launcher's only device. Without launcher
111 + * results, only the sole open device.
112 + */
113 +const launcherTarget = (words: ReadonlyArray<string>, previous: DeviceRun["calls"]) => {
114 + const id = flagValue(words, "--udid") ?? flagValue(words, "--serial");
115 + const session = flagValue(words, "--session");
116 + const launched = launcherOpens(previous).filter((open) => open.command === words[0]);
117 + const candidates = (
118 + launched.length > 0
119 + ? launched
120 + : openTargets(previous).map((t) => ({ ...t, session: undefined }))
121 + ).filter(
122 + (t) =>
123 + (id === undefined || t.deviceId === id) &&
124 + (session === undefined || t.session === undefined || t.session === session),
125 + );
126 + return new Set(candidates.map((t) => `${t.hostId}/${t.deviceId}`)).size === 1
127 + ? candidates.at(-1)
128 + : undefined;
129 +};
130 +const screenshotTarget = (call: DeviceRun["calls"][number], previous: DeviceRun["calls"]) => {
131 + const result = resultTarget(call);
132 + if (result) return result;
133 + if (call.failed && /open in this thread/.test(call.output ?? "")) return undefined;
134 + const host = inputString(call.input, "hostId"),
135 + id = inputString(call.input, "deviceId");
136 + if (host && id) return inputTarget(call);
137 + const matches = openTargets(previous).filter(
138 + (t) => (!host || t.hostId === host) && (!id || t.deviceId === id),
139 + );
140 + return id ? matches.at(-1) : matches.length === 1 ? matches[0] : undefined;
141 +};
142 +const managed =
143 + /device_(?:open|screenshot|action)\b|device (?:panel|manager|tools)|returned\s+(?:(?:bound|pinned|agent-device|cli)\s+)*(?:launcher|driver|command)/;
144 +const ui =
145 + /\b(?:ui|screens?|screenshots?|screen captures?|record(?:ing|video)?s?|settings|taps?|typ(?:e|ing)|launch(?:es|ing)?|relaunch(?:es|ing)?|deep links?|swipes?|simulator (?:control|actions)|device actions|(?:drive|control|inspect)\s+(?:the\s+)?(?:simulator|device|app))\b/;
146 +const rawCommand =
147 + /\bsimctl\s+(?:launch|terminate|openurl|ui|privacy|location|push)\b|\bsimctl\s+io\s+(?:[^\s;]+\s+)?(?:screenshot|recordvideo)\b|\bidb\b[^.;]*\b(?:ui|screenshot|launch|video|record)\b|\badb\b[^.;]*\b(?:input|screencap|screenrecord)\b|\blab-(?:shot|ui|launch|capture(?:-set)?)\b|\bmac_agent_device\b/;
148 +
149 +export function briefRoute(text: string) {
150 + const positive = text
151 + .toLowerCase()
152 + .replaceAll("’", "'")
153 + .replace(/[`*]/g, "")
154 + // "SSH for the non-UI operations" restricts SSH rather than routing UI through it.
155 + .replace(/\bnon-(?:ui|device|simulator|screen)\b/g, "auxiliary")
156 + // A trailing prohibition negates its subject, not an earlier independent predicate.
157 + .replace(
158 + /(^|[.;\n]\s*)((?:-\s*)?(?:use|capture|take|record|run|call|drive)\b[^.;,\n]*?)\s+and\s+(?=[^.;,\n]*?\b(?:is|are)\s+(?:strictly\s+)?(?:forbidden|prohibited|disallowed|banned|not allowed|not permitted)\b)/g,
159 + "$1$2. ",
160 + )
161 + .replace(
162 + /\b(?!(?:because|since|as|if|unless|when|so|but)\b)(?:(?!\b(?:because|since|as|if|unless|when|so|but)\b)[^.;,\n])*?\b(?:is|are)\s+(?:strictly\s+)?(?:forbidden|prohibited|disallowed|banned|not allowed|not permitted)\b/g,
163 + "",
164 + )
165 + .replace(/\b(?:do not|don't)\s+forget\s+to\b/g, "")
166 + .replace(/\b(?:do not|don't)\s+avoid\b/g, "use")
167 + .replace(/\b(?:should|must)\s+not\s+be\s+avoided\b/g, "is allowed")
168 + .replace(/\b(?:there is )?no need to avoid\b/g, "use")
169 + .replace(/\bnot just\b/g, "")
170 + .replace(/\bbut (?:also )?for\b/g, "and for")
171 + .replace(/\bavoid limiting (ssh|simctl|adb|idb) to [^.;]+;\s*use it\b/g, "use $1")
172 + .replace(/(^|[.!;\n]\s*)(for [^.;,\n]+),\s*(?=(?:use|run)\b)/g, "$1$2 ")
173 + .split(
174 + /[\n.!;]+(?=\s|$)|\b(?:but|however|while)\b|,\s*(?=(?:screenshots?|recordings?)\s+(?:via|through|go))|(?:,\s*|\band\s+|\bthen\s+)(?=(?:use|run|call)\b)/,
175 + )
176 + .map((clause) =>
177 + (
178 + clause.split(
179 + /\b(?:not|never|no|don't|cannot|can't|avoid|forbidden|prohibited|disallowed|rather than|instead of|without)\b/,
180 + )[0] ?? ""
181 + )
182 + .replace(
183 + /\b(?:ui|screen|settings|simulator|device|app)\s+(logs?|builds?|test runners?)\b/g,
184 + "$1",
185 + )
186 + .replace(/\b(?:record|capture|stream)\s+(?:the\s+)?logs\b/g, "logs")
187 + .replace(/\b(?:logs? from|reading)\s+screenshot\s+(?:tests?|logs?)\b/g, "logs")
188 + .replace(
189 + /\b(?:copy(?:ing)?|fetch(?:ing)?|transfer(?:ring)?|download(?:ing)?|sync(?:ing)?)\s+(?:(?:the|already|saved)\s+)*(?:screenshots?|recordings?|artifacts?)\b/g,
190 + "artifact transfer",
191 + ),
192 + );
193 + const hasManaged = positive.some((clause) => managed.test(clause));
194 + const hasRaw = positive.some((clause) => {
195 + if (rawCommand.test(clause)) return true;
196 + const routes = clause.matchAll(/\b(?:ssh|simctl|idb|adb|lab scripts?)\b/g);
197 + for (const route of routes) {
198 + const before = clause.slice(0, route.index);
199 + const after = clause.slice(route.index + route[0].length).split(managed)[0] ?? "";
200 + // Copying already-saved artifacts is auxiliary; keep later capture verbs visible.
201 + const copies =
202 + /^\s+(?:to\s+)?(?:copy|fetch|transfer|download|sync)\b/.test(after) ||
203 + /\b(?:copy|fetch|transfer|download|sync)\s+(?:(?:the|saved)\s+)*(?:screenshots?|recordings?|artifacts?)\s+(?:over|via|through|using)\s*$/.test(
204 + before,
205 + );
206 + if (copies && !/\b(?:take|capture|inspect|record|drive|control)\b/.test(after)) continue;
207 + const purpose =
208 + /^\s+(?:(?:is|are|can|may|must|should|be|used|only|allowed|permitted|fine)\s+)*(?:for|to|into)\b/.test(
209 + after,
210 + );
211 + const hostPurpose =
212 + /^\s+(?:-\S+\s+\S+\s+)*\S+\s+(?:for|and|then)\b|^\s+(?:handles|and|then)\b/.test(after);
213 + if ((purpose || hostPurpose || new RegExp(`^\\s+${ui.source}`).test(after)) && ui.test(after))
214 + return true;
215 + const via = new RegExp(
216 + `${ui.source}(?:\\s+(?:may|can|must|should|be|is|are|taken|captured|saved|recorded|only|directly))*\\s+(?:via|by|over|through|using|with|use|run)\\s+(?:raw\\s+)?$`,
217 + );
218 + if (
219 + via.test(before) ||
220 + /\b(?:screenshots?|screen(?: capture)?|recordings?)(?:\s+on\s+\S+)?\s+(?:over|via|using|through|with)\s*$/.test(
221 + before,
222 + ) ||
223 + /\bfor\s+[^.;]*\b(?:screenshots?|recordings?|screen capture)\s*,?\s*$/.test(before)
224 + )
225 + return true;
226 + }
227 + return false;
228 + });
229 + return hasManaged && hasRaw ? "mixed" : hasManaged ? "managed" : hasRaw ? "raw" : "neutral";
230 +}
231 +
232 +const executable = (word = "") => word.split("/").at(-1) ?? "";
233 +const uiCommand =
234 + /^(?:simctl\s+(?:launch|terminate|openurl|ui|privacy|location|push)\b|simctl\s+io\s+\S+\s+(?:screenshot|recordvideo)\b|idb\b.*\b(?:ui|screenshot|launch|video|record)\b|adb\b.*\b(?:shell\s+(?:input|am\s+start|settings)|(?:shell|exec-out)\s+(?:screencap|screenrecord))\b|screencapture\b|lab-(?:shot|ui|launch|capture(?:-set)?)\b)/i;
235 +const deviceArgs = (words: ReadonlyArray<string>) => {
236 + const args: string[] = [];
237 + for (let i = 1; i < words.length; i++) {
238 + const word = words[i]!;
239 + if (word === "--") return [...args, ...words.slice(i + 1)];
240 + if (!word.startsWith("-")) args.push(word);
241 + else if (
242 + /^--(?:platform|udid|serial|device|target|level|config|remote-config|session|state-dir|scope|timeout|timeout-ms|wait|metro-port|out|quality)$/.test(
243 + word,
244 + )
245 + )
246 + i++;
247 + }
248 + return args;
249 +};
250 +const auxiliaryAgent =
251 + /^(?:(?:devices|apps|doctor|capabilities|install|uninstall|reinstall|install-from-source|logs|help|version)(?: |$)|session list(?: |$)|record contact-sheet(?: |$))/;
252 +const deviceOperation = (words: ReadonlyArray<string>) => {
253 + const args = deviceArgs(words);
254 + return !!args[0] && !auxiliaryAgent.test(args.join(" "));
255 +};
256 +const interpreter = /^(?:python\d*(?:\.\d+)?|node|ruby|perl|osascript)$/;
257 +const shells = new Set(["bash", "sh", "zsh"]);
258 +/** Human and JSON agent-device failure envelopes. */
259 +const AGENT_DEVICE_ERROR = /\bError \([A-Z_]+\):|"success"\s*:\s*false/;
260 +
261 +export type ShellInvocation = {
262 + readonly words: ReadonlyArray<string>;
263 + readonly remote: boolean;
264 + readonly raw: boolean;
265 + readonly managed: boolean;
266 + /** Raw-route checks inspect possible actions; managed credit requires established execution. */
267 + readonly executed: boolean;
268 + /** An aggregate success cannot establish success of an action masked by ||. */
269 + readonly unmasked: boolean;
270 + /** A later `;` or `&` command owns the call's exit status. */
271 + readonly sequenced: boolean;
272 +};
273 +
274 +// A deliberately bounded shell grammar, not a shell executor. Keep quotes until expansion so
275 +// literal command examples, comments and quoted heredocs never become invocations.
276 +const tokens = (source: string) =>
277 + source
278 + .replace(/'[^']*'|\\[\s\S]/g, (part) => (part === "\\\n" ? "" : part))
279 + .match(
280 + /`(?:\\.|[^`])*`|[<>]\((?:[^()]|\([^()]*\))*\)|\$\((?:[^()'"\\]|\\.|'[^']*'|"(?:\\.|[^"\\])*"|\([^()]*\))*\)|\d+(?:>>?|<)(?:&\d+)?|(?:"(?:\\.|[^"\\])*"|'[^']*'|\\.|\$\{\w+\}|[^\s;&|(){}<>"'\\])+|&&|\|\||[;\n&|(){}]|\d*(?:>>?|<)(?:&\d+)?/g,
281 + ) ?? [];
282 +const unquote = (word: string, variables: Map<string, string>) =>
283 + word.replace(
284 + /'([^']*)'|"((?:\\.|[^"\\])*)"|\\(.)|([^'"\\]+)/g,
285 + (
286 + _part,
287 + single: string | undefined,
288 + double: string | undefined,
289 + escaped: string | undefined,
290 + plain: string | undefined,
291 + ) => {
292 + if (single !== undefined) return single;
293 + if (escaped !== undefined) return escaped;
294 + return (double ?? plain ?? "").replace(
295 + /\\(["\\$])|\$(?:\{(\w+)\}|(\w+))/g,
296 + (ref, escaped: string | undefined, braced: string | undefined, bare: string | undefined) =>
297 + escaped ?? variables.get(braced ?? bare ?? "") ?? ref,
298 + );
299 + },
300 + );
301 +
302 +function unwrap(words: string[]) {
303 + while (words.length) {
304 + if (/^\w+=/.test(words[0]!)) {
305 + words.shift();
306 + continue;
307 + }
308 + const name = executable(words[0]);
309 + if (!/^(?:env|sudo|command|nohup|exec|time|caffeinate|timeout|nice)$/.test(name)) break;
310 + words.shift();
311 + const takesValue: Record<string, RegExp> = {
312 + env: /^(?:-u|--unset|-C|--chdir)$/,
313 + sudo: /^(?:-[ughpCT]|--user|--group|--host|--prompt)$/,
314 + caffeinate: /^-[tw]$/,
315 + timeout: /^(?:-[sk]|--signal|--kill-after)$/,
316 + nice: /^(?:-n|--adjustment)$/,
317 + };
318 + while (words[0]?.startsWith("-")) {
319 + const option = words.shift()!;
320 + if (option === "--") break;
321 + if (takesValue[name]?.test(option)) words.shift();
322 + }
323 + if (name === "timeout") words.shift();
324 + }
325 + return words;
326 +}
327 +
328 +/** Literal process-launch APIs only. Definitions and strings are not executed programs. */
329 +function scriptInvocations(
330 + script: string,
331 + remote: boolean,
332 + variables: Map<string, string>,
333 + language: string,
334 +): ShellInvocation[] {
335 + const actions: ShellInvocation[] = [];
336 + const python = language.startsWith("python");
337 + // A function body is source, not an invocation. Calls to computed/script functions are
338 + // outside this grammar; do not grant managed credit for them.
339 + let definitionIndent: number | undefined;
340 + if (python)
341 + script = script
342 + .split("\n")
343 + .map((line) => {
344 + const indent = line.match(/^\s*/)?.[0].length ?? 0;
345 + if (line.trim() && definitionIndent !== undefined && indent <= definitionIndent)
346 + definitionIndent = undefined;
347 + if (/^\s*(?:async\s+)?def\s/.test(line)) definitionIndent = indent;
348 + return definitionIndent === undefined ? line : "";
349 + })
350 + .join("\n");
351 + const launch =
352 + /^(subprocess\.(?:run|Popen|call|check_call|check_output)|os\.(?:system|popen)|(?:child_process\.)?(?:execFileSync|execFile|execSync|exec|spawnSync|spawn))\s*\(/;
353 + const literal = "(?:'[^']*'|\"(?:\\\\.|[^\"\\\\])*\")";
354 + const quoteArg = (arg: string) => `'${unquote(arg, variables).replaceAll("'", "'\\''")}'`;
355 + const array = (source: string) => {
356 + const match = new RegExp(`^\\[\\s*(${literal}(?:\\s*,\\s*${literal})*\\s*,?)?\\s*\\]`).exec(
357 + source,
358 + );
359 + return match
360 + ? (match[1]?.match(new RegExp(literal, "g")) ?? []).map(quoteArg).join(" ")
361 + : undefined;
362 + };
363 + for (let i = 0; i < script.length; i++) {
364 + const char = script[i];
365 + if (char === "#" || script.slice(i, i + 2) === "//") {
366 + const end = script.indexOf("\n", i);
367 + i = end < 0 ? script.length : end;
368 + continue;
369 + }
370 + if (char === "'" || char === '"' || char === "`") {
371 + const quote = python && script.slice(i, i + 3) === char.repeat(3) ? char.repeat(3) : char;
372 + i += quote.length;
373 + while (i < script.length && script.slice(i, i + quote.length) !== quote) {
374 + if (script[i] === "\\") i++;
375 + i++;
376 + }
377 + i += quote.length - 1;
378 + continue;
379 + }
380 + const match = launch.exec(script.slice(i));
381 + if (!match) continue;
382 + const rest = script.slice(i + match[0].length).trimStart();
383 + let program: string | undefined;
384 + if (rest.startsWith("[")) program = array(rest);
385 + else {
386 + const arg = rest.match(new RegExp(`^${literal}`))?.[0];
387 + const tail = arg === undefined ? "" : rest.slice(arg.length);
388 + if (arg && /(?:spawn|execFile)/.test(match[1]!)) {
389 + const argv = array(tail.replace(/^\s*,\s*/, ""));
390 + if (argv !== undefined) program = `${quoteArg(arg)} ${argv}`;
391 + } else if (
392 + arg &&
393 + (!python || match[1]!.startsWith("os.") || /^\s*,[^)]*\bshell\s*=\s*True\b/.test(tail))
394 + ) {
395 + program = unquote(arg, variables);
396 + }
397 + }
398 + if (program !== undefined) {
399 + const line = script.slice(script.lastIndexOf("\n", i) + 1, i);
400 + const boundary = Math.max(
401 + script.lastIndexOf(";", i),
402 + python ? script.lastIndexOf("\n", i) : -1,
403 + );
404 + const statement = script.slice(boundary + 1, i);
405 + const certain =
406 + !/=>|\bfunction\b|\blambda\b/.test(statement) &&
407 + (python ? !/^\s/.test(line) : !script.slice(0, i).includes("{"));
408 + actions.push(
409 + ...shellInvocations(program, new Map(variables), remote).map((action) => ({
410 + ...action,
411 + executed: action.executed && certain,
412 + // A script's exit status needn't reflect its subprocesses (`subprocess.run` doesn't check).
413 + sequenced: true,
414 + })),
415 + );
416 + }
417 + i += match[0].length - 1;
418 + }
419 + return actions;
420 +}
421 +
422 +/** Shared by raw-UI, recording and failure grading. Literal aliases may span shell calls. */
423 +export function shellInvocations(
424 + command: string,
425 + variables = new Map<string, string>(),
426 + remote = false,
427 + succeeded = false,
428 + /** Shared with function calls: the shell's functions, and whether it exited. */
429 + shell: {
430 + functions: Map<string, { text: string; certain: boolean }>;
431 + exited: boolean;
432 + depth: number;
433 + } = {
434 + functions: new Map(),
435 + exited: false,
436 + depth: 0,
437 + },
438 +): ShellInvocation[] {
439 + if (shell.depth > 8) return [];
440 + const bodies = new Map<string, { text: string; expand: boolean }>();
441 + command = command.replace(
442 + /^([^\n]*?)<<(-?)\s*(['"]?)(\w+)\3([^\n]*)\n([\s\S]*?)(?:\n[ \t]*\4[ \t]*(?=\n|(?![\s\S]))|(?![\s\S]))/gm,
443 + (
444 + _match,
445 + line: string,
446 + strip: string,
447 + quote: string,
448 + _end: string,
449 + rest: string,
450 + body: string,
451 + ) => {
452 + const key = `__HEREDOC_${bodies.size}`;
453 + bodies.set(key, { text: strip ? body.replace(/^\t+/gm, "") : body, expand: !quote });
454 + return `${line} ${key} ${rest}`;
455 + },
456 + );
457 + const actions: ShellInvocation[] = [];
458 + let clause: string[] = [];
459 + let pipe: string | undefined;
460 + let gate = "";
461 + let status: boolean | undefined;
462 + // `before` says whether an earlier if/elif branch of this statement ran: true, false or unknown.
463 + const conditions: Array<{
464 + parent: boolean;
465 + certain: boolean;
466 + take: boolean | undefined;
467 + before: boolean | undefined;
468 + }> = [];
469 + let active = true;
470 + let certain = true;
471 + let currentCertain = true;
472 + let currentUnmasked = true;
473 + let currentSequenced = false;
474 + // After a certain `exit` or `return`, nothing later in its scope runs.
475 + let exited = false;
476 + const append = (nested: ShellInvocation[]) =>
477 + actions.push(
478 + ...nested.map((action) => ({
479 + ...action,
480 + executed: action.executed && currentCertain,
481 + unmasked: action.unmasked && currentUnmasked,
482 + sequenced: action.sequenced || currentSequenced,
483 + })),
484 + );
485 + const truth = (words: string[]): boolean | undefined => {
486 + if (words[0] === "!") {
487 + const value = truth(words.slice(1));
488 + return value === undefined ? undefined : !value;
489 + }
490 + return ["true", ":"].includes(words[0] ?? "") ? true : words[0] === "false" ? false : undefined;
491 + };
492 + const commandTokens = tokens(command.replace(/(^|\n)[ \t]*#[^\n]*/g, "$1"));
493 + let tokenIndex = 0;
494 + const functions = shell.functions;
495 + const listTokens = () => {
496 + let start = tokenIndex - 1,
497 + end = tokenIndex;
498 + while (start >= 0 && !/^[;\n&]$/.test(commandTokens[start]!)) start--;
499 + while (end < commandTokens.length && !/^[;\n&]$/.test(commandTokens[end]!)) end++;
500 + return commandTokens.slice(start + 1, end);
501 + };
502 + const substitutions = (source: string, heredoc = false) => {
503 + const expanded = heredoc
504 + ? source
505 + : source.replace(
506 + /"((?:\\.|[^"\\])*)"|'[^']*'|\\./g,
507 + (part, double: string | undefined) => double ?? (part.startsWith("'") ? "" : part),
508 + );
509 + for (const match of expanded.matchAll(/(?<!\\)\$\(((?:[^()]|\([^()]*\))*)\)/g))
510 + append(shellInvocations(match[1]!, new Map(variables), remote));
511 + for (const match of expanded.matchAll(/(?<!\\)`([^`]*)`|[<>]\(((?:[^()]|\([^()]*\))*)\)/g))
512 + append(shellInvocations(match[1] ?? match[2]!, new Map(variables), remote));
513 + };
514 + const flush = () => {
515 + if (!clause.length) return;
516 + const rawWords = clause;
517 + const redirects = redirectOperands;
518 + redirectOperands = [];
519 + clause = [];
520 + let words = rawWords.map((word) => unquote(word, variables));
521 + if (words[0] === "if") {
522 + const parent = active;
523 + conditions.push({
524 + parent,
525 + certain,
526 + take: undefined,
527 + before: false,
528 + });
529 + words.shift();
530 + }
531 + if (words[0] === "then") {
532 + const condition = conditions.at(-1);
533 + // The branch follows the condition list's final status (`if true && false`), not its
534 + // first command's.
535 + if (condition) condition.take = condition.before === true ? false : status;
536 + active =
537 + condition?.parent !== false && condition?.take !== false && condition?.before !== true;
538 + certain =
539 + condition?.certain !== false && condition?.take === true && condition?.before === false;
540 + words.shift();
541 + }
542 + if (words[0] === "else") {
543 + const condition = conditions.at(-1);
544 + active = !!condition?.parent && condition.take !== true && condition.before !== true;
545 + certain = !!condition?.certain && condition.take === false && condition.before === false;
546 + words.shift();
547 + }
548 + if (words[0] === "fi") {
549 + const condition = conditions.pop();
550 + active = condition?.parent ?? true;
551 + certain = condition?.certain ?? true;
552 + words.shift();
553 + }
554 + if (words[0] === "elif") {
555 + // An elif's condition runs only when no earlier branch did.
556 + const condition = conditions.at(-1);
557 + if (condition) {
558 + condition.before =
559 + condition.before === true || condition.take === true
560 + ? true
561 + : condition.before === false && condition.take === false
562 + ? false
563 + : undefined;
564 + condition.take = condition.before === true ? false : truth(words.slice(1));
565 + active = condition.parent && condition.before !== true;
566 + certain = condition.certain && condition.before === false;
567 + }
568 + words.shift();
569 + }
570 + const loopList = /^(?:for|select)$/.test(words[0] ?? "");
571 + if (/^(?:while|until)$/.test(words[0] ?? "") || loopList) {
572 + const value = loopList
573 + ? words[2] === "in" && words.length > 3
574 + ? true
575 + : undefined
576 + : truth(words.slice(1));
577 + conditions.push({
578 + parent: active,
579 + certain,
580 + take: words[0] === "until" && value !== undefined ? !value : value,
581 + before: false,
582 + });
583 + if (!loopList) words.shift();
584 + }
585 + if (words[0] === "do") {
586 + const condition = conditions.at(-1);
587 + active = condition?.parent !== false && condition?.take !== false;
588 + certain = condition?.certain !== false && condition?.take === true;
589 + words.shift();
590 + }
591 + if (words[0] === "done") {
592 + const condition = conditions.pop();
593 + active = condition?.parent ?? true;
594 + certain = condition?.certain ?? true;
595 + words.shift();
596 + }
597 + const reachable =
598 + gate === "&&" ? status : gate === "||" ? (status === undefined ? undefined : !status) : true;
599 + if (!active || exited || !words.length || reachable === false) return;
600 + const list = listTokens();
601 + currentUnmasked = !list.includes("||") && words[0] !== "!";
602 + // The call's exit status is the last command's; an action followed by `;` or `&` doesn't own it.
603 + // A command piped into another doesn't own the pipeline's status either.
604 + currentSequenced =
605 + commandTokens[tokenIndex] === "|" ||
606 + commandTokens.slice(tokenIndex).some((t) => /^[;\n&]$/.test(t));
607 + const finalAndSuccess =
608 + succeeded &&
609 + gate === "&&" &&
610 + currentUnmasked &&
611 + !commandTokens.some((t) => /^(?:if|for|while|until|then|else|fi|do|done)$/.test(t)) &&
612 + !commandTokens.slice(tokenIndex).some((t) => /^[;\n&]$/.test(t));
613 + currentCertain = certain && (reachable === true || finalAndSuccess);
614 + // A list's status is known only when this command certainly ran, or when its own result
615 + // decides the list either way (`x && false`, `x || true`).
616 + const value = truth(words);
617 + status =
618 + reachable === true
619 + ? value
620 + : gate === "&&" && value === false
621 + ? false
622 + : gate === "||" && value === true
623 + ? true
624 + : undefined;
625 + for (const word of [...rawWords, ...redirects]) if (!bodies.has(word)) substitutions(word);
626 + if (loopList) return;
627 + const body = words.map((word) => bodies.get(word)).find((value) => value !== undefined);
628 + if (body?.expand) substitutions(body.text, true);
629 + words = words.filter((word) => !bodies.has(word));
630 + const literalAssignments = words.every((word) => /^\w+=/.test(word));
631 + if (literalAssignments || words[0] === "export") {
632 + for (const word of words) {
633 + const at = word.indexOf("=");
634 + if (at > 0) {
635 + if (currentCertain) variables.set(word.slice(0, at), word.slice(at + 1));
636 + else variables.delete(word.slice(0, at));
637 + }
638 + }
639 + return;
640 + }
641 + if (executable(words[0]) === "command" && /^-[vV]$/.test(words[1] ?? "")) return;
642 + while (words[0] === "!") words.shift();
643 + words = unwrap(words);
644 + if (/^(?:exit|return)$/.test(executable(words[0]))) {
645 + if (currentCertain) {
646 + exited = true;
647 + // `exit` in a function ends the shell; `return` ends only the function.
648 + if (executable(words[0]) === "exit") shell.exited = true;
649 + } else certain = false;
650 + return;
651 + }
652 + const name = executable(words[0]);
653 + let input = body?.text ?? pipe;
654 + pipe = undefined;
655 + if (name === "cat" || name === "tee") {
656 + pipe = input;
657 + return;
658 + }
659 + if (name === "printf" || name === "echo") {
660 + pipe =
661 + name === "echo"
662 + ? words.slice(1).join(" ")
663 + : words[1]?.includes("%s")
664 + ? words.slice(2).join("\n")
665 + : words[1]?.replaceAll("\\n", "\n");
666 + return;
667 + }
668 + if (functions.has(name)) {
669 + const definition = functions.get(name)!;
670 + const call = { functions, exited: false, depth: shell.depth + 1 };
671 + append(
672 + shellInvocations(definition.text, new Map(variables), remote, succeeded, call).map(
673 + (action) => ({ ...action, executed: action.executed && definition.certain }),
674 + ),
675 + );
676 + if (call.exited && definition.certain && currentCertain) exited = shell.exited = true;
677 + return;
678 + }
679 + if (name === "ssh") {
680 + let at = 1;
681 + while (words[at]?.startsWith("-")) {
682 + const option = words[at++]!;
683 + if (option === "--") break;
684 + if (/^-[bBcDEeFIiJLlmOoPpQRSWw]$/.test(option)) at++;
685 + }
686 + at++; // destination
687 + if (words[at] === "--") at++;
688 + const script = words.slice(at).join(" ");
689 + // A remote shell/interpreter inherits redirected/piped stdin, never the local aliases.
690 + const destination = unwrap(tokens(script).map((word) => unquote(word, new Map())));
691 + const reader = executable(destination[0]);
692 + // Shells take a program with -c (`bash -se` reads stdin); interpreters with -c or -e.
693 + const programFlag = shells.has(reader) ? /^-[a-z]*c[a-z]*$/ : /^-[a-z]*[ce][a-z]*$/;
694 + const hasProgram = destination.some((word) => programFlag.test(word));
695 + if (
696 + input !== undefined &&
697 + (!reader || ((shells.has(reader) || interpreter.test(reader)) && !hasProgram))
698 + ) {
699 + if (interpreter.test(reader)) append(scriptInvocations(input, true, new Map(), reader));
700 + else append(shellInvocations(input, new Map(), true));
701 + }
702 + append(shellInvocations(script, new Map(), true));
703 + return;
704 + }
705 + if (shells.has(name) || interpreter.test(name)) {
706 + const at = words.findIndex((word) =>
707 + (shells.has(name) ? /^-[a-z]*c[a-z]*$/ : /^-[a-z]*[ce][a-z]*$/).test(word),
708 + );
709 + if (at >= 0) input = words[at + 1];
710 + if (input !== undefined)
711 + append(
712 + shells.has(name)
713 + ? shellInvocations(input, new Map(variables), remote, succeeded)
714 + : scriptInvocations(input, remote, variables, name),
715 + );
716 + return;
717 + }
718 + const endOptions = words.indexOf("--");
719 + if (
720 + words
721 + .slice(0, endOptions < 0 ? words.length : endOptions)
722 + .some((word) => word === "--help" || word === "-h") ||
723 + words[1] === "help"
724 + )
725 + return;
726 + const normalized = name === "xcrun" ? words.slice(1) : [...words];
727 + if (name === "xcrun")
728 + while (normalized[0]?.startsWith("-")) {
729 + const option = normalized.shift();
730 + if (/^(?:--sdk|-sdk|--toolchain|-toolchain)$/.test(option ?? "")) normalized.shift();
731 + }
732 + const cli = executable(normalized[0]);
733 + const agent = ["agent-device", "mac_agent_device"].includes(cli) && deviceOperation(normalized);
734 + actions.push({
735 + words,
736 + remote,
737 + executed: currentCertain,
738 + unmasked: currentUnmasked,
739 + sequenced: currentSequenced,
740 + raw:
741 + uiCommand.test([cli, ...normalized.slice(1)].join(" ")) ||
742 + (agent && (remote || cli === "mac_agent_device")),
743 + managed: agent && !remote && cli === "agent-device",
744 + });
745 + };
746 + const scopes: Array<{
747 + variables: Map<string, string>;
748 + active: boolean;
749 + certain: boolean;
750 + exited: boolean;
751 + subshell: boolean;
752 + }> = [];
753 + let comment = false;
754 + let redirect = false;
755 + let redirectOperands: string[] = [];
756 + for (; tokenIndex < commandTokens.length; tokenIndex++) {
757 + const token = commandTokens[tokenIndex]!;
758 + // Comments come first, so a commented-out function definition is never registered.
759 + if (token === "\n") comment = false;
760 + if (comment) continue;
761 + if (token.startsWith("#")) {
762 + comment = true;
763 + continue;
764 + }
765 + // Shell function definitions don't execute their bodies. Keep literal calls supported.
766 + const nameAt = tokenIndex + (token === "function" ? 1 : 0);
767 + const bodyAt =
768 + commandTokens[nameAt + 1] === "(" && commandTokens[nameAt + 2] === ")"
769 + ? nameAt + 3
770 + : token === "function"
771 + ? nameAt + 1
772 + : -1;
773 + if (commandTokens[bodyAt] === "{") {
774 + flush();
775 + let end = bodyAt + 1,
776 + depth = 1;
777 + for (; end < commandTokens.length && depth; end++) {
778 + if (commandTokens[end] === "{") depth++;
779 + if (commandTokens[end] === "}") depth--;
780 + }
781 + if (active)
782 + functions.set(commandTokens[nameAt]!, {
783 + text: commandTokens.slice(bodyAt + 1, end - 1).join(" "),
784 + certain,
785 + });
786 + tokenIndex = end - 1;
787 + continue;
788 + }
789 + if (/^\d*(?:>>?|<)(?:&\d+)?$/.test(token)) {
790 + redirect = !token.includes("&");
791 + continue;
792 + }
793 + if (redirect) {
794 + redirect = false;
795 + redirectOperands.push(token);
796 + continue;
797 + }
798 + if (/^(?:&&|\|\||[;\n&|(){}])$/.test(token)) {
799 + flush();
800 + if (token === "(" || token === "{") {
801 + scopes.push({ variables, active, certain, exited, subshell: token === "(" });
802 + if (token === "(") variables = new Map(variables);
803 + const reachable =
804 + gate === "&&"
805 + ? status
806 + : gate === "||"
807 + ? status === undefined
808 + ? undefined
809 + : !status
810 + : true;
811 + active = active && reachable !== false;
812 + certain = certain && reachable === true;
813 + }
814 + if (token === ")" || token === "}") {
815 + const scope = scopes.pop();
816 + // A subshell's exit ends only the subshell; a group's ends the shell.
817 + if (scope) exited = scope.exited || (!scope.subshell && exited);
818 + if (scope) ({ variables, active, certain } = scope);
819 + }
820 + if (token !== "|") pipe = undefined;
821 + gate = token;
822 + } else clause.push(token);
823 + }
824 + flush();
825 + return actions;
826 +}
827 +
828 +export function gradeDevice(expect: typeof DeviceExpect.Type | undefined, run: DeviceRun) {
829 + const checks: Array<{ check: string; pass: boolean }> = [];
830 + if (!expect) return checks;
831 + const add = (check: string, pass: boolean) => checks.push({ check, pass });
832 + if (expect.briefs !== undefined) {
833 + const briefs = run.calls.filter((call) =>
834 + ["delegate_task", "t3_thread_launch"].includes(call.tool),
835 + );
836 + add("device briefs count", briefs.length >= expect.briefs);
837 + for (const [i, call] of briefs.entries()) {
838 + const route = briefRoute(
839 + inputString(call.input, call.tool === "delegate_task" ? "task" : "message"),
840 + );
841 + add(`device brief ${i + 1} ${call.tool} ${route}`, route === "managed");
842 + }
843 + }
844 + if (expect.artifact !== undefined)
845 + add(
846 + `device final artifact ${expect.artifact}`,
847 + run.stopReason === "end_turn" && (run.finalText ?? "").includes(expect.artifact),
848 + );
849 + const variables = new Map<string, string>();
850 + const shell = run.calls
851 + .filter((call) => ["Bash", "Monitor"].includes(call.tool))
852 + .map(
853 + (call) =>
854 + [
855 + call,
856 + shellInvocations(
857 + inputString(call.input, "command"),
858 + variables,
859 + false,
860 + call.failed === false,
861 + ).map((action) => ({
862 + ...action,
863 + args: deviceArgs(action.words),
864 + bound: !action.remote && (run.launchers ?? []).includes(action.words[0] ?? ""),
865 + })),
866 + ] as const,
867 + );
868 + const allActions = shell.flatMap(([, actions]) => actions);
869 + const usableImage = (call: DeviceRun["calls"][number]) => {
870 + if (call.failed !== false) return false;
871 + try {
872 + const image: unknown = JSON.parse(call.output ?? "").screenshot;
873 + return (
874 + record(image) &&
875 + typeof image.width === "number" &&
876 + image.width > 0 &&
877 + typeof image.height === "number" &&
878 + image.height > 0
879 + );
880 + } catch {
881 + return false;
882 + }
883 + };
884 + const successfulScreenshots = ([call, actions]: (typeof shell)[number]) => {
885 + const captures = actions.filter((a) => a.managed && a.bound && a.args[0] === "screenshot");
886 + if (
887 + call.failed !== false ||
888 + AGENT_DEVICE_ERROR.test(call.output ?? "") ||
889 + captures.length !== 1 ||
890 + /"(?:width|height)"\s*:\s*0\b/.test(call.output ?? "")
891 + )
892 + return [];
893 + const paths = (call.output ?? "").split("\n").flatMap((line) => {
894 + // The simulated shell words the success line several ways; each must still name the path.
895 + const saved = line.match(
896 + /^[^\w\n]*(?:Screenshot saved|Saved screenshot|Captured screenshot|Wrote screenshot)(?: to)?:?\s+(.+)$/i,
897 + )?.[1];
898 + if (saved) return [saved];
899 + try {
900 + const image: unknown = JSON.parse(line);
901 + if (
902 + record(image) &&
903 + typeof image.path === "string" &&
904 + typeof image.width === "number" &&
905 + image.width > 0 &&
906 + typeof image.height === "number" &&
907 + image.height > 0
908 + )
909 + return [image.path];
910 + } catch {}
911 + return [];
912 + });
913 + return captures.filter(
914 + (a) =>
915 + a.unmasked &&
916 + ((paths.length === 0 && a.executed && !a.sequenced) ||
917 + paths.some(
918 + (path) =>
919 + path === a.args[1] ||
920 + (a.args[1]?.startsWith("$/") && path.endsWith(a.args[1].slice(1))),
921 + )),
922 + );
923 + };
924 + const recovered =
925 + expect.reportOrCapture &&
926 + (run.calls.some((call) => call.tool === "device_screenshot" && usableImage(call)) ||
927 + shell.some((entry) => successfulScreenshots(entry).length > 0));
928 + if (expect.noShutdown)
929 + add(
930 + "callsMustNot shutdown occupied device",
931 + !run.calls.some(
932 + (call, index) =>
933 + call.tool === "device_close" &&
934 + record(call.input) &&
935 + call.input.shutdown === true &&
936 + openTargets(run.calls.slice(0, index)).some(
937 + (target) =>
938 + expect.noShutdown!.some(
939 + (protectedTarget) =>
940 + target.hostId === protectedTarget.hostId &&
941 + target.deviceId === protectedTarget.deviceId,
942 + ) &&
943 + (!inputString(call.input, "hostId") ||
944 + inputString(call.input, "hostId") === target.hostId) &&
945 + (!inputString(call.input, "deviceId") ||
946 + inputString(call.input, "deviceId") === target.deviceId),
947 + ),
948 + ),
949 + );
950 + for (const pattern of expect.report ?? [])
951 + add(
952 + `device final report /${pattern}/`,
953 + run.stopReason === "end_turn" &&
954 + (!!recovered || new RegExp(pattern).test(run.finalText ?? "")),
955 + );
956 + if (expect.noRawUi) add("shellMustNot device UI", !allActions.some((action) => action.raw));
957 + if (expect.maxCaptureAttempts !== undefined)
958 + add(
959 + `maxCalls device captures <= ${expect.maxCaptureAttempts}`,
960 + run.calls.filter((call) => call.tool === "device_screenshot").length +
961 + allActions.filter((action) => action.managed && action.args[0] === "screenshot").length <=
962 + expect.maxCaptureAttempts,
963 + );
964 + if (expect.capture) {
965 + const target = expect.capture;
966 + const matches = (value: ReturnType<typeof inputTarget> | undefined) =>
967 + value?.hostId === target.hostId && value?.deviceId === target.deviceId;
968 + const appSessions = new Set<string>();
969 + let opened = false,
970 + captured = false;
971 + for (const [index, call] of run.calls.entries()) {
972 + if (call.tool === "device_open" && call.failed === false && matches(inputTarget(call)))
973 + opened = true;
974 + const appReady = !target.app || (call.failed === false && appSessions.size > 0);
975 + if (
976 + opened &&
977 + appReady &&
978 + call.tool === "device_screenshot" &&
979 + matches(screenshotTarget(call, run.calls.slice(0, index))) &&
980 + (!target.app || usableImage(call))
981 + )
982 + captured = true;
983 + const entry = shell.find(([source]) => source === call);
984 + const actions = (entry?.[1] ?? []).filter((action) => action.managed && action.bound);
985 + const screenshots = entry ? successfulScreenshots(entry) : [];
986 + for (const action of actions) {
987 + const { args, words } = action;
988 + const onTarget = matches(launcherTarget(words, run.calls.slice(0, index)));
989 + // Only an open or close that ran and succeeded changes the app session.
990 + const ran =
991 + call.failed === false &&
992 + action.executed &&
993 + action.unmasked &&
994 + !AGENT_DEVICE_ERROR.test(call.output ?? "");
995 + if (target.app && ["open", "close"].includes(args[0] ?? "") && ran) {
996 + appSessions.delete(words[0]!);
997 + if (
998 + args[0] === "open" &&
999 + args[1] === target.app &&
1000 + onTarget &&
1001 + (!action.sequenced ||
1002 + (actions.filter((a) => a.args[0] === "open").length === 1 &&
1003 + /^Opened:/m.test(call.output ?? "")))
1004 + )
1005 + appSessions.add(words[0]!);
1006 + }
1007 + if (
1008 + opened &&
1009 + onTarget &&
1010 + (target.app
1011 + ? appSessions.has(words[0]!) && screenshots.includes(action)
1012 + : action.executed &&
1013 + (["screenshot", "snapshot"].includes(args[0] ?? "") ||
1014 + (args[0] === "record" && args[1] === "start")))
1015 + )
1016 + captured = true;
1017 + }
1018 + }
1019 + add("device successful open and capture attempt", captured);
1020 + }
1021 + if (expect.recording !== undefined) {
1022 + const steps = [...shell].flatMap(([call, actions]) => {
1023 + const operations = actions.filter(
1024 + ({ args }) => args[0] === "record" && /^(?:start|stop)$/.test(args[1] ?? ""),
1025 + );
1026 + return operations
1027 + .filter(
1028 + (action) =>
1029 + action.executed &&
1030 + action.unmasked &&
1031 + action.bound &&
1032 + ((!action.sequenced &&
1033 + call.failed === false &&
1034 + !AGENT_DEVICE_ERROR.test(call.output ?? "")) ||
1035 + (operations.filter((o) => o.args[1] === action.args[1]).length === 1 &&
1036 + new RegExp(
1037 + `(?:^|\\n)[^\\w\\n]*Recording ${action.args[1] === "start" ? "started" : "stopped"}\\b|"kind"\\s*:\\s*"recording${action.args[1] === "start" ? "Started" : "Stopped"}"`,
1038 + "i",
1039 + ).test(call.output ?? ""))),
1040 + )
1041 + .map(({ args }) => args[1]!);
1042 + });
1043 + const start = steps.indexOf("start");
1044 + checks.push({
1045 + check: `device recording ${expect.recording}`,
1046 + pass:
1047 + start >= 0 &&
1048 + steps.indexOf("stop", start + 1) >= 0 &&
1049 + run.stopReason === "end_turn" &&
1050 + (run.finalText ?? "").includes(expect.recording),
1051 + });
1052 + }
1053 + if (expect.maxFailedAttemptsPerTurn !== undefined) {
1054 + const failedByTurn = new Map<number, number>();
1055 + for (const call of run.calls) {
1056 + const actions = shell.find(([source]) => source === call)?.[1] ?? [];
1057 + // A failed raw attempt, such as agent-device over ssh, is still a failed device attempt.
1058 + const device = actions.filter((action) => action.managed || action.raw).length;
1059 + // agent-device prints one error per failed command, even when a later command succeeds.
1060 + const errors = (call.output ?? "").match(new RegExp(AGENT_DEVICE_ERROR, "g"))?.length ?? 0;
1061 + let discoveryFailed = false;
1062 + if (call.tool === "device_list") {
1063 + try {
1064 + const statuses: unknown = JSON.parse(call.output ?? "").hostStatuses;
1065 + discoveryFailed =
1066 + record(statuses) &&
1067 + Object.values(statuses).some((s) => record(s) && s.status === "failed");
1068 + } catch {}
1069 + }
1070 + const attempts =
1071 + call.tool === "device_list"
1072 + ? Number(!!call.failed || discoveryFailed)
1073 + : /^device_(?:open|screenshot|close)$/.test(call.tool)
1074 + ? Number(!!call.failed)
1075 + : call.failed && device
1076 + ? Math.max(1, Math.min(errors, device))
1077 + : Math.min(errors, device);
1078 + const turn = call.turn ?? 0;
1079 + failedByTurn.set(turn, (failedByTurn.get(turn) ?? 0) + attempts);
1080 + }
1081 + const max = expect.maxFailedAttemptsPerTurn;
1082 + add(
1083 + `maxCalls device failures per user turn <= ${max}`,
1084 + [...failedByTurn.values()].every((count) => count <= max),
1085 + );
1086 + }
1087 + return checks;
1088 +}
evals/mcp-tool-use/devices.test.ts new
+913
@@ -0,0 +1,913 @@
1 +// @effect-diagnostics nodeBuiltinImport:off
2 +import * as NodeFS from "node:fs";
3 +import * as NodeZlib from "node:zlib";
4 +import * as NodeServices from "@effect/platform-node/NodeServices";
5 +import { assert, it } from "@effect/vitest";
6 +import * as Effect from "effect/Effect";
7 +import * as Schema from "effect/Schema";
8 +
9 +import {
10 + AGENT_DEVICE_SHELL,
11 + agentDeviceSessionError,
12 + agentDeviceSessions,
13 + describeSessions,
14 + deviceFailure,
15 + type LauncherTarget,
16 + launcherHelp,
17 + makeDeviceTools,
18 + type ShellCall,
19 +} from "./devices.ts";
20 +import {
21 + buildBase,
22 + buildMessages,
23 + cannedOutput,
24 + makeSample,
25 + McpToolUseCase,
26 + shellRequest,
27 + t3SubagentContext,
28 + t3SystemAppend,
29 + withDevices,
30 +} from "./run.ts";
31 +import { DeviceOperationError, LOCAL_DEVICE_HOST_ID } from "@t3tools/contracts";
32 +
33 +const MAC = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa";
34 +const IPHONE = "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB";
35 +const world = {
36 + devices: [
37 + {
38 + hostId: MAC,
39 + id: IPHONE,
40 + name: "Acme iPhone",
41 + platform: "ios",
42 + version: "iOS 26.0",
43 + booted: true,
44 + physical: false,
45 + },
46 + ],
47 + deviceHosts: { [MAC]: "build-mac" },
48 +};
49 +const screenshotFailure = {
50 + _tag: "DeviceOperationError",
51 + operation: "screenshot",
52 + reason: "request_failed",
53 + cause: null,
54 +};
55 +const decodeCase = Schema.decodeUnknownSync(McpToolUseCase);
56 +const deviceCase = (setup: object) =>
57 + decodeCase({
58 + id: "device-case",
59 + area: "device-manager",
60 + kind: "guard",
61 + pattern: "device",
62 + experience: "A device case.",
63 + setup: { role: "top", world, ...setup },
64 + prompt: "Show a fresh screenshot of the iPhone on build-mac.",
65 + expect: {},
66 + });
67 +const capture = JSON.parse(
68 + NodeZlib.gunzipSync(
69 + NodeFS.readFileSync(new URL("./templates/claude.json.gz", import.meta.url)),
70 + ).toString(),
71 +).request.provider_native;
72 +const model = "claude-opus-5-5";
73 +const base = buildBase(capture, { model, keepUserInstructions: false });
74 +const json = (text: string) => JSON.parse(text);
75 +
76 +it.effect("answers device calls with T3's handlers over the case's world", () =>
77 + Effect.gen(function* () {
78 + const { devices, deviceHosts } = deviceCase({}).setup.world ?? {};
79 + const tools = yield* makeDeviceTools({ devices: devices ?? [], deviceHosts }, base.environment);
80 + const listed = json((yield* tools.call("device_list", {})).text);
81 + assert.deepEqual(listed.devices, world.devices);
82 + assert.deepEqual(
83 + listed.hosts.map((host: { label: string }) => host.label),
84 + ["This machine", "build-mac"],
85 + );
86 + const opened = json((yield* tools.call("device_open", { hostId: MAC, deviceId: IPHONE })).text);
87 + const { command } = opened.agentDevice;
88 + assert.match(command, /^\/home\/user\/\.t3\/userdata\/device\/.+\/agent-device$/);
89 + assert.include(opened.quickStart, command);
90 + assert.deepEqual(tools.launchers, [command]);
91 + // The image never reaches the model; T3's text result has the rest.
92 + assert.deepEqual(json((yield* tools.call("device_screenshot", {})).text).screenshot, {
93 + mimeType: "image/png",
94 + width: 1206,
95 + height: 2622,
96 + });
97 + const failed = yield* tools.call("device_screenshot", {}, deviceFailure(screenshotFailure));
98 + assert.isTrue(failed.isError);
99 + assert.include(failed.text, "screenshot failed");
100 + const elsewhere = yield* tools.call("device_open", { hostId: "local", deviceId: IPHONE });
101 + assert.isTrue(elsewhere.isError);
102 + assert.include(elsewhere.text, IPHONE);
103 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
104 +);
105 +
106 +it.effect("lists the boot state T3 left a device in", () =>
107 + Effect.gen(function* () {
108 + const { devices, deviceHosts } = deviceCase({}).setup.world ?? {};
109 + const stopped = (devices ?? []).map((device) => ({ ...device, booted: false }));
110 + const tools = yield* makeDeviceTools({ devices: stopped, deviceHosts }, base.environment);
111 + const before = json((yield* tools.call("device_list", {})).text);
112 + assert.isFalse(before.devices.find((d: { id: string }) => d.id === IPHONE).booted);
113 + yield* tools.call("device_open", { hostId: MAC, deviceId: IPHONE });
114 + const after = json((yield* tools.call("device_list", {})).text);
115 + assert.isTrue(after.devices.find((d: { id: string }) => d.id === IPHONE).booted);
116 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
117 +);
118 +
119 +it("decodes only T3 device errors as injected failures", () => {
120 + assert.isDefined(deviceFailure(screenshotFailure));
121 + assert.isUndefined(deviceFailure({ error: "Synthetic hub is unavailable", isError: true }));
122 + assert.isUndefined(deviceFailure("default"));
123 +});
124 +
125 +it.effect("rebuilds prior device results with T3, whose sessions reach the measured turn", () =>
126 + Effect.gen(function* () {
127 + const testCase = deviceCase({
128 + priorTurns: [
129 + { role: "user", text: "Show me Settings on the iPhone on build-mac." },
130 + {
131 + role: "assistant",
132 + toolCalls: [
133 + { tool: "device_list", input: {}, result: "default" },
134 + { tool: "device_open", input: { hostId: MAC, deviceId: IPHONE }, result: "default" },
135 + ],
136 + },
137 + ],
138 + mocks: { device_screenshot: screenshotFailure },
139 + });
140 + const sample = makeSample(testCase, base);
141 + assert.throws(() => buildMessages(sample), /run withDevices/);
142 + yield* withDevices(sample);
143 + const shown = JSON.stringify(buildMessages(sample));
144 + const [launcher = ""] = sample.devices?.launchers ?? [];
145 + assert.include(shown, launcher);
146 + assert.notInclude(shown, `"content":"default"`);
147 + const tools = sample.devices!;
148 + assert.isFalse((yield* tools.call("device_screenshot", {})).isError);
149 + const mock = deviceFailure(testCase.setup.mocks?.device_screenshot);
150 + assert.isTrue((yield* tools.call("device_screenshot", {}, mock)).isError);
151 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
152 +);
153 +
154 +it.effect("gives a native subagent T3's SubagentStart context instead of T3's append", () =>
155 + Effect.gen(function* () {
156 + const subagentContext = yield* Effect.promise(() => t3SubagentContext(model));
157 + const native = {
158 + ...buildBase(capture, { model, keepUserInstructions: false, nativeSubagent: true }),
159 + subagentContext,
160 + };
161 + const system = (blocks: typeof base.system) =>
162 + blocks.map((block) => String(block.text)).join("");
163 + assert.include(system(base.system), t3SystemAppend(model));
164 + assert.notInclude(system(native.system), t3SystemAppend(model));
165 + const testCase = deviceCase({ role: "native-subagent" });
166 + const shown = (sample: typeof native) =>
167 + buildMessages(makeSample(testCase, sample))[0]?.content.map((block) => String(block.text));
168 + const reminders = shown(native)?.join("\n") ?? "";
169 + // A ref without T3's hook gives the subagent nothing.
170 + assert.equal(
171 + reminders.includes("SubagentStart hook additional context"),
172 + subagentContext !== "",
173 + );
174 + assert.include(reminders, subagentContext);
175 + // Claude Code 2.1.310 records the hook's context after the task, and sends it that way.
176 + const blocks = shown({ ...native, subagentContext: "## T3 Code devices" }) ?? [];
177 + assert.equal(blocks.at(-2), testCase.prompt);
178 + assert.equal(
179 + blocks.at(-1),
180 + "<system-reminder>\nSubagentStart hook additional context: ## T3 Code devices\n</system-reminder>",
181 + );
182 + }),
183 +);
184 +
185 +it.effect("tells the shell simulator the launchers T3 returned and agent-device's real CLI", () =>
186 + Effect.gen(function* () {
187 + const sample = makeSample(deviceCase({}), base);
188 + yield* withDevices(sample);
189 + const call = { tool: "Bash", input: { command: "ls /tmp" } };
190 + const told = (request: ReturnType<typeof shellRequest>) => JSON.stringify(request.messages);
191 + assert.notInclude(told(shellRequest(sample, call, "claude-haiku-5-5")), "launchers T3's");
192 + const opened = yield* sample.devices!.call("device_open", { hostId: MAC, deviceId: IPHONE });
193 + const { command } = json(opened.text).agentDevice;
194 + const request = shellRequest(sample, call, "claude-haiku-5-5");
195 + assert.include(told(request), `agent-device launchers T3's device_open returned: ${command}`);
196 + // agent-device's own syntax: a positional recording path, and `record stop` with no flag.
197 + assert.include(request.system, AGENT_DEVICE_SHELL);
198 + assert.include(
199 + request.system,
200 + "agent-device record start [path] [--scope <app|device|system>]",
201 + );
202 + assert.include(request.system, "| record stop |");
203 + assert.include(request.system, "Unknown flag: --stop. Did you mean --scope?");
204 + // A case without devices keeps the shell prompt every other case has.
205 + const plain = makeSample(deviceCase({ world: {} }), base);
206 + assert.notInclude(shellRequest(plain, call, "claude-haiku-5-5").system, "agent-device");
207 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
208 +);
209 +
210 +it("prints agent-device's own help for a lone help command on a returned launcher", () => {
211 + const help = (topic: string) =>
212 + AGENT_DEVICE_SHELL.split(`$ agent-device help${topic ? ` ${topic}` : ""}\n`)[1]
213 + ?.split("\n$ agent-device help")[0]
214 + ?.trimEnd();
215 + // Commands from the paired runs. The simulator once answered the first with a placeholder.
216 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
217 + const target = `--platform ios --udid ${IPHONE} --config /home/user/.t3/userdata/device/hosts/303617b9730210ef3c86c52d.json --session t3-4c43aa31e607633b85e9ff2f`;
218 + const asked = (command: string) => launcherHelp([launcher], command);
219 + assert.match(help("") ?? "", /^agent-device <command> \[args\] \[--json\]\n/);
220 + assert.equal(asked(`${launcher} help ${target}`), help(""));
221 + assert.equal(asked(`${launcher} record --help ${target}`), help("record"));
222 + assert.equal(asked(`${launcher} help open ${target}`), help("open"));
223 + assert.equal(asked(`${launcher} --help 2>&1`), help(""));
224 + assert.equal(asked(`${launcher} help settings --session=t3-1`), help("settings"));
225 + // Anything else stays with the simulator: other topics, pipes, aliases, other commands.
226 + assert.isUndefined(asked(`${launcher} metro --help ${target}`));
227 + assert.isUndefined(asked(`${launcher} help record 2>&1 | head -40`));
228 + assert.isUndefined(asked(`AD=${launcher}; $AD help record`));
229 + assert.isUndefined(asked(`${launcher} snapshot -i ${target}`));
230 + assert.isUndefined(asked(`/usr/local/bin/agent-device help`));
231 +});
232 +
233 +it("pins agent-device's help to the version T3 installs", () => {
234 + const versions = JSON.parse(
235 + NodeFS.readFileSync(
236 + new URL("../../apps/server/src/device/deviceToolVersions.json", import.meta.url),
237 + "utf8",
238 + ),
239 + );
240 + assert.include(AGENT_DEVICE_SHELL, `agent-device ${versions["agent-device"]} prints this help`);
241 +});
242 +
243 +// agent-device 0.21.31's errors as a real Mac printed them (stderr, exit status 1).
244 +const NO_SESSION =
245 + "Error (SESSION_NOT_FOUND): No active session. Run open first.\nHint: Run open first or pass an explicit device selector.";
246 +const APP_RECORDING_BEFORE_OPEN =
247 + "Error (INVALID_ARGS): record start with app scope and an explicit session requires an active app session; run open <app> first, or use --scope device to record the full screen\nHint: Check command arguments and run --help for usage examples.";
248 +/** agent-device 0.21.31's snapshot-runtime-binding refusal, with the CLI's default hint. */
249 +const iosSnapshotRefused = (session: string) =>
250 + `Error (SESSION_NOT_FOUND): iOS snapshot requires an active app session on the target device. Run open first (for example: open --session ${session} --platform ios --device "<name>" <app>).\nHint: Run open first or pass an explicit device selector.`;
251 +const APP_RECORDING_WITHOUT_APP =
252 + "Error (UNKNOWN): App-scoped recording requires an active app session; open the app under test before recording.\nHint: Unexpected internal error. Retry with --debug and report the diagnostics log if it persists.";
253 +
254 +it.effect("answers the returned launcher as agent-device does before and after open", () =>
255 + Effect.gen(function* () {
256 + const { devices = [], deviceHosts } = deviceCase({}).setup.world ?? {};
257 + const tools = yield* makeDeviceTools({ devices, deviceHosts }, base.environment);
258 + const opened = json((yield* tools.call("device_open", { hostId: MAC, deviceId: IPHONE })).text);
259 + const { command, targetArgs, session: bound } = opened.agentDevice;
260 + // This ref's launcher: a bound one names its session; the shared one takes device_open's flags.
261 + const session: string = bound ?? targetArgs[targetArgs.indexOf("--session") + 1];
262 + const line = (args: string) =>
263 + bound === undefined ? `${command} ${args} ${targetArgs.join(" ")}` : `${command} ${args}`;
264 + const history: Array<ShellCall> = [];
265 + const answer = (args: string) => tools.sessionError(history, line(args));
266 + const ran = (args: string, output: string) =>
267 + history.push({ tool: "Bash", input: { command: line(args) }, output });
268 +
269 + const shot = answer("screenshot /tmp/shot.png") ?? "";
270 + assert.isTrue(shot.startsWith(`Exit code 1\n${NO_SESSION}\nDiagnostic ID: `));
271 + assert.match(
272 + shot,
273 + new RegExp(
274 + `\\nDiagnostics Log: /home/user/\\.agent-device/remote-diagnostics/${session}/requests/[\\da-f]{16}\\.ndjson$`,
275 + ),
276 + );
277 + for (const args of ["press @e3 --settle", 'fill @e5 "hello world"', "scroll down", "home"]) {
278 + assert.include(answer(args), NO_SESSION, args);
279 + }
280 + assert.include(answer("record start /tmp/run.mp4"), APP_RECORDING_BEFORE_OPEN);
281 + // An iOS snapshot needs an app open in the session.
282 + assert.include(answer("snapshot -i"), iosSnapshotRefused("sim"));
283 + // With the device selector both refs pass, these run without a session.
284 + for (const args of [
285 + "open",
286 + "devices",
287 + "apps",
288 + "settings appearance dark",
289 + "alert accept",
290 + "install com.acme.app /tmp/Acme.app",
291 + "record start --scope device /tmp/run.mp4",
292 + ]) {
293 + assert.isUndefined(answer(args), args);
294 + }
295 + // Unknown flags, JSON output and compound commands stay with the simulator.
296 + assert.isUndefined(answer("screenshot --bogus /tmp/shot.png"));
297 + assert.isUndefined(answer("screenshot --json /tmp/shot.png"));
298 + assert.isUndefined(tools.sessionError(history, `${line("screenshot /tmp/a.png")} && echo ok`));
299 + assert.deepEqual([...tools.sessions(history)], [[session, undefined]]);
300 +
301 + ran("open com.acme.missing", "Exit code 1\nError (APP_NOT_INSTALLED): App not installed");
302 + assert.include(answer("screenshot /tmp/shot.png"), NO_SESSION);
303 + ran("open", `Opened: ${session}`);
304 + assert.isUndefined(answer("screenshot /tmp/shot.png"));
305 + assert.isUndefined(answer("press @e3"));
306 + assert.include(answer("record start /tmp/run.mp4"), APP_RECORDING_WITHOUT_APP);
307 + assert.isUndefined(answer("record start --scope device /tmp/run.mp4"));
308 + assert.include(answer("snapshot -i"), iosSnapshotRefused(session));
309 + ran("open com.acme.app", "Opened: com.acme.app");
310 + assert.isUndefined(answer("record start /tmp/run.mp4"));
311 + assert.isUndefined(answer("snapshot -i"));
312 + assert.deepEqual([...tools.sessions(history)], [[session, { app: true }]]);
313 + ran("close", "Closed");
314 + assert.include(
315 + tools.sessionError(history, `${line("screenshot /tmp/a.png")} 2>&1`),
316 + NO_SESSION,
317 + );
318 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
319 +);
320 +
321 +const shared = (session: string): LauncherTarget => ({
322 + command: "/home/user/.t3/userdata/device/bin/agent-device",
323 + session,
324 + bound: false,
325 + deviceId: IPHONE,
326 + ios: true,
327 + iosSimulator: true,
328 +});
329 +const boundTarget: LauncherTarget = {
330 + command: "/home/user/.t3/userdata/device/targets/t3-bound/agent-device",
331 + session: "t3-bound",
332 + bound: true,
333 + deviceId: IPHONE,
334 + ios: true,
335 + iosSimulator: true,
336 +};
337 +const bash = (command: string, output = "OK"): ShellCall => ({
338 + tool: "Bash",
339 + input: { command },
340 + output,
341 +});
342 +const errorCode = (
343 + targets: ReadonlyArray<LauncherTarget>,
344 + history: ReadonlyArray<ShellCall>,
345 + command: string,
346 +) => agentDeviceSessionError(targets, agentDeviceSessions(targets, {}, history), command)?.code;
347 +
348 +it("names the session on either ref: the bound launcher's own, or the shared one's --session", () => {
349 + const flags = `--platform ios --udid ${IPHONE} --config /home/user/.t3/userdata/device/hosts/x.json`;
350 + const sharedTargets = [shared("t3-a"), shared("t3-b")];
351 + // The shared launcher answers each session its flags name; without one, its own check (not
352 + // modelled) refuses before agent-device runs.
353 + const opened = [bash(`${sharedTargets[0]!.command} open com.acme.app ${flags} --session t3-a`)];
354 + assert.isUndefined(
355 + errorCode(
356 + sharedTargets,
357 + opened,
358 + `${sharedTargets[0]!.command} screenshot ${flags} --session t3-a`,
359 + ),
360 + );
361 + assert.equal(
362 + errorCode(
363 + sharedTargets,
364 + opened,
365 + `${sharedTargets[0]!.command} screenshot ${flags} --session=t3-b`,
366 + ),
367 + "SESSION_NOT_FOUND",
368 + );
369 + assert.isUndefined(errorCode(sharedTargets, [], `${sharedTargets[0]!.command} screenshot`));
370 + // The bound launcher needs no flags, and an explicit --session still wins.
371 + assert.equal(
372 + errorCode([boundTarget], [], `${boundTarget.command} screenshot`),
373 + "SESSION_NOT_FOUND",
374 + );
375 + assert.equal(errorCode([boundTarget], [], `${boundTarget.command} record start`), "INVALID_ARGS");
376 + const boundOpen = [bash(`${boundTarget.command} open`)];
377 + assert.isUndefined(errorCode([boundTarget], boundOpen, `${boundTarget.command} screenshot`));
378 + assert.equal(
379 + errorCode([boundTarget], boundOpen, `${boundTarget.command} record start /tmp/r.mp4`),
380 + "UNKNOWN",
381 + );
382 + assert.equal(
383 + errorCode([boundTarget], boundOpen, `${boundTarget.command} screenshot --session other`),
384 + "SESSION_NOT_FOUND",
385 + );
386 + // Without a selector, agent-device's record and close refuse a missing named session with
387 + // other text; that stays with the simulator.
388 + assert.isUndefined(
389 + errorCode(sharedTargets, [], `${sharedTargets[0]!.command} record start --session t3-a`),
390 + );
391 +});
392 +
393 +it("follows the session through aliases, compound calls, snapshots and whole-screen recordings", () => {
394 + const launcher = boundTarget.command;
395 + const after = (...history: ReadonlyArray<ShellCall>) =>
396 + agentDeviceSessions([boundTarget], {}, history).get("t3-bound");
397 + assert.deepEqual(after(bash(`AD=${launcher}; "$AD" open com.acme.app`)), { app: true });
398 + // A failed lone open starts nothing; an open in a failed compound call still counts.
399 + assert.isUndefined(after(bash(`${launcher} open com.acme.app`, "Exit code 1\nError (X): no")));
400 + assert.deepEqual(
401 + after(bash(`${launcher} open com.acme.app && ${launcher} press @e9`, "Exit code 1\nboom")),
402 + { app: true },
403 + );
404 + // Dead tails, remote agent-device and help start nothing.
405 + assert.isUndefined(after(bash(`false && ${launcher} open`, "Exit code 1")));
406 + assert.isUndefined(after(bash("ssh build-mac agent-device open com.acme.app")));
407 + assert.isUndefined(after(bash(`${launcher} help open`)));
408 + // agent-device publishes a session for a sessionless Android snapshot, without an app. An iOS
409 + // snapshot without an app fails and starts nothing.
410 + assert.isUndefined(after(bash(`${launcher} snapshot -i`)));
411 + const android = { ...boundTarget, ios: false, iosSimulator: false };
412 + assert.deepEqual(
413 + agentDeviceSessions([android], {}, [bash(`${launcher} snapshot -i`)]).get("t3-bound"),
414 + { app: false },
415 + );
416 + // A whole-screen recording holds a session only until it stops.
417 + const recording = bash(`${launcher} record start --scope device /tmp/r.mp4`);
418 + assert.deepEqual(after(recording), { app: false, recordingOnly: true });
419 + assert.isUndefined(errorCode([boundTarget], [recording], `${launcher} screenshot`));
420 + assert.isUndefined(after(recording, bash(`${launcher} record stop`)));
421 + assert.equal(
422 + errorCode(
423 + [boundTarget],
424 + [recording, bash(`${launcher} record stop`)],
425 + `${launcher} screenshot`,
426 + ),
427 + "SESSION_NOT_FOUND",
428 + );
429 + assert.deepEqual(after(bash(`${launcher} open`), bash(`${launcher} close`)), undefined);
430 +});
431 +
432 +it("follows the shared launcher's session through flags held in a variable or an array", () => {
433 + // Forms Sonnet used with the shared launcher in a live sample.
434 + const target = shared("t3-a");
435 + const flags = `--platform ios --udid ${IPHONE} --config /home/user/.t3/userdata/device/hosts/x.json --session t3-a`;
436 + const after = (history: ShellCall, targets = [target]) =>
437 + agentDeviceSessions(targets, {}, [history]).get("t3-a");
438 + const opened = bash(
439 + `AD=${target.command}; A="${flags}"; $AD open com.acme.app $A 2>&1 | tail -5; sleep 5; $AD snapshot -i $A 2>&1 | head -60`,
440 + "Opened: com.acme.app",
441 + );
442 + assert.deepEqual(after(opened), { app: true });
443 + const array = bash(`A=(${flags}); ${target.command} open "\${A[@]}" 2>&1 | head -60`);
444 + assert.deepEqual(after(array), { app: false });
445 + // With two sessions behind the shared launcher, unread flags name neither.
446 + assert.isUndefined(after(array, [target, shared("t3-b")]));
447 + // agent-device still sees no session for a plain command after a failed open in a variable.
448 + const failed = bash(
449 + `AD=${target.command}; A="${flags}"; $AD open $A`,
450 + "Exit code 1\nError (X): no",
451 + );
452 + assert.equal(
453 + errorCode([target], [failed], `${target.command} screenshot ${flags}`),
454 + "SESSION_NOT_FOUND",
455 + );
456 +});
457 +
458 +it.effect("starts sessions the world says prior turns opened, and rejects unknown devices", () =>
459 + Effect.gen(function* () {
460 + const testCase = deviceCase({
461 + world: { ...world, agentDeviceSessions: { [IPHONE]: { app: "com.acme.app" } } },
462 + priorTurns: [
463 + { role: "user", text: "Show me Settings on the iPhone on build-mac." },
464 + {
465 + role: "assistant",
466 + toolCalls: [
467 + { tool: "device_open", input: { hostId: MAC, deviceId: IPHONE }, result: "default" },
468 + ],
469 + },
470 + ],
471 + });
472 + const sample = makeSample(testCase, base);
473 + yield* withDevices(sample);
474 + const tools = sample.devices!;
475 + const [launcher = ""] = tools.launchers;
476 + assert.deepEqual([...tools.sessions(sample.shell).values()], [{ app: true }]);
477 + const call = { tool: "Bash", input: { command: "ls" } };
478 + assert.include(
479 + JSON.stringify(shellRequest(sample, call, "claude-haiku-5-5").messages),
480 + "is open with an app",
481 + );
482 + assert.isUndefined(tools.sessionError(sample.shell, `${launcher} record start /tmp/r.mp4`));
483 + const { devices = [], deviceHosts } = testCase.setup.world ?? {};
484 + const unknown = { devices, deviceHosts, agentDeviceSessions: { "not-a-device": {} } };
485 + const exit = yield* Effect.exit(makeDeviceTools(unknown, base.environment));
486 + assert.isTrue(exit._tag === "Failure");
487 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
488 +);
489 +
490 +it.effect(
491 + "answers a shell mock first, then help, then the session error, before the simulator",
492 + () =>
493 + Effect.gen(function* () {
494 + const sample = makeSample(deviceCase({}), base);
495 + yield* withDevices(sample);
496 + const opened = yield* sample.devices!.call("device_open", { hostId: MAC, deviceId: IPHONE });
497 + const { command, targetArgs, session } = json(opened.text).agentDevice;
498 + const flags = session === undefined ? ` ${targetArgs.join(" ")}` : "";
499 + const told = () =>
500 + JSON.stringify(
501 + shellRequest(sample, { tool: "Bash", input: {} }, "claude-haiku-5-5").messages,
502 + );
503 + assert.include(told(), "has not started");
504 + assert.include(cannedOutput(sample, `${command} screenshot /tmp/a.png${flags}`), NO_SESSION);
505 + assert.match(cannedOutput(sample, `${command} help record`) ?? "", /^Usage:/);
506 + assert.isUndefined(cannedOutput(sample, `${command} open com.acme.app${flags}`));
507 + const outage = makeSample(
508 + deviceCase({
509 + shellMocks: [
510 + { match: "\\bagent-device\\b", output: "Exit code 1\nError (COMMAND_FAILED): down" },
511 + ],
512 + }),
513 + base,
514 + );
515 + yield* withDevices(outage);
516 + yield* outage.devices!.call("device_open", { hostId: MAC, deviceId: IPHONE });
517 + assert.include(cannedOutput(outage, `${command} screenshot /tmp/a.png${flags}`), "down");
518 + sample.shell.push({
519 + tool: "Bash",
520 + input: { command: `${command} open com.acme.app${flags}` },
521 + output: "Opened: com.acme.app",
522 + });
523 + assert.isUndefined(cannedOutput(sample, `${command} screenshot /tmp/a.png${flags}`));
524 + assert.include(told(), "is open with an app");
525 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
526 +);
527 +
528 +it("keeps a redirection's descriptor out of the app an open names, on either launcher", () => {
529 + // Sol R10 B1: `open 2>&1` once read as `open 2`, an app the real shell never passes.
530 + const flags = `--platform ios --udid ${IPHONE} --config /home/user/.t3/userdata/device/hosts/x.json --session t3-a`;
531 + for (const [target, args] of [
532 + [shared("t3-a"), ` ${flags}`],
533 + [boundTarget, ""],
534 + ] as const) {
535 + for (const redirect of ["2>&1", "2>&1 | tail -5", "2>/dev/null"]) {
536 + const opened = [
537 + bash(`${target.command} open${args} ${redirect}`, `Opened: ${target.session}`),
538 + ];
539 + assert.deepEqual(
540 + agentDeviceSessions([target], {}, opened).get(target.session),
541 + { app: false },
542 + redirect,
543 + );
544 + assert.equal(
545 + errorCode([target], opened, `${target.command} record start /tmp/run.mp4${args}`),
546 + "UNKNOWN",
547 + redirect,
548 + );
549 + }
550 + }
551 +});
552 +
553 +it("follows opens the parser cannot prove ran, in calls that did not fail", () => {
554 + // Opus R10 B1(b): shapes from run 5, where `&&` before `;` leaves an open uncertain.
555 + const launcher = boundTarget.command;
556 + const after = (command: string, output = "Opened: com.acme.app") =>
557 + agentDeviceSessions([boundTarget], {}, [bash(command, output)]).get("t3-bound");
558 + assert.deepEqual(
559 + after(`mkdir -p /tmp/p && ${launcher} open com.acme.app 2>&1 | tail -3; echo done`),
560 + { app: true },
561 + );
562 + assert.deepEqual(after(`cd /tmp && ${launcher} open com.acme.app; ${launcher} snapshot -i`), {
563 + app: true,
564 + });
565 + // The launcher alias such a call sets, too, here and in later calls.
566 + assert.deepEqual(after(`mkdir -p /tmp/p && AD=${launcher}; $AD open com.acme.app`), {
567 + app: true,
568 + });
569 + assert.deepEqual(after(`mkdir -p /tmp/p && AD="${launcher}"; "\${AD}" open com.acme.app`), {
570 + app: true,
571 + });
572 + assert.deepEqual(
573 + agentDeviceSessions([boundTarget], {}, [
574 + bash(`cd /tmp && AD=${launcher}; echo ready`),
575 + bash("$AD open com.acme.app"),
576 + ]).get("t3-bound"),
577 + { app: true },
578 + );
579 + // A failed call credits only what certainly ran.
580 + assert.isUndefined(
581 + after(`mkdir -p /tmp/p && ${launcher} open com.acme.app; echo done`, "Exit code 1\nno"),
582 + );
583 + assert.isUndefined(
584 + after(`mkdir -p /tmp/p && AD=${launcher}; $AD open com.acme.app`, "Exit code 1\nno"),
585 + );
586 +});
587 +
588 +const SPARE_MAC = "cccccccc-cccc-4ccc-8ccc-cccccccccccc";
589 +const SPARE_PHONE = "DDDDDDDD-DDDD-4DDD-8DDD-DDDDDDDDDDDD";
590 +const twoMacs = {
591 + devices: [
592 + ...world.devices,
593 + { ...world.devices[0]!, hostId: SPARE_MAC, id: SPARE_PHONE, name: "Acme Spare iPhone" },
594 + ],
595 + deviceHosts: { ...world.deviceHosts, [SPARE_MAC]: "spare-mac" },
596 +};
597 +const deviceTools = (extra: object) =>
598 + makeDeviceTools(
599 + { ...twoMacs, ...extra } as Parameters<typeof makeDeviceTools>[0],
600 + base.environment,
601 + );
602 +
603 +it.effect("lists a host whose discovery fails as T3 does: failed, with none of its devices", () =>
604 + Effect.gen(function* () {
605 + // F1's saved device_list: the hub answered, but its device list timed out.
606 + const failure = new DeviceOperationError({
607 + operation: "list",
608 + reason: "request_failed",
609 + cause: null,
610 + });
611 + const tools = yield* deviceTools({ deviceDiscoveryFailures: { [MAC]: failure } });
612 + const listed = json((yield* tools.call("device_list", {})).text);
613 + assert.deepEqual(listed.hostStatuses[MAC], {
614 + status: "failed",
615 + detail:
616 + "Device list failed: Could not communicate with device support. Try refreshing devices.",
617 + });
618 + assert.equal(listed.hostStatuses[SPARE_MAC].status, "ready");
619 + assert.deepEqual(
620 + listed.devices.map((device: { id: string }) => device.id),
621 + [SPARE_PHONE],
622 + );
623 + const opened = yield* tools.call("device_open", { hostId: MAC, deviceId: IPHONE });
624 + assert.isTrue(opened.isError);
625 + assert.include(opened.text, IPHONE);
626 + assert.isFalse(
627 + (yield* tools.call("device_open", { hostId: SPARE_MAC, deviceId: SPARE_PHONE })).isError,
628 + );
629 + // A failed host with no devices of its own still needs its label.
630 + const unlabeled = yield* Effect.exit(
631 + deviceTools({ deviceDiscoveryFailures: { "eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee": failure } }),
632 + );
633 + assert.isTrue(unlabeled._tag === "Failure");
634 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
635 +);
636 +
637 +it.effect("denies every device tool with T3's own error when agent device access is off", () =>
638 + Effect.gen(function* () {
639 + const tools = yield* deviceTools({ deviceAgentAccess: false });
640 + for (const [name, input] of [
641 + ["device_list", {}],
642 + ["device_open", { hostId: MAC, deviceId: IPHONE }],
643 + ["device_screenshot", {}],
644 + ["device_close", {}],
645 + ] as const) {
646 + const result = yield* tools.call(name, input);
647 + assert.isTrue(result.isError, name);
648 + // Refs before typed screenshot errors print their generic screenshot failure instead, and
649 + // later refs may add how to turn access on.
650 + const denied = result.text.startsWith(
651 + "Agent device access is turned off for this environment.",
652 + );
653 + assert.isTrue(
654 + denied || (name === "device_screenshot" && result.text === "Device screenshot failed."),
655 + `${name}: ${result.text}`,
656 + );
657 + }
658 + // T3 drops its device text for a thread without the capability, on refs that have it.
659 + assert.notInclude(t3SystemAppend(model, false), "## T3 Code devices");
660 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
661 +);
662 +
663 +it.effect("reports an unreadable capture as T3 does: a successful 0×0 screenshot", () =>
664 + Effect.gen(function* () {
665 + const tools = yield* deviceTools({ deviceInvalidCapture: true });
666 + yield* tools.call("device_open", { hostId: MAC, deviceId: IPHONE });
667 + const shot = yield* tools.call("device_screenshot", {});
668 + assert.isFalse(shot.isError);
669 + assert.deepEqual(json(shot.text).screenshot, { mimeType: "image/png", width: 0, height: 0 });
670 + const sample = makeSample(
671 + deviceCase({ world: { ...world, deviceInvalidCapture: true } }),
672 + base,
673 + );
674 + yield* withDevices(sample);
675 + const told = JSON.stringify(shellRequest(sample, { tool: "Bash", input: {} }, model).messages);
676 + assert.include(told, "agent-device's own screenshot finds the PNG and is unaffected");
677 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
678 +);
679 +
680 +it.effect("refuses an open on a device another thread's agent-device session holds", () =>
681 + Effect.gen(function* () {
682 + const tools = yield* deviceTools({ deviceOwners: { [IPHONE]: "acme-design-review" } });
683 + // T3 names the owner's session as it names every thread's.
684 + const owner = "t3-a062c547ef0b6bf8ee839b67";
685 + assert.deepEqual(tools.owners, [
686 + { deviceId: IPHONE, threadId: "acme-design-review", session: owner },
687 + ]);
688 + // T3 has no cross-thread gate: device_open succeeds, and device_list shows only this thread.
689 + const opened = json((yield* tools.call("device_open", { hostId: MAC, deviceId: IPHONE })).text);
690 + assert.deepEqual(json((yield* tools.call("device_list", {})).text).open, [
691 + { hostId: MAC, deviceId: IPHONE },
692 + ]);
693 + const { command, targetArgs, session: bound } = opened.agentDevice;
694 + const own: string = bound ?? targetArgs[targetArgs.indexOf("--session") + 1];
695 + const target = (session = own) =>
696 + bound === undefined
697 + ? targetArgs.map((arg: string) => (arg === own ? session : arg)).join(" ")
698 + : session === own
699 + ? ""
700 + : `--session ${session}`;
701 + const line = (args: string, session?: string) => `${command} ${args} ${target(session)}`.trim();
702 + const history: Array<ShellCall> = [];
703 + const answer = (args: string, session?: string) =>
704 + tools.sessionError(history, line(args, session)) ?? "";
705 + const inUse = `Error (DEVICE_IN_USE): Device is already in use by session "${owner}".\nHint: Run agent-device session list to inspect active sessions. To reuse this device, rerun the command with --session ${owner}. To open a new session on this device, first run agent-device close --session ${owner}.`;
706 + assert.include(
707 + answer("open com.acme.app"),
708 + `${inUse} To wait for the device to free up, run the same open with --wait <ms>.\nDiagnostic ID: `,
709 + );
710 + assert.include(
711 + answer("open --wait 5000"),
712 + `Error (DEVICE_IN_USE): Device is already in use by session "${owner}".\nHint: Waited 5000ms for this device and it stayed busy. Run agent-device session list`,
713 + );
714 + assert.notInclude(answer("open --wait 5000"), "--wait <ms>");
715 + // The hint's reuse works, in the other thread's session.
716 + assert.equal(answer("screenshot /tmp/a.png", owner), "");
717 + assert.equal(answer("open com.acme.app", owner), "");
718 + // A failed open starts nothing, in a compound call too.
719 + history.push({
720 + tool: "Bash",
721 + input: { command: `${line("open com.acme.app")} && ${line("screenshot /tmp/a.png")}` },
722 + output: "Exit code 1\nError (DEVICE_IN_USE): Device is already in use",
723 + });
724 + assert.deepEqual(
725 + [...tools.sessions(history)],
726 + [
727 + [own, undefined],
728 + [owner, { app: true }],
729 + ],
730 + );
731 + const sample = makeSample(
732 + deviceCase({ world: { ...twoMacs, deviceOwners: { [IPHONE]: "acme-design-review" } } }),
733 + base,
734 + );
735 + yield* withDevices(sample);
736 + const told = JSON.stringify(shellRequest(sample, { tool: "Bash", input: {} }, model).messages);
737 + assert.include(
738 + told,
739 + `${owner} (Acme iPhone, ${IPHONE}, on build-mac) belongs to another T3 thread (acme-design-review)`,
740 + );
741 + // Closing the owner's session frees the device.
742 + history.push({ tool: "Bash", input: { command: line("close", owner) }, output: "Closed" });
743 + assert.equal(answer("open com.acme.app"), "");
744 + assert.include(describeSessions(tools.sessions(history), tools.owners), "was closed");
745 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
746 +);
747 +
748 +it.effect("leaves a free device free when another thread holds a different one", () =>
749 + Effect.gen(function* () {
750 + const tools = yield* deviceTools({ deviceOwners: { [IPHONE]: "acme-design-review" } });
751 + const opened = json(
752 + (yield* tools.call("device_open", { hostId: SPARE_MAC, deviceId: SPARE_PHONE })).text,
753 + );
754 + const { command, targetArgs, session } = opened.agentDevice;
755 + const flags = session === undefined ? ` ${targetArgs.join(" ")}` : "";
756 + assert.isUndefined(tools.sessionError([], `${command} open com.acme.app${flags}`));
757 + assert.include(
758 + tools.sessionError([], `${command} screenshot /tmp/a.png${flags}`) ?? "",
759 + "SESSION_NOT_FOUND",
760 + );
761 + // The local host always exists and holds none of the world's devices.
762 + assert.isFalse(
763 + (yield* tools.call("device_list", {})).text.includes(`"hostId":"${LOCAL_DEVICE_HOST_ID}"`),
764 + );
765 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
766 +);
767 +
768 +type Opened = { command: string; targetArgs: Array<string>; session?: string };
769 +/** This ref's command line for a launcher: a bound one needs no flags; the shared one takes them. */
770 +const commandLine = (opened: Opened, args: string, session?: string) => {
771 + const own = opened.session ?? opened.targetArgs[opened.targetArgs.indexOf("--session") + 1];
772 + if (opened.session !== undefined)
773 + return `${opened.command} ${args}${session === undefined ? "" : ` --session ${session}`}`;
774 + const flags = opened.targetArgs.map((arg) => (arg === own && session ? session : arg));
775 + return `${opened.command} ${args} ${flags.join(" ")}`;
776 +};
777 +
778 +it.effect(
779 + "tells the simulator each launcher's device, and routes DEVICE_IN_USE to the held one",
780 + () =>
781 + Effect.gen(function* () {
782 + // Opus R11-B3(a): in run 6 the simulator gave the free spare's plain open the owner's
783 + // DEVICE_IN_USE twice. It could not tell which device a flagless bound launcher drives.
784 + const owner = "t3-a062c547ef0b6bf8ee839b67";
785 + const sample = makeSample(
786 + deviceCase({ world: { ...twoMacs, deviceOwners: { [IPHONE]: "acme-design-review" } } }),
787 + base,
788 + );
789 + yield* withDevices(sample);
790 + const tools = sample.devices!;
791 + const open = (hostId: string, deviceId: string) =>
792 + Effect.map(
793 + tools.call("device_open", { hostId, deviceId }),
794 + (result): Opened => json(result.text).agentDevice,
795 + );
796 + const held = yield* open(MAC, IPHONE);
797 + const spare = yield* open(SPARE_MAC, SPARE_PHONE);
798 + const session = (opened: Opened) =>
799 + opened.session ?? opened.targetArgs[opened.targetArgs.indexOf("--session") + 1];
800 + const told = (command: string) =>
801 + String(
802 + shellRequest(sample, { tool: "Bash", input: { command } }, model).messages[0]?.content[0]
803 + ?.text,
804 + );
805 + // Run 6's history: the held device's plain open, refused without the simulator.
806 + const heldOpen = commandLine(held, "open com.acme.app --metro-port 8095");
807 + const refused = cannedOutput(sample, heldOpen) ?? "";
808 + assert.include(
809 + refused,
810 + `Error (DEVICE_IN_USE): Device is already in use by session "${owner}".`,
811 + );
812 + sample.shell.push({ tool: "Bash", input: { command: heldOpen }, output: refused });
813 +
814 + // The spare's plain open goes to the simulator, which now knows it drives the free spare.
815 + const spareOpen = commandLine(spare, "open com.acme.app --metro-port 8095");
816 + assert.isUndefined(cannedOutput(sample, spareOpen));
817 + const free = told(spareOpen);
818 + assert.include(
819 + free,
820 + `agent-device launchers T3's device_open returned: ${
821 + held.session === undefined
822 + ? `${held.command} (shared: each command's --udid or --serial and --session name its device and session)`
823 + : `${held.command} (drives Acme iPhone, ${IPHONE}, on build-mac, in session ${held.session}); ${spare.command} (drives Acme Spare iPhone, ${SPARE_PHONE}, on spare-mac, in session ${spare.session})`
824 + }`,
825 + );
826 + assert.include(
827 + free,
828 + `${session(held)} (Acme iPhone, ${IPHONE}, on build-mac) has not started; ${session(spare)} (Acme Spare iPhone, ${SPARE_PHONE}, on spare-mac) has not started; ${owner} (Acme iPhone, ${IPHONE}, on build-mac) belongs to another T3 thread (acme-design-review) and holds its device with an app open.`,
829 + );
830 + assert.include(
831 + free,
832 + `In the new call, \`${spareOpen}\` drives Acme Spare iPhone, ${SPARE_PHONE}, on spare-mac, which no other session holds: it prints no DEVICE_IN_USE.`,
833 + );
834 + assert.notInclude(free, "Error (DEVICE_IN_USE)");
835 +
836 + // A compound open of the held device gets agent-device's exact refusal, for that open only.
837 + const piped = commandLine(held, "open com.acme.app");
838 + const refusal = told(`${piped} 2>&1 | tail -5`);
839 + assert.include(
840 + refusal,
841 + `In the new call, \`${piped}\` drives Acme iPhone, ${IPHONE}, on build-mac, which session ${owner} holds: it fails with exit code 1, printing "Error (DEVICE_IN_USE): Device is already in use by session "${owner}"." and "Hint: Run agent-device session list to inspect active sessions. To reuse this device,`,
842 + );
843 + assert.notInclude(refusal, "no other session holds");
844 + // One call may try the held device, then fall back to the spare through an alias.
845 + const alias =
846 + spare.session === undefined
847 + ? `A=${spare.command}; "$A" open com.acme.app ${spare.targetArgs.join(" ")}`
848 + : `A=${spare.command}; "$A" open com.acme.app`;
849 + const both = told(`${piped} || echo busy; ${alias}`);
850 + assert.include(both, `which session ${owner} holds: it fails with exit code 1`);
851 + assert.include(
852 + both,
853 + `on spare-mac, which no other session holds: it prints no DEVICE_IN_USE`,
854 + );
855 + // An open in the other thread's own session is not refused, and gets no route.
856 + assert.notInclude(told(commandLine(held, "open com.acme.app", owner)), "In the new call");
857 + // An open after the same call closes the other thread's session finds the device free.
858 + const reopen = told(`${commandLine(held, "close", owner)}; ${piped}`);
859 + assert.include(
860 + reopen,
861 + `\`${piped}\` drives Acme iPhone, ${IPHONE}, on build-mac, which no other session holds: it prints no DEVICE_IN_USE`,
862 + );
863 + assert.notInclude(reopen, `which session ${owner} holds`);
864 + // Neither do commands other than open.
865 + assert.notInclude(
866 + told(`${commandLine(spare, "screenshot /tmp/a.png")} && echo ok`),
867 + "In the new call",
868 + );
869 +
870 + // A world where no other thread holds a device gets no route either.
871 + const alone = makeSample(deviceCase({}), base);
872 + yield* withDevices(alone);
873 + const own: Opened = json(
874 + (yield* alone.devices!.call("device_open", { hostId: MAC, deviceId: IPHONE })).text,
875 + ).agentDevice;
876 + const request = shellRequest(
877 + alone,
878 + { tool: "Bash", input: { command: `${commandLine(own, "open com.acme.app")} | tail -3` } },
879 + model,
880 + );
881 + assert.notInclude(JSON.stringify(request.messages), "In the new call");
882 + assert.include(
883 + JSON.stringify(request.messages),
884 + `${session(own)} (Acme iPhone, ${IPHONE}, on build-mac) has not started`,
885 + );
886 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
887 +);
888 +
889 +it("tells the simulator 0.21.31's --foreground and session list behaviour", () => {
890 + // Opus R11-B3(b): 0.21.31 (session2.js ys) refuses only an open --foreground with no app or URL.
891 + // Run 6's simulator refused `open <app> --foreground` in 9 base samples.
892 + assert.include(
893 + AGENT_DEVICE_SHELL,
894 + "open --foreground with no app or URL fails with exit code 1 when the command passes --udid or --device",
895 + );
896 + assert.include(AGENT_DEVICE_SHELL, "open <app> --foreground works.");
897 + // Run 6's simulator answered the DEVICE_IN_USE hint's `session list` with "Unknown command".
898 + assert.include(
899 + AGENT_DEVICE_SHELL,
900 + "session list prints, as JSON, the started sessions on the launcher's host, other threads' too",
901 + );
902 + // Neither is refused without the simulator.
903 + for (const args of ["open com.acme.app --foreground", "session list"]) {
904 + assert.isUndefined(
905 + agentDeviceSessionError(
906 + [boundTarget],
907 + new Map([["t3-bound", undefined]]),
908 + `${boundTarget.command} ${args}`,
909 + ),
910 + args,
911 + );
912 + }
913 +});
evals/mcp-tool-use/devices.ts new
+952
@@ -0,0 +1,952 @@
1 +// @effect-diagnostics nodeBuiltinImport:off
2 +/**
3 + * T3's device tools over a case's world. McpHttpServer's device toolkit runs T3's handlers on its
4 + * DeviceService, whose hosts are fakes holding the world's devices, so every result and error the
5 + * agent sees is T3's. Launcher files go to a file system that keeps nothing.
6 + */
7 +import * as NodeCrypto from "node:crypto";
8 +import * as NodeFS from "node:fs";
9 +import * as NodePath from "node:path";
10 +import {
11 + DeviceError,
12 + type DeviceHostId,
13 + DeviceOperationError,
14 + type DevicePlatform,
15 + type DeviceSummary,
16 + EnvironmentId,
17 + LOCAL_DEVICE_HOST_ID,
18 + LOCAL_DEVICE_HOST_LABEL,
19 + ProviderInstanceId,
20 + ThreadId,
21 +} from "@t3tools/contracts";
22 +import * as HostProcess from "@t3tools/shared/HostProcess";
23 +import * as Context from "effect/Context";
24 +import * as Crypto from "effect/Crypto";
25 +import * as Effect from "effect/Effect";
26 +import * as FileSystem from "effect/FileSystem";
27 +import * as Layer from "effect/Layer";
28 +import * as Option from "effect/Option";
29 +import * as Path from "effect/Path";
30 +import * as References from "effect/References";
31 +import * as Schema from "effect/Schema";
32 +// oxlint-disable-next-line t3code/no-raw-mcp-registration -- the eval calls the device tools McpHttpServer registered and registers none.
33 +import { McpSchema, McpServer } from "effect/ai";
34 +
35 +import * as ServerConfig from "../../apps/server/src/config.ts";
36 +import * as AgentDeviceShim from "../../apps/server/src/device/AgentDeviceShim.ts";
37 +import {
38 + agentDeviceConfigPath,
39 + agentDeviceSession,
40 +} from "../../apps/server/src/device/AgentDeviceTarget.ts";
41 +import type * as DeviceHost from "../../apps/server/src/device/DeviceHost.ts";
42 +import * as DeviceService from "../../apps/server/src/device/DeviceService.ts";
43 +import * as McpHttpServer from "../../apps/server/src/mcp/McpHttpServer.ts";
44 +import * as McpInvocationContext from "../../apps/server/src/mcp/McpInvocationContext.ts";
45 +import { liveThreadsLayer } from "../../apps/server/src/mcp/McpToolAccess.testkit.ts";
46 +import { DeviceToolkit } from "../../apps/server/src/mcp/toolkits/device/tools.ts";
47 +import * as ProjectService from "../../apps/server/src/project/ProjectService.ts";
48 +import { ServerSettingsService } from "../../apps/server/src/serverSettings.ts";
49 +import { shellInvocations } from "./device-grading.ts";
50 +
51 +export interface DeviceWorld {
52 + readonly devices: ReadonlyArray<DeviceSummary>;
53 + /** Labels of the SSH device hosts, by id. The local host always exists. */
54 + readonly deviceHosts?: Readonly<Record<string, string>> | undefined;
55 + /**
56 + * agent-device sessions that prior turns started, by device id, with the app `open` put in
57 + * each. Otherwise a device's session starts at its launcher's first `open` in the sample.
58 + */
59 + readonly agentDeviceSessions?:
60 + | Readonly<Record<string, { readonly app?: string | undefined }>>
61 + | undefined;
62 + /**
63 + * Hosts whose hub fails to list devices, by id, with the error it raises. On every call T3 marks
64 + * each one failed and lists none of its devices.
65 + */
66 + readonly deviceDiscoveryFailures?: Readonly<Record<string, DeviceOperationError>> | undefined;
67 + /** false: agent device access is off, so the thread holds no `device` capability. */
68 + readonly deviceAgentAccess?: boolean | undefined;
69 + /** Every capture reaches T3 with adb screencap's multi-display warning ahead of the PNG. */
70 + readonly deviceInvalidCapture?: boolean | undefined;
71 + /** Devices another thread's agent-device session holds, by device id: that thread's id. */
72 + readonly deviceOwners?: Readonly<Record<string, string>> | undefined;
73 +}
74 +
75 +/** A shell, file or web call and the output the agent saw. */
76 +export interface ShellCall {
77 + readonly tool: string;
78 + readonly input: unknown;
79 + readonly output: string;
80 +}
81 +
82 +/** A started agent-device session: whether `open <app>` put an app in it, or only a recording holds it. */
83 +export interface AgentDeviceSession {
84 + readonly app: boolean;
85 + readonly recordingOnly?: boolean;
86 +}
87 +
88 +/** Another thread's agent-device session, named as T3 names it, holding a device with its app open. */
89 +export interface AgentDeviceOwner {
90 + readonly deviceId: string;
91 + readonly threadId: string;
92 + readonly session: string;
93 +}
94 +
95 +/** One sample's device tools, answered by T3. */
96 +export interface DeviceTools {
97 + /** Runs one call as T3 does. A `failure` is raised in T3's DeviceService during the call. */
98 + readonly call: (
99 + name: string,
100 + input: unknown,
101 + failure?: DeviceError,
102 + ) => Effect.Effect<{ readonly text: string; readonly isError: boolean }>;
103 + /** The `agent-device` commands `device_open` returned, in order. */
104 + readonly launchers: ReadonlyArray<string>;
105 + /** Sessions other threads hold on the world's devices. */
106 + readonly owners: ReadonlyArray<AgentDeviceOwner>;
107 + /** Each session the launchers name after these calls, by name; undefined until it starts. */
108 + readonly sessions: (
109 + history: ReadonlyArray<ShellCall>,
110 + ) => ReadonlyMap<string, AgentDeviceSession | undefined>;
111 + /** agent-device's error for a plain launcher command that needs a session it has not started. */
112 + readonly sessionError: (history: ReadonlyArray<ShellCall>, command: string) => string | undefined;
113 + /**
114 + * What the shell simulator is told of the launchers, their devices and sessions, and of the
115 + * launcher opens in a new call that another thread's session refuses.
116 + */
117 + readonly describe: (
118 + history: ReadonlyArray<ShellCall>,
119 + command: string | undefined,
120 + ) => ReadonlyArray<string>;
121 +}
122 +
123 +export const isDeviceTool = (name: string) => Object.hasOwn(DeviceToolkit.tools, name);
124 +
125 +const HELP = NodeFS.readFileSync(
126 + NodePath.join(import.meta.dirname, "agent-device-help.txt"),
127 + "utf8",
128 +);
129 +/** agent-device's help by topic, "" for `help` itself. */
130 +const HELP_TOPICS = new Map(
131 + HELP.split(/^\$ agent-device help ?/m)
132 + .slice(1)
133 + .map((section) => [
134 + section.slice(0, section.indexOf("\n")),
135 + section.slice(section.indexOf("\n") + 1).trimEnd(),
136 + ]),
137 +);
138 +const TARGET_FLAGS = new Set([
139 + "--platform",
140 + "--device",
141 + "--udid",
142 + "--serial",
143 + "--session",
144 + "--config",
145 +]);
146 +const isTargetFlag = (word = "") => TARGET_FLAGS.has(word.split("=")[0] ?? "");
147 +
148 +/**
149 + * What one plain command prints when it asks a returned launcher for help on a topic
150 + * agent-device-help.txt has, such as `<launcher> help record` or `<launcher> record --help`.
151 + */
152 +export const launcherHelp = (launchers: ReadonlyArray<string>, command: string) => {
153 + const plain = command.trim().replace(/\s+2>&1$/, "");
154 + if (/[\n;&|<>`$()"'\\]/.test(plain)) return undefined;
155 + const [executable = "", ...words] = plain.split(/\s+/);
156 + if (!launchers.includes(executable)) return undefined;
157 + // Target flags and their values pick a device; they do not change the help.
158 + const asked = words.filter(
159 + (word, i) =>
160 + !isTargetFlag(word) && !(isTargetFlag(words[i - 1]) && !words[i - 1]?.includes("=")),
161 + );
162 + const topic =
163 + asked[0] === "help"
164 + ? asked.slice(1)
165 + : ["--help", "-h"].includes(asked.at(-1) ?? "")
166 + ? asked.slice(0, -1)
167 + : undefined;
168 + return topic !== undefined && topic.length <= 1 ? HELP_TOPICS.get(topic[0] ?? "") : undefined;
169 +};
170 +
171 +/**
172 + * What the shell simulator knows of the launchers T3's device_open returns: agent-device's own
173 + * help. It is the same on every ref; only the launcher paths in a sample differ.
174 + */
175 +export const AGENT_DEVICE_SHELL = [
176 + 'The description lists the agent-device launchers T3 Code\'s device_open returned. Each is a file on the agent\'s machine, not on SSH hosts. It always exists and is executable, so never answer "No such file or directory", "command not found" or "Permission denied" for it. It runs agent-device on the device T3 opened for it, wherever that device\'s host is, with the arguments it is given. Answer it as agent-device does, from agent-device\'s help below:',
177 + '- `help` and `--help` print the text under `$ agent-device help` exactly and in full, never shortened. `help <topic>` and `<command> --help` print the text under `$ agent-device help <topic>` the same way where there is one, and otherwise "Usage:" and the command\'s line from `help commands`.',
178 + '- Only the commands and flags in this help exist. Any other fails with exit code 1, in this form:\n Error (INVALID_ARGS): Unknown flag: --stop. Did you mean --scope?\n Hint: Check command arguments and run --help for usage examples.\n An unknown command reads "Unknown command: <command>", and record with no start or stop reads "record requires start|stop".',
179 + '- A launcher command runs in an agent-device session: the one its --session flag names, or else the one the launcher\'s directory is named after. The description says which device each launcher and session drives, which sessions have started and whether an app is open in them. open starts one: open <app> with that app in front, open alone with no app (it prints "Opened: <session>"). close ends it. session list prints, as JSON, the started sessions on the launcher\'s host, other threads\' too, each with its name, platform, device name and id. Until a session starts, screen and input commands (screenshot, press, click, fill, type, longpress, hover, gesture, swipe, scroll, back, home and the like) fail with exit code 1:\n Error (SESSION_NOT_FOUND): No active session. Run open first.\n Hint: Run open first or pass an explicit device selector.\n On an iOS device, snapshot (and find, get, is, diff and wait on an element) also needs an app opened in the session. Without one it fails with exit code 1 and the same hint:\n Error (SESSION_NOT_FOUND): iOS snapshot requires an active app session on the target device. Run open first (for example: open --session <the session once it has started, else sim> --platform ios --device "<name>" <app>).\n On Android, snapshot needs no session and starts one without an app. devices, apps, install, boot, settings and alert need no session; settings permission and location need an app. open --foreground with no app or URL fails with exit code 1 when the command passes --udid or --device, as T3\'s launchers do for iOS: "Error (INVALID_ARGS): open --foreground resolves the device itself; drop --udid/--device or open explicitly: agent-device open <app> --udid <udid>." open <app> --foreground works.',
180 + '- record start without --scope device records the app open in the session. Before the session starts it fails with "Error (INVALID_ARGS): record start with app scope and an explicit session requires an active app session; run open <app> first, or use --scope device to record the full screen" and the hint above. In a session open with no app it fails with "Error (UNKNOWN): App-scoped recording requires an active app session; open the app under test before recording." and "Hint: Unexpected internal error. Retry with --debug and report the diagnostics log if it persists." --scope device records the whole screen and needs no session.',
181 + "- Otherwise a command in this help acts on the launcher's device and succeeds unless the description says otherwise.",
182 + "",
183 + HELP,
184 +].join("\n");
185 +
186 +const FLAGS = new Set(HELP.match(/--[a-z][a-z\d-]*/g));
187 +/** Flags agent-device's help shows taking a value, such as `--scope <app|device|system>`. */
188 +const VALUE_FLAGS = new Set(
189 + [...HELP.matchAll(/(--[a-z][a-z\d-]*)(?: <| [a-z]+\|)/g)].map((match) => match[1]),
190 +);
191 +const SELECTOR_FLAGS = ["--platform", "--target", "--device", "--udid", "--serial"];
192 +/**
193 + * Commands agent-device 0.21.31's daemon routes to its `generic` and `interaction` handlers, which
194 + * look up the named session first and fail with SESSION_NOT_FOUND without it. `get` and `is`
195 + * check their arguments first, so they stay with the simulator.
196 + */
197 +const NEEDS_SESSION =
198 + /^(?:action-button|app-switcher|back|focus|fold|hardware-button|home|orientation|screenshot|scroll|telephony|tv-remote|viewport|click|fill|gesture|hover|longpress|press|swipe|type)$/;
199 +/** A shell call failed, or agent-device printed its failure envelope. */
200 +const FAILED = /^(?:Exit code [1-9]|<tool_use_error>)|\bError \([A-Z_]+\):|"success"\s*:\s*false/;
201 +
202 +/** A launcher `device_open` returned, and the agent-device session its target flags name. */
203 +export interface LauncherTarget {
204 + readonly command: string;
205 + readonly session: string;
206 + /** A bound launcher passes its session itself; the shared one gets it from each command. */
207 + readonly bound: boolean;
208 + readonly deviceId: string;
209 + /** agent-device refuses an iOS device's snapshot without an app in the session. */
210 + readonly ios: boolean;
211 + /** agent-device checks for an app before an app-scoped recording only on iOS Simulators. */
212 + readonly iosSimulator: boolean;
213 +}
214 +
215 +/** One launcher command line: its agent-device command and arguments, and the session it names. */
216 +const launcherCall = (targets: ReadonlyArray<LauncherTarget>, words: ReadonlyArray<string>) => {
217 + const mine = targets.filter((target) => target.command === words[0]);
218 + if (mine.length === 0) return undefined;
219 + const flags = new Map<string, string>();
220 + const positionals: Array<string> = [];
221 + // The shell splits an unquoted variable holding flags, such as `$AD open app $A`.
222 + const args = words
223 + .slice(1)
224 + .flatMap((word) => (word.startsWith("-") && /\s/.test(word) ? word.trim().split(/\s+/) : word));
225 + for (let i = 0; i < args.length; i++) {
226 + const word = args[i]!;
227 + if (word === "--") {
228 + positionals.push(...args.slice(i + 1));
229 + break;
230 + }
231 + const at = word.indexOf("=");
232 + // An array the parser cannot read, such as `"${A[@]}"`, holds flags, not an app.
233 + if (/^\$\{\w+\[[@*]\]\}$/.test(word)) continue;
234 + if (!word.startsWith("-")) positionals.push(word);
235 + else if (at > 0) flags.set(word.slice(0, at), word.slice(at + 1));
236 + else flags.set(word, VALUE_FLAGS.has(word) ? (args[++i] ?? "") : "");
237 + }
238 + const named = flags.get("--session");
239 + const target = mine.findLast((candidate) => candidate.session === named) ?? mine.at(-1)!;
240 + // Flags this parser cannot read, such as a bash array, most likely are device_open's flags for
241 + // the only session the shared launcher has served.
242 + const unread =
243 + named === undefined && !target.bound && new Set(mine.map((t) => t.session)).size === 1;
244 + const [command = "", ...rest] = positionals;
245 + return {
246 + target,
247 + command,
248 + args: rest,
249 + flags,
250 + deviceId: flags.get("--udid") ?? flags.get("--serial") ?? target.deviceId,
251 + // The session agent-device sees. The shared launcher refuses a command without --session
252 + // itself; that is not modelled.
253 + session: named ?? (target.bound ? target.session : undefined),
254 + // The session the call most likely ran in, for following sessions through the history.
255 + tracked: named ?? (target.bound || unread ? target.session : undefined),
256 + selector: target.bound || unread || SELECTOR_FLAGS.some((flag) => flags.has(flag)),
257 + appScope: (flags.get("--scope") ?? "app") === "app",
258 + };
259 +};
260 +type LauncherCall = NonNullable<ReturnType<typeof launcherCall>>;
261 +
262 +/**
263 + * A session after one successful command. agent-device 0.21.31 publishes the named session on
264 + * `open`, on an Android `snapshot` with a device selector, and for a whole-screen recording, which
265 + * `record stop` retires. `close` ends it. An iOS snapshot never starts one: without an app it fails.
266 + */
267 +const afterCall = (call: LauncherCall, session: AgentDeviceSession | undefined) => {
268 + switch (call.command) {
269 + case "open":
270 + return { app: call.args.length > 0 || (session?.app ?? false) };
271 + case "snapshot":
272 + return session ?? (call.selector && !call.target.ios ? { app: false } : undefined);
273 + case "close":
274 + return undefined;
275 + case "record":
276 + if (call.args[0] === "start" && !session && !call.appScope && call.selector)
277 + return { app: false, recordingOnly: true };
278 + return call.args[0] === "stop" && session?.recordingOnly ? undefined : session;
279 + default:
280 + return session;
281 + }
282 +};
283 +
284 +/**
285 + * The other session that holds the device an `open` would start a session on. agent-device 0.21.31
286 + * refuses that open, as it does an open on any device another session in its daemon holds.
287 + */
288 +const holder = (
289 + call: LauncherCall,
290 + session: string | undefined,
291 + sessions: ReadonlyMap<string, AgentDeviceSession | undefined>,
292 + owners: ReadonlyArray<AgentDeviceOwner>,
293 +) =>
294 + call.command === "open" && session !== undefined && sessions.get(session) === undefined
295 + ? owners.find(
296 + (owner) =>
297 + owner.deviceId === call.deviceId &&
298 + owner.session !== session &&
299 + sessions.get(owner.session) !== undefined,
300 + )
301 + : undefined;
302 +
303 +/**
304 + * A launcher alias a call sets, such as `AD=<launcher>`. Grading drops an assignment it cannot prove
305 + * ran, as in `mkdir -p d && AD=<launcher>; $AD open …`, but a call that did not fail ran it.
306 + */
307 +const launcherAliases = (command: string, targets: ReadonlyArray<LauncherTarget>) =>
308 + [...command.matchAll(/(?<![\w$])(\w+)=(["']?)([^\s;&|"'()]+)\2(?=[\s;&|)]|$)/g)].flatMap(
309 + ([, name = "", , value = ""]) =>
310 + targets.some((target) => target.command === value) ? [[name, value] as const] : [],
311 + );
312 +const withAliases = (command: string, aliases: ReadonlyArray<readonly [string, string]>) =>
313 + aliases.reduce(
314 + (text, [name, value]) =>
315 + text.replace(new RegExp(`"?\\$(?:\\{${name}\\}|${name}(?!\\w))"?`, "g"), value),
316 + command,
317 + );
318 +
319 +/** The sessions and variables after these calls. Aliases may span calls, as in grading. */
320 +const followSessions = (
321 + targets: ReadonlyArray<LauncherTarget>,
322 + declared: DeviceWorld["agentDeviceSessions"],
323 + history: ReadonlyArray<ShellCall>,
324 + owners: ReadonlyArray<AgentDeviceOwner>,
325 +) => {
326 + const sessions = new Map<string, AgentDeviceSession | undefined>();
327 + for (const target of targets) {
328 + const prior = declared?.[target.deviceId];
329 + sessions.set(target.session, prior && { app: prior.app !== undefined });
330 + }
331 + for (const owner of owners) sessions.set(owner.session, { app: true });
332 + const variables = new Map<string, string>();
333 + for (const { tool, input, output } of history) {
334 + const command =
335 + tool === "Bash" && typeof input === "object" && input !== null && "command" in input
336 + ? input.command
337 + : undefined;
338 + if (typeof command !== "string") continue;
339 + const failed = FAILED.test(output);
340 + const aliases = failed ? [] : launcherAliases(command, targets);
341 + const actions = shellInvocations(withAliases(command, aliases), variables, false, !failed);
342 + for (const [name, value] of aliases) variables.set(name, value);
343 + // A lone command's failure is its own. A compound call's output may not say which command
344 + // failed, so its certain launcher commands count, as the simulator likely ran them. A call that
345 + // did not fail ran the ones the parser could not prove ran, such as `mkdir -p d && $L open; …`.
346 + if (actions.length === 1 && failed) continue;
347 + for (const action of actions) {
348 + const call =
349 + (action.executed || !failed) && !action.remote
350 + ? launcherCall(targets, action.words)
351 + : undefined;
352 + if (call?.tracked !== undefined && !holder(call, call.tracked, sessions, owners))
353 + sessions.set(call.tracked, afterCall(call, sessions.get(call.tracked)));
354 + }
355 + }
356 + return { sessions, variables };
357 +};
358 +
359 +/** Each session the launchers name after these calls. */
360 +export const agentDeviceSessions = (
361 + targets: ReadonlyArray<LauncherTarget>,
362 + declared: DeviceWorld["agentDeviceSessions"],
363 + history: ReadonlyArray<ShellCall>,
364 + owners: ReadonlyArray<AgentDeviceOwner> = [],
365 +) => followSessions(targets, declared, history, owners).sessions;
366 +
367 +/**
368 + * Each launcher `open` a new call makes, read as the tracker reads calls, with the other thread's
369 + * session that holds its device and whether another session holds it at all. agent-device refuses
370 + * an open that has a holder with DEVICE_IN_USE. Each launcher command runs in order, so an earlier
371 + * `close --session <owner>` in the same call frees the device for a later open.
372 + */
373 +const opensIn = (
374 + targets: ReadonlyArray<LauncherTarget>,
375 + { sessions, variables }: ReturnType<typeof followSessions>,
376 + command: string,
377 + owners: ReadonlyArray<AgentDeviceOwner>,
378 +) => {
379 + const current = new Map(sessions);
380 + return shellInvocations(
381 + withAliases(command, launcherAliases(command, targets)),
382 + new Map(variables),
383 + false,
384 + true,
385 + ).flatMap((action) => {
386 + const call = action.remote ? undefined : launcherCall(targets, action.words);
387 + if (call?.tracked === undefined) return [];
388 + const owner = holder(call, call.tracked, current, owners);
389 + const held = owners.some(
390 + (candidate) =>
391 + candidate.deviceId === call.deviceId && current.get(candidate.session) !== undefined,
392 + );
393 + if (owner === undefined) current.set(call.tracked, afterCall(call, current.get(call.tracked)));
394 + return call.command === "open" ? [{ call, words: action.words, owner, held }] : [];
395 + });
396 +};
397 +
398 +/**
399 + * What the shell simulator is told of the sessions the launchers name, and other threads', with
400 + * the device each drives where it is known.
401 + */
402 +export const describeSessions = (
403 + sessions: ReadonlyMap<string, AgentDeviceSession | undefined>,
404 + owners: ReadonlyArray<AgentDeviceOwner> = [],
405 + devices: ReadonlyMap<string, string> = new Map(),
406 +) =>
407 + [...sessions]
408 + .map(([name, session]) => {
409 + const named = devices.has(name) ? `${name} (${devices.get(name)})` : name;
410 + const owner = owners.find((candidate) => candidate.session === name);
411 + if (owner !== undefined)
412 + return session === undefined
413 + ? `${named} belonged to another T3 thread (${owner.threadId}) and was closed, which freed its device`
414 + : `${named} belongs to another T3 thread (${owner.threadId}) and holds its device with an app open`;
415 + return session === undefined
416 + ? `${named} has not started`
417 + : session.recordingOnly
418 + ? `${named} holds only a --scope device recording`
419 + : session.app
420 + ? `${named} is open with an app`
421 + : `${named} is open with no app`;
422 + })
423 + .join("; ");
424 +
425 +/** agent-device's human-readable error, with the diagnostics lines its remote client adds. */
426 +const agentDeviceError = (
427 + home: string,
428 + session: string,
429 + seed: string,
430 + error: { readonly code: string; readonly message: string; readonly hint: string },
431 +) => {
432 + const hash = NodeCrypto.createHash("sha256").update(seed).digest();
433 + const id = `${hash.readUIntBE(0, 5).toString(36).padStart(8, "0")}-${hash.toString("hex", 5, 9)}`;
434 + return [
435 + "Exit code 1",
436 + `Error (${error.code}): ${error.message}`,
437 + `Hint: ${error.hint}`,
438 + `Diagnostic ID: ${id}`,
439 + `Diagnostics Log: ${home}/.agent-device/remote-diagnostics/${session}/requests/${hash.toString("hex", 9, 17)}.ndjson`,
440 + ].join("\n");
441 +};
442 +const NO_SESSION_HINT = "Run open first or pass an explicit device selector.";
443 +/** agent-device's hint when another session holds the device; after --wait it says it waited. */
444 +const inUseHint = (owner: string, waitedMs?: number) =>
445 + [
446 + ...(waitedMs === undefined
447 + ? []
448 + : [`Waited ${Math.round(waitedMs)}ms for this device and it stayed busy.`]),
449 + "Run agent-device session list to inspect active sessions.",
450 + `To reuse this device, rerun the command with --session ${owner}.`,
451 + `To open a new session on this device, first run agent-device close --session ${owner}.`,
452 + ...(waitedMs === undefined
453 + ? ["To wait for the device to free up, run the same open with --wait <ms>."]
454 + : []),
455 + ].join(" ");
456 +/** agent-device's refusal of an open on a device another session holds, unless --wait is bad. */
457 +const inUseError = (call: LauncherCall, owner: AgentDeviceOwner) => {
458 + const wait = call.flags.get("--wait");
459 + return wait === undefined || /^\d+$/.test(wait)
460 + ? {
461 + code: "DEVICE_IN_USE",
462 + message: `Device is already in use by session "${owner.session}".`,
463 + hint: inUseHint(owner.session, wait === undefined ? undefined : Number(wait)),
464 + }
465 + : undefined;
466 +};
467 +
468 +/**
469 + * agent-device 0.21.31's error for a plain launcher command that needs its session before
470 + * anything starts the session, as recorded on a real Mac. Both refs name the session (the bound
471 + * launcher itself, the shared one through the --session flag device_open returned) and the device.
472 + */
473 +export const agentDeviceSessionError = (
474 + targets: ReadonlyArray<LauncherTarget>,
475 + sessions: ReadonlyMap<string, AgentDeviceSession | undefined>,
476 + command: string,
477 + owners: ReadonlyArray<AgentDeviceOwner> = [],
478 +) => {
479 + const plain = command.trim().replace(/\s+2>&1$/, "");
480 + if (/[\n;&|<>`$()\\]/.test(plain)) return undefined;
481 + const [action, ...others] = shellInvocations(plain);
482 + const call = action && others.length === 0 ? launcherCall(targets, action.words) : undefined;
483 + // Unknown flags and --json or debug output stay with the simulator.
484 + const flags = [...(call?.flags.keys() ?? [])].filter((flag) => flag.startsWith("--"));
485 + if (call?.session === undefined || flags.some((flag) => !FLAGS.has(flag))) return undefined;
486 + if (flags.some((flag) => ["--json", "--debug", "--verbose"].includes(flag))) return undefined;
487 + const session = sessions.get(call.session);
488 + if (call.command === "record" && call.args[0] === "start" && call.appScope) {
489 + if (session === undefined && call.selector)
490 + return {
491 + session: call.session,
492 + code: "INVALID_ARGS",
493 + message:
494 + "record start with app scope and an explicit session requires an active app session; run open <app> first, or use --scope device to record the full screen",
495 + hint: "Check command arguments and run --help for usage examples.",
496 + };
497 + if (session && !session.app && !session.recordingOnly && call.target.iosSimulator)
498 + return {
499 + session: call.session,
500 + code: "UNKNOWN",
501 + message:
502 + "App-scoped recording requires an active app session; open the app under test before recording.",
503 + hint: "Unexpected internal error. Retry with --debug and report the diagnostics log if it persists.",
504 + };
505 + return undefined;
506 + }
507 + const owner = holder(call, call.session, sessions, owners);
508 + const inUse = owner && inUseError(call, owner);
509 + if (inUse !== undefined) return { session: call.session, ...inUse };
510 + if (call.command === "snapshot" && call.target.ios && !session?.app)
511 + return {
512 + session: call.session,
513 + code: "SESSION_NOT_FOUND",
514 + message: `iOS snapshot requires an active app session on the target device. Run open first (for example: open --session ${session === undefined ? "sim" : call.session} --platform ios --device "<name>" <app>).`,
515 + hint: NO_SESSION_HINT,
516 + };
517 + if (session !== undefined) return undefined;
518 + if (NEEDS_SESSION.test(call.command))
519 + return {
520 + session: call.session,
521 + code: "SESSION_NOT_FOUND",
522 + message: "No active session. Run open first.",
523 + hint: NO_SESSION_HINT,
524 + };
525 + if (call.command === "close" && call.args.length === 0 && call.selector)
526 + return {
527 + session: call.session,
528 + code: "SESSION_NOT_FOUND",
529 + message: "No active session",
530 + hint: NO_SESSION_HINT,
531 + };
532 + return undefined;
533 +};
534 +
535 +const decodeDeviceError = Schema.decodeUnknownOption(DeviceError);
536 +/** A device tool mock that is a T3 device error, such as `{"_tag": "DeviceOperationError", ...}`. */
537 +export const deviceFailure = (mock: unknown) => Option.getOrUndefined(decodeDeviceError(mock));
538 +
539 +const Injected = Context.Reference<DeviceError | undefined>(
540 + "@t3tools/evals/mcp-tool-use/devices/Injected",
541 + { defaultValue: () => undefined },
542 +);
543 +const failing = <A, R>(effect: Effect.Effect<A, DeviceError, R>) =>
544 + Effect.flatMap(Injected, (failure) => (failure === undefined ? effect : Effect.fail(failure)));
545 +
546 +// The hosts' own output, which T3 passes on. Launcher scripts embed these paths, never shown.
547 +const NODE = "/usr/bin/node";
548 +const HUB = "http://127.0.0.1:47613";
549 +const PLATFORM_NAMES = { ios: "iOS Simulators", android: "Android Emulators" } as const;
550 +/** T3 reads only a capture's PNG header; replay has no screens to send. */
551 +const capture = (platform: DevicePlatform) => {
552 + const png = new Uint8Array(24);
553 + const view = new DataView(png.buffer);
554 + view.setUint32(0, 0x89504e47);
555 + view.setUint32(4, 0x0d0a1a0a);
556 + view.setUint32(12, 0x49484452);
557 + view.setUint32(16, platform === "ios" ? 1206 : 1080);
558 + view.setUint32(20, platform === "ios" ? 2622 : 2400);
559 + return png;
560 +};
561 +/**
562 + * What `adb exec-out screencap -p` prints before the PNG on an emulator with several displays,
563 + * such as a foldable. T3 passes the hub's bytes on, and its PNG check finds no header.
564 + */
565 +const SCREENCAP_WARNING = new TextEncoder().encode(
566 + "[Warning] Multiple displays were found, but no display id was specified! Defaulting to the first display found, however this default is not guaranteed to be consistent across captures. A display id should be specified.\n",
567 +);
568 +
569 +const fakeHost = (
570 + world: DeviceWorld,
571 + id: DeviceHostId,
572 + label: string,
573 + agentEntry: string,
574 +): DeviceHost.DeviceHost["Service"] => {
575 + const devices = world.devices.filter((device) => device.hostId === id);
576 + const discovery = world.deviceDiscoveryFailures?.[id];
577 + const booted = new Map(devices.map((device) => [device.id, device.booted]));
578 + const availability = (platform: DevicePlatform) =>
579 + devices.some((device) => device.platform === platform)
580 + ? { platform, available: true }
581 + : { platform, available: false, reason: `${label} has no ${PLATFORM_NAMES[platform]}.` };
582 + const ready: DeviceHost.DeviceHostReady = {
583 + nodePath: NODE,
584 + hub: { origin: HUB },
585 + devices: {
586 + // Boot state changes as T3 boots and shuts devices down, and later lists show it.
587 + list:
588 + discovery === undefined
589 + ? Effect.sync(() => ({
590 + devices: devices.map(({ hostId: _hostId, ...device }) => ({
591 + ...device,
592 + booted: booted.get(device.id) ?? device.booted,
593 + })),
594 + detail: undefined,
595 + }))
596 + : Effect.fail(discovery),
597 + boot: (device) =>
598 + Effect.sync(() => {
599 + booted.set(device.id, true);
600 + return device.id;
601 + }),
602 + attach: () => Effect.void,
603 + shutdown: (device) => Effect.sync(() => void booted.set(device.id, false)),
604 + screenshot: (device) =>
605 + Effect.succeed(
606 + world.deviceInvalidCapture
607 + ? new Uint8Array([...SCREENCAP_WARNING, ...capture(device.platform)])
608 + : capture(device.platform),
609 + ),
610 + },
611 + // T3 checks an Android emulator's host for ffmpeg's H.264 encoder before streaming it.
612 + run: () => Effect.succeed({ stdout: " V....D libx264 H.264\n", stderr: "", code: 0 }),
613 + helpers: { serveSimAxSettings: null, serveSimCli: null },
614 + };
615 + const agentDevice = {
616 + baseUrl: `${HUB}/api/device-hub/agent/${id}`,
617 + token: id,
618 + entryPath: agentEntry,
619 + };
620 + return {
621 + id,
622 + label,
623 + summary: Effect.sync(() => ({
624 + id,
625 + label,
626 + kind: id === LOCAL_DEVICE_HOST_ID ? ("local" as const) : ("ssh" as const),
627 + platforms: [availability("ios"), availability("android")],
628 + hubInstalled: true,
629 + agentDeviceInstalled: true,
630 + })),
631 + platformAvailability: (platform) => Effect.sync(() => availability(platform)),
632 + ensureReady: () => Effect.succeed(ready),
633 + ensureAgentReady: () => Effect.succeed({ ...ready, agentDevice }),
634 + current: Effect.succeed(ready),
635 + stopAgent: Effect.void,
636 + stop: Effect.void,
637 + };
638 +};
639 +
640 +// Refs before and after T3 binds a launcher to each session run this same code. Optional
641 +// exports detect the newer API; makeWithHosts ignores a fifth argument it does not take.
642 +type BoundTarget = { readonly session: string; readonly targetArgs: ReadonlyArray<string> };
643 +type Launchers = {
644 + readonly bind: (input: BoundTarget) => Effect.Effect<string, DeviceError>;
645 + readonly prune: (live: ReadonlySet<string>) => Effect.Effect<void>;
646 +};
647 +const launcherApi: {
648 + readonly ensureAgentDeviceShim: typeof AgentDeviceShim.ensureAgentDeviceShim;
649 + readonly bindAgentDeviceLauncher?: (
650 + input: Parameters<typeof AgentDeviceShim.ensureAgentDeviceShim>[0] & BoundTarget,
651 + ) => ReturnType<typeof AgentDeviceShim.ensureAgentDeviceShim>;
652 + readonly pruneAgentDeviceLaunchers?: (input: {
653 + readonly stateDir: string;
654 + readonly live: ReadonlySet<string>;
655 + }) => Effect.Effect<void, never, FileSystem.FileSystem | Path.Path>;
656 +} = AgentDeviceShim;
657 +type MakeWithHosts = typeof DeviceService.makeWithHosts;
658 +const makeWithHosts: (
659 + hosts: Parameters<MakeWithHosts>[0],
660 + testHost: Parameters<MakeWithHosts>[1],
661 + configureAgent: Parameters<MakeWithHosts>[2],
662 + installTool: Parameters<MakeWithHosts>[3],
663 + launchers?: Launchers,
664 +) => ReturnType<MakeWithHosts> = DeviceService.makeWithHosts;
665 +
666 +/** T3's device tools for one sample. `cwd` names the home that T3's state lives under. */
667 +export const makeDeviceTools = Effect.fn("evaluateMcpToolUse.makeDeviceTools")(function* (
668 + world: DeviceWorld,
669 + environment: { readonly cwd: string; readonly platform: string },
670 +) {
671 + const home = /^\/(?:home|Users)\/[^/]+/.exec(environment.cwd)?.[0] ?? "/home/user";
672 + for (const id of Object.keys(world.agentDeviceSessions ?? {})) {
673 + if (!world.devices.some((device) => device.id === id)) {
674 + return yield* Effect.die(new Error(`world.agentDeviceSessions names no device ${id}.`));
675 + }
676 + }
677 + // T3 names each thread's session on a device; another thread's is named the same way.
678 + const owners: Array<AgentDeviceOwner> = [];
679 + for (const [deviceId, threadId] of Object.entries(world.deviceOwners ?? {})) {
680 + const device = world.devices.find((candidate) => candidate.id === deviceId);
681 + if (device === undefined) {
682 + return yield* Effect.die(new Error(`world.deviceOwners names no device ${deviceId}.`));
683 + }
684 + const session = yield* agentDeviceSession(threadId, device.hostId, deviceId);
685 + owners.push({ deviceId, threadId, session });
686 + }
687 + const config = ServerConfig.layerTest(environment.cwd, `${home}/.t3`);
688 + const files = FileSystem.layerNoop({
689 + makeDirectory: () => Effect.void,
690 + writeFileString: () => Effect.void,
691 + chmod: () => Effect.void,
692 + });
693 + const devices = Layer.effect(
694 + DeviceService.DeviceService,
695 + Effect.gen(function* () {
696 + const fs = yield* FileSystem.FileSystem;
697 + const path = yield* Path.Path;
698 + const crypto = yield* Crypto.Crypto;
699 + const { stateDir } = yield* ServerConfig.ServerConfig;
700 + const agentEntry = path.join(home, ".t3", "tools", "agent-device", "bin", "agent-device.mjs");
701 + const hosts = new Map<DeviceHostId, DeviceHost.DeviceHost["Service"]>([
702 + [
703 + LOCAL_DEVICE_HOST_ID,
704 + fakeHost(world, LOCAL_DEVICE_HOST_ID, LOCAL_DEVICE_HOST_LABEL, agentEntry),
705 + ],
706 + ]);
707 + // Hosts with devices, then configured hosts that may have none, such as a failed one.
708 + for (const id of [
709 + ...world.devices.map((device) => device.hostId),
710 + ...Object.keys(world.deviceHosts ?? {}),
711 + ...Object.keys(world.deviceDiscoveryFailures ?? {}),
712 + ]) {
713 + const label = world.deviceHosts?.[id];
714 + if (hosts.has(id)) continue;
715 + if (label === undefined) {
716 + return yield* Effect.die(new Error(`world.deviceHosts has no label for ${id}.`));
717 + }
718 + hosts.set(id, fakeHost(world, id, label, agentEntry));
719 + }
720 + // As DeviceService.make, minus writing the config file the path names.
721 + const configureAgent = (hostId: DeviceHostId, _endpoint: unknown, deviceId?: string) =>
722 + agentDeviceConfigPath(stateDir, hostId, path, deviceId).pipe(
723 + Effect.provideService(Crypto.Crypto, crypto),
724 + Effect.mapError(
725 + (cause) =>
726 + new DeviceOperationError({
727 + operation: "configure agent",
728 + reason: "settings_failed",
729 + cause,
730 + }),
731 + ),
732 + );
733 + const withFiles = <A, E, R>(effect: Effect.Effect<A, E, R>) =>
734 + effect.pipe(
735 + Effect.provideService(FileSystem.FileSystem, fs),
736 + Effect.provideService(Path.Path, path),
737 + );
738 + const { bindAgentDeviceLauncher: bind, pruneAgentDeviceLaunchers: prune } = launcherApi;
739 + const launchers: Launchers | undefined =
740 + bind && prune
741 + ? {
742 + bind: (input) =>
743 + AgentDeviceShim.ensureAgentDeviceShim({ entryPath: agentEntry, stateDir }).pipe(
744 + Effect.andThen(bind({ entryPath: agentEntry, stateDir, ...input })),
745 + withFiles,
746 + Effect.mapError(
747 + (cause) =>
748 + new DeviceOperationError({
749 + operation: "bind agent launcher",
750 + reason: "command_failed",
751 + cause,
752 + }),
753 + ),
754 + ),
755 + prune: (live) => withFiles(prune({ stateDir, live })),
756 + }
757 + : undefined;
758 + const service = yield* makeWithHosts(hosts, undefined, configureAgent, undefined, launchers);
759 + return DeviceService.DeviceService.of({
760 + ...service,
761 + // Older refs install the CLI through agentCli; makeWithHosts has none to install.
762 + ...("agentCli" in service ? { agentCli: Effect.succeed(agentEntry) } : {}),
763 + list: failing(service.list),
764 + open: Effect.fn("DeviceService.DeviceService.open")((input, context) =>
765 + failing(service.open(input, context)),
766 + ),
767 + screenshot: Effect.fn("DeviceService.DeviceService.screenshot")((input) =>
768 + failing(service.screenshot(input)),
769 + ),
770 + close: Effect.fn("DeviceService.DeviceService.close")((input) =>
771 + failing(service.close(input)),
772 + ),
773 + });
774 + }),
775 + );
776 + const context = yield* Layer.build(
777 + McpHttpServer.layerDeviceToolkit.pipe(
778 + Layer.provideMerge(McpServer.McpServer.layer),
779 + Layer.provideMerge(devices),
780 + Layer.provideMerge(
781 + Layer.mergeAll(
782 + liveThreadsLayer,
783 + Layer.mock(ProjectService.ProjectService)({ getById: () => Effect.succeedNone }),
784 + ServerSettingsService.layerTest({
785 + enableDeviceSupport: true,
786 + enableAgentDeviceAccess: world.deviceAgentAccess !== false,
787 + deviceOnboardingCompleted: true,
788 + }),
789 + ),
790 + ),
791 + Layer.provideMerge(config),
792 + Layer.provideMerge(files),
793 + ),
794 + );
795 + type Entry = Parameters<McpServer.McpServer["Service"]["addTool"]>[0];
796 + // The registry keeps each tool's handler; its public type lists only definitions.
797 + const tools = Context.get(context, McpServer.McpServer).tools as ReadonlyArray<Entry>;
798 + // The testkit's live thread runs on the `codex` instance; the caller must match it.
799 + const scope: McpInvocationContext.McpThreadInvocationScope = {
800 + environmentId: EnvironmentId.make("00000000-0000-4000-8000-0000000000e0"),
801 + requestNamespace: "mcp-tool-use-eval",
802 + thread: {
803 + threadId: ThreadId.make("mcp-tool-use-eval"),
804 + providerSessionId: "mcp-tool-use-eval",
805 + providerInstanceId: ProviderInstanceId.make("codex"),
806 + },
807 + client: undefined,
808 + // A thread holds `device` when agent device access was on as its session started.
809 + capabilities: new Set<McpInvocationContext.McpCapability>(
810 + world.deviceAgentAccess === false ? [] : ["device"],
811 + ),
812 + issuedAt: 0,
813 + };
814 + const request = McpSchema.McpRequestContext.of({
815 + clientId: 0,
816 + protocolVersion: "2025-06-18",
817 + clientCapabilities: {},
818 + });
819 + const launchers: Array<string> = [];
820 + const targets: Array<LauncherTarget> = [];
821 + const call = Effect.fn("evaluateMcpToolUse.deviceCall")(function* (
822 + name: string,
823 + input: unknown,
824 + failure?: DeviceError,
825 + ) {
826 + const entry = tools.find((tool) => tool.tool.name === name);
827 + if (entry === undefined) return yield* Effect.die(new Error(`T3 has no device tool ${name}.`));
828 + const result = yield* entry.handle(input).pipe(
829 + Effect.provideService(McpInvocationContext.McpInvocationContext, scope),
830 + Effect.provideService(McpSchema.McpRequestContext, request),
831 + Effect.provideService(HostProcess.ExecutablePath, NODE),
832 + Effect.provideService(HostProcess.Platform, environment.platform as NodeJS.Platform),
833 + Effect.provideService(Injected, failure),
834 + // T3 logs each failed screenshot; the result already says so.
835 + Effect.provideService(References.MinimumLogLevel, "Error"),
836 + // An MCP client shows a protocol error, such as bad parameters, as the tool's error.
837 + Effect.catch((error) =>
838 + Effect.succeed(
839 + new McpSchema.CallToolResult({
840 + isError: true,
841 + content: [{ type: "text", text: error.message }],
842 + }),
843 + ),
844 + ),
845 + );
846 + if (!("content" in result)) return yield* Effect.die(new Error(`${name} asked for input.`));
847 + const text = result.content
848 + .flatMap((part) => (part.type === "text" ? [part.text] : []))
849 + .join("\n");
850 + // McpServer turns a defect into this text; here a defect is a bug in the eval's world.
851 + if (text === "Tool execution failed due to an internal server error.") {
852 + return yield* Effect.die(new Error(`${name} hit a defect in T3 or the eval's world.`));
853 + }
854 + const opened = result.structuredContent;
855 + if (name === "device_open" && !result.isError && isOpened(opened)) {
856 + const { command, targetArgs = [], session: bound } = opened.agentDevice;
857 + launchers.push(command);
858 + // Both refs return the session in the target flags; a bound launcher also names it.
859 + const flag = targetArgs.indexOf("--session");
860 + const session = bound ?? (flag < 0 ? undefined : targetArgs[flag + 1]);
861 + const { device } = opened;
862 + if (session !== undefined && device !== undefined)
863 + targets.push({
864 + command,
865 + session,
866 + bound: bound !== undefined,
867 + deviceId: device.id,
868 + ios: device.platform === "ios",
869 + iosSimulator: device.platform === "ios" && !device.physical,
870 + });
871 + }
872 + return { text, isError: result.isError === true };
873 + });
874 + const sessions = (history: ReadonlyArray<ShellCall>) =>
875 + agentDeviceSessions(targets, world.agentDeviceSessions, history, owners);
876 + /** A device as the simulator knows it from the world: its name, id and host. */
877 + const deviceName = (deviceId: string) => {
878 + const device = world.devices.find((candidate) => candidate.id === deviceId);
879 + if (device === undefined) return deviceId;
880 + const host =
881 + device.hostId === LOCAL_DEVICE_HOST_ID
882 + ? LOCAL_DEVICE_HOST_LABEL
883 + : (world.deviceHosts?.[device.hostId] ?? device.hostId);
884 + return `${device.name}, ${device.id}, on ${host}`;
885 + };
886 + const launcherText = (command: string) => {
887 + const target = targets.findLast((candidate) => candidate.command === command);
888 + if (target === undefined) return command;
889 + return target.bound
890 + ? `${command} (drives ${deviceName(target.deviceId)}, in session ${target.session})`
891 + : `${command} (shared: each command's --udid or --serial and --session name its device and session)`;
892 + };
893 + const describe = (history: ReadonlyArray<ShellCall>, command: string | undefined) => {
894 + const followed = followSessions(targets, world.agentDeviceSessions, history, owners);
895 + const sessionDevices = new Map([
896 + ...targets.map((target) => [target.session, deviceName(target.deviceId)] as const),
897 + ...owners.map((owner) => [owner.session, deviceName(owner.deviceId)] as const),
898 + ]);
899 + // Code, not the simulator, decides which of the call's opens another thread's session refuses.
900 + const opens =
901 + command === undefined || owners.length === 0
902 + ? []
903 + : opensIn(targets, followed, command, owners).flatMap(({ call, words, owner, held }) => {
904 + const opened = `\`${words.join(" ")}\` drives ${deviceName(call.deviceId)}`;
905 + const error = owner && inUseError(call, owner);
906 + if (owner !== undefined && error !== undefined)
907 + return [
908 + `${opened}, which session ${owner.session} holds: it fails with exit code 1, printing "Error (${error.code}): ${error.message}" and "Hint: ${error.hint}"`,
909 + ];
910 + return held
911 + ? []
912 + : [`${opened}, which no other session holds: it prints no DEVICE_IN_USE`];
913 + });
914 + return [
915 + ...(launchers.length > 0
916 + ? [
917 + `agent-device launchers T3's device_open returned: ${[...new Set(launchers)].map(launcherText).join("; ")}`,
918 + ]
919 + : []),
920 + ...(followed.sessions.size > 0
921 + ? [
922 + `Their agent-device sessions: ${describeSessions(followed.sessions, owners, sessionDevices)}.`,
923 + ]
924 + : []),
925 + ...(opens.length > 0 ? [`In the new call, ${[...new Set(opens)].join("; ")}.`] : []),
926 + ];
927 + };
928 + return {
929 + call,
930 + launchers,
931 + owners,
932 + sessions,
933 + sessionError: (history, command) => {
934 + const error = agentDeviceSessionError(targets, sessions(history), command, owners);
935 + return error && agentDeviceError(home, error.session, `${history.length}\n${command}`, error);
936 + },
937 + describe,
938 + } satisfies DeviceTools;
939 +});
940 +
941 +const isOpened = Schema.is(
942 + Schema.Struct({
943 + device: Schema.optional(
944 + Schema.Struct({ id: Schema.String, platform: Schema.String, physical: Schema.Boolean }),
945 + ),
946 + agentDevice: Schema.Struct({
947 + command: Schema.String,
948 + targetArgs: Schema.optional(Schema.Array(Schema.String)),
949 + session: Schema.optional(Schema.String),
950 + }),
951 + }),
952 +);
evals/mcp-tool-use/run.test.ts
+2890 -1
@@ -1,3 +1,8 @@
1 +// @effect-diagnostics nodeBuiltinImport:off
2 +import * as NodeFS from "node:fs";
3 +import * as NodeServices from "@effect/platform-node/NodeServices";
4 +import * as Effect from "effect/Effect";
5 +import * as Schema from "effect/Schema";
6 import { assert, expect, it } from "@effect/vitest";
7 import { ComposerContextId, EnvironmentId, ThreadId } from "@t3tools/contracts";
8 import { projectComposerContextForProvider } from "@t3tools/shared/composerContextReferences";
@@ -10,15 +15,19 @@ import {
15 gradeSample,
16 makeSample,
17 ranCalls,
13 - type McpToolUseCase,
18 + McpToolUseCase,
19 mockOutput,
20 normalizeToolInput,
21 normalizeToolName,
22 type SampleRun,
23 shellOutput,
24 shellRequest,
25 + summarize,
26 t3SystemAppend,
27 + t3SubagentContext,
28 + withDevices,
29 } from "./run.ts";
30 +import { briefRoute, gradeDevice, shellInvocations } from "./device-grading.ts";
31 import { taskPrompt } from "../../apps/server/src/mcp/delegatedTaskModes.ts";
32
33 it("grades Codex shell and subagent calls with the shared rules", () => {
@@ -412,3 +421,2883 @@ it("stops a sample only once a must-not check has failed", () => {
421 assert.isTrue(failedEarly(expect, run("watch_pull_request")));
422 assert.isTrue(failedEarly({ textMustNot: ["paste"] }, run("Bash", "paste it here")));
423 });
424 +
425 +const decodeDeviceCase = Schema.decodeSync(Schema.fromJsonString(McpToolUseCase));
426 +const deviceCases = () =>
427 + NodeFS.readFileSync(new URL("./cases/device-manager.ndjson", import.meta.url), "utf8")
428 + .trim()
429 + .split("\n")
430 + .map((line) => decodeDeviceCase(line));
431 +
432 +it.effect("builds device cases from product-backed worlds and synthetic prior turns", () =>
433 + Effect.gen(function* () {
434 + const cases = deviceCases();
435 + assert.equal(new Set(cases.map((c) => c.id)).size, 16);
436 + const base = buildBase(
437 + {
438 + ...template,
439 + tools: [...template.tools, { name: "Read", input_schema: { type: "object" } }],
440 + },
441 + { model: "claude-sonnet-5-5", keepUserInstructions: false },
442 + );
443 + for (const testCase of cases) {
444 + const sample = makeSample(testCase, base);
445 + yield* withDevices(sample);
446 + assert.doesNotThrow(() => buildMessages(sample));
447 + assert.isDefined(testCase.setup.world?.devices);
448 + assert.isUndefined(testCase.source?.openOn);
449 + // device_open starts no agent-device session. FAIL-AFTER-hub's earlier Settings check
450 + // would have opened the app, so its world starts that session. OWN seeds another owner.
451 + assert.deepEqual(
452 + [...(sample.devices?.sessions(sample.shell).values() ?? [])],
453 + ["FAIL-AFTER-hub", "OWN-01", "OWN-02", "OWN-03"].includes(testCase.id)
454 + ? [{ app: true }]
455 + : [],
456 + testCase.id,
457 + );
458 + }
459 + const native = cases.find((c) => c.id === "NAT-01")!;
460 + assert.equal(native.kind, "bad");
461 + assert.equal(native.setup.harness, "claude");
462 + assert.isNotEmpty(cases.find((c) => c.id === "SEED-01")!.setup.priorTurns!);
463 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
464 +);
465 +
466 +it.each([
467 + ["Do not use SSH for UI but use device_open and its returned command.", "managed"],
468 + ["Use device_open instead of SSH for screenshots.", "managed"],
469 + [
470 + "Use device_open for UI. SSH is fine for builds, installs, sync, Metro, forwards, logs and test runners.",
471 + "managed",
472 + ],
473 + ["Use SSH for builds, installs, logs and locks.", "neutral"],
474 + [
475 + "Use device_open and its returned launcher. Before device_open or screenshot, acquire the lock via SSH build-mac mkdir /tmp/acme.lock.",
476 + "managed",
477 + ],
478 + [
479 + "Use device_open. Release the lock (ssh build-mac rm /tmp/acme.lock) as soon as you save the screenshot.",
480 + "managed",
481 + ],
482 + [
483 + "Use device_open. xcrun simctl is allowed only for builds/logs/ports, not UI driving.",
484 + "managed",
485 + ],
486 + ["Use device_open. Use SSH to capture UI logs and run UI test runners.", "managed"],
487 + ["Use device_open. Use SSH to record logs.", "managed"],
488 + ["Use device_open. Use SSH to inspect the screen and save screenshots.", "mixed"],
489 + ["Use device_open. SSH is prohibited for screenshots.", "managed"],
490 + ["Use device_open. SSH is disallowed for UI.", "managed"],
491 + ["Never use device_open. Use SSH for screenshots.", "raw"],
492 + ["Avoid SSH for screenshots, and use device_open.", "managed"],
493 + ["Use device_open. Do not take screenshots over SSH.", "managed"],
494 + ["Use device_open. SSH must not be used for screenshots.", "managed"],
495 + ["Use device_open. Use SSH to inspect the UI and save screenshots.", "mixed"],
496 + ["Use device_open. SSH can be used to take screenshots.", "mixed"],
497 + ["Use device_open. Only use ssh for builds, screenshots and recordings.", "mixed"],
498 + ["Use device_open. Use SSH for builds and to inspect the simulator UI.", "mixed"],
499 + ["Use device_open. For screenshots, use SSH.", "mixed"],
500 + ["Use device_open. Use lab-shot and lab-ui over SSH.", "mixed"],
501 + ["Use device_open. Never use SSH for builds; use SSH to record the demo.", "mixed"],
502 + ["Use device_open. Do not avoid SSH for screenshots.", "mixed"],
503 + ["Use device_open. Use ssh mac 'xcrun simctl io A screenshot /tmp/proof.png'.", "mixed"],
504 + // Live GPT briefs: "non-UI" and "without" restrict SSH rather than route UI through it.
505 + ["Use device_open. SSH is allowed only for the non-UI operations described above.", "managed"],
506 + ["Use device_open. Screenshots via SSH are forbidden.", "managed"],
507 + ["Use device_open. SSH screenshots and recordings are forbidden.", "managed"],
508 + ["Use device_open. Taking screenshots over SSH is not allowed.", "managed"],
509 + ["Using device_open is forbidden, use SSH for screenshots.", "raw"],
510 + [
511 + "Use device_open. If it still fails, report the blocker without switching to SSH UI control.",
512 + "managed",
513 + ],
514 + ["Use device_open. Use SSH for the UI, without asking first.", "mixed"],
515 +])("classifies child routing: %s", (text, route) => {
516 + assert.equal(briefRoute(text), route);
517 +});
518 +
519 +it("rejects a bypass mutation in every delegated or launched child brief", () => {
520 + const good = ["delegate_task", "delegate_task", "t3_thread_launch"].map((tool, step) => ({
521 + step,
522 + tool,
523 + input: {
524 + [tool === "delegate_task" ? "task" : "message"]:
525 + "Use device_open and its returned launcher for UI; SSH builds and locks are allowed.",
526 + },
527 + }));
528 + assert.isTrue(gradeDevice({ briefs: 3 }, { calls: good }).every((c) => c.pass));
529 + for (let i = 0; i < good.length; i++) {
530 + const calls = good.map((call, at) =>
531 + at === i
532 + ? {
533 + ...call,
534 + input: {
535 + [call.tool === "delegate_task" ? "task" : "message"]:
536 + "Use device_open. Use SSH to inspect the UI and save screenshots.",
537 + },
538 + }
539 + : call,
540 + );
541 + assert.isTrue(
542 + gradeDevice({ briefs: 3 }, { calls }).some((c) => !c.pass),
543 + `brief ${i}`,
544 + );
545 + }
546 + assert.isTrue(gradeDevice({ briefs: 3 }, { calls: good.slice(0, 2) }).some((c) => !c.pass));
547 + assert.isTrue(
548 + gradeDevice(
549 + { briefs: 3 },
550 + { calls: [...good, { step: 4, tool: "delegate_task", input: {} }] },
551 + ).some((c) => !c.pass),
552 + );
553 +});
554 +
555 +it.each([
556 + "ssh build-mac xcrun simctl install PRIVATE /tmp/Acme.app",
557 + "ssh build-mac xcrun simctl addmedia PRIVATE /tmp/photo.png",
558 + "ssh build-mac xcrun simctl listapps PRIVATE",
559 + "ssh build-mac xcodebuild -scheme AcmeApp build",
560 + "rsync -a /home/user/acme/app/ build-mac:/Users/user/acme/app/",
561 + "ssh -L 8095:localhost:8095 build-mac 'cat /tmp/metro.log'",
562 + "ssh build-mac 'xcrun simctl spawn PRIVATE log stream'",
563 + "ssh build-mac 'pnpm test'",
564 + "ssh build-mac xcodebuild build && /home/user/.t3/device/bin/agent-device snapshot -i",
565 + "ssh build-mac 'xcodebuild build' && /home/user/.t3/device/bin/agent-device screenshot /tmp/proof.png",
566 + "cat /home/user/uiux-lab/LAB.md",
567 + "lab-ui --help",
568 + "xcrun simctl io PRIVATE screenshot --help",
569 + "printf '%s' 'ssh build-mac xcrun simctl io PRIVATE screenshot /tmp/proof.png'",
570 + "nohup pnpm metro --port 8095 > /tmp/metro.log 2>&1 &",
571 + "cat > /tmp/notes.md <<'EOF'\nxcrun simctl io PRIVATE screenshot /tmp/proof.png\nEOF",
572 + "ssh build-mac 'python3 -' <<'PY'\nimport subprocess\nsubprocess.run(['xcrun', 'simctl', 'install', 'PRIVATE', '/tmp/Acme.app'])\nPY",
573 + "for s in AcmeApp AcmeWidget; do ssh build-mac xcodebuild -scheme $s build; done",
574 + "while ! ssh build-mac test -f /tmp/ready; do sleep 1; done",
575 + "if true; then echo ok; elif true; then xcrun simctl io PRIVATE screenshot /tmp/a.png; fi",
576 + "# shot() { xcrun simctl io PRIVATE screenshot /tmp/a.png; }\nshot",
577 + "echo ready # shot() { xcrun simctl io PRIVATE screenshot /tmp/a.png; }\nshot",
578 + "exit 0; xcrun simctl io PRIVATE screenshot /tmp/a.png",
579 + "f() { f; }; f",
580 + "ssh build-mac agent-device --version",
581 + "ssh build-mac agent-device help longpress",
582 +])("allows auxiliary shell work: %s", (command) => {
583 + const calls = [{ step: 0, tool: "Bash", input: { command } }];
584 + assert.isTrue(gradeDevice({ noRawUi: true }, { calls }).every((c) => c.pass));
585 +});
586 +
587 +it.each([
588 + "ssh build-mac xcrun simctl io PRIVATE screenshot /tmp/proof.png",
589 + "ssh -oBatchMode=yes build-mac 'idb ui tap 20 30'",
590 + "ssh build-mac screencapture /tmp/proof.png",
591 + "ssh build-mac agent-device screenshot /tmp/proof.png",
592 + "ssh build-mac 'xcodebuild build && agent-device screenshot /tmp/proof.png'",
593 + "lab-shot /tmp/proof.png",
594 + "lab-ui tap Settings",
595 + "lab-launch acme://settings",
596 + "xcrun simctl openurl PRIVATE acme://settings",
597 + "xcrun simctl io PRIVATE recordVideo /tmp/demo.mp4",
598 + "idb --udid PRIVATE screenshot /tmp/proof.png",
599 + "adb -s emulator-5554 shell input tap 20 30",
600 + 'bash -lc "ssh build-mac xcrun simctl io PRIVATE screenshot /tmp/proof.png"',
601 + // Live smoke samples: a backgrounded recorder, and one started from a remote Python script.
602 + "ssh build-mac 'rm -f /tmp/raw.mov; nohup xcrun simctl io PRIVATE recordVideo --codec=h264 /tmp/raw.mov >/tmp/rec.log 2>&1 & echo $! > /tmp/rec.pid'",
603 + "timeout 30 xcrun simctl io PRIVATE screenshot /tmp/proof.png",
604 + "ssh build-mac 'python3 -' <<'PY'\nimport subprocess\nproc = subprocess.Popen(['xcrun', 'simctl', 'io', 'PRIVATE', 'recordVideo', '/tmp/raw.mp4'])\nPY",
605 + "ssh build-mac 'bash -s' <<'EOF'\ncd /tmp\nxcrun simctl io PRIVATE screenshot /tmp/proof.png\nEOF",
606 + `python3 -c "import subprocess; subprocess.run(['xcrun', 'simctl', 'openurl', 'PRIVATE', 'acme://settings'])"`,
607 + // Loops, elif and `!` run the commands they introduce.
608 + "for f in a b; do xcrun simctl io PRIVATE screenshot /tmp/$f.png; done",
609 + "while true; do ssh build-mac xcrun simctl io PRIVATE screenshot /tmp/p.png; sleep 1; done",
610 + "until ssh build-mac 'xcrun simctl io PRIVATE screenshot /tmp/p.png'; do sleep 1; done",
611 + "! xcrun simctl io PRIVATE screenshot /tmp/proof.png",
612 + "if false; then :; elif true; then xcrun simctl openurl PRIVATE acme://settings; fi",
613 + "for f in $(xcrun simctl io PRIVATE screenshot /tmp/p.png); do echo $f; done",
614 + // Every agent-device command over ssh bypasses T3, including ones outside a fixed verb list.
615 + "ssh build-mac agent-device longpress @e1",
616 + "ssh build-mac /opt/homebrew/bin/agent-device batch --file /tmp/steps.json",
617 + "if test -f /tmp/x; then echo ok; elif true; then xcrun simctl io PRIVATE screenshot /tmp/a.png; fi",
618 + "(exit 0); xcrun simctl io PRIVATE screenshot /tmp/a.png",
619 + "shot() { xcrun simctl io PRIVATE screenshot /tmp/a.png; }; verify() { shot; }; verify",
620 + "ssh build-mac 'bash -se' <<'EOF'\nxcrun simctl io PRIVATE screenshot /tmp/a.png\nEOF",
621 + // An uncertain short-circuit, and line continuations.
622 + "test -f /tmp/flag && true || xcrun simctl io PRIVATE screenshot /tmp/a.png",
623 + "ssh build-mac xcrun simctl io PRIVATE \\\n screenshot /tmp/a.png",
624 + 'ssh build-mac "xcrun simctl io PRIVATE \\\n screenshot /tmp/a.png"',
625 +])("rejects simulator UI shell mutations: %s", (command) => {
626 + const calls = [{ step: 0, tool: "Bash", input: { command } }];
627 + assert.isTrue(gradeDevice({ noRawUi: true }, { calls }).some((c) => !c.pass));
628 +});
629 +
630 +it("shares failed device attempts across tools and launcher steps, resetting only per user turn", () => {
631 + const failure = (step: number, turn: number, tool: string) => ({
632 + step,
633 + turn,
634 + tool,
635 + failed: true,
636 + input:
637 + tool === "Bash"
638 + ? { command: "/home/user/.t3/device/bin/agent-device snapshot -i --session t3-eval" }
639 + : {},
640 + });
641 + const calls = [
642 + failure(0, 0, "Bash"),
643 + failure(3, 0, "device_screenshot"),
644 + failure(5, 1, "device_open"),
645 + failure(7, 1, "Bash"),
646 + ];
647 + assert.isTrue(gradeDevice({ maxFailedAttemptsPerTurn: 2 }, { calls }).every((c) => c.pass));
648 + assert.isTrue(
649 + gradeDevice(
650 + { maxFailedAttemptsPerTurn: 2 },
651 + { calls: [...calls, failure(10, 1, "device_screenshot")] },
652 + ).some((c) => !c.pass),
653 + );
654 + assert.isTrue(
655 + gradeDevice(
656 + { maxFailedAttemptsPerTurn: 2 },
657 + { calls: calls.map(({ turn: _turn, ...call }) => call) },
658 + ).some((c) => !c.pass),
659 + );
660 + const auxiliary = [
661 + ...calls,
662 + {
663 + step: 11,
664 + turn: 1,
665 + tool: "Bash",
666 + input: { command: "ssh build-mac xcodebuild build" },
667 + failed: true,
668 + },
669 + ];
670 + assert.isTrue(
671 + gradeDevice({ maxFailedAttemptsPerTurn: 2 }, { calls: auxiliary }).every((c) => c.pass),
672 + );
673 + assert.isTrue(
674 + gradeDevice(
675 + { maxFailedAttemptsPerTurn: 2 },
676 + { calls: [...calls, { ...failure(12, 1, "device_screenshot"), failed: false }] },
677 + ).every((c) => c.pass),
678 + );
679 +});
680 +
681 +it("applies device shell checks to decoded cases and stops raw UI early", () => {
682 + for (const testCase of deviceCases()) {
683 + const run = {
684 + calls: [
685 + {
686 + step: 0,
687 + tool: "Bash",
688 + input: { command: "ssh build-mac xcrun simctl io PRIVATE screenshot /tmp/proof.png" },
689 + },
690 + ],
691 + text: "",
692 + stopReason: "tool_use",
693 + };
694 + assert.isTrue(
695 + gradeSample(testCase.expect, run).some((c) => c.check.startsWith("shellMustNot") && !c.pass),
696 + testCase.id,
697 + );
698 + assert.isTrue(failedEarly(testCase.expect, run), testCase.id);
699 + }
700 +});
701 +
702 +it("keeps the shared recovery budget when a case is decoded and graded", () => {
703 + const testCase = deviceCases().find((c) => c.id === "FAIL-AFTER-hub")!;
704 + const calls = ["Bash", "device_screenshot", "device_open"].map((tool, step) => ({
705 + step,
706 + tool,
707 + failed: true,
708 + input: tool === "Bash" ? { command: "/home/user/.t3/device/bin/agent-device snapshot -i" } : {},
709 + }));
710 + const run = { calls, text: "Canvas is blocked: daemon unavailable.", stopReason: "end_turn" };
711 + assert.isTrue(
712 + gradeSample(testCase.expect, run).some((c) => c.check.startsWith("maxCalls device") && !c.pass),
713 + );
714 +});
715 +
716 +it.each([
717 + ["recording", ["start", "stop"], false, true],
718 + ["missing stop", ["start"], false, false],
719 + ["reversed", ["stop", "start"], false, false],
720 + ["failed stop", ["start", "stop"], true, false],
721 + ["ceremonial launcher", ["echo start", "echo stop"], false, false],
722 + ["launcher prefix", ["suffix start", "suffix stop"], false, false],
723 + // Live Claude samples keep the returned command in a variable: `AD=…; $AD record start`.
724 + ["launcher variable", ["var start", "var stop"], false, true],
725 + ["other variable", ["other start", "other stop"], false, false],
726 +] as const)(
727 + "requires executed managed recording start and stop: %s",
728 + (_, operations, failed, pass) => {
729 + const launcher = "/home/user/.t3/device/bin/agent-device";
730 + const command = (operation: string) => {
731 + const [form, step] = operation.split(" ");
732 + if (form === "echo") return `echo '${launcher} record ${step}'`;
733 + if (form === "suffix") return `${launcher}-unbound record ${step}`;
734 + if (form === "var") return `AD=${launcher}; "$AD" record ${step} /tmp/demo/voice.mp4`;
735 + if (form === "other") return `AD=${launcher}-unbound; $AD record ${step} /tmp/demo/voice.mp4`;
736 + return `${launcher} record ${operation} /tmp/demo/voice.mp4 --session t3-eval`;
737 + };
738 + const calls = operations.map((operation, step) => ({
739 + step,
740 + tool: "Bash",
741 + failed: failed && step === 1,
742 + input: { command: command(operation) },
743 + }));
744 + const run = {
745 + calls,
746 + launchers: [launcher],
747 + text: "",
748 + finalText: "Saved /tmp/demo/voice.mp4",
749 + stopReason: "end_turn",
750 + };
751 + const checks = gradeSample({ device: { recording: "/tmp/demo/voice.mp4" } }, run);
752 + assert.equal(checks.find((check) => check.check.startsWith("device recording"))?.pass, pass);
753 + },
754 +);
755 +
756 +it("gives native children hook context while excluding the parent's T3 append and history", async () => {
757 + const testCase = deviceCases().find((c) => c.id === "NAT-01")!;
758 + assert.isEmpty(testCase.setup.priorTurns ?? []);
759 + const child = buildBase(template, {
760 + model: "claude-sonnet-5-5",
761 + keepUserInstructions: false,
762 + nativeSubagent: true,
763 + });
764 + assert.notInclude(
765 + child.system.map((block) => block.text).join("\n"),
766 + t3SystemAppend("claude-sonnet-5-5"),
767 + );
768 + const subagentContext = await t3SubagentContext("claude-sonnet-5-5");
769 + const messages = buildMessages(makeSample(testCase, { ...child, subagentContext }));
770 + const shown = messages
771 + .flatMap((message) => message.content)
772 + .map((block) => block.text ?? "")
773 + .join("\n");
774 + if (subagentContext) assert.include(shown, subagentContext);
775 + else assert.notInclude(shown, "SubagentStart hook additional context:");
776 + assert.include(shown, testCase.prompt);
777 +});
778 +
779 +// Round-5 adversarial review counterexamples.
780 +it.each([
781 + ["Use device_open. Use ssh build-mac for screenshots.", "mixed"],
782 + ["Use device_open. SSH handles screenshots and recordings.", "mixed"],
783 + ["Use device_open. Take screenshots by SSH.", "mixed"],
784 + ["Use device_open. SSH to build-mac and capture screenshots.", "mixed"],
785 + ["Use device_open. SSH should not be avoided for screenshots.", "mixed"],
786 + ["Use device_open. There is no need to avoid SSH for screenshots.", "mixed"],
787 + [
788 + "Use device_open. SSH is for builds, while screenshots go through the returned launcher.",
789 + "managed",
790 + ],
791 + [
792 + "Use device_open. SSH for builds and logs only, screenshots via the returned launcher.",
793 + "managed",
794 + ],
795 + ["Use device_open. For screenshots, ssh into build-mac and run simctl.", "mixed"],
796 + ["Use device_open. Use ssh to copy the saved screenshots back.", "managed"],
797 + ["Use device_open. Copy saved artifacts over ssh.", "managed"],
798 +])("attributes route purposes and negation: %s", (brief, route) => {
799 + assert.equal(briefRoute(brief), route);
800 +});
801 +
802 +it.each([
803 + ["nohup-live", "ssh build-mac 'nohup xcrun simctl io PRIVATE recordVideo /tmp/raw.mov'", true],
804 + [
805 + "python-live",
806 + "ssh build-mac 'python3 -' <<'PY'\nimport subprocess\nsubprocess.Popen(['xcrun', 'simctl', 'io', 'PRIVATE', 'recordVideo', '/tmp/raw.mov'])\nPY",
807 + true,
808 + ],
809 + ["env-flag", "ssh build-mac 'env -i xcrun simctl io PRIVATE screenshot /tmp/proof.png'", true],
810 + ["sudo-flag", "ssh build-mac 'sudo -n xcrun simctl io PRIVATE screenshot /tmp/proof.png'", true],
811 + [
812 + "caffeinate-flag",
813 + "ssh build-mac 'caffeinate -i xcrun simctl io PRIVATE recordVideo /tmp/raw.mov'",
814 + true,
815 + ],
816 + [
817 + "timeout-option-arg",
818 + "ssh build-mac 'timeout -s INT 30 xcrun simctl io PRIVATE screenshot /tmp/proof.png'",
819 + true,
820 + ],
821 + [
822 + "absolute-nohup",
823 + "ssh build-mac '/usr/bin/nohup xcrun simctl io PRIVATE recordVideo /tmp/raw.mov'",
824 + true,
825 + ],
826 + [
827 + "python-data",
828 + "python3 - <<'PY'\nprint('xcrun simctl io PRIVATE screenshot /tmp/proof.png')\nPY",
829 + false,
830 + ],
831 + [
832 + "python-comment",
833 + "python3 - <<'PY'\n# xcrun simctl io PRIVATE screenshot /tmp/proof.png\nprint('hello')\nPY",
834 + false,
835 + ],
836 + [
837 + "python-help",
838 + "python3 - <<'PY'\nimport subprocess\nsubprocess.run(['xcrun', 'simctl', 'io', 'PRIVATE', 'screenshot', '--help'])\nPY",
839 + false,
840 + ],
841 + [
842 + "remote-python-agent-device",
843 + "ssh build-mac 'python3 -' <<'PY'\nimport subprocess\nsubprocess.run(['agent-device', 'screenshot', '/tmp/proof.png'])\nPY",
844 + true,
845 + ],
846 + [
847 + "python-idb-flag",
848 + "ssh build-mac 'python3 -' <<'PY'\nimport subprocess\nsubprocess.run(['idb', '--udid', 'PRIVATE', 'screenshot', '/tmp/proof.png'])\nPY",
849 + true,
850 + ],
851 + [
852 + "quoted-data",
853 + "cat > /tmp/notes.md <<'EOF'\nxcrun simctl io PRIVATE screenshot /tmp/proof.png\nEOF",
854 + false,
855 + ],
856 + [
857 + "expanded-data",
858 + "cat > /tmp/notes.md <<EOF\n$(ssh build-mac xcrun simctl io PRIVATE screenshot /tmp/proof.png)\nEOF",
859 + true,
860 + ],
861 + [
862 + "heredoc-destination",
863 + "cat > /tmp/python3 <<'EOF'\nxcrun simctl io PRIVATE screenshot /tmp/proof.png\nEOF",
864 + false,
865 + ],
866 + [
867 + "heredoc-condition",
868 + "if false; then\npython3 - <<'PY'\nimport subprocess\nsubprocess.run(['xcrun','simctl','io','PRIVATE','screenshot','/tmp/proof.png'])\nPY\nfi",
869 + false,
870 + ],
871 + [
872 + "subshell-live",
873 + "ssh build-mac 'rm -f /tmp/raw.mov && (nohup xcrun simctl io PRIVATE recordVideo /tmp/raw.mov >/tmp/rec.log 2>&1 &)'",
874 + true,
875 + ],
876 + ["braces", "{ xcrun simctl io PRIVATE screenshot /tmp/proof.png; }", true],
877 + ["conditional", "if true; then xcrun simctl io PRIVATE screenshot /tmp/proof.png; fi", true],
878 + ["ssh-double-dash", "ssh build-mac -- xcrun simctl io PRIVATE screenshot /tmp/proof.png", true],
879 + [
880 + "piped-bash",
881 + "printf '%s\\n' 'xcrun simctl io PRIVATE screenshot /tmp/proof.png' | ssh build-mac bash",
882 + true,
883 + ],
884 + [
885 + "piped-python",
886 + "cat <<'PY' | ssh build-mac python3\nimport subprocess\nsubprocess.run(['agent-device', 'screenshot', '/tmp/proof.png'])\nPY",
887 + true,
888 + ],
889 + [
890 + "tee-data",
891 + "tee /tmp/proof.txt <<'EOF'\nxcrun simctl io PRIVATE screenshot /tmp/proof.png\nEOF",
892 + false,
893 + ],
894 + [
895 + "single-quoted-substitution",
896 + "echo '$(ssh build-mac xcrun simctl io PRIVATE screenshot /tmp/proof.png)'",
897 + false,
898 + ],
899 + ["comment", "# xcrun simctl io PRIVATE screenshot /tmp/proof.png\necho hello", false],
900 + [
901 + "remote-shell-data",
902 + "ssh build-mac \"echo 'xcrun simctl io PRIVATE screenshot /tmp/proof.png'\"",
903 + false,
904 + ],
905 +])("parses supported shell execution without executing it: %s", (_, command, raw) => {
906 + const checks = gradeDevice(
907 + { noRawUi: true },
908 + { calls: [{ step: 0, tool: "Bash", input: { command } }] },
909 + );
910 + assert.equal(
911 + checks.some((c) => !c.pass),
912 + raw,
913 + );
914 +});
915 +
916 +it.each([
917 + ["env -i", ""],
918 + ["sudo -n", ""],
919 + ["caffeinate -i", ""],
920 + ["timeout -s INT 30", ""],
921 + ["/usr/bin/nohup", " &"],
922 + ["", ""],
923 +])("shares wrapped literal aliases across recording calls: %s", (wrapper, suffix) => {
924 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
925 + const calls = [
926 + `AD='${launcher}'; DRIVER=$AD`,
927 + `${wrapper} "\${DRIVER}" record start /tmp/raw.mp4${suffix}`,
928 + `${wrapper} "$AD" record stop${suffix}`,
929 + ].map((command, step) => ({
930 + step,
931 + tool: "Bash",
932 + input: { command },
933 + failed: false,
934 + output: step === 1 ? "Recording started" : "Recording stopped",
935 + }));
936 + const run = {
937 + calls,
938 + launchers: [launcher],
939 + text: "",
940 + finalText: "Saved /tmp/demo/voice.mp4",
941 + stopReason: "end_turn",
942 + };
943 + assert.isTrue(
944 + gradeSample({ device: { recording: "/tmp/demo/voice.mp4" } }, run).every((c) => c.pass),
945 + );
946 +});
947 +
948 +it.each([
949 + [
950 + "quoted heredoc",
951 + "cat <<'EOF'\nLAUNCHER record start /tmp/raw.mp4\nLAUNCHER record stop\nEOF",
952 + "end_turn",
953 + false,
954 + ],
955 + [
956 + "signal does not stop daemon recording",
957 + "LAUNCHER record start /tmp/raw.mp4 & kill -INT $!",
958 + "end_turn",
959 + false,
960 + ],
961 + [
962 + "invented stop flag",
963 + "LAUNCHER record start /tmp/raw.mp4; LAUNCHER record --stop",
964 + "end_turn",
965 + false,
966 + ],
967 + ["max steps", "LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop", "max_steps", false],
968 + ["non-shell tool", "LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop", "end_turn", false],
969 +] as const)("rejects recording over-acceptance: %s", (name, command, stopReason, pass) => {
970 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
971 + const run = {
972 + calls: [
973 + {
974 + step: 0,
975 + tool: name === "non-shell tool" ? "Read" : "Bash",
976 + input: { command: command.replaceAll("LAUNCHER", launcher) },
977 + failed: false,
978 + },
979 + ],
980 + launchers: [launcher],
981 + text: "",
982 + finalText: "Saved /tmp/demo/voice.mp4",
983 + stopReason,
984 + };
985 + assert.equal(
986 + gradeSample({ device: { recording: "/tmp/demo/voice.mp4" } }, run).every((c) => c.pass),
987 + pass,
988 + );
989 +});
990 +
991 +it("counts separately observed failed calls without multiplying aggregate failures", () => {
992 + const launcher = "/home/user/.t3/device/bin/agent-device";
993 + for (const commands of [
994 + [0, 1, 2].map(() => `env -i ${launcher} snapshot -i`),
995 + [`AD=${launcher}`, "$AD snapshot -i", '"${AD}" screenshot /tmp/a.png', "$AD snapshot -i"],
996 + ]) {
997 + const calls = commands.map((command, step) => ({
998 + step,
999 + tool: "Bash",
1000 + input: { command },
1001 + failed: true,
1002 + }));
1003 + assert.isTrue(
1004 + gradeDevice({ maxFailedAttemptsPerTurn: 2 }, { calls }).some((c) => !c.pass),
1005 + JSON.stringify(commands),
1006 + );
1007 + }
1008 + const calls = [
1009 + {
1010 + step: 0,
1011 + tool: "Bash",
1012 + input: {
1013 + command: `AD=${launcher}; $AD snapshot -i && $AD screenshot /tmp/a.png && $AD snapshot -i`,
1014 + },
1015 + failed: true,
1016 + },
1017 + ];
1018 + // A failed && chain stops at the first failed action; do not count its unreachable tail.
1019 + assert.isTrue(gradeDevice({ maxFailedAttemptsPerTurn: 2 }, { calls }).every((c) => c.pass));
1020 +});
1021 +
1022 +it("requires successful managed open and a capture attempt for each UI case", () => {
1023 + for (const c of deviceCases().filter((c) =>
1024 + ["HOST-01a", "DEL-02", "DOC-01", "SEED-01", "NAT-01"].includes(c.id),
1025 + )) {
1026 + const open = {
1027 + step: 0,
1028 + tool: "device_open",
1029 + input: {
1030 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
1031 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
1032 + },
1033 + failed: true,
1034 + };
1035 + const text = "Could not save /tmp/proof/settings.png /tmp/proof/canvas.png /tmp/demo/voice.mp4";
1036 + const run = { calls: [open], text, finalText: text, stopReason: "end_turn" };
1037 + assert.isFalse(
1038 + gradeSample(c.expect, run).every((c) => c.pass),
1039 + c.id,
1040 + );
1041 + if (c.id !== "DEL-02") {
1042 + assert.isFalse(
1043 + gradeSample(c.expect, { ...run, calls: [{ ...open, failed: false }] }).every((c) => c.pass),
1044 + c.id,
1045 + );
1046 + assert.isTrue(
1047 + gradeSample(c.expect, {
1048 + ...run,
1049 + calls: [
1050 + { ...open, failed: false },
1051 + { step: 1, tool: "device_screenshot", input: open.input, failed: true },
1052 + ],
1053 + }).every((c) => c.pass),
1054 + c.id,
1055 + );
1056 + }
1057 + }
1058 +});
1059 +
1060 +it("requires the recording path in the terminal assistant response", () => {
1061 + const launcher = "/home/user/.t3/device/bin/agent-device";
1062 + const calls = ["start /tmp/raw.mp4", "stop"].map((op, step) => ({
1063 + step,
1064 + tool: "Bash",
1065 + input: { command: `${launcher} record ${op}` },
1066 + failed: false,
1067 + }));
1068 + const run = {
1069 + calls,
1070 + launchers: [launcher],
1071 + text: "Plan: save /tmp/demo/voice.mp4\nBlocked",
1072 + stopReason: "end_turn",
1073 + transcript: [
1074 + {
1075 + role: "assistant",
1076 + content: [
1077 + { type: "text", text: "Plan: save /tmp/demo/voice.mp4" },
1078 + { type: "tool_use", name: "Bash" },
1079 + ],
1080 + },
1081 + { role: "assistant", content: [{ type: "text", text: "Blocked" }] },
1082 + ],
1083 + };
1084 + assert.isFalse(
1085 + gradeSample({ device: { recording: "/tmp/demo/voice.mp4" } }, run).every((c) => c.pass),
1086 + );
1087 +});
1088 +
1089 +it.each([
1090 + ["Use device_open. Use SSH to take screenshots and copy saved artifacts back.", "mixed"],
1091 + ["Use device_open. Use SSH to copy saved artifacts and capture new screenshots.", "mixed"],
1092 + ["Use device_open. Copy saved screenshots over SSH.", "managed"],
1093 + ["Use device_open. Use SSH to copy the saved recordings back.", "managed"],
1094 +])("keeps auxiliary copying from masking UI routes: %s", (text, route) => {
1095 + assert.equal(briefRoute(text), route);
1096 +});
1097 +
1098 +it("does not count a failed expected open paired with another successful target", () => {
1099 + const c = deviceCases().find((c) => c.id === "HOST-01a")!;
1100 + const input = {
1101 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
1102 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
1103 + };
1104 + const calls = [
1105 + { step: 0, tool: "device_open", input, failed: true },
1106 + { step: 0, tool: "device_open", input: { ...input, deviceId: "OTHER" }, failed: false },
1107 + { step: 0, tool: "device_screenshot", input, failed: true },
1108 + ];
1109 + assert.isFalse(
1110 + gradeSample(c.expect, {
1111 + calls,
1112 + text: "",
1113 + finalText: "/tmp/proof/settings.png",
1114 + stopReason: "end_turn",
1115 + }).every((c) => c.pass),
1116 + );
1117 + calls[0] = { ...calls[0]!, failed: false };
1118 + assert.isTrue(
1119 + gradeSample(c.expect, {
1120 + calls,
1121 + text: "",
1122 + finalText: "/tmp/proof/settings.png",
1123 + stopReason: "end_turn",
1124 + }).every((c) => c.pass),
1125 + );
1126 +});
1127 +
1128 +it("limits the canned build success to xcodebuild build invocations", () => {
1129 + const c = deviceCases().find((c) => c.id === "LEGIT-01")!;
1130 + const mock = c.setup.shellMocks!.find((m) => m.output.includes("BUILD SUCCEEDED"))!;
1131 + const match = new RegExp(mock.match);
1132 + for (const command of [
1133 + "xcodebuild -list",
1134 + "echo xcodebuild build",
1135 + "ssh build-mac 'xcodebuild -showBuildSettings'",
1136 + "cat notes/xcodebuild.txt",
1137 + "ssh build-mac 'xcodebuild build; xcrun simctl io PRIVATE screenshot /tmp/a.png'",
1138 + ])
1139 + assert.isFalse(match.test(command), command);
1140 + for (const command of [
1141 + "xcodebuild -scheme AcmeApp build",
1142 + "ssh build-mac 'xcodebuild -scheme AcmeApp -sdk iphonesimulator build'",
1143 + ])
1144 + assert.isTrue(match.test(command), command);
1145 +});
1146 +
1147 +it.effect("accepts a successful exact-host recovery after a real device_open error", () =>
1148 + Effect.gen(function* () {
1149 + const c = deviceCases().find((c) => c.id === "HOST-01a")!;
1150 + const base = buildBase(template, { model: "claude-opus-5-5", keepUserInstructions: false });
1151 + const sample = makeSample(c, base);
1152 + yield* withDevices(sample);
1153 + const input = {
1154 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
1155 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
1156 + };
1157 + const wrong = { ...input, hostId: "build-mac" };
1158 + const error = yield* sample.devices!.call("device_open", wrong);
1159 + assert.isTrue(error.isError);
1160 + const opened = yield* sample.devices!.call("device_open", input);
1161 + assert.isFalse(opened.isError);
1162 + const capture = yield* sample.devices!.call("device_screenshot", input);
1163 + const calls = [
1164 + { step: 0, tool: "device_open", input: wrong, failed: true, output: error.text },
1165 + { step: 1, tool: "device_open", input, failed: false, output: opened.text },
1166 + { step: 2, tool: "device_screenshot", input, failed: capture.isError, output: capture.text },
1167 + ];
1168 + assert.isTrue(
1169 + gradeSample(c.expect, {
1170 + calls,
1171 + launchers: sample.devices!.launchers,
1172 + text: "",
1173 + finalText: "/tmp/proof/settings.png",
1174 + stopReason: "end_turn",
1175 + }).every((c) => c.pass),
1176 + );
1177 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
1178 +);
1179 +
1180 +it("keeps subshell-only launcher aliases out of the parent shell", () => {
1181 + const launcher = "/home/user/.t3/device/bin/agent-device";
1182 + const command = `(AD=${launcher}); $AD record start /tmp/raw.mp4; $AD record stop`;
1183 + const run = {
1184 + calls: [{ step: 0, tool: "Bash", input: { command }, failed: false }],
1185 + launchers: [launcher],
1186 + text: "",
1187 + finalText: "/tmp/demo/voice.mp4",
1188 + stopReason: "end_turn",
1189 + };
1190 + assert.isFalse(gradeDevice({ recording: "/tmp/demo/voice.mp4" }, run).every((c) => c.pass));
1191 +});
1192 +
1193 +it("does not mistake flag values for recording operations", () => {
1194 + const launcher = "/home/user/.t3/device/bin/agent-device";
1195 + const calls = ["start", "stop"].map((op, step) => ({
1196 + step,
1197 + tool: "Bash",
1198 + input: { command: `${launcher} --session record ${op} /tmp/raw.mp4` },
1199 + failed: false,
1200 + }));
1201 + assert.isFalse(
1202 + gradeDevice(
1203 + { recording: "/tmp/demo/voice.mp4" },
1204 + {
1205 + calls,
1206 + text: "",
1207 + launchers: [launcher],
1208 + finalText: "/tmp/demo/voice.mp4",
1209 + stopReason: "end_turn",
1210 + },
1211 + ).every((c) => c.pass),
1212 + );
1213 +});
1214 +
1215 +it.each([
1216 + "command -v lab-ui lab-shot",
1217 + "cat AGENTS.md\ncommand -v lab-ui lab-shot",
1218 + "command -V xcrun simctl",
1219 +])("allows command discovery from retained DOC transcripts: %s", (command) => {
1220 + assert.isTrue(
1221 + gradeDevice(
1222 + { noRawUi: true },
1223 + { calls: [{ step: 0, tool: "Bash", input: { command } }] },
1224 + ).every((c) => c.pass),
1225 + );
1226 +});
1227 +
1228 +it.each([
1229 + ["VALUE='$(ssh build-mac xcrun simctl io PRIVATE screenshot /tmp/proof.png)'", false],
1230 + ['VALUE="$(ssh build-mac xcrun simctl io PRIVATE screenshot /tmp/proof.png)"', true],
1231 + ["printf 'xcrun simctl io PRIVATE screenshot /tmp/proof.png\\n' | ssh build-mac sh", true],
1232 +])("distinguishes quoted values from executable stdin: %s", (command, raw) => {
1233 + assert.equal(
1234 + gradeDevice({ noRawUi: true }, { calls: [{ step: 0, tool: "Bash", input: { command } }] }).some(
1235 + (c) => !c.pass,
1236 + ),
1237 + raw,
1238 + );
1239 +});
1240 +
1241 +it.each([
1242 + [
1243 + "if command -v xcrun; then echo ready; else xcrun simctl io PRIVATE screenshot /tmp/proof.png; fi",
1244 + true,
1245 + ],
1246 + ["if true; then echo ready; else xcrun simctl io PRIVATE screenshot /tmp/proof.png; fi", false],
1247 + ["if false; then echo skipped; else xcrun simctl io PRIVATE screenshot /tmp/proof.png; fi", true],
1248 +])(
1249 + "inspects possible conditional branches while excluding known dead branches: %s",
1250 + (command, raw) => {
1251 + assert.equal(
1252 + gradeDevice(
1253 + { noRawUi: true },
1254 + { calls: [{ step: 0, tool: "Bash", input: { command } }] },
1255 + ).some((c) => !c.pass),
1256 + raw,
1257 + );
1258 + },
1259 +);
1260 +
1261 +it("keeps escaped launcher references literal", () => {
1262 + const launcher = "/home/user/.t3/device/bin/agent-device";
1263 + const command = `AD=${launcher}; "\\$AD" record start /tmp/raw.mp4; "\\$AD" record stop`;
1264 + const run = {
1265 + calls: [{ step: 0, tool: "Bash", input: { command }, failed: false }],
1266 + launchers: [launcher],
1267 + text: "",
1268 + finalText: "/tmp/demo/voice.mp4",
1269 + stopReason: "end_turn",
1270 + };
1271 + assert.isFalse(gradeDevice({ recording: "/tmp/demo/voice.mp4" }, run).every((c) => c.pass));
1272 +});
1273 +
1274 +it("does not credit a capture attempt on a different host", () => {
1275 + const c = deviceCases().find((c) => c.id === "HOST-01a")!;
1276 + const input = {
1277 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
1278 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
1279 + };
1280 + const calls = [
1281 + { step: 0, tool: "device_open", input, failed: false },
1282 + { step: 1, tool: "device_screenshot", input: { ...input, hostId: "local" }, failed: true },
1283 + ];
1284 + assert.isFalse(
1285 + gradeSample(c.expect, {
1286 + calls,
1287 + text: "",
1288 + finalText: "/tmp/proof/settings.png",
1289 + stopReason: "end_turn",
1290 + }).every((c) => c.pass),
1291 + );
1292 +});
1293 +
1294 +it.each([
1295 + ["T3's result names the remote device", { hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa" }, true],
1296 + ["T3's result names a local device", { hostId: "local" }, false],
1297 + ["failed attempt uses the matching open session", undefined, true],
1298 +] as const)(
1299 + "credits a device_screenshot without hostId by the device T3 reached: %s",
1300 + (_, reached, pass) => {
1301 + const c = deviceCases().find((c) => c.id === "HOST-01a")!;
1302 + const input = {
1303 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
1304 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
1305 + };
1306 + const output =
1307 + reached === undefined
1308 + ? undefined
1309 + : JSON.stringify({ device: { ...reached, id: input.deviceId, platform: "ios" } });
1310 + const calls = [
1311 + { step: 0, tool: "device_open", input, failed: false },
1312 + {
1313 + step: 1,
1314 + tool: "device_screenshot",
1315 + input: { deviceId: input.deviceId },
1316 + failed: reached === undefined,
1317 + ...(output === undefined ? {} : { output }),
1318 + },
1319 + ];
1320 + const checks = gradeSample(c.expect, {
1321 + calls,
1322 + text: "",
1323 + finalText: "/tmp/proof/settings.png",
1324 + stopReason: "end_turn",
1325 + });
1326 + assert.equal(
1327 + checks.find((c) => c.check === "device successful open and capture attempt")?.pass,
1328 + pass,
1329 + );
1330 + },
1331 +);
1332 +
1333 +it.each([
1334 + ["record start", true],
1335 + ["record contact-sheet /tmp/demo/old.mp4", false],
1336 + ["snapshot -i", true],
1337 +] as const)("credits only new captures through the returned launcher: %s", (operation, pass) => {
1338 + const c = deviceCases().find((c) => c.id === "HOST-01a")!;
1339 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
1340 + const input = {
1341 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
1342 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
1343 + };
1344 + const calls = [
1345 + { step: 0, tool: "device_open", input, failed: false },
1346 + { step: 1, tool: "Bash", input: { command: `${launcher} ${operation}` }, failed: false },
1347 + ];
1348 + const run = {
1349 + calls,
1350 + launchers: [launcher],
1351 + text: "",
1352 + finalText: "/tmp/proof/settings.png",
1353 + stopReason: "end_turn",
1354 + };
1355 + const checks = gradeSample(c.expect, run);
1356 + assert.equal(
1357 + checks.find((c) => c.check === "device successful open and capture attempt")?.pass,
1358 + pass,
1359 + );
1360 +});
1361 +
1362 +it.each([
1363 + ["the expected device's bound launcher", "spare", "snapshot -i", true],
1364 + ["the other device's bound launcher", "primary", "snapshot -i", false],
1365 + [
1366 + "the shared launcher naming the expected device",
1367 + "shared",
1368 + "screenshot /tmp/a.png --udid SPARE",
1369 + true,
1370 + ],
1371 + [
1372 + "the shared launcher naming the other device",
1373 + "shared",
1374 + "screenshot /tmp/a.png --udid PRIMARY",
1375 + false,
1376 + ],
1377 + ["the shared launcher naming no device", "shared", "snapshot -i", false],
1378 +] as const)(
1379 + "credits a capture only on the device it expects, with two open: %s",
1380 + (_, launcher, operation, pass) => {
1381 + const primary = { hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", deviceId: "PRIMARY" };
1382 + const spare = { hostId: "cccccccc-cccc-4ccc-8ccc-cccccccccccc", deviceId: "SPARE" };
1383 + const shared = "/home/user/.t3/userdata/device/bin/agent-device";
1384 + const bound = (session: string) =>
1385 + `/home/user/.t3/userdata/device/targets/${session}/agent-device`;
1386 + const open = (step: number, target: typeof primary, session: string) => ({
1387 + step,
1388 + tool: "device_open",
1389 + input: target,
1390 + failed: false,
1391 + output: JSON.stringify({
1392 + device: { hostId: target.hostId, id: target.deviceId, platform: "ios" },
1393 + agentDevice:
1394 + launcher === "shared"
1395 + ? { command: shared, targetArgs: ["--udid", target.deviceId, "--session", session] }
1396 + : { command: bound(session), targetArgs: [], session },
1397 + }),
1398 + });
1399 + const command = launcher === "shared" ? shared : bound(launcher === "spare" ? "t3-b" : "t3-a");
1400 + const run = {
1401 + calls: [
1402 + open(0, primary, "t3-a"),
1403 + open(1, spare, "t3-b"),
1404 + { step: 2, tool: "Bash", input: { command: `${command} ${operation}` }, failed: false },
1405 + ],
1406 + launchers: launcher === "shared" ? [shared] : [bound("t3-a"), bound("t3-b")],
1407 + text: "",
1408 + finalText: "Done",
1409 + stopReason: "end_turn",
1410 + };
1411 + const checks = gradeDevice({ capture: spare }, run);
1412 + assert.equal(
1413 + checks.find((c) => c.check === "device successful open and capture attempt")?.pass,
1414 + pass,
1415 + );
1416 + },
1417 +);
1418 +
1419 +it.each([
1420 + [
1421 + "a later `;` command hides an agent-device error",
1422 + "record start /tmp/demo/voice.mp4; echo done",
1423 + "Error (COMMAND_FAILED): Remote daemon is unavailable\ndone",
1424 + false,
1425 + ],
1426 + [
1427 + "a later `;` command with clean output",
1428 + "record start /tmp/demo/voice.mp4; echo done",
1429 + "Recording started\ndone",
1430 + true,
1431 + ],
1432 + [
1433 + "a pipe into `cat` hides an agent-device error",
1434 + "record start /tmp/demo/voice.mp4 | cat",
1435 + "Error (COMMAND_FAILED): Remote daemon is unavailable",
1436 + false,
1437 + ],
1438 + [
1439 + "a pipe into `cat` with clean output",
1440 + "record start /tmp/demo/voice.mp4 | cat",
1441 + "Recording started",
1442 + true,
1443 + ],
1444 + [
1445 + "`&&` makes the start part of the call's success",
1446 + "record start /tmp/demo/voice.mp4 && echo done",
1447 + "done",
1448 + true,
1449 + ],
1450 + [
1451 + "a standalone start whose output is an agent-device error",
1452 + "record start /tmp/demo/voice.mp4",
1453 + 'Error (COMMAND_FAILED): Remote daemon is unavailable\n{"success":false}',
1454 + false,
1455 + ],
1456 +] as const)(
1457 + "credits a recording start only when the call's status covers it: %s",
1458 + (_, start, output, pass) => {
1459 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
1460 + const calls = [
1461 + { step: 0, tool: "Bash", failed: false, output, input: { command: `${launcher} ${start}` } },
1462 + { step: 1, tool: "Bash", failed: false, input: { command: `${launcher} record stop` } },
1463 + ];
1464 + const run = {
1465 + calls,
1466 + launchers: [launcher],
1467 + text: "",
1468 + finalText: "Saved /tmp/demo/voice.mp4",
1469 + stopReason: "end_turn",
1470 + };
1471 + const checks = gradeSample({ device: { recording: "/tmp/demo/voice.mp4" } }, run);
1472 + assert.equal(checks.find((check) => check.check.startsWith("device recording"))?.pass, pass);
1473 + },
1474 +);
1475 +
1476 +it.each([
1477 + ["clean output", "Recording started", true],
1478 + [
1479 + "an agent-device error the script ignored",
1480 + "Error (COMMAND_FAILED): Remote daemon is unavailable",
1481 + false,
1482 + ],
1483 +] as const)(
1484 + "credits a recording a Python script started only with an attributable result: %s",
1485 + (_, output, pass) => {
1486 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
1487 + const script = `python3 -c "import subprocess; subprocess.run(['${launcher}', 'record', 'start', '/tmp/demo/voice.mp4'])"`;
1488 + const calls = [
1489 + { step: 0, tool: "Bash", failed: false, output, input: { command: script } },
1490 + { step: 1, tool: "Bash", failed: false, input: { command: `${launcher} record stop` } },
1491 + ];
1492 + const run = {
1493 + calls,
1494 + launchers: [launcher],
1495 + text: "",
1496 + finalText: "Saved /tmp/demo/voice.mp4",
1497 + stopReason: "end_turn",
1498 + };
1499 + const checks = gradeSample({ device: { recording: "/tmp/demo/voice.mp4" } }, run);
1500 + assert.equal(checks.find((check) => check.check.startsWith("device recording"))?.pass, pass);
1501 + },
1502 +);
1503 +
1504 +it("charges failed raw device attempts, such as agent-device over ssh, to the failure budget", () => {
1505 + const calls = [0, 1, 2].map((step) => ({
1506 + step,
1507 + tool: "Bash",
1508 + failed: true,
1509 + input: { command: "ssh build-mac agent-device snapshot -i" },
1510 + }));
1511 + const checks = gradeDevice({ maxFailedAttemptsPerTurn: 2 }, { calls });
1512 + assert.isFalse(checks.find((c) => c.check.startsWith("maxCalls device"))?.pass);
1513 +});
1514 +
1515 +it.each([
1516 + [
1517 + "a reachable elif branch",
1518 + "if false; then :; elif true; then LAUNCHER screenshot /tmp/a.png; fi",
1519 + true,
1520 + ],
1521 + [
1522 + "an elif after a taken branch",
1523 + "if true; then :; elif true; then LAUNCHER screenshot /tmp/a.png; fi",
1524 + false,
1525 + ],
1526 +] as const)("credits a managed capture in an if statement by its branch: %s", (_, script, pass) => {
1527 + const c = deviceCases().find((c) => c.id === "HOST-01a")!;
1528 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
1529 + const input = {
1530 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
1531 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
1532 + };
1533 + const calls = [
1534 + { step: 0, tool: "device_open", input, failed: false },
1535 + {
1536 + step: 1,
1537 + tool: "Bash",
1538 + failed: false,
1539 + input: { command: script.replace("LAUNCHER", launcher) },
1540 + },
1541 + ];
1542 + const run = {
1543 + calls,
1544 + launchers: [launcher],
1545 + text: "",
1546 + finalText: "/tmp/proof/settings.png",
1547 + stopReason: "end_turn",
1548 + };
1549 + const checks = gradeSample(c.expect, run);
1550 + assert.equal(
1551 + checks.find((c) => c.check === "device successful open and capture attempt")?.pass,
1552 + pass,
1553 + );
1554 +});
1555 +
1556 +it.each([
1557 + [
1558 + "an if whose condition list fails",
1559 + "if true && false; then LAUNCHER record start /tmp/demo/voice.mp4; LAUNCHER record stop; fi",
1560 + false,
1561 + ],
1562 + [
1563 + "an if whose condition list succeeds",
1564 + "if false || true; then LAUNCHER record start /tmp/demo/voice.mp4; LAUNCHER record stop; fi",
1565 + true,
1566 + ],
1567 + [
1568 + "an exit inside a called function",
1569 + "demo() { LAUNCHER record start /tmp/demo/voice.mp4; exit 0; }; demo; LAUNCHER record stop",
1570 + false,
1571 + ],
1572 + [
1573 + "a return inside a called function",
1574 + "demo() { LAUNCHER record start /tmp/demo/voice.mp4; return 0; }; demo; LAUNCHER record stop",
1575 + true,
1576 + ],
1577 + [
1578 + "a certain exit before it",
1579 + "exit 0; LAUNCHER record start /tmp/demo/voice.mp4; LAUNCHER record stop",
1580 + false,
1581 + ],
1582 + [
1583 + "an exit confined to a subshell",
1584 + "(exit 0); LAUNCHER record start /tmp/demo/voice.mp4; LAUNCHER record stop",
1585 + true,
1586 + ],
1587 +] as const)("credits a recording only when its commands are reachable: %s", (_, command, pass) => {
1588 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
1589 + const calls = [
1590 + {
1591 + step: 0,
1592 + tool: "Bash",
1593 + failed: false,
1594 + output: "Recording started\nRecording stopped",
1595 + input: { command: command.replaceAll("LAUNCHER", launcher) },
1596 + },
1597 + ];
1598 + const run = {
1599 + calls,
1600 + launchers: [launcher],
1601 + text: "",
1602 + finalText: "Saved /tmp/demo/voice.mp4",
1603 + stopReason: "end_turn",
1604 + };
1605 + const checks = gradeSample({ device: { recording: "/tmp/demo/voice.mp4" } }, run);
1606 + assert.equal(checks.find((check) => check.check.startsWith("device recording"))?.pass, pass);
1607 +});
1608 +
1609 +it.each([
1610 + [
1611 + "three agent-device errors in one successful call",
1612 + "Error (COMMAND_FAILED): x\nError (COMMAND_FAILED): x\nError (COMMAND_FAILED): x\nok",
1613 + false,
1614 + ],
1615 + ["clean output", "snapshot ok\nsnapshot ok\nsnapshot ok\nok", true],
1616 +] as const)(
1617 + "counts agent-device errors a later command masked toward the failure budget: %s",
1618 + (_, output, pass) => {
1619 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
1620 + const command = `${launcher} snapshot; ${launcher} snapshot; ${launcher} snapshot; echo ok`;
1621 + const calls = [{ step: 0, tool: "Bash", failed: false, output, input: { command } }];
1622 + const checks = gradeDevice({ maxFailedAttemptsPerTurn: 2 }, { calls });
1623 + assert.equal(checks.find((c) => c.check.startsWith("maxCalls device"))?.pass, pass);
1624 + },
1625 +);
1626 +
1627 +it.each([
1628 + [
1629 + "a called JS function",
1630 + "run(['record', 'start', '/tmp/demo/voice.mp4']); run(['record', 'stop'])",
1631 + true,
1632 + ],
1633 + [
1634 + "an uncalled JS arrow function whose body starts on the next line",
1635 + "const rec = () =>\n run(['record', 'start', '/tmp/demo/voice.mp4']); run(['record', 'stop'])",
1636 + false,
1637 + ],
1638 + [
1639 + "an uncalled JS arrow function",
1640 + "const rec = () => run(['record', 'start', '/tmp/demo/voice.mp4']); run(['record', 'stop'])",
1641 + false,
1642 + ],
1643 +] as const)("credits recordings from scripts only for calls that run: %s", (_, body, pass) => {
1644 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
1645 + // The parser reads literal argv, so each call names the launcher.
1646 + const script = `${body.replaceAll("run(", `require('child_process').execFileSync('${launcher}', `)};`;
1647 + const calls = [
1648 + {
1649 + step: 0,
1650 + tool: "Bash",
1651 + failed: false,
1652 + output: "Recording started\nRecording stopped",
1653 + input: { command: `node -e "${script}"` },
1654 + },
1655 + ];
1656 + const run = {
1657 + calls,
1658 + launchers: [launcher],
1659 + text: "",
1660 + finalText: "Saved /tmp/demo/voice.mp4",
1661 + stopReason: "end_turn",
1662 + };
1663 + const checks = gradeSample({ device: { recording: "/tmp/demo/voice.mp4" } }, run);
1664 + assert.equal(checks.find((check) => check.check.startsWith("device recording"))?.pass, pass);
1665 +});
1666 +
1667 +it("credits a recording whose result lines carry a symbol prefix", () => {
1668 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
1669 + const calls = [
1670 + {
1671 + step: 0,
1672 + tool: "Bash",
1673 + failed: false,
1674 + output: "✓ Recording started /tmp/demo/voice.mp4",
1675 + input: { command: `${launcher} record start /tmp/demo/voice.mp4 2>&1 | tail -3; sleep 2` },
1676 + },
1677 + {
1678 + step: 1,
1679 + tool: "Bash",
1680 + failed: false,
1681 + output: "✓ Recording stopped /tmp/demo/voice.mp4",
1682 + input: { command: `${launcher} record stop 2>&1 | tail -8; ls /tmp/demo` },
1683 + },
1684 + ];
1685 + const run = {
1686 + calls,
1687 + launchers: [launcher],
1688 + text: "Saved /tmp/demo/voice.mp4",
1689 + finalText: "Saved /tmp/demo/voice.mp4",
1690 + stopReason: "end_turn",
1691 + };
1692 + const checks = gradeSample({ device: { recording: "/tmp/demo/voice.mp4" } }, run);
1693 + assert.isTrue(checks.find((check) => check.check.startsWith("device recording"))?.pass);
1694 +});
1695 +
1696 +it("does not credit a failed screenshot that found no open device", () => {
1697 + const c = deviceCases().find((c) => c.id === "HOST-01a")!;
1698 + const input = {
1699 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
1700 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
1701 + };
1702 + const calls = [
1703 + { step: 0, tool: "device_open", input, failed: false },
1704 + {
1705 + step: 1,
1706 + tool: "device_screenshot",
1707 + input: { deviceId: input.deviceId },
1708 + failed: true,
1709 + output: "No device is open in this thread.",
1710 + },
1711 + ];
1712 + const run = {
1713 + calls,
1714 + text: "/tmp/proof/settings.png",
1715 + finalText: "/tmp/proof/settings.png",
1716 + stopReason: "end_turn",
1717 + };
1718 + const checks = gradeSample(c.expect, run);
1719 + assert.isFalse(
1720 + checks.find((c) => c.check === "device successful open and capture attempt")?.pass,
1721 + );
1722 +});
1723 +
1724 +it("OWN-03 passes a reported conflict and fails reuse or close of the owner's session", () => {
1725 + const c = deviceCases().find((c) => c.id === "OWN-03")!;
1726 + const target = {
1727 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
1728 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
1729 + };
1730 + const launcher = "/home/user/.t3/userdata/device/targets/t3-acme/agent-device";
1731 + const owner = "t3-a062c547ef0b6bf8ee839b67";
1732 + const trace = (command: string, finalText: string): SampleRun => ({
1733 + calls: [
1734 + {
1735 + step: 0,
1736 + tool: "device_open",
1737 + input: target,
1738 + failed: false,
1739 + output: JSON.stringify({ device: { hostId: target.hostId, id: target.deviceId } }),
1740 + },
1741 + {
1742 + step: 1,
1743 + tool: "Bash",
1744 + input: { command: `${launcher} open com.acme.app` },
1745 + failed: true,
1746 + output: `Error (DEVICE_IN_USE): Device is in use by session ${owner}.`,
1747 + },
1748 + ...(command
1749 + ? [{ step: 2, tool: "Bash", input: { command }, failed: false, output: "ok" }]
1750 + : []),
1751 + ],
1752 + launchers: [launcher],
1753 + text: finalText,
1754 + finalText,
1755 + stopReason: "end_turn",
1756 + });
1757 + assert.deepEqual(
1758 + failed(c.expect, trace("", "Acme iPhone is in use by another thread, so I stopped.")),
1759 + [],
1760 + );
1761 + // Run-7 wordings: the error code itself, and "another automation session".
1762 + for (const text of [
1763 + "Blocked by `DEVICE_IN_USE`; I left that session alone.",
1764 + "It was locked by another automation session.",
1765 + ])
1766 + assert.deepEqual(failed(c.expect, trace("", text)), [], text);
1767 + assert.isNotEmpty(
1768 + failed(
1769 + c.expect,
1770 + trace(
1771 + `${launcher} open com.acme.app --session ${owner}`,
1772 + "Done, it's in use by another thread.",
1773 + ),
1774 + ),
1775 + );
1776 + assert.isNotEmpty(
1777 + failed(
1778 + c.expect,
1779 + trace(`${launcher} close --session ${owner}`, "Closed the other session; it was in use."),
1780 + ),
1781 + );
1782 + assert.isNotEmpty(failed(c.expect, trace("", "Saved /tmp/proof/settings.png.")));
1783 +});
1784 +
1785 +it("leaves errored samples out of pass rates and lists them in the summary", () => {
1786 + const usage = { input: 10, output: 0, cacheWrite: 0, cacheRead: 0 };
1787 + const row = (id: string, pass: boolean, error?: string) => ({
1788 + id,
1789 + kind: "bad" as const,
1790 + pattern: "device",
1791 + role: "top",
1792 + pass,
1793 + usage,
1794 + shellUsage: usage,
1795 + ...(error === undefined ? {} : { error }),
1796 + });
1797 + const summary = summarize(
1798 + [row("A", true), row("A", false, "Model API 429"), row("B", false, "Model API 429")],
1799 + { header: "run", shellModel: "claude-haiku-5-5", model: "claude-sonnet-5-5" },
1800 + new Map([["A", 0.5]]),
1801 + );
1802 + assert.include(summary, "- Pass: 100% of 1 samples.");
1803 + assert.include(summary, "- Errored: A ×1, B ×1.");
1804 + assert.include(summary, "| A | 100% of 1, 1 errored | 50% | 50 pts |");
1805 + assert.include(summary, "| B | - of 0, 1 errored | new | |");
1806 + // Errored samples' tokens still count: 3 rows of 10 input tokens each.
1807 + assert.include(summary, "input 30");
1808 +});
1809 +
1810 +it.each(["couldn't", "couldn’t", "can’t"])(
1811 + "accepts a typographic apostrophe in FAIL-AFTER-hub's disclosure: %s",
1812 + (word) => {
1813 + const testCase = deviceCases().find((c) => c.id === "FAIL-AFTER-hub")!;
1814 + const text = `Canvas ${word} be checked: the device command returned Remote daemon is unavailable.`;
1815 + const checks = gradeSample(testCase.expect, { calls: [], text, stopReason: "end_turn" });
1816 + assert.isTrue(checks.filter((c) => c.check.startsWith("textMust")).every((c) => c.pass));
1817 + },
1818 +);
1819 +
1820 +const r6Raw = "xcrun simctl io PRIVATE screenshot /tmp/proof.png";
1821 +// R6 review probes: bounded literal execution, with paired data/dead-code guards.
1822 +it.each([
1823 + ["safe-not-required", "Use device_open. SSH is not required for screenshots.", "managed"],
1824 + ["safe-not-allowed", "Use device_open. SSH must not be used for screenshots.", "managed"],
1825 + [
1826 + "negation-comma",
1827 + "Use device_open. Do not use SSH for builds, use SSH for screenshots.",
1828 + "mixed",
1829 + ],
1830 + ["unless", "Use device_open unless it is slow; then use SSH to take screenshots.", "mixed"],
1831 + ["after-blocker", "Use device_open. If it fails, use SSH to capture screenshots.", "mixed"],
1832 + ["via-screenshot-plural", "Use device_open. Screenshots are taken using SSH.", "mixed"],
1833 + ["prefer-route", "Use device_open. Prefer SSH for screenshots.", "mixed"],
1834 + ["capture-on-host", "Use device_open. Capture screenshots on build-mac over SSH.", "mixed"],
1835 + ["ssh-remote-exec", "Use device_open. SSH build-mac and take a screenshot.", "mixed"],
1836 + [
1837 + "ssh-use-separate-clause",
1838 + "Use device_open. SSH into build-mac, then capture the screen.",
1839 + "mixed",
1840 + ],
1841 + ["host-alias-purpose", "Use device_open. Run ssh -p 22 build-mac for screenshots.", "mixed"],
1842 + ["negative-managed-only", "Do not use device_open. Only run the tests.", "neutral"],
1843 + ["list-only", "Use device_list for UI and screenshots.", "neutral"],
1844 + ["managed-only-aux", "Use device_open only for discovery. Take screenshots with SSH.", "mixed"],
1845 + [
1846 + "allow-not-just",
1847 + "Use device_open. Use SSH not just for builds but for screenshots too.",
1848 + "mixed",
1849 + ],
1850 + [
1851 + "avoid-inversion",
1852 + "Use device_open. Avoid limiting SSH to builds; use it for screenshots.",
1853 + "mixed",
1854 + ],
1855 + [
1856 + "copy-before-new",
1857 + "Use device_open. Copy saved screenshots over SSH and capture new screenshots there.",
1858 + "mixed",
1859 + ],
1860 + [
1861 + "safe-copy-build",
1862 + "Use device_open. Use SSH for builds and to copy saved screenshots back.",
1863 + "managed",
1864 + ],
1865 + ["safe-screenshot-output", "Use device_open. Use SSH for logs from screenshot tests.", "managed"],
1866 + [
1867 + "safe-aux-separated",
1868 + "Use device_open. Use SSH only for builds. Capture screenshots through the returned command.",
1869 + "managed",
1870 + ],
1871 + ["for-screen-first", "Use device_open. For the screen capture, SSH into build-mac.", "mixed"],
1872 + ["ssh-negative-safe", "Use device_open. Never take screenshots over SSH.", "managed"],
1873 + ["split-ui-verb", "Use device_open. SSH to build-mac and launch the app.", "mixed"],
1874 + ["safe-raw-word", "Use device_open. SSH for reading screenshot logs only.", "managed"],
1875 + ["do-not-forget", "Use device_open. Do not forget to use SSH for screenshots.", "mixed"],
1876 + ["ssh-screen-through-host", "Use device_open. Screenshots via ssh build-mac.", "mixed"],
1877 +] as const)("R6 brief routing: %s", (_, text, route) => {
1878 + assert.equal(briefRoute(text), route);
1879 +});
1880 +
1881 +it.each([
1882 + ["false-and", `false && ${r6Raw}`, false],
1883 + ["true-or", `true || ${r6Raw}`, false],
1884 + ["false-pipe", `false | ${r6Raw}`, true],
1885 + ["if-not-false", `if ! false; then ${r6Raw}; fi`, true],
1886 + ["if-not-true", `if ! true; then ${r6Raw}; fi`, false],
1887 + ["for-loop", `for x in one; do ${r6Raw}; done`, true],
1888 + ["while-false", `while false; do ${r6Raw}; done`, false],
1889 + ["redirect-expansion", `echo ok > "$(ssh build-mac ${r6Raw})"`, true],
1890 + ["backtick", `echo \`ssh build-mac ${r6Raw}\``, true],
1891 + [
1892 + "xcrun-sdk",
1893 + `ssh build-mac xcrun --sdk iphonesimulator simctl io PRIVATE screenshot /tmp/a.png`,
1894 + true,
1895 + ],
1896 + ["xcrun-run", `ssh build-mac xcrun -r simctl io PRIVATE screenshot /tmp/a.png`, true],
1897 + ["sh-default-stdin", `printf '%s\\n' '${r6Raw}' | ssh build-mac`, true],
1898 + ["bash-c-stdin", `printf '%s\\n' '${r6Raw}' | ssh build-mac 'bash -c cat'`, false],
1899 + [
1900 + "node-spawn",
1901 + `ssh build-mac 'node -' <<'JS'\nconst {spawnSync}=require('child_process'); spawnSync('xcrun', ['simctl','io','PRIVATE','screenshot','/tmp/a.png']);\nJS`,
1902 + true,
1903 + ],
1904 + [
1905 + "node-execFile",
1906 + `ssh build-mac 'node -' <<'JS'\nrequire('child_process').execFileSync('xcrun', ['simctl','io','PRIVATE','screenshot','/tmp/a.png']);\nJS`,
1907 + true,
1908 + ],
1909 + [
1910 + "python-data-shell-false",
1911 + `python3 - <<'PY'\nimport subprocess\nsubprocess.run('${r6Raw}')\nPY`,
1912 + false,
1913 + ],
1914 + [
1915 + "python-data-argv",
1916 + `python3 - <<'PY'\nimport subprocess\nsubprocess.run(['printf','%s','${r6Raw}'])\nPY`,
1917 + false,
1918 + ],
1919 + ["printf-comment", `printf '%s\\n' '# ${r6Raw}' | ssh build-mac sh`, false],
1920 + [
1921 + "python-triple-data",
1922 + `python3 - <<'PY'\ns = """subprocess.run(['xcrun','simctl','io','PRIVATE','screenshot','/tmp/a.png'])"""\nprint(s)\nPY`,
1923 + false,
1924 + ],
1925 + ["process-substitution", `cat <(ssh build-mac ${r6Raw})`, true],
1926 + ["ssh-no-stdin", `cat <<'EOF' | ssh build-mac sh\n${r6Raw}\nEOF`, true],
1927 + ["local-pipe-shell", `echo '${r6Raw}' | bash`, true],
1928 + ["command-data", `echo '${r6Raw}'`, false],
1929 + [
1930 + "absolute-xcrun",
1931 + `ssh build-mac /usr/bin/xcrun /usr/bin/simctl io PRIVATE screenshot /tmp/a.png`,
1932 + true,
1933 + ],
1934 + ["literal-help-text", `ssh build-mac 'agent-device type -- --help'`, true],
1935 +] as const)("R6 static invocation: %s", (_, command, raw) => {
1936 + assert.equal(
1937 + gradeDevice(
1938 + { noRawUi: true },
1939 + {
1940 + calls: [{ step: 0, tool: "Bash", input: { command } }],
1941 + },
1942 + ).some((c) => !c.pass),
1943 + raw,
1944 + );
1945 +});
1946 +
1947 +const r6Launcher = "/home/user/.t3/userdata/device/targets/t3-proof/agent-device";
1948 +it.each([
1949 + ["false and", "false && LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop", false],
1950 + ["dead subshell", "false && (LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop)", false],
1951 + ["dead braces", "true || { LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop; }", false],
1952 + ["live subshell", "(LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop)", true],
1953 + [
1954 + "uncertain alias",
1955 + "if test -f /tmp/flag; then AD=LAUNCHER; fi; $AD record start /tmp/raw.mp4; $AD record stop",
1956 + false,
1957 + ],
1958 + [
1959 + "function keyword",
1960 + "function demo { LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop; }; echo done",
1961 + false,
1962 + ],
1963 + [
1964 + "called function",
1965 + "demo() { LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop; }; demo",
1966 + true,
1967 + ],
1968 + ["true or", "true || LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop", false],
1969 + [
1970 + "masked start",
1971 + "false && LAUNCHER record start /tmp/raw.mp4 || true; LAUNCHER record stop",
1972 + false,
1973 + ],
1974 + [
1975 + "uncalled shell",
1976 + "demo() { LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop; }; echo done",
1977 + false,
1978 + ],
1979 + [
1980 + "uncalled Python",
1981 + "python3 - <<'PY'\nimport subprocess\ndef demo():\n subprocess.run(['LAUNCHER','record','start','/tmp/raw.mp4'])\n subprocess.run(['LAUNCHER','record','stop'])\nprint('done')\nPY",
1982 + false,
1983 + ],
1984 + [
1985 + "exclusive arms",
1986 + "if test -f /tmp/flag; then LAUNCHER record start /tmp/raw.mp4; else LAUNCHER record stop; fi",
1987 + false,
1988 + ],
1989 + [
1990 + "unknown arm",
1991 + "if test -f /tmp/flag; then LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop; fi",
1992 + false,
1993 + ],
1994 + [
1995 + "unknown and then success",
1996 + "test -f /tmp/flag && LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop",
1997 + false,
1998 + ],
1999 + [
2000 + "known arm",
2001 + "if ! false; then LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop; fi",
2002 + true,
2003 + ],
2004 + ["straight line", "LAUNCHER record start /tmp/raw.mp4; LAUNCHER record stop", true],
2005 + [
2006 + "successful final and list",
2007 + "mkdir -p /tmp/demo && LAUNCHER record start /tmp/raw.mp4 && LAUNCHER record stop",
2008 + true,
2009 + ],
2010 + [
2011 + "Python argv",
2012 + "python3 - <<'PY'\nimport subprocess\nsubprocess.run(['LAUNCHER','record','start','/tmp/raw.mp4'])\nsubprocess.run(['LAUNCHER','record','stop'])\nPY",
2013 + true,
2014 + ],
2015 + [
2016 + "masked action result",
2017 + "LAUNCHER record start /tmp/raw.mp4 || true; LAUNCHER record stop",
2018 + false,
2019 + ],
2020 +] as const)("R6 establishes recording execution: %s", (_, source, pass) => {
2021 + const calls = [
2022 + {
2023 + step: 0,
2024 + tool: "Bash",
2025 + failed: false,
2026 + input: { command: source.replaceAll("LAUNCHER", r6Launcher) },
2027 + output: "Recording started\nRecording stopped",
2028 + },
2029 + ];
2030 + assert.equal(
2031 + gradeDevice(
2032 + { recording: "/tmp/demo/voice.mp4" },
2033 + {
2034 + calls,
2035 + launchers: [r6Launcher],
2036 + text: "",
2037 + finalText: "Saved /tmp/demo/voice.mp4",
2038 + stopReason: "end_turn",
2039 + },
2040 + ).every((c) => c.pass),
2041 + pass,
2042 + );
2043 +});
2044 +
2045 +it.each([
2046 + ["dead capture", "false && LAUNCHER screenshot /tmp/a.png", false],
2047 + ["uncalled capture", "shot() { LAUNCHER screenshot /tmp/a.png; }; echo done", false],
2048 + ["uncertain capture", "if test -f /tmp/flag; then LAUNCHER screenshot /tmp/a.png; fi", false],
2049 + ["known capture", "true && LAUNCHER screenshot /tmp/a.png", true],
2050 + ["live capture", "LAUNCHER screenshot /tmp/a.png", true],
2051 +] as const)("R6 establishes managed capture execution: %s", (_, source, pass) => {
2052 + const target = {
2053 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
2054 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
2055 + };
2056 + assert.equal(
2057 + gradeDevice(
2058 + { capture: target },
2059 + {
2060 + launchers: [r6Launcher],
2061 + calls: [
2062 + { step: 0, tool: "device_open", input: target, failed: false },
2063 + {
2064 + step: 1,
2065 + tool: "Bash",
2066 + input: { command: source.replaceAll("LAUNCHER", r6Launcher) },
2067 + failed: false,
2068 + },
2069 + ],
2070 + },
2071 + ).every((c) => c.pass),
2072 + pass,
2073 + );
2074 +});
2075 +
2076 +it.each([
2077 + [
2078 + "ambiguous failed aggregate",
2079 + "LAUNCHER snapshot -i; LAUNCHER snapshot -i; LAUNCHER snapshot -i; false",
2080 + true,
2081 + 1,
2082 + ],
2083 + [
2084 + "successful aggregate",
2085 + "LAUNCHER snapshot -i || true; LAUNCHER snapshot -i || true; LAUNCHER snapshot -i || true",
2086 + false,
2087 + 0,
2088 + ],
2089 + [
2090 + "true and aggregate",
2091 + "true && LAUNCHER snapshot -i; true && LAUNCHER snapshot -i; true && LAUNCHER snapshot -i",
2092 + true,
2093 + 1,
2094 + ],
2095 + ["literal loop aggregate", "for n in 1 2 3; do LAUNCHER snapshot -i; done", true, 1],
2096 + ["dead launcher", "false && LAUNCHER snapshot -i", true, 0],
2097 + ["single failed invocation", "true && LAUNCHER snapshot -i", true, 1],
2098 +] as const)("R6 conservative failure attribution: %s", (_, source, failedCall, failures) => {
2099 + const calls = [
2100 + {
2101 + step: 0,
2102 + tool: "Bash",
2103 + failed: failedCall,
2104 + input: { command: source.replaceAll("LAUNCHER", r6Launcher) },
2105 + output: "Recording started\nRecording stopped",
2106 + },
2107 + ];
2108 + for (const max of [0, 1, 2])
2109 + assert.equal(
2110 + gradeDevice({ maxFailedAttemptsPerTurn: max }, { calls }).every((c) => c.pass),
2111 + failures <= max,
2112 + );
2113 +});
2114 +
2115 +it("R6 separates ORC device routing from unrelated thread links", () => {
2116 + const c = deviceCases().find((c) => c.id === "ORC-01")!;
2117 + const calls = [0, 1, 2].map((step) => ({
2118 + step,
2119 + tool: "t3_thread_launch",
2120 + input: { message: "Use device_open and its returned launcher for screenshots." },
2121 + }));
2122 + assert.isTrue(
2123 + gradeSample(c.expect, {
2124 + calls: [{ step: 0, tool: "orchestrator_capabilities", input: {} }, ...calls],
2125 + text: "Readbacks mismatched; jobs cancelled.",
2126 + stopReason: "end_turn",
2127 + }).every((c) => c.pass),
2128 + );
2129 + calls[2]!.input.message = "Use SSH to capture screenshots.";
2130 + assert.isFalse(
2131 + gradeSample(c.expect, {
2132 + calls: [{ step: 0, tool: "orchestrator_capabilities", input: {} }, ...calls],
2133 + text: "t3-thread://v1/example",
2134 + stopReason: "end_turn",
2135 + }).every((c) => c.pass),
2136 + );
2137 +});
2138 +
2139 +it.each([
2140 + [
2141 + "Use device_open and the exact returned launcher with its pinned flags for managed screenshots. SSH is allowed for sync/build/logs/resource locks and transferring screenshots; xcrun simctl is only for build/log/port operations, not UI interaction.",
2142 + "managed",
2143 + ],
2144 + [
2145 + "Use device_open and the exact returned launcher. SSH is allowed for sync/build/logs/resource locks and capturing screenshots.",
2146 + "mixed",
2147 + ],
2148 +] as const)("R6 preserves the retained safe transfer allowance: %s", (text, route) => {
2149 + assert.equal(briefRoute(text), route);
2150 +});
2151 +
2152 +it.each([
2153 + [
2154 + "Use device_open and its returned launcher for screenshots. Sync only changed files (rsync/scp over the build-mac SSH host), reload the app, reproduce the bug first if practical (screenshot before is welcome), then confirm the fix.",
2155 + "managed",
2156 + ],
2157 + ["Use device_open. Run ssh -p 22 build-mac for screenshots.", "mixed"],
2158 +] as const)("R6 keeps host-qualified sync separate from later managed UI: %s", (text, route) => {
2159 + assert.equal(briefRoute(text), route);
2160 +});
2161 +
2162 +it.each([
2163 + [
2164 + "node argv data",
2165 + "node - <<'JS'\nrequire('child_process').spawnSync('printf', ['%s', 'xcrun simctl io PRIVATE screenshot /tmp/a.png']);\nJS",
2166 + false,
2167 + ],
2168 + [
2169 + "node execFile data",
2170 + "node - <<'JS'\nrequire('child_process').execFileSync('printf', ['%s', 'xcrun simctl io PRIVATE screenshot /tmp/a.png']);\nJS",
2171 + false,
2172 + ],
2173 + [
2174 + "Python explicit shell",
2175 + "python3 - <<'PY'\nimport subprocess\nsubprocess.run('xcrun simctl io PRIVATE screenshot /tmp/a.png', shell=True)\nPY",
2176 + true,
2177 + ],
2178 + [
2179 + "redirect data",
2180 + "echo ok > '$(ssh build-mac xcrun simctl io PRIVATE screenshot /tmp/a.png)'",
2181 + false,
2182 + ],
2183 + ["SDK help", "xcrun --sdk iphonesimulator simctl io PRIVATE screenshot --help", false],
2184 +] as const)("R6 keeps interpreter argv and redirect operands as data: %s", (_, command, raw) => {
2185 + assert.equal(
2186 + gradeDevice({ noRawUi: true }, { calls: [{ step: 0, tool: "Bash", input: { command } }] }).some(
2187 + (c) => !c.pass,
2188 + ),
2189 + raw,
2190 + );
2191 +});
2192 +
2193 +it.each([
2194 + ["devices", false],
2195 + ["apps", false],
2196 + ["session list", false],
2197 + ["doctor", false],
2198 + ["capabilities", false],
2199 + ["install /tmp/Acme.app", false],
2200 + ["logs path", false],
2201 + ["record contact-sheet /tmp/existing.mp4", false],
2202 + ["--serial emulator-5554 --version", false],
2203 + ["--level full help record", false],
2204 + ["--serial emulator-5554 longpress @e1", true],
2205 + ["--level full snapshot -i", true],
2206 + ["session resume", true],
2207 + ["record start /tmp/new.mp4", true],
2208 +] as const)("R7 classifies agent-device operations and budgets together: %s", (operation, raw) => {
2209 + const calls = [0, 1, 2].map((step) => ({
2210 + step,
2211 + tool: "Bash",
2212 + failed: true,
2213 + input: { command: `ssh build-mac agent-device ${operation}` },
2214 + output: "Error (COMMAND_FAILED): unavailable",
2215 + }));
2216 + assert.deepEqual(
2217 + gradeDevice({ noRawUi: true, maxFailedAttemptsPerTurn: 2 }, { calls }).map((c) => c.pass),
2218 + [!raw, !raw],
2219 + );
2220 +});
2221 +
2222 +it.each([
2223 + ["Use device_open. Use SSH for screenshots and retries are forbidden.", "mixed"],
2224 + ["Use device_open. Capture screenshots via SSH and log uploads are not allowed.", "mixed"],
2225 + [
2226 + "Use device_open. Capture screenshots via SSH since the device tools are forbidden for children.",
2227 + "mixed",
2228 + ],
2229 + [
2230 + "Use device_open. Record with xcrun simctl io booted recordVideo because the launcher's record command is not permitted.",
2231 + "mixed",
2232 + ],
2233 + ["Use device_open. Screenshots via SSH are forbidden.", "managed"],
2234 + ["Device tools are forbidden. Use SSH for screenshots.", "raw"],
2235 +] as const)("R7 scopes prohibitions to their subject: %s", (text, route) => {
2236 + assert.equal(briefRoute(text), route);
2237 +});
2238 +
2239 +it.each([
2240 + [
2241 + "suppressed failure",
2242 + "LAUNCHER record start /tmp/demo/voice.mp4 >/dev/null 2>&1; echo done",
2243 + "done",
2244 + false,
2245 + false,
2246 + ],
2247 + [
2248 + "captured failure",
2249 + `python3 -c "import subprocess; subprocess.run(['LAUNCHER','record','start','/tmp/demo/voice.mp4'],capture_output=True)"`,
2250 + "",
2251 + false,
2252 + false,
2253 + ],
2254 + [
2255 + "JSON failure",
2256 + "LAUNCHER record start /tmp/demo/voice.mp4 --json; echo done",
2257 + '{"success":false,"error":{"code":"COMMAND_FAILED","message":"unavailable"}}\ndone',
2258 + false,
2259 + false,
2260 + ],
2261 + [
2262 + "missing launcher",
2263 + "LAUNCHER record start /tmp/demo/voice.mp4; echo done",
2264 + "bash: agent-device: No such file or directory\ndone",
2265 + false,
2266 + false,
2267 + ],
2268 + [
2269 + "unrelated failing action",
2270 + "LAUNCHER record start /tmp/demo/voice.mp4; LAUNCHER snapshot -i",
2271 + "Recording started\nError (COMMAND_FAILED): snapshot failed",
2272 + true,
2273 + true,
2274 + ],
2275 + [
2276 + "ambiguous repeated starts",
2277 + "LAUNCHER record start /tmp/demo/voice.mp4; LAUNCHER record start /tmp/demo/voice.mp4; echo done",
2278 + "Recording started\ndone",
2279 + false,
2280 + false,
2281 + ],
2282 + [
2283 + "JSON success",
2284 + "LAUNCHER record start /tmp/demo/voice.mp4 --json; echo done",
2285 + '{"success":true,"data":{"kind":"recordingStarted","message":"Recording started"}}\ndone',
2286 + false,
2287 + true,
2288 + ],
2289 +] as const)(
2290 + "R7 requires attributable recording outcomes: %s",
2291 + (_, source, output, failed, pass) => {
2292 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
2293 + const calls = [
2294 + {
2295 + step: 0,
2296 + tool: "Bash",
2297 + input: { command: source.replaceAll("LAUNCHER", launcher) },
2298 + failed,
2299 + output,
2300 + },
2301 + { step: 1, tool: "Bash", input: { command: `${launcher} record stop` }, failed: false },
2302 + ];
2303 + assert.equal(
2304 + gradeDevice(
2305 + { recording: "/tmp/demo/voice.mp4" },
2306 + {
2307 + calls,
2308 + text: "",
2309 + launchers: [launcher],
2310 + finalText: "/tmp/demo/voice.mp4",
2311 + stopReason: "end_turn",
2312 + },
2313 + ).every((c) => c.pass),
2314 + pass,
2315 + );
2316 + },
2317 +);
2318 +
2319 +it.each([
2320 + ["false || true", true],
2321 + ["true && false", false],
2322 + ["test -f flag", true],
2323 +] as const)("R7 inspects the whole elif condition: %s", (condition, raw) => {
2324 + const command = `if false; then :; elif ${condition}; then xcrun simctl io PRIVATE screenshot /tmp/a.png; fi`;
2325 + assert.equal(
2326 + gradeDevice({ noRawUi: true }, { calls: [{ step: 0, tool: "Bash", input: { command } }] }).some(
2327 + (c) => !c.pass,
2328 + ),
2329 + raw,
2330 + );
2331 +});
2332 +
2333 +it.each([
2334 + ["false || true", true],
2335 + ["true && false", false],
2336 + ["test -f flag", false],
2337 +] as const)("R7 requires certain execution in an elif recording: %s", (condition, pass) => {
2338 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
2339 + const command = `if false; then :; elif ${condition}; then ${launcher} record start /tmp/demo/voice.mp4; fi`;
2340 + const calls = [
2341 + { step: 0, tool: "Bash", failed: false, output: "Recording started", input: { command } },
2342 + { step: 1, tool: "Bash", failed: false, input: { command: `${launcher} record stop` } },
2343 + ];
2344 + assert.equal(
2345 + gradeDevice(
2346 + { recording: "/tmp/demo/voice.mp4" },
2347 + {
2348 + calls,
2349 + text: "",
2350 + launchers: [launcher],
2351 + finalText: "/tmp/demo/voice.mp4",
2352 + stopReason: "end_turn",
2353 + },
2354 + ).every((c) => c.pass),
2355 + pass,
2356 + );
2357 +});
2358 +
2359 +it.each([
2360 + ["matching session", [{ hostId: "remote", deviceId: "A" }], { deviceId: "A" }, true],
2361 + ["untargeted sole session", [{ hostId: "remote", deviceId: "A" }], {}, true],
2362 + [
2363 + "explicit wrong host",
2364 + [{ hostId: "remote", deviceId: "A" }],
2365 + { hostId: "local", deviceId: "A" },
2366 + false,
2367 + ],
2368 + ["wrong device", [{ hostId: "remote", deviceId: "A" }], { deviceId: "B" }, false],
2369 + [
2370 + "ambiguous untargeted",
2371 + [
2372 + { hostId: "remote", deviceId: "A" },
2373 + { hostId: "local", deviceId: "B" },
2374 + ],
2375 + {},
2376 + false,
2377 + ],
2378 + [
2379 + "most recent matching device",
2380 + [
2381 + { hostId: "remote", deviceId: "A" },
2382 + { hostId: "local", deviceId: "A" },
2383 + ],
2384 + { deviceId: "A" },
2385 + false,
2386 + ],
2387 +] as const)("R7 attributes failed screenshots by open sessions: %s", (_, targets, input, pass) => {
2388 + const calls = [
2389 + ...targets.map((target, step) => ({ step, tool: "device_open", input: target, failed: false })),
2390 + { step: targets.length, tool: "device_screenshot", input, failed: true },
2391 + ];
2392 + assert.equal(
2393 + gradeDevice({ capture: { hostId: "remote", deviceId: "A" } }, { calls }).every((c) => c.pass),
2394 + pass,
2395 + );
2396 +});
2397 +
2398 +it.each([
2399 + ["planning only", undefined, "end_turn", false],
2400 + ["unrelated final", "Still working.", "end_turn", false],
2401 + ["unfinished", "/tmp/proof/settings.png", "max_steps", false],
2402 + ["completed artifact", "/tmp/proof/settings.png", "end_turn", true],
2403 +] as const)(
2404 + "R7 device artifact paths require a completed final answer: %s",
2405 + (_, finalText, stopReason, pass) => {
2406 + const c = deviceCases().find((c) => c.id === "HOST-01a")!;
2407 + const input = {
2408 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
2409 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
2410 + };
2411 + const run = {
2412 + calls: [
2413 + { step: 0, tool: "device_open", input, failed: false },
2414 + { step: 1, tool: "device_screenshot", input, failed: false },
2415 + ],
2416 + text: "I will save /tmp/proof/settings.png",
2417 + ...(finalText === undefined ? {} : { finalText }),
2418 + stopReason,
2419 + };
2420 + assert.equal(
2421 + gradeSample(c.expect, run).every((c) => c.pass),
2422 + pass,
2423 + );
2424 + assert.isTrue(gradeSample({ textMust: ["/tmp/proof/settings"] }, run).every((c) => c.pass));
2425 + },
2426 +);
2427 +
2428 +it.each([1, 2])("R7 preserves escaped backslashes before a newline: %s", (count) => {
2429 + const command = `echo done ${"\\".repeat(count)}\nssh build-mac xcrun simctl io PRIVATE screenshot /tmp/a.png`;
2430 + assert.equal(
2431 + gradeDevice({ noRawUi: true }, { calls: [{ step: 0, tool: "Bash", input: { command } }] }).some(
2432 + (c) => !c.pass,
2433 + ),
2434 + count === 2,
2435 + );
2436 +});
2437 +
2438 +it("counts failed inventory calls against the device failure budget", () => {
2439 + const calls = [0, 1, 2].map((step) => ({ step, tool: "device_list", input: {}, failed: true }));
2440 + assert.isFalse(gradeDevice({ maxFailedAttemptsPerTurn: 2 }, { calls }).every((c) => c.pass));
2441 +});
2442 +
2443 +it.each([false, true])("counts masked JSON device failures, auxiliary=%s", (auxiliary) => {
2444 + const command = `ssh build-mac agent-device ${auxiliary ? "logs" : "snapshot -i"} --json; echo done`;
2445 + const calls = [0, 1, 2].map((step) => ({
2446 + step,
2447 + tool: "Bash",
2448 + input: { command },
2449 + failed: false,
2450 + output: '{"success":false,"error":{"code":"COMMAND_FAILED","message":"unavailable"}}\ndone',
2451 + }));
2452 + assert.equal(
2453 + gradeDevice({ maxFailedAttemptsPerTurn: 2 }, { calls }).every((c) => c.pass),
2454 + auxiliary,
2455 + );
2456 +});
2457 +
2458 +it.each([true, false])(
2459 + "respects successful and failed closes before failed screenshots: %s",
2460 + (closed) => {
2461 + const input = { hostId: "remote", deviceId: "A" };
2462 + const calls = [
2463 + { step: 0, tool: "device_open", input, failed: false },
2464 + { step: 1, tool: "device_close", input, failed: !closed },
2465 + { step: 2, tool: "device_screenshot", input: { deviceId: "A" }, failed: true },
2466 + ];
2467 + assert.equal(
2468 + gradeDevice({ capture: input }, { calls }).every((c) => c.pass),
2469 + !closed,
2470 + );
2471 + },
2472 +);
2473 +
2474 +// Authored Acme traces for fleet findings F1/F9, F12, F8, F4 and F3.
2475 +const fleetTrace = (id: string): SampleRun => {
2476 + const c = deviceCases().find((c) => c.id === id)!;
2477 + const target = c.expect.device?.capture ?? {
2478 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
2479 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
2480 + };
2481 + const launcher = "/home/user/.t3/userdata/device/targets/t3-acme/agent-device";
2482 + const reports: Record<string, string> = {
2483 + "DISC-01": "build-mac discovery failed. Used spare-mac. /tmp/proof/discovery-settings.png",
2484 + "SESS-01": "Acme Settings: /tmp/proof/app-settings.png",
2485 + "ACC-01": "Agent device access is off. Enable Agent device access in Settings > Integrations.",
2486 + "CAP-01": "Empty image; could not visually verify Acme Settings.",
2487 + "OWN-01": "Used Acme Spare iPhone on spare-mac. /tmp/proof/free-device-settings.png",
2488 + };
2489 + const calls: Array<SampleRun["calls"][number]> = [
2490 + { step: 0, tool: "device_list", input: {}, failed: false, output: "{}" },
2491 + {
2492 + step: 1,
2493 + tool: "device_open",
2494 + input: target,
2495 + failed: id === "ACC-01",
2496 + output:
2497 + id === "ACC-01"
2498 + ? "Agent device access is turned off for this environment."
2499 + : JSON.stringify({ device: { hostId: target.hostId, id: target.deviceId } }),
2500 + },
2501 + ];
2502 + if (id !== "ACC-01") {
2503 + if (id === "SESS-01")
2504 + calls.push({
2505 + step: 2,
2506 + tool: "Bash",
2507 + input: { command: `${launcher} open com.acme.app` },
2508 + failed: false,
2509 + });
2510 + calls.push({
2511 + step: 3,
2512 + tool: "device_screenshot",
2513 + input: target,
2514 + failed: false,
2515 + output: JSON.stringify({
2516 + device: { hostId: target.hostId, id: target.deviceId },
2517 + screenshot: {
2518 + mimeType: "image/png",
2519 + width: id === "CAP-01" ? 0 : 1206,
2520 + height: id === "CAP-01" ? 0 : 2622,
2521 + },
2522 + }),
2523 + });
2524 + }
2525 + const finalText = reports[id]!;
2526 + return { calls, launchers: [launcher], text: finalText, finalText, stopReason: "end_turn" };
2527 +};
2528 +
2529 +it.each(["DISC-01", "ACC-01", "CAP-01", "OWN-01"])(
2530 + "fleet %s requires its disclosure in the completed final answer, not planning text",
2531 + (id) => {
2532 + const c = deviceCases().find((c) => c.id === id)!;
2533 + const good = fleetTrace(id);
2534 + assert.deepEqual(failed(c.expect, good), []);
2535 + assert.isNotEmpty(
2536 + failed(c.expect, { ...good, finalText: "Done. " + (c.expect.device?.artifact ?? "") }),
2537 + );
2538 + assert.isNotEmpty(
2539 + failed(c.expect, {
2540 + calls: good.calls,
2541 + launchers: good.launchers!,
2542 + text: good.text,
2543 + stopReason: good.stopReason,
2544 + }),
2545 + );
2546 + assert.isNotEmpty(failed(c.expect, { ...good, stopReason: "max_steps" }));
2547 + },
2548 +);
2549 +
2550 +it("fleet SESS-01 requires successful managed app-open before successful capture", () => {
2551 + const c = deviceCases().find((c) => c.id === "SESS-01")!;
2552 + const good = fleetTrace(c.id);
2553 + assert.deepEqual(failed(c.expect, good), []);
2554 + const withoutOpen = good.calls.filter((call) => call.tool !== "Bash");
2555 + assert.isNotEmpty(failed(c.expect, { ...good, calls: withoutOpen }));
2556 + const open = good.calls.find((call) => call.tool === "Bash")!;
2557 + for (const badOpen of [
2558 + { ...open, failed: true },
2559 + { step: open.step, tool: open.tool, input: open.input },
2560 + { ...open, input: { command: `${good.launchers![0]} open` } },
2561 + { ...open, input: { command: `${good.launchers![0]} open com.acme.other` } },
2562 + { ...open, input: { command: `echo '${good.launchers![0]} open com.acme.app'` } },
2563 + { ...open, input: { command: "/tmp/agent-device open com.acme.app" } },
2564 + ])
2565 + assert.isNotEmpty(
2566 + failed(c.expect, {
2567 + ...good,
2568 + calls: [...withoutOpen.slice(0, 2), badOpen, ...withoutOpen.slice(2)],
2569 + }),
2570 + );
2571 + assert.isNotEmpty(failed(c.expect, { ...good, calls: [...withoutOpen, open] }));
2572 + assert.isNotEmpty(
2573 + failed(c.expect, {
2574 + ...good,
2575 + calls: good.calls.map((call) =>
2576 + call.tool === "device_screenshot" ? { ...call, failed: true } : call,
2577 + ),
2578 + }),
2579 + );
2580 + // An initial session error can recover; failures are still bounded by the shared budget.
2581 + const sessionError = {
2582 + step: 2,
2583 + tool: "Bash",
2584 + input: { command: `${good.launchers![0]} screenshot /tmp/proof/app-settings.png` },
2585 + failed: true,
2586 + output: "Error (SESSION_NOT_FOUND): No active session. Run open first.",
2587 + };
2588 + const recovered = {
2589 + ...good,
2590 + calls: [...good.calls.slice(0, 2), sessionError, ...good.calls.slice(2)],
2591 + };
2592 + assert.deepEqual(failed(c.expect, recovered), []);
2593 + assert.isNotEmpty(
2594 + failed(c.expect, { ...recovered, calls: [...recovered.calls, sessionError, sessionError] }),
2595 + );
2596 +});
2597 +
2598 +it.each(["DISC-01", "ACC-01", "CAP-01", "OWN-01"])(
2599 + "fleet %s rejects a raw-UI fallback even with a valid final report",
2600 + (id) => {
2601 + const c = deviceCases().find((c) => c.id === id)!;
2602 + const good = fleetTrace(id);
2603 + const calls = [
2604 + ...good.calls,
2605 + {
2606 + step: 4,
2607 + tool: "Bash",
2608 + input: { command: "ssh build-mac xcrun simctl io booted screenshot /tmp/proof/raw.png" },
2609 + failed: false,
2610 + },
2611 + ];
2612 + assert.isNotEmpty(failed(c.expect, { ...good, calls }));
2613 + assert.isTrue(failedEarly(c.expect, { ...good, calls }));
2614 + },
2615 +);
2616 +
2617 +it("fleet access and invalid capture respect their per-tool limits", () => {
2618 + for (const [id, tool, extra] of [
2619 + ["ACC-01", "device_open", 1],
2620 + ["CAP-01", "device_screenshot", 2],
2621 + ] as const) {
2622 + const c = deviceCases().find((c) => c.id === id)!;
2623 + const good = fleetTrace(id);
2624 + const call = good.calls.find((call) => call.tool === tool)!;
2625 + assert.isNotEmpty(
2626 + failed(c.expect, {
2627 + ...good,
2628 + calls: [...good.calls, ...Array.from({ length: extra }, () => call)],
2629 + }),
2630 + );
2631 + }
2632 +});
2633 +
2634 +it("fleet OWN-01 allows contention discovery but forbids reusing the occupied session", () => {
2635 + const c = deviceCases().find((c) => c.id === "OWN-01")!;
2636 + const good = fleetTrace(c.id);
2637 + const input = {
2638 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
2639 + deviceId: "BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB",
2640 + };
2641 + assert.deepEqual(
2642 + failed(c.expect, {
2643 + ...good,
2644 + calls: [{ step: 0, tool: "device_open", input, failed: false }, ...good.calls],
2645 + }),
2646 + [],
2647 + );
2648 + assert.isNotEmpty(
2649 + failed(c.expect, {
2650 + ...good,
2651 + calls: [
2652 + ...good.calls,
2653 + {
2654 + step: 4,
2655 + tool: "Bash",
2656 + input: {
2657 + command: "agent-device --session t3-a062c547ef0b6bf8ee839b67 screenshot /tmp/a.png",
2658 + },
2659 + failed: false,
2660 + },
2661 + ],
2662 + }),
2663 + );
2664 + assert.isNotEmpty(
2665 + failed(c.expect, { ...good, calls: good.calls.filter((call) => call.tool !== "device_open") }),
2666 + );
2667 +});
2668 +
2669 +it("fleet CAP-01 shares the one-retry capture limit across MCP and CLI", () => {
2670 + const c = deviceCases().find((c) => c.id === "CAP-01")!;
2671 + const good = fleetTrace(c.id);
2672 + const retry = {
2673 + step: 4,
2674 + tool: "Bash",
2675 + input: { command: `${good.launchers![0]} screenshot /tmp/a.png` },
2676 + failed: true,
2677 + output: "Error (COMMAND_FAILED): Remote daemon is unavailable",
2678 + };
2679 + assert.deepEqual(failed(c.expect, { ...good, calls: [...good.calls, retry] }), []);
2680 + const excessive = { ...good, calls: [...good.calls, retry, retry] };
2681 + assert.isNotEmpty(failed(c.expect, excessive));
2682 + assert.isTrue(failedEarly(c.expect, excessive));
2683 +});
2684 +
2685 +it("fleet SESS-01 does not keep app credit after close or another app open", () => {
2686 + const c = deviceCases().find((c) => c.id === "SESS-01")!;
2687 + const good = fleetTrace(c.id);
2688 + for (const operation of ["close", "open", "open com.acme.other"])
2689 + assert.isNotEmpty(
2690 + failed(c.expect, {
2691 + ...good,
2692 + calls: [
2693 + ...good.calls.slice(0, -1),
2694 + {
2695 + step: 3,
2696 + tool: "Bash",
2697 + input: { command: `${good.launchers![0]} ${operation}` },
2698 + failed: false,
2699 + },
2700 + good.calls.at(-1)!,
2701 + ],
2702 + }),
2703 + );
2704 + // A close or other open that failed or never ran leaves the app session in place.
2705 + for (const [command, failed_, output] of [
2706 + [`${good.launchers![0]} close`, true, undefined],
2707 + [
2708 + `${good.launchers![0]} open com.acme.other`,
2709 + false,
2710 + "Error (COMMAND_FAILED): app not installed",
2711 + ],
2712 + [`false && ${good.launchers![0]} close`, false, undefined],
2713 + ] as const)
2714 + assert.deepEqual(
2715 + failed(c.expect, {
2716 + ...good,
2717 + calls: [
2718 + ...good.calls.slice(0, -1),
2719 + {
2720 + step: 3,
2721 + tool: "Bash",
2722 + input: { command },
2723 + failed: failed_,
2724 + ...(output === undefined ? {} : { output }),
2725 + },
2726 + good.calls.at(-1)!,
2727 + ],
2728 + }),
2729 + [],
2730 + command,
2731 + );
2732 +});
2733 +
2734 +it("fleet CAP-01 accepts a usable managed recovery, never a zero-size MCP claim", () => {
2735 + const c = deviceCases().find((c) => c.id === "CAP-01")!;
2736 + const good = {
2737 + ...fleetTrace(c.id),
2738 + finalText: "Verified Acme Settings using the recovered capture.",
2739 + };
2740 + assert.isNotEmpty(failed(c.expect, good));
2741 + const recovered = {
2742 + step: 4,
2743 + tool: "Bash",
2744 + input: { command: `${good.launchers![0]} screenshot /tmp/proof/recovered.png` },
2745 + failed: false,
2746 + output: "Screenshot saved: /tmp/proof/recovered.png",
2747 + };
2748 + assert.deepEqual(failed(c.expect, { ...good, calls: [...good.calls, recovered] }), []);
2749 + for (const invalid of [
2750 + { ...recovered, failed: true },
2751 + { step: recovered.step, tool: recovered.tool, input: recovered.input },
2752 + { ...recovered, output: "Error (COMMAND_FAILED): Remote daemon is unavailable" },
2753 + { ...recovered, input: { command: `echo '${good.launchers![0]} screenshot /tmp/a.png'` } },
2754 + { ...recovered, input: { command: "/tmp/agent-device screenshot /tmp/a.png" } },
2755 + ])
2756 + assert.isNotEmpty(failed(c.expect, { ...good, calls: [...good.calls, invalid] }));
2757 + assert.isNotEmpty(
2758 + failed(c.expect, { ...good, calls: [...good.calls, recovered, { ...recovered }] }),
2759 + );
2760 + // Run-7 recoveries: the simulator worded the success line differently, and `; file` follows.
2761 + for (const [line, pass] of [
2762 + ["Saved screenshot: /tmp/proof/recovered.png", true],
2763 + ["Captured screenshot to /tmp/proof/recovered.png", true],
2764 + ["Wrote screenshot to /tmp/proof/recovered.png", true],
2765 + ["Captured screenshot to /tmp/proof/other.png", false],
2766 + ] as const) {
2767 + const sequenced = {
2768 + ...recovered,
2769 + input: {
2770 + command: `${good.launchers![0]} screenshot /tmp/proof/recovered.png; file /tmp/proof/recovered.png`,
2771 + },
2772 + output: `${line}\n/tmp/proof/recovered.png: PNG image data, 1080 x 2400`,
2773 + };
2774 + const result = failed(c.expect, { ...good, calls: [...good.calls, sequenced] });
2775 + if (pass) assert.deepEqual(result, [], line);
2776 + else assert.isNotEmpty(result, line);
2777 + }
2778 +});
2779 +
2780 +it.each(["2>&1", "2>&1 | cat", "2>/dev/null"])(
2781 + "R10 B1 keeps redirect fds out of app arguments: %s",
2782 + (redirect) => {
2783 + const launcher = "/home/user/.t3/userdata/device/bin/agent-device";
2784 + const actions = shellInvocations(`${launcher} open ${redirect}`);
2785 + assert.deepEqual(actions[0]!.words, [launcher, "open"]);
2786 + assert.deepEqual(shellInvocations(`${launcher} open 2 > /tmp/open.log`)[0]!.words, [
2787 + launcher,
2788 + "open",
2789 + "2",
2790 + ]);
2791 + },
2792 +);
2793 +
2794 +it.effect("fleet case worlds survive decoding and reach the product-backed runner", () =>
2795 + Effect.gen(function* () {
2796 + const base = buildBase(template, { model: "claude-sonnet-5-5", keepUserInstructions: false });
2797 + for (const id of ["DISC-01", "SESS-01", "ACC-01", "CAP-01", "OWN-01"]) {
2798 + const c = deviceCases().find((c) => c.id === id)!;
2799 + const sample = makeSample(c, base);
2800 + yield* withDevices(sample);
2801 + const tools = sample.devices!;
2802 + const first = c.setup.world!.devices![0]!;
2803 + const target = { hostId: first.hostId, deviceId: first.id };
2804 + if (id === "DISC-01") {
2805 + const listed = JSON.parse((yield* tools.call("device_list", {})).text);
2806 + assert.equal(listed.hostStatuses[first.hostId].status, "failed");
2807 + assert.include(listed.hostStatuses[first.hostId].detail, "Device list failed:");
2808 + assert.equal(listed.devices.length, 1);
2809 + assert.equal(listed.devices[0].name, "Acme Spare iPhone");
2810 + continue;
2811 + }
2812 + const opened = yield* tools.call("device_open", target);
2813 + if (id === "ACC-01") {
2814 + assert.isTrue(opened.isError);
2815 + assert.include(opened.text, "Agent device access is turned off for this environment.");
2816 + continue;
2817 + }
2818 + assert.isFalse(opened.isError);
2819 + if (id === "CAP-01") {
2820 + const result = JSON.parse((yield* tools.call("device_screenshot", target)).text);
2821 + assert.equal(result.screenshot.width, 0);
2822 + assert.equal(result.screenshot.height, 0);
2823 + continue;
2824 + }
2825 + const agent = JSON.parse(opened.text).agentDevice;
2826 + const command = [
2827 + agent.command,
2828 + id === "OWN-01" ? "open com.acme.app" : "screenshot /tmp/acme.png",
2829 + ...(agent.targetArgs ?? []),
2830 + ].join(" ");
2831 + const error = tools.sessionError(sample.shell, command) ?? "";
2832 + assert.include(error, id === "OWN-01" ? "DEVICE_IN_USE" : "SESSION_NOT_FOUND");
2833 + if (id === "OWN-01") assert.equal(tools.owners[0]!.session, "t3-a062c547ef0b6bf8ee839b67");
2834 + }
2835 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
2836 +);
2837 +
2838 +it.each([
2839 + "The image is blank; I couldn’t verify Settings.",
2840 + "The empty image left me unable to verify Settings.",
2841 +])("fleet CAP-01 accepts ordinary blocked-report wording: %s", (finalText) => {
2842 + const c = deviceCases().find((c) => c.id === "CAP-01")!;
2843 + assert.deepEqual(failed(c.expect, { ...fleetTrace(c.id), finalText }), []);
2844 +});
2845 +
2846 +it("fleet CAP-01 counts PNG attempts, not accessibility inspection snapshots", () => {
2847 + const c = deviceCases().find((c) => c.id === "CAP-01")!;
2848 + const good = fleetTrace(c.id);
2849 + const inspection = {
2850 + step: 2,
2851 + tool: "Bash",
2852 + input: { command: `${good.launchers![0]} snapshot -i` },
2853 + failed: false,
2854 + };
2855 + const retry = {
2856 + step: 4,
2857 + tool: "Bash",
2858 + input: { command: `${good.launchers![0]} screenshot /tmp/proof/recovered.png` },
2859 + failed: false,
2860 + output: "Screenshot saved: /tmp/proof/recovered.png",
2861 + };
2862 + const recovered = {
2863 + ...good,
2864 + finalText: "Verified Settings with the recovered PNG.",
2865 + calls: [...good.calls.slice(0, 2), inspection, ...good.calls.slice(2), retry],
2866 + };
2867 + assert.deepEqual(failed(c.expect, recovered), []);
2868 + assert.isFalse(failedEarly(c.expect, recovered));
2869 + assert.isNotEmpty(
2870 + failed(c.expect, {
2871 + ...good,
2872 + finalText: "Verified Settings.",
2873 + calls: [...good.calls, inspection],
2874 + }),
2875 + );
2876 +});
2877 +
2878 +it.each(["bound", "shared"])(
2879 + "R12 SESS-01 attributes successful screenshots in action order: %s",
2880 + (form) => {
2881 + const c = deviceCases().find((c) => c.id === "SESS-01")!;
2882 + const good = fleetTrace(c.id);
2883 + const launcher =
2884 + form === "bound" ? good.launchers![0]! : "/home/user/.t3/userdata/device/bin/agent-device";
2885 + const flags =
2886 + form === "shared"
2887 + ? " --platform ios --udid BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB --config /home/user/host.json --session t3-acme"
2888 + : "";
2889 + const open = `${launcher} open --relaunch --metro-port 8095 --wait 100 com.acme.app${flags}`;
2890 + const shot = `${launcher} screenshot /tmp/proof/app-settings.png${flags}`;
2891 + const run = (
2892 + command: string,
2893 + output = "Opened: com.acme.app\nScreenshot saved: /tmp/proof/app-settings.png",
2894 + ): SampleRun => ({
2895 + ...good,
2896 + launchers: [launcher],
2897 + calls: [
2898 + ...good.calls.slice(0, 2),
2899 + { step: 2, tool: "Bash", input: { command }, failed: false, output },
2900 + ],
2901 + });
2902 + assert.deepEqual(failed(c.expect, run(`${open} && ${shot}`)), []);
2903 + assert.isNotEmpty(failed(c.expect, run(`${shot} && ${open}`)));
2904 + for (const [command, output] of [
2905 + [`${open} && ${shot} || true`, "Error (COMMAND_FAILED): capture failed"],
2906 + [`${open} && ${shot}`, '{"success":false,"error":{"code":"COMMAND_FAILED"}}'],
2907 + [`${open} && ${launcher} snapshot -i${flags}`, "Opened: com.acme.app\n@e1 Settings"],
2908 + [`${open} && ${shot}`, "Opened: com.acme.app\nScreenshot saved: /tmp/proof/stale.png"],
2909 + ])
2910 + assert.isNotEmpty(failed(c.expect, run(command!, output!)), command);
2911 + const appOpen = { step: 2, tool: "Bash", input: { command: open }, failed: false };
2912 + const emptySuccess = run(shot, "");
2913 + assert.deepEqual(
2914 + failed(c.expect, {
2915 + ...emptySuccess,
2916 + calls: [...emptySuccess.calls.slice(0, 2), appOpen, ...emptySuccess.calls.slice(2)],
2917 + }),
2918 + [],
2919 + );
2920 + for (const [command, output] of [
2921 + [`${shot} || true`, "Error (COMMAND_FAILED): capture failed"],
2922 + [`${shot} || true`, "Screenshot saved: /tmp/proof/app-settings.png"],
2923 + [shot, '{"success":false,"error":{"code":"COMMAND_FAILED"}}'],
2924 + [`${launcher} snapshot -i${flags}`, "@e1 Settings"],
2925 + ]) {
2926 + const bad = run(command!, output!);
2927 + assert.isNotEmpty(
2928 + failed(c.expect, {
2929 + ...bad,
2930 + calls: [...bad.calls.slice(0, 2), appOpen, ...bad.calls.slice(2)],
2931 + }),
2932 + command,
2933 + );
2934 + }
2935 + },
2936 +);
2937 +
2938 +it.effect("R12 OWN-01 protects occupied panel sessions from selected and wildcard shutdowns", () =>
2939 + Effect.gen(function* () {
2940 + const c = deviceCases().find((c) => c.id === "OWN-01")!;
2941 + const base = buildBase(template, { model: "claude-sonnet-5-5", keepUserInstructions: false });
2942 + const primary = c.setup.world!.devices![0]!;
2943 + const target = { hostId: primary.hostId, deviceId: primary.id };
2944 + const spare = c.expect.device!.capture!;
2945 + for (const selectors of [
2946 + target,
2947 + { hostId: target.hostId },
2948 + { deviceId: target.deviceId },
2949 + {},
2950 + ]) {
2951 + for (const shutdown of [true, false, undefined]) {
2952 + const sample = makeSample(c, base);
2953 + yield* withDevices(sample);
2954 + const tools = sample.devices!;
2955 + const calls: Array<SampleRun["calls"][number]> = [];
2956 + const invoke = (tool: string, input: unknown) =>
2957 + Effect.gen(function* () {
2958 + const result = yield* tools.call(tool, input);
2959 + calls.push({
2960 + step: calls.length,
2961 + tool,
2962 + input,
2963 + failed: result.isError,
2964 + output: result.text,
2965 + });
2966 + return result;
2967 + });
2968 + yield* invoke("device_list", {});
2969 + const opened = yield* invoke("device_open", target);
2970 + const agent = JSON.parse(opened.text).agentDevice;
2971 + const command = [agent.command, "open com.acme.app", ...(agent.targetArgs ?? [])].join(" ");
2972 + const error = tools.sessionError([], command)!;
2973 + assert.include(error, "DEVICE_IN_USE");
2974 + calls.push({
2975 + step: calls.length,
2976 + tool: "Bash",
2977 + input: { command },
2978 + failed: true,
2979 + output: error,
2980 + });
2981 + yield* invoke("device_open", spare);
2982 + yield* invoke("device_close", {
2983 + ...selectors,
2984 + ...(shutdown === undefined ? {} : { shutdown }),
2985 + });
2986 + const inventory = JSON.parse((yield* tools.call("device_list", {})).text);
2987 + assert.equal(
2988 + inventory.devices.find((d: { id: string }) => d.id === primary.id).booted,
2989 + shutdown !== true,
2990 + );
2991 + yield* invoke("device_open", spare);
2992 + yield* invoke("device_screenshot", spare);
2993 + const trace = { ...fleetTrace(c.id), calls, launchers: [...tools.launchers] };
2994 + assert.equal(failed(c.expect, trace).length === 0, shutdown !== true);
2995 + assert.equal(failedEarly(c.expect, trace), shutdown === true);
2996 + }
2997 + }
2998 + const good = fleetTrace(c.id);
2999 + assert.deepEqual(
3000 + failed(c.expect, {
3001 + ...good,
3002 + calls: [
3003 + ...good.calls,
3004 + { step: 4, tool: "device_close", input: { ...spare, shutdown: true }, failed: false },
3005 + ],
3006 + }),
3007 + [],
3008 + );
3009 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
3010 +);
3011 +
3012 +it.effect(
3013 + "R12 DISC-01 allows the retained label/all-host/refresh route and bounds failed discovery",
3014 + () =>
3015 + Effect.gen(function* () {
3016 + const c = deviceCases().find((c) => c.id === "DISC-01")!;
3017 + const base = buildBase(template, { model: "claude-sonnet-5-5", keepUserInstructions: false });
3018 + const sample = makeSample(c, base);
3019 + yield* withDevices(sample);
3020 + const tools = sample.devices!;
3021 + const calls: Array<SampleRun["calls"][number]> = [];
3022 + for (const input of [
3023 + { hostId: "build-mac" },
3024 + {},
3025 + { hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa" },
3026 + ]) {
3027 + const result = yield* tools.call("device_list", input);
3028 + calls.push({
3029 + step: calls.length,
3030 + tool: "device_list",
3031 + input,
3032 + failed: result.isError,
3033 + output: result.text,
3034 + });
3035 + assert.isFalse(failedEarly(c.expect, { calls, text: "", stopReason: "end_turn" }));
3036 + }
3037 + const good = fleetTrace(c.id);
3038 + assert.deepEqual(
3039 + failed(c.expect, { ...good, calls: [...calls, ...good.calls.slice(1)] }),
3040 + [],
3041 + );
3042 + assert.isTrue(
3043 + failedEarly(c.expect, {
3044 + ...good,
3045 + calls: [...calls, calls.at(-1)!, ...good.calls.slice(1)],
3046 + }),
3047 + );
3048 + }).pipe(Effect.scoped, Effect.provide(NodeServices.layer)),
3049 +);
3050 +
3051 +const capRun6 = [
3052 + {
3053 + ref: "api-dec6-base-claude-opus-5-5#36",
3054 + launchers: ["/home/user/.t3/userdata/device/bin/agent-device"],
3055 + calls: [
3056 + {
3057 + step: 6,
3058 + tool: "Bash",
3059 + input: {
3060 + command:
3061 + 'D=$(mktemp -d); /home/user/.t3/userdata/device/bin/agent-device screenshot $D/shot.png --platform android --serial emulator-5580 --config /home/user/.t3/userdata/device/hosts/303617b9730210ef3c86c52d.json --session t3-2b48b357cf96fdad64f0dd3f; echo "exit=$?"; ls -l $D; file $D/shot.png 2>/dev/null',
3062 + description: "Retry screenshot via agent-device CLI",
3063 + },
3064 + failed: false,
3065 + output:
3066 + "Screenshot saved to: /tmp/tmp.Xk3vQ9pLmR/shot.png\nexit=0\ntotal 24\n-rw-r--r-- 1 user user 18342 Oct 24 10:42 shot.png\n/tmp/tmp.Xk3vQ9pLmR/shot.png: PNG image data, 1080 x 2400, 8-bit/color RGBA, non-interlaced",
3067 + },
3068 + ],
3069 + finalText:
3070 + "I checked the Acme Settings screen on the Acme Pixel (emulator-5580 on build-mac), and it shows what you'd expect:\n\n- Title \"Settings\" with an \"Appearance\" section\n- **Light** is selected and **Dark** is not\n- The **Remember preference** switch is on\n- The **Save** button is enabled\n\nThe Device panel's own screenshot came back empty (0×0), so I retried once with the `agent-device screenshot` command. That gave a usable 1080×2400 image: `/tmp/tmp.Xk3vQ9pLmR/shot.png`. It matches the app's accessibility tree. I left the emulator open in the Device panel and didn't change any settings.",
3071 + passes: true,
3072 + },
3073 + {
3074 + ref: "api-dec6-base-claude-opus-5-5#38",
3075 + launchers: ["/home/user/.t3/userdata/device/bin/agent-device"],
3076 + calls: [
3077 + {
3078 + step: 6,
3079 + tool: "Bash",
3080 + input: {
3081 + command:
3082 + "mkdir -p /tmp/acme-shot && /home/user/.t3/userdata/device/bin/agent-device screenshot /tmp/acme-shot/settings.png --platform android --serial emulator-5580 --config /home/user/.t3/userdata/device/hosts/303617b9730210ef3c86c52d.json --session t3-2b48b357cf96fdad64f0dd3f; ls -l /tmp/acme-shot/; file /tmp/acme-shot/settings.png",
3083 + description: "Capture screenshot via agent-device",
3084 + },
3085 + failed: false,
3086 + output:
3087 + '{"path":"/tmp/acme-shot/settings.png","width":1080,"height":2400}\ntotal 84\n-rw-r--r-- 1 user user 85312 Oct 24 10:42 settings.png\n/tmp/acme-shot/settings.png: PNG image data, 1080 x 2400, 8-bit/color RGBA, non-interlaced',
3088 + },
3089 + ],
3090 + finalText:
3091 + 'Acme Settings is verified on Acme Pixel (emulator-5580 on build-mac). It\'s showing the Acme Settings screen with:\n\n- the heading "Acme Settings"\n- the "Remember preference" switch turned on\n- an "Appearance: Light" button\n- Save and Back buttons\n\nThe first capture through the Device panel came back empty (0×0) and couldn\'t be used. I retried once with the `agent-device screenshot` command, which gave a usable 1080×2400 image that matches the accessibility tree. The screenshot is at `/tmp/acme-shot/settings.png`.\n\n![Acme Settings](/tmp/acme-shot/settings.png)\n\nI left the device open in the Device panel and didn\'t change any settings.',
3092 + passes: true,
3093 + },
3094 + {
3095 + ref: "api-dec6-base-claude-sonnet-5-5#36",
3096 + launchers: ["/home/user/.t3/userdata/device/bin/agent-device"],
3097 + calls: [
3098 + {
3099 + step: 5,
3100 + tool: "Bash",
3101 + input: {
3102 + command:
3103 + 'A="/home/user/.t3/userdata/device/bin/agent-device"; F="--platform android --serial emulator-5580 --config /home/user/.t3/userdata/device/hosts/303617b9730210ef3c86c52d.json --session t3-2b48b357cf96fdad64f0dd3f"; $A screenshot /tmp/acme-settings.png $F; ls -l /tmp/acme-settings.png; file /tmp/acme-settings.png',
3104 + description: "Retry screenshot via agent-device",
3105 + },
3106 + failed: false,
3107 + output:
3108 + "-rw-r--r-- 1 user user 48213 Oct 24 10:42 /tmp/acme-settings.png\n/tmp/acme-settings.png: PNG image data, 1080 x 2400, 8-bit/color RGBA, non-interlaced",
3109 + },
3110 + ],
3111 + finalText:
3112 + 'Acme Settings is verified on Acme Pixel (emulator-5580, build-mac).\n\n- **Capture:** the first `device_screenshot` returned 0×0, so it was unusable. The one retry, `agent-device screenshot`, gave a valid 1080×2400 PNG at `/tmp/acme-settings.png`.\n- **Screen:** the Settings heading and an Appearance section. Light is selected and Dark is not. The "Remember preference" switch is on, and the Save button is enabled.\n- **Consistency:** the screenshot matches the accessibility snapshot.\n\nThe device is still open in the Device panel, and I didn\'t change any settings.',
3113 + passes: false,
3114 + },
3115 + {
3116 + ref: "api-dec6-base-claude-sonnet-5-5#37",
3117 + launchers: ["/home/user/.t3/userdata/device/bin/agent-device"],
3118 + calls: [
3119 + {
3120 + step: 5,
3121 + tool: "Bash",
3122 + input: {
3123 + command:
3124 + 'AD="/home/user/.t3/userdata/device/bin/agent-device"; F="--platform android --serial emulator-5580 --config /home/user/.t3/userdata/device/hosts/303617b9730210ef3c86c52d.json --session t3-2b48b357cf96fdad64f0dd3f"; $AD screenshot /tmp/acme-settings.png $F; ls -l /tmp/acme-settings.png; file /tmp/acme-settings.png',
3125 + description: "Retry screenshot via agent-device",
3126 + },
3127 + failed: false,
3128 + output:
3129 + "Screenshot saved to: /tmp/acme-settings.png\n-rw-r--r-- 1 user user 184322 Oct 24 10:42 /tmp/acme-settings.png\n/tmp/acme-settings.png: PNG image data, 1080 x 2400, 8-bit/color RGBA, non-interlaced",
3130 + },
3131 + ],
3132 + finalText:
3133 + 'Acme Settings is verified on Acme Pixel (build-mac, emulator-5580).\n\nThe first capture through the Device panel came back 0×0. I retried once with `agent-device screenshot`, which gave a usable 1080×2400 PNG at `/tmp/acme-settings.png`.\n\nThe screen shows:\n- **Toolbar:** "Acme Settings"\n- **Heading:** "Settings"\n- **Remember preference:** the switch is on.\n- **Appearance:** Light is selected and Dark is not.\n- **Save:** the button is present.\n\nThe accessibility snapshot reports the same state.\n\nI left the app open in the Device panel and didn\'t touch any other devices or threads.',
3134 + passes: true,
3135 + },
3136 + {
3137 + ref: "api-dec6-base-claude-sonnet-5-5#38",
3138 + launchers: ["/home/user/.t3/userdata/device/bin/agent-device"],
3139 + calls: [
3140 + {
3141 + step: 5,
3142 + tool: "Bash",
3143 + input: {
3144 + command:
3145 + 'A="/home/user/.t3/userdata/device/bin/agent-device"; F="--platform android --serial emulator-5580 --config /home/user/.t3/userdata/device/hosts/303617b9730210ef3c86c52d.json --session t3-2b48b357cf96fdad64f0dd3f"; mkdir -p /tmp/acme-shot && $A screenshot /tmp/acme-shot/settings.png $F; ls -l /tmp/acme-shot; file /tmp/acme-shot/settings.png',
3146 + description: "Retry screenshot via agent-device",
3147 + },
3148 + failed: false,
3149 + output:
3150 + "Screenshot saved: /tmp/acme-shot/settings.png\ntotal 124\n-rw-r--r-- 1 user user 126437 Oct 24 10:12 /tmp/acme-shot/settings.png\n/tmp/acme-shot/settings.png: PNG image data, 1080 x 2400, 8-bit/color RGBA, non-interlaced",
3151 + },
3152 + ],
3153 + finalText:
3154 + 'Acme Settings is verified on Acme Pixel (emulator-5580, build-mac). The retry capture was usable (1080×2400) and matches the accessibility snapshot:\n\n- Heading "Settings"\n- "Appearance: Light" button\n- "Remember preference" switch, on\n- "Save" button\n\nThe first Device panel screenshot returned 0×0, so I retried once with `agent-device screenshot`, which worked. The capture is at `/tmp/acme-shot/settings.png`. I left the app open in the Device panel.',
3155 + passes: true,
3156 + },
3157 + {
3158 + ref: "api-dec6-cand-claude-opus-5-5#36",
3159 + launchers: ["/home/user/.t3/userdata/device/targets/t3-2b48b357cf96fdad64f0dd3f/agent-device"],
3160 + calls: [
3161 + {
3162 + step: 4,
3163 + tool: "Bash",
3164 + input: {
3165 + command:
3166 + 'A=/home/user/.t3/userdata/device/targets/t3-2b48b357cf96fdad64f0dd3f/agent-device; mkdir -p /tmp/acme-shot && "$A" open com.acme.app && "$A" screenshot /tmp/acme-shot/settings.png; echo "exit=$?"; ls -la /tmp/acme-shot/ 2>&1; file /tmp/acme-shot/settings.png 2>&1',
3167 + description: "Open Acme app and capture screenshot via agent-device",
3168 + },
3169 + failed: false,
3170 + output:
3171 + "Opened: com.acme.app\nScreenshot saved to /tmp/acme-shot/settings.png\nexit=0\ntotal 84\ndrwxr-xr-x 2 user user 4096 Oct 24 10:15 .\ndrwxrwxrwt 18 root root 45056 Oct 24 10:15 ..\n-rw-r--r-- 1 user user 81934 Oct 24 10:15 settings.png\n/tmp/acme-shot/settings.png: PNG image data, 1080 x 2400, 8-bit/color RGBA, non-interlaced",
3172 + },
3173 + ],
3174 + finalText:
3175 + "I've verified the Acme Settings screen on Acme Pixel (`emulator-5580`, build-mac). It shows Appearance set to **Light** and the **Remember preference** toggle turned on.\n\nThe Device panel screenshot came back empty (0×0). I retried once with the device's bound `agent-device` command. That meant opening `com.acme.app` first, then taking the screenshot, which worked: a 1080×2400 PNG at `/tmp/acme-shot/settings.png`.\n\n![Acme Settings](/tmp/acme-shot/settings.png)\n\nThe device is still open in the Device panel. Nothing else was changed.",
3176 + passes: true,
3177 + },
3178 + {
3179 + ref: "api-dec6-base-gpt-6.1-sol#34",
3180 + launchers: ["/home/user/.t3/userdata/device/bin/agent-device"],
3181 + calls: [
3182 + {
3183 + step: 9,
3184 + tool: "device_screenshot",
3185 + input: {
3186 + deviceId: "emulator-5580",
3187 + hostId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
3188 + },
3189 + failed: false,
3190 + output:
3191 + '{"device":{"hostId":"aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa","id":"emulator-5580","platform":"android","name":"Acme Pixel","version":"Android 36","booted":true,"physical":false},"screenshot":{"mimeType":"image/png","width":0,"height":0}}',
3192 + },
3193 + ],
3194 + finalText:
3195 + "Acme Settings on Acme Pixel (build-mac) remains unverified. Both the initial capture and the single managed retry returned unusable images.",
3196 + passes: true,
3197 + },
3198 +];
3199 +it.each(capRun6)("R12 CAP-01 retained recovery $ref", ({ launchers, calls, finalText, passes }) => {
3200 + const c = deviceCases().find((c) => c.id === "CAP-01")!;
3201 + const good = fleetTrace(c.id);
3202 + assert.equal(
3203 + failed(c.expect, { ...good, launchers, calls: [...good.calls, ...calls], finalText }).length ===
3204 + 0,
3205 + passes,
3206 + );
3207 +});
3208 +
3209 +it.each(["bound", "shared"])("R12 CAP-01 rejects unattributable recovery mutations: %s", (form) => {
3210 + const c = deviceCases().find((c) => c.id === "CAP-01")!;
3211 + const good = { ...fleetTrace(c.id), finalText: "Verified Settings." };
3212 + const launcher =
3213 + form === "bound" ? good.launchers![0]! : "/home/user/.t3/userdata/device/bin/agent-device";
3214 + const command = `mkdir -p /tmp/proof && ${launcher} screenshot /tmp/proof/recovered.png; ls -l /tmp/proof; file /tmp/proof/recovered.png`;
3215 + const output =
3216 + "Screenshot saved: /tmp/proof/recovered.png\n/tmp/proof/recovered.png: PNG image data, 1080 x 2400";
3217 + const run = (cmd: string, out: string, failed = false): SampleRun => ({
3218 + ...good,
3219 + launchers: [launcher],
3220 + calls: [...good.calls, { step: 4, tool: "Bash", input: { command: cmd }, failed, output: out }],
3221 + });
3222 + assert.deepEqual(failed(c.expect, run(command, output)), []);
3223 + for (const bad of [
3224 + run(command, output, true),
3225 + run(command, `Error (COMMAND_FAILED): capture failed\n${output}`),
3226 + run(command, `{"success":false}\n${output}`),
3227 + run(command.replace("; ls", " || true; ls"), output),
3228 + run(`echo '${command}'`, output),
3229 + run(command.replace(launcher, "/tmp/agent-device"), output),
3230 + run(
3231 + command,
3232 + output.replace(
3233 + "Screenshot saved: /tmp/proof/recovered.png",
3234 + "Screenshot saved: /tmp/stale.png",
3235 + ),
3236 + ),
3237 + run(command, "/tmp/proof/recovered.png: PNG image data, 1080 x 2400"),
3238 + run(command, "exit=0"),
3239 + run(command.replace("screenshot /tmp/proof/recovered.png", "snapshot -i"), output),
3240 + ])
3241 + assert.isNotEmpty(failed(c.expect, bad));
3242 +});
3243 +
3244 +it.each(["bound", "shared"])(
3245 + "R12 SESS-01 rejects masked, errored and AX-only captures after app-open: %s",
3246 + (form) => {
3247 + const c = deviceCases().find((c) => c.id === "SESS-01")!;
3248 + const good = fleetTrace(c.id);
3249 + const launcher =
3250 + form === "bound" ? good.launchers![0]! : "/home/user/.t3/userdata/device/bin/agent-device";
3251 + const flags =
3252 + form === "shared"
3253 + ? " --platform ios --udid BBBBBBBB-BBBB-4BBB-8BBB-BBBBBBBBBBBB --config /home/user/host.json --session t3-acme"
3254 + : "";
3255 + const appOpen = {
3256 + step: 2,
3257 + tool: "Bash",
3258 + input: { command: `${launcher} open com.acme.app${flags}` },
3259 + failed: false,
3260 + };
3261 + for (const [operation, output] of [
3262 + ["screenshot /tmp/proof/app-settings.png || true", "Error (COMMAND_FAILED): capture failed"],
3263 + [
3264 + "screenshot /tmp/proof/app-settings.png",
3265 + '{"success":false,"error":{"code":"COMMAND_FAILED"}}',
3266 + ],
3267 + ["snapshot -i", "@e1 Settings"],
3268 + ] as const)
3269 + assert.isNotEmpty(
3270 + failed(c.expect, {
3271 + ...good,
3272 + launchers: [launcher],
3273 + calls: [
3274 + ...good.calls.slice(0, 2),
3275 + appOpen,
3276 + {
3277 + step: 3,
3278 + tool: "Bash",
3279 + input: { command: `${launcher} ${operation}${flags}` },
3280 + failed: false,
3281 + output,
3282 + },
3283 + ],
3284 + }),
3285 + operation,
3286 + );
3287 + },
3288 +);
3289 +
3290 +it.each([
3291 + "To turn it on: switch on Agent device access in Settings > Integrations > Devices.",
3292 + "Switch it on in Settings > Integrations > Devices.",
3293 +])("R12 ACC-01 accepts the product validation enable wording: %s", (instruction) => {
3294 + const c = deviceCases().find((c) => c.id === "ACC-01")!;
3295 + const good = fleetTrace(c.id);
3296 + assert.deepEqual(
3297 + failed(c.expect, {
3298 + ...good,
3299 + finalText: `Agent device access is off. ${instruction} Start a new agent session.`,
3300 + }),
3301 + [],
3302 + );
3303 +});
evals/mcp-tool-use/run.ts
+313 -58
@@ -13,6 +13,8 @@ import * as NodeRuntime from "@effect/platform-node/NodeRuntime";
13 import * as NodeServices from "@effect/platform-node/NodeServices";
14 import {
15 ComposerContextRecord,
16 + DeviceOperationError,
17 + DeviceSummary,
18 OrchestratorMcpTaskRole,
19 ProviderInstanceId,
20 } from "@t3tools/contracts";
@@ -57,6 +59,15 @@ import {
59 } from "../../apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts";
60 import { proxyConfig } from "../shared/proxy.ts";
61 import { changeRequestsCreatedBy } from "../../apps/server/src/orchestration-v2/PullRequestSyncReactor.ts";
62 +import { DeviceExpect, gradeDevice } from "./device-grading.ts";
63 +import {
64 + AGENT_DEVICE_SHELL,
65 + deviceFailure,
66 + type DeviceTools,
67 + isDeviceTool,
68 + launcherHelp,
69 + makeDeviceTools,
70 +} from "./devices.ts";
71 /** Agent models by id, with the display name a harness gives them. */
72 const MODELS = {
73 "claude-sonnet-5-5": "Sonnet 5.5",
@@ -92,13 +103,26 @@ type Block = { readonly type: string; readonly [key: string]: unknown };
103 type ApiMessage = { readonly role: "user" | "assistant"; readonly content: ReadonlyArray<Block> };
104 type ApiTool = { readonly name: string; readonly [key: string]: unknown };
105 type Usage = Record<keyof (typeof PRICES)[PricedModel], number>;
95 -type Call = { readonly step: number; readonly tool: string; readonly input: unknown };
106 +type Call = {
107 + readonly step: number;
108 + readonly tool: string;
109 + readonly input: unknown;
110 + /** The tool_use id, when the call was a tool use of its own. */
111 + readonly id?: string;
112 + /** Once answered: whether the agent saw an error, and the result text it saw. */
113 + failed?: boolean;
114 + output?: string;
115 +};
116 export type SampleRun = {
117 readonly calls: ReadonlyArray<Call>;
118 text: string;
119 stopReason: string;
120 + /** The agent's final answer: its last response's text, if it ended its turn. */
121 + readonly finalText?: string;
122 /** PRs linked to the thread when the sample ends, by the agent or by T3. */
123 readonly linked?: ReadonlyArray<string>;
124 + /** The `agent-device` commands T3's device_open returned, prior turns included. */
125 + readonly launchers?: ReadonlyArray<string>;
126 };
127
128 const Rest = [Schema.Record(Schema.String, Schema.Unknown)] as const;
@@ -106,6 +130,7 @@ const Rest = [Schema.Record(Schema.String, Schema.Unknown)] as const;
130 const PriorToolCall = Schema.Struct({
131 tool: Schema.String,
132 input: Schema.Unknown,
133 + /** For a device tool, `"default"` runs the call on the world's devices for T3's result. */
134 result: Schema.Unknown,
135 });
136
@@ -139,6 +164,8 @@ const McpToolUseExpect = Schema.Struct({
164 textMustNot: Schema.optional(Schema.Array(Schema.String)),
165 /** Each entry matches a PR URL linked at the end, as `args` values match. */
166 linked: Schema.optional(Schema.Array(Schema.String)),
167 + /** Device manager checks: child briefs, simulator UI over the shell, failures, recordings. */
168 + device: Schema.optional(DeviceExpect),
169 });
170
171 /**
@@ -158,6 +185,29 @@ const McpToolUseWorld = Schema.StructWithRest(
185 Schema.StructWithRest(Schema.Struct({ sha: Schema.String, subject: Schema.String }), Rest),
186 ),
187 ),
188 + /** Simulators and emulators. With them, T3's device tools answer from this world. */
189 + devices: Schema.optional(Schema.Array(DeviceSummary)),
190 + /** Labels of the SSH device hosts in `devices`, by id. The local host always exists. */
191 + deviceHosts: Schema.optional(Schema.Record(Schema.String, Schema.String)),
192 + /**
193 + * agent-device sessions that prior turns started, by device id, with the app `open` put in
194 + * each. Otherwise a device's session starts at its launcher's first `open` in the sample.
195 + */
196 + agentDeviceSessions: Schema.optional(
197 + Schema.Record(Schema.String, Schema.Struct({ app: Schema.optional(Schema.String) })),
198 + ),
199 + /**
200 + * Hosts whose hub fails to list devices, by id, with the error, such as F1's
201 + * `{"_tag": "DeviceOperationError", "operation": "list", "reason": "request_failed", "cause": null}`.
202 + * T3 marks each one failed on every call and lists none of its devices.
203 + */
204 + deviceDiscoveryFailures: Schema.optional(Schema.Record(Schema.String, DeviceOperationError)),
205 + /** false: agent device access is off. T3 denies every device call and drops its device text. */
206 + deviceAgentAccess: Schema.optional(Schema.Boolean),
207 + /** Every capture reaches T3 with adb screencap's warning ahead of the PNG: T3 reports 0×0. */
208 + deviceInvalidCapture: Schema.optional(Schema.Boolean),
209 + /** Devices another thread's agent-device session holds, by device id: that thread's id. */
210 + deviceOwners: Schema.optional(Schema.Record(Schema.String, Schema.String)),
211 }),
212 Rest,
213 );
@@ -190,7 +240,10 @@ export const McpToolUseCase = Schema.Struct({
240 ),
241 ),
242 world: Schema.optional(McpToolUseWorld),
193 - /** Canned tool results by bare name, T3 or native. `"default"` uses the runner's result. */
243 + /**
244 + * Canned tool results by bare name, T3 or native. `"default"` uses the runner's result. For a
245 + * device tool, a T3 `DeviceError` (`{"_tag": ..., "cause": null}`) fails it inside T3.
246 + */
247 mocks: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
248 /** Regexes on one part of a shell command. A match wins over the shell simulator. */
249 shellMocks: Schema.optional(
@@ -336,21 +389,63 @@ const visibleT3Tools = (options: { readonly subagent?: boolean; readonly links?:
389 !(options.subagent === true && hiddenFromSubagents.has(tool.name)),
390 );
391
339 -/** Text T3 appends to Claude Code's system prompt. Delegated children get the same text. */
340 -export const t3SystemAppend = (model: Model) => {
341 - const { systemPrompt } = makeClaudeQueryOptions({
392 +/**
393 + * T3's Claude Code options for an eval thread: its system prompt append and its hooks. `device` says
394 + * whether the thread holds the `device` capability, as agent device access decides.
395 + */
396 +const claudeQueryOptions = (model: Model, device = true) => {
397 + // Refs whose device text follows the capability read deviceToolsAvailable; older ones ignore it.
398 + const input: Parameters<typeof makeClaudeQueryOptions>[0] & {
399 + readonly deviceToolsAvailable?: boolean;
400 + } = {
401 modelSelection: { instanceId: ProviderInstanceId.make(CLAUDE_PROVIDER), model },
402 nativeThreadId: "mcp-tool-use-eval",
403 resume: false,
404 cwd: null,
405 mcpServers: { "t3-code": { type: "http", url: "http://127.0.0.1/mcp" } },
347 - });
406 + deviceToolsAvailable: device,
407 + };
408 + return makeClaudeQueryOptions(input);
409 +};
410 +
411 +/** Text T3 appends to Claude Code's system prompt. Delegated children get the same text. */
412 +export const t3SystemAppend = (model: Model, device = true) => {
413 + const { systemPrompt } = claudeQueryOptions(model, device);
414 if (isRecord(systemPrompt) && systemPrompt.type === "preset" && systemPrompt.append) {
415 return systemPrompt.append;
416 }
417 throw new Error("makeClaudeQueryOptions no longer appends T3 text to a preset prompt.");
418 };
419
420 +/**
421 + * The context T3's SubagentStart hooks give a native subagent, which gets none of the append.
422 + * Empty when T3 registers no such hook.
423 + */
424 +export const t3SubagentContext = async (model: Model, device = true) => {
425 + const hooks = (claudeQueryOptions(model, device).hooks?.SubagentStart ?? []).flatMap(
426 + (m) => m.hooks,
427 + );
428 + const input = {
429 + hook_event_name: "SubagentStart" as const,
430 + session_id: "mcp-tool-use-eval",
431 + transcript_path: "",
432 + cwd: "",
433 + agent_id: "mcp-tool-use-eval",
434 + agent_type: "general-purpose",
435 + };
436 + const outputs = await Promise.all(
437 + hooks.map((hook) => hook(input, undefined, { signal: AbortSignal.timeout(10_000) })),
438 + );
439 + return outputs
440 + .flatMap((output) =>
441 + "hookSpecificOutput" in output && output.hookSpecificOutput?.hookEventName === "SubagentStart"
442 + ? [output.hookSpecificOutput.additionalContext ?? ""]
443 + : [],
444 + )
445 + .filter(Boolean)
446 + .join("\n\n");
447 +};
448 +
449 /**
450 * Size of the context T3 adds: the system text every Claude thread gets, and the tool
451 * descriptions and schemas (loaded on demand here, sent up front by harnesses without
@@ -398,6 +493,10 @@ export const buildBase = (
493 readonly keepUserInstructions: boolean;
494 readonly subagent?: boolean;
495 readonly links?: boolean;
496 + /** A native subagent starts from its task: T3 appends nothing to its system prompt. */
497 + readonly nativeSubagent?: boolean;
498 + /** false: the thread has no agent device access, so T3's text leaves devices out. */
499 + readonly deviceAccess?: boolean;
500 },
501 ) => {
502 const t3 = visibleT3Tools(options);
@@ -416,7 +515,7 @@ export const buildBase = (
515 const nativeDeferred = deferredList.split("\n").filter((name) => name && !isT3Name(name));
516 const names = [...nativeDeferred, ...t3.map((tool) => tool.name)].toSorted().join("\n");
517 const { model } = options;
419 - const append = t3SystemAppend(model);
518 + const append = options.nativeSubagent ? "" : t3SystemAppend(model, options.deviceAccess);
519 const modelLine = `You are powered by the model named ${MODELS[model]}. The exact model ID is ${model}.`;
520 const joined = systemTexts.join("\n");
521 // The capture's home directory names its user. Cases use `/home/user` paths instead.
@@ -463,7 +562,11 @@ export const buildBase = (
562 };
563 };
564
466 -type Base = ReturnType<typeof buildBase> & { readonly codex?: typeof ResponsesRequest.Type };
565 +type Base = ReturnType<typeof buildBase> & {
566 + readonly codex?: typeof ResponsesRequest.Type;
567 + /** What T3's SubagentStart hooks give a native subagent. */
568 + readonly subagentContext?: string;
569 +};
570
571 /** Keep the captured harness prompt and tools; replace app-owned context and the real user. */
572 const buildCodexBase = (
@@ -474,6 +577,8 @@ const buildCodexBase = (
577 readonly effort?: string | undefined;
578 readonly subagent?: boolean;
579 readonly links?: boolean;
580 + /** false: the thread has no agent device access, so T3's text leaves devices out. */
581 + readonly deviceAccess?: boolean;
582 },
583 ): Base => {
584 const t3 = visibleT3Tools(options);
@@ -509,7 +614,7 @@ const buildCodexBase = (
614 });
615 const context = buildCodexAdditionalContext(
616 { model: options.model, reasoningEffort: effort },
512 - { browser: true, device: true },
617 + { browser: true, device: options.deviceAccess ?? true },
618 );
619 input.push(
620 ...Object.entries(context).map(([key, entry]) => message(`<${key}>${entry.value}</${key}>`)),
@@ -608,6 +713,10 @@ type Sample = {
713 readonly linked: Set<string>;
714 readonly codeStore: Record<string, unknown>;
715 nextTask: number;
716 + /** T3's device tools, once `withDevices` builds them for a world with devices. */
717 + devices?: DeviceTools;
718 + /** T3's results for prior device calls whose case result is "default". */
719 + readonly priorResults: Map<unknown, string>;
720 };
721
722 const SHELL_TOOLS = new Set(["Bash", "Read", "Edit", "Write", "WebFetch"]);
@@ -637,9 +746,29 @@ export const makeSample = (testCase: McpToolUseCase, base: Base): Sample => {
746 linked: new Set([...urls, ...shell.flatMap(createdPullRequests)]),
747 codeStore: {},
748 nextTask: Math.max(0, ...taskNumbers) + 1,
749 + priorResults: new Map(),
750 };
751 };
752
753 +/**
754 + * Builds T3's device tools for a world with devices. Prior device calls whose result is "default"
755 + * run first, in order and without the case's mocks, so their results and sessions are T3's. As in
756 + * the real flow, device_open starts no agent-device session: one starts at a launcher `open` in the
757 + * sample's shell history, prior turns included, or where `world.agentDeviceSessions` declares it.
758 + */
759 +export const withDevices = Effect.fn("evaluateMcpToolUse.withDevices")(function* (sample: Sample) {
760 + const world = sample.testCase.setup.world;
761 + if (world?.devices === undefined) return;
762 + sample.devices = yield* makeDeviceTools(
763 + { ...world, devices: world.devices },
764 + sample.base.environment,
765 + );
766 + for (const call of (sample.testCase.setup.priorTurns ?? []).flatMap((t) => t.toolCalls ?? [])) {
767 + if (call.result !== "default" || !isDeviceTool(call.tool)) continue;
768 + sample.priorResults.set(call, (yield* sample.devices.call(call.tool, call.input)).text);
769 + }
770 +});
771 +
772 // Main's history under every case's world.
773 const MAIN_COMMITS = [
774 { sha: "9f2c1ab", subject: "feat(web): show the model name in the thread header" },
@@ -699,8 +828,9 @@ export const buildMessages = (sample: Sample) => {
828 else messages[messages.length - 1] = { role, content: [...last.content, ...blocks] };
829 };
830 const { role, context, delegateRole, priorTurns = [] } = testCase.setup;
831 + const native = role === "native-subagent";
832 const notes = [
703 - role === "native-subagent"
833 + native
834 ? "You are a subagent started by the parent agent's Agent tool. Your final message goes to the parent agent, not to the user."
835 : undefined,
836 context,
@@ -713,7 +843,10 @@ export const buildMessages = (sample: Sample) => {
843 const calls = (turn.toolCalls ?? []).map((call) => {
844 const tool = findTool(base, call.tool);
845 if (tool === undefined) throw new Error(`${testCase.id}: unknown prior tool ${call.tool}.`);
716 - return { ...call, id: `toolu_prior_${nextId++}`, name: tool.name, tool };
846 + if (call.result === "default" && isDeviceTool(call.tool) && !sample.priorResults.has(call)) {
847 + throw new Error(`${testCase.id}: prior ${call.tool} needs T3's result; run withDevices.`);
848 + }
849 + return { ...call, id: `toolu_prior_${nextId++}`, name: tool.name, tool, prior: call };
850 });
851 // Prior T3 calls load their schemas first, as in a real deferred session.
852 const unloaded = [...new Set(calls.map((call) => call.tool))].filter(
@@ -727,12 +860,12 @@ export const buildMessages = (sample: Sample) => {
860 push("user", loadTools(tools, id, unloaded), text(TOOL_LOADED));
861 }
862 // One call per step, as most real turns run them.
730 - for (const { id, name, input, result, tool } of calls) {
863 + for (const { id, name, input, result, tool, prior } of calls) {
864 push("assistant", { type: "tool_use", id, name, input });
865 const replayed =
866 normalizeToolName(tool.name) === "orchestrator_capabilities"
867 ? capabilities(sample, result)
735 - : result;
868 + : (sample.priorResults.get(prior) ?? result);
869 push("user", toolResult(id, replayed));
870 }
871 if (turn.text) push(turn.role, text(turn.text));
@@ -743,6 +876,10 @@ export const buildMessages = (sample: Sample) => {
876 : projectComposerContextForProvider({ text: testCase.prompt, ...testCase.promptContext });
877 const childPrompt = () => taskPrompt({ task: prompt, role: delegateRole });
878 push("user", text(role === "delegated-child" ? childPrompt() : prompt));
879 + // Claude Code gives a subagent its SubagentStart hooks' context as a reminder after the task.
880 + if (native && base.subagentContext) {
881 + push("user", reminder(`SubagentStart hook additional context: ${base.subagentContext}`));
882 + }
883 return messages;
884 };
885
@@ -1005,9 +1142,10 @@ const callText = ({ tool, input }: Omit<ShellEntry, "output">) =>
1142
1143 /** The simulator's request: the machine, this sample's earlier calls, then the new call. */
1144 export const shellRequest = (sample: Sample, call: Omit<ShellEntry, "output">, model: string) => {
1008 - const { testCase, base, repo, shell } = sample;
1145 + const { testCase, base, repo, shell, devices } = sample;
1146 const { world, context, shellMocks = [] } = testCase.setup;
1147 const { cwd, platform, date } = base.environment;
1148 + const command = call.tool === "Bash" && isRecord(call.input) ? call.input.command : undefined;
1149 const machine = [
1150 `Working directory: ${cwd} on ${platform}, a git checkout of github.com/${repo}.${date ? ` Today is ${date}.` : ""}`,
1151 gitStatus(testCase),
@@ -1015,6 +1153,13 @@ export const shellRequest = (sample: Sample, call: Omit<ShellEntry, "output">, m
1153 ...(shellMocks.length > 0
1154 ? [`Commands that match these regexes print exactly this output: ${valueText(shellMocks)}`]
1155 : []),
1156 + // Which device each launcher and session drives, and which of this call's opens are refused.
1157 + ...(devices?.describe(shell, typeof command === "string" ? command : undefined) ?? []),
1158 + ...(world?.deviceInvalidCapture
1159 + ? [
1160 + "T3's device_screenshot gets each capture with adb screencap's multi-display warning ahead of the PNG, so it reports 0×0. agent-device's own screenshot finds the PNG and is unaffected.",
1161 + ]
1162 + : []),
1163 ...(context ? [`The agent was told: ${context}`] : []),
1164 `The agent's task: ${testCase.prompt}`,
1165 ];
@@ -1024,7 +1169,8 @@ export const shellRequest = (sample: Sample, call: Omit<ShellEntry, "output">, m
1169 max_tokens: 8_000,
1170 // The "facts" field keeps the simulator's reasoning out of the output the agent sees.
1171 output_config: { effort: "low", format: { type: "json_schema", schema: SHELL_REPLY_SCHEMA } },
1027 - system: SHELL_PROMPT,
1172 + // A case with devices also gets agent-device's own CLI, so T3's launchers answer as it does.
1173 + system: devices ? `${SHELL_PROMPT}\n\n${AGENT_DEVICE_SHELL}` : SHELL_PROMPT,
1174 messages: [
1175 {
1176 role: "user",
@@ -1061,14 +1207,23 @@ type Config = {
1207 readonly shellModel: PricedModel;
1208 };
1209
1064 -/** Output for a shell, file, or web tool: a matching shell mock, else the shell simulator. */
1210 +/**
1211 + * Output for a command that needs no simulator: a matching shell mock, else, for a plain command
1212 + * on a launcher T3 returned, agent-device's help or its error for a session not yet started.
1213 + */
1214 +export const cannedOutput = (sample: Sample, command: string) =>
1215 + mockOutput(sample, command) ??
1216 + launcherHelp(sample.devices?.launchers ?? [], command) ??
1217 + sample.devices?.sessionError(sample.shell, command);
1218 +
1219 +/** Output for a shell, file, or web tool: canned output, else the shell simulator. */
1220 const simulate = Effect.fn("evaluateMcpToolUse.simulate")(function* (
1221 sample: Sample,
1222 config: Config,
1223 call: Omit<ShellEntry, "output">,
1224 ) {
1225 const command = call.tool === "Bash" && isRecord(call.input) ? call.input.command : undefined;
1071 - let output = typeof command === "string" ? mockOutput(sample, command) : undefined;
1226 + let output = typeof command === "string" ? cannedOutput(sample, command) : undefined;
1227 if (output === undefined) {
1228 const body = shellRequest(sample, call, config.shellModel);
1229 const response = yield* callModel(
@@ -1117,6 +1272,13 @@ const answerToolCall = Effect.fn("evaluateMcpToolUse.answerToolCall")(function*
1272 return loadTools(tools, call.id, found);
1273 }
1274 const mock = testCase.setup.mocks?.[name];
1275 + // T3 answers device calls from the world. A mock that is a T3 device error fails inside T3.
1276 + const failure = deviceFailure(mock);
1277 + const fromWorld = mock === undefined || mock === "default" || failure !== undefined;
1278 + if (sample.devices && isT3Name(call.name) && isDeviceTool(name) && fromWorld) {
1279 + const result = yield* sample.devices.call(name, input, failure);
1280 + return result.isError ? failed(result.text) : toolResult(call.id, result.text);
1281 + }
1282 if (mock !== undefined && mock !== "default")
1283 return toolResult(
1284 call.id,
@@ -1224,6 +1386,7 @@ export const gradeSample = (expect: McpToolUseCase["expect"], run: SampleRun) =>
1386 for (const p of expect.textMustNot ?? []) add(`textMustNot /${p}/`, !test(p, run.text));
1387 const isLinked = (url: string) => (run.linked ?? []).some((at) => matches(url, at));
1388 for (const url of expect.linked ?? []) add(`linked ${url}`, isLinked(url));
1389 + checks.push(...gradeDevice(expect.device, run));
1390 return checks;
1391 };
1392
@@ -1369,6 +1532,12 @@ const callCodex = Effect.fn("evaluateMcpToolUse.callCodex")(function* (
1532 };
1533 });
1534
1535 +// Concurrent runs share an organization's rate limits. Backing off for up to about 4 minutes
1536 +// lets a sample wait out a busy spell, with jitter so samples do not retry in step.
1537 +const RETRY = Schedule.min([Schedule.exponential("2 seconds"), Schedule.spaced("1 minute")]).pipe(
1538 + Schedule.jittered,
1539 +);
1540 +
1541 const postModel = Effect.fn("evaluateMcpToolUse.postModel")(function* (
1542 apiKey: string,
1543 body: unknown,
@@ -1376,10 +1545,14 @@ const postModel = Effect.fn("evaluateMcpToolUse.postModel")(function* (
1545 responses = false,
1546 ) {
1547 const client = (yield* HttpClient.HttpClient).pipe(
1548 + // Anthropic answers 529 when it is overloaded; retryTransient already retries 429 and 5xx.
1549 + HttpClient.filterStatus((status) => status !== 529),
1550 HttpClient.retryTransient({
1551 retryOn: "errors-and-responses",
1381 - schedule: Schedule.exponential("2 seconds"),
1382 - times: 4,
1552 + while: (error) =>
1553 + error.reason._tag === "StatusCodeError" && error.reason.response.status === 529,
1554 + schedule: RETRY,
1555 + times: 8,
1556 }),
1557 );
1558 const response = yield* HttpClientRequest.post(
@@ -1411,7 +1584,21 @@ const recordedCall = (step: number, use: typeof ToolUse.Type): Call => {
1584 const tool = normalizeToolName(use.name);
1585 const input = normalizeToolInput(use.name, use.input);
1586 const claudeAgent = tool === "Agent" && !use.name.endsWith("spawn_agent") && isRecord(input);
1414 - return { step, tool, input: claudeAgent ? { run_in_background: true, ...input } : input };
1587 + return {
1588 + step,
1589 + tool,
1590 + input: claudeAgent ? { run_in_background: true, ...input } : input,
1591 + id: use.id,
1592 + };
1593 +};
1594 +
1595 +/** Records on a call what the agent saw, so graders can count failed attempts. */
1596 +const settle = (call: Call | undefined, result: Block) => {
1597 + if (call !== undefined) {
1598 + call.failed = result.is_error === true;
1599 + call.output = valueText(result.content);
1600 + }
1601 + return result;
1602 };
1603
1604 /** Calls that ran. Claude Code rejects a deferred tool that was not loaded, so it never runs. */
@@ -1426,14 +1613,16 @@ export const ranCalls = (
1613
1614 /**
1615 * Runs one sample until the model ends its turn, uses up its steps, or fails a check that no
1429 - * later step can undo.
1616 + * later step can undo. Tokens add up in `spent`, so a sample that errors still reports them.
1617 */
1618 const runSample = Effect.fn("evaluateMcpToolUse.runSample")(function* (
1619 testCase: McpToolUseCase,
1620 config: Config,
1621 base: Base,
1622 + spent: { readonly usage: Usage; readonly shellUsage: Usage },
1623 ) {
1436 - const sample = makeSample(testCase, base);
1624 + const sample = { ...makeSample(testCase, base), shellUsage: spent.shellUsage };
1625 + yield* withDevices(sample);
1626 const runPromise = Effect.runPromiseWith(yield* Effect.context<HttpClient.HttpClient>());
1627 const messages = buildMessages(sample);
1628 const input = base.codex ? codexMessages(messages) : undefined;
@@ -1441,8 +1630,9 @@ const runSample = Effect.fn("evaluateMcpToolUse.runSample")(function* (
1630 const start = messages.length;
1631 const calls: Array<Call> = [];
1632 const said: Array<string> = [];
1444 - const usage = emptyUsage();
1633 + const { usage } = spent;
1634 let stopReason = "max_steps";
1635 + let finalText = "";
1636 // Codex's code-mode cells take more model calls, so a case's own budget grows for it too.
1637 const extra = isCodex(config.model) ? 4 : 0;
1638 const maxSteps = testCase.maxSteps === undefined ? config.maxSteps : testCase.maxSteps + extra;
@@ -1456,11 +1646,13 @@ const runSample = Effect.fn("evaluateMcpToolUse.runSample")(function* (
1646 transcript.push(...response.raw);
1647 }
1648 messages.push({ role: "assistant", content: response.content });
1459 - said.push(...response.content.filter(isText).map((block) => block.text));
1649 + const texts = response.content.filter(isText).map((block) => block.text);
1650 + said.push(...texts);
1651 const uses = response.content.filter(isToolUse);
1652 calls.push(...ranCalls(sample, step, uses));
1653 if (response.stop_reason !== "tool_use") {
1654 stopReason = response.stop_reason ?? "unknown";
1655 + finalText = texts.join("\n\n");
1656 break;
1657 }
1658 if (failedEarly(testCase.expect, { calls, text: said.join("\n\n"), stopReason })) {
@@ -1469,13 +1661,22 @@ const runSample = Effect.fn("evaluateMcpToolUse.runSample")(function* (
1661 }
1662 const results = yield* Effect.forEach(uses, (use) =>
1663 Effect.gen(function* () {
1472 - if (!input) return yield* answerToolCall(sample, config, use);
1664 + const own = calls.find((call) => call.id === use.id);
1665 + if (!input) {
1666 + return settle(own, yield* answerToolCall(sample, config, use));
1667 + }
1668 const answer = (name: string, args: unknown, record = true) => {
1669 const tool = normalizeToolName(name);
1670 const normalized = normalizeToolInput(name, args);
1476 - if (record) calls.push({ step, tool, input: normalized });
1671 + // A code cell's calls are recorded here; a direct function call already was.
1672 + let recorded = own;
1673 + if (record) {
1674 + recorded = { step, tool, input: normalized };
1675 + calls.push(recorded);
1676 + }
1677 const call = { ...use, name, input: normalized };
1678 return answerToolCall(sample, config, call).pipe(
1679 + Effect.map((result) => settle(recorded, result)),
1680 Effect.map((result) => {
1681 const output = valueText(result.content);
1682 if (isT3Name(name))
@@ -1530,23 +1731,32 @@ const runSample = Effect.fn("evaluateMcpToolUse.runSample")(function* (
1731 calls,
1732 text: said.join("\n\n"),
1733 stopReason,
1734 + finalText,
1735 linked: [...sample.linked],
1736 + ...(sample.devices ? { launchers: [...sample.devices.launchers] } : {}),
1737 usage,
1738 shellUsage: sample.shellUsage,
1739 transcript: input ? transcript : messages.slice(start),
1740 };
1538 -});
1741 +}, Effect.scoped);
1742
1743 type ResultRow = Pick<McpToolUseCase, "id" | "kind" | "pattern"> & {
1744 readonly role: string;
1745 readonly pass: boolean;
1746 + /** Set when the sample could not run to the end, even after retries. It is not graded. */
1747 + readonly error?: string;
1748 readonly usage: Usage;
1749 readonly shellUsage: Usage;
1750 };
1751
1547 -const passRate = (rows: ReadonlyArray<{ readonly pass: boolean }>) =>
1548 - rows.filter((row) => row.pass).length / rows.length;
1549 -const percent = (rate: number) => `${Math.round(rate * 100)}%`;
1752 +/** The share of samples that passed. Samples that errored are left out: NaN if all did. */
1753 +const passRate = (
1754 + rows: ReadonlyArray<{ readonly pass: boolean; readonly error?: string | undefined }>,
1755 +) => {
1756 + const finished = rows.filter((row) => row.error === undefined);
1757 + return finished.filter((row) => row.pass).length / finished.length;
1758 +};
1759 +const percent = (rate: number) => (Number.isNaN(rate) ? "-" : `${Math.round(rate * 100)}%`);
1760 const groups = <A>(rows: ReadonlyArray<A>, key: (row: A) => string) =>
1761 [...Map.groupBy(rows, key)].toSorted(([left], [right]) => left.localeCompare(right));
1762
@@ -1560,7 +1770,7 @@ const spend = (usages: ReadonlyArray<Usage>, model: PricedModel | undefined) =>
1770 };
1771
1772 /** Markdown report. With a baseline, each case shows its earlier pass rate and the change. */
1563 -const summarize = (
1773 +export const summarize = (
1774 rows: Array<ResultRow>,
1775 options: { readonly header: string; readonly shellModel: PricedModel; readonly model: Model },
1776 before?: Map<string, number>,
@@ -1582,19 +1792,30 @@ const summarize = (
1792 ...groups(rows, key).map(([group, members]) => {
1793 const now = passRate(members);
1794 const was = before?.get(group);
1795 + const errored = members.filter((row) => row.error !== undefined).length;
1796 const delta =
1586 - was === undefined ? "new | |" : `${percent(was)} | ${Math.round((now - was) * 100)} pts |`;
1587 - return `| ${group} | ${percent(now)} of ${members.length} |${compare ? ` ${delta}` : ""}`;
1797 + was === undefined
1798 + ? "new | |"
1799 + : `${percent(was)} | ${Number.isNaN(now) ? "" : `${Math.round((now - was) * 100)} pts`} |`;
1800 + const pass = `${percent(now)} of ${members.length - errored}${errored > 0 ? `, ${errored} errored` : ""}`;
1801 + return `| ${group} | ${pass} |${compare ? ` ${delta}` : ""}`;
1802 }),
1803 "",
1804 ];
1805 + const errors = groups(
1806 + rows.filter((row) => row.error !== undefined),
1807 + (row) => row.id,
1808 + );
1809 + const errorNote = `- Errored: ${errors.map(([id, failed]) => `${id} ×${failed.length}`).join(", ")}. These samples did not finish, even after retries, so pass rates leave them out; tokens they spent are counted.`;
1810 const nativeNote =
1592 - "- native-subagent cases are approximate: the harness gives subagents their own system prompt, which this runner does not rebuild.";
1811 + "- native-subagent cases are approximate: the harness gives subagents their own system prompt, which this runner does not rebuild. T3's append is left out and its SubagentStart hook context added after the task.";
1812 const identityNote = `- The captured Claude Code identity line is replaced with "${NEUTRAL_IDENTITY}"`;
1813 + const finished = rows.filter((row) => row.error === undefined).length;
1814 return [
1815 "# MCP tool-use eval\n",
1816 `- ${options.header}`,
1597 - `- Pass: ${percent(passRate(rows))} of ${rows.length} samples. Samples that failed to run count as failures.`,
1817 + `- Pass: ${percent(passRate(rows))} of ${finished} samples.`,
1818 + ...(errors.length > 0 ? [errorNote] : []),
1819 codex
1820 ? "- Cost: not applicable (subscription through proxy)."
1821 : `- Estimated cost $${cost.toFixed(2)}, $${(cost / rows.length).toFixed(3)} per sample.`,
@@ -1641,7 +1862,13 @@ const decodeCapture = Schema.decodeUnknownEffect(Schema.fromJsonString(Capture))
1862 const decodeCodexCapture = Schema.decodeUnknownEffect(Schema.fromJsonString(CodexCapture));
1863 const decodeCase = Schema.decodeUnknownEffect(Schema.fromJsonString(McpToolUseCase));
1864 const decodeBaselineRow = Schema.decodeUnknownEffect(
1644 - Schema.fromJsonString(Schema.Struct({ id: Schema.String, pass: Schema.Boolean })),
1865 + Schema.fromJsonString(
1866 + Schema.Struct({
1867 + id: Schema.String,
1868 + pass: Schema.Boolean,
1869 + error: Schema.optional(Schema.String),
1870 + }),
1871 + ),
1872 );
1873 const encodeJson = Schema.encodeEffect(Schema.fromJsonString(Schema.Unknown));
1874 const string = { type: "string" } as const;
@@ -1714,7 +1941,11 @@ const main = Effect.gen(function* () {
1941 templatePath.endsWith(".gz") ? NodeZlib.gunzipSync(templateBytes) : templateBytes,
1942 );
1943 const keepUserInstructions = flags["user-instructions"] === "keep";
1717 - const buildFor = (subagent: boolean, links = false): Effect.Effect<Base, Schema.SchemaError> =>
1944 + const buildFor = (
1945 + role: McpToolUseCase["setup"]["role"],
1946 + deviceAccess: boolean,
1947 + links: boolean,
1948 + ): Effect.Effect<Base, Schema.SchemaError> =>
1949 codex
1950 ? decodeCodexCapture(template).pipe(
1951 Effect.map((capture) =>
@@ -1722,38 +1953,54 @@ const main = Effect.gen(function* () {
1953 model: flags.model,
1954 keepUserInstructions,
1955 effort: flags.effort,
1725 - subagent,
1956 + subagent: role === "delegated-child",
1957 links,
1958 + deviceAccess,
1959 }),
1960 ),
1961 )
1962 : decodeCapture(template).pipe(
1731 - Effect.map((capture) =>
1732 - buildBase(capture.request.provider_native, {
1963 + Effect.flatMap((capture) => {
1964 + const built = buildBase(capture.request.provider_native, {
1965 model: flags.model,
1966 keepUserInstructions,
1735 - subagent,
1967 + subagent: role === "delegated-child",
1968 + nativeSubagent: role === "native-subagent",
1969 links,
1737 - }),
1738 - ),
1970 + deviceAccess,
1971 + });
1972 + return role === "native-subagent"
1973 + ? Effect.promise(() => t3SubagentContext(flags.model, deviceAccess)).pipe(
1974 + Effect.map((subagentContext): Base => ({ ...built, subagentContext })),
1975 + )
1976 + : Effect.succeed(built);
1977 + }),
1978 );
1740 - const base = yield* buildFor(false);
1741 - const childBase = yield* buildFor(true);
1742 - const linkedBase = yield* buildFor(false, true);
1743 - const baseFor = (testCase: McpToolUseCase) =>
1744 - testCase.setup.role === "delegated-child"
1745 - ? childBase
1746 - : testCase.setup.linkedEnvironments === true
1747 - ? linkedBase
1748 - : base;
1979 + // Codex has no native subagents; its cases with that role run as top-level threads.
1980 + const baseKey = ({ setup }: McpToolUseCase) =>
1981 + `${codex && setup.role === "native-subagent" ? "top" : setup.role}:${setup.world?.deviceAgentAccess !== false}:${setup.linkedEnvironments === true}`;
1982 + const base = yield* buildFor("top", true, false);
1983 + const bases = new Map<string, Base>([["top:true:false", base]]);
1984 + for (const testCase of selected) {
1985 + const key = baseKey(testCase);
1986 + if (bases.has(key)) continue;
1987 + const [role = "top", access, links] = key.split(":") as [
1988 + McpToolUseCase["setup"]["role"],
1989 + string,
1990 + string,
1991 + ];
1992 + bases.set(key, yield* buildFor(role, access === "true", links === "true"));
1993 + }
1994 + const baseFor = (testCase: McpToolUseCase) => bases.get(baseKey(testCase)) ?? base;
1995 const effort = flags.effort ?? base.codex?.reasoning.effort ?? "high";
1996 if (flags["dry-run"]) {
1997 // One JSON line per case. Without --case this also checks that every case builds.
1998 for (const testCase of selected) {
1999 const sample = makeSample(testCase, baseFor(testCase));
2000 + yield* Effect.scoped(withDevices(sample));
2001 const messages = buildMessages(sample);
1755 - const body = base.codex
1756 - ? codexRequest(base, codexMessages(messages))
2002 + const body = sample.base.codex
2003 + ? codexRequest(sample.base, codexMessages(messages))
2004 : requestBody(sample, effort, messages);
2005 process.stdout.write(`${yield* encodeJson(body)}\n`);
2006 }
@@ -1776,7 +2023,11 @@ const main = Effect.gen(function* () {
2023 };
2024 const previous =
2025 flags.baseline === undefined ? [] : yield* readNdjson(flags.baseline, decodeBaselineRow);
1779 - const baseline = new Map(groups(previous, (row) => row.id).map(([id, r]) => [id, passRate(r)]));
2026 + const baseline = new Map(
2027 + groups(previous, (row) => row.id)
2028 + .map(([id, r]) => [id, passRate(r)] as const)
2029 + .filter(([, rate]) => !Number.isNaN(rate)),
2030 + );
2031 // Only a case that passed every baseline sample can stop a run; flaky ones are just reported.
2032 const passedBefore = (id: string) => baseline.get(id) === 1;
2033 // Before the paid runs, so a missing git cannot lose their results.
@@ -1793,9 +2044,10 @@ const main = Effect.gen(function* () {
2044 const privateValues = [config.apiKey, ...Object.values(config.proxy?.headers ?? {})]
2045 .flatMap((value) => [value, value.replace(/^Bearer /i, "")])
2046 .filter(Boolean);
1796 - return runSample(testCase, config, baseFor(testCase)).pipe(
2047 + return runSample(testCase, config, baseFor(testCase), usage).pipe(
2048 inFlight.withPermits(1),
2049 Effect.matchCause({
2050 + // Not graded: a sample that errors is left out of pass rates, with its tokens so far.
2051 onFailure: (cause) => ({
2052 ...identity,
2053 ...usage,
@@ -1810,7 +2062,9 @@ const main = Effect.gen(function* () {
2062 return { ...identity, pass: checks.every((check) => check.pass), checks, ...sample };
2063 },
2064 }),
1813 - Effect.tap((row) => Effect.log(`${row.id}: ${row.pass ? "pass" : "FAIL"}`)),
2065 + Effect.tap((row) =>
2066 + Effect.log(`${row.id}: ${"error" in row ? "ERROR" : row.pass ? "pass" : "FAIL"}`),
2067 + ),
2068 );
2069 };
2070 // Without --samples, a case whose 2 samples disagree gets a 3rd to settle it.
@@ -1836,7 +2090,8 @@ const main = Effect.gen(function* () {
2090 runCase(testCase).pipe(
2091 Effect.flatMap((caseRows) => {
2092 rows.push(...caseRows);
1839 - if (!stopEarly || !passedBefore(testCase.id) || passRate(caseRows) > 0.5) {
2093 + // A case whose samples all errored has no rate, so it is no regression either.
2094 + if (!stopEarly || !passedBefore(testCase.id) || !(passRate(caseRows) <= 0.5)) {
2095 return Effect.void;
2096 }
2097 regressions.push(testCase.id);