@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 03:37 UTC

main
md 156 lines 9.23 KB
Rendered Raw
1 # Devices
2
3 The environment server owns simulators and emulators the way it owns
4 terminals: discovery, streaming, and agent access all run there, and every
5 client reaches them through the environment connection. This is what makes the
6 Device panel work over Tailscale and T3 Connect, including when an SSH host runs the devices.
7
8 ## Two external tools, one seam
9
10 [expo-device-hub](../../apps/server/src/device/LocalDeviceHost.ts) streams and
11 [agent-device](../../apps/server/src/device/AgentDeviceShim.ts) drives. Each is
12 npm-installed at a pinned version into the T3 home after its matching Device
13 panel consent step. Manual setup installs and starts only expo-device-hub;
14 agent-device remains absent and stopped until agent access is granted. Both run
15 with the server's Node; `npx` would make the first `device_open` after a reboot
16 depend on the registry. The hub is a supervised child rather than an imported
17 middleware because serve-sim loads private CoreSimulator frameworks through a
18 native addon, and a crash there must not take the server down.
19
20 Everything platform-specific sits behind
21 [`DeviceHost`](../../apps/server/src/device/DeviceHost.ts). The service lists,
22 boots, and shuts down devices through the host's device operations, and the
23 proxy and the MCP tools only see a hub origin and an agent-device endpoint.
24 Hosts that run expo-device-hub share one hub-backed implementation of those
25 operations, so a host whose devices live elsewhere only replaces them.
26 SSH hosts forward both endpoints to server loopback. Every proxied request
27 also carries the host id; device ids alone are not unique across hosts.
28
29 ## Expo hosts run billed sessions
30
31 An [Expo host](../../apps/server/src/device/EasDeviceHost.ts) has no machine:
32 each opened device is an Expo Simulators session that bills the Expo account
33 until it stops. Every session is created with the host's run and idle limits,
34 the host refuses to start past its session limit, and closing the host stops
35 its sessions. Sessions it starts are recorded under the state directory and
36 stopped by the next server if this one dies; sessions other tools started on
37 the same account are never touched.
38
39 Each session brings its own preview host (serve-sim, or serve-emu for Android)
40 and agent-device daemon, with their own tokens. The proxy asks the host to
41 route each request (`DeviceHostReady.upstream`) and adds the session token server-side, so
42 clients never hold it. Agent config files are written per device
43 (`DeviceOperations.agentEndpoint`), which is why `device_open` opens the
44 device before targeting it. Device settings and tools run `simctl` or `adb` on
45 the device's machine, which Expo does not expose, so they are unavailable there.
46
47 ## Simlock hosts lease through the agent-device front
48
49 A [Simlock host](../../apps/server/src/device/SimlockDeviceHost.ts) (alpha) opens a
50 device by running T3's own agent-device CLI against the gateway's agent-device
51 front (`connect proxy`, then `open`), not with Simlock's lease HTTP API:
52 agent-device cannot adopt a lease it did not allocate. The lease is bound to
53 that one agent-device session and to the directory `open` ran in, so
54 `agentEndpoint` hands every thread the same session, a `stateDir` holding its
55 connection record, and no `--udid`/`--serial` (the gateway refuses selectors).
56 Installs work only from that directory; `device_open` names it.
57
58 Simlock serves a lease's screen and input to its holder in serve-sim's formats
59 on both platforms: AVCC envelopes at `/v1/leases/<id>/stream` and serve-sim's
60 input messages at `/v1/leases/<id>/input`, both with the agent token. Clients
61 therefore treat a Simlock Android lease like iOS (`serveSimOnly` in the stream
62 client), and `upstream` maps serve-sim's `stream.avcc` and `helper/ws` paths to
63 those routes. Simlock has no MJPEG, screenshot, or host command routes, so the
64 panel offers only input there.
65 Leases are renewed while tracked and released at the host's age limit; one a
66 crashed server left behind ends on its own one TTL later, so there is no ledger.
67
68 ## The hub is never exposed
69
70 serve-sim has a shell-exec route whose token is readable from its own
71 unauthenticated `/api`, and serve-emu's action routes have no auth at all. The
72 hub binds loopback and the only way in is the
73 [proxy](../../apps/server/src/device/DeviceHubProxy.ts), which allowlists the
74 stream, config, and screenshot routes and authenticates every request as an
75 environment session. `<img>` and `WebSocket` cannot carry headers, so the proxy
76 authenticates like the `/ws` upgrade: cookie, or a short-lived `wsTicket` that
77 bearer and DPoP clients mint over authenticated HTTP. The ticket is stripped
78 before the request reaches the hub.
79
80 Stream responses carry `Cache-Control: no-transform`; the compression
81 middleware would otherwise buffer an MJPEG body that never ends. Browsers paint an
82 MJPEG part only once the next part begins and the hub writes one only when the
83 screen changes, so the proxy
84 [re-sends the last part](../../apps/server/src/device/mjpegStillFrame.ts) after a
85 short quiet period. In browser dev, the Vite proxy must forward WebSocket
86 upgrades for `/api`, not only `/ws`.
87
88 ## Device settings never go through the hub
89
90 serve-sim's preview drives its Tools panel by sending shell commands over that
91 same exec channel. Proxying it, even allowlisted, would hand any environment
92 session arbitrary command execution on the host, so T3 does not. The
93 [`device.action`](../../apps/server/src/device/DeviceActions.ts) RPC runs the
94 underlying `simctl`, `adb`, and serve-sim helper binaries itself through
95 `DeviceHostReady.run`, one typed action per control, and returns the settings
96 it reads back. The proxy allowlist grows only with read routes (accessibility
97 tree, foreground app, event log) and refuses non-GET methods everywhere except
98 screenshot capture and stream tuning.
99
100 ## Agents drive through the CLI
101
102 The `device_*` toolkit is deliberately four tools: list, open, screenshot, and
103 close. Driving happens through the `agent-device` CLI, which has the semantic
104 snapshot model agents need and stays current with its own releases. T3 prepends
105 a shim directory to the provider's PATH. The CLI installs on the environment
106 server even when that server cannot run simulators. Hosts start on demand.
107
108 That environment is fixed when the provider subprocess spawns, so
109 [`prepareMcpSession`](../../apps/server/src/orchestration-v2/ProviderSessionManager.ts)
110 starts agent-device only when device support and agent access have both been
111 enabled, the session has the `device` capability, and the machine can run at
112 least one platform. Starting it later from `device_open` would leave the
113 already-running agent without the CLI.
114
115 How to drive a device is returned from `device_open`, not kept in an
116 always-loaded prompt or skill: it costs nothing in threads that never open a
117 device and cannot drift from the pinned CLI version. `device_open` returns a
118 launcher bound to its agent-device session, so agents need no target flags.
119 The launcher leaves out the flags a command refuses (`cdp` talks to Metro, not
120 to the device), refuses the commands that would replace T3's connection, and
121 checks that connection first: agent-device reports a sandboxed shell and a
122 dropped tunnel alike as an unreachable daemon, so the launcher names which one it
123 is. Ending a session (`device_close`, the Device panel, a shutdown) also closes
124 its agent-device session, which quits the app that session opened and releases the CLI's claim on the device.
125
126 Agents choose between these tools and ssh before they open anything, so
127 sessions that hold `device` get a short block from
128 [`orchestrationInstructions.ts`](../../packages/provider-core/src/server/orchestrationInstructions.ts)
129 through their adapter's instruction channel. It points simulator and emulator UI
130 work on local and SSH hosts at the tools; `simctl`, `adb` and ssh stay
131 available for what the tools do not cover. Pi and Muse do not send it yet.
132 Claude's native subagents never see the session's system prompt, so a
133 `SubagentStart` hook passes them the block, where the runtime runs hooks and the
134 subagent's context is not isolated.
135
136 ## The viewer decodes both vendored protocols
137
138 The hub vendors two streaming servers with different wire formats. iOS video is
139 an HTTP body of AVCC envelopes decoded with WebCodecs, with input on a separate
140 binary WebSocket; Android multiplexes SEMU-framed H.264 and JSON gestures over
141 one WebSocket. [`DeviceStreamView.tsx`](../../apps/web/src/components/device/DeviceStreamView.tsx)
142 speaks both so one panel covers both platforms.
143
144 WebCodecs is secure-context only, so the viewer probes `isConfigSupported` and falls back to
145 MJPEG on iOS when decoding is unavailable. Android's SEMU socket has no MJPEG;
146 there the panel reports that it cannot decode.
147
148 serve-sim can also be locked to WebRTC, as on Expo Simulators, and then refuses
149 AVCC with a 409 naming the transport. The viewer still asks for AVCC first, so
150 hubs that serve it pay nothing, and switches on that 409, or when its MJPEG
151 image fails in a browser that has `RTCPeerConnection`. Offers are not trickled:
152 serve-sim takes one offer with every candidate. The video element only feeds
153 the same canvas sink, so the panel and the mobile WebView keep one surface.
154 Android asks serve-emu's `/api` before opening its video socket and streams
155 WebRTC when it is advertised (Expo Simulators), keeping a separate `video=0`
156 socket for input so input can reconnect without restarting video.