The best way to control Claude Code, Codex and more from any computer or phone
Mirrored from https://github.com/Camo67/t3code.git
Synced Nov 21, 03:37 UTC
| 1 | # Devices |
| 2 | |
| 3 | The environment server owns simulators and emulators the way it owns |
| 4 | terminals: discovery, streaming, and agent access all run there, and every |
| 5 | client reaches them through the environment connection. This is what makes the |
| 6 | Device panel work over Tailscale and T3 Connect, including when an SSH host runs the devices. |
| 7 | |
| 8 | ## Two external tools, one seam |
| 9 | |
| 10 | [expo-device-hub](../../apps/server/src/device/LocalDeviceHost.ts) streams and |
| 11 | [agent-device](../../apps/server/src/device/AgentDeviceShim.ts) drives. Each is |
| 12 | npm-installed at a pinned version into the T3 home after its matching Device |
| 13 | panel consent step. Manual setup installs and starts only expo-device-hub; |
| 14 | agent-device remains absent and stopped until agent access is granted. Both run |
| 15 | with the server's Node; `npx` would make the first `device_open` after a reboot |
| 16 | depend on the registry. The hub is a supervised child rather than an imported |
| 17 | middleware because serve-sim loads private CoreSimulator frameworks through a |
| 18 | native addon, and a crash there must not take the server down. |
| 19 | |
| 20 | Everything platform-specific sits behind |
| 21 | [`DeviceHost`](../../apps/server/src/device/DeviceHost.ts). The service lists, |
| 22 | boots, and shuts down devices through the host's device operations, and the |
| 23 | proxy and the MCP tools only see a hub origin and an agent-device endpoint. |
| 24 | Hosts that run expo-device-hub share one hub-backed implementation of those |
| 25 | operations, so a host whose devices live elsewhere only replaces them. |
| 26 | SSH hosts forward both endpoints to server loopback. Every proxied request |
| 27 | also carries the host id; device ids alone are not unique across hosts. |
| 28 | |
| 29 | ## Expo hosts run billed sessions |
| 30 | |
| 31 | An [Expo host](../../apps/server/src/device/EasDeviceHost.ts) has no machine: |
| 32 | each opened device is an Expo Simulators session that bills the Expo account |
| 33 | until it stops. Every session is created with the host's run and idle limits, |
| 34 | the host refuses to start past its session limit, and closing the host stops |
| 35 | its sessions. Sessions it starts are recorded under the state directory and |
| 36 | stopped by the next server if this one dies; sessions other tools started on |
| 37 | the same account are never touched. |
| 38 | |
| 39 | Each session brings its own preview host (serve-sim, or serve-emu for Android) |
| 40 | and agent-device daemon, with their own tokens. The proxy asks the host to |
| 41 | route each request (`DeviceHostReady.upstream`) and adds the session token server-side, so |
| 42 | clients never hold it. Agent config files are written per device |
| 43 | (`DeviceOperations.agentEndpoint`), which is why `device_open` opens the |
| 44 | device before targeting it. Device settings and tools run `simctl` or `adb` on |
| 45 | the device's machine, which Expo does not expose, so they are unavailable there. |
| 46 | |
| 47 | ## Simlock hosts lease through the agent-device front |
| 48 | |
| 49 | A [Simlock host](../../apps/server/src/device/SimlockDeviceHost.ts) (alpha) opens a |
| 50 | device by running T3's own agent-device CLI against the gateway's agent-device |
| 51 | front (`connect proxy`, then `open`), not with Simlock's lease HTTP API: |
| 52 | agent-device cannot adopt a lease it did not allocate. The lease is bound to |
| 53 | that one agent-device session and to the directory `open` ran in, so |
| 54 | `agentEndpoint` hands every thread the same session, a `stateDir` holding its |
| 55 | connection record, and no `--udid`/`--serial` (the gateway refuses selectors). |
| 56 | Installs work only from that directory; `device_open` names it. |
| 57 | |
| 58 | Simlock serves a lease's screen and input to its holder in serve-sim's formats |
| 59 | on both platforms: AVCC envelopes at `/v1/leases/<id>/stream` and serve-sim's |
| 60 | input messages at `/v1/leases/<id>/input`, both with the agent token. Clients |
| 61 | therefore treat a Simlock Android lease like iOS (`serveSimOnly` in the stream |
| 62 | client), and `upstream` maps serve-sim's `stream.avcc` and `helper/ws` paths to |
| 63 | those routes. Simlock has no MJPEG, screenshot, or host command routes, so the |
| 64 | panel offers only input there. |
| 65 | Leases are renewed while tracked and released at the host's age limit; one a |
| 66 | crashed server left behind ends on its own one TTL later, so there is no ledger. |
| 67 | |
| 68 | ## The hub is never exposed |
| 69 | |
| 70 | serve-sim has a shell-exec route whose token is readable from its own |
| 71 | unauthenticated `/api`, and serve-emu's action routes have no auth at all. The |
| 72 | hub binds loopback and the only way in is the |
| 73 | [proxy](../../apps/server/src/device/DeviceHubProxy.ts), which allowlists the |
| 74 | stream, config, and screenshot routes and authenticates every request as an |
| 75 | environment session. `<img>` and `WebSocket` cannot carry headers, so the proxy |
| 76 | authenticates like the `/ws` upgrade: cookie, or a short-lived `wsTicket` that |
| 77 | bearer and DPoP clients mint over authenticated HTTP. The ticket is stripped |
| 78 | before the request reaches the hub. |
| 79 | |
| 80 | Stream responses carry `Cache-Control: no-transform`; the compression |
| 81 | middleware would otherwise buffer an MJPEG body that never ends. Browsers paint an |
| 82 | MJPEG part only once the next part begins and the hub writes one only when the |
| 83 | screen changes, so the proxy |
| 84 | [re-sends the last part](../../apps/server/src/device/mjpegStillFrame.ts) after a |
| 85 | short quiet period. In browser dev, the Vite proxy must forward WebSocket |
| 86 | upgrades for `/api`, not only `/ws`. |
| 87 | |
| 88 | ## Device settings never go through the hub |
| 89 | |
| 90 | serve-sim's preview drives its Tools panel by sending shell commands over that |
| 91 | same exec channel. Proxying it, even allowlisted, would hand any environment |
| 92 | session arbitrary command execution on the host, so T3 does not. The |
| 93 | [`device.action`](../../apps/server/src/device/DeviceActions.ts) RPC runs the |
| 94 | underlying `simctl`, `adb`, and serve-sim helper binaries itself through |
| 95 | `DeviceHostReady.run`, one typed action per control, and returns the settings |
| 96 | it reads back. The proxy allowlist grows only with read routes (accessibility |
| 97 | tree, foreground app, event log) and refuses non-GET methods everywhere except |
| 98 | screenshot capture and stream tuning. |
| 99 | |
| 100 | ## Agents drive through the CLI |
| 101 | |
| 102 | The `device_*` toolkit is deliberately four tools: list, open, screenshot, and |
| 103 | close. Driving happens through the `agent-device` CLI, which has the semantic |
| 104 | snapshot model agents need and stays current with its own releases. T3 prepends |
| 105 | a shim directory to the provider's PATH. The CLI installs on the environment |
| 106 | server even when that server cannot run simulators. Hosts start on demand. |
| 107 | |
| 108 | That environment is fixed when the provider subprocess spawns, so |
| 109 | [`prepareMcpSession`](../../apps/server/src/orchestration-v2/ProviderSessionManager.ts) |
| 110 | starts agent-device only when device support and agent access have both been |
| 111 | enabled, the session has the `device` capability, and the machine can run at |
| 112 | least one platform. Starting it later from `device_open` would leave the |
| 113 | already-running agent without the CLI. |
| 114 | |
| 115 | How to drive a device is returned from `device_open`, not kept in an |
| 116 | always-loaded prompt or skill: it costs nothing in threads that never open a |
| 117 | device and cannot drift from the pinned CLI version. `device_open` returns a |
| 118 | launcher bound to its agent-device session, so agents need no target flags. |
| 119 | The launcher leaves out the flags a command refuses (`cdp` talks to Metro, not |
| 120 | to the device), refuses the commands that would replace T3's connection, and |
| 121 | checks that connection first: agent-device reports a sandboxed shell and a |
| 122 | dropped tunnel alike as an unreachable daemon, so the launcher names which one it |
| 123 | is. Ending a session (`device_close`, the Device panel, a shutdown) also closes |
| 124 | its agent-device session, which quits the app that session opened and releases the CLI's claim on the device. |
| 125 | |
| 126 | Agents choose between these tools and ssh before they open anything, so |
| 127 | sessions that hold `device` get a short block from |
| 128 | [`orchestrationInstructions.ts`](../../packages/provider-core/src/server/orchestrationInstructions.ts) |
| 129 | through their adapter's instruction channel. It points simulator and emulator UI |
| 130 | work on local and SSH hosts at the tools; `simctl`, `adb` and ssh stay |
| 131 | available for what the tools do not cover. Pi and Muse do not send it yet. |
| 132 | Claude's native subagents never see the session's system prompt, so a |
| 133 | `SubagentStart` hook passes them the block, where the runtime runs hooks and the |
| 134 | subagent's context is not isolated. |
| 135 | |
| 136 | ## The viewer decodes both vendored protocols |
| 137 | |
| 138 | The hub vendors two streaming servers with different wire formats. iOS video is |
| 139 | an HTTP body of AVCC envelopes decoded with WebCodecs, with input on a separate |
| 140 | binary WebSocket; Android multiplexes SEMU-framed H.264 and JSON gestures over |
| 141 | one WebSocket. [`DeviceStreamView.tsx`](../../apps/web/src/components/device/DeviceStreamView.tsx) |
| 142 | speaks both so one panel covers both platforms. |
| 143 | |
| 144 | WebCodecs is secure-context only, so the viewer probes `isConfigSupported` and falls back to |
| 145 | MJPEG on iOS when decoding is unavailable. Android's SEMU socket has no MJPEG; |
| 146 | there the panel reports that it cannot decode. |
| 147 | |
| 148 | serve-sim can also be locked to WebRTC, as on Expo Simulators, and then refuses |
| 149 | AVCC with a 409 naming the transport. The viewer still asks for AVCC first, so |
| 150 | hubs that serve it pay nothing, and switches on that 409, or when its MJPEG |
| 151 | image fails in a browser that has `RTCPeerConnection`. Offers are not trickled: |
| 152 | serve-sim takes one offer with every candidate. The video element only feeds |
| 153 | the same canvas sink, so the panel and the mobile WebView keep one surface. |
| 154 | Android asks serve-emu's `/api` before opening its video socket and streams |
| 155 | WebRTC when it is advertised (Expo Simulators), keeping a separate `video=0` |
| 156 | socket for input so input can reconnect without restarting video. |