The best way to control Claude Code, Codex and more from any computer or phone

main
md 227 lines 13.8 KB
Rendered Raw
1 # T3 Connect
2
3 T3 Connect uses Clerk for cloud identity. The relay manages environment links,
4 credentials for reaching environments, and managed tunnel allocations. After
5 bootstrap, clients send application traffic through the environment's tunnel
6 hostname; the relay Worker does not proxy their HTTP or WebSocket sessions.
7 The one exception is automation webhooks: the relay forwards
8 `/v1/hooks/:endpointKey/:hookId/:token` to the environment's tunnel so
9 senders get a stable URL. `endpointKey` is the 16-hex hash suffix of the
10 managed tunnel's name, not the environment id. It keeps bodies and tokens out
11 of its traces and leaves token and signature verification to the environment
12 ([forwarder](../../infra/relay/src/hooks/HookForwarder.ts)).
13
14 By default the forwarder stores nothing. An environment can opt in to having
15 the relay hold requests while it is offline
16 (`hold_webhooks_while_offline` on its links). Only then does the relay store the
17 raw request, including the hook token in the path, in a Durable Object with
18 SQLite storage. There is one object per managed link, keyed by its endpoint
19 key: each account that links the environment gets its own held requests,
20 order, and caps. The object pushes held requests back through the tunnel from
21 its alarm, oldest first within each hook, and backs off while the environment
22 stays away. A hook that answers busy is skipped for the rest of that run, so
23 other hooks on the link can overtake it. When
24 the tunnel reconnects, the environment asks the relay to deliver right away to
25 each of its links. Requests are deleted once the environment answers, after 24
26 hours, when their link is unlinked, or when the environment turns holding off,
27 which clears every link it proved. Both clears are best effort: unlinking logs a
28 failed clear and continues, and turning holding off stops at the first link it
29 cannot clear, so those requests wait for the 24-hour expiry. The relay still
30 never checks the token;
31 delivery goes through the same environment route.
32 Every forward carries `x-t3-relay-delivery-id`, so a request that reached the
33 environment before a timeout and is delivered again later runs once
34 ([inbox object](../../infra/relay/src/hooks/HookInboxObject.ts)).
35
36 A Durable Object, not Postgres or Queues, because held requests are write-once,
37 read-once bodies of up to 1 MiB that need per-link order, caps, and retry
38 timing. Queues cap messages at 128 KB and cannot hold one link's requests back
39 while its environment is away.
40
41 Clerk, deployment, and native authentication setup live in the
42 [Connect setup runbook](../operations/connect-setup.md).
43
44 ## The relay is a trusted broker
45
46 An authenticated cloud user still needs an active environment link. The relay
47 asks that environment to mint a one-time bootstrap credential bound to the
48 client's DPoP key. The client exchanges it directly with the environment for an
49 [environment session](./environment-auth.md). The relay never receives that
50 session token, and possessing the bootstrap credential alone does not permit
51 redeeming it without the client's private key.
52
53 Both sides authenticate this exchange. The environment accepts only bounded,
54 replay-guarded relay proofs for its own identity, linked user, and requested
55 operation. Signed environment responses bind the result to the request nonce;
56 mint responses also bind the credential to the client proof key. The relay
57 verifies those bindings before returning a credential. This prevents a different
58 process behind the tunnel from impersonating the linked environment. The checks
59 meet in the
60 [environment link service](../../apps/server/src/cloud/CloudLink.ts) and
61 [relay connector](../../infra/relay/src/environments/EnvironmentConnector.ts).
62
63 The relay holds the signing authority for mint requests. DPoP protects an honest
64 exchange from credential reuse; it does not make a compromised relay signing
65 key harmless. Keep that trust assumption explicit when changing the protocol.
66
67 Device approval, off by default, limits what that authority buys. With
68 `requireConnectDeviceApproval` on, the environment trusts T3 Connect device
69 keys, meaning DPoP thumbprints, and not the relay. A mint for an untrusted key
70 still succeeds, so the relay protocol does not change. The credential it returns
71 cannot be spent until someone with `access:write` on the environment approves
72 that key, so a forged mint yields a request that waits for approval, never a
73 session. The waiting device retries the whole mint, and the environment hands
74 back the same open request, so approvers see one entry with a stable code
75 derived from the thumbprint. Approval is per key and does not expire. It ends
76 only when the key is revoked. Revoking a session bound to a key revokes the key,
77 all its sessions, and its unspent credentials, unless the caller's own session
78 holds the same key; otherwise the device would mint itself a new session within
79 seconds. While approval is off, every key that completes an
80 exchange is recorded as trusted. Turning approval on therefore keeps current
81 devices working and lists them for review, including any that got in before.
82 See `issueConnectCredential` in the
83 [environment auth service](../../apps/server/src/auth/EnvironmentAuth.ts).
84
85 Managed tunnels expose only this server's own loopback HTTP origin. The link
86 proof signs that origin from the server's listening address and ignores the
87 caller's, so any client holding `relay:configure`, local or remote, can link
88 without being able to point the tunnel at another service on the host. The
89 relay refuses to relink an active environment id under a different key: an
90 account bearer alone would otherwise take over that environment's tunnel and
91 trusted key with an origin it signed itself. Unlinking first releases the
92 tunnel. The
93 relay resolves endpoints from its own managed allocations rather than a
94 caller-supplied URL. Health and mint
95 requests must not follow redirects. These restrictions keep endpoint discovery
96 from turning into arbitrary relay egress or exposing another service on the
97 environment host.
98
99 ## A link outlives a connector process
100
101 CLI authorization, desired exposure, and a running connector have different
102 lifetimes. Linking can record intent while the server is stopped. Startup
103 reconciles that intent. CLI logout removes the stored cloud credential and
104 disables exposure without uninstalling the environment's background service.
105
106 Managed allocations belong to a user/environment pair. Provisioning checkpoints
107 external tunnel and DNS resources so retries can reconcile partial work. A
108 normal shutdown of a CLI-managed link releases its tunnel to avoid paying for
109 an idle resource, retaining the hostname reservation for the next startup.
110 It also retains the allocation record so the environment remains "offline"
111 rather than becoming "not authorized".
112
113 Two cases must retain the tunnel across shutdown. A link installed through a
114 client has no startup provisioning path and depends on its stored connector
115 token. An update handoff immediately starts a replacement server, and replacing
116 the tunnel would add routing propagation delay to every update. These exceptions
117 belong to [shutdown handling](../../apps/server/src/cloud/CloudLink.ts).
118
119 Release and unlink claim the allocation generation before deleting external
120 resources. A delayed cleanup must not delete a tunnel reused by a concurrent
121 restart or relink. Unlink commits authorization revocation before external
122 teardown, because a database failure must leave the active link usable. Failed
123 teardown retains enough state to retry. See the
124 [managed endpoint lifecycle](../../infra/relay/src/environments/ManagedEndpointProvider.ts).
125
126 ## Idle tunnels are reclaimed and recovered
127
128 Cloudflare bills a tunnel whether or not a connector is attached, so a laptop
129 that sleeps with a linked environment leaves a paid tunnel behind. The relay's
130 five-minute maintenance job can reclaim those tunnels. `RELAY_TUNNEL_CLEANUP_MODE`
131 selects `off`, `dry-run`, or `enabled`, with `off` as the default. The mode is
132 read at deploy time, so changing it means a relay deploy, not a variable flip.
133 A candidate is a same-stage tunnel that Cloudflare reports down for at least
134 five minutes, or one that never connected and is at least an hour old. The
135 longer grace for never-connected tunnels covers a pairing still in progress.
136
137 Cleanup deletes only tunnels whose host has registered recovery. Allocations
138 without recovery registration belong to hosts that cannot replace a deleted
139 tunnel and are left alone. Allocations with no recorded tunnel ID, or a
140 different tunnel ID, are skipped because a provision may own them. A tunnel with
141 no allocation row at all is counted as `skippedOrphan` and never deleted: there
142 is no row to lock, so a relink that adopts it by name could race the delete.
143 Clear those by hand. Each sweep is bounded: at most ten list requests, 100 deletions, a
144 two-minute deadline, and an early stop on a Cloudflare rate limit. Each sweep
145 starts one budget further along the candidate list, so a block of deletes that
146 keep failing cannot starve the tunnels listed after them. See the
147 [reaper](../../infra/relay/src/environments/ManagedEndpointReaper.ts).
148
149 A host registers recovery at startup by sending its tunnel ID and loopback
150 origin with a short-lived signature from the environment key. Registration
151 touches Cloudflare only when the local host or port changed, and once per
152 existing allocation on the first registration after the upgrade because the
153 stored origin is empty. First registrations are jittered so an auto-update wave
154 does not hit the relay at once. The host stores a confirmed-origin marker with
155 the connector config, and a later boot starts the connector before registration
156 only when that marker matches the current config and port. If registration
157 cannot reach the relay for ten minutes, the host starts its stored config anyway
158 and keeps registering in the background until it can reconcile the origin.
159 If the connector exits, or `cloudflared` reports repeated tunnel
160 rejections, the host asks the relay for a replacement, at most once every two
161 minutes. The relay
162 provisions under the same allocation, so the hostname and DNS record survive
163 and clients keep their bindings. Every mutation on an allocation bumps its
164 `generation`, and deletion locks the row at the generation it claimed, so a
165 host that reconnects mid-sweep wins.
166
167 ## A paid T3 Chat plan raises the tunnel cap
168
169 T3 Chat uses WorkOS, not Clerk, so the two accounts are linked by consent
170 rather than by a shared identity. The relay and T3 Chat share one secret
171 (`T3CHAT_LINK_SECRET` here, `T3CODE_LINK_SECRET` there). The relay signs a
172 short-lived token naming the Clerk user and email; the user confirms it while
173 signed in to T3 Chat, and T3 Chat stores the link. T3 Chat owns the link and the
174 plan, so the relay keeps no copy. It asks T3 Chat only when an account reaches
175 its normal cap for a new environment, and any failure keeps the normal cap.
176 An environment that already holds a counted slot restarts without asking, so a
177 downgrade or a T3 Chat outage only blocks new environments. A manual override
178 row replaces the default for new environments, and a T3 Chat plan does not
179 raise it
180 ([limits](../../infra/relay/src/environments/ManagedTunnelLimits.ts)).
181
182 ## The relay client follows the server's pin
183
184 The host runs `cloudflared` pinned by `CLOUDFLARED_VERSION` in
185 [`relayClient.ts`](../../packages/shared/src/relayClient.ts), and bumping it there
186 (version, URLs, and checksums) is the whole release step. A linked host that
187 starts a server with a new pin keeps its connector up on the newest older managed
188 release, installs the pinned one in the background, then starts a second
189 connector on it for the same tunnel. The old connector stops only after the new
190 one registers, so the swap never takes the host off the relay. If the new one
191 fails to spawn, the old one keeps serving and the install retries later. A host
192 with no relay client installs one first. A `cloudflared` on `PATH` or an explicit `T3CODE_CLOUDFLARED_PATH` is the
193 user's choice and is never replaced, since linking asks before downloading.
194
195 Once the pinned release registers a connection, managed releases older than the
196 pin are deleted except the newest of them. That one, and any newer release, can
197 belong to another server sharing the same T3 home, such as a rollback after a
198 failed update, which could not run again without it.
199
200 The version comes from running `cloudflared version`, not from the folder name.
201 Connectors from before `--no-autoupdate` replaced themselves in place, so a
202 managed folder can hold another release. Such a binary still runs as a fallback
203 while the pin installs. Every binary needs `CLOUDFLARED_MIN_VERSION`, the oldest
204 release that accepts every flag the connector is started with and rejects the
205 compromised eMudhra intermediates. No binary below it runs, even as a fallback,
206 so a host without an eligible one stays off the relay until it has one. Raise it
207 whenever a new flag or security fix needs it.
208
209 ## OAuth traps
210
211 Interactive clients and the headless CLI use the same Clerk application but
212 different credentials. The relay accepts both session-template JWTs and CLI
213 OAuth tokens; requiring a JWT template for the CLI would reject valid logins.
214 The CLI is a public OAuth client using PKCE and stores no client secret.
215
216 Loopback CLI authorization starts on the hosted `/connect` page so sign-in
217 completes before entering Clerk's authorize endpoint. Sending a signed-out
218 browser straight to that endpoint loses the authorize parameters during the
219 sign-in redirect. The [shared flow](../../packages/shared/src/connectAuth.ts)
220 preserves PKCE and state for the loopback callback.
221
222 SSH and headless sessions use Clerk's OAuth device authorization grant because
223 the browser cannot ordinarily reach a listener on the remote machine. The CLI
224 polls Clerk's token endpoint directly while the user approves a short code on
225 Clerk's hosted device page; the hosted app plays no part and there is no
226 redirect URI or PKCE. The grant must be enabled on the CLI OAuth application
227 or the device endpoint returns an error before any prompt is shown.