The best way to control Claude Code, Codex and more from any computer or phone

main
md 256 lines 15 KB
Rendered Raw
1 # Relay observability
2
3 > For maintainers. Using T3 Code? See [docs/user](../user/).
4
5 The relay Alchemy stack owns a shared Axiom trace setup:
6
7 - `t3-code-relay-traces-prod`, the OpenTelemetry trace dataset shared by the Worker, mobile app, and
8 first-party relay clients
9 - `t3-code-relay-otel-ingest-prod`, the dataset-scoped Worker ingest token
10 - `t3-code-mobile-otel-ingest-prod`, the dataset-scoped mobile ingest token
11 - `t3-code-relay-client-otel-ingest-prod`, the dataset-scoped first-party relay-client ingest token
12 - `t3-code-relay-recent-spans-prod`, a view of recent request and endpoint spans
13 - `t3-code-mobile-logs-prod` and `t3-code-mobile-metrics-prod`, the mobile app's crash, error, and
14 startup data from `expo-observe` (see [Mobile crashes and errors](#mobile-crashes-and-errors))
15 - `t3-code-mobile-telemetry-ingest-prod`, the ingest token the Worker forwards that data with
16 - `t3-code-mobile-sourcemaps-prod`, the R2 bucket of mobile JavaScript source maps, and
17 `t3-code-mobile-sourcemaps-upload-prod`, a Cloudflare token that can only write objects to it
18
19 Alchemy stages append their sanitized stage name to isolate resources, for example
20 `t3-code-relay-traces-dev-julius` for a personal stage.
21
22 Deploy from `infra/relay` with the normal Alchemy workflow:
23
24 ```sh
25 vp run deploy
26 ```
27
28 Alchemy resolves account-level Axiom deployment credentials through its provider. At runtime, the
29 Worker receives only its scoped ingest token. Mobile and relay clients use their own separately
30 provisioned scoped ingest tokens.
31
32 The Worker emits Effect's built-in HTTP server spans plus endpoint and database child spans.
33 Effect's OpenTelemetry exporter stores semantic HTTP attributes below the `attributes.` prefix.
34 For example:
35
36 ```apl
37 ['t3-code-relay-traces-prod']
38 | where kind == 'server'
39 | extend endpoint = column_ifexists('attributes.http.route', ''),
40 customAttributes = column_ifexists('attributes.custom', dynamic({}))
41 | project _time, name, trace_id, duration,
42 ['attributes.http.request.method'],
43 ['attributes.url.path'],
44 ['attributes.http.response.status_code'],
45 endpoint,
46 relayOperation = customAttributes['relay']['operation']
47 | order by _time desc
48 | limit 200
49 ```
50
51 The provisioned view also reads the endpoint from `attributes.http.route`. Relay-specific span
52 annotations are stored under `attributes.custom`; `relay.operation` is one of the emitted custom
53 attributes.
54
55 Agents should prefer the provisioned view or APL queries for completed incidents instead of
56 tailing the Cloudflare Worker. The stack does not provision a separate query token. Responders who
57 need scripted query access use the authorized account-level `AXIOM_TOKEN` together with
58 `AXIOM_ORG_ID`; scoped ingest tokens remain write-only credentials for their producers.
59
60 DPoP proof failures include the stable `relay.dpop.failure_code` span attribute. A `time_window`
61 failure means that a signed proof was too old or too far in the future for the relay's allowed
62 window. It can point to a date or time problem on either device, but it can also result from a
63 delayed request. The client uses this category, and the absence of a category from an older relay,
64 to decide whether clock skew is confirmed or only one possible cause.
65
66 ## Webhooks
67
68 A public webhook request is one `relay.hooks.forward` span. Its `relay.hook.outcome` says what
69 happened: `forwarded`, `held`, `rate_limited`, `inbox_full`, `not_found`, `payload_too_large`,
70 `environment_unavailable`, or `environment_timeout`. `relay.hook.endpoint_key` identifies the
71 managed endpoint, and with it the environment. On a forward, `relay.hook.upstream_status` or
72 `relay.hook.upstream_error` records the environment's answer, and
73 `relay.hook.upstream_outcome` what it did with the request (`accepted`, `duplicate`,
74 `prompt_too_long`, `queue_full`, `rejected_signature`, `expired`, `disabled`, ...), from its
75 `x-t3-hook-outcome` response header. A held request's delivery records the same on its
76 `relay.inbox.deliver` span. The relay sends its own `traceparent` with each forward and drops any a
77 sender supplied, and signs each forward (`x-t3-relay-delivery`) so the environment trusts the
78 relay's delivery id, receive time and trace context only from the relay, so on environments that export to T3 Connect, the environment's
79 `ScheduledTaskService.triggerWebhook` span lands in the same trace. `relay.hook.rate_limit` says which
80 budget ran out: `endpoint` or `hook`. `relay.hook.rate_limiter_failed_open` is set when the
81 Cloudflare rate limiter was unavailable and the request went through unlimited.
82
83 Held requests are handled in the endpoint's `HookInboxObject`, and each call into it is its own
84 trace, not a child of the forward span: `relay.inbox.hold`, `relay.inbox.wake`, and
85 `relay.inbox.deliver` for each alarm run. Each carries `relay.hook.endpoint_key`, so they join to
86 forwards. The details are on the child spans: `HookInboxStore.hold` carries
87 `relay.inbox.refused` when the inbox refused a request (`max_per_hook`, `max_requests`,
88 `max_bytes`, or `already_held`). `HookInboxStore.deliverDue` carries `relay.inbox.run_result`
89 (`drained`, `more_pending`, `busy`, or `unreachable`), how many requests it sent and delivered,
90 the consecutive failures behind the retry delay, the longest time a delivered request waited, and
91 the backlog left. A run that errored outright has `relay.inbox.run_result = failed` on the root
92 `relay.inbox.deliver` span instead.
93
94 Questions these answer:
95
96 ```apl
97 // Webhook outcomes per hour
98 ['t3-code-relay-traces-prod']
99 | where name == 'relay.hooks.forward'
100 | summarize count() by bin(_time, 1h), outcome = tostring(['attributes.custom']['relay.hook.outcome'])
101
102 // Inboxes stuck behind an unreachable environment, by endpoint
103 ['t3-code-relay-traces-prod']
104 | where name == 'HookInboxStore.deliverDue'
105 | extend c = ['attributes.custom']
106 | where tostring(c['relay.inbox.run_result']) == 'unreachable'
107 | project trace_id, failures = toint(c['relay.inbox.consecutive_failures']),
108 backlog = toint(c['relay.inbox.held_count'])
109 | join kind=inner (
110 ['t3-code-relay-traces-prod']
111 | where name == 'relay.inbox.deliver'
112 | project trace_id, endpoint = tostring(['attributes.custom']['relay.hook.endpoint_key'])
113 ) on trace_id
114 | summarize runs = count(), failures = max(failures), backlog = max(backlog) by endpoint
115 ```
116
117 ## Alerts
118
119 The stack provisions threshold monitors and a `t3-code-relay-health` dashboard from
120 [`infra/relay/src/monitoring.ts`](../../infra/relay/src/monitoring.ts) when the deployment sets
121 `RELAY_ALERT_NOTIFIER_ID`. Paging monitors go to that notifier and need two failing runs in a row.
122 Warnings go to `RELAY_WARNING_NOTIFIER_ID` when it is set and are otherwise only visible in Axiom.
123 Each monitor's description gives its healthy baseline.
124
125 Monitors count HTTP status codes, not span errors. A request the relay cuts off at its 9 s deadline
126 ends as an interrupted span without an error flag, but still answers 504. `/connect` 502s are
127 normal: offline hosts produce most of them, retried by their clients every few seconds.
128
129 The dashboard's "Connecting" charts show connections as users see them. Web, desktop, and mobile
130 export a root `relay.connection.attempt` span per attempt, which lasts until initial sync finishes, in
131 the same trace as the relay's and the host's spans. Most reconnects reuse a cached relay token and go
132 straight to the host, so relay latency only affects cold connects. A slow total with fast phases
133 usually means the client waited out an unreachable direct (LAN) route before falling back to the relay.
134
135 **Links failing / tunnel recovery failing.** Open the dashboard's "Link failures by reason" table.
136 `provisioning: ensure-tunnel` or `configure-tunnel` with Cloudflare 429s on the "Cloudflare API calls
137 by status" chart is the #16810 failure: the account's Cloudflare API rate limit. Ask Cloudflare to
138 raise it, and check whether something is calling the API in a loop (tunnel recovery from hosts is the
139 largest caller). Slow Postgres instead points at the database or Hyperdrive.
140
141 **Environments unreachable.** Clients reach fewer hosts than usual. Check Cloudflare Tunnel status and
142 recent relay deploys before suspecting hosts: a host-side problem does not move this many
143 environments at once.
144
145 **Push delivery failing.** Look at the `relay.apns_deliveries.process_signed_job` or `relay.fcm.process`
146 error messages. An expired APNs key or FCM service account fails every job.
147
148 **Cloudflare API rate limited / near its limit.** The warning fires on sustained 429s; the page fires
149 when they are frequent enough to stall provisioning. Compare the call volume on the dashboard with the
150 account limit, which Cloudflare raised to 20,000 calls per 5 minutes in October 2026 (peak use then
151 was about 2,100).
152
153 **Traffic dropped.** The Worker is down, a deploy broke it, or spans stopped reaching Axiom. Check
154 `https://relay.t3.codes/health` first.
155
156 **Tunnel cleanup stopped / tunnel deletions failing.** Monitors for the five-minute reaper that deletes
157 idle Cloudflare tunnels. "Stopped" pages after two hours without a finished sweep, so it also fires
158 when every sweep fails; a sweep fails when listing tunnels does, behind Cloudflare 429s or listings
159 that outlast the sweep's list timeout. "Deletions failing" is a warning. Idle tunnels are
160 billed until the reaper catches up, which the "Idle tunnels in Cloudflare" chart shows. A failed
161 sweep records no counts, so while sweeps fail the chart stops at the last good one; check the tunnel
162 list in the Cloudflare dashboard for current numbers. For failed
163 deletions, read the relay's "Failed to delete an inactive managed tunnel" warnings. To pause cleanup,
164 see [Managed tunnel cleanup rollout](release.md#managed-tunnel-cleanup-rollout).
165
166 **Links degraded / database slow.** Warnings for slow degradation. Link failures above 20% over six
167 hours have so far meant provisioning running into the 9 s deadline, with Postgres latency as the
168 usual cause.
169
170 ## Mobile crashes and errors
171
172 The mobile app reports through [`expo-observe`](https://docs.expo.dev/eas/observe/). Its
173 `endpointUrl` points at the relay (`/v1/mobile/observe`, set in `apps/mobile/app.config.ts` from
174 `T3CODE_RELAY_URL`), because Observe cannot send auth headers and prefixes every path with the EAS
175 project id. The relay forwards each request to Axiom with its own token: logs (JavaScript errors,
176 native crashes, log events) to `t3-code-mobile-logs-*`, metrics to `t3-code-mobile-metrics-*`, and
177 spans to the shared traces dataset. Expo's Observe dashboard receives nothing. The endpoint URL is
178 compiled into the native app, so pointing it elsewhere takes a store build.
179
180 Hermes stacks from release builds are minified bytecode offsets. On the way through, the relay
181 rewrites each JavaScript exception's `exception.stacktrace` with the bundle's source map from R2
182 and records the outcome on the log record:
183
184 - `relay.symbolication.status`: `symbolicated`, or why not. `no_bundle_id` is a development or
185 local build. `source_map_missing` means no map was uploaded for the bundle;
186 `source_map_too_large`, `source_map_invalid` and `source_map_unavailable` (R2 unreadable, retried
187 shortly) mean one was. `source_map_limit` is a request naming more bundles than the relay decodes
188 at once, and `no_bundle_frames` a stack with no frames from the bundle.
189 - `relay.symbolication.source_map`: the R2 key used
190 - `relay.symbolication.raw_stacktrace`: the original stack, when it was rewritten
191
192 iOS frames lose their install path (`/private/var/containers/Bundle/Application/<uuid>/...`), which
193 differs per install, so `exception.stacktrace` groups across users.
194
195 A fatal JavaScript error ends in a native abort, which Observe reports as a second crash with a
196 generic React Native stack. The relay marks that one `relay.crash.js_fatal` when both arrive in one
197 request, as they do on the next launch. Leave it out to count real native crashes:
198
199 ```apl
200 ['t3-code-mobile-logs-prod']
201 | where ['attributes.expo.error.source'] == 'nativeCrash'
202 | where isnull(['attributes.relay.crash.js_fatal'])
203 | summarize count() by ['attributes.exception.type'], ['attributes.exception.message']
204 ```
205
206 Native crash stacks (`expo.error.source == 'nativeCrash'`) are not symbolicated. On iOS, app frames
207 read like `main (T3Code + 19476)`: the offset into the app binary. The build hook uploads that
208 build's dSYMs to `ios/dsym/<easBuildId>.zip`; download it with an account R2 token, then:
209
210 ```sh
211 unzip <easBuildId>.zip
212 atos -arch arm64 -o T3Code.app.dSYM/Contents/Resources/DWARF/T3Code -l 0x100000000 \
213 $(printf '0x%x' $((0x100000000 + 19476)))
214 ```
215
216 Maps are keyed by the bundle that crashed: `<platform>/ota/<updateId>.map` for OTA updates and
217 `<platform>/build/<easBuildId>.map` for the bundle embedded in a store build. The production
218 mobile workflow uploads OTA maps after `eas update`, and the `eas-build-on-success` hook in
219 `apps/mobile/package.json` uploads a build's map, both with `apps/mobile/scripts/upload-sourcemaps.mts`.
220 They need the `MOBILE_SOURCEMAPS_R2` secret, a JSON value the stack outputs as
221 `mobileSourceMapsUploadCredentials` from a token that can only write to the bucket. Creating that
222 token needs **Account API Tokens: Edit** on the deploy token. Deploy output shows it redacted; read
223 it from state, where it is under `__redacted__`:
224
225 ```sh
226 cd infra/relay && vp exec alchemy state read T3CodeRelay/prod/output
227 ```
228
229 Set it as a GitHub Actions secret, for OTA updates, and as an EAS environment variable with
230 `secret` visibility in the `production` environment only, for builds. Keep it out of `preview`:
231 PR preview builds run the PR's code in that environment. In CI only the upload step gets it, not
232 `eas update` and the dependencies it runs. If the deploy replaces the token, update both.
233
234 The `t3-code-mobile-health` dashboard is the place to look every few days: crash and handled
235 error counts, top crashes with the installs and app versions they hit, crashes by version, whether
236 stacks are symbolicated, and whether reports reach the relay at all. A crash counts once: the
237 native abort after a JS crash (`relay.crash.js_fatal`) is left out. Startup times are metrics, so
238 they are not on it; query `t3-code-mobile-metrics-*` with MPL. Prod and PR previews get the
239 dashboard, and deployments with `RELAY_ALERT_NOTIFIER_ID` get these monitors
240 ([`infra/relay/src/mobileTelemetry/monitoring.ts`](../../infra/relay/src/mobileTelemetry/monitoring.ts)):
241
242 - **crash spike** (pages): crashes per 15 min well above the same time over the last 7 days. Until
243 a week of data exists it has no baseline. A spike right after a release usually means a bad OTA
244 update; roll it back in EAS.
245 - **one crash hitting many installs** (pages): a single crash on 25 installs within an hour.
246 - **reports stopped** (warning): no startup metrics forwarded for an hour, so crashes are not
247 arriving either.
248 - **crash stacks not symbolicated** (warning): over half of release JS stacks stayed minified for
249 6 h. `relay.symbolication.status` names the cause.
250
251 Thresholds are starting points, not measured baselines; revisit them after two weeks of
252 production data.
253
254 Without it, uploads are skipped with a warning and stacks stay minified. Each ingest request is one
255 `relay.mobile_telemetry.ingest` span; `relay.mobile_telemetry.outcome` is `forwarded`,
256 `rate_limited`, `body_too_large`, `upstream_unavailable`, or `upstream_timeout`.