The best way to control Claude Code, Codex and more from any computer or phone
| 1 | # Relay observability |
| 2 | |
| 3 | > For maintainers. Using T3 Code? See [docs/user](../user/). |
| 4 | |
| 5 | The relay Alchemy stack owns a shared Axiom trace setup: |
| 6 | |
| 7 | - `t3-code-relay-traces-prod`, the OpenTelemetry trace dataset shared by the Worker, mobile app, and |
| 8 | first-party relay clients |
| 9 | - `t3-code-relay-otel-ingest-prod`, the dataset-scoped Worker ingest token |
| 10 | - `t3-code-mobile-otel-ingest-prod`, the dataset-scoped mobile ingest token |
| 11 | - `t3-code-relay-client-otel-ingest-prod`, the dataset-scoped first-party relay-client ingest token |
| 12 | - `t3-code-relay-recent-spans-prod`, a view of recent request and endpoint spans |
| 13 | - `t3-code-mobile-logs-prod` and `t3-code-mobile-metrics-prod`, the mobile app's crash, error, and |
| 14 | startup data from `expo-observe` (see [Mobile crashes and errors](#mobile-crashes-and-errors)) |
| 15 | - `t3-code-mobile-telemetry-ingest-prod`, the ingest token the Worker forwards that data with |
| 16 | - `t3-code-mobile-sourcemaps-prod`, the R2 bucket of mobile JavaScript source maps, and |
| 17 | `t3-code-mobile-sourcemaps-upload-prod`, a Cloudflare token that can only write objects to it |
| 18 | |
| 19 | Alchemy stages append their sanitized stage name to isolate resources, for example |
| 20 | `t3-code-relay-traces-dev-julius` for a personal stage. |
| 21 | |
| 22 | Deploy from `infra/relay` with the normal Alchemy workflow: |
| 23 | |
| 24 | ```sh |
| 25 | vp run deploy |
| 26 | ``` |
| 27 | |
| 28 | Alchemy resolves account-level Axiom deployment credentials through its provider. At runtime, the |
| 29 | Worker receives only its scoped ingest token. Mobile and relay clients use their own separately |
| 30 | provisioned scoped ingest tokens. |
| 31 | |
| 32 | The Worker emits Effect's built-in HTTP server spans plus endpoint and database child spans. |
| 33 | Effect's OpenTelemetry exporter stores semantic HTTP attributes below the `attributes.` prefix. |
| 34 | For example: |
| 35 | |
| 36 | ```apl |
| 37 | ['t3-code-relay-traces-prod'] |
| 38 | | where kind == 'server' |
| 39 | | extend endpoint = column_ifexists('attributes.http.route', ''), |
| 40 | customAttributes = column_ifexists('attributes.custom', dynamic({})) |
| 41 | | project _time, name, trace_id, duration, |
| 42 | ['attributes.http.request.method'], |
| 43 | ['attributes.url.path'], |
| 44 | ['attributes.http.response.status_code'], |
| 45 | endpoint, |
| 46 | relayOperation = customAttributes['relay']['operation'] |
| 47 | | order by _time desc |
| 48 | | limit 200 |
| 49 | ``` |
| 50 | |
| 51 | The provisioned view also reads the endpoint from `attributes.http.route`. Relay-specific span |
| 52 | annotations are stored under `attributes.custom`; `relay.operation` is one of the emitted custom |
| 53 | attributes. |
| 54 | |
| 55 | Agents should prefer the provisioned view or APL queries for completed incidents instead of |
| 56 | tailing the Cloudflare Worker. The stack does not provision a separate query token. Responders who |
| 57 | need scripted query access use the authorized account-level `AXIOM_TOKEN` together with |
| 58 | `AXIOM_ORG_ID`; scoped ingest tokens remain write-only credentials for their producers. |
| 59 | |
| 60 | DPoP proof failures include the stable `relay.dpop.failure_code` span attribute. A `time_window` |
| 61 | failure means that a signed proof was too old or too far in the future for the relay's allowed |
| 62 | window. It can point to a date or time problem on either device, but it can also result from a |
| 63 | delayed request. The client uses this category, and the absence of a category from an older relay, |
| 64 | to decide whether clock skew is confirmed or only one possible cause. |
| 65 | |
| 66 | ## Webhooks |
| 67 | |
| 68 | A public webhook request is one `relay.hooks.forward` span. Its `relay.hook.outcome` says what |
| 69 | happened: `forwarded`, `held`, `rate_limited`, `inbox_full`, `not_found`, `payload_too_large`, |
| 70 | `environment_unavailable`, or `environment_timeout`. `relay.hook.endpoint_key` identifies the |
| 71 | managed endpoint, and with it the environment. On a forward, `relay.hook.upstream_status` or |
| 72 | `relay.hook.upstream_error` records the environment's answer, and |
| 73 | `relay.hook.upstream_outcome` what it did with the request (`accepted`, `duplicate`, |
| 74 | `prompt_too_long`, `queue_full`, `rejected_signature`, `expired`, `disabled`, ...), from its |
| 75 | `x-t3-hook-outcome` response header. A held request's delivery records the same on its |
| 76 | `relay.inbox.deliver` span. The relay sends its own `traceparent` with each forward and drops any a |
| 77 | sender supplied, and signs each forward (`x-t3-relay-delivery`) so the environment trusts the |
| 78 | relay's delivery id, receive time and trace context only from the relay, so on environments that export to T3 Connect, the environment's |
| 79 | `ScheduledTaskService.triggerWebhook` span lands in the same trace. `relay.hook.rate_limit` says which |
| 80 | budget ran out: `endpoint` or `hook`. `relay.hook.rate_limiter_failed_open` is set when the |
| 81 | Cloudflare rate limiter was unavailable and the request went through unlimited. |
| 82 | |
| 83 | Held requests are handled in the endpoint's `HookInboxObject`, and each call into it is its own |
| 84 | trace, not a child of the forward span: `relay.inbox.hold`, `relay.inbox.wake`, and |
| 85 | `relay.inbox.deliver` for each alarm run. Each carries `relay.hook.endpoint_key`, so they join to |
| 86 | forwards. The details are on the child spans: `HookInboxStore.hold` carries |
| 87 | `relay.inbox.refused` when the inbox refused a request (`max_per_hook`, `max_requests`, |
| 88 | `max_bytes`, or `already_held`). `HookInboxStore.deliverDue` carries `relay.inbox.run_result` |
| 89 | (`drained`, `more_pending`, `busy`, or `unreachable`), how many requests it sent and delivered, |
| 90 | the consecutive failures behind the retry delay, the longest time a delivered request waited, and |
| 91 | the backlog left. A run that errored outright has `relay.inbox.run_result = failed` on the root |
| 92 | `relay.inbox.deliver` span instead. |
| 93 | |
| 94 | Questions these answer: |
| 95 | |
| 96 | ```apl |
| 97 | // Webhook outcomes per hour |
| 98 | ['t3-code-relay-traces-prod'] |
| 99 | | where name == 'relay.hooks.forward' |
| 100 | | summarize count() by bin(_time, 1h), outcome = tostring(['attributes.custom']['relay.hook.outcome']) |
| 101 | |
| 102 | // Inboxes stuck behind an unreachable environment, by endpoint |
| 103 | ['t3-code-relay-traces-prod'] |
| 104 | | where name == 'HookInboxStore.deliverDue' |
| 105 | | extend c = ['attributes.custom'] |
| 106 | | where tostring(c['relay.inbox.run_result']) == 'unreachable' |
| 107 | | project trace_id, failures = toint(c['relay.inbox.consecutive_failures']), |
| 108 | backlog = toint(c['relay.inbox.held_count']) |
| 109 | | join kind=inner ( |
| 110 | ['t3-code-relay-traces-prod'] |
| 111 | | where name == 'relay.inbox.deliver' |
| 112 | | project trace_id, endpoint = tostring(['attributes.custom']['relay.hook.endpoint_key']) |
| 113 | ) on trace_id |
| 114 | | summarize runs = count(), failures = max(failures), backlog = max(backlog) by endpoint |
| 115 | ``` |
| 116 | |
| 117 | ## Alerts |
| 118 | |
| 119 | The stack provisions threshold monitors and a `t3-code-relay-health` dashboard from |
| 120 | [`infra/relay/src/monitoring.ts`](../../infra/relay/src/monitoring.ts) when the deployment sets |
| 121 | `RELAY_ALERT_NOTIFIER_ID`. Paging monitors go to that notifier and need two failing runs in a row. |
| 122 | Warnings go to `RELAY_WARNING_NOTIFIER_ID` when it is set and are otherwise only visible in Axiom. |
| 123 | Each monitor's description gives its healthy baseline. |
| 124 | |
| 125 | Monitors count HTTP status codes, not span errors. A request the relay cuts off at its 9 s deadline |
| 126 | ends as an interrupted span without an error flag, but still answers 504. `/connect` 502s are |
| 127 | normal: offline hosts produce most of them, retried by their clients every few seconds. |
| 128 | |
| 129 | The dashboard's "Connecting" charts show connections as users see them. Web, desktop, and mobile |
| 130 | export a root `relay.connection.attempt` span per attempt, which lasts until initial sync finishes, in |
| 131 | the same trace as the relay's and the host's spans. Most reconnects reuse a cached relay token and go |
| 132 | straight to the host, so relay latency only affects cold connects. A slow total with fast phases |
| 133 | usually means the client waited out an unreachable direct (LAN) route before falling back to the relay. |
| 134 | |
| 135 | **Links failing / tunnel recovery failing.** Open the dashboard's "Link failures by reason" table. |
| 136 | `provisioning: ensure-tunnel` or `configure-tunnel` with Cloudflare 429s on the "Cloudflare API calls |
| 137 | by status" chart is the #16810 failure: the account's Cloudflare API rate limit. Ask Cloudflare to |
| 138 | raise it, and check whether something is calling the API in a loop (tunnel recovery from hosts is the |
| 139 | largest caller). Slow Postgres instead points at the database or Hyperdrive. |
| 140 | |
| 141 | **Environments unreachable.** Clients reach fewer hosts than usual. Check Cloudflare Tunnel status and |
| 142 | recent relay deploys before suspecting hosts: a host-side problem does not move this many |
| 143 | environments at once. |
| 144 | |
| 145 | **Push delivery failing.** Look at the `relay.apns_deliveries.process_signed_job` or `relay.fcm.process` |
| 146 | error messages. An expired APNs key or FCM service account fails every job. |
| 147 | |
| 148 | **Cloudflare API rate limited / near its limit.** The warning fires on sustained 429s; the page fires |
| 149 | when they are frequent enough to stall provisioning. Compare the call volume on the dashboard with the |
| 150 | account limit, which Cloudflare raised to 20,000 calls per 5 minutes in October 2026 (peak use then |
| 151 | was about 2,100). |
| 152 | |
| 153 | **Traffic dropped.** The Worker is down, a deploy broke it, or spans stopped reaching Axiom. Check |
| 154 | `https://relay.t3.codes/health` first. |
| 155 | |
| 156 | **Tunnel cleanup stopped / tunnel deletions failing.** Monitors for the five-minute reaper that deletes |
| 157 | idle Cloudflare tunnels. "Stopped" pages after two hours without a finished sweep, so it also fires |
| 158 | when every sweep fails; a sweep fails when listing tunnels does, behind Cloudflare 429s or listings |
| 159 | that outlast the sweep's list timeout. "Deletions failing" is a warning. Idle tunnels are |
| 160 | billed until the reaper catches up, which the "Idle tunnels in Cloudflare" chart shows. A failed |
| 161 | sweep records no counts, so while sweeps fail the chart stops at the last good one; check the tunnel |
| 162 | list in the Cloudflare dashboard for current numbers. For failed |
| 163 | deletions, read the relay's "Failed to delete an inactive managed tunnel" warnings. To pause cleanup, |
| 164 | see [Managed tunnel cleanup rollout](release.md#managed-tunnel-cleanup-rollout). |
| 165 | |
| 166 | **Links degraded / database slow.** Warnings for slow degradation. Link failures above 20% over six |
| 167 | hours have so far meant provisioning running into the 9 s deadline, with Postgres latency as the |
| 168 | usual cause. |
| 169 | |
| 170 | ## Mobile crashes and errors |
| 171 | |
| 172 | The mobile app reports through [`expo-observe`](https://docs.expo.dev/eas/observe/). Its |
| 173 | `endpointUrl` points at the relay (`/v1/mobile/observe`, set in `apps/mobile/app.config.ts` from |
| 174 | `T3CODE_RELAY_URL`), because Observe cannot send auth headers and prefixes every path with the EAS |
| 175 | project id. The relay forwards each request to Axiom with its own token: logs (JavaScript errors, |
| 176 | native crashes, log events) to `t3-code-mobile-logs-*`, metrics to `t3-code-mobile-metrics-*`, and |
| 177 | spans to the shared traces dataset. Expo's Observe dashboard receives nothing. The endpoint URL is |
| 178 | compiled into the native app, so pointing it elsewhere takes a store build. |
| 179 | |
| 180 | Hermes stacks from release builds are minified bytecode offsets. On the way through, the relay |
| 181 | rewrites each JavaScript exception's `exception.stacktrace` with the bundle's source map from R2 |
| 182 | and records the outcome on the log record: |
| 183 | |
| 184 | - `relay.symbolication.status`: `symbolicated`, or why not. `no_bundle_id` is a development or |
| 185 | local build. `source_map_missing` means no map was uploaded for the bundle; |
| 186 | `source_map_too_large`, `source_map_invalid` and `source_map_unavailable` (R2 unreadable, retried |
| 187 | shortly) mean one was. `source_map_limit` is a request naming more bundles than the relay decodes |
| 188 | at once, and `no_bundle_frames` a stack with no frames from the bundle. |
| 189 | - `relay.symbolication.source_map`: the R2 key used |
| 190 | - `relay.symbolication.raw_stacktrace`: the original stack, when it was rewritten |
| 191 | |
| 192 | iOS frames lose their install path (`/private/var/containers/Bundle/Application/<uuid>/...`), which |
| 193 | differs per install, so `exception.stacktrace` groups across users. |
| 194 | |
| 195 | A fatal JavaScript error ends in a native abort, which Observe reports as a second crash with a |
| 196 | generic React Native stack. The relay marks that one `relay.crash.js_fatal` when both arrive in one |
| 197 | request, as they do on the next launch. Leave it out to count real native crashes: |
| 198 | |
| 199 | ```apl |
| 200 | ['t3-code-mobile-logs-prod'] |
| 201 | | where ['attributes.expo.error.source'] == 'nativeCrash' |
| 202 | | where isnull(['attributes.relay.crash.js_fatal']) |
| 203 | | summarize count() by ['attributes.exception.type'], ['attributes.exception.message'] |
| 204 | ``` |
| 205 | |
| 206 | Native crash stacks (`expo.error.source == 'nativeCrash'`) are not symbolicated. On iOS, app frames |
| 207 | read like `main (T3Code + 19476)`: the offset into the app binary. The build hook uploads that |
| 208 | build's dSYMs to `ios/dsym/<easBuildId>.zip`; download it with an account R2 token, then: |
| 209 | |
| 210 | ```sh |
| 211 | unzip <easBuildId>.zip |
| 212 | atos -arch arm64 -o T3Code.app.dSYM/Contents/Resources/DWARF/T3Code -l 0x100000000 \ |
| 213 | $(printf '0x%x' $((0x100000000 + 19476))) |
| 214 | ``` |
| 215 | |
| 216 | Maps are keyed by the bundle that crashed: `<platform>/ota/<updateId>.map` for OTA updates and |
| 217 | `<platform>/build/<easBuildId>.map` for the bundle embedded in a store build. The production |
| 218 | mobile workflow uploads OTA maps after `eas update`, and the `eas-build-on-success` hook in |
| 219 | `apps/mobile/package.json` uploads a build's map, both with `apps/mobile/scripts/upload-sourcemaps.mts`. |
| 220 | They need the `MOBILE_SOURCEMAPS_R2` secret, a JSON value the stack outputs as |
| 221 | `mobileSourceMapsUploadCredentials` from a token that can only write to the bucket. Creating that |
| 222 | token needs **Account API Tokens: Edit** on the deploy token. Deploy output shows it redacted; read |
| 223 | it from state, where it is under `__redacted__`: |
| 224 | |
| 225 | ```sh |
| 226 | cd infra/relay && vp exec alchemy state read T3CodeRelay/prod/output |
| 227 | ``` |
| 228 | |
| 229 | Set it as a GitHub Actions secret, for OTA updates, and as an EAS environment variable with |
| 230 | `secret` visibility in the `production` environment only, for builds. Keep it out of `preview`: |
| 231 | PR preview builds run the PR's code in that environment. In CI only the upload step gets it, not |
| 232 | `eas update` and the dependencies it runs. If the deploy replaces the token, update both. |
| 233 | |
| 234 | The `t3-code-mobile-health` dashboard is the place to look every few days: crash and handled |
| 235 | error counts, top crashes with the installs and app versions they hit, crashes by version, whether |
| 236 | stacks are symbolicated, and whether reports reach the relay at all. A crash counts once: the |
| 237 | native abort after a JS crash (`relay.crash.js_fatal`) is left out. Startup times are metrics, so |
| 238 | they are not on it; query `t3-code-mobile-metrics-*` with MPL. Prod and PR previews get the |
| 239 | dashboard, and deployments with `RELAY_ALERT_NOTIFIER_ID` get these monitors |
| 240 | ([`infra/relay/src/mobileTelemetry/monitoring.ts`](../../infra/relay/src/mobileTelemetry/monitoring.ts)): |
| 241 | |
| 242 | - **crash spike** (pages): crashes per 15 min well above the same time over the last 7 days. Until |
| 243 | a week of data exists it has no baseline. A spike right after a release usually means a bad OTA |
| 244 | update; roll it back in EAS. |
| 245 | - **one crash hitting many installs** (pages): a single crash on 25 installs within an hour. |
| 246 | - **reports stopped** (warning): no startup metrics forwarded for an hour, so crashes are not |
| 247 | arriving either. |
| 248 | - **crash stacks not symbolicated** (warning): over half of release JS stacks stayed minified for |
| 249 | 6 h. `relay.symbolication.status` names the cause. |
| 250 | |
| 251 | Thresholds are starting points, not measured baselines; revisit them after two weeks of |
| 252 | production data. |
| 253 | |
| 254 | Without it, uploads are skipped with a warning and stacks stay minified. Each ingest request is one |
| 255 | `relay.mobile_telemetry.ingest` span; `relay.mobile_telemetry.outcome` is `forwarded`, |
| 256 | `rate_limited`, `body_too_large`, `upstream_unavailable`, or `upstream_timeout`. |