The best way to control Claude Code, Codex and more from any computer or phone
| 1 | # Observability |
| 2 | |
| 3 | > For maintainers. Using T3 Code? See [docs/user](../user/). |
| 4 | |
| 5 | T3 Code has one server-side observability model: |
| 6 | |
| 7 | - pretty logs go to stdout for humans |
| 8 | - completed spans go to a local NDJSON trace file |
| 9 | - traces, metrics, and logs can also be exported over OTLP to a real backend like Grafana LGTM |
| 10 | |
| 11 | The local trace file is the persisted source of truth for normal local launches. Those launches do not |
| 12 | write a separate server log file, but SSH-managed launches also persist the remote process's |
| 13 | stdout/stderr at `~/.t3/ssh-launch/<state>/server.log`. Each managed launch starts that file |
| 14 | empty, so it holds only the latest launch's output. |
| 15 | |
| 16 | ## Where To Find Things |
| 17 | |
| 18 | ### Logs |
| 19 | |
| 20 | Logs are human-facing: |
| 21 | |
| 22 | - destination: stdout |
| 23 | - format: `Logger.consolePretty()` |
| 24 | - normal local persistence: none |
| 25 | - SSH-managed launch persistence: `~/.t3/ssh-launch/<state>/server.log` |
| 26 | - remote export: OTLP only, when configured |
| 27 | |
| 28 | If you want a log message to show up in the trace file, emit it inside an active span with `Effect.log...`. `Logger.tracerLogger` will attach it as a span event. |
| 29 | |
| 30 | Configuring a logs endpoint takes over that job. The server then exports log records, which cover |
| 31 | every message instead of only the ones inside an active span and carry the trace and span ids so |
| 32 | they still line up with the trace. `Logger.tracerLogger` is dropped in that mode, so the same |
| 33 | message is not exported twice and the trace file stops carrying log messages. stdout output and |
| 34 | SSH-managed launch persistence stay unchanged either way. |
| 35 | |
| 36 | ### Traces |
| 37 | |
| 38 | Completed spans are written as NDJSON records to `serverTracePath`. The default depends on how the |
| 39 | server starts: production and explicitly configured homes use |
| 40 | `<home>/userdata/logs/server.trace.ndjson` (so `~/.t3/userdata/...` by default, or |
| 41 | `/custom/path/userdata/...` with `--home-dir /custom/path`), a linked worktree dev run uses |
| 42 | `<worktree>/.t3/userdata/logs/server.trace.ndjson`, and an implicit dev run outside a linked |
| 43 | worktree uses `~/.t3/dev/logs/server.trace.ndjson`. |
| 44 | |
| 45 | Important fields common to both record types: |
| 46 | |
| 47 | - `type`: `effect-span` or `otlp-span` |
| 48 | - `name`: span name |
| 49 | - `traceId`, `spanId`, `parentSpanId`: correlation |
| 50 | - `durationMs`: elapsed time |
| 51 | - `attributes`: structured context |
| 52 | - `events`: embedded logs and custom events |
| 53 | |
| 54 | `effect-span` records also contain `exit` with `Success`, `Failure`, or `Interrupted`. `otlp-span` |
| 55 | records instead carry OTLP resource, scope, and optional status fields. |
| 56 | |
| 57 | The `TraceRecord`, `EffectTraceRecord`, and `OtlpTraceRecord` schemas live in |
| 58 | `packages/shared/src/observability.ts`. |
| 59 | |
| 60 | SQL statement spans (named `sqlite`) and `sql.transaction` spans carry `db.connection` and |
| 61 | `db.lock_wait_ms`. `db.connection` is `writer` for the read-write connection or `reader` for a |
| 62 | read-only one. For a database file both run in worker threads; an in-memory database has only a |
| 63 | writer, on the event loop. `db.lock_wait_ms` is the part of the |
| 64 | span spent waiting for that connection, behind other SQL. That wait does not block the event loop, |
| 65 | so subtract it before you blame a slow span on the query. Statements inside a transaction use its |
| 66 | connection and do not carry these attributes. |
| 67 | |
| 68 | An MCP request over HTTP has a `McpHttpServer.authenticateRequest` span that covers only admitting |
| 69 | it: the credential check, the mode limit, and the Linked environments gate. The request runs after |
| 70 | it, under the `McpServer.…/<method>` span, and each tool call has its own `mcp.tool.<name>` span |
| 71 | that times it. That span's attributes name the tool, never its arguments. |
| 72 | |
| 73 | DPoP proof failures include the safe `environment.dpop.failure_code` span |
| 74 | attribute. A `time_window` failure means that a signed proof was too old or too |
| 75 | far in the future for the environment server's allowed window. It can point to |
| 76 | a date or time problem on either device, but it can also result from a delayed |
| 77 | request. |
| 78 | |
| 79 | #### Summarize the trace file |
| 80 | |
| 81 | `t3 trace summary` reads the trace file and its rotated backups directly, so it works while the |
| 82 | server is stalled or stopped. It prints counts, rates, and latency percentiles per span name. Use |
| 83 | it to measure background work or to compare two builds. |
| 84 | |
| 85 | ```bash |
| 86 | t3 trace summary --since 30m --limit 40 |
| 87 | ``` |
| 88 | |
| 89 | It reads `T3CODE_TRACE_FILE` if set, else `<home>/userdata/logs/server.trace.ndjson` for |
| 90 | `--base-dir` or `T3CODE_HOME`, plus the `T3CODE_TRACE_MAX_FILES` rotated backups. For a dev run or |
| 91 | a copied file, set `T3CODE_TRACE_FILE`. `--since 30m` keeps spans that ended in the last 30 |
| 92 | minutes. The rate is per minute between the first and last span end. |
| 93 | |
| 94 | ### Metrics |
| 95 | |
| 96 | Metrics are not written to a local file. |
| 97 | |
| 98 | - local persistence: none |
| 99 | - remote export: OTLP only, when configured |
| 100 | - current definitions: `apps/server/src/observability/Metrics.ts` |
| 101 | |
| 102 | If OTLP is not configured, metrics still exist in-process, but you will not have a local artifact to inspect. |
| 103 | |
| 104 | ### Event Loop Stalls |
| 105 | |
| 106 | `apps/server/src/observability/EventLoopMonitor.ts` samples the server's event loop every 30 s. When |
| 107 | the loop stalled for more than 2 s since the previous sample, it records a root |
| 108 | `server.eventLoop.stall` span with a warning. The span has trace level `Warn`, so it stays when |
| 109 | `T3CODE_TRACE_MIN_LEVEL` is `Warn`. The warning shows in Settings > Diagnostics unless OTLP logs are |
| 110 | on. The span time is when the sample ran, not when the stall happened. |
| 111 | |
| 112 | The trace file rotates in well under an hour on a busy server, so each stall is also appended as one |
| 113 | NDJSON line to `event-loop-stalls.ndjson`, next to the trace file. A line has `time` and the span |
| 114 | attributes. The file keeps the last 1000 or more stalls, with one rotated backup |
| 115 | (`event-loop-stalls.ndjson.1`). Use it to see how often a server stalls over days. |
| 116 | |
| 117 | Some delay is not recorded: |
| 118 | |
| 119 | - `delayMaxMs` is the longest stall, and can undercount it by up to 1 s. The 2 s threshold applies to |
| 120 | this value, so a stall over 3 s is normally recorded, and a shorter one can be missed. A stall |
| 121 | that ends just as a sample runs can be missed too. |
| 122 | - Time the computer spends asleep reads as delay on macOS and Windows. So a sample only counts when |
| 123 | the loop was busy, not waiting for events, for at least `delayMaxMs`. Busy time covers the whole |
| 124 | window, so a short sleep in an otherwise busy window can still record a false stall. The span then |
| 125 | shows CPU time far below `delayMaxMs`. |
| 126 | - The first sample after launch is skipped. Startup work such as migrations and projection bootstrap |
| 127 | can block the loop for seconds on a large database. |
| 128 | |
| 129 | CPU times and page faults cover the whole process over the whole window since the previous sample. |
| 130 | The window is nominally 30 s, but a long stall delays the sample and makes the window longer. Other |
| 131 | work in the window can hide a wait, so only CPU time far below `delayMaxMs` proves the thread was |
| 132 | waiting. Read CPU together with page faults: |
| 133 | |
| 134 | - High `cpuSystemMs` with many page faults means memory pressure. Major faults are reads from disk or swap. |
| 135 | On macOS, reads from compressed memory are minor faults plus system CPU. |
| 136 | - High `cpuUserMs` with few page faults means JavaScript work or garbage collection. |
| 137 | - Low CPU with few major page faults points at synchronous disk I/O, such as SQLite reads or trace |
| 138 | file writes. |
| 139 | - Many `involuntaryContextSwitches` mean other processes were competing for the CPU. |
| 140 | |
| 141 | ### Related Artifacts |
| 142 | |
| 143 | Provider event NDJSON files still exist for provider runtime streams. Those are separate from the main server trace file. |
| 144 | |
| 145 | ## Run The Server In Instrumented Mode |
| 146 | |
| 147 | There are two useful modes: |
| 148 | |
| 149 | - local-only: stdout + local `server.trace.ndjson` |
| 150 | - full local observability: stdout + local trace file + OTLP export to Grafana/Tempo/Prometheus |
| 151 | |
| 152 | The local trace file is always on. OTLP export is opt-in. |
| 153 | |
| 154 | ### Option 1: Local Traces Only |
| 155 | |
| 156 | You do not need any extra env vars. Just run the app normally and inspect `server.trace.ndjson`. |
| 157 | |
| 158 | Examples: |
| 159 | |
| 160 | ```bash |
| 161 | t3 |
| 162 | ``` |
| 163 | |
| 164 | ```bash |
| 165 | node --run dev |
| 166 | ``` |
| 167 | |
| 168 | ```bash |
| 169 | node --run dev:desktop |
| 170 | ``` |
| 171 | |
| 172 | ### Option 2: Run With A Local LGTM Stack |
| 173 | |
| 174 | #### 1. Start Grafana LGTM |
| 175 | |
| 176 | ```bash |
| 177 | docker run --name lgtm \ |
| 178 | -p 3000:3000 \ |
| 179 | -p 4317:4317 \ |
| 180 | -p 4318:4318 \ |
| 181 | --rm -ti \ |
| 182 | grafana/otel-lgtm |
| 183 | ``` |
| 184 | |
| 185 | Then open `http://localhost:3000`. |
| 186 | |
| 187 | Default Grafana login: |
| 188 | |
| 189 | - username: `admin` |
| 190 | - password: `admin` |
| 191 | |
| 192 | #### 2. Export OTLP env vars |
| 193 | |
| 194 | For endpoint settings and startup precedence, see the |
| 195 | [user telemetry guide](../user/telemetry.md#export-diagnostics-to-your-own-receiver). |
| 196 | In a worktree, use `vp run dev` instead of `t3` to keep state isolated. |
| 197 | |
| 198 | ```bash |
| 199 | export T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces |
| 200 | export T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics |
| 201 | export T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs |
| 202 | export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=development |
| 203 | ``` |
| 204 | |
| 205 | Optional: |
| 206 | |
| 207 | ```bash |
| 208 | export T3CODE_TRACE_MIN_LEVEL=Info |
| 209 | export T3CODE_TRACE_TIMING_ENABLED=true |
| 210 | ``` |
| 211 | |
| 212 | #### 3. Launch the app from that same shell |
| 213 | |
| 214 | CLI: |
| 215 | |
| 216 | ```bash |
| 217 | t3 |
| 218 | ``` |
| 219 | |
| 220 | Monorepo web/server dev: |
| 221 | |
| 222 | ```bash |
| 223 | node --run dev |
| 224 | ``` |
| 225 | |
| 226 | Monorepo desktop dev: |
| 227 | |
| 228 | ```bash |
| 229 | node --run dev:desktop |
| 230 | ``` |
| 231 | |
| 232 | Packaged desktop app: |
| 233 | |
| 234 | Launch the actual app executable from the same shell so the desktop app and embedded backend inherit `T3CODE_OTLP_*`. |
| 235 | |
| 236 | macOS app bundle example: |
| 237 | |
| 238 | ```bash |
| 239 | T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces \ |
| 240 | T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics \ |
| 241 | T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs \ |
| 242 | "/Applications/T3 Code.app/Contents/MacOS/T3 Code" |
| 243 | ``` |
| 244 | |
| 245 | Direct binary example: |
| 246 | |
| 247 | ```bash |
| 248 | T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces \ |
| 249 | T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics \ |
| 250 | T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs \ |
| 251 | ./path/to/your/desktop-app-binary |
| 252 | ``` |
| 253 | |
| 254 | Do not rely on launching from Finder, Spotlight, the dock, or the Start menu after setting shell env vars. Those launches usually will not pick them up. |
| 255 | |
| 256 | #### 4. Fully restart after changing env |
| 257 | |
| 258 | The backend reads observability config at process start. If you change OTLP env vars, stop the app completely and start it again. |
| 259 | |
| 260 | ## How To Use Traces And Metrics To Debug The Server |
| 261 | |
| 262 | ### Start With The Local Trace File |
| 263 | |
| 264 | The trace file is the fastest way to inspect raw span data. |
| 265 | |
| 266 | Resolve the path for the launch mode once. Production and explicitly configured homes store runtime |
| 267 | state under the base directory's `userdata` folder: |
| 268 | |
| 269 | ```bash |
| 270 | TRACE_FILE="${T3CODE_HOME:-$HOME/.t3}/userdata/logs/server.trace.ndjson" |
| 271 | ``` |
| 272 | |
| 273 | A dev server started from a linked worktree defaults to that worktree's local home: |
| 274 | |
| 275 | ```bash |
| 276 | TRACE_FILE="$WORKTREE/.t3/userdata/logs/server.trace.ndjson" |
| 277 | ``` |
| 278 | |
| 279 | Only an implicit dev run outside a linked worktree uses the shared dev directory: |
| 280 | |
| 281 | ```bash |
| 282 | TRACE_FILE="$HOME/.t3/dev/logs/server.trace.ndjson" |
| 283 | ``` |
| 284 | |
| 285 | Tail the selected file: |
| 286 | |
| 287 | ```bash |
| 288 | tail -f "$TRACE_FILE" |
| 289 | ``` |
| 290 | |
| 291 | Show failed spans: |
| 292 | |
| 293 | ```bash |
| 294 | jq -c 'select(.type == "effect-span" and .exit._tag != "Success") | { |
| 295 | name, |
| 296 | durationMs, |
| 297 | exit, |
| 298 | attributes |
| 299 | }' "$TRACE_FILE" |
| 300 | ``` |
| 301 | |
| 302 | Show slow spans: |
| 303 | |
| 304 | ```bash |
| 305 | jq -c 'select(.durationMs > 1000) | { |
| 306 | name, |
| 307 | durationMs, |
| 308 | traceId, |
| 309 | spanId |
| 310 | }' "$TRACE_FILE" |
| 311 | ``` |
| 312 | |
| 313 | Inspect embedded log events: |
| 314 | |
| 315 | ```bash |
| 316 | jq -c 'select(any(.events[]?; .attributes["effect.logLevel"] != null)) | { |
| 317 | name, |
| 318 | durationMs, |
| 319 | events: [ |
| 320 | .events[] |
| 321 | | select(.attributes["effect.logLevel"] != null) |
| 322 | | { |
| 323 | message: .name, |
| 324 | level: .attributes["effect.logLevel"] |
| 325 | } |
| 326 | ] |
| 327 | }' "$TRACE_FILE" |
| 328 | ``` |
| 329 | |
| 330 | Follow one trace: |
| 331 | |
| 332 | ```bash |
| 333 | jq -r 'select(.traceId == "TRACE_ID_HERE") | [ |
| 334 | .name, |
| 335 | .spanId, |
| 336 | (.parentSpanId // "-"), |
| 337 | .durationMs |
| 338 | ] | @tsv' "$TRACE_FILE" |
| 339 | ``` |
| 340 | |
| 341 | Filter orchestration commands: |
| 342 | |
| 343 | ```bash |
| 344 | jq -c 'select(.attributes["orchestration_v2.command_type"] != null) | { |
| 345 | name, |
| 346 | durationMs, |
| 347 | commandType: .attributes["orchestration_v2.command_type"], |
| 348 | threadId: .attributes["orchestration_v2.thread_id"] |
| 349 | }' "$TRACE_FILE" |
| 350 | ``` |
| 351 | |
| 352 | Filter git activity: |
| 353 | |
| 354 | ```bash |
| 355 | jq -c 'select(.attributes["git.operation"] != null) | { |
| 356 | name, |
| 357 | durationMs, |
| 358 | operation: .attributes["git.operation"], |
| 359 | cwd: .attributes["git.cwd"], |
| 360 | hookEvents: [ |
| 361 | .events[] |
| 362 | | select(.name == "git.hook.started" or .name == "git.hook.finished") |
| 363 | ] |
| 364 | }' "$TRACE_FILE" |
| 365 | ``` |
| 366 | |
| 367 | ### Use Tempo When You Need A Real Trace Viewer |
| 368 | |
| 369 | Tempo is better than raw NDJSON when you want to: |
| 370 | |
| 371 | - search across many traces |
| 372 | - inspect parent/child relationships visually |
| 373 | - compare many slow traces |
| 374 | - drill into one failing request without hand-joining by `traceId` |
| 375 | |
| 376 | Recommended flow in Grafana: |
| 377 | |
| 378 | 1. Open `Explore`. |
| 379 | 2. Pick the `Tempo` data source. |
| 380 | 3. Set the time range to something recent like `Last 15 minutes`. |
| 381 | 4. Start broad. Do not begin with a very narrow query. |
| 382 | 5. Look for spans from the `t3code-server` or `t3code-desktop` service, then narrow by span name or |
| 383 | attributes. |
| 384 | |
| 385 | Good first searches: |
| 386 | |
| 387 | - service name `t3code-server` or `t3code-desktop`, plus a resource attribute such as |
| 388 | `deployment.environment.name` |
| 389 | - span names like `sendTurn` or a Git operation such as `GitVcsDriver.statusDetails.status` |
| 390 | - Git spans whose `git.operation` attribute identifies the operation |
| 391 | - orchestration spans with attributes like `orchestration_v2.command_type` |
| 392 | |
| 393 | Once you know traces are arriving, narrower TraceQL queries for names such as `sendTurn` or Git |
| 394 | operation names become useful. |
| 395 | |
| 396 | ### Use Metrics To See Systemic Problems |
| 397 | |
| 398 | Traces are best for one request. Metrics are best for trends. |
| 399 | |
| 400 | Good metric families to watch: |
| 401 | |
| 402 | - `t3_rpc_request_duration` |
| 403 | - `t3_orchestration_command_duration` |
| 404 | - `t3_provider_turn_duration` (how long the provider adapter takes to start a turn, not the turn's run time) |
| 405 | - `t3_git_command_duration` |
| 406 | |
| 407 | Counters tell you volume and failure rate: |
| 408 | |
| 409 | - `t3_rpc_requests_total` |
| 410 | - `t3_orchestration_commands_total` |
| 411 | - `t3_provider_turns_total` |
| 412 | - `t3_git_commands_total` |
| 413 | |
| 414 | Webhooks have their own families: |
| 415 | |
| 416 | - `t3_webhook_deliveries_total` by `outcome` and `source` (`relay` or `direct`). Beyond what the |
| 417 | sender sees, `queue_full` means a task already had its limit of deliveries waiting, |
| 418 | `prompt_too_long` means the filled-in prompt passed the provider limit, and `duplicate` means |
| 419 | the relay delivered a request this environment had already run. |
| 420 | - `t3_webhook_runs_total` by `outcome` (`started`, `skipped`, `failed`) for the runs those |
| 421 | deliveries start, which happen after the sender has its answer. |
| 422 | - `t3_webhook_held_delay` for how long requests the relay held waited before arriving. |
| 423 | |
| 424 | - `t3_secret_requests_total` by `status` (`saved`, `declined`, `cancelled`, `timed_out`) for secrets |
| 425 | agents asked users for, and `t3_secret_refs_consumed_total` by `result` (`used`, `rejected`) for |
| 426 | tools redeeming them. Neither ever carries a value. |
| 427 | |
| 428 | `ScheduledTaskService.triggerWebhook` spans carry the same outcome per request, and each run |
| 429 | started from a delivery is its own `ScheduledTaskService.runWebhookDelivery` trace. For a request |
| 430 | the relay forwarded, the span also goes to the T3 Connect trace export as a child of the relay's |
| 431 | span; requests that reach the environment directly never join a sender's trace. |
| 432 | |
| 433 | Use metrics when the question is: |
| 434 | |
| 435 | - "is this always slow?" |
| 436 | - "did this get worse after a change?" |
| 437 | - "which command type is failing most often?" |
| 438 | |
| 439 | Use traces when the question is: |
| 440 | |
| 441 | - "what happened in this specific request?" |
| 442 | - "which child span caused this one slow interaction?" |
| 443 | - "what logs were emitted inside the failing flow?" |
| 444 | |
| 445 | ## Common Workflows |
| 446 | |
| 447 | ### "Why did this request fail?" |
| 448 | |
| 449 | 1. Start with the local NDJSON file. |
| 450 | 2. Find `effect-span` records where `exit._tag != "Success"`. |
| 451 | 3. Group by `traceId`. |
| 452 | 4. Inspect sibling spans and span events. |
| 453 | 5. If needed, move to Tempo for the full trace tree. |
| 454 | |
| 455 | ### "Why is the UI feeling slow?" |
| 456 | |
| 457 | 1. Search for slow top-level spans in the trace file or Tempo. |
| 458 | 2. Check child spans for sqlite, git, provider, or terminal work. |
| 459 | 3. Look at the matching duration metrics to see whether the slowness is systemic. |
| 460 | |
| 461 | ### "Are git hooks causing latency?" |
| 462 | |
| 463 | 1. Filter `git.operation` spans. |
| 464 | 2. Inspect `git.hook.started` and `git.hook.finished` events. |
| 465 | 3. Compare hook timing to the enclosing git span duration. |
| 466 | |
| 467 | ### "Why do I have spans locally but nothing in Grafana?" |
| 468 | |
| 469 | Usually one of these is true: |
| 470 | |
| 471 | - `T3CODE_OTLP_TRACES_URL` was not set |
| 472 | - the app was launched from a different environment than the one where you exported the vars |
| 473 | - the app was not fully restarted after changing env |
| 474 | - Grafana is looking at the wrong time range or service name |
| 475 | |
| 476 | If the local NDJSON file is updating, local tracing is working. The problem is almost always OTLP export configuration or process startup. |
| 477 | |
| 478 | If metrics arrive but logs or traces have gaps, read `t3_otlp_export_attempt_failures_total` by |
| 479 | `signal` and `reason` (`status_<code>`, `transport`, `encode`). The exporter drops a batch once it |
| 480 | fails (transport errors and 408, 429, 500, 502, 503 and 504 after 3 retries; other statuses and |
| 481 | encoding at once) and then drops that signal for 60 s, logging it only at Debug, so at the default |
| 482 | log level this counter is the only record. Every retry counts, so one dropped batch reads as up to |
| 483 | four. |
| 484 | |
| 485 | ## How To Think About Adding Tracing To Future Code |
| 486 | |
| 487 | ### Prefer Boundaries Over Tiny Helpers |
| 488 | |
| 489 | Good span boundaries: |
| 490 | |
| 491 | - RPC methods |
| 492 | - orchestration command handling |
| 493 | - provider adapter calls |
| 494 | - external process calls |
| 495 | - persistence writes |
| 496 | - queue handoffs |
| 497 | |
| 498 | Avoid tracing every tiny helper. Most helpers should inherit the active span rather than create a new one. |
| 499 | |
| 500 | ### Reuse `Effect.fn(...)` Where It Already Exists |
| 501 | |
| 502 | The codebase already uses `Effect.fn("name")` heavily. That should usually be your first tracing boundary. |
| 503 | |
| 504 | For ad hoc work: |
| 505 | |
| 506 | ```ts |
| 507 | import { Effect } from "effect"; |
| 508 | |
| 509 | const runThing = Effect.gen(function* () { |
| 510 | yield* Effect.annotateCurrentSpan({ |
| 511 | "thing.id": "abc123", |
| 512 | "thing.kind": "example", |
| 513 | }); |
| 514 | |
| 515 | yield* Effect.logInfo("starting thing"); |
| 516 | return yield* doWork(); |
| 517 | }).pipe(Effect.withSpan("thing.run")); |
| 518 | ``` |
| 519 | |
| 520 | ### Put High-Cardinality Detail On Spans |
| 521 | |
| 522 | Use span annotations for IDs, paths, and other detailed context: |
| 523 | |
| 524 | ```ts |
| 525 | yield * |
| 526 | Effect.annotateCurrentSpan({ |
| 527 | "provider.thread_id": input.threadId, |
| 528 | "provider.request_id": input.requestId, |
| 529 | "git.cwd": input.cwd, |
| 530 | }); |
| 531 | ``` |
| 532 | |
| 533 | ### Keep Metric Labels Low Cardinality |
| 534 | |
| 535 | Good metric labels: |
| 536 | |
| 537 | - operation kind |
| 538 | - method name |
| 539 | - provider kind |
| 540 | - aggregate kind |
| 541 | - outcome |
| 542 | |
| 543 | Bad metric labels: |
| 544 | |
| 545 | - raw thread IDs |
| 546 | - command IDs |
| 547 | - file paths |
| 548 | - cwd |
| 549 | - full prompts |
| 550 | - full model strings when a normalized family label would do |
| 551 | |
| 552 | Detailed context belongs on spans, not metrics. |
| 553 | |
| 554 | ### Use Logs As Span Events |
| 555 | |
| 556 | Logs inside a span become part of the trace story: |
| 557 | |
| 558 | ```ts |
| 559 | yield * Effect.logInfo("starting provider turn"); |
| 560 | yield * Effect.logDebug("waiting for approval response"); |
| 561 | ``` |
| 562 | |
| 563 | Those messages show up as span events because `Logger.tracerLogger` is installed. |
| 564 | |
| 565 | ### Use The Pipeable Metrics API |
| 566 | |
| 567 | `withMetrics(...)` is the default way to attach a counter and timer to an effect: |
| 568 | |
| 569 | ```ts |
| 570 | import { someCounter, someDuration, withMetrics } from "../observability/Metrics.ts"; |
| 571 | |
| 572 | const program = doWork().pipe( |
| 573 | withMetrics({ |
| 574 | counter: someCounter, |
| 575 | timer: someDuration, |
| 576 | attributes: { |
| 577 | operation: "work", |
| 578 | }, |
| 579 | }), |
| 580 | ); |
| 581 | ``` |
| 582 | |
| 583 | ## Detailed API Reference |
| 584 | |
| 585 | ### Runtime Wiring |
| 586 | |
| 587 | The server observability layer is assembled in `apps/server/src/observability/Observability.ts`. |
| 588 | |
| 589 | It provides: |
| 590 | |
| 591 | - pretty stdout logger |
| 592 | - `Logger.tracerLogger` |
| 593 | - local NDJSON tracer |
| 594 | - optional OTLP trace exporter |
| 595 | - optional OTLP metrics exporter |
| 596 | - optional OTLP log exporter |
| 597 | - Effect trace-level and timing refs |
| 598 | |
| 599 | The desktop main process is a second producer, assembled in |
| 600 | `apps/desktop/src/app/DesktopObservability.ts`. It reads the same `T3CODE_OTLP_*` names and the same |
| 601 | Settings entries as the backend it supervises, and covers work the backend cannot see: app startup, |
| 602 | window and menu handling, backend supervision, and updates. It reports as service |
| 603 | `t3code-desktop`, so a collector shows it alongside the backend rather than mixed into it. It |
| 604 | exports traces and logs only; the main process records no metrics, so the metrics endpoint applies |
| 605 | to the backend alone. |
| 606 | |
| 607 | ### Env Vars |
| 608 | |
| 609 | Local trace file: |
| 610 | |
| 611 | - `T3CODE_TRACE_FILE`: override trace file path |
| 612 | - `T3CODE_TRACE_MAX_BYTES`: per-file rotation size, default `10485760` |
| 613 | - `T3CODE_TRACE_MAX_FILES`: rotated file count, default `10` |
| 614 | - `T3CODE_TRACE_BATCH_WINDOW_MS`: flush window, default `1000` |
| 615 | - `T3CODE_TRACE_MIN_LEVEL`: minimum trace level, default `Info` |
| 616 | - `T3CODE_TRACE_TIMING_ENABLED`: enable timing metadata, default `true` |
| 617 | |
| 618 | OTLP export: |
| 619 | |
| 620 | - `T3CODE_OTLP_TRACES_URL`: OTLP trace endpoint |
| 621 | - `T3CODE_OTLP_METRICS_URL`: OTLP metric endpoint |
| 622 | - `T3CODE_OTLP_LOGS_URL`: OTLP log endpoint |
| 623 | - `T3CODE_OTLP_EXPORT_INTERVAL_MS`: export interval, default `10000` |
| 624 | - `T3CODE_OTLP_HEADERS`: extra headers for all three exporters, same format as |
| 625 | `OTEL_EXPORTER_OTLP_HEADERS`: comma-separated `key=value` pairs with percent-encoded values. |
| 626 | - `T3CODE_OTLP_PROTOCOL`: `http/json` (default) or `http/protobuf` |
| 627 | |
| 628 | The server and the desktop app also read the standard |
| 629 | `OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT` and generic `OTEL_EXPORTER_OTLP_ENDPOINT` (with |
| 630 | `/v1/traces`, `/v1/metrics`, or `/v1/logs` appended), for a collector expecting those instead. A |
| 631 | non-blank `T3CODE_OTLP_*_URL` wins over either, and a per-signal endpoint wins over the generic one |
| 632 | for its signal. A blank value counts as unset. A signal with an OTEL endpoint takes its headers from |
| 633 | `OTEL_EXPORTER_OTLP_HEADERS` and its protocol from `OTEL_EXPORTER_OTLP_PROTOCOL` (default |
| 634 | `http/protobuf`, read case-insensitively), and a per-signal |
| 635 | `OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_HEADERS` or `_PROTOCOL` wins over the generic one for its |
| 636 | signal. `T3CODE_OTLP_HEADERS` and `T3CODE_OTLP_PROTOCOL` never apply to it. An endpoint that is not |
| 637 | an `http` or `https` URL, a protocol other than `http/protobuf` or `http/json` such as `grpc`, or |
| 638 | headers that are not `key=value` pairs with percent-encoded values turn that signal's export off |
| 639 | with a startup warning, rather than sending it to the Settings endpoint. |
| 640 | |
| 641 | Service names are fixed: `t3code-server` for the backend and `t3code-desktop` for the desktop main |
| 642 | process, both in `service.namespace` `t3code`. `OTEL_SERVICE_NAME` and a `service.name` or |
| 643 | `service.namespace` in `OTEL_RESOURCE_ATTRIBUTES` are ignored. Tell installations apart with other |
| 644 | resource attributes, such as `OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=staging`. |
| 645 | |
| 646 | Each process reports its own build, not the one it talks to: the server, the desktop main process, |
| 647 | the UI, and the mobile app each carry `t3code.release.channel` (`dev`, `stable`, `nightly`, or |
| 648 | `preview`, from their own version; desktop PR builds count as `preview`, and a server not built by a |
| 649 | release workflow reports `dev`) and `deployment.environment.name` (`development`, `production`, or |
| 650 | `staging` for `preview`). On the server and the desktop main process either key in |
| 651 | `OTEL_RESOURCE_ATTRIBUTES` wins. The UI and mobile app never read that variable: Vercel PR previews |
| 652 | set `VITE_RELEASE_CHANNEL=preview` because their `APP_VERSION` must stay comparable with the |
| 653 | server's, and the mobile app maps its build variant (`development`, `preview`, `production`), or |
| 654 | reports `dev` from a development bundle. |
| 655 | |
| 656 | Attributes specific to T3 Code use the `t3code.` prefix. The UI runs as `t3code-web` in both a |
| 657 | browser and the desktop window, since it is the same code, and `t3code.client.surface` is `desktop` |
| 658 | or `web`. The server's `t3code.server.managed_by` is `desktop` when the desktop app launched it and |
| 659 | `standalone` otherwise. The desktop main process reports `t3code.desktop.mode` as `development` or |
| 660 | `packaged` (formerly `service.mode`; dashboards filtering on that key must switch). Node processes |
| 661 | also report the semantic `process.runtime.name`, `process.runtime.version`, and |
| 662 | `process.runtime.description`, and UI traces report `browser.language`, `user_agent.original`, and, |
| 663 | in Chromium, `browser.platform`, `browser.mobile`, and `browser.brands`. |
| 664 | |
| 665 | Every resource also carries a `service.instance.id`: random per process for the server and desktop, |
| 666 | per page load for the UI, and the Expo session on mobile. A `service.instance.id` in |
| 667 | `OTEL_RESOURCE_ATTRIBUTES` replaces the generated one on the server and desktop. |
| 668 | |
| 669 | If the OTLP URLs are unset, local tracing still works, metrics stay in-process only, and logs stay |
| 670 | on stdout only. |
| 671 | |
| 672 | ### The Kill Switch |
| 673 | |
| 674 | `T3CODE_OTEL_SDK_DISABLED` and `OTEL_SDK_DISABLED` turn off every OTLP export in both the server and |
| 675 | the desktop main process, overriding any endpoint from the environment or Settings. Local trace |
| 676 | files and stdout logs are unaffected. |
| 677 | |
| 678 | `T3CODE_OTEL_SDK_DISABLED` wins when set, so `T3CODE_OTEL_SDK_DISABLED=false` re-enables export on a |
| 679 | machine that sets `OTEL_SDK_DISABLED` for everything else. It accepts the usual boolean spellings |
| 680 | (`true`/`false`, `yes`/`no`, `on`/`off`, `1`/`0`, `y`/`n`). `OTEL_SDK_DISABLED` follows the |
| 681 | OpenTelemetry specification and only `true` disables export, so `OTEL_SDK_DISABLED=1` does not. |
| 682 | Values are case-insensitive and trimmed. An unrecognized value is ignored with a startup warning. |
| 683 | |
| 684 | `OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, or `OTEL_LOGS_EXPORTER` set to `none` turns off |
| 685 | just that signal, overriding an OTEL endpoint and the Settings endpoint. A `T3CODE_OTLP_*_URL` still |
| 686 | wins for its signal. `otlp` is the default, and any other exporter name, such as `console` or |
| 687 | `prometheus`, is ignored with a startup warning. |
| 688 | |
| 689 | ### What Is Instrumented Today |
| 690 | |
| 691 | Current high-value span and metric boundaries include: |
| 692 | |
| 693 | - WebSocket RPC request spans (`ws.rpc.<method>`) and metrics in |
| 694 | `apps/server/src/observability/RpcInstrumentation.ts` |
| 695 | - startup phases |
| 696 | - orchestration command processing |
| 697 | - provider session and turn operations |
| 698 | - git command execution and git hook events |
| 699 | - terminal session lifecycle |
| 700 | - sqlite query execution, except the statements inside `orchestrationV2.EventSink.write`, which |
| 701 | its `sql.transaction` span times as one |
| 702 | - event loop stalls (`server.eventLoop.stall`) |
| 703 | |
| 704 | ### Current Constraints |
| 705 | |
| 706 | - logs outside spans are not persisted in the trace file; SSH-managed launch stdout/stderr is still |
| 707 | captured in its launcher log |
| 708 | - metrics are not snapshotted locally |
| 709 | |
| 710 | ## Heap Snapshots |
| 711 | |
| 712 | To see what a long-running server holds in memory, send it `SIGUSR2`. The server writes a V8 heap |
| 713 | snapshot to its logs dir and logs the path. This works for desktop, `npx t3`, and service installs |
| 714 | on macOS and Linux. Windows has no `SIGUSR2`. |
| 715 | |
| 716 | Send the signal to the server pid in `server-runtime.json`, which sits in the server's state dir |
| 717 | next to the `logs` dir. For a dev server or a `--home-dir` launch, use that server's state dir from |
| 718 | [Traces](#traces). Do not send it to the desktop app or the service launcher: a process without the |
| 719 | handler exits on `SIGUSR2`. After a crash the file can keep a stale pid that now belongs to a |
| 720 | different process, so check the pid first. |
| 721 | |
| 722 | ```bash |
| 723 | pid="$(jq .pid "${T3CODE_HOME:-$HOME/.t3}/userdata/server-runtime.json")" |
| 724 | ps -p "$pid" -o command= |
| 725 | ``` |
| 726 | |
| 727 | If `ps` shows the T3 Code server, send the signal: |
| 728 | |
| 729 | ```bash |
| 730 | kill -USR2 "$pid" |
| 731 | ``` |
| 732 | |
| 733 | The file is `<logsDir>/server-<pid>-<timestamp>.heapsnapshot`, next to `server.trace.ndjson`. To |
| 734 | open it, use the Memory tab in Chrome DevTools and select Load. |
| 735 | |
| 736 | Before you take one: |
| 737 | |
| 738 | - The server stops while it writes the file. For a large heap this can take a minute or more. |
| 739 | Connected clients can reconnect during the pause, and an event loop monitor, if the server has |
| 740 | one, records the pause as a stall. Send the signal once. A second signal sent during a write |
| 741 | takes another snapshot after the first one finishes. |
| 742 | - The write needs about as much free memory as the heap uses. On a machine that is already |
| 743 | swapping, it can make the problem worse or crash the server. |
| 744 | - The file contains everything in server memory, including tokens, secrets, and thread content. Do |
| 745 | not share it publicly. Delete it when you are done, because storage cleanup does not remove it. |