@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 03:37 UTC

main
md 745 lines 27.2 KB
Rendered Raw
1 # Observability
2
3 > For maintainers. Using T3 Code? See [docs/user](../user/).
4
5 T3 Code has one server-side observability model:
6
7 - pretty logs go to stdout for humans
8 - completed spans go to a local NDJSON trace file
9 - traces, metrics, and logs can also be exported over OTLP to a real backend like Grafana LGTM
10
11 The local trace file is the persisted source of truth for normal local launches. Those launches do not
12 write a separate server log file, but SSH-managed launches also persist the remote process's
13 stdout/stderr at `~/.t3/ssh-launch/<state>/server.log`. Each managed launch starts that file
14 empty, so it holds only the latest launch's output.
15
16 ## Where To Find Things
17
18 ### Logs
19
20 Logs are human-facing:
21
22 - destination: stdout
23 - format: `Logger.consolePretty()`
24 - normal local persistence: none
25 - SSH-managed launch persistence: `~/.t3/ssh-launch/<state>/server.log`
26 - remote export: OTLP only, when configured
27
28 If you want a log message to show up in the trace file, emit it inside an active span with `Effect.log...`. `Logger.tracerLogger` will attach it as a span event.
29
30 Configuring a logs endpoint takes over that job. The server then exports log records, which cover
31 every message instead of only the ones inside an active span and carry the trace and span ids so
32 they still line up with the trace. `Logger.tracerLogger` is dropped in that mode, so the same
33 message is not exported twice and the trace file stops carrying log messages. stdout output and
34 SSH-managed launch persistence stay unchanged either way.
35
36 ### Traces
37
38 Completed spans are written as NDJSON records to `serverTracePath`. The default depends on how the
39 server starts: production and explicitly configured homes use
40 `<home>/userdata/logs/server.trace.ndjson` (so `~/.t3/userdata/...` by default, or
41 `/custom/path/userdata/...` with `--home-dir /custom/path`), a linked worktree dev run uses
42 `<worktree>/.t3/userdata/logs/server.trace.ndjson`, and an implicit dev run outside a linked
43 worktree uses `~/.t3/dev/logs/server.trace.ndjson`.
44
45 Important fields common to both record types:
46
47 - `type`: `effect-span` or `otlp-span`
48 - `name`: span name
49 - `traceId`, `spanId`, `parentSpanId`: correlation
50 - `durationMs`: elapsed time
51 - `attributes`: structured context
52 - `events`: embedded logs and custom events
53
54 `effect-span` records also contain `exit` with `Success`, `Failure`, or `Interrupted`. `otlp-span`
55 records instead carry OTLP resource, scope, and optional status fields.
56
57 The `TraceRecord`, `EffectTraceRecord`, and `OtlpTraceRecord` schemas live in
58 `packages/shared/src/observability.ts`.
59
60 SQL statement spans (named `sqlite`) and `sql.transaction` spans carry `db.connection` and
61 `db.lock_wait_ms`. `db.connection` is `writer` for the read-write connection or `reader` for a
62 read-only one. For a database file both run in worker threads; an in-memory database has only a
63 writer, on the event loop. `db.lock_wait_ms` is the part of the
64 span spent waiting for that connection, behind other SQL. That wait does not block the event loop,
65 so subtract it before you blame a slow span on the query. Statements inside a transaction use its
66 connection and do not carry these attributes.
67
68 An MCP request over HTTP has a `McpHttpServer.authenticateRequest` span that covers only admitting
69 it: the credential check, the mode limit, and the Linked environments gate. The request runs after
70 it, under the `McpServer.…/<method>` span, and each tool call has its own `mcp.tool.<name>` span
71 that times it. That span's attributes name the tool, never its arguments.
72
73 DPoP proof failures include the safe `environment.dpop.failure_code` span
74 attribute. A `time_window` failure means that a signed proof was too old or too
75 far in the future for the environment server's allowed window. It can point to
76 a date or time problem on either device, but it can also result from a delayed
77 request.
78
79 #### Summarize the trace file
80
81 `t3 trace summary` reads the trace file and its rotated backups directly, so it works while the
82 server is stalled or stopped. It prints counts, rates, and latency percentiles per span name. Use
83 it to measure background work or to compare two builds.
84
85 ```bash
86 t3 trace summary --since 30m --limit 40
87 ```
88
89 It reads `T3CODE_TRACE_FILE` if set, else `<home>/userdata/logs/server.trace.ndjson` for
90 `--base-dir` or `T3CODE_HOME`, plus the `T3CODE_TRACE_MAX_FILES` rotated backups. For a dev run or
91 a copied file, set `T3CODE_TRACE_FILE`. `--since 30m` keeps spans that ended in the last 30
92 minutes. The rate is per minute between the first and last span end.
93
94 ### Metrics
95
96 Metrics are not written to a local file.
97
98 - local persistence: none
99 - remote export: OTLP only, when configured
100 - current definitions: `apps/server/src/observability/Metrics.ts`
101
102 If OTLP is not configured, metrics still exist in-process, but you will not have a local artifact to inspect.
103
104 ### Event Loop Stalls
105
106 `apps/server/src/observability/EventLoopMonitor.ts` samples the server's event loop every 30 s. When
107 the loop stalled for more than 2 s since the previous sample, it records a root
108 `server.eventLoop.stall` span with a warning. The span has trace level `Warn`, so it stays when
109 `T3CODE_TRACE_MIN_LEVEL` is `Warn`. The warning shows in Settings > Diagnostics unless OTLP logs are
110 on. The span time is when the sample ran, not when the stall happened.
111
112 The trace file rotates in well under an hour on a busy server, so each stall is also appended as one
113 NDJSON line to `event-loop-stalls.ndjson`, next to the trace file. A line has `time` and the span
114 attributes. The file keeps the last 1000 or more stalls, with one rotated backup
115 (`event-loop-stalls.ndjson.1`). Use it to see how often a server stalls over days.
116
117 Some delay is not recorded:
118
119 - `delayMaxMs` is the longest stall, and can undercount it by up to 1 s. The 2 s threshold applies to
120 this value, so a stall over 3 s is normally recorded, and a shorter one can be missed. A stall
121 that ends just as a sample runs can be missed too.
122 - Time the computer spends asleep reads as delay on macOS and Windows. So a sample only counts when
123 the loop was busy, not waiting for events, for at least `delayMaxMs`. Busy time covers the whole
124 window, so a short sleep in an otherwise busy window can still record a false stall. The span then
125 shows CPU time far below `delayMaxMs`.
126 - The first sample after launch is skipped. Startup work such as migrations and projection bootstrap
127 can block the loop for seconds on a large database.
128
129 CPU times and page faults cover the whole process over the whole window since the previous sample.
130 The window is nominally 30 s, but a long stall delays the sample and makes the window longer. Other
131 work in the window can hide a wait, so only CPU time far below `delayMaxMs` proves the thread was
132 waiting. Read CPU together with page faults:
133
134 - High `cpuSystemMs` with many page faults means memory pressure. Major faults are reads from disk or swap.
135 On macOS, reads from compressed memory are minor faults plus system CPU.
136 - High `cpuUserMs` with few page faults means JavaScript work or garbage collection.
137 - Low CPU with few major page faults points at synchronous disk I/O, such as SQLite reads or trace
138 file writes.
139 - Many `involuntaryContextSwitches` mean other processes were competing for the CPU.
140
141 ### Related Artifacts
142
143 Provider event NDJSON files still exist for provider runtime streams. Those are separate from the main server trace file.
144
145 ## Run The Server In Instrumented Mode
146
147 There are two useful modes:
148
149 - local-only: stdout + local `server.trace.ndjson`
150 - full local observability: stdout + local trace file + OTLP export to Grafana/Tempo/Prometheus
151
152 The local trace file is always on. OTLP export is opt-in.
153
154 ### Option 1: Local Traces Only
155
156 You do not need any extra env vars. Just run the app normally and inspect `server.trace.ndjson`.
157
158 Examples:
159
160 ```bash
161 t3
162 ```
163
164 ```bash
165 node --run dev
166 ```
167
168 ```bash
169 node --run dev:desktop
170 ```
171
172 ### Option 2: Run With A Local LGTM Stack
173
174 #### 1. Start Grafana LGTM
175
176 ```bash
177 docker run --name lgtm \
178 -p 3000:3000 \
179 -p 4317:4317 \
180 -p 4318:4318 \
181 --rm -ti \
182 grafana/otel-lgtm
183 ```
184
185 Then open `http://localhost:3000`.
186
187 Default Grafana login:
188
189 - username: `admin`
190 - password: `admin`
191
192 #### 2. Export OTLP env vars
193
194 For endpoint settings and startup precedence, see the
195 [user telemetry guide](../user/telemetry.md#export-diagnostics-to-your-own-receiver).
196 In a worktree, use `vp run dev` instead of `t3` to keep state isolated.
197
198 ```bash
199 export T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces
200 export T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics
201 export T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs
202 export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=development
203 ```
204
205 Optional:
206
207 ```bash
208 export T3CODE_TRACE_MIN_LEVEL=Info
209 export T3CODE_TRACE_TIMING_ENABLED=true
210 ```
211
212 #### 3. Launch the app from that same shell
213
214 CLI:
215
216 ```bash
217 t3
218 ```
219
220 Monorepo web/server dev:
221
222 ```bash
223 node --run dev
224 ```
225
226 Monorepo desktop dev:
227
228 ```bash
229 node --run dev:desktop
230 ```
231
232 Packaged desktop app:
233
234 Launch the actual app executable from the same shell so the desktop app and embedded backend inherit `T3CODE_OTLP_*`.
235
236 macOS app bundle example:
237
238 ```bash
239 T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces \
240 T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics \
241 T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs \
242 "/Applications/T3 Code.app/Contents/MacOS/T3 Code"
243 ```
244
245 Direct binary example:
246
247 ```bash
248 T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces \
249 T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics \
250 T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs \
251 ./path/to/your/desktop-app-binary
252 ```
253
254 Do not rely on launching from Finder, Spotlight, the dock, or the Start menu after setting shell env vars. Those launches usually will not pick them up.
255
256 #### 4. Fully restart after changing env
257
258 The backend reads observability config at process start. If you change OTLP env vars, stop the app completely and start it again.
259
260 ## How To Use Traces And Metrics To Debug The Server
261
262 ### Start With The Local Trace File
263
264 The trace file is the fastest way to inspect raw span data.
265
266 Resolve the path for the launch mode once. Production and explicitly configured homes store runtime
267 state under the base directory's `userdata` folder:
268
269 ```bash
270 TRACE_FILE="${T3CODE_HOME:-$HOME/.t3}/userdata/logs/server.trace.ndjson"
271 ```
272
273 A dev server started from a linked worktree defaults to that worktree's local home:
274
275 ```bash
276 TRACE_FILE="$WORKTREE/.t3/userdata/logs/server.trace.ndjson"
277 ```
278
279 Only an implicit dev run outside a linked worktree uses the shared dev directory:
280
281 ```bash
282 TRACE_FILE="$HOME/.t3/dev/logs/server.trace.ndjson"
283 ```
284
285 Tail the selected file:
286
287 ```bash
288 tail -f "$TRACE_FILE"
289 ```
290
291 Show failed spans:
292
293 ```bash
294 jq -c 'select(.type == "effect-span" and .exit._tag != "Success") | {
295 name,
296 durationMs,
297 exit,
298 attributes
299 }' "$TRACE_FILE"
300 ```
301
302 Show slow spans:
303
304 ```bash
305 jq -c 'select(.durationMs > 1000) | {
306 name,
307 durationMs,
308 traceId,
309 spanId
310 }' "$TRACE_FILE"
311 ```
312
313 Inspect embedded log events:
314
315 ```bash
316 jq -c 'select(any(.events[]?; .attributes["effect.logLevel"] != null)) | {
317 name,
318 durationMs,
319 events: [
320 .events[]
321 | select(.attributes["effect.logLevel"] != null)
322 | {
323 message: .name,
324 level: .attributes["effect.logLevel"]
325 }
326 ]
327 }' "$TRACE_FILE"
328 ```
329
330 Follow one trace:
331
332 ```bash
333 jq -r 'select(.traceId == "TRACE_ID_HERE") | [
334 .name,
335 .spanId,
336 (.parentSpanId // "-"),
337 .durationMs
338 ] | @tsv' "$TRACE_FILE"
339 ```
340
341 Filter orchestration commands:
342
343 ```bash
344 jq -c 'select(.attributes["orchestration_v2.command_type"] != null) | {
345 name,
346 durationMs,
347 commandType: .attributes["orchestration_v2.command_type"],
348 threadId: .attributes["orchestration_v2.thread_id"]
349 }' "$TRACE_FILE"
350 ```
351
352 Filter git activity:
353
354 ```bash
355 jq -c 'select(.attributes["git.operation"] != null) | {
356 name,
357 durationMs,
358 operation: .attributes["git.operation"],
359 cwd: .attributes["git.cwd"],
360 hookEvents: [
361 .events[]
362 | select(.name == "git.hook.started" or .name == "git.hook.finished")
363 ]
364 }' "$TRACE_FILE"
365 ```
366
367 ### Use Tempo When You Need A Real Trace Viewer
368
369 Tempo is better than raw NDJSON when you want to:
370
371 - search across many traces
372 - inspect parent/child relationships visually
373 - compare many slow traces
374 - drill into one failing request without hand-joining by `traceId`
375
376 Recommended flow in Grafana:
377
378 1. Open `Explore`.
379 2. Pick the `Tempo` data source.
380 3. Set the time range to something recent like `Last 15 minutes`.
381 4. Start broad. Do not begin with a very narrow query.
382 5. Look for spans from the `t3code-server` or `t3code-desktop` service, then narrow by span name or
383 attributes.
384
385 Good first searches:
386
387 - service name `t3code-server` or `t3code-desktop`, plus a resource attribute such as
388 `deployment.environment.name`
389 - span names like `sendTurn` or a Git operation such as `GitVcsDriver.statusDetails.status`
390 - Git spans whose `git.operation` attribute identifies the operation
391 - orchestration spans with attributes like `orchestration_v2.command_type`
392
393 Once you know traces are arriving, narrower TraceQL queries for names such as `sendTurn` or Git
394 operation names become useful.
395
396 ### Use Metrics To See Systemic Problems
397
398 Traces are best for one request. Metrics are best for trends.
399
400 Good metric families to watch:
401
402 - `t3_rpc_request_duration`
403 - `t3_orchestration_command_duration`
404 - `t3_provider_turn_duration` (how long the provider adapter takes to start a turn, not the turn's run time)
405 - `t3_git_command_duration`
406
407 Counters tell you volume and failure rate:
408
409 - `t3_rpc_requests_total`
410 - `t3_orchestration_commands_total`
411 - `t3_provider_turns_total`
412 - `t3_git_commands_total`
413
414 Webhooks have their own families:
415
416 - `t3_webhook_deliveries_total` by `outcome` and `source` (`relay` or `direct`). Beyond what the
417 sender sees, `queue_full` means a task already had its limit of deliveries waiting,
418 `prompt_too_long` means the filled-in prompt passed the provider limit, and `duplicate` means
419 the relay delivered a request this environment had already run.
420 - `t3_webhook_runs_total` by `outcome` (`started`, `skipped`, `failed`) for the runs those
421 deliveries start, which happen after the sender has its answer.
422 - `t3_webhook_held_delay` for how long requests the relay held waited before arriving.
423
424 - `t3_secret_requests_total` by `status` (`saved`, `declined`, `cancelled`, `timed_out`) for secrets
425 agents asked users for, and `t3_secret_refs_consumed_total` by `result` (`used`, `rejected`) for
426 tools redeeming them. Neither ever carries a value.
427
428 `ScheduledTaskService.triggerWebhook` spans carry the same outcome per request, and each run
429 started from a delivery is its own `ScheduledTaskService.runWebhookDelivery` trace. For a request
430 the relay forwarded, the span also goes to the T3 Connect trace export as a child of the relay's
431 span; requests that reach the environment directly never join a sender's trace.
432
433 Use metrics when the question is:
434
435 - "is this always slow?"
436 - "did this get worse after a change?"
437 - "which command type is failing most often?"
438
439 Use traces when the question is:
440
441 - "what happened in this specific request?"
442 - "which child span caused this one slow interaction?"
443 - "what logs were emitted inside the failing flow?"
444
445 ## Common Workflows
446
447 ### "Why did this request fail?"
448
449 1. Start with the local NDJSON file.
450 2. Find `effect-span` records where `exit._tag != "Success"`.
451 3. Group by `traceId`.
452 4. Inspect sibling spans and span events.
453 5. If needed, move to Tempo for the full trace tree.
454
455 ### "Why is the UI feeling slow?"
456
457 1. Search for slow top-level spans in the trace file or Tempo.
458 2. Check child spans for sqlite, git, provider, or terminal work.
459 3. Look at the matching duration metrics to see whether the slowness is systemic.
460
461 ### "Are git hooks causing latency?"
462
463 1. Filter `git.operation` spans.
464 2. Inspect `git.hook.started` and `git.hook.finished` events.
465 3. Compare hook timing to the enclosing git span duration.
466
467 ### "Why do I have spans locally but nothing in Grafana?"
468
469 Usually one of these is true:
470
471 - `T3CODE_OTLP_TRACES_URL` was not set
472 - the app was launched from a different environment than the one where you exported the vars
473 - the app was not fully restarted after changing env
474 - Grafana is looking at the wrong time range or service name
475
476 If the local NDJSON file is updating, local tracing is working. The problem is almost always OTLP export configuration or process startup.
477
478 If metrics arrive but logs or traces have gaps, read `t3_otlp_export_attempt_failures_total` by
479 `signal` and `reason` (`status_<code>`, `transport`, `encode`). The exporter drops a batch once it
480 fails (transport errors and 408, 429, 500, 502, 503 and 504 after 3 retries; other statuses and
481 encoding at once) and then drops that signal for 60 s, logging it only at Debug, so at the default
482 log level this counter is the only record. Every retry counts, so one dropped batch reads as up to
483 four.
484
485 ## How To Think About Adding Tracing To Future Code
486
487 ### Prefer Boundaries Over Tiny Helpers
488
489 Good span boundaries:
490
491 - RPC methods
492 - orchestration command handling
493 - provider adapter calls
494 - external process calls
495 - persistence writes
496 - queue handoffs
497
498 Avoid tracing every tiny helper. Most helpers should inherit the active span rather than create a new one.
499
500 ### Reuse `Effect.fn(...)` Where It Already Exists
501
502 The codebase already uses `Effect.fn("name")` heavily. That should usually be your first tracing boundary.
503
504 For ad hoc work:
505
506 ```ts
507 import { Effect } from "effect";
508
509 const runThing = Effect.gen(function* () {
510 yield* Effect.annotateCurrentSpan({
511 "thing.id": "abc123",
512 "thing.kind": "example",
513 });
514
515 yield* Effect.logInfo("starting thing");
516 return yield* doWork();
517 }).pipe(Effect.withSpan("thing.run"));
518 ```
519
520 ### Put High-Cardinality Detail On Spans
521
522 Use span annotations for IDs, paths, and other detailed context:
523
524 ```ts
525 yield *
526 Effect.annotateCurrentSpan({
527 "provider.thread_id": input.threadId,
528 "provider.request_id": input.requestId,
529 "git.cwd": input.cwd,
530 });
531 ```
532
533 ### Keep Metric Labels Low Cardinality
534
535 Good metric labels:
536
537 - operation kind
538 - method name
539 - provider kind
540 - aggregate kind
541 - outcome
542
543 Bad metric labels:
544
545 - raw thread IDs
546 - command IDs
547 - file paths
548 - cwd
549 - full prompts
550 - full model strings when a normalized family label would do
551
552 Detailed context belongs on spans, not metrics.
553
554 ### Use Logs As Span Events
555
556 Logs inside a span become part of the trace story:
557
558 ```ts
559 yield * Effect.logInfo("starting provider turn");
560 yield * Effect.logDebug("waiting for approval response");
561 ```
562
563 Those messages show up as span events because `Logger.tracerLogger` is installed.
564
565 ### Use The Pipeable Metrics API
566
567 `withMetrics(...)` is the default way to attach a counter and timer to an effect:
568
569 ```ts
570 import { someCounter, someDuration, withMetrics } from "../observability/Metrics.ts";
571
572 const program = doWork().pipe(
573 withMetrics({
574 counter: someCounter,
575 timer: someDuration,
576 attributes: {
577 operation: "work",
578 },
579 }),
580 );
581 ```
582
583 ## Detailed API Reference
584
585 ### Runtime Wiring
586
587 The server observability layer is assembled in `apps/server/src/observability/Observability.ts`.
588
589 It provides:
590
591 - pretty stdout logger
592 - `Logger.tracerLogger`
593 - local NDJSON tracer
594 - optional OTLP trace exporter
595 - optional OTLP metrics exporter
596 - optional OTLP log exporter
597 - Effect trace-level and timing refs
598
599 The desktop main process is a second producer, assembled in
600 `apps/desktop/src/app/DesktopObservability.ts`. It reads the same `T3CODE_OTLP_*` names and the same
601 Settings entries as the backend it supervises, and covers work the backend cannot see: app startup,
602 window and menu handling, backend supervision, and updates. It reports as service
603 `t3code-desktop`, so a collector shows it alongside the backend rather than mixed into it. It
604 exports traces and logs only; the main process records no metrics, so the metrics endpoint applies
605 to the backend alone.
606
607 ### Env Vars
608
609 Local trace file:
610
611 - `T3CODE_TRACE_FILE`: override trace file path
612 - `T3CODE_TRACE_MAX_BYTES`: per-file rotation size, default `10485760`
613 - `T3CODE_TRACE_MAX_FILES`: rotated file count, default `10`
614 - `T3CODE_TRACE_BATCH_WINDOW_MS`: flush window, default `1000`
615 - `T3CODE_TRACE_MIN_LEVEL`: minimum trace level, default `Info`
616 - `T3CODE_TRACE_TIMING_ENABLED`: enable timing metadata, default `true`
617
618 OTLP export:
619
620 - `T3CODE_OTLP_TRACES_URL`: OTLP trace endpoint
621 - `T3CODE_OTLP_METRICS_URL`: OTLP metric endpoint
622 - `T3CODE_OTLP_LOGS_URL`: OTLP log endpoint
623 - `T3CODE_OTLP_EXPORT_INTERVAL_MS`: export interval, default `10000`
624 - `T3CODE_OTLP_HEADERS`: extra headers for all three exporters, same format as
625 `OTEL_EXPORTER_OTLP_HEADERS`: comma-separated `key=value` pairs with percent-encoded values.
626 - `T3CODE_OTLP_PROTOCOL`: `http/json` (default) or `http/protobuf`
627
628 The server and the desktop app also read the standard
629 `OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT` and generic `OTEL_EXPORTER_OTLP_ENDPOINT` (with
630 `/v1/traces`, `/v1/metrics`, or `/v1/logs` appended), for a collector expecting those instead. A
631 non-blank `T3CODE_OTLP_*_URL` wins over either, and a per-signal endpoint wins over the generic one
632 for its signal. A blank value counts as unset. A signal with an OTEL endpoint takes its headers from
633 `OTEL_EXPORTER_OTLP_HEADERS` and its protocol from `OTEL_EXPORTER_OTLP_PROTOCOL` (default
634 `http/protobuf`, read case-insensitively), and a per-signal
635 `OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_HEADERS` or `_PROTOCOL` wins over the generic one for its
636 signal. `T3CODE_OTLP_HEADERS` and `T3CODE_OTLP_PROTOCOL` never apply to it. An endpoint that is not
637 an `http` or `https` URL, a protocol other than `http/protobuf` or `http/json` such as `grpc`, or
638 headers that are not `key=value` pairs with percent-encoded values turn that signal's export off
639 with a startup warning, rather than sending it to the Settings endpoint.
640
641 Service names are fixed: `t3code-server` for the backend and `t3code-desktop` for the desktop main
642 process, both in `service.namespace` `t3code`. `OTEL_SERVICE_NAME` and a `service.name` or
643 `service.namespace` in `OTEL_RESOURCE_ATTRIBUTES` are ignored. Tell installations apart with other
644 resource attributes, such as `OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=staging`.
645
646 Each process reports its own build, not the one it talks to: the server, the desktop main process,
647 the UI, and the mobile app each carry `t3code.release.channel` (`dev`, `stable`, `nightly`, or
648 `preview`, from their own version; desktop PR builds count as `preview`, and a server not built by a
649 release workflow reports `dev`) and `deployment.environment.name` (`development`, `production`, or
650 `staging` for `preview`). On the server and the desktop main process either key in
651 `OTEL_RESOURCE_ATTRIBUTES` wins. The UI and mobile app never read that variable: Vercel PR previews
652 set `VITE_RELEASE_CHANNEL=preview` because their `APP_VERSION` must stay comparable with the
653 server's, and the mobile app maps its build variant (`development`, `preview`, `production`), or
654 reports `dev` from a development bundle.
655
656 Attributes specific to T3 Code use the `t3code.` prefix. The UI runs as `t3code-web` in both a
657 browser and the desktop window, since it is the same code, and `t3code.client.surface` is `desktop`
658 or `web`. The server's `t3code.server.managed_by` is `desktop` when the desktop app launched it and
659 `standalone` otherwise. The desktop main process reports `t3code.desktop.mode` as `development` or
660 `packaged` (formerly `service.mode`; dashboards filtering on that key must switch). Node processes
661 also report the semantic `process.runtime.name`, `process.runtime.version`, and
662 `process.runtime.description`, and UI traces report `browser.language`, `user_agent.original`, and,
663 in Chromium, `browser.platform`, `browser.mobile`, and `browser.brands`.
664
665 Every resource also carries a `service.instance.id`: random per process for the server and desktop,
666 per page load for the UI, and the Expo session on mobile. A `service.instance.id` in
667 `OTEL_RESOURCE_ATTRIBUTES` replaces the generated one on the server and desktop.
668
669 If the OTLP URLs are unset, local tracing still works, metrics stay in-process only, and logs stay
670 on stdout only.
671
672 ### The Kill Switch
673
674 `T3CODE_OTEL_SDK_DISABLED` and `OTEL_SDK_DISABLED` turn off every OTLP export in both the server and
675 the desktop main process, overriding any endpoint from the environment or Settings. Local trace
676 files and stdout logs are unaffected.
677
678 `T3CODE_OTEL_SDK_DISABLED` wins when set, so `T3CODE_OTEL_SDK_DISABLED=false` re-enables export on a
679 machine that sets `OTEL_SDK_DISABLED` for everything else. It accepts the usual boolean spellings
680 (`true`/`false`, `yes`/`no`, `on`/`off`, `1`/`0`, `y`/`n`). `OTEL_SDK_DISABLED` follows the
681 OpenTelemetry specification and only `true` disables export, so `OTEL_SDK_DISABLED=1` does not.
682 Values are case-insensitive and trimmed. An unrecognized value is ignored with a startup warning.
683
684 `OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, or `OTEL_LOGS_EXPORTER` set to `none` turns off
685 just that signal, overriding an OTEL endpoint and the Settings endpoint. A `T3CODE_OTLP_*_URL` still
686 wins for its signal. `otlp` is the default, and any other exporter name, such as `console` or
687 `prometheus`, is ignored with a startup warning.
688
689 ### What Is Instrumented Today
690
691 Current high-value span and metric boundaries include:
692
693 - WebSocket RPC request spans (`ws.rpc.<method>`) and metrics in
694 `apps/server/src/observability/RpcInstrumentation.ts`
695 - startup phases
696 - orchestration command processing
697 - provider session and turn operations
698 - git command execution and git hook events
699 - terminal session lifecycle
700 - sqlite query execution, except the statements inside `orchestrationV2.EventSink.write`, which
701 its `sql.transaction` span times as one
702 - event loop stalls (`server.eventLoop.stall`)
703
704 ### Current Constraints
705
706 - logs outside spans are not persisted in the trace file; SSH-managed launch stdout/stderr is still
707 captured in its launcher log
708 - metrics are not snapshotted locally
709
710 ## Heap Snapshots
711
712 To see what a long-running server holds in memory, send it `SIGUSR2`. The server writes a V8 heap
713 snapshot to its logs dir and logs the path. This works for desktop, `npx t3`, and service installs
714 on macOS and Linux. Windows has no `SIGUSR2`.
715
716 Send the signal to the server pid in `server-runtime.json`, which sits in the server's state dir
717 next to the `logs` dir. For a dev server or a `--home-dir` launch, use that server's state dir from
718 [Traces](#traces). Do not send it to the desktop app or the service launcher: a process without the
719 handler exits on `SIGUSR2`. After a crash the file can keep a stale pid that now belongs to a
720 different process, so check the pid first.
721
722 ```bash
723 pid="$(jq .pid "${T3CODE_HOME:-$HOME/.t3}/userdata/server-runtime.json")"
724 ps -p "$pid" -o command=
725 ```
726
727 If `ps` shows the T3 Code server, send the signal:
728
729 ```bash
730 kill -USR2 "$pid"
731 ```
732
733 The file is `<logsDir>/server-<pid>-<timestamp>.heapsnapshot`, next to `server.trace.ndjson`. To
734 open it, use the Memory tab in Chrome DevTools and select Load.
735
736 Before you take one:
737
738 - The server stops while it writes the file. For a large heap this can take a minute or more.
739 Connected clients can reconnect during the pause, and an event loop monitor, if the server has
740 one, records the pause as a stall. Send the signal once. A second signal sent during a write
741 takes another snapshot after the first one finishes.
742 - The write needs about as much free memory as the heap uses. On a machine that is already
743 swapping, it can make the problem worse or crash the server.
744 - The file contains everything in server memory, including tokens, secrets, and thread content. Do
745 not share it publicly. Delete it when you are done, because storage cleanup does not remove it.