@devries-cameron20 / t3code Mirror · read-only
Fork 0

The best way to control Claude Code, Codex and more from any computer or phone

Mirrored from https://github.com/Camo67/t3code.git

Synced Nov 21, 03:37 UTC

main
md 156 lines 10.1 KB
Rendered Raw
1 # Server updates
2
3 A [stable launcher](../../apps/server/src/serviceLauncher.ts) owns the runtime
4 selected by systemd, launchd, or Task Scheduler. It is the only runtime writer of durable service
5 state. Server children request updates over inherited IPC; they never rewrite
6 their service definition or select their own replacement. Local service commands
7 may replace the launcher and state, optionally deferring the restart. Foreground
8 CLI processes do not self-update.
9
10 Both writers serialize state and unit writes through
11 [file leases](../../apps/server/src/cloud/serviceSelectionLock.ts). The installer
12 takes the state lease before the unit lease; the launcher never holds both, so a
13 unit failure cannot block a durable commit. Manager stop/start commands run outside
14 both leases, since stopping a launcher that waits on a lease would deadlock. A
15 running launcher refuses further updates once a local command changes its selection.
16 A protocol-3 launcher takes no lease, so a local command can still race its updates.
17
18 Exact-version installs keep restarts independent of npm cache eviction or a moving
19 release tag. Installation and preflight happen in staging before publishing an
20 immutable runtime. Preflight checks the launcher protocol because a target that
21 needs new rollback guarantees cannot safely run under an older launcher. A remote
22 update replaces only the child, so upgrading the launcher requires a local service
23 update with a restart. Every installed launcher is therefore one protocol behind
24 after a bump, and a target that refuses the previous protocol stops remote updates
25 for every existing service. Refuse it only when the child cannot work under it.
26 Protocol 4 changed the launcher alone, so its runtime still runs under protocol 3.
27 The server names its launcher's protocol to preflight, not its own, since the two differ
28 after the first such update.
29
30 A client can stage the target early with `stageOnly`, which installs and preflights the
31 runtime and restarts nothing. "Update everything" does this while provider installs run,
32 because a restart would kill them. A server without the `serverUpdateStaging` capability
33 ignores the flag and runs the whole update, so send it only to a server that advertises it.
34
35 After a commit or rollback the launcher points the unit at the selected runtime and
36 reloads the manager, so the next service start runs that version's launcher; the
37 protocol check is what makes that safe. Each start after a finished update retries this, because
38 a previous reload may have failed, and leaves a unit naming a newer runtime or another
39 T3 home alone. A protocol-3 launcher does none of this: its unit keeps naming the
40 runtime that installed it. Until a deferred local upgrade restarts, systemd (before
41 daemon-reload) and launchd (before bootstrap) can respawn the old launcher, which
42 rejects the state the new installer wrote.
43
44 Task Scheduler tracks only the console host it started, not the launcher. `schtasks /End`
45 kills that host and nothing else; the launcher takes its closed console as SIGTERM. The task
46 reads as stopped at once, and `/Run` then starts a second launcher beside the first, so a
47 Windows launcher holds a lock file for as long as it runs. A replacement waits for that lock,
48 and the CLI treats its release as the stop having finished.
49
50 Task Scheduler also runs its own copy of the task, and registering one enables it. So every
51 write of the task file registers it, including an install that defers its restart. The CLI
52 disables the task under the unit lease, and a registration under that lease keeps a disabled
53 task disabled, so the repeating trigger cannot restart a service the CLI is holding stopped.
54
55 ## Commit boundary
56
57 The launcher durably records the pending update before acknowledging it, then
58 stops the old child and starts the target as a trial. Service-state writes use
59 same-directory replacement with file and directory fsync. Invalid state stops
60 startup rather than guessing which runtime to boot.
61
62 The trial must finish migrations, acquire dependencies, bind HTTP, and park every
63 long-running root at the activation gate before reporting `prepared`. The launcher
64 then commits the target version durably and replies `committed`. Only then may the
65 child release its gates, accept commands, and publish ready. Keep fallible startup
66 acquisitions before this boundary. A listener alone does not prove the runtime is
67 ready to commit.
68
69 A failed or timed-out trial returns to the old version. After commit, the target
70 is authoritative and the service manager's ordinary restart policy applies.
71
72 The prepared deadline is long because migrations can scale with the database
73 and block the trial's main thread, so it cannot report progress. A missed
74 deadline restores the snapshot, and the retry pays the same cost again. A
75 crashed trial is caught when it exits; the deadline bounds only a stuck one.
76 Client update waits must outlast the deadline, or a slow update shows as failed.
77
78 ## Database rollback
79
80 Before stopping the old child, the launcher refuses an update whose snapshot will
81 not fit on disk, so a full disk does not cause an outage. After the old child
82 exits, the launcher snapshots SQLite's main file, WAL, and shared-memory file. This makes trial migrations reversible without down
83 migrations. The snapshot is made once per update and survives launcher restarts;
84 replacing it during a retry could capture changes from the failed trial.
85
86 Rollback stops the trial before restoring. A durable restore marker makes an
87 interrupted restore finish before either version boots. Keep the snapshot until
88 commit, or until both restoration and the terminal rollback state are durable.
89 Attachments and other files outside SQLite are outside this rollback boundary.
90
91 ## Client acknowledgement
92
93 An accepted update is still pending. Clients correlate the launcher's update ID
94 with the ready event after reconnecting, then check the outcome and target version.
95 A reconnect alone cannot distinguish successful replacement from rollback. Older
96 servers without an update ID retain version-only correlation.
97
98 Desktop updates have a separate two-phase handoff because installing the app stops
99 its bundled backend. Preparation returns a token while the connection is alive;
100 the client commits that token only after receiving it. Otherwise backend shutdown
101 could lose the only successful RPC result. The client must then observe the
102 prepared version after reconnecting. If installation fails, desktop restarts the
103 stopped backends and replays the failure for the same token.
104
105 ## Recovering interrupted threads
106
107 Restart continuation is an environment-owned preference, on by default. A
108 `settings.json` that fails to decode turns it off, since the file may hold an
109 opt-out. The
110 [v2 recovery service](../../apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts)
111 requires matching durable run, provider thread, session, and native resume identity.
112 Queued runs never started, so recovery holds them and continues the run they wait
113 behind. A settled run qualifies only when the restart cancels work the agent waits on
114 (a subagent, a background task, or a shell or monitor the agent marked with
115 `t3_background_wait`) and its session was live, so stale records from an older crash
116 never wake it. Its continuation prompt is the note of what will not report back. An
117 unmarked shell or monitor does not wake a settled run: most are dev servers and their
118 watchdogs that nobody waits on, so that note waits for the next user message.
119
120 A run is marked running before its provider accepts the prompt, so recovery treats a run
121 without a provider turn as never started. A user's prompt the provider never received
122 joins the held queue. With restart continuation on, a delegated, agent-sent, or scheduled
123 prompt has nobody to review it, so recovery starts it again under the same run. The resend
124 skips every thread and Stop that a continuation skips, and gives up after three tries so a
125 start that crashes the server cannot loop. A crash after the provider received the prompt
126 but before its turn was recorded can repeat that prompt.
127
128 Recovery retires effects tied to the lost process and records continuation intent
129 in the durable outbox. That intent survives another restart before provider startup.
130 Continuation effects wait for activation; a slow provider must not delay the server's
131 readiness or the launcher's commit boundary. Graceful shutdown captures intent before
132 closing providers, then reconciles after ingestion has stopped so a late completion
133 cannot be overwritten by a stale cancellation.
134
135 The [continuation handler](../../apps/server/src/orchestration-v2/RestartContinuation.ts)
136 rechecks the preference, provider selection, newer user work, a stop the user requested,
137 and maintenance turns such as `/compact` before dispatching. Intent and delivery both
138 refuse a thread that is archived, settled, or still snoozed, a cut turn whose provider
139 turn already ended, and a run waiting on the user: an approval, question, or secret that was pending,
140 or that recovery closed on the cut turn. Message-capable questions stay answerable, so
141 they do not block. New continuation intents persist a cumulative process-loss count;
142 delivery stops before the fourth crash recovery of an unfinished turn, regardless
143 of elapsed time. Planned shutdowns and completed turns reset it. A live continuation
144 retains its preceding intent past outbox retention so cleanup cannot erase the count.
145 Legacy intents without a count keep the five-minute chain guard. Stable
146 command and message IDs prevent duplicate submissions after an outbox retry. Codex
147 resumes without adding provider prompt text unless the turn lost background work;
148 other adapters receive the continuation message through their normal turn path.
149
150 Delegated tasks (`delegate_task` child threads) are reconciled as their own threads,
151 never as the parent's background work. The orchestrator settles child results and
152 completion deliveries in a startup pass after reconciliation, because the terminal-run
153 listener ignores reconciliation's cancellations. A cancelled child whose restart
154 continuation is still pending in the outbox is not a result yet; the continuation's run
155 settles it, or the handler settles it when it declines to continue. Schedulers wait for
156 activation so they cannot start runs that reconciliation would then cancel.