The best way to control Claude Code, Codex and more from any computer or phone
Mirrored from https://github.com/Camo67/t3code.git
Synced Nov 21, 03:37 UTC
| 1 | # Server updates |
| 2 | |
| 3 | A [stable launcher](../../apps/server/src/serviceLauncher.ts) owns the runtime |
| 4 | selected by systemd, launchd, or Task Scheduler. It is the only runtime writer of durable service |
| 5 | state. Server children request updates over inherited IPC; they never rewrite |
| 6 | their service definition or select their own replacement. Local service commands |
| 7 | may replace the launcher and state, optionally deferring the restart. Foreground |
| 8 | CLI processes do not self-update. |
| 9 | |
| 10 | Both writers serialize state and unit writes through |
| 11 | [file leases](../../apps/server/src/cloud/serviceSelectionLock.ts). The installer |
| 12 | takes the state lease before the unit lease; the launcher never holds both, so a |
| 13 | unit failure cannot block a durable commit. Manager stop/start commands run outside |
| 14 | both leases, since stopping a launcher that waits on a lease would deadlock. A |
| 15 | running launcher refuses further updates once a local command changes its selection. |
| 16 | A protocol-3 launcher takes no lease, so a local command can still race its updates. |
| 17 | |
| 18 | Exact-version installs keep restarts independent of npm cache eviction or a moving |
| 19 | release tag. Installation and preflight happen in staging before publishing an |
| 20 | immutable runtime. Preflight checks the launcher protocol because a target that |
| 21 | needs new rollback guarantees cannot safely run under an older launcher. A remote |
| 22 | update replaces only the child, so upgrading the launcher requires a local service |
| 23 | update with a restart. Every installed launcher is therefore one protocol behind |
| 24 | after a bump, and a target that refuses the previous protocol stops remote updates |
| 25 | for every existing service. Refuse it only when the child cannot work under it. |
| 26 | Protocol 4 changed the launcher alone, so its runtime still runs under protocol 3. |
| 27 | The server names its launcher's protocol to preflight, not its own, since the two differ |
| 28 | after the first such update. |
| 29 | |
| 30 | A client can stage the target early with `stageOnly`, which installs and preflights the |
| 31 | runtime and restarts nothing. "Update everything" does this while provider installs run, |
| 32 | because a restart would kill them. A server without the `serverUpdateStaging` capability |
| 33 | ignores the flag and runs the whole update, so send it only to a server that advertises it. |
| 34 | |
| 35 | After a commit or rollback the launcher points the unit at the selected runtime and |
| 36 | reloads the manager, so the next service start runs that version's launcher; the |
| 37 | protocol check is what makes that safe. Each start after a finished update retries this, because |
| 38 | a previous reload may have failed, and leaves a unit naming a newer runtime or another |
| 39 | T3 home alone. A protocol-3 launcher does none of this: its unit keeps naming the |
| 40 | runtime that installed it. Until a deferred local upgrade restarts, systemd (before |
| 41 | daemon-reload) and launchd (before bootstrap) can respawn the old launcher, which |
| 42 | rejects the state the new installer wrote. |
| 43 | |
| 44 | Task Scheduler tracks only the console host it started, not the launcher. `schtasks /End` |
| 45 | kills that host and nothing else; the launcher takes its closed console as SIGTERM. The task |
| 46 | reads as stopped at once, and `/Run` then starts a second launcher beside the first, so a |
| 47 | Windows launcher holds a lock file for as long as it runs. A replacement waits for that lock, |
| 48 | and the CLI treats its release as the stop having finished. |
| 49 | |
| 50 | Task Scheduler also runs its own copy of the task, and registering one enables it. So every |
| 51 | write of the task file registers it, including an install that defers its restart. The CLI |
| 52 | disables the task under the unit lease, and a registration under that lease keeps a disabled |
| 53 | task disabled, so the repeating trigger cannot restart a service the CLI is holding stopped. |
| 54 | |
| 55 | ## Commit boundary |
| 56 | |
| 57 | The launcher durably records the pending update before acknowledging it, then |
| 58 | stops the old child and starts the target as a trial. Service-state writes use |
| 59 | same-directory replacement with file and directory fsync. Invalid state stops |
| 60 | startup rather than guessing which runtime to boot. |
| 61 | |
| 62 | The trial must finish migrations, acquire dependencies, bind HTTP, and park every |
| 63 | long-running root at the activation gate before reporting `prepared`. The launcher |
| 64 | then commits the target version durably and replies `committed`. Only then may the |
| 65 | child release its gates, accept commands, and publish ready. Keep fallible startup |
| 66 | acquisitions before this boundary. A listener alone does not prove the runtime is |
| 67 | ready to commit. |
| 68 | |
| 69 | A failed or timed-out trial returns to the old version. After commit, the target |
| 70 | is authoritative and the service manager's ordinary restart policy applies. |
| 71 | |
| 72 | The prepared deadline is long because migrations can scale with the database |
| 73 | and block the trial's main thread, so it cannot report progress. A missed |
| 74 | deadline restores the snapshot, and the retry pays the same cost again. A |
| 75 | crashed trial is caught when it exits; the deadline bounds only a stuck one. |
| 76 | Client update waits must outlast the deadline, or a slow update shows as failed. |
| 77 | |
| 78 | ## Database rollback |
| 79 | |
| 80 | Before stopping the old child, the launcher refuses an update whose snapshot will |
| 81 | not fit on disk, so a full disk does not cause an outage. After the old child |
| 82 | exits, the launcher snapshots SQLite's main file, WAL, and shared-memory file. This makes trial migrations reversible without down |
| 83 | migrations. The snapshot is made once per update and survives launcher restarts; |
| 84 | replacing it during a retry could capture changes from the failed trial. |
| 85 | |
| 86 | Rollback stops the trial before restoring. A durable restore marker makes an |
| 87 | interrupted restore finish before either version boots. Keep the snapshot until |
| 88 | commit, or until both restoration and the terminal rollback state are durable. |
| 89 | Attachments and other files outside SQLite are outside this rollback boundary. |
| 90 | |
| 91 | ## Client acknowledgement |
| 92 | |
| 93 | An accepted update is still pending. Clients correlate the launcher's update ID |
| 94 | with the ready event after reconnecting, then check the outcome and target version. |
| 95 | A reconnect alone cannot distinguish successful replacement from rollback. Older |
| 96 | servers without an update ID retain version-only correlation. |
| 97 | |
| 98 | Desktop updates have a separate two-phase handoff because installing the app stops |
| 99 | its bundled backend. Preparation returns a token while the connection is alive; |
| 100 | the client commits that token only after receiving it. Otherwise backend shutdown |
| 101 | could lose the only successful RPC result. The client must then observe the |
| 102 | prepared version after reconnecting. If installation fails, desktop restarts the |
| 103 | stopped backends and replays the failure for the same token. |
| 104 | |
| 105 | ## Recovering interrupted threads |
| 106 | |
| 107 | Restart continuation is an environment-owned preference, on by default. A |
| 108 | `settings.json` that fails to decode turns it off, since the file may hold an |
| 109 | opt-out. The |
| 110 | [v2 recovery service](../../apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts) |
| 111 | requires matching durable run, provider thread, session, and native resume identity. |
| 112 | Queued runs never started, so recovery holds them and continues the run they wait |
| 113 | behind. A settled run qualifies only when the restart cancels work the agent waits on |
| 114 | (a subagent, a background task, or a shell or monitor the agent marked with |
| 115 | `t3_background_wait`) and its session was live, so stale records from an older crash |
| 116 | never wake it. Its continuation prompt is the note of what will not report back. An |
| 117 | unmarked shell or monitor does not wake a settled run: most are dev servers and their |
| 118 | watchdogs that nobody waits on, so that note waits for the next user message. |
| 119 | |
| 120 | A run is marked running before its provider accepts the prompt, so recovery treats a run |
| 121 | without a provider turn as never started. A user's prompt the provider never received |
| 122 | joins the held queue. With restart continuation on, a delegated, agent-sent, or scheduled |
| 123 | prompt has nobody to review it, so recovery starts it again under the same run. The resend |
| 124 | skips every thread and Stop that a continuation skips, and gives up after three tries so a |
| 125 | start that crashes the server cannot loop. A crash after the provider received the prompt |
| 126 | but before its turn was recorded can repeat that prompt. |
| 127 | |
| 128 | Recovery retires effects tied to the lost process and records continuation intent |
| 129 | in the durable outbox. That intent survives another restart before provider startup. |
| 130 | Continuation effects wait for activation; a slow provider must not delay the server's |
| 131 | readiness or the launcher's commit boundary. Graceful shutdown captures intent before |
| 132 | closing providers, then reconciles after ingestion has stopped so a late completion |
| 133 | cannot be overwritten by a stale cancellation. |
| 134 | |
| 135 | The [continuation handler](../../apps/server/src/orchestration-v2/RestartContinuation.ts) |
| 136 | rechecks the preference, provider selection, newer user work, a stop the user requested, |
| 137 | and maintenance turns such as `/compact` before dispatching. Intent and delivery both |
| 138 | refuse a thread that is archived, settled, or still snoozed, a cut turn whose provider |
| 139 | turn already ended, and a run waiting on the user: an approval, question, or secret that was pending, |
| 140 | or that recovery closed on the cut turn. Message-capable questions stay answerable, so |
| 141 | they do not block. New continuation intents persist a cumulative process-loss count; |
| 142 | delivery stops before the fourth crash recovery of an unfinished turn, regardless |
| 143 | of elapsed time. Planned shutdowns and completed turns reset it. A live continuation |
| 144 | retains its preceding intent past outbox retention so cleanup cannot erase the count. |
| 145 | Legacy intents without a count keep the five-minute chain guard. Stable |
| 146 | command and message IDs prevent duplicate submissions after an outbox retry. Codex |
| 147 | resumes without adding provider prompt text unless the turn lost background work; |
| 148 | other adapters receive the continuation message through their normal turn path. |
| 149 | |
| 150 | Delegated tasks (`delegate_task` child threads) are reconciled as their own threads, |
| 151 | never as the parent's background work. The orchestrator settles child results and |
| 152 | completion deliveries in a startup pass after reconciliation, because the terminal-run |
| 153 | listener ignores reconciliation's cancellations. A cancelled child whose restart |
| 154 | continuation is still pending in the outbox is not a result yet; the continuation's run |
| 155 | settles it, or the handler settles it when it declines to continue. Schedulers wait for |
| 156 | activation so they cannot start runs that reconciliation would then cancel. |