The best way to control Claude Code, Codex and more from any computer or phone

main
md 91 lines 5.53 KB
Rendered Raw
1 # Voice input
2
3 Transcription edits a composer draft. It does not submit an agent turn. Audio is
4 temporary client input, and only normal message submission sends the resulting
5 text to the agent. Mobile defaults to on-device transcription. Sending audio off
6 the device requires an explicit grant naming one environment and, for cloud
7 transcription, one provider.
8
9 The [shared controller](../../packages/client-runtime/src/voice-input/controller.ts)
10 owns the operation while the client supplies capture and transcription. The
11 transcriber is bound to the recording when it starts. Capture can begin while
12 the locale and destination are still being resolved. Draft ownership,
13 text, and revision are captured before recording and checked before insertion, so
14 a late transcript cannot overwrite a draft that was edited or replaced. A
15 transcriber may expose a streaming session, in which case audio leaves during
16 capture and `finish` replaces the file-based `transcribe`.
17
18 ## Where audio may go
19
20 On mobile, on-device transcription needs no audio grant. Environment and cloud
21 transcription share a recorder and upload only finished recordings. The client
22 keeps the grant in device-bound secure storage, outside preference backups, and
23 rechecks it after native decoding and before upload.
24
25 The macOS desktop app adds a `device` destination, Apple Speech. It exists only in
26 the web grant schema, not in `VoiceTranscriptionDestination`, because no environment
27 or other client can act on it. The desktop main process runs a
28 [Swift helper](../../native/apple-speech/AppleSpeech.swift) that the macOS release
29 runners compile; the Speech framework exists only in the macOS SDK.
30
31 The [web grant](../../apps/web/src/speech/voiceConsent.ts) lives outside the preferences snapshot so another
32 window saving unrelated settings cannot resurrect a revoked grant. Each new choice
33 gets a distinct grant, even if the destination was previously selected. The
34 [web hook](../../apps/web/src/speech/useEnvironmentSpeechInput.ts) binds the
35 controller to that grant and the selected environment's connection. It requires
36 `voiceTranscription` to use the environment's model or `cloudVoiceTranscription`
37 to forward audio to the selected provider. Changing the grant or losing the
38 connection, permission, or required capability discards both the active recording
39 and any recording retained for retry. It never chooses a fallback destination.
40 The settings page's transcription test uses the same hook. The microphone loopback
41 test captures separately for local playback and sends nothing. The transport
42 refuses redirects so credentials and audio cannot follow one to another host.
43
44 Transcript cleanup requires its own grant, off by default.
45 It sends text, never audio, to the text generation model of the draft's
46 own environment, because that is where its credentials live.
47
48 A cloud destination binds both the forwarding environment and the named provider.
49 It uses a separate, batch-only endpoint: changing a server key cannot upgrade
50 an environment-local grant, and a cloud recording never uses the streaming
51 socket or falls back to another provider or the environment's native model.
52
53 OpenRouter routes to many upstreams, so its grant also names the exact model and
54 the model travels with every upload as a query parameter. The server accepts only
55 ids present in OpenRouter's public ZDR-eligible transcription catalog,
56 which it caches for ten minutes after a successful read;
57 router aliases such as `openrouter/auto` or `:nitro` fail schema validation before
58 that. Requests set `provider.zdr` and carry no `models` fallback list. OpenRouter
59 applies the account's safety-retention policy when choosing an endpoint. The catalog
60 holds no credential or consent state, which is why it may be cached while key
61 presence is read fresh on every request.
62
63 ## Server side
64
65 The [speech service](../../apps/server/src/speech/SpeechService.ts) runs
66 transcribe-cpp in a child process so a hung native call can be killed without
67 taking the server down; see [native.ts](../../apps/server/src/speech/native.ts)
68 for the packaging constraints (Node SEA, Electron ASAR). One native request runs
69 at a time. Batch uploads carry their options in the body framing defined in
70 [shared/speech.ts](../../packages/shared/src/speech.ts); streaming sends them in
71 the socket's first command. Model files and the selected model, hardware
72 placement, and unload timeout are environment state in server settings.
73 A model file from another app is installed as a symlink only after its SHA-256
74 matches a catalog entry, so the store never loads a file the catalog does not pin
75 and removing a model never deletes a file T3 Code did not download.
76
77 The Silero silence detector is optional. If its download fails, recordings use
78 raw audio during the retry cooldown so a detector outage cannot block dictation.
79 Explicitly installing a speech model also retries the detector without waiting.
80
81 ## Cancellation and resource ownership
82
83 Cancellation invalidates a result immediately, but resources stay owned until the
84 underlying work settles. Apple's native transcription call cannot be interrupted
85 once started. Releasing the session or deleting its recording when the abort signal
86 fires would race that work. The [transcription contract](../../packages/client-runtime/src/voice-input/transcription.ts)
87 therefore requires implementations to settle only after their work has stopped;
88 the [Apple binding](../../apps/mobile/src/native/voiceTranscription.ios.ts) checks
89 cancellation between native calls and discards late results. On the server,
90 interrupting a batch request kills the worker process instead; the next request
91 reloads the model.