A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Docs] Fix formatting of transcription doc (#24676)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Sep 11, 2025 at 19:18 UTC
361ae27f8a336b82efb932458e3e79e06027d4ce
1 file changed
+163
-159
docs/contributing/model/transcription.md
+163
-159
@@ -3,149 +3,151 @@
3
This document walks you through the steps to add support for speech-to-text (ASR) models to vLLM’s transcription and translation APIs by implementing [SupportsTranscription][vllm.model_executor.models.interfaces.SupportsTranscription].
4
Please refer to the [supported models](../../models/supported_models.md#transcription) for further guidance.
5
6
-## 1. Update the base vLLM model
6
+## Update the base vLLM model
7
8
It is assumed you have already implemented your model in vLLM according to the basic model guide. Extend your model with the [SupportsTranscription][vllm.model_executor.models.interfaces.SupportsTranscription] interface and implement the following class attributes and methods.
9
10
-- Declare supported languages and capabilities:
10
+### `supported_languages` and `supports_transcription_only`
11
12
- ??? code
12
+Declare supported languages and capabilities:
13
14
- ```python
15
- from typing import ClassVar, Mapping, Optional, Literal
16
- import numpy as np
17
- import torch
18
- from torch import nn
19
-
20
- from vllm.config import ModelConfig, SpeechToTextConfig
21
- from vllm.inputs.data import PromptType
22
- from vllm.model_executor.models.interfaces import SupportsTranscription
14
+- The `supported_languages` mapping is validated at init time.
15
+- Set `supports_transcription_only=True` if the model should not serve text generation (eg Whisper).
16
+
17
+??? code "supported_languages and supports_transcription_only"
18
+ ```python
19
+ from typing import ClassVar, Mapping, Optional, Literal
20
+ import numpy as np
21
+ import torch
22
+ from torch import nn
23
+
24
+ from vllm.config import ModelConfig, SpeechToTextConfig
25
+ from vllm.inputs.data import PromptType
26
+ from vllm.model_executor.models.interfaces import SupportsTranscription
27
+
28
+ class YourASRModel(nn.Module, SupportsTranscription):
29
+ # Map of ISO 639-1 language codes to language names
30
+ supported_languages: ClassVar[Mapping[str, str]] = {
31
+ "en": "English",
32
+ "it": "Italian",
33
+ # ... add more as needed
34
+ }
35
24
- class YourASRModel(nn.Module, SupportsTranscription):
25
- # Map of ISO 639-1 language codes to language names
26
- supported_languages: ClassVar[Mapping[str, str]] = {
27
- "en": "English",
28
- "it": "Italian",
29
- # ... add more as needed
36
+ # If your model only supports audio-conditioned generation
37
+ # (no text-only generation), enable this flag.
38
+ supports_transcription_only: ClassVar[bool] = True
39
+ ```
40
+
41
+Provide an ASR configuration via [get_speech_to_text_config][vllm.model_executor.models.interfaces.SupportsTranscription.get_speech_to_text_config].
42
+
43
+This is for controlling general behavior of the API when serving your model:
44
+
45
+??? code "get_speech_to_text_config()"
46
+ ```python
47
+ class YourASRModel(nn.Module, SupportsTranscription):
48
+ ...
49
+
50
+ @classmethod
51
+ def get_speech_to_text_config(
52
+ cls,
53
+ model_config: ModelConfig,
54
+ task_type: Literal["transcribe", "translate"],
55
+ ) -> SpeechToTextConfig:
56
+ return SpeechToTextConfig(
57
+ sample_rate=16_000,
58
+ max_audio_clip_s=30,
59
+ # Set to None to disable server-side chunking if your
60
+ # model/processor handles it already
61
+ min_energy_split_window_size=None,
62
+ )
63
+ ```
64
+
65
+See [Audio preprocessing and chunking](#audio-preprocessing-and-chunking) for what each field controls.
66
+
67
+Implement the prompt construction via [get_generation_prompt][vllm.model_executor.models.interfaces.SupportsTranscription.get_generation_prompt]. The server passes you the resampled waveform and task parameters; you return a valid [PromptType][vllm.inputs.data.PromptType]. There are two common patterns:
68
+
69
+#### Multimodal LLM with audio embeddings (e.g., Voxtral, Gemma3n)
70
+
71
+Return a dict containing `multi_modal_data` with the audio, and either a `prompt` string or `prompt_token_ids`:
72
+
73
+??? code "get_generation_prompt()"
74
+ ```python
75
+ class YourASRModel(nn.Module, SupportsTranscription):
76
+ ...
77
+
78
+ @classmethod
79
+ def get_generation_prompt(
80
+ cls,
81
+ audio: np.ndarray,
82
+ stt_config: SpeechToTextConfig,
83
+ model_config: ModelConfig,
84
+ language: Optional[str],
85
+ task_type: Literal["transcribe", "translate"],
86
+ request_prompt: str,
87
+ to_language: Optional[str],
88
+ ) -> PromptType:
89
+ # Example with a free-form instruction prompt
90
+ task_word = "Transcribe" if task_type == "transcribe" else "Translate"
91
+ prompt = (
92
+ "<start_of_turn>user\n"
93
+ f"{task_word} this audio: <audio_soft_token>"
94
+ "<end_of_turn>\n<start_of_turn>model\n"
95
+ )
96
+
97
+ return {
98
+ "multi_modal_data": {"audio": (audio, stt_config.sample_rate)},
99
+ "prompt": prompt,
100
}
31
-
32
- # If your model only supports audio-conditioned generation
33
- # (no text-only generation), enable this flag.
34
- supports_transcription_only: ClassVar[bool] = True
35
- ```
36
-
37
- - The `supported_languages` mapping is validated at init time.
38
- - Set `supports_transcription_only=True` if the model should not serve text generation (eg Whisper).
39
-
40
-- Provide an ASR configuration via [get_speech_to_text_config][vllm.model_executor.models.interfaces.SupportsTranscription.get_speech_to_text_config].
41
- This is for controlling general behavior of the API when serving your model:
42
-
43
- ??? code
44
-
45
- ```python
46
- class YourASRModel(nn.Module, SupportsTranscription):
47
- ...
48
-
49
- @classmethod
50
- def get_speech_to_text_config(
51
- cls,
52
- model_config: ModelConfig,
53
- task_type: Literal["transcribe", "translate"],
54
- ) -> SpeechToTextConfig:
55
- return SpeechToTextConfig(
56
- sample_rate=16_000,
57
- max_audio_clip_s=30,
58
- # Set to None to disable server-side chunking if your
59
- # model/processor handles it already
60
- min_energy_split_window_size=None,
61
- )
62
- ```
63
-
64
- See the “Audio preprocessing and chunking” section for what each field controls.
65
-
66
-- Implement the prompt construction via [get_generation_prompt][vllm.model_executor.models.interfaces.SupportsTranscription.get_generation_prompt]. The server passes you the resampled waveform and task parameters; you return a valid [PromptType][vllm.inputs.data.PromptType]. There are two common patterns:
67
-
68
-### A. Multimodal LLM with audio embeddings (e.g., Voxtral, Gemma3n)
69
-
70
- Return a dict containing `multi_modal_data` with the audio, and either a `prompt` string or `prompt_token_ids`:
71
-
72
- ??? code
73
-
74
- ```python
75
- class YourASRModel(nn.Module, SupportsTranscription):
76
- ...
77
-
78
- @classmethod
79
- def get_generation_prompt(
80
- cls,
81
- audio: np.ndarray,
82
- stt_config: SpeechToTextConfig,
83
- model_config: ModelConfig,
84
- language: Optional[str],
85
- task_type: Literal["transcribe", "translate"],
86
- request_prompt: str,
87
- to_language: Optional[str],
88
- ) -> PromptType:
89
- # Example with a free-form instruction prompt
90
- task_word = "Transcribe" if task_type == "transcribe" else "Translate"
91
- prompt = (
92
- "<start_of_turn>user\n"
93
- f"{task_word} this audio: <audio_soft_token>"
94
- "<end_of_turn>\n<start_of_turn>model\n"
95
- )
96
-
97
- return {
98
- "multi_modal_data": {"audio": (audio, stt_config.sample_rate)},
99
- "prompt": prompt,
100
- }
101
- ```
102
-
103
- For further clarification on multi modal inputs, please refer to [Multi-Modal Inputs](../../features/multimodal_inputs.md).
104
-
105
-### B. Encoder–decoder audio-only (e.g., Whisper)
106
-
107
- Return a dict with separate `encoder_prompt` and `decoder_prompt` entries:
108
-
109
- ??? code
110
-
111
- ```python
112
- class YourASRModel(nn.Module, SupportsTranscription):
113
- ...
114
-
115
- @classmethod
116
- def get_generation_prompt(
117
- cls,
118
- audio: np.ndarray,
119
- stt_config: SpeechToTextConfig,
120
- model_config: ModelConfig,
121
- language: Optional[str],
122
- task_type: Literal["transcribe", "translate"],
123
- request_prompt: str,
124
- to_language: Optional[str],
125
- ) -> PromptType:
126
- if language is None:
127
- raise ValueError("Language must be specified")
128
-
129
- prompt = {
130
- "encoder_prompt": {
131
- "prompt": "",
132
- "multi_modal_data": {
133
- "audio": (audio, stt_config.sample_rate),
134
- },
101
+ ```
102
+
103
+ For further clarification on multi modal inputs, please refer to [Multi-Modal Inputs](../../features/multimodal_inputs.md).
104
+
105
+#### Encoder–decoder audio-only (e.g., Whisper)
106
+
107
+Return a dict with separate `encoder_prompt` and `decoder_prompt` entries:
108
+
109
+??? code "get_generation_prompt()"
110
+ ```python
111
+ class YourASRModel(nn.Module, SupportsTranscription):
112
+ ...
113
+
114
+ @classmethod
115
+ def get_generation_prompt(
116
+ cls,
117
+ audio: np.ndarray,
118
+ stt_config: SpeechToTextConfig,
119
+ model_config: ModelConfig,
120
+ language: Optional[str],
121
+ task_type: Literal["transcribe", "translate"],
122
+ request_prompt: str,
123
+ to_language: Optional[str],
124
+ ) -> PromptType:
125
+ if language is None:
126
+ raise ValueError("Language must be specified")
127
+
128
+ prompt = {
129
+ "encoder_prompt": {
130
+ "prompt": "",
131
+ "multi_modal_data": {
132
+ "audio": (audio, stt_config.sample_rate),
133
},
136
- "decoder_prompt": (
137
- (f"<|prev|>{request_prompt}" if request_prompt else "")
138
- + f"<|startoftranscript|><|{language}|>"
139
- + f"<|{task_type}|><|notimestamps|>"
140
- ),
141
- }
142
- return cast(PromptType, prompt)
143
- ```
134
+ },
135
+ "decoder_prompt": (
136
+ (f"<|prev|>{request_prompt}" if request_prompt else "")
137
+ + f"<|startoftranscript|><|{language}|>"
138
+ + f"<|{task_type}|><|notimestamps|>"
139
+ ),
140
+ }
141
+ return cast(PromptType, prompt)
142
+ ```
143
145
-- (Optional) Language validation via [validate_language][vllm.model_executor.models.interfaces.SupportsTranscription.validate_language]
144
+### `validate_language` (optional)
145
147
- If your model requires a language and you want a default, override this method (see Whisper):
146
+Language validation via [validate_language][vllm.model_executor.models.interfaces.SupportsTranscription.validate_language]
147
148
+If your model requires a language and you want a default, override this method (see Whisper):
149
+
150
+??? code "validate_language()"
151
```python
152
@classmethod
153
def validate_language(cls, language: Optional[str]) -> Optional[str]:
@@ -156,27 +158,29 @@ It is assumed you have already implemented your model in vLLM according to the b
158
return super().validate_language(language)
159
```
160
159
-- (Optional) Token accounting for streaming via [get_num_audio_tokens][vllm.model_executor.models.interfaces.SupportsTranscription.get_num_audio_tokens]
161
+### `get_num_audio_tokens` (optional)
162
+
163
+Token accounting for streaming via [get_num_audio_tokens][vllm.model_executor.models.interfaces.SupportsTranscription.get_num_audio_tokens]
164
161
- Provide a fast duration→token estimate to improve streaming usage statistics:
165
+Provide a fast duration→token estimate to improve streaming usage statistics:
166
163
- ??? code
164
- ```python
165
- class YourASRModel(nn.Module, SupportsTranscription):
166
- ...
167
+??? code "get_num_audio_tokens()"
168
+ ```python
169
+ class YourASRModel(nn.Module, SupportsTranscription):
170
+ ...
171
168
- @classmethod
169
- def get_num_audio_tokens(
170
- cls,
171
- audio_duration_s: float,
172
- stt_config: SpeechToTextConfig,
173
- model_config: ModelConfig,
174
- ) -> Optional[int]:
175
- # Return None if unknown; otherwise return an estimate.
176
- return int(audio_duration_s * stt_config.sample_rate // 320) # example
177
- ```
172
+ @classmethod
173
+ def get_num_audio_tokens(
174
+ cls,
175
+ audio_duration_s: float,
176
+ stt_config: SpeechToTextConfig,
177
+ model_config: ModelConfig,
178
+ ) -> Optional[int]:
179
+ # Return None if unknown; otherwise return an estimate.
180
+ return int(audio_duration_s * stt_config.sample_rate // 320) # example
181
+ ```
182
179
-## 2. Audio preprocessing and chunking
183
+## Audio preprocessing and chunking
184
185
The API server takes care of basic audio I/O and optional chunking before building prompts:
186
@@ -185,7 +189,8 @@ The API server takes care of basic audio I/O and optional chunking before buildi
189
- Energy-aware splitting: When `min_energy_split_window_size` is set, the server finds low-energy regions to minimize cutting within words.
190
191
Relevant server logic:
188
-??? code
192
+
193
+??? code "_preprocess_speech_to_text()"
194
```python
195
# vllm/entrypoints/openai/speech_to_text.py
196
async def _preprocess_speech_to_text(...):
@@ -211,9 +216,9 @@ Relevant server logic:
216
return prompts, duration
217
```
218
214
-## 3. Exposing tasks automatically
219
+## Exposing tasks automatically
220
216
-- vLLM automatically advertises transcription support if your model implements the interface:
221
+vLLM automatically advertises transcription support if your model implements the interface:
222
223
```python
224
if supports_transcription(model):
@@ -222,7 +227,7 @@ if supports_transcription(model):
227
supported_tasks.append("transcription")
228
```
229
225
-- When enabled, the server initializes the transcription and translation handlers:
230
+When enabled, the server initializes the transcription and translation handlers:
231
232
```python
233
state.openai_serving_transcription = OpenAIServingTranscription(...) if "transcription" in supported_tasks else None
@@ -231,13 +236,13 @@ state.openai_serving_translation = OpenAIServingTranslation(...) if "transcripti
236
237
No extra registration is required beyond having your model class available via the model registry and implementing `SupportsTranscription`.
238
234
-## 4. Examples in-tree
239
+## Examples in-tree
240
241
- Whisper encoder–decoder (audio-only): <gh-file:vllm/model_executor/models/whisper.py>
242
- Voxtral decoder-only (audio embeddings + LLM): <gh-file:vllm/model_executor/models/voxtral.py>
243
- Gemma3n decoder-only with fixed instruction prompt: <gh-file:vllm/model_executor/models/gemma3n_mm.py>
244
240
-## 5. Test with the API
245
+## Test with the API
246
247
Once your model implements `SupportsTranscription`, you can test the endpoints (API mimics OpenAI):
248
@@ -266,7 +271,6 @@ Once your model implements `SupportsTranscription`, you can test the endpoints (
271
Or check out more examples in <gh-file:examples/online_serving>.
272
273
!!! note
269
-
270
-- If your model handles chunking internally (e.g., via its processor or encoder), set `min_energy_split_window_size=None` in the returned `SpeechToTextConfig` to disable server-side chunking.
271
-- Implementing `get_num_audio_tokens` improves accuracy of streaming usage metrics (`prompt_tokens`) without an extra forward pass.
272
-- For multilingual behavior, keep `supported_languages` aligned with actual model capabilities.
274
+ - If your model handles chunking internally (e.g., via its processor or encoder), set `min_energy_split_window_size=None` in the returned `SpeechToTextConfig` to disable server-side chunking.
275
+ - Implementing `get_num_audio_tokens` improves accuracy of streaming usage metrics (`prompt_tokens`) without an extra forward pass.
276
+ - For multilingual behavior, keep `supported_languages` aligned with actual model capabilities.