@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 286 lines 11.4 KB
Rendered Raw
1 # Speech-to-Text (Transcription/Translation) Support
2
3 This document walks you through the steps to add support for speech-to-text (ASR) models to vLLM’s transcription and translation APIs by implementing [SupportsTranscription][vllm.model_executor.models.interfaces.SupportsTranscription].
4 Please refer to the [supported models](../../models/supported_models.md#transcription) for further guidance.
5
6 ## Update the base vLLM model
7
8 It is assumed you have already implemented your model in vLLM according to the basic model guide. Extend your model with the [SupportsTranscription][vllm.model_executor.models.interfaces.SupportsTranscription] interface and implement the following class attributes and methods.
9
10 ### `supported_languages` and `supports_transcription_only`
11
12 Declare supported languages and capabilities:
13
14 - The `supported_languages` mapping is validated at init time.
15 - Set `supports_transcription_only=True` if the model should not serve text generation (eg Whisper).
16
17 ??? code "supported_languages and supports_transcription_only"
18
19 ```python
20 from typing import ClassVar, Mapping, Literal
21 import numpy as np
22 import torch
23 from torch import nn
24
25 from vllm.config import ModelConfig, SpeechToTextConfig
26 from vllm.inputs import PromptType
27 from vllm.model_executor.models.interfaces import SupportsTranscription
28
29 class YourASRModel(nn.Module, SupportsTranscription):
30 # Map of ISO 639-1 language codes to language names
31 supported_languages: ClassVar[Mapping[str, str]] = {
32 "en": "English",
33 "it": "Italian",
34 # ... add more as needed
35 }
36
37 # If your model only supports audio-conditioned generation
38 # (no text-only generation), enable this flag.
39 supports_transcription_only: ClassVar[bool] = True
40 ```
41
42 Provide an ASR configuration via [get_speech_to_text_config][vllm.model_executor.models.interfaces.SupportsTranscription.get_speech_to_text_config].
43
44 This is for controlling general behavior of the API when serving your model:
45
46 ??? code "get_speech_to_text_config()"
47
48 ```python
49 class YourASRModel(nn.Module, SupportsTranscription):
50 ...
51
52 @classmethod
53 def get_speech_to_text_config(
54 cls,
55 model_config: ModelConfig,
56 task_type: Literal["transcribe", "translate"],
57 ) -> SpeechToTextConfig:
58 return SpeechToTextConfig(
59 sample_rate=16_000,
60 max_audio_clip_s=30,
61 # Set to None to disable server-side chunking if your
62 # model/processor handles it already
63 min_energy_split_window_size=None,
64 )
65 ```
66
67 See [Audio preprocessing and chunking](#audio-preprocessing-and-chunking) for what each field controls.
68
69 Implement the prompt construction via [get_generation_prompt][vllm.model_executor.models.interfaces.SupportsTranscription.get_generation_prompt]. The server builds a [SpeechToTextParams][vllm.config.speech_to_text.SpeechToTextParams] object that bundles the resampled waveform, task parameters, and request-specific options. Your model receives this single object and returns a valid [PromptType][vllm.inputs.llm.PromptType]. There are two common patterns:
70
71 #### Multimodal LLM with audio embeddings (e.g., Voxtral, Gemma3n)
72
73 Return a dict containing `multi_modal_data` with the audio, and either a `prompt` string or `prompt_token_ids`:
74
75 ??? code "get_generation_prompt()"
76
77 ```python
78 from vllm.config.speech_to_text import SpeechToTextParams
79
80 class YourASRModel(nn.Module, SupportsTranscription):
81 ...
82
83 @classmethod
84 def get_generation_prompt(
85 cls,
86 stt_params: SpeechToTextParams,
87 ) -> PromptType:
88 audio = stt_params.audio
89 stt_config = stt_params.stt_config
90 task_type = stt_params.task_type
91
92 task_word = "Transcribe" if task_type == "transcribe" else "Translate"
93 prompt = (
94 "<start_of_turn>user\n"
95 f"{task_word} this audio: <audio_soft_token>"
96 "<end_of_turn>\n<start_of_turn>model\n"
97 )
98
99 return {
100 "multi_modal_data": {"audio": (audio, stt_config.sample_rate)},
101 "prompt": prompt,
102 }
103 ```
104
105 For further clarification on multi modal inputs, please refer to [Multi-Modal Inputs](../../features/multimodal_inputs.md).
106
107 #### Encoder–decoder audio-only (e.g., Whisper)
108
109 Return a dict with separate `encoder_prompt` and `decoder_prompt` entries:
110
111 ??? code "get_generation_prompt()"
112
113 ```python
114 from vllm.config.speech_to_text import SpeechToTextParams
115
116 class YourASRModel(nn.Module, SupportsTranscription):
117 ...
118
119 @classmethod
120 def get_generation_prompt(
121 cls,
122 stt_params: SpeechToTextParams,
123 ) -> PromptType:
124 audio = stt_params.audio
125 stt_config = stt_params.stt_config
126 language = stt_params.language
127 task_type = stt_params.task_type
128 request_prompt = stt_params.request_prompt
129
130 if language is None:
131 raise ValueError("Language must be specified")
132
133 prompt = {
134 "encoder_prompt": {
135 "prompt": "",
136 "multi_modal_data": {
137 "audio": (audio, stt_config.sample_rate),
138 },
139 },
140 "decoder_prompt": (
141 (f"<|prev|>{request_prompt}" if request_prompt else "")
142 + f"<|startoftranscript|><|{language}|>"
143 + f"<|{task_type}|><|notimestamps|>"
144 ),
145 }
146 return cast(PromptType, prompt)
147 ```
148
149 ### `validate_language` (optional)
150
151 Language validation via [validate_language][vllm.model_executor.models.interfaces.SupportsTranscription.validate_language]
152
153 If your model requires a language and you want a default, override this method (see Whisper):
154
155 ??? code "validate_language()"
156
157 ```python
158 @classmethod
159 def validate_language(cls, language: str | None) -> str | None:
160 if language is None:
161 logger.warning(
162 "Defaulting to language='en'. If you wish to transcribe "
163 "audio in a different language, pass the `language` field "
164 "in the TranscriptionRequest."
165 )
166 language = "en"
167 return super().validate_language(language)
168 ```
169
170 ### `get_num_audio_tokens` (optional)
171
172 Token accounting for streaming via [get_num_audio_tokens][vllm.model_executor.models.interfaces.SupportsTranscription.get_num_audio_tokens]
173
174 Provide a fast duration→token estimate to improve streaming usage statistics:
175
176 ??? code "get_num_audio_tokens()"
177
178 ```python
179 class YourASRModel(nn.Module, SupportsTranscription):
180 ...
181
182 @classmethod
183 def get_num_audio_tokens(
184 cls,
185 audio_duration_s: float,
186 stt_config: SpeechToTextConfig,
187 model_config: ModelConfig,
188 ) -> int | None:
189 # Return None if unknown; otherwise return an estimate.
190 return int(audio_duration_s * stt_config.sample_rate // 320) # example
191 ```
192
193 ## Audio preprocessing and chunking
194
195 The API server takes care of basic audio I/O and optional chunking before building prompts:
196
197 - Resampling: Input audio is resampled to `SpeechToTextConfig.sample_rate` using `AudioResampler`.
198 - Chunking: If `SpeechToTextConfig.allow_audio_chunking` is True and the duration exceeds `max_audio_clip_s`, the server splits the audio into chunks and generates a prompt per chunk. There is no overlap between chunks, overlap_chunk_second controls the size of the search window used to find the split point.
199 - Energy-aware splitting: When `min_energy_split_window_size` is set, the server finds low-energy regions to minimize cutting within words.
200
201 Relevant server logic:
202
203 ??? code "_preprocess_speech_to_text()"
204
205 ```python
206 # vllm/entrypoints/openai/speech_to_text.py
207 async def _preprocess_speech_to_text(...):
208 language = self.model_cls.validate_language(request.language)
209 ...
210 y, sr = load_audio(bytes_, sr=self.asr_config.sample_rate)
211 duration = get_audio_duration(y=y, sr=sr)
212 do_split_audio = (self.asr_config.allow_audio_chunking
213 and duration > self.asr_config.max_audio_clip_s)
214 chunks = [y] if not do_split_audio else self._split_audio(y, int(sr))
215 prompts = []
216 for chunk in chunks:
217 stt_params = request.build_stt_params(
218 audio=chunk,
219 stt_config=self.asr_config,
220 model_config=self.model_config,
221 task_type=self.task_type,
222 )
223 prompt = self.model_cls.get_generation_prompt(stt_params)
224 prompts.append(prompt)
225 return prompts, duration
226 ```
227
228 ## Exposing tasks automatically
229
230 vLLM automatically advertises transcription support if your model implements the interface:
231
232 ```python
233 if supports_transcription(model):
234 if model.supports_transcription_only:
235 return ["transcription"]
236 supported_tasks.append("transcription")
237 ```
238
239 When enabled, the server initializes the transcription and translation handlers:
240
241 ```python
242 state.openai_serving_transcription = OpenAIServingTranscription(...) if "transcription" in supported_tasks else None
243 state.openai_serving_translation = OpenAIServingTranslation(...) if "transcription" in supported_tasks else None
244 ```
245
246 No extra registration is required beyond having your model class available via the model registry and implementing `SupportsTranscription`.
247
248 ## Examples in-tree
249
250 - Whisper encoder–decoder (audio-only): [vllm/model_executor/models/whisper.py](../../../vllm/model_executor/models/whisper.py)
251 - Voxtral decoder-only (audio embeddings + LLM): [vllm/model_executor/models/voxtral.py](../../../vllm/model_executor/models/voxtral.py). Make sure to have installed `mistral-common[audio]`.
252 - Gemma3n decoder-only with fixed instruction prompt: [vllm/model_executor/models/gemma3n_mm.py](../../../vllm/model_executor/models/gemma3n_mm.py)
253 - Qwen3-Omni multimodal with audio embeddings: [vllm/model_executor/models/qwen3_omni_moe_thinker.py](../../../vllm/model_executor/models/qwen3_omni_moe_thinker.py)
254
255 ## Test with the API
256
257 Once your model implements `SupportsTranscription`, you can test the endpoints (API mimics OpenAI):
258
259 - Transcription (ASR):
260
261 ```bash
262 curl -s -X POST \
263 -H "Authorization: Bearer $VLLM_API_KEY" \
264 -H "Content-Type: multipart/form-data" \
265 -F "file=@/path/to/audio.wav" \
266 -F "model=$MODEL_ID" \
267 http://localhost:8000/v1/audio/transcriptions
268 ```
269
270 - Translation (source → English unless otherwise supported):
271
272 ```bash
273 curl -s -X POST \
274 -H "Authorization: Bearer $VLLM_API_KEY" \
275 -H "Content-Type: multipart/form-data" \
276 -F "file=@/path/to/audio.wav" \
277 -F "model=$MODEL_ID" \
278 http://localhost:8000/v1/audio/translations
279 ```
280
281 Or check out more examples in [examples/speech_to_text](../../../examples/speech_to_text).
282
283 !!! note
284 - If your model handles chunking internally (e.g., via its processor or encoder), set `min_energy_split_window_size=None` in the returned `SpeechToTextConfig` to disable server-side chunking.
285 - Implementing `get_num_audio_tokens` improves accuracy of streaming usage metrics (`prompt_tokens`) without an extra forward pass.
286 - For multilingual behavior, keep `supported_languages` aligned with actual model capabilities.