@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 404c85ca72
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

[Docs] Add transcription support to model (#24664)

Signed-off-by: NickLucche <nlucches@redhat.com>

Nicolò Lucchesi committed Sep 11, 2025 at 16:39 UTC 404c85ca7290d314bbbf4d130bb55becc437c4c2
3 files changed +275
docs/.nav.yml
+1
@@ -44,6 +44,7 @@ nav:
44 - contributing/model/registration.md
45 - contributing/model/tests.md
46 - contributing/model/multimodal.md
47 + - contributing/model/transcription.md
48 - CI: contributing/ci
49 - Design Documents: design
50 - API Reference:
docs/contributing/model/README.md
+1
@@ -15,6 +15,7 @@ Read through these pages for a step-by-step guide:
15 - [Registering a Model](registration.md)
16 - [Unit Testing](tests.md)
17 - [Multi-Modal Support](multimodal.md)
18 +- [Speech-to-Text Support](transcription.md)
19
20 !!! tip
21 If you are encountering issues while integrating your model into vLLM, feel free to open a [GitHub issue](https://github.com/vllm-project/vllm/issues)
docs/contributing/model/transcription.md new
+273
@@ -0,0 +1,273 @@
1 +# Speech-to-Text (Transcription/Translation) Support
2 +
3 +This document walks you through the steps to add support for speech-to-text (ASR) models to vLLM’s transcription and translation APIs by implementing [SupportsTranscription][vllm.model_executor.models.interfaces.SupportsTranscription].
4 +Please refer to the [supported models](../../models/supported_models.md#transcription) for further guidance.
5 +
6 +## 1. Update the base vLLM model
7 +
8 +It is assumed you have already implemented your model in vLLM according to the basic model guide. Extend your model with the [SupportsTranscription][vllm.model_executor.models.interfaces.SupportsTranscription] interface and implement the following class attributes and methods.
9 +
10 +- Declare supported languages and capabilities:
11 +
12 + ??? code
13 +
14 + ```python
15 + from typing import ClassVar, Mapping, Optional, Literal
16 + import numpy as np
17 + import torch
18 + from torch import nn
19 +
20 + from vllm.config import ModelConfig, SpeechToTextConfig
21 + from vllm.inputs.data import PromptType
22 + from vllm.model_executor.models.interfaces import SupportsTranscription
23 +
24 + class YourASRModel(nn.Module, SupportsTranscription):
25 + # Map of ISO 639-1 language codes to language names
26 + supported_languages: ClassVar[Mapping[str, str]] = {
27 + "en": "English",
28 + "it": "Italian",
29 + # ... add more as needed
30 + }
31 +
32 + # If your model only supports audio-conditioned generation
33 + # (no text-only generation), enable this flag.
34 + supports_transcription_only: ClassVar[bool] = True
35 + ```
36 +
37 + - The `supported_languages` mapping is validated at init time.
38 + - Set `supports_transcription_only=True` if the model should not serve text generation (eg Whisper).
39 +
40 +- Provide an ASR configuration via [get_speech_to_text_config][vllm.model_executor.models.interfaces.SupportsTranscription.get_speech_to_text_config].
41 + This is for controlling general behavior of the API when serving your model:
42 +
43 + ??? code
44 +
45 + ```python
46 + class YourASRModel(nn.Module, SupportsTranscription):
47 + ...
48 +
49 + @classmethod
50 + def get_speech_to_text_config(
51 + cls,
52 + model_config: ModelConfig,
53 + task_type: Literal["transcribe", "translate"],
54 + ) -> SpeechToTextConfig:
55 + return SpeechToTextConfig(
56 + sample_rate=16_000,
57 + max_audio_clip_s=30,
58 + # Set to None to disable server-side chunking if your
59 + # model/processor handles it already
60 + min_energy_split_window_size=None,
61 + )
62 + ```
63 +
64 + See the “Audio preprocessing and chunking” section for what each field controls.
65 +
66 +- Implement the prompt construction via [get_generation_prompt][vllm.model_executor.models.interfaces.SupportsTranscription.get_generation_prompt]. The server passes you the resampled waveform and task parameters; you return a valid [PromptType][vllm.inputs.data.PromptType]. There are two common patterns:
67 +
68 + #### A. Multimodal LLM with audio embeddings (e.g., Voxtral, Gemma3n)
69 +
70 + Return a dict containing `multi_modal_data` with the audio, and either a `prompt` string or `prompt_token_ids`:
71 +
72 + ??? code
73 +
74 + ```python
75 + class YourASRModel(nn.Module, SupportsTranscription):
76 + ...
77 +
78 + @classmethod
79 + def get_generation_prompt(
80 + cls,
81 + audio: np.ndarray,
82 + stt_config: SpeechToTextConfig,
83 + model_config: ModelConfig,
84 + language: Optional[str],
85 + task_type: Literal["transcribe", "translate"],
86 + request_prompt: str,
87 + to_language: Optional[str],
88 + ) -> PromptType:
89 + # Example with a free-form instruction prompt
90 + task_word = "Transcribe" if task_type == "transcribe" else "Translate"
91 + prompt = (
92 + "<start_of_turn>user\n"
93 + f"{task_word} this audio: <audio_soft_token>"
94 + "<end_of_turn>\n<start_of_turn>model\n"
95 + )
96 +
97 + return {
98 + "multi_modal_data": {"audio": (audio, stt_config.sample_rate)},
99 + "prompt": prompt,
100 + }
101 + ```
102 +
103 + For further clarification on multi modal inputs, please refer to [Multi-Modal Inputs](../../features/multimodal_inputs.md).
104 +
105 + #### B. Encoder–decoder audio-only (e.g., Whisper)
106 +
107 + Return a dict with separate `encoder_prompt` and `decoder_prompt` entries:
108 +
109 + ??? code
110 +
111 + ```python
112 + class YourASRModel(nn.Module, SupportsTranscription):
113 + ...
114 +
115 + @classmethod
116 + def get_generation_prompt(
117 + cls,
118 + audio: np.ndarray,
119 + stt_config: SpeechToTextConfig,
120 + model_config: ModelConfig,
121 + language: Optional[str],
122 + task_type: Literal["transcribe", "translate"],
123 + request_prompt: str,
124 + to_language: Optional[str],
125 + ) -> PromptType:
126 + if language is None:
127 + raise ValueError("Language must be specified")
128 +
129 + prompt = {
130 + "encoder_prompt": {
131 + "prompt": "",
132 + "multi_modal_data": {
133 + "audio": (audio, stt_config.sample_rate),
134 + },
135 + },
136 + "decoder_prompt": (
137 + (f"<|prev|>{request_prompt}" if request_prompt else "")
138 + + f"<|startoftranscript|><|{language}|>"
139 + + f"<|{task_type}|><|notimestamps|>"
140 + ),
141 + }
142 + return cast(PromptType, prompt)
143 + ```
144 +
145 +
146 +- (Optional) Language validation via [validate_language][vllm.model_executor.models.interfaces.SupportsTranscription.validate_language]
147 +
148 + If your model requires a language and you want a default, override this method (see Whisper):
149 +
150 + ```python
151 + @classmethod
152 + def validate_language(cls, language: Optional[str]) -> Optional[str]:
153 + if language is None:
154 + logger.warning(
155 + "Defaulting to language='en'. If you wish to transcribe audio in a different language, pass the `language` field.")
156 + language = "en"
157 + return super().validate_language(language)
158 + ```
159 +
160 +- (Optional) Token accounting for streaming via [get_num_audio_tokens][vllm.model_executor.models.interfaces.SupportsTranscription.get_num_audio_tokens]
161 +
162 + Provide a fast duration→token estimate to improve streaming usage statistics:
163 +
164 + ??? code
165 + ```python
166 + class YourASRModel(nn.Module, SupportsTranscription):
167 + ...
168 +
169 + @classmethod
170 + def get_num_audio_tokens(
171 + cls,
172 + audio_duration_s: float,
173 + stt_config: SpeechToTextConfig,
174 + model_config: ModelConfig,
175 + ) -> Optional[int]:
176 + # Return None if unknown; otherwise return an estimate.
177 + return int(audio_duration_s * stt_config.sample_rate // 320) # example
178 + ```
179 +
180 +
181 +## 2. Audio preprocessing and chunking
182 +
183 +The API server takes care of basic audio I/O and optional chunking before building prompts:
184 +
185 +- Resampling: Input audio is resampled to `SpeechToTextConfig.sample_rate` using `librosa`.
186 +- Chunking: If `SpeechToTextConfig.allow_audio_chunking` is True and the duration exceeds `max_audio_clip_s`, the server splits the audio into overlapping chunks and generates a prompt per chunk. Overlap is controlled by `overlap_chunk_second`.
187 +- Energy-aware splitting: When `min_energy_split_window_size` is set, the server finds low-energy regions to minimize cutting within words.
188 +
189 +Relevant server logic:
190 +??? code
191 + ```python
192 + # vllm/entrypoints/openai/speech_to_text.py
193 + async def _preprocess_speech_to_text(...):
194 + language = self.model_cls.validate_language(request.language)
195 + ...
196 + y, sr = librosa.load(bytes_, sr=self.asr_config.sample_rate)
197 + duration = librosa.get_duration(y=y, sr=sr)
198 + do_split_audio = (self.asr_config.allow_audio_chunking
199 + and duration > self.asr_config.max_audio_clip_s)
200 + chunks = [y] if not do_split_audio else self._split_audio(y, int(sr))
201 + prompts = []
202 + for chunk in chunks:
203 + prompt = self.model_cls.get_generation_prompt(
204 + audio=chunk,
205 + stt_config=self.asr_config,
206 + model_config=self.model_config,
207 + language=language,
208 + task_type=self.task_type,
209 + request_prompt=request.prompt,
210 + to_language=to_language,
211 + )
212 + prompts.append(prompt)
213 + return prompts, duration
214 + ```
215 +
216 +## 3. Exposing tasks automatically
217 +
218 +- vLLM automatically advertises transcription support if your model implements the interface:
219 +
220 +```python
221 +if supports_transcription(model):
222 + if model.supports_transcription_only:
223 + return ["transcription"]
224 + supported_tasks.append("transcription")
225 +```
226 +
227 +- When enabled, the server initializes the transcription and translation handlers:
228 +
229 +```python
230 +state.openai_serving_transcription = OpenAIServingTranscription(...) if "transcription" in supported_tasks else None
231 +state.openai_serving_translation = OpenAIServingTranslation(...) if "transcription" in supported_tasks else None
232 +```
233 +
234 +No extra registration is required beyond having your model class available via the model registry and implementing `SupportsTranscription`.
235 +
236 +## 4. Examples in-tree
237 +
238 +- Whisper encoder–decoder (audio-only): <gh-file:vllm/model_executor/models/whisper.py>
239 +- Voxtral decoder-only (audio embeddings + LLM): <gh-file:vllm/model_executor/models/voxtral.py>
240 +- Gemma3n decoder-only with fixed instruction prompt: <gh-file:vllm/model_executor/models/gemma3n_mm.py>
241 +
242 +## 5. Test with the API
243 +
244 +Once your model implements `SupportsTranscription`, you can test the endpoints (API mimics OpenAI):
245 +
246 +- Transcription (ASR):
247 +
248 + ```bash
249 + curl -s -X POST \
250 + -H "Authorization: Bearer $VLLM_API_KEY" \
251 + -H "Content-Type: multipart/form-data" \
252 + -F "file=@/path/to/audio.wav" \
253 + -F "model=$MODEL_ID" \
254 + http://localhost:8000/v1/audio/transcriptions
255 + ```
256 +
257 +- Translation (source → English unless otherwise supported):
258 +
259 + ```bash
260 + curl -s -X POST \
261 + -H "Authorization: Bearer $VLLM_API_KEY" \
262 + -H "Content-Type: multipart/form-data" \
263 + -F "file=@/path/to/audio.wav" \
264 + -F "model=$MODEL_ID" \
265 + http://localhost:8000/v1/audio/translations
266 + ```
267 +Or check out more examples in <gh-file:examples/online_serving>.
268 +
269 +!!! note
270 +
271 +- If your model handles chunking internally (e.g., via its processor or encoder), set `min_energy_split_window_size=None` in the returned `SpeechToTextConfig` to disable server-side chunking.
272 +- Implementing `get_num_audio_tokens` improves accuracy of streaming usage metrics (`prompt_tokens`) without an extra forward pass.
273 +- For multilingual behavior, keep `supported_languages` aligned with actual model capabilities.
\ No newline at end of file