| 1 | # Speech to Text APIs |
| 2 | |
| 3 | ## Transcriptions API |
| 4 | |
| 5 | Our Transcriptions API is compatible with [OpenAI's Transcriptions API](https://platform.openai.com/docs/api-reference/audio/createTranscription); |
| 6 | you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. |
| 7 | |
| 8 | !!! note |
| 9 | To use the Transcriptions API, please install with extra audio dependencies using `pip install vllm[audio]`. |
| 10 | |
| 11 | Code example: [examples/speech_to_text/openai/openai_transcription_client.py](../../../examples/speech_to_text/openai/openai_transcription_client.py) |
| 12 | |
| 13 | NOTE: beam search is currently supported in the transcriptions endpoint for encoder-decoder multimodal models, e.g., whisper, but highly inefficient as work for handling the encoder/decoder cache is actively ongoing. This is an active point of ongoing optimization and will be handled properly in the very near future. |
| 14 | |
| 15 | ### API Enforced Limits |
| 16 | |
| 17 | Set the maximum audio file size (in MB) that VLLM will accept, via the |
| 18 | `VLLM_MAX_AUDIO_CLIP_FILESIZE_MB` environment variable. Default is 25 MB. |
| 19 | |
| 20 | ### Uploading Audio Files |
| 21 | |
| 22 | The Transcriptions API supports uploading audio files in various formats including FLAC, MP3, MP4, MPEG, MPGA, M4A, OGG, WAV, and WEBM. |
| 23 | |
| 24 | **Using OpenAI Python Client:** |
| 25 | |
| 26 | ??? code |
| 27 | |
| 28 | ```python |
| 29 | from openai import OpenAI |
| 30 | |
| 31 | client = OpenAI( |
| 32 | base_url="http://localhost:8000/v1", |
| 33 | api_key="token-abc123", |
| 34 | ) |
| 35 | |
| 36 | # Upload audio file from disk |
| 37 | with open("audio.mp3", "rb") as audio_file: |
| 38 | transcription = client.audio.transcriptions.create( |
| 39 | model="openai/whisper-large-v3-turbo", |
| 40 | file=audio_file, |
| 41 | language="en", |
| 42 | response_format="verbose_json", |
| 43 | ) |
| 44 | |
| 45 | print(transcription.text) |
| 46 | ``` |
| 47 | |
| 48 | **Using curl with multipart/form-data:** |
| 49 | |
| 50 | ??? code |
| 51 | |
| 52 | ```bash |
| 53 | curl -X POST "http://localhost:8000/v1/audio/transcriptions" \ |
| 54 | -H "Authorization: Bearer token-abc123" \ |
| 55 | -F "file=@audio.mp3" \ |
| 56 | -F "model=openai/whisper-large-v3-turbo" \ |
| 57 | -F "language=en" \ |
| 58 | -F "response_format=verbose_json" |
| 59 | ``` |
| 60 | |
| 61 | **Supported Parameters:** |
| 62 | |
| 63 | - `file`: The audio file to transcribe (required) |
| 64 | - `model`: The model to use for transcription (required) |
| 65 | - `language`: The language code (e.g., "en", "zh") (optional) |
| 66 | - `prompt`: Optional text to guide the transcription style (optional) |
| 67 | - `response_format`: Format of the response ("json", "text", "verbose_json", or |
| 68 | "diarized_json") (optional) |
| 69 | - `temperature`: Sampling temperature between 0 and 1 (optional) |
| 70 | |
| 71 | For the complete list of supported parameters including sampling parameters and vLLM extensions, see the [protocol definitions](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/speech_to_text/transcription/protocol.py). |
| 72 | |
| 73 | **Response Format:** |
| 74 | |
| 75 | For `verbose_json` response format: |
| 76 | |
| 77 | ??? code |
| 78 | |
| 79 | ```json |
| 80 | { |
| 81 | "text": "Hello, this is a transcription of the audio file.", |
| 82 | "language": "en", |
| 83 | "duration": 5.42, |
| 84 | "segments": [ |
| 85 | { |
| 86 | "id": 0, |
| 87 | "seek": 0, |
| 88 | "start": 0.0, |
| 89 | "end": 2.5, |
| 90 | "text": "Hello, this is a transcription", |
| 91 | "tokens": [50364, 938, 428, 307, 275, 28347], |
| 92 | "temperature": 0.0, |
| 93 | "avg_logprob": -0.245, |
| 94 | "compression_ratio": 1.235, |
| 95 | "no_speech_prob": 0.012 |
| 96 | } |
| 97 | ] |
| 98 | } |
| 99 | ``` |
| 100 | Currently “verbose_json” response format doesn’t support no_speech_prob. |
| 101 | |
| 102 | For models with diarization support, `diarized_json` returns OpenAI-compatible |
| 103 | speaker segments. Currently, this is supported by |
| 104 | `OpenMOSS-Team/MOSS-Transcribe-Diarize`. |
| 105 | |
| 106 | ```json |
| 107 | { |
| 108 | "task": "transcribe", |
| 109 | "duration": 6.1, |
| 110 | "text": "Hello. Hi, how are you?", |
| 111 | "segments": [ |
| 112 | { |
| 113 | "type": "transcript.text.segment", |
| 114 | "id": "segment_0", |
| 115 | "start": 0.0, |
| 116 | "end": 2.8, |
| 117 | "text": "Hello.", |
| 118 | "speaker": "S01" |
| 119 | } |
| 120 | ], |
| 121 | "usage": {"type": "duration", "seconds": 7} |
| 122 | } |
| 123 | ``` |
| 124 | |
| 125 | ### Extra Parameters |
| 126 | |
| 127 | The following [sampling parameters](../../api/README.md#inference-parameters) are supported. |
| 128 | |
| 129 | ??? code |
| 130 | |
| 131 | ```python |
| 132 | --8<-- "vllm/entrypoints/speech_to_text/transcription/protocol.py:transcription-sampling-params" |
| 133 | ``` |
| 134 | |
| 135 | The following extra parameters are supported: |
| 136 | |
| 137 | ??? code |
| 138 | |
| 139 | ```python |
| 140 | --8<-- "vllm/entrypoints/speech_to_text/transcription/protocol.py:transcription-extra-params" |
| 141 | ``` |
| 142 | |
| 143 | ## Translations API |
| 144 | |
| 145 | Our Translation API is compatible with [OpenAI's Translations API](https://platform.openai.com/docs/api-reference/audio/createTranslation); |
| 146 | you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. |
| 147 | Whisper models can translate audio from one of the 55 non-English supported languages into English. |
| 148 | Please mind that the popular `openai/whisper-large-v3-turbo` model does not support translating. |
| 149 | |
| 150 | !!! note |
| 151 | To use the Translation API, please install with extra audio dependencies using `pip install vllm[audio]`. |
| 152 | |
| 153 | Code example: [examples/speech_to_text/openai/openai_translation_client.py](../../../examples/speech_to_text/openai/openai_translation_client.py) |
| 154 | |
| 155 | ### Extra Parameters |
| 156 | |
| 157 | The following [sampling parameters](../../api/README.md#inference-parameters) are supported. |
| 158 | |
| 159 | ```python |
| 160 | --8<-- "vllm/entrypoints/speech_to_text/translation/protocol.py:translation-sampling-params" |
| 161 | ``` |
| 162 | |
| 163 | The following extra parameters are supported: |
| 164 | |
| 165 | ```python |
| 166 | --8<-- "vllm/entrypoints/speech_to_text/translation/protocol.py:translation-extra-params" |
| 167 | ``` |
| 168 | |
| 169 | ## Realtime API |
| 170 | |
| 171 | The Realtime API provides WebSocket-based streaming audio transcription, allowing real-time speech-to-text as audio is being recorded. |
| 172 | |
| 173 | !!! note |
| 174 | To use the Realtime API, please install with extra audio dependencies using `uv pip install vllm[audio]`. |
| 175 | |
| 176 | ### Audio Format |
| 177 | |
| 178 | Audio must be sent as base64-encoded PCM16 audio at 16kHz sample rate, mono channel. |
| 179 | |
| 180 | ### Protocol Overview |
| 181 | |
| 182 | 1. Client connects to `ws://host/v1/realtime` |
| 183 | 2. Server sends `session.created` event |
| 184 | 3. Client optionally sends `session.update` with model/params |
| 185 | 4. Client sends `input_audio_buffer.commit` when ready |
| 186 | 5. Client sends `input_audio_buffer.append` events with base64 PCM16 chunks |
| 187 | 6. Server sends `transcription.delta` events with incremental text |
| 188 | 7. Server sends `transcription.done` with final text + usage |
| 189 | 8. Repeat from step 5 for next utterance |
| 190 | 9. Optionally, client sends input_audio_buffer.commit with final=True |
| 191 | to signal audio input is finished. Useful when streaming audio files |
| 192 | |
| 193 | ### Client → Server Events |
| 194 | |
| 195 | | Event | Description | |
| 196 | | ----- | ----------- | |
| 197 | | `input_audio_buffer.append` | Send base64-encoded audio chunk: `{"type": "input_audio_buffer.append", "audio": "<base64>"}` | |
| 198 | | `input_audio_buffer.commit` | Trigger transcription processing or end: `{"type": "input_audio_buffer.commit", "final": bool}` | |
| 199 | | `session.update` | Configure session: `{"type": "session.update", "model": "model-name"}` | |
| 200 | |
| 201 | ### Server → Client Events |
| 202 | |
| 203 | | Event | Description | |
| 204 | | ----- | ----------- | |
| 205 | | `session.created` | Connection established with session ID and timestamp | |
| 206 | | `transcription.delta` | Incremental transcription text: `{"type": "transcription.delta", "delta": "text"}` | |
| 207 | | `transcription.done` | Final transcription with usage stats | |
| 208 | | `error` | Error notification with message and optional code | |
| 209 | |
| 210 | #### Example Clients |
| 211 | |
| 212 | - [openai_realtime_client.py](https://github.com/vllm-project/vllm/tree/main/examples/speech_to_text/realtime/openai_realtime_client.py) - Upload and transcribe an audio file |
| 213 | - [openai_realtime_microphone_client.py](https://github.com/vllm-project/vllm/tree/main/examples/speech_to_text/realtime/openai_realtime_microphone_client.py) - Gradio demo for live microphone transcription |