| 1 | # Online Serving |
| 2 | |
| 3 | vLLM provides an HTTP server that is compatible with many interfaces! |
| 4 | |
| 5 | ## OpenAI-Compatible Server |
| 6 | |
| 7 | We currently support the following OpenAI APIs: |
| 8 | |
| 9 | - [Completions API](./openai_compatible_server.md#completions-api) (`/v1/completions`) |
| 10 | - Only applicable to [text generation models](../../models/generative_models.md). |
| 11 | - *Note: `suffix` parameter is not supported.* |
| 12 | - [Chat Completions API](./openai_compatible_server.md#chat-api) (`/v1/chat/completions`) |
| 13 | - Only applicable to [text generation models](../../models/generative_models.md) with a [chat template](#chat-template). |
| 14 | - *Note: `user` parameter is ignored.* |
| 15 | - *Note:* Setting the `parallel_tool_calls` parameter to `false` ensures vLLM only returns zero or one tool call per request. Setting it to `true` (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to `true`, as that behavior is model dependent and not all models are designed to support parallel tool calls. |
| 16 | - [Chat Completions batch API](./openai_compatible_server.md#chat-api) (`/v1/chat/completions/batch`) |
| 17 | - [Responses API](./openai_compatible_server.md#responses-api) (`/v1/responses`, `/v1/responses/{response_id}`, `/v1/responses/{response_id}/cancel`) |
| 18 | - Only applicable to [text generation models](../../models/generative_models.md). |
| 19 | - [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) |
| 20 | - Only applicable to [embedding models](../../models/pooling_models/embed.md). |
| 21 | - [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`) |
| 22 | - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). |
| 23 | - [Translation API](./speech_to_text.md#translations-api) (`/v1/audio/translations`) |
| 24 | - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). |
| 25 | |
| 26 | ## Anthropic APIs |
| 27 | |
| 28 | - Anthropic messages API (`/v1/messages`, `/v1/messages/count_tokens`) |
| 29 | |
| 30 | ## Cohere APIs |
| 31 | |
| 32 | - [Cohere Embed API](../../models/pooling_models/embed.md#cohere-embed-api) (`/v2/embed`) |
| 33 | - Compatible with [Cohere's Embed API](https://docs.cohere.com/reference/embed) |
| 34 | - Works with any [embedding model](../../models/pooling_models/embed.md#supported-models), including multimodal models. |
| 35 | - [Cohere Rerank API](../../models/pooling_models/scoring.md#cohere-rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) |
| 36 | - Implements [Jina AI's v1 rerank API](https://jina.ai/reranker/) |
| 37 | - compatible with [Cohere's v1 & v2 rerank APIs](https://docs.cohere.com/v2/reference/rerank) |
| 38 | |
| 39 | ## Pooling APIs |
| 40 | |
| 41 | For further details on pooling models, please refer to [this page](../../models/pooling_models/README.md). |
| 42 | |
| 43 | - [Classification Usages](../../models/pooling_models/classify.md) |
| 44 | - [Classification API](../../models/pooling_models/classify.md#online-serving) (`/classify`) |
| 45 | - Only applicable to [classification models](../../models/pooling_models/classify.md). |
| 46 | - [Embedding Usages](../../models/pooling_models/embed.md) |
| 47 | - [Cohere Embed API](../../models/pooling_models/embed.md#cohere-embed-api) (`/v2/embed`) |
| 48 | - [OpenAI-compatible Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) |
| 49 | - Only applicable to [embedding models](../../models/pooling_models/embed.md). |
| 50 | - [Scoring Usages](../../models/pooling_models/scoring.md) |
| 51 | - [Score API](../../models/pooling_models/scoring.md#score-api) (`/score`, `/v1/score`) |
| 52 | - [Cohere Rerank API](../../models/pooling_models/scoring.md#cohere-rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) |
| 53 | - Applicable to [score models](../../models/pooling_models/scoring.md) (cross-encoder, bi-encoder, late-interaction). |
| 54 | - [Pooling API](../../models/pooling_models/README.md#pooling-api) (`/pooling`) |
| 55 | - Applicable to all [pooling models](../../models/pooling_models/README.md). |
| 56 | |
| 57 | ## Speech to Text APIs |
| 58 | |
| 59 | For further details on speech to text, please refer to [this page](speech_to_text.md). |
| 60 | |
| 61 | - [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`) |
| 62 | - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). |
| 63 | - [Translation API](./speech_to_text.md#translations-api) (`/v1/audio/translations`) |
| 64 | - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). |
| 65 | - [Realtime API](./speech_to_text.md#realtime-api) (`/v1/realtime`) |
| 66 | - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#realtime-transcription). |
| 67 | |
| 68 | ## Custom APIs |
| 69 | |
| 70 | - [Classification API](../../models/pooling_models/classify.md#classification-api) (`/classify`) |
| 71 | - Only applicable to [classification models](../../models/pooling_models/classify.md). |
| 72 | - [Score API](../../models/pooling_models/scoring.md#score-api) (`/score`, `/v1/score`) |
| 73 | - Applicable to [score models](../../models/pooling_models/scoring.md) (cross-encoder, bi-encoder, late-interaction). |
| 74 | - [Pooling API](../../models/pooling_models/README.md#pooling-api) (`/pooling`) |
| 75 | - Applicable to all [pooling models](../../models/pooling_models/README.md). |
| 76 | - [Generative Scoring API](generative_scoring.md) (`/generative_scoring`) |
| 77 | - Applicable to [CausalLM models](../../models/generative_models.md) (task `"generate"`). |
| 78 | - Computes next-token probabilities for specified `label_token_ids`. |
| 79 | |
| 80 | ## Instrumentator APIs |
| 81 | |
| 82 | ### Basic APIs |
| 83 | |
| 84 | - `/version` - Version information |
| 85 | - `/load` - Server load metrics |
| 86 | - `/v1/models` - List available models |
| 87 | - `/health` - Health check |
| 88 | |
| 89 | ### Metrics APIs |
| 90 | |
| 91 | For further details on metrics, please refer to [this page](../../design/metrics.md). |
| 92 | |
| 93 | - `/metrics` - Prometheus-compatible metrics HTTP endpoint |
| 94 | |
| 95 | ### Offline API Documentation |
| 96 | |
| 97 | The FastAPI `/docs` endpoint requires an internet connection by default. To enable offline access in air-gapped environments, use the `--enable-offline-docs` flag: |
| 98 | |
| 99 | ```bash |
| 100 | vllm serve NousResearch/Meta-Llama-3-8B-Instruct --enable-offline-docs |
| 101 | ``` |
| 102 | |
| 103 | ### LoRA dynamic loading |
| 104 | |
| 105 | LoRA dynamic loading & unloading is enabled in the API server. This should ONLY be used for local development! |
| 106 | |
| 107 | - `/v1/load_lora_adapter` - LoRA dynamic loading |
| 108 | - `/v1/unload_lora_adapter` - LoRA dynamic unloading |
| 109 | |
| 110 | ### Profiling APIs |
| 111 | |
| 112 | For further details on profiling vLLM, please refer to [this page](../../contributing/profiling.md). |
| 113 | |
| 114 | - `/start_profile` - Start PyTorch profiler |
| 115 | - `/stop_profile` - Stop PyTorch profiler |
| 116 | |
| 117 | ### SageMaker APIs |
| 118 | |
| 119 | - `/ping` - SageMaker health check |
| 120 | - `/invocations` - SageMaker-compatible endpoint (routes to the same inference functions as `/v1` endpoints) |
| 121 | |
| 122 | ## Scale-Out APIs |
| 123 | |
| 124 | Scale-out APIs are disabled by default on `vllm serve`. Set `--enable-scale-out` to register the endpoints below. The dedicated `vllm launch render` and `vllm serve --tokens-only` modes always register their required endpoints regardless of `--enable-scale-out`. |
| 125 | |
| 126 | ### Tokens IN <> Tokens OUT APIs |
| 127 | |
| 128 | - `/inference/v1/generate` - Generate completions |
| 129 | - `/abort_requests` - Abort in-flight requests (only when `--tokens-only` is also set) |
| 130 | |
| 131 | ### Renderer APIs |
| 132 | |
| 133 | Renderer APIs preprocess completion, chat, and Responses requests without running |
| 134 | inference. They handle tokenization, model-specific prompt formatting, and |
| 135 | multimodal preprocessing, returning prompt token IDs, sampling parameters, and |
| 136 | any processed multimodal inputs for generation. |
| 137 | |
| 138 | See the [renderer guide](renderer.md) for setup instructions and examples. |
| 139 | |
| 140 | - [Completions Render API](renderer.md) (`/v1/completions/render`) |
| 141 | - Render completion requests |
| 142 | - [Chat Completions Render API](renderer.md) (`/v1/chat/completions/render`) |
| 143 | - Render chat completions |
| 144 | - [Responses Render API](renderer.md) (`/v1/responses/render`) |
| 145 | - Render self-contained Responses requests |
| 146 | |
| 147 | ### Derenderer APIs |
| 148 | |
| 149 | For further details on derenderer APIs, please refer to [this page](derenderer.md). |
| 150 | |
| 151 | - [Chat Completions Derender API](derenderer.md) (`/v1/chat/completions/derender`) |
| 152 | - Derender chat completion requests |
| 153 | - [Completions Derender API](derenderer.md) (`/v1/completions/derender`) |
| 154 | - Derender completion requests |
| 155 | |
| 156 | ## Tokenize APIs |
| 157 | |
| 158 | - `/tokenize` - Tokenize text |
| 159 | - `/detokenize` - Detokenize tokens |
| 160 | - `/tokenizer_info` - Get comprehensive tokenizer information including chat templates and configuration |
| 161 | |
| 162 | ## Elastic Expert Parallelism (EEP) |
| 163 | |
| 164 | - `/scale_elastic_ep` - Trigger scaling operations |
| 165 | - `/is_scaling_elastic_ep` - Check if scaling is in progress |
| 166 | |
| 167 | ## Server in development mode |
| 168 | |
| 169 | When using the flag VLLM_SERVER_DEV_MODE=1, you enable development endpoints. |
| 170 | |
| 171 | **SECURITY WARNING: These endpoints should NOT be used in production!** |
| 172 | |
| 173 | ### Cache Management APIs |
| 174 | |
| 175 | - `/reset_prefix_cache` - Reset prefix cache (can disrupt service) |
| 176 | - `/reset_mm_cache` - Reset multimodal cache (can disrupt service) |
| 177 | - `/reset_encoder_cache` - Reset encoder cache (can disrupt service) |
| 178 | |
| 179 | ### Weight Transfer APIs (RL Training) |
| 180 | |
| 181 | For further details on Weight Transfer, please refer to [this page](../../training/weight_transfer/README.md). |
| 182 | |
| 183 | - `/pause` - Pause generation (causes denial of service) |
| 184 | - `/resume` - Resume generation |
| 185 | - `/is_paused` - Check if generation is paused |
| 186 | - `/abort_requests` - Abort in-flight requests (all in-flight, or the given `request_ids`) without pausing the scheduler |
| 187 | - `/init_weight_transfer_engine` - Initialize weight transfer engine for RLHF |
| 188 | - `/start_weight_update` - Prepares the inference engine for a weight update. |
| 189 | - `/update_weights` - Update model weights (can alter model behavior) |
| 190 | - `/finish_weight_update` - Finalizes the weight update |
| 191 | - `/update_weight_version` - Set the weight version without updating model weights |
| 192 | - `/weight_info` - Get the latest committed weight version |
| 193 | - `/get_world_size` - Get distributed world size |
| 194 | |
| 195 | ### Collective RPC |
| 196 | |
| 197 | - `/collective_rpc` - Execute arbitrary RPC methods on the engine (extremely dangerous) |
| 198 | |
| 199 | ### Server info |
| 200 | |
| 201 | - `/server_info` - Get detailed server configuration |
| 202 | |
| 203 | ### Sleep Mode APIs |
| 204 | |
| 205 | For further details on sleep mode, please refer to [this page](../../features/sleep_mode.md). |
| 206 | |
| 207 | - `/sleep` - Put engine to sleep (causes denial of service) |
| 208 | - `/wake_up` - Wake engine from sleep |
| 209 | - `/is_sleeping` - Check if engine is sleeping |
| 210 | |
| 211 | ## Chat Template |
| 212 | |
| 213 | In order for the language model to support chat protocol, vLLM requires the model to include |
| 214 | a chat template in its tokenizer configuration. The chat template is a Jinja2 template that |
| 215 | specifies how roles, messages, and other chat-specific tokens are encoded in the input. |
| 216 | |
| 217 | An example chat template for `NousResearch/Meta-Llama-3-8B-Instruct` can be found [here](https://llama.com/docs/model-cards-and-prompt-formats/meta-llama-3/#prompt-template-for-meta-llama-3) |
| 218 | |
| 219 | Some models do not provide a chat template even though they are instruction/chat fine-tuned. For those models, |
| 220 | you can manually specify their chat template in the `--chat-template` parameter with the file path to the chat |
| 221 | template, or the template in string form. Without a chat template, the server will not be able to process chat |
| 222 | and all chat requests will error. |
| 223 | |
| 224 | ```bash |
| 225 | vllm serve <model> --chat-template ./path-to-chat-template.jinja |
| 226 | ``` |
| 227 | |
| 228 | vLLM community provides a set of chat templates for popular models. You can find them under the [examples](../../../examples) directory. |
| 229 | |
| 230 | With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies |
| 231 | both a `type` and a `text` field. An example is provided below: |
| 232 | |
| 233 | ```python |
| 234 | completion = client.chat.completions.create( |
| 235 | model="NousResearch/Meta-Llama-3-8B-Instruct", |
| 236 | messages=[ |
| 237 | { |
| 238 | "role": "user", |
| 239 | "content": [ |
| 240 | {"type": "text", "text": "Classify this sentiment: vLLM is wonderful!"}, |
| 241 | ], |
| 242 | }, |
| 243 | ], |
| 244 | ) |
| 245 | ``` |
| 246 | |
| 247 | Most chat templates for LLMs expect the `content` field to be a string, but there are some newer models like |
| 248 | `meta-llama/Llama-Guard-3-1B` that expect the content to be formatted according to the OpenAI schema in the |
| 249 | request. vLLM provides best-effort support to detect this automatically, which is logged as a string like |
| 250 | *"Detected the chat template content format to be..."*, and internally converts incoming requests to match |
| 251 | the detected format, which can be one of: |
| 252 | |
| 253 | - `"string"`: A string. |
| 254 | - Example: `"Hello world"` |
| 255 | - `"openai"`: A list of dictionaries, similar to OpenAI schema. |
| 256 | - Example: `[{"type": "text", "text": "Hello world!"}]` |
| 257 | |
| 258 | If the result is not what you expect, you can set the `--chat-template-content-format` CLI argument |
| 259 | to override which format to use. |
| 260 | |
| 261 | ## Ray Serve LLM |
| 262 | |
| 263 | Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure. |
| 264 | |
| 265 | Key capabilities: |
| 266 | |
| 267 | - Exposes an OpenAI-compatible HTTP API as well as a Pythonic API. |
| 268 | - Scales from a single GPU to a multi-node cluster without code changes. |
| 269 | - Provides observability and autoscaling policies through Ray dashboards and metrics. |
| 270 | |
| 271 | The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: [examples/ray_serving/ray_serve_deepseek.py](../../../examples/ray_serving/ray_serve_deepseek.py). |
| 272 | |
| 273 | Learn more about Ray Serve LLM with the official [Ray Serve LLM documentation](https://docs.ray.io/en/latest/serve/llm/index.html). |