A high-throughput and memory-efficient inference and serving engine for LLMs
[Docs] Update the online serving docs. (#45676)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
wang.yuqi committed
Jun 15, 2026 at 17:23 UTC
1d88c4daddb267173f69901dfbcbb20b21046fa4
4 files changed
+83
-37
docs/models/pooling_models/README.md
+1
-1
@@ -184,7 +184,7 @@ Our online Server provides endpoints that correspond to the offline APIs:
184
- Corresponding to `LLM.classify`:
185
- [Classification API](classify.md#online-serving)(`/classify`)
186
- Corresponding to `LLM.score`:
187
- - [Score API](scoring.md#score-api)(`/score`)
187
+ - [Score API](scoring.md#score-api) (`/score`, `/v1/score`)
188
- [Cohere Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)
189
- Pooling API (`/pooling`) is similar to `LLM.encode`, being applicable to all types of pooling models.
190
docs/models/pooling_models/scoring.md
+2
-2
@@ -19,7 +19,7 @@ The score models is designed to compute similarity scores between two input prom
19
- Offline APIs:
20
- `LLM.score`
21
- Online APIs:
22
- - [Score API](scoring.md#score-api) (`/score`)
22
+ - [Score API](scoring.md#score-api) (`/score`, `/v1/score`)
23
- [Cohere Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)
24
25
!!! note
@@ -157,7 +157,7 @@ A code example can be found here: [examples/basic/offline_inference/score.py](..
157
158
### Score API
159
160
-Our Score API (`/score`) is similar to `LLM.score`, compute similarity scores between two input prompts.
160
+Our Score API (`/score`, `/v1/score`) is similar to `LLM.score`, compute similarity scores between two input prompts.
161
162
#### Parameters
163
docs/serving/online_serving/README.md
+77
-32
@@ -9,12 +9,13 @@ We currently support the following OpenAI APIs:
9
- [Completions API](./openai_compatible_server.md#completions-api) (`/v1/completions`)
10
- Only applicable to [text generation models](../../models/generative_models.md).
11
- *Note: `suffix` parameter is not supported.*
12
-- [Responses API](./openai_compatible_server.md#responses-api) (`/v1/responses`)
13
- - Only applicable to [text generation models](../../models/generative_models.md).
12
- [Chat Completions API](./openai_compatible_server.md#chat-api) (`/v1/chat/completions`)
13
- Only applicable to [text generation models](../../models/generative_models.md) with a [chat template](./openai_compatible_server.md#chat-template).
14
- *Note: `user` parameter is ignored.*
15
- *Note:* Setting the `parallel_tool_calls` parameter to `false` ensures vLLM only returns zero or one tool call per request. Setting it to `true` (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to `true`, as that behavior is model dependent and not all models are designed to support parallel tool calls.
16
+- [Chat Completions batch API](./openai_compatible_server.md#chat-api) (`/v1/chat/completions/batch`)
17
+- [Responses API](./openai_compatible_server.md#responses-api) (`/v1/responses`, `/v1/responses/{response_id}`, `/v1/responses/{response_id}/cancel`)
18
+ - Only applicable to [text generation models](../../models/generative_models.md).
19
- [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)
20
- Only applicable to [embedding models](../../models/pooling_models/embed.md).
21
- [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`)
@@ -24,7 +25,7 @@ We currently support the following OpenAI APIs:
25
26
## Anthropic APIs
27
27
-- Anthropic messages API (`/v1/messages`)
28
+- Anthropic messages API (`/v1/messages`, `/v1/messages/count_tokens`)
29
30
## Cohere APIs
31
@@ -35,10 +36,6 @@ We currently support the following OpenAI APIs:
36
- Implements [Jina AI's v1 rerank API](https://jina.ai/reranker/)
37
- compatible with [Cohere's v1 & v2 rerank APIs](https://docs.cohere.com/v2/reference/rerank)
38
38
-## SageMaker APIs
39
-
40
-- `/invocations` - SageMaker-compatible endpoint (routes to the same inference functions as `/v1` endpoints)
41
-
39
## Pooling APIs
40
41
For further details on pooling models, please refer to [this page](../../models/pooling_models/README.md).
@@ -51,7 +48,7 @@ For further details on pooling models, please refer to [this page](../../models/
48
- [OpenAI-compatible Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)
49
- Only applicable to [embedding models](../../models/pooling_models/embed.md).
50
- [Scoring Usages](../../models/pooling_models/scoring.md)
54
- - [Score API](../../models/pooling_models/scoring.md#score-api) (`/score`)
51
+ - [Score API](../../models/pooling_models/scoring.md#score-api) (`/score`, `/v1/score`)
52
- [Cohere Rerank API](../../models/pooling_models/scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)
53
- Applicable to [score models](../../models/pooling_models/scoring.md) (cross-encoder, bi-encoder, late-interaction).
54
- [Pooling API](../../models/pooling_models/README.md#pooling-api) (`/pooling`)
@@ -68,17 +65,6 @@ For further details on speech to text, please refer to [this page](speech_to_tex
65
- [Realtime API](./speech_to_text.md#realtime-api) (`/v1/realtime`)
66
- Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#realtime-transcription).
67
71
-## Disaggregated APIs
72
-
73
-### Renderer APIs
74
-
75
-For further details on renderer APIs, please refer to [this page](renderer.md).
76
-
77
-- [Completions Render API](renderer.md) (`/v1/completions/render`)
78
- - Render completion requests
79
-- [Chat Completions Render API](renderer.md) (`/v1/chat/completions/render`)
80
- - Render chat completions
81
-
68
## Custom APIs
69
70
- [Classification API](../../models/pooling_models/classify.md#classification-api) (`/classify`)
@@ -91,14 +77,79 @@ For further details on renderer APIs, please refer to [this page](renderer.md).
77
- Applicable to [CausalLM models](../../models/generative_models.md) (task `"generate"`).
78
- Computes next-token probabilities for specified `label_token_ids`.
79
94
-## Utility APIs
80
+## Instrumentator APIs
81
+
82
+### Basic APIs
83
96
-- `/tokenize` - Tokenize text
97
-- `/detokenize` - Detokenize tokens
98
-- `/health` - Health check
99
-- `/ping` - SageMaker health check
84
- `/version` - Version information
85
- `/load` - Server load metrics
86
+- `/v1/models` - List available models
87
+- `/health` - Health check
88
+
89
+### Metrics APIs
90
+
91
+For further details on metrics, please refer to [this page](../../design/metrics.md).
92
+
93
+- `/metrics` - Prometheus-compatible metrics HTTP endpoint
94
+
95
+### Offline API Documentation
96
+
97
+The FastAPI `/docs` endpoint requires an internet connection by default. To enable offline access in air-gapped environments, use the `--enable-offline-docs` flag:
98
+
99
+```bash
100
+vllm serve NousResearch/Meta-Llama-3-8B-Instruct --enable-offline-docs
101
+```
102
+
103
+### LoRA dynamic loading
104
+
105
+LoRA dynamic loading & unloading is enabled in the API server. This should ONLY be used for local development!
106
+
107
+- `/v1/load_lora_adapter` - LoRA dynamic loading
108
+- `/v1/unload_lora_adapter` - LoRA dynamic unloading
109
+
110
+### Profiling APIs
111
+
112
+For further details on profiling vLLM, please refer to [this page](../../contributing/profiling.md).
113
+
114
+- `/start_profile` - Start PyTorch profiler
115
+- `/stop_profile` - Stop PyTorch profiler
116
+
117
+### SageMaker APIs
118
+
119
+- `/ping` - SageMaker health check
120
+- `/invocations` - SageMaker-compatible endpoint (routes to the same inference functions as `/v1` endpoints)
121
+
122
+## Disaggregated Everything
123
+
124
+### Tokens IN <> Tokens OUT
125
+
126
+- `/inference/v1/generate` - Generate completions
127
+- `/abort_requests` - Abort in-flight requests (only when `--tokens-only` is also set)
128
+
129
+### Renderer APIs
130
+
131
+For further details on renderer APIs, please refer to [this page](renderer.md).
132
+
133
+- [Completions Render API](renderer.md) (`/v1/completions/render`)
134
+ - Render completion requests
135
+- [Chat Completions Render API](renderer.md) (`/v1/chat/completions/render`)
136
+ - Render chat completions
137
+
138
+### Derenderer APIs
139
+
140
+- `/v1/completions/derender` - Derenderer completion requests
141
+- `/v1/chat/completions/derender` - Derenderer chat completion requests
142
+
143
+## Tokenize APIs
144
+
145
+- `/tokenize` - Tokenize text
146
+- `/detokenize` - Detokenize tokens
147
+- `/tokenizer_info` - Get comprehensive tokenizer information including chat templates and configuration
148
+
149
+## Elastic Expert Parallelism (EEP)
150
+
151
+- `/scale_elastic_ep` - Trigger scaling operations
152
+- `/is_scaling_elastic_ep` - Check if scaling is in progress
153
154
## Server in development mode
155
@@ -120,7 +171,9 @@ For further details on Weight Transfer, please refer to [this page](../../traini
171
- `/resume` - Resume generation
172
- `/is_paused` - Check if generation is paused
173
- `/init_weight_transfer_engine` - Initialize weight transfer engine for RLHF
174
+- `/start_weight_update` - Prepares the inference engine for a weight update.
175
- `/update_weights` - Update model weights (can alter model behavior)
176
+- `/finish_weight_update` - Finalizes the weight update
177
- `/get_world_size` - Get distributed world size
178
179
### Collective RPC
@@ -189,14 +242,6 @@ the detected format, which can be one of:
242
If the result is not what you expect, you can set the `--chat-template-content-format` CLI argument
243
to override which format to use.
244
192
-## Offline API Documentation
193
-
194
-The FastAPI `/docs` endpoint requires an internet connection by default. To enable offline access in air-gapped environments, use the `--enable-offline-docs` flag:
195
-
196
-```bash
197
-vllm serve NousResearch/Meta-Llama-3-8B-Instruct --enable-offline-docs
198
-```
199
-
245
## Ray Serve LLM
246
247
Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure.
docs/serving/online_serving/openai_compatible_server.md
+3
-2
@@ -9,12 +9,13 @@ We currently support the following OpenAI APIs:
9
- [Completions API](#completions-api) (`/v1/completions`)
10
- Only applicable to [text generation models](../../models/generative_models.md).
11
- *Note: `suffix` parameter is not supported.*
12
-- [Responses API](#responses-api) (`/v1/responses`)
13
- - Only applicable to [text generation models](../../models/generative_models.md).
12
- [Chat Completions API](#chat-api) (`/v1/chat/completions`)
13
- Only applicable to [text generation models](../../models/generative_models.md) with a [chat template](../online_serving/README.md#chat-template).
14
- *Note: `user` parameter is ignored.*
15
- *Note:* Setting the `parallel_tool_calls` parameter to `false` ensures vLLM only returns zero or one tool call per request. Setting it to `true` (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to `true`, as that behavior is model dependent and not all models are designed to support parallel tool calls.
16
+- [Chat Completions batch API](#chat-api) (`/v1/chat/completions/batch`)
17
+- [Responses API](#responses-api) (`/v1/responses`, `/v1/responses/{response_id}`, `/v1/responses/{response_id}/cancel`)
18
+ - Only applicable to [text generation models](../../models/generative_models.md).
19
- [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)
20
- Only applicable to [embedding models](../../models/pooling_models/embed.md).
21
- [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`)