A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 23:45 UTC
[Docs] Switch to better markdown linting pre-commit hook (#21851)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Jul 30, 2025 at 03:45 UTC
ba5c5e5404d2d3fdee02e163fc75a44bd960935f
55 files changed
+274
-199
.buildkite/nightly-benchmarks/README.md
+5
@@ -28,6 +28,7 @@ See [vLLM performance dashboard](https://perf.vllm.ai) for the latest performanc
28
## Trigger the benchmark
29
30
Performance benchmark will be triggered when:
31
+
32
- A PR being merged into vllm.
33
- Every commit for those PRs with `perf-benchmarks` label AND `ready` label.
34
@@ -38,6 +39,7 @@ bash .buildkite/nightly-benchmarks/scripts/run-performance-benchmarks.sh
39
```
40
41
Runtime environment variables:
42
+
43
- `ON_CPU`: set the value to '1' on Intel® Xeon® Processors. Default value is 0.
44
- `SERVING_JSON`: JSON file to use for the serving tests. Default value is empty string (use default file).
45
- `LATENCY_JSON`: JSON file to use for the latency tests. Default value is empty string (use default file).
@@ -46,12 +48,14 @@ Runtime environment variables:
48
- `REMOTE_PORT`: Port for the remote vLLM service to benchmark. Default value is empty string.
49
50
Nightly benchmark will be triggered when:
51
+
52
- Every commit for those PRs with `perf-benchmarks` label and `nightly-benchmarks` label.
53
54
## Performance benchmark details
55
56
See [performance-benchmarks-descriptions.md](performance-benchmarks-descriptions.md) for detailed descriptions, and use `tests/latency-tests.json`, `tests/throughput-tests.json`, `tests/serving-tests.json` to configure the test cases.
57
> NOTE: For Intel® Xeon® Processors, use `tests/latency-tests-cpu.json`, `tests/throughput-tests-cpu.json`, `tests/serving-tests-cpu.json` instead.
58
+>
59
### Latency test
60
61
Here is an example of one test inside `latency-tests.json`:
@@ -149,6 +153,7 @@ Here is an example using the script to compare result_a and result_b without det
153
154
Here is an example using the script to compare result_a and result_b with detail test name.
155
`python3 compare-json-results.py -f results_a/benchmark_results.json -f results_b/benchmark_results.json`
156
+
157
| | results_a/benchmark_results.json_name | results_a/benchmark_results.json | results_b/benchmark_results.json_name | results_b/benchmark_results.json | perf_ratio |
158
|---|---------------------------------------------|----------------------------------------|---------------------------------------------|----------------------------------------|----------|
159
| 0 | serving_llama8B_tp1_sharegpt_qps_1 | 142.633982 | serving_llama8B_tp1_sharegpt_qps_1 | 156.526018 | 1.097396 |
.buildkite/nightly-benchmarks/nightly-annotation.md
+11
-10
@@ -1,3 +1,4 @@
1
+# Nightly benchmark annotation
2
3
## Description
4
@@ -13,15 +14,15 @@ Please download the visualization scripts in the post
14
15
- Find the docker we use in `benchmarking pipeline`
16
- Deploy the docker, and inside the docker:
16
- - Download `nightly-benchmarks.zip`.
17
- - In the same folder, run the following code:
18
-
19
- ```bash
20
- export HF_TOKEN=<your HF token>
21
- apt update
22
- apt install -y git
23
- unzip nightly-benchmarks.zip
24
- VLLM_SOURCE_CODE_LOC=./ bash .buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh
25
- ```
17
+ - Download `nightly-benchmarks.zip`.
18
+ - In the same folder, run the following code:
19
+
20
+ ```bash
21
+ export HF_TOKEN=<your HF token>
22
+ apt update
23
+ apt install -y git
24
+ unzip nightly-benchmarks.zip
25
+ VLLM_SOURCE_CODE_LOC=./ bash .buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh
26
+ ```
27
28
And the results will be inside `./benchmarks/results`.
.buildkite/nightly-benchmarks/nightly-descriptions.md
+17
-17
@@ -13,25 +13,25 @@ Latest reproduction guilde: [github issue link](https://github.com/vllm-project/
13
## Setup
14
15
- Docker images:
16
- - vLLM: `vllm/vllm-openai:v0.6.2`
17
- - SGLang: `lmsysorg/sglang:v0.3.2-cu121`
18
- - LMDeploy: `openmmlab/lmdeploy:v0.6.1-cu12`
19
- - TensorRT-LLM: `nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3`
20
- - *NOTE: we uses r24.07 as the current implementation only works for this version. We are going to bump this up.*
21
- - Check [nightly-pipeline.yaml](nightly-pipeline.yaml) for the concrete docker images, specs and commands we use for the benchmark.
16
+ - vLLM: `vllm/vllm-openai:v0.6.2`
17
+ - SGLang: `lmsysorg/sglang:v0.3.2-cu121`
18
+ - LMDeploy: `openmmlab/lmdeploy:v0.6.1-cu12`
19
+ - TensorRT-LLM: `nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3`
20
+ - *NOTE: we uses r24.07 as the current implementation only works for this version. We are going to bump this up.*
21
+ - Check [nightly-pipeline.yaml](nightly-pipeline.yaml) for the concrete docker images, specs and commands we use for the benchmark.
22
- Hardware
23
- - 8x Nvidia A100 GPUs
23
+ - 8x Nvidia A100 GPUs
24
- Workload:
25
- - Dataset
26
- - ShareGPT dataset
27
- - Prefill-heavy dataset (in average 462 input tokens, 16 tokens as output)
28
- - Decode-heavy dataset (in average 462 input tokens, 256 output tokens)
29
- - Check [nightly-tests.json](tests/nightly-tests.json) for the concrete configuration of datasets we use.
30
- - Models: llama-3 8B, llama-3 70B.
31
- - We do not use llama 3.1 as it is incompatible with trt-llm r24.07. ([issue](https://github.com/NVIDIA/TensorRT-LLM/issues/2105)).
32
- - Average QPS (query per second): 2, 4, 8, 16, 32 and inf.
33
- - Queries are randomly sampled, and arrival patterns are determined via Poisson process, but all with fixed random seed.
34
- - Evaluation metrics: Throughput (higher the better), TTFT (time to the first token, lower the better), ITL (inter-token latency, lower the better).
25
+ - Dataset
26
+ - ShareGPT dataset
27
+ - Prefill-heavy dataset (in average 462 input tokens, 16 tokens as output)
28
+ - Decode-heavy dataset (in average 462 input tokens, 256 output tokens)
29
+ - Check [nightly-tests.json](tests/nightly-tests.json) for the concrete configuration of datasets we use.
30
+ - Models: llama-3 8B, llama-3 70B.
31
+ - We do not use llama 3.1 as it is incompatible with trt-llm r24.07. ([issue](https://github.com/NVIDIA/TensorRT-LLM/issues/2105)).
32
+ - Average QPS (query per second): 2, 4, 8, 16, 32 and inf.
33
+ - Queries are randomly sampled, and arrival patterns are determined via Poisson process, but all with fixed random seed.
34
+ - Evaluation metrics: Throughput (higher the better), TTFT (time to the first token, lower the better), ITL (inter-token latency, lower the better).
35
36
## Known issues
37
.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md
+1
@@ -1,3 +1,4 @@
1
+# Performance benchmarks descriptions
2
3
## Latency tests
4
.github/PULL_REQUEST_TEMPLATE.md
+2
-2
@@ -1,4 +1,5 @@
1
-## Essential Elements of an Effective PR Description Checklist
1
+# Essential Elements of an Effective PR Description Checklist
2
+
3
- [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
4
- [ ] The test plan, such as providing test command.
5
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
@@ -14,5 +15,4 @@ PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS ABOVE HAVE B
15
16
## (Optional) Documentation Update
17
17
-<!--- pyml disable-next-line no-emphasis-as-heading -->
18
**BEFORE SUBMITTING, PLEASE READ <https://docs.vllm.ai/en/latest/contributing>** (anything written below this line will be removed by GitHub Actions)
.markdownlint.yaml
new
+13
@@ -0,0 +1,13 @@
1
+MD007:
2
+ indent: 4
3
+MD013: false
4
+MD024:
5
+ siblings_only: true
6
+MD033: false
7
+MD042: false
8
+MD045: false
9
+MD046: false
10
+MD051: false
11
+MD052: false
12
+MD053: false
13
+MD059: false
.pre-commit-config.yaml
+3
-4
@@ -35,12 +35,11 @@ repos:
35
exclude: 'csrc/(moe/topk_softmax_kernels.cu|quantization/gguf/(ggml-common.h|dequantize.cuh|vecdotq.cuh|mmq.cuh|mmvq.cuh))|vllm/third_party/.*'
36
types_or: [c++, cuda]
37
args: [--style=file, --verbose]
38
-- repo: https://github.com/jackdewinter/pymarkdown
39
- rev: v0.9.29
38
+- repo: https://github.com/igorshubovych/markdownlint-cli
39
+ rev: v0.45.0
40
hooks:
41
- - id: pymarkdown
41
+ - id: markdownlint-fix
42
exclude: '.*\.inc\.md'
43
- args: [fix]
43
- repo: https://github.com/rhysd/actionlint
44
rev: v1.7.7
45
hooks:
README.md
+7
@@ -1,3 +1,4 @@
1
+<!-- markdownlint-disable MD001 MD041 -->
2
<p align="center">
3
<picture>
4
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/assets/logos/vllm-logo-text-dark.png">
@@ -16,6 +17,7 @@ Easy, fast, and cheap LLM serving for everyone
17
---
18
19
*Latest News* 🔥
20
+
21
- [2025/05] We hosted [NYC vLLM Meetup](https://lu.ma/c1rqyf1f)! Please find the meetup slides [here](https://docs.google.com/presentation/d/1_q_aW_ioMJWUImf1s1YM-ZhjXz8cUeL0IJvaquOYBeA/edit?usp=sharing).
22
- [2025/05] vLLM is now a hosted project under PyTorch Foundation! Please find the announcement [here](https://pytorch.org/blog/pytorch-foundation-welcomes-vllm/).
23
- [2025/04] We hosted [Asia Developer Day](https://www.sginnovate.com/event/limited-availability-morning-evening-slots-remaining-inaugural-vllm-asia-developer-day)! Please find the meetup slides from the vLLM team [here](https://docs.google.com/presentation/d/19cp6Qu8u48ihB91A064XfaXruNYiBOUKrBxAmDOllOo/edit?usp=sharing).
@@ -46,6 +48,7 @@ Easy, fast, and cheap LLM serving for everyone
48
</details>
49
50
---
51
+
52
## About
53
54
vLLM is a fast and easy-to-use library for LLM inference and serving.
@@ -75,6 +78,7 @@ vLLM is flexible and easy to use with:
78
- Multi-LoRA support
79
80
vLLM seamlessly supports most popular open-source models on HuggingFace, including:
81
+
82
- Transformer-like LLMs (e.g., Llama)
83
- Mixture-of-Expert LLMs (e.g., Mixtral, Deepseek-V2 and V3)
84
- Embedding Models (e.g., E5-Mistral)
@@ -91,6 +95,7 @@ pip install vllm
95
```
96
97
Visit our [documentation](https://docs.vllm.ai/en/latest/) to learn more.
98
+
99
- [Installation](https://docs.vllm.ai/en/latest/getting_started/installation.html)
100
- [Quickstart](https://docs.vllm.ai/en/latest/getting_started/quickstart.html)
101
- [List of Supported Models](https://docs.vllm.ai/en/latest/models/supported_models.html)
@@ -107,6 +112,7 @@ vLLM is a community project. Our compute resources for development and testing a
112
<!-- Note: Please sort them in alphabetical order. -->
113
<!-- Note: Please keep these consistent with docs/community/sponsors.md -->
114
Cash Donations:
115
+
116
- a16z
117
- Dropbox
118
- Sequoia Capital
@@ -114,6 +120,7 @@ Cash Donations:
120
- ZhenFund
121
122
Compute Resources:
123
+
124
- AMD
125
- Anyscale
126
- AWS
RELEASE.md
+4
-1
@@ -60,9 +60,10 @@ Please note: **No feature work allowed for cherry picks**. All PRs that are cons
60
Before each release, we perform end-to-end performance validation to ensure no regressions are introduced. This validation uses the [vllm-benchmark workflow](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) on PyTorch CI.
61
62
**Current Coverage:**
63
+
64
* Models: Llama3, Llama4, and Mixtral
65
* Hardware: NVIDIA H100 and AMD MI300x
65
-* *Note: Coverage may change based on new model releases and hardware availability*
66
+* _Note: Coverage may change based on new model releases and hardware availability_
67
68
**Performance Validation Process:**
69
@@ -71,11 +72,13 @@ Request write access to the [pytorch/pytorch-integration-testing](https://github
72
73
**Step 2: Review Benchmark Setup**
74
Familiarize yourself with the benchmark configurations:
75
+
76
* [CUDA setup](https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks/cuda)
77
* [ROCm setup](https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks/rocm)
78
79
**Step 3: Run the Benchmark**
80
Navigate to the [vllm-benchmark workflow](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) and configure:
81
+
82
* **vLLM branch**: Set to the release branch (e.g., `releases/v0.9.2`)
83
* **vLLM commit**: Set to the RC commit hash
84
benchmarks/README.md
+56
-43
@@ -4,7 +4,7 @@ This README guides you through running benchmark tests with the extensive
4
datasets supported on vLLM. It’s a living document, updated as new features and datasets
5
become available.
6
7
-**Dataset Overview**
7
+## Dataset Overview
8
9
<table style="width:100%; border-collapse: collapse;">
10
<thead>
@@ -81,9 +81,10 @@ become available.
81
82
**Note**: HuggingFace dataset's `dataset-name` should be set to `hf`
83
84
----
84
+## 🚀 Example - Online Benchmark
85
+
86
<details>
86
-<summary><b>🚀 Example - Online Benchmark</b></summary>
87
+<summary>Show more</summary>
88
89
<br/>
90
@@ -109,7 +110,7 @@ vllm bench serve \
110
111
If successful, you will see the following output
112
112
-```
113
+```text
114
============ Serving Benchmark Result ============
115
Successful requests: 10
116
Benchmark duration (s): 5.78
@@ -133,11 +134,11 @@ P99 ITL (ms): 8.39
134
==================================================
135
```
136
136
-**Custom Dataset**
137
+### Custom Dataset
138
139
If the dataset you want to benchmark is not supported yet in vLLM, even then you can benchmark on it using `CustomDataset`. Your data needs to be in `.jsonl` format and needs to have "prompt" field per entry, e.g., data.jsonl
140
140
-```
141
+```json
142
{"prompt": "What is the capital of India?"}
143
{"prompt": "What is the capital of Iran?"}
144
{"prompt": "What is the capital of China?"}
@@ -166,7 +167,7 @@ vllm bench serve --port 9001 --save-result --save-detailed \
167
168
You can skip applying chat template if your data already has it by using `--custom-skip-chat-template`.
169
169
-**VisionArena Benchmark for Vision Language Models**
170
+### VisionArena Benchmark for Vision Language Models
171
172
```bash
173
# need a model with vision capability here
@@ -184,7 +185,7 @@ vllm bench serve \
185
--num-prompts 1000
186
```
187
187
-**InstructCoder Benchmark with Speculative Decoding**
188
+### InstructCoder Benchmark with Speculative Decoding
189
190
``` bash
191
VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
@@ -201,13 +202,13 @@ vllm bench serve \
202
--num-prompts 2048
203
```
204
204
-**Other HuggingFaceDataset Examples**
205
+### Other HuggingFaceDataset Examples
206
207
```bash
208
vllm serve Qwen/Qwen2-VL-7B-Instruct --disable-log-requests
209
```
210
210
-**`lmms-lab/LLaVA-OneVision-Data`**
211
+`lmms-lab/LLaVA-OneVision-Data`:
212
213
```bash
214
vllm bench serve \
@@ -221,7 +222,7 @@ vllm bench serve \
222
--num-prompts 10
223
```
224
224
-**`Aeala/ShareGPT_Vicuna_unfiltered`**
225
+`Aeala/ShareGPT_Vicuna_unfiltered`:
226
227
```bash
228
vllm bench serve \
@@ -234,7 +235,7 @@ vllm bench serve \
235
--num-prompts 10
236
```
237
237
-**`AI-MO/aimo-validation-aime`**
238
+`AI-MO/aimo-validation-aime`:
239
240
``` bash
241
vllm bench serve \
@@ -245,7 +246,7 @@ vllm bench serve \
246
--seed 42
247
```
248
248
-**`philschmid/mt-bench`**
249
+`philschmid/mt-bench`:
250
251
``` bash
252
vllm bench serve \
@@ -255,7 +256,7 @@ vllm bench serve \
256
--num-prompts 80
257
```
258
258
-**Running With Sampling Parameters**
259
+### Running With Sampling Parameters
260
261
When using OpenAI-compatible backends such as `vllm`, optional sampling
262
parameters can be specified. Example client command:
@@ -273,25 +274,29 @@ vllm bench serve \
274
--num-prompts 10
275
```
276
276
-**Running With Ramp-Up Request Rate**
277
+### Running With Ramp-Up Request Rate
278
279
The benchmark tool also supports ramping up the request rate over the
280
duration of the benchmark run. This can be useful for stress testing the
281
server or finding the maximum throughput that it can handle, given some latency budget.
282
283
Two ramp-up strategies are supported:
284
+
285
- `linear`: Increases the request rate linearly from a start value to an end value.
286
- `exponential`: Increases the request rate exponentially.
287
288
The following arguments can be used to control the ramp-up:
289
+
290
- `--ramp-up-strategy`: The ramp-up strategy to use (`linear` or `exponential`).
291
- `--ramp-up-start-rps`: The request rate at the beginning of the benchmark.
292
- `--ramp-up-end-rps`: The request rate at the end of the benchmark.
293
294
</details>
295
296
+## 📈 Example - Offline Throughput Benchmark
297
+
298
<details>
294
-<summary><b>📈 Example - Offline Throughput Benchmark</b></summary>
299
+<summary>Show more</summary>
300
301
<br/>
302
@@ -305,15 +310,15 @@ vllm bench throughput \
310
311
If successful, you will see the following output
312
308
-```
313
+```text
314
Throughput: 7.15 requests/s, 4656.00 total tokens/s, 1072.15 output tokens/s
315
Total num prompt tokens: 5014
316
Total num output tokens: 1500
317
```
318
314
-**VisionArena Benchmark for Vision Language Models**
319
+### VisionArena Benchmark for Vision Language Models
320
316
-``` bash
321
+```bash
322
vllm bench throughput \
323
--model Qwen/Qwen2-VL-7B-Instruct \
324
--backend vllm-chat \
@@ -325,13 +330,13 @@ vllm bench throughput \
330
331
The `num prompt tokens` now includes image token counts
332
328
-```
333
+```text
334
Throughput: 2.55 requests/s, 4036.92 total tokens/s, 326.90 output tokens/s
335
Total num prompt tokens: 14527
336
Total num output tokens: 1280
337
```
338
334
-**InstructCoder Benchmark with Speculative Decoding**
339
+### InstructCoder Benchmark with Speculative Decoding
340
341
``` bash
342
VLLM_WORKER_MULTIPROC_METHOD=spawn \
@@ -349,15 +354,15 @@ vllm bench throughput \
354
"prompt_lookup_min": 2}'
355
```
356
352
-```
357
+```text
358
Throughput: 104.77 requests/s, 23836.22 total tokens/s, 10477.10 output tokens/s
359
Total num prompt tokens: 261136
360
Total num output tokens: 204800
361
```
362
358
-**Other HuggingFaceDataset Examples**
363
+### Other HuggingFaceDataset Examples
364
360
-**`lmms-lab/LLaVA-OneVision-Data`**
365
+`lmms-lab/LLaVA-OneVision-Data`:
366
367
```bash
368
vllm bench throughput \
@@ -370,7 +375,7 @@ vllm bench throughput \
375
--num-prompts 10
376
```
377
373
-**`Aeala/ShareGPT_Vicuna_unfiltered`**
378
+`Aeala/ShareGPT_Vicuna_unfiltered`:
379
380
```bash
381
vllm bench throughput \
@@ -382,7 +387,7 @@ vllm bench throughput \
387
--num-prompts 10
388
```
389
385
-**`AI-MO/aimo-validation-aime`**
390
+`AI-MO/aimo-validation-aime`:
391
392
```bash
393
vllm bench throughput \
@@ -394,7 +399,7 @@ vllm bench throughput \
399
--num-prompts 10
400
```
401
397
-**Benchmark with LoRA Adapters**
402
+Benchmark with LoRA adapters:
403
404
``` bash
405
# download dataset
@@ -413,20 +418,22 @@ vllm bench throughput \
418
419
</details>
420
421
+## 🛠️ Example - Structured Output Benchmark
422
+
423
<details>
417
-<summary><b>🛠️ Example - Structured Output Benchmark</b></summary>
424
+<summary>Show more</summary>
425
426
<br/>
427
428
Benchmark the performance of structured output generation (JSON, grammar, regex).
429
423
-**Server Setup**
430
+### Server Setup
431
432
```bash
433
vllm serve NousResearch/Hermes-3-Llama-3.1-8B --disable-log-requests
434
```
435
429
-**JSON Schema Benchmark**
436
+### JSON Schema Benchmark
437
438
```bash
439
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -438,7 +445,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
445
--num-prompts 1000
446
```
447
441
-**Grammar-based Generation Benchmark**
448
+### Grammar-based Generation Benchmark
449
450
```bash
451
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -450,7 +457,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
457
--num-prompts 1000
458
```
459
453
-**Regex-based Generation Benchmark**
460
+### Regex-based Generation Benchmark
461
462
```bash
463
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -461,7 +468,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
468
--num-prompts 1000
469
```
470
464
-**Choice-based Generation Benchmark**
471
+### Choice-based Generation Benchmark
472
473
```bash
474
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -472,7 +479,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
479
--num-prompts 1000
480
```
481
475
-**XGrammar Benchmark Dataset**
482
+### XGrammar Benchmark Dataset
483
484
```bash
485
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -485,14 +492,16 @@ python3 benchmarks/benchmark_serving_structured_output.py \
492
493
</details>
494
495
+## 📚 Example - Long Document QA Benchmark
496
+
497
<details>
489
-<summary><b>📚 Example - Long Document QA Benchmark</b></summary>
498
+<summary>Show more</summary>
499
500
<br/>
501
502
Benchmark the performance of long document question-answering with prefix caching.
503
495
-**Basic Long Document QA Test**
504
+### Basic Long Document QA Test
505
506
```bash
507
python3 benchmarks/benchmark_long_document_qa_throughput.py \
@@ -504,7 +513,7 @@ python3 benchmarks/benchmark_long_document_qa_throughput.py \
513
--repeat-count 5
514
```
515
507
-**Different Repeat Modes**
516
+### Different Repeat Modes
517
518
```bash
519
# Random mode (default) - shuffle prompts randomly
@@ -537,14 +546,16 @@ python3 benchmarks/benchmark_long_document_qa_throughput.py \
546
547
</details>
548
549
+## 🗂️ Example - Prefix Caching Benchmark
550
+
551
<details>
541
-<summary><b>🗂️ Example - Prefix Caching Benchmark</b></summary>
552
+<summary>Show more</summary>
553
554
<br/>
555
556
Benchmark the efficiency of automatic prefix caching.
557
547
-**Fixed Prompt with Prefix Caching**
558
+### Fixed Prompt with Prefix Caching
559
560
```bash
561
python3 benchmarks/benchmark_prefix_caching.py \
@@ -555,7 +566,7 @@ python3 benchmarks/benchmark_prefix_caching.py \
566
--input-length-range 128:256
567
```
568
558
-**ShareGPT Dataset with Prefix Caching**
569
+### ShareGPT Dataset with Prefix Caching
570
571
```bash
572
# download dataset
@@ -572,14 +583,16 @@ python3 benchmarks/benchmark_prefix_caching.py \
583
584
</details>
585
586
+## ⚡ Example - Request Prioritization Benchmark
587
+
588
<details>
576
-<summary><b>⚡ Example - Request Prioritization Benchmark</b></summary>
589
+<summary>Show more</summary>
590
591
<br/>
592
593
Benchmark the performance of request prioritization in vLLM.
594
582
-**Basic Prioritization Test**
595
+### Basic Prioritization Test
596
597
```bash
598
python3 benchmarks/benchmark_prioritization.py \
@@ -590,7 +603,7 @@ python3 benchmarks/benchmark_prioritization.py \
603
--scheduling-policy priority
604
```
605
593
-**Multiple Sequences per Prompt**
606
+### Multiple Sequences per Prompt
607
608
```bash
609
python3 benchmarks/benchmark_prioritization.py \
benchmarks/auto_tune/README.md
+6
-2
@@ -3,6 +3,7 @@
3
This script automates the process of finding the optimal server parameter combination (`max-num-seqs` and `max-num-batched-tokens`) to maximize throughput for a vLLM server. It also supports additional constraints such as E2E latency and prefix cache hit rate.
4
5
## Table of Contents
6
+
7
- [Prerequisites](#prerequisites)
8
- [Configuration](#configuration)
9
- [How to Run](#how-to-run)
@@ -52,7 +53,7 @@ You must set the following variables at the top of the script before execution.
53
1. **Configure**: Edit the script and set the variables in the [Configuration](#configuration) section.
54
2. **Execute**: Run the script. Since the process can take a long time, it is highly recommended to use a terminal multiplexer like `tmux` or `screen` to prevent the script from stopping if your connection is lost.
55
55
-```
56
+```bash
57
cd <FOLDER_OF_THIS_SCRIPT>
58
bash auto_tune.sh
59
```
@@ -64,6 +65,7 @@ bash auto_tune.sh
65
Here are a few examples of how to configure the script for different goals:
66
67
### 1. Maximize Throughput (No Latency Constraint)
68
+
69
- **Goal**: Find the best `max-num-seqs` and `max-num-batched-tokens` to get the highest possible throughput for 1800 input tokens and 20 output tokens.
70
- **Configuration**:
71
@@ -76,6 +78,7 @@ MAX_LATENCY_ALLOWED_MS=100000000000 # A very large number
78
```
79
80
#### 2. Maximize Throughput with a Latency Requirement
81
+
82
- **Goal**: Find the best server parameters when P99 end-to-end latency must be below 500ms.
83
- **Configuration**:
84
@@ -88,6 +91,7 @@ MAX_LATENCY_ALLOWED_MS=500
91
```
92
93
#### 3. Maximize Throughput with Prefix Caching and Latency Requirements
94
+
95
- **Goal**: Find the best server parameters assuming a 60% prefix cache hit rate and a latency requirement of 500ms.
96
- **Configuration**:
97
@@ -109,7 +113,7 @@ After the script finishes, you will find the results in a new, timestamped direc
113
114
- **Final Result Summary**: A file named `result.txt` is created in the log directory. It contains a summary of each tested combination and concludes with the overall best parameters found.
115
112
-```
116
+```text
117
# Example result.txt content
118
hash:a1b2c3d4...
119
max_num_seqs: 128, max_num_batched_tokens: 2048, request_rate: 10.0, e2el: 450.5, throughput: 9.8, goodput: 9.8
benchmarks/kernels/deepgemm/README.md
+2
-2
@@ -8,7 +8,7 @@ Currently this just includes dense GEMMs and only works on Hopper GPUs.
8
9
You need to install vLLM in your usual fashion, then install DeepGEMM from source in its own directory:
10
11
-```
11
+```bash
12
git clone --recursive https://github.com/deepseek-ai/DeepGEMM
13
cd DeepGEMM
14
python setup.py install
@@ -17,7 +17,7 @@ uv pip install -e .
17
18
## Usage
19
20
-```
20
+```console
21
python benchmark_fp8_block_dense_gemm.py
22
INFO 02-26 21:55:13 [__init__.py:207] Automatically detected platform cuda.
23
===== STARTING FP8 GEMM BENCHMARK =====
csrc/quantization/cutlass_w8a8/Epilogues.md
+3
-2
@@ -86,6 +86,7 @@ D = s_a s_b \widehat A \widehat B
86
```
87
88
Epilogue parameters:
89
+
90
- `scale_a` is the scale for activations, can be per-tensor (scalar) or per-token (column-vector).
91
- `scale_b` is the scale for weights, can be per-tensor (scalar) or per-channel (row-vector).
92
@@ -135,7 +136,7 @@ That is precomputed and stored in `azp_with_adj` as a row-vector.
136
Epilogue parameters:
137
138
- `scale_a` is the scale for activations, can be per-tensor (scalar) or per-token (column-vector).
138
- - Generally this will be per-tensor as the zero-points are per-tensor.
139
+ - Generally this will be per-tensor as the zero-points are per-tensor.
140
- `scale_b` is the scale for weights, can be per-tensor (scalar) or per-channel (row-vector).
141
- `azp_with_adj` is the precomputed zero-point term ($` z_a J_a \widehat B `$), is per-channel (row-vector).
142
- `bias` is the bias, is always per-channel (row-vector).
@@ -152,7 +153,7 @@ That means the zero-point term $` z_a J_a \widehat B `$ becomes an outer product
153
Epilogue parameters:
154
155
- `scale_a` is the scale for activations, can be per-tensor (scalar) or per-token (column-vector).
155
- - Generally this will be per-token as the zero-points are per-token.
156
+ - Generally this will be per-token as the zero-points are per-token.
157
- `scale_b` is the scale for weights, can be per-tensor (scalar) or per-channel (row-vector).
158
- `azp_adj` is the precomputed zero-point adjustment term ($` \mathbf 1 \widehat B `$), is per-channel (row-vector).
159
- `azp` is the zero-point (`z_a`), is per-token (column-vector).
docs/cli/README.md
+2
-2
@@ -6,13 +6,13 @@ toc_depth: 4
6
7
The vllm command-line tool is used to run and manage vLLM models. You can start by viewing the help message with:
8
9
-```
9
+```bash
10
vllm --help
11
```
12
13
Available Commands:
14
15
-```
15
+```bash
16
vllm {chat,complete,serve,bench,collect-env,run-batch}
17
```
18
docs/configuration/tpu.md
+11
-4
@@ -40,6 +40,7 @@ Although the first compilation can take some time, for all subsequent server lau
40
Use `VLLM_XLA_CACHE_PATH` environment variable to write to shareable storage for future deployed nodes (like when using autoscaling).
41
42
#### Reducing compilation time
43
+
44
This initial compilation time ranges significantly and is impacted by many of the arguments discussed in this optimization doc. Factors that influence the length of time to compile are things like model size and `--max-num-batch-tokens`. Other arguments you can tune are things like `VLLM_TPU_MOST_MODEL_LEN`.
45
46
### Optimize based on your data
@@ -71,12 +72,15 @@ The fewer tokens we pad, the less unnecessary computation TPU does, the better p
72
73
However, you need to be careful to choose the padding gap. If the gap is too small, it means the number of buckets is large, leading to increased warmup (precompile) time and higher memory to store the compiled graph. Too many compilaed graphs may lead to HBM OOM. Conversely, an overly large gap yields no performance improvement compared to the default exponential padding.
74
74
-**If possible, use the precision that matches the chip’s hardware acceleration**
75
+#### Quantization
76
+
77
+If possible, use the precision that matches the chip’s hardware acceleration:
78
79
- v5e has int4/int8 hardware acceleration in the MXU
80
- v6e has int4/int8 hardware acceleration in the MXU
81
79
-Supported quantized formats and features in vLLM on TPU [Jul '25]
82
+Supported quantized formats and features in vLLM on TPU [Jul '25]:
83
+
84
- INT8 W8A8
85
- INT8 W8A16
86
- FP8 KV cache
@@ -84,11 +88,13 @@ Supported quantized formats and features in vLLM on TPU [Jul '25]
88
- [WIP] AWQ
89
- [WIP] FP4 W4A8
90
87
-**Don't set TP to be less than the number of chips on a single-host deployment**
91
+#### Parallelization
92
+
93
+Don't set TP to be less than the number of chips on a single-host deployment.
94
95
Although it’s common to do this with GPUs, don't try to fragment 2 or 8 different workloads across 8 chips on a single host. If you need 1 or 4 chips, just create an instance with 1 or 4 chips (these are partial-host machine types).
96
91
-### Tune your workloads!
97
+### Tune your workloads
98
99
Although we try to have great default configs, we strongly recommend you check out the [vLLM auto-tuner](../../benchmarks/auto_tune/README.md) to optimize your workloads for your use case.
100
@@ -99,6 +105,7 @@ Although we try to have great default configs, we strongly recommend you check o
105
The auto-tuner provides a profile of optimized configurations as its final step. However, interpreting this profile can be challenging for new users. We plan to expand this section in the future with more detailed guidance. In the meantime, you can learn how to collect a TPU profile using vLLM's native profiling tools [here](../examples/offline_inference/profiling_tpu.md). This profile can provide valuable insights into your workload's performance.
106
107
#### SPMD
108
+
109
More details to come.
110
111
**Want us to cover something that isn't listed here? Open up an issue please and cite this doc. We'd love to hear your questions or tips.**
docs/contributing/ci/failures.md
+4
-4
@@ -20,19 +20,19 @@ the failure?
20
21
- **Use this title format:**
22
23
- ```
23
+ ```text
24
[CI Failure]: failing-test-job - regex/matching/failing:test
25
```
26
27
- **For the environment field:**
28
29
- ```
30
- Still failing on main as of commit abcdef123
29
+ ```text
30
+ Still failing on main as of commit abcdef123
31
```
32
33
- **In the description, include failing tests:**
34
35
- ```
35
+ ```text
36
FAILED failing/test.py:failing_test1 - Failure description
37
FAILED failing/test.py:failing_test2 - Failure description
38
https://github.com/orgs/vllm-project/projects/20
docs/contributing/ci/update_pytorch_version.md
+3
-1
@@ -106,6 +106,7 @@ releases (which would take too much time), they can be built from
106
source to unblock the update process.
107
108
### FlashInfer
109
+
110
Here is how to build and install it from source with `torch2.7.0+cu128` in vLLM [Dockerfile](https://github.com/vllm-project/vllm/blob/27bebcd89792d5c4b08af7a65095759526f2f9e1/docker/Dockerfile#L259-L271):
111
112
```bash
@@ -121,6 +122,7 @@ public location for immediate installation, such as [this FlashInfer wheel link]
122
team if you want to get the package published there.
123
124
### xFormers
125
+
126
Similar to FlashInfer, here is how to build and install xFormers from source:
127
128
```bash
@@ -138,7 +140,7 @@ uv pip install --system \
140
141
### causal-conv1d
142
141
-```
143
+```bash
144
uv pip install 'git+https://github.com/Dao-AILab/causal-conv1d@v1.5.0.post8'
145
```
146
docs/contributing/deprecation_policy.md
+3
-3
@@ -31,7 +31,7 @@ Features that fall under this policy include (at a minimum) the following:
31
The deprecation process consists of several clearly defined stages that span
32
multiple Y releases:
33
34
-**1. Deprecated (Still On By Default)**
34
+### 1. Deprecated (Still On By Default)
35
36
- **Action**: Feature is marked as deprecated.
37
- **Timeline**: A removal version is explicitly stated in the deprecation
@@ -46,7 +46,7 @@ warning (e.g., "This will be removed in v0.10.0").
46
- GitHub Issue (RFC) for feedback
47
- Documentation and use of the `@typing_extensions.deprecated` decorator for Python APIs
48
49
-**2.Deprecated (Off By Default)**
49
+### 2.Deprecated (Off By Default)
50
51
- **Action**: Feature is disabled by default, but can still be re-enabled via a
52
CLI flag or environment variable. Feature throws an error when used without
@@ -55,7 +55,7 @@ re-enabling.
55
while signaling imminent removal. Ensures any remaining usage is clearly
56
surfaced and blocks silent breakage before full removal.
57
58
-**3. Removed**
58
+### 3. Removed
59
60
- **Action**: Feature is completely removed from the codebase.
61
- **Note**: Only features that have passed through the previous deprecation
docs/contributing/profiling.md
+2
-2
@@ -112,13 +112,13 @@ vllm bench serve \
112
113
In practice, you should set the `--duration` argument to a large value. Whenever you want the server to stop profiling, run:
114
115
-```
115
+```bash
116
nsys sessions list
117
```
118
119
to get the session id in the form of `profile-XXXXX`, then run:
120
121
-```
121
+```bash
122
nsys stop --session=profile-XXXXX
123
```
124
docs/contributing/vulnerability_management.md
+3
-3
@@ -32,9 +32,9 @@ We prefer to keep all vulnerability-related communication on the security report
32
on GitHub. However, if you need to contact the VMT directly for an urgent issue,
33
you may contact the following individuals:
34
35
-- Simon Mo - simon.mo@hey.com
36
-- Russell Bryant - rbryant@redhat.com
37
-- Huzaifa Sidhpurwala - huzaifas@redhat.com
35
+- Simon Mo - <simon.mo@hey.com>
36
+- Russell Bryant - <rbryant@redhat.com>
37
+- Huzaifa Sidhpurwala - <huzaifas@redhat.com>
38
39
## Slack Discussion
40
docs/deployment/frameworks/anything-llm.md
+6
-6
@@ -19,9 +19,9 @@ vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096
19
- Download and install [Anything LLM desktop](https://anythingllm.com/desktop).
20
21
- On the bottom left of open settings, AI Prooviders --> LLM:
22
- - LLM Provider: Generic OpenAI
23
- - Base URL: http://{vllm server host}:{vllm server port}/v1
24
- - Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`
22
+ - LLM Provider: Generic OpenAI
23
+ - Base URL: http://{vllm server host}:{vllm server port}/v1
24
+ - Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`
25
26
27
@@ -30,9 +30,9 @@ vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096
30
31
32
- Click the upload button:
33
- - upload the doc
34
- - select the doc and move to the workspace
35
- - save and embed
33
+ - upload the doc
34
+ - select the doc and move to the workspace
35
+ - save and embed
36
37
38
docs/deployment/frameworks/chatbox.md
+5
-5
@@ -19,11 +19,11 @@ vllm serve qwen/Qwen1.5-0.5B-Chat
19
- Download and install [Chatbox desktop](https://chatboxai.app/en#download).
20
21
- On the bottom left of settings, Add Custom Provider
22
- - API Mode: `OpenAI API Compatible`
23
- - Name: vllm
24
- - API Host: `http://{vllm server host}:{vllm server port}/v1`
25
- - API Path: `/chat/completions`
26
- - Model: `qwen/Qwen1.5-0.5B-Chat`
22
+ - API Mode: `OpenAI API Compatible`
23
+ - Name: vllm
24
+ - API Host: `http://{vllm server host}:{vllm server port}/v1`
25
+ - API Path: `/chat/completions`
26
+ - Model: `qwen/Qwen1.5-0.5B-Chat`
27
28
29
docs/deployment/frameworks/dify.md
+5
-5
@@ -34,11 +34,11 @@ docker compose up -d
34
- In the top-right user menu (under the profile icon), go to Settings, then click `Model Provider`, and locate the `vLLM` provider to install it.
35
36
- Fill in the model provider details as follows:
37
- - **Model Type**: `LLM`
38
- - **Model Name**: `Qwen/Qwen1.5-7B-Chat`
39
- - **API Endpoint URL**: `http://{vllm_server_host}:{vllm_server_port}/v1`
40
- - **Model Name for API Endpoint**: `Qwen/Qwen1.5-7B-Chat`
41
- - **Completion Mode**: `Completion`
37
+ - **Model Type**: `LLM`
38
+ - **Model Name**: `Qwen/Qwen1.5-7B-Chat`
39
+ - **API Endpoint URL**: `http://{vllm_server_host}:{vllm_server_port}/v1`
40
+ - **Model Name for API Endpoint**: `Qwen/Qwen1.5-7B-Chat`
41
+ - **Completion Mode**: `Completion`
42
43
44
docs/deployment/frameworks/haystack.md
-2
@@ -1,7 +1,5 @@
1
# Haystack
2
3
-# Haystack
4
-
3
[Haystack](https://github.com/deepset-ai/haystack) is an end-to-end LLM framework that allows you to build applications powered by LLMs, Transformer models, vector search and more. Whether you want to perform retrieval-augmented generation (RAG), document search, question answering or answer generation, Haystack can orchestrate state-of-the-art embedding models and LLMs into pipelines to build end-to-end NLP applications and solve your use case.
4
5
It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints.
docs/deployment/frameworks/retrieval_augmented_generation.md
+1
@@ -3,6 +3,7 @@
3
[Retrieval-augmented generation (RAG)](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) is a technique that enables generative artificial intelligence (Gen AI) models to retrieve and incorporate new information. It modifies interactions with a large language model (LLM) so that the model responds to user queries with reference to a specified set of documents, using this information to supplement information from its pre-existing training data. This allows LLMs to use domain-specific and/or updated information. Use cases include providing chatbot access to internal company data or generating responses based on authoritative sources.
4
5
Here are the integrations:
6
+
7
- vLLM + [langchain](https://github.com/langchain-ai/langchain) + [milvus](https://github.com/milvus-io/milvus)
8
- vLLM + [llamaindex](https://github.com/run-llama/llama_index) + [milvus](https://github.com/milvus-io/milvus)
9
docs/deployment/integrations/production-stack.md
+5
-4
@@ -140,11 +140,12 @@ The core vLLM production stack configuration is managed with YAML. Here is the e
140
```
141
142
In this YAML configuration:
143
+
144
* **`modelSpec`** includes:
144
- * `name`: A nickname that you prefer to call the model.
145
- * `repository`: Docker repository of vLLM.
146
- * `tag`: Docker image tag.
147
- * `modelURL`: The LLM model that you want to use.
145
+ * `name`: A nickname that you prefer to call the model.
146
+ * `repository`: Docker repository of vLLM.
147
+ * `tag`: Docker image tag.
148
+ * `modelURL`: The LLM model that you want to use.
149
* **`replicaCount`**: Number of replicas.
150
* **`requestCPU` and `requestMemory`**: Specifies the CPU and memory resource requests for the pod.
151
* **`requestGPU`**: Specifies the number of GPUs required.
docs/deployment/k8s.md
+1
-1
@@ -5,7 +5,7 @@ Deploying vLLM on Kubernetes is a scalable and efficient way to serve machine le
5
- [Deployment with CPUs](#deployment-with-cpus)
6
- [Deployment with GPUs](#deployment-with-gpus)
7
- [Troubleshooting](#troubleshooting)
8
- - [Startup Probe or Readiness Probe Failure, container log contains "KeyboardInterrupt: terminated"](#startup-probe-or-readiness-probe-failure-container-log-contains-keyboardinterrupt-terminated)
8
+ - [Startup Probe or Readiness Probe Failure, container log contains "KeyboardInterrupt: terminated"](#startup-probe-or-readiness-probe-failure-container-log-contains-keyboardinterrupt-terminated)
9
- [Conclusion](#conclusion)
10
11
Alternatively, you can deploy vLLM to Kubernetes using any of the following:
docs/design/metrics.md
+2
-2
@@ -361,7 +361,7 @@ instances in Prometheus.
361
362
We use this concept for the `vllm:cache_config_info` metric:
363
364
-```
364
+```text
365
# HELP vllm:cache_config_info Information of the LLMEngine CacheConfig
366
# TYPE vllm:cache_config_info gauge
367
vllm:cache_config_info{block_size="16",cache_dtype="auto",calculate_kv_scales="False",cpu_offload_gb="0",enable_prefix_caching="False",gpu_memory_utilization="0.9",...} 1.0
@@ -686,7 +686,7 @@ documentation for this option states:
686
The metrics were added by <gh-pr:7089> and who up in an OpenTelemetry trace
687
as:
688
689
-```
689
+```text
690
-> gen_ai.latency.time_in_scheduler: Double(0.017550230026245117)
691
-> gen_ai.latency.time_in_model_forward: Double(3.151565277099609)
692
-> gen_ai.latency.time_in_model_execute: Double(3.6468167304992676)
docs/design/p2p_nccl_connector.md
+3
-1
@@ -5,6 +5,7 @@ An implementation of xPyD with dynamic scaling based on point-to-point communica
5
## Detailed Design
6
7
### Overall Process
8
+
9
As shown in Figure 1, the overall process of this **PD disaggregation** solution is described through a request flow:
10
11
1. The client sends an HTTP request to the Proxy/Router's `/v1/completions` interface.
@@ -23,7 +24,7 @@ A simple HTTP service acts as the entry point for client requests and starts a b
24
25
The Proxy/Router is responsible for selecting 1P1D based on the characteristics of the client request, such as the prompt, and generating a corresponding `request_id`, for example:
26
26
-```
27
+```text
28
cmpl-___prefill_addr_10.0.1.2:21001___decode_addr_10.0.1.3:22001_93923d63113b4b338973f24d19d4bf11-0
29
```
30
@@ -70,6 +71,7 @@ pip install "vllm>=0.9.2"
71
## Run xPyD
72
73
### Instructions
74
+
75
- The following examples are run on an A800 (80GB) device, using the Meta-Llama-3.1-8B-Instruct model.
76
- Pay attention to the setting of the `kv_buffer_size` (in bytes). The empirical value is 10% of the GPU memory size. This is related to the kvcache size. If it is too small, the GPU memory buffer for temporarily storing the received kvcache will overflow, causing the kvcache to be stored in the tensor memory pool, which increases latency. If it is too large, the kvcache available for inference will be reduced, leading to a smaller batch size and decreased throughput.
77
- For Prefill instances, when using non-GET mode, the `kv_buffer_size` can be set to 1, as Prefill currently does not need to receive kvcache. However, when using GET mode, a larger `kv_buffer_size` is required because it needs to store the kvcache sent to the D instance.
docs/design/prefix_caching.md
+7
-4
@@ -18,10 +18,12 @@ In the example above, the KV cache in the first block can be uniquely identified
18
* Block tokens: A tuple of tokens in this block. The reason to include the exact tokens is to reduce potential hash value collision.
19
* Extra hashes: Other values required to make this block unique, such as LoRA IDs, multi-modality input hashes (see the example below), and cache salts to isolate caches in multi-tenant environments.
20
21
-> **Note 1:** We only cache full blocks.
21
+!!! note "Note 1"
22
+ We only cache full blocks.
23
23
-> **Note 2:** The above hash key structure is not 100% collision free. Theoretically it’s still possible for the different prefix tokens to have the same hash value. To avoid any hash collisions **in a multi-tenant setup, we advise to use SHA256** as hash function instead of the default builtin hash.
24
-SHA256 is supported since vLLM v0.8.3 and must be enabled with a command line argument. It comes with a performance impact of about 100-200ns per token (~6ms for 50k tokens of context).
24
+!!! note "Note 2"
25
+ The above hash key structure is not 100% collision free. Theoretically it’s still possible for the different prefix tokens to have the same hash value. To avoid any hash collisions **in a multi-tenant setup, we advise to use SHA256** as hash function instead of the default builtin hash.
26
+ SHA256 is supported since vLLM v0.8.3 and must be enabled with a command line argument. It comes with a performance impact of about 100-200ns per token (~6ms for 50k tokens of context).
27
28
**A hashing example with multi-modality inputs**
29
In this example, we illustrate how prefix caching works with multi-modality inputs (e.g., images). Assuming we have a request with the following messages:
@@ -92,7 +94,8 @@ To improve privacy in shared environments, vLLM supports isolating prefix cache
94
95
With this setup, cache sharing is limited to users or requests that explicitly agree on a common salt, enabling cache reuse within a trust group while isolating others.
96
95
-> **Note:** Cache isolation is not supported in engine V0.
97
+!!! note
98
+ Cache isolation is not supported in engine V0.
99
100
## Data Structure
101
docs/design/torch_compile.md
+3
-3
@@ -8,7 +8,7 @@ Throughout the example, we will run a common Llama model using v1, and turn on d
8
9
In the very verbose logs, we can see:
10
11
-```
11
+```console
12
INFO 03-07 03:06:55 [backends.py:409] Using cache directory: ~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0 for vLLM's torch.compile
13
```
14
@@ -75,7 +75,7 @@ Every submodule can be identified by its index, and will be processed individual
75
76
In the very verbose logs, we can also see:
77
78
-```
78
+```console
79
DEBUG 03-07 03:52:37 [backends.py:134] store the 0-th graph for shape None from inductor via handle ('fpegyiq3v3wzjzphd45wkflpabggdbjpylgr7tta4hj6uplstsiw', '~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/inductor_cache/iw/ciwzrk3ittdqatuzwonnajywvno3llvjcs2vfdldzwzozn3zi3iy.py')
80
DEBUG 03-07 03:52:39 [backends.py:134] store the 1-th graph for shape None from inductor via handle ('f7fmlodmf3h3by5iiu2c4zarwoxbg4eytwr3ujdd2jphl4pospfd', '~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/inductor_cache/ly/clyfzxldfsj7ehaluis2mca2omqka4r7mgcedlf6xfjh645nw6k2.py')
81
...
@@ -93,7 +93,7 @@ One more detail: you can see that the 1-th graph and the 15-th graph have the sa
93
94
If we already have the cache directory (e.g. run the same code for the second time), we will see the following logs:
95
96
-```
96
+```console
97
DEBUG 03-07 04:00:45 [backends.py:86] Directly load the 0-th graph for shape None from inductor via handle ('fpegyiq3v3wzjzphd45wkflpabggdbjpylgr7tta4hj6uplstsiw', '~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/inductor_cache/iw/ciwzrk3ittdqatuzwonnajywvno3llvjcs2vfdldzwzozn3zi3iy.py')
98
```
99
docs/features/compatibility_matrix.md
+3
-3
@@ -36,9 +36,9 @@ th:not(:first-child) {
36
37
| Feature | [CP][chunked-prefill] | [APC](automatic_prefix_caching.md) | [LoRA](lora.md) | [SD](spec_decode.md) | CUDA graph | [pooling](../models/pooling_models.md) | <abbr title="Encoder-Decoder Models">enc-dec</abbr> | <abbr title="Logprobs">logP</abbr> | <abbr title="Prompt Logprobs">prmpt logP</abbr> | <abbr title="Async Output Processing">async output</abbr> | multi-step | <abbr title="Multimodal Inputs">mm</abbr> | best-of | beam-search |
38
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
39
-| [CP][chunked-prefill] | ✅ | | | | | | | | | | | | | | |
40
-| [APC](automatic_prefix_caching.md) | ✅ | ✅ | | | | | | | | | | | | | |
41
-| [LoRA](lora.md) | ✅ | ✅ | ✅ | | | | | | | | | | | | |
39
+| [CP][chunked-prefill] | ✅ | | | | | | | | | | | | | |
40
+| [APC](automatic_prefix_caching.md) | ✅ | ✅ | | | | | | | | | | | | |
41
+| [LoRA](lora.md) | ✅ | ✅ | ✅ | | | | | | | | | | | |
42
| [SD](spec_decode.md) | ✅ | ✅ | ❌ | ✅ | | | | | | | | | | |
43
| CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | | | |
44
| [pooling](../models/pooling_models.md) | ✅\* | ✅\* | ✅ | ❌ | ✅ | ✅ | | | | | | | | |
docs/features/lora.md
+2
@@ -119,6 +119,7 @@ export VLLM_ALLOW_RUNTIME_LORA_UPDATING=True
119
```
120
121
### Using API Endpoints
122
+
123
Loading a LoRA Adapter:
124
125
To dynamically load a LoRA adapter, send a POST request to the `/v1/load_lora_adapter` endpoint with the necessary
@@ -156,6 +157,7 @@ curl -X POST http://localhost:8000/v1/unload_lora_adapter \
157
```
158
159
### Using Plugins
160
+
161
Alternatively, you can use the LoRAResolver plugin to dynamically load LoRA adapters. LoRAResolver plugins enable you to load LoRA adapters from both local and remote sources such as local file system and S3. On every request, when there's a new model name that hasn't been loaded yet, the LoRAResolver will try to resolve and load the corresponding LoRA adapter.
162
163
You can set up multiple LoRAResolver plugins if you want to load LoRA adapters from different sources. For example, you might have one resolver for local files and another for S3 storage. vLLM will load the first LoRA adapter that it finds.
docs/features/multimodal_inputs.md
+2
@@ -588,7 +588,9 @@ Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for
588
589
To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
590
pass a tensor of shape to the corresponding field of the multi-modal dictionary.
591
+
592
#### Image Embedding Inputs
593
+
594
For image embeddings, you can pass the base64-encoded tensor to the `image_embeds` field.
595
The following example demonstrates how to pass image embeddings to the OpenAI server:
596
docs/features/quantization/auto_round.md
+1
-1
@@ -97,7 +97,7 @@ for output in outputs:
97
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
98
```
99
100
-# Acknowledgement
100
+## Acknowledgement
101
102
Special thanks to open-source low precision libraries such as AutoGPTQ, AutoAWQ, GPTQModel, Triton, Marlin, and
103
ExLLaMAV2 for providing low-precision CUDA kernels, which are leveraged in AutoRound.
docs/features/quantization/int4.md
+2
-2
@@ -134,8 +134,8 @@ lm_eval --model vllm \
134
- Employ the chat template or instruction template that the model was trained with
135
- If you've fine-tuned a model, consider using a sample of your training data for calibration
136
- Tune key hyperparameters to the quantization algorithm:
137
- - `dampening_frac` sets how much influence the GPTQ algorithm has. Lower values can improve accuracy, but can lead to numerical instabilities that cause the algorithm to fail.
138
- - `actorder` sets the activation ordering. When compressing the weights of a layer weight, the order in which channels are quantized matters. Setting `actorder="weight"` can improve accuracy without added latency.
137
+ - `dampening_frac` sets how much influence the GPTQ algorithm has. Lower values can improve accuracy, but can lead to numerical instabilities that cause the algorithm to fail.
138
+ - `actorder` sets the activation ordering. When compressing the weights of a layer weight, the order in which channels are quantized matters. Setting `actorder="weight"` can improve accuracy without added latency.
139
140
The following is an example of an expanded quantization recipe you can tune to your own use case:
141
docs/features/quantization/quantized_kvcache.md
+1
@@ -50,6 +50,7 @@ Here is an example of how to enable FP8 quantization:
50
```
51
52
The `kv_cache_dtype` argument specifies the data type for KV cache storage:
53
+
54
- `"auto"`: Uses the model's default "unquantized" data type
55
- `"fp8"` or `"fp8_e4m3"`: Supported on CUDA 11.8+ and ROCm (AMD GPU)
56
- `"fp8_e5m2"`: Supported on CUDA 11.8+
docs/features/quantization/quark.md
+1
@@ -213,6 +213,7 @@ lm_eval --model vllm \
213
```
214
215
## Quark Quantization Script
216
+
217
In addition to the example of Python API above, Quark also offers a
218
[quantization script](https://quark.docs.amd.com/latest/pytorch/example_quark_torch_llm_ptq.html)
219
to quantize large language models more conveniently. It supports quantizing models with variety
docs/features/quantization/torchao.md
+1
@@ -13,6 +13,7 @@ pip install \
13
```
14
15
## Quantizing HuggingFace Models
16
+
17
You can quantize your own huggingface model with torchao, e.g. [transformers](https://huggingface.co/docs/transformers/main/en/quantization/torchao) and [diffusers](https://huggingface.co/docs/diffusers/en/quantization/torchao), and save the checkpoint to huggingface hub like [this](https://huggingface.co/jerryzh168/llama3-8b-int8wo) with the following example code:
18
19
??? code
docs/getting_started/installation/cpu.md
+3
-3
@@ -164,7 +164,7 @@ Note, it is recommended to manually reserve 1 CPU for vLLM front-end process whe
164
165
### How to decide `VLLM_CPU_KVCACHE_SPACE`?
166
167
- - This value is 4GB by default. Larger space can support more concurrent requests, longer context length. However, users should take care of memory capacity of each NUMA node. The memory usage of each TP rank is the sum of `weight shard size` and `VLLM_CPU_KVCACHE_SPACE`, if it exceeds the capacity of a single NUMA node, the TP worker will be killed with `exitcode 9` due to out-of-memory.
167
+This value is 4GB by default. Larger space can support more concurrent requests, longer context length. However, users should take care of memory capacity of each NUMA node. The memory usage of each TP rank is the sum of `weight shard size` and `VLLM_CPU_KVCACHE_SPACE`, if it exceeds the capacity of a single NUMA node, the TP worker will be killed with `exitcode 9` due to out-of-memory.
168
169
### How to do performance tuning for vLLM CPU?
170
@@ -183,13 +183,13 @@ vLLM CPU supports tensor parallel (TP) and pipeline parallel (PP) to leverage mu
183
184
### Which quantization configs does vLLM CPU support?
185
186
- - vLLM CPU supports quantizations:
186
+- vLLM CPU supports quantizations:
187
- AWQ (x86 only)
188
- GPTQ (x86 only)
189
- compressed-tensor INT8 W8A8 (x86, s390x)
190
191
### (x86 only) What is the purpose of `VLLM_CPU_MOE_PREPACK` and `VLLM_CPU_SGL_KERNEL`?
192
193
- - Both of them requires `amx` CPU flag.
193
+- Both of them requires `amx` CPU flag.
194
- `VLLM_CPU_MOE_PREPACK` can provides better performance for MoE models
195
- `VLLM_CPU_SGL_KERNEL` can provides better performance for MoE models and small-batch scenarios.
docs/getting_started/installation/intel_gaudi.md
+4
-4
@@ -339,13 +339,13 @@ Each described step is logged by vLLM server, as follows (negative values corres
339
340
- `VLLM_{phase}_{dim}_BUCKET_{param}` - collection of 12 environment variables configuring ranges of bucketing mechanism
341
342
- * `{phase}` is either `PROMPT` or `DECODE`
342
+ - `{phase}` is either `PROMPT` or `DECODE`
343
344
- * `{dim}` is either `BS`, `SEQ` or `BLOCK`
344
+ - `{dim}` is either `BS`, `SEQ` or `BLOCK`
345
346
- * `{param}` is either `MIN`, `STEP` or `MAX`
346
+ - `{param}` is either `MIN`, `STEP` or `MAX`
347
348
- * Default values:
348
+ - Default values:
349
350
| `{phase}` | Parameter | Env Variable | Value Expression |
351
|-----------|-----------|--------------|------------------|
docs/models/hardware_supported_models/tpu.md
+3
-2
@@ -1,7 +1,8 @@
1
# TPU
2
3
-# TPU Supported Models
4
-## Text-only Language Models
3
+## Supported Models
4
+
5
+### Text-only Language Models
6
7
| Model | Architecture | Supported |
8
|-----------------------------------------------------|--------------------------------|-----------|
docs/models/supported_models.md
+7
-7
@@ -45,10 +45,10 @@ If a model is neither supported natively by vLLM or Transformers, it can still b
45
For a model to be compatible with the Transformers backend for vLLM it must:
46
47
- be a Transformers compatible custom model (see [Transformers - Customizing models](https://huggingface.co/docs/transformers/en/custom_models)):
48
- * The model directory must have the correct structure (e.g. `config.json` is present).
49
- * `config.json` must contain `auto_map.AutoModel`.
48
+ - The model directory must have the correct structure (e.g. `config.json` is present).
49
+ - `config.json` must contain `auto_map.AutoModel`.
50
- be a Transformers backend for vLLM compatible model (see [writing-custom-models][writing-custom-models]):
51
- * Customisation should be done in the base model (e.g. in `MyModel`, not `MyModelForCausalLM`).
51
+ - Customisation should be done in the base model (e.g. in `MyModel`, not `MyModelForCausalLM`).
52
53
If the compatible model is:
54
@@ -134,10 +134,10 @@ class MyConfig(PretrainedConfig):
134
135
- `base_model_tp_plan` is a `dict` that maps fully qualified layer name patterns to tensor parallel styles (currently only `"colwise"` and `"rowwise"` are supported).
136
- `base_model_pp_plan` is a `dict` that maps direct child layer names to `tuple`s of `list`s of `str`s:
137
- * You only need to do this for layers which are not present on all pipeline stages
138
- * vLLM assumes that there will be only one `nn.ModuleList`, which is distributed across the pipeline stages
139
- * The `list` in the first element of the `tuple` contains the names of the input arguments
140
- * The `list` in the last element of the `tuple` contains the names of the variables the layer outputs to in your modeling code
137
+ - You only need to do this for layers which are not present on all pipeline stages
138
+ - vLLM assumes that there will be only one `nn.ModuleList`, which is distributed across the pipeline stages
139
+ - The `list` in the first element of the `tuple` contains the names of the input arguments
140
+ - The `list` in the last element of the `tuple` contains the names of the variables the layer outputs to in your modeling code
141
142
## Loading a Model
143
docs/serving/distributed_serving.md
+1
-1
@@ -99,7 +99,7 @@ From any node, enter a container and run `ray status` and `ray list nodes` to ve
99
### Running vLLM on a Ray cluster
100
101
!!! tip
102
- If Ray is running inside containers, run the commands in the remainder of this guide _inside the containers_, not on the host. To open a shell inside a container, connect to a node and use `docker exec -it <container_name> /bin/bash`.
102
+ If Ray is running inside containers, run the commands in the remainder of this guide *inside the containers*, not on the host. To open a shell inside a container, connect to a node and use `docker exec -it <container_name> /bin/bash`.
103
104
Once a Ray cluster is running, use vLLM as you would in a single-node setting. All resources across the Ray cluster are visible to vLLM, so a single `vllm` command on a single node is sufficient.
105
docs/serving/expert_parallel_deployment.md
+2
-1
@@ -31,11 +31,12 @@ vLLM provides three communication backends for EP:
31
32
Enable EP by setting the `--enable-expert-parallel` flag. The EP size is automatically calculated as:
33
34
-```
34
+```text
35
EP_SIZE = TP_SIZE × DP_SIZE
36
```
37
38
Where:
39
+
40
- `TP_SIZE`: Tensor parallel size (always 1 for now)
41
- `DP_SIZE`: Data parallel size
42
- `EP_SIZE`: Expert parallel size (computed automatically)
docs/serving/openai_compatible_server.md
+1
@@ -206,6 +206,7 @@ you can use the [official OpenAI Python client](https://github.com/openai/openai
206
We support both [Vision](https://platform.openai.com/docs/guides/vision)- and
207
[Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters;
208
see our [Multimodal Inputs](../features/multimodal_inputs.md) guide for more information.
209
+
210
- *Note: `image_url.detail` parameter is not supported.*
211
212
Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
docs/usage/security.md
+19
-13
@@ -13,15 +13,18 @@ All communications between nodes in a multi-node vLLM deployment are **insecure
13
The following options control inter-node communications in vLLM:
14
15
#### 1. **Environment Variables:**
16
- - `VLLM_HOST_IP`: Sets the IP address for vLLM processes to communicate on
16
+
17
+- `VLLM_HOST_IP`: Sets the IP address for vLLM processes to communicate on
18
19
#### 2. **KV Cache Transfer Configuration:**
19
- - `--kv-ip`: The IP address for KV cache transfer communications (default: 127.0.0.1)
20
- - `--kv-port`: The port for KV cache transfer communications (default: 14579)
20
+
21
+- `--kv-ip`: The IP address for KV cache transfer communications (default: 127.0.0.1)
22
+- `--kv-port`: The port for KV cache transfer communications (default: 14579)
23
24
#### 3. **Data Parallel Configuration:**
23
- - `data_parallel_master_ip`: IP of the data parallel master (default: 127.0.0.1)
24
- - `data_parallel_master_port`: Port of the data parallel master (default: 29500)
25
+
26
+- `data_parallel_master_ip`: IP of the data parallel master (default: 127.0.0.1)
27
+- `data_parallel_master_port`: Port of the data parallel master (default: 29500)
28
29
### Notes on PyTorch Distributed
30
@@ -41,18 +44,21 @@ Key points from the PyTorch security guide:
44
### Security Recommendations
45
46
#### 1. **Network Isolation:**
44
- - Deploy vLLM nodes on a dedicated, isolated network
45
- - Use network segmentation to prevent unauthorized access
46
- - Implement appropriate firewall rules
47
+
48
+- Deploy vLLM nodes on a dedicated, isolated network
49
+- Use network segmentation to prevent unauthorized access
50
+- Implement appropriate firewall rules
51
52
#### 2. **Configuration Best Practices:**
49
- - Always set `VLLM_HOST_IP` to a specific IP address rather than using defaults
50
- - Configure firewalls to only allow necessary ports between nodes
53
+
54
+- Always set `VLLM_HOST_IP` to a specific IP address rather than using defaults
55
+- Configure firewalls to only allow necessary ports between nodes
56
57
#### 3. **Access Control:**
53
- - Restrict physical and network access to the deployment environment
54
- - Implement proper authentication and authorization for management interfaces
55
- - Follow the principle of least privilege for all system components
58
+
59
+- Restrict physical and network access to the deployment environment
60
+- Implement proper authentication and authorization for management interfaces
61
+- Follow the principle of least privilege for all system components
62
63
## Security and Firewalls: Protecting Exposed vLLM Systems
64
docs/usage/v1_guide.md
+5
-5
@@ -148,7 +148,7 @@ are not yet supported.
148
vLLM V1 supports logprobs and prompt logprobs. However, there are some important semantic
149
differences compared to V0:
150
151
-**Logprobs Calculation**
151
+##### Logprobs Calculation
152
153
Logprobs in V1 are now returned immediately once computed from the model’s raw output (i.e.
154
before applying any logits post-processing such as temperature scaling or penalty
@@ -157,7 +157,7 @@ probabilities used during sampling.
157
158
Support for logprobs with post-sampling adjustments is in progress and will be added in future updates.
159
160
-**Prompt Logprobs with Prefix Caching**
160
+##### Prompt Logprobs with Prefix Caching
161
162
Currently prompt logprobs are only supported when prefix caching is turned off via `--no-enable-prefix-caching`. In a future release, prompt logprobs will be compatible with prefix caching, but a recomputation will be triggered to recover the full prompt logprobs even upon a prefix cache hit. See details in [RFC #13414](gh-issue:13414).
163
@@ -165,7 +165,7 @@ Currently prompt logprobs are only supported when prefix caching is turned off v
165
166
As part of the major architectural rework in vLLM V1, several legacy features have been deprecated.
167
168
-**Sampling features**
168
+##### Sampling features
169
170
- **best_of**: This feature has been deprecated due to limited usage. See details at [RFC #13361](gh-issue:13361).
171
- **Per-Request Logits Processors**: In V0, users could pass custom
@@ -173,11 +173,11 @@ As part of the major architectural rework in vLLM V1, several legacy features ha
173
feature has been deprecated. Instead, the design is moving toward supporting **global logits
174
processors**, a feature the team is actively working on for future releases. See details at [RFC #13360](gh-pr:13360).
175
176
-**KV Cache features**
176
+##### KV Cache features
177
178
- **GPU <> CPU KV Cache Swapping**: with the new simplified core architecture, vLLM V1 no longer requires KV cache swapping
179
to handle request preemptions.
180
181
-**Structured Output features**
181
+##### Structured Output features
182
183
- **Request-level Structured Output Backend**: Deprecated, alternative backends (outlines, guidance) with fallbacks is supported now.
examples/offline_inference/disaggregated-prefill-v1/README.md
+1
-1
@@ -5,6 +5,6 @@ This example contains scripts that demonstrate disaggregated prefill in the offl
5
## Files
6
7
- `run.sh` - A helper script that will run `prefill_example.py` and `decode_example.py` sequentially.
8
- - Make sure you are in the `examples/offline_inference/disaggregated-prefill-v1` directory before running `run.sh`.
8
+ - Make sure you are in the `examples/offline_inference/disaggregated-prefill-v1` directory before running `run.sh`.
9
- `prefill_example.py` - A script which performs prefill only, saving the KV state to the `local_storage` directory and the prompts to `output.txt`.
10
- `decode_example.py` - A script which performs decode only, loading the KV state from the `local_storage` directory and the prompts from `output.txt`.
examples/offline_inference/openai_batch/README.md
+4
-4
@@ -19,9 +19,9 @@ We currently support `/v1/chat/completions`, `/v1/embeddings`, and `/v1/score` e
19
## Pre-requisites
20
21
* The examples in this document use `meta-llama/Meta-Llama-3-8B-Instruct`.
22
- - Create a [user access token](https://huggingface.co/docs/hub/en/security-tokens)
23
- - Install the token on your machine (Run `huggingface-cli login`).
24
- - Get access to the gated model by [visiting the model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and agreeing to the terms and conditions.
22
+ * Create a [user access token](https://huggingface.co/docs/hub/en/security-tokens)
23
+ * Install the token on your machine (Run `huggingface-cli login`).
24
+ * Get access to the gated model by [visiting the model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and agreeing to the terms and conditions.
25
26
## Example 1: Running with a local file
27
@@ -105,7 +105,7 @@ To integrate with cloud blob storage, we recommend using presigned urls.
105
106
* [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
107
* The `awscli` package (Run `pip install awscli`) to configure your credentials and interactively use s3.
108
- - [Configure your credentials](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).
108
+ * [Configure your credentials](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).
109
* The `boto3` python package (Run `pip install boto3`) to generate presigned urls.
110
111
### Step 1: Upload your input script
examples/others/lmcache/README.md
+4
@@ -28,16 +28,20 @@ to run disaggregated prefill and benchmark the performance.
28
### Components
29
30
#### Server Scripts
31
+
32
- `disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh` - Launches individual vLLM servers for prefill/decode, and also launches the proxy server.
33
- `disagg_prefill_lmcache_v1/disagg_proxy_server.py` - FastAPI proxy server that coordinates between prefiller and decoder
34
- `disagg_prefill_lmcache_v1/disagg_example_nixl.sh` - Main script to run the example
35
36
#### Configuration
37
+
38
- `disagg_prefill_lmcache_v1/configs/lmcache-prefiller-config.yaml` - Configuration for prefiller server
39
- `disagg_prefill_lmcache_v1/configs/lmcache-decoder-config.yaml` - Configuration for decoder server
40
41
#### Log Files
42
+
43
The main script generates several log files:
44
+
45
- `prefiller.log` - Logs from the prefill server
46
- `decoder.log` - Logs from the decode server
47
- `proxy.log` - Logs from the proxy server
examples/others/logging_configuration.md
+3
-3
@@ -8,11 +8,11 @@ of logging configurations that range from simple-and-inflexible to
8
more-complex-and-more-flexible.
9
10
- No vLLM logging (simple and inflexible)
11
- - Set `VLLM_CONFIGURE_LOGGING=0` (leaving `VLLM_LOGGING_CONFIG_PATH` unset)
11
+ - Set `VLLM_CONFIGURE_LOGGING=0` (leaving `VLLM_LOGGING_CONFIG_PATH` unset)
12
- vLLM's default logging configuration (simple and inflexible)
13
- - Leave `VLLM_CONFIGURE_LOGGING` unset or set `VLLM_CONFIGURE_LOGGING=1`
13
+ - Leave `VLLM_CONFIGURE_LOGGING` unset or set `VLLM_CONFIGURE_LOGGING=1`
14
- Fine-grained custom logging configuration (more complex, more flexible)
15
- - Leave `VLLM_CONFIGURE_LOGGING` unset or set `VLLM_CONFIGURE_LOGGING=1` and
15
+ - Leave `VLLM_CONFIGURE_LOGGING` unset or set `VLLM_CONFIGURE_LOGGING=1` and
16
set `VLLM_LOGGING_CONFIG_PATH=<path-to-logging-config.json>`
17
18
## Logging Configuration Environment Variables
pyproject.toml
-10
@@ -156,16 +156,6 @@ markers = [
156
"optional: optional tests that are automatically skipped, include --optional to run them",
157
]
158
159
-[tool.pymarkdown]
160
-plugins.md004.style = "sublist" # ul-style
161
-plugins.md007.indent = 4 # ul-indent
162
-plugins.md007.start_indented = true # ul-indent
163
-plugins.md013.enabled = false # line-length
164
-plugins.md041.enabled = false # first-line-h1
165
-plugins.md033.enabled = false # inline-html
166
-plugins.md046.enabled = false # code-block-style
167
-plugins.md024.allow_different_nesting = true # no-duplicate-headers
168
-
159
[tool.ty.src]
160
root = "./vllm"
161
respect-ignore-files = true
tools/ep_kernels/README.md
+6
-3
@@ -1,6 +1,9 @@
1
+# Expert parallel kernels
2
+
3
Large-scale cluster-level expert parallel, as described in the [DeepSeek-V3 Technical Report](http://arxiv.org/abs/2412.19437), is an efficient way to deploy sparse MoE models with many experts. However, such deployment requires many components beyond a normal Python package, including system package support and system driver support. It is impossible to bundle all these components into a Python package.
4
5
Here we break down the requirements in 2 steps:
6
+
7
1. Build and install the Python libraries (both [pplx-kernels](https://github.com/ppl-ai/pplx-kernels) and [DeepEP](https://github.com/deepseek-ai/DeepEP)), including necessary dependencies like NVSHMEM. This step does not require any privileged access. Any user can do this.
8
2. Configure NVIDIA driver to enable IBGDA. This step requires root access, and must be done on the host machine.
9
@@ -8,15 +11,15 @@ Here we break down the requirements in 2 steps:
11
12
All scripts accept a positional argument as workspace path for staging the build, defaulting to `$(pwd)/ep_kernels_workspace`.
13
11
-# Usage
14
+## Usage
15
13
-## Single-node
16
+### Single-node
17
18
```bash
19
bash install_python_libraries.sh
20
```
21
19
-## Multi-node
22
+### Multi-node
23
24
```bash
25
bash install_python_libraries.sh
vllm/plugins/lora_resolvers/README.md
+2
-1
@@ -6,7 +6,8 @@ via the LoRAResolver plugin framework.
6
Note that `VLLM_ALLOW_RUNTIME_LORA_UPDATING` must be set to true to allow LoRA resolver plugins
7
to work, and `VLLM_PLUGINS` must be set to include the desired resolver plugins.
8
9
-# lora_filesystem_resolver
9
+## lora_filesystem_resolver
10
+
11
This LoRA Resolver is installed with vLLM by default.
12
To use, set `VLLM_PLUGIN_LORA_CACHE_DIR` to a local directory. When vLLM receives a request
13
for a LoRA adapter `foobar` it doesn't currently recognize, it will look in that local directory