@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / a0f44bb616

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 7, 01:30 UTC

Allow `markdownlint` to run locally (#36398)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Harry Mellor committed Mar 9, 2026 at 03:05 UTC a0f44bb6169dcd6225d2efc0a59dd343a8d4a38e

47 files changed +394 -392

.github/mergify.yml

+2 -4

@@ -38,15 +38,13 @@ pull_request_rules:

39 > [!TIP]

40 > <details>

41 - > <summary>Is <code>mypy</code> or <code>markdownlint</code> failing?</summary>

41 + > <summary>Is <code>mypy</code> failing?</summary>

42 >

43 - > <code>mypy</code> and <code>markdownlint</code> are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

43 + > <code>mypy</code> is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

44 >

45 > ```bash

46 > # For mypy (substitute "3.10" with the failing version if needed)

47 > pre-commit run --hook-stage manual mypy-3.10

48 - > # For markdownlint

49 - > pre-commit run --hook-stage manual markdownlint

48 > ```

49 > </details>

.pre-commit-config.yaml

+5 -5

@@ -24,12 +24,12 @@ repos:

25 types_or: [c++, cuda]

26 args: [--style=file, --verbose]

27 -- repo: https://github.com/igorshubovych/markdownlint-cli

28 - rev: v0.45.0

27 +- repo: https://github.com/DavidAnson/markdownlint-cli2

28 + rev: v0.21.0

29 hooks:

30 - - id: markdownlint

31 - exclude: '.*\.inc\.md'

32 - stages: [manual] # Only run in CI

30 + - id: markdownlint-cli2

31 + language_version: lts

32 + args: [--fix]

33 - repo: https://github.com/rhysd/actionlint

34 rev: v1.7.7

35 hooks:

benchmarks/attention_benchmarks/README.md

+1 -1

@@ -187,7 +187,7 @@ python benchmark.py \

187 ## Hardware Requirements

188

189 | Backend | Hardware |

190 -|---------|----------|

190 +| ------- | -------- |

191 | Flash/Triton/FlashInfer | Any CUDA GPU |

192 | CUTLASS MLA | Blackwell (SM100+) |

193 | FlashAttn MLA | Hopper (SM90+) |

benchmarks/auto_tune/README.md

+1 -1

@@ -41,7 +41,7 @@ MODEL=meta-llama/Llama-3.3-70B-Instruct SYSTEM=TPU TP=8 DOWNLOAD_DIR='' INPUT_LE

41 | --- | --- | --- |

42 | `BASE` | **Required.** The absolute path to the parent directory of your vLLM repository directory. | `"$HOME"` |

43 | `MODEL` | **Required.** The Hugging Face model identifier to be served by vllm. | `"meta-llama/Llama-3.1-8B-Instruct"` |

44 -| `SYSTEM`| **Required.** The hardware you are running on. Choices: `TPU` or `GPU`. (For other systems, it might not support saving profiles) | `"TPU"` |

44 +| `SYSTEM` | **Required.** The hardware you are running on. Choices: `TPU` or `GPU`. (For other systems, it might not support saving profiles) | `"TPU"` |

45 | `TP` | **Required.** The tensor-parallelism size. | `1` |

46 | `DOWNLOAD_DIR` | **Required.** Directory to download and load model weights from. | `""` (default download path) |

47 | `INPUT_LEN` | **Required.** Request input length. | `4000` |

docs/benchmarking/cli.md

+9 -9

@@ -18,7 +18,7 @@ th {

18 </style>

21 -|---------|--------|---------|-----------|

21 +| ------- | ------ | ------- | --------- |

22 | ShareGPT | ✅ | ✅ | `wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json` |

23 | ShareGPT4V (Image) | ✅ | ✅ | `wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json` Note that the images need to be downloaded separately. For example, to download COCO's 2017 Train images: `wget http://images.cocodataset.org/zips/train2017.zip` |

24 | ShareGPT4Video (Video) | ✅ | ✅ | `git clone https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video` |

@@ -383,14 +383,14 @@ The `--burstiness` parameter mathematically controls request arrival patterns us

383

384 Load Pattern Recommendations by Use Case:

385

387 -| --- | --- | --- | --- | --- |

387 +| --- | --- | --- | --- | --- |

391 -| Latency Profiling | 2.0-5.0 | Low (1-10) | Infinite | Uniform load for consistent timing analysis |

391 +| Latency Profiling | 2.0-5.0 | Low (1-10) | Infinite | Uniform load for consistent timing analysis |

394

395 These load patterns help evaluate different aspects of your vLLM deployment, from basic performance characteristics to resilience under challenging traffic conditions.

396

@@ -941,7 +941,7 @@ Benchmark per-stage latency of the multimodal (MM) input processor pipeline, inc

941 The benchmark measures the following stages for each request:

942

943 | Stage | Description |

944 -|-------|-------------|

944 +| ----- | ----------- |

945 | `get_mm_hashes_secs` | Time spent hashing multimodal inputs |

946 | `get_cache_missing_items_secs` | Time spent looking up the processor cache |

947 | `apply_hf_processor_secs` | Time spent in the HuggingFace processor |

docs/benchmarking/dashboard.md

+6 -6

@@ -60,12 +60,12 @@ Here is an example using the script to compare result_a and result_b with max co

61 ***Output Tput (tok/s) — Model : [ meta-llama/Llama-3.1-8B-Instruct ] , Dataset Name : [ random ] , Input Len : [ 2048.0 ] , Output Len : [ 2048.0 ]***

64 -|----|------|-----|-----------|----------|----------|

65 -| 0 | 12 | inf | 24.98 | 186.03 | 7.45 |

66 -| 1 | 16 | inf| 25.49 | 246.92 | 9.69 |

67 -| 2 | 24 | inf| 27.74 | 293.34 | 10.57 |

68 -| 3 | 32 | inf| 28.61 |306.69 | 10.72 |

64 +| | -------------------- | --- | -------------------------------- | -------------------------------- | ---------- |

65 +| 0 | 12 | inf | 24.98 | 186.03 | 7.45 |

66 +| 1 | 16 | inf | 25.49 | 246.92 | 9.69 |

67 +| 2 | 24 | inf | 27.74 | 293.34 | 10.57 |

68 +| 3 | 32 | inf | 28.61 |306.69 | 10.72 |

70 ***compare-json-results.py – Command-Line Parameters***

docs/cli/bench/mm_processor.md

+1 -1

@@ -29,7 +29,7 @@ vllm bench mm-processor \

29 ## Measured Stages

31 | Stage | Description |

32 -|-------|-------------|

32 +| ----- | ----------- |

33 | `get_mm_hashes_secs` | Time spent hashing multimodal inputs |

34 | `get_cache_missing_items_secs` | Time spent looking up the processor cache |

35 | `apply_hf_processor_secs` | Time spent in the HuggingFace processor |

docs/cli/json_tip.inc.md

+2 -1

@@ -1,3 +1,4 @@

1 +

2 When passing JSON CLI arguments, the following sets of arguments are equivalent:

4 - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'`

@@ -6,4 +7,4 @@ When passing JSON CLI arguments, the following sets of arguments are equivalent:

7 Additionally, list elements can be passed individually using `+`:

9 - `--json-arg '{"key4": ["value3", "value4", "value5"]}'`

9 -- `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'`

\ No newline at end of file

10 +- `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'`

docs/configuration/optimization.md

+1 -1

@@ -293,7 +293,7 @@ llm = LLM(

293 Based on the configuration, the content of the multi-modal caches on `P0` and `P1` are as follows:

294

296 -|-------------------|-------------|------------|------------|-------------|-------------|

296 +| ----------------- | ----------- | ---------- | ---------- | ----------- | ----------- |

297 | lru | Processor Caching | K + V | N/A | N/A | `mm_processor_cache_gb * data_parallel_size` |

298 | lru | Key-Replicated Caching | K | K + V | N/A | `mm_processor_cache_gb * api_server_count` |

299 | shm | Shared Memory Caching | K | N/A | V | `mm_processor_cache_gb * api_server_count` |

docs/contributing/README.md

-1

@@ -94,7 +94,6 @@ vLLM's `pre-commit` hooks will now run automatically every time you commit.

94 Some `pre-commit` hooks only run in CI. If you need to, you can run them locally with:

96 ```bash

97 - pre-commit run --hook-stage manual markdownlint

97 pre-commit run --hook-stage manual mypy-3.10

98 ```

docs/contributing/ci/update_pytorch_version.md

+4 -4

@@ -66,12 +66,12 @@ This complicates the process as we cannot use the out-of-the-box

66 - Important indexes at the moment include:

68 | Platform | `--extra-index-url` |

69 -|----------|-----------------|

70 -| CUDA 12.8| [https://download.pytorch.org/whl/cu128](https://download.pytorch.org/whl/cu128)|

71 -| CPU | [https://download.pytorch.org/whl/cpu](https://download.pytorch.org/whl/cpu)|

69 +| -------- | ------------------- |

70 +| CUDA 12.8 | [https://download.pytorch.org/whl/cu128](https://download.pytorch.org/whl/cu128) |

71 +| CPU | [https://download.pytorch.org/whl/cpu](https://download.pytorch.org/whl/cpu) |

72 | ROCm 6.2 | [https://download.pytorch.org/whl/rocm6.2.4](https://download.pytorch.org/whl/rocm6.2.4) |

73 | ROCm 6.3 | [https://download.pytorch.org/whl/rocm6.3](https://download.pytorch.org/whl/rocm6.3) |

74 -| XPU | [https://download.pytorch.org/whl/xpu](https://download.pytorch.org/whl/xpu) |

74 +| XPU | [https://download.pytorch.org/whl/xpu](https://download.pytorch.org/whl/xpu) |

76 - Update the below files to match the CUDA version from step 1. This makes sure that the release vLLM wheel is tested on CI.

77 - `.buildkite/release-pipeline.yaml`

docs/contributing/deprecation_policy.md

+1 -1

@@ -66,7 +66,7 @@ stages will be removed.

66 Assume a feature is deprecated in `v0.9.0`.

68 | Release | Status |

69 -|---------------|-------------------------------------------------------------------------------------------------|

69 +| ------------- | ----------------------------------------------------------------------------------------------- |

70 | `v0.9.0` | Feature is deprecated with clear removal version listed. |

71 | `v0.10.0` | Feature is now off by default, throws an error when used, and can be re-enabled for legacy use. |

72 | `v0.11.0` | Feature is removed. |

docs/deployment/frameworks/helm.md

+1 -1

@@ -49,7 +49,7 @@ chart **including persistent volumes** and deletes the release.

49 The following table describes configurable parameters of the chart in `values.yaml`:

52 -|-----|------|---------|-------------|

52 +| --- | ---- | ------- | ----------- |

53 | autoscaling | object | {"enabled":false,"maxReplicas":100,"minReplicas":1,"targetCPUUtilizationPercentage":80} | Autoscaling configuration |

55 | autoscaling.maxReplicas | int | 100 | Maximum replicas |

docs/deployment/integrations/kuberay.md

+1 -1

@@ -6,7 +6,7 @@ A Ray cluster can be declared in YAML, and the operator then handles pod schedul

6 ## Why KubeRay instead of manual scripts?

8 | Feature | Manual scripts | KubeRay |

9 -|---------|-----------------------------------------------------------|---------|

9 +| ------- | --------------------------------------------------------- | ------- |

10 | Cluster bootstrap | Manually SSH into every node and run a script | One command to create or update the whole cluster: `kubectl apply -f cluster.yaml` |

11 | Autoscaling | Manual | Automatically patches CRDs for adjusting cluster size |

12 | Upgrades | Tear down & re-create manually | Blue/green deployment updates supported |

docs/design/arch_overview.md

+1 -1

@@ -119,7 +119,7 @@ The code can be found in [vllm/v1/engine/coordinator.py](../../vllm/v1/engine/co

119 For a deployment with `N` GPUs, `TP` tensor parallel size, `DP` data parallel size, and `A` API server count:

120

121 | Process Type | Count | Notes |

122 -|---|---|---|

122 +| - | - | - |

123 | API Server | `A` (default `DP`) | Handles HTTP requests and input processing |

124 | Engine Core | `DP` (default 1) | Scheduler and KV cache management |

125 | GPU Worker | `N` (= `DP x PP x TP`) | One per GPU, executes model forward passes |

docs/design/attention_backends.md

+17 -17

@@ -101,7 +101,7 @@ Priority is **1 = highest** (tried first).

101 **Blackwell (SM 10.x):**

102

103 | Priority | Backend |

104 -|----------|---------|

104 +| -------- | ------- |

105 | 1 | `FLASHINFER` |

106 | 2 | `FLASH_ATTN` |

107 | 3 | `TRITON_ATTN` |

@@ -110,7 +110,7 @@ Priority is **1 = highest** (tried first).

110 **Ampere/Hopper (SM 8.x-9.x):**

111

112 | Priority | Backend |

113 -|----------|---------|

113 +| -------- | ------- |

114 | 1 | `FLASH_ATTN` |

115 | 2 | `FLASHINFER` |

116 | 3 | `TRITON_ATTN` |

@@ -121,7 +121,7 @@ Priority is **1 = highest** (tried first).

121 **Blackwell (SM 10.x):**

122

123 | Priority | Backend |

124 -|----------|---------|

124 +| -------- | ------- |

125 | 1 | `FLASHINFER_MLA` |

126 | 2 | `CUTLASS_MLA` |

127 | 3 | `FLASH_ATTN_MLA` |

@@ -133,7 +133,7 @@ Priority is **1 = highest** (tried first).

133 **Ampere/Hopper (SM 8.x-9.x):**

134

135 | Priority | Backend |

136 -|----------|---------|

136 +| -------- | ------- |

137 | 1 | `FLASH_ATTN_MLA` |

138 | 2 | `FLASHMLA` |

139 | 3 | `FLASHINFER_MLA` |

@@ -145,7 +145,7 @@ Priority is **1 = highest** (tried first).

145 ## Legend

146

147 | Column | Description |

148 -|--------|-------------|

148 +| ------ | ----------- |

149 | **Dtypes** | Supported model data types (fp16, bf16, fp32) |

150 | **KV Dtypes** | Supported KV cache data types (`auto`, `fp8`, `fp8_e4m3`, etc.) |

151 | **Block Sizes** | Supported KV cache block sizes (%N means multiples of N) |

@@ -162,20 +162,20 @@ Priority is **1 = highest** (tried first).

162 ## Standard Attention (MHA, MQA, GQA) Backends

163

165 -|---------|---------|--------|-----------|-------------|------------|------|-----------|-----|-----------------|--------------|

166 -| `CPU_ATTN` | | fp16, bf16, fp32 | `auto` | Any | 32, 64, 80, 96, 112, 128, 160, 192, 224, 256 | ❌ | ❌ | ❌ | All | N/A |

165 +| ------- | ------- | ------ | --------- | ----------- | ---------- | ---- | --------- | --- | --------------- | ------------ |

166 +| `CPU_ATTN` | | fp16, bf16, fp32 | `auto` | Any | 32, 64, 80, 96, 112, 128, 160, 192, 224, 256 | ❌ | ❌ | ❌ | All | N/A |

167 | `FLASHINFER` | Native† | fp16, bf16 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` | 16, 32, 64 | 64, 128, 256 | ❌ | ❌ | ✅ | Decoder | 7.x-9.x |

168 | `FLASHINFER` | TRTLLM† | fp16, bf16 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` | 16, 32, 64 | 64, 128, 256 | ✅ | ❌ | ✅ | Decoder | 10.x |

169 | `FLASH_ATTN` | FA2* | fp16, bf16 | `auto`, `bfloat16` | %16 | Any | ❌ | ❌ | ✅ | All | ≥8.0 |

170 | `FLASH_ATTN` | FA3* | fp16, bf16 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` | %16 | Any | ✅ | ❌ | ✅ | All | 9.x |

171 | `FLASH_ATTN` | FA4* | fp16, bf16 | `auto`, `bfloat16` | %16 | Any | ❌ | ❌ | ✅ | All | ≥10.0 |

172 -| `FLASH_ATTN_DIFFKV` | | fp16, bf16 | `auto` | Any | Any | ❌ | ❌ | ✅ | Decoder | Any |

173 -| `FLEX_ATTENTION` | | fp16, bf16, fp32 | `auto`, `bfloat16` | Any | Any | ❌ | ✅ | ❌ | Decoder, Encoder Only | Any |

174 -| `ROCM_AITER_FA` | | fp16, bf16 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` | 16, 32 | 64, 128, 256 | ❌ | ❌ | ❌ | Decoder, Enc-Dec | N/A |

175 -| `ROCM_AITER_UNIFIED_ATTN` | | fp16, bf16 | `auto` | %16 | Any | ✅ | ✅ | ❌ | All | N/A |

176 -| `ROCM_ATTN` | | fp16, bf16, fp32 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` | 16, 32, 544 | 32, 64, 80, 96, 128, 160, 192, 224, 256 | ✅ | ✅ | ❌ | All | N/A |

177 -| `TREE_ATTN` | | fp16, bf16 | `auto` | %16 | 32, 64, 96, 128, 160, 192, 224, 256 | ❌ | ❌ | ❌ | Decoder | Any |

178 -| `TRITON_ATTN` | | fp16, bf16, fp32 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` | %16 | Any | ✅ | ✅ | ❌ | All | Any |

172 +| `FLASH_ATTN_DIFFKV` | | fp16, bf16 | `auto` | Any | Any | ❌ | ❌ | ✅ | Decoder | Any |

173 +| `FLEX_ATTENTION` | | fp16, bf16, fp32 | `auto`, `bfloat16` | Any | Any | ❌ | ✅ | ❌ | Decoder, Encoder Only | Any |

174 +| `ROCM_AITER_FA` | | fp16, bf16 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` | 16, 32 | 64, 128, 256 | ❌ | ❌ | ❌ | Decoder, Enc-Dec | N/A |

175 +| `ROCM_AITER_UNIFIED_ATTN` | | fp16, bf16 | `auto` | %16 | Any | ✅ | ✅ | ❌ | All | N/A |

176 +| `ROCM_ATTN` | | fp16, bf16, fp32 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` | 16, 32, 544 | 32, 64, 80, 96, 128, 160, 192, 224, 256 | ✅ | ✅ | ❌ | All | N/A |

177 +| `TREE_ATTN` | | fp16, bf16 | `auto` | %16 | 32, 64, 96, 128, 160, 192, 224, 256 | ❌ | ❌ | ❌ | Decoder | Any |

178 +| `TRITON_ATTN` | | fp16, bf16, fp32 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` | %16 | Any | ✅ | ✅ | ❌ | All | Any |

179

180 > **†** FlashInfer uses TRTLLM attention on Blackwell (SM100), which supports sinks. Disable via `--attention-config.use_trtllm_attention=0`.

181 >

@@ -191,10 +191,10 @@ The prefill backend is selected at runtime based on hardware and

191 configuration.

192

194 -|---------|-------------|--------------|--------|---------|-------|

194 +| ------- | ----------- | ------------ | ------ | ------- | ----- |

199

200 > **‡** TRT-LLM Ragged is the default on Blackwell (SM100).

@@ -203,7 +203,7 @@ configuration.

203 ### Decode Backends

204

206 -|---------|--------|-----------|-------------|------------|------|--------|-----------|-----|-----------------|--------------|

206 +| ------- | ------ | --------- | ----------- | ---------- | ---- | ------ | --------- | --- | --------------- | ------------ |

207 | `CUTLASS_MLA` | fp16, bf16 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3` | 128 | Any | ❌ | ❌ | ❌ | ✅ | Decoder | 10.x |

208 | `FLASHINFER_MLA` | fp16, bf16 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3` | 32, 64 | Any | ❌ | ❌ | ❌ | ❌ | Decoder | 10.x |

209 | `FLASHINFER_MLA_SPARSE` | fp16, bf16 | `auto`, `bfloat16`, `fp8`, `fp8_e4m3` | 32, 64 | 576 | ❌ | ✅ | ❌ | ❌ | Decoder | 10.x |

docs/design/cuda_graphs.md

+3 -3

@@ -174,18 +174,18 @@ Suppose we have hybrid attention backends (e.g., in mamba mixer models). In that

174 The following table lists backends that support full CUDA Graphs at the time of writing.

175

176 | Attention Backend | cudagraph_support | Comments |

177 -|:---|:---|:---|

177 +| :---------------- | :---------------- | :------- |

178 | FlashAttention v2 | `UNIFORM_BATCH` | Actually `ALWAYS` but workaround to fallback to `FULL_AND_PIECEWISE` for performance reason |

179 | FlashAttention v3 | `ALWAYS` | has unified routine for both batches, so `FULL` mode is good |

180 | Triton Attention | `ALWAYS` | prefer `FULL_AND_PIECEWISE` since it has different kernels for prefill/mixed and pure decode batches |

181 -| AITER FlashAttention | `UNIFORM_BATCH`| |

181 +| AITER FlashAttention | `UNIFORM_BATCH` | |

182 | FlashInfer | `UNIFORM_SINGLE_TOKEN_DECODE` | Will be set to `UNIFORM_BATCH` when using TRTLLM attention on Blackwell |

183 | FlashMLA | `UNIFORM_BATCH` | |

184 | FlashInferMLA | `UNIFORM_BATCH` | |

185 | FlashInferMLASparse | `UNIFORM_BATCH` | |

186 | AITER MLA | `UNIFORM_SINGLE_TOKEN_DECODE` | |

187 | CUTLASS MLA | `UNIFORM_SINGLE_TOKEN_DECODE` | |

188 -| Mamba attention| `UNIFORM_SINGLE_TOKEN_DECODE` | |

188 +| Mamba attention | `UNIFORM_SINGLE_TOKEN_DECODE` | |

189

190 Unlisted backends are all declared as `NEVER`.

191

docs/design/debug_vllm_compile.md

+6 -6

@@ -5,12 +5,12 @@ TL;DR:

5 - use tlparse to acquire torch.compile logs. Include these logs in bug reports and/or support asks.

6 - The vLLM-torch.compile integration is multiple pieces. vLLM exposes flags to turn off each piece:

8 -| Online Flag | Offline Flag | Result |

9 -|----------|----------|-------------|

10 -| --enforce-eager | enforce_eager=True | Turn off torch.compile and CUDAGraphs |

11 -| -cc.mode=0 | mode=CompilationMode.NONE | Turn off torch.compile only |

12 -| -cc.cudagraph_mode=NONE | compilation_config=CompilationConfig(cudagraph_mode=CUDAGraphMode.NONE) | Turn off CUDAGraphs only |

13 -| -cc.backend=eager | compilation_config=CompilationConfig(backend='eager') | Turn off TorchInductor |

8 +| Online Flag | Offline Flag | Result |

9 +| ----------- | ------------ | ------ |

10 +| --enforce-eager | enforce_eager=True | Turn off torch.compile and CUDAGraphs |

11 +| -cc.mode=0 | mode=CompilationMode.NONE | Turn off torch.compile only |

12 +| -cc.cudagraph_mode=NONE | compilation_config=CompilationConfig(cudagraph_mode=CUDAGraphMode.NONE) | Turn off CUDAGraphs only |

13 +| -cc.backend=eager | compilation_config=CompilationConfig(backend='eager') | Turn off TorchInductor |

15 ## vLLM-torch.compile overview

docs/design/fusions.md

+2 -2

@@ -19,7 +19,7 @@ or just on the low or high end.

19 If tuning performance by hand, always benchmark your exact use-case with and without the fusion to verify the impact.

22 -|--------------------------------------------------------------------------------|------------------------------|------------------------------------------------|--------------------------------|--------------------|-----------|--------------|

22 +| ------------------------------------------------------------------------------ | ---------------------------- | ---------------------------------------------- | ------------------------------ | ------------------ | --------- | ------------ |

23 | [AllReduce + RMSNorm](#allreduce--rmsnorm-fuse_allreduce_rms) | `fuse_allreduce_rms` | All-reduce → RMSNorm (+residual_add) (→ quant) | O2 (Hopper/Blackwell + TP > 1) | 5-20% | No | Low |

25 | [RoPE + KV-Cache Update](#rope--kv-cache-update-fuse_rope_kvcache) | `fuse_rope_kvcache` | Rotary embedding → KV cache write | O1 (ROCm/AITER only) | TBD | No | Low |

@@ -37,7 +37,7 @@ The table below lists the quantization schemes supported by each fusion on each

37 [#36066](https://github.com/vllm-project/vllm/issues/36066)

40 -|------------------------------|------------------------------------------|------------------------------------------|------------------------------------------|---------------|------------------------------------------|

40 +| ---------------------------- | ---------------------------------------- | ---------------------------------------- | ---------------------------------------- | ------------- | ---------------------------------------- |

41 | `fuse_allreduce_rms` | FP16/BF16, FP8 static, NVFP4 | FP16/BF16, FP8 static | — | — | — |

43 | `fuse_rope_kvcache` | — | — | — | — | FP16/BF16 |

docs/design/moe_kernel_features.md

+5 -5

@@ -31,7 +31,7 @@ th {

31 </style>

34 -|---------|--------------------|--------------|---------------|-------|-----------------------|-----------|

34 +| ------- | ------------------ | ------------ | ------------- | ----- | --------------------- | --------- |

35 | naive | standard | all1 | G,A,T | N | 6 | [layer.py][vllm.model_executor.layers.fused_moe.layer.FusedMoE] |

36 | deepep_high_throughput | standard | fp8 | G(128),A,T2 | Y | Y | [`DeepEPHTPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.deepep_ht_prepare_finalize.DeepEPHTPrepareAndFinalize] |

37 | deepep_low_latency | batched | fp8 | G(128),A,T3 | Y | Y | [`DeepEPLLPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.deepep_ll_prepare_finalize.DeepEPLLPrepareAndFinalize] |

@@ -78,7 +78,7 @@ Most experts flavors include an equivalent modular interface which will be a sub

78 To be used with a particular `FusedMoEPrepareAndFinalizeModular` subclass, MoE kernels must have compatible activation formats, quantization types and quantization formats.

81 -|--------|-------------------|--------------|---------------|---------------------|-----------------------|---------|--------|

81 +| ------ | ----------------- | ------------ | ------------- | ------------------- | --------------------- | ------- | ------ |

82 | triton | standard | all1 | G,A,T | silu, gelu,swigluoai,silu_no_mul,gelu_no_mul | Y | Y | [`fused_experts`][vllm.model_executor.layers.fused_moe.fused_moe.fused_experts],[`TritonExperts`][vllm.model_executor.layers.fused_moe.fused_moe.TritonExperts] |

83 | triton (batched) | batched | all1 | G,A,T | silu, gelu | 6 | Y | [`BatchedTritonExperts`][vllm.model_executor.layers.fused_moe.fused_batched_moe.BatchedTritonExperts] |

84 | deep gemm | standard,batched | fp8 | G(128),A,T | silu, gelu | 6 | Y | [`DeepGemmExperts`][vllm.model_executor.layers.fused_moe.deep_gemm_moe.DeepGemmExperts],[`BatchedDeepGemmExperts`][vllm.model_executor.layers.fused_moe.batched_deep_gemm_moe.BatchedDeepGemmExperts] |

@@ -105,7 +105,7 @@ To be used with a particular `FusedMoEPrepareAndFinalizeModular` subclass, MoE k

105 The following table shows "families" of modular kernels that are intended to work together. There are some combinations which may work but have not yet been tested, e.g. flashinfer with other fp8 experts. Note that the "naive" backend will work with any non-modular experts.

106

107 | backend | `FusedMoEPrepareAndFinalizeModular` subclasses | `FusedMoEExpertsModular` subclasses |

108 -|---------|-----------------------------------------|----------------------------------------------|

109 -| deepep_high_throughput | `DeepEPHTPrepareAndFinalize` | `DeepGemmExperts`,`TritonExperts`,`TritonOrDeepGemmExperts`,`CutlassExpertsFp8`, `MarlinExperts` |

110 -| deepep_low_latency | `DeepEPLLPrepareAndFinalize` | `BatchedDeepGemmExperts`,`BatchedTritonExperts`,`CutlassBatchedExpertsFp8`,`BatchedMarlinExperts` |

108 +| ------- | ---------------------------------------------- | ----------------------------------- |

109 +| deepep_high_throughput | `DeepEPHTPrepareAndFinalize` | `DeepGemmExperts`,`TritonExperts`,`TritonOrDeepGemmExperts`,`CutlassExpertsFp8`, `MarlinExperts` |

110 +| deepep_low_latency | `DeepEPLLPrepareAndFinalize` | `BatchedDeepGemmExperts`,`BatchedTritonExperts`,`CutlassBatchedExpertsFp8`,`BatchedMarlinExperts` |

111 | flashinfer | `FlashInferCutlassMoEPrepareAndFinalize` | `FlashInferExperts` |

docs/features/README.md

+18 -18

@@ -37,7 +37,7 @@ th:not(:first-child) {

37 </style>

40 -|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

40 +| - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |

41 | [CP](../configuration/optimization.md#chunked-prefill) | ✅ | | | | | | | | | | | | | | |

42 | [APC](automatic_prefix_caching.md) | ✅ | ✅ | | | | | | | | | | | | | |

43 | [LoRA](lora.md) | ✅ | ✅ | ✅ | | | | | | | | | | | | |

@@ -59,23 +59,23 @@ th:not(:first-child) {

60 ### Feature x Hardware

63 -|-----------------------------------------------------------|---------------------|-----------|-----------|--------|------------|--------------------|--------| ------------|

64 -| [CP](../configuration/optimization.md#chunked-prefill) | [❌](https://github.com/vllm-project/vllm/issues/2729) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

65 -| [APC](automatic_prefix_caching.md) | [❌](https://github.com/vllm-project/vllm/issues/3687) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

66 -| [LoRA](lora.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

67 -| [SD](speculative_decoding/README.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |

68 -| CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | [❌](https://github.com/vllm-project/vllm/issues/26970) |

69 -| [pooling](../models/pooling_models.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

70 -| <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |

71 -| [mm](multimodal_inputs.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

72 -| [prompt-embeds](prompt_embeds.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |

73 -| <abbr title="Logprobs">logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

74 -| <abbr title="Prompt Logprobs">prmpt logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

75 -| <abbr title="Async Output Processing">async output</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |

76 -| multi-step | ✅ | ✅ | ✅ | ✅ | ✅ | [❌](https://github.com/vllm-project/vllm/issues/8477) | ✅ | ✅ |

77 -| best-of | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

78 -| beam-search | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

63 +| ------- | ----- | ------ | ------ | --- | ------ | --- | --- | --------- |

64 +| [CP](../configuration/optimization.md#chunked-prefill) | [❌](https://github.com/vllm-project/vllm/issues/2729) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

65 +| [APC](automatic_prefix_caching.md) | [❌](https://github.com/vllm-project/vllm/issues/3687) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

66 +| [LoRA](lora.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

67 +| [SD](speculative_decoding/README.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |

68 +| CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | [❌](https://github.com/vllm-project/vllm/issues/26970) |

69 +| [pooling](../models/pooling_models.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

70 +| <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |

71 +| [mm](multimodal_inputs.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

72 +| [prompt-embeds](prompt_embeds.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ |

73 +| <abbr title="Logprobs">logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

74 +| <abbr title="Prompt Logprobs">prmpt logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

75 +| <abbr title="Async Output Processing">async output</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |

76 +| multi-step | ✅ | ✅ | ✅ | ✅ | ✅ | [❌](https://github.com/vllm-project/vllm/issues/8477) | ✅ | ✅ |

77 +| best-of | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

78 +| beam-search | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

80 !!! note

81 For information on feature support on Google TPU, please refer to the [TPU-Inference Recommended Models and Features](https://docs.vllm.ai/projects/tpu/en/latest/recommended_models_features/) documentation.

docs/features/interleaved_thinking.md

+3 -3

@@ -20,9 +20,9 @@ With interleaved thinking, the model can:

20 vLLM currently supports the following interleaved thinking models:

22 | Model Series | Reasoning Parser Name |

23 -|--------------|-----------------------|

24 -| moonshotai/Kimi-K2-Thinking | kimi_k2 |

25 -| MiniMaxAI/MiniMax-M2 | minimax_m2 |

23 +| ------------ | --------------------- |

24 +| moonshotai/Kimi-K2-Thinking | kimi_k2 |

25 +| MiniMaxAI/MiniMax-M2 | minimax_m2 |

27 ## Example Usage

docs/features/quantization/README.md

+11 -11

@@ -44,16 +44,16 @@ th:not(:first-child) {

44 }

45 </style>

48 -|-----------------------|---------|----------|----------|-------|----------|-----------|-------------|-----------|

49 -| AWQ | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ |

50 -| GPTQ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ |

51 -| Marlin (GPTQ/AWQ/FP8/FP4) | ❌ | ✅︎* | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |

52 -| INT8 (W8A8) | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ✅︎ |

53 -| FP8 (W8A8) | ❌ | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ |

54 -| bitsandbytes | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |

55 -| DeepSpeedFP | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |

56 -| GGUF | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ |

48 +| ------------------------- | ----- | ------ | ------ | --- | ------ | ------- | --------- | ------- |

49 +| AWQ | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ |

50 +| GPTQ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ |

51 +| Marlin (GPTQ/AWQ/FP8/FP4) | ❌ | ✅︎* | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |

52 +| INT8 (W8A8) | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ✅︎ |

53 +| FP8 (W8A8) | ❌ | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ |

54 +| bitsandbytes | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |

55 +| DeepSpeedFP | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |

56 +| GGUF | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ |

58 - Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.

59 - ✅︎ indicates that the quantization method is supported on the specified hardware.

@@ -131,7 +131,7 @@ class MyQuantConfig(QuantizationConfig):

131 Your custom `QuantizationConfig` subclass must implement these abstract methods:

132

133 | Method | Description |

134 -|--------|-------------|

134 +| ------ | ----------- |

135 | `get_name()` | Returns the name of the quantization method |

136 | `get_supported_act_dtypes()` | Returns list of supported activation dtypes (e.g., `torch.float16`) |

137 | `get_min_capability()` | Returns minimum GPU compute capability (e.g., 80 for Ampere, -1 for no restriction) |

docs/features/quantization/fp8.md

+1 -1

@@ -114,7 +114,7 @@ Here's an example of the resulting scores:

114

115 ```text

117 -|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|

117 +| --- |------:| -------------- |-----:| --------- | - |----:| - |-----:|

118 |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.768|± |0.0268|

119 | | |strict-match | 5|exact_match|↑ |0.768|± |0.0268|

120 ```

docs/features/reasoning_outputs.md

+1 -1

@@ -12,7 +12,7 @@ Reasoning models return an additional `reasoning` field in their outputs, which

12 vLLM currently supports the following reasoning models:

15 -|--------------|-------------|------------------|-------------|

15 +| ------------ | ----------- | ---------------- | ----------- |

docs/getting_started/installation/cpu.apple.inc.md

+18 -17

@@ -1,4 +1,5 @@

1 -# --8<-- [start:installation]

1 +

2 +--8<-- [start:installation]

4 vLLM has experimental support for macOS with Apple Silicon. For now, users must build from source to natively run on macOS.

@@ -7,23 +8,23 @@ Currently the CPU implementation for macOS supports FP32 and FP16 datatypes.

8 !!! tip "GPU-Accelerated Inference with vLLM-Metal"

9 For GPU-accelerated inference on Apple Silicon using Metal, check out [vllm-metal](https://github.com/vllm-project/vllm-metal), a community-maintained hardware plugin that uses MLX as the compute backend.

10 -# --8<-- [end:installation]

11 -# --8<-- [start:requirements]

11 +--8<-- [end:installation]

12 +--8<-- [start:requirements]

14 - OS: `macOS Sonoma` or later

15 - SDK: `XCode 15.4` or later with Command Line Tools

16 - Compiler: `Apple Clang >= 15.0.0`

17 -# --8<-- [end:requirements]

18 -# --8<-- [start:set-up-using-python]

18 +--8<-- [end:requirements]

19 +--8<-- [start:set-up-using-python]

20 -# --8<-- [end:set-up-using-python]

21 -# --8<-- [start:pre-built-wheels]

21 +--8<-- [end:set-up-using-python]

22 +--8<-- [start:pre-built-wheels]

24 Currently, there are no pre-built Apple silicon CPU wheels.

25 -# --8<-- [end:pre-built-wheels]

26 -# --8<-- [start:build-wheel-from-source]

26 +--8<-- [end:pre-built-wheels]

27 +--8<-- [start:build-wheel-from-source]

29 After installation of XCode and the Command Line Tools, which include Apple Clang, execute the following commands to build and install vLLM from source.

@@ -36,7 +37,7 @@ uv pip install -e .

38 !!! tip

39 The `--index-strategy unsafe-best-match` flag is needed to resolve dependencies across multiple package indexes (PyTorch CPU index and PyPI). Without this flag, you may encounter `typing-extensions` version conflicts.

39 -

40 +

41 The term "unsafe" refers to the package resolution strategy, not security. By default, `uv` only searches the first index where a package is found to prevent dependency confusion attacks. This flag allows `uv` to search all configured indexes to find the best compatible versions. Since both PyTorch and PyPI are trusted package sources, using this strategy is safe and appropriate for vLLM installation.

43 !!! note

@@ -77,14 +78,14 @@ uv pip install -e .

78 ```

79 On Apple Clang 16 you should see: `#define __cplusplus 201703L`

80 -# --8<-- [end:build-wheel-from-source]

81 -# --8<-- [start:pre-built-images]

81 +--8<-- [end:build-wheel-from-source]

82 +--8<-- [start:pre-built-images]

84 Currently, there are no pre-built Arm silicon CPU images.

85 -# --8<-- [end:pre-built-images]

86 -# --8<-- [start:build-image-from-source]

86 +--8<-- [end:pre-built-images]

87 +--8<-- [start:build-image-from-source]

88 -# --8<-- [end:build-image-from-source]

89 -# --8<-- [start:extra-information]

90 -# --8<-- [end:extra-information]

89 +--8<-- [end:build-image-from-source]

90 +--8<-- [start:extra-information]

91 +--8<-- [end:extra-information]

docs/getting_started/installation/cpu.arm.inc.md

+27 -25

@@ -1,19 +1,20 @@

1 -# --8<-- [start:installation]

1 +

2 +--8<-- [start:installation]

4 vLLM offers basic model inferencing and serving on Arm CPU platform, with support for NEON, data types FP32, FP16 and BF16.

5 -# --8<-- [end:installation]

6 -# --8<-- [start:requirements]

6 +--8<-- [end:installation]

7 +--8<-- [start:requirements]

9 - OS: Linux

10 - Compiler: `gcc/g++ >= 12.3.0` (optional, recommended)

11 - Instruction Set Architecture (ISA): NEON support is required

12 -# --8<-- [end:requirements]

13 -# --8<-- [start:set-up-using-python]

13 +--8<-- [end:requirements]

14 +--8<-- [start:set-up-using-python]

15 -# --8<-- [end:set-up-using-python]

16 -# --8<-- [start:pre-built-wheels]

16 +--8<-- [end:set-up-using-python]

17 +--8<-- [start:pre-built-wheels]

19 Pre-built vLLM wheels for Arm are available since version 0.11.2. These wheels contain pre-compiled C++ binaries.

@@ -43,13 +44,14 @@ uv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VE

45 The `uv` approach works for vLLM `v0.6.6` and later. A unique feature of `uv` is that packages in `--extra-index-url` have [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes). If the latest public release is `v0.6.6.post1`, `uv`'s behavior allows installing a commit before `v0.6.6.post1` by specifying the `--extra-index-url`. In contrast, `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install a development version prior to the released version.

46 -**Install the latest code**

47 +#### Install the latest code

49 LLM inference is a fast-evolving field, and the latest code may contain bug fixes, performance improvements, and new features that are not released yet. To allow users to try the latest code without waiting for the next release, vLLM provides working pre-built Arm CPU wheels for every commit since `v0.11.2` on <https://wheels.vllm.ai/nightly>. For native CPU wheels, this index should be used:

50 -* `https://wheels.vllm.ai/nightly/cpu/vllm`

51 +- `https://wheels.vllm.ai/nightly/cpu/vllm`

53 To install from nightly index, run:

54 +

55 ```bash

56 uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu --index-strategy first-index

57 ```

@@ -64,7 +66,7 @@ uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu --index

66 pip install https://wheels.vllm.ai/4fa7ce46f31cbd97b4651694caf9991cc395a259/vllm-0.13.0rc2.dev104%2Bg4fa7ce46f.cpu-cp38-abi3-manylinux_2_35_aarch64.whl # current nightly build (the filename will change!)

67 ```

67 -**Install specific revisions**

69 +#### Install specific revisions

71 If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL:

@@ -73,8 +75,8 @@ export VLLM_COMMIT=730bd35378bf2a5b56b6d3a45be28b3092d26519 # use full commit ha

75 uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}/cpu --index-strategy first-index

76 ```

76 -# --8<-- [end:pre-built-wheels]

77 -# --8<-- [start:build-wheel-from-source]

78 +--8<-- [end:pre-built-wheels]

79 +--8<-- [start:build-wheel-from-source]

81 First, install the recommended compiler. We recommend using `gcc/g++ >= 12.3.0` as the default compiler to avoid potential problems. For example, on Ubuntu 22.4, you can run:

@@ -133,8 +135,8 @@ Testing has been conducted on AWS Graviton3 instances for compatibility.

135 export LD_PRELOAD="$TC_PATH:$LD_PRELOAD"

136 ```

137

136 -# --8<-- [end:build-wheel-from-source]

137 -# --8<-- [start:pre-built-images]

138 +--8<-- [end:build-wheel-from-source]

139 +--8<-- [start:pre-built-images]

140

141 To pull the latest image from Docker Hub:

142

@@ -170,10 +172,10 @@ export VLLM_COMMIT=6299628d326f429eba78736acb44e76749b281f5 # use full commit ha

172 docker pull public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:${VLLM_COMMIT}-arm64-cpu

173 ```

174

173 -# --8<-- [end:pre-built-images]

174 -# --8<-- [start:build-image-from-source]

175 +--8<-- [end:pre-built-images]

176 +--8<-- [start:build-image-from-source]

177

176 -## Building for your target ARM CPU

178 +#### Building for your target ARM CPU

179

180 ```bash

181 docker build -f docker/Dockerfile.cpu \

@@ -189,9 +191,9 @@ docker build -f docker/Dockerfile.cpu \

191 - `VLLM_CPU_ARM_BF16=true` - Force-enable ARM BF16 support (build with BF16 regardless of build system capabilities)

192 - `VLLM_CPU_ARM_BF16=false` - Rely on auto-detection (default)

193

192 -### Examples

194 +##### Examples

195

194 -**Auto-detection build (native ARM)**

196 +###### Auto-detection build (native ARM)

197

198 ```bash

199 # Building on ARM64 system - platform auto-detected

@@ -200,7 +202,7 @@ docker build -f docker/Dockerfile.cpu \

202 --target vllm-openai .

203 ```

204

203 -**Cross-compile for ARM with BF16 support**

205 +###### Cross-compile for ARM with BF16 support

206

207 ```bash

208 # Building on ARM64 for newer ARM CPUs with BF16

@@ -210,7 +212,7 @@ docker build -f docker/Dockerfile.cpu \

212 --target vllm-openai .

213 ```

214

213 -**Cross-compile from x86_64 to ARM64 with BF16**

215 +###### Cross-compile from x86_64 to ARM64 with BF16

216

217 ```bash

218 # Requires Docker buildx with ARM emulation (QEMU)

@@ -226,7 +228,7 @@ docker buildx build -f docker/Dockerfile.cpu \

228 !!! note "ARM BF16 requirements"

229 ARM BF16 support requires ARMv8.6-A or later (FEAT_BF16). Supported on AWS Graviton3/4, AmpereOne, and other recent ARM processors.

230

229 -## Launching the OpenAI server

231 +#### Launching the OpenAI server

232

233 ```bash

234 docker run --rm \

@@ -245,6 +247,6 @@ docker run --rm \

247 !!! tip "Alternative to --privileged"

248 Instead of `--privileged=true`, use `--cap-add SYS_NICE --security-opt seccomp=unconfined` for better security.

249

248 -# --8<-- [end:build-image-from-source]

249 -# --8<-- [start:extra-information]

250 -# --8<-- [end:extra-information]

250 +--8<-- [end:build-image-from-source]

251 +--8<-- [start:extra-information]

252 +--8<-- [end:extra-information]

docs/getting_started/installation/cpu.md

@@ -1,3 +1,7 @@

1 +---

2 +toc_depth: 3

3 +---

4 +

5 # CPU

7 vLLM is a Python library that supports the following CPU variants. Select your CPU type to see vendor specific instructions:

docs/getting_started/installation/cpu.s390x.inc.md

+17 -16

@@ -1,27 +1,28 @@

1 -# --8<-- [start:installation]

1 +

2 +--8<-- [start:installation]

4 vLLM has experimental support for s390x architecture on IBM Z platform. For now, users must build from source to natively run on IBM Z platform.

6 Currently, the CPU implementation for s390x architecture supports FP32 datatype only.

7 -# --8<-- [end:installation]

8 -# --8<-- [start:requirements]

8 +--8<-- [end:installation]

9 +--8<-- [start:requirements]

11 - OS: `Linux`

12 - SDK: `gcc/g++ >= 12.3.0` or later with Command Line Tools

13 - Instruction Set Architecture (ISA): VXE support is required. Works with Z14 and above.

14 - Build install python packages: `pyarrow`, `torch` and `torchvision`

15 -# --8<-- [end:requirements]

16 -# --8<-- [start:set-up-using-python]

16 +--8<-- [end:requirements]

17 +--8<-- [start:set-up-using-python]

18 -# --8<-- [end:set-up-using-python]

19 -# --8<-- [start:pre-built-wheels]

19 +--8<-- [end:set-up-using-python]

20 +--8<-- [start:pre-built-wheels]

22 Currently, there are no pre-built IBM Z CPU wheels.

23 -# --8<-- [end:pre-built-wheels]

24 -# --8<-- [start:build-wheel-from-source]

24 +--8<-- [end:pre-built-wheels]

25 +--8<-- [start:build-wheel-from-source]

27 Install the following packages from the package manager before building the vLLM. For example on RHEL 9.4:

@@ -65,13 +66,13 @@ Execute the following commands to build and install vLLM from source.

66 pip install dist/*.whl

67 ```

68 -# --8<-- [end:build-wheel-from-source]

69 -# --8<-- [start:pre-built-images]

69 +--8<-- [end:build-wheel-from-source]

70 +--8<-- [start:pre-built-images]

72 Currently, there are no pre-built IBM Z CPU images.

73 -# --8<-- [end:pre-built-images]

74 -# --8<-- [start:build-image-from-source]

74 +--8<-- [end:pre-built-images]

75 +--8<-- [start:build-image-from-source]

77 ```bash

78 docker build -f docker/Dockerfile.s390x \

@@ -93,6 +94,6 @@ docker run --rm \

94 !!! tip

95 An alternative of `--privileged true` is `--cap-add SYS_NICE --security-opt seccomp=unconfined`.

96 -# --8<-- [end:build-image-from-source]

97 -# --8<-- [start:extra-information]

98 -# --8<-- [end:extra-information]

97 +--8<-- [end:build-image-from-source]

98 +--8<-- [start:extra-information]

99 +--8<-- [end:extra-information]

docs/getting_started/installation/cpu.x86.inc.md

+26 -24

@@ -1,9 +1,10 @@

1 -# --8<-- [start:installation]

1 +

2 +--8<-- [start:installation]

4 vLLM supports basic model inferencing and serving on x86 CPU platform, with data types FP32, FP16 and BF16.

5 -# --8<-- [end:installation]

6 -# --8<-- [start:requirements]

6 +--8<-- [end:installation]

7 +--8<-- [start:requirements]

9 - OS: Linux

10 - CPU flags: `avx512f` (Recommended), `avx512_bf16` (Optional), `avx512_vnni` (Optional)

@@ -11,11 +12,11 @@ vLLM supports basic model inferencing and serving on x86 CPU platform, with data

12 !!! tip

13 Use `lscpu` to check the CPU flags.

14 -# --8<-- [end:requirements]

15 -# --8<-- [start:set-up-using-python]

15 +--8<-- [end:requirements]

16 +--8<-- [start:set-up-using-python]

17 -# --8<-- [end:set-up-using-python]

18 -# --8<-- [start:pre-built-wheels]

18 +--8<-- [end:set-up-using-python]

19 +--8<-- [start:pre-built-wheels]

21 Pre-built vLLM wheels for x86 with AVX512 are available since version 0.13.0. To install release wheels:

@@ -25,6 +26,7 @@ export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/rel

26 # use uv

27 uv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cpu-cp38-abi3-manylinux_2_35_x86_64.whl --torch-backend cpu

28 ```

29 +

30 ??? console "pip"

31 ```bash

32 # use pip

@@ -46,7 +48,7 @@ uv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VE

48 export LD_PRELOAD="$TC_PATH:$IOMP_PATH:$LD_PRELOAD"

49 ```

49 -**Install the latest code**

51 +#### Install the latest code

53 To install the wheel built from the latest main branch:

@@ -54,7 +56,7 @@ To install the wheel built from the latest main branch:

56 uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu --index-strategy first-index --torch-backend cpu

57 ```

57 -**Install specific revisions**

59 +#### Install specific revisions

61 If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL:

@@ -63,8 +65,8 @@ export VLLM_COMMIT=730bd35378bf2a5b56b6d3a45be28b3092d26519 # use full commit ha

65 uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}/cpu --index-strategy first-index --torch-backend cpu

66 ```

66 -# --8<-- [end:pre-built-wheels]

67 -# --8<-- [start:build-wheel-from-source]

68 +--8<-- [end:pre-built-wheels]

69 +--8<-- [start:build-wheel-from-source]

71 Install recommended compiler. We recommend to use `gcc/g++ >= 12.3.0` as the default compiler to avoid potential problems. For example, on Ubuntu 22.4, you can run:

@@ -158,8 +160,8 @@ uv pip install dist/*.whl

160 ]

161 ```

162

161 -# --8<-- [end:build-wheel-from-source]

162 -# --8<-- [start:pre-built-images]

163 +--8<-- [end:build-wheel-from-source]

164 +--8<-- [start:pre-built-images]

165

166 You can pull the latest available CPU image from Docker Hub:

167

@@ -189,10 +191,10 @@ vllm/vllm-openai-cpu:latest-x86_64 <args...>

191 !!! warning

192 If deploying the pre-built images on machines without `avx512f`, `avx512_bf16`, or `avx512_vnni` support, an `Illegal instruction` error may be raised. See the build-image-from-source section below for build arguments to match your target CPU capabilities.

193

192 -# --8<-- [end:pre-built-images]

193 -# --8<-- [start:build-image-from-source]

194 +--8<-- [end:pre-built-images]

195 +--8<-- [start:build-image-from-source]

196

195 -## Building for your target CPU

197 +#### Building for your target CPU

198

199 ```bash

200 docker build -f docker/Dockerfile.cpu \

@@ -212,15 +214,15 @@ docker build -f docker/Dockerfile.cpu \

214 - `VLLM_CPU_{ISA}=true` - Force-enable the instruction set (build with ISA regardless of build system capabilities)

215 - `VLLM_CPU_{ISA}=false` - Rely on auto-detection (default)

216

215 -### Examples

217 +##### Examples

218

217 -**Auto-detection build (default)**

219 +###### Auto-detection build (default)

220

221 ```bash

222 docker build -f docker/Dockerfile.cpu --tag vllm-cpu-env --target vllm-openai .

223 ```

224

223 -**Cross-compile for AVX512**

225 +###### Cross-compile for AVX512

226

227 ```bash

228 docker build -f docker/Dockerfile.cpu \

@@ -231,7 +233,7 @@ docker build -f docker/Dockerfile.cpu \

233 --target vllm-openai .

234 ```

235

234 -**Cross-compile for AVX2**

236 +###### Cross-compile for AVX2

237

238 ```bash

239 docker build -f docker/Dockerfile.cpu \

@@ -240,7 +242,7 @@ docker build -f docker/Dockerfile.cpu \

242 --target vllm-openai .

243 ```

244

243 -## Launching the OpenAI server

245 +#### Launching the OpenAI server

246

247 ```bash

248 docker run --rm \

@@ -255,6 +257,6 @@ docker run --rm \

257 other vLLM OpenAI server arguments

258 ```

259

258 -# --8<-- [end:build-image-from-source]

259 -# --8<-- [start:extra-information]

260 -# --8<-- [end:extra-information]

\ No newline at end of file

260 +--8<-- [end:build-image-from-source]

261 +--8<-- [start:extra-information]

262 +--8<-- [end:extra-information]

docs/getting_started/installation/gpu.cuda.inc.md

+22 -21

@@ -1,14 +1,15 @@

1 -# --8<-- [start:installation]

1 +

2 +--8<-- [start:installation]

4 vLLM contains pre-compiled C++ and CUDA (12.8) binaries.

5 -# --8<-- [end:installation]

6 -# --8<-- [start:requirements]

6 +--8<-- [end:installation]

7 +--8<-- [start:requirements]

9 - GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)

10 -# --8<-- [end:requirements]

11 -# --8<-- [start:set-up-using-python]

11 +--8<-- [end:requirements]

12 +--8<-- [start:set-up-using-python]

14 !!! note

15 PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <https://github.com/vllm-project/vllm/issues/8420> for more details.

@@ -17,8 +18,8 @@ In order to be performant, vLLM has to compile many cuda kernels. The compilatio

19 Therefore, it is recommended to install vLLM with a **fresh new** environment. If either you have a different CUDA version or you want to use an existing PyTorch installation, you need to build vLLM from source. See [below](#build-wheel-from-source) for more details.

20 -# --8<-- [end:set-up-using-python]

21 -# --8<-- [start:pre-built-wheels]

21 +--8<-- [end:set-up-using-python]

22 +--8<-- [start:pre-built-wheels]

24 ```bash

25 uv pip install vllm --torch-backend=auto

@@ -49,8 +50,8 @@ uv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VE

51 LLM inference is a fast-evolving field, and the latest code may contain bug fixes, performance improvements, and new features that are not released yet. To allow users to try the latest code without waiting for the next release, vLLM provides wheels for every commit since `v0.5.3` on <https://wheels.vllm.ai/nightly>. There are multiple indices that could be used:

52 -* `https://wheels.vllm.ai/nightly`: the default variant (CUDA with version specified in `VLLM_MAIN_CUDA_VERSION`) built with the last commit on the `main` branch. Currently it is CUDA 12.9.

53 -* `https://wheels.vllm.ai/nightly/<variant>`: all other variants. Now this includes `cu130`, and `cpu`. The default variant (`cu129`) also has a subdirectory to keep consistency.

53 +- `https://wheels.vllm.ai/nightly`: the default variant (CUDA with version specified in `VLLM_MAIN_CUDA_VERSION`) built with the last commit on the `main` branch. Currently it is CUDA 12.9.

54 +- `https://wheels.vllm.ai/nightly/<variant>`: all other variants. Now this includes `cu130`, and `cpu`. The default variant (`cu129`) also has a subdirectory to keep consistency.

56 To install from nightly index, run:

@@ -82,8 +83,8 @@ uv pip install vllm \

83 --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT} # add variant subdirectory here if needed

84 ```

85 -# --8<-- [end:pre-built-wheels]

86 -# --8<-- [start:build-wheel-from-source]

86 +--8<-- [end:pre-built-wheels]

87 +--8<-- [start:build-wheel-from-source]

89 #### Set up using Python-only build (without compilation) {#python-only-build}

@@ -116,9 +117,9 @@ uv pip install --editable .

117

118 There are more environment variables to control the behavior of Python-only build:

119

119 -* `VLLM_PRECOMPILED_WHEEL_LOCATION`: specify the exact wheel URL or local file path of a pre-compiled wheel to use. All other logic to find the wheel will be skipped.

120 -* `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the commit hash to download the pre-compiled wheel. It can be `nightly` to use the last **already built** commit on the main branch.

121 -* `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the variant subdirectory to use on the nightly index, e.g., `cu129`, `cu130`, `cpu`. If not specified, the variant is auto-detected based on your system's CUDA version (from PyTorch or nvidia-smi). You can also set `VLLM_MAIN_CUDA_VERSION` to override auto-detection.

120 +- `VLLM_PRECOMPILED_WHEEL_LOCATION`: specify the exact wheel URL or local file path of a pre-compiled wheel to use. All other logic to find the wheel will be skipped.

121 +- `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the commit hash to download the pre-compiled wheel. It can be `nightly` to use the last **already built** commit on the main branch.

122 +- `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the variant subdirectory to use on the nightly index, e.g., `cu129`, `cu130`, `cpu`. If not specified, the variant is auto-detected based on your system's CUDA version (from PyTorch or nvidia-smi). You can also set `VLLM_MAIN_CUDA_VERSION` to override auto-detection.

123

124 You can find more information about vLLM's wheels in [Install the latest code](#install-the-latest-code).

125

@@ -236,8 +237,8 @@ export VLLM_TARGET_DEVICE=empty

237 uv pip install -e .

238 ```

239

239 -# --8<-- [end:build-wheel-from-source]

240 -# --8<-- [start:pre-built-images]

240 +--8<-- [end:build-wheel-from-source]

241 +--8<-- [start:pre-built-images]

242

243 vLLM offers an official Docker image for deployment.

244 The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai](https://hub.docker.com/r/vllm/vllm-openai/tags).

@@ -314,8 +315,8 @@ docker run --runtime nvidia --gpus all \

315

316 This will automatically configure `LD_LIBRARY_PATH` to point to the compatibility libraries before loading PyTorch and other dependencies.

317

317 -# --8<-- [end:pre-built-images]

318 -# --8<-- [start:build-image-from-source]

318 +--8<-- [end:pre-built-images]

319 +--8<-- [start:build-image-from-source]

320

321 You can build and run vLLM from source via the provided [docker/Dockerfile](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile). To build vLLM:

322

@@ -415,9 +416,9 @@ The argument `vllm/vllm-openai` specifies the image to run, and should be replac

416 !!! note

417 **For version 0.4.1 and 0.4.2 only** - the vLLM docker images under these versions are supposed to be run under the root user since a library under the root user's home directory, i.e. `/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` is required to be loaded during runtime. If you are running the container under a different user, you may need to first change the permissions of the library (and all the parent directories) to allow the user to access it, then run vLLM with environment variable `VLLM_NCCL_SO_PATH=/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` .

418

418 -# --8<-- [end:build-image-from-source]

419 -# --8<-- [start:supported-features]

419 +--8<-- [end:build-image-from-source]

420 +--8<-- [start:supported-features]

421

422 See [Feature x Hardware](../../features/README.md#feature-x-hardware) compatibility matrix for feature support information.

423

423 -# --8<-- [end:supported-features]

\ No newline at end of file

424 +--8<-- [end:supported-features]

docs/getting_started/installation/gpu.md

+4 -10

@@ -88,8 +88,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G

89 ### Pre-built images

91 -

92 -# --8<-- [start:pre-built-images]

91 +--8<-- [start:pre-built-images]

93 === "NVIDIA CUDA"

@@ -103,15 +102,11 @@ vLLM is a Python library that supports the following GPU variants. Select your G

102

103 --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:pre-built-images"

104

106 -# --8<-- [end:pre-built-images]

107 -

105 +--8<-- [end:pre-built-images]

106

109 -

107 ### Build image from source

111 -

108

113 -

114 -# --8<-- [start:build-image-from-source]

109 +--8<-- [start:build-image-from-source]

110

111 === "NVIDIA CUDA"

112

@@ -125,8 +120,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G

120

121 --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:build-image-from-source"

122

128 -# --8<-- [end:build-image-from-source]

129 -

123 +--8<-- [end:build-image-from-source]

124

125 ## Supported features

126

docs/getting_started/installation/gpu.rocm.inc.md

+18 -21

@@ -1,23 +1,24 @@

1 -# --8<-- [start:installation]

1 +

2 +--8<-- [start:installation]

4 vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 7.0.

5 -# --8<-- [end:installation]

6 -# --8<-- [start:requirements]

6 +--8<-- [end:installation]

7 +--8<-- [start:requirements]

9 - GPU: MI200s (gfx90a), MI300 (gfx942), MI350 (gfx950), Radeon RX 7900 series (gfx1100/1101), Radeon RX 9000 series (gfx1200/1201), Ryzen AI MAX / AI 300 Series (gfx1151/1150)

10 - ROCm 6.3 or above

11 - MI350 requires ROCm 7.0 or above

12 - Ryzen AI MAX / AI 300 Series requires ROCm 7.0.2 or above

13 -# --8<-- [end:requirements]

14 -# --8<-- [start:set-up-using-python]

14 +--8<-- [end:requirements]

15 +--8<-- [start:set-up-using-python]

17 The vLLM wheel bundles PyTorch and all required dependencies, and you should use the included PyTorch for compatibility. Because vLLM compiles many ROCm kernels to ensure a validated, high‑performance stack, the resulting binaries may not be compatible with other ROCm or PyTorch builds.

18 If you need a different ROCm version or want to use an existing PyTorch installation, you’ll need to build vLLM from source. See [below](#build-wheel-from-source) for more details.

19 -# --8<-- [end:set-up-using-python]

20 -# --8<-- [start:pre-built-wheels]

20 +--8<-- [end:set-up-using-python]

21 +--8<-- [start:pre-built-wheels]

23 To install the latest version of vLLM for Python 3.12, ROCm 7.0 and `glibc >= 2.35`.

@@ -34,7 +35,7 @@ To install a specific version and ROCm variant of vLLM wheel.

35 uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.15.0/rocm700

36 ```

37 -!!! warning "Caveats for using `pip`"

38 +!!! warning "Caveats for using `pip`"

40 We recommend leveraging `uv` to install vLLM wheel. Using `pip` to install from custom indices is cumbersome, because `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install wheel from custom index if exact versions of all packages are specified exactly. In contrast, `uv` gives the extra index [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes).

@@ -44,8 +45,8 @@ uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.15.0/rocm700

45 pip install vllm==0.15.0+rocm700 --extra-index-url https://wheels.vllm.ai/rocm/0.15.0/rocm700

46 ```

47 -# --8<-- [end:pre-built-wheels]

48 -# --8<-- [start:build-wheel-from-source]

48 +--8<-- [end:pre-built-wheels]

49 +--8<-- [start:build-wheel-from-source]

51 !!! tip

52 - If you found that the following installation step does not work for you, please refer to [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). Dockerfile is a form of installation steps.

@@ -104,7 +105,6 @@ uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.15.0/rocm700

105 !!! note

106 - The validated `$FA_BRANCH` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).

107

107 -

108 3. Optionally, if you choose to build AITER yourself to use a certain branch or commit, you can build AITER using the following steps:

109

110 ```bash

@@ -120,7 +120,6 @@ uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.15.0/rocm700

120 - You will need to config the `$AITER_BRANCH_OR_COMMIT` for your purpose.

121 - The validated `$AITER_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).

122

123 -

123 4. Optionally, if you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps:

124

125 ```bash

@@ -135,7 +134,6 @@ uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.15.0/rocm700

134 - You will need to config the `$MORI_BRANCH_OR_COMMIT` for your purpose.

135 - The validated `$MORI_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).

136

138 -

137 5. Build vLLM. For example, vLLM on ROCM 7.0 can be built with the following steps:

138

139 ???+ console "Commands"

@@ -171,8 +169,8 @@ uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.15.0/rocm700

169 - For MI300x (gfx942) users, to achieve optimal performance, please refer to [MI300x tuning guide](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html) for performance optimization and tuning tips on system and workflow level.

170 For vLLM, please refer to [vLLM performance optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html).

171

174 -# --8<-- [end:build-wheel-from-source]

175 -# --8<-- [start:pre-built-images]

172 +--8<-- [end:build-wheel-from-source]

173 +--8<-- [start:pre-built-images]

174

175 vLLM offers an official Docker image for deployment.

176 The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags).

@@ -217,8 +215,8 @@ rocm/vllm-dev:nightly

215 Please check [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/performance-validation/mi300x/vllm-benchmark.html)

216 for instructions on how to use this prebuilt docker image.

217

220 -# --8<-- [end:pre-built-images]

221 -# --8<-- [start:build-image-from-source]

218 +--8<-- [end:pre-built-images]

219 +--8<-- [start:build-image-from-source]

220

221 You can build and run vLLM from source via the provided [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm).

222

@@ -271,7 +269,6 @@ To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default (w

269 DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm/vllm-openai-rocm .

270 ```

271

274 -

272 To run vLLM with the custom-built Docker image:

273

274 ```bash

@@ -308,9 +305,9 @@ To use the docker image as base for development, you can launch it in interactiv

305 vllm/vllm-openai-rocm

306 ```

307

311 -# --8<-- [end:build-image-from-source]

312 -# --8<-- [start:supported-features]

308 +--8<-- [end:build-image-from-source]

309 +--8<-- [start:supported-features]

310

311 See [Feature x Hardware](../../features/README.md#feature-x-hardware) compatibility matrix for feature support information.

312

316 -# --8<-- [end:supported-features]

313 +--8<-- [end:supported-features]

docs/getting_started/installation/gpu.xpu.inc.md

+20 -19

@@ -1,29 +1,30 @@

1 -# --8<-- [start:installation]

1 +

2 +--8<-- [start:installation]

4 vLLM initially supports basic model inference and serving on Intel GPU platform.

5 -# --8<-- [end:installation]

6 -# --8<-- [start:requirements]

6 +--8<-- [end:installation]

7 +--8<-- [start:requirements]

9 - Supported Hardware: Intel Data Center GPU, Intel ARC GPU

10 - OneAPI requirements: oneAPI 2025.3

10 -- Dependency: [vllm-xpu-kernels](https://github.com/vllm-project/vllm-xpu-kernels): a package provide all necessary vllm custom kernel when running vLLM on Intel GPU platform,

11 +- Dependency: [vllm-xpu-kernels](https://github.com/vllm-project/vllm-xpu-kernels): a package provide all necessary vllm custom kernel when running vLLM on Intel GPU platform,

12 - Python: 3.12

13 !!! warning

14 The provided vllm-xpu-kernels whl is Python3.12 specific so this version is a MUST.

15 -# --8<-- [end:requirements]

16 -# --8<-- [start:set-up-using-python]

16 +--8<-- [end:requirements]

17 +--8<-- [start:set-up-using-python]

19 There is no extra information on creating a new Python environment for this device.

20 -# --8<-- [end:set-up-using-python]

21 -# --8<-- [start:pre-built-wheels]

21 +--8<-- [end:set-up-using-python]

22 +--8<-- [start:pre-built-wheels]

24 Currently, there are no pre-built XPU wheels.

25 -# --8<-- [end:pre-built-wheels]

26 -# --8<-- [start:build-wheel-from-source]

26 +--8<-- [end:pre-built-wheels]

27 +--8<-- [start:build-wheel-from-source]

29 - First, install required [driver](https://dgpu-docs.intel.com/driver/installation.html#installing-gpu-drivers) and [Intel OneAPI](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html) 2025.3 or later.

30 - Second, install Python packages for vLLM XPU backend building:

@@ -54,13 +55,13 @@ pip install -v -r requirements/xpu.txt

55 VLLM_TARGET_DEVICE=xpu pip install --no-build-isolation -e . -v

56 ```

57 -# --8<-- [end:build-wheel-from-source]

58 -# --8<-- [start:pre-built-images]

58 +--8<-- [end:build-wheel-from-source]

59 +--8<-- [start:pre-built-images]

61 Currently, we release prebuilt XPU images at docker [hub](https://hub.docker.com/r/intel/vllm/tags) based on vLLM released version. For more information, please refer release [note](https://github.com/intel/ai-containers/blob/main/vllm).

62 -# --8<-- [end:pre-built-images]

63 -# --8<-- [start:build-image-from-source]

63 +--8<-- [end:pre-built-images]

64 +--8<-- [start:build-image-from-source]

66 ```bash

67 docker build -f docker/Dockerfile.xpu -t vllm-xpu-env --shm-size=4g .

@@ -74,8 +75,8 @@ docker run -it \

75 vllm-xpu-env

76 ```

77 -# --8<-- [end:build-image-from-source]

78 -# --8<-- [start:supported-features]

78 +--8<-- [end:build-image-from-source]

79 +--8<-- [start:supported-features]

81 XPU platform supports **tensor parallel** inference/serving and also supports **pipeline parallel** as a beta feature for online serving. For **pipeline parallel**, we support it on single node with mp as the backend. For example, a reference execution like following:

@@ -90,9 +91,9 @@ vllm serve facebook/opt-13b \

92 By default, a ray instance will be launched automatically if no existing one is detected in the system, with `num-gpus` equals to `parallel_config.world_size`. We recommend properly starting a ray cluster before execution, referring to the [examples/online_serving/run_cluster.sh](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/run_cluster.sh) helper script.

93 -# --8<-- [end:supported-features]

94 -# --8<-- [start:distributed-backend]

94 +--8<-- [end:supported-features]

95 +--8<-- [start:distributed-backend]

97 XPU platform uses **torch-ccl** for torch<2.8 and **xccl** for torch>=2.8 as distributed backend, since torch 2.8 supports **xccl** as built-in backend for XPU.

98 -# --8<-- [end:distributed-backend]

99 +--8<-- [end:distributed-backend]

docs/getting_started/installation/python_env_setup.inc.md

@@ -1,3 +1,4 @@

1 +

2 It's recommended to use [uv](https://docs.astral.sh/uv/), a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment using the following commands:

4 ```bash

docs/models/hardware_supported_models/cpu.md

+16 -16

@@ -2,32 +2,32 @@

3 ## Validated Hardware

5 -| Hardware |

6 -| ----------------------------------------- |

7 -| [Intel® Xeon® 6 Processors](https://www.intel.com/content/www/us/en/products/details/processors/xeon.html) |

8 -| [Intel® Xeon® 5 Processors](https://www.intel.com/content/www/us/en/products/docs/processors/xeon/5th-gen-xeon-scalable-processors.html) |

5 +| Hardware |

6 +| -------- |

7 +| [Intel® Xeon® 6 Processors](https://www.intel.com/content/www/us/en/products/details/processors/xeon.html) |

8 +| [Intel® Xeon® 5 Processors](https://www.intel.com/content/www/us/en/products/docs/processors/xeon/5th-gen-xeon-scalable-processors.html) |

10 ## Recommended Models

12 ### Text-only Language Models

14 | Model | Architecture | Supported |

15 -|--------------------------------------|-------------------------------------------|-----------|

16 -| meta-llama/Llama-3.1-8B-Instruct | LlamaForCausalLM | ✅ |

17 -| meta-llama/Llama-3.2-3B-Instruct | LlamaForCausalLM | ✅ |

18 -| ibm-granite/granite-3.2-2b-instruct | GraniteForCausalLM | ✅ |

19 -| Qwen/Qwen3-1.7B | Qwen3ForCausalLM | ✅ |

20 -| Qwen/Qwen3-4B | Qwen3ForCausalLM | ✅ |

21 -| Qwen/Qwen3-8B | Qwen3ForCausalLM | ✅ |

22 -| zai-org/glm-4-9b-hf | GLMForCausalLM | ✅ |

23 -| google/gemma-7b | GemmaForCausalLM | ✅ |

15 +| ------------------------------------ | ---------------------------------------- | --------- |

16 +| meta-llama/Llama-3.1-8B-Instruct | LlamaForCausalLM | ✅ |

17 +| meta-llama/Llama-3.2-3B-Instruct | LlamaForCausalLM | ✅ |

18 +| ibm-granite/granite-3.2-2b-instruct | GraniteForCausalLM | ✅ |

19 +| Qwen/Qwen3-1.7B | Qwen3ForCausalLM | ✅ |

20 +| Qwen/Qwen3-4B | Qwen3ForCausalLM | ✅ |

21 +| Qwen/Qwen3-8B | Qwen3ForCausalLM | ✅ |

22 +| zai-org/glm-4-9b-hf | GLMForCausalLM | ✅ |

23 +| google/gemma-7b | GemmaForCausalLM | ✅ |

25 ### Multimodal Language Models

27 | Model | Architecture | Supported |

28 -|--------------------------------------|-------------------------------------------|-----------|

29 -| Qwen/Qwen2.5-VL-7B-Instruct | Qwen2VLForConditionalGeneration | ✅ |

30 -| openai/whisper-large-v3 | WhisperForConditionalGeneration | ✅ |

28 +| ------------------------------------ | ---------------------------------------- | --------- |

29 +| Qwen/Qwen2.5-VL-7B-Instruct | Qwen2VLForConditionalGeneration | ✅ |

30 +| openai/whisper-large-v3 | WhisperForConditionalGeneration | ✅ |

32 ✅ Runs and optimized.

33 🟨 Runs and correct but not optimized to green yet.

docs/models/hardware_supported_models/xpu.md

+40 -40

@@ -2,9 +2,9 @@

3 ## Validated Hardware

5 -| Hardware |

6 -| ----------------------------------------- |

7 -| [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/products/docs/discrete-gpus/arc/workstations/b-series/overview.html) |

5 +| Hardware |

6 +| -------- |

7 +| [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/products/docs/discrete-gpus/arc/workstations/b-series/overview.html) |

9 ## Recommended Models

@@ -12,53 +12,53 @@

14 | ----------------------------------------- | ---------------------------------------------------- | ---- | ----------- | ----- |

15 -| openai/gpt-oss-20b | GPTForCausalLM | | | ✅ |

16 -| openai/gpt-oss-120b | GPTForCausalLM | | | ✅ |

17 -| deepseek-ai/DeepSeek-R1-Distill-Llama-8B | LlamaForCausalLM | ✅ | ✅ | |

18 -| deepseek-ai/DeepSeek-R1-Distill-Qwen-14B | QwenForCausalLM | ✅ | ✅ | |

19 -| deepseek-ai/DeepSeek-R1-Distill-Qwen-32B | QwenForCausalLM | ✅ | ✅ | |

20 -| deepseek-ai/DeepSeek-R1-Distill-Llama-70B | LlamaForCausalLM | ✅ | ✅ | |

21 -| Qwen/Qwen2.5-72B-Instruct | Qwen2ForCausalLM | ✅ | ✅ | |

22 -| Qwen/Qwen3-14B | Qwen3ForCausalLM | ✅ | ✅ | |

23 -| Qwen/Qwen3-32B | Qwen3ForCausalLM | ✅ | ✅ | |

24 -| Qwen/Qwen3-30B-A3B | Qwen3ForCausalLM | ✅ | ✅ | |

25 -| Qwen/Qwen3-30B-A3B-GPTQ-Int4 | Qwen3ForCausalLM | ✅ | ✅ | |

26 -| Qwen/Qwen3-coder-30B-A3B-Instruct | Qwen3ForCausalLM | ✅ | ✅ | |

27 -| Qwen/QwQ-32B | QwenForCausalLM | ✅ | ✅ | |

28 -| deepseek-ai/DeepSeek-V2-Lite | DeepSeekForCausalLM | ✅ | ✅ | |

29 -| meta-llama/Llama-3.1-8B-Instruct | LlamaForCausalLM | ✅ | ✅ | |

30 -| baichuan-inc/Baichuan2-13B-Chat | BaichuanForCausalLM | ✅ | ✅ | |

31 -| THUDM/GLM-4-9B-chat | GLMForCausalLM | ✅ | ✅ | |

32 -| THUDM/CodeGeex4-All-9B | CodeGeexForCausalLM | ✅ | ✅ | |

33 -| chuhac/TeleChat2-35B | LlamaForCausalLM (TeleChat2 based on Llama arch) | ✅ | ✅ | |

34 -| 01-ai/Yi1.5-34B-Chat | YiForCausalLM | ✅ | ✅ | |

35 -| THUDM/CodeGeex4-All-9B | CodeGeexForCausalLM | ✅ | ✅ | |

36 -| deepseek-ai/DeepSeek-Coder-33B-base | DeepSeekCoderForCausalLM | ✅ | ✅ | |

37 -| baichuan-inc/Baichuan2-13B-Chat | BaichuanForCausalLM | ✅ | ✅ | |

38 -| meta-llama/Llama-2-13b-chat-hf | LlamaForCausalLM | ✅ | ✅ | |

39 -| THUDM/CodeGeex4-All-9B | CodeGeexForCausalLM | ✅ | ✅ | |

40 -| Qwen/Qwen1.5-14B-Chat | QwenForCausalLM | ✅ | ✅ | |

41 -| Qwen/Qwen1.5-32B-Chat | QwenForCausalLM | ✅ | ✅ | |

15 +| openai/gpt-oss-20b | GPTForCausalLM | | | ✅ |

16 +| openai/gpt-oss-120b | GPTForCausalLM | | | ✅ |

17 +| deepseek-ai/DeepSeek-R1-Distill-Llama-8B | LlamaForCausalLM | ✅ | ✅ | |

18 +| deepseek-ai/DeepSeek-R1-Distill-Qwen-14B | QwenForCausalLM | ✅ | ✅ | |

19 +| deepseek-ai/DeepSeek-R1-Distill-Qwen-32B | QwenForCausalLM | ✅ | ✅ | |

20 +| deepseek-ai/DeepSeek-R1-Distill-Llama-70B | LlamaForCausalLM | ✅ | ✅ | |

21 +| Qwen/Qwen2.5-72B-Instruct | Qwen2ForCausalLM | ✅ | ✅ | |

22 +| Qwen/Qwen3-14B | Qwen3ForCausalLM | ✅ | ✅ | |

23 +| Qwen/Qwen3-32B | Qwen3ForCausalLM | ✅ | ✅ | |

24 +| Qwen/Qwen3-30B-A3B | Qwen3ForCausalLM | ✅ | ✅ | |

25 +| Qwen/Qwen3-30B-A3B-GPTQ-Int4 | Qwen3ForCausalLM | ✅ | ✅ | |

26 +| Qwen/Qwen3-coder-30B-A3B-Instruct | Qwen3ForCausalLM | ✅ | ✅ | |

27 +| Qwen/QwQ-32B | QwenForCausalLM | ✅ | ✅ | |

28 +| deepseek-ai/DeepSeek-V2-Lite | DeepSeekForCausalLM | ✅ | ✅ | |

29 +| meta-llama/Llama-3.1-8B-Instruct | LlamaForCausalLM | ✅ | ✅ | |

30 +| baichuan-inc/Baichuan2-13B-Chat | BaichuanForCausalLM | ✅ | ✅ | |

31 +| THUDM/GLM-4-9B-chat | GLMForCausalLM | ✅ | ✅ | |

32 +| THUDM/CodeGeex4-All-9B | CodeGeexForCausalLM | ✅ | ✅ | |

33 +| chuhac/TeleChat2-35B | LlamaForCausalLM (TeleChat2 based on Llama arch) | ✅ | ✅ | |

34 +| 01-ai/Yi1.5-34B-Chat | YiForCausalLM | ✅ | ✅ | |

35 +| THUDM/CodeGeex4-All-9B | CodeGeexForCausalLM | ✅ | ✅ | |

36 +| deepseek-ai/DeepSeek-Coder-33B-base | DeepSeekCoderForCausalLM | ✅ | ✅ | |

37 +| baichuan-inc/Baichuan2-13B-Chat | BaichuanForCausalLM | ✅ | ✅ | |

38 +| meta-llama/Llama-2-13b-chat-hf | LlamaForCausalLM | ✅ | ✅ | |

39 +| THUDM/CodeGeex4-All-9B | CodeGeexForCausalLM | ✅ | ✅ | |

40 +| Qwen/Qwen1.5-14B-Chat | QwenForCausalLM | ✅ | ✅ | |

41 +| Qwen/Qwen1.5-32B-Chat | QwenForCausalLM | ✅ | ✅ | |

43 ### Multimodal Language Models

46 | ---------------------------- | -------------------------------- | ---- | ----------- | ----- |

47 -| OpenGVLab/InternVL3_5-8B | InternVLForConditionalGeneration | ✅ | ✅ | |

48 -| OpenGVLab/InternVL3_5-14B | InternVLForConditionalGeneration | ✅ | ✅ | |

49 -| OpenGVLab/InternVL3_5-38B | InternVLForConditionalGeneration | ✅ | ✅ | |

50 -| Qwen/Qwen2-VL-7B-Instruct | Qwen2VLForConditionalGeneration | ✅ | ✅ | |

51 -| Qwen/Qwen2.5-VL-72B-Instruct | Qwen2VLForConditionalGeneration | ✅ | ✅ | |

52 -| Qwen/Qwen2.5-VL-32B-Instruct | Qwen2VLForConditionalGeneration | ✅ | ✅ | |

53 -| THUDM/GLM-4v-9B | GLM4vForConditionalGeneration | ✅ | ✅ | |

54 -| openbmb/MiniCPM-V-4 | MiniCPMVForConditionalGeneration | ✅ | ✅ | |

47 +| OpenGVLab/InternVL3_5-8B | InternVLForConditionalGeneration | ✅ | ✅ | |

48 +| OpenGVLab/InternVL3_5-14B | InternVLForConditionalGeneration | ✅ | ✅ | |

49 +| OpenGVLab/InternVL3_5-38B | InternVLForConditionalGeneration | ✅ | ✅ | |

50 +| Qwen/Qwen2-VL-7B-Instruct | Qwen2VLForConditionalGeneration | ✅ | ✅ | |

51 +| Qwen/Qwen2.5-VL-72B-Instruct | Qwen2VLForConditionalGeneration | ✅ | ✅ | |

52 +| Qwen/Qwen2.5-VL-32B-Instruct | Qwen2VLForConditionalGeneration | ✅ | ✅ | |

53 +| THUDM/GLM-4v-9B | GLM4vForConditionalGeneration | ✅ | ✅ | |

54 +| openbmb/MiniCPM-V-4 | MiniCPMVForConditionalGeneration | ✅ | ✅ | |

56 ### Embedding and Reranker Language Models

59 | ----------------------- | ------------------------------ | ---- | ----------- | ----- |

60 -| Qwen/Qwen3-Embedding-8B | Qwen3ForTextEmbedding | ✅ | ✅ | |

61 -| Qwen/Qwen3-Reranker-8B | Qwen3ForSequenceClassification | ✅ | ✅ | |

60 +| Qwen/Qwen3-Embedding-8B | Qwen3ForTextEmbedding | ✅ | ✅ | |

61 +| Qwen/Qwen3-Reranker-8B | Qwen3ForSequenceClassification | ✅ | ✅ | |

63 ✅ Runs and optimized.

64 🟨 Runs and correct but not optimized to green yet.

docs/models/pooling_models.md

+7 -7

@@ -31,7 +31,7 @@ vLLM will attempt to automatically convert the model according to the architectu

31 shown in the table below.

33 | Architecture | `--convert` | Supported pooling tasks |

34 -|-------------------------------------------------|-------------|---------------------------------------|

34 +| ----------------------------------------------- | ----------- | ------------------------------------- |

35 | `*ForTextEncoding`, `*EmbeddingModel`, `*Model` | `embed` | `token_embed`, `embed` |

36 | `*ForRewardModeling`, `*RewardModel` | `embed` | `token_embed`, `embed` |

37 | `*For*Classification`, `*ClassificationModel` | `classify` | `token_classify`, `classify`, `score` |

@@ -46,7 +46,7 @@ Each pooling model in vLLM supports one or more of these tasks according to

46 enabling the corresponding APIs:

48 | Task | APIs |

49 -|------------------|-------------------------------------------------------------------------------|

49 +| ---------------- | ----------------------------------------------------------------------------- |

50 | `embed` | `LLM.embed(...)`, `LLM.score(...)`\*, `LLM.encode(..., pooling_task="embed")` |

51 | `classify` | `LLM.classify(...)`, `LLM.encode(..., pooling_task="classify")` |

52 | `score` | `LLM.score(...)` |

@@ -69,7 +69,7 @@ If the model has been converted via `--convert` (see above),

69 the pooler assigned to each task has the following attributes by default:

72 -|------------|--------------|---------------|---------|

72 +| ---------- | ------------ | ------------- | ------- |

73 | `embed` | `LAST` | ✅︎ | ❌ |

74 | `classify` | `LAST` | ❌ | ✅︎ |

@@ -314,7 +314,7 @@ An OpenAI client example can be found here: [examples/pooling/embed/openai_embed

314 vLLM supports ColBERT models with multiple encoder backbones:

315

316 | Architecture | Backbone | Example HF Models |

317 -|---|---|---|

317 +| - | - | - |

318 | `HF_ColBERT` | BERT | `answerdotai/answerai-colbert-small-v1`, `colbert-ir/colbertv2.0` |

319 | `ColBERTModernBertModel` | ModernBERT | `lightonai/GTE-ModernColBERT-v1` |

320 | `ColBERTJinaRobertaModel` | Jina XLM-RoBERTa | `jinaai/jina-colbert-v2` |

@@ -379,7 +379,7 @@ An example can be found here: [examples/pooling/score/colbert_rerank_online.py](

379 ColQwen3 is based on [ColPali](https://arxiv.org/abs/2407.01449), which extends ColBERT's late interaction approach to **multi-modal** inputs. While ColBERT operates on text-only token embeddings, ColPali/ColQwen3 can embed both **text and images** (e.g. PDF pages, screenshots, diagrams) into per-token L2-normalized vectors and compute relevance via MaxSim scoring. ColQwen3 specifically uses Qwen3-VL as its vision-language backbone.

380

381 | Architecture | Backbone | Example HF Models |

382 -|---|---|---|

382 +| - | - | - |

383 | `ColQwen3` | Qwen3-VL | `TomoroAI/tomoro-colqwen3-embed-4b`, `TomoroAI/tomoro-colqwen3-embed-8b` |

384 | `OpsColQwen3Model` | Qwen3-VL | `OpenSearch-AI/Ops-Colqwen3-4B`, `OpenSearch-AI/Ops-Colqwen3-8B` |

385 | `Qwen3VLNemotronEmbedModel` | Qwen3-VL | `nvidia/nemotron-colembed-vl-4b-v2`, `nvidia/nemotron-colembed-vl-8b-v2` |

@@ -507,7 +507,7 @@ Llama Nemotron VL Embedding models combine the bidirectional Llama embedding bac

507 single-vector embeddings from text and/or images.

508

509 | Architecture | Backbone | Example HF Models |

510 -|---|---|---|

510 +| - | - | - |

511 | `LlamaNemotronVLModel` | Bidirectional Llama + SigLIP | `nvidia/llama-nemotron-embed-vl-1b-v2` |

512

513 Start the server:

@@ -567,7 +567,7 @@ Llama Nemotron VL reranker models combine the same bidirectional Llama + SigLIP

567 backbone with a sequence-classification head for cross-encoder scoring and reranking.

568

569 | Architecture | Backbone | Example HF Models |

570 -|---|---|---|

570 +| - | - | - |

571 | `LlamaNemotronVLForSequenceClassification` | Bidirectional Llama + SigLIP | `nvidia/llama-nemotron-rerank-vl-1b-v2` |

572

573 Start the server:

docs/models/supported_models.md

+31 -31

@@ -179,7 +179,7 @@ class MyConfig(PretrainedConfig):

179 Some model architectures are supported via vLLM plugins. These plugins extend vLLM's capabilities through the [plugin system](../design/plugin_system.md).

180

181 | Architecture | Models | Plugin Repository |

182 -|--------------|--------|-------------------|

182 +| ------------ | ------ | ----------------- |

183 | `BartForConditionalGeneration` | BART | [bart-plugin](https://github.com/vllm-project/bart-plugin) |

184 | `Florence2ForConditionalGeneration` | Florence-2 | [bart-plugin](https://github.com/vllm-project/bart-plugin) |

185

@@ -363,7 +363,7 @@ th {

363 </style>

364

366 -|--------------|--------|-------------------|----------------------|---------------------------|

366 +| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

367 | `AfmoeForCausalLM` | Afmoe | TBA | ✅︎ | ✅︎ |

368 | `ApertusForCausalLM` | Apertus | `swiss-ai/Apertus-8B-2509`, `swiss-ai/Apertus-70B-Instruct-2509`, etc. | ✅︎ | ✅︎ |

369 | `AquilaForCausalLM` | Aquila, Aquila2 | `BAAI/Aquila-7B`, `BAAI/AquilaChat-7B`, etc. | ✅︎ | ✅︎ |

@@ -387,7 +387,7 @@ th {

388 | `DotsOCRForCausalLM` | dots_ocr | `rednote-hilab/dots.ocr` | ✅︎ | ✅︎ |

389 | `Ernie4_5ForCausalLM` | Ernie4.5 | `baidu/ERNIE-4.5-0.3B-PT`, etc. | ✅︎ | ✅︎ |

390 -| `Ernie4_5_MoeForCausalLM` | Ernie4.5MoE | `baidu/ERNIE-4.5-21B-A3B-PT`, `baidu/ERNIE-4.5-300B-A47B-PT`, etc. |✅︎| ✅︎ |

390 +| `Ernie4_5_MoeForCausalLM` | Ernie4.5MoE | `baidu/ERNIE-4.5-21B-A3B-PT`, `baidu/ERNIE-4.5-300B-A47B-PT`, etc. | ✅︎ | ✅︎ |

391 | `ExaoneForCausalLM` | EXAONE-3 | `LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc. | ✅︎ | ✅︎ |

393 | `Exaone4ForCausalLM` | EXAONE-4 | `LGAI-EXAONE/EXAONE-4.0-32B`, etc. | ✅︎ | ✅︎ |

@@ -427,18 +427,18 @@ th {

428 | `JambaForCausalLM` | Jamba | `ai21labs/AI21-Jamba-1.5-Large`, `ai21labs/AI21-Jamba-1.5-Mini`, `ai21labs/Jamba-v0.1`, etc. | ✅︎ | ✅︎ |

430 -| `Lfm2ForCausalLM` | LFM2 | `LiquidAI/LFM2-1.2B`, `LiquidAI/LFM2-700M`, `LiquidAI/LFM2-350M`, etc. | ✅︎ | ✅︎ |

431 -| `Lfm2MoeForCausalLM` | LFM2MoE | `LiquidAI/LFM2-8B-A1B-preview`, etc. | ✅︎ | ✅︎ |

430 +| `Lfm2ForCausalLM` | LFM2 | `LiquidAI/LFM2-1.2B`, `LiquidAI/LFM2-700M`, `LiquidAI/LFM2-350M`, etc. | ✅︎ | ✅︎ |

431 +| `Lfm2MoeForCausalLM` | LFM2MoE | `LiquidAI/LFM2-8B-A1B-preview`, etc. | ✅︎ | ✅︎ |

432 | `LlamaForCausalLM` | Llama 3.1, Llama 3, Llama 2, LLaMA, Yi | `meta-llama/Meta-Llama-3.1-405B-Instruct`, `meta-llama/Meta-Llama-3.1-70B`, `meta-llama/Meta-Llama-3-70B-Instruct`, `meta-llama/Llama-2-70b-hf`, `01-ai/Yi-34B`, etc. | ✅︎ | ✅︎ |

433 | `LongcatFlashForCausalLM` | LongCat-Flash | `meituan-longcat/LongCat-Flash-Chat`, `meituan-longcat/LongCat-Flash-Chat-FP8` | ✅︎ | ✅︎ |

436 | `MiMoForCausalLM` | MiMo | `XiaomiMiMo/MiMo-7B-RL`, etc. | ✅︎ | ✅︎ |

437 -| `MiMoV2FlashForCausalLM` | MiMoV2Flash | `XiaomiMiMo/MiMo-V2-Flash`, etc. | ︎| ✅︎ |

438 | `MiniCPMForCausalLM` | MiniCPM | `openbmb/MiniCPM-2B-sft-bf16`, `openbmb/MiniCPM-2B-dpo-bf16`, `openbmb/MiniCPM-S-1B-sft`, etc. | ✅︎ | ✅︎ |

439 | `MiniCPM3ForCausalLM` | MiniCPM3 | `openbmb/MiniCPM3-4B`, etc. | ✅︎ | ✅︎ |

441 -| `MiniMaxM2ForCausalLM` | MiniMax-M2, MiniMax-M2.1 |`MiniMaxAI/MiniMax-M2`, etc. | ✅︎ | ✅︎ |

441 +| `MiniMaxM2ForCausalLM` | MiniMax-M2, MiniMax-M2.1 | `MiniMaxAI/MiniMax-M2`, etc. | ✅︎ | ✅︎ |

442 | `MistralForCausalLM` | Ministral-3, Mistral, Mistral-Instruct | `mistralai/Ministral-3-3B-Instruct-2512`, `mistralai/Mistral-7B-v0.1`, `mistralai/Mistral-7B-Instruct-v0.1`, etc. | ✅︎ | ✅︎ |

443 | `MistralLarge3ForCausalLM` | Mistral-Large-3-675B-Base-2512, Mistral-Large-3-675B-Instruct-2512 | `mistralai/Mistral-Large-3-675B-Base-2512`, `mistralai/Mistral-Large-3-675B-Instruct-2512`, etc. | ✅︎ | ✅︎ |

444 | `MixtralForCausalLM` | Mixtral-8x7B, Mixtral-8x7B-Instruct | `mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, `mistral-community/Mixtral-8x22B-v0.1`, etc. | ✅︎ | ✅︎ |

@@ -453,9 +453,9 @@ th {

453 | `OPTForCausalLM` | OPT, OPT-IML | `facebook/opt-66b`, `facebook/opt-iml-max-30b`, etc. | ✅︎ | ✅︎ |

456 -| `PanguEmbeddedForCausalLM` |openPangu-Embedded-7B | `FreedomIntelligence/openPangu-Embedded-7B-V1.1` | ✅︎ | ✅︎ |

457 -| `PanguProMoEV2ForCausalLM` |openpangu-pro-moe-v2 | | ✅︎ | ✅︎ |

458 -| `PanguUltraMoEForCausalLM` |openpangu-ultra-moe-718b-model | `FreedomIntelligence/openPangu-Ultra-MoE-718B-V1.1` | ✅︎ | ✅︎ |

456 +| `PanguEmbeddedForCausalLM` | openPangu-Embedded-7B | `FreedomIntelligence/openPangu-Embedded-7B-V1.1` | ✅︎ | ✅︎ |

457 +| `PanguProMoEV2ForCausalLM` | openpangu-pro-moe-v2 | | ✅︎ | ✅︎ |

458 +| `PanguUltraMoEForCausalLM` | openpangu-ultra-moe-718b-model | `FreedomIntelligence/openPangu-Ultra-MoE-718B-V1.1` | ✅︎ | ✅︎ |

459 | `PhiForCausalLM` | Phi | `microsoft/phi-1_5`, `microsoft/phi-2`, etc. | ✅︎ | ✅︎ |

460 | `Phi3ForCausalLM` | Phi-4, Phi-3 | `microsoft/Phi-4-mini-instruct`, `microsoft/Phi-4`, `microsoft/Phi-3-mini-4k-instruct`, `microsoft/Phi-3-mini-128k-instruct`, `microsoft/Phi-3-medium-128k-instruct`, etc. | ✅︎ | ✅︎ |

461 | `PhiMoEForCausalLM` | Phi-3.5-MoE | `microsoft/Phi-3.5-MoE-instruct`, etc. | ✅︎ | ✅︎ |

@@ -477,7 +477,7 @@ th {

479 | `Step1ForCausalLM` | Step-Audio | `stepfun-ai/Step-Audio-EditX`, etc. | ✅︎ | ✅︎ |

481 | `TeleChatForCausalLM` | TeleChat | `chuhac/TeleChat2-35B`, etc. | ✅︎ | ✅︎ |

482 | `TeleChat2ForCausalLM` | TeleChat2 | `Tele-AI/TeleChat2-3B`, `Tele-AI/TeleChat2-7B`, `Tele-AI/TeleChat2-35B`, etc. | ✅︎ | ✅︎ |

483 | `TeleFLMForCausalLM` | TeleFLM | `CofeAI/FLM-2-52B-Instruct-2407`, `CofeAI/Tele-FLM`, etc. | ✅︎ | ✅︎ |

@@ -492,7 +492,7 @@ th {

492 Some models are supported only via the [Transformers modeling backend](#transformers). The purpose of the table below is to acknowledge models which we officially support in this way. The logs will say that the Transformers modeling backend is being used, and you will see no warning that this is fallback behaviour. This means that, if you have issues with any of the models listed below, please [make an issue](https://github.com/vllm-project/vllm/issues/new/choose) and we'll do our best to fix it!

493

495 -|--------------|--------|-------------------|----------------------|---------------------------|

495 +| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

496 | `SmolLM3ForCausalLM` | SmolLM3 | `HuggingFaceTB/SmolLM3-3B` | ✅︎ | ✅︎ |

497

498 !!! note

@@ -511,16 +511,16 @@ See [this page](./pooling_models.md) for more information on how to use pooling

511 These models primarily support the [`LLM.embed`](./pooling_models.md#llmembed) API.

512

514 -|--------------|--------|-------------------|----------------------|---------------------------|

514 +| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

517 | `Gemma2Model`C | Gemma 2-based | `BAAI/bge-multilingual-gemma2`, etc. | ✅︎ | ✅︎ |

518 | `Gemma3TextModel`C | Gemma 3-based | `google/embeddinggemma-300m`, etc. | ✅︎ | ✅︎ |

519 | `GritLM` | GritLM | `parasail-ai/GritLM-7B-vllm`. | ✅︎ | ✅︎ |

524 | `LlamaBidirectionalModel`C | Llama-based with bidirectional attention | `nvidia/llama-nemotron-embed-1b-v2`, etc. | ✅︎ | ✅︎ |

525 | `LlamaModel`C, `LlamaForCausalLM`C, `MistralModel`C, etc. | Llama-based | `intfloat/e5-mistral-7b-instruct`, etc. | ✅︎ | ✅︎ |

526 | `Qwen2Model`C, `Qwen2ForCausalLM`C | Qwen2-based | `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc. | ✅︎ | ✅︎ |

@@ -555,7 +555,7 @@ of the whole prompt are extracted from the normalized hidden state corresponding

555 These models primarily support the [`LLM.classify`](./pooling_models.md#llmclassify) API.

556

558 -|--------------|--------|-------------------|----------------------|---------------------------|

558 +| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

559 | `JambaForSequenceClassification` | Jamba | `ai21labs/Jamba-tiny-reward-dev`, etc. | ✅︎ | ✅︎ |

560 | `GPT2ForSequenceClassification` | GPT2 | `nie3e/sentiment-polish-gpt2-small` | | |

561 | `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* |

@@ -572,7 +572,7 @@ Cross-encoder and reranker models are a subset of classification models that acc

572 These models primarily support the [`LLM.score`](./pooling_models.md#llmscore) API.

573

575 -|--------------|--------|-------------------|---------------------------|-----------------------------|-----------------------------------------|

575 +| ------------ | ------ | ----------------- | ------------------------- | --------------------------- | --------------------------------------- |

577 | `GemmaForSequenceClassification` | Gemma-based | `BAAI/bge-reranker-v2-gemma`(see note), etc. | [bge-reranker-v2-gemma.jinja](../../examples/pooling/score/template/bge-reranker-v2-gemma.jinja) | ✅︎ | ✅︎ |

@@ -622,7 +622,7 @@ These models primarily support the [`LLM.score`](./pooling_models.md#llmscore) A

622 These models primarily support the [`LLM.reward`](./pooling_models.md#llmreward) API.

623

625 -|--------------|--------|-------------------|----------------------|---------------------------|

625 +| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

626 | `InternLM2ForRewardModel` | InternLM2-based | `internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc. | ✅︎ | ✅︎ |

627 | `LlamaForCausalLM` | Llama-based | `peiyi9979/math-shepherd-mistral-7b-prm`, etc. | ✅︎ | ✅︎ |

628 | `Qwen2ForRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-RM-72B`, etc. | ✅︎ | ✅︎ |

@@ -637,9 +637,9 @@ These models primarily support the [`LLM.reward`](./pooling_models.md#llmreward)

637 These models primarily support the [`LLM.encode`](./pooling_models.md#llmencode) API.

638

640 -|--------------|--------|-------------------|-----------------------------|-----------------------------------------|

640 +| ------------ | ------ | ----------------- | --------------------------- | --------------------------------------- |

643

644 !!! note

645 Named Entity Recognition (NER) usage, please refer to [examples/pooling/token_classify/ner_offline.py](../../examples/pooling/token_classify/ner_offline.py), [examples/pooling/token_classify/ner_online.py](../../examples/pooling/token_classify/ner_online.py).

@@ -678,7 +678,7 @@ See [this page](generative_models.md) for more information on how to use generat

678 These models primarily accept the [`LLM.generate`](./generative_models.md#llmgenerate) API. Chat/Instruct models additionally support the [`LLM.chat`](./generative_models.md#llmchat) API.

679

681 -|--------------|--------|--------|-------------------|----------------------|---------------------------|

681 +| ------------ | ------ | ------ | ----------------- | -------------------- | ------------------------- |

683 | `AudioFlamingo3ForConditionalGeneration` | AudioFlamingo3 | T + A | `nvidia/audio-flamingo-3-hf`, `nvidia/music-flamingo-2601-hf` | ✅︎ | ✅︎ |

684 | `AyaVisionForConditionalGeneration` | Aya Vision | T + I+ | `CohereLabs/aya-vision-8b`, `CohereLabs/aya-vision-32b`, etc. | | ✅︎ |

@@ -698,7 +698,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen

698 | `GLM4VForCausalLM`^ | GLM-4V | T + I | `zai-org/glm-4v-9b`, `zai-org/cogagent-9b-20241220`, etc. | ✅︎ | ✅︎ |

699 | `Glm4vForConditionalGeneration` | GLM-4.1V-Thinking | T + IE+ + VE+ | `zai-org/GLM-4.1V-9B-Thinking`, etc. | ✅︎ | ✅︎ |

700 | `Glm4vMoeForConditionalGeneration` | GLM-4.5V | T + IE+ + VE+ | `zai-org/GLM-4.5V`, etc. | ✅︎ | ✅︎ |

701 -| `GlmOcrForConditionalGeneration` | GLM-OCR | T + IE+ | `zai-org/GLM-OCR`, etc. | ✅︎ | ✅︎ |

701 +| `GlmOcrForConditionalGeneration` | GLM-OCR | T + IE+ | `zai-org/GLM-OCR`, etc. | ✅︎ | ✅︎ |

702 | `GraniteSpeechForConditionalGeneration` | Granite Speech | T + A | `ibm-granite/granite-speech-3.3-8b` | ✅︎ | ✅︎ |

704 | `H2OVLChatModel` | H2OVL | T + IE+ | `h2oai/h2ovl-mississippi-800m`, `h2oai/h2ovl-mississippi-2b`, etc. | | ✅︎ |

@@ -714,7 +714,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen

714 | `KeyeVL1_5ForConditionalGeneration` | Keye-VL-1_5-8B | T + IE+ + VE+ | `Kwai-Keye/Keye-VL-1_5-8B` | ✅︎ | ✅︎ |

715 | `KimiVLForConditionalGeneration` | Kimi-VL-A3B-Instruct, Kimi-VL-A3B-Thinking | T + I+ | `moonshotai/Kimi-VL-A3B-Instruct`, `moonshotai/Kimi-VL-A3B-Thinking` | | ✅︎ |

716 | `KimiK25ForConditionalGeneration` | Kimi-K2.5 | T + I+ | `moonshotai/Kimi-K2.5` | | ✅︎ |

717 -| `LightOnOCRForConditionalGeneration` | LightOnOCR-1B | T + I+ | `lightonai/LightOnOCR-1B`, etc | ✅︎ | ✅︎ |

717 +| `LightOnOCRForConditionalGeneration` | LightOnOCR-1B | T + I+ | `lightonai/LightOnOCR-1B`, etc | ✅︎ | ✅︎ |

718 | `Lfm2VlForConditionalGeneration` | LFM2-VL | T + I+ | `LiquidAI/LFM2-VL-450M`, `LiquidAI/LFM2-VL-3B`, `LiquidAI/LFM2-VL-8B-A1B`, etc. | ✅︎ | ✅︎ |

719 | `Llama4ForConditionalGeneration` | Llama 4 | T + I+ | `meta-llama/Llama-4-Scout-17B-16E-Instruct`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct`, etc. | ✅︎ | ✅︎ |

720 | `Llama_Nemotron_Nano_VL` | Llama Nemotron Nano VL | T + IE+ | `nvidia/Llama-3.1-Nemotron-Nano-VL-8B-V1` | ✅︎ | ✅︎ |

@@ -731,7 +731,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen

731 | `Molmo2ForConditionalGeneration` | Molmo2 | T + I+ / V | `allenai/Molmo2-4B`, `allenai/Molmo2-8B`, `allenai/Molmo2-O-7B` | ✅︎ | ✅︎ |

732 | `NVLM_D_Model` | NVLM-D 1.0 | T + I+ | `nvidia/NVLM-D-72B`, etc. | | ✅︎ |

733 | `OpenCUAForConditionalGeneration` | OpenCUA-7B | T + IE+ | `xlangai/OpenCUA-7B` | ✅︎ | ✅︎ |

734 -| `OpenPanguVLForConditionalGeneration` | openpangu-VL | T + IE+ + VE+ |`FreedomIntelligence/openPangu-VL-7B` | ✅︎ | ✅︎ |

734 +| `OpenPanguVLForConditionalGeneration` | openpangu-VL | T + IE+ + VE+ | `FreedomIntelligence/openPangu-VL-7B` | ✅︎ | ✅︎ |

735 | `Ovis` | Ovis2, Ovis1.6 | T + I+ | `AIDC-AI/Ovis2-1B`, `AIDC-AI/Ovis1.6-Llama3.2-3B`, etc. | | ✅︎ |

@@ -764,7 +764,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen

764 Some models are supported only via the [Transformers modeling backend](#transformers). The purpose of the table below is to acknowledge models which we officially support in this way. The logs will say that the Transformers modeling backend is being used, and you will see no warning that this is fallback behaviour. This means that, if you have issues with any of the models listed below, please [make an issue](https://github.com/vllm-project/vllm/issues/new/choose) and we'll do our best to fix it!

765

767 -|--------------|--------|--------|-------------------|-----------------------------|-----------------------------------------|

767 +| ------------ | ------ | ------ | ----------------- | --------------------------- | --------------------------------------- |

768 | `Emu3ForConditionalGeneration` | Emu3 | T + I | `BAAI/Emu3-Chat-hf` | ✅︎ | ✅︎ |

769

770 ^ You need to set the architecture name via `--hf-overrides` to match the one in vLLM.

@@ -795,7 +795,7 @@ Some models are supported only via the [Transformers modeling backend](#transfor

795 Speech2Text models trained specifically for Automatic Speech Recognition.

796

798 -|--------------|--------|-------------------|----------------------|---------------------------|

798 +| ------------ | ------ | ----------------- | -------------------- | ------------------------- |

@@ -823,7 +823,7 @@ These models primarily support the [`LLM.embed`](./pooling_models.md#llmembed) A

823 The following table lists those that are tested in vLLM.

824

826 -|--------------|--------|--------|-------------------|----------------------|---------------------------|

826 +| ------------ | ------ | ------ | ----------------- | -------------------- | ------------------------- |

@@ -844,7 +844,7 @@ Cross-encoder and reranker models are a subset of classification models that acc

844 These models primarily support the [`LLM.score`](./pooling_models.md#llmscore) API.

845

847 -|--------------|--------|--------|-------------------|----------------------|---------------------------|

847 +| ------------ | ------ | ------ | ----------------- | -------------------- | ------------------------- |

848 | `JinaVLForSequenceClassification` | JinaVL-based | T + IE+ | `jinaai/jina-reranker-m0`, etc. | ✅︎ | ✅︎ |

850 | `Qwen3VLForSequenceClassification` | Qwen3-VL-Reranker | T + IE+ + VE+ | `Qwen/Qwen3-VL-Reranker-2B`(see note), etc. | ✅︎ | ✅︎ |

docs/serving/expert_parallel_deployment.md

+5 -5

@@ -17,7 +17,7 @@ Before using EP, you need to install the necessary dependencies. We are actively

17 vLLM provides multiple communication backends for EP. Use `--all2all-backend` to select one:

20 -|---------|----------|----------|----------|

20 +| ------- | -------- | -------- | -------- |

@@ -48,7 +48,7 @@ Where:

48 When EP is enabled, different layers in MoE models behave differently:

50 | Layer Type | Behavior | Parallelism Used |

51 -|------------|----------|------------------|

51 +| ---------- | -------- | ---------------- |

52 | **Expert (MoE) Layers** | Sharded across all EP ranks | Expert Parallel (EP) of size `TP × DP` |

53 | **Attention Layers** | Behavior depends on TP size | See below |

@@ -146,9 +146,9 @@ When enabled, vLLM collects load statistics with every forward pass and periodic

146 Configure EPLB with the `--eplb-config` argument, which accepts a JSON string. The available keys and their descriptions are:

147

148 | Parameter | Description | Default |

149 -|-----------|-------------|---------|

150 -| `window_size`| Number of engine steps to track for rebalancing decisions | 1000 |

151 -| `step_interval`| Frequency of rebalancing (every N engine steps) | 3000 |

149 +| --------- | ----------- | ------- |

150 +| `window_size` | Number of engine steps to track for rebalancing decisions | 1000 |

151 +| `step_interval` | Frequency of rebalancing (every N engine steps) | 3000 |

152 | `log_balancedness` | Log balancedness metrics (avg tokens per expert ÷ max tokens per expert) | `false` |

153 | `num_redundant_experts` | Additional global experts per EP rank beyond equal distribution | `0` |

154 | `use_async` | Use non-blocking EPLB for reduced latency overhead | `false` |

docs/serving/openai_compatible_server.md

+2 -2

@@ -596,7 +596,7 @@ Audio must be sent as base64-encoded PCM16 audio at 16kHz sample rate, mono chan

596 #### Client → Server Events

597

598 | Event | Description |

599 -|-------|-------------|

599 +| ----- | ----------- |

600 | `input_audio_buffer.append` | Send base64-encoded audio chunk: `{"type": "input_audio_buffer.append", "audio": "<base64>"}` |

601 | `input_audio_buffer.commit` | Trigger transcription processing or end: `{"type": "input_audio_buffer.commit", "final": bool}` |

602 | `session.update` | Configure session: `{"type": "session.update", "model": "model-name"}` |

@@ -604,7 +604,7 @@ Audio must be sent as base64-encoded PCM16 audio at 16kHz sample rate, mono chan

604 #### Server → Client Events

605

606 | Event | Description |

607 -|-------|-------------|

607 +| ----- | ----------- |

608 | `session.created` | Connection established with session ID and timestamp |

609 | `transcription.delta` | Incremental transcription text: `{"type": "transcription.delta", "delta": "text"}` |

610 | `transcription.done` | Final transcription with usage stats |

docs/usage/v1_guide.md

+15 -15

@@ -83,13 +83,13 @@ based on assigned priority, with FCFS as a tie-breaker), configurable via the

84 ### Hardware

86 -| Hardware | Status |

87 -|------------------|-----------------------------------------------|

88 -| **NVIDIA** | 🟢 |

89 -| **AMD** | 🟢 |

90 -| **INTEL GPU** | 🟢 |

91 -| **TPU** | 🟢 |

92 -| **CPU** | 🟢 |

86 +| Hardware | Status |

87 +| --------------| --------------- |

88 +| **NVIDIA** | 🟢 |

89 +| **AMD** | 🟢 |

90 +| **INTEL GPU** | 🟢 |

91 +| **TPU** | 🟢 |

92 +| **CPU** | 🟢 |

94 !!! note

@@ -104,13 +104,13 @@ based on assigned priority, with FCFS as a tie-breaker), configurable via the

104

105 ### Models

106

107 -| Model Type | Status |

108 -|-----------------------------|-------------------------------------------------------------------------|

109 -| **Decoder-only Models** | 🟢 |

110 -| **Encoder-Decoder Models** | 🟢 (Whisper), 🔴 (Others) |

111 -| **Pooling Models** | 🟢 |

112 -| **Mamba Models** | 🟢 |

113 -| **Multimodal Models** | 🟢 |

107 +| Model Type | Status |

108 +| -------------------------- | --------------------------------------- |

109 +| **Decoder-only Models** | 🟢 |

110 +| **Encoder-Decoder Models** | 🟢 (Whisper), 🔴 (Others) |

111 +| **Pooling Models** | 🟢 |

112 +| **Mamba Models** | 🟢 |

113 +| **Multimodal Models** | 🟢 |

114

115 See below for the status of models that are not yet supported or have more features planned in V1.

116

@@ -145,7 +145,7 @@ following a similar pattern by implementing support through the [plugin system](

145 ### Features

146

147 | Feature | Status |

148 -|---------------------------------------------|-----------------------------------------------------------------------------------|

148 +| ------------------------------------------- | --------------------------------------------------------------------------------- |

149 | **Prefix Caching** | 🟢 Functional |

150 | **Chunked Prefill** | 🟢 Functional |

151 | **LoRA** | 🟢 Functional |

examples/online_serving/dashboards/README.md

+1 -1

@@ -34,7 +34,7 @@ deployment methods:

34 Both platforms provide equivalent monitoring capabilities:

36 | Dashboard | Description |

37 -|-----------|-------------|

37 +| --------- | ----------- |

38 | **Performance Statistics** | Tracks latency, throughput, and performance metrics |

39 | **Query Statistics** | Monitors request volume, query performance, and KPIs |

examples/online_serving/disaggregated_encoder/README.md

+1 -1

@@ -95,7 +95,7 @@ If you enable prefill instance (`--prefill-servers-urls` not disabled), you will

95 ## Proxy Instance Flags (`disagg_epd_proxy.py`)

97 | Flag | Description |

98 -|------|-------------|

98 +| ---- | ----------- |

99 | `--encode-servers-urls` | Comma-separated list of encoder endpoints. Every multimodal item extracted from the request is fanned out to one of these URLs in a round-robin fashion. |

100 | `--prefill-servers-urls` | Comma-separated list of prefill endpoints. Set to `disable`, `none`, or `""` to skip the dedicated prefill phase and run E+PD (encoder + combined prefill/decode). |

101 | `--decode-servers-urls` | Comma-separated list of decode endpoints. Non-stream and stream paths both round-robin over this list. |

examples/pooling/embed/openai_embedding_long_text/README.md

+4 -4

@@ -34,7 +34,7 @@ python client.py

34 ## 📁 Files

36 | File | Description |

37 -|------|-------------|

37 +| ---- | ----------- |

38 | `service.sh` | Server startup script with chunked processing enabled |

39 | `client.py` | Comprehensive test client for long text embedding |

@@ -61,7 +61,7 @@ The key parameters for chunked processing are in the `--pooler-config`:

61 Chunked processing uses **MEAN aggregation** for cross-chunk combination when input exceeds the model's native maximum length:

63 | Component | Behavior | Description |

64 -|-----------|----------|-------------|

64 +| --------- | -------- | ----------- |

65 | **Within chunks** | Model's native pooling | Uses the model's configured pooling strategy |

66 | **Cross-chunk aggregation** | Always MEAN | Weighted averaging based on chunk token counts |

67 | **Performance** | Optimal | All chunks processed for complete semantic coverage |

@@ -69,7 +69,7 @@ Chunked processing uses **MEAN aggregation** for cross-chunk combination when in

69 ### Environment Variables

71 | Variable | Default | Description |

72 -|----------|---------|-------------|

72 +| -------- | ------- | ----------- |

73 | `MODEL_NAME` | `intfloat/multilingual-e5-large` | Embedding model to use (supports multiple models) |

74 | `PORT` | `31090` | Server port |

75 | `GPU_COUNT` | `1` | Number of GPUs to use |

@@ -106,7 +106,7 @@ With `MAX_EMBED_LEN=3072000`, you can process:

106 ### Chunked Processing Performance

107

108 | Aspect | Behavior | Performance |

109 -|--------|----------|-------------|

109 +| ------ | -------- | ----------- |

110 | **Chunk Processing** | All chunks processed with native pooling | Consistent with input length |

111 | **Cross-chunk Aggregation** | MEAN weighted averaging | Minimal overhead |

112 | **Memory Usage** | Proportional to number of chunks | Moderate, scalable |

tools/pre_commit/generate_attention_backend_docs.py

+6 -6

@@ -1153,11 +1153,11 @@ def _render_table(

1153 ) -> list[str]:

1154 """Render a markdown table from column specs and backend data."""

1155 header = "| " + " | ".join(name for name, _ in columns) + " |"

1156 - sep = "|" + "|".join("-" * (len(name) + 2) for name, _ in columns) + "|"

1156 + sep = "| " + " | ".join("-" * len(name) for name, _ in columns) + " |"

1157 lines = [header, sep]

1158 for info in sorted(backends, key=_sort_key):

1159 row = "| " + " | ".join(fmt(info) for _, fmt in columns) + " |"

1160 - lines.append(row)

1160 + lines.append(row.replace(" ", " "))

1161 return lines

1162

1163

@@ -1268,7 +1268,7 @@ def _priority_table(title: str, backends: list[str]) -> list[str]:

1268 f"**{title}:**",

1269 "",

1270 "| Priority | Backend |",

1271 - "|----------|---------|",

1271 + "| -------- | ------- |",

1272 *[f"| {i} | `{b}` |" for i, b in enumerate(backends, 1)],

1273 "",

1274 ]

@@ -1317,7 +1317,7 @@ def generate_legend() -> str:

1317 return """## Legend

1318

1319 | Column | Description |

1320 -|--------|-------------|

1320 +| ------ | ----------- |

1321 | **Dtypes** | Supported model data types (fp16, bf16, fp32) |

1322 | **KV Dtypes** | Supported KV cache data types (`auto`, `fp8`, `fp8_e4m3`, etc.) |

1323 | **Block Sizes** | Supported KV cache block sizes (%N means multiples of N) |

@@ -1348,7 +1348,7 @@ def generate_mla_section(

1348 "configuration.",

1349 "",

1351 - "|---------|-------------|--------------|--------|---------|-------|",

1351 + "| ------- | ----------- | ------------ | ------ | ------- | ----- |",

1352 ]

1353

1354 for backend in prefill_backends:

@@ -1360,7 +1360,7 @@ def generate_mla_section(

1360 backend["disable"],

1361 backend.get("notes", ""),

1362 )

1363 - lines.append(row)

1363 + lines.append(row.replace(" ", " "))

1364

1365 lines.extend(

1366 [

vllm/lora/ops/triton_ops/README_TUNING.md

+6 -6

@@ -43,14 +43,14 @@ Multi-lora shrink/expand Triton kernel tuning follows a similar methodology from

44 ### File Naming

46 -| Kernel Type | File Name Template | Example |

47 -|---------------------------|--------------------------------------------|---------------------------------------------|

48 -| shrink | `{gpu_name}_SHRINK.json` | `NVIDIA_H200_SHRINK.json` |

49 -| expand | `{gpu_name}_EXPAND_{add_input}.json` | `NVIDIA_H200_EXPAND_TRUE.json` |

46 +| Kernel Type | File Name Template | Example |

47 +| ------------------------- | ------------------------------------------- | -------------------------------------------- |

48 +| shrink | `{gpu_name}_SHRINK.json` | `NVIDIA_H200_SHRINK.json` |

49 +| expand | `{gpu_name}_EXPAND_{add_input}.json` | `NVIDIA_H200_EXPAND_TRUE.json` |

50 | fused_moe_lora_w13_shrink | `{gpu_name}_FUSED_MOE_LORA_W13_SHRINK.json` | `NVIDIA_H200_FUSED_MOE_LORA_W13_SHRINK.json` |

51 | fused_moe_lora_w13_expand | `{gpu_name}_FUSED_MOE_LORA_W13_EXPAND.json` | `NVIDIA_H200_FUSED_MOE_LORA_W13_EXPAND.json` |

52 -| fused_moe_lora_w2_shrink | `{gpu_name}_FUSED_MOE_LORA_W2_SHRINK.json` | `NVIDIA_H200_FUSED_MOE_LORA_W2_SHRINK.json` |

53 -| fused_moe_lora_w2_expand | `{gpu_name}_FUSED_MOE_LORA_W2_EXPAND.json` | `NVIDIA_H200_FUSED_MOE_LORA_W2_EXPAND.json` |

52 +| fused_moe_lora_w2_shrink | `{gpu_name}_FUSED_MOE_LORA_W2_SHRINK.json` | `NVIDIA_H200_FUSED_MOE_LORA_W2_SHRINK.json` |

53 +| fused_moe_lora_w2_expand | `{gpu_name}_FUSED_MOE_LORA_W2_EXPAND.json` | `NVIDIA_H200_FUSED_MOE_LORA_W2_EXPAND.json` |

55 The `gpu_name` can be automatically detected by calling `torch.cuda.get_device_name()`.