@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 389aa1b2eb
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 12, 16:00 UTC

[Doc] Update more docs with respect to V1 (#29188)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Nov 23, 2025 at 10:58 UTC 389aa1b2ebf3726fef6aac737e0020075324d138
6 files changed +89 -100
docs/configuration/conserving_memory.md
-3
@@ -49,9 +49,6 @@ llm = LLM(model="adept/fuyu-8b", max_model_len=2048, max_num_seqs=2)
49
50 By default, we optimize model inference using CUDA graphs which take up extra memory in the GPU.
51
52 -!!! warning
53 - CUDA graph capture takes up more memory in V1 than in V0.
54 -
52 You can adjust `compilation_config` to achieve a better balance between inference speed and memory usage:
53
54 ??? code
docs/configuration/optimization.md
+1 -3
@@ -31,9 +31,7 @@ In vLLM V1, the default preemption mode is `RECOMPUTE` rather than `SWAP`, as re
31
32 Chunked prefill allows vLLM to process large prefills in smaller chunks and batch them together with decode requests. This feature helps improve both throughput and latency by better balancing compute-bound (prefill) and memory-bound (decode) operations.
33
34 -In vLLM V1, **chunked prefill is always enabled by default**. This is different from vLLM V0, where it was conditionally enabled based on model characteristics.
35 -
36 -With chunked prefill enabled, the scheduling policy prioritizes decode requests. It batches all pending decode requests before scheduling any prefill operations. When there are available tokens in the `max_num_batched_tokens` budget, it schedules pending prefills. If a pending prefill request cannot fit into `max_num_batched_tokens`, it automatically chunks it.
34 +In V1, **chunked prefill is enabled by default whenever possible**. With chunked prefill enabled, the scheduling policy prioritizes decode requests. It batches all pending decode requests before scheduling any prefill operations. When there are available tokens in the `max_num_batched_tokens` budget, it schedules pending prefills. If a pending prefill request cannot fit into `max_num_batched_tokens`, it automatically chunks it.
35
36 This policy has two benefits:
37
docs/usage/reproducibility.md
+14 -20
@@ -1,21 +1,23 @@
1 # Reproducibility
2
3 vLLM does not guarantee the reproducibility of the results by default, for the sake of performance. To achieve
4 -reproducible results, you need to turn off multiprocessing to make the scheduling deterministic by setting `VLLM_ENABLE_V1_MULTIPROCESSING=0`.
4 +reproducible results:
5 +
6 +- In offline mode, you can either set `VLLM_ENABLE_V1_MULTIPROCESSING=0` which makes scheduling deterministic,
7 + or enable [batch invariance](../features/batch_invariance.md) to make the outputs insensitive to scheduling.
8 +- In online mode, you can only enable [batch invariance](../features/batch_invariance.md).
9
10 Example: [examples/offline_inference/reproducibility.py](../../examples/offline_inference/reproducibility.py)
11
12 !!! warning
13
10 - Applying the above settings [changes the random state in user code](#locality-of-random-state).
14 + Setting `VLLM_ENABLE_V1_MULTIPROCESSING=0` will change the random state of user code
15 + (i.e. the code that constructs [LLM][vllm.LLM] class).
16
17 !!! note
18
19 Even with the above settings, vLLM only provides reproducibility
20 when it runs on the same hardware and the same vLLM version.
16 - Also, the online serving API (`vllm serve`) does not support reproducibility
17 - because it is almost impossible to make the scheduling deterministic in the
18 - online setting.
21
22 ## Setting the global seed
23
@@ -23,25 +25,17 @@ The `seed` parameter in vLLM is used to control the random states for various ra
25
26 If a specific seed value is provided, the random states for `random`, `np.random`, and `torch.manual_seed` will be set accordingly.
27
26 -However, in some cases, setting the seed will also [change the random state in user code](#locality-of-random-state).
27 -
28 ### Default Behavior
29
30 In V1, the `seed` parameter defaults to `0` which sets the random state for each worker, so the results will remain consistent for each vLLM run even if `temperature > 0`.
31
32 -!!! note
33 -
34 - It is impossible to un-specify a seed for V1 because different workers need to sample the same outputs
35 - for workflows such as speculative decoding.
36 -
37 - For more information, see: <https://github.com/vllm-project/vllm/pull/17929>
38 -
39 -### Locality of random state
32 +It is impossible to un-specify a seed for V1 because different workers need to sample the same outputs
33 +for workflows such as speculative decoding. For more information, see: <https://github.com/vllm-project/vllm/pull/17929>
34
41 -The random state in user code (i.e. the code that constructs [LLM][vllm.LLM] class) is updated by vLLM under the following conditions:
35 +!!! note
36
43 -- For V0: The seed is specified.
44 -- For V1: The workers are run in the same process as user code, i.e.: `VLLM_ENABLE_V1_MULTIPROCESSING=0`.
37 + The random state in user code (i.e. the code that constructs [LLM][vllm.LLM] class) is updated by vLLM
38 + only if the workers are run in the same process as user code, i.e.: `VLLM_ENABLE_V1_MULTIPROCESSING=0`.
39
46 -By default, these conditions are not active so you can use vLLM without having to worry about
47 -accidentally making deterministic subsequent operations that rely on random state.
40 + By default, `VLLM_ENABLE_V1_MULTIPROCESSING=1` so you can use vLLM without having to worry about
41 + accidentally making deterministic subsequent operations that rely on random state.
docs/usage/v1_guide.md
+70 -66
@@ -4,9 +4,7 @@
4
5 We have fully deprecated V0. Please read [RFC #18571](https://github.com/vllm-project/vllm/issues/18571) for more details.
6
7 -V1 is now enabled by default for all supported use cases, and we will gradually enable it for every use case we plan to support. Please share any feedback on [GitHub](https://github.com/vllm-project/vllm) or in the [vLLM Slack](https://inviter.co/vllm-slack).
8 -
9 -## Why vLLM V1?
7 + If you have a use case that works on V0 Engine but not V1, please share it on [GitHub](https://github.com/vllm-project/vllm) or in the [vLLM Slack](https://inviter.co/vllm-slack).
8
9 vLLM V0 successfully supported a wide range of models and hardware, but as new features were developed independently, the system grew increasingly complex. This complexity made it harder to integrate new capabilities and introduced technical debt, revealing the need for a more streamlined and unified design.
10
@@ -32,16 +30,44 @@ Upgrade to vLLM’s Core Architecture](https://blog.vllm.ai/2025/01/27/v1-alpha-
30
31 This living user guide outlines a few known **important changes and limitations** introduced by vLLM V1. The team has been working actively to bring V1 as the default engine, therefore this guide will be updated constantly as more features get supported on vLLM V1.
32
35 -## Current Status
33 +## Differences from V0
34 +
35 +This section lists some differences in behavior between V0 and V1.
36 +
37 +### Chunked Prefill
38 +
39 +Chunked prefill is enabled by default whenever possible, unlike in V0 where it was conditionally enabled based on model characteristics.
40 +
41 +### CUDA Graphs
42 +
43 +CUDA graph capture takes up more memory in V1 than in V0.
44 +
45 +### Semantic Changes to Logprobs
46 +
47 +#### Logprobs Calculation
48 +
49 +By default, logprobs in V1 are now returned immediately once computed from the model’s raw output (i.e.
50 +before applying any logits post-processing such as temperature scaling or penalty
51 +adjustments). As a result, the returned logprobs do not reflect the final adjusted
52 +probabilities used during sampling.
53 +
54 +You can adjust this behavior by setting the `--logprobs-mode` flag.
55 +Four modes are supported: `raw_logprobs` (default), `processed_logprobs`, `raw_logits`, `processed_logits`.
56 +Raw means the values before applying any logit processors, like bad words.
57 +Processed means the values after applying all processors, including temperature and top_k/top_p.
58 +
59 +#### Prompt Logprobs with Prefix Caching
60 +
61 +While V1 supports passing prompt logprobs with prefix caching enabled, it no longer caches the logprobs.
62 +For a request requiring prompt logprobs, the engine will ignore the prefix cache and recompute the prefill of full prompt to generate the logprobs.
63 +
64 +## Feature Support
65
37 -For each item, our progress towards V1 support falls into one of the following states:
66 +For each item, its support in vLLM V1 falls into one of the following states:
67
39 -- **🚀 Optimized**: Nearly fully optimized, with no further work currently planned.
40 -- **🟢 Functional**: Fully operational, with ongoing optimizations.
41 -- **🚧 WIP**: Under active development.
42 -- **🟡 Planned**: Scheduled for future implementation (some may have open PRs/RFCs).
43 -- **🟠 Delayed**: Temporarily dropped in V1 but planned to be re-introduced later.
44 -- **🔴 Deprecated**: Not planned for V1 unless there is strong demand.
68 +- **🟢 Functional**: Fully operational with optimizations comparable to or better than V0.
69 +- **🟡 In Progress**: Planned to be in vLLM V1, with open PRs/RFCs.
70 +- **🔴 Removed**: Dropped from vLLM V1. Will only consider re-introducing if there is strong demand.
71
72 !!! note
73 vLLM V1’s unified scheduler treats both prompt and output tokens the same
@@ -57,13 +83,13 @@ based on assigned priority, with FCFS as a tie-breaker), configurable via the
83
84 ### Hardware
85
60 -| Hardware | Status |
61 -|------------|-----------------------------------------------|
62 -| **NVIDIA** | <nobr>🚀</nobr> |
63 -| **AMD** | <nobr>🟢</nobr> |
86 +| Hardware | Status |
87 +|------------------|-----------------------------------------------|
88 +| **NVIDIA** | <nobr>🟢</nobr> |
89 +| **AMD** | <nobr>🟢</nobr> |
90 | **INTEL GPU** | <nobr>🟢</nobr> |
65 -| **TPU** | <nobr>🟢</nobr> |
66 -| **CPU** | <nobr>🟢 (x86\_64/aarch64) 🟡 (MacOS) </nobr> |
91 +| **TPU** | <nobr>🟢</nobr> |
92 +| **CPU** | <nobr>🟢</nobr> |
93
94 !!! note
95
@@ -78,23 +104,21 @@ based on assigned priority, with FCFS as a tie-breaker), configurable via the
104
105 ### Models
106
81 -| Model Type | Status |
82 -|-----------------------------|------------------------------------------------------------------------------------|
83 -| **Decoder-only Models** | <nobr>🚀 Optimized</nobr> |
84 -| **Encoder-Decoder Models** | <nobr>🟢 Whisper only</nobr> |
85 -| **Embedding Models** | <nobr>🟢 Functional</nobr> |
86 -| **Mamba Models** | <nobr>🟢 (Mamba-2), 🟢 (Mamba-1)</nobr> |
87 -| **Multimodal Models** | <nobr>🟢 Functional</nobr> |
107 +| Model Type | Status |
108 +|-----------------------------|-------------------------------------------------------------------------|
109 +| **Decoder-only Models** | <nobr>🟢</nobr> |
110 +| **Encoder-Decoder Models** | <nobr>🟢 (Whisper), 🔴 (Others) </nobr> |
111 +| **Pooling Models** | <nobr>🟢</nobr> |
112 +| **Mamba Models** | <nobr>🟢</nobr> |
113 +| **Multimodal Models** | <nobr>🟢</nobr> |
114
115 See below for the status of models that are not yet supported or have more features planned in V1.
116
91 -#### Embedding Models
117 +#### Pooling Models
118
93 -The initial basic support is now functional.
119 +Now fully supported, with prefix caching and chunked prefill newly available for last-pooling models.
120
95 -Later, we will consider using [hidden states processor](https://github.com/vllm-project/vllm/issues/12249),
96 -which is based on [global logits processor](https://github.com/vllm-project/vllm/pull/13360)
97 -to enable simultaneous generation and embedding using the same engine instance in V1.
121 +We are working on enabling prefix caching and chunked prefill for more categories of pooling models.
122
123 #### Mamba Models
124
@@ -112,24 +136,25 @@ Please note that prefix caching is not yet supported for any of the above models
136
137 Whisper is supported. Other models requiring cross-attention between separate
138 encoder and decoder (e.g., `BartForConditionalGeneration`,
115 -`MllamaForConditionalGeneration`) are not supported.
139 +`MllamaForConditionalGeneration`) are no longer supported.
140
141 ### Features
142
143 | Feature | Status |
144 |---------------------------------------------|-----------------------------------------------------------------------------------|
121 -| **Prefix Caching** | <nobr>🚀 Optimized</nobr> |
122 -| **Chunked Prefill** | <nobr>🚀 Optimized</nobr> |
123 -| **LoRA** | <nobr>🚀 Optimized</nobr> |
145 +| **Prefix Caching** | <nobr>🟢 Functional</nobr> |
146 +| **Chunked Prefill** | <nobr>🟢 Functional</nobr> |
147 +| **LoRA** | <nobr>🟢 Functional</nobr> |
148 | **Logprobs Calculation** | <nobr>🟢 Functional</nobr> |
125 -| **FP8 KV Cache** | <nobr>🟢 Functional on Hopper devices (<https://github.com/vllm-project/vllm/pull/15191>)</nobr>|
126 -| **Spec Decode** | <nobr>🚀 Optimized</nobr> |
127 -| **Prompt Logprobs with Prefix Caching** | <nobr>🟡 Planned ([RFC #13414](https://github.com/vllm-project/vllm/issues/13414))</nobr>|
149 +| **FP8 KV Cache** | <nobr>🟢 Functional</nobr> |
150 +| **Spec Decode** | <nobr>🟢 Functional</nobr> |
151 +| **Prompt Logprobs with Prefix Caching** | <nobr>🟢 Functional</nobr> |
152 | **Structured Output Alternative Backends** | <nobr>🟢 Functional</nobr> |
129 -| **Request-level Structured Output Backend** | <nobr>🔴 Deprecated</nobr> |
130 -| **best_of** | <nobr>🔴 Deprecated ([RFC #13361](https://github.com/vllm-project/vllm/issues/13361))</nobr>|
131 -| **Per-Request Logits Processors** | <nobr>🔴 Deprecated ([RFC #13360](https://github.com/vllm-project/vllm/pull/13360))</nobr> |
132 -| **GPU <> CPU KV Cache Swapping** | <nobr>🔴 Deprecated</nobr> |
153 +| **Concurrent Partial Prefills** | <nobr>🟡 [In Progress](https://github.com/vllm-project/vllm/issues/14003)</nobr> |
154 +| **best_of** | <nobr>🔴 [Removed](https://github.com/vllm-project/vllm/issues/13361)</nobr> |
155 +| **Per-Request Logits Processors** | <nobr>🔴 [Removed](https://github.com/vllm-project/vllm/pull/13360)</nobr> |
156 +| **GPU <> CPU KV Cache Swapping** | <nobr>🔴 Removed</nobr> |
157 +| **Request-level Structured Output Backend** | <nobr>🔴 Removed</nobr> |
158
159 !!! note
160
@@ -139,37 +164,16 @@ encoder and decoder (e.g., `BartForConditionalGeneration`,
164 prefix caching, and speculative decoding without a strict separation between prefill
165 and decode phases.
166
142 -#### Semantic Changes to Logprobs
143 -
144 -vLLM V1 supports logprobs and prompt logprobs. However, there are some important semantic
145 -differences compared to V0:
146 -
147 -##### Logprobs Calculation
148 -
149 -By default, logprobs in V1 are now returned immediately once computed from the model’s raw output (i.e.
150 -before applying any logits post-processing such as temperature scaling or penalty
151 -adjustments). As a result, the returned logprobs do not reflect the final adjusted
152 -probabilities used during sampling.
153 -
154 -You can adjust this behavior by setting the `--logprobs-mode` flag.
155 -Four modes are supported: `raw_logprobs` (default), `processed_logprobs`, `raw_logits`, `processed_logits`.
156 -Raw means the values before applying any logit processors, like bad words.
157 -Processed means the values after applying all processors, including temperature and top_k/top_p.
158 -
159 -##### Prompt Logprobs with Prefix Caching
160 -
161 -Logprobs are not cached. For a request requiring prompt logprobs, the engine will ignore the prefix cache and recompute the prefill of full prompt to generate the logprobs.
162 -
163 -#### Deprecated Features
167 +#### Removed Features
168
165 -As part of the major architectural rework in vLLM V1, several legacy features have been deprecated.
169 +As part of the major architectural rework in vLLM V1, several legacy features have been removed.
170
171 ##### Sampling features
172
169 -- **best_of**: This feature has been deprecated due to limited usage. See details at [RFC #13361](https://github.com/vllm-project/vllm/issues/13361).
173 +- **best_of**: This feature has been removed due to limited usage. See details at [RFC #13361](https://github.com/vllm-project/vllm/issues/13361).
174 - **Per-Request Logits Processors**: In V0, users could pass custom
175 processing functions to adjust logits on a per-request basis. In vLLM V1, this
172 - feature has been deprecated. Instead, we now support **global logits processors**
176 + feature has been removed. Instead, we now support **global logits processors**
177 which are set at startup time, see [RFC #17799](https://github.com/vllm-project/vllm/issues/17799).
178
179 ##### KV Cache features
@@ -179,4 +183,4 @@ to handle request preemptions.
183
184 ##### Structured Output features
185
182 -- **Request-level Structured Output Backend**: Deprecated, alternative backends (outlines, guidance) with fallbacks is supported now.
186 +- **Request-level Structured Output Backend**: Removed; alternative backends (outlines, guidance) with fallbacks are supported now.
examples/offline_inference/reproducibility.py
+4 -1
@@ -11,8 +11,11 @@ import random
11
12 from vllm import LLM, SamplingParams
13
14 -# Turn off multiprocessing to make the scheduling deterministic.
14 +# Either:
15 +## Turn off multiprocessing to make the scheduling deterministic, or
16 os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"
17 +## Enable batch invariance to get consistent results regardless of scheduling.
18 +os.environ["VLLM_BATCH_INVARIANT"] = "1"
19
20 prompts = [
21 "Hello, my name is",
tests/models/language/generation/test_common.py
-7
@@ -10,13 +10,6 @@ from ....utils import large_gpu_mark
10 from ...registry import HF_EXAMPLE_MODELS
11 from ...utils import check_logprobs_close
12
13 -# These have unsupported head_dim for FA. We do not
14 -# have a clean way to fall back, so we fail with
15 -# a clear msg when it happens.
16 -# https://github.com/vllm-project/vllm/issues/14524
17 -# NOTE(woosuk): Skipping these tests until V1 supports them.
18 -# REQUIRES_V0 = ["microsoft/phi-2", "stabilityai/stablelm-3b-4e1t"]
19 -
13 # This list contains the model that are using AITER kernel.
14 # Skip model that are not using AITER tests.
15 # When more AITER kernels are added, this list will not be