@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 236 lines 13.1 KB
Rendered Raw
1 # Speculative Decoding
2
3 This document shows how to use [Speculative Decoding](https://arxiv.org/pdf/2302.01318) with vLLM to reduce inter-token latency under medium-to-low QPS (queries per second), memory-bound workloads.
4
5 To train your own draft models for optimized speculative decoding, see [vllm-project/speculators](speculators.md) for seamless training and integration with vLLM.
6
7 ## vLLM Speculation Methods
8
9 vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models, PARD and MLP provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
10
11 - [EAGLE](eagle.md)
12 - [LiLiCorr](lilicorr.md)
13 - [Multi-Token Prediction (MTP)](mtp.md)
14 - [Draft Model](draft_model.md)
15 - [Parallel Draft Model (PARD)](parallel_draft_model.md)
16 - [Multi-Layer Perceptron](mlp.md)
17 - [N-Gram](n_gram.md)
18 - [Suffix Decoding](suffix.md)
19 - [Hidden State Extraction](extract_hidden_states.md)
20 - [Custom Proposer Backend (Experimental)](#custom-proposer-backend-experimental)
21 - [Dynamic Speculative Decoding](dynamic_speculative_decoding.md)
22 - [Adaptive Verification](adaptive_verification.md)
23 - [Per-Request Acceptance Metrics](acceptance_metrics.md)
24
25 ## Method Selection at a Glance
26
27 Use this qualitative table as a starting point for method selection. Real gains
28 depend on your model family, traffic pattern, hardware, and sampling settings.
29
30 | Method | Low QPS (latency focused) | High QPS (throughput focused) | Notes |
31 | --- | --- | --- | --- |
32 | EAGLE | High gain | Medium to high gain | Strong general-purpose model-based method. |
33 | MTP | High gain | Medium to high gain | Best when the target model has native MTP support. |
34 | Draft model | High gain | Medium gain | Needs a separate draft model. |
35 | Parallel Draft Model | High gain | Medium to high gain | Low draft model latency. |
36 | MLP speculator | Medium to high gain | Medium gain | Good when compatible MLP speculators are available. |
37 | N-gram | Low to medium gain | Medium gain | Lightweight and easy to enable. |
38 | Suffix decoding | Low to medium gain | Medium gain | No extra draft model; dynamic speculation depth. |
39 | Custom Proposer | Varies | Varies | Bring your own proposer class (experimental). |
40 | Dynamic Speculative Decoding | High gain | Higher than base SD method | Useful for RL or workload with fluctuating QPS |
41 | Adaptive Verification | High gain | Higher than base SD method | Sizes verification per request from drafter confidence; currently DSpark only. |
42
43 For reproducible measurements in your environment, use
44 [`examples/features/speculative_decoding/spec_decode_offline.py`](../../../examples/features/speculative_decoding/spec_decode_offline.py)
45 or the [benchmark CLI guide](../../benchmarking/cli.md).
46
47 ## Custom Proposer Backend (Experimental)
48
49 You can plug in your own custom proposer class for speculative decoding by setting the method to `custom_class` and providing the full module path to your class.
50 Your custom class must accept a `VllmConfig` upon instantiation and implement a `propose` method.
51
52 **Example configuration:**
53
54 - `speculative_config.method = "custom_class"`
55 - `speculative_config.model = "your_module.YourCustomProposerClass"`
56
57 ## `--speculative-config` schema
58
59 Use `--speculative-config` to pass speculative decoding settings as a JSON
60 object on the CLI:
61
62 ```bash
63 vllm serve <target-model> \
64 --speculative-config '{
65 "method": "draft_model",
66 "model": "<draft-model>",
67 "num_speculative_tokens": 5
68 }'
69 ```
70
71 The same keys are accepted from Python via `LLM(..., speculative_config={...})`.
72 The tables below highlight common user-facing keys accepted in this JSON
73 object; they are not an exhaustive schema reference.
74 For more details, see the generated [engine arguments reference](../../configuration/engine_args.md)
75 and the API docs for [vllm.config.SpeculativeConfig][].
76
77 ### Common keys
78
79 These keys are commonly used across speculative decoding setups, though some
80 only apply to model-based methods such as `draft_model`, `mtp`, `eagle3`, and
81 `dflash`.
82
83 | Key | Type | Default | Allowed values / meaning |
84 | --- | --- | --- | --- |
85 | `method` | `string` | `None` | Speculation method. Common values include `draft_model`, `ngram`, `suffix`, `mtp`, `eagle3`, and `dflash`. If omitted, vLLM infers the method from the provided configuration when possible. |
86 | `model` | `string` | `None` | Draft model, EAGLE head, or auxiliary model identifier. For `ngram`, `ngram_gpu`, `suffix`, and `mtp`, this can often be omitted. |
87 | `num_speculative_tokens` | `integer > 0` | `None` | Number of speculative tokens to propose per step. Required for methods that do not infer it from model metadata. |
88 | `draft_tensor_parallel_size` | `integer >= 1` | `None` | Tensor parallel size for the draft model. |
89 | `max_model_len` | `integer >= 1` | `None` | Maximum context length for the draft model. |
90 | `parallel_drafting` | `boolean` | `false` | Enable parallel draft token generation. Only compatible with EAGLE and draft-model methods. |
91 | `rejection_sample_method` | `string` | `standard` | `standard`, `synthetic`, or `block`. |
92 | `synthetic_acceptance_rates` | `list[float]` | `None` | Per-position unconditional acceptance rates for `synthetic` rejection sampling. Each entry in `[0, 1]`; length must equal `num_speculative_tokens`; must be non-increasing. |
93 | `synthetic_acceptance_length` | `float` | `None` | Target mean acceptance length for `synthetic`; in `[1, num_speculative_tokens + 1]`. Mutually exclusive with `synthetic_acceptance_rates`. |
94 | `use_heterogeneous_vocab` | `boolean` | `false` | Allow draft and target models with different vocabularies. Builds a token-level intersection at initialisation and constrains draft logits to shared tokens only. Only compatible with `method=draft_model`. Probabilistic draft sampling (`draft_sample_method='probabilistic'`) is not yet supported when this option is enabled. |
95
96 !!! note
97 Gemma 4 assistant checkpoints are handled as Gemma 4 MTP speculators, not
98 as generic draft models. Use `"method": "mtp"` with the assistant
99 checkpoint in `model`, as shown in the [MTP guide](mtp.md#gemma-4-assistant-models).
100
101 If startup logs show `SpeculativeConfig(method='draft_model', ...)` for a
102 Gemma 4 assistant checkpoint, the installed vLLM version does not include
103 Gemma 4 MTP support for that path. Upgrade to a version that includes
104 Gemma 4 MTP support instead of forcing the assistant checkpoint through
105 generic draft-model speculative decoding.
106
107 ### Method-specific keys
108
109 #### N-gram
110
111 | Key | Type | Default | Meaning |
112 | --- | --- | --- | --- |
113 | `prompt_lookup_max` | `integer >= 1` | `5` if both lookup bounds are omitted; otherwise mirrors `prompt_lookup_min` when omitted | Maximum n-gram window size. |
114 | `prompt_lookup_min` | `integer >= 1` | `5` if both lookup bounds are omitted; otherwise mirrors `prompt_lookup_max` when omitted | Minimum n-gram window size. |
115
116 Example:
117
118 ```bash
119 vllm serve <target-model> \
120 --speculative-config '{
121 "method": "ngram",
122 "num_speculative_tokens": 4,
123 "prompt_lookup_min": 2,
124 "prompt_lookup_max": 5
125 }'
126 ```
127
128 #### Suffix decoding
129
130 | Key | Type | Default | Meaning |
131 | --- | --- | --- | --- |
132 | `suffix_decoding_max_tree_depth` | `integer` | `24` | Maximum combined prefix-match and speculation tree depth. |
133 | `suffix_decoding_max_cached_requests` | `integer` | `10000` | Maximum number of requests cached in the global suffix tree. Set `0` to disable the global cache. |
134 | `suffix_decoding_max_spec_factor` | `float` | `1.0` | Caps speculative length as a multiple of prefix-match length. |
135 | `suffix_decoding_min_token_prob` | `float` | `0.1` | Minimum estimated token probability required to speculate a token. |
136
137 Example:
138
139 ```bash
140 vllm serve <target-model> \
141 --speculative-config '{
142 "method": "suffix",
143 "num_speculative_tokens": 8,
144 "suffix_decoding_max_tree_depth": 24,
145 "suffix_decoding_max_cached_requests": 10000,
146 "suffix_decoding_max_spec_factor": 1.0,
147 "suffix_decoding_min_token_prob": 0.1
148 }'
149 ```
150
151 #### Cross-Vocabulary Draft Models (TLI)
152
153 By default, vLLM requires the draft and target models to share the same
154 vocabulary. Setting `use_heterogeneous_vocab: true` enables the
155 **Token-Level Intersection (TLI)** algorithm, which allows draft models
156 from a different model family with a different tokenizer.
157
158 At initialisation, vLLM builds a mapping between the two vocabularies by
159 normalising token strings and computing their intersection. Draft logits are
160 constrained to the shared tokens before sampling, and the sampled token IDs
161 are translated to the target vocabulary before rejection sampling.
162
163 ```python
164 from vllm import LLM, SamplingParams
165
166 llm = LLM(
167 model="Qwen/Qwen3-8B",
168 speculative_config={
169 "method": "draft_model",
170 "model": "HuggingFaceTB/SmolLM2-135M-Instruct",
171 "num_speculative_tokens": 3,
172 "use_heterogeneous_vocab": True,
173 },
174 gpu_memory_utilization=0.5,
175 )
176 ```
177
178 ### Notes
179
180 - `--speculative-config` expects a JSON object on the CLI. In YAML config
181 files, use a nested mapping instead of an escaped JSON string.
182 - `tensor_parallel_size` is not a valid key in `speculative_config`. Use
183 `draft_tensor_parallel_size` instead.
184 - Keys such as `temperature` and `top_p` are sampling parameters, not
185 `--speculative-config` fields.
186 - Internal fields such as `target_model_config`, `draft_model_config`,
187 `target_parallel_config`, `draft_parallel_config`, and `draft_load_config`
188 are populated by vLLM and are not intended to be set by users.
189 - `use_heterogeneous_vocab` currently supports greedy draft sampling only. Probabilistic acceptance (temperature > 0 draft sampling) is not yet supported and will be added in a future release.
190
191 ## Lossless guarantees of Speculative Decoding
192
193 In vLLM, speculative decoding aims to enhance inference efficiency while maintaining accuracy. This section addresses the lossless guarantees of
194 speculative decoding, breaking down the guarantees into three key areas:
195
196 1. **Theoretical Losslessness**
197 \- Speculative decoding sampling is theoretically lossless up to the precision limits of hardware numerics. Floating-point errors might
198 cause slight variations in output distributions, as discussed
199 in [Accelerating Large Language Model Decoding with Speculative Sampling](https://arxiv.org/pdf/2302.01318)
200
201 2. **Algorithmic Losslessness**
202 \- vLLM’s implementation of speculative decoding is algorithmically validated to be lossless. Key validation tests include:
203
204 > - **Rejection Sampler Convergence**: Ensures that samples from vLLM’s rejection sampler align with the target
205 > distribution. [View Test Code](https://github.com/vllm-project/vllm/blob/47b65a550866c7ffbd076ecb74106714838ce7da/tests/samplers/test_rejection_sampler.py#L252)
206 > - **Greedy Sampling Equality**: Confirms that greedy sampling with speculative decoding matches greedy sampling
207 > without it. This verifies that vLLM's speculative decoding framework, when integrated with the vLLM forward pass and the vLLM rejection sampler,
208 > provides a lossless guarantee. Almost all of the tests in [tests/spec_decode/e2e](../../../tests/v1/spec_decode)
209 > verify this property using [this assertion implementation](https://github.com/vllm-project/vllm/blob/b67ae00cdbbe1a58ffc8ff170f0c8d79044a684a/tests/spec_decode/e2e/conftest.py#L291)
210
211 3. **vLLM Logprob Stability**
212 \- vLLM does not currently guarantee stable token log probabilities (logprobs). This can result in different outputs for the
213 same request across runs. For more details, see the FAQ section
214 titled *Can the output of a prompt vary across runs in vLLM?* in the [FAQs](../../usage/faq.md).
215
216 While vLLM strives to ensure losslessness in speculative decoding, variations in generated outputs with and without speculative decoding
217 can occur due to following factors:
218
219 - **Floating-Point Precision**: Differences in hardware numerical precision may lead to slight discrepancies in the output distribution.
220 - **Batch Size and Numerical Stability**: Changes in batch size may cause variations in logprobs and output probabilities, potentially
221 due to non-deterministic behavior in batched operations or numerical instability.
222
223 For mitigation strategies, please refer to the FAQ entry *Can the output of a prompt vary across runs in vLLM?* in the [FAQs](../../usage/faq.md).
224
225 ## Known Feature Incompatibility
226
227 1. Pipeline parallelism is not composable with speculative decoding as of `vllm<=0.15.0`
228 2. Speculative decoding with draft models is not supported in `vllm<=0.10.0`
229
230 ## Resources for vLLM contributors
231
232 - [[vLLM Office Hours #40] Intro to Speculators](https://www.youtube.com/watch?v=2ISAr_JVGLs)
233 - [A Hacker's Guide to Speculative Decoding in vLLM](https://www.youtube.com/watch?v=9wNAgpX6z_4)
234 - [What is Lookahead Scheduling in vLLM?](https://docs.google.com/document/d/1Z9TvqzzBPnh5WHcRwjvK2UEeFeq5zMZb5mFE8jR0HCs/edit#heading=h.1fjfb0donq5a)
235 - [Information on batch expansion](https://docs.google.com/document/d/1T-JaS2T1NRfdP51qzqpyakoCXxSXTtORppiwaj5asxA/edit#heading=h.kk7dq05lc6q8)
236 - [Dynamic speculative decoding](https://github.com/vllm-project/vllm/issues/4565)