A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 7, 06:30 UTC
[Doc] Add MTP docs and update speculative decoding guidance (#35197)
Signed-off-by: liuxing <945764858@qq.com>
Xing Liu committed
Mar 5, 2026 at 01:23 UTC
7cc6058ac69009b7d595c891f0b439d1d6b0351d
3 files changed
+79
-4
docs/features/speculative_decoding/README.md
+20
-1
@@ -6,14 +6,33 @@ To train your own draft models for optimized speculative decoding, see [vllm-pro
6
7
## vLLM Speculation Methods
8
9
-vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, draft models, and mlp provide the best latency reduction, while simpler methods such as n-gram and and suffix decoding provide modest speedups without increasing workload during peak traffic.
9
+vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models, and MLP provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
10
11
- [EAGLE](eagle.md)
12
+- [Multi-Token Prediction (MTP)](mtp.md)
13
- [Draft Model](draft_model.md)
14
- [Multi-Layer Perceptron](mlp.md)
15
- [N-Gram](n_gram.md)
16
- [Suffix Decoding](suffix.md)
17
18
+## Method Selection at a Glance
19
+
20
+Use this qualitative table as a starting point for method selection. Real gains
21
+depend on your model family, traffic pattern, hardware, and sampling settings.
22
+
23
+| Method | Low QPS (latency focused) | High QPS (throughput focused) | Notes |
24
+| --- | --- | --- | --- |
25
+| EAGLE | High gain | Medium to high gain | Strong general-purpose model-based method. |
26
+| MTP | High gain | Medium to high gain | Best when the target model has native MTP support. |
27
+| Draft model | High gain | Medium gain | Needs a separate draft model. |
28
+| MLP speculator | Medium to high gain | Medium gain | Good when compatible MLP speculators are available. |
29
+| N-gram | Low to medium gain | Medium gain | Lightweight and easy to enable. |
30
+| Suffix decoding | Low to medium gain | Medium gain | No extra draft model; dynamic speculation depth. |
31
+
32
+For reproducible measurements in your environment, use
33
+[`examples/offline_inference/spec_decode.py`](../../../examples/offline_inference/spec_decode.py)
34
+or the [benchmark CLI guide](../../benchmarking/cli.md).
35
+
36
## Lossless guarantees of Speculative Decoding
37
38
In vLLM, speculative decoding aims to enhance inference efficiency while maintaining accuracy. This section addresses the lossless guarantees of
docs/features/speculative_decoding/mlp.md
+9
-3
@@ -11,10 +11,10 @@ prompts = ["The future of AI is"]
11
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
12
13
llm = LLM(
14
- model="meta-llama/Meta-Llama-3.1-70B-Instruct",
15
- tensor_parallel_size=4,
14
+ model="meta-llama/Meta-Llama-3.1-8B-Instruct",
15
+ tensor_parallel_size=1,
16
speculative_config={
17
- "model": "ibm-ai-platform/llama3-70b-accelerator",
17
+ "model": "ibm-ai-platform/llama3-8b-accelerator",
18
"draft_tensor_parallel_size": 1,
19
"method": "mlp_speculator",
20
},
@@ -27,6 +27,12 @@ for output in outputs:
27
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
28
```
29
30
+!!! warning "Known issue"
31
+ `ibm-ai-platform/llama3-70b-accelerator` can fail with:
32
+ `AttributeError: 'MLPSpeculatorConfig' object has no attribute 'num_attention_heads'`.
33
+ Track status in [#34106](https://github.com/vllm-project/vllm/issues/34106)
34
+ and [#34163](https://github.com/vllm-project/vllm/pull/34163).
35
+
36
## Pre-Trained MLP Drafter Models
37
38
A variety of speculative models of this type are available on HF hub:
docs/features/speculative_decoding/mtp.md
new
+50
@@ -0,0 +1,50 @@
1
+# MTP (Multi-Token Prediction)
2
+
3
+MTP is a speculative decoding method where the target model includes native
4
+multi-token prediction capability. Unlike draft-model-based methods, you do not
5
+need to provide a separate draft model.
6
+
7
+MTP is useful when:
8
+
9
+- Your model natively supports MTP.
10
+- You want model-based speculative decoding with minimal extra configuration.
11
+
12
+## Offline Example
13
+
14
+```python
15
+from vllm import LLM, SamplingParams
16
+
17
+prompts = ["The future of AI is"]
18
+sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
19
+
20
+llm = LLM(
21
+ model="XiaomiMiMo/MiMo-7B-Base",
22
+ tensor_parallel_size=1,
23
+ speculative_config={
24
+ "method": "mtp",
25
+ "num_speculative_tokens": 1,
26
+ },
27
+)
28
+outputs = llm.generate(prompts, sampling_params)
29
+
30
+for output in outputs:
31
+ prompt = output.prompt
32
+ generated_text = output.outputs[0].text
33
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
34
+```
35
+
36
+## Online Example
37
+
38
+```bash
39
+vllm serve XiaomiMiMo/MiMo-7B-Base \
40
+ --tensor-parallel-size 1 \
41
+ --speculative_config '{"method":"mtp","num_speculative_tokens":1}'
42
+```
43
+
44
+## Notes
45
+
46
+- MTP only works for model families that support MTP in vLLM.
47
+- `num_speculative_tokens` controls speculative depth. A small value like `1`
48
+ is a good default to start with.
49
+- If your model does not support MTP, use another method such as EAGLE or draft
50
+ model speculation.