A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 7, 05:15 UTC
[Doc] Add Parallel Draft Models (#35973)
Signed-off-by: <zihaoan2@amd.com> Signed-off-by: zihaoanllm <zihaoan2@amd.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
zihaoanllm committed
Mar 5, 2026 at 13:44 UTC
d106bf39f56cdc59d08a84094c0de41a0be9ad0f
3 files changed
+51
-1
docs/features/speculative_decoding/README.md
+3
-1
@@ -6,11 +6,12 @@ To train your own draft models for optimized speculative decoding, see [vllm-pro
6
7
## vLLM Speculation Methods
8
9
-vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models, and MLP provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
9
+vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models, PARD and MLP provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
10
11
- [EAGLE](eagle.md)
12
- [Multi-Token Prediction (MTP)](mtp.md)
13
- [Draft Model](draft_model.md)
14
+- [Parallel Draft Model (PARD)](parallel_draft_model.md)
15
- [Multi-Layer Perceptron](mlp.md)
16
- [N-Gram](n_gram.md)
17
- [Suffix Decoding](suffix.md)
@@ -25,6 +26,7 @@ depend on your model family, traffic pattern, hardware, and sampling settings.
26
| EAGLE | High gain | Medium to high gain | Strong general-purpose model-based method. |
27
| MTP | High gain | Medium to high gain | Best when the target model has native MTP support. |
28
| Draft model | High gain | Medium gain | Needs a separate draft model. |
29
+| Parallel Draft Model | High gain | Medium to high gain | Low draft model latency. |
30
| MLP speculator | Medium to high gain | Medium gain | Good when compatible MLP speculators are available. |
31
| N-gram | Low to medium gain | Medium gain | Lightweight and easy to enable. |
32
| Suffix decoding | Low to medium gain | Medium gain | No extra draft model; dynamic speculation depth. |
docs/features/speculative_decoding/parallel_draft_model.md
new
+46
@@ -0,0 +1,46 @@
1
+# Parallel Draft Models
2
+
3
+The following code configures vLLM to use speculative decoding where proposals are generated by [PARD](https://arxiv.org/pdf/2504.18583) (Parallel Draft Models).
4
+
5
+## PARD Offline Mode Example
6
+
7
+```python
8
+from vllm import LLM, SamplingParams
9
+
10
+prompts = ["The future of AI is"]
11
+sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
12
+
13
+llm = LLM(
14
+ model="Qwen/Qwen3-8B",
15
+ tensor_parallel_size=1,
16
+ speculative_config={
17
+ "model": "amd/PARD-Qwen3-0.6B",
18
+ "num_speculative_tokens": 12,
19
+ "method": "draft_model",
20
+ "parallel_drafting": True,
21
+ },
22
+)
23
+outputs = llm.generate(prompts, sampling_params)
24
+
25
+for output in outputs:
26
+ prompt = output.prompt
27
+ generated_text = output.outputs[0].text
28
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
29
+```
30
+
31
+## PARD Online Mode Example
32
+
33
+```bash
34
+vllm serve Qwen/Qwen3-4B \
35
+ --host 0.0.0.0 \
36
+ --port 8000 \
37
+ --seed 42 \
38
+ -tp 1 \
39
+ --max_model_len 2048 \
40
+ --gpu_memory_utilization 0.8 \
41
+ --speculative_config '{"model": "amd/PARD-Qwen3-0.6B", "num_speculative_tokens": 12, "method": "draft_model", "parallel_drafting": true}'
42
+```
43
+
44
+## Pre-trained PARD weights
45
+
46
+- [amd/pard](https://huggingface.co/collections/amd/pard)
pyproject.toml
+2
@@ -175,6 +175,8 @@ tme = "tme"
175
dout = "dout"
176
Pn = "Pn"
177
arange = "arange"
178
+PARD = "PARD"
179
+pard = "pard"
180
181
[tool.typos.type.py]
182
extend-glob = []