A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 4, 20:00 UTC
[Doc] Clarify supported keys for --speculative-config (#40455)
Signed-off-by: Wangxiaoxiaoa <Wangxiaoxiaoa@users.noreply.github.com> Co-authored-by: Wangxiaoxiaoa <Wangxiaoxiaoa@users.noreply.github.com>
xiao committed
Apr 22, 2026 at 19:36 UTC
ecbe42e9911bd0c0ae471ed573ea8b2d050a0e8e
4 files changed
+105
-8
docs/features/speculative_decoding/README.md
+93
@@ -35,6 +35,99 @@ For reproducible measurements in your environment, use
35
[`examples/offline_inference/spec_decode.py`](../../../examples/offline_inference/spec_decode.py)
36
or the [benchmark CLI guide](../../benchmarking/cli.md).
37
38
+## `--speculative-config` schema
39
+
40
+Use `--speculative-config` to pass speculative decoding settings as a JSON
41
+object on the CLI:
42
+
43
+```bash
44
+vllm serve <target-model> \
45
+ --speculative-config '{
46
+ "method": "draft_model",
47
+ "model": "<draft-model>",
48
+ "num_speculative_tokens": 5
49
+ }'
50
+```
51
+
52
+The same keys are accepted from Python via `LLM(..., speculative_config={...})`.
53
+The tables below highlight common user-facing keys accepted in this JSON
54
+object; they are not an exhaustive schema reference.
55
+For more details, see the generated [engine arguments reference](../../configuration/engine_args.md)
56
+and the API docs for [vllm.config.SpeculativeConfig][].
57
+
58
+### Common keys
59
+
60
+These keys are commonly used across speculative decoding setups, though some
61
+only apply to model-based methods such as `draft_model`, `mtp`, `eagle3`, and
62
+`dflash`.
63
+
64
+| Key | Type | Default | Allowed values / meaning |
65
+| --- | --- | --- | --- |
66
+| `method` | `string` | `None` | Speculation method. Common values include `draft_model`, `ngram`, `suffix`, `mtp`, `eagle3`, and `dflash`. If omitted, vLLM infers the method from the provided configuration when possible. |
67
+| `model` | `string` | `None` | Draft model, EAGLE head, or auxiliary model identifier. For `ngram`, `ngram_gpu`, `suffix`, and `mtp`, this can often be omitted. |
68
+| `num_speculative_tokens` | `integer > 0` | `None` | Number of speculative tokens to propose per step. Required for methods that do not infer it from model metadata. |
69
+| `draft_tensor_parallel_size` | `integer >= 1` | `None` | Tensor parallel size for the draft model. |
70
+| `max_model_len` | `integer >= 1` | `None` | Maximum context length for the draft model. |
71
+| `parallel_drafting` | `boolean` | `false` | Enable parallel draft token generation. Only compatible with EAGLE and draft-model methods. |
72
+| `rejection_sample_method` | `string` | `strict` | `strict`, `probabilistic`, or `synthetic`. |
73
+| `synthetic_acceptance_rate` | `float` | `None` | Average acceptance rate to target when `rejection_sample_method` is `synthetic`. Valid range is `[0, 1]`. |
74
+
75
+### Method-specific keys
76
+
77
+#### N-gram
78
+
79
+| Key | Type | Default | Meaning |
80
+| --- | --- | --- | --- |
81
+| `prompt_lookup_max` | `integer >= 1` | `5` if both lookup bounds are omitted; otherwise mirrors `prompt_lookup_min` when omitted | Maximum n-gram window size. |
82
+| `prompt_lookup_min` | `integer >= 1` | `5` if both lookup bounds are omitted; otherwise mirrors `prompt_lookup_max` when omitted | Minimum n-gram window size. |
83
+
84
+Example:
85
+
86
+```bash
87
+vllm serve <target-model> \
88
+ --speculative-config '{
89
+ "method": "ngram",
90
+ "num_speculative_tokens": 4,
91
+ "prompt_lookup_min": 2,
92
+ "prompt_lookup_max": 5
93
+ }'
94
+```
95
+
96
+#### Suffix decoding
97
+
98
+| Key | Type | Default | Meaning |
99
+| --- | --- | --- | --- |
100
+| `suffix_decoding_max_tree_depth` | `integer` | `24` | Maximum combined prefix-match and speculation tree depth. |
101
+| `suffix_decoding_max_cached_requests` | `integer` | `10000` | Maximum number of requests cached in the global suffix tree. Set `0` to disable the global cache. |
102
+| `suffix_decoding_max_spec_factor` | `float` | `1.0` | Caps speculative length as a multiple of prefix-match length. |
103
+| `suffix_decoding_min_token_prob` | `float` | `0.1` | Minimum estimated token probability required to speculate a token. |
104
+
105
+Example:
106
+
107
+```bash
108
+vllm serve <target-model> \
109
+ --speculative-config '{
110
+ "method": "suffix",
111
+ "num_speculative_tokens": 8,
112
+ "suffix_decoding_max_tree_depth": 24,
113
+ "suffix_decoding_max_cached_requests": 10000,
114
+ "suffix_decoding_max_spec_factor": 1.0,
115
+ "suffix_decoding_min_token_prob": 0.1
116
+ }'
117
+```
118
+
119
+### Notes
120
+
121
+- `--speculative-config` expects a JSON object on the CLI. In YAML config
122
+ files, use a nested mapping instead of an escaped JSON string.
123
+- `tensor_parallel_size` is not a valid key in `speculative_config`. Use
124
+ `draft_tensor_parallel_size` instead.
125
+- Keys such as `temperature` and `top_p` are sampling parameters, not
126
+ `--speculative-config` fields.
127
+- Internal fields such as `target_model_config`, `draft_model_config`,
128
+ `target_parallel_config`, `draft_parallel_config`, and `draft_load_config`
129
+ are populated by vLLM and are not intended to be set by users.
130
+
131
## Lossless guarantees of Speculative Decoding
132
133
In vLLM, speculative decoding aims to enhance inference efficiency while maintaining accuracy. This section addresses the lossless guarantees of
docs/features/speculative_decoding/draft_model.md
+8
-4
@@ -33,9 +33,9 @@ vllm serve Qwen/Qwen3-4B-Thinking-2507 \
33
--port 8000 \
34
--seed 42 \
35
-tp 1 \
36
- --max_model_len 2048 \
37
- --gpu_memory_utilization 0.8 \
38
- --speculative_config '{"model": "Qwen/Qwen3-0.6B", "num_speculative_tokens": 5, "method": "draft_model"}'
36
+ --max-model-len 2048 \
37
+ --gpu-memory-utilization 0.8 \
38
+ --speculative-config '{"model": "Qwen/Qwen3-0.6B", "num_speculative_tokens": 5, "method": "draft_model"}'
39
```
40
41
The code used to request as completions as a client remains unchanged:
@@ -77,4 +77,8 @@ The code used to request as completions as a client remains unchanged:
77
```
78
79
!!! warning
80
- Note: Please use `--speculative_config` to set all configurations related to speculative decoding. The previous method of specifying the model through `--speculative_model` and adding related parameters (e.g., `--num_speculative_tokens`) separately has been deprecated.
80
+ Note: Please use `--speculative-config` to set all configurations related
81
+ to speculative decoding. The previous method of specifying the model
82
+ through `--speculative-model` and adding related parameters such as
83
+ `--num-speculative-tokens` separately has been deprecated. For supported
84
+ keys and examples, see the [`--speculative-config` schema](README.md#--speculative-config-schema).
docs/features/speculative_decoding/mtp.md
+1
-1
@@ -38,7 +38,7 @@ for output in outputs:
38
```bash
39
vllm serve XiaomiMiMo/MiMo-7B-Base \
40
--tensor-parallel-size 1 \
41
- --speculative_config '{"method":"mtp","num_speculative_tokens":1}'
41
+ --speculative-config '{"method":"mtp","num_speculative_tokens":1}'
42
```
43
44
## Notes
docs/features/speculative_decoding/parallel_draft_model.md
+3
-3
@@ -36,9 +36,9 @@ vllm serve Qwen/Qwen3-4B \
36
--port 8000 \
37
--seed 42 \
38
-tp 1 \
39
- --max_model_len 2048 \
40
- --gpu_memory_utilization 0.8 \
41
- --speculative_config '{"model": "amd/PARD-Qwen3-0.6B", "num_speculative_tokens": 12, "method": "draft_model", "parallel_drafting": true}'
39
+ --max-model-len 2048 \
40
+ --gpu-memory-utilization 0.8 \
41
+ --speculative-config '{"model": "amd/PARD-Qwen3-0.6B", "num_speculative_tokens": 12, "method": "draft_model", "parallel_drafting": true}'
42
```
43
44
## Pre-trained PARD weights