A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 4, 00:00 UTC
docs: clarify Gemma 4 assistant speculative decoding (#42180)
Signed-off-by: AbhiOnGithub <abhiOnGithub@users.noreply.github.com> Co-authored-by: AbhiOnGithub <abhiOnGithub@users.noreply.github.com>
Abhishek Gupta committed
May 9, 2026 at 20:08 UTC
27d3bac272597bf1cbbd070ae0b81c0f68e13788
3 files changed
+39
docs/features/speculative_decoding/README.md
+11
@@ -72,6 +72,17 @@ only apply to model-based methods such as `draft_model`, `mtp`, `eagle3`, and
72
| `rejection_sample_method` | `string` | `strict` | `strict`, `probabilistic`, or `synthetic`. |
73
| `synthetic_acceptance_rate` | `float` | `None` | Average acceptance rate to target when `rejection_sample_method` is `synthetic`. Valid range is `[0, 1]`. |
74
75
+!!! note
76
+ Gemma 4 assistant checkpoints are handled as Gemma 4 MTP speculators, not
77
+ as generic draft models. Use `"method": "mtp"` with the assistant
78
+ checkpoint in `model`, as shown in the [MTP guide](mtp.md#gemma-4-assistant-models).
79
+
80
+ If startup logs show `SpeculativeConfig(method='draft_model', ...)` for a
81
+ Gemma 4 assistant checkpoint, the installed vLLM version does not include
82
+ Gemma 4 MTP support for that path. Upgrade to a version that includes
83
+ Gemma 4 MTP support instead of forcing the assistant checkpoint through
84
+ generic draft-model speculative decoding.
85
+
86
### Method-specific keys
87
88
#### N-gram
docs/features/speculative_decoding/mtp.md
+25
@@ -9,6 +9,31 @@ MTP is useful when:
9
- Your model natively supports MTP.
10
- You want model-based speculative decoding with minimal extra configuration.
11
12
+## Gemma 4 Assistant Models
13
+
14
+Gemma 4 assistant checkpoints use vLLM's Gemma 4 MTP path. They are not generic
15
+draft models, even though they are passed through the `model` field in
16
+`--speculative-config`.
17
+
18
+Use `"method": "mtp"` when serving Gemma 4 with an assistant checkpoint:
19
+
20
+```bash
21
+vllm serve google/gemma-4-E2B-it \
22
+ --tensor-parallel-size 1 \
23
+ --max-model-len 8192 \
24
+ --speculative-config '{"method":"mtp","model":"gg-hf-am/gemma-4-E2B-it-assistant","num_speculative_tokens":1}'
25
+```
26
+
27
+The E2B, E4B, 26B-A4B, and 31B Gemma 4 IT assistant checkpoints are supported
28
+when their configuration uses `model_type: gemma4_assistant`. vLLM maps those
29
+checkpoints to `Gemma4MTPModel` internally and wires the assistant layers to
30
+share KV cache with the target model.
31
+
32
+If an older vLLM release logs `SpeculativeConfig(method='draft_model', ...)`
33
+for a Gemma 4 assistant checkpoint, that release is treating the assistant as a
34
+generic draft model and may fail during initialization for multimodal Gemma 4
35
+targets. Upgrade to a version with Gemma 4 MTP support instead.
36
+
37
## Offline Example
38
39
```python
docs/models/supported_models.md
+3
@@ -659,6 +659,9 @@ Some models are supported only via the [Transformers modeling backend](#transfor
659
For `Gemma4ForConditionalGeneration`:
660
- audio input is only supported by the `gemma-4-E2B` and `gemma-4-E4B` variants.
661
- The model does not ingest videos directly. However, vLLM’s Gemma 4 implementation supports video inputs by handling video processing internally. Users can send videos directly in the message structure to vLLM, where they are converted into text and image frames before being passed to the model.
662
+ - Gemma 4 assistant checkpoints for speculative decoding use vLLM's Gemma
663
+ 4 MTP path, not generic draft-model speculative decoding. See the
664
+ [Gemma 4 assistant model MTP example](../features/speculative_decoding/mtp.md#gemma-4-assistant-models).
665
666
!!! note
667
For `InternVLChatModel`, only InternVL2.5 with Qwen2.5 text backbone (`OpenGVLab/InternVL2.5-1B` etc.), InternVL3 and InternVL3.5 have video inputs support currently.