A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 22:35 UTC
[Doc] Added warning of speculating with draft model (#22047)
Signed-off-by: Dilute-l <dilu2333@163.com> Co-authored-by: Dilute-l <dilu2333@163.com>
WeiQing Chen committed
Aug 1, 2025 at 17:11 UTC
49314869887e169be080201ab8bcda14e745c080
1 file changed
+4
docs/features/spec_decode.md
+4
@@ -15,6 +15,10 @@ Speculative decoding is a technique which improves inter-token latency in memory
15
16
The following code configures vLLM in an offline mode to use speculative decoding with a draft model, speculating 5 tokens at a time.
17
18
+!!! warning
19
+ In vllm v0.10.0, speculative decoding with a draft model is not supported.
20
+ If you use the following code, you will get a `NotImplementedError`.
21
+
22
??? code
23
24
```python