A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Doc] Added unmentioned required option "method" in the usage of EAGLE-3 based models (#21737)
Signed-off-by: Dilute-l <dilu2333@163.com> Co-authored-by: Dilute-l <dilu2333@163.com>
Hongsheng Liu committed
Aug 12, 2025 at 15:14 UTC
3a7e3bbdd255b470d37727a31cc0471aa0fe6ecb
1 file changed
+4
docs/features/spec_decode.md
+4
@@ -203,6 +203,7 @@ an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https
203
"model": "yuhuili/EAGLE-LLaMA3-Instruct-8B",
204
"draft_tensor_parallel_size": 1,
205
"num_speculative_tokens": 2,
206
+ "method": "eagle",
207
},
208
)
209
@@ -231,6 +232,9 @@ A few important things to consider when using the EAGLE based draft models:
232
reported in the reference implementation [here](https://github.com/SafeAILab/EAGLE). This issue is under
233
investigation and tracked here: <gh-issue:9565>.
234
235
+4. When using EAGLE-3 based draft model, option "method" must be set to "eagle3".
236
+ That is, to specify `"method": "eagle3"` in `speculative_config`.
237
+
238
A variety of EAGLE draft models are available on the Hugging Face hub:
239
240
| Base Model | EAGLE on Hugging Face | # EAGLE Parameters |