@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 973f5dc581
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 03:15 UTC

[Doc]Add documentation for using EAGLE in vLLM (#11417)

Signed-off-by: Sourashis Roy <sroy@roblox.com>

sroy745 committed Jan 7, 2025 at 11:19 UTC 973f5dc581c35a9c5b9176116e2f42f3f97d0d01
1 file changed +66
docs/source/features/spec_decode.md
+66
@@ -159,6 +159,72 @@ A variety of speculative models of this type are available on HF hub:
159 - [granite-7b-instruct-accelerator](https://huggingface.co/ibm-granite/granite-7b-instruct-accelerator)
160 - [granite-20b-code-instruct-accelerator](https://huggingface.co/ibm-granite/granite-20b-code-instruct-accelerator)
161
162 +## Speculating using EAGLE based draft models
163 +
164 +The following code configures vLLM to use speculative decoding where proposals are generated by
165 +an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https://arxiv.org/pdf/2401.15077) based draft model.
166 +
167 +```python
168 +from vllm import LLM, SamplingParams
169 +
170 +prompts = [
171 + "The future of AI is",
172 +]
173 +sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
174 +
175 +llm = LLM(
176 + model="meta-llama/Meta-Llama-3-8B-Instruct",
177 + tensor_parallel_size=4,
178 + speculative_model="path/to/modified/eagle/model",
179 + speculative_draft_tensor_parallel_size=1,
180 +)
181 +
182 +outputs = llm.generate(prompts, sampling_params)
183 +
184 +for output in outputs:
185 + prompt = output.prompt
186 + generated_text = output.outputs[0].text
187 + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
188 +
189 +```
190 +
191 +A few important things to consider when using the EAGLE based draft models:
192 +
193 +1. The EAGLE draft models available in the [HF repository for EAGLE models](https://huggingface.co/yuhuili) cannot be
194 + used directly with vLLM due to differences in the expected layer names and model definition.
195 + To use these models with vLLM, use the [following script](https://gist.github.com/abhigoyal1997/1e7a4109ccb7704fbc67f625e86b2d6d)
196 + to convert them. Note that this script does not modify the model's weights.
197 +
198 + In the above example, use the script to first convert
199 + the [yuhuili/EAGLE-LLaMA3-Instruct-8B](https://huggingface.co/yuhuili/EAGLE-LLaMA3-Instruct-8B) model
200 + and then use the converted checkpoint as the draft model in vLLM.
201 +
202 +2. The EAGLE based draft models need to be run without tensor parallelism
203 + (i.e. speculative_draft_tensor_parallel_size is set to 1), although
204 + it is possible to run the main model using tensor parallelism (see example above).
205 +
206 +3. When using EAGLE-based speculators with vLLM, the observed speedup is lower than what is
207 + reported in the reference implementation [here](https://github.com/SafeAILab/EAGLE). This issue is under
208 + investigation and tracked here: [https://github.com/vllm-project/vllm/issues/9565](https://github.com/vllm-project/vllm/issues/9565).
209 +
210 +
211 +A variety of EAGLE draft models are available on the Hugging Face hub:
212 +
213 +| Base Model | EAGLE on Hugging Face | # EAGLE Parameters |
214 +|---------------------------------------------------------------------|-------------------------------------------|--------------------|
215 +| Vicuna-7B-v1.3 | yuhuili/EAGLE-Vicuna-7B-v1.3 | 0.24B |
216 +| Vicuna-13B-v1.3 | yuhuili/EAGLE-Vicuna-13B-v1.3 | 0.37B |
217 +| Vicuna-33B-v1.3 | yuhuili/EAGLE-Vicuna-33B-v1.3 | 0.56B |
218 +| LLaMA2-Chat 7B | yuhuili/EAGLE-llama2-chat-7B | 0.24B |
219 +| LLaMA2-Chat 13B | yuhuili/EAGLE-llama2-chat-13B | 0.37B |
220 +| LLaMA2-Chat 70B | yuhuili/EAGLE-llama2-chat-70B | 0.99B |
221 +| Mixtral-8x7B-Instruct-v0.1 | yuhuili/EAGLE-mixtral-instruct-8x7B | 0.28B |
222 +| LLaMA3-Instruct 8B | yuhuili/EAGLE-LLaMA3-Instruct-8B | 0.25B |
223 +| LLaMA3-Instruct 70B | yuhuili/EAGLE-LLaMA3-Instruct-70B | 0.99B |
224 +| Qwen2-7B-Instruct | yuhuili/EAGLE-Qwen2-7B-Instruct | 0.26B |
225 +| Qwen2-72B-Instruct | yuhuili/EAGLE-Qwen2-72B-Instruct | 1.05B |
226 +
227 +
228 ## Lossless guarantees of Speculative Decoding
229
230 In vLLM, speculative decoding aims to enhance inference efficiency while maintaining accuracy. This section addresses the lossless guarantees of