A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
[Docs] [Spec decode] Fix docs error in code example (#5427)
Cade Daniel committed
Jun 11, 2024 at 10:31 UTC
89ec06c33b9b3e64a6562a82049de44cbf7f9e09
1 file changed
+2
docs/source/models/spec_decode.rst
+2
@@ -17,6 +17,7 @@ Speculating with a draft model
17
The following code configures vLLM to use speculative decoding with a draft model, speculating 5 tokens at a time.
18
19
.. code-block:: python
20
+
21
from vllm import LLM, SamplingParams
22
23
prompts = [
@@ -45,6 +46,7 @@ The following code configures vLLM to use speculative decoding where proposals a
46
matching n-grams in the prompt. For more information read `this thread. <https://x.com/joao_gante/status/1747322413006643259>`_
47
48
.. code-block:: python
49
+
50
from vllm import LLM, SamplingParams
51
52
prompts = [