@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 89ec06c33b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

[Docs] [Spec decode] Fix docs error in code example (#5427)

Cade Daniel committed Jun 11, 2024 at 10:31 UTC 89ec06c33b9b3e64a6562a82049de44cbf7f9e09
1 file changed +2
docs/source/models/spec_decode.rst
+2
@@ -17,6 +17,7 @@ Speculating with a draft model
17 The following code configures vLLM to use speculative decoding with a draft model, speculating 5 tokens at a time.
18
19 .. code-block:: python
20 +
21 from vllm import LLM, SamplingParams
22
23 prompts = [
@@ -45,6 +46,7 @@ The following code configures vLLM to use speculative decoding where proposals a
46 matching n-grams in the prompt. For more information read `this thread. <https://x.com/joao_gante/status/1747322413006643259>`_
47
48 .. code-block:: python
49 +
50 from vllm import LLM, SamplingParams
51
52 prompts = [