A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
[Speculative decoding] Initial spec decode docs (#5400)
Cade Daniel committed
Jun 11, 2024 at 10:15 UTC
4c2ffb28ffe7270b49ac7cf5324978950a28e7e1
2 files changed
+76
docs/source/index.rst
+1
@@ -90,6 +90,7 @@ Documentation
90
models/engine_args
91
models/lora
92
models/vlm
93
+ models/spec_decode
94
models/performance
95
96
.. toctree::
docs/source/models/spec_decode.rst
new
+75
@@ -0,0 +1,75 @@
1
+.. _spec_decode:
2
+
3
+Speculative decoding in vLLM
4
+============================
5
+
6
+.. warning::
7
+ Please note that speculative decoding in vLLM is not yet optimized and does
8
+ not usually yield inter-token latency reductions for all prompt datasets or sampling parameters. The work
9
+ to optimize it is ongoing and can be followed in `this issue. <https://github.com/vllm-project/vllm/issues/4630>`_
10
+
11
+This document shows how to use `Speculative Decoding <https://x.com/karpathy/status/1697318534555336961>`_ with vLLM.
12
+Speculative decoding is a technique which improves inter-token latency in memory-bound LLM inference.
13
+
14
+Speculating with a draft model
15
+------------------------------
16
+
17
+The following code configures vLLM to use speculative decoding with a draft model, speculating 5 tokens at a time.
18
+
19
+.. code-block:: python
20
+ from vllm import LLM, SamplingParams
21
+
22
+ prompts = [
23
+ "The future of AI is",
24
+ ]
25
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
26
+
27
+ llm = LLM(
28
+ model="facebook/opt-6.7b",
29
+ tensor_parallel_size=1,
30
+ speculative_model="facebook/opt-125m",
31
+ num_speculative_tokens=5,
32
+ use_v2_block_manager=True,
33
+ )
34
+ outputs = llm.generate(prompts, sampling_params)
35
+
36
+ for output in outputs:
37
+ prompt = output.prompt
38
+ generated_text = output.outputs[0].text
39
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
40
+
41
+Speculating by matching n-grams in the prompt
42
+---------------------------------------------
43
+
44
+The following code configures vLLM to use speculative decoding where proposals are generated by
45
+matching n-grams in the prompt. For more information read `this thread. <https://x.com/joao_gante/status/1747322413006643259>`_
46
+
47
+.. code-block:: python
48
+ from vllm import LLM, SamplingParams
49
+
50
+ prompts = [
51
+ "The future of AI is",
52
+ ]
53
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
54
+
55
+ llm = LLM(
56
+ model="facebook/opt-6.7b",
57
+ tensor_parallel_size=1,
58
+ speculative_model="[ngram]",
59
+ num_speculative_tokens=5,
60
+ ngram_prompt_lookup_max=4,
61
+ use_v2_block_manager=True,
62
+ )
63
+ outputs = llm.generate(prompts, sampling_params)
64
+
65
+ for output in outputs:
66
+ prompt = output.prompt
67
+ generated_text = output.outputs[0].text
68
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
69
+
70
+Resources for vLLM contributors
71
+-------------------------------
72
+* `A Hacker's Guide to Speculative Decoding in vLLM <https://www.youtube.com/watch?v=9wNAgpX6z_4>`_
73
+* `What is Lookahead Scheduling in vLLM? <https://docs.google.com/document/d/1Z9TvqzzBPnh5WHcRwjvK2UEeFeq5zMZb5mFE8jR0HCs/edit#heading=h.1fjfb0donq5a>`_
74
+* `Information on batch expansion. <https://docs.google.com/document/d/1T-JaS2T1NRfdP51qzqpyakoCXxSXTtORppiwaj5asxA/edit#heading=h.kk7dq05lc6q8>`_
75
+* `Dynamic speculative decoding <https://github.com/vllm-project/vllm/issues/4565>`_