A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 19:30 UTC
[Doc] add online speculative decoding example (#7243)
Stas Bekman committed
Aug 7, 2024 at 09:58 UTC
0e12cd67a8f84047e6d28084bfe94a8278e10218
1 file changed
+55
-11
docs/source/models/spec_decode.rst
+55
-11
@@ -14,17 +14,17 @@ Speculative decoding is a technique which improves inter-token latency in memory
14
Speculating with a draft model
15
------------------------------
16
17
-The following code configures vLLM to use speculative decoding with a draft model, speculating 5 tokens at a time.
17
+The following code configures vLLM in an offline mode to use speculative decoding with a draft model, speculating 5 tokens at a time.
18
19
.. code-block:: python
20
21
from vllm import LLM, SamplingParams
22
-
22
+
23
prompts = [
24
"The future of AI is",
25
]
26
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
27
-
27
+
28
llm = LLM(
29
model="facebook/opt-6.7b",
30
tensor_parallel_size=1,
@@ -33,12 +33,56 @@ The following code configures vLLM to use speculative decoding with a draft mode
33
use_v2_block_manager=True,
34
)
35
outputs = llm.generate(prompts, sampling_params)
36
-
36
+
37
for output in outputs:
38
prompt = output.prompt
39
generated_text = output.outputs[0].text
40
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
41
42
+To perform the same with an online mode launch the server:
43
+
44
+.. code-block:: bash
45
+
46
+ python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 8000 --model facebook/opt-6.7b \
47
+ --seed 42 -tp 1 --speculative_model facebook/opt-125m --use-v2-block-manager \
48
+ --num_speculative_tokens 5 --gpu_memory_utilization 0.8
49
+
50
+ Then use a client:
51
+
52
+.. code-block:: python
53
+
54
+ from openai import OpenAI
55
+
56
+ # Modify OpenAI's API key and API base to use vLLM's API server.
57
+ openai_api_key = "EMPTY"
58
+ openai_api_base = "http://localhost:8000/v1"
59
+
60
+ client = OpenAI(
61
+ # defaults to os.environ.get("OPENAI_API_KEY")
62
+ api_key=openai_api_key,
63
+ base_url=openai_api_base,
64
+ )
65
+
66
+ models = client.models.list()
67
+ model = models.data[0].id
68
+
69
+ # Completion API
70
+ stream = False
71
+ completion = client.completions.create(
72
+ model=model,
73
+ prompt="The future of AI is",
74
+ echo=False,
75
+ n=1,
76
+ stream=stream,
77
+ )
78
+
79
+ print("Completion results:")
80
+ if stream:
81
+ for c in completion:
82
+ print(c)
83
+ else:
84
+ print(completion)
85
+
86
Speculating by matching n-grams in the prompt
87
---------------------------------------------
88
@@ -48,12 +92,12 @@ matching n-grams in the prompt. For more information read `this thread. <https:/
92
.. code-block:: python
93
94
from vllm import LLM, SamplingParams
51
-
95
+
96
prompts = [
97
"The future of AI is",
98
]
99
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
56
-
100
+
101
llm = LLM(
102
model="facebook/opt-6.7b",
103
tensor_parallel_size=1,
@@ -63,7 +107,7 @@ matching n-grams in the prompt. For more information read `this thread. <https:/
107
use_v2_block_manager=True,
108
)
109
outputs = llm.generate(prompts, sampling_params)
66
-
110
+
111
for output in outputs:
112
prompt = output.prompt
113
generated_text = output.outputs[0].text
@@ -74,7 +118,7 @@ Speculating using MLP speculators
118
119
The following code configures vLLM to use speculative decoding where proposals are generated by
120
draft models that conditioning draft predictions on both context vectors and sampled tokens.
77
-For more information see `this blog <https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/>`_ or
121
+For more information see `this blog <https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/>`_ or
122
`this technical report <https://arxiv.org/abs/2404.19124>`_.
123
124
.. code-block:: python
@@ -100,9 +144,9 @@ For more information see `this blog <https://pytorch.org/blog/hitchhikers-guide-
144
generated_text = output.outputs[0].text
145
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
146
103
-Note that these speculative models currently need to be run without tensor parallelism, although
104
-it is possible to run the main model using tensor parallelism (see example above). Since the
105
-speculative models are relatively small, we still see significant speedups. However, this
147
+Note that these speculative models currently need to be run without tensor parallelism, although
148
+it is possible to run the main model using tensor parallelism (see example above). Since the
149
+speculative models are relatively small, we still see significant speedups. However, this
150
limitation will be fixed in a future release.
151
152
A variety of speculative models of this type are available on HF hub: