@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 0e12cd67a8
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 19:30 UTC

[Doc] add online speculative decoding example (#7243)

Stas Bekman committed Aug 7, 2024 at 09:58 UTC 0e12cd67a8f84047e6d28084bfe94a8278e10218
1 file changed +55 -11
docs/source/models/spec_decode.rst
+55 -11
@@ -14,17 +14,17 @@ Speculative decoding is a technique which improves inter-token latency in memory
14 Speculating with a draft model
15 ------------------------------
16
17 -The following code configures vLLM to use speculative decoding with a draft model, speculating 5 tokens at a time.
17 +The following code configures vLLM in an offline mode to use speculative decoding with a draft model, speculating 5 tokens at a time.
18
19 .. code-block:: python
20
21 from vllm import LLM, SamplingParams
22 -
22 +
23 prompts = [
24 "The future of AI is",
25 ]
26 sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
27 -
27 +
28 llm = LLM(
29 model="facebook/opt-6.7b",
30 tensor_parallel_size=1,
@@ -33,12 +33,56 @@ The following code configures vLLM to use speculative decoding with a draft mode
33 use_v2_block_manager=True,
34 )
35 outputs = llm.generate(prompts, sampling_params)
36 -
36 +
37 for output in outputs:
38 prompt = output.prompt
39 generated_text = output.outputs[0].text
40 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
41
42 +To perform the same with an online mode launch the server:
43 +
44 +.. code-block:: bash
45 +
46 + python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 8000 --model facebook/opt-6.7b \
47 + --seed 42 -tp 1 --speculative_model facebook/opt-125m --use-v2-block-manager \
48 + --num_speculative_tokens 5 --gpu_memory_utilization 0.8
49 +
50 + Then use a client:
51 +
52 +.. code-block:: python
53 +
54 + from openai import OpenAI
55 +
56 + # Modify OpenAI's API key and API base to use vLLM's API server.
57 + openai_api_key = "EMPTY"
58 + openai_api_base = "http://localhost:8000/v1"
59 +
60 + client = OpenAI(
61 + # defaults to os.environ.get("OPENAI_API_KEY")
62 + api_key=openai_api_key,
63 + base_url=openai_api_base,
64 + )
65 +
66 + models = client.models.list()
67 + model = models.data[0].id
68 +
69 + # Completion API
70 + stream = False
71 + completion = client.completions.create(
72 + model=model,
73 + prompt="The future of AI is",
74 + echo=False,
75 + n=1,
76 + stream=stream,
77 + )
78 +
79 + print("Completion results:")
80 + if stream:
81 + for c in completion:
82 + print(c)
83 + else:
84 + print(completion)
85 +
86 Speculating by matching n-grams in the prompt
87 ---------------------------------------------
88
@@ -48,12 +92,12 @@ matching n-grams in the prompt. For more information read `this thread. <https:/
92 .. code-block:: python
93
94 from vllm import LLM, SamplingParams
51 -
95 +
96 prompts = [
97 "The future of AI is",
98 ]
99 sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
56 -
100 +
101 llm = LLM(
102 model="facebook/opt-6.7b",
103 tensor_parallel_size=1,
@@ -63,7 +107,7 @@ matching n-grams in the prompt. For more information read `this thread. <https:/
107 use_v2_block_manager=True,
108 )
109 outputs = llm.generate(prompts, sampling_params)
66 -
110 +
111 for output in outputs:
112 prompt = output.prompt
113 generated_text = output.outputs[0].text
@@ -74,7 +118,7 @@ Speculating using MLP speculators
118
119 The following code configures vLLM to use speculative decoding where proposals are generated by
120 draft models that conditioning draft predictions on both context vectors and sampled tokens.
77 -For more information see `this blog <https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/>`_ or
121 +For more information see `this blog <https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/>`_ or
122 `this technical report <https://arxiv.org/abs/2404.19124>`_.
123
124 .. code-block:: python
@@ -100,9 +144,9 @@ For more information see `this blog <https://pytorch.org/blog/hitchhikers-guide-
144 generated_text = output.outputs[0].text
145 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
146
103 -Note that these speculative models currently need to be run without tensor parallelism, although
104 -it is possible to run the main model using tensor parallelism (see example above). Since the
105 -speculative models are relatively small, we still see significant speedups. However, this
147 +Note that these speculative models currently need to be run without tensor parallelism, although
148 +it is possible to run the main model using tensor parallelism (see example above). Since the
149 +speculative models are relatively small, we still see significant speedups. However, this
150 limitation will be fixed in a future release.
151
152 A variety of speculative models of this type are available on HF hub: