@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 144 lines 5.65 KB
Rendered Raw
1 # Generative Models
2
3 vLLM provides first-class support for generative models, which covers most of LLMs.
4
5 In vLLM, generative models implement the [VllmModelForTextGeneration][vllm.model_executor.models.VllmModelForTextGeneration] interface.
6 Based on the final hidden states of the input, these models output log probabilities of the tokens to generate,
7 which are then passed through [Sampler][vllm.v1.sample.sampler.Sampler] to obtain the final text.
8
9 ## Configuration
10
11 ### Model Runner (`--runner`)
12
13 Run a model in generation mode via the option `--runner generate`.
14
15 !!! tip
16 There is no need to set this option in the vast majority of cases as vLLM can automatically
17 detect the model runner to use via `--runner auto`.
18
19 ## Offline Inference
20
21 The [LLM][vllm.LLM] class provides various methods for offline inference.
22 See [configuration](../api/README.md#configuration) for a list of options when initializing the model.
23
24 ### `LLM.generate`
25
26 The [generate][vllm.LLM.generate] method is available to all generative models in vLLM.
27 It is similar to [its counterpart in HF Transformers](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation#transformers.GenerationMixin.generate),
28 except that tokenization and detokenization are also performed automatically.
29
30 ```python
31 from vllm import LLM
32
33 llm = LLM(model="facebook/opt-125m")
34 outputs = llm.generate("Hello, my name is")
35
36 for output in outputs:
37 prompt = output.prompt
38 generated_text = output.outputs[0].text
39 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
40 ```
41
42 You can optionally control the language generation by passing [SamplingParams][vllm.SamplingParams].
43 For example, you can use greedy sampling by setting `temperature=0`:
44
45 ```python
46 from vllm import LLM, SamplingParams
47
48 llm = LLM(model="facebook/opt-125m")
49 params = SamplingParams(temperature=0)
50 outputs = llm.generate("Hello, my name is", params)
51
52 for output in outputs:
53 prompt = output.prompt
54 generated_text = output.outputs[0].text
55 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
56 ```
57
58 !!! important
59 By default, vLLM will use sampling parameters recommended by model creator by applying the `generation_config.json` from the huggingface model repository if it exists. In most cases, this will provide you with the best results by default if [SamplingParams][vllm.SamplingParams] is not specified.
60
61 However, if vLLM's default sampling parameters are preferred, please pass `generation_config="vllm"` when creating the [LLM][vllm.LLM] instance.
62 A code example can be found here: [examples/basic/offline_inference/basic.py](../../examples/basic/offline_inference/basic.py)
63
64 ### `LLM.beam_search`
65
66 The [beam_search][vllm.LLM.beam_search] method implements [beam search](https://huggingface.co/docs/transformers/en/generation_strategies#beam-search) on top of [generate][vllm.LLM.generate].
67 For example, to search using 5 beams and output at most 50 tokens:
68
69 ```python
70 from vllm import LLM
71 from vllm.sampling_params import BeamSearchParams
72
73 llm = LLM(model="facebook/opt-125m")
74 params = BeamSearchParams(beam_width=5, max_tokens=50)
75 outputs = llm.beam_search([{"prompt": "Hello, my name is "}], params)
76
77 for output in outputs:
78 generated_text = output.sequences[0].text
79 print(f"Generated text: {generated_text!r}")
80 ```
81
82 ### `LLM.chat`
83
84 The [chat][vllm.LLM.chat] method implements chat functionality on top of [generate][vllm.LLM.generate].
85 In particular, it accepts input similar to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat)
86 and automatically applies the model's [chat template](https://huggingface.co/docs/transformers/en/chat_templating) to format the prompt.
87
88 !!! important
89 In general, only instruction-tuned models have a chat template.
90 Base models may perform poorly as they are not trained to respond to the chat conversation.
91
92 ??? code
93
94 ```python
95 from vllm import LLM
96
97 llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
98 conversation = [
99 {
100 "role": "system",
101 "content": "You are a helpful assistant",
102 },
103 {
104 "role": "user",
105 "content": "Hello",
106 },
107 {
108 "role": "assistant",
109 "content": "Hello! How can I assist you today?",
110 },
111 {
112 "role": "user",
113 "content": "Write an essay about the importance of higher education.",
114 },
115 ]
116 outputs = llm.chat(conversation)
117
118 for output in outputs:
119 prompt = output.prompt
120 generated_text = output.outputs[0].text
121 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
122 ```
123
124 A code example can be found here: [examples/basic/offline_inference/chat.py](../../examples/basic/offline_inference/chat.py)
125
126 If the model doesn't have a chat template or you want to specify another one,
127 you can explicitly pass a chat template:
128
129 ```python
130 from vllm.entrypoints.chat_utils import load_chat_template
131
132 # You can find a list of existing chat templates under `examples/`
133 custom_template = load_chat_template(chat_template="<path_to_template>")
134 print("Loaded chat template:", custom_template)
135
136 outputs = llm.chat(conversation, chat_template=custom_template)
137 ```
138
139 ## Online Serving
140
141 Our [OpenAI-Compatible Server](../serving/online_serving/openai_compatible_server.md) provides endpoints that correspond to the offline APIs:
142
143 - [Completions API](../serving/online_serving/openai_compatible_server.md#completions-api) is similar to `LLM.generate` but only accepts text.
144 - [Chat API](../serving/online_serving/openai_compatible_server.md#chat-api) is similar to `LLM.chat`, accepting both text and [multi-modal inputs](../features/multimodal_inputs.md) for models with a chat template.