@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 333 lines 15.3 KB
Rendered Raw
1 # Quickstart
2
3 This guide will help you quickly get started with vLLM to perform:
4
5 - [Offline batched inference](#offline-batched-inference)
6 - [Online serving](#online-serving)
7
8 ## Prerequisites
9
10 - OS: Linux
11 - Python: 3.10 -- 3.13
12
13 !!! note
14 vLLM also works on macOS with [vLLM-Metal](https://github.com/vllm-project/vllm-metal) for Apple Silicon GPU acceleration. See the [GPU installation guide](installation/gpu.md) and select the "Apple Silicon" tab.
15
16 ## Installation
17
18 === "NVIDIA CUDA"
19
20 If you are using NVIDIA GPUs, you can install vLLM using [pip](https://pypi.org/project/vllm/) directly.
21
22 It's recommended to use [uv](https://docs.astral.sh/uv/), a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment and install vLLM using the following commands:
23
24 ```bash
25 uv venv --python 3.12 --seed
26 source .venv/bin/activate
27 uv pip install vllm --torch-backend=auto
28 ```
29
30 `uv` can [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu126`), set `--torch-backend=cu126` (or `UV_TORCH_BACKEND=cu126`).
31
32 Another delightful way is to use `uv run` with `--with [dependency]` option, which allows you to run commands such as `vllm serve` without creating any permanent environment:
33
34 ```bash
35 uv run --with vllm vllm --help
36 ```
37
38 You can also use [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) to create and manage Python environments. You can install `uv` to the conda environment through `pip` if you want to manage it within the environment.
39
40 ```bash
41 conda create -n myenv python=3.12 -y
42 conda activate myenv
43 pip install --upgrade uv
44 uv pip install vllm --torch-backend=auto
45 ```
46
47 === "AMD ROCm"
48
49 If you are using AMD GPUs, you can install vLLM using `uv`.
50
51 It's recommended to use [uv](https://docs.astral.sh/uv/), as it gives the extra index [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes). `uv` is also a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment and install vLLM using the following commands:
52
53 ```bash
54 uv venv --python 3.12 --seed
55 source .venv/bin/activate
56 uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
57 ```
58
59 !!! note
60 It currently supports Python 3.12, ROCm 7.0 and `glibc >= 2.35`.
61
62 !!! note
63 Note that, previously, docker images were published using AMD's docker release pipeline and were located `rocm/vllm-dev`. This is being deprecated by using vLLM's docker release pipeline.
64
65 !!! tip
66 A nightly Docker image is also available as [vllm/vllm-openai-rocm:nightly](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags) for testing the latest development builds.
67
68 === "Intel GPU"
69
70 vLLM supports Intel GPUs through the XPU backend. Pre-built XPU wheels will be available soon.
71
72 Official Docker images for Intel GPUs are added to the vLLM release starting from v0.26.0. Nightly Docker image is also available as [vllm/vllm-openai-xpu:nightly](https://hub.docker.com/r/vllm/vllm-openai-xpu/tags).
73
74 !!! tip
75 For more detailed instructions, including building from source and Docker image setup, please refer to the [GPU installation guide](installation/gpu.md) and select the "Intel XPU" tab.
76
77 === "Google TPU"
78
79 To run vLLM on Google TPUs, you need to install the `vllm-tpu` package.
80
81 ```bash
82 uv pip install vllm-tpu
83 ```
84
85 !!! note
86 For more detailed instructions, including Docker, installing from source, and troubleshooting, please refer to the [vLLM on TPU documentation](https://docs.vllm.ai/projects/tpu/en/latest/).
87
88 === "Ascend NPU"
89
90 If you are using Ascend NPUs, you can run vLLM through [vLLM Ascend](https://github.com/vllm-project/vllm-ascend), a community-maintained hardware plugin.
91
92 Follow the installation instructions in the [vLLM Ascend quick start](https://docs.vllm.ai/projects/ascend/en/latest/quick_start.html).
93
94 !!! note
95 Ascend setup depends on your NPU hardware and CANN version. For supported versions, Docker images, and troubleshooting, please refer to the [vLLM Ascend documentation](https://docs.vllm.ai/projects/ascend/en/latest/).
96
97 === "Apple Silicon (Mac)"
98
99 If you are using Apple Silicon Macs, you can use vLLM-Metal for GPU-accelerated inference via Apple's Metal framework.
100
101 Follow the installation instructions in the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal#installation).
102
103 !!! note
104 vLLM-Metal uses MLX instead of PyTorch as the compute backend and requires MLX-optimized models from the [mlx-community](https://huggingface.co/mlx-community) on Hugging Face.
105
106 !!! tip
107 For more detailed instructions, please refer to the [GPU installation guide](installation/gpu.md) and select the "Apple Silicon" tab.
108
109 !!! note
110 For more detail and non-CUDA platforms, please refer to the [installation guide](installation/README.md) for specific instructions on how to install vLLM.
111
112 ## Offline Batched Inference
113
114 With vLLM installed, you can start generating texts for list of input prompts (i.e. offline batch inferencing). See the example script: [examples/basic/offline_inference/basic.py](../../examples/basic/offline_inference/basic.py)
115
116 The first line of this example imports the classes [LLM][vllm.LLM] and [SamplingParams][vllm.SamplingParams]:
117
118 - [LLM][vllm.LLM] is the main class for running offline inference with vLLM engine.
119 - [SamplingParams][vllm.SamplingParams] specifies the parameters for the sampling process.
120
121 ```python
122 from vllm import LLM, SamplingParams
123 ```
124
125 The next section defines a list of input prompts and sampling parameters for text generation. The [sampling temperature](https://arxiv.org/html/2402.05201v1) is set to `0.8` and the [nucleus sampling probability](https://en.wikipedia.org/wiki/Top-p_sampling) is set to `0.95`. You can find more information about the sampling parameters [here](../api/README.md#inference-parameters).
126
127 !!! important
128 By default, vLLM will use sampling parameters recommended by model creator by applying the `generation_config.json` from the Hugging Face model repository if it exists. In most cases, this will provide you with the best results by default if [SamplingParams][vllm.SamplingParams] is not specified.
129
130 However, if vLLM's default sampling parameters are preferred, please set `generation_config="vllm"` when creating the [LLM][vllm.LLM] instance.
131
132 ```python
133 prompts = [
134 "Hello, my name is",
135 "The president of the United States is",
136 "The capital of France is",
137 "The future of AI is",
138 ]
139 sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
140 ```
141
142 The [LLM][vllm.LLM] class initializes vLLM's engine and the [OPT-125M model](https://arxiv.org/abs/2205.01068) for offline inference. The list of supported models can be found [here](../models/supported_models.md).
143
144 ```python
145 llm = LLM(model="facebook/opt-125m")
146 ```
147
148 !!! note
149 By default, vLLM downloads models from [Hugging Face](https://huggingface.co/). If you would like to use models from [ModelScope](https://www.modelscope.cn), set the environment variable `VLLM_USE_MODELSCOPE` before initializing the engine.
150
151 ```shell
152 export VLLM_USE_MODELSCOPE=True
153 ```
154
155 Now, the fun part! The outputs are generated using `llm.generate`. It adds the input prompts to the vLLM engine's waiting queue and executes the vLLM engine to generate the outputs with high throughput. The outputs are returned as a list of `RequestOutput` objects, which include all of the output tokens.
156
157 ```python
158 outputs = llm.generate(prompts, sampling_params)
159
160 for output in outputs:
161 prompt = output.prompt
162 generated_text = output.outputs[0].text
163 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
164 ```
165
166 !!! note
167 The `llm.generate` method does not automatically apply the model's chat template to the input prompt. Therefore, if you are using an Instruct model or Chat model, you should manually apply the corresponding chat template to ensure the expected behavior. Alternatively, you can use the `llm.chat` method and pass a list of messages which have the same format as those passed to OpenAI's `client.chat.completions`:
168
169 ??? code
170
171 ```python
172 # Using tokenizer to apply chat template
173 from transformers import AutoTokenizer
174
175 tokenizer = AutoTokenizer.from_pretrained("/path/to/chat_model")
176 messages_list = [
177 [{"role": "user", "content": prompt}]
178 for prompt in prompts
179 ]
180 texts = tokenizer.apply_chat_template(
181 messages_list,
182 tokenize=False,
183 add_generation_prompt=True,
184 )
185
186 # Generate outputs
187 outputs = llm.generate(texts, sampling_params)
188
189 # Print the outputs.
190 for output in outputs:
191 prompt = output.prompt
192 generated_text = output.outputs[0].text
193 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
194
195 # Using chat interface.
196 outputs = llm.chat(messages_list, sampling_params)
197 for idx, output in enumerate(outputs):
198 prompt = prompts[idx]
199 generated_text = output.outputs[0].text
200 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
201 ```
202
203 ## Online Serving
204
205 vLLM can be deployed as a server that implements the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API.
206 By default, it starts the server at `http://localhost:8000`. You can specify the address with `--host` and `--port` arguments. The server currently hosts one model at a time and implements endpoints such as [list models](https://platform.openai.com/docs/api-reference/models/list), [create chat completion](https://platform.openai.com/docs/api-reference/chat/completions/create), and [create completion](https://platform.openai.com/docs/api-reference/completions/create) endpoints.
207
208 Run the following command to start the vLLM server with the [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) model:
209
210 ```bash
211 vllm serve Qwen/Qwen2.5-1.5B-Instruct
212 ```
213
214 !!! note
215 By default, the server uses a predefined chat template stored in the tokenizer.
216 You can learn about overriding it [here](../serving/online_serving/README.md#chat-template).
217 !!! important
218 By default, the server applies `generation_config.json` from the huggingface model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator.
219
220 To disable this behavior, please pass `--generation-config vllm` when launching the server.
221
222 This server can be queried in the same format as OpenAI API. For example, to list the models:
223
224 ```bash
225 curl http://localhost:8000/v1/models
226 ```
227
228 You can pass in the argument `--api-key` or environment variable `VLLM_API_KEY` to enable the server to check for API key in the header.
229 You can pass multiple keys after `--api-key`, and the server will accept any of the keys passed, this can be useful for key rotation.
230
231 ### OpenAI Completions API with vLLM
232
233 Once your server is started, you can query the model with input prompts:
234
235 ```bash
236 curl http://localhost:8000/v1/completions \
237 -H "Content-Type: application/json" \
238 -d '{
239 "model": "Qwen/Qwen2.5-1.5B-Instruct",
240 "prompt": "San Francisco is a",
241 "max_tokens": 7,
242 "temperature": 0
243 }'
244 ```
245
246 Since this server is compatible with OpenAI API, you can use it as a drop-in replacement for any applications using OpenAI API. For example, another way to query the server is via the `openai` Python package:
247
248 ??? code
249
250 ```python
251 from openai import OpenAI
252
253 # Modify OpenAI's API key and API base to use vLLM's API server.
254 openai_api_key = "EMPTY"
255 openai_api_base = "http://localhost:8000/v1"
256 client = OpenAI(
257 api_key=openai_api_key,
258 base_url=openai_api_base,
259 )
260 completion = client.completions.create(
261 model="Qwen/Qwen2.5-1.5B-Instruct",
262 prompt="San Francisco is a",
263 )
264 print("Completion result:", completion)
265 ```
266
267 A more detailed client example can be found here: [examples/basic/offline_inference/basic.py](../../examples/basic/offline_inference/basic.py)
268
269 ### OpenAI Chat Completions API with vLLM
270
271 vLLM is designed to also support the OpenAI Chat Completions API. The chat interface is a more dynamic, interactive way to communicate with the model, allowing back-and-forth exchanges that can be stored in the chat history. This is useful for tasks that require context or more detailed explanations.
272
273 You can use the [create chat completion](https://platform.openai.com/docs/api-reference/chat/completions/create) endpoint to interact with the model:
274
275 ```bash
276 curl http://localhost:8000/v1/chat/completions \
277 -H "Content-Type: application/json" \
278 -d '{
279 "model": "Qwen/Qwen2.5-1.5B-Instruct",
280 "messages": [
281 {"role": "system", "content": "You are a helpful assistant."},
282 {"role": "user", "content": "Who won the world series in 2020?"}
283 ]
284 }'
285 ```
286
287 Alternatively, you can use the `openai` Python package:
288
289 ??? code
290
291 ```python
292 from openai import OpenAI
293 # Set OpenAI's API key and API base to use vLLM's API server.
294 openai_api_key = "EMPTY"
295 openai_api_base = "http://localhost:8000/v1"
296
297 client = OpenAI(
298 api_key=openai_api_key,
299 base_url=openai_api_base,
300 )
301
302 chat_response = client.chat.completions.create(
303 model="Qwen/Qwen2.5-1.5B-Instruct",
304 messages=[
305 {"role": "system", "content": "You are a helpful assistant."},
306 {"role": "user", "content": "Tell me a joke."},
307 ],
308 )
309 print("Chat response:", chat_response)
310 ```
311
312 ## On Attention Backends
313
314 Currently, vLLM supports multiple backends for efficient Attention computation across different platforms and accelerator architectures. It automatically selects the most performant backend compatible with your system and model specifications.
315
316 If desired, you can also manually set the backend of your choice using the `--attention-backend` CLI argument:
317
318 ```bash
319 # For online serving
320 vllm serve Qwen/Qwen2.5-1.5B-Instruct --attention-backend FLASH_ATTN
321
322 # For offline inference
323 python script.py --attention-backend FLASHINFER
324 ```
325
326 Some of the available backend options include:
327
328 - On NVIDIA CUDA: `FLASH_ATTN` or `FLASHINFER`.
329 - On AMD ROCm: `TRITON_ATTN`, `ROCM_ATTN`, `ROCM_AITER_FA`, `ROCM_AITER_UNIFIED_ATTN`, `TRITON_MLA`, `ROCM_AITER_MLA` or `ROCM_AITER_TRITON_MLA`.
330 - On Intel XPU: `FLASH_ATTN`, `TRITON_ATTN`, `TRITON_MLA`, `XPU_MLA_SPARSE`, `TORCH_SDPA` or `TURBOQUANT`.
331
332 !!! warning
333 There are no pre-built vllm wheels containing Flash Infer, so you must install it in your environment first. Refer to the [Flash Infer official docs](https://docs.flashinfer.ai/) or see [docker/Dockerfile](../../docker/Dockerfile) for instructions on how to install it.