A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 02:00 UTC
[Doc] Organise installation documentation into categories and tabs (#11935)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Jan 13, 2025 at 12:27 UTC
e8c23ff989d4b061726315bbf74d0bca7136fdc4
21 files changed
+1241
-392
docs/source/conf.py
+1
-1
@@ -56,7 +56,7 @@ templates_path = ['_templates']
56
# List of patterns, relative to source directory, that match files and
57
# directories to ignore when looking for source files.
58
# This pattern also affects html_static_path and html_extra_path.
59
-exclude_patterns: List[str] = ["**/*.template.md"]
59
+exclude_patterns: List[str] = ["**/*.template.md", "**/*.inc.md"]
60
61
# Exclude the prompt "$" when copying code
62
copybutton_prompt_text = r"\$ "
docs/source/deployment/docker.md
+4
@@ -2,6 +2,8 @@
2
3
# Using Docker
4
5
+(deployment-docker-pre-built-image)=
6
+
7
## Use vLLM's Official Docker Image
8
9
vLLM offers an official Docker image for deployment.
@@ -23,6 +25,8 @@ container to access the host's shared memory. vLLM uses PyTorch, which uses shar
25
memory to share data between processes under the hood, particularly for tensor parallel inference.
26
```
27
28
+(deployment-docker-build-image-from-source)=
29
+
30
## Building vLLM's Docker Image from Source
31
32
You can build and run vLLM from source via the provided <gh-file:Dockerfile>. To build vLLM:
docs/source/features/compatibility_matrix.md
+3
-1
@@ -322,7 +322,9 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
322
323
```
324
325
-### Feature x Hardware
325
+(feature-x-hardware)=
326
+
327
+## Feature x Hardware
328
329
```{list-table}
330
:header-rows: 1
docs/source/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md
renamed
+41
-31
@@ -1,10 +1,13 @@
1
-(installation-gaudi)=
1
+# Installation
2
3
-# Installation for Intel® Gaudi®
3
+This tab provides instructions on running vLLM with Intel Gaudi devices.
4
5
-This README provides instructions on running vLLM with Intel Gaudi devices.
5
+## Requirements
6
7
-## Requirements and Installation
7
+- OS: Ubuntu 22.04 LTS
8
+- Python: 3.10
9
+- Intel Gaudi accelerator
10
+- Intel Gaudi software version 1.18.0
11
12
Please follow the instructions provided in the [Gaudi Installation
13
Guide](https://docs.habana.ai/en/latest/Installation_Guide/index.html)
@@ -12,27 +15,9 @@ to set up the execution environment. To achieve the best performance,
15
please follow the methods outlined in the [Optimizing Training Platform
16
Guide](https://docs.habana.ai/en/latest/PyTorch/Model_Optimization_PyTorch/Optimization_in_Training_Platform.html).
17
15
-### Requirements
16
-
17
-- OS: Ubuntu 22.04 LTS
18
-- Python: 3.10
19
-- Intel Gaudi accelerator
20
-- Intel Gaudi software version 1.18.0
21
-
22
-### Quick start using Dockerfile
23
-
24
-```console
25
-docker build -f Dockerfile.hpu -t vllm-hpu-env .
26
-docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --rm vllm-hpu-env
27
-```
28
-
29
-```{tip}
30
-If you're observing the following error: `docker: Error response from daemon: Unknown runtime specified habana.`, please refer to "Install Using Containers" section of [Intel Gaudi Software Stack and Driver Installation](https://docs.habana.ai/en/v1.18.0/Installation_Guide/Bare_Metal_Fresh_OS.html). Make sure you have `habana-container-runtime` package installed and that `habana` container runtime is registered.
31
-```
18
+## Configure a new environment
19
33
-### Build from source
34
-
35
-#### Environment verification
20
+### Environment verification
21
22
To verify that the Intel Gaudi software was correctly installed, run:
23
@@ -47,7 +32,7 @@ Refer to [Intel Gaudi Software Stack
32
Verification](https://docs.habana.ai/en/latest/Installation_Guide/SW_Verification.html#platform-upgrade)
33
for more details.
34
50
-#### Run Docker Image
35
+### Run Docker Image
36
37
It is highly recommended to use the latest Docker image from Intel Gaudi
38
vault. Refer to the [Intel Gaudi
@@ -61,7 +46,13 @@ docker pull vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-i
46
docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --ipc=host vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest
47
```
48
64
-#### Build and Install vLLM
49
+## Set up using Python
50
+
51
+### Pre-built wheels
52
+
53
+Currently, there are no pre-built Intel Gaudi wheels.
54
+
55
+### Build wheel from source
56
57
To build and install vLLM from source, run:
58
@@ -80,7 +71,26 @@ git checkout habana_main
71
python setup.py develop
72
```
73
83
-## Supported Features
74
+## Set up using Docker
75
+
76
+### Pre-built images
77
+
78
+Currently, there are no pre-built Intel Gaudi images.
79
+
80
+### Build image from source
81
+
82
+```console
83
+docker build -f Dockerfile.hpu -t vllm-hpu-env .
84
+docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --rm vllm-hpu-env
85
+```
86
+
87
+```{tip}
88
+If you're observing the following error: `docker: Error response from daemon: Unknown runtime specified habana.`, please refer to "Install Using Containers" section of [Intel Gaudi Software Stack and Driver Installation](https://docs.habana.ai/en/v1.18.0/Installation_Guide/Bare_Metal_Fresh_OS.html). Make sure you have `habana-container-runtime` package installed and that `habana` container runtime is registered.
89
+```
90
+
91
+## Extra information
92
+
93
+## Supported features
94
95
- [Offline inference](#offline-inference)
96
- Online serving via [OpenAI-Compatible Server](#openai-compatible-server)
@@ -94,14 +104,14 @@ python setup.py develop
104
for accelerating low-batch latency and throughput
105
- Attention with Linear Biases (ALiBi)
106
97
-## Unsupported Features
107
+## Unsupported features
108
109
- Beam search
110
- LoRA adapters
111
- Quantization
112
- Prefill chunking (mixed-batch inferencing)
113
104
-## Supported Configurations
114
+## Supported configurations
115
116
The following configurations have been validated to be function with
117
Gaudi2 devices. Configurations that are not listed may or may not work.
@@ -137,7 +147,7 @@ Gaudi2 devices. Configurations that are not listed may or may not work.
147
- [meta-llama/Meta-Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct)
148
with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
149
140
-## Performance Tuning
150
+## Performance tuning
151
152
### Execution modes
153
@@ -368,7 +378,7 @@ Additionally, there are HPU PyTorch Bridge environment variables impacting vLLM
378
- `PT_HPU_LAZY_MODE`: if `0`, PyTorch Eager backend for Gaudi will be used, if `1` PyTorch Lazy backend for Gaudi will be used, `1` is default
379
- `PT_HPU_ENABLE_LAZY_COLLECTIVES`: required to be `true` for tensor parallel inference with HPU Graphs
380
371
-## Troubleshooting: Tweaking HPU Graphs
381
+## Troubleshooting: tweaking HPU graphs
382
383
If you experience device out-of-memory issues or want to attempt
384
inference at higher batch sizes, try tweaking HPU Graphs by following
docs/source/getting_started/installation/ai_accelerator/index.md
new
+375
@@ -0,0 +1,375 @@
1
+# Other AI accelerators
2
+
3
+vLLM is a Python library that supports the following AI accelerators. Select your AI accelerator type to see vendor specific instructions:
4
+
5
+::::{tab-set}
6
+:sync-group: device
7
+
8
+:::{tab-item} TPU
9
+:sync: tpu
10
+
11
+```{include} tpu.inc.md
12
+:start-after: "# Installation"
13
+:end-before: "## Requirements"
14
+```
15
+
16
+:::
17
+
18
+:::{tab-item} Intel Gaudi
19
+:sync: hpu-gaudi
20
+
21
+```{include} hpu-gaudi.inc.md
22
+:start-after: "# Installation"
23
+:end-before: "## Requirements"
24
+```
25
+
26
+:::
27
+
28
+:::{tab-item} Neuron
29
+:sync: neuron
30
+
31
+```{include} neuron.inc.md
32
+:start-after: "# Installation"
33
+:end-before: "## Requirements"
34
+```
35
+
36
+:::
37
+
38
+:::{tab-item} OpenVINO
39
+:sync: openvino
40
+
41
+```{include} openvino.inc.md
42
+:start-after: "# Installation"
43
+:end-before: "## Requirements"
44
+```
45
+
46
+:::
47
+
48
+::::
49
+
50
+## Requirements
51
+
52
+::::{tab-set}
53
+:sync-group: device
54
+
55
+:::{tab-item} TPU
56
+:sync: tpu
57
+
58
+```{include} tpu.inc.md
59
+:start-after: "## Requirements"
60
+:end-before: "## Configure a new environment"
61
+```
62
+
63
+:::
64
+
65
+:::{tab-item} Intel Gaudi
66
+:sync: hpu-gaudi
67
+
68
+```{include} hpu-gaudi.inc.md
69
+:start-after: "## Requirements"
70
+:end-before: "## Configure a new environment"
71
+```
72
+
73
+:::
74
+
75
+:::{tab-item} Neuron
76
+:sync: neuron
77
+
78
+```{include} neuron.inc.md
79
+:start-after: "## Requirements"
80
+:end-before: "## Configure a new environment"
81
+```
82
+
83
+:::
84
+
85
+:::{tab-item} OpenVINO
86
+:sync: openvino
87
+
88
+```{include} openvino.inc.md
89
+:start-after: "## Requirements"
90
+:end-before: "## Set up using Python"
91
+```
92
+
93
+:::
94
+
95
+::::
96
+
97
+## Configure a new environment
98
+
99
+::::{tab-set}
100
+:sync-group: device
101
+
102
+:::{tab-item} TPU
103
+:sync: tpu
104
+
105
+```{include} tpu.inc.md
106
+:start-after: "## Configure a new environment"
107
+:end-before: "## Set up using Python"
108
+```
109
+
110
+:::
111
+
112
+:::{tab-item} Intel Gaudi
113
+:sync: hpu-gaudi
114
+
115
+```{include} hpu-gaudi.inc.md
116
+:start-after: "## Configure a new environment"
117
+:end-before: "## Set up using Python"
118
+```
119
+
120
+:::
121
+
122
+:::{tab-item} Neuron
123
+:sync: neuron
124
+
125
+```{include} neuron.inc.md
126
+:start-after: "## Configure a new environment"
127
+:end-before: "## Set up using Python"
128
+```
129
+
130
+:::
131
+
132
+:::{tab-item} OpenVINO
133
+:sync: openvino
134
+
135
+```{include} ../python_env_setup.inc.md
136
+```
137
+
138
+:::
139
+
140
+::::
141
+
142
+## Set up using Python
143
+
144
+### Pre-built wheels
145
+
146
+::::{tab-set}
147
+:sync-group: device
148
+
149
+:::{tab-item} TPU
150
+:sync: tpu
151
+
152
+```{include} tpu.inc.md
153
+:start-after: "### Pre-built wheels"
154
+:end-before: "### Build wheel from source"
155
+```
156
+
157
+:::
158
+
159
+:::{tab-item} Intel Gaudi
160
+:sync: hpu-gaudi
161
+
162
+```{include} hpu-gaudi.inc.md
163
+:start-after: "### Pre-built wheels"
164
+:end-before: "### Build wheel from source"
165
+```
166
+
167
+:::
168
+
169
+:::{tab-item} Neuron
170
+:sync: neuron
171
+
172
+```{include} neuron.inc.md
173
+:start-after: "### Pre-built wheels"
174
+:end-before: "### Build wheel from source"
175
+```
176
+
177
+:::
178
+
179
+:::{tab-item} OpenVINO
180
+:sync: openvino
181
+
182
+```{include} openvino.inc.md
183
+:start-after: "### Pre-built wheels"
184
+:end-before: "### Build wheel from source"
185
+```
186
+
187
+:::
188
+
189
+::::
190
+
191
+### Build wheel from source
192
+
193
+::::{tab-set}
194
+:sync-group: device
195
+
196
+:::{tab-item} TPU
197
+:sync: tpu
198
+
199
+```{include} tpu.inc.md
200
+:start-after: "### Build wheel from source"
201
+:end-before: "## Set up using Docker"
202
+```
203
+
204
+:::
205
+
206
+:::{tab-item} Intel Gaudi
207
+:sync: hpu-gaudi
208
+
209
+```{include} hpu-gaudi.inc.md
210
+:start-after: "### Build wheel from source"
211
+:end-before: "## Set up using Docker"
212
+```
213
+
214
+:::
215
+
216
+:::{tab-item} Neuron
217
+:sync: neuron
218
+
219
+```{include} neuron.inc.md
220
+:start-after: "### Build wheel from source"
221
+:end-before: "## Set up using Docker"
222
+```
223
+
224
+:::
225
+
226
+:::{tab-item} OpenVINO
227
+:sync: openvino
228
+
229
+```{include} openvino.inc.md
230
+:start-after: "### Build wheel from source"
231
+:end-before: "## Set up using Docker"
232
+```
233
+
234
+:::
235
+
236
+::::
237
+
238
+## Set up using Docker
239
+
240
+### Pre-built images
241
+
242
+::::{tab-set}
243
+:sync-group: device
244
+
245
+:::{tab-item} TPU
246
+:sync: tpu
247
+
248
+```{include} tpu.inc.md
249
+:start-after: "### Pre-built images"
250
+:end-before: "### Build image from source"
251
+```
252
+
253
+:::
254
+
255
+:::{tab-item} Intel Gaudi
256
+:sync: hpu-gaudi
257
+
258
+```{include} hpu-gaudi.inc.md
259
+:start-after: "### Pre-built images"
260
+:end-before: "### Build image from source"
261
+```
262
+
263
+:::
264
+
265
+:::{tab-item} Neuron
266
+:sync: neuron
267
+
268
+```{include} neuron.inc.md
269
+:start-after: "### Pre-built images"
270
+:end-before: "### Build image from source"
271
+```
272
+
273
+:::
274
+
275
+:::{tab-item} OpenVINO
276
+:sync: openvino
277
+
278
+```{include} openvino.inc.md
279
+:start-after: "### Pre-built images"
280
+:end-before: "### Build image from source"
281
+```
282
+
283
+:::
284
+
285
+::::
286
+
287
+### Build image from source
288
+
289
+::::{tab-set}
290
+:sync-group: device
291
+
292
+:::{tab-item} TPU
293
+:sync: tpu
294
+
295
+```{include} tpu.inc.md
296
+:start-after: "### Build image from source"
297
+:end-before: "## Extra information"
298
+```
299
+
300
+:::
301
+
302
+:::{tab-item} Intel Gaudi
303
+:sync: hpu-gaudi
304
+
305
+```{include} hpu-gaudi.inc.md
306
+:start-after: "### Build image from source"
307
+:end-before: "## Extra information"
308
+```
309
+
310
+:::
311
+
312
+:::{tab-item} Neuron
313
+:sync: neuron
314
+
315
+```{include} neuron.inc.md
316
+:start-after: "### Build image from source"
317
+:end-before: "## Extra information"
318
+```
319
+
320
+:::
321
+
322
+:::{tab-item} OpenVINO
323
+:sync: openvino
324
+
325
+```{include} openvino.inc.md
326
+:start-after: "### Build image from source"
327
+:end-before: "## Extra information"
328
+```
329
+
330
+:::
331
+
332
+::::
333
+
334
+## Extra information
335
+
336
+::::{tab-set}
337
+:sync-group: device
338
+
339
+:::{tab-item} TPU
340
+:sync: tpu
341
+
342
+```{include} tpu.inc.md
343
+:start-after: "## Extra information"
344
+```
345
+
346
+:::
347
+
348
+:::{tab-item} Intel Gaudi
349
+:sync: hpu-gaudi
350
+
351
+```{include} hpu-gaudi.inc.md
352
+:start-after: "## Extra information"
353
+```
354
+
355
+:::
356
+
357
+:::{tab-item} Neuron
358
+:sync: neuron
359
+
360
+```{include} neuron.inc.md
361
+:start-after: "## Extra information"
362
+```
363
+
364
+:::
365
+
366
+:::{tab-item} OpenVINO
367
+:sync: openvino
368
+
369
+```{include} openvino.inc.md
370
+:start-after: "## Extra information"
371
+```
372
+
373
+:::
374
+
375
+::::
docs/source/getting_started/installation/ai_accelerator/neuron.inc.md
renamed
+35
-32
@@ -1,6 +1,4 @@
1
-(installation-neuron)=
2
-
3
-# Installation for Neuron
1
+# Installation
2
3
vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Inferentia with Neuron SDK with continuous batching.
4
Paged Attention and Chunked Prefill are currently in development and will be available soon.
@@ -14,28 +12,9 @@ Data types currently supported in Neuron SDK are FP16 and BF16.
12
- Pytorch 2.0.1/2.1.1
13
- AWS Neuron SDK 2.16/2.17 (Verified on python 3.8)
14
17
-Installation steps:
18
-
19
-- [Build from source](#build-from-source-neuron)
20
-
21
- - [Step 0. Launch Trn1/Inf2 instances](#launch-instances)
22
- - [Step 1. Install drivers and tools](#install-drivers)
23
- - [Step 2. Install transformers-neuronx and its dependencies](#install-tnx)
24
- - [Step 3. Install vLLM from source](#install-vllm)
25
-
26
-(build-from-source-neuron)=
27
-
28
-```{note}
29
-The currently supported version of Pytorch for Neuron installs `triton` version `2.1.0`. This is incompatible with `vllm >= 0.5.3`. You may see an error `cannot import name 'default_dump_dir...`. To work around this, run a `pip install --upgrade triton==3.0.0` after installing the vLLM wheel.
30
-```
31
-
32
-## Build from source
33
-
34
-Following instructions are applicable to Neuron SDK 2.16 and beyond.
35
-
36
-(launch-instances)=
15
+## Configure a new environment
16
38
-### Step 0. Launch Trn1/Inf2 instances
17
+### Launch Trn1/Inf2 instances
18
19
Here are the steps to launch trn1/inf2 instances, in order to install [PyTorch Neuron ("torch-neuronx") Setup on Ubuntu 22.04 LTS](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/setup/neuron-setup/pytorch/neuronx/ubuntu/torch-neuronx-ubuntu22.html).
20
@@ -45,9 +24,7 @@ Here are the steps to launch trn1/inf2 instances, in order to install [PyTorch N
24
- When launching a Trn1/Inf2, please adjust your primary EBS volume size to a minimum of 512GB.
25
- After launching the instance, follow the instructions in [Connect to your instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html) to connect to the instance
26
48
-(install-drivers)=
49
-
50
-### Step 1. Install drivers and tools
27
+### Install drivers and tools
28
29
The installation of drivers and tools wouldn't be necessary, if [Deep Learning AMI Neuron](https://docs.aws.amazon.com/dlami/latest/devguide/appendix-ami-release-notes.html) is installed. In case the drivers and tools are not installed on the operating system, follow the steps below:
30
@@ -82,9 +59,21 @@ sudo apt-get install aws-neuronx-tools=2.* -y
59
export PATH=/opt/aws/neuron/bin:$PATH
60
```
61
85
-(install-tnx)=
62
+## Set up using Python
63
+
64
+### Pre-built wheels
65
87
-### Step 2. Install transformers-neuronx and its dependencies
66
+Currently, there are no pre-built Neuron wheels.
67
+
68
+### Build wheel from source
69
+
70
+```{note}
71
+The currently supported version of Pytorch for Neuron installs `triton` version `2.1.0`. This is incompatible with `vllm >= 0.5.3`. You may see an error `cannot import name 'default_dump_dir...`. To work around this, run a `pip install --upgrade triton==3.0.0` after installing the vLLM wheel.
72
+```
73
+
74
+Following instructions are applicable to Neuron SDK 2.16 and beyond.
75
+
76
+#### Install transformers-neuronx and its dependencies
77
78
[transformers-neuronx](https://github.com/aws-neuron/transformers-neuronx) will be the backend to support inference on trn1/inf2 instances.
79
Follow the steps below to install transformer-neuronx package and its dependencies.
@@ -116,9 +105,7 @@ python -m pip install awscli
105
python -m pip install --upgrade neuronx-cc==2.* --pre torch-neuronx==2.1.* torchvision transformers-neuronx
106
```
107
119
-(install-vllm)=
120
-
121
-### Step 3. Install vLLM from source
108
+#### Install vLLM from source
109
110
Once neuronx-cc and transformers-neuronx packages are installed, we will be able to install vllm as follows:
111
@@ -130,3 +117,19 @@ VLLM_TARGET_DEVICE="neuron" pip install .
117
```
118
119
If neuron packages are detected correctly in the installation process, `vllm-0.3.0+neuron212` will be installed.
120
+
121
+## Set up using Docker
122
+
123
+### Pre-built images
124
+
125
+Currently, there are no pre-built Neuron images.
126
+
127
+### Build image from source
128
+
129
+See <project:#deployment-docker-build-image-from-source> for instructions on building the Docker image.
130
+
131
+Make sure to use <gh-file:Dockerfile.neuron> in place of the default Dockerfile.
132
+
133
+## Extra information
134
+
135
+There is no extra information for this device.
docs/source/getting_started/installation/ai_accelerator/openvino.inc.md
renamed
+39
-39
@@ -1,63 +1,65 @@
1
-(installation-openvino)=
1
+# Installation
2
3
-# Installation for OpenVINO
3
+vLLM powered by OpenVINO supports all LLM models from [vLLM supported models list](#supported-models) and can perform optimal model serving on all x86-64 CPUs with, at least, AVX2 support, as well as on both integrated and discrete Intel® GPUs ([the list of supported GPUs](https://docs.openvino.ai/2024/about-openvino/release-notes-openvino/system-requirements.html#gpu)).
4
5
-vLLM powered by OpenVINO supports all LLM models from [vLLM supported models list](#supported-models) and can perform optimal model serving on all x86-64 CPUs with, at least, AVX2 support, as well as on both integrated and discrete Intel® GPUs ([the list of supported GPUs](https://docs.openvino.ai/2024/about-openvino/release-notes-openvino/system-requirements.html#gpu)). OpenVINO vLLM backend supports the following advanced vLLM features:
5
+## Requirements
6
7
-- Prefix caching (`--enable-prefix-caching`)
8
-- Chunked prefill (`--enable-chunked-prefill`)
7
+- OS: Linux
8
+- Instruction set architecture (ISA) requirement: at least AVX2.
9
10
-**Table of contents**:
10
+## Set up using Python
11
12
-- [Requirements](#openvino-backend-requirements)
13
-- [Quick start using Dockerfile](#openvino-backend-quick-start-dockerfile)
14
-- [Build from source](#install-openvino-backend-from-source)
15
-- [Performance tips](#openvino-backend-performance-tips)
16
-- [Limitations](#openvino-backend-limitations)
12
+### Pre-built wheels
13
18
-(openvino-backend-requirements)=
14
+Currently, there are no pre-built OpenVINO wheels.
15
20
-## Requirements
16
+### Build wheel from source
17
22
-- OS: Linux
23
-- Instruction set architecture (ISA) requirement: at least AVX2.
18
+First, install Python. For example, on Ubuntu 22.04, you can run:
19
25
-(openvino-backend-quick-start-dockerfile)=
20
+```console
21
+sudo apt-get update -y
22
+sudo apt-get install python3
23
+```
24
27
-## Quick start using Dockerfile
25
+Second, install prerequisites vLLM OpenVINO backend installation:
26
27
```console
30
-docker build -f Dockerfile.openvino -t vllm-openvino-env .
31
-docker run -it --rm vllm-openvino-env
28
+pip install --upgrade pip
29
+pip install -r requirements-build.txt --extra-index-url https://download.pytorch.org/whl/cpu
30
```
31
34
-(install-openvino-backend-from-source)=
32
+Finally, install vLLM with OpenVINO backend:
33
36
-## Install from source
34
+```console
35
+PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu" VLLM_TARGET_DEVICE=openvino python -m pip install -v .
36
+```
37
38
-- First, install Python. For example, on Ubuntu 22.04, you can run:
38
+:::{tip}
39
+To use vLLM OpenVINO backend with a GPU device, ensure your system is properly set up. Follow the instructions provided here: [https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html](https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html).
40
+:::
41
40
- ```console
41
- sudo apt-get update -y
42
- sudo apt-get install python3
43
- ```
42
+## Set up using Docker
43
45
-- Second, install prerequisites vLLM OpenVINO backend installation:
44
+### Pre-built images
45
47
- ```console
48
- pip install --upgrade pip
49
- pip install -r requirements-build.txt --extra-index-url https://download.pytorch.org/whl/cpu
50
- ```
46
+Currently, there are no pre-built OpenVINO images.
47
52
-- Finally, install vLLM with OpenVINO backend:
48
+### Build image from source
49
54
- ```console
55
- PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu" VLLM_TARGET_DEVICE=openvino python -m pip install -v .
56
- ```
50
+```console
51
+docker build -f Dockerfile.openvino -t vllm-openvino-env .
52
+docker run -it --rm vllm-openvino-env
53
+```
54
58
-- [Optional] To use vLLM OpenVINO backend with a GPU device, ensure your system is properly set up. Follow the instructions provided here: [https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html](https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html).
55
+## Extra information
56
60
-(openvino-backend-performance-tips)=
57
+## Supported features
58
+
59
+OpenVINO vLLM backend supports the following advanced vLLM features:
60
+
61
+- Prefix caching (`--enable-prefix-caching`)
62
+- Chunked prefill (`--enable-chunked-prefill`)
63
64
## Performance tips
65
@@ -95,8 +97,6 @@ $ VLLM_OPENVINO_DEVICE=GPU VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON \
97
python3 vllm/benchmarks/benchmark_throughput.py --model meta-llama/Llama-2-7b-chat-hf --dataset vllm/benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json
98
```
99
98
-(openvino-backend-limitations)=
99
-
100
## Limitations
101
102
- LoRA serving is not supported.
docs/source/getting_started/installation/ai_accelerator/tpu.inc.md
renamed
+27
-11
@@ -1,6 +1,4 @@
1
-(installation-tpu)=
2
-
3
-# Installation for TPUs
1
+# Installation
2
3
Tensor Processing Units (TPUs) are Google's custom-developed application-specific
4
integrated circuits (ASICs) used to accelerate machine learning workloads. TPUs
@@ -54,7 +52,16 @@ In all of the following commands, replace the ALL CAPS parameter names with
52
appropriate values. See the parameter descriptions table for more information.
53
```
54
57
-## Provision a Cloud TPU with the queued resource API
55
+### Provision Cloud TPUs with GKE
56
+
57
+For more information about using TPUs with GKE, see:
58
+- <https://cloud.google.com/kubernetes-engine/docs/how-to/tpus>
59
+- <https://cloud.google.com/kubernetes-engine/docs/concepts/tpus>
60
+- <https://cloud.google.com/kubernetes-engine/docs/concepts/plan-tpus>
61
+
62
+## Configure a new environment
63
+
64
+### Provision a Cloud TPU with the queued resource API
65
66
Create a TPU v5e with 4 TPU chips:
67
@@ -102,6 +109,14 @@ Connect to your TPU using SSH:
109
gcloud compute tpus tpu-vm ssh TPU_NAME --zone ZONE
110
```
111
112
+## Set up using Python
113
+
114
+### Pre-built wheels
115
+
116
+Currently, there are no pre-built TPU wheels.
117
+
118
+### Build wheel from source
119
+
120
Install Miniconda:
121
122
```bash
@@ -142,16 +157,13 @@ Run the setup script:
157
VLLM_TARGET_DEVICE="tpu" python setup.py develop
158
```
159
145
-## Provision Cloud TPUs with GKE
160
+## Set up using Docker
161
147
-For more information about using TPUs with GKE, see
148
-<https://cloud.google.com/kubernetes-engine/docs/how-to/tpus>
149
-<https://cloud.google.com/kubernetes-engine/docs/concepts/tpus>
150
-<https://cloud.google.com/kubernetes-engine/docs/concepts/plan-tpus>
162
+### Pre-built images
163
152
-(build-docker-tpu)=
164
+See <project:#deployment-docker-pre-built-image> for instructions on using the official Docker image, making sure to substitute the image name `vllm/vllm-openai` with `vllm/vllm-tpu`.
165
154
-## Build a docker image with {code}`Dockerfile.tpu`
166
+### Build image from source
167
168
You can use <gh-file:Dockerfile.tpu> to build a Docker image with TPU support.
169
@@ -189,3 +201,7 @@ Install OpenBLAS with the following command:
201
$ sudo apt-get install libopenblas-base libopenmpi-dev libomp-dev
202
```
203
````
204
+
205
+## Extra information
206
+
207
+There is no extra information for this device.
docs/source/getting_started/installation/cpu-arm.md
deleted
-46
@@ -1,46 +0,0 @@
1
-(installation-arm)=
2
-
3
-# Installation for ARM CPUs
4
-
5
-vLLM has been adapted to work on ARM64 CPUs with NEON support, leveraging the CPU backend initially developed for the x86 platform. This guide provides installation instructions specific to ARM (which also apply to Apple Silicon, see [Installation for macOS](#installation-apple) for more). For additional details on supported features, refer to the [x86 CPU documentation](#installation-x86) covering:
6
-
7
-- CPU backend inference capabilities
8
-- Relevant runtime environment variables
9
-- Performance optimization tips
10
-
11
-ARM CPU backend currently supports Float32, FP16 and BFloat16 datatypes.
12
-Contents:
13
-
14
-1. [Requirements](#arm-backend-requirements)
15
-2. [Quick Start with Dockerfile](#arm-backend-quick-start-dockerfile)
16
-3. [Building from Source](#build-arm-backend-from-source)
17
-
18
-(arm-backend-requirements)=
19
-
20
-## Requirements
21
-
22
-- **Operating System**: Linux or macOS
23
-- **Compilers**: `gcc/g++ >= 12.3.0` (optional, but recommended) or `Apple Clang >= 15.0.0` for macOS
24
-- **Instruction Set Architecture (ISA)**: NEON support is required
25
-
26
-(arm-backend-quick-start-dockerfile)=
27
-
28
-## Quick Start with Dockerfile
29
-
30
-You can quickly set up vLLM on ARM using Docker:
31
-
32
-```console
33
-$ docker build -f Dockerfile.arm -t vllm-cpu-env --shm-size=4g .
34
-$ docker run -it \
35
- --rm \
36
- --network=host \
37
- --cpuset-cpus=<cpu-id-list, optional> \
38
- --cpuset-mems=<memory-node, optional> \
39
- vllm-cpu-env
40
-```
41
-
42
-(build-arm-backend-from-source)=
43
-
44
-## Building from Source
45
-
46
-To build vLLM from source on Ubuntu 22.04 or other Linux distributions, follow a similar process as with x86. Testing has been conducted on AWS Graviton3 instances for compatibility.
docs/source/getting_started/installation/cpu/apple.inc.md
renamed
+18
-10
@@ -1,20 +1,20 @@
1
-(installation-apple)=
1
+# Installation
2
3
-# Installation for macOS
4
-
5
-vLLM has experimental support for macOS with Apple Silicon. For now, users shall build from the source vLLM to natively run on macOS. For more details, like running on vLLM in a docker container, see [ARM CPU Documentation](installation-arm)
3
+vLLM has experimental support for macOS with Apple silicon. For now, users shall build from the source vLLM to natively run on macOS.
4
5
Currently the CPU implementation for macOS supports FP32 and FP16 datatypes.
6
7
## Requirements
8
11
-- **Operating System**: `macOS Sonoma` or later
12
-- **SDK** `XCode 15.4` or later with Command Line Tools
13
-- **Compilers**: `Apple Clang >= 15.0.0`
9
+- OS: `macOS Sonoma` or later
10
+- SDK: `XCode 15.4` or later with Command Line Tools
11
+- Compiler: `Apple Clang >= 15.0.0`
12
+
13
+## Set up using Python
14
15
-<!-- (arm-backend-quick-start-dockerfile)= -->
15
+### Pre-built wheels
16
17
-## Build and installation
17
+### Build wheel from source
18
19
After installation of XCode and the Command Line Tools, which include Apple Clang, execute the following commands to build and install vLLM from the source.
20
@@ -29,7 +29,7 @@ pip install -e .
29
On macOS the `VLLM_TARGET_DEVICE` is automatically set to `cpu`, which currently is the only supported device.
30
```
31
32
-## Troubleshooting
32
+#### Troubleshooting
33
34
If the build has error like the following snippet where standard C++ headers cannot be found, try to remove and reinstall your
35
[Command Line Tools for Xcode](https://developer.apple.com/download/all/).
@@ -46,3 +46,11 @@ If the build has error like the following snippet where standard C++ headers can
46
| ^~~~~~~~~
47
1 error generated.
48
```
49
+
50
+## Set up using Docker
51
+
52
+### Pre-built images
53
+
54
+### Build image from source
55
+
56
+## Extra information
docs/source/getting_started/installation/cpu/arm.inc.md
new
+30
@@ -0,0 +1,30 @@
1
+# Installation
2
+
3
+vLLM has been adapted to work on ARM64 CPUs with NEON support, leveraging the CPU backend initially developed for the x86 platform.
4
+
5
+ARM CPU backend currently supports Float32, FP16 and BFloat16 datatypes.
6
+
7
+## Requirements
8
+
9
+- OS: Linux
10
+- Compiler: `gcc/g++ >= 12.3.0` (optional, recommended)
11
+- Instruction Set Architecture (ISA): NEON support is required
12
+
13
+## Set up using Python
14
+
15
+### Pre-built wheels
16
+
17
+### Build wheel from source
18
+
19
+:::{include} build.inc.md
20
+:::
21
+
22
+Testing has been conducted on AWS Graviton3 instances for compatibility.
23
+
24
+## Set up using Docker
25
+
26
+### Pre-built images
27
+
28
+### Build image from source
29
+
30
+## Extra information
docs/source/getting_started/installation/cpu/build.inc.md
new
+21
@@ -0,0 +1,21 @@
1
+First, install recommended compiler. We recommend to use `gcc/g++ >= 12.3.0` as the default compiler to avoid potential problems. For example, on Ubuntu 22.4, you can run:
2
+
3
+```console
4
+sudo apt-get update -y
5
+sudo apt-get install -y gcc-12 g++-12 libnuma-dev
6
+sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12
7
+```
8
+
9
+Second, install Python packages for vLLM CPU backend building:
10
+
11
+```console
12
+pip install --upgrade pip
13
+pip install cmake>=3.26 wheel packaging ninja "setuptools-scm>=8" numpy
14
+pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
15
+```
16
+
17
+Finally, build and install vLLM CPU backend:
18
+
19
+```console
20
+VLLM_TARGET_DEVICE=cpu python setup.py install
21
+```
docs/source/getting_started/installation/cpu/index.md
renamed
+132
-58
@@ -1,91 +1,165 @@
1
-(installation-x86)=
1
+# CPU
2
3
-# Installation for x86 CPUs
3
+vLLM is a Python library that supports the following CPU variants. Select your CPU type to see vendor specific instructions:
4
5
-vLLM initially supports basic model inferencing and serving on x86 CPU platform, with data types FP32, FP16 and BF16. vLLM CPU backend supports the following vLLM features:
5
+::::{tab-set}
6
+:sync-group: device
7
7
-- Tensor Parallel
8
-- Model Quantization (`INT8 W8A8, AWQ, GPTQ`)
9
-- Chunked-prefill
10
-- Prefix-caching
11
-- FP8-E5M2 KV-Caching (TODO)
8
+:::{tab-item} x86
9
+:sync: x86
10
+
11
+```{include} x86.inc.md
12
+:start-after: "# Installation"
13
+:end-before: "## Requirements"
14
+```
15
13
-Table of contents:
16
+:::
17
+
18
+:::{tab-item} ARM
19
+:sync: arm
20
+
21
+```{include} arm.inc.md
22
+:start-after: "# Installation"
23
+:end-before: "## Requirements"
24
+```
25
+
26
+:::
27
+
28
+:::{tab-item} Apple silicon
29
+:sync: apple
30
+
31
+```{include} apple.inc.md
32
+:start-after: "# Installation"
33
+:end-before: "## Requirements"
34
+```
35
15
-1. [Requirements](#cpu-backend-requirements)
16
-2. [Quick start using Dockerfile](#cpu-backend-quick-start-dockerfile)
17
-3. [Build from source](#build-cpu-backend-from-source)
18
-4. [Related runtime environment variables](#env-intro)
19
-5. [Intel Extension for PyTorch](#ipex-guidance)
20
-6. [Performance tips](#cpu-backend-performance-tips)
36
+:::
37
22
-(cpu-backend-requirements)=
38
+::::
39
40
## Requirements
41
26
-- OS: Linux
27
-- Compiler: `gcc/g++>=12.3.0` (optional, recommended)
28
-- Instruction set architecture (ISA) requirement: AVX512 (optional, recommended)
42
+- Python: 3.9 -- 3.12
43
30
-(cpu-backend-quick-start-dockerfile)=
44
+::::{tab-set}
45
+:sync-group: device
46
32
-## Quick start using Dockerfile
47
+:::{tab-item} x86
48
+:sync: x86
49
34
-```console
35
-docker build -f Dockerfile.cpu -t vllm-cpu-env --shm-size=4g .
36
-docker run -it \
37
- --rm \
38
- --network=host \
39
- --cpuset-cpus=<cpu-id-list, optional> \
40
- --cpuset-mems=<memory-node, optional> \
41
- vllm-cpu-env
50
+```{include} x86.inc.md
51
+:start-after: "## Requirements"
52
+:end-before: "## Set up using Python"
53
```
54
44
-(build-cpu-backend-from-source)=
55
+:::
56
+
57
+:::{tab-item} ARM
58
+:sync: arm
59
46
-## Build from source
60
+```{include} arm.inc.md
61
+:start-after: "## Requirements"
62
+:end-before: "## Set up using Python"
63
+```
64
48
-- First, install recommended compiler. We recommend to use `gcc/g++ >= 12.3.0` as the default compiler to avoid potential problems. For example, on Ubuntu 22.4, you can run:
65
+:::
66
50
-```console
51
-sudo apt-get update -y
52
-sudo apt-get install -y gcc-12 g++-12 libnuma-dev
53
-sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12
67
+:::{tab-item} Apple silicon
68
+:sync: apple
69
+
70
+```{include} apple.inc.md
71
+:start-after: "## Requirements"
72
+:end-before: "## Set up using Python"
73
```
74
56
-- Second, install Python packages for vLLM CPU backend building:
75
+:::
76
58
-```console
59
-pip install --upgrade pip
60
-pip install cmake>=3.26 wheel packaging ninja "setuptools-scm>=8" numpy
61
-pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
77
+::::
78
+
79
+## Set up using Python
80
+
81
+### Create a new Python environment
82
+
83
+```{include} ../python_env_setup.inc.md
84
```
85
64
-- Finally, build and install vLLM CPU backend:
86
+### Pre-built wheels
87
66
-```console
67
-VLLM_TARGET_DEVICE=cpu python setup.py install
88
+Currently, there are no pre-built CPU wheels.
89
+
90
+### Build wheel from source
91
+
92
+::::{tab-set}
93
+:sync-group: device
94
+
95
+:::{tab-item} x86
96
+:sync: x86
97
+
98
+```{include} x86.inc.md
99
+:start-after: "### Build wheel from source"
100
+:end-before: "## Set up using Docker"
101
```
102
70
-```{note}
71
-- AVX512_BF16 is an extension ISA provides native BF16 data type conversion and vector product instructions, will brings some performance improvement compared with pure AVX512. The CPU backend build script will check the host CPU flags to determine whether to enable AVX512_BF16.
72
-- If you want to force enable AVX512_BF16 for the cross-compilation, please set environment variable `VLLM_CPU_AVX512BF16=1` before the building.
103
+:::
104
+
105
+:::{tab-item} ARM
106
+:sync: arm
107
+
108
+```{include} arm.inc.md
109
+:start-after: "### Build wheel from source"
110
+:end-before: "## Set up using Docker"
111
```
112
75
-(env-intro)=
113
+:::
114
77
-## Related runtime environment variables
115
+:::{tab-item} Apple silicon
116
+:sync: apple
117
79
-- `VLLM_CPU_KVCACHE_SPACE`: specify the KV Cache size (e.g, `VLLM_CPU_KVCACHE_SPACE=40` means 40 GB space for KV cache), larger setting will allow vLLM running more requests in parallel. This parameter should be set based on the hardware configuration and memory management pattern of users.
80
-- `VLLM_CPU_OMP_THREADS_BIND`: specify the CPU cores dedicated to the OpenMP threads. For example, `VLLM_CPU_OMP_THREADS_BIND=0-31` means there will be 32 OpenMP threads bound on 0-31 CPU cores. `VLLM_CPU_OMP_THREADS_BIND=0-31|32-63` means there will be 2 tensor parallel processes, 32 OpenMP threads of rank0 are bound on 0-31 CPU cores, and the OpenMP threads of rank1 are bound on 32-63 CPU cores.
118
+```{include} apple.inc.md
119
+:start-after: "### Build wheel from source"
120
+:end-before: "## Set up using Docker"
121
+```
122
+
123
+:::
124
+
125
+::::
126
+
127
+## Set up using Docker
128
+
129
+### Pre-built images
130
+
131
+Currently, there are no pre-build CPU images.
132
+
133
+### Build image from source
134
82
-(ipex-guidance)=
135
+```console
136
+$ docker build -f Dockerfile.cpu -t vllm-cpu-env --shm-size=4g .
137
+$ docker run -it \
138
+ --rm \
139
+ --network=host \
140
+ --cpuset-cpus=<cpu-id-list, optional> \
141
+ --cpuset-mems=<memory-node, optional> \
142
+ vllm-cpu-env
143
+```
144
84
-## Intel Extension for PyTorch
145
+:::{tip}
146
+For ARM or Apple silicon, use `Dockerfile.arm`
147
+:::
148
86
-- [Intel Extension for PyTorch (IPEX)](https://github.com/intel/intel-extension-for-pytorch) extends PyTorch with up-to-date features optimizations for an extra performance boost on Intel hardware.
149
+## Supported features
150
88
-(cpu-backend-performance-tips)=
151
+vLLM CPU backend supports the following vLLM features:
152
+
153
+- Tensor Parallel
154
+- Model Quantization (`INT8 W8A8, AWQ, GPTQ`)
155
+- Chunked-prefill
156
+- Prefix-caching
157
+- FP8-E5M2 KV-Caching (TODO)
158
+
159
+## Related runtime environment variables
160
+
161
+- `VLLM_CPU_KVCACHE_SPACE`: specify the KV Cache size (e.g, `VLLM_CPU_KVCACHE_SPACE=40` means 40 GB space for KV cache), larger setting will allow vLLM running more requests in parallel. This parameter should be set based on the hardware configuration and memory management pattern of users.
162
+- `VLLM_CPU_OMP_THREADS_BIND`: specify the CPU cores dedicated to the OpenMP threads. For example, `VLLM_CPU_OMP_THREADS_BIND=0-31` means there will be 32 OpenMP threads bound on 0-31 CPU cores. `VLLM_CPU_OMP_THREADS_BIND=0-31|32-63` means there will be 2 tensor parallel processes, 32 OpenMP threads of rank0 are bound on 0-31 CPU cores, and the OpenMP threads of rank1 are bound on 32-63 CPU cores.
163
164
## Performance tips
165
@@ -137,13 +211,13 @@ $ python examples/offline_inference/basic.py
211
212
- If using vLLM CPU backend on a multi-socket machine with NUMA, be aware to set CPU cores using `VLLM_CPU_OMP_THREADS_BIND` to avoid cross NUMA node memory access.
213
140
-## CPU Backend Considerations
214
+## Other considerations
215
216
- The CPU backend significantly differs from the GPU backend since the vLLM architecture was originally optimized for GPU use. A number of optimizations are needed to enhance its performance.
217
218
- Decouple the HTTP serving components from the inference components. In a GPU backend configuration, the HTTP serving and tokenization tasks operate on the CPU, while inference runs on the GPU, which typically does not pose a problem. However, in a CPU-based setup, the HTTP serving and tokenization can cause significant context switching and reduced cache efficiency. Therefore, it is strongly recommended to segregate these two components for improved performance.
219
146
-- On CPU based setup with NUMA enabled, the memory access performance may be largely impacted by the [topology](https://github.com/intel/intel-extension-for-pytorch/blob/main/docs/tutorials/performance_tuning/tuning_guide.md#non-uniform-memory-access-numa). For NUMA architecture, two optimizations are to recommended: Tensor Parallel or Data Parallel.
220
+- On CPU based setup with NUMA enabled, the memory access performance may be largely impacted by the [topology](https://github.com/intel/intel-extension-for-pytorch/blob/main/docs/tutorials/performance_tuning/tuning_guide.inc.md#non-uniform-memory-access-numa). For NUMA architecture, two optimizations are to recommended: Tensor Parallel or Data Parallel.
221
222
- Using Tensor Parallel for a latency constraints deployment: following GPU backend design, a Megatron-LM's parallel algorithm will be used to shard the model, based on the number of NUMA nodes (e.g. TP = 2 for a two NUMA node system). With [TP feature on CPU](gh-pr:6125) merged, Tensor Parallel is supported for serving and offline inferencing. In general each NUMA node is treated as one GPU card. Below is the example script to enable Tensor Parallel = 2 for serving:
223
@@ -151,4 +225,4 @@ $ python examples/offline_inference/basic.py
225
VLLM_CPU_KVCACHE_SPACE=40 VLLM_CPU_OMP_THREADS_BIND="0-31|32-63" vllm serve meta-llama/Llama-2-7b-chat-hf -tp=2 --distributed-executor-backend mp
226
```
227
154
- - Using Data Parallel for maximum throughput: to launch an LLM serving endpoint on each NUMA node along with one additional load balancer to dispatch the requests to those endpoints. Common solutions like [Nginx](#nginxloadbalancer) or HAProxy are recommended. Anyscale Ray project provides the feature on LLM [serving](https://docs.ray.io/en/latest/serve/index.html). Here is the example to setup a scalable LLM serving with [Ray Serve](https://github.com/intel/llm-on-ray/blob/main/docs/setup.md).
228
+ - Using Data Parallel for maximum throughput: to launch an LLM serving endpoint on each NUMA node along with one additional load balancer to dispatch the requests to those endpoints. Common solutions like [Nginx](#nginxloadbalancer) or HAProxy are recommended. Anyscale Ray project provides the feature on LLM [serving](https://docs.ray.io/en/latest/serve/index.html). Here is the example to setup a scalable LLM serving with [Ray Serve](https://github.com/intel/llm-on-ray/blob/main/docs/setup.inc.md).
docs/source/getting_started/installation/cpu/x86.inc.md
new
+35
@@ -0,0 +1,35 @@
1
+# Installation
2
+
3
+vLLM initially supports basic model inferencing and serving on x86 CPU platform, with data types FP32, FP16 and BF16.
4
+
5
+## Requirements
6
+
7
+- OS: Linux
8
+- Compiler: `gcc/g++ >= 12.3.0` (optional, recommended)
9
+- Instruction Set Architecture (ISA): AVX512 (optional, recommended)
10
+
11
+## Set up using Python
12
+
13
+### Pre-built wheels
14
+
15
+### Build wheel from source
16
+
17
+:::{include} build.inc.md
18
+:::
19
+
20
+```{note}
21
+- AVX512_BF16 is an extension ISA provides native BF16 data type conversion and vector product instructions, will brings some performance improvement compared with pure AVX512. The CPU backend build script will check the host CPU flags to determine whether to enable AVX512_BF16.
22
+- If you want to force enable AVX512_BF16 for the cross-compilation, please set environment variable `VLLM_CPU_AVX512BF16=1` before the building.
23
+```
24
+
25
+## Set up using Docker
26
+
27
+### Pre-built images
28
+
29
+### Build image from source
30
+
31
+## Extra information
32
+
33
+## Intel Extension for PyTorch
34
+
35
+- [Intel Extension for PyTorch (IPEX)](https://github.com/intel/intel-extension-for-pytorch) extends PyTorch with up-to-date features optimizations for an extra performance boost on Intel hardware.
docs/source/getting_started/installation/device.template.md
new
+17
@@ -0,0 +1,17 @@
1
+# Installation
2
+
3
+## Requirements
4
+
5
+## Set up using Python
6
+
7
+### Pre-built wheels
8
+
9
+### Build wheel from source
10
+
11
+## Set up using Docker
12
+
13
+### Pre-built images
14
+
15
+### Build image from source
16
+
17
+## Extra information
docs/source/getting_started/installation/gpu/cuda.inc.md
renamed
+42
-54
@@ -1,44 +1,24 @@
1
-(installation-cuda)=
1
+# Installation
2
3
-# Installation for CUDA
4
-
5
-vLLM is a Python library that also contains pre-compiled C++ and CUDA (12.1) binaries.
3
+vLLM contains pre-compiled C++ and CUDA (12.1) binaries.
4
5
## Requirements
6
9
-- OS: Linux
10
-- Python: 3.9 -- 3.12
7
- GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)
8
13
-## Install released versions
9
+## Set up using Python
10
11
### Create a new Python environment
12
17
-You can create a new Python environment using `conda`:
18
-
19
-```console
20
-# (Recommended) Create a new conda environment.
21
-conda create -n myenv python=3.12 -y
22
-conda activate myenv
23
-```
24
-
13
```{note}
26
-[PyTorch has deprecated the conda release channel](https://github.com/pytorch/pytorch/issues/138506). If you use `conda`, please only use it to create Python environment rather than installing packages. In particular, the PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
27
-```
28
-
29
-Or you can create a new Python environment using [uv](https://docs.astral.sh/uv/), a very fast Python environment manager. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment using the following command:
30
-
31
-```console
32
-# (Recommended) Create a new uv environment. Use `--seed` to install `pip` and `setuptools` in the environment.
33
-uv venv myenv --python 3.12 --seed
34
-source myenv/bin/activate
14
+PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
15
```
16
17
In order to be performant, vLLM has to compile many cuda kernels. The compilation unfortunately introduces binary incompatibility with other CUDA versions and PyTorch versions, even for the same PyTorch version with different building configurations.
18
19
Therefore, it is recommended to install vLLM with a **fresh new** environment. If either you have a different CUDA version or you want to use an existing PyTorch installation, you need to build vLLM from source. See [below](#build-from-source) for more details.
20
41
-### Install vLLM
21
+### Pre-built wheels
22
23
You can install vLLM using either `pip` or `uv pip`:
24
@@ -59,11 +39,11 @@ pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSI
39
40
(install-the-latest-code)=
41
62
-## Install the latest code
42
+#### Install the latest code
43
44
LLM inference is a fast-evolving field, and the latest code may contain bug fixes, performance improvements, and new features that are not released yet. To allow users to try the latest code without waiting for the next release, vLLM provides wheels for Linux running on a x86 platform with CUDA 12 for every commit since `v0.5.3`.
45
66
-### Install the latest code using `pip`
46
+##### Install the latest code using `pip`
47
48
```console
49
pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
@@ -80,7 +60,7 @@ pip install https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manyl
60
61
Note that the wheels are built with Python 3.8 ABI (see [PEP 425](https://peps.python.org/pep-0425/) for more details about ABI), so **they are compatible with Python 3.8 and later**. The version string in the wheel file name (`1.0.0.dev`) is just a placeholder to have a unified URL for the wheels, the actual versions of wheels are contained in the wheel metadata (the wheels listed in the extra index url have correct versions). Although we don't support Python 3.8 any more (because PyTorch 2.5 dropped support for Python 3.8), the wheels are still built with Python 3.8 ABI to keep the same wheel name as before.
62
83
-### Install the latest code using `uv`
63
+##### Install the latest code using `uv`
64
65
Another way to install the latest code is to use `uv`:
66
@@ -97,26 +77,9 @@ uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}
77
78
The `uv` approach works for vLLM `v0.6.6` and later and offers an easy-to-remember command. A unique feature of `uv` is that packages in `--extra-index-url` have [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes). If the latest public release is `v0.6.6.post1`, `uv`'s behavior allows installing a commit before `v0.6.6.post1` by specifying the `--extra-index-url`. In contrast, `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install a development version prior to the released version.
79
100
-### Install the latest code using `docker`
101
-
102
-Another way to access the latest code is to use the docker images:
103
-
104
-```console
105
-export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch
106
-docker pull public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:${VLLM_COMMIT}
107
-```
108
-
109
-These docker images are used for CI and testing only, and they are not intended for production use. They will be expired after several days.
110
-
111
-The latest code can contain bugs and may not be stable. Please use it with caution.
112
-
113
-(build-from-source)=
80
+### Build wheel from source
81
115
-## Build from source
116
-
117
-(python-only-build)=
118
-
119
-### Python-only build (without compilation)
82
+#### Set up using Python-only build (without compilation)
83
84
If you only need to change Python code, you can build and install vLLM without compilation. Using `pip`'s [`--editable` flag](https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs), changes you make to the code will be reflected when you run vLLM:
85
@@ -135,14 +98,14 @@ export VLLM_PRECOMPILED_WHEEL_LOCATION=https://files.pythonhosted.org/packages/4
98
pip install --editable .
99
```
100
138
-You can find more information about vLLM's wheels [above](#install-the-latest-code).
101
+You can find more information about vLLM's wheels in <project:#install-the-latest-code>.
102
103
```{note}
104
There is a possibility that your source code may have a different commit ID compared to the latest vLLM wheel, which could potentially lead to unknown errors.
142
-It is recommended to use the same commit ID for the source code as the vLLM wheel you have installed. Please refer to [the section above](#install-the-latest-code) for instructions on how to install a specified wheel.
105
+It is recommended to use the same commit ID for the source code as the vLLM wheel you have installed. Please refer to <project:#install-the-latest-code> for instructions on how to install a specified wheel.
106
```
107
145
-### Full build (with compilation)
108
+#### Full build (with compilation)
109
110
If you want to modify C++ or CUDA code, you'll need to build vLLM from source. This can take several minutes:
111
@@ -162,7 +125,7 @@ As long as `which ccache` command can find the `ccache` binary, it will be used
125
The following environment variables can be set to configure the vLLM `sccache` remote: `SCCACHE_BUCKET=vllm-build-sccache SCCACHE_REGION=us-west-2 SCCACHE_S3_NO_CREDENTIALS=1`. We also recommend setting `SCCACHE_IDLE_TIMEOUT=0`.
126
```
127
165
-#### Use an existing PyTorch installation
128
+##### Use an existing PyTorch installation
129
130
There are scenarios where the PyTorch dependency cannot be easily installed via pip, e.g.:
131
@@ -179,7 +142,7 @@ pip install -r requirements-build.txt
142
pip install -e . --no-build-isolation
143
```
144
182
-#### Use the local cutlass for compilation
145
+##### Use the local cutlass for compilation
146
147
Currently, before starting the build process, vLLM fetches cutlass code from GitHub. However, there may be scenarios where you want to use a local version of cutlass instead.
148
To achieve this, you can set the environment variable VLLM_CUTLASS_SRC_DIR to point to your local cutlass directory.
@@ -190,7 +153,7 @@ cd vllm
153
VLLM_CUTLASS_SRC_DIR=/path/to/cutlass pip install -e .
154
```
155
193
-#### Troubleshooting
156
+##### Troubleshooting
157
158
To avoid your system being overloaded, you can limit the number of compilation jobs
159
to be run simultaneously, via the environment variable `MAX_JOBS`. For example:
@@ -224,7 +187,7 @@ nvcc --version # verify that nvcc is in your PATH
187
${CUDA_HOME}/bin/nvcc --version # verify that nvcc is in your CUDA_HOME
188
```
189
227
-### Unsupported OS build
190
+#### Unsupported OS build
191
192
vLLM can fully run only on Linux but for development purposes, you can still build it on other systems (for example, macOS), allowing for imports and a more convenient development environment. The binaries will not be compiled and won't work on non-Linux systems.
193
@@ -234,3 +197,28 @@ Simply disable the `VLLM_TARGET_DEVICE` environment variable before installing:
197
export VLLM_TARGET_DEVICE=empty
198
pip install -e .
199
```
200
+
201
+## Set up using Docker
202
+
203
+### Pre-built images
204
+
205
+See <project:#deployment-docker-pre-built-image> for instructions on using the official Docker image.
206
+
207
+Another way to access the latest code is to use the docker images:
208
+
209
+```console
210
+export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch
211
+docker pull public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:${VLLM_COMMIT}
212
+```
213
+
214
+These docker images are used for CI and testing only, and they are not intended for production use. They will be expired after several days.
215
+
216
+The latest code can contain bugs and may not be stable. Please use it with caution.
217
+
218
+### Build image from source
219
+
220
+See <project:#deployment-docker-build-image-from-source> for instructions on building the Docker image.
221
+
222
+## Supported features
223
+
224
+See <project:#feature-x-hardware> compatibility matrix for feature support information.
docs/source/getting_started/installation/gpu/index.md
new
+300
@@ -0,0 +1,300 @@
1
+# GPU
2
+
3
+vLLM is a Python library that supports the following GPU variants. Select your GPU type to see vendor specific instructions:
4
+
5
+::::{tab-set}
6
+:sync-group: device
7
+
8
+:::{tab-item} CUDA
9
+:sync: cuda
10
+
11
+```{include} cuda.inc.md
12
+:start-after: "# Installation"
13
+:end-before: "## Requirements"
14
+```
15
+
16
+:::
17
+
18
+:::{tab-item} ROCm
19
+:sync: rocm
20
+
21
+```{include} rocm.inc.md
22
+:start-after: "# Installation"
23
+:end-before: "## Requirements"
24
+```
25
+
26
+:::
27
+
28
+:::{tab-item} XPU
29
+:sync: xpu
30
+
31
+```{include} xpu.inc.md
32
+:start-after: "# Installation"
33
+:end-before: "## Requirements"
34
+```
35
+
36
+:::
37
+
38
+::::
39
+
40
+## Requirements
41
+
42
+- OS: Linux
43
+- Python: 3.9 -- 3.12
44
+
45
+::::{tab-set}
46
+:sync-group: device
47
+
48
+:::{tab-item} CUDA
49
+:sync: cuda
50
+
51
+```{include} cuda.inc.md
52
+:start-after: "## Requirements"
53
+:end-before: "## Set up using Python"
54
+```
55
+
56
+:::
57
+
58
+:::{tab-item} ROCm
59
+:sync: rocm
60
+
61
+```{include} rocm.inc.md
62
+:start-after: "## Requirements"
63
+:end-before: "## Set up using Python"
64
+```
65
+
66
+:::
67
+
68
+:::{tab-item} XPU
69
+:sync: xpu
70
+
71
+```{include} xpu.inc.md
72
+:start-after: "## Requirements"
73
+:end-before: "## Set up using Python"
74
+```
75
+
76
+:::
77
+
78
+::::
79
+
80
+## Set up using Python
81
+
82
+### Create a new Python environment
83
+
84
+```{include} ../python_env_setup.inc.md
85
+```
86
+
87
+::::{tab-set}
88
+:sync-group: device
89
+
90
+:::{tab-item} CUDA
91
+:sync: cuda
92
+
93
+```{include} cuda.inc.md
94
+:start-after: "## Create a new Python environment"
95
+:end-before: "### Pre-built wheels"
96
+```
97
+
98
+:::
99
+
100
+:::{tab-item} ROCm
101
+:sync: rocm
102
+
103
+There is no extra information on creating a new Python environment for this device.
104
+
105
+:::
106
+
107
+:::{tab-item} XPU
108
+:sync: xpu
109
+
110
+There is no extra information on creating a new Python environment for this device.
111
+
112
+:::
113
+
114
+::::
115
+
116
+### Pre-built wheels
117
+
118
+::::{tab-set}
119
+:sync-group: device
120
+
121
+:::{tab-item} CUDA
122
+:sync: cuda
123
+
124
+```{include} cuda.inc.md
125
+:start-after: "### Pre-built wheels"
126
+:end-before: "### Build wheel from source"
127
+```
128
+
129
+:::
130
+
131
+:::{tab-item} ROCm
132
+:sync: rocm
133
+
134
+```{include} rocm.inc.md
135
+:start-after: "### Pre-built wheels"
136
+:end-before: "### Build wheel from source"
137
+```
138
+
139
+:::
140
+
141
+:::{tab-item} XPU
142
+:sync: xpu
143
+
144
+```{include} xpu.inc.md
145
+:start-after: "### Pre-built wheels"
146
+:end-before: "### Build wheel from source"
147
+```
148
+
149
+:::
150
+
151
+::::
152
+
153
+(build-from-source)=
154
+
155
+### Build wheel from source
156
+
157
+::::{tab-set}
158
+:sync-group: device
159
+
160
+:::{tab-item} CUDA
161
+:sync: cuda
162
+
163
+```{include} cuda.inc.md
164
+:start-after: "### Build wheel from source"
165
+:end-before: "## Set up using Docker"
166
+```
167
+
168
+:::
169
+
170
+:::{tab-item} ROCm
171
+:sync: rocm
172
+
173
+```{include} rocm.inc.md
174
+:start-after: "### Build wheel from source"
175
+:end-before: "## Set up using Docker"
176
+```
177
+
178
+:::
179
+
180
+:::{tab-item} XPU
181
+:sync: xpu
182
+
183
+```{include} xpu.inc.md
184
+:start-after: "### Build wheel from source"
185
+:end-before: "## Set up using Docker"
186
+```
187
+
188
+:::
189
+
190
+::::
191
+
192
+## Set up using Docker
193
+
194
+### Pre-built images
195
+
196
+::::{tab-set}
197
+:sync-group: device
198
+
199
+:::{tab-item} CUDA
200
+:sync: cuda
201
+
202
+```{include} cuda.inc.md
203
+:start-after: "### Pre-built images"
204
+:end-before: "### Build image from source"
205
+```
206
+
207
+:::
208
+
209
+:::{tab-item} ROCm
210
+:sync: rocm
211
+
212
+```{include} rocm.inc.md
213
+:start-after: "### Pre-built images"
214
+:end-before: "### Build image from source"
215
+```
216
+
217
+:::
218
+
219
+:::{tab-item} XPU
220
+:sync: xpu
221
+
222
+```{include} xpu.inc.md
223
+:start-after: "### Pre-built images"
224
+:end-before: "### Build image from source"
225
+```
226
+
227
+:::
228
+
229
+::::
230
+
231
+### Build image from source
232
+
233
+::::{tab-set}
234
+:sync-group: device
235
+
236
+:::{tab-item} CUDA
237
+:sync: cuda
238
+
239
+```{include} cuda.inc.md
240
+:start-after: "### Build image from source"
241
+:end-before: "## Supported features"
242
+```
243
+
244
+:::
245
+
246
+:::{tab-item} ROCm
247
+:sync: rocm
248
+
249
+```{include} rocm.inc.md
250
+:start-after: "### Build image from source"
251
+:end-before: "## Supported features"
252
+```
253
+
254
+:::
255
+
256
+:::{tab-item} XPU
257
+:sync: xpu
258
+
259
+```{include} xpu.inc.md
260
+:start-after: "### Build image from source"
261
+:end-before: "## Supported features"
262
+```
263
+
264
+:::
265
+
266
+::::
267
+
268
+## Supported features
269
+
270
+::::{tab-set}
271
+:sync-group: device
272
+
273
+:::{tab-item} CUDA
274
+:sync: cuda
275
+
276
+```{include} cuda.inc.md
277
+:start-after: "## Supported features"
278
+```
279
+
280
+:::
281
+
282
+:::{tab-item} ROCm
283
+:sync: rocm
284
+
285
+```{include} rocm.inc.md
286
+:start-after: "## Supported features"
287
+```
288
+
289
+:::
290
+
291
+:::{tab-item} XPU
292
+:sync: xpu
293
+
294
+```{include} xpu.inc.md
295
+:start-after: "## Supported features"
296
+```
297
+
298
+:::
299
+
300
+::::
docs/source/getting_started/installation/gpu/rocm.inc.md
renamed
+75
-72
@@ -1,82 +1,19 @@
1
-(installation-rocm)=
2
-
3
-# Installation for ROCm
1
+# Installation
2
3
vLLM supports AMD GPUs with ROCm 6.2.
4
5
## Requirements
6
9
-- OS: Linux
10
-- Python: 3.9 -- 3.12
7
- GPU: MI200s (gfx90a), MI300 (gfx942), Radeon RX 7900 series (gfx1100)
8
- ROCm 6.2
9
14
-Installation options:
15
-
16
-1. [Build from source with docker](#build-from-source-docker-rocm)
17
-2. [Build from source](#build-from-source-rocm)
18
-
19
-(build-from-source-docker-rocm)=
20
-
21
-## Option 1: Build from source with docker (recommended)
22
-
23
-You can build and install vLLM from source.
24
-
25
-First, build a docker image from <gh-file:Dockerfile.rocm> and launch a docker container from the image.
26
-It is important that the user kicks off the docker build using buildkit. Either the user put DOCKER_BUILDKIT=1 as environment variable when calling docker build command, or the user needs to setup buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:
27
-
28
-```console
29
-{
30
- "features": {
31
- "buildkit": true
32
- }
33
-}
34
-```
35
-
36
-<gh-file:Dockerfile.rocm> uses ROCm 6.2 by default, but also supports ROCm 5.7, 6.0 and 6.1 in older vLLM branches.
37
-It provides flexibility to customize the build of docker image using the following arguments:
38
-
39
-- `BASE_IMAGE`: specifies the base image used when running `docker build`, specifically the PyTorch on ROCm base image.
40
-- `BUILD_FA`: specifies whether to build CK flash-attention. The default is 1. For [Radeon RX 7900 series (gfx1100)](https://rocm.docs.amd.com/projects/radeon/en/latest/index.html), this should be set to 0 before flash-attention supports this target.
41
-- `FX_GFX_ARCHS`: specifies the GFX architecture that is used to build CK flash-attention, for example, `gfx90a;gfx942` for MI200 and MI300. The default is `gfx90a;gfx942`
42
-- `FA_BRANCH`: specifies the branch used to build the CK flash-attention in [ROCm's flash-attention repo](https://github.com/ROCmSoftwarePlatform/flash-attention). The default is `ae7928c`
43
-- `BUILD_TRITON`: specifies whether to build triton flash-attention. The default value is 1.
44
-
45
-Their values can be passed in when running `docker build` with `--build-arg` options.
46
-
47
-To build vllm on ROCm 6.2 for MI200 and MI300 series, you can use the default:
48
-
49
-```console
50
-DOCKER_BUILDKIT=1 docker build -f Dockerfile.rocm -t vllm-rocm .
51
-```
10
+## Set up using Python
11
53
-To build vllm on ROCm 6.2 for Radeon RX7900 series (gfx1100), you should specify `BUILD_FA` as below:
12
+### Pre-built wheels
13
55
-```console
56
-DOCKER_BUILDKIT=1 docker build --build-arg BUILD_FA="0" -f Dockerfile.rocm -t vllm-rocm .
57
-```
14
+Currently, there are no pre-built ROCm wheels.
15
59
-To run the above docker image `vllm-rocm`, use the below command:
60
-
61
-```console
62
-$ docker run -it \
63
- --network=host \
64
- --group-add=video \
65
- --ipc=host \
66
- --cap-add=SYS_PTRACE \
67
- --security-opt seccomp=unconfined \
68
- --device /dev/kfd \
69
- --device /dev/dri \
70
- -v <path/to/model>:/app/model \
71
- vllm-rocm \
72
- bash
73
-```
74
-
75
-Where the `<path/to/model>` is the location where the model is stored, for example, the weights for llama2 or llama3 models.
76
-
77
-(build-from-source-rocm)=
78
-
79
-## Option 2: Build from source
16
+### Build wheel from source
17
18
0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
19
@@ -157,7 +94,73 @@ Where the `<path/to/model>` is the location where the model is stored, for examp
94
- The ROCm version of PyTorch, ideally, should match the ROCm driver version.
95
```
96
160
- ```{tip}
161
- - For MI300x (gfx942) users, to achieve optimal performance, please refer to [MI300x tuning guide](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html) for performance optimization and tuning tips on system and workflow level.
162
- For vLLM, please refer to [vLLM performance optimization](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/workload.html#vllm-performance-optimization).
163
- ```
97
+```{tip}
98
+- For MI300x (gfx942) users, to achieve optimal performance, please refer to [MI300x tuning guide](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html) for performance optimization and tuning tips on system and workflow level.
99
+ For vLLM, please refer to [vLLM performance optimization](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/workload.html#vllm-performance-optimization).
100
+```
101
+
102
+## Set up using Docker
103
+
104
+### Pre-built images
105
+
106
+Currently, there are no pre-built ROCm images.
107
+
108
+### Build image from source
109
+
110
+Building the Docker image from source is the recommended way to use vLLM with ROCm.
111
+
112
+First, build a docker image from <gh-file:Dockerfile.rocm> and launch a docker container from the image.
113
+It is important that the user kicks off the docker build using buildkit. Either the user put DOCKER_BUILDKIT=1 as environment variable when calling docker build command, or the user needs to setup buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:
114
+
115
+```console
116
+{
117
+ "features": {
118
+ "buildkit": true
119
+ }
120
+}
121
+```
122
+
123
+<gh-file:Dockerfile.rocm> uses ROCm 6.2 by default, but also supports ROCm 5.7, 6.0 and 6.1 in older vLLM branches.
124
+It provides flexibility to customize the build of docker image using the following arguments:
125
+
126
+- `BASE_IMAGE`: specifies the base image used when running `docker build`, specifically the PyTorch on ROCm base image.
127
+- `BUILD_FA`: specifies whether to build CK flash-attention. The default is 1. For [Radeon RX 7900 series (gfx1100)](https://rocm.docs.amd.com/projects/radeon/en/latest/index.html), this should be set to 0 before flash-attention supports this target.
128
+- `FX_GFX_ARCHS`: specifies the GFX architecture that is used to build CK flash-attention, for example, `gfx90a;gfx942` for MI200 and MI300. The default is `gfx90a;gfx942`
129
+- `FA_BRANCH`: specifies the branch used to build the CK flash-attention in [ROCm's flash-attention repo](https://github.com/ROCmSoftwarePlatform/flash-attention). The default is `ae7928c`
130
+- `BUILD_TRITON`: specifies whether to build triton flash-attention. The default value is 1.
131
+
132
+Their values can be passed in when running `docker build` with `--build-arg` options.
133
+
134
+To build vllm on ROCm 6.2 for MI200 and MI300 series, you can use the default:
135
+
136
+```console
137
+DOCKER_BUILDKIT=1 docker build -f Dockerfile.rocm -t vllm-rocm .
138
+```
139
+
140
+To build vllm on ROCm 6.2 for Radeon RX7900 series (gfx1100), you should specify `BUILD_FA` as below:
141
+
142
+```console
143
+DOCKER_BUILDKIT=1 docker build --build-arg BUILD_FA="0" -f Dockerfile.rocm -t vllm-rocm .
144
+```
145
+
146
+To run the above docker image `vllm-rocm`, use the below command:
147
+
148
+```console
149
+docker run -it \
150
+ --network=host \
151
+ --group-add=video \
152
+ --ipc=host \
153
+ --cap-add=SYS_PTRACE \
154
+ --security-opt seccomp=unconfined \
155
+ --device /dev/kfd \
156
+ --device /dev/dri \
157
+ -v <path/to/model>:/app/model \
158
+ vllm-rocm \
159
+ bash
160
+```
161
+
162
+Where the `<path/to/model>` is the location where the model is stored, for example, the weights for llama2 or llama3 models.
163
+
164
+## Supported features
165
+
166
+See <project:#feature-x-hardware> compatibility matrix for feature support information.
docs/source/getting_started/installation/gpu/xpu.inc.md
renamed
+24
-27
@@ -1,40 +1,19 @@
1
-(installation-xpu)=
2
-
3
-# Installation for XPUs
1
+# Installation
2
3
vLLM initially supports basic model inferencing and serving on Intel GPU platform.
4
7
-Table of contents:
8
-
9
-1. [Requirements](#xpu-backend-requirements)
10
-2. [Quick start using Dockerfile](#xpu-backend-quick-start-dockerfile)
11
-3. [Build from source](#build-xpu-backend-from-source)
12
-
13
-(xpu-backend-requirements)=
14
-
5
## Requirements
6
17
-- OS: Linux
7
- Supported Hardware: Intel Data Center GPU, Intel ARC GPU
8
- OneAPI requirements: oneAPI 2024.2
9
21
-(xpu-backend-quick-start-dockerfile)=
10
+## Set up using Python
11
23
-## Quick start using Dockerfile
24
-
25
-```console
26
-$ docker build -f Dockerfile.xpu -t vllm-xpu-env --shm-size=4g .
27
-$ docker run -it \
28
- --rm \
29
- --network=host \
30
- --device /dev/dri \
31
- -v /dev/dri/by-path:/dev/dri/by-path \
32
- vllm-xpu-env
33
-```
12
+### Pre-built wheels
13
35
-(build-xpu-backend-from-source)=
14
+Currently, there are no pre-built XPU wheels.
15
37
-## Build from source
16
+### Build wheel from source
17
18
- First, install required driver and intel OneAPI 2024.2 or later.
19
- Second, install Python packages for vLLM XPU backend building:
@@ -56,7 +35,25 @@ VLLM_TARGET_DEVICE=xpu python setup.py install
35
type will be supported in the future.
36
```
37
59
-## Distributed inference and serving
38
+## Set up using Docker
39
+
40
+### Pre-built images
41
+
42
+Currently, there are no pre-built XPU images.
43
+
44
+### Build image from source
45
+
46
+```console
47
+$ docker build -f Dockerfile.xpu -t vllm-xpu-env --shm-size=4g .
48
+$ docker run -it \
49
+ --rm \
50
+ --network=host \
51
+ --device /dev/dri \
52
+ -v /dev/dri/by-path:/dev/dri/by-path \
53
+ vllm-xpu-env
54
+```
55
+
56
+## Supported features
57
58
XPU platform supports tensor-parallel inference/serving and also supports pipeline parallel as a beta feature for online serving. We requires Ray as the distributed runtime backend. For example, a reference execution likes following:
59
docs/source/getting_started/installation/index.md
+3
-10
@@ -7,14 +7,7 @@ vLLM supports the following hardware platforms:
7
```{toctree}
8
:maxdepth: 1
9
10
-gpu-cuda
11
-gpu-rocm
12
-cpu-x86
13
-cpu-arm
14
-cpu-apple
15
-hpu-gaudi
16
-tpu
17
-xpu
18
-openvino
19
-neuron
10
+gpu/index
11
+cpu/index
12
+ai_accelerator/index
13
```
docs/source/getting_started/installation/python_env_setup.inc.md
new
+19
@@ -0,0 +1,19 @@
1
+You can create a new Python environment using `conda`:
2
+
3
+```console
4
+# (Recommended) Create a new conda environment.
5
+conda create -n myenv python=3.12 -y
6
+conda activate myenv
7
+```
8
+
9
+```{note}
10
+[PyTorch has deprecated the conda release channel](https://github.com/pytorch/pytorch/issues/138506). If you use `conda`, please only use it to create Python environment rather than installing packages.
11
+```
12
+
13
+Or you can create a new Python environment using [uv](https://docs.astral.sh/uv/), a very fast Python environment manager. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment using the following command:
14
+
15
+```console
16
+# (Recommended) Create a new uv environment. Use `--seed` to install `pip` and `setuptools` in the environment.
17
+uv venv myenv --python 3.12 --seed
18
+source myenv/bin/activate
19
+```