A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 00:45 UTC
[Doc] Improve installation signposting (#12575)
- Make device tab names more explicit - Add comprehensive list of devices to https://docs.vllm.ai/en/latest/getting_started/installation/index.html - Add `attention` blocks to the intro of all devices that don't have pre-built wheels/images --------- Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Jan 31, 2025 at 23:38 UTC
60808bd4c7a27b6d28f82657e38a5b303f7534a9
13 files changed
+111
-59
docs/source/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md
+4
@@ -2,6 +2,10 @@
2
3
This tab provides instructions on running vLLM with Intel Gaudi devices.
4
5
+:::{attention}
6
+There are no pre-built wheels or images for this device, so you must build vLLM from source.
7
+:::
8
+
9
## Requirements
10
11
- OS: Ubuntu 22.04 LTS
docs/source/getting_started/installation/ai_accelerator/index.md
+17
-16
@@ -5,7 +5,8 @@ vLLM is a Python library that supports the following AI accelerators. Select you
5
:::::{tab-set}
6
:sync-group: device
7
8
-::::{tab-item} TPU
8
+::::{tab-item} Google TPU
9
+:selected:
10
:sync: tpu
11
12
:::{include} tpu.inc.md
@@ -25,7 +26,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
26
27
::::
28
28
-::::{tab-item} Neuron
29
+::::{tab-item} AWS Neuron
30
:sync: neuron
31
32
:::{include} neuron.inc.md
@@ -52,7 +53,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
53
:::::{tab-set}
54
:sync-group: device
55
55
-::::{tab-item} TPU
56
+::::{tab-item} Google TPU
57
:sync: tpu
58
59
:::{include} tpu.inc.md
@@ -72,7 +73,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
73
74
::::
75
75
-::::{tab-item} Neuron
76
+::::{tab-item} AWS Neuron
77
:sync: neuron
78
79
:::{include} neuron.inc.md
@@ -99,7 +100,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
100
:::::{tab-set}
101
:sync-group: device
102
102
-::::{tab-item} TPU
103
+::::{tab-item} Google TPU
104
:sync: tpu
105
106
:::{include} tpu.inc.md
@@ -119,7 +120,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
120
121
::::
122
122
-::::{tab-item} Neuron
123
+::::{tab-item} AWS Neuron
124
:sync: neuron
125
126
:::{include} neuron.inc.md
@@ -146,7 +147,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
147
:::::{tab-set}
148
:sync-group: device
149
149
-::::{tab-item} TPU
150
+::::{tab-item} Google TPU
151
:sync: tpu
152
153
:::{include} tpu.inc.md
@@ -166,7 +167,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
167
168
::::
169
169
-::::{tab-item} Neuron
170
+::::{tab-item} AWS Neuron
171
:sync: neuron
172
173
:::{include} neuron.inc.md
@@ -193,7 +194,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
194
:::::{tab-set}
195
:sync-group: device
196
196
-::::{tab-item} TPU
197
+::::{tab-item} Google TPU
198
:sync: tpu
199
200
:::{include} tpu.inc.md
@@ -213,7 +214,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
214
215
::::
216
216
-::::{tab-item} Neuron
217
+::::{tab-item} AWS Neuron
218
:sync: neuron
219
220
:::{include} neuron.inc.md
@@ -242,7 +243,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
243
:::::{tab-set}
244
:sync-group: device
245
245
-::::{tab-item} TPU
246
+::::{tab-item} Google TPU
247
:sync: tpu
248
249
:::{include} tpu.inc.md
@@ -262,7 +263,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
263
264
::::
265
265
-::::{tab-item} Neuron
266
+::::{tab-item} AWS Neuron
267
:sync: neuron
268
269
:::{include} neuron.inc.md
@@ -289,7 +290,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
290
:::::{tab-set}
291
:sync-group: device
292
292
-::::{tab-item} TPU
293
+::::{tab-item} Google TPU
294
:sync: tpu
295
296
:::{include} tpu.inc.md
@@ -309,7 +310,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
310
311
::::
312
312
-::::{tab-item} Neuron
313
+::::{tab-item} AWS Neuron
314
:sync: neuron
315
316
:::{include} neuron.inc.md
@@ -336,7 +337,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
337
:::::{tab-set}
338
:sync-group: device
339
339
-::::{tab-item} TPU
340
+::::{tab-item} Google TPU
341
:sync: tpu
342
343
:::{include} tpu.inc.md
@@ -354,7 +355,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
355
356
::::
357
357
-::::{tab-item} Neuron
358
+::::{tab-item} AWS Neuron
359
:sync: neuron
360
361
:::{include} neuron.inc.md
docs/source/getting_started/installation/ai_accelerator/neuron.inc.md
+4
@@ -4,6 +4,10 @@ vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Infere
4
Paged Attention and Chunked Prefill are currently in development and will be available soon.
5
Data types currently supported in Neuron SDK are FP16 and BF16.
6
7
+:::{attention}
8
+There are no pre-built wheels or images for this device, so you must build vLLM from source.
9
+:::
10
+
11
## Requirements
12
13
- OS: Linux
docs/source/getting_started/installation/ai_accelerator/openvino.inc.md
+4
@@ -2,6 +2,10 @@
2
3
vLLM powered by OpenVINO supports all LLM models from [vLLM supported models list](#supported-models) and can perform optimal model serving on all x86-64 CPUs with, at least, AVX2 support, as well as on both integrated and discrete Intel® GPUs ([the list of supported GPUs](https://docs.openvino.ai/2024/about-openvino/release-notes-openvino/system-requirements.html#gpu)).
4
5
+:::{attention}
6
+There are no pre-built wheels or images for this device, so you must build vLLM from source.
7
+:::
8
+
9
## Requirements
10
11
- OS: Linux
docs/source/getting_started/installation/ai_accelerator/tpu.inc.md
+4
@@ -30,6 +30,10 @@ For TPU pricing information, see [Cloud TPU pricing](https://cloud.google.com/tp
30
You may need additional persistent storage for your TPU VMs. For more
31
information, see [Storage options for Cloud TPU data](https://cloud.devsite.corp.google.com/tpu/docs/storage-options).
32
33
+:::{attention}
34
+There are no pre-built wheels for this device, so you must either use the pre-built Docker image or build vLLM from source.
35
+:::
36
+
37
## Requirements
38
39
- Google Cloud TPU VM
docs/source/getting_started/installation/cpu/apple.inc.md
+4
@@ -4,6 +4,10 @@ vLLM has experimental support for macOS with Apple silicon. For now, users shall
4
5
Currently the CPU implementation for macOS supports FP32 and FP16 datatypes.
6
7
+:::{attention}
8
+There are no pre-built wheels or images for this device, so you must build vLLM from source.
9
+:::
10
+
11
## Requirements
12
13
- OS: `macOS Sonoma` or later
docs/source/getting_started/installation/cpu/arm.inc.md
+4
@@ -4,6 +4,10 @@ vLLM has been adapted to work on ARM64 CPUs with NEON support, leveraging the CP
4
5
ARM CPU backend currently supports Float32, FP16 and BFloat16 datatypes.
6
7
+:::{attention}
8
+There are no pre-built wheels or images for this device, so you must build vLLM from source.
9
+:::
10
+
11
## Requirements
12
13
- OS: Linux
docs/source/getting_started/installation/cpu/index.md
+7
-6
@@ -5,7 +5,8 @@ vLLM is a Python library that supports the following CPU variants. Select your C
5
:::::{tab-set}
6
:sync-group: device
7
8
-::::{tab-item} x86
8
+::::{tab-item} Intel/AMD x86
9
+:selected:
10
:sync: x86
11
12
:::{include} x86.inc.md
@@ -15,7 +16,7 @@ vLLM is a Python library that supports the following CPU variants. Select your C
16
17
::::
18
18
-::::{tab-item} ARM
19
+::::{tab-item} ARM AArch64
20
:sync: arm
21
22
:::{include} arm.inc.md
@@ -44,7 +45,7 @@ vLLM is a Python library that supports the following CPU variants. Select your C
45
:::::{tab-set}
46
:sync-group: device
47
47
-::::{tab-item} x86
48
+::::{tab-item} Intel/AMD x86
49
:sync: x86
50
51
:::{include} x86.inc.md
@@ -54,7 +55,7 @@ vLLM is a Python library that supports the following CPU variants. Select your C
55
56
::::
57
57
-::::{tab-item} ARM
58
+::::{tab-item} ARM AArch64
59
:sync: arm
60
61
:::{include} arm.inc.md
@@ -92,7 +93,7 @@ Currently, there are no pre-built CPU wheels.
93
:::::{tab-set}
94
:sync-group: device
95
95
-::::{tab-item} x86
96
+::::{tab-item} Intel/AMD x86
97
:sync: x86
98
99
:::{include} x86.inc.md
@@ -102,7 +103,7 @@ Currently, there are no pre-built CPU wheels.
103
104
::::
105
105
-::::{tab-item} ARM
106
+::::{tab-item} ARM AArch64
107
:sync: arm
108
109
:::{include} arm.inc.md
docs/source/getting_started/installation/cpu/x86.inc.md
+8
-4
@@ -2,12 +2,20 @@
2
3
vLLM initially supports basic model inferencing and serving on x86 CPU platform, with data types FP32, FP16 and BF16.
4
5
+:::{attention}
6
+There are no pre-built wheels or images for this device, so you must build vLLM from source.
7
+:::
8
+
9
## Requirements
10
11
- OS: Linux
12
- Compiler: `gcc/g++ >= 12.3.0` (optional, recommended)
13
- Instruction Set Architecture (ISA): AVX512 (optional, recommended)
14
15
+:::{tip}
16
+[Intel Extension for PyTorch (IPEX)](https://github.com/intel/intel-extension-for-pytorch) extends PyTorch with up-to-date features optimizations for an extra performance boost on Intel hardware.
17
+:::
18
+
19
## Set up using Python
20
21
### Pre-built wheels
@@ -29,7 +37,3 @@ vLLM initially supports basic model inferencing and serving on x86 CPU platform,
37
### Build image from source
38
39
## Extra information
32
-
33
-## Intel Extension for PyTorch
34
-
35
-- [Intel Extension for PyTorch (IPEX)](https://github.com/intel/intel-extension-for-pytorch) extends PyTorch with up-to-date features optimizations for an extra performance boost on Intel hardware.
docs/source/getting_started/installation/gpu/index.md
+25
-24
@@ -5,7 +5,8 @@ vLLM is a Python library that supports the following GPU variants. Select your G
5
:::::{tab-set}
6
:sync-group: device
7
8
-::::{tab-item} CUDA
8
+::::{tab-item} NVIDIA CUDA
9
+:selected:
10
:sync: cuda
11
12
:::{include} cuda.inc.md
@@ -15,7 +16,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
16
17
::::
18
18
-::::{tab-item} ROCm
19
+::::{tab-item} AMD ROCm
20
:sync: rocm
21
22
:::{include} rocm.inc.md
@@ -25,7 +26,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
26
27
::::
28
28
-::::{tab-item} XPU
29
+::::{tab-item} Intel XPU
30
:sync: xpu
31
32
:::{include} xpu.inc.md
@@ -45,7 +46,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
46
:::::{tab-set}
47
:sync-group: device
48
48
-::::{tab-item} CUDA
49
+::::{tab-item} NVIDIA CUDA
50
:sync: cuda
51
52
:::{include} cuda.inc.md
@@ -55,7 +56,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
56
57
::::
58
58
-::::{tab-item} ROCm
59
+::::{tab-item} AMD ROCm
60
:sync: rocm
61
62
:::{include} rocm.inc.md
@@ -65,7 +66,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
66
67
::::
68
68
-::::{tab-item} XPU
69
+::::{tab-item} Intel XPU
70
:sync: xpu
71
72
:::{include} xpu.inc.md
@@ -87,7 +88,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
88
:::::{tab-set}
89
:sync-group: device
90
90
-::::{tab-item} CUDA
91
+::::{tab-item} NVIDIA CUDA
92
:sync: cuda
93
94
:::{include} cuda.inc.md
@@ -97,14 +98,14 @@ vLLM is a Python library that supports the following GPU variants. Select your G
98
99
::::
100
100
-::::{tab-item} ROCm
101
+::::{tab-item} AMD ROCm
102
:sync: rocm
103
104
There is no extra information on creating a new Python environment for this device.
105
106
::::
107
107
-::::{tab-item} XPU
108
+::::{tab-item} Intel XPU
109
:sync: xpu
110
111
There is no extra information on creating a new Python environment for this device.
@@ -118,7 +119,7 @@ There is no extra information on creating a new Python environment for this devi
119
:::::{tab-set}
120
:sync-group: device
121
121
-::::{tab-item} CUDA
122
+::::{tab-item} NVIDIA CUDA
123
:sync: cuda
124
125
:::{include} cuda.inc.md
@@ -128,7 +129,7 @@ There is no extra information on creating a new Python environment for this devi
129
130
::::
131
131
-::::{tab-item} ROCm
132
+::::{tab-item} AMD ROCm
133
:sync: rocm
134
135
:::{include} rocm.inc.md
@@ -138,7 +139,7 @@ There is no extra information on creating a new Python environment for this devi
139
140
::::
141
141
-::::{tab-item} XPU
142
+::::{tab-item} Intel XPU
143
:sync: xpu
144
145
:::{include} xpu.inc.md
@@ -157,7 +158,7 @@ There is no extra information on creating a new Python environment for this devi
158
:::::{tab-set}
159
:sync-group: device
160
160
-::::{tab-item} CUDA
161
+::::{tab-item} NVIDIA CUDA
162
:sync: cuda
163
164
:::{include} cuda.inc.md
@@ -167,7 +168,7 @@ There is no extra information on creating a new Python environment for this devi
168
169
::::
170
170
-::::{tab-item} ROCm
171
+::::{tab-item} AMD ROCm
172
:sync: rocm
173
174
:::{include} rocm.inc.md
@@ -177,7 +178,7 @@ There is no extra information on creating a new Python environment for this devi
178
179
::::
180
180
-::::{tab-item} XPU
181
+::::{tab-item} Intel XPU
182
:sync: xpu
183
184
:::{include} xpu.inc.md
@@ -196,7 +197,7 @@ There is no extra information on creating a new Python environment for this devi
197
:::::{tab-set}
198
:sync-group: device
199
199
-::::{tab-item} CUDA
200
+::::{tab-item} NVIDIA CUDA
201
:sync: cuda
202
203
:::{include} cuda.inc.md
@@ -206,7 +207,7 @@ There is no extra information on creating a new Python environment for this devi
207
208
::::
209
209
-::::{tab-item} ROCm
210
+::::{tab-item} AMD ROCm
211
:sync: rocm
212
213
:::{include} rocm.inc.md
@@ -216,7 +217,7 @@ There is no extra information on creating a new Python environment for this devi
217
218
::::
219
219
-::::{tab-item} XPU
220
+::::{tab-item} Intel XPU
221
:sync: xpu
222
223
:::{include} xpu.inc.md
@@ -233,7 +234,7 @@ There is no extra information on creating a new Python environment for this devi
234
:::::{tab-set}
235
:sync-group: device
236
236
-::::{tab-item} CUDA
237
+::::{tab-item} NVIDIA CUDA
238
:sync: cuda
239
240
:::{include} cuda.inc.md
@@ -243,7 +244,7 @@ There is no extra information on creating a new Python environment for this devi
244
245
::::
246
246
-::::{tab-item} ROCm
247
+::::{tab-item} AMD ROCm
248
:sync: rocm
249
250
:::{include} rocm.inc.md
@@ -253,7 +254,7 @@ There is no extra information on creating a new Python environment for this devi
254
255
::::
256
256
-::::{tab-item} XPU
257
+::::{tab-item} Intel XPU
258
:sync: xpu
259
260
:::{include} xpu.inc.md
@@ -270,7 +271,7 @@ There is no extra information on creating a new Python environment for this devi
271
:::::{tab-set}
272
:sync-group: device
273
273
-::::{tab-item} CUDA
274
+::::{tab-item} NVIDIA CUDA
275
:sync: cuda
276
277
:::{include} cuda.inc.md
@@ -279,7 +280,7 @@ There is no extra information on creating a new Python environment for this devi
280
281
::::
282
282
-::::{tab-item} ROCm
283
+::::{tab-item} AMD ROCm
284
:sync: rocm
285
286
:::{include} rocm.inc.md
@@ -288,7 +289,7 @@ There is no extra information on creating a new Python environment for this devi
289
290
::::
291
291
-::::{tab-item} XPU
292
+::::{tab-item} Intel XPU
293
:sync: xpu
294
295
:::{include} xpu.inc.md
docs/source/getting_started/installation/gpu/rocm.inc.md
+11
-9
@@ -2,6 +2,10 @@
2
3
vLLM supports AMD GPUs with ROCm 6.2.
4
5
+:::{attention}
6
+There are no pre-built wheels for this device, so you must either use the pre-built Docker image or build vLLM from source.
7
+:::
8
+
9
## Requirements
10
11
- GPU: MI200s (gfx90a), MI300 (gfx942), Radeon RX 7900 series (gfx1100)
@@ -13,14 +17,6 @@ vLLM supports AMD GPUs with ROCm 6.2.
17
18
Currently, there are no pre-built ROCm wheels.
19
16
-However, the [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offers a prebuilt, optimized
17
-docker image designed for validating inference performance on the AMD Instinct™ MI300X accelerator.
18
-
19
-:::{tip}
20
-Please check [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/performance-validation/mi300x/vllm-benchmark.html)
21
-for instructions on how to use this prebuilt docker image.
22
-:::
23
-
20
### Build wheel from source
21
22
0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
@@ -112,7 +108,13 @@ for instructions on how to use this prebuilt docker image.
108
109
### Pre-built images
110
115
-Currently, there are no pre-built ROCm images.
111
+The [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offers a prebuilt, optimized
112
+docker image designed for validating inference performance on the AMD Instinct™ MI300X accelerator.
113
+
114
+:::{tip}
115
+Please check [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/performance-validation/mi300x/vllm-benchmark.html)
116
+for instructions on how to use this prebuilt docker image.
117
+:::
118
119
### Build image from source
120
docs/source/getting_started/installation/gpu/xpu.inc.md
+4
@@ -2,6 +2,10 @@
2
3
vLLM initially supports basic model inferencing and serving on Intel GPU platform.
4
5
+:::{attention}
6
+There are no pre-built wheels or images for this device, so you must build vLLM from source.
7
+:::
8
+
9
## Requirements
10
11
- Supported Hardware: Intel Data Center GPU, Intel ARC GPU
docs/source/getting_started/installation/index.md
+15
@@ -6,8 +6,23 @@ vLLM supports the following hardware platforms:
6
7
:::{toctree}
8
:maxdepth: 1
9
+:hidden:
10
11
gpu/index
12
cpu/index
13
ai_accelerator/index
14
:::
15
+
16
+- <project:gpu/index.md>
17
+ - NVIDIA CUDA
18
+ - AMD ROCm
19
+ - Intel XPU
20
+- <project:cpu/index.md>
21
+ - Intel/AMD x86
22
+ - ARM AArch64
23
+ - Apple silicon
24
+- <project:ai_accelerator/index.md>
25
+ - Google TPU
26
+ - Intel Gaudi
27
+ - AWS Neuron
28
+ - OpenVINO