@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 60808bd4c7
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 00:45 UTC

[Doc] Improve installation signposting (#12575)

- Make device tab names more explicit - Add comprehensive list of devices to https://docs.vllm.ai/en/latest/getting_started/installation/index.html - Add `attention` blocks to the intro of all devices that don't have pre-built wheels/images --------- Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Harry Mellor committed Jan 31, 2025 at 23:38 UTC 60808bd4c7a27b6d28f82657e38a5b303f7534a9
13 files changed +111 -59
docs/source/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md
+4
@@ -2,6 +2,10 @@
2
3 This tab provides instructions on running vLLM with Intel Gaudi devices.
4
5 +:::{attention}
6 +There are no pre-built wheels or images for this device, so you must build vLLM from source.
7 +:::
8 +
9 ## Requirements
10
11 - OS: Ubuntu 22.04 LTS
docs/source/getting_started/installation/ai_accelerator/index.md
+17 -16
@@ -5,7 +5,8 @@ vLLM is a Python library that supports the following AI accelerators. Select you
5 :::::{tab-set}
6 :sync-group: device
7
8 -::::{tab-item} TPU
8 +::::{tab-item} Google TPU
9 +:selected:
10 :sync: tpu
11
12 :::{include} tpu.inc.md
@@ -25,7 +26,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
26
27 ::::
28
28 -::::{tab-item} Neuron
29 +::::{tab-item} AWS Neuron
30 :sync: neuron
31
32 :::{include} neuron.inc.md
@@ -52,7 +53,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
53 :::::{tab-set}
54 :sync-group: device
55
55 -::::{tab-item} TPU
56 +::::{tab-item} Google TPU
57 :sync: tpu
58
59 :::{include} tpu.inc.md
@@ -72,7 +73,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
73
74 ::::
75
75 -::::{tab-item} Neuron
76 +::::{tab-item} AWS Neuron
77 :sync: neuron
78
79 :::{include} neuron.inc.md
@@ -99,7 +100,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
100 :::::{tab-set}
101 :sync-group: device
102
102 -::::{tab-item} TPU
103 +::::{tab-item} Google TPU
104 :sync: tpu
105
106 :::{include} tpu.inc.md
@@ -119,7 +120,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
120
121 ::::
122
122 -::::{tab-item} Neuron
123 +::::{tab-item} AWS Neuron
124 :sync: neuron
125
126 :::{include} neuron.inc.md
@@ -146,7 +147,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
147 :::::{tab-set}
148 :sync-group: device
149
149 -::::{tab-item} TPU
150 +::::{tab-item} Google TPU
151 :sync: tpu
152
153 :::{include} tpu.inc.md
@@ -166,7 +167,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
167
168 ::::
169
169 -::::{tab-item} Neuron
170 +::::{tab-item} AWS Neuron
171 :sync: neuron
172
173 :::{include} neuron.inc.md
@@ -193,7 +194,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
194 :::::{tab-set}
195 :sync-group: device
196
196 -::::{tab-item} TPU
197 +::::{tab-item} Google TPU
198 :sync: tpu
199
200 :::{include} tpu.inc.md
@@ -213,7 +214,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
214
215 ::::
216
216 -::::{tab-item} Neuron
217 +::::{tab-item} AWS Neuron
218 :sync: neuron
219
220 :::{include} neuron.inc.md
@@ -242,7 +243,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
243 :::::{tab-set}
244 :sync-group: device
245
245 -::::{tab-item} TPU
246 +::::{tab-item} Google TPU
247 :sync: tpu
248
249 :::{include} tpu.inc.md
@@ -262,7 +263,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
263
264 ::::
265
265 -::::{tab-item} Neuron
266 +::::{tab-item} AWS Neuron
267 :sync: neuron
268
269 :::{include} neuron.inc.md
@@ -289,7 +290,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
290 :::::{tab-set}
291 :sync-group: device
292
292 -::::{tab-item} TPU
293 +::::{tab-item} Google TPU
294 :sync: tpu
295
296 :::{include} tpu.inc.md
@@ -309,7 +310,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
310
311 ::::
312
312 -::::{tab-item} Neuron
313 +::::{tab-item} AWS Neuron
314 :sync: neuron
315
316 :::{include} neuron.inc.md
@@ -336,7 +337,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
337 :::::{tab-set}
338 :sync-group: device
339
339 -::::{tab-item} TPU
340 +::::{tab-item} Google TPU
341 :sync: tpu
342
343 :::{include} tpu.inc.md
@@ -354,7 +355,7 @@ vLLM is a Python library that supports the following AI accelerators. Select you
355
356 ::::
357
357 -::::{tab-item} Neuron
358 +::::{tab-item} AWS Neuron
359 :sync: neuron
360
361 :::{include} neuron.inc.md
docs/source/getting_started/installation/ai_accelerator/neuron.inc.md
+4
@@ -4,6 +4,10 @@ vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Infere
4 Paged Attention and Chunked Prefill are currently in development and will be available soon.
5 Data types currently supported in Neuron SDK are FP16 and BF16.
6
7 +:::{attention}
8 +There are no pre-built wheels or images for this device, so you must build vLLM from source.
9 +:::
10 +
11 ## Requirements
12
13 - OS: Linux
docs/source/getting_started/installation/ai_accelerator/openvino.inc.md
+4
@@ -2,6 +2,10 @@
2
3 vLLM powered by OpenVINO supports all LLM models from [vLLM supported models list](#supported-models) and can perform optimal model serving on all x86-64 CPUs with, at least, AVX2 support, as well as on both integrated and discrete Intel® GPUs ([the list of supported GPUs](https://docs.openvino.ai/2024/about-openvino/release-notes-openvino/system-requirements.html#gpu)).
4
5 +:::{attention}
6 +There are no pre-built wheels or images for this device, so you must build vLLM from source.
7 +:::
8 +
9 ## Requirements
10
11 - OS: Linux
docs/source/getting_started/installation/ai_accelerator/tpu.inc.md
+4
@@ -30,6 +30,10 @@ For TPU pricing information, see [Cloud TPU pricing](https://cloud.google.com/tp
30 You may need additional persistent storage for your TPU VMs. For more
31 information, see [Storage options for Cloud TPU data](https://cloud.devsite.corp.google.com/tpu/docs/storage-options).
32
33 +:::{attention}
34 +There are no pre-built wheels for this device, so you must either use the pre-built Docker image or build vLLM from source.
35 +:::
36 +
37 ## Requirements
38
39 - Google Cloud TPU VM
docs/source/getting_started/installation/cpu/apple.inc.md
+4
@@ -4,6 +4,10 @@ vLLM has experimental support for macOS with Apple silicon. For now, users shall
4
5 Currently the CPU implementation for macOS supports FP32 and FP16 datatypes.
6
7 +:::{attention}
8 +There are no pre-built wheels or images for this device, so you must build vLLM from source.
9 +:::
10 +
11 ## Requirements
12
13 - OS: `macOS Sonoma` or later
docs/source/getting_started/installation/cpu/arm.inc.md
+4
@@ -4,6 +4,10 @@ vLLM has been adapted to work on ARM64 CPUs with NEON support, leveraging the CP
4
5 ARM CPU backend currently supports Float32, FP16 and BFloat16 datatypes.
6
7 +:::{attention}
8 +There are no pre-built wheels or images for this device, so you must build vLLM from source.
9 +:::
10 +
11 ## Requirements
12
13 - OS: Linux
docs/source/getting_started/installation/cpu/index.md
+7 -6
@@ -5,7 +5,8 @@ vLLM is a Python library that supports the following CPU variants. Select your C
5 :::::{tab-set}
6 :sync-group: device
7
8 -::::{tab-item} x86
8 +::::{tab-item} Intel/AMD x86
9 +:selected:
10 :sync: x86
11
12 :::{include} x86.inc.md
@@ -15,7 +16,7 @@ vLLM is a Python library that supports the following CPU variants. Select your C
16
17 ::::
18
18 -::::{tab-item} ARM
19 +::::{tab-item} ARM AArch64
20 :sync: arm
21
22 :::{include} arm.inc.md
@@ -44,7 +45,7 @@ vLLM is a Python library that supports the following CPU variants. Select your C
45 :::::{tab-set}
46 :sync-group: device
47
47 -::::{tab-item} x86
48 +::::{tab-item} Intel/AMD x86
49 :sync: x86
50
51 :::{include} x86.inc.md
@@ -54,7 +55,7 @@ vLLM is a Python library that supports the following CPU variants. Select your C
55
56 ::::
57
57 -::::{tab-item} ARM
58 +::::{tab-item} ARM AArch64
59 :sync: arm
60
61 :::{include} arm.inc.md
@@ -92,7 +93,7 @@ Currently, there are no pre-built CPU wheels.
93 :::::{tab-set}
94 :sync-group: device
95
95 -::::{tab-item} x86
96 +::::{tab-item} Intel/AMD x86
97 :sync: x86
98
99 :::{include} x86.inc.md
@@ -102,7 +103,7 @@ Currently, there are no pre-built CPU wheels.
103
104 ::::
105
105 -::::{tab-item} ARM
106 +::::{tab-item} ARM AArch64
107 :sync: arm
108
109 :::{include} arm.inc.md
docs/source/getting_started/installation/cpu/x86.inc.md
+8 -4
@@ -2,12 +2,20 @@
2
3 vLLM initially supports basic model inferencing and serving on x86 CPU platform, with data types FP32, FP16 and BF16.
4
5 +:::{attention}
6 +There are no pre-built wheels or images for this device, so you must build vLLM from source.
7 +:::
8 +
9 ## Requirements
10
11 - OS: Linux
12 - Compiler: `gcc/g++ >= 12.3.0` (optional, recommended)
13 - Instruction Set Architecture (ISA): AVX512 (optional, recommended)
14
15 +:::{tip}
16 +[Intel Extension for PyTorch (IPEX)](https://github.com/intel/intel-extension-for-pytorch) extends PyTorch with up-to-date features optimizations for an extra performance boost on Intel hardware.
17 +:::
18 +
19 ## Set up using Python
20
21 ### Pre-built wheels
@@ -29,7 +37,3 @@ vLLM initially supports basic model inferencing and serving on x86 CPU platform,
37 ### Build image from source
38
39 ## Extra information
32 -
33 -## Intel Extension for PyTorch
34 -
35 -- [Intel Extension for PyTorch (IPEX)](https://github.com/intel/intel-extension-for-pytorch) extends PyTorch with up-to-date features optimizations for an extra performance boost on Intel hardware.
docs/source/getting_started/installation/gpu/index.md
+25 -24
@@ -5,7 +5,8 @@ vLLM is a Python library that supports the following GPU variants. Select your G
5 :::::{tab-set}
6 :sync-group: device
7
8 -::::{tab-item} CUDA
8 +::::{tab-item} NVIDIA CUDA
9 +:selected:
10 :sync: cuda
11
12 :::{include} cuda.inc.md
@@ -15,7 +16,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
16
17 ::::
18
18 -::::{tab-item} ROCm
19 +::::{tab-item} AMD ROCm
20 :sync: rocm
21
22 :::{include} rocm.inc.md
@@ -25,7 +26,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
26
27 ::::
28
28 -::::{tab-item} XPU
29 +::::{tab-item} Intel XPU
30 :sync: xpu
31
32 :::{include} xpu.inc.md
@@ -45,7 +46,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
46 :::::{tab-set}
47 :sync-group: device
48
48 -::::{tab-item} CUDA
49 +::::{tab-item} NVIDIA CUDA
50 :sync: cuda
51
52 :::{include} cuda.inc.md
@@ -55,7 +56,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
56
57 ::::
58
58 -::::{tab-item} ROCm
59 +::::{tab-item} AMD ROCm
60 :sync: rocm
61
62 :::{include} rocm.inc.md
@@ -65,7 +66,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
66
67 ::::
68
68 -::::{tab-item} XPU
69 +::::{tab-item} Intel XPU
70 :sync: xpu
71
72 :::{include} xpu.inc.md
@@ -87,7 +88,7 @@ vLLM is a Python library that supports the following GPU variants. Select your G
88 :::::{tab-set}
89 :sync-group: device
90
90 -::::{tab-item} CUDA
91 +::::{tab-item} NVIDIA CUDA
92 :sync: cuda
93
94 :::{include} cuda.inc.md
@@ -97,14 +98,14 @@ vLLM is a Python library that supports the following GPU variants. Select your G
98
99 ::::
100
100 -::::{tab-item} ROCm
101 +::::{tab-item} AMD ROCm
102 :sync: rocm
103
104 There is no extra information on creating a new Python environment for this device.
105
106 ::::
107
107 -::::{tab-item} XPU
108 +::::{tab-item} Intel XPU
109 :sync: xpu
110
111 There is no extra information on creating a new Python environment for this device.
@@ -118,7 +119,7 @@ There is no extra information on creating a new Python environment for this devi
119 :::::{tab-set}
120 :sync-group: device
121
121 -::::{tab-item} CUDA
122 +::::{tab-item} NVIDIA CUDA
123 :sync: cuda
124
125 :::{include} cuda.inc.md
@@ -128,7 +129,7 @@ There is no extra information on creating a new Python environment for this devi
129
130 ::::
131
131 -::::{tab-item} ROCm
132 +::::{tab-item} AMD ROCm
133 :sync: rocm
134
135 :::{include} rocm.inc.md
@@ -138,7 +139,7 @@ There is no extra information on creating a new Python environment for this devi
139
140 ::::
141
141 -::::{tab-item} XPU
142 +::::{tab-item} Intel XPU
143 :sync: xpu
144
145 :::{include} xpu.inc.md
@@ -157,7 +158,7 @@ There is no extra information on creating a new Python environment for this devi
158 :::::{tab-set}
159 :sync-group: device
160
160 -::::{tab-item} CUDA
161 +::::{tab-item} NVIDIA CUDA
162 :sync: cuda
163
164 :::{include} cuda.inc.md
@@ -167,7 +168,7 @@ There is no extra information on creating a new Python environment for this devi
168
169 ::::
170
170 -::::{tab-item} ROCm
171 +::::{tab-item} AMD ROCm
172 :sync: rocm
173
174 :::{include} rocm.inc.md
@@ -177,7 +178,7 @@ There is no extra information on creating a new Python environment for this devi
178
179 ::::
180
180 -::::{tab-item} XPU
181 +::::{tab-item} Intel XPU
182 :sync: xpu
183
184 :::{include} xpu.inc.md
@@ -196,7 +197,7 @@ There is no extra information on creating a new Python environment for this devi
197 :::::{tab-set}
198 :sync-group: device
199
199 -::::{tab-item} CUDA
200 +::::{tab-item} NVIDIA CUDA
201 :sync: cuda
202
203 :::{include} cuda.inc.md
@@ -206,7 +207,7 @@ There is no extra information on creating a new Python environment for this devi
207
208 ::::
209
209 -::::{tab-item} ROCm
210 +::::{tab-item} AMD ROCm
211 :sync: rocm
212
213 :::{include} rocm.inc.md
@@ -216,7 +217,7 @@ There is no extra information on creating a new Python environment for this devi
217
218 ::::
219
219 -::::{tab-item} XPU
220 +::::{tab-item} Intel XPU
221 :sync: xpu
222
223 :::{include} xpu.inc.md
@@ -233,7 +234,7 @@ There is no extra information on creating a new Python environment for this devi
234 :::::{tab-set}
235 :sync-group: device
236
236 -::::{tab-item} CUDA
237 +::::{tab-item} NVIDIA CUDA
238 :sync: cuda
239
240 :::{include} cuda.inc.md
@@ -243,7 +244,7 @@ There is no extra information on creating a new Python environment for this devi
244
245 ::::
246
246 -::::{tab-item} ROCm
247 +::::{tab-item} AMD ROCm
248 :sync: rocm
249
250 :::{include} rocm.inc.md
@@ -253,7 +254,7 @@ There is no extra information on creating a new Python environment for this devi
254
255 ::::
256
256 -::::{tab-item} XPU
257 +::::{tab-item} Intel XPU
258 :sync: xpu
259
260 :::{include} xpu.inc.md
@@ -270,7 +271,7 @@ There is no extra information on creating a new Python environment for this devi
271 :::::{tab-set}
272 :sync-group: device
273
273 -::::{tab-item} CUDA
274 +::::{tab-item} NVIDIA CUDA
275 :sync: cuda
276
277 :::{include} cuda.inc.md
@@ -279,7 +280,7 @@ There is no extra information on creating a new Python environment for this devi
280
281 ::::
282
282 -::::{tab-item} ROCm
283 +::::{tab-item} AMD ROCm
284 :sync: rocm
285
286 :::{include} rocm.inc.md
@@ -288,7 +289,7 @@ There is no extra information on creating a new Python environment for this devi
289
290 ::::
291
291 -::::{tab-item} XPU
292 +::::{tab-item} Intel XPU
293 :sync: xpu
294
295 :::{include} xpu.inc.md
docs/source/getting_started/installation/gpu/rocm.inc.md
+11 -9
@@ -2,6 +2,10 @@
2
3 vLLM supports AMD GPUs with ROCm 6.2.
4
5 +:::{attention}
6 +There are no pre-built wheels for this device, so you must either use the pre-built Docker image or build vLLM from source.
7 +:::
8 +
9 ## Requirements
10
11 - GPU: MI200s (gfx90a), MI300 (gfx942), Radeon RX 7900 series (gfx1100)
@@ -13,14 +17,6 @@ vLLM supports AMD GPUs with ROCm 6.2.
17
18 Currently, there are no pre-built ROCm wheels.
19
16 -However, the [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offers a prebuilt, optimized
17 -docker image designed for validating inference performance on the AMD Instinct™ MI300X accelerator.
18 -
19 -:::{tip}
20 -Please check [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/performance-validation/mi300x/vllm-benchmark.html)
21 -for instructions on how to use this prebuilt docker image.
22 -:::
23 -
20 ### Build wheel from source
21
22 0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
@@ -112,7 +108,13 @@ for instructions on how to use this prebuilt docker image.
108
109 ### Pre-built images
110
115 -Currently, there are no pre-built ROCm images.
111 +The [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offers a prebuilt, optimized
112 +docker image designed for validating inference performance on the AMD Instinct™ MI300X accelerator.
113 +
114 +:::{tip}
115 +Please check [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/performance-validation/mi300x/vllm-benchmark.html)
116 +for instructions on how to use this prebuilt docker image.
117 +:::
118
119 ### Build image from source
120
docs/source/getting_started/installation/gpu/xpu.inc.md
+4
@@ -2,6 +2,10 @@
2
3 vLLM initially supports basic model inferencing and serving on Intel GPU platform.
4
5 +:::{attention}
6 +There are no pre-built wheels or images for this device, so you must build vLLM from source.
7 +:::
8 +
9 ## Requirements
10
11 - Supported Hardware: Intel Data Center GPU, Intel ARC GPU
docs/source/getting_started/installation/index.md
+15
@@ -6,8 +6,23 @@ vLLM supports the following hardware platforms:
6
7 :::{toctree}
8 :maxdepth: 1
9 +:hidden:
10
11 gpu/index
12 cpu/index
13 ai_accelerator/index
14 :::
15 +
16 +- <project:gpu/index.md>
17 + - NVIDIA CUDA
18 + - AMD ROCm
19 + - Intel XPU
20 +- <project:cpu/index.md>
21 + - Intel/AMD x86
22 + - ARM AArch64
23 + - Apple silicon
24 +- <project:ai_accelerator/index.md>
25 + - Google TPU
26 + - Intel Gaudi
27 + - AWS Neuron
28 + - OpenVINO