@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 522 lines 26.8 KB
Rendered Raw
1 <!-- markdownlint-disable MD041 MD051 -->
2 --8<-- [start:installation]
3
4 vLLM contains pre-compiled C++ and CUDA (12.9) binaries.
5
6 --8<-- [end:installation]
7 --8<-- [start:requirements]
8
9 - GPU: compute capability 7.5 or higher (e.g., T4, RTX20xx, A100, L4, H100, B200, etc.)
10
11 --8<-- [end:requirements]
12 --8<-- [start:set-up-using-python]
13
14 !!! note
15 PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <https://github.com/vllm-project/vllm/issues/8420> for more details.
16
17 In order to be performant, vLLM has to compile many cuda kernels. The compilation unfortunately introduces binary incompatibility with other CUDA versions and PyTorch versions, even for the same PyTorch version with different building configurations.
18
19 Therefore, it is recommended to install vLLM with a **fresh new** environment. If either you have a different CUDA version or you want to use an existing PyTorch installation, you need to build vLLM from source. See [below](#build-wheel-from-source) for more details.
20
21 --8<-- [end:set-up-using-python]
22 --8<-- [start:pre-built-wheels]
23
24 ```bash
25 uv pip install vllm --torch-backend=auto
26 ```
27
28 ??? console "pip"
29 ```bash
30 # Install vLLM with CUDA 12.9.
31 pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
32 ```
33
34 We recommend leveraging `uv` to [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu130`), set `--torch-backend=cu130` (or `UV_TORCH_BACKEND=cu130`). If this doesn't work, try running `uv self update` to update `uv` first.
35
36 !!! note
37 NVIDIA Blackwell GPUs (B200, GB200) require a minimum of CUDA 12.8, so make sure you are installing PyTorch wheels with at least that version. PyTorch itself offers a [dedicated interface](https://pytorch.org/get-started/locally/) to determine the appropriate pip command to run for a given target configuration.
38
39 As of now, vLLM's binaries are compiled with CUDA 12.9 and public PyTorch release versions by default. We also provide vLLM binaries compiled with CUDA 12.8, 13.0, and public PyTorch release versions:
40
41 ```bash
42 # Install vLLM with a specific CUDA version (e.g., 13.0).
43 export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')
44 export CUDA_VERSION=130 # or other
45 export CPU_ARCH=$(uname -m) # x86_64 or aarch64
46 uv pip install "https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu${CUDA_VERSION}-cp38-abi3-manylinux_2_28_${CPU_ARCH}.whl" --extra-index-url "https://download.pytorch.org/whl/cu${CUDA_VERSION}"
47 ```
48
49 #### Install the latest code
50
51 LLM inference is a fast-evolving field, and the latest code may contain bug fixes, performance improvements, and new features that are not released yet. To allow users to try the latest code without waiting for the next release, vLLM provides wheels for every commit since `v0.5.3` on <https://wheels.vllm.ai/nightly>. There are multiple indices that could be used:
52
53 - `https://wheels.vllm.ai/nightly`: the default variant (CUDA with version specified in `VLLM_MAIN_CUDA_VERSION`) built with the last commit on the `main` branch. Currently it is CUDA 12.9.
54 - `https://wheels.vllm.ai/nightly/<variant>`: all other variants. Now this includes `cu130`, and `cpu`. The default variant (`cu129`) also has a subdirectory to keep consistency.
55
56 To install from nightly index, run:
57
58 ```bash
59 uv pip install -U vllm \
60 --torch-backend=auto \
61 --extra-index-url https://wheels.vllm.ai/nightly # add variant subdirectory here if needed
62 ```
63
64 !!! warning "`pip` caveat"
65
66 Using `pip` to install from nightly indices is _not supported_, because `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install a development version prior to the released version. In contrast, `uv` gives the extra index [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes).
67
68 If you insist on using `pip`, you have to specify the full URL of the wheel file (which can be obtained from the web page).
69
70 ```bash
71 pip install -U https://wheels.vllm.ai/2f3f441f84bd5b35ec8aa9fcfffb540f107da8a7/vllm-0.23.1rc1.dev901%2Bg2f3f441f8-cp38-abi3-manylinux_2_28_x86_64.whl # current nightly build (the filename will change!)
72 pip install -U https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-0.23.1rc1.dev901%2Bg2f3f441f8-cp38-abi3-manylinux_2_28_x86_64.whl # from specific commit
73 ```
74
75 ##### Install specific revisions
76
77 If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL:
78
79 ```bash
80 export VLLM_COMMIT=72d9c316d3f6ede485146fe5aabd4e61dbc59069 # use full commit hash from the main branch
81 uv pip install vllm \
82 --torch-backend=auto \
83 --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT} # add variant subdirectory here if needed
84 ```
85
86 --8<-- [end:pre-built-wheels]
87 --8<-- [start:build-wheel-from-source]
88
89 #### Set up using Python-only build (without compilation) {#python-only-build}
90
91 If you only need to change Python code, you can build and install vLLM without compilation. Using `uv pip`'s [`--editable` flag](https://docs.astral.sh/uv/pip/packages/#editable-packages), changes you make to the code will be reflected when you run vLLM:
92
93 ```bash
94 git clone https://github.com/vllm-project/vllm.git
95 cd vllm
96 VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
97 ```
98
99 This command will do the following:
100
101 1. Look for the current branch in your vLLM clone.
102 1. Identify the corresponding base commit in the main branch.
103 1. Download the pre-built wheel of the base commit.
104 1. Use its compiled libraries and `vllm-rs` binary in the installation.
105
106 !!! note
107 1. If you change C++ or kernel code, you cannot use Python-only build; otherwise you will see an import error about library not found or undefined symbol.
108 2. If you rebase your dev branch, it is recommended to uninstall vllm and re-run the above command to make sure your libraries are up to date.
109
110 !!! tip "Rebuilding the Rust frontend"
111 If you need to recompile the `vllm-rs` Rust frontend binary, you can rebuild and install it without re-running the full pip install:
112
113 ```bash
114 ./tools/build_rust.sh # release build
115 ./tools/build_rust.sh --debug # faster build for development
116 ```
117
118 This will install the required Rust toolchain if needed, build the binary, and place it in `vllm/vllm-rs`.
119
120 In case you see an error about wheel not found when running the above command, it might be because the commit you based on in the `main` branch was just merged and its precompiled wheel is not available yet. You can wait around an hour and retry, or set `VLLM_PRECOMPILED_WHEEL_COMMIT=nightly` to automatically select the most recent already-built commit on `main`.
121
122 ```bash
123 export VLLM_PRECOMPILED_WHEEL_COMMIT=nightly
124 export VLLM_USE_PRECOMPILED=1
125 uv pip install --editable .
126 ```
127
128 There are more environment variables to control the behavior of Python-only build:
129
130 - `VLLM_PRECOMPILED_WHEEL_LOCATION`: specify the exact wheel URL or local file path of a pre-compiled wheel to use. All other logic to find the wheel will be skipped.
131 - `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the commit hash to download the pre-compiled wheel. It can be `nightly` to use the last **already built** commit on the main branch.
132 - `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the variant subdirectory to use on the nightly index, e.g., `cu129`, `cu130`, `cpu`. If not specified, the variant is auto-detected based on your system's CUDA version (from PyTorch or nvidia-smi). You can also set `VLLM_MAIN_CUDA_VERSION` to override auto-detection.
133
134 You can find more information about vLLM's wheels in [Install the latest code](#install-the-latest-code).
135
136 !!! note
137 There is a possibility that your source code may have a different commit ID compared to the latest vLLM wheel, which could potentially lead to unknown errors.
138 It is recommended to use the same commit ID for the source code as the vLLM wheel you have installed. Please refer to [Install the latest code](#install-the-latest-code) for instructions on how to install a specified wheel.
139
140 #### Full build (with compilation) {#full-build}
141
142 !!! note "Compiler requirement"
143 Building from source requires GCC/G++ ≥ 11.3. PyTorch's C++20 headers are
144 not compatible with GCC 10 or GCC < 11.3. On Ubuntu 22.04:
145 ```bash
146 sudo apt-get install -y gcc-11 g++-11
147 sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 110 \
148 --slave /usr/bin/g++ g++ /usr/bin/g++-11
149 ```
150
151 If you want to modify C++ or CUDA code, you'll need to build vLLM from source. This can take several minutes:
152
153 ```bash
154 git clone https://github.com/vllm-project/vllm.git
155 cd vllm
156 uv pip install -e . --torch-backend=auto
157 ```
158
159 !!! note "CUDA Architecture & PTX Flags"
160 vLLM normalizes CUDA architectures on a per-source basis to optimize build times and wheel sizes. Global `+PTX` requests in `TORCH_CUDA_ARCH_LIST` (e.g., `TORCH_CUDA_ARCH_LIST="8.0+PTX"`) are ignored for general extension targets; vLLM generates PTX only for specific internal kernels that require it.
161
162 !!! tip
163 Building from source requires a lot of compilation. If you are building from source repeatedly, it's more efficient to cache the compilation results.
164
165 For example, you can install [ccache](https://github.com/ccache/ccache) using `conda install ccache` or `apt install ccache` .
166 As long as `which ccache` command can find the `ccache` binary, it will be used automatically by the build system. After the first build, subsequent builds will be much faster.
167
168 When using `ccache` with `pip install -e .`, you should run `CCACHE_NOHASHDIR="true" pip install --no-build-isolation -e .`. This is because `pip` creates a new folder with a random name for each build, preventing `ccache` from recognizing that the same files are being built.
169
170 [sccache](https://github.com/mozilla/sccache) works similarly to `ccache`, but has the capability to utilize caching in remote storage environments.
171 The following environment variables can be set to configure the vLLM `sccache` remote: `SCCACHE_BUCKET=vllm-build-sccache SCCACHE_REGION=us-west-2 SCCACHE_S3_NO_CREDENTIALS=1`. We also recommend setting `SCCACHE_IDLE_TIMEOUT=0`.
172
173 !!! note "Faster Kernel Development"
174 For frequent C++/CUDA kernel changes, after the initial `uv pip install -e .` setup, consider using the [Incremental Compilation Workflow](../../contributing/incremental_build.md) for significantly faster rebuilds of only the modified kernel code.
175
176 ##### Use an existing PyTorch installation
177
178 There are scenarios where the PyTorch dependency cannot be easily installed with `uv`, for example, when building vLLM with non-default PyTorch builds (like nightly or a custom build).
179
180 To build vLLM using an existing PyTorch installation:
181
182 ```bash
183 # install PyTorch first, either from PyPI or from source
184 git clone https://github.com/vllm-project/vllm.git
185 cd vllm
186 python tools/use_existing_torch.py
187 uv pip install -r requirements/build/cuda.txt
188 uv pip install --no-build-isolation -e .
189 ```
190
191 Alternatively: if you are exclusively using `uv` to create and manage virtual environments, it has [a unique mechanism](https://docs.astral.sh/uv/concepts/projects/config/#disabling-build-isolation)
192 for disabling build isolation for specific packages. vLLM can leverage this mechanism to specify `torch` as the package to disable build isolation for:
193
194 ```bash
195 # install PyTorch first, either from PyPI or from source
196 git clone https://github.com/vllm-project/vllm.git
197 cd vllm
198 # pip install -e . does not work directly, only uv can do this
199 uv pip install -e .
200 ```
201
202 ##### Use the local cutlass for compilation
203
204 Currently, before starting the build process, vLLM fetches cutlass code from GitHub. However, there may be scenarios where you want to use a local version of cutlass instead.
205 To achieve this, you can set the environment variable VLLM_CUTLASS_SRC_DIR to point to your local cutlass directory.
206
207 ```bash
208 git clone https://github.com/vllm-project/vllm.git
209 cd vllm
210 VLLM_CUTLASS_SRC_DIR=/path/to/cutlass uv pip install -e . --torch-backend=auto
211 ```
212
213 ##### Troubleshooting
214
215 To avoid your system being overloaded, you can limit the number of compilation jobs
216 to be run simultaneously, via the environment variable `MAX_JOBS`. For example:
217
218 ```bash
219 export MAX_JOBS=6
220 uv pip install -e .
221 ```
222
223 This is especially useful when you are building on less powerful machines. For example, when you use WSL it only [assigns 50% of the total memory by default](https://learn.microsoft.com/en-us/windows/wsl/wsl-config#main-wsl-settings), so using `export MAX_JOBS=1` can avoid compiling multiple files simultaneously and running out of memory.
224 A side effect is a much slower build process.
225
226 Additionally, if you have trouble building vLLM, we recommend using the NVIDIA PyTorch Docker image.
227
228 ```bash
229 # Use `--ipc=host` to make sure the shared memory is large enough.
230 docker run \
231 --gpus all \
232 -it \
233 --rm \
234 --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
235 ```
236
237 If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from [the official website](https://developer.nvidia.com/cuda-toolkit-archive). After installation, set the environment variable `CUDA_HOME` to the installation path of CUDA Toolkit, and make sure that the `nvcc` compiler is in your `PATH`, e.g.:
238
239 ```bash
240 export CUDA_HOME=/usr/local/cuda
241 export PATH="${CUDA_HOME}/bin:$PATH"
242 ```
243
244 Here is a sanity check to verify that the CUDA Toolkit is correctly installed:
245
246 ```bash
247 nvcc --version # verify that nvcc is in your PATH
248 ${CUDA_HOME}/bin/nvcc --version # verify that nvcc is in your CUDA_HOME
249 ```
250
251 #### Unsupported OS build
252
253 vLLM can fully run only on Linux but for development purposes, you can still build it on other systems (for example, macOS), allowing for imports and a more convenient development environment. The binaries will not be compiled and won't work on non-Linux systems.
254
255 Simply disable the `VLLM_TARGET_DEVICE` environment variable before installing:
256
257 ```bash
258 export VLLM_TARGET_DEVICE=empty
259 uv pip install -e .
260 ```
261
262 --8<-- [end:build-wheel-from-source]
263 --8<-- [start:pre-built-images]
264
265 vLLM offers an official Docker image for deployment.
266 The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai](https://hub.docker.com/r/vllm/vllm-openai/tags).
267
268 ```bash
269 docker run --runtime nvidia --gpus all \
270 -v ~/.cache/huggingface:/root/.cache/huggingface \
271 --env "HF_TOKEN=$HF_TOKEN" \
272 -p 8000:8000 \
273 --ipc=host \
274 vllm/vllm-openai:latest \
275 --model Qwen/Qwen3-0.6B
276 ```
277
278 This image can also be used with other container engines such as [Podman](https://podman.io/).
279
280 ```bash
281 podman run --device nvidia.com/gpu=all \
282 -v ~/.cache/huggingface:/root/.cache/huggingface \
283 --env "HF_TOKEN=$HF_TOKEN" \
284 -p 8000:8000 \
285 --ipc=host \
286 docker.io/vllm/vllm-openai:latest \
287 --model Qwen/Qwen3-0.6B
288 ```
289
290 You can add any other [engine-args](https://docs.vllm.ai/en/latest/configuration/engine_args/) you need after the image tag (`vllm/vllm-openai:latest`).
291
292 !!! note
293 You can either use the `ipc=host` flag or `--shm-size` flag to allow the
294 container to access the host's shared memory. vLLM uses PyTorch, which uses shared
295 memory to share data between processes under the hood, particularly for tensor parallel inference.
296
297 !!! note
298 Optional dependencies are not included in order to avoid licensing issues (e.g. <https://github.com/vllm-project/vllm/issues/8030>).
299
300 If you need to use those dependencies (having accepted the license terms),
301 create a custom Dockerfile on top of the base image with an extra layer that installs them:
302
303 ```Dockerfile
304 FROM vllm/vllm-openai:v0.11.0
305
306 # e.g. install the `audio` optional dependencies
307 # NOTE: Make sure the version of vLLM matches the base image!
308 RUN uv pip install --system vllm[audio]==0.11.0
309 ```
310
311 !!! tip
312 Some new models may only be available on the main branch of [HF Transformers](https://github.com/huggingface/transformers).
313
314 To use the development version of `transformers`, create a custom Dockerfile on top of the base image
315 with an extra layer that installs their code from source:
316
317 ```Dockerfile
318 FROM vllm/vllm-openai:latest
319
320 RUN uv pip install --system git+https://github.com/huggingface/transformers.git
321 ```
322
323 #### Running on Systems with Older CUDA Drivers
324
325 vLLM's Docker image comes with [CUDA compatibility libraries](https://docs.nvidia.com/deploy/cuda-compatibility/index.html) pre-installed. This allows you to run vLLM on systems with NVIDIA drivers that are older than the CUDA Toolkit version used in the image, but only supports select professional and datacenter NVIDIA GPUs.
326
327 For CUDA 13 images, the minimum host kernel is Linux 4.15 when running normally because CUDA 13 requires an R580 or newer driver. Compatibility mode supports R535 and R570 host drivers; R535 lowers the minimum host kernel to Linux 3.10, while R570 still requires Linux 4.15. Upgrading the container userland to Ubuntu 24.04 does not raise these driver-defined kernel requirements. See NVIDIA's [forward compatibility matrix](https://docs.nvidia.com/deploy/cuda-compatibility/forward-compatibility.html#use-the-right-cuda-forward-compatibility-package), [R535 requirements](https://download.nvidia.com/XFree86/Linux-x86_64/535.104.05/README/minimumrequirements.html), and [R580 requirements](https://download.nvidia.com/XFree86/Linux-x86_64/580.76.05/README/minimumrequirements.html).
328
329 To enable this feature, set the `VLLM_ENABLE_CUDA_COMPATIBILITY` environment variable to `1` or `true` when running the container:
330
331 ```bash
332 docker run --runtime nvidia --gpus all \
333 -v ~/.cache/huggingface:/root/.cache/huggingface \
334 -p 8000:8000 \
335 --env "HF_TOKEN=<secret>" \
336 --env "VLLM_ENABLE_CUDA_COMPATIBILITY=1" \
337 vllm/vllm-openai <args...>
338 ```
339
340 This will automatically configure `LD_LIBRARY_PATH` to point to the compatibility libraries before loading PyTorch and other dependencies.
341
342 --8<-- [end:pre-built-images]
343 --8<-- [start:build-image-from-source]
344
345 You can build and run vLLM from source via the provided [docker/Dockerfile](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile). To build vLLM:
346
347 ```bash
348 # optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2
349 DOCKER_BUILDKIT=1 docker build . \
350 --target vllm-openai \
351 --tag vllm/vllm-openai \
352 --file docker/Dockerfile
353 ```
354
355 !!! note
356 By default vLLM will build for all GPU types for widest distribution. If you are just building for the
357 current GPU type, you can add `--build-arg torch_cuda_arch_list=""` to delegate architecture selection
358 to PyTorch. This requires the GPU to be visible to the container build; standard Docker BuildKit builds
359 do not expose it and PyTorch instead falls back to its common architecture list.
360
361 If you are using Podman instead of Docker, you might need to disable SELinux labeling by
362 adding `--security-opt label=disable` when running `podman build` command to avoid certain [existing issues](https://github.com/containers/buildah/discussions/4184).
363
364 !!! note
365 If you have not changed any C++ or CUDA kernel code, you can use precompiled wheels to significantly reduce Docker build time.
366
367 * **Enable the feature** by adding the build argument: `--build-arg VLLM_USE_PRECOMPILED="1"`.
368 * **How it works**: By default, vLLM automatically finds the correct wheels from our [Nightly Builds](https://docs.vllm.ai/en/latest/contributing/ci/nightly_builds/) by using the merge-base commit with the upstream `main` branch.
369 * **Override commit**: To use wheels from a specific commit, provide the `--build-arg VLLM_PRECOMPILED_WHEEL_COMMIT=<commit_hash>` argument.
370
371 For a detailed explanation, refer to the documentation on 'Set up using Python-only build (without compilation)' part in [Build wheel from source](https://docs.vllm.ai/en/latest/contributing/ci/nightly_builds/#precompiled-wheels-usage), these args are similar.
372
373 #### Building vLLM's Docker Image from Source for Arm64/aarch64
374
375 A docker container can be built for aarch64 systems such as the Nvidia Grace-Hopper and Grace-Blackwell. Using the flag `--platform "linux/arm64"` will build for arm64.
376
377 !!! note
378 Multiple modules must be compiled, so this process can take a while. Recommend using `--build-arg max_jobs=` & `--build-arg nvcc_threads=`
379 flags to speed up build process. However, ensure your `max_jobs` is substantially larger than `nvcc_threads` to get the most benefits.
380 Keep an eye on memory usage with parallel jobs as it can be substantial (see example below).
381
382 ??? console "Command"
383
384 ```bash
385 # Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)
386 DOCKER_BUILDKIT=1 docker build . \
387 --file docker/Dockerfile \
388 --target vllm-openai \
389 --platform "linux/arm64" \
390 -t vllm/vllm-gh200-openai:latest \
391 --build-arg max_jobs=66 \
392 --build-arg nvcc_threads=2 \
393 --build-arg BUILD_BASE_IMAGE=pytorch/manylinuxaarch64-builder:cuda13.0-78e737ad29420ffc4800e677c51e2a852caf8359 \
394 --build-arg torch_cuda_arch_list="9.0 10.0+PTX"
395 ```
396
397 For (G)B300, we recommend using CUDA 13, as shown in the following command.
398
399 ??? console "Command"
400
401 ```bash
402 DOCKER_BUILDKIT=1 docker build \
403 --build-arg CUDA_VERSION=13.0.2 \
404 --build-arg BUILD_BASE_IMAGE=pytorch/manylinuxaarch64-builder:cuda13.0-78e737ad29420ffc4800e677c51e2a852caf8359 \
405 --build-arg max_jobs=256 \
406 --build-arg nvcc_threads=2 \
407 --build-arg torch_cuda_arch_list='9.0 10.0+PTX' \
408 --platform "linux/arm64" \
409 --tag vllm/vllm-gb300-openai:latest \
410 --target vllm-openai \
411 -f docker/Dockerfile \
412 .
413 ```
414
415 !!! note
416 If you are building the `linux/arm64` image on a non-ARM host (e.g., an x86_64 machine), you need to ensure your system is set up for cross-compilation using QEMU. This allows your host machine to emulate ARM64 execution.
417
418 Run the following command on your host machine to register QEMU user static handlers:
419
420 ```bash
421 docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
422 ```
423
424 After setting up QEMU, you can use the `--platform "linux/arm64"` flag in your `docker build` command.
425
426 #### [Preview] Building vLLM's Docker Image from Source for NVIDIA Rubin GPU Architecture
427
428 Set `INSTALL_RUBIN_PRERELEASE=true` to enable the Rubin build path.
429
430 Triton must currently be installed from source for Rubin compatibility.
431 Specify its repository with `TRITON_INSTALL_FROM_SOURCE_REPO`; an empty
432 `TRITON_INSTALL_FROM_SOURCE_REVISION` selects the repository's latest `main`,
433 while a commit, branch, or tag selects that revision. The tested revision
434 lowers SM107 through LLVM's SM100 target and uses the final CUDA image's
435 version-matched `ptxas` for SM107 assembly. This enables vLLM's default compiled
436 mode on VR200 and R100.
437
438 BuildKit does not automatically invalidate cached layers when a mutable Git
439 ref changes. Use `--no-cache-filter extensions-build` to refresh an empty,
440 branch, or tag revision.
441
442 For `FINAL_BASE_IMAGE`, use the public, multi-arch
443 `nvcr.io/nvidia/cuda-dl-base:26.08-cuda13.4-devel-ubuntu24.04` image.
444 For `BUILD_BASE_IMAGE`, use:
445
446 - `pytorch/manylinux2_28-builder:cuda13.4` for x86_64 CPUs.
447 - `pytorch/manylinuxaarch64-builder:cuda13.4` for ARM64/AArch64 CPUs.
448
449 ??? console "ARM64/AArch64 build command"
450
451 ```bash
452 docker buildx build --progress=plain --load \
453 --file docker/Dockerfile \
454 --target vllm-openai \
455 --platform "linux/arm64" \
456 --tag "vllm/vllm-rubin-openai:prerelease-cu134-public-arm64" \
457 --build-arg max_jobs="$(nproc)" \
458 --build-arg nvcc_threads=2 \
459 --build-arg RUN_WHEEL_CHECK=false \
460 --build-arg INSTALL_RUBIN_PRERELEASE=true \
461 --build-arg TRITON_INSTALL_FROM_SOURCE_REPO=https://github.com/triton-lang/triton.git \
462 --build-arg TRITON_INSTALL_FROM_SOURCE_REVISION=3f6e41132b5edf639bfb872ad73d4688765e08b8 \
463 --build-arg CUDA_VERSION=13.4 \
464 --build-arg BUILD_BASE_IMAGE="pytorch/manylinuxaarch64-builder:cuda13.4" \
465 --build-arg FINAL_BASE_IMAGE="nvcr.io/nvidia/cuda-dl-base:26.08-cuda13.4-devel-ubuntu24.04" \
466 .
467 ```
468
469 ??? console "x86_64 build command"
470
471 ```bash
472 docker buildx build --progress=plain --load \
473 --file docker/Dockerfile \
474 --target vllm-openai \
475 --platform "linux/amd64" \
476 --tag "vllm/vllm-rubin-openai:prerelease-cu134-public-amd64" \
477 --build-arg max_jobs="$(nproc)" \
478 --build-arg nvcc_threads=2 \
479 --build-arg RUN_WHEEL_CHECK=false \
480 --build-arg INSTALL_RUBIN_PRERELEASE=true \
481 --build-arg TRITON_INSTALL_FROM_SOURCE_REPO=https://github.com/triton-lang/triton.git \
482 --build-arg TRITON_INSTALL_FROM_SOURCE_REVISION=3f6e41132b5edf639bfb872ad73d4688765e08b8 \
483 --build-arg CUDA_VERSION=13.4 \
484 --build-arg BUILD_BASE_IMAGE="pytorch/manylinux2_28-builder:cuda13.4" \
485 --build-arg FINAL_BASE_IMAGE="nvcr.io/nvidia/cuda-dl-base:26.08-cuda13.4-devel-ubuntu24.04" \
486 .
487 ```
488
489 !!! note
490 Keep the default explicit `torch_cuda_arch_list`. GPU-less BuildKit builds
491 cannot inspect the host GPU. R100 and VR200 report compute capability 10.7,
492 for which the generic `10.0` target provides family-compatible kernels.
493 The Ubuntu `devel` final image is also required: the corresponding `base`
494 image lacks the CUDA runtime/JIT package closure used by vLLM and the
495 prerelease PyTorch wheel.
496
497 `RUN_WHEEL_CHECK=false` disables only the PyPI publication-size guard for
498 this private staging image.
499
500 #### Use the custom-built vLLM Docker image**
501
502 To run vLLM with the custom-built Docker image:
503
504 ```bash
505 docker run --runtime nvidia --gpus all \
506 -v ~/.cache/huggingface:/root/.cache/huggingface \
507 -p 8000:8000 \
508 --env "HF_TOKEN=<secret>" \
509 vllm/vllm-openai <args...>
510 ```
511
512 The argument `vllm/vllm-openai` specifies the image to run, and should be replaced with the name of the custom-built image (the `-t` tag from the build command).
513
514 !!! note
515 **For version 0.4.1 and 0.4.2 only** - the vLLM docker images under these versions are supposed to be run under the root user since a library under the root user's home directory, i.e. `/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` is required to be loaded during runtime. If you are running the container under a different user, you may need to first change the permissions of the library (and all the parent directories) to allow the user to access it, then run vLLM with environment variable `VLLM_NCCL_SO_PATH=/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` .
516
517 --8<-- [end:build-image-from-source]
518 --8<-- [start:supported-features]
519
520 See [Feature x Hardware](../../features/README.md#feature-x-hardware) compatibility matrix for feature support information.
521
522 --8<-- [end:supported-features]