A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 02:15 UTC
[doc] update doc format (#20673)
Signed-off-by: reidliu41 <reid201711@gmail.com>
Reid committed
Jul 9, 2025 at 23:08 UTC
efe73d0575951767180468dac8202739cb479074
1 file changed
+51
-27
docs/contributing/ci/update_pytorch_version.md
+51
-27
@@ -16,11 +16,12 @@ by waiting for the next release or by implementing hacky workarounds in vLLM.
16
The better solution is to test vLLM with PyTorch release candidates (RC) to ensure
17
compatibility before each release.
18
19
-PyTorch release candidates can be downloaded from PyTorch test index at https://download.pytorch.org/whl/test.
20
-For example, torch2.7.0+cu12.8 RC can be installed using the following command:
19
+PyTorch release candidates can be downloaded from [PyTorch test index](https://download.pytorch.org/whl/test).
20
+For example, `torch2.7.0+cu12.8` RC can be installed using the following command:
21
22
-```
23
-uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/test/cu128
22
+```bash
23
+uv pip install torch torchvision torchaudio \
24
+ --index-url https://download.pytorch.org/whl/test/cu128
25
```
26
27
When the final RC is ready for testing, it will be announced to the community
@@ -28,13 +29,28 @@ on the [PyTorch dev-discuss forum](https://dev-discuss.pytorch.org/c/release-ann
29
After this announcement, we can begin testing vLLM integration by drafting a pull request
30
following this 3-step process:
31
31
-1. Update requirements files in https://github.com/vllm-project/vllm/tree/main/requirements
32
-to point to the new releases for torch, torchvision, and torchaudio.
33
-2. Use `--extra-index-url https://download.pytorch.org/whl/test/<PLATFORM>` to
34
-get the final release candidates' wheels. Some common platforms are `cpu`, `cu128`,
35
-and `rocm6.2.4`.
36
-3. As vLLM uses uv, make sure that `unsafe-best-match` strategy is set either
37
-via `UV_INDEX_STRATEGY` env variable or via `--index-strategy unsafe-best-match`.
32
+1. Update [requirements files](https://github.com/vllm-project/vllm/tree/main/requirements)
33
+to point to the new releases for `torch`, `torchvision`, and `torchaudio`.
34
+
35
+2. Use the following option to get the final release candidates' wheels. Some common platforms are `cpu`, `cu128`, and `rocm6.2.4`.
36
+
37
+ ```bash
38
+ --extra-index-url https://download.pytorch.org/whl/test/<PLATFORM>
39
+ ```
40
+
41
+3. Since vLLM uses `uv`, ensure the following index strategy is applied:
42
+
43
+ - Via environment variable:
44
+
45
+ ```bash
46
+ export UV_INDEX_STRATEGY=unsafe-best-match
47
+ ```
48
+
49
+ - Or via CLI flag:
50
+
51
+ ```bash
52
+ --index-strategy unsafe-best-match
53
+ ```
54
55
If failures are found in the pull request, raise them as issues on vLLM and
56
cc the PyTorch release team to initiate discussion on how to address them.
@@ -42,20 +58,25 @@ cc the PyTorch release team to initiate discussion on how to address them.
58
## Update CUDA version
59
60
The PyTorch release matrix includes both stable and experimental [CUDA versions](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix). Due to limitations, only the latest stable CUDA version (for example,
45
-torch2.7.0+cu12.6) is uploaded to PyPI. However, vLLM may require a different CUDA version,
61
+`torch2.7.0+cu12.6`) is uploaded to PyPI. However, vLLM may require a different CUDA version,
62
such as 12.8 for Blackwell support.
63
This complicates the process as we cannot use the out-of-the-box
64
`pip install torch torchvision torchaudio` command. The solution is to use
65
`--extra-index-url` in vLLM's Dockerfiles.
66
51
-1. Use `--extra-index-url https://download.pytorch.org/whl/cu128` to install torch+cu128.
52
-2. Other important indexes at the moment include:
53
- 1. CPU ‒ https://download.pytorch.org/whl/cpu
54
- 2. ROCm ‒ https://download.pytorch.org/whl/rocm6.2.4 and https://download.pytorch.org/whl/rocm6.3
55
- 3. XPU ‒ https://download.pytorch.org/whl/xpu
56
-3. Update .buildkite/release-pipeline.yaml and .buildkite/scripts/upload-wheels.sh to
57
-match the CUDA version from step 1. This makes sure that the release vLLM wheel is tested
58
-on CI.
67
+- Important indexes at the moment include:
68
+
69
+| Platform | `--extra-index-url` |
70
+|----------|-----------------|
71
+| CUDA 12.8| [https://download.pytorch.org/whl/cu128](https://download.pytorch.org/whl/cu128)|
72
+| CPU | [https://download.pytorch.org/whl/cpu](https://download.pytorch.org/whl/cpu)|
73
+| ROCm 6.2 | [https://download.pytorch.org/whl/rocm6.2.4](https://download.pytorch.org/whl/rocm6.2.4) |
74
+| ROCm 6.3 | [https://download.pytorch.org/whl/rocm6.3](https://download.pytorch.org/whl/rocm6.3) |
75
+| XPU | [https://download.pytorch.org/whl/xpu](https://download.pytorch.org/whl/xpu) |
76
+
77
+- Update the below files to match the CUDA version from step 1. This makes sure that the release vLLM wheel is tested on CI.
78
+ - `.buildkite/release-pipeline.yaml`
79
+ - `.buildkite/scripts/upload-wheels.sh`
80
81
## Address long vLLM build time
82
@@ -66,7 +87,7 @@ it doesn't populate the cache, so re-running it to warm up the cache
87
is ineffective.
88
89
While ongoing efforts like [#17419](gh-issue:17419)
69
-address the long build time at its source, the current workaround is to set VLLM_CI_BRANCH
90
+address the long build time at its source, the current workaround is to set `VLLM_CI_BRANCH`
91
to a custom branch provided by @khluu (`VLLM_CI_BRANCH=khluu/use_postmerge_q`)
92
when manually triggering a build on Buildkite. This branch accomplishes two things:
93
@@ -86,17 +107,18 @@ releases (which would take too much time), they can be built from
107
source to unblock the update process.
108
109
### FlashInfer
89
-Here is how to build and install it from source with torch2.7.0+cu128 in vLLM [Dockerfile](https://github.com/vllm-project/vllm/blob/27bebcd89792d5c4b08af7a65095759526f2f9e1/docker/Dockerfile#L259-L271):
110
+Here is how to build and install it from source with `torch2.7.0+cu128` in vLLM [Dockerfile](https://github.com/vllm-project/vllm/blob/27bebcd89792d5c4b08af7a65095759526f2f9e1/docker/Dockerfile#L259-L271):
111
112
```bash
113
export TORCH_CUDA_ARCH_LIST='7.5 8.0 8.9 9.0 10.0+PTX'
114
export FLASHINFER_ENABLE_SM90=1
94
-uv pip install --system --no-build-isolation "git+https://github.com/flashinfer-ai/flashinfer@v0.2.6.post1"
115
+uv pip install --system \
116
+ --no-build-isolation "git+https://github.com/flashinfer-ai/flashinfer@v0.2.6.post1"
117
```
118
119
One caveat is that building FlashInfer from source adds approximately 30
120
minutes to the vLLM build time. Therefore, it's preferable to cache the wheel in a
99
-public location for immediate installation, such as https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.6.post1%2Bcu128torch2.7-cp39-abi3-linux_x86_64.whl. For future releases, contact the PyTorch release
121
+public location for immediate installation, such as [this FlashInfer wheel link](https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.6.post1%2Bcu128torch2.7-cp39-abi3-linux_x86_64.whl). For future releases, contact the PyTorch release
122
team if you want to get the package published there.
123
124
### xFormers
@@ -104,13 +126,15 @@ Similar to FlashInfer, here is how to build and install xFormers from source:
126
127
```bash
128
export TORCH_CUDA_ARCH_LIST='7.0 7.5 8.0 8.9 9.0 10.0+PTX'
107
-MAX_JOBS=16 uv pip install --system --no-build-isolation "git+https://github.com/facebookresearch/xformers@v0.0.30"
129
+MAX_JOBS=16 uv pip install --system \
130
+ --no-build-isolation "git+https://github.com/facebookresearch/xformers@v0.0.30"
131
```
132
133
### Mamba
134
135
```bash
113
-uv pip install --system --no-build-isolation "git+https://github.com/state-spaces/mamba@v2.2.4"
136
+uv pip install --system \
137
+ --no-build-isolation "git+https://github.com/state-spaces/mamba@v2.2.4"
138
```
139
140
### causal-conv1d
@@ -125,6 +149,6 @@ Rather than attempting to update all vLLM platforms in a single pull request, it
149
to handle some platforms separately. The separation of requirements and Dockerfiles
150
for different platforms in vLLM CI/CD allows us to selectively choose
151
which platforms to update. For instance, updating XPU requires the corresponding
128
-release from https://github.com/intel/intel-extension-for-pytorch by Intel.
152
+release from [Intel Extension for PyTorch](https://github.com/intel/intel-extension-for-pytorch) by Intel.
153
While <gh-pr:16859> updated vLLM to PyTorch 2.7.0 on CPU, CUDA, and ROCm,
154
<gh-pr:17444> completed the update for XPU.