| 1 | # Update PyTorch version on vLLM OSS CI/CD |
| 2 | |
| 3 | vLLM's current policy is to always use the latest PyTorch stable |
| 4 | release in CI/CD. It is standard practice to submit a PR to update the |
| 5 | PyTorch version as early as possible when a new [PyTorch stable |
| 6 | release](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-cadence) becomes available. |
| 7 | This process is non-trivial due to the gap between PyTorch |
| 8 | releases. Using <https://github.com/vllm-project/vllm/pull/16859> as an example, this document outlines common steps to achieve this |
| 9 | update along with a list of potential issues and how to address them. |
| 10 | |
| 11 | ## Test PyTorch release candidates (RCs) |
| 12 | |
| 13 | Updating PyTorch in vLLM after the official release is not |
| 14 | ideal because any issues discovered at that point can only be resolved |
| 15 | by waiting for the next release or by implementing hacky workarounds in vLLM. |
| 16 | The better solution is to test vLLM with PyTorch release candidates (RC) to ensure |
| 17 | compatibility before each release. |
| 18 | |
| 19 | PyTorch release candidates can be downloaded from [PyTorch test index](https://download.pytorch.org/whl/test). |
| 20 | For example, `torch2.7.0+cu12.8` RC can be installed using the following command: |
| 21 | |
| 22 | ```bash |
| 23 | uv pip install torch torchvision torchaudio \ |
| 24 | --index-url https://download.pytorch.org/whl/test/cu128 |
| 25 | ``` |
| 26 | |
| 27 | When the final RC is ready for testing, it will be announced to the community |
| 28 | on the [PyTorch dev-discuss forum](https://dev-discuss.pytorch.org/c/release-announcements). |
| 29 | After this announcement, we can begin testing vLLM integration by drafting a pull request |
| 30 | following this 3-step process: |
| 31 | |
| 32 | 1. Update [requirements files](https://github.com/vllm-project/vllm/tree/main/requirements) |
| 33 | to point to the new releases for `torch`, `torchvision`, and `torchaudio`. |
| 34 | |
| 35 | 2. Use the following option to get the final release candidates' wheels. Some common platforms are `cpu`, `cu128`, and `rocm6.2.4`. |
| 36 | |
| 37 | ```bash |
| 38 | --extra-index-url https://download.pytorch.org/whl/test/<PLATFORM> |
| 39 | ``` |
| 40 | |
| 41 | 3. Since vLLM uses `uv`, ensure the following index strategy is applied: |
| 42 | |
| 43 | - Via environment variable: |
| 44 | |
| 45 | ```bash |
| 46 | export UV_INDEX_STRATEGY=unsafe-best-match |
| 47 | ``` |
| 48 | |
| 49 | - Or via CLI flag: |
| 50 | |
| 51 | ```bash |
| 52 | --index-strategy unsafe-best-match |
| 53 | ``` |
| 54 | |
| 55 | If failures are found in the pull request, raise them as issues on vLLM and |
| 56 | cc the PyTorch release team to initiate discussion on how to address them. |
| 57 | |
| 58 | ## Update CUDA version |
| 59 | |
| 60 | The PyTorch release matrix includes both stable and experimental [CUDA versions](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix). Due to limitations, only the latest stable CUDA version (for example, torch `2.7.1+cu126`) is uploaded to PyPI. However, vLLM may require a different CUDA version, |
| 61 | such as 12.8 for Blackwell support. |
| 62 | This complicates the process as we cannot use the out-of-the-box |
| 63 | `pip install torch torchvision torchaudio` command. The solution is to use |
| 64 | `--extra-index-url` in vLLM's Dockerfiles. |
| 65 | |
| 66 | - Important indexes at the moment include: |
| 67 | |
| 68 | | Platform | `--extra-index-url` | |
| 69 | | -------- | ------------------- | |
| 70 | | CUDA 12.8 | [https://download.pytorch.org/whl/cu128](https://download.pytorch.org/whl/cu128) | |
| 71 | | CPU | [https://download.pytorch.org/whl/cpu](https://download.pytorch.org/whl/cpu) | |
| 72 | | ROCm 6.2 | [https://download.pytorch.org/whl/rocm6.2.4](https://download.pytorch.org/whl/rocm6.2.4) | |
| 73 | | ROCm 6.3 | [https://download.pytorch.org/whl/rocm6.3](https://download.pytorch.org/whl/rocm6.3) | |
| 74 | | XPU | [https://download.pytorch.org/whl/xpu](https://download.pytorch.org/whl/xpu) | |
| 75 | |
| 76 | - Update the below files to match the CUDA version from step 1. This makes sure that the release vLLM wheel is tested on CI. |
| 77 | - `.buildkite/release-pipeline.yaml` |
| 78 | - `.buildkite/scripts/upload-wheels.sh` |
| 79 | |
| 80 | ## Manually running vLLM builds on BuildKiteCI |
| 81 | |
| 82 | When building vLLM with a new PyTorch/CUDA version, the vLLM sccache S3 bucket |
| 83 | will not have any cached artifacts, which can cause CI build jobs to exceed 5 hours. |
| 84 | Furthermore, vLLM's fastcheck pipeline operates in read-only mode and does not |
| 85 | populate the cache, making it ineffective for cache warm-up purposes. |
| 86 | |
| 87 | To address this, manually trigger a build on Buildkite to accomplish two objectives: |
| 88 | |
| 89 | 1. Run the complete test suite against the PyTorch RC build by setting the environment variables: `RUN_ALL=1` and `NIGHTLY=1` |
| 90 | 2. Populate the vLLM sccache S3 bucket with compiled artifacts, enabling faster subsequent builds |
| 91 | |
| 92 | <p align="center" width="100%"> |
| 93 | <img width="60%" alt="Buildkite new build popup" src="https://github.com/user-attachments/assets/3b07f71b-bb18-4ca3-aeaf-da0fe79d315f" /> |
| 94 | </p> |
| 95 | |
| 96 | You can also trigger this build from the command line with |
| 97 | [`.buildkite/scripts/trigger-ci-build.sh`](../../../.buildkite/scripts/trigger-ci-build.sh) |
| 98 | (dry-run by default; pass `--execute` to actually trigger it). |
| 99 | |
| 100 | ## Test against PyTorch nightly |
| 101 | |
| 102 | The steps above test against a specific PyTorch RC/stable wheel pinned in the |
| 103 | requirements files. To instead build and run the CI suite against the latest |
| 104 | PyTorch **nightly** wheels, set the `TORCH_NIGHTLY=1` environment variable on |
| 105 | the build (or apply the `ready-torch-nightly` label to the PR). |
| 106 | |
| 107 | When `TORCH_NIGHTLY=1`, the base CI image is built against PyTorch nightly |
| 108 | (`image_build_torch_nightly.sh`, `PYTORCH_NIGHTLY=1`, CUDA 13.0) and tagged at |
| 109 | the normal image tag, so the entire existing pipeline runs on nightly torch -- |
| 110 | there is no separate pipeline section to trigger. Combine it with `RUN_ALL=1` |
| 111 | to run the full suite (the `ready-torch-nightly` label and |
| 112 | `trigger-ci-build.sh --torch-nightly` both set this for you). This is the |
| 113 | configuration to use for a scheduled "vLLM vs PyTorch nightly" run. |
| 114 | |
| 115 | Use `.buildkite/scripts/trigger-ci-build.sh --torch-nightly` to trigger it from |
| 116 | the command line. |
| 117 | |
| 118 | ## Update all the different vLLM platforms |
| 119 | |
| 120 | Rather than attempting to update all vLLM platforms in a single pull request, it's more manageable |
| 121 | to handle some platforms separately. The separation of requirements and Dockerfiles |
| 122 | for different platforms in vLLM CI/CD allows us to selectively choose |
| 123 | which platforms to update. For instance, updating XPU requires the corresponding |
| 124 | release from [Intel Extension for PyTorch](https://github.com/intel/intel-extension-for-pytorch) by Intel. |
| 125 | While <https://github.com/vllm-project/vllm/pull/16859> updated vLLM to PyTorch 2.7.0 on CPU, CUDA, and ROCm, |
| 126 | <https://github.com/vllm-project/vllm/pull/17444> completed the update for XPU. |