A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 01:00 UTC
Add the instruction to run e2e validation manually before release (#21023)
Signed-off-by: Huy Do <huydhn@gmail.com>
Huy Do committed
Jul 20, 2025 at 22:29 UTC
940af1f03a6d47415655ba32c0ba551b24161faa
1 file changed
+33
RELEASE.md
+33
@@ -52,3 +52,36 @@ After branch cut, we approach finalizing the release branch with clear criteria
52
* Release branch specific changes (e.g. change version identifiers or CI fixes)
53
54
Please note: **No feature work allowed for cherry picks**. All PRs that are considered for cherry-picks need to be merged on trunk, the only exception are Release branch specific changes.
55
+
56
+## Manual validations
57
+
58
+### E2E Performance Validation
59
+
60
+Before each release, we perform end-to-end performance validation to ensure no regressions are introduced. This validation uses the [vllm-benchmark workflow](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) on PyTorch CI.
61
+
62
+**Current Coverage:**
63
+* Models: Llama3, Llama4, and Mixtral
64
+* Hardware: NVIDIA H100 and AMD MI300x
65
+* *Note: Coverage may change based on new model releases and hardware availability*
66
+
67
+**Performance Validation Process:**
68
+
69
+**Step 1: Get Access**
70
+Request write access to the [pytorch/pytorch-integration-testing](https://github.com/pytorch/pytorch-integration-testing) repository to run the benchmark workflow.
71
+
72
+**Step 2: Review Benchmark Setup**
73
+Familiarize yourself with the benchmark configurations:
74
+* [CUDA setup](https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks/cuda)
75
+* [ROCm setup](https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks/rocm)
76
+
77
+**Step 3: Run the Benchmark**
78
+Navigate to the [vllm-benchmark workflow](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) and configure:
79
+* **vLLM branch**: Set to the release branch (e.g., `releases/v0.9.2`)
80
+* **vLLM commit**: Set to the RC commit hash
81
+
82
+**Step 4: Review Results**
83
+Once the workflow completes, benchmark results will be available on the [vLLM benchmark dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm) under the corresponding branch and commit.
84
+
85
+**Step 5: Performance Comparison**
86
+Compare the current results against the previous release to verify no performance regressions have occurred. Here is an
87
+example of [v0.9.1 vs v0.9.2](https://hud.pytorch.org/benchmark/llms?startTime=Thu%2C%2017%20Apr%202025%2021%3A43%3A50%20GMT&stopTime=Wed%2C%2016%20Jul%202025%2021%3A43%3A50%20GMT&granularity=week&lBranch=releases/v0.9.1&lCommit=b6553be1bc75f046b00046a4ad7576364d03c835&rBranch=releases/v0.9.2&rCommit=a5dd03c1ebc5e4f56f3c9d3dc0436e9c582c978f&repoName=vllm-project%2Fvllm&benchmarkName=&modelName=All%20Models&backendName=All%20Backends&modeName=All%20Modes&dtypeName=All%20DType&deviceName=All%20Devices&archName=All%20Platforms).