@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 267 lines 10.6 KB
Rendered Raw
1 # Parameter Sweeps
2
3 `vllm bench sweep` is a suite of commands designed to run benchmarks across multiple configurations and compare them by visualizing the results.
4
5 ## Online Benchmark
6
7 ### Basic
8
9 `vllm bench sweep serve` starts `vllm serve` and iteratively runs `vllm bench serve` for each server configuration.
10
11 !!! tip
12 If you only need to run benchmarks for a single server configuration, consider using [GuideLLM](https://github.com/vllm-project/guidellm), an established performance benchmarking framework with live progress updates and automatic report generation. It is also more flexible than `vllm bench serve` in terms of dataset loading, request formatting, and workload patterns.
13
14 Follow these steps to run the script:
15
16 1. Construct the base command to `vllm serve`, and pass it to the `--serve-cmd` option.
17 2. Construct the base command to `vllm bench serve`, and pass it to the `--bench-cmd` option.
18 3. (Optional) If you would like to vary the settings of `vllm serve`, create a new JSON file and populate it with the parameter combinations you want to test. Pass the file path to `--serve-params`.
19
20 - Example: Tuning `--max-num-seqs` and `--max-num-batched-tokens`:
21
22 ```json
23 [
24 {
25 "max_num_seqs": 32,
26 "max_num_batched_tokens": 1024
27 },
28 {
29 "max_num_seqs": 64,
30 "max_num_batched_tokens": 1024
31 },
32 {
33 "max_num_seqs": 64,
34 "max_num_batched_tokens": 2048
35 },
36 {
37 "max_num_seqs": 128,
38 "max_num_batched_tokens": 2048
39 },
40 {
41 "max_num_seqs": 128,
42 "max_num_batched_tokens": 4096
43 },
44 {
45 "max_num_seqs": 256,
46 "max_num_batched_tokens": 4096
47 }
48 ]
49 ```
50
51 4. (Optional) If you would like to vary the settings of `vllm bench serve`, create a new JSON file and populate it with the parameter combinations you want to test. Pass the file path to `--bench-params`.
52
53 - Example: Using different input/output lengths for random dataset:
54
55 ```json
56 [
57 {
58 "_benchmark_name": "scenario_A",
59 "random_input_len": 128,
60 "random_output_len": 32
61 },
62 {
63 "_benchmark_name": "scenario_B",
64 "random_input_len": 256,
65 "random_output_len": 64
66 },
67 {
68 "_benchmark_name": "scenario_C",
69 "random_input_len": 512,
70 "random_output_len": 128
71 }
72 ]
73 ```
74
75 5. Set `--output-dir` and optionally `--experiment-name` to control where to save the results.
76
77 Example command:
78
79 ```bash
80 vllm bench sweep serve \
81 --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \
82 --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json' \
83 --serve-params benchmarks/serve_hparams.json \
84 --bench-params benchmarks/bench_hparams.json \
85 --output-dir benchmarks/results \
86 --experiment-name demo
87 ```
88
89 By default, each parameter combination is benchmarked 3 times to make the results more reliable. You can adjust the number of runs by setting `--num-runs`.
90
91 You can optionally run an unmeasured warmup before each parameter combination by setting `--warmup-num-prompts`. The warmup result is saved as `warmup.json` for inspection, but is excluded from the measured summary.
92
93 You can use `--continue-on-error` to record a failed warmup, benchmark run, or server configuration and continue processing the remaining sweep combinations. Failure details are saved in `.failure.json` files.
94
95 Both options are also supported by `vllm bench sweep serve_workload`.
96
97 !!! important
98 If both `--serve-params` and `--bench-params` are passed, the script will iterate over the Cartesian product between them.
99 You can use `--dry-run` to preview the commands to be run.
100
101 We only start the server once for each `--serve-params`, and keep it running for multiple `--bench-params`.
102 Between each benchmark run, we call all `/reset_*_cache` endpoints to get a clean slate for the next run.
103 In case you are using a custom `--serve-cmd`, you can override the commands used for resetting the state by setting `--after-bench-cmd`.
104
105 !!! note
106 You should set `_benchmark_name` to provide a human-readable name for parameter combinations involving many variables.
107 This becomes mandatory if the file name would otherwise exceed the maximum path length allowed by the filesystem.
108
109 !!! tip
110 You can use the `--resume` option to continue the parameter sweep if an unexpected error occurs, e.g., timeout when connecting to HF Hub.
111
112 ### Workload Explorer
113
114 `vllm bench sweep serve_workload` is a variant of `vllm bench sweep serve` that explores different workload levels in order to find the tradeoff between latency and throughput. The results can also be [visualized](#visualization) to determine the feasible SLAs.
115
116 The workload can be expressed in terms of request rate or concurrency (choose using `--workload-var`).
117
118 Example command:
119
120 ```bash
121 vllm bench sweep serve_workload \
122 --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \
123 --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 100' \
124 --workload-var max_concurrency \
125 --serve-params benchmarks/serve_hparams.json \
126 --bench-params benchmarks/bench_hparams.json \
127 --num-runs 1 \
128 --output-dir benchmarks/results \
129 --experiment-name demo
130 ```
131
132 The algorithm for exploring different workload levels can be summarized as follows:
133
134 1. Run the benchmark by sending requests one at a time (serial inference, lowest workload). This results in the lowest possible latency and throughput.
135 2. Run the benchmark by sending all requests at once (batch inference, highest workload). This results in the highest possible latency and throughput.
136 3. Estimate the value of `workload_var` corresponding to Step 2.
137 4. Run the benchmark over intermediate values of `workload_var` uniformly using the remaining iterations.
138
139 You can override the number of iterations in the algorithm by setting `--workload-iters`.
140
141 !!! tip
142 This is our equivalent of [GuideLLM's `--profile sweep`](https://github.com/vllm-project/guidellm/blob/v0.5.3/src/guidellm/benchmark/profiles.py#L575).
143
144 In general, `--workload-var max_concurrency` produces more reliable results because it directly controls the workload imposed on the vLLM engine.
145 Nevertheless, we default to `--workload-var request_rate` to maintain similar behavior as GuideLLM.
146
147 ## Startup Benchmark
148
149 `vllm bench sweep startup` runs `vllm bench startup` across parameter combinations to compare cold/warm startup time for different engine settings.
150
151 Follow these steps to run the script:
152
153 1. (Optional) Construct the base command to `vllm bench startup`, and pass it to `--startup-cmd` (default: `vllm bench startup`).
154 2. (Optional) Reuse a `--serve-params` JSON from `vllm bench sweep serve` to vary engine settings. Only parameters supported by `vllm bench startup` are applied.
155 3. (Optional) Create a `--startup-params` JSON to vary startup-specific options like iteration counts.
156 4. Determine where you want to save the results, and pass that to `--output-dir`.
157
158 Example `--serve-params`:
159
160 ```json
161 [
162 {
163 "_benchmark_name": "tp1",
164 "model": "Qwen/Qwen3-0.6B",
165 "tensor_parallel_size": 1,
166 "gpu_memory_utilization": 0.9
167 },
168 {
169 "_benchmark_name": "tp2",
170 "model": "Qwen/Qwen3-0.6B",
171 "tensor_parallel_size": 2,
172 "gpu_memory_utilization": 0.9
173 }
174 ]
175 ```
176
177 Example `--startup-params`:
178
179 ```json
180 [
181 {
182 "_benchmark_name": "qwen3-0.6",
183 "num_iters_cold": 2,
184 "num_iters_warmup": 1,
185 "num_iters_warm": 2
186 }
187 ]
188 ```
189
190 Example command:
191
192 ```bash
193 vllm bench sweep startup \
194 --startup-cmd 'vllm bench startup --model Qwen/Qwen3-0.6B' \
195 --serve-params benchmarks/serve_hparams.json \
196 --startup-params benchmarks/startup_hparams.json \
197 --output-dir benchmarks/results \
198 --experiment-name demo
199 ```
200
201 !!! important
202 By default, unsupported parameters in `--serve-params` or `--startup-params` are ignored with a warning.
203 Use `--strict-params` to fail fast on unknown keys.
204
205 ## Visualization
206
207 ### Basic
208
209 `vllm bench sweep plot` can be used to plot performance curves from parameter sweep results.
210
211 Control the variables to plot via `--var-x` and `--var-y`, optionally applying `--filter-by` and `--bin-by` to the values. The plot is organized according to `--fig-by`, `--row-by`, `--col-by`, and `--curve-by`.
212
213 Example commands for visualizing [Workload Explorer](#workload-explorer) results:
214
215 ```bash
216 EXPERIMENT_DIR=${1:-"benchmarks/results/demo"}
217
218 # Latency increases as the workload increases
219 vllm bench sweep plot $EXPERIMENT_DIR \
220 --var-x max_concurrency \
221 --var-y median_ttft_ms \
222 --col-by _benchmark_name \
223 --curve-by max_num_seqs,max_num_batched_tokens \
224 --fig-name latency_curve
225
226 # Throughput saturates as workload increases
227 vllm bench sweep plot $EXPERIMENT_DIR \
228 --var-x max_concurrency \
229 --var-y total_token_throughput \
230 --col-by _benchmark_name \
231 --curve-by max_num_seqs,max_num_batched_tokens \
232 --fig-name throughput_curve
233
234 # Tradeoff between latency and throughput
235 vllm bench sweep plot $EXPERIMENT_DIR \
236 --var-x total_token_throughput \
237 --var-y median_ttft_ms \
238 --col-by _benchmark_name \
239 --curve-by max_num_seqs,max_num_batched_tokens \
240 --fig-name latency_throughput
241 ```
242
243 !!! tip
244 You can use `--dry-run` to preview the figures to be plotted.
245
246 ### Pareto chart
247
248 `vllm bench sweep plot_pareto` helps pick configurations that balance per-user and per-GPU throughput.
249
250 Higher concurrency or batch size can raise GPU efficiency (per-GPU), but can add per user latency; lower concurrency improves per-user rate but underutilizes GPUs; The Pareto frontier shows the best achievable pairs across your runs.
251
252 - x-axis: tokens/s/user = `output_throughput` ÷ concurrency (`--user-count-var`, default `max_concurrency`, fallback `max_concurrent_requests`).
253 - y-axis: tokens/s/GPU = `output_throughput` ÷ GPU count (`--gpu-count-var` if set; else gpu_count is TP×PP*DP).
254 - Output: a single figure at `OUTPUT_DIR/pareto/PARETO.png`.
255 - Show the configuration used in each data point `--label-by` (default: `max_concurrency,gpu_count`).
256
257 Example:
258
259 ```bash
260 EXPERIMENT_DIR=${1:-"benchmarks/results/demo"}
261
262 vllm bench sweep plot_pareto $EXPERIMENT_DIR \
263 --label-by max_concurrency,tensor_parallel_size,pipeline_parallel_size
264 ```
265
266 !!! tip
267 You can use `--dry-run` to preview the figures to be plotted.