@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / b3aba04e5a
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

[Benchmark] Convenience script for multiple parameter combinations (#27085)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Oct 19, 2025 at 14:57 UTC b3aba04e5a04ede5218ef91fccab67d3bb9c1c1a
3 files changed +1312 -3
docs/contributing/benchmarks.md
+145 -3
@@ -6,7 +6,8 @@ toc_depth: 4
6
7 vLLM provides comprehensive benchmarking tools for performance testing and evaluation:
8
9 -- **[Benchmark CLI]**: `vllm bench` CLI tools and specialized benchmark scripts for interactive performance testing
9 +- **[Benchmark CLI](#benchmark-cli)**: `vllm bench` CLI tools and specialized benchmark scripts for interactive performance testing
10 +- **[Batch Scripts](#batch-scripts)**: Run `vllm bench` against multiple configurations conveniently
11 - **[Performance benchmarks](#performance-benchmarks)**: Automated CI benchmarks for development
12 - **[Nightly benchmarks](#nightly-benchmarks)**: Comparative benchmarks against alternatives
13
@@ -29,7 +30,7 @@ th {
30 | Dataset | Online | Offline | Data Path |
31 |---------|--------|---------|-----------|
32 | ShareGPT | ✅ | ✅ | `wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json` |
32 -| ShareGPT4V (Image) | ✅ | ✅ | `wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/blob/main/sharegpt4v_instruct_gpt4-vision_cap100k.json`<br>Note that the images need to be downloaded separately. For example, to download COCO's 2017 Train images:<br>`wget http://images.cocodataset.org/zips/train2017.zip` |
33 +| ShareGPT4V (Image) | ✅ | ✅ | `wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json`<br>Note that the images need to be downloaded separately. For example, to download COCO's 2017 Train images:<br>`wget http://images.cocodataset.org/zips/train2017.zip` |
34 | ShareGPT4Video (Video) | ✅ | ✅ | `git clone https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video` |
35 | BurstGPT | ✅ | ✅ | `wget https://github.com/HPMLL/BurstGPT/releases/download/v1.1/BurstGPT_without_fails_2.csv` |
36 | Sonnet (deprecated) | ✅ | ✅ | Local file: `benchmarks/sonnet.txt` |
@@ -714,7 +715,7 @@ Generate synthetic image inputs alongside random text prompts to stress-test vis
715
716 Notes:
717
717 -- Works only with online benchmark via the OpenAI backend (`--backend openai-chat`) and endpoint `/v1/chat/completions`.
718 +- Works only with online benchmark via the OpenAI backend (`--backend openai-chat`) and endpoint `/v1/chat/completions`.
719 - Video sampling is not yet implemented.
720
721 Start the server (example):
@@ -924,6 +925,147 @@ throughput numbers correctly is also adjusted.
925
926 </details>
927
928 +## Batch Scripts
929 +
930 +### Batch Serving Script
931 +
932 +[`vllm/benchmarks/serve_multi.py`](../../vllm/benchmarks/serve_multi.py) automatically starts `vllm serve` and runs `vllm bench serve` over multiple configurations.
933 +
934 +#### Batch Mode
935 +
936 +The basic purpose of this script is to evaluate vLLM under different settings. Follows these steps to run the script:
937 +
938 +1. Construct the base command to `vllm serve`, and pass it to the `--serve-cmd` option.
939 +2. Construct the base command to `vllm bench serve`, and pass it to the `--bench-cmd` option.
940 +3. (Optional) If you would like to vary the settings of `vllm serve`, create a new JSON file and populate it with the parameter combinations you want to test. Pass the file path to `--serve-params`.
941 +
942 + - Example: Tuning `--max-num-seqs` and `--max-num-batched-tokens`:
943 +
944 + ```json
945 + [
946 + {
947 + "max_num_seqs": 32,
948 + "max_num_batched_tokens": 1024
949 + },
950 + {
951 + "max_num_seqs": 64,
952 + "max_num_batched_tokens": 1024
953 + },
954 + {
955 + "max_num_seqs": 64,
956 + "max_num_batched_tokens": 2048
957 + },
958 + {
959 + "max_num_seqs": 128,
960 + "max_num_batched_tokens": 2048
961 + },
962 + {
963 + "max_num_seqs": 128,
964 + "max_num_batched_tokens": 4096
965 + },
966 + {
967 + "max_num_seqs": 256,
968 + "max_num_batched_tokens": 4096
969 + }
970 + ]
971 + ```
972 +
973 +4. (Optional) If you would like to vary the settings of `vllm bench serve`, create a new JSON file and populate it with the parameter combinations you want to test. Pass the file path to `--bench-params`.
974 +
975 + - Example: Using different input/output lengths for random dataset:
976 +
977 + ```json
978 + [
979 + {
980 + "random_input_len": 128,
981 + "random_output_len": 32
982 + },
983 + {
984 + "random_input_len": 256,
985 + "random_output_len": 64
986 + },
987 + {
988 + "random_input_len": 512,
989 + "random_output_len": 128
990 + }
991 + ]
992 + ```
993 +
994 +5. Determine where you want to save the results, and pass that to `--output-dir`.
995 +
996 +Example command:
997 +
998 +```bash
999 +python vllm/benchmarks/serve_multi.py \
1000 + --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \
1001 + --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json' \
1002 + --serve-params benchmarks/serve_hparams.json \
1003 + --bench-params benchmarks/bench_hparams.json \
1004 + -o benchmarks/results
1005 +```
1006 +
1007 +!!! important
1008 + If both `--serve-params` and `--bench-params` are passed, the script will iterate over the Cartesian product between them.
1009 + You can use `--dry-run` to preview the commands to be run.
1010 +
1011 + We only start the server once for each `--serve-params`, and keep it running for multiple `--bench-params`.
1012 + Between each benchmark run, we call the `/reset_prefix_cache` and `/reset_mm_cache` endpoints to get a clean slate for the next run.
1013 + In case you are using a custom `--serve-cmd`, you can override the commands used for resetting the state by setting `--after-bench-cmd`.
1014 +
1015 +!!! note
1016 + By default, each parameter combination is run 3 times to make the results more reliable. You can adjust the number of runs by setting `--num-runs`.
1017 +
1018 +!!! tip
1019 + You can use the `--resume` option to continue the parameter sweep if one of the runs failed.
1020 +
1021 +#### SLA Mode
1022 +
1023 +By passing SLA constraints via `--sla-params`, you can run this script in SLA mode, causing it to adjust either the request rate or concurrency (choose using `--sla-variable`) in order to satisfy the SLA constraints.
1024 +
1025 +For example, to ensure E2E latency within different target values for 99% of requests:
1026 +
1027 +```json
1028 +[
1029 + {
1030 + "p99_e2el_ms": "<=200"
1031 + },
1032 + {
1033 + "p99_e2el_ms": "<=500"
1034 + },
1035 + {
1036 + "p99_e2el_ms": "<=1000"
1037 + },
1038 + {
1039 + "p99_e2el_ms": "<=2000"
1040 + }
1041 +]
1042 +```
1043 +
1044 +Example command:
1045 +
1046 +```bash
1047 +python vllm/benchmarks/serve_multi.py \
1048 + --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \
1049 + --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json' \
1050 + --serve-params benchmarks/serve_hparams.json \
1051 + --bench-params benchmarks/bench_hparams.json \
1052 + --sla-params benchmarks/sla_hparams.json \
1053 + --sla-variable max_concurrency \
1054 + -o benchmarks/results
1055 +```
1056 +
1057 +The algorithm for adjusting the SLA variable is as follows:
1058 +
1059 +1. Run the benchmark with infinite QPS, and use the corresponding metrics to determine the initial value of the variable.
1060 + - For example, the initial request rate is set to the concurrency under infinite QPS.
1061 +2. If the SLA is still satisfied, keep doubling the value until the SLA is no longer satisfied. This gives a relatively narrow window that contains the point where the SLA is barely satisfied.
1062 +3. Apply binary search over the window to find the maximum value that still satisfies the SLA.
1063 +
1064 +!!! important
1065 + SLA tuning is applied over each combination of `--serve-params`, `--bench-params`, and `--sla-params`.
1066 +
1067 + For a given combination of `--serve-params` and `--bench-params`, we share the benchmark results across `--sla-params` to avoid rerunning benchmarks with the same SLA variable value.
1068 +
1069 ## Performance Benchmarks
1070
1071 The performance benchmarks are used for development to confirm whether new changes improve performance under various workloads. They are triggered on every commit with both the `perf-benchmarks` and `ready` labels, and when a PR is merged into vLLM.
vllm/benchmarks/serve_multi.py new
+1157
@@ -0,0 +1,1157 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +import argparse
4 +import contextlib
5 +import json
6 +import math
7 +import os
8 +import shlex
9 +import signal
10 +import subprocess
11 +from abc import ABC, abstractmethod
12 +from datetime import datetime
13 +from pathlib import Path
14 +from typing import Literal, get_args
15 +
16 +import pandas as pd
17 +import requests
18 +import seaborn as sns
19 +from typing_extensions import assert_never, override
20 +
21 +_BAD_PARAMS_TYPE_MSG = (
22 + "The parameters to vary should be expressed as a JSON list of dictionaries."
23 +)
24 +
25 +
26 +def _parse_params(params: list[dict[str, object]]):
27 + if not isinstance(params, list):
28 + raise TypeError(f"{_BAD_PARAMS_TYPE_MSG} Found JSON type {type(params)}")
29 +
30 + for comb in params:
31 + if not isinstance(comb, dict):
32 + raise TypeError(f"{_BAD_PARAMS_TYPE_MSG} Found item type {type(comb)}")
33 +
34 + return params
35 +
36 +
37 +class SLACriterionBase(ABC):
38 + def __init__(self, target: float) -> None:
39 + super().__init__()
40 +
41 + self.target = target
42 +
43 + @abstractmethod
44 + def validate(self, actual: float) -> bool:
45 + """Return `True` if this criterion is met; otherwise `False`."""
46 + raise NotImplementedError
47 +
48 + @abstractmethod
49 + def format_cond(self, lhs: str) -> str:
50 + raise NotImplementedError
51 +
52 + def print_and_validate(
53 + self,
54 + metrics: dict[str, float],
55 + metrics_key: str,
56 + ) -> bool:
57 + metric = metrics[metrics_key]
58 + result = self.validate(metric)
59 +
60 + cond = self.format_cond(f"{metrics_key} = {metric:.2f}")
61 + print(f"Validating SLA: {cond} | " + ("PASSED" if result else "FAILED"))
62 +
63 + return result
64 +
65 +
66 +class SLALessThan(SLACriterionBase):
67 + @override
68 + def validate(self, actual: float) -> bool:
69 + return actual < self.target
70 +
71 + @override
72 + def format_cond(self, lhs: str) -> str:
73 + return f"{lhs}<{self.target:.2f}"
74 +
75 +
76 +class SLALessThanOrEqual(SLACriterionBase):
77 + @override
78 + def validate(self, actual: float) -> bool:
79 + return actual <= self.target
80 +
81 + @override
82 + def format_cond(self, lhs: str) -> str:
83 + return f"{lhs}<={self.target:.2f}"
84 +
85 +
86 +class SLAGreaterThan(SLACriterionBase):
87 + @override
88 + def validate(self, actual: float) -> bool:
89 + return actual > self.target
90 +
91 + @override
92 + def format_cond(self, lhs: str) -> str:
93 + return f"{lhs}>{self.target:.2f}"
94 +
95 +
96 +class SLAGreaterThanOrEqual(SLACriterionBase):
97 + @override
98 + def validate(self, actual: float) -> bool:
99 + return actual >= self.target
100 +
101 + @override
102 + def format_cond(self, lhs: str) -> str:
103 + return f"{lhs}>={self.target:.2f}"
104 +
105 +
106 +# NOTE: The ordering is important! Match longer op_keys first
107 +SLA_CRITERIA: dict[str, type[SLACriterionBase]] = {
108 + "<=": SLALessThanOrEqual,
109 + ">=": SLAGreaterThanOrEqual,
110 + "<": SLALessThan,
111 + ">": SLAGreaterThan,
112 +}
113 +
114 +
115 +def _parse_sla_item(sla_item: dict[str, str]):
116 + sla_criteria: dict[str, SLACriterionBase] = {}
117 +
118 + for metric_key, metric_value in sla_item.items():
119 + for op_key in SLA_CRITERIA:
120 + if metric_value.startswith(op_key):
121 + sla_criteria[metric_key] = SLA_CRITERIA[op_key](
122 + float(metric_value.removeprefix(op_key))
123 + )
124 + break
125 + else:
126 + raise ValueError(
127 + f"Invalid operator for SLA constraint '{metric_key}={metric_value}'. "
128 + f"Valid operators are: {set(SLA_CRITERIA)}",
129 + )
130 +
131 + return sla_criteria
132 +
133 +
134 +def _parse_sla(sla: list[dict[str, str]]):
135 + return [_parse_sla_item(item) for item in sla]
136 +
137 +
138 +# In JSON, we prefer "_"
139 +def _iter_param_key_candidates(param_key: str):
140 + yield param_key
141 + yield param_key.replace("-", "_")
142 + yield param_key.replace("_", "-")
143 +
144 +
145 +# In CLI, we prefer "-"
146 +def _iter_cmd_key_candidates(param_key: str):
147 + for k in reversed(tuple(_iter_param_key_candidates(param_key))):
148 + yield "--" + k
149 +
150 +
151 +def _normalize_cmd_key(param_key: str):
152 + return next(_iter_cmd_key_candidates(param_key))
153 +
154 +
155 +def _override_args(cmd: list[str], params: dict[str, object]):
156 + cmd = list(cmd)
157 +
158 + for k, v in params.items():
159 + for k_candidate in _iter_cmd_key_candidates(k):
160 + try:
161 + k_idx = cmd.index(k_candidate)
162 +
163 + if isinstance(v, bool):
164 + cmd[k_idx] = _normalize_cmd_key(k if v else "no-" + k)
165 + else:
166 + cmd[k_idx + 1] = str(v)
167 +
168 + break
169 + except ValueError:
170 + continue
171 + else:
172 + if isinstance(v, bool):
173 + cmd.append(_normalize_cmd_key(k if v else "no-" + k))
174 + else:
175 + cmd.extend([_normalize_cmd_key(k), str(v)])
176 +
177 + return cmd
178 +
179 +
180 +class ServerWrapper:
181 + def __init__(
182 + self,
183 + server_cmd: list[str],
184 + after_bench_cmd: list[str],
185 + *,
186 + show_stdout: bool,
187 + ) -> None:
188 + super().__init__()
189 +
190 + self.server_cmd = server_cmd
191 + self.after_bench_cmd = after_bench_cmd
192 + self.show_stdout = show_stdout
193 +
194 + def run_subcommand(self, cmd: list[str]):
195 + return subprocess.run(
196 + cmd,
197 + stdout=None if self.show_stdout else subprocess.DEVNULL,
198 + check=True,
199 + )
200 +
201 + def after_bench(self) -> None:
202 + if not self.after_bench_cmd:
203 + self.reset_caches()
204 + return
205 +
206 + self.run_subcommand(self.after_bench_cmd)
207 +
208 + def _get_vllm_server_address(self) -> str:
209 + server_cmd = self.server_cmd
210 +
211 + for host_key in ("--host",):
212 + if host_key in server_cmd:
213 + host = server_cmd[server_cmd.index(host_key) + 1]
214 + break
215 + else:
216 + host = "localhost"
217 +
218 + for port_key in ("-p", "--port"):
219 + if port_key in server_cmd:
220 + port = int(server_cmd[server_cmd.index(port_key) + 1])
221 + break
222 + else:
223 + port = 8000 # The default value in vllm serve
224 +
225 + return f"http://{host}:{port}"
226 +
227 + def reset_caches(self) -> None:
228 + server_cmd = self.server_cmd
229 +
230 + # Use `.endswith()` to match `/bin/...`
231 + if server_cmd[0].endswith("vllm"):
232 + server_address = self._get_vllm_server_address()
233 + print(f"Resetting caches at {server_address}")
234 +
235 + res = requests.post(f"{server_address}/reset_prefix_cache")
236 + res.raise_for_status()
237 +
238 + res = requests.post(f"{server_address}/reset_mm_cache")
239 + res.raise_for_status()
240 + elif server_cmd[0].endswith("infinity_emb"):
241 + if "--vector-disk-cache" in server_cmd:
242 + raise NotImplementedError(
243 + "Infinity server uses caching but does not expose a method "
244 + "to reset the cache"
245 + )
246 + else:
247 + raise NotImplementedError(
248 + f"No implementation of `reset_caches` for `{server_cmd[0]}` server. "
249 + "Please specify a custom command via `--after-bench-cmd`."
250 + )
251 +
252 +
253 +@contextlib.contextmanager
254 +def _run_server(
255 + serve_cmd: list[str],
256 + after_bench_cmd: list[str],
257 + *,
258 + show_stdout: bool,
259 + serve_overrides: dict[str, object],
260 + dry_run: bool,
261 +):
262 + server_cmd = _override_args(serve_cmd, serve_overrides)
263 +
264 + print("[BEGIN SERVER]")
265 + print(f"Server overrides: {serve_overrides}")
266 + print(f"Server command: {server_cmd}")
267 +
268 + if dry_run:
269 + yield None
270 + print("[END SERVER]")
271 + return
272 +
273 + # Create new process group for clean termination
274 + server_process = subprocess.Popen(
275 + server_cmd,
276 + start_new_session=True,
277 + stdout=None if show_stdout else subprocess.DEVNULL,
278 + # Need VLLM_SERVER_DEV_MODE=1 for `_reset_caches`
279 + env={**os.environ, "VLLM_SERVER_DEV_MODE": "1"},
280 + )
281 +
282 + try:
283 + yield ServerWrapper(
284 + server_cmd,
285 + after_bench_cmd,
286 + show_stdout=show_stdout,
287 + )
288 + finally:
289 + if server_process.poll() is None:
290 + # In case only some processes have been terminated
291 + with contextlib.suppress(ProcessLookupError):
292 + # We need to kill both API Server and Engine processes
293 + os.killpg(os.getpgid(server_process.pid), signal.SIGKILL)
294 +
295 + print("[END SERVER]")
296 +
297 +
298 +def _run_benchmark(
299 + server: ServerWrapper | None,
300 + bench_cmd: list[str],
301 + *,
302 + serve_overrides: dict[str, object],
303 + bench_overrides: dict[str, object],
304 + run_number: int,
305 + output_path: Path,
306 + dry_run: bool,
307 +):
308 + benchmark_cmd = [
309 + *_override_args(bench_cmd, bench_overrides),
310 + "--save-result",
311 + "--result-dir",
312 + str(output_path.parent),
313 + "--result-filename",
314 + output_path.name,
315 + ]
316 +
317 + print("[BEGIN BENCHMARK]")
318 + print(f"Benchmark overrides: {bench_overrides}")
319 + print(f"Run Number: {run_number}")
320 + print(f"Benchmark command: {benchmark_cmd}")
321 + print(f"Output file: {output_path}")
322 +
323 + run_data: dict[str, object]
324 +
325 + if output_path.exists():
326 + print("Found existing results. Skipping.")
327 +
328 + with output_path.open("rb") as f:
329 + run_data = json.load(f)
330 + return run_data
331 +
332 + if server is None:
333 + assert dry_run
334 + print("[END BENCHMARK]")
335 + return None
336 +
337 + output_path.parent.mkdir(parents=True, exist_ok=True)
338 +
339 + server.run_subcommand(benchmark_cmd)
340 + server.after_bench()
341 +
342 + with output_path.open("rb") as f:
343 + run_data = json.load(f)
344 +
345 + run_data["run_number"] = run_number
346 + run_data.update(serve_overrides)
347 +
348 + with output_path.open("w") as f:
349 + json.dump(run_data, f, indent=4)
350 +
351 + print("[END BENCHMARK]")
352 +
353 + return run_data
354 +
355 +
356 +def _get_comb_base_path(
357 + output_dir: Path,
358 + serve_comb: dict[str, object],
359 + bench_comb: dict[str, object],
360 +):
361 + return output_dir / "-".join(
362 + (
363 + "SERVE",
364 + *(f"{k}={v}" for k, v in serve_comb.items()),
365 + "BENCH",
366 + *(f"{k}={v}" for k, v in bench_comb.items()),
367 + )
368 + ).replace("/", "_").replace("..", "__") # Sanitize
369 +
370 +
371 +def _get_comb_run_path(base_path: Path, run_number: int | None):
372 + if run_number is None:
373 + return base_path / "summary.json"
374 +
375 + return base_path / f"run={run_number}.json"
376 +
377 +
378 +def _comb_needs_server(
379 + serve_comb: dict[str, object],
380 + bench_combs: list[dict[str, object]],
381 + output_dir: Path,
382 +):
383 + for bench_comb in bench_combs:
384 + base_path = _get_comb_base_path(output_dir, serve_comb, bench_comb)
385 + if not _get_comb_run_path(base_path, run_number=None).exists():
386 + return True
387 +
388 + return False
389 +
390 +
391 +def _run_comb(
392 + server: ServerWrapper | None,
393 + bench_cmd: list[str],
394 + *,
395 + serve_comb: dict[str, object],
396 + bench_comb: dict[str, object],
397 + base_path: Path,
398 + num_runs: int,
399 + dry_run: bool,
400 +):
401 + comb_data = list[dict[str, object]]()
402 +
403 + for run_number in range(num_runs):
404 + run_data = _run_benchmark(
405 + server,
406 + bench_cmd,
407 + serve_overrides=serve_comb,
408 + bench_overrides=bench_comb,
409 + run_number=run_number,
410 + output_path=_get_comb_run_path(base_path, run_number),
411 + dry_run=dry_run,
412 + )
413 +
414 + if run_data is not None:
415 + comb_data.append(run_data)
416 +
417 + if dry_run:
418 + return None
419 +
420 + with _get_comb_run_path(base_path, run_number=None).open("w") as f:
421 + json.dump(comb_data, f, indent=4)
422 +
423 + return comb_data
424 +
425 +
426 +def run_combs(
427 + serve_cmd: list[str],
428 + bench_cmd: list[str],
429 + after_bench_cmd: list[str],
430 + *,
431 + show_stdout: bool,
432 + serve_params: list[dict[str, object]],
433 + bench_params: list[dict[str, object]],
434 + output_dir: Path,
435 + num_runs: int,
436 + dry_run: bool,
437 +):
438 + all_data = list[dict[str, object]]()
439 + for serve_comb in serve_params:
440 + with (
441 + _run_server(
442 + serve_cmd,
443 + after_bench_cmd,
444 + show_stdout=show_stdout,
445 + serve_overrides=serve_comb,
446 + dry_run=dry_run,
447 + )
448 + if _comb_needs_server(serve_comb, bench_params, output_dir)
449 + else contextlib.nullcontext()
450 + ) as server:
451 + for bench_comb in bench_params:
452 + base_path = _get_comb_base_path(output_dir, serve_comb, bench_comb)
453 +
454 + comb_data = _run_comb(
455 + server,
456 + bench_cmd,
457 + serve_comb=serve_comb,
458 + bench_comb=bench_comb,
459 + base_path=base_path,
460 + num_runs=num_runs,
461 + dry_run=dry_run,
462 + )
463 +
464 + if comb_data is not None:
465 + all_data.extend(comb_data)
466 +
467 + if dry_run:
468 + return None
469 +
470 + combined_df = pd.DataFrame.from_records(all_data)
471 + combined_df.to_csv(output_dir / "summary.csv")
472 +
473 + return combined_df
474 +
475 +
476 +def _get_sla_base_path(
477 + output_dir: Path,
478 + serve_comb: dict[str, object],
479 + bench_comb: dict[str, object],
480 +):
481 + return output_dir / "-".join(
482 + (
483 + "SERVE",
484 + *(f"{k}={v}" for k, v in serve_comb.items()),
485 + "BENCH",
486 + *(f"{k}={v}" for k, v in bench_comb.items()),
487 + )
488 + ).replace("/", "_").replace("..", "__") # Sanitize
489 +
490 +
491 +def _get_sla_iter_path(
492 + base_path: Path,
493 + sla_comb: dict[str, SLACriterionBase],
494 + sla_variable: str,
495 + sla_value: int | None,
496 +):
497 + if sla_value is None:
498 + prefix = "-".join(v.format_cond(k) for k, v in sla_comb.items())
499 + return base_path / f"SLA-{prefix}.json"
500 +
501 + return base_path / f"{sla_variable}={sla_value}"
502 +
503 +
504 +def _get_sla_run_path(iter_path: Path, run_number: int | None):
505 + if run_number is None:
506 + return iter_path / "summary.json"
507 +
508 + return iter_path / f"run={run_number}.json"
509 +
510 +
511 +def _sla_needs_server(
512 + serve_comb: dict[str, object],
513 + bench_combs: list[dict[str, object]],
514 + sla_combs: list[dict[str, SLACriterionBase]],
515 + sla_variable: str,
516 + output_dir: Path,
517 +):
518 + for bench_comb in bench_combs:
519 + base_path = _get_sla_base_path(output_dir, serve_comb, bench_comb)
520 + for sla_comb in sla_combs:
521 + if not _get_sla_iter_path(
522 + base_path,
523 + sla_comb,
524 + sla_variable,
525 + sla_value=None,
526 + ).exists():
527 + return True
528 +
529 + return False
530 +
531 +
532 +def _run_sla(
533 + server: ServerWrapper | None,
534 + bench_cmd: list[str],
535 + *,
536 + serve_comb: dict[str, object],
537 + bench_comb: dict[str, object],
538 + iter_path: Path,
539 + num_runs: int,
540 + dry_run: bool,
541 +):
542 + iter_data = list[dict[str, object]]()
543 +
544 + for run_number in range(num_runs):
545 + run_data = _run_benchmark(
546 + server,
547 + bench_cmd,
548 + serve_overrides=serve_comb,
549 + bench_overrides=bench_comb,
550 + run_number=run_number,
551 + output_path=_get_sla_run_path(iter_path, run_number),
552 + dry_run=dry_run,
553 + )
554 +
555 + if run_data is not None:
556 + iter_data.append(run_data)
557 +
558 + if dry_run:
559 + return None
560 +
561 + with _get_sla_run_path(iter_path, run_number=None).open("w") as f:
562 + json.dump(iter_data, f, indent=4)
563 +
564 + return iter_data
565 +
566 +
567 +SLAVariable = Literal["request_rate", "max_concurrency"]
568 +
569 +
570 +def _estimate_sla_value(run_data: dict[str, object], sla_variable: SLAVariable):
571 + request_throughput = float(run_data["request_throughput"]) # type: ignore
572 + if sla_variable == "request_rate":
573 + return request_throughput
574 + if sla_variable == "max_concurrency":
575 + mean_latency_ms = float(run_data["mean_e2el_ms"]) # type: ignore
576 + return request_throughput * mean_latency_ms / 1000
577 +
578 + assert_never(sla_variable)
579 +
580 +
581 +def _estimate_sla_bounds(
582 + server: ServerWrapper | None,
583 + bench_cmd: list[str],
584 + *,
585 + serve_comb: dict[str, object],
586 + bench_comb: dict[str, object],
587 + sla_comb: dict[str, SLACriterionBase],
588 + base_path: Path,
589 + num_runs: int,
590 + dry_run: bool,
591 + sla_variable: SLAVariable,
592 + init_value: int,
593 + max_value: int,
594 +):
595 + sla_data = list[dict[str, object]]()
596 +
597 + max_passing: int = 0
598 + min_failing: int = 0
599 +
600 + val: int = init_value
601 + assert val > 0
602 +
603 + while True:
604 + print(f"Testing {sla_variable}: {val} req/s")
605 +
606 + iter_data = _run_sla(
607 + server,
608 + bench_cmd,
609 + serve_comb=serve_comb,
610 + bench_comb={**bench_comb, sla_variable: val},
611 + iter_path=_get_sla_iter_path(base_path, sla_comb, sla_variable, val),
612 + num_runs=num_runs,
613 + dry_run=dry_run,
614 + )
615 +
616 + assert iter_data is not None
617 + sla_data.extend(iter_data)
618 +
619 + iter_data_mean = {
620 + k: sum(float(run_data[k]) for run_data in iter_data) / len(iter_data) # type: ignore
621 + for k in sla_comb
622 + }
623 +
624 + sla_results = [
625 + criterion.print_and_validate(iter_data_mean, k)
626 + for k, criterion in sla_comb.items()
627 + ]
628 +
629 + if all(sla_results):
630 + print("SLA criteria are met.")
631 + max_passing = val
632 + val *= 2
633 + else:
634 + print("SLA criteria are not met.")
635 + min_failing = val
636 + break
637 +
638 + if val >= max_value:
639 + break
640 +
641 + return sla_data, (max_passing, min_failing)
642 +
643 +
644 +def _find_sla_value(
645 + server: ServerWrapper | None,
646 + bench_cmd: list[str],
647 + *,
648 + serve_comb: dict[str, object],
649 + bench_comb: dict[str, object],
650 + sla_comb: dict[str, SLACriterionBase],
651 + base_path: Path,
652 + num_runs: int,
653 + dry_run: bool,
654 + sla_variable: SLAVariable,
655 + min_value: int,
656 + max_value: int,
657 +):
658 + sla_data = list[dict[str, object]]()
659 +
660 + left: int = min_value
661 + right: int = max_value
662 +
663 + while True:
664 + val = (left + right) // 2
665 + print(f"Testing {sla_variable}: {val} req/s")
666 +
667 + iter_data = _run_sla(
668 + server,
669 + bench_cmd,
670 + serve_comb=serve_comb,
671 + bench_comb={**bench_comb, sla_variable: val},
672 + iter_path=_get_sla_iter_path(base_path, sla_comb, sla_variable, val),
673 + num_runs=num_runs,
674 + dry_run=dry_run,
675 + )
676 +
677 + assert iter_data is not None
678 + sla_data.extend(iter_data)
679 +
680 + iter_data_mean = {
681 + k: sum(float(run_data[k]) for run_data in iter_data) / len(iter_data) # type: ignore
682 + for k in sla_comb
683 + }
684 +
685 + sla_results = [
686 + criterion.print_and_validate(iter_data_mean, k)
687 + for k, criterion in sla_comb.items()
688 + ]
689 +
690 + if all(sla_results):
691 + print("SLA criteria are met.")
692 + left = val
693 + else:
694 + print("SLA criteria are not met.")
695 + right = val
696 +
697 + if right - left <= 1:
698 + break
699 +
700 + return sla_data, left
701 +
702 +
703 +def _search_sla(
704 + server: ServerWrapper | None,
705 + bench_cmd: list[str],
706 + *,
707 + serve_comb: dict[str, object],
708 + bench_comb: dict[str, object],
709 + sla_comb: dict[str, SLACriterionBase],
710 + sla_variable: SLAVariable,
711 + sla_inf_value: int = 65536, # The value that represents infinite QPS
712 + base_path: Path,
713 + num_runs: int,
714 + dry_run: bool,
715 +):
716 + print("[SLA START]")
717 + print(f"SLA criteria: {', '.join(v.format_cond(k) for k, v in sla_comb.items())}")
718 +
719 + sla_data_0 = _run_sla(
720 + server,
721 + bench_cmd,
722 + serve_comb=serve_comb,
723 + bench_comb={**bench_comb, sla_variable: sla_inf_value},
724 + iter_path=_get_sla_iter_path(base_path, sla_comb, sla_variable, sla_inf_value),
725 + num_runs=num_runs,
726 + dry_run=dry_run,
727 + )
728 + if sla_data_0 is None:
729 + assert dry_run
730 + print("Omitting SLA search.")
731 + print("[SLA END]")
732 + return None
733 +
734 + sla_init_value = math.ceil(
735 + sum(_estimate_sla_value(item, sla_variable) for item in sla_data_0)
736 + / len(sla_data_0)
737 + )
738 + print(f"Initial {sla_variable} to search: {sla_init_value} req/s.")
739 +
740 + sla_data_1, (sla_min, sla_max) = _estimate_sla_bounds(
741 + server,
742 + bench_cmd,
743 + serve_comb=serve_comb,
744 + bench_comb=bench_comb,
745 + sla_comb=sla_comb,
746 + base_path=base_path,
747 + num_runs=num_runs,
748 + dry_run=dry_run,
749 + sla_variable=sla_variable,
750 + init_value=sla_init_value,
751 + max_value=sla_inf_value,
752 + )
753 + print(f"Range of {sla_variable} to search: [{sla_min}, {sla_max}] req/s.")
754 +
755 + sla_data_2, sla_value = _find_sla_value(
756 + server,
757 + bench_cmd,
758 + serve_comb=serve_comb,
759 + bench_comb=bench_comb,
760 + sla_comb=sla_comb,
761 + base_path=base_path,
762 + num_runs=num_runs,
763 + dry_run=dry_run,
764 + sla_variable=sla_variable,
765 + min_value=sla_min,
766 + max_value=sla_max,
767 + )
768 +
769 + sla_data = sla_data_0 + sla_data_1 + sla_data_2
770 + print(f"Maximum {sla_variable} for SLA: {sla_value} req/s.")
771 +
772 + with _get_sla_iter_path(
773 + base_path,
774 + sla_comb,
775 + sla_variable,
776 + sla_value=None,
777 + ).open("w") as f:
778 + json.dump(sla_data, f, indent=4)
779 +
780 + print("[SLA END]")
781 +
782 + return sla_data
783 +
784 +
785 +def _plot_throughput_latency_curve(
786 + all_data: list[dict[str, object]],
787 + serve_combs: list[dict[str, object]],
788 + bench_comb: dict[str, object],
789 + output_dir: Path,
790 +):
791 + fig_path = output_dir / "-".join(
792 + (
793 + "BENCH",
794 + *(f"{k}={v}" for k, v in bench_comb.items()),
795 + )
796 + ).replace("/", "_").replace("..", "__") # Sanitize
797 +
798 + df = pd.DataFrame.from_records(
799 + [item for item in all_data if all(item[k] == bench_comb[k] for k in bench_comb)]
800 + )
801 +
802 + # Group together points with similar throughput
803 + df["request_throughput"] = df["request_throughput"].round()
804 +
805 + # Preserve the key order using dictionary
806 + all_comb_keys = {k: None for comb in serve_combs for k in comb}
807 + for k in all_comb_keys:
808 + df[k] = df[k].astype(str)
809 +
810 + keys_per_comb = [comb.keys() for comb in serve_combs]
811 + if (
812 + all(ks == keys_per_comb[0] for ks in keys_per_comb)
813 + and len(keys_per_comb[0]) <= 3
814 + ):
815 + hue, style, size, *_ = (*keys_per_comb[0], None, None)
816 + ax = sns.lineplot(
817 + df,
818 + x="request_throughput",
819 + y="p99_e2el_ms",
820 + hue=hue,
821 + style=style,
822 + size=size,
823 + markers=True,
824 + )
825 + else:
826 + df["category"] = df[list(all_comb_keys)].agg("-".join, axis=1)
827 + ax = sns.lineplot(
828 + df,
829 + x="request_throughput",
830 + y="p99_e2el_ms",
831 + hue="category",
832 + markers=True,
833 + )
834 +
835 + sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
836 +
837 + fig = ax.get_figure()
838 + assert fig is not None
839 +
840 + fig.tight_layout()
841 + fig.savefig(fig_path)
842 +
843 +
844 +def _plot_throughput_latency_curves(
845 + all_data: list[dict[str, object]],
846 + serve_combs: list[dict[str, object]],
847 + bench_combs: list[dict[str, object]],
848 + output_dir: Path,
849 +):
850 + for bench_comb in bench_combs:
851 + _plot_throughput_latency_curve(all_data, serve_combs, bench_comb, output_dir)
852 +
853 +
854 +def run_slas(
855 + serve_cmd: list[str],
856 + bench_cmd: list[str],
857 + after_bench_cmd: list[str],
858 + *,
859 + show_stdout: bool,
860 + serve_params: list[dict[str, object]],
861 + bench_params: list[dict[str, object]],
862 + sla_params: list[dict[str, SLACriterionBase]],
863 + sla_variable: SLAVariable,
864 + output_dir: Path,
865 + num_runs: int,
866 + dry_run: bool,
867 +):
868 + if any(
869 + k in bench_comb
870 + for bench_comb in bench_params
871 + for k in _iter_param_key_candidates(sla_variable)
872 + ):
873 + raise ValueError(
874 + f"You should not override `{sla_variable}` in `bench_params` in SLA mode, "
875 + "since it is supposed to be determined automatically."
876 + )
877 +
878 + all_data = list[dict[str, object]]()
879 + for serve_comb in serve_params:
880 + with (
881 + _run_server(
882 + serve_cmd,
883 + after_bench_cmd,
884 + show_stdout=show_stdout,
885 + serve_overrides=serve_comb,
886 + dry_run=dry_run,
887 + )
888 + if _sla_needs_server(
889 + serve_comb,
890 + bench_params,
891 + sla_params,
892 + sla_variable,
893 + output_dir,
894 + )
895 + else contextlib.nullcontext()
896 + ) as server:
897 + for bench_comb in bench_params:
898 + for sla_comb in sla_params:
899 + base_path = _get_sla_base_path(output_dir, serve_comb, bench_comb)
900 +
901 + comb_data = _search_sla(
902 + server,
903 + bench_cmd,
904 + serve_comb=serve_comb,
905 + bench_comb=bench_comb,
906 + sla_comb=sla_comb,
907 + sla_variable=sla_variable,
908 + base_path=base_path,
909 + num_runs=num_runs,
910 + dry_run=dry_run,
911 + )
912 +
913 + if comb_data is not None:
914 + all_data.extend(comb_data)
915 +
916 + if dry_run:
917 + return None
918 +
919 + combined_df = pd.DataFrame.from_records(all_data)
920 + combined_df.to_csv(output_dir / "summary.csv")
921 +
922 + _plot_throughput_latency_curves(all_data, serve_params, bench_params, output_dir)
923 +
924 + return combined_df
925 +
926 +
927 +def _run_main(
928 + serve_cmd: list[str],
929 + bench_cmd: list[str],
930 + after_bench_cmd: list[str],
931 + *,
932 + show_stdout: bool,
933 + serve_params: list[dict[str, object]],
934 + bench_params: list[dict[str, object]],
935 + sla_params: list[dict[str, SLACriterionBase]],
936 + sla_variable: SLAVariable,
937 + output_dir: Path,
938 + num_runs: int,
939 + dry_run: bool,
940 +):
941 + if sla_params:
942 + return run_slas(
943 + serve_cmd=serve_cmd,
944 + bench_cmd=bench_cmd,
945 + after_bench_cmd=after_bench_cmd,
946 + show_stdout=show_stdout,
947 + serve_params=serve_params,
948 + bench_params=bench_params,
949 + sla_params=sla_params,
950 + sla_variable=sla_variable,
951 + output_dir=output_dir,
952 + num_runs=num_runs,
953 + dry_run=dry_run,
954 + )
955 +
956 + return run_combs(
957 + serve_cmd=serve_cmd,
958 + bench_cmd=bench_cmd,
959 + after_bench_cmd=after_bench_cmd,
960 + show_stdout=show_stdout,
961 + serve_params=serve_params,
962 + bench_params=bench_params,
963 + output_dir=output_dir,
964 + num_runs=num_runs,
965 + dry_run=dry_run,
966 + )
967 +
968 +
969 +def run_main(
970 + serve_cmd: list[str],
971 + bench_cmd: list[str],
972 + after_bench_cmd: list[str],
973 + *,
974 + show_stdout: bool,
975 + serve_params: list[dict[str, object]],
976 + bench_params: list[dict[str, object]],
977 + sla_params: list[dict[str, SLACriterionBase]],
978 + sla_variable: SLAVariable,
979 + output_dir: Path,
980 + num_runs: int,
981 + dry_run: bool,
982 + resume: str | None,
983 +):
984 + timestamp = resume or datetime.now().strftime("%Y%m%d_%H%M%S")
985 + output_dir = output_dir / timestamp
986 +
987 + if resume and not output_dir.exists():
988 + raise ValueError(f"Cannot resume from non-existent directory ({output_dir})")
989 +
990 + try:
991 + return _run_main(
992 + serve_cmd=serve_cmd,
993 + bench_cmd=bench_cmd,
994 + after_bench_cmd=after_bench_cmd,
995 + show_stdout=show_stdout,
996 + serve_params=serve_params,
997 + bench_params=bench_params,
998 + sla_params=sla_params,
999 + sla_variable=sla_variable,
1000 + output_dir=output_dir,
1001 + num_runs=num_runs,
1002 + dry_run=dry_run,
1003 + )
1004 + except BaseException as exc:
1005 + raise RuntimeError(
1006 + f"The script was terminated early. Use `--resume {timestamp}` "
1007 + f"to continue the script from its last checkpoint."
1008 + ) from exc
1009 +
1010 +
1011 +def main():
1012 + parser = argparse.ArgumentParser(
1013 + description="Run vLLM server benchmark on a parameter grid of settings."
1014 + )
1015 + parser.add_argument(
1016 + "--serve-cmd",
1017 + type=str,
1018 + required=True,
1019 + help="The command used to run the server: `vllm serve ...`",
1020 + )
1021 + parser.add_argument(
1022 + "--bench-cmd",
1023 + type=str,
1024 + required=True,
1025 + help="The command used to run the benchmark: `vllm bench serve ...`",
1026 + )
1027 + parser.add_argument(
1028 + "--after-bench-cmd",
1029 + type=str,
1030 + default=None,
1031 + help="After a benchmark run is complete, invoke this command instead of the "
1032 + "default `ServerWrapper.clear_cache()`.",
1033 + )
1034 + parser.add_argument(
1035 + "--show-stdout",
1036 + action="store_true",
1037 + help="If set, logs the standard output of subcommands. "
1038 + "Useful for debugging but can be quite spammy.",
1039 + )
1040 + parser.add_argument(
1041 + "--serve-params",
1042 + type=str,
1043 + default=None,
1044 + help="Path to JSON file containing a list of parameter combinations "
1045 + "for the `vllm serve` command. "
1046 + "If both `serve_params` and `bench_params` are given, "
1047 + "this script will iterate over their Cartesian product.",
1048 + )
1049 + parser.add_argument(
1050 + "--bench-params",
1051 + type=str,
1052 + default=None,
1053 + help="Path to JSON file containing a list of parameter combinations "
1054 + "for the `vllm bench serve` command. "
1055 + "If both `serve_params` and `bench_params` are given, "
1056 + "this script will iterate over their Cartesian product.",
1057 + )
1058 + parser.add_argument(
1059 + "--sla-params",
1060 + type=str,
1061 + default=None,
1062 + help="Path to JSON file containing a list of SLA constraints to satisfy. "
1063 + 'Each constraint is expressed in `{"<KEY>": "<OP><VALUE>"}` format, '
1064 + 'e.g.: `{"p99_e2el_ms": "<=500"}` means that '
1065 + "the E2E latency should be less than 500ms 99% of the time. "
1066 + "Setting this option runs this script in SLA mode, which searches for the "
1067 + "maximum `sla_variable` that satisfies the constraints for each combination "
1068 + "of `serve_params`, `bench_params`, and `sla_params`.",
1069 + )
1070 + parser.add_argument(
1071 + "--sla-variable",
1072 + type=str,
1073 + choices=get_args(SLAVariable),
1074 + default="request_rate",
1075 + help="Whether to tune request rate or maximum concurrency to satisfy "
1076 + "the SLA constraints.",
1077 + )
1078 + parser.add_argument(
1079 + "-o",
1080 + "--output-dir",
1081 + type=str,
1082 + default="results",
1083 + help="The directory to which results are written.",
1084 + )
1085 + parser.add_argument(
1086 + "--num-runs",
1087 + type=int,
1088 + default=3,
1089 + help="Number of runs per parameter combination.",
1090 + )
1091 + parser.add_argument(
1092 + "--dry-run",
1093 + action="store_true",
1094 + help="If set, prints the commands to run then exits without running them.",
1095 + )
1096 + parser.add_argument(
1097 + "--resume",
1098 + type=str,
1099 + default=None,
1100 + help="Set this to the name of a directory under `output_dir` (which is a "
1101 + "timestamp) to resume a previous execution of this script, i.e., only run "
1102 + "parameter combinations for which there are still no output files.",
1103 + )
1104 +
1105 + args = parser.parse_args()
1106 +
1107 + serve_cmd = shlex.split(args.serve_cmd)
1108 + bench_cmd = shlex.split(args.bench_cmd)
1109 + after_bench_cmd = (
1110 + [] if args.after_bench_cmd is None else shlex.split(args.after_bench_cmd)
1111 + )
1112 +
1113 + serve_params: list[dict[str, object]]
1114 + if args.serve_params:
1115 + with open(args.serve_params, "rb") as f:
1116 + serve_params = _parse_params(json.load(f))
1117 + else:
1118 + # i.e.: run serve_cmd without any modification
1119 + serve_params = [{}]
1120 +
1121 + bench_params: list[dict[str, object]]
1122 + if args.bench_params:
1123 + with open(args.bench_params, "rb") as f:
1124 + bench_params = _parse_params(json.load(f))
1125 + else:
1126 + # i.e.: run bench_cmd without any modification
1127 + bench_params = [{}]
1128 +
1129 + sla_params: list[dict[str, SLACriterionBase]]
1130 + if args.sla_params:
1131 + with open(args.sla_params, "rb") as f:
1132 + sla_params = _parse_sla(json.load(f))
1133 + else:
1134 + sla_params = []
1135 +
1136 + num_runs = args.num_runs
1137 + if num_runs < 1:
1138 + raise ValueError("`num_runs` should be at least 1.")
1139 +
1140 + run_main(
1141 + serve_cmd=serve_cmd,
1142 + bench_cmd=bench_cmd,
1143 + after_bench_cmd=after_bench_cmd,
1144 + show_stdout=args.show_stdout,
1145 + serve_params=serve_params,
1146 + bench_params=bench_params,
1147 + sla_params=sla_params,
1148 + sla_variable=args.sla_variable,
1149 + output_dir=Path(args.output_dir),
1150 + num_runs=num_runs,
1151 + dry_run=args.dry_run,
1152 + resume=args.resume,
1153 + )
1154 +
1155 +
1156 +if __name__ == "__main__":
1157 + main()
vllm/entrypoints/openai/api_server.py
+10
@@ -994,6 +994,16 @@ if envs.VLLM_SERVER_DEV_MODE:
994 await engine_client(raw_request).reset_prefix_cache(device)
995 return Response(status_code=200)
996
997 + @router.post("/reset_mm_cache")
998 + async def reset_mm_cache(raw_request: Request):
999 + """
1000 + Reset the multi-modal cache. Note that we currently do not check if the
1001 + multi-modal cache is successfully reset in the API server.
1002 + """
1003 + logger.info("Resetting multi-modal cache...")
1004 + await engine_client(raw_request).reset_mm_cache()
1005 + return Response(status_code=200)
1006 +
1007 @router.post("/sleep")
1008 async def sleep(raw_request: Request):
1009 # get POST params