@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / ceacedc1f9

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

[Benchmark] Add plot utility for parameter sweep (#27168)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Oct 22, 2025 at 11:30 UTC ceacedc1f9d64acb467145f76fa484d61647160c

10 files changed +1788 -1168

docs/contributing/benchmarks.md

+27 -11

@@ -7,7 +7,7 @@ toc_depth: 4

7 vLLM provides comprehensive benchmarking tools for performance testing and evaluation:

9 - **[Benchmark CLI](#benchmark-cli)**: `vllm bench` CLI tools and specialized benchmark scripts for interactive performance testing

10 -- **[Batch Scripts](#batch-scripts)**: Run `vllm bench` against multiple configurations conveniently

10 +- **[Parameter sweeps](#parameter-sweeps)**: Automate `vllm bench` runs for multiple configurations

11 - **[Performance benchmarks](#performance-benchmarks)**: Automated CI benchmarks for development

12 - **[Nightly benchmarks](#nightly-benchmarks)**: Comparative benchmarks against alternatives

@@ -925,15 +925,13 @@ throughput numbers correctly is also adjusted.

925

926 </details>

927

928 -## Batch Scripts

928 +## Parameter Sweeps

929

930 -### Batch Serving Script

930 +### Online Benchmark

931

932 -[`vllm/benchmarks/serve_multi.py`](../../vllm/benchmarks/serve_multi.py) automatically starts `vllm serve` and runs `vllm bench serve` over multiple configurations.

932 +[`vllm/benchmarks/sweep/serve.py`](../../vllm/benchmarks/sweep/serve.py) automatically starts `vllm serve` and runs `vllm bench serve` to evaluate vLLM over multiple configurations.

933

934 -#### Batch Mode

935 -

936 -The basic purpose of this script is to evaluate vLLM under different settings. Follows these steps to run the script:

934 +Follow these steps to run the script:

935

936 1. Construct the base command to `vllm serve`, and pass it to the `--serve-cmd` option.

937 2. Construct the base command to `vllm bench serve`, and pass it to the `--bench-cmd` option.

@@ -996,7 +994,7 @@ The basic purpose of this script is to evaluate vLLM under different settings. F

994 Example command:

995

996 ```bash

999 -python vllm/benchmarks/serve_multi.py \

997 +python -m vllm.benchmarks.sweep.serve \

998 --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \

999 --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json' \

1000 --serve-params benchmarks/serve_hparams.json \

@@ -1018,9 +1016,9 @@ python vllm/benchmarks/serve_multi.py \

1016 !!! tip

1017 You can use the `--resume` option to continue the parameter sweep if one of the runs failed.

1018

1021 -#### SLA Mode

1019 +### SLA Auto-Tuner

1020

1023 -By passing SLA constraints via `--sla-params`, you can run this script in SLA mode, causing it to adjust either the request rate or concurrency (choose using `--sla-variable`) in order to satisfy the SLA constraints.

1021 +[`vllm/benchmarks/sweep/serve_sla.py`](../../vllm/benchmarks/sweep/serve_sla.py) is a wrapper over [`vllm/benchmarks/sweep/serve.py`](../../vllm/benchmarks/sweep/serve.py) that tunes either the request rate or concurrency (choose using `--sla-variable`) in order to satisfy the SLA constraints given by `--sla-params`.

1022

1023 For example, to ensure E2E latency within different target values for 99% of requests:

1024

@@ -1044,7 +1042,7 @@ For example, to ensure E2E latency within different target values for 99% of req

1042 Example command:

1043

1044 ```bash

1047 -python vllm/benchmarks/serve_multi.py \

1045 +python -m vllm.benchmarks.sweep.serve_sla \

1046 --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \

1047 --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json' \

1048 --serve-params benchmarks/serve_hparams.json \

@@ -1066,6 +1064,24 @@ The algorithm for adjusting the SLA variable is as follows:

1064

1065 For a given combination of `--serve-params` and `--bench-params`, we share the benchmark results across `--sla-params` to avoid rerunning benchmarks with the same SLA variable value.

1066

1067 +### Visualizer

1068 +

1069 +[`vllm/benchmarks/sweep/plot.py`](../../vllm/benchmarks/sweep/plot.py) can be used to plot performance curves from parameter sweep results.

1070 +

1071 +Example command:

1072 +

1073 +```bash

1074 +python -m vllm.benchmarks.sweep.plot benchmarks/results/<timestamp> \

1075 + --var-x max_concurrency \

1076 + --row-by random_input_len \

1077 + --col-by random_output_len \

1078 + --curve-by api_server_count,max_num_batched_tokens \

1079 + --filter-by 'max_concurrency<=1024'

1080 +```

1081 +

1082 +!!! tip

1083 + You can use `--dry-run` to preview the figures to be plotted.

1084 +

1085 ## Performance Benchmarks

1086

1087 The performance benchmarks are used for development to confirm whether new changes improve performance under various workloads. They are triggered on every commit with both the `perf-benchmarks` and `ready` labels, and when a PR is merged into vLLM.

vllm/benchmarks/serve_multi.py deleted

-1157

@@ -1,1157 +0,0 @@

1 -# SPDX-License-Identifier: Apache-2.0

2 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 -import argparse

4 -import contextlib

5 -import json

6 -import math

7 -import os

8 -import shlex

9 -import signal

10 -import subprocess

11 -from abc import ABC, abstractmethod

12 -from datetime import datetime

13 -from pathlib import Path

14 -from typing import Literal, get_args

15 -

16 -import pandas as pd

17 -import requests

18 -import seaborn as sns

19 -from typing_extensions import assert_never, override

20 -

21 -_BAD_PARAMS_TYPE_MSG = (

22 - "The parameters to vary should be expressed as a JSON list of dictionaries."

23 -)

24 -

25 -

26 -def _parse_params(params: list[dict[str, object]]):

27 - if not isinstance(params, list):

28 - raise TypeError(f"{_BAD_PARAMS_TYPE_MSG} Found JSON type {type(params)}")

29 -

30 - for comb in params:

31 - if not isinstance(comb, dict):

32 - raise TypeError(f"{_BAD_PARAMS_TYPE_MSG} Found item type {type(comb)}")

33 -

34 - return params

35 -

36 -

37 -class SLACriterionBase(ABC):

38 - def __init__(self, target: float) -> None:

39 - super().__init__()

40 -

41 - self.target = target

42 -

43 - @abstractmethod

44 - def validate(self, actual: float) -> bool:

45 - """Return `True` if this criterion is met; otherwise `False`."""

46 - raise NotImplementedError

47 -

48 - @abstractmethod

49 - def format_cond(self, lhs: str) -> str:

50 - raise NotImplementedError

51 -

52 - def print_and_validate(

53 - self,

54 - metrics: dict[str, float],

55 - metrics_key: str,

56 - ) -> bool:

57 - metric = metrics[metrics_key]

58 - result = self.validate(metric)

59 -

60 - cond = self.format_cond(f"{metrics_key} = {metric:.2f}")

61 - print(f"Validating SLA: {cond} | " + ("PASSED" if result else "FAILED"))

62 -

63 - return result

64 -

65 -

66 -class SLALessThan(SLACriterionBase):

67 - @override

68 - def validate(self, actual: float) -> bool:

69 - return actual < self.target

70 -

71 - @override

72 - def format_cond(self, lhs: str) -> str:

73 - return f"{lhs}<{self.target:.2f}"

74 -

75 -

76 -class SLALessThanOrEqual(SLACriterionBase):

77 - @override

78 - def validate(self, actual: float) -> bool:

79 - return actual <= self.target

80 -

81 - @override

82 - def format_cond(self, lhs: str) -> str:

83 - return f"{lhs}<={self.target:.2f}"

84 -

85 -

86 -class SLAGreaterThan(SLACriterionBase):

87 - @override

88 - def validate(self, actual: float) -> bool:

89 - return actual > self.target

90 -

91 - @override

92 - def format_cond(self, lhs: str) -> str:

93 - return f"{lhs}>{self.target:.2f}"

94 -

95 -

96 -class SLAGreaterThanOrEqual(SLACriterionBase):

97 - @override

98 - def validate(self, actual: float) -> bool:

99 - return actual >= self.target

100 -

101 - @override

102 - def format_cond(self, lhs: str) -> str:

103 - return f"{lhs}>={self.target:.2f}"

104 -

105 -

106 -# NOTE: The ordering is important! Match longer op_keys first

107 -SLA_CRITERIA: dict[str, type[SLACriterionBase]] = {

108 - "<=": SLALessThanOrEqual,

109 - ">=": SLAGreaterThanOrEqual,

110 - "<": SLALessThan,

111 - ">": SLAGreaterThan,

112 -}

113 -

114 -

115 -def _parse_sla_item(sla_item: dict[str, str]):

116 - sla_criteria: dict[str, SLACriterionBase] = {}

117 -

118 - for metric_key, metric_value in sla_item.items():

119 - for op_key in SLA_CRITERIA:

120 - if metric_value.startswith(op_key):

121 - sla_criteria[metric_key] = SLA_CRITERIA[op_key](

122 - float(metric_value.removeprefix(op_key))

123 - )

124 - break

125 - else:

126 - raise ValueError(

127 - f"Invalid operator for SLA constraint '{metric_key}={metric_value}'. "

128 - f"Valid operators are: {set(SLA_CRITERIA)}",

129 - )

130 -

131 - return sla_criteria

132 -

133 -

134 -def _parse_sla(sla: list[dict[str, str]]):

135 - return [_parse_sla_item(item) for item in sla]

136 -

137 -

138 -# In JSON, we prefer "_"

139 -def _iter_param_key_candidates(param_key: str):

140 - yield param_key

141 - yield param_key.replace("-", "_")

142 - yield param_key.replace("_", "-")

143 -

144 -

145 -# In CLI, we prefer "-"

146 -def _iter_cmd_key_candidates(param_key: str):

147 - for k in reversed(tuple(_iter_param_key_candidates(param_key))):

148 - yield "--" + k

149 -

150 -

151 -def _normalize_cmd_key(param_key: str):

152 - return next(_iter_cmd_key_candidates(param_key))

153 -

154 -

155 -def _override_args(cmd: list[str], params: dict[str, object]):

156 - cmd = list(cmd)

157 -

158 - for k, v in params.items():

159 - for k_candidate in _iter_cmd_key_candidates(k):

160 - try:

161 - k_idx = cmd.index(k_candidate)

162 -

163 - if isinstance(v, bool):

164 - cmd[k_idx] = _normalize_cmd_key(k if v else "no-" + k)

165 - else:

166 - cmd[k_idx + 1] = str(v)

167 -

168 - break

169 - except ValueError:

170 - continue

171 - else:

172 - if isinstance(v, bool):

173 - cmd.append(_normalize_cmd_key(k if v else "no-" + k))

174 - else:

175 - cmd.extend([_normalize_cmd_key(k), str(v)])

176 -

177 - return cmd

178 -

179 -

180 -class ServerWrapper:

181 - def __init__(

182 - self,

183 - server_cmd: list[str],

184 - after_bench_cmd: list[str],

185 - *,

186 - show_stdout: bool,

187 - ) -> None:

188 - super().__init__()

189 -

190 - self.server_cmd = server_cmd

191 - self.after_bench_cmd = after_bench_cmd

192 - self.show_stdout = show_stdout

193 -

194 - def run_subcommand(self, cmd: list[str]):

195 - return subprocess.run(

196 - cmd,

197 - stdout=None if self.show_stdout else subprocess.DEVNULL,

198 - check=True,

199 - )

200 -

201 - def after_bench(self) -> None:

202 - if not self.after_bench_cmd:

203 - self.reset_caches()

204 - return

205 -

206 - self.run_subcommand(self.after_bench_cmd)

207 -

208 - def _get_vllm_server_address(self) -> str:

209 - server_cmd = self.server_cmd

210 -

211 - for host_key in ("--host",):

212 - if host_key in server_cmd:

213 - host = server_cmd[server_cmd.index(host_key) + 1]

214 - break

215 - else:

216 - host = "localhost"

217 -

218 - for port_key in ("-p", "--port"):

219 - if port_key in server_cmd:

220 - port = int(server_cmd[server_cmd.index(port_key) + 1])

221 - break

222 - else:

223 - port = 8000 # The default value in vllm serve

224 -

225 - return f"http://{host}:{port}"

226 -

227 - def reset_caches(self) -> None:

228 - server_cmd = self.server_cmd

229 -

230 - # Use `.endswith()` to match `/bin/...`

231 - if server_cmd[0].endswith("vllm"):

232 - server_address = self._get_vllm_server_address()

233 - print(f"Resetting caches at {server_address}")

234 -

235 - res = requests.post(f"{server_address}/reset_prefix_cache")

236 - res.raise_for_status()

237 -

238 - res = requests.post(f"{server_address}/reset_mm_cache")

239 - res.raise_for_status()

240 - elif server_cmd[0].endswith("infinity_emb"):

241 - if "--vector-disk-cache" in server_cmd:

242 - raise NotImplementedError(

243 - "Infinity server uses caching but does not expose a method "

244 - "to reset the cache"

245 - )

246 - else:

247 - raise NotImplementedError(

248 - f"No implementation of `reset_caches` for `{server_cmd[0]}` server. "

249 - "Please specify a custom command via `--after-bench-cmd`."

250 - )

251 -

252 -

253 -@contextlib.contextmanager

254 -def _run_server(

255 - serve_cmd: list[str],

256 - after_bench_cmd: list[str],

257 - *,

258 - show_stdout: bool,

259 - serve_overrides: dict[str, object],

260 - dry_run: bool,

261 -):

262 - server_cmd = _override_args(serve_cmd, serve_overrides)

263 -

264 - print("[BEGIN SERVER]")

265 - print(f"Server overrides: {serve_overrides}")

266 - print(f"Server command: {server_cmd}")

267 -

268 - if dry_run:

269 - yield None

270 - print("[END SERVER]")

271 - return

272 -

273 - # Create new process group for clean termination

274 - server_process = subprocess.Popen(

275 - server_cmd,

276 - start_new_session=True,

277 - stdout=None if show_stdout else subprocess.DEVNULL,

278 - # Need VLLM_SERVER_DEV_MODE=1 for `_reset_caches`

279 - env={**os.environ, "VLLM_SERVER_DEV_MODE": "1"},

280 - )

281 -

282 - try:

283 - yield ServerWrapper(

284 - server_cmd,

285 - after_bench_cmd,

286 - show_stdout=show_stdout,

287 - )

288 - finally:

289 - if server_process.poll() is None:

290 - # In case only some processes have been terminated

291 - with contextlib.suppress(ProcessLookupError):

292 - # We need to kill both API Server and Engine processes

293 - os.killpg(os.getpgid(server_process.pid), signal.SIGKILL)

294 -

295 - print("[END SERVER]")

296 -

297 -

298 -def _run_benchmark(

299 - server: ServerWrapper | None,

300 - bench_cmd: list[str],

301 - *,

302 - serve_overrides: dict[str, object],

303 - bench_overrides: dict[str, object],

304 - run_number: int,

305 - output_path: Path,

306 - dry_run: bool,

307 -):

308 - benchmark_cmd = [

309 - *_override_args(bench_cmd, bench_overrides),

310 - "--save-result",

311 - "--result-dir",

312 - str(output_path.parent),

313 - "--result-filename",

314 - output_path.name,

315 - ]

316 -

317 - print("[BEGIN BENCHMARK]")

318 - print(f"Benchmark overrides: {bench_overrides}")

319 - print(f"Run Number: {run_number}")

320 - print(f"Benchmark command: {benchmark_cmd}")

321 - print(f"Output file: {output_path}")

322 -

323 - run_data: dict[str, object]

324 -

325 - if output_path.exists():

326 - print("Found existing results. Skipping.")

327 -

328 - with output_path.open("rb") as f:

329 - run_data = json.load(f)

330 - return run_data

331 -

332 - if server is None:

333 - assert dry_run

334 - print("[END BENCHMARK]")

335 - return None

336 -

337 - output_path.parent.mkdir(parents=True, exist_ok=True)

338 -

339 - server.run_subcommand(benchmark_cmd)

340 - server.after_bench()

341 -

342 - with output_path.open("rb") as f:

343 - run_data = json.load(f)

344 -

345 - run_data["run_number"] = run_number

346 - run_data.update(serve_overrides)

347 -

348 - with output_path.open("w") as f:

349 - json.dump(run_data, f, indent=4)

350 -

351 - print("[END BENCHMARK]")

352 -

353 - return run_data

354 -

355 -

356 -def _get_comb_base_path(

357 - output_dir: Path,

358 - serve_comb: dict[str, object],

359 - bench_comb: dict[str, object],

360 -):

361 - return output_dir / "-".join(

362 - (

363 - "SERVE",

364 - *(f"{k}={v}" for k, v in serve_comb.items()),

365 - "BENCH",

366 - *(f"{k}={v}" for k, v in bench_comb.items()),

367 - )

368 - ).replace("/", "_").replace("..", "__") # Sanitize

369 -

370 -

371 -def _get_comb_run_path(base_path: Path, run_number: int | None):

372 - if run_number is None:

373 - return base_path / "summary.json"

374 -

375 - return base_path / f"run={run_number}.json"

376 -

377 -

378 -def _comb_needs_server(

379 - serve_comb: dict[str, object],

380 - bench_combs: list[dict[str, object]],

381 - output_dir: Path,

382 -):

383 - for bench_comb in bench_combs:

384 - base_path = _get_comb_base_path(output_dir, serve_comb, bench_comb)

385 - if not _get_comb_run_path(base_path, run_number=None).exists():

386 - return True

387 -

388 - return False

389 -

390 -

391 -def _run_comb(

392 - server: ServerWrapper | None,

393 - bench_cmd: list[str],

394 - *,

395 - serve_comb: dict[str, object],

396 - bench_comb: dict[str, object],

397 - base_path: Path,

398 - num_runs: int,

399 - dry_run: bool,

400 -):

401 - comb_data = list[dict[str, object]]()

402 -

403 - for run_number in range(num_runs):

404 - run_data = _run_benchmark(

405 - server,

406 - bench_cmd,

407 - serve_overrides=serve_comb,

408 - bench_overrides=bench_comb,

409 - run_number=run_number,

410 - output_path=_get_comb_run_path(base_path, run_number),

411 - dry_run=dry_run,

412 - )

413 -

414 - if run_data is not None:

415 - comb_data.append(run_data)

416 -

417 - if dry_run:

418 - return None

419 -

420 - with _get_comb_run_path(base_path, run_number=None).open("w") as f:

421 - json.dump(comb_data, f, indent=4)

422 -

423 - return comb_data

424 -

425 -

426 -def run_combs(

427 - serve_cmd: list[str],

428 - bench_cmd: list[str],

429 - after_bench_cmd: list[str],

430 - *,

431 - show_stdout: bool,

432 - serve_params: list[dict[str, object]],

433 - bench_params: list[dict[str, object]],

434 - output_dir: Path,

435 - num_runs: int,

436 - dry_run: bool,

437 -):

438 - all_data = list[dict[str, object]]()

439 - for serve_comb in serve_params:

440 - with (

441 - _run_server(

442 - serve_cmd,

443 - after_bench_cmd,

444 - show_stdout=show_stdout,

445 - serve_overrides=serve_comb,

446 - dry_run=dry_run,

447 - )

448 - if _comb_needs_server(serve_comb, bench_params, output_dir)

449 - else contextlib.nullcontext()

450 - ) as server:

451 - for bench_comb in bench_params:

452 - base_path = _get_comb_base_path(output_dir, serve_comb, bench_comb)

453 -

454 - comb_data = _run_comb(

455 - server,

456 - bench_cmd,

457 - serve_comb=serve_comb,

458 - bench_comb=bench_comb,

459 - base_path=base_path,

460 - num_runs=num_runs,

461 - dry_run=dry_run,

462 - )

463 -

464 - if comb_data is not None:

465 - all_data.extend(comb_data)

466 -

467 - if dry_run:

468 - return None

469 -

470 - combined_df = pd.DataFrame.from_records(all_data)

471 - combined_df.to_csv(output_dir / "summary.csv")

472 -

473 - return combined_df

474 -

475 -

476 -def _get_sla_base_path(

477 - output_dir: Path,

478 - serve_comb: dict[str, object],

479 - bench_comb: dict[str, object],

480 -):

481 - return output_dir / "-".join(

482 - (

483 - "SERVE",

484 - *(f"{k}={v}" for k, v in serve_comb.items()),

485 - "BENCH",

486 - *(f"{k}={v}" for k, v in bench_comb.items()),

487 - )

488 - ).replace("/", "_").replace("..", "__") # Sanitize

489 -

490 -

491 -def _get_sla_iter_path(

492 - base_path: Path,

493 - sla_comb: dict[str, SLACriterionBase],

494 - sla_variable: str,

495 - sla_value: int | None,

496 -):

497 - if sla_value is None:

498 - prefix = "-".join(v.format_cond(k) for k, v in sla_comb.items())

499 - return base_path / f"SLA-{prefix}.json"

500 -

501 - return base_path / f"{sla_variable}={sla_value}"

502 -

503 -

504 -def _get_sla_run_path(iter_path: Path, run_number: int | None):

505 - if run_number is None:

506 - return iter_path / "summary.json"

507 -

508 - return iter_path / f"run={run_number}.json"

509 -

510 -

511 -def _sla_needs_server(

512 - serve_comb: dict[str, object],

513 - bench_combs: list[dict[str, object]],

514 - sla_combs: list[dict[str, SLACriterionBase]],

515 - sla_variable: str,

516 - output_dir: Path,

517 -):

518 - for bench_comb in bench_combs:

519 - base_path = _get_sla_base_path(output_dir, serve_comb, bench_comb)

520 - for sla_comb in sla_combs:

521 - if not _get_sla_iter_path(

522 - base_path,

523 - sla_comb,

524 - sla_variable,

525 - sla_value=None,

526 - ).exists():

527 - return True

528 -

529 - return False

530 -

531 -

532 -def _run_sla(

533 - server: ServerWrapper | None,

534 - bench_cmd: list[str],

535 - *,

536 - serve_comb: dict[str, object],

537 - bench_comb: dict[str, object],

538 - iter_path: Path,

539 - num_runs: int,

540 - dry_run: bool,

541 -):

542 - iter_data = list[dict[str, object]]()

543 -

544 - for run_number in range(num_runs):

545 - run_data = _run_benchmark(

546 - server,

547 - bench_cmd,

548 - serve_overrides=serve_comb,

549 - bench_overrides=bench_comb,

550 - run_number=run_number,

551 - output_path=_get_sla_run_path(iter_path, run_number),

552 - dry_run=dry_run,

553 - )

554 -

555 - if run_data is not None:

556 - iter_data.append(run_data)

557 -

558 - if dry_run:

559 - return None

560 -

561 - with _get_sla_run_path(iter_path, run_number=None).open("w") as f:

562 - json.dump(iter_data, f, indent=4)

563 -

564 - return iter_data

565 -

566 -

567 -SLAVariable = Literal["request_rate", "max_concurrency"]

568 -

569 -

570 -def _estimate_sla_value(run_data: dict[str, object], sla_variable: SLAVariable):

571 - request_throughput = float(run_data["request_throughput"]) # type: ignore

572 - if sla_variable == "request_rate":

573 - return request_throughput

574 - if sla_variable == "max_concurrency":

575 - mean_latency_ms = float(run_data["mean_e2el_ms"]) # type: ignore

576 - return request_throughput * mean_latency_ms / 1000

577 -

578 - assert_never(sla_variable)

579 -

580 -

581 -def _estimate_sla_bounds(

582 - server: ServerWrapper | None,

583 - bench_cmd: list[str],

584 - *,

585 - serve_comb: dict[str, object],

586 - bench_comb: dict[str, object],

587 - sla_comb: dict[str, SLACriterionBase],

588 - base_path: Path,

589 - num_runs: int,

590 - dry_run: bool,

591 - sla_variable: SLAVariable,

592 - init_value: int,

593 - max_value: int,

594 -):

595 - sla_data = list[dict[str, object]]()

596 -

597 - max_passing: int = 0

598 - min_failing: int = 0

599 -

600 - val: int = init_value

601 - assert val > 0

602 -

603 - while True:

604 - print(f"Testing {sla_variable}: {val} req/s")

605 -

606 - iter_data = _run_sla(

607 - server,

608 - bench_cmd,

609 - serve_comb=serve_comb,

610 - bench_comb={**bench_comb, sla_variable: val},

611 - iter_path=_get_sla_iter_path(base_path, sla_comb, sla_variable, val),

612 - num_runs=num_runs,

613 - dry_run=dry_run,

614 - )

615 -

616 - assert iter_data is not None

617 - sla_data.extend(iter_data)

618 -

619 - iter_data_mean = {

620 - k: sum(float(run_data[k]) for run_data in iter_data) / len(iter_data) # type: ignore

621 - for k in sla_comb

622 - }

623 -

624 - sla_results = [

625 - criterion.print_and_validate(iter_data_mean, k)

626 - for k, criterion in sla_comb.items()

627 - ]

628 -

629 - if all(sla_results):

630 - print("SLA criteria are met.")

631 - max_passing = val

632 - val *= 2

633 - else:

634 - print("SLA criteria are not met.")

635 - min_failing = val

636 - break

637 -

638 - if val >= max_value:

639 - break

640 -

641 - return sla_data, (max_passing, min_failing)

642 -

643 -

644 -def _find_sla_value(

645 - server: ServerWrapper | None,

646 - bench_cmd: list[str],

647 - *,

648 - serve_comb: dict[str, object],

649 - bench_comb: dict[str, object],

650 - sla_comb: dict[str, SLACriterionBase],

651 - base_path: Path,

652 - num_runs: int,

653 - dry_run: bool,

654 - sla_variable: SLAVariable,

655 - min_value: int,

656 - max_value: int,

657 -):

658 - sla_data = list[dict[str, object]]()

659 -

660 - left: int = min_value

661 - right: int = max_value

662 -

663 - while True:

664 - val = (left + right) // 2

665 - print(f"Testing {sla_variable}: {val} req/s")

666 -

667 - iter_data = _run_sla(

668 - server,

669 - bench_cmd,

670 - serve_comb=serve_comb,

671 - bench_comb={**bench_comb, sla_variable: val},

672 - iter_path=_get_sla_iter_path(base_path, sla_comb, sla_variable, val),

673 - num_runs=num_runs,

674 - dry_run=dry_run,

675 - )

676 -

677 - assert iter_data is not None

678 - sla_data.extend(iter_data)

679 -

680 - iter_data_mean = {

681 - k: sum(float(run_data[k]) for run_data in iter_data) / len(iter_data) # type: ignore

682 - for k in sla_comb

683 - }

684 -

685 - sla_results = [

686 - criterion.print_and_validate(iter_data_mean, k)

687 - for k, criterion in sla_comb.items()

688 - ]

689 -

690 - if all(sla_results):

691 - print("SLA criteria are met.")

692 - left = val

693 - else:

694 - print("SLA criteria are not met.")

695 - right = val

696 -

697 - if right - left <= 1:

698 - break

699 -

700 - return sla_data, left

701 -

702 -

703 -def _search_sla(

704 - server: ServerWrapper | None,

705 - bench_cmd: list[str],

706 - *,

707 - serve_comb: dict[str, object],

708 - bench_comb: dict[str, object],

709 - sla_comb: dict[str, SLACriterionBase],

710 - sla_variable: SLAVariable,

711 - sla_inf_value: int = 65536, # The value that represents infinite QPS

712 - base_path: Path,

713 - num_runs: int,

714 - dry_run: bool,

715 -):

716 - print("[SLA START]")

717 - print(f"SLA criteria: {', '.join(v.format_cond(k) for k, v in sla_comb.items())}")

718 -

719 - sla_data_0 = _run_sla(

720 - server,

721 - bench_cmd,

722 - serve_comb=serve_comb,

723 - bench_comb={**bench_comb, sla_variable: sla_inf_value},

724 - iter_path=_get_sla_iter_path(base_path, sla_comb, sla_variable, sla_inf_value),

725 - num_runs=num_runs,

726 - dry_run=dry_run,

727 - )

728 - if sla_data_0 is None:

729 - assert dry_run

730 - print("Omitting SLA search.")

731 - print("[SLA END]")

732 - return None

733 -

734 - sla_init_value = math.ceil(

735 - sum(_estimate_sla_value(item, sla_variable) for item in sla_data_0)

736 - / len(sla_data_0)

737 - )

738 - print(f"Initial {sla_variable} to search: {sla_init_value} req/s.")

739 -

740 - sla_data_1, (sla_min, sla_max) = _estimate_sla_bounds(

741 - server,

742 - bench_cmd,

743 - serve_comb=serve_comb,

744 - bench_comb=bench_comb,

745 - sla_comb=sla_comb,

746 - base_path=base_path,

747 - num_runs=num_runs,

748 - dry_run=dry_run,

749 - sla_variable=sla_variable,

750 - init_value=sla_init_value,

751 - max_value=sla_inf_value,

752 - )

753 - print(f"Range of {sla_variable} to search: [{sla_min}, {sla_max}] req/s.")

754 -

755 - sla_data_2, sla_value = _find_sla_value(

756 - server,

757 - bench_cmd,

758 - serve_comb=serve_comb,

759 - bench_comb=bench_comb,

760 - sla_comb=sla_comb,

761 - base_path=base_path,

762 - num_runs=num_runs,

763 - dry_run=dry_run,

764 - sla_variable=sla_variable,

765 - min_value=sla_min,

766 - max_value=sla_max,

767 - )

768 -

769 - sla_data = sla_data_0 + sla_data_1 + sla_data_2

770 - print(f"Maximum {sla_variable} for SLA: {sla_value} req/s.")

771 -

772 - with _get_sla_iter_path(

773 - base_path,

774 - sla_comb,

775 - sla_variable,

776 - sla_value=None,

777 - ).open("w") as f:

778 - json.dump(sla_data, f, indent=4)

779 -

780 - print("[SLA END]")

781 -

782 - return sla_data

783 -

784 -

785 -def _plot_throughput_latency_curve(

786 - all_data: list[dict[str, object]],

787 - serve_combs: list[dict[str, object]],

788 - bench_comb: dict[str, object],

789 - output_dir: Path,

790 -):

791 - fig_path = output_dir / "-".join(

792 - (

793 - "BENCH",

794 - *(f"{k}={v}" for k, v in bench_comb.items()),

795 - )

796 - ).replace("/", "_").replace("..", "__") # Sanitize

797 -

798 - df = pd.DataFrame.from_records(

799 - [item for item in all_data if all(item[k] == bench_comb[k] for k in bench_comb)]

800 - )

801 -

802 - # Group together points with similar throughput

803 - df["request_throughput"] = df["request_throughput"].round()

804 -

805 - # Preserve the key order using dictionary

806 - all_comb_keys = {k: None for comb in serve_combs for k in comb}

807 - for k in all_comb_keys:

808 - df[k] = df[k].astype(str)

809 -

810 - keys_per_comb = [comb.keys() for comb in serve_combs]

811 - if (

812 - all(ks == keys_per_comb[0] for ks in keys_per_comb)

813 - and len(keys_per_comb[0]) <= 3

814 - ):

815 - hue, style, size, *_ = (*keys_per_comb[0], None, None)

816 - ax = sns.lineplot(

817 - df,

818 - x="request_throughput",

819 - y="p99_e2el_ms",

820 - hue=hue,

821 - style=style,

822 - size=size,

823 - markers=True,

824 - )

825 - else:

826 - df["category"] = df[list(all_comb_keys)].agg("-".join, axis=1)

827 - ax = sns.lineplot(

828 - df,

829 - x="request_throughput",

830 - y="p99_e2el_ms",

831 - hue="category",

832 - markers=True,

833 - )

834 -

835 - sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))

836 -

837 - fig = ax.get_figure()

838 - assert fig is not None

839 -

840 - fig.tight_layout()

841 - fig.savefig(fig_path)

842 -

843 -

844 -def _plot_throughput_latency_curves(

845 - all_data: list[dict[str, object]],

846 - serve_combs: list[dict[str, object]],

847 - bench_combs: list[dict[str, object]],

848 - output_dir: Path,

849 -):

850 - for bench_comb in bench_combs:

851 - _plot_throughput_latency_curve(all_data, serve_combs, bench_comb, output_dir)

852 -

853 -

854 -def run_slas(

855 - serve_cmd: list[str],

856 - bench_cmd: list[str],

857 - after_bench_cmd: list[str],

858 - *,

859 - show_stdout: bool,

860 - serve_params: list[dict[str, object]],

861 - bench_params: list[dict[str, object]],

862 - sla_params: list[dict[str, SLACriterionBase]],

863 - sla_variable: SLAVariable,

864 - output_dir: Path,

865 - num_runs: int,

866 - dry_run: bool,

867 -):

868 - if any(

869 - k in bench_comb

870 - for bench_comb in bench_params

871 - for k in _iter_param_key_candidates(sla_variable)

872 - ):

873 - raise ValueError(

874 - f"You should not override `{sla_variable}` in `bench_params` in SLA mode, "

875 - "since it is supposed to be determined automatically."

876 - )

877 -

878 - all_data = list[dict[str, object]]()

879 - for serve_comb in serve_params:

880 - with (

881 - _run_server(

882 - serve_cmd,

883 - after_bench_cmd,

884 - show_stdout=show_stdout,

885 - serve_overrides=serve_comb,

886 - dry_run=dry_run,

887 - )

888 - if _sla_needs_server(

889 - serve_comb,

890 - bench_params,

891 - sla_params,

892 - sla_variable,

893 - output_dir,

894 - )

895 - else contextlib.nullcontext()

896 - ) as server:

897 - for bench_comb in bench_params:

898 - for sla_comb in sla_params:

899 - base_path = _get_sla_base_path(output_dir, serve_comb, bench_comb)

900 -

901 - comb_data = _search_sla(

902 - server,

903 - bench_cmd,

904 - serve_comb=serve_comb,

905 - bench_comb=bench_comb,

906 - sla_comb=sla_comb,

907 - sla_variable=sla_variable,

908 - base_path=base_path,

909 - num_runs=num_runs,

910 - dry_run=dry_run,

911 - )

912 -

913 - if comb_data is not None:

914 - all_data.extend(comb_data)

915 -

916 - if dry_run:

917 - return None

918 -

919 - combined_df = pd.DataFrame.from_records(all_data)

920 - combined_df.to_csv(output_dir / "summary.csv")

921 -

922 - _plot_throughput_latency_curves(all_data, serve_params, bench_params, output_dir)

923 -

924 - return combined_df

925 -

926 -

927 -def _run_main(

928 - serve_cmd: list[str],

929 - bench_cmd: list[str],

930 - after_bench_cmd: list[str],

931 - *,

932 - show_stdout: bool,

933 - serve_params: list[dict[str, object]],

934 - bench_params: list[dict[str, object]],

935 - sla_params: list[dict[str, SLACriterionBase]],

936 - sla_variable: SLAVariable,

937 - output_dir: Path,

938 - num_runs: int,

939 - dry_run: bool,

940 -):

941 - if sla_params:

942 - return run_slas(

943 - serve_cmd=serve_cmd,

944 - bench_cmd=bench_cmd,

945 - after_bench_cmd=after_bench_cmd,

946 - show_stdout=show_stdout,

947 - serve_params=serve_params,

948 - bench_params=bench_params,

949 - sla_params=sla_params,

950 - sla_variable=sla_variable,

951 - output_dir=output_dir,

952 - num_runs=num_runs,

953 - dry_run=dry_run,

954 - )

955 -

956 - return run_combs(

957 - serve_cmd=serve_cmd,

958 - bench_cmd=bench_cmd,

959 - after_bench_cmd=after_bench_cmd,

960 - show_stdout=show_stdout,

961 - serve_params=serve_params,

962 - bench_params=bench_params,

963 - output_dir=output_dir,

964 - num_runs=num_runs,

965 - dry_run=dry_run,

966 - )

967 -

968 -

969 -def run_main(

970 - serve_cmd: list[str],

971 - bench_cmd: list[str],

972 - after_bench_cmd: list[str],

973 - *,

974 - show_stdout: bool,

975 - serve_params: list[dict[str, object]],

976 - bench_params: list[dict[str, object]],

977 - sla_params: list[dict[str, SLACriterionBase]],

978 - sla_variable: SLAVariable,

979 - output_dir: Path,

980 - num_runs: int,

981 - dry_run: bool,

982 - resume: str | None,

983 -):

984 - timestamp = resume or datetime.now().strftime("%Y%m%d_%H%M%S")

985 - output_dir = output_dir / timestamp

986 -

987 - if resume and not output_dir.exists():

988 - raise ValueError(f"Cannot resume from non-existent directory ({output_dir})")

989 -

990 - try:

991 - return _run_main(

992 - serve_cmd=serve_cmd,

993 - bench_cmd=bench_cmd,

994 - after_bench_cmd=after_bench_cmd,

995 - show_stdout=show_stdout,

996 - serve_params=serve_params,

997 - bench_params=bench_params,

998 - sla_params=sla_params,

999 - sla_variable=sla_variable,

1000 - output_dir=output_dir,

1001 - num_runs=num_runs,

1002 - dry_run=dry_run,

1003 - )

1004 - except BaseException as exc:

1005 - raise RuntimeError(

1006 - f"The script was terminated early. Use `--resume {timestamp}` "

1007 - f"to continue the script from its last checkpoint."

1008 - ) from exc

1009 -

1010 -

1011 -def main():

1012 - parser = argparse.ArgumentParser(

1013 - description="Run vLLM server benchmark on a parameter grid of settings."

1014 - )

1015 - parser.add_argument(

1016 - "--serve-cmd",

1017 - type=str,

1018 - required=True,

1019 - help="The command used to run the server: `vllm serve ...`",

1020 - )

1021 - parser.add_argument(

1022 - "--bench-cmd",

1023 - type=str,

1024 - required=True,

1025 - help="The command used to run the benchmark: `vllm bench serve ...`",

1026 - )

1027 - parser.add_argument(

1028 - "--after-bench-cmd",

1029 - type=str,

1030 - default=None,

1031 - help="After a benchmark run is complete, invoke this command instead of the "

1032 - "default `ServerWrapper.clear_cache()`.",

1033 - )

1034 - parser.add_argument(

1035 - "--show-stdout",

1036 - action="store_true",

1037 - help="If set, logs the standard output of subcommands. "

1038 - "Useful for debugging but can be quite spammy.",

1039 - )

1040 - parser.add_argument(

1041 - "--serve-params",

1042 - type=str,

1043 - default=None,

1044 - help="Path to JSON file containing a list of parameter combinations "

1045 - "for the `vllm serve` command. "

1046 - "If both `serve_params` and `bench_params` are given, "

1047 - "this script will iterate over their Cartesian product.",

1048 - )

1049 - parser.add_argument(

1050 - "--bench-params",

1051 - type=str,

1052 - default=None,

1053 - help="Path to JSON file containing a list of parameter combinations "

1054 - "for the `vllm bench serve` command. "

1055 - "If both `serve_params` and `bench_params` are given, "

1056 - "this script will iterate over their Cartesian product.",

1057 - )

1058 - parser.add_argument(

1059 - "--sla-params",

1060 - type=str,

1061 - default=None,

1062 - help="Path to JSON file containing a list of SLA constraints to satisfy. "

1063 - 'Each constraint is expressed in `{"<KEY>": "<OP><VALUE>"}` format, '

1064 - 'e.g.: `{"p99_e2el_ms": "<=500"}` means that '

1065 - "the E2E latency should be less than 500ms 99% of the time. "

1066 - "Setting this option runs this script in SLA mode, which searches for the "

1067 - "maximum `sla_variable` that satisfies the constraints for each combination "

1068 - "of `serve_params`, `bench_params`, and `sla_params`.",

1069 - )

1070 - parser.add_argument(

1071 - "--sla-variable",

1072 - type=str,

1073 - choices=get_args(SLAVariable),

1074 - default="request_rate",

1075 - help="Whether to tune request rate or maximum concurrency to satisfy "

1076 - "the SLA constraints.",

1077 - )

1078 - parser.add_argument(

1079 - "-o",

1080 - "--output-dir",

1081 - type=str,

1082 - default="results",

1083 - help="The directory to which results are written.",

1084 - )

1085 - parser.add_argument(

1086 - "--num-runs",

1087 - type=int,

1088 - default=3,

1089 - help="Number of runs per parameter combination.",

1090 - )

1091 - parser.add_argument(

1092 - "--dry-run",

1093 - action="store_true",

1094 - help="If set, prints the commands to run then exits without running them.",

1095 - )

1096 - parser.add_argument(

1097 - "--resume",

1098 - type=str,

1099 - default=None,

1100 - help="Set this to the name of a directory under `output_dir` (which is a "

1101 - "timestamp) to resume a previous execution of this script, i.e., only run "

1102 - "parameter combinations for which there are still no output files.",

1103 - )

1104 -

1105 - args = parser.parse_args()

1106 -

1107 - serve_cmd = shlex.split(args.serve_cmd)

1108 - bench_cmd = shlex.split(args.bench_cmd)

1109 - after_bench_cmd = (

1110 - [] if args.after_bench_cmd is None else shlex.split(args.after_bench_cmd)

1111 - )

1112 -

1113 - serve_params: list[dict[str, object]]

1114 - if args.serve_params:

1115 - with open(args.serve_params, "rb") as f:

1116 - serve_params = _parse_params(json.load(f))

1117 - else:

1118 - # i.e.: run serve_cmd without any modification

1119 - serve_params = [{}]

1120 -

1121 - bench_params: list[dict[str, object]]

1122 - if args.bench_params:

1123 - with open(args.bench_params, "rb") as f:

1124 - bench_params = _parse_params(json.load(f))

1125 - else:

1126 - # i.e.: run bench_cmd without any modification

1127 - bench_params = [{}]

1128 -

1129 - sla_params: list[dict[str, SLACriterionBase]]

1130 - if args.sla_params:

1131 - with open(args.sla_params, "rb") as f:

1132 - sla_params = _parse_sla(json.load(f))

1133 - else:

1134 - sla_params = []

1135 -

1136 - num_runs = args.num_runs

1137 - if num_runs < 1:

1138 - raise ValueError("`num_runs` should be at least 1.")

1139 -

1140 - run_main(

1141 - serve_cmd=serve_cmd,

1142 - bench_cmd=bench_cmd,

1143 - after_bench_cmd=after_bench_cmd,

1144 - show_stdout=args.show_stdout,

1145 - serve_params=serve_params,

1146 - bench_params=bench_params,

1147 - sla_params=sla_params,

1148 - sla_variable=args.sla_variable,

1149 - output_dir=Path(args.output_dir),

1150 - num_runs=num_runs,

1151 - dry_run=args.dry_run,

1152 - resume=args.resume,

1153 - )

1154 -

1155 -

1156 -if __name__ == "__main__":

1157 - main()

vllm/benchmarks/sweep/__init__.py

vllm/benchmarks/sweep/param_sweep.py new

+91

@@ -0,0 +1,91 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +import json

4 +import os

5 +from typing import Any

6 +

7 +

8 +class ParameterSweep(list["ParameterSweepItem"]):

9 + @classmethod

10 + def read_json(cls, filepath: os.PathLike):

11 + with open(filepath, "rb") as f:

12 + records = json.load(f)

13 +

14 + return cls.from_records(records)

15 +

16 + @classmethod

17 + def from_records(cls, records: list[dict[str, object]]):

18 + if not isinstance(records, list):

19 + raise TypeError(

20 + f"The parameter sweep should be a list of dictionaries, "

21 + f"but found type: {type(records)}"

22 + )

23 +

24 + return cls(ParameterSweepItem.from_record(record) for record in records)

25 +

26 +

27 +class ParameterSweepItem(dict[str, object]):

28 + @classmethod

29 + def from_record(cls, record: dict[str, object]):

30 + if not isinstance(record, dict):

31 + raise TypeError(

32 + f"Each item in the parameter sweep should be a dictionary, "

33 + f"but found type: {type(record)}"

34 + )

35 +

36 + return cls(record)

37 +

38 + def __or__(self, other: dict[str, Any]):

39 + return type(self)(super().__or__(other))

40 +

41 + # In JSON, we prefer "_"

42 + def _iter_param_key_candidates(self, param_key: str):

43 + # Inner config arguments are not converted by the CLI

44 + if "." in param_key:

45 + prefix, rest = param_key.split(".", 1)

46 + for prefix_candidate in self._iter_param_key_candidates(prefix):

47 + yield prefix_candidate + "." + rest

48 +

49 + return

50 +

51 + yield param_key

52 + yield param_key.replace("-", "_")

53 + yield param_key.replace("_", "-")

54 +

55 + # In CLI, we prefer "-"

56 + def _iter_cmd_key_candidates(self, param_key: str):

57 + for k in reversed(tuple(self._iter_param_key_candidates(param_key))):

58 + yield "--" + k

59 +

60 + def _normalize_cmd_key(self, param_key: str):

61 + return next(self._iter_cmd_key_candidates(param_key))

62 +

63 + def has_param(self, param_key: str) -> bool:

64 + return any(k in self for k in self._iter_param_key_candidates(param_key))

65 +

66 + def apply_to_cmd(self, cmd: list[str]) -> list[str]:

67 + cmd = list(cmd)

68 +

69 + for k, v in self.items():

70 + for k_candidate in self._iter_cmd_key_candidates(k):

71 + try:

72 + k_idx = cmd.index(k_candidate)

73 +

74 + if isinstance(v, bool):

75 + cmd[k_idx] = self._normalize_cmd_key(k if v else "no-" + k)

76 + else:

77 + cmd[k_idx + 1] = str(v)

78 +

79 + break

80 + except ValueError:

81 + continue

82 + else:

83 + if isinstance(v, bool):

84 + cmd.append(self._normalize_cmd_key(k if v else "no-" + k))

85 + else:

86 + cmd.extend([self._normalize_cmd_key(k), str(v)])

87 +

88 + return cmd

89 +

90 + def as_text(self, sep: str = ", ") -> str:

91 + return sep.join(f"{k}={v}" for k, v in self.items())

vllm/benchmarks/sweep/plot.py new

+530

@@ -0,0 +1,530 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +import argparse

4 +import json

5 +from abc import ABC, abstractmethod

6 +from concurrent.futures import ProcessPoolExecutor

7 +from dataclasses import dataclass

8 +from functools import partial

9 +from pathlib import Path

10 +from types import TracebackType

11 +

12 +import matplotlib.pyplot as plt

13 +import pandas as pd

14 +import seaborn as sns

15 +from typing_extensions import Self, override

16 +

17 +from vllm.utils.collection_utils import full_groupby

18 +

19 +from .utils import sanitize_filename

20 +

21 +

22 +@dataclass

23 +class PlotFilterBase(ABC):

24 + var: str

25 + target: str

26 +

27 + @classmethod

28 + def parse_str(cls, s: str):

29 + for op_key in PLOT_FILTERS:

30 + if op_key in s:

31 + key, value = s.split(op_key)

32 + return PLOT_FILTERS[op_key](

33 + key,

34 + value.removeprefix(op_key).strip("'").strip('"'),

35 + )

36 + else:

37 + raise ValueError(

38 + f"Invalid operator for plot filter '{s}'. "

39 + f"Valid operators are: {sorted(PLOT_FILTERS)}",

40 + )

41 +

42 + @abstractmethod

43 + def apply(self, df: pd.DataFrame) -> pd.DataFrame:

44 + """Applies this filter to a DataFrame."""

45 + raise NotImplementedError

46 +

47 +

48 +@dataclass

49 +class PlotEqualTo(PlotFilterBase):

50 + @override

51 + def apply(self, df: pd.DataFrame) -> pd.DataFrame:

52 + try:

53 + target = float(self.target)

54 + except ValueError:

55 + target = self.target

56 +

57 + return df[df[self.var] == target]

58 +

59 +

60 +@dataclass

61 +class PlotLessThan(PlotFilterBase):

62 + @override

63 + def apply(self, df: pd.DataFrame) -> pd.DataFrame:

64 + return df[df[self.var] < float(self.target)]

65 +

66 +

67 +@dataclass

68 +class PlotLessThanOrEqualTo(PlotFilterBase):

69 + @override

70 + def apply(self, df: pd.DataFrame) -> pd.DataFrame:

71 + return df[df[self.var] <= float(self.target)]

72 +

73 +

74 +@dataclass

75 +class PlotGreaterThan(PlotFilterBase):

76 + @override

77 + def apply(self, df: pd.DataFrame) -> pd.DataFrame:

78 + return df[df[self.var] > float(self.target)]

79 +

80 +

81 +@dataclass

82 +class PlotGreaterThanOrEqualTo(PlotFilterBase):

83 + @override

84 + def apply(self, df: pd.DataFrame) -> pd.DataFrame:

85 + return df[df[self.var] >= float(self.target)]

86 +

87 +

88 +# NOTE: The ordering is important! Match longer op_keys first

89 +PLOT_FILTERS: dict[str, type[PlotFilterBase]] = {

90 + "==": PlotEqualTo,

91 + "<=": PlotLessThanOrEqualTo,

92 + ">=": PlotGreaterThanOrEqualTo,

93 + "<": PlotLessThan,

94 + ">": PlotGreaterThan,

95 +}

96 +

97 +

98 +class PlotFilters(list[PlotFilterBase]):

99 + @classmethod

100 + def parse_str(cls, s: str):

101 + if not s:

102 + return cls()

103 +

104 + return cls(PlotFilterBase.parse_str(e) for e in s.split(","))

105 +

106 + def apply(self, df: pd.DataFrame) -> pd.DataFrame:

107 + for item in self:

108 + df = item.apply(df)

109 +

110 + return df

111 +

112 +

113 +@dataclass

114 +class PlotBinner:

115 + var: str

116 + bin_size: float

117 +

118 + @classmethod

119 + def parse_str(cls, s: str):

120 + for op_key in PLOT_BINNERS:

121 + if op_key in s:

122 + key, value = s.split(op_key)

123 + return PLOT_BINNERS[op_key](key, float(value.removeprefix(op_key)))

124 + else:

125 + raise ValueError(

126 + f"Invalid operator for plot binner '{s}'. "

127 + f"Valid operators are: {sorted(PLOT_BINNERS)}",

128 + )

129 +

130 + def apply(self, df: pd.DataFrame) -> pd.DataFrame:

131 + """Applies this binner to a DataFrame."""

132 + df = df.copy()

133 + df[self.var] = df[self.var] // self.bin_size * self.bin_size

134 + return df

135 +

136 +

137 +PLOT_BINNERS: dict[str, type[PlotBinner]] = {

138 + "%": PlotBinner,

139 +}

140 +

141 +

142 +class PlotBinners(list[PlotBinner]):

143 + @classmethod

144 + def parse_str(cls, s: str):

145 + if not s:

146 + return cls()

147 +

148 + return cls(PlotBinner.parse_str(e) for e in s.split(","))

149 +

150 + def apply(self, df: pd.DataFrame) -> pd.DataFrame:

151 + for item in self:

152 + df = item.apply(df)

153 +

154 + return df

155 +

156 +

157 +def _json_load_bytes(path: Path) -> list[dict[str, object]]:

158 + with path.open("rb") as f:

159 + return json.load(f)

160 +

161 +

162 +def _get_metric(run_data: dict[str, object], metric_key: str):

163 + try:

164 + return run_data[metric_key]

165 + except KeyError as exc:

166 + raise ValueError(f"Cannot find metric {metric_key!r} in {run_data=}") from exc

167 +

168 +

169 +def _get_group(run_data: dict[str, object], group_keys: list[str]):

170 + return tuple((k, str(_get_metric(run_data, k))) for k in group_keys)

171 +

172 +

173 +def _get_fig_path(fig_dir: Path, group: tuple[tuple[str, str], ...]):

174 + parts = list[str]()

175 + if group:

176 + parts.extend(("FIGURE-", *(f"{k}={v}" for k, v in group)))

177 + else:

178 + parts.append("figure")

179 +

180 + return fig_dir / sanitize_filename("-".join(parts) + ".png")

181 +

182 +

183 +class DummyExecutor:

184 + map = map

185 +

186 + def __enter__(self) -> Self:

187 + return self

188 +

189 + def __exit__(

190 + self,

191 + exc_type: type[BaseException] | None,

192 + exc_value: BaseException | None,

193 + exc_traceback: TracebackType | None,

194 + ) -> None:

195 + return None

196 +

197 +

198 +def _plot_fig(

199 + fig_dir: Path,

200 + fig_group_data: tuple[tuple[tuple[str, str], ...], list[dict[str, object]]],

201 + row_by: list[str],

202 + col_by: list[str],

203 + curve_by: list[str],

204 + *,

205 + var_x: str,

206 + var_y: str,

207 + filter_by: PlotFilters,

208 + bin_by: PlotBinners,

209 + scale_x: str | None,

210 + scale_y: str | None,

211 + dry_run: bool,

212 +):

213 + fig_group, fig_data = fig_group_data

214 +

215 + row_groups = full_groupby(

216 + fig_data,

217 + key=lambda item: _get_group(item, row_by),

218 + )

219 + num_rows = len(row_groups)

220 + num_cols = max(

221 + len(full_groupby(row_data, key=lambda item: _get_group(item, col_by)))

222 + for _, row_data in row_groups

223 + )

224 +

225 + fig_path = _get_fig_path(fig_dir, fig_group)

226 +

227 + print("[BEGIN FIGURE]")

228 + print(f"Group: {dict(fig_group)}")

229 + print(f"Grid: {num_rows} rows x {num_cols} cols")

230 + print(f"Output file: {fig_path}")

231 +

232 + if dry_run:

233 + print("[END FIGURE]")

234 + return

235 +

236 + df = pd.DataFrame.from_records(fig_data)

237 +

238 + if var_x not in df.columns:

239 + raise ValueError(

240 + f"Cannot find {var_x=!r} in parameter sweep results. "

241 + f"Available variables: {df.columns.tolist()}"

242 + )

243 + if var_y not in df.columns:

244 + raise ValueError(

245 + f"Cannot find {var_y=!r} in parameter sweep results. "

246 + f"Available variables: {df.columns.tolist()}"

247 + )

248 + for k in row_by:

249 + if k not in df.columns:

250 + raise ValueError(

251 + f"Cannot find row_by={k!r} in parameter sweep results. "

252 + f"Available variables: {df.columns.tolist()}"

253 + )

254 + for k in col_by:

255 + if k not in df.columns:

256 + raise ValueError(

257 + f"Cannot find col_by={k!r} in parameter sweep results. "

258 + f"Available variables: {df.columns.tolist()}"

259 + )

260 + for k in curve_by:

261 + if k not in df.columns:

262 + raise ValueError(

263 + f"Cannot find curve_by={k!r} in parameter sweep results. "

264 + f"Available variables: {df.columns.tolist()}"

265 + )

266 +

267 + df = filter_by.apply(df)

268 + df = bin_by.apply(df)

269 +

270 + df["row_group"] = (

271 + pd.concat(

272 + [k + "=" + df[k].astype(str) for k in row_by],

273 + axis=1,

274 + ).agg("\n".join, axis=1)

275 + if row_by

276 + else "(All)"

277 + )

278 +

279 + df["col_group"] = (

280 + pd.concat(

281 + [k + "=" + df[k].astype(str) for k in col_by],

282 + axis=1,

283 + ).agg("\n".join, axis=1)

284 + if col_by

285 + else "(All)"

286 + )

287 +

288 + g = sns.FacetGrid(df, row="row_group", col="col_group")

289 +

290 + if row_by and col_by:

291 + g.set_titles("{row_name}\n{col_name}")

292 + elif row_by:

293 + g.set_titles("{row_name}")

294 + elif col_by:

295 + g.set_titles("{col_name}")

296 + else:

297 + g.set_titles("")

298 +

299 + if scale_x:

300 + g.set(xscale=scale_x)

301 + if scale_y:

302 + g.set(yscale=scale_y)

303 +

304 + if len(curve_by) <= 3:

305 + hue, style, size, *_ = (*curve_by, None, None, None)

306 +

307 + g.map_dataframe(

308 + sns.lineplot,

309 + x=var_x,

310 + y=var_y,

311 + hue=hue,

312 + style=style,

313 + size=size,

314 + markers=True,

315 + )

316 +

317 + g.add_legend(title=hue)

318 + else:

319 + df["curve_group"] = (

320 + pd.concat(

321 + [k + "=" + df[k].astype(str) for k in curve_by],

322 + axis=1,

323 + ).agg("\n".join, axis=1)

324 + if curve_by

325 + else "(All)"

326 + )

327 +

328 + g.map_dataframe(

329 + sns.lineplot,

330 + x=var_x,

331 + y=var_y,

332 + hue="curve_group",

333 + markers=True,

334 + )

335 +

336 + g.add_legend()

337 +

338 + g.savefig(fig_path)

339 + plt.close(g.figure)

340 +

341 + print("[END FIGURE]")

342 +

343 +

344 +def plot(

345 + output_dir: Path,

346 + fig_dir: Path,

347 + fig_by: list[str],

348 + row_by: list[str],

349 + col_by: list[str],

350 + curve_by: list[str],

351 + *,

352 + var_x: str,

353 + var_y: str,

354 + filter_by: PlotFilters,

355 + bin_by: PlotBinners,

356 + scale_x: str | None,

357 + scale_y: str | None,

358 + dry_run: bool,

359 +):

360 + all_data = [

361 + run_data

362 + for path in output_dir.rglob("**/summary.json")

363 + for run_data in _json_load_bytes(path)

364 + ]

365 +

366 + if not all_data:

367 + raise ValueError(f"Did not find any parameter sweep results under {output_dir}")

368 +

369 + fig_dir.mkdir(parents=True, exist_ok=True)

370 +

371 + fig_groups = full_groupby(

372 + all_data,

373 + key=lambda item: _get_group(item, fig_by),

374 + )

375 +

376 + with DummyExecutor() if len(fig_groups) <= 1 else ProcessPoolExecutor() as executor:

377 + # Resolve the iterable to ensure that the workers are run

378 + all(

379 + executor.map(

380 + partial(

381 + _plot_fig,

382 + fig_dir,

383 + row_by=row_by,

384 + col_by=col_by,

385 + curve_by=curve_by,

386 + var_x=var_x,

387 + var_y=var_y,

388 + filter_by=filter_by,

389 + bin_by=bin_by,

390 + scale_x=scale_x,

391 + scale_y=scale_y,

392 + dry_run=dry_run,

393 + ),

394 + fig_groups,

395 + )

396 + )

397 +

398 +

399 +def add_cli_args(parser: argparse.ArgumentParser):

400 + parser.add_argument(

401 + "OUTPUT_DIR",

402 + type=str,

403 + default="results",

404 + help="The directory containing the results to plot, "

405 + "i.e., the `--output-dir` argument to the parameter sweep script.",

406 + )

407 + parser.add_argument(

408 + "--fig-dir",

409 + type=str,

410 + default="",

411 + help="The directory to save the figures, relative to `OUTPUT_DIR`. "

412 + "By default, the same directory is used.",

413 + )

414 + parser.add_argument(

415 + "--fig-by",

416 + type=str,

417 + default="",

418 + help="A comma-separated list of variables, such that a separate figure "

419 + "is created for each combination of these variables.",

420 + )

421 + parser.add_argument(

422 + "--row-by",

423 + type=str,

424 + default="",

425 + help="A comma-separated list of variables, such that a separate row "

426 + "is created for each combination of these variables.",

427 + )

428 + parser.add_argument(

429 + "--col-by",

430 + type=str,

431 + default="",

432 + help="A comma-separated list of variables, such that a separate column "

433 + "is created for each combination of these variables.",

434 + )

435 + parser.add_argument(

436 + "--curve-by",

437 + type=str,

438 + default=None,

439 + help="A comma-separated list of variables, such that a separate curve "

440 + "is created for each combination of these variables.",

441 + )

442 + parser.add_argument(

443 + "--var-x",

444 + type=str,

445 + default="request_throughput",

446 + help="The variable for the x-axis.",

447 + )

448 + parser.add_argument(

449 + "--var-y",

450 + type=str,

451 + default="p99_e2el_ms",

452 + help="The variable for the y-axis",

453 + )

454 + parser.add_argument(

455 + "--filter-by",

456 + type=str,

457 + default="",

458 + help="A comma-separated list of statements indicating values to filter by. "

459 + "This is useful to remove outliers. "

460 + "Example: `max_concurrency<1000,max_num_batched_tokens<=4096` means "

461 + "plot only the points where `max_concurrency` is less than 1000 and "

462 + "`max_num_batched_tokens` is no greater than 4096.",

463 + )

464 + parser.add_argument(

465 + "--bin-by",

466 + type=str,

467 + default="",

468 + help="A comma-separated list of statements indicating values to bin by. "

469 + "This is useful to avoid plotting points that are too close together. "

470 + "Example: `request_throughput%1` means "

471 + "use a bin size of 1 for the `request_throughput` variable.",

472 + )

473 + parser.add_argument(

474 + "--scale-x",

475 + type=str,

476 + default=None,

477 + help="The scale to use for the x-axis. "

478 + "Currently only accepts string values such as 'log' and 'sqrt'. "

479 + "See also: https://seaborn.pydata.org/generated/seaborn.objects.Plot.scale.html",

480 + )

481 + parser.add_argument(

482 + "--scale-y",

483 + type=str,

484 + default=None,

485 + help="The scale to use for the y-axis. "

486 + "Currently only accepts string values such as 'log' and 'sqrt'. "

487 + "See also: https://seaborn.pydata.org/generated/seaborn.objects.Plot.scale.html",

488 + )

489 + parser.add_argument(

490 + "--dry-run",

491 + action="store_true",

492 + help="If set, prints the information about each figure to plot, "

493 + "then exits without drawing them.",

494 + )

495 +

496 +

497 +def main(args: argparse.Namespace):

498 + output_dir = Path(args.OUTPUT_DIR)

499 + if not output_dir.exists():

500 + raise ValueError(f"No parameter sweep results under {output_dir}")

501 +

502 + curve_by = [] if not args.curve_by else args.curve_by.split(",")

503 + row_by = [] if not args.row_by else args.row_by.split(",")

504 + col_by = [] if not args.col_by else args.col_by.split(",")

505 + fig_by = [] if not args.fig_by else args.fig_by.split(",")

506 +

507 + plot(

508 + output_dir=output_dir,

509 + fig_dir=output_dir / args.fig_dir,

510 + fig_by=fig_by,

511 + row_by=row_by,

512 + col_by=col_by,

513 + curve_by=curve_by,

514 + var_x=args.var_x,

515 + var_y=args.var_y,

516 + filter_by=PlotFilters.parse_str(args.filter_by),

517 + bin_by=PlotBinners.parse_str(args.bin_by),

518 + scale_x=args.scale_x,

519 + scale_y=args.scale_y,

520 + dry_run=args.dry_run,

521 + )

522 +

523 +

524 +if __name__ == "__main__":

525 + parser = argparse.ArgumentParser(

526 + description="Plot performance curves from parameter sweep results."

527 + )

528 + add_cli_args(parser)

529 +

530 + main(parser.parse_args())

vllm/benchmarks/sweep/serve.py new

+407

@@ -0,0 +1,407 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +import argparse

4 +import contextlib

5 +import json

6 +import shlex

7 +from dataclasses import dataclass

8 +from datetime import datetime

9 +from pathlib import Path

10 +

11 +import pandas as pd

12 +

13 +from .param_sweep import ParameterSweep, ParameterSweepItem

14 +from .server import ServerProcess

15 +from .utils import sanitize_filename

16 +

17 +

18 +@contextlib.contextmanager

19 +def run_server(

20 + serve_cmd: list[str],

21 + after_bench_cmd: list[str],

22 + *,

23 + show_stdout: bool,

24 + serve_overrides: ParameterSweepItem,

25 + dry_run: bool,

26 +):

27 + server_cmd = serve_overrides.apply_to_cmd(serve_cmd)

28 +

29 + print("[BEGIN SERVER]")

30 + print(f"Server overrides: {serve_overrides}")

31 + print(f"Server command: {server_cmd}")

32 +

33 + if dry_run:

34 + yield None

35 + print("[END SERVER]")

36 + return

37 +

38 + with ServerProcess(server_cmd, after_bench_cmd, show_stdout=show_stdout) as server:

39 + yield server

40 +

41 + print("[END SERVER]")

42 +

43 +

44 +def _update_run_data(

45 + run_data: dict[str, object],

46 + serve_overrides: ParameterSweepItem,

47 + bench_overrides: ParameterSweepItem,

48 + run_number: int,

49 +):

50 + run_data["run_number"] = run_number

51 + run_data.update(serve_overrides)

52 + run_data.update(bench_overrides)

53 +

54 + return run_data

55 +

56 +

57 +def run_benchmark(

58 + server: ServerProcess | None,

59 + bench_cmd: list[str],

60 + *,

61 + serve_overrides: ParameterSweepItem,

62 + bench_overrides: ParameterSweepItem,

63 + run_number: int,

64 + output_path: Path,

65 + dry_run: bool,

66 +):

67 + benchmark_cmd = [

68 + *bench_overrides.apply_to_cmd(bench_cmd),

69 + "--save-result",

70 + "--result-dir",

71 + str(output_path.parent),

72 + "--result-filename",

73 + output_path.name,

74 + ]

75 +

76 + print("[BEGIN BENCHMARK]")

77 + print(f"Benchmark overrides: {bench_overrides}")

78 + print(f"Run Number: {run_number}")

79 + print(f"Benchmark command: {benchmark_cmd}")

80 + print(f"Output file: {output_path}")

81 +

82 + run_data: dict[str, object]

83 +

84 + if output_path.exists():

85 + print("Found existing results. Skipping.")

86 +

87 + with output_path.open("rb") as f:

88 + run_data = json.load(f)

89 + return _update_run_data(

90 + run_data,

91 + serve_overrides,

92 + bench_overrides,

93 + run_number,

94 + )

95 +

96 + if server is None:

97 + if not dry_run:

98 + raise ValueError(f"Cannot find results at {output_path}")

99 +

100 + print("[END BENCHMARK]")

101 + return None

102 +

103 + output_path.parent.mkdir(parents=True, exist_ok=True)

104 +

105 + server.run_subcommand(benchmark_cmd)

106 + server.after_bench()

107 +

108 + with output_path.open("rb") as f:

109 + run_data = json.load(f)

110 +

111 + run_data = _update_run_data(

112 + run_data,

113 + serve_overrides,

114 + bench_overrides,

115 + run_number,

116 + )

117 +

118 + with output_path.open("w") as f:

119 + json.dump(run_data, f, indent=4)

120 +

121 + print("[END BENCHMARK]")

122 +

123 + return run_data

124 +

125 +

126 +def _get_comb_base_path(

127 + output_dir: Path,

128 + serve_comb: ParameterSweepItem,

129 + bench_comb: ParameterSweepItem,

130 +):

131 + parts = list[str]()

132 + if serve_comb:

133 + parts.extend(("SERVE-", serve_comb.as_text(sep="-")))

134 + if bench_comb:

135 + parts.extend(("BENCH-", bench_comb.as_text(sep="-")))

136 +

137 + return output_dir / sanitize_filename("-".join(parts))

138 +

139 +

140 +def _get_comb_run_path(base_path: Path, run_number: int | None):

141 + if run_number is None:

142 + return base_path / "summary.json"

143 +

144 + return base_path / f"run={run_number}.json"

145 +

146 +

147 +def _comb_needs_server(

148 + serve_comb: ParameterSweepItem,

149 + bench_combs: ParameterSweep,

150 + output_dir: Path,

151 +):

152 + for bench_comb in bench_combs:

153 + base_path = _get_comb_base_path(output_dir, serve_comb, bench_comb)

154 + if not _get_comb_run_path(base_path, run_number=None).exists():

155 + return True

156 +

157 + return False

158 +

159 +

160 +def run_comb(

161 + server: ServerProcess | None,

162 + bench_cmd: list[str],

163 + *,

164 + serve_comb: ParameterSweepItem,

165 + bench_comb: ParameterSweepItem,

166 + base_path: Path,

167 + num_runs: int,

168 + dry_run: bool,

169 +):

170 + comb_data = list[dict[str, object]]()

171 +

172 + for run_number in range(num_runs):

173 + run_data = run_benchmark(

174 + server,

175 + bench_cmd,

176 + serve_overrides=serve_comb,

177 + bench_overrides=bench_comb,

178 + run_number=run_number,

179 + output_path=_get_comb_run_path(base_path, run_number),

180 + dry_run=dry_run,

181 + )

182 +

183 + if run_data is not None:

184 + comb_data.append(run_data)

185 +

186 + if dry_run:

187 + return None

188 +

189 + with _get_comb_run_path(base_path, run_number=None).open("w") as f:

190 + json.dump(comb_data, f, indent=4)

191 +

192 + return comb_data

193 +

194 +

195 +def run_combs(

196 + serve_cmd: list[str],

197 + bench_cmd: list[str],

198 + after_bench_cmd: list[str],

199 + *,

200 + show_stdout: bool,

201 + serve_params: ParameterSweep,

202 + bench_params: ParameterSweep,

203 + output_dir: Path,

204 + num_runs: int,

205 + dry_run: bool,

206 +):

207 + all_data = list[dict[str, object]]()

208 + for serve_comb in serve_params:

209 + with (

210 + run_server(

211 + serve_cmd,

212 + after_bench_cmd,

213 + show_stdout=show_stdout,

214 + serve_overrides=serve_comb,

215 + dry_run=dry_run,

216 + )

217 + if _comb_needs_server(serve_comb, bench_params, output_dir)

218 + else contextlib.nullcontext()

219 + ) as server:

220 + for bench_comb in bench_params:

221 + base_path = _get_comb_base_path(output_dir, serve_comb, bench_comb)

222 +

223 + comb_data = run_comb(

224 + server,

225 + bench_cmd,

226 + serve_comb=serve_comb,

227 + bench_comb=bench_comb,

228 + base_path=base_path,

229 + num_runs=num_runs,

230 + dry_run=dry_run,

231 + )

232 +

233 + if comb_data is not None:

234 + all_data.extend(comb_data)

235 +

236 + if dry_run:

237 + return None

238 +

239 + combined_df = pd.DataFrame.from_records(all_data)

240 + combined_df.to_csv(output_dir / "summary.csv")

241 +

242 + return combined_df

243 +

244 +

245 +@dataclass

246 +class SweepServeArgs:

247 + serve_cmd: list[str]

248 + bench_cmd: list[str]

249 + after_bench_cmd: list[str]

250 + show_stdout: bool

251 + serve_params: ParameterSweep

252 + bench_params: ParameterSweep

253 + output_dir: Path

254 + num_runs: int

255 + dry_run: bool

256 + resume: str | None

257 +

258 + @classmethod

259 + def from_cli_args(cls, args: argparse.Namespace):

260 + serve_cmd = shlex.split(args.serve_cmd)

261 + bench_cmd = shlex.split(args.bench_cmd)

262 + after_bench_cmd = (

263 + [] if args.after_bench_cmd is None else shlex.split(args.after_bench_cmd)

264 + )

265 +

266 + if args.serve_params:

267 + serve_params = ParameterSweep.read_json(args.serve_params)

268 + else:

269 + # i.e.: run serve_cmd without any modification

270 + serve_params = ParameterSweep.from_records([{}])

271 +

272 + if args.bench_params:

273 + bench_params = ParameterSweep.read_json(args.bench_params)

274 + else:

275 + # i.e.: run bench_cmd without any modification

276 + bench_params = ParameterSweep.from_records([{}])

277 +

278 + num_runs = args.num_runs

279 + if num_runs < 1:

280 + raise ValueError("`num_runs` should be at least 1.")

281 +

282 + return cls(

283 + serve_cmd=serve_cmd,

284 + bench_cmd=bench_cmd,

285 + after_bench_cmd=after_bench_cmd,

286 + show_stdout=args.show_stdout,

287 + serve_params=serve_params,

288 + bench_params=bench_params,

289 + output_dir=Path(args.output_dir),

290 + num_runs=num_runs,

291 + dry_run=args.dry_run,

292 + resume=args.resume,

293 + )

294 +

295 + @classmethod

296 + def add_cli_args(cls, parser: argparse.ArgumentParser) -> argparse.ArgumentParser:

297 + parser.add_argument(

298 + "--serve-cmd",

299 + type=str,

300 + required=True,

301 + help="The command used to run the server: `vllm serve ...`",

302 + )

303 + parser.add_argument(

304 + "--bench-cmd",

305 + type=str,

306 + required=True,

307 + help="The command used to run the benchmark: `vllm bench serve ...`",

308 + )

309 + parser.add_argument(

310 + "--after-bench-cmd",

311 + type=str,

312 + default=None,

313 + help="After a benchmark run is complete, invoke this command instead of "

314 + "the default `ServerWrapper.clear_cache()`.",

315 + )

316 + parser.add_argument(

317 + "--show-stdout",

318 + action="store_true",

319 + help="If set, logs the standard output of subcommands. "

320 + "Useful for debugging but can be quite spammy.",

321 + )

322 + parser.add_argument(

323 + "--serve-params",

324 + type=str,

325 + default=None,

326 + help="Path to JSON file containing a list of parameter combinations "

327 + "for the `vllm serve` command. "

328 + "If both `serve_params` and `bench_params` are given, "

329 + "this script will iterate over their Cartesian product.",

330 + )

331 + parser.add_argument(

332 + "--bench-params",

333 + type=str,

334 + default=None,

335 + help="Path to JSON file containing a list of parameter combinations "

336 + "for the `vllm bench serve` command. "

337 + "If both `serve_params` and `bench_params` are given, "

338 + "this script will iterate over their Cartesian product.",

339 + )

340 + parser.add_argument(

341 + "-o",

342 + "--output-dir",

343 + type=str,

344 + default="results",

345 + help="The directory to which results are written.",

346 + )

347 + parser.add_argument(

348 + "--num-runs",

349 + type=int,

350 + default=3,

351 + help="Number of runs per parameter combination.",

352 + )

353 + parser.add_argument(

354 + "--dry-run",

355 + action="store_true",

356 + help="If set, prints the commands to run, "

357 + "then exits without executing them.",

358 + )

359 + parser.add_argument(

360 + "--resume",

361 + type=str,

362 + default=None,

363 + help="Set this to the name of a directory under `output_dir` (which is a "

364 + "timestamp) to resume a previous execution of this script, i.e., only run "

365 + "parameter combinations for which there are still no output files.",

366 + )

367 +

368 + return parser

369 +

370 +

371 +def run_main(args: SweepServeArgs):

372 + timestamp = args.resume or datetime.now().strftime("%Y%m%d_%H%M%S")

373 + output_dir = args.output_dir / timestamp

374 +

375 + if args.resume and not output_dir.exists():

376 + raise ValueError(f"Cannot resume from non-existent directory ({output_dir})")

377 +

378 + try:

379 + return run_combs(

380 + serve_cmd=args.serve_cmd,

381 + bench_cmd=args.bench_cmd,

382 + after_bench_cmd=args.after_bench_cmd,

383 + show_stdout=args.show_stdout,

384 + serve_params=args.serve_params,

385 + bench_params=args.bench_params,

386 + output_dir=output_dir,

387 + num_runs=args.num_runs,

388 + dry_run=args.dry_run,

389 + )

390 + except BaseException as exc:

391 + raise RuntimeError(

392 + f"The script was terminated early. Use `--resume {timestamp}` "

393 + f"to continue the script from its last checkpoint."

394 + ) from exc

395 +

396 +

397 +def main(args: argparse.Namespace):

398 + run_main(SweepServeArgs.from_cli_args(args))

399 +

400 +

401 +if __name__ == "__main__":

402 + parser = argparse.ArgumentParser(

403 + description="Run vLLM server benchmark under multiple settings."

404 + )

405 + SweepServeArgs.add_cli_args(parser)

406 +

407 + main(parser.parse_args())

vllm/benchmarks/sweep/serve_sla.py new

+483

@@ -0,0 +1,483 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +import argparse

4 +import contextlib

5 +import json

6 +import math

7 +from dataclasses import asdict, dataclass

8 +from datetime import datetime

9 +from pathlib import Path

10 +from typing import Literal, get_args

11 +

12 +import pandas as pd

13 +from typing_extensions import assert_never

14 +

15 +from .param_sweep import ParameterSweep, ParameterSweepItem

16 +from .serve import SweepServeArgs, run_benchmark, run_server

17 +from .server import ServerProcess

18 +from .sla_sweep import SLASweep, SLASweepItem

19 +from .utils import sanitize_filename

20 +

21 +

22 +def _get_sla_base_path(

23 + output_dir: Path,

24 + serve_comb: ParameterSweepItem,

25 + bench_comb: ParameterSweepItem,

26 +):

27 + parts = list[str]()

28 + if serve_comb:

29 + parts.extend(("SERVE-", serve_comb.as_text(sep="-")))

30 + if bench_comb:

31 + parts.extend(("BENCH-", bench_comb.as_text(sep="-")))

32 +

33 + return output_dir / sanitize_filename("-".join(parts))

34 +

35 +

36 +def _get_sla_iter_path(

37 + base_path: Path,

38 + sla_comb: SLASweepItem,

39 + sla_variable: str,

40 + sla_value: int | None,

41 +):

42 + if sla_value is None:

43 + prefix = sla_comb.as_text(sep="-")

44 + return base_path / f"SLA--{prefix}.json"

45 +

46 + return base_path / f"{sla_variable}={sla_value}"

47 +

48 +

49 +def _get_sla_run_path(iter_path: Path, run_number: int | None):

50 + if run_number is None:

51 + return iter_path / "summary.json"

52 +

53 + return iter_path / f"run={run_number}.json"

54 +

55 +

56 +def _sla_needs_server(

57 + serve_comb: ParameterSweepItem,

58 + bench_combs: ParameterSweep,

59 + sla_combs: SLASweep,

60 + sla_variable: str,

61 + output_dir: Path,

62 +):

63 + for bench_comb in bench_combs:

64 + base_path = _get_sla_base_path(output_dir, serve_comb, bench_comb)

65 + for sla_comb in sla_combs:

66 + if not _get_sla_iter_path(

67 + base_path,

68 + sla_comb,

69 + sla_variable,

70 + sla_value=None,

71 + ).exists():

72 + return True

73 +

74 + return False

75 +

76 +

77 +def run_sla(

78 + server: ServerProcess | None,

79 + bench_cmd: list[str],

80 + *,

81 + serve_comb: ParameterSweepItem,

82 + bench_comb: ParameterSweepItem,

83 + iter_path: Path,

84 + num_runs: int,

85 + dry_run: bool,

86 +):

87 + iter_data = list[dict[str, object]]()

88 +

89 + for run_number in range(num_runs):

90 + run_data = run_benchmark(

91 + server,

92 + bench_cmd,

93 + serve_overrides=serve_comb,

94 + bench_overrides=bench_comb,

95 + run_number=run_number,

96 + output_path=_get_sla_run_path(iter_path, run_number),

97 + dry_run=dry_run,

98 + )

99 +

100 + if run_data is not None:

101 + iter_data.append(run_data)

102 +

103 + if dry_run:

104 + return None

105 +

106 + with _get_sla_run_path(iter_path, run_number=None).open("w") as f:

107 + json.dump(iter_data, f, indent=4)

108 +

109 + return iter_data

110 +

111 +

112 +SLAVariable = Literal["request_rate", "max_concurrency"]

113 +

114 +

115 +def _estimate_sla_value(run_data: dict[str, object], sla_variable: SLAVariable):

116 + request_throughput = float(run_data["request_throughput"]) # type: ignore

117 + if sla_variable == "request_rate":

118 + return request_throughput

119 + if sla_variable == "max_concurrency":

120 + mean_latency_ms = float(run_data["mean_e2el_ms"]) # type: ignore

121 + return request_throughput * mean_latency_ms / 1000

122 +

123 + assert_never(sla_variable)

124 +

125 +

126 +def _estimate_sla_bounds(

127 + server: ServerProcess | None,

128 + bench_cmd: list[str],

129 + *,

130 + serve_comb: ParameterSweepItem,

131 + bench_comb: ParameterSweepItem,

132 + sla_comb: SLASweepItem,

133 + base_path: Path,

134 + num_runs: int,

135 + dry_run: bool,

136 + sla_variable: SLAVariable,

137 + init_value: int,

138 + max_value: int,

139 +):

140 + sla_data = list[dict[str, object]]()

141 +

142 + max_passing: int = 0

143 + min_failing: int = 0

144 +

145 + val: int = init_value

146 + assert val > 0

147 +

148 + while True:

149 + print(f"Testing {sla_variable}: {val} req/s")

150 +

151 + iter_data = run_sla(

152 + server,

153 + bench_cmd,

154 + serve_comb=serve_comb,

155 + bench_comb=bench_comb | {sla_variable: val},

156 + iter_path=_get_sla_iter_path(base_path, sla_comb, sla_variable, val),

157 + num_runs=num_runs,

158 + dry_run=dry_run,

159 + )

160 +

161 + assert iter_data is not None

162 + sla_data.extend(iter_data)

163 +

164 + iter_data_mean = {

165 + k: sum(float(run_data[k]) for run_data in iter_data) / len(iter_data) # type: ignore

166 + for k in sla_comb

167 + }

168 +

169 + sla_results = [

170 + criterion.print_and_validate(iter_data_mean, k)

171 + for k, criterion in sla_comb.items()

172 + ]

173 +

174 + if all(sla_results):

175 + print("SLA criteria are met.")

176 + max_passing = val

177 + val *= 2

178 + else:

179 + print("SLA criteria are not met.")

180 + min_failing = val

181 + break

182 +

183 + if val >= max_value:

184 + break

185 +

186 + return sla_data, (max_passing, min_failing)

187 +

188 +

189 +def _find_sla_value(

190 + server: ServerProcess | None,

191 + bench_cmd: list[str],

192 + *,

193 + serve_comb: ParameterSweepItem,

194 + bench_comb: ParameterSweepItem,

195 + sla_comb: SLASweepItem,

196 + base_path: Path,

197 + num_runs: int,

198 + dry_run: bool,

199 + sla_variable: SLAVariable,

200 + min_value: int,

201 + max_value: int,

202 +):

203 + sla_data = list[dict[str, object]]()

204 +

205 + left: int = min_value

206 + right: int = max_value

207 +

208 + while True:

209 + val = (left + right) // 2

210 + print(f"Testing {sla_variable}: {val} req/s")

211 +

212 + iter_data = run_sla(

213 + server,

214 + bench_cmd,

215 + serve_comb=serve_comb,

216 + bench_comb=bench_comb | {sla_variable: val},

217 + iter_path=_get_sla_iter_path(base_path, sla_comb, sla_variable, val),

218 + num_runs=num_runs,

219 + dry_run=dry_run,

220 + )

221 +

222 + assert iter_data is not None

223 + sla_data.extend(iter_data)

224 +

225 + iter_data_mean = {

226 + k: sum(float(run_data[k]) for run_data in iter_data) / len(iter_data) # type: ignore

227 + for k in sla_comb

228 + }

229 +

230 + sla_results = [

231 + criterion.print_and_validate(iter_data_mean, k)

232 + for k, criterion in sla_comb.items()

233 + ]

234 +

235 + if all(sla_results):

236 + print("SLA criteria are met.")

237 + left = val

238 + else:

239 + print("SLA criteria are not met.")

240 + right = val

241 +

242 + if right - left <= 1:

243 + break

244 +

245 + return sla_data, left

246 +

247 +

248 +def search_sla(

249 + server: ServerProcess | None,

250 + bench_cmd: list[str],

251 + *,

252 + serve_comb: ParameterSweepItem,

253 + bench_comb: ParameterSweepItem,

254 + sla_comb: SLASweepItem,

255 + sla_variable: SLAVariable,

256 + sla_inf_value: int = 65536, # The value that represents infinite QPS

257 + base_path: Path,

258 + num_runs: int,

259 + dry_run: bool,

260 +):

261 + print("[SLA START]")

262 + print(f"SLA criteria: {sla_comb.as_text()}")

263 +

264 + sla_data_0 = run_sla(

265 + server,

266 + bench_cmd,

267 + serve_comb=serve_comb,

268 + bench_comb=bench_comb | {sla_variable: sla_inf_value},

269 + iter_path=_get_sla_iter_path(base_path, sla_comb, sla_variable, sla_inf_value),

270 + num_runs=num_runs,

271 + dry_run=dry_run,

272 + )

273 + if sla_data_0 is None:

274 + assert dry_run

275 + print("Omitting SLA search.")

276 + print("[SLA END]")

277 + return None

278 +

279 + sla_init_value = math.ceil(

280 + sum(_estimate_sla_value(item, sla_variable) for item in sla_data_0)

281 + / len(sla_data_0)

282 + )

283 + print(f"Initial {sla_variable} to search: {sla_init_value} req/s.")

284 +

285 + sla_data_1, (sla_min, sla_max) = _estimate_sla_bounds(

286 + server,

287 + bench_cmd,

288 + serve_comb=serve_comb,

289 + bench_comb=bench_comb,

290 + sla_comb=sla_comb,

291 + base_path=base_path,

292 + num_runs=num_runs,

293 + dry_run=dry_run,

294 + sla_variable=sla_variable,

295 + init_value=sla_init_value,

296 + max_value=sla_inf_value,

297 + )

298 + print(f"Range of {sla_variable} to search: [{sla_min}, {sla_max}] req/s.")

299 +

300 + sla_data_2, sla_value = _find_sla_value(

301 + server,

302 + bench_cmd,

303 + serve_comb=serve_comb,

304 + bench_comb=bench_comb,

305 + sla_comb=sla_comb,

306 + base_path=base_path,

307 + num_runs=num_runs,

308 + dry_run=dry_run,

309 + sla_variable=sla_variable,

310 + min_value=sla_min,

311 + max_value=sla_max,

312 + )

313 +

314 + sla_data = sla_data_0 + sla_data_1 + sla_data_2

315 + print(f"Maximum {sla_variable} for SLA: {sla_value} req/s.")

316 +

317 + with _get_sla_iter_path(

318 + base_path,

319 + sla_comb,

320 + sla_variable,

321 + sla_value=None,

322 + ).open("w") as f:

323 + json.dump(sla_data, f, indent=4)

324 +

325 + print("[SLA END]")

326 +

327 + return sla_data

328 +

329 +

330 +def run_slas(

331 + serve_cmd: list[str],

332 + bench_cmd: list[str],

333 + after_bench_cmd: list[str],

334 + *,

335 + show_stdout: bool,

336 + serve_params: ParameterSweep,

337 + bench_params: ParameterSweep,

338 + sla_params: SLASweep,

339 + sla_variable: SLAVariable,

340 + output_dir: Path,

341 + num_runs: int,

342 + dry_run: bool,

343 +):

344 + if any(bench_comb.has_param(sla_variable) for bench_comb in bench_params):

345 + raise ValueError(

346 + f"You should not override `{sla_variable}` in `bench_params` in SLA mode, "

347 + "since it is supposed to be determined automatically."

348 + )

349 +

350 + all_data = list[dict[str, object]]()

351 + for serve_comb in serve_params:

352 + with (

353 + run_server(

354 + serve_cmd,

355 + after_bench_cmd,

356 + show_stdout=show_stdout,

357 + serve_overrides=serve_comb,

358 + dry_run=dry_run,

359 + )

360 + if _sla_needs_server(

361 + serve_comb,

362 + bench_params,

363 + sla_params,

364 + sla_variable,

365 + output_dir,

366 + )

367 + else contextlib.nullcontext()

368 + ) as server:

369 + for bench_comb in bench_params:

370 + for sla_comb in sla_params:

371 + base_path = _get_sla_base_path(output_dir, serve_comb, bench_comb)

372 +

373 + comb_data = search_sla(

374 + server,

375 + bench_cmd,

376 + serve_comb=serve_comb,

377 + bench_comb=bench_comb,

378 + sla_comb=sla_comb,

379 + sla_variable=sla_variable,

380 + base_path=base_path,

381 + num_runs=num_runs,

382 + dry_run=dry_run,

383 + )

384 +

385 + if comb_data is not None:

386 + all_data.extend(comb_data)

387 +

388 + if dry_run:

389 + return None

390 +

391 + combined_df = pd.DataFrame.from_records(all_data)

392 + combined_df.to_csv(output_dir / "summary.csv")

393 +

394 + return combined_df

395 +

396 +

397 +@dataclass

398 +class SweepServeSLAArgs(SweepServeArgs):

399 + sla_params: SLASweep

400 + sla_variable: SLAVariable

401 +

402 + @classmethod

403 + def from_cli_args(cls, args: argparse.Namespace):

404 + base_args = super().from_cli_args(args)

405 +

406 + if args.sla_params:

407 + sla_params = SLASweep.read_json(args.sla_params)

408 + else:

409 + sla_params = SLASweep.from_records([])

410 +

411 + return cls(

412 + **asdict(base_args),

413 + sla_params=sla_params,

414 + sla_variable=args.sla_variable,

415 + )

416 +

417 + @classmethod

418 + def add_cli_args(cls, parser: argparse.ArgumentParser) -> argparse.ArgumentParser:

419 + parser = super().add_cli_args(parser)

420 +

421 + parser.add_argument(

422 + "--sla-params",

423 + type=str,

424 + required=True,

425 + help="Path to JSON file containing a list of SLA constraints to satisfy. "

426 + 'Each constraint is expressed in `{"<KEY>": "<OP><VALUE>"}` format, '

427 + 'e.g.: `{"p99_e2el_ms": "<=500"}` means that '

428 + "the E2E latency should be less than 500ms 99%% of the time. "

429 + "Setting this option runs this script in SLA mode, which searches for "

430 + "the maximum `sla_variable` that satisfies the constraints for "

431 + "each combination of `serve_params`, `bench_params`, and `sla_params`.",

432 + )

433 + parser.add_argument(

434 + "--sla-variable",

435 + type=str,

436 + choices=get_args(SLAVariable),

437 + default="request_rate",

438 + help="Whether to tune request rate or maximum concurrency to satisfy "

439 + "the SLA constraints.",

440 + )

441 +

442 + return parser

443 +

444 +

445 +def run_main(args: SweepServeSLAArgs):

446 + timestamp = args.resume or datetime.now().strftime("%Y%m%d_%H%M%S")

447 + output_dir = args.output_dir / timestamp

448 +

449 + if args.resume and not output_dir.exists():

450 + raise ValueError(f"Cannot resume from non-existent directory ({output_dir})")

451 +

452 + try:

453 + return run_slas(

454 + serve_cmd=args.serve_cmd,

455 + bench_cmd=args.bench_cmd,

456 + after_bench_cmd=args.after_bench_cmd,

457 + show_stdout=args.show_stdout,

458 + serve_params=args.serve_params,

459 + bench_params=args.bench_params,

460 + sla_params=args.sla_params,

461 + sla_variable=args.sla_variable,

462 + output_dir=output_dir,

463 + num_runs=args.num_runs,

464 + dry_run=args.dry_run,

465 + )

466 + except BaseException as exc:

467 + raise RuntimeError(

468 + f"The script was terminated early. Use `--resume {timestamp}` "

469 + f"to continue the script from its last checkpoint."

470 + ) from exc

471 +

472 +

473 +def main(args: argparse.Namespace):

474 + run_main(SweepServeSLAArgs.from_cli_args(args))

475 +

476 +

477 +if __name__ == "__main__":

478 + parser = argparse.ArgumentParser(

479 + description="Tune a variable to meet SLAs under multiple settings."

480 + )

481 + SweepServeSLAArgs.add_cli_args(parser)

482 +

483 + main(parser.parse_args())

vllm/benchmarks/sweep/server.py new

+114

@@ -0,0 +1,114 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +import contextlib

4 +import os

5 +import signal

6 +import subprocess

7 +from types import TracebackType

8 +

9 +import requests

10 +from typing_extensions import Self

11 +

12 +

13 +class ServerProcess:

14 + def __init__(

15 + self,

16 + server_cmd: list[str],

17 + after_bench_cmd: list[str],

18 + *,

19 + show_stdout: bool,

20 + ) -> None:

21 + super().__init__()

22 +

23 + self.server_cmd = server_cmd

24 + self.after_bench_cmd = after_bench_cmd

25 + self.show_stdout = show_stdout

26 +

27 + def __enter__(self) -> Self:

28 + self.start()

29 + return self

30 +

31 + def __exit__(

32 + self,

33 + exc_type: type[BaseException] | None,

34 + exc_value: BaseException | None,

35 + exc_traceback: TracebackType | None,

36 + ) -> None:

37 + self.stop()

38 +

39 + def start(self):

40 + # Create new process for clean termination

41 + self._server_process = subprocess.Popen(

42 + self.server_cmd,

43 + start_new_session=True,

44 + stdout=None if self.show_stdout else subprocess.DEVNULL,

45 + # Need `VLLM_SERVER_DEV_MODE=1` for `_reset_caches`

46 + env=os.environ | {"VLLM_SERVER_DEV_MODE": "1"},

47 + )

48 +

49 + def stop(self):

50 + server_process = self._server_process

51 +

52 + if server_process.poll() is None:

53 + # In case only some processes have been terminated

54 + with contextlib.suppress(ProcessLookupError):

55 + # We need to kill both API Server and Engine processes

56 + os.killpg(os.getpgid(server_process.pid), signal.SIGKILL)

57 +

58 + def run_subcommand(self, cmd: list[str]):

59 + return subprocess.run(

60 + cmd,

61 + stdout=None if self.show_stdout else subprocess.DEVNULL,

62 + check=True,

63 + )

64 +

65 + def after_bench(self) -> None:

66 + if not self.after_bench_cmd:

67 + self.reset_caches()

68 + return

69 +

70 + self.run_subcommand(self.after_bench_cmd)

71 +

72 + def _get_vllm_server_address(self) -> str:

73 + server_cmd = self.server_cmd

74 +

75 + for host_key in ("--host",):

76 + if host_key in server_cmd:

77 + host = server_cmd[server_cmd.index(host_key) + 1]

78 + break

79 + else:

80 + host = "localhost"

81 +

82 + for port_key in ("-p", "--port"):

83 + if port_key in server_cmd:

84 + port = int(server_cmd[server_cmd.index(port_key) + 1])

85 + break

86 + else:

87 + port = 8000 # The default value in vllm serve

88 +

89 + return f"http://{host}:{port}"

90 +

91 + def reset_caches(self) -> None:

92 + server_cmd = self.server_cmd

93 +

94 + # Use `.endswith()` to match `/bin/...`

95 + if server_cmd[0].endswith("vllm"):

96 + server_address = self._get_vllm_server_address()

97 + print(f"Resetting caches at {server_address}")

98 +

99 + res = requests.post(f"{server_address}/reset_prefix_cache")

100 + res.raise_for_status()

101 +

102 + res = requests.post(f"{server_address}/reset_mm_cache")

103 + res.raise_for_status()

104 + elif server_cmd[0].endswith("infinity_emb"):

105 + if "--vector-disk-cache" in server_cmd:

106 + raise NotImplementedError(

107 + "Infinity server uses caching but does not expose a method "

108 + "to reset the cache"

109 + )

110 + else:

111 + raise NotImplementedError(

112 + f"No implementation of `reset_caches` for `{server_cmd[0]}` server. "

113 + "Please specify a custom command via `--after-bench-cmd`."

114 + )

vllm/benchmarks/sweep/sla_sweep.py new

+132

@@ -0,0 +1,132 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +import json

4 +import os

5 +from abc import ABC, abstractmethod

6 +from dataclasses import dataclass

7 +

8 +from typing_extensions import override

9 +

10 +

11 +@dataclass

12 +class SLACriterionBase(ABC):

13 + target: float

14 +

15 + @abstractmethod

16 + def validate(self, actual: float) -> bool:

17 + """Return `True` if this criterion is met; otherwise `False`."""

18 + raise NotImplementedError

19 +

20 + @abstractmethod

21 + def format_cond(self, lhs: str) -> str:

22 + raise NotImplementedError

23 +

24 + def print_and_validate(

25 + self,

26 + metrics: dict[str, float],

27 + metrics_key: str,

28 + ) -> bool:

29 + metric = metrics[metrics_key]

30 + result = self.validate(metric)

31 +

32 + cond = self.format_cond(f"{metrics_key} = {metric:.2f}")

33 + print(f"Validating SLA: {cond} | " + ("PASSED" if result else "FAILED"))

34 +

35 + return result

36 +

37 +

38 +@dataclass

39 +class SLALessThan(SLACriterionBase):

40 + @override

41 + def validate(self, actual: float) -> bool:

42 + return actual < self.target

43 +

44 + @override

45 + def format_cond(self, lhs: str) -> str:

46 + return f"{lhs}<{self.target:.2f}"

47 +

48 +

49 +@dataclass

50 +class SLALessThanOrEqualTo(SLACriterionBase):

51 + @override

52 + def validate(self, actual: float) -> bool:

53 + return actual <= self.target

54 +

55 + @override

56 + def format_cond(self, lhs: str) -> str:

57 + return f"{lhs}<={self.target:.2f}"

58 +

59 +

60 +@dataclass

61 +class SLAGreaterThan(SLACriterionBase):

62 + @override

63 + def validate(self, actual: float) -> bool:

64 + return actual > self.target

65 +

66 + @override

67 + def format_cond(self, lhs: str) -> str:

68 + return f"{lhs}>{self.target:.2f}"

69 +

70 +

71 +@dataclass

72 +class SLAGreaterThanOrEqualTo(SLACriterionBase):

73 + @override

74 + def validate(self, actual: float) -> bool:

75 + return actual >= self.target

76 +

77 + @override

78 + def format_cond(self, lhs: str) -> str:

79 + return f"{lhs}>={self.target:.2f}"

80 +

81 +

82 +# NOTE: The ordering is important! Match longer op_keys first

83 +SLA_CRITERIA: dict[str, type[SLACriterionBase]] = {

84 + "<=": SLALessThanOrEqualTo,

85 + ">=": SLAGreaterThanOrEqualTo,

86 + "<": SLALessThan,

87 + ">": SLAGreaterThan,

88 +}

89 +

90 +

91 +class SLASweep(list["SLASweepItem"]):

92 + @classmethod

93 + def read_json(cls, filepath: os.PathLike):

94 + with open(filepath, "rb") as f:

95 + records = json.load(f)

96 +

97 + return cls.from_records(records)

98 +

99 + @classmethod

100 + def from_records(cls, records: list[dict[str, str]]):

101 + if not isinstance(records, list):

102 + raise TypeError(

103 + f"The SLA sweep should be a list of dictionaries, "

104 + f"but found type: {type(records)}"

105 + )

106 +

107 + return cls(SLASweepItem.from_record(record) for record in records)

108 +

109 +

110 +class SLASweepItem(dict[str, SLACriterionBase]):

111 + @classmethod

112 + def from_record(cls, record: dict[str, str]):

113 + sla_criteria: dict[str, SLACriterionBase] = {}

114 +

115 + for metric_key, metric_value in record.items():

116 + for op_key in SLA_CRITERIA:

117 + if metric_value.startswith(op_key):

118 + sla_criteria[metric_key] = SLA_CRITERIA[op_key](

119 + float(metric_value.removeprefix(op_key))

120 + )

121 + break

122 + else:

123 + raise ValueError(

124 + f"Invalid operator for "

125 + f"SLA constraint '{metric_key}={metric_value}'. "

126 + f"Valid operators are: {sorted(SLA_CRITERIA)}",

127 + )

128 +

129 + return cls(sla_criteria)

130 +

131 + def as_text(self, sep: str = ", ") -> str:

132 + return sep.join(v.format_cond(k) for k, v in self.items())

vllm/benchmarks/sweep/utils.py new

@@ -0,0 +1,4 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +def sanitize_filename(filename: str) -> str:

4 + return filename.replace("/", "_").replace("..", "__").strip("'").strip('"')