@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / d3a51da92a
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 7, 16:30 UTC

[Benchmark] Simplify SLA scan (#35306)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Feb 26, 2026 at 14:35 UTC d3a51da92a031f6c1758771a2b13976ace2eece2
8 files changed +254 -800
docs/benchmarking/cli.md
+5
@@ -4,6 +4,11 @@ This section guides you through running benchmark tests with the extensive datas
4
5 It's a living document, updated as new features and datasets become available.
6
7 +!!! tip
8 + The benchmarks described on this page are mainly for evaluating specific vLLM features as well as regression testing.
9 +
10 + For benchmarking production vLLM servers, we recommend [GuideLLM](https://github.com/vllm-project/guidellm), an established performance benchmarking framework with live progress updates and automatic report generation. It is also more flexible than `vllm bench serve` in terms of dataset loading, request formatting, and workload patterns.
11 +
12 ## Dataset Overview
13
14 <style>
docs/benchmarking/sweeps.md
+47 -41
@@ -1,10 +1,15 @@
1 # Parameter Sweeps
2
3 +`vllm bench sweep` is a suite of commands designed to run benchmarks across multiple configurations and compare them by visualizing the results.
4 +
5 ## Online Benchmark
6
7 ### Basic
8
7 -`vllm bench sweep serve` automatically starts `vllm serve` and runs `vllm bench serve` to evaluate vLLM over multiple configurations.
9 +`vllm bench sweep serve` starts `vllm serve` and iteratively runs `vllm bench serve` for each server configuration.
10 +
11 +!!! tip
12 + If you only need to run benchmarks for a single server configuration, consider using [GuideLLM](https://github.com/vllm-project/guidellm), an established performance benchmarking framework with live progress updates and automatic report generation. It is also more flexible than `vllm bench serve` in terms of dataset loading, request formatting, and workload patterns.
13
14 Follow these steps to run the script:
15
@@ -50,14 +55,17 @@ Follow these steps to run the script:
55 ```json
56 [
57 {
58 + "_benchmark_name": "scenario_A",
59 "random_input_len": 128,
60 "random_output_len": 32
61 },
62 {
63 + "_benchmark_name": "scenario_B",
64 "random_input_len": 256,
65 "random_output_len": 64
66 },
67 {
68 + "_benchmark_name": "scenario_C",
69 "random_input_len": 512,
70 "random_output_len": 128
71 }
@@ -77,6 +85,8 @@ vllm bench sweep serve \
85 -o benchmarks/results
86 ```
87
88 +By default, each parameter combination is benchmarked 3 times to make the results more reliable. You can adjust the number of runs by setting `--num-runs`.
89 +
90 !!! important
91 If both `--serve-params` and `--bench-params` are passed, the script will iterate over the Cartesian product between them.
92 You can use `--dry-run` to preview the commands to be run.
@@ -86,60 +96,40 @@ vllm bench sweep serve \
96 In case you are using a custom `--serve-cmd`, you can override the commands used for resetting the state by setting `--after-bench-cmd`.
97
98 !!! note
89 - By default, each parameter combination is run 3 times to make the results more reliable. You can adjust the number of runs by setting `--num-runs`.
99 + You should set `_benchmark_name` to provide a human-readable name for parameter combinations involving many variables.
100 + This becomes mandatory if the file name would otherwise exceed the maximum path length allowed by the filesystem.
101
102 !!! tip
92 - You can use the `--resume` option to continue the parameter sweep if one of the runs failed.
93 -
94 -### SLA auto-tuner
103 + You can use the `--resume` option to continue the parameter sweep if an unexpected error occurs, e.g., timeout when connecting to HF Hub.
104
96 -`vllm bench sweep serve_sla` is a wrapper over `vllm bench sweep serve` that tunes either the request rate or concurrency (choose using `--sla-variable`) in order to satisfy the SLA constraints given by `--sla-params`.
105 +### SLA Scanner
106
98 -For example, to ensure E2E latency within different target values for 99% of requests:
99 -
100 -```json
101 -[
102 - {
103 - "p99_e2el_ms": "<=200"
104 - },
105 - {
106 - "p99_e2el_ms": "<=500"
107 - },
108 - {
109 - "p99_e2el_ms": "<=1000"
110 - },
111 - {
112 - "p99_e2el_ms": "<=2000"
113 - }
114 -]
115 -```
107 +`vllm bench sweep serve_sla` is a variant of `vllm bench sweep serve` that scans through values of request rate or concurrency (choose using `--sla-variable`) in order to find the tradeoff between latency and throughput. The results can then be [visualized](#visualization) to determine the feasible SLAs.
108
109 Example command:
110
111 ```bash
112 vllm bench sweep serve_sla \
113 --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \
122 - --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json' \
114 + --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 100' \
115 --serve-params benchmarks/serve_hparams.json \
124 - --bench-params benchmarks/bench_hparams.json \
125 - --sla-params benchmarks/sla_hparams.json \
126 - --sla-variable max_concurrency \
116 + --bench-params benchmarks/bench_hparams.json
117 -o benchmarks/results
118 ```
119
130 -The algorithm for adjusting the SLA variable is as follows:
120 +The algorithm for scanning through different values of `sla_variable` can be summarized as follows:
121
132 -1. Run the benchmark once with maximum possible QPS, and once with minimum possible QPS. For each run, calculate the distance of the SLA metrics from their targets, resulting in data points of QPS vs SLA distance.
133 -2. Perform spline interpolation between the data points to estimate the QPS that results in zero SLA distance.
134 -3. Run the benchmark with the estimated QPS and add the resulting data point to the history.
135 -4. Repeat Steps 2 and 3 until the maximum QPS that passes SLA and the minimum QPS that fails SLA in the history are close enough to each other.
122 +1. Run the benchmark once with `sla_variable = 1` to simulate serial inference. This results in the lowest possible latency and throughput.
123 +2. Run the benchmark once with `sla_variable = num_prompts` to simulate batch inference over the whole dataset. This results in the highest possible latency and throughput.
124 +3. Estimate the maximum value of `sla_variable` that can be supported by the server without oversaturating it.
125 +4. Run the benchmark over intermediate values of `sla_variable` uniformly using the remaining iterations.
126
137 -!!! important
138 - SLA tuning is applied over each combination of `--serve-params`, `--bench-params`, and `--sla-params`.
127 +You can override the number of iterations in the algorithm by setting `--sla-iters`.
128
140 - For a given combination of `--serve-params` and `--bench-params`, we share the benchmark results across `--sla-params` to avoid rerunning benchmarks with the same SLA variable value.
129 +!!! tip
130 + This is our equivalent of [GuideLLM's `--profile sweep`](https://github.com/vllm-project/guidellm/blob/v0.5.3/src/guidellm/benchmark/profiles.py#L575).
131
142 -### Startup
132 +## Startup Benchmark
133
134 `vllm bench sweep startup` runs `vllm bench startup` across parameter combinations to compare cold/warm startup time for different engine settings.
135
@@ -202,15 +192,28 @@ vllm bench sweep startup \
192
193 `vllm bench sweep plot` can be used to plot performance curves from parameter sweep results.
194
205 -Example command:
195 +Control the variables to plot via `--var-x` and `--var-y`, optionally applying `--filter-by` and `--bin-by` to the values. The plot is organized according to `--fig-by`, `--row-by`, `--col-by`, and `--curve-by`.
196 +
197 +Example commands for visualizing [SLA Scanner](#sla-scanner) results:
198
199 ```bash
200 +# Latency increases as the request rate increases
201 vllm bench sweep plot benchmarks/results/<timestamp> \
209 - --var-x max_concurrency \
202 + --var-x request_rate \
203 + --var-y p99_ttft_ms \
204 --row-by random_input_len \
205 --col-by random_output_len \
212 - --curve-by api_server_count,max_num_batched_tokens \
213 - --filter-by 'max_concurrency<=1024'
206 + --curve-by max_num_seqs,max_num_batched_tokens \
207 + --filter-by 'request_rate<=128'
208 +
209 +# Tradeoff between latency and throughput
210 +vllm bench sweep plot benchmarks/results/<timestamp> \
211 + --var-x request_throughput \
212 + --var-y median_ttft_ms \
213 + --row-by random_input_len \
214 + --col-by random_output_len \
215 + --curve-by max_num_seqs,max_num_batched_tokens \
216 + --filter-by 'request_rate<=128'
217 ```
218
219 !!! tip
@@ -233,3 +236,6 @@ Example:
236 vllm bench sweep plot_pareto benchmarks/results/<timestamp> \
237 --label-by max_concurrency,tensor_parallel_size,pipeline_parallel_size
238 ```
239 +
240 +!!! tip
241 + You can use `--dry-run` to preview the figures to be plotted.
tests/benchmarks/sweep/test_serve_sla.py deleted
-298
@@ -1,298 +0,0 @@
1 -# SPDX-License-Identifier: Apache-2.0
2 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 -import json
4 -from collections.abc import Callable
5 -from pathlib import Path
6 -from unittest.mock import patch
7 -
8 -from vllm.benchmarks.sweep.param_sweep import ParameterSweepItem
9 -from vllm.benchmarks.sweep.serve_sla import _get_sla_run_path, solve_sla
10 -from vllm.benchmarks.sweep.server import ServerProcess
11 -from vllm.benchmarks.sweep.sla_sweep import (
12 - SLACriterionBase,
13 - SLALessThan,
14 - SLALessThanOrEqualTo,
15 - SLASweepItem,
16 -)
17 -
18 -
19 -def _set_return_value(
20 - var2metric: Callable[[ParameterSweepItem], list[dict[str, float]]],
21 -):
22 - """
23 - Create a patch for run_sla with a specific function
24 - indicating the relationship between the benchmark combination
25 - (which includes the SLA variable) and the SLA criterion.
26 - """
27 -
28 - def mock_run_sla(
29 - server: ServerProcess | None,
30 - bench_cmd: list[str],
31 - *,
32 - serve_comb: ParameterSweepItem,
33 - bench_comb: ParameterSweepItem,
34 - iter_path: Path,
35 - num_runs: int,
36 - dry_run: bool,
37 - ):
38 - iter_data = var2metric(bench_comb)
39 -
40 - summary_path = _get_sla_run_path(iter_path, run_number=None)
41 - summary_path.parent.mkdir(parents=True, exist_ok=True)
42 - with summary_path.open("w") as f:
43 - json.dump(iter_data, f, indent=4)
44 -
45 - return iter_data
46 -
47 - return patch("vllm.benchmarks.sweep.serve_sla.run_sla", side_effect=mock_run_sla)
48 -
49 -
50 -def _var2metric_linear():
51 - def wrapped(bench_comb):
52 - x = float(bench_comb["request_rate"])
53 - y = x
54 -
55 - return [{"request_throughput": y}]
56 -
57 - return wrapped
58 -
59 -
60 -def _var2metric_concave(elbow_point: float):
61 - def wrapped(bench_comb):
62 - x = float(bench_comb["request_rate"])
63 - if x < elbow_point:
64 - y = 0.5 * (x - elbow_point) + elbow_point
65 - else:
66 - y = 1.5 * (x - elbow_point) + elbow_point
67 -
68 - return [{"request_throughput": y}]
69 -
70 - return wrapped
71 -
72 -
73 -def _var2metric_convex(elbow_point: float):
74 - def wrapped(bench_comb):
75 - x = float(bench_comb["request_rate"])
76 - if x < elbow_point:
77 - y = 1.5 * (x - elbow_point) + elbow_point
78 - else:
79 - y = 0.5 * (x - elbow_point) + elbow_point
80 -
81 - return [{"request_throughput": y}]
82 -
83 - return wrapped
84 -
85 -
86 -def _var2metric_quadratic(y_intercept: float):
87 - def wrapped(bench_comb):
88 - x = float(bench_comb["request_rate"])
89 - y = y_intercept + 0.1 * x**2
90 -
91 - return [{"request_throughput": y}]
92 -
93 - return wrapped
94 -
95 -
96 -def _var2metric_sqrt(y_intercept: float):
97 - def wrapped(bench_comb):
98 - x = float(bench_comb["request_rate"])
99 - y = y_intercept + 10 * x**0.5
100 -
101 - return [{"request_throughput": y}]
102 -
103 - return wrapped
104 -
105 -
106 -def _run_solve_sla(
107 - var2metric: Callable[[ParameterSweepItem], list[dict[str, float]]],
108 - criterion: SLACriterionBase,
109 - base_path: Path,
110 - min_value: int = 1,
111 - max_value: int = 100,
112 -):
113 - with _set_return_value(var2metric):
114 - result = solve_sla(
115 - server=None,
116 - bench_cmd=[],
117 - serve_comb=ParameterSweepItem(),
118 - bench_comb=ParameterSweepItem(),
119 - sla_comb=SLASweepItem({"request_throughput": criterion}),
120 - base_path=base_path,
121 - num_runs=1,
122 - dry_run=False,
123 - sla_variable="request_rate",
124 - sla_min_value=min_value,
125 - sla_max_value=max_value,
126 - )
127 - assert result is not None
128 -
129 - return result
130 -
131 -
132 -def test_solve_linear_sla_le(tmp_path):
133 - sla_data, history = _run_solve_sla(
134 - _var2metric_linear(),
135 - SLALessThanOrEqualTo(target=32),
136 - tmp_path,
137 - )
138 -
139 - assert history.get_max_passing() == 32
140 -
141 - assert {val: margin <= 0 for val, margin in history.items()} == {
142 - 100: False,
143 - 1: True,
144 - 32: True,
145 - 33: False,
146 - }
147 -
148 -
149 -def test_solve_linear_sla_lt(tmp_path):
150 - sla_data, history = _run_solve_sla(
151 - _var2metric_linear(),
152 - SLALessThan(target=32),
153 - tmp_path,
154 - )
155 -
156 - assert history.get_max_passing() == 31
157 -
158 - assert {val: margin <= 0 for val, margin in history.items()} == {
159 - 100: False,
160 - 1: True,
161 - 31: True,
162 - 32: False,
163 - }
164 -
165 -
166 -def test_solve_linear_sla_oob(tmp_path):
167 - sla_data, history = _run_solve_sla(
168 - _var2metric_linear(),
169 - SLALessThanOrEqualTo(target=32),
170 - tmp_path,
171 - min_value=64,
172 - )
173 -
174 - assert history.get_max_passing() == 64
175 - assert history.get_min_failing() == 64
176 -
177 - assert {val: margin <= 0 for val, margin in history.items()} == {
178 - 100: False,
179 - 64: False,
180 - }
181 -
182 -
183 -def test_solve_concave_sla_le(tmp_path):
184 - sla_data, history = _run_solve_sla(
185 - _var2metric_concave(elbow_point=32),
186 - SLALessThanOrEqualTo(target=24),
187 - tmp_path,
188 - )
189 -
190 - assert history.get_max_passing() == 16
191 -
192 - assert {val: margin <= 0 for val, margin in history.items()} == {
193 - 100: False,
194 - 1: True,
195 - 7: True,
196 - 13: True,
197 - 15: True,
198 - 16: True,
199 - 17: False,
200 - }
201 -
202 -
203 -def test_solve_convex_sla_le(tmp_path):
204 - sla_data, history = _run_solve_sla(
205 - _var2metric_convex(elbow_point=32),
206 - SLALessThanOrEqualTo(target=24),
207 - tmp_path,
208 - )
209 -
210 - assert history.get_max_passing() == 26
211 -
212 - assert {val: margin <= 0 for val, margin in history.items()} == {
213 - 100: False,
214 - 1: True,
215 - 48: False,
216 - 30: False,
217 - 24: True,
218 - 26: True,
219 - 27: False,
220 - }
221 -
222 -
223 -def test_solve_quadratic_sla_le(tmp_path):
224 - sla_data, history = _run_solve_sla(
225 - _var2metric_quadratic(y_intercept=10),
226 - SLALessThanOrEqualTo(target=50),
227 - tmp_path,
228 - )
229 -
230 - assert history.get_max_passing() == 20
231 -
232 - assert {val: margin <= 0 for val, margin in history.items()} == {
233 - 100: False,
234 - 1: True,
235 - 4: True,
236 - 20: True,
237 - 21: False,
238 - }
239 -
240 -
241 -def test_solve_sqrt_sla_le(tmp_path):
242 - sla_data, history = _run_solve_sla(
243 - _var2metric_sqrt(y_intercept=10),
244 - SLALessThanOrEqualTo(target=100),
245 - tmp_path,
246 - )
247 -
248 - assert history.get_max_passing() == 81
249 -
250 - assert {val: margin <= 0 for val, margin in history.items()} == {
251 - 100: False,
252 - 1: True,
253 - 89: False,
254 - 81: True,
255 - 82: False,
256 - }
257 -
258 -
259 -def test_solve_reuse_history(tmp_path):
260 - sla_data, history = _run_solve_sla(
261 - _var2metric_linear(),
262 - SLALessThanOrEqualTo(target=10),
263 - tmp_path,
264 - min_value=1,
265 - max_value=20,
266 - )
267 -
268 - assert history.get_max_passing() == 10
269 -
270 - assert {val: margin <= 0 for val, margin in history.items()} == {
271 - 20: False,
272 - 1: True,
273 - 10: True,
274 - 11: False,
275 - }
276 -
277 - sla_data, history = _run_solve_sla(
278 - _var2metric_linear(),
279 - SLALessThanOrEqualTo(target=30),
280 - tmp_path,
281 - min_value=21,
282 - max_value=40,
283 - )
284 -
285 - assert history.get_max_passing() == 30
286 -
287 - assert {val: margin <= 0 for val, margin in history.items()} == {
288 - # Items from the past run
289 - # (the margins are different because the target changed)
290 - 20: True,
291 - 1: True,
292 - 10: True,
293 - 11: True,
294 - # Items from this run
295 - 40: False,
296 - 30: True,
297 - 31: False,
298 - }
vllm/benchmarks/sweep/plot.py
+1 -1
@@ -576,7 +576,7 @@ class SweepPlotArgs:
576 parser.add_argument(
577 "--var-y",
578 type=str,
579 - default="p99_e2el_ms",
579 + default="p99_ttft_ms",
580 help="The variable for the y-axis",
581 )
582 parser.add_argument(
vllm/benchmarks/sweep/serve.py
+61 -26
@@ -92,7 +92,8 @@ def run_benchmark(
92 run_data: dict[str, object]
93
94 if output_path.exists():
95 - print("Found existing results. Skipping.")
95 + print("Found existing results.")
96 + print("[SKIPPED BENCHMARK]")
97
98 with output_path.open("rb") as f:
99 run_data = json.load(f)
@@ -167,6 +168,43 @@ def _comb_needs_server(
168 return False
169
170
171 +def server_ctx(
172 + serve_cmd: list[str],
173 + after_bench_cmd: list[str],
174 + *,
175 + show_stdout: bool,
176 + serve_comb: ParameterSweepItem,
177 + bench_params: ParameterSweep,
178 + output_dir: Path,
179 + dry_run: bool,
180 + server_ready_timeout: int = 300,
181 +):
182 + if not _comb_needs_server(serve_comb, bench_params, output_dir):
183 + return contextlib.nullcontext()
184 +
185 + return run_server(
186 + serve_cmd,
187 + after_bench_cmd,
188 + show_stdout=show_stdout,
189 + serve_overrides=serve_comb,
190 + dry_run=dry_run,
191 + server_ready_timeout=server_ready_timeout,
192 + )
193 +
194 +
195 +def _comb_is_valid(
196 + serve_comb: ParameterSweepItem,
197 + bench_comb: ParameterSweepItem,
198 + link_vars: list[tuple[str, str]],
199 +) -> bool:
200 + return all(
201 + serve_key in serve_comb
202 + and bench_key in bench_comb
203 + and serve_comb[serve_key] == bench_comb[bench_key]
204 + for serve_key, bench_key in link_vars
205 + )
206 +
207 +
208 def run_comb(
209 server: ServerProcess | None,
210 bench_cmd: list[str],
@@ -176,7 +214,11 @@ def run_comb(
214 base_path: Path,
215 num_runs: int,
216 dry_run: bool,
217 + link_vars: list[tuple[str, str]],
218 ):
219 + if not _comb_is_valid(serve_comb, bench_comb, link_vars):
220 + return None
221 +
222 comb_data = list[dict[str, object]]()
223
224 for run_number in range(num_runs):
@@ -208,37 +250,27 @@ def run_combs(
250 after_bench_cmd: list[str],
251 *,
252 show_stdout: bool,
253 + server_ready_timeout: int,
254 serve_params: ParameterSweep,
255 bench_params: ParameterSweep,
256 output_dir: Path,
257 num_runs: int,
258 dry_run: bool,
216 - links: list[tuple[str, str]],
217 - server_ready_timeout: int = 300,
259 + link_vars: list[tuple[str, str]],
260 ):
261 all_data = list[dict[str, object]]()
262 for serve_comb in serve_params:
221 - with (
222 - run_server(
223 - serve_cmd,
224 - after_bench_cmd,
225 - show_stdout=show_stdout,
226 - serve_overrides=serve_comb,
227 - dry_run=dry_run,
228 - server_ready_timeout=server_ready_timeout,
229 - )
230 - if _comb_needs_server(serve_comb, bench_params, output_dir)
231 - else contextlib.nullcontext()
263 + with server_ctx(
264 + serve_cmd,
265 + after_bench_cmd,
266 + show_stdout=show_stdout,
267 + serve_comb=serve_comb,
268 + bench_params=bench_params,
269 + output_dir=output_dir,
270 + dry_run=dry_run,
271 + server_ready_timeout=server_ready_timeout,
272 ) as server:
273 for bench_comb in bench_params:
234 - should_run = all(
235 - serve_key in serve_comb
236 - and bench_key in bench_comb
237 - and serve_comb[serve_key] == bench_comb[bench_key]
238 - for serve_key, bench_key in links
239 - )
240 - if not should_run:
241 - continue
274 base_path = _get_comb_base_path(output_dir, serve_comb, bench_comb)
275
276 comb_data = run_comb(
@@ -249,6 +281,7 @@ def run_combs(
281 base_path=base_path,
282 num_runs=num_runs,
283 dry_run=dry_run,
284 + link_vars=link_vars,
285 )
286
287 if comb_data is not None:
@@ -269,14 +302,14 @@ class SweepServeArgs:
302 bench_cmd: list[str]
303 after_bench_cmd: list[str]
304 show_stdout: bool
305 + server_ready_timeout: int
306 serve_params: ParameterSweep
307 bench_params: ParameterSweep
308 output_dir: Path
309 num_runs: int
310 dry_run: bool
311 resume: str | None
278 - link_vars: list[tuple[str, str]] | None
279 - server_ready_timeout: int
312 + link_vars: list[tuple[str, str]]
313
314 parser_name: ClassVar[str] = "serve"
315 parser_help: ClassVar[str] = "Run vLLM server benchmark under multiple settings."
@@ -300,7 +333,9 @@ class SweepServeArgs:
333 else:
334 # i.e.: run bench_cmd without any modification
335 bench_params = ParameterSweep.from_records([{}])
336 +
337 link_vars = cls.parse_link_vars(args.link_vars)
338 +
339 num_runs = args.num_runs
340 if num_runs < 1:
341 raise ValueError("`num_runs` should be at least 1.")
@@ -437,13 +472,13 @@ def run_main(args: SweepServeArgs):
472 bench_cmd=args.bench_cmd,
473 after_bench_cmd=args.after_bench_cmd,
474 show_stdout=args.show_stdout,
475 + server_ready_timeout=args.server_ready_timeout,
476 serve_params=args.serve_params,
477 bench_params=args.bench_params,
478 output_dir=output_dir,
479 num_runs=args.num_runs,
480 dry_run=args.dry_run,
445 - links=args.link_vars,
446 - server_ready_timeout=args.server_ready_timeout,
481 + link_vars=args.link_vars,
482 )
483 except BaseException as exc:
484 raise RuntimeError(
vllm/benchmarks/sweep/serve_sla.py
+138 -295
@@ -1,306 +1,162 @@
1 # SPDX-License-Identifier: Apache-2.0
2 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 import argparse
4 -import contextlib
5 -import json
4 +import math
5 from dataclasses import asdict, dataclass
6 from datetime import datetime
7 from pathlib import Path
8 from typing import ClassVar, Literal, get_args
9
10 +import numpy as np
11 +from typing_extensions import assert_never
12 +
13 from vllm.utils.import_utils import PlaceholderModule
14
15 from .param_sweep import ParameterSweep, ParameterSweepItem
14 -from .serve import SweepServeArgs, run_benchmark, run_server
16 +from .serve import (
17 + SweepServeArgs,
18 + _get_comb_base_path,
19 + run_comb,
20 + server_ctx,
21 +)
22 from .server import ServerProcess
16 -from .sla_sweep import SLASweep, SLASweepItem
17 -from .utils import sanitize_filename
23
24 try:
25 import pandas as pd
26 except ImportError:
27 pd = PlaceholderModule("pandas")
28
24 -try:
25 - from scipy.interpolate import PchipInterpolator
26 -except ImportError:
27 - PchipInterpolator = (
28 - PlaceholderModule("scipy")
29 - .placeholder_attr("interpolate")
30 - .placeholder_attr("PchipInterpolator")
31 - )
32 -
33 -
34 -def _get_sla_base_path(
35 - output_dir: Path,
36 - serve_comb: ParameterSweepItem,
37 - bench_comb: ParameterSweepItem,
38 -):
39 - parts = list[str]()
40 - if serve_comb:
41 - parts.extend(("SERVE-", serve_comb.as_text(sep="-")))
42 - if bench_comb:
43 - parts.extend(("BENCH-", bench_comb.as_text(sep="-")))
44 -
45 - return output_dir / sanitize_filename("-".join(parts))
46 -
47 -
48 -def _get_sla_iter_path(
49 - base_path: Path,
50 - sla_comb: SLASweepItem,
51 - sla_variable: str,
52 - sla_value: int | None,
53 -):
54 - if sla_value is None:
55 - prefix = sla_comb.as_text(sep="-")
56 - return base_path / f"SLA--{prefix}.json"
57 -
58 - return base_path / f"{sla_variable}={sla_value}"
59 -
60 -
61 -def _get_sla_run_path(iter_path: Path, run_number: int | None):
62 - if run_number is None:
63 - return iter_path / "summary.json"
64 -
65 - return iter_path / f"run={run_number}.json"
66 -
67 -
68 -def _iter_sla_val_paths(base_path: Path, sla_variable: str):
69 - for iter_path in base_path.glob(f"{sla_variable}=*"):
70 - sla_value = int(iter_path.name.removeprefix(f"{sla_variable}="))
71 - summary_path = iter_path / "summary.json"
72 - if summary_path.exists():
73 - yield sla_value, summary_path
74 -
75 -
76 -def _sla_needs_server(
77 - serve_comb: ParameterSweepItem,
78 - bench_combs: ParameterSweep,
79 - sla_combs: SLASweep,
80 - sla_variable: str,
81 - output_dir: Path,
82 -):
83 - for bench_comb in bench_combs:
84 - base_path = _get_sla_base_path(output_dir, serve_comb, bench_comb)
85 - for sla_comb in sla_combs:
86 - if not _get_sla_iter_path(
87 - base_path,
88 - sla_comb,
89 - sla_variable,
90 - sla_value=None,
91 - ).exists():
92 - return True
93 -
94 - return False
95 -
96 -
97 -def run_sla(
98 - server: ServerProcess | None,
99 - bench_cmd: list[str],
100 - *,
101 - serve_comb: ParameterSweepItem,
102 - bench_comb: ParameterSweepItem,
103 - iter_path: Path,
104 - num_runs: int,
105 - dry_run: bool,
106 -):
107 - iter_data = list[dict[str, object]]()
108 -
109 - for run_number in range(num_runs):
110 - run_data = run_benchmark(
111 - server,
112 - bench_cmd,
113 - serve_overrides=serve_comb,
114 - bench_overrides=bench_comb,
115 - run_number=run_number,
116 - output_path=_get_sla_run_path(iter_path, run_number),
117 - dry_run=dry_run,
118 - )
119 -
120 - if run_data is not None:
121 - iter_data.append(run_data)
122 -
123 - if dry_run:
124 - return None
125 -
126 - with _get_sla_run_path(iter_path, run_number=None).open("w") as f:
127 - json.dump(iter_data, f, indent=4)
128 -
129 - return iter_data
130 -
29
30 SLAVariable = Literal["request_rate", "max_concurrency"]
31
32
135 -class SLAHistory(dict[int, float]):
136 - def __init__(self, min_value: int, max_value: int) -> None:
137 - super().__init__()
138 -
139 - self.min_value = min_value
140 - self.max_value = max_value
141 -
142 - def get_xy(self) -> tuple[list[int], list[float]]:
143 - xs = list[int]()
144 - ys = list[float]()
145 - for x, y in sorted(self.items()):
146 - xs.append(x)
147 - ys.append(y)
33 +def _estimate_sla_value(run_data: dict[str, object], sla_variable: SLAVariable):
34 + request_throughput = float(run_data["request_throughput"]) # type: ignore
35 + if sla_variable == "request_rate":
36 + return request_throughput
37 + if sla_variable == "max_concurrency":
38 + mean_latency_ms = float(run_data["mean_e2el_ms"]) # type: ignore
39 + return request_throughput * mean_latency_ms / 1000
40
149 - return xs, ys
41 + assert_never(sla_variable)
42
151 - def get_max_passing(self) -> float:
152 - return max(
153 - (val for val, margin in self.items() if margin <= 0),
154 - default=self.min_value,
155 - )
43
157 - def get_min_failing(self) -> float:
158 - return min(
159 - (val for val, margin in self.items() if margin > 0),
160 - default=self.max_value,
161 - )
44 +def _estimate_sla_avg(runs: list[dict[str, object]], sla_variable: SLAVariable):
45 + return sum(_estimate_sla_value(run, sla_variable) for run in runs) / len(runs)
46
47
164 -def _compute_margin(
165 - sla_comb: SLASweepItem,
166 - iter_data: list[dict[str, object]],
167 -):
168 - assert iter_data, "Summary should not be empty"
169 -
170 - iter_data_mean = {
171 - k: sum(float(run_data[k]) for run_data in iter_data) / len(iter_data) # type: ignore
172 - for k in sla_comb
173 - }
174 -
175 - sla_margins = [
176 - criterion.print_and_compute_margin(iter_data_mean, k)
177 - for k, criterion in sla_comb.items()
178 - ]
179 -
180 - return max(sla_margins)
181 -
182 -
183 -def solve_sla(
48 +def run_comb_sla(
49 server: ServerProcess | None,
50 bench_cmd: list[str],
51 *,
52 serve_comb: ParameterSweepItem,
53 bench_comb: ParameterSweepItem,
189 - sla_comb: SLASweepItem,
190 - base_path: Path,
54 + output_dir: Path,
55 num_runs: int,
56 dry_run: bool,
57 + link_vars: list[tuple[str, str]],
58 sla_variable: SLAVariable,
194 - sla_min_value: int = 1,
195 - sla_max_value: int = 8192, # The value that represents infinite QPS
196 -):
197 - sla_data = list[dict[str, object]]()
198 - history = SLAHistory(min_value=sla_min_value, max_value=sla_max_value)
199 -
200 - # Use results from previous runs
201 - for past_sla_value, path in _iter_sla_val_paths(base_path, sla_variable):
202 - with path.open("rb") as f:
203 - past_iter_data = json.load(f)
204 -
205 - sla_data.append(past_iter_data)
206 - history[past_sla_value] = _compute_margin(sla_comb, past_iter_data)
207 -
208 - # NOTE: We don't use equality here to be more robust against noisy results
209 - while history.get_max_passing() + 1 < history.get_min_failing():
210 - if max(history, default=sla_min_value) < sla_max_value:
211 - val = sla_max_value
212 - elif min(history, default=sla_max_value) > sla_min_value:
213 - val = sla_min_value
214 - else:
215 - spl = PchipInterpolator(*history.get_xy(), extrapolate=False)
216 - spl_roots = spl.solve()
217 - if len(spl_roots) == 0:
218 - # Fallback to binary search
219 - val = int((history.get_max_passing() + history.get_min_failing()) / 2)
220 - else:
221 - val = int(spl_roots[0])
222 -
223 - if val in history:
224 - # Cover both sides (floor and ceil) of the root to be sure
225 - # that it is indeed the target value
226 - val += 1
227 -
228 - val = max(sla_min_value, min(val, sla_max_value))
229 - print(f"Testing {sla_variable}: {val} req/s")
230 -
231 - iter_data = run_sla(
232 - server,
233 - bench_cmd,
234 - serve_comb=serve_comb,
235 - bench_comb=bench_comb | {sla_variable: val},
236 - iter_path=_get_sla_iter_path(base_path, sla_comb, sla_variable, val),
237 - num_runs=num_runs,
238 - dry_run=dry_run,
239 - )
240 - if iter_data is None:
241 - return None
59 + sla_value: int,
60 +) -> list[dict[str, object]] | None:
61 + bench_comb_sla = bench_comb | {sla_variable: sla_value}
62
243 - margin = _compute_margin(sla_comb, iter_data)
244 - if margin <= 0:
245 - print(f"SLA criteria are met. ({margin=:.2f})")
246 - else:
247 - print(f"SLA criteria are not met. ({margin=:.2f})")
248 -
249 - sla_data.extend(iter_data)
250 - history[val] = margin
251 -
252 - return sla_data, history
63 + return run_comb(
64 + server,
65 + bench_cmd,
66 + serve_comb=serve_comb,
67 + bench_comb=bench_comb_sla,
68 + base_path=_get_comb_base_path(output_dir, serve_comb, bench_comb_sla),
69 + num_runs=num_runs,
70 + dry_run=dry_run,
71 + link_vars=link_vars,
72 + )
73
74
255 -def search_sla(
75 +def explore_sla(
76 server: ServerProcess | None,
77 bench_cmd: list[str],
78 *,
79 serve_comb: ParameterSweepItem,
80 bench_comb: ParameterSweepItem,
261 - sla_comb: SLASweepItem,
81 sla_variable: SLAVariable,
263 - base_path: Path,
82 + sla_iters: int,
83 + output_dir: Path,
84 num_runs: int,
85 dry_run: bool,
86 + link_vars: list[tuple[str, str]],
87 ):
88 print("[SLA START]")
89 print(f"Serve parameters: {serve_comb.as_text() or '(None)'}")
90 print(f"Bench parameters: {bench_comb.as_text() or '(None)'}")
270 - print(f"SLA criteria: {sla_comb.as_text()}")
91 + print(f"Number of SLA iterations: {sla_iters}")
92 +
93 + if sla_iters < 2:
94 + raise ValueError("`sla_iters` should be at least 2")
95
272 - result = solve_sla(
96 + serial_comb_data = run_comb_sla(
97 + server,
98 + bench_cmd,
99 + serve_comb=serve_comb,
100 + bench_comb=bench_comb,
101 + output_dir=output_dir,
102 + num_runs=num_runs,
103 + dry_run=dry_run,
104 + link_vars=link_vars,
105 + sla_variable=sla_variable,
106 + sla_value=1,
107 + )
108 + batch_comb_data = run_comb_sla(
109 server,
110 bench_cmd,
111 serve_comb=serve_comb,
112 bench_comb=bench_comb,
277 - sla_comb=sla_comb,
278 - base_path=base_path,
113 + output_dir=output_dir,
114 num_runs=num_runs,
115 dry_run=dry_run,
116 + link_vars=link_vars,
117 sla_variable=sla_variable,
118 + sla_value=int(bench_comb.get("num_prompts", 1000)), # type: ignore
119 )
283 - if result is None:
284 - assert dry_run
285 - print("Omitting SLA search.")
286 - print("[SLA END]")
120 +
121 + if serial_comb_data is None or batch_comb_data is None:
122 + if dry_run:
123 + print("Omitting intermediate SLA iterations.")
124 + print("[SLA END]")
125 +
126 return
127
289 - sla_data, sla_history = result
290 - sla_value = sla_history.get_max_passing()
291 - print(f"Maximum {sla_variable} for SLA: {sla_value} req/s.")
128 + serial_sla_value = math.ceil(_estimate_sla_avg(serial_comb_data, sla_variable))
129 + print(f"Serial inference: {sla_variable}={serial_sla_value}")
130
293 - with _get_sla_iter_path(
294 - base_path,
295 - sla_comb,
296 - sla_variable,
297 - sla_value=None,
298 - ).open("w") as f:
299 - json.dump(sla_data, f, indent=4)
131 + batch_sla_value = math.floor(_estimate_sla_avg(batch_comb_data, sla_variable))
132 + print(f"Batch inference: {sla_variable}={batch_sla_value}")
133 +
134 + # Avoid duplicated runs for intermediate values if the range between
135 + # `serial_sla_value` and `batch_sla_value` is small
136 + inter_sla_values = np.linspace(serial_sla_value, batch_sla_value, sla_iters)[1:-1]
137 + inter_sla_values = sorted(set(map(round, inter_sla_values)))
138 +
139 + inter_combs_data: list[dict[str, object]] = []
140 + for inter_sla_value in inter_sla_values:
141 + print(f"Exploring: {sla_variable}={inter_sla_value}")
142 + inter_comb_data = run_comb_sla(
143 + server,
144 + bench_cmd,
145 + serve_comb=serve_comb,
146 + bench_comb=bench_comb,
147 + output_dir=output_dir,
148 + num_runs=num_runs,
149 + dry_run=dry_run,
150 + link_vars=link_vars,
151 + sla_variable=sla_variable,
152 + sla_value=inter_sla_value,
153 + )
154 + if inter_comb_data is not None:
155 + inter_combs_data.extend(inter_comb_data)
156
157 print("[SLA END]")
158
303 - return sla_data
159 + return serial_comb_data + inter_combs_data + batch_comb_data
160
161
162 def run_slas(
@@ -309,13 +165,15 @@ def run_slas(
165 after_bench_cmd: list[str],
166 *,
167 show_stdout: bool,
168 + server_ready_timeout: int,
169 serve_params: ParameterSweep,
170 bench_params: ParameterSweep,
314 - sla_params: SLASweep,
171 sla_variable: SLAVariable,
172 + sla_iters: int,
173 output_dir: Path,
174 num_runs: int,
175 dry_run: bool,
176 + link_vars: list[tuple[str, str]],
177 ):
178 if any(bench_comb.has_param(sla_variable) for bench_comb in bench_params):
179 raise ValueError(
@@ -325,41 +183,32 @@ def run_slas(
183
184 all_data = list[dict[str, object]]()
185 for serve_comb in serve_params:
328 - with (
329 - run_server(
330 - serve_cmd,
331 - after_bench_cmd,
332 - show_stdout=show_stdout,
333 - serve_overrides=serve_comb,
334 - dry_run=dry_run,
335 - )
336 - if _sla_needs_server(
337 - serve_comb,
338 - bench_params,
339 - sla_params,
340 - sla_variable,
341 - output_dir,
342 - )
343 - else contextlib.nullcontext()
186 + with server_ctx(
187 + serve_cmd,
188 + after_bench_cmd,
189 + show_stdout=show_stdout,
190 + server_ready_timeout=server_ready_timeout,
191 + serve_comb=serve_comb,
192 + bench_params=bench_params,
193 + output_dir=output_dir,
194 + dry_run=dry_run,
195 ) as server:
196 for bench_comb in bench_params:
346 - for sla_comb in sla_params:
347 - base_path = _get_sla_base_path(output_dir, serve_comb, bench_comb)
348 -
349 - comb_data = search_sla(
350 - server,
351 - bench_cmd,
352 - serve_comb=serve_comb,
353 - bench_comb=bench_comb,
354 - sla_comb=sla_comb,
355 - sla_variable=sla_variable,
356 - base_path=base_path,
357 - num_runs=num_runs,
358 - dry_run=dry_run,
359 - )
360 -
361 - if comb_data is not None:
362 - all_data.extend(comb_data)
197 + comb_data = explore_sla(
198 + server,
199 + bench_cmd,
200 + serve_comb=serve_comb,
201 + bench_comb=bench_comb,
202 + sla_variable=sla_variable,
203 + sla_iters=sla_iters,
204 + output_dir=output_dir,
205 + num_runs=num_runs,
206 + dry_run=dry_run,
207 + link_vars=link_vars,
208 + )
209 +
210 + if comb_data is not None:
211 + all_data.extend(comb_data)
212
213 if dry_run:
214 return None
@@ -372,26 +221,23 @@ def run_slas(
221
222 @dataclass
223 class SweepServeSLAArgs(SweepServeArgs):
375 - sla_params: SLASweep
224 sla_variable: SLAVariable
225 + sla_iters: int
226
227 parser_name: ClassVar[str] = "serve_sla"
379 - parser_help: ClassVar[str] = "Tune a variable to meet SLAs under multiple settings."
228 + parser_help: ClassVar[str] = (
229 + "Explore the latency-throughput space for determining SLAs."
230 + )
231
232 @classmethod
233 def from_cli_args(cls, args: argparse.Namespace):
234 # NOTE: Don't use super() as `from_cli_args` calls `cls()`
235 base_args = SweepServeArgs.from_cli_args(args)
236
386 - if args.sla_params:
387 - sla_params = SLASweep.read_json(args.sla_params)
388 - else:
389 - sla_params = SLASweep.from_records([])
390 -
237 return cls(
238 **asdict(base_args),
393 - sla_params=sla_params,
239 sla_variable=args.sla_variable,
240 + sla_iters=args.sla_iters,
241 )
242
243 @classmethod
@@ -399,25 +245,20 @@ class SweepServeSLAArgs(SweepServeArgs):
245 parser = super().add_cli_args(parser)
246
247 sla_group = parser.add_argument_group("sla options")
402 - sla_group.add_argument(
403 - "--sla-params",
404 - type=str,
405 - required=True,
406 - help="Path to JSON file containing a list of SLA constraints to satisfy. "
407 - 'Each constraint is expressed in `{"<KEY>": "<OP><VALUE>"}` format, '
408 - 'e.g.: `{"p99_e2el_ms": "<=500"}` means that '
409 - "the E2E latency should be less than 500ms 99%% of the time. "
410 - "Setting this option runs this script in SLA mode, which searches for "
411 - "the maximum `sla_variable` that satisfies the constraints for "
412 - "each combination of `serve_params`, `bench_params`, and `sla_params`.",
413 - )
248 sla_group.add_argument(
249 "--sla-variable",
250 type=str,
251 choices=get_args(SLAVariable),
252 default="request_rate",
419 - help="Whether to tune request rate or maximum concurrency to satisfy "
420 - "the SLA constraints.",
253 + help="The variable to adjust in each iteration.",
254 + )
255 + sla_group.add_argument(
256 + "--sla-iters",
257 + type=int,
258 + default=10,
259 + help="Number of iterations used to explore the latency-throughput space. "
260 + "This includes the first two iterations used to interpolate the value of "
261 + "`sla_variable` for remaining iterations.",
262 )
263
264 return parser
@@ -436,13 +277,15 @@ def run_main(args: SweepServeSLAArgs):
277 bench_cmd=args.bench_cmd,
278 after_bench_cmd=args.after_bench_cmd,
279 show_stdout=args.show_stdout,
280 + server_ready_timeout=args.server_ready_timeout,
281 serve_params=args.serve_params,
282 bench_params=args.bench_params,
441 - sla_params=args.sla_params,
283 sla_variable=args.sla_variable,
284 + sla_iters=args.sla_iters,
285 output_dir=output_dir,
286 num_runs=args.num_runs,
287 dry_run=args.dry_run,
288 + link_vars=args.link_vars,
289 )
290 except BaseException as exc:
291 raise RuntimeError(
vllm/benchmarks/sweep/sla_sweep.py deleted
-138
@@ -1,138 +0,0 @@
1 -# SPDX-License-Identifier: Apache-2.0
2 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 -import json
4 -import os
5 -from abc import ABC, abstractmethod
6 -from dataclasses import dataclass
7 -
8 -from typing_extensions import override
9 -
10 -SLA_EPS = 1e-8
11 -"""Offset used to differentiate margins for equality checks."""
12 -
13 -
14 -@dataclass
15 -class SLACriterionBase(ABC):
16 - target: float
17 -
18 - @abstractmethod
19 - def compute_margin(self, actual: float) -> float:
20 - """
21 - Return a negative value or `0` if this criterion is met;
22 - otherwise a positive value indicating the distance to the target.
23 - """
24 - raise NotImplementedError
25 -
26 - @abstractmethod
27 - def format_cond(self, lhs: str) -> str:
28 - raise NotImplementedError
29 -
30 - def print_and_compute_margin(
31 - self,
32 - metrics: dict[str, float],
33 - metrics_key: str,
34 - ) -> float:
35 - metric = metrics[metrics_key]
36 - margin = self.compute_margin(metric)
37 -
38 - cond = self.format_cond(f"{metrics_key} = {metric:.2f}")
39 - print(f"Validating SLA: {cond} | " + ("PASSED" if margin <= 0 else "FAILED"))
40 -
41 - return margin
42 -
43 -
44 -@dataclass
45 -class SLALessThan(SLACriterionBase):
46 - @override
47 - def compute_margin(self, actual: float) -> float:
48 - return actual + SLA_EPS - self.target
49 -
50 - @override
51 - def format_cond(self, lhs: str) -> str:
52 - return f"{lhs}<{self.target:.2f}"
53 -
54 -
55 -@dataclass
56 -class SLALessThanOrEqualTo(SLACriterionBase):
57 - @override
58 - def compute_margin(self, actual: float) -> float:
59 - return actual - self.target
60 -
61 - @override
62 - def format_cond(self, lhs: str) -> str:
63 - return f"{lhs}<={self.target:.2f}"
64 -
65 -
66 -@dataclass
67 -class SLAGreaterThan(SLACriterionBase):
68 - @override
69 - def compute_margin(self, actual: float) -> float:
70 - return self.target + SLA_EPS - actual
71 -
72 - @override
73 - def format_cond(self, lhs: str) -> str:
74 - return f"{lhs}>{self.target:.2f}"
75 -
76 -
77 -@dataclass
78 -class SLAGreaterThanOrEqualTo(SLACriterionBase):
79 - @override
80 - def compute_margin(self, actual: float) -> float:
81 - return self.target - actual
82 -
83 - @override
84 - def format_cond(self, lhs: str) -> str:
85 - return f"{lhs}>={self.target:.2f}"
86 -
87 -
88 -# NOTE: The ordering is important! Match longer op_keys first
89 -SLA_CRITERIA: dict[str, type[SLACriterionBase]] = {
90 - "<=": SLALessThanOrEqualTo,
91 - ">=": SLAGreaterThanOrEqualTo,
92 - "<": SLALessThan,
93 - ">": SLAGreaterThan,
94 -}
95 -
96 -
97 -class SLASweep(list["SLASweepItem"]):
98 - @classmethod
99 - def read_json(cls, filepath: os.PathLike):
100 - with open(filepath, "rb") as f:
101 - records = json.load(f)
102 -
103 - return cls.from_records(records)
104 -
105 - @classmethod
106 - def from_records(cls, records: list[dict[str, str]]):
107 - if not isinstance(records, list):
108 - raise TypeError(
109 - f"The SLA sweep should be a list of dictionaries, "
110 - f"but found type: {type(records)}"
111 - )
112 -
113 - return cls(SLASweepItem.from_record(record) for record in records)
114 -
115 -
116 -class SLASweepItem(dict[str, SLACriterionBase]):
117 - @classmethod
118 - def from_record(cls, record: dict[str, str]):
119 - sla_criteria: dict[str, SLACriterionBase] = {}
120 -
121 - for metric_key, metric_value in record.items():
122 - for op_key in SLA_CRITERIA:
123 - if metric_value.startswith(op_key):
124 - sla_criteria[metric_key] = SLA_CRITERIA[op_key](
125 - float(metric_value.removeprefix(op_key))
126 - )
127 - break
128 - else:
129 - raise ValueError(
130 - f"Invalid operator for "
131 - f"SLA constraint '{metric_key}={metric_value}'. "
132 - f"Valid operators are: {sorted(SLA_CRITERIA)}",
133 - )
134 -
135 - return cls(sla_criteria)
136 -
137 - def as_text(self, sep: str = ", ") -> str:
138 - return sep.join(v.format_cond(k) for k, v in self.items())
vllm/benchmarks/sweep/startup.py
+2 -1
@@ -151,7 +151,8 @@ def run_benchmark(
151 print(f"Output file: {output_path}")
152
153 if output_path.exists():
154 - print("Found existing results. Skipping.")
154 + print("Found existing results.")
155 + print("[SKIPPED BENCHMARK]")
156
157 with output_path.open("r", encoding="utf-8") as f:
158 run_data = json.load(f)