A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 7, 12:45 UTC
[Benchmark] Rename SLA Finder to Workload Explorer (#35586)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Feb 28, 2026 at 15:31 UTC
24d6ea8afdb13ceee95b36645ba61a641f9a2f7f
6 files changed
+125
-107
docs/benchmarking/sweeps.md
+17
-14
@@ -102,36 +102,39 @@ By default, each parameter combination is benchmarked 3 times to make the result
102
!!! tip
103
You can use the `--resume` option to continue the parameter sweep if an unexpected error occurs, e.g., timeout when connecting to HF Hub.
104
105
-### SLA Scanner
105
+### Workload Explorer
106
107
-`vllm bench sweep serve_sla` is a variant of `vllm bench sweep serve` that scans through values of request rate or concurrency (choose using `--sla-variable`) in order to find the tradeoff between latency and throughput. The results can then be [visualized](#visualization) to determine the feasible SLAs.
107
+`vllm bench sweep serve_workload` is a variant of `vllm bench sweep serve` that explores different workload levels in order to find the tradeoff between latency and throughput. The results can also be [visualized](#visualization) to determine the feasible SLAs.
108
+
109
+The workload can be expressed in terms of request rate or concurrency (choose using `--workload-var`).
110
111
Example command:
112
113
```bash
112
-vllm bench sweep serve_sla \
114
+vllm bench sweep serve_workload \
115
--serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \
116
--bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 100' \
115
- --sla-variable max_concurrency \
117
+ --workload-var max_concurrency \
118
--serve-params benchmarks/serve_hparams.json \
117
- --bench-params benchmarks/bench_hparams.json
119
+ --bench-params benchmarks/bench_hparams.json \
120
+ --num-runs 1 \
121
-o benchmarks/results
122
```
123
121
-The algorithm for scanning through different values of `sla_variable` can be summarized as follows:
124
+The algorithm for exploring different workload levels can be summarized as follows:
125
123
-1. Run the benchmark by sending requests one at a time (serial inference). This results in the lowest possible latency and throughput.
124
-2. Run the benchmark by sending all requests at once (batch inference). This results in the highest possible latency and throughput.
125
-3. Estimate the maximum value of `sla_variable` that can be supported by the server without oversaturating it.
126
-4. Run the benchmark over intermediate values of `sla_variable` uniformly using the remaining iterations.
126
+1. Run the benchmark by sending requests one at a time (serial inference, lowest workload). This results in the lowest possible latency and throughput.
127
+2. Run the benchmark by sending all requests at once (batch inference, highest workload). This results in the highest possible latency and throughput.
128
+3. Estimate the value of `workload_var` corresponding to Step 2.
129
+4. Run the benchmark over intermediate values of `workload_var` uniformly using the remaining iterations.
130
128
-You can override the number of iterations in the algorithm by setting `--sla-iters`.
131
+You can override the number of iterations in the algorithm by setting `--workload-iters`.
132
133
!!! tip
134
This is our equivalent of [GuideLLM's `--profile sweep`](https://github.com/vllm-project/guidellm/blob/v0.5.3/src/guidellm/benchmark/profiles.py#L575).
135
133
- In general, `--sla-variable max_concurrency` produces more reliable results because it directly controls the workload imposed on the vLLM engine.
134
- Nevertheless, we default to `--sla-variable request_rate` to maintain similar behavior as GuideLLM.
136
+ In general, `--workload-var max_concurrency` produces more reliable results because it directly controls the workload imposed on the vLLM engine.
137
+ Nevertheless, we default to `--workload-var request_rate` to maintain similar behavior as GuideLLM.
138
139
## Startup Benchmark
140
@@ -198,7 +201,7 @@ vllm bench sweep startup \
201
202
Control the variables to plot via `--var-x` and `--var-y`, optionally applying `--filter-by` and `--bin-by` to the values. The plot is organized according to `--fig-by`, `--row-by`, `--col-by`, and `--curve-by`.
203
201
-Example commands for visualizing [SLA Scanner](#sla-scanner) results:
204
+Example commands for visualizing [Workload Explorer](#workload-explorer) results:
205
206
```bash
207
# Name of the directory that stores the results
docs/cli/bench/sweep/serve_sla.md
deleted
-9
@@ -1,9 +0,0 @@
1
-# vllm bench sweep serve_sla
2
-
3
-## JSON CLI Arguments
4
-
5
---8<-- "docs/cli/json_tip.inc.md"
6
-
7
-## Arguments
8
-
9
---8<-- "docs/generated/argparse/bench_sweep_serve_sla.inc.md"
docs/cli/bench/sweep/serve_workload.md
new
+9
@@ -0,0 +1,9 @@
1
+# vllm bench sweep serve_workload
2
+
3
+## JSON CLI Arguments
4
+
5
+--8<-- "docs/cli/json_tip.inc.md"
6
+
7
+## Arguments
8
+
9
+--8<-- "docs/generated/argparse/bench_sweep_serve_workload.inc.md"
docs/mkdocs/hooks/generate_argparse.py
+5
-3
@@ -100,8 +100,8 @@ bench_sweep_plot_pareto = auto_mock(
100
"vllm.benchmarks.sweep.plot_pareto", "SweepPlotParetoArgs"
101
)
102
bench_sweep_serve = auto_mock("vllm.benchmarks.sweep.serve", "SweepServeArgs")
103
-bench_sweep_serve_sla = auto_mock(
104
- "vllm.benchmarks.sweep.serve_sla", "SweepServeSLAArgs"
103
+bench_sweep_serve_workload = auto_mock(
104
+ "vllm.benchmarks.sweep.serve_workload", "SweepServeWorkloadArgs"
105
)
106
bench_throughput = auto_mock("vllm.benchmarks", "throughput")
107
AsyncEngineArgs = auto_mock("vllm.engine.arg_utils", "AsyncEngineArgs")
@@ -229,7 +229,9 @@ def on_startup(command: Literal["build", "gh-deploy", "serve"], dirty: bool):
229
"bench_sweep_plot": create_parser(bench_sweep_plot.add_cli_args),
230
"bench_sweep_plot_pareto": create_parser(bench_sweep_plot_pareto.add_cli_args),
231
"bench_sweep_serve": create_parser(bench_sweep_serve.add_cli_args),
232
- "bench_sweep_serve_sla": create_parser(bench_sweep_serve_sla.add_cli_args),
232
+ "bench_sweep_serve_workload": create_parser(
233
+ bench_sweep_serve_workload.add_cli_args
234
+ ),
235
"bench_throughput": create_parser(bench_throughput.add_cli_args),
236
}
237
vllm/benchmarks/sweep/cli.py
+3
-3
@@ -10,14 +10,14 @@ from .plot_pareto import SweepPlotParetoArgs
10
from .plot_pareto import main as plot_pareto_main
11
from .serve import SweepServeArgs
12
from .serve import main as serve_main
13
-from .serve_sla import SweepServeSLAArgs
14
-from .serve_sla import main as serve_sla_main
13
+from .serve_workload import SweepServeWorkloadArgs
14
+from .serve_workload import main as serve_workload_main
15
from .startup import SweepStartupArgs
16
from .startup import main as startup_main
17
18
SUBCOMMANDS = (
19
(SweepServeArgs, serve_main),
20
- (SweepServeSLAArgs, serve_sla_main),
20
+ (SweepServeWorkloadArgs, serve_workload_main),
21
(SweepStartupArgs, startup_main),
22
(SweepPlotArgs, plot_main),
23
(SweepPlotParetoArgs, plot_pareto_main),
vllm/benchmarks/sweep/serve_workload.py
renamed
+91
-78
@@ -28,25 +28,32 @@ except ImportError:
28
pd = PlaceholderModule("pandas")
29
30
31
-SLAVariable = Literal["request_rate", "max_concurrency"]
31
+WorkloadVariable = Literal["request_rate", "max_concurrency"]
32
33
34
-def _estimate_sla_value(run_data: dict[str, object], sla_variable: SLAVariable):
34
+def _estimate_workload_value(
35
+ run_data: dict[str, object],
36
+ workload_var: WorkloadVariable,
37
+):
38
request_throughput = float(run_data["request_throughput"]) # type: ignore
36
- if sla_variable == "request_rate":
39
+ if workload_var == "request_rate":
40
return request_throughput
38
- if sla_variable == "max_concurrency":
41
+ if workload_var == "max_concurrency":
42
mean_latency_ms = float(run_data["mean_e2el_ms"]) # type: ignore
43
return request_throughput * mean_latency_ms / 1000
44
42
- assert_never(sla_variable)
45
+ assert_never(workload_var)
46
47
45
-def _estimate_sla_avg(runs: list[dict[str, object]], sla_variable: SLAVariable):
46
- return sum(_estimate_sla_value(run, sla_variable) for run in runs) / len(runs)
48
+def _estimate_workload_avg(
49
+ runs: list[dict[str, object]],
50
+ workload_var: WorkloadVariable,
51
+):
52
+ total = sum(_estimate_workload_value(run, workload_var) for run in runs)
53
+ return total / len(runs)
54
55
49
-def run_comb_sla(
56
+def run_comb_workload(
57
server: ServerProcess | None,
58
bench_cmd: list[str],
59
*,
@@ -56,21 +63,21 @@ def run_comb_sla(
63
num_runs: int,
64
dry_run: bool,
65
link_vars: list[tuple[str, str]],
59
- sla_variable: SLAVariable,
60
- sla_value: int,
66
+ workload_var: WorkloadVariable,
67
+ workload_value: int,
68
) -> list[dict[str, object]] | None:
62
- bench_comb_sla = bench_comb | {sla_variable: sla_value}
69
+ bench_comb_workload = bench_comb | {workload_var: workload_value}
70
71
return run_comb(
72
server,
73
bench_cmd,
74
serve_comb=serve_comb,
68
- bench_comb=bench_comb_sla,
75
+ bench_comb=bench_comb_workload,
76
base_path=_get_comb_base_path(
77
output_dir,
78
serve_comb,
79
bench_comb,
73
- extra_parts=("SLA-", f"{sla_variable}={sla_value}"),
80
+ extra_parts=("WL-", f"{workload_var}={workload_value}"),
81
),
82
num_runs=num_runs,
83
dry_run=dry_run,
@@ -78,26 +85,26 @@ def run_comb_sla(
85
)
86
87
81
-def explore_sla(
88
+def explore_comb_workloads(
89
server: ServerProcess | None,
90
bench_cmd: list[str],
91
*,
92
serve_comb: ParameterSweepItem,
93
bench_comb: ParameterSweepItem,
87
- sla_variable: SLAVariable,
88
- sla_iters: int,
94
+ workload_var: WorkloadVariable,
95
+ workload_iters: int,
96
output_dir: Path,
97
num_runs: int,
98
dry_run: bool,
99
link_vars: list[tuple[str, str]],
100
):
94
- print("[SLA START]")
101
+ print("[WL START]")
102
print(f"Serve parameters: {serve_comb.as_text() or '(None)'}")
103
print(f"Bench parameters: {bench_comb.as_text() or '(None)'}")
97
- print(f"Number of SLA iterations: {sla_iters}")
104
+ print(f"Number of workload iterations: {workload_iters}")
105
99
- if sla_iters < 2:
100
- raise ValueError("`sla_iters` should be at least 2")
106
+ if workload_iters < 2:
107
+ raise ValueError("`workload_iters` should be at least 2")
108
109
dataset_size = DEFAULT_NUM_PROMPTS
110
if "num_prompts" in bench_comb:
@@ -113,7 +120,7 @@ def explore_sla(
120
121
print(f"Dataset size: {dataset_size}")
122
116
- serial_comb_data = run_comb_sla(
123
+ serial_workload_data = run_comb_workload(
124
server,
125
bench_cmd,
126
serve_comb=serve_comb,
@@ -122,10 +129,10 @@ def explore_sla(
129
num_runs=num_runs,
130
dry_run=dry_run,
131
link_vars=link_vars,
125
- sla_variable=sla_variable,
126
- sla_value=1,
132
+ workload_var=workload_var,
133
+ workload_value=1,
134
)
128
- batch_comb_data = run_comb_sla(
135
+ batch_workload_data = run_comb_workload(
136
server,
137
bench_cmd,
138
serve_comb=serve_comb,
@@ -134,32 +141,38 @@ def explore_sla(
141
num_runs=num_runs,
142
dry_run=dry_run,
143
link_vars=link_vars,
137
- sla_variable=sla_variable,
138
- sla_value=dataset_size,
144
+ workload_var=workload_var,
145
+ workload_value=dataset_size,
146
)
147
141
- if serial_comb_data is None or batch_comb_data is None:
148
+ if serial_workload_data is None or batch_workload_data is None:
149
if dry_run:
143
- print("Omitting intermediate SLA iterations.")
144
- print("[SLA END]")
150
+ print("Omitting intermediate Workload iterations.")
151
+ print("[WL END]")
152
153
return
154
148
- serial_sla_value = math.ceil(_estimate_sla_avg(serial_comb_data, sla_variable))
149
- print(f"Serial inference: {sla_variable}={serial_sla_value}")
155
+ serial_workload_value = math.ceil(
156
+ _estimate_workload_avg(serial_workload_data, workload_var)
157
+ )
158
+ print(f"Serial inference: {workload_var}={serial_workload_value}")
159
151
- batch_sla_value = math.floor(_estimate_sla_avg(batch_comb_data, sla_variable))
152
- print(f"Batch inference: {sla_variable}={batch_sla_value}")
160
+ batch_workload_value = math.floor(
161
+ _estimate_workload_avg(batch_workload_data, workload_var)
162
+ )
163
+ print(f"Batch inference: {workload_var}={batch_workload_value}")
164
165
# Avoid duplicated runs for intermediate values if the range between
155
- # `serial_sla_value` and `batch_sla_value` is small
156
- inter_sla_values = np.linspace(serial_sla_value, batch_sla_value, sla_iters)[1:-1]
157
- inter_sla_values = sorted(set(map(round, inter_sla_values)))
158
-
159
- inter_combs_data: list[dict[str, object]] = []
160
- for inter_sla_value in inter_sla_values:
161
- print(f"Exploring: {sla_variable}={inter_sla_value}")
162
- inter_comb_data = run_comb_sla(
166
+ # `serial_workload_value` and `batch_workload_value` is small
167
+ inter_workload_values = np.linspace(
168
+ serial_workload_value, batch_workload_value, workload_iters
169
+ )[1:-1]
170
+ inter_workload_values = sorted(set(map(round, inter_workload_values)))
171
+
172
+ inter_workloads_data: list[dict[str, object]] = []
173
+ for inter_workload_value in inter_workload_values:
174
+ print(f"Exploring: {workload_var}={inter_workload_value}")
175
+ inter_workload_data = run_comb_workload(
176
server,
177
bench_cmd,
178
serve_comb=serve_comb,
@@ -168,18 +181,18 @@ def explore_sla(
181
num_runs=num_runs,
182
dry_run=dry_run,
183
link_vars=link_vars,
171
- sla_variable=sla_variable,
172
- sla_value=inter_sla_value,
184
+ workload_var=workload_var,
185
+ workload_value=inter_workload_value,
186
)
174
- if inter_comb_data is not None:
175
- inter_combs_data.extend(inter_comb_data)
187
+ if inter_workload_data is not None:
188
+ inter_workloads_data.extend(inter_workload_data)
189
177
- print("[SLA END]")
190
+ print("[WL END]")
191
179
- return serial_comb_data + inter_combs_data + batch_comb_data
192
+ return serial_workload_data + inter_workloads_data + batch_workload_data
193
194
182
-def run_slas(
195
+def explore_combs_workloads(
196
serve_cmd: list[str],
197
bench_cmd: list[str],
198
after_bench_cmd: list[str],
@@ -188,17 +201,17 @@ def run_slas(
201
server_ready_timeout: int,
202
serve_params: ParameterSweep,
203
bench_params: ParameterSweep,
191
- sla_variable: SLAVariable,
192
- sla_iters: int,
204
+ workload_var: WorkloadVariable,
205
+ workload_iters: int,
206
output_dir: Path,
207
num_runs: int,
208
dry_run: bool,
209
link_vars: list[tuple[str, str]],
210
):
198
- if any(bench_comb.has_param(sla_variable) for bench_comb in bench_params):
211
+ if any(bench_comb.has_param(workload_var) for bench_comb in bench_params):
212
raise ValueError(
200
- f"You should not override `{sla_variable}` in `bench_params` in SLA mode, "
201
- "since it is supposed to be determined automatically."
213
+ f"You should not override `{workload_var}` in `bench_params` "
214
+ "since it is supposed to be explored automatically."
215
)
216
217
all_data = list[dict[str, object]]()
@@ -214,13 +227,13 @@ def run_slas(
227
dry_run=dry_run,
228
) as server:
229
for bench_comb in bench_params:
217
- comb_data = explore_sla(
230
+ comb_data = explore_comb_workloads(
231
server,
232
bench_cmd,
233
serve_comb=serve_comb,
234
bench_comb=bench_comb,
222
- sla_variable=sla_variable,
223
- sla_iters=sla_iters,
235
+ workload_var=workload_var,
236
+ workload_iters=workload_iters,
237
output_dir=output_dir,
238
num_runs=num_runs,
239
dry_run=dry_run,
@@ -240,13 +253,13 @@ def run_slas(
253
254
255
@dataclass
243
-class SweepServeSLAArgs(SweepServeArgs):
244
- sla_variable: SLAVariable
245
- sla_iters: int
256
+class SweepServeWorkloadArgs(SweepServeArgs):
257
+ workload_var: WorkloadVariable
258
+ workload_iters: int
259
247
- parser_name: ClassVar[str] = "serve_sla"
260
+ parser_name: ClassVar[str] = "serve_workload"
261
parser_help: ClassVar[str] = (
249
- "Explore the latency-throughput space for determining SLAs."
262
+ "Explore the latency-throughput tradeoff for different workload levels."
263
)
264
265
@classmethod
@@ -256,35 +269,35 @@ class SweepServeSLAArgs(SweepServeArgs):
269
270
return cls(
271
**asdict(base_args),
259
- sla_variable=args.sla_variable,
260
- sla_iters=args.sla_iters,
272
+ workload_var=args.workload_var,
273
+ workload_iters=args.workload_iters,
274
)
275
276
@classmethod
277
def add_cli_args(cls, parser: argparse.ArgumentParser) -> argparse.ArgumentParser:
278
parser = super().add_cli_args(parser)
279
267
- sla_group = parser.add_argument_group("sla options")
268
- sla_group.add_argument(
269
- "--sla-variable",
280
+ workload_group = parser.add_argument_group("workload options")
281
+ workload_group.add_argument(
282
+ "--workload-var",
283
type=str,
271
- choices=get_args(SLAVariable),
284
+ choices=get_args(WorkloadVariable),
285
default="request_rate",
286
help="The variable to adjust in each iteration.",
287
)
275
- sla_group.add_argument(
276
- "--sla-iters",
288
+ workload_group.add_argument(
289
+ "--workload-iters",
290
type=int,
291
default=10,
279
- help="Number of iterations used to explore the latency-throughput space. "
292
+ help="Number of workload levels to explore. "
293
"This includes the first two iterations used to interpolate the value of "
281
- "`sla_variable` for remaining iterations.",
294
+ "`workload_var` for remaining iterations.",
295
)
296
297
return parser
298
299
287
-def run_main(args: SweepServeSLAArgs):
300
+def run_main(args: SweepServeWorkloadArgs):
301
timestamp = args.resume or datetime.now().strftime("%Y%m%d_%H%M%S")
302
output_dir = args.output_dir / timestamp
303
@@ -292,7 +305,7 @@ def run_main(args: SweepServeSLAArgs):
305
raise ValueError(f"Cannot resume from non-existent directory ({output_dir})")
306
307
try:
295
- return run_slas(
308
+ return explore_combs_workloads(
309
serve_cmd=args.serve_cmd,
310
bench_cmd=args.bench_cmd,
311
after_bench_cmd=args.after_bench_cmd,
@@ -300,8 +313,8 @@ def run_main(args: SweepServeSLAArgs):
313
server_ready_timeout=args.server_ready_timeout,
314
serve_params=args.serve_params,
315
bench_params=args.bench_params,
303
- sla_variable=args.sla_variable,
304
- sla_iters=args.sla_iters,
316
+ workload_var=args.workload_var,
317
+ workload_iters=args.workload_iters,
318
output_dir=output_dir,
319
num_runs=args.num_runs,
320
dry_run=args.dry_run,
@@ -315,11 +328,11 @@ def run_main(args: SweepServeSLAArgs):
328
329
330
def main(args: argparse.Namespace):
318
- run_main(SweepServeSLAArgs.from_cli_args(args))
331
+ run_main(SweepServeWorkloadArgs.from_cli_args(args))
332
333
334
if __name__ == "__main__":
322
- parser = argparse.ArgumentParser(description=SweepServeSLAArgs.parser_help)
323
- SweepServeSLAArgs.add_cli_args(parser)
335
+ parser = argparse.ArgumentParser(description=SweepServeWorkloadArgs.parser_help)
336
+ SweepServeWorkloadArgs.add_cli_args(parser)
337
338
main(parser.parse_args())