@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 879 lines 41 KB
Rendered Raw
1 # vllm-bench
2
3 High-performance Rust benchmark client for vLLM serving endpoints. A drop-in replacement for `vllm bench serve` with near-instant startup, parallel dataset generation, and a fraction of the memory overhead — and no Python at runtime.
4
5 ```bash
6 vllm-bench --backend vllm --base-url http://127.0.0.1:8000 \
7 --model <model> --dataset-name random \
8 --random-input-len 1024 --random-output-len 128 \
9 --num-prompts 1000 --max-concurrency 200
10 ```
11
12 ## Highlights
13
14 - **Fast** — ~7 ms startup, single ~7 MB static binary, no Python imports.
15 - **Scales** — `Arc<str>` prompt sharing + mimalloc keep memory <100 MB at 1400+ concurrency.
16 - **Many datasets** — `random`, `random-mm` (VLM), `sharegpt`, `sonnet`, `speed-bench`, and any HuggingFace dataset.
17 - **Many backends** — completions, chat, embeddings, pooling, and rerank.
18 - **Beyond a single run** — concurrency/rate **sweeps**, **multi-run** stats, **multi-turn** conversations, **LoRA** multi-adapter, and result **comparison**.
19 - **Steady-state metrics** — throughput/latency measured over the saturated plateau, excluding ramp-up and drain.
20 - **Parity** — JSON output schema and timing semantics match Python `vllm bench serve` exactly.
21 - **Multimodal preprocessing** — `mm-processor` subcommand measures per-stage VLM preprocessing latency against a managed headless engine (mirrors `vllm bench mm-processor`).
22
23 ### Performance vs. Python
24
25 | Metric | Python | Rust |
26 | -------- | -------- | ------ |
27 | Startup time | Multi-second (import vllm + numpy + aiohttp) | ~7 ms |
28 | 100k random prompts (input_len=8192) | Minutes | Seconds (rayon parallelism) |
29 | Binary size | — | ~7 MB |
30 | Peak memory at 1400 concurrency | High (GIL + per-object overhead) | <100 MB (`Arc<str>` prompt sharing) |
31
32 ## Contents
33
34 - [Install](#install)
35 - [Quick Start](#quick-start)
36 - [Usage Examples](#usage-examples)
37 - [Supported Backends](#supported-backends)
38 - [Supported Datasets](#supported-datasets)
39 - [Metrics](#metrics)
40 - [CLI Reference](#cli-reference)
41 - [Tokenizer Support](#tokenizer-support)
42 - [Output Format](#output-format)
43 - [Architecture](#architecture)
44 - [Environment Variables](#environment-variables)
45
46 ## Install
47
48 ### Prebuilt binaries (Linux)
49
50 ```bash
51 curl -fsSL https://github.com/vllm-project/vllm-bench/releases/latest/download/vllm-bench-$(uname -m)-linux-musl -o vllm-bench && chmod +x vllm-bench
52 ```
53
54 ### With Cargo
55
56 Install straight from the repository (builds from source; requires [Rust](https://rustup.rs/) stable and a C compiler for the native tokenizer dependency):
57
58 ```bash
59 cargo install --git https://github.com/vllm-project/vllm-bench vllm-bench
60 ```
61
62 The trailing `vllm-bench` selects the package — the repo also ships a `mock-llm-server` binary, so omitting it fails with `multiple packages with binaries found`. The binary is installed to `~/.cargo/bin/`.
63
64 ### Build from source
65
66 Requires [Rust](https://rustup.rs/) (stable).
67
68 ```bash
69 git clone https://github.com/vllm-project/vllm-bench.git
70 cd vllm-bench
71 ./install.sh # builds release and installs to ~/.local/bin
72 # or: ./install.sh --to ~/bin
73 ```
74
75 ## Quick Start
76
77 Point it at a running vLLM server and benchmark with synthetic prompts:
78
79 ```bash
80 vllm-bench \
81 --backend vllm \
82 --base-url http://127.0.0.1:8000 \
83 --model <model-name> \
84 --dataset-name random \
85 --random-input-len 1024 \
86 --random-output-len 128 \
87 --num-prompts 1000 \
88 --max-concurrency 200
89 ```
90
91 > **Tip:** prefer `127.0.0.1` over `localhost` — some systems resolve `localhost` to IPv6 `::1` while vLLM listens on IPv4 only.
92
93 Add `--save-result` to write a JSON file, or `--dry-run` to generate and inspect the dataset without sending any requests.
94
95 ## Usage Examples
96
97 <details open>
98 <summary><b>Generation (completions / chat)</b></summary>
99
100 ```bash
101 # Full production-style run with percentile metrics and result file
102 vllm-bench \
103 --backend vllm \
104 --base-url http://127.0.0.1:8000 \
105 --model nvidia/Kimi-K2.5-NVFP4 \
106 --dataset-name random \
107 --random-input-len 8192 \
108 --random-output-len 1024 \
109 --ignore-eos \
110 --num-prompts 4096 \
111 --percentile-metrics "ttft,tpot,itl,e2el" \
112 --save-result \
113 --max-concurrency 1400
114
115 # Send token IDs instead of text (pure vLLM: skips server-side tokenization,
116 # exact token counts, faster). Random dataset only.
117 vllm-bench \
118 --backend vllm \
119 --base-url http://127.0.0.1:8000 \
120 --model <model-name> \
121 --dataset-name random \
122 --random-input-len 1024 \
123 --prompt-token-ids \
124 --num-prompts 1000
125 ```
126
127 </details>
128
129 <details>
130 <summary><b>Datasets (ShareGPT / Sonnet / HuggingFace / SPEED-Bench)</b></summary>
131
132 ```bash
133 # ShareGPT (auto-downloads from HuggingFace on first run, cached afterwards)
134 vllm-bench \
135 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
136 --dataset-name sharegpt --num-prompts 500 --save-result
137
138 # ShareGPT with an explicit local file
139 vllm-bench \
140 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
141 --dataset-name sharegpt --dataset-path /path/to/ShareGPT_V3.json \
142 --num-prompts 500 --save-result
143
144 # Sonnet — built-in Shakespeare sonnets, no dataset file needed.
145 # Generates prompts of a controllable token length with a shared prefix.
146 vllm-bench \
147 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
148 --dataset-name sonnet \
149 --sonnet-input-len 550 --sonnet-output-len 150 --sonnet-prefix-len 200 \
150 --num-prompts 500
151
152 # Any HuggingFace dataset (downloads parquet shards into the standard HF hub
153 # cache — shared with `hf download` and Python tooling — auto-detects columns)
154 vllm-bench \
155 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
156 --dataset-name hf --dataset-path allenai/WildChat-4.8M \
157 --hf-split train --num-prompts 1000 --save-result
158
159 # HuggingFace dataset with subset + fixed output length (LongBench)
160 vllm-bench \
161 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
162 --dataset-name hf --dataset-path THUDM/LongBench \
163 --hf-subset narrativeqa --hf-split test --hf-output-len 512 --num-prompts 200
164
165 # Gated HuggingFace dataset (auth via HF_TOKEN or the `hf auth login` token file).
166 # Private datasets work too when the repo stores native parquet/json/jsonl files.
167 HF_TOKEN=hf_xxx vllm-bench \
168 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
169 --dataset-name hf --dataset-path lmsys/lmsys-chat-1m \
170 --hf-split train --hf-output-len 256 --num-prompts 1000
171
172 # SPEED-Bench for speculative decoding evaluation (auto-downloads, cached)
173 vllm-bench \
174 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
175 --dataset-name speed-bench --speed-bench-config qualitative \
176 --num-prompts 200 --output-len 256 --save-result
177
178 # SPEED-Bench throughput split with entropy category filter + input truncation
179 vllm-bench \
180 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
181 --dataset-name speed-bench --speed-bench-config throughput_16k \
182 --speed-bench-max-input-len 10240 --speed-bench-category low_entropy \
183 --num-prompts 500 --output-len 256 --max-concurrency 200 --save-result
184 ```
185
186 </details>
187
188 <details>
189 <summary><b>Multimodal (VLM with synthetic images)</b></summary>
190
191 ```bash
192 # One synthetic image per request
193 vllm-bench \
194 --backend openai-chat --base-url http://127.0.0.1:8000 \
195 --model Qwen/Qwen2.5-VL-7B-Instruct \
196 --dataset-name random-mm \
197 --random-input-len 512 --random-output-len 128 --num-prompts 100 \
198 --random-mm-base-items-per-request 1 \
199 --random-mm-limit-mm-per-prompt '{"image": 1, "video": 0}' \
200 --random-mm-bucket-config '{(1024, 800, 1): 1.0}'
201
202 # Multiple images per request, mixed resolutions
203 vllm-bench \
204 --backend openai-chat --base-url http://127.0.0.1:8000 \
205 --model Qwen/Qwen2.5-VL-7B-Instruct \
206 --dataset-name random-mm \
207 --random-input-len 256 --random-output-len 128 --num-prompts 50 \
208 --random-mm-base-items-per-request 3 \
209 --random-mm-limit-mm-per-prompt '{"image": 5, "video": 0}' \
210 --random-mm-bucket-config '{(256,256,1): 0.5, (720,1280,1): 0.5}'
211 ```
212
213 </details>
214
215 <details>
216 <summary><b>Embedding / Pooling / Rerank</b></summary>
217
218 ```bash
219 # Text embedding
220 vllm-bench \
221 --backend openai-embeddings --base-url http://127.0.0.1:8000 \
222 --model BAAI/bge-large-en-v1.5 \
223 --dataset-name random --random-input-len 512 --num-prompts 1000 \
224 --max-concurrency 200 --save-result
225
226 # Chat-format embedding (supports multimodal content)
227 vllm-bench \
228 --backend openai-embeddings-chat --base-url http://127.0.0.1:8000 \
229 --model BAAI/bge-large-en-v1.5 \
230 --dataset-name sharegpt --num-prompts 500 --save-result
231
232 # vLLM native pooling endpoint
233 vllm-bench \
234 --backend vllm-pooling --base-url http://127.0.0.1:8000 \
235 --model BAAI/bge-large-en-v1.5 \
236 --dataset-name random --random-input-len 256 --num-prompts 1000 --save-result
237
238 # Rerank (query from dataset, documents via --extra-body)
239 vllm-bench \
240 --backend vllm-rerank --base-url http://127.0.0.1:8000 \
241 --model BAAI/bge-reranker-v2-m3 \
242 --dataset-name sharegpt --num-prompts 500 \
243 --extra-body '{"documents": ["document to rerank"]}' --save-result
244 ```
245
246 </details>
247
248 <details>
249 <summary><b>Rate control, ramp-up &amp; goodput</b></summary>
250
251 ```bash
252 # Ramp from 10 → 100 RPS with goodput SLO tracking
253 vllm-bench \
254 --backend vllm --base-url http://127.0.0.1:8000 --model <model-name> \
255 --num-prompts 2000 \
256 --ramp-up-strategy linear --ramp-up-start-rps 10 --ramp-up-end-rps 100 \
257 --goodput ttft:200 e2el:5000 \
258 --save-result
259
260 # Fixed Poisson arrival rate at 50 RPS
261 vllm-bench \
262 --backend vllm --base-url http://127.0.0.1:8000 --model <model-name> \
263 --num-prompts 2000 --request-rate 50 --burstiness 1.0
264 ```
265
266 </details>
267
268 <details>
269 <summary><b>Sweep — find the optimal concurrency / rate</b></summary>
270
271 ```bash
272 # Sweep over concurrency values
273 vllm-bench \
274 --backend vllm --base-url http://127.0.0.1:8000 --model <model-name> \
275 --num-prompts 500 \
276 --sweep-max-concurrency 1,10,50,100,200,500,1000
277
278 # Sweep over request rates
279 vllm-bench \
280 --backend vllm --base-url http://127.0.0.1:8000 --model <model-name> \
281 --num-prompts 500 \
282 --sweep-request-rate 1,10,50,100,inf
283
284 # Scale work with concurrency and reset the prefix cache between points
285 # (--sweep-num-prompts-factor sets num_prompts = concurrency * factor;
286 # --reset-prefix-cache requires VLLM_SERVER_DEV_MODE=1 on the server)
287 vllm-bench \
288 --backend vllm --base-url http://127.0.0.1:8000 --model <model-name> \
289 --sweep-max-concurrency 1,10,50,100 \
290 --sweep-num-prompts-factor 20 \
291 --reset-prefix-cache
292 ```
293
294 </details>
295
296 <details>
297 <summary><b>Multi-run &amp; comparison</b></summary>
298
299 ```bash
300 # Run 5 times, report mean/std/min/max with coefficient of variation
301 vllm-bench \
302 --backend vllm --base-url http://127.0.0.1:8000 --model <model-name> \
303 --num-prompts 1000 --max-concurrency 200 --num-runs 5
304
305 # Compare two saved result files side-by-side (no server needed)
306 vllm-bench --compare baseline.json optimized.json
307 ```
308
309 </details>
310
311 <details>
312 <summary><b>Multi-turn conversations</b></summary>
313
314 ```bash
315 # Synthetic multi-turn (controllable per-turn token lengths)
316 vllm-bench \
317 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
318 --dataset-name random --multi-turn --multi-turn-num-turns 5 \
319 --random-input-len 512 --random-output-len 256 \
320 --num-prompts 50 --multi-turn-concurrency 10 \
321 --percentile-metrics "ttft,tpot,itl,e2el" --save-result
322
323 # Variable turn count per conversation + per-turn input length for turns 1+
324 vllm-bench \
325 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
326 --dataset-name random --multi-turn \
327 --multi-turn-min-turns 2 --multi-turn-max-turns 8 \
328 --random-input-len 2048 --per-turn-input-len 256 --random-output-len 128 \
329 --num-prompts 100 --multi-turn-concurrency 20
330
331 # ShareGPT conversations (loads all turns, not just the first two)
332 vllm-bench \
333 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
334 --dataset-name sharegpt --multi-turn \
335 --num-prompts 50 --multi-turn-concurrency 10 --save-result
336
337 # Think time between turns
338 vllm-bench \
339 --backend openai-chat --base-url http://127.0.0.1:8000 --model <model-name> \
340 --multi-turn --multi-turn-num-turns 3 --multi-turn-delay-ms 500 \
341 --num-prompts 100 --multi-turn-concurrency 20
342 ```
343
344 </details>
345
346 <details>
347 <summary><b>LoRA multi-adapter</b></summary>
348
349 ```bash
350 # Distribute requests across N adapters registered on the server.
351 # --model stays the BASE model (tokenizer / readiness / /tokenize use it);
352 # the per-request `model` field is rewritten to one of --lora-modules.
353 vllm-bench \
354 --backend openai-chat --base-url http://127.0.0.1:8000 \
355 --model Qwen/Qwen3-30B-A3B \
356 --lora-modules sql-lora-1 sql-lora-2 sql-lora-3 sql-lora-4 \
357 sql-lora-5 sql-lora-6 sql-lora-7 sql-lora-8 \
358 --lora-assignment random \
359 --dataset-name random --random-input-len 1024 --random-output-len 256 \
360 --num-prompts 1000 --max-concurrency 64 --save-result
361
362 # Deterministic round-robin assignment (request i -> adapter[i % N])
363 vllm-bench \
364 --backend openai-chat --base-url http://127.0.0.1:8000 \
365 --model Qwen/Qwen3-30B-A3B \
366 --lora-modules sql-lora-1 sql-lora-2 sql-lora-3 sql-lora-4 \
367 --lora-assignment round-robin \
368 --dataset-name random --num-prompts 1000
369 ```
370
371 Server side — start vLLM with `--enable-lora` and one `name=path` pair per adapter:
372
373 ```bash
374 vllm serve <base-model> \
375 --enable-lora --max-loras 8 --max-lora-rank 16 \
376 --lora-modules \
377 sql-lora-1=jeeejeee/qwen3-moe-text2sql-spider \
378 sql-lora-2=jeeejeee/qwen3-moe-text2sql-spider \
379 ...
380 ```
381
382 Set `--max-loras` ≥ number of adapter names to keep them all resident (clean steady-state numbers), or lower to stress the LoRA swap path.
383
384 </details>
385
386 <details>
387 <summary><b>Profiling &amp; dry-run</b></summary>
388
389 ```bash
390 # Trigger vLLM server-side profiling (start before, stop after the benchmark)
391 vllm-bench \
392 --backend vllm --base-url http://127.0.0.1:8000 --model <model-name> \
393 --num-prompts 100 --profile
394
395 # Defer profiling until the server batch is full, then capture for 10s
396 vllm-bench \
397 --backend vllm --base-url http://127.0.0.1:8000 --model <model-name> \
398 --num-prompts 2000 --max-concurrency 256 \
399 --profile --profile-batch-threshold 200 --profile-duration 10
400
401 # Dry run: generate dataset, print stats, send nothing
402 vllm-bench \
403 --model <model-name> --num-prompts 100000 --random-input-len 8192 --dry-run
404 ```
405
406 </details>
407
408 <details>
409 <summary><b>Multimodal preprocessing (offline)</b></summary>
410
411 The `mm-processor` subcommand benchmarks the multimodal preprocessing pipeline
412 itself (media fetch → per-modality preprocess → prompt expansion), mirroring
413 `vllm bench mm-processor`. It spawns a managed headless Python engine (real
414 weights, real encoder) — no serving endpoint needed — and reports per-stage
415 timing (mean/median/std/percentiles) plus end-to-end latency.
416
417 ```bash
418 vllm-bench mm-processor \
419 --model Qwen/Qwen2.5-VL-7B-Instruct \
420 --num-prompts 100 --num-warmups 5 \
421 --random-input-len 512 --random-output-len 128 \
422 --random-mm-base-items-per-request 1 \
423 --random-mm-limit-mm-per-prompt '{"image": 1, "video": 0}' \
424 --random-mm-bucket-config '{(1024, 800, 1): 1.0}' \
425 --metric-percentiles "50,90,99" \
426 --output-json mm_processor_stats.json
427 ```
428
429 `--max-concurrency` defaults to `1`, matching Python's serial driver; raise it
430 to measure a concurrent serving-style workload.
431
432 </details>
433
434 ## Supported Backends
435
436 ### Generation
437
438 | Backend | API Endpoint | Description |
439 | --------- | ------------- | ------------- |
440 | `vllm` / `openai` | `/v1/completions` | OpenAI-compatible completions (streaming) |
441 | `openai-chat` | `/v1/chat/completions` | OpenAI-compatible chat completions (streaming, multimodal) |
442
443 ### Embedding / Pooling
444
445 | Backend | API Endpoint | Description |
446 | --------- | ------------- | ------------- |
447 | `openai-embeddings` | `/v1/embeddings` | Text embedding (accepts text or token IDs) |
448 | `openai-embeddings-chat` | `/v1/embeddings` | Chat-format embedding (supports multimodal content) |
449 | `vllm-pooling` | `/pooling` | vLLM native pooling endpoint |
450 | `vllm-rerank` | `/v1/rerank` | vLLM reranking (query from prompt, documents via `--extra-body`) |
451
452 Pooling backends are non-streaming and report E2EL (end-to-end latency) only. Use `--dataset-name sharegpt`, `sonnet`, or `hf` for text-based embedding/rerank benchmarks, or `random` for token-ID-based embedding benchmarks.
453
454 ## Supported Datasets
455
456 | Dataset | Description |
457 | --------- | ------------- |
458 | `random` | Synthetic prompts with exact token-length matching (default) |
459 | `random-mm` | Synthetic multimodal prompts with random JPEG images for VLM benchmarking (requires `openai-chat`) |
460 | `sharegpt` | Real conversations from ShareGPT (auto-downloads from HuggingFace, or use `--dataset-path`) |
461 | `sonnet` | Built-in Shakespeare sonnets; controllable token length + shared prefix, no dataset file needed |
462 | `speed-bench` | NVIDIA SPEED-Bench for speculative decoding evaluation (auto-downloads, 11 categories) |
463 | `hf` | Any HuggingFace dataset with parquet data (downloads parquet shards via hf-hub into the standard HF hub cache, auto-detects chat/text columns) |
464
465 ## Metrics
466
467 ### Generation backends
468
469 - **TTFT** (Time to First Token) — latency from request send to first token received
470 - **TPOT** (Time per Output Token) — average time between output tokens
471 - **ITL** (Inter-Token Latency) — per-token latency distribution
472 - **E2EL** (End-to-End Latency) — total request latency
473 - **Throughput** — requests/sec, output tokens/sec, peak output tokens/sec, total tokens/sec
474 - **Concurrency** — peak concurrent requests
475 - **Goodput** — requests/sec meeting all specified SLOs (with `--goodput`)
476
477 ### Pooling / embedding backends
478
479 - **E2EL** — total request latency (mean, median, std, percentiles)
480 - **Throughput** — requests/sec, input tokens/sec
481 - **Concurrency** — peak concurrent requests
482
483 ### Steady-state metrics
484
485 When `--max-concurrency` is set and `--request-rate` is `inf` (closed-loop mode), the benchmark automatically reports an additional **Steady-State Metrics** block. It measures throughput and latency only over the window during which in-flight concurrency stays at or above a fraction of `--max-concurrency`, excluding the ramp-up and drain phases. This sharply reduces run-to-run variance at very high concurrency.
486
487 The block reports request/input/output/total token throughput plus TTFT (mean, median, percentiles) and TPOT (mean, median, P90, P99) over the detected plateau, along with the window bounds and how many requests fell inside it. Tune it with `--steady-state-threshold` (default `0.95`) and `--steady-state-min-window`, or disable with `--no-steady-state`. The result JSON carries a `steady_state` object (null when not computed).
488
489 ## CLI Reference
490
491 Run `vllm-bench --help` for the authoritative list. Grouped reference below.
492
493 <details>
494 <summary><b>Server connection</b></summary>
495
496 | Flag | Default | Description |
497 | ------ | --------- | ------------- |
498 | `--backend` | `openai` | Backend type (`vllm`, `openai`, `openai-chat`, `openai-embeddings`, `openai-embeddings-chat`, `vllm-pooling`, `vllm-rerank`) |
499 | `--base-url` | — | Server base URL (overrides `--host`/`--port`) |
500 | `--host` | `127.0.0.1` | Server host |
501 | `--port` | `8000` | Server port |
502 | `--endpoint` | Auto | API endpoint path (auto-selected per backend) |
503 | `--insecure` | `false` | Disable SSL certificate verification |
504
505 </details>
506
507 <details>
508 <summary><b>Model &amp; tokenizer</b></summary>
509
510 | Flag | Default | Description |
511 | ------ | --------- | ------------- |
512 | `--model` | Auto-detect | Model name (fetched from `/v1/models` if omitted) |
513 | `--served-model-name` | — | Model name used in API requests |
514 | `--tokenizer` | Same as model | Tokenizer name or path (supports HF, tiktoken, server fallback) |
515 | `--tokenizer-mode` | `auto` | Accepted for Python CLI compatibility but ignored (warns on non-`auto`): resolution always follows the HF `tokenizer.json` → tiktoken → server-side `/tokenize` chain, so `mistral_common` (tekken) tokenizers fall back to server-side tokenization |
516 | `--trust-remote-code` | `false` | Trust remote code for tokenizer |
517 | `--skip-tokenizer-init` | `false` | Skip tokenizer initialization |
518
519 </details>
520
521 <details>
522 <summary><b>Dataset</b></summary>
523
524 | Flag | Default | Description |
525 | ------ | --------- | ------------- |
526 | `--dataset-name` | `random` | Dataset type (`random`, `random-mm`, `sharegpt`, `sonnet`, `speed-bench`, `hf`) |
527 | `--dataset-path` | — | Path to dataset file (optional for `sharegpt`/`sonnet`, which auto-source) |
528 | `--num-prompts` | `1000` | Number of prompts to generate (conversations in multi-turn mode) |
529 | `--max-model-len` | — | Filter out requests where `prompt_len + output_len` exceeds this context length |
530 | `--input-len` | — | Override input length (general) |
531 | `--output-len` | — | Override output length (general) |
532 | `--no-oversample` | `false` | Don't oversample if dataset is smaller than `--num-prompts` |
533 | `--disable-shuffle` | `false` | Don't shuffle the dataset |
534 | `--seed` | `0` | Random seed for reproducibility |
535 | **Random** | | |
536 | `--random-input-len` | `1024` | Input token length |
537 | `--random-output-len` | `128` | Output token length |
538 | `--random-prefix-len` | `0` | Shared prefix length |
539 | `--random-range-ratio` | `0.0` | Length jitter `r`, range `[0, 1)`. Lengths sampled uniformly from `[len×(1−r), len×(1+r)]`; `0.0` = exact target lengths. Also accepts `'{"input": r1, "output": r2}'` for independent input/output jitter |
540 | `--prompt-token-ids` | `false` | Send prompts as token-ID arrays (skips server-side tokenization, exact counts). Random dataset only |
541 | **Random multimodal** | | |
542 | `--random-mm-base-items-per-request` | `1` | Base number of multimodal items (images) per request |
543 | `--random-mm-num-mm-items-range-ratio` | `0.0` | Range ratio for varying item count per request |
544 | `--random-mm-limit-mm-per-prompt` | `{"image": 255, "video": 1}` | Per-modality hard caps (JSON) |
545 | `--random-mm-bucket-config` | `{(256,256,1): 0.5, (720,1280,1): 0.5}` | `(height,width,frames)` → probability (Python tuple syntax; frames=1 = image) |
546 | **ShareGPT** | | |
547 | `--sharegpt-output-len` | — | Override output length |
548 | **Sonnet** | | |
549 | `--sonnet-input-len` | `550` | Input tokens per request |
550 | `--sonnet-output-len` | `150` | Output tokens per request |
551 | `--sonnet-prefix-len` | `200` | Prefix tokens shared across requests |
552 | **SPEED-Bench** | | |
553 | `--speed-bench-config` | `qualitative` | Split (`qualitative`, `throughput_1k`/`2k`/`8k`/`16k`/`32k`); alias: `--speed-bench-dataset-subset` (Python name) |
554 | `--speed-bench-output-len` | `4096` | Output tokens per request (matches Python default) |
555 | `--speed-bench-category` | — | Filter by category (`low_entropy`, `high_entropy`, `mixed_entropy`, `coding`, `math`, …) |
556 | `--speed-bench-max-input-len` | — | Truncate prompts to at most N tokens |
557 | **HuggingFace** | | |
558 | `--hf-split` | Auto | Split (`train`, `test`, `validation`); auto-detected if omitted |
559 | `--hf-subset` | — | Subset/config name (e.g. `narrativeqa` for LongBench) |
560 | `--hf-output-len` | — | Fixed output length for all requests (overrides dataset-derived length) |
561 | `--hf-text-column` | Auto | Column containing prompt text; auto-detected from common patterns |
562
563 </details>
564
565 <details>
566 <summary><b>Rate control</b></summary>
567
568 | Flag | Default | Description |
569 | ------ | --------- | ------------- |
570 | `--request-rate` | `inf` | Requests per second (`inf` = all at once) |
571 | `--burstiness` | `1.0` | Burstiness factor (1.0 = Poisson, >1 = bursty) |
572 | `--max-concurrency` | `num-prompts` | Maximum concurrent requests (semaphore) |
573 | `--ramp-up-strategy` | — | Ramp-up mode (`linear` or `exponential`) |
574 | `--ramp-up-start-rps` | — | Starting request rate for ramp-up |
575 | `--ramp-up-end-rps` | — | Ending request rate for ramp-up |
576
577 </details>
578
579 <details>
580 <summary><b>Sampling parameters</b></summary>
581
582 | Flag | Description |
583 | ------ | ------------- |
584 | `--temperature` | Temperature (server default if omitted) |
585 | `--top-p` | Top-p (nucleus) sampling |
586 | `--top-k` | Top-k sampling |
587 | `--min-p` | Min-p sampling |
588 | `--frequency-penalty` | Frequency penalty |
589 | `--presence-penalty` | Presence penalty |
590 | `--repetition-penalty` | Repetition penalty |
591
592 Merged into the request body. Only effective with generation backends (`vllm`, `openai`, `openai-chat`); ignored by pooling/embedding backends.
593
594 </details>
595
596 <details>
597 <summary><b>Output &amp; results</b></summary>
598
599 | Flag | Default | Description |
600 | ------ | --------- | ------------- |
601 | `--save-result` | `false` | Save results to JSON file |
602 | `--save-detailed` | `false` | Include per-request data in JSON (input/output lens, ITLs, texts) |
603 | `--append-result` | `false` | Append to existing JSON file (JSONL format) |
604 | `--result-dir` | — | Directory for result files |
605 | `--result-filename` | Auto | Custom result filename |
606 | `--percentile-metrics` | `ttft,tpot,itl,e2el` | Metrics for percentile reporting (pooling defaults to `e2el` only) |
607 | `--metric-percentiles` | `99` | Percentile values to compute |
608 | `--sweep-summary-percentiles` | — | Extra percentiles for sweep summary tables (auto-added to computed set) |
609 | `--goodput` | — | SLO pairs for goodput (`ttft:100 tpot:50 e2el:500`, values in ms) |
610 | `--disable-tqdm` | `false` | Disable progress bar |
611 | `--label` | — | Label prefix for result files |
612 | `--metadata` | — | Key-value metadata (`KEY=VALUE`, repeatable) |
613
614 </details>
615
616 <details>
617 <summary><b>Request options</b></summary>
618
619 | Flag | Default | Description |
620 | ------ | --------- | ------------- |
621 | `--ignore-eos` | `false` | Ignore EOS token (force full output length) |
622 | `--logprobs` | — | Number of logprobs per token |
623 | `--num-warmups` | `0` | Warmup requests before benchmarking |
624 | `--ready-check-timeout-sec` | `0` | Endpoint readiness timeout (0 = skip) |
625 | `--request-id-prefix` | Auto (UUID) | Prefix for request IDs |
626 | `--header` | — | Extra headers (`KEY=VALUE`, repeatable) |
627 | `--extra-body` | — | Extra JSON body parameters |
628 | `--dry-run` | `false` | Generate dataset only, skip benchmark |
629
630 </details>
631
632 <details>
633 <summary><b>Steady-state metrics</b></summary>
634
635 | Flag | Default | Description |
636 | ------ | --------- | ------------- |
637 | `--steady-state-threshold` | `0.95` | Fraction of `--max-concurrency` at which the steady-state window opens, range (0, 1] |
638 | `--steady-state-min-window` | Auto | Minimum window duration (s) below which a warning is attached. Default `max(10, 0.1 × run_duration)` |
639 | `--no-steady-state` | `false` | Disable steady-state metrics computation |
640
641 Computed only when `--max-concurrency` is set and `--request-rate` is `inf`.
642
643 </details>
644
645 <details>
646 <summary><b>Profiling</b></summary>
647
648 | Flag | Default | Description |
649 | ------ | --------- | ------------- |
650 | `--profile` | `false` | Trigger vLLM server-side profiling (`/start_profile` before, `/stop_profile` after) |
651 | `--profile-batch-threshold` | — | Defer profiling until `/metrics` reports ≥ N running requests, then capture. Requires `--profile` |
652 | `--profile-duration` | `5.0` | Seconds to capture once the batch threshold is reached. Requires `--profile-batch-threshold` |
653
654 </details>
655
656 <details>
657 <summary><b>Sweep mode</b></summary>
658
659 | Flag | Default | Description |
660 | ------ | --------- | ------------- |
661 | `--sweep-max-concurrency` | — | Comma-separated concurrency values to sweep (e.g. `1,10,50,100,500`) |
662 | `--sweep-request-rate` | — | Comma-separated rate values to sweep, supports `inf` (e.g. `1,10,100,inf`) |
663 | `--sweep-num-prompts-factor` | — | Set `num_prompts = concurrency × factor` per concurrency sweep point |
664 | `--reset-prefix-cache` | `false` | Reset the server's prefix cache before each sweep iteration (requires `VLLM_SERVER_DEV_MODE=1`) |
665
666 Runs the benchmark once per value, then prints a summary table comparing throughput and latency across all sweep points and identifies the best-throughput configuration. Works in multi-turn mode too. `--sweep-summary-percentiles` appends extra TTFT/TPOT/E2EL columns to the summary, auto-adding any missing percentiles to the computed set so they also appear in result JSON.
667
668 </details>
669
670 <details>
671 <summary><b>Multi-turn conversation benchmark</b></summary>
672
673 | Flag | Default | Description |
674 | ------ | --------- | ------------- |
675 | `--multi-turn` | `false` | Enable multi-turn conversation mode (requires `--backend openai-chat`) |
676 | `--multi-turn-num-turns` | `3` | Turns per conversation (synthetic mode) |
677 | `--multi-turn-min-turns` | `0` | Minimum turns per conversation (0 = use `--multi-turn-num-turns`) |
678 | `--multi-turn-max-turns` | `0` | Maximum turns per conversation (0 = `--multi-turn-num-turns` synthetic / uncapped ShareGPT) |
679 | `--multi-turn-concurrency` | — | Concurrent conversations (defaults to `--max-concurrency` or `--num-prompts`) |
680 | `--multi-turn-delay-ms` | `0` | Delay between turns in ms (simulates user think time) |
681 | `--per-turn-input-len` | `0` | Input token length for turns 1+ (0 = use `--random-input-len` for all turns) |
682 | `--multi-turn-prefix-global-ratio` | `0.0` | Fraction of per-turn input shared across all conversations (random dataset only) |
683 | `--multi-turn-prefix-conversation-ratio` | `0.0` | Fraction shared within each conversation (random dataset only) |
684
685 With `--multi-turn`, `--num-prompts` controls the number of **conversations**, not individual requests.
686
687 **How it works:**
688
689 - Turn 1: send `[user_1]`, get `assistant_1`
690 - Turn 2: send `[user_1, assistant_1, user_2]`, get `assistant_2`
691 - Turn N: send full history + `user_N` — measures growing-context performance
692
693 **Data sources:**
694
695 - `--dataset-name random` — synthetic conversations with controllable per-turn token lengths. Auto-sets `min_tokens` to enforce output length without `ignore_eos`.
696 - `--dataset-name sharegpt` — loads all turns (not just the first two); filters for entries with ≥ 2 real turns.
697 - `--dataset-name hf` — downloads a ShareGPT-format HuggingFace config, then loads it like `sharegpt`.
698
699 **Prefix sharing** (random dataset): when `--multi-turn-prefix-global-ratio` or `--multi-turn-prefix-conversation-ratio` is > 0, each turn sends a fixed-length message (no history accumulation) composed of a global prefix + per-conversation prefix + unique suffix. The two ratios must sum to < 1.0.
700
701 **Router affinity:** every turn sends `X-Session-ID: {conversation_id}` for KV-cache reuse behind a vLLM router.
702
703 **Output:** overall metrics plus a per-turn breakdown (TTFT/TPOT/ITL/E2EL by turn index). Expect TTFT to climb across turns due to growing context. JSON includes a `per_turn_metrics` array.
704
705 </details>
706
707 <details>
708 <summary><b>LoRA multi-adapter</b></summary>
709
710 | Flag | Default | Description |
711 | ------ | --------- | ------------- |
712 | `--lora-modules` | — | Adapter names registered on the server (`vllm serve --lora-modules name=path`). Each request's `model` field is rewritten to one of these. Repeatable |
713 | `--lora-assignment` | `random` | Distribution: `random` (uniform, seeded by `--seed`) or `round-robin` (deterministic `i % N`) |
714
715 `--model` must stay the **base** model — its tokenizer builds prompts, and `/v1/models`, `/tokenize`, ready check, and warmup all use it. Only the per-request `model` field in completions/chat payloads is rewritten to the assigned adapter (vLLM routes by name).
716
717 **Assignment scope:** per request in single-shot mode; **per conversation** (sticky across all turns) in multi-turn mode, to avoid breaking prefix-cache reuse mid-dialog.
718
719 **Reproducibility:** with `--lora-assignment random`, the same `--seed` + same `--lora-modules` list yields identical request-to-adapter mappings. Pooling/embedding backends are rejected — LoRA routing applies to generative paths only.
720
721 </details>
722
723 <details>
724 <summary><b>Multi-run &amp; comparison</b></summary>
725
726 | Flag | Default | Description |
727 | ------ | --------- | ------------- |
728 | `--num-runs` | `1` | Run benchmark N times; report mean/std/min/max with CV |
729 | `--compare` | — | Compare two result JSON files side-by-side (skips benchmarking) |
730
731 `--num-runs` aggregates metrics across runs and reports the coefficient of variation (CV) for throughput stability. `--compare` reads two previously-saved result files and prints a diff with delta, % change, and improvement/regression markers.
732
733 </details>
734
735 <details>
736 <summary><b><code>mm-processor</code> subcommand</b></summary>
737
738 Offline multimodal preprocessing latency benchmark (mirrors `vllm bench mm-processor`). Spawns a managed headless Python engine and drives the full chat preprocessing pipeline.
739
740 | Flag | Default | Description |
741 | ------ | --------- | ------------- |
742 | `--model` | (required) | Model to serve and benchmark (HF id or local path) |
743 | `--num-prompts` | `10` | Prompts to process, excluding warmups |
744 | `--num-warmups` | `1` | Warmup prompts processed and discarded before timing |
745 | `--metric-percentiles` | `99` | Comma-separated percentiles to report |
746 | `--output-json` | — | Path to write aggregate stats as JSON (same schema as Python) |
747 | `--random-input-len` | `1024` | Text input length per prompt |
748 | `--random-output-len` | `128` | Expected output length per prompt |
749 | `--random-prefix-len` | `0` | Prefix token length per prompt |
750 | `--random-range-ratio` | `0.0` | Input/output length range ratio (float or `{"input": i, "output": o}`) |
751 | `--random-mm-base-items-per-request` | `1` | Base number of multimodal items per request |
752 | `--random-mm-num-mm-items-range-ratio` | `0.0` | Range ratio (in [0, 1]) for the number of mm items per request |
753 | `--random-mm-limit-mm-per-prompt` | `{"image": 255, "video": 1}` | Per-modality item caps |
754 | `--random-mm-bucket-config` | `{(256,256,1): 0.5, (720,1280,1): 0.5}` | Bucket config |
755 | `--seed` | `0` | Seed for dataset generation |
756 | `--request-id-prefix` | `mm-proc-` | Request-id prefix for per-request timing keys |
757 | `--max-concurrency` | `1` | Maximum concurrent requests (1 matches Python's serial driver) |
758 | `--trust-remote-code` | `false` | Trust remote code for the tokenizer and the managed engine |
759 | `--chat-template` | — | Chat-template override (inline text or path) |
760
761 Managed headless-engine options (e.g. `--python`, `--data-parallel-address`, `--data-parallel-rpc-port`, `--data-parallel-size`, `--max-model-len`, `--python-args`) are shared with other managed-engine commands; see `vllm-bench mm-processor --help`.
762
763 </details>
764
765 ## Tokenizer Support
766
767 Tokenizers are loaded with a three-tier fallback chain:
768
769 1. **Local HuggingFace** — `tokenizer.json` from a local path or the Hub (fastest)
770 2. **Tiktoken** — `.tiktoken` / `.model` format for Kimi, Qwen, etc. (auto-extracts `pat_str` from Python source)
771 3. **Server-side** — falls back to vLLM's `/tokenize` + `/detokenize` endpoints
772
773 For the `random` dataset, prompt token lengths are verified against the server on the first run and cached; subsequent runs with the same model+server skip verification. Verification is also skipped when `--prompt-token-ids` is set (token counts are exact by construction).
774
775 Models without `tokenizer.json` (e.g. `nvidia/Kimi-K2.5-NVFP4`) fall back to server-side tokenization automatically; you can also point `--tokenizer` at a model that ships `tokenizer.json`.
776
777 ## Output Format
778
779 JSON output is compatible with the `vllm bench serve` Python schema. Result files are named:
780
781 ```text
782 {label}-{rate}qps-concurrency{max_concurrency}-{model}-{timestamp}.json
783 ```
784
785 Use `--append-result` to append multiple runs to the same file in JSONL format. `--save-detailed` adds per-request arrays (input/output lengths, ITLs, generated text).
786
787 ## Architecture
788
789 <details>
790 <summary><b>Source layout</b></summary>
791
792 ```text
793 src/
794 ├── main.rs # Entry point, mimalloc, tokio runtime, mode/subcommand dispatch
795 ├── cli.rs # clap CLI argument definitions
796 ├── config.rs # Validated config, goodput/ramp-up parsing
797 ├── benchmark.rs # Core orchestrator (schedule, spawn, collect, verify, profile)
798 ├── mm_processor.rs # Offline mm preprocessing benchmark (mm-processor subcommand)
799 ├── multi_turn.rs # Multi-turn conversation orchestrator (channel workers)
800 ├── compare.rs # Result diff (--compare file_a.json file_b.json)
801 ├── sweep.rs # Parameter sweep (--sweep-max-concurrency, --sweep-request-rate)
802 ├── multi_run.rs # Multi-run statistics (--num-runs N)
803 ├── rate_control.rs # Gamma/Poisson scheduling + linear/exponential ramp-up
804 ├── ready_checker.rs # Endpoint readiness with retry
805 ├── tokenizer.rs # Tokenizer abstraction (HF, tiktoken, server)
806 ├── tiktoken.rs # Tiktoken BPE loader with pat_str extraction
807 ├── error.rs # Error types
808 ├── backends/
809 │ ├── mod.rs # Backend enum dispatch, typed SSE structs
810 │ ├── streaming.rs # SSE stream parser with speculative JSON parse
811 │ ├── openai_completions.rs # /v1/completions backend
812 │ ├── openai_chat.rs # /v1/chat/completions backend
813 │ └── pooling.rs # Embedding/pooling/rerank backends (non-streaming)
814 ├── datasets/
815 │ ├── mod.rs # SampleRequest, ConversationTurn, MultiTurnConversation types
816 │ ├── random.rs # Random dataset with rayon parallelism
817 │ ├── random_mm.rs # Random multimodal dataset (JPEG generation, bucket sampling)
818 │ ├── multi_turn.rs # Multi-turn synthetic + ShareGPT conversation generators
819 │ ├── sharegpt.rs # ShareGPT JSON dataset loader
820 │ ├── sonnet.rs # Sonnet dataset (built-in Shakespeare sonnets)
821 │ ├── speed_bench.rs # NVIDIA SPEED-Bench loader (auto-download + cache)
822 │ └── hf_dataset.rs # Generic HuggingFace dataset (auto-download, column detection)
823 ├── metrics/
824 │ ├── mod.rs # BenchmarkMetrics, MultiTurnMetrics structs
825 │ ├── calculator.rs # Percentile/throughput/goodput/peak/multi-turn computation
826 │ └── steady_state.rs # Steady-state window detection + plateau metrics
827 └── output/
828 ├── mod.rs
829 ├── console.rs # Terminal output (matches Python format)
830 └── json.rs # JSON result serialization (Python-compatible schema)
831 ```
832
833 </details>
834
835 ### Key design decisions
836
837 - **reqwest + tokio** — HTTP client with connection pooling, forced HTTP/1.1, TCP_NODELAY to match Python's aiohttp and avoid Nagle latency inflation on TTFT
838 - **mimalloc** — global allocator to reduce contention under high concurrency (1400+ tasks); page-agnostic, runs on aarch64 4K- and 64K-page kernels
839 - **`Arc<str>` prompts** — zero-copy prompt sharing across tokio tasks, eliminating ~3 GB peak memory at 100k requests with 8k-token prompts
840 - **Spawn-per-request** — `tokio::spawn` per request with a `Semaphore` for concurrency control (matches Python's asyncio pattern)
841 - **rayon** — parallel dataset generation across CPU cores (200–500× faster than Python for 100k+ prompts)
842 - **Enum dispatch** — backend variants instead of trait objects (avoids async trait-object limitations)
843 - **Typed SSE deserialization** — `CompletionChunk`/`ChatChunk` structs skip unused JSON fields (cheaper than `serde_json::Value`)
844 - **Speculative JSON parse** — SSE handler uses `serde_json::value::RawValue` to detect complete JSON before `\n\n` arrives, improving TTFT/ITL accuracy when TCP segments split
845 - **Connection error retry** — automatic retry with backoff on connection reset/timeout/refused (up to 3 attempts)
846 - **Tokenizer verification cache** — server-side token-length verification is cached per model+server pair
847
848 ### Behavioral parity with Python
849
850 The Rust implementation matches Python `vllm bench serve` in:
851
852 - SSE streaming protocol handling (including speculative parse for split TCP segments)
853 - Timing semantics (monotonic `Instant` matching Python's `time.perf_counter()`)
854 - Chat vs. completions differences (`max_completion_tokens` vs. `max_tokens`, Content-Type, timestamp placement)
855 - JSON output schema (all fields, key naming, `request_rate` as the string `"inf"`)
856 - Rate control (Gamma distribution, normalization, burstiness, linear/exponential ramp-up)
857 - Metrics (TTFT/TPOT/ITL/E2EL percentiles, peak tokens/sec, peak concurrency, goodput)
858 - Sampling parameters merged into the request body via `extra_body` (same precedence rules)
859 - HF dataset prompt construction (token-identical input totals vs. Python on full GSM8k / MT-Bench splits; Python-only per-dataset chat templating is intentionally not applied — prompts are sent raw)
860 - CLI flags: a superset of Python's, enforced by `tests/cli_parity.rs` against a
861 snapshot of the Python parser (`tests/python_serve_flags.txt`, regenerated by
862 `tests/benchmarks/test_rust_bench_cli_parity.py` in the repo root) with an
863 explicit allowlist for the Python-only remainder — so `VLLM_USE_RUST_BENCH=1`
864 delegation cannot silently reject documented flags
865
866 ## Environment Variables
867
868 | Variable | Description |
869 | ---------- | ------------- |
870 | `OPENAI_API_KEY` | API key for authenticated endpoints (cached, not read per-request) |
871 | `HF_TOKEN` | HuggingFace token for gated/private tokenizers and datasets (falls back to the `hf auth login` token file) |
872 | `HF_HOME` | Overrides the HuggingFace cache location (default `~/.cache/huggingface`) used for tokenizers and dataset parquet shards |
873 | `TOKIO_WORKER_THREADS` | Override tokio worker thread count (default: physical cores) |
874
875 ## License
876
877 Apache-2.0
878 </content>
879 </invoke>