A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 23:45 UTC
[CI/Build][Doc] Clean up more docs that point to old bench scripts (#21667)
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
Ye (Charlotte) Qi committed
Jul 26, 2025 at 21:02 UTC
01a395e9e720ec06db299ddb561da8e00b094182
9 files changed
+66
-63
.buildkite/nightly-benchmarks/README.md
+5
-5
@@ -74,7 +74,7 @@ Here is an example of one test inside `latency-tests.json`:
74
In this example:
75
76
- The `test_name` attributes is a unique identifier for the test. In `latency-tests.json`, it must start with `latency_`.
77
-- The `parameters` attribute control the command line arguments to be used for `benchmark_latency.py`. Note that please use underline `_` instead of the dash `-` when specifying the command line arguments, and `run-performance-benchmarks.sh` will convert the underline to dash when feeding the arguments to `benchmark_latency.py`. For example, the corresponding command line arguments for `benchmark_latency.py` will be `--model meta-llama/Meta-Llama-3-8B --tensor-parallel-size 1 --load-format dummy --num-iters-warmup 5 --num-iters 15`
77
+- The `parameters` attribute control the command line arguments to be used for `vllm bench latency`. Note that please use underline `_` instead of the dash `-` when specifying the command line arguments, and `run-performance-benchmarks.sh` will convert the underline to dash when feeding the arguments to `vllm bench latency`. For example, the corresponding command line arguments for `vllm bench latency` will be `--model meta-llama/Meta-Llama-3-8B --tensor-parallel-size 1 --load-format dummy --num-iters-warmup 5 --num-iters 15`
78
79
Note that the performance numbers are highly sensitive to the value of the parameters. Please make sure the parameters are set correctly.
80
@@ -82,13 +82,13 @@ WARNING: The benchmarking script will save json results by itself, so please do
82
83
### Throughput test
84
85
-The tests are specified in `throughput-tests.json`. The syntax is similar to `latency-tests.json`, except for that the parameters will be fed forward to `benchmark_throughput.py`.
85
+The tests are specified in `throughput-tests.json`. The syntax is similar to `latency-tests.json`, except for that the parameters will be fed forward to `vllm bench throughput`.
86
87
The number of this test is also stable -- a slight change on the value of this number might vary the performance numbers by a lot.
88
89
### Serving test
90
91
-We test the throughput by using `benchmark_serving.py` with request rate = inf to cover the online serving overhead. The corresponding parameters are in `serving-tests.json`, and here is an example:
91
+We test the throughput by using `vllm bench serve` with request rate = inf to cover the online serving overhead. The corresponding parameters are in `serving-tests.json`, and here is an example:
92
93
```json
94
[
@@ -118,8 +118,8 @@ Inside this example:
118
119
- The `test_name` attribute is also a unique identifier for the test. It must start with `serving_`.
120
- The `server-parameters` includes the command line arguments for vLLM server.
121
-- The `client-parameters` includes the command line arguments for `benchmark_serving.py`.
122
-- The `qps_list` controls the list of qps for test. It will be used to configure the `--request-rate` parameter in `benchmark_serving.py`
121
+- The `client-parameters` includes the command line arguments for `vllm bench serve`.
122
+- The `qps_list` controls the list of qps for test. It will be used to configure the `--request-rate` parameter in `vllm bench serve`
123
124
The number of this test is less stable compared to the delay and latency benchmarks (due to randomized sharegpt dataset sampling inside `benchmark_serving.py`), but a large change on this number (e.g. 5% change) still vary the output greatly.
125
.buildkite/nightly-benchmarks/scripts/convert-results-json-to-markdown.py
+3
-3
@@ -100,7 +100,7 @@ if __name__ == "__main__":
100
raw_result = json.loads(f.read())
101
102
if "serving" in str(test_file):
103
- # this result is generated via `benchmark_serving.py`
103
+ # this result is generated via `vllm bench serve` command
104
105
# attach the benchmarking command to raw_result
106
try:
@@ -120,7 +120,7 @@ if __name__ == "__main__":
120
continue
121
122
elif "latency" in f.name:
123
- # this result is generated via `benchmark_latency.py`
123
+ # this result is generated via `vllm bench latency` command
124
125
# attach the benchmarking command to raw_result
126
try:
@@ -148,7 +148,7 @@ if __name__ == "__main__":
148
continue
149
150
elif "throughput" in f.name:
151
- # this result is generated via `benchmark_throughput.py`
151
+ # this result is generated via `vllm bench throughput` command
152
153
# attach the benchmarking command to raw_result
154
try:
.buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh
+1
-1
@@ -127,7 +127,7 @@ ensure_installed() {
127
}
128
129
run_serving_tests() {
130
- # run serving tests using `benchmark_serving.py`
130
+ # run serving tests using `vllm bench serve` command
131
# $1: a json file specifying serving test cases
132
133
local serving_test_file
.buildkite/nightly-benchmarks/scripts/run-performance-benchmarks.sh
+4
-4
@@ -165,7 +165,7 @@ upload_to_buildkite() {
165
}
166
167
run_latency_tests() {
168
- # run latency tests using `benchmark_latency.py`
168
+ # run latency tests using `vllm bench latency` command
169
# $1: a json file specifying latency test cases
170
171
local latency_test_file
@@ -232,7 +232,7 @@ run_latency_tests() {
232
}
233
234
run_throughput_tests() {
235
- # run throughput tests using `benchmark_throughput.py`
235
+ # run throughput tests using `vllm bench throughput`
236
# $1: a json file specifying throughput test cases
237
238
local throughput_test_file
@@ -298,7 +298,7 @@ run_throughput_tests() {
298
}
299
300
run_serving_tests() {
301
- # run serving tests using `benchmark_serving.py`
301
+ # run serving tests using `vllm bench serve` command
302
# $1: a json file specifying serving test cases
303
304
local serving_test_file
@@ -448,7 +448,7 @@ main() {
448
(which jq) || (apt-get update && apt-get -y install jq)
449
(which lsof) || (apt-get update && apt-get install -y lsof)
450
451
- # get the current IP address, required by benchmark_serving.py
451
+ # get the current IP address, required by `vllm bench serve` command
452
export VLLM_HOST_IP=$(hostname -I | awk '{print $1}')
453
# turn of the reporting of the status of each request, to clean up the terminal output
454
export VLLM_LOGGING_LEVEL="WARNING"
benchmarks/auto_tune/README.md
+1
-1
@@ -105,7 +105,7 @@ After the script finishes, you will find the results in a new, timestamped direc
105
106
- **Log Files**: The directory (`$BASE/auto-benchmark/YYYY_MM_DD_HH_MM/`) contains detailed logs for each run:
107
- `vllm_log_...txt`: The log output from the vLLM server for each parameter combination.
108
- - `bm_log_...txt`: The log output from the `benchmark_serving.py` script for each benchmark run.
108
+ - `bm_log_...txt`: The log output from the `vllm bench serve` command for each benchmark run.
109
110
- **Final Result Summary**: A file named `result.txt` is created in the log directory. It contains a summary of each tested combination and concludes with the overall best parameters found.
111
benchmarks/disagg_benchmarks/disagg_overhead_benchmark.sh
+30
-30
@@ -3,7 +3,7 @@
3
# benchmark the overhead of disaggregated prefill.
4
# methodology:
5
# - send all request to prefill vLLM instance. It will buffer KV cache.
6
-# - then send all request to decode instance.
6
+# - then send all request to decode instance.
7
# - The TTFT of decode instance is the overhead.
8
9
set -ex
@@ -63,7 +63,7 @@ benchmark() {
63
--gpu-memory-utilization 0.6 \
64
--kv-transfer-config \
65
'{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":5e9}' &
66
-
66
+
67
68
CUDA_VISIBLE_DEVICES=1 python3 \
69
-m vllm.entrypoints.openai.api_server \
@@ -78,38 +78,38 @@ benchmark() {
78
wait_for_server 8200
79
80
# let the prefill instance finish prefill
81
- python3 ../benchmark_serving.py \
82
- --backend vllm \
83
- --model $model \
84
- --dataset-name $dataset_name \
85
- --dataset-path $dataset_path \
86
- --sonnet-input-len $input_len \
87
- --sonnet-output-len "$output_len" \
88
- --sonnet-prefix-len $prefix_len \
89
- --num-prompts $num_prompts \
90
- --port 8100 \
91
- --save-result \
92
- --result-dir $results_folder \
93
- --result-filename disagg_prefill_tp1.json \
94
- --request-rate "inf"
81
+ vllm bench serve \
82
+ --backend vllm \
83
+ --model $model \
84
+ --dataset-name $dataset_name \
85
+ --dataset-path $dataset_path \
86
+ --sonnet-input-len $input_len \
87
+ --sonnet-output-len "$output_len" \
88
+ --sonnet-prefix-len $prefix_len \
89
+ --num-prompts $num_prompts \
90
+ --port 8100 \
91
+ --save-result \
92
+ --result-dir $results_folder \
93
+ --result-filename disagg_prefill_tp1.json \
94
+ --request-rate "inf"
95
96
97
# send the request to decode.
98
# The TTFT of this command will be the overhead of disagg prefill impl.
99
- python3 ../benchmark_serving.py \
100
- --backend vllm \
101
- --model $model \
102
- --dataset-name $dataset_name \
103
- --dataset-path $dataset_path \
104
- --sonnet-input-len $input_len \
105
- --sonnet-output-len "$output_len" \
106
- --sonnet-prefix-len $prefix_len \
107
- --num-prompts $num_prompts \
108
- --port 8200 \
109
- --save-result \
110
- --result-dir $results_folder \
111
- --result-filename disagg_prefill_tp1_overhead.json \
112
- --request-rate "$qps"
99
+ vllm bench serve \
100
+ --backend vllm \
101
+ --model $model \
102
+ --dataset-name $dataset_name \
103
+ --dataset-path $dataset_path \
104
+ --sonnet-input-len $input_len \
105
+ --sonnet-output-len "$output_len" \
106
+ --sonnet-prefix-len $prefix_len \
107
+ --num-prompts $num_prompts \
108
+ --port 8200 \
109
+ --save-result \
110
+ --result-dir $results_folder \
111
+ --result-filename disagg_prefill_tp1_overhead.json \
112
+ --request-rate "$qps"
113
kill_gpu_processes
114
115
}
benchmarks/disagg_benchmarks/disagg_performance_benchmark.sh
+15
-15
@@ -60,7 +60,7 @@ launch_chunked_prefill() {
60
61
62
launch_disagg_prefill() {
63
- model="meta-llama/Meta-Llama-3.1-8B-Instruct"
63
+ model="meta-llama/Meta-Llama-3.1-8B-Instruct"
64
# disagg prefill
65
CUDA_VISIBLE_DEVICES=0 python3 \
66
-m vllm.entrypoints.openai.api_server \
@@ -99,20 +99,20 @@ benchmark() {
99
output_len=$2
100
tag=$3
101
102
- python3 ../benchmark_serving.py \
103
- --backend vllm \
104
- --model $model \
105
- --dataset-name $dataset_name \
106
- --dataset-path $dataset_path \
107
- --sonnet-input-len $input_len \
108
- --sonnet-output-len "$output_len" \
109
- --sonnet-prefix-len $prefix_len \
110
- --num-prompts $num_prompts \
111
- --port 8000 \
112
- --save-result \
113
- --result-dir $results_folder \
114
- --result-filename "$tag"-qps-"$qps".json \
115
- --request-rate "$qps"
102
+ vllm bench serve \
103
+ --backend vllm \
104
+ --model $model \
105
+ --dataset-name $dataset_name \
106
+ --dataset-path $dataset_path \
107
+ --sonnet-input-len $input_len \
108
+ --sonnet-output-len "$output_len" \
109
+ --sonnet-prefix-len $prefix_len \
110
+ --num-prompts $num_prompts \
111
+ --port 8000 \
112
+ --save-result \
113
+ --result-dir $results_folder \
114
+ --result-filename "$tag"-qps-"$qps".json \
115
+ --request-rate "$qps"
116
117
sleep 2
118
}
docs/contributing/profiling.md
+6
-3
@@ -9,10 +9,13 @@ We support tracing vLLM workers using the `torch.profiler` module. You can enabl
9
10
The OpenAI server also needs to be started with the `VLLM_TORCH_PROFILER_DIR` environment variable set.
11
12
-When using `benchmarks/benchmark_serving.py`, you can enable profiling by passing the `--profile` flag.
12
+When using `vllm bench serve`, you can enable profiling by passing the `--profile` flag.
13
14
Traces can be visualized using <https://ui.perfetto.dev/>.
15
16
+!!! tip
17
+You can directly call bench module without installing vllm using `python -m vllm.entrypoints.cli.main bench`.
18
+
19
!!! tip
20
Only send a few requests through vLLM when profiling, as the traces can get quite large. Also, no need to untar the traces, they can be viewed directly.
21
@@ -35,7 +38,7 @@ VLLM_TORCH_PROFILER_DIR=./vllm_profile \
38
--model meta-llama/Meta-Llama-3-70B
39
```
40
38
-benchmark_serving.py:
41
+vllm bench command:
42
43
```bash
44
vllm bench serve \
@@ -69,7 +72,7 @@ apt install nsight-systems-cli
72
73
For basic usage, you can just append `nsys profile -o report.nsys-rep --trace-fork-before-exec=true --cuda-graph-trace=node` before any existing script you would run for offline inference.
74
72
-The following is an example using the `benchmarks/benchmark_latency.py` script:
75
+The following is an example using the `vllm bench latency` script:
76
77
```bash
78
nsys profile -o report.nsys-rep \
examples/online_serving/prometheus_grafana/README.md
+1
-1
@@ -28,7 +28,7 @@ Submit some sample requests to the server:
28
```bash
29
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
30
31
-python3 ../../../benchmarks/benchmark_serving.py \
31
+vllm bench serve \
32
--model mistralai/Mistral-7B-v0.1 \
33
--tokenizer mistralai/Mistral-7B-v0.1 \
34
--endpoint /v1/completions \