@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 00e3f9da46
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

vLLM Benchmark suite improvement (#22119)

vLLM Benchmark suite improvement (#22119) Signed-off-by: Tsai, Louie <louie.tsai@intel.com> Signed-off-by: Louie Tsai <louie.tsai@intel.com> Co-authored-by: Li, Jiang <bigpyj64@gmail.com>

Louie Tsai committed Aug 14, 2025 at 00:12 UTC 00e3f9da462b31f271d9d9fdb526f148572609a9
10 files changed +452 -152
.buildkite/nightly-benchmarks/README.md
+12 -20
@@ -7,7 +7,7 @@ This directory contains two sets of benchmark for vllm.
7 - Performance benchmark: benchmark vllm's performance under various workload, for **developers** to gain clarity on whether their PR improves/degrades vllm's performance
8 - Nightly benchmark: compare vllm's performance against alternatives (tgi, trt-llm and lmdeploy), for **the public** to know when to choose vllm.
9
10 -See [vLLM performance dashboard](https://perf.vllm.ai) for the latest performance benchmark results and [vLLM GitHub README](https://github.com/vllm-project/vllm/blob/main/README.md) for latest nightly benchmark results.
10 +See [vLLM performance dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm) for the latest performance benchmark results and [vLLM GitHub README](https://github.com/vllm-project/vllm/blob/main/README.md) for latest nightly benchmark results.
11
12 ## Performance benchmark quick overview
13
@@ -138,28 +138,20 @@ The raw benchmarking results (in the format of json files) are in the `Artifacts
138
139 The `compare-json-results.py` helps to compare benchmark results JSON files converted using `convert-results-json-to-markdown.py`.
140 When run, benchmark script generates results under `benchmark/results` folder, along with the `benchmark_results.md` and `benchmark_results.json`.
141 -`compare-json-results.py` compares two `benchmark_results.json` files and provides performance ratio e.g. for Output Tput, Median TTFT and Median TPOT.
141 +`compare-json-results.py` compares two `benchmark_results.json` files and provides performance ratio e.g. for Output Tput, Median TTFT and Median TPOT.
142 +If only one benchmark_results.json is passed, `compare-json-results.py` compares different TP and PP configurations in the benchmark_results.json instead.
143
143 -Here is an example using the script to compare result_a and result_b without detail test name.
144 -`python3 compare-json-results.py -f results_a/benchmark_results.json -f results_b/benchmark_results.json --ignore_test_name`
145 -
146 -| | results_a/benchmark_results.json | results_b/benchmark_results.json | perf_ratio |
147 -|----|----------------------------------------|----------------------------------------|----------|
148 -| 0 | 142.633982 | 156.526018 | 1.097396 |
149 -| 1 | 241.620334 | 294.018783 | 1.216863 |
150 -| 2 | 218.298905 | 262.664916 | 1.203235 |
151 -| 3 | 242.743860 | 299.816190 | 1.235113 |
152 -
153 -Here is an example using the script to compare result_a and result_b with detail test name.
144 +Here is an example using the script to compare result_a and result_b with Model, Dataset name, input/output lenght, max concurrency and qps.
145 `python3 compare-json-results.py -f results_a/benchmark_results.json -f results_b/benchmark_results.json`
146
156 -| | results_a/benchmark_results.json_name | results_a/benchmark_results.json | results_b/benchmark_results.json_name | results_b/benchmark_results.json | perf_ratio |
157 -|---|---------------------------------------------|----------------------------------------|---------------------------------------------|----------------------------------------|----------|
158 -| 0 | serving_llama8B_tp1_sharegpt_qps_1 | 142.633982 | serving_llama8B_tp1_sharegpt_qps_1 | 156.526018 | 1.097396 |
159 -| 1 | serving_llama8B_tp1_sharegpt_qps_16 | 241.620334 | serving_llama8B_tp1_sharegpt_qps_16 | 294.018783 | 1.216863 |
160 -| 2 | serving_llama8B_tp1_sharegpt_qps_4 | 218.298905 | serving_llama8B_tp1_sharegpt_qps_4 | 262.664916 | 1.203235 |
161 -| 3 | serving_llama8B_tp1_sharegpt_qps_inf | 242.743860 | serving_llama8B_tp1_sharegpt_qps_inf | 299.816190 | 1.235113 |
162 -| 4 | serving_llama8B_tp2_random_1024_128_qps_1 | 96.613390 | serving_llama8B_tp4_random_1024_128_qps_1 | 108.404853 | 1.122048 |
147 +| | Model | Dataset Name | Input Len | Output Len | # of max concurrency | qps | results_a/benchmark_results.json | results_b/benchmark_results.json | perf_ratio |
148 +|----|---------------------------------------|--------|-----|-----|------|-----|-----------|----------|----------|
149 +| 0 | meta-llama/Meta-Llama-3.1-8B-Instruct | random | 128 | 128 | 1000 | 1 | 142.633982 | 156.526018 | 1.097396 |
150 +| 1 | meta-llama/Meta-Llama-3.1-8B-Instruct | random | 128 | 128 | 1000 | inf| 241.620334 | 294.018783 | 1.216863 |
151 +
152 +A comparison diagram will be generated below the table.
153 +Here is an example to compare between 96c/results_gnr_96c_091_tp2pp3 and 128c/results_gnr_128c_091_tp2pp3
154 +<img width="1886" height="828" alt="image" src="https://github.com/user-attachments/assets/c02a43ef-25d0-4fd6-90e5-2169a28682dd" />
155
156 ## Nightly test details
157
.buildkite/nightly-benchmarks/scripts/compare-json-results.py
+162 -13
@@ -1,24 +1,38 @@
1 # SPDX-License-Identifier: Apache-2.0
2 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 import argparse
4 +import json
5 +import os
6
7 import pandas as pd
8
9
10 def compare_data_columns(
9 - files, name_column, data_column, drop_column, ignore_test_name=False
11 + files, name_column, data_column, info_cols, drop_column, debug=False
12 ):
13 print("\ncompare_data_column: " + data_column)
14 frames = []
15 + raw_data_cols = []
16 compare_frames = []
17 for file in files:
18 data_df = pd.read_json(file)
19 serving_df = data_df.dropna(subset=[drop_column], ignore_index=True)
17 - if ignore_test_name is False:
20 + # Show all info columns in the first couple columns
21 + if not frames:
22 + for col in info_cols:
23 + if col not in serving_df.columns:
24 + print(f"Skipping missing column: {col}")
25 + continue
26 + frames.append(serving_df[col])
27 + # only show test name under debug mode
28 + if debug is True:
29 serving_df = serving_df.rename(columns={name_column: file + "_name"})
30 frames.append(serving_df[file + "_name"])
31 +
32 + file = "/".join(file.split("/")[:-1])
33 serving_df = serving_df.rename(columns={data_column: file})
34 frames.append(serving_df[file])
35 + raw_data_cols.append(file)
36 compare_frames.append(serving_df[file])
37 if len(compare_frames) >= 2:
38 # Compare numbers among two files
@@ -27,7 +41,68 @@ def compare_data_columns(
41 compare_frames.pop(1)
42
43 concat_df = pd.concat(frames, axis=1)
30 - return concat_df
44 + print(raw_data_cols)
45 + return concat_df, raw_data_cols
46 +
47 +
48 +def split_json_by_tp_pp(
49 + input_file: str = "benchmark_results.json", output_root: str = "."
50 +) -> list[str]:
51 + """
52 + Split a benchmark JSON into separate folders by (TP Size, PP Size).
53 +
54 + Creates: <output_root>/tp{TP}_pp{PP}/benchmark_results.json
55 + Returns: list of file paths written.
56 + """
57 + # Load JSON data into DataFrame
58 + with open(input_file, encoding="utf-8") as f:
59 + data = json.load(f)
60 +
61 + # If the JSON is a dict with a list under common keys, use that list
62 + if isinstance(data, dict):
63 + for key in ("results", "serving_results", "benchmarks", "data"):
64 + if isinstance(data.get(key), list):
65 + data = data[key]
66 + break
67 +
68 + df = pd.DataFrame(data)
69 +
70 + # Handle alias column names
71 + rename_map = {
72 + "tp_size": "TP Size",
73 + "tensor_parallel_size": "TP Size",
74 + "pp_size": "PP Size",
75 + "pipeline_parallel_size": "PP Size",
76 + }
77 + df.rename(
78 + columns={k: v for k, v in rename_map.items() if k in df.columns}, inplace=True
79 + )
80 +
81 + # Ensure TP/PP columns exist (default to 1 if missing)
82 + if "TP Size" not in df.columns:
83 + df["TP Size"] = 1
84 + if "PP Size" not in df.columns:
85 + df["PP Size"] = 1
86 +
87 + # make sure TP/PP are numeric ints with no NaN
88 + df["TP Size"] = (
89 + pd.to_numeric(df.get("TP Size", 1), errors="coerce").fillna(1).astype(int)
90 + )
91 + df["PP Size"] = (
92 + pd.to_numeric(df.get("PP Size", 1), errors="coerce").fillna(1).astype(int)
93 + )
94 +
95 + # Split into separate folders
96 + saved_paths: list[str] = []
97 + for (tp, pp), group_df in df.groupby(["TP Size", "PP Size"], dropna=False):
98 + folder_name = os.path.join(output_root, f"tp{int(tp)}_pp{int(pp)}")
99 + os.makedirs(folder_name, exist_ok=True)
100 + filepath = os.path.join(folder_name, "benchmark_results.json")
101 + group_df.to_json(filepath, orient="records", indent=2, force_ascii=False)
102 + print(f"Saved: {filepath}")
103 + saved_paths.append(filepath)
104 +
105 + return saved_paths
106
107
108 if __name__ == "__main__":
@@ -36,31 +111,105 @@ if __name__ == "__main__":
111 "-f", "--file", action="append", type=str, help="input file name"
112 )
113 parser.add_argument(
39 - "--ignore_test_name", action="store_true", help="ignore_test_name or not"
114 + "--debug", action="store_true", help="show all information for debugging"
115 + )
116 + parser.add_argument(
117 + "--plot",
118 + action=argparse.BooleanOptionalAction,
119 + default=True,
120 + help="plot perf diagrams or not --no-plot --plot",
121 + )
122 + parser.add_argument(
123 + "-x",
124 + "--xaxis",
125 + type=str,
126 + default="# of max concurrency.",
127 + help="column name to use as X Axis in comparision graph",
128 )
129 args = parser.parse_args()
42 - files = args.file
43 - print("comparing : " + ", ".join(files))
130
131 drop_column = "P99"
132 name_column = "Test name"
133 + info_cols = [
134 + "Model",
135 + "Dataset Name",
136 + "Input Len",
137 + "Output Len",
138 + "TP Size",
139 + "PP Size",
140 + "# of max concurrency.",
141 + "qps",
142 + ]
143 data_cols_to_compare = ["Output Tput (tok/s)", "Median TTFT (ms)", "Median"]
144 html_msgs_for_data_cols = [
145 "Compare Output Tokens /n",
146 "Median TTFT /n",
147 "Median TPOT /n",
148 ]
53 - ignore_test_name = args.ignore_test_name
149 +
150 + if len(args.file) == 1:
151 + files = split_json_by_tp_pp(args.file[0], output_root="splits")
152 + info_cols = [c for c in info_cols if c not in ("TP Size", "PP Size")]
153 + else:
154 + files = args.file
155 + print("comparing : " + ", ".join(files))
156 + debug = args.debug
157 + plot = args.plot
158 + # For Plot feature, assign y axis from one of info_cols
159 + y_axis_index = info_cols.index(args.xaxis) if args.xaxis in info_cols else 6
160 with open("perf_comparison.html", "w") as text_file:
161 for i in range(len(data_cols_to_compare)):
56 - output_df = compare_data_columns(
162 + output_df, raw_data_cols = compare_data_columns(
163 files,
164 name_column,
165 data_cols_to_compare[i],
166 + info_cols,
167 drop_column,
61 - ignore_test_name=ignore_test_name,
168 + debug=debug,
169 )
63 - print(output_df)
64 - html = output_df.to_html()
65 - text_file.write(html_msgs_for_data_cols[i])
66 - text_file.write(html)
170 +
171 + # For Plot feature, insert y axis from one of info_cols
172 + raw_data_cols.insert(0, info_cols[y_axis_index])
173 +
174 + filtered_info_cols = info_cols[:-2]
175 + existing_group_cols = [
176 + c for c in filtered_info_cols if c in output_df.columns
177 + ]
178 + if not existing_group_cols:
179 + raise ValueError(
180 + f"No valid group-by columns "
181 + f"Expected subset: {filtered_info_cols}, "
182 + f"but DataFrame has: {list(output_df.columns)}"
183 + )
184 +
185 + output_df_sorted = output_df.sort_values(by=existing_group_cols)
186 + output_groups = output_df_sorted.groupby(existing_group_cols, dropna=False)
187 + for name, group in output_groups:
188 + html = group.to_html()
189 + text_file.write(html_msgs_for_data_cols[i])
190 + text_file.write(html)
191 +
192 + if plot is True:
193 + import pandas as pd
194 + import plotly.express as px
195 +
196 + df = group[raw_data_cols]
197 + df_sorted = df.sort_values(by=info_cols[y_axis_index])
198 + # Melt DataFrame for plotting
199 + df_melted = df_sorted.melt(
200 + id_vars=info_cols[y_axis_index],
201 + var_name="Configuration",
202 + value_name=data_cols_to_compare[i],
203 + )
204 + title = data_cols_to_compare[i] + " vs " + info_cols[y_axis_index]
205 + # Create Plotly line chart
206 + fig = px.line(
207 + df_melted,
208 + x=info_cols[y_axis_index],
209 + y=data_cols_to_compare[i],
210 + color="Configuration",
211 + title=title,
212 + markers=True,
213 + )
214 + # Export to HTML
215 + text_file.write(fig.to_html(full_html=True, include_plotlyjs="cdn"))
.buildkite/nightly-benchmarks/scripts/convert-results-json-to-markdown.py
+153 -10
@@ -1,17 +1,19 @@
1 # SPDX-License-Identifier: Apache-2.0
2 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
4 +import argparse
5 import json
6 import os
7 +import re
8 +import shlex
9 from importlib import util
10 from pathlib import Path
11 +from typing import Any
12
13 import pandas as pd
14 import psutil
15 from tabulate import tabulate
16
13 -results_folder = Path("results/")
14 -
17 # latency results and the keys that will be printed into markdown
18 latency_results = []
19 latency_column_mapping = {
@@ -42,14 +44,22 @@ throughput_results_column_mapping = {
44 serving_results = []
45 serving_column_mapping = {
46 "test_name": "Test name",
47 + "model_id": "Model",
48 + "dataset_name": "Dataset Name",
49 + "input_len": "Input Len",
50 + "output_len": "Output Len",
51 + "tp_size": "TP Size",
52 + "pp_size": "PP Size",
53 + "dtype": "dtype",
54 "gpu_type": "GPU",
55 "completed": "# of req.",
56 + "qps": "qps",
57 "max_concurrency": "# of max concurrency.",
58 "request_throughput": "Tput (req/s)",
59 "total_token_throughput": "Total Token Tput (tok/s)",
60 "output_throughput": "Output Tput (tok/s)",
51 - "total_input_tokens": "Total input tokens",
52 - "total_output_tokens": "Total output tokens",
61 + # "total_input_tokens": "Total input tokens",
62 + # "total_output_tokens": "Total output tokens",
63 "mean_ttft_ms": "Mean TTFT (ms)",
64 "median_ttft_ms": "Median TTFT (ms)",
65 "p99_ttft_ms": "P99 TTFT (ms)",
@@ -94,7 +104,104 @@ def get_size_with_unit(bytes, suffix="B"):
104 bytes /= factor
105
106
107 +def _coerce(val: str) -> Any:
108 + """Best-effort type coercion from string to Python types."""
109 + low = val.lower()
110 + if low == "null":
111 + return None
112 + if low == "true":
113 + return True
114 + if low == "false":
115 + return False
116 + # integers
117 + if re.fullmatch(r"[+-]?\d+", val):
118 + try:
119 + return int(val)
120 + except ValueError:
121 + pass
122 + # floats (keep 'inf'/'-inf'/'nan' as strings)
123 + if re.fullmatch(r"[+-]?\d*\.\d+", val):
124 + try:
125 + return float(val)
126 + except ValueError:
127 + pass
128 + return val
129 +
130 +
131 +def parse_client_command(cmd: str) -> dict[str, Any]:
132 + """Parse the client_command shell string into {executable, script, args}."""
133 + toks = shlex.split(cmd)
134 + if len(toks) < 2:
135 + raise ValueError("client_command must include an executable and a script")
136 + executable, script = toks[0], toks[1]
137 + args: dict[str, Any] = {}
138 +
139 + i = 2
140 + while i < len(toks):
141 + t = toks[i]
142 + if t.startswith("--"):
143 + # --key=value or --key (value) or boolean flag
144 + if "=" in t:
145 + key, val = t.split("=", 1)
146 + if key == "--metadata":
147 + md = {}
148 + if val:
149 + if "=" in val:
150 + k, v = val.split("=", 1)
151 + md[k] = _coerce(v)
152 + else:
153 + md[val] = True
154 + args[key] = md
155 + else:
156 + args[key] = _coerce(val)
157 + i += 1
158 + continue
159 +
160 + key = t
161 +
162 + # Special: consume metadata k=v pairs until next --flag
163 + if key == "--metadata":
164 + i += 1
165 + md = {}
166 + while i < len(toks) and not toks[i].startswith("--"):
167 + pair = toks[i]
168 + if "=" in pair:
169 + k, v = pair.split("=", 1)
170 + md[k] = _coerce(v)
171 + else:
172 + md[pair] = True
173 + i += 1
174 + args[key] = md
175 + continue
176 +
177 + # Standard: check if next token is a value (not a flag)
178 + if i + 1 < len(toks) and not toks[i + 1].startswith("--"):
179 + args[key] = _coerce(toks[i + 1])
180 + i += 2
181 + else:
182 + # lone flag -> True
183 + args[key] = True
184 + i += 1
185 + else:
186 + # unexpected positional; skip
187 + i += 1
188 +
189 + return {"executable": executable, "script": script, "args": args}
190 +
191 +
192 if __name__ == "__main__":
193 + parser = argparse.ArgumentParser()
194 + parser.add_argument(
195 + "-r",
196 + "--result",
197 + type=str,
198 + default="results",
199 + help="Folder name for benchmark output results.",
200 + )
201 + args = parser.parse_args()
202 + results_folder = Path(args.result)
203 + if not results_folder.exists():
204 + raise FileNotFoundError(f"results folder does not exist: {results_folder}")
205 # collect results
206 for test_file in results_folder.glob("*.json"):
207 with open(test_file) as f:
@@ -102,7 +209,6 @@ if __name__ == "__main__":
209
210 if "serving" in str(test_file):
211 # this result is generated via `vllm bench serve` command
105 -
212 # attach the benchmarking command to raw_result
213 try:
214 with open(test_file.with_suffix(".commands")) as f:
@@ -110,12 +216,44 @@ if __name__ == "__main__":
216 except OSError as e:
217 print(e)
218 continue
113 -
219 + # Parse Server Command Arg
220 + out: dict[str, Any] = {
221 + "server_command": parse_client_command(command["server_command"])
222 + }
223 + parse_args = [
224 + "--tensor-parallel-size",
225 + "--pipeline-parallel-size",
226 + "--dtype",
227 + ]
228 + col_mapping = ["tp_size", "pp_size", "dtype"]
229 + for index, arg in enumerate(parse_args):
230 + if arg in out["server_command"]["args"]:
231 + raw_result.update(
232 + {col_mapping[index]: out["server_command"]["args"][arg]}
233 + )
234 +
235 + # Parse Client Command Arg
236 + out: dict[str, Any] = {
237 + "client_command": parse_client_command(command["client_command"])
238 + }
239 + parse_args = [
240 + "--dataset-name",
241 + "--random-input-len",
242 + "--random-output-len",
243 + "--request-rate",
244 + ]
245 + col_mapping = ["dataset_name", "input_len", "output_len", "qps"]
246 +
247 + for index, arg in enumerate(parse_args):
248 + if arg in out["client_command"]["args"]:
249 + raw_result.update(
250 + {col_mapping[index]: out["client_command"]["args"][arg]}
251 + )
252 + # Add Server, Client command
253 raw_result.update(command)
254
255 # update the test name of this result
256 raw_result.update({"test_name": test_file.stem})
118 -
257 # add the result to raw_result
258 serving_results.append(raw_result)
259 continue
@@ -205,7 +343,10 @@ if __name__ == "__main__":
343 columns=latency_column_mapping
344 )
345 if not serving_results.empty:
208 - serving_results = serving_results[list(serving_column_mapping.keys())].rename(
346 + valid_columns = [
347 + col for col in serving_column_mapping if col in serving_results.columns
348 + ]
349 + serving_results = serving_results[valid_columns].rename(
350 columns=serving_column_mapping
351 )
352 if not throughput_results.empty:
@@ -245,7 +386,9 @@ if __name__ == "__main__":
386 )
387
388 # document the result
248 - with open(results_folder / "benchmark_results.md", "w") as f:
389 + md_file = "benchmark_results.md"
390 + json_file = "benchmark_results.json"
391 + with open(results_folder / md_file, "w") as f:
392 results = read_markdown(
393 "../.buildkite/nightly-benchmarks/"
394 + "performance-benchmarks-descriptions.md"
@@ -260,7 +403,7 @@ if __name__ == "__main__":
403 f.write(results)
404
405 # document benchmarking results in json
263 - with open(results_folder / "benchmark_results.json", "w") as f:
406 + with open(results_folder / json_file, "w") as f:
407 results = (
408 latency_results.to_dict(orient="records")
409 + throughput_results.to_dict(orient="records")
.buildkite/nightly-benchmarks/scripts/run-performance-benchmarks.sh
+55 -38
@@ -194,9 +194,11 @@ run_latency_tests() {
194
195 # check if there is enough GPU to run the test
196 tp=$(echo "$latency_params" | jq -r '.tensor_parallel_size')
197 - if [ "$ON_CPU" == "1" ];then
198 - if [[ $numa_count -lt $tp ]]; then
199 - echo "Required tensor-parallel-size $tp but only $numa_count NUMA nodes found. Skip testcase $test_name."
197 + if [ "$ON_CPU" == "1" ]; then
198 + pp=$(echo "$latency_params" | jq -r '.pipeline_parallel_size')
199 + world_size=$(($tp*$pp))
200 + if [[ $numa_count -lt $world_size && -z "${REMOTE_HOST}" ]]; then
201 + echo "Required world-size $world_size but only $numa_count NUMA nodes found. Skip testcase $test_name."
202 continue
203 fi
204 else
@@ -261,9 +263,11 @@ run_throughput_tests() {
263
264 # check if there is enough GPU to run the test
265 tp=$(echo "$throughput_params" | jq -r '.tensor_parallel_size')
264 - if [ "$ON_CPU" == "1" ];then
265 - if [[ $numa_count -lt $tp ]]; then
266 - echo "Required tensor-parallel-size $tp but only $numa_count NUMA nodes found. Skip testcase $test_name."
266 + if [ "$ON_CPU" == "1" ]; then
267 + pp=$(echo "$throughput_params" | jq -r '.pipeline_parallel_size')
268 + world_size=$(($tp*$pp))
269 + if [[ $numa_count -lt $world_size && -z "${REMOTE_HOST}" ]]; then
270 + echo "Required world-size $world_size but only $numa_count NUMA nodes found. Skip testcase $test_name."
271 continue
272 fi
273 else
@@ -329,12 +333,21 @@ run_serving_tests() {
333 qps_list=$(echo "$params" | jq -r '.qps_list')
334 qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
335 echo "Running over qps list $qps_list"
336 + max_concurrency_list=$(echo "$params" | jq -r '.max_concurrency_list')
337 + if [[ -z "$max_concurrency_list" || "$max_concurrency_list" == "null" ]]; then
338 + num_prompts=$(echo "$client_params" | jq -r '.num_prompts')
339 + max_concurrency_list="[$num_prompts]"
340 + fi
341 + max_concurrency_list=$(echo "$max_concurrency_list" | jq -r '.[] | @sh')
342 + echo "Running over max concurrency list $max_concurrency_list"
343
344 # check if there is enough resources to run the test
345 tp=$(echo "$server_params" | jq -r '.tensor_parallel_size')
335 - if [ "$ON_CPU" == "1" ];then
336 - if [[ $numa_count -lt $tp ]]; then
337 - echo "Required tensor-parallel-size $tp but only $numa_count NUMA nodes found. Skip testcase $test_name."
346 + if [ "$ON_CPU" == "1" ]; then
347 + pp=$(echo "$server_params" | jq -r '.pipeline_parallel_size')
348 + world_size=$(($tp*$pp))
349 + if [[ $numa_count -lt $world_size && -z "${REMOTE_HOST}" ]]; then
350 + echo "Required world-size $world_size but only $numa_count NUMA nodes found. Skip testcase $test_name."
351 continue
352 fi
353 else
@@ -390,35 +403,39 @@ run_serving_tests() {
403 echo "now qps is $qps"
404 fi
405
393 - new_test_name=$test_name"_qps_"$qps
394 -
395 - # pass the tensor parallel size to the client so that it can be displayed
396 - # on the benchmark dashboard
397 - client_command="vllm bench serve \
398 - --save-result \
399 - --result-dir $RESULTS_FOLDER \
400 - --result-filename ${new_test_name}.json \
401 - --request-rate $qps \
402 - --metadata "tensor_parallel_size=$tp" \
403 - $client_args $client_remote_args "
404 -
405 - echo "Running test case $test_name with qps $qps"
406 - echo "Client command: $client_command"
407 -
408 - bash -c "$client_command"
409 -
410 - # record the benchmarking commands
411 - jq_output=$(jq -n \
412 - --arg server "$server_command" \
413 - --arg client "$client_command" \
414 - --arg gpu "$gpu_type" \
415 - '{
416 - server_command: $server,
417 - client_command: $client,
418 - gpu_type: $gpu
419 - }')
420 - echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
421 -
406 + # iterate over different max_concurrency
407 + for max_concurrency in $max_concurrency_list; do
408 + new_test_name=$test_name"_qps_"$qps"_concurrency_"$max_concurrency
409 + echo " new test name $new_test_name"
410 + # pass the tensor parallel size to the client so that it can be displayed
411 + # on the benchmark dashboard
412 + client_command="vllm bench serve \
413 + --save-result \
414 + --result-dir $RESULTS_FOLDER \
415 + --result-filename ${new_test_name}.json \
416 + --request-rate $qps \
417 + --max-concurrency $max_concurrency \
418 + --metadata "tensor_parallel_size=$tp" \
419 + $client_args $client_remote_args "
420 +
421 + echo "Running test case $test_name with qps $qps"
422 + echo "Client command: $client_command"
423 +
424 + bash -c "$client_command"
425 +
426 + # record the benchmarking commands
427 + jq_output=$(jq -n \
428 + --arg server "$server_command" \
429 + --arg client "$client_command" \
430 + --arg gpu "$gpu_type" \
431 + '{
432 + server_command: $server,
433 + client_command: $client,
434 + gpu_type: $gpu
435 + }')
436 + echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
437 +
438 + done
439 done
440
441 # clean up
.buildkite/nightly-benchmarks/tests/latency-tests-cpu.json
+2 -2
@@ -6,7 +6,7 @@
6 "VLLM_CPU_KVCACHE_SPACE": 40
7 },
8 "parameters": {
9 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
9 + "model": "meta-llama/Llama-3.1-8B-Instruct",
10 "tensor_parallel_size": 1,
11 "load_format": "dummy",
12 "num_iters_warmup": 5,
@@ -20,7 +20,7 @@
20 "VLLM_CPU_KVCACHE_SPACE": 40
21 },
22 "parameters": {
23 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
23 + "model": "meta-llama/Llama-3.1-8B-Instruct",
24 "tensor_parallel_size": 4,
25 "load_format": "dummy",
26 "num_iters_warmup": 5,
.buildkite/nightly-benchmarks/tests/serving-tests-cpu-snc2.json
+24 -25
@@ -1,7 +1,8 @@
1 [
2 {
3 "test_name": "serving_llama8B_tp1_sharegpt",
4 - "qps_list": [1, 4, 16, "inf"],
4 + "qps_list": ["inf"],
5 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
6 "server_environment_variables": {
7 "VLLM_RPC_TIMEOUT": 100000,
8 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -10,7 +11,7 @@
11 "VLLM_CPU_KVCACHE_SPACE": 40
12 },
13 "server_parameters": {
13 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
14 + "model": "meta-llama/Llama-3.1-8B-Instruct",
15 "tensor_parallel_size": 1,
16 "dtype": "bfloat16",
17 "distributed_executor_backend": "mp",
@@ -23,17 +24,17 @@
24 "load_format": "dummy"
25 },
26 "client_parameters": {
26 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
27 + "model": "meta-llama/Llama-3.1-8B-Instruct",
28 "backend": "vllm",
29 "dataset_name": "sharegpt",
30 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
30 - "max_concurrency": 60,
31 "num_prompts": 200
32 }
33 },
34 {
35 "test_name": "serving_llama8B_tp2_sharegpt",
36 - "qps_list": [1, 4, 16, "inf"],
36 + "qps_list": ["inf"],
37 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
38 "server_environment_variables": {
39 "VLLM_RPC_TIMEOUT": 100000,
40 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -42,7 +43,7 @@
43 "VLLM_CPU_KVCACHE_SPACE": 40
44 },
45 "server_parameters": {
45 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
46 + "model": "meta-llama/Llama-3.1-8B-Instruct",
47 "tensor_parallel_size": 2,
48 "dtype": "bfloat16",
49 "distributed_executor_backend": "mp",
@@ -55,17 +56,17 @@
56 "load_format": "dummy"
57 },
58 "client_parameters": {
58 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
59 + "model": "meta-llama/Llama-3.1-8B-Instruct",
60 "backend": "vllm",
61 "dataset_name": "sharegpt",
62 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
62 - "max_concurrency": 60,
63 "num_prompts": 200
64 }
65 },
66 {
67 "test_name": "serving_llama8B_tp4_sharegpt",
68 - "qps_list": [1, 4, 16, "inf"],
68 + "qps_list": ["inf"],
69 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
70 "server_environment_variables": {
71 "VLLM_RPC_TIMEOUT": 100000,
72 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -74,7 +75,7 @@
75 "VLLM_CPU_KVCACHE_SPACE": 40
76 },
77 "server_parameters": {
77 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
78 + "model": "meta-llama/Llama-3.1-8B-Instruct",
79 "tensor_parallel_size": 4,
80 "dtype": "bfloat16",
81 "distributed_executor_backend": "mp",
@@ -87,17 +88,17 @@
88 "load_format": "dummy"
89 },
90 "client_parameters": {
90 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
91 + "model": "meta-llama/Llama-3.1-8B-Instruct",
92 "backend": "vllm",
93 "dataset_name": "sharegpt",
94 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
94 - "max_concurrency": 60,
95 "num_prompts": 200
96 }
97 },
98 {
99 "test_name": "serving_llama8B_tp1_random_128_128",
100 - "qps_list": [1, 4, 16, "inf"],
100 + "qps_list": ["inf"],
101 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
102 "server_environment_variables": {
103 "VLLM_RPC_TIMEOUT": 100000,
104 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -106,7 +107,7 @@
107 "VLLM_CPU_KVCACHE_SPACE": 40
108 },
109 "server_parameters": {
109 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
110 + "model": "meta-llama/Llama-3.1-8B-Instruct",
111 "tensor_parallel_size": 1,
112 "dtype": "bfloat16",
113 "distributed_executor_backend": "mp",
@@ -120,19 +121,19 @@
121 "load_format": "dummy"
122 },
123 "client_parameters": {
123 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
124 + "model": "meta-llama/Llama-3.1-8B-Instruct",
125 "backend": "vllm",
126 "dataset_name": "random",
127 "random-input-len": 128,
128 "random-output-len": 128,
129 "ignore-eos": "",
129 - "max_concurrency": 1000,
130 "num_prompts": 1000
131 }
132 },
133 {
134 "test_name": "serving_llama8B_tp2_random_128_128",
135 - "qps_list": [1, 4, 16, "inf"],
135 + "qps_list": ["inf"],
136 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
137 "server_environment_variables": {
138 "VLLM_RPC_TIMEOUT": 100000,
139 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -141,7 +142,7 @@
142 "VLLM_CPU_KVCACHE_SPACE": 40
143 },
144 "server_parameters": {
144 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
145 + "model": "meta-llama/Llama-3.1-8B-Instruct",
146 "tensor_parallel_size": 2,
147 "dtype": "bfloat16",
148 "distributed_executor_backend": "mp",
@@ -155,19 +156,19 @@
156 "load_format": "dummy"
157 },
158 "client_parameters": {
158 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
159 + "model": "meta-llama/Llama-3.1-8B-Instruct",
160 "backend": "vllm",
161 "dataset_name": "random",
162 "random-input-len": 128,
163 "random-output-len": 128,
164 "ignore-eos": "",
164 - "max_concurrency": 1000,
165 "num_prompts": 1000
166 }
167 },
168 {
169 "test_name": "serving_llama8B_tp4_random_128_128",
170 - "qps_list": [1, 4, 16, "inf"],
170 + "qps_list": ["inf"],
171 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
172 "server_environment_variables": {
173 "VLLM_RPC_TIMEOUT": 100000,
174 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -176,7 +177,7 @@
177 "VLLM_CPU_KVCACHE_SPACE": 40
178 },
179 "server_parameters": {
179 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
180 + "model": "meta-llama/Llama-3.1-8B-Instruct",
181 "tensor_parallel_size": 4,
182 "dtype": "bfloat16",
183 "distributed_executor_backend": "mp",
@@ -190,13 +191,11 @@
191 "load_format": "dummy"
192 },
193 "client_parameters": {
193 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
194 + "model": "meta-llama/Llama-3.1-8B-Instruct",
195 "backend": "vllm",
196 "dataset_name": "random",
197 "random-input-len": 128,
198 "random-output-len": 128,
198 - "ignore-eos": "",
199 - "max_concurrency": 1000,
199 "num_prompts": 1000
200 }
201 }
.buildkite/nightly-benchmarks/tests/serving-tests-cpu-snc3.json
+26 -26
@@ -1,7 +1,8 @@
1 [
2 {
3 "test_name": "serving_llama8B_pp1_sharegpt",
4 - "qps_list": [1, 4, 16, "inf"],
4 + "qps_list": ["inf"],
5 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
6 "server_environment_variables": {
7 "VLLM_RPC_TIMEOUT": 100000,
8 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -10,7 +11,7 @@
11 "VLLM_CPU_KVCACHE_SPACE": 40
12 },
13 "server_parameters": {
13 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
14 + "model": "meta-llama/Llama-3.1-8B-Instruct",
15 "pipeline_parallel_size": 1,
16 "dtype": "bfloat16",
17 "distributed_executor_backend": "mp",
@@ -23,17 +24,17 @@
24 "load_format": "dummy"
25 },
26 "client_parameters": {
26 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
27 + "model": "meta-llama/Llama-3.1-8B-Instruct",
28 "backend": "vllm",
29 "dataset_name": "sharegpt",
30 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
30 - "max_concurrency": 60,
31 "num_prompts": 200
32 }
33 },
34 {
35 "test_name": "serving_llama8B_pp3_sharegpt",
36 - "qps_list": [1, 4, 16, "inf"],
36 + "qps_list": ["inf"],
37 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
38 "server_environment_variables": {
39 "VLLM_RPC_TIMEOUT": 100000,
40 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -42,7 +43,7 @@
43 "VLLM_CPU_KVCACHE_SPACE": 40
44 },
45 "server_parameters": {
45 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
46 + "model": "meta-llama/Llama-3.1-8B-Instruct",
47 "pipeline_parallel_size": 3,
48 "dtype": "bfloat16",
49 "distributed_executor_backend": "mp",
@@ -55,17 +56,17 @@
56 "load_format": "dummy"
57 },
58 "client_parameters": {
58 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
59 + "model": "meta-llama/Llama-3.1-8B-Instruct",
60 "backend": "vllm",
61 "dataset_name": "sharegpt",
62 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
62 - "max_concurrency": 60,
63 "num_prompts": 200
64 }
65 },
66 {
67 - "test_name": "serving_llama8B_tp2pp6_sharegpt",
68 - "qps_list": [1, 4, 16, "inf"],
67 + "test_name": "serving_llama8B_tp2pp3_sharegpt",
68 + "qps_list": ["inf"],
69 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
70 "server_environment_variables": {
71 "VLLM_RPC_TIMEOUT": 100000,
72 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -74,7 +75,7 @@
75 "VLLM_CPU_KVCACHE_SPACE": 40
76 },
77 "server_parameters": {
77 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
78 + "model": "meta-llama/Llama-3.1-8B-Instruct",
79 "tensor_parallel_size": 2,
80 "pipeline_parallel_size": 3,
81 "dtype": "bfloat16",
@@ -88,17 +89,17 @@
89 "load_format": "dummy"
90 },
91 "client_parameters": {
91 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
92 + "model": "meta-llama/Llama-3.1-8B-Instruct",
93 "backend": "vllm",
94 "dataset_name": "sharegpt",
95 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
95 - "max_concurrency": 60,
96 "num_prompts": 200
97 }
98 },
99 {
100 "test_name": "serving_llama8B_pp1_random_128_128",
101 - "qps_list": [1, 4, 16, "inf"],
101 + "qps_list": ["inf"],
102 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
103 "server_environment_variables": {
104 "VLLM_RPC_TIMEOUT": 100000,
105 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -107,7 +108,7 @@
108 "VLLM_CPU_KVCACHE_SPACE": 40
109 },
110 "server_parameters": {
110 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
111 + "model": "meta-llama/Llama-3.1-8B-Instruct",
112 "pipeline_parallel_size": 1,
113 "dtype": "bfloat16",
114 "distributed_executor_backend": "mp",
@@ -121,28 +122,28 @@
122 "load_format": "dummy"
123 },
124 "client_parameters": {
124 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
125 + "model": "meta-llama/Llama-3.1-8B-Instruct",
126 "backend": "vllm",
127 "dataset_name": "random",
128 "random-input-len": 128,
129 "random-output-len": 128,
130 "ignore-eos": "",
130 - "max_concurrency": 1000,
131 "num_prompts": 1000
132 }
133 },
134 {
135 "test_name": "serving_llama8B_pp3_random_128_128",
136 - "qps_list": [1, 4, 16, "inf"],
136 + "qps_list": ["inf"],
137 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
138 "server_environment_variables": {
139 "VLLM_RPC_TIMEOUT": 100000,
140 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
141 "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
141 - "VLLM_CPU_SGL_KERNEL:": 1,
142 + "VLLM_CPU_SGL_KERNEL": 1,
143 "VLLM_CPU_KVCACHE_SPACE": 40
144 },
145 "server_parameters": {
145 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
146 + "model": "meta-llama/Llama-3.1-8B-Instruct",
147 "pipeline_parallel_size": 3,
148 "dtype": "bfloat16",
149 "distributed_executor_backend": "mp",
@@ -156,19 +157,19 @@
157 "load_format": "dummy"
158 },
159 "client_parameters": {
159 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
160 + "model": "meta-llama/Llama-3.1-8B-Instruct",
161 "backend": "vllm",
162 "dataset_name": "random",
163 "random-input-len": 128,
164 "random-output-len": 128,
165 "ignore-eos": "",
165 - "max_concurrency": 1000,
166 "num_prompts": 1000
167 }
168 },
169 {
170 "test_name": "serving_llama8B_tp2pp3_random_128_128",
171 - "qps_list": [1, 4, 16, "inf"],
171 + "qps_list": ["inf"],
172 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
173 "server_environment_variables": {
174 "VLLM_RPC_TIMEOUT": 100000,
175 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -177,7 +178,7 @@
178 "VLLM_CPU_KVCACHE_SPACE": 40
179 },
180 "server_parameters": {
180 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
181 + "model": "meta-llama/Llama-3.1-8B-Instruct",
182 "tensor_parallel_size": 2,
183 "pipeline_parallel_size": 3,
184 "dtype": "bfloat16",
@@ -192,13 +193,12 @@
193 "load_format": "dummy"
194 },
195 "client_parameters": {
195 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
196 + "model": "meta-llama/Llama-3.1-8B-Instruct",
197 "backend": "vllm",
198 "dataset_name": "random",
199 "random-input-len": 128,
200 "random-output-len": 128,
201 "ignore-eos": "",
201 - "max_concurrency": 1000,
202 "num_prompts": 1000
203 }
204 }
.buildkite/nightly-benchmarks/tests/serving-tests-cpu.json
+15 -15
@@ -2,6 +2,7 @@
2 {
3 "test_name": "serving_llama8B_tp1_sharegpt",
4 "qps_list": [1, 4, 16, "inf"],
5 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
6 "server_environment_variables": {
7 "VLLM_RPC_TIMEOUT": 100000,
8 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -10,7 +11,7 @@
11 "VLLM_CPU_KVCACHE_SPACE": 40
12 },
13 "server_parameters": {
13 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
14 + "model": "meta-llama/Llama-3.1-8B-Instruct",
15 "tensor_parallel_size": 1,
16 "dtype": "bfloat16",
17 "distributed_executor_backend": "mp",
@@ -23,17 +24,17 @@
24 "load_format": "dummy"
25 },
26 "client_parameters": {
26 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
27 + "model": "meta-llama/Llama-3.1-8B-Instruct",
28 "backend": "vllm",
29 "dataset_name": "sharegpt",
30 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
30 - "max_concurrency": 60,
31 "num_prompts": 200
32 }
33 },
34 {
35 "test_name": "serving_llama8B_tp2_sharegpt",
36 "qps_list": [1, 4, 16, "inf"],
37 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
38 "server_environment_variables": {
39 "VLLM_RPC_TIMEOUT": 100000,
40 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -42,7 +43,7 @@
43 "VLLM_CPU_KVCACHE_SPACE": 40
44 },
45 "server_parameters": {
45 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
46 + "model": "meta-llama/Llama-3.1-8B-Instruct",
47 "tensor_parallel_size": 2,
48 "dtype": "bfloat16",
49 "distributed_executor_backend": "mp",
@@ -55,17 +56,17 @@
56 "load_format": "dummy"
57 },
58 "client_parameters": {
58 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
59 + "model": "meta-llama/Llama-3.1-8B-Instruct",
60 "backend": "vllm",
61 "dataset_name": "sharegpt",
62 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
62 - "max_concurrency": 60,
63 "num_prompts": 200
64 }
65 },
66 {
67 "test_name": "serving_llama8B_tp4_sharegpt",
68 "qps_list": [1, 4, 16, "inf"],
69 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
70 "server_environment_variables": {
71 "VLLM_RPC_TIMEOUT": 100000,
72 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -74,7 +75,7 @@
75 "VLLM_CPU_KVCACHE_SPACE": 40
76 },
77 "server_parameters": {
77 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
78 + "model": "meta-llama/Llama-3.1-8B-Instruct",
79 "tensor_parallel_size": 4,
80 "dtype": "bfloat16",
81 "distributed_executor_backend": "mp",
@@ -87,17 +88,17 @@
88 "load_format": "dummy"
89 },
90 "client_parameters": {
90 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
91 + "model": "meta-llama/Llama-3.1-8B-Instruct",
92 "backend": "vllm",
93 "dataset_name": "sharegpt",
94 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
94 - "max_concurrency": 60,
95 "num_prompts": 200
96 }
97 },
98 {
99 "test_name": "serving_llama8B_tp4_random_1024_128",
100 "qps_list": [1, 4, 16, "inf"],
101 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
102 "server_environment_variables": {
103 "VLLM_RPC_TIMEOUT": 100000,
104 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -106,7 +107,7 @@
107 "VLLM_CPU_KVCACHE_SPACE": 40
108 },
109 "server_parameters": {
109 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
110 + "model": "meta-llama/Llama-3.1-8B-Instruct",
111 "tensor_parallel_size": 4,
112 "dtype": "bfloat16",
113 "distributed_executor_backend": "mp",
@@ -120,19 +121,19 @@
121 "load_format": "dummy"
122 },
123 "client_parameters": {
123 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
124 + "model": "meta-llama/Llama-3.1-8B-Instruct",
125 "backend": "vllm",
126 "dataset_name": "random",
127 "random-input-len": 1024,
128 "random-output-len": 128,
129 "ignore-eos": "",
129 - "max_concurrency": 100,
130 "num_prompts": 100
131 }
132 },
133 {
134 "test_name": "serving_llama8B_pp6_random_1024_128",
135 "qps_list": [1, 4, 16, "inf"],
136 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
137 "server_environment_variables": {
138 "VLLM_RPC_TIMEOUT": 100000,
139 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
@@ -141,7 +142,7 @@
142 "VLLM_CPU_KVCACHE_SPACE": 40
143 },
144 "server_parameters": {
144 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
145 + "model": "meta-llama/Llama-3.1-8B-Instruct",
146 "pipeline_parallel_size": 6,
147 "dtype": "bfloat16",
148 "distributed_executor_backend": "mp",
@@ -155,13 +156,12 @@
156 "load_format": "dummy"
157 },
158 "client_parameters": {
158 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
159 + "model": "meta-llama/Llama-3.1-8B-Instruct",
160 "backend": "vllm",
161 "dataset_name": "random",
162 "random-input-len": 1024,
163 "random-output-len": 128,
164 "ignore-eos": "",
164 - "max_concurrency": 100,
165 "num_prompts": 100
166 }
167 }
.buildkite/nightly-benchmarks/tests/throughput-tests-cpu.json
+2 -2
@@ -6,7 +6,7 @@
6 "VLLM_CPU_KVCACHE_SPACE": 40
7 },
8 "parameters": {
9 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
9 + "model": "meta-llama/Llama-3.1-8B-Instruct",
10 "tensor_parallel_size": 1,
11 "load_format": "dummy",
12 "dataset": "./ShareGPT_V3_unfiltered_cleaned_split.json",
@@ -21,7 +21,7 @@
21 "VLLM_CPU_KVCACHE_SPACE": 40
22 },
23 "parameters": {
24 - "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
24 + "model": "meta-llama/Llama-3.1-8B-Instruct",
25 "tensor_parallel_size": 4,
26 "load_format": "dummy",
27 "dataset": "./ShareGPT_V3_unfiltered_cleaned_split.json",
docs/contributing/benchmarks.md
+1 -1
@@ -11,7 +11,7 @@ vLLM contains two sets of benchmarks:
11
12 The performance benchmarks are used for development to confirm whether new changes improve performance under various workloads. They are triggered on every commit with both the `perf-benchmarks` and `ready` labels, and when a PR is merged into vLLM.
13
14 -The latest performance results are hosted on the public [vLLM Performance Dashboard](https://perf.vllm.ai).
14 +The latest performance results are hosted on the public [vLLM Performance Dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm).
15
16 More information on the performance benchmarks and their parameters can be found [here](gh-file:.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md).
17