@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / f15185fbdb

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 10, 22:45 UTC

[Benchmark Suite] improve cpu Benchmark Suite tests and comparison report for 0.12.0 (#30994)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com>

Louie Tsai committed Dec 25, 2025 at 00:51 UTC f15185fbdbc48d32315a1fa3c8f4b579eb68184e

4 files changed +669 -225

.buildkite/performance-benchmarks/README.md

+3 -16

@@ -176,19 +176,6 @@ If you do not see the table, please wait till the benchmark finish running.

176 The json version of the table (together with the json version of the benchmark) will be also attached to the markdown file.

177 The raw benchmarking results (in the format of json files) are in the `Artifacts` tab of the benchmarking.

178

179 -The `compare-json-results.py` helps to compare benchmark results JSON files converted using `convert-results-json-to-markdown.py`.

180 -When run, benchmark script generates results under `benchmark/results` folder, along with the `benchmark_results.md` and `benchmark_results.json`.

181 -`compare-json-results.py` compares two `benchmark_results.json` files and provides performance ratio e.g. for Output Tput, Median TTFT and Median TPOT.

182 -If only one benchmark_results.json is passed, `compare-json-results.py` compares different TP and PP configurations in the benchmark_results.json instead.

183 -

184 -Here is an example using the script to compare result_a and result_b with Model, Dataset name, input/output length, max concurrency and qps.

185 -`python3 compare-json-results.py -f results_a/benchmark_results.json -f results_b/benchmark_results.json`

186 -

188 -|----|---------------------------------------|--------|-----|-----|------|-----|-----------|----------|----------|

189 -| 0 | meta-llama/Meta-Llama-3.1-8B-Instruct | random | 128 | 128 | 1000 | 1 | 142.633982 | 156.526018 | 1.097396 |

190 -| 1 | meta-llama/Meta-Llama-3.1-8B-Instruct | random | 128 | 128 | 1000 | inf| 241.620334 | 294.018783 | 1.216863 |

191 -

192 -A comparison diagram will be generated below the table.

193 -Here is an example to compare between 96c/results_gnr_96c_091_tp2pp3 and 128c/results_gnr_128c_091_tp2pp3

194 -<img width="1886" height="828" alt="image" src="https://github.com/user-attachments/assets/c02a43ef-25d0-4fd6-90e5-2169a28682dd" />

179 +#### Performance Results Comparison

180 +

181 +Follow the instructions in [performance results comparison](https://docs.vllm.ai/en/latest/benchmarking/dashboard/#performance-results-comparison) to analyze performance results and the sizing guide.

.buildkite/performance-benchmarks/scripts/compare-json-results.py

+574 -205

@@ -1,8 +1,13 @@

1 # SPDX-License-Identifier: Apache-2.0

2 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +

4 +from __future__ import annotations

5 +

6 import argparse

7 +import html as _html

8 import json

9 import os

10 +from dataclasses import dataclass

11 from importlib import util

13 import pandas as pd

@@ -10,27 +15,49 @@ import pandas as pd

15 pd.options.display.float_format = "{:.2f}".format

16 plotly_found = util.find_spec("plotly.express") is not None

13 -

18 +DEFAULT_INFO_COLS = [

19 + "Model",

20 + "Dataset Name",

21 + "Input Len",

22 + "Output Len",

23 + # "TP Size",

24 + # "PP Size",

25 + "# of max concurrency.",

26 + "qps",

27 +]

28 +

29 +# Safety net: if any DataFrame leaks into to_html(), keep precision at 2.

30 +pd.set_option("display.precision", 2)

31 +pd.set_option("display.float_format", lambda x: f"{x:.2f}")

32 +

33 +

34 +# -----------------------------

35 +# Core data compare

36 +# -----------------------------

37 def compare_data_columns(

15 - files, name_column, data_column, info_cols, drop_column, debug=False

38 + files: list[str],

39 + name_column: str,

40 + data_column: str,

41 + info_cols: list[str],

42 + drop_column: str,

43 + debug: bool = False,

44 ):

45 """

46 Align concatenation by keys derived from info_cols instead of row order.

47 - Pick one canonical key list: subset of info_cols present in ALL files.

48 - For each file: set index to those keys, aggregate duplicates

21 - - (mean for metric, first for names).

49 + (mean for metric, first for names).

50 - Concat along axis=1 (indexes align), then reset_index so callers can

23 - - group by columns.

51 + group by columns.

52 - If --debug, add a <file_label>_name column per file.

53 """

54 print("\ncompare_data_column:", data_column)

56 frames = []

29 - raw_data_cols = []

57 + raw_data_cols: list[str] = []

58 compare_frames = []

32 - # 1) choose a canonical key list from info_cols that exists in ALL files

33 - cols_per_file = []

60 + cols_per_file: list[set] = []

61 for f in files:

62 try:

63 df_tmp = pd.read_json(f, orient="records")

@@ -40,24 +67,20 @@ def compare_data_columns(

68 key_cols = [c for c in info_cols if all(c in cset for cset in cols_per_file)]

69 if not key_cols:

43 - # soft fallback: use any info_cols present in the first file

70 key_cols = [c for c in info_cols if c in list(cols_per_file[0])]

71 if not key_cols:

72 raise ValueError(

73 "No common key columns found from info_cols across the input files."

74 )

50 - # 2) build a single "meta" block (keys as columns) once, aligned by the key index

76 meta_added = False

78 for file in files:

79 df = pd.read_json(file, orient="records")

56 - # Keep rows that actually have the compared metric (same as original behavior)

81 if drop_column in df.columns:

82 df = df.dropna(subset=[drop_column], ignore_index=True)

60 - # Stabilize numeric key columns (harmless if missing)

84 for c in (

85 "Input Len",

86 "Output Len",

@@ -69,32 +92,26 @@ def compare_data_columns(

92 if c in df.columns:

93 df[c] = pd.to_numeric(df[c], errors="coerce")

72 - # Ensure all key columns exist

95 for c in key_cols:

96 if c not in df.columns:

97 df[c] = pd.NA

77 - # Set index = key_cols and aggregate duplicates → unique MultiIndex

99 df_idx = df.set_index(key_cols, drop=False)

100

80 - # meta (key columns), unique per key

101 meta = df_idx[key_cols]

102 if not meta.index.is_unique:

103 meta = meta.groupby(level=key_cols, dropna=False).first()

104

85 - # metric series for this file, aggregated to one row per key

105 file_label = "/".join(file.split("/")[:-1]) or os.path.basename(file)

106 s = df_idx[data_column]

107 if not s.index.is_unique:

108 s = s.groupby(level=key_cols, dropna=False).mean()

90 - s.name = file_label # column label like original

109 + s.name = file_label

110

92 - # add meta once (from first file) so keys are the leftmost columns

111 if not meta_added:

112 frames.append(meta)

113 meta_added = True

114

97 - # (NEW) debug: aligned test-name column per file

115 if debug and name_column in df_idx.columns:

116 name_s = df_idx[name_column]

117 if not name_s.index.is_unique:

@@ -106,26 +123,19 @@ def compare_data_columns(

123 raw_data_cols.append(file_label)

124 compare_frames.append(s)

125

109 - # Generalize ratio: for any file N>=2, add ratio (fileN / file1)

126 if len(compare_frames) >= 2:

127 base = compare_frames[0]

128 current = compare_frames[-1]

129 if "P99" in data_column or "Median" in data_column:

114 - ratio = base / current # for latency

130 + ratio = base / current

131 else:

132 ratio = current / base

117 - ratio = ratio.mask(base == 0) # avoid inf when baseline is 0

133 + ratio = ratio.mask(base == 0)

134 ratio.name = f"Ratio 1 vs {len(compare_frames)}"

135 frames.append(ratio)

136

121 - # 4) concat on columns with aligned MultiIndex;

122 - # then reset_index to return keys as columns

123 - concat_df = pd.concat(frames, axis=1)

124 - concat_df = concat_df.reset_index(drop=True).reset_index()

125 - if "index" in concat_df.columns:

126 - concat_df = concat_df.drop(columns=["index"])

137 + concat_df = pd.concat(frames, axis=1).reset_index(drop=True)

138

128 - # Ensure key/info columns appear first (in your info_cols order)

139 front = [c for c in info_cols if c in concat_df.columns]

140 rest = [c for c in concat_df.columns if c not in front]

141 concat_df = concat_df[front + rest]

@@ -134,20 +144,15 @@ def compare_data_columns(

144 return concat_df, raw_data_cols

145

146

147 +# -----------------------------

148 +# Split helper

149 +# -----------------------------

150 def split_json_by_tp_pp(

151 input_file: str = "benchmark_results.json", output_root: str = "."

152 ) -> list[str]:

140 - """

141 - Split a benchmark JSON into separate folders by (TP Size, PP Size).

142 -

143 - Creates: <output_root>/tp{TP}_pp{PP}/benchmark_results.json

144 - Returns: list of file paths written.

145 - """

146 - # Load JSON data into DataFrame

153 with open(input_file, encoding="utf-8") as f:

154 data = json.load(f)

155

150 - # If the JSON is a dict with a list under common keys, use that list

156 if isinstance(data, dict):

157 for key in ("results", "serving_results", "benchmarks", "data"):

158 if isinstance(data.get(key), list):

@@ -156,7 +161,6 @@ def split_json_by_tp_pp(

161

162 df = pd.DataFrame(data)

163

159 - # Keep only "serving" tests

164 name_col = next(

165 (c for c in ["Test name", "test_name", "Test Name"] if c in df.columns), None

166 )

@@ -165,7 +169,6 @@ def split_json_by_tp_pp(

169 df[name_col].astype(str).str.contains(r"serving", case=False, na=False)

170 ].copy()

171

168 - # Handle alias column names

172 rename_map = {

173 "tp_size": "TP Size",

174 "tensor_parallel_size": "TP Size",

@@ -176,21 +179,14 @@ def split_json_by_tp_pp(

179 columns={k: v for k, v in rename_map.items() if k in df.columns}, inplace=True

180 )

181

179 - # Ensure TP/PP columns exist (default to 1 if missing)

182 if "TP Size" not in df.columns:

183 df["TP Size"] = 1

184 if "PP Size" not in df.columns:

185 df["PP Size"] = 1

186

185 - # make sure TP/PP are numeric ints with no NaN

186 - df["TP Size"] = (

187 - pd.to_numeric(df.get("TP Size", 1), errors="coerce").fillna(1).astype(int)

188 - )

189 - df["PP Size"] = (

190 - pd.to_numeric(df.get("PP Size", 1), errors="coerce").fillna(1).astype(int)

191 - )

187 + df["TP Size"] = pd.to_numeric(df["TP Size"], errors="coerce").fillna(1).astype(int)

188 + df["PP Size"] = pd.to_numeric(df["PP Size"], errors="coerce").fillna(1).astype(int)

189

193 - # Split into separate folders

190 saved_paths: list[str] = []

191 for (tp, pp), group_df in df.groupby(["TP Size", "PP Size"], dropna=False):

192 folder_name = os.path.join(output_root, f"tp{int(tp)}_pp{int(pp)}")

@@ -203,32 +199,9 @@ def split_json_by_tp_pp(

199 return saved_paths

200

201

206 -def _add_limit_line(fig, y_value, label):

207 - # Visible dashed line + annotation

208 - fig.add_hline(

209 - y=y_value,

210 - line_dash="dash",

211 - line_color="red" if "ttft" in label.lower() else "blue",

212 - annotation_text=f"{label}: {y_value} ms",

213 - annotation_position="top left",

214 - )

215 - # Optional: add a legend item (as a transparent helper trace)

216 - if plot and plotly_found:

217 - import plotly.graph_objects as go

218 -

219 - fig.add_trace(

220 - go.Scatter(

221 - x=[None],

222 - y=[None],

223 - mode="lines",

224 - line=dict(

225 - dash="dash", color="red" if "ttft" in label.lower() else "blue"

226 - ),

227 - name=f"{label}",

228 - )

229 - )

230 -

231 -

202 +# -----------------------------

203 +# Styling helpers

204 +# -----------------------------

205 def _find_concurrency_col(df: pd.DataFrame) -> str:

206 for c in [

207 "# of max concurrency.",

@@ -239,7 +212,6 @@ def _find_concurrency_col(df: pd.DataFrame) -> str:

212 ]:

213 if c in df.columns:

214 return c

242 - # Fallback: guess an integer-like column (harmless if unused)

215 for c in df.columns:

216 if df[c].dtype.kind in "iu" and df[c].nunique() > 1 and df[c].min() >= 1:

217 return c

@@ -248,8 +220,7 @@ def _find_concurrency_col(df: pd.DataFrame) -> str:

220

221 def _highlight_threshold(

222 df: pd.DataFrame, threshold: float

251 -) -> "pd.io.formats.style.Styler":

252 - """Highlight numeric per-configuration columns with value <= threshold."""

223 +) -> pd.io.formats.style.Styler:

224 conc_col = _find_concurrency_col(df)

225 key_cols = [

226 c

@@ -260,6 +231,7 @@ def _highlight_threshold(

231 c for c in df.columns if c not in key_cols and not str(c).startswith("Ratio")

232 ]

233 conf_cols = [c for c in conf_cols if pd.api.types.is_numeric_dtype(df[c])]

234 +

235 return df.style.map(

236 lambda v: "background-color:#e6ffe6;font-weight:bold;"

237 if pd.notna(v) and v <= threshold

@@ -268,7 +240,264 @@ def _highlight_threshold(

240 )

241

242

271 -if __name__ == "__main__":

243 +def highlight_ratio_columns(styler: pd.io.formats.style.Styler):

244 + ratio_cols = [c for c in styler.data.columns if "ratio" in str(c).lower()]

245 + if not ratio_cols:

246 + return styler

247 +

248 + styler = styler.apply(

249 + lambda _: ["background-color: #fff3b0"] * len(styler.data),

250 + subset=ratio_cols,

251 + axis=0,

252 + )

253 +

254 + styler = styler.set_table_styles(

255 + [

256 + {

257 + "selector": f"th.col_heading.level0.col{i}",

258 + "props": [("background-color", "#fff3b0")],

259 + }

260 + for i, col in enumerate(styler.data.columns)

261 + if col in ratio_cols

262 + ],

263 + overwrite=False,

264 + )

265 + return styler

266 +

267 +

268 +def _apply_two_decimals(

269 + styler: pd.io.formats.style.Styler,

270 +) -> pd.io.formats.style.Styler:

271 + df = styler.data

272 + num_cols = df.select_dtypes("number").columns

273 + if len(num_cols) == 0:

274 + return styler

275 + return styler.format({c: "{:.2f}" for c in num_cols}, na_rep="")

276 +

277 +

278 +# -----------------------------

279 +# Valid max concurrency summary helpers

280 +# -----------------------------

281 +def _config_value_columns(df: pd.DataFrame, conc_col: str) -> list[str]:

282 + key_cols = [

283 + c

284 + for c in ["Model", "Dataset Name", "Input Len", "Output Len"]

285 + if c in df.columns

286 + ]

287 + exclude = set(key_cols + [conc_col, "qps", "QPS"])

288 +

289 + cols: list[str] = []

290 + for c in df.columns:

291 + if c in exclude:

292 + continue

293 + lc = str(c).lower()

294 + if lc.startswith("ratio"):

295 + continue

296 + if lc.endswith("_name") or lc == "test name" or lc == "test_name":

297 + continue

298 + if pd.api.types.is_numeric_dtype(df[c]):

299 + cols.append(c)

300 + return cols

301 +

302 +

303 +def _max_concurrency_ok(

304 + df: pd.DataFrame, conc_col: str, cfg_col: str, threshold: float

305 +):

306 + if df is None or conc_col not in df.columns or cfg_col not in df.columns:

307 + return pd.NA

308 +

309 + d = df[[conc_col, cfg_col]].copy()

310 + d[conc_col] = pd.to_numeric(d[conc_col], errors="coerce")

311 + d[cfg_col] = pd.to_numeric(d[cfg_col], errors="coerce")

312 + d = d.dropna(subset=[conc_col, cfg_col])

313 +

314 + if d.empty:

315 + return pd.NA

316 +

317 + ok = d[d[cfg_col] <= threshold]

318 + if ok.empty:

319 + return pd.NA

320 +

321 + return ok[conc_col].max()

322 +

323 +

324 +def _value_at_concurrency(df: pd.DataFrame, conc_col: str, cfg_col: str, conc_value):

325 + if (

326 + df is None

327 + or conc_col not in df.columns

328 + or cfg_col not in df.columns

329 + or pd.isna(conc_value)

330 + ):

331 + return pd.NA

332 +

333 + d = df[[conc_col, cfg_col]].copy()

334 + d[conc_col] = pd.to_numeric(d[conc_col], errors="coerce")

335 + d[cfg_col] = pd.to_numeric(d[cfg_col], errors="coerce")

336 +

337 + conc_value = pd.to_numeric(conc_value, errors="coerce")

338 + if pd.isna(conc_value):

339 + return pd.NA

340 +

341 + hit = d[d[conc_col] == conc_value]

342 + if hit.empty:

343 + return pd.NA

344 + return hit[cfg_col].iloc[0]

345 +

346 +

347 +def build_valid_max_concurrency_summary_html(

348 + tput_group_df: pd.DataFrame | None,

349 + ttft_group_df: pd.DataFrame | None,

350 + tpot_group_df: pd.DataFrame | None,

351 + conc_col: str,

352 + args,

353 +) -> str:

354 + if ttft_group_df is None and tpot_group_df is None:

355 + return ""

356 +

357 + ttft_cols = (

358 + _config_value_columns(ttft_group_df, conc_col)

359 + if ttft_group_df is not None

360 + else []

361 + )

362 + tpot_cols = (

363 + _config_value_columns(tpot_group_df, conc_col)

364 + if tpot_group_df is not None

365 + else []

366 + )

367 + tput_cols = (

368 + _config_value_columns(tput_group_df, conc_col)

369 + if tput_group_df is not None

370 + else []

371 + )

372 +

373 + if ttft_group_df is not None and tpot_group_df is not None:

374 + cfg_cols = [c for c in ttft_cols if c in tpot_cols]

375 + if tput_group_df is not None:

376 + cfg_cols = [c for c in cfg_cols if c in tput_cols] or cfg_cols

377 + else:

378 + cfg_cols = ttft_cols or tpot_cols

379 +

380 + if not cfg_cols:

381 + cfg_cols = sorted(set(ttft_cols) | set(tpot_cols) | set(tput_cols), key=str)

382 +

383 + rows = []

384 + for cfg in cfg_cols:

385 + ttft_max = (

386 + _max_concurrency_ok(ttft_group_df, conc_col, cfg, args.ttft_max_ms)

387 + if ttft_group_df is not None

388 + else pd.NA

389 + )

390 + tpot_max = (

391 + _max_concurrency_ok(tpot_group_df, conc_col, cfg, args.tpot_max_ms)

392 + if tpot_group_df is not None

393 + else pd.NA

394 + )

395 + both = (

396 + pd.NA

397 + if (pd.isna(ttft_max) or pd.isna(tpot_max))

398 + else min(ttft_max, tpot_max)

399 + )

400 +

401 + tput_at_both = (

402 + _value_at_concurrency(tput_group_df, conc_col, cfg, both)

403 + if tput_group_df is not None

404 + else pd.NA

405 + )

406 + ttft_at_both = (

407 + _value_at_concurrency(ttft_group_df, conc_col, cfg, both)

408 + if ttft_group_df is not None

409 + else pd.NA

410 + )

411 + tpot_at_both = (

412 + _value_at_concurrency(tpot_group_df, conc_col, cfg, both)

413 + if tpot_group_df is not None

414 + else pd.NA

415 + )

416 +

417 + rows.append(

418 + {

419 + "Configuration": cfg,

420 + f"Max {conc_col} (TTFT ≤ {args.ttft_max_ms:g} ms)": ttft_max,

421 + f"Max {conc_col} (TPOT ≤ {args.tpot_max_ms:g} ms)": tpot_max,

422 + f"Max {conc_col} (Both)": both,

423 + "Output Tput @ Both (tok/s)": tput_at_both,

424 + "TTFT @ Both (ms)": ttft_at_both,

425 + "TPOT @ Both (ms)": tpot_at_both,

426 + }

427 + )

428 +

429 + summary_df = pd.DataFrame(rows)

430 +

431 + # --- Coerce numeric columns so Styler doesn't miss them due to object dtype ---

432 + for c in summary_df.columns:

433 + if c == "Configuration":

434 + continue

435 + summary_df[c] = pd.to_numeric(summary_df[c], errors="coerce")

436 +

437 + both_col = f"Max {conc_col} (Both)"

438 +

439 + # --- Strict 2-decimal formatting for ALL non-Configuration columns ---

440 + formatters = {}

441 + for c in summary_df.columns:

442 + if c == "Configuration":

443 + continue

444 + # default argument binds per-column formatter correctly

445 + formatters[c] = lambda v: "" if pd.isna(v) else f"{float(v):.2f}"

446 +

447 + styler = summary_df.style.format(formatters)

448 +

449 + def _green(v):

450 + return "background-color:#e6ffe6;font-weight:bold;" if pd.notna(v) else ""

451 +

452 + if both_col in summary_df.columns:

453 + styler = styler.map(_green, subset=[both_col])

454 +

455 + title = (

456 + '<div style="font-size: 1.15em; font-weight: 700; margin: 12px 0 6px 0;">'

457 + "Valid Max Concurrency Summary"

458 + "</div>\n"

459 + )

460 + return title + styler.to_html(table_attributes='border="1" class="dataframe"')

461 +

462 +

463 +# -----------------------------

464 +# Plot helper

465 +# -----------------------------

466 +def _add_limit_line(fig, y_value: float, label: str):

467 + fig.add_hline(

468 + y=y_value,

469 + line_dash="dash",

470 + line_color="red" if "ttft" in label.lower() else "blue",

471 + annotation_text=f"{label}: {y_value} ms",

472 + annotation_position="top left",

473 + )

474 + if plotly_found:

475 + import plotly.graph_objects as go

476 +

477 + fig.add_trace(

478 + go.Scatter(

479 + x=[None],

480 + y=[None],

481 + mode="lines",

482 + line=dict(

483 + dash="dash",

484 + color="red" if "ttft" in label.lower() else "blue",

485 + ),

486 + name=label,

487 + )

488 + )

489 +

490 +

491 +# -----------------------------

492 +# Refactored main + group-first report

493 +# -----------------------------

494 +@dataclass(frozen=True)

495 +class MetricPlan:

496 + data_cols: list[str]

497 + drop_column: str

498 +

499 +

500 +def build_parser() -> argparse.ArgumentParser:

501 parser = argparse.ArgumentParser()

502 parser.add_argument(

503 "-f", "--file", action="append", type=str, help="input file name"

@@ -308,149 +537,289 @@ if __name__ == "__main__":

537 default=100.0,

538 help="Reference limit for TPOT plots (ms)",

539 )

540 + return parser

541

312 - args = parser.parse_args()

542

543 +def choose_metrics(latency: str) -> MetricPlan:

544 + latency = (latency or "").lower()

545 drop_column = "P99"

315 - name_column = "Test name"

316 - info_cols = [

317 - "Model",

318 - "Dataset Name",

319 - "Input Len",

320 - "Output Len",

321 - "TP Size",

322 - "PP Size",

323 - "# of max concurrency.",

324 - "qps",

325 - ]

546

327 - if "median" in args.latency:

328 - data_cols_to_compare = ["Output Tput (tok/s)", "Median TTFT (ms)", "Median"]

329 - html_msgs_for_data_cols = [

330 - "Compare Output Tokens /n",

331 - "Median TTFT /n",

332 - "Median TPOT /n",

333 - ]

334 - drop_column = "P99"

335 - elif "p99" in args.latency:

336 - data_cols_to_compare = ["Output Tput (tok/s)", "P99 TTFT (ms)", "P99"]

337 - html_msgs_for_data_cols = [

338 - "Compare Output Tokens /n",

339 - "P99 TTFT /n",

340 - "P99 TPOT /n",

341 - ]

547 + if "median" in latency:

548 + return MetricPlan(

549 + data_cols=["Output Tput (tok/s)", "Median TTFT (ms)", "Median"],

550 + drop_column=drop_column,

551 + )

552 +

553 + return MetricPlan(

554 + data_cols=["Output Tput (tok/s)", "P99 TTFT (ms)", "P99"],

555 + drop_column=drop_column,

556 + )

557 +

558 +

559 +def prepare_input_files(args, info_cols: list[str]) -> tuple[list[str], list[str]]:

560 + if not args.file:

561 + raise ValueError("No input files provided. Use -f/--file.")

562

563 if len(args.file) == 1:

564 files = split_json_by_tp_pp(args.file[0], output_root="splits")

565 info_cols = [c for c in info_cols if c not in ("TP Size", "PP Size")]

566 else:

567 files = args.file

568 +

569 + return files, info_cols

570 +

571 +

572 +def get_y_axis_col(info_cols: list[str], xaxis: str) -> str:

573 + y_axis_index = info_cols.index(xaxis) if xaxis in info_cols else 6

574 + return info_cols[y_axis_index]

575 +

576 +

577 +def get_group_cols(output_df: pd.DataFrame, info_cols: list[str]) -> list[str]:

578 + filtered_info_cols = info_cols[:4]

579 + group_cols = [c for c in filtered_info_cols if c in output_df.columns]

580 + if not group_cols:

581 + raise ValueError(

582 + f"No valid group-by columns. Expected subset: {filtered_info_cols}, "

583 + f"but DataFrame has: {list(output_df.columns)}"

584 + )

585 + return group_cols

586 +

587 +

588 +def normalize_group_key(name):

589 + return name if isinstance(name, tuple) else (name,)

590 +

591 +

592 +def group_filename(name, prefix: str = "perf_comparison_") -> str:

593 + name_vals = normalize_group_key(name)

594 + safe = ",".join(map(str, name_vals)).replace(",", "_").replace("/", "-")

595 + return f"{prefix}{safe}.html"

596 +

597 +

598 +def build_group_suffix(group_cols: list[str], name) -> str:

599 + name_vals = normalize_group_key(name)

600 + return " , ".join(f"{col} : [ {val} ] " for col, val in zip(group_cols, name_vals))

601 +

602 +

603 +def render_metric_table_html(

604 + display_group: pd.DataFrame,

605 + metric_label: str,

606 + group_suffix: str,

607 + args,

608 +) -> str:

609 + title = (

610 + f'<div style="font-size: 1.25em; font-weight: 600; margin: 12px 0;">'

611 + f"{_html.escape(metric_label)}"

612 + f" — {_html.escape(group_suffix)}"

613 + f"</div>\n"

614 + )

615 +

616 + metric_name = metric_label.lower()

617 + if "ttft" in metric_name:

618 + styler = _highlight_threshold(display_group, args.ttft_max_ms)

619 + elif ("tpot" in metric_name) or ("median" in metric_name) or ("p99" in metric_name):

620 + styler = _highlight_threshold(display_group, args.tpot_max_ms)

621 + else:

622 + styler = display_group.style

623 +

624 + styler = _apply_two_decimals(styler)

625 + styler = highlight_ratio_columns(styler)

626 +

627 + return title + styler.to_html(table_attributes='border="1" class="dataframe"')

628 +

629 +

630 +def maybe_write_plot(

631 + main_fh,

632 + sub_fh,

633 + group_df: pd.DataFrame,

634 + raw_data_cols: list[str],

635 + metric_label: str,

636 + y_axis_col: str,

637 + args,

638 +):

639 + if not (args.plot and plotly_found):

640 + return

641 +

642 + import plotly.express as px

643 +

644 + df = group_df[raw_data_cols].sort_values(by=y_axis_col)

645 + df_melted = df.melt(

646 + id_vars=y_axis_col,

647 + var_name="Configuration",

648 + value_name=metric_label,

649 + )

650 +

651 + fig = px.line(

652 + df_melted,

653 + x=y_axis_col,

654 + y=metric_label,

655 + color="Configuration",

656 + title=f"{metric_label} vs {y_axis_col}",

657 + markers=True,

658 + )

659 +

660 + # Ensure plot hover + y tick labels are also 2 decimals.

661 + fig.update_traces(hovertemplate="%{y:.2f}<extra></extra>")

662 + fig.update_yaxes(tickformat=".2f")

663 +

664 + metric_name = metric_label.lower()

665 + if "ttft" in metric_name:

666 + _add_limit_line(fig, args.ttft_max_ms, "TTFT limit")

667 + elif ("tpot" in metric_name) or ("median" in metric_name) or ("p99" in metric_name):

668 + _add_limit_line(fig, args.tpot_max_ms, "TPOT limit")

669 +

670 + html = fig.to_html(full_html=True, include_plotlyjs="cdn")

671 + main_fh.write(html)

672 + sub_fh.write(html)

673 +

674 +

675 +def build_group_keys(

676 + df: pd.DataFrame, group_cols: list[str], sort_cols: list[str] | None = None

677 +):

678 + if sort_cols:

679 + df = df.sort_values(by=sort_cols)

680 + gb = df.groupby(group_cols, dropna=False)

681 + return [k for k, _ in gb]

682 +

683 +

684 +def write_report_group_first(

685 + files: list[str], info_cols: list[str], plan: MetricPlan, args

686 +):

687 + name_column = "Test name"

688 + y_axis_col = get_y_axis_col(info_cols, args.xaxis)

689 +

690 print("comparing : " + ", ".join(files))

349 - debug = args.debug

350 - plot = args.plot

351 - # For Plot feature, assign y axis from one of info_cols

352 - y_axis_index = info_cols.index(args.xaxis) if args.xaxis in info_cols else 6

353 - with open("perf_comparison.html", "w") as text_file:

354 - for i in range(len(data_cols_to_compare)):

355 - output_df, raw_data_cols = compare_data_columns(

356 - files,

357 - name_column,

358 - data_cols_to_compare[i],

359 - info_cols,

360 - drop_column,

361 - debug=debug,

691 +

692 + metric_cache: dict[str, tuple[pd.DataFrame, list[str]]] = {}

693 + group_cols_canonical: list[str] | None = None

694 +

695 + for metric_label in plan.data_cols:

696 + output_df, raw_data_cols = compare_data_columns(

697 + files,

698 + name_column,

699 + metric_label,

700 + info_cols,

701 + plan.drop_column,

702 + debug=args.debug,

703 + )

704 +

705 + raw_data_cols = list(raw_data_cols)

706 + raw_data_cols.insert(0, y_axis_col)

707 +

708 + group_cols = get_group_cols(output_df, info_cols)

709 + if group_cols_canonical is None:

710 + group_cols_canonical = group_cols

711 + else:

712 + group_cols_canonical = [c for c in group_cols_canonical if c in group_cols]

713 +

714 + metric_cache[metric_label] = (

715 + output_df.sort_values(by=args.xaxis),

716 + raw_data_cols,

717 + )

718 +

719 + if not group_cols_canonical:

720 + raise ValueError("No canonical group columns found across metrics.")

721 +

722 + first_metric = plan.data_cols[0]

723 + first_df_sorted, _ = metric_cache[first_metric]

724 + group_keys = build_group_keys(

725 + first_df_sorted, group_cols_canonical, sort_cols=[args.xaxis]

726 + )

727 +

728 + metric_groupbys = {

729 + metric_label: df.groupby(group_cols_canonical, dropna=False)

730 + for metric_label, (df, _) in metric_cache.items()

731 + }

732 +

733 + with open("perf_comparison.html", "w", encoding="utf-8") as main_fh:

734 + main_fh.write('<meta charset="utf-8">\n')

735 + for gkey in group_keys:

736 + gkey_tuple = normalize_group_key(gkey)

737 + suffix = build_group_suffix(group_cols_canonical, gkey_tuple)

738 + sub_path = group_filename(gkey_tuple)

739 + group_header = (

740 + '<div style="font-size: 1.4em; font-weight: 700; '

741 + 'margin: 18px 0 10px 0;">'

742 + f"{_html.escape(suffix)}"

743 + "</div>\n"

744 )

745

364 - # For Plot feature, insert y axis from one of info_cols

365 - raw_data_cols.insert(0, info_cols[y_axis_index])

366 -

367 - filtered_info_cols = info_cols[:-2]

368 - existing_group_cols = [

369 - c for c in filtered_info_cols if c in output_df.columns

370 - ]

371 - if not existing_group_cols:

372 - raise ValueError(

373 - f"No valid group-by columns "

374 - f"Expected subset: {filtered_info_cols}, "

375 - f"but DataFrame has: {list(output_df.columns)}"

376 - )

377 - # output_df_sorted = output_df.sort_values(by=existing_group_cols)

378 - output_df_sorted = output_df.sort_values(by=args.xaxis)

379 - output_groups = output_df_sorted.groupby(existing_group_cols, dropna=False)

380 - for name, group in output_groups:

381 - group_name = (

382 - ",".join(map(str, name)).replace(",", "_").replace("/", "-")

383 - )

384 - group_html_name = "perf_comparison_" + group_name + ".html"

385 -

386 - metric_name = str(data_cols_to_compare[i]).lower()

387 - if "tok/s" in metric_name:

388 - html = group.to_html()

389 - elif "ttft" in metric_name:

390 - styler = _highlight_threshold(group, args.ttft_max_ms).format(

391 - {c: "{:.2f}" for c in group.select_dtypes("number").columns},

392 - na_rep="—",

393 - )

394 - html = styler.to_html(

395 - table_attributes='border="1" class="dataframe"'

746 + main_fh.write(group_header)

747 + with open(sub_path, "w", encoding="utf-8") as sub_fh:

748 + sub_fh.write('<meta charset="utf-8">\n')

749 + sub_fh.write(group_header)

750 + tput_group_df = None

751 + ttft_group_df = None

752 + tpot_group_df = None

753 + conc_col = args.xaxis

754 +

755 + for metric_label in plan.data_cols:

756 + gb = metric_groupbys[metric_label]

757 + df_sorted, raw_data_cols = metric_cache[metric_label]

758 +

759 + try:

760 + group_df = gb.get_group(gkey)

761 + except KeyError:

762 + missing = (

763 + '<div style="font-size: 1.1em; font-weight: 600; '

764 + 'margin: 10px 0;">'

765 + f"{_html.escape(metric_label)} — missing for this group"

766 + "</div>\n"

767 + )

768 +

769 + main_fh.write(missing)

770 + sub_fh.write(missing)

771 + continue

772 +

773 + if conc_col not in group_df.columns:

774 + conc_col = _find_concurrency_col(group_df)

775 +

776 + mn = metric_label.lower().strip()

777 + if "tok/s" in mn:

778 + tput_group_df = group_df

779 + elif "ttft" in mn:

780 + ttft_group_df = group_df

781 + elif mn in ("p99", "median") or "tpot" in mn:

782 + tpot_group_df = group_df

783 +

784 + display_group = group_df.drop(

785 + columns=group_cols_canonical, errors="ignore"

786 )

397 - elif (

398 - "tpot" in metric_name

399 - or "median" in metric_name

400 - or "p99" in metric_name

401 - ):

402 - styler = _highlight_threshold(group, args.tpot_max_ms).format(

403 - {c: "{:.2f}" for c in group.select_dtypes("number").columns},

404 - na_rep="—",

787 +

788 + html = render_metric_table_html(

789 + display_group, metric_label, suffix, args

790 )

406 - html = styler.to_html(

407 - table_attributes='border="1" class="dataframe"'

791 + main_fh.write(html)

792 + sub_fh.write(html)

793 +

794 + maybe_write_plot(

795 + main_fh,

796 + sub_fh,

797 + group_df=group_df,

798 + raw_data_cols=raw_data_cols,

799 + metric_label=metric_label,

800 + y_axis_col=y_axis_col,

801 + args=args,

802 )

803

410 - text_file.write(html_msgs_for_data_cols[i])

411 - text_file.write(html)

412 - with open(group_html_name, "a+") as sub_text_file:

413 - sub_text_file.write(html_msgs_for_data_cols[i])

414 - sub_text_file.write(html)

415 -

416 - if plot and plotly_found:

417 - import plotly.express as px

418 -

419 - df = group[raw_data_cols]

420 - df_sorted = df.sort_values(by=info_cols[y_axis_index])

421 - # Melt DataFrame for plotting

422 - df_melted = df_sorted.melt(

423 - id_vars=info_cols[y_axis_index],

424 - var_name="Configuration",

425 - value_name=data_cols_to_compare[i],

426 - )

427 - title = (

428 - data_cols_to_compare[i] + " vs " + info_cols[y_axis_index]

429 - )

430 - # Create Plotly line chart

431 - fig = px.line(

432 - df_melted,

433 - x=info_cols[y_axis_index],

434 - y=data_cols_to_compare[i],

435 - color="Configuration",

436 - title=title,

437 - markers=True,

438 - )

804 + summary_html = build_valid_max_concurrency_summary_html(

805 + tput_group_df=tput_group_df,

806 + ttft_group_df=ttft_group_df,

807 + tpot_group_df=tpot_group_df,

808 + conc_col=conc_col,

809 + args=args,

810 + )

811 + if summary_html:

812 + main_fh.write(summary_html)

813 + sub_fh.write(summary_html)

814

440 - # ---- Add threshold lines based on metric name ----

441 - if "ttft" in metric_name:

442 - _add_limit_line(fig, args.ttft_max_ms, "TTFT limit")

443 - elif (

444 - "tpot" in metric_name

445 - or "median" in metric_name

446 - or "p99" in metric_name

447 - ):

448 - _add_limit_line(fig, args.tpot_max_ms, "TPOT limit")

449 -

450 - # Export to HTML

451 - text_file.write(

452 - fig.to_html(full_html=True, include_plotlyjs="cdn")

453 - )

454 - sub_text_file.write(

455 - fig.to_html(full_html=True, include_plotlyjs="cdn")

456 - )

815 +

816 +def main():

817 + args = build_parser().parse_args()

818 + info_cols = list(DEFAULT_INFO_COLS)

819 + plan = choose_metrics(args.latency)

820 + files, info_cols = prepare_input_files(args, info_cols)

821 + write_report_group_first(files, info_cols, plan, args)

822 +

823 +

824 +if __name__ == "__main__":

825 + main()

.buildkite/performance-benchmarks/tests/serving-tests-cpu.json

+40 -3

@@ -19,10 +19,8 @@

19 "block_size": 128,

20 "trust_remote_code": "",

21 "disable_log_stats": "",

22 - "enforce_eager": "",

22 "max_num_batched_tokens": 2048,

24 - "max_num_seqs": 256,

25 - "load_format": "dummy"

23 + "max_num_seqs": 256

24 },

25 "client_parameters": {

26 "model": "meta-llama/Llama-3.1-8B-Instruct",

@@ -151,6 +149,45 @@

149 "random-output-len": 128

150 }

151 },

152 + {

153 + "test_name": "serving_llama8B_int4_tp1_random_128_128",

154 + "server_parameters": {

155 + "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",

156 + "tensor_parallel_size": 1

157 + },

158 + "client_parameters": {

159 + "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",

160 + "dataset_name": "random",

161 + "random-input-len": 128,

162 + "random-output-len": 128

163 + }

164 + },

165 + {

166 + "test_name": "serving_llama8B_int4_tp2_random_128_128",

167 + "server_parameters": {

168 + "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",

169 + "tensor_parallel_size": 2

170 + },

171 + "client_parameters": {

172 + "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",

173 + "dataset_name": "random",

174 + "random-input-len": 128,

175 + "random-output-len": 128

176 + }

177 + },

178 + {

179 + "test_name": "serving_llama8B_int4_tp4_random_128_128",

180 + "server_parameters": {

181 + "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",

182 + "tensor_parallel_size": 4

183 + },

184 + "client_parameters": {

185 + "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",

186 + "dataset_name": "random",

187 + "random-input-len": 128,

188 + "random-output-len": 128

189 + }

190 + },

191 {

192 "test_name": "serving_llama3B_tp1_random_128_128",

193 "server_parameters": {

docs/benchmarking/dashboard.md

+52 -1

@@ -40,7 +40,58 @@ When run, benchmark script generates results under **benchmark/results** folder,

40 - `REMOTE_HOST`: IP for the remote vLLM service to benchmark. Default value is empty string.

41 - `REMOTE_PORT`: Port for the remote vLLM service to benchmark. Default value is empty string.

43 -For more results visualization, check the [visualizing the results](https://github.com/intel-ai-tce/vllm/blob/more_cpu_models/.buildkite/nightly-benchmarks/README.md#visualizing-the-results).

43 +### Visualization

44 +

45 +The `convert-results-json-to-markdown.py` helps you put the benchmarking results inside a markdown table with real benchmarking results.

46 +You can find the result presented as a table inside the `buildkite/performance-benchmark` job page.

47 +If you do not see the table, please wait till the benchmark finish running.

48 +The json version of the table (together with the json version of the benchmark) will be also attached to the markdown file.

49 +The raw benchmarking results (in the format of json files) are in the `Artifacts` tab of the benchmarking.

50 +

51 +#### Performance Results Comparison

52 +

53 +The `compare-json-results.py` helps to compare benchmark results JSON files converted using `convert-results-json-to-markdown.py`.

54 +When run, benchmark script generates results under `benchmark/results` folder, along with the `benchmark_results.md` and `benchmark_results.json`.

55 +`compare-json-results.py` compares two `benchmark_results.json` files and provides performance ratio e.g. for Output Tput, Median TTFT and Median TPOT.

56 +If only one benchmark_results.json is passed, `compare-json-results.py` compares different TP and PP configurations in the benchmark_results.json instead.

57 +

58 +Here is an example using the script to compare result_a and result_b with max concurrency and qps for same Model, Dataset name, input/output length.

59 +`python3 compare-json-results.py -f results_a/benchmark_results.json -f results_b/benchmark_results.json`

60 +

61 +***Output Tput (tok/s) — Model : [ meta-llama/Llama-3.1-8B-Instruct ] , Dataset Name : [ random ] , Input Len : [ 2048.0 ] , Output Len : [ 2048.0 ]***

62 +

64 +|----|------|-----|-----------|----------|----------|

65 +| 0 | 12 | inf | 24.98 | 186.03 | 7.45 |

66 +| 1 | 16 | inf| 25.49 | 246.92 | 9.69 |

67 +| 2 | 24 | inf| 27.74 | 293.34 | 10.57 |

68 +| 3 | 32 | inf| 28.61 |306.69 | 10.72 |

69 +

70 +***compare-json-results.py – Command-Line Parameters***

71 +

72 +compare-json-results.py provides configurable parameters to compare one or more benchmark_results.json files and generate summary tables and plots.

73 +In most cases, users only need to specify --file to parse the desired benchmark results.

74 +

76 +| ---------------------- | ------------------ | ----------------------- | ----------------------------------------------------------------------------------------------------- |

81 +| `--latency` | `str` | `p99` | Latency aggregation method used for TTFT/TPOT. Supported values: `median` or `p99`. |

84 +

85 +***Valid Max Concurrency Summary***

86 +

87 +Based on the configured TTFT and TPOT SLA thresholds, compare-json-results.py computes the maximum valid concurrency for each benchmark result.

88 +The “Max # of max concurrency. (Both)” column represents the highest concurrency level that satisfies both TTFT and TPOT constraints simultaneously.

89 +This value is typically used in capacity planning and sizing guides.

90 +

92 +| - | -------------- | ------------------------------------------- | ----------------------------------------- | -------------------------------- | -------------------------- | ---------------- | ---------------- |

93 +| 0 | results-a | 128.00 | 12.00 | 12.00 | 127.76 | 3000.82 | 93.24 |

94 +| 1 | results-b | 128.00 | 32.00 | 32.00 | 371.42 | 2261.53 | 81.74 |

96 More information on the performance benchmarks and their parameters can be found in [Benchmark README](https://github.com/intel-ai-tce/vllm/blob/more_cpu_models/.buildkite/nightly-benchmarks/README.md) and [performance benchmark description](../../.buildkite/performance-benchmarks/performance-benchmarks-descriptions.md).