@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / a6e0d096dd

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 19:45 UTC

[Feature] Add visionarena offline support for benchmark_throughput (#14654)

Signed-off-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com> Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com> Co-authored-by: Jennifer Zhao <7443418+JenZhao@users.noreply.github.com> Co-authored-by: Jennifer Zhao <JenZhao@users.noreply.github.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>

Jennifer Zhao committed Mar 13, 2025 at 21:07 UTC a6e0d096dd48d1190fb548dd81fdd31f310391e3

3 files changed +291 -99

benchmarks/README.md

+47 -11

@@ -43,20 +43,26 @@ become available.

43 <tr>

44 <td><strong>HuggingFace</strong></td>

45 <td style="text-align: center;">✅</td>

46 - <td style="text-align: center;">🚧</td>

46 + <td style="text-align: center;">🟡</td>

47 <td>Specify your dataset path on HuggingFace</td>

48 </tr>

49 <tr>

50 <td><strong>VisionArena</strong></td>

51 <td style="text-align: center;">✅</td>

52 - <td style="text-align: center;">🚧</td>

52 + <td style="text-align: center;">✅</td>

53 <td><code>lmarena-ai/vision-arena-bench-v0.1</code> (a HuggingFace dataset)</td>

54 </tr>

55 </tbody>

56 </table>

57 -✅: supported

57 +

58 +✅: supported

59 +

60 🚧: to be supported

62 +🟡: Partial support. Currently, HuggingFaceDataset only supports dataset formats

63 +similar to `lmms-lab/LLaVA-OneVision-Data`. If you need support for other dataset

64 +formats, please consider contributing.

65 +

66 **Note**: VisionArena’s `dataset-name` should be set to `hf`

68 ---

@@ -79,7 +85,7 @@ NUM_PROMPTS=10

85 BACKEND="openai-chat"

86 DATASET_NAME="sharegpt"

87 DATASET_PATH="<your data path>/ShareGPT_V3_unfiltered_cleaned_split.json"

82 -python3 benchmarks/benchmark_serving.py --backend ${BACKEND} --model ${MODEL_NAME} --endpoint /v1/chat/completions --dataset-name ${DATASET_NAME} --dataset-path ${DATASET_PATH} --num-prompts ${NUM_PROMPTS}

88 +python3 vllm/benchmarks/benchmark_serving.py --backend ${BACKEND} --model ${MODEL_NAME} --endpoint /v1/chat/completions --dataset-name ${DATASET_NAME} --dataset-path ${DATASET_PATH} --num-prompts ${NUM_PROMPTS}

89 ```

91 If successful, you will see the following output

@@ -123,7 +129,7 @@ DATASET_NAME="hf"

129 DATASET_PATH="lmarena-ai/vision-arena-bench-v0.1"

130 DATASET_SPLIT='train'

131

126 -python3 benchmarks/benchmark_serving.py \

132 +python3 vllm/benchmarks/benchmark_serving.py \

133 --backend "${BACKEND}" \

134 --model "${MODEL_NAME}" \

135 --endpoint "/v1/chat/completions" \

@@ -140,35 +146,65 @@ python3 benchmarks/benchmark_serving.py \

146 MODEL_NAME="NousResearch/Hermes-3-Llama-3.1-8B"

147 NUM_PROMPTS=10

148 DATASET_NAME="sonnet"

143 -DATASET_PATH="benchmarks/sonnet.txt"

149 +DATASET_PATH="vllm/benchmarks/sonnet.txt"

150

145 -python3 benchmarks/benchmark_throughput.py \

151 +python3 vllm/benchmarks/benchmark_throughput.py \

152 --model "${MODEL_NAME}" \

153 --dataset-name "${DATASET_NAME}" \

154 --dataset-path "${DATASET_PATH}" \

155 --num-prompts "${NUM_PROMPTS}"

150 - ```

156 +```

157

158 If successful, you will see the following output

159

160 ```

155 -Throughput: 7.35 requests/s, 4789.20 total tokens/s, 1102.83 output tokens/s

161 +Throughput: 7.15 requests/s, 4656.00 total tokens/s, 1072.15 output tokens/s

162 +Total num prompt tokens: 5014

163 +Total num output tokens: 1500

164 +```

165 +

166 +### VisionArena Benchmark for Vision Language Models

167 +

168 +``` bash

169 +MODEL_NAME="Qwen/Qwen2-VL-7B-Instruct"

170 +NUM_PROMPTS=10

171 +DATASET_NAME="hf"

172 +DATASET_PATH="lmarena-ai/vision-arena-bench-v0.1"

173 +DATASET_SPLIT="train"

174 +

175 +python3 vllm/benchmarks/benchmark_throughput.py \

176 + --model "${MODEL_NAME}" \

177 + --backend "vllm-chat" \

178 + --dataset-name "${DATASET_NAME}" \

179 + --dataset-path "${DATASET_PATH}" \

180 + --num-prompts "${NUM_PROMPTS}" \

181 + --hf-split "${DATASET_SPLIT}"

182 +```

183 +

184 +The `num prompt tokens` now includes image token counts

185 +

186 +```

187 +Throughput: 2.55 requests/s, 4036.92 total tokens/s, 326.90 output tokens/s

188 +Total num prompt tokens: 14527

189 +Total num output tokens: 1280

190 ```

191

192 ### Benchmark with LoRA Adapters

193

194 ``` bash

195 +# download dataset

196 +# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

197 MODEL_NAME="meta-llama/Llama-2-7b-hf"

198 BACKEND="vllm"

199 DATASET_NAME="sharegpt"

164 -DATASET_PATH="/home/jovyan/data/vllm_benchmark_datasets/ShareGPT_V3_unfiltered_cleaned_split.json"

200 +DATASET_PATH="<your data path>/ShareGPT_V3_unfiltered_cleaned_split.json"

201 NUM_PROMPTS=10

202 MAX_LORAS=2

203 MAX_LORA_RANK=8

204 ENABLE_LORA="--enable-lora"

205 LORA_PATH="yard1/llama-2-7b-sql-lora-test"

206

171 -python3 benchmarks/benchmark_throughput.py \

207 +python3 vllm/benchmarks/benchmark_throughput.py \

208 --model "${MODEL_NAME}" \

209 --backend "${BACKEND}" \

210 --dataset_path "${DATASET_PATH}" \

benchmarks/benchmark_dataset.py

+43 -22

@@ -46,7 +46,7 @@ class SampleRequest:

46 Represents a single inference request for benchmarking.

47 """

49 - prompt: str

49 + prompt: Union[str, Any]

50 prompt_len: int

51 expected_output_len: int

52 multi_modal_data: Optional[Union[MultiModalDataDict, dict]] = None

@@ -84,6 +84,20 @@ class BenchmarkDataset(ABC):

84 if random_seed is not None else self.DEFAULT_SEED)

85 self.data = None

87 + def apply_multimodal_chat_transformation(

88 + self,

89 + prompt: str,

90 + mm_content: Optional[MultiModalDataDict] = None) -> list[dict]:

91 + """

92 + Transform a prompt and optional multimodal content into a chat format.

93 + This method is used for chat models that expect a specific

94 + conversation format.

95 + """

96 + content = [{"text": prompt, "type": "text"}]

97 + if mm_content is not None:

98 + content.append(mm_content)

99 + return [{"role": "user", "content": content}]

100 +

101 def load_data(self) -> None:

102 """

103 Load data from the dataset path into self.data.

@@ -338,6 +352,7 @@ class ShareGPTDataset(BenchmarkDataset):

352 lora_path: Optional[str] = None,

353 max_loras: Optional[int] = None,

354 output_len: Optional[int] = None,

355 + enable_multimodal_chat: bool = False,

356 **kwargs) -> list:

357 samples: list = []

358 for entry in self.data:

@@ -358,6 +373,9 @@ class ShareGPTDataset(BenchmarkDataset):

373 skip_min_output_len_check=output_len

374 is not None):

375 continue

376 + if enable_multimodal_chat:

377 + prompt = self.apply_multimodal_chat_transformation(

378 + prompt, None)

379 samples.append(

380 SampleRequest(

381 prompt=prompt,

@@ -550,10 +568,13 @@ class HuggingFaceDataset(BenchmarkDataset):

568 split=self.dataset_split,

569 streaming=True,

570 )

553 -

554 - if "conversations" not in self.data.features:

555 - raise ValueError("HF Dataset must have a 'conversations' column.")

556 -

571 + if self.data.features is None or "conversations" \

572 + not in self.data.features:

573 + raise ValueError(

574 + "HuggingFaceDataset currently only supports datasets with "

575 + "a 'conversations' column like lmms-lab/LLaVA-OneVision-Data. "

576 + "Please consider contributing if you would like to add "

577 + "support for additional dataset formats.")

578 # Shuffle and filter examples with at least 2 conversations.

579 self.data = self.data.shuffle(seed=self.random_seed).filter(

580 lambda x: len(x["conversations"]) >= 2)

@@ -561,9 +582,8 @@ class HuggingFaceDataset(BenchmarkDataset):

582 def sample(self,

583 tokenizer: PreTrainedTokenizerBase,

584 num_requests: int,

564 - lora_path: Optional[str] = None,

565 - max_loras: Optional[int] = None,

585 output_len: Optional[int] = None,

586 + enable_multimodal_chat: bool = False,

587 **kwargs) -> list:

588 sampled_requests = []

589 dynamic_output = output_len is None

@@ -571,13 +591,9 @@ class HuggingFaceDataset(BenchmarkDataset):

591 for item in self.data:

592 if len(sampled_requests) >= num_requests:

593 break

574 -

594 conv = item["conversations"]

595 prompt, completion = conv[0]["value"], conv[1]["value"]

596

578 - lora_request, tokenizer = self.get_random_lora_request(

579 - tokenizer, lora_path=lora_path, max_loras=max_loras)

580 -

597 prompt_ids = tokenizer(prompt).input_ids

598 completion_ids = tokenizer(completion).input_ids

599 prompt_len = len(prompt_ids)

@@ -587,16 +603,20 @@ class HuggingFaceDataset(BenchmarkDataset):

603 if dynamic_output and not is_valid_sequence(

604 prompt_len, completion_len):

605 continue

590 -

606 mm_content = process_image(

607 item["image"]) if "image" in item else None

608 + if enable_multimodal_chat:

609 + # Note: when chat is enabled the request prompt_len is no longer

610 + # accurate and we will be using request output to count the

611 + # actual prompt len and output len

612 + prompt = self.apply_multimodal_chat_transformation(

613 + prompt, mm_content)

614 sampled_requests.append(

615 SampleRequest(

616 prompt=prompt,

617 prompt_len=prompt_len,

618 expected_output_len=output_len,

619 multi_modal_data=mm_content,

599 - lora_request=lora_request,

620 ))

621 return sampled_requests

622

@@ -606,7 +626,7 @@ class HuggingFaceDataset(BenchmarkDataset):

626 # -----------------------------------------------------------------------------

627

628

609 -class VisionArenaDataset(BenchmarkDataset):

629 +class VisionArenaDataset(HuggingFaceDataset):

630 """

631 Vision Arena Dataset.

632 """

@@ -617,14 +637,9 @@ class VisionArenaDataset(BenchmarkDataset):

637

638 def __init__(

639 self,

620 - dataset_split: str,

621 - dataset_subset: Optional[str] = None,

640 **kwargs,

641 ) -> None:

642 super().__init__(**kwargs)

625 - self.dataset_split = dataset_split

626 - self.dataset_subset = dataset_subset

627 -

643 if self.dataset_path != self.VISION_ARENA_DATASET_PATH:

644 raise ValueError(f"Only support Vision Arena dataset.\

645 This data path {self.dataset_path} is not valid.")

@@ -645,9 +660,9 @@ class VisionArenaDataset(BenchmarkDataset):

660 def sample(self,

661 tokenizer: PreTrainedTokenizerBase,

662 num_requests: int,

648 - output_len: int = DEFAULT_OUTPUT_LEN,

663 + output_len: Optional[int] = None,

664 + enable_multimodal_chat: bool = False,

665 **kwargs) -> list:

650 - # TODO (jenniferzhao): Add support for offline benchmark sampling

666 output_len = (output_len

667 if output_len is not None else self.DEFAULT_OUTPUT_LEN)

668 sampled_requests = []

@@ -655,8 +670,14 @@ class VisionArenaDataset(BenchmarkDataset):

670 if len(sampled_requests) >= num_requests:

671 break

672 prompt = item["turns"][0][0]["content"]

658 - prompt_len = len(tokenizer(prompt).input_ids)

673 mm_content = process_image(item["images"][0])

674 + prompt_len = len(tokenizer(prompt).input_ids)

675 + if enable_multimodal_chat:

676 + # Note: when chat is enabled the request prompt_len is no longer

677 + # accurate and we will be using request output to count the

678 + # actual prompt len

679 + prompt = self.apply_multimodal_chat_transformation(

680 + prompt, mm_content)

681 sampled_requests.append(

682 SampleRequest(

683 prompt=prompt,

benchmarks/benchmark_throughput.py

+201 -66

@@ -11,8 +11,9 @@ from typing import Any, Optional, Union

12 import torch

13 import uvloop

14 -from benchmark_dataset import (BurstGPTDataset, RandomDataset, SampleRequest,

15 - ShareGPTDataset, SonnetDataset)

14 +from benchmark_dataset import (BurstGPTDataset, HuggingFaceDataset,

15 + RandomDataset, SampleRequest, ShareGPTDataset,

16 + SonnetDataset, VisionArenaDataset)

17 from benchmark_utils import convert_to_pytorch_benchmark_format, write_to_json

18 from tqdm import tqdm

19 from transformers import (AutoModelForCausalLM, AutoTokenizer,

@@ -23,6 +24,7 @@ from vllm.entrypoints.openai.api_server import (

24 build_async_engine_client_from_engine_args)

25 from vllm.inputs import TextPrompt, TokensPrompt

26 from vllm.lora.request import LoRARequest

27 +from vllm.outputs import RequestOutput

28 from vllm.sampling_params import BeamSearchParams

29 from vllm.utils import FlexibleArgumentParser, merge_async_iterators

@@ -32,7 +34,7 @@ def run_vllm(

34 n: int,

35 engine_args: EngineArgs,

36 disable_detokenize: bool = False,

35 -) -> float:

37 +) -> tuple[float, Optional[list[RequestOutput]]]:

38 from vllm import LLM, SamplingParams

39 llm = LLM(**dataclasses.asdict(engine_args))

40 assert all(

@@ -66,12 +68,13 @@ def run_vllm(

69 use_beam_search = False

71 + outputs = None

72 if not use_beam_search:

73 start = time.perf_counter()

71 - llm.generate(prompts,

72 - sampling_params,

73 - lora_request=lora_requests,

74 - use_tqdm=True)

74 + outputs = llm.generate(prompts,

75 + sampling_params,

76 + lora_request=lora_requests,

77 + use_tqdm=True)

78 end = time.perf_counter()

79 else:

80 assert lora_requests is None, "BeamSearch API does not support LoRA"

@@ -89,7 +92,46 @@ def run_vllm(

92 ignore_eos=True,

93 ))

94 end = time.perf_counter()

92 - return end - start

95 + return end - start, outputs

96 +

97 +

98 +def run_vllm_chat(

99 + requests: list[SampleRequest],

100 + n: int,

101 + engine_args: EngineArgs,

102 + disable_detokenize: bool = False) -> tuple[float, list[RequestOutput]]:

103 + """

104 + Run vLLM chat benchmark. This function is recommended ONLY for benchmarking

105 + multimodal models as it properly handles multimodal inputs and chat

106 + formatting. For non-multimodal models, use run_vllm() instead.

107 + """

108 + from vllm import LLM, SamplingParams

109 + llm = LLM(**dataclasses.asdict(engine_args))

110 +

111 + assert all(

112 + llm.llm_engine.model_config.max_model_len >= (

113 + request.prompt_len + request.expected_output_len)

114 + for request in requests), (

115 + "Please ensure that max_model_len is greater than the sum of "

116 + "prompt_len and expected_output_len for all requests.")

117 +

118 + prompts = []

119 + sampling_params: list[SamplingParams] = []

120 + for request in requests:

121 + prompts.append(request.prompt)

122 + sampling_params.append(

123 + SamplingParams(

124 + n=n,

125 + temperature=1.0,

126 + top_p=1.0,

127 + ignore_eos=True,

128 + max_tokens=request.expected_output_len,

129 + detokenize=not disable_detokenize,

130 + ))

131 + start = time.perf_counter()

132 + outputs = llm.chat(prompts, sampling_params, use_tqdm=True)

133 + end = time.perf_counter()

134 + return end - start, outputs

135

136

137 async def run_vllm_async(

@@ -264,6 +306,8 @@ def get_requests(args, tokenizer):

306 dataset_cls = RandomDataset

307 elif args.dataset_name == "sharegpt":

308 dataset_cls = ShareGPTDataset

309 + if args.backend == "vllm-chat":

310 + sample_kwargs["enable_multimodal_chat"] = True

311 elif args.dataset_name == "sonnet":

312 assert tokenizer.chat_template or tokenizer.default_chat_template, (

313 "Tokenizer/model must have chat template for sonnet dataset.")

@@ -272,6 +316,19 @@ def get_requests(args, tokenizer):

316 sample_kwargs["return_prompt_formatted"] = True

317 elif args.dataset_name == "burstgpt":

318 dataset_cls = BurstGPTDataset

319 + elif args.dataset_name == "hf":

320 + if args.backend != "vllm-chat":

321 + raise ValueError(

322 + "hf datasets only are supported by vllm-chat backend")

323 + # Choose between VisionArenaDataset and HuggingFaceDataset based on

324 + # provided parameters.

325 + dataset_cls = (VisionArenaDataset if args.dataset_path

326 + == VisionArenaDataset.VISION_ARENA_DATASET_PATH

327 + and args.hf_subset is None else HuggingFaceDataset)

328 + common_kwargs['dataset_subset'] = args.hf_subset

329 + common_kwargs['dataset_split'] = args.hf_split

330 + sample_kwargs["enable_multimodal_chat"] = True

331 +

332 else:

333 raise ValueError(f"Unknown dataset name: {args.dataset_name}")

334 # Remove None values

@@ -290,6 +347,7 @@ def main(args: argparse.Namespace):

347 requests = get_requests(args, tokenizer)

348 is_multi_modal = any(request.multi_modal_data is not None

349 for request in requests)

350 + request_outputs: Optional[list[RequestOutput]] = None

351 if args.backend == "vllm":

352 if args.async_engine:

353 elapsed_time = uvloop.run(

@@ -301,9 +359,9 @@ def main(args: argparse.Namespace):

359 args.disable_detokenize,

360 ))

361 else:

304 - elapsed_time = run_vllm(requests, args.n,

305 - EngineArgs.from_cli_args(args),

306 - args.disable_detokenize)

362 + elapsed_time, request_outputs = run_vllm(

363 + requests, args.n, EngineArgs.from_cli_args(args),

364 + args.disable_detokenize)

365 elif args.backend == "hf":

366 assert args.tensor_parallel_size == 1

367 elapsed_time = run_hf(requests, args.model, tokenizer, args.n,

@@ -312,20 +370,45 @@ def main(args: argparse.Namespace):

370 elif args.backend == "mii":

371 elapsed_time = run_mii(requests, args.model, args.tensor_parallel_size,

372 args.output_len)

373 + elif args.backend == "vllm-chat":

374 + elapsed_time, request_outputs = run_vllm_chat(

375 + requests, args.n, EngineArgs.from_cli_args(args),

376 + args.disable_detokenize)

377 else:

378 raise ValueError(f"Unknown backend: {args.backend}")

317 - total_num_tokens = sum(request.prompt_len + request.expected_output_len

318 - for request in requests)

319 - total_output_tokens = sum(request.expected_output_len

320 - for request in requests)

321 - if is_multi_modal:

322 - print("\033[91mWARNING\033[0m: Multi-modal request detected. The "

379 +

380 + if request_outputs:

381 + # Note: with the vllm and vllm-chat backends,

382 + # we have request_outputs, which we use to count tokens.

383 + total_prompt_tokens = 0

384 + total_output_tokens = 0

385 + for ro in request_outputs:

386 + if not isinstance(ro, RequestOutput):

387 + continue

388 + total_prompt_tokens += len(

389 + ro.prompt_token_ids) if ro.prompt_token_ids else 0

390 + total_output_tokens += sum(

391 + len(o.token_ids) for o in ro.outputs if o)

392 + total_num_tokens = total_prompt_tokens + total_output_tokens

393 + else:

394 + total_num_tokens = sum(r.prompt_len + r.expected_output_len

395 + for r in requests)

396 + total_output_tokens = sum(r.expected_output_len for r in requests)

397 + total_prompt_tokens = total_num_tokens - total_output_tokens

398 +

399 + if is_multi_modal and args.backend != "vllm-chat":

400 + print("\033[91mWARNING\033[0m: Multi-modal request with "

401 + f"{args.backend} backend detected. The "

402 "following metrics are not accurate because image tokens are not"

403 " counted. See vllm-project/vllm/issues/9778 for details.")

404 # TODO(vllm-project/vllm/issues/9778): Count multi-modal token length.

405 + # vllm-chat backend counts the image tokens now

406 +

407 print(f"Throughput: {len(requests) / elapsed_time:.2f} requests/s, "

408 f"{total_num_tokens / elapsed_time:.2f} total tokens/s, "

409 f"{total_output_tokens / elapsed_time:.2f} output tokens/s")

410 + print(f"Total num prompt tokens: {total_prompt_tokens}")

411 + print(f"Total num output tokens: {total_output_tokens}")

412

413 # Output JSON results if specified

414 if args.output_json:

@@ -341,17 +424,100 @@ def main(args: argparse.Namespace):

424 save_to_pytorch_benchmark_format(args, results)

425

426

427 +def validate_args(args):

428 + """

429 + Validate command-line arguments.

430 + """

431 +

432 + # === Deprecation and Defaulting ===

433 + if args.dataset is not None:

434 + warnings.warn(

435 + "The '--dataset' argument will be deprecated in the next release. "

436 + "Please use '--dataset-name' and '--dataset-path' instead.",

437 + stacklevel=2)

438 + args.dataset_path = args.dataset

439 +

440 + if not getattr(args, "tokenizer", None):

441 + args.tokenizer = args.model

442 +

443 + # === Backend Validation ===

444 + valid_backends = {"vllm", "hf", "mii", "vllm-chat"}

445 + if args.backend not in valid_backends:

446 + raise ValueError(f"Unsupported backend: {args.backend}")

447 +

448 + # === Dataset Configuration ===

449 + if not args.dataset and not args.dataset_path:

450 + print(

451 + "When dataset path is not set, it will default to random dataset")

452 + args.dataset_name = 'random'

453 + if args.input_len is None:

454 + raise ValueError("input_len must be provided for a random dataset")

455 +

456 + # === Dataset Name Specific Checks ===

457 + # --hf-subset and --hf-split: only used

458 + # when dataset_name is 'hf'

459 + if args.dataset_name != "hf" and (

460 + getattr(args, "hf_subset", None) is not None

461 + or getattr(args, "hf_split", None) is not None):

462 + warnings.warn("--hf-subset and --hf-split will be ignored \

463 + since --dataset-name is not 'hf'.",

464 + stacklevel=2)

465 + elif args.dataset_name == "hf" and args.backend != "vllm-chat":

466 + raise ValueError(

467 + "When --dataset-name is 'hf', backend must be 'vllm-chat'")

468 +

469 + # --random-range-ratio: only used when dataset_name is 'random'

470 + if args.dataset_name != 'random' and args.random_range_ratio is not None:

471 + warnings.warn("--random-range-ratio will be ignored since \

472 + --dataset-name is not 'random'.",

473 + stacklevel=2)

474 +

475 + # --prefix-len: only used when dataset_name is 'random', 'sonnet', or not

476 + # set.

477 + if args.dataset_name not in {"random", "sonnet", None

478 + } and args.prefix_len is not None:

479 + warnings.warn("--prefix-len will be ignored since --dataset-name\

480 + is not 'random', 'sonnet', or not set.",

481 + stacklevel=2)

482 +

483 + # === LoRA Settings ===

484 + if getattr(args, "enable_lora", False) and args.backend != "vllm":

485 + raise ValueError(

486 + "LoRA benchmarking is only supported for vLLM backend")

487 + if getattr(args, "enable_lora", False) and args.lora_path is None:

488 + raise ValueError("LoRA path must be provided when enable_lora is True")

489 +

490 + # === Backend-specific Validations ===

491 + if args.backend == "hf" and args.hf_max_batch_size is None:

492 + raise ValueError("HF max batch size is required for HF backend")

493 + if args.backend != "hf" and args.hf_max_batch_size is not None:

494 + raise ValueError("HF max batch size is only for HF backend.")

495 +

496 + if args.backend in {"hf", "mii"} and getattr(args, "quantization",

497 + None) is not None:

498 + raise ValueError("Quantization is only for vLLM backend.")

499 +

500 + if args.backend == "mii" and args.dtype != "auto":

501 + raise ValueError("dtype must be auto for MII backend.")

502 + if args.backend == "mii" and args.n != 1:

503 + raise ValueError("n must be 1 for MII backend.")

504 + if args.backend == "mii" and args.tokenizer != args.model:

505 + raise ValueError(

506 + "Tokenizer must be the same as the model for MII backend.")

507 +

508 +

509 if __name__ == "__main__":

510 parser = FlexibleArgumentParser(description="Benchmark the throughput.")

511 parser.add_argument("--backend",

512 type=str,

348 - choices=["vllm", "hf", "mii"],

513 + choices=["vllm", "hf", "mii", "vllm-chat"],

514 default="vllm")

350 - parser.add_argument("--dataset-name",

351 - type=str,

352 - choices=["sharegpt", "random", "sonnet", "burstgpt"],

353 - help="Name of the dataset to benchmark on.",

354 - default="sharegpt")

515 + parser.add_argument(

516 + "--dataset-name",

517 + type=str,

518 + choices=["sharegpt", "random", "sonnet", "burstgpt", "hf"],

519 + help="Name of the dataset to benchmark on.",

520 + default="sharegpt")

521 parser.add_argument(

522 "--dataset",

523 type=str,

@@ -419,55 +585,24 @@ if __name__ == "__main__":

585 parser.add_argument(

586 "--random-range-ratio",

587 type=float,

422 - default=1.0,

588 + default=None,

589 help="Range of sampled ratio of input/output length, "

590 "used only for RandomDataSet.",

591 )

592

593 + # hf dtaset

594 + parser.add_argument("--hf-subset",

595 + type=str,

596 + default=None,

597 + help="Subset of the HF dataset.")

598 + parser.add_argument("--hf-split",

599 + type=str,

600 + default=None,

601 + help="Split of the HF dataset.")

602 +

603 parser = AsyncEngineArgs.add_cli_args(parser)

604 args = parser.parse_args()

605 if args.tokenizer is None:

606 args.tokenizer = args.model

431 - if args.dataset is not None:

432 - warnings.warn(

433 - "The '--dataset' argument will be deprecated in the next "

434 - "release. Please use '--dataset-name' and "

435 - "'--dataset-path' in the future runs.",

436 - stacklevel=2)

437 - args.dataset_path = args.dataset

438 - if args.dataset is None and args.dataset_path is None:

439 - # for random dataset, the default sampling setting is in

440 - # benchmark_dataset.RandomDataset

441 - print("When dataset is not set, it will default to random dataset")

442 - else:

443 - assert args.input_len is None

444 - if args.enable_lora:

445 - assert args.lora_path is not None

446 -

447 - if args.backend == "vllm":

448 - if args.hf_max_batch_size is not None:

449 - raise ValueError("HF max batch size is only for HF backend.")

450 - elif args.backend == "hf":

451 - if args.hf_max_batch_size is None:

452 - raise ValueError("HF max batch size is required for HF backend.")

453 - if args.quantization is not None:

454 - raise ValueError("Quantization is only for vLLM backend.")

455 - if args.enable_lora is not None:

456 - raise ValueError("LoRA benchmarking is only supported for vLLM"

457 - " backend")

458 - elif args.backend == "mii":

459 - if args.dtype != "auto":

460 - raise ValueError("dtype must be auto for MII backend.")

461 - if args.n != 1:

462 - raise ValueError("n must be 1 for MII backend.")

463 - if args.quantization is not None:

464 - raise ValueError("Quantization is only for vLLM backend.")

465 - if args.hf_max_batch_size is not None:

466 - raise ValueError("HF max batch size is only for HF backend.")

467 - if args.tokenizer != args.model:

468 - raise ValueError("Tokenizer must be the same as the model for MII "

469 - "backend.")

470 - if args.enable_lora is not None:

471 - raise ValueError("LoRA benchmarking is only supported for vLLM"

472 - " backend")

607 + validate_args(args)

608 main(args)