A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 17:15 UTC
[Benchmark] Update Vision Arena Dataset and HuggingFaceDataset Setup (#15748)
Signed-off-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>
Jennifer Zhao committed
Mar 31, 2025 at 00:38 UTC
effc5d24fae10b29996256eb7a88668ff7941aed
4 files changed
+227
-215
benchmarks/README.md
+134
-106
@@ -41,29 +41,33 @@ become available.
41
<td><code>synthetic</code></td>
42
</tr>
43
<tr>
44
- <td><strong>HuggingFace</strong></td>
45
- <td style="text-align: center;">🟡</td>
46
- <td style="text-align: center;">🟡</td>
47
- <td>Specify your dataset path on HuggingFace</td>
44
+ <td><strong>HuggingFace-VisionArena</strong></td>
45
+ <td style="text-align: center;">✅</td>
46
+ <td style="text-align: center;">✅</td>
47
+ <td><code>lmarena-ai/VisionArena-Chat</code></td>
48
+ </tr>
49
+ <tr>
50
+ <td><strong>HuggingFace-InstructCoder</strong></td>
51
+ <td style="text-align: center;">✅</td>
52
+ <td style="text-align: center;">✅</td>
53
+ <td><code>likaixin/InstructCoder</code></td>
54
</tr>
55
<tr>
50
- <td><strong>VisionArena</strong></td>
56
+ <td><strong>HuggingFace-Other</strong></td>
57
<td style="text-align: center;">✅</td>
58
<td style="text-align: center;">✅</td>
53
- <td><code>lmarena-ai/vision-arena-bench-v0.1</code> (a HuggingFace dataset)</td>
59
+ <td><code>lmms-lab/LLaVA-OneVision-Data</code>, <code>Aeala/ShareGPT_Vicuna_unfiltered</code></td>
60
</tr>
61
</tbody>
62
</table>
63
64
✅: supported
65
60
-🚧: to be supported
66
+🟡: Partial support
67
62
-🟡: Partial support. Currently, HuggingFaceDataset only supports dataset formats
63
-similar to `lmms-lab/LLaVA-OneVision-Data` and `Aeala/ShareGPT_Vicuna_unfiltered`.
64
-If you need support for other dataset formats, please consider contributing.
68
+🚧: to be supported
69
66
-**Note**: VisionArena’s `dataset-name` should be set to `hf`
70
+**Note**: HuggingFace dataset's `dataset-name` should be set to `hf`
71
72
---
73
## Example - Online Benchmark
@@ -71,8 +75,7 @@ If you need support for other dataset formats, please consider contributing.
75
First start serving your model
76
77
```bash
74
-MODEL_NAME="NousResearch/Hermes-3-Llama-3.1-8B"
75
-vllm serve ${MODEL_NAME} --disable-log-requests
78
+vllm serve NousResearch/Hermes-3-Llama-3.1-8B --disable-log-requests
79
```
80
81
Then run the benchmarking script
@@ -80,12 +83,13 @@ Then run the benchmarking script
83
```bash
84
# download dataset
85
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
83
-MODEL_NAME="NousResearch/Hermes-3-Llama-3.1-8B"
84
-NUM_PROMPTS=10
85
-BACKEND="vllm"
86
-DATASET_NAME="sharegpt"
87
-DATASET_PATH="<your data path>/ShareGPT_V3_unfiltered_cleaned_split.json"
88
-python3 vllm/benchmarks/benchmark_serving.py --backend ${BACKEND} --model ${MODEL_NAME} --endpoint /v1/completions --dataset-name ${DATASET_NAME} --dataset-path ${DATASET_PATH} --num-prompts ${NUM_PROMPTS}
86
+python3 vllm/benchmarks/benchmark_serving.py \
87
+ --backend vllm \
88
+ --model NousResearch/Hermes-3-Llama-3.1-8B \
89
+ --endpoint /v1/completions \
90
+ --dataset-name sharegpt \
91
+ --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
92
+ --num-prompts 10
93
```
94
95
If successful, you will see the following output
@@ -122,88 +126,76 @@ vllm serve Qwen/Qwen2-VL-7B-Instruct --disable-log-requests
126
```
127
128
```bash
125
-MODEL_NAME="Qwen/Qwen2-VL-7B-Instruct"
126
-NUM_PROMPTS=10
127
-BACKEND="openai-chat"
128
-DATASET_NAME="hf"
129
-DATASET_PATH="lmarena-ai/vision-arena-bench-v0.1"
130
-DATASET_SPLIT='train'
131
-
129
python3 vllm/benchmarks/benchmark_serving.py \
133
- --backend "${BACKEND}" \
134
- --model "${MODEL_NAME}" \
135
- --endpoint "/v1/chat/completions" \
136
- --dataset-name "${DATASET_NAME}" \
137
- --dataset-path "${DATASET_PATH}" \
138
- --hf-split "${DATASET_SPLIT}" \
139
- --num-prompts "${NUM_PROMPTS}"
130
+ --backend openai-chat \
131
+ --model Qwen/Qwen2-VL-7B-Instruct \
132
+ --endpoint /v1/chat/completions \
133
+ --dataset-name hf \
134
+ --dataset-path lmarena-ai/VisionArena-Chat \
135
+ --hf-split train \
136
+ --num-prompts 1000
137
```
138
142
-### HuggingFaceDataset Examples
139
+### InstructCoder Benchmark with Speculative Decoding
140
144
-Currently, HuggingFaceDataset only supports dataset formats
145
-similar to `lmms-lab/LLaVA-OneVision-Data` and `Aeala/ShareGPT_Vicuna_unfiltered`. If you need support for other dataset
146
-formats, please consider contributing.
141
+``` bash
142
+VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
143
+ --speculative-model "[ngram]" \
144
+ --ngram_prompt_lookup_min 2 \
145
+ --ngram-prompt-lookup-max 5 \
146
+ --num_speculative_tokens 5
147
+```
148
+
149
+``` bash
150
+python3 benchmarks/benchmark_serving.py \
151
+ --model meta-llama/Meta-Llama-3-8B-Instruct \
152
+ --dataset-name hf \
153
+ --dataset-path likaixin/InstructCoder \
154
+ --num-prompts 2048
155
+```
156
+
157
+### Other HuggingFaceDataset Examples
158
159
```bash
149
-# need a model with vision capability here
160
vllm serve Qwen/Qwen2-VL-7B-Instruct --disable-log-requests
161
```
162
163
**`lmms-lab/LLaVA-OneVision-Data`**
164
165
```bash
156
-MODEL_NAME="Qwen/Qwen2-VL-7B-Instruct"
157
-NUM_PROMPTS=10
158
-BACKEND="openai-chat"
159
-DATASET_NAME="hf"
160
-DATASET_PATH="lmms-lab/LLaVA-OneVision-Data"
161
-DATASET_SPLIT='train'
162
-DATASET_SUBSET='chart2text(cauldron)'
166
python3 vllm/benchmarks/benchmark_serving.py \
164
- --backend "${BACKEND}" \
165
- --model "${MODEL_NAME}" \
166
- --endpoint "/v1/chat/completions" \
167
- --dataset-name "${DATASET_NAME}" \
168
- --dataset-path "${DATASET_PATH}" \
169
- --hf-split "${DATASET_SPLIT}" \
170
- --num-prompts "${NUM_PROMPTS}" \
171
- --hf-subset "${DATASET_SUBSET}"
167
+ --backend openai-chat \
168
+ --model Qwen/Qwen2-VL-7B-Instruct \
169
+ --endpoint /v1/chat/completions \
170
+ --dataset-name hf \
171
+ --dataset-path lmms-lab/LLaVA-OneVision-Data \
172
+ --hf-split train \
173
+ --hf-subset "chart2text(cauldron)" \
174
+ --num-prompts 10
175
```
176
177
**`Aeala/ShareGPT_Vicuna_unfiltered`**
178
179
```bash
177
-MODEL_NAME="Qwen/Qwen2-VL-7B-Instruct"
178
-NUM_PROMPTS=10
179
-BACKEND="openai-chat"
180
-DATASET_NAME="hf"
181
-DATASET_PATH="Aeala/ShareGPT_Vicuna_unfiltered"
182
-DATASET_SPLIT='train'
180
python3 vllm/benchmarks/benchmark_serving.py \
184
- --backend "${BACKEND}" \
185
- --model "${MODEL_NAME}" \
186
- --endpoint "/v1/chat/completions" \
187
- --dataset-name "${DATASET_NAME}" \
188
- --dataset-path "${DATASET_PATH}" \
189
- --hf-split "${DATASET_SPLIT}" \
190
- --num-prompts "${NUM_PROMPTS}" \
181
+ --backend openai-chat \
182
+ --model Qwen/Qwen2-VL-7B-Instruct \
183
+ --endpoint /v1/chat/completions \
184
+ --dataset-name hf \
185
+ --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
186
+ --hf-split train \
187
+ --num-prompts 10
188
```
189
190
---
191
## Example - Offline Throughput Benchmark
192
193
```bash
197
-MODEL_NAME="NousResearch/Hermes-3-Llama-3.1-8B"
198
-NUM_PROMPTS=10
199
-DATASET_NAME="sonnet"
200
-DATASET_PATH="vllm/benchmarks/sonnet.txt"
201
-
194
python3 vllm/benchmarks/benchmark_throughput.py \
203
- --model "${MODEL_NAME}" \
204
- --dataset-name "${DATASET_NAME}" \
205
- --dataset-path "${DATASET_PATH}" \
206
- --num-prompts "${NUM_PROMPTS}"
195
+ --model NousResearch/Hermes-3-Llama-3.1-8B \
196
+ --dataset-name sonnet \
197
+ --dataset-path vllm/benchmarks/sonnet.txt \
198
+ --num-prompts 10
199
```
200
201
If successful, you will see the following output
@@ -217,19 +209,13 @@ Total num output tokens: 1500
209
### VisionArena Benchmark for Vision Language Models
210
211
``` bash
220
-MODEL_NAME="Qwen/Qwen2-VL-7B-Instruct"
221
-NUM_PROMPTS=10
222
-DATASET_NAME="hf"
223
-DATASET_PATH="lmarena-ai/vision-arena-bench-v0.1"
224
-DATASET_SPLIT="train"
225
-
212
python3 vllm/benchmarks/benchmark_throughput.py \
227
- --model "${MODEL_NAME}" \
228
- --backend "vllm-chat" \
229
- --dataset-name "${DATASET_NAME}" \
230
- --dataset-path "${DATASET_PATH}" \
231
- --num-prompts "${NUM_PROMPTS}" \
232
- --hf-split "${DATASET_SPLIT}"
213
+ --model Qwen/Qwen2-VL-7B-Instruct \
214
+ --backend vllm-chat \
215
+ --dataset-name hf \
216
+ --dataset-path lmarena-ai/VisionArena-Chat \
217
+ --num-prompts 1000 \
218
+ --hf-split train
219
```
220
221
The `num prompt tokens` now includes image token counts
@@ -240,29 +226,71 @@ Total num prompt tokens: 14527
226
Total num output tokens: 1280
227
```
228
229
+### InstructCoder Benchmark with Speculative Decoding
230
+
231
+``` bash
232
+VLLM_WORKER_MULTIPROC_METHOD=spawn \
233
+VLLM_USE_V1=1 \
234
+python3 vllm/benchmarks/benchmark_throughput.py \
235
+ --dataset-name=hf \
236
+ --dataset-path=likaixin/InstructCoder \
237
+ --model=meta-llama/Meta-Llama-3-8B-Instruct \
238
+ --input-len=1000 \
239
+ --output-len=100 \
240
+ --num-prompts=2048 \
241
+ --async-engine \
242
+ --speculative-model="[ngram]" \
243
+ --ngram_prompt_lookup_min=2 \
244
+ --ngram-prompt-lookup-max=5 \
245
+ --num_speculative_tokens=5
246
+```
247
+
248
+```
249
+Throughput: 104.77 requests/s, 23836.22 total tokens/s, 10477.10 output tokens/s
250
+Total num prompt tokens: 261136
251
+Total num output tokens: 204800
252
+```
253
+
254
+### Other HuggingFaceDataset Examples
255
+
256
+**`lmms-lab/LLaVA-OneVision-Data`**
257
+
258
+```bash
259
+python3 vllm/benchmarks/benchmark_throughput.py \
260
+ --model Qwen/Qwen2-VL-7B-Instruct \
261
+ --backend vllm-chat \
262
+ --dataset-name hf \
263
+ --dataset-path lmms-lab/LLaVA-OneVision-Data \
264
+ --hf-split train \
265
+ --hf-subset "chart2text(cauldron)" \
266
+ --num-prompts 10
267
+```
268
+
269
+**`Aeala/ShareGPT_Vicuna_unfiltered`**
270
+
271
+```bash
272
+python3 vllm/benchmarks/benchmark_throughput.py \
273
+ --model Qwen/Qwen2-VL-7B-Instruct \
274
+ --backend vllm-chat \
275
+ --dataset-name hf \
276
+ --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
277
+ --hf-split train \
278
+ --num-prompts 10
279
+```
280
+
281
### Benchmark with LoRA Adapters
282
283
``` bash
284
# download dataset
285
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
248
-MODEL_NAME="meta-llama/Llama-2-7b-hf"
249
-BACKEND="vllm"
250
-DATASET_NAME="sharegpt"
251
-DATASET_PATH="<your data path>/ShareGPT_V3_unfiltered_cleaned_split.json"
252
-NUM_PROMPTS=10
253
-MAX_LORAS=2
254
-MAX_LORA_RANK=8
255
-ENABLE_LORA="--enable-lora"
256
-LORA_PATH="yard1/llama-2-7b-sql-lora-test"
257
-
286
python3 vllm/benchmarks/benchmark_throughput.py \
259
- --model "${MODEL_NAME}" \
260
- --backend "${BACKEND}" \
261
- --dataset_path "${DATASET_PATH}" \
262
- --dataset_name "${DATASET_NAME}" \
263
- --num-prompts "${NUM_PROMPTS}" \
264
- --max-loras "${MAX_LORAS}" \
265
- --max-lora-rank "${MAX_LORA_RANK}" \
266
- ${ENABLE_LORA} \
267
- --lora-path "${LORA_PATH}"
287
+ --model meta-llama/Llama-2-7b-hf \
288
+ --backend vllm \
289
+ --dataset_path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
290
+ --dataset_name sharegpt \
291
+ --num-prompts 10 \
292
+ --max-loras 2 \
293
+ --max-lora-rank 8 \
294
+ --enable-lora \
295
+ --lora-path yard1/llama-2-7b-sql-lora-test
296
```
benchmarks/benchmark_dataset.py
+69
-86
@@ -23,7 +23,8 @@ from abc import ABC, abstractmethod
23
from collections.abc import Mapping
24
from dataclasses import dataclass
25
from functools import cache
26
-from typing import Any, Optional, Union
26
+from io import BytesIO
27
+from typing import Any, Callable, Optional, Union
28
29
import numpy as np
30
import pandas as pd
@@ -239,21 +240,24 @@ def process_image(image: Any) -> Mapping[str, Any]:
240
"""
241
Process a single image input and return a multimedia content dictionary.
242
242
- For a PIL.Image.Image input:
243
- - Converts the image to RGB.
244
- - Saves the image as a JPEG in-memory.
245
- - Encodes the JPEG data as a base64 string.
246
- - Returns a dictionary with the image as a base64 data URL.
243
+ Supports three input types:
244
248
- For a string input:
249
- - Treats the string as a URL or file path.
250
- - Prepends "file://" if the string doesn't start with "http://" or
251
- "file://".
252
- - Returns a dictionary with the image URL.
245
+ 1. Dictionary with raw image bytes: - Expects a dict with a 'bytes' key
246
+ containing raw image data. - Loads the bytes as a PIL.Image.Image.
247
+
248
+ 2. PIL.Image.Image input: - Converts the image to RGB. - Saves the image as
249
+ a JPEG in memory. - Encodes the JPEG data as a base64 string. - Returns
250
+ a dictionary with the image as a base64 data URL.
251
+
252
+ 3. String input: - Treats the string as a URL or local file path. -
253
+ Prepends "file://" if the string doesn't start with "http://" or
254
+ "file://". - Returns a dictionary with the image URL.
255
256
Raises:
255
- ValueError: If the input is neither a PIL.Image.Image nor a string.
257
+ ValueError: If the input is not a supported type.
258
"""
259
+ if isinstance(image, dict) and 'bytes' in image:
260
+ image = Image.open(BytesIO(image['bytes']))
261
if isinstance(image, Image.Image):
262
image = image.convert("RGB")
263
with io.BytesIO() as image_data:
@@ -272,8 +276,8 @@ def process_image(image: Any) -> Mapping[str, Any]:
276
("http://", "file://")) else f"file://{image}")
277
return {"type": "image_url", "image_url": {"url": image_url}}
278
275
- raise ValueError(
276
- f"Invalid image input {image}. Must be a PIL.Image.Image or str.")
279
+ raise ValueError(f"Invalid image input {image}. Must be a PIL.Image.Image"
280
+ " or str or dictionary with raw image bytes.")
281
282
283
# -----------------------------------------------------------------------------
@@ -562,48 +566,56 @@ class BurstGPTDataset(BenchmarkDataset):
566
567
568
# -----------------------------------------------------------------------------
565
-# HuggingFace Dataset Implementation
569
+# HuggingFace Dataset Base Implementation
570
# -----------------------------------------------------------------------------
567
-
568
-
571
class HuggingFaceDataset(BenchmarkDataset):
570
- """
571
- Dataset class for processing a HuggingFace dataset with conversation data
572
- and optional images.
573
- """
572
+ """Base class for datasets hosted on HuggingFace."""
573
+
574
+ SUPPORTED_DATASET_PATHS: Union[set[str], dict[str, Callable]] = set()
575
576
def __init__(
577
self,
578
+ dataset_path: str,
579
dataset_split: str,
580
dataset_subset: Optional[str] = None,
581
**kwargs,
582
) -> None:
581
- super().__init__(**kwargs)
583
+ super().__init__(dataset_path=dataset_path, **kwargs)
584
+
585
+ # Validate dataset path
586
+ if self.SUPPORTED_DATASET_PATHS and \
587
+ self.dataset_path not in self.SUPPORTED_DATASET_PATHS:
588
+ raise ValueError(
589
+ f"{self.__class__.__name__} "
590
+ f"only supports: {', '.join(self.SUPPORTED_DATASET_PATHS)}. "
591
+ "Please consider contributing if you would "
592
+ "like to add support for additional dataset formats.")
593
+
594
self.dataset_split = dataset_split
595
self.dataset_subset = dataset_subset
584
-
596
self.load_data()
597
598
def load_data(self) -> None:
588
- if not self.dataset_path:
589
- raise ValueError("dataset_path must be provided for loading data.")
590
-
599
+ """Load data from HuggingFace datasets."""
600
self.data = load_dataset(
601
self.dataset_path,
602
name=self.dataset_subset,
603
split=self.dataset_split,
604
streaming=True,
605
)
597
- if self.data.features is None or "conversations" \
598
- not in self.data.features:
599
- raise ValueError(
600
- "HuggingFaceDataset currently only supports datasets with "
601
- "a 'conversations' column like lmms-lab/LLaVA-OneVision-Data. "
602
- "Please consider contributing if you would like to add "
603
- "support for additional dataset formats.")
604
- # Shuffle and filter examples with at least 2 conversations.
605
- self.data = self.data.shuffle(seed=self.random_seed).filter(
606
- lambda x: len(x["conversations"]) >= 2)
606
+ self.data = self.data.shuffle(seed=self.random_seed)
607
+
608
+
609
+# -----------------------------------------------------------------------------
610
+# Conversation Dataset Implementation
611
+# -----------------------------------------------------------------------------
612
+
613
+
614
+class ConversationDataset(HuggingFaceDataset):
615
+ """Dataset for conversation data with multimodal support."""
616
+ SUPPORTED_DATASET_PATHS = {
617
+ 'lmms-lab/LLaVA-OneVision-Data', 'Aeala/ShareGPT_Vicuna_unfiltered'
618
+ }
619
620
def sample(self,
621
tokenizer: PreTrainedTokenizerBase,
@@ -611,10 +623,13 @@ class HuggingFaceDataset(BenchmarkDataset):
623
output_len: Optional[int] = None,
624
enable_multimodal_chat: bool = False,
625
**kwargs) -> list:
626
+ # Filter examples with at least 2 conversations
627
+ filtered_data = self.data.filter(
628
+ lambda x: len(x["conversations"]) >= 2)
629
sampled_requests = []
630
dynamic_output = output_len is None
631
617
- for item in self.data:
632
+ for item in filtered_data:
633
if len(sampled_requests) >= num_requests:
634
break
635
conv = item["conversations"]
@@ -659,29 +674,12 @@ class VisionArenaDataset(HuggingFaceDataset):
674
"""
675
676
DEFAULT_OUTPUT_LEN = 128
662
- VISION_ARENA_DATASET_PATH = "lmarena-ai/vision-arena-bench-v0.1"
663
-
664
- def __init__(
665
- self,
666
- **kwargs,
667
- ) -> None:
668
- super().__init__(**kwargs)
669
- if self.dataset_path != self.VISION_ARENA_DATASET_PATH:
670
- raise ValueError(f"Only support Vision Arena dataset.\
671
- This data path {self.dataset_path} is not valid.")
672
- if self.dataset_subset is None and self.dataset_split != "train":
673
- raise ValueError("Dataset split must be 'train'.")
674
-
675
- self.load_data()
676
-
677
- def load_data(self) -> None:
678
- dataset = load_dataset(
679
- self.dataset_path,
680
- name=self.dataset_subset,
681
- split=self.dataset_split,
682
- streaming=True,
683
- )
684
- self.data = dataset.shuffle(seed=self.random_seed)
677
+ SUPPORTED_DATASET_PATHS = {
678
+ "lmarena-ai/VisionArena-Chat":
679
+ lambda x: x["conversation"][0][0]["content"],
680
+ "lmarena-ai/vision-arena-bench-v0.1":
681
+ lambda x: x["turns"][0][0]["content"]
682
+ }
683
684
def sample(
685
self,
@@ -697,7 +695,11 @@ class VisionArenaDataset(HuggingFaceDataset):
695
for item in self.data:
696
if len(sampled_requests) >= num_requests:
697
break
700
- prompt = item["turns"][0][0]["content"]
698
+ parser_fn = self.SUPPORTED_DATASET_PATHS.get(self.dataset_path)
699
+ if parser_fn is None:
700
+ raise ValueError(
701
+ f"Unsupported dataset path: {self.dataset_path}")
702
+ prompt = parser_fn(item)
703
mm_content = process_image(item["images"][0])
704
prompt_len = len(tokenizer(prompt).input_ids)
705
if enable_multimodal_chat:
@@ -727,34 +729,15 @@ class InstructCoderDataset(HuggingFaceDataset):
729
InstructCoder Dataset.
730
https://huggingface.co/datasets/likaixin/InstructCoder
731
730
- InstructCoder is the dataset designed for general code editing.
731
- It consists of 114,239 instruction-input-output triplets,
732
- and covers multiple distinct code editing scenario.
732
+ InstructCoder is the dataset designed for general code editing. It consists
733
+ of 114,239 instruction-input-output triplets, and covers multiple distinct
734
+ code editing scenario.
735
"""
736
737
DEFAULT_OUTPUT_LEN = 200 # this is the average default output length
736
- DEFAULT_NUM_REQUESTS = 1000
737
- INSTRUCT_CODER_DATASET_PATH = "likaixin/InstructCoder"
738
-
739
- def __init__(
740
- self,
741
- **kwargs,
742
- ) -> None:
743
- super().__init__(**kwargs)
744
- if self.dataset_path != self.INSTRUCT_CODER_DATASET_PATH:
745
- raise ValueError(f"Only support likaixin/InstructCoder dataset.\
746
- This data path {self.dataset_path} is not valid.")
747
- if self.dataset_subset is None and self.dataset_split != "train":
748
- raise ValueError("Dataset split must be 'train'.")
749
-
750
- def load_data(self) -> None:
751
- dataset = load_dataset(
752
- self.dataset_path,
753
- name=self.dataset_subset,
754
- split=self.dataset_split,
755
- streaming=True,
756
- )
757
- self.data = dataset.shuffle(seed=self.random_seed)
738
+ SUPPORTED_DATASET_PATHS = {
739
+ "likaixin/InstructCoder",
740
+ }
741
742
def sample(self,
743
tokenizer: PreTrainedTokenizerBase,
benchmarks/benchmark_serving.py
+9
-8
@@ -49,7 +49,7 @@ try:
49
except ImportError:
50
from argparse import ArgumentParser as FlexibleArgumentParser
51
52
-from benchmark_dataset import (BurstGPTDataset, HuggingFaceDataset,
52
+from benchmark_dataset import (BurstGPTDataset, ConversationDataset,
53
InstructCoderDataset, RandomDataset,
54
SampleRequest, ShareGPTDataset, SonnetDataset,
55
VisionArenaDataset)
@@ -584,16 +584,17 @@ def main(args: argparse.Namespace):
584
return_prompt_formatted=True)
585
586
elif args.dataset_name == "hf":
587
- # Choose between VisionArenaDataset
588
- # and HuggingFaceDataset based on provided parameters.
589
- dataset_class = HuggingFaceDataset
590
- if args.dataset_path == VisionArenaDataset.VISION_ARENA_DATASET_PATH:
591
- assert args.hf_subset is None, "VisionArenaDataset needs hf_subset to be None." #noqa: E501
587
+ # all following datasets are implemented from the
588
+ # HuggingFaceDataset base class
589
+ if args.dataset_path in VisionArenaDataset.SUPPORTED_DATASET_PATHS:
590
dataset_class = VisionArenaDataset
593
- elif args.dataset_path == "likaixin/InstructCoder":
591
+ args.hf_split = "train"
592
+ args.hf_subset = None
593
+ elif args.dataset_path in InstructCoderDataset.SUPPORTED_DATASET_PATHS:
594
dataset_class = InstructCoderDataset
595
args.hf_split = "train"
596
-
596
+ elif args.dataset_path in ConversationDataset.SUPPORTED_DATASET_PATHS:
597
+ dataset_class = ConversationDataset
598
input_requests = dataset_class(
599
dataset_path=args.dataset_path,
600
dataset_subset=args.hf_subset,
benchmarks/benchmark_throughput.py
+15
-15
@@ -11,7 +11,7 @@ from typing import Any, Optional, Union
11
12
import torch
13
import uvloop
14
-from benchmark_dataset import (BurstGPTDataset, HuggingFaceDataset,
14
+from benchmark_dataset import (BurstGPTDataset, ConversationDataset,
15
InstructCoderDataset, RandomDataset,
16
SampleRequest, ShareGPTDataset, SonnetDataset,
17
VisionArenaDataset)
@@ -319,21 +319,19 @@ def get_requests(args, tokenizer):
319
elif args.dataset_name == "burstgpt":
320
dataset_cls = BurstGPTDataset
321
elif args.dataset_name == "hf":
322
- if args.dataset_path == VisionArenaDataset.VISION_ARENA_DATASET_PATH:
323
- if args.args.backend == "vllm-chat":
324
- raise ValueError(
325
- "hf datasets only are supported by vllm-chat backend")
326
- # Choose between VisionArenaDataset and HuggingFaceDataset based on
327
- # provided parameters.
328
- dataset_cls = (VisionArenaDataset if args.dataset_path
329
- == VisionArenaDataset.VISION_ARENA_DATASET_PATH
330
- and args.hf_subset is None else HuggingFaceDataset)
331
- common_kwargs['dataset_subset'] = args.hf_subset
332
- common_kwargs['dataset_split'] = args.hf_split
322
+ if args.dataset_path in VisionArenaDataset.SUPPORTED_DATASET_PATHS:
323
+ dataset_cls = VisionArenaDataset
324
+ common_kwargs['dataset_subset'] = None
325
+ common_kwargs['dataset_split'] = "train"
326
sample_kwargs["enable_multimodal_chat"] = True
334
- elif args.dataset_path == "likaixin/InstructCoder":
327
+ elif args.dataset_path in InstructCoderDataset.SUPPORTED_DATASET_PATHS:
328
dataset_cls = InstructCoderDataset
329
common_kwargs['dataset_split'] = "train"
330
+ elif args.dataset_path in ConversationDataset.SUPPORTED_DATASET_PATHS:
331
+ dataset_cls = ConversationDataset
332
+ common_kwargs['dataset_subset'] = args.hf_subset
333
+ common_kwargs['dataset_split'] = args.hf_split
334
+ sample_kwargs["enable_multimodal_chat"] = True
335
336
else:
337
raise ValueError(f"Unknown dataset name: {args.dataset_name}")
@@ -469,10 +467,12 @@ def validate_args(args):
467
since --dataset-name is not 'hf'.",
468
stacklevel=2)
469
elif args.dataset_name == "hf":
472
- if args.dataset_path == VisionArenaDataset.VISION_ARENA_DATASET_PATH:
470
+ if args.dataset_path in VisionArenaDataset.SUPPORTED_DATASET_PATHS:
471
assert args.backend == "vllm-chat", "VisionArenaDataset needs to use vllm-chat as the backend." #noqa: E501
474
- elif args.dataset_path == "likaixin/InstructCoder":
472
+ elif args.dataset_path in InstructCoderDataset.SUPPORTED_DATASET_PATHS:
473
assert args.backend == "vllm", "InstructCoder dataset needs to use vllm as the backend." #noqa: E501
474
+ elif args.dataset_path in ConversationDataset.SUPPORTED_DATASET_PATHS:
475
+ assert args.backend == "vllm-chat", "ConversationDataset needs to use vllm-chat as the backend." #noqa: E501
476
else:
477
raise ValueError(
478
f"{args.dataset_path} is not supported by hf dataset.")