@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 0cb7b065c3
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

Feature/benchmark/random mm data/images (#23119)

Signed-off-by: breno.skuk <breno.skuk@hcompany.ai>

Breno Baldas Skuk committed Aug 25, 2025 at 10:28 UTC 0cb7b065c3c2c3e9fa269f48fb4e945616f7f5b9
3 files changed +1051 -56
benchmarks/README.md
+77
@@ -59,6 +59,12 @@ become available.
59 <td style="text-align: center;">✅</td>
60 <td><code>synthetic</code></td>
61 </tr>
62 + <tr>
63 + <td><strong>RandomMultiModal (Image/Video)</strong></td>
64 + <td style="text-align: center;">🟡</td>
65 + <td style="text-align: center;">🚧</td>
66 + <td><code>synthetic</code> </td>
67 + </tr>
68 <tr>
69 <td><strong>Prefix Repetition</strong></td>
70 <td style="text-align: center;">✅</td>
@@ -722,4 +728,75 @@ python benchmarks/benchmark_serving.py \
728 --endpoint /v1/chat/completion
729 ```
730
731 +### Synthetic Random Images (random-mm)
732 +
733 +Generate synthetic image inputs alongside random text prompts to stress-test vision models without external datasets.
734 +
735 +Notes:
736 +
737 +- Works only with online benchmark via the OpenAI backend (`--backend openai-chat`) and endpoint `/v1/chat/completions`.
738 +- Video sampling is not yet implemented.
739 +
740 +Start the server (example):
741 +
742 +```bash
743 +vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
744 + --dtype bfloat16 \
745 + --max-model-len 16384 \
746 + --limit-mm-per-prompt '{"image": 3, "video": 0}' \
747 + --mm-processor-kwargs max_pixels=1003520
748 +```
749 +
750 +Benchmark. It is recommended to use the flag `--ignore-eos` to simulate real responses. You can set the size of the output via the arg `random-output-len`.
751 +
752 +Ex.1: Fixed number of items and a single image resolutionm, enforcing generation of approx 40 tokens:
753 +
754 +```bash
755 +vllm bench serve \
756 + --backend openai-chat \
757 + --model Qwen/Qwen2.5-VL-3B-Instruct \
758 + --endpoint /v1/chat/completions \
759 + --dataset-name random-mm \
760 + --num-prompts 100 \
761 + --max-concurrency 10 \
762 + --random-prefix-len 25 \
763 + --random-input-len 300 \
764 + --random-output-len 40 \
765 + --random-range-ratio 0.2 \
766 + --random-mm-base-items-per-request 2 \
767 + --random-mm-limit-mm-per-prompt '{"image": 3, "video": 0}' \
768 + --random-mm-bucket-config '{(224, 224, 1): 1.0}' \
769 + --request-rate inf \
770 + --ignore-eos \
771 + --seed 42
772 +```
773 +
774 +The number of items per request can be controlled by passing multiple image buckets:
775 +
776 +```bash
777 + --random-mm-base-items-per-request 2 \
778 + --random-mm-num-mm-items-range-ratio 0.5 \
779 + --random-mm-limit-mm-per-prompt '{"image": 4, "video": 0}' \
780 + --random-mm-bucket-config '{(256, 256, 1): 0.7, (720, 1280, 1): 0.3}' \
781 +```
782 +
783 +Flags specific to `random-mm`:
784 +
785 +- `--random-mm-base-items-per-request`: base number of multimodal items per request.
786 +- `--random-mm-num-mm-items-range-ratio`: vary item count uniformly in the closed integer range [floor(n·(1−r)), ceil(n·(1+r))]. Set r=0 to keep it fixed; r=1 allows 0 items.
787 +- `--random-mm-limit-mm-per-prompt`: per-modality hard caps, e.g. '{"image": 3, "video": 0}'.
788 +- `--random-mm-bucket-config`: dict mapping (H, W, T) → probability. Entries with probability 0 are removed; remaining probabilities are renormalized to sum to 1. Use T=1 for images. Set any T>1 for videos (video sampling not yet supported).
789 +
790 +Behavioral notes:
791 +
792 +- If the requested base item count cannot be satisfied under the provided per-prompt limits, the tool raises an error rather than silently clamping.
793 +
794 +How sampling works:
795 +
796 +- Determine per-request item count k by sampling uniformly from the integer range defined by `--random-mm-base-items-per-request` and `--random-mm-num-mm-items-range-ratio`, then clamp k to at most the sum of per-modality limits.
797 +- For each of the k items, sample a bucket (H, W, T) according to the normalized probabilities in `--random-mm-bucket-config`, while tracking how many items of each modality have been added.
798 +- If a modality (e.g., image) reaches its limit from `--random-mm-limit-mm-per-prompt`, all buckets of that modality are excluded and the remaining bucket probabilities are renormalized before continuing.
799 +This should be seen as an edge case, and if this behavior can be avoided by setting `--random-mm-limit-mm-per-prompt` to a large number. Note that this might result in errors due to engine config `--limit-mm-per-prompt`.
800 +- The resulting request contains synthetic image data in `multi_modal_data` (OpenAI Chat format). When `random-mm` is used with the OpenAI Chat backend, prompts remain text and MM content is attached via `multi_modal_data`.
801 +
802 </details>
tests/benchmarks/test_random_dataset.py new
+344
@@ -0,0 +1,344 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +import random
4 +from typing import Any, NamedTuple, Optional, cast
5 +
6 +import numpy as np
7 +import pytest
8 +from transformers import AutoTokenizer, PreTrainedTokenizerBase
9 +
10 +from vllm.benchmarks.datasets import (RandomDataset, RandomMultiModalDataset,
11 + SampleRequest)
12 +
13 +
14 +@pytest.fixture(scope="session")
15 +def hf_tokenizer() -> PreTrainedTokenizerBase:
16 + # Use a small, commonly available tokenizer
17 + return AutoTokenizer.from_pretrained("gpt2")
18 +
19 +
20 +class Params(NamedTuple):
21 + num_requests: int
22 + prefix_len: int
23 + range_ratio: float
24 + input_len: int
25 + output_len: int
26 +
27 +
28 +@pytest.fixture(scope="session")
29 +def random_dataset_params() -> Params:
30 + return Params(num_requests=16,
31 + prefix_len=7,
32 + range_ratio=0.3,
33 + input_len=50,
34 + output_len=20)
35 +
36 +
37 +def _fingerprint_sample(req: SampleRequest) -> tuple[str, int, int]:
38 + """Project a SampleRequest into a comparable tuple."""
39 + return (req.prompt, req.prompt_len, req.expected_output_len)
40 +
41 +
42 +def _collect_samples(dataset: RandomDataset,
43 + tokenizer: PreTrainedTokenizerBase,
44 + num_requests: int = 16,
45 + prefix_len: int = 7,
46 + range_ratio: float = 0.3,
47 + input_len: int = 50,
48 + output_len: int = 20) -> list[tuple[str, int, int]]:
49 + samples = dataset.sample(
50 + tokenizer=tokenizer,
51 + num_requests=num_requests,
52 + prefix_len=prefix_len,
53 + range_ratio=range_ratio,
54 + input_len=input_len,
55 + output_len=output_len,
56 + )
57 + return [_fingerprint_sample(s) for s in samples]
58 +
59 +
60 +@pytest.mark.benchmark
61 +def test_random_dataset_same_seed(
62 + hf_tokenizer: PreTrainedTokenizerBase,
63 + random_dataset_params: Params) -> None:
64 + """Same seed should yield identical outputs, even if global RNGs change.
65 +
66 + This guards against accidental reliance on Python's random or np.random
67 + in RandomDataset after moving to numpy.default_rng.
68 + """
69 + p = random_dataset_params
70 + common_seed = 123
71 + dataset_a = RandomDataset(random_seed=common_seed)
72 + dataset_b = RandomDataset(random_seed=common_seed)
73 + a = _collect_samples(dataset_a,
74 + hf_tokenizer,
75 + num_requests=p.num_requests,
76 + prefix_len=p.prefix_len,
77 + range_ratio=p.range_ratio,
78 + input_len=p.input_len,
79 + output_len=p.output_len)
80 +
81 + # Perturb global RNG state to ensure isolation
82 + random.seed(999)
83 + _ = [random.random() for _ in range(100)]
84 + np.random.seed(888)
85 + _ = [np.random.random() for _ in range(100)]
86 +
87 + b = _collect_samples(dataset_b,
88 + hf_tokenizer,
89 + num_requests=p.num_requests,
90 + prefix_len=p.prefix_len,
91 + range_ratio=p.range_ratio,
92 + input_len=p.input_len,
93 + output_len=p.output_len)
94 + assert a == b
95 +
96 +@pytest.mark.benchmark
97 +def test_random_dataset_different_seeds(
98 + hf_tokenizer: PreTrainedTokenizerBase,
99 + random_dataset_params: Params) -> None:
100 + """Different seeds should change outputs with overwhelming likelihood."""
101 + p = random_dataset_params
102 + seed_a = 0
103 + dataset_a = RandomDataset(random_seed=seed_a)
104 + a = _collect_samples(dataset_a,
105 + hf_tokenizer,
106 + num_requests=p.num_requests,
107 + prefix_len=p.prefix_len,
108 + range_ratio=p.range_ratio,
109 + input_len=p.input_len,
110 + output_len=p.output_len)
111 +
112 + seed_b = 999
113 + dataset_b = RandomDataset(random_seed=seed_b)
114 + # Perturb global RNG with same seed as dataset_a to ensure isolation
115 + random.seed(seed_a)
116 + np.random.seed(seed_a)
117 + b = _collect_samples(dataset_b,
118 + hf_tokenizer,
119 + num_requests=p.num_requests,
120 + prefix_len=p.prefix_len,
121 + range_ratio=p.range_ratio,
122 + input_len=p.input_len,
123 + output_len=p.output_len)
124 + assert a != b
125 +
126 +
127 +# -----------------------------
128 +# RandomMultiModalDataset tests
129 +# -----------------------------
130 +
131 +def _mm_fingerprint_sample(
132 + req: SampleRequest,
133 +) -> tuple[str, int, int, int, list[str]]:
134 + """Create a compact fingerprint for multimodal samples.
135 +
136 + Includes:
137 + - prompt string
138 + - prompt_len
139 + - expected_output_len
140 + - count of multimodal items
141 + - per-item type and URL prefix (e.g., 'data:image/jpeg;base64,')
142 + """
143 + items = req.multi_modal_data or []
144 + item_prefixes: list[str] = []
145 + for it in items:
146 + if isinstance(it, dict) and it.get("type") == "image_url":
147 + url = it.get("image_url", {}).get("url", "")
148 + # Only keep a short identifying prefix to avoid huge strings
149 + item_prefixes.append(f"image:{url[:22]}")
150 + elif isinstance(it, dict) and it.get("type") == "video_url":
151 + url = it.get("video_url", {}).get("url", "")
152 + item_prefixes.append(f"video:{url[:22]}")
153 + else:
154 + item_prefixes.append("unknown:")
155 + return (req.prompt, req.prompt_len, req.expected_output_len, len(items),
156 + item_prefixes)
157 +
158 +
159 +def _collect_mm_samples(
160 + dataset: RandomMultiModalDataset,
161 + tokenizer: PreTrainedTokenizerBase,
162 + *,
163 + num_requests: int = 8,
164 + prefix_len: int = 3,
165 + range_ratio: float = 0.0,
166 + input_len: int = 20,
167 + output_len: int = 5,
168 + base_items_per_request: int = 2,
169 + num_mm_items_range_ratio: float = 0.0,
170 + limit_mm_per_prompt: Optional[dict[str, int]] = None,
171 + bucket_config: Optional[dict[tuple[int, int, int], float]] = None,
172 + enable_multimodal_chat: bool = False,
173 +) -> list[SampleRequest]:
174 + if limit_mm_per_prompt is None:
175 + limit_mm_per_prompt = {"image": 5, "video": 0}
176 + if bucket_config is None:
177 + bucket_config = {(32, 32, 1): 0.5, (52, 64, 1): 0.5}
178 + return dataset.sample(
179 + tokenizer=tokenizer,
180 + num_requests=num_requests,
181 + prefix_len=prefix_len,
182 + range_ratio=range_ratio,
183 + input_len=input_len,
184 + output_len=output_len,
185 + base_items_per_request=base_items_per_request,
186 + num_mm_items_range_ratio=num_mm_items_range_ratio,
187 + limit_mm_per_prompt=limit_mm_per_prompt,
188 + bucket_config=bucket_config,
189 + enable_multimodal_chat=enable_multimodal_chat,
190 + )
191 +
192 +
193 +@pytest.mark.benchmark
194 +def test_random_mm_same_seed(hf_tokenizer: PreTrainedTokenizerBase) -> None:
195 + seed = 42
196 + ds_a = RandomMultiModalDataset(random_seed=seed)
197 + ds_b = RandomMultiModalDataset(random_seed=seed)
198 + a = _collect_mm_samples(ds_a, hf_tokenizer)
199 + b = _collect_mm_samples(ds_b, hf_tokenizer)
200 + fa = [_mm_fingerprint_sample(s) for s in a]
201 + fb = [_mm_fingerprint_sample(s) for s in b]
202 + assert fa == fb
203 +
204 +
205 +@pytest.mark.benchmark
206 +def test_random_mm_different_seeds(
207 + hf_tokenizer: PreTrainedTokenizerBase,
208 +) -> None:
209 + ds_a = RandomMultiModalDataset(random_seed=0)
210 + ds_b = RandomMultiModalDataset(random_seed=999)
211 + a = _collect_mm_samples(ds_a, hf_tokenizer)
212 + b = _collect_mm_samples(ds_b, hf_tokenizer)
213 + fa = [_mm_fingerprint_sample(s) for s in a]
214 + fb = [_mm_fingerprint_sample(s) for s in b]
215 + assert fa != fb
216 +
217 +@pytest.mark.benchmark
218 +def test_random_mm_respects_limits(
219 + hf_tokenizer: PreTrainedTokenizerBase,
220 +) -> None:
221 + ds = RandomMultiModalDataset(random_seed=0)
222 + # Requesting 3 items with a per-prompt limit of 1 should error per current
223 + # design (dataset refuses to silently clamp below the requested baseline).
224 + with pytest.raises(ValueError):
225 + _collect_mm_samples(
226 + ds,
227 + hf_tokenizer,
228 + num_requests=12,
229 + base_items_per_request=3,
230 + num_mm_items_range_ratio=0.0,
231 + limit_mm_per_prompt={"image": 1, "video": 0},
232 + bucket_config={(32, 32, 1): 1.0},
233 + )
234 +
235 +
236 +@pytest.mark.benchmark
237 +def test_random_mm_zero_prob_entries_are_removed(
238 + hf_tokenizer: PreTrainedTokenizerBase,
239 +) -> None:
240 + ds = RandomMultiModalDataset(random_seed=0)
241 + # Second bucket has zero probability and should be ignored after
242 + # normalization
243 + samples = _collect_mm_samples(
244 + ds,
245 + hf_tokenizer,
246 + num_requests=6,
247 + base_items_per_request=2,
248 + num_mm_items_range_ratio=0.0,
249 + limit_mm_per_prompt={"image": 10, "video": 0},
250 + bucket_config={(32, 32, 1): 1.0, (52, 64, 1): 0.0},
251 + )
252 + for s in samples:
253 + assert isinstance(s.multi_modal_data, list)
254 + typed_mm = cast(list[dict[str, Any]], s.multi_modal_data)
255 + for it in typed_mm:
256 + assert it.get("type") == "image_url"
257 +
258 +
259 +@pytest.mark.benchmark
260 +def test_random_mm_zero_items(hf_tokenizer: PreTrainedTokenizerBase) -> None:
261 + ds = RandomMultiModalDataset(random_seed=0)
262 + samples = _collect_mm_samples(
263 + ds,
264 + hf_tokenizer,
265 + num_requests=5,
266 + base_items_per_request=0,
267 + num_mm_items_range_ratio=0.0,
268 + limit_mm_per_prompt={"image": 5, "video": 0},
269 + bucket_config={(32, 32, 1): 1.0},
270 + )
271 + for s in samples:
272 + assert s.multi_modal_data == []
273 +
274 +@pytest.mark.benchmark
275 +def test_random_mm_num_items_per_prompt(
276 + hf_tokenizer: PreTrainedTokenizerBase) -> None:
277 + ds = RandomMultiModalDataset(random_seed=0)
278 + # Fixed number of images per prompt
279 + # set num_mm_items_range_ratio to 0.0
280 + # TODO: modify video values when video sampling is implemented
281 + samples_fixed_items = _collect_mm_samples(
282 + ds,
283 + hf_tokenizer,
284 + num_requests=5,
285 + base_items_per_request=3,
286 + num_mm_items_range_ratio=0.0,
287 + limit_mm_per_prompt={"image": 3, "video": 0},
288 + bucket_config={(32, 32, 1): 1.0},
289 + )
290 + # Must have 5 requests each with 3 mm items per prompt
291 + assert len(samples_fixed_items) == 5
292 + for s in samples_fixed_items:
293 + mm_data = cast(list[dict[str, Any]], s.multi_modal_data)
294 + assert len(mm_data) == 3
295 + for it in mm_data:
296 + assert it.get("type") == "image_url"
297 +
298 +
299 +@pytest.mark.benchmark
300 +def test_random_mm_bucket_config_not_mutated(
301 + hf_tokenizer: PreTrainedTokenizerBase,
302 +) -> None:
303 +
304 + ds = RandomMultiModalDataset(random_seed=0)
305 + # This bucket config is not normalized to sum to 1
306 + # and has more buckets than requested images
307 + original = {(32, 32, 1): 0.2, (52, 64, 1): 6, (25, 64, 1): 3}
308 + # Keep a snapshot to compare after sampling
309 + snapshot = dict(original)
310 +
311 + _ = _collect_mm_samples(
312 + ds,
313 + hf_tokenizer,
314 + num_requests=4,
315 + base_items_per_request=1,
316 + num_mm_items_range_ratio=0.0,
317 + limit_mm_per_prompt={"image": 1, "video": 0},
318 + bucket_config=original,
319 + )
320 +
321 + # Ensure the original dict content is unchanged
322 + assert original == snapshot
323 +
324 +
325 + # Vary number of mm items per prompt
326 + # set num_mm_items_range_ratio to 0.5
327 + samples_varying_items = _collect_mm_samples(
328 + ds,
329 + hf_tokenizer,
330 + num_requests=5,
331 + base_items_per_request=2,
332 + num_mm_items_range_ratio=0.5,
333 + limit_mm_per_prompt={"image": 4, "video": 0},
334 + bucket_config={(32, 32, 1): 1.0},
335 + )
336 + # Must have 5 requests each with less than 4 mm items per prompt
337 + # but at least 1 mm item per prompt
338 + assert len(samples_varying_items) == 5
339 + for s in samples_varying_items:
340 + mm_data = cast(list[dict[str, Any]], s.multi_modal_data)
341 + assert len(mm_data) <= 4
342 + assert len(mm_data) >= 1
343 + for it in mm_data:
344 + assert it.get("type") == "image_url"
vllm/benchmarks/datasets.py
+630 -56
@@ -11,18 +11,21 @@ generation. Supported dataset types include:
11 - HuggingFace
12 - VisionArena
13 """
14 +import ast
15 import base64
16 import io
17 import json
18 import logging
19 +import math
20 import random
21 from abc import ABC, abstractmethod
20 -from collections.abc import Mapping
22 +from collections.abc import Iterator, Mapping
23 +from contextlib import suppress
24 from copy import deepcopy
25 from dataclasses import dataclass
26 from functools import cache
27 from io import BytesIO
25 -from typing import Any, Callable, Optional, Union
28 +from typing import Any, Callable, Optional, Union, cast
29
30 import numpy as np
31 from PIL import Image
@@ -114,7 +117,9 @@ class BenchmarkDataset(ABC):
117 def apply_multimodal_chat_transformation(
118 self,
119 prompt: str,
117 - mm_content: Optional[MultiModalDataDict] = None) -> list[dict]:
120 + mm_content: Optional[
121 + Union[MultiModalDataDict, dict, list[dict]]
122 + ] = None) -> list[dict]:
123 """
124 Transform a prompt and optional multimodal content into a chat format.
125 This method is used for chat models that expect a specific conversation
@@ -122,7 +127,15 @@ class BenchmarkDataset(ABC):
127 """
128 content = [{"text": prompt, "type": "text"}]
129 if mm_content is not None:
125 - content.append(mm_content)
130 + if isinstance(mm_content, list):
131 + content.extend(cast(list[dict[str, Any]], mm_content))
132 + elif isinstance(mm_content, dict):
133 + content.append(mm_content)
134 + else:
135 + raise TypeError(
136 + "Could not process multimodal content of type: " +
137 + f"{type(mm_content)}"
138 + )
139 return [{"role": "user", "content": content}]
140
141 def load_data(self) -> None:
@@ -362,90 +375,536 @@ def process_video(video: Any) -> Mapping[str, Any]:
375
376
377 class RandomDataset(BenchmarkDataset):
378 + """
379 + Synthetic text-only dataset for serving/throughput benchmarks.
380 +
381 + Strategy:
382 + - Sample input/output token lengths per request from integer-uniform ranges
383 + around configured means (controlled by range_ratio).
384 + - Prepend a fixed random prefix of length prefix_len.
385 + - Generate the remaining tokens as a reproducible sequence:
386 + (offset + index + arange(input_len)) % vocab_size.
387 + - Decode then re-encode/truncate to ensure prompt token counts match.
388 + - Uses numpy.default_rng seeded with random_seed for reproducible sampling.
389 + """
390 # Default values copied from benchmark_serving.py for the random dataset.
391 DEFAULT_PREFIX_LEN = 0
392 DEFAULT_RANGE_RATIO = 0.0
393 DEFAULT_INPUT_LEN = 1024
394 DEFAULT_OUTPUT_LEN = 128
395
371 - def __init__(
372 - self,
373 - **kwargs,
374 - ) -> None:
396 + def __init__(self, **kwargs) -> None:
397 super().__init__(**kwargs)
376 - random.seed(self.random_seed)
377 - np.random.seed(self.random_seed)
398 + # Use numpy's default_rng for deterministic sampling
399 + # Do not use random.seed() or np.random.seed() elsewhere in this class.
400 + # This ensures that the RNG is isolated from global RNG state.
401 + self._rng = np.random.default_rng(self.random_seed)
402
403 def sample(
404 self,
405 tokenizer: PreTrainedTokenizerBase,
406 num_requests: int,
407 + request_id_prefix: str = "",
408 prefix_len: int = DEFAULT_PREFIX_LEN,
409 range_ratio: float = DEFAULT_RANGE_RATIO,
410 input_len: int = DEFAULT_INPUT_LEN,
411 output_len: int = DEFAULT_OUTPUT_LEN,
387 - request_id_prefix: str = "",
412 **kwargs,
413 ) -> list[SampleRequest]:
390 - # Enforce range_ratio < 1
391 - assert range_ratio < 1.0, (
392 - "random_range_ratio must be < 1.0 to ensure a valid sampling range"
414 +
415 + input_lens, output_lens, offsets = self.get_sampling_params(
416 + num_requests, range_ratio, input_len, output_len, tokenizer
417 )
418
419 + # Generate prefix once
420 + prefix_token_ids = self.get_prefix(tokenizer, prefix_len)
421 vocab_size = tokenizer.vocab_size
396 - num_special_tokens = tokenizer.num_special_tokens_to_add()
397 - real_input_len = input_len - num_special_tokens
422
399 - prefix_token_ids = (np.random.randint(
400 - 0, vocab_size, size=prefix_len).tolist() if prefix_len > 0 else [])
423 + requests = []
424 + for i in range(num_requests):
425 + prompt, total_input_len = self.generate_token_sequence(
426 + tokenizer=tokenizer,
427 + prefix_token_ids=prefix_token_ids,
428 + prefix_len=prefix_len,
429 + vocab_size=vocab_size,
430 + input_len=int(input_lens[i]),
431 + offset=int(offsets[i]),
432 + index=i,
433 + )
434 + requests.append(
435 + SampleRequest(
436 + prompt=prompt,
437 + prompt_len=total_input_len,
438 + expected_output_len=int(output_lens[i]),
439 + request_id=request_id_prefix + str(i),
440 + )
441 + )
442 + return requests
443 +
444 + def get_prefix(
445 + self, tokenizer: PreTrainedTokenizerBase, prefix_len: int
446 + ) -> list[int]:
447 + """
448 + Get the prefix for the dataset.
449 + """
450 + return (
451 + self._rng.integers(
452 + 0, tokenizer.vocab_size, size=prefix_len).tolist()
453 + if prefix_len > 0
454 + else []
455 + )
456
402 - # New sampling logic: [X * (1 - b), X * (1 + b)]
403 - input_low = int(real_input_len * (1 - range_ratio))
404 - input_high = int(real_input_len * (1 + range_ratio))
405 - output_low = int(output_len * (1 - range_ratio))
406 - output_high = int(output_len * (1 + range_ratio))
457 + def get_sampling_params(
458 + self,
459 + num_requests: int,
460 + range_ratio: float,
461 + input_len: int,
462 + output_len: int,
463 + tokenizer: PreTrainedTokenizerBase,
464 + ) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
465 + """
466 + Get the sampling parameters for the dataset.
467 + """
468 + # Enforce range_ratio < 1
469 + if not (0.0 <= range_ratio < 1.0):
470 + raise ValueError("range_ratio must be in [0, 1).")
471 + num_special_tokens = int(tokenizer.num_special_tokens_to_add())
472 + real_input_len = max(0, int(input_len) - num_special_tokens)
473 + # Bounds use floor for low and ceil for high
474 + input_low = math.floor(real_input_len * (1 - range_ratio))
475 + input_high = math.ceil(real_input_len * (1 + range_ratio))
476 + output_low = math.floor(output_len * (1 - range_ratio))
477 + output_high = math.ceil(output_len * (1 + range_ratio))
478 + # Ensure the lower bound for output length is at least 1 to
479 + # prevent sampling 0 tokens.
480 + output_low = max(output_low, 1)
481 +
482 + if input_low > input_high:
483 + raise ValueError(
484 + "Invalid input sampling interval: "
485 + f"low={input_low} > high={input_high}"
486 + )
487 + if output_low > output_high:
488 + raise ValueError(
489 + "Invalid output sampling interval: "
490 + f"low={output_low} > high={output_high}"
491 + )
492
408 - # Add logging for debugging
493 logger.info(
494 "Sampling input_len from [%s, %s] and output_len from [%s, %s]",
411 - input_low, input_high, output_low, output_high)
495 + input_low,
496 + input_high,
497 + output_low,
498 + output_high,
499 + )
500
413 - input_lens = np.random.randint(input_low,
414 - input_high + 1,
415 - size=num_requests)
416 - output_lens = np.random.randint(output_low,
417 - output_high + 1,
501 + input_lens = self._rng.integers(input_low, input_high + 1,
502 + size=num_requests)
503 + output_lens = self._rng.integers(output_low, output_high + 1,
504 + size=num_requests)
505 + offsets = self._rng.integers(0, tokenizer.vocab_size,
506 size=num_requests)
419 - offsets = np.random.randint(0, vocab_size, size=num_requests)
507 + return input_lens, output_lens, offsets
508
421 - requests = []
509 +
510 + def generate_token_sequence(
511 + self,
512 + *,
513 + tokenizer: PreTrainedTokenizerBase,
514 + prefix_token_ids: list[int],
515 + prefix_len: int,
516 + vocab_size: int,
517 + input_len: int,
518 + offset: int,
519 + index: int,
520 + ) -> tuple[str, int]:
521 + """
522 + Returns (prompt, total_input_len).
523 +
524 + NOTE: After decoding the prompt we have to encode and decode it again.
525 + This is done because in some cases N consecutive tokens
526 + give a string tokenized into != N number of tokens.
527 + For example for GPT2Tokenizer:
528 + [6880, 6881] -> ['Ġcalls', 'here'] ->
529 + [1650, 939, 486] -> ['Ġcall', 'sh', 'ere']
530 + To avoid uncontrolled change of the prompt length,
531 + the encoded sequence is truncated before being decode again.
532 + """
533 + # Build the inner sequence by sampling sequentially from the vocab
534 + inner_seq = ((offset + index + np.arange(input_len))
535 + % vocab_size).tolist()
536 + token_sequence = prefix_token_ids + inner_seq
537 +
538 + # Decode, then re-encode and truncate to preserve token count invariants
539 + prompt = tokenizer.decode(token_sequence)
540 + total_input_len = prefix_len + int(input_len)
541 +
542 + re_encoded_sequence = tokenizer.encode(
543 + prompt, add_special_tokens=False)[:total_input_len]
544 + prompt = tokenizer.decode(re_encoded_sequence)
545 + total_input_len = len(re_encoded_sequence)
546 +
547 + return prompt, total_input_len
548 +
549 +
550 +# -----------------------------------------------------------------------------
551 +# MultiModalDataset Implementation
552 +# -----------------------------------------------------------------------------
553 +
554 +class RandomMultiModalDataset(RandomDataset):
555 + """
556 + Synthetic multimodal dataset (text + images) that extends RandomDataset.
557 +
558 + Status:
559 + - Images: supported via synthetic RGB data.
560 + - Video: not yet supported (TODO: implement video generation method).
561 + - Audio: not yet supported.
562 +
563 + Sampling overview:
564 + 1) Number of items per request is sampled uniformly from the integer range
565 + [floor(n·(1−r)), ceil(n·(1+r))], where n is the base count and r is
566 + `num_mm_items_range_ratio` in [0, 1]. r=0 keeps it fixed; r=1 allows 0.
567 + The maximum is further clamped to the sum of per-modality limits.
568 + 2) Each item’s modality and shape is sampled from `bucket_config`, a dict
569 + mapping (height, width, num_frames) → probability. We treat
570 + `num_frames`=1 as image and and `num_frames` > 1 as video.
571 + Entries with zero probability are removed and the rest are renormalized
572 + to sum to 1.
573 + 3) Per-modality hard caps are enforced via `limit_mm_per_prompt`.
574 + When a modality reaches its cap, all of its buckets are excluded and the
575 + remaining probabilities are renormalized.
576 +
577 + Example bucket configuration:
578 + {(256, 256, 1): 0.5, (720, 1280, 1): 0.4, (720, 1280, 16): 0.1}
579 + - Two image buckets (`num_frames`=1) and one video bucket
580 + (`num_frames`=16).
581 + OBS.: Only image sampling is supported for now.
582 + """
583 +
584 + IS_MULTIMODAL = True
585 + # NOTE: video sampling is WIP. Setting it to 0.
586 + DEFAULT_LIMIT_MM_PER_PROMPT = {"image": 255, "video": 0}
587 +
588 + DEFAULT_BASE_ITEMS_PER_REQUEST = 1
589 + DEFAULT_NUM_MM_ITEMS_RANGE_RATIO = 0.0
590 + DEFAULT_MM_ITEM_BUCKET_CONFIG = {
591 + (256, 256, 1): 0.5,
592 + (720, 1280, 1): 0.5,
593 + (720, 1280, 16): 0.0,
594 + }
595 + DEFAULT_ENABLE_MULTIMODAL_CHAT = False
596 +
597 + def __init__(self, **kwargs) -> None:
598 + super().__init__(**kwargs)
599 +
600 +
601 + def generate_synthetic_image(self, width: int, height: int) -> Image.Image:
602 + """Generate synthetic PIL image with random RGB values.
603 +
604 + NOTE: iid pixel sampling results in worst-case compression
605 + (good for stressing I/O), but very unlike real photos.
606 + We could consider a “low-freq” mode (e.g., noise blur)
607 + to emulate network realism instead of max stress.
608 + """
609 + random_pixels = self._rng.integers(
610 + 0,
611 + 256,
612 + (height, width, 3),
613 + dtype=np.uint8,
614 + )
615 + return Image.fromarray(random_pixels)
616 +
617 + def generate_synthetic_video(self, width: int,
618 + height: int,
619 + num_frames: int) -> Any:
620 + """Generate synthetic video with random values.
621 +
622 + TODO: Finish this method.
623 + """
624 + raise NotImplementedError("Video sampling is WIP.")
625 +
626 + def map_config_to_modality(self, config: tuple[int, int, int]) -> str:
627 + """Map the configuration to the modality."""
628 + if config[-1] == 1:
629 + return "image"
630 + elif config[-1] > 1:
631 + return "video"
632 + else:
633 + raise ValueError(f"Invalid multimodal item configuration: {config}")
634 +
635 + def normalize_bucket_config(self, bucket_config: dict[tuple[int, int, int],
636 + float]) -> dict[tuple[int, int, int], float]:
637 + """
638 + Remove zero probability entries
639 + and normalize the bucket config to sum to 1.
640 + """
641 + # Raise error if value is negative
642 + if any(v < 0 for v in bucket_config.values()):
643 + raise ValueError("Bucket config values must be non-negative.")
644 + # Remove zero probability entries
645 + bucket_config = {k: v for k, v in bucket_config.items() if v > 0}
646 + # if bucket config is empty, raise error
647 + if not bucket_config:
648 + raise ValueError("Got invalid bucket config. "
649 + "Bucket config values must be non-zero.")
650 + # Normalize the remaining bucket config to sum to 1
651 + total = sum(bucket_config.values())
652 + return {k: v / total for k, v in bucket_config.items()}
653 +
654 +
655 + def generate_mm_item(self,
656 + mm_item_config: tuple[int, int, int],
657 + ) -> Mapping[str, Any]:
658 + """
659 + Create synthetic images and videos and
660 + apply process_image/process_video respectively.
661 + This follows the OpenAI API chat completions
662 + https://github.com/openai/openai-python
663 + """
664 +
665 + if self.map_config_to_modality(mm_item_config) == "image":
666 + return process_image(self.generate_synthetic_image(
667 + mm_item_config[1],
668 + mm_item_config[0]))
669 + elif self.map_config_to_modality(mm_item_config) == "video":
670 + return process_video(self.generate_synthetic_video(
671 + mm_item_config[1],
672 + mm_item_config[0],
673 + mm_item_config[2]))
674 + else:
675 + raise ValueError(f"Invalid multimodal item configuration: "
676 + f"{mm_item_config}")
677 +
678 +
679 + def get_mm_item_sampling_params(
680 + self,
681 + base_items_per_request: int,
682 + num_mm_items_range_ratio: float,
683 + limit_mm_per_prompt: dict[str, int],
684 + bucket_config: dict[tuple[int, int, int], float],
685 + ) -> tuple[int, int, dict[str, int], dict[tuple[int, int, int], float]]:
686 + """
687 + Get the sampling parameters for the multimodal items.
688 + """
689 + # Enforce num_mm_items_range_ratio <= 1
690 + if not (0.0 <= num_mm_items_range_ratio <= 1.0):
691 + raise ValueError("num_mm_items_range_ratio must be in [0, 1].")
692 +
693 + # Ensure modalities to sample are in limit_mm_per_prompt
694 + for k, v in bucket_config.items():
695 + # get modality from bucket config
696 + modality = self.map_config_to_modality(k)
697 + if modality not in limit_mm_per_prompt:
698 + raise ValueError(f"Modality {modality} is not in "
699 + f"limit_mm_per_prompt: "
700 + f"{limit_mm_per_prompt.keys()}")
701 +
702 + # Remove zero probability entries
703 + # and normalize bucket config to sum to 1
704 + bucket_config = self.normalize_bucket_config(bucket_config)
705 + logger.info(
706 + "Normalized bucket config: %s", bucket_config,
707 + )
708 + # Only consider limit per prompt for modalities in bucket config
709 + allowed_modalities = {self.map_config_to_modality(cfg)
710 + for cfg in bucket_config}
711 + limit_mm_per_prompt = {
712 + k: v for k, v in limit_mm_per_prompt.items()
713 + if k in allowed_modalities}
714 + if not limit_mm_per_prompt:
715 + raise ValueError("No valid limits for modalities present in "
716 + "bucket_config.")
717 +
718 + logger.info(
719 + "Updated mm-limit-per-prompt: %s", limit_mm_per_prompt,
720 + )
721 +
722 + # Get max and min num mm items and ensure
723 + # it is at most the sum of limit_mm_per_prompt for all modalities
724 + max_num_mm_items = min(
725 + sum(limit_mm_per_prompt.values()),
726 + math.ceil(base_items_per_request * (1 + num_mm_items_range_ratio))
727 + )
728 + # Ensure min num mm items is at least 0
729 + min_num_mm_items = max(
730 + 0,
731 + math.floor(base_items_per_request * (1 - num_mm_items_range_ratio))
732 + )
733 + # Raise error if min num mm items is greater than max num mm items
734 + if min_num_mm_items > max_num_mm_items:
735 + raise ValueError(f"Min num mm items is greater than max mm items: "
736 + f"{min_num_mm_items} > {max_num_mm_items}")
737 +
738 + logger.info(
739 + "Sampling number of multimodal items from [%s, %s]",
740 + min_num_mm_items, max_num_mm_items,
741 + )
742 +
743 + return (
744 + min_num_mm_items,
745 + max_num_mm_items,
746 + limit_mm_per_prompt,
747 + bucket_config,
748 + )
749 +
750 + def get_mm_item_iterator(
751 + self,
752 + min_num_mm_items: int,
753 + max_num_mm_items: int,
754 + bucket_config: dict[tuple[int, int, int], float],
755 + limit_mm_per_prompt: dict[str, int],
756 + ) -> Iterator[tuple[int,int, int]]:
757 + """
758 + Iterator over the multimodal items for each request
759 + whose size is between min_num_mm_items and max_num_mm_items.
760 +
761 + Loop over the bucket config and sample a multimodal item.
762 + Loop until the number of multimodal items sampled is equal to
763 + request_num_mm_items or limit of multimodal items per prompt
764 + for all modalities is reached.
765 +
766 + Note:
767 + - This function operates on a per-request shallow copy of
768 + `bucket_config` (tuple->float). The original dict passed to
769 + `sample` is not mutated. If this ever changes, a test
770 + is implemented and will fail.
771 + """
772 + # Get the number of multimodal items to sample
773 + request_num_mm_items = int(
774 + self._rng.integers(min_num_mm_items, max_num_mm_items + 1)
775 + )
776 + # If request_num_mm_items is 0, yield an empty iterator
777 + if request_num_mm_items == 0:
778 + return
779 + # Initialize modality counters
780 + modality_counter = {self.map_config_to_modality(k): 0
781 + for k in bucket_config}
782 + # Copy the bucket config to avoid modifying the original
783 + bucket_config_copy = bucket_config.copy()
784 + # Loop over the number of multimodal items to sample
785 + while sum(modality_counter.values()) < request_num_mm_items:
786 + # Sample a multimodal item config
787 + mm_item_config = self._rng.choice(list(bucket_config_copy.keys()),
788 + p=list(bucket_config_copy.values()))
789 + modality = self.map_config_to_modality(mm_item_config)
790 + # Check that modality count is less than limit per prompt
791 + if modality_counter[modality] < limit_mm_per_prompt[modality]:
792 + modality_counter[modality] += 1
793 + yield (
794 + mm_item_config
795 + )
796 + else:
797 + # If the counter is greater than the limit per prompt
798 + # set all multimodal items of this modality to 0
799 + for k, v in bucket_config_copy.items():
800 + if self.map_config_to_modality(k) == modality:
801 + bucket_config_copy[k] = 0
802 + # If all configs are 0, break the loop
803 + # This should not happen as request_num_mm_items is at most
804 + # the sum of limit_mm_per_prompt for all modalities
805 + if all(v == 0 for v in bucket_config_copy.values()):
806 + logger.warning("Exhausted all multimodal items "
807 + "of modality %s",
808 + modality)
809 + break
810 + # Renormalize the bucket config
811 + bucket_config_copy = self.normalize_bucket_config(
812 + bucket_config_copy)
813 +
814 +
815 + def sample(
816 + self,
817 + tokenizer: PreTrainedTokenizerBase,
818 + num_requests: int,
819 + request_id_prefix: str = "",
820 + prefix_len: int = RandomDataset.DEFAULT_PREFIX_LEN,
821 + range_ratio: float = RandomDataset.DEFAULT_RANGE_RATIO,
822 + input_len: int = RandomDataset.DEFAULT_INPUT_LEN,
823 + output_len: int = RandomDataset.DEFAULT_OUTPUT_LEN,
824 + limit_mm_per_prompt: dict[str, int] = DEFAULT_LIMIT_MM_PER_PROMPT,
825 + base_items_per_request: int = DEFAULT_BASE_ITEMS_PER_REQUEST,
826 + num_mm_items_range_ratio: float = DEFAULT_NUM_MM_ITEMS_RANGE_RATIO,
827 + bucket_config: dict[tuple[int, int, int], float] =
828 + DEFAULT_MM_ITEM_BUCKET_CONFIG,
829 + enable_multimodal_chat: bool = DEFAULT_ENABLE_MULTIMODAL_CHAT,
830 + **kwargs,
831 + ) -> list[SampleRequest]:
832 +
833 + # NOTE: Video sampling is WIP. Raise error if video is in bucket config
834 + # and probability is non-zero.
835 + if any(self.map_config_to_modality(cfg) == "video" and p > 0
836 + for cfg, p in bucket_config.items()):
837 + raise NotImplementedError("Video sampling not implemented; "
838 + "set its probability to 0.")
839 +
840 + # Get the sampling parameters for the dataset
841 + input_lens, output_lens, offsets = self.get_sampling_params(
842 + num_requests, range_ratio, input_len, output_len, tokenizer
843 + )
844 +
845 + (
846 + min_num_mm_items,
847 + max_num_mm_items,
848 + limit_mm_per_prompt,
849 + bucket_config,
850 + ) = self.get_mm_item_sampling_params(
851 + base_items_per_request,
852 + num_mm_items_range_ratio,
853 + limit_mm_per_prompt,
854 + bucket_config,
855 + )
856 +
857 + # Generate prefix once
858 + prefix_token_ids = self.get_prefix(tokenizer, prefix_len)
859 + vocab_size = tokenizer.vocab_size
860 + # Add synthetic multimodal items to each request
861 + mm_requests = []
862 for i in range(num_requests):
423 - inner_seq = ((offsets[i] + i + np.arange(input_lens[i])) %
424 - vocab_size).tolist()
425 - token_sequence = prefix_token_ids + inner_seq
426 - prompt = tokenizer.decode(token_sequence)
427 - # After decoding the prompt we have to encode and decode it again.
428 - # This is done because in some cases N consecutive tokens
429 - # give a string tokenized into != N number of tokens.
430 - # For example for GPT2Tokenizer:
431 - # [6880, 6881] -> ['Ġcalls', 'here'] ->
432 - # [1650, 939, 486] -> ['Ġcall', 'sh', 'ere']
433 - # To avoid uncontrolled change of the prompt length,
434 - # the encoded sequence is truncated before being decode again.
435 - total_input_len = prefix_len + int(input_lens[i])
436 - re_encoded_sequence = tokenizer.encode(
437 - prompt, add_special_tokens=False)[:total_input_len]
438 - prompt = tokenizer.decode(re_encoded_sequence)
439 - total_input_len = len(re_encoded_sequence)
440 - requests.append(
441 - SampleRequest(
863 + prompt, total_input_len = self.generate_token_sequence(
864 + tokenizer=tokenizer,
865 + prefix_token_ids=prefix_token_ids,
866 + prefix_len=prefix_len,
867 + vocab_size=vocab_size,
868 + input_len=int(input_lens[i]),
869 + offset=int(offsets[i]),
870 + index=i,
871 + )
872 + # Get multimodal item iterator for a given request
873 + mm_item_iterator = self.get_mm_item_iterator(
874 + min_num_mm_items,
875 + max_num_mm_items,
876 + bucket_config,
877 + limit_mm_per_prompt,
878 + )
879 +
880 + mm_content = cast(list[dict[str, Any]], [
881 + self.generate_mm_item(mm_item_config)
882 + for mm_item_config in mm_item_iterator
883 + ])
884 +
885 + if enable_multimodal_chat:
886 + # NOTE: For now this option is only provided for completeness
887 + # given that the serve.py benchmark currently does not use it.
888 + mm_chat_prompt: Any = prompt
889 + mm_chat_prompt = self.apply_multimodal_chat_transformation(
890 + prompt, mm_content)
891 + sample_request = SampleRequest(
892 + prompt=mm_chat_prompt,
893 + prompt_len=total_input_len,
894 + expected_output_len=int(output_lens[i]),
895 + multi_modal_data=None,
896 + request_id=request_id_prefix + str(i),
897 + )
898 + else:
899 + sample_request = SampleRequest(
900 prompt=prompt,
901 prompt_len=total_input_len,
902 expected_output_len=int(output_lens[i]),
903 + multi_modal_data=mm_content,
904 request_id=request_id_prefix + str(i),
446 - ))
447 - return requests
448 -
905 + )
906 + mm_requests.append(sample_request)
907 + return mm_requests
908
909 # -----------------------------------------------------------------------------
910 # ShareGPT Dataset Implementation
@@ -545,8 +1004,8 @@ def add_dataset_parser(parser: FlexibleArgumentParser):
1004 type=str,
1005 default="random",
1006 choices=[
548 - "sharegpt", "burstgpt", "sonnet", "random", "hf", "custom",
549 - "prefix_repetition"
1007 + "sharegpt", "burstgpt", "sonnet", "random", "random-mm", "hf",
1008 + "custom", "prefix_repetition"
1009 ],
1010 help="Name of the dataset to benchmark on.",
1011 )
@@ -647,6 +1106,98 @@ def add_dataset_parser(parser: FlexibleArgumentParser):
1106 "input_len * (1 + range_ratio)]."),
1107 )
1108
1109 + # random multimodal dataset options
1110 + random_mm_group = parser.add_argument_group(
1111 + "random multimodal dataset options extended from random dataset")
1112 + random_mm_group.add_argument(
1113 + "--random-mm-base-items-per-request",
1114 + type=int,
1115 + default=RandomMultiModalDataset.DEFAULT_BASE_ITEMS_PER_REQUEST,
1116 + help=(
1117 + "Base number of multimodal items per request for random-mm. "
1118 + "Actual per-request count is sampled around this base using "
1119 + "--random-mm-num-mm-items-range-ratio."
1120 + ),
1121 + )
1122 + random_mm_group.add_argument(
1123 + "--random-mm-num-mm-items-range-ratio",
1124 + type=float,
1125 + default=RandomMultiModalDataset.DEFAULT_NUM_MM_ITEMS_RANGE_RATIO,
1126 + help=(
1127 + "Range ratio r in [0, 1] for sampling items per request. "
1128 + "We sample uniformly from the closed integer range "
1129 + "[floor(n*(1-r)), ceil(n*(1+r))] "
1130 + "where n is the base items per request. "
1131 + "r=0 keeps it fixed; r=1 allows 0 items. The maximum is clamped "
1132 + "to the sum of per-modality limits from "
1133 + "--random-mm-limit-mm-per-prompt. "
1134 + "An error is raised if the computed min exceeds the max."
1135 + ),
1136 + )
1137 + random_mm_group.add_argument(
1138 + "--random-mm-limit-mm-per-prompt",
1139 + type=json.loads,
1140 + default=RandomMultiModalDataset.DEFAULT_LIMIT_MM_PER_PROMPT,
1141 + help=(
1142 + "Per-modality hard caps for items attached per request, e.g. "
1143 + "'{\"image\": 3, \"video\": 0}'. The sampled per-request item "
1144 + "count is clamped to the sum of these limits. When a modality "
1145 + "reaches its cap, its buckets are excluded and probabilities are "
1146 + "renormalized."
1147 + "OBS.: Only image sampling is supported for now."
1148 + ),
1149 + )
1150 +
1151 + def _parse_mm_bucket_config(v: object) -> dict[tuple[int, int, int], float]:
1152 + # If already a dict (e.g., programmatic call), normalize keys
1153 + def normalize(d: dict) -> dict[tuple[int, int, int], float]:
1154 + out: dict[tuple[int, int, int], float] = {}
1155 + for k, val in d.items():
1156 + key = k
1157 + if isinstance(key, str):
1158 + with suppress(Exception):
1159 + key = ast.literal_eval(key)
1160 + if not (isinstance(key, tuple) and len(key) == 3
1161 + and all(isinstance(x, int) for x in key)):
1162 + raise ValueError(
1163 + f"Invalid bucket key {k!r}. Expected tuple (H, W, T)."
1164 + )
1165 + out[(int(key[0]), int(key[1]), int(key[2]))] = float(val)
1166 + return out
1167 +
1168 + if isinstance(v, dict):
1169 + return normalize(v)
1170 + if isinstance(v, str):
1171 + # Python literal (supports tuple keys)
1172 + parsed = ast.literal_eval(v)
1173 + if not isinstance(parsed, dict):
1174 + raise ValueError("Bucket config must parse to a dict.")
1175 + return normalize(parsed)
1176 + raise ValueError("Unsupported value for --random-mm-bucket-config.")
1177 +
1178 + random_mm_group.add_argument(
1179 + "--random-mm-bucket-config",
1180 + type=_parse_mm_bucket_config,
1181 + default=RandomMultiModalDataset.DEFAULT_MM_ITEM_BUCKET_CONFIG,
1182 + help=(
1183 + "The bucket config is a dictionary mapping a multimodal item"
1184 + "sampling configuration to a probability."
1185 + "Currently allows for 2 modalities: images and videos. "
1186 + "An bucket key is a tuple of (height, width, num_frames)"
1187 + "The value is the probability of sampling that specific item. "
1188 + "Example: "
1189 + "--random-mm-bucket-config "
1190 + "{(256, 256, 1): 0.5, (720, 1280, 1): 0.4, (720, 1280, 16): 0.10} "
1191 + "First item: images with resolution 256x256 w.p. 0.5"
1192 + "Second item: images with resolution 720x1280 w.p. 0.4 "
1193 + "Third item: videos with resolution 720x1280 and 16 frames w.p. 0.1"
1194 + "OBS.: If the probabilities do not sum to 1, they are normalized."
1195 + "OBS bis.: Only image sampling is supported for now."
1196 + ),
1197 + )
1198 +
1199 +
1200 +
1201 hf_group = parser.add_argument_group("hf dataset options")
1202 hf_group.add_argument("--hf-subset",
1203 type=str,
@@ -821,6 +1372,22 @@ def get_samples(args, tokenizer) -> list[SampleRequest]:
1372 range_ratio=args.random_range_ratio,
1373 request_id_prefix=args.request_id_prefix,
1374 ),
1375 + "random-mm":
1376 + lambda: RandomMultiModalDataset(
1377 + random_seed=args.seed, dataset_path=args.dataset_path
1378 + ).sample(
1379 + tokenizer=tokenizer,
1380 + num_requests=args.num_prompts,
1381 + prefix_len=args.random_prefix_len,
1382 + range_ratio=args.random_range_ratio,
1383 + input_len=args.random_input_len,
1384 + output_len=args.random_output_len,
1385 + base_items_per_request=args.random_mm_base_items_per_request,
1386 + limit_mm_per_prompt=args.random_mm_limit_mm_per_prompt,
1387 + num_mm_items_range_ratio=args.random_mm_num_mm_items_range_ratio,
1388 + bucket_config=args.random_mm_bucket_config,
1389 + request_id_prefix=args.request_id_prefix,
1390 + ),
1391 "prefix_repetition":
1392 lambda: PrefixRepetitionRandomDataset(
1393 random_seed=args.seed, dataset_path=args.dataset_path
@@ -836,6 +1403,13 @@ def get_samples(args, tokenizer) -> list[SampleRequest]:
1403 }
1404
1405 try:
1406 + # Enforce endpoint compatibility for multimodal datasets.
1407 + if args.dataset_name == "random-mm" and args.endpoint_type not in [
1408 + "openai-chat"]:
1409 + raise ValueError(
1410 + "Multi-modal content (images) is only supported on "
1411 + "'openai-chat' backend."
1412 + )
1413 input_requests = dataset_mapping[args.dataset_name]()
1414 except KeyError as err:
1415 raise ValueError(f"Unknown dataset: {args.dataset_name}") from err