@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 44b9af5bb2
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

[Benchmark] Enable MM Embedding benchmarks (#26310)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Oct 7, 2025 at 03:51 UTC 44b9af5bb24279556e4b03d1174c6243da1ad143
5 files changed +358 -107
docs/contributing/benchmarks.md
+104 -7
@@ -67,13 +67,13 @@ Legend:
67 <details class="admonition abstract" markdown="1">
68 <summary>Show more</summary>
69
70 -First start serving your model
70 +First start serving your model:
71
72 ```bash
73 vllm serve NousResearch/Hermes-3-Llama-3.1-8B
74 ```
75
76 -Then run the benchmarking script
76 +Then run the benchmarking script:
77
78 ```bash
79 # download dataset
@@ -87,7 +87,7 @@ vllm bench serve \
87 --num-prompts 10
88 ```
89
90 -If successful, you will see the following output
90 +If successful, you will see the following output:
91
92 ```text
93 ============ Serving Benchmark Result ============
@@ -125,7 +125,7 @@ If the dataset you want to benchmark is not supported yet in vLLM, even then you
125
126 ```bash
127 # start server
128 -VLLM_USE_V1=1 vllm serve meta-llama/Llama-3.1-8B-Instruct
128 +vllm serve meta-llama/Llama-3.1-8B-Instruct
129 ```
130
131 ```bash
@@ -167,7 +167,7 @@ vllm bench serve \
167 ##### InstructCoder Benchmark with Speculative Decoding
168
169 ``` bash
170 -VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
170 +vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
171 --speculative-config $'{"method": "ngram",
172 "num_speculative_tokens": 5, "prompt_lookup_max": 5,
173 "prompt_lookup_min": 2}'
@@ -184,7 +184,7 @@ vllm bench serve \
184 ##### Spec Bench Benchmark with Speculative Decoding
185
186 ``` bash
187 -VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
187 +vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
188 --speculative-config $'{"method": "ngram",
189 "num_speculative_tokens": 5, "prompt_lookup_max": 5,
190 "prompt_lookup_min": 2}'
@@ -366,7 +366,6 @@ Total num output tokens: 1280
366
367 ``` bash
368 VLLM_WORKER_MULTIPROC_METHOD=spawn \
369 -VLLM_USE_V1=1 \
369 vllm bench throughput \
370 --dataset-name=hf \
371 --dataset-path=likaixin/InstructCoder \
@@ -781,6 +780,104 @@ This should be seen as an edge case, and if this behavior can be avoided by sett
780
781 </details>
782
783 +#### Embedding Benchmark
784 +
785 +Benchmark the performance of embedding requests in vLLM.
786 +
787 +<details class="admonition abstract" markdown="1">
788 +<summary>Show more</summary>
789 +
790 +##### Text Embeddings
791 +
792 +Unlike generative models which use Completions API or Chat Completions API,
793 +you should set `--backend openai-embeddings` and `--endpoint /v1/embeddings` to use the Embeddings API.
794 +
795 +You can use any text dataset to benchmark the model, such as ShareGPT.
796 +
797 +Start the server:
798 +
799 +```bash
800 +vllm serve jinaai/jina-embeddings-v3 --trust-remote-code
801 +```
802 +
803 +Run the benchmark:
804 +
805 +```bash
806 +# download dataset
807 +# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
808 +vllm bench serve \
809 + --model jinaai/jina-embeddings-v3 \
810 + --backend openai-embeddings \
811 + --endpoint /v1/embeddings \
812 + --dataset-name sharegpt \
813 + --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json
814 +```
815 +
816 +##### Multi-modal Embeddings
817 +
818 +Unlike generative models which use Completions API or Chat Completions API,
819 +you should set `--endpoint /v1/embeddings` to use the Embeddings API. The backend to use depends on the model:
820 +
821 +- CLIP: `--backend openai-embeddings-clip`
822 +- VLM2Vec: `--backend openai-embeddings-vlm2vec`
823 +
824 +For other models, please add your own implementation inside <gh-file:vllm/benchmarks/lib/endpoint_request_func.py> to match the expected instruction format.
825 +
826 +You can use any text or multi-modal dataset to benchmark the model, as long as the model supports it.
827 +For example, you can use ShareGPT and VisionArena to benchmark vision-language embeddings.
828 +
829 +Serve and benchmark CLIP:
830 +
831 +```bash
832 +# Run this in another process
833 +vllm serve openai/clip-vit-base-patch32
834 +
835 +# Run these one by one after the server is up
836 +# download dataset
837 +# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
838 +vllm bench serve \
839 + --model openai/clip-vit-base-patch32 \
840 + --backend openai-embeddings-clip \
841 + --endpoint /v1/embeddings \
842 + --dataset-name sharegpt \
843 + --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json
844 +
845 +vllm bench serve \
846 + --model openai/clip-vit-base-patch32 \
847 + --backend openai-embeddings-clip \
848 + --endpoint /v1/embeddings \
849 + --dataset-name hf \
850 + --dataset-path lmarena-ai/VisionArena-Chat
851 +```
852 +
853 +Serve and benchmark VLM2Vec:
854 +
855 +```bash
856 +# Run this in another process
857 +vllm serve TIGER-Lab/VLM2Vec-Full --runner pooling \
858 + --trust-remote-code \
859 + --chat-template examples/template_vlm2vec_phi3v.jinja
860 +
861 +# Run these one by one after the server is up
862 +# download dataset
863 +# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
864 +vllm bench serve \
865 + --model TIGER-Lab/VLM2Vec-Full \
866 + --backend openai-embeddings-vlm2vec \
867 + --endpoint /v1/embeddings \
868 + --dataset-name sharegpt \
869 + --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json
870 +
871 +vllm bench serve \
872 + --model TIGER-Lab/VLM2Vec-Full \
873 + --backend openai-embeddings-vlm2vec \
874 + --endpoint /v1/embeddings \
875 + --dataset-name hf \
876 + --dataset-path lmarena-ai/VisionArena-Chat
877 +```
878 +
879 +</details>
880 +
881 [](){ #performance-benchmarks }
882
883 ## Performance Benchmarks
vllm/benchmarks/datasets.py
+4 -4
@@ -1582,10 +1582,10 @@ def get_samples(args, tokenizer) -> list[SampleRequest]:
1582 "like to add support for additional dataset formats."
1583 )
1584
1585 - if dataset_class.IS_MULTIMODAL and args.backend not in [
1586 - "openai-chat",
1587 - "openai-audio",
1588 - ]:
1585 + if dataset_class.IS_MULTIMODAL and not (
1586 + args.backend in ("openai-chat", "openai-audio")
1587 + or "openai-embeddings-" in args.backend
1588 + ):
1589 # multi-modal benchmark is only available on OpenAI Chat
1590 # endpoint-type.
1591 raise ValueError(
vllm/benchmarks/lib/endpoint_request_func.py
+214 -60
@@ -10,9 +10,10 @@ import time
10 import traceback
11 from collections.abc import Awaitable
12 from dataclasses import dataclass, field
13 -from typing import Optional, Protocol, Union
13 +from typing import Any, Literal, Optional, Protocol, Union
14
15 import aiohttp
16 +import regex as re
17 from tqdm.asyncio import tqdm
18
19 AIOHTTP_TIMEOUT = aiohttp.ClientTimeout(total=6 * 60 * 60)
@@ -103,6 +104,40 @@ class RequestFunc(Protocol):
104 ) -> Awaitable[RequestFuncOutput]: ...
105
106
107 +def _validate_api_url(
108 + api_url: str,
109 + api_name: str,
110 + expected_suffixes: Union[str, set[str]],
111 +) -> None:
112 + if isinstance(expected_suffixes, str):
113 + expected_suffixes = {expected_suffixes}
114 +
115 + expected_suffixes = {*expected_suffixes, "profile"}
116 +
117 + if not api_url.endswith(tuple(expected_suffixes)):
118 + raise ValueError(f"{api_name} URL must end with one of: {expected_suffixes}.")
119 +
120 +
121 +def _update_payload_common(
122 + payload: dict[str, Any],
123 + request_func_input: RequestFuncInput,
124 +) -> None:
125 + if request_func_input.ignore_eos:
126 + payload["ignore_eos"] = request_func_input.ignore_eos
127 + if request_func_input.extra_body:
128 + payload.update(request_func_input.extra_body)
129 +
130 +
131 +def _update_headers_common(
132 + headers: dict[str, Any],
133 + request_func_input: RequestFuncInput,
134 +) -> None:
135 + if request_func_input.extra_headers:
136 + headers |= request_func_input.extra_headers
137 + if request_func_input.request_id:
138 + headers["x-request-id"] = request_func_input.request_id
139 +
140 +
141 async def async_request_openai_completions(
142 request_func_input: RequestFuncInput,
143 session: aiohttp.ClientSession,
@@ -118,9 +153,7 @@ async def async_request_openai_completions(
153 The output of the request function.
154 """
155 api_url = request_func_input.api_url
121 - assert api_url.endswith(("completions", "profile")), (
122 - "OpenAI Completions API URL must end with 'completions' or 'profile'."
123 - )
156 + _validate_api_url(api_url, "OpenAI Completions API", "completions")
157
158 payload = {
159 "model": request_func_input.model_name
@@ -136,15 +169,12 @@ async def async_request_openai_completions(
169 "include_usage": True,
170 },
171 }
139 - if request_func_input.ignore_eos:
140 - payload["ignore_eos"] = request_func_input.ignore_eos
141 - if request_func_input.extra_body:
142 - payload.update(request_func_input.extra_body)
143 - headers = {"Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}"}
144 - if request_func_input.extra_headers:
145 - headers |= request_func_input.extra_headers
146 - if request_func_input.request_id:
147 - headers["x-request-id"] = request_func_input.request_id
172 + _update_payload_common(payload, request_func_input)
173 +
174 + headers = {
175 + "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}",
176 + }
177 + _update_headers_common(headers, request_func_input)
178
179 output = RequestFuncOutput()
180 output.prompt_len = request_func_input.prompt_len
@@ -222,27 +252,41 @@ async def async_request_openai_completions(
252 return output
253
254
225 -async def async_request_openai_chat_completions(
255 +def _get_chat_content(
256 request_func_input: RequestFuncInput,
227 - session: aiohttp.ClientSession,
228 - pbar: Optional[tqdm] = None,
229 -) -> RequestFuncOutput:
230 - api_url = request_func_input.api_url
231 - assert api_url.endswith(("chat/completions", "profile")), (
232 - "OpenAI Chat Completions API URL must end with 'chat/completions'."
233 - )
257 + mm_position: Literal["first", "last"] = "last",
258 +) -> list[dict[str, Any]]:
259 + text_contents = [{"type": "text", "text": request_func_input.prompt}]
260
235 - content = [{"type": "text", "text": request_func_input.prompt}]
261 + mm_contents = []
262 if request_func_input.multi_modal_content:
263 mm_content = request_func_input.multi_modal_content
264 if isinstance(mm_content, list):
239 - content.extend(mm_content)
265 + mm_contents.extend(request_func_input.multi_modal_content)
266 elif isinstance(mm_content, dict):
241 - content.append(mm_content)
267 + mm_contents.append(request_func_input.multi_modal_content)
268 else:
269 raise TypeError(
270 "multi_modal_content must be a dict or list[dict] for openai-chat"
271 )
272 +
273 + if mm_position == "first":
274 + return mm_contents + text_contents
275 +
276 + return text_contents + mm_contents
277 +
278 +
279 +async def async_request_openai_chat_completions(
280 + request_func_input: RequestFuncInput,
281 + session: aiohttp.ClientSession,
282 + pbar: Optional[tqdm] = None,
283 + mm_position: Literal["first", "last"] = "last",
284 +) -> RequestFuncOutput:
285 + api_url = request_func_input.api_url
286 + _validate_api_url(api_url, "OpenAI Chat Completions API", "chat/completions")
287 +
288 + content = _get_chat_content(request_func_input, mm_position=mm_position)
289 +
290 payload = {
291 "model": request_func_input.model_name
292 if request_func_input.model_name
@@ -257,18 +301,13 @@ async def async_request_openai_chat_completions(
301 "include_usage": True,
302 },
303 }
260 - if request_func_input.ignore_eos:
261 - payload["ignore_eos"] = request_func_input.ignore_eos
262 - if request_func_input.extra_body:
263 - payload.update(request_func_input.extra_body)
304 + _update_payload_common(payload, request_func_input)
305 +
306 headers = {
307 "Content-Type": "application/json",
308 "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}",
309 }
268 - if request_func_input.extra_headers:
269 - headers |= request_func_input.extra_headers
270 - if request_func_input.request_id:
271 - headers["x-request-id"] = request_func_input.request_id
310 + _update_headers_common(headers, request_func_input)
311
312 output = RequestFuncOutput()
313 output.prompt_len = request_func_input.prompt_len
@@ -343,10 +382,7 @@ async def async_request_openai_audio(
382 import soundfile
383
384 api_url = request_func_input.api_url
346 - assert api_url.endswith(("transcriptions", "translations")), (
347 - "OpenAI Chat Completions API URL must end with 'transcriptions' "
348 - )
349 - "or `translations`."
385 + _validate_api_url(api_url, "OpenAI Audio API", {"transcriptions", "translations"})
386
387 content = [{"type": "text", "text": request_func_input.prompt}]
388 payload = {
@@ -361,15 +397,12 @@ async def async_request_openai_audio(
397 "stream_include_usage": True,
398 "stream_continuous_usage_stats": True,
399 }
364 - if request_func_input.extra_body:
365 - payload.update(request_func_input.extra_body)
400 + _update_payload_common(payload, request_func_input)
401 +
402 headers = {
403 "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}",
404 }
369 - if request_func_input.extra_headers:
370 - headers |= request_func_input.extra_headers
371 - if request_func_input.request_id:
372 - headers["x-request-id"] = request_func_input.request_id
405 + _update_headers_common(headers, request_func_input)
406
407 # Send audio file
408 def to_bytes(y, sr):
@@ -451,26 +484,13 @@ async def async_request_openai_audio(
484 return output
485
486
454 -async def async_request_openai_embeddings(
455 - request_func_input: RequestFuncInput,
487 +async def _run_openai_embeddings(
488 session: aiohttp.ClientSession,
489 + api_url: str,
490 + payload: dict[str, Any],
491 + headers: dict[str, Any],
492 pbar: Optional[tqdm] = None,
458 -):
459 - api_url = request_func_input.api_url
460 - assert api_url.endswith("embeddings"), (
461 - "OpenAI Embeddings API URL must end with 'embeddings'."
462 - )
463 -
464 - headers = {
465 - "Content-Type": "application/json",
466 - "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}",
467 - }
468 -
469 - payload = {
470 - "model": request_func_input.model,
471 - "input": request_func_input.prompt,
472 - }
473 -
493 +) -> RequestFuncOutput:
494 output = RequestFuncOutput()
495 st = time.perf_counter()
496 output.start_time = st
@@ -494,6 +514,137 @@ async def async_request_openai_embeddings(
514 return output
515
516
517 +async def async_request_openai_embeddings(
518 + request_func_input: RequestFuncInput,
519 + session: aiohttp.ClientSession,
520 + pbar: Optional[tqdm] = None,
521 +) -> RequestFuncOutput:
522 + api_url = request_func_input.api_url
523 + _validate_api_url(api_url, "OpenAI Embeddings API", "embeddings")
524 +
525 + payload = {
526 + "model": request_func_input.model_name
527 + if request_func_input.model_name
528 + else request_func_input.model,
529 + "input": request_func_input.prompt,
530 + }
531 + _update_payload_common(payload, request_func_input)
532 +
533 + headers = {
534 + "Content-Type": "application/json",
535 + "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}",
536 + }
537 + _update_headers_common(headers, request_func_input)
538 +
539 + return await _run_openai_embeddings(
540 + session,
541 + api_url,
542 + payload=payload,
543 + headers=headers,
544 + pbar=pbar,
545 + )
546 +
547 +
548 +async def async_request_openai_embeddings_chat(
549 + request_func_input: RequestFuncInput,
550 + session: aiohttp.ClientSession,
551 + pbar: Optional[tqdm] = None,
552 + mm_position: Literal["first", "last"] = "last",
553 +) -> RequestFuncOutput:
554 + api_url = request_func_input.api_url
555 + _validate_api_url(api_url, "OpenAI Embeddings API", "embeddings")
556 +
557 + content = _get_chat_content(request_func_input, mm_position=mm_position)
558 +
559 + payload = {
560 + "model": request_func_input.model_name
561 + if request_func_input.model_name
562 + else request_func_input.model,
563 + "messages": [
564 + {"role": "user", "content": content},
565 + ],
566 + }
567 + _update_payload_common(payload, request_func_input)
568 +
569 + headers = {
570 + "Content-Type": "application/json",
571 + "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}",
572 + }
573 + _update_headers_common(headers, request_func_input)
574 +
575 + return await _run_openai_embeddings(
576 + session,
577 + api_url,
578 + payload=payload,
579 + headers=headers,
580 + pbar=pbar,
581 + )
582 +
583 +
584 +async def async_request_openai_embeddings_clip(
585 + request_func_input: RequestFuncInput,
586 + session: aiohttp.ClientSession,
587 + pbar: Optional[tqdm] = None,
588 +) -> RequestFuncOutput:
589 + if request_func_input.multi_modal_content:
590 + # Image input
591 + request_func_input.prompt = ""
592 +
593 + # max_model_len=77 is too short for most datasets,
594 + # so by default we truncate the prompt to max_model_len
595 + if request_func_input.extra_body is None:
596 + request_func_input.extra_body = {}
597 + if "truncate_prompt_tokens" not in request_func_input.extra_body:
598 + request_func_input.extra_body["truncate_prompt_tokens"] = -1
599 +
600 + return await async_request_openai_embeddings_chat(
601 + request_func_input,
602 + session,
603 + pbar=pbar,
604 + )
605 +
606 +
607 +def _try_extract_request_idx(request_func_input: RequestFuncInput):
608 + if request_func_input.request_id:
609 + match = re.search(r"(\d+)$", request_func_input.request_id)
610 + if match:
611 + try:
612 + return int(match.group(1))
613 + except ValueError:
614 + pass
615 +
616 + return None
617 +
618 +
619 +async def async_request_openai_embeddings_vlm2vec(
620 + request_func_input: RequestFuncInput,
621 + session: aiohttp.ClientSession,
622 + pbar: Optional[tqdm] = None,
623 +) -> RequestFuncOutput:
624 + if request_func_input.multi_modal_content:
625 + request_idx = _try_extract_request_idx(request_func_input)
626 +
627 + # Adjust the ratio manually if needed.
628 + use_image_only_prompt = request_idx is None or request_idx % 2 == 0
629 +
630 + if use_image_only_prompt:
631 + # Image input
632 + request_func_input.prompt = "Represent the given image."
633 + else:
634 + # Text+Image input
635 + request_func_input.prompt = (
636 + f"Represent the given image with the following question: "
637 + f"{request_func_input.prompt}"
638 + )
639 +
640 + return await async_request_openai_embeddings_chat(
641 + request_func_input,
642 + session,
643 + pbar=pbar,
644 + mm_position="first",
645 + )
646 +
647 +
648 # TODO: Add more request functions for different API protocols.
649 ASYNC_REQUEST_FUNCS: dict[str, RequestFunc] = {
650 "vllm": async_request_openai_completions,
@@ -501,6 +652,9 @@ ASYNC_REQUEST_FUNCS: dict[str, RequestFunc] = {
652 "openai-chat": async_request_openai_chat_completions,
653 "openai-audio": async_request_openai_audio,
654 "openai-embeddings": async_request_openai_embeddings,
655 + "openai-embeddings-chat": async_request_openai_embeddings_chat,
656 + "openai-embeddings-clip": async_request_openai_embeddings_clip,
657 + "openai-embeddings-vlm2vec": async_request_openai_embeddings_vlm2vec,
658 }
659
660 OPENAI_COMPATIBLE_BACKENDS = [
vllm/benchmarks/serve.py
+34 -34
@@ -465,6 +465,7 @@ def calculate_metrics(
465
466
467 async def benchmark(
468 + task_type: TaskType,
469 endpoint_type: str,
470 api_url: str,
471 base_url: str,
@@ -490,18 +491,10 @@ async def benchmark(
491 ramp_up_end_rps: Optional[int] = None,
492 ready_check_timeout_sec: int = 600,
493 ):
493 - task_type = (
494 - TaskType.EMBEDDING
495 - if api_url.endswith("/v1/embeddings")
496 - else TaskType.GENERATION
497 - )
498 - if endpoint_type in ASYNC_REQUEST_FUNCS:
499 - if task_type == TaskType.EMBEDDING:
500 - request_func = ASYNC_REQUEST_FUNCS["openai-embeddings"]
501 - else:
502 - request_func = ASYNC_REQUEST_FUNCS[endpoint_type]
503 - else:
504 - raise ValueError(f"Unknown backend: {endpoint_type}")
494 + try:
495 + request_func = ASYNC_REQUEST_FUNCS[endpoint_type]
496 + except KeyError:
497 + raise ValueError(f"Unknown backend: {endpoint_type}") from None
498
499 # Reuses connections across requests to reduce TLS handshake overhead.
500 connector = aiohttp.TCPConnector(
@@ -1310,36 +1303,43 @@ async def main_async(args: argparse.Namespace) -> dict[str, Any]:
1303 input_requests = get_samples(args, tokenizer)
1304 goodput_config_dict = check_goodput_args(args)
1305
1306 + backend = args.backend
1307 + task_type = TaskType.EMBEDDING if "embeddings" in backend else TaskType.GENERATION
1308 +
1309 # Collect the sampling parameters.
1314 - sampling_params = {
1315 - k: v
1316 - for k, v in {
1317 - "top_p": args.top_p,
1318 - "top_k": args.top_k,
1319 - "min_p": args.min_p,
1320 - "temperature": args.temperature,
1321 - "frequency_penalty": args.frequency_penalty,
1322 - "presence_penalty": args.presence_penalty,
1323 - "repetition_penalty": args.repetition_penalty,
1324 - }.items()
1325 - if v is not None
1326 - }
1327 -
1328 - # Sampling parameters are only supported by openai-compatible backend.
1329 - if sampling_params and args.backend not in OPENAI_COMPATIBLE_BACKENDS:
1330 - raise ValueError(
1331 - "Sampling parameters are only supported by openai-compatible backends."
1332 - )
1310 + if task_type == TaskType.GENERATION:
1311 + sampling_params = {
1312 + k: v
1313 + for k, v in {
1314 + "top_p": args.top_p,
1315 + "top_k": args.top_k,
1316 + "min_p": args.min_p,
1317 + "temperature": args.temperature,
1318 + "frequency_penalty": args.frequency_penalty,
1319 + "presence_penalty": args.presence_penalty,
1320 + "repetition_penalty": args.repetition_penalty,
1321 + }.items()
1322 + if v is not None
1323 + }
1324 +
1325 + # Sampling parameters are only supported by openai-compatible backend.
1326 + if sampling_params and args.backend not in OPENAI_COMPATIBLE_BACKENDS:
1327 + raise ValueError(
1328 + "Sampling parameters are only supported by openai-compatible backends."
1329 + )
1330
1334 - if "temperature" not in sampling_params:
1335 - sampling_params["temperature"] = 0.0 # Default to greedy decoding.
1331 + if "temperature" not in sampling_params:
1332 + sampling_params["temperature"] = 0.0 # Default to greedy decoding.
1333 + else:
1334 + sampling_params = {}
1335
1336 # Avoid GC processing "static" data - reduce pause times.
1337 gc.collect()
1338 gc.freeze()
1339
1340 benchmark_result = await benchmark(
1342 - endpoint_type=args.backend,
1341 + task_type=task_type,
1342 + endpoint_type=backend,
1343 api_url=api_url,
1344 base_url=base_url,
1345 model_id=model_id,
vllm/entrypoints/chat_utils.py
+2 -2
@@ -498,14 +498,14 @@ def resolve_hf_chat_template(
498 tokenizer_name_or_path=model_config.tokenizer,
499 )
500 if path is not None:
501 - logger.info(
501 + logger.info_once(
502 "Loading chat template fallback for %s as there isn't one "
503 "defined on HF Hub.",
504 tokenizer.name_or_path,
505 )
506 chat_template = load_chat_template(path)
507 else:
508 - logger.debug(
508 + logger.debug_once(
509 "There is no chat template fallback for %s", tokenizer.name_or_path
510 )
511