A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Benchmark] Add flag --served-model-name to benchmark_serving_multi_turn (#22889)
Signed-off-by: daniels <daniels@pliops.com>
Daniel Serebrenik committed
Aug 19, 2025 at 10:48 UTC
3c8a7872471a96b414f9861653dded55aa431795
2 files changed
+20
-6
benchmarks/multi_turn/README.md
+7
-5
@@ -5,11 +5,13 @@ The requirements (pip) for `benchmark_serving_multi_turn.py` can be found in `re
5
First start serving your model
6
7
```bash
8
-export MODEL_NAME=/models/meta-llama/Meta-Llama-3.1-8B-Instruct/
8
+export MODEL_PATH=/models/meta-llama/Meta-Llama-3.1-8B-Instruct/
9
10
-vllm serve $MODEL_NAME --disable-log-requests
10
+vllm serve $MODEL_PATH --served-model-name Llama --disable-log-requests
11
```
12
13
+The variable `MODEL_PATH` should be a path to the model files (e.g. downloaded from huggingface).
14
+
15
## Synthetic Multi-Turn Conversations
16
17
Download the following text file (used for generation of synthetic conversations)
@@ -26,10 +28,10 @@ But you may use other text files if you prefer (using this specific file is not
28
Then run the benchmarking script
29
30
```bash
29
-export MODEL_NAME=/models/meta-llama/Meta-Llama-3.1-8B-Instruct/
31
+export MODEL_PATH=/models/meta-llama/Meta-Llama-3.1-8B-Instruct/
32
31
-python benchmark_serving_multi_turn.py --model $MODEL_NAME --input-file generate_multi_turn.json \
32
---num-clients 2 --max-active-conversations 6
33
+python benchmark_serving_multi_turn.py --model $MODEL_PATH --served-model-name Llama \
34
+--input-file generate_multi_turn.json --num-clients 2 --max-active-conversations 6
35
```
36
37
You can edit the file `generate_multi_turn.json` to change the conversation parameters (number of turns, etc.).
benchmarks/multi_turn/benchmark_serving_multi_turn.py
+13
-1
@@ -825,9 +825,11 @@ def get_client_config(
825
826
# Arguments for API requests
827
chat_url = f"{args.url}/v1/chat/completions"
828
+ model_name = args.served_model_name if args.served_model_name else args.model
829
+
830
req_args = RequestArgs(
831
chat_url=chat_url,
830
- model=args.model,
832
+ model=model_name,
833
stream=not args.no_stream,
834
limit_min_tokens=args.limit_min_tokens,
835
limit_max_tokens=args.limit_max_tokens,
@@ -1247,9 +1249,19 @@ async def main() -> None:
1249
default=0,
1250
help="Seed for random number generators (default: 0)",
1251
)
1252
+
1253
parser.add_argument(
1254
"-m", "--model", type=str, required=True, help="Path of the LLM model"
1255
)
1256
+ parser.add_argument(
1257
+ "--served-model-name",
1258
+ type=str,
1259
+ default=None,
1260
+ help="The model name used in the API. "
1261
+ "If not specified, the model name will be the "
1262
+ "same as the ``--model`` argument. ",
1263
+ )
1264
+
1265
parser.add_argument(
1266
"-u",
1267
"--url",