A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 18:30 UTC
[Misc][Doc] Minor benchmark README update (#14874)
Signed-off-by: Roger Wang <ywang@roblox.com>
Roger Wang committed
Mar 15, 2025 at 18:46 UTC
3453b964a3ed84d99c9ae33bc0fae00790df36ef
1 file changed
+2
-2
benchmarks/README.md
+2
-2
@@ -82,10 +82,10 @@ Then run the benchmarking script
82
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
83
MODEL_NAME="NousResearch/Hermes-3-Llama-3.1-8B"
84
NUM_PROMPTS=10
85
-BACKEND="openai-chat"
85
+BACKEND="vllm"
86
DATASET_NAME="sharegpt"
87
DATASET_PATH="<your data path>/ShareGPT_V3_unfiltered_cleaned_split.json"
88
-python3 vllm/benchmarks/benchmark_serving.py --backend ${BACKEND} --model ${MODEL_NAME} --endpoint /v1/chat/completions --dataset-name ${DATASET_NAME} --dataset-path ${DATASET_PATH} --num-prompts ${NUM_PROMPTS}
88
+python3 vllm/benchmarks/benchmark_serving.py --backend ${BACKEND} --model ${MODEL_NAME} --endpoint /v1/completions --dataset-name ${DATASET_NAME} --dataset-path ${DATASET_PATH} --num-prompts ${NUM_PROMPTS}
89
```
90
91
If successful, you will see the following output