@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 8a579408f3
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 03:15 UTC

[Misc] Update benchmark_prefix_caching.py fixed example usage (#11920)

Signed-off-by: Ren MinMin <renmm6@chinaunicom.cn> Co-authored-by: Ren MinMin <renmm6@chinaunicom.cn>

minmin committed Jan 11, 2025 at 04:39 UTC 8a579408f33e2f98a89d57418342b53a41622d2f
1 file changed +2 -1
benchmarks/benchmark_prefix_caching.py
+2 -1
@@ -10,7 +10,8 @@ Fixed example usage:
10 --model meta-llama/Llama-2-7b-chat-hf \
11 --enable-prefix-caching \
12 --num-prompts 1 \
13 - --repeat-count 100
13 + --repeat-count 100 \
14 + --input-length-range 128:256
15
16 ShareGPT example usage:
17 # This command samples 20 prompts with input lengths