A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
[Misc] Update benchmark_prefix_caching.py fixed example usage (#11920)
Signed-off-by: Ren MinMin <renmm6@chinaunicom.cn> Co-authored-by: Ren MinMin <renmm6@chinaunicom.cn>
minmin committed
Jan 11, 2025 at 04:39 UTC
8a579408f33e2f98a89d57418342b53a41622d2f
1 file changed
+2
-1
benchmarks/benchmark_prefix_caching.py
+2
-1
@@ -10,7 +10,8 @@ Fixed example usage:
10
--model meta-llama/Llama-2-7b-chat-hf \
11
--enable-prefix-caching \
12
--num-prompts 1 \
13
- --repeat-count 100
13
+ --repeat-count 100 \
14
+ --input-length-range 128:256
15
16
ShareGPT example usage:
17
# This command samples 20 prompts with input lengths