@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / d7f396486e
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 04:15 UTC

Update comment (#2934)

Ronen Schaffer committed Feb 22, 2024 at 04:18 UTC d7f396486e3e9b4dd31020c81c6eb446593b586d
1 file changed +1 -1
benchmarks/benchmark_serving.py
+1 -1
@@ -7,7 +7,7 @@ On the server side, run one of the following commands:
7 --disable-log-requests
8
9 (TGI backend)
10 - ./launch_hf_server.sh <your_model>
10 + ./launch_tgi_server.sh <your_model> <max_batch_total_tokens>
11
12 On the client side, run:
13 python benchmarks/benchmark_serving.py \