A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 04:15 UTC
Update comment (#2934)
Ronen Schaffer committed
Feb 22, 2024 at 04:18 UTC
d7f396486e3e9b4dd31020c81c6eb446593b586d
1 file changed
+1
-1
benchmarks/benchmark_serving.py
+1
-1
@@ -7,7 +7,7 @@ On the server side, run one of the following commands:
7
--disable-log-requests
8
9
(TGI backend)
10
- ./launch_hf_server.sh <your_model>
10
+ ./launch_tgi_server.sh <your_model> <max_batch_total_tokens>
11
12
On the client side, run:
13
python benchmarks/benchmark_serving.py \