A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
Doc fix in `benchmark_long_document_qa_throughput.py` (#11933)
Signed-off-by: Kuntai Du <kuntai@uchicago.edu>
Kuntai Du committed
Jan 10, 2025 at 23:51 UTC
5959564f94180a6a50e0d394e35a035c0c98a7fb
1 file changed
+1
-2
benchmarks/benchmark_long_document_qa_throughput.py
+1
-2
@@ -2,8 +2,7 @@
2
Offline benchmark to test the long document QA throughput.
3
4
Example usage:
5
- # This command run the vllm with 50GB CPU memory for offloading
6
- # The workload samples 8 different prompts with a default input
5
+ # This workload samples 8 different prompts with a default input
6
# length of 20000 tokens, then replicates each prompt 2 times
7
# in random order.
8
python benchmark_long_document_qa_throughput.py \