@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 5959564f94
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 03:15 UTC

Doc fix in `benchmark_long_document_qa_throughput.py` (#11933)

Signed-off-by: Kuntai Du <kuntai@uchicago.edu>

Kuntai Du committed Jan 10, 2025 at 23:51 UTC 5959564f94180a6a50e0d394e35a035c0c98a7fb
1 file changed +1 -2
benchmarks/benchmark_long_document_qa_throughput.py
+1 -2
@@ -2,8 +2,7 @@
2 Offline benchmark to test the long document QA throughput.
3
4 Example usage:
5 - # This command run the vllm with 50GB CPU memory for offloading
6 - # The workload samples 8 different prompts with a default input
5 + # This workload samples 8 different prompts with a default input
6 # length of 20000 tokens, then replicates each prompt 2 times
7 # in random order.
8 python benchmark_long_document_qa_throughput.py \