@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / bd0e7802e0
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 00:30 UTC

[Bugfix] Add warmup for prefix caching example (#5235)

Zhuohan Li committed Jun 3, 2024 at 19:36 UTC bd0e7802e09e40060b857b85227fd93a6a739467
1 file changed +4 -2
examples/offline_inference_with_prefix.py
+4 -2
@@ -51,8 +51,10 @@ for output in outputs:
51
52 print("-" * 80)
53
54 -# The llm.generate call will batch all prompts and send the batch at once
55 -# if resources allow.
54 +# Warmup so that the shared prompt's KV cache is computed.
55 +prefix_cached_llm.generate(generating_prompts[0], sampling_params)
56 +
57 +# Generate with prefix caching.
58 start_time_cached = time()
59 outputs = prefix_cached_llm.generate(generating_prompts, sampling_params)
60 duration_cached = time() - start_time_cached