[Benchmark] Warn on warm prefix cache for random serve runs (#53920)
Signed-off-by: Zupeng Wang <71580390+zupengwang@users.noreply.github.com> Co-authored-by: OpenAI <support@openai.com>
Zupeng Wang committed
Aug 27, 2026 at 13:51 UTC
833fa6233c61b34ea37397a778c00e545c4732f4
1 file changed
+9
docs/benchmarking/cli.md
+9
@@ -111,6 +111,15 @@ P99 ITL (ms): 8.39
111
==================================================
112
```
113
114
+!!! warning
115
+ Repeating `vllm bench serve` against the same server can reuse prompts left
116
+ in the prefix cache and inflate throughput. This can affect any reproducible
117
+ dataset; the synthetic random dataset is reproducible for a fixed `--seed`,
118
+ which defaults to `0`. Prefix cache hits can also come from shared prefixes
119
+ within the run, so interpret cache metrics in the context of the workload.
120
+ If cache reuse is not intended, vary `--seed`, reset or restart the server,
121
+ or use `vllm bench sweep serve`, which resets server caches between runs.
122
+
123
#### Understanding the Latency Metrics
124
125
`vllm bench serve` measures latency at the benchmark client: