A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 09:45 UTC
[Doc] Update prefix cache metrics to counting tokens (#18138)
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Chen Zhang committed
May 14, 2025 at 23:23 UTC
964472b9667508b1d4a7ed92068ff81740ae0036
1 file changed
+2
-2
docs/source/design/v1/metrics.md
+2
-2
@@ -415,8 +415,8 @@ The discussion in <gh-issue:10582> about adding prefix cache metrics yielded
415
some interesting points which may be relevant to how we approach
416
future metrics.
417
418
-Every time the prefix cache is queried, we record the number of blocks
419
-queried and the number of queried blocks present in the cache
418
+Every time the prefix cache is queried, we record the number of tokens
419
+queried and the number of queried tokens present in the cache
420
(i.e. hits).
421
422
However, the metric of interest is the hit rate - i.e. the number of