A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 11, 23:45 UTC
docs: update metrics design doc to use new vllm:kv_cache_usage_perc (#30041)
Signed-off-by: Tim <tim.wang03@sap.com>
TimWang committed
Dec 5, 2025 at 07:37 UTC
690cc3ef20eec0d080b8e2fce397bf4f981beaf1
1 file changed
+1
-1
docs/design/metrics.md
+1
-1
@@ -62,7 +62,7 @@ The subset of metrics exposed in the Grafana dashboard gives us an indication of
62
- `vllm:time_per_output_token_seconds` - Inter-token latency (Time Per Output Token, TPOT) in seconds.
63
- `vllm:time_to_first_token_seconds` - Time to First Token (TTFT) latency in seconds.
64
- `vllm:num_requests_running` (also, `_swapped` and `_waiting`) - Number of requests in the RUNNING, WAITING, and SWAPPED states.
65
-- `vllm:gpu_cache_usage_perc` - Percentage of used cache blocks by vLLM.
65
+- `vllm:kv_cache_usage_perc` - Percentage of used cache blocks by vLLM.
66
- `vllm:request_prompt_tokens` - Request prompt length.
67
- `vllm:request_generation_tokens` - Request generation length.
68
- `vllm:request_success` - Number of finished requests by their finish reason: either an EOS token was generated or the max sequence length was reached.