A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 12, 02:15 UTC
[docs] Remove _total from counter metrics names (#30028)
In Prometheus Counters always expose their actual numeric value with a metric name that ends in _total. We should document the base name, as this what appears in the get_metrics() API. Signed-off-by: CYJiang <86391540+googs1025@users.noreply.github.com>
CYJiang committed
Dec 4, 2025 at 15:46 UTC
fd68e909db1804f211707bb027a49b82bb5c2d8f
1 file changed
+6
-6
docs/design/metrics.md
+6
-6
@@ -57,15 +57,15 @@ vLLM also provides [a reference example](../../examples/online_serving/prometheu
57
The subset of metrics exposed in the Grafana dashboard gives us an indication of which metrics are especially important:
58
59
- `vllm:e2e_request_latency_seconds_bucket` - End to end request latency measured in seconds.
60
-- `vllm:prompt_tokens_total` - Prompt tokens.
61
-- `vllm:generation_tokens_total` - Generation tokens.
60
+- `vllm:prompt_tokens` - Prompt tokens.
61
+- `vllm:generation_tokens` - Generation tokens.
62
- `vllm:time_per_output_token_seconds` - Inter-token latency (Time Per Output Token, TPOT) in seconds.
63
- `vllm:time_to_first_token_seconds` - Time to First Token (TTFT) latency in seconds.
64
- `vllm:num_requests_running` (also, `_swapped` and `_waiting`) - Number of requests in the RUNNING, WAITING, and SWAPPED states.
65
- `vllm:gpu_cache_usage_perc` - Percentage of used cache blocks by vLLM.
66
- `vllm:request_prompt_tokens` - Request prompt length.
67
- `vllm:request_generation_tokens` - Request generation length.
68
-- `vllm:request_success_total` - Number of finished requests by their finish reason: either an EOS token was generated or the max sequence length was reached.
68
+- `vllm:request_success` - Number of finished requests by their finish reason: either an EOS token was generated or the max sequence length was reached.
69
- `vllm:request_queue_time_seconds` - Queue time.
70
- `vllm:request_prefill_time_seconds` - Requests prefill time.
71
- `vllm:request_decode_time_seconds` - Requests decode time.
@@ -571,9 +571,9 @@ model and then validate those tokens with the larger model.
571
572
- `vllm:spec_decode_draft_acceptance_rate` (Gauge)
573
- `vllm:spec_decode_efficiency` (Gauge)
574
-- `vllm:spec_decode_num_accepted_tokens_total` (Counter)
575
-- `vllm:spec_decode_num_draft_tokens_total` (Counter)
576
-- `vllm:spec_decode_num_emitted_tokens_total` (Counter)
574
+- `vllm:spec_decode_num_accepted_tokens` (Counter)
575
+- `vllm:spec_decode_num_draft_tokens` (Counter)
576
+- `vllm:spec_decode_num_emitted_tokens` (Counter)
577
578
There is a PR under review (<https://github.com/vllm-project/vllm/pull/12193>) to add "prompt lookup (ngram)"
579
speculative decoding to v1. Other techniques will follow. We should