A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 14:45 UTC
[Doc] Changed explanation of generation_tokens_total and prompt_tokens_total counter type metrics to avoid confusion (#16784)
Signed-off-by: insukim1994 <insu.kim@moreh.io>
Insu Kim committed
Apr 17, 2025 at 23:10 UTC
7c02d6a137867a2c4007c6ff991922426af55124
1 file changed
+2
-2
docs/source/design/v1/metrics.md
+2
-2
@@ -66,8 +66,8 @@ vLLM also provides [a reference example](https://docs.vllm.ai/en/latest/getting_
66
The subset of metrics exposed in the Grafana dashboard gives us an indication of which metrics are especially important:
67
68
- `vllm:e2e_request_latency_seconds_bucket` - End to end request latency measured in seconds
69
-- `vllm:prompt_tokens_total` - Prompt Tokens/Sec
70
-- `vllm:generation_tokens_total` - Generation Tokens/Sec
69
+- `vllm:prompt_tokens_total` - Prompt Tokens
70
+- `vllm:generation_tokens_total` - Generation Tokens
71
- `vllm:time_per_output_token_seconds` - Inter token latency (Time Per Output Token, TPOT) in second.
72
- `vllm:time_to_first_token_seconds` - Time to First Token (TTFT) latency in seconds.
73
- `vllm:num_requests_running` (also, `_swapped` and `_waiting`) - Number of requests in RUNNING, WAITING, and SWAPPED state