@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 124 lines 5.51 KB
Rendered Raw
1 # Production Metrics
2
3 vLLM exposes a number of metrics that can be used to monitor the health of the
4 system. These metrics are exposed via the `/metrics` endpoint on the vLLM
5 OpenAI compatible API server.
6
7 You can start the server using Python, or using [Docker](../deployment/docker.md):
8
9 ```bash
10 vllm serve unsloth/Llama-3.2-1B-Instruct
11 ```
12
13 Then query the endpoint to get the latest metrics from the server:
14
15 ??? console "Output"
16
17 ```console
18 $ curl http://0.0.0.0:8000/metrics
19
20 # HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
21 # TYPE vllm:iteration_tokens_total histogram
22 vllm:iteration_tokens_total_sum{model_name="unsloth/Llama-3.2-1B-Instruct"} 0.0
23 vllm:iteration_tokens_total_bucket{le="1.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
24 vllm:iteration_tokens_total_bucket{le="8.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
25 vllm:iteration_tokens_total_bucket{le="16.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
26 vllm:iteration_tokens_total_bucket{le="32.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
27 vllm:iteration_tokens_total_bucket{le="64.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
28 vllm:iteration_tokens_total_bucket{le="128.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
29 vllm:iteration_tokens_total_bucket{le="256.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
30 vllm:iteration_tokens_total_bucket{le="512.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
31 ...
32 ```
33
34 The following metrics are exposed:
35
36 ## General Metrics
37
38 --8<-- "gen:metrics-general"
39
40 ## Speculative Decoding Metrics
41
42 --8<-- "gen:metrics-spec-decode"
43
44 ## NIXL KV Connector Metrics
45
46 --8<-- "gen:metrics-nixl"
47
48 ## Simple CPU Offload Connector Metrics
49
50 These metrics are exposed when the `SimpleCPUOffloadConnector` KV connector
51 is configured (e.g. `--kv-transfer-config='{"kv_connector":
52 "SimpleCPUOffloadConnector", "kv_role": "kv_both", "kv_connector_extra_config":
53 {"kv_offload_backend": "disk", "disk_path": "/mnt/nvme/kv"}}'`). They are
54 updated once per engine step.
55
56 Caveats to keep in mind when interpreting them:
57
58 - A "completed" store means the write syscalls returned; it is **not**
59 fsync-durable, and the disk backend's file is process-lifetime scratch,
60 unlinked at startup and shutdown.
61 - `save_outcomes_total` classifies eager-mode boundary hand-off stores;
62 lazy-mode stores are not classified.
63 - `used_blocks` counts blocks pinned by in-flight transfers or cache hits;
64 warm cached blocks that are evictable are not counted. Use the
65 `capacity_blocks` label of `simple_kv_offload_info` as the denominator.
66 - Counters and gauges are quantized to engine steps. They are reported by the
67 scheduler process and reflect engine-wide logical block counts, not values
68 pooled from individual tensor-parallel workers.
69
70 --8<-- "gen:metrics-simple-kv-offload"
71
72 ## Model Flops Utilization (MFU) Performance Metrics
73
74 These metrics are available via `--enable-mfu-metrics`:
75
76 --8<-- "gen:metrics-mfu"
77
78 ## Custom Histogram Buckets
79
80 The core engine histograms ship with default bucket boundaries tuned for
81 typical serving workloads. The `--custom-histogram-buckets` option replaces
82 the boundaries of one or more *bucket families* — exactly the histograms
83 listed in the table below — with your own list; histograms owned by other
84 subsystems (for example, the NIXL connector metrics) are not affected. Use it,
85 for example, to track sub-300ms latency SLOs with the request-phase
86 histograms, whose smallest default boundary is 0.3s:
87
88 ```bash
89 vllm serve Qwen/Qwen3-0.6B \
90 --custom-histogram-buckets '{"request_latency": [0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0, 30.0]}'
91 ```
92
93 Each family key overrides a group of related histograms:
94
95 | Family key | Histograms |
96 | --- | --- |
97 | `request_latency` | `vllm:e2e_request_latency_seconds`, `vllm:request_queue_time_seconds`, `vllm:request_inference_time_seconds`, `vllm:request_prefill_time_seconds`, `vllm:request_decode_time_seconds` |
98 | `time_to_first_token` | `vllm:time_to_first_token_seconds` |
99 | `inter_token_latency` | `vllm:inter_token_latency_seconds`, `vllm:request_time_per_output_token_seconds` |
100 | `iteration_tokens` | `vllm:iteration_tokens_total` |
101 | `request_params_n` | `vllm:request_params_n` |
102 | `request_num_preemptions` | `vllm:request_num_preemptions` |
103 | `request_tokens` | `vllm:request_prompt_tokens`, `vllm:request_generation_tokens`, `vllm:request_max_num_generation_tokens`, `vllm:request_params_max_tokens`, `vllm:request_prefill_kv_computed_tokens` |
104 | `kv_cache_residency` | `vllm:kv_block_lifetime_seconds`, `vllm:kv_block_idle_before_evict_seconds`, `vllm:kv_block_reuse_gap_seconds` |
105
106 Bucket values must be positive, finite, and strictly increasing; unknown
107 family keys are rejected at startup. Families you do not list keep their
108 default boundaries. The `request_tokens` defaults normally scale with
109 `--max-model-len`; an override replaces that computed list. The
110 `kv_cache_residency` family only takes effect when `--kv-cache-metrics` is
111 enabled.
112
113 !!! warning "Bucket cardinality"
114 Every bucket boundary creates one extra time series per metric and per
115 label combination (model and engine index, multiplied under data-parallel
116 deployments). Long bucket lists inflate Prometheus storage, scrape sizes,
117 and query costs. Keep custom lists short, and only override the families
118 you actively monitor.
119
120 ## Deprecation Policy
121
122 Note: when metrics are deprecated in version `X.Y`, they are hidden in version `X.Y+1`
123 but can be re-enabled using the `--show-hidden-metrics-for-version=X.Y` escape hatch,
124 and are then removed in version `X.Y+2`.