@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 148 lines 5.67 KB
Rendered Raw
1 # Per-Request Metrics
2
3 vLLM can return per-request timing metrics directly in API responses.
4 This is useful for billing, SLA monitoring, and latency analysis at the
5 individual request level, as a complement to the server-aggregated Prometheus
6 metrics exposed at `/metrics`.
7
8 ## Enabling
9
10 Start the server with `--enable-per-request-metrics`:
11
12 ```bash
13 vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-per-request-metrics
14 ```
15
16 When this flag is set, supported API responses include metrics for each
17 attributable request.
18
19 !!! note
20 At high concurrency, enabling per-request metrics computation may introduce
21 non-negligible CPU overhead. Benchmark your specific workload to evaluate the
22 impact before enabling in production.
23
24 ## Response Format
25
26 When per-request metrics are enabled, the response includes a `metrics` object:
27
28 ```json
29 {
30 "id": "chatcmpl-abc123",
31 "object": "chat.completion",
32 "model": "meta-llama/Llama-3.1-8B-Instruct",
33 "choices": [ ... ],
34 "usage": {
35 "prompt_tokens": 42,
36 "completion_tokens": 128,
37 "total_tokens": 170
38 },
39 "metrics": {
40 "time_to_first_token_ms": 85.2,
41 "generation_time_ms": 1240.5,
42 "queue_time_ms": 12.3,
43 "mean_itl_ms": 9.1,
44 "tokens_per_second": 103.2
45 }
46 }
47 ```
48
49 | Field | Description |
50 | --- | --- |
51 | `time_to_first_token_ms` | Time from when the request was scheduled until the first output token was generated (TTFT). |
52 | `generation_time_ms` | Decode time: time from the first output token to the last output token. Excludes both queue wait and prefill/TTFT. |
53 | `queue_time_ms` | Time the request spent waiting in the scheduler queue before processing began. |
54 | `mean_itl_ms` | Mean inter-token latency (average time between successive output tokens) during the decode phase. `null` for single-token responses. |
55 | `tokens_per_second` | Overall output token throughput: all generated tokens over the inference interval (scheduling to last output token). Unlike `generation_time_ms`, this includes the prefill phase, so it reflects end-to-end generation speed rather than pure decode speed. |
56
57 All fields are `null` if the underlying timing data is not available for that
58 request.
59
60 !!! note
61 Timing metrics describe a single generation stream, so they are only
62 returned when the request maps to exactly one. They are suppressed (the
63 `metrics` object is `null`) for requests with `n > 1`, because the
64 underlying timing data reflects only one of the `n` sequences and cannot be
65 accurately attributed to the request as a whole. Token usage
66 (`prompt_tokens`, `completion_tokens`) remains accurate in these cases.
67 Per-request metrics also require server-side statistics logging, which is
68 on by default. vLLM rejects `--enable-per-request-metrics` when
69 `--disable-log-stats` is also set.
70
71 ## Example Request
72
73 === "Non-streaming"
74
75 ```python
76 from openai import OpenAI
77
78 client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")
79
80 response = client.chat.completions.create(
81 model="meta-llama/Llama-3.1-8B-Instruct",
82 messages=[{"role": "user", "content": "What is the capital of France?"}],
83 )
84
85 print(response.usage)
86 print(response.model_extra.get("metrics"))
87 ```
88
89 === "Streaming"
90
91 In streaming responses, metrics are attached to the final usage chunk (the
92 chunk sent after all content chunks). That chunk is only emitted when usage
93 reporting is enabled with `stream_options.include_usage: true` or forced
94 server-side with `--enable-force-include-usage`. Without forced usage, a
95 streaming client must set `stream_options.include_usage: true` to receive
96 metrics.
97
98 ```python
99 from openai import OpenAI
100
101 client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")
102
103 stream = client.chat.completions.create(
104 model="meta-llama/Llama-3.1-8B-Instruct",
105 messages=[{"role": "user", "content": "What is the capital of France?"}],
106 stream=True,
107 stream_options={"include_usage": True},
108 )
109
110 for chunk in stream:
111 if chunk.usage:
112 print("Usage:", chunk.usage)
113 print("Metrics:", chunk.model_extra.get("metrics"))
114 ```
115
116 ## Completions API
117
118 Per-request metrics are also available on the `/v1/completions` endpoint using
119 the same `metrics` response field. As with `n > 1`, metrics are omitted for
120 requests with multiple prompts, because the timing data cannot be attributed to
121 a single prompt's generation.
122
123 ## Responses API
124
125 Per-request metrics are available on the `/v1/responses` endpoint using the
126 common `metrics` object described above. Non-streaming responses include it at
127 the top level. Streaming responses include it in the final response carried by
128 the `response.completed` event; intermediate events do not include metrics.
129
130 Metrics are omitted for Responses requests that perform multiple
131 model-generation turns, such as built-in tool-call workflows, because the
132 response retains timing data for only one generation turn while token usage is
133 accumulated across all turns.
134
135 ## Relationship to Prometheus Metrics
136
137 The `metrics` response field provides per-request values for a single request.
138 The `/metrics` Prometheus endpoint exposes server-level histograms (e.g.
139 `vllm:time_to_first_token_seconds`) that aggregate across all requests.
140
141 ## Speculative Decoding Acceptance
142
143 When speculative decoding is enabled, per-request acceptance metrics
144 (mean acceptance length and the accepted-draft-length distribution) can be
145 returned via `--per-request-spec-decode-metrics`. They share this `metrics`
146 object as `metrics.speculative_decoding`, and — like the timing fields — are
147 reported only for single-sequence (`n == 1`) requests. See
148 [Per-Request Acceptance Metrics](speculative_decoding/acceptance_metrics.md).