Add Understanding the Latency Metrics docs (#50600)
Signed-off-by: mgoin <mgoin64@gmail.com>
Michael Goin committed
Aug 1, 2026 at 02:22 UTC
4ee9702bee668a447e9983a6aefc16ebbc3ad32e
3 files changed
+193
docs/assets/benchmarking/latency-metrics-speculative-decoding-dark.svg
new
+77
@@ -0,0 +1,77 @@
1
+<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="560" viewBox="0 0 1200 560" role="img" aria-labelledby="title desc">
2
+ <title id="title">Latency metrics with token bundling</title>
3
+ <desc id="desc">The client receives token 1 at 100 milliseconds, tokens 2 through 4 together at 140 milliseconds, and token 5 at 180 milliseconds. The two observed ITL gaps average 40 milliseconds. TPOT divides the 80 millisecond generation span by the four tokens after the first, giving 20 milliseconds per token.</desc>
4
+ <defs>
5
+ <marker id="axis-arrow" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto">
6
+ <path d="M0,0 L8,4 L0,8 Z" fill="#94a3b8"/>
7
+ </marker>
8
+ <marker id="itl-arrow" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto">
9
+ <path d="M0,0 L8,4 L0,8 Z" fill="#30a2ff"/>
10
+ </marker>
11
+ </defs>
12
+
13
+ <rect width="1200" height="560" rx="18" fill="#111827"/>
14
+ <text x="56" y="48" fill="#f5f7f9" font-family="Inter, system-ui, sans-serif" font-size="21" font-weight="600">Latency with token bundling</text>
15
+ <text x="1144" y="48" text-anchor="end" fill="#aeb8c5" font-family="Inter, system-ui, sans-serif" font-size="14">5 tokens · 3 stream outputs</text>
16
+
17
+ <line x1="88" y1="210" x2="1118" y2="210" stroke="#64748b" stroke-width="2" marker-end="url(#axis-arrow)"/>
18
+
19
+ <text x="100" y="153" text-anchor="middle" fill="#f5f7f9" font-family="Inter, system-ui, sans-serif" font-size="15" font-weight="600">Request</text>
20
+ <line x1="100" y1="170" x2="100" y2="225" stroke="#94a3b8" stroke-width="2"/>
21
+ <circle cx="100" cy="210" r="6" fill="#94a3b8"/>
22
+ <text x="100" y="246" text-anchor="middle" fill="#aeb8c5" font-family="Inter, system-ui, sans-serif" font-size="14">0 ms</text>
23
+
24
+ <text x="350" y="92" text-anchor="middle" fill="#f5f7f9" font-family="Inter, system-ui, sans-serif" font-size="15" font-weight="600">Output 1</text>
25
+ <rect x="314" y="106" width="72" height="54" rx="10" fill="#1d2733" stroke="#30a2ff" stroke-width="2"/>
26
+ <rect x="328" y="117" width="44" height="32" rx="6" fill="#173b58"/>
27
+ <text x="350" y="139" text-anchor="middle" fill="#a8d9ff" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">T1</text>
28
+ <line x1="350" y1="160" x2="350" y2="225" stroke="#30a2ff" stroke-width="2"/>
29
+ <circle cx="350" cy="210" r="6" fill="#30a2ff"/>
30
+ <text x="350" y="246" text-anchor="middle" fill="#aeb8c5" font-family="Inter, system-ui, sans-serif" font-size="14">100 ms</text>
31
+
32
+ <text x="700" y="80" text-anchor="middle" fill="#f5f7f9" font-family="Inter, system-ui, sans-serif" font-size="15" font-weight="600">Output 2 · bundled</text>
33
+ <rect x="610" y="94" width="180" height="66" rx="10" fill="#1d2733" stroke="#30a2ff" stroke-width="2"/>
34
+ <rect x="628" y="111" width="44" height="32" rx="6" fill="#173b58"/>
35
+ <rect x="678" y="111" width="44" height="32" rx="6" fill="#173b58"/>
36
+ <rect x="728" y="111" width="44" height="32" rx="6" fill="#173b58"/>
37
+ <text x="650" y="133" text-anchor="middle" fill="#a8d9ff" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">T2</text>
38
+ <text x="700" y="133" text-anchor="middle" fill="#a8d9ff" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">T3</text>
39
+ <text x="750" y="133" text-anchor="middle" fill="#a8d9ff" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">T4</text>
40
+ <line x1="700" y1="160" x2="700" y2="225" stroke="#30a2ff" stroke-width="2"/>
41
+ <circle cx="700" cy="210" r="6" fill="#30a2ff"/>
42
+ <text x="700" y="246" text-anchor="middle" fill="#aeb8c5" font-family="Inter, system-ui, sans-serif" font-size="14">140 ms</text>
43
+
44
+ <text x="1050" y="92" text-anchor="middle" fill="#f5f7f9" font-family="Inter, system-ui, sans-serif" font-size="15" font-weight="600">Output 3</text>
45
+ <rect x="1014" y="106" width="72" height="54" rx="10" fill="#1d2733" stroke="#30a2ff" stroke-width="2"/>
46
+ <rect x="1028" y="117" width="44" height="32" rx="6" fill="#173b58"/>
47
+ <text x="1050" y="139" text-anchor="middle" fill="#a8d9ff" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">T5</text>
48
+ <line x1="1050" y1="160" x2="1050" y2="225" stroke="#30a2ff" stroke-width="2"/>
49
+ <circle cx="1050" cy="210" r="6" fill="#30a2ff"/>
50
+ <text x="1050" y="246" text-anchor="middle" fill="#aeb8c5" font-family="Inter, system-ui, sans-serif" font-size="14">180 ms</text>
51
+
52
+ <path d="M100 274 H350" fill="none" stroke="#94a3b8" stroke-width="2"/>
53
+ <path d="M100 268 V280 M350 268 V280" fill="none" stroke="#94a3b8" stroke-width="2"/>
54
+ <text x="225" y="302" text-anchor="middle" fill="#aeb8c5" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">TTFT 100 ms</text>
55
+
56
+ <line x1="365" y1="274" x2="685" y2="274" stroke="#30a2ff" stroke-width="3" marker-end="url(#itl-arrow)"/>
57
+ <text x="525" y="302" text-anchor="middle" fill="#8dceff" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">ITL 40 ms</text>
58
+ <line x1="715" y1="274" x2="1035" y2="274" stroke="#30a2ff" stroke-width="3" marker-end="url(#itl-arrow)"/>
59
+ <text x="875" y="302" text-anchor="middle" fill="#8dceff" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">ITL 40 ms</text>
60
+
61
+ <path d="M350 326 V338 H1050 V326" fill="none" stroke="#ebb431" stroke-width="3"/>
62
+ <text x="700" y="365" text-anchor="middle" fill="#f1cf72" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">80 ms generation span</text>
63
+
64
+ <rect x="56" y="398" width="520" height="124" rx="12" fill="#172b3c" stroke="#285878"/>
65
+ <rect x="56" y="398" width="6" height="124" rx="3" fill="#30a2ff"/>
66
+ <text x="86" y="431" fill="#8dceff" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="600">Mean ITL</text>
67
+ <text x="86" y="473" fill="#f5f7f9" font-family="Inter, system-ui, sans-serif" font-size="20">(40 ms + 40 ms) ÷ 2</text>
68
+ <text x="546" y="468" text-anchor="end" fill="#8dceff" font-family="Inter, system-ui, sans-serif" font-size="30" font-weight="600">40 ms</text>
69
+ <text x="86" y="500" fill="#aeb8c5" font-family="Inter, system-ui, sans-serif" font-size="13">2 observed gaps between stream outputs</text>
70
+
71
+ <rect x="624" y="398" width="520" height="124" rx="12" fill="#2b2718" stroke="#66562b"/>
72
+ <rect x="624" y="398" width="6" height="124" rx="3" fill="#ebb431"/>
73
+ <text x="654" y="431" fill="#f1cf72" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="600">TPOT</text>
74
+ <text x="654" y="473" fill="#f5f7f9" font-family="Inter, system-ui, sans-serif" font-size="20">(180 − 100) ms ÷ (5 − 1)</text>
75
+ <text x="1114" y="468" text-anchor="end" fill="#f1cf72" font-family="Inter, system-ui, sans-serif" font-size="30" font-weight="600">20 ms/token</text>
76
+ <text x="654" y="500" fill="#aeb8c5" font-family="Inter, system-ui, sans-serif" font-size="13">4 output tokens after the first</text>
77
+</svg>
docs/assets/benchmarking/latency-metrics-speculative-decoding-light.svg
new
+77
@@ -0,0 +1,77 @@
1
+<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="560" viewBox="0 0 1200 560" role="img" aria-labelledby="title desc">
2
+ <title id="title">Latency metrics with token bundling</title>
3
+ <desc id="desc">The client receives token 1 at 100 milliseconds, tokens 2 through 4 together at 140 milliseconds, and token 5 at 180 milliseconds. The two observed ITL gaps average 40 milliseconds. TPOT divides the 80 millisecond generation span by the four tokens after the first, giving 20 milliseconds per token.</desc>
4
+ <defs>
5
+ <marker id="axis-arrow" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto">
6
+ <path d="M0,0 L8,4 L0,8 Z" fill="#94a3b8"/>
7
+ </marker>
8
+ <marker id="itl-arrow" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto">
9
+ <path d="M0,0 L8,4 L0,8 Z" fill="#30a2ff"/>
10
+ </marker>
11
+ </defs>
12
+
13
+ <rect width="1200" height="560" rx="18" fill="#f8fafc"/>
14
+ <text x="56" y="48" fill="#454545" font-family="Inter, system-ui, sans-serif" font-size="21" font-weight="600">Latency with token bundling</text>
15
+ <text x="1144" y="48" text-anchor="end" fill="#64748b" font-family="Inter, system-ui, sans-serif" font-size="14">5 tokens · 3 stream outputs</text>
16
+
17
+ <line x1="88" y1="210" x2="1118" y2="210" stroke="#94a3b8" stroke-width="2" marker-end="url(#axis-arrow)"/>
18
+
19
+ <text x="100" y="153" text-anchor="middle" fill="#454545" font-family="Inter, system-ui, sans-serif" font-size="15" font-weight="600">Request</text>
20
+ <line x1="100" y1="170" x2="100" y2="225" stroke="#64748b" stroke-width="2"/>
21
+ <circle cx="100" cy="210" r="6" fill="#64748b"/>
22
+ <text x="100" y="246" text-anchor="middle" fill="#64748b" font-family="Inter, system-ui, sans-serif" font-size="14">0 ms</text>
23
+
24
+ <text x="350" y="92" text-anchor="middle" fill="#454545" font-family="Inter, system-ui, sans-serif" font-size="15" font-weight="600">Output 1</text>
25
+ <rect x="314" y="106" width="72" height="54" rx="10" fill="#ffffff" stroke="#30a2ff" stroke-width="2"/>
26
+ <rect x="328" y="117" width="44" height="32" rx="6" fill="#eaf6ff"/>
27
+ <text x="350" y="139" text-anchor="middle" fill="#1769aa" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">T1</text>
28
+ <line x1="350" y1="160" x2="350" y2="225" stroke="#30a2ff" stroke-width="2"/>
29
+ <circle cx="350" cy="210" r="6" fill="#30a2ff"/>
30
+ <text x="350" y="246" text-anchor="middle" fill="#64748b" font-family="Inter, system-ui, sans-serif" font-size="14">100 ms</text>
31
+
32
+ <text x="700" y="80" text-anchor="middle" fill="#454545" font-family="Inter, system-ui, sans-serif" font-size="15" font-weight="600">Output 2 · bundled</text>
33
+ <rect x="610" y="94" width="180" height="66" rx="10" fill="#ffffff" stroke="#30a2ff" stroke-width="2"/>
34
+ <rect x="628" y="111" width="44" height="32" rx="6" fill="#eaf6ff"/>
35
+ <rect x="678" y="111" width="44" height="32" rx="6" fill="#eaf6ff"/>
36
+ <rect x="728" y="111" width="44" height="32" rx="6" fill="#eaf6ff"/>
37
+ <text x="650" y="133" text-anchor="middle" fill="#1769aa" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">T2</text>
38
+ <text x="700" y="133" text-anchor="middle" fill="#1769aa" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">T3</text>
39
+ <text x="750" y="133" text-anchor="middle" fill="#1769aa" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">T4</text>
40
+ <line x1="700" y1="160" x2="700" y2="225" stroke="#30a2ff" stroke-width="2"/>
41
+ <circle cx="700" cy="210" r="6" fill="#30a2ff"/>
42
+ <text x="700" y="246" text-anchor="middle" fill="#64748b" font-family="Inter, system-ui, sans-serif" font-size="14">140 ms</text>
43
+
44
+ <text x="1050" y="92" text-anchor="middle" fill="#454545" font-family="Inter, system-ui, sans-serif" font-size="15" font-weight="600">Output 3</text>
45
+ <rect x="1014" y="106" width="72" height="54" rx="10" fill="#ffffff" stroke="#30a2ff" stroke-width="2"/>
46
+ <rect x="1028" y="117" width="44" height="32" rx="6" fill="#eaf6ff"/>
47
+ <text x="1050" y="139" text-anchor="middle" fill="#1769aa" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">T5</text>
48
+ <line x1="1050" y1="160" x2="1050" y2="225" stroke="#30a2ff" stroke-width="2"/>
49
+ <circle cx="1050" cy="210" r="6" fill="#30a2ff"/>
50
+ <text x="1050" y="246" text-anchor="middle" fill="#64748b" font-family="Inter, system-ui, sans-serif" font-size="14">180 ms</text>
51
+
52
+ <path d="M100 274 H350" fill="none" stroke="#64748b" stroke-width="2"/>
53
+ <path d="M100 268 V280 M350 268 V280" fill="none" stroke="#64748b" stroke-width="2"/>
54
+ <text x="225" y="302" text-anchor="middle" fill="#64748b" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">TTFT 100 ms</text>
55
+
56
+ <line x1="365" y1="274" x2="685" y2="274" stroke="#30a2ff" stroke-width="3" marker-end="url(#itl-arrow)"/>
57
+ <text x="525" y="302" text-anchor="middle" fill="#1769aa" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">ITL 40 ms</text>
58
+ <line x1="715" y1="274" x2="1035" y2="274" stroke="#30a2ff" stroke-width="3" marker-end="url(#itl-arrow)"/>
59
+ <text x="875" y="302" text-anchor="middle" fill="#1769aa" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">ITL 40 ms</text>
60
+
61
+ <path d="M350 326 V338 H1050 V326" fill="none" stroke="#ebb431" stroke-width="3"/>
62
+ <text x="700" y="365" text-anchor="middle" fill="#8a6500" font-family="Inter, system-ui, sans-serif" font-size="14" font-weight="600">80 ms generation span</text>
63
+
64
+ <rect x="56" y="398" width="520" height="124" rx="12" fill="#edf7ff" stroke="#b8ddfa"/>
65
+ <rect x="56" y="398" width="6" height="124" rx="3" fill="#30a2ff"/>
66
+ <text x="86" y="431" fill="#1769aa" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="600">Mean ITL</text>
67
+ <text x="86" y="473" fill="#454545" font-family="Inter, system-ui, sans-serif" font-size="20">(40 ms + 40 ms) ÷ 2</text>
68
+ <text x="546" y="468" text-anchor="end" fill="#1769aa" font-family="Inter, system-ui, sans-serif" font-size="30" font-weight="600">40 ms</text>
69
+ <text x="86" y="500" fill="#64748b" font-family="Inter, system-ui, sans-serif" font-size="13">2 observed gaps between stream outputs</text>
70
+
71
+ <rect x="624" y="398" width="520" height="124" rx="12" fill="#fff9e8" stroke="#ead79c"/>
72
+ <rect x="624" y="398" width="6" height="124" rx="3" fill="#ebb431"/>
73
+ <text x="654" y="431" fill="#8a6500" font-family="Inter, system-ui, sans-serif" font-size="16" font-weight="600">TPOT</text>
74
+ <text x="654" y="473" fill="#454545" font-family="Inter, system-ui, sans-serif" font-size="20">(180 − 100) ms ÷ (5 − 1)</text>
75
+ <text x="1114" y="468" text-anchor="end" fill="#8a6500" font-family="Inter, system-ui, sans-serif" font-size="30" font-weight="600">20 ms/token</text>
76
+ <text x="654" y="500" fill="#64748b" font-family="Inter, system-ui, sans-serif" font-size="13">4 output tokens after the first</text>
77
+</svg>
docs/benchmarking/cli.md
+39
@@ -111,6 +111,45 @@ P99 ITL (ms): 8.39
111
==================================================
112
```
113
114
+#### Understanding the Latency Metrics
115
+
116
+`vllm bench serve` measures latency at the benchmark client:
117
+
118
+!!! note
119
+ Metric terminology is not standardized across benchmarking tools. When
120
+ comparing results, use the measurement points and formulas rather than the
121
+ metric names alone. This section explains how we refer to them in vLLM.
122
+
123
+- **Time to first token (TTFT)** is the time from sending a request to receiving
124
+ its first streamed output.
125
+- **Inter-token latency (ITL)** records the time between consecutive streamed
126
+ outputs. The reported ITL statistics aggregate these individual gaps across
127
+ all successful requests.
128
+- **Time per output token (TPOT)** is calculated once per request, excluding the
129
+ first token, and then aggregated across requests:
130
+
131
+ $$
132
+ \text{TPOT} =
133
+ \frac{\text{end-to-end latency} - \text{TTFT}}
134
+ {\text{number of output tokens} - 1}
135
+ $$
136
+
137
+With standard decoding, each streamed output usually contains one token, so ITL
138
+and TPOT are typically similar.
139
+
140
+With speculative decoding, one streamed output can contain multiple tokens,
141
+such as several accepted draft tokens within a single engine tstep. ITL records
142
+only the gaps between streamed outputs; it does not add zero-duration gaps for
143
+tokens in the same output. TPOT instead amortizes the request's decoding time
144
+over every output token.
145
+
146
+
147
+
148
+
149
+In this example, the benchmark observes two 40 ms ITL samples. The three tokens
150
+in the second streamed output do not create additional ITL samples, so mean ITL
151
+is 40 ms. TPOT is `(180 ms - 100 ms) / (5 - 1) = 20 ms/token`.
152
+
153
#### Results Visualization
154
155
The `--plot-timeline` and `--plot-dataset-stats` can be used to generate respectively the requests completion timeline and dataset prompt and output tokens statistics, which can be useful for debugging purpose or for deeper analysis.