@tduncan / Nvidia-Optimization-Stack-VLLM / commits / 7cfb97e337

[Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses (#48915)

Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com> Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Matthew Kotila committed Aug 20, 2026 at 15:42 UTC 7cfb97e33791a348cd5d7b622cca521d82d8399f

25 files changed +798 -76

docs/features/per_request_metrics.md

@@ -125,3 +125,12 @@ a single prompt's generation.

125 The `metrics` response field provides per-request values for a single request.

126 The `/metrics` Prometheus endpoint exposes server-level histograms (e.g.

127 `vllm:time_to_first_token_seconds`) that aggregate across all requests.

128 +

129 +## Speculative Decoding Acceptance

130 +

131 +When speculative decoding is enabled, per-request acceptance metrics

132 +(mean acceptance length and the accepted-draft-length distribution) can be

133 +returned via `--per-request-spec-decode-metrics`. They share this `metrics`

134 +object as `metrics.speculative_decoding`, and — like the timing fields — are

135 +reported only for single-sequence (`n == 1`) requests. See

136 +[Per-Request Acceptance Metrics](speculative_decoding/acceptance_metrics.md).

docs/features/speculative_decoding/README.md

@@ -19,6 +19,7 @@ vLLM supports a variety of methods of speculative decoding. Model-based methods

19 - [Custom Proposer Backend (Experimental)](#custom-proposer-backend-experimental)

20 - [Dynamic Speculative Decoding](dynamic_speculative_decoding.md)

21 - [Adaptive Verification](adaptive_verification.md)

22 +- [Per-Request Acceptance Metrics](acceptance_metrics.md)

24 ## Method Selection at a Glance

docs/features/speculative_decoding/acceptance_metrics.md new

+100

@@ -0,0 +1,100 @@

1 +# Per-Request Acceptance Metrics

2 +

3 +When speculative decoding is enabled, vLLM can report per-request acceptance

4 +metrics in the response, under `metrics.speculative_decoding`. This lets a

5 +client compute the mean acceptance length and the accepted-draft-length

6 +distribution for an individual request, as a complement to the server-aggregated

7 +spec-decode metrics exposed at `/metrics`.

8 +

9 +!!! warning "Experimental"

10 + `metrics.speculative_decoding` is experimental and its shape may change in a

11 + future release. Pin to a vLLM version if you depend on it.

12 +

13 +## Enabling

14 +

15 +Start the server with `--per-request-spec-decode-metrics` set to `summary` or

16 +`detailed` (default `none`):

17 +

18 +```bash

19 +vllm serve <target-model> \

20 + --speculative-config '{"method": "ngram", "num_speculative_tokens": 3, "prompt_lookup_min": 1, "prompt_lookup_max": 3}' \

21 + --per-request-spec-decode-metrics summary

22 +```

23 +

24 +| Level | Behavior |

25 +| --- | --- |

26 +| `none` (default) | No collection; responses are unchanged. |

27 +| `summary` | Acceptance metrics per request. |

28 +| `detailed` | `summary` plus ordered per-step arrays. |

29 +

30 +Collection is gated at the source: with `none`, nothing is accumulated.

31 +

32 +## Response Format

33 +

34 +Acceptance metrics share the top-level `metrics` object with the timing

35 +[per-request metrics](../per_request_metrics.md) — `metrics.speculative_decoding`

36 +sits alongside the timing fields. Like timing, they describe a single generation

37 +stream, so they are reported only for single-sequence requests and are `null`

38 +for `n > 1`.

39 +

40 +A `summary` response's `metrics` looks like:

41 +

42 +```json

43 +{

44 + "choices": [ ... ],

45 + "usage": { ... },

46 + "metrics": {

47 + "speculative_decoding": {

48 + "mean_acceptance_length": 1.2325581395348837,

49 + "draft_acceptance_rate": 0.07751937984496124,

50 + "acceptance_histogram": [39, 1, 0, 3],

51 + "num_spec_steps": 43,

52 + "num_accepted_draft_tokens": 10,

53 + "num_draft_tokens": 129,

54 + "num_spec_tokens": 3

55 + }

56 + }

57 +}

58 +```

59 +

60 +| Field | Description |

61 +| --- | --- |

62 +| `mean_acceptance_length` | Mean tokens emitted per verification step, including the bonus token: `1 + num_accepted_draft_tokens / num_spec_steps`. Ranges from `1.0` (nothing accepted) to `num_spec_tokens + 1`. |

63 +| `draft_acceptance_rate` | Fraction of proposed draft tokens accepted: `num_accepted_draft_tokens / num_draft_tokens`. |

64 +| `acceptance_histogram` | Dense list of length `num_spec_tokens + 1`; index `j` is the number of steps that accepted exactly `j` draft tokens. Excludes the always-accepted bonus token. |

65 +| `num_spec_steps` | Number of verification steps for this request (the sum of the histogram). |

66 +| `num_accepted_draft_tokens` | Total accepted draft tokens, excluding bonus tokens. |

67 +| `num_draft_tokens` | Total proposed draft tokens, after subtracting drafts invalidated by structured-output constraints. |

68 +| `num_spec_tokens` | Configured `num_speculative_tokens` (`k`), i.e. the maximum draft length per step. |

69 +

70 +With `detailed`, two ordered arrays are added, one entry per verification step:

71 +

72 +| Field | Description |

73 +| --- | --- |

74 +| `per_step_accepted` | Accepted draft count at each step. |

75 +| `per_step_drafted` | Proposed draft count at each step. Records the effective proposal length per step, so variable-length drafting (e.g. adaptive speculation) is represented without a schema change. |

76 +

77 +`metrics.speculative_decoding` is present whenever `--per-request-spec-decode-metrics`

78 +is `summary`/`detailed`, speculative decoding is enabled, and `n == 1` (with an

79 +all-zero histogram if the request drafted nothing). It is `null` otherwise.

80 +

81 +## Streaming

82 +

83 +In streaming responses, `metrics` (including `speculative_decoding`) rides the

84 +final usage chunk, which is only emitted when usage reporting is enabled — set

85 +`stream_options.include_usage: true` or start the server with

86 +`--enable-force-include-usage`.

87 +

88 +## Relationship to Prometheus metrics

89 +

90 +The per-request fields are the individual-request counterpart of the

91 +server-aggregated spec-decode counters at `/metrics`. Summed across the

92 +single-sequence requests that report them, they reconcile with the aggregate

93 +counters (which also count `n > 1` requests, so the totals match only for

94 +all-`n == 1` workloads):

95 +

96 +| Per-request field (summed) | Prometheus counter |

97 +| --- | --- |

98 +| `num_spec_steps` | `vllm:spec_decode_num_drafts_total` |

99 +| `num_draft_tokens` | `vllm:spec_decode_num_draft_tokens_total` |

100 +| `num_accepted_draft_tokens` | `vllm:spec_decode_num_accepted_tokens_total` |

rust/src/engine-core-client/src/protocol/output.rs

@@ -125,6 +125,11 @@ pub struct EngineCoreOutput {

125 pub mm_cache_miss_hashes: Option<Vec<String>>,

126 #[serde(default)]

127 pub new_sampling_mask: Option<OpaqueValue>,

128 + /// Per-request speculative-decoding acceptance metrics, set on the final

129 + /// output when `--per-request-spec-decode-metrics` is enabled. Opaque here;

130 + /// the Rust frontend does not yet surface it in responses.

131 + #[serde(default)]

132 + pub spec_decode_metrics: Option<OpaqueValue>,

133 }

134

135 impl EngineCoreOutput {

@@ -443,6 +448,7 @@ mod tests {

448 num_nans_in_logits: 0,

449 mm_cache_miss_hashes: None,

450 new_sampling_mask: None,

451 + spec_decode_metrics: None,

452 },

453 ],

454 scheduler_stats: None,

rust/src/engine-core-client/src/tests/client.rs

@@ -2743,6 +2743,7 @@ fn python_msgpack_fixtures_match_rust_encoding() {

2743 num_nans_in_logits: 0,

2744 mm_cache_miss_hashes: None,

2745 new_sampling_mask: None,

2746 + spec_decode_metrics: None,

2747 },

2748 ],

2749 scheduler_stats: None,

tests/entrypoints/openai/completion/test_completion_error.py

+109 -1

@@ -23,7 +23,7 @@ from vllm.renderers.hf import HfRenderer

23 from vllm.renderers.online_renderer import OnlineRenderer

24 from vllm.tokenizers.registry import cached_tokenizer_from_config

25 from vllm.v1.engine.async_llm import AsyncLLM

26 -from vllm.v1.metrics.stats import RequestStateStats

26 +from vllm.v1.metrics.stats import RequestSpecDecodeMetrics, RequestStateStats

28 MODEL_NAME = "openai-community/gpt2"

29 MODEL_NAME_SHORT = "gpt2"

@@ -196,6 +196,114 @@ def test_completion_per_request_metrics_suppressed_for_multiple_prompts():

196 assert response.metrics is None

197

198

199 +def _spec_decode_metrics() -> RequestSpecDecodeMetrics:

200 + # Two verify steps: accept 3 drafts, then 1 -> histogram [0, 1, 0, 1].

201 + m = RequestSpecDecodeMetrics.new(num_spec_tokens=3)

202 + m.observe(num_draft_tokens=3, num_accepted=3)

203 + m.observe(num_draft_tokens=3, num_accepted=1)

204 + return m

205 +

206 +

207 +def _make_spec_decode_request_output(

208 + num_seqs: int = 1, with_metrics: bool = True

209 +) -> RequestOutput:

210 + outputs = [

211 + CompletionOutput(

212 + index=i,

213 + text="Hello",

214 + token_ids=[100, 101],

215 + cumulative_logprob=None,

216 + logprobs=None,

217 + finish_reason="stop",

218 + spec_decode_metrics=_spec_decode_metrics() if with_metrics else None,

219 + )

220 + for i in range(num_seqs)

221 + ]

222 + return RequestOutput(

223 + request_id="test-id",

224 + prompt="Test prompt",

225 + prompt_token_ids=[1, 2, 3],

226 + prompt_logprobs=None,

227 + outputs=outputs,

228 + finished=True,

229 + metrics=None,

230 + )

231 +

232 +

233 +def _completion_response(serving, request, request_output):

234 + return serving.request_output_to_completion_response(

235 + [request_output],

236 + request,

237 + "cmpl-test-id",

238 + 0,

239 + MODEL_NAME,

240 + None,

241 + RequestResponseMetadata(request_id="cmpl-test-id"),

242 + )

243 +

244 +

245 +def test_completion_spec_decode_metrics_present_for_single_sequence():

246 + # Timing off, but the sequence carries acceptance metrics -> the metrics

247 + # object is created just to hold metrics.speculative_decoding.

248 + serving = _build_minimal_metrics_serving_completion(

249 + enable_per_request_metrics=False

250 + )

251 + response = _completion_response(

252 + serving,

253 + CompletionRequest(model=MODEL_NAME, prompt="Test prompt", max_tokens=10),

254 + _make_spec_decode_request_output(num_seqs=1),

255 + )

256 + assert response.metrics is not None

257 + assert response.metrics.time_to_first_token_ms is None # timing not requested

258 + spec = response.metrics.speculative_decoding

259 + assert spec is not None

260 + assert spec.acceptance_histogram == [0, 1, 0, 1] # dense, index j

261 + assert spec.num_spec_steps == 2

262 + assert spec.num_spec_tokens == 3

263 + assert spec.mean_acceptance_length == pytest.approx(3.0) # 1 + (3 + 1) / 2

264 +

265 +

266 +def test_completion_spec_decode_metrics_suppressed_for_n_gt_1():

267 + # Per-request metrics can't be attributed to one of the n sequences.

268 + serving = _build_minimal_metrics_serving_completion(

269 + enable_per_request_metrics=False

270 + )

271 + response = _completion_response(

272 + serving,

273 + CompletionRequest(model=MODEL_NAME, prompt="Test prompt", n=2, max_tokens=10),

274 + _make_spec_decode_request_output(num_seqs=2),

275 + )

276 + assert response.metrics is None

277 +

278 +

279 +def test_completion_spec_decode_metrics_absent_when_not_collected():

280 + # Flag off -> the sequence carries no acceptance metrics -> no metrics object.

281 + serving = _build_minimal_metrics_serving_completion(

282 + enable_per_request_metrics=False

283 + )

284 + response = _completion_response(

285 + serving,

286 + CompletionRequest(model=MODEL_NAME, prompt="Test prompt", max_tokens=10),

287 + _make_spec_decode_request_output(num_seqs=1, with_metrics=False),

288 + )

289 + assert response.metrics is None

290 +

291 +

292 +def test_completion_metrics_carries_both_timing_and_spec_decode():

293 + serving = _build_minimal_metrics_serving_completion(enable_per_request_metrics=True)

294 + request_output = _make_spec_decode_request_output(num_seqs=1)

295 + request_output.metrics = _PER_REQUEST_STATS # timing source

296 + response = _completion_response(

297 + serving,

298 + CompletionRequest(model=MODEL_NAME, prompt="Test prompt", max_tokens=10),

299 + request_output,

300 + )

301 + assert response.metrics is not None

302 + assert response.metrics.time_to_first_token_ms == pytest.approx(500.0)

303 + assert response.metrics.speculative_decoding is not None

304 + assert response.metrics.speculative_decoding.num_spec_steps == 2

305 +

306 +

307 @pytest.mark.asyncio

308 async def test_completion_error_non_stream():

309 """test finish_reason='error' returns 500 InternalServerError (non-streaming)"""

tests/test_config.py

+14

@@ -20,6 +20,7 @@ from vllm.config import (

20 CompilationConfig,

21 KernelConfig,

22 ModelConfig,

23 + ObservabilityConfig,

24 ParallelConfig,

25 PoolerConfig,

26 SchedulerConfig,

@@ -85,6 +86,19 @@ def test_kda_recoverssm_derivation_is_revalidated():

86 VllmConfig.validate_mamba_cached_kernel(config)

89 +def test_per_request_spec_decode_metrics_requires_spec_decode():

90 + # The flag only makes sense with speculative decoding configured; enabling

91 + # it without --speculative-config should fail fast rather than silently

92 + # produce no metrics.

93 + for level in ("summary", "detailed"):

94 + with pytest.raises(ValueError, match="speculative"):

95 + VllmConfig(

96 + observability_config=ObservabilityConfig(

97 + per_request_spec_decode_metrics=level

98 + )

99 + )

100 +

101 +

102 def test_compile_config_repr_succeeds():

103 # setup: VllmBackend mutates the config object

104 config = VllmConfig()

tests/v1/core/test_scheduler.py

+113 -9

@@ -1306,29 +1306,42 @@ def test_draft_slots_budgeted_per_scheduled_request(tmp_path, monkeypatch):

1306

1307 # Note - these test cases mirror some of those in test_rejection_sampler.py

1308 @pytest.mark.parametrize(

1309 - "spec_tokens,output_tokens,expected",

1309 + "spec_tokens,output_tokens,expected,expected_per_req",

1310 [

1311 - ([[1, 2, 3]], [[1, 2, 3, 4]], (1, 3, 3, [1, 1, 1])), # perfect match

1312 - ([[1, 2, 3]], [[1, 5]], (1, 3, 1, [1, 0, 0])), # early mismatch

1313 - ([[1, 2], [3]], [[1, 2, 5], [3, 4]], (2, 3, 3, [2, 1])), # multiple sequences

1314 - ([[1]], [[1, 2]], (1, 1, 1, [1])), # single token sequence

1315 - ([[]], [[5]], (0, 0, 0, [0])), # empty sequence

1311 + ([[1, 2, 3]], [[1, 2, 3, 4]], (1, 3, 3, [1, 1, 1]), [[0, 0, 0, 1]]), # perfect

1312 + ([[1, 2, 3]], [[1, 5]], (1, 3, 1, [1, 0, 0]), [[0, 1, 0, 0]]), # early mismatch

1313 + (

1314 + [[1, 2], [3]],

1315 + [[1, 2, 5], [3, 4]],

1316 + (2, 3, 3, [2, 1]),

1317 + [[0, 0, 1], [0, 1, 0]],

1318 + ), # multiple sequences

1319 + ([[1]], [[1, 2]], (1, 1, 1, [1]), [[0, 1]]), # single token sequence

1320 + ([[]], [[5]], (0, 0, 0, [0]), [[0, 0]]), # empty sequence -> empty accumulator

1321 (

1322 [[1, 2, 3], [4, 5, 6]],

1323 [[1, 2, 7], [4, 8]],

1324 (2, 6, 3, [2, 1, 0]),

1325 + [[0, 0, 1, 0], [0, 1, 0, 0]],

1326 ), # multiple mismatches

1327 ],

1328 )

1323 -def test_schedule_spec_decoding_stats(spec_tokens, output_tokens, expected):

1329 +def test_schedule_spec_decoding_stats(

1330 + spec_tokens, output_tokens, expected, expected_per_req

1331 +):

1332 """Test scheduling behavior with speculative decoding.

1333

1334 This test verifies that:

1335 1. Speculated tokens get scheduled correctly

1328 - 2. Spec decoding stats properly count number of draft and accepted tokens

1336 + 2. The aggregate SpecDecodingStats count draft and accepted tokens

1337 + 3. The per-request accumulator (enabled via per_request_spec_decode_metrics)

1338 + buckets the same acceptance by accepted draft count (j)

1339 """

1340 num_spec_tokens = max(1, max(len(t) for t in spec_tokens))

1331 - scheduler = create_scheduler(num_speculative_tokens=num_spec_tokens)

1341 + scheduler = create_scheduler(

1342 + num_speculative_tokens=num_spec_tokens,

1343 + per_request_spec_decode_metrics="summary",

1344 + )

1345 requests = create_requests(num_requests=len(spec_tokens), num_tokens=1)

1346 req_ids = []

1347 req_to_index = {}

@@ -1414,6 +1427,97 @@ def test_schedule_spec_decoding_stats(spec_tokens, output_tokens, expected):

1427 assert stats.num_accepted_tokens == expected[2]

1428 assert stats.num_accepted_tokens_per_pos == expected[3]

1429

1430 + # Per-request accumulator: the same acceptance, bucketed by accepted draft

1431 + # count (j) on each request rather than summed across the batch. The

1432 + # accumulator is created eagerly on add_request, so every request has one

1433 + # (an empty histogram when it drafted nothing).

1434 + for i, req_id in enumerate(req_ids):

1435 + payload = scheduler.requests[req_id].spec_decode_metrics.to_dict()

1436 + assert payload["acceptance_histogram"] == expected_per_req[i]

1437 + assert payload["num_draft_tokens"] == len(spec_tokens[i])

1438 + assert "per_step_accepted" not in payload # summary level

1439 +

1440 +

1441 +def _run_spec_verify_steps(scheduler, rounds, num_invalid_per_round=None):

1442 + """Drive prefill + one draft/verify step per round for a single request.

1443 +

1444 + ``rounds`` is a list of ``(spec_token_ids, output_token_ids)`` -- one verify

1445 + step each. ``num_invalid_per_round`` optionally injects the grammar-invalid

1446 + draft count structured-output decoding would set on the verify output.

1447 + Returns the request.

1448 + """

1449 + [req] = create_requests(num_requests=1, num_tokens=1)

1450 + scheduler.add_request(req)

1451 + rid = req.request_id

1452 + req_to_index = {rid: 0}

1453 +

1454 + def _mk_output(sampled):

1455 + return ModelRunnerOutput(

1456 + req_ids=[rid],

1457 + req_id_to_index=req_to_index,

1458 + sampled_token_ids=sampled,

1459 + logprobs=None,

1460 + prompt_logprobs_dict={},

1461 + pooler_output=[],

1462 + )

1463 +

1464 + # Prefill: sample one token.

1465 + scheduler.update_from_output(scheduler.schedule(), _mk_output([[0]]))

1466 + for i, (spec, out) in enumerate(rounds):

1467 + scheduler.update_draft_token_ids(DraftTokenIds([rid], [spec]))

1468 + output = scheduler.schedule()

1469 + if num_invalid_per_round is not None and num_invalid_per_round[i]:

1470 + output.num_invalid_spec_tokens = {rid: num_invalid_per_round[i]}

1471 + scheduler.update_from_output(output, _mk_output([out]))

1472 + return req

1473 +

1474 +

1475 +def test_per_request_spec_decode_detailed_records_per_step():

1476 + scheduler = create_scheduler(

1477 + num_speculative_tokens=3,

1478 + per_request_spec_decode_metrics="detailed",

1479 + )

1480 + # Three verify steps for one request; num_accepted = len(output) - 1, so the

1481 + # outputs below accept 3, 1, then 0 drafts across the steps.

1482 + req = _run_spec_verify_steps(

1483 + scheduler,

1484 + [

1485 + ([1, 2, 3], [1, 2, 3, 4]), # accept 3

1486 + ([5, 6, 7], [5, 8]), # accept 1

1487 + ([9, 10, 11], [12]), # accept 0

1488 + ],

1489 + )

1490 + payload = scheduler.requests[req.request_id].spec_decode_metrics.to_dict()

1491 + assert payload["per_step_accepted"] == [3, 1, 0]

1492 + assert payload["per_step_drafted"] == [3, 3, 3]

1493 + assert payload["num_spec_steps"] == 3

1494 +

1495 +

1496 +def test_per_request_spec_decode_subtracts_invalid_drafts():

1497 + # Grammar-invalidated drafts (num_invalid_spec_tokens, set by structured

1498 + # output) are excluded from the proposed count, mirroring the aggregate.

1499 + scheduler = create_scheduler(

1500 + num_speculative_tokens=3,

1501 + per_request_spec_decode_metrics="summary",

1502 + )

1503 + # One verify step: 3 drafted, 1 grammar-invalid, output accepts 2.

1504 + req = _run_spec_verify_steps(

1505 + scheduler,

1506 + [([1, 2, 3], [1, 2, 5])],

1507 + num_invalid_per_round=[1],

1508 + )

1509 + payload = scheduler.requests[req.request_id].spec_decode_metrics.to_dict()

1510 + assert payload["num_draft_tokens"] == 2 # 3 drafted - 1 invalid

1511 + assert payload["num_accepted_draft_tokens"] == 2 # len([1,2,5]) - 1

1512 + assert payload["acceptance_histogram"] == [0, 0, 1, 0]

1513 +

1514 +

1515 +def test_per_request_spec_decode_acceptance_disabled_by_default():

1516 + scheduler = create_scheduler(num_speculative_tokens=3)

1517 + assert scheduler.spec_decode_metrics_level == "none"

1518 + req = _run_spec_verify_steps(scheduler, [([1, 2, 3], [1, 2, 3, 4])])

1519 + assert scheduler.requests[req.request_id].spec_decode_metrics is None

1520 +

1521

1522 def test_spec_decoding_stats_empty_output():

1523 """Test that spec decoding stats handle empty output tokens gracefully.

tests/v1/core/utils.py

@@ -10,6 +10,7 @@ from vllm.config import (

10 ECTransferConfig,

11 KVTransferConfig,

12 ModelConfig,

13 + ObservabilityConfig,

14 ParallelConfig,

15 SchedulerConfig,

16 SpeculativeConfig,

@@ -71,6 +72,7 @@ def create_scheduler(

72 ec_role: str | None = None,

73 use_v2_model_runner: bool | None = None,

74 kv_cache_spec: KVCacheSpec | None = None,

75 + per_request_spec_decode_metrics: str = "none",

76 ) -> Scheduler | AsyncScheduler:

77 """Create scheduler under test.

@@ -175,6 +177,9 @@ def create_scheduler(

177 kv_transfer_config=kv_transfer_config,

178 speculative_config=speculative_config,

179 ec_transfer_config=ec_transfer_config,

180 + observability_config=ObservabilityConfig(

181 + per_request_spec_decode_metrics=per_request_spec_decode_metrics,

182 + ),

183 )

184 if kv_cache_spec is None:

185 kv_cache_spec = FullAttentionSpec(

tests/v1/spec_decode/test_request_acceptance.py new

+150

@@ -0,0 +1,150 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +"""Unit tests for per-request speculative-decoding metrics accumulation.

4 +

5 +These cover the pure histogram/per-step math (no GPU / no model): the engine-core

6 +accumulator ``RequestSpecDecodeMetrics`` and its ``to_dict`` payload surfaced in the

7 +response as ``metrics.speculative_decoding``.

8 +"""

9 +

10 +import msgspec

11 +import pytest

12 +

13 +from vllm.outputs import CompletionOutput

14 +from vllm.v1.engine import EngineCoreOutput

15 +from vllm.v1.metrics.stats import RequestSpecDecodeMetrics

16 +

17 +

18 +def _metrics(pairs, num_spec_tokens=3, detailed=False):

19 + s = RequestSpecDecodeMetrics.new(num_spec_tokens)

20 + for k, j in pairs:

21 + s.observe(num_draft_tokens=k, num_accepted=j, detailed=detailed)

22 + return s

23 +

24 +

25 +def test_new_allocates_dense_histogram_of_k_plus_one():

26 + s = RequestSpecDecodeMetrics.new(num_spec_tokens=3)

27 + assert s.num_spec_tokens == 3

28 + assert s.histogram == [0, 0, 0, 0]

29 + assert s.num_draft_tokens == 0

30 + assert s.per_step_accepted == []

31 +

32 +

33 +def test_observe_buckets_by_accepted_draft_count():

34 + s = _metrics([(3, 0), (3, 3), (3, 2), (3, 3), (3, 1)])

35 + # j=0 ->1, j=1 ->1, j=2 ->1, j=3 ->2

36 + assert s.histogram == [1, 1, 1, 2]

37 + assert s.num_draft_tokens == 15

38 + # summary level does not record the ordered per-step arrays

39 + assert s.per_step_accepted == []

40 + assert s.per_step_drafted == []

41 +

42 +

43 +def test_observe_detailed_records_ordered_per_step_arrays():

44 + # Distinct step count (3), max draft length (k=4), and per-step drafted

45 + # counts (4, 2, 4) so no accidental "everything is 3" pattern is implied.

46 + s = _metrics([(4, 3), (2, 2), (4, 0)], num_spec_tokens=4, detailed=True)

47 + assert s.per_step_accepted == [3, 2, 0]

48 + assert s.per_step_drafted == [4, 2, 4]

49 + # histogram (indexed by accepted j, length k+1=5) is still maintained

50 + assert s.histogram == [1, 0, 1, 1, 0]

51 +

52 +

53 +def test_to_dict_summary_omits_per_step_arrays():

54 + d = _metrics([(3, 0), (3, 3), (3, 2), (3, 3), (3, 1)]).to_dict()

55 + assert d == {

56 + "mean_acceptance_length": pytest.approx(1 + 9 / 5), # j+1

57 + "draft_acceptance_rate": pytest.approx(9 / 15),

58 + "acceptance_histogram": [1, 1, 1, 2], # dense, index j = step count

59 + "num_spec_steps": 5,

60 + "num_accepted_draft_tokens": 9,

61 + "num_draft_tokens": 15,

62 + "num_spec_tokens": 3,

63 + }

64 +

65 +

66 +def test_to_dict_detailed_appends_per_step_arrays():

67 + d = _metrics([(3, 3), (3, 2), (3, 0)], detailed=True).to_dict()

68 + assert d["per_step_accepted"] == [3, 2, 0]

69 + assert d["per_step_drafted"] == [3, 3, 3]

70 + # summary fields still present in detailed mode

71 + assert d["num_spec_steps"] == 3

72 + assert d["num_accepted_draft_tokens"] == 5

73 +

74 +

75 +def test_to_dict_histogram_is_dense_list_indexed_by_j():

76 + # length k+1, index j holds the step count that accepted exactly j drafts

77 + d = _metrics([(3, 0), (3, 0), (3, 3)]).to_dict()

78 + assert d["acceptance_histogram"] == [2, 0, 0, 1]

79 +

80 +

81 +def test_all_rejected_gives_mean_one_and_rate_zero():

82 + d = _metrics([(2, 0), (2, 0), (2, 0), (2, 0)], num_spec_tokens=2).to_dict()

83 + assert d["num_spec_steps"] == 4

84 + assert d["num_accepted_draft_tokens"] == 0

85 + assert d["acceptance_histogram"] == [4, 0, 0]

86 + assert d["mean_acceptance_length"] == pytest.approx(1.0)

87 + assert d["draft_acceptance_rate"] == pytest.approx(0.0)

88 +

89 +

90 +def test_empty_metrics_do_not_divide_by_zero():

91 + d = RequestSpecDecodeMetrics.new(3).to_dict()

92 + assert d["num_spec_steps"] == 0

93 + assert d["num_draft_tokens"] == 0

94 + assert d["draft_acceptance_rate"] == 0.0

95 + assert d["mean_acceptance_length"] == 1.0

96 + assert "per_step_accepted" not in d

97 +

98 +

99 +def test_observe_records_proposed_and_accepted_independently():

100 + # observe() takes proposed and accepted as independent inputs: the histogram

101 + # is keyed by accepted, num_draft_tokens sums the proposed as given. (The

102 + # grammar-invalidated-draft subtraction happens in the scheduler before

103 + # observe() -- see test_per_request_spec_decode_subtracts_invalid_drafts.)

104 + s = RequestSpecDecodeMetrics.new(num_spec_tokens=3)

105 + s.observe(num_draft_tokens=2, num_accepted=1)

106 + s.observe(num_draft_tokens=3, num_accepted=1)

107 + d = s.to_dict()

108 + assert d["acceptance_histogram"] == [0, 2, 0, 0] # both steps accepted 1

109 + assert d["num_draft_tokens"] == 5 # proposed summed independently: 2 + 3

110 +

111 +

112 +def test_engine_core_output_round_trips_spec_decode_metrics():

113 + # The accumulator rides EngineCoreOutput (msgspec, array_like) to the

114 + # frontend; verify it serializes (incl. per-step arrays) and is omitted

115 + # when absent. ``new_token_ids`` is EngineCoreOutput's required "tokens

116 + # generated this step" field -- a dummy value here since we only exercise

117 + # spec_decode_metrics.

118 + metrics = _metrics([(3, 0), (3, 3), (3, 2)], detailed=True)

119 + out = EngineCoreOutput(

120 + request_id="r1", new_token_ids=[1, 2], spec_decode_metrics=metrics

121 + )

122 + decoder = msgspec.msgpack.Decoder(EngineCoreOutput)

123 + decoded = decoder.decode(msgspec.msgpack.encode(out))

124 + assert decoded.spec_decode_metrics.histogram == [1, 0, 1, 1]

125 + assert decoded.spec_decode_metrics.num_draft_tokens == 9

126 + assert decoded.spec_decode_metrics.per_step_accepted == [0, 3, 2]

127 +

128 + without = EngineCoreOutput(request_id="r2", new_token_ids=[1])

129 + decoded_without = decoder.decode(msgspec.msgpack.encode(without))

130 + assert decoded_without.spec_decode_metrics is None

131 +

132 +

133 +def _completion_output(**kwargs):

134 + return CompletionOutput(

135 + index=0,

136 + text="",

137 + token_ids=[],

138 + cumulative_logprob=None,

139 + logprobs=None,

140 + **kwargs,

141 + )

142 +

143 +

144 +def test_completion_output_carries_spec_decode_metrics():

145 + # Metrics are per output sequence, so they ride the CompletionOutput

146 + # (=> choices[i]), not the request-level RequestOutput.

147 + metrics = _metrics([(3, 3), (3, 1)])

148 + out = _completion_output(spec_decode_metrics=metrics)

149 + assert out.spec_decode_metrics is metrics

150 + assert _completion_output().spec_decode_metrics is None

vllm/config/observability.py

+12

@@ -45,6 +45,18 @@ class ObservabilityConfig:

45 Note that collecting detailed timing information for each request can be

46 expensive."""

48 + per_request_spec_decode_metrics: Literal["none", "summary", "detailed"] = "none"

49 + """Include per-request speculative-decoding acceptance metrics in the

50 + response under `metrics.speculative_decoding`. `none` disables; `summary` adds mean

51 + acceptance length, draft acceptance rate, and the step-by-draft-length

52 + histogram; `detailed` additionally records the ordered per-step

53 + accepted/proposed arrays (one entry per verify step). Only reported for

54 + single-sequence requests (`n == 1`), mirroring the timing metrics. No effect

55 + unless speculative decoding is enabled. Independent of `--disable-log-stats`.

56 + This is the per-request response-body counterpart of the aggregate

57 + `vllm:spec_decode_*` Prometheus metrics. The response field is experimental

58 + and its shape may change in a future release."""

59 +

60 kv_cache_metrics: bool = False

61 """Enable KV cache residency metrics (lifetime, idle time, reuse gaps).

62 Uses sampling to minimize overhead.

vllm/config/vllm.py

@@ -1331,6 +1331,15 @@ class VllmConfig:

1331 )

1332 self.model_config.disable_cascade_attn = True

1333

1334 + if (

1335 + self.observability_config.per_request_spec_decode_metrics != "none"

1336 + and self.speculative_config is None

1337 + ):

1338 + raise ValueError(

1339 + "--per-request-spec-decode-metrics requires speculative decoding "

1340 + "to be enabled (via --speculative-config)."

1341 + )

1342 +

1343 if (

1344 self.model_config is not None

1345 and self.model_config.multimodal_config is not None

vllm/engine/arg_utils.py

@@ -656,6 +656,9 @@ class EngineArgs:

656 collect_detailed_traces: list[DetailedTraceModules] | None = (

657 ObservabilityConfig.collect_detailed_traces

658 )

659 + per_request_spec_decode_metrics: Literal["none", "summary", "detailed"] = (

660 + ObservabilityConfig.per_request_spec_decode_metrics

661 + )

662 kv_cache_metrics: bool = ObservabilityConfig.kv_cache_metrics

663 kv_cache_metrics_sample: float = get_field(

664 ObservabilityConfig, "kv_cache_metrics_sample"

@@ -1476,6 +1479,10 @@ class EngineArgs:

1479 "--collect-detailed-traces",

1480 **observability_kwargs["collect_detailed_traces"],

1481 )

1482 + observability_group.add_argument(

1483 + "--per-request-spec-decode-metrics",

1484 + **observability_kwargs["per_request_spec_decode_metrics"],

1485 + )

1486 observability_group.add_argument(

1487 "--kv-cache-metrics", **observability_kwargs["kv_cache_metrics"]

1488 )

@@ -1934,6 +1941,7 @@ class EngineArgs:

1941 show_hidden_metrics_for_version=self.show_hidden_metrics_for_version,

1942 otlp_traces_endpoint=self.otlp_traces_endpoint,

1943 collect_detailed_traces=self.collect_detailed_traces,

1944 + per_request_spec_decode_metrics=self.per_request_spec_decode_metrics,

1945 kv_cache_metrics=self.kv_cache_metrics,

1946 kv_cache_metrics_sample=self.kv_cache_metrics_sample,

1947 cudagraph_metrics=self.cudagraph_metrics,

vllm/entrypoints/generate/base/serving.py

+26 -5

@@ -5,7 +5,7 @@ import time

5 from collections.abc import Awaitable, Mapping

6 from dataclasses import dataclass, field

7 from http import HTTPStatus

8 -from typing import ClassVar, Generic, TypeVar

8 +from typing import TYPE_CHECKING, ClassVar, Generic, TypeVar

10 from fastapi import Request

11 from pydantic import ConfigDict

@@ -18,7 +18,8 @@ from vllm.entrypoints.openai.completion.protocol import CompletionRequest

18 from vllm.entrypoints.openai.engine.protocol import (

19 ErrorResponse,

20 GenerationError,

21 - PerRequestTimingMetrics,

21 + PerRequestMetrics,

22 + SpeculativeDecodingMetrics,

23 )

24 from vllm.entrypoints.openai.models.serving import OpenAIServingModels

25 from vllm.entrypoints.openai.responses.protocol import ResponsesRequest

@@ -37,6 +38,9 @@ from vllm.tracing import (

38 )

39 from vllm.v1.metrics.stats import RequestStateStats

41 +if TYPE_CHECKING:

42 + from vllm.outputs import RequestOutput

43 +

44 logger = init_logger(__name__)

46 RequestT = TypeVar("RequestT", bound=AnyRequest)

@@ -48,7 +52,7 @@ PRIORITY_HEADER = "X-Vllm-Priority"

52 def build_per_request_timing_metrics(

53 metrics: RequestStateStats | None,

54 num_generation_tokens: int,

51 -) -> PerRequestTimingMetrics:

55 +) -> PerRequestMetrics:

56 """Build per-request timing metrics from ``RequestStateStats``.

58 ``generation_time_ms`` is the decode interval only (first output token to

@@ -60,7 +64,7 @@ def build_per_request_timing_metrics(

64 unavailable.

65 """

66 if metrics is None:

63 - return PerRequestTimingMetrics()

67 + return PerRequestMetrics()

69 queued_ts = metrics.queued_ts

70 scheduled_ts = metrics.scheduled_ts

@@ -91,7 +95,7 @@ def build_per_request_timing_metrics(

95 if inference_time_ms > 0:

96 tokens_per_second = num_generation_tokens / inference_time_ms * 1000

94 - return PerRequestTimingMetrics(

98 + return PerRequestMetrics(

99 time_to_first_token_ms=time_to_first_token_ms,

100 generation_time_ms=generation_time_ms,

101 queue_time_ms=queue_time_ms,

@@ -100,6 +104,23 @@ def build_per_request_timing_metrics(

104 )

105

106

107 +def build_spec_decoding_metrics(

108 + final_res: "RequestOutput | None",

109 +) -> SpeculativeDecodingMetrics | None:

110 + """Build per-request spec-decode acceptance metrics from the single output

111 + sequence, or ``None`` when unavailable (metrics disabled, or no sequence

112 + yet).

113 +

114 + Only meaningful for single-sequence requests; callers suppress it for n>1.

115 + """

116 + if final_res is None or not final_res.outputs:

117 + return None

118 + metrics = final_res.outputs[0].spec_decode_metrics

119 + if metrics is None:

120 + return None

121 + return SpeculativeDecodingMetrics(**metrics.to_dict())

122 +

123 +

124 @dataclass(kw_only=True)

125 class ServeContext(Generic[RequestT]):

126 request: RequestT

vllm/entrypoints/openai/chat_completion/protocol.py

+3 -3

@@ -29,7 +29,7 @@ from vllm.entrypoints.openai.engine.protocol import (

29 FunctionCall,

30 FunctionDefinition,

31 OpenAIBaseModel,

32 - PerRequestTimingMetrics,

32 + PerRequestMetrics,

33 StopParam,

34 StreamOptions,

35 ToolCall,

@@ -146,7 +146,7 @@ class ChatCompletionResponse(OpenAIBaseModel):

146 ec_transfer_params: dict[str, Any] | None = Field(

147 default=None, description="ECTransfer parameters."

148 )

149 - metrics: PerRequestTimingMetrics | None = None

149 + metrics: PerRequestMetrics | None = None

150

151

152 class ChatCompletionResponseStreamChoice(OpenAIBaseModel):

@@ -178,7 +178,7 @@ class ChatCompletionStreamResponse(OpenAIBaseModel):

178 # Rendered prompt text from chat templating (only set when

179 # ``return_prompt_text=True`` on the request); only sent on the first chunk.

180 prompt_text: str | None = None

181 - metrics: PerRequestTimingMetrics | None = None

181 + metrics: PerRequestMetrics | None = None

182

183

184 class ChatCompletionToolsParam(OpenAIBaseModel):

vllm/entrypoints/openai/chat_completion/serving.py

+31 -22

@@ -20,6 +20,7 @@ from vllm.entrypoints.generate.base.serving import (

20 GenerateBaseServing,

21 GenerationError,

22 build_per_request_timing_metrics,

23 + build_spec_decoding_metrics,

24 clamp_prompt_logprobs,

25 format_token_id_placeholder,

26 )

@@ -40,7 +41,7 @@ from vllm.entrypoints.openai.engine.protocol import (

41 DeltaMessage,

42 ErrorResponse,

43 FunctionCall,

43 - PerRequestTimingMetrics,

44 + PerRequestMetrics,

45 PromptTokenUsageInfo,

46 RequestResponseMetadata,

47 ToolCall,

@@ -819,16 +820,21 @@ class OpenAIServingChat(GenerateBaseServing):

820 # only emitted when usage reporting is enabled (i.e.

821 # ``stream_options.include_usage=true`` or

822 # ``--enable-force-include-usage``).

822 - stream_per_request_metrics: PerRequestTimingMetrics | None = None

823 - if (

824 - self.enable_per_request_metrics

825 - # See note in chat_completion_full_generator: suppress for n>1.

826 - and (request.n or 1) == 1

827 - ):

828 - last_metrics = last_res.metrics if last_res is not None else None

829 - stream_per_request_metrics = build_per_request_timing_metrics(

830 - last_metrics, completion_tokens

831 - )

823 + stream_per_request_metrics: PerRequestMetrics | None = None

824 + # See note in chat_completion_full_generator: suppress for n>1.

825 + if (request.n or 1) == 1:

826 + if self.enable_per_request_metrics:

827 + last_metrics = (

828 + last_res.metrics if last_res is not None else None

829 + )

830 + stream_per_request_metrics = build_per_request_timing_metrics(

831 + last_metrics, completion_tokens

832 + )

833 + spec_stats = build_spec_decoding_metrics(last_res)

834 + if spec_stats is not None:

835 + if stream_per_request_metrics is None:

836 + stream_per_request_metrics = PerRequestMetrics()

837 + stream_per_request_metrics.speculative_decoding = spec_stats

838

839 final_usage_chunk = ChatCompletionStreamResponse(

840 id=request_id,

@@ -1118,17 +1124,20 @@ class OpenAIServingChat(GenerateBaseServing):

1124

1125 request_metadata.final_usage_info = usage

1126

1121 - per_request_metrics: PerRequestTimingMetrics | None = None

1122 - if (

1123 - self.enable_per_request_metrics

1124 - # Timing metrics describe a single generation stream. For n>1 the

1125 - # returned stats belong to only one of the n sequences, so they

1126 - # cannot be accurately attributed to the request; suppress instead.

1127 - and (request.n or 1) == 1

1128 - ):

1129 - per_request_metrics = build_per_request_timing_metrics(

1130 - final_res.metrics, num_generated_tokens

1131 - )

1127 + per_request_metrics: PerRequestMetrics | None = None

1128 + # Per-request metrics (timing + spec-decode acceptance) describe a single

1129 + # generation stream. For n>1 the stats belong to only one of the n

1130 + # sequences, so they cannot be attributed to the request; suppress.

1131 + if (request.n or 1) == 1:

1132 + if self.enable_per_request_metrics:

1133 + per_request_metrics = build_per_request_timing_metrics(

1134 + final_res.metrics, num_generated_tokens

1135 + )

1136 + spec_stats = build_spec_decoding_metrics(final_res)

1137 + if spec_stats is not None:

1138 + if per_request_metrics is None:

1139 + per_request_metrics = PerRequestMetrics()

1140 + per_request_metrics.speculative_decoding = spec_stats

1141

1142 # ``final_res.prompt`` is the rendered chat-templated prompt text

1143 prompt_text = final_res.prompt if request.return_prompt_text else None

vllm/entrypoints/openai/completion/protocol.py

+3 -3

@@ -13,7 +13,7 @@ from vllm.config import ModelConfig

13 from vllm.entrypoints.openai.engine.protocol import (

14 AnyResponseFormat,

15 OpenAIBaseModel,

16 - PerRequestTimingMetrics,

16 + PerRequestMetrics,

17 StopParam,

18 StreamOptions,

19 UsageInfo,

@@ -660,7 +660,7 @@ class CompletionResponse(OpenAIBaseModel):

660 ec_transfer_params: dict[str, Any] | None = Field(

661 default=None, description="ECTransfer parameters."

662 )

663 - metrics: PerRequestTimingMetrics | None = None

663 + metrics: PerRequestMetrics | None = None

664

665

666 class CompletionResponseStreamChoice(OpenAIBaseModel):

@@ -692,4 +692,4 @@ class CompletionStreamResponse(OpenAIBaseModel):

692 # Set only on the final chunk of a stream to mirror non-streaming responses

693 # without the per-chunk serialization overhead.

694 system_fingerprint: str | None = None

695 - metrics: PerRequestTimingMetrics | None = None

695 + metrics: PerRequestMetrics | None = None

vllm/entrypoints/openai/completion/serving.py

+36 -28

@@ -14,6 +14,7 @@ from vllm.entrypoints.generate.base.serving import (

14 GenerateBaseServing,

15 GenerationError,

16 build_per_request_timing_metrics,

17 + build_spec_decoding_metrics,

18 clamp_prompt_logprobs,

19 format_token_id_placeholder,

20 )

@@ -27,7 +28,7 @@ from vllm.entrypoints.openai.completion.protocol import (

28 )

29 from vllm.entrypoints.openai.engine.protocol import (

30 ErrorResponse,

30 - PerRequestTimingMetrics,

31 + PerRequestMetrics,

32 PromptTokenUsageInfo,

33 RequestResponseMetadata,

34 UsageInfo,

@@ -460,18 +461,22 @@ class OpenAIServingCompletion(GenerateBaseServing):

461 # only emitted when usage reporting is enabled (i.e.

462 # ``stream_options.include_usage=true`` or

463 # ``--enable-force-include-usage``).

463 - stream_per_request_metrics: PerRequestTimingMetrics | None = None

464 - if (

465 - self.enable_per_request_metrics

466 - # See note in request_output_to_completion_response: suppress

467 - # when not attributable to one stream (multi-prompt or n>1).

468 - and num_prompts == 1

469 - and (request.n or 1) == 1

470 - ):

471 - last_metrics = last_res.metrics if last_res is not None else None

472 - stream_per_request_metrics = build_per_request_timing_metrics(

473 - last_metrics, total_completion_tokens

474 - )

464 + stream_per_request_metrics: PerRequestMetrics | None = None

465 + # See note in request_output_to_completion_response: suppress when

466 + # not attributable to one stream (multi-prompt or n>1).

467 + if num_prompts == 1 and (request.n or 1) == 1:

468 + if self.enable_per_request_metrics:

469 + last_metrics = (

470 + last_res.metrics if last_res is not None else None

471 + )

472 + stream_per_request_metrics = build_per_request_timing_metrics(

473 + last_metrics, total_completion_tokens

474 + )

475 + spec_stats = build_spec_decoding_metrics(last_res)

476 + if spec_stats is not None:

477 + if stream_per_request_metrics is None:

478 + stream_per_request_metrics = PerRequestMetrics()

479 + stream_per_request_metrics.speculative_decoding = spec_stats

480

481 final_usage_chunk = CompletionStreamResponse(

482 id=request_id,

@@ -612,21 +617,24 @@ class OpenAIServingCompletion(GenerateBaseServing):

617

618 request_metadata.final_usage_info = usage

619

615 - per_request_metrics: PerRequestTimingMetrics | None = None

616 - if (

617 - self.enable_per_request_metrics

618 - # Metrics describe a single generation stream, so suppress them when

619 - # they cannot be attributed to one: multiple prompts (timestamps

620 - # span prompts) or n>1 (stats belong to one of the n sequences).

621 - and len(final_res_batch) == 1

622 - and (request.n or 1) == 1

623 - ):

624 - last_metrics = (

625 - last_final_res.metrics if last_final_res is not None else None

626 - )

627 - per_request_metrics = build_per_request_timing_metrics(

628 - last_metrics, num_generated_tokens

629 - )

620 + per_request_metrics: PerRequestMetrics | None = None

621 + # Per-request metrics (timing + spec-decode acceptance) describe a single

622 + # generation stream, so suppress them when they cannot be attributed to

623 + # one: multiple prompts (timestamps span prompts) or n>1 (stats belong to

624 + # one of the n sequences).

625 + if len(final_res_batch) == 1 and (request.n or 1) == 1:

626 + if self.enable_per_request_metrics:

627 + last_metrics = (

628 + last_final_res.metrics if last_final_res is not None else None

629 + )

630 + per_request_metrics = build_per_request_timing_metrics(

631 + last_metrics, num_generated_tokens

632 + )

633 + spec_stats = build_spec_decoding_metrics(last_final_res)

634 + if spec_stats is not None:

635 + if per_request_metrics is None:

636 + per_request_metrics = PerRequestMetrics()

637 + per_request_metrics.speculative_decoding = spec_stats

638

639 if final_res_batch:

640 kv_transfer_params = final_res_batch[0].kv_transfer_params

vllm/entrypoints/openai/engine/protocol.py

+25 -1

@@ -129,12 +129,36 @@ class UsageInfo(OpenAIBaseModel):

129 completion_tokens_details: CompletionTokenUsageInfo | None = None

130

131

132 -class PerRequestTimingMetrics(OpenAIBaseModel):

132 +class SpeculativeDecodingMetrics(OpenAIBaseModel):

133 + """Per-request speculative-decoding acceptance metrics.

134 +

135 + Experimental, subject to change. Only populated for single-sequence requests

136 + (`n == 1`); `null` for `n > 1`, mirroring the timing metrics.

137 + """

138 +

139 + mean_acceptance_length: float

140 + draft_acceptance_rate: float

141 + # Dense histogram: index j holds the number of verify steps that accepted

142 + # exactly j draft tokens (length num_spec_tokens + 1). Excludes the

143 + # always-accepted bonus token.

144 + acceptance_histogram: list[int]

145 + num_spec_steps: int

146 + num_accepted_draft_tokens: int

147 + num_draft_tokens: int

148 + num_spec_tokens: int

149 + # Ordered per-verify-step arrays; populated only at the `detailed` level.

150 + per_step_accepted: list[int] | None = None

151 + per_step_drafted: list[int] | None = None

152 +

153 +

154 +class PerRequestMetrics(OpenAIBaseModel):

155 time_to_first_token_ms: float | None = None

156 generation_time_ms: float | None = None

157 queue_time_ms: float | None = None

158 mean_itl_ms: float | None = None

159 tokens_per_second: float | None = None

160 + # Experimental, subject to change.

161 + speculative_decoding: SpeculativeDecodingMetrics | None = None

162

163

164 class RequestResponseMetadata(BaseModel):

vllm/outputs.py

+7 -1

@@ -13,7 +13,7 @@ from typing_extensions import TypeVar

13 from vllm.logger import init_logger

14 from vllm.logprobs import PromptLogprobs, SampleLogprobs

15 from vllm.lora.request import LoRARequest

16 -from vllm.v1.metrics.stats import RequestStateStats

16 +from vllm.v1.metrics.stats import RequestSpecDecodeMetrics, RequestStateStats

18 logger = init_logger(__name__)

@@ -48,6 +48,11 @@ class CompletionOutput:

48 to stop, None if the completion finished for some other reason

49 including encountering the EOS token.

50 lora_request: The LoRA request that was used to generate the output.

51 + spec_decode_metrics: Per-sequence speculative-decoding acceptance metrics,

52 + populated on finish when speculative decoding ran and

53 + ``--per-request-spec-decode-metrics`` is enabled; None otherwise.

54 + Surfaced in the response as ``metrics.speculative_decoding`` for

55 + single-sequence (``n == 1``) requests.

56 """

58 index: int

@@ -60,6 +65,7 @@ class CompletionOutput:

65 stop_reason: int | str | None = None

66 lora_request: LoRARequest | None = None

67 sampling_mask: SamplingMask | None = None

68 + spec_decode_metrics: RequestSpecDecodeMetrics | None = None

70 def finished(self) -> bool:

71 return self.finish_reason is not None

vllm/v1/core/sched/scheduler.py

+31 -1

@@ -55,7 +55,11 @@ from vllm.v1.core.sched.utils import check_stop, remove_all

55 from vllm.v1.engine import EngineCoreEventType, EngineCoreOutput, EngineCoreOutputs

56 from vllm.v1.kv_cache_interface import KVCacheConfig

57 from vllm.v1.metrics.perf import ModelMetrics, PerfStats

58 -from vllm.v1.metrics.stats import PrefixCacheStats, SchedulerStats

58 +from vllm.v1.metrics.stats import (

59 + PrefixCacheStats,

60 + RequestSpecDecodeMetrics,

61 + SchedulerStats,

62 +)

63 from vllm.v1.outputs import DraftTokenIds, KVConnectorOutput, ModelRunnerOutput

64 from vllm.v1.request import Request, RequestStatus, StreamingUpdate

65 from vllm.v1.spec_decode.dynamic.utils import build_dynamic_sd_schedule_lookup

@@ -88,6 +92,9 @@ class Scheduler(SchedulerInterface):

92 self.parallel_config = vllm_config.parallel_config

93 self.log_stats = log_stats

94 self.observability_config = vllm_config.observability_config

95 + self.spec_decode_metrics_level = (

96 + self.observability_config.per_request_spec_decode_metrics

97 + )

98 self.kv_metrics_collector: KVCacheMetricsCollector | None = None

99 if self.observability_config.kv_cache_metrics:

100 self.kv_metrics_collector = KVCacheMetricsCollector(

@@ -1857,6 +1864,20 @@ class Scheduler(SchedulerInterface):

1864 num_invalid_spec_tokens=scheduler_output.num_invalid_spec_tokens,

1865 request_id=req_id,

1866 )

1867 + if request.spec_decode_metrics is not None:

1868 + # Exclude grammar-invalidated drafts from the proposed

1869 + # count, mirroring make_spec_decoding_stats; the accepted

1870 + # bucket (j) is unaffected.

1871 + adj_draft_tokens = num_draft_tokens

1872 + if scheduler_output.num_invalid_spec_tokens:

1873 + adj_draft_tokens -= (

1874 + scheduler_output.num_invalid_spec_tokens.get(req_id, 0)

1875 + )

1876 + request.spec_decode_metrics.observe(

1877 + num_draft_tokens=adj_draft_tokens,

1878 + num_accepted=num_accepted,

1879 + detailed=self.spec_decode_metrics_level == "detailed",

1880 + )

1881

1882 # Free encoder inputs only after the step has actually executed.

1883 if request.has_encoder_inputs:

@@ -2020,6 +2041,11 @@ class Scheduler(SchedulerInterface):

2041 stop_reason=request.stop_reason,

2042 events=request.take_events(),

2043 prefill_stats=prefill_stats,

2044 + spec_decode_metrics=(

2045 + request.spec_decode_metrics

2046 + if finish_reason is not None

2047 + else None

2048 + ),

2049 kv_transfer_params=kv_transfer_params,

2050 ec_transfer_params=ec_transfer_params,

2051 trace_headers=request.trace_headers,

@@ -2321,6 +2347,10 @@ class Scheduler(SchedulerInterface):

2347 request.streaming_queue = deque()

2348 self._enqueue_waiting_request(request)

2349 self.requests[request.request_id] = request

2350 + if self.spec_decode_metrics_level != "none":

2351 + request.spec_decode_metrics = RequestSpecDecodeMetrics.new(

2352 + self.num_spec_tokens

2353 + )

2354 if self.connector is not None:

2355 self.connector.on_new_request(request)

2356 if self.log_stats:

vllm/v1/engine/__init__.py

+9 -1

@@ -16,7 +16,11 @@ from vllm.lora.request import LoRARequest

16 from vllm.multimodal.inputs import MultiModalFeatureSpec

17 from vllm.pooling_params import PoolingParams

18 from vllm.sampling_params import SamplingParams

19 -from vllm.v1.metrics.stats import PrefillStats, SchedulerStats

19 +from vllm.v1.metrics.stats import (

20 + PrefillStats,

21 + RequestSpecDecodeMetrics,

22 + SchedulerStats,

23 +)

24 from vllm.v1.outputs import LogprobsLists, LogprobsTensors, SamplingMaskLists

25 from vllm.v1.serial_utils import UtilityResult

@@ -225,6 +229,10 @@ class EngineCoreOutput(

229

230 new_sampling_mask: SamplingMaskLists | None = None

231

232 + # Per-request spec-decode acceptance; attached only on the final output.

233 + # Appended last so `array_like` positional serialization stays compatible.

234 + spec_decode_metrics: RequestSpecDecodeMetrics | None = None

235 +

236 @property

237 def finished(self) -> bool:

238 return self.finish_reason is not None

vllm/v1/engine/output_processor.py

@@ -35,6 +35,7 @@ from vllm.v1.engine.parallel_sampling import ParentRequest

35 from vllm.v1.metrics.stats import (

36 IterationStats,

37 LoRARequestStates,

38 + RequestSpecDecodeMetrics,

39 RequestStateStats,

40 SchedulerStats,

41 )

@@ -175,6 +176,9 @@ class RequestState:

176 self.queue = queue

177 self.num_cached_tokens = 0

178 self.num_cache_creation_tokens = 0

179 + # Per-sequence spec-decode accumulator; arrives once (on finish) via

180 + # EngineCoreOutput, then attached to this sequence's CompletionOutput.

181 + self.spec_decode_metrics: RequestSpecDecodeMetrics | None = None

182

183 self.stats = RequestStateStats(arrival_time=arrival_time) if log_stats else None

184

@@ -429,6 +433,7 @@ class RequestState:

433 cumulative_logprob=self.logprobs_processor.cumulative_logprob,

434 finish_reason=str(finish_reason) if finished else None,

435 stop_reason=stop_reason if finished else None,

436 + spec_decode_metrics=self.spec_decode_metrics if finished else None,

437 )

438

439 def _new_pooling_output(self, pooling_output: torch.Tensor) -> PoolingOutput:

@@ -658,6 +663,9 @@ class OutputProcessor:

663 )

664 req_state.is_prefilling = False

665

666 + if engine_core_output.spec_decode_metrics is not None:

667 + req_state.spec_decode_metrics = engine_core_output.spec_decode_metrics

668 +

669 if pooling_output is None:

670 assert req_state.detokenizer is not None

671 assert req_state.logprobs_processor is not None

vllm/v1/metrics/stats.py

+76

@@ -297,6 +297,82 @@ class PrefillStats:

297 )

298

299

300 +@dataclass

301 +class RequestSpecDecodeMetrics:

302 + """Per-output-sequence speculative-decoding statistics accumulator.

303 +

304 + Accumulates, over one sequence's verify steps, a histogram of accepted

305 + draft-token counts (``j``, draft-only) and the total number of proposed

306 + draft tokens. When ``detailed`` is requested it also records the ordered

307 + per-step accepted/proposed sequences (``summary`` omits them). Tracked per

308 + engine ``Request`` (one per sampled sequence, so ``n > 1`` yields one per

309 + child), surfaced via ``EngineCoreOutput`` and the response

310 + ``metrics.speculative_decoding`` for single-sequence requests (see

311 + ``to_dict``).

312 +

313 + Fields:

314 + num_spec_tokens: Configured ``num_speculative_tokens`` (the max ``k``);

315 + also the histogram's upper bound.

316 + histogram: Dense counts indexed by accepted draft tokens ``j``

317 + (length ``num_spec_tokens + 1``).

318 + num_draft_tokens: Total proposed draft tokens, after the

319 + grammar-invalidated (``num_invalid_spec_tokens``) adjustment.

320 + per_step_accepted: Ordered accepted-draft count per verify step

321 + (``detailed`` only; empty otherwise).

322 + per_step_drafted: Ordered proposed-draft count per verify step

323 + (``detailed`` only; empty otherwise).

324 + """

325 +

326 + num_spec_tokens: int

327 + histogram: list[int] = field(default_factory=list)

328 + num_draft_tokens: int = 0

329 + per_step_accepted: list[int] = field(default_factory=list)

330 + per_step_drafted: list[int] = field(default_factory=list)

331 +

332 + @classmethod

333 + def new(cls, num_spec_tokens: int) -> "RequestSpecDecodeMetrics":

334 + return cls(

335 + num_spec_tokens=num_spec_tokens,

336 + histogram=[0] * (num_spec_tokens + 1),

337 + )

338 +

339 + def observe(

340 + self, num_draft_tokens: int, num_accepted: int, detailed: bool = False

341 + ) -> None:

342 + self.histogram[num_accepted] += 1

343 + self.num_draft_tokens += num_draft_tokens

344 + if detailed:

345 + self.per_step_accepted.append(num_accepted)

346 + self.per_step_drafted.append(num_draft_tokens)

347 +

348 + def to_dict(self) -> dict[str, Any]:

349 + """Payload matching ``SpeculativeDecodingMetrics`` for the response.

350 +

351 + ``acceptance_histogram`` is a dense list indexed by accepted draft count

352 + ``j`` (length ``num_spec_tokens + 1``). ``mean_acceptance_length``

353 + includes the bonus token (``j + 1``); ``draft_acceptance_rate`` is

354 + draft-only, full precision. Per-step arrays are included only when

355 + populated (``detailed`` level).

356 + """

357 + num_spec_steps = sum(self.histogram)

358 + num_accepted = sum(j * count for j, count in enumerate(self.histogram))

359 + mean_al = 1.0 + num_accepted / num_spec_steps if num_spec_steps else 1.0

360 + rate = num_accepted / self.num_draft_tokens if self.num_draft_tokens else 0.0

361 + result: dict[str, Any] = {

362 + "mean_acceptance_length": mean_al,

363 + "draft_acceptance_rate": rate,

364 + "acceptance_histogram": list(self.histogram),

365 + "num_spec_steps": num_spec_steps,

366 + "num_accepted_draft_tokens": num_accepted,

367 + "num_draft_tokens": self.num_draft_tokens,

368 + "num_spec_tokens": self.num_spec_tokens,

369 + }

370 + if self.per_step_accepted:

371 + result["per_step_accepted"] = self.per_step_accepted

372 + result["per_step_drafted"] = self.per_step_drafted

373 + return result

374 +

375 +

376 @dataclass

377 class PromptTokenStats:

378 """Breakdown of prompt tokens by source.

vllm/v1/request.py

+6 -1

@@ -20,7 +20,7 @@ from vllm.v1.engine import (

20 EngineCoreRequest,

21 FinishReason,

22 )

23 -from vllm.v1.metrics.stats import PrefillStats

23 +from vllm.v1.metrics.stats import PrefillStats, RequestSpecDecodeMetrics

24 from vllm.v1.structured_output.request import StructuredOutputRequest

25 from vllm.v1.utils import ConstantList

@@ -211,6 +211,11 @@ class Request:

211

212 self.prefill_stats: PrefillStats | None = PrefillStats()

213

214 + # Per-request speculative-decoding acceptance accumulator. Populated by

215 + # the scheduler when --per-request-spec-decode-metrics is set (eagerly on

216 + # add_request, then observed each verify step); stays None otherwise.

217 + self.spec_decode_metrics: RequestSpecDecodeMetrics | None = None

218 +

219 self.block_hashes: list[BlockHash] = []

220 # Store the block hasher without binding self to avoid creating a

221 # reference cycle (Request -> partial -> Request) that prevents