| 1 | # Per-Request Acceptance Metrics |
| 2 | |
| 3 | When speculative decoding is enabled, vLLM can report per-request acceptance |
| 4 | metrics in the response, under `metrics.speculative_decoding`. This lets a |
| 5 | client compute the mean acceptance length and the accepted-draft-length |
| 6 | distribution for an individual request, as a complement to the server-aggregated |
| 7 | spec-decode metrics exposed at `/metrics`. |
| 8 | |
| 9 | !!! warning "Experimental" |
| 10 | `metrics.speculative_decoding` is experimental and its shape may change in a |
| 11 | future release. Pin to a vLLM version if you depend on it. |
| 12 | |
| 13 | ## Enabling |
| 14 | |
| 15 | Start the server with `--per-request-spec-decode-metrics` set to `summary` or |
| 16 | `detailed` (default `none`): |
| 17 | |
| 18 | ```bash |
| 19 | vllm serve <target-model> \ |
| 20 | --speculative-config '{"method": "ngram", "num_speculative_tokens": 3, "prompt_lookup_min": 1, "prompt_lookup_max": 3}' \ |
| 21 | --per-request-spec-decode-metrics summary |
| 22 | ``` |
| 23 | |
| 24 | | Level | Behavior | |
| 25 | | --- | --- | |
| 26 | | `none` (default) | No collection; responses are unchanged. | |
| 27 | | `summary` | Acceptance metrics per request. | |
| 28 | | `detailed` | `summary` plus ordered per-step arrays. | |
| 29 | |
| 30 | Collection is gated at the source: with `none`, nothing is accumulated. |
| 31 | |
| 32 | ## Response Format |
| 33 | |
| 34 | Acceptance metrics share the top-level `metrics` object with the timing |
| 35 | [per-request metrics](../per_request_metrics.md) — `metrics.speculative_decoding` |
| 36 | sits alongside the timing fields. Like timing, they describe a single generation |
| 37 | stream, so they are reported only for single-sequence requests and are `null` |
| 38 | for `n > 1`. |
| 39 | |
| 40 | A `summary` response's `metrics` looks like: |
| 41 | |
| 42 | ```json |
| 43 | { |
| 44 | "choices": [ ... ], |
| 45 | "usage": { ... }, |
| 46 | "metrics": { |
| 47 | "speculative_decoding": { |
| 48 | "mean_acceptance_length": 1.2325581395348837, |
| 49 | "draft_acceptance_rate": 0.07751937984496124, |
| 50 | "acceptance_histogram": [39, 1, 0, 3], |
| 51 | "num_spec_steps": 43, |
| 52 | "num_accepted_draft_tokens": 10, |
| 53 | "num_draft_tokens": 129, |
| 54 | "num_spec_tokens": 3 |
| 55 | } |
| 56 | } |
| 57 | } |
| 58 | ``` |
| 59 | |
| 60 | | Field | Description | |
| 61 | | --- | --- | |
| 62 | | `mean_acceptance_length` | Mean tokens emitted per verification step, including the bonus token: `1 + num_accepted_draft_tokens / num_spec_steps`. Ranges from `1.0` (nothing accepted) to `num_spec_tokens + 1`. | |
| 63 | | `draft_acceptance_rate` | Fraction of proposed draft tokens accepted: `num_accepted_draft_tokens / num_draft_tokens`. | |
| 64 | | `acceptance_histogram` | Dense list of length `num_spec_tokens + 1`; index `j` is the number of steps that accepted exactly `j` draft tokens. Excludes the always-accepted bonus token. | |
| 65 | | `num_spec_steps` | Number of verification steps for this request (the sum of the histogram). | |
| 66 | | `num_accepted_draft_tokens` | Total accepted draft tokens, excluding bonus tokens. | |
| 67 | | `num_draft_tokens` | Total proposed draft tokens, after subtracting drafts invalidated by structured-output constraints. | |
| 68 | | `num_spec_tokens` | Configured `num_speculative_tokens` (`k`), i.e. the maximum draft length per step. | |
| 69 | |
| 70 | With `detailed`, two ordered arrays are added, one entry per verification step: |
| 71 | |
| 72 | | Field | Description | |
| 73 | | --- | --- | |
| 74 | | `per_step_accepted` | Accepted draft count at each step. | |
| 75 | | `per_step_drafted` | Proposed draft count at each step. Records the effective proposal length per step, so variable-length drafting (e.g. adaptive speculation) is represented without a schema change. | |
| 76 | |
| 77 | `metrics.speculative_decoding` is present whenever `--per-request-spec-decode-metrics` |
| 78 | is `summary`/`detailed`, speculative decoding is enabled, and `n == 1` (with an |
| 79 | all-zero histogram if the request drafted nothing). It is `null` otherwise. |
| 80 | |
| 81 | ## Streaming |
| 82 | |
| 83 | In streaming responses, `metrics` (including `speculative_decoding`) rides the |
| 84 | final usage chunk, which is only emitted when usage reporting is enabled — set |
| 85 | `stream_options.include_usage: true` or start the server with |
| 86 | `--enable-force-include-usage`. |
| 87 | |
| 88 | ## Relationship to Prometheus metrics |
| 89 | |
| 90 | The per-request fields are the individual-request counterpart of the |
| 91 | server-aggregated spec-decode counters at `/metrics`. Summed across the |
| 92 | single-sequence requests that report them, they reconcile with the aggregate |
| 93 | counters (which also count `n > 1` requests, so the totals match only for |
| 94 | all-`n == 1` workloads): |
| 95 | |
| 96 | | Per-request field (summed) | Prometheus counter | |
| 97 | | --- | --- | |
| 98 | | `num_spec_steps` | `vllm:spec_decode_num_drafts_total` | |
| 99 | | `num_draft_tokens` | `vllm:spec_decode_num_draft_tokens_total` | |
| 100 | | `num_accepted_draft_tokens` | `vllm:spec_decode_num_accepted_tokens_total` | |