| 1 | # Trace Replay |
| 2 | |
| 3 | Trace replay forces the engine to emit a predetermined token sequence during decoding while computing real logprobs from the model's unmodified logit distribution. The primary use case is comparing logprob distributions across different configurations: |
| 4 | |
| 5 | - **Inference config diff**: compare logprobs between different quantization schemes, tensor parallelism layouts, or attention backends for the same model. |
| 6 | - **Train vs inference diff**: replay a training-time token sequence through the inference engine to detect logprob divergence caused by numerical differences between training and serving frameworks. |
| 7 | |
| 8 | ## Requirements |
| 9 | |
| 10 | Trace replay reserves a per-request trace buffer, so it is off by default. Enable it with `--enable-trace-replay` (or `LLM(..., enable_trace_replay=True)`). It is only supported by model runner V2; requests are rejected when either condition is not met. |
| 11 | |
| 12 | ## Usage |
| 13 | |
| 14 | ```python |
| 15 | from vllm import LLM, SamplingParams |
| 16 | |
| 17 | llm = LLM(model="Qwen/Qwen3-0.6B", enable_trace_replay=True) |
| 18 | |
| 19 | # Token sequence captured from a previous run or training log |
| 20 | trace_tokens = [15, 284, 1026, 374] |
| 21 | |
| 22 | params = SamplingParams( |
| 23 | trace_decode_token_ids=trace_tokens, |
| 24 | logprobs=5, |
| 25 | ) |
| 26 | outputs = llm.generate(["Once upon a time"], sampling_params=params) |
| 27 | |
| 28 | for token, logprob in zip( |
| 29 | outputs[0].outputs[0].token_ids, |
| 30 | outputs[0].outputs[0].logprobs, |
| 31 | ): |
| 32 | print(f"token={token} logprob={logprob[token].logprob:.4f}") |
| 33 | ``` |
| 34 | |
| 35 | The output tokens will always be `[15, 284, 1026, 374]`. The logprobs reflect the model's true probability for each forced token under the current inference configuration. |
| 36 | |
| 37 | To compare configurations, run the same trace against two engine setups and diff the per-token logprobs. |
| 38 | |
| 39 | ## Behavior |
| 40 | |
| 41 | When `trace_decode_token_ids` is set: |
| 42 | |
| 43 | - `max_tokens` is automatically set to the trace length. |
| 44 | - The trace is truncated if it does not fit within `max_model_len` given the prompt length. |
| 45 | - All stop conditions (EOS, stop strings, stop token IDs) are disabled. |
| 46 | - Generation produces exactly the trace tokens, then stops. |
| 47 | |
| 48 | ## Limitations |
| 49 | |
| 50 | `trace_decode_token_ids` is incompatible with the following features (raises `ValueError`): |
| 51 | |
| 52 | | Feature | Reason | |
| 53 | | --- | --- | |
| 54 | | `n > 1` | Trace replay produces a single deterministic sequence | |
| 55 | | `prompt_logprobs` | Expanded logit layout conflicts with trace kernel indexing | |
| 56 | | Speculative decoding | Multi-token speculation conflicts with single-token trace stepping | |
| 57 | | Structured outputs | Grammar constraints conflict with forced token injection | |
| 58 | | `repetition_detection` | Would terminate the request on repeated patterns in the trace | |
| 59 | | `thinking_token_budget` | Logit masking corrupts logprobs for trace tokens | |
| 60 | | `bad_words` | Logit masking corrupts logprobs for trace tokens | |