[Doc] Score centering via top-k processed logprobs (#59361)
Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
vx120 committed
Oct 1, 2026 at 08:57 UTC
a47e4a5568b4791db0c9633b5dc7531128f98bae
1 file changed
+33
docs/training/sampling_mask.md
+33
@@ -104,6 +104,39 @@ log_prob = log_softmax(masked_logits)[sampled_token_id]
104
Both sides normalize over the same token set, so the importance ratio is
105
consistent.
106
107
+## Score centering with top-k logprobs
108
+
109
+[Score centering](https://arxiv.org/abs/2609.20807) needs the sampler's top-k
110
+token IDs and logprobs plus the sampled token's logprob; the paper uses
111
+`k=128` and does not need the exact sampling support. vLLM already returns
112
+this through the ordinary `logprobs` parameter:
113
+
114
+```bash
115
+vllm serve <model> --logprobs-mode processed_logprobs --max-logprobs 128
116
+```
117
+
118
+```json
119
+{"sampling_params": {"temperature": 1.0, "top_p": 0.9, "logprobs": 128}}
120
+```
121
+
122
+Each position then carries the sampled token's logprob and the top-128
123
+`(token_id, logprob)` pairs, normalized over the post-truncation support:
124
+
125
+```json
126
+{"token": "token_id:11", "logprob": -0.757,
127
+ "top_logprobs": [{"token": "token_id:11", "logprob": -0.757},
128
+ {"token": "token_id:13", "logprob": -0.632},
129
+ {"token": "token_id:0", "logprob": -9999.0}, ...]}
130
+```
131
+
132
+- `--max-logprobs` defaults to 20; raise it to the requested `k`.
133
+- Tokens outside the support are returned as `-9999.0` with arbitrary IDs;
134
+ drop them. The finite entries are the sampling mask when it has at most `k`
135
+ tokens.
136
+- The sampled token is always the first entry and may repeat inside the top-k.
137
+- `/inference/v1/generate` encodes IDs as `token_id:<id>`; the OpenAI
138
+ endpoints need `return_tokens_as_token_ids`.
139
+
140
## Limitations
141
142
- **Engine-level flag:** `--return-sampling-mask` globally disables the