A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 7, 06:30 UTC
docs: add README for logits_processor examples (#35933)
Dr Alex Mitre committed
Mar 4, 2026 at 11:09 UTC
3417ba5648b73b8125bdd20a2b9bb11ac35b9ab7
1 file changed
+40
examples/offline_inference/logits_processor/README.md
new
+40
@@ -0,0 +1,40 @@
1
+# Custom Logits Processors
2
+
3
+This directory contains examples demonstrating how to use custom logits processors with vLLM's offline inference API. Logits processors allow you to modify the model's output distribution before sampling, enabling controlled generation behaviors like token masking, constrained decoding, and custom sampling strategies.
4
+
5
+## Scripts
6
+
7
+### `custom.py` — Engine-level logits processor
8
+
9
+Demonstrates how to instantiate vLLM with a custom logits processor class that operates at the batch level. The example uses a `DummyLogitsProcessor` that masks out all tokens except a specified `target_token` when passed via `SamplingParams.extra_args`.
10
+
11
+```bash
12
+python examples/offline_inference/logits_processor/custom.py
13
+```
14
+
15
+### `custom_req.py` — Request-level logits processor wrapper
16
+
17
+Shows how to wrap a request-level logits processor (which operates on individual requests) to be compatible with vLLM's batch-level logits processing interface.
18
+
19
+```bash
20
+python examples/offline_inference/logits_processor/custom_req.py
21
+```
22
+
23
+### `custom_req_init.py` — Request-level processor with engine config
24
+
25
+A special case of wrapping a request-level logits processor where the processor needs access to engine configuration or model metadata during initialization (e.g., vocabulary size, tokenizer info).
26
+
27
+```bash
28
+python examples/offline_inference/logits_processor/custom_req_init.py
29
+```
30
+
31
+## Key Concepts
32
+
33
+- **Batch-level vs. request-level**: vLLM processes logits at the batch level for efficiency. If you have a per-request processor, you need to wrap it using the patterns shown in `custom_req.py` and `custom_req_init.py`.
34
+- **`SamplingParams.extra_args`**: Use this to pass custom keyword arguments to your logits processor on a per-request basis (e.g., `target_token`).
35
+- **`DummyLogitsProcessor`**: A reference implementation available in `vllm/test_utils.py` that can be used as a starting point for custom processors.
36
+
37
+## Further Reading
38
+
39
+- [vLLM Sampling Parameters](https://docs.vllm.ai/en/latest/api/inference_params.html#sampling-parameters)
40
+- [vLLM LLM API](https://docs.vllm.ai/en/latest/api/offline_inference/llm.html)