A high-throughput and memory-efficient inference and serving engine for LLMs
docs, kv_offloading: add docs for selective offload (#45279)
Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com>
Angelo Ruocco committed
Jun 17, 2026 at 13:58 UTC
ee0fd6984ad6579204a388cd80021fea081c8869
1 file changed
+23
docs/features/kv_offloading_usage.md
+23
@@ -127,6 +127,29 @@ PYTHONHASHSEED=0 vllm serve ...
127
- FS thread counts: tune `n_read_threads` and `n_write_threads` to the parallelism your storage can sustain. Reads are latency-sensitive on the prefill path, so prefer more read threads when prefill hit rates are high.
128
- Sharing `root_dir` across runs: runs with the same model, `block_size`, parallelism layout, and dtype share files under the same `<digest>` subdirectory. Changing any of these produces a new subdirectory; old ones are orphaned but harmless. Delete them to reclaim disk.
129
130
+## Per-Request Selective Offload
131
+
132
+Individual requests can cap how many of their tokens are eligible for offload by setting `max_offload_tokens` in the request's `kv_transfer_params`. Only the first `max_offload_tokens` tokens of the request are offloaded; blocks beyond that point are skipped on the store path. This is useful when a known prefix (e.g., a system prompt or shared context) is worth caching but later request-specific tokens are not.
133
+
134
+| Key | Type | Notes |
135
+| --- | --- | --- |
136
+| `max_offload_tokens` | non-negative `int` | Upper bound on tokens to offload for this request. `0` disables offload for the request entirely; omit the key (or set to `None`) for no cap. Non-`int`, negative, or `bool` values are rejected with a warning and treated as no cap. |
137
+
138
+!!! note
139
+ `max_offload_tokens` is experimental and subject to change.
140
+
141
+Example (OpenAI-compatible completions request):
142
+
143
+```json
144
+{
145
+ "model": "<model>",
146
+ "prompt": "...",
147
+ "kv_transfer_params": {
148
+ "max_offload_tokens": 1024
149
+ }
150
+}
151
+```
152
+
153
## Further Reading
154
155
- [vLLM blog: KV Offloading Connector](https://vllm.ai/blog/2026-01-08-kv-offloading-connector) — motivation, architecture (DMA-based async transfer), and benchmarks (TTFT and throughput).