A high-throughput and memory-efficient inference and serving engine for LLMs
[Docs] Add KV offloading usage guide (single- and multi-tier) (#44415)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Ronen Schaffer committed
Jun 9, 2026 at 19:20 UTC
3d119f78f77cd460c39a1c8ba8303724ad1f88bc
2 files changed
+134
docs/features/disagg_prefill.md
+2
@@ -41,6 +41,8 @@ Now supports 9 types of connectors:
41
--kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"block_size": 64, "cpu_bytes_to_use": 1000000000}}'
42
```
43
44
+ For multi-tier offloading (e.g., CPU + filesystem tier) and the full configuration reference, see the [KV Offloading Usage Guide](kv_offloading_usage.md).
45
+
46
- **FlexKVConnectorV1**: refer to [examples/disaggregated/flexkv_connector/prefix_caching_flexkv.py](../../examples/disaggregated/flexkv_connector/prefix_caching_flexkv.py) for the example usage of FlexKVConnectorV1. FlexKV is a distributed KV Store and multi-level cache management system for ultra-large-scale LLM inference.
47
48
```bash
docs/features/kv_offloading_usage.md
new
+132
@@ -0,0 +1,132 @@
1
+# KV Offloading Usage Guide
2
+
3
+This guide covers configuration of the [`OffloadingConnector`](disagg_prefill.md), which extends the prefix cache by offloading completed KV blocks to slower but larger tiers (CPU host memory, plus optional secondary tiers) as they are produced. Hits in the offload tiers are promoted back to GPU on demand. Transfers between GPU and CPU use DMA (`cudaMemcpyAsync`) and run asynchronously alongside model computation, so offloading adds minimal CPU- and GPU-core overhead.
4
+
5
+!!! note
6
+ The `OffloadingConnector` currently supports CUDA, ROCm, and XPU only.
7
+
8
+## Overview
9
+
10
+Two specs are available, selected by the `spec_name` key in `kv_connector_extra_config`:
11
+
12
+- `CPUOffloadingSpec` (default): single CPU tier. Completed GPU blocks are copied into pinned host memory.
13
+- `TieringOffloadingSpec`: multi-tier. A CPU primary tier plus one or more secondary tiers.
14
+
15
+Only the CPU primary tier has direct GPU access. Secondary tiers cannot read from or write to GPU memory; all GPU↔secondary transfers are staged through the CPU primary tier.
16
+
17
+```mermaid
18
+flowchart LR
19
+ GPU <--> CPU["CPU primary tier"]
20
+ CPU <--> S0["Secondary tier 0"]
21
+ CPU <--> S1["Secondary tier 1"]
22
+ CPU <--> SN["..."]
23
+```
24
+
25
+## Single-Tier Setup (CPU Only)
26
+
27
+```bash
28
+vllm serve <model> \
29
+ --kv-transfer-config '{
30
+ "kv_connector": "OffloadingConnector",
31
+ "kv_role": "kv_both",
32
+ "kv_connector_extra_config": {
33
+ "block_size": 64,
34
+ "cpu_bytes_to_use": 1000000000
35
+ }
36
+ }'
37
+```
38
+
39
+## Multi-Tier Setup
40
+
41
+Set `spec_name` to `"TieringOffloadingSpec"` and supply a `secondary_tiers` list. Each entry is a dict with a required `type` key plus tier-specific fields. The list is ordered: tier 0 is consulted before tier 1, and so on. See [Secondary Tiers](#secondary-tiers) for tier-specific keys.
42
+
43
+```bash
44
+vllm serve <model> \
45
+ --kv-transfer-config '{
46
+ "kv_connector": "OffloadingConnector",
47
+ "kv_role": "kv_both",
48
+ "kv_connector_extra_config": {
49
+ "spec_name": "TieringOffloadingSpec",
50
+ "cpu_bytes_to_use": 10737418240,
51
+ "block_size": 16,
52
+ "eviction_policy": "lru",
53
+ "secondary_tiers": [
54
+ {
55
+ "type": "fs",
56
+ "root_dir": "/mnt/kv_cache",
57
+ "n_read_threads": 32,
58
+ "n_write_threads": 16
59
+ }
60
+ ]
61
+ }
62
+ }'
63
+```
64
+
65
+## `kv_connector_extra_config` Reference
66
+
67
+| Key | Required | Default | Scope | Notes |
68
+| --- | --- | --- | --- | --- |
69
+| `spec_name` | no | `CPUOffloadingSpec` | both | Set to `TieringOffloadingSpec` for multi-tier. |
70
+| `cpu_bytes_to_use` | yes | — | both | Total bytes of host memory reserved for the CPU tier across all workers (not per-worker). |
71
+| `block_size` | no | GPU block size | both | Offloaded block size in tokens; must be a multiple of the GPU block size. |
72
+| `eviction_policy` | no | `lru` | both | Primary tier policy: `lru` or `arc`. |
73
+| `store_threshold` | no | `0` | single-tier | Min lookups before a block is offloaded. Values ≥ 2 are rejected by `TieringOffloadingSpec`. |
74
+| `max_tracker_size` | no | `64000` | single-tier | Max entries in the lookup tracker. |
75
+| `secondary_tiers` | no | `[]` | multi-tier | List of secondary tier configs (see below). |
76
+| `offload_prompt_only` | no | `true` | both | If `true`, only prompt (prefill) blocks are offloaded; decode blocks are skipped. |
77
+| `spec_module_path` | no | — | both | Python import path for a custom `OffloadingSpec` not in the built-in registry. Required only when `spec_name` is not built-in (advanced). |
78
+
79
+## Secondary Tiers
80
+
81
+Each entry in `secondary_tiers` is a dict with a required `type` field plus tier-specific fields.
82
+
83
+### Filesystem (FS)
84
+
85
+The filesystem tier (`type: "fs"`) writes blocks to a directory on local storage.
86
+
87
+| Key | Required | Default | Notes |
88
+| --- | --- | --- | --- |
89
+| `type` | yes | — | Must be `fs`. |
90
+| `root_dir` | yes | — | Base directory; vLLM creates subdirectories beneath it (see [On-Disk Layout](#on-disk-layout)). |
91
+| `n_read_threads` | no | `16` | Read-priority I/O threads (load path). |
92
+| `n_write_threads` | no | `16` | Write-priority I/O threads (store path). |
93
+
94
+Each thread group prefers its own queue but pulls from the other when its primary queue is empty, so a write-heavy or read-heavy burst won't leave the off-priority queue waiting. Size the totals to your storage's effective concurrency.
95
+
96
+#### On-Disk Layout
97
+
98
+Under `root_dir`, vLLM creates a subdirectory `<model>_<digest>`, where `<model>` is the model name with `/` replaced by `_` (so HuggingFace IDs like `meta-llama/Llama-3-8B` don't nest), and `<digest>` is a short SHA256 prefix derived from the run configuration (model, block size, parallelism, dtype, etc.). Runs with the same configuration share the same subdirectory; runs with different configurations live side-by-side under the same `root_dir` without colliding.
99
+
100
+Inside that subdirectory, blocks are sharded across hash-prefix subdirectories to limit directory fan-out:
101
+
102
+```text
103
+<root_dir>/
104
+ <model>_<digest>/
105
+ config.json
106
+ <model>_<digest>_r<rank>/
107
+ <hhh>/ # first 3 hex chars of the block hash
108
+ <hh>_g<group_idx>/ # next 2 hex chars + KV cache group index
109
+ <hash_hex>.bin # full block hash (in hex)
110
+```
111
+
112
+`config.json` records the run (block size, number of KV groups, etc.) and is written on first start. Each rank writes blocks under its own `_r<rank>` sibling directory, so multiple ranks can safely share the same `root_dir`.
113
+
114
+#### Cross-Process Sharing
115
+
116
+To enable KV cache sharing between multiple vLLM instances using the same `root_dir` (e.g., via a shared PVC), the `PYTHONHASHSEED` environment variable must be set to the same fixed value (e.g., `"0"`) on every instance. Without this, each process initializes `NONE_HASH` (the chain-hash seed for block content hashes) with random bytes, producing different block filenames for identical token content.
117
+
118
+```bash
119
+PYTHONHASHSEED=0 vllm serve ...
120
+```
121
+
122
+## Tuning Tips
123
+
124
+- `cpu_bytes_to_use`: a bigger CPU tier means fewer trips to slower secondary tiers and a higher hit rate. The value is total across all workers, not per-worker. Leave headroom for the rest of the host workload.
125
+- For single-tier (CPU-only) setups, set `cpu_bytes_to_use` larger than the aggregate GPU KV cache. Because offloading is immediate, a smaller CPU tier just mirrors what the GPU already holds and adds no hit rate.
126
+- `block_size`: larger offloaded blocks reduce per-block bookkeeping overhead but increase the granularity of lookups. Must be a multiple of the GPU block size.
127
+- FS thread counts: tune `n_read_threads` and `n_write_threads` to the parallelism your storage can sustain. Reads are latency-sensitive on the prefill path, so prefer more read threads when prefill hit rates are high.
128
+- Sharing `root_dir` across runs: runs with the same model, `block_size`, parallelism layout, and dtype share files under the same `<digest>` subdirectory. Changing any of these produces a new subdirectory; old ones are orphaned but harmless. Delete them to reclaim disk.
129
+
130
+## Further Reading
131
+
132
+- [vLLM blog: KV Offloading Connector](https://vllm.ai/blog/2026-01-08-kv-offloading-connector) — motivation, architecture (DMA-based async transfer), and benchmarks (TTFT and throughput).