| 1 | # MooncakeStoreConnector Usage Guide |
| 2 | |
| 3 | MooncakeStoreConnector is a KV cache connector that uses [MooncakeDistributedStore](https://github.com/kvcache-ai/Mooncake) as a shared KV cache pool. Unlike `MooncakeConnector` which does direct point-to-point KV transfer between prefiller and decoder, MooncakeStoreConnector enables KV cache offloading to an external distributed store, supporting: |
| 4 | |
| 5 | - **CPU/disk offloading**: Extend effective KV cache capacity by offloading to CPU memory or disk via Mooncake's transfer engine. |
| 6 | - **Prefix caching across instances**: Hash-based deduplication allows multiple vLLM instances to share cached KV blocks through the store. |
| 7 | - **Single-node and multi-node deployment**: Works both as a standalone KV cache extension and in disaggregated prefill-decode setups. |
| 8 | |
| 9 | ## Prerequisites |
| 10 | |
| 11 | ### Install Mooncake |
| 12 | |
| 13 | Install mooncake through pip: |
| 14 | |
| 15 | ```bash |
| 16 | uv pip install mooncake-transfer-engine |
| 17 | ``` |
| 18 | |
| 19 | Refer to the [Mooncake official repository](https://github.com/kvcache-ai/Mooncake) for more installation instructions and building from source. |
| 20 | |
| 21 | ### Start the Mooncake Master Server |
| 22 | |
| 23 | The Mooncake master manages metadata and coordinates the distributed store. Start it before launching vLLM: |
| 24 | |
| 25 | ```bash |
| 26 | mooncake_master --port 50051 |
| 27 | ``` |
| 28 | |
| 29 | Default ports: |
| 30 | |
| 31 | - RPC: 50051 |
| 32 | |
| 33 | Multiple vLLM instances can share the same master server. |
| 34 | |
| 35 | ### Configure Mooncake |
| 36 | |
| 37 | Create a JSON configuration file (e.g., `mooncake_config.json`): |
| 38 | |
| 39 | ```json |
| 40 | { |
| 41 | "mode": "embedded", |
| 42 | "metadata_server": "P2PHANDSHAKE", |
| 43 | "master_server_address": "127.0.0.1:50051", |
| 44 | "global_segment_size": "80GB", |
| 45 | "local_buffer_size": "4GB", |
| 46 | "protocol": "rdma", |
| 47 | "device_name": "", |
| 48 | "enable_offload": false |
| 49 | } |
| 50 | ``` |
| 51 | |
| 52 | - `mode`: Topology selection. `"embedded"` (default, PR-40900 baseline) has each |
| 53 | vLLM rank contribute `global_segment_size` to the pool in-process. |
| 54 | `"standalone-store"` makes ranks pure requesters — an external |
| 55 | `mooncake_client` process owns the CPU pool and (optionally) the SSD tier. |
| 56 | - `protocol`: Use `"rdma"` for best performance. `"tcp"` works as a fallback. |
| 57 | - `global_segment_size`: CPU memory contributed to the distributed pool (per |
| 58 | GPU). Must be `> 0` in `embedded` mode and `0` in `standalone-store` mode. |
| 59 | - `local_buffer_size`: Private buffer for this node's own operations (per GPU). |
| 60 | - `enable_offload`: When `true`, vLLM allocates a DirectIO staging buffer so |
| 61 | large prefills do not exceed the owner's SSD-write budget. Set this together |
| 62 | with the matching `--enable_offload=true` flag on `mooncake_master` and on |
| 63 | the external `mooncake_client` (if any). |
| 64 | - `tenant_id`: Optional Mooncake tenant namespace. Producers and consumers |
| 65 | that should share store data must use the same tenant id. Default: |
| 66 | `"default"`. |
| 67 | |
| 68 | Set the config path via environment variable: |
| 69 | |
| 70 | ```bash |
| 71 | export MOONCAKE_CONFIG_PATH=/path/to/mooncake_config.json |
| 72 | ``` |
| 73 | |
| 74 | ## Usage |
| 75 | |
| 76 | ### Single-Node KV Cache Offloading |
| 77 | |
| 78 | Use MooncakeStoreConnector to offload KV cache to CPU memory, extending the effective cache size: |
| 79 | |
| 80 | ```bash |
| 81 | MOONCAKE_CONFIG_PATH=mooncake_config.json \ |
| 82 | vllm serve meta-llama/Llama-3.1-8B-Instruct \ |
| 83 | --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both"}' |
| 84 | ``` |
| 85 | |
| 86 | ### Disaggregated Prefill-Decode (XpYd) |
| 87 | |
| 88 | In disaggregated prefill-decode mode, use `MultiConnector` to combine `MooncakeConnector` (point-to-point KV transfer) with `MooncakeStoreConnector` (shared KV cache pool). This enables both direct P2P transfer between prefiller and decoder, and cross-instance prefix cache sharing via the distributed store. |
| 89 | **Prefiller Node:** |
| 90 | |
| 91 | ```bash |
| 92 | MOONCAKE_CONFIG_PATH=mooncake_config.json \ |
| 93 | VLLM_MOONCAKE_BOOTSTRAP_PORT=50052 \ |
| 94 | vllm serve meta-llama/Llama-3.1-8B-Instruct \ |
| 95 | --port 8100 \ |
| 96 | --kv-transfer-config '{ |
| 97 | "kv_connector": "MultiConnector", |
| 98 | "kv_role": "kv_producer", |
| 99 | "kv_connector_extra_config": { |
| 100 | "connectors": [ |
| 101 | { |
| 102 | "kv_connector": "MooncakeConnector", |
| 103 | "kv_role": "kv_producer" |
| 104 | }, |
| 105 | { |
| 106 | "kv_connector": "MooncakeStoreConnector", |
| 107 | "kv_role": "kv_both" |
| 108 | } |
| 109 | ] |
| 110 | } |
| 111 | }' |
| 112 | ``` |
| 113 | |
| 114 | **Decoder Node:** |
| 115 | |
| 116 | ```bash |
| 117 | MOONCAKE_CONFIG_PATH=mooncake_config.json \ |
| 118 | VLLM_MOONCAKE_BOOTSTRAP_PORT=50053 \ |
| 119 | vllm serve meta-llama/Llama-3.1-8B-Instruct \ |
| 120 | --port 8200 \ |
| 121 | --kv-transfer-config '{ |
| 122 | "kv_connector": "MultiConnector", |
| 123 | "kv_role": "kv_consumer", |
| 124 | "kv_connector_extra_config": { |
| 125 | "connectors": [ |
| 126 | { |
| 127 | "kv_connector": "MooncakeConnector", |
| 128 | "kv_role": "kv_consumer" |
| 129 | }, |
| 130 | { |
| 131 | "kv_connector": "MooncakeStoreConnector", |
| 132 | "kv_role": "kv_consumer" |
| 133 | } |
| 134 | ] |
| 135 | } |
| 136 | }' |
| 137 | ``` |
| 138 | |
| 139 | To also offload newly completed decode KV blocks, add the following extra |
| 140 | configuration to the decoder's `MooncakeStoreConnector` entry. |
| 141 | |
| 142 | When decode processing starts, the consumer checks the block-aligned prompt |
| 143 | prefix and fills any blocks missing from the Store. Subsequent saves append |
| 144 | newly completed decode blocks. This keeps a complete, reusable prefix in the |
| 145 | Store. It also covers prompt KV delivered directly by `MooncakeConnector`. |
| 146 | |
| 147 | ```json |
| 148 | { |
| 149 | "kv_connector_extra_config": { |
| 150 | "save_decode_cache": true |
| 151 | } |
| 152 | } |
| 153 | ``` |
| 154 | |
| 155 | #### Sharing one Store across multiple Prefill TP sizes |
| 156 | |
| 157 | Heterogeneous-TP sharing normally uses a fixed `store_tp_size`. When several |
| 158 | prefillers use different TP sizes, opt in to a common Store TP derived from |
| 159 | their least common multiple: |
| 160 | |
| 161 | ```json |
| 162 | { |
| 163 | "kv_connector_extra_config": { |
| 164 | "enable_store_tp_lcm": true, |
| 165 | "prefill_tp_sizes": [4, 2] |
| 166 | } |
| 167 | } |
| 168 | ``` |
| 169 | |
| 170 | Every prefiller and decoder that shares these entries must use the same list. |
| 171 | The example selects Store TP 4: a TP4 endpoint maps each rank to one Store |
| 172 | shard, while TP2 endpoints map each rank to two Store shards. Runtime TP sizes |
| 173 | remain unchanged. A decoder configured with `"save_decode_cache": true` uses |
| 174 | the same Store TP for decode KV from every prefiller. |
| 175 | |
| 176 | The list may contain positive integer TP sizes. Sharing requires a Store TP that |
| 177 | is at least the local TP and divisible by it, an LBHNC or LBNHC local KV cache, |
| 178 | and the existing topology and KV-head constraints. The Store namespace includes |
| 179 | the attention backend's selected layout. Different layouts use separate Store |
| 180 | entries. Malformed lists and unsupported endpoints use an isolated rank-local |
| 181 | key layout. When |
| 182 | `enable_store_tp_lcm` is absent or false, `prefill_tp_sizes` has no effect and |
| 183 | the existing `store_tp_size` behavior is unchanged. |
| 184 | |
| 185 | **Proxy:** |
| 186 | |
| 187 | A disaggregation proxy routes requests between prefiller and decoder nodes. |
| 188 | When `MooncakeConnector` is also used for direct P2P transfer, refer to its |
| 189 | [usage guide](mooncake_connector_usage.md) for proxy setup details. |
| 190 | |
| 191 | ### Disk Offloading |
| 192 | |
| 193 | Disk offloading is most commonly run in `standalone-store` mode: an external |
| 194 | `mooncake_client` process owns the CPU pool and the SSD tier, and each vLLM |
| 195 | rank is a pure requester. This avoids per-rank duplication of the SSD pool |
| 196 | and keeps DirectIO budget tracking on a single process. |
| 197 | |
| 198 | Three things need to be aligned for end-to-end disk offloading: |
| 199 | |
| 200 | 1. **`mooncake_master`** is started with `--enable_offload=true`. |
| 201 | 2. **`mooncake_client`** (the owner) is started with `--enable_offload=true` |
| 202 | plus an SSD path via `MOONCAKE_OFFLOAD_FILE_STORAGE_PATH`. |
| 203 | 3. **vLLM-side** sets `"enable_offload": true` in the JSON config file (this is |
| 204 | read by the connector and is **not** an environment variable). |
| 205 | |
| 206 | Example `mooncake_config.json` for the vLLM side: |
| 207 | |
| 208 | ```json |
| 209 | { |
| 210 | "mode": "standalone-store", |
| 211 | "metadata_server": "P2PHANDSHAKE", |
| 212 | "master_server_address": "127.0.0.1:50051", |
| 213 | "global_segment_size": 0, |
| 214 | "local_buffer_size": "4GB", |
| 215 | "protocol": "rdma", |
| 216 | "device_name": "mlx5_0", |
| 217 | "enable_offload": true |
| 218 | } |
| 219 | ``` |
| 220 | |
| 221 | Steer this rank to the local owner segment with: |
| 222 | |
| 223 | ```bash |
| 224 | export MOONCAKE_PREFERRED_SEGMENT=127.0.0.1:50053 |
| 225 | ``` |
| 226 | |
| 227 | The owner's SSD directory, on-disk eviction policy, and the DirectIO staging |
| 228 | buffer size are controlled on the `mooncake_client` side via the standard |
| 229 | Mooncake environment variables (`MOONCAKE_OFFLOAD_FILE_STORAGE_PATH`, |
| 230 | `MOONCAKE_BUCKET_EVICTION_POLICY`, `MOONCAKE_USE_URING`, |
| 231 | `MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES`, |
| 232 | `MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES`, etc.). Those are independent of |
| 233 | the vLLM JSON config. |
| 234 | |
| 235 | ### Tenant Isolation |
| 236 | |
| 237 | Set `tenant_id` in the Mooncake JSON config when different vLLM deployments should use separate Mooncake tenant namespaces: |
| 238 | |
| 239 | ```json |
| 240 | { |
| 241 | "mode": "embedded", |
| 242 | "metadata_server": "P2PHANDSHAKE", |
| 243 | "master_server_address": "127.0.0.1:50051", |
| 244 | "global_segment_size": "80GB", |
| 245 | "local_buffer_size": "4GB", |
| 246 | "protocol": "rdma", |
| 247 | "device_name": "", |
| 248 | "enable_offload": false, |
| 249 | "tenant_id": "tenant-a" |
| 250 | } |
| 251 | ``` |
| 252 | |
| 253 | Strict isolation requires a Mooncake master started with `--enable_multi_tenants=true` and a tenant quota policy that registers each tenant. Non-default `tenant_id` also requires a Mooncake version whose `MooncakeDistributedStore.setup()` accepts the `tenant_id` parameter. In `standalone-store` mode, start the external `mooncake_client` with the matching tenant id because that process owns the real store client. |
| 254 | |
| 255 | ## Environment Variables |
| 256 | |
| 257 | | Variable | Description | Default | |
| 258 | | --- | --- | --- | |
| 259 | | `MOONCAKE_CONFIG_PATH` | Path to Mooncake JSON config file | (required) | |
| 260 | | `VLLM_MOONCAKE_BOOTSTRAP_PORT` | Bootstrap port for MooncakeConnector P2P transfer (disagg mode only) | 8998 | |
| 261 | | `MOONCAKE_PREFERRED_SEGMENT` | Pin this rank's replicas to a specific owner segment (`host:port`); used in `standalone-store` mode | — | |
| 262 | | `MOONCAKE_REQUESTER_LOCAL_HOSTNAME` | Override the hostname the vLLM rank registers with Mooncake as a requester. Defaults to the rank's resolved IP. | — | |
| 263 | | `VLLM_MOONCAKE_STORE_TIER_LOG` | When `1`, logs a per-batch tier summary (memory vs disk hits) for observability | disabled | |
| 264 | | `VLLM_MOONCAKE_DISK_STAGING_USABLE_RATIO` | Fraction of the owner's DirectIO staging buffer that the requester will fill in a single `batch_get_into_multi_buffers` call. Lower → more conservative pre-split, more round trips. | 0.9 | |
| 265 | |
| 266 | ## KV Transfer Config |
| 267 | |
| 268 | ### KV Role Options |
| 269 | |
| 270 | - **kv_producer**: For instances that store KV caches to the pool. |
| 271 | - **kv_consumer**: For instances that load KV caches from the pool. |
| 272 | - **kv_both**: The instance both stores and loads KV caches. Use this for single-node CPU offloading or prefiller instances. |
| 273 | |
| 274 | ### kv_connector_extra_config |
| 275 | |
| 276 | - `load_async` (bool): Enable asynchronous loading for better compute-I/O overlap. Default: `true`. |
| 277 | - `lookup_async` (bool): Run the external prefix-cache lookup on a background thread so it never blocks the scheduler step. The request is held until the in-flight lookup completes, then resumed on a later step. Default: `false`. |
| 278 | - `lookup_rpc_port` (int): Custom port for the ZMQ lookup RPC socket. Default: `0`. |
| 279 | - `cache_prefix` (str): Namespace prepended to every store key. Lets separate deployments share one Mooncake master without polluting each other — instances configured with different prefixes never see each other's cached blocks, even for identical prompts. All instances that should share a prefix cache must use the same value. Default: `""` (no prefix; keys are byte-identical to the unprefixed format). |
| 280 | - `save_decode_cache` (bool): Enable offloading decode tokens' KV cache. A `kv_consumer` does not save during prefill; when decode starts, it fills any missing block-aligned prompt prefix before appending completed decode blocks. Default: `false`. |
| 281 | - `store_tp_size` (int): Common Store TP for endpoints with different local TP sizes. It supports LBHNC and LBNHC local KV caches, with `store_tp_size >= local_tp_size` and `store_tp_size % local_tp_size == 0`. The current topology is one full-attention cache group, PCP/DCP disabled, and cross-layer blocks disabled. For GQA and MHA, the total KV-head count must be divisible by `store_tp_size`. Store shards contain fixed global KV-head ranges in the local layout. Shared endpoints use the same KV cache layout, pipeline-parallel size, and Store TP. The Store namespace includes the layout and PP size. Unsupported configurations use a topology-specific rank-local namespace. |
| 282 | |
| 283 | LBHNC/HND is strongly recommended for TP-sharded Store when supported. LBNHC/NHD |
| 284 | creates many transfer segments and may significantly reduce PUT/GET performance. |
| 285 | |
| 286 | For example, with prefill TP 4, decode TP 2, and eight KV heads, set |
| 287 | `store_tp_size` to 4 on both instances. Each decode rank reads and writes two |
| 288 | of the four Store shards. |
| 289 | |
| 290 | MQA with one total KV head uses a replicated-head layout. For the supported |
| 291 | prefill TP 4 to decode TP 2 case, every rank uses the |
| 292 | same rank-0 key namespace. The four prefill replicas stripe block PUTs so each |
| 293 | object is stored once, while both decode ranks GET every block into their local |
| 294 | KV replica. `store_tp_size` does not appear in MQA keys, so identical MQA |
| 295 | objects written at different store TP sizes share the same pool entry when PP |
| 296 | sizes match. |
| 297 | |
| 298 | Tensor-parallel collectives and low-precision arithmetic are not bitwise |
| 299 | invariant across TP sizes, so heterogeneous-TP reuse does not guarantee the |
| 300 | same greedy output as recomputing the prefix at the decode TP size. |
| 301 | |
| 302 | ## Notes |
| 303 | |
| 304 | ### Reproducible Block Hashes Across Processes |
| 305 | |
| 306 | The `MooncakeStoreConnector` relies on consistent block hashes across all vLLM processes sharing the distributed store. Block hashes chain from `NONE_HASH`, which is derived from a fixed default seed, so identical prompts produce identical block hashes across processes by default — enabling cross-process prefix cache hits without extra configuration. |
| 307 | |
| 308 | The exception is the non-cryptographic `xxhash`/`xxhash_cbor` values of `--prefix-caching-hash-algo`, which seed `NONE_HASH` randomly per process; sharing a store with those requires `PYTHONHASHSEED`. |
| 309 | |
| 310 | To use a custom shared seed, set the same `PYTHONHASHSEED` on every instance that shares the store (DP ranks, separate prefiller/decoder nodes, and any other vLLM process pointed at the same Mooncake store): |
| 311 | |
| 312 | ```bash |
| 313 | PYTHONHASHSEED=<shared-value> vllm serve ... |
| 314 | ``` |