A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 3, 06:45 UTC
[Docs] Fix MooncakeStoreConnector role in disaggregated example (#42994)
Signed-off-by: Dao Le <Dao007forever@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
Dao007forever committed
May 19, 2026 at 11:14 UTC
aed2eb355a9d9136c8e17690b932983b55fb343f
1 file changed
+8
-8
docs/features/mooncake_store_connector_usage.md
+8
-8
@@ -101,7 +101,7 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct \
101
},
102
{
103
"kv_connector": "MooncakeStoreConnector",
104
- "kv_role": "kv_producer"
104
+ "kv_role": "kv_both"
105
}
106
]
107
}
@@ -196,9 +196,9 @@ the vLLM JSON config.
196
197
### KV Role Options
198
199
-- **kv_producer**: For prefiller instances that store KV caches to the pool.
200
-- **kv_consumer**: For decoder instances that load KV caches from the pool.
201
-- **kv_both**: The instance both stores and loads KV caches. Use this for single-node CPU offloading.
199
+- **kv_producer**: For instances that store KV caches to the pool.
200
+- **kv_consumer**: For instances that load KV caches from the pool.
201
+- **kv_both**: The instance both stores and loads KV caches. Use this for single-node CPU offloading or prefiller instances.
202
203
### kv_connector_extra_config
204
@@ -209,12 +209,12 @@ the vLLM JSON config.
209
210
## Notes
211
212
-### Cross-DP Prefix Cache Hits
212
+### Reproducible Block Hashes Across Processes
213
+
214
+The `MooncakeStoreConnector` relies on consistent block hashes across all vLLM processes sharing the distributed store. Because Python randomizes its hash seed per process by default, identical prompts can produce different block hashes on different processes — preventing cross-process prefix cache hits.
215
214
-When running with data parallelism, set a fixed `PYTHONHASHSEED` so that block hashes are consistent across DP ranks:
216
+Set a fixed `PYTHONHASHSEED` on every instance that shares the store (DP ranks, separate prefiller/decoder nodes, and any other vLLM process pointed at the same Mooncake store):
217
218
```bash
219
PYTHONHASHSEED=0 vllm serve ...
220
```
219
-
220
-Without this, identical prompts may produce different block hashes on different DP ranks, preventing cross-instance prefix cache hits.