@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 89 lines 5.6 KB
Rendered Raw
1 # MooncakeConnector Usage Guide
2
3 ## About Mooncake
4
5 Mooncake aims to enhance the inference efficiency of large language models (LLMs), especially in slow object storage environments, by constructing a multi-level caching pool on high-speed interconnected DRAM/SSD resources. Compared to traditional caching systems, Mooncake utilizes (GPUDirect) RDMA technology to transfer data directly in a zero-copy manner, while maximizing the use of multi-NIC resources on a single machine.
6
7 For more details about Mooncake, please refer to [Mooncake project](https://github.com/kvcache-ai/Mooncake) and [Mooncake documents](https://kvcache-ai.github.io/Mooncake/).
8
9 ## Prerequisites
10
11 ### Installation
12
13 Install mooncake through pip: `uv pip install mooncake-transfer-engine-cuda13`.
14
15 vLLM defaults to CUDA 13. On a CUDA 12 environment install `mooncake-transfer-engine` instead — the two are the same release built against different CUDA majors, and the wrong one fails to import with `libcudart.so.<major>: cannot open shared object file`.
16
17 Refer to [Mooncake official repository](https://github.com/kvcache-ai/Mooncake) for more installation instructions
18
19 ## Usage
20
21 ### Prefiller Node (192.168.0.2)
22
23 ```bash
24 vllm serve Qwen/Qwen2.5-7B-Instruct --port 8010 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer"}'
25 ```
26
27 ### Decoder Node (192.168.0.3)
28
29 ```bash
30 vllm serve Qwen/Qwen2.5-7B-Instruct --port 8020 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer"}'
31 ```
32
33 ### Proxy
34
35 ```bash
36 python examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py --prefill http://192.168.0.2:8010 --decode http://192.168.0.3:8020
37 ```
38
39 Now you can send requests to the proxy server through port 8000.
40
41 For multimodal Encode-Prefill-Decode (encoder embeddings rather than KV
42 cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
43
44 ## Environment Variables
45
46 - `VLLM_MOONCAKE_BOOTSTRAP_PORT`: Port for Mooncake bootstrap server
47 - Default: 8998
48 - Required only for prefiller instances
49 - For headless instances, must be the same as the master instance
50 - Each instance needs a unique port on its host; using the same port number across different hosts is fine
51
52 - `VLLM_MOONCAKE_CONNECTOR_TIMEOUT`: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup. (Optional)
53 - Default: 30.0
54 - Global rank 0 hosts the bootstrap server in the same process that mounts the Mooncake transfer engine's host segment, so a large `MOONCAKE_GLOBAL_SEGMENT_SIZE` can delay responses for several seconds during startup
55 - Raise this if bootstrap server timeout warnings appear on hosts with very large host-memory segments
56 - Bootstrap registration uses up to 3 attempts internally, with exponential backoff capped at 10 seconds; connection errors and timeouts are both retried
57 - Once the attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging
58
59 - `WITH_NVIDIA_PEERMEM`: Selects how mooncake registers GPU memory for RDMA. Read by mooncake, not vLLM.
60 - Default: 1, which uses `ibv_reg_mr()` and requires the `nvidia-peermem` kernel module to be loaded
61 - Set to 0 to use the DMA-BUF path, which does not need that module. Required on hosts where `nvidia-peermem` is not loaded, such as GB200
62 - With the container image, pass it at run time: `docker run -e WITH_NVIDIA_PEERMEM=0 ...`
63 - Symptom when left unset on such a host: `Failed to register memory <addr>: Bad address [14]` from `rdma_context.cpp`, and KV transfers fail
64
65 - `VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT`: Timeout (in seconds) for automatically releasing the prefiller’s KV cache for a particular request. (Optional)
66 - Default: 480
67 - If a request is aborted and the decoder has not yet notified the prefiller, the prefill instance will release its KV-cache blocks after this timeout to avoid holding them indefinitely.
68
69 ## KV Transfer Config
70
71 ### KV Role Options
72
73 - **kv_producer**: For prefiller instances that generate KV caches
74 - **kv_consumer**: For decoder instances that consume KV caches from prefiller
75 - **kv_both**: Enables symmetric functionality where the connector can act as both producer and consumer. This provides flexibility for experimental setups and scenarios where the role distinction is not predetermined.
76
77 ### kv_connector_extra_config
78
79 - **num_workers**: Size of thread pool for one prefiller worker to transfer KV caches by mooncake. (default 10)
80 - **mooncake_protocol**: Mooncake connector protocol. (default "rdma")
81 - **device_name**: Comma-separated whitelist of RDMA devices (e.g. `"mlx5_0,mlx5_1"`) to restrict topology discovery to. Empty discovers every device. Useful on hosts exposing a mix of InfiniBand and RoCE ports, where both peers must settle on the same link layer.
82 - **bootstrap_server_address**: Optional `"host:port"` of an already running Mooncake bootstrap server. All prefiller ranks register with this endpoint instead of launching an embedded server. The caller owns the server lifetime and must configure the proxy to use the same endpoint. The example proxy assumes the bootstrap server is on the prefiller API host; a different host requires a proxy that supplies that address in `remote_bootstrap_addr`. Omit this option to use the embedded server and `VLLM_MOONCAKE_BOOTSTRAP_PORT`.
83
84 ## Example Scripts/Code
85
86 Refer to these example scripts in the vLLM repository:
87
88 - [run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh)
89 - [mooncake_connector_proxy.py](../../examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py)