@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 264 lines 10.8 KB
Rendered Raw
1 # MoRIIOConnector Usage Guide
2
3 `MoRIIOConnector` is a high-performance KV connector used for KV cache transfer in PD disaggregated deployments, built on ROCm's [MoRI-IO](https://github.com/rocm/mori) communication library for point-to-point communication with ultra-low overhead.
4
5 ## Prerequisites
6
7 ### Installation
8
9 **Docker:** MoRI is shipped with the official ROCm vLLM image: `vllm/vllm-openai-rocm:nightly`.
10
11 **Manual installation:** MoRI wheel can be installed with
12
13 ```bash
14 pip install amd_mori
15 ```
16
17 Refer to the [Dockerfile.rocm_base](../../docker/Dockerfile.rocm_base) for more information, or [official MoRI repository](https://github.com/rocm/mori) for instructions on how to build MoRI from source.
18
19 For instructions on installing appropriate NIC userspace libraries, see [Installing NIC userspace libraries](#appendix-installing-nic-userspace-libraries).
20
21 ## Basic usage (single host)
22
23 Start the proxy first; the producer and consumer instances will retry registration until the proxy is reachable.
24
25 ### Producer (prefiller) configuration
26
27 Start a prefiller instance that produces KV caches
28
29 ```bash
30 # Prefill instance (GPU 0-3)
31 export VLLM_ROCM_USE_AITER=1
32 export CUDA_VISIBLE_DEVICES=0,1,2,3
33 export HIP_VISIBLE_DEVICES=0,1,2,3
34
35 vllm serve Qwen/Qwen3-235B-A22B-FP8 \
36 -tp 4 \
37 --port 20005 \
38 --gpu-memory-utilization 0.9 \
39 --kv-transfer-config '{
40 "kv_connector": "MoRIIOConnector",
41 "kv_role": "kv_producer",
42 "kv_connector_extra_config": {
43 "proxy_ip": "127.0.0.1",
44 "proxy_ping_port": "36367",
45 "http_port": "20005",
46 "handshake_port": "6301",
47 "notify_port": "6105"
48 }
49 }'
50 ```
51
52 ### Consumer (decoder) configuration
53
54 Start a decoder instance that consumes KV caches:
55
56 ```bash
57 # Decode instance (GPU 4-7)
58 export VLLM_ROCM_USE_AITER=1
59 export CUDA_VISIBLE_DEVICES=4,5,6,7
60 export HIP_VISIBLE_DEVICES=4,5,6,7
61
62 vllm serve Qwen/Qwen3-235B-A22B-FP8 \
63 -tp 4 \
64 --port 40005 \
65 --gpu-memory-utilization 0.9 \
66 --kv-transfer-config '{
67 "kv_connector": "MoRIIOConnector",
68 "kv_role": "kv_consumer",
69 "kv_connector_extra_config": {
70 "proxy_ip": "127.0.0.1",
71 "http_port": "40005",
72 "proxy_ping_port": "36367",
73 "handshake_port": "7301",
74 "notify_port": "7501"
75 }
76 }'
77 ```
78
79 ### Proxy server
80
81 The proxy fronts the producer and consumer instances and routes incoming requests to them. `vllm-router` is the recommended proxy; it can be installed manually or run as a Docker container. Note that the port `36367` below is the `proxy_ping_port` configured on each vLLM instance.
82
83 **Docker:**
84
85 ```bash
86 docker run \
87 --network host \
88 vllm/vllm-router:nightly \
89 vllm-router \
90 --vllm-pd-disaggregation \
91 --kv-connector moriio \
92 --vllm-discovery-address "0.0.0.0:36367"
93 ```
94
95 **Manual install:**
96
97 ```bash
98 pip install vllm-router
99 vllm-router \
100 --vllm-pd-disaggregation \
101 --kv-connector moriio \
102 --vllm-discovery-address "0.0.0.0:36367"
103 ```
104
105 Alternatively, you can use the reference implementation proxy shipped with vLLM:
106
107 ```bash
108 cd <path_to>/vllm
109 pip install quart aiohttp msgpack
110 python examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
111 ```
112
113 ## Configuration
114
115 The connector is configured at two levels: the application level and the transport level.
116
117 ### Application-level configuration
118
119 **Modes:** MoRI has two modes of operation: WRITE and READ mode.
120
121 - In WRITE mode, the producer actively pushes computed KV blocks after every layer into the consumer's memory.
122 - In READ mode, the consumer pulls the KV blocks from the producer all at once, as soon as it has been notified those blocks are ready.
123
124 WRITE mode is used by default. READ mode can be configured by setting `--kv-transfer-config.kv_connector_extra_config.read_mode true`.
125
126 **Control-plane configuration:** MoRI moves KV bytes over RDMA/xGMI, but producers and consumers also need out-of-band TCP channels for handshake, block id exchange, liveness, and completion signaling. These keys live under `kv_connector_extra_config`:
127
128 - `proxy_ip`: IP address of the disaggregation proxy/router that fronts the prefiller and decoder. Each vLLM instance uses it to register itself and to send heartbeats so the proxy knows where to route incoming requests.
129 - `proxy_ping_port`: TCP port on `proxy_ip` where the proxy listens for instance heartbeats and registration messages. Used to detect dead vLLM instances and keep routing tables fresh.
130 - `http_port`: HTTP port that this vLLM instance exposes its OpenAI-compatible API on. The proxy registers this port, and forwards user requests to this port once it has picked an instance.
131 - `handshake_port`: TCP port used for the one-time MoRI engine handshake between a prefiller and a decoder. The two sides exchange RDMA engine descriptors here before any KV transfer can happen.
132 - `notify_port`: TCP port used for control and synchronization messages between prefiller and decoder. Used differently in the two modes:
133 - WRITE mode: **Block allocation:** the decoder notifies the prefiller about its block ids, so the prefiller can push its computed KV blocks into the correct place on the decoder instance. **Completion:** once all blocks have been transferred, the prefiller notifies the decoder that it's safe to use its blocks.
134 - READ mode: **Completion:** once the decoder has read all blocks from the prefiller, it notifies the prefiller so it can free its KV cache blocks.
135
136 !!! note
137 `notify_port` is used as a *base* port: each (DP rank, TP rank) pair within an instance uses `notify_port + offset` where the offset is based on the rank. Make sure the range starting at `notify_port` is free on the host.
138
139 ### Transport configuration
140
141 MoRI has two transport backends: RDMA and xGMI. You can select backend using `--kv-transfer-config.kv_connector_extra_config.backend $BACKEND`, with `$BACKEND` being `rdma` or `xgmi`. RDMA is the default backend and should be used in multi-node deployments.
142
143 The configuration options for each backend are as follows.
144
145 #### RDMA backend
146
147 - `qp_per_transfer`: number of RDMA Queue Pairs (QPs) used per transfer. More QPs let a single transfer be striped over multiple QPs to increase NIC concurrency, at the cost of more RDMA resources.
148 - `post_batch_size`: how many RDMA Work Requests (WR) are batched into one `ibv_post_send` doorbell. Defaults to -1, meaning the backend default. Larger batches reduce the posting overhead per WR.
149 - `num_workers`: number of worker threads MoRI uses to post and poll transfer completions.
150
151 Advanced users can also configure MoRI itself using environment variables such as `MORI_IO_QP_MAX_SEND_WR`, `MORI_IO_QP_MAX_CQE`, etc. These are MoRI library variables and are separate from vLLM's own `VLLM_MORIIO_*` settings. Refer to the [MoRI repository](https://github.com/rocm/mori) for more information.
152
153 #### xGMI backend
154
155 Use xGMI when the prefiller and decoder run on the same physical host so transfers go over the AMD GPU fabric and skip the NIC entirely. Currently only configured using MoRI-specific environment variables; see the [MoRI repository](https://github.com/rocm/mori).
156
157 ## Multi-node deployment
158
159 The example below shows how to run a 1P1D deployment on two nodes. We run the proxy on the same node as the prefill instance.
160
161 ### On both nodes
162
163 ```bash
164 # Set on both nodes before running any command
165 export PREFILL_IP=<node1-ip>
166 export DECODE_IP=<node2-ip>
167 ```
168
169 ### On node 1
170
171 Start the proxy first as described in [Proxy server](#proxy-server), then start the prefill instance:
172
173 ```bash
174 docker run \
175 --name moriio-prefill \
176 --init --network host --ipc host --privileged \
177 --security-opt seccomp=unconfined \
178 --ulimit memlock=-1 --ulimit stack=67108864 --shm-size 256G \
179 --group-add video --group-add render \
180 --device /dev/kfd --device /dev/dri --device /dev/infiniband \
181 -e VLLM_ROCM_USE_AITER=1 \
182 vllm/vllm-openai-rocm:nightly \
183 deepseek-ai/DeepSeek-R1-0528 \
184 --port 8100 \
185 --tensor-parallel-size 8 \
186 --enable-expert-parallel \
187 --gpu-memory-utilization 0.8 \
188 --trust-remote-code \
189 --kv-transfer-config '{
190 "kv_connector": "MoRIIOConnector",
191 "kv_role": "kv_producer",
192 "kv_connector_extra_config": {
193 "proxy_ip": "'"${PREFILL_IP}"'",
194 "proxy_ping_port": "36367",
195 "http_port": "8100",
196 "handshake_port": "6301",
197 "notify_port": "61005"
198 }
199 }'
200 ```
201
202 ### On node 2
203
204 Decode instance:
205
206 ```bash
207 docker run \
208 --name moriio-decode \
209 --init --network host --ipc host --privileged \
210 --security-opt seccomp=unconfined \
211 --ulimit memlock=-1 --ulimit stack=67108864 --shm-size 256G \
212 --group-add video --group-add render \
213 --device /dev/kfd --device /dev/dri --device /dev/infiniband \
214 -e VLLM_ROCM_USE_AITER=1 \
215 vllm/vllm-openai-rocm:nightly \
216 deepseek-ai/DeepSeek-R1-0528 \
217 --port 8200 \
218 --tensor-parallel-size 8 \
219 --gpu-memory-utilization 0.8 \
220 --trust-remote-code \
221 --enable-expert-parallel \
222 --kv-transfer-config '{
223 "kv_connector": "MoRIIOConnector",
224 "kv_role": "kv_consumer",
225 "kv_connector_extra_config": {
226 "proxy_ip": "'"${PREFILL_IP}"'",
227 "proxy_ping_port": "36367",
228 "http_port": "8200",
229 "handshake_port": "6301",
230 "notify_port": "61005"
231 }
232 }'
233 ```
234
235 ## Troubleshooting
236
237 ### `availDevices.size() > 0` assertion failure
238
239 **Problem:** vLLM fails to launch with the following log:
240
241 ```bash
242 libibverbs: Warning: Driver bnxt_re does not support the kernel ABI of 6 (supports 1 to 1) for device /sys/class/infiniband/rdma4
243 ...
244 ker: /app/mori/src/io/rdma/backend_impl.cpp: mori::io::RdmaManager::RdmaManager(const RdmaBackendConfig, application::RdmaContext *): Assertion `availDevices.size() > 0' failed.
245 ```
246
247 **Fix:** The installed RDMA userspace libraries do not match the driver and firmware version installed on the host. You must install NIC userspace libraries corresponding to your RDMA kernel module and firmware version. See [Installing NIC userspace
248 libraries](#appendix-installing-nic-userspace-libraries) for more information.
249
250 ## Appendix: installing NIC userspace libraries
251
252 To run MoRI with RDMA, your environment must have the necessary RDMA userspace libraries installed that match the associated kernel module and firmware version.
253
254 The official image `vllm/vllm-openai-rocm:nightly` comes pre-installed with userspace libraries for the following NICs and kernel module versions:
255
256 - AINIC (AMD Pensando Pollara): version `1.117.5-a-77`, ships `libionic1=54.0-187-1`, tested with `ionic-dkms=26.03.3.001`
257 - Thor2 (Broadcom): version `235.2.86.0`, tested with `bnxt-en-dkms=1.10.3.235.2.86.0`, `bnxt-re-dkms=235.2.86.0`
258
259 Refer to [Dockerfile.rocm](../../docker/Dockerfile.rocm) for more details. For users with NICs, kernel modules, and/or FW other than those stated above we refer to
260 the vendors' own installation instructions.
261
262 ## Further reading
263
264 - [Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode Disaggregation](https://vllm.ai/blog/2026-04-07-moriio-kv-connector).