@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 158 lines 6.04 KB
Rendered Raw
1 # ECMooncakeConnector Usage Guide
2
3 `ECMooncakeConnector` moves multimodal **encoder embeddings** (encoder cache)
4 between a disaggregated encoder instance and a language-model instance over
5 [Mooncake Transfer Engine](https://github.com/kvcache-ai/Mooncake). Use it
6 when the vision encoder should scale independently of prefill and decode.
7
8 For KV-cache transfer between prefiller and decoder, see
9 [MooncakeConnector Usage Guide](mooncake_connector_usage.md). That path uses
10 `MooncakeConnector`, not `ECMooncakeConnector`.
11
12 Added in [PR #41567](https://github.com/vllm-project/vllm/pull/41567).
13 Overview of disaggregated encoding:
14 [Disaggregated Encoder](disagg_encoder.md).
15
16 !!! important
17 Static proxy flags (`--ec-consumer-zmq-addrs`) support Mooncake EC only in
18 **1E + 1PD** mode (`--prefill-servers-urls disable`). For 1E + 1P + 1D,
19 register `ec_zmq_addrs` on the **prefill** consumer through the proxy's
20 dynamic registration API instead of the static CLI flag. See
21 `examples/disaggregated/disaggregated_encoder/README.md`.
22
23 ## Prerequisites
24
25 Install Mooncake Transfer Engine:
26
27 ```bash
28 # CUDA 13 (vLLM default)
29 uv pip install mooncake-transfer-engine-cuda13
30
31 # CUDA 12
32 uv pip install mooncake-transfer-engine
33 ```
34
35 The two wheels are the same release built against different CUDA majors.
36 Importing the wrong one fails with
37 `libcudart.so.<major>: cannot open shared object file`.
38
39 ## 1E + 1PD
40
41 One encoder producer and one combined prefill/decode consumer. The consumer
42 listens on `ec_ip` + `ec_port` (ZMQ control). The proxy must advertise that
43 address with `--ec-consumer-zmq-addrs`.
44
45 ### Encoder (producer)
46
47 The producer must run with `tensor_parallel_size=1`,
48 `pipeline_parallel_size=1`, and `data_parallel_size=1`.
49
50 ```bash
51 CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
52 --port 19534 \
53 --gpu-memory-utilization 0.35 \
54 --enable-request-id-headers \
55 --no-enable-prefix-caching \
56 --max-num-batched-tokens 16384 \
57 --allowed-local-media-path /path/to/media \
58 --ec-transfer-config '{
59 "ec_connector": "ECMooncakeConnector",
60 "ec_role": "ec_producer",
61 "ec_connector_extra_config": {
62 "mooncake_protocol": "rdma"
63 }
64 }'
65 ```
66
67 ### Prefill+Decode (consumer)
68
69 ```bash
70 CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
71 --port 19537 \
72 --gpu-memory-utilization 0.75 \
73 --enable-mm-embeds \
74 --enable-request-id-headers \
75 --no-enable-prefix-caching \
76 --allowed-local-media-path /path/to/media \
77 --ec-transfer-config '{
78 "ec_connector": "ECMooncakeConnector",
79 "ec_role": "ec_consumer",
80 "ec_ip": "127.0.0.1",
81 "ec_port": 19019,
82 "ec_connector_extra_config": {
83 "mooncake_protocol": "rdma"
84 }
85 }'
86 ```
87
88 `--enable-mm-embeds` lets the PD instance inject transferred embeddings
89 instead of re-running the vision encoder.
90
91 ### Proxy
92
93 ```bash
94 python examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py \
95 --host 0.0.0.0 \
96 --port 10002 \
97 --encode-servers-urls http://127.0.0.1:19534 \
98 --prefill-servers-urls disable \
99 --decode-servers-urls http://127.0.0.1:19537 \
100 --ec-consumer-zmq-addrs tcp://127.0.0.1:19019
101 ```
102
103 `--ec-consumer-zmq-addrs` must align with `--decode-servers-urls` (the
104 combined PD instances). For data-parallel consumers, list each replica
105 consecutively and set `--ec-consumer-dp-size`.
106
107 ### Correctness script
108
109 ```bash
110 # TCP (no RDMA device required; CI default)
111 MOONCAKE_EC_PROTOCOL=tcp \
112 bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
113
114 # RDMA
115 MOONCAKE_EC_PROTOCOL=rdma \
116 bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
117 ```
118
119 The TCP path sets `MC_FORCE_TCP=1` so Transfer Engine does not auto-select
120 RDMA when an HCA is present.
121
122 ## Configuration
123
124 Producer and consumer must use the same `mooncake_protocol`.
125
126 | Setting | Where | Default | Notes |
127 | --------- | ------- | --------- | ------- |
128 | `mooncake_protocol` | `ec_connector_extra_config` | `rdma` | Set `tcp` when no verbs device is available. With TCP, also export `MC_FORCE_TCP=1`. |
129 | `ec_ip` / `ec_port` | consumer `ECTransferConfig` | `127.0.0.1` / `14579` | First ZMQ control port. Tensor-parallel rank `r` listens on `ec_port + dp_index * tp_size + r`. |
130 | `ec_buffer_size` | `ECTransferConfig` | `1e9` | Registered receive arena on the consumer, in bytes. |
131 | `control_timeout_s` | `ec_connector_extra_config` | `30` | Reservation RPC timeout. |
132 | `push_wait_timeout_s` | `ec_connector_extra_config` | `60` | How long the consumer waits for a matching encoder push. |
133 | `WITH_NVIDIA_PEERMEM` | process environment | `1` | Read by Mooncake. Set `0` on hosts without `nvidia-peermem` (DMA-BUF path). |
134 | `--ec-consumer-zmq-addrs` | proxy | empty | `tcp://<ec_ip>:<ec_port>` for each PD replica. Required for Mooncake EC. |
135
136 ## 1E + 1P + 1D
137
138 The filesystem example is `examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh`
139 (`ECExampleConnector` for encoder cache, `NixlConnector` for KV). P→D KV can
140 use `MooncakeConnector` instead; see
141 [MooncakeConnector Usage Guide](mooncake_connector_usage.md).
142
143 Do **not** pass `--ec-consumer-zmq-addrs` together with independent prefill
144 URLs: the static proxy flag is E+PD only. For Mooncake EC with a dedicated
145 prefill, use dynamic registration and set `ec_zmq_addrs` on the prefill
146 consumer as described in
147 `examples/disaggregated/disaggregated_encoder/README.md`.
148
149 ## Troubleshooting
150
151 | Symptom | Likely cause |
152 | --------- | ---------------- |
153 | `Install mooncake-transfer-engine` / `libcudart.so.*` | Wrong or missing Mooncake wheel for this CUDA major. |
154 | `Failed to register memory ... Bad address [14]` | GPUDirect registration failed. Set `WITH_NVIDIA_PEERMEM=0`. |
155 | Proxy startup error about E+PD only | `--ec-consumer-zmq-addrs` used with a real `--prefill-servers-urls`. |
156 | Consumer waits then errors; encoder never finishes the push | `--ec-consumer-zmq-addrs` does not match consumer `ec_ip`/`ec_port`, or producer/consumer protocols differ. |
157 | TCP transfers stall on a host that also has RDMA | Export `MC_FORCE_TCP=1`. |
158 | Producer rejected at startup | Encoder used tensor, pipeline, or data parallelism. Keep the producer unsharded. |