[Doc] Document ECMooncakeConnector for EPD disaggregation (#59196)
Signed-off-by: Teng Ma <stmatengss@gmail.com> Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Teng Ma committed
Oct 2, 2026 at 12:10 UTC
1a9eaa3a686233610b75b6cc7605500f8903c9b9
5 files changed
+174
-5
docs/features/disagg_encoder.md
+4
-3
@@ -45,9 +45,10 @@ The following scripts demonstrate the workflow with **ExampleConnector**:
45
### ECMooncakeConnector
46
47
**ECMooncakeConnector** transfers encoder outputs using the Mooncake TransferEngine.
48
-See the [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
49
-for a complete 1 Encoder instance + 1 PD instance setup, including the producer and
50
-consumer `--ec-transfer-config` settings and proxy configuration.
48
+See the [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md) for
49
+producer/consumer `--ec-transfer-config`, proxy `--ec-consumer-zmq-addrs`, and
50
+transport options. The [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
51
+is a complete 1 Encoder instance + 1 PD instance setup.
52
53
With vLLM and Mooncake installed, run the example from the repository root:
54
docs/features/disagg_prefill.md
+1
-1
@@ -27,7 +27,7 @@ Now supports 9 types of connectors:
27
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_buffer_device":"cuda", "kv_connector_extra_config":{"backends":["UCX", "GDS"]}}'
28
```
29
30
-- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md).
30
+- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md). For multimodal encoder-cache transfer, see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
31
- **MoRIIOConnector** (ROCm only): see [MoRI-IO Usage Guide](moriio_connector_usage.md) for example usage and detailed documentation.
32
- **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
33
docs/features/mooncake_connector_usage.md
+3
@@ -38,6 +38,9 @@ python examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py --p
38
39
Now you can send requests to the proxy server through port 8000.
40
41
+For multimodal Encode-Prefill-Decode (encoder embeddings rather than KV
42
+cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
43
+
44
## Environment Variables
45
46
- `VLLM_MOONCAKE_BOOTSTRAP_PORT`: Port for Mooncake bootstrap server
docs/features/mooncake_ec_connector_usage.md
new
+158
@@ -0,0 +1,158 @@
1
+# ECMooncakeConnector Usage Guide
2
+
3
+`ECMooncakeConnector` moves multimodal **encoder embeddings** (encoder cache)
4
+between a disaggregated encoder instance and a language-model instance over
5
+[Mooncake Transfer Engine](https://github.com/kvcache-ai/Mooncake). Use it
6
+when the vision encoder should scale independently of prefill and decode.
7
+
8
+For KV-cache transfer between prefiller and decoder, see
9
+[MooncakeConnector Usage Guide](mooncake_connector_usage.md). That path uses
10
+`MooncakeConnector`, not `ECMooncakeConnector`.
11
+
12
+Added in [PR #41567](https://github.com/vllm-project/vllm/pull/41567).
13
+Overview of disaggregated encoding:
14
+[Disaggregated Encoder](disagg_encoder.md).
15
+
16
+!!! important
17
+ Static proxy flags (`--ec-consumer-zmq-addrs`) support Mooncake EC only in
18
+ **1E + 1PD** mode (`--prefill-servers-urls disable`). For 1E + 1P + 1D,
19
+ register `ec_zmq_addrs` on the **prefill** consumer through the proxy's
20
+ dynamic registration API instead of the static CLI flag. See
21
+ `examples/disaggregated/disaggregated_encoder/README.md`.
22
+
23
+## Prerequisites
24
+
25
+Install Mooncake Transfer Engine:
26
+
27
+```bash
28
+# CUDA 13 (vLLM default)
29
+uv pip install mooncake-transfer-engine-cuda13
30
+
31
+# CUDA 12
32
+uv pip install mooncake-transfer-engine
33
+```
34
+
35
+The two wheels are the same release built against different CUDA majors.
36
+Importing the wrong one fails with
37
+`libcudart.so.<major>: cannot open shared object file`.
38
+
39
+## 1E + 1PD
40
+
41
+One encoder producer and one combined prefill/decode consumer. The consumer
42
+listens on `ec_ip` + `ec_port` (ZMQ control). The proxy must advertise that
43
+address with `--ec-consumer-zmq-addrs`.
44
+
45
+### Encoder (producer)
46
+
47
+The producer must run with `tensor_parallel_size=1`,
48
+`pipeline_parallel_size=1`, and `data_parallel_size=1`.
49
+
50
+```bash
51
+CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
52
+ --port 19534 \
53
+ --gpu-memory-utilization 0.35 \
54
+ --enable-request-id-headers \
55
+ --no-enable-prefix-caching \
56
+ --max-num-batched-tokens 16384 \
57
+ --allowed-local-media-path /path/to/media \
58
+ --ec-transfer-config '{
59
+ "ec_connector": "ECMooncakeConnector",
60
+ "ec_role": "ec_producer",
61
+ "ec_connector_extra_config": {
62
+ "mooncake_protocol": "rdma"
63
+ }
64
+ }'
65
+```
66
+
67
+### Prefill+Decode (consumer)
68
+
69
+```bash
70
+CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
71
+ --port 19537 \
72
+ --gpu-memory-utilization 0.75 \
73
+ --enable-mm-embeds \
74
+ --enable-request-id-headers \
75
+ --no-enable-prefix-caching \
76
+ --allowed-local-media-path /path/to/media \
77
+ --ec-transfer-config '{
78
+ "ec_connector": "ECMooncakeConnector",
79
+ "ec_role": "ec_consumer",
80
+ "ec_ip": "127.0.0.1",
81
+ "ec_port": 19019,
82
+ "ec_connector_extra_config": {
83
+ "mooncake_protocol": "rdma"
84
+ }
85
+ }'
86
+```
87
+
88
+`--enable-mm-embeds` lets the PD instance inject transferred embeddings
89
+instead of re-running the vision encoder.
90
+
91
+### Proxy
92
+
93
+```bash
94
+python examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py \
95
+ --host 0.0.0.0 \
96
+ --port 10002 \
97
+ --encode-servers-urls http://127.0.0.1:19534 \
98
+ --prefill-servers-urls disable \
99
+ --decode-servers-urls http://127.0.0.1:19537 \
100
+ --ec-consumer-zmq-addrs tcp://127.0.0.1:19019
101
+```
102
+
103
+`--ec-consumer-zmq-addrs` must align with `--decode-servers-urls` (the
104
+combined PD instances). For data-parallel consumers, list each replica
105
+consecutively and set `--ec-consumer-dp-size`.
106
+
107
+### Correctness script
108
+
109
+```bash
110
+# TCP (no RDMA device required; CI default)
111
+MOONCAKE_EC_PROTOCOL=tcp \
112
+ bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
113
+
114
+# RDMA
115
+MOONCAKE_EC_PROTOCOL=rdma \
116
+ bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
117
+```
118
+
119
+The TCP path sets `MC_FORCE_TCP=1` so Transfer Engine does not auto-select
120
+RDMA when an HCA is present.
121
+
122
+## Configuration
123
+
124
+Producer and consumer must use the same `mooncake_protocol`.
125
+
126
+| Setting | Where | Default | Notes |
127
+| --------- | ------- | --------- | ------- |
128
+| `mooncake_protocol` | `ec_connector_extra_config` | `rdma` | Set `tcp` when no verbs device is available. With TCP, also export `MC_FORCE_TCP=1`. |
129
+| `ec_ip` / `ec_port` | consumer `ECTransferConfig` | `127.0.0.1` / `14579` | First ZMQ control port. Tensor-parallel rank `r` listens on `ec_port + dp_index * tp_size + r`. |
130
+| `ec_buffer_size` | `ECTransferConfig` | `1e9` | Registered receive arena on the consumer, in bytes. |
131
+| `control_timeout_s` | `ec_connector_extra_config` | `30` | Reservation RPC timeout. |
132
+| `push_wait_timeout_s` | `ec_connector_extra_config` | `60` | How long the consumer waits for a matching encoder push. |
133
+| `WITH_NVIDIA_PEERMEM` | process environment | `1` | Read by Mooncake. Set `0` on hosts without `nvidia-peermem` (DMA-BUF path). |
134
+| `--ec-consumer-zmq-addrs` | proxy | empty | `tcp://<ec_ip>:<ec_port>` for each PD replica. Required for Mooncake EC. |
135
+
136
+## 1E + 1P + 1D
137
+
138
+The filesystem example is `examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh`
139
+(`ECExampleConnector` for encoder cache, `NixlConnector` for KV). P→D KV can
140
+use `MooncakeConnector` instead; see
141
+[MooncakeConnector Usage Guide](mooncake_connector_usage.md).
142
+
143
+Do **not** pass `--ec-consumer-zmq-addrs` together with independent prefill
144
+URLs: the static proxy flag is E+PD only. For Mooncake EC with a dedicated
145
+prefill, use dynamic registration and set `ec_zmq_addrs` on the prefill
146
+consumer as described in
147
+`examples/disaggregated/disaggregated_encoder/README.md`.
148
+
149
+## Troubleshooting
150
+
151
+| Symptom | Likely cause |
152
+| --------- | ---------------- |
153
+| `Install mooncake-transfer-engine` / `libcudart.so.*` | Wrong or missing Mooncake wheel for this CUDA major. |
154
+| `Failed to register memory ... Bad address [14]` | GPUDirect registration failed. Set `WITH_NVIDIA_PEERMEM=0`. |
155
+| Proxy startup error about E+PD only | `--ec-consumer-zmq-addrs` used with a real `--prefill-servers-urls`. |
156
+| Consumer waits then errors; encoder never finishes the push | `--ec-consumer-zmq-addrs` does not match consumer `ec_ip`/`ec_port`, or producer/consumer protocols differ. |
157
+| TCP transfers stall on a host that also has RDMA | Export `MC_FORCE_TCP=1`. |
158
+| Producer rejected at startup | Encoder used tensor, pipeline, or data parallelism. Keep the producer unsharded. |
examples/disaggregated/disaggregated_encoder/README.md
+8
-1
@@ -57,7 +57,12 @@ The vllm instances and `disagg_encoder_proxy` supports local URIs with ```{"url"
57
58
## EC connector and KV transfer
59
60
-The `ECExampleonnector` is used to store the encoder cache on local disk and facilitate transfer. To enable the encoder disaggregation feature, add the following configuration:
60
+`ECExampleConnector` stores encoder cache on local disk. For RDMA/TCP transfer
61
+without a shared filesystem, use `ECMooncakeConnector` (1E + 1PD only through
62
+this proxy). See
63
+[ECMooncakeConnector Usage Guide](../../../docs/features/mooncake_ec_connector_usage.md).
64
+
65
+To enable filesystem encoder disaggregation, add:
66
67
```bash
68
# Add to encoder instance:
@@ -104,6 +109,8 @@ If you enable prefill instance (`--prefill-servers-urls` not disabled), you will
109
| `--encode-servers-urls` | Comma-separated list of encoder endpoints. Every multimodal item extracted from the request is fanned out to one of these URLs in a round-robin fashion. |
110
| `--prefill-servers-urls` | Comma-separated list of prefill endpoints. Set to `disable`, `none`, or `""` to skip the dedicated prefill phase and run E+PD (encoder + combined prefill/decode). |
111
| `--decode-servers-urls` | Comma-separated list of decode endpoints. Non-stream and stream paths both round-robin over this list. |
112
+| `--ec-consumer-zmq-addrs` | Mooncake EC consumer control addresses, aligned with `--decode-servers-urls`. Required for `ECMooncakeConnector`. Supported only in E+PD mode (`--prefill-servers-urls disable`). |
113
+| `--ec-consumer-dp-size` | Data-parallel replicas per EC consumer (default 1). |
114
| `--host`, `--port` | Bind address for the proxy itself (defaults: `0.0.0.0:8000`). |
115
116
### Dynamic registration