@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 225 lines 9.78 KB
Rendered Raw
1 # Disaggregated Encoder
2
3 These example scripts that demonstrate the disaggregated encoder (EPD) features of vLLM.
4
5 For a detailed explanation of the EPD features, please refer to the [Disaggregated Encoder Feature Documentation](../../../docs/features/disagg_encoder.md).
6
7 ## Files
8
9 - `disagg_epd_proxy.py` - Proxy script that demonstrates the XeYpZd setup (X encode instances, Y prefill instances, Z decode instances). Currently stable for the 1e1p1d configuration.
10
11 - `disagg_1e1p1d_example.sh` - Sets up the 1e1p1d configuration, runs the VisionArena benchmark, and processes a single request with a local image.
12
13 - `disagg_1e1pd_example.sh` - Sets up the 1e1pd configuration, runs the VisionArena benchmark, and processes a single request with a local image.
14
15 ### Custom Configuration
16
17 ```bash
18 # Use specific GPUs
19 GPU_E=0 GPU_PD=1 GPU_P=1 GPU_D=2 bash disagg_1e1p1d_example.sh
20
21 # Use specific ports
22 ENDPOINT_PORT=10001 bash disagg_1e1p1d_example.sh
23
24 # Use specific model
25 MODEL="Qwen/Qwen2.5-VL-3B-Instruct" bash disagg_1e1p1d_example.sh
26
27 # Use specific storage path
28 EC_SHARED_STORAGE_PATH="/tmp/my_ec_cache" bash disagg_1e1p1d_example.sh
29
30 # Run on XPU; scripts switch from CUDA_VISIBLE_DEVICES to ZE_AFFINITY_MASK
31 DEVICE_PLATFORM=xpu GPU_E=0 GPU_PD=1 bash disagg_1e1pd_example.sh
32 ```
33
34 `DEVICE_PLATFORM` defaults to `cuda`. Set `DEVICE_PLATFORM=xpu` when running these examples on Intel GPUs so the scripts use `ZE_AFFINITY_MASK` instead of `CUDA_VISIBLE_DEVICES` for device selection.
35
36 ## Encoder Instances
37
38 Encoder engines should be launched with the following flags:
39
40 - `--enforce-eager` **(required)** – The current EPD implementation is only compatible with encoder instances running in this mode.
41
42 - `--no-enable-prefix-caching` **(required)** – Encoder instances do not consume KV cache; prefix caching is disabled to avoid conflicts with other features.
43
44 - `--max-num-batched-tokens=<large value>` **(default: 2048)** – This flag controls the token scheduling budget per decoding step and is irrelevant to encoder-only instances. **Set it to a very high value (effectively unlimited) to bypass scheduler limitations.** The actual token budget is managed by the encoder cache manager.
45
46 - `--mm-encoder-only` **(Optional)** - If possible, skips the language model during initialization to reduce device memory usage.
47
48 ## Local media inputs
49
50 To support local image inputs (from your ```MEDIA_PATH``` directory), add the following flag to the encoder instance:
51
52 ```bash
53 --allowed-local-media-path $MEDIA_PATH
54 ```
55
56 The vllm instances and `disagg_encoder_proxy` supports local URIs with ```{"url": "file://'"$MEDIA_PATH_FILENAME"'}``` as multimodal inputs. Each URI is passed unchanged from the `disagg_encoder_proxy` to the encoder instance so that the encoder can load the media locally.
57
58 ## EC connector and KV transfer
59
60 `ECExampleConnector` stores encoder cache on local disk. For RDMA/TCP transfer
61 without a shared filesystem, use `ECMooncakeConnector` (1E + 1PD only through
62 this proxy). See
63 [ECMooncakeConnector Usage Guide](../../../docs/features/mooncake_ec_connector_usage.md).
64
65 To enable filesystem encoder disaggregation, add:
66
67 ```bash
68 # Add to encoder instance:
69 --ec-transfer-config '{
70 "ec_connector": "ECExampleConnector",
71 "ec_role": "ec_producer",
72 "ec_connector_extra_config": {
73 "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
74 }
75 }'
76
77 # Add to prefill/prefill+decode instance:
78 --ec-transfer-config '{
79 "ec_connector": "ECExampleConnector",
80 "ec_role": "ec_consumer",
81 "ec_connector_extra_config": {
82 "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
83 }
84 }'
85 ```
86
87 `$EC_SHARED_STORAGE_PATH` is the path where the EC connector temporarily stores the cache.
88
89 If you enable prefill instance (`--prefill-servers-urls` not disabled), you will need --kv-transfer-config to facilitate the PD disaggregation. Currently, we use the `NixlConnector` for this purpose. Refer to `tests/v1/kv_connector/nixl_integration` for more example codes on PD disaggregation with Nixl.
90
91 ```bash
92 # Add to prefill instance:
93 --kv-transfer-config '{
94 "kv_connector": "NixlConnector",
95 "kv_role": "kv_producer"
96 }'
97
98 # Add to decode instance:
99 --kv-transfer-config '{
100 "kv_connector": "NixlConnector",
101 "kv_role": "kv_consumer"
102 }'
103 ```
104
105 ## Proxy Instance Flags (`disagg_epd_proxy.py`)
106
107 | Flag | Description |
108 | ---- | ----------- |
109 | `--encode-servers-urls` | Comma-separated list of encoder endpoints. Every multimodal item extracted from the request is fanned out to one of these URLs in a round-robin fashion. |
110 | `--prefill-servers-urls` | Comma-separated list of prefill endpoints. Set to `disable`, `none`, or `""` to skip the dedicated prefill phase and run E+PD (encoder + combined prefill/decode). |
111 | `--decode-servers-urls` | Comma-separated list of decode endpoints. Non-stream and stream paths both round-robin over this list. |
112 | `--ec-consumer-zmq-addrs` | Mooncake EC consumer control addresses, aligned with `--decode-servers-urls`. Required for `ECMooncakeConnector`. Supported only in E+PD mode (`--prefill-servers-urls disable`). |
113 | `--ec-consumer-dp-size` | Data-parallel replicas per EC consumer (default 1). |
114 | `--host`, `--port` | Bind address for the proxy itself (defaults: `0.0.0.0:8000`). |
115
116 ### Dynamic registration
117
118 Alternatively, let the external launcher register ready instances over HTTP.
119 No vLLM configuration changes or worker registration threads are required.
120 Set `ADMIN_API_KEY` on the proxy and supply it as `X-API-Key` for registration
121 and removal. These are trusted control-plane APIs; do not expose them publicly.
122
123 ```bash
124 export ADMIN_API_KEY="your-admin-key"
125 python disagg_epd_proxy.py --port 8000 --dynamic-registration
126 ```
127
128 Omit the static server URL flags and register every stage. The roles determine
129 the topology: `encode` + `prefill_decode` gives E+PD; `encode` + `prefill` +
130 `decode` gives E+P+D. Standalone `decode` always requires an available `prefill`,
131 even when D registers first or all P instances go offline; otherwise requests
132 return `503`. The proxy rejects mixing combined PD with standalone P/D,
133 including unhealthy instances still in the registry. Explicitly remove the
134 old topology's P/D or PD registrations before switching topologies.
135
136 After each instance is ready, the launcher registers its reachable HTTP URL:
137
138 ```bash
139 curl --fail-with-body http://proxy-host:8000/instances \
140 -H "X-API-Key: $ADMIN_API_KEY" -H 'Content-Type: application/json' \
141 -d '{"role":"encode","url":"http://e-host:8001"}'
142 curl --fail-with-body http://proxy-host:8000/instances \
143 -H "X-API-Key: $ADMIN_API_KEY" -H 'Content-Type: application/json' \
144 -d '{"role":"prefill_decode","url":"http://pd-host:8002"}'
145 ```
146
147 For E+P+D, register P with `role: "prefill"` and D with `role: "decode"`.
148 Example and NIXL EC connectors need no additional registration fields.
149 For Mooncake, register `ec_zmq_addrs` on the **EC consumer** (`prefill_decode`
150 or `prefill`), using its configured `ec_ip` and `ec_port`. Keep the fixed-port
151 layout: supply the TP-rank-0 address for each DP replica, in DP-rank order;
152 replica `r` uses `ec_port + r * tensor_parallel_size`. The connector discovers
153 the remaining TP ranks itself. For example, DP=2, TP=2, `ec_port=19019`:
154
155 ```json
156 {
157 "role": "prefill_decode",
158 "url": "http://pd-host:8002",
159 "dp_size": 2,
160 "ec_zmq_addrs": ["tcp://pd-host:19019", "tcp://pd-host:19021"]
161 }
162 ```
163
164 The proxy selects one consumer replica and uses it for both the encoder push
165 and the consumer HTTP request. Standalone D does not need EC control addresses.
166 Port allocation and avoiding collisions remain the launcher's responsibility.
167
168 Inspect or remove instances without restarting the proxy:
169
170 ```bash
171 curl http://proxy-host:8000/instances
172 curl --fail-with-body -X DELETE \
173 'http://proxy-host:8000/instances?url=http://e-host:8001' \
174 -H "X-API-Key: $ADMIN_API_KEY"
175 ```
176
177 Registration is idempotent. The proxy probes registered instances every
178 `--probe-interval` seconds (default 5), with `--probe-timeout` seconds per probe
179 (default 2). After `--fail-threshold` consecutive failures (default 3), it stops
180 sending new requests to that instance. Healthy instances rejoin automatically;
181 unreachable ones are forgotten after `--evicted-ttl` seconds (default 900;
182 `0` retains them indefinitely). Removal stops new routing; already routed
183 requests retain their selected endpoints, so drain requests before stopping
184 the instance. The launcher must re-register instances after a proxy restart.
185
186 The example launch scripts also support this flow: export `ADMIN_API_KEY` and
187 set `DYNAMIC_REGISTRATION=1` when running `disagg_1e1pd_example.sh` or
188 `disagg_1e1p1d_example.sh`. Their default remains static routing.
189
190 ### Static configuration
191
192 The proxy batches images from the same user request assigned to the same encoder.
193 Set the proxy environment variable `ENCODER_MAX_BATCH_SIZE` to limit the number
194 of images per encoder subrequest. It defaults to `0` (unlimited); `1` sends each
195 image separately. Audio and video remain separate subrequests.
196
197 For encoders with a smaller `--limit-mm-per-prompt` image limit than P/PD, set
198 `ENCODER_MAX_BATCH_SIZE` no higher than the smallest encoder image limit.
199 For example, for encoders configured with `--limit-mm-per-prompt '{"image": 2}'`:
200
201 ```bash
202 ENCODER_MAX_BATCH_SIZE=2 python disagg_epd_proxy.py \
203 --encode-servers-urls "http://e1:8001,http://e2:8002" \
204 --prefill-servers-urls disable \
205 --decode-servers-urls "http://pd1:8003"
206 ```
207
208 Example usage:
209 For E + PD setup:
210
211 ```bash
212 $ python disagg_encoder_proxy.py \
213 --encode-servers-urls "http://e1:8001,http://e2:8002" \
214 --prefill-servers-urls "disable" \
215 --decode-servers-urls "http://pd1:8003,http://pd2:8004"
216 ```
217
218 For E + P + D setup:
219
220 ```bash
221 $ python disagg_encoder_proxy.py \
222 --encode-servers-urls "http://e1:8001,http://e2:8001" \
223 --prefill-servers-urls "http://p1:8003,http://p2:8004" \
224 --decode-servers-urls "http://d1:8005,http://d2:8006"
225 ```