@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 112 lines 4.88 KB
Rendered Raw
1 # Disaggregated Encoder
2
3 A **disaggregated encoder** runs the vision-encoder stage of a multimodal LLM in a process that is separate from the pre-fill / decoder stage. Deploying these two stages in independent vLLM instances brings three practical benefits:
4
5 1. **Independent, fine-grained scaling**
6 2. **Lower time-to-first-token (TTFT)**
7 3. **Cross-process reuse and caching of encoder outputs**
8
9 Design doc: <https://docs.google.com/document/d/1aed8KtC6XkXtdoV87pWT0a8OJlZ-CpnuLLzmR8l9BAE>
10
11 ---
12
13 ## 1 Motivation
14
15 ### 1. Independent, fine-grained scaling
16
17 * Vision encoders are lightweight, while language models are orders of magnitude larger.
18 * The language model can be parallelised without affecting the encoder fleet.
19 * Encoder nodes can be added or removed independently.
20
21 ### 2. Lower time-to-first-token (TTFT)
22
23 * Language-only requests bypass the vision encoder entirely.
24 * Encoder output is injected only at required attention layers, shortening the pre-fill critical path.
25
26 ### 3. Cross-process reuse and caching
27
28 * In-process encoders confine reuse to a single worker.
29 * A remote, shared cache lets any worker retrieve existing embeddings, eliminating redundant computation.
30
31 ---
32
33 ## 2 Usage Example
34
35 ### ExampleConnector
36
37 The following scripts demonstrate the workflow with **ExampleConnector**:
38
39 1 Encoder instance + 1 PD instance:
40 `examples/disaggregated/disaggregated_encoder/disagg_1e1pd_example.sh`
41
42 1 Encoder instance + 1 Prefill instance + 1 Decode instance:
43 `examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh`
44
45 ### ECMooncakeConnector
46
47 **ECMooncakeConnector** transfers encoder outputs using the Mooncake TransferEngine.
48 See the [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md) for
49 producer/consumer `--ec-transfer-config`, proxy `--ec-consumer-zmq-addrs`, and
50 transport options. The [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
51 is a complete 1 Encoder instance + 1 PD instance setup.
52
53 With vLLM and Mooncake installed, run the example from the repository root:
54
55 ```bash
56 GPU_E=0 GPU_PD=1 MOONCAKE_EC_PROTOCOL=tcp \
57 bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
58 ```
59
60 The script uses `Qwen/Qwen2.5-VL-3B-Instruct` by default, runs a single-GPU baseline,
61 and compares the disaggregated outputs against it. Set `MODEL` to use another model
62 or `MOONCAKE_EC_PROTOCOL=rdma` to use RDMA on supported hardware. See the script for
63 additional configuration options.
64
65 ### Audio inputs
66
67 The Python EPD proxy also rewrites audio in `/v1/chat/completions` into
68 metadata-only references for Qwen2-Audio, AudioFlamingo3, Ultravox,
69 Qwen2.5-Omni, and Qwen3-Omni. The encoder publishes `audio_num_tokens`
70 (the placeholder token count of each audio) together with any feature
71 lengths the model needs; the consumer uses them to reconstruct the audio
72 placeholders and receives the embeddings through its EC connector without
73 repeating audio preprocessing.
74
75 This covers pure audio inputs, not video with an embedded audio track. It does
76 not add `/v1/audio/transcriptions` or realtime routes to the example proxy.
77
78 ---
79
80 ## 3 Test Script
81
82 For optional cross-Encoder output reuse while retaining Mooncake P2P delivery,
83 see [Cross-encoder output reuse](cross_encoder_cache.md).
84
85 Please refer to the directories `tests/v1/ec_connector`
86
87 ## 4 Development
88
89 Disaggregated encoding is implemented by running two parts:
90
91 * **Encoder instance** – a vLLM instance to performs vision encoding.
92 * **Prefill/Decode (PD) instance(s)** – runs language pre-fill and decode.
93 * PD can be in either a single normal instance with `disagg_encoder_example.sh` (E->PD) or in disaggregated instances with `disagg_epd_example.sh` (E->P->D)
94
95 A connector transfers encoder-cache (EC) embeddings from the encoder instance to the PD instance.
96 All related code is under `vllm/distributed/ec_transfer`.
97
98 ### Key abstractions
99
100 * **ECConnector** – interface for retrieving EC caches produced by the encoder.
101 * *Scheduler role* – checks cache existence and schedules loads.
102 * *Worker role* – loads the embeddings into memory.
103
104 Here is a figure illustrating disaggregate encoder flow:
105
106 ![Disaggregated Encoder Flow](../assets/features/disagg_encoder/disagg_encoder_flow.png)
107
108 For the PD disaggregation part, the Prefill instance receives cache exactly the same as the disaggregated encoder flow above. Prefill instance executes 1 step (prefill -> 1 token output) and then transfers KV cache to the Decode instance for the remaining execution. The KV transfer part purely happens after the execution of the PD instance.
109
110 `docs/features/disagg_prefill.md` shows the brief idea about the disaggregated prefill (v0)
111
112 We create the example setup with the **NixlConnector** from `vllm/distributed/kv_transfer/kv_connector/v1/nixl/` and referred to the `tests/v1/kv_connector/nixl_integration/toy_proxy_server.py` to facilitate the kv transfer between P and D;