A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Docs] Update features/disagg_prefill, add v1 examples and development (#22165)
Signed-off-by: David Chen <530634352@qq.com>
WeiQing Chen committed
Aug 7, 2025 at 15:59 UTC
289b18e670c2439dfc1f4f80df782de9ad112762
3 files changed
+25
docs/assets/features/disagg_prefill/high_level_design.png
Binary files /dev/null and b/docs/assets/features/disagg_prefill/high_level_design.png differ
docs/assets/features/disagg_prefill/workflow.png
Binary files /dev/null and b/docs/assets/features/disagg_prefill/workflow.png differ
docs/features/disagg_prefill.md
+25
@@ -19,6 +19,18 @@ Two main reasons:
19
20
Please refer to <gh-file:examples/online_serving/disaggregated_prefill.sh> for the example usage of disaggregated prefilling.
21
22
+Now supports 5 types of connectors:
23
+
24
+- **SharedStorageConnector**: refer to <gh-file:examples/offline_inference/disaggregated-prefill-v1/run.sh> for the example usage of SharedStorageConnector disaggregated prefilling.
25
+- **LMCacheConnectorV1**: refer to <gh-file:examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh> for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission.
26
+- **NixlConnector**: refer to <gh-file:tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh> for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv.
27
+- **P2pNcclConnector**: refer to <gh-file:examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh> for the example usage of P2pNcclConnector disaggregated prefilling.
28
+- **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
29
+
30
+ ```bash
31
+ --kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both"},{"kv_connector":"SharedStorageConnector","kv_role":"kv_both","kv_connector_extra_config":{"shared_storage_path":"local_storage"}}]}}'
32
+ ```
33
+
34
## Benchmarks
35
36
Please refer to <gh-file:benchmarks/disagg_benchmarks> for disaggregated prefilling benchmarks.
@@ -48,6 +60,19 @@ The workflow of disaggregated prefilling is as follows:
60
61
The `buffer` corresponds to `insert` API in LookupBuffer, and the `drop_select` corresponds to `drop_select` API in LookupBuffer.
62
63
+Now every process in vLLM will have a corresponding connector. Specifically, we have:
64
+
65
+- Scheduler connector: the connector that locates in the same process as the scheduler process. It schedules the KV cache transfer ops.
66
+- Worker connectors: the connectors that locate in the worker processes. They execute KV cache transfer ops.
67
+
68
+Here is a figure illustrating how the above 2 connectors are organized:
69
+
70
+
71
+
72
+The figure below shows how the worker connector works with the attention module to achieve layer-by-layer KV cache store and load:
73
+
74
+
75
+
76
## Third-party contributions
77
78
Disaggregated prefilling is highly related to infrastructure, so vLLM relies on third-party connectors for production-level disaggregated prefilling (and vLLM team will actively review and merge new PRs for third-party connectors).