A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 5, 18:30 UTC
[Docs] PD with Nixl compat matrix (#38628)
Signed-off-by: NickLucche <nlucches@redhat.com>
Nicolò Lucchesi committed
Mar 31, 2026 at 17:01 UTC
7337ff7f03569f74676cbb64da0839ce4be82839
3 files changed
+107
-1
docs/features/disagg_prefill.md
+1
-1
@@ -23,7 +23,7 @@ Now supports 6 types of connectors:
23
24
- **ExampleConnector**: refer to [examples/offline_inference/disaggregated-prefill-v1/run.sh](../../examples/offline_inference/disaggregated-prefill-v1/run.sh) for the example usage of ExampleConnector disaggregated prefilling.
25
- **LMCacheConnectorV1**: refer to [examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh](../../examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh) for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission.
26
-- **NixlConnector**: refer to [tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh) for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. For detailed usage guide, see [NixlConnector Usage Guide](nixl_connector_usage.md).
26
+- **NixlConnector**: refer to [tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh) for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. For detailed usage guide, see [NixlConnector Usage Guide](nixl_connector_usage.md). For feature compatibility details, see [NixlConnector Compatibility Matrix](nixl_connector_compatibility.md).
27
- **P2pNcclConnector**: refer to [examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh](../../examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh) for the example usage of P2pNcclConnector disaggregated prefilling.
28
- **MooncakeConnector**: refer to [examples/online_serving/disaggregated_serving/mooncake_connector/run_mooncake_connector.sh](../../examples/online_serving/disaggregated_serving/mooncake_connector/run_mooncake_connector.sh) for the example usage of ExampleConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md).
29
- **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
docs/features/nixl_connector_compatibility.md
new
+104
@@ -0,0 +1,104 @@
1
+# NixlConnector Compatibility Matrix
2
+
3
+This page documents the feature compatibility of **disaggregated prefilling with the NixlConnector**. For general usage instructions, see the [NixlConnector Usage Guide](nixl_connector_usage.md). For an overview of disaggregated prefilling, see [Disaggregated Prefilling](disagg_prefill.md).
4
+
5
+!!! note
6
+ This page reflects the current state of the codebase and is subject to change as features evolve. Entries marked 🟠 or ❌ may link to tracking issues. See the [NIXL connector roadmap](https://github.com/vllm-project/vllm/issues/33702) for upcoming feature development.
7
+
8
+**Legend:**
9
+
10
+- ✅ = Fully supported
11
+- 🟠 = Partial support (see footnotes)
12
+- ❌ = Not supported
13
+- ❔ = Unknown / not yet validated
14
+- 🚧 = Work in progress
15
+
16
+!!! info "Universally supported features"
17
+ The following features work with **all** model architectures when using NixlConnector PD disaggregated serving:
18
+
19
+ [Chunked Prefill](../configuration/optimization.md#chunked-prefill) |
20
+ [APC (Prefix Caching)](automatic_prefix_caching.md) |
21
+ [Data Parallel](../serving/data_parallel_deployment.md) |
22
+ CUDA graph |
23
+ Logprobs |
24
+ Prompt Logprobs |
25
+ [Prompt Embeds](prompt_embeds.md) |
26
+ Multiple NIXL backends (UCX, GDS, LIBFABRIC, etc.)
27
+
28
+## Model Architecture x Capability
29
+
30
+<style>
31
+td:not(:first-child) {
32
+ text-align: center !important;
33
+}
34
+td {
35
+ padding: 0.5rem !important;
36
+ white-space: nowrap;
37
+}
38
+
39
+th {
40
+ padding: 0.5rem !important;
41
+ min-width: 0 !important;
42
+}
43
+
44
+th:not(:first-child) {
45
+ writing-mode: vertical-lr;
46
+ transform: rotate(180deg)
47
+}
48
+</style>
49
+
50
+| Model type | <abbr title="Basic Prefill/Decode disaggregation">Basic PD</abbr> | <abbr title="Speculative Decoding">Spec Decode</abbr> | <abbr title="Heterogeneous Tensor Parallelism (P TP != D TP)">Hetero TP</abbr> | <abbr title="Cross-layer blocks optimization">Cross-layer blocks</abbr> | <abbr title="Sliding Window Attention">SWA</abbr> | <abbr title="CPU host buffer offload (e.g. TPU)">Host buffer</abbr> | <abbr title="Different block sizes on P and D">Hetero block size</abbr> |
51
+| - | - | - | - | - | - | - | - |
52
+| Dense Transformers | ✅ | ✅<sup>1</sup> | ✅ | ✅<sup>2</sup> | ✅ | ✅ | 🟠<sup>3</sup> |
53
+| MLA (e.g. DeepSeek-V2/V3) | ✅ | ✅<sup>1</sup> | 🟠<sup>4</sup> | ✅<sup>2</sup> | ✅ | ✅ | 🟠<sup>3</sup> |
54
+| Sparse MLA (e.g. DeepSeek-V3.2) | ✅ | ✅<sup>1</sup> | 🟠<sup>4</sup> | ✅<sup>2</sup> | ✅ | ✅ | 🟠<sup>3</sup> |
55
+| Hybrid SSM / Mamba | ✅ | ❔ | 🚧<sup>5</sup> | ❌ | ✅ | ✅ | ❌<sup>6</sup> |
56
+| MoE | ✅ | ✅<sup>1</sup> | ✅ | ✅<sup>2</sup> | ✅ | ✅ | 🟠<sup>3</sup> |
57
+| Multimodal | ❔ | ❔ | ❔ | ❔ | ❔ | ❔ | ❔ |
58
+| Encoder-Decoder | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
59
+
60
+<sup>1</sup> P and D instances must use the same speculation configuration.
61
+
62
+<sup>2</sup> Requires `FLASH_ATTN` or `FLASHINFER` backend **and** `HND` KV cache layout. Enable via `--kv-transfer-config '{"kv_connector_extra_config": {"enable_cross_layers_blocks": "True"}}'`.
63
+
64
+<sup>3</sup> Supported only when HMA is **not** required (i.e., non-hybrid models). Block IDs are remapped automatically. Only P block size < D block size is supported.
65
+
66
+<sup>4</sup> MLA KV cache is replicated across TP workers, so heterogeneous TP works but there is no head-splitting. When P TP > D TP, only a single read is executed (redundant ranks are skipped). D TP > P TP also works.
67
+
68
+<sup>5</sup> Hybrid SSM (Mamba) models require **homogeneous TP** (`P TP == D TP`). Heterogeneous TP is not yet supported for Mamba layers.
69
+
70
+<sup>6</sup> HMA (required by hybrid models) does not support different remote block sizes.
71
+
72
+## Configuration Notes
73
+
74
+### What must match between P and D
75
+
76
+By default, a **compatibility hash** is checked during handshake. P and D instances must agree on:
77
+
78
+- vLLM version and NIXL connector version
79
+- Model (architecture, dtype, number of KV heads, head size, number of hidden layers)
80
+- Attention backend
81
+- KV cache dtype (`cache_dtype`)
82
+
83
+!!! warning
84
+ Disable the hash check with `--kv-transfer-config '{"kv_connector_extra_config": {"enforce_handshake_compat": false}}'` at your own risk.
85
+
86
+### What can safely differ between P and D
87
+
88
+- `tensor-parallel-size` (heterogeneous TP, subject to model restrictions above)
89
+- `block-size` (heterogeneous block size, subject to restrictions above)
90
+- Number of KV cache blocks (determined by available memory on each instance)
91
+
92
+### KV cache layout
93
+
94
+- NixlConnector defaults to **`HND`** layout for optimal transfer performance (non-MLA models).
95
+- `NHD` layout is supported but does **not** allow heterogeneous TP head splitting.
96
+- Experimental `HND` ↔ `NHD` permute: enable via `--kv-transfer-config '{"enable_permute_local_kv": true}'`. Not supported with HMA.
97
+
98
+### Quantized KV cache
99
+
100
+[Quantized KV cache](quantization/quantized_kvcache.md) (e.g., FP8) requires both P and D instances to use the **same** `cache_dtype`. Mismatched cache dtypes will fail the compatibility hash check during handshake.
101
+
102
+- **Static quantization** (scales loaded from checkpoint): ✅ Supported. Scales are loaded independently by each instance from the model checkpoint.
103
+- **Dynamic quantization** (scales computed at runtime): ❌ Not supported. Per-block scales are not transferred alongside KV cache data.
104
+- **Packed-layout scales** (scales stored inline with weights): ✅ Supported. Scales are transferred together with the KV cache blocks.
docs/features/nixl_connector_usage.md
+2
@@ -2,6 +2,8 @@
2
3
NixlConnector is a high-performance KV cache transfer connector for vLLM's disaggregated prefilling feature. It provides fully asynchronous send/receive operations using the NIXL library for efficient cross-process KV cache transfer.
4
5
+For feature compatibility details (supported model architectures, TP configurations, and feature interactions), see the [NixlConnector Compatibility Matrix](nixl_connector_compatibility.md).
6
+
7
## Prerequisites
8
9
### Installation