| 1 | # NixlConnector Compatibility Matrix |
| 2 | |
| 3 | This page documents the feature compatibility of **disaggregated prefilling with the NixlConnector**. For general usage instructions, see the [NixlConnector Usage Guide](nixl_connector_usage.md). For an overview of disaggregated prefilling, see [Disaggregated Prefilling](disagg_prefill.md). |
| 4 | |
| 5 | !!! note |
| 6 | This page reflects the current state of the codebase and is subject to change as features evolve. Entries marked 🟠 or ❌ may link to tracking issues. See the [NIXL connector roadmap](https://github.com/vllm-project/vllm/issues/33702) for upcoming feature development. |
| 7 | |
| 8 | **Legend:** |
| 9 | |
| 10 | - ✅ = Fully supported |
| 11 | - 🟠 = Partial support (see footnotes) |
| 12 | - ❌ = Not supported |
| 13 | - ❔ = Unknown / not yet validated |
| 14 | - 🚧 = Work in progress |
| 15 | |
| 16 | !!! info "Universally supported features" |
| 17 | The following features work with **all** model architectures when using NixlConnector PD disaggregated serving: |
| 18 | |
| 19 | [Chunked Prefill](../configuration/optimization.md#chunked-prefill) | |
| 20 | [APC (Prefix Caching)](automatic_prefix_caching.md) | |
| 21 | [Data Parallel](../serving/data_parallel_deployment.md) | |
| 22 | CUDA graph | |
| 23 | Logprobs | |
| 24 | Prompt Logprobs | |
| 25 | [Prompt Embeds](prompt_embeds.md) | |
| 26 | Multiple NIXL backends (UCX, GDS, LIBFABRIC, etc.) |
| 27 | |
| 28 | ## Model Architecture x Capability |
| 29 | |
| 30 | <style> |
| 31 | td:not(:first-child) { |
| 32 | text-align: center !important; |
| 33 | } |
| 34 | td { |
| 35 | padding: 0.5rem !important; |
| 36 | white-space: nowrap; |
| 37 | } |
| 38 | |
| 39 | th { |
| 40 | padding: 0.5rem !important; |
| 41 | min-width: 0 !important; |
| 42 | } |
| 43 | |
| 44 | th:not(:first-child) { |
| 45 | writing-mode: vertical-lr; |
| 46 | transform: rotate(180deg) |
| 47 | } |
| 48 | </style> |
| 49 | |
| 50 | | Model type | <abbr title="Basic Prefill/Decode disaggregation">Basic PD</abbr> | <abbr title="Speculative Decoding">Spec Decode</abbr> | <abbr title="Heterogeneous Tensor Parallelism (P TP != D TP)">Hetero TP</abbr> | <abbr title="Cross-layer blocks optimization">Cross-layer blocks</abbr> | <abbr title="Sliding Window Attention">SWA</abbr> | <abbr title="CPU host buffer offload (e.g. TPU)">Host buffer</abbr> | <abbr title="Different block sizes on P and D">Hetero block size</abbr> | |
| 51 | | - | - | - | - | - | - | - | - | |
| 52 | | Dense Transformers | ✅ | ✅<sup>1</sup> | ✅ | ✅<sup>2</sup> | ✅ | ✅ | 🟠<sup>3</sup> | |
| 53 | | MLA (e.g. DeepSeek-V2/V3) | ✅ | ✅<sup>1</sup> | 🟠<sup>4</sup> | ✅<sup>2</sup> | ✅ | ✅ | 🟠<sup>3</sup> | |
| 54 | | Sparse MLA (e.g. DeepSeek-V3.2) | ✅ | ✅<sup>1</sup> | 🟠<sup>4</sup> | ✅<sup>2</sup> | ✅ | ✅ | 🟠<sup>3</sup> | |
| 55 | | Hybrid SSM / Mamba | ✅ | ❔ | 🚧<sup>5</sup> | ❌ | ✅ | ✅ | ❌<sup>6</sup> | |
| 56 | | MoE | ✅ | ✅<sup>1</sup> | ✅ | ✅<sup>2</sup> | ✅ | ✅ | 🟠<sup>3</sup> | |
| 57 | | Multimodal | ❔ | ❔ | ❔ | ❔ | ❔ | ❔ | ❔ | |
| 58 | | Encoder-Decoder | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
| 59 | |
| 60 | <sup>1</sup> P and D instances must use compatible speculation configurations. |
| 61 | See [Configuration Notes](#configuration-notes) below for what must match and |
| 62 | what may differ. |
| 63 | |
| 64 | <sup>2</sup> Cross-layer contiguity is achieved by using a `BLHNC` layout (set via `VLLM_KV_CACHE_LAYOUT=BLHNC`). |
| 65 | |
| 66 | <sup>3</sup> Supported only when HMA is **not** required (i.e., non-hybrid models). Block IDs are remapped automatically. Only P block size < D block size is supported. |
| 67 | |
| 68 | <sup>4</sup> MLA KV cache is replicated across TP workers, so heterogeneous TP works but there is no head-splitting. When P TP > D TP, only a single read is executed (redundant ranks are skipped). D TP > P TP also works. |
| 69 | |
| 70 | <sup>5</sup> Hybrid SSM (Mamba) models require **homogeneous TP** (`P TP == D TP`). Heterogeneous TP is not yet supported for Mamba layers. |
| 71 | |
| 72 | <sup>6</sup> HMA (required by hybrid models) does not support different remote block sizes. |
| 73 | |
| 74 | ## Configuration Notes |
| 75 | |
| 76 | ### What must match between P and D |
| 77 | |
| 78 | By default, a **compatibility hash** is checked during handshake. P and D instances must agree on: |
| 79 | |
| 80 | - vLLM version and NIXL connector version |
| 81 | - Model (architecture, dtype, number of KV heads, head size, number of hidden layers) |
| 82 | - Attention backend |
| 83 | - KV cache dtype (`cache_dtype`) |
| 84 | - EAGLE/MTP-style speculative method and draft-model configuration |
| 85 | - NIXL transfer mode (push vs pull) — a push (WRITE) connector and a pull (READ) |
| 86 | connector use incompatible transfer protocols and must never be paired |
| 87 | |
| 88 | !!! warning |
| 89 | Disable the hash check with `--kv-transfer-config '{"kv_connector_extra_config": {"enforce_handshake_compat": false}}'` at your own risk. |
| 90 | |
| 91 | ### What can safely differ between P and D |
| 92 | |
| 93 | - `tensor-parallel-size` (heterogeneous TP, subject to model restrictions above) |
| 94 | - `block-size` (heterogeneous block size, subject to restrictions above) |
| 95 | - Number of KV cache blocks (determined by available memory on each instance) |
| 96 | - `num_speculative_tokens` (prefill and decode may use different draft depths) |
| 97 | - Draft-model `attention_backend` (each instance auto-selects independently; the |
| 98 | resulting KV block layout is validated at handshake time rather than via the |
| 99 | compatibility hash) |
| 100 | |
| 101 | ### KV cache layout |
| 102 | |
| 103 | - NixlConnector defaults to **`LBHNC`** (head-major, formerly `HND`) layout for optimal transfer performance (non-MLA models). |
| 104 | - `LBNHC` (token-major, formerly `NHD`) layout is supported but does **not** allow heterogeneous TP head splitting. |
| 105 | - Experimental `LBHNC` ↔ `LBNHC` permute: enable via `--kv-transfer-config '{"enable_permute_local_kv": true}'`. Not supported with HMA. |
| 106 | |
| 107 | ### Pipeline parallelism |
| 108 | |
| 109 | The default (pull) NixlConnector does **not** support |
| 110 | `pipeline-parallel-size > 1` together with a hybrid KV cache (HMA) |
| 111 | layout: region indices are not a stable identity across the |
| 112 | prefill/decode layer split, so the connector raises at startup. Use the |
| 113 | push connector (`NixlPushConnector`) for pipeline parallelism with |
| 114 | hybrid KV caches — it routes transfers by layer-name (member) identity, |
| 115 | so a PP-sharded prefiller can write into a `PP=1` decoder. See |
| 116 | [NIXL push-mode KV transfer](../design/nixl_kv_push_connector.md). |
| 117 | |
| 118 | Current push PP + HMA limitations: |
| 119 | |
| 120 | - Only the prefiller (producer) may be PP-sharded; decode-side PP is not supported. |
| 121 | - Hybrid SSM/Mamba layouts are not supported under PP. |
| 122 | - HMA requires the same block size on P and D. |
| 123 | - Non-MLA attention-HMA layer routing requires decode TP to be no greater than prefill TP. |
| 124 | - Packed layouts under PP require MLA caches. Local and remote packed block strides may differ; layer pages must have equal sizes. MLA supports decode TP greater than prefill TP because its KV is replicated. |
| 125 | |
| 126 | ### Quantized KV cache |
| 127 | |
| 128 | [Quantized KV cache](quantization/quantized_kvcache.md) (e.g., FP8) requires both P and D instances to use the **same** `cache_dtype`. Mismatched cache dtypes will fail the compatibility hash check during handshake. |
| 129 | |
| 130 | - **Static quantization** (scales loaded from checkpoint): ✅ Supported. Scales are loaded independently by each instance from the model checkpoint. |
| 131 | - **Dynamic quantization** (scales computed at runtime): ❌ Not supported. Per-block scales are not transferred alongside KV cache data. |
| 132 | - **Packed-layout scales** (scales stored inline with weights): ✅ Supported. Scales are transferred together with the KV cache blocks. |