| 1 | # NixlConnector Usage Guide |
| 2 | |
| 3 | NixlConnector is a high-performance KV cache transfer connector for vLLM's disaggregated prefilling feature. It provides fully asynchronous send/receive operations using the NIXL library for efficient cross-process KV cache transfer. |
| 4 | |
| 5 | For feature compatibility details (supported model architectures, TP configurations, and feature interactions), see the [NixlConnector Compatibility Matrix](nixl_connector_compatibility.md). |
| 6 | |
| 7 | ## Prerequisites |
| 8 | |
| 9 | ### Installation |
| 10 | |
| 11 | Install the NIXL library: `uv pip install nixl`, as a quick start on Nvidia platform. |
| 12 | |
| 13 | - Refer to [NIXL official repository](https://github.com/ai-dynamo/nixl) for more installation instructions |
| 14 | - The specified required NIXL version can be found in [requirements/kv_connectors.txt](../../requirements/kv_connectors.txt) and other relevant config files |
| 15 | |
| 16 | For ROCm, the [ROCm Dockerfile](../../docker/Dockerfile.rocm) builds NIXL and UCX with ROCm support from source. |
| 17 | |
| 18 | For non-cuda platform, please install nixl with ucx build from source, instructed as below. |
| 19 | |
| 20 | ```bash |
| 21 | python tools/install_nixl_from_source_ubuntu.py |
| 22 | ``` |
| 23 | |
| 24 | ### Transport Configuration |
| 25 | |
| 26 | NixlConnector uses NIXL library for underlying communication, which supports multiple transport backends. UCX (Unified Communication X) is the primary default transport library used by NIXL. Configure transport environment variables: |
| 27 | |
| 28 | ```bash |
| 29 | # Example UCX configuration, adjust according to your environment |
| 30 | export UCX_TLS=all # or specify specific transports like "rc,ud,sm,^cuda_ipc" ..etc |
| 31 | export UCX_NET_DEVICES=all # or specify network devices like "mlx5_0:1,mlx5_1:1" |
| 32 | ``` |
| 33 | |
| 34 | !!! tip |
| 35 | When using UCX as the transport backend, NCCL environment variables (like `NCCL_IB_HCA`, `NCCL_SOCKET_IFNAME`) are not applicable to NixlConnector, so configure UCX-specific environment variables instead of NCCL variables. |
| 36 | |
| 37 | #### Selecting a NIXL transport backend (plugin) |
| 38 | |
| 39 | NixlConnector can use different NIXL transport backends (plugins). By default, NixlConnector uses UCX as the transport backend. |
| 40 | |
| 41 | To select a different backend, set `kv_connector_extra_config.backends` in `--kv-transfer-config`. |
| 42 | |
| 43 | ### Example: using LIBFABRIC backend |
| 44 | |
| 45 | ```bash |
| 46 | vllm serve <MODEL> \ |
| 47 | --kv-transfer-config '{ |
| 48 | "kv_connector":"NixlConnector", |
| 49 | "kv_role":"kv_producer", |
| 50 | "kv_connector_extra_config":{"backends":["LIBFABRIC"]} |
| 51 | }' |
| 52 | ``` |
| 53 | |
| 54 | You can also pass JSON keys individually using dotted arguments, and you can append list elements using `+`: |
| 55 | |
| 56 | ```bash |
| 57 | vllm serve <MODEL> \ |
| 58 | --kv-transfer-config.kv_connector NixlConnector \ |
| 59 | --kv-transfer-config.kv_role kv_producer \ |
| 60 | --kv-transfer-config.kv_connector_extra_config.backends+ LIBFABRIC |
| 61 | ``` |
| 62 | |
| 63 | !!! note |
| 64 | Backend availability depends on how NIXL was built and what plugins are present in your environment. Refer to the [NIXL repository](https://github.com/ai-dynamo/nixl) for available backends and build instructions. |
| 65 | |
| 66 | ## Basic Usage (on the same host) |
| 67 | |
| 68 | ### Producer (Prefiller) Configuration |
| 69 | |
| 70 | Start a prefiller instance that produces KV caches |
| 71 | |
| 72 | ```bash |
| 73 | # 1st GPU as prefiller |
| 74 | CUDA_VISIBLE_DEVICES=0 \ |
| 75 | UCX_NET_DEVICES=all \ |
| 76 | VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \ |
| 77 | vllm serve Qwen/Qwen3-0.6B \ |
| 78 | --port 8100 \ |
| 79 | --enforce-eager \ |
| 80 | --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail"}' |
| 81 | ``` |
| 82 | |
| 83 | ### Consumer (Decoder) Configuration |
| 84 | |
| 85 | Start a decoder instance that consumes KV caches: |
| 86 | |
| 87 | ```bash |
| 88 | # 2nd GPU as decoder |
| 89 | CUDA_VISIBLE_DEVICES=1 \ |
| 90 | UCX_NET_DEVICES=all \ |
| 91 | VLLM_NIXL_SIDE_CHANNEL_PORT=5601 \ |
| 92 | vllm serve Qwen/Qwen3-0.6B \ |
| 93 | --port 8200 \ |
| 94 | --enforce-eager \ |
| 95 | --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"}' |
| 96 | ``` |
| 97 | |
| 98 | ### Proxy Server |
| 99 | |
| 100 | Use a proxy server to route requests between prefiller and decoder: |
| 101 | |
| 102 | ```bash |
| 103 | python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \ |
| 104 | --port 8192 \ |
| 105 | --prefiller-hosts localhost \ |
| 106 | --prefiller-ports 8100 \ |
| 107 | --decoder-hosts localhost \ |
| 108 | --decoder-ports 8200 |
| 109 | ``` |
| 110 | |
| 111 | ## Environment Variables |
| 112 | |
| 113 | - `VLLM_NIXL_SIDE_CHANNEL_PORT`: Port for NIXL handshake communication |
| 114 | - Default: 5600 |
| 115 | - **Required for both prefiller and decoder instances** |
| 116 | - Each vLLM worker needs a unique port on its host; using the same port number across different hosts is fine |
| 117 | - For TP/DP deployments, each worker's port on a node is computed as: base_port + dp_rank (e.g., with `--data-parallel-size=2` and base_port=5600, dp_rank 0..1 use port 5600, 5601 on that node). |
| 118 | - Used for the initial NIXL handshake between the prefiller and the decoder |
| 119 | |
| 120 | - `VLLM_NIXL_SIDE_CHANNEL_HOST`: Host for side channel communication |
| 121 | - Default: "localhost" |
| 122 | - Set when prefiller and decoder are on different machines |
| 123 | - Connection info is passed via KVTransferParams from prefiller to decoder for handshake |
| 124 | |
| 125 | - `kv_lease_duration` (via `kv_connector_extra_config`): Lease duration (in seconds) for the prefiller's KV cache blocks. (Optional) |
| 126 | - Default: 30 |
| 127 | - When a prefill request finishes, its KV blocks are held for this duration waiting for the decoder to read them. While the request is queued on the decoder, periodic heartbeats automatically extend the lease. If neither a heartbeat nor a read notification arrives before the lease expires, the blocks are freed. The heartbeat interval and extension amount are derived automatically from this value. |
| 128 | - Example: `--kv-transfer-config '{"kv_connector_extra_config": {"kv_lease_duration": 60}}'` |
| 129 | |
| 130 | - `decoder_kv_blocks_ttl` (via `kv_connector_extra_config`): TTL (in seconds) for KV blocks cached on the decoder in bidirectional transfer mode. (Optional) |
| 131 | - Default: 480 |
| 132 | - In bidirectional mode, the decoder caches KV blocks for multi-turn conversations. This TTL controls how long those blocks are held before being released. Unlike the prefiller lease, this TTL is not renewed via heartbeats. |
| 133 | - Example: `--kv-transfer-config '{"kv_connector_extra_config": {"decoder_kv_blocks_ttl": 600}}'` |
| 134 | |
| 135 | ## Bidirectional KV Transfer (Multi-turn) |
| 136 | |
| 137 | In standard disaggregated prefilling, KV cache flows in one direction: Prefill (P) computes the KV cache and Decode (D) reads from P. For multi-turn conversations this is wasteful — D already holds the KV cache corresponding to the generated tokens from prior turns, yet P must recompute it from scratch on every new turn. Bidirectional KV transfer lets P **pull** existing KV blocks from D via RDMA before computing only the new tokens, significantly reducing Time-To-First-Token (TTFT) for long-prefill such as **multi-turn heavy scenarios**. |
| 138 | |
| 139 | ### How it works |
| 140 | |
| 141 | The feature relies on a **stateful proxy** that sits between the client and the P/D instances. The proxy tracks `kv_transfer_params` returned by D at the end of each turn, and attaches them to the next turn's request so P knows which blocks to pull from D. |
| 142 | |
| 143 | ```mermaid |
| 144 | sequenceDiagram |
| 145 | participant Client |
| 146 | participant Proxy |
| 147 | participant P as Prefill (P) |
| 148 | participant D as Decode (D) |
| 149 | |
| 150 | rect rgb(240, 240, 250) |
| 151 | note right of Client: Turn 1 — Cache Miss |
| 152 | Client->>Proxy: chat request + conversation_id |
| 153 | Proxy->>P: request (no remote blocks) |
| 154 | activate P |
| 155 | note over P: full prefill |
| 156 | P-->>Proxy: kv_transfer_params (P's blocks) |
| 157 | deactivate P |
| 158 | Proxy->>D: request + P's kv_transfer_params |
| 159 | activate D |
| 160 | D-->P: RDMA read (D pulls KV from P) |
| 161 | note over D: decode |
| 162 | D-->>Proxy: stream response + kv_transfer_params |
| 163 | deactivate D |
| 164 | note over Proxy: cache D's kv_transfer_params |
| 165 | Proxy-->>Client: response |
| 166 | end |
| 167 | |
| 168 | rect rgb(255, 245, 235) |
| 169 | note right of Client: Turn 2+ — Cache Hit (Bidirectional) |
| 170 | Client->>Proxy: chat request + conversation_id |
| 171 | note over Proxy: lookup cached D blocks |
| 172 | Proxy->>P: request + D's remote_block_ids |
| 173 | activate P |
| 174 | P-->D: RDMA read (P pulls KV from D) |
| 175 | note over P: prefill new tokens only |
| 176 | P-->>Proxy: kv_transfer_params (P's blocks) |
| 177 | deactivate P |
| 178 | Proxy->>D: request + P's kv_transfer_params |
| 179 | activate D |
| 180 | D-->P: RDMA read (D pulls new KV from P) |
| 181 | note over D: decode |
| 182 | D-->>Proxy: stream response + kv_transfer_params |
| 183 | deactivate D |
| 184 | note over Proxy: update cached kv_transfer_params |
| 185 | Proxy-->>Client: response |
| 186 | end |
| 187 | ``` |
| 188 | |
| 189 | **Turn 1 (cache miss):** |
| 190 | |
| 191 | 1. Client sends a chat request with a `conversation_id` to the proxy. |
| 192 | 2. Proxy forwards the request to P with no remote block info — P computes the full KV cache. |
| 193 | 3. Proxy forwards the request to D along with P's `kv_transfer_params` (block IDs, engine ID, host/port). |
| 194 | 4. D reads KV blocks from P via RDMA (peer-to-peer pull), then generates the response. |
| 195 | 5. D streams the response back through the proxy. The final chunk includes D's own `kv_transfer_params`. |
| 196 | 6. Proxy caches D's `kv_transfer_params` keyed by `conversation_id`, then returns the response to the client. |
| 197 | |
| 198 | **Turn 2+ (cache hit — bidirectional):** |
| 199 | |
| 200 | 1. Client sends the next turn with the same `conversation_id`. |
| 201 | 2. Proxy looks up cached `kv_transfer_params` from the previous turn and attaches D's `remote_block_ids` to the request sent to P. |
| 202 | 3. P reads the existing KV cache from D via RDMA (D→P pull), then computes KV only for the new tokens. |
| 203 | 4. Proxy forwards the request to D with P's updated `kv_transfer_params`. |
| 204 | 5. D reads the new KV blocks from P, generates the response, and returns updated `kv_transfer_params` which the proxy caches for the next turn. |
| 205 | |
| 206 | ### Configuration |
| 207 | |
| 208 | Enable bidirectional KV transfer by setting `bidirectional_kv_xfer` in `kv_connector_extra_config` on **both** P and D instances: |
| 209 | |
| 210 | ```bash |
| 211 | # Prefill instance |
| 212 | vllm serve <MODEL> \ |
| 213 | --kv-transfer-config '{ |
| 214 | "kv_connector": "NixlConnector", |
| 215 | "kv_role": "kv_producer", |
| 216 | "kv_connector_extra_config": { |
| 217 | "bidirectional_kv_xfer": true |
| 218 | } |
| 219 | }' |
| 220 | |
| 221 | # Decode instance |
| 222 | vllm serve <MODEL> \ |
| 223 | --kv-transfer-config '{ |
| 224 | "kv_connector": "NixlConnector", |
| 225 | "kv_role": "kv_consumer", |
| 226 | "kv_connector_extra_config": { |
| 227 | "bidirectional_kv_xfer": true |
| 228 | } |
| 229 | }' |
| 230 | ``` |
| 231 | |
| 232 | Additional configuration options in `kv_connector_extra_config`: |
| 233 | |
| 234 | | Parameter | Default | Description | |
| 235 | | --------- | ------- | ----------- | |
| 236 | | `bidirectional_kv_xfer` | `false` | Enable bidirectional D→P KV transfer. | |
| 237 | | `kv_recompute_threshold` | `64` | Minimum number of remote tokens required to trigger a D→P pull. Below this threshold, P recomputes locally instead of pulling (to amortize transfer latency). | |
| 238 | | `decoder_kv_blocks_ttl` | `480` | TTL (seconds) for KV blocks cached on D for bidirectional reuse. Blocks are released after this duration. Not renewed via heartbeats. | |
| 239 | |
| 240 | ### Multi-turn proxy setup |
| 241 | |
| 242 | Use the provided multi-turn proxy to manage `kv_transfer_params` caching across conversation turns: |
| 243 | |
| 244 | ```bash |
| 245 | python examples/disaggregated/disaggregated_serving/disagg_proxy_multiturn.py \ |
| 246 | --host 0.0.0.0 --port 8000 \ |
| 247 | --prefiller-host <P_IP> --prefiller-port 8100 \ |
| 248 | --decoder-host <D_IP> --decoder-port 8200 |
| 249 | ``` |
| 250 | |
| 251 | The proxy supports multiple P and D instances via round-robin: |
| 252 | |
| 253 | ```bash |
| 254 | python examples/disaggregated/disaggregated_serving/disagg_proxy_multiturn.py \ |
| 255 | --host 0.0.0.0 --port 8000 \ |
| 256 | --prefiller-hosts <P_IP1> <P_IP2> --prefiller-ports 8100 8100 \ |
| 257 | --decoder-hosts <D_IP1> <D_IP2> --decoder-ports 8200 8200 |
| 258 | ``` |
| 259 | |
| 260 | ### Client usage |
| 261 | |
| 262 | Include a `conversation_id` field in the request body to enable cross-turn KV reuse. Without it, the proxy cannot link turns and falls back to full recomputation. |
| 263 | |
| 264 | ```bash |
| 265 | # Turn 1 |
| 266 | curl http://localhost:8000/v1/chat/completions \ |
| 267 | -H "Content-Type: application/json" \ |
| 268 | -d '{ |
| 269 | "model": "Qwen/Qwen3-0.6B", |
| 270 | "conversation_id": "session-42", |
| 271 | "messages": [ |
| 272 | {"role": "user", "content": "What is vLLM?"} |
| 273 | ] |
| 274 | }' |
| 275 | |
| 276 | # Turn 2 — same conversation_id triggers bidirectional KV pull |
| 277 | curl http://localhost:8000/v1/chat/completions \ |
| 278 | -H "Content-Type: application/json" \ |
| 279 | -d '{ |
| 280 | "model": "Qwen/Qwen3-0.6B", |
| 281 | "conversation_id": "session-42", |
| 282 | "messages": [ |
| 283 | {"role": "user", "content": "What is vLLM?"}, |
| 284 | {"role": "assistant", "content": "vLLM is a high-throughput LLM serving engine..."}, |
| 285 | {"role": "user", "content": "How does disaggregated prefilling work?"} |
| 286 | ] |
| 287 | }' |
| 288 | ``` |
| 289 | |
| 290 | !!! note |
| 291 | The `conversation_id` field is a non-standard extension to the OpenAI API. It is consumed by the proxy and not forwarded to the vLLM engine. |
| 292 | |
| 293 | ### Benchmarking the multi-turn proxy |
| 294 | |
| 295 | [`benchmarks/multi_turn/benchmark_serving_multi_turn.py`](../../benchmarks/multi_turn/benchmark_serving_multi_turn.py) supports targeting the disaggregated multi-turn proxy with the `--send-conversation-id` flag, which injects a per-conversation `conversation_id` into every request payload so the proxy can key cross-turn KV cache reuse. |
| 296 | |
| 297 | The flag is **off by default** so the benchmark is compatible with strict OpenAI-compatible frontends that reject unknown top-level fields. When benchmarking the multi-turn proxy you must pass it explicitly — otherwise every turn lands as a cache MISS and the bidirectional KV transfer path is never exercised. |
| 298 | |
| 299 | ```bash |
| 300 | python benchmarks/multi_turn/benchmark_serving_multi_turn.py \ |
| 301 | --model <MODEL> --served-model-name <NAME> \ |
| 302 | --url http://<proxy_host>:8000 \ |
| 303 | --input-file benchmarks/multi_turn/generate_multi_turn.json \ |
| 304 | --num-clients 2 --max-active-conversations 6 \ |
| 305 | --send-conversation-id |
| 306 | ``` |
| 307 | |
| 308 | ### Limitations |
| 309 | |
| 310 | - Requires a stateful proxy (or equivalent router) to track and forward `kv_transfer_params` between turns. |
| 311 | - Currently supported on CUDA with device-buffer KV cache. Host-buffer support (e.g., for Intel XPU) is planned for future work. |
| 312 | |
| 313 | !!! warning "Reasoning models with stripped thinking traces" |
| 314 | When using reasoning models (e.g. DeepSeek-R1) that produce thinking traces |
| 315 | (`<think>...</think>`), D's KV blocks cover the full token sequence including |
| 316 | thinking tokens. If the client strips thinking traces from the conversation |
| 317 | history before sending the next turn, the prompt P receives will be missing |
| 318 | tokens from the middle of what D generated. The block-alignment logic assumes |
| 319 | P's prompt is a prefix of D's sequence, so pulling KV blocks from D in this |
| 320 | case transfers cache computed for the wrong token positions, producing |
| 321 | incorrect results. |
| 322 | |
| 323 | We currently assume the router is able to detect such mismatch across turns. See [#43094](https://github.com/vllm-project/vllm/issues/43094). |
| 324 | |
| 325 | ## Multi-Instance Setup |
| 326 | |
| 327 | ### Multiple Prefiller Instances on Different Machines |
| 328 | |
| 329 | ```bash |
| 330 | # Prefiller 1 on Machine A (example IP: ${IP1}) |
| 331 | VLLM_NIXL_SIDE_CHANNEL_HOST=${IP1} \ |
| 332 | VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \ |
| 333 | UCX_NET_DEVICES=all \ |
| 334 | vllm serve Qwen/Qwen3-0.6B --port 8000 \ |
| 335 | --tensor-parallel-size 8 \ |
| 336 | --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail"}' |
| 337 | |
| 338 | # Prefiller 2 on Machine B (example IP: ${IP2}) |
| 339 | VLLM_NIXL_SIDE_CHANNEL_HOST=${IP2} \ |
| 340 | VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \ |
| 341 | UCX_NET_DEVICES=all \ |
| 342 | vllm serve Qwen/Qwen3-0.6B --port 8000 \ |
| 343 | --tensor-parallel-size 8 \ |
| 344 | --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail"}' |
| 345 | ``` |
| 346 | |
| 347 | ### Multiple Decoder Instances on Different Machines |
| 348 | |
| 349 | ```bash |
| 350 | # Decoder 1 on Machine C (example IP: ${IP3}) |
| 351 | VLLM_NIXL_SIDE_CHANNEL_HOST=${IP3} \ |
| 352 | VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \ |
| 353 | UCX_NET_DEVICES=all \ |
| 354 | vllm serve Qwen/Qwen3-0.6B --port 8000 \ |
| 355 | --tensor-parallel-size 8 \ |
| 356 | --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"}' |
| 357 | |
| 358 | # Decoder 2 on Machine D (example IP: ${IP4}) |
| 359 | VLLM_NIXL_SIDE_CHANNEL_HOST=${IP4} \ |
| 360 | VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \ |
| 361 | UCX_NET_DEVICES=all \ |
| 362 | vllm serve Qwen/Qwen3-0.6B --port 8000 \ |
| 363 | --tensor-parallel-size 8 \ |
| 364 | --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"}' |
| 365 | ``` |
| 366 | |
| 367 | ### Proxy for Multiple Instances |
| 368 | |
| 369 | ```bash |
| 370 | python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \ |
| 371 | --port 8192 \ |
| 372 | --prefiller-hosts ${IP1} ${IP2} \ |
| 373 | --prefiller-ports 8000 8000 \ |
| 374 | --decoder-hosts ${IP3} ${IP4} \ |
| 375 | --decoder-ports 8000 8000 |
| 376 | ``` |
| 377 | |
| 378 | For multi-host DP deployment, only need to provide the host/port of the head instances. |
| 379 | |
| 380 | ### KV Role Options |
| 381 | |
| 382 | - **kv_producer**: For prefiller instances that generate KV caches |
| 383 | - **kv_consumer**: For decoder instances that consume KV caches from prefiller |
| 384 | - **kv_both** (deprecated): Previously used as a catch-all when the role was not predetermined. This value is now deprecated for NixlConnector and will be removed in a future release. |
| 385 | |
| 386 | !!! warning |
| 387 | `kv_role="kv_both"` is deprecated for NixlConnector. Please set `kv_role="kv_producer"` for prefill instances and `kv_role="kv_consumer"` for decode instances. See [#33702](https://github.com/vllm-project/vllm/issues/33702) for details. |
| 388 | |
| 389 | ### KV Load Failure Policy |
| 390 | |
| 391 | The `kv_load_failure_policy` setting controls how the system handles failures when the decoder instance loads KV cache blocks from the prefiller instance: |
| 392 | |
| 393 | - **fail** (default): Immediately fail the request with an error when KV load fails. This prevents performance degradation by avoiding recomputation of prefill work on the decode instance. |
| 394 | - **recompute**: Recompute failed blocks locally on the decode instance. This may cause performance _jitter_ on decode instances as the scheduled prefill will delay and interfere with other decodes. Furthermore, decode instances are typically configured with low-latency optimizations. |
| 395 | |
| 396 | !!! warning |
| 397 | Using `kv_load_failure_policy="recompute"` can lead to performance degradation in production deployments. When KV loads fail, the decode instance will execute prefill work with decode-optimized configurations, which is inefficient and defeats the purpose of disaggregated prefilling. This also increases tail latency for other ongoing decode requests. |
| 398 | |
| 399 | ### For NVIDIA GB-series GPUs |
| 400 | |
| 401 | GB-series GPUs support multi-node NVLink. NIXL supports this capability, but KVCache must be registered as VMM during KVCache registration. To enable this feature, you need to set `--enable-cumem-allocator` or `--enable-sleep-mode` flags, and set `UCX_CUDA_IPC_ENABLE_MNNVL: 'y'` env. Otherwise, NIXL can only use RDMA/TCP for cross-node KVCache transfers. |
| 402 | |
| 403 | ## Experimental Feature |
| 404 | |
| 405 | ### Heterogeneous KV Layout support |
| 406 | |
| 407 | Support use case: Prefill with `LBHNC` and decode with `LBNHC` with experimental configuration |
| 408 | |
| 409 | ```bash |
| 410 | --kv-transfer-config '{..., "enable_permute_local_kv":"True"}' |
| 411 | ``` |
| 412 | |
| 413 | ## Metrics Reference |
| 414 | |
| 415 | vLLM periodically logs a `KV Transfer metrics` line summarising NIXL transfer |
| 416 | activity for the last reporting interval. Example output: |
| 417 | |
| 418 | ```text |
| 419 | KV Transfer metrics: Num successful transfers=4, Avg xfer time (ms)=1.381, |
| 420 | P90 xfer time (ms)=2.601, Avg post time (ms)=0.672, P90 post time (ms)=0.801, |
| 421 | Avg MB per transfer=2.25, Throughput (MB/s)=1629.549, Avg number of descriptors=72.0, |
| 422 | Num failed transfers=0, Num KV expired reqs=0 |
| 423 | ``` |
| 424 | |
| 425 | The table below describes each field. All timing values cover only the |
| 426 | successful transfers recorded in the current interval; failures are reported |
| 427 | as interval counts in the same line and via Prometheus (see |
| 428 | [Prometheus metrics](#prometheus-metrics) below). |
| 429 | |
| 430 | | Metric | Unit | Description | |
| 431 | | -------- | ------ | ------------- | |
| 432 | | `Num successful transfers` | count | Number of NIXL KV-block transfers that completed without error during the interval. A transfer corresponds to one prefill request's worth of KV cache being moved from the prefiller to the decoder (or vice versa in bidirectional mode). | |
| 433 | | `Avg xfer time (ms)` | ms | Mean end-to-end transfer duration (`xferDuration` in NIXL telemetry, converted from µs). Measured from when the request is posted to when the backend reports completion, so it includes both the posting step and the actual data movement. | |
| 434 | | `P90 xfer time (ms)` | ms | 90th-percentile transfer duration. Use this to identify tail latency: a large gap between average and P90 suggests occasional stragglers (e.g., network congestion or large KV blocks). | |
| 435 | | `Avg post time (ms)` | ms | Mean time to submit the transfer request to the RDMA backend (`postDuration` in NIXL telemetry). This is the synchronous cost of posting work to the NIC queue (descriptor setup, etc.) before the async data movement begins. | |
| 436 | | `P90 post time (ms)` | ms | 90th-percentile request-posting duration. Elevated P90 here (with low xfer P90) points to overhead in submitting requests rather than in the data transfer itself. | |
| 437 | | `Avg MB per transfer` | MB | Mean payload size per transfer, computed as `total bytes transferred / number of transfers`. Reflects the average KV cache footprint of a single request (sequence length × layers × head dimension × dtype bytes). | |
| 438 | | `Throughput (MB/s)` | MB/s | Effective bandwidth over the interval: `total MB transferred / total xfer time (s)` across all successful transfers. This is aggregate throughput, not per-request bandwidth. | |
| 439 | | `Avg number of descriptors` | count | Mean number of NIXL memory descriptors (scatter-gather segments) submitted per transfer. More descriptors indicate more fragmented or larger KV cache allocations; very high counts can increase descriptor-registration overhead. | |
| 440 | | `Num failed transfers` | count | Failed NIXL transfers, handshakes, and completion notifications (`send_notif`) in the interval, grouped because all are sporadic lower-transport-layer events. | |
| 441 | | `Num KV expired reqs` | count | Requests whose KV blocks expired before being read. Reported separately from the failure counts above because it is an autoscaler/lease-tuning signal rather than a transport issue. | |
| 442 | |
| 443 | ### Prometheus metrics |
| 444 | |
| 445 | In addition to the periodic log line, the following Prometheus metrics are |
| 446 | exported when NixlConnector is active: |
| 447 | |
| 448 | | Metric name | Type | Description | |
| 449 | | ------------- | ------ | ------------- | |
| 450 | | `vllm:nixl_xfer_time_seconds` | Histogram | Per-transfer RDMA copy duration (seconds). | |
| 451 | | `vllm:nixl_post_time_seconds` | Histogram | Time to submit the transfer request to the RDMA backend (seconds). | |
| 452 | | `vllm:nixl_bytes_transferred` | Histogram | Bytes moved per transfer. | |
| 453 | | `vllm:nixl_num_descriptors` | Histogram | Descriptor count per transfer. | |
| 454 | | `vllm:nixl_num_failed_transfers` | Counter | Cumulative count of failed NIXL KV-block transfers, including handshake and notification (`send_notif`) failures. These are grouped because all are sporadic lower-transport-layer events. | |
| 455 | | `vllm:nixl_num_failed_notifications` | Counter | Cumulative count of failed notifications (`send_notif`), retained for compatibility. These failures are also included in `vllm:nixl_num_failed_transfers`; do not sum the two counters. | |
| 456 | | `vllm:nixl_num_kv_expired_reqs` | Counter | Requests whose KV blocks expired on the prefiller before the decoder read them (tracked on the P instance). Kept separate from the failure counters above: KV expiry is an autoscaler/lease-tuning signal, not a transport issue. | |
| 457 | | `vllm:nixl_num_notifications_after_expiry` | Counter | Completion notifications that arrive for a request that is no longer tracked, usually because its KV lease expired first. Pull connector: counted on P, where it means the decoder may have read blocks that were already reused; each matches a "Potentially invalid KV blocks" error log. Push connector: counted on whichever instance tracks the request as neither a send nor a receive (P after the lease expired, or D after the receive completed or was abandoned); each matches an "Unrecognized request" error log. Counted per notification, not per request: each remote rank notifies separately, so with TP > 1 or heterogeneous TP one late request counts several times, and the value is not comparable to `vllm:nixl_num_kv_expired_reqs`. | |
| 458 | |
| 459 | !!! tip |
| 460 | High `vllm:nixl_num_kv_expired_reqs` indicates that the prefiller's lease |
| 461 | duration (`kv_lease_duration`) is too short for your network or workload. |
| 462 | Increase it via `--kv-transfer-config '{"kv_connector_extra_config": |
| 463 | {"kv_lease_duration": <seconds>}}'`. |
| 464 | |
| 465 | ## Example Scripts/Code |
| 466 | |
| 467 | Refer to these example scripts in the vLLM repository: |
| 468 | |
| 469 | - [run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh) |
| 470 | - [toy_proxy_server.py](../../tests/v1/kv_connector/nixl_integration/toy_proxy_server.py) |
| 471 | - [test_accuracy.py](../../tests/v1/kv_connector/nixl_integration/test_accuracy.py) |