A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 10, 06:30 UTC
[Docs] Nixl Usage recommend `fail` kv_load_failure_policy (#32198)
Signed-off-by: NickLucche <nlucches@redhat.com>
Nicolò Lucchesi committed
Jan 13, 2026 at 13:51 UTC
8c8653b67217ad41f89c10bc7fc097d0881bb63d
1 file changed
+16
-6
docs/features/nixl_connector_usage.md
+16
-6
@@ -50,7 +50,7 @@ VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
50
vllm serve Qwen/Qwen3-0.6B \
51
--port 8100 \
52
--enforce-eager \
53
- --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
53
+ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail"}'
54
```
55
56
### Consumer (Decoder) Configuration
@@ -65,7 +65,7 @@ VLLM_NIXL_SIDE_CHANNEL_PORT=5601 \
65
vllm serve Qwen/Qwen3-0.6B \
66
--port 8200 \
67
--enforce-eager \
68
- --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
68
+ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail"}'
69
```
70
71
### Proxy Server
@@ -110,7 +110,7 @@ VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
110
UCX_NET_DEVICES=all \
111
vllm serve Qwen/Qwen3-0.6B --port 8000 \
112
--tensor-parallel-size 8 \
113
- --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'
113
+ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail"}'
114
115
# Prefiller 2 on Machine B (example IP: ${IP2})
116
VLLM_NIXL_SIDE_CHANNEL_HOST=${IP2} \
@@ -118,7 +118,7 @@ VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
118
UCX_NET_DEVICES=all \
119
vllm serve Qwen/Qwen3-0.6B --port 8000 \
120
--tensor-parallel-size 8 \
121
- --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'
121
+ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail"}'
122
```
123
124
### Multiple Decoder Instances on Different Machines
@@ -130,7 +130,7 @@ VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
130
UCX_NET_DEVICES=all \
131
vllm serve Qwen/Qwen3-0.6B --port 8000 \
132
--tensor-parallel-size 8 \
133
- --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'
133
+ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"}'
134
135
# Decoder 2 on Machine D (example IP: ${IP4})
136
VLLM_NIXL_SIDE_CHANNEL_HOST=${IP4} \
@@ -138,7 +138,7 @@ VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
138
UCX_NET_DEVICES=all \
139
vllm serve Qwen/Qwen3-0.6B --port 8000 \
140
--tensor-parallel-size 8 \
141
- --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'
141
+ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"}'
142
```
143
144
### Proxy for Multiple Instances
@@ -164,6 +164,16 @@ For multi-host DP deployment, only need to provide the host/port of the head ins
164
NixlConnector currently does not distinguish `kv_role`; the actual prefiller/decoder roles are determined by the upper-level proxy (e.g., `toy_proxy_server.py` using `--prefiller-hosts` and `--decoder-hosts`).
165
Therefore, `kv_role` in `--kv-transfer-config` is effectively a placeholder and does not affect NixlConnector's behavior.
166
167
+### KV Load Failure Policy
168
+
169
+The `kv_load_failure_policy` setting controls how the system handles failures when the decoder instance loads KV cache blocks from the prefiller instance:
170
+
171
+- **fail** (recommended): Immediately fail the request with an error when KV load fails. This prevents performance degradation by avoiding recomputation of prefill work on the decode instance.
172
+- **recompute** (default): Recompute failed blocks locally on the decode instance. This may cause performance _jitter_ on decode instances as the scheduled prefill will delay and interfere with other decodes. Furthermore, decode instances are typically configured with low-latency optimizations.
173
+
174
+!!! warning
175
+ Using `kv_load_failure_policy="recompute"` can lead to performance degradation in production deployments. When KV loads fail, the decode instance will execute prefill work with decode-optimized configurations, which is inefficient and defeats the purpose of disaggregated prefilling. This also increases tail latency for other ongoing decode requests.
176
+
177
## Experimental Feature
178
179
### Heterogeneous KV Layout support