A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 3, 06:45 UTC
[Docs][PD][NIXL] Bidirectional kv-cache transfer (#43097)
Signed-off-by: NickLucche <nlucches@redhat.com>
Nicolò Lucchesi committed
May 20, 2026 at 09:02 UTC
40651c020772b80f9ca80272aebe749fe01cd38a
1 file changed
+164
docs/features/nixl_connector_usage.md
+164
@@ -136,6 +136,170 @@ python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
136
- In bidirectional mode, the decoder caches KV blocks for multi-turn conversations. This TTL controls how long those blocks are held before being released. Unlike the prefiller lease, this TTL is not renewed via heartbeats.
137
- Example: `--kv-transfer-config '{"kv_connector_extra_config": {"decoder_kv_blocks_ttl": 600}}'`
138
139
+## Bidirectional KV Transfer (Multi-turn)
140
+
141
+In standard disaggregated prefilling, KV cache flows in one direction: Prefill (P) computes the KV cache and Decode (D) reads from P. For multi-turn conversations this is wasteful — D already holds the KV cache corresponding to the generated tokens from prior turns, yet P must recompute it from scratch on every new turn. Bidirectional KV transfer lets P **pull** existing KV blocks from D via RDMA before computing only the new tokens, significantly reducing Time-To-First-Token (TTFT) for long-prefill such as **multi-turn heavy scenarios**.
142
+
143
+### How it works
144
+
145
+The feature relies on a **stateful proxy** that sits between the client and the P/D instances. The proxy tracks `kv_transfer_params` returned by D at the end of each turn, and attaches them to the next turn's request so P knows which blocks to pull from D.
146
+
147
+```mermaid
148
+sequenceDiagram
149
+ participant Client
150
+ participant Proxy
151
+ participant P as Prefill (P)
152
+ participant D as Decode (D)
153
+
154
+ rect rgb(240, 240, 250)
155
+ note right of Client: Turn 1 — Cache Miss
156
+ Client->>Proxy: chat request + conversation_id
157
+ Proxy->>P: request (no remote blocks)
158
+ activate P
159
+ note over P: full prefill
160
+ P-->>Proxy: kv_transfer_params (P's blocks)
161
+ deactivate P
162
+ Proxy->>D: request + P's kv_transfer_params
163
+ activate D
164
+ D-->P: RDMA read (D pulls KV from P)
165
+ note over D: decode
166
+ D-->>Proxy: stream response + kv_transfer_params
167
+ deactivate D
168
+ note over Proxy: cache D's kv_transfer_params
169
+ Proxy-->>Client: response
170
+ end
171
+
172
+ rect rgb(255, 245, 235)
173
+ note right of Client: Turn 2+ — Cache Hit (Bidirectional)
174
+ Client->>Proxy: chat request + conversation_id
175
+ note over Proxy: lookup cached D blocks
176
+ Proxy->>P: request + D's remote_block_ids
177
+ activate P
178
+ P-->D: RDMA read (P pulls KV from D)
179
+ note over P: prefill new tokens only
180
+ P-->>Proxy: kv_transfer_params (P's blocks)
181
+ deactivate P
182
+ Proxy->>D: request + P's kv_transfer_params
183
+ activate D
184
+ D-->P: RDMA read (D pulls new KV from P)
185
+ note over D: decode
186
+ D-->>Proxy: stream response + kv_transfer_params
187
+ deactivate D
188
+ note over Proxy: update cached kv_transfer_params
189
+ Proxy-->>Client: response
190
+ end
191
+```
192
+
193
+**Turn 1 (cache miss):**
194
+
195
+1. Client sends a chat request with a `conversation_id` to the proxy.
196
+2. Proxy forwards the request to P with no remote block info — P computes the full KV cache.
197
+3. Proxy forwards the request to D along with P's `kv_transfer_params` (block IDs, engine ID, host/port).
198
+4. D reads KV blocks from P via RDMA (peer-to-peer pull), then generates the response.
199
+5. D streams the response back through the proxy. The final chunk includes D's own `kv_transfer_params`.
200
+6. Proxy caches D's `kv_transfer_params` keyed by `conversation_id`, then returns the response to the client.
201
+
202
+**Turn 2+ (cache hit — bidirectional):**
203
+
204
+1. Client sends the next turn with the same `conversation_id`.
205
+2. Proxy looks up cached `kv_transfer_params` from the previous turn and attaches D's `remote_block_ids` to the request sent to P.
206
+3. P reads the existing KV cache from D via RDMA (D→P pull), then computes KV only for the new tokens.
207
+4. Proxy forwards the request to D with P's updated `kv_transfer_params`.
208
+5. D reads the new KV blocks from P, generates the response, and returns updated `kv_transfer_params` which the proxy caches for the next turn.
209
+
210
+### Configuration
211
+
212
+Enable bidirectional KV transfer by setting `bidirectional_kv_xfer` in `kv_connector_extra_config` on **both** P and D instances:
213
+
214
+```bash
215
+vllm serve <MODEL> \
216
+ --kv-transfer-config '{
217
+ "kv_connector": "NixlConnector",
218
+ "kv_role": "kv_both",
219
+ "kv_connector_extra_config": {
220
+ "bidirectional_kv_xfer": true
221
+ }
222
+ }'
223
+```
224
+
225
+Additional configuration options in `kv_connector_extra_config`:
226
+
227
+| Parameter | Default | Description |
228
+| --------- | ------- | ----------- |
229
+| `bidirectional_kv_xfer` | `false` | Enable bidirectional D→P KV transfer. |
230
+| `kv_recompute_threshold` | `64` | Minimum number of remote tokens required to trigger a D→P pull. Below this threshold, P recomputes locally instead of pulling (to amortize transfer latency). |
231
+| `decoder_kv_blocks_ttl` | `480` | TTL (seconds) for KV blocks cached on D for bidirectional reuse. Blocks are released after this duration. Not renewed via heartbeats. |
232
+
233
+### Multi-turn proxy setup
234
+
235
+Use the provided multi-turn proxy to manage `kv_transfer_params` caching across conversation turns:
236
+
237
+```bash
238
+python examples/disaggregated/disaggregated_serving/disagg_proxy_multiturn.py \
239
+ --host 0.0.0.0 --port 8000 \
240
+ --prefiller-host <P_IP> --prefiller-port 8100 \
241
+ --decoder-host <D_IP> --decoder-port 8200
242
+```
243
+
244
+The proxy supports multiple P and D instances via round-robin:
245
+
246
+```bash
247
+python examples/disaggregated/disaggregated_serving/disagg_proxy_multiturn.py \
248
+ --host 0.0.0.0 --port 8000 \
249
+ --prefiller-hosts <P_IP1> <P_IP2> --prefiller-ports 8100 8100 \
250
+ --decoder-hosts <D_IP1> <D_IP2> --decoder-ports 8200 8200
251
+```
252
+
253
+### Client usage
254
+
255
+Include a `conversation_id` field in the request body to enable cross-turn KV reuse. Without it, the proxy cannot link turns and falls back to full recomputation.
256
+
257
+```bash
258
+# Turn 1
259
+curl http://localhost:8000/v1/chat/completions \
260
+ -H "Content-Type: application/json" \
261
+ -d '{
262
+ "model": "Qwen/Qwen3-0.6B",
263
+ "conversation_id": "session-42",
264
+ "messages": [
265
+ {"role": "user", "content": "What is vLLM?"}
266
+ ]
267
+ }'
268
+
269
+# Turn 2 — same conversation_id triggers bidirectional KV pull
270
+curl http://localhost:8000/v1/chat/completions \
271
+ -H "Content-Type: application/json" \
272
+ -d '{
273
+ "model": "Qwen/Qwen3-0.6B",
274
+ "conversation_id": "session-42",
275
+ "messages": [
276
+ {"role": "user", "content": "What is vLLM?"},
277
+ {"role": "assistant", "content": "vLLM is a high-throughput LLM serving engine..."},
278
+ {"role": "user", "content": "How does disaggregated prefilling work?"}
279
+ ]
280
+ }'
281
+```
282
+
283
+!!! note
284
+ The `conversation_id` field is a non-standard extension to the OpenAI API. It is consumed by the proxy and not forwarded to the vLLM engine.
285
+
286
+### Limitations
287
+
288
+- Requires a stateful proxy (or equivalent router) to track and forward `kv_transfer_params` between turns.
289
+- Currently supported on CUDA with device-buffer KV cache. Host-buffer support (e.g., for Intel XPU) is planned for future work.
290
+
291
+!!! warning "Reasoning models with stripped thinking traces"
292
+ When using reasoning models (e.g. DeepSeek-R1) that produce thinking traces
293
+ (`<think>...</think>`), D's KV blocks cover the full token sequence including
294
+ thinking tokens. If the client strips thinking traces from the conversation
295
+ history before sending the next turn, the prompt P receives will be missing
296
+ tokens from the middle of what D generated. The block-alignment logic assumes
297
+ P's prompt is a prefix of D's sequence, so pulling KV blocks from D in this
298
+ case transfers cache computed for the wrong token positions, producing
299
+ incorrect results.
300
+
301
+ We currently assume the router is able to detect such mismatch across turns. See [#43094](https://github.com/vllm-project/vllm/issues/43094).
302
+
303
## Multi-Instance Setup
304
305
### Multiple Prefiller Instances on Different Machines