A high-throughput and memory-efficient inference and serving engine for LLMs
[Docs][KV Connector][NIXL] document KV Transfer stat logging and Prometheus metrics (#44055)
Signed-off-by: Sai Sridhar <tarrasridhar1154@gmail.com>
Sai Sridhar Tarra committed
Jun 12, 2026 at 21:09 UTC
a30addc7548a9a8b9b3323a7bc3eb7d7c4895d1c
1 file changed
+48
docs/features/nixl_connector_usage.md
+48
@@ -423,6 +423,54 @@ To enable this feature:
423
--kv-transfer-config '{..., "kv_connector_extra_config": {"enable_cross_layers_blocks": "True"}}'
424
```
425
426
+## Metrics Reference
427
+
428
+vLLM periodically logs a `KV Transfer metrics` line summarising NIXL transfer
429
+activity for the last reporting interval. Example output:
430
+
431
+```text
432
+KV Transfer metrics: Num successful transfers=4, Avg xfer time (ms)=1.381,
433
+P90 xfer time (ms)=2.601, Avg post time (ms)=0.672, P90 post time (ms)=0.801,
434
+Avg MB per transfer=2.25, Throughput (MB/s)=1629.549, Avg number of descriptors=72.0
435
+```
436
+
437
+The table below describes each field. All timing values cover only the
438
+successful transfers recorded in the current interval; failed transfers are
439
+counted separately via Prometheus (see
440
+[Prometheus metrics](#prometheus-metrics) below).
441
+
442
+| Metric | Unit | Description |
443
+| -------- | ------ | ------------- |
444
+| `Num successful transfers` | count | Number of NIXL KV-block transfers that completed without error during the interval. A transfer corresponds to one prefill request's worth of KV cache being moved from the prefiller to the decoder (or vice versa in bidirectional mode). |
445
+| `Avg xfer time (ms)` | ms | Mean end-to-end transfer duration (`xferDuration` in NIXL telemetry, converted from µs). Measured from when the request is posted to when the backend reports completion, so it includes both the posting step and the actual data movement. |
446
+| `P90 xfer time (ms)` | ms | 90th-percentile transfer duration. Use this to identify tail latency: a large gap between average and P90 suggests occasional stragglers (e.g., network congestion or large KV blocks). |
447
+| `Avg post time (ms)` | ms | Mean time to submit the transfer request to the RDMA backend (`postDuration` in NIXL telemetry). This is the synchronous cost of posting work to the NIC queue (descriptor setup, etc.) before the async data movement begins. |
448
+| `P90 post time (ms)` | ms | 90th-percentile request-posting duration. Elevated P90 here (with low xfer P90) points to overhead in submitting requests rather than in the data transfer itself. |
449
+| `Avg MB per transfer` | MB | Mean payload size per transfer, computed as `total bytes transferred / number of transfers`. Reflects the average KV cache footprint of a single request (sequence length × layers × head dimension × dtype bytes). |
450
+| `Throughput (MB/s)` | MB/s | Effective bandwidth over the interval: `total MB transferred / total xfer time (s)` across all successful transfers. This is aggregate throughput, not per-request bandwidth. |
451
+| `Avg number of descriptors` | count | Mean number of NIXL memory descriptors (scatter-gather segments) submitted per transfer. More descriptors indicate more fragmented or larger KV cache allocations; very high counts can increase descriptor-registration overhead. |
452
+
453
+### Prometheus metrics
454
+
455
+In addition to the periodic log line, the following Prometheus metrics are
456
+exported when NixlConnector is active:
457
+
458
+| Metric name | Type | Description |
459
+| ------------- | ------ | ------------- |
460
+| `vllm:nixl_xfer_time_seconds` | Histogram | Per-transfer RDMA copy duration (seconds). |
461
+| `vllm:nixl_post_time_seconds` | Histogram | Time to submit the transfer request to the RDMA backend (seconds). |
462
+| `vllm:nixl_bytes_transferred` | Histogram | Bytes moved per transfer. |
463
+| `vllm:nixl_num_descriptors` | Histogram | Descriptor count per transfer. |
464
+| `vllm:nixl_num_failed_transfers` | Counter | Cumulative count of failed NIXL KV-block transfers. |
465
+| `vllm:nixl_num_failed_notifications` | Counter | Cumulative count of failed completion notifications (`send_notif`). |
466
+| `vllm:nixl_num_kv_expired_reqs` | Counter | Requests whose KV blocks expired on the prefiller before the decoder read them (tracked on the P instance). |
467
+
468
+!!! tip
469
+ High `vllm:nixl_num_kv_expired_reqs` indicates that the prefiller's lease
470
+ duration (`kv_lease_duration`) is too short for your network or workload.
471
+ Increase it via `--kv-transfer-config '{"kv_connector_extra_config":
472
+ {"kv_lease_duration": <seconds>}}'`.
473
+
474
## Example Scripts/Code
475
476
Refer to these example scripts in the vLLM repository: