[Docs] Clarify admission control limits apply server-wide, not per DP rank (#55124)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Nicolò Lucchesi committed
Sep 7, 2026 at 16:08 UTC
cd64c2dea9c72af333de7ec05293d54bbf1bd128
2 files changed
+14
-1
docs/serving/data_parallel_deployment.md
+2
@@ -26,6 +26,8 @@ vLLM supports "self-contained" data parallel deployments that expose a single AP
26
27
It can be configured by simply including e.g. `--data-parallel-size=4` in the vllm serve command line arguments. This will require 4 GPUs. It can be combined with tensor parallel, for example `--data-parallel-size=4 --tensor-parallel-size=2`, which would require 8 GPUs. When sizing DP deployments, remember that `--max-num-seqs` applies per DP rank.
28
29
+By contrast, the admission control limits `--max-num-queued-reqs` and `--max-num-queued-tokens` apply to the whole server, not per DP rank: each API server process counts in-flight requests and prefill backlog across all DP ranks it routes to. For example, with `--data-parallel-size=4 --max-num-seqs=256 --max-num-queued-reqs=256`, the deployment rejects new requests once 256 are in-flight in total, even though ranks could jointly run 1024. To keep ranks saturating, size the cap as roughly `data-parallel-size * max-num-seqs` plus the desired queue depth.
30
+
31
Running a single data parallel deployment across multiple nodes requires a different `vllm serve` to be run on each node, specifying which DP ranks should run on that node. In this case, there will still be a single HTTP entrypoint - the API server(s) will run only on one node, but it doesn't necessarily need to be co-located with the DP ranks.
32
33
This will run DP=4, TP=2 on a single 8-GPU node:
vllm/config/scheduler.py
+12
-1
@@ -76,7 +76,13 @@ class SchedulerConfig:
76
at the same time, or None for no limit. When the limit is reached, new
77
requests are rejected with HTTP 503 so the client can retry on another
78
instance. This bounds vLLM's otherwise unbounded request queue and is
79
- primarily a coarse capacity valve."""
79
+ primarily a coarse capacity valve.
80
+
81
+ Unlike ``max_num_seqs``, which applies per data-parallel rank, this
82
+ limit is enforced in the API server process and counts in-flight
83
+ requests across all DP ranks it routes to. Size it as roughly
84
+ ``data_parallel_size * max_num_seqs`` plus the desired queue depth if
85
+ it should not bind before per-rank admission does."""
86
87
max_num_queued_tokens: int | None = Field(default=None, ge=0)
88
"""Maximum total prompt tokens of requests currently in the prefill
@@ -89,6 +95,11 @@ class SchedulerConfig:
95
prefill-decode setup this maps directly to the prefill pool's
96
capacity.
97
98
+ Like ``max_num_queued_reqs``, this limit is enforced in the API
99
+ server process and covers the prefill backlog across all DP ranks it
100
+ routes to, so ``prefill_throughput`` in the formula above is the
101
+ aggregate throughput of the deployment.
102
+
103
Note: the count is conservative. A partially prefilled request
104
still contributes its full ``prompt_len`` until it transitions out
105
of the prefill phase, because the scheduler's per-iteration