A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 9, 14:45 UTC
[DOC]: Add warning about max_num_batched_tokens and max_model_len when chunked prefill is disabled (#33109)
Signed-off-by: Vincent Gimenes <147169146+VincentG1234@users.noreply.github.com>
Vincent Gimenes committed
Jan 27, 2026 at 04:05 UTC
0b53bec60b9b41c9e16cefa2db367afb6a60628d
1 file changed
+4
docs/configuration/optimization.md
+4
@@ -47,6 +47,10 @@ You can tune the performance by adjusting `max_num_batched_tokens`:
47
- For optimal throughput, we recommend setting `max_num_batched_tokens > 8192` especially for smaller models on large GPUs.
48
- If `max_num_batched_tokens` is the same as `max_model_len`, that's almost the equivalent to the V0 default scheduling policy (except that it still prioritizes decodes).
49
50
+!!! warning
51
+ When chunked prefill is disabled, `max_num_batched_tokens` must be greater than `max_model_len`.
52
+ In that case, if `max_num_batched_tokens < max_model_len`, vLLM may crash at server start‑up.
53
+
54
```python
55
from vllm import LLM
56