A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Doc] use power of 2 (#23172)
Tialo committed
Aug 19, 2025 at 13:16 UTC
2c3f557f08880b3cff86470b7ee358a047072990
1 file changed
+1
-1
docs/configuration/optimization.md
+1
-1
@@ -48,7 +48,7 @@ You can tune the performance by adjusting `max_num_batched_tokens`:
48
49
- Smaller values (e.g., 2048) achieve better inter-token latency (ITL) because there are fewer prefills slowing down decodes.
50
- Higher values achieve better time to first token (TTFT) as you can process more prefill tokens in a batch.
51
-- For optimal throughput, we recommend setting `max_num_batched_tokens > 8096` especially for smaller models on large GPUs.
51
+- For optimal throughput, we recommend setting `max_num_batched_tokens > 8192` especially for smaller models on large GPUs.
52
- If `max_num_batched_tokens` is the same as `max_model_len`, that's almost the equivalent to the V0 default scheduling policy (except that it still prioritizes decodes).
53
54
```python