A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 2, 11:00 UTC
docs: clarify ITL acronym in optimization docs (#43922)
Signed-off-by: chunyang.wen <chunyang.wen@gmail.com>
Chunyang Wen committed
May 29, 2026 at 22:40 UTC
f191d5630e371a575bee2ab4eedc9332dcfd43f9
1 file changed
+2
-2
docs/configuration/optimization.md
+2
-2
@@ -46,14 +46,14 @@ In V1, **chunked prefill is enabled by default whenever possible**. With chunked
46
47
This policy has two benefits:
48
49
-- It improves ITL and generation decode because decode requests are prioritized.
49
+- It improves inter-token latency (ITL) and generation decode because decode requests are prioritized.
50
- It helps achieve better GPU utilization by locating compute-bound (prefill) and memory-bound (decode) requests to the same batch.
51
52
### Performance Tuning with Chunked Prefill
53
54
You can tune the performance by adjusting `max_num_batched_tokens`:
55
56
-- Smaller values (e.g., 2048) achieve better inter-token latency (ITL) because there are fewer prefills slowing down decodes.
56
+- Smaller values (e.g., 2048) achieve better ITL because there are fewer prefills slowing down decodes.
57
- Higher values achieve better time to first token (TTFT) as you can process more prefill tokens in a batch.
58
- For optimal throughput, we recommend setting `max_num_batched_tokens > 8192` especially for smaller models on large GPUs.
59
- If `max_num_batched_tokens` is the same as `max_model_len`, that's almost the equivalent to the V0 default scheduling policy (except that it still prioritizes decodes).