A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
Update default max_num_batch_tokens for chunked prefill (#11694)
Sachin Varghese committed
Jan 2, 2025 at 19:25 UTC
2f1e8e8f54032e38998e90427aedf649c0beee39
1 file changed
+4
-5
docs/source/usage/performance.md
+4
-5
@@ -32,8 +32,8 @@ You can enable the feature by specifying `--enable-chunked-prefill` in the comma
32
```python
33
llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_chunked_prefill=True)
34
# Set max_num_batched_tokens to tune performance.
35
-# NOTE: 512 is the default max_num_batched_tokens for chunked prefill.
36
-# llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_chunked_prefill=True, max_num_batched_tokens=512)
35
+# NOTE: 2048 is the default max_num_batched_tokens for chunked prefill.
36
+# llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_chunked_prefill=True, max_num_batched_tokens=2048)
37
```
38
39
By default, vLLM scheduler prioritizes prefills and doesn't batch prefill and decode to the same batch.
@@ -49,13 +49,12 @@ This policy has two benefits:
49
- It improves ITL and generation decode because decode requests are prioritized.
50
- It helps achieve better GPU utilization by locating compute-bound (prefill) and memory-bound (decode) requests to the same batch.
51
52
-You can tune the performance by changing `max_num_batched_tokens`.
53
-By default, it is set to 512, which has the best ITL on A100 in the initial benchmark (llama 70B and mixtral 8x22B).
52
+You can tune the performance by changing `max_num_batched_tokens`. By default, it is set to 2048.
53
Smaller `max_num_batched_tokens` achieves better ITL because there are fewer prefills interrupting decodes.
54
Higher `max_num_batched_tokens` achieves better TTFT as you can put more prefill to the batch.
55
56
- If `max_num_batched_tokens` is the same as `max_model_len`, that's almost the equivalent to the default scheduling policy (except that it still prioritizes decodes).
58
-- Note that the default value (512) of `max_num_batched_tokens` is optimized for ITL, and it may have lower throughput than the default scheduler.
57
+- Note that the default value (2048) of `max_num_batched_tokens` is optimized for ITL, and it may have lower throughput than the default scheduler.
58
59
We recommend you set `max_num_batched_tokens > 2048` for throughput.
60