@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 2f1e8e8f54
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 03:15 UTC

Update default max_num_batch_tokens for chunked prefill (#11694)

Sachin Varghese committed Jan 2, 2025 at 19:25 UTC 2f1e8e8f54032e38998e90427aedf649c0beee39
1 file changed +4 -5
docs/source/usage/performance.md
+4 -5
@@ -32,8 +32,8 @@ You can enable the feature by specifying `--enable-chunked-prefill` in the comma
32 ```python
33 llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_chunked_prefill=True)
34 # Set max_num_batched_tokens to tune performance.
35 -# NOTE: 512 is the default max_num_batched_tokens for chunked prefill.
36 -# llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_chunked_prefill=True, max_num_batched_tokens=512)
35 +# NOTE: 2048 is the default max_num_batched_tokens for chunked prefill.
36 +# llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_chunked_prefill=True, max_num_batched_tokens=2048)
37 ```
38
39 By default, vLLM scheduler prioritizes prefills and doesn't batch prefill and decode to the same batch.
@@ -49,13 +49,12 @@ This policy has two benefits:
49 - It improves ITL and generation decode because decode requests are prioritized.
50 - It helps achieve better GPU utilization by locating compute-bound (prefill) and memory-bound (decode) requests to the same batch.
51
52 -You can tune the performance by changing `max_num_batched_tokens`.
53 -By default, it is set to 512, which has the best ITL on A100 in the initial benchmark (llama 70B and mixtral 8x22B).
52 +You can tune the performance by changing `max_num_batched_tokens`. By default, it is set to 2048.
53 Smaller `max_num_batched_tokens` achieves better ITL because there are fewer prefills interrupting decodes.
54 Higher `max_num_batched_tokens` achieves better TTFT as you can put more prefill to the batch.
55
56 - If `max_num_batched_tokens` is the same as `max_model_len`, that's almost the equivalent to the default scheduling policy (except that it still prioritizes decodes).
58 -- Note that the default value (512) of `max_num_batched_tokens` is optimized for ITL, and it may have lower throughput than the default scheduler.
57 +- Note that the default value (2048) of `max_num_batched_tokens` is optimized for ITL, and it may have lower throughput than the default scheduler.
58
59 We recommend you set `max_num_batched_tokens > 2048` for throughput.
60