@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / cb97f2bfc5

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 03:30 UTC

[Docs] Replace two list with tables in intel_gaudi.md (#20414)

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

Michael Yao committed Jul 3, 2025 at 15:48 UTC cb97f2bfc54f0aff5a19d6bbdfdde1d9becbe738

1 file changed +24 -19

docs/getting_started/installation/intel_gaudi.md

+24 -19

@@ -198,7 +198,12 @@ INFO 08-01 21:37:59 hpu_model_runner.py:504] Decode bucket config (min, step, ma

198 INFO 08-01 21:37:59 hpu_model_runner.py:509] Generated 48 decode buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]

199 ```

200

201 -`min` determines the lowest value of the bucket. `step` determines the interval between buckets, and `max` determines the upper bound of the bucket. Furthermore, interval between `min` and `step` has special handling -- `min` gets multiplied by consecutive powers of two, until `step` gets reached. We call this the ramp-up phase and it is used for handling lower batch sizes with minimum wastage, while allowing larger padding on larger batch sizes.

201 +| Parameter | Description |

202 +|----------------|-----------------------------------------------------------------------------|

203 +| `min` | Determines the lowest value of the bucket. |

204 +| `step` | Determines the interval between buckets. |

205 +| `max` | Determines the upper bound of the bucket. |

206 +| Ramp-up phase | A special handling phase applied between `min` and `step`:<br/>- `min` is multiplied by consecutive powers of two until `step` is reached.<br/>- Minimizes resource wastage for small batch sizes.<br/>- Allows larger padding for larger batches. |

207

208 Example (with ramp-up):

209

@@ -349,28 +354,28 @@ Each described step is logged by vLLM server, as follows (negative values corres

354

355 - `VLLM_{phase}_{dim}_BUCKET_{param}` - collection of 12 environment variables configuring ranges of bucketing mechanism

356

352 - * `{phase}` is either `PROMPT` or `DECODE`

357 + * `{phase}` is either `PROMPT` or `DECODE`

358

354 - * `{dim}` is either `BS`, `SEQ` or `BLOCK`

359 + * `{dim}` is either `BS`, `SEQ` or `BLOCK`

360

356 - * `{param}` is either `MIN`, `STEP` or `MAX`

361 + * `{param}` is either `MIN`, `STEP` or `MAX`

362

358 - * Default values:

363 + * Default values:

364

360 - - Prompt:

361 - - batch size min (`VLLM_PROMPT_BS_BUCKET_MIN`): `1`

362 - - batch size step (`VLLM_PROMPT_BS_BUCKET_STEP`): `min(max_num_seqs, 32)`

363 - - batch size max (`VLLM_PROMPT_BS_BUCKET_MAX`): `min(max_num_seqs, 64)`

364 - - sequence length min (`VLLM_PROMPT_SEQ_BUCKET_MIN`): `block_size`

365 - - sequence length step (`VLLM_PROMPT_SEQ_BUCKET_STEP`): `block_size`

366 - - sequence length max (`VLLM_PROMPT_SEQ_BUCKET_MAX`): `max_model_len`

367 - - Decode:

368 - - batch size min (`VLLM_DECODE_BS_BUCKET_MIN`): `1`

369 - - batch size step (`VLLM_DECODE_BS_BUCKET_STEP`): `min(max_num_seqs, 32)`

370 - - batch size max (`VLLM_DECODE_BS_BUCKET_MAX`): `max_num_seqs`

371 - - sequence length min (`VLLM_DECODE_BLOCK_BUCKET_MIN`): `block_size`

372 - - sequence length step (`VLLM_DECODE_BLOCK_BUCKET_STEP`): `block_size`

373 - - sequence length max (`VLLM_DECODE_BLOCK_BUCKET_MAX`): `max(128, (max_num_seqs*max_model_len)/block_size)`

365 +| `{phase}` | Parameter | Env Variable | Value Expression |

366 +|-----------|-----------|--------------|------------------|

379

380 Additionally, there are HPU PyTorch Bridge environment variables impacting vLLM execution:

381