@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 47455c424f

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

[Doc: ]fix various typos in multiple files (#23487)

Signed-off-by: Didier Durand <durand.didier@gmail.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Didier Durand committed Aug 25, 2025 at 02:04 UTC 47455c424f62a20b75a7cfd872e17c5ba11c9f3a

9 files changed +12 -12

.buildkite/nightly-benchmarks/nightly-descriptions.md

+1 -1

@@ -17,7 +17,7 @@ Latest reproduction guilde: [github issue link](https://github.com/vllm-project/

17 - SGLang: `lmsysorg/sglang:v0.3.2-cu121`

18 - LMDeploy: `openmmlab/lmdeploy:v0.6.1-cu12`

19 - TensorRT-LLM: `nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3`

20 - - *NOTE: we uses r24.07 as the current implementation only works for this version. We are going to bump this up.*

20 + - *NOTE: we use r24.07 as the current implementation only works for this version. We are going to bump this up.*

21 - Check [nightly-pipeline.yaml](nightly-pipeline.yaml) for the concrete docker images, specs and commands we use for the benchmark.

22 - Hardware

23 - 8x Nvidia A100 GPUs

docs/deployment/frameworks/anything-llm.md

+1 -1

@@ -18,7 +18,7 @@ vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096

19 - Download and install [Anything LLM desktop](https://anythingllm.com/desktop).

21 -- On the bottom left of open settings, AI Prooviders --> LLM:

21 +- On the bottom left of open settings, AI Providers --> LLM:

22 - LLM Provider: Generic OpenAI

23 - Base URL: http://{vllm server host}:{vllm server port}/v1

24 - Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`

docs/design/fused_moe_modular_kernel.md

+1 -1

@@ -226,7 +226,7 @@ Doing this will add the new implementation to the test suite.

226

227 The unit test file [test_modular_kernel_combinations.py](gh-file:tests/kernels/moe/test_modular_kernel_combinations.py) can also be executed as a standalone script.

228 Example: `python3 -m tests.kernels.moe.test_modular_kernel_combinations --pf-type PplxPrepareAndFinalize --experts-type BatchedTritonExperts`

229 -As a side-effect, this script can be used to test `FusedMoEPrepareAndFinalize` & `FusedMoEPermuteExpertsUnpermute` compatibility. When invoked

229 +As a side effect, this script can be used to test `FusedMoEPrepareAndFinalize` & `FusedMoEPermuteExpertsUnpermute` compatibility. When invoked

230 with incompatible types, the script will error.

231

232 ### How To Profile

docs/design/metrics.md

+1 -1

@@ -565,7 +565,7 @@ model and then validate those tokens with the larger model.

565 - `vllm:spec_decode_num_emitted_tokens_total` (Counter)

566

567 There is a PR under review (<gh-pr:12193>) to add "prompt lookup (ngram)"

568 -seculative decoding to v1. Other techniques will follow. We should

568 +speculative decoding to v1. Other techniques will follow. We should

569 revisit the v0 metrics in this context.

570

571 !!! note

docs/design/paged_attention.md

+1 -1

@@ -422,7 +422,7 @@ a total of 128 * 16 / 256 = 8 inner iterations for a warp to handle

422 a whole block of value tokens. And each `accs` in each thread

423 contains 8 elements that accumulated at 8 different head positions.

424 For the thread 0, the `accs` variable will have 8 elements, which

425 -are 0th, 32th … 224th elements of a value head that are accumulated

425 +are 0th, 32nd … 224th elements of a value head that are accumulated

426 from all assigned 8 tokens.

427

428 ## LV

docs/features/quantization/inc.md

+1 -1

@@ -7,7 +7,7 @@ Intel Gaudi supports quantization of various modules and functions, including, b

7 [Supported Modules\\Supported Functions\\Custom Patched Modules](https://docs.habana.ai/en/latest/PyTorch/Inference_on_PyTorch/Quantization/Inference_Using_FP8.html#supported-modules).

9 !!! note

10 - Measurement files are required to run quantized models with vLLM on Gaudi accelerators. The FP8 model calibration procedure is described in the [vllm-hpu-extention](https://github.com/HabanaAI/vllm-hpu-extension/tree/main/calibration/README.md) package.

10 + Measurement files are required to run quantized models with vLLM on Gaudi accelerators. The FP8 model calibration procedure is described in the [vLLM HPU extension](https://github.com/HabanaAI/vllm-hpu-extension/tree/main/calibration/README.md) package.

12 !!! note

13 `QUANT_CONFIG` is an environment variable that points to the measurement or quantization [JSON config file](https://docs.habana.ai/en/latest/PyTorch/Inference_on_PyTorch/Quantization/Inference_Using_FP8.html#supported-json-config-file-options).

docs/getting_started/installation/cpu.md

+3 -3

@@ -170,7 +170,7 @@ This value is 4GB by default. Larger space can support more concurrent requests,

170

171 First of all, please make sure the thread-binding and KV cache space are properly set and take effect. You can check the thread-binding by running a vLLM benchmark and observing CPU cores usage via `htop`.

172

173 -Inference batch size is a important parameter for the performance. Larger batch usually provides higher throughput, smaller batch provides lower latency. Tuning max batch size starts from default value to balance throughput and latency is an effective way to improve vLLM CPU performance on specific platforms. There are two important related parameters in vLLM:

173 +Inference batch size is an important parameter for the performance. Larger batch usually provides higher throughput, smaller batch provides lower latency. Tuning max batch size starts from default value to balance throughput and latency is an effective way to improve vLLM CPU performance on specific platforms. There are two important related parameters in vLLM:

174

175 - `--max-num-batched-tokens`, defines the limit of token numbers in a single batch, has more impacts on the first token performance. The default value is set as:

176 - Offline Inference: `4096 * world_size`

@@ -179,7 +179,7 @@ Inference batch size is a important parameter for the performance. Larger batch

179 - Offline Inference: `256 * world_size`

180 - Online Serving: `128 * world_size`

181

182 -vLLM CPU supports tensor parallel (TP) and pipeline parallel (PP) to leverage multiple CPU sockets and memory nodes. For more detials of tuning TP and PP, please refer to [Optimization and Tuning](../../configuration/optimization.md). For vLLM CPU, it is recommend to use TP and PP togther if there are enough CPU sockets and memory nodes.

182 +vLLM CPU supports tensor parallel (TP) and pipeline parallel (PP) to leverage multiple CPU sockets and memory nodes. For more details of tuning TP and PP, please refer to [Optimization and Tuning](../../configuration/optimization.md). For vLLM CPU, it is recommend to use TP and PP together if there are enough CPU sockets and memory nodes.

183

184 ### Which quantization configs does vLLM CPU support?

185

@@ -190,6 +190,6 @@ vLLM CPU supports tensor parallel (TP) and pipeline parallel (PP) to leverage mu

190

191 ### (x86 only) What is the purpose of `VLLM_CPU_MOE_PREPACK` and `VLLM_CPU_SGL_KERNEL`?

192

193 -- Both of them requires `amx` CPU flag.

193 +- Both of them require `amx` CPU flag.

194 - `VLLM_CPU_MOE_PREPACK` can provides better performance for MoE models

195 - `VLLM_CPU_SGL_KERNEL` can provides better performance for MoE models and small-batch scenarios.

docs/getting_started/installation/intel_gaudi.md

+2 -2

@@ -261,13 +261,13 @@ Lower value corresponds to less usable graph memory reserved for prefill stage,

261

262 User can also configure the strategy for capturing HPU Graphs for prompt and decode stages separately. Strategy affects the order of capturing graphs. There are two strategies implemented:

263

264 -- `max_bs` - graph capture queue will sorted in descending order by their batch sizes. Buckets with equal batch sizes are sorted by sequence length in ascending order (e.g. `(64, 128)`, `(64, 256)`, `(32, 128)`, `(32, 256)`, `(1, 128)`, `(1,256)`), default strategy for decode

264 +- `max_bs` - graph capture queue will be sorted in descending order by their batch sizes. Buckets with equal batch sizes are sorted by sequence length in ascending order (e.g. `(64, 128)`, `(64, 256)`, `(32, 128)`, `(32, 256)`, `(1, 128)`, `(1,256)`), default strategy for decode

265 - `min_tokens` - graph capture queue will be sorted in ascending order by the number of tokens each graph processes (`batch_size*sequence_length`), default strategy for prompt

266

267 When there's large amount of requests pending, vLLM scheduler will attempt to fill the maximum batch size for decode as soon as possible. When a request is finished, decode batch size decreases. When that happens, vLLM will attempt to schedule a prefill iteration for requests in the waiting queue, to fill the decode batch size to its previous state. This means that in a full load scenario, decode batch size is often at its maximum, which makes large batch size HPU Graphs crucial to capture, as reflected by `max_bs` strategy. On the other hand, prefills will be executed most frequently with very low batch sizes (1-4), which is reflected in `min_tokens` strategy.

268

269 !!! note

270 - `VLLM_GRAPH_PROMPT_RATIO` does not set a hard limit on memory taken by graphs for each stage (prefill and decode). vLLM will first attempt to use up entirety of usable prefill graph memory (usable graph memory * `VLLM_GRAPH_PROMPT_RATIO`) for capturing prefill HPU Graphs, next it will attempt do the same for decode graphs and usable decode graph memory pool. If one stage is fully captured, and there is unused memory left within usable graph memory pool, vLLM will attempt further graph capture for the other stage, until no more HPU Graphs can be captured without exceeding reserved memory pool. The behavior on that mechanism can be observed in the example below.

270 + `VLLM_GRAPH_PROMPT_RATIO` does not set a hard limit on memory taken by graphs for each stage (prefill and decode). vLLM will first attempt to use up entirety of usable prefill graph memory (usable graph memory * `VLLM_GRAPH_PROMPT_RATIO`) for capturing prefill HPU Graphs, next it will attempt to do the same for decode graphs and usable decode graph memory pool. If one stage is fully captured, and there is unused memory left within usable graph memory pool, vLLM will attempt further graph capture for the other stage, until no more HPU Graphs can be captured without exceeding reserved memory pool. The behavior on that mechanism can be observed in the example below.

271

272 Each described step is logged by vLLM server, as follows (negative values correspond to memory being released):

273

vllm/config/cache.py

+1 -1

@@ -116,7 +116,7 @@ class CacheConfig:

116 In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254),

117 some layers can skip tokens corresponding to prefill. This flag enables

118 attention metadata for eligible layers to be overriden with metadata

119 - necessary for implementating this optimization in some models (e.g. Gemma3n)

119 + necessary for implementing this optimization in some models (e.g. Gemma3n)

120 """

121

122 def compute_hash(self) -> str: