33f50773cb · tduncan/Nvidia-Optimization-Stack-VLLM

docs/benchmarking/cli.md

+4 -4

@@ -338,7 +338,7 @@ vllm bench serve \

338 --model meta-llama/Meta-Llama-3-8B-Instruct \

339 --dataset-name spec_bench \

340 --dataset-path "<YOUR_DOWNLOADED_PATH>/data/spec_bench/question.jsonl" \

341 - --num-prompts -1

341 + --num-prompts -1 \

342 --spec-bench-category "summarization"

343 ```

344

@@ -352,7 +352,7 @@ vllm bench serve \

352 First, download the dataset to a folder, using this one liner:

353

354 ```bash

355 -curl -LsSf https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py \| python3 -

355 +curl -LsSf https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py | python3 -

356 ```

357

358 The command supports also the following arguments:

@@ -388,7 +388,7 @@ vllm bench serve \

388 --model meta-llama/Llama-3.3-70B-Instruct \

389 --dataset-name speed_bench \

390 --dataset-path "<YOUR_DOWNLOADED_PATH>/data/speed_bench" \

391 - --num-prompts -1

391 + --num-prompts -1 \

392 --speed-bench-category "multilingual"

393 ```

394

@@ -398,7 +398,7 @@ Run all categories in the Throughput split (2k ISL):

398 vllm bench serve \

399 --model meta-llama/Llama-3.3-70B-Instruct \

400 --dataset-name speed_bench \

401 - --speed-bench-dataset-subset throughput_2k

401 + --speed-bench-dataset-subset throughput_2k \

402 --dataset-path "<YOUR_DOWNLOADED_PATH>/data/speed_bench/" \

403 --num-prompts -1

404 ```

docs/configuration/optimization.md

+1 -1

@@ -109,7 +109,7 @@ from vllm import LLM

109

110 # Combine pipeline and tensor parallelism

111 llm = LLM(

112 - model="meta-llama/Llama-3.3-70B-Instruct,

112 + model="meta-llama/Llama-3.3-70B-Instruct",

113 tensor_parallel_size=4,

114 pipeline_parallel_size=2,

115 )

docs/design/cuda_graphs.md

+2 -2

@@ -161,11 +161,11 @@ class AttentionCGSupport(enum.Enum):

161 ALWAYS = 3

162 """CUDA Graphs always supported; supports mixed-prefill-decode"""

163 UNIFORM_BATCH = 2

164 - """CUDA Graphs supported for batches the only contain query lengths that are

164 + """CUDA Graphs supported for batches that only contain query lengths that are

165 the same, this can be used for spec-decode

166 i.e. "decodes" are 1 + num_speculative_tokens"""

167 UNIFORM_SINGLE_TOKEN_DECODE = 1

168 - """CUDA Graphs supported for batches the only contain query_len==1 decodes"""

168 + """CUDA Graphs supported for batches that only contain query_len==1 decodes"""

169 NEVER = 0

170 """NO CUDA Graphs support"""

171 ```

docs/design/metrics.md

+1 -1

@@ -685,7 +685,7 @@ documentation for this option states:

685 > use of possibly costly and or blocking operations and hence might

686 > have a performance impact.

687

688 -The metrics were added by <https://github.com/vllm-project/vllm/pull/7089> and who up in an OpenTelemetry trace

688 +The metrics were added by <https://github.com/vllm-project/vllm/pull/7089> and show up in an OpenTelemetry trace

689 as:

690

691 ```text

docs/design/prefix_caching.md

+2 -2

@@ -27,7 +27,7 @@ In the example above, the KV cache in the first block can be uniquely identified

27 For `vllm serve`, you can control the hashing algorithm via `--prefix-caching-hash-algo`:

28 - `sha256` (default): Uses Python's `pickle` for serialization. Hashes may not be reproducible across different Python or vLLM versions.

29 - `sha256_cbor`: Uses `cbor2` for serialization, providing a reproducible, cross-language compatible hash. This is recommended for deterministic caching across environments.

30 - - `xxhash`: `Uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional `xxhash` package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on.

30 + - `xxhash`: Uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional `xxhash` package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on.

31 - `xxhash_cbor` combines canonical CBOR serialization with xxHash for reproducible hashing. Requires the optional `xxhash` package.

32

33 **A hashing example with multi-modality inputs**

@@ -197,7 +197,7 @@ As can be seen, block 3 is a new full block and is cached. However, it is redund

197

198 When a request is finished, we free all its blocks if no other requests are using them (reference count = 0). In this example, we free request 1 and block 2, 3, 4, 8 associated with it. We can see that the freed blocks are added to the tail of the free queue in the *reverse* order. This is because the last block of a request must hash more tokens and is less likely to be reused by other requests. As a result, it should be evicted first.

199

200 -![Free queue after a request us freed](../assets/design/prefix_caching/free.png)

200 +![Free queue after a request is freed](../assets/design/prefix_caching/free.png)

201

202 ### Eviction (LRU)

203

docs/features/quantization/gptqmodel.md

+1 -1

@@ -55,7 +55,7 @@ Here is an example of how to quantize `meta-llama/Llama-3.2-1B-Instruct`:

55

56 ## Running a quantized model with vLLM

57

58 -To run an GPTQModel quantized model with vLLM, you can use [DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2](https://huggingface.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2) with the following command:

58 +To run a GPTQModel quantized model with vLLM, you can use [DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2](https://huggingface.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2) with the following command:

59

60 ```bash

61 python examples/deployment/llm_engine_example.py \

docs/features/quantization/llm_compressor/int8_w8a8.md

-2

@@ -78,8 +78,6 @@ def tokenize(sample):

78 ds = ds.map(tokenize, remove_columns=ds.column_names)

79 ```

80

81 -</details>

82 -

81 ### 3. Applying Quantization

82

83 Now, apply the quantization algorithms:

docs/features/speculative_decoding/README.md

+1 -1

@@ -190,7 +190,7 @@ For mitigation strategies, please refer to the FAQ entry *Can the output of a pr

190

191 ## Known Feature Incompatibility

192

193 -1. Pipeline parallelism is not composible with speculative decoding as of `vllm<=0.15.0`

193 +1. Pipeline parallelism is not composable with speculative decoding as of `vllm<=0.15.0`

194 2. Speculative decoding with a draft models is not supported in `vllm<=0.10.0`

195

196 ## Resources for vLLM contributors

docs/features/tool_calling.md

+1 -1

@@ -338,7 +338,7 @@ Supported models:

338

339 Flags: `--tool-call-parser deepseek_v31 --chat-template {see_above}`

340

341 -### OpenAI OSS Models ('openai`)

341 +### OpenAI OSS Models (`openai`)

342

343 Supported models:

344

docs/models/pooling_models/README.md

+2 -2

@@ -143,7 +143,7 @@ enabling the corresponding APIs.

143

144 The [classify][vllm.LLM.classify] method outputs a probability vector for each prompt.

145 It is primarily designed for [classification models](classify.md).

146 -For more information about `LLM.embed`, see [this page](classify.md#offline-inference).

146 +For more information about `LLM.classify`, see [this page](classify.md#offline-inference).

147

148 ### `LLM.embed`

149

@@ -302,7 +302,7 @@ Pooling models now support token-wise task.

302

303 ### Score task

304

305 -`score` task have has been removed in v0.21, use `classify` instead. Only when a classification model outputs num_labels

305 +`score` task has been removed in v0.21, use `classify` instead. Only when a classification model outputs num_labels

306 equal to 1 can it be used as a scoring model and have its scoring API enabled.

307

308 ### Pooling multitask support

docs/models/pooling_models/scoring.md

+1 -1

@@ -440,7 +440,7 @@ More examples can be found here: [examples/pooling/score](../../../examples/pool

440

441 ## Supported Features

442

443 -AS cross-encoder models are a subset of classification models that accept two prompts as input and output num_labels equal to 1, cross-encoder features should be consistent with (sequence) classification. For more information, see [this page](classify.md#supported-features).

443 +As cross-encoder models are a subset of classification models that accept two prompts as input and output num_labels equal to 1, cross-encoder features should be consistent with (sequence) classification. For more information, see [this page](classify.md#supported-features).

444

445 ### Score Template

446

[Doc] Fix typos, grammar, and broken commands across docs (#46398)