A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 12, 23:30 UTC
[Doc]: fix typos in various files (#29010)
Signed-off-by: Didier Durand <durand.didier@gmail.com>
Didier Durand committed
Nov 19, 2025 at 13:56 UTC
09540cd918a5f7d776d7f7e0abec78fbc03938ad
6 files changed
+6
-6
docs/deployment/frameworks/skypilot.md
+1
-1
@@ -4,7 +4,7 @@
4
<img src="https://imgur.com/yxtzPEu.png" alt="vLLM"/>
5
</p>
6
7
-vLLM can be **run and scaled to multiple service replicas on clouds and Kubernetes** with [SkyPilot](https://github.com/skypilot-org/skypilot), an open-source framework for running LLMs on any cloud. More examples for various open models, such as Llama-3, Mixtral, etc, can be found in [SkyPilot AI gallery](https://skypilot.readthedocs.io/en/latest/gallery/index.html).
7
+vLLM can be **run and scaled to multiple service replicas on clouds and Kubernetes** with [SkyPilot](https://github.com/skypilot-org/skypilot), an open-source framework for running LLMs on any cloud. More examples for various open models, such as Llama-3, Mixtral, etc., can be found in [SkyPilot AI gallery](https://skypilot.readthedocs.io/en/latest/gallery/index.html).
8
9
## Prerequisites
10
docs/design/prefix_caching.md
+1
-1
@@ -1,6 +1,6 @@
1
# Automatic Prefix Caching
2
3
-Prefix caching kv-cache blocks is a popular optimization in LLM inference to avoid redundant prompt computations. The core idea is simple – we cache the kv-cache blocks of processed requests, and reuse these blocks when a new request comes in with the same prefix as previous requests. Since prefix caching is almost a free lunch and won’t change model outputs, it has been widely used by many public endpoints (e.g., OpenAI, Anthropic, etc) and most open source LLM inference frameworks (e.g., SGLang).
3
+Prefix caching kv-cache blocks is a popular optimization in LLM inference to avoid redundant prompt computations. The core idea is simple – we cache the kv-cache blocks of processed requests, and reuse these blocks when a new request comes in with the same prefix as previous requests. Since prefix caching is almost a free lunch and won’t change model outputs, it has been widely used by many public endpoints (e.g., OpenAI, Anthropic, etc.) and most open source LLM inference frameworks (e.g., SGLang).
4
5
While there are many ways to implement prefix caching, vLLM chooses a hash-based approach. Specifically, we hash each kv-cache block by the tokens in the block and the tokens in the prefix before the block:
6
docs/features/nixl_connector_usage.md
+1
-1
@@ -158,7 +158,7 @@ python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
158
159
## Experimental Feature
160
161
-### Heterogenuous KV Layout support
161
+### Heterogeneous KV Layout support
162
163
Support use case: Prefill with 'HND' and decode with 'NHD' with experimental configuration
164
docs/getting_started/quickstart.md
+1
-1
@@ -286,7 +286,7 @@ If desired, you can also manually set the backend of your choice by configuring
286
- On NVIDIA CUDA: `FLASH_ATTN`, `FLASHINFER` or `XFORMERS`.
287
- On AMD ROCm: `TRITON_ATTN`, `ROCM_ATTN`, `ROCM_AITER_FA` or `ROCM_AITER_UNIFIED_ATTN`.
288
289
-For AMD ROCm, you can futher control the specific Attention implementation using the following variables:
289
+For AMD ROCm, you can further control the specific Attention implementation using the following variables:
290
291
- Triton Unified Attention: `VLLM_ROCM_USE_AITER=0 VLLM_V1_USE_PREFILL_DECODE_ATTENTION=0 VLLM_ROCM_USE_AITER_MHA=0`
292
- AITER Unified Attention: `VLLM_ROCM_USE_AITER=1 VLLM_USE_AITER_UNIFIED_ATTENTION=1 VLLM_V1_USE_PREFILL_DECODE_ATTENTION=0 VLLM_ROCM_USE_AITER_MHA=0`
tests/v1/ec_connector/integration/README.md
+1
-1
@@ -113,7 +113,7 @@ Quick sanity check:
113
114
- Outputs differ between baseline and disagg
115
- Server startup fails
116
-- Encoder cache not found (should fallback to local execution)
116
+- Encoder cache not found (should fall back to local execution)
117
- Proxy routing errors
118
119
## Notes
vllm/multimodal/evs.py
+1
-1
@@ -185,7 +185,7 @@ def recompute_mrope_positions(
185
186
Args:
187
input_ids: (N,) All input tokens of the prompt (entire sequence).
188
- multimodal_positions: List of mrope positsions for each media.
188
+ multimodal_positions: List of mrope positions for each media.
189
mrope_positions: Existing mrope positions (4, N) for entire sequence.
190
num_computed_tokens: A number of computed tokens so far.
191
vision_start_token_id: Token indicating start of vision media.