1a55cfafcb · tduncan/Nvidia-Optimization-Stack-VLLM

docs/configuration/optimization.md

+1 -1

@@ -7,7 +7,7 @@ This guide covers optimization strategies and performance tuning for vLLM V1.

7

8 ## Preemption

9

10 -Due to the auto-regressive nature of transformer architecture, there are times when KV cache space is insufficient to handle all batched requests.

10 +Due to the autoregressive nature of transformer architecture, there are times when KV cache space is insufficient to handle all batched requests.

11 In such cases, vLLM can preempt requests to free up KV cache space for other requests. Preempted requests are recomputed when sufficient KV cache space becomes

12 available again. When this occurs, you may see the following warning:

13

docs/deployment/integrations/production-stack.md

+1 -1

@@ -4,7 +4,7 @@ Deploying vLLM on Kubernetes is a scalable and efficient way to serve machine le

4

5 * **Upstream vLLM compatibility** – It wraps around upstream vLLM without modifying its code.

6 * **Ease of use** – Simplified deployment via Helm charts and observability through Grafana dashboards.

7 -* **High performance** – Optimized for LLM workloads with features like multi-model support, model-aware and prefix-aware routing, fast vLLM bootstrapping, and KV cache offloading with [LMCache](https://github.com/LMCache/LMCache), among others.

7 +* **High performance** – Optimized for LLM workloads with features like multimodel support, model-aware and prefix-aware routing, fast vLLM bootstrapping, and KV cache offloading with [LMCache](https://github.com/LMCache/LMCache), among others.

8

9 If you are new to Kubernetes, don't worry: in the vLLM production stack [repo](https://github.com/vllm-project/production-stack), we provide a step-by-step [guide](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md) and a [short video](https://www.youtube.com/watch?v=EsTJbQtzj0g) to set up everything and get started in **4 minutes**!

10

docs/design/cuda_graphs.md

+2 -2

@@ -41,7 +41,7 @@ These features allow the most flexibility for cudagraph capture and compilation

41 * `NONE` — turn CUDA Graphs off. Good for debugging.

42 * `PIECEWISE` — a single-mode strategy (and past default). It is the most flexible: attention or other CUDA Graphs-incompatible operations stay eager, everything else goes into CUDA Graphs. Requires piecewise compilation.

43 * `FULL` — a single-mode strategy, which only captures full CUDA Graphs for non-uniform batches, then uniform-decode batches reuse the CUDA Graph of non-uniform batch of the same batch_size, since they are compatible; can be good for small models or workloads with small prompts.

44 -* `FULL_DECODE_ONLY` — full CUDA Graph for uniform decode, no cudagraph for prefill/mixed etc; suitable for decode instances in a P/D setup where prefill is not as important, this way we can save the memory needed for `PIECEWISE` CUDA Graphs.

44 +* `FULL_DECODE_ONLY` — full CUDA Graph for uniform decode, no cudagraph for prefill/mixed etc.; suitable for decode instances in a P/D setup where prefill is not as important, this way we can save the memory needed for `PIECEWISE` CUDA Graphs.

45 * `FULL_AND_PIECEWISE` — (default mode) full CUDA Graph for uniform decode, piecewise CUDA Graphs for others; generally the most performant setting, especially for low latency with small models or MoEs, but also requires the most memory and takes the longest to capture.

46

47 Defaults: If you’re on v1 with piecewise compilation, we default to `FULL_AND_PIECEWISE` for better performance, (for pooling models, it's still `PIECEWISE`). Otherwise, e.g. if piecewise compilation unavailable, we default to `NONE`.

@@ -49,7 +49,7 @@ Defaults: If you’re on v1 with piecewise compilation, we default to `FULL_AND_

49 While `NONE` , `PIECEWISE`, and `FULL` are single-mode configurations and simply equivalent to past implementations of eager execution, piecewise CUDA Graphs, and full CUDA Graphs respectively, `FULL_DECODE_ONLY` and `FULL_AND_PIECEWISE` are newly appended dual-mode configurations, which require dispatching to switch between concrete runtime modes according to runtime batches dynamically.

50

51 !!! note

52 - Here, the single-modes `NONE`, `PIECEWISE`, and `FULL` are treated as the runtime modes for CUDA Graphs dispatching. If using a dual-mode, the dispatcher will always dispatch to one of its member modes (plus a potantial `NONE` if no suitable CUDA Graph available), depending on the batch composition.

52 + Here, the single-modes `NONE`, `PIECEWISE`, and `FULL` are treated as the runtime modes for CUDA Graphs dispatching. If using a dual-mode, the dispatcher will always dispatch to one of its member modes (plus a potential `NONE` if no suitable CUDA Graph available), depending on the batch composition.

53

54 While cascade attention is not cudagraph compatible, it is now compatible with all possible cudagraph mode configurations. If a batch uses cascade attention, it always gets dispatched to `PIECEWISE` mode if available (otherwise `NONE`).

55

docs/design/optimization_levels.md

+1 -1

@@ -4,7 +4,7 @@

4

5 ## Overview

6

7 -vLLM now supports optimization levels (`-O0`, `-O1`, `-O2`, `-O3`). Optimization levels provide an intuitive mechnaism for users to trade startup time for performance. Higher levels have better performance but worse startup time. These optimization levels have associated defaults to help users get desired out of the box performance. Importantly, defaults set by optimization levels are purely defaults; explicit user settings will not be overwritten.

7 +vLLM now supports optimization levels (`-O0`, `-O1`, `-O2`, `-O3`). Optimization levels provide an intuitive mechanism for users to trade startup time for performance. Higher levels have better performance but worse startup time. These optimization levels have associated defaults to help users get desired out-of-the-box performance. Importantly, defaults set by optimization levels are purely defaults; explicit user settings will not be overwritten.

8

9 ## Level Summaries and Usage Examples

10 ```bash

docs/design/paged_attention.md

+3 -3

@@ -36,7 +36,7 @@ the input pointers `q`, `k_cache`, and `v_cache`, which point

36 to query, key, and value data on global memory that need to be read

37 and processed. The output pointer `out` points to global memory

38 where the result should be written. These four pointers actually

39 -refer to multi-dimensional arrays, but each thread only accesses the

39 +refer to multidimensional arrays, but each thread only accesses the

40 portion of data assigned to it. I have omitted all other runtime

41 parameters here for simplicity.

42

@@ -229,7 +229,7 @@ manner.

229

230 ## QK

231

232 -As shown the pseudo code below, before the entire for loop block, we

232 +As shown the pseudocode below, before the entire for loop block, we

233 fetch the query data for one token and store it in `q_vecs`. Then,

234 in the outer for loop, we iterate through different `k_ptrs` that

235 point to different tokens and prepare the `k_vecs` in the inner for

@@ -403,7 +403,7 @@ for ... { // Iteration over different blocks.

403 }

404 ```

405

406 -As shown in the above pseudo code, in the outer loop, similar to

406 +As shown in the above pseudocode, in the outer loop, similar to

407 `k_ptr`, `logits_vec` iterates over different blocks and reads

408 `V_VEC_SIZE` elements from `logits`. In the inner loop, each

409 thread reads `V_VEC_SIZE` elements from the same tokens as a

docs/models/supported_models.md

+1 -1

@@ -743,7 +743,7 @@ Some models are supported only via the [Transformers modeling backend](#transfor

743 - There's no PLE caching or out-of-memory swapping support, as described in [Google's blog](https://developers.googleblog.com/en/introducing-gemma-3n/). These features might be too model-specific for vLLM, and swapping in particular may be better suited for constrained setups.

744

745 !!! note

746 - For `InternVLChatModel`, only InternVL2.5 with Qwen2.5 text backbone (`OpenGVLab/InternVL2.5-1B` etc), InternVL3 and InternVL3.5 have video inputs support currently.

746 + For `InternVLChatModel`, only InternVL2.5 with Qwen2.5 text backbone (`OpenGVLab/InternVL2.5-1B` etc.), InternVL3 and InternVL3.5 have video inputs support currently.

747

748 !!! note

749 To use `TIGER-Lab/Mantis-8B-siglip-llama3`, you have to pass `--hf_overrides '{"architectures": ["MantisForConditionalGeneration"]}'` when running vLLM.

docs/serving/parallelism_scaling.md

+1 -1

@@ -154,7 +154,7 @@ vllm serve /path/to/the/model/in/the/container \

154

155 ## Optimizing network communication for tensor parallelism

156

157 -Efficient tensor parallelism requires fast inter-node communication, preferably through high-speed network adapters such as InfiniBand.

157 +Efficient tensor parallelism requires fast internode communication, preferably through high-speed network adapters such as InfiniBand.

158 To set up the cluster to use InfiniBand, append additional arguments like `--privileged -e NCCL_IB_HCA=mlx5` to the

159 [examples/online_serving/run_cluster.sh](../../examples/online_serving/run_cluster.sh) helper script.

160 Contact your system administrator for more information about the required flags.

docs/usage/security.md

+2 -2

@@ -10,7 +10,7 @@ All communications between nodes in a multi-node vLLM deployment are **insecure

10

11 ### Configuration Options for Inter-Node Communications

12

13 -The following options control inter-node communications in vLLM:

13 +The following options control internode communications in vLLM:

14

15 #### 1. **Environment Variables:**

16

@@ -28,7 +28,7 @@ The following options control inter-node communications in vLLM:

28

29 ### Notes on PyTorch Distributed

30

31 -vLLM uses PyTorch's distributed features for some inter-node communication. For

31 +vLLM uses PyTorch's distributed features for some internode communication. For

32 detailed information about PyTorch Distributed security considerations, please

33 refer to the [PyTorch Security

34 Guide](https://github.com/pytorch/pytorch/security/policy#using-distributed-features).

examples/online_serving/structured_outputs/structured_outputs.py

+1 -1

@@ -112,7 +112,7 @@ PARAMS: dict[ConstraintsFormat, dict[str, Any]] = {

112 "messages": [

113 {

114 "role": "user",

115 - "content": "Generate an SQL query to show the 'username' and 'email'from the 'users' table.",

115 + "content": "Generate an SQL query to show the 'username' and 'email' from the 'users' table.",

116 }

117 ],

118 "extra_body": {

vllm/entrypoints/openai/serving_responses.py

+1 -1

@@ -420,7 +420,7 @@ class OpenAIServingResponses(OpenAIServing):

420 context = HarmonyContext(messages, available_tools)

421 else:

422 if envs.VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT:

423 - # This is an feature in development for parsing

423 + # This is a feature in development for parsing

424 # tokens during generation instead of at the end

425 context = ParsableContext(

426 response_messages=messages,

vllm/model_executor/layers/fused_moe/shared_fused_moe.py

+2 -2

@@ -30,8 +30,8 @@ class SharedFusedMoE(FusedMoE):

30

31 # Disable shared expert overlap if:

32 # - we are using eplb, because of correctness issues

33 - # - we are using flashinfer with DP, since there nothint to gain

34 - # - we are using marlin kjernels

33 + # - we are using flashinfer with DP, since there nothing to gain

34 + # - we are using marlin kernels

35 self.use_overlapped = (

36 use_overlapped

37 and not (

vllm/model_executor/layers/quantization/kernels/scaled_mm/__init__.py

+1 -1

@@ -62,7 +62,7 @@ def choose_scaled_mm_linear_kernel(

62 continue

63

64 # If the current platform uses compute_capability,

65 - # make sure the kernel supports the compute cability.

65 + # make sure the kernel supports the compute capability.

66 is_supported, reason = kernel.is_supported(compute_capability)

67 if not is_supported:

68 failure_reasons.append(f"{kernel.__name__}: {reason}")

[Doc]: fixing typos in various files (#30540)