8c853050e7 · tduncan/Nvidia-Optimization-Stack-VLLM

.readthedocs.yaml

+1

@@ -13,6 +13,7 @@ build:

13

14 mkdocs:

15 configuration: mkdocs.yaml

16 + fail_on_warning: true

17

18 # Optionally declare the Python requirements required to build your docs

19 python:

docs/features/nixl_connector_usage.md

+4 -4

@@ -9,7 +9,7 @@ NixlConnector is a high-performance KV cache transfer connector for vLLM's disag

9 Install the NIXL library: `uv pip install nixl`, as a quick start.

10

11 - Refer to [NIXL official repository](https://github.com/ai-dynamo/nixl) for more installation instructions

12 -- The specified required NIXL version can be found in [requirements/kv_connectors.txt](../../requirements/kv_connectors.txt) and other relevant config files

12 +- The specified required NIXL version can be found in [requirements/kv_connectors.txt](gh-file:requirements/kv_connectors.txt) and other relevant config files

13

14 ### Transport Configuration

15

@@ -154,6 +154,6 @@ python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \

154

155 Refer to these example scripts in the vLLM repository:

156

157 -- [run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh)

158 -- [toy_proxy_server.py](../../tests/v1/kv_connector/nixl_integration/toy_proxy_server.py)

159 -- [test_accuracy.py](../../tests/v1/kv_connector/nixl_integration/test_accuracy.py)

157 +- [run_accuracy_test.sh](gh-file:tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh)

158 +- [toy_proxy_server.py](gh-file:tests/v1/kv_connector/nixl_integration/toy_proxy_server.py)

159 +- [test_accuracy.py](gh-file:tests/v1/kv_connector/nixl_integration/test_accuracy.py)

docs/mkdocs/hooks/generate_argparse.py

+3 -2

@@ -32,8 +32,9 @@ def auto_mock(module, attr, max_mocks=50):

32 for _ in range(max_mocks):

33 try:

34 # First treat attr as an attr, then as a submodule

35 - return getattr(importlib.import_module(module), attr,

36 - importlib.import_module(f"{module}.{attr}"))

35 + with patch("importlib.metadata.version", return_value="0.0.0"):

36 + return getattr(importlib.import_module(module), attr,

37 + importlib.import_module(f"{module}.{attr}"))

38 except importlib.metadata.PackageNotFoundError as e:

39 raise e

40 except ModuleNotFoundError as e:

docs/models/generative_models.md

+1 -1

@@ -4,7 +4,7 @@ vLLM provides first-class support for generative models, which covers most of LL

4

5 In vLLM, generative models implement the[VllmModelForTextGeneration][vllm.model_executor.models.VllmModelForTextGeneration] interface.

6 Based on the final hidden states of the input, these models output log probabilities of the tokens to generate,

7 -which are then passed through [Sampler][vllm.model_executor.layers.sampler.Sampler] to obtain the final text.

7 +which are then passed through [Sampler][vllm.v1.sample.sampler.Sampler] to obtain the final text.

8

9 ## Configuration

10

docs/models/supported_models.md

+1 -1

@@ -29,7 +29,7 @@ _*Vision-language models currently accept only image inputs. Support for video i

29

30 If the Transformers model implementation follows all the steps in [writing a custom model](#writing-custom-models) then, when used with the Transformers backend, it will be compatible with the following features of vLLM:

31

32 -- All the features listed in the [compatibility matrix](../features/compatibility_matrix.md#feature-x-feature)

32 +- All the features listed in the [compatibility matrix](../features/README.md#feature-x-feature)

33 - Any combination of the following vLLM parallelisation schemes:

34 - Pipeline parallel

35 - Tensor parallel

docs/usage/README.md

+1 -1

@@ -1,6 +1,6 @@

1 # Using vLLM

2

3 -First, vLLM must be [installed](../getting_started/installation) for your chosen device in either a Python or Docker environment.

3 +First, vLLM must be [installed](../getting_started/installation/) for your chosen device in either a Python or Docker environment.

4

5 Then, vLLM supports the following usage patterns:

6

examples/online_serving/dashboards/grafana/README.md

+2 -2

@@ -11,9 +11,9 @@ vLLM performance and metrics.

11

12 ## Dashboard Descriptions

13

14 -- **[performance_statistics.json](./performance_statistics.json)**: Tracks performance metrics including latency and

14 +- **performance_statistics.json**: Tracks performance metrics including latency and

15 throughput for your vLLM service.

16 -- **[query_statistics.json](./query_statistics.json)**: Tracks query performance, request volume, and key

16 +- **query_statistics.json**: Tracks query performance, request volume, and key

17 performance indicators for your vLLM service.

18

19 ## Deployment Options

examples/online_serving/dashboards/perses/README.md

+2 -2

@@ -21,9 +21,9 @@ deployment methods:

21

22 ## Dashboard Descriptions

23

24 -- **[performance_statistics.yaml](./performance_statistics.yaml)**: Performance metrics with aggregated latency

24 +- **performance_statistics.yaml**: Performance metrics with aggregated latency

25 statistics

26 -- **[query_statistics.yaml](./query_statistics.yaml)**: Query performance and deployment metrics

26 +- **query_statistics.yaml**: Query performance and deployment metrics

27

28 ## Deployment Options

29

vllm/attention/ops/common.py

+19 -17

@@ -18,12 +18,14 @@ def _correct_attn_cp_out_kernel(outputs_ptr, new_output_ptr, lses_ptr,

18 final attention output.

19

20 Args:

21 - output: [ B, H, D ]

22 - lses : [ N, B, H ]

23 - cp, batch, q_heads, v_head_dim

24 - Return:

25 - output: [ B, H, D ]

26 - lse : [ B, H ]

21 + outputs_ptr (triton.PointerType):

22 + Pointer to input tensor of shape [ B, H, D ]

23 + lses_ptr (triton.PointerType):

24 + Pointer to input tensor of shape [ N, B, H ]

25 + new_output_ptr (triton.PointerType):

26 + Pointer to output tensor of shape [ B, H, D ]

27 + vlse_ptr (triton.PointerType):

28 + Pointer to output tensor of shape [ B, H ]

29 """

30 batch_idx = tl.program_id(axis=0).to(tl.int64)

31 head_idx = tl.program_id(axis=1).to(tl.int64)

@@ -81,19 +83,19 @@ class CPTritonContext:

83 self.inner_kernel[grid](*regular_args)

84

85

84 -def correct_attn_out(out: torch.Tensor, lses: torch.Tensor, cp_rank: int,

85 - ctx: CPTritonContext):

86 - """

87 - Apply the all-gathered lses to correct each local rank's attention

88 - output. we still need perform a cross-rank reduction to obtain the

89 - final attention output.

86 +def correct_attn_out(

87 + out: torch.Tensor, lses: torch.Tensor, cp_rank: int,

88 + ctx: CPTritonContext) -> tuple[torch.Tensor, torch.Tensor]:

89 + """Correct the attention output using the all-gathered lses.

90

91 Args:

92 - output: [ B, H, D ]

93 - lses : [ N, B, H ]

94 - Return:

95 - output: [ B, H, D ]

96 - lse : [ B, H ]

92 + out: Tensor of shape [ B, H, D ]

93 + lses: Tensor of shape [ N, B, H ]

94 + cp_rank: Current rank in the context-parallel group

95 + ctx: Triton context to avoid recompilation

96 +

97 + Returns:

98 + Tuple of (out, lse) with corrected attention and final log-sum-exp.

99 """

100 if ctx is None:

101 ctx = CPTritonContext()

vllm/inputs/data.py

+2 -2

@@ -287,8 +287,8 @@ class EncoderDecoderInputs(TypedDict):

287

288 SingletonInputs = Union[TokenInputs, EmbedsInputs, "MultiModalInputs"]

289 """

290 -A processed [`SingletonPrompt`][vllm.inputs.data.SingletonPrompt] which can be

291 -passed to [`vllm.sequence.Sequence`][].

290 +A processed [`SingletonPrompt`][vllm.inputs.data.SingletonPrompt] which can be

291 +passed to [`Sequence`][collections.abc.Sequence].

292 """

293

294 ProcessorInputs = Union[DecoderOnlyInputs, EncoderDecoderInputs]

vllm/model_executor/layers/fused_moe/layer.py

+6 -6

@@ -57,7 +57,7 @@ else:

57 FusedMoEPermuteExpertsUnpermute = None # type: ignore

58 FusedMoEPrepareAndFinalize = None # type: ignore

59

60 - def eplb_map_to_physical_and_record(

60 + def _eplb_map_to_physical_and_record(

61 topk_ids: torch.Tensor, expert_load_view: torch.Tensor,

62 logical_to_physical_map: torch.Tensor,

63 logical_replica_count: torch.Tensor,

@@ -65,6 +65,7 @@ else:

65 # CPU fallback: no EPLB so just return as is

66 return topk_ids

67

68 + eplb_map_to_physical_and_record = _eplb_map_to_physical_and_record

69

70 if is_rocm_aiter_moe_enabled():

71 from vllm.model_executor.layers.fused_moe.rocm_aiter_fused_moe import ( # noqa: E501

@@ -807,12 +808,11 @@ def maybe_roundup_hidden_size(

808 if necessary.

809

810 Args:

810 - hidden_size(int): Layer hidden-size

811 + hidden_size: Layer hidden-size

812 act_dtype: Data type of the layer activations.

812 - quant_config(FusedMoEQuantConfig): Fused MoE quantization configuration.

813 - moe_parallel_config(FusedMoEParallelConfig): Fused MoE parallelization

814 - strategy configuration.

815 -

813 + quant_config: Fused MoE quantization configuration.

814 + moe_parallel_config: Fused MoE parallelization strategy configuration.

815 +

816 Return:

817 Rounded up hidden_size if rounding up is required based on the configs.

818 Original hidden size otherwise.

vllm/model_executor/model_loader/weight_utils.py

+2 -2

@@ -13,7 +13,7 @@ from collections import defaultdict

13 from collections.abc import Generator

14 from contextlib import contextmanager

15 from pathlib import Path

16 -from typing import Any, Callable, Optional, Union

16 +from typing import IO, Any, Callable, Optional, Union

17

18 import filelock

19 import huggingface_hub.constants

@@ -102,7 +102,7 @@ def get_lock(model_name_or_path: Union[str, Path],

102 @contextmanager

103 def atomic_writer(filepath: Union[str, Path],

104 mode: str = 'w',

105 - encoding: Optional[str] = None):

105 + encoding: Optional[str] = None) -> Generator[IO]:

106 """

107 Context manager that provides an atomic file writing routine.

108

vllm/model_executor/models/qwen3_vl.py

+12 -8

@@ -1445,14 +1445,18 @@ class Qwen3VLForConditionalGeneration(nn.Module, SupportsMultiModal,

1445 **NOTE**: If mrope is enabled (default setting for Qwen3VL

1446 opensource models), the shape will be `(3, seq_len)`,

1447 otherwise it will be `(seq_len,).

1448 - pixel_values: Pixel values to be fed to a model.

1449 - `None` if no images are passed.

1450 - image_grid_thw: Tensor `(n_images, 3)` of image 3D grid in LLM.

1451 - `None` if no images are passed.

1452 - pixel_values_videos: Pixel values of videos to be fed to a model.

1453 - `None` if no videos are passed.

1454 - video_grid_thw: Tensor `(n_videos, 3)` of video 3D grid in LLM.

1455 - `None` if no videos are passed.

1448 + intermediate_tensors: Intermediate tensors from previous pipeline

1449 + stages.

1450 + inputs_embeds: Pre-computed input embeddings.

1451 + **kwargs: Additional keyword arguments including:

1452 + - pixel_values: Pixel values to be fed to a model.

1453 + `None` if no images are passed.

1454 + - image_grid_thw: Tensor `(n_images, 3)` of image 3D grid in

1455 + LLM. `None` if no images are passed.

1456 + - pixel_values_videos: Pixel values of videos to be fed to a

1457 + model. `None` if no videos are passed.

1458 + - video_grid_thw: Tensor `(n_videos, 3)` of video 3D grid in

1459 + LLM. `None` if no videos are passed.

1460 """

1461

1462 if intermediate_tensors is not None:

vllm/model_executor/models/zamba2.py

+2 -3

@@ -944,11 +944,10 @@ class Zamba2ForCausalLM(nn.Module, HasInnerState, IsHybrid):

944 hidden_states: torch.Tensor,

945 ) -> Optional[torch.Tensor]:

946 """Compute logits for next token prediction.

947 -

947 +

948 Args:

949 hidden_states: Hidden states from model forward pass

950 - sampling_metadata: Metadata for sampling process

951 -

950 +

951 Returns:

952 Logits for next token prediction

953 """

vllm/reasoning/granite_reasoning_parser.py

+5 -5

@@ -278,11 +278,11 @@ class GraniteReasoningParser(ReasoningParser):

278 content and normal (response) content.

279

280 Args:

281 - delta_text (str): Text to consider and parse content from.

282 - reasoning_content (str): reasoning content from current_text.

283 - response_content (str): response content from current_text.

284 - current_text (str): The full previous + delta text.

285 - response_seq_len(str): Len of the complete response sequence used.

281 + delta_text: Text to consider and parse content from.

282 + reasoning_content: reasoning content from current_text.

283 + response_content: response content from current_text.

284 + current_text: The full previous + delta text.

285 + response_seq_len: Len of the complete response sequence used.

286

287 Returns:

288 DeltaMessage: Message containing the parsed content.

vllm/transformers_utils/configs/radio.py

+17 -30

@@ -27,36 +27,23 @@ class RadioConfig(PretrainedConfig):

27 specified arguments, defining the model architecture.

28

29 Args:

30 - model_name (`str`, *optional*, defaults to "vit_base_patch16_224"):

31 - Name of the vision transformer model (e.g., "vit_base_patch16_224").

32 - Used to determine architecture dimensions from

33 - `VIT_TIMM_DIM_BY_NAME`.

34 - image_size (`int`, *optional*, defaults to 224):

35 - The size (resolution) of each image.

36 - patch_size (`int`, *optional*, defaults to 16):

37 - The size (resolution) of each patch.

38 - qkv_bias (`bool`, *optional*, defaults to True):

39 - Whether to add a bias to the queries, keys and values.

40 - qk_normalization (`bool`, *optional*, defaults to False):

41 - Whether to apply normalization to queries and keys.

42 - norm_type (`str`, *optional*, defaults to "layer_norm"):

43 - The normalization type to use.

44 - layer_norm_eps (`float`, *optional*, defaults to 1e-6):

45 - The epsilon used by the layer normalization layers.

46 - initializer_factor (`float`, *optional*, defaults to 1.0):

47 - A factor for initializing all weight matrices.

48 - hidden_act (`str`, *optional*, defaults to "gelu"):

49 - The non-linear activation function in the encoder.

50 - max_img_size (`int`, *optional*, defaults to 2048):

51 - Maximum image size for position embeddings.

52 - norm_mean (`tuple` or `list`, *optional*,

53 - defaults to (0.48145466, 0.4578275, 0.40821073)):

54 - Mean values for image normalization (RGB channels).

55 - norm_std (`tuple` or `list`, *optional*,

56 - defaults to (0.26862954, 0.26130258, 0.27577711)):

57 - Standard deviation values for image normalization (RGB channels).

58 - reg_tokens (`int`, *optional*):

59 - Number of register tokens to use.

30 + model_name: Name of the vision transformer model

31 + (e.g., "vit_base_patch16_224"). Used to determine architecture

32 + dimensions from `VIT_TIMM_DIM_BY_NAME`.

33 + image_size: The size (resolution) of each image.

34 + patch_size: The size (resolution) of each patch.

35 + qkv_bias: Whether to add a bias to the queries, keys and values.

36 + qk_normalization: Whether to apply normalization to queries and keys.

37 + norm_type: The normalization type to use.

38 + layer_norm_eps: The epsilon used by the layer normalization layers.

39 + initializer_factor: A factor for initializing all weight matrices.

40 + hidden_act: The non-linear activation function in the encoder.

41 + max_img_size: Maximum image size for position embeddings.

42 + norm_mean: Mean values for image normalization (RGB channels).

43 + Defaults to (0.48145466, 0.4578275, 0.40821073)).

44 + norm_std: Standard deviation values for image normalization

45 + (RGB channels). Defaults to (0.26862954, 0.26130258, 0.27577711)).

46 + reg_tokens: Number of register tokens to use.

47 """

48

49 model_type = "radio"

vllm/transformers_utils/dynamic_module.py

+1 -1

@@ -27,7 +27,7 @@ def try_get_class_from_dynamic_module(

27 **kwargs,

28 ) -> Optional[type]:

29 """

30 - As [transformers.dynamic_module_utils.get_class_from_dynamic_module][],

30 + As `transformers.dynamic_module_utils.get_class_from_dynamic_module`,

31 but ignoring any errors.

32 """

33 try:

vllm/v1/kv_offload/__init__.py

vllm/v1/kv_offload/backends/__init__.py

vllm/v1/kv_offload/worker/__init__.py

[Docs] Enable `fail_on_warning` for the docs build in CI (#25580)