f243abc92d · tduncan/Nvidia-Optimization-Stack-VLLM

docs/contributing/deprecation_policy.md

+1 -1

@@ -46,7 +46,7 @@ warning (e.g., "This will be removed in v0.10.0").

46 - GitHub Issue (RFC) for feedback

47 - Documentation and use of the `@typing_extensions.deprecated` decorator for Python APIs

48

49 -### 2.Deprecated (Off By Default)

49 +### 2. Deprecated (Off By Default)

50

51 - **Action**: Feature is disabled by default, but can still be re-enabled via a

52 CLI flag or environment variable. Feature throws an error when used without

docs/contributing/model/basic.md

+1 -1

@@ -118,7 +118,7 @@ To support a model with interleaving sliding windows, we need to take care of th

118 - Make sure the model's `config.json` contains `layer_types`.

119 - In the modeling code, parse the correct sliding window value for every layer, and pass it to the attention layer's `per_layer_sliding_window` argument. For reference, check [this line](https://github.com/vllm-project/vllm/blob/996357e4808ca5eab97d4c97c7d25b3073f46aab/vllm/model_executor/models/llama.py#L171).

120

121 -With these two steps, interleave sliding windows should work with the model.

121 +With these two steps, interleaved sliding windows should work with the model.

122

123 ### How to support models that use Mamba?

124

docs/deployment/frameworks/cerebrium.md

+1 -1

@@ -59,7 +59,7 @@ Then, run the following code to deploy it to the cloud:

59 cerebrium deploy

60 ```

61

62 -If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case`/run`)

62 +If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case `/run`)

63

64 ??? console "Command"

65

docs/deployment/frameworks/hf_inference_endpoints.md

+1 -1

@@ -70,7 +70,7 @@ This method applies to models with the [`transformers` library tag](https://hugg

70

71 ![Locate deploy button](../../assets/deployment/hf-inference-endpoints-locate-deploy-button.png)

72

73 -3. Click to **Deploy** button > **HF Inference Endpoints**. You will be taken to the Inference Endpoints interface to configure the deployment.

73 +3. Click the **Deploy** button > **HF Inference Endpoints**. You will be taken to the Inference Endpoints interface to configure the deployment.

74

75 ![Click deploy button](../../assets/deployment/hf-inference-endpoints-click-deploy-button.png)

76

docs/deployment/integrations/production-stack.md

+1 -1

@@ -10,7 +10,7 @@ If you are new to Kubernetes, don't worry: in the vLLM production stack [repo](h

10

11 ## Pre-requisite

12

13 -Ensure that you have a running Kubernetes environment with GPU (you can follow [this tutorial](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md) to install a Kubernetes environment on a bare-medal GPU machine).

13 +Ensure that you have a running Kubernetes environment with GPU (you can follow [this tutorial](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md) to install a Kubernetes environment on a bare-metal GPU machine).

14

15 ## Deployment using vLLM production stack

16

docs/design/custom_op.md

+2 -2

@@ -40,9 +40,9 @@ Furthermore, vLLM decides whether to enable or disable a `CustomOp` based on `co

40 By default, if `compilation_config.backend == "inductor"` and `compilation_config.mode != CompilationMode.NONE`, a `none` will be appended into `compilation_config.custom_ops`, otherwise a `all` will be appended. In other words, this means `CustomOp` will be disabled in some platforms (i.e., those use `inductor` as dafault backend for `torch.compile`) when running with torch compile mode. In this case, Inductor generates (fused) Triton kernels for those disabled custom ops.

41

42 !!! note

43 - For multi-modal models, vLLM has enforece enabled some custom ops to use device-specific deep-optimized kernels for better performance in ViT part, such as `MMEncoderAttention` and `ApplyRotaryEmb`. We can also pass a `enforce_enable=True` param to the `__init__()` method of the `CustomOp` to enforce enable itself at object-level.

43 + For multi-modal models, vLLM has enforced the enabling of some custom ops to use device-specific deep-optimized kernels for better performance in ViT part, such as `MMEncoderAttention` and `ApplyRotaryEmb`. We can also pass a `enforce_enable=True` param to the `__init__()` method of the `CustomOp` to enforce enable itself at object-level.

44

45 - Note that this `enforce_enable` mechanism will be removed after we adding a separate `compilation_config` for multi-modal part.

45 + Note that this `enforce_enable` mechanism will be removed after we add a separate `compilation_config` for multi-modal part.

46

47 ## How to Customise Your Configuration for CustomOp

48

docs/design/fused_moe_modular_kernel.md

+1 -1

@@ -2,7 +2,7 @@

2

3 ## Introduction

4

5 -FusedMoEModularKernel is implemented [here](../..//vllm/model_executor/layers/fused_moe/modular_kernel.py)

5 +FusedMoEModularKernel is implemented [here](../../vllm/model_executor/layers/fused_moe/modular_kernel.py)

6

7 Based on the format of the input activations, FusedMoE implementations are broadly classified into 2 types.

8

docs/design/logits_processors.md

+1 -1

@@ -138,7 +138,7 @@ Note that the sampler will access the logits processors via `SamplingMetadata.lo

138 # ...return sampler output data structure...

139

140

141 - def sample(self, logits, sampling_metadta)

141 + def sample(self, logits, sampling_metadata)

142

143 ...

144

docs/features/disagg_encoder.md

+1 -1

@@ -68,7 +68,7 @@ Here is a figure illustrating disaggregate encoder flow:

68

69 ![Disaggregated Encoder Flow](../assets/features/disagg_encoder/disagg_encoder_flow.png)

70

71 -For the PD disaggregation part, the Prefill instance receive cache exactly the same as the disaggregate encoder flow above. Prefill instance executes 1 step (prefill -> 1 token output) and then transfer KV cache to the Decode instance for the remaining execution. The KV transfer part purely happens after the execute of the PDinstance.

71 +For the PD disaggregation part, the Prefill instance receives cache exactly the same as the disaggregated encoder flow above. Prefill instance executes 1 step (prefill -> 1 token output) and then transfers KV cache to the Decode instance for the remaining execution. The KV transfer part purely happens after the execution of the PD instance.

72

73 `docs/features/disagg_prefill.md` shows the brief idea about the disaggregated prefill (v0)

74

docs/features/disagg_prefill.md

+1 -1

@@ -1,6 +1,6 @@

1 # Disaggregated Prefilling (experimental)

2

3 -This page introduces you the disaggregated prefilling feature in vLLM.

3 +This page introduces you to the disaggregated prefilling feature in vLLM.

4

5 !!! note

6 This feature is experimental and subject to change.

docs/features/quantization/inc.md

+1 -1

@@ -19,7 +19,7 @@ Once you've completed the model calibration process and collected the measuremen

19

20 ```bash

21 export QUANT_CONFIG=/path/to/quant/config/inc/meta-llama-3.1-405b-instruct/maxabs_measure_g3.json

22 -vllm serve meta-llama/Llama-3.1-405B-Instruct --quantization inc --kv-cache-dtype fp8_inc --tensor_paralel_size 8

22 +vllm serve meta-llama/Llama-3.1-405B-Instruct --quantization inc --kv-cache-dtype fp8_inc --tensor-parallel-size 8

23 ```

24

25 !!! tip

docs/features/spec_decode.md

+1 -1

@@ -173,7 +173,7 @@ Suffix Decoding can achieve better performance for tasks with high repetition, s

173 ## Speculating using MLP speculators

174

175 The following code configures vLLM to use speculative decoding where proposals are generated by

176 -draft models that conditioning draft predictions on both context vectors and sampled tokens.

176 +draft models that condition draft predictions on both context vectors and sampled tokens.

177 For more information see [this blog](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/) or

178 [this technical report](https://arxiv.org/abs/2404.19124).

179

docs/features/structured_outputs.md

+4 -4

@@ -39,7 +39,7 @@ request. You may also choose a specific backend, along with

39 some options. A full set of options is available in the `vllm serve --help`

40 text.

41

42 -Now let´s see an example for each of the cases, starting with the `choice`, as it´s the easiest one:

42 +Now let's see an example for each of the cases, starting with the `choice`, as it's the easiest one:

43

44 ??? code

45

@@ -126,12 +126,12 @@ The next example shows how to use the `response_format` parameter with a Pydanti

126 ```

127

128 !!! tip

129 - While not strictly necessary, normally it´s better to indicate in the prompt the

129 + While not strictly necessary, normally it's better to indicate in the prompt the

130 JSON schema and how the fields should be populated. This can improve the

131 results notably in most cases.

132

133 Finally we have the `grammar` option, which is probably the most

134 -difficult to use, but it´s really powerful. It allows us to define complete

134 +difficult to use, but it's really powerful. It allows us to define complete

135 languages like SQL queries. It works by using a context free EBNF grammar.

136 As an example, we can use to define a specific format of simplified SQL queries:

137

@@ -303,7 +303,7 @@ An example of using `structural_tag` can be found here: [examples/online_serving

303 ## Offline Inference

304

305 Offline inference allows for the same types of structured outputs.

306 -To use it, we´ll need to configure the structured outputs using the class `StructuredOutputsParams` inside `SamplingParams`.

306 +To use it, we'll need to configure the structured outputs using the class `StructuredOutputsParams` inside `SamplingParams`.

307 The main available options inside `StructuredOutputsParams` are:

308

309 - `json`

docs/getting_started/installation/cpu.arm.inc.md

+1 -1

@@ -1,6 +1,6 @@

1 # --8<-- [start:installation]

2

3 -vLLM offers basic model inferencing and serving on Arm CPU platform, with support NEON, data types FP32, FP16 and BF16.

3 +vLLM offers basic model inferencing and serving on Arm CPU platform, with support for NEON, data types FP32, FP16 and BF16.

4

5 # --8<-- [end:installation]

6 # --8<-- [start:requirements]

docs/getting_started/quickstart.md

+1 -1

@@ -75,7 +75,7 @@ This guide will help you quickly get started with vLLM to perform:

75 For more detailed instructions, including Docker, installing from source, and troubleshooting, please refer to the [vLLM on TPU documentation](https://docs.vllm.ai/projects/tpu/en/latest/).

76

77 !!! note

78 - For more detail and non-CUDA platforms, please refer [here](installation/README.md) for specific instructions on how to install vLLM.

78 + For more detail and non-CUDA platforms, please refer to the [installation guide](installation/README.md) for specific instructions on how to install vLLM.

79

80 ## Offline Batched Inference

81

docs/governance/collaboration.md

+2 -2

@@ -18,7 +18,7 @@ For features that you intend to maintain, please feel free to add yourself in [`

18 If you use vLLM, we recommend you making the model work with vLLM by following the [model registration](../contributing/model/registration.md) process before you release it publicly.

19

20 The vLLM team helps with new model architectures not supported by vLLM, especially models pushing architectural frontiers.

21 -Here's how the vLLM team works with model providers. The vLLM team includes all [committers](./committers.md) of the project. model providers can exclude certain members but shouldn't, as this may harm release timelines due to missing expertise. Contact [project leads](./process.md) if you want to collaborate.

21 +Here's how the vLLM team works with model providers. The vLLM team includes all [committers](./committers.md) of the project. Model providers can exclude certain members but shouldn't, as this may harm release timelines due to missing expertise. Contact [project leads](./process.md) if you want to collaborate.

22

23 Once we establish the connection between the vLLM team and model provider:

24

@@ -30,7 +30,7 @@ The vLLM team works with model providers on features, integrations, and release

30

31 The vLLM maintainers will not publicly share details about model architecture, release timelines, or upcoming releases. We maintain model weights on secure servers with security measures (though we can work with security reviews and testing without certification). We delete pre-release weights or artifacts upon request.

32

33 -The vLLM team collaborates on marketing and promotional efforts for model releases. model providers can use vLLM's trademark and logo in publications and materials.

33 +The vLLM team collaborates on marketing and promotional efforts for model releases. Model providers can use vLLM's trademark and logo in publications and materials.

34

35 ## Adding New Hardware

36

docs/models/extensions/fastsafetensor.md

+1 -1

@@ -1,4 +1,4 @@

1 -Loading Model weights with fastsafetensors

1 +Loading model weights with fastsafetensors

2 ===================================================================

3

4 Using fastsafetensors library enables loading model weights to GPU memory by leveraging GPU direct storage. See [their GitHub repository](https://github.com/foundation-model-stack/fastsafetensors) for more details.

docs/models/generative_models.md

+1 -1

@@ -2,7 +2,7 @@

2

3 vLLM provides first-class support for generative models, which covers most of LLMs.

4

5 -In vLLM, generative models implement the[VllmModelForTextGeneration][vllm.model_executor.models.VllmModelForTextGeneration] interface.

5 +In vLLM, generative models implement the [VllmModelForTextGeneration][vllm.model_executor.models.VllmModelForTextGeneration] interface.

6 Based on the final hidden states of the input, these models output log probabilities of the tokens to generate,

7 which are then passed through [Sampler][vllm.v1.sample.sampler.Sampler] to obtain the final text.

8

docs/serving/openai_compatible_server.md

+1 -1

@@ -874,7 +874,7 @@ You can pass multi-modal inputs to scoring models by passing `content` including

874 Full example:

875

876 - [examples/pooling/score/vision_score_api_online.py](../../examples/pooling/score/vision_score_api_online.py)

877 -- examples/pooling/score/vision_rerank_api_online.py](../../examples/pooling/score/vision_rerank_api_online.py)

877 +- [examples/pooling/score/vision_rerank_api_online.py](../../examples/pooling/score/vision_rerank_api_online.py)

878

879 #### Extra parameters

880

docs/usage/troubleshooting.md

+1 -1

@@ -338,7 +338,7 @@ If you use triton kernels with cuda 13, you might see an error like `ptxas fatal

338 vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

339 ```

340

341 -It means that the ptxas in triton bundle not compatible with your device. You need to set `TRITON_PTXAS_PATH` environment variable to use cuda toolkit's ptxas manually instead:

341 +It means that the ptxas in the triton bundle is not compatible with your device. You need to set `TRITON_PTXAS_PATH` environment variable to use cuda toolkit's ptxas manually instead:

342

343 ```shell

344 export CUDA_HOME=/usr/local/cuda

docs/usage/v1_guide.md

+1 -1

@@ -123,7 +123,7 @@ We are working on enabling prefix caching and chunked prefill for more categorie

123 #### Mamba Models

124

125 Models using selective state-space mechanisms instead of standard transformer attention are supported.

126 -Models that use Mamba-2 and Mamba-1 layers (e.g., `Mamba2ForCausalLM`, `MambaForCausalLM`,`FalconMambaForCausalLM`) are supported.

126 +Models that use Mamba-2 and Mamba-1 layers (e.g., `Mamba2ForCausalLM`, `MambaForCausalLM`, `FalconMambaForCausalLM`) are supported.

127

128 Hybrid models that combine Mamba-2 and Mamba-1 layers with standard attention layers are also supported (e.g., `BambaForCausalLM`,

129 `Zamba2ForCausalLM`, `NemotronHForCausalLM`, `FalconH1ForCausalLM` and `GraniteMoeHybridForCausalLM`, `JambaForCausalLM`, `Plamo2ForCausalLM`).

Fix various typos found in `docs` (#32212)