@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 248 lines 9.81 KB
Rendered Raw
1 # Quantization
2
3 Quantization trades off model precision for smaller memory footprint, allowing large models to be run on a wider range of devices.
4
5 !!! tip
6 To get started with quantization, see [LLM Compressor](llm_compressor/README.md), a library for optimizing models for deployment with vLLM that supports FP8, INT8, INT4, and other quantization formats.
7
8 The following are the supported quantization formats for vLLM:
9
10 - [AutoAWQ](auto_awq.md)
11 - [BitsAndBytes](bnb.md)
12 - [GPTQModel](gptqmodel.md)
13 - [Intel Neural Compressor](inc.md)
14 - [LLM Compressor](llm_compressor/README.md)
15 - [FP8 W8A8](llm_compressor/fp8.md)
16 - [INT4 W4A16](llm_compressor/int4.md)
17 - [INT8 W4A8](llm_compressor/int8_w4a8.md)
18 - [INT8 W8A8](llm_compressor/int8_w8a8.md)
19 - [NVIDIA Model Optimizer](modelopt.md)
20 - [Online Quantization](online.md)
21 - [AMD Quark](quark.md)
22 - [Quantized KV Cache](quantized_kvcache.md)
23 - [TorchAO](torchao.md)
24 - [FP8 ViT Encoder Attention](fp8_vit_attn.md)
25
26 ## Selecting Linear Backends per Quantization
27
28 `--linear-backend` selects one backend for all quantized linear layers. For
29 mixed-precision models that use more than one linear quantization scheme, use
30 `linear_backend_per_quant` to override the backend for individual schemes:
31
32 ```bash
33 vllm serve <model> \
34 --linear-backend cutlass \
35 --kernel-config '{"linear_backend_per_quant":{"nvfp4_w4a16":"humming"}}'
36 ```
37
38 Here, NVFP4 W4A16 linear layers use Humming, while all other quantized linear
39 layers use CUTLASS. Per-quantization overrides take precedence over
40 `--linear-backend`; schemes without an override continue to use the global
41 setting, including automatic selection when it is `auto`.
42
43 ## Supported Hardware
44
45 The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
46
47 <style>
48 td:not(:first-child) {
49 text-align: center !important;
50 }
51 td {
52 padding: 0.5rem !important;
53 white-space: nowrap;
54 }
55
56 th {
57 padding: 0.5rem !important;
58 min-width: 0 !important;
59 }
60
61 th:not(:first-child) {
62 writing-mode: vertical-lr;
63 transform: rotate(180deg)
64 }
65 </style>
66
67 | Implementation | Volta | Turing | Ampere | Ada | Hopper | AMD GPU | Intel GPU | x86 CPU | Arm CPU |
68 | ------------------------- | ----- | ------ | ------ | --- | ------ | ------- | --------- | ------- | ------- |
69 | AWQ | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ | ❌ |
70 | GPTQ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ | ❌ |
71 | Marlin (GPTQ/AWQ/FP8/FP4) | ❌ | ✅︎* | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ |
72 | llm-compressor INT8 (W8A8)| ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ✅︎ | ✅︎ |
73 | llm-compressor INT8 (W4A8)| ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅︎ |
74 | llm-compressor FP8 (W8A8) | ❌ | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
75 | bitsandbytes | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ |
76 | DeepSpeedFP | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ |
77 | GGUF | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
78
79 - Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.
80 - ✅︎ indicates that the quantization method is supported on the specified hardware.
81 - ❌ indicates that the quantization method is not supported on the specified hardware.
82 - All Intel Gaudi quantization support has been migrated to [vLLM-Gaudi](https://github.com/vllm-project/vllm-gaudi).
83 - *Turing does not support Marlin MXFP4.
84
85 !!! note
86 For information on quantization support on Google TPU, please refer to the [TPU-Inference Recommended Models and Features](https://docs.vllm.ai/projects/tpu/en/latest/recommended_models_features/) documentation.
87
88 !!! note
89 This compatibility chart is subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
90
91 For the most up-to-date information on hardware support and quantization methods, please refer to [vllm/model_executor/layers/quantization](../../../vllm/model_executor/layers/quantization) or consult with the vLLM development team.
92
93 ## Out-of-Tree Quantization Plugins
94
95 vLLM supports registering custom, out-of-tree quantization methods using the `@register_quantization_config` decorator. This allows you to implement and use your own quantization schemes without modifying the vLLM codebase.
96
97 ### Registering a Custom Quantization Method
98
99 To register a custom quantization method, create a class that inherits from `QuantizationConfig` and decorate it with `@register_quantization_config`. The `get_quant_method` dispatches to the appropriate quantize method based on the layer type:
100
101 ```python
102 import torch
103 from vllm.model_executor.layers.quantization import (
104 register_quantization_config,
105 )
106 from vllm.model_executor.layers.quantization.base_config import (
107 QuantizationConfig,
108 QuantizeMethodBase,
109 )
110 from vllm.model_executor.layers.linear import LinearBase
111 from vllm.model_executor.layers.fused_moe import FusedMoE
112
113 @register_quantization_config("my_quant")
114 class MyQuantConfig(QuantizationConfig):
115 """Custom quantization config."""
116
117 def get_name(self) -> str:
118 return "my_quant"
119
120 def get_supported_act_dtypes(self) -> list:
121 return [torch.float16, torch.bfloat16]
122
123 @classmethod
124 def get_min_capability(cls) -> int:
125 # Minimum GPU compute capability, -1 for no restriction
126 return -1
127
128 @staticmethod
129 def get_config_filenames() -> list[str]:
130 # Config files to search for in model directory
131 return []
132
133 @classmethod
134 def from_config(cls, config: dict) -> "MyQuantConfig":
135 # Create config from model's quantization config
136 return cls()
137
138 def get_quant_method(
139 self, layer: torch.nn.Module, prefix: str
140 ) -> QuantizeMethodBase | None:
141 # Dispatch based on layer type
142 # NOTE: you only need to implement methods you care about
143 if isinstance(layer, LinearBase):
144 return MyQuantLinearMethod()
145 elif isinstance(layer, FusedMoE):
146 return MyQuantMoEMethod(layer.moe_config)
147 return None
148 ```
149
150 ### Required QuantizationConfig Methods
151
152 Your custom `QuantizationConfig` subclass must implement these abstract methods:
153
154 | Method | Description |
155 | ------ | ----------- |
156 | `get_name()` | Returns the name of the quantization method |
157 | `get_supported_act_dtypes()` | Returns list of supported activation dtypes (e.g., `torch.float16`) |
158 | `get_min_capability()` | Returns minimum GPU compute capability (e.g., 80 for Ampere, -1 for no restriction) |
159 | `get_config_filenames()` | Returns list of config filenames to search for in model directory |
160 | `from_config(config)` | Class method to create config from model's quantization config dict |
161 | `get_quant_method(layer, prefix)` | Returns the quantization method for a given layer, or `None` to skip |
162
163 ### Implementing a Quantized Linear Method
164
165 For linear layers, return a `QuantizeMethodBase` subclass from `get_quant_method`. You can extend `UnquantizedLinearMethod` as a starting point:
166
167 ```python
168 from vllm.model_executor.layers.linear import UnquantizedLinearMethod
169
170 class MyQuantLinearMethod(UnquantizedLinearMethod):
171 """Custom quantization method for linear layers."""
172
173 def create_weights(
174 self, layer: torch.nn.Module, *weight_args, **extra_weight_attrs
175 ):
176 # Create quantized weights for the layer
177 ...
178
179 def apply(
180 self,
181 layer: torch.nn.Module,
182 x: torch.Tensor,
183 bias: torch.Tensor | None = None,
184 ) -> torch.Tensor:
185 # Apply custom quantization logic here
186 ...
187 ```
188
189 ### Implementing a Quantized MoE Method
190
191 For Mixture of Experts (MoE) models, return a `FusedMoEMethodBase` subclass from `get_quant_method`. You can use `UnquantizedFusedMoEMethod` to skip MoE quantization:
192
193 ```python
194 from vllm.model_executor.layers.fused_moe.layer import UnquantizedFusedMoEMethod
195 from vllm.model_executor.layers.fused_moe.fused_moe_method_base import (
196 FusedMoEMethodBase,
197 )
198 from vllm.model_executor.layers.fused_moe.config import FusedMoEQuantConfig
199
200 class MyQuantMoEMethod(FusedMoEMethodBase):
201 """Custom quantization method for MoE layers."""
202
203 def create_weights(
204 self,
205 layer: torch.nn.Module,
206 num_experts: int,
207 hidden_size: int,
208 intermediate_size_per_partition: int,
209 params_dtype: torch.dtype,
210 **extra_weight_attrs,
211 ):
212 # Create quantized weights for the MoE layer
213 ...
214
215 def apply(
216 self,
217 layer: torch.nn.Module,
218 router: "FusedMoERouter",
219 x: torch.Tensor,
220 router_logits: torch.Tensor,
221 ) -> torch.Tensor:
222 # Apply MoE computation with quantized weights
223 ...
224
225 def get_fused_moe_quant_config(
226 self, layer: torch.nn.Module
227 ) -> FusedMoEQuantConfig | None:
228 # Return the MoE quantization configuration
229 ...
230 ```
231
232 See existing implementations like `Fp8MoEMethod` in `vllm/model_executor/layers/quantization/fp8.py` for reference.
233
234 ### Using the Plugin
235
236 Once registered, you can use your custom quantization method with vLLM:
237
238 ```python
239 # Register your quantization method (import the module containing your config)
240 import my_quant_plugin
241
242 from vllm import LLM
243
244 # Use the custom quantization method
245 llm = LLM(model="your-model", quantization="my_quant")
246 ```
247
248 For more information on the plugin system, see the [Plugin System documentation](../../design/plugin_system.md).