@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 218 lines 8.61 KB
Rendered Raw
1 # Online Quantization
2
3 Online quantization lets you take a BF16/FP16 model and quantize its Linear
4 and MoE weights to lower precision (such as FP8) at load time, without needing
5 a pre-quantized checkpoint or calibration data. Weights are converted during
6 model loading and activations are dynamically scaled during each forward pass.
7
8 ## Quick Start
9
10 Pass a scheme name to the `quantization` parameter:
11
12 ```python
13 from vllm import LLM
14
15 # Per-tensor FP8 quantization (one scale per weight tensor)
16 llm = LLM("meta-llama/Llama-3.1-8B", quantization="fp8_per_tensor")
17
18 # Per-block FP8 quantization (128x128 block scaling for weights and 1x128 block scaling for activations)
19 llm = LLM("meta-llama/Llama-3.1-8B", quantization="fp8_per_block")
20
21 # MXFP8 quantization for weights and activations
22 llm = LLM("meta-llama/Llama-3.1-8B", quantization="mxfp8")
23
24 # MXFP4 weight; activation quantization depends on the `linear_backend` picked
25 llm = LLM("meta-llama/Llama-3.1-8B", quantization="mxfp4")
26
27 # MXFP4 MOE-only weight and activation quantization
28 llm = LLM(
29 "Qwen/Qwen3.5-35B-A3B",
30 quantization="mxfp4",
31 quantization_config={"linear": {"activation": None, "weight": None}}
32 )
33 ```
34
35 Or with the CLI:
36
37 ```bash
38 vllm serve meta-llama/Llama-3.1-8B --quantization fp8_per_tensor
39 vllm serve meta-llama/Llama-3.1-8B --quantization fp8_per_block
40 vllm serve meta-llama/Llama-3.1-8B --quantization mxfp8
41 vllm serve meta-llama/Llama-3.1-8B --quantization mxfp4
42
43 vllm serve Qwen/Qwen3.5-35B-A3B --quantization mxfp4 \
44 --quantization-config '{"linear":{"activation":null,"weight":null}}'
45 ```
46
47 ## Supported Schemes
48
49 | Scheme | Weight recipe | Activation recipe | Notes |
50 | ------ | ------------- | ------------------ | ----- |
51 | `fp8_per_tensor` | fp8_e4m3 data, fp32 per-tensor scale | fp8_e4m3 data, fp32 per-tensor scale | On some GPUs (Ada, Hopper) linear activations use per-token scaling for better performance |
52 | `fp8_per_block` | fp8_e4m3 data, fp32 per-128x128-block scale | fp8_e4m3 data, fp32 per-1x128-block scale | |
53 | `mxfp8` | fp8_e4m3 data, e8m0 per-1x32-block scale | fp8_e4m3 data, e8m0 per-1x32-block scale | Requires SM 100+ (Blackwell or newer) for w8a8, other GPUs use a w8a16 fallback |
54 | `mxfp4` | fp4_e2m1 data, e8m0 per-1x32-block scale ([OCP MX specs](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf)) | - linear: fp4_e2m1 data, e8m0 per-1x32-block scale in some backends, or BF16. <br> - MOE: fp4_e2m1 data, e8m0 per-1x32-block scale. | Linear MXFP4 backend is auto-selected per platform, not enforcing activation dtype. Some use BF16 activation. Use `--linear-backend` to pin one (e.g. `--linear-backend flashinfer`). |
55
56 ## Advanced Configuration
57
58 For fine-grained control, use a `quantization_config` dictionary.
59
60 ### Schema
61
62 ```yaml
63 quantization_config:
64 linear:
65 weight: <name> # see QUANT_KEY_NAMES in vllm/config/quantization.py
66 activation: <name>
67 moe:
68 weight: <name>
69 activation: <name>
70 ignore: [<layer-name-or-regex-or-fnmatch-pattern>, ...]
71 ```
72
73 `linear` and `moe` accept a full `{weight, activation}` dict, or a bare
74 string. A string resolves first against the `--quantization` shorthands
75 (taking the matching layer-kind slot), then against `QUANT_KEY_NAMES` as a
76 weight name. Unset fields fall back to the `--quantization` shorthand's
77 defaults, or for already-quantized checkpoints to whatever the checkpoint
78 declares.
79
80 On XPU, non-block FP8 scaled-mm linear layers default to W8A16; setting `--linear-backend xpu` forces W8A8. Use `--linear-backend xpu_woq` to explicitly select weight-only quantization (W8A16). Setting `--linear-backend torch` also forces W8A8 but runs the GEMM through `torch._scaled_mm` instead of the custom XPU kernel.
81
82 The CLI accepts the same shape as JSON or as dotted keys:
83
84 ```bash
85 vllm serve <model> --quantization-config '{"moe":{"activation":"mxfp8"}}'
86 vllm serve <model> --quantization-config.moe.activation mxfp8
87 ```
88
89 ### Activation overrides on already-quantized checkpoints
90
91 For checkpoint-quantized models, `quantization_config` lets you pick an
92 activation format independently of the baked-in weights. The supported
93 overrides are checkpoint-specific; today this is wired up for MXFP4 MoE
94 checkpoints (gpt-oss) where you can opt into FP8 activations:
95
96 ```bash
97 vllm serve openai/gpt-oss-20b --quantization-config.moe.activation mxfp8
98 ```
99
100 Combine with `--moe-backend` to pin a specific kernel family.
101
102 ### Online quantization on unquantized layers from partially-quantized checkpoints
103
104 Online quantization can be used on already quantized checkpoints independently of their original `quant_method` (`modelopt`, `compressed-tensors`, `quark`, etc.), for layers that are left unquantized in the original checkpoint.
105
106 The checkpoint `quant_method` remains responsible for its quantized layers, while the selected unquantized layers use the requested online method.
107
108 For example:
109
110 ```bash
111 vllm serve amd/Qwen3.5-35B-A3B-MXFP4 \
112 --quantization-config.linear mxfp8
113 ```
114
115 adds MXFP8 quantization to the dense linear layers of a Quark checkpoint where only MOE experts are quantized.
116
117 !!! info
118 `quantization_config.ignore` is an online-only exclusion: the original `quant_method` relies solely on its own ignore implementation and on the ignored layers specified in `config.json`.
119
120 ### Separate Schemes for Dense and MoE Layers
121
122 You can apply different quantization schemes to dense linear layers and MoE expert layers via the `linear` and `moe` fields. Each accepts either a full spec dict, or a bare string naming an online shorthand (e.g. `"fp8_per_block"`) or weight format (e.g. `"fp8_per_block_static"`); fields not set fall back to the shorthand defaults.
123
124 ```python
125 from vllm import LLM
126
127 # Linear: per-block FP8; MoE: per-tensor FP8 (inherited from the shorthand)
128 llm = LLM(
129 "ibm-granite/granite-3.0-1b-a400m-base",
130 quantization="fp8_per_tensor",
131 quantization_config={
132 "linear": "fp8_per_block",
133 },
134 )
135 ```
136
137 Or,
138
139 ```python
140 from vllm import LLM
141
142 # Linear: per-tensor FP8 (inherited); MoE: per-block FP8
143 llm = LLM(
144 "ibm-granite/granite-3.0-1b-a400m-base",
145 quantization="fp8_per_tensor",
146 quantization_config={
147 "moe": "fp8_per_block",
148 },
149 )
150 ```
151
152 ### Excluding Layers from Quantization
153
154 Use the `ignore` parameter to skip specific layers. It accepts exact layer names, regex patterns (prefixed with `re:`), and patterns understood by [`fnmatch.fnmatch`](https://docs.python.org/3/library/fnmatch.html#fnmatch.fnmatch):
155
156 ```python
157 from vllm import LLM
158
159 llm = LLM(
160 "ibm-granite/granite-3.0-1b-a400m-base",
161 quantization="fp8_per_tensor",
162 quantization_config={
163 "ignore": [
164 # exact layer name
165 "model.layers.1.self_attn.o_proj",
166 # regex: skip all QKV projections
167 "re:.*[qkv]_proj",
168 # fnmatch: skip all MoE experts
169 "*mlp.experts*",
170 ],
171 },
172 )
173 ```
174
175 !!! note
176 For fused layers (e.g., `qkv_proj` which fuses `q_proj`, `k_proj`, `v_proj`), patterns may match the fused name directly or all of its unfused shard names.
177
178 ### Fine-Grained Per-Layer Quantization Schemes
179
180 Use the `targets` parameter to apply different online shorthands to different layers, instead of one scheme applied everywhere via `linear`/`moe`. Keys are exact layer names, regex patterns (prefixed with `re:`), or patterns understood by [`fnmatch.fnmatch`](https://docs.python.org/3/library/fnmatch.html#fnmatch.fnmatch); values are shorthand names (`fp8_per_tensor`, `fp8_per_block`, `fp8_per_channel`, `mxfp8`, `int8_per_channel_weight_only`, `nvfp4_per_token`).
181
182 Example:
183
184 ```python
185 from vllm import LLM
186
187 llm = LLM(
188 "Qwen/Qwen3.5-35B-A3B",
189 quantization="online",
190 quantization_config={
191 "targets": {
192 # exact layer name
193 "model.layers.0.self_attn.o_proj": "fp8_per_tensor",
194 # regex: quantize all QKV projections
195 r"re:.*self_attn\.qkv_proj.*": "mxfp8",
196 # fnmatch: quantize all MoE experts
197 "*mlp.experts*": "mxfp4",
198 },
199 },
200 )
201 ```
202
203 Or with the CLI:
204
205 ```bash
206 vllm serve Qwen/Qwen3.5-35B-A3B \
207 --quantization online \
208 --quantization-config '{"targets":{"model.layers.0.self_attn.o_proj":"fp8_per_tensor","re:.*self_attn\\.qkv_proj.*":"mxfp8","*mlp.experts*":"mxfp4"}}'
209 ```
210
211 !!! info
212
213 - `targets` is mutually exclusive with online `linear` and `moe`: set one or the other, not both.
214 - A layer that matches no `targets` pattern is left unchanged from its checkpoint dtype.
215 - A layer name may not match both `targets` and `ignore`, raising an error.
216 - A layer may not match more than one `targets` pattern, raising an error.
217 - fnmatch-style patterns are supported only by online quantization; they are
218 not applied to Quark or compressed-tensors configurations.