A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 5, 02:15 UTC
[Doc] add docs for online quant frontend (#39736)
Signed-off-by: Vasiliy Kuznetsov <vasiliy@meta.com>
Vasiliy Kuznetsov committed
Apr 16, 2026 at 10:52 UTC
5e5afafa21031abd7fb0e0707116e34be6f29af5
2 files changed
+95
docs/features/quantization/README.md
+1
@@ -16,6 +16,7 @@ The following are the supported quantization formats for vLLM:
16
- [INT8 W8A8](int8.md)
17
- [FP8 W8A8](fp8.md)
18
- [NVIDIA Model Optimizer](modelopt.md)
19
+- [Online Quantization](online.md)
20
- [AMD Quark](quark.md)
21
- [Quantized KV Cache](quantized_kvcache.md)
22
- [TorchAO](torchao.md)
docs/features/quantization/online.md
new
+94
@@ -0,0 +1,94 @@
1
+# Online Quantization
2
+
3
+Online quantization lets you take a BF16/FP16 model and quantize its Linear
4
+and MoE weights to lower precision (such as FP8) at load time, without needing
5
+a pre-quantized checkpoint or calibration data. Weights are converted during
6
+model loading and activations are dynamically scaled during each forward pass.
7
+
8
+## Quick Start
9
+
10
+Pass a scheme name to the `quantization` parameter:
11
+
12
+```python
13
+from vllm import LLM
14
+
15
+# Per-tensor FP8 quantization (one scale per weight tensor)
16
+llm = LLM("meta-llama/Llama-3.1-8B", quantization="fp8_per_tensor")
17
+
18
+# Per-block FP8 quantization (128x128 block scaling for weights and 1x128 block scaling for activations)
19
+llm = LLM("meta-llama/Llama-3.1-8B", quantization="fp8_per_block")
20
+```
21
+
22
+Or with the CLI:
23
+
24
+```bash
25
+vllm serve meta-llama/Llama-3.1-8B --quantization fp8_per_tensor
26
+vllm serve meta-llama/Llama-3.1-8B --quantization fp8_per_block
27
+```
28
+
29
+## Supported Schemes
30
+
31
+| Scheme | Weight recipe | Activation recipe | Notes |
32
+| ------ | ------------- | ------------------ | ----- |
33
+| `fp8_per_tensor` | fp8_e4m3 data, fp32 per-tensor scale | fp8_e4m3 data, fp32 per-tensor scale | On some GPUs (Ada, Hopper) linear activations use per-token scaling for better performance |
34
+| `fp8_per_block` | fp8_e4m3 data, fp32 per-128x128-block scale | fp8_e4m3 data, fp32 per-1x128-block scale | |
35
+
36
+Support for additional schemes will be added in future versions of vllm.
37
+
38
+## Advanced Configuration
39
+
40
+For fine-grained control, use a `quantization_config` dictionary.
41
+
42
+### Separate Schemes for Dense and MoE Layers
43
+
44
+You can apply different quantization schemes to dense linear layers and MoE expert layers:
45
+
46
+```python
47
+from vllm import LLM
48
+
49
+llm = LLM(
50
+ "ibm-granite/granite-3.0-1b-a400m-base",
51
+ quantization="fp8_per_tensor",
52
+ quantization_config={
53
+ "linear_scheme_override": "fp8_per_block",
54
+ },
55
+)
56
+```
57
+
58
+Or,
59
+
60
+```python
61
+from vllm import LLM
62
+
63
+llm = LLM(
64
+ "ibm-granite/granite-3.0-1b-a400m-base",
65
+ quantization="fp8_per_tensor",
66
+ quantization_config={
67
+ "moe_scheme_override": "fp8_per_block",
68
+ },
69
+)
70
+```
71
+
72
+### Excluding Layers from Quantization
73
+
74
+Use the `ignore` parameter to skip specific layers. It accepts exact layer names and regex patterns (prefixed with `re:`):
75
+
76
+```python
77
+from vllm import LLM
78
+
79
+llm = LLM(
80
+ "ibm-granite/granite-3.0-1b-a400m-base",
81
+ quantization="fp8_per_tensor",
82
+ quantization_config={
83
+ "ignore": [
84
+ # exact layer name
85
+ "model.layers.1.self_attn.o_proj",
86
+ # regex: skip all QKV projections
87
+ "re:.*[qkv]_proj",
88
+ ],
89
+ },
90
+)
91
+```
92
+
93
+!!! note
94
+ For fused layers (e.g., `qkv_proj` which fuses `q_proj`, `k_proj`, `v_proj`), the ignore pattern must match the **unfused** shard names (`q_proj`, `k_proj`, `v_proj`), not the fused name.