[Doc] Document FP8 GEMM kernel selection and Blackwell support (#49936)
Signed-off-by: harjoth <harjoth.khara@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harjoth Khara committed
Sep 1, 2026 at 05:22 UTC
40824284bcb2f50047a48307ed39ce441bb15b0b
2 files changed
+21
-2
.github/CODEOWNERS
+1
@@ -127,6 +127,7 @@
127
/tests/models/transformers @hmellor
128
129
# Docs
130
+/docs/features/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye
131
/docs/mkdocs @hmellor
132
/docs/**/*.yml @hmellor
133
/requirements/docs.txt @hmellor
docs/features/quantization/llm_compressor/fp8.md
+20
-2
@@ -1,7 +1,7 @@
1
# FP8 W8A8
2
3
vLLM supports FP8 (8-bit floating point) weight and activation quantization using hardware acceleration on GPUs such as Nvidia H100 and AMD MI300x.
4
-Currently, only Hopper and Ada Lovelace GPUs are officially supported for W8A8.
4
+Ada Lovelace, Hopper, and Blackwell GPUs are supported for W8A8.
5
Turing/Ampere GPUs are supported for W8A16 (weight-only FP8) utilizing Marlin kernels.
6
Quantization of models with FP8 allows for a 2x reduction in model memory requirements and up to a 1.6x improvement in throughput with minimal impact on accuracy.
7
@@ -13,9 +13,27 @@ The FP8 types typically supported in hardware have two distinct representations,
13
- **E5M2**: Consists of 1 sign bit, 5 exponent bits, and 2 bits of mantissa. It can store values up to +/-57344, +/- `inf`, and `nan`. The tradeoff for the increased dynamic range is lower precision of the stored values.
14
15
!!! note
16
- FP8 computation is supported on NVIDIA GPUs with compute capability >= 8.9 (Ada Lovelace, Hopper).
16
+ FP8 computation is supported on NVIDIA GPUs with compute capability >= 8.9 (Ada Lovelace, Hopper, Blackwell).
17
FP8 models will run on compute capability >= 7.5 (Turing) as weight-only W8A16, utilizing FP8 Marlin.
18
19
+!!! note "GEMM kernel selection"
20
+ vLLM picks an FP8 GEMM kernel automatically at load time and logs a
21
+ `Selected <kernel> for <module>` line at startup. For block-quantized
22
+ checkpoints on CUDA it tries, in order: a FlashInfer/DeepGEMM hybrid
23
+ (Hopper only), DeepGEMM, CUTLASS, Marlin, Triton, Humming, then a
24
+ PyTorch fallback. GPUs without native FP8 support (e.g. Turing/Ampere)
25
+ land on weight-only (W8A16) Marlin.
26
+
27
+ If inference hangs with no error, try `VLLM_USE_DEEP_GEMM=0` or
28
+ `--linear-backend cutlass`.
29
+
30
+ `--linear-backend` only affects quantized linear layers; MoE experts
31
+ use the separate `--moe-backend`. An explicit backend that is not
32
+ supported on your hardware raises an error rather than falling back.
33
+ The full list is documented under
34
+ [`KernelConfig`](../../../cli/serve.md#kernelconfig) in the CLI
35
+ reference and shown by `vllm serve --help=KernelConfig`.
36
+
37
## Installation
38
39
To produce performant FP8 quantized models with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library: