[Docs] Document NVFP4 GEMM kernel selection and Marlin weight-only fallback (#49376)
Signed-off-by: harjoth <harjoth.khara@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Harjoth Khara committed
Jul 26, 2026 at 20:53 UTC
8de50e46d4a0a227db1888410cb6ac93058fb6cf
1 file changed
+14
docs/features/quantization/modelopt.md
+14
@@ -19,6 +19,20 @@ following `quantization.quant_algo` values:
19
- `NVFP4`: ModelOpt NVFP4 checkpoints (use `quantization="modelopt_fp4"`).
20
- `MXFP8`: ModelOpt MXFP8 checkpoints (use `quantization="modelopt_mxfp8"`).
21
22
+!!! note
23
+ For NVFP4 checkpoints, vLLM selects a GEMM kernel automatically at load
24
+ time from the backends available on the current platform (CUTLASS,
25
+ FlashInfer, Marlin, and others). On GPUs without a supported native FP4
26
+ GEMM kernel, vLLM falls back to weight-only (W4A16) execution via Marlin
27
+ and logs a warning; this may reduce throughput for compute-heavy
28
+ workloads. Use `--linear-backend` to override the automatic selection
29
+ (this replaces the deprecated `VLLM_NVFP4_GEMM_BACKEND` environment
30
+ variable). Values relevant to NVFP4 include `cutlass`,
31
+ `flashinfer_cutlass`, `flashinfer_trtllm`, `flashinfer_cudnn`, and
32
+ `marlin`; the full list is documented under `KernelConfig` on the
33
+ [Engine Arguments](../../configuration/engine_args.md) page and shown by
34
+ `vllm serve --help=KernelConfig`.
35
+
36
## Quantizing HuggingFace Models with PTQ
37
38
You can quantize HuggingFace models using the example scripts provided in the Model Optimizer repository. The primary script for LLM PTQ is typically found within the `examples/llm_ptq` directory.