A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
Fix INT8 quantization error on Blackwell GPUs (SM100+) (#25935)
Signed-off-by: padg9912 <phone.and.desktop@gmail.com>
Param committed
Sep 30, 2025 at 22:19 UTC
99028fda449d920237561da163daf2e908e25535
2 files changed
+9
-2
csrc/quantization/cutlass_w8a8/c3x/scaled_mm_helper.hpp
+4
-1
@@ -25,7 +25,10 @@ void dispatch_scaled_mm(torch::Tensor& c, torch::Tensor const& a,
25
if constexpr (!std::is_same_v<Int8Func, std::nullptr_t>) {
26
int8_func(c, a, b, a_scales, b_scales, bias);
27
} else {
28
- TORCH_CHECK(false, "Int8 not supported for this architecture");
28
+ int32_t version_num = get_sm_version_num();
29
+ TORCH_CHECK(
30
+ false, "Int8 not supported on SM", version_num,
31
+ ". Use FP8 quantization instead, or run on older arch (SM < 100).");
32
}
33
}
34
} else {
docs/features/quantization/int8.md
+5
-1
@@ -6,7 +6,11 @@ This quantization method is particularly useful for reducing model size while ma
6
Please visit the HF collection of [quantized INT8 checkpoints of popular LLMs ready to use with vLLM](https://huggingface.co/collections/neuralmagic/int8-llms-for-vllm-668ec32c049dca0369816415).
7
8
!!! note
9
- INT8 computation is supported on NVIDIA GPUs with compute capability > 7.5 (Turing, Ampere, Ada Lovelace, Hopper, Blackwell).
9
+ INT8 computation is supported on NVIDIA GPUs with compute capability > 7.5 (Turing, Ampere, Ada Lovelace, Hopper).
10
+
11
+!!! warning
12
+ **Blackwell GPU Limitation**: INT8 is not supported on compute capability >= 100 (e.g., RTX 6000 Blackwell).
13
+ Use [FP8 quantization](fp8.md) instead, or run on Hopper/Ada/Ampere architectures.
14
15
## Prerequisites
16