A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Docs] Move quant supported hardware table to README (#23663)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Aug 26, 2025 at 23:26 UTC
6421b66bf4894a3e1e22d17c78901e3974173e09
3 files changed
+48
-34
docs/features/quantization/README.md
+47
-1
@@ -4,7 +4,6 @@ Quantization trades off model precision for smaller memory footprint, allowing l
4
5
Contents:
6
7
-- [Supported Hardware](supported_hardware.md)
7
- [AutoAWQ](auto_awq.md)
8
- [AutoRound](auto_round.md)
9
- [BitsAndBytes](bnb.md)
@@ -19,3 +18,50 @@ Contents:
18
- [AMD Quark](quark.md)
19
- [Quantized KV Cache](quantized_kvcache.md)
20
- [TorchAO](torchao.md)
21
+
22
+## Supported Hardware
23
+
24
+The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
25
+
26
+<style>
27
+td:not(:first-child) {
28
+ text-align: center !important;
29
+}
30
+td {
31
+ padding: 0.5rem !important;
32
+ white-space: nowrap;
33
+}
34
+
35
+th {
36
+ padding: 0.5rem !important;
37
+ min-width: 0 !important;
38
+}
39
+
40
+th:not(:first-child) {
41
+ writing-mode: vertical-lr;
42
+ transform: rotate(180deg)
43
+}
44
+</style>
45
+
46
+| Implementation | Volta | Turing | Ampere | Ada | Hopper | AMD GPU | Intel GPU | Intel Gaudi | x86 CPU | AWS Neuron | Google TPU |
47
+|-----------------------|---------|----------|----------|-------|----------|-----------|-------------|-------------|-----------|--------------|--------------|
48
+| AWQ | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ❌ | ✅︎ | ❌ | ❌ |
49
+| GPTQ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ❌ | ✅︎ | ❌ | ❌ |
50
+| Marlin (GPTQ/AWQ/FP8) | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
51
+| INT8 (W8A8) | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ |
52
+| FP8 (W8A8) | ❌ | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ✅︎ | ❌ |
53
+| BitBLAS | ✅︎ | ✅ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
54
+| BitBLAS (GPTQ) | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
55
+| bitsandbytes | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
56
+| DeepSpeedFP | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
57
+| GGUF | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ | ❌ |
58
+| INC (W8A8) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅︎ | ❌ | ❌ | ❌ |
59
+
60
+- Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.
61
+- ✅︎ indicates that the quantization method is supported on the specified hardware.
62
+- ❌ indicates that the quantization method is not supported on the specified hardware.
63
+
64
+!!! note
65
+ This compatibility chart is subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
66
+
67
+ For the most up-to-date information on hardware support and quantization methods, please refer to <gh-dir:vllm/model_executor/layers/quantization> or consult with the vLLM development team.
docs/features/quantization/bitblas.md
+1
-1
@@ -5,7 +5,7 @@ vLLM now supports [BitBLAS](https://github.com/microsoft/BitBLAS) for more effic
5
!!! note
6
Ensure your hardware supports the selected `dtype` (`torch.bfloat16` or `torch.float16`).
7
Most recent NVIDIA GPUs support `float16`, while `bfloat16` is more common on newer architectures like Ampere or Hopper.
8
- For details see [supported hardware](supported_hardware.md).
8
+ For details see [supported hardware](README.md#supported-hardware).
9
10
Below are the steps to utilize BitBLAS with vLLM.
11
docs/features/quantization/supported_hardware.md
deleted
-32
@@ -1,32 +0,0 @@
1
-# Supported Hardware
2
-
3
-The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
4
-
5
-<style>
6
-th {
7
- white-space: nowrap;
8
- min-width: 0 !important;
9
-}
10
-</style>
11
-
12
-| Implementation | Volta | Turing | Ampere | Ada | Hopper | AMD GPU | Intel GPU | Intel Gaudi | x86 CPU | AWS Neuron | Google TPU |
13
-|-----------------------|---------|----------|----------|-------|----------|-----------|-------------|-------------|-----------|--------------|--------------|
14
-| AWQ | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ❌ | ✅︎ | ❌ | ❌ |
15
-| GPTQ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ❌ | ✅︎ | ❌ | ❌ |
16
-| Marlin (GPTQ/AWQ/FP8) | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
17
-| INT8 (W8A8) | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ |
18
-| FP8 (W8A8) | ❌ | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ✅︎ | ❌ |
19
-| BitBLAS (GPTQ) | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
20
-| bitsandbytes | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
21
-| DeepSpeedFP | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
22
-| GGUF | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ | ❌ |
23
-| INC (W8A8) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅︎ | ❌ | ❌ | ❌ |
24
-
25
-- Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.
26
-- ✅︎ indicates that the quantization method is supported on the specified hardware.
27
-- ❌ indicates that the quantization method is not supported on the specified hardware.
28
-
29
-!!! note
30
- This compatibility chart is subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
31
-
32
- For the most up-to-date information on hardware support and quantization methods, please refer to <gh-dir:vllm/model_executor/layers/quantization> or consult with the vLLM development team.