A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 18:15 UTC
[Doc] Update quantization supported hardware table (#7595)
Michael Goin committed
Aug 16, 2024 at 16:59 UTC
d4f0f17b025487e78525ba13133a161b20538d41
1 file changed
+127
-14
docs/source/quantization/supported_hardware.rst
+127
-14
@@ -5,25 +5,138 @@ Supported Hardware for Quantization Kernels
5
6
The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
7
8
-============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
9
-Implementation Volta Turing Ampere Ada Hopper AMD GPU Intel GPU x86 CPU AWS Inferentia Google TPU
10
-============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
11
-AQLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
12
-AWQ ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
13
-DeepSpeedFP ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
14
-FP8 ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
15
-Marlin ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
16
-GPTQ ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
17
-SqueezeLLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
18
-bitsandbytes ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
19
-============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
8
+.. list-table::
9
+ :header-rows: 1
10
+ :widths: 20 8 8 8 8 8 8 8 8 8 8
11
+
12
+ * - Implementation
13
+ - Volta
14
+ - Turing
15
+ - Ampere
16
+ - Ada
17
+ - Hopper
18
+ - AMD GPU
19
+ - Intel GPU
20
+ - x86 CPU
21
+ - AWS Inferentia
22
+ - Google TPU
23
+ * - AWQ
24
+ - ✗
25
+ - ✅︎
26
+ - ✅︎
27
+ - ✅︎
28
+ - ✅︎
29
+ - ✗
30
+ - ✗
31
+ - ✗
32
+ - ✗
33
+ - ✗
34
+ * - GPTQ
35
+ - ✅︎
36
+ - ✅︎
37
+ - ✅︎
38
+ - ✅︎
39
+ - ✅︎
40
+ - ✗
41
+ - ✗
42
+ - ✗
43
+ - ✗
44
+ - ✗
45
+ * - Marlin (GPTQ/AWQ/FP8)
46
+ - ✗
47
+ - ✗
48
+ - ✅︎
49
+ - ✅︎
50
+ - ✅︎
51
+ - ✗
52
+ - ✗
53
+ - ✗
54
+ - ✗
55
+ - ✗
56
+ * - INT8 (W8A8)
57
+ - ✗
58
+ - ✅︎
59
+ - ✅︎
60
+ - ✅︎
61
+ - ✅︎
62
+ - ✗
63
+ - ✗
64
+ - ✗
65
+ - ✗
66
+ - ✗
67
+ * - FP8 (W8A8)
68
+ - ✗
69
+ - ✗
70
+ - ✗
71
+ - ✅︎
72
+ - ✅︎
73
+ - ✅︎
74
+ - ✗
75
+ - ✗
76
+ - ✗
77
+ - ✗
78
+ * - AQLM
79
+ - ✅︎
80
+ - ✅︎
81
+ - ✅︎
82
+ - ✅︎
83
+ - ✅︎
84
+ - ✗
85
+ - ✗
86
+ - ✗
87
+ - ✗
88
+ - ✗
89
+ * - bitsandbytes
90
+ - ✅︎
91
+ - ✅︎
92
+ - ✅︎
93
+ - ✅︎
94
+ - ✅︎
95
+ - ✗
96
+ - ✗
97
+ - ✗
98
+ - ✗
99
+ - ✗
100
+ * - DeepSpeedFP
101
+ - ✅︎
102
+ - ✅︎
103
+ - ✅︎
104
+ - ✅︎
105
+ - ✅︎
106
+ - ✗
107
+ - ✗
108
+ - ✗
109
+ - ✗
110
+ - ✗
111
+ * - GGUF
112
+ - ✅︎
113
+ - ✅︎
114
+ - ✅︎
115
+ - ✅︎
116
+ - ✅︎
117
+ - ✗
118
+ - ✗
119
+ - ✗
120
+ - ✗
121
+ - ✗
122
+ * - SqueezeLLM
123
+ - ✅︎
124
+ - ✅︎
125
+ - ✅︎
126
+ - ✅︎
127
+ - ✅︎
128
+ - ✗
129
+ - ✗
130
+ - ✗
131
+ - ✗
132
+ - ✗
133
134
Notes:
135
^^^^^^
136
137
- Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.
25
-- "✅" indicates that the quantization method is supported on the specified hardware.
26
-- "❌" indicates that the quantization method is not supported on the specified hardware.
138
+- "✅︎" indicates that the quantization method is supported on the specified hardware.
139
+- "✗" indicates that the quantization method is not supported on the specified hardware.
140
141
Please note that this compatibility chart may be subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
142