A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 19:30 UTC
[Doc] Update supported_hardware.rst (#7276)
Michael Goin committed
Aug 7, 2024 at 17:21 UTC
6d94420246f2d88137c04f357502a6a8c38a2e8e
1 file changed
+15
-13
docs/source/quantization/supported_hardware.rst
+15
-13
@@ -5,18 +5,20 @@ Supported Hardware for Quantization Kernels
5
6
The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
7
8
-============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
9
-Implementation Volta Turing Ampere Ada Hopper AMD GPU Intel GPU x86 CPU AWS Inferentia Google TPU
10
-============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
11
-AQLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
12
-AWQ ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
13
-DeepSpeedFP ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
14
-FP8 ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
15
-Marlin ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
16
-GPTQ ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
17
-SqueezeLLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
18
-bitsandbytes ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
19
-============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
8
+===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
9
+Implementation Volta Turing Ampere Ada Hopper AMD GPU Intel GPU x86 CPU AWS Inferentia Google TPU
10
+===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
11
+AWQ ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
12
+GPTQ ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
13
+Marlin (GPTQ/AWQ/FP8) ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
14
+INT8 (W8A8) ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
15
+FP8 (W8A8) ❌ ❌ ❌ ✅ ✅ ❌ ❌ ❌ ❌ ❌
16
+AQLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
17
+bitsandbytes ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
18
+DeepSpeedFP ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
19
+GGUF ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
20
+SqueezeLLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
21
+===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
22
23
Notes:
24
^^^^^^
@@ -27,4 +29,4 @@ Notes:
29
30
Please note that this compatibility chart may be subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
31
30
-For the most up-to-date information on hardware support and quantization methods, please check the `quantization directory <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization>`_ or consult with the vLLM development team.
\ No newline at end of file
32
+For the most up-to-date information on hardware support and quantization methods, please check the `quantization directory <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization>`_ or consult with the vLLM development team.