A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 19:30 UTC
Revert "[Doc] Update supported_hardware.rst (#7276)" (#7467)
Woosuk Kwon committed
Aug 13, 2024 at 01:37 UTC
e20233d361b4e6a7cb8e37c6d7f85e9900527802
1 file changed
+13
-15
docs/source/quantization/supported_hardware.rst
+13
-15
@@ -5,20 +5,18 @@ Supported Hardware for Quantization Kernels
5
6
The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
7
8
-===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
9
-Implementation Volta Turing Ampere Ada Hopper AMD GPU Intel GPU x86 CPU AWS Inferentia Google TPU
10
-===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
11
-AWQ ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
12
-GPTQ ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
13
-Marlin (GPTQ/AWQ/FP8) ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
14
-INT8 (W8A8) ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
15
-FP8 (W8A8) ❌ ❌ ❌ ✅ ✅ ❌ ❌ ❌ ❌ ❌
16
-AQLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
17
-bitsandbytes ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
18
-DeepSpeedFP ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
19
-GGUF ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
20
-SqueezeLLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
21
-===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
8
+============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
9
+Implementation Volta Turing Ampere Ada Hopper AMD GPU Intel GPU x86 CPU AWS Inferentia Google TPU
10
+============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
11
+AQLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
12
+AWQ ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
13
+DeepSpeedFP ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
14
+FP8 ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
15
+Marlin ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
16
+GPTQ ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
17
+SqueezeLLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
18
+bitsandbytes ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
19
+============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
20
21
Notes:
22
^^^^^^
@@ -29,4 +27,4 @@ Notes:
27
28
Please note that this compatibility chart may be subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
29
32
-For the most up-to-date information on hardware support and quantization methods, please check the `quantization directory <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization>`_ or consult with the vLLM development team.
30
+For the most up-to-date information on hardware support and quantization methods, please check the `quantization directory <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization>`_ or consult with the vLLM development team.
\ No newline at end of file