@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / e20233d361
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 19:30 UTC

Revert "[Doc] Update supported_hardware.rst (#7276)" (#7467)

Woosuk Kwon committed Aug 13, 2024 at 01:37 UTC e20233d361b4e6a7cb8e37c6d7f85e9900527802
1 file changed +13 -15
docs/source/quantization/supported_hardware.rst
+13 -15
@@ -5,20 +5,18 @@ Supported Hardware for Quantization Kernels
5
6 The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
7
8 -===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
9 -Implementation Volta Turing Ampere Ada Hopper AMD GPU Intel GPU x86 CPU AWS Inferentia Google TPU
10 -===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
11 -AWQ ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
12 -GPTQ ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
13 -Marlin (GPTQ/AWQ/FP8) ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
14 -INT8 (W8A8) ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
15 -FP8 (W8A8) ❌ ❌ ❌ ✅ ✅ ❌ ❌ ❌ ❌ ❌
16 -AQLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
17 -bitsandbytes ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
18 -DeepSpeedFP ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
19 -GGUF ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
20 -SqueezeLLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
21 -===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
8 +============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
9 +Implementation Volta Turing Ampere Ada Hopper AMD GPU Intel GPU x86 CPU AWS Inferentia Google TPU
10 +============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
11 +AQLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
12 +AWQ ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
13 +DeepSpeedFP ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
14 +FP8 ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
15 +Marlin ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
16 +GPTQ ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
17 +SqueezeLLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
18 +bitsandbytes ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
19 +============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
20
21 Notes:
22 ^^^^^^
@@ -29,4 +27,4 @@ Notes:
27
28 Please note that this compatibility chart may be subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
29
32 -For the most up-to-date information on hardware support and quantization methods, please check the `quantization directory <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization>`_ or consult with the vLLM development team.
30 +For the most up-to-date information on hardware support and quantization methods, please check the `quantization directory <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization>`_ or consult with the vLLM development team.
\ No newline at end of file