@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 6d94420246
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 19:30 UTC

[Doc] Update supported_hardware.rst (#7276)

Michael Goin committed Aug 7, 2024 at 17:21 UTC 6d94420246f2d88137c04f357502a6a8c38a2e8e
1 file changed +15 -13
docs/source/quantization/supported_hardware.rst
+15 -13
@@ -5,18 +5,20 @@ Supported Hardware for Quantization Kernels
5
6 The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
7
8 -============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
9 -Implementation Volta Turing Ampere Ada Hopper AMD GPU Intel GPU x86 CPU AWS Inferentia Google TPU
10 -============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
11 -AQLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
12 -AWQ ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
13 -DeepSpeedFP ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
14 -FP8 ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
15 -Marlin ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
16 -GPTQ ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
17 -SqueezeLLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
18 -bitsandbytes ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
19 -============== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
8 +===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
9 +Implementation Volta Turing Ampere Ada Hopper AMD GPU Intel GPU x86 CPU AWS Inferentia Google TPU
10 +===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
11 +AWQ ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
12 +GPTQ ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
13 +Marlin (GPTQ/AWQ/FP8) ❌ ❌ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
14 +INT8 (W8A8) ❌ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
15 +FP8 (W8A8) ❌ ❌ ❌ ✅ ✅ ❌ ❌ ❌ ❌ ❌
16 +AQLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
17 +bitsandbytes ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
18 +DeepSpeedFP ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
19 +GGUF ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
20 +SqueezeLLM ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌ ❌
21 +===================== ====== ======= ======= ===== ====== ======= ========= ======= ============== ==========
22
23 Notes:
24 ^^^^^^
@@ -27,4 +29,4 @@ Notes:
29
30 Please note that this compatibility chart may be subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
31
30 -For the most up-to-date information on hardware support and quantization methods, please check the `quantization directory <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization>`_ or consult with the vLLM development team.
\ No newline at end of file
32 +For the most up-to-date information on hardware support and quantization methods, please check the `quantization directory <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization>`_ or consult with the vLLM development team.