@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 4cefa9b49b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 06:45 UTC

[Docs] Update the AWQ documentation to highlight performance issue (#1883)

Simon Mo committed Dec 2, 2023 at 15:52 UTC 4cefa9b49b6cb2be6d7eac88315df65e0f0d8c9a
1 file changed +6
docs/source/quantization/auto_awq.rst
+6
@@ -3,6 +3,12 @@
3 AutoAWQ
4 ==================
5
6 +.. warning::
7 +
8 + Please note that AWQ support in vLLM is under-optimized at the moment. We would recommend using the unquantized version of the model for better
9 + accuracy and higher throughput. Currently, you can use AWQ as a way to reduce memory footprint. As of now, it is more suitable for low latency
10 + inference with small number of concurrent requests. vLLM's AWQ implementation have lower throughput than unquantized version.
11 +
12 To create a new 4-bit quantized model, you can leverage `AutoAWQ <https://github.com/casper-hansen/AutoAWQ>`_.
13 Quantizing reduces the model's precision from FP16 to INT4 which effectively reduces the file size by ~70%.
14 The main benefits are lower latency and memory usage.