@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / c53dc466b1
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 00:45 UTC

[Doc] Remove performance warning for auto_awq.md (#12743)

Michael Goin committed Feb 5, 2025 at 01:43 UTC c53dc466b1b802d45d0b61cce36908334ea7a23e
1 file changed -6
docs/source/features/quantization/auto_awq.md
-6
@@ -2,12 +2,6 @@
2
3 # AutoAWQ
4
5 -:::{warning}
6 -Please note that AWQ support in vLLM is under-optimized at the moment. We would recommend using the unquantized version of the model for better
7 -accuracy and higher throughput. Currently, you can use AWQ as a way to reduce memory footprint. As of now, it is more suitable for low latency
8 -inference with small number of concurrent requests. vLLM's AWQ implementation have lower throughput than unquantized version.
9 -:::
10 -
5 To create a new 4-bit quantized model, you can leverage [AutoAWQ](https://github.com/casper-hansen/AutoAWQ).
6 Quantizing reduces the model's precision from FP16 to INT4 which effectively reduces the file size by ~70%.
7 The main benefits are lower latency and memory usage.