| 1 | # LLM Compressor |
| 2 | |
| 3 | [LLM Compressor](https://docs.vllm.ai/projects/llm-compressor/en/latest/) is a library for optimizing models for deployment with vLLM. |
| 4 | It provides a comprehensive set of quantization algorithms, including support for techniques such as FP4, FP8, INT8, and INT4 quantization. |
| 5 | |
| 6 | ## Why use LLM Compressor? |
| 7 | |
| 8 | Modern LLMs often contain billions of parameters stored in 16-bit or 32-bit floating point, requiring substantial GPU memory and limiting deployment options. |
| 9 | Quantization lowers memory requirements while maintaining inference output quality by reducing the precision of model weights and activations to smaller data types. |
| 10 | |
| 11 | LLM Compressor provides the following benefits: |
| 12 | |
| 13 | - **Reduced memory footprint**: Run larger models on smaller GPUs. |
| 14 | - **Lower inference costs**: Serve more concurrent users per GPU, directly reducing the cost per query in production deployments. |
| 15 | - **Faster inference**: Smaller data types mean less memory bandwidth consumed, which often translates to higher throughput, especially for memory-bound workloads. |
| 16 | |
| 17 | LLM Compressor handles the complexity of quantization, calibration, and format conversion, producing models ready for immediate use with vLLM. |
| 18 | |
| 19 | ## Key features |
| 20 | |
| 21 | - **Multiple Quantization Algorithms**: Support for AWQ, GPTQ, AutoRound, and Round-to-Nearest. |
| 22 | Also includes support for QuIP and SpinQuant-style transforms as well as KV cache and attention quantization. |
| 23 | - **Multiple Quantization Methods**: Support for FP8, INT8, INT4, NVFP4, MXFP4, and mixed-precision quantization |
| 24 | - **One-Shot Quantization**: Quantize models quickly with minimal calibration data |
| 25 | - **vLLM Integration**: Seamlessly deploy quantized models with vLLM using the compressed-tensors format |
| 26 | - **Hugging Face Compatibility**: Works with models from the Hugging Face Hub |
| 27 | |
| 28 | ## Resources |
| 29 | |
| 30 | - [LLM Compressor examples](https://github.com/vllm-project/llm-compressor/tree/main/examples) |
| 31 | - [GitHub Repository](https://github.com/vllm-project/llm-compressor) |