A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 23:45 UTC
[Docs] add auto-round quantization readme (#21600)
Signed-off-by: Wenhua Cheng <wenhua.cheng@intel.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Wenhua Cheng committed
Jul 25, 2025 at 23:52 UTC
5ac3168ee342f4cae17b0b67375e647bd5dd9151
2 files changed
+104
docs/features/quantization/README.md
+1
@@ -6,6 +6,7 @@ Contents:
6
7
- [Supported Hardware](supported_hardware.md)
8
- [AutoAWQ](auto_awq.md)
9
+- [AutoRound](auto_round.md)
10
- [BitsAndBytes](bnb.md)
11
- [BitBLAS](bitblas.md)
12
- [GGUF](gguf.md)
docs/features/quantization/auto_round.md
new
+103
@@ -0,0 +1,103 @@
1
+# AutoRound
2
+
3
+[AutoRound](https://github.com/intel/auto-round) is Intel’s advanced quantization algorithm designed to produce highly efficient **INT2, INT3, INT4, and INT8**
4
+quantized large language models—striking an optimal balance between accuracy and deployment performance.
5
+
6
+AutoRound applies weight-only quantization to transformer-based models, enabling significant memory savings and faster
7
+inference while maintaining near-original accuracy. It supports a wide range of hardware platforms, including **CPUs,
8
+Intel GPUs, HPUs, and CUDA-enabled devices**.
9
+
10
+Please refer to the [AutoRound guide](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md) for more details.
11
+
12
+Key Features:
13
+
14
+✅ **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** are supported
15
+
16
+✅ **10+ vision-language models (VLMs)** are supported
17
+
18
+✅ **Per-layer mixed-bit quantization** for fine-grained control
19
+
20
+✅ **RTN (Round-To-Nearest) mode** for quick quantization with slight accuracy loss
21
+
22
+✅ **Multiple quantization recipes**: best, base, and light
23
+
24
+✅ Advanced utilities such as immediate packing and support for **10+ backends**
25
+
26
+## Installation
27
+
28
+```bash
29
+uv pip install auto-round
30
+```
31
+
32
+## Quantizing a model
33
+
34
+For VLMs, please change to `auto-round-mllm` in CLI usage and `AutoRoundMLLM` in API usage.
35
+
36
+### CLI usage
37
+
38
+```bash
39
+auto-round \
40
+ --model Qwen/Qwen3-0.6B \
41
+ --bits 4 \
42
+ --group_size 128 \
43
+ --format "auto_round" \
44
+ --output_dir ./tmp_autoround
45
+```
46
+
47
+```bash
48
+auto-round \
49
+ --model Qwen/Qwen3-0.6B \
50
+ --format "gguf:q4_k_m" \
51
+ --output_dir ./tmp_autoround
52
+```
53
+
54
+### API usage
55
+
56
+```python
57
+from transformers import AutoModelForCausalLM, AutoTokenizer
58
+from auto_round import AutoRound
59
+
60
+model_name = "Qwen/Qwen3-0.6B"
61
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")
62
+tokenizer = AutoTokenizer.from_pretrained(model_name)
63
+
64
+bits, group_size, sym = 4, 128, True
65
+autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, sym=sym)
66
+
67
+# the best accuracy, 4-5X slower, low_gpu_mem_usage could save ~20G but ~30% slower
68
+# autoround = AutoRound(model, tokenizer, nsamples=512, iters=1000, low_gpu_mem_usage=True, bits=bits, group_size=group_size, sym=sym)
69
+
70
+# 2-3X speedup, slight accuracy drop at W4G128
71
+# autoround = AutoRound(model, tokenizer, nsamples=128, iters=50, lr=5e-3, bits=bits, group_size=group_size, sym=sym )
72
+
73
+output_dir = "./tmp_autoround"
74
+# format= 'auto_round'(default), 'auto_gptq', 'auto_awq'
75
+autoround.quantize_and_save(output_dir, format="auto_round")
76
+```
77
+
78
+## Running a quantized model with vLLM
79
+
80
+Here is some example code to run auto-round format in vLLM:
81
+
82
+```python
83
+from vllm import LLM, SamplingParams
84
+
85
+prompts = [
86
+ "Hello, my name is",
87
+]
88
+sampling_params = SamplingParams(temperature=0.6, top_p=0.95)
89
+model_name = "Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound"
90
+llm = LLM(model=model_name)
91
+
92
+outputs = llm.generate(prompts, sampling_params)
93
+
94
+for output in outputs:
95
+ prompt = output.prompt
96
+ generated_text = output.outputs[0].text
97
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
98
+```
99
+
100
+# Acknowledgement
101
+
102
+Special thanks to open-source low precision libraries such as AutoGPTQ, AutoAWQ, GPTQModel, Triton, Marlin, and
103
+ExLLaMAV2 for providing low-precision CUDA kernels, which are leveraged in AutoRound.