| 1 | # Intel Quantization Support |
| 2 | |
| 3 | [AutoRound](https://github.com/intel/auto-round) is Intel’s advanced quantization algorithm designed for large language models(LLMs). It produces highly efficient **INT2, INT3, INT4, INT8, MXFP8, MXFP4, NVFP4**, and **GGUF** quantized models, balancing accuracy and inference performance. AutoRound is also part of the [Intel® Neural Compressor](https://github.com/intel/neural-compressor). For a deeper introduction, see the [AutoRound step-by-step guide](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md). |
| 4 | |
| 5 | ## Key Features |
| 6 | |
| 7 | ✅ Superior Accuracy Delivers strong performance even at 2–3 bits [example models](https://huggingface.co/collections/OPEA/2-3-bits) |
| 8 | |
| 9 | ✅ Fast Mixed `Bits`/`Dtypes` Scheme Generation Automatically configure in minutes |
| 10 | |
| 11 | ✅ Support for exporting **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** formats |
| 12 | |
| 13 | ✅ **10+ vision-language models (VLMs)** are supported |
| 14 | |
| 15 | ✅ **Per-layer mixed-bit quantization** for fine-grained control |
| 16 | |
| 17 | ✅ **RTN (Round-To-Nearest) mode** for quick quantization with slight accuracy loss |
| 18 | |
| 19 | ✅ **Multiple quantization recipes**: best, base, and light |
| 20 | |
| 21 | ✅ Advanced utilities such as immediate packing and support for **10+ backends** |
| 22 | |
| 23 | ## Supported Recipes on Intel Platforms |
| 24 | |
| 25 | On Intel platforms, AutoRound recipes are being enabled progressively by format and hardware. Currently, vLLM supports: |
| 26 | |
| 27 | - **`W4A16`**: weight-only, 4-bit weights with 16-bit activations |
| 28 | - **`W8A16`**: weight-only, 8-bit weights with 16-bit activations |
| 29 | |
| 30 | Additional recipes and formats will be supported in future releases. |
| 31 | |
| 32 | ## Quantizing a Model |
| 33 | |
| 34 | ### Installation |
| 35 | |
| 36 | ```bash |
| 37 | uv pip install auto-round |
| 38 | ``` |
| 39 | |
| 40 | ### Quantize with CLI |
| 41 | |
| 42 | ```bash |
| 43 | auto-round \ |
| 44 | --model Qwen/Qwen3-0.6B \ |
| 45 | --scheme W4A16 \ |
| 46 | --format auto_round \ |
| 47 | --output_dir ./tmp_autoround |
| 48 | ``` |
| 49 | |
| 50 | ### Quantize with Python API |
| 51 | |
| 52 | ```python |
| 53 | from transformers import AutoModelForCausalLM, AutoTokenizer |
| 54 | from auto_round import AutoRound |
| 55 | |
| 56 | model_name = "Qwen/Qwen3-0.6B" |
| 57 | autoround = AutoRound(model_name, scheme="W4A16") |
| 58 | |
| 59 | # the best accuracy, 4-5X slower, low_gpu_mem_usage could save ~20G but ~30% slower |
| 60 | # autoround = AutoRound(model, tokenizer, nsamples=512, iters=1000, low_gpu_mem_usage=True, bits=bits, group_size=group_size, sym=sym) |
| 61 | |
| 62 | # 2-3X speedup, slight accuracy drop at W4G128 |
| 63 | # autoround = AutoRound(model, tokenizer, nsamples=128, iters=50, lr=5e-3, bits=bits, group_size=group_size, sym=sym ) |
| 64 | |
| 65 | output_dir = "./tmp_autoround" |
| 66 | # format= 'auto_round'(default), 'auto_gptq', 'auto_awq' |
| 67 | autoround.quantize_and_save(output_dir, format="auto_round") |
| 68 | ``` |
| 69 | |
| 70 | ## Deploying AutoRound Quantized Models in vLLM |
| 71 | |
| 72 | ```bash |
| 73 | vllm serve Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound \ |
| 74 | --gpu-memory-utilization 0.8 \ |
| 75 | --max-model-len 4096 |
| 76 | ``` |
| 77 | |
| 78 | ## Evaluating the Quantized Model with vLLM |
| 79 | |
| 80 | ```bash |
| 81 | lm_eval --model vllm \ |
| 82 | --model_args pretrained="Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound,max_model_len=8192,max_num_batched_tokens=32768,max_num_seqs=128,gpu_memory_utilization=0.8,dtype=bfloat16,max_gen_toks=2048" \ |
| 83 | --tasks gsm8k \ |
| 84 | --num_fewshot 5 \ |
| 85 | --batch_size 128 |
| 86 | ``` |