1
+(int4)=
2
+
3
+# INT4 W4A16
4
+
5
+vLLM supports quantizing weights to INT4 for memory savings and inference acceleration. This quantization method is particularly useful for reducing model size and maintaining low latency in workloads with low queries per second (QPS).
6
+
7
+Please visit the HF collection of [quantized INT4 checkpoints of popular LLMs ready to use with vLLM](https://huggingface.co/collections/neuralmagic/int4-llms-for-vllm-668ec34bf3c9fa45f857df2c).
8
+
9
+:::{note}
10
+INT4 computation is supported on NVIDIA GPUs with compute capability > 8.0 (Ampere, Ada Lovelace, Hopper, Blackwell).
11
+:::
12
+
13
+## Prerequisites
14
+
15
+To use INT4 quantization with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library:
16
+
17
+```console
18
+pip install llmcompressor
19
+```
20
+
21
+## Quantization Process
22
+
23
+The quantization process involves four main steps:
24
+
25
+1. Loading the model
26
+2. Preparing calibration data
27
+3. Applying quantization
28
+4. Evaluating accuracy in vLLM
29
+
30
+### 1. Loading the Model
31
+
32
+Load your model and tokenizer using the standard `transformers` AutoModel classes:
33
+
34
+```python
35
+from transformers import AutoTokenizer, AutoModelForCausalLM
36
+
37
+MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
38
+model = AutoModelForCausalLM.from_pretrained(
39
+ MODEL_ID, device_map="auto", torch_dtype="auto",
40
+)
41
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
42
+```
43
+
44
+### 2. Preparing Calibration Data
45
+
46
+When quantizing weights to INT4, you need sample data to estimate the weight updates and calibrated scales.
47
+It's best to use calibration data that closely matches your deployment data.
48
+For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
49
+
50
+```python
51
+from datasets import load_dataset
52
+
53
+NUM_CALIBRATION_SAMPLES = 512
54
+MAX_SEQUENCE_LENGTH = 2048
55
+
56
+# Load and preprocess the dataset
57
+ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
58
+ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
59
+
60
+def preprocess(example):
61
+ return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
62
+ds = ds.map(preprocess)
63
+
64
+def tokenize(sample):
65
+ return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
66
+ds = ds.map(tokenize, remove_columns=ds.column_names)
67
+```
68
+
69
+### 3. Applying Quantization
70
+
71
+Now, apply the quantization algorithms:
72
+
73
+```python
74
+from llmcompressor.transformers import oneshot
75
+from llmcompressor.modifiers.quantization import GPTQModifier
76
+from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
77
+
78
+# Configure the quantization algorithms
79
+recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
80
+
81
+# Apply quantization
82
+oneshot(
83
+ model=model,
84
+ dataset=ds,
85
+ recipe=recipe,
86
+ max_seq_length=MAX_SEQUENCE_LENGTH,
87
+ num_calibration_samples=NUM_CALIBRATION_SAMPLES,
88
+)
89
+
90
+# Save the compressed model
91
+SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A16-G128"
92
+model.save_pretrained(SAVE_DIR, save_compressed=True)
93
+tokenizer.save_pretrained(SAVE_DIR)
94
+```
95
+
96
+This process creates a W4A16 model with weights quantized to 4-bit integers.
97
+
98
+### 4. Evaluating Accuracy
99
+
100
+After quantization, you can load and run the model in vLLM:
101
+
102
+```python
103
+from vllm import LLM
104
+model = LLM("./Meta-Llama-3-8B-Instruct-W4A16-G128")
105
+```
106
+
107
+To evaluate accuracy, you can use `lm_eval`:
108
+
109
+```console
110
+$ lm_eval --model vllm \
111
+ --model_args pretrained="./Meta-Llama-3-8B-Instruct-W4A16-G128",add_bos_token=true \
112
+ --tasks gsm8k \
113
+ --num_fewshot 5 \
114
+ --limit 250 \
115
+ --batch_size 'auto'
116
+```
117
+
118
+:::{note}
119
+Quantized models can be sensitive to the presence of the `bos` token. Make sure to include the `add_bos_token=True` argument when running evaluations.
120
+:::
121
+
122
+## Best Practices
123
+
124
+- Start with 512 samples for calibration data, and increase if accuracy drops
125
+- Ensure the calibration data contains a high variety of samples to prevent overfitting towards a specific use case
126
+- Use a sequence length of 2048 as a starting point
127
+- Employ the chat template or instruction template that the model was trained with
128
+- If you've fine-tuned a model, consider using a sample of your training data for calibration
129
+- Tune key hyperparameters to the quantization algorithm:
130
+ - `dampening_frac` sets how much influence the GPTQ algorithm has. Lower values can improve accuracy, but can lead to numerical instabilities that cause the algorithm to fail.
131
+ - `actorder` sets the activation ordering. When compressing the weights of a layer weight, the order in which channels are quantized matters. Setting `actorder="weight"` can improve accuracy without added latency.
132
+
133
+The following is an example of an expanded quantization recipe you can tune to your own use case:
134
+
135
+```python
136
+from compressed_tensors.quantization import (
137
+ QuantizationArgs,
138
+ QuantizationScheme,
139
+ QuantizationStrategy,
140
+ QuantizationType,
141
+)
142
+recipe = GPTQModifier(
143
+ targets="Linear",
144
+ config_groups={
145
+ "config_group": QuantizationScheme(
146
+ targets=["Linear"],
147
+ weights=QuantizationArgs(
148
+ num_bits=4,
149
+ type=QuantizationType.INT,
150
+ strategy=QuantizationStrategy.GROUP,
151
+ group_size=128,
152
+ symmetric=True,
153
+ dynamic=False,
154
+ actorder="weight",
155
+ ),
156
+ ),
157
+ },
158
+ ignore=["lm_head"],
159
+ update_size=NUM_CALIBRATION_SAMPLES,
160
+ dampening_frac=0.01
161
+)
162
+```
163
+
164
+## Troubleshooting and Support
165
+
166
+If you encounter any issues or have feature requests, please open an issue on the [`vllm-project/llm-compressor`](https://github.com/vllm-project/llm-compressor) GitHub repository. The full INT4 quantization example in `llm-compressor` is available [here](https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w4a16/llama3_example.py).