@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 173 lines 5.92 KB
Rendered Raw
1 # INT4 W4A16
2
3 vLLM supports quantizing weights to INT4 for memory savings and inference acceleration. This quantization method is particularly useful for reducing model size and maintaining low latency in workloads with low queries per second (QPS).
4
5 Please visit the HF collection of [quantized INT4 checkpoints of popular LLMs ready to use with vLLM](https://huggingface.co/collections/neuralmagic/int4-llms-for-vllm-668ec34bf3c9fa45f857df2c).
6
7 !!! note
8 INT4 computation is supported on NVIDIA GPUs with compute capability > 8.0 (Ampere, Ada Lovelace, Hopper, Blackwell).
9
10 ## Prerequisites
11
12 To use INT4 quantization with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library:
13
14 ```bash
15 (venv-llm-compressor) pip install llmcompressor
16 ```
17
18 Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:
19
20 ```bash
21 (venv-vllm) pip install vllm "lm-eval[api]>=0.4.12"
22 ```
23
24 Please use separate environments for vLLM and llm-compressor as they might not work together.
25
26 ## Quantization Process
27
28 The quantization process involves four main steps:
29
30 1. Loading the model
31 2. Preparing calibration data
32 3. Applying quantization
33 4. Evaluating accuracy in vLLM
34
35 ### 1. Loading the Model
36
37 Load your model and tokenizer using the standard `transformers` AutoModel classes:
38
39 ```python
40 from transformers import AutoTokenizer, AutoModelForCausalLM
41
42 MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
43 model = AutoModelForCausalLM.from_pretrained(
44 MODEL_ID,
45 device_map="auto",
46 dtype="auto",
47 )
48 tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
49 ```
50
51 ### 2. Preparing Calibration Data
52
53 When quantizing weights to INT4, you need sample data to estimate the weight updates and calibrated scales.
54 It's best to use calibration data that closely matches your deployment data.
55 For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
56
57 ```python
58 from datasets import load_dataset
59
60 NUM_CALIBRATION_SAMPLES = 512
61 MAX_SEQUENCE_LENGTH = 2048
62
63 # Load and preprocess the dataset
64 ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
65 ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
66
67 def preprocess(example):
68 return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
69 ds = ds.map(preprocess)
70
71 def tokenize(sample):
72 return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
73 ds = ds.map(tokenize, remove_columns=ds.column_names)
74 ```
75
76 ### 3. Applying Quantization
77
78 Now, apply the quantization algorithms:
79
80 ```python
81 from llmcompressor import oneshot
82 from llmcompressor.modifiers.quantization import GPTQModifier
83 from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
84
85 # Configure the quantization algorithms
86 recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
87
88 # Apply quantization
89 oneshot(
90 model=model,
91 dataset=ds,
92 recipe=recipe,
93 max_seq_length=MAX_SEQUENCE_LENGTH,
94 num_calibration_samples=NUM_CALIBRATION_SAMPLES,
95 )
96
97 # Save the compressed model: Meta-Llama-3-8B-Instruct-W4A16-G128
98 SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A16-G128"
99 model.save_pretrained(SAVE_DIR, save_compressed=True)
100 tokenizer.save_pretrained(SAVE_DIR)
101 ```
102
103 This process creates a W4A16 model with weights quantized to 4-bit integers.
104
105 ### 4. Evaluating Accuracy
106
107 After quantization, you can load and run the model in vLLM:
108
109 ```python
110 from vllm import LLM
111
112 llm = LLM("./Meta-Llama-3-8B-Instruct-W4A16-G128")
113 ```
114
115 To evaluate accuracy, you can use `lm_eval`:
116
117 ```bash
118 lm_eval --model vllm \
119 --model_args pretrained="./Meta-Llama-3-8B-Instruct-W4A16-G128",add_bos_token=true \
120 --tasks gsm8k \
121 --num_fewshot 5 \
122 --limit 250 \
123 --batch_size 'auto'
124 ```
125
126 !!! note
127 Quantized models can be sensitive to the presence of the `bos` token. Make sure to include the `add_bos_token=True` argument when running evaluations.
128
129 ## Best Practices
130
131 - Start with 512 samples for calibration data, and increase if accuracy drops
132 - Ensure the calibration data contains a high variety of samples to prevent overfitting towards a specific use case
133 - Use a sequence length of 2048 as a starting point
134 - Employ the chat template or instruction template that the model was trained with
135 - If you've fine-tuned a model, consider using a sample of your training data for calibration
136 - Tune key hyperparameters to the quantization algorithm:
137 - `dampening_frac` sets how much influence the GPTQ algorithm has. Lower values can improve accuracy, but can lead to numerical instabilities that cause the algorithm to fail.
138 - `actorder` sets the activation ordering. When compressing the weights of a layer weight, the order in which channels are quantized matters. Setting `actorder="weight"` can improve accuracy without added latency.
139
140 The following is an example of an expanded quantization recipe you can tune to your own use case:
141
142 ```python
143 from compressed_tensors.quantization import (
144 QuantizationArgs,
145 QuantizationScheme,
146 QuantizationStrategy,
147 QuantizationType,
148 )
149 recipe = GPTQModifier(
150 targets="Linear",
151 config_groups={
152 "config_group": QuantizationScheme(
153 targets=["Linear"],
154 weights=QuantizationArgs(
155 num_bits=4,
156 type=QuantizationType.INT,
157 strategy=QuantizationStrategy.GROUP,
158 group_size=128,
159 symmetric=True,
160 dynamic=False,
161 actorder="weight",
162 ),
163 ),
164 },
165 ignore=["lm_head"],
166 update_size=NUM_CALIBRATION_SAMPLES,
167 dampening_frac=0.01,
168 )
169 ```
170
171 ## Troubleshooting and Support
172
173 If you encounter any issues or have feature requests, please open an issue on the [vllm-project/llm-compressor](https://github.com/vllm-project/llm-compressor/issues) GitHub repository. The full INT4 quantization example in `llm-compressor` is available [here](https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w4a16/llama3_example.py).