A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 1, 22:30 UTC
[DOC] Add INT8 W4A8 docs and Arm's supported quantization schemes (#34894)
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
Fadi Arafeh committed
Jun 4, 2026 at 17:27 UTC
3da29aa4a5509b068d378bd8aedbe4837cecf6a6
7 files changed
+372
-155
docs/features/quantization/README.md
+17
-14
@@ -3,7 +3,7 @@
3
Quantization trades off model precision for smaller memory footprint, allowing large models to be run on a wider range of devices.
4
5
!!! tip
6
- To get started with quantization, see [LLM Compressor](llm_compressor.md), a library for optimizing models for deployment with vLLM that supports FP8, INT8, INT4, and other quantization formats.
6
+ To get started with quantization, see [LLM Compressor](llm_compressor/README.md), a library for optimizing models for deployment with vLLM that supports FP8, INT8, INT4, and other quantization formats.
7
8
The following are the supported quantization formats for vLLM:
9
@@ -12,9 +12,11 @@ The following are the supported quantization formats for vLLM:
12
- [GGUF](gguf.md)
13
- [GPTQModel](gptqmodel.md)
14
- [Intel Neural Compressor](inc.md)
15
-- [INT4 W4A16](int4.md)
16
-- [INT8 W8A8](int8.md)
17
-- [FP8 W8A8](fp8.md)
15
+- [LLM Compressor](llm_compressor/README.md)
16
+ - [FP8 W8A8](llm_compressor/fp8.md)
17
+ - [INT4 W4A16](llm_compressor/int4.md)
18
+ - [INT8 W4A8](llm_compressor/int8_w4a8.md)
19
+ - [INT8 W8A8](llm_compressor/int8_w8a8.md)
20
- [NVIDIA Model Optimizer](modelopt.md)
21
- [Online Quantization](online.md)
22
- [AMD Quark](quark.md)
@@ -46,16 +48,17 @@ th:not(:first-child) {
48
}
49
</style>
50
49
-| Implementation | Volta | Turing | Ampere | Ada | Hopper | AMD GPU | Intel GPU | x86 CPU |
50
-| ------------------------- | ----- | ------ | ------ | --- | ------ | ------- | --------- | ------- |
51
-| AWQ | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ |
52
-| GPTQ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ |
53
-| Marlin (GPTQ/AWQ/FP8/FP4) | ❌ | ✅︎* | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
54
-| INT8 (W8A8) | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ✅︎ |
55
-| FP8 (W8A8) | ❌ | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ |
56
-| bitsandbytes | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
57
-| DeepSpeedFP | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
58
-| GGUF | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ |
51
+| Implementation | Volta | Turing | Ampere | Ada | Hopper | AMD GPU | Intel GPU | x86 CPU | Arm CPU |
52
+| ------------------------- | ----- | ------ | ------ | --- | ------ | ------- | --------- | ------- | ------- |
53
+| AWQ | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ | ❌ |
54
+| GPTQ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ | ❌ |
55
+| Marlin (GPTQ/AWQ/FP8/FP4) | ❌ | ✅︎* | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ |
56
+| llm-compressor INT8 (W8A8)| ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ✅︎ | ✅︎ |
57
+| llm-compressor INT8 (W4A8)| ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅︎ |
58
+| llm-compressor FP8 (W8A8) | ❌ | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
59
+| bitsandbytes | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ |
60
+| DeepSpeedFP | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ |
61
+| GGUF | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
62
63
- Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.
64
- ✅︎ indicates that the quantization method is supported on the specified hardware.
docs/features/quantization/llm_compressor/README.md
renamed
docs/features/quantization/llm_compressor/fp8.md
renamed
+26
-26
@@ -21,9 +21,17 @@ The FP8 types typically supported in hardware have two distinct representations,
21
To produce performant FP8 quantized models with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library:
22
23
```bash
24
-pip install llmcompressor
24
+(venv-llm-compressor) pip install llmcompressor
25
```
26
27
+Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:
28
+
29
+```bash
30
+(venv-vllm) pip install vllm "lm-eval[api]>=0.4.12"
31
+```
32
+
33
+Please use separate environments for vLLM and llm-compressor as they might not work together.
34
+
35
## Quantization Process
36
37
The quantization process involves three main steps:
@@ -57,36 +65,28 @@ For FP8 quantization, we can recover accuracy with simple RTN quantization. We r
65
66
Since simple RTN does not require data for weight quantization and the activations are quantized dynamically, we do not need any calibration data for this quantization flow.
67
60
-??? code
61
-
62
- ```python
63
- from llmcompressor import oneshot
64
- from llmcompressor.modifiers.quantization import QuantizationModifier
65
-
66
- # Configure the simple PTQ quantization
67
- recipe = QuantizationModifier(
68
- targets="Linear",
69
- scheme="FP8_DYNAMIC",
70
- ignore=["lm_head"],
71
- )
68
+```python
69
+from llmcompressor import oneshot
70
+from llmcompressor.modifiers.quantization import QuantizationModifier
71
+
72
+# Configure the simple PTQ quantization
73
+recipe = QuantizationModifier(
74
+ targets="Linear",
75
+ scheme="FP8_DYNAMIC",
76
+ ignore=["lm_head"],
77
+)
78
73
- # Apply the quantization algorithm.
74
- oneshot(model=model, recipe=recipe)
79
+# Apply the quantization algorithm.
80
+oneshot(model=model, recipe=recipe)
81
76
- # Save the model: Meta-Llama-3-8B-Instruct-FP8-Dynamic
77
- SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
78
- model.save_pretrained(SAVE_DIR)
79
- tokenizer.save_pretrained(SAVE_DIR)
80
- ```
82
+# Save the model: Meta-Llama-3-8B-Instruct-FP8-Dynamic
83
+SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
84
+model.save_pretrained(SAVE_DIR)
85
+tokenizer.save_pretrained(SAVE_DIR)
86
+```
87
88
### 3. Evaluating Accuracy
89
84
-Install `vllm` and `lm-evaluation-harness` for evaluation:
85
-
86
-```bash
87
-pip install vllm "lm-eval[api]>=0.4.12"
88
-```
89
-
90
Load and run the model in `vllm`:
91
92
```python
docs/features/quantization/llm_compressor/int4.md
renamed
+66
-70
@@ -12,15 +12,17 @@ Please visit the HF collection of [quantized INT4 checkpoints of popular LLMs re
12
To use INT4 quantization with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library:
13
14
```bash
15
-pip install llmcompressor
15
+(venv-llm-compressor) pip install llmcompressor
16
```
17
18
Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:
19
20
```bash
21
-pip install vllm "lm-eval[api]>=0.4.12"
21
+(venv-vllm) pip install vllm "lm-eval[api]>=0.4.12"
22
```
23
24
+Please use separate environments for vLLM and llm-compressor as they might not work together.
25
+
26
## Quantization Process
27
28
The quantization process involves four main steps:
@@ -52,55 +54,51 @@ When quantizing weights to INT4, you need sample data to estimate the weight upd
54
It's best to use calibration data that closely matches your deployment data.
55
For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
56
55
-??? code
56
-
57
- ```python
58
- from datasets import load_dataset
57
+```python
58
+from datasets import load_dataset
59
60
- NUM_CALIBRATION_SAMPLES = 512
61
- MAX_SEQUENCE_LENGTH = 2048
60
+NUM_CALIBRATION_SAMPLES = 512
61
+MAX_SEQUENCE_LENGTH = 2048
62
63
- # Load and preprocess the dataset
64
- ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
65
- ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
63
+# Load and preprocess the dataset
64
+ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
65
+ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
66
67
- def preprocess(example):
68
- return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
69
- ds = ds.map(preprocess)
67
+def preprocess(example):
68
+ return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
69
+ds = ds.map(preprocess)
70
71
- def tokenize(sample):
72
- return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
73
- ds = ds.map(tokenize, remove_columns=ds.column_names)
74
- ```
71
+def tokenize(sample):
72
+ return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
73
+ds = ds.map(tokenize, remove_columns=ds.column_names)
74
+```
75
76
### 3. Applying Quantization
77
78
Now, apply the quantization algorithms:
79
80
-??? code
81
-
82
- ```python
83
- from llmcompressor import oneshot
84
- from llmcompressor.modifiers.quantization import GPTQModifier
85
- from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
86
-
87
- # Configure the quantization algorithms
88
- recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
89
-
90
- # Apply quantization
91
- oneshot(
92
- model=model,
93
- dataset=ds,
94
- recipe=recipe,
95
- max_seq_length=MAX_SEQUENCE_LENGTH,
96
- num_calibration_samples=NUM_CALIBRATION_SAMPLES,
97
- )
80
+```python
81
+from llmcompressor import oneshot
82
+from llmcompressor.modifiers.quantization import GPTQModifier
83
+from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
84
+
85
+# Configure the quantization algorithms
86
+recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
87
+
88
+# Apply quantization
89
+oneshot(
90
+ model=model,
91
+ dataset=ds,
92
+ recipe=recipe,
93
+ max_seq_length=MAX_SEQUENCE_LENGTH,
94
+ num_calibration_samples=NUM_CALIBRATION_SAMPLES,
95
+)
96
99
- # Save the compressed model: Meta-Llama-3-8B-Instruct-W4A16-G128
100
- SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A16-G128"
101
- model.save_pretrained(SAVE_DIR, save_compressed=True)
102
- tokenizer.save_pretrained(SAVE_DIR)
103
- ```
97
+# Save the compressed model: Meta-Llama-3-8B-Instruct-W4A16-G128
98
+SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A16-G128"
99
+model.save_pretrained(SAVE_DIR, save_compressed=True)
100
+tokenizer.save_pretrained(SAVE_DIR)
101
+```
102
103
This process creates a W4A16 model with weights quantized to 4-bit integers.
104
@@ -141,36 +139,34 @@ lm_eval --model vllm \
139
140
The following is an example of an expanded quantization recipe you can tune to your own use case:
141
144
-??? code
145
-
146
- ```python
147
- from compressed_tensors.quantization import (
148
- QuantizationArgs,
149
- QuantizationScheme,
150
- QuantizationStrategy,
151
- QuantizationType,
152
- )
153
- recipe = GPTQModifier(
154
- targets="Linear",
155
- config_groups={
156
- "config_group": QuantizationScheme(
157
- targets=["Linear"],
158
- weights=QuantizationArgs(
159
- num_bits=4,
160
- type=QuantizationType.INT,
161
- strategy=QuantizationStrategy.GROUP,
162
- group_size=128,
163
- symmetric=True,
164
- dynamic=False,
165
- actorder="weight",
166
- ),
142
+```python
143
+from compressed_tensors.quantization import (
144
+ QuantizationArgs,
145
+ QuantizationScheme,
146
+ QuantizationStrategy,
147
+ QuantizationType,
148
+)
149
+recipe = GPTQModifier(
150
+ targets="Linear",
151
+ config_groups={
152
+ "config_group": QuantizationScheme(
153
+ targets=["Linear"],
154
+ weights=QuantizationArgs(
155
+ num_bits=4,
156
+ type=QuantizationType.INT,
157
+ strategy=QuantizationStrategy.GROUP,
158
+ group_size=128,
159
+ symmetric=True,
160
+ dynamic=False,
161
+ actorder="weight",
162
),
168
- },
169
- ignore=["lm_head"],
170
- update_size=NUM_CALIBRATION_SAMPLES,
171
- dampening_frac=0.01,
172
- )
173
- ```
163
+ ),
164
+ },
165
+ ignore=["lm_head"],
166
+ update_size=NUM_CALIBRATION_SAMPLES,
167
+ dampening_frac=0.01,
168
+)
169
+```
170
171
## Troubleshooting and Support
172
docs/features/quantization/llm_compressor/int8_w4a8.md
new
+217
@@ -0,0 +1,217 @@
1
+# INT8 W4A8
2
+
3
+vLLM supports quantizing weights to INT4 and activations to INT8 for memory savings and inference acceleration.
4
+This quantization method is particularly useful for reducing model size while maintaining good performance.
5
+
6
+## Prerequisites
7
+
8
+To use INT8 W4A8 quantization with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library.
9
+
10
+```bash
11
+(venv-llm-compressor) pip install llmcompressor
12
+```
13
+
14
+Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:
15
+
16
+```bash
17
+(venv-vllm) pip install vllm "lm-eval[api]>=0.4.12"
18
+```
19
+
20
+Please use separate environments for vLLM and llm-compressor as they might not work together.
21
+
22
+## Quantization Process
23
+
24
+The quantization process involves four main steps:
25
+
26
+1. Loading the model
27
+2. Preparing calibration data
28
+3. Applying quantization
29
+4. Evaluating accuracy in vLLM
30
+
31
+### 1. Loading the Model
32
+
33
+Load your model and tokenizer using the standard `transformers` AutoModel classes:
34
+
35
+```python
36
+from transformers import AutoTokenizer, AutoModelForCausalLM
37
+
38
+MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
39
+model = AutoModelForCausalLM.from_pretrained(
40
+ MODEL_ID,
41
+ dtype="auto",
42
+)
43
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
44
+```
45
+
46
+### 2. Preparing Calibration Data
47
+
48
+When quantizing activations to INT8 and weights to INT4, you need sample data to estimate the activation scales.
49
+It's best to use calibration data that closely matches your deployment data.
50
+For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
51
+
52
+```python
53
+from datasets import load_dataset
54
+
55
+NUM_CALIBRATION_SAMPLES = 512
56
+MAX_SEQUENCE_LENGTH = 2048
57
+
58
+# Load and preprocess the dataset
59
+ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
60
+ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
61
+
62
+def preprocess(example):
63
+ return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
64
+ds = ds.map(preprocess)
65
+
66
+def tokenize(sample):
67
+ return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
68
+ds = ds.map(tokenize, remove_columns=ds.column_names)
69
+```
70
+
71
+### 3. Applying Quantization
72
+
73
+Now, apply the quantization algorithms.
74
+
75
+The following recipes create W4A8 models (int4 weights, int8 activations). On Arm® CPUs, this is accelerated through [KleidiAI](https://github.com/ARM-software/kleidiai).
76
+
77
+Use groupwise for best accuracy, and channelwise for best inference performance.
78
+
79
+=== "Groupwise"
80
+
81
+ ```python
82
+ from llmcompressor import oneshot
83
+ from llmcompressor.modifiers.quantization import GPTQModifier
84
+
85
+ # Configure the quantization algorithms
86
+ recipe = [
87
+ GPTQModifier(
88
+ targets="Linear",
89
+ scheme="W4A8",
90
+ ignore=["lm_head"],
91
+ dampening_frac=0.01
92
+ ),
93
+ ]
94
+
95
+ # Apply quantization
96
+ oneshot(
97
+ model=model,
98
+ dataset=ds,
99
+ recipe=recipe,
100
+ max_seq_length=MAX_SEQUENCE_LENGTH,
101
+ num_calibration_samples=NUM_CALIBRATION_SAMPLES,
102
+ )
103
+
104
+ # Save the compressed model: Meta-Llama-3-8B-Instruct-W4A8-G128-Dynamic-Per-Token
105
+ SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A8-G128-Dynamic-Per-Token"
106
+ model.save_pretrained(SAVE_DIR, save_compressed=True)
107
+ tokenizer.save_pretrained(SAVE_DIR)
108
+ ```
109
+
110
+=== "Channelwise"
111
+
112
+ ```python
113
+ from llmcompressor import oneshot
114
+ from llmcompressor.modifiers.quantization import GPTQModifier
115
+ from compressed_tensors.quantization import QuantizationStrategy, QuantizationType
116
+
117
+ scheme = {
118
+ "targets": ["Linear"],
119
+ "weights": {
120
+ "num_bits": 4,
121
+ "type": QuantizationType.INT,
122
+ "strategy": QuantizationStrategy.CHANNEL,
123
+ "symmetric": True,
124
+ "dynamic": False,
125
+ "group_size": None,
126
+ },
127
+ "input_activations": {
128
+ "num_bits": 8,
129
+ "type": QuantizationType.INT,
130
+ "strategy": QuantizationStrategy.TOKEN,
131
+ "dynamic": True,
132
+ "symmetric": False,
133
+ "observer": None,
134
+ },
135
+ "output_activations": None,
136
+ }
137
+
138
+ recipe = [
139
+ GPTQModifier(
140
+ targets="Linear",
141
+ config_groups={"group_0": scheme},
142
+ ignore=["lm_head"],
143
+ dampening_frac=0.01,
144
+ ),
145
+ ]
146
+
147
+ oneshot(
148
+ model=model,
149
+ dataset=ds,
150
+ recipe=recipe,
151
+ max_seq_length=MAX_SEQUENCE_LENGTH,
152
+ num_calibration_samples=NUM_CALIBRATION_SAMPLES,
153
+ )
154
+
155
+ # Save the compressed model: Meta-Llama-3-8B-Instruct-W4A8-Channelwise-Dynamic-Per-Token
156
+ SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A8-Channelwise-Dynamic-Per-Token"
157
+ model.save_pretrained(SAVE_DIR, save_compressed=True)
158
+ tokenizer.save_pretrained(SAVE_DIR)
159
+ ```
160
+
161
+### 4. Evaluating Accuracy
162
+
163
+=== "Groupwise"
164
+
165
+ After quantization, you can load and run the model in vLLM:
166
+
167
+ ```python
168
+ from vllm import LLM
169
+
170
+ llm = LLM("./Meta-Llama-3-8B-Instruct-W4A8-G128-Dynamic-Per-Token")
171
+ ```
172
+
173
+ To evaluate accuracy, you can use `lm_eval`:
174
+
175
+ ```bash
176
+ lm_eval --model vllm \
177
+ --model_args pretrained="./Meta-Llama-3-8B-Instruct-W4A8-G128-Dynamic-Per-Token",add_bos_token=true \
178
+ --tasks gsm8k \
179
+ --num_fewshot 5 \
180
+ --limit 250 \
181
+ --batch_size 'auto'
182
+ ```
183
+
184
+=== "Channelwise"
185
+
186
+ After quantization, you can load and run the model in vLLM:
187
+
188
+ ```python
189
+ from vllm import LLM
190
+
191
+ llm = LLM("./Meta-Llama-3-8B-Instruct-W4A8-Channelwise-Dynamic-Per-Token")
192
+ ```
193
+
194
+ To evaluate accuracy, you can use `lm_eval`:
195
+
196
+ ```bash
197
+ lm_eval --model vllm \
198
+ --model_args pretrained="./Meta-Llama-3-8B-Instruct-W4A8-Channelwise-Dynamic-Per-Token",add_bos_token=true \
199
+ --tasks gsm8k \
200
+ --num_fewshot 5 \
201
+ --limit 250 \
202
+ --batch_size 'auto'
203
+ ```
204
+
205
+!!! note
206
+ Quantized models can be sensitive to the presence of the `bos` token. Make sure to include the `add_bos_token=True` argument when running evaluations.
207
+
208
+## Best Practices
209
+
210
+- Start with 512 samples for calibration data (increase if accuracy drops)
211
+- Use a sequence length of 2048 as a starting point
212
+- Employ the chat template or instruction template that the model was trained with
213
+- If you've fine-tuned a model, consider using a sample of your training data for calibration
214
+
215
+## Troubleshooting and Support
216
+
217
+If you encounter any issues or have feature requests, please open an issue on the [vllm-project/llm-compressor](https://github.com/vllm-project/llm-compressor/issues) GitHub repository.
docs/features/quantization/llm_compressor/int8_w8a8.md
renamed
+43
-45
@@ -17,15 +17,17 @@ Please visit the HF collection of [quantized INT8 checkpoints of popular LLMs re
17
To use INT8 quantization with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library:
18
19
```bash
20
-pip install llmcompressor
20
+(venv-llm-compressor) pip install llmcompressor
21
```
22
23
Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:
24
25
```bash
26
-pip install vllm "lm-eval[api]>=0.4.12"
26
+(venv-vllm) pip install vllm "lm-eval[api]>=0.4.12"
27
```
28
29
+Please use separate environments for vLLM and llm-compressor as they might not work together.
30
+
31
## Quantization Process
32
33
The quantization process involves four main steps:
@@ -57,26 +59,24 @@ When quantizing activations to INT8, you need sample data to estimate the activa
59
It's best to use calibration data that closely matches your deployment data.
60
For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
61
60
-??? code
61
-
62
- ```python
63
- from datasets import load_dataset
62
+```python
63
+from datasets import load_dataset
64
65
- NUM_CALIBRATION_SAMPLES = 512
66
- MAX_SEQUENCE_LENGTH = 2048
65
+NUM_CALIBRATION_SAMPLES = 512
66
+MAX_SEQUENCE_LENGTH = 2048
67
68
- # Load and preprocess the dataset
69
- ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
70
- ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
68
+# Load and preprocess the dataset
69
+ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
70
+ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
71
72
- def preprocess(example):
73
- return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
74
- ds = ds.map(preprocess)
72
+def preprocess(example):
73
+ return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
74
+ds = ds.map(preprocess)
75
76
- def tokenize(sample):
77
- return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
78
- ds = ds.map(tokenize, remove_columns=ds.column_names)
79
- ```
76
+def tokenize(sample):
77
+ return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
78
+ds = ds.map(tokenize, remove_columns=ds.column_names)
79
+```
80
81
</details>
82
@@ -84,33 +84,31 @@ For a general-purpose instruction-tuned model, you can use a dataset like `ultra
84
85
Now, apply the quantization algorithms:
86
87
-??? code
88
-
89
- ```python
90
- from llmcompressor import oneshot
91
- from llmcompressor.modifiers.quantization import GPTQModifier
92
- from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
93
-
94
- # Configure the quantization algorithms
95
- recipe = [
96
- SmoothQuantModifier(smoothing_strength=0.8),
97
- GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
98
- ]
99
-
100
- # Apply quantization
101
- oneshot(
102
- model=model,
103
- dataset=ds,
104
- recipe=recipe,
105
- max_seq_length=MAX_SEQUENCE_LENGTH,
106
- num_calibration_samples=NUM_CALIBRATION_SAMPLES,
107
- )
108
-
109
- # Save the compressed model: Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token
110
- SAVE_DIR = MODEL_ID.split("/")[1] + "-W8A8-Dynamic-Per-Token"
111
- model.save_pretrained(SAVE_DIR, save_compressed=True)
112
- tokenizer.save_pretrained(SAVE_DIR)
113
- ```
87
+```python
88
+from llmcompressor import oneshot
89
+from llmcompressor.modifiers.quantization import GPTQModifier
90
+from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
91
+
92
+# Configure the quantization algorithms
93
+recipe = [
94
+ SmoothQuantModifier(smoothing_strength=0.8),
95
+ GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
96
+]
97
+
98
+# Apply quantization
99
+oneshot(
100
+ model=model,
101
+ dataset=ds,
102
+ recipe=recipe,
103
+ max_seq_length=MAX_SEQUENCE_LENGTH,
104
+ num_calibration_samples=NUM_CALIBRATION_SAMPLES,
105
+)
106
+
107
+# Save the compressed model: Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token
108
+SAVE_DIR = MODEL_ID.split("/")[1] + "-W8A8-Dynamic-Per-Token"
109
+model.save_pretrained(SAVE_DIR, save_compressed=True)
110
+tokenizer.save_pretrained(SAVE_DIR)
111
+```
112
113
This process creates a W8A8 model with weights and activations quantized to 8-bit integers.
114
mkdocs.yaml
+3
@@ -110,6 +110,9 @@ plugins:
110
redirect_maps:
111
features/spec_decode/README.md: features/speculative_decoding/README.md
112
features/spec_decode/speculators.md: features/speculative_decoding/speculators.md
113
+ features/quantization/fp8.md: features/quantization/llm_compressor/fp8.md
114
+ features/quantization/int4.md: features/quantization/llm_compressor/int4.md
115
+ features/quantization/int8.md: features/quantization/llm_compressor/int8_w8a8.md
116
serving/openai_compatible_server.md: serving/online_serving/README.md
117
118
markdown_extensions: