A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 18:15 UTC
[Doc] Add docs for llmcompressor INT8 and FP8 checkpoints (#7444)
Michael Goin committed
Aug 16, 2024 at 16:59 UTC
b3f4e179350140d415f03e25261edf4b0b152ed4
3 files changed
+252
-111
docs/source/index.rst
+1
@@ -107,6 +107,7 @@ Documentation
107
quantization/supported_hardware
108
quantization/auto_awq
109
quantization/bnb
110
+ quantization/int8
111
quantization/fp8
112
quantization/fp8_e5m2_kvcache
113
quantization/fp8_e4m3_kvcache
docs/source/quantization/fp8.rst
+106
-111
@@ -1,6 +1,6 @@
1
.. _fp8:
2
3
-FP8
3
+FP8 W8A8
4
==================
5
6
vLLM supports FP8 (8-bit floating point) weight and activation quantization using hardware acceleration on GPUs such as Nvidia H100 and AMD MI300x.
@@ -15,6 +15,11 @@ The FP8 types typically supported in hardware have two distinct representations,
15
- **E4M3**: Consists of 1 sign bit, 4 exponent bits, and 3 bits of mantissa. It can store values up to +/-448 and ``nan``.
16
- **E5M2**: Consists of 1 sign bit, 5 exponent bits, and 2 bits of mantissa. It can store values up to +/-57344, +/- ``inf``, and ``nan``. The tradeoff for the increased dynamic range is lower precision of the stored values.
17
18
+.. note::
19
+
20
+ FP8 computation is supported on NVIDIA GPUs with compute capability > 8.9 (Ada Lovelace, Hopper).
21
+ FP8 models will run on compute capability > 8.0 (Ampere) as weight-only W8A16, utilizing FP8 Marlin.
22
+
23
Quick Start with Online Dynamic Quantization
24
--------------------------------------------
25
@@ -33,106 +38,134 @@ In this mode, all Linear modules (except for the final ``lm_head``) have their w
38
39
Currently, we load the model at original precision before quantizing down to 8-bits, so you need enough memory to load the whole model.
40
36
-Offline Quantization
41
+Installation
42
+------------
43
+
44
+To produce performant FP8 quantized models with vLLM, you'll need to install the `llm-compressor <https://github.com/vllm-project/llm-compressor/>`_ library:
45
+
46
+.. code-block:: console
47
+
48
+ $ pip install llmcompressor==0.1.0
49
+
50
+Quantization Process
51
--------------------
52
39
-For offline quantization to FP8, please install the `AutoFP8 library <https://github.com/neuralmagic/autofp8>`_.
53
+The quantization process involves three main steps:
54
41
-.. code-block:: bash
55
+1. Loading the model
56
+2. Applying quantization
57
+3. Evaluating accuracy in vLLM
58
43
- git clone https://github.com/neuralmagic/AutoFP8.git
44
- pip install -e AutoFP8
59
+1. Loading the Model
60
+^^^^^^^^^^^^^^^^^^^^
61
46
-This package introduces the ``AutoFP8ForCausalLM`` and ``BaseQuantizeConfig`` objects for managing how your model will be compressed.
62
+Use ``SparseAutoModelForCausalLM``, which wraps ``AutoModelForCausalLM``, for saving and loading quantized models:
63
48
-Offline Quantization with Dynamic Activation Scaling Factors
49
-------------------------------------------------------------
64
+.. code-block:: python
65
51
-You can use AutoFP8 to produce checkpoints with their weights quantized to FP8 ahead of time and let vLLM handle calculating dynamic scales for the activations at runtime for maximum accuracy. You can enable this with the ``activation_scheme="dynamic"`` argument.
66
+ from llmcompressor.transformers import SparseAutoModelForCausalLM
67
+ from transformers import AutoTokenizer
68
53
-.. warning::
69
+ MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
70
+
71
+ model = SparseAutoModelForCausalLM.from_pretrained(
72
+ MODEL_ID, device_map="auto", torch_dtype="auto")
73
+ tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
74
+
75
+2. Applying Quantization
76
+^^^^^^^^^^^^^^^^^^^^^^^^
77
55
- Please note that although this mode doesn't give you better performance, it reduces memory footprint compared to online quantization.
78
+For FP8 quantization, we can recover accuracy with simple RTN quantization. We recommend targeting all ``Linear`` layers using the ``FP8_DYNAMIC`` scheme, which uses:
79
+
80
+- Static, per-channel quantization on the weights
81
+- Dynamic, per-token quantization on the activations
82
+
83
+Since simple RTN does not require data for weight quantization and the activations are quantized dynamically, we do not need any calibration data for this quantization flow.
84
85
.. code-block:: python
86
59
- from auto_fp8 import AutoFP8ForCausalLM, BaseQuantizeConfig
87
+ from llmcompressor.transformers import oneshot
88
+ from llmcompressor.modifiers.quantization import QuantizationModifier
89
61
- pretrained_model_dir = "meta-llama/Meta-Llama-3-8B-Instruct"
62
- quantized_model_dir = "Meta-Llama-3-8B-Instruct-FP8-Dynamic"
90
+ # Configure the simple PTQ quantization
91
+ recipe = QuantizationModifier(
92
+ targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
93
64
- # Define quantization config with static activation scales
65
- quantize_config = BaseQuantizeConfig(quant_method="fp8", activation_scheme="dynamic")
66
- # For dynamic activation scales, there is no need for calbration examples
67
- examples = []
94
+ # Apply the quantization algorithm.
95
+ oneshot(model=model, recipe=recipe)
96
69
- # Load the model, quantize, and save checkpoint
70
- model = AutoFP8ForCausalLM.from_pretrained(pretrained_model_dir, quantize_config)
71
- model.quantize(examples)
72
- model.save_quantized(quantized_model_dir)
97
+ # Save the model.
98
+ SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
99
+ model.save_pretrained(SAVE_DIR)
100
+ tokenizer.save_pretrained(SAVE_DIR)
101
+
102
+3. Evaluating Accuracy
103
+^^^^^^^^^^^^^^^^^^^^^^
104
+
105
+Install ``vllm`` and ``lm-evaluation-harness``:
106
+
107
+.. code-block:: console
108
+
109
+ $ pip install vllm lm_eval==0.4.3
110
+
111
+Load and run the model in ``vllm``:
112
+
113
+.. code-block:: python
114
+
115
+ from vllm import LLM
116
+ model = LLM("./Meta-Llama-3-8B-Instruct-FP8-Dynamic")
117
+ model.generate("Hello my name is")
118
+
119
+Evaluate accuracy with ``lm_eval`` (for example on 250 samples of ``gsm8k``):
120
+
121
+.. note::
122
+
123
+ Quantized models can be sensitive to the presence of the ``bos`` token. ``lm_eval`` does not add a ``bos`` token by default, so make sure to include the ``add_bos_token=True`` argument when running your evaluations.
124
+
125
+.. code-block:: console
126
+
127
+ $ MODEL=$PWD/Meta-Llama-3-8B-Instruct-FP8-Dynamic
128
+ $ lm_eval \
129
+ --model vllm \
130
+ --model_args pretrained=$MODEL,add_bos_token=True \
131
+ --tasks gsm8k --num_fewshot 5 --batch_size auto --limit 250
132
74
-In the output of the above script, you should be able to see the quantized Linear modules (FP8DynamicLinear) replaced in the model definition.
75
-Note that the ``lm_head`` Linear module at the end is currently skipped by default.
133
+Here's an example of the resulting scores:
134
135
.. code-block:: text
136
79
- LlamaForCausalLM(
80
- (model): LlamaModel(
81
- (embed_tokens): Embedding(128256, 4096)
82
- (layers): ModuleList(
83
- (0-31): 32 x LlamaDecoderLayer(
84
- (self_attn): LlamaSdpaAttention(
85
- (q_proj): FP8DynamicLinear()
86
- (k_proj): FP8DynamicLinear()
87
- (v_proj): FP8DynamicLinear()
88
- (o_proj): FP8DynamicLinear()
89
- (rotary_emb): LlamaRotaryEmbedding()
90
- )
91
- (mlp): LlamaMLP(
92
- (gate_proj): FP8DynamicLinear()
93
- (up_proj): FP8DynamicLinear()
94
- (down_proj): FP8DynamicLinear()
95
- (act_fn): SiLU()
96
- )
97
- (input_layernorm): LlamaRMSNorm()
98
- (post_attention_layernorm): LlamaRMSNorm()
99
- )
100
- )
101
- (norm): LlamaRMSNorm()
102
- )
103
- (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
104
- )
105
- Saving the model to Meta-Llama-3-8B-Instruct-FP8-Dynamic
106
-
107
-Your model checkpoint with quantized weights should be available at ``Meta-Llama-3-8B-Instruct-FP8/``.
108
-We can see that the weights are smaller than the original BF16 precision.
137
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
138
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
139
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.768|± |0.0268|
140
+ | | |strict-match | 5|exact_match|↑ |0.768|± |0.0268|
141
110
-.. code-block:: bash
142
+Troubleshooting and Support
143
+---------------------------
144
112
- ls -lh Meta-Llama-3-8B-Instruct-FP8-Dynamic/
113
- total 8.5G
114
- -rw-rw-r-- 1 user user 869 Jun 7 14:43 config.json
115
- -rw-rw-r-- 1 user user 194 Jun 7 14:43 generation_config.json
116
- -rw-rw-r-- 1 user user 4.7G Jun 7 14:43 model-00001-of-00002.safetensors
117
- -rw-rw-r-- 1 user user 3.9G Jun 7 14:43 model-00002-of-00002.safetensors
118
- -rw-rw-r-- 1 user user 43K Jun 7 14:43 model.safetensors.index.json
119
- -rw-rw-r-- 1 user user 296 Jun 7 14:43 special_tokens_map.json
120
- -rw-rw-r-- 1 user user 50K Jun 7 14:43 tokenizer_config.json
121
- -rw-rw-r-- 1 user user 8.7M Jun 7 14:43 tokenizer.json
145
+If you encounter any issues or have feature requests, please open an issue on the ``vllm-project/llm-compressor`` GitHub repository.
146
123
-Finally, you can load the quantized model checkpoint directly in vLLM.
147
125
-.. code-block:: python
148
+Deprecated Flow
149
+------------------
150
127
- from vllm import LLM
128
- model = LLM(model="Meta-Llama-3-8B-Instruct-FP8-Dynamic/")
129
- # INFO 06-10 21:15:41 model_runner.py:159] Loading model weights took 8.4596 GB
130
- result = model.generate("Hello, my name is")
151
+.. note::
152
+
153
+ The following information is preserved for reference and search purposes.
154
+ The quantization method described below is deprecated in favor of the ``llmcompressor`` method described above.
155
+
156
+For static per-tensor offline quantization to FP8, please install the `AutoFP8 library <https://github.com/neuralmagic/autofp8>`_.
157
+
158
+.. code-block:: bash
159
+
160
+ git clone https://github.com/neuralmagic/AutoFP8.git
161
+ pip install -e AutoFP8
162
+
163
+This package introduces the ``AutoFP8ForCausalLM`` and ``BaseQuantizeConfig`` objects for managing how your model will be compressed.
164
165
Offline Quantization with Static Activation Scaling Factors
166
-----------------------------------------------------------
167
135
-For the best inference performance, you can use AutoFP8 with calibration data to produce per-tensor static scales for both the weights and activations by enabling the ``activation_scheme="static"`` argument.
168
+You can use AutoFP8 with calibration data to produce per-tensor static scales for both the weights and activations by enabling the ``activation_scheme="static"`` argument.
169
170
.. code-block:: python
171
@@ -169,41 +202,3 @@ Finally, you can load the quantized model checkpoint directly in vLLM.
202
# INFO 06-10 21:15:41 model_runner.py:159] Loading model weights took 8.4596 GB
203
result = model.generate("Hello, my name is")
204
172
-FP8 checkpoint structure explanation
173
------------------------------------------------------------
174
-
175
-Here we detail the structure for the FP8 checkpoints.
176
-
177
-The following is necessary to be present in the model's ``config.json``:
178
-
179
-.. code-block:: text
180
-
181
- "quantization_config": {
182
- "quant_method": "fp8",
183
- "activation_scheme": "static" or "dynamic"
184
- }
185
-
186
-
187
-Each quantized layer in the state_dict will have these tensors:
188
-
189
-* If the config has ``"activation_scheme": "static"``:
190
-
191
-.. code-block:: text
192
-
193
- model.layers.0.mlp.down_proj.weight < F8_E4M3
194
- model.layers.0.mlp.down_proj.input_scale < F32
195
- model.layers.0.mlp.down_proj.weight_scale < F32
196
-
197
-* If the config has ``"activation_scheme": "dynamic"``:
198
-
199
-.. code-block:: text
200
-
201
- model.layers.0.mlp.down_proj.weight < F8_E4M3
202
- model.layers.0.mlp.down_proj.weight_scale < F32
203
-
204
-
205
-Additionally, there can be `FP8 kv-cache scaling factors <https://github.com/vllm-project/vllm/pull/4893>`_ contained within quantized checkpoints specified through the ``.kv_scale`` parameter present on the Attention Module, such as:
206
-
207
-.. code-block:: text
208
-
209
- model.layers.0.self_attn.kv_scale < F32
docs/source/quantization/int8.rst
new
+145
@@ -0,0 +1,145 @@
1
+.. _int8:
2
+
3
+INT8 W8A8
4
+==================
5
+
6
+vLLM supports quantizing weights and activations to INT8 for memory savings and inference acceleration.
7
+This quantization method is particularly useful for reducing model size while maintaining good performance.
8
+
9
+Please visit the HF collection of `quantized INT8 checkpoints of popular LLMs ready to use with vLLM <https://huggingface.co/collections/neuralmagic/int8-llms-for-vllm-668ec32c049dca0369816415>`_.
10
+
11
+.. note::
12
+
13
+ INT8 computation is supported on NVIDIA GPUs with compute capability > 7.5 (Turing, Ampere, Ada Lovelace, Hopper).
14
+
15
+Prerequisites
16
+-------------
17
+
18
+To use INT8 quantization with vLLM, you'll need to install the `llm-compressor <https://github.com/vllm-project/llm-compressor/>`_ library:
19
+
20
+.. code-block:: console
21
+
22
+ $ pip install llmcompressor==0.1.0
23
+
24
+Quantization Process
25
+--------------------
26
+
27
+The quantization process involves four main steps:
28
+
29
+1. Loading the model
30
+2. Preparing calibration data
31
+3. Applying quantization
32
+4. Evaluating accuracy in vLLM
33
+
34
+1. Loading the Model
35
+^^^^^^^^^^^^^^^^^^^^
36
+
37
+Use ``SparseAutoModelForCausalLM``, which wraps ``AutoModelForCausalLM``, for saving and loading quantized models:
38
+
39
+.. code-block:: python
40
+
41
+ from llmcompressor.transformers import SparseAutoModelForCausalLM
42
+ from transformers import AutoTokenizer
43
+
44
+ MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
45
+ model = SparseAutoModelForCausalLM.from_pretrained(
46
+ MODEL_ID, device_map="auto", torch_dtype="auto",
47
+ )
48
+ tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
49
+
50
+2. Preparing Calibration Data
51
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
52
+
53
+When quantizing activations to INT8, you need sample data to estimate the activation scales.
54
+It's best to use calibration data that closely matches your deployment data.
55
+For a general-purpose instruction-tuned model, you can use a dataset like ``ultrachat``:
56
+
57
+.. code-block:: python
58
+
59
+ from datasets import load_dataset
60
+
61
+ NUM_CALIBRATION_SAMPLES = 512
62
+ MAX_SEQUENCE_LENGTH = 2048
63
+
64
+ # Load and preprocess the dataset
65
+ ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
66
+ ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
67
+
68
+ def preprocess(example):
69
+ return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
70
+ ds = ds.map(preprocess)
71
+
72
+ def tokenize(sample):
73
+ return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
74
+ ds = ds.map(tokenize, remove_columns=ds.column_names)
75
+
76
+3. Applying Quantization
77
+^^^^^^^^^^^^^^^^^^^^^^^^
78
+
79
+Now, apply the quantization algorithms:
80
+
81
+.. code-block:: python
82
+
83
+ from llmcompressor.transformers import oneshot
84
+ from llmcompressor.modifiers.quantization import GPTQModifier
85
+ from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
86
+
87
+ # Configure the quantization algorithms
88
+ recipe = [
89
+ SmoothQuantModifier(smoothing_strength=0.8),
90
+ GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
91
+ ]
92
+
93
+ # Apply quantization
94
+ oneshot(
95
+ model=model,
96
+ dataset=ds,
97
+ recipe=recipe,
98
+ max_seq_length=MAX_SEQUENCE_LENGTH,
99
+ num_calibration_samples=NUM_CALIBRATION_SAMPLES,
100
+ )
101
+
102
+ # Save the compressed model
103
+ SAVE_DIR = MODEL_ID.split("/")[1] + "-W8A8-Dynamic-Per-Token"
104
+ model.save_pretrained(SAVE_DIR, save_compressed=True)
105
+ tokenizer.save_pretrained(SAVE_DIR)
106
+
107
+This process creates a W8A8 model with weights and activations quantized to 8-bit integers.
108
+
109
+4. Evaluating Accuracy
110
+^^^^^^^^^^^^^^^^^^^^^^
111
+
112
+After quantization, you can load and run the model in vLLM:
113
+
114
+.. code-block:: python
115
+
116
+ from vllm import LLM
117
+ model = LLM("./Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token")
118
+
119
+To evaluate accuracy, you can use ``lm_eval``:
120
+
121
+.. code-block:: console
122
+
123
+ $ lm_eval --model vllm \
124
+ --model_args pretrained="./Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token",add_bos_token=true \
125
+ --tasks gsm8k \
126
+ --num_fewshot 5 \
127
+ --limit 250 \
128
+ --batch_size 'auto'
129
+
130
+.. note::
131
+
132
+ Quantized models can be sensitive to the presence of the ``bos`` token. Make sure to include the ``add_bos_token=True`` argument when running evaluations.
133
+
134
+Best Practices
135
+--------------
136
+
137
+- Start with 512 samples for calibration data (increase if accuracy drops)
138
+- Use a sequence length of 2048 as a starting point
139
+- Employ the chat template or instruction template that the model was trained with
140
+- If you've fine-tuned a model, consider using a sample of your training data for calibration
141
+
142
+Troubleshooting and Support
143
+---------------------------
144
+
145
+If you encounter any issues or have feature requests, please open an issue on the ``vllm-project/llm-compressor`` GitHub repository.
\ No newline at end of file