@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 2041c0e360
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 16:00 UTC

[Doc] Quark quantization documentation (#15861)

Signed-off-by: chaow <chaow@amd.com>

chaow-amd committed Apr 1, 2025 at 23:32 UTC 2041c0e360bf69955d0752e604289edd5547f122
2 files changed +218
docs/source/features/quantization/index.md
+1
@@ -16,5 +16,6 @@ gptqmodel
16 int4
17 int8
18 fp8
19 +quark
20 quantized_kvcache
21 :::
docs/source/features/quantization/quark.md new
+217
@@ -0,0 +1,217 @@
1 +(quark)=
2 +
3 +# AMD QUARK
4 +
5 +Quantization can effectively reduce memory and bandwidth usage, accelerate computation and improve
6 +throughput while with minimal accuracy loss. vLLM can leverage [Quark](https://quark.docs.amd.com/latest/),
7 +the flexible and powerful quantization toolkit, to produce performant quantized models to run on AMD GPUs. Quark has specialized support for quantizing large language models with weight,
8 +activation and kv-cache quantization and cutting-edge quantization algorithms like
9 +AWQ, GPTQ, Rotation and SmoothQuant.
10 +
11 +## Quark Installation
12 +
13 +Before quantizing models, you need to install Quark. The latest release of Quark can be installed with pip:
14 +
15 +```console
16 +pip install amd-quark
17 +```
18 +
19 +You can refer to [Quark installation guide](https://quark.docs.amd.com/latest/install.html)
20 +for more installation details.
21 +
22 +## Quantization Process
23 +
24 +After installing Quark, we will use an example to illustrate how to use Quark.
25 +The Quark quantization process can be listed for 5 steps as below:
26 +
27 +1. Load the model
28 +2. Prepare the calibration dataloader
29 +3. Set the quantization configuration
30 +4. Quantize the model and export
31 +5. Evaluation in vLLM
32 +
33 +### 1. Load the Model
34 +
35 +Quark uses [Transformers](https://huggingface.co/docs/transformers/en/index)
36 +to fetch model and tokenizer.
37 +
38 +```python
39 +from transformers import AutoTokenizer, AutoModelForCausalLM
40 +
41 +MODEL_ID = "meta-llama/Llama-2-70b-chat-hf"
42 +MAX_SEQ_LEN = 512
43 +
44 +model = AutoModelForCausalLM.from_pretrained(
45 + MODEL_ID, device_map="auto", torch_dtype="auto",
46 +)
47 +model.eval()
48 +
49 +tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, model_max_length=MAX_SEQ_LEN)
50 +tokenizer.pad_token = tokenizer.eos_token
51 +```
52 +
53 +### 2. Prepare the Calibration Dataloader
54 +
55 +Quark uses the [PyTorch Dataloader](https://pytorch.org/tutorials/beginner/basics/data_tutorial.html)
56 +to load calibration data. For more details about how to use calibration datasets efficiently, please refer
57 +to [Adding Calibration Datasets](https://quark.docs.amd.com/latest/pytorch/calibration_datasets.html).
58 +
59 +```python
60 +from datasets import load_dataset
61 +from torch.utils.data import DataLoader
62 +
63 +BATCH_SIZE = 1
64 +NUM_CALIBRATION_DATA = 512
65 +
66 +# Load the dataset and get calibration data.
67 +dataset = load_dataset("mit-han-lab/pile-val-backup", split="validation")
68 +text_data = dataset["text"][:NUM_CALIBRATION_DATA]
69 +
70 +tokenized_outputs = tokenizer(text_data, return_tensors="pt",
71 + padding=True, truncation=True, max_length=MAX_SEQ_LEN)
72 +calib_dataloader = DataLoader(tokenized_outputs['input_ids'],
73 + batch_size=BATCH_SIZE, drop_last=True)
74 +```
75 +
76 +### 3. Set the Quantization Configuration
77 +
78 +We need to set the quantization configuration, you can check
79 +[quark config guide](https://quark.docs.amd.com/latest/pytorch/user_guide_config_description.html)
80 +for further details. Here we use FP8 per-tensor quantization on weight, activation,
81 +kv-cache and the quantization algorithm is AutoSmoothQuant.
82 +
83 +:::{note}
84 +Note the quantization algorithm needs a JSON config file and the config file is located in
85 +[Quark Pytorch examples](https://quark.docs.amd.com/latest/pytorch/pytorch_examples.html),
86 +under the directory `examples/torch/language_modeling/llm_ptq/models`. For example,
87 +AutoSmoothQuant config file for Llama is
88 +`examples/torch/language_modeling/llm_ptq/models/llama/autosmoothquant_config.json`.
89 +:::
90 +
91 +```python
92 +from quark.torch.quantization import (Config, QuantizationConfig,
93 + FP8E4M3PerTensorSpec,
94 + load_quant_algo_config_from_file)
95 +
96 +# Define fp8/per-tensor/static spec.
97 +FP8_PER_TENSOR_SPEC = FP8E4M3PerTensorSpec(observer_method="min_max",
98 + is_dynamic=False).to_quantization_spec()
99 +
100 +# Define global quantization config, input tensors and weight apply FP8_PER_TENSOR_SPEC.
101 +global_quant_config = QuantizationConfig(input_tensors=FP8_PER_TENSOR_SPEC,
102 + weight=FP8_PER_TENSOR_SPEC)
103 +
104 +# Define quantization config for kv-cache layers, output tensors apply FP8_PER_TENSOR_SPEC.
105 +KV_CACHE_SPEC = FP8_PER_TENSOR_SPEC
106 +kv_cache_layer_names_for_llama = ["*k_proj", "*v_proj"]
107 +kv_cache_quant_config = {name :
108 + QuantizationConfig(input_tensors=global_quant_config.input_tensors,
109 + weight=global_quant_config.weight,
110 + output_tensors=KV_CACHE_SPEC)
111 + for name in kv_cache_layer_names_for_llama}
112 +layer_quant_config = kv_cache_quant_config.copy()
113 +
114 +# Define algorithm config by config file.
115 +LLAMA_AUTOSMOOTHQUANT_CONFIG_FILE =
116 + 'examples/torch/language_modeling/llm_ptq/models/llama/autosmoothquant_config.json'
117 +algo_config = load_quant_algo_config_from_file(LLAMA_AUTOSMOOTHQUANT_CONFIG_FILE)
118 +
119 +EXCLUDE_LAYERS = ["lm_head"]
120 +quant_config = Config(
121 + global_quant_config=global_quant_config,
122 + layer_quant_config=layer_quant_config,
123 + kv_cache_quant_config=kv_cache_quant_config,
124 + exclude=EXCLUDE_LAYERS,
125 + algo_config=algo_config)
126 +```
127 +
128 +### 4. Quantize the Model and Export
129 +
130 +Then we can apply the quantization. After quantizing, we need to freeze the
131 +quantized model first before exporting. Note that we need to export model with format of
132 +HuggingFace `safetensors`, you can refer to
133 +[HuggingFace format exporting](https://quark.docs.amd.com/latest/pytorch/export/quark_export_hf.html)
134 +for more exporting format details.
135 +
136 +```python
137 +import torch
138 +from quark.torch import ModelQuantizer, ModelExporter
139 +from quark.torch.export import ExporterConfig, JsonExporterConfig
140 +
141 +# Apply quantization.
142 +quantizer = ModelQuantizer(quant_config)
143 +quant_model = quantizer.quantize_model(model, calib_dataloader)
144 +
145 +# Freeze quantized model to export.
146 +freezed_model = quantizer.freeze(model)
147 +
148 +# Define export config.
149 +LLAMA_KV_CACHE_GROUP = ["*k_proj", "*v_proj"]
150 +export_config = ExporterConfig(json_export_config=JsonExporterConfig())
151 +export_config.json_export_config.kv_cache_group = LLAMA_KV_CACHE_GROUP
152 +
153 +EXPORT_DIR = MODEL_ID.split("/")[1] + "-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant"
154 +exporter = ModelExporter(config=export_config, export_dir=EXPORT_DIR)
155 +with torch.no_grad():
156 + exporter.export_safetensors_model(freezed_model,
157 + quant_config=quant_config, tokenizer=tokenizer)
158 +```
159 +
160 +### 5. Evaluation in vLLM
161 +
162 +Now, you can load and run the Quark quantized model directly through the LLM entrypoint:
163 +
164 +```python
165 +from vllm import LLM, SamplingParams
166 +
167 +# Sample prompts.
168 +prompts = [
169 + "Hello, my name is",
170 + "The president of the United States is",
171 + "The capital of France is",
172 + "The future of AI is",
173 +]
174 +# Create a sampling params object.
175 +sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
176 +
177 +# Create an LLM.
178 +llm = LLM(model="Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant",
179 + kv_cache_dtype='fp8',quantization='quark')
180 +# Generate texts from the prompts. The output is a list of RequestOutput objects
181 +# that contain the prompt, generated text, and other information.
182 +outputs = llm.generate(prompts, sampling_params)
183 +# Print the outputs.
184 +print("\nGenerated Outputs:\n" + "-" * 60)
185 +for output in outputs:
186 + prompt = output.prompt
187 + generated_text = output.outputs[0].text
188 + print(f"Prompt: {prompt!r}")
189 + print(f"Output: {generated_text!r}")
190 + print("-" * 60)
191 +```
192 +
193 +Or, you can use `lm_eval` to evaluate accuracy:
194 +
195 +```console
196 +$ lm_eval --model vllm \
197 + --model_args pretrained=Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant,kv_cache_dtype='fp8',quantization='quark' \
198 + --tasks gsm8k
199 +```
200 +
201 +## Quark Quantization Script
202 +In addition to the example of Python API above, Quark also offers a
203 +[quantization script](https://quark.docs.amd.com/latest/pytorch/example_quark_torch_llm_ptq.html)
204 +to quantize large language models more conveniently. It supports quantizing models with variety
205 +of different quantization schemes and optimization algorithms. It can export the quantized model
206 +and run evaluation tasks on the fly. With the script, the example above can be:
207 +
208 +```console
209 +python3 quantize_quark.py --model_dir meta-llama/Llama-2-70b-chat-hf \
210 + --output_dir /path/to/output \
211 + --quant_scheme w_fp8_a_fp8 \
212 + --kv_cache_dtype fp8 \
213 + --quant_algo autosmoothquant \
214 + --num_calib_data 512 \
215 + --model_export hf_format \
216 + --tasks gsm8k
217 +```