A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 18:15 UTC
[Doc] fix the autoAWQ example (#7937)
Stas Bekman committed
Aug 28, 2024 at 05:12 UTC
98c12cffe57be141b64d47c82e65b64948446699
1 file changed
+12
-8
docs/source/quantization/auto_awq.rst
+12
-8
@@ -19,27 +19,31 @@ You can quantize your own models by installing AutoAWQ or picking one of the `40
19
20
$ pip install autoawq
21
22
-After installing AutoAWQ, you are ready to quantize a model. Here is an example of how to quantize Vicuna 7B v1.5:
22
+After installing AutoAWQ, you are ready to quantize a model. Here is an example of how to quantize `mistralai/Mistral-7B-Instruct-v0.2`:
23
24
.. code-block:: python
25
26
from awq import AutoAWQForCausalLM
27
from transformers import AutoTokenizer
28
-
29
- model_path = 'lmsys/vicuna-7b-v1.5'
30
- quant_path = 'vicuna-7b-v1.5-awq'
28
+
29
+ model_path = 'mistralai/Mistral-7B-Instruct-v0.2'
30
+ quant_path = 'mistral-instruct-v0.2-awq'
31
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
32
-
32
+
33
# Load model
34
- model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True})
34
+ model = AutoAWQForCausalLM.from_pretrained(
35
+ model_path, **{"low_cpu_mem_usage": True, "use_cache": False}
36
+ )
37
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
36
-
38
+
39
# Quantize
40
model.quantize(tokenizer, quant_config=quant_config)
39
-
41
+
42
# Save quantized model
43
model.save_quantized(quant_path)
44
tokenizer.save_pretrained(quant_path)
45
+
46
+ print(f'Model is quantized and saved at "{quant_path}"')
47
48
To run an AWQ model with vLLM, you can use `TheBloke/Llama-2-7b-Chat-AWQ <https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ>`_ with the following command:
49