@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 98c12cffe5
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 18:15 UTC

[Doc] fix the autoAWQ example (#7937)

Stas Bekman committed Aug 28, 2024 at 05:12 UTC 98c12cffe57be141b64d47c82e65b64948446699
1 file changed +12 -8
docs/source/quantization/auto_awq.rst
+12 -8
@@ -19,27 +19,31 @@ You can quantize your own models by installing AutoAWQ or picking one of the `40
19
20 $ pip install autoawq
21
22 -After installing AutoAWQ, you are ready to quantize a model. Here is an example of how to quantize Vicuna 7B v1.5:
22 +After installing AutoAWQ, you are ready to quantize a model. Here is an example of how to quantize `mistralai/Mistral-7B-Instruct-v0.2`:
23
24 .. code-block:: python
25
26 from awq import AutoAWQForCausalLM
27 from transformers import AutoTokenizer
28 -
29 - model_path = 'lmsys/vicuna-7b-v1.5'
30 - quant_path = 'vicuna-7b-v1.5-awq'
28 +
29 + model_path = 'mistralai/Mistral-7B-Instruct-v0.2'
30 + quant_path = 'mistral-instruct-v0.2-awq'
31 quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
32 -
32 +
33 # Load model
34 - model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True})
34 + model = AutoAWQForCausalLM.from_pretrained(
35 + model_path, **{"low_cpu_mem_usage": True, "use_cache": False}
36 + )
37 tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
36 -
38 +
39 # Quantize
40 model.quantize(tokenizer, quant_config=quant_config)
39 -
41 +
42 # Save quantized model
43 model.save_quantized(quant_path)
44 tokenizer.save_pretrained(quant_path)
45 +
46 + print(f'Model is quantized and saved at "{quant_path}"')
47
48 To run an AWQ model with vLLM, you can use `TheBloke/Llama-2-7b-Chat-AWQ <https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ>`_ with the following command:
49