@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 82 lines 3.21 KB
Rendered Raw
1 # AutoAWQ
2
3 > ⚠️ **Warning:**
4 The `AutoAWQ` library is deprecated. This functionality has been adopted by the vLLM project in [`llm-compressor`](https://github.com/vllm-project/llm-compressor/tree/main/examples/awq).
5 For the recommended quantization workflow, please see the AWQ examples in [`llm-compressor`](https://github.com/vllm-project/llm-compressor/tree/main/examples/awq). For more details on the deprecation, refer to the original [AutoAWQ repository](https://github.com/casper-hansen/AutoAWQ).
6
7 To create a new 4-bit quantized model, you can leverage [AutoAWQ](https://github.com/casper-hansen/AutoAWQ).
8 Quantization reduces the model's precision from BF16/FP16 to INT4 which effectively reduces the total model memory footprint.
9 The main benefits are lower latency and memory usage.
10
11 You can quantize your own models by installing AutoAWQ or picking one of the [6500+ models on Huggingface](https://huggingface.co/models?search=awq).
12
13 ```bash
14 pip install autoawq
15 ```
16
17 After installing AutoAWQ, you are ready to quantize a model. Please refer to the [AutoAWQ documentation](https://casper-hansen.github.io/AutoAWQ/examples/#basic-quantization) for further details. Here is an example of how to quantize `mistralai/Mistral-7B-Instruct-v0.2`:
18
19 ??? code
20
21 ```python
22 from awq import AutoAWQForCausalLM
23 from transformers import AutoTokenizer
24
25 model_path = "mistralai/Mistral-7B-Instruct-v0.2"
26 quant_path = "mistral-instruct-v0.2-awq"
27 quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"}
28
29 # Load model
30 model = AutoAWQForCausalLM.from_pretrained(
31 model_path,
32 low_cpu_mem_usage=True,
33 use_cache=False,
34 )
35 tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
36
37 # Quantize
38 model.quantize(tokenizer, quant_config=quant_config)
39
40 # Save quantized model
41 model.save_quantized(quant_path)
42 tokenizer.save_pretrained(quant_path)
43
44 print(f'Model is quantized and saved at "{quant_path}"')
45 ```
46
47 To run an AWQ model with vLLM, you can use [TheBloke/Llama-2-7b-Chat-AWQ](https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ) with the following command:
48
49 ```bash
50 python examples/deployment/llm_engine_example.py \
51 --model TheBloke/Llama-2-7b-Chat-AWQ \
52 --quantization auto_awq
53 ```
54
55 AWQ models are also supported directly through the LLM entrypoint:
56
57 ??? code
58
59 ```python
60 from vllm import LLM, SamplingParams
61
62 # Sample prompts.
63 prompts = [
64 "Hello, my name is",
65 "The president of the United States is",
66 "The capital of France is",
67 "The future of AI is",
68 ]
69 # Create a sampling params object.
70 sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
71
72 # Create an LLM.
73 llm = LLM(model="TheBloke/Llama-2-7b-Chat-AWQ", quantization="auto_awq")
74 # Generate texts from the prompts. The output is a list of RequestOutput objects
75 # that contain the prompt, generated text, and other information.
76 outputs = llm.generate(prompts, sampling_params)
77 # Print the outputs.
78 for output in outputs:
79 prompt = output.prompt
80 generated_text = output.outputs[0].text
81 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
82 ```