A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 06:45 UTC
Add Quantization and AutoAWQ to docs (#1235)
Casper committed
Nov 5, 2023 at 06:43 UTC
8516999495114926c9838c2d6e0feb580d4d983f
2 files changed
+75
docs/source/index.rst
+6
@@ -73,3 +73,9 @@ Documentation
73
74
models/supported_models
75
models/adding_model
76
+
77
+.. toctree::
78
+ :maxdepth: 1
79
+ :caption: Quantization
80
+
81
+ quantization/auto_awq
\ No newline at end of file
docs/source/quantization/auto_awq.rst
new
+69
@@ -0,0 +1,69 @@
1
+.. _auto_awq:
2
+
3
+AutoAWQ
4
+==================
5
+
6
+To create a new 4-bit quantized model, you can leverage `AutoAWQ <https://github.com/casper-hansen/AutoAWQ>`_.
7
+Quantizing reduces the model's precision from FP16 to INT4 which effectively reduces the file size by ~70%.
8
+The main benefits are lower latency and memory usage.
9
+
10
+You can quantize your own models by installing AutoAWQ or picking one of the `400+ models on Huggingface <https://huggingface.co/models?sort=trending&search=awq>`_.
11
+
12
+.. code-block:: console
13
+
14
+ $ pip install autoawq
15
+
16
+After installing AutoAWQ, you are ready to quantize a model. Here is an example of how to quantize Vicuna 7B v1.5:
17
+
18
+.. code-block:: python
19
+
20
+ from awq import AutoAWQForCausalLM
21
+ from transformers import AutoTokenizer
22
+
23
+ model_path = 'lmsys/vicuna-7b-v1.5'
24
+ quant_path = 'vicuna-7b-v1.5-awq'
25
+ quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
26
+
27
+ # Load model
28
+ model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True})
29
+ tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
30
+
31
+ # Quantize
32
+ model.quantize(tokenizer, quant_config=quant_config)
33
+
34
+ # Save quantized model
35
+ model.save_quantized(quant_path)
36
+ tokenizer.save_pretrained(quant_path)
37
+
38
+To run an AWQ model with vLLM, you can use `TheBloke/Llama-2-7b-Chat-AWQ <https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ>`_ with the following command:
39
+
40
+.. code-block:: console
41
+
42
+ $ python examples/llm_engine_example.py --model TheBloke/Llama-2-7b-Chat-AWQ --quantization awq
43
+
44
+AWQ models are also supported directly through the LLM entrypoint:
45
+
46
+.. code-block:: python
47
+
48
+ from vllm import LLM, SamplingParams
49
+
50
+ # Sample prompts.
51
+ prompts = [
52
+ "Hello, my name is",
53
+ "The president of the United States is",
54
+ "The capital of France is",
55
+ "The future of AI is",
56
+ ]
57
+ # Create a sampling params object.
58
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
59
+
60
+ # Create an LLM.
61
+ llm = LLM(model="TheBloke/Llama-2-7b-Chat-AWQ", quantization="AWQ")
62
+ # Generate texts from the prompts. The output is a list of RequestOutput objects
63
+ # that contain the prompt, generated text, and other information.
64
+ outputs = llm.generate(prompts, sampling_params)
65
+ # Print the outputs.
66
+ for output in outputs:
67
+ prompt = output.prompt
68
+ generated_text = output.outputs[0].text
69
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")