@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 8516999495
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 06:45 UTC

Add Quantization and AutoAWQ to docs (#1235)

Casper committed Nov 5, 2023 at 06:43 UTC 8516999495114926c9838c2d6e0feb580d4d983f
2 files changed +75
docs/source/index.rst
+6
@@ -73,3 +73,9 @@ Documentation
73
74 models/supported_models
75 models/adding_model
76 +
77 +.. toctree::
78 + :maxdepth: 1
79 + :caption: Quantization
80 +
81 + quantization/auto_awq
\ No newline at end of file
docs/source/quantization/auto_awq.rst new
+69
@@ -0,0 +1,69 @@
1 +.. _auto_awq:
2 +
3 +AutoAWQ
4 +==================
5 +
6 +To create a new 4-bit quantized model, you can leverage `AutoAWQ <https://github.com/casper-hansen/AutoAWQ>`_.
7 +Quantizing reduces the model's precision from FP16 to INT4 which effectively reduces the file size by ~70%.
8 +The main benefits are lower latency and memory usage.
9 +
10 +You can quantize your own models by installing AutoAWQ or picking one of the `400+ models on Huggingface <https://huggingface.co/models?sort=trending&search=awq>`_.
11 +
12 +.. code-block:: console
13 +
14 + $ pip install autoawq
15 +
16 +After installing AutoAWQ, you are ready to quantize a model. Here is an example of how to quantize Vicuna 7B v1.5:
17 +
18 +.. code-block:: python
19 +
20 + from awq import AutoAWQForCausalLM
21 + from transformers import AutoTokenizer
22 +
23 + model_path = 'lmsys/vicuna-7b-v1.5'
24 + quant_path = 'vicuna-7b-v1.5-awq'
25 + quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
26 +
27 + # Load model
28 + model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True})
29 + tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
30 +
31 + # Quantize
32 + model.quantize(tokenizer, quant_config=quant_config)
33 +
34 + # Save quantized model
35 + model.save_quantized(quant_path)
36 + tokenizer.save_pretrained(quant_path)
37 +
38 +To run an AWQ model with vLLM, you can use `TheBloke/Llama-2-7b-Chat-AWQ <https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ>`_ with the following command:
39 +
40 +.. code-block:: console
41 +
42 + $ python examples/llm_engine_example.py --model TheBloke/Llama-2-7b-Chat-AWQ --quantization awq
43 +
44 +AWQ models are also supported directly through the LLM entrypoint:
45 +
46 +.. code-block:: python
47 +
48 + from vllm import LLM, SamplingParams
49 +
50 + # Sample prompts.
51 + prompts = [
52 + "Hello, my name is",
53 + "The president of the United States is",
54 + "The capital of France is",
55 + "The future of AI is",
56 + ]
57 + # Create a sampling params object.
58 + sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
59 +
60 + # Create an LLM.
61 + llm = LLM(model="TheBloke/Llama-2-7b-Chat-AWQ", quantization="AWQ")
62 + # Generate texts from the prompts. The output is a list of RequestOutput objects
63 + # that contain the prompt, generated text, and other information.
64 + outputs = llm.generate(prompts, sampling_params)
65 + # Print the outputs.
66 + for output in outputs:
67 + prompt = output.prompt
68 + generated_text = output.outputs[0].text
69 + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")