@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 99 lines 3.93 KB
Rendered Raw
1 # GPTQModel
2
3 To create a new 4-bit or 8-bit GPTQ quantized model, you can leverage [GPTQModel](https://github.com/ModelCloud/GPTQModel) from ModelCloud.AI.
4
5 Quantization reduces the model's precision from BF16/FP16 (16-bits) to INT4 (4-bits) or INT8 (8-bits) which significantly reduces the
6 total model memory footprint while at-the-same-time increasing inference performance.
7
8 Compatible GPTQModel quantized models can leverage the `Marlin` and `Machete` vLLM custom kernels to maximize batching
9 transactions-per-second `tps` and token-latency performance for both Ampere (A100+) and Hopper (H100+) Nvidia GPUs.
10 These two kernels are highly optimized by vLLM and NeuralMagic (now part of Redhat) to allow world-class inference performance of quantized GPTQ
11 models.
12
13 GPTQModel is one of the few quantization toolkits in the world that allows `Dynamic` per-module quantization where different layers and/or modules within a llm model can be further optimized with custom quantization parameters. `Dynamic` quantization
14 is fully integrated into vLLM and backed up by support from the ModelCloud.AI team. Please refer to [GPTQModel readme](https://github.com/ModelCloud/GPTQModel?tab=readme-ov-file#dynamic-quantization-per-module-quantizeconfig-override)
15 for more details on this and other advanced features.
16
17 ## Installation
18
19 You can quantize your own models by installing [GPTQModel](https://github.com/ModelCloud/GPTQModel) or picking one of the [5000+ models on Huggingface](https://huggingface.co/models?search=gptq).
20
21 ```bash
22 pip install -U gptqmodel --no-build-isolation -v
23 ```
24
25 ## Quantizing a model
26
27 After installing GPTQModel, you are ready to quantize a model. Please refer to the [GPTQModel readme](https://github.com/ModelCloud/GPTQModel/?tab=readme-ov-file#quantization) for further details.
28
29 Here is an example of how to quantize `meta-llama/Llama-3.2-1B-Instruct`:
30
31 ??? code
32
33 ```python
34 from datasets import load_dataset
35 from gptqmodel import GPTQModel, QuantizeConfig
36
37 model_id = "meta-llama/Llama-3.2-1B-Instruct"
38 quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"
39
40 calibration_dataset = load_dataset(
41 "allenai/c4",
42 data_files="en/c4-train.00001-of-01024.json.gz",
43 split="train",
44 ).select(range(1024))["text"]
45
46 quant_config = QuantizeConfig(bits=4, group_size=128)
47
48 model = GPTQModel.load(model_id, quant_config)
49
50 # increase `batch_size` to match gpu/vram specs to speed up quantization
51 model.quantize(calibration_dataset, batch_size=2)
52
53 model.save(quant_path)
54 ```
55
56 ## Running a quantized model with vLLM
57
58 To run a GPTQModel quantized model with vLLM, you can use [DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2](https://huggingface.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2) with the following command:
59
60 ```bash
61 python examples/deployment/llm_engine_example.py \
62 --model ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2
63 ```
64
65 ## Using GPTQModel with vLLM's Python API
66
67 GPTQModel quantized models are also supported directly through the LLM entrypoint:
68
69 ??? code
70
71 ```python
72 from vllm import LLM, SamplingParams
73
74 # Sample prompts.
75 prompts = [
76 "Hello, my name is",
77 "The president of the United States is",
78 "The capital of France is",
79 "The future of AI is",
80 ]
81
82 # Create a sampling params object.
83 sampling_params = SamplingParams(temperature=0.6, top_p=0.9)
84
85 # Create an LLM.
86 llm = LLM(model="ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2")
87
88 # Generate texts from the prompts. The output is a list of RequestOutput objects
89 # that contain the prompt, generated text, and other information.
90 outputs = llm.generate(prompts, sampling_params)
91
92 # Print the outputs.
93 print("-"*50)
94 for output in outputs:
95 prompt = output.prompt
96 generated_text = output.outputs[0].text
97 print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}")
98 print("-"*50)
99 ```