A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 13:30 UTC
[Doc] Add headings to improve gptqmodel.md (#17164)
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
Michael Yao committed
Apr 25, 2025 at 16:13 UTC
ef19e67d2c8d275dd0a40492830aa3e26dd5e7c0
1 file changed
+11
docs/source/features/quantization/gptqmodel.md
+11
@@ -16,12 +16,16 @@ GPTQModel is one of the few quantization toolkits in the world that allows `Dyna
16
is fully integrated into vLLM and backed up by support from the ModelCloud.AI team. Please refer to [GPTQModel readme](https://github.com/ModelCloud/GPTQModel?tab=readme-ov-file#dynamic-quantization-per-module-quantizeconfig-override)
17
for more details on this and other advanced features.
18
19
+## Installation
20
+
21
You can quantize your own models by installing [GPTQModel](https://github.com/ModelCloud/GPTQModel) or picking one of the [5000+ models on Huggingface](https://huggingface.co/models?sort=trending&search=gptq).
22
23
```console
24
pip install -U gptqmodel --no-build-isolation -v
25
```
26
27
+## Quantizing a model
28
+
29
After installing GPTQModel, you are ready to quantize a model. Please refer to the [GPTQModel readme](https://github.com/ModelCloud/GPTQModel/?tab=readme-ov-file#quantization) for further details.
30
31
Here is an example of how to quantize `meta-llama/Llama-3.2-1B-Instruct`:
@@ -49,12 +53,16 @@ model.quantize(calibration_dataset, batch_size=2)
53
model.save(quant_path)
54
```
55
56
+## Running a quantized model with vLLM
57
+
58
To run an GPTQModel quantized model with vLLM, you can use [DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2](https://huggingface.co/ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2) with the following command:
59
60
```console
61
python examples/offline_inference/llm_engine_example.py --model DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2
62
```
63
64
+## Using GPTQModel with vLLM's Python API
65
+
66
GPTQModel quantized models are also supported directly through the LLM entrypoint:
67
68
```python
@@ -67,14 +75,17 @@ prompts = [
75
"The capital of France is",
76
"The future of AI is",
77
]
78
+
79
# Create a sampling params object.
80
sampling_params = SamplingParams(temperature=0.6, top_p=0.9)
81
82
# Create an LLM.
83
llm = LLM(model="DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2")
84
+
85
# Generate texts from the prompts. The output is a list of RequestOutput objects
86
# that contain the prompt, generated text, and other information.
87
outputs = llm.generate(prompts, sampling_params)
88
+
89
# Print the outputs.
90
for output in outputs:
91
prompt = output.prompt