@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 146 lines 4.67 KB
Rendered Raw
1 # INT8 W8A8
2
3 vLLM supports quantizing weights and activations to INT8 for memory savings and inference acceleration.
4 This quantization method is particularly useful for reducing model size while maintaining good performance.
5
6 Please visit the HF collection of [quantized INT8 checkpoints of popular LLMs ready to use with vLLM](https://huggingface.co/collections/neuralmagic/int8-llms-for-vllm-668ec32c049dca0369816415).
7
8 !!! note
9 INT8 computation is supported on NVIDIA GPUs with compute capability > 7.5 (Turing, Ampere, Ada Lovelace, Hopper).
10
11 !!! warning
12 **Blackwell GPU Limitation**: INT8 is not supported on compute capability >= 10.0 (e.g., RTX 6000 Blackwell).
13 Use [FP8 quantization](fp8.md) instead, or run on Hopper/Ada/Ampere architectures.
14
15 ## Prerequisites
16
17 To use INT8 quantization with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library:
18
19 ```bash
20 (venv-llm-compressor) pip install llmcompressor
21 ```
22
23 Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:
24
25 ```bash
26 (venv-vllm) pip install vllm "lm-eval[api]>=0.4.12"
27 ```
28
29 Please use separate environments for vLLM and llm-compressor as they might not work together.
30
31 ## Quantization Process
32
33 The quantization process involves four main steps:
34
35 1. Loading the model
36 2. Preparing calibration data
37 3. Applying quantization
38 4. Evaluating accuracy in vLLM
39
40 ### 1. Loading the Model
41
42 Load your model and tokenizer using the standard `transformers` AutoModel classes:
43
44 ```python
45 from transformers import AutoTokenizer, AutoModelForCausalLM
46
47 MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
48 model = AutoModelForCausalLM.from_pretrained(
49 MODEL_ID,
50 device_map="auto",
51 dtype="auto",
52 )
53 tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
54 ```
55
56 ### 2. Preparing Calibration Data
57
58 When quantizing activations to INT8, you need sample data to estimate the activation scales.
59 It's best to use calibration data that closely matches your deployment data.
60 For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
61
62 ```python
63 from datasets import load_dataset
64
65 NUM_CALIBRATION_SAMPLES = 512
66 MAX_SEQUENCE_LENGTH = 2048
67
68 # Load and preprocess the dataset
69 ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
70 ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
71
72 def preprocess(example):
73 return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
74 ds = ds.map(preprocess)
75
76 def tokenize(sample):
77 return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
78 ds = ds.map(tokenize, remove_columns=ds.column_names)
79 ```
80
81 ### 3. Applying Quantization
82
83 Now, apply the quantization algorithms:
84
85 ```python
86 from llmcompressor import oneshot
87 from llmcompressor.modifiers.quantization import GPTQModifier
88 from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
89
90 # Configure the quantization algorithms
91 recipe = [
92 SmoothQuantModifier(smoothing_strength=0.8),
93 GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
94 ]
95
96 # Apply quantization
97 oneshot(
98 model=model,
99 dataset=ds,
100 recipe=recipe,
101 max_seq_length=MAX_SEQUENCE_LENGTH,
102 num_calibration_samples=NUM_CALIBRATION_SAMPLES,
103 )
104
105 # Save the compressed model: Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token
106 SAVE_DIR = MODEL_ID.split("/")[1] + "-W8A8-Dynamic-Per-Token"
107 model.save_pretrained(SAVE_DIR, save_compressed=True)
108 tokenizer.save_pretrained(SAVE_DIR)
109 ```
110
111 This process creates a W8A8 model with weights and activations quantized to 8-bit integers.
112
113 ### 4. Evaluating Accuracy
114
115 After quantization, you can load and run the model in vLLM:
116
117 ```python
118 from vllm import LLM
119
120 llm = LLM("./Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token")
121 ```
122
123 To evaluate accuracy, you can use `lm_eval`:
124
125 ```bash
126 lm_eval --model vllm \
127 --model_args pretrained="./Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token",add_bos_token=true \
128 --tasks gsm8k \
129 --num_fewshot 5 \
130 --limit 250 \
131 --batch_size 'auto'
132 ```
133
134 !!! note
135 Quantized models can be sensitive to the presence of the `bos` token. Make sure to include the `add_bos_token=True` argument when running evaluations.
136
137 ## Best Practices
138
139 - Start with 512 samples for calibration data (increase if accuracy drops)
140 - Use a sequence length of 2048 as a starting point
141 - Employ the chat template or instruction template that the model was trained with
142 - If you've fine-tuned a model, consider using a sample of your training data for calibration
143
144 ## Troubleshooting and Support
145
146 If you encounter any issues or have feature requests, please open an issue on the [vllm-project/llm-compressor](https://github.com/vllm-project/llm-compressor/issues) GitHub repository.