A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 10, 05:15 UTC
Consolidate Intel Quantization Toolkit Integration in vLLM (#31716)
Signed-off-by: yiliu30 <yi4.liu@intel.com>
Yi Liu committed
Jan 14, 2026 at 15:11 UTC
50632adc583fe1af8e2ee112b48c61c14153574f
10 files changed
+530
-659
docs/features/quantization/README.md
+14
-15
@@ -5,12 +5,11 @@ Quantization trades off model precision for smaller memory footprint, allowing l
5
Contents:
6
7
- [AutoAWQ](auto_awq.md)
8
-- [AutoRound](auto_round.md)
8
- [BitsAndBytes](bnb.md)
9
- [BitBLAS](bitblas.md)
10
- [GGUF](gguf.md)
11
- [GPTQModel](gptqmodel.md)
13
-- [INC](inc.md)
12
+- [Intel Neural Compressor](inc.md)
13
- [INT4 W4A16](int4.md)
14
- [INT8 W8A8](int8.md)
15
- [FP8 W8A8](fp8.md)
@@ -43,23 +42,23 @@ th:not(:first-child) {
42
}
43
</style>
44
46
-| Implementation | Volta | Turing | Ampere | Ada | Hopper | AMD GPU | Intel GPU | Intel Gaudi | x86 CPU |
47
-|-----------------------|---------|----------|----------|-------|----------|-----------|-------------|-------------|-----------|
48
-| AWQ | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ❌ | ✅︎ |
49
-| GPTQ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ❌ | ✅︎ |
50
-| Marlin (GPTQ/AWQ/FP8) | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ |
51
-| INT8 (W8A8) | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ✅︎ |
52
-| FP8 (W8A8) | ❌ | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
53
-| BitBLAS | ✅︎ | ✅ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ |
54
-| BitBLAS (GPTQ) | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ |
55
-| bitsandbytes | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ |
56
-| DeepSpeedFP | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ | ❌ |
57
-| GGUF | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
58
-| INC (W8A8) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅︎ | ❌ |
45
+| Implementation | Volta | Turing | Ampere | Ada | Hopper | AMD GPU | Intel GPU | x86 CPU |
46
+|-----------------------|---------|----------|----------|-------|----------|-----------|-------------|-----------|
47
+| AWQ | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ |
48
+| GPTQ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ✅︎ | ✅︎ |
49
+| Marlin (GPTQ/AWQ/FP8) | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
50
+| INT8 (W8A8) | ❌ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ✅︎ |
51
+| FP8 (W8A8) | ❌ | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ |
52
+| BitBLAS | ✅︎ | ✅ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
53
+| BitBLAS (GPTQ) | ❌ | ❌ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
54
+| bitsandbytes | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
55
+| DeepSpeedFP | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ | ❌ |
56
+| GGUF | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ✅︎ | ❌ | ❌ |
57
58
- Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.
59
- ✅︎ indicates that the quantization method is supported on the specified hardware.
60
- ❌ indicates that the quantization method is not supported on the specified hardware.
61
+- All Intel Gaudi quantization support has been migrated to [vLLM-Gaudi](https://github.com/vllm-project/vllm-gaudi).
62
63
!!! note
64
For information on quantization support on Google TPU, please refer to the [TPU-Inference Recommended Models and Features](https://docs.vllm.ai/projects/tpu/en/latest/recommended_models_features/) documentation.
docs/features/quantization/auto_round.md
deleted
-103
@@ -1,103 +0,0 @@
1
-# AutoRound
2
-
3
-[AutoRound](https://github.com/intel/auto-round) is Intel’s advanced quantization algorithm designed to produce highly efficient **INT2, INT3, INT4, and INT8**
4
-quantized large language models—striking an optimal balance between accuracy and deployment performance.
5
-
6
-AutoRound applies weight-only quantization to transformer-based models, enabling significant memory savings and faster
7
-inference while maintaining near-original accuracy. It supports a wide range of hardware platforms, including **CPUs,
8
-Intel GPUs, HPUs, and CUDA-enabled devices**.
9
-
10
-Please refer to the [AutoRound guide](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md) for more details.
11
-
12
-Key Features:
13
-
14
-✅ **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** are supported
15
-
16
-✅ **10+ vision-language models (VLMs)** are supported
17
-
18
-✅ **Per-layer mixed-bit quantization** for fine-grained control
19
-
20
-✅ **RTN (Round-To-Nearest) mode** for quick quantization with slight accuracy loss
21
-
22
-✅ **Multiple quantization recipes**: best, base, and light
23
-
24
-✅ Advanced utilities such as immediate packing and support for **10+ backends**
25
-
26
-## Installation
27
-
28
-```bash
29
-uv pip install auto-round
30
-```
31
-
32
-## Quantizing a model
33
-
34
-For VLMs, please change to `auto-round-mllm` in CLI usage and `AutoRoundMLLM` in API usage.
35
-
36
-### CLI usage
37
-
38
-```bash
39
-auto-round \
40
- --model Qwen/Qwen3-0.6B \
41
- --bits 4 \
42
- --group_size 128 \
43
- --format "auto_round" \
44
- --output_dir ./tmp_autoround
45
-```
46
-
47
-```bash
48
-auto-round \
49
- --model Qwen/Qwen3-0.6B \
50
- --format "gguf:q4_k_m" \
51
- --output_dir ./tmp_autoround
52
-```
53
-
54
-### API usage
55
-
56
-```python
57
-from transformers import AutoModelForCausalLM, AutoTokenizer
58
-from auto_round import AutoRound
59
-
60
-model_name = "Qwen/Qwen3-0.6B"
61
-model = AutoModelForCausalLM.from_pretrained(model_name, dtype="auto")
62
-tokenizer = AutoTokenizer.from_pretrained(model_name)
63
-
64
-bits, group_size, sym = 4, 128, True
65
-autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, sym=sym)
66
-
67
-# the best accuracy, 4-5X slower, low_gpu_mem_usage could save ~20G but ~30% slower
68
-# autoround = AutoRound(model, tokenizer, nsamples=512, iters=1000, low_gpu_mem_usage=True, bits=bits, group_size=group_size, sym=sym)
69
-
70
-# 2-3X speedup, slight accuracy drop at W4G128
71
-# autoround = AutoRound(model, tokenizer, nsamples=128, iters=50, lr=5e-3, bits=bits, group_size=group_size, sym=sym )
72
-
73
-output_dir = "./tmp_autoround"
74
-# format= 'auto_round'(default), 'auto_gptq', 'auto_awq'
75
-autoround.quantize_and_save(output_dir, format="auto_round")
76
-```
77
-
78
-## Running a quantized model with vLLM
79
-
80
-Here is some example code to run auto-round format in vLLM:
81
-
82
-```python
83
-from vllm import LLM, SamplingParams
84
-
85
-prompts = [
86
- "Hello, my name is",
87
-]
88
-sampling_params = SamplingParams(temperature=0.6, top_p=0.95)
89
-model_name = "Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound"
90
-llm = LLM(model=model_name)
91
-
92
-outputs = llm.generate(prompts, sampling_params)
93
-
94
-for output in outputs:
95
- prompt = output.prompt
96
- generated_text = output.outputs[0].text
97
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
98
-```
99
-
100
-## Acknowledgement
101
-
102
-Special thanks to open-source low precision libraries such as AutoGPTQ, AutoAWQ, GPTQModel, Triton, Marlin, and
103
-ExLLaMAV2 for providing low-precision CUDA kernels, which are leveraged in AutoRound.
docs/features/quantization/inc.md
+72
-33
@@ -1,50 +1,89 @@
1
-# FP8 INC
1
+# Intel Quantization Support
2
3
-vLLM supports FP8 (8-bit floating point) weight and activation quantization using Intel® Neural Compressor (INC) on Intel® Gaudi® 2 and Intel® Gaudi® 3 AI accelerators.
4
-Currently, quantization is validated only in Llama models.
3
+[AutoRound](https://github.com/intel/auto-round) is Intel’s advanced quantization algorithm designed for large language models(LLMs). It produces highly efficient **INT2, INT3, INT4, INT8, MXFP8, MXFP4, NVFP4**, and **GGUF** quantized models, balancing accuracy and inference performance. AutoRound is also part of the [Intel® Neural Compressor](https://github.com/intel/neural-compressor). For a deeper introduction, see the [AutoRound step-by-step guide](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md).
4
6
-Intel Gaudi supports quantization of various modules and functions, including, but not limited to `Linear`, `KVCache`, `Matmul` and `Softmax`. For more information, please refer to:
7
-[Supported Modules\\Supported Functions\\Custom Patched Modules](https://docs.habana.ai/en/latest/PyTorch/Inference_on_PyTorch/Quantization/Inference_Using_FP8.html#supported-modules).
5
+## Key Features
6
9
-!!! note
10
- Measurement files are required to run quantized models with vLLM on Gaudi accelerators. The FP8 model calibration procedure is described in the [vLLM HPU extension](https://github.com/HabanaAI/vllm-hpu-extension/tree/main/calibration/README.md) package.
7
+✅ Superior Accuracy Delivers strong performance even at 2–3 bits [example models](https://huggingface.co/collections/OPEA/2-3-bits)
8
12
-!!! note
13
- `QUANT_CONFIG` is an environment variable that points to the measurement or quantization [JSON config file](https://docs.habana.ai/en/latest/PyTorch/Inference_on_PyTorch/Quantization/Inference_Using_FP8.html#supported-json-config-file-options).
14
- The measurement configuration file is used during the calibration procedure to collect measurements for a given model. The quantization configuration is used during inference.
9
+✅ Fast Mixed `Bits`/`Dtypes` Scheme Generation Automatically configure in minutes
10
+
11
+✅ Support for exporting **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** formats
12
+
13
+✅ **10+ vision-language models (VLMs)** are supported
14
+
15
+✅ **Per-layer mixed-bit quantization** for fine-grained control
16
+
17
+✅ **RTN (Round-To-Nearest) mode** for quick quantization with slight accuracy loss
18
+
19
+✅ **Multiple quantization recipes**: best, base, and light
20
16
-## Run Online Inference Using FP8
21
+✅ Advanced utilities such as immediate packing and support for **10+ backends**
22
18
-Once you've completed the model calibration process and collected the measurements, you can run FP8 inference with vLLM using the following command:
23
+## Supported Recipes on Intel Platforms
24
+
25
+On Intel platforms, AutoRound recipes are being enabled progressively by format and hardware. Currently, vLLM supports:
26
+
27
+- **`W4A16`**: weight-only, 4-bit weights with 16-bit activations
28
+- **`W8A16`**: weight-only, 8-bit weights with 16-bit activations
29
+
30
+Additional recipes and formats will be supported in future releases.
31
+
32
+## Quantizing a Model
33
+
34
+### Installation
35
36
```bash
21
-export QUANT_CONFIG=/path/to/quant/config/inc/meta-llama-3.1-405b-instruct/maxabs_measure_g3.json
22
-vllm serve meta-llama/Llama-3.1-405B-Instruct --quantization inc --kv-cache-dtype fp8_inc --tensor-parallel-size 8
37
+uv pip install auto-round
38
```
39
25
-!!! tip
26
- When using FP8 models, you may experience timeouts caused by the long compilation time of FP8 operations. To mitigate this problem, you can use the below environment variables:
27
- `VLLM_ENGINE_ITERATION_TIMEOUT_S` - to adjust the vLLM server timeout. You can set the value in seconds, e.g., 600 equals 10 minutes.
28
- `VLLM_RPC_TIMEOUT` - to adjust the RPC protocol timeout used by the OpenAI-compatible API. This value is in microseconds, e.g., 600000 equals 10 minutes.
29
-
30
-## Run Offline Inference Using FP8
40
+### Quantize with CLI
41
32
-To run offline inference (after completing the model calibration process):
42
+```bash
43
+auto-round \
44
+ --model Qwen/Qwen3-0.6B \
45
+ --scheme W4A16 \
46
+ --format auto_round \
47
+ --output_dir ./tmp_autoround
48
+```
49
34
-* Set the "QUANT_CONFIG" environment variable to point to a JSON configuration file with QUANTIZE mode.
35
-* Pass `quantization=inc` and `kv_cache_dtype=fp8_inc` as parameters to the `LLM` object.
36
-* Call shutdown method of the model_executor at the end of the run.
50
+### Quantize with Python API
51
52
```python
39
-from vllm import LLM
40
-llm = LLM("llama3.1/Meta-Llama-3.1-8B-Instruct", quantization="inc", kv_cache_dtype="fp8_inc")
41
-...
42
-# Call llm.generate on the required prompts and sampling params.
43
-...
44
-llm.llm_engine.model_executor.shutdown()
53
+from transformers import AutoModelForCausalLM, AutoTokenizer
54
+from auto_round import AutoRound
55
+
56
+model_name = "Qwen/Qwen3-0.6B"
57
+autoround = AutoRound(model_name, scheme="W4A16")
58
+
59
+# the best accuracy, 4-5X slower, low_gpu_mem_usage could save ~20G but ~30% slower
60
+# autoround = AutoRound(model, tokenizer, nsamples=512, iters=1000, low_gpu_mem_usage=True, bits=bits, group_size=group_size, sym=sym)
61
+
62
+# 2-3X speedup, slight accuracy drop at W4G128
63
+# autoround = AutoRound(model, tokenizer, nsamples=128, iters=50, lr=5e-3, bits=bits, group_size=group_size, sym=sym )
64
+
65
+output_dir = "./tmp_autoround"
66
+# format= 'auto_round'(default), 'auto_gptq', 'auto_awq'
67
+autoround.quantize_and_save(output_dir, format="auto_round")
68
```
69
47
-## Device for the Model's Weights Uploading
70
+## Deploying AutoRound Quantized Models in vLLM
71
49
-The unquantized weights are first loaded onto the CPU, then quantized and transferred to the target device (HPU) for model execution.
50
-This reduces the device memory footprint of model weights, as only quantized weights are stored in the device memory.
72
+```bash
73
+vllm serve Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound \
74
+ --gpu-memory-utilization 0.8 \
75
+ --max-model-len 4096
76
+```
77
+
78
+!!! note
79
+ To deploy `wNa16` models on Intel GPU/CPU, please add `--enforce-eager` for now.
80
+
81
+## Evaluating the Quantized Model with vLLM
82
+
83
+```bash
84
+lm_eval --model vllm \
85
+ --model_args pretrained="Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound,max_model_len=8192,max_num_batched_tokens=32768,max_num_seqs=128,gpu_memory_utilization=0.8,dtype=bfloat16,max_gen_toks=2048,enforce_eager=True" \
86
+ --tasks gsm8k \
87
+ --num_fewshot 5 \
88
+ --batch_size 128
89
+```
tests/quantization/test_auto_round.py
+1
-3
@@ -26,9 +26,7 @@ MODELS = [
26
)
27
@pytest.mark.parametrize("model", MODELS)
28
def test_auto_round(vllm_runner, model):
29
- with vllm_runner(
30
- model, enforce_eager=True, allow_deprecated_quantization=True
31
- ) as llm:
29
+ with vllm_runner(model, enforce_eager=True) as llm:
30
output = llm.generate_greedy(["The capital of France is"], max_tokens=8)
31
assert output
32
print(f"{output[0][1]}")
vllm/config/model.py
+1
@@ -884,6 +884,7 @@ class ModelConfig:
884
"gptq_bitblas",
885
"awq_marlin",
886
"ipex",
887
+ "inc",
888
"moe_wna16",
889
"modelopt",
890
"modelopt_fp4",
vllm/engine/arg_utils.py
-5
@@ -223,10 +223,6 @@ def get_type_hints(type_hint: TypeHint) -> set[TypeHint]:
223
return type_hints
224
225
226
-def is_online_quantization(quantization: Any) -> bool:
227
- return quantization in ["inc"]
228
-
229
-
226
NEEDS_HELP = (
227
any("--help" in arg for arg in sys.argv) # vllm SUBCOMMAND --help
228
or (argv0 := sys.argv[0]).endswith("mkdocs") # mkdocs SUBCOMMAND
@@ -1304,7 +1300,6 @@ class EngineArgs:
1300
load_format=self.load_format,
1301
download_dir=self.download_dir,
1302
safetensors_load_strategy=self.safetensors_load_strategy,
1307
- device="cpu" if is_online_quantization(self.quantization) else None,
1303
model_loader_extra_config=self.model_loader_extra_config,
1304
ignore_patterns=self.ignore_patterns,
1305
use_tqdm_on_load=self.use_tqdm_on_load,
vllm/model_executor/layers/quantization/__init__.py
+1
-4
@@ -33,7 +33,6 @@ QuantizationMethods = Literal[
33
"quark",
34
"moe_wna16",
35
"torchao",
36
- "auto-round",
36
"rtn",
37
"inc",
38
"mxfp4",
@@ -54,7 +53,6 @@ DEPRECATED_QUANTIZATION_METHODS = [
53
"hqq",
54
"experts_int8",
55
"ipex",
57
- "auto-round",
56
"rtn",
57
"petit_nvfp4",
58
]
@@ -120,7 +118,6 @@ def get_quantization_config(quantization: str) -> type[QuantizationConfig]:
118
# lazy import to avoid triggering `torch.compile` too early
119
from vllm.model_executor.layers.quantization.quark.quark import QuarkConfig
120
123
- from .auto_round import AutoRoundConfig
121
from .awq import AWQConfig
122
from .awq_marlin import AWQMarlinConfig
123
from .bitblas import BitBLASConfig
@@ -174,8 +171,8 @@ def get_quantization_config(quantization: str) -> type[QuantizationConfig]:
171
"quark": QuarkConfig,
172
"moe_wna16": MoeWNA16Config,
173
"torchao": TorchAOConfig,
177
- "auto-round": AutoRoundConfig,
174
"rtn": RTNConfig,
175
+ "auto-round": INCConfig,
176
"inc": INCConfig,
177
"mxfp4": Mxfp4Config,
178
"petit_nvfp4": PetitNvFp4Config,
vllm/model_executor/layers/quantization/auto_round.py
deleted
-454
@@ -1,454 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-from fractions import Fraction
5
-from typing import TYPE_CHECKING, Any
6
-
7
-import regex as re
8
-import torch
9
-
10
-from vllm.logger import init_logger
11
-from vllm.model_executor.layers.linear import LinearBase, UnquantizedLinearMethod
12
-from vllm.model_executor.layers.quantization import (
13
- QuantizationConfig,
14
- QuantizationMethods,
15
-)
16
-from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead
17
-from vllm.platforms import current_platform
18
-from vllm.scalar_type import scalar_types
19
-
20
-if TYPE_CHECKING:
21
- from vllm.model_executor.models.utils import WeightsMapper
22
-
23
-logger = init_logger(__name__)
24
-
25
-
26
-class AutoRoundConfig(QuantizationConfig):
27
- """Config class for AutoRound.
28
- Reference: https://arxiv.org/pdf/2309.05516
29
- """
30
-
31
- SUPPORTED_BITS = {2, 3, 4, 8}
32
- SUPPORTED_DTYPES = {"int"}
33
- SUPPORTED_FORMATS = {"auto_round:auto_gptq", "auto_round:auto_awq"}
34
- SUPPORTED_BACKENDS = {
35
- "auto",
36
- "gptq",
37
- "gptq:marlin",
38
- "awq",
39
- "awq:marlin",
40
- "marlin",
41
- "ipex",
42
- }
43
-
44
- def __init__(
45
- self,
46
- weight_bits: int,
47
- group_size: int,
48
- sym: bool = True,
49
- packing_format: str = "auto_round:auto_gptq",
50
- block_name_to_quantize: str | list[str] | None = None,
51
- extra_config: dict[str, Any] | None = None,
52
- data_type: str = "int",
53
- backend: str = "auto",
54
- ) -> None:
55
- super().__init__()
56
- if weight_bits not in self.SUPPORTED_BITS:
57
- raise ValueError(
58
- f"Unsupported weight_bits: {weight_bits}, "
59
- f"currently only support {self.SUPPORTED_BITS}."
60
- )
61
- if data_type not in self.SUPPORTED_DTYPES:
62
- raise ValueError(
63
- f"Unsupported data_type: {data_type}, "
64
- f"currently only support {self.SUPPORTED_DTYPES}."
65
- )
66
- if packing_format not in self.SUPPORTED_FORMATS:
67
- raise ValueError(
68
- f"Unsupported packing_format: {packing_format}, "
69
- f"currently only support {self.SUPPORTED_FORMATS}."
70
- )
71
- if backend not in self.SUPPORTED_BACKENDS:
72
- raise ValueError(
73
- f"Unsupported backend: {backend}, "
74
- f"currently only support {self.SUPPORTED_BACKENDS}."
75
- )
76
-
77
- self.weight_bits = weight_bits
78
- self.group_size = group_size
79
- self.sym = sym
80
- self.packing_format = packing_format
81
- self.block_name_to_quantize = (
82
- block_name_to_quantize.split(",")
83
- if isinstance(block_name_to_quantize, str)
84
- else block_name_to_quantize
85
- )
86
- self.extra_config = extra_config
87
- self.data_type = data_type
88
- self.backend = backend
89
- self.pack_factor = Fraction(32, weight_bits)
90
-
91
- def __repr__(self) -> str:
92
- return (
93
- f"AutoRoundConfig(weight_bits={self.weight_bits}, "
94
- f"group_size={self.group_size}, sym={self.sym})"
95
- )
96
-
97
- @classmethod
98
- def get_name(cls) -> QuantizationMethods:
99
- return "auto-round"
100
-
101
- @classmethod
102
- def get_supported_act_dtypes(cls) -> list[torch.dtype]:
103
- return [torch.half, torch.bfloat16]
104
-
105
- @classmethod
106
- def get_min_capability(cls) -> int:
107
- return 60
108
-
109
- @classmethod
110
- def get_config_filenames(cls) -> list[str]:
111
- return ["quantization_config.json"]
112
-
113
- @classmethod
114
- def from_config(cls, config: dict[str, Any]) -> "AutoRoundConfig":
115
- return cls(
116
- weight_bits=cls.get_from_keys(config, ["bits"]),
117
- group_size=cls.get_from_keys(config, ["group_size"]),
118
- sym=cls.get_from_keys(config, ["sym"]),
119
- packing_format=cls.get_from_keys_or(
120
- config, ["packing_format"], "auto_round:auto_gptq"
121
- ),
122
- block_name_to_quantize=cls.get_from_keys_or(
123
- config, ["block_name_to_quantize", "to_quant_block_names"], None
124
- ),
125
- extra_config=cls.get_from_keys_or(config, ["extra_config"], None),
126
- data_type=cls.get_from_keys_or(config, ["data_type"], "int"),
127
- backend=cls.get_from_keys_or(config, ["backend", "vllm_backend"], "auto"),
128
- )
129
-
130
- def get_layer_config(self, layer, layer_name: str):
131
- def get_config(name: str, quantized: bool = True):
132
- if not self.extra_config:
133
- return (
134
- self.weight_bits if quantized else 16,
135
- self.group_size if quantized else -1,
136
- self.sym if quantized else True,
137
- )
138
-
139
- # exact match first
140
- if name in self.extra_config:
141
- cfg = self.extra_config[name]
142
- return (
143
- cfg.get("bits", self.weight_bits if quantized else 16),
144
- cfg.get("group_size", self.group_size if quantized else -1),
145
- cfg.get("sym", self.sym if quantized else True),
146
- )
147
-
148
- REGEX_SPECIAL_CHARS = set(r"*+?^$()[]{}|\\")
149
- for pattern, cfg in self.extra_config.items():
150
- if not isinstance(pattern, str) or not any(
151
- c in REGEX_SPECIAL_CHARS for c in pattern
152
- ):
153
- continue
154
-
155
- try:
156
- if re.search(re.compile(pattern), name) is not None:
157
- return (
158
- cfg.get("bits", self.weight_bits if quantized else 16),
159
- cfg.get("group_size", self.group_size if quantized else -1),
160
- cfg.get("sym", self.sym if quantized else True),
161
- )
162
- except re.error:
163
- # Invalid regex, ignore.
164
- continue
165
-
166
- return (
167
- self.weight_bits if quantized else 16,
168
- self.group_size if quantized else -1,
169
- self.sym if quantized else True,
170
- )
171
-
172
- # 1. Exact match from config
173
- if self.extra_config and layer_name in self.extra_config:
174
- return get_config(layer_name)
175
-
176
- # 2. Determine whether layer should be quantized
177
- quantized = not isinstance(layer, ParallelLMHead)
178
- if self.block_name_to_quantize:
179
- quantized = any(
180
- layer_name.startswith(name) for name in self.block_name_to_quantize
181
- )
182
-
183
- # 3. Handle fused MoE
184
- if self.extra_config and "fusedmoe" in layer.__class__.__name__.lower():
185
- moe_configs = [
186
- get_config(name, quantized)
187
- for name in self.extra_config
188
- if name.startswith(layer_name)
189
- ]
190
- if moe_configs:
191
- if len(set(moe_configs)) == 1:
192
- return moe_configs[0]
193
- raise ValueError(
194
- f"Fused MoE layer '{layer_name}' requires "
195
- f"consistent quant config for all sub-layers"
196
- )
197
-
198
- # 4. Handle fused QKV or other patterns
199
- if self.extra_config:
200
- for fusion_key, sub_keys in self.packed_modules_mapping.items():
201
- if fusion_key in layer_name and layer_name.count(fusion_key) == 1:
202
- sub_names = [
203
- layer_name.replace(fusion_key, sub_key) for sub_key in sub_keys
204
- ]
205
- sub_configs = [get_config(name, quantized) for name in sub_names]
206
- if len(set(sub_configs)) == 1:
207
- return sub_configs[0]
208
- raise ValueError(
209
- f"Fused module '{layer_name}' requires "
210
- f"consistent quant config for {sub_names}"
211
- )
212
-
213
- # 5. Fallback or try a regular expression match
214
- return get_config(layer_name, quantized)
215
-
216
- def check_quantized(self, weight_bits: int) -> bool:
217
- return weight_bits < 16
218
-
219
- def apply_vllm_mapper(self, hf_to_vllm_mapper: "WeightsMapper"):
220
- if self.block_name_to_quantize is not None:
221
- self.block_name_to_quantize = hf_to_vllm_mapper.apply_list(
222
- self.block_name_to_quantize
223
- )
224
- if self.extra_config is not None:
225
- self.extra_config = hf_to_vllm_mapper.apply_dict(self.extra_config)
226
-
227
- def apply_awq_quant_layer(self, layer, prefix: str, backend: str = "auto"):
228
- from vllm.model_executor.layers.fused_moe import FusedMoE
229
- from vllm.model_executor.layers.quantization.utils.marlin_utils import (
230
- check_marlin_supported,
231
- check_moe_marlin_supports_layer,
232
- )
233
-
234
- weight_bits, group_size, sym = self.get_layer_config(layer, prefix)
235
- if not self.check_quantized(weight_bits):
236
- if isinstance(layer, (LinearBase, ParallelLMHead)):
237
- return UnquantizedLinearMethod()
238
- else:
239
- return None
240
-
241
- logger.debug(
242
- "[%s] Type: %s, Bits: %s, Group Size: %s, Sym: %s",
243
- prefix,
244
- layer.__class__.__name__,
245
- weight_bits,
246
- group_size,
247
- sym,
248
- )
249
- if backend == "auto" or "marlin" in backend:
250
- AWQ_TYPE_MAP = {
251
- 4: scalar_types.uint4,
252
- 8: scalar_types.uint8,
253
- }
254
- use_marlin = (weight_bits in AWQ_TYPE_MAP) and check_marlin_supported(
255
- AWQ_TYPE_MAP[weight_bits], group_size, not sym
256
- )
257
-
258
- if isinstance(layer, FusedMoE):
259
- use_marlin = use_marlin and check_moe_marlin_supports_layer(
260
- layer, group_size
261
- )
262
-
263
- else:
264
- use_marlin = False
265
- if use_marlin:
266
- from vllm.model_executor.layers.quantization.awq_marlin import (
267
- AWQMarlinConfig,
268
- AWQMarlinLinearMethod,
269
- AWQMarlinMoEMethod,
270
- )
271
-
272
- quant_args_marlin = AWQMarlinConfig(
273
- weight_bits=weight_bits,
274
- group_size=group_size,
275
- zero_point=not sym,
276
- lm_head_quantized=False,
277
- full_config={},
278
- modules_to_not_convert=[],
279
- )
280
- else:
281
- from vllm.model_executor.layers.quantization.awq import (
282
- AWQConfig,
283
- AWQLinearMethod,
284
- )
285
-
286
- quant_args = AWQConfig(
287
- weight_bits=weight_bits,
288
- group_size=group_size,
289
- zero_point=not sym,
290
- )
291
-
292
- if isinstance(layer, FusedMoE):
293
- if use_marlin:
294
- return AWQMarlinMoEMethod(quant_args_marlin, layer.moe)
295
- from vllm.model_executor.layers.quantization.moe_wna16 import MoeWNA16Config
296
-
297
- config = {
298
- "quant_method": "awq",
299
- "bits": weight_bits,
300
- "group_size": group_size,
301
- "zero_point": not sym,
302
- "lm_head": False,
303
- }
304
- return MoeWNA16Config.from_config(config).get_quant_method(layer, prefix)
305
-
306
- if isinstance(layer, (LinearBase, ParallelLMHead)):
307
- if use_marlin:
308
- return AWQMarlinLinearMethod(quant_args_marlin)
309
- else:
310
- return AWQLinearMethod(quant_args)
311
- return None
312
-
313
- def apply_gptq_quant_layer(self, layer, prefix: str, backend: str = "auto"):
314
- from vllm.model_executor.layers.fused_moe import FusedMoE
315
- from vllm.model_executor.layers.quantization.utils.marlin_utils import (
316
- check_marlin_supported,
317
- check_moe_marlin_supports_layer,
318
- )
319
-
320
- weight_bits, group_size, sym = self.get_layer_config(layer, prefix)
321
- if not self.check_quantized(weight_bits):
322
- if isinstance(layer, (LinearBase, ParallelLMHead)):
323
- return UnquantizedLinearMethod()
324
- else:
325
- return None
326
-
327
- logger.debug(
328
- "[%s] Type: %s, Bits: %s, Group Size: %s, Sym: %s",
329
- prefix,
330
- layer.__class__.__name__,
331
- weight_bits,
332
- group_size,
333
- sym,
334
- )
335
- if backend == "auto" or "marlin" in backend:
336
- GPTQ_TYPE_MAP = {
337
- (4, True): scalar_types.uint4b8,
338
- (8, True): scalar_types.uint8b128,
339
- }
340
- use_marlin = (weight_bits, sym) in GPTQ_TYPE_MAP and check_marlin_supported(
341
- GPTQ_TYPE_MAP[(weight_bits, sym)], group_size, has_zp=not sym
342
- )
343
- if isinstance(layer, FusedMoE):
344
- use_marlin = use_marlin and check_moe_marlin_supports_layer(
345
- layer, group_size
346
- )
347
- else:
348
- use_marlin = False
349
- if use_marlin:
350
- from vllm.model_executor.layers.quantization.gptq_marlin import (
351
- GPTQMarlinConfig,
352
- GPTQMarlinLinearMethod,
353
- GPTQMarlinMoEMethod,
354
- )
355
-
356
- quant_args_marlin = GPTQMarlinConfig(
357
- weight_bits=weight_bits,
358
- group_size=group_size,
359
- is_sym=sym,
360
- lm_head_quantized=False,
361
- desc_act=False,
362
- dynamic={},
363
- full_config={},
364
- )
365
- else:
366
- from vllm.model_executor.layers.quantization.gptq import (
367
- GPTQConfig,
368
- GPTQLinearMethod,
369
- )
370
-
371
- quant_args = GPTQConfig(
372
- weight_bits=weight_bits,
373
- group_size=group_size,
374
- lm_head_quantized=False,
375
- desc_act=False,
376
- dynamic={},
377
- )
378
-
379
- if isinstance(layer, FusedMoE):
380
- if use_marlin:
381
- return GPTQMarlinMoEMethod(quant_args_marlin, layer.moe_config)
382
- else:
383
- from vllm.model_executor.layers.quantization.moe_wna16 import (
384
- MoeWNA16Config,
385
- )
386
-
387
- config = {
388
- "quant_method": "gptq",
389
- "bits": weight_bits,
390
- "group_size": group_size,
391
- "sym": sym,
392
- "lm_head": False,
393
- }
394
- return MoeWNA16Config.from_config(config).get_quant_method(
395
- layer, prefix
396
- )
397
-
398
- if isinstance(layer, (LinearBase, ParallelLMHead)):
399
- if use_marlin:
400
- return GPTQMarlinLinearMethod(quant_args_marlin)
401
- else:
402
- return GPTQLinearMethod(quant_args)
403
-
404
- return None
405
-
406
- def apply_ipex_quant_layer(self, layer, prefix: str):
407
- weight_bits, group_size, sym = self.get_layer_config(layer, prefix)
408
- if not self.check_quantized(weight_bits):
409
- if isinstance(layer, (LinearBase, ParallelLMHead)):
410
- return UnquantizedLinearMethod()
411
- else:
412
- return None
413
- from vllm.model_executor.layers.quantization.ipex_quant import (
414
- IPEXAWQLinearMethod,
415
- IPEXConfig,
416
- IPEXGPTQLinearMethod,
417
- )
418
-
419
- if isinstance(layer, (LinearBase, ParallelLMHead)):
420
- if "awq" in self.packing_format:
421
- config = IPEXConfig(
422
- method="awq", weight_bits=weight_bits, group_size=group_size
423
- )
424
- return IPEXAWQLinearMethod(config)
425
- elif "gptq" in self.packing_format:
426
- config = IPEXConfig(
427
- method="gptq", weight_bits=weight_bits, group_size=group_size
428
- )
429
- return IPEXGPTQLinearMethod(config)
430
- else:
431
- raise ValueError(
432
- f"ipex backend only supports awq "
433
- f"and gtpq format,but got {self.packing_format}"
434
- )
435
- else:
436
- return None
437
-
438
- def get_quant_method(self, layer: torch.nn.Module, prefix: str):
439
- if prefix and self.extra_config:
440
- for layer_name in self.extra_config:
441
- if (
442
- layer_name == prefix or layer_name == f"model.{prefix}"
443
- ) and self.extra_config[layer_name].get("bits", 16) >= 16:
444
- return UnquantizedLinearMethod()
445
- if (
446
- current_platform.is_cpu()
447
- or current_platform.is_xpu()
448
- or self.backend == "ipex"
449
- ):
450
- return self.apply_ipex_quant_layer(layer, prefix)
451
- if "gptq" in self.packing_format or "gptq" in self.backend:
452
- return self.apply_gptq_quant_layer(layer, prefix)
453
- if "awq" in self.packing_format or "awq" in self.backend:
454
- return self.apply_awq_quant_layer(layer, prefix)
vllm/model_executor/layers/quantization/inc.py
+440
-41
@@ -1,39 +1,98 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-#
4
-# Intel Gaudi supports quantization of various modules and functions,
5
-# including, but not limited to `Linear`, `KVCache`, `Matmul` and `Softmax`.
6
-# During model loading,
7
-# INC will patch layers with quantization/dequantization operators.
8
-# Meanwhile, INC will convert original weight to target datatype
9
-# and loading to target device.
10
-# static scaling should be provided through Quant_CONFIG:
11
-# `QUANT_CONFIG` is an environment variable,
12
-# that points to the measurement or quantization JSON config file.
13
-# The measurement configuration file is used during the calibration procedure,
14
-# to collect measurements for a given model.
15
-# The quantization configuration is used during inference.
16
-# For more information, please refer to:
17
-# https://docs.habana.ai/en/v1.21.1/PyTorch/vLLM_Inference/vLLM_FP8_Inference.html
18
-
19
-from typing import Any, Optional
3
4
+from fractions import Fraction
5
+from typing import TYPE_CHECKING, Any, Optional
6
+
7
+import regex as re
8
import torch
9
23
-from vllm.model_executor.layers.fused_moe.layer import (
24
- FusedMoE,
25
- UnquantizedFusedMoEMethod,
26
-)
10
+from vllm.logger import init_logger
11
from vllm.model_executor.layers.linear import LinearBase, UnquantizedLinearMethod
28
-from vllm.model_executor.layers.quantization import QuantizationMethods
29
-from vllm.model_executor.layers.quantization.base_config import (
12
+from vllm.model_executor.layers.quantization import (
13
QuantizationConfig,
31
- QuantizeMethodBase,
14
+ QuantizationMethods,
15
)
16
+from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead
17
+from vllm.platforms import current_platform
18
+from vllm.scalar_type import scalar_types
19
+
20
+if TYPE_CHECKING:
21
+ from vllm.model_executor.models.utils import WeightsMapper
22
+
23
+logger = init_logger(__name__)
24
25
26
class INCConfig(QuantizationConfig):
36
- """Config class for FP8 using Intel Neural Compressor."""
27
+ """Config class for Intel Neural Compressor (INC).
28
+ Repo: https://github.com/intel/neural-compressor
29
+ """
30
+
31
+ SUPPORTED_BITS = {2, 3, 4, 8}
32
+ SUPPORTED_DTYPES = {"int"}
33
+ SUPPORTED_FORMATS = {"auto_round:auto_gptq", "auto_round:auto_awq"}
34
+ SUPPORTED_BACKENDS = {
35
+ "auto",
36
+ "gptq",
37
+ "gptq:marlin",
38
+ "awq",
39
+ "awq:marlin",
40
+ "marlin",
41
+ "ipex",
42
+ }
43
+
44
+ def __init__(
45
+ self,
46
+ weight_bits: int,
47
+ group_size: int,
48
+ sym: bool = True,
49
+ packing_format: str = "auto_round:auto_gptq",
50
+ block_name_to_quantize: str | list[str] | None = None,
51
+ extra_config: dict[str, Any] | None = None,
52
+ data_type: str = "int",
53
+ backend: str = "auto",
54
+ ) -> None:
55
+ super().__init__()
56
+ if weight_bits not in self.SUPPORTED_BITS:
57
+ raise ValueError(
58
+ f"Unsupported weight_bits: {weight_bits}, "
59
+ f"currently only support {self.SUPPORTED_BITS}."
60
+ )
61
+ if data_type not in self.SUPPORTED_DTYPES:
62
+ raise ValueError(
63
+ f"Unsupported data_type: {data_type},"
64
+ f" currently only support {self.SUPPORTED_DTYPES}."
65
+ )
66
+ if packing_format not in self.SUPPORTED_FORMATS:
67
+ raise ValueError(
68
+ f"Unsupported packing_format: {packing_format}, "
69
+ f"currently only support {self.SUPPORTED_FORMATS}."
70
+ )
71
+ if backend not in self.SUPPORTED_BACKENDS:
72
+ raise ValueError(
73
+ f"Unsupported backend: {backend}, "
74
+ f"currently only support {self.SUPPORTED_BACKENDS}."
75
+ )
76
+
77
+ self.weight_bits = weight_bits
78
+ self.group_size = group_size
79
+ self.sym = sym
80
+ self.packing_format = packing_format
81
+ self.block_name_to_quantize = (
82
+ block_name_to_quantize.split(",")
83
+ if isinstance(block_name_to_quantize, str)
84
+ else block_name_to_quantize
85
+ )
86
+ self.extra_config = extra_config
87
+ self.data_type = data_type
88
+ self.backend = backend
89
+ self.pack_factor = Fraction(32, weight_bits)
90
+
91
+ def __repr__(self) -> str:
92
+ return (
93
+ f"INCConfig(weight_bits={self.weight_bits}, "
94
+ f"group_size={self.group_size}, sym={self.sym})"
95
+ )
96
97
@classmethod
98
def get_name(cls) -> QuantizationMethods:
@@ -41,25 +100,365 @@ class INCConfig(QuantizationConfig):
100
101
@classmethod
102
def get_supported_act_dtypes(cls) -> list[torch.dtype]:
44
- return [torch.bfloat16]
103
+ return [torch.half, torch.bfloat16]
104
+
105
+ @classmethod
106
+ def get_min_capability(cls) -> int:
107
+ return 60
108
+
109
+ @classmethod
110
+ def get_config_filenames(cls) -> list[str]:
111
+ return ["quantization_config.json"]
112
113
@classmethod
114
def from_config(cls, config: dict[str, Any]) -> "INCConfig":
48
- raise AssertionError
49
-
50
- def get_quant_method(
51
- self, layer: torch.nn.Module, prefix: str
52
- ) -> Optional["QuantizeMethodBase"]:
53
- if isinstance(layer, LinearBase):
54
- return UnquantizedLinearMethod()
55
- elif isinstance(layer, FusedMoE):
56
- return UnquantizedFusedMoEMethod(layer.moe_config)
115
+ return cls(
116
+ weight_bits=cls.get_from_keys(config, ["bits"]),
117
+ group_size=cls.get_from_keys(config, ["group_size"]),
118
+ sym=cls.get_from_keys(config, ["sym"]),
119
+ packing_format=cls.get_from_keys_or(
120
+ config, ["packing_format"], "auto_round:auto_gptq"
121
+ ),
122
+ block_name_to_quantize=cls.get_from_keys_or(
123
+ config, ["block_name_to_quantize", "to_quant_block_names"], None
124
+ ),
125
+ extra_config=cls.get_from_keys_or(config, ["extra_config"], None),
126
+ data_type=cls.get_from_keys_or(config, ["data_type"], "int"),
127
+ backend=cls.get_from_keys_or(config, ["backend", "vllm_backend"], "auto"),
128
+ )
129
+
130
+ def get_layer_config(self, layer, layer_name: str):
131
+ def get_config(name: str, quantized: bool = True):
132
+ if not self.extra_config:
133
+ return (
134
+ self.weight_bits if quantized else 16,
135
+ self.group_size if quantized else -1,
136
+ self.sym if quantized else True,
137
+ )
138
+
139
+ # exact match first
140
+ if name in self.extra_config:
141
+ cfg = self.extra_config[name]
142
+ return (
143
+ cfg.get("bits", self.weight_bits if quantized else 16),
144
+ cfg.get("group_size", self.group_size if quantized else -1),
145
+ cfg.get("sym", self.sym if quantized else True),
146
+ )
147
+
148
+ REGEX_SPECIAL_CHARS = set(r"*+?^$()[]{}|\\")
149
+ for pattern, cfg in self.extra_config.items():
150
+ if not isinstance(pattern, str) or not any(
151
+ c in REGEX_SPECIAL_CHARS for c in pattern
152
+ ):
153
+ continue
154
+
155
+ try:
156
+ if re.search(re.compile(pattern), name) is not None:
157
+ return (
158
+ cfg.get("bits", self.weight_bits if quantized else 16),
159
+ cfg.get("group_size", self.group_size if quantized else -1),
160
+ cfg.get("sym", self.sym if quantized else True),
161
+ )
162
+ except re.error:
163
+ # Invalid regex, ignore.
164
+ continue
165
+
166
+ return (
167
+ self.weight_bits if quantized else 16,
168
+ self.group_size if quantized else -1,
169
+ self.sym if quantized else True,
170
+ )
171
+
172
+ # 1. Exact match from config
173
+ if self.extra_config and layer_name in self.extra_config:
174
+ return get_config(layer_name)
175
+
176
+ # 2. Determine whether layer should be quantized
177
+ quantized = not isinstance(layer, ParallelLMHead)
178
+ if self.block_name_to_quantize:
179
+ quantized = any(
180
+ layer_name.startswith(name) for name in self.block_name_to_quantize
181
+ )
182
+
183
+ # 3. Handle fused MoE
184
+ if self.extra_config and "fusedmoe" in layer.__class__.__name__.lower():
185
+ moe_configs = [
186
+ get_config(name, quantized)
187
+ for name in self.extra_config
188
+ if name.startswith(layer_name)
189
+ ]
190
+ if moe_configs:
191
+ if len(set(moe_configs)) == 1:
192
+ return moe_configs[0]
193
+ raise ValueError(
194
+ f"Fused MoE layer '{layer_name}' requires "
195
+ f"consistent quant config for all sub-layers"
196
+ )
197
+
198
+ # 4. Handle fused QKV or other patterns
199
+ if self.extra_config:
200
+ for fusion_key, sub_keys in self.packed_modules_mapping.items():
201
+ if fusion_key in layer_name and layer_name.count(fusion_key) == 1:
202
+ sub_names = [
203
+ layer_name.replace(fusion_key, sub_key) for sub_key in sub_keys
204
+ ]
205
+ sub_configs = [get_config(name, quantized) for name in sub_names]
206
+ if len(set(sub_configs)) == 1:
207
+ return sub_configs[0]
208
+ raise ValueError(
209
+ f"Fused module '{layer_name}' requires "
210
+ f"consistent quant config for {sub_names}"
211
+ )
212
+
213
+ # 5. Fallback or try a regular expression match
214
+ return get_config(layer_name, quantized)
215
+
216
+ def check_quantized(self, weight_bits: int) -> bool:
217
+ return weight_bits < 16
218
+
219
+ def apply_vllm_mapper(self, hf_to_vllm_mapper: "WeightsMapper"):
220
+ if self.block_name_to_quantize is not None:
221
+ self.block_name_to_quantize = hf_to_vllm_mapper.apply_list(
222
+ self.block_name_to_quantize
223
+ )
224
+ if self.extra_config is not None:
225
+ self.extra_config = hf_to_vllm_mapper.apply_dict(self.extra_config)
226
+
227
+ def apply_awq_quant_layer(self, layer, prefix: str, backend: str = "auto"):
228
+ from vllm.model_executor.layers.fused_moe import FusedMoE
229
+ from vllm.model_executor.layers.quantization.utils.marlin_utils import (
230
+ check_marlin_supported,
231
+ check_moe_marlin_supports_layer,
232
+ )
233
+
234
+ weight_bits, group_size, sym = self.get_layer_config(layer, prefix)
235
+ if not self.check_quantized(weight_bits):
236
+ if isinstance(layer, (LinearBase, ParallelLMHead)):
237
+ return UnquantizedLinearMethod()
238
+ else:
239
+ return None
240
+
241
+ logger.debug(
242
+ "[%s] Type: %s, Bits: %s, Group Size: %s, Sym: %s",
243
+ prefix,
244
+ layer.__class__.__name__,
245
+ weight_bits,
246
+ group_size,
247
+ sym,
248
+ )
249
+ if backend == "auto" or "marlin" in backend:
250
+ AWQ_TYPE_MAP = {
251
+ 4: scalar_types.uint4,
252
+ 8: scalar_types.uint8,
253
+ }
254
+ use_marlin = (weight_bits in AWQ_TYPE_MAP) and check_marlin_supported(
255
+ AWQ_TYPE_MAP[weight_bits], group_size, not sym
256
+ )
257
+
258
+ if isinstance(layer, FusedMoE):
259
+ use_marlin = use_marlin and check_moe_marlin_supports_layer(
260
+ layer, group_size
261
+ )
262
+
263
+ else:
264
+ use_marlin = False
265
+ if use_marlin:
266
+ from vllm.model_executor.layers.quantization.awq_marlin import (
267
+ AWQMarlinConfig,
268
+ AWQMarlinLinearMethod,
269
+ AWQMarlinMoEMethod,
270
+ )
271
+
272
+ quant_args_marlin = AWQMarlinConfig(
273
+ weight_bits=weight_bits,
274
+ group_size=group_size,
275
+ zero_point=not sym,
276
+ lm_head_quantized=False,
277
+ full_config={},
278
+ modules_to_not_convert=[],
279
+ )
280
+ else:
281
+ from vllm.model_executor.layers.quantization.awq import (
282
+ AWQConfig,
283
+ AWQLinearMethod,
284
+ )
285
+
286
+ quant_args = AWQConfig(
287
+ weight_bits=weight_bits,
288
+ group_size=group_size,
289
+ zero_point=not sym,
290
+ )
291
+
292
+ if isinstance(layer, FusedMoE):
293
+ if use_marlin:
294
+ return AWQMarlinMoEMethod(quant_args_marlin, layer.moe_config)
295
+ from vllm.model_executor.layers.quantization.moe_wna16 import MoeWNA16Config
296
+
297
+ config = {
298
+ "quant_method": "awq",
299
+ "bits": weight_bits,
300
+ "group_size": group_size,
301
+ "zero_point": not sym,
302
+ "lm_head": False,
303
+ }
304
+ return MoeWNA16Config.from_config(config).get_quant_method(layer, prefix)
305
+
306
+ if isinstance(layer, (LinearBase, ParallelLMHead)):
307
+ if use_marlin:
308
+ return AWQMarlinLinearMethod(quant_args_marlin)
309
+ else:
310
+ return AWQLinearMethod(quant_args)
311
return None
312
59
- @classmethod
60
- def get_min_capability(cls) -> int:
61
- raise AssertionError
313
+ def apply_gptq_quant_layer(self, layer, prefix: str, backend: str = "auto"):
314
+ from vllm.model_executor.layers.fused_moe import FusedMoE
315
+ from vllm.model_executor.layers.quantization.utils.marlin_utils import (
316
+ check_marlin_supported,
317
+ check_moe_marlin_supports_layer,
318
+ )
319
+
320
+ weight_bits, group_size, sym = self.get_layer_config(layer, prefix)
321
+ if not self.check_quantized(weight_bits):
322
+ if isinstance(layer, (LinearBase, ParallelLMHead)):
323
+ return UnquantizedLinearMethod()
324
+ else:
325
+ return None
326
+
327
+ logger.debug(
328
+ "[%s] Type: %s, Bits: %s, Group Size: %s, Sym: %s",
329
+ prefix,
330
+ layer.__class__.__name__,
331
+ weight_bits,
332
+ group_size,
333
+ sym,
334
+ )
335
+ if backend == "auto" or "marlin" in backend:
336
+ GPTQ_TYPE_MAP = {
337
+ (4, True): scalar_types.uint4b8,
338
+ (8, True): scalar_types.uint8b128,
339
+ }
340
+ use_marlin = (weight_bits, sym) in GPTQ_TYPE_MAP and check_marlin_supported(
341
+ GPTQ_TYPE_MAP[(weight_bits, sym)], group_size, has_zp=not sym
342
+ )
343
+ if isinstance(layer, FusedMoE):
344
+ use_marlin = use_marlin and check_moe_marlin_supports_layer(
345
+ layer, group_size
346
+ )
347
+ else:
348
+ use_marlin = False
349
+ if use_marlin:
350
+ from vllm.model_executor.layers.quantization.gptq_marlin import (
351
+ GPTQMarlinConfig,
352
+ GPTQMarlinLinearMethod,
353
+ GPTQMarlinMoEMethod,
354
+ )
355
63
- @staticmethod
64
- def get_config_filenames() -> list[str]:
65
- return []
356
+ quant_args_marlin = GPTQMarlinConfig(
357
+ weight_bits=weight_bits,
358
+ group_size=group_size,
359
+ is_sym=sym,
360
+ lm_head_quantized=False,
361
+ desc_act=False,
362
+ dynamic={},
363
+ full_config={},
364
+ )
365
+ else:
366
+ from vllm.model_executor.layers.quantization.gptq import (
367
+ GPTQConfig,
368
+ GPTQLinearMethod,
369
+ )
370
+
371
+ quant_args = GPTQConfig(
372
+ weight_bits=weight_bits,
373
+ group_size=group_size,
374
+ lm_head_quantized=False,
375
+ desc_act=False,
376
+ dynamic={},
377
+ )
378
+
379
+ if isinstance(layer, FusedMoE):
380
+ if use_marlin:
381
+ return GPTQMarlinMoEMethod(quant_args_marlin, layer.moe_config)
382
+ else:
383
+ from vllm.model_executor.layers.quantization.moe_wna16 import (
384
+ MoeWNA16Config,
385
+ )
386
+
387
+ config = {
388
+ "quant_method": "gptq",
389
+ "bits": weight_bits,
390
+ "group_size": group_size,
391
+ "sym": sym,
392
+ "lm_head": False,
393
+ }
394
+ return MoeWNA16Config.from_config(config).get_quant_method(
395
+ layer, prefix
396
+ )
397
+
398
+ if isinstance(layer, (LinearBase, ParallelLMHead)):
399
+ if use_marlin:
400
+ return GPTQMarlinLinearMethod(quant_args_marlin)
401
+ else:
402
+ return GPTQLinearMethod(quant_args)
403
+
404
+ return None
405
+
406
+ def apply_ipex_quant_layer(self, layer, prefix: str):
407
+ weight_bits, group_size, sym = self.get_layer_config(layer, prefix)
408
+ if not self.check_quantized(weight_bits):
409
+ if isinstance(layer, (LinearBase, ParallelLMHead)):
410
+ return UnquantizedLinearMethod()
411
+ else:
412
+ return None
413
+ from vllm.model_executor.layers.quantization.ipex_quant import (
414
+ IPEXAWQLinearMethod,
415
+ IPEXConfig,
416
+ IPEXGPTQLinearMethod,
417
+ )
418
+
419
+ if isinstance(layer, (LinearBase, ParallelLMHead)):
420
+ if "awq" in self.packing_format:
421
+ config = IPEXConfig(
422
+ method="awq", weight_bits=weight_bits, group_size=group_size
423
+ )
424
+ return IPEXAWQLinearMethod(config)
425
+ elif "gptq" in self.packing_format:
426
+ config = IPEXConfig(
427
+ method="gptq", weight_bits=weight_bits, group_size=group_size
428
+ )
429
+ return IPEXGPTQLinearMethod(config)
430
+ else:
431
+ raise ValueError(
432
+ f"ipex backend only supports awq "
433
+ f"and gptq format,but got {self.packing_format}"
434
+ )
435
+ else:
436
+ return None
437
+
438
+ def get_quant_method(self, layer: torch.nn.Module, prefix: str):
439
+ if prefix and self.extra_config:
440
+ for layer_name in self.extra_config:
441
+ if (
442
+ layer_name == prefix or layer_name == f"model.{prefix}"
443
+ ) and self.extra_config[layer_name].get("bits", 16) >= 16:
444
+ return UnquantizedLinearMethod()
445
+ if (
446
+ current_platform.is_cpu()
447
+ or current_platform.is_xpu()
448
+ or self.backend == "ipex"
449
+ ):
450
+ return self.apply_ipex_quant_layer(layer, prefix)
451
+ if "gptq" in self.packing_format or "gptq" in self.backend:
452
+ return self.apply_gptq_quant_layer(layer, prefix)
453
+ if "awq" in self.packing_format or "awq" in self.backend:
454
+ return self.apply_awq_quant_layer(layer, prefix)
455
+
456
+ @classmethod
457
+ def override_quantization_method(
458
+ cls, hf_quant_cfg, user_quant
459
+ ) -> Optional["QuantizationMethods"]:
460
+ """Override the `auto-round` method to `inc`."""
461
+ is_auto_round_format = hf_quant_cfg.get("quant_method", None) == "auto-round"
462
+ if is_auto_round_format:
463
+ return cls.get_name()
464
+ return None
vllm/model_executor/model_loader/weight_utils.py
+1
-1
@@ -233,7 +233,7 @@ def get_quant_config(
233
quant_cls = get_quantization_config(model_config.quantization)
234
235
# GGUF doesn't have config file
236
- if model_config.quantization in ("gguf", "inc"):
236
+ if model_config.quantization == "gguf":
237
return quant_cls()
238
239
# Read the quantization config from the HF model config, if available.