@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / cbe94391eb
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 02:00 UTC

Fix: cases with empty sparsity config (#12057)

Signed-off-by: Rahul Tuli <rahul@neuralmagic.com>

Rahul Tuli committed Jan 15, 2025 at 04:41 UTC cbe94391eb04aa9ae1be15711fec4eb453c1e053
1 file changed +1 -1
vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py
+1 -1
@@ -113,7 +113,7 @@ class CompressedTensorsConfig(QuantizationConfig):
113 :return: A dictionary mapping target layer names to their corresponding
114 sparsity compression configurations
115 """
116 - if (sparsity_config := config.get(SPARSITY_CONFIG_NAME)) is None:
116 + if not (sparsity_config := config.get(SPARSITY_CONFIG_NAME)):
117 return dict()
118
119 sparsity_config = SparsityCompressionConfig.model_validate(