@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / ebd8c669ef
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 02:00 UTC

[Bugfix] Fix _get_lora_device for HQQ marlin (#12090)

Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com> Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>

Varun Sundar Rabindranath committed Jan 16, 2025 at 01:29 UTC ebd8c669efa54a218eb83735fd7ba40922f5f3ad
1 file changed +3
vllm/lora/layers.py
+3
@@ -51,6 +51,9 @@ def _get_lora_device(base_layer: nn.Module) -> torch.device:
51 # marlin
52 elif hasattr(base_layer, "B"):
53 return base_layer.B.device
54 + # HQQ marlin
55 + elif hasattr(base_layer, "W_q"):
56 + return base_layer.W_q.device
57 else:
58 raise ValueError(f"Unsupported base layer: {base_layer}")
59