A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 06:00 UTC
[BugFix] Force registration of w8a8_block_fp8_matmul_deepgemm via lazy import (#19514)
Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com> Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Varun Sundar Rabindranath committed
Jun 12, 2025 at 00:28 UTC
e5d35d62f5bd308abf05ee0bfc23bc17c5c46be2
1 file changed
+1
vllm/model_executor/layers/quantization/utils/fp8_utils.py
+1
@@ -143,6 +143,7 @@ def apply_w8a8_block_fp8_linear(
143
column_major_scales=True,
144
)
145
146
+ import vllm.model_executor.layers.quantization.deepgemm # noqa: F401
147
output = torch.ops.vllm.w8a8_block_fp8_matmul_deepgemm(
148
q_input,
149
weight,