@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 18511aeda6
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 10:45 UTC

[Bugfix] Fix Machete unittests failing with `NotImplementedError` (#9218)

Lucas Wilkinson committed Oct 10, 2024 at 13:39 UTC 18511aeda64b473314bb7727a97a220565e0af41
1 file changed +4
csrc/quantization/machete/machete_pytorch.cu
+4
@@ -89,6 +89,10 @@ torch::Tensor prepack_B(torch::Tensor const& B,
89 TORCH_LIBRARY_IMPL_EXPAND(TORCH_EXTENSION_NAME, CUDA, m) {
90 m.impl("machete_prepack_B", &prepack_B);
91 m.impl("machete_gemm", &gemm);
92 +}
93 +
94 +// use CatchAll since supported_schedules has no tensor arguments
95 +TORCH_LIBRARY_IMPL(TORCH_EXTENSION_NAME, CatchAll, m) {
96 m.impl("machete_supported_schedules", &supported_schedules);
97 }
98