@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / e2251109c7
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 05:45 UTC

[Kernel] Remove if-else with identical branches in marlin 2:4 (#10687)

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

Tyler Michael Smith committed Nov 27, 2024 at 01:55 UTC e2251109c746f0d08ab9b37b5abcf44ca105d426
1 file changed +3 -7
csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu
+3 -7
@@ -296,13 +296,9 @@ __global__ void Marlin_24(
296 // We use a different scale layout for grouped and column-wise quantization as
297 // we scale a `half2` tile in column-major layout in the former and in
298 // row-major in the latter case.
299 - if (group_blocks != -1) {
300 - s_sh_rd = 8 * ((threadIdx.x / 32) % (thread_n_blocks / 4)) +
301 - (threadIdx.x % 32) / 4;
302 - } else {
303 - s_sh_rd = 8 * ((threadIdx.x / 32) % (thread_n_blocks / 4)) +
304 - (threadIdx.x % 32) / 4;
305 - }
299 + s_sh_rd = 8 * ((threadIdx.x / 32) % (thread_n_blocks / 4)) +
300 + (threadIdx.x % 32) / 4; // Note that in the original Marlin kernel
301 + // this is (threadIdx.x % 32) / 4
302
303 // Precompute which thread should not read memory in which iterations; this is
304 // needed if there are more threads than required for a certain tilesize or