A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Bugfix] [B200] cutlass_mla - ensure kv_split == 1 for batch size > 1 (#25509)
Signed-off-by: Alexander Matveev <amatveev@redhat.com>
Alexander Matveev committed
Sep 23, 2025 at 19:57 UTC
1210e4d95b515507cd151ee76ef947999fdcbe59
1 file changed
+2
-2
csrc/attention/mla/cutlass_sm100_mla/device/sm100_mla.hpp
+2
-2
@@ -135,10 +135,10 @@ public:
135
max_splits = min(16, max_splits);
136
137
// TODO: This avoids a hang when the batch size larger than 1 and
138
- // there is more than 4 kv_splits.
138
+ // there is more than 1 kv_splits.
139
// Discuss with NVIDIA how this can be fixed.
140
if (B > 1) {
141
- max_splits = min(2, max_splits);
141
+ max_splits = min(1, max_splits);
142
}
143
144
// printf(" max_splits = %d\n", max_splits);