@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / f7dac83d95
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 22:00 UTC

[Kernel] Raise an exception in MoE kernel if the batch size is larger then 65k (#5939)

Cody Yu committed Jun 29, 2024 at 06:04 UTC f7dac83d95ae38973b425a8bb2d3a3df9fe9a9c2
1 file changed +5
vllm/model_executor/layers/fused_moe/fused_moe.py
+5
@@ -423,6 +423,11 @@ def fused_experts(hidden_states: torch.Tensor,
423 M, _ = hidden_states.shape
424 E, N, _ = w1.shape
425
426 + if M > 65536:
427 + # https://github.com/vllm-project/vllm/issues/5938
428 + raise ValueError("MoE kernel does not support more than 65536 tokens, "
429 + f"but got {M}")
430 +
431 if override_config:
432 config = override_config
433 else: