@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 0ee349b553
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 00:45 UTC

[V1][Bugfix] Fix assertion when mm hashing is turned off (#12439)

Signed-off-by: Roger Wang <ywang@roblox.com>

Roger Wang committed Jan 26, 2025 at 00:47 UTC 0ee349b5534e3d02b499b1126f2abde73b798fe9
1 file changed +2 -1
vllm/v1/request.py
+2 -1
@@ -58,7 +58,8 @@ class Request:
58
59 # Sanity check
60 assert len(self.mm_inputs) == len(self.mm_positions)
61 - assert len(self.mm_inputs) == len(self.mm_hashes)
61 + if self.mm_hashes:
62 + assert len(self.mm_inputs) == len(self.mm_hashes)
63
64 # Cache the computed kv block hashes of the request to avoid
65 # recomputing.