A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 11:00 UTC
[Bugfix][V1] Only get input embeddings w/ multi-modal models if first PP (#17916)
Signed-off-by: Jin Huang <jinhun@amazon.com> Co-authored-by: Jin Huang <jinhun@amazon.com>
Jin Huang committed
May 13, 2025 at 03:10 UTC
8dd0671baca64527ddc53dbe93a78b800fdc3e54
1 file changed
+1
-1
vllm/v1/worker/gpu_model_runner.py
+1
-1
@@ -1107,7 +1107,7 @@ class GPUModelRunner(LoRAModelRunnerMixin):
1107
else:
1108
mm_embeds = []
1109
1110
- if self.is_multimodal_model:
1110
+ if self.is_multimodal_model and get_pp_group().is_first_rank:
1111
# NOTE(woosuk): To unify token ids and soft tokens (vision
1112
# embeddings), we always use embeddings (rather than token ids)
1113
# as input to the multimodal model, even when the input is text.