@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / d1adb9b403
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 02:00 UTC

[BugFix] add more `is not None` check in VllmConfig.__post_init__ (#12138)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

Chen Zhang committed Jan 17, 2025 at 13:33 UTC d1adb9b4032dd430bb28b8e91feb8164c3a1ca9c
1 file changed +2 -1
vllm/config.py
+2 -1
@@ -3174,7 +3174,8 @@ class VllmConfig:
3174
3175 if self.compilation_config is None:
3176 self.compilation_config = CompilationConfig()
3177 - if envs.VLLM_USE_V1 and not self.model_config.enforce_eager:
3177 + if envs.VLLM_USE_V1 and self.model_config is not None and \
3178 + not self.model_config.enforce_eager:
3179 # NOTE(woosuk): Currently, we use inductor because the piecewise
3180 # CUDA graphs do not work properly with the custom CUDA kernels.
3181 # FIXME(woosuk): Disable inductor to reduce the compilation time