@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 3d158cdc8d
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 07:00 UTC

Add default value to avoid Falcon crash (#5363) (#10347)

Signed-off-by: wchen61 <wchen61@foxmail.com>

wchen61 committed Nov 15, 2024 at 16:52 UTC 3d158cdc8dad62dfed45d5d808ae9f14f16e4dae
1 file changed +3
vllm/model_executor/models/falcon.py
+3
@@ -250,6 +250,9 @@ class FalconDecoderLayer(nn.Module):
250 self.mlp = FalconMLP(config, quant_config)
251 self.config = config
252
253 + if (not hasattr(config, "num_ln_in_parallel_attn")):
254 + config.num_ln_in_parallel_attn = None
255 +
256 if (config.num_ln_in_parallel_attn is None
257 and config.new_decoder_architecture):
258 config.num_ln_in_parallel_attn = 2