@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / ad9776353e
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 23:30 UTC

Set `torch_dtype` in `TransformersModel` (#13088)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Harry Mellor committed Feb 11, 2025 at 15:51 UTC ad9776353e6b00d019415e94fd17c78ad4575ff7
1 file changed +1
vllm/model_executor/models/transformers.py
+1
@@ -143,6 +143,7 @@ class TransformersModel(nn.Module):
143 self.model: PreTrainedModel = AutoModel.from_config(
144 self.config,
145 attn_implementation="vllm",
146 + torch_dtype=vllm_config.model_config.dtype,
147 trust_remote_code=vllm_config.model_config.trust_remote_code,
148 )
149 prefix = self.model.base_model_prefix