@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 9c8e2d1161
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 18:15 UTC

[Bugfix][Harmless] Fix float16 dtype for model_is_embedding (#7566)

Michael Goin committed Aug 15, 2024 at 21:26 UTC 9c8e2d11611ca5313710df06f215b41f05dde1b4
1 file changed +1 -1
vllm/entrypoints/openai/api_server.py
+1 -1
@@ -66,7 +66,7 @@ def model_is_embedding(model_name: str, trust_remote_code: bool) -> bool:
66 tokenizer_mode="auto",
67 trust_remote_code=trust_remote_code,
68 seed=0,
69 - dtype="float16").embedding_mode
69 + dtype="auto").embedding_mode
70
71
72 @asynccontextmanager