A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 19:45 UTC
[Docs] Mention `model_impl` arg when explaining Transformers fallback (#14552)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Mar 10, 2025 at 13:13 UTC
60a98b2de5ffab549ec768275f08c2dfb236f99b
1 file changed
+4
docs/source/models/supported_models.md
+4
@@ -59,6 +59,10 @@ llm.apply_model(lambda model: print(type(model)))
59
60
If it is `TransformersModel` then it means it's based on Transformers!
61
62
+:::{tip}
63
+You can force the use of `TransformersModel` by setting `model_impl="transformers"` for <project:#offline-inference> or `--model-impl transformers` for the <project:#openai-compatible-server>.
64
+:::
65
+
66
:::{note}
67
vLLM may not fully optimise the Transformers implementation so you may see degraded performance if comparing a native model to a Transformers model in vLLM.
68
:::