A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 14:45 UTC
Help user create custom model for Transformers backend remote code models (#16719)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Apr 17, 2025 at 02:05 UTC
3cd91dc9555e6f10e55f23d37782c65b0366f7cf
1 file changed
+4
docs/source/models/supported_models.md
+4
@@ -55,6 +55,10 @@ If your model is neither supported natively by vLLM or Transformers, you can sti
55
Simply set `trust_remote_code=True` and vLLM will run any model on the Model Hub that is compatible with Transformers.
56
Provided that the model writer implements their model in a compatible way, this means that you can run new models before they are officially supported in Transformers or vLLM!
57
58
+:::{tip}
59
+If you have not yet created your custom model, you can follow this guide on [customising models in Transformers](https://huggingface.co/docs/transformers/en/custom_models).
60
+:::
61
+
62
```python
63
from vllm import LLM
64
llm = LLM(model=..., task="generate", trust_remote_code=True) # Name or path of your model