A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 08:30 UTC
[Doc] Update OOT model docs (#18742)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
May 27, 2025 at 14:30 UTC
25a817f202ae86ab884ab83a10ff64435e92a5dc
2 files changed
+19
-18
docs/contributing/model/registration.md
+15
-16
@@ -23,33 +23,32 @@ Finally, update our [list of supported models][supported-models] to promote your
23
24
## Out-of-tree models
25
26
-You can load an external model using a plugin without modifying the vLLM codebase.
27
-
28
-!!! info
29
- [vLLM's Plugin System][plugin-system]
26
+You can load an external model [using a plugin][plugin-system] without modifying the vLLM codebase.
27
28
To register the model, use the following code:
29
30
```python
34
-from vllm import ModelRegistry
35
-from your_code import YourModelForCausalLM
36
-ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)
31
+# The entrypoint of your plugin
32
+def register():
33
+ from vllm import ModelRegistry
34
+ from your_code import YourModelForCausalLM
35
+
36
+ ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)
37
```
38
39
If your model imports modules that initialize CUDA, consider lazy-importing it to avoid errors like `RuntimeError: Cannot re-initialize CUDA in forked subprocess`:
40
41
```python
42
-from vllm import ModelRegistry
43
-
44
-ModelRegistry.register_model(
45
- "YourModelForCausalLM",
46
- "your_code:YourModelForCausalLM"
47
-)
42
+# The entrypoint of your plugin
43
+def register():
44
+ from vllm import ModelRegistry
45
+
46
+ ModelRegistry.register_model(
47
+ "YourModelForCausalLM",
48
+ "your_code:YourModelForCausalLM"
49
+ )
50
```
51
52
!!! warning
53
If your model is a multimodal model, ensure the model class implements the [SupportsMultiModal][vllm.model_executor.models.interfaces.SupportsMultiModal] interface.
54
Read more about that [here][supports-multimodal].
53
-
54
-!!! note
55
- Although you can directly put these code snippets in your script using `vllm.LLM`, the recommended way is to place these snippets in a vLLM plugin. This ensures compatibility with various vLLM features like distributed inference and the API server.
docs/design/plugin_system.md
+4
-2
@@ -30,8 +30,10 @@ def register():
30
from vllm import ModelRegistry
31
32
if "MyLlava" not in ModelRegistry.get_supported_archs():
33
- ModelRegistry.register_model("MyLlava",
34
- "vllm_add_dummy_model.my_llava:MyLlava")
33
+ ModelRegistry.register_model(
34
+ "MyLlava",
35
+ "vllm_add_dummy_model.my_llava:MyLlava",
36
+ )
37
```
38
39
For more information on adding entry points to your package, please check the [official documentation](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).