@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 755b85359b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 07:00 UTC

[doc] add doc for the plugin system (#10372)

Signed-off-by: youkaichao <youkaichao@gmail.com>

youkaichao committed Nov 15, 2024 at 21:46 UTC 755b85359be910fabe39a75299439fc11beb57d4
5 files changed +84 -22
docs/source/design/class_hierarchy.rst
+2
@@ -1,3 +1,5 @@
1 +.. _class_hierarchy:
2 +
3 vLLM's Class Hierarchy
4 =======================
5
docs/source/design/plugin_system.rst new
+62
@@ -0,0 +1,62 @@
1 +.. _plugin_system:
2 +
3 +vLLM's Plugin System
4 +====================
5 +
6 +The community frequently requests the ability to extend vLLM with custom features. To facilitate this, vLLM includes a plugin system that allows users to add custom features without modifying the vLLM codebase. This document explains how plugins work in vLLM and how to create a plugin for vLLM.
7 +
8 +How Plugins Work in vLLM
9 +------------------------
10 +
11 +Plugins are user-registered code that vLLM executes. Given vLLM's architecture (see :ref:`class_hierarchy`), multiple processes may be involved, especially when using distributed inference with various parallelism techniques. To enable plugins successfully, every process created by vLLM needs to load the plugin. This is done by the `load_general_plugins <https://github.com/vllm-project/vllm/blob/c76ac49d266e27aa3fea84ef2df1f813d24c91c7/vllm/plugins/__init__.py#L16>`__ function in the ``vllm.plugins`` module. This function is called for every process created by vLLM before it starts any work.
12 +
13 +How vLLM Discovers Plugins
14 +--------------------------
15 +
16 +vLLM's plugin system uses the standard Python ``entry_points`` mechanism. This mechanism allows developers to register functions in their Python packages for use by other packages. An example of a plugin:
17 +
18 +.. code-block:: python
19 +
20 + # inside `setup.py` file
21 + from setuptools import setup
22 +
23 + setup(name='vllm_add_dummy_model',
24 + version='0.1',
25 + packages=['vllm_add_dummy_model'],
26 + entry_points={
27 + 'vllm.general_plugins':
28 + ["register_dummy_model = vllm_add_dummy_model:register"]
29 + })
30 +
31 + # inside `vllm_add_dummy_model.py` file
32 + def register():
33 + from vllm import ModelRegistry
34 +
35 + if "MyLlava" not in ModelRegistry.get_supported_archs():
36 + ModelRegistry.register_model("MyLlava",
37 + "vllm_add_dummy_model.my_llava:MyLlava")
38 +
39 +For more information on adding entry points to your package, please check the `official documentation <https://setuptools.pypa.io/en/latest/userguide/entry_point.html>`__.
40 +
41 +Every plugin has three parts:
42 +
43 +1. **Plugin group**: The name of the entry point group. vLLM uses the entry point group ``vllm.general_plugins`` to register general plugins. This is the key of ``entry_points`` in the ``setup.py`` file. Always use ``vllm.general_plugins`` for vLLM's general plugins.
44 +
45 +2. **Plugin name**: The name of the plugin. This is the value in the dictionary of the ``entry_points`` dictionary. In the example above, the plugin name is ``register_dummy_model``. Plugins can be filtered by their names using the ``VLLM_PLUGINS`` environment variable. To load only a specific plugin, set ``VLLM_PLUGINS`` to the plugin name.
46 +
47 +3. **Plugin value**: The fully qualified name of the function to register in the plugin system. In the example above, the plugin value is ``vllm_add_dummy_model:register``, which refers to a function named ``register`` in the ``vllm_add_dummy_model`` module.
48 +
49 +What Can Plugins Do?
50 +--------------------
51 +
52 +Currently, the primary use case for plugins is to register custom, out-of-the-tree models into vLLM. This is done by calling ``ModelRegistry.register_model`` to register the model. In the future, the plugin system may be extended to support more features, such as swapping in custom implementations for certain classes in vLLM.
53 +
54 +Guidelines for Writing Plugins
55 +------------------------------
56 +
57 +- **Being re-entrant**: The function specified in the entry point should be re-entrant, meaning it can be called multiple times without causing issues. This is necessary because the function might be called multiple times in some processes.
58 +
59 +Compatibility Guarantee
60 +-----------------------
61 +
62 +vLLM guarantees the interface of documented plugins, such as ``ModelRegistry.register_model``, will always be available for plugins to register models. However, it is the responsibility of plugin developers to ensure their plugins are compatible with the version of vLLM they are targeting. For example, ``"vllm_add_dummy_model.my_llava:MyLlava"`` should be compatible with the version of vLLM that the plugin targets. The interface for the model may change during vLLM's development.
\ No newline at end of file
docs/source/index.rst
+1
@@ -158,6 +158,7 @@ Documentation
158
159 design/class_hierarchy
160 design/huggingface_integration
161 + design/plugin_system
162 design/input_processing/model_inputs_index
163 design/kernel/paged_attention
164 design/multimodal/multimodal_index
docs/source/models/adding_model.rst
+7 -18
@@ -102,11 +102,11 @@ This method should load the weights from the HuggingFace's checkpoint file and a
102 Finally, register your :code:`*ForCausalLM` class to the :code:`_VLLM_MODELS` in `vllm/model_executor/models/registry.py <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/registry.py>`_.
103
104 6. Out-of-Tree Model Integration
105 ---------------------------------------------
105 +--------------------------------
106
107 -We also provide a way to integrate a model without modifying the vLLM codebase. Step 2, 3, 4 are still required, but you can skip step 1 and 5.
107 +You can integrate a model without modifying the vLLM codebase. Steps 2, 3, and 4 are still required, but you can skip steps 1 and 5. Instead, write a plugin to register your model. For general introduction of the plugin system, see :ref:`plugin_system`.
108
109 -Just add the following lines in your code:
109 +To register the model, use the following code:
110
111 .. code-block:: python
112
@@ -114,7 +114,7 @@ Just add the following lines in your code:
114 from your_code import YourModelForCausalLM
115 ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)
116
117 -If your model imports modules that initialize CUDA, consider instead lazy-importing it to avoid an error like :code:`RuntimeError: Cannot re-initialize CUDA in forked subprocess`:
117 +If your model imports modules that initialize CUDA, consider lazy-importing it to avoid errors like :code:`RuntimeError: Cannot re-initialize CUDA in forked subprocess`:
118
119 .. code-block:: python
120
@@ -123,19 +123,8 @@ If your model imports modules that initialize CUDA, consider instead lazy-import
123 ModelRegistry.register_model("YourModelForCausalLM", "your_code:YourModelForCausalLM")
124
125 .. important::
126 - If your model is a multimodal model, make sure the model class implements the :class:`~vllm.model_executor.models.interfaces.SupportsMultiModal` interface.
126 + If your model is a multimodal model, ensure the model class implements the :class:`~vllm.model_executor.models.interfaces.SupportsMultiModal` interface.
127 Read more about that :ref:`here <enabling_multimodal_inputs>`.
128
129 -If you are running api server with :code:`vllm serve <args>`, you can wrap the entrypoint with the following code:
130 -
131 -.. code-block:: python
132 -
133 - from vllm import ModelRegistry
134 - from your_code import YourModelForCausalLM
135 - ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)
136 -
137 - if __name__ == '__main__':
138 - import runpy
139 - runpy.run_module('vllm.entrypoints.openai.api_server', run_name='__main__')
140 -
141 -Save the above code in a file and run it with :code:`python your_file.py <args>`.
129 +.. note::
130 + Although you can directly put these code snippets in your script using ``vllm.LLM``, the recommended way is to place these snippets in a vLLM plugin. This ensures compatibility with various vLLM features like distributed inference and the API server.
vllm/plugins/__init__.py
+12 -4
@@ -27,16 +27,24 @@ def load_general_plugins():
27 allowed_plugins = envs.VLLM_PLUGINS
28
29 discovered_plugins = entry_points(group='vllm.general_plugins')
30 + logger.info("Available plugins:")
31 + for plugin in discovered_plugins:
32 + logger.info("name=%s, value=%s, group=%s", plugin.name, plugin.value,
33 + plugin.group)
34 + if allowed_plugins is None:
35 + logger.info("all available plugins will be loaded.")
36 + logger.info("set environment variable VLLM_PLUGINS to control"
37 + " which plugins to load.")
38 + else:
39 + logger.info("plugins to load: %s", allowed_plugins)
40 for plugin in discovered_plugins:
31 - logger.info("Found general plugin: %s", plugin.name)
41 if allowed_plugins is None or plugin.name in allowed_plugins:
42 try:
43 func = plugin.load()
44 func()
36 - logger.info("Loaded general plugin: %s", plugin.name)
45 + logger.info("plugin %s loaded.", plugin.name)
46 except Exception:
38 - logger.exception("Failed to load general plugin: %s",
39 - plugin.name)
47 + logger.exception("Failed to load plugin %s", plugin.name)
48
49
50 _torch_compile_backend: Optional[Union[Callable, str]] = None