| 1 | # Endpoint Plugins |
| 2 | |
| 3 | Endpoint plugins let out-of-tree packages add HTTP routes to the OpenAI compatible API server. Their scope is |
| 4 | the **HTTP surface only** registering routes and optionally per app state used by those routes. A plugin reaches the engine the same way an in-tree serving handler does, through the `EngineClient` it is handed at startup (e.g. `engine_client.collective_rpc(...)`). No new engine access path is introduced. |
| 5 | |
| 6 | !!! warning "Security" |
| 7 | Endpoint plugins are **not loaded by default** and must be explicitly allowlisted. Read [Endpoint Plugins security posture](../usage/security.md#endpoint-plugins) before enabling one, especially the route shadowing warning. |
| 8 | |
| 9 | ## The `EndpointPlugin` protocol |
| 10 | |
| 11 | Endpoint plugins implement the [`EndpointPlugin`][vllm.plugins.endpoint_plugins.interface.EndpointPlugin] runtime checkable `Protocol`: |
| 12 | |
| 13 | ```python |
| 14 | class EndpointPlugin(Protocol): |
| 15 | name: str |
| 16 | required_tasks: tuple[SupportedTask, ...] | None |
| 17 | |
| 18 | def attach_router(self, app: FastAPI) -> None: ... |
| 19 | |
| 20 | async def init_state( |
| 21 | self, engine_client: EngineClient | None, state: State, args: Namespace |
| 22 | ) -> None: ... |
| 23 | ``` |
| 24 | |
| 25 | - `name`: a unique identifier used in logs and for `VLLM_PLUGINS` allowlisting |
| 26 | - `required_tasks`: the tasks the server must support for this plugin to load. `None` means the plugin has no task requirement |
| 27 | - `attach_router`: registers routes on `app` |
| 28 | - `init_state`: initializes per app state the routes read at request time |
| 29 | |
| 30 | ## The two phase lifecycle |
| 31 | |
| 32 | Routes are registered before the engine exists. This means the interface has to expose two hooks that run at two different points in server startup: |
| 33 | |
| 34 | | Phase | Called from | `engine_client` available? | Work | |
| 35 | | --- | --- | --- | --- | |
| 36 | | A. Route registration | `build_app()` | No | `attach_router(app)` add routes. Do not touch the engine here. | |
| 37 | | B. State init | `init_app_state()` | Usually but `None` on the CPU only render server | `init_state(engine_client, state, args)` build a serving handler holding `engine_client` and store it on `state`. | |
| 38 | |
| 39 | Because `app.state` *is* the `state` object passed to `init_app_state()`, an object stored during phase A is visible in phase B and an object stored in phase B is visible to route handlers at request time via `request.app.state`. This is the same pattern in-tree endpoints already use. |
| 40 | |
| 41 | ### Engine less servers (the render server) |
| 42 | |
| 43 | The CPU only render server (`init_render_app_state()`) has no `EngineClient`. It still runs both phases for any plugin eligible for the `render` task (`required_tasks` is `None` or includes `"render"`). `attach_router` is called as usual but `init_state` is called with `engine_client=None`. |
| 44 | |
| 45 | A plugin that needs an engine to function has two options: |
| 46 | |
| 47 | - Exclude `"render"` from `required_tasks` so it is never loaded on the render server in the first place |
| 48 | - Accept being loaded on `render` and check for `None` in `init_state` or in the route handler returning an error response (e.g. HTTP 503) instead of dereferencing a client that doesn't exist |
| 49 | |
| 50 | `tests/plugins/vllm_add_dummy_endpoint_plugin` demonstrates the second option. Its route handler returns a 503 when `state.dummy_engine_client` is `None`. |
| 51 | |
| 52 | ### Reaching the engine from a route handler |
| 53 | |
| 54 | `init_state` is where a plugin captures `engine_client` into a small serving handler and stashes it on `state`. The route added in `attach_router` reads that handler off `request.app.state` at request time and calls the engine through it, typically via `engine_client.collective_rpc(...)`. |
| 55 | |
| 56 | This minimal example omits the `None` check from the previous section for brevity since `required_tasks` is `None` here. It is in fact eligible for `render` and should handle `engine_client=None` the way `tests/plugins/vllm_add_dummy_endpoint_plugin` does before shipping it: |
| 57 | |
| 58 | ```python |
| 59 | from fastapi import FastAPI, Request |
| 60 | |
| 61 | |
| 62 | class MyAdminEndpointPlugin: |
| 63 | name = "my_admin_endpoint_plugin" |
| 64 | required_tasks: tuple[str, ...] | None = None |
| 65 | |
| 66 | def attach_router(self, app: FastAPI) -> None: |
| 67 | @app.get("/plugins/my_admin_endpoint_plugin/scheduler_config") |
| 68 | async def scheduler_config(raw_request: Request): |
| 69 | engine_client = raw_request.app.state.my_engine_client |
| 70 | results = await engine_client.collective_rpc("get_scheduler_config") |
| 71 | return {"scheduler_config": results} |
| 72 | |
| 73 | async def init_state(self, engine_client, state, args) -> None: |
| 74 | state.my_engine_client = engine_client |
| 75 | ``` |
| 76 | |
| 77 | A complete and tested version of this example is in-repo as `tests/plugins/vllm_add_dummy_endpoint_plugin` and is exercised e2e (including a real HTTP request) in `tests/plugins_tests/test_endpoint_plugins.py`. |
| 78 | |
| 79 | ## Registering the entry point |
| 80 | |
| 81 | Register a zero argument factory (a class or function) under the `vllm.endpoint_plugins` group. The factory must return an object satisfying `EndpointPlugin`: |
| 82 | |
| 83 | ```toml |
| 84 | # pyproject.toml |
| 85 | [project.entry-points."vllm.endpoint_plugins"] |
| 86 | my_admin_api = "my_pkg.endpoints:MyAdminEndpointPlugin" |
| 87 | ``` |
| 88 | |
| 89 | ```python |
| 90 | # setup.py equivalent |
| 91 | setup( |
| 92 | name="my_pkg", |
| 93 | entry_points={ |
| 94 | "vllm.endpoint_plugins": [ |
| 95 | "my_admin_api = my_pkg.endpoints:MyAdminEndpointPlugin" |
| 96 | ] |
| 97 | }, |
| 98 | ) |
| 99 | ``` |
| 100 | |
| 101 | The entry point name (`my_admin_api` above) is independent of the plugin's `name` attribute. `VLLM_PLUGINS` allowlisting matches on the **entry point name** following the same convention as `vllm.general_plugins` (see [Plugin System](plugin_system.md)). |
| 102 | |
| 103 | ## Gating: `VLLM_PLUGINS` and `required_tasks` |
| 104 | |
| 105 | Endpoint plugins are discovered and gated by [`load_endpoint_plugins`][vllm.plugins.load_endpoint_plugins] which is stricter than the loader used for other plugin groups: |
| 106 | |
| 107 | - **Nothing loads unless `VLLM_PLUGINS` is set and names the plugin.** Other plugin groups load everything unless `VLLM_PLUGINS` narrows the set. Endpoint plugins invert that default because they add network exposed surface. See [Security](../usage/security.md#endpoint-plugins). |
| 108 | - **`required_tasks` must intersect the server's supported tasks** unless it is `None`. Use this to keep a plugin from attaching routes on a server that can't service them (e.g. a pooling only deployment). |
| 109 | - A factory that raises an issue during instantiation is logged and skipped. It does not abort server startup. |
| 110 | |
| 111 | Only the front end API server process loads endpoint plugins. There is no need to guard for worker or engine core processes. |
| 112 | |
| 113 | ## Pairing with `vllm.general_plugins` |
| 114 | |
| 115 | Endpoint plugins cover the HTTP surface only. If a plugin also needs new engine side behavior (a new worker-side RPC method, a custom stat) that half ships separately through the existing `vllm.general_plugins` group which loads in worker processes (see [Plugin System](plugin_system.md)). The two entry points are registered and loaded **independently**. Neither implies the other. The recommended distribution shape is a single package exposing both: |
| 116 | |
| 117 | ```toml |
| 118 | [project.entry-points."vllm.general_plugins"] |
| 119 | my_admin_engine = "my_pkg.engine:register" # adds the worker side method |
| 120 | |
| 121 | [project.entry-points."vllm.endpoint_plugins"] |
| 122 | my_admin_api = "my_pkg.endpoints:MyAdminEndpointPlugin" # adds the HTTP route |
| 123 | ``` |
| 124 | |
| 125 | Do not expect a single endpoint plugin to also mutate engine/worker state. If your route needs a worker side method that doesn't already exist then add it via a paired `general_plugins` entry point. |
| 126 | |
| 127 | ## Path-prefix convention |
| 128 | |
| 129 | There is currently no route conflict enforcement (tracked as a follow-up to RFC [#46565](https://github.com/vllm-project/vllm/issues/46565)). A plugin's `attach_router` can register a path that collides with a core route and routes attached later win. To avoid surprising operators: |
| 130 | |
| 131 | - Namespace your routes under a distinct prefix, e.g. `/plugins/<plugin-name>/...`, rather than reusing `/v1/...` or other core prefixes |
| 132 | - Only register routes under a core prefix (like the worked example's `/v1/admin/scheduler_config`) if you specifically intend to override or extend existing behavior and document that clearly for operators allowlisting your plugin |
| 133 | |
| 134 | ## Compatibility |
| 135 | |
| 136 | `state`/serving handler internals (e.g. the shape of in-tree `OpenAIServing*` classes) are not a stable public contract yet. Treat them as use-at-your-own-risk and expect them to change between vLLM versions. `FastAPI`, `EngineClient` and the `EndpointPlugin` protocol itself are the supported surface. |