@tduncan / Nvidia-Optimization-Stack-VLLM / commits / f7fc0ca993

[Frontend] Add endpoint plugins framework (#47454)

Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>

Martin Hickey committed Jul 8, 2026 at 03:00 UTC f7fc0ca9933ff4903dc6240a62e4194e38283c70
12 files changed +660 -1
.buildkite/test-amd.yaml
+5
@@ -1938,6 +1938,11 @@ steps:
1938 - pytest -v -s plugins_tests/test_stats_logger_plugins.py
1939 - pip uninstall dummy_stat_logger -y
1940 # END: `stat_logger` plugins test
1941 + # BEGIN: `endpoint` plugins test
1942 + - pip install -e ./plugins/vllm_add_dummy_endpoint_plugin
1943 + - pytest -v -s plugins_tests/test_endpoint_plugins.py
1944 + - pip uninstall vllm_add_dummy_endpoint_plugin -y
1945 + # END: `endpoint` plugins test
1946 # BEGIN: other tests
1947 - pytest -v -s plugins_tests/test_scheduler_plugins.py
1948 - pip install -e ./plugins/vllm_add_dummy_model
.buildkite/test_areas/plugins.yaml
+5
@@ -37,6 +37,11 @@ steps:
37 - pytest -v -s plugins_tests/test_stats_logger_plugins.py
38 - pip uninstall dummy_stat_logger -y
39 # end stat_logger plugins test
40 + # begin endpoint plugins test
41 + - pip install -e ./plugins/vllm_add_dummy_endpoint_plugin
42 + - pytest -v -s plugins_tests/test_endpoint_plugins.py
43 + - pip uninstall vllm_add_dummy_endpoint_plugin -y
44 + # end endpoint plugins test
45 # other tests continue here:
46 - pytest -v -s plugins_tests/test_scheduler_plugins.py
47 - pip install -e ./plugins/vllm_add_dummy_model
docs/design/endpoint_plugins.md new
+136
@@ -0,0 +1,136 @@
1 +# Endpoint Plugins
2 +
3 +Endpoint plugins let out-of-tree packages add HTTP routes to the OpenAI compatible API server without editing `vllm/entrypoints/openai/api_server.py`. Their scope is
4 +the **HTTP surface only** registering routes and optionally per app state used by those routes. A plugin reaches the engine the same way an in-tree serving handler does, through the `EngineClient` it is handed at startup (e.g. `engine_client.collective_rpc(...)`). No new engine access path is introduced.
5 +
6 +!!! warning "Security"
7 + Endpoint plugins are **not loaded by default** and must be explicitly allowlisted. Read [Endpoint Plugins security posture](../usage/security.md#endpoint-plugins) before enabling one, especially the route shadowing warning.
8 +
9 +## The `EndpointPlugin` protocol
10 +
11 +Endpoint plugins implement the [`EndpointPlugin`][vllm.plugins.endpoint_plugins.interface.EndpointPlugin] runtime checkable `Protocol`:
12 +
13 +```python
14 +class EndpointPlugin(Protocol):
15 + name: str
16 + required_tasks: tuple[SupportedTask, ...] | None
17 +
18 + def attach_router(self, app: FastAPI) -> None: ...
19 +
20 + async def init_state(
21 + self, engine_client: EngineClient | None, state: State, args: Namespace
22 + ) -> None: ...
23 +```
24 +
25 +- `name`: a unique identifier used in logs and for `VLLM_PLUGINS` allowlisting
26 +- `required_tasks`: the tasks the server must support for this plugin to load. `None` means the plugin has no task requirement
27 +- `attach_router`: registers routes on `app`
28 +- `init_state`: initializes per app state the routes read at request time
29 +
30 +## The two phase lifecycle
31 +
32 +Routes are registered before the engine exists. This means the interface has to expose two hooks that run at two different points in server startup:
33 +
34 +| Phase | Called from | `engine_client` available? | Work |
35 +| --- | --- | --- | --- |
36 +| A. Route registration | `build_app()` | No | `attach_router(app)` add routes. Do not touch the engine here. |
37 +| B. State init | `init_app_state()` | Usually but `None` on the CPU only render server | `init_state(engine_client, state, args)` build a serving handler holding `engine_client` and store it on `state`. |
38 +
39 +Because `app.state` *is* the `state` object passed to `init_app_state()`, an object stored during phase A is visible in phase B and an object stored in phase B is visible to route handlers at request time via `request.app.state`. This is the same pattern in-tree endpoints already use.
40 +
41 +### Engine less servers (the render server)
42 +
43 +The CPU only render server (`init_render_app_state()`) has no `EngineClient`. It still runs both phases for any plugin eligible for the `render` task (`required_tasks` is `None` or includes `"render"`). `attach_router` is called as usual but `init_state` is called with `engine_client=None`.
44 +
45 +A plugin that needs an engine to function has two options:
46 +
47 +- Exclude `"render"` from `required_tasks` so it is never loaded on the render server in the first place
48 +- Accept being loaded on `render` and check for `None` in `init_state` or in the route handler returning an error response (e.g. HTTP 503) instead of dereferencing a client that doesn't exist
49 +
50 +`tests/plugins/vllm_add_dummy_endpoint_plugin` demonstrates the second option. Its route handler returns a 503 when `state.dummy_engine_client` is `None`.
51 +
52 +### Reaching the engine from a route handler
53 +
54 +`init_state` is where a plugin captures `engine_client` into a small serving handler and stashes it on `state`. The route added in `attach_router` reads that handler off `request.app.state` at request time and calls the engine through it, typically via `engine_client.collective_rpc(...)`.
55 +
56 +This minimal example omits the `None` check from the previous section for brevity since `required_tasks` is `None` here. It is in fact eligible for `render` and should handle `engine_client=None` the way `tests/plugins/vllm_add_dummy_endpoint_plugin` does before shipping it:
57 +
58 +```python
59 +from fastapi import FastAPI, Request
60 +
61 +
62 +class MyAdminEndpointPlugin:
63 + name = "my_admin_endpoint_plugin"
64 + required_tasks: tuple[str, ...] | None = None
65 +
66 + def attach_router(self, app: FastAPI) -> None:
67 + @app.get("/plugins/my_admin_endpoint_plugin/scheduler_config")
68 + async def scheduler_config(raw_request: Request):
69 + engine_client = raw_request.app.state.my_engine_client
70 + results = await engine_client.collective_rpc("get_scheduler_config")
71 + return {"scheduler_config": results}
72 +
73 + async def init_state(self, engine_client, state, args) -> None:
74 + state.my_engine_client = engine_client
75 +```
76 +
77 +A complete and tested version of this example is in-repo as `tests/plugins/vllm_add_dummy_endpoint_plugin` and is exercised e2e (including a real HTTP request) in `tests/plugins_tests/test_endpoint_plugins.py`.
78 +
79 +## Registering the entry point
80 +
81 +Register a zero argument factory (a class or function) under the `vllm.endpoint_plugins` group. The factory must return an object satisfying `EndpointPlugin`:
82 +
83 +```toml
84 +# pyproject.toml
85 +[project.entry-points."vllm.endpoint_plugins"]
86 +my_admin_api = "my_pkg.endpoints:MyAdminEndpointPlugin"
87 +```
88 +
89 +```python
90 +# setup.py equivalent
91 +setup(
92 + name="my_pkg",
93 + entry_points={
94 + "vllm.endpoint_plugins": [
95 + "my_admin_api = my_pkg.endpoints:MyAdminEndpointPlugin"
96 + ]
97 + },
98 +)
99 +```
100 +
101 +The entry point name (`my_admin_api` above) is independent of the plugin's `name` attribute. `VLLM_PLUGINS` allowlisting matches on the **entry point name** following the same convention as `vllm.general_plugins` (see [Plugin System](plugin_system.md)).
102 +
103 +## Gating: `VLLM_PLUGINS` and `required_tasks`
104 +
105 +Endpoint plugins are discovered and gated by [`load_endpoint_plugins`][vllm.plugins.load_endpoint_plugins] which is stricter than the loader used for other plugin groups:
106 +
107 +- **Nothing loads unless `VLLM_PLUGINS` is set and names the plugin.** Other plugin groups load everything unless `VLLM_PLUGINS` narrows the set. Endpoint plugins invert that default because they add network exposed surface. See [Security](../usage/security.md#endpoint-plugins).
108 +- **`required_tasks` must intersect the server's supported tasks** unless it is `None`. Use this to keep a plugin from attaching routes on a server that can't service them (e.g. a pooling only deployment).
109 +- A factory that raises an issue during instantiation is logged and skipped. It does not abort server startup.
110 +
111 +Only the front end API server process loads endpoint plugins. There is no need to guard for worker or engine core processes.
112 +
113 +## Pairing with `vllm.general_plugins`
114 +
115 +Endpoint plugins cover the HTTP surface only. If a plugin also needs new engine side behavior (a new worker-side RPC method, a custom stat) that half ships separately through the existing `vllm.general_plugins` group which loads in worker processes (see [Plugin System](plugin_system.md)). The two entry points are registered and loaded **independently**. Neither implies the other. The recommended distribution shape is a single package exposing both:
116 +
117 +```toml
118 +[project.entry-points."vllm.general_plugins"]
119 +my_admin_engine = "my_pkg.engine:register" # adds the worker side method
120 +
121 +[project.entry-points."vllm.endpoint_plugins"]
122 +my_admin_api = "my_pkg.endpoints:MyAdminEndpointPlugin" # adds the HTTP route
123 +```
124 +
125 +Do not expect a single endpoint plugin to also mutate engine/worker state. If your route needs a worker side method that doesn't already exist then add it via a paired `general_plugins` entry point.
126 +
127 +## Path-prefix convention
128 +
129 +There is currently no route conflict enforcement (tracked as a follow-up to RFC [#46565](https://github.com/vllm-project/vllm/issues/46565)). A plugin's `attach_router` can register a path that collides with a core route and routes attached later win. To avoid surprising operators:
130 +
131 +- Namespace your routes under a distinct prefix, e.g. `/plugins/<plugin-name>/...`, rather than reusing `/v1/...` or other core prefixes
132 +- Only register routes under a core prefix (like the worked example's `/v1/admin/scheduler_config`) if you specifically intend to override or extend existing behavior and document that clearly for operators allowlisting your plugin
133 +
134 +## Compatibility
135 +
136 +`state`/serving handler internals (e.g. the shape of in-tree `OpenAIServing*` classes) are not a stable public contract yet. Treat them as use-at-your-own-risk and expect them to change between vLLM versions. `FastAPI`, `EngineClient` and the `EndpointPlugin` protocol itself are the supported surface.
docs/design/plugin_system.md
+2
@@ -53,6 +53,8 @@ Every plugin has three parts:
53
54 - **Stat logger plugins** (with group name `vllm.stat_logger_plugins`): The primary use case for these plugins is to register custom, out-of-the-tree loggers into vLLM. The entry point should be a class that subclasses StatLoggerBase.
55
56 +- **Endpoint plugins** (with group name `vllm.endpoint_plugins`): The primary use case for these plugins is to register custom, out-of-the-tree HTTP routes on the OpenAI compatible API server. Unlike the other plugin groups above, endpoint plugins are loaded only in the API server front end process and are **not loaded by default**. See [Endpoint Plugins](endpoint_plugins.md) for the interface and [Security](../usage/security.md#endpoint-plugins) for the opt-in and trust model.
57 +
58 ## Guidelines for Writing Plugins
59
60 - **Being re-entrant**: The function specified in the entry point should be re-entrant, meaning it can be called multiple times without causing issues. This is necessary because the function might be called multiple times in some processes.
docs/usage/security.md
+13
@@ -326,6 +326,19 @@ vLLM supports dynamically loading and unloading LoRA adapters at runtime via the
326
327 **Warning:** Dynamic LoRA loading is not a secure operation and should not be enabled in deployments exposed to untrusted clients. If you must enable dynamic LoRA loading, restrict access to the `/v1/load_lora_adapter` and `/v1/unload_lora_adapter` endpoints to trusted administrators only, using a reverse proxy or network-level access controls. Do not expose these endpoints to end users. For details on configuring LoRA adapters, see the [LoRA Adapters documentation](../features/lora.md).
328
329 +## Endpoint Plugins
330 +
331 +vLLM supports loading out-of-tree HTTP routes via the `vllm.endpoint_plugins` entry point group (see [Endpoint Plugins](../design/endpoint_plugins.md) for how to write one). An endpoint plugin can register arbitrary FastAPI routes, including routes that reach the engine via `EngineClient.collective_rpc`, so it must be treated as part of the server's trusted code base and not as sandboxed or reviewed input.
332 +
333 +**Endpoint plugins are not loaded by default.** Unlike other vLLM plugin groups (`vllm.general_plugins`, `vllm.platform_plugins`, etc.), which load every discovered plugin unless `VLLM_PLUGINS` narrows the set, endpoint plugins load **none** unless `VLLM_PLUGINS` is set and explicitly names them. This mirrors the "off by default in production" posture used for development endpoints gated behind `VLLM_SERVER_DEV_MODE`. Both surfaces are only present when an operator has explicitly opted in.
334 +
335 +### Recommended Security Practices
336 +
337 +1. **Only allowlist plugins you trust.** Set `VLLM_PLUGINS` to the exact plugin names you intend to run and never wildcard or copy an allowlist between deployments without reviewing what each named plugin does.
338 +2. **Audit routes before deploying.** A plugin's `attach_router` can add routes under any path, including ones that duplicate existing `/v1/*` paths. There is currently no route conflict enforcement (tracked as a follow-up to RFC [#46565](https://github.com/vllm-project/vllm/issues/46565)), so a malicious or buggy plugin can **shadow a core route** and silently replace its behavior. Prefer plugins that namespace their routes under a distinct prefix (e.g. `/plugins/<plugin-name>/...`) instead of reusing `/v1/...` and review `app.routes` after startup if you need certainty about what is actually being served.
339 +3. **Treat plugin routes like any other unauthenticated by default surface.** `--api-key` only protects the `/v1`, `/v2`, and `/inference` path prefixes (see [API Key Authentication Limitations](#api-key-authentication-limitations)). A plugin route outside those prefixes is unauthenticated unless the plugin implements its own authentication. Deploy behind a reverse proxy that allowlists only the plugin routes you intend to expose externally.
340 +4. **Remember the `vllm.general_plugins` pairing.** A plugin that also needs new engine side behavior ships that half separately via `vllm.general_plugins` which loads in every worker process under the default (load all unless restricted) posture. Allowlisting the endpoint plugin does not by itself restrict its paired engine side plugin. Need to review both.
341 +
342 ## gRPC Interface
343
344 vLLM provides an optional gRPC Generate service on a separate TCP port, enabled via the `--grpc-port` flag. When not specified, no gRPC server is started. The gRPC listener binds to the same host address as the HTTP server.
tests/plugins/vllm_add_dummy_endpoint_plugin/setup.py new
+15
@@ -0,0 +1,15 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +
4 +from setuptools import setup
5 +
6 +setup(
7 + name="vllm_add_dummy_endpoint_plugin",
8 + version="0.1",
9 + packages=["vllm_add_dummy_endpoint_plugin"],
10 + entry_points={
11 + "vllm.endpoint_plugins": [
12 + "dummy_admin_endpoint_plugin = vllm_add_dummy_endpoint_plugin:DummyAdminEndpointPlugin" # noqa
13 + ]
14 + },
15 +)
tests/plugins/vllm_add_dummy_endpoint_plugin/vllm_add_dummy_endpoint_plugin/__init__.py new
+37
@@ -0,0 +1,37 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +"""Worked example `vllm.endpoint_plugins` entry point.
4 +
5 +Reports scheduler config via `collective_rpc`. Demonstrates the full
6 +contract: `attach_router` registers the route at Phase A (`build_app`) and
7 +`init_state` stashes the `EngineClient` the route handler needs at Phase B
8 +(`init_app_state`).
9 +
10 +`required_tasks` is `None`, so this plugin is also eligible on the CPU only
11 +render server which has no `EngineClient`. `init_state` is called with
12 +`engine_client=None` in that case and the route handler returns 503 rather
13 +than reaching for a client that doesn't exist.
14 +"""
15 +
16 +from fastapi import FastAPI, HTTPException, Request
17 +
18 +
19 +class DummyAdminEndpointPlugin:
20 + name = "dummy_admin_endpoint_plugin"
21 + required_tasks: tuple[str, ...] | None = None
22 +
23 + def attach_router(self, app: FastAPI) -> None:
24 + @app.get("/v1/admin/scheduler_config")
25 + async def scheduler_config(raw_request: Request):
26 + engine_client = raw_request.app.state.dummy_engine_client
27 + if engine_client is None:
28 + raise HTTPException(
29 + status_code=503,
30 + detail="scheduler_config requires an engine, which this "
31 + "server does not have",
32 + )
33 + results = await engine_client.collective_rpc("get_scheduler_config")
34 + return {"scheduler_config": results}
35 +
36 + async def init_state(self, engine_client, state, args) -> None:
37 + state.dummy_engine_client = engine_client
tests/plugins_tests/test_endpoint_plugins.py new
+236
@@ -0,0 +1,236 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +"""Tests for the `vllm.endpoint_plugins` framework (RFC #46565).
4 +
5 +Uses the worked in repo example plugin (`vllm_add_dummy_endpoint_plugin`,
6 +installed via `tests/plugins/vllm_add_dummy_endpoint_plugin`) exercising both
7 +`EndpointPlugin` hooks against a fake `EngineClient`, unit tests for the
8 +`load_endpoint_plugins` gating matrix and an e2e test that drives a real HTTP
9 +request through the plugin's route.
10 +"""
11 +
12 +from argparse import Namespace
13 +from typing import Any
14 +
15 +import httpx
16 +import pytest
17 +from fastapi import FastAPI
18 +from vllm_add_dummy_endpoint_plugin import DummyAdminEndpointPlugin
19 +
20 +from vllm.entrypoints.openai.api_server import (
21 + _attach_endpoint_plugins,
22 + _init_endpoint_plugins_state,
23 + build_app,
24 +)
25 +from vllm.entrypoints.openai.cli_args import make_arg_parser
26 +from vllm.plugins import load_endpoint_plugins
27 +from vllm.plugins.endpoint_plugins.interface import EndpointPlugin
28 +from vllm.utils.argparse_utils import FlexibleArgumentParser
29 +
30 +
31 +class _RaisingEndpointPlugin:
32 + """Factory that raises to exercise the "instantiation fails" path."""
33 +
34 + name = "raising_endpoint_plugin"
35 + required_tasks = None
36 +
37 + def __init__(self):
38 + raise RuntimeError("boom")
39 +
40 +
41 +class _FakeEngineClient:
42 + """Minimal stand in exercising `collective_rpc`. Not a real engine."""
43 +
44 + def __init__(self, rpc_result: Any = None):
45 + self.rpc_result = rpc_result
46 + self.rpc_calls: list[tuple[str, tuple, dict]] = []
47 +
48 + async def collective_rpc(self, method, timeout=None, args=(), kwargs=None):
49 + self.rpc_calls.append((method, args, kwargs or {}))
50 + return self.rpc_result
51 +
52 +
53 +def _build_args() -> Namespace:
54 + parser = FlexibleArgumentParser()
55 + subparsers = parser.add_subparsers()
56 + serve_parser = subparsers.add_parser("serve")
57 + make_arg_parser(serve_parser)
58 + return serve_parser.parse_args([])
59 +
60 +
61 +def _fake_loader(factories: dict[str, Any]):
62 + def _load_plugins_by_group(group: str) -> dict[str, Any]:
63 + assert group == "vllm.endpoint_plugins"
64 + return factories
65 +
66 + return _load_plugins_by_group
67 +
68 +
69 +def test_dummy_plugin_satisfies_protocol():
70 + assert isinstance(DummyAdminEndpointPlugin(), EndpointPlugin)
71 +
72 +
73 +def test_no_plugins_loaded_when_allowlist_unset(monkeypatch: pytest.MonkeyPatch):
74 + monkeypatch.delenv("VLLM_PLUGINS", raising=False)
75 +
76 + assert load_endpoint_plugins(("generate",)) == []
77 +
78 +
79 +def test_no_plugins_loaded_when_allowlist_is_empty_string(
80 + monkeypatch: pytest.MonkeyPatch,
81 +):
82 + """`VLLM_PLUGINS=""` parses to `[""]`, not `None` (see `vllm.envs`), so it
83 + must be treated as a (non strict) allowlist matching no plugin name, not
84 + as "unset"."""
85 + monkeypatch.setenv("VLLM_PLUGINS", "")
86 +
87 + assert load_endpoint_plugins(("generate",)) == []
88 +
89 +
90 +def test_plugin_loaded_when_allowlisted_and_task_matches(
91 + monkeypatch: pytest.MonkeyPatch,
92 +):
93 + monkeypatch.setenv("VLLM_PLUGINS", "dummy_admin_endpoint_plugin")
94 +
95 + plugins = load_endpoint_plugins(("generate",))
96 +
97 + assert len(plugins) == 1
98 + assert isinstance(plugins[0], DummyAdminEndpointPlugin)
99 +
100 +
101 +def test_plugin_skipped_when_required_tasks_miss(monkeypatch: pytest.MonkeyPatch):
102 + class _GenerateOnlyPlugin(DummyAdminEndpointPlugin):
103 + required_tasks = ("generate",)
104 +
105 + monkeypatch.setenv("VLLM_PLUGINS", "dummy_admin_endpoint_plugin")
106 + monkeypatch.setattr(
107 + "vllm.plugins.load_plugins_by_group",
108 + _fake_loader({"dummy_admin_endpoint_plugin": _GenerateOnlyPlugin}),
109 + )
110 +
111 + assert load_endpoint_plugins(("embed",)) == []
112 + assert len(load_endpoint_plugins(("generate",))) == 1
113 +
114 +
115 +def test_plugin_loaded_when_required_tasks_is_none(monkeypatch: pytest.MonkeyPatch):
116 + monkeypatch.setenv("VLLM_PLUGINS", "dummy_admin_endpoint_plugin")
117 +
118 + assert len(load_endpoint_plugins(supported_tasks=None)) == 1
119 +
120 +
121 +def test_plugin_skipped_when_required_tasks_set_but_supported_tasks_none(
122 + monkeypatch: pytest.MonkeyPatch,
123 +):
124 + class _GenerateOnlyPlugin(DummyAdminEndpointPlugin):
125 + required_tasks = ("generate",)
126 +
127 + monkeypatch.setenv("VLLM_PLUGINS", "dummy_admin_endpoint_plugin")
128 + monkeypatch.setattr(
129 + "vllm.plugins.load_plugins_by_group",
130 + _fake_loader({"dummy_admin_endpoint_plugin": _GenerateOnlyPlugin}),
131 + )
132 +
133 + assert load_endpoint_plugins(supported_tasks=None) == []
134 +
135 +
136 +def test_factory_raising_is_logged_and_skipped(monkeypatch: pytest.MonkeyPatch):
137 + monkeypatch.setenv(
138 + "VLLM_PLUGINS", "raising_endpoint_plugin,dummy_admin_endpoint_plugin"
139 + )
140 + monkeypatch.setattr(
141 + "vllm.plugins.load_plugins_by_group",
142 + _fake_loader(
143 + {
144 + "raising_endpoint_plugin": _RaisingEndpointPlugin,
145 + "dummy_admin_endpoint_plugin": DummyAdminEndpointPlugin,
146 + }
147 + ),
148 + )
149 +
150 + plugins = load_endpoint_plugins(("generate",))
151 +
152 + assert len(plugins) == 1
153 + assert isinstance(plugins[0], DummyAdminEndpointPlugin)
154 +
155 +
156 +def test_attach_is_noop_when_nothing_discovered(monkeypatch: pytest.MonkeyPatch):
157 + monkeypatch.delenv("VLLM_PLUGINS", raising=False)
158 +
159 + app = FastAPI()
160 + _attach_endpoint_plugins(app, ("generate",))
161 +
162 + assert app.state.endpoint_plugins == []
163 +
164 +
165 +@pytest.mark.asyncio
166 +async def test_init_state_is_noop_without_phase_a(monkeypatch: pytest.MonkeyPatch):
167 + """`init_app_state` callers that never ran `build_app` (e.g.
168 + `run_batch.py`, which builds a bare `State()`) must not crash just
169 + because `state.endpoint_plugins` was never set."""
170 + from starlette.datastructures import State
171 +
172 + monkeypatch.setenv("VLLM_PLUGINS", "dummy_admin_endpoint_plugin")
173 +
174 + state = State()
175 + await _init_endpoint_plugins_state(_FakeEngineClient(), state, _build_args())
176 +
177 + assert not hasattr(state, "dummy_engine_client")
178 +
179 +
180 +@pytest.mark.asyncio
181 +async def test_render_server_attaches_endpoint_plugins_with_no_engine_client(
182 + monkeypatch: pytest.MonkeyPatch,
183 +):
184 + """The CPU only render server has no `EngineClient` but a plugin eligible
185 + for the `render` task (`required_tasks` is `None` or includes `"render"`)
186 + still gets its routes attached at Phase A. Phase B passes `None` for
187 + `engine_client` and it's up to the plugin to handle that."""
188 + monkeypatch.setenv("VLLM_PLUGINS", "dummy_admin_endpoint_plugin")
189 +
190 + args = _build_args()
191 + app = build_app(args, ("render",))
192 +
193 + assert len(app.state.endpoint_plugins) == 1
194 + assert any(
195 + getattr(route, "path", None) == "/v1/admin/scheduler_config"
196 + for route in app.routes
197 + )
198 +
199 + await _init_endpoint_plugins_state(None, app.state, args)
200 +
201 + assert app.state.dummy_engine_client is None
202 +
203 + transport = httpx.ASGITransport(app=app)
204 + async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
205 + response = await client.get("/v1/admin/scheduler_config")
206 +
207 + assert response.status_code == 503
208 +
209 +
210 +@pytest.mark.asyncio
211 +async def test_endpoint_plugin_end_to_end(monkeypatch: pytest.MonkeyPatch):
212 + """Phase A (attach) + Phase B (init) wired through `build_app` then
213 + exercised with a real HTTP request against the worked example plugin."""
214 + monkeypatch.setenv("VLLM_PLUGINS", "dummy_admin_endpoint_plugin")
215 +
216 + args = _build_args()
217 + app = build_app(args, supported_tasks=())
218 +
219 + assert len(app.state.endpoint_plugins) == 1
220 + assert any(
221 + getattr(route, "path", None) == "/v1/admin/scheduler_config"
222 + for route in app.routes
223 + )
224 +
225 + fake_engine_client = _FakeEngineClient(rpc_result=["cfg-a", "cfg-b"])
226 + await _init_endpoint_plugins_state(fake_engine_client, app.state, args)
227 +
228 + assert app.state.dummy_engine_client is fake_engine_client
229 +
230 + transport = httpx.ASGITransport(app=app)
231 + async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
232 + response = await client.get("/v1/admin/scheduler_config")
233 +
234 + assert response.status_code == 200
235 + assert response.json() == {"scheduler_config": ["cfg-a", "cfg-b"]}
236 + assert fake_engine_client.rpc_calls == [("get_scheduler_config", (), {})]
vllm/entrypoints/openai/api_server.py
+47
@@ -74,6 +74,41 @@ logger = init_logger("vllm.entrypoints.openai.api_server")
74 _FALLBACK_SUPPORTED_TASKS: tuple[SupportedTask, ...] = ("generate",)
75
76
77 +def _attach_endpoint_plugins(
78 + app: FastAPI, supported_tasks: tuple["SupportedTask", ...]
79 +) -> None:
80 + """Phase A of endpoint plugin wiring: discover, gate and attach routes.
81 +
82 + Attached last after all core routers. This is so endpoint plugin routes can
83 + shadow core routes with the same path (see `EndpointPlugin.attach_router`
84 + docstring). No-ops when no plugins are discovered/allowlisted.
85 + """
86 + from vllm.plugins import load_endpoint_plugins
87 +
88 + endpoint_plugins = load_endpoint_plugins(supported_tasks)
89 + for plugin in endpoint_plugins:
90 + plugin.attach_router(app)
91 + app.state.endpoint_plugins = endpoint_plugins
92 +
93 +
94 +async def _init_endpoint_plugins_state(
95 + engine_client: EngineClient | None, state: State, args: Namespace
96 +) -> None:
97 + """Phase B of endpoint plugin wiring: initialize per app plugin state.
98 +
99 + `state.endpoint_plugins` is set by `_attach_endpoint_plugins` (Phase A)
100 + in `build_app`. Some `init_app_state` callers (e.g. `run_batch.py`)
101 + build their own bare `State` without going through `build_app`. As a result
102 + `endpoint_plugins` may be absent and are treated that the same as "none attached".
103 +
104 + `engine_client` is `None` for the CPU only render server which has no
105 + engine (see `init_render_app_state`). Plugins must handle a `None`
106 + `engine_client` themselves (see `EndpointPlugin.init_state`).
107 + """
108 + for plugin in getattr(state, "endpoint_plugins", []):
109 + await plugin.init_state(engine_client, state, args)
110 +
111 +
112 @asynccontextmanager
113 async def build_async_engine_client(
114 args: Namespace,
@@ -230,6 +265,12 @@ def build_app(
265
266 register_pooling_api_routers(app, supported_tasks, model_config)
267
268 + # Endpoint plugins are attached last so their routes are registered after all core
269 + # routers. This runs even for the CPU only render server. A plugin eligible for
270 + # the `render` task still gets its routes registered. It receives
271 + # `engine_client=None` at Phase B (see `_init_endpoint_plugins_state`).
272 + _attach_endpoint_plugins(app, supported_tasks)
273 +
274 app.root_path = args.root_path
275 app.add_middleware(
276 CORSMiddleware,
@@ -417,6 +458,8 @@ async def init_app_state(
458
459 init_pooling_state(engine_client, state, args, request_logger, supported_tasks)
460
461 + await _init_endpoint_plugins_state(engine_client, state, args)
462 +
463 state.enable_server_load_tracking = args.enable_server_load_tracking
464 state.server_load_metrics = 0
465
@@ -508,6 +551,10 @@ async def init_render_app_state(
551 state.enable_server_load_tracking = False
552 state.server_load_metrics = 0
553
554 + # No `EngineClient` exists for the render server, so plugins get `None` and
555 + # must handle it themselves (see `EndpointPlugin.init_state`).
556 + await _init_endpoint_plugins_state(None, state, args)
557 +
558
559 def create_server_socket(
560 addr: tuple[str, int],
vllm/plugins/__init__.py
+77 -1
@@ -3,10 +3,14 @@
3
4 import logging
5 from collections.abc import Callable
6 -from typing import Any
6 +from typing import TYPE_CHECKING, Any
7
8 import vllm.envs as envs
9
10 +if TYPE_CHECKING:
11 + from vllm.plugins.endpoint_plugins.interface import EndpointPlugin
12 + from vllm.tasks import SupportedTask
13 +
14 logger = logging.getLogger(__name__)
15
16 # Default plugins group will be loaded in all processes(process0, engine core
@@ -20,6 +24,10 @@ PLATFORM_PLUGINS_GROUP = "vllm.platform_plugins"
24 # Stat logger plugins group will be loaded in process0 only when serve vLLM with
25 # async mode.
26 STAT_LOGGER_PLUGINS_GROUP = "vllm.stat_logger_plugins"
27 +# Endpoint plugins group is loaded in the API server front end process only.
28 +# Each entry point resolves to a factory returning an `EndpointPlugin`
29 +# (see `vllm/plugins/endpoint_plugins/interface.py`).
30 +ENDPOINT_PLUGINS_GROUP = "vllm.endpoint_plugins"
31
32 # make sure one process only loads plugins once
33 plugins_loaded = False
@@ -80,3 +88,71 @@ def load_general_plugins():
88 # general plugins, we only need to execute the loaded functions
89 for func in plugins.values():
90 func()
91 +
92 +
93 +def load_endpoint_plugins(
94 + supported_tasks: "tuple[SupportedTask, ...] | None" = None,
95 +) -> "list[EndpointPlugin]":
96 + """Discover, gate and instantiate `vllm.endpoint_plugins` entry points.
97 +
98 + Endpoint plugins add HTTP routes to the API server, so they default to
99 + not loading. Unlike other plugin groups, a plugin here is only
100 + considered when it is explicitly named in `VLLM_PLUGINS`. This is a
101 + stricter posture than `load_plugins_by_group` which "load everything unless
102 + an allowlist says otherwise". This posture is taken to handle potentially
103 + larger exposed network surface.
104 +
105 + A discovered plugin is loaded only if both hold:
106 + - it is named in `VLLM_PLUGINS` (enforced by not calling the loader
107 + at all when `VLLM_PLUGINS` is unset). Note that `VLLM_PLUGINS=""`
108 + parses to `[""]`, not `None`, so it is treated as a (non strict)
109 + allowlist that matches no plugin name, not as "unset".
110 + - its `required_tasks` is `None` or intersects `supported_tasks`.
111 +
112 + Args:
113 + supported_tasks: Tasks the server supports. `None` means no plugin
114 + with a non `None` `required_tasks` will be loaded.
115 +
116 + Returns:
117 + Instantiated plugins that passed gating in discovery order.
118 + """
119 + from importlib.metadata import entry_points
120 +
121 + if envs.VLLM_PLUGINS is None:
122 + discovered = entry_points(group=ENDPOINT_PLUGINS_GROUP)
123 + if discovered:
124 + logger.warning(
125 + "Found endpoint plugin(s) %s but VLLM_PLUGINS is not set. "
126 + "Endpoint plugins add HTTP routes and must be explicitly "
127 + "allowlisted via VLLM_PLUGINS to be loaded.",
128 + [p.name for p in discovered],
129 + )
130 + return []
131 +
132 + factories = load_plugins_by_group(ENDPOINT_PLUGINS_GROUP)
133 +
134 + endpoint_plugins: list[EndpointPlugin] = []
135 + for name, factory in factories.items():
136 + try:
137 + plugin = factory()
138 + except Exception:
139 + logger.exception("Failed to instantiate endpoint plugin %s", name)
140 + continue
141 +
142 + required_tasks = plugin.required_tasks
143 + if required_tasks is not None and (
144 + supported_tasks is None or not set(required_tasks) & set(supported_tasks)
145 + ):
146 + logger.info(
147 + "Skipping endpoint plugin %s: requires one of tasks %s, "
148 + "server supports %s",
149 + name,
150 + required_tasks,
151 + supported_tasks,
152 + )
153 + continue
154 +
155 + logger.info("Loaded endpoint plugin %s", name)
156 + endpoint_plugins.append(plugin)
157 +
158 + return endpoint_plugins
vllm/plugins/endpoint_plugins/__init__.py
vllm/plugins/endpoint_plugins/interface.py new
+87
@@ -0,0 +1,87 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +"""Contract for `vllm.endpoint_plugins` entry points.
4 +
5 +An endpoint plugin adds HTTP routes to the OpenAI compatible API server.
6 +Its scope is HTTP surface only. It registers routes and optionally
7 +per app state used by those routes. It must not open new paths into the
8 +engine by reaching the engine the same way an in-tree serving handler does
9 +via `EngineClient` (e.g. `engine_client.collective_rpc(...)`).
10 +
11 +If a plugin also needs engine side behavior (a new worker side RPC method,
12 +a custom stat, etc.) pair this entry point with one registered under
13 +`vllm.general_plugins` (see `vllm/plugins/__init__.py`). The
14 +`general_plugins` entry installs the engine side method and the
15 +`endpoint_plugins` entry exposes it over HTTP. The two are registered and
16 +loaded independently where neither implies the other.
17 +
18 +Plugins are opt-in. See `load_endpoint_plugins` in `vllm/plugins/__init__.py`
19 +for the loading/gating rules and `docs/usage/security.md` for the security
20 +posture of exposing plugin defined routes.
21 +
22 +The CPU only render server (see `build_and_serve_renderer` in
23 +`vllm/entrypoints/openai/api_server.py`) has no `EngineClient`. A plugin
24 +eligible for the `render` task (`required_tasks` is `None` or includes
25 +`"render"`) still gets `attach_router` called but `init_state` receives
26 +`engine_client=None`. Plugins that cannot function without an engine should
27 +either exclude `"render"` from `required_tasks` or check for `None` in
28 +`init_state`/their route handlers and degrade gracefully.
29 +"""
30 +
31 +from argparse import Namespace
32 +from typing import TYPE_CHECKING, Protocol, runtime_checkable
33 +
34 +from fastapi import FastAPI
35 +from starlette.datastructures import State
36 +
37 +if TYPE_CHECKING:
38 + from vllm.engine.protocol import EngineClient
39 + from vllm.tasks import SupportedTask
40 +
41 +
42 +@runtime_checkable
43 +class EndpointPlugin(Protocol):
44 + """Protocol implemented by `vllm.endpoint_plugins` entry point factories.
45 +
46 + An entry point registered under the `vllm.endpoint_plugins` group must
47 + resolve to a zero argument callable (a class or factory function) that
48 + returns an object satisfying this protocol.
49 + """
50 +
51 + name: str
52 + """Unique plugin name used in logs and for `VLLM_PLUGINS` allowlisting."""
53 +
54 + required_tasks: "tuple[SupportedTask, ...] | None"
55 + """Tasks the server must support for this plugin to be loaded.
56 +
57 + The plugin is loaded only if this set intersects the server's
58 + `supported_tasks`. `None` means the plugin has no task requirement and
59 + is always eligible (subject to the `VLLM_PLUGINS` allowlist).
60 + """
61 +
62 + def attach_router(self, app: FastAPI) -> None:
63 + """Register this plugin's routes on `app`.
64 +
65 + Called once during `build_app()` after all core routers have been
66 + attached. Routes attached here can shadow core routes with the same
67 + path. There is currently no conflict enforcement (see RFC #46565 follow ups).
68 + """
69 + ...
70 +
71 + async def init_state(
72 + self, engine_client: "EngineClient | None", state: State, args: Namespace
73 + ) -> None:
74 + """Initialize per app state consumed by this plugin's routes.
75 +
76 + Called once during `init_app_state()` after core state has been
77 + initialized. Use `engine_client` (e.g. `collective_rpc`) to reach
78 + the engine. Do not open new engine access paths.
79 +
80 + `engine_client` is `None` on the CPU only render server which has
81 + no engine. This only happens for plugins eligible for the `render`
82 + task (`required_tasks` is `None` or includes `"render"`). Handle
83 + `None` explicitly (e.g. skip engine dependent setup, or have route
84 + handlers return an error) if the plugin is loadable for `render` but
85 + cannot function without an engine.
86 + """
87 + ...