[Frontend] Gate scale-out endpoints behind opt-in flag (#54579)
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
Francisco Javier Arceo committed
Sep 2, 2026 at 02:31 UTC
c00091e026709b149d80246a18928f514871c160
20 files changed
+386
-39
docs/serving/online_serving/README.md
+7
@@ -121,6 +121,13 @@ For further details on profiling vLLM, please refer to [this page](../../contrib
121
122
## Scale-Out APIs
123
124
+Scale-out APIs are disabled by default on `vllm serve`. The environment
125
+variable accepts only `0` or `1`; set `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1` to
126
+register the endpoints below. The
127
+dedicated `vllm launch render` and `vllm serve --tokens-only` modes are explicit
128
+opt-ins and enable their required endpoints when the variable is unset; an
129
+explicit value of `0` is rejected for those modes.
130
+
131
### Tokens IN <> Tokens OUT APIs
132
133
- `/inference/v1/generate` - Generate completions
docs/serving/online_serving/derenderer.md
+8
@@ -56,6 +56,14 @@ Oversized payloads are rejected with a `400` before any `tokenizer.decode()` or
56
57
The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
58
59
+Launch the two servers first:
60
+
61
+```bash
62
+vllm launch render meta-llama/Llama-3.2-1B-Instruct --port 8100
63
+VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve \
64
+ meta-llama/Llama-3.2-1B-Instruct --tokens-only --port 8200
65
+```
66
+
67
```python
68
import httpx
69
docs/serving/online_serving/renderer.md
+13
@@ -6,6 +6,19 @@ Our renderer API is designed to disaggregate the render phase(preprocessing) and
6
- Disaggregated tokenization: Support use cases such as llm-d, Dynamo, and custom frontends that need to leverage vLLM's preprocessing logic without running the full inference engine.
7
- Tokens-in / tokens-out engine: Make the engine a pure token-in / token-out service, decoupled from request preprocessing.
8
9
+The dedicated `vllm launch render` server always exposes the `/render` and
10
+`/derender` endpoints when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS` is unset or set to
11
+`1`. An explicit value of `0` conflicts with the renderer command and is
12
+rejected at startup.
13
+
14
+Scale-out endpoints, including `/render`, `/derender`, and
15
+`/inference/v1/generate`, are disabled by default on a standard inference
16
+server. To expose them with `vllm serve`, opt in explicitly:
17
+
18
+```bash
19
+VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve <model>
20
+```
21
+
22
## API Reference
23
24
- [Completions Render API](renderer.md) (`/v1/completions/render`)
docs/usage/security.md
+9
-7
@@ -158,15 +158,16 @@ When `--api-key` is configured, the following `/v1` endpoints require Bearer tok
158
- `/v1/models` - List available models
159
- `/v1/chat/completions` - Chat completions
160
- `/v1/chat/completions/batch` - Batch chat completions
161
-- `/v1/chat/completions/render` - Render chat completion requests
162
-- `/v1/chat/completions/derender` - Derender chat completion requests
161
+- `/v1/chat/completions/render` - Render chat completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
162
+- `/v1/chat/completions/derender` - Derender chat completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
163
- `/v1/completions` - Text completions
164
-- `/v1/completions/render` - Render completion requests
165
-- `/v1/completions/derender` - Derender completion requests
164
+- `/v1/completions/render` - Render completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
165
+- `/v1/completions/derender` - Derender completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
166
- `/v1/embeddings` - Generate embeddings
167
- `/v1/audio/transcriptions` - Audio transcription
168
- `/v1/audio/translations` - Audio translation
169
- `/v1/messages` - Anthropic-compatible messages API
170
+- `/v1/messages/render` - Render Anthropic-compatible messages (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
171
- `/v1/messages/count_tokens` - Count tokens for Anthropic messages
172
- `/v1/responses` - Create a response
173
- `/v1/responses/{response_id}` - Retrieve a response
@@ -175,7 +176,8 @@ When `--api-key` is configured, the following `/v1` endpoints require Bearer tok
176
- `/v1/rerank` - Reranking API
177
- `/v1/load_lora_adapter` - Load a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
178
- `/v1/unload_lora_adapter` - Unload a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
178
-- `/inference/v1/generate` - Generate completions
179
+- `/inference/v1/generate` - Generate completions (available when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or with `--tokens-only` unless explicitly disabled)
180
+- `/cohere/v2/chat/render` - Render Cohere Chat v2 requests (requires both `VLLM_ENABLE_COHERE_API=1` and `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`)
181
- `/v2/embed` - Cohere Embed API
182
- `/v2/rerank` - Cohere Rerank API
183
@@ -203,11 +205,11 @@ The following endpoints **do not require authentication** even when `--api-key`
205
- `/init_weight_transfer_engine` - Initialize weight transfer engine for RLHF
206
- `/update_weights` - Update model weights (can alter model behavior)
207
- `/get_world_size` - Get distributed world size
206
-- `/abort_requests` - Abort in-flight requests (only when `--tokens-only` is also set)
208
+- `/abort_requests` - Abort in-flight requests (available with `--tokens-only` unless `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=0`)
209
210
**Utility endpoints:**
211
210
-- `/tokenize` - Tokenize text
212
+- `/tokenize` - Tokenize text (not disabled by `VLLM_ENABLE_SCALE_OUT_ENDPOINTS`)
213
- `/detokenize` - Detokenize tokens
214
- `/health` - Health check
215
- `/ping` - SageMaker health check
examples/scale_out/example_mm_serve.py
+2
-1
@@ -13,7 +13,8 @@ zero client-side transformation.
13
14
Launch the server first:
15
16
- vllm serve Qwen/Qwen3-VL-2B-Instruct \
16
+ VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 \
17
+ vllm serve Qwen/Qwen3-VL-2B-Instruct \
18
--dtype bfloat16 --max-model-len 4096 --enforce-eager
19
20
Then run this script:
examples/scale_out/token_generation_client.py
+5
@@ -1,5 +1,10 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+"""Call a token-in/token-out server started with:
4
+
5
+VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve Qwen/Qwen3-0.6B
6
+"""
7
+
8
import httpx
9
from transformers import AutoTokenizer
10
rust/src/server/src/routes.rs
+42
-4
@@ -26,7 +26,7 @@ use axum::extract::DefaultBodyLimit;
26
use axum::middleware::{from_fn, from_fn_with_state};
27
use axum::routing::{get, post};
28
use tower_http::trace::TraceLayer;
29
-use tracing::warn;
29
+use tracing::{info, warn};
30
31
use crate::DEFAULT_REQUEST_BODY_LIMIT_BYTES;
32
use crate::middleware;
@@ -45,12 +45,33 @@ fn runtime_lora_updating_enabled() -> bool {
45
.is_some_and(|value| matches!(value.trim().to_lowercase().as_str(), "1" | "true"))
46
}
47
48
+fn parse_scale_out_endpoints_flag(value: Option<&str>) -> Result<bool, String> {
49
+ let Some(value) = value else {
50
+ return Ok(false);
51
+ };
52
+ if value.is_empty() {
53
+ return Ok(false);
54
+ }
55
+
56
+ match value.trim() {
57
+ "0" => Ok(false),
58
+ "1" => Ok(true),
59
+ _ => Err("VLLM_ENABLE_SCALE_OUT_ENDPOINTS must be 0 or 1".to_string()),
60
+ }
61
+}
62
+
63
+fn scale_out_endpoints_enabled() -> bool {
64
+ let value = std::env::var("VLLM_ENABLE_SCALE_OUT_ENDPOINTS").ok();
65
+ parse_scale_out_endpoints_flag(value.as_deref()).unwrap_or_else(|message| panic!("{message}"))
66
+}
67
+
68
/// Build the minimal OpenAI-compatible router for one configured model.
69
pub fn build_router(state: Arc<AppState>) -> Router {
70
build_router_with_options(
71
state,
72
server_dev_mode_enabled(),
73
runtime_lora_updating_enabled(),
74
+ scale_out_endpoints_enabled(),
75
)
76
}
77
@@ -65,13 +86,22 @@ fn build_router_with_dev_mode_and_lora(
86
dev_mode_enabled: bool,
87
runtime_lora_updating_enabled: bool,
88
) -> Router {
68
- build_router_with_options(state, dev_mode_enabled, runtime_lora_updating_enabled)
89
+ build_router_with_options(state, dev_mode_enabled, runtime_lora_updating_enabled, true)
90
+}
91
+
92
+#[cfg(test)]
93
+fn build_router_with_scale_out_endpoints(
94
+ state: Arc<AppState>,
95
+ scale_out_endpoints_enabled: bool,
96
+) -> Router {
97
+ build_router_with_options(state, false, false, scale_out_endpoints_enabled)
98
}
99
100
fn build_router_with_options(
101
state: Arc<AppState>,
102
dev_mode_enabled: bool,
103
runtime_lora_updating_enabled: bool,
104
+ scale_out_endpoints_enabled: bool,
105
) -> Router {
106
let mut router = Router::new()
107
// Health & monitoring
@@ -85,8 +115,16 @@ fn build_router_with_options(
115
.route("/v1/chat/completions", post(openai::chat_completions))
116
// vLLM specific endpoints
117
.route("/tokenize", post(tokenize::tokenize))
88
- .route("/detokenize", post(tokenize::detokenize))
89
- .route("/inference/v1/generate", post(inference::generate));
118
+ .route("/detokenize", post(tokenize::detokenize));
119
+
120
+ if scale_out_endpoints_enabled {
121
+ router = router.route("/inference/v1/generate", post(inference::generate));
122
+ } else {
123
+ info!(
124
+ "scale-out endpoints are disabled; set \
125
+ VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 to enable them"
126
+ );
127
+ }
128
129
if runtime_lora_updating_enabled {
130
router = router
rust/src/server/src/routes/tests.rs
+103
-16
@@ -54,6 +54,7 @@ use zeromq::{DealerSocket, PushSocket, ZmqMessage};
54
55
use super::{
56
build_router, build_router_with_dev_mode, build_router_with_dev_mode_and_lora,
57
+ build_router_with_scale_out_endpoints, parse_scale_out_endpoints_flag,
58
render::build_router as build_render_router,
59
};
60
use crate::config::{ApiServerOptions, CorsConfig};
@@ -716,6 +717,83 @@ async fn test_app_with_dev_mode(dev_mode_enabled: bool) -> axum::Router {
717
)
718
}
719
720
+#[test]
721
+fn scale_out_flag_accepts_unset_empty_zero_one_and_whitespace() {
722
+ assert_eq!(parse_scale_out_endpoints_flag(None), Ok(false));
723
+ assert_eq!(parse_scale_out_endpoints_flag(Some("")), Ok(false));
724
+ assert_eq!(parse_scale_out_endpoints_flag(Some("0")), Ok(false));
725
+ assert_eq!(parse_scale_out_endpoints_flag(Some("1")), Ok(true));
726
+ assert_eq!(parse_scale_out_endpoints_flag(Some(" 1 ")), Ok(true));
727
+}
728
+
729
+#[test]
730
+fn scale_out_flag_rejects_values_other_than_zero_or_one() {
731
+ for value in ["-1", "2", "01", "+1", "invalid", " "] {
732
+ assert_eq!(
733
+ parse_scale_out_endpoints_flag(Some(value)),
734
+ Err("VLLM_ENABLE_SCALE_OUT_ENDPOINTS must be 0 or 1".to_string())
735
+ );
736
+ }
737
+}
738
+
739
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
740
+#[serial]
741
+async fn scale_out_generate_route_is_disabled_by_default() {
742
+ let (chat, _engine_task) = test_models_with_engine_outputs_and_backend(
743
+ b"engine-scale-out-disabled",
744
+ default_stream_output_specs(),
745
+ Arc::new(FakeChatBackend::new()),
746
+ )
747
+ .await;
748
+ let mut app = build_router_with_scale_out_endpoints(
749
+ Arc::new(AppState::new(vec!["model".to_string()], chat)),
750
+ false,
751
+ );
752
+
753
+ let response = app
754
+ .call(
755
+ Request::builder()
756
+ .method("POST")
757
+ .uri("/inference/v1/generate")
758
+ .header("content-type", "application/json")
759
+ .body(Body::from("{}"))
760
+ .expect("build request"),
761
+ )
762
+ .await
763
+ .expect("call app");
764
+
765
+ assert_eq!(response.status(), StatusCode::NOT_FOUND);
766
+}
767
+
768
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
769
+#[serial]
770
+async fn scale_out_generate_route_can_be_enabled() {
771
+ let (chat, _engine_task) = test_models_with_engine_outputs_and_backend(
772
+ b"engine-scale-out-enabled",
773
+ default_stream_output_specs(),
774
+ Arc::new(FakeChatBackend::new()),
775
+ )
776
+ .await;
777
+ let mut app = build_router_with_scale_out_endpoints(
778
+ Arc::new(AppState::new(vec!["model".to_string()], chat)),
779
+ true,
780
+ );
781
+
782
+ let response = app
783
+ .call(
784
+ Request::builder()
785
+ .method("POST")
786
+ .uri("/inference/v1/generate")
787
+ .header("content-type", "application/json")
788
+ .body(Body::from("{}"))
789
+ .expect("build request"),
790
+ )
791
+ .await
792
+ .expect("call app");
793
+
794
+ assert_ne!(response.status(), StatusCode::NOT_FOUND);
795
+}
796
+
797
/// Build a dev-mode router backed by a mock engine using a custom ready
798
/// response, returning the router and the engine task handle so the engine
799
/// stays alive for the duration of the test.
@@ -916,10 +994,13 @@ async fn test_app_with_stream_output_specs(
994
)
995
.await;
996
(
919
- build_router(Arc::new(AppState::new(
920
- vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()],
921
- chat,
922
- ))),
997
+ build_router_with_scale_out_endpoints(
998
+ Arc::new(AppState::new(
999
+ vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()],
1000
+ chat,
1001
+ )),
1002
+ true,
1003
+ ),
1004
engine_task,
1005
)
1006
}
@@ -1236,10 +1317,10 @@ async fn render_chat_response_can_be_submitted_to_generate_unchanged() {
1317
Arc::new(FakeChatBackend::new()),
1318
)
1319
.await;
1239
- let mut generate_app = build_router(Arc::new(AppState::new(
1240
- vec!["render-model".to_string()],
1241
- chat,
1242
- )));
1320
+ let mut generate_app = build_router_with_scale_out_endpoints(
1321
+ Arc::new(AppState::new(vec!["render-model".to_string()], chat)),
1322
+ true,
1323
+ );
1324
let generate_response = generate_app
1325
.call(
1326
Request::builder()
@@ -3986,10 +4067,13 @@ async fn non_stream_raw_generate_returns_token_output_envelope() {
4067
.await
4068
.expect("connect client");
4069
let chat = ChatLlm::from_shared_backend(Llm::new(client), Arc::new(FakeChatBackend::new()));
3989
- let mut app = build_router(Arc::new(AppState::new(
3990
- vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()],
3991
- chat,
3992
- )));
4070
+ let mut app = build_router_with_scale_out_endpoints(
4071
+ Arc::new(AppState::new(
4072
+ vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()],
4073
+ chat,
4074
+ )),
4075
+ true,
4076
+ );
4077
4078
let response = app
4079
.call(
@@ -4100,10 +4184,13 @@ async fn stream_raw_generate_returns_sse_chunks_and_usage() {
4184
.await
4185
.expect("connect client");
4186
let chat = ChatLlm::from_shared_backend(Llm::new(client), Arc::new(FakeChatBackend::new()));
4103
- let mut app = build_router(Arc::new(AppState::new(
4104
- vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()],
4105
- chat,
4106
- )));
4187
+ let mut app = build_router_with_scale_out_endpoints(
4188
+ Arc::new(AppState::new(
4189
+ vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()],
4190
+ chat,
4191
+ )),
4192
+ true,
4193
+ );
4194
4195
let response = app
4196
.call(
tests/entrypoints/cohere/test_api_router.py
+3
-1
@@ -58,6 +58,7 @@ def _enable_cohere_api(monkeypatch):
58
flag inside the test body.
59
"""
60
monkeypatch.setenv("VLLM_ENABLE_COHERE_API", "1")
61
+ monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "1")
62
63
64
# ----------------------------------------------------------------------
@@ -402,11 +403,12 @@ class TestRenderEndpoint:
403
assert "/cohere/v2/chat/render" in paths
404
405
def test_route_not_registered_when_flag_unset(self, monkeypatch):
405
- monkeypatch.delenv("VLLM_ENABLE_COHERE_API", raising=False)
406
+ monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
407
app = FastAPI()
408
attach_router(app)
409
paths = [getattr(r, "path", None) for r in app.routes]
410
assert "/cohere/v2/chat/render" not in paths
411
+ assert "/cohere/v2/chat" in paths
412
413
def test_returns_generate_request_json(self):
414
render_handler = _RenderHandler(_generate_request())
tests/entrypoints/cohere/test_chat_v2.py
+1
@@ -71,6 +71,7 @@ def server():
71
env_dict = {
72
"VLLM_CPU_KVCACHE_SPACE": "1",
73
"VLLM_ENABLE_COHERE_API": "1",
74
+ "VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1",
75
}
76
77
with RemoteOpenAIServer(MODEL_NAME, args, env_dict=env_dict) as remote_server:
tests/entrypoints/scale_out/derender/test_derender_parity.py
+5
-1
@@ -54,7 +54,11 @@ FORCE_WEATHER_TOOL = {"type": "function", "function": {"name": "get_weather"}}
54
55
@pytest.fixture(scope="module")
56
def server():
57
- with RemoteOpenAIServer(MODEL, ARGS) as remote_server:
57
+ with RemoteOpenAIServer(
58
+ MODEL,
59
+ ARGS,
60
+ env_dict={"VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1"},
61
+ ) as remote_server:
62
yield remote_server
63
64
tests/entrypoints/scale_out/render/test_render_multimodal.py
+1
-1
@@ -29,7 +29,7 @@ def vision_server():
29
"0",
30
]
31
32
- env_overrides: dict[str, str] = {}
32
+ env_overrides = {"VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1"}
33
34
with RemoteOpenAIServer(
35
VISION_MODEL_NAME,
tests/entrypoints/scale_out/test_factories.py
new
+89
@@ -0,0 +1,89 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+
4
+from argparse import Namespace
5
+
6
+import pytest
7
+from fastapi import FastAPI
8
+
9
+from vllm.entrypoints.launchers.api_server.routers import register_api_routers
10
+from vllm.tasks import SupportedTask
11
+
12
+RENDER_PATHS = {
13
+ "/v1/chat/completions/render",
14
+ "/v1/chat/completions/derender",
15
+ "/v1/completions/render",
16
+ "/v1/completions/derender",
17
+ "/v1/messages/render",
18
+}
19
+GENERATE_PATH = "/inference/v1/generate"
20
+SCALE_OUT_PATHS = RENDER_PATHS | {GENERATE_PATH}
21
+
22
+
23
+def registered_paths(
24
+ supported_tasks: tuple[SupportedTask, ...], *, tokens_only: bool = False
25
+) -> set[str]:
26
+ app = FastAPI()
27
+ args = Namespace(tokens_only=tokens_only, enable_fault_tolerance=False)
28
+ app.state.args = args
29
+ register_api_routers(args, app, supported_tasks)
30
+ return {route.path for route in app.routes}
31
+
32
+
33
+def test_scale_out_routes_are_disabled_by_default(monkeypatch):
34
+ monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
35
+
36
+ paths = registered_paths(("generate",))
37
+
38
+ assert SCALE_OUT_PATHS.isdisjoint(paths)
39
+
40
+
41
+def test_disabled_scale_out_routes_are_logged(monkeypatch, caplog):
42
+ monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
43
+
44
+ with caplog.at_level("INFO", logger="vllm.entrypoints.scale_out.factories"):
45
+ registered_paths(("generate",))
46
+
47
+ assert any(
48
+ "VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1" in record.message
49
+ for record in caplog.records
50
+ )
51
+
52
+
53
+def test_scale_out_routes_can_be_enabled_for_generate_server(monkeypatch):
54
+ monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "1")
55
+
56
+ paths = registered_paths(("generate",))
57
+
58
+ assert paths >= SCALE_OUT_PATHS
59
+
60
+
61
+def test_render_routes_remain_enabled_for_render_server(monkeypatch):
62
+ monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
63
+
64
+ paths = registered_paths(("render",))
65
+
66
+ assert paths >= RENDER_PATHS
67
+ assert GENERATE_PATH not in paths
68
+
69
+
70
+def test_render_server_rejects_explicitly_disabled_scale_out_routes(monkeypatch):
71
+ monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "0")
72
+
73
+ with pytest.raises(ValueError, match="VLLM_ENABLE_SCALE_OUT_ENDPOINTS=0"):
74
+ registered_paths(("render",))
75
+
76
+
77
+def test_tokens_only_mode_enables_generate_routes_when_flag_is_unset(monkeypatch):
78
+ monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
79
+
80
+ paths = registered_paths(("generate",), tokens_only=True)
81
+
82
+ assert paths >= {GENERATE_PATH, "/abort_requests"}
83
+
84
+
85
+def test_tokens_only_mode_rejects_explicitly_disabled_scale_out_routes(monkeypatch):
86
+ monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "0")
87
+
88
+ with pytest.raises(ValueError, match="--tokens-only"):
89
+ registered_paths(("generate",), tokens_only=True)
tests/entrypoints/scale_out/token_in_token_out/test_return_routed_experts.py
+5
-1
@@ -35,7 +35,11 @@ def server():
35
"--hf-overrides",
36
'{"sliding_window": null}',
37
]
38
- with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
38
+ with RemoteOpenAIServer(
39
+ MODEL_NAME,
40
+ args,
41
+ env_dict={"VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1"},
42
+ ) as remote_server:
43
yield remote_server
44
45
tests/entrypoints/scale_out/token_in_token_out/test_serving_multimodal_tokens.py
+5
-1
@@ -39,7 +39,11 @@ def server():
39
"--no-enable-prefix-caching",
40
]
41
42
- with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
42
+ with RemoteOpenAIServer(
43
+ MODEL_NAME,
44
+ args,
45
+ env_dict={"VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1"},
46
+ ) as remote_server:
47
yield remote_server
48
49
tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py
+5
-1
@@ -79,7 +79,11 @@ def server(request):
79
else [str(extra_args)]
80
)
81
82
- with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
82
+ with RemoteOpenAIServer(
83
+ MODEL_NAME,
84
+ args,
85
+ env_dict={"VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1"},
86
+ ) as remote_server:
87
yield remote_server
88
89
tests/test_envs.py
+35
@@ -43,6 +43,41 @@ def test_api_key_is_not_compile_factor(monkeypatch: pytest.MonkeyPatch):
43
assert "VLLM_API_KEY" not in envs.compile_factors()
44
45
46
+def test_scale_out_endpoints_flag_is_runtime_only(monkeypatch: pytest.MonkeyPatch):
47
+ monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "1")
48
+
49
+ assert envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS is True
50
+ assert "VLLM_ENABLE_SCALE_OUT_ENDPOINTS" not in envs.compile_factors()
51
+
52
+
53
+def test_scale_out_endpoints_flag_distinguishes_unset_from_disabled(
54
+ monkeypatch: pytest.MonkeyPatch,
55
+):
56
+ monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
57
+ assert envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS is None
58
+
59
+ monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "0")
60
+ assert envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS is False
61
+
62
+
63
+def test_scale_out_endpoints_flag_treats_empty_as_unset(
64
+ monkeypatch: pytest.MonkeyPatch,
65
+):
66
+ monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "")
67
+
68
+ assert envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS is None
69
+
70
+
71
+@pytest.mark.parametrize("value", ["-1", "2", "01", "+1", "invalid", " "])
72
+def test_scale_out_endpoints_flag_rejects_values_other_than_zero_or_one(
73
+ monkeypatch: pytest.MonkeyPatch, value: str
74
+):
75
+ monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", value)
76
+
77
+ with pytest.raises(ValueError, match="must be 0 or 1"):
78
+ _ = envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS
79
+
80
+
81
def test_p2p_side_channel_defaults_and_override(monkeypatch: pytest.MonkeyPatch):
82
monkeypatch.delenv("VLLM_P2P_SIDE_CHANNEL_HOST", raising=False)
83
monkeypatch.delenv("VLLM_P2P_SIDE_CHANNEL_PORT", raising=False)
vllm/entrypoints/cohere/api_router.py
+8
-5
@@ -17,9 +17,9 @@ log) and vLLM continues to boot normally.
17
18
Even when the SDK is installed, :func:`attach_router` also requires
19
``VLLM_ENABLE_COHERE_API=1`` in the environment before it will expose
20
-the routes. This keeps non-Cohere deployments that pull in the SDK for
21
-unrelated reasons (e.g. test dependencies) from accidentally exposing
22
-the api.
20
+the chat route. The render route additionally requires
21
+``VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1``. These gates keep unrelated
22
+deployments from accidentally exposing the APIs.
23
24
Note: the handlers must live at module scope (not inside
25
``attach_router``) so that FastAPI's ``typing.get_type_hints`` resolves
@@ -71,6 +71,7 @@ if _SDK_AVAILABLE:
71
from vllm.entrypoints.scale_out.token_in_token_out.protocol import GenerateRequest
72
73
router = APIRouter()
74
+ render_router = APIRouter()
75
76
def _serving(request: Request) -> CohereServingChatV2 | None:
77
return getattr(request.app.state, "cohere_serving_chat_v2", None)
@@ -154,7 +155,7 @@ if _SDK_AVAILABLE:
155
case _:
156
return StreamingResponse(content=result, media_type="text/event-stream")
157
157
- @router.post(
158
+ @render_router.post(
159
"/cohere/v2/chat/render",
160
dependencies=[Depends(validate_json_request)],
161
response_model=GenerateRequest,
@@ -273,7 +274,7 @@ if _SDK_AVAILABLE:
274
275
276
def attach_router(app: FastAPI) -> None:
276
- """Register the ``/cohere/v2/chat`` routes on ``app``.
277
+ """Register the enabled ``/cohere/v2/chat`` routes on ``app``.
278
279
No-op when either:
280
@@ -305,4 +306,6 @@ def attach_router(app: FastAPI) -> None:
306
)
307
return
308
app.include_router(router)
309
+ if envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS:
310
+ app.include_router(render_router)
311
app.add_middleware(CohereErrorEnvelopeMiddleware)
vllm/entrypoints/scale_out/factories.py
+23
@@ -5,7 +5,9 @@ from typing import TYPE_CHECKING
5
6
from fastapi import FastAPI
7
8
+from vllm import envs
9
from vllm.engine.protocol import EngineClient
10
+from vllm.logger import init_logger
11
from vllm.tasks import SupportedTask
12
13
if TYPE_CHECKING:
@@ -15,6 +17,8 @@ if TYPE_CHECKING:
17
else:
18
RequestLogger = object
19
20
+logger = init_logger(__name__)
21
+
22
23
def init_render_state(
24
state: "State",
@@ -62,6 +66,25 @@ def register_scale_out_api_routers(
66
app: FastAPI,
67
supported_tasks: tuple["SupportedTask", ...],
68
):
69
+ dedicated_mode = None
70
+ if "render" in supported_tasks:
71
+ dedicated_mode = "`vllm launch render`"
72
+ elif getattr(app.state.args, "tokens_only", False):
73
+ dedicated_mode = "`--tokens-only`"
74
+
75
+ enabled = envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS
76
+ if dedicated_mode is not None and enabled is False:
77
+ raise ValueError(
78
+ "`VLLM_ENABLE_SCALE_OUT_ENDPOINTS=0` conflicts with "
79
+ f"{dedicated_mode}; unset it or set it to 1"
80
+ )
81
+ if dedicated_mode is None and not enabled:
82
+ logger.info(
83
+ "Scale-out endpoints are disabled. Set "
84
+ "VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 to enable them."
85
+ )
86
+ return
87
+
88
from .render.api_router import router as render_render
89
90
app.include_router(render_render)
vllm/envs.py
+17
@@ -261,6 +261,7 @@ if TYPE_CHECKING:
261
VLLM_ALLOW_CHUNKED_LOCAL_ATTN_WITH_HYBRID_KV_CACHE: bool = True
262
VLLM_ENABLE_RESPONSES_API_STORE: bool = False
263
VLLM_ENABLE_COHERE_API: bool = False
264
+ VLLM_ENABLE_SCALE_OUT_ENDPOINTS: bool | None = None
265
VLLM_HAS_FLASHINFER_CUBIN: bool = False
266
VLLM_ROCM_FP8_MFMA_PAGE_ATTN: bool = False
267
VLLM_ALLREDUCE_USE_SYMM_MEM: bool = True
@@ -353,6 +354,15 @@ def maybe_convert_bool(value: str | None) -> bool | None:
354
return bool(int(value))
355
356
357
+def maybe_convert_scale_out_endpoints(value: str | None) -> bool | None:
358
+ if value is None:
359
+ return None
360
+ normalized = value.strip()
361
+ if normalized not in ("0", "1"):
362
+ raise ValueError("VLLM_ENABLE_SCALE_OUT_ENDPOINTS must be 0 or 1")
363
+ return maybe_convert_bool(normalized)
364
+
365
+
366
def maybe_convert_json_str_or_file(value: str | None) -> dict[str, Any] | None:
367
if value is None:
368
return None
@@ -1864,6 +1874,12 @@ environment_variables: dict[str, Callable[[], Any]] = {
1874
"VLLM_ENABLE_COHERE_API": lambda: bool(
1875
int(os.getenv("VLLM_ENABLE_COHERE_API", "0"))
1876
),
1877
+ # If set to 1, expose the scale-out endpoints on `vllm serve`.
1878
+ # The dedicated `vllm launch render` server exposes render and derender
1879
+ # endpoints when this variable is unset or set to 1.
1880
+ "VLLM_ENABLE_SCALE_OUT_ENDPOINTS": lambda: maybe_convert_scale_out_endpoints(
1881
+ os.getenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS") or None
1882
+ ),
1883
# If set, use the fp8 mfma in rocm paged attention.
1884
"VLLM_ROCM_FP8_MFMA_PAGE_ATTN": lambda: bool(
1885
int(os.getenv("VLLM_ROCM_FP8_MFMA_PAGE_ATTN", "0"))
@@ -2291,6 +2307,7 @@ def compile_factors() -> dict[str, object]:
2307
"VLLM_CONFIG_ROOT",
2308
"LD_LIBRARY_PATH",
2309
"VLLM_SERVER_DEV_MODE",
2310
+ "VLLM_ENABLE_SCALE_OUT_ENDPOINTS",
2311
"VLLM_DP_MASTER_IP",
2312
"VLLM_DP_MASTER_PORT",
2313
"VLLM_NIXL_SIDE_CHANNEL_HOST",