[Frontend] Replace `VLLM_ENABLE_SCALE_OUT_ENDPOINTS` with `--enable-scale-out` (#55176)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Martin Hickey committed
Sep 14, 2026 at 11:02 UTC
934b1fcbcbbac6e68da7a4246928ac797fe54690
24 files changed
+130
-215
docs/serving/online_serving/README.md
+1
-6
@@ -121,12 +121,7 @@ For further details on profiling vLLM, please refer to [this page](../../contrib
121
122
## Scale-Out APIs
123
124
-Scale-out APIs are disabled by default on `vllm serve`. The environment
125
-variable accepts only `0` or `1`; set `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1` to
126
-register the endpoints below. The
127
-dedicated `vllm launch render` and `vllm serve --tokens-only` modes are explicit
128
-opt-ins and enable their required endpoints when the variable is unset; an
129
-explicit value of `0` is rejected for those modes.
124
+Scale-out APIs are disabled by default on `vllm serve`. Set `--enable-scale-out` to register the endpoints below. The dedicated `vllm launch render` and `vllm serve --tokens-only` modes always register their required endpoints regardless of `--enable-scale-out`.
125
126
### Tokens IN <> Tokens OUT APIs
127
docs/serving/online_serving/derenderer.md
+1
-2
@@ -60,8 +60,7 @@ Launch the two servers first:
60
61
```bash
62
vllm launch render meta-llama/Llama-3.2-1B-Instruct --port 8100
63
-VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve \
64
- meta-llama/Llama-3.2-1B-Instruct --tokens-only --port 8200
63
+vllm serve meta-llama/Llama-3.2-1B-Instruct --tokens-only --port 8200
64
```
65
66
```python
docs/serving/online_serving/renderer.md
+2
-4
@@ -7,16 +7,14 @@ Our renderer API is designed to disaggregate the render phase(preprocessing) and
7
- Tokens-in / tokens-out engine: Make the engine a pure token-in / token-out service, decoupled from request preprocessing.
8
9
The dedicated `vllm launch render` server always exposes the `/render` and
10
-`/derender` endpoints when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS` is unset or set to
11
-`1`. An explicit value of `0` conflicts with the renderer command and is
12
-rejected at startup.
10
+`/derender` endpoints.
11
12
Scale-out endpoints, including `/render`, `/derender`, and
13
`/inference/v1/generate`, are disabled by default on a standard inference
14
server. To expose them with `vllm serve`, opt in explicitly:
15
16
```bash
19
-VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve <model>
17
+vllm serve <model> --enable-scale-out
18
```
19
20
## API Reference
docs/usage/security.md
+9
-9
@@ -158,16 +158,16 @@ When `--api-key` is configured, the following endpoints require Bearer token aut
158
- `/v1/models` - List available models
159
- `/v1/chat/completions` - Chat completions
160
- `/v1/chat/completions/batch` - Batch chat completions
161
-- `/v1/chat/completions/render` - Render chat completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
162
-- `/v1/chat/completions/derender` - Derender chat completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
161
+- `/v1/chat/completions/render` - Render chat completion requests (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render`)
162
+- `/v1/chat/completions/derender` - Derender chat completion requests (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render`)
163
- `/v1/completions` - Text completions
164
-- `/v1/completions/render` - Render completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
165
-- `/v1/completions/derender` - Derender completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
164
+- `/v1/completions/render` - Render completion requests (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render`)
165
+- `/v1/completions/derender` - Derender completion requests (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render`)
166
- `/v1/embeddings` - Generate embeddings
167
- `/v1/audio/transcriptions` - Audio transcription
168
- `/v1/audio/translations` - Audio translation
169
- `/v1/messages` - Anthropic-compatible messages API
170
-- `/v1/messages/render` - Render Anthropic-compatible messages (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
170
+- `/v1/messages/render` - Render Anthropic-compatible messages (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render`)
171
- `/v1/messages/count_tokens` - Count tokens for Anthropic messages
172
- `/v1/responses` - Create a response
173
- `/v1/responses/render` - Render a self-contained response request (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
@@ -177,8 +177,8 @@ When `--api-key` is configured, the following endpoints require Bearer token aut
177
- `/v1/rerank` - Reranking API
178
- `/v1/load_lora_adapter` - Load a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
179
- `/v1/unload_lora_adapter` - Unload a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
180
-- `/inference/v1/generate` - Generate completions (available when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or with `--tokens-only` unless explicitly disabled)
181
-- `/cohere/v2/chat/render` - Render Cohere Chat v2 requests (requires both `VLLM_ENABLE_COHERE_API=1` and `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`)
180
+- `/inference/v1/generate` - Generate completions (available when `--enable-scale-out` is set, or with `--tokens-only`)
181
+- `/cohere/v2/chat/render` - Render Cohere Chat v2 requests (requires `VLLM_ENABLE_COHERE_API=1` and either `--enable-scale-out` or `--tokens-only`)
182
- `/v2/embed` - Cohere Embed API
183
- `/v2/rerank` - Cohere Rerank API
184
@@ -206,11 +206,11 @@ The following endpoints **do not require authentication** even when `--api-key`
206
- `/init_weight_transfer_engine` - Initialize weight transfer engine for RLHF
207
- `/update_weights` - Update model weights (can alter model behavior)
208
- `/get_world_size` - Get distributed world size
209
-- `/abort_requests` - Abort in-flight requests (available with `--tokens-only` unless `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=0`)
209
+- `/abort_requests` - Abort in-flight requests (available with `--tokens-only`)
210
211
**Utility endpoints:**
212
213
-- `/tokenize` - Tokenize text (not disabled by `VLLM_ENABLE_SCALE_OUT_ENDPOINTS`)
213
+- `/tokenize` - Tokenize text (not gated by `--enable-scale-out`)
214
- `/detokenize` - Detokenize tokens
215
- `/health` - Health check
216
- `/ping` - SageMaker health check
examples/scale_out/example_mm_serve.py
+3
-3
@@ -13,9 +13,9 @@ zero client-side transformation.
13
14
Launch the server first:
15
16
- VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 \
17
- vllm serve Qwen/Qwen3-VL-2B-Instruct \
18
- --dtype bfloat16 --max-model-len 4096 --enforce-eager
16
+ vllm serve Qwen/Qwen3-VL-2B-Instruct \
17
+ --dtype bfloat16 --max-model-len 4096 --enforce-eager \
18
+ --enable-scale-out
19
20
Then run this script:
21
examples/scale_out/token_generation_client.py
+1
-1
@@ -2,7 +2,7 @@
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
"""Call a token-in/token-out server started with:
4
5
-VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve Qwen/Qwen3-0.6B
5
+vllm serve Qwen/Qwen3-0.6B --enable-scale-out
6
"""
7
8
import httpx
rust/src/cmd/src/cli.rs
+10
@@ -329,6 +329,15 @@ pub struct SharedRuntimeArgs {
329
#[serde(default)]
330
pub enable_request_id_headers: bool,
331
332
+ /// Register the scale-out `/inference/v1/generate` endpoint.
333
+ #[arg(
334
+ long,
335
+ default_missing_value = "true",
336
+ num_args = 0..=1
337
+ )]
338
+ #[serde(default)]
339
+ pub enable_scale_out: bool,
340
+
341
/// If provided, the server will require one of these keys to be presented
342
/// in the Authorization header.
343
#[educe(Debug(ignore))]
@@ -577,6 +586,7 @@ impl SharedRuntimeArgs {
586
enable_log_requests: self.enable_log_requests,
587
enable_prompt_tokens_details: self.enable_prompt_tokens_details,
588
enable_request_id_headers: self.enable_request_id_headers,
589
+ enable_scale_out: self.enable_scale_out,
590
}
591
}
592
rust/src/cmd/src/cli/tests.rs
+6
@@ -195,6 +195,7 @@ fn serve_args_forward_python_flags_with_separator() {
195
enable_log_requests: false,
196
enable_prompt_tokens_details: false,
197
enable_request_id_headers: false,
198
+ enable_scale_out: false,
199
disable_log_stats: false,
200
served_model_name: [],
201
allowed_origins: JsonStringList(
@@ -1030,6 +1031,7 @@ fn frontend_args_accept_json() {
1031
enable_log_requests: false,
1032
enable_prompt_tokens_details: false,
1033
enable_request_id_headers: false,
1034
+ enable_scale_out: false,
1035
disable_log_stats: false,
1036
served_model_name: [],
1037
allowed_origins: JsonStringList(
@@ -1629,6 +1631,7 @@ fn serve_args_accept_handshake_aliases() {
1631
enable_log_requests: false,
1632
enable_prompt_tokens_details: false,
1633
enable_request_id_headers: false,
1634
+ enable_scale_out: false,
1635
disable_log_stats: false,
1636
served_model_name: [],
1637
allowed_origins: JsonStringList(
@@ -1780,6 +1783,7 @@ fn serve_frontend_config_uses_dp_address_as_advertised_host() {
1783
enable_log_requests: false,
1784
enable_prompt_tokens_details: false,
1785
enable_request_id_headers: false,
1786
+ enable_scale_out: false,
1787
},
1788
cors: CorsConfig {
1789
allow_origins: [
@@ -1868,6 +1872,7 @@ fn serve_frontend_config_keeps_tcp_transport_for_non_local_only_topology() {
1872
enable_log_requests: false,
1873
enable_prompt_tokens_details: false,
1874
enable_request_id_headers: false,
1875
+ enable_scale_out: false,
1876
},
1877
cors: CorsConfig {
1878
allow_origins: [
@@ -1977,6 +1982,7 @@ fn frontend_config_uses_external_coordinator_when_coordinator_address_is_present
1982
enable_log_requests: false,
1983
enable_prompt_tokens_details: false,
1984
enable_request_id_headers: false,
1985
+ enable_scale_out: false,
1986
},
1987
cors: CorsConfig {
1988
allow_origins: [
rust/src/server/src/config.rs
+2
@@ -55,6 +55,8 @@ pub struct ApiServerOptions {
55
pub enable_prompt_tokens_details: bool,
56
/// When `true`, set `X-Request-Id` on every HTTP response.
57
pub enable_request_id_headers: bool,
58
+ /// When `true`, register the scale-out `/inference/v1/generate` route.
59
+ pub enable_scale_out: bool,
60
}
61
62
/// CORS settings mirroring Python's `CORSMiddleware`; the default is permissive.
rust/src/server/src/routes.rs
+3
-25
@@ -46,33 +46,14 @@ fn runtime_lora_updating_enabled() -> bool {
46
.is_some_and(|value| matches!(value.trim().to_lowercase().as_str(), "1" | "true"))
47
}
48
49
-fn parse_scale_out_endpoints_flag(value: Option<&str>) -> Result<bool, String> {
50
- let Some(value) = value else {
51
- return Ok(false);
52
- };
53
- if value.is_empty() {
54
- return Ok(false);
55
- }
56
-
57
- match value.trim() {
58
- "0" => Ok(false),
59
- "1" => Ok(true),
60
- _ => Err("VLLM_ENABLE_SCALE_OUT_ENDPOINTS must be 0 or 1".to_string()),
61
- }
62
-}
63
-
64
-fn scale_out_endpoints_enabled() -> bool {
65
- let value = std::env::var("VLLM_ENABLE_SCALE_OUT_ENDPOINTS").ok();
66
- parse_scale_out_endpoints_flag(value.as_deref()).unwrap_or_else(|message| panic!("{message}"))
67
-}
68
-
49
/// Build the minimal OpenAI-compatible router for one configured model.
50
pub fn build_router(state: Arc<AppState>) -> Router {
51
+ let scale_out_endpoints_enabled = state.api_server_options.enable_scale_out;
52
build_router_with_options(
53
state,
54
server_dev_mode_enabled(),
55
runtime_lora_updating_enabled(),
75
- scale_out_endpoints_enabled(),
56
+ scale_out_endpoints_enabled,
57
)
58
}
59
@@ -121,10 +102,7 @@ fn build_router_with_options(
102
if scale_out_endpoints_enabled {
103
router = router.route("/inference/v1/generate", post(inference::generate));
104
} else {
124
- info!(
125
- "scale-out endpoints are disabled; set \
126
- VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 to enable them"
127
- );
105
+ info!("scale-out endpoints are disabled; pass --enable-scale-out to enable them");
106
}
107
108
if runtime_lora_updating_enabled {
rust/src/server/src/routes/tests.rs
+30
-18
@@ -54,8 +54,7 @@ use zeromq::{DealerSocket, PushSocket, ZmqMessage};
54
55
use super::{
56
build_router, build_router_with_dev_mode, build_router_with_dev_mode_and_lora,
57
- build_router_with_scale_out_endpoints, parse_scale_out_endpoints_flag,
58
- render::build_router as build_render_router,
57
+ build_router_with_scale_out_endpoints, render::build_router as build_render_router,
58
};
59
use crate::config::{ApiServerOptions, CorsConfig, LoraModulePath};
60
use crate::lora::LoadLoraError;
@@ -731,23 +730,36 @@ async fn test_app_with_dev_mode(dev_mode_enabled: bool) -> axum::Router {
730
)
731
}
732
734
-#[test]
735
-fn scale_out_flag_accepts_unset_empty_zero_one_and_whitespace() {
736
- assert_eq!(parse_scale_out_endpoints_flag(None), Ok(false));
737
- assert_eq!(parse_scale_out_endpoints_flag(Some("")), Ok(false));
738
- assert_eq!(parse_scale_out_endpoints_flag(Some("0")), Ok(false));
739
- assert_eq!(parse_scale_out_endpoints_flag(Some("1")), Ok(true));
740
- assert_eq!(parse_scale_out_endpoints_flag(Some(" 1 ")), Ok(true));
741
-}
733
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
734
+#[serial]
735
+async fn scale_out_generate_route_follows_api_server_options() {
736
+ let (chat, _engine_task) = test_models_with_engine_outputs_and_backend(
737
+ b"engine-scale-out-api-server-options",
738
+ default_stream_output_specs(),
739
+ Arc::new(FakeChatBackend::new()),
740
+ )
741
+ .await;
742
+ let state = Arc::new(
743
+ AppState::new(vec!["model".to_string()], chat).with_api_server_options(ApiServerOptions {
744
+ enable_scale_out: true,
745
+ ..Default::default()
746
+ }),
747
+ );
748
+ let mut app = build_router(state);
749
743
-#[test]
744
-fn scale_out_flag_rejects_values_other_than_zero_or_one() {
745
- for value in ["-1", "2", "01", "+1", "invalid", " "] {
746
- assert_eq!(
747
- parse_scale_out_endpoints_flag(Some(value)),
748
- Err("VLLM_ENABLE_SCALE_OUT_ENDPOINTS must be 0 or 1".to_string())
749
- );
750
- }
750
+ let response = app
751
+ .call(
752
+ Request::builder()
753
+ .method("POST")
754
+ .uri("/inference/v1/generate")
755
+ .header("content-type", "application/json")
756
+ .body(Body::from("{}"))
757
+ .expect("build request"),
758
+ )
759
+ .await
760
+ .expect("call app");
761
+
762
+ assert_ne!(response.status(), StatusCode::NOT_FOUND);
763
}
764
765
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
tests/entrypoints/cohere/test_api_router.py
+13
-10
@@ -58,7 +58,6 @@ def _enable_cohere_api(monkeypatch):
58
flag inside the test body.
59
"""
60
monkeypatch.setenv("VLLM_ENABLE_COHERE_API", "1")
61
- monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "1")
61
62
63
# ----------------------------------------------------------------------
@@ -128,15 +127,19 @@ def _generate_request() -> GenerateRequest:
127
)
128
129
131
-def _build_app(handler: _Handler | None) -> FastAPI:
130
+def _build_app(handler: _Handler | None, *, enable_scale_out: bool = True) -> FastAPI:
131
app = FastAPI()
132
+ app.state.args = Namespace(enable_scale_out=enable_scale_out, tokens_only=False)
133
attach_router(app)
134
app.state.cohere_serving_chat_v2 = handler
135
return app
136
137
138
-def _build_render_app(chat_handler, render_handler) -> FastAPI:
138
+def _build_render_app(
139
+ chat_handler, render_handler, *, enable_scale_out: bool = True
140
+) -> FastAPI:
141
app = FastAPI()
142
+ app.state.args = Namespace(enable_scale_out=enable_scale_out, tokens_only=False)
143
attach_router(app)
144
app.state.cohere_serving_chat_v2 = chat_handler
145
app.state.serving_render = render_handler
@@ -151,11 +154,13 @@ def _build_app_with_vllm_handlers(handler: _Handler | None) -> FastAPI:
154
into the ``CohereError`` wire shape.
155
"""
156
app = FastAPI()
154
- attach_router(app)
155
- app.state.cohere_serving_chat_v2 = handler
157
# ``validation_exception_handler`` reads ``req.app.state.args``; the
158
# real cli builds this via argparse.
158
- app.state.args = Namespace(log_error_stack=False)
159
+ app.state.args = Namespace(
160
+ log_error_stack=False, enable_scale_out=True, tokens_only=False
161
+ )
162
+ attach_router(app)
163
+ app.state.cohere_serving_chat_v2 = handler
164
app.exception_handler(RequestValidationError)(validation_exception_handler)
165
app.exception_handler(HTTPException)(http_exception_handler)
166
app.exception_handler(VLLMError)(vllm_error_handler)
@@ -402,10 +407,8 @@ class TestRenderEndpoint:
407
paths = [getattr(r, "path", None) for r in app.routes]
408
assert "/cohere/v2/chat/render" in paths
409
405
- def test_route_not_registered_when_flag_unset(self, monkeypatch):
406
- monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
407
- app = FastAPI()
408
- attach_router(app)
410
+ def test_route_not_registered_when_flag_unset(self):
411
+ app = _build_app(handler=None, enable_scale_out=False)
412
paths = [getattr(r, "path", None) for r in app.routes]
413
assert "/cohere/v2/chat/render" not in paths
414
assert "/cohere/v2/chat" in paths
tests/entrypoints/cohere/test_chat_v2.py
+2
-1
@@ -57,6 +57,8 @@ def server():
57
# the conversation to surface a thinking block (SmolLM2 doesn't
58
# emit Cohere-style reasoning tokens out of the box).
59
"--no-cohere-is-reasoning-model",
60
+ # Registers /cohere/v2/chat/render alongside /cohere/v2/chat.
61
+ "--enable-scale-out",
62
]
63
64
# Cap the CPU KV-cache pool the vLLM CPU backend reserves at
@@ -71,7 +73,6 @@ def server():
73
env_dict = {
74
"VLLM_CPU_KVCACHE_SPACE": "1",
75
"VLLM_ENABLE_COHERE_API": "1",
74
- "VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1",
76
}
77
78
with RemoteOpenAIServer(MODEL_NAME, args, env_dict=env_dict) as remote_server:
tests/entrypoints/scale_out/derender/test_derender_parity.py
+2
-5
@@ -34,6 +34,7 @@ ARGS = [
34
"hermes",
35
"--reasoning-parser",
36
"deepseek_r1",
37
+ "--enable-scale-out",
38
]
39
40
TOOLS = [
@@ -54,11 +55,7 @@ FORCE_WEATHER_TOOL = {"type": "function", "function": {"name": "get_weather"}}
55
56
@pytest.fixture(scope="module")
57
def server():
57
- with RemoteOpenAIServer(
58
- MODEL,
59
- ARGS,
60
- env_dict={"VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1"},
61
- ) as remote_server:
58
+ with RemoteOpenAIServer(MODEL, ARGS) as remote_server:
59
yield remote_server
60
61
tests/entrypoints/scale_out/render/test_render_multimodal.py
+2
-7
@@ -27,15 +27,10 @@ def vision_server():
27
"1",
28
"--limit-mm-per-prompt.video",
29
"0",
30
+ "--enable-scale-out",
31
]
32
32
- env_overrides = {"VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1"}
33
-
34
- with RemoteOpenAIServer(
35
- VISION_MODEL_NAME,
36
- args,
37
- env_dict=env_overrides,
38
- ) as remote_server:
33
+ with RemoteOpenAIServer(VISION_MODEL_NAME, args) as remote_server:
34
yield remote_server
35
36
tests/entrypoints/scale_out/test_factories.py
+16
-37
@@ -3,7 +3,6 @@
3
4
from argparse import Namespace
5
6
-import pytest
6
from fastapi import FastAPI
7
8
from vllm.entrypoints.launchers.api_server.routers import register_api_routers
@@ -22,69 +21,49 @@ SCALE_OUT_PATHS = RENDER_PATHS | {GENERATE_PATH}
21
22
23
def registered_paths(
25
- supported_tasks: tuple[SupportedTask, ...], *, tokens_only: bool = False
24
+ supported_tasks: tuple[SupportedTask, ...],
25
+ *,
26
+ tokens_only: bool = False,
27
+ enable_scale_out: bool = False,
28
) -> set[str]:
29
app = FastAPI()
28
- args = Namespace(tokens_only=tokens_only, enable_fault_tolerance=False)
30
+ args = Namespace(
31
+ tokens_only=tokens_only,
32
+ enable_scale_out=enable_scale_out,
33
+ enable_fault_tolerance=False,
34
+ )
35
app.state.args = args
36
register_api_routers(args, app, supported_tasks)
37
return {route.path for route in app.routes}
38
39
34
-def test_scale_out_routes_are_disabled_by_default(monkeypatch):
35
- monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
36
-
40
+def test_scale_out_routes_are_disabled_by_default():
41
paths = registered_paths(("generate",))
42
43
assert SCALE_OUT_PATHS.isdisjoint(paths)
44
45
42
-def test_disabled_scale_out_routes_are_logged(monkeypatch, caplog):
43
- monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
44
-
46
+def test_disabled_scale_out_routes_are_logged(caplog):
47
with caplog.at_level("INFO", logger="vllm.entrypoints.scale_out.factories"):
48
registered_paths(("generate",))
49
48
- assert any(
49
- "VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1" in record.message
50
- for record in caplog.records
51
- )
50
+ assert any("--enable-scale-out" in record.message for record in caplog.records)
51
52
54
-def test_scale_out_routes_can_be_enabled_for_generate_server(monkeypatch):
55
- monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "1")
56
-
57
- paths = registered_paths(("generate",))
53
+def test_scale_out_routes_can_be_enabled_for_generate_server():
54
+ paths = registered_paths(("generate",), enable_scale_out=True)
55
56
assert paths >= SCALE_OUT_PATHS
57
58
62
-def test_render_routes_remain_enabled_for_render_server(monkeypatch):
63
- monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
64
-
59
+def test_render_routes_remain_enabled_for_render_server():
60
paths = registered_paths(("render",))
61
62
assert paths >= RENDER_PATHS
63
assert GENERATE_PATH not in paths
64
65
71
-def test_render_server_rejects_explicitly_disabled_scale_out_routes(monkeypatch):
72
- monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "0")
73
-
74
- with pytest.raises(ValueError, match="VLLM_ENABLE_SCALE_OUT_ENDPOINTS=0"):
75
- registered_paths(("render",))
76
-
77
-
78
-def test_tokens_only_mode_enables_generate_routes_when_flag_is_unset(monkeypatch):
79
- monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
80
-
66
+def test_tokens_only_mode_enables_generate_routes_when_flag_is_unset():
67
paths = registered_paths(("generate",), tokens_only=True)
68
69
assert paths >= {GENERATE_PATH, "/abort_requests"}
84
-
85
-
86
-def test_tokens_only_mode_rejects_explicitly_disabled_scale_out_routes(monkeypatch):
87
- monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "0")
88
-
89
- with pytest.raises(ValueError, match="--tokens-only"):
90
- registered_paths(("generate",), tokens_only=True)
tests/entrypoints/scale_out/token_in_token_out/test_return_routed_experts.py
+2
-5
@@ -34,12 +34,9 @@ def server():
34
"--enable-return-routed-experts",
35
"--hf-overrides",
36
'{"sliding_window": null}',
37
+ "--enable-scale-out",
38
]
38
- with RemoteOpenAIServer(
39
- MODEL_NAME,
40
- args,
41
- env_dict={"VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1"},
42
- ) as remote_server:
39
+ with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
40
yield remote_server
41
42
tests/entrypoints/scale_out/token_in_token_out/test_serving_multimodal_tokens.py
+2
-5
@@ -37,13 +37,10 @@ def server():
37
"4096",
38
"--enforce-eager",
39
"--no-enable-prefix-caching",
40
+ "--enable-scale-out",
41
]
42
42
- with RemoteOpenAIServer(
43
- MODEL_NAME,
44
- args,
45
- env_dict={"VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1"},
46
- ) as remote_server:
43
+ with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
44
yield remote_server
45
46
tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py
+2
-5
@@ -69,6 +69,7 @@ def server(request):
69
# Option B: deterministic prefix caching
70
# "--enable-prefix-caching",
71
# "--deterministic-prefix-caching",
72
+ "--enable-scale-out",
73
]
74
75
extra_args = getattr(request, "param", None)
@@ -79,11 +80,7 @@ def server(request):
80
else [str(extra_args)]
81
)
82
82
- with RemoteOpenAIServer(
83
- MODEL_NAME,
84
- args,
85
- env_dict={"VLLM_ENABLE_SCALE_OUT_ENDPOINTS": "1"},
86
- ) as remote_server:
83
+ with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
84
yield remote_server
85
86
tests/test_envs.py
-35
@@ -43,41 +43,6 @@ def test_api_key_is_not_compile_factor(monkeypatch: pytest.MonkeyPatch):
43
assert "VLLM_API_KEY" not in envs.compile_factors()
44
45
46
-def test_scale_out_endpoints_flag_is_runtime_only(monkeypatch: pytest.MonkeyPatch):
47
- monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "1")
48
-
49
- assert envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS is True
50
- assert "VLLM_ENABLE_SCALE_OUT_ENDPOINTS" not in envs.compile_factors()
51
-
52
-
53
-def test_scale_out_endpoints_flag_distinguishes_unset_from_disabled(
54
- monkeypatch: pytest.MonkeyPatch,
55
-):
56
- monkeypatch.delenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", raising=False)
57
- assert envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS is None
58
-
59
- monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "0")
60
- assert envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS is False
61
-
62
-
63
-def test_scale_out_endpoints_flag_treats_empty_as_unset(
64
- monkeypatch: pytest.MonkeyPatch,
65
-):
66
- monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", "")
67
-
68
- assert envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS is None
69
-
70
-
71
-@pytest.mark.parametrize("value", ["-1", "2", "01", "+1", "invalid", " "])
72
-def test_scale_out_endpoints_flag_rejects_values_other_than_zero_or_one(
73
- monkeypatch: pytest.MonkeyPatch, value: str
74
-):
75
- monkeypatch.setenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS", value)
76
-
77
- with pytest.raises(ValueError, match="must be 0 or 1"):
78
- _ = envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS
79
-
80
-
46
def test_p2p_side_channel_defaults_and_override(monkeypatch: pytest.MonkeyPatch):
47
monkeypatch.delenv("VLLM_P2P_SIDE_CHANNEL_HOST", raising=False)
48
monkeypatch.delenv("VLLM_P2P_SIDE_CHANNEL_PORT", raising=False)
vllm/entrypoints/cohere/api_router.py
+6
-4
@@ -17,9 +17,10 @@ log) and vLLM continues to boot normally.
17
18
Even when the SDK is installed, :func:`attach_router` also requires
19
``VLLM_ENABLE_COHERE_API=1`` in the environment before it will expose
20
-the chat route. The render route additionally requires
21
-``VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1``. These gates keep unrelated
22
-deployments from accidentally exposing the APIs.
20
+the chat route. The render route additionally requires scale-out
21
+endpoints to be enabled, i.e. ``--enable-scale-out`` or
22
+``--tokens-only``. These gates keep unrelated deployments from
23
+accidentally exposing the APIs.
24
25
Note: the handlers must live at module scope (not inside
26
``attach_router``) so that FastAPI's ``typing.get_type_hints`` resolves
@@ -306,6 +307,7 @@ def attach_router(app: FastAPI) -> None:
307
)
308
return
309
app.include_router(router)
309
- if envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS:
310
+ args = getattr(app.state, "args", None)
311
+ if getattr(args, "enable_scale_out", False) or getattr(args, "tokens_only", False):
312
app.include_router(render_router)
313
app.add_middleware(CohereErrorEnvelopeMiddleware)
vllm/entrypoints/launchers/cli_args.py
+7
@@ -181,6 +181,13 @@ class BaseFrontendArgs:
181
If set to True, only enable the Tokens In<>Out endpoint.
182
This is intended for use in a Disaggregated Everything setup.
183
"""
184
+ enable_scale_out: bool = False
185
+ """
186
+ If set to True, register the scale-out endpoints (`/render`, `/derender`,
187
+ and `/inference/v1/generate`) on `vllm serve`. Has no effect on
188
+ `vllm launch render` or `vllm serve --tokens-only`, which always register
189
+ their required endpoints regardless of this flag.
190
+ """
191
fingerprint_mode: Literal["full", "hash", "custom", "none"] = "full"
192
"""Controls the ``system_fingerprint`` field on responses.
193
vllm/entrypoints/scale_out/factories.py
+8
-16
@@ -5,7 +5,6 @@ from typing import TYPE_CHECKING
5
6
from fastapi import FastAPI
7
8
-from vllm import envs
8
from vllm.engine.protocol import EngineClient
9
from vllm.logger import init_logger
10
from vllm.tasks import SupportedTask
@@ -67,22 +66,15 @@ def register_scale_out_api_routers(
66
app: FastAPI,
67
supported_tasks: tuple["SupportedTask", ...],
68
):
70
- dedicated_mode = None
71
- if "render" in supported_tasks:
72
- dedicated_mode = "`vllm launch render`"
73
- elif getattr(app.state.args, "tokens_only", False):
74
- dedicated_mode = "`--tokens-only`"
75
-
76
- enabled = envs.VLLM_ENABLE_SCALE_OUT_ENDPOINTS
77
- if dedicated_mode is not None and enabled is False:
78
- raise ValueError(
79
- "`VLLM_ENABLE_SCALE_OUT_ENDPOINTS=0` conflicts with "
80
- f"{dedicated_mode}; unset it or set it to 1"
81
- )
82
- if dedicated_mode is None and not enabled:
69
+ args = getattr(app.state, "args", None)
70
+ # `vllm launch render` and `vllm serve --tokens-only` are dedicated to
71
+ # serving these endpoints, so they always register them regardless of
72
+ # `--enable-scale-out`.
73
+ dedicated_mode = "render" in supported_tasks or getattr(args, "tokens_only", False)
74
+ enabled = dedicated_mode or getattr(args, "enable_scale_out", False)
75
+ if not enabled:
76
logger.info(
84
- "Scale-out endpoints are disabled. Set "
85
- "VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 to enable them."
77
+ "Scale-out endpoints are disabled. Set --enable-scale-out to enable them."
78
)
79
return
80
vllm/envs.py
-17
@@ -261,7 +261,6 @@ if TYPE_CHECKING:
261
VLLM_ALLOW_CHUNKED_LOCAL_ATTN_WITH_HYBRID_KV_CACHE: bool = True
262
VLLM_ENABLE_RESPONSES_API_STORE: bool = False
263
VLLM_ENABLE_COHERE_API: bool = False
264
- VLLM_ENABLE_SCALE_OUT_ENDPOINTS: bool | None = None
264
VLLM_HAS_FLASHINFER_CUBIN: bool = False
265
VLLM_ROCM_FP8_MFMA_PAGE_ATTN: bool = False
266
VLLM_ALLREDUCE_USE_SYMM_MEM: bool = True
@@ -352,15 +351,6 @@ def maybe_convert_bool(value: str | None) -> bool | None:
351
return bool(int(value))
352
353
355
-def maybe_convert_scale_out_endpoints(value: str | None) -> bool | None:
356
- if value is None:
357
- return None
358
- normalized = value.strip()
359
- if normalized not in ("0", "1"):
360
- raise ValueError("VLLM_ENABLE_SCALE_OUT_ENDPOINTS must be 0 or 1")
361
- return maybe_convert_bool(normalized)
362
-
363
-
354
def maybe_convert_json_str_or_file(value: str | None) -> dict[str, Any] | None:
355
if value is None:
356
return None
@@ -1858,12 +1848,6 @@ environment_variables: dict[str, Callable[[], Any]] = {
1848
"VLLM_ENABLE_COHERE_API": lambda: bool(
1849
int(os.getenv("VLLM_ENABLE_COHERE_API", "0"))
1850
),
1861
- # If set to 1, expose the scale-out endpoints on `vllm serve`.
1862
- # The dedicated `vllm launch render` server exposes render and derender
1863
- # endpoints when this variable is unset or set to 1.
1864
- "VLLM_ENABLE_SCALE_OUT_ENDPOINTS": lambda: maybe_convert_scale_out_endpoints(
1865
- os.getenv("VLLM_ENABLE_SCALE_OUT_ENDPOINTS") or None
1866
- ),
1851
# If set, use the fp8 mfma in rocm paged attention.
1852
"VLLM_ROCM_FP8_MFMA_PAGE_ATTN": lambda: bool(
1853
int(os.getenv("VLLM_ROCM_FP8_MFMA_PAGE_ATTN", "0"))
@@ -2289,7 +2273,6 @@ def compile_factors() -> dict[str, object]:
2273
"VLLM_CONFIG_ROOT",
2274
"LD_LIBRARY_PATH",
2275
"VLLM_SERVER_DEV_MODE",
2292
- "VLLM_ENABLE_SCALE_OUT_ENDPOINTS",
2276
"VLLM_DP_MASTER_IP",
2277
"VLLM_DP_MASTER_PORT",
2278
"VLLM_NIXL_SIDE_CHANNEL_HOST",