| 1 | # Security |
| 2 | |
| 3 | ## Inter-Node Communication |
| 4 | |
| 5 | All communications between nodes in a multi-node vLLM deployment are **insecure by default** and must be protected by placing the nodes on an isolated network. This includes: |
| 6 | |
| 7 | 1. PyTorch Distributed communications |
| 8 | 2. KV cache transfer communications |
| 9 | 3. Tensor, Pipeline, and Data parallel communications |
| 10 | |
| 11 | ### Configuration Options for Inter-Node Communications |
| 12 | |
| 13 | The following options control internode communications in vLLM: |
| 14 | |
| 15 | #### 1. **Environment Variables:** |
| 16 | |
| 17 | - `VLLM_HOST_IP`: Sets the IP address for vLLM processes to communicate on |
| 18 | |
| 19 | #### 2. **KV Cache Transfer Configuration:** |
| 20 | |
| 21 | - `--kv-ip`: The IP address for KV cache transfer communications (default: 127.0.0.1) |
| 22 | - `--kv-port`: The port for KV cache transfer communications (default: 14579) |
| 23 | |
| 24 | #### 3. **Data Parallel Configuration:** |
| 25 | |
| 26 | - `data_parallel_master_ip`: IP of the data parallel master (default: 127.0.0.1) |
| 27 | - `data_parallel_master_port`: Port of the data parallel master (default: 29500) |
| 28 | |
| 29 | ### Notes on PyTorch Distributed |
| 30 | |
| 31 | vLLM uses PyTorch's distributed features for some internode communication. For |
| 32 | detailed information about PyTorch Distributed security considerations, please |
| 33 | refer to the [PyTorch Security |
| 34 | Guide](https://github.com/pytorch/pytorch/security/policy#using-distributed-features). |
| 35 | |
| 36 | Key points from the PyTorch security guide: |
| 37 | |
| 38 | - PyTorch Distributed features are intended for internal communication only |
| 39 | - They are not built for use in untrusted environments or networks |
| 40 | - No authorization protocol is included for performance reasons |
| 41 | - Messages are sent unencrypted |
| 42 | - Connections are accepted from anywhere without checks |
| 43 | |
| 44 | ## Security Recommendations |
| 45 | |
| 46 | ### 1. **Network Isolation:** |
| 47 | |
| 48 | - Deploy vLLM nodes on a dedicated, isolated network |
| 49 | - Use network segmentation to prevent unauthorized access |
| 50 | - Implement appropriate firewall rules |
| 51 | |
| 52 | ### 2. **Configuration Best Practices:** |
| 53 | |
| 54 | - Always set `VLLM_HOST_IP` to a specific IP address rather than using defaults |
| 55 | - Configure firewalls to only allow necessary ports between nodes |
| 56 | |
| 57 | ### 3. **Access Control:** |
| 58 | |
| 59 | - Restrict physical and network access to the deployment environment |
| 60 | - Implement proper authentication and authorization for management interfaces |
| 61 | - Follow the principle of least privilege for all system components |
| 62 | |
| 63 | ### 4. **Restrict Domains Access for Media URLs:** |
| 64 | |
| 65 | Restrict domains that vLLM can access for media URLs by setting |
| 66 | `--allowed-media-domains` to prevent Server-Side Request Forgery (SSRF) attacks. |
| 67 | (e.g. `--allowed-media-domains upload.wikimedia.org github.com www.bogotobogo.com`) |
| 68 | |
| 69 | This protection applies to both the online serving API (multimodal inputs) and |
| 70 | the **batch runner** (`vllm run-batch`), where `file_url` values in batch |
| 71 | transcription/translation requests are validated against the same allowlist. |
| 72 | |
| 73 | Without domain restrictions, a malicious user could supply URLs that: |
| 74 | |
| 75 | - **Target internal services**: Access internal network endpoints, cloud metadata |
| 76 | services (e.g. `169.254.169.254`), or other services not intended to be |
| 77 | publicly reachable (SSRF). |
| 78 | - **Consume excessive resources**: Point to extremely large files or slow |
| 79 | endpoints, causing the server to download unbounded amounts of data and |
| 80 | exhausting memory, disk, or network bandwidth. |
| 81 | |
| 82 | By explicitly allowlisting only the domains you expect media to come from, you |
| 83 | significantly reduce the attack surface for these types of abuse. |
| 84 | |
| 85 | Also, consider setting `VLLM_MEDIA_URL_ALLOW_REDIRECTS=0` to prevent HTTP |
| 86 | redirects from being followed to bypass domain restrictions. |
| 87 | |
| 88 | ### 5. **Restrict Media Download and Decode Sizes:** |
| 89 | |
| 90 | Remote media responses and compressed media files can expand into gigabytes of |
| 91 | memory. vLLM enforces download and decode-size limits to prevent out-of-memory |
| 92 | denial of service: |
| 93 | |
| 94 | | Environment Variable | Default | Description | |
| 95 | | --- | --- | --- | |
| 96 | | `VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB` | `256` | Maximum size in MB for a single remote media response. Oversized responses are rejected while streaming before the full body is materialized in memory. | |
| 97 | | `VLLM_MAX_IMAGE_PIXELS` | `178956970` (~179M pixels) | Maximum decoded image size in pixels. Images exceeding this are rejected before raster memory is allocated. Default matches PIL's built-in 2x decompression-bomb threshold (~680 MB for RGB). | |
| 98 | | `VLLM_MAX_AUDIO_CLIP_FILESIZE_MB` | `25` | Maximum compressed filesize in MB for a single audio file. Enforced on all audio inputs (multimodal chat URLs, speech-to-text uploads, data: URLs, and local file paths) before decoding begins. | |
| 99 | | `VLLM_MAX_AUDIO_DECODE_DURATION_S` | `600` | Maximum decoded audio duration in seconds. Prevents compressed audio from expanding into gigabytes of float32 PCM. | |
| 100 | | `VLLM_MAX_AUDIO_DECODE_BYTES` | `268435456` (256 MiB) | Maximum float32 PCM bytes that audio decoding may allocate. Guards against sample-rate forgery where an inflated header sample rate bypasses the duration guard while the actual frame count causes a multi-GiB allocation. | |
| 101 | | `VLLM_MAX_EMBED_DECODE_BYTES` | `2147483648` (2 GiB) | Maximum bytes a client-supplied embedding payload (`prompt_embeds`, `image_embeds`, `audio_embeds`, `video_embeds`) may allocate once densified. A sparse tensor carries its own declared shape, so a payload of a few hundred bytes can expand into hundreds of GiB. Checked before `to_dense()`, so the memory is never allocated. Set to `0` to disable. | |
| 102 | |
| 103 | Setting any of these to `0` disables the corresponding limit. This is **not |
| 104 | recommended** for deployments exposed to untrusted users, as it removes the |
| 105 | protection against resource-exhaustion attacks. |
| 106 | |
| 107 | ## Security and Firewalls: Protecting Exposed vLLM Systems |
| 108 | |
| 109 | While vLLM is designed to allow unsafe network services to be isolated to |
| 110 | private networks, there are components—such as dependencies and underlying |
| 111 | frameworks—that may open insecure services listening on all network interfaces, |
| 112 | sometimes outside of vLLM's direct control. |
| 113 | |
| 114 | A major concern is the use of `torch.distributed`, which vLLM leverages for |
| 115 | distributed communication, including when using vLLM on a single host. When vLLM |
| 116 | uses TCP initialization (see [PyTorch TCP Initialization |
| 117 | documentation](https://docs.pytorch.org/docs/stable/distributed.html#tcp-initialization)), |
| 118 | PyTorch creates a `TCPStore` that, by default, listens on all network |
| 119 | interfaces. This means that unless additional protections are put in place, |
| 120 | these services may be accessible to any host that can reach your machine via any |
| 121 | network interface. |
| 122 | |
| 123 | **From a PyTorch perspective, any use of `torch.distributed` should be |
| 124 | considered insecure by default.** This is a known and intentional behavior from |
| 125 | the PyTorch team. |
| 126 | |
| 127 | ### Firewall Configuration Guidance |
| 128 | |
| 129 | The best way to protect your vLLM system is to carefully configure a firewall to |
| 130 | expose only the minimum network surface area necessary. In most cases, this |
| 131 | means: |
| 132 | |
| 133 | - **Block all incoming connections except to the TCP port the API server is |
| 134 | listening on.** |
| 135 | |
| 136 | - Ensure that ports used for internal communication (such as those for |
| 137 | `torch.distributed` and KV cache transfer) are only accessible from trusted |
| 138 | hosts or networks. |
| 139 | |
| 140 | - Never expose these internal ports to the public internet or untrusted |
| 141 | networks. |
| 142 | |
| 143 | Consult your operating system or application platform documentation for specific |
| 144 | firewall configuration instructions. |
| 145 | |
| 146 | ## API Key Authentication Limitations |
| 147 | |
| 148 | ### Overview |
| 149 | |
| 150 | The `--api-key` flag (or `VLLM_API_KEY` environment variable) provides authentication for vLLM's HTTP server, but **only for endpoints under the `/v1`, `/v2`, `/inference`, and `/cohere` path prefixes**. Many other sensitive endpoints are exposed on the same HTTP server without any authentication enforcement. |
| 151 | |
| 152 | **Important:** Do not rely exclusively on `--api-key` for securing access to vLLM. Additional security measures are required for production deployments. |
| 153 | |
| 154 | ### Protected Endpoints (Require API Key) |
| 155 | |
| 156 | When `--api-key` is configured, the following endpoints require Bearer token authentication: |
| 157 | |
| 158 | - `/v1/models` - List available models |
| 159 | - `/v1/chat/completions` - Chat completions |
| 160 | - `/v1/chat/completions/batch` - Batch chat completions |
| 161 | - `/v1/chat/completions/render` - Render chat completion requests (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render`) |
| 162 | - `/v1/chat/completions/derender` - Derender chat completion requests (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render`) |
| 163 | - `/v1/completions` - Text completions |
| 164 | - `/v1/completions/render` - Render completion requests (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render`) |
| 165 | - `/v1/completions/derender` - Derender completion requests (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render`) |
| 166 | - `/v1/embeddings` - Generate embeddings |
| 167 | - `/v1/audio/transcriptions` - Audio transcription |
| 168 | - `/v1/audio/translations` - Audio translation |
| 169 | - `/v1/messages` - Anthropic-compatible messages API |
| 170 | - `/v1/messages/render` - Render Anthropic-compatible messages (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render`) |
| 171 | - `/v1/messages/count_tokens` - Count tokens for Anthropic messages |
| 172 | - `/v1/responses` - Create a response |
| 173 | - `/v1/responses/render` - Render a self-contained response request (available on `vllm serve` only when `--enable-scale-out` is set, or on `vllm launch render` unless explicitly disabled) |
| 174 | - `/v1/responses/{response_id}` - Retrieve a response |
| 175 | - `/v1/responses/{response_id}/cancel` - Cancel a response |
| 176 | - `/v1/score` - Scoring API |
| 177 | - `/v1/rerank` - Reranking API |
| 178 | - `/v1/load_lora_adapter` - Load a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`) |
| 179 | - `/v1/unload_lora_adapter` - Unload a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`) |
| 180 | - `/inference/v1/generate` - Generate completions (available when `--enable-scale-out` is set, or with `--tokens-only`) |
| 181 | - `/cohere/v2/chat/render` - Render Cohere Chat v2 requests (requires `VLLM_ENABLE_COHERE_API=1` and either `--enable-scale-out` or `--tokens-only`) |
| 182 | - `/v2/embed` - Cohere Embed API |
| 183 | - `/v2/rerank` - Cohere Rerank API |
| 184 | |
| 185 | ### Unprotected Endpoints (No API Key Required) |
| 186 | |
| 187 | The following endpoints **do not require authentication** even when `--api-key` is configured: |
| 188 | |
| 189 | **Inference endpoints:** |
| 190 | |
| 191 | - `/invocations` - SageMaker-compatible endpoint (routes to the same inference functions as `/v1` endpoints) |
| 192 | - `/generative_scoring` - Generative scoring API |
| 193 | - `/pooling` - Pooling API |
| 194 | - `/classify` - Classification API |
| 195 | - `/score` - Scoring API (non-`/v1` variant) |
| 196 | - `/rerank` - Reranking API (non-`/v1` variant) |
| 197 | |
| 198 | **Operational control endpoints (only when `"generate"` task is supported):** |
| 199 | |
| 200 | - `/pause` - Pause generation (causes denial of service) |
| 201 | - `/resume` - Resume generation |
| 202 | - `/is_paused` - Check if generation is paused |
| 203 | - `/abort_requests` - Abort in-flight requests (causes loss of in-flight work) |
| 204 | - `/scale_elastic_ep` - Trigger scaling operations |
| 205 | - `/is_scaling_elastic_ep` - Check if scaling is in progress |
| 206 | - `/init_weight_transfer_engine` - Initialize weight transfer engine for RLHF |
| 207 | - `/update_weights` - Update model weights (can alter model behavior) |
| 208 | - `/get_world_size` - Get distributed world size |
| 209 | - `/abort_requests` - Abort in-flight requests (available with `--tokens-only`) |
| 210 | |
| 211 | **Utility endpoints:** |
| 212 | |
| 213 | - `/tokenize` - Tokenize text (not gated by `--enable-scale-out`) |
| 214 | - `/detokenize` - Detokenize tokens |
| 215 | - `/health` - Health check |
| 216 | - `/ping` - SageMaker health check |
| 217 | - `/version` - Version information |
| 218 | - `/load` - Server load metrics |
| 219 | |
| 220 | **Tokenizer information endpoint (only when `--enable-tokenizer-info-endpoint` is set):** |
| 221 | |
| 222 | This endpoint is **only available when the `--enable-tokenizer-info-endpoint` flag is set**. It may expose sensitive information such as chat templates and tokenizer configuration: |
| 223 | |
| 224 | - `/tokenizer_info` - Get comprehensive tokenizer information including chat templates and configuration |
| 225 | |
| 226 | **Development endpoints (only when `VLLM_SERVER_DEV_MODE=1`):** |
| 227 | |
| 228 | These endpoints are **only available when the environment variable `VLLM_SERVER_DEV_MODE` is set to `1`**. They are intended for development and debugging purposes and should never be enabled in production: |
| 229 | |
| 230 | - `/server_info` - Get detailed server configuration |
| 231 | - `/reset_prefix_cache` - Reset prefix cache (can disrupt service) |
| 232 | - `/reset_mm_cache` - Reset multimodal cache (can disrupt service) |
| 233 | - `/reset_encoder_cache` - Reset encoder cache (can disrupt service) |
| 234 | - `/sleep` - Put engine to sleep (causes denial of service) |
| 235 | - `/wake_up` - Wake engine from sleep |
| 236 | - `/is_sleeping` - Check if engine is sleeping |
| 237 | - `/collective_rpc` - Execute arbitrary RPC methods on the engine (extremely dangerous) |
| 238 | |
| 239 | **Profiler endpoints (only when profiling is enabled via `--profiler-config`):** |
| 240 | |
| 241 | These endpoints are only available when profiling is enabled and should only be used for local development: |
| 242 | |
| 243 | - `/start_profile` - Start PyTorch profiler |
| 244 | - `/stop_profile` - Stop PyTorch profiler |
| 245 | |
| 246 | **Note:** The `/invocations` endpoint is particularly concerning as it provides unauthenticated access to the same inference capabilities as the protected `/v1` endpoints. |
| 247 | |
| 248 | ### Security Implications |
| 249 | |
| 250 | An attacker who can reach the vLLM HTTP server can: |
| 251 | |
| 252 | 1. **Bypass authentication** by using endpoints outside the protected path prefixes, such as `/invocations`, `/generative_scoring`, `/pooling`, `/classify`, `/score`, or `/rerank`, to run arbitrary inference without credentials |
| 253 | 2. **Cause denial of service** by calling `/pause`, `/scale_elastic_ep`, or `/abort_requests` without a token |
| 254 | 3. **Access operational controls** to manipulate server state (e.g., pausing generation, updating model weights via `/update_weights`) |
| 255 | 4. **If `--enable-tokenizer-info-endpoint` is set:** Access sensitive tokenizer configuration including chat templates, which may reveal prompt engineering strategies or other implementation details |
| 256 | 5. **If `VLLM_SERVER_DEV_MODE=1` is set:** Execute arbitrary RPC commands via `/collective_rpc`, reset caches, put the engine to sleep, and access detailed server configuration |
| 257 | |
| 258 | ### Recommended Security Practices |
| 259 | |
| 260 | #### 1. Minimize Exposed Endpoints |
| 261 | |
| 262 | **CRITICAL:** Never set `VLLM_SERVER_DEV_MODE=1` in production environments. Development endpoints expose extremely dangerous functionality including: |
| 263 | |
| 264 | - Arbitrary RPC execution via `/collective_rpc` |
| 265 | - Cache manipulation that can disrupt service |
| 266 | - Detailed server configuration disclosure |
| 267 | |
| 268 | Similarly, never enable profiler endpoints in production. |
| 269 | |
| 270 | **Be cautious with `--enable-tokenizer-info-endpoint`:** Only enable the `/tokenizer_info` endpoint if you need to expose tokenizer configuration information. This endpoint reveals chat templates and tokenizer settings that may contain sensitive implementation details or prompt engineering strategies. |
| 271 | |
| 272 | #### 2. Deploy Behind a Reverse Proxy |
| 273 | |
| 274 | The most effective approach is to deploy vLLM behind a reverse proxy (such as nginx, Envoy, or a Kubernetes Gateway) that: |
| 275 | |
| 276 | - Explicitly allowlists only the endpoints you want to expose to end users |
| 277 | - Blocks all other endpoints, including the unauthenticated inference and operational control endpoints |
| 278 | - Implements additional authentication, rate limiting, and logging at the proxy layer |
| 279 | |
| 280 | ## Request Parameter Resource Limits |
| 281 | |
| 282 | Certain API request parameters can have a large impact on resource consumption and may be abused to exhaust server resources. The `n` parameter in the `/v1/completions` and `/v1/chat/completions` endpoints controls how many independent output sequences are generated per request. A very large value causes the engine to allocate memory, CPU, and GPU time proportional to `n`, which can lead to out-of-memory conditions on the host and block the server from processing other requests. |
| 283 | |
| 284 | To mitigate this, vLLM enforces a configurable upper bound on the `n` parameter via the `VLLM_MAX_N_SEQUENCES` environment variable (default: **16384**). Requests exceeding this limit are rejected before reaching the engine. |
| 285 | |
| 286 | ### Recommendations |
| 287 | |
| 288 | - **Public-facing deployments:** Consider setting `VLLM_MAX_N_SEQUENCES` to a value appropriate for your workload (e.g., `64` or `128`) to limit the blast radius of a single request. |
| 289 | - **Reverse proxy layer:** In addition to vLLM's built-in limit, consider enforcing request body validation and rate limiting at your reverse proxy to further constrain abusive payloads. |
| 290 | - **Monitoring:** Monitor per-request resource consumption to detect anomalous patterns that may indicate abuse. |
| 291 | |
| 292 | ### Per-request multimodal arguments |
| 293 | |
| 294 | API server endpoints reject non-empty per-request `mm_processor_kwargs` and |
| 295 | `media_io_kwargs` by default. These arguments can change image, video, or audio |
| 296 | loading, sizing, sampling, and preprocessing behavior, causing excessive CPU, |
| 297 | GPU, or memory use when controlled by an untrusted client. Server-level |
| 298 | `--mm-processor-kwargs` and `--media-io-kwargs` remain available for deployment |
| 299 | configuration. |
| 300 | |
| 301 | Only deployments whose API clients are trusted should start the server with |
| 302 | `--trust-request-mm-kwargs` to restore per-request overrides. Do not enable |
| 303 | this option on an endpoint exposed to untrusted clients. |
| 304 | |
| 305 | ## Tool Server and MCP Security |
| 306 | |
| 307 | vLLM supports connecting to external tool servers via the `--tool-server` argument. This enables models to call tools through the Responses API (`/v1/responses`). Tool server support works with all models — it is not limited to specific model architectures. |
| 308 | |
| 309 | **Important:** No tool servers are enabled by default. They must be explicitly opted into via configuration. |
| 310 | |
| 311 | ### Built-in Demo Tools (GPT-OSS) |
| 312 | |
| 313 | Passing `--tool-server demo` enables built-in demo tools that work with any model that supports tool calling. The tool implementations are not part of vLLM — they are provided by the separately installed [`gpt-oss`](https://github.com/openai/gpt-oss) package. vLLM provides thin wrappers that delegate to `gpt-oss`. |
| 314 | |
| 315 | - **Code interpreter** (`python`): Python execution via Docker (via `gpt_oss.tools.python_docker`) |
| 316 | - **Web browser** (`browser`): Search via Exa API, requires `EXA_API_KEY` (via `gpt_oss.tools.simple_browser`) |
| 317 | |
| 318 | #### Code Interpreter (Python Tool) Security Risks |
| 319 | |
| 320 | The code interpreter executes model-generated code inside a Docker container. However, the container is **not configured with network isolation by default**. It inherits the host's Docker networking configuration (e.g., default bridge network or `--network=host`), which means: |
| 321 | |
| 322 | - The container may be able to access the host network and LAN. |
| 323 | - Internal services reachable from the container may be exploited via SSRF (Server-Side Request Forgery). |
| 324 | - Cloud metadata services (e.g., `169.254.169.254`) may be accessible. |
| 325 | - If vulnerable internal services (such as `torch.distributed` endpoints) are reachable from the container, this could be used to attack them. |
| 326 | |
| 327 | This is particularly concerning because the code being executed is generated by the model, which may be influenced by adversarial inputs (prompt injection). |
| 328 | |
| 329 | #### Controlling Built-in Tool Availability |
| 330 | |
| 331 | Built-in demo tools are controlled by two settings: |
| 332 | |
| 333 | 1. **`--tool-server demo`**: Enables the built-in demo tools (browser and Python code interpreter). |
| 334 | |
| 335 | 2. **`VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS`**: When built-in tools are requested via the `mcp` tool type in the Responses API, this comma-separated allowlist controls which tool labels are permitted. Valid values are: |
| 336 | - `container` - Container tool |
| 337 | - `code_interpreter` - Python code execution tool |
| 338 | - `web_search_preview` - Web search/browser tool |
| 339 | |
| 340 | If this variable is not set or is empty, no built-in tools requested via MCP tool type will be enabled. |
| 341 | |
| 342 | To disable the Python code interpreter specifically, omit `code_interpreter` from `VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS`. |
| 343 | |
| 344 | **Consider a custom implementation**: The GPT-OSS Python tool is a reference implementation. For production deployments, consider implementing a custom code execution sandbox with stricter isolation guarantees. See the [GPT-OSS documentation](https://github.com/openai/gpt-oss?tab=readme-ov-file#python) for guidance. |
| 345 | |
| 346 | ## Dynamic LoRA Loading |
| 347 | |
| 348 | vLLM supports dynamically loading and unloading LoRA adapters at runtime via the `/v1/load_lora_adapter` and `/v1/unload_lora_adapter` API endpoints. This functionality is **not enabled by default** — it requires both `--enable-lora` and the environment variable `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` to be set. |
| 349 | |
| 350 | **Warning:** Dynamic LoRA loading is not a secure operation and should not be enabled in deployments exposed to untrusted clients. If you must enable dynamic LoRA loading, restrict access to the `/v1/load_lora_adapter` and `/v1/unload_lora_adapter` endpoints to trusted administrators only, using a reverse proxy or network-level access controls. Do not expose these endpoints to end users. For details on configuring LoRA adapters, see the [LoRA Adapters documentation](../features/lora.md). |
| 351 | |
| 352 | ## Endpoint Plugins |
| 353 | |
| 354 | vLLM supports loading out-of-tree HTTP routes via the `vllm.endpoint_plugins` entry point group (see [Endpoint Plugins](../design/endpoint_plugins.md) for how to write one). An endpoint plugin can register arbitrary FastAPI routes, including routes that reach the engine via `EngineClient.collective_rpc`, so it must be treated as part of the server's trusted code base and not as sandboxed or reviewed input. |
| 355 | |
| 356 | **Endpoint plugins are not loaded by default.** Unlike other vLLM plugin groups (`vllm.general_plugins`, `vllm.platform_plugins`, etc.), which load every discovered plugin unless `VLLM_PLUGINS` narrows the set, endpoint plugins load **none** unless `VLLM_PLUGINS` is set and explicitly names them. This mirrors the "off by default in production" posture used for development endpoints gated behind `VLLM_SERVER_DEV_MODE`. Both surfaces are only present when an operator has explicitly opted in. |
| 357 | |
| 358 | ### Recommended Security Practices |
| 359 | |
| 360 | 1. **Only allowlist plugins you trust.** Set `VLLM_PLUGINS` to the exact plugin names you intend to run and never wildcard or copy an allowlist between deployments without reviewing what each named plugin does. |
| 361 | 2. **Audit routes before deploying.** A plugin's `attach_router` can add routes under any path, including ones that duplicate existing `/v1/*` paths. There is currently no route conflict enforcement (tracked as a follow-up to RFC [#46565](https://github.com/vllm-project/vllm/issues/46565)), so a malicious or buggy plugin can **shadow a core route** and silently replace its behavior. Prefer plugins that namespace their routes under a distinct prefix (e.g. `/plugins/<plugin-name>/...`) instead of reusing `/v1/...` and review `app.routes` after startup if you need certainty about what is actually being served. |
| 362 | 3. **Treat plugin routes like any other unauthenticated by default surface.** `--api-key` only protects the `/v1`, `/v2`, `/inference`, and `/cohere` path prefixes (see [API Key Authentication Limitations](#api-key-authentication-limitations)). A plugin route outside those prefixes is unauthenticated unless the plugin implements its own authentication. Deploy behind a reverse proxy that allowlists only the plugin routes you intend to expose externally. |
| 363 | 4. **Remember the `vllm.general_plugins` pairing.** A plugin that also needs new engine side behavior ships that half separately via `vllm.general_plugins` which loads in every worker process under the default (load all unless restricted) posture. Allowlisting the endpoint plugin does not by itself restrict its paired engine side plugin. Need to review both. |
| 364 | |
| 365 | ## gRPC Interface |
| 366 | |
| 367 | vLLM provides optional gRPC `Inference` and `Control` services on a separate TCP port, enabled via the `--grpc-port` flag. When not specified, no gRPC server is started. The gRPC listener binds to the same host address as the HTTP server. |
| 368 | |
| 369 | **Warning:** The gRPC interface is **insecure by default** — it does not implement authentication, authorization, or encryption. It should be considered a private, internal interface intended for use only between co-located services within a trusted network. Do not expose the gRPC port to the public internet or untrusted clients. If you enable the gRPC interface, protect it via network-level access controls such as firewall rules, network segmentation, or deployment on an isolated private network. |
| 370 | |
| 371 | ### Security Implications |
| 372 | |
| 373 | An attacker who can reach the gRPC port can: |
| 374 | |
| 375 | 1. **Run arbitrary inference** via the `Generate` and `GenerateStream` RPCs without any credentials |
| 376 | 2. **Mutate engine state** by pausing generation, sleeping the engine, or initiating configured RL weight updates through the `Control` service |
| 377 | 3. **Consume GPU and compute resources** by submitting unbounded generation requests |
| 378 | 4. **Stop a managed engine** through `Control.Shutdown`, or cause denial of service by exploiting bugs in the gRPC interface. |
| 379 | |
| 380 | ### Recommendations |
| 381 | |
| 382 | - Only enable `--grpc-port` when you have a specific need for gRPC-based inference |
| 383 | - Ensure the gRPC port is only accessible from trusted hosts or services |
| 384 | - Use firewall rules to block external access to the gRPC port |
| 385 | - Consider deploying the gRPC interface on a dedicated internal network interface |
| 386 | |
| 387 | ## Cache Directory Security |
| 388 | |
| 389 | vLLM assumes that its cache directories are **private and trusted**. Cache contents are loaded without cryptographic integrity verification, including formats that support arbitrary code execution. If an untrusted user or process can write to vLLM's cache directories, they may be able to crash vLLM or cause it to execute arbitrary code. |
| 390 | |
| 391 | **Do not share vLLM cache directories with untrusted users or mount them from untrusted storage.** Treat the cache directory with the same care as the vLLM installation itself. |
| 392 | |
| 393 | ### Cache Directory Configuration |
| 394 | |
| 395 | Most cache paths default to subdirectories under a single root. Changing `VLLM_CACHE_ROOT` changes the default location for all features that inherit from it. When `torch.compile` caching is enabled (the default), vLLM also redirects `TRITON_CACHE_DIR` into this tree. If compile caching is disabled, Triton falls back to its own default location (`~/.triton/cache`). |
| 396 | |
| 397 | | Environment Variable | Default | Description | |
| 398 | | --- | --- | --- | |
| 399 | | `VLLM_CACHE_ROOT` | `~/.cache/vllm` | Base cache directory. Respects `XDG_CACHE_HOME` if set. All paths below inherit from this unless explicitly overridden. | |
| 400 | | *(torch.compile)* | `$VLLM_CACHE_ROOT/torch_compile_cache/` | Compilation cache for AOT-compiled models, Inductor graphs, and Triton kernels. Controlled by `VLLM_DISABLE_COMPILE_CACHE` (set to `1` to disable). | |
| 401 | | `VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR` | `$VLLM_CACHE_ROOT/flashinfer_autotune_cache/<flashinfer-version>/<arch>/<cache-hash>/` | FlashInfer autotune config cache. | |
| 402 | | `VLLM_ASSETS_CACHE` | `$VLLM_CACHE_ROOT/assets/` | Downloaded assets (e.g., tokenizer files). | |
| 403 | | `VLLM_XLA_CACHE_PATH` | `$VLLM_CACHE_ROOT/xla_cache/` | XLA/TPU compilation cache. | |
| 404 | | `VLLM_MEDIA_CACHE` | *(disabled)* | Optional cache for downloaded media (images, video, audio). Not enabled unless explicitly set. | |
| 405 | |
| 406 | ### Recommendations |
| 407 | |
| 408 | - **Restrict file permissions** on `VLLM_CACHE_ROOT` (and any other cache directories used by dependencies, such as `~/.triton` if compile caching is disabled) so that only the vLLM process owner can read and write to them. |
| 409 | - **Do not copy cache contents from untrusted sources.** If you distribute cache artifacts between environments, ensure they originate from a trusted build pipeline. |
| 410 | - **Container deployments:** If mounting cache directories into containers, ensure the volume source is trusted. |
| 411 | |
| 412 | ## FIPS Compatibility |
| 413 | |
| 414 | FIPS compliance depends on many factors, so a vLLM deployment is not automatically FIPS compliant. Recent changes have improved vLLM's *tolerance* of FIPS-enabled hosts — that is, avoiding crashes when non-approved algorithms are blocked — but tolerance is not the same as compliance. Whether a deployment satisfies FIPS requirements depends on the host operating system, the OpenSSL provider backing Python's `hashlib` and `ssl` modules, and which optional dependencies are installed. |
| 415 | |
| 416 | ### FIPS-relevant configuration |
| 417 | |
| 418 | Operators running vLLM on FIPS-enabled hosts should select FIPS-approved algorithms via the following knobs: |
| 419 | |
| 420 | - **Multimodal input hashing** — `--mm-hasher-algorithm` (config field `mm_hasher_algorithm`) defaults to `blake3`, which is not FIPS-approved. Set it to `sha256` or `sha512` in FIPS-enabled environments. |
| 421 | - **Prefix-cache hashing** — set `--prefix-caching-hash-algo` (config field `prefix_caching_hash_algo`) to `sha256` or `sha256_cbor`. The `xxhash` and `xxhash_cbor` options are not FIPS-approved. |
| 422 | - **TLS ciphers** — use `--ssl-ciphers` to restrict the API server's TLS handshake to FIPS-approved cipher suites that match your environment's policy. |
| 423 | |
| 424 | ### Automatic fallback for non-security MD5 use |
| 425 | |
| 426 | vLLM uses MD5 in a few places to derive non-security cache keys (for example, configuration hashes). These call sites pass `usedforsecurity=False` and additionally fall back to SHA-256 when the underlying OpenSSL provider refuses MD5 outright (see `safe_hash()` in `vllm/utils/hashing.py`). No user action is required; this behavior is documented so that auditors and security reviewers can identify the MD5 references and understand their purpose. |
| 427 | |
| 428 | ### Dependencies that provide non-FIPS hash implementations |
| 429 | |
| 430 | Some dependencies expose hash implementations that are not FIPS-approved. vLLM only invokes them when the corresponding algorithm is selected, but operators with strict cryptographic controls may want to ensure the code paths are not exercised — and, where policy requires, that the packages themselves are absent: |
| 431 | |
| 432 | - `blake3` — currently listed in `requirements/common.txt`, so a standard |
| 433 | install pulls it in. It is imported lazily and only used when |
| 434 | `mm_hasher_algorithm=blake3` (the default). Setting |
| 435 | `--mm-hasher-algorithm sha256` or `--mm-hasher-algorithm sha512` is sufficient |
| 436 | to keep the non-FIPS code path dormant. If your policy additionally forbids |
| 437 | the package being present, uninstall it after installation; vLLM will |
| 438 | continue to function as long as a non-blake3 algorithm is selected. |
| 439 | - `xxhash` — a true optional dependency (not in `requirements/common.txt`). It is only imported when an `xxhash`-based prefix-cache algorithm is selected. Leave it uninstalled and select a `sha256`-based prefix-cache algorithm. |
| 440 | |
| 441 | ### Beyond hashing: other FIPS considerations |
| 442 | |
| 443 | Hashing is the area where vLLM has explicit FIPS-aware code, but a FIPS-compliant deployment depends on several factors that sit outside vLLM itself. Operators should evaluate the following with their platform and security teams: |
| 444 | |
| 445 | - **Host crypto provider.** Python's `hashlib` and `ssl` modules are FIPS-aware only when Python is linked against a FIPS-validated OpenSSL (or equivalent) provider supplied by the host OS. vLLM inherits whatever provider the host configures — it does not bundle one. |
| 446 | - **API server TLS.** TLS termination for the OpenAI-compatible API server uses the host's OpenSSL via Python's `ssl` module. Restrict the cipher suite with `--ssl-ciphers` to match your environment's FIPS policy, and ensure server certificates are issued with FIPS-approved algorithms and key sizes. |
| 447 | - **Outbound HTTPS.** Model and asset downloads (for example, via `huggingface_hub`) use the same host TLS stack. The same provider/cipher considerations apply. |
| 448 | - **Inter-node communication is unencrypted by default.** As described in [Inter-Node Communication](#inter-node-communication), PyTorch Distributed, KV-cache transfer, and data-parallel channels do not encrypt traffic. FIPS environments that require FIPS-approved cryptography for data in transit must provide that protection externally — for example, via an mTLS sidecar or IPsec terminated by a FIPS-validated module — since vLLM's internal channels cannot satisfy the requirement on their own. Network isolation alone is not cryptography and does not meet a "FIPS-approved cryptography for data in transit" requirement, though it remains a useful defense-in-depth measure. |
| 449 | - **Dependencies that bundle their own OpenSSL.** Some Python wheels statically link OpenSSL builds that fail the kernel FIPS self-test on FIPS-enabled hosts (`FATAL FIPS SELFTEST FAILURE`). `opencv-python-headless` is a known example; other manylinux wheels may behave similarly. Audit your installed wheels for bundled crypto libraries when troubleshooting FIPS startup failures. |
| 450 | - **Accelerator and ML libraries.** PyTorch, CUDA, cuDNN, NCCL, and similar components have their own crypto and FIPS posture independent of vLLM. NVIDIA publishes FIPS-validated builds for some libraries; vLLM does not pin to those builds, so selecting and validating them is the operator's responsibility. |
| 451 | - **What is *not* a FIPS concern in vLLM.** Random number generation used for token sampling (Python/NumPy/PyTorch RNGs) is not a cryptographic use and is out of scope for FIPS. Pickled cache artifacts are a separate security concern covered under [Cache Directory Security](#cache-directory-security). |
| 452 | |
| 453 | In short: the configuration knobs above let vLLM avoid non-approved algorithms, and the automatic fallbacks let it run without crashing on FIPS-enabled hosts. End-to-end FIPS compliance, however, is a property of the full deployment — host OS, crypto provider, transitive dependencies, and network architecture — not of vLLM alone. |
| 454 | |
| 455 | ## Ray Cluster Trust Model and Environment Variable Propagation |
| 456 | |
| 457 | ### Trust Assumption |
| 458 | |
| 459 | vLLM treats the entire Ray cluster as a single trust domain. Any principal |
| 460 | with the ability to execute code within the Ray cluster (e.g. submit actors |
| 461 | or tasks) is considered to have the same level of trust as the driver/API |
| 462 | server process. This means that vLLM does **not** attempt to isolate |
| 463 | driver-side credentials from worker-side processes within the same Ray |
| 464 | cluster. |
| 465 | |
| 466 | This assumption is consistent with |
| 467 | [Ray's own security model](https://docs.ray.io/en/latest/ray-core/security.html), |
| 468 | which states that any user who can connect to a Ray cluster can run arbitrary |
| 469 | code on any node in that cluster. In other words, Ray cluster access already |
| 470 | implies full code execution on worker nodes, so restricting environment |
| 471 | variable propagation alone would not constitute a meaningful security |
| 472 | boundary. |
| 473 | |
| 474 | ### Driver-to-Worker Environment Variable Propagation |
| 475 | |
| 476 | When using `RayExecutorV2` in multi-node deployments, vLLM propagates |
| 477 | environment variables from the driver process to remote Ray workers so that |
| 478 | workers have the configuration they need to function correctly (e.g. vLLM |
| 479 | settings, NCCL tuning, Hugging Face tokens for gated model downloads). |
| 480 | |
| 481 | The propagation uses a **copy-all-except-denylist** policy via |
| 482 | `get_driver_env_vars()` in `vllm/v1/executor/ray_env_utils.py`: every |
| 483 | environment variable present in the driver's `os.environ` is sent to |
| 484 | workers, except for a small set of worker-specific variables and any |
| 485 | names the operator has explicitly excluded. |
| 486 | |
| 487 | On the worker side, propagated variables are applied with `setdefault` |
| 488 | semantics — they fill in missing variables but never overwrite values |
| 489 | already present in the worker's environment. |
| 490 | |
| 491 | ### When This Matters |
| 492 | |
| 493 | In deployments where operators intentionally scope credentials (such as |
| 494 | `HF_TOKEN`, cloud storage keys, registry tokens, or internal service |
| 495 | tokens) to the driver/API server alone — for example, when GPU workers |
| 496 | run in a different node, pod, or trust domain — the default propagation |
| 497 | behavior will copy those credentials into worker environments. A process |
| 498 | running on a worker node under the same OS user may then be able to read |
| 499 | those credentials (e.g. via `/proc/<pid>/environ` on Linux). |
| 500 | |
| 501 | If your deployment treats Ray workers as less trusted than the driver, be |
| 502 | aware that environment-variable isolation is **not** enforced by default. |
| 503 | |
| 504 | ### Hardening Recommendations |
| 505 | |
| 506 | Operators who want to limit which environment variables are propagated to |
| 507 | Ray workers can use the following mechanisms: |
| 508 | |
| 509 | #### 1. Denylist via Configuration File |
| 510 | |
| 511 | Create a JSON file at `$VLLM_CONFIG_ROOT/ray_non_carry_over_env_vars.json` |
| 512 | (default: `~/.config/vllm/ray_non_carry_over_env_vars.json`) containing an |
| 513 | array of environment variable names to exclude from propagation: |
| 514 | |
| 515 | ```json |
| 516 | [ |
| 517 | "HF_TOKEN", |
| 518 | "AWS_SECRET_ACCESS_KEY", |
| 519 | "AWS_SESSION_TOKEN", |
| 520 | "GOOGLE_APPLICATION_CREDENTIALS", |
| 521 | "AZURE_CLIENT_SECRET", |
| 522 | "REGISTRY_TOKEN", |
| 523 | "MY_INTERNAL_SERVICE_KEY" |
| 524 | ] |
| 525 | ``` |
| 526 | |
| 527 | Any variable listed here will **not** be copied from the driver to workers. |
| 528 | |
| 529 | #### 2. Minimize Driver Environment |
| 530 | |
| 531 | Rather than setting credentials in the driver's shell environment, inject |
| 532 | them through a secrets manager, a mounted file, or a short-lived |
| 533 | subprocess so that they are not present in `os.environ` when vLLM starts. |
| 534 | |
| 535 | #### 3. Network and Process Isolation |
| 536 | |
| 537 | - Restrict `procfs` visibility on worker nodes (e.g. mount `/proc` with |
| 538 | `hidepid=2` or use a container runtime that isolates `/proc` between |
| 539 | pods) so that same-UID processes cannot read each other's |
| 540 | `/proc/<pid>/environ`. |
| 541 | - Run the driver and workers under different OS users or in separate |
| 542 | containers with non-overlapping UIDs. |
| 543 | |
| 544 | #### 4. Limit Ray Cluster Access |
| 545 | |
| 546 | Because Ray cluster access is equivalent to arbitrary code execution, |
| 547 | ensure that only trusted principals can submit work to the cluster: |
| 548 | |
| 549 | - Use Ray's TLS authentication to restrict cluster membership. |
| 550 | - Place the Ray cluster on an isolated network segment. |
| 551 | - Do not expose the Ray client port or dashboard to untrusted networks. |
| 552 | |
| 553 | ## Prefix Cache Timing Side-Channel Mitigation (Cache Salting) |
| 554 | |
| 555 | ### Background |
| 556 | |
| 557 | Prefix caching reuses KV cache blocks across requests that share a common prompt prefix. In multi-tenant deployments, that reuse is a timing side channel ([CVE-2025-46570](https://github.com/vllm-project/vllm/security/advisories/GHSA-4qjh-9fv9-r85r)). |
| 558 | |
| 559 | An attacker sharing the same backend can measure differences in Time to First Token (TTFT) to infer whether a guessed prompt prefix matches another user's cached prompt: when the prefix is already cached, prefill is faster. Research has shown this signal is nearly perfectly distinguishable (ROC AUC of 0.99) at prefix lengths of just 8 tokens. See [Leaking Secrets from Prefix Caches](https://arxiv.org/html/2411.18191v1) for details. |
| 560 | |
| 561 | ### Cache Salting |
| 562 | |
| 563 | vLLM accepts an optional `cache_salt` parameter on requests. The salt is mixed into the hash of the first KV cache block, so only requests carrying the same salt can share cached prefix blocks. See [Automatic Prefix Caching](../design/prefix_caching.md) for the implementation details. |
| 564 | |
| 565 | `cache_salt` is accepted by the OpenAI-compatible chat completions, completions, responses, and pooling (embeddings, classification, scoring) endpoints, and by the Anthropic `/v1/messages` endpoint. |
| 566 | |
| 567 | #### Usage with the OpenAI Python client |
| 568 | |
| 569 | ```python |
| 570 | response = client.chat.completions.create( |
| 571 | model=model, |
| 572 | messages=messages, |
| 573 | extra_body={ |
| 574 | "cache_salt": "per-user-or-per-tenant-secret", |
| 575 | }, |
| 576 | ) |
| 577 | ``` |
| 578 | |
| 579 | #### Usage with a raw request |
| 580 | |
| 581 | ```json |
| 582 | { |
| 583 | "model": "meta-llama/Llama-3-8b", |
| 584 | "messages": [ |
| 585 | {"role": "user", "content": "Hello"} |
| 586 | ], |
| 587 | "cache_salt": "per-user-or-per-tenant-secret" |
| 588 | } |
| 589 | ``` |
| 590 | |
| 591 | ### How to choose a salt value |
| 592 | |
| 593 | Treat the salt as a secret. An attacker who can guess or obtain the salt used by another tenant can still mount the timing attack against that tenant, so use random values that are long enough to be unpredictable (e.g. 43 base64 characters, 256 bits) rather than predictable identifiers such as a user name or account ID. |
| 594 | |
| 595 | Scope the salt to the isolation boundary you need: |
| 596 | |
| 597 | - **Per-user isolation**: A unique random salt per user prevents any cross-user cache inference. |
| 598 | - **Per-group sharing**: A shared random salt for users who are allowed to benefit from each other's cached prefixes, such as users within the same organization. |
| 599 | - **No salt**: Omitting `cache_salt` preserves the default behavior where all requests can share cached prefixes. This is appropriate for single-tenant deployments or when prefix privacy is not a concern. |
| 600 | |
| 601 | ### Recommendations |
| 602 | |
| 603 | - **Multi-tenant deployments**: Set `cache_salt` on every request, using a secret scoped to the tenant boundary you want to enforce. |
| 604 | - **Single-tenant deployments**: Cache salting is unnecessary and can be omitted to maximize cache hit rates. |
| 605 | - Salting reduces cache efficiency, since cached blocks are only reusable by requests with the same salt. Choose the granularity of your salt values to balance privacy against performance. |
| 606 | |
| 607 | ## Multimodal Media UUID Security |
| 608 | |
| 609 | ### Background |
| 610 | |
| 611 | Multimodal content parts (`image_url`, `input_audio`, `video`, `image_embeds`, `audio_embeds`, `vision_chunk`) accept an optional `uuid` field. When provided, vLLM uses this value as the cache identity for the media item instead of hashing the raw media bytes. This avoids re-hashing large media payloads on repeated requests and is the primary mechanism for client-side cache control of multimodal inputs. |
| 612 | |
| 613 | ### Client Responsibility |
| 614 | |
| 615 | **It is the client's responsibility to generate UUIDs that cannot be guessed by others.** Use cryptographically random values — for example, UUIDv4 via Python's `uuid.uuid4()` — rather than sequential counters, short strings, or predictable identifiers such as filenames or user IDs. |
| 616 | |
| 617 | ### Multi-Tenant Risk |
| 618 | |
| 619 | In multi-tenant deployments where multiple callers share the same vLLM server, two callers who present the same `uuid` for different media will share a single cache entry. The media from whichever request arrives first is served to both, which means: |
| 620 | |
| 621 | - **Integrity**: A later caller's media is silently discarded and replaced by the earlier caller's cached output. |
| 622 | - **Confidentiality**: A caller who deliberately reuses another caller's UUID receives output derived from that caller's media. |
| 623 | |
| 624 | This applies to the multimodal processor cache, the encoder output cache, and the prefix cache block hashes. Clients must ensure UUID uniqueness across all callers on the same server. |
| 625 | |
| 626 | ### Extra Protection with `cache_salt` |
| 627 | |
| 628 | For additional cross-tenant isolation, set `cache_salt` on each request (see [Prefix Cache Timing Side-Channel Mitigation](#prefix-cache-timing-side-channel-mitigation-cache-salting) above). The salt is mixed into the prefix cache block hashes, so requests with different salts cannot share cached prefix blocks even if UUIDs collide. |
| 629 | |
| 630 | `cache_salt` is opt-in and not passed by default. |
| 631 | |
| 632 | ### Recommendations |
| 633 | |
| 634 | - **Multi-tenant deployments**: Always generate cryptographically random UUIDs per media item. Additionally, set `cache_salt` to a per-tenant secret for defense in depth. |
| 635 | - **Single-tenant deployments**: Ensure UUIDs are unique per distinct media content. `cache_salt` is unnecessary when there is no cross-tenant threat. |
| 636 | - **Default behavior**: Omitting `uuid` entirely preserves the default content-hash-based identity, which is safe against this class of collision but requires hashing the media bytes on every request. |
| 637 | |
| 638 | ## Reporting Security Vulnerabilities |
| 639 | |
| 640 | If you believe you have found a security vulnerability in vLLM, please report it following the project's security policy. For more information on how to report security issues and the project's security policy, please see the [vLLM Security Policy](https://github.com/vllm-project/vllm/blob/main/SECURITY.md). |