@tduncan / Nvidia-Optimization-Stack-VLLM / commits / 89d97d9a16

docs(security): document Ray cluster trust model and env var propagation (#50397)

Signed-off-by: jperezde <jperezde@redhat.com>

Juan Pérez de Algaba committed Jul 30, 2026 at 13:52 UTC 89d97d9a165a7fb2c84ffb77c8da816093bba0ed

1 file changed +98

docs/usage/security.md

+98

@@ -425,6 +425,104 @@ Hashing is the area where vLLM has explicit FIPS-aware code, but a FIPS-complian

425

426 In short: the configuration knobs above let vLLM avoid non-approved algorithms, and the automatic fallbacks let it run without crashing on FIPS-enabled hosts. End-to-end FIPS compliance, however, is a property of the full deployment — host OS, crypto provider, transitive dependencies, and network architecture — not of vLLM alone.

427

428 +## Ray Cluster Trust Model and Environment Variable Propagation

429 +

430 +### Trust Assumption

431 +

432 +vLLM treats the entire Ray cluster as a single trust domain. Any principal

433 +with the ability to execute code within the Ray cluster (e.g. submit actors

434 +or tasks) is considered to have the same level of trust as the driver/API

435 +server process. This means that vLLM does **not** attempt to isolate

436 +driver-side credentials from worker-side processes within the same Ray

437 +cluster.

438 +

439 +This assumption is consistent with

440 +[Ray's own security model](https://docs.ray.io/en/latest/ray-core/security.html),

441 +which states that any user who can connect to a Ray cluster can run arbitrary

442 +code on any node in that cluster. In other words, Ray cluster access already

443 +implies full code execution on worker nodes, so restricting environment

444 +variable propagation alone would not constitute a meaningful security

445 +boundary.

446 +

447 +### Driver-to-Worker Environment Variable Propagation

448 +

449 +When using `RayExecutorV2` in multi-node deployments, vLLM propagates

450 +environment variables from the driver process to remote Ray workers so that

451 +workers have the configuration they need to function correctly (e.g. vLLM

452 +settings, NCCL tuning, Hugging Face tokens for gated model downloads).

453 +

454 +The propagation uses a **copy-all-except-denylist** policy via

455 +`get_driver_env_vars()` in `vllm/v1/executor/ray_env_utils.py`: every

456 +environment variable present in the driver's `os.environ` is sent to

457 +workers, except for a small set of worker-specific variables and any

458 +names the operator has explicitly excluded.

459 +

460 +On the worker side, propagated variables are applied with `setdefault`

461 +semantics — they fill in missing variables but never overwrite values

462 +already present in the worker's environment.

463 +

464 +### When This Matters

465 +

466 +In deployments where operators intentionally scope credentials (such as

467 +`HF_TOKEN`, cloud storage keys, registry tokens, or internal service

468 +tokens) to the driver/API server alone — for example, when GPU workers

469 +run in a different node, pod, or trust domain — the default propagation

470 +behavior will copy those credentials into worker environments. A process

471 +running on a worker node under the same OS user may then be able to read

472 +those credentials (e.g. via `/proc/<pid>/environ` on Linux).

473 +

474 +If your deployment treats Ray workers as less trusted than the driver, be

475 +aware that environment-variable isolation is **not** enforced by default.

476 +

477 +### Hardening Recommendations

478 +

479 +Operators who want to limit which environment variables are propagated to

480 +Ray workers can use the following mechanisms:

481 +

482 +#### 1. Denylist via Configuration File

483 +

484 +Create a JSON file at `$VLLM_CONFIG_ROOT/ray_non_carry_over_env_vars.json`

485 +(default: `~/.config/vllm/ray_non_carry_over_env_vars.json`) containing an

486 +array of environment variable names to exclude from propagation:

487 +

488 +```json

489 +[

490 + "HF_TOKEN",

491 + "AWS_SECRET_ACCESS_KEY",

492 + "AWS_SESSION_TOKEN",

493 + "GOOGLE_APPLICATION_CREDENTIALS",

494 + "AZURE_CLIENT_SECRET",

495 + "REGISTRY_TOKEN",

496 + "MY_INTERNAL_SERVICE_KEY"

497 +]

498 +```

499 +

500 +Any variable listed here will **not** be copied from the driver to workers.

501 +

502 +#### 2. Minimize Driver Environment

503 +

504 +Rather than setting credentials in the driver's shell environment, inject

505 +them through a secrets manager, a mounted file, or a short-lived

506 +subprocess so that they are not present in `os.environ` when vLLM starts.

507 +

508 +#### 3. Network and Process Isolation

509 +

510 +- Restrict `procfs` visibility on worker nodes (e.g. mount `/proc` with

511 + `hidepid=2` or use a container runtime that isolates `/proc` between

512 + pods) so that same-UID processes cannot read each other's

513 + `/proc/<pid>/environ`.

514 +- Run the driver and workers under different OS users or in separate

515 + containers with non-overlapping UIDs.

516 +

517 +#### 4. Limit Ray Cluster Access

518 +

519 +Because Ray cluster access is equivalent to arbitrary code execution,

520 +ensure that only trusted principals can submit work to the cluster:

521 +

522 +- Use Ray's TLS authentication to restrict cluster membership.

523 +- Place the Ray cluster on an isolated network segment.

524 +- Do not expose the Ray client port or dashboard to untrusted networks.

525 +

526 ## Reporting Security Vulnerabilities

527

528 If you believe you have found a security vulnerability in vLLM, please report it following the project's security policy. For more information on how to report security issues and the project's security policy, please see the [vLLM Security Policy](https://github.com/vllm-project/vllm/blob/main/SECURITY.md).