A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 7, 17:45 UTC
[Bugfix] Fix CUDA compatibility path setting for both datacenter and consumer NVIDIA GPUs (#33992)
Signed-off-by: Seungmin Kim <8457324+ehfd@users.noreply.github.com> Signed-off-by: Andrew Mello <19512127+88plug@users.noreply.github.com> Co-authored-by: 88plug <19512127+88plug@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>
Seungmin Kim committed
Feb 26, 2026 at 11:15 UTC
160424a937d373101818876103227cc986887b55
6 files changed
+334
-5
docker/Dockerfile
+8
-4
@@ -132,8 +132,10 @@ ENV UV_LINK_MODE=copy
132
# Verify GCC version
133
RUN gcc --version
134
135
-# Ensure CUDA compatibility library is loaded
136
-RUN echo "/usr/local/cuda-$(echo "$CUDA_VERSION" | cut -d. -f1,2)/compat/" > /etc/ld.so.conf.d/cuda-compat.conf && ldconfig
135
+# Enable CUDA forward compatibility by setting '-e VLLM_ENABLE_CUDA_COMPATIBILITY=1'
136
+# Only needed for datacenter/professional GPUs with older drivers.
137
+# See: https://docs.nvidia.com/deploy/cuda-compatibility/
138
+ENV VLLM_ENABLE_CUDA_COMPATIBILITY=0
139
140
# ============================================================
141
# SLOW-CHANGING DEPENDENCIES BELOW
@@ -560,8 +562,10 @@ ENV UV_HTTP_TIMEOUT=500
562
ENV UV_INDEX_STRATEGY="unsafe-best-match"
563
ENV UV_LINK_MODE=copy
564
563
-# Ensure CUDA compatibility library is loaded
564
-RUN echo "/usr/local/cuda-$(echo "$CUDA_VERSION" | cut -d. -f1,2)/compat/" > /etc/ld.so.conf.d/cuda-compat.conf && ldconfig
565
+# Enable CUDA forward compatibility by setting '-e VLLM_ENABLE_CUDA_COMPATIBILITY=1'
566
+# Only needed for datacenter/professional GPUs with older drivers.
567
+# See: https://docs.nvidia.com/deploy/cuda-compatibility/
568
+ENV VLLM_ENABLE_CUDA_COMPATIBILITY=0
569
570
# ============================================================
571
# SLOW-CHANGING DEPENDENCIES BELOW
docs/getting_started/installation/gpu.cuda.inc.md
+17
@@ -297,6 +297,23 @@ You can add any other [engine-args](https://docs.vllm.ai/en/latest/configuration
297
RUN uv pip install --system git+https://github.com/huggingface/transformers.git
298
```
299
300
+#### Running on Systems with Older CUDA Drivers
301
+
302
+vLLM's Docker image comes with [CUDA compatibility libraries](https://docs.nvidia.com/deploy/cuda-compatibility/index.html) pre-installed. This allows you to run vLLM on systems with NVIDIA drivers that are older than the CUDA Toolkit version used in the image, but only supports select professional and datacenter NVIDIA GPUs.
303
+
304
+To enable this feature, set the `VLLM_ENABLE_CUDA_COMPATIBILITY` environment variable to `1` or `true` when running the container:
305
+
306
+```bash
307
+docker run --runtime nvidia --gpus all \
308
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
309
+ -p 8000:8000 \
310
+ --env "HF_TOKEN=<secret>" \
311
+ --env "VLLM_ENABLE_CUDA_COMPATIBILITY=1" \
312
+ vllm/vllm-openai <args...>
313
+```
314
+
315
+This will automatically configure `LD_LIBRARY_PATH` to point to the compatibility libraries before loading PyTorch and other dependencies.
316
+
317
# --8<-- [end:pre-built-images]
318
# --8<-- [start:build-image-from-source]
319
docs/usage/troubleshooting.md
+26
-1
@@ -318,7 +318,32 @@ This indicates vLLM failed to initialize the NCCL communicator, possibly due to
318
319
## CUDA error: the provided PTX was compiled with an unsupported toolchain
320
321
-If you see an error like `RuntimeError: CUDA error: the provided PTX was compiled with an unsupported toolchain.`, it means that the CUDA PTX in vLLM's wheels was compiled with a toolchain unsupported by your system. The released vLLM wheels have to be compiled with a specific version of CUDA toolkit, and the compiled code might fail to run on lower versions of CUDA drivers. Read [cuda compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/) for more details. The solution is to install `cuda-compat` package from your package manager. For example, on Ubuntu, you can run `sudo apt-get install cuda-compat-12-9`, and then add `export LD_LIBRARY_PATH=/usr/local/cuda-12.9/compat:$LD_LIBRARY_PATH` to your `.bashrc` file. When successfully installed, you should see that the output of `nvidia-smi` will show `CUDA Version: 12.9`. Note that we use CUDA 12.9 as an example here, you may want to install a higher version of cuda-compat package in case vLLM's default CUDA version goes higher.
321
+If you see an error like `RuntimeError: CUDA error: the provided PTX was compiled with an unsupported toolchain`, it means that the CUDA PTX in vLLM's wheels was compiled with a toolchain unsupported by your system. This section also applies if you get the error `RuntimeError: The NVIDIA driver on your system is too old`.
322
+
323
+The released vLLM wheels are compiled with a specific version of CUDA toolkit, and the compiled code might fail to run on lower versions of CUDA drivers. Read [CUDA compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/) for more details. **This is only supported on select professional and datacenter NVIDIA GPUs.**
324
+
325
+If you are using the vLLM official Docker image, you can solve this by adding `-e VLLM_ENABLE_CUDA_COMPATIBILITY=1` to your `docker run` command. This will enable the pre-installed CUDA forward compatibility libraries.
326
+
327
+If you are running vLLM outside of Docker, the solution is to install the `cuda-compat` package from your package manager with the [CUDA repository](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) enabled. For example, on Ubuntu, you can run `sudo apt-get install cuda-compat-12-9`, and then set `export VLLM_ENABLE_CUDA_COMPATIBILITY=1` and `export VLLM_CUDA_COMPATIBILITY_PATH="/usr/local/cuda-12.9/compat"`.
328
+
329
+On Conda, you can install the `conda-forge::cuda-compat` package (e.g., `conda install -c conda-forge cuda-compat=12.9`), then after activating the environment, set `export VLLM_ENABLE_CUDA_COMPATIBILITY=1` and `export VLLM_CUDA_COMPATIBILITY_PATH="${CONDA_PREFIX}/cuda-compat"`.
330
+
331
+You can verify the configuration works by running a minimal Python script that initializes CUDA via vLLM:
332
+
333
+```bash
334
+export VLLM_ENABLE_CUDA_COMPATIBILITY=1
335
+export VLLM_CUDA_COMPATIBILITY_PATH="/usr/local/cuda-12.9/compat"
336
+
337
+python3 - << 'EOF'
338
+import vllm
339
+import torch
340
+
341
+print(f"CUDA available: {torch.cuda.is_available()}")
342
+print(f"CUDA device count: {torch.cuda.device_count()}")
343
+EOF
344
+```
345
+
346
+Note that we use CUDA 12.9 as an example here, and you may want to install a higher version of cuda-compat package in case vLLM's default CUDA version goes higher.
347
348
## ptxas fatal: Value 'sm_110a' is not defined for option 'gpu-name'
349
tests/cuda/test_cuda_compatibility_path.py
new
+187
@@ -0,0 +1,187 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+"""Tests for CUDA forward compatibility path logic in env_override.py.
4
+
5
+Verifies the opt-in LD_LIBRARY_PATH manipulation for CUDA compat libs,
6
+including env var parsing, path detection, and deduplication.
7
+"""
8
+
9
+import os
10
+from unittest.mock import patch
11
+
12
+import pytest
13
+
14
+# Import the functions directly (they're module-level in env_override)
15
+# We must import them without triggering the module-level side effects,
16
+# so we import the functions by name after the module is already loaded.
17
+from vllm.env_override import (
18
+ _get_torch_cuda_version,
19
+ _maybe_set_cuda_compatibility_path,
20
+)
21
+
22
+
23
+class TestCudaCompatibilityEnvParsing:
24
+ """Test VLLM_ENABLE_CUDA_COMPATIBILITY env var parsing."""
25
+
26
+ def test_disabled_by_default(self, monkeypatch):
27
+ """Compat path is NOT set when env var is absent."""
28
+ monkeypatch.delenv("VLLM_ENABLE_CUDA_COMPATIBILITY", raising=False)
29
+ monkeypatch.delenv("LD_LIBRARY_PATH", raising=False)
30
+ _maybe_set_cuda_compatibility_path()
31
+ assert (
32
+ "LD_LIBRARY_PATH" not in os.environ
33
+ or os.environ.get("LD_LIBRARY_PATH", "") == ""
34
+ )
35
+
36
+ @pytest.mark.parametrize("value", ["0", "false", "False", "no", ""])
37
+ def test_disabled_values(self, monkeypatch, value):
38
+ """Various falsy values should not activate compat path."""
39
+ monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", value)
40
+ monkeypatch.delenv("LD_LIBRARY_PATH", raising=False)
41
+ _maybe_set_cuda_compatibility_path()
42
+ # LD_LIBRARY_PATH should not be set (or remain empty)
43
+ ld_path = os.environ.get("LD_LIBRARY_PATH", "")
44
+ assert "compat" not in ld_path
45
+
46
+ @pytest.mark.parametrize("value", ["1", "true", "True", " 1 ", " TRUE "])
47
+ def test_enabled_values_with_valid_path(self, monkeypatch, tmp_path, value):
48
+ """Truthy values activate compat path when a valid path exists."""
49
+ compat_dir = tmp_path / "compat"
50
+ compat_dir.mkdir()
51
+ monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", value)
52
+ monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(compat_dir))
53
+ monkeypatch.delenv("LD_LIBRARY_PATH", raising=False)
54
+ _maybe_set_cuda_compatibility_path()
55
+ ld_path = os.environ.get("LD_LIBRARY_PATH", "")
56
+ assert str(compat_dir) in ld_path
57
+
58
+
59
+class TestCudaCompatibilityPathDetection:
60
+ """Test path detection: custom override, conda, default."""
61
+
62
+ def test_custom_path_override(self, monkeypatch, tmp_path):
63
+ """VLLM_CUDA_COMPATIBILITY_PATH takes highest priority."""
64
+ custom_dir = tmp_path / "my-compat"
65
+ custom_dir.mkdir()
66
+ monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1")
67
+ monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(custom_dir))
68
+ monkeypatch.delenv("LD_LIBRARY_PATH", raising=False)
69
+ _maybe_set_cuda_compatibility_path()
70
+ ld_path = os.environ.get("LD_LIBRARY_PATH", "")
71
+ assert ld_path.startswith(str(custom_dir))
72
+
73
+ def test_conda_prefix_fallback(self, monkeypatch, tmp_path):
74
+ """Falls back to $CONDA_PREFIX/cuda-compat if custom not set."""
75
+ conda_dir = tmp_path / "conda-env"
76
+ compat_dir = conda_dir / "cuda-compat"
77
+ compat_dir.mkdir(parents=True)
78
+ monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1")
79
+ monkeypatch.delenv("VLLM_CUDA_COMPATIBILITY_PATH", raising=False)
80
+ monkeypatch.setenv("CONDA_PREFIX", str(conda_dir))
81
+ monkeypatch.delenv("LD_LIBRARY_PATH", raising=False)
82
+ _maybe_set_cuda_compatibility_path()
83
+ ld_path = os.environ.get("LD_LIBRARY_PATH", "")
84
+ assert str(compat_dir) in ld_path
85
+
86
+ def test_no_valid_path_does_nothing(self, monkeypatch):
87
+ """When enabled but no valid path exists, LD_LIBRARY_PATH unchanged."""
88
+ monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1")
89
+ monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", "/nonexistent/path")
90
+ monkeypatch.delenv("CONDA_PREFIX", raising=False)
91
+ monkeypatch.delenv("LD_LIBRARY_PATH", raising=False)
92
+ with patch("vllm.env_override._get_torch_cuda_version", return_value=None):
93
+ _maybe_set_cuda_compatibility_path()
94
+ assert os.environ.get("LD_LIBRARY_PATH", "") == ""
95
+
96
+ def test_default_cuda_path_fallback(self, monkeypatch, tmp_path):
97
+ """Falls back to /usr/local/cuda-{ver}/compat via torch version."""
98
+ fake_cuda = tmp_path / "cuda-12.8" / "compat"
99
+ fake_cuda.mkdir(parents=True)
100
+ monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1")
101
+ monkeypatch.delenv("VLLM_CUDA_COMPATIBILITY_PATH", raising=False)
102
+ monkeypatch.delenv("CONDA_PREFIX", raising=False)
103
+ monkeypatch.delenv("LD_LIBRARY_PATH", raising=False)
104
+ with (
105
+ patch("vllm.env_override._get_torch_cuda_version", return_value="12.8"),
106
+ patch(
107
+ "vllm.env_override.os.path.isdir",
108
+ side_effect=lambda p: p == "/usr/local/cuda-12.8/compat"
109
+ or os.path.isdir(p),
110
+ ),
111
+ ):
112
+ _maybe_set_cuda_compatibility_path()
113
+ ld_path = os.environ.get("LD_LIBRARY_PATH", "")
114
+ assert "/usr/local/cuda-12.8/compat" in ld_path
115
+
116
+
117
+class TestCudaCompatibilityLdPathManipulation:
118
+ """Test LD_LIBRARY_PATH prepend and deduplication logic."""
119
+
120
+ def test_prepends_to_empty_ld_path(self, monkeypatch, tmp_path):
121
+ """Compat path is set when LD_LIBRARY_PATH is empty."""
122
+ compat_dir = tmp_path / "compat"
123
+ compat_dir.mkdir()
124
+ monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1")
125
+ monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(compat_dir))
126
+ monkeypatch.delenv("LD_LIBRARY_PATH", raising=False)
127
+ _maybe_set_cuda_compatibility_path()
128
+ assert os.environ["LD_LIBRARY_PATH"] == str(compat_dir)
129
+
130
+ def test_prepends_to_existing_ld_path(self, monkeypatch, tmp_path):
131
+ """Compat path is prepended before existing entries."""
132
+ compat_dir = tmp_path / "compat"
133
+ compat_dir.mkdir()
134
+ monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1")
135
+ monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(compat_dir))
136
+ monkeypatch.setenv("LD_LIBRARY_PATH", "/usr/lib:/other/lib")
137
+ _maybe_set_cuda_compatibility_path()
138
+ ld_path = os.environ["LD_LIBRARY_PATH"]
139
+ parts = ld_path.split(os.pathsep)
140
+ assert parts[0] == str(compat_dir)
141
+ assert "/usr/lib" in parts
142
+ assert "/other/lib" in parts
143
+
144
+ def test_deduplicates_existing_compat_path(self, monkeypatch, tmp_path):
145
+ """If compat path already in LD_LIBRARY_PATH, move to front."""
146
+ compat_dir = tmp_path / "compat"
147
+ compat_dir.mkdir()
148
+ monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1")
149
+ monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(compat_dir))
150
+ monkeypatch.setenv(
151
+ "LD_LIBRARY_PATH",
152
+ f"/usr/lib:{compat_dir}:/other/lib",
153
+ )
154
+ _maybe_set_cuda_compatibility_path()
155
+ ld_path = os.environ["LD_LIBRARY_PATH"]
156
+ parts = ld_path.split(os.pathsep)
157
+ assert parts[0] == str(compat_dir)
158
+ assert parts.count(str(compat_dir)) == 1
159
+
160
+ def test_already_at_front_is_noop(self, monkeypatch, tmp_path):
161
+ """If compat path is already first, don't modify LD_LIBRARY_PATH."""
162
+ compat_dir = tmp_path / "compat"
163
+ compat_dir.mkdir()
164
+ original = f"{compat_dir}:/usr/lib"
165
+ monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1")
166
+ monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(compat_dir))
167
+ monkeypatch.setenv("LD_LIBRARY_PATH", original)
168
+ _maybe_set_cuda_compatibility_path()
169
+ assert os.environ["LD_LIBRARY_PATH"] == original
170
+
171
+
172
+class TestGetTorchCudaVersion:
173
+ """Test _get_torch_cuda_version() helper."""
174
+
175
+ def test_returns_string_when_torch_available(self):
176
+ """Should return a CUDA version string like '12.8'."""
177
+ version = _get_torch_cuda_version()
178
+ # torch is installed in vllm's environment
179
+ assert version is None or isinstance(version, str)
180
+
181
+ def test_returns_none_when_torch_missing(self):
182
+ """Should return None when torch is not importable."""
183
+ with patch(
184
+ "vllm.env_override.importlib.util.find_spec",
185
+ return_value=None,
186
+ ):
187
+ assert _get_torch_cuda_version() is None
vllm/env_override.py
+82
@@ -1,7 +1,89 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+# ruff: noqa: E402
4
+import importlib.util
5
import os
6
7
+
8
+def _get_torch_cuda_version():
9
+ """Peripheral function to _maybe_set_cuda_compatibility_path().
10
+ PyTorch version must not be determined by importing directly
11
+ because it will trigger the CUDA initialization, losing the
12
+ chance to set the LD_LIBRARY_PATH beforehand.
13
+ """
14
+ try:
15
+ spec = importlib.util.find_spec("torch")
16
+ if not spec:
17
+ return None
18
+ if spec.origin:
19
+ torch_root = os.path.dirname(spec.origin)
20
+ elif spec.submodule_search_locations:
21
+ torch_root = spec.submodule_search_locations[0]
22
+ else:
23
+ return None
24
+ version_path = os.path.join(torch_root, "version.py")
25
+ if not os.path.exists(version_path):
26
+ return None
27
+ # Load the version module without importing torch
28
+ ver_spec = importlib.util.spec_from_file_location("torch.version", version_path)
29
+ if not ver_spec or not ver_spec.loader:
30
+ return None
31
+ module = importlib.util.module_from_spec(ver_spec)
32
+ # Avoid registering in sys.modules to not confuse future imports
33
+ ver_spec.loader.exec_module(module)
34
+ return getattr(module, "cuda", None)
35
+ except Exception:
36
+ return None
37
+
38
+
39
+def _maybe_set_cuda_compatibility_path():
40
+ """Set LD_LIBRARY_PATH for CUDA forward compatibility if enabled.
41
+
42
+ Must run before 'import torch' since torch loads CUDA shared libraries
43
+ at import time and the dynamic linker only consults LD_LIBRARY_PATH when
44
+ a library is first loaded.
45
+
46
+ CUDA forward compatibility is only supported on select professional and
47
+ datacenter NVIDIA GPUs. Consumer GPUs (GeForce, RTX) do not support it
48
+ and will get Error 803 if compat libs are loaded.
49
+ """
50
+ enable = os.environ.get("VLLM_ENABLE_CUDA_COMPATIBILITY", "0").strip().lower() in (
51
+ "1",
52
+ "true",
53
+ )
54
+ if not enable:
55
+ return
56
+
57
+ cuda_compat_path = os.environ.get("VLLM_CUDA_COMPATIBILITY_PATH", "")
58
+ if not cuda_compat_path or not os.path.isdir(cuda_compat_path):
59
+ conda_prefix = os.environ.get("CONDA_PREFIX", "")
60
+ conda_compat = os.path.join(conda_prefix, "cuda-compat")
61
+ if conda_prefix and os.path.isdir(conda_compat):
62
+ cuda_compat_path = conda_compat
63
+ if not cuda_compat_path or not os.path.isdir(cuda_compat_path):
64
+ torch_cuda_version = _get_torch_cuda_version()
65
+ if torch_cuda_version:
66
+ default_path = f"/usr/local/cuda-{torch_cuda_version}/compat"
67
+ if os.path.isdir(default_path):
68
+ cuda_compat_path = default_path
69
+ if not cuda_compat_path or not os.path.isdir(cuda_compat_path):
70
+ return
71
+
72
+ norm_path = os.path.normpath(cuda_compat_path)
73
+ existing = os.environ.get("LD_LIBRARY_PATH", "")
74
+ ld_paths = existing.split(os.pathsep) if existing else []
75
+
76
+ if ld_paths and ld_paths[0] and os.path.normpath(ld_paths[0]) == norm_path:
77
+ return # Already at the front
78
+
79
+ new_paths = [norm_path] + [
80
+ p for p in ld_paths if not p or os.path.normpath(p) != norm_path
81
+ ]
82
+ os.environ["LD_LIBRARY_PATH"] = os.pathsep.join(new_paths)
83
+
84
+
85
+_maybe_set_cuda_compatibility_path()
86
+
87
import torch
88
89
from vllm.logger import init_logger
vllm/envs.py
+14
@@ -239,6 +239,8 @@ if TYPE_CHECKING:
239
VLLM_WEIGHT_OFFLOADING_DISABLE_UVA: bool = False
240
VLLM_DISABLE_LOG_LOGO: bool = False
241
VLLM_LORA_DISABLE_PDL: bool = False
242
+ VLLM_ENABLE_CUDA_COMPATIBILITY: bool = False
243
+ VLLM_CUDA_COMPATIBILITY_PATH: str | None = None
244
245
246
def get_default_cache_root():
@@ -1591,6 +1593,16 @@ environment_variables: dict[str, Callable[[], Any]] = {
1593
# Disable PDL for LoRA, as enabling PDL with LoRA on SM100 causes
1594
# Triton compilation to fail.
1595
"VLLM_LORA_DISABLE_PDL": lambda: bool(int(os.getenv("VLLM_LORA_DISABLE_PDL", "0"))),
1596
+ # Enable CUDA compatibility mode for datacenter GPUs with older
1597
+ # driver versions than the CUDA toolkit major version of vLLM.
1598
+ "VLLM_ENABLE_CUDA_COMPATIBILITY": lambda: (
1599
+ os.environ.get("VLLM_ENABLE_CUDA_COMPATIBILITY", "0").strip().lower()
1600
+ in ("1", "true")
1601
+ ),
1602
+ # Path to the CUDA compatibility libraries when CUDA compatibility is enabled.
1603
+ "VLLM_CUDA_COMPATIBILITY_PATH": lambda: os.environ.get(
1604
+ "VLLM_CUDA_COMPATIBILITY_PATH", None
1605
+ ),
1606
}
1607
1608
@@ -1731,6 +1743,8 @@ def compile_factors() -> dict[str, object]:
1743
"VLLM_CPU_MOE_PREPACK",
1744
"VLLM_CPU_SGL_KERNEL",
1745
"VLLM_TEST_FORCE_LOAD_FORMAT",
1746
+ "VLLM_ENABLE_CUDA_COMPATIBILITY",
1747
+ "VLLM_CUDA_COMPATIBILITY_PATH",
1748
"LOCAL_RANK",
1749
"CUDA_VISIBLE_DEVICES",
1750
"NO_COLOR",