A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 9, 14:45 UTC
[Misc] HF Hub LoRA Resolver (#20320)
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
Alex Brooks committed
Jan 26, 2026 at 06:56 UTC
9ac818a5517199f6f13aac32b4980d52b418bd6c
8 files changed
+280
-9
.buildkite/test-pipeline.yaml
+1
-1
@@ -1316,7 +1316,7 @@ steps:
1316
- pytest -v -s distributed/test_distributed_oot.py
1317
- pytest -v -s entrypoints/openai/test_oot_registration.py # it needs a clean process
1318
- pytest -v -s models/test_oot_registration.py # it needs a clean process
1319
- - pytest -v -s plugins/lora_resolvers # unit tests for in-tree lora resolver plugins
1319
+ - pytest -v -s plugins/lora_resolvers # unit tests for lora resolver plugins
1320
1321
- label: Pipeline + Context Parallelism Test # 45min
1322
timeout_in_minutes: 60
docs/design/lora_resolver_plugins.md
+1
-1
@@ -10,7 +10,7 @@ receives a request for a LoRA adapter that hasn't been loaded yet, the resolver
10
to locate and load the adapter from their configured storage locations. This enables:
11
12
- **Dynamic LoRA Loading**: Load adapters on-demand without server restarts
13
-- **Multiple Storage Backends**: Support for filesystem, S3, and custom backends. The built-in `lora_filesystem_resolver` requires a local storage path, but custom resolvers can be implemented to fetch from any source.
13
+- **Multiple Storage Backends**: Support for filesystem, S3, and custom backends. The built-in `lora_filesystem_resolver` requires a local storage path, while the built-in `hf_hub_resolver` will pull LoRA adapters from Huggingface Hub and proceed in an identical manner. In general, custom resolvers can be implemented to fetch from any source.
14
- **Automatic Discovery**: Seamless integration with existing LoRA workflows
15
- **Scalable Deployment**: Centralized adapter management across multiple vLLM instances
16
docs/features/lora.md
+6
-4
@@ -159,10 +159,12 @@ Alternatively, you can use the LoRAResolver plugin to dynamically load LoRA adap
159
160
You can set up multiple LoRAResolver plugins if you want to load LoRA adapters from different sources. For example, you might have one resolver for local files and another for S3 storage. vLLM will load the first LoRA adapter that it finds.
161
162
-You can either install existing plugins or implement your own. By default, vLLM comes with a [resolver plugin to load LoRA adapters from a local directory.](https://github.com/vllm-project/vllm/tree/main/vllm/plugins/lora_resolvers)
163
-To enable this resolver, set `VLLM_ALLOW_RUNTIME_LORA_UPDATING` to True, set `VLLM_PLUGINS` to include `lora_filesystem_resolver`, and then set `VLLM_LORA_RESOLVER_CACHE_DIR` to a local directory. When vLLM receives a request using a LoRA adapter `foobar`,
164
-it will first look in the local directory for a directory `foobar`, and attempt to load the contents of that directory as a LoRA adapter. If successful, the request will complete as normal and
165
-that adapter will then be available for normal use on the server.
162
+You can either install existing plugins or implement your own. By default, vLLM comes with a [resolver plugin to load LoRA adapters from a local directory, as well as a resolver plugin to load LoRA adapters from repositories on Hugging Face Hub](https://github.com/vllm-project/vllm/tree/main/vllm/plugins/lora_resolvers)
163
+To enable either of these resolvers, you must `set VLLM_ALLOW_RUNTIME_LORA_UPDATING` to True.
164
+
165
+- To leverage a local directory, set `VLLM_PLUGINS` to include `lora_filesystem_resolver` and set `VLLM_LORA_RESOLVER_CACHE_DIR` to a local directory. When vLLM receives a request using a LoRA adapter `foobar`,
166
+it will first look in the local directory for a directory `foobar`, and attempt to load the contents of that directory as a LoRA adapter. If successful, the request will complete as normal and that adapter will then be available for normal use on the server.
167
+- To leverage repositories on Hugging Face Hub, set `VLLM_PLUGINS` to include `lora_hf_hub_resolver` and set `VLLM_LORA_RESOLVER_HF_REPO_LIST` to a comma separated list of repository IDs on Hugging Face Hub. When vLLM receives a request for the LoRA adapter `my/repo/subpath`, it will download the adapter at the `subpath` of `my/repo` if it exists and contains an `adapter_config.json`, then build a request to the cached dir for the adapter, similar to the `lora_filesystem_resolver`. Please note that enabling remote downloads is insecure and not intended for use in production environments.
168
169
Alternatively, follow these example steps to implement your own plugin:
170
pyproject.toml
+1
@@ -44,6 +44,7 @@ vllm = "vllm.entrypoints.cli.main:main"
44
45
[project.entry-points."vllm.general_plugins"]
46
lora_filesystem_resolver = "vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver"
47
+lora_hf_hub_resolver = "vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver"
48
49
[tool.setuptools_scm]
50
# no extra settings needed, presence enables setuptools-scm
tests/plugins/lora_resolvers/test_hf_hub_resolver.py
new
+107
@@ -0,0 +1,107 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+import os
4
+
5
+import pytest
6
+from huggingface_hub.constants import HF_HUB_CACHE
7
+
8
+from vllm.plugins.lora_resolvers.hf_hub_resolver import HfHubResolver
9
+
10
+LORA_LIB_MODEL_NAME = "ibm-granite/granite-3.3-8b-instruct"
11
+# Repo with multiple LoRAs contained in it
12
+LORA_LIB = "ibm-granite/granite-3.3-8b-rag-agent-lib"
13
+LORA_NAME = "ibm-granite/granite-3.3-8b-rag-agent-lib/answerability_prediction_lora" # noqa: E501
14
+NON_LORA_SUBPATH = "ibm-granite/granite-3.3-8b-rag-agent-lib/README.md"
15
+LIB_DOWNLOAD_DIR = os.path.join(
16
+ HF_HUB_CACHE, "models--ibm-granite--granite-3.3-8b-rag-agent-lib"
17
+)
18
+INVALID_REPO_NAME = "thisrepodoesnotexist"
19
+
20
+# Repo with only one LoRA in the root dir
21
+LORA_REPO_MODEL_NAME = "meta-llama/Llama-2-7b-hf"
22
+LORA_REPO = "yard1/llama-2-7b-sql-lora-test"
23
+REPO_DOWNLOAD_DIR = os.path.join(
24
+ HF_HUB_CACHE, "models--yard1--llama-2-7b-sql-lora-test"
25
+)
26
+
27
+
28
+@pytest.mark.asyncio
29
+async def test_hf_resolver_with_direct_path():
30
+ hf_resolver = HfHubResolver([LORA_REPO])
31
+ assert hf_resolver is not None
32
+
33
+ lora_request = await hf_resolver.resolve_lora(LORA_REPO_MODEL_NAME, LORA_REPO)
34
+ assert lora_request.lora_name == LORA_REPO
35
+ assert REPO_DOWNLOAD_DIR in lora_request.lora_path
36
+ assert "adapter_config.json" in os.listdir(lora_request.lora_path)
37
+
38
+
39
+@pytest.mark.asyncio
40
+async def test_hf_resolver_with_nested_paths():
41
+ hf_resolver = HfHubResolver([LORA_LIB])
42
+ assert hf_resolver is not None
43
+
44
+ lora_request = await hf_resolver.resolve_lora(LORA_LIB_MODEL_NAME, LORA_NAME)
45
+ assert lora_request is not None
46
+ assert lora_request.lora_name == LORA_NAME
47
+ assert LIB_DOWNLOAD_DIR in lora_request.lora_path
48
+ assert "adapter_config.json" in os.listdir(lora_request.lora_path)
49
+
50
+
51
+@pytest.mark.asyncio
52
+async def test_hf_resolver_with_multiple_repos():
53
+ hf_resolver = HfHubResolver([LORA_LIB, LORA_REPO])
54
+ assert hf_resolver is not None
55
+
56
+ lora_request = await hf_resolver.resolve_lora(LORA_LIB_MODEL_NAME, LORA_NAME)
57
+ assert lora_request is not None
58
+ assert lora_request.lora_name == LORA_NAME
59
+ assert LIB_DOWNLOAD_DIR in lora_request.lora_path
60
+ assert "adapter_config.json" in os.listdir(lora_request.lora_path)
61
+
62
+
63
+@pytest.mark.asyncio
64
+async def test_missing_adapter():
65
+ hf_resolver = HfHubResolver([LORA_LIB])
66
+ assert hf_resolver is not None
67
+
68
+ missing_lora_request = await hf_resolver.resolve_lora(LORA_LIB_MODEL_NAME, "foobar")
69
+ assert missing_lora_request is None
70
+
71
+
72
+@pytest.mark.asyncio
73
+async def test_nonlora_adapter():
74
+ hf_resolver = HfHubResolver([LORA_LIB])
75
+ assert hf_resolver is not None
76
+
77
+ readme_request = await hf_resolver.resolve_lora(
78
+ LORA_LIB_MODEL_NAME, NON_LORA_SUBPATH
79
+ )
80
+ assert readme_request is None
81
+
82
+
83
+@pytest.mark.asyncio
84
+async def test_invalid_repo():
85
+ hf_resolver = HfHubResolver([LORA_LIB])
86
+ assert hf_resolver is not None
87
+
88
+ invalid_repo_req = await hf_resolver.resolve_lora(
89
+ INVALID_REPO_NAME,
90
+ f"{INVALID_REPO_NAME}/foo",
91
+ )
92
+ assert invalid_repo_req is None
93
+
94
+
95
+@pytest.mark.asyncio
96
+async def test_trailing_slash():
97
+ hf_resolver = HfHubResolver([LORA_LIB])
98
+ assert hf_resolver is not None
99
+
100
+ lora_request = await hf_resolver.resolve_lora(
101
+ LORA_LIB_MODEL_NAME,
102
+ f"{LORA_NAME}/",
103
+ )
104
+ assert lora_request is not None
105
+ assert lora_request.lora_name == f"{LORA_NAME}/"
106
+ assert LIB_DOWNLOAD_DIR in lora_request.lora_path
107
+ assert "adapter_config.json" in os.listdir(lora_request.lora_path)
vllm/envs.py
+8
@@ -87,6 +87,7 @@ if TYPE_CHECKING:
87
VLLM_HTTP_TIMEOUT_KEEP_ALIVE: int = 5 # seconds
88
VLLM_PLUGINS: list[str] | None = None
89
VLLM_LORA_RESOLVER_CACHE_DIR: str | None = None
90
+ VLLM_LORA_RESOLVER_HF_REPO_LIST: str | None = None
91
# Deprecated env variables for profiling, kept for backward compatibility
92
# See also vllm/config/profiler.py and `--profiler-config` argument
93
VLLM_TORCH_CUDA_PROFILE: str | None = None
@@ -873,6 +874,13 @@ environment_variables: dict[str, Callable[[], Any]] = {
874
"VLLM_LORA_RESOLVER_CACHE_DIR": lambda: os.getenv(
875
"VLLM_LORA_RESOLVER_CACHE_DIR", None
876
),
877
+ # A remote HF repo(s) containing one or more LoRA adapters, which
878
+ # may be downloaded and leveraged as needed. Only works if plugins
879
+ # are enabled and VLLM_ALLOW_RUNTIME_LORA_UPDATING is enabled.
880
+ # Values should be comma separated.
881
+ "VLLM_LORA_RESOLVER_HF_REPO_LIST": lambda: os.getenv(
882
+ "VLLM_LORA_RESOLVER_HF_REPO_LIST", None
883
+ ),
884
# Enables torch CUDA profiling if set to 1.
885
# Deprecated, see profiler_config.
886
"VLLM_TORCH_CUDA_PROFILE": lambda: os.getenv("VLLM_TORCH_CUDA_PROFILE"),
vllm/plugins/lora_resolvers/filesystem_resolver.py
+13
-3
@@ -16,10 +16,20 @@ class FilesystemResolver(LoRAResolver):
16
self, base_model_name: str, lora_name: str
17
) -> LoRARequest | None:
18
lora_path = os.path.join(self.lora_cache_dir, lora_name)
19
+ maybe_lora_request = await self._get_lora_req_from_path(
20
+ lora_name, lora_path, base_model_name
21
+ )
22
+ return maybe_lora_request
23
+
24
+ async def _get_lora_req_from_path(
25
+ self, lora_name: str, lora_path: str, base_model_name: str
26
+ ) -> LoRARequest | None:
27
+ """Builds a LoraRequest pointing to the lora path if it's a valid
28
+ LoRA adapter and has a matching base_model_name.
29
+ """
30
if os.path.exists(lora_path):
20
- adapter_config_path = os.path.join(
21
- self.lora_cache_dir, lora_name, "adapter_config.json"
22
- )
31
+ adapter_config_path = os.path.join(lora_path, "adapter_config.json")
32
+
33
if os.path.exists(adapter_config_path):
34
with open(adapter_config_path) as file:
35
adapter_config = json.load(file)
vllm/plugins/lora_resolvers/hf_hub_resolver.py
new
+143
@@ -0,0 +1,143 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+import asyncio
4
+import os
5
+
6
+from huggingface_hub import HfApi, snapshot_download
7
+
8
+import vllm.envs as envs
9
+from vllm.logger import init_logger
10
+from vllm.lora.request import LoRARequest
11
+from vllm.lora.resolver import LoRAResolverRegistry
12
+from vllm.plugins.lora_resolvers.filesystem_resolver import FilesystemResolver
13
+
14
+logger = init_logger(__name__)
15
+
16
+
17
+class HfHubResolver(FilesystemResolver):
18
+ def __init__(self, repo_list: list[str]):
19
+ logger.warning(
20
+ "LoRA is allowing resolution from the following repositories on"
21
+ " HF Hub: %s please note that allowing remote downloads"
22
+ " is not secure, and that this plugin is not intended for use in"
23
+ " production environments.",
24
+ repo_list,
25
+ )
26
+
27
+ self.repo_list: list[str] = repo_list
28
+ self.adapter_dirs: dict[str, set[str]] = {}
29
+
30
+ async def resolve_lora(
31
+ self, base_model_name: str, lora_name: str
32
+ ) -> LoRARequest | None:
33
+ """Resolves potential LoRA requests in a remote repo on HF Hub.
34
+ This is effectively the same behavior as the filesystem resolver, but
35
+ with a snapshot_download on dirs containing an adapter config prior
36
+ to inspecting the cached dir to build a potential LoRA
37
+ request.
38
+ """
39
+ # If a LoRA name begins with the repository name, it's disambiguated
40
+ maybe_repo = await self._resolve_repo(lora_name)
41
+
42
+ # If we haven't inspected this repo before, save available adapter dirs
43
+ if maybe_repo is not None and maybe_repo not in self.adapter_dirs:
44
+ self.adapter_dirs[maybe_repo] = await self._get_adapter_dirs(maybe_repo)
45
+
46
+ maybe_subpath = await self._resolve_repo_subpath(lora_name, maybe_repo)
47
+
48
+ if maybe_repo is None or maybe_subpath is None:
49
+ return None
50
+
51
+ repo_path = await asyncio.to_thread(
52
+ snapshot_download,
53
+ repo_id=maybe_repo,
54
+ allow_patterns=f"{maybe_subpath}/*" if maybe_subpath != "." else "*",
55
+ )
56
+
57
+ lora_path = os.path.join(repo_path, maybe_subpath)
58
+ maybe_lora_request = await self._get_lora_req_from_path(
59
+ lora_name, lora_path, base_model_name
60
+ )
61
+ return maybe_lora_request
62
+
63
+ async def _resolve_repo(self, lora_name: str) -> str | None:
64
+ """Given a fully qualified path to a LoRA with respect to its HF Hub
65
+ repo, match the right repo to potentially download from if one exists.
66
+
67
+ Args:
68
+ lora_name: Path to LoRA in HF Hub, e.g., <org>/<repo>/<subpath>,
69
+ match on <org>/<repo> (if it contains an adapter directly) or
70
+ <org>/<repo>/ if it may have one in subdirs.
71
+ """
72
+ for potential_repo in self.repo_list:
73
+ if lora_name.startswith(potential_repo) and (
74
+ len(lora_name) == len(potential_repo)
75
+ or lora_name[len(potential_repo)] == "/"
76
+ ):
77
+ return potential_repo
78
+ return None
79
+
80
+ async def _resolve_repo_subpath(
81
+ self, lora_name: str, maybe_repo: str | None
82
+ ) -> str | None:
83
+ """Given the fully qualified path of the LoRA with respect to the HF
84
+ Repo, get the subpath to download from assuming it's actually got an
85
+ adapter in it.
86
+
87
+ Args:
88
+ lora_name: Path to LoRA in HF Hub, e.g., <org>/<repo>/<subpath>
89
+ maybe_repo: Path to the repo to match against if one exists.
90
+ """
91
+ if maybe_repo is None:
92
+ return None
93
+ repo_len = len(maybe_repo)
94
+ if lora_name == maybe_repo or (
95
+ len(lora_name) == repo_len + 1 and lora_name[-1] == "/"
96
+ ):
97
+ # Resolves to the root of the directory
98
+ adapter_dir = "."
99
+ else:
100
+ # It's a subpath; removing trailing slashes if there are any
101
+ adapter_dir = lora_name[repo_len + 1 :].rstrip("/")
102
+
103
+ # Only download if the directory actually contains an adapter
104
+ is_adapter = adapter_dir in self.adapter_dirs[maybe_repo]
105
+ return adapter_dir if is_adapter else None
106
+
107
+ async def _get_adapter_dirs(self, repo_name: str) -> set[str]:
108
+ """Gets the subpaths within a HF repo that contain an adapter config.
109
+
110
+ Args:
111
+ repo_name: Name of the HF hub repo to inspect.
112
+ """
113
+ repo_files = await asyncio.to_thread(HfApi().list_repo_files, repo_id=repo_name)
114
+ adapter_dirs = {
115
+ os.path.dirname(name)
116
+ for name in repo_files
117
+ if name.endswith("adapter_config.json")
118
+ }
119
+ if "adapter_config.json" in repo_files:
120
+ adapter_dirs.add(".")
121
+ return adapter_dirs
122
+
123
+
124
+def register_hf_hub_resolver():
125
+ """Register the Hf hub LoRA Resolver with vLLM"""
126
+
127
+ hf_repo_list = envs.VLLM_LORA_RESOLVER_HF_REPO_LIST
128
+ is_enabled = (
129
+ envs.VLLM_PLUGINS is not None and "lora_hf_hub_resolver" in envs.VLLM_PLUGINS
130
+ )
131
+ if hf_repo_list:
132
+ if not is_enabled:
133
+ logger.warning(
134
+ "It appears that VLLM_LORA_RESOLVER_HF_REPO_LIST is set, but "
135
+ "lora_hf_hub_resolver is not enabled in VLLM_PLUGINS; you must"
136
+ " enable this resolver directly in VLLM_PLUGINS to use it "
137
+ " because it allows remote downloads."
138
+ )
139
+ else:
140
+ hf_hub_resolver = HfHubResolver(hf_repo_list.split(","))
141
+ LoRAResolverRegistry.register_resolver("Hf Hub Resolver", hf_hub_resolver)
142
+
143
+ return