A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 23:45 UTC
[V0 Deprecation] Remove Prompt Adapters (#20588)
Signed-off-by: mgoin <mgoin64@gmail.com>
Michael Goin committed
Jul 23, 2025 at 19:36 UTC
82ec66f514c99546779c47e0da4eae28bd6353dc
60 files changed
+126
-1727
docs/api/README.md
-1
@@ -14,7 +14,6 @@ API documentation for vLLM's configuration classes.
14
- [vllm.config.DeviceConfig][]
15
- [vllm.config.SpeculativeConfig][]
16
- [vllm.config.LoRAConfig][]
17
-- [vllm.config.PromptAdapterConfig][]
17
- [vllm.config.MultiModalConfig][]
18
- [vllm.config.PoolerConfig][]
19
- [vllm.config.DecodingConfig][]
docs/features/compatibility_matrix.md
+16
-18
@@ -34,23 +34,22 @@ th:not(:first-child) {
34
}
35
</style>
36
37
-| Feature | [CP][chunked-prefill] | [APC](automatic_prefix_caching.md) | [LoRA](lora.md) | <abbr title="Prompt Adapter">prmpt adptr</abbr> | [SD](spec_decode.md) | CUDA graph | <abbr title="Pooling Models">pooling</abbr> | <abbr title="Encoder-Decoder Models">enc-dec</abbr> | <abbr title="Logprobs">logP</abbr> | <abbr title="Prompt Logprobs">prmpt logP</abbr> | <abbr title="Async Output Processing">async output</abbr> | multi-step | <abbr title="Multimodal Inputs">mm</abbr> | best-of | beam-search |
38
-|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
37
+| Feature | [CP][chunked-prefill] | [APC](automatic_prefix_caching.md) | [LoRA](lora.md) | [SD](spec_decode.md) | CUDA graph | <abbr title="Pooling Models">pooling</abbr> | <abbr title="Encoder-Decoder Models">enc-dec</abbr> | <abbr title="Logprobs">logP</abbr> | <abbr title="Prompt Logprobs">prmpt logP</abbr> | <abbr title="Async Output Processing">async output</abbr> | multi-step | <abbr title="Multimodal Inputs">mm</abbr> | best-of | beam-search |
38
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
39
| [CP][chunked-prefill] | ✅ | | | | | | | | | | | | | | |
40
| [APC](automatic_prefix_caching.md) | ✅ | ✅ | | | | | | | | | | | | | |
41
| [LoRA](lora.md) | ✅ | ✅ | ✅ | | | | | | | | | | | | |
42
-| <abbr title="Prompt Adapter">prmpt adptr</abbr> | ✅ | ✅ | ✅ | ✅ | | | | | | | | | | | |
43
-| [SD](spec_decode.md) | ✅ | ✅ | ❌ | ✅ | ✅ | | | | | | | | | | |
44
-| CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | | | |
45
-| <abbr title="Pooling Models">pooling</abbr> | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | | | | | | | | |
46
-| <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ❌ | [❌](gh-issue:7366) | ❌ | ❌ | [❌](gh-issue:7366) | ✅ | ✅ | ✅ | | | | | | | |
47
-| <abbr title="Logprobs">logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | | | | | | |
48
-| <abbr title="Prompt Logprobs">prmpt logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | | | | | |
49
-| <abbr title="Async Output Processing">async output</abbr> | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | | | | |
50
-| multi-step | ❌ | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | | | |
51
-| <abbr title="Multimodal Inputs">mm</abbr> | ✅ | [🟠](gh-pr:8348) | [🟠](gh-pr:4194) | ❔ | ❔ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | | |
52
-| best-of | ✅ | ✅ | ✅ | ✅ | [❌](gh-issue:6137) | ✅ | ❌ | ✅ | ✅ | ✅ | ❔ | [❌](gh-issue:7968) | ✅ | ✅ | |
53
-| beam-search | ✅ | ✅ | ✅ | ✅ | [❌](gh-issue:6137) | ✅ | ❌ | ✅ | ✅ | ✅ | ❔ | [❌](gh-issue:7968) | ❔ | ✅ | ✅ |
42
+| [SD](spec_decode.md) | ✅ | ✅ | ❌ | ✅ | | | | | | | | | | |
43
+| CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | | | |
44
+| <abbr title="Pooling Models">pooling</abbr> | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | | | | | | | | |
45
+| <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ❌ | [❌](gh-issue:7366) | ❌ | [❌](gh-issue:7366) | ✅ | ✅ | ✅ | | | | | | | |
46
+| <abbr title="Logprobs">logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | | | | | | |
47
+| <abbr title="Prompt Logprobs">prmpt logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | | | | | |
48
+| <abbr title="Async Output Processing">async output</abbr> | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | | | | |
49
+| multi-step | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | | | |
50
+| <abbr title="Multimodal Inputs">mm</abbr> | ✅ | [🟠](gh-pr:8348) | [🟠](gh-pr:4194) | ❔ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | | |
51
+| best-of | ✅ | ✅ | ✅ | [❌](gh-issue:6137) | ✅ | ❌ | ✅ | ✅ | ✅ | ❔ | [❌](gh-issue:7968) | ✅ | ✅ | |
52
+| beam-search | ✅ | ✅ | ✅ | [❌](gh-issue:6137) | ✅ | ❌ | ✅ | ✅ | ✅ | ❔ | [❌](gh-issue:7968) | ❔ | ✅ | ✅ |
53
54
[](){ #feature-x-hardware }
55
@@ -59,10 +58,9 @@ th:not(:first-child) {
58
| Feature | Volta | Turing | Ampere | Ada | Hopper | CPU | AMD | TPU |
59
|-----------------------------------------------------------|---------------------|-----------|-----------|--------|------------|--------------------|--------|-----|
60
| [CP][chunked-prefill] | [❌](gh-issue:2729) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
62
-| [APC](automatic_prefix_caching.md) | [❌](gh-issue:3687) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
63
-| [LoRA](lora.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
64
-| <abbr title="Prompt Adapter">prmpt adptr</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | [❌](gh-issue:8475) | ✅ | ❌ |
65
-| [SD](spec_decode.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
61
+| [APC](automatic_prefix_caching.md) | [❌](gh-issue:3687) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
62
+| [LoRA](lora.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
63
+| [SD](spec_decode.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
64
| CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
65
| <abbr title="Pooling Models">pooling</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ❌ |
66
| <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
pyproject.toml
-1
@@ -72,7 +72,6 @@ line-length = 80
72
"vllm/core/**/*.py" = ["UP006", "UP035"]
73
"vllm/engine/**/*.py" = ["UP006", "UP035"]
74
"vllm/executor/**/*.py" = ["UP006", "UP035"]
75
-"vllm/prompt_adapter/**/*.py" = ["UP006", "UP035"]
75
"vllm/worker/**/*.py" = ["UP006", "UP035"]
76
# Python 3.8 typing - skip utils for ROCm
77
"vllm/utils/__init__.py" = ["UP006", "UP035"]
tests/entrypoints/openai/test_completion.py
+30
-42
@@ -2,6 +2,7 @@
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
# imports for guided decoding tests
4
import json
5
+import os
6
import shutil
7
from tempfile import TemporaryDirectory
8
from typing import Optional
@@ -26,10 +27,6 @@ MODEL_NAME = "HuggingFaceH4/zephyr-7b-beta"
27
# technically these adapters use a different base model,
28
# but we're not testing generation quality here
29
LORA_NAME = "typeof/zephyr-7b-beta-lora"
29
-PA_NAME = "swapnilbp/llama_tweet_ptune"
30
-# if PA_NAME changes, PA_NUM_VIRTUAL_TOKENS might also
31
-# need to change to match the prompt adapter
32
-PA_NUM_VIRTUAL_TOKENS = 8
30
31
GUIDED_DECODING_BACKENDS = ["outlines", "lm-format-enforcer", "xgrammar"]
32
@@ -56,13 +53,7 @@ def zephyr_lora_added_tokens_files(zephyr_lora_files):
53
54
55
@pytest.fixture(scope="module")
59
-def zephyr_pa_files():
60
- return snapshot_download(repo_id=PA_NAME)
61
-
62
-
63
-@pytest.fixture(scope="module")
64
-def default_server_args(zephyr_lora_files, zephyr_lora_added_tokens_files,
65
- zephyr_pa_files):
56
+def default_server_args(zephyr_lora_files, zephyr_lora_added_tokens_files):
57
return [
58
# use half precision for speed and memory savings in CI environment
59
"--dtype",
@@ -81,15 +72,6 @@ def default_server_args(zephyr_lora_files, zephyr_lora_added_tokens_files,
72
"64",
73
"--max-cpu-loras",
74
"2",
84
- # pa config
85
- "--enable-prompt-adapter",
86
- "--prompt-adapters",
87
- f"zephyr-pa={zephyr_pa_files}",
88
- f"zephyr-pa2={zephyr_pa_files}",
89
- "--max-prompt-adapters",
90
- "2",
91
- "--max-prompt-adapter-token",
92
- "128",
75
]
76
77
@@ -98,8 +80,19 @@ def default_server_args(zephyr_lora_files, zephyr_lora_added_tokens_files,
80
def server(default_server_args, request):
81
if request.param:
82
default_server_args.append(request.param)
101
- with RemoteOpenAIServer(MODEL_NAME, default_server_args) as remote_server:
102
- yield remote_server
83
+
84
+ original_value = os.environ.get('VLLM_USE_V1')
85
+ os.environ['VLLM_USE_V1'] = '0'
86
+ try:
87
+ with RemoteOpenAIServer(MODEL_NAME,
88
+ default_server_args) as remote_server:
89
+ yield remote_server
90
+ finally:
91
+ # Restore original env value
92
+ if original_value is None:
93
+ os.environ.pop('VLLM_USE_V1', None)
94
+ else:
95
+ os.environ['VLLM_USE_V1'] = original_value
96
97
98
@pytest_asyncio.fixture
@@ -110,14 +103,11 @@ async def client(server):
103
104
@pytest.mark.asyncio
105
@pytest.mark.parametrize(
113
- # first test base model, then test loras, then test prompt adapters
114
- "model_name,num_virtual_tokens",
115
- [(MODEL_NAME, 0), ("zephyr-lora", 0), ("zephyr-lora2", 0),
116
- ("zephyr-pa", PA_NUM_VIRTUAL_TOKENS),
117
- ("zephyr-pa2", PA_NUM_VIRTUAL_TOKENS)],
106
+ # first test base model, then test loras
107
+ "model_name",
108
+ [MODEL_NAME, "zephyr-lora", "zephyr-lora2"],
109
)
119
-async def test_single_completion(client: openai.AsyncOpenAI, model_name: str,
120
- num_virtual_tokens: int):
110
+async def test_single_completion(client: openai.AsyncOpenAI, model_name: str):
111
completion = await client.completions.create(model=model_name,
112
prompt="Hello, my name is",
113
max_tokens=5,
@@ -130,9 +120,7 @@ async def test_single_completion(client: openai.AsyncOpenAI, model_name: str,
120
assert len(choice.text) >= 5
121
assert choice.finish_reason == "length"
122
assert completion.usage == openai.types.CompletionUsage(
133
- completion_tokens=5,
134
- prompt_tokens=6 + num_virtual_tokens,
135
- total_tokens=11 + num_virtual_tokens)
123
+ completion_tokens=5, prompt_tokens=6, total_tokens=11)
124
125
# test using token IDs
126
completion = await client.completions.create(
@@ -175,9 +163,9 @@ async def test_added_lora_tokens_base_model(client: openai.AsyncOpenAI):
163
164
@pytest.mark.asyncio
165
@pytest.mark.parametrize(
178
- # first test base model, then test loras, then test prompt adapters
166
+ # first test base model, then test loras
167
"model_name",
180
- [MODEL_NAME, "zephyr-lora", "zephyr-lora2", "zephyr-pa", "zephyr-pa2"],
168
+ [MODEL_NAME, "zephyr-lora", "zephyr-lora2"],
169
)
170
async def test_no_logprobs(client: openai.AsyncOpenAI, model_name: str):
171
# test using token IDs
@@ -194,9 +182,9 @@ async def test_no_logprobs(client: openai.AsyncOpenAI, model_name: str):
182
183
@pytest.mark.asyncio
184
@pytest.mark.parametrize(
197
- # just test 1 lora and 1 pa hereafter
185
+ # just test 1 lora
186
"model_name",
199
- [MODEL_NAME, "zephyr-lora", "zephyr-pa"],
187
+ [MODEL_NAME, "zephyr-lora"],
188
)
189
async def test_zero_logprobs(client: openai.AsyncOpenAI, model_name: str):
190
# test using token IDs
@@ -217,7 +205,7 @@ async def test_zero_logprobs(client: openai.AsyncOpenAI, model_name: str):
205
@pytest.mark.asyncio
206
@pytest.mark.parametrize(
207
"model_name",
220
- [MODEL_NAME, "zephyr-lora", "zephyr-pa"],
208
+ [MODEL_NAME, "zephyr-lora"],
209
)
210
async def test_some_logprobs(client: openai.AsyncOpenAI, model_name: str):
211
# test using token IDs
@@ -238,7 +226,7 @@ async def test_some_logprobs(client: openai.AsyncOpenAI, model_name: str):
226
@pytest.mark.asyncio
227
@pytest.mark.parametrize(
228
"model_name",
241
- [MODEL_NAME, "zephyr-lora", "zephyr-pa"],
229
+ [MODEL_NAME, "zephyr-lora"],
230
)
231
async def test_too_many_completion_logprobs(client: openai.AsyncOpenAI,
232
model_name: str):
@@ -314,7 +302,7 @@ async def test_prompt_logprobs_completion(client: openai.AsyncOpenAI,
302
@pytest.mark.asyncio
303
@pytest.mark.parametrize(
304
"model_name",
317
- [MODEL_NAME, "zephyr-lora", "zephyr-pa"],
305
+ [MODEL_NAME, "zephyr-lora"],
306
)
307
async def test_completion_streaming(client: openai.AsyncOpenAI,
308
model_name: str):
@@ -348,7 +336,7 @@ async def test_completion_streaming(client: openai.AsyncOpenAI,
336
@pytest.mark.asyncio
337
@pytest.mark.parametrize(
338
"model_name",
351
- [MODEL_NAME, "zephyr-lora", "zephyr-pa"],
339
+ [MODEL_NAME, "zephyr-lora"],
340
)
341
async def test_parallel_streaming(client: openai.AsyncOpenAI, model_name: str):
342
"""Streaming for parallel sampling.
@@ -382,7 +370,7 @@ async def test_parallel_streaming(client: openai.AsyncOpenAI, model_name: str):
370
@pytest.mark.asyncio
371
@pytest.mark.parametrize(
372
"model_name",
385
- [MODEL_NAME, "zephyr-lora", "zephyr-pa"],
373
+ [MODEL_NAME, "zephyr-lora"],
374
)
375
async def test_completion_stream_options(client: openai.AsyncOpenAI,
376
model_name: str):
@@ -519,7 +507,7 @@ async def test_completion_stream_options(client: openai.AsyncOpenAI,
507
@pytest.mark.asyncio
508
@pytest.mark.parametrize(
509
"model_name",
522
- [MODEL_NAME, "zephyr-lora", "zephyr-pa"],
510
+ [MODEL_NAME, "zephyr-lora"],
511
)
512
async def test_batch_completions(client: openai.AsyncOpenAI, model_name: str):
513
# test both text and token IDs
tests/entrypoints/openai/test_return_tokens_as_ids.py
-1
@@ -13,7 +13,6 @@ from ...utils import RemoteOpenAIServer
13
from .test_completion import default_server_args # noqa: F401
14
from .test_completion import zephyr_lora_added_tokens_files # noqa: F401
15
from .test_completion import zephyr_lora_files # noqa: F401
16
-from .test_completion import zephyr_pa_files # noqa: F401
16
from .test_completion import MODEL_NAME
17
18
tests/entrypoints/openai/test_serving_models.py
+1
-2
@@ -32,8 +32,7 @@ async def _async_serving_models_init() -> OpenAIServingModels:
32
serving_models = OpenAIServingModels(engine_client=mock_engine_client,
33
base_model_paths=BASE_MODEL_PATHS,
34
model_config=mock_model_config,
35
- lora_modules=None,
36
- prompt_adapters=None)
35
+ lora_modules=None)
36
await serving_models.init_static_loras()
37
38
return serving_models
tests/prompt_adapter/test_bloom.py
deleted
-48
@@ -1,48 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-import pytest
5
-
6
-import vllm
7
-from vllm.prompt_adapter.request import PromptAdapterRequest
8
-
9
-MODEL_PATH = "bigscience/bloomz-560m"
10
-PA_PATH = 'stevhliu/bloomz-560m_PROMPT_TUNING_CAUSAL_LM'
11
-
12
-
13
-def do_sample(llm, pa_name: str, pa_id: int):
14
-
15
- prompts = [
16
- "Tweet text : @nationalgridus I have no water and the bill is \
17
- current and paid. Can you do something about this? Label : ",
18
- "Tweet text : @nationalgridus Looks good thanks! Label : "
19
- ]
20
- sampling_params = vllm.SamplingParams(temperature=0.0,
21
- max_tokens=3,
22
- stop_token_ids=[3])
23
-
24
- outputs = llm.generate(prompts,
25
- sampling_params,
26
- prompt_adapter_request=PromptAdapterRequest(
27
- pa_name, pa_id, PA_PATH, 8) if pa_id else None)
28
-
29
- # Print the outputs.
30
- generated_texts = []
31
- for output in outputs:
32
- prompt = output.prompt
33
- generated_text = output.outputs[0].text.strip()
34
- generated_texts.append(generated_text)
35
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
36
- return generated_texts
37
-
38
-
39
-@pytest.mark.parametrize("enforce_eager", [True, False])
40
-def test_twitter_prompt_adapter(enforce_eager: bool):
41
- llm = vllm.LLM(MODEL_PATH,
42
- enforce_eager=enforce_eager,
43
- enable_prompt_adapter=True,
44
- max_prompt_adapter_token=8)
45
-
46
- expected_output = ['complaint', 'no complaint']
47
-
48
- assert do_sample(llm, "twitter_pa", pa_id=1) == expected_output
tests/prompt_adapter/test_multi_adapter_inference.py
deleted
-56
@@ -1,56 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-from vllm import EngineArgs, LLMEngine, SamplingParams
5
-from vllm.prompt_adapter.request import PromptAdapterRequest
6
-
7
-MODEL_PATH = "bigscience/bloomz-560m"
8
-pa_path = 'stevhliu/bloomz-560m_PROMPT_TUNING_CAUSAL_LM'
9
-pa_path2 = 'swapnilbp/angry_tweet_ptune'
10
-
11
-
12
-def do_sample(engine):
13
-
14
- prompts = [
15
- ("Tweet text: I have complaints! Label: ",
16
- SamplingParams(temperature=0.0, max_tokens=3, stop_token_ids=[3]),
17
- PromptAdapterRequest("hate_speech", 1, pa_path2, 8)),
18
- ("Tweet text: I have no problems Label: ",
19
- SamplingParams(temperature=0.0, max_tokens=3, stop_token_ids=[3]),
20
- PromptAdapterRequest("hate_speech2", 2, pa_path2, 8)),
21
- ("Tweet text: I have complaints! Label: ",
22
- SamplingParams(temperature=0.0, max_tokens=3), None),
23
- ("Tweet text: I have no problems Label: ",
24
- SamplingParams(temperature=0.0, max_tokens=3, stop_token_ids=[3]),
25
- PromptAdapterRequest("complain", 3, pa_path, 8)),
26
- ]
27
-
28
- request_id = 0
29
- results = set()
30
- while prompts or engine.has_unfinished_requests():
31
- if prompts:
32
- prompt, sampling_params, pa_request = prompts.pop(0)
33
- engine.add_request(str(request_id),
34
- prompt,
35
- sampling_params,
36
- prompt_adapter_request=pa_request)
37
- request_id += 1
38
-
39
- request_outputs = engine.step()
40
-
41
- for request_output in request_outputs:
42
- if request_output.finished:
43
- results.add(request_output.outputs[0].text)
44
- return results
45
-
46
-
47
-def test_multi_prompt_adapters():
48
- engine_args = EngineArgs(model=MODEL_PATH,
49
- max_prompt_adapters=3,
50
- enable_prompt_adapter=True,
51
- max_prompt_adapter_token=8)
52
- engine = LLMEngine.from_engine_args(engine_args)
53
- expected_output = {
54
- ' quot;I', 'hate speech', 'no complaint', 'not hate speech'
55
- }
56
- assert do_sample(engine) == expected_output
tests/prompt_adapter/test_pa_lora.py
deleted
-64
@@ -1,64 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-from huggingface_hub import snapshot_download
5
-
6
-from vllm import EngineArgs, LLMEngine, SamplingParams
7
-from vllm.lora.request import LoRARequest
8
-from vllm.prompt_adapter.request import PromptAdapterRequest
9
-
10
-MODEL_PATH = "meta-llama/Llama-2-7b-hf"
11
-pa_path = snapshot_download(repo_id="swapnilbp/llama_tweet_ptune")
12
-lora_path = snapshot_download(repo_id="yard1/llama-2-7b-sql-lora-test")
13
-
14
-
15
-def do_sample(engine):
16
-
17
- prompt_text = "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_74 (icao VARCHAR, airport VARCHAR)\n\n question: Name the ICAO for lilongwe international airport [/user] [assistant]" # noqa: E501
18
-
19
- # first prompt with a prompt adapter and second without adapter
20
- prompts = [
21
- (prompt_text,
22
- SamplingParams(temperature=0.0, max_tokens=100,
23
- stop=["[/assistant]"]),
24
- PromptAdapterRequest("hate_speech", 1, pa_path,
25
- 8), LoRARequest("sql_test", 1, lora_path)),
26
- (prompt_text,
27
- SamplingParams(temperature=0.0, max_tokens=100,
28
- stop=["[/assistant]"]), None,
29
- LoRARequest("sql_test", 1, lora_path)),
30
- ]
31
-
32
- request_id = 0
33
- results = set()
34
- while prompts or engine.has_unfinished_requests():
35
- if prompts:
36
- prompt, sampling_params, pa_request, lora_request = prompts.pop(0)
37
- engine.add_request(str(request_id),
38
- prompt,
39
- sampling_params,
40
- prompt_adapter_request=pa_request,
41
- lora_request=lora_request)
42
- request_id += 1
43
-
44
- request_outputs = engine.step()
45
-
46
- for request_output in request_outputs:
47
- if request_output.finished:
48
- results.add(request_output.outputs[0].text)
49
- return results
50
-
51
-
52
-def test_lora_prompt_adapter():
53
- engine_args = EngineArgs(model=MODEL_PATH,
54
- enable_prompt_adapter=True,
55
- enable_lora=True,
56
- max_num_seqs=60,
57
- max_prompt_adapter_token=8)
58
- engine = LLMEngine.from_engine_args(engine_args)
59
- result = do_sample(engine)
60
-
61
- expected_output = {
62
- " SELECT icao FROM table_name_74 WHERE airport = 'lilongwe international airport' " # noqa: E501
63
- }
64
- assert result == expected_output
tools/mypy.sh
-1
@@ -31,6 +31,5 @@ run_mypy vllm/inputs
31
run_mypy vllm/lora
32
run_mypy vllm/model_executor
33
run_mypy vllm/plugins
34
-run_mypy vllm/prompt_adapter
34
run_mypy vllm/worker
35
run_mypy vllm/v1
vllm/config.py
-62
@@ -3143,59 +3143,6 @@ class LoRAConfig:
3143
self.lora_dtype = getattr(torch, self.lora_dtype)
3144
3145
3146
-@config
3147
-@dataclass(config=ConfigDict(arbitrary_types_allowed=True))
3148
-class PromptAdapterConfig:
3149
- """Configuration for PromptAdapters."""
3150
-
3151
- max_prompt_adapters: int = 1
3152
- """Max number of PromptAdapters in a batch."""
3153
- max_prompt_adapter_token: int = 0
3154
- """Max number of PromptAdapters tokens."""
3155
- max_cpu_prompt_adapters: Optional[int] = None
3156
- """Maximum number of PromptAdapters to store in CPU memory. Must be >= than
3157
- `max_prompt_adapters`."""
3158
- prompt_adapter_dtype: Union[torch.dtype, str] = "auto"
3159
- """Data type for PromptAdapter. If auto, will default to base model dtype.
3160
- """
3161
-
3162
- def compute_hash(self) -> str:
3163
- """
3164
- WARNING: Whenever a new field is added to this config,
3165
- ensure that it is included in the factors list if
3166
- it affects the computation graph.
3167
-
3168
- Provide a hash that uniquely identifies all the configs
3169
- that affect the structure of the computation
3170
- graph from input ids/embeddings to the final hidden states,
3171
- excluding anything before input ids/embeddings and after
3172
- the final hidden states.
3173
- """
3174
- # no factors to consider.
3175
- # this config will not affect the computation graph.
3176
- factors: list[Any] = []
3177
- hash_str = hashlib.md5(str(factors).encode(),
3178
- usedforsecurity=False).hexdigest()
3179
- return hash_str
3180
-
3181
- def __post_init__(self):
3182
-
3183
- if self.max_prompt_adapters < 1:
3184
- raise ValueError(f"max_prompt_adapters "
3185
- f"({self.max_prompt_adapters}) must be >= 1.")
3186
- if self.max_prompt_adapter_token == 0:
3187
- raise ValueError("max_prompt_adapter_token must be set.")
3188
- if self.max_cpu_prompt_adapters is None:
3189
- self.max_cpu_prompt_adapters = self.max_prompt_adapters
3190
-
3191
- def verify_with_model_config(self, model_config: ModelConfig):
3192
- if self.prompt_adapter_dtype == "auto":
3193
- self.prompt_adapter_dtype = model_config.dtype
3194
- elif isinstance(self.prompt_adapter_dtype, str):
3195
- self.prompt_adapter_dtype = getattr(torch,
3196
- self.prompt_adapter_dtype)
3197
-
3198
-
3146
@config
3147
@dataclass
3148
class MultiModalConfig:
@@ -4402,8 +4349,6 @@ class VllmConfig:
4349
"""Decoding configuration."""
4350
observability_config: Optional[ObservabilityConfig] = None
4351
"""Observability configuration."""
4405
- prompt_adapter_config: Optional[PromptAdapterConfig] = None
4406
- """Prompt adapter configuration."""
4352
quant_config: Optional[QuantizationConfig] = None
4353
"""Quantization configuration."""
4354
compilation_config: CompilationConfig = field(
@@ -4500,10 +4445,6 @@ class VllmConfig:
4445
vllm_factors.append(self.observability_config.compute_hash())
4446
else:
4447
vllm_factors.append("None")
4503
- if self.prompt_adapter_config:
4504
- vllm_factors.append(self.prompt_adapter_config.compute_hash())
4505
- else:
4506
- vllm_factors.append("None")
4448
if self.quant_config:
4449
pass # should be captured by model_config.quantization
4450
if self.compilation_config:
@@ -4611,9 +4552,6 @@ class VllmConfig:
4552
if self.lora_config is not None:
4553
self.lora_config.verify_with_cache_config(self.cache_config)
4554
self.lora_config.verify_with_model_config(self.model_config)
4614
- if self.prompt_adapter_config is not None:
4615
- self.prompt_adapter_config.verify_with_model_config(
4616
- self.model_config)
4555
4556
if self.quant_config is None and self.model_config is not None:
4557
self.quant_config = VllmConfig._get_quantization_config(
vllm/core/scheduler.py
-12
@@ -15,7 +15,6 @@ from vllm.config import CacheConfig, LoRAConfig, SchedulerConfig
15
from vllm.core.interfaces import AllocStatus, BlockSpaceManager
16
from vllm.logger import init_logger
17
from vllm.lora.request import LoRARequest
18
-from vllm.prompt_adapter.request import PromptAdapterRequest
18
from vllm.sequence import (Sequence, SequenceData, SequenceGroup,
19
SequenceGroupBase, SequenceGroupMetadata,
20
SequenceGroupMetadataDelta, SequenceStage,
@@ -165,8 +164,6 @@ class SchedulerOutputs:
164
if self.num_loras > 0:
165
self._sort_by_lora_ids()
166
168
- self.num_prompt_adapters: int = len(self.prompt_adapter_requests)
169
-
167
def is_empty(self) -> bool:
168
# NOTE: We do not consider the ignored sequence groups.
169
return (not self.scheduled_seq_groups and not self.blocks_to_swap_in
@@ -194,14 +191,6 @@ class SchedulerOutputs:
191
if g.seq_group.lora_request is not None
192
}
193
197
- @property
198
- def prompt_adapter_requests(self) -> Set[PromptAdapterRequest]:
199
- return {
200
- g.seq_group.prompt_adapter_request
201
- for g in self.scheduled_seq_groups
202
- if g.seq_group.prompt_adapter_request is not None
203
- }
204
-
194
195
@dataclass
196
class SchedulerRunningOutputs:
@@ -1648,7 +1637,6 @@ class Scheduler:
1637
multi_modal_placeholders=(
1638
seq_group.multi_modal_placeholders
1639
if scheduler_outputs.num_prefill_groups > 0 else None),
1651
- prompt_adapter_request=seq_group.prompt_adapter_request,
1640
)
1641
else:
1642
# When SPMD mode is enabled, we only send delta data except for
vllm/engine/arg_utils.py
+11
-38
@@ -30,9 +30,9 @@ from vllm.config import (BlockSize, CacheConfig, CacheDType, CompilationConfig,
30
LogprobsMode, LoRAConfig, ModelConfig, ModelDType,
31
ModelImpl, MultiModalConfig, ObservabilityConfig,
32
ParallelConfig, PoolerConfig, PrefixCachingHashAlgo,
33
- PromptAdapterConfig, SchedulerConfig, SchedulerPolicy,
34
- SpeculativeConfig, TaskOption, TokenizerMode,
35
- VllmConfig, get_attr_docs, get_field)
33
+ SchedulerConfig, SchedulerPolicy, SpeculativeConfig,
34
+ TaskOption, TokenizerMode, VllmConfig, get_attr_docs,
35
+ get_field)
36
from vllm.logger import init_logger
37
from vllm.platforms import CpuArchEnum, current_platform
38
from vllm.plugins import load_general_plugins
@@ -358,11 +358,6 @@ class EngineArgs:
358
max_cpu_loras: Optional[int] = LoRAConfig.max_cpu_loras
359
lora_dtype: Optional[Union[str, torch.dtype]] = LoRAConfig.lora_dtype
360
lora_extra_vocab_size: int = LoRAConfig.lora_extra_vocab_size
361
- # PromptAdapter fields
362
- enable_prompt_adapter: bool = False
363
- max_prompt_adapters: int = PromptAdapterConfig.max_prompt_adapters
364
- max_prompt_adapter_token: int = \
365
- PromptAdapterConfig.max_prompt_adapter_token
361
362
num_scheduler_steps: int = SchedulerConfig.num_scheduler_steps
363
multi_step_stream_outputs: bool = SchedulerConfig.multi_step_stream_outputs
@@ -437,6 +432,8 @@ class EngineArgs:
432
ParallelConfig.enable_multimodal_encoder_data_parallel
433
434
async_scheduling: bool = SchedulerConfig.async_scheduling
435
+ # DEPRECATED
436
+ enable_prompt_adapter: bool = False
437
438
def __post_init__(self):
439
# support `EngineArgs(compilation_config={...})`
@@ -729,23 +726,6 @@ class EngineArgs:
726
lora_group.add_argument("--default-mm-loras",
727
**lora_kwargs["default_mm_loras"])
728
732
- # PromptAdapter related configs
733
- prompt_adapter_kwargs = get_kwargs(PromptAdapterConfig)
734
- prompt_adapter_group = parser.add_argument_group(
735
- title="PromptAdapterConfig",
736
- description=PromptAdapterConfig.__doc__,
737
- )
738
- prompt_adapter_group.add_argument(
739
- "--enable-prompt-adapter",
740
- action=argparse.BooleanOptionalAction,
741
- help="If True, enable handling of PromptAdapters.")
742
- prompt_adapter_group.add_argument(
743
- "--max-prompt-adapters",
744
- **prompt_adapter_kwargs["max_prompt_adapters"])
745
- prompt_adapter_group.add_argument(
746
- "--max-prompt-adapter-token",
747
- **prompt_adapter_kwargs["max_prompt_adapter_token"])
748
-
729
# Speculative arguments
730
speculative_group = parser.add_argument_group(
731
title="SpeculativeConfig",
@@ -850,6 +830,12 @@ class EngineArgs:
830
parser.add_argument('--disable-log-stats',
831
action='store_true',
832
help='Disable logging statistics.')
833
+ parser.add_argument('--enable-prompt-adapter',
834
+ action='store_true',
835
+ deprecated=True,
836
+ help='[DEPRECATED] Prompt adapter has been '
837
+ 'removed. Setting this flag to True or False'
838
+ ' has no effect on vLLM behavior.')
839
840
return parser
841
@@ -1234,11 +1220,6 @@ class EngineArgs:
1220
1221
load_config = self.create_load_config()
1222
1237
- prompt_adapter_config = PromptAdapterConfig(
1238
- max_prompt_adapters=self.max_prompt_adapters,
1239
- max_prompt_adapter_token=self.max_prompt_adapter_token) \
1240
- if self.enable_prompt_adapter else None
1241
-
1223
decoding_config = DecodingConfig(
1224
backend=self.guided_decoding_backend,
1225
disable_fallback=self.guided_decoding_disable_fallback,
@@ -1266,7 +1247,6 @@ class EngineArgs:
1247
load_config=load_config,
1248
decoding_config=decoding_config,
1249
observability_config=observability_config,
1269
- prompt_adapter_config=prompt_adapter_config,
1250
compilation_config=self.compilation_config,
1251
kv_transfer_config=self.kv_transfer_config,
1252
kv_events_config=self.kv_events_config,
@@ -1342,12 +1322,6 @@ class EngineArgs:
1322
recommend_to_remove=False)
1323
return False
1324
1345
- # No Prompt Adapter so far.
1346
- if self.enable_prompt_adapter:
1347
- _raise_or_fallback(feature_name="--enable-prompt-adapter",
1348
- recommend_to_remove=False)
1349
- return False
1350
-
1325
# No text embedding inputs so far.
1326
if self.enable_prompt_embeds:
1327
_raise_or_fallback(feature_name="--enable-prompt-embeds",
@@ -1469,7 +1443,6 @@ class EngineArgs:
1443
1444
if (is_gpu and not use_sliding_window and not use_spec_decode
1445
and not self.enable_lora
1472
- and not self.enable_prompt_adapter
1446
and model_config.runner_type != "pooling"):
1447
self.enable_chunked_prefill = True
1448
logger.warning(
vllm/engine/async_llm_engine.py
-10
@@ -29,7 +29,6 @@ from vllm.model_executor.guided_decoding import (
29
from vllm.model_executor.layers.sampler import SamplerOutput
30
from vllm.outputs import PoolingRequestOutput, RequestOutput
31
from vllm.pooling_params import PoolingParams
32
-from vllm.prompt_adapter.request import PromptAdapterRequest
32
from vllm.sampling_params import SamplingParams
33
from vllm.sequence import ExecuteModelRequest
34
from vllm.transformers_utils.tokenizer import AnyTokenizer
@@ -435,7 +434,6 @@ class _AsyncLLMEngine(LLMEngine):
434
arrival_time: Optional[float] = None,
435
lora_request: Optional[LoRARequest] = None,
436
trace_headers: Optional[Mapping[str, str]] = None,
438
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
437
priority: int = 0,
438
data_parallel_rank: Optional[int] = None,
439
tokenization_kwargs: Optional[dict[str, Any]] = None,
@@ -468,7 +466,6 @@ class _AsyncLLMEngine(LLMEngine):
466
processed_inputs = await self.input_preprocessor.preprocess_async(
467
prompt,
468
lora_request=lora_request,
471
- prompt_adapter_request=prompt_adapter_request,
469
tokenization_kwargs=tokenization_kwargs,
470
)
471
@@ -491,7 +488,6 @@ class _AsyncLLMEngine(LLMEngine):
488
params=params,
489
arrival_time=arrival_time,
490
lora_request=lora_request,
494
- prompt_adapter_request=prompt_adapter_request,
491
trace_headers=trace_headers,
492
priority=priority,
493
)
@@ -861,7 +857,6 @@ class AsyncLLMEngine(EngineClient):
857
arrival_time: Optional[float] = None,
858
lora_request: Optional[LoRARequest] = None,
859
trace_headers: Optional[Mapping[str, str]] = None,
864
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
860
priority: int = 0,
861
data_parallel_rank: Optional[int] = None,
862
tokenization_kwargs: Optional[dict[str, Any]] = None,
@@ -889,7 +884,6 @@ class AsyncLLMEngine(EngineClient):
884
arrival_time=arrival_time or time.time(),
885
lora_request=lora_request,
886
trace_headers=trace_headers,
892
- prompt_adapter_request=prompt_adapter_request,
887
priority=priority,
888
data_parallel_rank=data_parallel_rank,
889
tokenization_kwargs=tokenization_kwargs,
@@ -904,7 +898,6 @@ class AsyncLLMEngine(EngineClient):
898
request_id: str,
899
lora_request: Optional[LoRARequest] = None,
900
trace_headers: Optional[Mapping[str, str]] = None,
907
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
901
priority: int = 0,
902
data_parallel_rank: Optional[int] = None,
903
) -> AsyncGenerator[RequestOutput, None]:
@@ -922,8 +915,6 @@ class AsyncLLMEngine(EngineClient):
915
request_id: The unique id of the request.
916
lora_request: LoRA request to use for generation, if any.
917
trace_headers: OpenTelemetry trace headers.
925
- prompt_adapter_request: Prompt Adapter request to use
926
- for generation, if any.
918
priority: The priority of the request.
919
Only applicable with priority scheduling.
920
data_parallel_rank: The (global) data parallel rank that must
@@ -983,7 +974,6 @@ class AsyncLLMEngine(EngineClient):
974
sampling_params,
975
lora_request=lora_request,
976
trace_headers=trace_headers,
986
- prompt_adapter_request=prompt_adapter_request,
977
priority=priority,
978
data_parallel_rank=data_parallel_rank,
979
):
vllm/engine/llm_engine.py
+18
-50
@@ -44,7 +44,6 @@ from vllm.multimodal.processing import EncDecMultiModalProcessor
44
from vllm.outputs import (PoolingRequestOutput, RequestOutput,
45
RequestOutputFactory)
46
from vllm.pooling_params import PoolingParams
47
-from vllm.prompt_adapter.request import PromptAdapterRequest
47
from vllm.sampling_params import RequestOutputKind, SamplingParams
48
from vllm.sequence import (ExecuteModelRequest, ParallelSampleSequenceGroup,
49
PoolingSequenceGroupOutput, Sequence, SequenceGroup,
@@ -223,7 +222,6 @@ class LLMEngine:
222
self.load_config = vllm_config.load_config
223
self.decoding_config = vllm_config.decoding_config or DecodingConfig( # noqa
224
)
226
- self.prompt_adapter_config = vllm_config.prompt_adapter_config # noqa
225
self.observability_config = vllm_config.observability_config or ObservabilityConfig( # noqa
226
)
227
@@ -294,8 +292,6 @@ class LLMEngine:
292
# Feature flags
293
"enable_lora":
294
bool(self.lora_config),
297
- "enable_prompt_adapter":
298
- bool(self.prompt_adapter_config),
295
"enable_prefix_caching":
296
self.cache_config.enable_prefix_caching,
297
"enforce_eager":
@@ -542,9 +538,6 @@ class LLMEngine:
538
self.lora_config.verify_with_model_config(self.model_config)
539
self.lora_config.verify_with_scheduler_config(
540
self.scheduler_config)
545
- if self.prompt_adapter_config:
546
- self.prompt_adapter_config.verify_with_model_config(
547
- self.model_config)
541
542
def _add_processed_request(
543
self,
@@ -553,7 +546,6 @@ class LLMEngine:
546
params: Union[SamplingParams, PoolingParams],
547
arrival_time: float,
548
lora_request: Optional[LoRARequest],
556
- prompt_adapter_request: Optional[PromptAdapterRequest],
549
trace_headers: Optional[Mapping[str, str]] = None,
550
priority: int = 0,
551
) -> Optional[SequenceGroup]:
@@ -569,7 +561,6 @@ class LLMEngine:
561
arrival_time=arrival_time,
562
lora_request=lora_request,
563
trace_headers=trace_headers,
572
- prompt_adapter_request=prompt_adapter_request,
564
priority=priority,
565
)
566
return None
@@ -583,11 +574,10 @@ class LLMEngine:
574
encoder_inputs, decoder_inputs = split_enc_dec_inputs(processed_inputs)
575
576
seq = Sequence(seq_id, decoder_inputs, block_size, eos_token_id,
586
- lora_request, prompt_adapter_request)
577
+ lora_request)
578
579
encoder_seq = (None if encoder_inputs is None else Sequence(
589
- seq_id, encoder_inputs, block_size, eos_token_id, lora_request,
590
- prompt_adapter_request))
580
+ seq_id, encoder_inputs, block_size, eos_token_id, lora_request))
581
582
# Create a SequenceGroup based on SamplingParams or PoolingParams
583
if isinstance(params, SamplingParams):
@@ -598,7 +588,6 @@ class LLMEngine:
588
arrival_time=arrival_time,
589
lora_request=lora_request,
590
trace_headers=trace_headers,
601
- prompt_adapter_request=prompt_adapter_request,
591
encoder_seq=encoder_seq,
592
priority=priority)
593
elif isinstance(params, PoolingParams):
@@ -608,7 +597,6 @@ class LLMEngine:
597
params,
598
arrival_time=arrival_time,
599
lora_request=lora_request,
611
- prompt_adapter_request=prompt_adapter_request,
600
encoder_seq=encoder_seq,
601
priority=priority)
602
else:
@@ -637,7 +625,6 @@ class LLMEngine:
625
lora_request: Optional[LoRARequest] = None,
626
tokenization_kwargs: Optional[dict[str, Any]] = None,
627
trace_headers: Optional[Mapping[str, str]] = None,
640
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
628
priority: int = 0,
629
) -> None:
630
"""Add a request to the engine's request pool.
@@ -658,7 +645,6 @@ class LLMEngine:
645
the current monotonic time.
646
lora_request: The LoRA request to add.
647
trace_headers: OpenTelemetry trace headers.
661
- prompt_adapter_request: The prompt adapter request to add.
648
priority: The priority of the request.
649
Only applicable with priority scheduling.
650
@@ -719,7 +705,6 @@ class LLMEngine:
705
prompt,
706
tokenization_kwargs=tokenization_kwargs,
707
lora_request=lora_request,
722
- prompt_adapter_request=prompt_adapter_request,
708
)
709
710
self._add_processed_request(
@@ -728,7 +713,6 @@ class LLMEngine:
713
params=params,
714
arrival_time=arrival_time,
715
lora_request=lora_request,
731
- prompt_adapter_request=prompt_adapter_request,
716
trace_headers=trace_headers,
717
priority=priority,
718
)
@@ -741,7 +725,6 @@ class LLMEngine:
725
arrival_time: float,
726
lora_request: Optional[LoRARequest],
727
trace_headers: Optional[Mapping[str, str]] = None,
744
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
728
encoder_seq: Optional[Sequence] = None,
729
priority: int = 0,
730
) -> SequenceGroup:
@@ -769,17 +752,15 @@ class LLMEngine:
752
if self.vllm_config.speculative_config is not None:
753
draft_size = \
754
self.vllm_config.speculative_config.num_speculative_tokens + 1
772
- seq_group = SequenceGroup(
773
- request_id=request_id,
774
- seqs=[seq],
775
- arrival_time=arrival_time,
776
- sampling_params=sampling_params,
777
- lora_request=lora_request,
778
- trace_headers=trace_headers,
779
- prompt_adapter_request=prompt_adapter_request,
780
- encoder_seq=encoder_seq,
781
- priority=priority,
782
- draft_size=draft_size)
755
+ seq_group = SequenceGroup(request_id=request_id,
756
+ seqs=[seq],
757
+ arrival_time=arrival_time,
758
+ sampling_params=sampling_params,
759
+ lora_request=lora_request,
760
+ trace_headers=trace_headers,
761
+ encoder_seq=encoder_seq,
762
+ priority=priority,
763
+ draft_size=draft_size)
764
765
return seq_group
766
@@ -790,7 +771,6 @@ class LLMEngine:
771
pooling_params: PoolingParams,
772
arrival_time: float,
773
lora_request: Optional[LoRARequest],
793
- prompt_adapter_request: Optional[PromptAdapterRequest],
774
encoder_seq: Optional[Sequence] = None,
775
priority: int = 0,
776
) -> SequenceGroup:
@@ -798,15 +778,13 @@ class LLMEngine:
778
# Defensive copy of PoolingParams, which are used by the pooler
779
pooling_params = pooling_params.clone()
780
# Create the sequence group.
801
- seq_group = SequenceGroup(
802
- request_id=request_id,
803
- seqs=[seq],
804
- arrival_time=arrival_time,
805
- lora_request=lora_request,
806
- pooling_params=pooling_params,
807
- prompt_adapter_request=prompt_adapter_request,
808
- encoder_seq=encoder_seq,
809
- priority=priority)
781
+ seq_group = SequenceGroup(request_id=request_id,
782
+ seqs=[seq],
783
+ arrival_time=arrival_time,
784
+ lora_request=lora_request,
785
+ pooling_params=pooling_params,
786
+ encoder_seq=encoder_seq,
787
+ priority=priority)
788
return seq_group
789
790
def abort_request(self, request_id: Union[str, Iterable[str]]) -> None:
@@ -1834,16 +1812,6 @@ class LLMEngine:
1812
def pin_lora(self, lora_id: int) -> bool:
1813
return self.model_executor.pin_lora(lora_id)
1814
1837
- def add_prompt_adapter(
1838
- self, prompt_adapter_request: PromptAdapterRequest) -> bool:
1839
- return self.model_executor.add_prompt_adapter(prompt_adapter_request)
1840
-
1841
- def remove_prompt_adapter(self, prompt_adapter_id: int) -> bool:
1842
- return self.model_executor.remove_prompt_adapter(prompt_adapter_id)
1843
-
1844
- def list_prompt_adapters(self) -> List[int]:
1845
- return self.model_executor.list_prompt_adapters()
1846
-
1815
def start_profile(self) -> None:
1816
self.model_executor.start_profile()
1817
vllm/engine/multiprocessing/__init__.py
-4
@@ -10,7 +10,6 @@ from vllm import PoolingParams
10
from vllm.inputs import PromptType
11
from vllm.lora.request import LoRARequest
12
from vllm.outputs import RequestOutput
13
-from vllm.prompt_adapter.request import PromptAdapterRequest
13
from vllm.sampling_params import SamplingParams
14
from vllm.utils import Device
15
@@ -33,7 +32,6 @@ class RPCProcessRequest:
32
request_id: str
33
lora_request: Optional[LoRARequest] = None
34
trace_headers: Optional[Mapping[str, str]] = None
36
- prompt_adapter_request: Optional[PromptAdapterRequest] = None
35
priority: int = 0
36
37
def __init__(
@@ -43,7 +41,6 @@ class RPCProcessRequest:
41
request_id: str,
42
lora_request: Optional[LoRARequest] = None,
43
trace_headers: Optional[Mapping[str, str]] = None,
46
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
44
priority: int = 0,
45
) -> None:
46
super().__init__()
@@ -53,7 +50,6 @@ class RPCProcessRequest:
50
self.request_id = request_id
51
self.lora_request = lora_request
52
self.trace_headers = trace_headers
56
- self.prompt_adapter_request = prompt_adapter_request
53
self.priority = priority
54
55
vllm/engine/multiprocessing/client.py
+1
-8
@@ -45,7 +45,6 @@ from vllm.logger import init_logger
45
from vllm.lora.request import LoRARequest
46
from vllm.model_executor.layers.sampler import SamplerOutput
47
from vllm.outputs import PoolingRequestOutput, RequestOutput
48
-from vllm.prompt_adapter.request import PromptAdapterRequest
48
from vllm.sampling_params import SamplingParams
49
from vllm.transformers_utils.tokenizer_group import init_tokenizer_from_configs
50
from vllm.utils import Device
@@ -448,7 +447,6 @@ class MQLLMEngineClient(EngineClient):
447
request_id: str,
448
lora_request: Optional[LoRARequest] = None,
449
trace_headers: Optional[Mapping[str, str]] = None,
451
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
450
priority: int = 0,
451
) -> AsyncGenerator[RequestOutput, None]:
452
"""Generate outputs for a request.
@@ -465,8 +463,6 @@ class MQLLMEngineClient(EngineClient):
463
request_id: The unique id of the request.
464
lora_request: LoRA request to use for generation, if any.
465
trace_headers: OpenTelemetry trace headers.
468
- prompt_adapter_request: Prompt Adapter request to use
469
- for generation, if any.
466
priority: Priority of the request (lower means earlier handling).
467
Any priority other than 0 will lead to an error if the
468
scheduling policy is not "priority".
@@ -474,8 +470,7 @@ class MQLLMEngineClient(EngineClient):
470
return cast(
471
AsyncGenerator[RequestOutput, None],
472
self._process_request(prompt, sampling_params, request_id,
477
- lora_request, trace_headers,
478
- prompt_adapter_request, priority))
473
+ lora_request, trace_headers, priority))
474
475
def encode(
476
self,
@@ -521,7 +516,6 @@ class MQLLMEngineClient(EngineClient):
516
request_id: str,
517
lora_request: Optional[LoRARequest] = None,
518
trace_headers: Optional[Mapping[str, str]] = None,
524
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
519
priority: int = 0,
520
) -> Union[AsyncGenerator[RequestOutput, None], AsyncGenerator[
521
PoolingRequestOutput, None]]:
@@ -575,7 +569,6 @@ class MQLLMEngineClient(EngineClient):
569
request_id=request_id,
570
lora_request=lora_request,
571
trace_headers=trace_headers,
578
- prompt_adapter_request=prompt_adapter_request,
572
priority=priority,
573
))
574
vllm/engine/multiprocessing/engine.py
+6
-8
@@ -304,14 +304,12 @@ class MQLLMEngine:
304
self._send_outputs(rpc_err)
305
306
try:
307
- self.engine.add_request(
308
- request_id=request_id,
309
- prompt=request.prompt,
310
- params=request.params,
311
- lora_request=request.lora_request,
312
- trace_headers=request.trace_headers,
313
- prompt_adapter_request=request.prompt_adapter_request,
314
- priority=request.priority)
307
+ self.engine.add_request(request_id=request_id,
308
+ prompt=request.prompt,
309
+ params=request.params,
310
+ lora_request=request.lora_request,
311
+ trace_headers=request.trace_headers,
312
+ priority=request.priority)
313
314
if self.log_requests:
315
logger.info("Added request %s.", request.request_id)
vllm/engine/protocol.py
-2
@@ -16,7 +16,6 @@ from vllm.lora.request import LoRARequest
16
from vllm.model_executor.layers.sampler import SamplerOutput
17
from vllm.outputs import CompletionOutput, PoolingRequestOutput, RequestOutput
18
from vllm.pooling_params import PoolingParams
19
-from vllm.prompt_adapter.request import PromptAdapterRequest
19
from vllm.sampling_params import BeamSearchParams, SamplingParams
20
from vllm.transformers_utils.tokenizer import AnyTokenizer
21
from vllm.utils import Device, collect_from_async_generator, random_uuid
@@ -55,7 +54,6 @@ class EngineClient(ABC):
54
request_id: str,
55
lora_request: Optional[LoRARequest] = None,
56
trace_headers: Optional[Mapping[str, str]] = None,
58
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
57
priority: int = 0,
58
) -> AsyncGenerator[RequestOutput, None]:
59
"""Generate outputs for a request."""
vllm/entrypoints/llm.py
+2
-44
@@ -45,7 +45,6 @@ from vllm.outputs import (ClassificationRequestOutput, EmbeddingRequestOutput,
45
PoolingRequestOutput, RequestOutput,
46
ScoringRequestOutput)
47
from vllm.pooling_params import PoolingParams, PoolingTask
48
-from vllm.prompt_adapter.request import PromptAdapterRequest
48
from vllm.sampling_params import (BeamSearchParams, GuidedDecodingParams,
49
RequestOutputKind, SamplingParams)
50
from vllm.transformers_utils.tokenizer import (AnyTokenizer, MistralTokenizer,
@@ -314,7 +313,6 @@ class LLM:
313
*,
314
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
315
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
317
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
316
guided_options_request: Optional[Union[LLMGuidedOptions,
317
GuidedDecodingRequest]] = None,
318
) -> list[RequestOutput]:
@@ -330,7 +328,6 @@ class LLM:
328
prompt_token_ids: Optional[list[int]] = None,
329
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
330
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
333
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
331
guided_options_request: Optional[Union[LLMGuidedOptions,
332
GuidedDecodingRequest]] = None,
333
) -> list[RequestOutput]:
@@ -346,7 +343,6 @@ class LLM:
343
prompt_token_ids: Optional[list[list[int]]] = None,
344
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
345
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
349
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
346
guided_options_request: Optional[Union[LLMGuidedOptions,
347
GuidedDecodingRequest]] = None,
348
) -> list[RequestOutput]:
@@ -363,7 +359,6 @@ class LLM:
359
prompt_token_ids: list[int],
360
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
361
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
366
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
362
guided_options_request: Optional[Union[LLMGuidedOptions,
363
GuidedDecodingRequest]] = None,
364
) -> list[RequestOutput]:
@@ -380,7 +375,6 @@ class LLM:
375
prompt_token_ids: list[list[int]],
376
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
377
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
383
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
378
guided_options_request: Optional[Union[LLMGuidedOptions,
379
GuidedDecodingRequest]] = None,
380
) -> list[RequestOutput]:
@@ -395,7 +389,6 @@ class LLM:
389
prompt_token_ids: Union[list[int], list[list[int]]],
390
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
391
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
398
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
392
guided_options_request: Optional[Union[LLMGuidedOptions,
393
GuidedDecodingRequest]] = None,
394
) -> list[RequestOutput]:
@@ -415,7 +408,6 @@ class LLM:
408
prompt_token_ids: Optional[Union[list[int], list[list[int]]]] = None,
409
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
410
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
418
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
411
guided_options_request: Optional[Union[LLMGuidedOptions,
412
GuidedDecodingRequest]] = None,
413
priority: Optional[list[int]] = None,
@@ -440,8 +432,6 @@ class LLM:
432
it is used to create the progress bar.
433
If `False`, no progress bar is created.
434
lora_request: LoRA request to use for generation, if any.
443
- prompt_adapter_request: Prompt Adapter request to use for
444
- generation, if any.
435
priority: The priority of the requests, if any.
436
Only applicable when priority scheduling policy is enabled.
437
@@ -507,7 +497,6 @@ class LLM:
497
params=sampling_params,
498
use_tqdm=use_tqdm,
499
lora_request=lora_request,
510
- prompt_adapter_request=prompt_adapter_request,
500
guided_options=guided_options_request,
501
tokenization_kwargs=tokenization_kwargs,
502
priority=priority,
@@ -963,7 +952,6 @@ class LLM:
952
truncate_prompt_tokens: Optional[int] = None,
953
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
954
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
966
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
955
pooling_task: PoolingTask = "encode",
956
tokenization_kwargs: Optional[dict[str, Any]] = None,
957
) -> list[PoolingRequestOutput]:
@@ -980,7 +968,6 @@ class LLM:
968
truncate_prompt_tokens: Optional[int] = None,
969
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
970
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
983
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
971
pooling_task: PoolingTask = "encode",
972
tokenization_kwargs: Optional[dict[str, Any]] = None,
973
) -> list[PoolingRequestOutput]:
@@ -997,7 +984,6 @@ class LLM:
984
truncate_prompt_tokens: Optional[int] = None,
985
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
986
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
1000
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
987
pooling_task: PoolingTask = "encode",
988
tokenization_kwargs: Optional[dict[str, Any]] = None,
989
) -> list[PoolingRequestOutput]:
@@ -1015,7 +1001,6 @@ class LLM:
1001
truncate_prompt_tokens: Optional[int] = None,
1002
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
1003
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
1018
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
1004
pooling_task: PoolingTask = "encode",
1005
tokenization_kwargs: Optional[dict[str, Any]] = None,
1006
) -> list[PoolingRequestOutput]:
@@ -1033,7 +1018,6 @@ class LLM:
1018
truncate_prompt_tokens: Optional[int] = None,
1019
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
1020
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
1036
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
1021
pooling_task: PoolingTask = "encode",
1022
tokenization_kwargs: Optional[dict[str, Any]] = None,
1023
) -> list[PoolingRequestOutput]:
@@ -1049,7 +1033,6 @@ class LLM:
1033
truncate_prompt_tokens: Optional[int] = None,
1034
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
1035
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
1052
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
1036
pooling_task: PoolingTask = "encode",
1037
tokenization_kwargs: Optional[dict[str, Any]] = None,
1038
) -> list[PoolingRequestOutput]:
@@ -1070,7 +1053,6 @@ class LLM:
1053
truncate_prompt_tokens: Optional[int] = None,
1054
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
1055
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
1073
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
1056
pooling_task: PoolingTask = "encode",
1057
tokenization_kwargs: Optional[dict[str, Any]] = None,
1058
) -> list[PoolingRequestOutput]:
@@ -1092,8 +1074,6 @@ class LLM:
1074
it is used to create the progress bar.
1075
If `False`, no progress bar is created.
1076
lora_request: LoRA request to use for generation, if any.
1095
- prompt_adapter_request: Prompt Adapter request to use for
1096
- generation, if any.
1077
pooling_task: Override the pooling task to use.
1078
1079
Returns:
@@ -1150,7 +1130,6 @@ class LLM:
1130
use_tqdm=use_tqdm,
1131
lora_request=lora_request,
1132
tokenization_kwargs=tokenization_kwargs,
1153
- prompt_adapter_request=prompt_adapter_request,
1133
)
1134
1135
outputs = self._run_engine(use_tqdm=use_tqdm)
@@ -1167,7 +1146,6 @@ class LLM:
1146
pooling_params: Optional[Union[PoolingParams,
1147
Sequence[PoolingParams]]] = None,
1148
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
1170
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
1149
) -> list[EmbeddingRequestOutput]:
1150
"""
1151
Generate an embedding vector for each prompt.
@@ -1187,8 +1165,6 @@ class LLM:
1165
it is used to create the progress bar.
1166
If `False`, no progress bar is created.
1167
lora_request: LoRA request to use for generation, if any.
1190
- prompt_adapter_request: Prompt Adapter request to use for
1191
- generation, if any.
1168
1169
Returns:
1170
A list of `EmbeddingRequestOutput` objects containing the
@@ -1205,7 +1181,6 @@ class LLM:
1181
use_tqdm=use_tqdm,
1182
pooling_params=pooling_params,
1183
lora_request=lora_request,
1208
- prompt_adapter_request=prompt_adapter_request,
1184
pooling_task="embed",
1185
)
1186
@@ -1218,7 +1193,6 @@ class LLM:
1193
*,
1194
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
1195
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
1221
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
1196
) -> list[ClassificationRequestOutput]:
1197
"""
1198
Generate class logits for each prompt.
@@ -1236,8 +1210,6 @@ class LLM:
1210
it is used to create the progress bar.
1211
If `False`, no progress bar is created.
1212
lora_request: LoRA request to use for generation, if any.
1239
- prompt_adapter_request: Prompt Adapter request to use for
1240
- generation, if any.
1213
1214
Returns:
1215
A list of `ClassificationRequestOutput` objects containing the
@@ -1253,7 +1225,6 @@ class LLM:
1225
prompts,
1226
use_tqdm=use_tqdm,
1227
lora_request=lora_request,
1256
- prompt_adapter_request=prompt_adapter_request,
1228
pooling_task="classify",
1229
)
1230
@@ -1267,7 +1238,6 @@ class LLM:
1238
truncate_prompt_tokens: Optional[int] = None,
1239
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
1240
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
1270
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
1241
) -> list[ScoringRequestOutput]:
1242
1243
encoded_output: list[PoolingRequestOutput] = self.encode(
@@ -1275,7 +1245,6 @@ class LLM:
1245
truncate_prompt_tokens=truncate_prompt_tokens,
1246
use_tqdm=use_tqdm,
1247
lora_request=lora_request,
1278
- prompt_adapter_request=prompt_adapter_request,
1248
pooling_task="embed",
1249
)
1250
@@ -1303,7 +1272,6 @@ class LLM:
1272
truncate_prompt_tokens: Optional[int] = None,
1273
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
1274
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
1306
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
1275
) -> list[ScoringRequestOutput]:
1276
1277
if isinstance(tokenizer, MistralTokenizer):
@@ -1361,7 +1329,6 @@ class LLM:
1329
params=pooling_params,
1330
use_tqdm=use_tqdm,
1331
lora_request=lora_request,
1364
- prompt_adapter_request=prompt_adapter_request,
1332
)
1333
1334
outputs = self._run_engine(use_tqdm=use_tqdm)
@@ -1381,7 +1348,6 @@ class LLM:
1348
truncate_prompt_tokens: Optional[int] = None,
1349
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
1350
lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,
1384
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
1351
) -> list[ScoringRequestOutput]:
1352
"""Generate similarity scores for all pairs `<text,text_pair>` or
1353
`<multi-modal data, multi-modal data pair>`.
@@ -1412,8 +1378,6 @@ class LLM:
1378
it is used to create the progress bar.
1379
If `False`, no progress bar is created.
1380
lora_request: LoRA request to use for generation, if any.
1415
- prompt_adapter_request: Prompt Adapter request to use for
1416
- generation, if any.
1381
1382
Returns:
1383
A list of `ScoringRequestOutput` objects containing the
@@ -1504,8 +1468,7 @@ class LLM:
1468
data_2, # type: ignore[arg-type]
1469
truncate_prompt_tokens,
1470
use_tqdm,
1507
- lora_request,
1508
- prompt_adapter_request)
1471
+ lora_request)
1472
else:
1473
return self._embedding_score(
1474
tokenizer,
@@ -1513,8 +1476,7 @@ class LLM:
1476
data_2, # type: ignore[arg-type]
1477
truncate_prompt_tokens,
1478
use_tqdm,
1516
- lora_request,
1517
- prompt_adapter_request)
1479
+ lora_request)
1480
1481
def start_profile(self) -> None:
1482
self.llm_engine.start_profile()
@@ -1625,7 +1587,6 @@ class LLM:
1587
*,
1588
use_tqdm: Union[bool, Callable[..., tqdm]] = True,
1589
lora_request: Optional[Union[Sequence[LoRARequest], LoRARequest]],
1628
- prompt_adapter_request: Optional[PromptAdapterRequest],
1590
tokenization_kwargs: Optional[dict[str, Any]] = None,
1591
guided_options: Optional[GuidedDecodingRequest] = None,
1592
priority: Optional[list[int]] = None,
@@ -1671,7 +1632,6 @@ class LLM:
1632
tokenization_kwargs=tokenization_kwargs,
1633
lora_request=lora_request[i] if isinstance(
1634
lora_request, Sequence) else lora_request,
1674
- prompt_adapter_request=prompt_adapter_request,
1635
priority=priority[i] if priority else 0,
1636
)
1637
@@ -1681,7 +1641,6 @@ class LLM:
1641
params: Union[SamplingParams, PoolingParams],
1642
tokenization_kwargs: Optional[dict[str, Any]] = None,
1643
lora_request: Optional[LoRARequest] = None,
1684
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
1644
priority: int = 0,
1645
) -> None:
1646
request_id = str(next(self.request_counter))
@@ -1691,7 +1650,6 @@ class LLM:
1650
params,
1651
lora_request=lora_request,
1652
tokenization_kwargs=tokenization_kwargs,
1694
- prompt_adapter_request=prompt_adapter_request,
1653
priority=priority,
1654
)
1655
vllm/entrypoints/logger.py
+2
-5
@@ -8,7 +8,6 @@ import torch
8
from vllm.logger import init_logger
9
from vllm.lora.request import LoRARequest
10
from vllm.pooling_params import PoolingParams
11
-from vllm.prompt_adapter.request import PromptAdapterRequest
11
from vllm.sampling_params import BeamSearchParams, SamplingParams
12
13
logger = init_logger(__name__)
@@ -30,7 +29,6 @@ class RequestLogger:
29
params: Optional[Union[SamplingParams, PoolingParams,
30
BeamSearchParams]],
31
lora_request: Optional[LoRARequest],
33
- prompt_adapter_request: Optional[PromptAdapterRequest],
32
) -> None:
33
max_log_len = self.max_log_len
34
if max_log_len is not None:
@@ -44,7 +42,6 @@ class RequestLogger:
42
"Received request %s: prompt: %r, "
43
"params: %s, prompt_token_ids: %s, "
44
"prompt_embeds shape: %s, "
47
- "lora_request: %s, prompt_adapter_request: %s.", request_id,
48
- prompt, params, prompt_token_ids,
45
+ "lora_request: %s.", request_id, prompt, params, prompt_token_ids,
46
prompt_embeds.shape if prompt_embeds is not None else None,
50
- lora_request, prompt_adapter_request)
47
+ lora_request)
vllm/entrypoints/openai/api_server.py
-1
@@ -1620,7 +1620,6 @@ async def init_app_state(
1620
model_config=model_config,
1621
base_model_paths=base_model_paths,
1622
lora_modules=lora_modules,
1623
- prompt_adapters=args.prompt_adapters,
1623
)
1624
await state.openai_serving_models.init_static_loras()
1625
state.openai_serving_responses = OpenAIServingResponses(
vllm/entrypoints/openai/cli_args.py
+1
-35
@@ -20,8 +20,7 @@ from vllm.config import config
20
from vllm.engine.arg_utils import AsyncEngineArgs, optional_type
21
from vllm.entrypoints.chat_utils import (ChatTemplateContentFormatOption,
22
validate_chat_template)
23
-from vllm.entrypoints.openai.serving_models import (LoRAModulePath,
24
- PromptAdapterPath)
23
+from vllm.entrypoints.openai.serving_models import LoRAModulePath
24
from vllm.entrypoints.openai.tool_parsers import ToolParserManager
25
from vllm.logger import init_logger
26
from vllm.utils import FlexibleArgumentParser
@@ -65,27 +64,6 @@ class LoRAParserAction(argparse.Action):
64
setattr(namespace, self.dest, lora_list)
65
66
68
-class PromptAdapterParserAction(argparse.Action):
69
-
70
- def __call__(
71
- self,
72
- parser: argparse.ArgumentParser,
73
- namespace: argparse.Namespace,
74
- values: Optional[Union[str, Sequence[str]]],
75
- option_string: Optional[str] = None,
76
- ):
77
- if values is None:
78
- values = []
79
- if isinstance(values, str):
80
- raise TypeError("Expected values to be a list")
81
-
82
- adapter_list: list[PromptAdapterPath] = []
83
- for item in values:
84
- name, path = item.split('=')
85
- adapter_list.append(PromptAdapterPath(name, path))
86
- setattr(namespace, self.dest, adapter_list)
87
-
88
-
67
@config
68
@dataclass
69
class FrontendArgs:
@@ -115,9 +93,6 @@ class FrontendArgs:
93
or JSON list format. Example (old format): `'name=path'` Example (new
94
format): `{\"name\": \"name\", \"path\": \"lora_path\",
95
\"base_model_name\": \"id\"}`"""
118
- prompt_adapters: Optional[list[PromptAdapterPath]] = None
119
- """Prompt adapter configurations in the format name=path. Multiple adapters
120
- can be specified."""
96
chat_template: Optional[str] = None
97
"""The file path to the chat template, or the template in single-line form
98
for the specified model."""
@@ -207,12 +182,6 @@ schema. Example: `[{"type": "text", "text": "Hello world!"}]`"""
182
frontend_kwargs["lora_modules"]["type"] = optional_type(str)
183
frontend_kwargs["lora_modules"]["action"] = LoRAParserAction
184
210
- # Special case: Prompt adapters need custom parser action and
211
- # optional_type(str)
212
- frontend_kwargs["prompt_adapters"]["type"] = optional_type(str)
213
- frontend_kwargs["prompt_adapters"][
214
- "action"] = PromptAdapterParserAction
215
-
185
# Special case: Middleware needs append action
186
frontend_kwargs["middleware"]["action"] = "append"
187
frontend_kwargs["middleware"]["type"] = str
@@ -288,9 +257,6 @@ def validate_parsed_serve_args(args: argparse.Namespace):
257
if args.enable_auto_tool_choice and not args.tool_call_parser:
258
raise TypeError("Error: --enable-auto-tool-choice requires "
259
"--tool-call-parser")
291
- if args.enable_prompt_embeds and args.enable_prompt_adapter:
292
- raise ValueError(
293
- "Cannot use prompt embeds and prompt adapter at the same time.")
260
261
262
def log_non_default_args(args: argparse.Namespace):
vllm/entrypoints/openai/run_batch.py
-1
@@ -337,7 +337,6 @@ async def main(args):
337
model_config=model_config,
338
base_model_paths=base_model_paths,
339
lora_modules=None,
340
- prompt_adapters=None,
340
)
341
openai_serving_chat = OpenAIServingChat(
342
engine,
vllm/entrypoints/openai/serving_chat.py
+3
-8
@@ -147,11 +147,8 @@ class OpenAIServingChat(OpenAIServing):
147
raise self.engine_client.dead_error
148
149
try:
150
- (
151
- lora_request,
152
- prompt_adapter_request,
153
- ) = self._maybe_get_adapters(request,
154
- supports_default_mm_loras=True)
150
+ lora_request = self._maybe_get_adapters(
151
+ request, supports_default_mm_loras=True)
152
153
model_name = self._get_model_name(request.model, lora_request)
154
@@ -239,8 +236,7 @@ class OpenAIServingChat(OpenAIServing):
236
self._log_inputs(request_id,
237
request_prompts[i],
238
params=sampling_params,
242
- lora_request=lora_request,
243
- prompt_adapter_request=prompt_adapter_request)
239
+ lora_request=lora_request)
240
241
trace_headers = (None if raw_request is None else await
242
self._get_trace_headers(raw_request.headers))
@@ -259,7 +255,6 @@ class OpenAIServingChat(OpenAIServing):
255
request_id,
256
lora_request=lora_request,
257
trace_headers=trace_headers,
262
- prompt_adapter_request=prompt_adapter_request,
258
priority=request.priority,
259
)
260
vllm/entrypoints/openai/serving_classification.py
+1
-9
@@ -49,19 +49,11 @@ class ClassificationMixin(OpenAIServing):
49
return None
50
51
try:
52
- (
53
- ctx.lora_request,
54
- ctx.prompt_adapter_request,
55
- ) = self._maybe_get_adapters(ctx.request)
52
+ ctx.lora_request = self._maybe_get_adapters(ctx.request)
53
54
ctx.tokenizer = await self.engine_client.get_tokenizer(
55
ctx.lora_request)
56
60
- if ctx.prompt_adapter_request is not None:
61
- raise NotImplementedError(
62
- "Prompt adapter is not supported for classification models"
63
- )
64
-
57
(
58
ctx.request_prompts,
59
ctx.engine_prompts,
vllm/entrypoints/openai/serving_completion.py
+1
-6
@@ -121,10 +121,7 @@ class OpenAIServingCompletion(OpenAIServing):
121
raw_request.state.request_metadata = request_metadata
122
123
try:
124
- (
125
- lora_request,
126
- prompt_adapter_request,
127
- ) = self._maybe_get_adapters(request)
124
+ lora_request = self._maybe_get_adapters(request)
125
126
tokenizer = await self.engine_client.get_tokenizer(lora_request)
127
@@ -197,7 +194,6 @@ class OpenAIServingCompletion(OpenAIServing):
194
request_prompts[i],
195
params=sampling_params,
196
lora_request=lora_request,
200
- prompt_adapter_request=prompt_adapter_request,
197
)
198
199
trace_headers = (None if raw_request is None else await
@@ -221,7 +217,6 @@ class OpenAIServingCompletion(OpenAIServing):
217
sampling_params,
218
request_id_item,
219
lora_request=lora_request,
224
- prompt_adapter_request=prompt_adapter_request,
220
trace_headers=trace_headers,
221
priority=request.priority,
222
)
vllm/entrypoints/openai/serving_embedding.py
+1
-8
@@ -53,18 +53,11 @@ class EmbeddingMixin(OpenAIServing):
53
) -> Optional[ErrorResponse]:
54
ctx = cast(EmbeddingServeContext, ctx)
55
try:
56
- (
57
- ctx.lora_request,
58
- ctx.prompt_adapter_request,
59
- ) = self._maybe_get_adapters(ctx.request)
56
+ ctx.lora_request = self._maybe_get_adapters(ctx.request)
57
58
tokenizer = await self.engine_client.get_tokenizer(ctx.lora_request
59
)
60
64
- if ctx.prompt_adapter_request is not None:
65
- raise NotImplementedError("Prompt adapter is not supported "
66
- "for embedding models")
67
-
61
if isinstance(ctx.request, EmbeddingChatRequest):
62
(
63
_,
vllm/entrypoints/openai/serving_engine.py
+8
-23
@@ -68,7 +68,6 @@ from vllm.multimodal import ( # noqa: F401 - Required to resolve Pydantic error
68
MultiModalDataDict)
69
from vllm.outputs import PoolingRequestOutput, RequestOutput
70
from vllm.pooling_params import PoolingParams
71
-from vllm.prompt_adapter.request import PromptAdapterRequest
71
from vllm.sampling_params import BeamSearchParams, SamplingParams
72
from vllm.sequence import Logprob, PromptLogprobs
73
from vllm.tracing import (contains_trace_headers, extract_trace_headers,
@@ -161,7 +160,6 @@ class ServeContext(RequestProcessingMixin, ResponseGenerationMixin, BaseModel,
160
request_id: str
161
created_time: int = Field(default_factory=lambda: int(time.time()))
162
lora_request: Optional[LoRARequest] = None
164
- prompt_adapter_request: Optional[PromptAdapterRequest] = None
163
164
# Shared across most requests
165
tokenizer: Optional[AnyTokenizer] = None
@@ -343,12 +341,10 @@ class OpenAIServing:
341
return self.create_error_response(
342
"Request prompts not available")
343
346
- self._log_inputs(
347
- request_id_item,
348
- ctx.request_prompts[i],
349
- params=pooling_params,
350
- lora_request=ctx.lora_request,
351
- prompt_adapter_request=ctx.prompt_adapter_request)
344
+ self._log_inputs(request_id_item,
345
+ ctx.request_prompts[i],
346
+ params=pooling_params,
347
+ lora_request=ctx.lora_request)
348
349
# Mypy has an existing bug related to inferring the variance of
350
# TypedDicts with `builtins.enumerate`:
@@ -450,11 +446,6 @@ class OpenAIServing:
446
if isinstance(load_result, ErrorResponse) and \
447
load_result.code == HTTPStatus.BAD_REQUEST.value:
448
error_response = load_result
453
- if request.model in [
454
- prompt_adapter.prompt_adapter_name
455
- for prompt_adapter in self.models.prompt_adapter_requests
456
- ]:
457
- return None
449
450
return error_response or self.create_error_response(
451
message=f"The model `{request.model}` does not exist.",
@@ -489,25 +480,21 @@ class OpenAIServing:
480
self,
481
request: AnyRequest,
482
supports_default_mm_loras: bool = False,
492
- ) -> Union[tuple[None, None], tuple[LoRARequest, None], tuple[
493
- None, PromptAdapterRequest]]:
483
+ ) -> Optional[LoRARequest]:
484
485
if request.model in self.models.lora_requests:
496
- return self.models.lora_requests[request.model], None
486
+ return self.models.lora_requests[request.model]
487
488
# Currently only support default modality specific loras
489
# if we have exactly one lora matched on the request.
490
if supports_default_mm_loras:
491
default_mm_lora = self._get_active_default_mm_loras(request)
492
if default_mm_lora is not None:
503
- return default_mm_lora, None
493
+ return default_mm_lora
494
495
if self._is_model_supported(request.model):
506
- return None, None
496
+ return None
497
508
- for prompt_adapter in self.models.prompt_adapter_requests:
509
- if request.model == prompt_adapter.prompt_adapter_name:
510
- return None, prompt_adapter
498
# if _check_model has been called earlier, this will be unreachable
499
raise ValueError(f"The model `{request.model}` does not exist.")
500
@@ -987,7 +974,6 @@ class OpenAIServing:
974
params: Optional[Union[SamplingParams, PoolingParams,
975
BeamSearchParams]],
976
lora_request: Optional[LoRARequest],
990
- prompt_adapter_request: Optional[PromptAdapterRequest],
977
) -> None:
978
if self.request_logger is None:
979
return
@@ -1009,7 +995,6 @@ class OpenAIServing:
995
prompt_embeds,
996
params=params,
997
lora_request=lora_request,
1012
- prompt_adapter_request=prompt_adapter_request,
998
)
999
1000
async def _get_trace_headers(
vllm/entrypoints/openai/serving_models.py
-31
@@ -1,8 +1,6 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
4
-import json
5
-import pathlib
4
from asyncio import Lock
5
from collections import defaultdict
6
from dataclasses import dataclass
@@ -19,7 +17,6 @@ from vllm.entrypoints.openai.protocol import (ErrorResponse,
17
from vllm.logger import init_logger
18
from vllm.lora.request import LoRARequest
19
from vllm.lora.resolver import LoRAResolver, LoRAResolverRegistry
22
-from vllm.prompt_adapter.request import PromptAdapterRequest
20
from vllm.utils import AtomicCounter
21
22
logger = init_logger(__name__)
@@ -31,12 +28,6 @@ class BaseModelPath:
28
model_path: str
29
30
34
-@dataclass
35
-class PromptAdapterPath:
36
- name: str
37
- local_path: str
38
-
39
-
31
@dataclass
32
class LoRAModulePath:
33
name: str
@@ -60,7 +51,6 @@ class OpenAIServingModels:
51
base_model_paths: list[BaseModelPath],
52
*,
53
lora_modules: Optional[list[LoRAModulePath]] = None,
63
- prompt_adapters: Optional[list[PromptAdapterPath]] = None,
54
):
55
super().__init__()
56
@@ -81,20 +71,6 @@ class OpenAIServingModels:
71
LoRAResolverRegistry.get_resolver(lora_resolver_name))
72
self.lora_resolver_lock: dict[str, Lock] = defaultdict(Lock)
73
84
- self.prompt_adapter_requests = []
85
- if prompt_adapters is not None:
86
- for i, prompt_adapter in enumerate(prompt_adapters, start=1):
87
- with pathlib.Path(prompt_adapter.local_path,
88
- "adapter_config.json").open() as f:
89
- adapter_config = json.load(f)
90
- num_virtual_tokens = adapter_config["num_virtual_tokens"]
91
- self.prompt_adapter_requests.append(
92
- PromptAdapterRequest(
93
- prompt_adapter_name=prompt_adapter.name,
94
- prompt_adapter_id=i,
95
- prompt_adapter_local_path=prompt_adapter.local_path,
96
- prompt_adapter_num_virtual_tokens=num_virtual_tokens))
97
-
74
async def init_static_loras(self):
75
"""Loads all static LoRA modules.
76
Raises if any fail to load"""
@@ -141,14 +117,7 @@ class OpenAIServingModels:
117
permission=[ModelPermission()])
118
for lora in self.lora_requests.values()
119
]
144
- prompt_adapter_cards = [
145
- ModelCard(id=prompt_adapter.prompt_adapter_name,
146
- root=self.base_model_paths[0].name,
147
- permission=[ModelPermission()])
148
- for prompt_adapter in self.prompt_adapter_requests
149
- ]
120
model_cards.extend(lora_cards)
151
- model_cards.extend(prompt_adapter_cards)
121
return ModelList(data=model_cards)
122
123
async def load_lora_adapter(
vllm/entrypoints/openai/serving_pooling.py
+2
-10
@@ -94,17 +94,10 @@ class OpenAIServingPooling(OpenAIServing):
94
try:
95
truncate_prompt_tokens = _validate_truncation_size(
96
self.max_model_len, truncate_prompt_tokens)
97
- (
98
- lora_request,
99
- prompt_adapter_request,
100
- ) = self._maybe_get_adapters(request)
97
+ lora_request = self._maybe_get_adapters(request)
98
99
tokenizer = await self.engine_client.get_tokenizer(lora_request)
100
104
- if prompt_adapter_request is not None:
105
- raise NotImplementedError("Prompt adapter is not supported "
106
- "for pooling models")
107
-
101
if isinstance(request, PoolingChatRequest):
102
(
103
_,
@@ -153,8 +146,7 @@ class OpenAIServingPooling(OpenAIServing):
146
self._log_inputs(request_id_item,
147
request_prompts[i],
148
params=pooling_params,
156
- lora_request=lora_request,
157
- prompt_adapter_request=prompt_adapter_request)
149
+ lora_request=lora_request)
150
151
trace_headers = (None if raw_request is None else await
152
self._get_trace_headers(raw_request.headers))
vllm/entrypoints/openai/serving_responses.py
+2
-7
@@ -133,10 +133,7 @@ class OpenAIServingResponses(OpenAIServing):
133
messages = self._construct_input_messages(request, prev_response)
134
135
try:
136
- (
137
- lora_request,
138
- prompt_adapter_request,
139
- ) = self._maybe_get_adapters(request)
136
+ lora_request = self._maybe_get_adapters(request)
137
model_name = self._get_model_name(request.model, lora_request)
138
tokenizer = await self.engine_client.get_tokenizer(lora_request)
139
@@ -169,8 +166,7 @@ class OpenAIServingResponses(OpenAIServing):
166
self._log_inputs(request.request_id,
167
request_prompts[i],
168
params=sampling_params,
172
- lora_request=lora_request,
173
- prompt_adapter_request=prompt_adapter_request)
169
+ lora_request=lora_request)
170
171
trace_headers = (None if raw_request is None else await
172
self._get_trace_headers(raw_request.headers))
@@ -181,7 +177,6 @@ class OpenAIServingResponses(OpenAIServing):
177
request.request_id,
178
lora_request=lora_request,
179
trace_headers=trace_headers,
184
- prompt_adapter_request=prompt_adapter_request,
180
priority=request.priority,
181
)
182
generators.append(generator)
vllm/entrypoints/openai/serving_score.py
+3
-19
@@ -27,7 +27,6 @@ from vllm.inputs.data import TokensPrompt
27
from vllm.logger import init_logger
28
from vllm.lora.request import LoRARequest
29
from vllm.outputs import PoolingRequestOutput, ScoringRequestOutput
30
-from vllm.prompt_adapter.request import PromptAdapterRequest
30
from vllm.transformers_utils.tokenizer import AnyTokenizer, MistralTokenizer
31
from vllm.utils import make_async, merge_async_iterators
32
@@ -58,8 +57,6 @@ class ServingScores(OpenAIServing):
57
request_id: str,
58
tokenization_kwargs: Optional[dict[str, Any]] = None,
59
lora_request: Optional[Union[LoRARequest, None]] = None,
61
- prompt_adapter_request: Optional[Union[PromptAdapterRequest,
62
- None]] = None,
60
trace_headers: Optional[Mapping[str, str]] = None,
61
) -> Union[list[PoolingRequestOutput], ErrorResponse]:
62
input_texts = texts_1 + texts_2
@@ -100,8 +97,7 @@ class ServingScores(OpenAIServing):
97
self._log_inputs(request_id_item,
98
input_texts[i],
99
params=pooling_params,
103
- lora_request=lora_request,
104
- prompt_adapter_request=prompt_adapter_request)
100
+ lora_request=lora_request)
101
102
generators.append(
103
self.engine_client.encode(
@@ -176,8 +172,6 @@ class ServingScores(OpenAIServing):
172
request_id: str,
173
tokenization_kwargs: Optional[dict[str, Any]] = None,
174
lora_request: Optional[Union[LoRARequest, None]] = None,
179
- prompt_adapter_request: Optional[Union[PromptAdapterRequest,
180
- None]] = None,
175
trace_headers: Optional[Mapping[str, str]] = None,
176
) -> Union[list[PoolingRequestOutput], ErrorResponse]:
177
request_prompts: list[str] = []
@@ -261,8 +255,7 @@ class ServingScores(OpenAIServing):
255
self._log_inputs(request_id_item,
256
request_prompts[i],
257
params=pooling_params,
264
- lora_request=lora_request,
265
- prompt_adapter_request=prompt_adapter_request)
258
+ lora_request=lora_request)
259
260
generator = self.engine_client.encode(
261
engine_prompt,
@@ -295,14 +288,7 @@ class ServingScores(OpenAIServing):
288
raw_request: Optional[Request] = None,
289
truncate_prompt_tokens: Optional[int] = None,
290
) -> Union[list[PoolingRequestOutput], ErrorResponse]:
298
- (
299
- lora_request,
300
- prompt_adapter_request,
301
- ) = self._maybe_get_adapters(request)
302
-
303
- if prompt_adapter_request is not None:
304
- raise NotImplementedError("Prompt adapter is not supported "
305
- "for scoring models")
291
+ lora_request = self._maybe_get_adapters(request)
292
293
tokenizer = await self.engine_client.get_tokenizer(lora_request)
294
@@ -340,7 +326,6 @@ class ServingScores(OpenAIServing):
326
request_id=request_id,
327
tokenization_kwargs=tokenization_kwargs,
328
lora_request=lora_request,
343
- prompt_adapter_request=prompt_adapter_request,
329
trace_headers=trace_headers)
330
331
else:
@@ -352,7 +337,6 @@ class ServingScores(OpenAIServing):
337
request_id=request_id,
338
tokenization_kwargs=tokenization_kwargs,
339
lora_request=lora_request,
355
- prompt_adapter_request=prompt_adapter_request,
340
trace_headers=trace_headers)
341
342
async def create_score(
vllm/entrypoints/openai/serving_tokenization.py
+4
-17
@@ -60,10 +60,7 @@ class OpenAIServingTokenization(OpenAIServing):
60
request_id = f"tokn-{self._base_request_id(raw_request)}"
61
62
try:
63
- (
64
- lora_request,
65
- prompt_adapter_request,
66
- ) = self._maybe_get_adapters(request)
63
+ lora_request = self._maybe_get_adapters(request)
64
65
tokenizer = await self.engine_client.get_tokenizer(lora_request)
66
@@ -104,11 +101,8 @@ class OpenAIServingTokenization(OpenAIServing):
101
self._log_inputs(request_id,
102
request_prompts[i],
103
params=None,
107
- lora_request=lora_request,
108
- prompt_adapter_request=prompt_adapter_request)
104
+ lora_request=lora_request)
105
110
- # Silently ignore prompt adapter since it does not affect
111
- # tokenization (Unlike in Embeddings API where an error is raised)
106
if isinstance(engine_prompt,
107
dict) and "prompt_token_ids" in engine_prompt:
108
input_ids.extend(engine_prompt["prompt_token_ids"])
@@ -133,21 +127,14 @@ class OpenAIServingTokenization(OpenAIServing):
127
128
request_id = f"tokn-{self._base_request_id(raw_request)}"
129
136
- (
137
- lora_request,
138
- prompt_adapter_request,
139
- ) = self._maybe_get_adapters(request)
130
+ lora_request = self._maybe_get_adapters(request)
131
132
tokenizer = await self.engine_client.get_tokenizer(lora_request)
133
134
self._log_inputs(request_id,
135
request.tokens,
136
params=None,
146
- lora_request=lora_request,
147
- prompt_adapter_request=prompt_adapter_request)
148
-
149
- # Silently ignore prompt adapter since it does not affect tokenization
150
- # (Unlike in Embeddings API where an error is raised)
137
+ lora_request=lora_request)
138
139
prompt_input = await self._tokenize_prompt_input_async(
140
request,
vllm/entrypoints/openai/speech_to_text.py
+2
-10
@@ -150,19 +150,12 @@ class OpenAISpeechToText(OpenAIServing):
150
raw_request.state.request_metadata = request_metadata
151
152
try:
153
- (
154
- lora_request,
155
- prompt_adapter_request,
156
- ) = self._maybe_get_adapters(request)
153
+ lora_request = self._maybe_get_adapters(request)
154
155
if lora_request:
156
return self.create_error_response(
157
"Currently do not support LoRA for "
158
f"{self.task_type.title()}.")
162
- if prompt_adapter_request:
163
- return self.create_error_response(
164
- f"Currently do not support PromptAdapter for "
165
- f"{self.task_type.title()}.")
159
160
prompts, duration_s = await self._preprocess_speech_to_text(
161
request=request,
@@ -188,8 +181,7 @@ class OpenAISpeechToText(OpenAIServing):
181
# It will not display special tokens like <|startoftranscript|>
182
request.prompt,
183
params=sampling_params,
191
- lora_request=None,
192
- prompt_adapter_request=None)
184
+ lora_request=None)
185
186
list_result_generator = [
187
self.engine_client.generate(
vllm/executor/executor_base.py
-31
@@ -17,7 +17,6 @@ from vllm.logger import init_logger
17
from vllm.lora.request import LoRARequest
18
from vllm.model_executor.layers.sampler import SamplerOutput
19
from vllm.pooling_params import PoolingTask
20
-from vllm.prompt_adapter.request import PromptAdapterRequest
20
from vllm.sequence import ExecuteModelRequest, PoolerOutput
21
from vllm.utils import make_async
22
from vllm.worker.worker_base import WorkerBase
@@ -50,7 +49,6 @@ class ExecutorBase(ABC):
49
self.scheduler_config = vllm_config.scheduler_config
50
self.device_config = vllm_config.device_config
51
self.speculative_config = vllm_config.speculative_config
53
- self.prompt_adapter_config = vllm_config.prompt_adapter_config
52
self.observability_config = vllm_config.observability_config
53
self._init_executor()
54
self.is_sleeping = False
@@ -171,35 +169,6 @@ class ExecutorBase(ABC):
169
assert s == sets[0], "All workers should have the same LORAs."
170
return sets[0]
171
174
- def add_prompt_adapter(
175
- self, prompt_adapter_request: PromptAdapterRequest) -> bool:
176
- assert prompt_adapter_request.prompt_adapter_id > 0, \
177
- "prompt_adapter_id must be greater than 0."
178
- return all(
179
- self.collective_rpc("add_prompt_adapter",
180
- args=(prompt_adapter_request, )))
181
-
182
- def remove_prompt_adapter(self, prompt_adapter_id: int) -> bool:
183
- assert prompt_adapter_id > 0, \
184
- "prompt_adapter_id must be greater than 0."
185
- return all(
186
- self.collective_rpc("remove_prompt_adapter",
187
- args=(prompt_adapter_id, )))
188
-
189
- def pin_prompt_adapter(self, prompt_adapter_id: int) -> bool:
190
- assert prompt_adapter_id > 0, \
191
- "prompt_adapter_id must be greater than 0."
192
- return all(
193
- self.collective_rpc("pin_prompt_adapter",
194
- args=(prompt_adapter_id, )))
195
-
196
- def list_prompt_adapters(self) -> Set[int]:
197
- sets = self.collective_rpc("list_prompt_adapters")
198
- for s in sets:
199
- assert (s == sets[0]
200
- ), "All workers should have the same prompt adapters."
201
- return sets[0]
202
-
172
def start_profile(self) -> None:
173
self.collective_rpc("start_profile")
174
vllm/inputs/preprocess.py
+2
-33
@@ -13,7 +13,6 @@ from vllm.lora.request import LoRARequest
13
from vllm.multimodal import MULTIMODAL_REGISTRY, MultiModalRegistry
14
from vllm.multimodal.inputs import (MultiModalDataDict, MultiModalEncDecInputs,
15
MultiModalInputs)
16
-from vllm.prompt_adapter.request import PromptAdapterRequest
16
from vllm.transformers_utils.tokenizer import AnyTokenizer
17
from vllm.transformers_utils.tokenizer_group import TokenizerGroup
18
@@ -168,18 +167,6 @@ class InputPreprocessor:
167
168
return decoder_input_ids
169
171
- def _apply_prompt_adapter(
172
- self,
173
- prompt_token_ids: list[int],
174
- prompt_adapter_request: Optional[PromptAdapterRequest],
175
- ) -> list[int]:
176
- if prompt_adapter_request:
177
- prompt_token_ids = (
178
- [0] * prompt_adapter_request.prompt_adapter_num_virtual_tokens
179
- + prompt_token_ids)
180
-
181
- return prompt_token_ids
182
-
170
def _get_tokenization_kw(
171
self,
172
overrides: Optional[dict[str, Any]] = None,
@@ -786,15 +773,10 @@ class InputPreprocessor:
773
def _build_decoder_only_llm_inputs(
774
self,
775
prompt_inputs: DecoderOnlyInputs,
789
- prompt_adapter_request: Optional[PromptAdapterRequest],
776
) -> DecoderOnlyInputs:
777
if "prompt_token_ids" in prompt_inputs:
778
prompt_inputs = cast(Union[TokenInputs, MultiModalInputs],
779
prompt_inputs) # Needed for mypy
794
- prompt_inputs["prompt_token_ids"] = self._apply_prompt_adapter(
795
- prompt_inputs["prompt_token_ids"],
796
- prompt_adapter_request=prompt_adapter_request,
797
- )
780
781
return prompt_inputs
782
@@ -803,7 +785,6 @@ class InputPreprocessor:
785
prompt: SingletonPrompt,
786
tokenization_kwargs: Optional[dict[str, Any]] = None,
787
lora_request: Optional[LoRARequest] = None,
806
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
788
return_mm_hashes: bool = False,
789
) -> DecoderOnlyInputs:
790
"""
@@ -815,7 +796,6 @@ class InputPreprocessor:
796
797
* prompt: input prompt
798
* lora_request
818
- * prompt_adapter_request
799
* return_mm_hashes
800
801
Returns:
@@ -830,17 +810,13 @@ class InputPreprocessor:
810
return_mm_hashes=return_mm_hashes,
811
)
812
833
- return self._build_decoder_only_llm_inputs(
834
- prompt_comps,
835
- prompt_adapter_request=prompt_adapter_request,
836
- )
813
+ return self._build_decoder_only_llm_inputs(prompt_comps)
814
815
async def _process_decoder_only_prompt_async(
816
self,
817
prompt: SingletonPrompt,
818
tokenization_kwargs: Optional[dict[str, Any]] = None,
819
lora_request: Optional[LoRARequest] = None,
843
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
820
return_mm_hashes: bool = False,
821
) -> DecoderOnlyInputs:
822
"""
@@ -854,17 +830,13 @@ class InputPreprocessor:
830
return_mm_hashes=return_mm_hashes,
831
)
832
857
- return self._build_decoder_only_llm_inputs(
858
- prompt_comps,
859
- prompt_adapter_request=prompt_adapter_request,
860
- )
833
+ return self._build_decoder_only_llm_inputs(prompt_comps)
834
835
def preprocess(
836
self,
837
prompt: PromptType,
838
tokenization_kwargs: Optional[dict[str, Any]] = None,
839
lora_request: Optional[LoRARequest] = None,
867
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
840
return_mm_hashes: bool = False,
841
) -> ProcessorInputs:
842
"""Preprocess the input prompt."""
@@ -886,7 +858,6 @@ class InputPreprocessor:
858
prompt,
859
tokenization_kwargs=tokenization_kwargs,
860
lora_request=lora_request,
889
- prompt_adapter_request=prompt_adapter_request,
861
return_mm_hashes=return_mm_hashes,
862
)
863
@@ -895,7 +866,6 @@ class InputPreprocessor:
866
prompt: PromptType,
867
tokenization_kwargs: Optional[dict[str, Any]] = None,
868
lora_request: Optional[LoRARequest] = None,
898
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
869
return_mm_hashes: bool = False,
870
) -> ProcessorInputs:
871
"""
@@ -919,6 +889,5 @@ class InputPreprocessor:
889
prompt,
890
tokenization_kwargs=tokenization_kwargs,
891
lora_request=lora_request,
922
- prompt_adapter_request=prompt_adapter_request,
892
return_mm_hashes=return_mm_hashes,
893
)
vllm/prompt_adapter/__init__.py
vllm/prompt_adapter/layers.py
deleted
-83
@@ -1,83 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-from dataclasses import dataclass
5
-from typing import Optional
6
-
7
-import torch
8
-from torch import nn
9
-
10
-from vllm.adapter_commons.layers import AdapterMapping
11
-from vllm.config import PromptAdapterConfig
12
-from vllm.model_executor.layers.vocab_parallel_embedding import (
13
- VocabParallelEmbedding)
14
-
15
-
16
-@dataclass
17
-class PromptAdapterMapping(AdapterMapping):
18
- pass
19
-
20
-
21
-class VocabParallelEmbeddingWithPromptAdapter(nn.Module):
22
-
23
- def __init__(self, base_layer: VocabParallelEmbedding) -> None:
24
- super().__init__()
25
- self.base_layer = base_layer
26
- self.emb_layer = self.base_layer
27
- if 'LoRA' in base_layer.__class__.__name__:
28
- self.emb_layer = self.base_layer.base_layer
29
-
30
- def create_prompt_adapter_weights(
31
- self, prompt_adapter_config: PromptAdapterConfig):
32
- self.embeddings_tensors = torch.zeros(
33
- (
34
- prompt_adapter_config.max_prompt_adapters,
35
- prompt_adapter_config.max_prompt_adapter_token,
36
- self.emb_layer.embedding_dim,
37
- ),
38
- dtype=self.emb_layer.weight.dtype,
39
- device=self.emb_layer.weight.device,
40
- )
41
- self.adapter_lengths = torch.zeros(
42
- prompt_adapter_config.max_prompt_adapters,
43
- dtype=torch.long,
44
- device=self.emb_layer.weight.device)
45
-
46
- self.indices_gpu: torch.Tensor
47
- self.embedding_indices_gpu: torch.Tensor
48
-
49
- def reset_prompt_adapter(self, index: int):
50
- self.embeddings_tensors[index] = 0
51
-
52
- def set_prompt_adapter(
53
- self,
54
- index: int,
55
- adapter_model: Optional[torch.Tensor],
56
- ):
57
- self.reset_prompt_adapter(index)
58
- if adapter_model is not None:
59
- length = adapter_model.shape[0]
60
- self.embeddings_tensors[index, :length] = adapter_model
61
- self.adapter_lengths[index] = length
62
-
63
- def set_mapping(
64
- self,
65
- prompt_indices: torch.Tensor,
66
- prompt_embedding_indices: torch.Tensor,
67
- ):
68
- self.indices_gpu = prompt_indices.to(
69
- device=self.emb_layer.weight.device)
70
- self.embedding_indices_gpu = prompt_embedding_indices.to(
71
- device=self.emb_layer.weight.device)
72
-
73
- def forward(self, x: torch.Tensor) -> torch.Tensor:
74
- hidden_states = self.base_layer(x)
75
- if self.embedding_indices_gpu.ndim > 1:
76
- valid_mask = self.indices_gpu != -1
77
- gathered_embeddings = self.embeddings_tensors[
78
- self.embedding_indices_gpu[:, 0],
79
- self.embedding_indices_gpu[:, 1]]
80
-
81
- # Update hidden states
82
- hidden_states[valid_mask] = gathered_embeddings
83
- return hidden_states
\ No newline at end of file
vllm/prompt_adapter/models.py
deleted
-358
@@ -1,358 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-import logging
5
-import math
6
-from typing import Any, Callable, Dict, List, Optional, Type
7
-
8
-import torch
9
-from torch import nn
10
-
11
-from vllm.adapter_commons.models import (AdapterLRUCache, AdapterModel,
12
- AdapterModelManager)
13
-from vllm.adapter_commons.utils import (add_adapter, deactivate_adapter,
14
- get_adapter, list_adapters,
15
- remove_adapter, set_adapter_mapping)
16
-from vllm.config import PromptAdapterConfig
17
-from vllm.prompt_adapter.layers import (
18
- VocabParallelEmbeddingWithPromptAdapter) # yapf: disable
19
-from vllm.prompt_adapter.layers import PromptAdapterMapping
20
-from vllm.prompt_adapter.utils import load_peft_weights
21
-
22
-logger = logging.getLogger(__name__)
23
-
24
-_GLOBAL_PROMPT_ADAPTER_ID = 0
25
-
26
-
27
-def get_prompt_adapter_id():
28
- global _GLOBAL_PROMPT_ADAPTER_ID
29
- _GLOBAL_PROMPT_ADAPTER_ID += 1
30
- return _GLOBAL_PROMPT_ADAPTER_ID
31
-
32
-
33
-def convert_to_embedding_indices(indices):
34
- embedding_indices = []
35
- count = 0
36
-
37
- for value in indices:
38
- if value == -1:
39
- count = 0
40
- else:
41
- embedding_indices.append([value, count])
42
- count += 1
43
-
44
- return torch.tensor(embedding_indices)
45
-
46
-
47
-def convert_mapping(
48
- mapping: PromptAdapterMapping,
49
- prompt_adapter_index_to_id: List[Optional[int]],
50
-) -> torch.Tensor:
51
- """Converts PromptAdapterMapping to index tensors.
52
-
53
- Args:
54
- mapping: PromptAdapterMapping mapping rows in a
55
- batch to PromptAdapter ids.
56
- prompt_adapter_index_to_id: List mapping PromptAdapter
57
- ids to PromptAdapter indices.
58
-
59
- Returns:
60
- pa_indices: Tensor of shape [batch_size] mapping batch rows to
61
- PromptAdapter indices.
62
- """
63
- id_to_index = {
64
- id_: idx
65
- for idx, id_ in enumerate(prompt_adapter_index_to_id)
66
- if id_ is not None
67
- }
68
- pa_indices = ([
69
- id_to_index.get(id_, -1) if id_ > 0 else -1
70
- for id_ in mapping.index_mapping
71
- ])
72
-
73
- pa_embedding_mapping = convert_to_embedding_indices(pa_indices)
74
- pa_indices = torch.tensor(pa_indices)
75
- return pa_indices, pa_embedding_mapping
76
-
77
-
78
-class PromptAdapterModel(AdapterModel):
79
-
80
- def __init__(self,
81
- prompt_adapter_id=None,
82
- num_virtual_tokens=None,
83
- prompt_embedding=None) -> None:
84
- self.id = prompt_adapter_id
85
- self.prompt_embedding = prompt_embedding
86
- self.num_virtual_tokens = num_virtual_tokens
87
-
88
- @classmethod
89
- def from_local_checkpoint(
90
- cls,
91
- adapter_model_path: str,
92
- prompt_adapter_id: int,
93
- num_virtual_tokens: int,
94
- config: PromptAdapterConfig,
95
- device: str = "cuda",
96
- ) -> "PromptAdapterModel":
97
-
98
- if num_virtual_tokens > config.max_prompt_adapter_token:
99
- raise ValueError(
100
- f'num_virtual_tokens ({num_virtual_tokens}) should be <= '
101
- f'max_prompt_adapter_token({config.max_prompt_adapter_token})')
102
-
103
- adapters_weights = load_peft_weights(adapter_model_path, device)
104
- prompt_embedding = adapters_weights["prompt_embeddings"].to(
105
- config.prompt_adapter_dtype)
106
-
107
- return cls(prompt_adapter_id, num_virtual_tokens, prompt_embedding)
108
-
109
-
110
-class PromptAdapterModelManager(AdapterModelManager):
111
- """A manager that manages multiple Prompt Adapter models."""
112
-
113
- def __init__(
114
- self,
115
- model: nn.Module,
116
- max_num_seqs: int,
117
- max_num_batched_tokens: int,
118
- prompt_adapter_config: PromptAdapterConfig,
119
- ):
120
- """Create a PromptAdapterModel and adapter for a given model.
121
-
122
- Args:
123
- model: the model to be adapted.
124
- max_num_seqs: the maximum number of sequences model can run in a
125
- single batch.
126
- max_num_batched_tokens: the maximum number of tokens model can run
127
- in a single batch.
128
- prompt_adapter_config: the PromptAdapter config,
129
- """
130
- self.model: nn.Module = model
131
- # Dict instead of a Set for compatibility with LRUCache.
132
- self.prompt_adapter_index_to_id: List[
133
- Optional[int]] = [None] * self.prompt_adapter_slots
134
- self.max_num_seqs = max_num_seqs
135
- self.max_num_batched_tokens = math.ceil(max_num_batched_tokens / 8) * 8
136
- self.prompt_adapter_config = prompt_adapter_config
137
- self.model.prompt_adapter_manager = self
138
- self.adapter_type = 'PromptAdapter'
139
-
140
- self.base_indices = torch.tensor([-1])
141
- self.base_embedding_indices = torch.tensor([])
142
-
143
- self.modules: Dict[str, nn.Module] = {}
144
- self._create_prompt_adapter_modules()
145
- self._last_mapping: Optional[PromptAdapterMapping] = None
146
-
147
- @property
148
- def prompt_adapter_slots(self) -> int:
149
- return self.prompt_adapter_config.max_prompt_adapters
150
-
151
- @property
152
- def adapter_slots(self) -> int:
153
- return self.prompt_adapter_slots
154
-
155
- @property
156
- def capacity(self) -> int:
157
- return self.prompt_adapter_config.max_cpu_prompt_adapters
158
-
159
- def activate_adapter(
160
- self,
161
- prompt_adapter_id: int,
162
- ) -> bool:
163
- """Move PromptAdapter into a GPU buffer
164
- to be used in the forward pass."""
165
- if prompt_adapter_id in self._active_adapters:
166
- return False
167
- first_free_slot = next(
168
- ((i, prompt_adapter_id) for i, prompt_adapter_id in enumerate(
169
- self.prompt_adapter_index_to_id) if prompt_adapter_id is None),
170
- None)
171
- if first_free_slot is None:
172
- raise ValueError("No free prompt_adapter slots")
173
- index, _ = first_free_slot
174
- self._active_adapters[prompt_adapter_id] = None
175
- prompt_adapter_model = (self._registered_adapters[prompt_adapter_id])
176
- logger.debug("Activating prompt_adapter. int id: %d, slot index: %d",
177
- prompt_adapter_model.id, index)
178
- self.prompt_adapter_index_to_id[index] = prompt_adapter_model.id
179
- for _, v in self.modules.items():
180
- v.set_prompt_adapter(index, prompt_adapter_model.prompt_embedding)
181
- return True
182
-
183
- def _deactivate_adapter(self, prompt_adapter_id: int):
184
- try:
185
- index = self.prompt_adapter_index_to_id.index(prompt_adapter_id)
186
- self.prompt_adapter_index_to_id[index] = None
187
- for _, v in self.modules.items():
188
- v.reset_prompt_adapter(index)
189
- except ValueError:
190
- pass
191
-
192
- def _add_adapter(self, prompt_adapter: PromptAdapterModel):
193
- self._registered_adapters[prompt_adapter.id] = prompt_adapter
194
-
195
- def _set_adapter_mapping(self, mapping: PromptAdapterMapping) -> None:
196
- base_indices, base_embedding_indices = convert_mapping(
197
- mapping, self.prompt_adapter_index_to_id)
198
- for k, v in self.modules.items():
199
- v.set_mapping(base_indices, base_embedding_indices)
200
-
201
- def _create_prompt_adapter_modules(self):
202
- for module_name, module in self.model.named_modules(
203
- remove_duplicate=False):
204
- if "VocabParallel" in module.__class__.__name__:
205
- new_module = VocabParallelEmbeddingWithPromptAdapter(module)
206
- new_module.create_prompt_adapter_weights(
207
- self.prompt_adapter_config)
208
- replaced_module = self.replace_submodule(
209
- self.model, module_name, new_module)
210
- self.register_module(module.__class__.__name__,
211
- replaced_module)
212
- replaced_module.set_mapping(self.base_indices,
213
- self.base_embedding_indices)
214
- break
215
-
216
- def replace_submodule(self, model: nn.Module, module_name: str,
217
- new_module: nn.Module) -> nn.Module:
218
- """Replace a submodule in a model with a new module."""
219
- parent = model.get_submodule(".".join(module_name.split(".")[:-1]))
220
- target_name = module_name.split(".")[-1]
221
- setattr(parent, target_name, new_module)
222
- return new_module
223
-
224
- def register_module(self, module_name: str, module: nn.Module):
225
- self.modules[module_name] = module
226
-
227
- def pin_adapter(self, prompt_adapter_id: int) -> bool:
228
- """Pin a PromptAdapterModel in the manager cache."""
229
- raise NotImplementedError(
230
- "Pinning is not supported in PromptAdapterModelManager. "
231
- "Use LRUCachePromptAdapterModelManager for pinning"
232
- ) # type: ignore
233
-
234
- def remove_all_adapters(self):
235
- """Remove all PromptAdapterModel from the manager."""
236
- self._registered_adapters.clear()
237
- self.prompt_adapter_index_to_id = [None] * self.prompt_adapter_slots
238
- self._active_adapters.clear()
239
-
240
- def deactivate_adapter(self, adapter_id: int) -> bool:
241
- return deactivate_adapter(adapter_id, self._active_adapters,
242
- self._deactivate_adapter)
243
-
244
- def add_adapter(self, adapter: PromptAdapterModel) -> bool:
245
- return add_adapter(adapter, self._registered_adapters, self.capacity,
246
- self._add_adapter)
247
-
248
- def set_adapter_mapping(self, mapping: PromptAdapterMapping) -> None:
249
- self._last_mapping = set_adapter_mapping(mapping, self._last_mapping,
250
- self._set_adapter_mapping)
251
-
252
- def remove_adapter(self, adapter_id: int) -> bool:
253
- return remove_adapter(adapter_id, self._registered_adapters,
254
- self.deactivate_adapter)
255
-
256
- def list_adapters(self) -> Dict[int, Any]:
257
- return list_adapters(self._registered_adapters)
258
-
259
- def get_adapter(self, adapter_id: int) -> Optional[Any]:
260
- return get_adapter(adapter_id, self._registered_adapters)
261
-
262
-
263
-class PromptAdapterLRUCache(AdapterLRUCache[PromptAdapterModel]):
264
-
265
- def __init__(self, capacity: int,
266
- deactivate_prompt_adapter_fn: Callable[[int], bool]):
267
- super().__init__(capacity, deactivate_prompt_adapter_fn)
268
-
269
-
270
-class LRUCachePromptAdapterModelManager(PromptAdapterModelManager):
271
- """A model manager that manages multiple prompt_adapters with LRU cache."""
272
-
273
- def __init__(
274
- self,
275
- model: nn.Module,
276
- max_num_seqs: int,
277
- max_num_batched_tokens: int,
278
- prompt_adapter_config: PromptAdapterConfig,
279
- ):
280
- self.prompt_adapter_config = prompt_adapter_config
281
- super().__init__(model, max_num_seqs, max_num_batched_tokens,
282
- prompt_adapter_config)
283
- self._registered_adapters = PromptAdapterLRUCache(
284
- self.capacity, self.deactivate_adapter)
285
- self._active_adapters = PromptAdapterLRUCache(
286
- self.prompt_adapter_slots, self._deactivate_adapter)
287
-
288
- def list_adapters(self) -> Dict[int, PromptAdapterModel]:
289
- """List all registered PromptAdapterModel."""
290
- return dict(self._registered_adapters.cache)
291
-
292
- def add_adapter(self, prompt_adapter: PromptAdapterModel) -> bool:
293
- """Add a PromptAdapterModel to the manager."""
294
- if prompt_adapter.id not in self._registered_adapters:
295
- self._add_adapter(prompt_adapter)
296
- was_added = True
297
- else:
298
- # We always touch to update the LRU cache order
299
- self._registered_adapters.touch(prompt_adapter.id)
300
- was_added = False
301
- return was_added
302
-
303
- def activate_adapter(
304
- self,
305
- prompt_adapter_id: int,
306
- ) -> bool:
307
- if prompt_adapter_id not in self._active_adapters and len(
308
- self._active_adapters) >= self.prompt_adapter_slots:
309
- self._active_adapters.remove_oldest()
310
- result = super().activate_adapter(prompt_adapter_id)
311
- # We always touch to update the LRU cache order
312
- self._active_adapters.touch(prompt_adapter_id)
313
- return result
314
-
315
- def remove_oldest_adapter(self) -> bool:
316
- if len(self._registered_adapters) > 0:
317
- self._registered_adapters.remove_oldest()
318
- return True
319
- return False
320
-
321
- def pin_adapter(self, prompt_adapter_id: int) -> bool:
322
- """Pin a PromptAdapterModel in the manager cache."""
323
- self._pin_prompt_adapter_in_cpu_cache(prompt_adapter_id)
324
- self._pin_prompt_adapter_in_gpu_cache(prompt_adapter_id)
325
- return True
326
-
327
- def _pin_prompt_adapter_in_cpu_cache(self, prompt_adapter_id: int):
328
- try:
329
- self._registered_adapters.pin(prompt_adapter_id)
330
- except ValueError as err:
331
- raise ValueError(
332
- "Pinning failed. "
333
- f"Prompt Adapter {prompt_adapter_id} is not registered."
334
- ) from err
335
-
336
- def _pin_prompt_adapter_in_gpu_cache(self, prompt_adapter_id: int):
337
- if prompt_adapter_id not in self._active_adapters:
338
- # move adapter to gpu if not already active
339
- self.activate_adapter(prompt_adapter_id)
340
- self._active_adapters.pin(prompt_adapter_id)
341
-
342
-
343
-def create_prompt_adapter_manager(
344
- model: nn.Module,
345
- max_num_seqs: int,
346
- max_num_batched_tokens: int,
347
- prompt_adapter_config: PromptAdapterConfig,
348
- prompt_adapter_manager_cls: Type[
349
- PromptAdapterModelManager] = PromptAdapterModelManager,
350
- **kwargs) -> PromptAdapterModelManager:
351
- """Create a PromptAdapterModel for a given model."""
352
- prompt_adapter_manager = prompt_adapter_manager_cls(
353
- model=model,
354
- max_num_seqs=max_num_seqs,
355
- max_num_batched_tokens=max_num_batched_tokens,
356
- prompt_adapter_config=prompt_adapter_config,
357
- **kwargs)
358
- return prompt_adapter_manager
vllm/prompt_adapter/request.py
deleted
-37
@@ -1,37 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-import msgspec
5
-
6
-from vllm.adapter_commons.request import AdapterRequest
7
-
8
-
9
-class PromptAdapterRequest(
10
- msgspec.Struct,
11
- array_like=True, # type: ignore[call-arg]
12
- omit_defaults=True, # type: ignore[call-arg]
13
- frozen=True): # type: ignore[call-arg]
14
- """
15
- Request for a Prompt adapter.
16
- """
17
- __metaclass__ = AdapterRequest
18
-
19
- prompt_adapter_name: str
20
- prompt_adapter_id: int
21
- prompt_adapter_local_path: str
22
- prompt_adapter_num_virtual_tokens: int
23
-
24
- def __hash__(self):
25
- return super().__hash__()
26
-
27
- @property
28
- def adapter_id(self):
29
- return self.prompt_adapter_id
30
-
31
- @property
32
- def name(self):
33
- return self.prompt_adapter_name
34
-
35
- @property
36
- def local_path(self):
37
- return self.prompt_adapter_local_path
vllm/prompt_adapter/utils.py
deleted
-98
@@ -1,98 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-# code borrowed from: https://github.com/huggingface/peft/blob/v0.12.0/src/peft/utils/save_and_load.py#L420
5
-
6
-import os
7
-from typing import Optional
8
-
9
-import torch
10
-from huggingface_hub import file_exists, hf_hub_download
11
-from huggingface_hub.utils import EntryNotFoundError
12
-from safetensors.torch import load_file as safe_load_file
13
-
14
-from vllm.platforms import current_platform
15
-
16
-WEIGHTS_NAME = "adapter_model.bin"
17
-SAFETENSORS_WEIGHTS_NAME = "adapter_model.safetensors"
18
-
19
-
20
-# Get current device name based on available devices
21
-def infer_device() -> str:
22
- if current_platform.is_cuda_alike():
23
- return "cuda"
24
- return "cpu"
25
-
26
-
27
-def load_peft_weights(model_id: str,
28
- device: Optional[str] = None,
29
- **hf_hub_download_kwargs) -> dict:
30
- r"""
31
- A helper method to load the PEFT weights from the HuggingFace Hub or locally
32
-
33
- Args:
34
- model_id (`str`):
35
- The local path to the adapter weights or the name of the adapter to
36
- load from the HuggingFace Hub.
37
- device (`str`):
38
- The device to load the weights onto.
39
- hf_hub_download_kwargs (`dict`):
40
- Additional arguments to pass to the `hf_hub_download` method when
41
- loading from the HuggingFace Hub.
42
- """
43
- path = (os.path.join(model_id, hf_hub_download_kwargs["subfolder"]) if
44
- hf_hub_download_kwargs.get("subfolder") is not None else model_id)
45
-
46
- if device is None:
47
- device = infer_device()
48
-
49
- if os.path.exists(os.path.join(path, SAFETENSORS_WEIGHTS_NAME)):
50
- filename = os.path.join(path, SAFETENSORS_WEIGHTS_NAME)
51
- use_safetensors = True
52
- elif os.path.exists(os.path.join(path, WEIGHTS_NAME)):
53
- filename = os.path.join(path, WEIGHTS_NAME)
54
- use_safetensors = False
55
- else:
56
- token = hf_hub_download_kwargs.get("token")
57
- if token is None:
58
- token = hf_hub_download_kwargs.get("use_auth_token")
59
-
60
- hub_filename = (os.path.join(hf_hub_download_kwargs["subfolder"],
61
- SAFETENSORS_WEIGHTS_NAME)
62
- if hf_hub_download_kwargs.get("subfolder") is not None
63
- else SAFETENSORS_WEIGHTS_NAME)
64
- has_remote_safetensors_file = file_exists(
65
- repo_id=model_id,
66
- filename=hub_filename,
67
- revision=hf_hub_download_kwargs.get("revision"),
68
- repo_type=hf_hub_download_kwargs.get("repo_type"),
69
- token=token,
70
- )
71
- use_safetensors = has_remote_safetensors_file
72
-
73
- if has_remote_safetensors_file:
74
- # Priority 1: load safetensors weights
75
- filename = hf_hub_download(
76
- model_id,
77
- SAFETENSORS_WEIGHTS_NAME,
78
- **hf_hub_download_kwargs,
79
- )
80
- else:
81
- try:
82
- filename = hf_hub_download(model_id, WEIGHTS_NAME,
83
- **hf_hub_download_kwargs)
84
- except EntryNotFoundError:
85
- raise ValueError( # noqa: B904
86
- f"Can't find weights for {model_id} in {model_id} or \
87
- in the Hugging Face Hub. "
88
- f"Please check that the file {WEIGHTS_NAME} or \
89
- {SAFETENSORS_WEIGHTS_NAME} is present at {model_id}.")
90
-
91
- if use_safetensors:
92
- adapters_weights = safe_load_file(filename, device=device)
93
- else:
94
- adapters_weights = torch.load(filename,
95
- map_location=torch.device(device),
96
- weights_only=True)
97
-
98
- return adapters_weights
vllm/prompt_adapter/worker_manager.py
deleted
-179
@@ -1,179 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-import logging
5
-from typing import Any, Optional, Set, Type
6
-
7
-import torch
8
-
9
-from vllm.adapter_commons.utils import (add_adapter_worker,
10
- apply_adapters_worker,
11
- list_adapters_worker,
12
- set_active_adapters_worker)
13
-from vllm.adapter_commons.worker_manager import AbstractWorkerManager
14
-from vllm.config import PromptAdapterConfig
15
-from vllm.prompt_adapter.models import (LRUCachePromptAdapterModelManager,
16
- PromptAdapterModel,
17
- PromptAdapterModelManager,
18
- create_prompt_adapter_manager)
19
-from vllm.prompt_adapter.request import PromptAdapterRequest
20
-
21
-logger = logging.getLogger(__name__)
22
-
23
-
24
-class WorkerPromptAdapterManager(AbstractWorkerManager):
25
- """WorkerPromptAdapterManager that manages
26
- prompt_adapter models on the worker side.
27
-
28
- Every request, the requested prompt_adapters will be
29
- loaded (unless they are already loaded),
30
- and every other prompt_adapter will be unloaded."""
31
-
32
- _manager_cls: Type[PromptAdapterModelManager] = PromptAdapterModelManager
33
-
34
- def __init__(
35
- self,
36
- max_num_seqs: int,
37
- max_num_batched_tokens: int,
38
- device: torch.device,
39
- prompt_adapter_config: PromptAdapterConfig,
40
- prompt_adapter_model_cls: Type[PromptAdapterModel] = PromptAdapterModel
41
- ):
42
- self._adapter_manager: PromptAdapterModelManager
43
- self.max_num_seqs = max_num_seqs
44
- self.max_num_batched_tokens = max_num_batched_tokens
45
- self._prompt_adapter_model_cls = prompt_adapter_model_cls
46
- self.prompt_adapter_config = prompt_adapter_config
47
- super().__init__(device)
48
-
49
- @property
50
- def is_enabled(self) -> bool:
51
- return True
52
-
53
- def create_prompt_adapter_manager(
54
- self,
55
- model: torch.nn.Module,
56
- ) -> Any:
57
- prompt_adapter_manager = create_prompt_adapter_manager(
58
- model,
59
- max_num_seqs=self.max_num_seqs,
60
- max_num_batched_tokens=self.max_num_batched_tokens,
61
- prompt_adapter_config=self.prompt_adapter_config,
62
- prompt_adapter_manager_cls=self._manager_cls,
63
- )
64
- self._adapter_manager = prompt_adapter_manager
65
- return prompt_adapter_manager.model
66
-
67
- def _load_adapter(
68
- self, prompt_adapter_request: PromptAdapterRequest
69
- ) -> PromptAdapterModel:
70
- try:
71
- prompt_adapter = (
72
- self._prompt_adapter_model_cls.from_local_checkpoint(
73
- prompt_adapter_request.prompt_adapter_local_path,
74
- prompt_adapter_id=prompt_adapter_request.prompt_adapter_id,
75
- num_virtual_tokens=prompt_adapter_request.
76
- prompt_adapter_num_virtual_tokens,
77
- config=self.prompt_adapter_config,
78
- device=str(self.device),
79
- ))
80
- except Exception as e:
81
- raise RuntimeError(
82
- f"Loading prompt_adapter "
83
- f"{prompt_adapter_request.prompt_adapter_local_path}"
84
- f" failed") from e
85
- return prompt_adapter
86
-
87
- def add_dummy_prompt_adapter(
88
- self, prompt_adapter_request: PromptAdapterRequest) -> bool:
89
- return True
90
-
91
- def pin_adapter(self, adapter_id: int) -> bool:
92
- return self._adapter_manager.pin_adapter(adapter_id)
93
-
94
- def set_active_adapters(self, requests: Set[Any],
95
- mapping: Optional[Any]) -> None:
96
- set_active_adapters_worker(requests, mapping, self._apply_adapters,
97
- self._adapter_manager.set_adapter_mapping)
98
-
99
- def add_adapter(self, adapter_request: Any) -> bool:
100
- return add_adapter_worker(adapter_request, self.list_adapters,
101
- self._load_adapter,
102
- self._adapter_manager.add_adapter,
103
- self._adapter_manager.activate_adapter)
104
-
105
- def _apply_adapters(self, adapter_requests: Set[Any]) -> None:
106
- apply_adapters_worker(adapter_requests, self.list_adapters,
107
- self._adapter_manager.adapter_slots,
108
- self.remove_adapter, self.add_adapter)
109
-
110
- def remove_adapter(self, adapter_id: int) -> bool:
111
- return self._adapter_manager.remove_adapter(adapter_id)
112
-
113
- def remove_all_adapters(self):
114
- self._adapter_manager.remove_all_adapters()
115
-
116
- def list_adapters(self) -> Set[int]:
117
- return list_adapters_worker(self._adapter_manager.list_adapters)
118
-
119
-
120
-class LRUCacheWorkerPromptAdapterManager(WorkerPromptAdapterManager):
121
- """WorkerPromptAdapterManager that manages
122
- prompt_adapter models on the worker side.
123
-
124
- Uses an LRU Cache. Every request, the requested
125
- prompt_adapters will be loaded (unless they are already loaded)
126
- and least recently used prompt_adapters will
127
- be unloaded if the cache is above capacity."""
128
-
129
- _prompt_adapter_manager_cls: Type[
130
- LRUCachePromptAdapterModelManager] = LRUCachePromptAdapterModelManager
131
-
132
- def create_prompt_adapter_manager(
133
- self,
134
- model: torch.nn.Module,
135
- ) -> Any:
136
- prompt_adapter_manager = create_prompt_adapter_manager(
137
- model,
138
- max_num_seqs=self.max_num_seqs,
139
- max_num_batched_tokens=self.max_num_batched_tokens,
140
- prompt_adapter_config=self.prompt_adapter_config,
141
- prompt_adapter_manager_cls=self._prompt_adapter_manager_cls)
142
- self._adapter_manager: LRUCachePromptAdapterModelManager = (
143
- prompt_adapter_manager)
144
- return prompt_adapter_manager.model
145
-
146
- def _apply_adapters(
147
- self, prompt_adapter_requests: Set[PromptAdapterRequest]) -> None:
148
- prompt_adapters_map = {
149
- prompt_adapter_request.prompt_adapter_id: prompt_adapter_request
150
- for prompt_adapter_request in prompt_adapter_requests
151
- if prompt_adapter_request
152
- }
153
- if len(prompt_adapters_map
154
- ) > self._adapter_manager.prompt_adapter_slots:
155
- raise RuntimeError(
156
- f"Number of requested prompt_adapters "
157
- f"({len(prompt_adapters_map)}) is greater "
158
- "than the number of GPU prompt_adapter slots "
159
- f"({self._adapter_manager.prompt_adapter_slots}).")
160
- for prompt_adapter in prompt_adapters_map.values():
161
- self.add_adapter(prompt_adapter)
162
-
163
- def add_adapter(self,
164
- prompt_adapter_request: PromptAdapterRequest) -> bool:
165
- if prompt_adapter_request.prompt_adapter_id not in self.list_adapters(
166
- ):
167
- # Remove before we load the new prompt_adapter to save memory
168
- if len(self._adapter_manager) + 1 > self._adapter_manager.capacity:
169
- self._adapter_manager.remove_oldest_adapter()
170
- prompt_adapter = self._load_adapter(prompt_adapter_request)
171
- loaded = self._adapter_manager.add_adapter(prompt_adapter)
172
- else:
173
- # If the prompt_adapter is already loaded, just touch it to
174
- # update its position in the caches
175
- loaded = self._adapter_manager.get_adapter(
176
- prompt_adapter_request.prompt_adapter_id) is not None
177
- self._adapter_manager.activate_adapter(
178
- prompt_adapter_request.prompt_adapter_id)
179
- return loaded
vllm/sequence.py
+2
-37
@@ -19,7 +19,6 @@ from vllm.inputs import SingletonInputs
19
from vllm.lora.request import LoRARequest
20
from vllm.multimodal import MultiModalKwargs, MultiModalPlaceholderDict
21
from vllm.pooling_params import PoolingParams
22
-from vllm.prompt_adapter.request import PromptAdapterRequest
22
from vllm.sampling_params import RequestOutputKind, SamplingParams
23
24
VLLM_TOKEN_ID_ARRAY_TYPE = "l"
@@ -458,7 +457,6 @@ class Sequence:
457
block size used by the block manager and cache engine.
458
eos_token_id: The end-of-sequence (EOS) token id recognized by this LLM.
459
lora_request: LoRA request.
461
- prompt_adapter_request: Prompt Adapter request.
460
"""
461
462
def __init__(
@@ -468,14 +466,12 @@ class Sequence:
466
block_size: int,
467
eos_token_id: Optional[int] = None,
468
lora_request: Optional[LoRARequest] = None,
471
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
469
) -> None:
470
self.seq_id = seq_id
471
self.inputs = inputs
472
self.block_size = block_size
473
self.eos_token_id = eos_token_id
474
self.lora_request = lora_request
478
- self.prompt_adapter_request = prompt_adapter_request
475
476
self.data = SequenceData.from_seqs(
477
self.prompt_token_ids,
@@ -537,11 +533,6 @@ class Sequence:
533
def lora_int_id(self) -> int:
534
return self.lora_request.lora_int_id if self.lora_request else 0
535
540
- @property
541
- def prompt_adapter_id(self) -> int:
542
- return self.prompt_adapter_request.prompt_adapter_id \
543
- if self.prompt_adapter_request else 0
544
-
536
def get_output_text_to_return(self, buffer_length: int,
537
delta: bool) -> str:
538
"""If delta is True, only new text since the last call to
@@ -601,12 +592,12 @@ class Sequence:
592
designed for prefix caching mode. The final sequence hash is determined
593
by applying token_ids from the sequence's blocks.
594
"""
604
- if self.prompt_adapter_id == 0 and self.lora_int_id == 0:
595
+ if self.lora_int_id == 0:
596
return None
597
598
# NOTE: If there are additional factors influencing the block aside from
599
# token_ids, include them as input parameters to the hash.
609
- return hash((self.prompt_adapter_id, self.lora_int_id))
600
+ return hash(self.lora_int_id)
601
602
def num_hashed_tokens_of_block(self, logical_idx: int):
603
return logical_idx * self.block_size + self.block_size
@@ -707,7 +698,6 @@ class SequenceGroup:
698
encoder_seq: Optional, the single encoder sequence. Should be None
699
unless you are working with an encoder/decoder model.
700
trace_headers: OpenTelemetry trace headers.
710
- prompt_adapter_request: Prompt Adapter request.
701
priority: User-defined priority of the request.
702
draft_size: The number of speculative tokens plus one from the target
703
model; equal to max number of tokens a step can generate
@@ -725,7 +715,6 @@ class SequenceGroup:
715
pooled_data: Optional[torch.Tensor] = None,
716
encoder_seq: Optional[Sequence] = None,
717
trace_headers: Optional[Mapping[str, str]] = None,
728
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
718
priority: int = 0,
719
draft_size: int = 1) -> None:
720
self.request_id = request_id
@@ -747,7 +736,6 @@ class SequenceGroup:
736
self.state = SequenceGroupState()
737
self.pooling_params = pooling_params
738
self.pooled_data = pooled_data
750
- self.prompt_adapter_request = prompt_adapter_request
739
self.encoder_seq = encoder_seq
740
self.trace_headers = trace_headers
741
self.priority = priority
@@ -802,16 +790,6 @@ class SequenceGroup:
790
def lora_int_id(self) -> int:
791
return self.lora_request.lora_int_id if self.lora_request else 0
792
805
- @property
806
- def prompt_adapter_id(self) -> int:
807
- return self.prompt_adapter_request.prompt_adapter_id \
808
- if self.prompt_adapter_request else 0
809
-
810
- @property
811
- def prompt_adapter_num_virtual_tokens(self) -> int:
812
- return self.prompt_adapter_request.prompt_adapter_num_virtual_tokens\
813
- if self.prompt_adapter_request else 0
814
-
793
def init_multi_step(self, num_steps: int) -> None:
794
self.state.num_steps = num_steps
795
self.state.current_step = 0
@@ -1011,7 +989,6 @@ class SequenceGroupMetadata(
989
(SequenceGroup.encoder_seq). Should be None
990
unless you are working with an encoder/decoder
991
model.
1014
- prompt_adapter_request: Prompt Adapter request.
992
"""
993
994
request_id: str
@@ -1030,7 +1007,6 @@ class SequenceGroupMetadata(
1007
multi_modal_placeholders: Optional[MultiModalPlaceholderDict] = None
1008
encoder_seq_data: Optional[SequenceData] = None
1009
cross_block_table: Optional[list[int]] = None
1033
- prompt_adapter_request: Optional[PromptAdapterRequest] = None
1010
token_chunk_size: Optional[int] = None
1011
1012
### Stateful fields that are lazily defined. ###
@@ -1052,16 +1028,6 @@ class SequenceGroupMetadata(
1028
def lora_int_id(self) -> int:
1029
return self.lora_request.lora_int_id if self.lora_request else 0
1030
1055
- @property
1056
- def prompt_adapter_id(self) -> int:
1057
- return self.prompt_adapter_request.prompt_adapter_id \
1058
- if self.prompt_adapter_request else 0
1059
-
1060
- @property
1061
- def prompt_adapter_num_virtual_tokens(self) -> int:
1062
- return self.prompt_adapter_request.prompt_adapter_num_virtual_tokens \
1063
- if self.prompt_adapter_request else 0
1064
-
1031
# Multi-Step Chunked-Prefill property
1032
@property
1033
def is_single_step_prompt(self) -> bool:
@@ -1525,7 +1491,6 @@ class ParallelSampleSequenceGroup(SequenceGroupBase):
1491
pooled_data=seq_group.pooled_data,
1492
encoder_seq=seq_group.encoder_seq,
1493
trace_headers=seq_group.trace_headers,
1528
- prompt_adapter_request=seq_group.prompt_adapter_request,
1494
priority=seq_group.priority,
1495
)
1496
vllm/utils/__init__.py
-5
@@ -128,10 +128,6 @@ STR_NOT_IMPL_ENC_DEC_BACKEND = ("XFormers and Flash-Attention are the only "
128
"backends currently supported with encoder/"
129
"decoder models.")
130
131
-STR_NOT_IMPL_ENC_DEC_PROMPT_ADAPTER = ("Prompt adapters are not "
132
- "currently supported with encoder/"
133
- "decoder models.")
134
-
131
# Efficiently import all enc/dec error strings
132
# rather than having to import all of the above
133
STR_NOT_IMPL_ENC_DEC_ERR_STRS = {
@@ -145,7 +141,6 @@ STR_NOT_IMPL_ENC_DEC_ERR_STRS = {
141
"STR_NOT_IMPL_ENC_DEC_MM": STR_NOT_IMPL_ENC_DEC_MM,
142
"STR_NOT_IMPL_ENC_DEC_SPEC_DEC": STR_NOT_IMPL_ENC_DEC_SPEC_DEC,
143
"STR_NOT_IMPL_ENC_DEC_BACKEND": STR_NOT_IMPL_ENC_DEC_BACKEND,
148
- "STR_NOT_IMPL_ENC_DEC_PROMPT_ADAPTER": STR_NOT_IMPL_ENC_DEC_PROMPT_ADAPTER,
144
}
145
146
# Constants related to forcing the attention backend selection
vllm/v1/engine/async_llm.py
+1
-6
@@ -20,7 +20,6 @@ from vllm.lora.request import LoRARequest
20
from vllm.multimodal import MULTIMODAL_REGISTRY, MultiModalRegistry
21
from vllm.outputs import PoolingRequestOutput, RequestOutput
22
from vllm.pooling_params import PoolingParams
23
-from vllm.prompt_adapter.request import PromptAdapterRequest
23
from vllm.sampling_params import SamplingParams
24
from vllm.transformers_utils.config import (
25
maybe_register_config_serialize_by_value)
@@ -221,7 +220,6 @@ class AsyncLLM(EngineClient):
220
lora_request: Optional[LoRARequest] = None,
221
tokenization_kwargs: Optional[dict[str, Any]] = None,
222
trace_headers: Optional[Mapping[str, str]] = None,
224
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
223
priority: int = 0,
224
data_parallel_rank: Optional[int] = None,
225
) -> RequestOutputCollector:
@@ -238,8 +236,7 @@ class AsyncLLM(EngineClient):
236
# Convert Input --> Request.
237
prompt_str, request = self.processor.process_inputs(
238
request_id, prompt, params, arrival_time, lora_request,
241
- tokenization_kwargs, trace_headers, prompt_adapter_request,
242
- priority, data_parallel_rank)
239
+ tokenization_kwargs, trace_headers, priority, data_parallel_rank)
240
241
if is_pooling or params.n == 1:
242
await self._add_request(request, prompt_str, None, 0, queue)
@@ -283,7 +280,6 @@ class AsyncLLM(EngineClient):
280
request_id: str,
281
lora_request: Optional[LoRARequest] = None,
282
trace_headers: Optional[Mapping[str, str]] = None,
286
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
283
priority: int = 0,
284
data_parallel_rank: Optional[int] = None,
285
) -> AsyncGenerator[RequestOutput, None]:
@@ -314,7 +310,6 @@ class AsyncLLM(EngineClient):
310
sampling_params,
311
lora_request=lora_request,
312
trace_headers=trace_headers,
317
- prompt_adapter_request=prompt_adapter_request,
313
priority=priority,
314
data_parallel_rank=data_parallel_rank,
315
)
vllm/v1/engine/llm_engine.py
+1
-4
@@ -17,7 +17,6 @@ from vllm.lora.request import LoRARequest
17
from vllm.multimodal import MULTIMODAL_REGISTRY, MultiModalRegistry
18
from vllm.outputs import PoolingRequestOutput, RequestOutput
19
from vllm.pooling_params import PoolingParams
20
-from vllm.prompt_adapter.request import PromptAdapterRequest
20
from vllm.sampling_params import SamplingParams
21
from vllm.transformers_utils.tokenizer_group import (
22
TokenizerGroup, init_tokenizer_from_configs)
@@ -192,7 +191,6 @@ class LLMEngine:
191
lora_request: Optional[LoRARequest] = None,
192
tokenization_kwargs: Optional[dict[str, Any]] = None,
193
trace_headers: Optional[Mapping[str, str]] = None,
195
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
194
priority: int = 0,
195
) -> None:
196
# Validate the request_id type.
@@ -203,8 +201,7 @@ class LLMEngine:
201
# Process raw inputs into the request.
202
prompt_str, request = self.processor.process_inputs(
203
request_id, prompt, params, arrival_time, lora_request,
206
- tokenization_kwargs, trace_headers, prompt_adapter_request,
207
- priority)
204
+ tokenization_kwargs, trace_headers, priority)
205
206
n = params.n if isinstance(params, SamplingParams) else 1
207
vllm/v1/engine/processor.py
-6
@@ -16,7 +16,6 @@ from vllm.multimodal.inputs import PlaceholderRange
16
from vllm.multimodal.processing import EncDecMultiModalProcessor
17
from vllm.multimodal.utils import merge_and_sort_multimodal_metadata
18
from vllm.pooling_params import PoolingParams
19
-from vllm.prompt_adapter.request import PromptAdapterRequest
19
from vllm.sampling_params import SamplingParams
20
from vllm.transformers_utils.tokenizer_group import TokenizerGroup
21
from vllm.v1.engine import EngineCoreRequest
@@ -226,7 +225,6 @@ class Processor:
225
lora_request: Optional[LoRARequest] = None,
226
tokenization_kwargs: Optional[dict[str, Any]] = None,
227
trace_headers: Optional[Mapping[str, str]] = None,
229
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
228
priority: int = 0,
229
data_parallel_rank: Optional[int] = None,
230
) -> tuple[Optional[str], EngineCoreRequest]:
@@ -237,8 +235,6 @@ class Processor:
235
self._validate_params(params, lora_request)
236
if trace_headers is not None:
237
raise ValueError("V1 does not support tracing yet.")
240
- if prompt_adapter_request is not None:
241
- raise ValueError("V1 does not support prompt_adapter_request.")
238
239
data_parallel_size = self.vllm_config.parallel_config.data_parallel_size
240
if data_parallel_rank is not None and not (0 <= data_parallel_rank <
@@ -253,12 +249,10 @@ class Processor:
249
# 1. Tokenize text prompt, with LoRA request if one exists.
250
# 2. For multimodal models with a merged preprocessor, preprocess
251
# multimodal data and expand prompt token ids accordingly.
256
- # 3. Apply prompt adapter to prompt token ids if one exists.
252
processed_inputs: ProcessorInputs = self.input_preprocessor.preprocess(
253
prompt,
254
tokenization_kwargs=tokenization_kwargs,
255
lora_request=lora_request,
261
- prompt_adapter_request=prompt_adapter_request,
256
return_mm_hashes=self.use_hash,
257
)
258
from vllm.platforms import current_platform
vllm/v1/utils.py
-2
@@ -318,8 +318,6 @@ def report_usage_stats(
318
# Feature flags
319
"enable_lora":
320
bool(vllm_config.lora_config),
321
- "enable_prompt_adapter":
322
- bool(vllm_config.prompt_adapter_config),
321
"enable_prefix_caching":
322
vllm_config.cache_config.enable_prefix_caching,
323
"enforce_eager":
vllm/v1/worker/gpu_model_runner.py
-1
@@ -104,7 +104,6 @@ class GPUModelRunner(LoRAModelRunnerMixin):
104
self.parallel_config = vllm_config.parallel_config
105
self.scheduler_config = vllm_config.scheduler_config
106
self.speculative_config = vllm_config.speculative_config
107
- self.prompt_adapter_config = vllm_config.prompt_adapter_config
107
self.observability_config = vllm_config.observability_config
108
109
from vllm.model_executor.models.utils import set_cpu_offload_max_bytes
vllm/v1/worker/tpu_model_runner.py
-1
@@ -114,7 +114,6 @@ class TPUModelRunner(LoRAModelRunnerMixin):
114
self.original_parallel_config = original_parallel_config
115
self.scheduler_config = vllm_config.scheduler_config
116
self.speculative_config = vllm_config.speculative_config
117
- self.prompt_adapter_config = vllm_config.prompt_adapter_config
117
self.observability_config = vllm_config.observability_config
118
self.device_config = vllm_config.device_config
119
vllm/v1/worker/tpu_worker.py
-1
@@ -62,7 +62,6 @@ class TPUWorker:
62
self.scheduler_config = vllm_config.scheduler_config
63
self.device_config = vllm_config.device_config
64
self.speculative_config = vllm_config.speculative_config
65
- self.prompt_adapter_config = vllm_config.prompt_adapter_config
65
self.observability_config = vllm_config.observability_config
66
67
self.parallel_config.rank = rank
vllm/worker/enc_dec_model_runner.py
+3
-4
@@ -91,10 +91,9 @@ class EncoderDecoderModelRunner(GPUModelRunnerBase[EncoderDecoderModelInput]):
91
'''
92
EncoderDecoderModelRunner constructor.
93
94
- `lora_config` and `prompt_adapter_config` are
95
- unused (since these features are not yet supported for encoder/decoder
96
- models) but these arguments are present here for compatibility with
97
- the base-class constructor.
94
+ `lora_config` is unused (since these features are not yet supported
95
+ for encoder/decoder models) but these arguments are present here for
96
+ compatibility with the base-class constructor.
97
'''
98
self._maybe_force_supported_attention_backend()
99
vllm/worker/model_runner.py
+2
-149
@@ -45,10 +45,6 @@ from vllm.model_executor.models.utils import set_cpu_offload_max_bytes
45
from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensorInputs,
46
MultiModalKwargs, MultiModalPlaceholderMap,
47
MultiModalRegistry)
48
-from vllm.prompt_adapter.layers import PromptAdapterMapping
49
-from vllm.prompt_adapter.request import PromptAdapterRequest
50
-from vllm.prompt_adapter.worker_manager import (
51
- LRUCacheWorkerPromptAdapterManager)
48
from vllm.sampling_params import SamplingParams
49
from vllm.sequence import IntermediateTensors, SequenceGroupMetadata
50
from vllm.utils import (DeviceMemoryProfiler, GiB_bytes, PyObjectCache,
@@ -95,8 +91,6 @@ class ModelInputForGPU(ModelRunnerInputBase):
91
lora_mapping: Optional["LoRAMapping"] = None
92
lora_requests: Optional[Set[LoRARequest]] = None
93
attn_metadata: Optional["AttentionMetadata"] = None
98
- prompt_adapter_mapping: Optional[PromptAdapterMapping] = None
99
- prompt_adapter_requests: Optional[Set[PromptAdapterRequest]] = None
94
multi_modal_kwargs: Optional[BatchedTensorInputs] = None
95
request_ids_to_seq_ids: Optional[Dict[str, List[int]]] = None
96
finished_requests_ids: Optional[List[str]] = None
@@ -113,8 +107,6 @@ class ModelInputForGPU(ModelRunnerInputBase):
107
"lora_requests": self.lora_requests,
108
"lora_mapping": self.lora_mapping,
109
"multi_modal_kwargs": self.multi_modal_kwargs,
116
- "prompt_adapter_mapping": self.prompt_adapter_mapping,
117
- "prompt_adapter_requests": self.prompt_adapter_requests,
110
"virtual_engine": self.virtual_engine,
111
"request_ids_to_seq_ids": self.request_ids_to_seq_ids,
112
"finished_requests_ids": self.finished_requests_ids,
@@ -164,8 +156,6 @@ class ModelInputForGPUWithSamplingMetadata(ModelInputForGPU):
156
"lora_requests": self.lora_requests,
157
"lora_mapping": self.lora_mapping,
158
"multi_modal_kwargs": self.multi_modal_kwargs,
167
- "prompt_adapter_mapping": self.prompt_adapter_mapping,
168
- "prompt_adapter_requests": self.prompt_adapter_requests,
159
"virtual_engine": self.virtual_engine,
160
"request_ids_to_seq_ids": self.request_ids_to_seq_ids,
161
"finished_requests_ids": self.finished_requests_ids,
@@ -212,8 +202,6 @@ class ModelInputForGPUBuilder(ModelRunnerInputBuilderBase[ModelInputForGPU]):
202
self.lora_index_mapping.clear() # type: ignore
203
self.lora_prompt_mapping.clear() # type: ignore
204
self.lora_requests.clear() # type: ignore
215
- self.prompt_adapter_index_mapping.clear() # type: ignore
216
- self.prompt_adapter_prompt_mapping.clear() # type: ignore
205
206
def __init__(
207
self,
@@ -252,11 +240,6 @@ class ModelInputForGPUBuilder(ModelRunnerInputBuilderBase[ModelInputForGPU]):
240
lora_prompt_mapping: Optional[List[List[int]]] = None,
241
lora_requests: Optional[Set[LoRARequest]] = None,
242
255
- # Prompt adapter inputs.
256
- prompt_adapter_index_mapping: Optional[List[int]] = None,
257
- prompt_adapter_prompt_mapping: Optional[List[int]] = None,
258
- prompt_adapter_request: Optional[PromptAdapterRequest] = None,
259
-
243
# Multi-modal inputs.
244
multi_modal_kwargs: Optional[MultiModalKwargs] = None,
245
multi_modal_placeholder_maps: Optional[Dict[
@@ -360,18 +343,6 @@ class ModelInputForGPUBuilder(ModelRunnerInputBuilderBase[ModelInputForGPU]):
343
else:
344
self.lora_requests.clear()
345
363
- if prompt_adapter_index_mapping:
364
- self.prompt_adapter_index_mapping = \
365
- prompt_adapter_index_mapping
366
- else:
367
- self.prompt_adapter_index_mapping.clear()
368
-
369
- if prompt_adapter_prompt_mapping:
370
- self.prompt_adapter_prompt_mapping = \
371
- prompt_adapter_prompt_mapping
372
- else:
373
- self.prompt_adapter_prompt_mapping.clear()
374
-
346
else:
347
self.input_tokens = input_tokens or []
348
self.inputs_embeds = inputs_embeds
@@ -390,12 +361,6 @@ class ModelInputForGPUBuilder(ModelRunnerInputBuilderBase[ModelInputForGPU]):
361
self.lora_prompt_mapping = lora_prompt_mapping or []
362
self.lora_requests = lora_requests or set()
363
393
- self.prompt_adapter_index_mapping = (
394
- prompt_adapter_index_mapping or [])
395
- self.prompt_adapter_prompt_mapping = (
396
- prompt_adapter_prompt_mapping or [])
397
-
398
- self.prompt_adapter_request = prompt_adapter_request
364
self.multi_modal_kwargs = multi_modal_kwargs
365
self.multi_modal_placeholder_maps = multi_modal_placeholder_maps
366
self.prefix_cache_hit = prefix_cache_hit
@@ -485,7 +450,6 @@ class ModelInputForGPUBuilder(ModelRunnerInputBuilderBase[ModelInputForGPU]):
450
# Compute functions for each sequence group.
451
# WARNING: The order of the functions matters!
452
self.per_seq_group_compute_fns = [
488
- self._compute_prompt_adapter_input,
453
self._compute_multi_modal_input,
454
]
455
@@ -496,8 +460,6 @@ class ModelInputForGPUBuilder(ModelRunnerInputBuilderBase[ModelInputForGPU]):
460
self.sliding_window = self.runner.sliding_window
461
self.block_size = self.runner.block_size
462
self.enable_lora = self.runner.lora_config is not None
499
- self.enable_prompt_adapter = (self.runner.prompt_adapter_config
500
- is not None)
463
464
# Attention metadata inputs.
465
if self.attn_backend is not None:
@@ -693,34 +655,6 @@ class ModelInputForGPUBuilder(ModelRunnerInputBuilderBase[ModelInputForGPU]):
655
else:
656
inter_data.lora_prompt_mapping.append([])
657
696
- def _compute_prompt_adapter_input(
697
- self, inter_data: InterDataForSeqGroup,
698
- seq_group_metadata: SequenceGroupMetadata):
699
- """If prompt adapter is enabled, compute index and prompt mapping.
700
- """
701
- # Note that when is_prompt=True, we expect only one sequence
702
- # in the group.
703
- if not self.enable_prompt_adapter:
704
- return
705
-
706
- prompt_adapter_id = seq_group_metadata.prompt_adapter_id
707
- if prompt_adapter_id <= 0 or not inter_data.is_prompt:
708
- return
709
-
710
- # We expect only one sequence in the group when is_prompt=True.
711
- assert inter_data.n_seqs == 1
712
- query_len = inter_data.query_lens[0]
713
- inter_data.prompt_adapter_request = (
714
- seq_group_metadata.prompt_adapter_request)
715
-
716
- num_tokens = seq_group_metadata.prompt_adapter_num_virtual_tokens
717
- inter_data.prompt_adapter_index_mapping = [
718
- prompt_adapter_id
719
- ] * num_tokens + [0] * (query_len - num_tokens)
720
- inter_data.prompt_adapter_prompt_mapping = [prompt_adapter_id] * (
721
- query_len if seq_group_metadata.sampling_params
722
- and seq_group_metadata.sampling_params.prompt_logprobs else 1)
723
-
658
def _compute_multi_modal_input(self, inter_data: InterDataForSeqGroup,
659
seq_group_metadata: SequenceGroupMetadata):
660
"""If multi-modal data is given, add it to the input."""
@@ -1009,29 +943,6 @@ class ModelInputForGPUBuilder(ModelRunnerInputBuilderBase[ModelInputForGPU]):
943
prompt_mapping=lora_prompt_mapping,
944
is_prefill=not self.decode_only))
945
1012
- # Prompt adapter data.
1013
- prompt_adapter_requests: Set[PromptAdapterRequest] = set()
1014
- prompt_adapter_mapping = None
1015
- if self.enable_prompt_adapter:
1016
- prompt_adapter_requests = set(
1017
- data.prompt_adapter_request for data in self.inter_data_list
1018
- if data.prompt_adapter_request is not None)
1019
- prompt_adapter_index_mapping = flatten_2d_lists([
1020
- inter_data.prompt_adapter_index_mapping
1021
- for inter_data in self.inter_data_list
1022
- ])
1023
- if cuda_graph_pad_size:
1024
- prompt_adapter_index_mapping.extend(
1025
- itertools.repeat(0, cuda_graph_pad_size))
1026
- prompt_adapter_prompt_mapping = flatten_2d_lists([
1027
- inter_data.prompt_adapter_prompt_mapping
1028
- for inter_data in self.inter_data_list
1029
- ])
1030
- prompt_adapter_mapping = PromptAdapterMapping(
1031
- prompt_adapter_index_mapping,
1032
- prompt_adapter_prompt_mapping,
1033
- )
1034
-
946
# Multi-modal data.
947
multi_modal_kwargs_list = [
948
data.multi_modal_kwargs for data in self.inter_data_list
@@ -1051,9 +962,7 @@ class ModelInputForGPUBuilder(ModelRunnerInputBuilderBase[ModelInputForGPU]):
962
lora_requests=lora_requests,
963
multi_modal_kwargs=multi_modal_kwargs,
964
request_ids_to_seq_ids=request_ids_to_seq_ids,
1054
- finished_requests_ids=self.finished_requests_ids,
1055
- prompt_adapter_mapping=prompt_adapter_mapping,
1056
- prompt_adapter_requests=prompt_adapter_requests)
965
+ finished_requests_ids=self.finished_requests_ids)
966
967
968
class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
@@ -1148,7 +1057,6 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
1057
self.model: nn.Module # Set after load_model
1058
# Set after load_model.
1059
self.lora_manager: Optional[LRUCacheWorkerLoRAManager] = None
1151
- self.prompt_adapter_manager: LRUCacheWorkerPromptAdapterManager = None
1060
self.sampler = get_sampler()
1061
1062
set_cpu_offload_max_bytes(
@@ -1207,14 +1115,7 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
1115
logger.info("Model loading took %.4f GiB and %.6f seconds",
1116
self.model_memory_usage / GiB_bytes,
1117
time_after_load - time_before_load)
1210
- if self.prompt_adapter_config:
1211
- self.prompt_adapter_manager = LRUCacheWorkerPromptAdapterManager(
1212
- self.scheduler_config.max_num_seqs,
1213
- self.scheduler_config.max_num_batched_tokens, self.device,
1214
- self.prompt_adapter_config)
1215
- self.model = (
1216
- self.prompt_adapter_manager.create_prompt_adapter_manager(
1217
- self.model))
1118
+
1119
1120
if self.vllm_config.compilation_config.level ==\
1121
CompilationLevel.DYNAMO_AS_IS and supports_dynamo():
@@ -1466,40 +1367,6 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
1367
raise RuntimeError("LoRA is not enabled.")
1368
return self.lora_manager.list_adapters()
1369
1469
- def remove_all_prompt_adapters(self):
1470
- if not self.prompt_adapter_manager:
1471
- raise RuntimeError("PromptAdapter is not enabled.")
1472
- self.prompt_adapter_manager.remove_all_adapters()
1473
-
1474
- def set_active_prompt_adapters(
1475
- self, prompt_adapter_requests: Set[PromptAdapterRequest],
1476
- prompt_adapter_mapping: PromptAdapterMapping) -> None:
1477
- if not self.prompt_adapter_manager:
1478
- raise RuntimeError("PromptAdapter is not enabled.")
1479
- self.prompt_adapter_manager.set_active_adapters(
1480
- prompt_adapter_requests, prompt_adapter_mapping)
1481
-
1482
- def add_prompt_adapter(
1483
- self, prompt_adapter_request: PromptAdapterRequest) -> bool:
1484
- if not self.prompt_adapter_manager:
1485
- raise RuntimeError("PromptAdapter is not enabled.")
1486
- return self.prompt_adapter_manager.add_adapter(prompt_adapter_request)
1487
-
1488
- def remove_prompt_adapter(self, prompt_adapter_id: int) -> bool:
1489
- if not self.prompt_adapter_manager:
1490
- raise RuntimeError("PromptAdapter is not enabled.")
1491
- return self.prompt_adapter_manager.remove_adapter(prompt_adapter_id)
1492
-
1493
- def pin_prompt_adapter(self, prompt_adapter_id: int) -> bool:
1494
- if not self.prompt_adapter_manager:
1495
- raise RuntimeError("PromptAdapter is not enabled.")
1496
- return self.prompt_adapter_manager.pin_adapter(prompt_adapter_id)
1497
-
1498
- def list_prompt_adapters(self) -> Set[int]:
1499
- if not self.prompt_adapter_manager:
1500
- raise RuntimeError("PromptAdapter is not enabled.")
1501
- return self.prompt_adapter_manager.list_adapters()
1502
-
1370
@torch.inference_mode()
1371
def capture_model(self, kv_caches: List[List[torch.Tensor]]) -> None:
1372
"""Cuda graph capture a model.
@@ -1609,13 +1476,6 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
1476
self.set_active_loras(set([dummy_lora_request]),
1477
lora_mapping)
1478
1612
- if self.prompt_adapter_config:
1613
- prompt_adapter_mapping = PromptAdapterMapping(
1614
- [-1] * batch_size,
1615
- [-1] * batch_size,
1616
- )
1617
- self.set_active_prompt_adapters(
1618
- set(), prompt_adapter_mapping)
1479
graph_runner = CUDAGraphRunner(
1480
self.model, self.attn_backend.get_name(),
1481
self.attn_state.graph_clone(batch_size),
@@ -1776,13 +1636,6 @@ class ModelRunner(GPUModelRunnerBase[ModelInputForGPUWithSamplingMetadata]):
1636
self.set_active_loras(model_input.lora_requests,
1637
model_input.lora_mapping)
1638
1779
- if self.prompt_adapter_config:
1780
- assert model_input.prompt_adapter_requests is not None
1781
- assert model_input.prompt_adapter_mapping is not None
1782
- self.set_active_prompt_adapters(
1783
- model_input.prompt_adapter_requests,
1784
- model_input.prompt_adapter_mapping)
1785
-
1639
self.attn_state.begin_forward(model_input)
1640
1641
# Currently cuda graph is only supported by the decode phase.
vllm/worker/model_runner_base.py
-1
@@ -190,7 +190,6 @@ class ModelRunnerBase(ABC, Generic[T]):
190
self.scheduler_config = vllm_config.scheduler_config
191
self.device_config = vllm_config.device_config
192
self.speculative_config = vllm_config.speculative_config
193
- self.prompt_adapter_config = vllm_config.prompt_adapter_config
193
self.observability_config = vllm_config.observability_config
194
195
# Map of request_id -> generator used for seeded random sampling
vllm/worker/multi_step_model_runner.py
-3
@@ -288,9 +288,6 @@ class StatefulModelInput(BroadcastableModelInput):
288
assert fmi.lora_requests is not None
289
assert len(fmi.lora_requests) == 0
290
assert fmi.attn_metadata is not None
291
- assert fmi.prompt_adapter_mapping is None
292
- assert fmi.prompt_adapter_requests is not None
293
- assert len(fmi.prompt_adapter_requests) == 0
291
assert fmi.multi_modal_kwargs is not None
292
assert len(fmi.multi_modal_kwargs) == 0
293
vllm/worker/pooling_model_runner.py
-7
@@ -64,13 +64,6 @@ class PoolingModelRunner(
64
self.set_active_loras(model_input.lora_requests,
65
model_input.lora_mapping)
66
67
- if self.prompt_adapter_config:
68
- assert model_input.prompt_adapter_requests is not None
69
- assert model_input.prompt_adapter_mapping is not None
70
- self.set_active_prompt_adapters(
71
- model_input.prompt_adapter_requests,
72
- model_input.prompt_adapter_mapping)
73
-
67
# Currently cuda graph is only supported by the decode phase.
68
assert model_input.attn_metadata is not None
69
prefill_meta = model_input.attn_metadata.prefill_metadata
vllm/worker/utils.py
-4
@@ -47,7 +47,3 @@ def assert_enc_dec_mr_supported_scenario(
47
if enc_dec_mr.scheduler_config.num_lookahead_slots > 0:
48
raise NotImplementedError(
49
STR_NOT_IMPL_ENC_DEC_ERR_STRS['STR_NOT_IMPL_ENC_DEC_SPEC_DEC'])
50
-
51
- if enc_dec_mr.prompt_adapter_config is not None:
52
- raise NotImplementedError(STR_NOT_IMPL_ENC_DEC_ERR_STRS[
53
- 'STR_NOT_IMPL_ENC_DEC_PROMPT_ADAPTER'])
vllm/worker/worker.py
-14
@@ -22,7 +22,6 @@ from vllm.model_executor import set_random_seed
22
from vllm.model_executor.layers.sampler import SamplerOutput
23
from vllm.model_executor.model_loader.tensorizer import TensorizerConfig
24
from vllm.platforms import current_platform
25
-from vllm.prompt_adapter.request import PromptAdapterRequest
25
from vllm.sequence import (ExecuteModelRequest, IntermediateTensors,
26
SequenceGroupMetadata, SequenceGroupMetadataDelta)
27
from vllm.utils import (GiB_bytes, MemorySnapshot, bind_kv_cache,
@@ -513,19 +512,6 @@ class Worker(LocalOrDistributedWorkerBase):
512
def list_loras(self) -> Set[int]:
513
return self.model_runner.list_loras()
514
516
- def add_prompt_adapter(
517
- self, prompt_adapter_request: PromptAdapterRequest) -> bool:
518
- return self.model_runner.add_prompt_adapter(prompt_adapter_request)
519
-
520
- def remove_prompt_adapter(self, prompt_adapter_id: int) -> bool:
521
- return self.model_runner.remove_lora(prompt_adapter_id)
522
-
523
- def pin_prompt_adapter(self, prompt_adapter_id: int) -> bool:
524
- return self.model_runner.pin_prompt_adapter(prompt_adapter_id)
525
-
526
- def list_prompt_adapters(self) -> Set[int]:
527
- return self.model_runner.list_prompt_adapters()
528
-
515
@property
516
def max_model_len(self) -> int:
517
return self.model_config.max_model_len
vllm/worker/worker_base.py
-1
@@ -49,7 +49,6 @@ class WorkerBase:
49
self.scheduler_config = vllm_config.scheduler_config
50
self.device_config = vllm_config.device_config
51
self.speculative_config = vllm_config.speculative_config
52
- self.prompt_adapter_config = vllm_config.prompt_adapter_config
52
self.observability_config = vllm_config.observability_config
53
self.kv_transfer_config = vllm_config.kv_transfer_config
54
self.compilation_config = vllm_config.compilation_config