A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 11, 20:00 UTC
kv_transfer: Rename the shared storage connectors (#30201)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Or Ozeri committed
Dec 9, 2025 at 06:46 UTC
4c6fd258808ed42fc98a94f3a849f5fc9efebc20
27 files changed
+129
-129
.buildkite/scripts/hardware_ci/run-xpu-test.sh
+1
-1
@@ -47,6 +47,6 @@ docker run \
47
pytest -v -s v1/worker --ignore=v1/worker/test_gpu_model_runner.py
48
pytest -v -s v1/structured_output
49
pytest -v -s v1/spec_decode --ignore=v1/spec_decode/test_max_len.py --ignore=v1/spec_decode/test_tree_attention.py --ignore=v1/spec_decode/test_speculators_eagle3.py
50
- pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector.py --ignore=v1/kv_connector/unit/test_shared_storage_connector.py --ignore=v1/kv_connector/unit/test_lmcache_integration.py
50
+ pytest -v -s v1/kv_connector/unit --ignore=v1/kv_connector/unit/test_multi_connector.py --ignore=v1/kv_connector/unit/test_nixl_connector.py --ignore=v1/kv_connector/unit/test_example_connector.py --ignore=v1/kv_connector/unit/test_lmcache_integration.py
51
pytest -v -s v1/test_serial_utils.py
52
'
docs/features/disagg_encoder.md
+3
-3
@@ -32,14 +32,14 @@ Design doc: <https://docs.google.com/document/d/1aed8KtC6XkXtdoV87pWT0a8OJlZ-Cpn
32
33
## 2 Usage Example
34
35
-The current reference pathway is **SharedStorageConnector**.
35
+The current reference pathway is **ExampleConnector**.
36
Below ready-to-run scripts shows the workflow:
37
38
1 Encoder instance + 1 PD instance:
39
-`examples/online_serving/disaggregated_encoder/shared_storage_connector/disagg_encoder_example.sh`
39
+`examples/online_serving/disaggregated_encoder/disagg_1e1pd_example.sh`
40
41
1 Encoder instance + 1 Prefill instance + 1 Decode instance:
42
-`examples/online_serving/disaggregated_encoder/shared_storage_connector/disagg_epd_example.sh`
42
+`examples/online_serving/disaggregated_encoder/disagg_1e1p1d_example.sh`
43
44
---
45
docs/features/disagg_prefill.md
+2
-2
@@ -21,14 +21,14 @@ Please refer to [examples/online_serving/disaggregated_prefill.sh](../../example
21
22
Now supports 5 types of connectors:
23
24
-- **SharedStorageConnector**: refer to [examples/offline_inference/disaggregated-prefill-v1/run.sh](../../examples/offline_inference/disaggregated-prefill-v1/run.sh) for the example usage of SharedStorageConnector disaggregated prefilling.
24
+- **ExampleConnector**: refer to [examples/offline_inference/disaggregated-prefill-v1/run.sh](../../examples/offline_inference/disaggregated-prefill-v1/run.sh) for the example usage of ExampleConnector disaggregated prefilling.
25
- **LMCacheConnectorV1**: refer to [examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh](../../examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh) for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission.
26
- **NixlConnector**: refer to [tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh) for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. For detailed usage guide, see [NixlConnector Usage Guide](nixl_connector_usage.md).
27
- **P2pNcclConnector**: refer to [examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh](../../examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh) for the example usage of P2pNcclConnector disaggregated prefilling.
28
- **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
29
30
```bash
31
- --kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both"},{"kv_connector":"SharedStorageConnector","kv_role":"kv_both","kv_connector_extra_config":{"shared_storage_path":"local_storage"}}]}}'
31
+ --kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both"},{"kv_connector":"ExampleConnector","kv_role":"kv_both","kv_connector_extra_config":{"shared_storage_path":"local_storage"}}]}}'
32
```
33
34
For NixlConnector, you may also specify one or multiple NIXL_Backend. Such as:
examples/offline_inference/disaggregated-prefill-v1/decode_example.py
+1
-1
@@ -30,7 +30,7 @@ def main():
30
max_num_batched_tokens=64,
31
max_num_seqs=16,
32
kv_transfer_config=KVTransferConfig(
33
- kv_connector="SharedStorageConnector",
33
+ kv_connector="ExampleConnector",
34
kv_role="kv_both",
35
kv_connector_extra_config={"shared_storage_path": "local_storage"},
36
),
examples/offline_inference/disaggregated-prefill-v1/prefill_example.py
+1
-1
@@ -26,7 +26,7 @@ def main():
26
enforce_eager=True,
27
gpu_memory_utilization=0.8,
28
kv_transfer_config=KVTransferConfig(
29
- kv_connector="SharedStorageConnector",
29
+ kv_connector="ExampleConnector",
30
kv_role="kv_both",
31
kv_connector_extra_config={"shared_storage_path": "local_storage"},
32
),
examples/offline_inference/kv_load_failure_recovery/README.md
+2
-2
@@ -10,7 +10,7 @@ It demonstrates vLLM's ability to recover from KV load failures in both synchron
10
- `decode_example.py` – performs the decode stage. Accepts:
11
- `--simulate-failure`: simulates KV load failure using a custom connector.
12
- `--async-load`: enables asynchronous KV loading mode.
13
-- `rogue_shared_storage_connector.py` – defines `RogueSharedStorageConnector`, a subclass of `SharedStorageConnector`, that simulates missing or corrupted external KV blocks by failing to load blocks for the first decode request.
13
+- `load_recovery_example_connector.py` – defines `LoadRecoveryExampleConnector`, a subclass of `ExampleConnector`, that simulates missing or corrupted external KV blocks by failing to load blocks for the first decode request.
14
- `run.sh` – orchestrates the test: runs the prefill stage, then three decode stages:
15
1. Normal decode (baseline).
16
2. Decode with simulated sync KV load failure.
@@ -20,7 +20,7 @@ It demonstrates vLLM's ability to recover from KV load failures in both synchron
20
21
## How It Works
22
23
-- The test dynamically loads `RogueSharedStorageConnector` via `KVTransferConfig.kv_connector_module_path`, enabling controlled simulation of load failures without modifying the original connector.
23
+- The test dynamically loads `LoadRecoveryExampleConnector` via `KVTransferConfig.kv_connector_module_path`, enabling controlled simulation of load failures without modifying the original connector.
24
- The decode stages that simulate failure are expected to trigger recovery logic in vLLM, resulting in the same output as the baseline decode.
25
- If recovery fails, the script prints a unified diff of the output mismatch and exits with error.
26
examples/offline_inference/kv_load_failure_recovery/decode_example.py
+3
-3
@@ -35,13 +35,13 @@ def main():
35
36
if args.simulate_failure:
37
ktc = KVTransferConfig(
38
- kv_connector="RogueSharedStorageConnector",
38
+ kv_connector="LoadRecoveryExampleConnector",
39
kv_role="kv_both",
40
kv_connector_extra_config={
41
"shared_storage_path": "local_storage",
42
"async_load": args.async_load,
43
},
44
- kv_connector_module_path="rogue_shared_storage_connector",
44
+ kv_connector_module_path="load_recovery_example_connector",
45
)
46
out_file = (
47
"async_decode_recovered_output.txt"
@@ -50,7 +50,7 @@ def main():
50
)
51
else:
52
ktc = KVTransferConfig(
53
- kv_connector="SharedStorageConnector",
53
+ kv_connector="ExampleConnector",
54
kv_role="kv_both",
55
kv_connector_extra_config={
56
"shared_storage_path": "local_storage",
examples/offline_inference/kv_load_failure_recovery/load_recovery_example_connector.py
renamed
+10
-10
@@ -10,9 +10,9 @@ from vllm.distributed.kv_transfer.kv_connector.v1.base import (
10
KVConnectorMetadata,
11
KVConnectorRole,
12
)
13
-from vllm.distributed.kv_transfer.kv_connector.v1.shared_storage_connector import (
14
- SharedStorageConnector,
15
- SharedStorageConnectorMetadata,
13
+from vllm.distributed.kv_transfer.kv_connector.v1.example_connector import (
14
+ ExampleConnector,
15
+ ExampleConnectorMetadata,
16
)
17
from vllm.forward_context import ForwardContext
18
from vllm.v1.core.kv_cache_manager import KVCacheBlocks
@@ -26,15 +26,15 @@ logging.basicConfig(level=logging.INFO)
26
27
28
@dataclass
29
-class RogueSharedStorageConnectorMetadata(SharedStorageConnectorMetadata):
29
+class LoadRecoveryExampleConnectorMetadata(ExampleConnectorMetadata):
30
req_to_block_ids: dict[str, set[int]] = field(default_factory=dict)
31
32
@classmethod
33
- def from_base(cls, base: SharedStorageConnectorMetadata):
33
+ def from_base(cls, base: ExampleConnectorMetadata):
34
return cls(requests=base.requests)
35
36
37
-class RogueSharedStorageConnector(SharedStorageConnector):
37
+class LoadRecoveryExampleConnector(ExampleConnector):
38
def __init__(self, vllm_config: "VllmConfig", role: KVConnectorRole):
39
super().__init__(vllm_config=vllm_config, role=role)
40
self._async_load = vllm_config.kv_transfer_config.get_from_extra_config(
@@ -45,7 +45,7 @@ class RogueSharedStorageConnector(SharedStorageConnector):
45
self._req_to_block_ids: dict[str, list[int]] = dict()
46
47
def bind_connector_metadata(self, connector_metadata: KVConnectorMetadata) -> None:
48
- assert isinstance(connector_metadata, RogueSharedStorageConnectorMetadata)
48
+ assert isinstance(connector_metadata, LoadRecoveryExampleConnectorMetadata)
49
index, failed_request = next(
50
(
51
(i, x)
@@ -84,7 +84,7 @@ class RogueSharedStorageConnector(SharedStorageConnector):
84
) -> tuple[set[str] | None, set[str] | None]:
85
if self._async_load:
86
meta = self._get_connector_metadata()
87
- assert isinstance(meta, RogueSharedStorageConnectorMetadata)
87
+ assert isinstance(meta, LoadRecoveryExampleConnectorMetadata)
88
if meta.req_to_block_ids:
89
return None, set(meta.req_to_block_ids)
90
@@ -126,9 +126,9 @@ class RogueSharedStorageConnector(SharedStorageConnector):
126
) -> KVConnectorMetadata:
127
if not self._async_load:
128
base = super().build_connector_meta(scheduler_output)
129
- meta = RogueSharedStorageConnectorMetadata.from_base(base)
129
+ meta = LoadRecoveryExampleConnectorMetadata.from_base(base)
130
else:
131
- meta = RogueSharedStorageConnectorMetadata()
131
+ meta = LoadRecoveryExampleConnectorMetadata()
132
if self._requests_need_load:
133
for req_id, request in self._requests_need_load.items():
134
meta.add_request(
examples/offline_inference/kv_load_failure_recovery/prefill_example.py
+1
-1
@@ -26,7 +26,7 @@ def main():
26
enforce_eager=True,
27
gpu_memory_utilization=0.8,
28
kv_transfer_config=KVTransferConfig(
29
- kv_connector="SharedStorageConnector",
29
+ kv_connector="ExampleConnector",
30
kv_role="kv_both",
31
kv_connector_extra_config={"shared_storage_path": "local_storage"},
32
),
examples/online_serving/disaggregated_encoder/README.md
+3
-3
@@ -50,12 +50,12 @@ The vllm instances and `disagg_encoder_proxy` supports local URIs with ```{"url"
50
51
## EC connector and KV transfer
52
53
-The `ECSharedStorageConnector` is used to store the encoder cache on local disk and facilitate transfer. To enable the encoder disaggregation feature, add the following configuration:
53
+The `ECExampleonnector` is used to store the encoder cache on local disk and facilitate transfer. To enable the encoder disaggregation feature, add the following configuration:
54
55
```bash
56
# Add to encoder instance:
57
--ec-transfer-config '{
58
- "ec_connector": "ECSharedStorageConnector",
58
+ "ec_connector": "ECExampleConnector",
59
"ec_role": "ec_producer",
60
"ec_connector_extra_config": {
61
"shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
@@ -64,7 +64,7 @@ The `ECSharedStorageConnector` is used to store the encoder cache on local disk
64
65
# Add to prefill/prefill+decode instance:
66
--ec-transfer-config '{
67
- "ec_connector": "ECSharedStorageConnector",
67
+ "ec_connector": "ECExampleConnector",
68
"ec_role": "ec_consumer",
69
"ec_connector_extra_config": {
70
"shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
examples/online_serving/disaggregated_encoder/disagg_1e1p1d_example.sh
+2
-2
@@ -102,7 +102,7 @@ CUDA_VISIBLE_DEVICES="$GPU_E" vllm serve "$MODEL" \
102
--max-num-seqs 128 \
103
--allowed-local-media-path ${GIT_ROOT}/tests/v1/ec_connector/integration \
104
--ec-transfer-config '{
105
- "ec_connector": "ECSharedStorageConnector",
105
+ "ec_connector": "ECExampleConnector",
106
"ec_role": "ec_producer",
107
"ec_connector_extra_config": {
108
"shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
@@ -126,7 +126,7 @@ vllm serve "$MODEL" \
126
--max-num-seqs 128 \
127
--allowed-local-media-path ${GIT_ROOT}/tests/v1/ec_connector/integration \
128
--ec-transfer-config '{
129
- "ec_connector": "ECSharedStorageConnector",
129
+ "ec_connector": "ECExampleConnector",
130
"ec_role": "ec_consumer",
131
"ec_connector_extra_config": {
132
"shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
examples/online_serving/disaggregated_encoder/disagg_1e1pd_example.sh
+2
-2
@@ -96,7 +96,7 @@ CUDA_VISIBLE_DEVICES="$GPU_E" vllm serve "$MODEL" \
96
--max-num-seqs 128 \
97
--allowed-local-media-path ${GIT_ROOT}/tests/v1/ec_connector/integration \
98
--ec-transfer-config '{
99
- "ec_connector": "ECSharedStorageConnector",
99
+ "ec_connector": "ECExampleConnector",
100
"ec_role": "ec_producer",
101
"ec_connector_extra_config": {
102
"shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
@@ -117,7 +117,7 @@ CUDA_VISIBLE_DEVICES="$GPU_PD" vllm serve "$MODEL" \
117
--max-num-seqs 128 \
118
--allowed-local-media-path ${GIT_ROOT}/tests/v1/ec_connector/integration \
119
--ec-transfer-config '{
120
- "ec_connector": "ECSharedStorageConnector",
120
+ "ec_connector": "ECExampleConnector",
121
"ec_role": "ec_consumer",
122
"ec_connector_extra_config": {
123
"shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
tests/distributed/test_kvlayout.py
+1
-1
@@ -61,7 +61,7 @@ def test_get_kv_connector_cache_layout_with_multi_connector():
61
kv_role="kv_both",
62
kv_connector_extra_config={
63
"connectors": [
64
- {"kv_connector": "SharedStorageConnector", "kv_role": "kv_both"},
64
+ {"kv_connector": "ExampleConnector", "kv_role": "kv_both"},
65
{"kv_connector": "NixlConnector", "kv_role": "kv_both"},
66
]
67
},
tests/v1/core/test_scheduler.py
+3
-3
@@ -1536,7 +1536,7 @@ def create_scheduler_with_priority(
1536
)
1537
kv_transfer_config = (
1538
KVTransferConfig(
1539
- kv_connector="SharedStorageConnector",
1539
+ kv_connector="ExampleConnector",
1540
kv_role="kv_both",
1541
kv_connector_extra_config={"shared_storage_path": "local_storage"},
1542
)
@@ -1552,7 +1552,7 @@ def create_scheduler_with_priority(
1552
1553
ec_transfer_config = (
1554
ECTransferConfig(
1555
- ec_connector="ECSharedStorageConnector",
1555
+ ec_connector="ECExampleConnector",
1556
ec_role=ec_role,
1557
ec_connector_extra_config={"shared_storage_path": "/tmp/ec_test"},
1558
)
@@ -2413,7 +2413,7 @@ def _assert_right_ec_connector_metadata(
2413
metadata_dict = {mm_data.mm_hash: mm_data for mm_data in metadata.mm_datas}
2414
2415
# Check all required identifiers exist in metadata; and no extra
2416
- # In ECSharedStorageConnector format
2416
+ # In ECExampleConnector format
2417
# NOTE: even having same identifier, the mm_features can be different
2418
# since their mm_position can be in different offsets, etc
2419
identifiers_dict = {f.identifier for f in mm_features_list}
tests/v1/core/utils.py
+2
-2
@@ -108,7 +108,7 @@ def create_scheduler(
108
)
109
elif use_kv_connector:
110
kv_transfer_config = KVTransferConfig(
111
- kv_connector="SharedStorageConnector",
111
+ kv_connector="ExampleConnector",
112
kv_role="kv_both",
113
kv_connector_extra_config={"shared_storage_path": "local_storage"},
114
)
@@ -121,7 +121,7 @@ def create_scheduler(
121
122
ec_transfer_config = (
123
ECTransferConfig(
124
- ec_connector="ECSharedStorageConnector",
124
+ ec_connector="ECExampleConnector",
125
ec_role=ec_role,
126
ec_connector_extra_config={"shared_storage_path": "/tmp/ec_test"},
127
)
tests/v1/ec_connector/integration/run_epd_correctness_test.sh
+4
-4
@@ -148,7 +148,7 @@ run_epd_1e_1pd() {
148
--max-num-seqs 128 \
149
--allowed-local-media-path ${GIT_ROOT}/tests/v1/ec_connector/integration \
150
--ec-transfer-config '{
151
- "ec_connector": "ECSharedStorageConnector",
151
+ "ec_connector": "ECExampleConnector",
152
"ec_role": "ec_producer",
153
"ec_connector_extra_config": {
154
"shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
@@ -167,7 +167,7 @@ run_epd_1e_1pd() {
167
--max-num-seqs 128 \
168
--allowed-local-media-path ${GIT_ROOT}/tests/v1/ec_connector/integration \
169
--ec-transfer-config '{
170
- "ec_connector": "ECSharedStorageConnector",
170
+ "ec_connector": "ECExampleConnector",
171
"ec_role": "ec_consumer",
172
"ec_connector_extra_config": {
173
"shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
@@ -348,7 +348,7 @@ run_epd_1e_1p_1d() {
348
--max-num-seqs 128 \
349
--allowed-local-media-path ${GIT_ROOT}/tests/v1/ec_connector/integration \
350
--ec-transfer-config '{
351
- "ec_connector": "ECSharedStorageConnector",
351
+ "ec_connector": "ECExampleConnector",
352
"ec_role": "ec_producer",
353
"ec_connector_extra_config": {
354
"shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
@@ -369,7 +369,7 @@ run_epd_1e_1p_1d() {
369
--max-num-seqs 128 \
370
--allowed-local-media-path ${GIT_ROOT}/tests/v1/ec_connector/integration \
371
--ec-transfer-config '{
372
- "ec_connector": "ECSharedStorageConnector",
372
+ "ec_connector": "ECExampleConnector",
373
"ec_role": "ec_consumer",
374
"ec_connector_extra_config": {
375
"shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
tests/v1/ec_connector/unit/test_ec_example_connector.py
renamed
+43
-43
@@ -1,7 +1,7 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
"""
4
-Unit tests for ECSharedStorageConnector.
4
+Unit tests for ECExampleConnector.
5
"""
6
7
import os
@@ -13,9 +13,9 @@ import torch
13
14
from vllm.config import VllmConfig
15
from vllm.distributed.ec_transfer.ec_connector.base import ECConnectorRole
16
-from vllm.distributed.ec_transfer.ec_connector.shared_storage_connector import (
17
- ECSharedStorageConnector,
18
- ECSharedStorageConnectorMetadata,
16
+from vllm.distributed.ec_transfer.ec_connector.example_connector import (
17
+ ECExampleConnector,
18
+ ECExampleConnectorMetadata,
19
MMMeta,
20
)
21
from vllm.multimodal.inputs import MultiModalFeatureSpec, PlaceholderRange
@@ -81,12 +81,12 @@ def mock_request_with_3_mm():
81
82
83
# ------------------ Unit Tests ------------------ #
84
-class TestECSharedStorageConnectorBasics:
84
+class TestECExampleConnectorBasics:
85
"""Test basic EC connector functionality."""
86
87
def test_initialization_producer(self, mock_vllm_config_producer, temp_storage):
88
"""Test connector initializes correctly as producer."""
89
- connector = ECSharedStorageConnector(
89
+ connector = ECExampleConnector(
90
vllm_config=mock_vllm_config_producer,
91
role=ECConnectorRole.SCHEDULER,
92
)
@@ -98,7 +98,7 @@ class TestECSharedStorageConnectorBasics:
98
99
def test_initialization_consumer(self, mock_vllm_config_consumer, temp_storage):
100
"""Test connector initializes correctly as consumer."""
101
- connector = ECSharedStorageConnector(
101
+ connector = ECExampleConnector(
102
vllm_config=mock_vllm_config_consumer,
103
role=ECConnectorRole.WORKER,
104
)
@@ -109,11 +109,11 @@ class TestECSharedStorageConnectorBasics:
109
110
def test_role_assignment(self, mock_vllm_config_producer):
111
"""Test role is correctly assigned."""
112
- scheduler_connector = ECSharedStorageConnector(
112
+ scheduler_connector = ECExampleConnector(
113
vllm_config=mock_vllm_config_producer,
114
role=ECConnectorRole.SCHEDULER,
115
)
116
- worker_connector = ECSharedStorageConnector(
116
+ worker_connector = ECExampleConnector(
117
vllm_config=mock_vllm_config_producer,
118
role=ECConnectorRole.WORKER,
119
)
@@ -133,7 +133,7 @@ class TestCacheExistence:
133
):
134
"""Test has_caches returns True when all 3 caches exist."""
135
# Test for producer first
136
- producer = ECSharedStorageConnector(
136
+ producer = ECExampleConnector(
137
vllm_config=mock_vllm_config_producer,
138
role=ECConnectorRole.SCHEDULER,
139
)
@@ -154,7 +154,7 @@ class TestCacheExistence:
154
assert all(producer_result), f"Expected all True, got {producer_result}"
155
156
# Also test consumer can check if cache exists
157
- consumer = ECSharedStorageConnector(
157
+ consumer = ECExampleConnector(
158
vllm_config=mock_vllm_config_consumer,
159
role=ECConnectorRole.SCHEDULER,
160
)
@@ -170,7 +170,7 @@ class TestCacheExistence:
170
self, mock_vllm_config_producer, mock_request_with_3_mm
171
):
172
"""Test has_caches returns False when no caches exist."""
173
- connector = ECSharedStorageConnector(
173
+ connector = ECExampleConnector(
174
vllm_config=mock_vllm_config_producer,
175
role=ECConnectorRole.SCHEDULER,
176
)
@@ -186,7 +186,7 @@ class TestCacheExistence:
186
self, mock_vllm_config_producer, mock_request_with_3_mm
187
):
188
"""Test has_caches with some caches existing (1 of 3)."""
189
- connector = ECSharedStorageConnector(
189
+ connector = ECExampleConnector(
190
vllm_config=mock_vllm_config_producer,
191
role=ECConnectorRole.SCHEDULER,
192
)
@@ -213,7 +213,7 @@ class TestStateManagement:
213
self, mock_vllm_config_producer, mock_request_with_3_mm
214
):
215
"""Test state update after allocation for 3 MM items."""
216
- connector = ECSharedStorageConnector(
216
+ connector = ECExampleConnector(
217
vllm_config=mock_vllm_config_producer,
218
role=ECConnectorRole.SCHEDULER,
219
)
@@ -238,7 +238,7 @@ class TestStateManagement:
238
self, mock_vllm_config_producer, mock_request_with_3_mm
239
):
240
"""Test metadata building for 3 MM items."""
241
- connector = ECSharedStorageConnector(
241
+ connector = ECExampleConnector(
242
vllm_config=mock_vllm_config_producer,
243
role=ECConnectorRole.SCHEDULER,
244
)
@@ -252,7 +252,7 @@ class TestStateManagement:
252
metadata = connector.build_connector_meta(scheduler_output)
253
254
# Assert
255
- assert isinstance(metadata, ECSharedStorageConnectorMetadata)
255
+ assert isinstance(metadata, ECExampleConnectorMetadata)
256
assert len(metadata.mm_datas) == 3
257
assert metadata.mm_datas[0].mm_hash == "img_hash_1"
258
assert metadata.mm_datas[0].num_token == 100
@@ -266,7 +266,7 @@ class TestStateManagement:
266
267
def test_build_connector_meta_empty(self, mock_vllm_config_producer):
268
"""Test metadata building with empty state."""
269
- connector = ECSharedStorageConnector(
269
+ connector = ECExampleConnector(
270
vllm_config=mock_vllm_config_producer,
271
role=ECConnectorRole.SCHEDULER,
272
)
@@ -274,14 +274,14 @@ class TestStateManagement:
274
scheduler_output = Mock(spec=SchedulerOutput)
275
metadata = connector.build_connector_meta(scheduler_output)
276
277
- assert isinstance(metadata, ECSharedStorageConnectorMetadata)
277
+ assert isinstance(metadata, ECExampleConnectorMetadata)
278
assert len(metadata.mm_datas) == 0
279
280
def test_state_cleared_after_metadata_build(
281
self, mock_vllm_config_producer, mock_request_with_3_mm
282
):
283
"""Test that state is properly cleared after building metadata."""
284
- connector = ECSharedStorageConnector(
284
+ connector = ECExampleConnector(
285
vllm_config=mock_vllm_config_producer,
286
role=ECConnectorRole.SCHEDULER,
287
)
@@ -310,7 +310,7 @@ class TestCacheSaving:
310
self, mock_vllm_config_producer, mock_request_with_3_mm, temp_storage
311
):
312
"""Test cache saving as producer for 3 different MM items."""
313
- connector = ECSharedStorageConnector(
313
+ connector = ECExampleConnector(
314
vllm_config=mock_vllm_config_producer,
315
role=ECConnectorRole.WORKER,
316
)
@@ -336,7 +336,7 @@ class TestCacheSaving:
336
337
def test_save_caches_consumer_skips(self, mock_vllm_config_consumer):
338
"""Test cache saving is skipped for consumer."""
339
- connector = ECSharedStorageConnector(
339
+ connector = ECExampleConnector(
340
vllm_config=mock_vllm_config_consumer,
341
role=ECConnectorRole.WORKER,
342
)
@@ -366,7 +366,7 @@ class TestCacheLoading:
366
):
367
"""Test consumer loads 3 caches from storage."""
368
# First, create producer to save caches
369
- producer = ECSharedStorageConnector(
369
+ producer = ECExampleConnector(
370
vllm_config=mock_vllm_config_producer,
371
role=ECConnectorRole.WORKER,
372
)
@@ -379,13 +379,13 @@ class TestCacheLoading:
379
producer.save_caches(saved_caches, mm_hash)
380
381
# Now consumer loads
382
- consumer = ECSharedStorageConnector(
382
+ consumer = ECExampleConnector(
383
vllm_config=mock_vllm_config_consumer,
384
role=ECConnectorRole.WORKER,
385
)
386
387
# Setup metadata for all 3
388
- metadata = ECSharedStorageConnectorMetadata()
388
+ metadata = ECExampleConnectorMetadata()
389
for mm_hash in mm_hashes:
390
metadata.add_mm_data(MMMeta.make_meta(mm_hash, 100))
391
consumer.bind_connector_metadata(metadata)
@@ -410,7 +410,7 @@ class TestCacheLoading:
410
):
411
"""Test cache loading skips already cached items."""
412
# Setup: producer saves cache
413
- producer = ECSharedStorageConnector(
413
+ producer = ECExampleConnector(
414
vllm_config=mock_vllm_config_producer,
415
role=ECConnectorRole.WORKER,
416
)
@@ -420,12 +420,12 @@ class TestCacheLoading:
420
producer.save_caches({mm_hash: saved_cache}, mm_hash)
421
422
# Consumer setup
423
- consumer = ECSharedStorageConnector(
423
+ consumer = ECExampleConnector(
424
vllm_config=mock_vllm_config_consumer,
425
role=ECConnectorRole.WORKER,
426
)
427
428
- metadata = ECSharedStorageConnectorMetadata()
428
+ metadata = ECExampleConnectorMetadata()
429
metadata.add_mm_data(MMMeta.make_meta(mm_hash, 100))
430
consumer.bind_connector_metadata(metadata)
431
@@ -444,13 +444,13 @@ class TestCacheLoading:
444
445
def test_start_load_caches_empty_metadata(self, mock_vllm_config_consumer):
446
"""Test loading with empty metadata does nothing."""
447
- consumer = ECSharedStorageConnector(
447
+ consumer = ECExampleConnector(
448
vllm_config=mock_vllm_config_consumer,
449
role=ECConnectorRole.WORKER,
450
)
451
452
# Setup empty metadata
453
- metadata = ECSharedStorageConnectorMetadata()
453
+ metadata = ECExampleConnectorMetadata()
454
consumer.bind_connector_metadata(metadata)
455
456
# Load (should not raise)
@@ -466,7 +466,7 @@ class TestFilenameGeneration:
466
467
def test_generate_foldername(self, mock_vllm_config_producer, temp_storage):
468
"""Test folder name generation."""
469
- connector = ECSharedStorageConnector(
469
+ connector = ECExampleConnector(
470
vllm_config=mock_vllm_config_producer,
471
role=ECConnectorRole.WORKER,
472
)
@@ -479,7 +479,7 @@ class TestFilenameGeneration:
479
480
def test_generate_filename(self, mock_vllm_config_producer, temp_storage):
481
"""Test filename generation."""
482
- connector = ECSharedStorageConnector(
482
+ connector = ECExampleConnector(
483
vllm_config=mock_vllm_config_producer,
484
role=ECConnectorRole.WORKER,
485
)
@@ -493,7 +493,7 @@ class TestFilenameGeneration:
493
494
def test_generate_filename_consistency(self, mock_vllm_config_producer):
495
"""Test filename generation is consistent."""
496
- connector = ECSharedStorageConnector(
496
+ connector = ECExampleConnector(
497
vllm_config=mock_vllm_config_producer,
498
role=ECConnectorRole.WORKER,
499
)
@@ -510,12 +510,12 @@ class TestMetadataBindingLifecycle:
510
511
def test_bind_connector_metadata(self, mock_vllm_config_consumer):
512
"""Test binding connector metadata."""
513
- connector = ECSharedStorageConnector(
513
+ connector = ECExampleConnector(
514
vllm_config=mock_vllm_config_consumer,
515
role=ECConnectorRole.WORKER,
516
)
517
518
- metadata = ECSharedStorageConnectorMetadata()
518
+ metadata = ECExampleConnectorMetadata()
519
metadata.add_mm_data(MMMeta.make_meta("hash_1", 100))
520
521
connector.bind_connector_metadata(metadata)
@@ -524,12 +524,12 @@ class TestMetadataBindingLifecycle:
524
525
def test_clear_connector_metadata(self, mock_vllm_config_consumer):
526
"""Test clearing connector metadata."""
527
- connector = ECSharedStorageConnector(
527
+ connector = ECExampleConnector(
528
vllm_config=mock_vllm_config_consumer,
529
role=ECConnectorRole.WORKER,
530
)
531
532
- metadata = ECSharedStorageConnectorMetadata()
532
+ metadata = ECExampleConnectorMetadata()
533
connector.bind_connector_metadata(metadata)
534
535
connector.clear_connector_metadata()
@@ -538,12 +538,12 @@ class TestMetadataBindingLifecycle:
538
539
def test_get_connector_metadata(self, mock_vllm_config_consumer):
540
"""Test getting connector metadata."""
541
- connector = ECSharedStorageConnector(
541
+ connector = ECExampleConnector(
542
vllm_config=mock_vllm_config_consumer,
543
role=ECConnectorRole.WORKER,
544
)
545
546
- metadata = ECSharedStorageConnectorMetadata()
546
+ metadata = ECExampleConnectorMetadata()
547
connector.bind_connector_metadata(metadata)
548
549
retrieved = connector._get_connector_metadata()
@@ -552,7 +552,7 @@ class TestMetadataBindingLifecycle:
552
553
def test_get_connector_metadata_not_set(self, mock_vllm_config_consumer):
554
"""Test getting metadata when not set raises."""
555
- connector = ECSharedStorageConnector(
555
+ connector = ECExampleConnector(
556
vllm_config=mock_vllm_config_consumer,
557
role=ECConnectorRole.WORKER,
558
)
@@ -566,7 +566,7 @@ class TestEdgeCases:
566
567
def test_save_empty_cache(self, mock_vllm_config_producer):
568
"""Test saving empty tensor."""
569
- connector = ECSharedStorageConnector(
569
+ connector = ECExampleConnector(
570
vllm_config=mock_vllm_config_producer,
571
role=ECConnectorRole.WORKER,
572
)
@@ -579,12 +579,12 @@ class TestEdgeCases:
579
580
def test_load_nonexistent_cache(self, mock_vllm_config_consumer):
581
"""Test loading cache that doesn't exist raises error."""
582
- connector = ECSharedStorageConnector(
582
+ connector = ECExampleConnector(
583
vllm_config=mock_vllm_config_consumer,
584
role=ECConnectorRole.WORKER,
585
)
586
587
- metadata = ECSharedStorageConnectorMetadata()
587
+ metadata = ECExampleConnectorMetadata()
588
metadata.add_mm_data(MMMeta.make_meta("nonexistent_hash", 100))
589
connector.bind_connector_metadata(metadata)
590
@@ -596,7 +596,7 @@ class TestEdgeCases:
596
597
def test_has_caches_empty_request(self, mock_vllm_config_producer):
598
"""Test has_caches with request that has no MM data."""
599
- connector = ECSharedStorageConnector(
599
+ connector = ECExampleConnector(
600
vllm_config=mock_vllm_config_producer,
601
role=ECConnectorRole.SCHEDULER,
602
)
tests/v1/engine/test_engine_core.py
+2
-2
@@ -507,7 +507,7 @@ def test_encoder_instance_zero_kv_cache(
507
)
508
kv_transfer_config = (
509
KVTransferConfig(
510
- kv_connector="SharedStorageConnector",
510
+ kv_connector="ExampleConnector",
511
kv_role="kv_both",
512
kv_connector_extra_config={"shared_storage_path": "local_storage"},
513
)
@@ -515,7 +515,7 @@ def test_encoder_instance_zero_kv_cache(
515
else None
516
)
517
ec_transfer_config = ECTransferConfig(
518
- ec_connector="ECSharedStorageConnector",
518
+ ec_connector="ECExampleConnector",
519
ec_role=ec_role,
520
ec_connector_extra_config={"shared_storage_path": "/tmp/ec_test_encoder"},
521
)
tests/v1/kv_connector/unit/test_backwards_compatibility.py
+4
-4
@@ -218,12 +218,12 @@ def test_internal_connector_uses_new_signature():
218
Test that internal connectors (registered in factory) always use the new
219
signature and get kv_cache_config.
220
"""
221
- from vllm.distributed.kv_transfer.kv_connector.v1.shared_storage_connector import (
222
- SharedStorageConnector,
221
+ from vllm.distributed.kv_transfer.kv_connector.v1.example_connector import (
222
+ ExampleConnector,
223
)
224
225
vllm_config = create_vllm_config()
226
- vllm_config.kv_transfer_config.kv_connector = "SharedStorageConnector"
226
+ vllm_config.kv_transfer_config.kv_connector = "ExampleConnector"
227
228
scheduler = create_scheduler(vllm_config)
229
kv_cache_config = scheduler.kv_cache_config
@@ -233,7 +233,7 @@ def test_internal_connector_uses_new_signature():
233
)
234
235
assert connector is not None
236
- assert isinstance(connector, SharedStorageConnector)
236
+ assert isinstance(connector, ExampleConnector)
237
assert connector._kv_cache_config is not None
238
assert connector._kv_cache_config == kv_cache_config
239
tests/v1/kv_connector/unit/test_example_connector.py
renamed
+3
-3
@@ -119,16 +119,16 @@ def process_prompt(processor, llm: LLM, question: str, image_urls: list[Image]):
119
)
120
def test_shared_storage_connector_hashes(tmp_path):
121
"""
122
- Tests that SharedStorageConnector saves KV to the storage locations
122
+ Tests that ExampleConnector saves KV to the storage locations
123
with proper hashes; that are unique for inputs with identical text but
124
different images (same size), or same multiple images but different orders.
125
"""
126
# Using tmp_path as the storage path to store KV
127
print(f"KV storage path at: {str(tmp_path)}")
128
129
- # Configure the SharedStorageConnector
129
+ # Configure the ExampleConnector
130
kv_transfer_config = KVTransferConfig(
131
- kv_connector="SharedStorageConnector",
131
+ kv_connector="ExampleConnector",
132
kv_role="kv_both",
133
kv_connector_extra_config={"shared_storage_path": str(tmp_path)},
134
)
tests/v1/kv_connector/unit/test_kv_connector_lifecyle.py
+5
-5
@@ -1,8 +1,8 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
4
-from vllm.distributed.kv_transfer.kv_connector.v1.shared_storage_connector import ( # noqa: E501
5
- SharedStorageConnectorMetadata,
4
+from vllm.distributed.kv_transfer.kv_connector.v1.example_connector import ( # noqa: E501
5
+ ExampleConnectorMetadata,
6
)
7
from vllm.distributed.kv_transfer.kv_transfer_state import (
8
ensure_kv_transfer_initialized,
@@ -11,7 +11,7 @@ from vllm.distributed.kv_transfer.kv_transfer_state import (
11
from vllm.v1.core.sched.output import CachedRequestData, SchedulerOutput
12
from vllm.v1.worker.kv_connector_model_runner_mixin import KVConnectorModelRunnerMixin
13
14
-# Importing utils registers TestSharedStorageConnector with the factory
14
+# Importing utils registers TestExampleConnector with the factory
15
from .utils import create_vllm_config
16
17
@@ -26,13 +26,13 @@ def _make_empty_scheduler_output():
26
num_common_prefix_blocks=[],
27
finished_req_ids=set(),
28
free_encoder_mm_hashes=[],
29
- kv_connector_metadata=SharedStorageConnectorMetadata(),
29
+ kv_connector_metadata=ExampleConnectorMetadata(),
30
)
31
32
33
def test_kv_connector_mixin_clears_metadata():
34
vllm_config = create_vllm_config()
35
- vllm_config.kv_transfer_config.kv_connector = "TestSharedStorageConnector"
35
+ vllm_config.kv_transfer_config.kv_connector = "TestExampleConnector"
36
vllm_config.kv_transfer_config.kv_role = "kv_both"
37
vllm_config.kv_transfer_config.kv_connector_extra_config["name"] = "unit"
38
tests/v1/kv_connector/unit/test_multi_connector.py
+11
-11
@@ -77,9 +77,9 @@ def _compare_directories(dir1: Path, dir2: Path) -> bool:
77
"https://github.com/ROCm/pytorch/issues/2822"
78
),
79
)
80
-def test_multi_shared_storage_connector_consistency():
80
+def test_multi_example_connector_consistency():
81
"""
82
- Tests that MultiConnector with two SharedStorageConnectors saves
82
+ Tests that MultiConnector with two ExampleConnectors saves
83
identical KV cache data to separate storage locations.
84
"""
85
storage_1_path = Path("storage_1/")
@@ -89,14 +89,14 @@ def test_multi_shared_storage_connector_consistency():
89
storage_1_path.mkdir()
90
storage_2_path.mkdir()
91
92
- # Configure MultiConnector with two SharedStorageConnectors
92
+ # Configure MultiConnector with two ExampleConnectors
93
kv_transfer_config = KVTransferConfig(
94
kv_connector="MultiConnector",
95
kv_role="kv_both",
96
kv_connector_extra_config={
97
"connectors": [
98
{
99
- "kv_connector": "TestSharedStorageConnector",
99
+ "kv_connector": "TestExampleConnector",
100
"kv_role": "kv_both",
101
"kv_connector_extra_config": {
102
"shared_storage_path": str(storage_1_path),
@@ -105,7 +105,7 @@ def test_multi_shared_storage_connector_consistency():
105
"kv_connector_module_path": "tests.v1.kv_connector.unit.utils",
106
},
107
{
108
- "kv_connector": "TestSharedStorageConnector",
108
+ "kv_connector": "TestExampleConnector",
109
"kv_role": "kv_both",
110
"kv_connector_extra_config": {
111
"shared_storage_path": str(storage_2_path),
@@ -427,7 +427,7 @@ class TestMultiConnectorStats:
427
428
def test_build_kv_connector_stats_skips_connectors_without_custom_stats(self):
429
"""Test that connectors without custom stats (return None) are skipped."""
430
- # SharedStorageConnector doesn't override build_kv_connector_stats,
430
+ # ExampleConnector doesn't override build_kv_connector_stats,
431
# so it returns None and should be skipped
432
serialized_data = {
433
"NixlConnector": {
@@ -440,7 +440,7 @@ class TestMultiConnectorStats:
440
"num_failed_notifications": [],
441
}
442
},
443
- "SharedStorageConnector": {"data": {"some_field": [1, 2, 3]}},
443
+ "ExampleConnector": {"data": {"some_field": [1, 2, 3]}},
444
}
445
446
stats = MultiConnector.build_kv_connector_stats(data=serialized_data)
@@ -451,8 +451,8 @@ class TestMultiConnectorStats:
451
assert len(stats.data) == 1
452
assert "NixlConnector" in stats.data
453
assert isinstance(stats.data["NixlConnector"], NixlKVConnectorStats)
454
- # SharedStorageConnector should be skipped (returns None)
455
- assert "SharedStorageConnector" not in stats.data
454
+ # ExampleConnector should be skipped (returns None)
455
+ assert "ExampleConnector" not in stats.data
456
457
def test_build_kv_connector_stats_handles_malformed_data(self):
458
"""Test that malformed data raises appropriate errors."""
@@ -527,13 +527,13 @@ class TestMultiConnectorStats:
527
)
528
529
stats2 = MultiKVConnectorStats(
530
- data={"SharedStorageConnector": KVConnectorStats(data={"field": [1, 2]})}
530
+ data={"ExampleConnector": KVConnectorStats(data={"field": [1, 2]})}
531
)
532
533
result = stats1.aggregate(stats2)
534
535
assert "NixlConnector" in result.data
536
- assert "SharedStorageConnector" in result.data
536
+ assert "ExampleConnector" in result.data
537
538
def test_reduce(self):
539
"""Test that reduce() correctly reduces all nested connector stats."""
tests/v1/kv_connector/unit/utils.py
+5
-5
@@ -24,8 +24,8 @@ from vllm.distributed.kv_transfer.kv_connector.v1.base import (
24
KVConnectorMetadata,
25
KVConnectorRole,
26
)
27
-from vllm.distributed.kv_transfer.kv_connector.v1.shared_storage_connector import ( # noqa
28
- SharedStorageConnector,
27
+from vllm.distributed.kv_transfer.kv_connector.v1.example_connector import ( # noqa
28
+ ExampleConnector,
29
)
30
from vllm.utils.hashing import sha256
31
from vllm.v1.core.kv_cache_manager import KVCacheBlocks
@@ -264,10 +264,10 @@ def create_model_runner_output(
264
)
265
266
267
-class TestSharedStorageConnector(SharedStorageConnector):
267
+class TestExampleConnector(ExampleConnector):
268
def __init__(self, config: VllmConfig, role, kv_cache_config):
269
self.name = config.kv_transfer_config.kv_connector_extra_config["name"]
270
- self._connector = SharedStorageConnector(config, role)
270
+ self._connector = ExampleConnector(config, role)
271
self.call_record: dict[str, int] = defaultdict(int)
272
# Use a unique temp file per connector
273
self._event_file = (
@@ -394,7 +394,7 @@ class MockKVConnector(KVConnectorBase_V1):
394
395
396
KVConnectorFactory.register_connector(
397
- "TestSharedStorageConnector", __name__, TestSharedStorageConnector.__name__
397
+ "TestExampleConnector", __name__, TestExampleConnector.__name__
398
)
399
400
KVConnectorFactory.register_connector(
vllm/distributed/ec_transfer/ec_connector/example_connector.py
renamed
+4
-4
@@ -32,7 +32,7 @@ class MMMeta:
32
33
34
@dataclass
35
-class ECSharedStorageConnectorMetadata(ECConnectorMetadata):
35
+class ECExampleConnectorMetadata(ECConnectorMetadata):
36
mm_datas: list[MMMeta]
37
38
def __init__(self):
@@ -42,7 +42,7 @@ class ECSharedStorageConnectorMetadata(ECConnectorMetadata):
42
self.mm_datas.append(mm_data)
43
44
45
-class ECSharedStorageConnector(ECConnectorBase):
45
+class ECExampleConnector(ECConnectorBase):
46
# NOTE: This is Simple debug implementation of the EC connector.
47
# It save / load the EC cache to / from the disk.
48
@@ -76,7 +76,7 @@ class ECSharedStorageConnector(ECConnectorBase):
76
77
# Get the metadata
78
metadata: ECConnectorMetadata = self._get_connector_metadata()
79
- assert isinstance(metadata, ECSharedStorageConnectorMetadata)
79
+ assert isinstance(metadata, ECExampleConnectorMetadata)
80
assert encoder_cache is not None
81
if metadata is None:
82
logger.warning(
@@ -160,7 +160,7 @@ class ECSharedStorageConnector(ECConnectorBase):
160
Args:
161
scheduler_output (SchedulerOutput): the scheduler output object.
162
"""
163
- meta = ECSharedStorageConnectorMetadata()
163
+ meta = ECExampleConnectorMetadata()
164
for mm_hash, num_encoder_token in self._mm_datas_need_loads.items():
165
meta.add_mm_data(MMMeta.make_meta(mm_hash, num_encoder_token))
166
self._mm_datas_need_loads.clear()
vllm/distributed/ec_transfer/ec_connector/factory.py
+3
-3
@@ -79,7 +79,7 @@ class ECConnectorFactory:
79
# only load the files corresponding to the current connector.
80
81
ECConnectorFactory.register_connector(
82
- "ECSharedStorageConnector",
83
- "vllm.distributed.ec_transfer.ec_connector.shared_storage_connector",
84
- "ECSharedStorageConnector",
82
+ "ECExampleConnector",
83
+ "vllm.distributed.ec_transfer.ec_connector.example_connector",
84
+ "ECExampleConnector",
85
)
vllm/distributed/kv_transfer/kv_connector/factory.py
+3
-3
@@ -144,9 +144,9 @@ class KVConnectorFactory:
144
# only load the files corresponding to the current connector.
145
146
KVConnectorFactory.register_connector(
147
- "SharedStorageConnector",
148
- "vllm.distributed.kv_transfer.kv_connector.v1.shared_storage_connector",
149
- "SharedStorageConnector",
147
+ "ExampleConnector",
148
+ "vllm.distributed.kv_transfer.kv_connector.v1.example_connector",
149
+ "ExampleConnector",
150
)
151
152
KVConnectorFactory.register_connector(
vllm/distributed/kv_transfer/kv_connector/v1/example_connector.py
renamed
+5
-5
@@ -65,7 +65,7 @@ class ReqMeta:
65
66
67
@dataclass
68
-class SharedStorageConnectorMetadata(KVConnectorMetadata):
68
+class ExampleConnectorMetadata(KVConnectorMetadata):
69
requests: list[ReqMeta] = field(default_factory=list)
70
71
def add_request(
@@ -81,7 +81,7 @@ class SharedStorageConnectorMetadata(KVConnectorMetadata):
81
)
82
83
84
-class SharedStorageConnector(KVConnectorBase_V1):
84
+class ExampleConnector(KVConnectorBase_V1):
85
# NOTE: This is Simple debug implementation of the KV connector.
86
# It save / load the KV cache to / from the disk.
87
# It does extra work which will overwrite the existing prefix-cache in GPU
@@ -157,7 +157,7 @@ class SharedStorageConnector(KVConnectorBase_V1):
157
158
# Get the metadata
159
metadata: KVConnectorMetadata = self._get_connector_metadata()
160
- assert isinstance(metadata, SharedStorageConnectorMetadata)
160
+ assert isinstance(metadata, ExampleConnectorMetadata)
161
162
if metadata is None:
163
logger.warning(
@@ -241,7 +241,7 @@ class SharedStorageConnector(KVConnectorBase_V1):
241
return layer.reshape(2, num_pages * page_size, -1)[:, slot_mapping, ...]
242
243
connector_metadata = self._get_connector_metadata()
244
- assert isinstance(connector_metadata, SharedStorageConnectorMetadata)
244
+ assert isinstance(connector_metadata, ExampleConnectorMetadata)
245
for request in connector_metadata.requests:
246
if request.is_store:
247
filename = self._generate_filename_debug(
@@ -315,7 +315,7 @@ class SharedStorageConnector(KVConnectorBase_V1):
315
Args:
316
scheduler_output (SchedulerOutput): the scheduler output object.
317
"""
318
- meta = SharedStorageConnectorMetadata()
318
+ meta = ExampleConnectorMetadata()
319
320
total_need_load = 0
321
for new_req in scheduler_output.scheduled_new_reqs: