A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Core] Store only the keys for multi-modal data in P0 (#22198)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Aug 7, 2025 at 16:45 UTC
766bc8162cb37ad32605eee051d4f049ec325926
17 files changed
+320
-229
docs/configuration/conserving_memory.md
+15
-17
@@ -86,7 +86,7 @@ llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct",
86
87
If you run out of CPU RAM, try the following options:
88
89
-- (Multi-modal models only) you can set the size of multi-modal input cache using `VLLM_MM_INPUT_CACHE_GIB` environment variable (default 4 GiB).
89
+- (Multi-modal models only) you can set the size of multi-modal processor cache using `VLLM_MM_INPUT_CACHE_GIB` environment variable (default 4 GiB per API process + 4 GiB per engine core process)
90
- (CPU backend only) you can set the size of KV cache using `VLLM_CPU_KVCACHE_SPACE` environment variable (default 4 GiB).
91
92
## Multi-modal input limits
@@ -129,20 +129,18 @@ reduce the size of the processed multi-modal inputs, which in turn saves memory.
129
130
Here are some examples:
131
132
-??? code
133
-
134
- ```python
135
- from vllm import LLM
132
+```python
133
+from vllm import LLM
134
137
- # Available for Qwen2-VL series models
138
- llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
139
- mm_processor_kwargs={
140
- "max_pixels": 768 * 768, # Default is 1280 * 28 * 28
141
- })
142
-
143
- # Available for InternVL series models
144
- llm = LLM(model="OpenGVLab/InternVL2-2B",
145
- mm_processor_kwargs={
146
- "max_dynamic_patch": 4, # Default is 12
147
- })
148
- ```
135
+# Available for Qwen2-VL series models
136
+llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
137
+ mm_processor_kwargs={
138
+ "max_pixels": 768 * 768, # Default is 1280 * 28 * 28
139
+ })
140
+
141
+# Available for InternVL series models
142
+llm = LLM(model="OpenGVLab/InternVL2-2B",
143
+ mm_processor_kwargs={
144
+ "max_dynamic_patch": 4, # Default is 12
145
+ })
146
+```
docs/configuration/optimization.md
+29
-44
@@ -2,6 +2,9 @@
2
3
This guide covers optimization strategies and performance tuning for vLLM V1.
4
5
+!!! tip
6
+ Running out of memory? Consult [this guide](./conserving_memory.md) on how to conserve memory.
7
+
8
## Preemption
9
10
Due to the auto-regressive nature of transformer architecture, there are times when KV cache space is insufficient to handle all batched requests.
@@ -126,62 +129,44 @@ Data parallelism replicates the entire model across multiple GPU sets and proces
129
Data parallelism can be combined with the other parallelism strategies and is set by `data_parallel_size=N`.
130
Note that MoE layers will be sharded according to the product of the tensor parallel size and data parallel size.
131
129
-## Reducing Memory Usage
130
-
131
-If you encounter out-of-memory issues, consider these strategies:
132
+## Input Processing
133
133
-### Context Length and Batch Size
134
+### Parallel Processing
135
135
-You can reduce memory usage by limiting the context length and batch size:
136
+You can run input processing in parallel via [API server scale-out](../serving/data_parallel_deployment.md#internal-load-balancing).
137
+This is useful when input processing (which is run inside the API server)
138
+becomes a bottleneck compared to model execution (which is run inside engine core)
139
+and you have excess CPU capacity.
140
137
-```python
138
-from vllm import LLM
141
+```console
142
+# Run 4 API processes and 1 engine core process
143
+vllm serve Qwen/Qwen2.5-VL-3B-Instruct --api-server-count 4
144
140
-llm = LLM(
141
- model="meta-llama/Llama-3.1-8B-Instruct",
142
- max_model_len=2048, # Limit context window
143
- max_num_seqs=4 # Limit batch size
144
-)
145
+# Run 4 API processes and 2 engine core processes
146
+vllm serve Qwen/Qwen2.5-VL-3B-Instruct --api-server-count 4 -dp 2
147
```
148
147
-### Adjust CUDA Graph Compilation
149
+!!! note
150
+ API server scale-out is only available for online inference.
151
149
-CUDA graph compilation in V1 uses more memory than in V0. You can reduce memory usage by adjusting the compilation level:
150
-
151
-```python
152
-from vllm import LLM
153
-from vllm.config import CompilationConfig, CompilationLevel
154
-
155
-llm = LLM(
156
- model="meta-llama/Llama-3.1-8B-Instruct",
157
- compilation_config=CompilationConfig(
158
- level=CompilationLevel.PIECEWISE,
159
- cudagraph_capture_sizes=[1, 2, 4, 8] # Capture fewer batch sizes
160
- )
161
-)
162
-```
152
+!!! note
153
+ [Multi-modal processor cache](#processor-cache) is disabled when API server scale-out is enabled
154
+ because it requires a one-to-one correspondance between API and engine core processes.
155
164
-Or, if you are not concerned about latency or overall performance, disable CUDA graph compilation entirely with `enforce_eager=True`:
156
+## Multi-Modal Caching
157
166
-```python
167
-from vllm import LLM
158
+### Processor Cache
159
169
-llm = LLM(
170
- model="meta-llama/Llama-3.1-8B-Instruct",
171
- enforce_eager=True # Disable CUDA graph compilation
172
-)
173
-```
160
+By default, the multi-modal processor cache is enabled to avoid repeatedly processing
161
+the same multi-modal inputs via Hugging Face `AutoProcessor`,
162
+which commonly occurs in multi-turn conversations.
163
175
-### Multimodal Models
164
+You can adjust the size of the cache via `VLLM_MM_INPUT_CACHE_GIB` environment variable
165
+(default 4 GiB per API process + 4 GiB per engine core process).
166
177
-For multi-modal models, you can reduce memory usage by limiting the number of images/videos per request:
167
+If you do not benefit much from the cache, you can disable it completely via `disable_mm_preprocessor_cache`:
168
169
```python
180
-from vllm import LLM
181
-
182
-# Accept up to 2 images per prompt
183
-llm = LLM(
184
- model="Qwen/Qwen2.5-VL-3B-Instruct",
185
- limit_mm_per_prompt={"image": 2}
186
-)
170
+llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
171
+ disable_mm_preprocessor_cache=True)
172
```
examples/offline_inference/mistral-small.py
+1
-1
@@ -166,7 +166,7 @@ def parse_args():
166
parser.add_argument(
167
"--disable-mm-preprocessor-cache",
168
action="store_true",
169
- help="If True, disables caching of multi-modal preprocessor/mapper.",
169
+ help="If True, disables caching of multi-modal processor.",
170
)
171
return parser.parse_args()
172
examples/offline_inference/vision_language.py
+1
-1
@@ -1565,7 +1565,7 @@ def parse_args():
1565
parser.add_argument(
1566
"--disable-mm-preprocessor-cache",
1567
action="store_true",
1568
- help="If True, disables caching of multi-modal preprocessor/mapper.",
1568
+ help="If True, disables caching of multi-modal processor.",
1569
)
1570
1571
parser.add_argument(
tests/models/utils.py
+3
-2
@@ -9,7 +9,7 @@ import torch
9
import torch.nn.functional as F
10
from transformers import PretrainedConfig
11
12
-from vllm.config import ModelConfig, RunnerOption
12
+from vllm.config import ModelConfig, ModelDType, RunnerOption
13
from vllm.inputs import InputContext
14
from vllm.sequence import Logprob, PromptLogprobs, SampleLogprobs
15
@@ -257,7 +257,7 @@ def check_logprobs_close(
257
def build_model_context(
258
model_id: str,
259
runner: RunnerOption = "auto",
260
- dtype: Union[str, torch.dtype] = "auto",
260
+ dtype: ModelDType = "auto",
261
model_config_kwargs: Optional[dict[str, Any]] = None,
262
mm_processor_kwargs: Optional[dict[str, Any]] = None,
263
limit_mm_per_prompt: Optional[dict[str, int]] = None,
@@ -279,6 +279,7 @@ def build_model_context(
279
model_info.check_transformers_version(on_fail="skip")
280
281
model_config_kwargs = model_config_kwargs or {}
282
+ limit_mm_per_prompt = limit_mm_per_prompt or {}
283
model_config = ModelConfig(
284
model_id,
285
runner=runner,
tests/multimodal/test_cache.py
new
+51
@@ -0,0 +1,51 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+import pytest
4
+import torch
5
+
6
+from vllm.multimodal.cache import MultiModalCache, MultiModalCacheItemMetadata
7
+from vllm.multimodal.inputs import (MultiModalFieldElem, MultiModalKwargs,
8
+ MultiModalKwargsItem,
9
+ MultiModalSharedField)
10
+
11
+
12
+def _dummy_elem(modality: str, key: str, size: int):
13
+ return MultiModalFieldElem(
14
+ modality=modality,
15
+ key=key,
16
+ data=torch.empty((size, ), dtype=torch.int8),
17
+ field=MultiModalSharedField(1),
18
+ )
19
+
20
+
21
+def _dummy_item(modality: str, size_by_key: dict[str, int]):
22
+ return MultiModalKwargsItem.from_elems([
23
+ _dummy_elem(modality, key, size) for key, size in size_by_key.items()
24
+ ])
25
+
26
+
27
+def _dummy_kw(size_by_key_modality: dict[str, dict[str, int]]):
28
+ return MultiModalKwargs.from_items([
29
+ _dummy_item(modality, size_by_key)
30
+ for modality, size_by_key in size_by_key_modality.items()
31
+ ])
32
+
33
+
34
+# yapf: disable
35
+@pytest.mark.parametrize(
36
+ ("item", "expected_size"),
37
+ [
38
+ (_dummy_item("a", {"a1": 100}), 100),
39
+ (_dummy_item("a", {"a1": 100, "a2": 110}), 210),
40
+ (_dummy_kw({"a": {"a1": 100, "a2": 110}, "b": {"b1": 120, "b2": 130}}), 460), # noqa: E501
41
+ ],
42
+)
43
+# yapf: enable
44
+def test_cache_item_size(item, expected_size):
45
+ cache = MultiModalCache.get_lru_cache(2048, type(item))
46
+
47
+ cache[""] = item
48
+ assert cache.currsize == expected_size
49
+
50
+ cache[""] = MultiModalCacheItemMetadata.wraps(item)
51
+ assert cache.currsize == expected_size
tests/multimodal/test_processing.py
+2
-46
@@ -6,20 +6,15 @@ from typing import Optional, cast
6
7
import numpy as np
8
import pytest
9
-import torch
9
10
from vllm.config import ModelConfig
11
from vllm.inputs import InputProcessingContext
12
from vllm.multimodal import MULTIMODAL_REGISTRY
14
-from vllm.multimodal.inputs import (MultiModalFieldElem, MultiModalKwargs,
15
- MultiModalKwargsItem,
16
- MultiModalSharedField)
13
# yapf conflicts with isort for this block
14
# yapf: disable
15
from vllm.multimodal.processing import (PlaceholderFeaturesInfo,
20
- ProcessingCache, PromptIndexTargets,
21
- PromptInsertion, PromptReplacement,
22
- apply_text_matches,
16
+ PromptIndexTargets, PromptInsertion,
17
+ PromptReplacement, apply_text_matches,
18
apply_token_matches,
19
find_mm_placeholders,
20
find_text_matches, find_token_matches,
@@ -902,45 +897,6 @@ def test_find_mm_placeholders(
897
assert result == expected
898
899
905
-def _dummy_elem(modality: str, key: str, size: int):
906
- return MultiModalFieldElem(
907
- modality=modality,
908
- key=key,
909
- data=torch.empty((size, ), dtype=torch.int8),
910
- field=MultiModalSharedField(1),
911
- )
912
-
913
-
914
-def _dummy_item(modality: str, size_by_key: dict[str, int]):
915
- return MultiModalKwargsItem.from_elems([
916
- _dummy_elem(modality, key, size) for key, size in size_by_key.items()
917
- ])
918
-
919
-
920
-def _dummy_kw(size_by_key_modality: dict[str, dict[str, int]]):
921
- return MultiModalKwargs.from_items([
922
- _dummy_item(modality, size_by_key)
923
- for modality, size_by_key in size_by_key_modality.items()
924
- ])
925
-
926
-
927
-# yapf: disable
928
-@pytest.mark.parametrize(
929
- ("item", "expected_size"),
930
- [
931
- (_dummy_item("a", {"a1": 100}), 100),
932
- (_dummy_item("a", {"a1": 100, "a2": 110}), 210),
933
- (_dummy_kw({"a": {"a1": 100, "a2": 110}, "b": {"b1": 120, "b2": 130}}), 460), # noqa: E501
934
- ],
935
-)
936
-# yapf: enable
937
-def test_cache_item_size(item, expected_size):
938
- cache = ProcessingCache.get_lru_cache(2048, type(item))
939
- cache[""] = item
940
-
941
- assert cache.currsize == expected_size
942
-
943
-
900
@pytest.mark.parametrize("model_id", ["llava-hf/llava-v1.6-mistral-7b-hf"])
901
@pytest.mark.parametrize(
902
("limit", "num_supported", "is_valid"),
vllm/config.py
+27
-3
@@ -444,8 +444,7 @@ class ModelConfig:
444
model that is being run. For example, for Phi-3-Vision: `{"num_crops": 4}`.
445
"""
446
disable_mm_preprocessor_cache: bool = False
447
- """If `True`, disable caching of the multi-modal preprocessor/mapper (not
448
- recommended)."""
447
+ """If `True`, disable caching of the multi-modal processor."""
448
override_neuron_config: dict[str, Any] = field(default_factory=dict)
449
"""Initialize non-default neuron config or override default neuron config
450
that are specific to Neuron devices, this argument will be used to
@@ -1692,6 +1691,31 @@ class ModelConfig:
1691
def is_multimodal_model(self) -> bool:
1692
return self.multimodal_config is not None
1693
1694
+ @property
1695
+ def processor_return_mm_hashes(self) -> bool:
1696
+ """Whether the multi-modal processor should output hashes."""
1697
+ mm_config = self.multimodal_config
1698
+ if mm_config is None:
1699
+ return False
1700
+
1701
+ return not mm_config.disable_mm_preprocessor_cache
1702
+
1703
+ @property
1704
+ def enable_mm_input_cache(self) -> bool:
1705
+ """Whether the multi-modal input cache should be enabled."""
1706
+ mm_config = self.multimodal_config
1707
+ if mm_config is None:
1708
+ return False
1709
+
1710
+ return not mm_config.disable_mm_preprocessor_cache
1711
+
1712
+ def get_mm_input_cache_gb(self) -> int:
1713
+ mm_config = self.multimodal_config
1714
+ if mm_config is None:
1715
+ return 0
1716
+
1717
+ return envs.VLLM_MM_INPUT_CACHE_GIB
1718
+
1719
@property
1720
def is_cross_encoder(self) -> bool:
1721
return (self._model_info.supports_cross_encoding
@@ -3369,7 +3393,7 @@ class MultiModalConfig:
3393
3394
disable_mm_preprocessor_cache: bool = False
3395
"""
3372
- If `True`, disable caching of the processed multi-modal inputs.
3396
+ If `True`, disable caching of the multi-modal processor.
3397
"""
3398
3399
interleave_mm_strings: bool = False
vllm/engine/arg_utils.py
+11
-11
@@ -1230,17 +1230,17 @@ class EngineArgs:
1230
enable_multimodal_encoder_data_parallel,
1231
)
1232
1233
- supports_mm_preprocessor_cache = (self.data_parallel_size == 1
1234
- or data_parallel_external_lb)
1235
- if (not supports_mm_preprocessor_cache
1236
- and model_config.is_multimodal_model
1237
- and not model_config.disable_mm_preprocessor_cache):
1238
- logger.warning(
1239
- "Multi-modal preprocessor cache is not compatible "
1240
- "with data parallelism when there does not exist a "
1241
- "one-to-one correspondance between API process and "
1242
- "EngineCore process, so the cache will be disabled.")
1243
- model_config.set_disable_mm_preprocessor_cache(True)
1233
+ if model_config.is_multimodal_model:
1234
+ dp_supports_mm_processor_cache = (self.data_parallel_size == 1
1235
+ or data_parallel_external_lb)
1236
+ if (not dp_supports_mm_processor_cache
1237
+ and not model_config.disable_mm_preprocessor_cache):
1238
+ logger.warning(
1239
+ "Multi-modal processor cache is disabled because "
1240
+ "it is not compatible with data parallelism when "
1241
+ "there does not exist a one-to-one correspondance "
1242
+ "between API and engine core processes.")
1243
+ model_config.set_disable_mm_preprocessor_cache(True)
1244
1245
speculative_config = self.create_speculative_config(
1246
target_model_config=model_config,
vllm/entrypoints/cli/serve.py
+2
-3
@@ -163,9 +163,8 @@ def run_multi_api_server(args: argparse.Namespace):
163
164
if model_config.is_multimodal_model and not (
165
orig_disable_mm_preprocessor_cache):
166
- logger.warning(
167
- "Multi-modal preprocessor cache is not compatible "
168
- "with api_server_count > 1, so the cache will be disabled.")
166
+ logger.warning("Multi-modal processor cache is disabled because "
167
+ "it is not compatible with `api_server_count > 1`.")
168
169
executor_class = Executor.get_class(vllm_config)
170
log_stats = not engine_args.disable_log_stats
vllm/envs.py
+3
-3
@@ -65,7 +65,7 @@ if TYPE_CHECKING:
65
VLLM_AUDIO_FETCH_TIMEOUT: int = 10
66
VLLM_MAX_AUDIO_CLIP_FILESIZE_MB: int = 25
67
VLLM_VIDEO_LOADER_BACKEND: str = "opencv"
68
- VLLM_MM_INPUT_CACHE_GIB: int = 8
68
+ VLLM_MM_INPUT_CACHE_GIB: int = 4
69
VLLM_TARGET_DEVICE: str = "cuda"
70
MAX_JOBS: Optional[str] = None
71
NVCC_THREADS: Optional[str] = None
@@ -561,8 +561,8 @@ environment_variables: dict[str, Callable[[], Any]] = {
561
"VLLM_VIDEO_LOADER_BACKEND":
562
lambda: os.getenv("VLLM_VIDEO_LOADER_BACKEND", "opencv"),
563
564
- # Cache size (in GiB) for multimodal input cache
565
- # Default is 4 GiB
564
+ # Cache size (in GiB per process) for multimodal input cache
565
+ # Default is 4 GiB per API process + 4 GiB per engine core process
566
"VLLM_MM_INPUT_CACHE_GIB":
567
lambda: int(os.getenv("VLLM_MM_INPUT_CACHE_GIB", "4")),
568
vllm/multimodal/cache.py
new
+95
@@ -0,0 +1,95 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+import sys
4
+from collections.abc import Mapping
5
+from dataclasses import dataclass
6
+from typing import TypeVar, Union
7
+
8
+import torch
9
+
10
+from vllm.jsontree import json_map_leaves, json_reduce_leaves
11
+from vllm.logger import init_logger
12
+from vllm.utils import GiB_bytes, LRUCache
13
+
14
+from .inputs import MultiModalKwargs, MultiModalKwargsItem, NestedTensors
15
+
16
+logger = init_logger(__name__)
17
+
18
+
19
+@dataclass
20
+class MultiModalCacheItemMetadata:
21
+ size: int
22
+
23
+ @classmethod
24
+ def wraps(cls, value: "MultiModalCacheValue"):
25
+ return cls(size=MultiModalCache.get_item_size(value))
26
+
27
+
28
+MultiModalCacheValue = Union[
29
+ MultiModalKwargs,
30
+ MultiModalKwargsItem,
31
+ Mapping[str, NestedTensors],
32
+ MultiModalCacheItemMetadata,
33
+]
34
+
35
+_V = TypeVar("_V", bound=MultiModalCacheValue)
36
+
37
+
38
+class MultiModalCache:
39
+
40
+ @classmethod
41
+ def get_leaf_size(
42
+ cls,
43
+ leaf: object,
44
+ *,
45
+ debug: bool = False,
46
+ ) -> int:
47
+ # MultiModalKwargs is not a subclass of dict
48
+ if isinstance(leaf, MultiModalKwargs):
49
+ return cls.get_item_size(leaf.data, debug=debug)
50
+
51
+ # MultiModalKwargsItem is not a subclass of dict
52
+ if isinstance(leaf, MultiModalKwargsItem):
53
+ leaf_data = {k: v.data for k, v in leaf.items()}
54
+ return cls.get_item_size(leaf_data, debug=debug)
55
+
56
+ # sys.getsizeof doesn't work for tensors
57
+ if isinstance(leaf, torch.Tensor):
58
+ return leaf.nbytes
59
+
60
+ if isinstance(leaf, MultiModalCacheItemMetadata):
61
+ return leaf.size
62
+
63
+ return sys.getsizeof(leaf)
64
+
65
+ @classmethod
66
+ def get_item_size(
67
+ cls,
68
+ value: MultiModalCacheValue,
69
+ *,
70
+ debug: bool = False,
71
+ ) -> int:
72
+ size = json_reduce_leaves(
73
+ lambda a, b: a + b,
74
+ json_map_leaves(lambda x: cls.get_leaf_size(x, debug=debug),
75
+ value),
76
+ )
77
+
78
+ if debug:
79
+ logger.debug("Calculated size of %s to be %.2f GiB", type(value),
80
+ size / GiB_bytes)
81
+
82
+ return size
83
+
84
+ @classmethod
85
+ def get_lru_cache(
86
+ cls,
87
+ capacity_gb: float,
88
+ value_type: type[_V],
89
+ *,
90
+ debug: bool = False,
91
+ ) -> LRUCache[str, _V]:
92
+ return LRUCache(
93
+ GiB_bytes * capacity_gb,
94
+ getsizeof=lambda x: cls.get_item_size(x, debug=debug),
95
+ )
vllm/multimodal/processing.py
+4
-49
@@ -1,6 +1,5 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-import sys
3
from abc import ABC, abstractmethod
4
from collections import defaultdict
5
from collections.abc import (Callable, Generator, ItemsView, Iterable, Mapping,
@@ -16,16 +15,16 @@ import torch
15
from typing_extensions import assert_never
16
17
from vllm.inputs import InputProcessingContext
19
-from vllm.jsontree import json_map_leaves, json_reduce_leaves
18
from vllm.logger import init_logger
19
from vllm.transformers_utils.tokenizer import (AnyTokenizer, decode_tokens,
20
encode_tokens)
23
-from vllm.utils import GiB_bytes, LRUCache, flatten_2d_lists, full_groupby
21
+from vllm.utils import GiB_bytes, flatten_2d_lists, full_groupby
22
23
+from .cache import MultiModalCache
24
from .hasher import MultiModalHasher
25
from .inputs import (MultiModalDataDict, MultiModalEncDecInputs,
26
MultiModalFieldConfig, MultiModalInputs, MultiModalKwargs,
28
- MultiModalKwargsItem, NestedTensors, PlaceholderRange)
27
+ MultiModalKwargsItem, PlaceholderRange)
28
from .parse import (DictEmbeddingItems, EmbeddingItems, MultiModalDataItems,
29
MultiModalDataParser)
30
@@ -888,9 +887,6 @@ def find_mm_placeholders(
887
return dict(full_groupby_modality(it))
888
889
891
-_V = TypeVar("_V", bound="Union[MultiModalKwargs, MultiModalKwargsItem]")
892
-
893
-
890
class ProcessingCacheOptionalItem(NamedTuple):
891
key: str
892
value: Optional[MultiModalKwargsItem]
@@ -901,48 +897,7 @@ class ProcessingCacheItem(NamedTuple):
897
value: MultiModalKwargsItem
898
899
904
-class ProcessingCache:
905
-
906
- @staticmethod
907
- def get_lru_cache(
908
- capacity_gb: float,
909
- value_type: type[_V],
910
- *,
911
- debug: bool = False,
912
- ) -> LRUCache[str, _V]:
913
-
914
- def get_leaf_size(leaf: object) -> int:
915
- # MultiModalKwargs is not a subclass of dict
916
- if isinstance(leaf, MultiModalKwargs):
917
- return get_item_size(leaf.data)
918
-
919
- # MultiModalKwargsItem is not a subclass of dict
920
- if isinstance(leaf, MultiModalKwargsItem):
921
- leaf_data = {k: v.data for k, v in leaf.items()}
922
- return get_item_size(leaf_data)
923
-
924
- # sys.getsizeof doesn't work for tensors
925
- if isinstance(leaf, torch.Tensor):
926
- return leaf.nbytes
927
-
928
- return sys.getsizeof(leaf)
929
-
930
- def get_item_size(
931
- value: Union[MultiModalKwargs, MultiModalKwargsItem,
932
- Mapping[str, NestedTensors]]
933
- ) -> int:
934
- size = json_reduce_leaves(
935
- lambda a, b: a + b,
936
- json_map_leaves(get_leaf_size, value),
937
- )
938
-
939
- if debug:
940
- logger.debug("Calculated size of %s to be %.2f GiB",
941
- type(value), size / GiB_bytes)
942
-
943
- return size
944
-
945
- return LRUCache(GiB_bytes * capacity_gb, getsizeof=get_item_size)
900
+class ProcessingCache(MultiModalCache):
901
902
def __init__(
903
self,
vllm/v1/core/kv_cache_utils.py
+2
-2
@@ -429,8 +429,8 @@ def _gen_mm_extra_hash_keys(request: Request, start_token_idx: int,
429
if mm_positions and len(mm_positions) != len(mm_hashes):
430
raise ValueError(
431
"The number of multi-modal positions and hashes must match. This "
432
- "is likely because you do not enable MM preprocessor hashing. "
433
- "Please set disable_mm_preprocessor_cache=False.")
432
+ "is likely because you did not enable MM hashing. "
433
+ "Please set `disable_mm_preprocessor_cache=False`.")
434
435
# Note that we assume mm_positions is sorted by offset.
436
# We do not need to check all mm inputs if the start token index is out of
vllm/v1/engine/core.py
+3
-4
@@ -35,7 +35,7 @@ from vllm.v1.engine import (EngineCoreOutputs, EngineCoreRequest,
35
EngineCoreRequestType,
36
ReconfigureDistributedRequest, ReconfigureRankType,
37
UtilityOutput, UtilityResult)
38
-from vllm.v1.engine.mm_input_cache import MirroredProcessingCache
38
+from vllm.v1.engine.mm_input_cache import MultiModalInputCacheServer
39
from vllm.v1.engine.utils import EngineHandshakeMetadata, EngineZmqAddresses
40
from vllm.v1.executor.abstract import Executor
41
from vllm.v1.kv_cache_interface import KVCacheConfig
@@ -124,8 +124,7 @@ class EngineCore:
124
log_stats=self.log_stats,
125
)
126
127
- # Setup MM Input Mapper.
128
- self.mm_input_cache_server = MirroredProcessingCache(
127
+ self.mm_input_cache_server = MultiModalInputCacheServer(
128
vllm_config.model_config)
129
130
# Setup batch queue for pipeline parallelism.
@@ -413,7 +412,7 @@ class EngineCore:
412
# Note on thread safety: no race condition.
413
# `mm_input_cache_server` is reset at the end of LLMEngine init,
414
# and will only accessed in the input processing thread afterwards.
416
- request.mm_inputs = self.mm_input_cache_server.get_and_update_p1(
415
+ request.mm_inputs = self.mm_input_cache_server.get_and_update(
416
request.mm_inputs, request.mm_hashes)
417
418
req = Request.from_engine_core_request(request)
vllm/v1/engine/mm_input_cache.py
+63
-34
@@ -1,54 +1,68 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
from collections.abc import Sequence
4
-from typing import Optional
4
+from typing import TYPE_CHECKING, Optional
5
6
-from vllm.envs import VLLM_MM_INPUT_CACHE_GIB
6
from vllm.multimodal import MultiModalKwargs
8
-from vllm.multimodal.processing import ProcessingCache
7
+from vllm.multimodal.cache import MultiModalCache, MultiModalCacheItemMetadata
8
from vllm.utils import is_list_of
9
11
-# The idea of multimodal preprocessing caching is based on having a client and
10
+if TYPE_CHECKING:
11
+ from vllm.config import ModelConfig
12
+
13
+# The idea of multimodal input caching is based on having a client and
14
# a server, where the client executes in the frontend process (=P0) and the
15
# server in the core process (=P1).
16
#
15
-# -- Client:
16
-# - BaseMultiModalProcessor to process MultiModalData into MultiModalKwargs
17
-# with built-in caching functionality, with mm_hash as its identifier.
18
-# - MirroredProcessingCache to keep track of the cached entries and
19
-# determine whether to send the MultiModalKwargs to P1.
17
+# -- P0:
18
+# - BaseMultiModalProcessor calls MultiModalHasher to get the `mm_hash` of
19
+# each input multi-modal item (e.g. image),
20
+# - BaseMultiModalProcessor processes the input items into `mm_inputs`,
21
+# which are MultiModalKwargsItem instances that each correspond to an
22
+# input multi-modal item.
23
+# - MultiModalInputCacheClient accepts the `mm_inputs` and corresponding
24
+# `mm_hash` for each item. It stores the `mm_hash` as keys and the size
25
+# of `mm_inputs`, but not the `mm_inputs` themselves, to avoid taking
26
+# up additional memory in P0.
27
+# - The `mm_hash` is always sent to P1.
28
+# - The corresponding `mm_inputs` are only sent to P1 if they are not cached
29
+# in MultiModalInputCacheServer.
30
#
21
-# -- Server:
22
-# - MirroredProcessingCache to store the MultiModalKwargs from P0.
31
+# -- P1:
32
+# - If the `mm_hash` is cached (i.e. `mm_inputs` are not sent from P0),
33
+# MultiModalInputCacheServer retrieves the corresponding `mm_inputs`.
34
+# - If the `mm_hash` is not cached (i.e. `mm_inputs` are sent from P0),
35
+# MultiModalInputCacheServer stores `mm_inputs` under the key `mm_hash`.
36
+# - Either way, the `mm_hash` and corresponding `mm_inputs` are sent to
37
+# the engine for model execution.
38
#
24
-# The caching for both client and server is mirrored, and this allows us
25
-# to avoid the serialization of "mm_inputs" (like pixel values) between
26
-# client (=P0) and server (=P1) processes if the mm_hash is found in the client
27
-# cache.
39
+# Both Client and Server must perform cache update and eviction based on the
40
+# same item size. This ensures that the keys of MultiModalInputCacheClient
41
+# and MultiModalInputCacheServer are mirrored, allowing us to determine in P0
42
+# whether a key is cached in MultiModalInputCacheServer by querying
43
+# MultiModalInputCacheClient without having to communicate with P1.
44
29
-# Both Client and Server must use the same cache size
30
-# (to perform mirrored caching). This cache size is set by the environment
31
-# variable VLLM_MM_INPUT_CACHE_GIB.
45
46
+class MultiModalInputCacheClient:
47
+ """Used by P0 to check whether multi-modal kwargs are cached in P1."""
48
34
-class MirroredProcessingCache:
49
+ def __init__(self, model_config: "ModelConfig") -> None:
50
+ super().__init__()
51
36
- def __init__(self, model_config):
37
- mm_config = model_config.multimodal_config
38
- disable_mm_preprocessor_cache = (
39
- mm_config is not None and mm_config.disable_mm_preprocessor_cache)
40
- self.use_cache = not disable_mm_preprocessor_cache
41
- self.mm_cache = ProcessingCache.get_lru_cache(VLLM_MM_INPUT_CACHE_GIB,
42
- MultiModalKwargs)
52
+ self.enabled = model_config.enable_mm_input_cache
53
+ self.mm_cache = MultiModalCache.get_lru_cache(
54
+ model_config.get_mm_input_cache_gb(),
55
+ MultiModalCacheItemMetadata,
56
+ )
57
44
- def get_and_update_p0(
58
+ def get_and_update(
59
self,
60
mm_inputs: Sequence[MultiModalKwargs],
61
mm_hashes: list[str],
62
) -> Sequence[Optional[MultiModalKwargs]]:
63
assert len(mm_inputs) == len(mm_hashes)
64
51
- if not self.use_cache:
65
+ if not self.enabled:
66
assert is_list_of(mm_inputs, MultiModalKwargs)
67
return mm_inputs
68
@@ -57,20 +71,37 @@ class MirroredProcessingCache:
71
if self.mm_cache.get(mm_hash) is not None:
72
mm_input = None
73
else:
60
- self.mm_cache[mm_hash] = mm_input
74
+ self.mm_cache[mm_hash] = \
75
+ MultiModalCacheItemMetadata.wraps(mm_input)
76
77
full_mm_inputs.append(mm_input)
78
79
return full_mm_inputs
80
66
- def get_and_update_p1(
81
+ def reset(self) -> None:
82
+ self.mm_cache.clear()
83
+
84
+
85
+class MultiModalInputCacheServer:
86
+ """Used by P1 to avoid requiring past multi-modal kwargs from P0."""
87
+
88
+ def __init__(self, model_config: "ModelConfig") -> None:
89
+ super().__init__()
90
+
91
+ self.enabled = model_config.enable_mm_input_cache
92
+ self.mm_cache = MultiModalCache.get_lru_cache(
93
+ model_config.get_mm_input_cache_gb(),
94
+ MultiModalKwargs,
95
+ )
96
+
97
+ def get_and_update(
98
self,
99
mm_inputs: Sequence[Optional[MultiModalKwargs]],
100
mm_hashes: list[str],
101
) -> Sequence[MultiModalKwargs]:
102
assert len(mm_inputs) == len(mm_hashes)
103
73
- if not self.use_cache:
104
+ if not self.enabled:
105
assert is_list_of(mm_inputs, MultiModalKwargs)
106
return mm_inputs
107
@@ -85,7 +116,5 @@ class MirroredProcessingCache:
116
117
return full_mm_inputs
118
88
- def reset(self) -> bool:
119
+ def reset(self) -> None:
120
self.mm_cache.clear()
90
-
91
- return True
vllm/v1/engine/processor.py
+8
-9
@@ -19,7 +19,7 @@ from vllm.pooling_params import PoolingParams
19
from vllm.sampling_params import SamplingParams
20
from vllm.transformers_utils.tokenizer_group import TokenizerGroup
21
from vllm.v1.engine import EngineCoreRequest
22
-from vllm.v1.engine.mm_input_cache import MirroredProcessingCache
22
+from vllm.v1.engine.mm_input_cache import MultiModalInputCacheClient
23
from vllm.v1.structured_output.backend_guidance import (
24
validate_guidance_grammar)
25
from vllm.v1.structured_output.backend_outlines import (
@@ -50,11 +50,8 @@ class Processor:
50
self.tokenizer,
51
mm_registry)
52
53
- self.mm_input_cache_client = MirroredProcessingCache(self.model_config)
54
-
55
- # Multi-modal hasher (for images)
56
- self.use_hash = self.mm_input_cache_client.use_cache or \
57
- self.cache_config.enable_prefix_caching
53
+ self.mm_input_cache_client = MultiModalInputCacheClient(
54
+ self.model_config)
55
56
@property
57
def mm_registry(self):
@@ -256,11 +253,13 @@ class Processor:
253
# 1. Tokenize text prompt, with LoRA request if one exists.
254
# 2. For multimodal models with a merged preprocessor, preprocess
255
# multimodal data and expand prompt token ids accordingly.
256
+ return_mm_hashes = (self.model_config.processor_return_mm_hashes
257
+ or bool(self.cache_config.enable_prefix_caching))
258
processed_inputs: ProcessorInputs = self.input_preprocessor.preprocess(
259
prompt,
260
tokenization_kwargs=tokenization_kwargs,
261
lora_request=lora_request,
263
- return_mm_hashes=self.use_hash,
262
+ return_mm_hashes=return_mm_hashes,
263
)
264
from vllm.platforms import current_platform
265
current_platform.validate_request(
@@ -312,7 +311,7 @@ class Processor:
311
sorted_mm_hashes,
312
) = merge_and_sort_multimodal_metadata(
313
decoder_inputs["mm_placeholders"],
315
- decoder_inputs["mm_hashes"] if self.use_hash else None,
314
+ decoder_inputs["mm_hashes"] if return_mm_hashes else None,
315
)
316
317
# The output of merged multi-modal processor (`decoder_mm_inputs`)
@@ -339,7 +338,7 @@ class Processor:
338
]
339
340
if sorted_mm_hashes is not None:
342
- sorted_mm_inputs = self.mm_input_cache_client.get_and_update_p0(
341
+ sorted_mm_inputs = self.mm_input_cache_client.get_and_update(
342
orig_sorted_mm_inputs, sorted_mm_hashes)
343
else:
344
sorted_mm_inputs = orig_sorted_mm_inputs