A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 10:45 UTC
[OpenVINO] Enable GPU support for OpenVINO vLLM backend (#8192)
Sergey Shlyapnikov committed
Oct 3, 2024 at 01:50 UTC
f58d4fccc9b270838be438f5f0db71bea156a56d
8 files changed
+446
-107
docs/source/getting_started/openvino-installation.rst
+29
-6
@@ -3,7 +3,7 @@
3
Installation with OpenVINO
4
==========================
5
6
-vLLM powered by OpenVINO supports all LLM models from :doc:`vLLM supported models list <../models/supported_models>` and can perform optimal model serving on all x86-64 CPUs with, at least, AVX2 support. OpenVINO vLLM backend supports the following advanced vLLM features:
6
+vLLM powered by OpenVINO supports all LLM models from :doc:`vLLM supported models list <../models/supported_models>` and can perform optimal model serving on all x86-64 CPUs with, at least, AVX2 support, as well as on both integrated and discrete Intel® GPUs (`the list of supported GPUs <https://docs.openvino.ai/2024/about-openvino/release-notes-openvino/system-requirements.html#gpu>`_). OpenVINO vLLM backend supports the following advanced vLLM features:
7
8
- Prefix caching (``--enable-prefix-caching``)
9
- Chunked prefill (``--enable-chunked-prefill``)
@@ -53,34 +53,57 @@ Install from source
53
$ pip install --upgrade pip
54
$ pip install -r requirements-build.txt --extra-index-url https://download.pytorch.org/whl/cpu
55
56
-- Finally, install vLLM with OpenVINO backend:
56
+- Finally, install vLLM with OpenVINO backend:
57
58
.. code-block:: console
59
60
$ PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu" VLLM_TARGET_DEVICE=openvino python -m pip install -v .
61
62
+- [Optional] To use vLLM OpenVINO backend with a GPU device, ensure your system is properly set up. Follow the instructions provided here: `https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html <https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html>`_.
63
+
64
.. _openvino_backend_performance_tips:
65
66
Performance tips
67
----------------
68
67
-vLLM OpenVINO backend uses the following environment variables to control behavior:
69
+vLLM OpenVINO backend environment variables
70
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
71
+
72
+- ``VLLM_OPENVINO_DEVICE`` to specify which device utilize for the inference. If there are multiple GPUs in the system, additional indexes can be used to choose the proper one (e.g, ``VLLM_OPENVINO_DEVICE=GPU.1``). If the value is not specified, CPU device is used by default.
73
+
74
+- ``VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON`` to enable U8 weights compression during model loading stage. By default, compression is turned off. You can also export model with different compression techniques using `optimum-cli` and pass exported folder as `<model_id>`
75
+
76
+CPU performance tips
77
+~~~~~~~~~~~~~~~~~~~~
78
+
79
+CPU uses the following environment variables to control behavior:
80
81
- ``VLLM_OPENVINO_KVCACHE_SPACE`` to specify the KV Cache size (e.g, ``VLLM_OPENVINO_KVCACHE_SPACE=40`` means 40 GB space for KV cache), larger setting will allow vLLM running more requests in parallel. This parameter should be set based on the hardware configuration and memory management pattern of users.
82
83
- ``VLLM_OPENVINO_CPU_KV_CACHE_PRECISION=u8`` to control KV cache precision. By default, FP16 / BF16 is used depending on platform.
84
73
-- ``VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON`` to enable U8 weights compression during model loading stage. By default, compression is turned off. You can also export model with different compression techniques using `optimum-cli` and pass exported folder as `<model_id>`
74
-
85
To enable better TPOT / TTFT latency, you can use vLLM's chunked prefill feature (``--enable-chunked-prefill``). Based on the experiments, the recommended batch size is ``256`` (``--max-num-batched-tokens``)
86
77
-OpenVINO best known configuration is:
87
+OpenVINO best known configuration for CPU is:
88
89
.. code-block:: console
90
91
$ VLLM_OPENVINO_KVCACHE_SPACE=100 VLLM_OPENVINO_CPU_KV_CACHE_PRECISION=u8 VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON \
92
python3 vllm/benchmarks/benchmark_throughput.py --model meta-llama/Llama-2-7b-chat-hf --dataset vllm/benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json --enable-chunked-prefill --max-num-batched-tokens 256
93
94
+GPU performance tips
95
+~~~~~~~~~~~~~~~~~~~~
96
+GPU device implements the logic for automatic detection of available GPU memory and, by default, tries to reserve as much memory as possible for the KV cache (taking into account ``gpu_memory_utilization`` option). However, this behavior can be overridden by explicitly specifying the desired amount of memory for the KV cache using ``VLLM_OPENVINO_KVCACHE_SPACE`` environment variable (e.g, ``VLLM_OPENVINO_KVCACHE_SPACE=8`` means 8 GB space for KV cache).
97
+
98
+Currently, the best performance using GPU can be achieved with the default vLLM execution parameters for models with quantized weights (8 and 4-bit integer data types are supported) and `preemption-mode=swap`.
99
+
100
+OpenVINO best known configuration for GPU is:
101
+
102
+.. code-block:: console
103
+
104
+ $ VLLM_OPENVINO_DEVICE=GPU VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON \
105
+ python3 vllm/benchmarks/benchmark_throughput.py --model meta-llama/Llama-2-7b-chat-hf --dataset vllm/benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json
106
+
107
.. _openvino_backend_limitations:
108
109
Limitations
requirements-openvino.txt
+3
-2
@@ -3,5 +3,6 @@
3
4
# OpenVINO dependencies
5
torch >= 2.1.2
6
-openvino ~= 2024.3.0
7
-optimum-intel[openvino] >= 1.18.2
6
+openvino ~= 2024.4.0
7
+openvino-tokenizers[transformers] ~= 2024.4.0
8
+optimum-intel[openvino] >= 1.19.0
vllm/attention/backends/openvino.py
+32
-8
@@ -9,6 +9,31 @@ from vllm.attention.backends.abstract import (AttentionBackend,
9
from vllm.attention.backends.utils import CommonAttentionState
10
11
12
+def copy_cache_block(src_tensor: ov.Tensor, dst_tensor: ov.Tensor,
13
+ src_offset: int, dst_offset: int) -> None:
14
+
15
+ def create_roi_tensor(
16
+ tensor: ov.Tensor,
17
+ block_number: int,
18
+ ) -> ov.Tensor:
19
+ roi_begin = ov.runtime.Coordinate([0, 0, 0, 0])
20
+ roi_end = ov.runtime.Coordinate(tensor.get_shape())
21
+
22
+ roi_begin[0] = block_number
23
+ roi_end[0] = block_number + 1
24
+
25
+ if isinstance(tensor, ov.Tensor):
26
+ return ov.Tensor(tensor, roi_begin, roi_end)
27
+ else:
28
+ return ov.RemoteTensor(tensor, roi_begin, roi_end)
29
+
30
+ src_roi_tensor = \
31
+ create_roi_tensor(src_tensor, src_offset)
32
+ dst_roi_tensor = \
33
+ create_roi_tensor(dst_tensor, dst_offset)
34
+ src_roi_tensor.copy_to(dst_roi_tensor)
35
+
36
+
37
class OpenVINOAttentionBackend(AttentionBackend):
38
39
@staticmethod
@@ -44,13 +69,12 @@ class OpenVINOAttentionBackend(AttentionBackend):
69
70
@staticmethod
71
def swap_blocks(
47
- src_kv_cache: ov.Tensor,
48
- dst_kv_cache: ov.Tensor,
49
- src_to_dst: torch.Tensor,
72
+ src_tensor: ov.Tensor,
73
+ dst_tensor: ov.Tensor,
74
+ src_to_dists: List[Tuple[int, int]],
75
) -> None:
51
- # OpenVINO currently supports only CPU, which does not require
52
- # swap of KV cache blocks
53
- raise NotImplementedError
76
+ for src, dst in src_to_dists:
77
+ copy_cache_block(src_tensor, dst_tensor, src, dst)
78
79
@staticmethod
80
def copy_blocks(
@@ -59,8 +83,8 @@ class OpenVINOAttentionBackend(AttentionBackend):
83
) -> None:
84
for src, dst in src_to_dists:
85
for key_cache, value_cache in kv_caches:
62
- key_cache.data[dst, :] = key_cache.data[src, :]
63
- value_cache.data[dst, :] = value_cache.data[src, :]
86
+ copy_cache_block(key_cache, key_cache, src, dst)
87
+ copy_cache_block(value_cache, value_cache, src, dst)
88
89
90
@dataclass
vllm/envs.py
+6
@@ -35,6 +35,7 @@ if TYPE_CHECKING:
35
VLLM_PP_LAYER_PARTITION: Optional[str] = None
36
VLLM_CPU_KVCACHE_SPACE: int = 0
37
VLLM_CPU_OMP_THREADS_BIND: str = ""
38
+ VLLM_OPENVINO_DEVICE: str = "CPU"
39
VLLM_OPENVINO_KVCACHE_SPACE: int = 0
40
VLLM_OPENVINO_CPU_KV_CACHE_PRECISION: Optional[str] = None
41
VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS: bool = False
@@ -302,6 +303,11 @@ environment_variables: Dict[str, Callable[[], Any]] = {
303
"VLLM_CPU_OMP_THREADS_BIND":
304
lambda: os.getenv("VLLM_CPU_OMP_THREADS_BIND", "all"),
305
306
+ # OpenVINO device selection
307
+ # default is CPU
308
+ "VLLM_OPENVINO_DEVICE":
309
+ lambda: os.getenv("VLLM_OPENVINO_DEVICE", "CPU").upper(),
310
+
311
# OpenVINO key-value cache space
312
# default is 4GB
313
"VLLM_OPENVINO_KVCACHE_SPACE":
vllm/executor/openvino_executor.py
+53
-20
@@ -17,6 +17,14 @@ from vllm.utils import (GiB_bytes, get_distributed_init_method, get_ip,
17
logger = init_logger(__name__)
18
19
20
+def is_openvino_cpu() -> bool:
21
+ return "CPU" in envs.VLLM_OPENVINO_DEVICE
22
+
23
+
24
+def is_openvino_gpu() -> bool:
25
+ return "GPU" in envs.VLLM_OPENVINO_DEVICE
26
+
27
+
28
class OpenVINOExecutor(ExecutorBase):
29
30
uses_ray: bool = False
@@ -24,8 +32,13 @@ class OpenVINOExecutor(ExecutorBase):
32
def _init_executor(self) -> None:
33
assert self.device_config.device_type == "openvino"
34
assert self.lora_config is None, "OpenVINO backend doesn't support LoRA"
35
+ assert is_openvino_cpu() or is_openvino_gpu(), \
36
+ "OpenVINO backend supports only CPU and GPU devices"
37
+
38
+ self.ov_core = ov.Core()
39
self.model_config = _verify_and_get_model_config(self.model_config)
28
- self.cache_config = _verify_and_get_cache_config(self.cache_config)
40
+ self.cache_config = _verify_and_get_cache_config(
41
+ self.ov_core, self.cache_config)
42
43
# Instantiate the worker and load the model to CPU.
44
self._init_worker()
@@ -40,6 +53,7 @@ class OpenVINOExecutor(ExecutorBase):
53
distributed_init_method = get_distributed_init_method(
54
get_ip(), get_open_port())
55
self.driver_worker = OpenVINOWorker(
56
+ ov_core=self.ov_core,
57
model_config=self.model_config,
58
parallel_config=self.parallel_config,
59
scheduler_config=self.scheduler_config,
@@ -68,10 +82,13 @@ class OpenVINOExecutor(ExecutorBase):
82
# NOTE: We log here to avoid multiple logs when number of workers is
83
# greater than one. We could log in the engine, but not all executors
84
# have GPUs.
71
- # NOTE: `cpu block` for OpenVINO backend is located on CPU memory but is
72
- # referred as `gpu block`. Because we want to reuse the existing block
73
- # management procedure.
74
- logger.info("# CPU blocks: %d", num_gpu_blocks)
85
+ # NOTE: In case of a CPU device, `cpu block` for OpenVINO backend
86
+ # is located on CPU memory but is referred as `gpu block`.
87
+ # Because we want to reuse the existing block management procedure.
88
+ device_blocks = num_gpu_blocks
89
+ swap_blocks = num_cpu_blocks
90
+ logger.info("OpenVINO %s: # device blocks: %d; # swap blocks: %d",
91
+ envs.VLLM_OPENVINO_DEVICE, device_blocks, swap_blocks)
92
self.driver_worker.initialize_cache(num_gpu_blocks, num_cpu_blocks)
93
94
def execute_model(
@@ -143,29 +160,45 @@ def _verify_and_get_model_config(config: ModelConfig) -> ModelConfig:
160
return config
161
162
146
-def _verify_and_get_cache_config(config: CacheConfig) -> CacheConfig:
163
+def _verify_and_get_cache_config(ov_core: ov.Core,
164
+ config: CacheConfig) -> CacheConfig:
165
if envs.VLLM_OPENVINO_CPU_KV_CACHE_PRECISION == "u8":
148
- logger.info("KV cache type is overried to u8 via "
149
- "VLLM_OPENVINO_CPU_KV_CACHE_PRECISION env var.")
150
- config.cache_dtype = ov.Type.u8
166
+ if not is_openvino_cpu():
167
+ logger.info("VLLM_OPENVINO_CPU_KV_CACHE_PRECISION is"
168
+ "ignored for GPU, f16 data type will be used.")
169
+ config.cache_dtype = ov.Type.f16
170
+ else:
171
+ logger.info("KV cache type is overridden to u8 via "
172
+ "VLLM_OPENVINO_CPU_KV_CACHE_PRECISION env var.")
173
+ config.cache_dtype = ov.Type.u8
174
else:
152
- core = ov.Core()
153
- inference_precision = core.get_property("CPU",
154
- hints.inference_precision)
155
- if inference_precision == ov.Type.bf16:
156
- config.cache_dtype = ov.Type.bf16
175
+ if is_openvino_cpu():
176
+ ov_device = envs.VLLM_OPENVINO_DEVICE
177
+ inference_precision = ov_core.get_property(
178
+ ov_device, hints.inference_precision)
179
+ if inference_precision == ov.Type.bf16:
180
+ config.cache_dtype = ov.Type.bf16
181
+ else:
182
+ config.cache_dtype = ov.Type.f16
183
else:
184
config.cache_dtype = ov.Type.f16
185
160
- if config.block_size != 32:
161
- logger.info(
162
- f"OpenVINO optimal block size is 32, overriding currently set {config.block_size}" # noqa: G004, E501
163
- )
164
- config.block_size = 32
186
+ if is_openvino_cpu():
187
+ if config.block_size != 32:
188
+ logger.info(
189
+ f"OpenVINO CPU optimal block size is 32, overriding currently set {config.block_size}" # noqa: G004, E501
190
+ )
191
+ config.block_size = 32
192
+ else:
193
+ if config.block_size != 16:
194
+ logger.info(
195
+ f"OpenVINO GPU optimal block size is 16, overriding currently set {config.block_size}" # noqa: G004, E501
196
+ )
197
+ config.block_size = 16
198
199
kv_cache_space = envs.VLLM_OPENVINO_KVCACHE_SPACE
200
if kv_cache_space >= 0:
168
- if kv_cache_space == 0:
201
+ if kv_cache_space == 0 and is_openvino_cpu():
202
config.openvino_kvcache_space_bytes = 4 * GiB_bytes # type: ignore
203
logger.warning(
204
"Environment variable VLLM_OPENVINO_KVCACHE_SPACE (GB) "
vllm/model_executor/model_loader/openvino.py
+10
-16
@@ -12,6 +12,7 @@ from torch import nn
12
import vllm.envs as envs
13
from vllm.attention.backends.openvino import OpenVINOAttentionMetadata
14
from vllm.config import DeviceConfig, ModelConfig
15
+from vllm.executor.openvino_executor import is_openvino_cpu
16
from vllm.logger import init_logger
17
from vllm.model_executor.layers.logits_processor import (LogitsProcessor,
18
_prune_hidden_states)
@@ -51,25 +52,15 @@ def _modify_cache_parameters(model: ov.Model, kv_cache_dtype: ov.Type,
52
shape = parameter.get_partial_shape()
53
# use real block size if available, just a placeholder
54
# to provide the expected rank
54
- x_size = 1
55
num_blocks = ov.Dimension()
56
block_size = ov.Dimension()
57
head_size = ov.Dimension()
58
- # TODO: Negotiate required layout with plugins (CPU is ~OK, GPU is TBD),
59
- # pass more parameters to this function to set more static dimensions
58
if input_name.startswith("key_cache."):
59
cpu_shape = [num_blocks, shape[1], block_size, head_size]
62
- gpu_shape = [
63
- num_blocks,
64
- shape[1],
65
- shape[2].get_length() //
66
- x_size if shape[2].is_static else ov.Dimension(),
67
- block_size,
68
- x_size,
69
- ]
60
+ gpu_shape = [num_blocks, shape[1], shape[2], block_size]
61
elif input_name.startswith("value_cache."):
62
cpu_shape = [num_blocks, shape[1], block_size, head_size]
72
- gpu_shape = [num_blocks, shape[1], shape[2], block_size]
63
+ gpu_shape = [num_blocks, shape[1], block_size, shape[2]]
64
else:
65
continue
66
parameter.set_partial_shape(
@@ -108,6 +99,7 @@ class OpenVINOCasualLM(nn.Module):
99
100
def __init__(
101
self,
102
+ ov_core: ov.Core,
103
model_config: ModelConfig,
104
device_config: DeviceConfig,
105
kv_cache_dtype: ov.Type,
@@ -141,12 +133,12 @@ class OpenVINOCasualLM(nn.Module):
133
trust_remote_code=model_config.trust_remote_code,
134
)
135
136
+ ov_device = envs.VLLM_OPENVINO_DEVICE
137
paged_attention_transformation(pt_model.model)
138
_modify_cache_parameters(pt_model.model, kv_cache_dtype,
146
- device_config.device.type == "cpu")
139
+ is_openvino_cpu())
140
148
- core = ov.Core()
149
- ov_compiled = core.compile_model(pt_model.model, "CPU")
141
+ ov_compiled = ov_core.compile_model(pt_model.model, ov_device)
142
self.ov_request = ov_compiled.create_infer_request()
143
144
def forward(
@@ -199,6 +191,7 @@ def get_model(
191
**kwargs,
192
) -> torch.nn.Module:
193
lora_config = kwargs.get("lora_config", None)
194
+ ov_core = kwargs.get("ov_core")
195
if lora_config:
196
raise ValueError(
197
"OpenVINO modeling does not support LoRA, "
@@ -206,4 +199,5 @@ def get_model(
199
"be added in the future. If this is important to you, "
200
"please open an issue on github.")
201
209
- return OpenVINOCasualLM(model_config, device_config, kv_cache_dtype)
202
+ return OpenVINOCasualLM(ov_core, model_config, device_config,
203
+ kv_cache_dtype)
vllm/worker/openvino_model_runner.py
+6
-5
@@ -42,6 +42,7 @@ class OpenVINOModelRunner:
42
43
def __init__(
44
self,
45
+ ov_core: ov.Core,
46
model_config: ModelConfig,
47
parallel_config: ParallelConfig,
48
scheduler_config: SchedulerConfig,
@@ -55,6 +56,7 @@ class OpenVINOModelRunner:
56
*args,
57
**kwargs,
58
):
59
+ self.ov_core = ov_core
60
self.model_config = model_config
61
self.parallel_config = parallel_config
62
self.scheduler_config = scheduler_config
@@ -89,11 +91,10 @@ class OpenVINOModelRunner:
91
self.model: nn.Module # Set after init_Model
92
93
def load_model(self) -> None:
92
- self.model = get_model(
93
- model_config=self.model_config,
94
- device_config=self.device_config,
95
- kv_cache_dtype=self.kv_cache_dtype,
96
- )
94
+ self.model = get_model(model_config=self.model_config,
95
+ device_config=self.device_config,
96
+ kv_cache_dtype=self.kv_cache_dtype,
97
+ ov_core=self.ov_core)
98
99
def _prepare_model_input(
100
self,
vllm/worker/openvino_worker.py
+307
-50
@@ -5,6 +5,7 @@ import openvino as ov
5
import torch
6
import torch.distributed
7
8
+import vllm.envs as envs
9
from vllm.attention import get_attn_backend
10
from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
11
ModelConfig, MultiModalConfig, ParallelConfig,
@@ -12,10 +13,14 @@ from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
13
from vllm.distributed import (broadcast_tensor_dict,
14
ensure_model_parallel_initialized,
15
init_distributed_environment)
16
+from vllm.executor.openvino_executor import is_openvino_cpu
17
+from vllm.inputs import INPUT_REGISTRY
18
from vllm.logger import init_logger
19
from vllm.model_executor import set_random_seed
20
from vllm.model_executor.layers.sampler import SamplerOutput
18
-from vllm.sequence import ExecuteModelRequest
21
+from vllm.multimodal import MULTIMODAL_REGISTRY
22
+from vllm.sampling_params import SamplingParams
23
+from vllm.sequence import ExecuteModelRequest, SequenceGroupMetadata
24
from vllm.worker.openvino_model_runner import OpenVINOModelRunner
25
from vllm.worker.worker_base import LoraNotSupportedWorkerBase
26
@@ -36,6 +41,8 @@ class OpenVINOCacheEngine:
41
model_config: ModelConfig,
42
parallel_config: ParallelConfig,
43
device_config: DeviceConfig,
44
+ ov_core: ov.Core,
45
+ ov_device: str,
46
) -> None:
47
assert device_config.device_type == "openvino"
48
self.cache_config = cache_config
@@ -56,9 +63,10 @@ class OpenVINOCacheEngine:
63
64
self.block_size = cache_config.block_size
65
# Note: In CacheConfig, num_gpu_blocks actual is num_cpu_blocks
59
- # for OpenVINO backend, because we want to reuse KV cache management
60
- # in the scheduler.
61
- self.num_cpu_blocks = cache_config.num_gpu_blocks
66
+ # for OpenVINO backend with a CPU target device, because we want
67
+ # to reuse KV cache management in the scheduler.
68
+ self.num_device_blocks = cache_config.num_gpu_blocks
69
+ self.num_swap_blocks = cache_config.num_cpu_blocks
70
71
# Get attention backend.
72
self.attn_backend = get_attn_backend(
@@ -74,34 +82,100 @@ class OpenVINOCacheEngine:
82
# Initialize the cache.
83
self.kv_cache: List[Tuple[ov.Tensor,
84
ov.Tensor]] = self._allocate_kv_cache(
77
- self.num_cpu_blocks)
85
+ self.num_device_blocks, ov_core,
86
+ ov_device)
87
+
88
+ # Initialize the swap.
89
+ self.swap_cache: List[Tuple[ov.Tensor,
90
+ ov.Tensor]] = self._allocate_swap_cache(
91
+ self.num_swap_blocks, ov_device)
92
93
def _allocate_kv_cache(
94
self,
95
num_blocks: int,
96
+ ov_core: ov.Core,
97
+ ov_device: str,
98
) -> List[Tuple[ov.Tensor, ov.Tensor]]:
99
"""Allocates KV cache."""
100
k_block_shape = v_block_shape = self.attn_backend.get_kv_cache_shape(
101
num_blocks, self.block_size, self.num_kv_heads, self.head_size)[1:]
102
kv_cache: List[Tuple[ov.Tensor, ov.Tensor]] = []
103
+
104
+ if is_openvino_cpu():
105
+ for _ in range(self.num_layers):
106
+ key_blocks = ov.Tensor(self.cache_config.cache_dtype,
107
+ k_block_shape)
108
+ value_blocks = ov.Tensor(self.cache_config.cache_dtype,
109
+ v_block_shape)
110
+ kv_cache.append((key_blocks, value_blocks))
111
+ else:
112
+ # Update key_cache shape:
113
+ k_block_shape = (v_block_shape[0], v_block_shape[1],
114
+ v_block_shape[3], v_block_shape[2])
115
+
116
+ remote_context = ov_core.get_default_context(ov_device)
117
+
118
+ for _ in range(self.num_layers):
119
+ key_blocks = \
120
+ remote_context.create_tensor(self.cache_config.cache_dtype,
121
+ ov.Shape(k_block_shape),
122
+ {})
123
+
124
+ value_blocks = \
125
+ remote_context.create_tensor(self.cache_config.cache_dtype,
126
+ ov.Shape(v_block_shape),
127
+ {})
128
+
129
+ kv_cache.append((key_blocks, value_blocks))
130
+
131
+ return kv_cache
132
+
133
+ def _allocate_swap_cache(
134
+ self,
135
+ num_blocks: int,
136
+ ov_device: str,
137
+ ) -> List[Tuple[ov.Tensor, ov.Tensor]]:
138
+ """Allocates swap cache."""
139
+ k_block_shape = v_block_shape = self.attn_backend.get_kv_cache_shape(
140
+ num_blocks, self.block_size, self.num_kv_heads, self.head_size)[1:]
141
+ swap_cache: List[Tuple[ov.Tensor, ov.Tensor]] = []
142
+
143
+ if num_blocks == 0:
144
+ return swap_cache
145
+
146
+ assert not is_openvino_cpu(), \
147
+ "CPU device isn't supposed to have swap cache"
148
+
149
+ # Update key_cache shape:
150
+ k_block_shape = (v_block_shape[0], v_block_shape[1], v_block_shape[3],
151
+ v_block_shape[2])
152
+
153
for _ in range(self.num_layers):
154
key_blocks = ov.Tensor(self.cache_config.cache_dtype,
155
k_block_shape)
156
value_blocks = ov.Tensor(self.cache_config.cache_dtype,
157
v_block_shape)
92
- kv_cache.append((key_blocks, value_blocks))
93
- return kv_cache
158
+ swap_cache.append((key_blocks, value_blocks))
159
+
160
+ return swap_cache
161
95
- def swap_in(self, src_to_dst: Dict[int, int]) -> None:
96
- raise NotImplementedError(
97
- "Swap is not supported in OpenVINOCacheEngine.")
162
+ def swap_in(self, src_to_dst: List[Tuple[int, int]]) -> None:
163
+ for i in range(self.num_layers):
164
+ for swap_tensor, kv_tensor in zip(self.swap_cache[i],
165
+ self.kv_cache[i]):
166
+ self.attn_backend.swap_blocks(swap_tensor, kv_tensor,
167
+ src_to_dst)
168
99
- def swap_out(self, src_to_dst: Dict[int, int]) -> None:
100
- raise NotImplementedError(
101
- "Swap is not supported in OpenVINOCacheEngine.")
169
+ def swap_out(self, src_to_dst: List[Tuple[int, int]]) -> None:
170
+ for i in range(self.num_layers):
171
+ for swap_tensor, kv_tensor in zip(self.swap_cache[i],
172
+ self.kv_cache[i]):
173
+ self.attn_backend.swap_blocks(kv_tensor, swap_tensor,
174
+ src_to_dst)
175
103
- def copy(self, src_to_dsts: Dict[int, List[int]]) -> None:
104
- self.attn_backend.copy_blocks(self.kv_cache, src_to_dsts)
176
+ def copy(self, src_to_dsts: List[Tuple[int, int]]) -> None:
177
+ if (len(src_to_dsts) > 0):
178
+ self.attn_backend.copy_blocks(self.kv_cache, src_to_dsts)
179
180
@staticmethod
181
def get_cache_block_size(
@@ -139,6 +213,7 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
213
214
def __init__(
215
self,
216
+ ov_core: ov.Core,
217
model_config: ModelConfig,
218
parallel_config: ParallelConfig,
219
scheduler_config: SchedulerConfig,
@@ -153,6 +228,7 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
228
kv_cache_dtype: Optional[ov.Type] = ov.Type.undefined,
229
is_driver_worker: bool = False,
230
) -> None:
231
+ self.ov_core = ov_core
232
self.model_config = model_config
233
self.parallel_config = parallel_config
234
self.parallel_config.rank = rank
@@ -175,6 +251,7 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
251
252
init_cached_hf_modules()
253
self.model_runner = OpenVINOModelRunner(
254
+ self.ov_core,
255
model_config,
256
parallel_config,
257
scheduler_config,
@@ -204,56 +281,69 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
281
282
This determines how many KV blocks can fit into the configured
283
KV cache space.
207
-
208
- Note that since vLLM assumes a block resides on GPU if it can be
209
- modified, we return num_gpu_blocks=num_cpu_blocks and num_cpu_blocks=0.
210
- This allows us to reuse the scheduler of vLLM without generalizing it
211
- to different devices.
284
"""
213
- # For OpenVINO backend, the block number will be calculated based on the
214
- # openvino_kvcache_space_bytes.
285
+ # For OpenVINO backend, in case of CPU device, the block number will be
286
+ # calculated based on the openvino_kvcache_space_bytes.
287
cache_block_size = self.get_cache_block_size_bytes()
216
- num_cpu_blocks = int(self.cache_config.openvino_kvcache_space_bytes //
217
- cache_block_size)
218
- num_cpu_blocks = max(num_cpu_blocks, 0)
288
+ kvcache_space_bytes = self.cache_config.openvino_kvcache_space_bytes
289
220
- # Note: To reuse the cache management procedure,
221
- # use cpu cache as 'gpu cache'.
222
- num_gpu_blocks = num_cpu_blocks
223
- num_cpu_blocks = 0
224
- return num_gpu_blocks, num_cpu_blocks
290
+ if is_openvino_cpu():
291
+ num_device_blocks = int(kvcache_space_bytes // cache_block_size)
292
+ num_swap_blocks = 0
293
+ else:
294
+ if kvcache_space_bytes > 0:
295
+ logger.info("KV_CACHE size was explicitly configured via "
296
+ "VLLM_OPENVINO_KVCACHE_SPACE environment "
297
+ "variable, ignoring profiling run.")
298
+ kv_cache_size = kvcache_space_bytes
299
+ else:
300
+ try:
301
+ kv_cache_size = self.profile_run()
302
+ except Exception as err:
303
+ raise RuntimeError(
304
+ "The error occurred during profile run. This might be "
305
+ "due to insufficient GPU memory. Consider decreasing "
306
+ "`max_model_len` to limit the maximum simultaneously "
307
+ "processed tokens.") from err
308
+
309
+ num_device_blocks = int(kv_cache_size // cache_block_size)
310
+ num_swap_blocks = int(self.cache_config.swap_space_bytes //
311
+ cache_block_size)
312
+
313
+ return num_device_blocks, num_swap_blocks
314
315
def initialize_cache(self, num_gpu_blocks: int,
316
num_cpu_blocks: int) -> None:
228
- """Initialize the KV cache. Currently, swappable CPU memory is not
229
- supported.
317
+ """Initialize the KV cache. Swappable CPU memory is only
318
+ supported on GPU.
319
231
- Since this worker does not support GPUs, we use the num_gpu_blocks to
320
+ For CPU, we use the num_gpu_blocks to
321
determine how many non-swappable CPU blocks to allocate.
322
"""
234
- assert (num_cpu_blocks == 0
235
- ), f"{type(self)} does not support swappable cache"
323
237
- # Note: To reuse the cache management procedure,
238
- # use cpu cache as 'gpu cache'.
239
- num_cpu_blocks = num_gpu_blocks
324
+ num_device_blocks = num_gpu_blocks
325
+ num_swap_blocks = num_cpu_blocks
326
+
327
+ if is_openvino_cpu():
328
+ assert (num_swap_blocks == 0
329
+ ), f"{type(self)} does not support swappable cache for CPU"
330
241
- self._validate_num_cpu_blocks(num_cpu_blocks)
242
- self.cache_config.num_gpu_blocks = num_cpu_blocks
243
- self.cache_config.num_cpu_blocks = 0
331
+ self._validate_num_blocks(num_device_blocks)
332
+ self.cache_config.num_gpu_blocks = num_device_blocks
333
+ self.cache_config.num_cpu_blocks = num_swap_blocks
334
335
# Initialize the cache.
336
self._init_cache_engine()
337
248
- def _validate_num_cpu_blocks(self, num_cpu_blocks: int) -> None:
249
- """Raise errors if the num_cpu_blocks is invalid."""
250
- if num_cpu_blocks <= 0:
338
+ def _validate_num_blocks(self, num_blocks: int) -> None:
339
+ """Raise errors if the num_blocks is invalid."""
340
+ if num_blocks <= 0:
341
raise ValueError(
342
"No available memory for the cache blocks. "
343
"Try increasing `VLLM_OPENVINO_KVCACHE_SPACE` when "
344
"initializing the engine.")
345
256
- max_seq_len = self.cache_config.block_size * num_cpu_blocks
346
+ max_seq_len = self.cache_config.block_size * num_blocks
347
if self.model_config.max_model_len > max_seq_len:
348
raise ValueError(
349
f"The model's max seq len ({self.model_config.max_model_len}) "
@@ -263,11 +353,14 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
353
"when initializing the engine.")
354
355
def _init_cache_engine(self) -> None:
356
+ ov_device = envs.VLLM_OPENVINO_DEVICE
357
self.cache_engine = OpenVINOCacheEngine(
358
self.cache_config,
359
self.model_config,
360
self.parallel_config,
361
self.device_config,
362
+ self.ov_core,
363
+ ov_device,
364
)
365
self.kv_cache = self.cache_engine.kv_cache
366
self.model_runner.block_size = self.cache_engine.block_size
@@ -275,9 +368,16 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
368
assert self.kv_cache is not None
369
370
# Populate the cache to warmup the memory
278
- for key_cache, value_cache in self.kv_cache:
279
- key_cache.data[:] = 0
280
- value_cache.data[:] = 0
371
+ if is_openvino_cpu():
372
+ for key_cache, value_cache in self.kv_cache:
373
+ key_cache.data[:] = 0
374
+ value_cache.data[:] = 0
375
+
376
+ def cache_swap_in(self, src_to_dst: List[Tuple[int, int]]) -> None:
377
+ self.cache_engine.swap_in(src_to_dst)
378
+
379
+ def cache_swap_out(self, src_to_dst: List[Tuple[int, int]]) -> None:
380
+ self.cache_engine.swap_out(src_to_dst)
381
382
def cache_copy(
383
self,
@@ -300,17 +400,28 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
400
num_seq_groups: int = len(seq_group_metadata_list)
401
assert execute_model_req is not None
402
blocks_to_copy = execute_model_req.blocks_to_copy
303
- assert len(execute_model_req.blocks_to_swap_in) == 0
304
- assert len(execute_model_req.blocks_to_swap_out) == 0
403
+ blocks_to_swap_in = execute_model_req.blocks_to_swap_in
404
+ blocks_to_swap_out = execute_model_req.blocks_to_swap_out
405
data: Dict[str, Any] = {
406
"num_seq_groups": num_seq_groups,
407
"blocks_to_copy": execute_model_req.blocks_to_copy,
408
+ "blocks_to_swap_in": execute_model_req.blocks_to_swap_in,
409
+ "blocks_to_swap_out": execute_model_req.blocks_to_swap_out,
410
}
411
broadcast_tensor_dict(data, src=0)
412
else:
413
data = broadcast_tensor_dict(src=0)
414
num_seq_groups = data["num_seq_groups"]
415
blocks_to_copy = data["blocks_to_copy"]
416
+ blocks_to_swap_in = data["blocks_to_swap_in"]
417
+ blocks_to_swap_out = data["blocks_to_swap_out"]
418
+
419
+ if is_openvino_cpu():
420
+ assert len(execute_model_req.blocks_to_swap_in) == 0
421
+ assert len(execute_model_req.blocks_to_swap_out) == 0
422
+ else:
423
+ self.cache_swap_in(blocks_to_swap_in)
424
+ self.cache_swap_out(blocks_to_swap_out)
425
426
self.cache_copy(blocks_to_copy)
427
@@ -353,3 +464,149 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
464
self.model_config,
465
self.parallel_config,
466
)
467
+
468
+ def profile_run(self) -> int:
469
+ ov_device = envs.VLLM_OPENVINO_DEVICE
470
+
471
+ assert not is_openvino_cpu(), \
472
+ "CPU device isn't supposed to use profile run."
473
+
474
+ import openvino.properties.device as device
475
+ import openvino.properties.intel_gpu as intel_gpu
476
+
477
+ ov_core = self.ov_core
478
+ cache_config = self.cache_config
479
+ model_config = self.model_config
480
+ parallel_config = self.parallel_config
481
+ device_config = self.device_config
482
+ input_registry = INPUT_REGISTRY
483
+ mm_registry = MULTIMODAL_REGISTRY
484
+ mm_registry.init_mm_limits_per_prompt(model_config)
485
+
486
+ # Execute a forward pass with dummy inputs to profile the memory usage
487
+ # of the model.
488
+ def model_profile_run():
489
+ top_k = model_config.get_vocab_size() - 1
490
+ sampling_params = SamplingParams(top_p=0.99, top_k=top_k)
491
+
492
+ max_num_batched_tokens = \
493
+ self.scheduler_config.max_num_batched_tokens
494
+ max_num_seqs = self.scheduler_config.max_num_seqs
495
+ tmp_cache_config = CacheConfig(cache_config.block_size,
496
+ cache_config.gpu_memory_utilization,
497
+ cache_config.swap_space_bytes,
498
+ "auto")
499
+ tmp_cache_config.num_gpu_blocks = 1
500
+ tmp_cache_config.num_cpu_blocks = 0
501
+ tmp_cache_config.cache_dtype = cache_config.cache_dtype
502
+
503
+ profiling_cache_engine = OpenVINOCacheEngine(
504
+ tmp_cache_config, model_config, parallel_config, device_config,
505
+ ov_core, ov_device)
506
+
507
+ # Profile memory usage with max_num_sequences sequences and the
508
+ # total # number of tokens equal to max_num_batched_tokens.
509
+ seqs: List[SequenceGroupMetadata] = []
510
+ for group_id in range(max_num_seqs):
511
+ seq_len = (max_num_batched_tokens // max_num_seqs +
512
+ (group_id < max_num_batched_tokens % max_num_seqs))
513
+ block_size = cache_config.block_size
514
+ seq_num_blocks = (seq_len + block_size - 1) // block_size
515
+
516
+ seq_data, dummy_multi_modal_data = input_registry \
517
+ .dummy_data_for_profiling(model_config,
518
+ seq_len,
519
+ mm_registry)
520
+
521
+ block_tables = [[0] * seq_num_blocks] * max_num_seqs
522
+ seq = SequenceGroupMetadata(
523
+ request_id=str(group_id),
524
+ is_prompt=True,
525
+ seq_data={group_id: seq_data},
526
+ sampling_params=sampling_params,
527
+ block_tables=block_tables,
528
+ lora_request=None,
529
+ multi_modal_data=dummy_multi_modal_data)
530
+ seqs.append(seq)
531
+
532
+ self.model_runner.block_size = tmp_cache_config.block_size
533
+
534
+ # Run the model with the dummy inputs.
535
+ self.model_runner.execute_model(seqs,
536
+ profiling_cache_engine.kv_cache)
537
+
538
+ # explicitly delete temporary KV cache manager to free KV cache
539
+ # when real inputs will be passed to OV
540
+ del profiling_cache_engine
541
+
542
+ logger.info(
543
+ "Start profiling run with dummy inputs to evaluate "
544
+ "memory usage for %s. It might take a while.", ov_device)
545
+
546
+ model_profile_run()
547
+
548
+ gpu_device_type = ov_core.get_property(ov_device, device.type)
549
+ memory_statistics = \
550
+ ov_core.get_property(ov_device, intel_gpu.memory_statistics)
551
+ memory_utilization = cache_config.gpu_memory_utilization
552
+
553
+ if gpu_device_type == device.Type.INTEGRATED and \
554
+ memory_utilization >= 0.9:
555
+ logger.warning(
556
+ "iGPU is used with high gpu_memory_utilization=%f "
557
+ "value. This may cause low performance due to "
558
+ "occupying the majority of available system "
559
+ "memory. Please consider decreasing "
560
+ "gpu_memory_utilization or explicitly setting"
561
+ "`VLLM_OPENVINO_KVCACHE_SPACE` (GB) environment "
562
+ "variable.", memory_utilization)
563
+
564
+ # sum up all used device memory
565
+ device_memory_types = ["cl_mem", "usm_device"]
566
+ used_device_mem = \
567
+ sum(memory_statistics.get(key, 0) for key in device_memory_types)
568
+
569
+ if gpu_device_type == device.Type.INTEGRATED:
570
+ used_device_mem += memory_statistics.get("usm_host", 0)
571
+
572
+ # there could be unaccounted extra memory reserved by kernels, kept
573
+ # in memory pools, etc
574
+ # therefore, add a threshold to account for this
575
+ used_memory_threshold = 1.1
576
+ used_device_mem *= used_memory_threshold
577
+
578
+ total_device_memory = \
579
+ ov_core.get_property(ov_device, intel_gpu.device_total_mem_size)
580
+
581
+ def format_memory_size(size) -> str:
582
+ units = ["B", "KB", "MB", "GB"]
583
+ unit_index = 0
584
+
585
+ while size > 1024 and unit_index < len(units) - 1:
586
+ size /= 1024
587
+ unit_index += 1
588
+
589
+ return f"{size:.2f} {units[unit_index]}"
590
+
591
+ total_device_memory_str = \
592
+ format(format_memory_size(total_device_memory))
593
+ used_device_memory_str = \
594
+ format(format_memory_size(used_device_mem))
595
+
596
+ logger.info(
597
+ "Total %s memory: %s. "
598
+ "Amount of memory required to run the model with "
599
+ "max_num_batched_tokens=%d: %s.", ov_device,
600
+ total_device_memory_str,
601
+ self.scheduler_config.max_num_batched_tokens,
602
+ used_device_memory_str)
603
+
604
+ if used_device_mem >= total_device_memory:
605
+ raise RuntimeError(
606
+ f"The required memory size {used_device_memory_str} for model "
607
+ "is higher than the total available device "
608
+ "memory {total_device_memory_str}. Please consider to "
609
+ "decrease `max_num_batched_tokens` or increase "
610
+ "`gpu_memory_utilization`")
611
+
612
+ return total_device_memory * memory_utilization - used_device_mem