@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / f58d4fccc9
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 10:45 UTC

[OpenVINO] Enable GPU support for OpenVINO vLLM backend (#8192)

Sergey Shlyapnikov committed Oct 3, 2024 at 01:50 UTC f58d4fccc9b270838be438f5f0db71bea156a56d
8 files changed +446 -107
docs/source/getting_started/openvino-installation.rst
+29 -6
@@ -3,7 +3,7 @@
3 Installation with OpenVINO
4 ==========================
5
6 -vLLM powered by OpenVINO supports all LLM models from :doc:`vLLM supported models list <../models/supported_models>` and can perform optimal model serving on all x86-64 CPUs with, at least, AVX2 support. OpenVINO vLLM backend supports the following advanced vLLM features:
6 +vLLM powered by OpenVINO supports all LLM models from :doc:`vLLM supported models list <../models/supported_models>` and can perform optimal model serving on all x86-64 CPUs with, at least, AVX2 support, as well as on both integrated and discrete Intel® GPUs (`the list of supported GPUs <https://docs.openvino.ai/2024/about-openvino/release-notes-openvino/system-requirements.html#gpu>`_). OpenVINO vLLM backend supports the following advanced vLLM features:
7
8 - Prefix caching (``--enable-prefix-caching``)
9 - Chunked prefill (``--enable-chunked-prefill``)
@@ -53,34 +53,57 @@ Install from source
53 $ pip install --upgrade pip
54 $ pip install -r requirements-build.txt --extra-index-url https://download.pytorch.org/whl/cpu
55
56 -- Finally, install vLLM with OpenVINO backend:
56 +- Finally, install vLLM with OpenVINO backend:
57
58 .. code-block:: console
59
60 $ PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu" VLLM_TARGET_DEVICE=openvino python -m pip install -v .
61
62 +- [Optional] To use vLLM OpenVINO backend with a GPU device, ensure your system is properly set up. Follow the instructions provided here: `https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html <https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html>`_.
63 +
64 .. _openvino_backend_performance_tips:
65
66 Performance tips
67 ----------------
68
67 -vLLM OpenVINO backend uses the following environment variables to control behavior:
69 +vLLM OpenVINO backend environment variables
70 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
71 +
72 +- ``VLLM_OPENVINO_DEVICE`` to specify which device utilize for the inference. If there are multiple GPUs in the system, additional indexes can be used to choose the proper one (e.g, ``VLLM_OPENVINO_DEVICE=GPU.1``). If the value is not specified, CPU device is used by default.
73 +
74 +- ``VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON`` to enable U8 weights compression during model loading stage. By default, compression is turned off. You can also export model with different compression techniques using `optimum-cli` and pass exported folder as `<model_id>`
75 +
76 +CPU performance tips
77 +~~~~~~~~~~~~~~~~~~~~
78 +
79 +CPU uses the following environment variables to control behavior:
80
81 - ``VLLM_OPENVINO_KVCACHE_SPACE`` to specify the KV Cache size (e.g, ``VLLM_OPENVINO_KVCACHE_SPACE=40`` means 40 GB space for KV cache), larger setting will allow vLLM running more requests in parallel. This parameter should be set based on the hardware configuration and memory management pattern of users.
82
83 - ``VLLM_OPENVINO_CPU_KV_CACHE_PRECISION=u8`` to control KV cache precision. By default, FP16 / BF16 is used depending on platform.
84
73 -- ``VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON`` to enable U8 weights compression during model loading stage. By default, compression is turned off. You can also export model with different compression techniques using `optimum-cli` and pass exported folder as `<model_id>`
74 -
85 To enable better TPOT / TTFT latency, you can use vLLM's chunked prefill feature (``--enable-chunked-prefill``). Based on the experiments, the recommended batch size is ``256`` (``--max-num-batched-tokens``)
86
77 -OpenVINO best known configuration is:
87 +OpenVINO best known configuration for CPU is:
88
89 .. code-block:: console
90
91 $ VLLM_OPENVINO_KVCACHE_SPACE=100 VLLM_OPENVINO_CPU_KV_CACHE_PRECISION=u8 VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON \
92 python3 vllm/benchmarks/benchmark_throughput.py --model meta-llama/Llama-2-7b-chat-hf --dataset vllm/benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json --enable-chunked-prefill --max-num-batched-tokens 256
93
94 +GPU performance tips
95 +~~~~~~~~~~~~~~~~~~~~
96 +GPU device implements the logic for automatic detection of available GPU memory and, by default, tries to reserve as much memory as possible for the KV cache (taking into account ``gpu_memory_utilization`` option). However, this behavior can be overridden by explicitly specifying the desired amount of memory for the KV cache using ``VLLM_OPENVINO_KVCACHE_SPACE`` environment variable (e.g, ``VLLM_OPENVINO_KVCACHE_SPACE=8`` means 8 GB space for KV cache).
97 +
98 +Currently, the best performance using GPU can be achieved with the default vLLM execution parameters for models with quantized weights (8 and 4-bit integer data types are supported) and `preemption-mode=swap`.
99 +
100 +OpenVINO best known configuration for GPU is:
101 +
102 +.. code-block:: console
103 +
104 + $ VLLM_OPENVINO_DEVICE=GPU VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON \
105 + python3 vllm/benchmarks/benchmark_throughput.py --model meta-llama/Llama-2-7b-chat-hf --dataset vllm/benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json
106 +
107 .. _openvino_backend_limitations:
108
109 Limitations
requirements-openvino.txt
+3 -2
@@ -3,5 +3,6 @@
3
4 # OpenVINO dependencies
5 torch >= 2.1.2
6 -openvino ~= 2024.3.0
7 -optimum-intel[openvino] >= 1.18.2
6 +openvino ~= 2024.4.0
7 +openvino-tokenizers[transformers] ~= 2024.4.0
8 +optimum-intel[openvino] >= 1.19.0
vllm/attention/backends/openvino.py
+32 -8
@@ -9,6 +9,31 @@ from vllm.attention.backends.abstract import (AttentionBackend,
9 from vllm.attention.backends.utils import CommonAttentionState
10
11
12 +def copy_cache_block(src_tensor: ov.Tensor, dst_tensor: ov.Tensor,
13 + src_offset: int, dst_offset: int) -> None:
14 +
15 + def create_roi_tensor(
16 + tensor: ov.Tensor,
17 + block_number: int,
18 + ) -> ov.Tensor:
19 + roi_begin = ov.runtime.Coordinate([0, 0, 0, 0])
20 + roi_end = ov.runtime.Coordinate(tensor.get_shape())
21 +
22 + roi_begin[0] = block_number
23 + roi_end[0] = block_number + 1
24 +
25 + if isinstance(tensor, ov.Tensor):
26 + return ov.Tensor(tensor, roi_begin, roi_end)
27 + else:
28 + return ov.RemoteTensor(tensor, roi_begin, roi_end)
29 +
30 + src_roi_tensor = \
31 + create_roi_tensor(src_tensor, src_offset)
32 + dst_roi_tensor = \
33 + create_roi_tensor(dst_tensor, dst_offset)
34 + src_roi_tensor.copy_to(dst_roi_tensor)
35 +
36 +
37 class OpenVINOAttentionBackend(AttentionBackend):
38
39 @staticmethod
@@ -44,13 +69,12 @@ class OpenVINOAttentionBackend(AttentionBackend):
69
70 @staticmethod
71 def swap_blocks(
47 - src_kv_cache: ov.Tensor,
48 - dst_kv_cache: ov.Tensor,
49 - src_to_dst: torch.Tensor,
72 + src_tensor: ov.Tensor,
73 + dst_tensor: ov.Tensor,
74 + src_to_dists: List[Tuple[int, int]],
75 ) -> None:
51 - # OpenVINO currently supports only CPU, which does not require
52 - # swap of KV cache blocks
53 - raise NotImplementedError
76 + for src, dst in src_to_dists:
77 + copy_cache_block(src_tensor, dst_tensor, src, dst)
78
79 @staticmethod
80 def copy_blocks(
@@ -59,8 +83,8 @@ class OpenVINOAttentionBackend(AttentionBackend):
83 ) -> None:
84 for src, dst in src_to_dists:
85 for key_cache, value_cache in kv_caches:
62 - key_cache.data[dst, :] = key_cache.data[src, :]
63 - value_cache.data[dst, :] = value_cache.data[src, :]
86 + copy_cache_block(key_cache, key_cache, src, dst)
87 + copy_cache_block(value_cache, value_cache, src, dst)
88
89
90 @dataclass
vllm/envs.py
+6
@@ -35,6 +35,7 @@ if TYPE_CHECKING:
35 VLLM_PP_LAYER_PARTITION: Optional[str] = None
36 VLLM_CPU_KVCACHE_SPACE: int = 0
37 VLLM_CPU_OMP_THREADS_BIND: str = ""
38 + VLLM_OPENVINO_DEVICE: str = "CPU"
39 VLLM_OPENVINO_KVCACHE_SPACE: int = 0
40 VLLM_OPENVINO_CPU_KV_CACHE_PRECISION: Optional[str] = None
41 VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS: bool = False
@@ -302,6 +303,11 @@ environment_variables: Dict[str, Callable[[], Any]] = {
303 "VLLM_CPU_OMP_THREADS_BIND":
304 lambda: os.getenv("VLLM_CPU_OMP_THREADS_BIND", "all"),
305
306 + # OpenVINO device selection
307 + # default is CPU
308 + "VLLM_OPENVINO_DEVICE":
309 + lambda: os.getenv("VLLM_OPENVINO_DEVICE", "CPU").upper(),
310 +
311 # OpenVINO key-value cache space
312 # default is 4GB
313 "VLLM_OPENVINO_KVCACHE_SPACE":
vllm/executor/openvino_executor.py
+53 -20
@@ -17,6 +17,14 @@ from vllm.utils import (GiB_bytes, get_distributed_init_method, get_ip,
17 logger = init_logger(__name__)
18
19
20 +def is_openvino_cpu() -> bool:
21 + return "CPU" in envs.VLLM_OPENVINO_DEVICE
22 +
23 +
24 +def is_openvino_gpu() -> bool:
25 + return "GPU" in envs.VLLM_OPENVINO_DEVICE
26 +
27 +
28 class OpenVINOExecutor(ExecutorBase):
29
30 uses_ray: bool = False
@@ -24,8 +32,13 @@ class OpenVINOExecutor(ExecutorBase):
32 def _init_executor(self) -> None:
33 assert self.device_config.device_type == "openvino"
34 assert self.lora_config is None, "OpenVINO backend doesn't support LoRA"
35 + assert is_openvino_cpu() or is_openvino_gpu(), \
36 + "OpenVINO backend supports only CPU and GPU devices"
37 +
38 + self.ov_core = ov.Core()
39 self.model_config = _verify_and_get_model_config(self.model_config)
28 - self.cache_config = _verify_and_get_cache_config(self.cache_config)
40 + self.cache_config = _verify_and_get_cache_config(
41 + self.ov_core, self.cache_config)
42
43 # Instantiate the worker and load the model to CPU.
44 self._init_worker()
@@ -40,6 +53,7 @@ class OpenVINOExecutor(ExecutorBase):
53 distributed_init_method = get_distributed_init_method(
54 get_ip(), get_open_port())
55 self.driver_worker = OpenVINOWorker(
56 + ov_core=self.ov_core,
57 model_config=self.model_config,
58 parallel_config=self.parallel_config,
59 scheduler_config=self.scheduler_config,
@@ -68,10 +82,13 @@ class OpenVINOExecutor(ExecutorBase):
82 # NOTE: We log here to avoid multiple logs when number of workers is
83 # greater than one. We could log in the engine, but not all executors
84 # have GPUs.
71 - # NOTE: `cpu block` for OpenVINO backend is located on CPU memory but is
72 - # referred as `gpu block`. Because we want to reuse the existing block
73 - # management procedure.
74 - logger.info("# CPU blocks: %d", num_gpu_blocks)
85 + # NOTE: In case of a CPU device, `cpu block` for OpenVINO backend
86 + # is located on CPU memory but is referred as `gpu block`.
87 + # Because we want to reuse the existing block management procedure.
88 + device_blocks = num_gpu_blocks
89 + swap_blocks = num_cpu_blocks
90 + logger.info("OpenVINO %s: # device blocks: %d; # swap blocks: %d",
91 + envs.VLLM_OPENVINO_DEVICE, device_blocks, swap_blocks)
92 self.driver_worker.initialize_cache(num_gpu_blocks, num_cpu_blocks)
93
94 def execute_model(
@@ -143,29 +160,45 @@ def _verify_and_get_model_config(config: ModelConfig) -> ModelConfig:
160 return config
161
162
146 -def _verify_and_get_cache_config(config: CacheConfig) -> CacheConfig:
163 +def _verify_and_get_cache_config(ov_core: ov.Core,
164 + config: CacheConfig) -> CacheConfig:
165 if envs.VLLM_OPENVINO_CPU_KV_CACHE_PRECISION == "u8":
148 - logger.info("KV cache type is overried to u8 via "
149 - "VLLM_OPENVINO_CPU_KV_CACHE_PRECISION env var.")
150 - config.cache_dtype = ov.Type.u8
166 + if not is_openvino_cpu():
167 + logger.info("VLLM_OPENVINO_CPU_KV_CACHE_PRECISION is"
168 + "ignored for GPU, f16 data type will be used.")
169 + config.cache_dtype = ov.Type.f16
170 + else:
171 + logger.info("KV cache type is overridden to u8 via "
172 + "VLLM_OPENVINO_CPU_KV_CACHE_PRECISION env var.")
173 + config.cache_dtype = ov.Type.u8
174 else:
152 - core = ov.Core()
153 - inference_precision = core.get_property("CPU",
154 - hints.inference_precision)
155 - if inference_precision == ov.Type.bf16:
156 - config.cache_dtype = ov.Type.bf16
175 + if is_openvino_cpu():
176 + ov_device = envs.VLLM_OPENVINO_DEVICE
177 + inference_precision = ov_core.get_property(
178 + ov_device, hints.inference_precision)
179 + if inference_precision == ov.Type.bf16:
180 + config.cache_dtype = ov.Type.bf16
181 + else:
182 + config.cache_dtype = ov.Type.f16
183 else:
184 config.cache_dtype = ov.Type.f16
185
160 - if config.block_size != 32:
161 - logger.info(
162 - f"OpenVINO optimal block size is 32, overriding currently set {config.block_size}" # noqa: G004, E501
163 - )
164 - config.block_size = 32
186 + if is_openvino_cpu():
187 + if config.block_size != 32:
188 + logger.info(
189 + f"OpenVINO CPU optimal block size is 32, overriding currently set {config.block_size}" # noqa: G004, E501
190 + )
191 + config.block_size = 32
192 + else:
193 + if config.block_size != 16:
194 + logger.info(
195 + f"OpenVINO GPU optimal block size is 16, overriding currently set {config.block_size}" # noqa: G004, E501
196 + )
197 + config.block_size = 16
198
199 kv_cache_space = envs.VLLM_OPENVINO_KVCACHE_SPACE
200 if kv_cache_space >= 0:
168 - if kv_cache_space == 0:
201 + if kv_cache_space == 0 and is_openvino_cpu():
202 config.openvino_kvcache_space_bytes = 4 * GiB_bytes # type: ignore
203 logger.warning(
204 "Environment variable VLLM_OPENVINO_KVCACHE_SPACE (GB) "
vllm/model_executor/model_loader/openvino.py
+10 -16
@@ -12,6 +12,7 @@ from torch import nn
12 import vllm.envs as envs
13 from vllm.attention.backends.openvino import OpenVINOAttentionMetadata
14 from vllm.config import DeviceConfig, ModelConfig
15 +from vllm.executor.openvino_executor import is_openvino_cpu
16 from vllm.logger import init_logger
17 from vllm.model_executor.layers.logits_processor import (LogitsProcessor,
18 _prune_hidden_states)
@@ -51,25 +52,15 @@ def _modify_cache_parameters(model: ov.Model, kv_cache_dtype: ov.Type,
52 shape = parameter.get_partial_shape()
53 # use real block size if available, just a placeholder
54 # to provide the expected rank
54 - x_size = 1
55 num_blocks = ov.Dimension()
56 block_size = ov.Dimension()
57 head_size = ov.Dimension()
58 - # TODO: Negotiate required layout with plugins (CPU is ~OK, GPU is TBD),
59 - # pass more parameters to this function to set more static dimensions
58 if input_name.startswith("key_cache."):
59 cpu_shape = [num_blocks, shape[1], block_size, head_size]
62 - gpu_shape = [
63 - num_blocks,
64 - shape[1],
65 - shape[2].get_length() //
66 - x_size if shape[2].is_static else ov.Dimension(),
67 - block_size,
68 - x_size,
69 - ]
60 + gpu_shape = [num_blocks, shape[1], shape[2], block_size]
61 elif input_name.startswith("value_cache."):
62 cpu_shape = [num_blocks, shape[1], block_size, head_size]
72 - gpu_shape = [num_blocks, shape[1], shape[2], block_size]
63 + gpu_shape = [num_blocks, shape[1], block_size, shape[2]]
64 else:
65 continue
66 parameter.set_partial_shape(
@@ -108,6 +99,7 @@ class OpenVINOCasualLM(nn.Module):
99
100 def __init__(
101 self,
102 + ov_core: ov.Core,
103 model_config: ModelConfig,
104 device_config: DeviceConfig,
105 kv_cache_dtype: ov.Type,
@@ -141,12 +133,12 @@ class OpenVINOCasualLM(nn.Module):
133 trust_remote_code=model_config.trust_remote_code,
134 )
135
136 + ov_device = envs.VLLM_OPENVINO_DEVICE
137 paged_attention_transformation(pt_model.model)
138 _modify_cache_parameters(pt_model.model, kv_cache_dtype,
146 - device_config.device.type == "cpu")
139 + is_openvino_cpu())
140
148 - core = ov.Core()
149 - ov_compiled = core.compile_model(pt_model.model, "CPU")
141 + ov_compiled = ov_core.compile_model(pt_model.model, ov_device)
142 self.ov_request = ov_compiled.create_infer_request()
143
144 def forward(
@@ -199,6 +191,7 @@ def get_model(
191 **kwargs,
192 ) -> torch.nn.Module:
193 lora_config = kwargs.get("lora_config", None)
194 + ov_core = kwargs.get("ov_core")
195 if lora_config:
196 raise ValueError(
197 "OpenVINO modeling does not support LoRA, "
@@ -206,4 +199,5 @@ def get_model(
199 "be added in the future. If this is important to you, "
200 "please open an issue on github.")
201
209 - return OpenVINOCasualLM(model_config, device_config, kv_cache_dtype)
202 + return OpenVINOCasualLM(ov_core, model_config, device_config,
203 + kv_cache_dtype)
vllm/worker/openvino_model_runner.py
+6 -5
@@ -42,6 +42,7 @@ class OpenVINOModelRunner:
42
43 def __init__(
44 self,
45 + ov_core: ov.Core,
46 model_config: ModelConfig,
47 parallel_config: ParallelConfig,
48 scheduler_config: SchedulerConfig,
@@ -55,6 +56,7 @@ class OpenVINOModelRunner:
56 *args,
57 **kwargs,
58 ):
59 + self.ov_core = ov_core
60 self.model_config = model_config
61 self.parallel_config = parallel_config
62 self.scheduler_config = scheduler_config
@@ -89,11 +91,10 @@ class OpenVINOModelRunner:
91 self.model: nn.Module # Set after init_Model
92
93 def load_model(self) -> None:
92 - self.model = get_model(
93 - model_config=self.model_config,
94 - device_config=self.device_config,
95 - kv_cache_dtype=self.kv_cache_dtype,
96 - )
94 + self.model = get_model(model_config=self.model_config,
95 + device_config=self.device_config,
96 + kv_cache_dtype=self.kv_cache_dtype,
97 + ov_core=self.ov_core)
98
99 def _prepare_model_input(
100 self,
vllm/worker/openvino_worker.py
+307 -50
@@ -5,6 +5,7 @@ import openvino as ov
5 import torch
6 import torch.distributed
7
8 +import vllm.envs as envs
9 from vllm.attention import get_attn_backend
10 from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
11 ModelConfig, MultiModalConfig, ParallelConfig,
@@ -12,10 +13,14 @@ from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
13 from vllm.distributed import (broadcast_tensor_dict,
14 ensure_model_parallel_initialized,
15 init_distributed_environment)
16 +from vllm.executor.openvino_executor import is_openvino_cpu
17 +from vllm.inputs import INPUT_REGISTRY
18 from vllm.logger import init_logger
19 from vllm.model_executor import set_random_seed
20 from vllm.model_executor.layers.sampler import SamplerOutput
18 -from vllm.sequence import ExecuteModelRequest
21 +from vllm.multimodal import MULTIMODAL_REGISTRY
22 +from vllm.sampling_params import SamplingParams
23 +from vllm.sequence import ExecuteModelRequest, SequenceGroupMetadata
24 from vllm.worker.openvino_model_runner import OpenVINOModelRunner
25 from vllm.worker.worker_base import LoraNotSupportedWorkerBase
26
@@ -36,6 +41,8 @@ class OpenVINOCacheEngine:
41 model_config: ModelConfig,
42 parallel_config: ParallelConfig,
43 device_config: DeviceConfig,
44 + ov_core: ov.Core,
45 + ov_device: str,
46 ) -> None:
47 assert device_config.device_type == "openvino"
48 self.cache_config = cache_config
@@ -56,9 +63,10 @@ class OpenVINOCacheEngine:
63
64 self.block_size = cache_config.block_size
65 # Note: In CacheConfig, num_gpu_blocks actual is num_cpu_blocks
59 - # for OpenVINO backend, because we want to reuse KV cache management
60 - # in the scheduler.
61 - self.num_cpu_blocks = cache_config.num_gpu_blocks
66 + # for OpenVINO backend with a CPU target device, because we want
67 + # to reuse KV cache management in the scheduler.
68 + self.num_device_blocks = cache_config.num_gpu_blocks
69 + self.num_swap_blocks = cache_config.num_cpu_blocks
70
71 # Get attention backend.
72 self.attn_backend = get_attn_backend(
@@ -74,34 +82,100 @@ class OpenVINOCacheEngine:
82 # Initialize the cache.
83 self.kv_cache: List[Tuple[ov.Tensor,
84 ov.Tensor]] = self._allocate_kv_cache(
77 - self.num_cpu_blocks)
85 + self.num_device_blocks, ov_core,
86 + ov_device)
87 +
88 + # Initialize the swap.
89 + self.swap_cache: List[Tuple[ov.Tensor,
90 + ov.Tensor]] = self._allocate_swap_cache(
91 + self.num_swap_blocks, ov_device)
92
93 def _allocate_kv_cache(
94 self,
95 num_blocks: int,
96 + ov_core: ov.Core,
97 + ov_device: str,
98 ) -> List[Tuple[ov.Tensor, ov.Tensor]]:
99 """Allocates KV cache."""
100 k_block_shape = v_block_shape = self.attn_backend.get_kv_cache_shape(
101 num_blocks, self.block_size, self.num_kv_heads, self.head_size)[1:]
102 kv_cache: List[Tuple[ov.Tensor, ov.Tensor]] = []
103 +
104 + if is_openvino_cpu():
105 + for _ in range(self.num_layers):
106 + key_blocks = ov.Tensor(self.cache_config.cache_dtype,
107 + k_block_shape)
108 + value_blocks = ov.Tensor(self.cache_config.cache_dtype,
109 + v_block_shape)
110 + kv_cache.append((key_blocks, value_blocks))
111 + else:
112 + # Update key_cache shape:
113 + k_block_shape = (v_block_shape[0], v_block_shape[1],
114 + v_block_shape[3], v_block_shape[2])
115 +
116 + remote_context = ov_core.get_default_context(ov_device)
117 +
118 + for _ in range(self.num_layers):
119 + key_blocks = \
120 + remote_context.create_tensor(self.cache_config.cache_dtype,
121 + ov.Shape(k_block_shape),
122 + {})
123 +
124 + value_blocks = \
125 + remote_context.create_tensor(self.cache_config.cache_dtype,
126 + ov.Shape(v_block_shape),
127 + {})
128 +
129 + kv_cache.append((key_blocks, value_blocks))
130 +
131 + return kv_cache
132 +
133 + def _allocate_swap_cache(
134 + self,
135 + num_blocks: int,
136 + ov_device: str,
137 + ) -> List[Tuple[ov.Tensor, ov.Tensor]]:
138 + """Allocates swap cache."""
139 + k_block_shape = v_block_shape = self.attn_backend.get_kv_cache_shape(
140 + num_blocks, self.block_size, self.num_kv_heads, self.head_size)[1:]
141 + swap_cache: List[Tuple[ov.Tensor, ov.Tensor]] = []
142 +
143 + if num_blocks == 0:
144 + return swap_cache
145 +
146 + assert not is_openvino_cpu(), \
147 + "CPU device isn't supposed to have swap cache"
148 +
149 + # Update key_cache shape:
150 + k_block_shape = (v_block_shape[0], v_block_shape[1], v_block_shape[3],
151 + v_block_shape[2])
152 +
153 for _ in range(self.num_layers):
154 key_blocks = ov.Tensor(self.cache_config.cache_dtype,
155 k_block_shape)
156 value_blocks = ov.Tensor(self.cache_config.cache_dtype,
157 v_block_shape)
92 - kv_cache.append((key_blocks, value_blocks))
93 - return kv_cache
158 + swap_cache.append((key_blocks, value_blocks))
159 +
160 + return swap_cache
161
95 - def swap_in(self, src_to_dst: Dict[int, int]) -> None:
96 - raise NotImplementedError(
97 - "Swap is not supported in OpenVINOCacheEngine.")
162 + def swap_in(self, src_to_dst: List[Tuple[int, int]]) -> None:
163 + for i in range(self.num_layers):
164 + for swap_tensor, kv_tensor in zip(self.swap_cache[i],
165 + self.kv_cache[i]):
166 + self.attn_backend.swap_blocks(swap_tensor, kv_tensor,
167 + src_to_dst)
168
99 - def swap_out(self, src_to_dst: Dict[int, int]) -> None:
100 - raise NotImplementedError(
101 - "Swap is not supported in OpenVINOCacheEngine.")
169 + def swap_out(self, src_to_dst: List[Tuple[int, int]]) -> None:
170 + for i in range(self.num_layers):
171 + for swap_tensor, kv_tensor in zip(self.swap_cache[i],
172 + self.kv_cache[i]):
173 + self.attn_backend.swap_blocks(kv_tensor, swap_tensor,
174 + src_to_dst)
175
103 - def copy(self, src_to_dsts: Dict[int, List[int]]) -> None:
104 - self.attn_backend.copy_blocks(self.kv_cache, src_to_dsts)
176 + def copy(self, src_to_dsts: List[Tuple[int, int]]) -> None:
177 + if (len(src_to_dsts) > 0):
178 + self.attn_backend.copy_blocks(self.kv_cache, src_to_dsts)
179
180 @staticmethod
181 def get_cache_block_size(
@@ -139,6 +213,7 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
213
214 def __init__(
215 self,
216 + ov_core: ov.Core,
217 model_config: ModelConfig,
218 parallel_config: ParallelConfig,
219 scheduler_config: SchedulerConfig,
@@ -153,6 +228,7 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
228 kv_cache_dtype: Optional[ov.Type] = ov.Type.undefined,
229 is_driver_worker: bool = False,
230 ) -> None:
231 + self.ov_core = ov_core
232 self.model_config = model_config
233 self.parallel_config = parallel_config
234 self.parallel_config.rank = rank
@@ -175,6 +251,7 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
251
252 init_cached_hf_modules()
253 self.model_runner = OpenVINOModelRunner(
254 + self.ov_core,
255 model_config,
256 parallel_config,
257 scheduler_config,
@@ -204,56 +281,69 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
281
282 This determines how many KV blocks can fit into the configured
283 KV cache space.
207 -
208 - Note that since vLLM assumes a block resides on GPU if it can be
209 - modified, we return num_gpu_blocks=num_cpu_blocks and num_cpu_blocks=0.
210 - This allows us to reuse the scheduler of vLLM without generalizing it
211 - to different devices.
284 """
213 - # For OpenVINO backend, the block number will be calculated based on the
214 - # openvino_kvcache_space_bytes.
285 + # For OpenVINO backend, in case of CPU device, the block number will be
286 + # calculated based on the openvino_kvcache_space_bytes.
287 cache_block_size = self.get_cache_block_size_bytes()
216 - num_cpu_blocks = int(self.cache_config.openvino_kvcache_space_bytes //
217 - cache_block_size)
218 - num_cpu_blocks = max(num_cpu_blocks, 0)
288 + kvcache_space_bytes = self.cache_config.openvino_kvcache_space_bytes
289
220 - # Note: To reuse the cache management procedure,
221 - # use cpu cache as 'gpu cache'.
222 - num_gpu_blocks = num_cpu_blocks
223 - num_cpu_blocks = 0
224 - return num_gpu_blocks, num_cpu_blocks
290 + if is_openvino_cpu():
291 + num_device_blocks = int(kvcache_space_bytes // cache_block_size)
292 + num_swap_blocks = 0
293 + else:
294 + if kvcache_space_bytes > 0:
295 + logger.info("KV_CACHE size was explicitly configured via "
296 + "VLLM_OPENVINO_KVCACHE_SPACE environment "
297 + "variable, ignoring profiling run.")
298 + kv_cache_size = kvcache_space_bytes
299 + else:
300 + try:
301 + kv_cache_size = self.profile_run()
302 + except Exception as err:
303 + raise RuntimeError(
304 + "The error occurred during profile run. This might be "
305 + "due to insufficient GPU memory. Consider decreasing "
306 + "`max_model_len` to limit the maximum simultaneously "
307 + "processed tokens.") from err
308 +
309 + num_device_blocks = int(kv_cache_size // cache_block_size)
310 + num_swap_blocks = int(self.cache_config.swap_space_bytes //
311 + cache_block_size)
312 +
313 + return num_device_blocks, num_swap_blocks
314
315 def initialize_cache(self, num_gpu_blocks: int,
316 num_cpu_blocks: int) -> None:
228 - """Initialize the KV cache. Currently, swappable CPU memory is not
229 - supported.
317 + """Initialize the KV cache. Swappable CPU memory is only
318 + supported on GPU.
319
231 - Since this worker does not support GPUs, we use the num_gpu_blocks to
320 + For CPU, we use the num_gpu_blocks to
321 determine how many non-swappable CPU blocks to allocate.
322 """
234 - assert (num_cpu_blocks == 0
235 - ), f"{type(self)} does not support swappable cache"
323
237 - # Note: To reuse the cache management procedure,
238 - # use cpu cache as 'gpu cache'.
239 - num_cpu_blocks = num_gpu_blocks
324 + num_device_blocks = num_gpu_blocks
325 + num_swap_blocks = num_cpu_blocks
326 +
327 + if is_openvino_cpu():
328 + assert (num_swap_blocks == 0
329 + ), f"{type(self)} does not support swappable cache for CPU"
330
241 - self._validate_num_cpu_blocks(num_cpu_blocks)
242 - self.cache_config.num_gpu_blocks = num_cpu_blocks
243 - self.cache_config.num_cpu_blocks = 0
331 + self._validate_num_blocks(num_device_blocks)
332 + self.cache_config.num_gpu_blocks = num_device_blocks
333 + self.cache_config.num_cpu_blocks = num_swap_blocks
334
335 # Initialize the cache.
336 self._init_cache_engine()
337
248 - def _validate_num_cpu_blocks(self, num_cpu_blocks: int) -> None:
249 - """Raise errors if the num_cpu_blocks is invalid."""
250 - if num_cpu_blocks <= 0:
338 + def _validate_num_blocks(self, num_blocks: int) -> None:
339 + """Raise errors if the num_blocks is invalid."""
340 + if num_blocks <= 0:
341 raise ValueError(
342 "No available memory for the cache blocks. "
343 "Try increasing `VLLM_OPENVINO_KVCACHE_SPACE` when "
344 "initializing the engine.")
345
256 - max_seq_len = self.cache_config.block_size * num_cpu_blocks
346 + max_seq_len = self.cache_config.block_size * num_blocks
347 if self.model_config.max_model_len > max_seq_len:
348 raise ValueError(
349 f"The model's max seq len ({self.model_config.max_model_len}) "
@@ -263,11 +353,14 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
353 "when initializing the engine.")
354
355 def _init_cache_engine(self) -> None:
356 + ov_device = envs.VLLM_OPENVINO_DEVICE
357 self.cache_engine = OpenVINOCacheEngine(
358 self.cache_config,
359 self.model_config,
360 self.parallel_config,
361 self.device_config,
362 + self.ov_core,
363 + ov_device,
364 )
365 self.kv_cache = self.cache_engine.kv_cache
366 self.model_runner.block_size = self.cache_engine.block_size
@@ -275,9 +368,16 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
368 assert self.kv_cache is not None
369
370 # Populate the cache to warmup the memory
278 - for key_cache, value_cache in self.kv_cache:
279 - key_cache.data[:] = 0
280 - value_cache.data[:] = 0
371 + if is_openvino_cpu():
372 + for key_cache, value_cache in self.kv_cache:
373 + key_cache.data[:] = 0
374 + value_cache.data[:] = 0
375 +
376 + def cache_swap_in(self, src_to_dst: List[Tuple[int, int]]) -> None:
377 + self.cache_engine.swap_in(src_to_dst)
378 +
379 + def cache_swap_out(self, src_to_dst: List[Tuple[int, int]]) -> None:
380 + self.cache_engine.swap_out(src_to_dst)
381
382 def cache_copy(
383 self,
@@ -300,17 +400,28 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
400 num_seq_groups: int = len(seq_group_metadata_list)
401 assert execute_model_req is not None
402 blocks_to_copy = execute_model_req.blocks_to_copy
303 - assert len(execute_model_req.blocks_to_swap_in) == 0
304 - assert len(execute_model_req.blocks_to_swap_out) == 0
403 + blocks_to_swap_in = execute_model_req.blocks_to_swap_in
404 + blocks_to_swap_out = execute_model_req.blocks_to_swap_out
405 data: Dict[str, Any] = {
406 "num_seq_groups": num_seq_groups,
407 "blocks_to_copy": execute_model_req.blocks_to_copy,
408 + "blocks_to_swap_in": execute_model_req.blocks_to_swap_in,
409 + "blocks_to_swap_out": execute_model_req.blocks_to_swap_out,
410 }
411 broadcast_tensor_dict(data, src=0)
412 else:
413 data = broadcast_tensor_dict(src=0)
414 num_seq_groups = data["num_seq_groups"]
415 blocks_to_copy = data["blocks_to_copy"]
416 + blocks_to_swap_in = data["blocks_to_swap_in"]
417 + blocks_to_swap_out = data["blocks_to_swap_out"]
418 +
419 + if is_openvino_cpu():
420 + assert len(execute_model_req.blocks_to_swap_in) == 0
421 + assert len(execute_model_req.blocks_to_swap_out) == 0
422 + else:
423 + self.cache_swap_in(blocks_to_swap_in)
424 + self.cache_swap_out(blocks_to_swap_out)
425
426 self.cache_copy(blocks_to_copy)
427
@@ -353,3 +464,149 @@ class OpenVINOWorker(LoraNotSupportedWorkerBase):
464 self.model_config,
465 self.parallel_config,
466 )
467 +
468 + def profile_run(self) -> int:
469 + ov_device = envs.VLLM_OPENVINO_DEVICE
470 +
471 + assert not is_openvino_cpu(), \
472 + "CPU device isn't supposed to use profile run."
473 +
474 + import openvino.properties.device as device
475 + import openvino.properties.intel_gpu as intel_gpu
476 +
477 + ov_core = self.ov_core
478 + cache_config = self.cache_config
479 + model_config = self.model_config
480 + parallel_config = self.parallel_config
481 + device_config = self.device_config
482 + input_registry = INPUT_REGISTRY
483 + mm_registry = MULTIMODAL_REGISTRY
484 + mm_registry.init_mm_limits_per_prompt(model_config)
485 +
486 + # Execute a forward pass with dummy inputs to profile the memory usage
487 + # of the model.
488 + def model_profile_run():
489 + top_k = model_config.get_vocab_size() - 1
490 + sampling_params = SamplingParams(top_p=0.99, top_k=top_k)
491 +
492 + max_num_batched_tokens = \
493 + self.scheduler_config.max_num_batched_tokens
494 + max_num_seqs = self.scheduler_config.max_num_seqs
495 + tmp_cache_config = CacheConfig(cache_config.block_size,
496 + cache_config.gpu_memory_utilization,
497 + cache_config.swap_space_bytes,
498 + "auto")
499 + tmp_cache_config.num_gpu_blocks = 1
500 + tmp_cache_config.num_cpu_blocks = 0
501 + tmp_cache_config.cache_dtype = cache_config.cache_dtype
502 +
503 + profiling_cache_engine = OpenVINOCacheEngine(
504 + tmp_cache_config, model_config, parallel_config, device_config,
505 + ov_core, ov_device)
506 +
507 + # Profile memory usage with max_num_sequences sequences and the
508 + # total # number of tokens equal to max_num_batched_tokens.
509 + seqs: List[SequenceGroupMetadata] = []
510 + for group_id in range(max_num_seqs):
511 + seq_len = (max_num_batched_tokens // max_num_seqs +
512 + (group_id < max_num_batched_tokens % max_num_seqs))
513 + block_size = cache_config.block_size
514 + seq_num_blocks = (seq_len + block_size - 1) // block_size
515 +
516 + seq_data, dummy_multi_modal_data = input_registry \
517 + .dummy_data_for_profiling(model_config,
518 + seq_len,
519 + mm_registry)
520 +
521 + block_tables = [[0] * seq_num_blocks] * max_num_seqs
522 + seq = SequenceGroupMetadata(
523 + request_id=str(group_id),
524 + is_prompt=True,
525 + seq_data={group_id: seq_data},
526 + sampling_params=sampling_params,
527 + block_tables=block_tables,
528 + lora_request=None,
529 + multi_modal_data=dummy_multi_modal_data)
530 + seqs.append(seq)
531 +
532 + self.model_runner.block_size = tmp_cache_config.block_size
533 +
534 + # Run the model with the dummy inputs.
535 + self.model_runner.execute_model(seqs,
536 + profiling_cache_engine.kv_cache)
537 +
538 + # explicitly delete temporary KV cache manager to free KV cache
539 + # when real inputs will be passed to OV
540 + del profiling_cache_engine
541 +
542 + logger.info(
543 + "Start profiling run with dummy inputs to evaluate "
544 + "memory usage for %s. It might take a while.", ov_device)
545 +
546 + model_profile_run()
547 +
548 + gpu_device_type = ov_core.get_property(ov_device, device.type)
549 + memory_statistics = \
550 + ov_core.get_property(ov_device, intel_gpu.memory_statistics)
551 + memory_utilization = cache_config.gpu_memory_utilization
552 +
553 + if gpu_device_type == device.Type.INTEGRATED and \
554 + memory_utilization >= 0.9:
555 + logger.warning(
556 + "iGPU is used with high gpu_memory_utilization=%f "
557 + "value. This may cause low performance due to "
558 + "occupying the majority of available system "
559 + "memory. Please consider decreasing "
560 + "gpu_memory_utilization or explicitly setting"
561 + "`VLLM_OPENVINO_KVCACHE_SPACE` (GB) environment "
562 + "variable.", memory_utilization)
563 +
564 + # sum up all used device memory
565 + device_memory_types = ["cl_mem", "usm_device"]
566 + used_device_mem = \
567 + sum(memory_statistics.get(key, 0) for key in device_memory_types)
568 +
569 + if gpu_device_type == device.Type.INTEGRATED:
570 + used_device_mem += memory_statistics.get("usm_host", 0)
571 +
572 + # there could be unaccounted extra memory reserved by kernels, kept
573 + # in memory pools, etc
574 + # therefore, add a threshold to account for this
575 + used_memory_threshold = 1.1
576 + used_device_mem *= used_memory_threshold
577 +
578 + total_device_memory = \
579 + ov_core.get_property(ov_device, intel_gpu.device_total_mem_size)
580 +
581 + def format_memory_size(size) -> str:
582 + units = ["B", "KB", "MB", "GB"]
583 + unit_index = 0
584 +
585 + while size > 1024 and unit_index < len(units) - 1:
586 + size /= 1024
587 + unit_index += 1
588 +
589 + return f"{size:.2f} {units[unit_index]}"
590 +
591 + total_device_memory_str = \
592 + format(format_memory_size(total_device_memory))
593 + used_device_memory_str = \
594 + format(format_memory_size(used_device_mem))
595 +
596 + logger.info(
597 + "Total %s memory: %s. "
598 + "Amount of memory required to run the model with "
599 + "max_num_batched_tokens=%d: %s.", ov_device,
600 + total_device_memory_str,
601 + self.scheduler_config.max_num_batched_tokens,
602 + used_device_memory_str)
603 +
604 + if used_device_mem >= total_device_memory:
605 + raise RuntimeError(
606 + f"The required memory size {used_device_memory_str} for model "
607 + "is higher than the total available device "
608 + "memory {total_device_memory_str}. Please consider to "
609 + "decrease `max_num_batched_tokens` or increase "
610 + "`gpu_memory_utilization`")
611 +
612 + return total_device_memory * memory_utilization - used_device_mem