@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 57f09a419c
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

[Hardware][Intel] OpenVINO vLLM backend (#5379)

Ilya Lavrenov committed Jun 28, 2024 at 17:50 UTC 57f09a419c04ecec4718ea9d5be1e6f4a8cc336e
22 files changed +1393 -23
.buildkite/run-openvino-test.sh new
+14
@@ -0,0 +1,14 @@
1 +# This script build the OpenVINO docker image and run the offline inference inside the container.
2 +# It serves a sanity check for compilation and basic model usage.
3 +set -ex
4 +
5 +# Try building the docker image
6 +docker build -t openvino-test -f Dockerfile.openvino .
7 +
8 +# Setup cleanup
9 +remove_docker_container() { docker rm -f openvino-test || true; }
10 +trap remove_docker_container EXIT
11 +remove_docker_container
12 +
13 +# Run the image and launch offline inference
14 +docker run --network host --env VLLM_OPENVINO_KVCACHE_SPACE=1 --name openvino-test openvino-test python3 /workspace/vllm/examples/offline_inference.py
Dockerfile.openvino new
+26
@@ -0,0 +1,26 @@
1 +# The vLLM Dockerfile is used to construct vLLM image that can be directly used
2 +# to run the OpenAI compatible server.
3 +
4 +FROM ubuntu:22.04 AS dev
5 +
6 +RUN apt-get update -y && \
7 + apt-get install -y python3-pip git
8 +WORKDIR /workspace
9 +
10 +# copy requirements
11 +COPY requirements-build.txt /workspace/vllm/
12 +COPY requirements-common.txt /workspace/vllm/
13 +COPY requirements-openvino.txt /workspace/vllm/
14 +
15 +COPY vllm/ /workspace/vllm/vllm
16 +COPY setup.py /workspace/vllm/
17 +
18 +# install build requirements
19 +RUN PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu" python3 -m pip install -r /workspace/vllm/requirements-build.txt
20 +# build vLLM with OpenVINO backend
21 +RUN PIP_PRE=1 PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu https://storage.openvinotoolkit.org/simple/wheels/nightly/" VLLM_TARGET_DEVICE="openvino" python3 -m pip install /workspace/vllm/
22 +
23 +COPY examples/ /workspace/vllm/examples
24 +COPY benchmarks/ /workspace/vllm/benchmarks
25 +
26 +CMD ["/bin/bash"]
benchmarks/benchmark_latency.py
+4 -3
@@ -207,9 +207,10 @@ if __name__ == '__main__':
207 parser.add_argument(
208 "--device",
209 type=str,
210 - default="cuda",
211 - choices=["cuda", "cpu", "tpu", "xpu"],
212 - help='device type for vLLM execution, supporting CUDA and CPU.')
210 + default="auto",
211 + choices=["auto", "cuda", "cpu", "openvino", "tpu", "xpu"],
212 + help='device type for vLLM execution, supporting CUDA, OpenVINO and '
213 + 'CPU.')
214 parser.add_argument('--block-size',
215 type=int,
216 default=16,
benchmarks/benchmark_throughput.py
+4 -3
@@ -349,9 +349,10 @@ if __name__ == "__main__":
349 parser.add_argument(
350 "--device",
351 type=str,
352 - default="cuda",
353 - choices=["cuda", "cpu", "tpu", "xpu"],
354 - help='device type for vLLM execution, supporting CUDA and CPU.')
352 + default="auto",
353 + choices=["auto", "cuda", "cpu", "openvino", "tpu", "xpu"],
354 + help='device type for vLLM execution, supporting CUDA, OpenVINO and '
355 + 'CPU.')
356 parser.add_argument(
357 "--enable-prefix-caching",
358 action='store_true',
docs/source/getting_started/openvino-installation.rst new
+95
@@ -0,0 +1,95 @@
1 +.. _installation_openvino:
2 +
3 +Installation with OpenVINO
4 +==========================
5 +
6 +vLLM powered by OpenVINO supports all LLM models from :doc:`vLLM supported models list <../models/supported_models>` and can perform optimal model serving on all x86-64 CPUs with, at least, AVX2 support. OpenVINO vLLM backend supports the following advanced vLLM features:
7 +
8 +- Prefix caching (``--enable-prefix-caching``)
9 +- Chunked prefill (``--enable-chunked-prefill``)
10 +
11 +**Table of contents**:
12 +
13 +- :ref:`Requirements <openvino_backend_requirements>`
14 +- :ref:`Quick start using Dockerfile <openvino_backend_quick_start_dockerfile>`
15 +- :ref:`Build from source <install_openvino_backend_from_source>`
16 +- :ref:`Performance tips <openvino_backend_performance_tips>`
17 +- :ref:`Limitations <openvino_backend_limitations>`
18 +
19 +.. _openvino_backend_requirements:
20 +
21 +Requirements
22 +------------
23 +
24 +* OS: Linux
25 +* Instruction set architecture (ISA) requirement: at least AVX2.
26 +
27 +.. _openvino_backend_quick_start_dockerfile:
28 +
29 +Quick start using Dockerfile
30 +----------------------------
31 +
32 +.. code-block:: console
33 +
34 + $ docker build -f Dockerfile.openvino -t vllm-openvino-env .
35 + $ docker run -it --rm vllm-openvino-env
36 +
37 +.. _install_openvino_backend_from_source:
38 +
39 +Install from source
40 +-------------------
41 +
42 +- First, install Python. For example, on Ubuntu 22.04, you can run:
43 +
44 + .. code-block:: console
45 +
46 + $ sudo apt-get update -y
47 + $ sudo apt-get install python3
48 +
49 +- Second, install prerequisites vLLM OpenVINO backend installation:
50 +
51 + .. code-block:: console
52 +
53 + $ pip install --upgrade pip
54 + $ pip install -r requirements-build.txt --extra-index-url https://download.pytorch.org/whl/cpu
55 +
56 +- Finally, install vLLM with OpenVINO backend:
57 +
58 + .. code-block:: console
59 +
60 + $ PIP_PRE=1 PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu https://storage.openvinotoolkit.org/simple/wheels/nightly/" VLLM_TARGET_DEVICE=openvino python -m pip install -v .
61 +
62 +.. _openvino_backend_performance_tips:
63 +
64 +Performance tips
65 +----------------
66 +
67 +vLLM OpenVINO backend uses the following environment variables to control behavior:
68 +
69 +- ``VLLM_OPENVINO_KVCACHE_SPACE`` to specify the KV Cache size (e.g, ``VLLM_OPENVINO_KVCACHE_SPACE=40`` means 40 GB space for KV cache), larger setting will allow vLLM running more requests in parallel. This parameter should be set based on the hardware configuration and memory management pattern of users.
70 +
71 +- ``VLLM_OPENVINO_CPU_KV_CACHE_PRECISION=u8`` to control KV cache precision. By default, FP16 / BF16 is used depending on platform.
72 +
73 +- ``VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON`` to enable U8 weights compression during model loading stage. By default, compression is turned off.
74 +
75 +To enable better TPOT / TTFT latency, you can use vLLM's chunked prefill feature (``--enable-chunked-prefill``). Based on the experiments, the recommended batch size is ``256`` (``--max-num-batched-tokens``)
76 +
77 +OpenVINO best known configuration is:
78 +
79 +.. code-block:: console
80 +
81 + $ VLLM_OPENVINO_KVCACHE_SPACE=100 VLLM_OPENVINO_CPU_KV_CACHE_PRECISION=u8 VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON \
82 + python3 vllm/benchmarks/benchmark_throughput.py --model meta-llama/Llama-2-7b-chat-hf --dataset vllm/benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json --enable-chunked-prefill --max-num-batched-tokens 256
83 +
84 +.. _openvino_backend_limitations:
85 +
86 +Limitations
87 +-----------
88 +
89 +- LoRA serving is not supported.
90 +
91 +- Only LLM models are currently supported. LLaVa and encoder-decoder models are not currently enabled in vLLM OpenVINO integration.
92 +
93 +- Tensor and pipeline parallelism are not currently enabled in vLLM integration.
94 +
95 +- Speculative sampling is not tested within vLLM integration.
docs/source/index.rst
+1
@@ -63,6 +63,7 @@ Documentation
63
64 getting_started/installation
65 getting_started/amd-installation
66 + getting_started/openvino-installation
67 getting_started/cpu-installation
68 getting_started/neuron-installation
69 getting_started/tpu-installation
requirements-openvino.txt new
+9
@@ -0,0 +1,9 @@
1 +# Common dependencies
2 +-r requirements-common.txt
3 +
4 +# OpenVINO dependencies
5 +torch >= 2.1.2
6 +openvino ~= 2024.3.0.dev
7 +optimum-intel[openvino] >= 1.17.2
8 +
9 +triton >= 2.2.0 # FIXME(woosuk): This is a hack to avoid import error.
setup.py
+10 -1
@@ -233,6 +233,10 @@ def _is_cpu() -> bool:
233 return VLLM_TARGET_DEVICE == "cpu"
234
235
236 +def _is_openvino() -> bool:
237 + return VLLM_TARGET_DEVICE == "openvino"
238 +
239 +
240 def _is_xpu() -> bool:
241 return VLLM_TARGET_DEVICE == "xpu"
242
@@ -337,6 +341,8 @@ def get_vllm_version() -> str:
341 if neuron_version != MAIN_CUDA_VERSION:
342 neuron_version_str = neuron_version.replace(".", "")[:3]
343 version += f"+neuron{neuron_version_str}"
344 + elif _is_openvino():
345 + version += "+openvino"
346 elif _is_tpu():
347 version += "+tpu"
348 elif _is_cpu():
@@ -388,6 +394,8 @@ def get_requirements() -> List[str]:
394 requirements = _read_requirements("requirements-rocm.txt")
395 elif _is_neuron():
396 requirements = _read_requirements("requirements-neuron.txt")
397 + elif _is_openvino():
398 + requirements = _read_requirements("requirements-openvino.txt")
399 elif _is_tpu():
400 requirements = _read_requirements("requirements-tpu.txt")
401 elif _is_cpu():
@@ -396,7 +404,8 @@ def get_requirements() -> List[str]:
404 requirements = _read_requirements("requirements-xpu.txt")
405 else:
406 raise ValueError(
399 - "Unsupported platform, please use CUDA, ROCm, Neuron, or CPU.")
407 + "Unsupported platform, please use CUDA, ROCm, Neuron, "
408 + "OpenVINO, or CPU.")
409 return requirements
410
411
tests/kernels/test_attention_selector.py
+7 -2
@@ -9,8 +9,8 @@ from vllm.attention.selector import which_attn_to_use
9
10
11 @pytest.mark.parametrize(
12 - "name", ["TORCH_SDPA", "ROCM_FLASH", "XFORMERS", "FLASHINFER"])
13 -@pytest.mark.parametrize("device", ["cpu", "hip", "cuda"])
12 + "name", ["TORCH_SDPA", "ROCM_FLASH", "XFORMERS", "FLASHINFER", "OPENVINO"])
13 +@pytest.mark.parametrize("device", ["cpu", "openvino", "hip", "cuda"])
14 def test_env(name: str, device: str, monkeypatch):
15 """Test that the attention selector can be set via environment variable.
16 Note that we do not test FlashAttn because it is the default backend.
@@ -28,6 +28,11 @@ def test_env(name: str, device: str, monkeypatch):
28 backend = which_attn_to_use(8, 16, 8, None, torch.float16,
29 torch.float16, 16)
30 assert backend.name == "ROCM_FLASH"
31 + elif device == "openvino":
32 + with patch("vllm.attention.selector.is_openvino", return_value=True):
33 + backend = which_attn_to_use(8, 16, 8, None, torch.float16,
34 + torch.float16, 16)
35 + assert backend.name == "OPENVINO"
36 else:
37 backend = which_attn_to_use(8, 16, 8, None, torch.float16,
38 torch.float16, 16)
vllm/attention/backends/openvino.py new
+101
@@ -0,0 +1,101 @@
1 +from dataclasses import dataclass
2 +from typing import List, Tuple
3 +
4 +import openvino as ov
5 +import torch
6 +
7 +from vllm.attention.backends.abstract import (AttentionBackend,
8 + AttentionMetadata)
9 +
10 +
11 +class OpenVINOAttentionBackend(AttentionBackend):
12 +
13 + @staticmethod
14 + def get_name() -> str:
15 + return "openvino"
16 +
17 + @staticmethod
18 + def get_impl_cls():
19 + # OpenVINO implements PagedAttention as part of the Optimum
20 + # exported model
21 + raise NotImplementedError
22 +
23 + @staticmethod
24 + def make_metadata(*args, **kwargs) -> "AttentionMetadata":
25 + raise NotImplementedError
26 +
27 + @staticmethod
28 + def make_openvino_metadata(*args, **kwargs) -> "OpenVINOAttentionMetadata":
29 + return OpenVINOAttentionMetadata(*args, **kwargs)
30 +
31 + @staticmethod
32 + def get_kv_cache_shape(
33 + num_blocks: int,
34 + block_size: int,
35 + num_kv_heads: int,
36 + head_size: int,
37 + ) -> Tuple[int, ...]:
38 + return (2, num_blocks, num_kv_heads, block_size, head_size)
39 +
40 + @staticmethod
41 + def swap_blocks(
42 + src_kv_cache: ov.Tensor,
43 + dst_kv_cache: ov.Tensor,
44 + src_to_dst: torch.Tensor,
45 + ) -> None:
46 + # OpenVINO currently supports only CPU, which does not require
47 + # swap of KV cache blocks
48 + raise NotImplementedError
49 +
50 + @staticmethod
51 + def copy_blocks(
52 + kv_caches: List[Tuple[ov.Tensor, ov.Tensor]],
53 + src_to_dists: List[Tuple[int, int]],
54 + ) -> None:
55 + for src, dst in src_to_dists:
56 + for key_cache, value_cache in kv_caches:
57 + key_cache.data[dst, :] = key_cache.data[src, :]
58 + value_cache.data[dst, :] = value_cache.data[src, :]
59 +
60 +
61 +@dataclass
62 +class OpenVINOAttentionMetadata:
63 + """Metadata for OpenVINOAttentionBackend.
64 +
65 + Basic terms used below:
66 + - batch_size_in_sequences - total number of sequences to execute​
67 + - prompt_lens – per sequence size number of scheduled tokens​
68 + - batch_size_in_tokens = sum(prompt_lens)​
69 + - max_context_len = max(context_lens)​
70 + - max_num_blocks = div_up(max_context_len / BLOCK_SIZE)​
71 + - num_blocks – total number of blocks in block_indices​
72 + """
73 +
74 + # Describes past KV cache size for each sequence within a batch
75 + # Shape: [batch_size_in_sequences]
76 + # Type: i32​
77 + past_lens: torch.Tensor
78 +
79 + # Describes start indices of input / speculative tokens from
80 + # current sequences within a batch sequence​
81 + # Shape: [batch_size_in_sequences + 1]​
82 + # Type: i32
83 + subsequence_begins: torch.Tensor
84 +
85 + # Describes block tables for each sequence within a batch​ -
86 + # indices along 0th dimension in key_cache and value_cache inputs​
87 + # Shape: [num_blocks]
88 + # Type: i32​
89 + block_indices: torch.Tensor
90 +
91 + # Describes block tables for each sequence within a batch​ -
92 + # for i-th element, it is an index in block_indices with the
93 + # first block belonging to i-th sequence​
94 + # Shape: [batch_size_in_sequences + 1]
95 + # Type: i32​
96 + block_indices_begins: torch.Tensor
97 +
98 + # Describes max context length
99 + # Shape: scalar
100 + # Type: i32
101 + max_context_len: torch.Tensor
vllm/attention/selector.py
+11 -1
@@ -7,7 +7,7 @@ import torch
7 import vllm.envs as envs
8 from vllm.attention.backends.abstract import AttentionBackend
9 from vllm.logger import init_logger
10 -from vllm.utils import is_cpu, is_hip, is_tpu, is_xpu
10 +from vllm.utils import is_cpu, is_hip, is_openvino, is_tpu, is_xpu
11
12 logger = init_logger(__name__)
13
@@ -17,6 +17,7 @@ class _Backend(enum.Enum):
17 XFORMERS = enum.auto()
18 ROCM_FLASH = enum.auto()
19 TORCH_SDPA = enum.auto()
20 + OPENVINO = enum.auto()
21 FLASHINFER = enum.auto()
22 PALLAS = enum.auto()
23 IPEX = enum.auto()
@@ -64,6 +65,10 @@ def get_attn_backend(
65 logger.info("Using Torch SDPA backend.")
66 from vllm.attention.backends.torch_sdpa import TorchSDPABackend
67 return TorchSDPABackend
68 + elif backend == _Backend.OPENVINO:
69 + logger.info("Using OpenVINO Attention backend.")
70 + from vllm.attention.backends.openvino import OpenVINOAttentionBackend
71 + return OpenVINOAttentionBackend
72 elif backend == _Backend.IPEX:
73 assert is_xpu(), RuntimeError(
74 "IPEX attention backend is only used for the XPU device.")
@@ -113,6 +118,11 @@ def which_attn_to_use(
118 logger.info("Cannot use %s backend on CPU.", selected_backend)
119 return _Backend.TORCH_SDPA
120
121 + if is_openvino():
122 + if selected_backend != _Backend.OPENVINO:
123 + logger.info("Cannot use %s backend on OpenVINO.", selected_backend)
124 + return _Backend.OPENVINO
125 +
126 if is_xpu():
127 if selected_backend != _Backend.IPEX:
128 logger.info("Cannot use %s backend on XPU.", selected_backend)
vllm/config.py
+5 -3
@@ -14,8 +14,8 @@ from vllm.model_executor.models import ModelRegistry
14 from vllm.tracing import is_otel_installed
15 from vllm.transformers_utils.config import get_config, get_hf_text_config
16 from vllm.utils import (cuda_device_count_stateless, get_cpu_memory, is_cpu,
17 - is_hip, is_neuron, is_tpu, is_xpu, print_warning_once,
18 - update_environment_variables)
17 + is_hip, is_neuron, is_openvino, is_tpu, is_xpu,
18 + print_warning_once, update_environment_variables)
19
20 if TYPE_CHECKING:
21 from ray.util.placement_group import PlacementGroup
@@ -781,6 +781,8 @@ class DeviceConfig:
781 # Automated device type detection
782 if is_neuron():
783 self.device_type = "neuron"
784 + elif is_openvino():
785 + self.device_type = "openvino"
786 elif is_tpu():
787 self.device_type = "tpu"
788 elif is_cpu():
@@ -796,7 +798,7 @@ class DeviceConfig:
798 self.device_type = device
799
800 # Some device types require processing inputs on CPU
799 - if self.device_type in ["neuron"]:
801 + if self.device_type in ["neuron", "openvino"]:
802 self.device = torch.device("cpu")
803 elif self.device_type in ["tpu"]:
804 self.device = None
vllm/engine/arg_utils.py
+8 -6
@@ -504,12 +504,14 @@ class EngineArgs:
504 'Enabling this will use the fully sharded layers. '
505 'At high sequence length, max rank or '
506 'tensor parallel size, this is likely faster.'))
507 - parser.add_argument(
508 - "--device",
509 - type=str,
510 - default=EngineArgs.device,
511 - choices=["auto", "cuda", "neuron", "cpu", "tpu", "xpu"],
512 - help='Device type for vLLM execution.')
507 + parser.add_argument("--device",
508 + type=str,
509 + default=EngineArgs.device,
510 + choices=[
511 + "auto", "cuda", "neuron", "cpu", "openvino",
512 + "tpu", "xpu"
513 + ],
514 + help='Device type for vLLM execution.')
515
516 # Related to Vision-language models such as llava
517 parser = EngineArgs.add_cli_args_for_vlm(parser)
vllm/engine/async_llm_engine.py
+6
@@ -393,6 +393,12 @@ class AsyncLLMEngine:
393 "Distributed execution is not supported with the CPU backend.")
394 from vllm.executor.cpu_executor import CPUExecutorAsync
395 executor_class = CPUExecutorAsync
396 + elif engine_config.device_config.device_type == "openvino":
397 + assert distributed_executor_backend is None, (
398 + "Distributed execution is not supported with "
399 + "the OpenVINO backend.")
400 + from vllm.executor.openvino_executor import OpenVINOExecutorAsync
401 + executor_class = OpenVINOExecutorAsync
402 elif engine_config.device_config.device_type == "xpu":
403 if distributed_executor_backend is None:
404 from vllm.executor.xpu_executor import XPUExecutorAsync
vllm/engine/llm_engine.py
+3
@@ -363,6 +363,9 @@ class LLMEngine:
363 elif engine_config.device_config.device_type == "cpu":
364 from vllm.executor.cpu_executor import CPUExecutor
365 executor_class = CPUExecutor
366 + elif engine_config.device_config.device_type == "openvino":
367 + from vllm.executor.openvino_executor import OpenVINOExecutor
368 + executor_class = OpenVINOExecutor
369 elif engine_config.device_config.device_type == "xpu":
370 if distributed_executor_backend == "ray":
371 initialize_ray_cluster(engine_config.parallel_config)
vllm/envs.py
+21 -1
@@ -28,6 +28,9 @@ if TYPE_CHECKING:
28 VLLM_TRACE_FUNCTION: int = 0
29 VLLM_ATTENTION_BACKEND: Optional[str] = None
30 VLLM_CPU_KVCACHE_SPACE: int = 0
31 + VLLM_OPENVINO_KVCACHE_SPACE: int = 0
32 + VLLM_OPENVINO_CPU_KV_CACHE_PRECISION: Optional[str] = None
33 + VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS: bool = False
34 VLLM_XLA_CACHE_PATH: str = "~/.vllm/xla_cache/"
35 VLLM_USE_RAY_COMPILED_DAG: bool = False
36 VLLM_WORKER_MULTIPROC_METHOD: str = "fork"
@@ -49,7 +52,8 @@ environment_variables: Dict[str, Callable[[], Any]] = {
52
53 # ================== Installation Time Env Vars ==================
54
52 - # Target device of vLLM, supporting [cuda (by default), rocm, neuron, cpu]
55 + # Target device of vLLM, supporting [cuda (by default),
56 + # rocm, neuron, cpu, openvino]
57 "VLLM_TARGET_DEVICE":
58 lambda: os.getenv("VLLM_TARGET_DEVICE", "cuda"),
59
@@ -208,6 +212,22 @@ environment_variables: Dict[str, Callable[[], Any]] = {
212 "VLLM_CPU_KVCACHE_SPACE":
213 lambda: int(os.getenv("VLLM_CPU_KVCACHE_SPACE", "0")),
214
215 + # OpenVINO key-value cache space
216 + # default is 4GB
217 + "VLLM_OPENVINO_KVCACHE_SPACE":
218 + lambda: int(os.getenv("VLLM_OPENVINO_KVCACHE_SPACE", "0")),
219 +
220 + # OpenVINO KV cache precision
221 + # default is bf16 if natively supported by platform, otherwise f16
222 + # To enable KV cache compression, please, explicitly specify u8
223 + "VLLM_OPENVINO_CPU_KV_CACHE_PRECISION":
224 + lambda: os.getenv("VLLM_OPENVINO_CPU_KV_CACHE_PRECISION", None),
225 +
226 + # Enables weights compression during model export via HF Optimum
227 + # default is False
228 + "VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS":
229 + lambda: bool(os.getenv("VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS", False)),
230 +
231 # If the env var is set, it uses the Ray's compiled DAG API
232 # which optimizes the control plane overhead.
233 # Run vLLM with VLLM_USE_RAY_COMPILED_DAG=1 to enable it.
vllm/executor/openvino_executor.py new
+163
@@ -0,0 +1,163 @@
1 +from typing import List, Set, Tuple
2 +
3 +import openvino as ov
4 +import openvino.properties.hint as hints
5 +import torch
6 +
7 +import vllm.envs as envs
8 +from vllm.config import CacheConfig, ModelConfig
9 +from vllm.executor.executor_base import ExecutorAsyncBase, ExecutorBase
10 +from vllm.logger import init_logger
11 +from vllm.lora.request import LoRARequest
12 +from vllm.sequence import ExecuteModelRequest, SamplerOutput
13 +from vllm.utils import (get_distributed_init_method, get_ip, get_open_port,
14 + make_async)
15 +
16 +logger = init_logger(__name__)
17 +
18 +
19 +class OpenVINOExecutor(ExecutorBase):
20 +
21 + def _init_executor(self) -> None:
22 + assert self.device_config.device_type == "openvino"
23 + assert self.lora_config is None, "OpenVINO backend doesn't support LoRA"
24 + self.model_config = _verify_and_get_model_config(self.model_config)
25 + self.cache_config = _verify_and_get_cache_config(self.cache_config)
26 +
27 + # Instantiate the worker and load the model to CPU.
28 + self._init_worker()
29 +
30 + def _init_worker(self):
31 + from vllm.worker.openvino_worker import OpenVINOWorker
32 +
33 + assert (
34 + self.parallel_config.world_size == 1
35 + ), "OpenVINOExecutor only supports single CPU socket currently."
36 +
37 + distributed_init_method = get_distributed_init_method(
38 + get_ip(), get_open_port())
39 + self.driver_worker = OpenVINOWorker(
40 + model_config=self.model_config,
41 + parallel_config=self.parallel_config,
42 + scheduler_config=self.scheduler_config,
43 + device_config=self.device_config,
44 + cache_config=self.cache_config,
45 + load_config=self.load_config,
46 + local_rank=0,
47 + rank=0,
48 + distributed_init_method=distributed_init_method,
49 + lora_config=self.lora_config,
50 + vision_language_config=self.vision_language_config,
51 + kv_cache_dtype=self.cache_config.cache_dtype,
52 + is_driver_worker=True,
53 + )
54 + self.driver_worker.init_device()
55 + self.driver_worker.load_model()
56 +
57 + def determine_num_available_blocks(self) -> Tuple[int, int]:
58 + """Determine the number of available KV blocks by invoking the
59 + underlying worker.
60 + """
61 + return self.driver_worker.determine_num_available_blocks()
62 +
63 + def initialize_cache(self, num_gpu_blocks: int,
64 + num_cpu_blocks: int) -> None:
65 + """Initialize the KV cache by invoking the underlying worker."""
66 + # NOTE: We log here to avoid multiple logs when number of workers is
67 + # greater than one. We could log in the engine, but not all executors
68 + # have GPUs.
69 + # NOTE: `cpu block` for OpenVINO backend is located on CPU memory but is
70 + # referred as `gpu block`. Because we want to reuse the existing block
71 + # management procedure.
72 + logger.info("# CPU blocks: %d", num_gpu_blocks)
73 + self.driver_worker.initialize_cache(num_gpu_blocks, num_cpu_blocks)
74 +
75 + def execute_model(
76 + self,
77 + execute_model_req: ExecuteModelRequest) -> List[SamplerOutput]:
78 + output = self.driver_worker.execute_model(execute_model_req)
79 + return output
80 +
81 + def add_lora(self, lora_request: LoRARequest) -> bool:
82 + return self.driver_worker.add_lora(lora_request)
83 +
84 + def remove_lora(self, lora_id: int) -> bool:
85 + return self.driver_worker.remove_lora(lora_id)
86 +
87 + def pin_lora(self, lora_id: int) -> bool:
88 + return self.driver_worker.pin_lora(lora_id)
89 +
90 + def list_loras(self) -> Set[int]:
91 + return self.driver_worker.list_loras()
92 +
93 + def check_health(self) -> None:
94 + # OpenVINOExecutor will always be healthy as long as
95 + # it's running.
96 + return
97 +
98 +
99 +class OpenVINOExecutorAsync(OpenVINOExecutor, ExecutorAsyncBase):
100 +
101 + async def execute_model_async(
102 + self,
103 + execute_model_req: ExecuteModelRequest) -> List[SamplerOutput]:
104 + output = await make_async(self.driver_worker.execute_model
105 + )(execute_model_req=execute_model_req, )
106 + return output
107 +
108 + async def check_health_async(self) -> None:
109 + # OpenVINOExecutor will always be healthy as long as
110 + # it's running.
111 + return
112 +
113 +
114 +def _verify_and_get_model_config(config: ModelConfig) -> ModelConfig:
115 + if config.dtype != torch.float32:
116 + logger.warning(
117 + f"Only float32 dtype is supported on OpenVINO, casting from {config.dtype}." # noqa: G004, E501
118 + )
119 + config.dtype = torch.float32
120 + if not config.enforce_eager:
121 + logger.warning(
122 + "CUDA graph is not supported on OpenVINO backend, fallback to the "
123 + "eager mode.")
124 + config.enforce_eager = True
125 + return config
126 +
127 +
128 +def _verify_and_get_cache_config(config: CacheConfig) -> CacheConfig:
129 + if envs.VLLM_OPENVINO_CPU_KV_CACHE_PRECISION == "u8":
130 + logger.info("KV cache type is overried to u8 via "
131 + "VLLM_OPENVINO_CPU_KV_CACHE_PRECISION env var.")
132 + config.cache_dtype = ov.Type.u8
133 + else:
134 + core = ov.Core()
135 + inference_precision = core.get_property("CPU",
136 + hints.inference_precision)
137 + if inference_precision == ov.Type.bf16:
138 + config.cache_dtype = ov.Type.bf16
139 + else:
140 + config.cache_dtype = ov.Type.f16
141 +
142 + if config.block_size != 32:
143 + logger.info(
144 + f"OpenVINO optimal block size is 32, overriding currently set {config.block_size}" # noqa: G004, E501
145 + )
146 + config.block_size = 32
147 +
148 + kv_cache_space = envs.VLLM_OPENVINO_KVCACHE_SPACE
149 + if kv_cache_space >= 0:
150 + _GB = 1 << 30
151 + if kv_cache_space == 0:
152 + config.openvino_kvcache_space_bytes = 4 * _GB # type: ignore
153 + logger.warning(
154 + "Environment variable VLLM_OPENVINO_KVCACHE_SPACE (GB) "
155 + "for OpenVINO backend is not set, using 4 by default.")
156 + else:
157 + config.openvino_kvcache_space_bytes = kv_cache_space * _GB # type: ignore
158 + else:
159 + raise RuntimeError(
160 + "Invalid environment variable VLLM_OPENVINO_KVCACHE_SPACE"
161 + f" {kv_cache_space}, expect a positive integer value.")
162 +
163 + return config
vllm/model_executor/layers/sampler.py
+2 -2
@@ -679,7 +679,7 @@ def _get_ranks(x: torch.Tensor, indices: torch.Tensor) -> torch.Tensor:
679
680 Returns:
681 torch.Tensor: 1D tensor of shape (N,) where N is the no. of tokens.
682 - Each element in the returned tensor represents the rank
682 + Each element in the returned tensor represents the rank
683 of the chosen token in the input logprob tensor.
684 """
685 vals = x[torch.arange(0, len(x), device=x.device, dtype=indices.dtype),
@@ -965,7 +965,7 @@ def _modify_greedy_probs_inplace(logprobs: torch.Tensor, probs: torch.Tensor,
965 distribution.
966 - Greedy sampling performs `argmax` to obtain the token with the
967 highest likelihood.
968 -
968 +
969 Ignoring greedy sampling for a moment, we find that the computed probability
970 distribution has the following property: we can sample from it independently
971 and find that the token sampled by the Sampler has a frequency corresponding
vllm/model_executor/model_loader/openvino.py new
+210
@@ -0,0 +1,210 @@
1 +# ruff: noqa: SIM117
2 +from pathlib import Path
3 +from typing import List, Optional, Tuple
4 +
5 +import openvino as ov
6 +import torch
7 +from huggingface_hub import HfApi
8 +from openvino._offline_transformations import paged_attention_transformation
9 +from optimum.intel import OVModelForCausalLM
10 +from torch import nn
11 +
12 +import vllm.envs as envs
13 +from vllm.attention.backends.openvino import OpenVINOAttentionMetadata
14 +from vllm.config import DeviceConfig, ModelConfig
15 +from vllm.logger import init_logger
16 +from vllm.model_executor.layers.logits_processor import (LogitsProcessor,
17 + _prune_hidden_states)
18 +from vllm.model_executor.layers.sampler import Sampler
19 +from vllm.model_executor.sampling_metadata import SamplingMetadata
20 +from vllm.sequence import SamplerOutput
21 +
22 +logger = init_logger(__name__)
23 +
24 +
25 +def _flattenize_inputs(inputs):
26 + """
27 + Helper function for making nested inputs flattens
28 + """
29 + flatten_inputs = []
30 + for input_data in inputs:
31 + if input_data is None:
32 + continue
33 + if isinstance(input_data, (list, tuple)):
34 + flatten_inputs.extend(_flattenize_inputs(input_data))
35 + elif isinstance(input_data, dict):
36 + flatten_inputs.extend(_flattenize_inputs(list(
37 + input_data.values())))
38 + else:
39 + flatten_inputs.append(input_data)
40 + return flatten_inputs
41 +
42 +
43 +def _modify_cache_parameters(model: ov.Model, kv_cache_dtype: ov.Type,
44 + is_cpu: bool):
45 + # Apply hardware dependent modifications to KV tensors
46 + for parameter in model.get_parameters():
47 + input = parameter.get_output_tensor(0)
48 + input_names = input.get_names()
49 + if len(input_names) != 1:
50 + continue
51 + input_name = next(iter(input_names))
52 + shape = parameter.get_partial_shape()
53 + # use real block size if available, just a placeholder
54 + # to provide the expected rank
55 + x_size = 1
56 + num_blocks = ov.Dimension()
57 + block_size = ov.Dimension()
58 + head_size = ov.Dimension()
59 + # TODO: Negotiate required layout with plugins (CPU is ~OK, GPU is TBD),
60 + # pass more parameters to this function to set more static dimensions
61 + if input_name.startswith("key_cache."):
62 + cpu_shape = [num_blocks, shape[1], block_size, head_size]
63 + gpu_shape = [
64 + num_blocks,
65 + shape[1],
66 + shape[2].get_length() //
67 + x_size if shape[2].is_static else ov.Dimension(),
68 + block_size,
69 + x_size,
70 + ]
71 + elif input_name.startswith("value_cache."):
72 + cpu_shape = [num_blocks, shape[1], block_size, head_size]
73 + gpu_shape = [num_blocks, shape[1], shape[2], block_size]
74 + else:
75 + continue
76 + parameter.set_partial_shape(
77 + ov.PartialShape(cpu_shape if is_cpu else gpu_shape))
78 + parameter.set_element_type(kv_cache_dtype)
79 + model.validate_nodes_and_infer_types()
80 +
81 +
82 +def _require_model_export(model_id, revision=None, subfolder=None):
83 + model_dir = Path(model_id)
84 + if subfolder is not None:
85 + model_dir = model_dir / subfolder
86 + if model_dir.is_dir():
87 + return (not (model_dir / "openvino_model.xml").exists()
88 + or not (model_dir / "openvino_model.bin").exists())
89 +
90 + hf_api = HfApi()
91 + try:
92 + model_info = hf_api.model_info(model_id, revision=revision or "main")
93 + normalized_subfolder = (None if subfolder is None else
94 + Path(subfolder).as_posix())
95 + model_files = [
96 + file.rfilename for file in model_info.siblings
97 + if normalized_subfolder is None
98 + or file.rfilename.startswith(normalized_subfolder)
99 + ]
100 + ov_model_path = ("openvino_model.xml" if normalized_subfolder is None
101 + else f"{normalized_subfolder}/openvino_model.xml")
102 + return (ov_model_path not in model_files
103 + or ov_model_path.replace(".xml", ".bin") not in model_files)
104 + except Exception:
105 + return True
106 +
107 +
108 +class OpenVINOCasualLM(nn.Module):
109 +
110 + def __init__(
111 + self,
112 + model_config: ModelConfig,
113 + device_config: DeviceConfig,
114 + kv_cache_dtype: ov.Type,
115 + ) -> None:
116 + super().__init__()
117 + self.logits_processor = LogitsProcessor(
118 + model_config.hf_config.vocab_size, logits_as_input=True)
119 + self.sampler = Sampler()
120 +
121 + export = _require_model_export(model_config.model)
122 + if export:
123 + logger.warning(
124 + f"Provided model id {model_config.model} does not " # noqa: G004
125 + "contain OpenVINO IR, the model will be converted to IR with "
126 + "default options. If you need to use specific options for "
127 + "model conversion, use optimum-cli export openvino with "
128 + "desired options.")
129 + else:
130 + logger.warning(
131 + "OpenVINO IR is available for provided model id " # noqa: G004
132 + f"{model_config.model}. This IR will be used for inference "
133 + "as-is, all possible options that may affect model conversion "
134 + "are ignored.")
135 +
136 + load_in_8bit = envs.VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS
137 + pt_model = OVModelForCausalLM.from_pretrained(
138 + model_config.model,
139 + export=export,
140 + compile=False,
141 + load_in_8bit=load_in_8bit,
142 + trust_remote_code=model_config.trust_remote_code,
143 + )
144 +
145 + paged_attention_transformation(pt_model.model)
146 + _modify_cache_parameters(pt_model.model, kv_cache_dtype,
147 + device_config.device.type == "cpu")
148 +
149 + core = ov.Core()
150 + ov_compiled = core.compile_model(pt_model.model, "CPU")
151 + self.ov_request = ov_compiled.create_infer_request()
152 +
153 + def forward(
154 + self,
155 + input_ids: torch.Tensor,
156 + positions: torch.Tensor,
157 + kv_caches: List[Tuple[ov.Tensor, ov.Tensor]],
158 + attn_metadata: OpenVINOAttentionMetadata,
159 + ) -> torch.Tensor:
160 + flatten_kv_cache = _flattenize_inputs(kv_caches)
161 +
162 + inputs = [
163 + input_ids,
164 + positions,
165 + *flatten_kv_cache,
166 + attn_metadata.past_lens,
167 + attn_metadata.subsequence_begins,
168 + attn_metadata.block_indices,
169 + attn_metadata.block_indices_begins,
170 + attn_metadata.max_context_len,
171 + ]
172 +
173 + self.ov_request.start_async(inputs, share_inputs=True)
174 + self.ov_request.wait()
175 +
176 + logits = torch.from_numpy(self.ov_request.get_tensor("logits").data)
177 +
178 + # TODO: remove 'view' once OpenVINO PA will drop 'seq_len' dimension
179 + return logits.view(-1, logits.shape[-1])
180 +
181 + def compute_logits(self, hidden_states: torch.Tensor,
182 + sampling_metadata: SamplingMetadata) -> torch.Tensor:
183 + hidden_states = _prune_hidden_states(hidden_states, sampling_metadata)
184 + logits = self.logits_processor(None, hidden_states, sampling_metadata)
185 + return logits
186 +
187 + def sample(
188 + self,
189 + logits: torch.Tensor,
190 + sampling_metadata: SamplingMetadata,
191 + ) -> Optional[SamplerOutput]:
192 + next_tokens = self.sampler(logits, sampling_metadata)
193 + return next_tokens
194 +
195 +
196 +def get_model(
197 + model_config: ModelConfig,
198 + device_config: DeviceConfig,
199 + kv_cache_dtype: ov.Type,
200 + **kwargs,
201 +) -> torch.nn.Module:
202 + lora_config = kwargs.get("lora_config", None)
203 + if lora_config:
204 + raise ValueError(
205 + "OpenVINO modeling does not support LoRA, "
206 + "but LoRA is enabled. Support for this model may "
207 + "be added in the future. If this is important to you, "
208 + "please open an issue on github.")
209 +
210 + return OpenVINOCasualLM(model_config, device_config, kv_cache_dtype)
vllm/utils.py
+10 -1
@@ -176,6 +176,15 @@ def is_cpu() -> bool:
176 return False
177
178
179 +@lru_cache(maxsize=None)
180 +def is_openvino() -> bool:
181 + from importlib.metadata import PackageNotFoundError, version
182 + try:
183 + return "openvino" in version("vllm")
184 + except PackageNotFoundError:
185 + return False
186 +
187 +
188 @lru_cache(maxsize=None)
189 def is_neuron() -> bool:
190 try:
@@ -546,7 +555,7 @@ def is_pin_memory_available() -> bool:
555 elif is_neuron():
556 print_warning_once("Pin memory is not supported on Neuron.")
557 return False
549 - elif is_cpu():
558 + elif is_cpu() or is_openvino():
559 return False
560 return True
561
vllm/worker/openvino_model_runner.py new
+330
@@ -0,0 +1,330 @@
1 +from typing import List, NamedTuple, Optional, Tuple
2 +
3 +import openvino as ov
4 +import torch
5 +from torch import nn
6 +
7 +from vllm.attention import get_attn_backend
8 +from vllm.attention.backends.openvino import OpenVINOAttentionMetadata
9 +from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
10 + ModelConfig, ParallelConfig, SchedulerConfig,
11 + VisionLanguageConfig)
12 +from vllm.logger import init_logger
13 +from vllm.model_executor import SamplingMetadata
14 +from vllm.model_executor.model_loader.openvino import get_model
15 +from vllm.sequence import SamplerOutput, SequenceGroupMetadata
16 +
17 +logger = init_logger(__name__)
18 +
19 +
20 +class ModelInput(NamedTuple):
21 + input_tokens: torch.Tensor
22 + input_positions: torch.Tensor
23 + attn_metadata: Optional[OpenVINOAttentionMetadata]
24 + seq_lens: List[int]
25 + query_lens: List[int]
26 + multi_modal_input: Optional[torch.Tensor]
27 +
28 + @classmethod
29 + def empty(cls, device):
30 + return ModelInput(input_tokens=torch.empty(0, device=device),
31 + input_positions=torch.empty(0, device=device),
32 + attn_metadata=None,
33 + seq_lens=[],
34 + query_lens=[],
35 + multi_modal_input=None)
36 +
37 +
38 +class OpenVINOModelRunner:
39 +
40 + def __init__(
41 + self,
42 + model_config: ModelConfig,
43 + parallel_config: ParallelConfig,
44 + scheduler_config: SchedulerConfig,
45 + device_config: DeviceConfig,
46 + cache_config: CacheConfig,
47 + load_config: LoadConfig,
48 + lora_config: Optional[LoRAConfig],
49 + vision_language_config: Optional[VisionLanguageConfig],
50 + kv_cache_dtype: Optional[str] = "auto",
51 + is_driver_worker: bool = False,
52 + *args,
53 + **kwargs,
54 + ):
55 + self.model_config = model_config
56 + self.parallel_config = parallel_config
57 + self.scheduler_config = scheduler_config
58 + self.device_config = device_config
59 + self.cache_config = cache_config
60 + self.lora_config = lora_config
61 + self.vision_language_config = vision_language_config
62 + self.load_config = load_config
63 + self.is_driver_worker = is_driver_worker
64 +
65 + self.device = self.device_config.device
66 +
67 + self.kv_cache_dtype = kv_cache_dtype
68 + self.sliding_window = model_config.get_sliding_window()
69 + self.block_size = cache_config.block_size
70 +
71 + self.attn_backend = get_attn_backend(
72 + self.model_config.get_num_attention_heads(self.parallel_config),
73 + self.model_config.get_head_size(),
74 + self.model_config.get_num_kv_heads(self.parallel_config),
75 + self.model_config.get_sliding_window(),
76 + self.model_config.dtype,
77 + self.kv_cache_dtype,
78 + self.block_size,
79 + )
80 +
81 + # Lazy initialization.
82 + self.model: nn.Module # Set after init_Model
83 +
84 + def load_model(self) -> None:
85 + self.model = get_model(
86 + model_config=self.model_config,
87 + device_config=self.device_config,
88 + kv_cache_dtype=self.kv_cache_dtype,
89 + )
90 +
91 + def _prepare_model_input(
92 + self,
93 + seq_group_metadata_list: List[SequenceGroupMetadata],
94 + ) -> ModelInput:
95 + """Prepare the model input based on a given sequence group.
96 +
97 + The API assumes seq_group_metadata_list is sorted by prefill -> decode.
98 +
99 + The result tensors and data structure also batches input in prefill
100 + -> decode order. For example,
101 +
102 + - input_tokens[:num_prefill_tokens] contains prefill tokens.
103 + - input_tokens[num_prefill_tokens:] contains decode tokens.
104 + """
105 + input_tokens: List[int] = []
106 + input_positions: List[int] = []
107 +
108 + seq_lens: List[int] = []
109 + past_lens: List[int] = []
110 + query_lens: List[int] = []
111 + subsequence_begins: List[int] = []
112 + block_indices: List[int] = []
113 + block_indices_begins: List[int] = []
114 +
115 + # initialize beginning of prefix sums
116 + subsequence_begins.append(0)
117 + block_indices_begins.append(0)
118 +
119 + if len(seq_group_metadata_list) == 0:
120 + return ModelInput.empty(self.device)
121 +
122 + for seq_group_metadata in seq_group_metadata_list:
123 + seq_ids = list(seq_group_metadata.seq_data.keys())
124 + is_prompt = seq_group_metadata.is_prompt
125 +
126 + for seq_id in seq_ids:
127 + computed_block_nums = seq_group_metadata.computed_block_nums
128 + if (self.scheduler_config is not None
129 + and self.scheduler_config.chunked_prefill_enabled
130 + and not (computed_block_nums is None
131 + or computed_block_nums == [])):
132 + raise RuntimeError(
133 + "chunked prefill cannot be used with prefix caching "
134 + "now.")
135 +
136 + seq_data = seq_group_metadata.seq_data[seq_id]
137 + if is_prompt:
138 + computed_len = seq_data.get_num_computed_tokens()
139 + else:
140 + # get_num_computed_tokens is incorrect for spec decoding.
141 + # So, we should have a special logic here.
142 + # TODO(sang): Fix it.
143 + computed_len = seq_data.get_len() - 1
144 +
145 + seq_len = min(
146 + seq_data.get_len(),
147 + computed_len + seq_group_metadata.token_chunk_size,
148 + )
149 + if is_prompt:
150 + tokens = seq_data.get_token_ids()[computed_len:seq_len]
151 + else:
152 + # Optimization. get_token_ids requires the entire copy of
153 + # tokens.
154 + tokens = [seq_data.get_last_token_id()]
155 +
156 + # Prefix cache was hit.
157 + # Prefix is not supported with sliding_window
158 + prefix_cache_hit = (computed_block_nums is not None
159 + and len(computed_block_nums) > 0
160 + and self.sliding_window is None
161 + and is_prompt)
162 +
163 + block_table = seq_group_metadata.block_tables[seq_id]
164 + # TODO(sang): Combine chunked prefill and prefix caching by
165 + # only allowing multiple of block_size chunk size.
166 + # NOTE: This only works for oooooooxxx style attention.
167 + if prefix_cache_hit:
168 + assert computed_block_nums is not None
169 + computed_len = len(computed_block_nums) * self.block_size
170 + tokens = tokens[computed_len:]
171 + elif (self.scheduler_config.chunked_prefill_enabled
172 + or not is_prompt):
173 + if seq_group_metadata.block_tables is not None:
174 + # chunked prefill or decode
175 + block_table = seq_group_metadata.block_tables[seq_id]
176 + if self.sliding_window is not None:
177 + # chunked prefill doesn't support sliding window.
178 + assert not self.scheduler_config.chunked_prefill_enabled # noqa: E501
179 + sliding_window_blocks = (self.sliding_window //
180 + self.block_size)
181 + block_table = block_table[-sliding_window_blocks:]
182 + else:
183 + # Only happens when memory profiling runs.
184 + block_table = []
185 + else:
186 + # prompt phase w/o prefix_caching, chunked_prefill
187 + pass
188 +
189 + block_indices.extend(block_table)
190 + block_indices_begins.append(block_indices_begins[-1] +
191 + len(block_table))
192 +
193 + # TODO(sang): This is a hack to make sliding window work with
194 + # paged attn. We can remove it if we make paged attn kernel
195 + # to properly handle slinding window attn.
196 + if self.sliding_window is not None and not is_prompt:
197 + seq_len = min(seq_len, self.sliding_window)
198 + computed_len = seq_len - 1
199 +
200 + seq_lens.append(seq_len)
201 +
202 + query_len = seq_len - computed_len
203 + query_lens.append(query_len)
204 +
205 + input_tokens.extend(tokens)
206 + input_positions.extend(list(range(computed_len, seq_len)))
207 +
208 + past_lens.append(computed_len)
209 + subsequence_begins.append(subsequence_begins[-1] + query_len)
210 +
211 + if is_prompt:
212 + assert len(seq_ids) == 1
213 + else:
214 + assert (
215 + query_len == 1
216 + ), "seq_len: {}, computed_len: {}, query_len: {}".format(
217 + seq_len, computed_len, query_len)
218 +
219 + max_query_len = max(query_lens)
220 + assert max_query_len > 0, "query_lens: {}".format(query_lens)
221 +
222 + input_tokens = torch.tensor(input_tokens,
223 + dtype=torch.long,
224 + device=self.device) # type: ignore
225 + input_positions = torch.tensor(input_positions,
226 + dtype=torch.long,
227 + device=self.device) # type: ignore
228 +
229 + past_lens_tensor = torch.tensor(past_lens,
230 + dtype=torch.int32,
231 + device=self.device) # type: ignore
232 + subsequence_begins_tensor = torch.tensor(
233 + subsequence_begins, dtype=torch.int32,
234 + device=self.device) # type: ignore
235 + block_indices_tensor = torch.tensor(block_indices,
236 + dtype=torch.int32,
237 + device=self.device) # type: ignore
238 + block_indices_begins_tensor = torch.tensor(
239 + block_indices_begins, dtype=torch.int32,
240 + device=self.device) # type: ignore
241 +
242 + max_context_len = max(seq_lens)
243 + max_context_len_tensor = torch.tensor(
244 + max_context_len, dtype=torch.int32,
245 + device=self.device) # type: ignore
246 +
247 + attn_metadata = self.attn_backend.make_openvino_metadata(
248 + past_lens=past_lens_tensor,
249 + subsequence_begins=subsequence_begins_tensor,
250 + block_indices=block_indices_tensor,
251 + block_indices_begins=block_indices_begins_tensor,
252 + max_context_len=max_context_len_tensor,
253 + )
254 + return ModelInput(
255 + input_tokens,
256 + input_positions,
257 + attn_metadata,
258 + seq_lens,
259 + query_lens,
260 + None,
261 + )
262 +
263 + def prepare_input_tensors(
264 + self,
265 + seq_group_metadata_list: List[SequenceGroupMetadata],
266 + ) -> Tuple[torch.Tensor, torch.Tensor, OpenVINOAttentionMetadata,
267 + SamplingMetadata, Optional[torch.Tensor], ]:
268 + multi_modal_input = None
269 +
270 + # Prepare input tensors.
271 + (
272 + input_tokens,
273 + input_positions,
274 + attn_metadata,
275 + seq_lens,
276 + query_lens,
277 + multi_modal_input,
278 + ) = self._prepare_model_input(seq_group_metadata_list)
279 +
280 + sampling_metadata = SamplingMetadata.prepare(
281 + seq_group_metadata_list,
282 + seq_lens,
283 + query_lens,
284 + self.device,
285 + pin_memory=False,
286 + )
287 +
288 + return (
289 + input_tokens,
290 + input_positions,
291 + attn_metadata,
292 + sampling_metadata,
293 + multi_modal_input,
294 + )
295 +
296 + @torch.inference_mode()
297 + def execute_model(
298 + self,
299 + seq_group_metadata_list: List[SequenceGroupMetadata],
300 + kv_caches: List[Tuple["ov.Tensor", "ov.Tensor"]],
301 + ) -> Optional[SamplerOutput]:
302 + (
303 + input_tokens,
304 + input_positions,
305 + attn_metadata,
306 + sampling_metadata,
307 + multi_modal_input,
308 + ) = self.prepare_input_tensors(seq_group_metadata_list)
309 +
310 + model_executable = self.model
311 + execute_model_kwargs = {
312 + "input_ids": input_tokens,
313 + "positions": input_positions,
314 + "kv_caches": kv_caches,
315 + "attn_metadata": attn_metadata,
316 + }
317 + if self.vision_language_config:
318 + execute_model_kwargs.update({"image_input": multi_modal_input})
319 +
320 + hidden_states = model_executable(**execute_model_kwargs)
321 +
322 + # Compute the logits.
323 + logits = self.model.compute_logits(hidden_states, sampling_metadata)
324 +
325 + # Sample the next token.
326 + output = self.model.sample(
327 + logits=logits,
328 + sampling_metadata=sampling_metadata,
329 + )
330 + return output
vllm/worker/openvino_worker.py new
+353
@@ -0,0 +1,353 @@
1 +"""An OpenVINO worker class."""
2 +from typing import Any, Dict, List, Optional, Tuple
3 +
4 +import openvino as ov
5 +import torch
6 +import torch.distributed
7 +
8 +from vllm.attention import get_attn_backend
9 +from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
10 + ModelConfig, ParallelConfig, SchedulerConfig,
11 + VisionLanguageConfig)
12 +from vllm.distributed import (broadcast_tensor_dict,
13 + ensure_model_parallel_initialized,
14 + init_distributed_environment)
15 +from vllm.logger import init_logger
16 +from vllm.model_executor import set_random_seed
17 +from vllm.sequence import ExecuteModelRequest, SamplerOutput
18 +from vllm.worker.openvino_model_runner import OpenVINOModelRunner
19 +from vllm.worker.worker_base import LoraNotSupportedWorkerBase
20 +
21 +logger = init_logger(__name__)
22 +
23 +
24 +class OpenVINOCacheEngine:
25 + """Manages the KV cache for OpenVINO backend.
26 +
27 + This class is responsible for initializing and managing CPU KV
28 + caches. It also provides methods for performing KV cache operations, such
29 + as copying.
30 + """
31 +
32 + def __init__(
33 + self,
34 + cache_config: CacheConfig,
35 + model_config: ModelConfig,
36 + parallel_config: ParallelConfig,
37 + device_config: DeviceConfig,
38 + ) -> None:
39 + assert device_config.device_type == "openvino"
40 + self.cache_config = cache_config
41 + self.model_config = model_config
42 + self.parallel_config = parallel_config
43 +
44 + self.head_size = model_config.get_head_size()
45 + if device_config.device.type == "cpu" and \
46 + cache_config.cache_dtype == ov.Type.u8:
47 + # Scale, zero point and quantized data will be stored together.
48 + # The layout for per token per head:
49 + # |scale(f32)|zeropoint(f32)|quantized data(u8,idx_1)|quantized data(u8,idx_2)|...|quantized data(u8,idx_head_size)| # noqa: E501
50 + # so, we have to extend head_size by 8, which is sizeof(float)
51 + # for scale and sizeof(float) for zeropoint
52 + self.head_size += 8
53 + self.num_layers = model_config.get_num_layers(parallel_config)
54 + self.num_kv_heads = model_config.get_num_kv_heads(parallel_config)
55 +
56 + self.block_size = cache_config.block_size
57 + # Note: In CacheConfig, num_gpu_blocks actual is num_cpu_blocks
58 + # for OpenVINO backend, because we want to reuse KV cache management
59 + # in the scheduler.
60 + self.num_cpu_blocks = cache_config.num_gpu_blocks
61 +
62 + # Get attention backend.
63 + self.attn_backend = get_attn_backend(
64 + self.model_config.get_num_attention_heads(self.parallel_config),
65 + self.head_size,
66 + self.model_config.get_num_kv_heads(self.parallel_config),
67 + self.model_config.get_sliding_window(),
68 + self.model_config.dtype,
69 + self.cache_config.cache_dtype,
70 + self.block_size,
71 + )
72 +
73 + # Initialize the cache.
74 + self.kv_cache: List[Tuple[ov.Tensor,
75 + ov.Tensor]] = self._allocate_kv_cache(
76 + self.num_cpu_blocks)
77 +
78 + def _allocate_kv_cache(
79 + self,
80 + num_blocks: int,
81 + ) -> List[Tuple[ov.Tensor, ov.Tensor]]:
82 + """Allocates KV cache."""
83 + k_block_shape = v_block_shape = self.attn_backend.get_kv_cache_shape(
84 + num_blocks, self.block_size, self.num_kv_heads, self.head_size)[1:]
85 + kv_cache: List[Tuple[ov.Tensor, ov.Tensor]] = []
86 + for _ in range(self.num_layers):
87 + key_blocks = ov.Tensor(self.cache_config.cache_dtype,
88 + k_block_shape)
89 + value_blocks = ov.Tensor(self.cache_config.cache_dtype,
90 + v_block_shape)
91 + kv_cache.append((key_blocks, value_blocks))
92 + return kv_cache
93 +
94 + def swap_in(self, src_to_dst: Dict[int, int]) -> None:
95 + raise NotImplementedError(
96 + "Swap is not supported in OpenVINOCacheEngine.")
97 +
98 + def swap_out(self, src_to_dst: Dict[int, int]) -> None:
99 + raise NotImplementedError(
100 + "Swap is not supported in OpenVINOCacheEngine.")
101 +
102 + def copy(self, src_to_dsts: Dict[int, List[int]]) -> None:
103 + self.attn_backend.copy_blocks(self.kv_cache, src_to_dsts)
104 +
105 + @staticmethod
106 + def get_cache_block_size(
107 + block_size: int,
108 + cache_dtype: ov.Type,
109 + model_config: ModelConfig,
110 + parallel_config: ParallelConfig,
111 + ) -> int:
112 + head_size = model_config.get_head_size()
113 + num_kv_heads = model_config.get_num_kv_heads(parallel_config)
114 + num_layers = model_config.get_num_layers(parallel_config)
115 +
116 + if cache_dtype == ov.Type.u8:
117 + # Scale, zero point and quantized data will be stored together.
118 + # The layout for per token per head:
119 + # |scale(f32)|zeropoint(f32)|quantized data(u8,idx_1)|quantized data(u8,idx_2)|...|quantized data(u8,idx_head_size)| # noqa: E501
120 + # so, we have to extend head_size by 8, which is sizeof(float)
121 + # for scale and sizeof(float) for zeropoint
122 + head_size += 8
123 +
124 + key_cache_block = block_size * num_kv_heads * head_size
125 + value_cache_block = key_cache_block
126 + total = num_layers * (key_cache_block + value_cache_block)
127 + dtype_size = cache_dtype.size
128 + return dtype_size * total
129 +
130 +
131 +class OpenVINOWorker(LoraNotSupportedWorkerBase):
132 + """A worker class that executes the model on OpenVINO backend.
133 +
134 + Each worker is associated with a single OpenVINO device. The worker is
135 + responsible for maintaining the KV cache and executing the model on the
136 + OpenVINO backend.
137 + """
138 +
139 + def __init__(
140 + self,
141 + model_config: ModelConfig,
142 + parallel_config: ParallelConfig,
143 + scheduler_config: SchedulerConfig,
144 + device_config: DeviceConfig,
145 + cache_config: CacheConfig,
146 + load_config: LoadConfig,
147 + local_rank: int,
148 + rank: int,
149 + distributed_init_method: str,
150 + lora_config: Optional[LoRAConfig] = None,
151 + vision_language_config: Optional[VisionLanguageConfig] = None,
152 + kv_cache_dtype: Optional[ov.Type] = ov.Type.undefined,
153 + is_driver_worker: bool = False,
154 + ) -> None:
155 + self.model_config = model_config
156 + self.parallel_config = parallel_config
157 + self.scheduler_config = scheduler_config
158 + self.device_config = device_config
159 + self.cache_config = cache_config
160 + self.load_config = load_config
161 + self.local_rank = local_rank
162 + self.rank = rank
163 + self.distributed_init_method = distributed_init_method
164 + self.lora_config = lora_config
165 + self.vision_language_config = vision_language_config
166 + self.is_driver_worker = is_driver_worker
167 + if self.is_driver_worker:
168 + assert self.rank == 0, "The driver worker must have rank 0."
169 +
170 + if self.model_config.trust_remote_code:
171 + # note: lazy import to avoid importing torch before initializing
172 + from vllm.utils import init_cached_hf_modules
173 +
174 + init_cached_hf_modules()
175 + self.model_runner = OpenVINOModelRunner(
176 + model_config,
177 + parallel_config,
178 + scheduler_config,
179 + device_config,
180 + cache_config,
181 + load_config=self.load_config,
182 + lora_config=self.lora_config,
183 + vision_language_config=self.vision_language_config,
184 + kv_cache_dtype=kv_cache_dtype,
185 + is_driver_worker=is_driver_worker,
186 + )
187 + # Uninitialized cache engine. Will be initialized by
188 + # initialize_cache.
189 + self.cache_engine: OpenVINOCacheEngine
190 + self.kv_cache: List[Tuple[ov.Tensor, ov.Tensor]]
191 +
192 + def init_device(self) -> None:
193 + self.init_distributed_environment()
194 + # Set random seed.
195 + set_random_seed(self.model_config.seed)
196 +
197 + def load_model(self):
198 + self.model_runner.load_model()
199 +
200 + def determine_num_available_blocks(self) -> Tuple[int, int]:
201 + """Determine the number of blocks available for the KV cache.
202 +
203 + This determines how many KV blocks can fit into the configured
204 + KV cache space.
205 +
206 + Note that since vLLM assumes a block resides on GPU if it can be
207 + modified, we return num_gpu_blocks=num_cpu_blocks and num_cpu_blocks=0.
208 + This allows us to reuse the scheduler of vLLM without generalizing it
209 + to different devices.
210 + """
211 + # For OpenVINO backend, the block number will be calculated based on the
212 + # openvino_kvcache_space_bytes.
213 + cache_block_size = self.get_cache_block_size_bytes()
214 + num_cpu_blocks = int(self.cache_config.openvino_kvcache_space_bytes //
215 + cache_block_size)
216 + num_cpu_blocks = max(num_cpu_blocks, 0)
217 +
218 + # Note: To reuse the cache management procedure,
219 + # use cpu cache as 'gpu cache'.
220 + num_gpu_blocks = num_cpu_blocks
221 + num_cpu_blocks = 0
222 + return num_gpu_blocks, num_cpu_blocks
223 +
224 + def initialize_cache(self, num_gpu_blocks: int,
225 + num_cpu_blocks: int) -> None:
226 + """Initialize the KV cache. Currently, swappable CPU memory is not
227 + supported.
228 +
229 + Since this worker does not support GPUs, we use the num_gpu_blocks to
230 + determine how many non-swappable CPU blocks to allocate.
231 + """
232 + assert (num_cpu_blocks == 0
233 + ), f"{type(self)} does not support swappable cache"
234 +
235 + # Note: To reuse the cache management procedure,
236 + # use cpu cache as 'gpu cache'.
237 + num_cpu_blocks = num_gpu_blocks
238 +
239 + self._validate_num_cpu_blocks(num_cpu_blocks)
240 + self.cache_config.num_gpu_blocks = num_cpu_blocks
241 + self.cache_config.num_cpu_blocks = 0
242 +
243 + # Initialize the cache.
244 + self._init_cache_engine()
245 +
246 + def _validate_num_cpu_blocks(self, num_cpu_blocks: int) -> None:
247 + """Raise errors if the num_cpu_blocks is invalid."""
248 + if num_cpu_blocks <= 0:
249 + raise ValueError(
250 + "No available memory for the cache blocks. "
251 + "Try increasing `VLLM_OPENVINO_KVCACHE_SPACE` when "
252 + "initializing the engine.")
253 +
254 + max_seq_len = self.cache_config.block_size * num_cpu_blocks
255 + if self.model_config.max_model_len > max_seq_len:
256 + raise ValueError(
257 + f"The model's max seq len ({self.model_config.max_model_len}) "
258 + "is larger than the maximum number of tokens that can be "
259 + f"stored in KV cache ({max_seq_len}). Try increasing "
260 + "`VLLM_OPENVINO_KVCACHE_SPACE` or decreasing `max_model_len` "
261 + "when initializing the engine.")
262 +
263 + def _init_cache_engine(self) -> None:
264 + self.cache_engine = OpenVINOCacheEngine(
265 + self.cache_config,
266 + self.model_config,
267 + self.parallel_config,
268 + self.device_config,
269 + )
270 + self.kv_cache = self.cache_engine.kv_cache
271 + self.model_runner.block_size = self.cache_engine.block_size
272 +
273 + assert self.kv_cache is not None
274 +
275 + # Populate the cache to warmup the memory
276 + for key_cache, value_cache in self.kv_cache:
277 + key_cache.data[:] = 0
278 + value_cache.data[:] = 0
279 +
280 + def cache_copy(
281 + self,
282 + blocks_to_copy: List[Tuple[int, int]],
283 + ) -> None:
284 + self.cache_engine.copy(blocks_to_copy) # type: ignore
285 +
286 + @torch.inference_mode()
287 + def execute_model(
288 + self,
289 + execute_model_req: Optional[ExecuteModelRequest] = None,
290 + ) -> List[SamplerOutput]:
291 + if execute_model_req is None:
292 + seq_group_metadata_list = None
293 + else:
294 + seq_group_metadata_list = execute_model_req.seq_group_metadata_list
295 +
296 + if self.is_driver_worker:
297 + assert seq_group_metadata_list is not None
298 + num_seq_groups: int = len(seq_group_metadata_list)
299 + assert execute_model_req is not None
300 + blocks_to_copy = execute_model_req.blocks_to_copy
301 + assert len(execute_model_req.blocks_to_swap_in) == 0
302 + assert len(execute_model_req.blocks_to_swap_out) == 0
303 + data: Dict[str, Any] = {
304 + "num_seq_groups": num_seq_groups,
305 + "blocks_to_copy": execute_model_req.blocks_to_copy,
306 + }
307 + broadcast_tensor_dict(data, src=0)
308 + else:
309 + data = broadcast_tensor_dict(src=0)
310 + num_seq_groups = data["num_seq_groups"]
311 + blocks_to_copy = data["blocks_to_copy"]
312 +
313 + self.cache_copy(blocks_to_copy)
314 +
315 + # If there is no input, we don't need to execute the model.
316 + if num_seq_groups == 0:
317 + return []
318 +
319 + output = self.model_runner.execute_model(seq_group_metadata_list,
320 + self.kv_cache)
321 +
322 + # OpenVINO worker only supports single-step execution.
323 + return [output]
324 +
325 + def init_distributed_environment(self) -> None:
326 + """Initialize the distributed environment."""
327 +
328 + parallel_config = self.parallel_config
329 + rank = self.rank
330 + distributed_init_method = self.distributed_init_method
331 + init_distributed_environment(
332 + world_size=parallel_config.world_size,
333 + rank=rank,
334 + distributed_init_method=distributed_init_method,
335 + backend="gloo",
336 + )
337 +
338 + # A small all_reduce for warmup.
339 + torch.distributed.all_reduce(torch.zeros(1).cpu())
340 +
341 + ensure_model_parallel_initialized(
342 + parallel_config.tensor_parallel_size,
343 + parallel_config.pipeline_parallel_size,
344 + )
345 +
346 + def get_cache_block_size_bytes(self) -> int:
347 + """Return the size in bytes of a single KV cache block."""
348 + return OpenVINOCacheEngine.get_cache_block_size(
349 + self.cache_config.block_size,
350 + self.cache_config.cache_dtype,
351 + self.model_config,
352 + self.parallel_config,
353 + )