@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / a02a50e6e5
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 08:15 UTC

[Hardware][Intel-Gaudi] Add Intel Gaudi (HPU) inference backend (#6143)

Signed-off-by: yuwenzho <yuwen.zhou@intel.com> Signed-off-by: Chendi.Xue <chendi.xue@intel.com> Signed-off-by: Bob Zhu <bob.zhu@intel.com> Signed-off-by: zehao-intel <zehao.huang@intel.com> Signed-off-by: Konrad Zawora <kzawora@habana.ai> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com> Co-authored-by: Sanju C Sudhakaran <scsudhakaran@habana.ai> Co-authored-by: Michal Adamczyk <madamczyk@habana.ai> Co-authored-by: Marceli Fylcek <mfylcek@habana.ai> Co-authored-by: Himangshu Lahkar <49579433+hlahkar@users.noreply.github.com> Co-authored-by: Vivek Goel <vgoel@habana.ai> Co-authored-by: yuwenzho <yuwen.zhou@intel.com> Co-authored-by: Dominika Olszewska <dolszewska@habana.ai> Co-authored-by: barak goldberg <149692267+bgoldberg-habana@users.noreply.github.com> Co-authored-by: Michal Szutenberg <37601244+szutenberg@users.noreply.github.com> Co-authored-by: Jan Kaniecki <jkaniecki@habana.ai> Co-authored-by: Agata Dobrzyniewicz <160237065+adobrzyniewicz-habana@users.noreply.github.com> Co-authored-by: Krzysztof Wisniewski <kwisniewski@habana.ai> Co-authored-by: Dudi Lester <160421192+dudilester@users.noreply.github.com> Co-authored-by: Ilia Taraban <tarabanil@gmail.com> Co-authored-by: Chendi.Xue <chendi.xue@intel.com> Co-authored-by: Michał Kuligowski <mkuligowski@habana.ai> Co-authored-by: Jakub Maksymczuk <jmaksymczuk@habana.ai> Co-authored-by: Tomasz Zielinski <85164140+tzielinski-habana@users.noreply.github.com> Co-authored-by: Sun Choi <schoi@habana.ai> Co-authored-by: Iryna Boiko <iboiko@habana.ai> Co-authored-by: Bob Zhu <41610754+czhu15@users.noreply.github.com> Co-authored-by: hlin99 <73271530+hlin99@users.noreply.github.com> Co-authored-by: Zehao Huang <zehao.huang@intel.com> Co-authored-by: Andrzej Kotłowski <Andrzej.Kotlowski@intel.com> Co-authored-by: Yan Tomsinsky <73292515+Yantom1@users.noreply.github.com> Co-authored-by: Nir David <ndavid@habana.ai> Co-authored-by: Yu-Zhou <yu.zhou@intel.com> Co-authored-by: Ruheena Suhani Shaik <rsshaik@habana.ai> Co-authored-by: Karol Damaszke <kdamaszke@habana.ai> Co-authored-by: Marcin Swiniarski <mswiniarski@habana.ai> Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> Co-authored-by: Jacek Czaja <jacek.czaja@intel.com> Co-authored-by: Jacek Czaja <jczaja@habana.ai> Co-authored-by: Yuan <yuan.zhou@outlook.com>

Konrad Zawora committed Nov 6, 2024 at 10:09 UTC a02a50e6e5bb74f9d48f75942e47197d22ec6444
31 files changed +4279 -20
Dockerfile.hpu new
+16
@@ -0,0 +1,16 @@
1 +FROM vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest
2 +
3 +COPY ./ /workspace/vllm
4 +
5 +WORKDIR /workspace/vllm
6 +
7 +RUN pip install -v -r requirements-hpu.txt
8 +
9 +ENV no_proxy=localhost,127.0.0.1
10 +ENV PT_HPU_ENABLE_LAZY_COLLECTIVES=true
11 +
12 +RUN VLLM_TARGET_DEVICE=hpu python3 setup.py install
13 +
14 +WORKDIR /workspace/
15 +
16 +ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"]
docs/source/getting_started/gaudi-installation.rst new
+402
@@ -0,0 +1,402 @@
1 +Installation with Intel® Gaudi® AI Accelerators
2 +===============================================
3 +
4 +This README provides instructions on running vLLM with Intel Gaudi devices.
5 +
6 +Requirements and Installation
7 +=============================
8 +
9 +Please follow the instructions provided in the `Gaudi Installation
10 +Guide <https://docs.habana.ai/en/latest/Installation_Guide/index.html>`__
11 +to set up the execution environment. To achieve the best performance,
12 +please follow the methods outlined in the `Optimizing Training Platform
13 +Guide <https://docs.habana.ai/en/latest/PyTorch/Model_Optimization_PyTorch/Optimization_in_Training_Platform.html>`__.
14 +
15 +Requirements
16 +------------
17 +
18 +- OS: Ubuntu 22.04 LTS
19 +- Python: 3.10
20 +- Intel Gaudi accelerator
21 +- Intel Gaudi software version 1.18.0
22 +
23 +
24 +Quick start using Dockerfile
25 +----------------------------
26 +.. code:: console
27 +
28 + $ docker build -f Dockerfile.hpu -t vllm-hpu-env .
29 + $ docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --rm vllm-hpu-env
30 +
31 +
32 +.. tip::
33 + If you're observing the following error: ``docker: Error response from daemon: Unknown runtime specified habana.``, please refer to "Install Using Containers" section of `Intel Gaudi Software Stack and Driver Installation <https://docs.habana.ai/en/v1.18.0/Installation_Guide/Bare_Metal_Fresh_OS.html>`__. Make sure you have ``habana-container-runtime`` package installed and that ``habana`` container runtime is registered.
34 +
35 +
36 +Build from source
37 +-----------------
38 +
39 +Environment verification
40 +~~~~~~~~~~~~~~~~~~~~~~~~
41 +
42 +To verify that the Intel Gaudi software was correctly installed, run:
43 +
44 +.. code:: console
45 +
46 + $ hl-smi # verify that hl-smi is in your PATH and each Gaudi accelerator is visible
47 + $ apt list --installed | grep habana # verify that habanalabs-firmware-tools, habanalabs-graph, habanalabs-rdma-core, habanalabs-thunk and habanalabs-container-runtime are installed
48 + $ pip list | grep habana # verify that habana-torch-plugin, habana-torch-dataloader, habana-pyhlml and habana-media-loader are installed
49 + $ pip list | grep neural # verify that neural_compressor is installed
50 +
51 +Refer to `Intel Gaudi Software Stack
52 +Verification <https://docs.habana.ai/en/latest/Installation_Guide/SW_Verification.html#platform-upgrade>`__
53 +for more details.
54 +
55 +Run Docker Image
56 +~~~~~~~~~~~~~~~~
57 +
58 +It is highly recommended to use the latest Docker image from Intel Gaudi
59 +vault. Refer to the `Intel Gaudi
60 +documentation <https://docs.habana.ai/en/latest/Installation_Guide/Bare_Metal_Fresh_OS.html#pull-prebuilt-containers>`__
61 +for more details.
62 +
63 +Use the following commands to run a Docker image:
64 +
65 +.. code:: console
66 +
67 + $ docker pull vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest
68 + $ docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --ipc=host vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest
69 +
70 +Build and Install vLLM
71 +~~~~~~~~~~~~~~~~~~~~~~
72 +
73 +To build and install vLLM from source, run:
74 +
75 +.. code:: console
76 +
77 + $ git clone https://github.com/vllm-project/vllm.git
78 + $ cd vllm
79 + $ python setup.py develop
80 +
81 +
82 +Currently, the latest features and performance optimizations are developed in Gaudi's `vLLM-fork <https://github.com/HabanaAI/vllm-fork>`__ and we periodically upstream them to vLLM main repo. To install latest `HabanaAI/vLLM-fork <https://github.com/HabanaAI/vllm-fork>`__, run the following:
83 +
84 +.. code:: console
85 +
86 + $ git clone https://github.com/HabanaAI/vllm-fork.git
87 + $ cd vllm-fork
88 + $ git checkout habana_main
89 + $ python setup.py develop
90 +
91 +
92 +Supported Features
93 +==================
94 +
95 +- `Offline batched
96 + inference <https://docs.vllm.ai/en/latest/getting_started/quickstart.html#offline-batched-inference>`__
97 +- Online inference via `OpenAI-Compatible
98 + Server <https://docs.vllm.ai/en/latest/getting_started/quickstart.html#openai-compatible-server>`__
99 +- HPU autodetection - no need to manually select device within vLLM
100 +- Paged KV cache with algorithms enabled for Intel Gaudi accelerators
101 +- Custom Intel Gaudi implementations of Paged Attention, KV cache ops,
102 + prefill attention, Root Mean Square Layer Normalization, Rotary
103 + Positional Encoding
104 +- Tensor parallelism support for multi-card inference
105 +- Inference with `HPU Graphs <https://docs.habana.ai/en/latest/PyTorch/Inference_on_PyTorch/Inference_Using_HPU_Graphs.html>`__
106 + for accelerating low-batch latency and throughput
107 +- Attention with Linear Biases (ALiBi)
108 +
109 +Unsupported Features
110 +====================
111 +
112 +- Beam search
113 +- LoRA adapters
114 +- Quantization
115 +- Prefill chunking (mixed-batch inferencing)
116 +
117 +Supported Configurations
118 +========================
119 +
120 +The following configurations have been validated to be function with
121 +Gaudi2 devices. Configurations that are not listed may or may not work.
122 +
123 +- `meta-llama/Llama-2-7b <https://huggingface.co/meta-llama/Llama-2-7b>`__
124 + on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
125 + datatype with random or greedy sampling
126 +- `meta-llama/Llama-2-7b-chat-hf <https://huggingface.co/meta-llama/Llama-2-7b-chat-hf>`__
127 + on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
128 + datatype with random or greedy sampling
129 +- `meta-llama/Meta-Llama-3-8B <https://huggingface.co/meta-llama/Meta-Llama-3-8B>`__
130 + on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
131 + datatype with random or greedy sampling
132 +- `meta-llama/Meta-Llama-3-8B-Instruct <https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct>`__
133 + on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
134 + datatype with random or greedy sampling
135 +- `meta-llama/Meta-Llama-3.1-8B <https://huggingface.co/meta-llama/Meta-Llama-3.1-8B>`__
136 + on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
137 + datatype with random or greedy sampling
138 +- `meta-llama/Meta-Llama-3.1-8B-Instruct <https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct>`__
139 + on single HPU, or with tensor parallelism on 2x and 8x HPU, BF16
140 + datatype with random or greedy sampling
141 +- `meta-llama/Llama-2-70b <https://huggingface.co/meta-llama/Llama-2-70b>`__
142 + with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
143 +- `meta-llama/Llama-2-70b-chat-hf <https://huggingface.co/meta-llama/Llama-2-70b-chat-hf>`__
144 + with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
145 +- `meta-llama/Meta-Llama-3-70B <https://huggingface.co/meta-llama/Meta-Llama-3-70B>`__
146 + with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
147 +- `meta-llama/Meta-Llama-3-70B-Instruct <https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct>`__
148 + with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
149 +- `meta-llama/Meta-Llama-3.1-70B <https://huggingface.co/meta-llama/Meta-Llama-3.1-70B>`__
150 + with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
151 +- `meta-llama/Meta-Llama-3.1-70B-Instruct <https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct>`__
152 + with tensor parallelism on 8x HPU, BF16 datatype with random or greedy sampling
153 +
154 +Performance Tuning
155 +==================
156 +
157 +Execution modes
158 +---------------
159 +
160 +Currently in vLLM for HPU we support four execution modes, depending on selected HPU PyTorch Bridge backend (via ``PT_HPU_LAZY_MODE`` environment variable), and ``--enforce-eager`` flag.
161 +
162 +.. list-table:: vLLM execution modes
163 + :widths: 25 25 50
164 + :header-rows: 1
165 +
166 + * - ``PT_HPU_LAZY_MODE``
167 + - ``enforce_eager``
168 + - execution mode
169 + * - 0
170 + - 0
171 + - torch.compile
172 + * - 0
173 + - 1
174 + - PyTorch eager mode
175 + * - 1
176 + - 0
177 + - HPU Graphs
178 + * - 1
179 + - 1
180 + - PyTorch lazy mode
181 +
182 +.. warning::
183 + In 1.18.0, all modes utilizing ``PT_HPU_LAZY_MODE=0`` are highly experimental and should be only used for validating functional correctness. Their performance will be improved in the next releases. For obtaining the best performance in 1.18.0, please use HPU Graphs, or PyTorch lazy mode.
184 +
185 +
186 +Bucketing mechanism
187 +-------------------
188 +
189 +Intel Gaudi accelerators work best when operating on models with fixed tensor shapes. `Intel Gaudi Graph Compiler <https://docs.habana.ai/en/latest/Gaudi_Overview/Intel_Gaudi_Software_Suite.html#graph-compiler-and-runtime>`__ is responsible for generating optimized binary code that implements the given model topology on Gaudi. In its default configuration, the produced binary code may be heavily dependent on input and output tensor shapes, and can require graph recompilation when encountering differently shaped tensors within the same topology. While the resulting binaries utilize Gaudi efficiently, the compilation itself may introduce a noticeable overhead in end-to-end execution.
190 +In a dynamic inference serving scenario, there is a need to minimize the number of graph compilations and reduce the risk of graph compilation occurring during server runtime. Currently it is achieved by "bucketing" model's forward pass across two dimensions - ``batch_size`` and ``sequence_length``.
191 +
192 +.. note::
193 + Bucketing allows us to reduce the number of required graphs significantly, but it does not handle any graph compilation and device code generation - this is done in warmup and HPUGraph capture phase.
194 +
195 +Bucketing ranges are determined with 3 parameters - ``min``, ``step`` and ``max``. They can be set separately for prompt and decode phase, and for batch size and sequence length dimension. These parameters can be observed in logs during vLLM startup:
196 +
197 +.. code-block::
198 +
199 + INFO 08-01 21:37:59 hpu_model_runner.py:493] Prompt bucket config (min, step, max_warmup) bs:[1, 32, 4], seq:[128, 128, 1024]
200 + INFO 08-01 21:37:59 hpu_model_runner.py:499] Generated 24 prompt buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024)]
201 + INFO 08-01 21:37:59 hpu_model_runner.py:504] Decode bucket config (min, step, max_warmup) bs:[1, 128, 4], seq:[128, 128, 2048]
202 + INFO 08-01 21:37:59 hpu_model_runner.py:509] Generated 48 decode buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]
203 +
204 +``min`` determines the lowest value of the bucket. ``step`` determines the interval between buckets, and ``max`` determines the upper bound of the bucket. Furthermore, interval between ``min`` and ``step`` has special handling - ``min`` gets multiplied by consecutive powers of two, until ``step`` gets reached. We call this the ramp-up phase and it is used for handling lower batch sizes with minimum wastage, while allowing larger padding on larger batch sizes.
205 +
206 +Example (with ramp-up)
207 +
208 +.. code-block::
209 +
210 + min = 2, step = 32, max = 64
211 + => ramp_up = (2, 4, 8, 16)
212 + => stable = (32, 64)
213 + => buckets = ramp_up + stable => (2, 4, 8, 16, 32, 64)
214 +
215 +Example (without ramp-up)
216 +
217 +.. code-block::
218 +
219 + min = 128, step = 128, max = 512
220 + => ramp_up = ()
221 + => stable = (128, 256, 384, 512)
222 + => buckets = ramp_up + stable => (128, 256, 384, 512)
223 +
224 +
225 +In the logged scenario, 24 buckets were generated for prompt (prefill) runs, and 48 buckets for decode runs. Each bucket corresponds to a separate optimized device binary for a given model with specified tensor shapes. Whenever a batch of requests is processed, it is padded across batch and sequence length dimension to the smallest possible bucket.
226 +
227 +.. warning::
228 + If a request exceeds maximum bucket size in any dimension, it will be processed without padding, and its processing may require a graph compilation, potentially significantly increasing end-to-end latency. The boundaries of the buckets are user-configurable via environment variables, and upper bucket boundaries can be increased to avoid such scenario.
229 +
230 +As an example, if a request of 3 sequences, with max sequence length of 412 comes in to an idle vLLM server, it will be padded executed as ``(4, 512)`` prefill bucket, as ``batch_size`` (number of sequences) will be padded to 4 (closest batch_size dimension higher than 3), and max sequence length will be padded to 512 (closest sequence length dimension higher than 412). After prefill stage, it will be executed as ``(4, 512)`` decode bucket and will continue as that bucket until either batch dimension changes (due to request being finished) - in which case it will become a ``(2, 512)`` bucket, or context length increases above 512 tokens, in which case it will become ``(4, 640)`` bucket.
231 +
232 +.. note::
233 + Bucketing is transparent to a client - padding in sequence length dimension is never returned to the client, and padding in batch dimension does not create new requests.
234 +
235 +Warmup
236 +------
237 +
238 +Warmup is an optional, but highly recommended step occurring before vLLM server starts listening. It executes a forward pass for each bucket with dummy data. The goal is to pre-compile all graphs and not incur any graph compilation overheads within bucket boundaries during server runtime. Each warmup step is logged during vLLM startup:
239 +
240 +.. code-block::
241 +
242 + INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:79.16 GiB
243 + INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][2/24] batch_size:4 seq_len:896 free_mem:55.43 GiB
244 + INFO 08-01 22:26:48 hpu_model_runner.py:1066] [Warmup][Prompt][3/24] batch_size:4 seq_len:768 free_mem:55.43 GiB
245 + ...
246 + INFO 08-01 22:26:59 hpu_model_runner.py:1066] [Warmup][Prompt][24/24] batch_size:1 seq_len:128 free_mem:55.43 GiB
247 + INFO 08-01 22:27:00 hpu_model_runner.py:1066] [Warmup][Decode][1/48] batch_size:4 seq_len:2048 free_mem:55.43 GiB
248 + INFO 08-01 22:27:00 hpu_model_runner.py:1066] [Warmup][Decode][2/48] batch_size:4 seq_len:1920 free_mem:55.43 GiB
249 + INFO 08-01 22:27:01 hpu_model_runner.py:1066] [Warmup][Decode][3/48] batch_size:4 seq_len:1792 free_mem:55.43 GiB
250 + ...
251 + INFO 08-01 22:27:16 hpu_model_runner.py:1066] [Warmup][Decode][47/48] batch_size:2 seq_len:128 free_mem:55.43 GiB
252 + INFO 08-01 22:27:16 hpu_model_runner.py:1066] [Warmup][Decode][48/48] batch_size:1 seq_len:128 free_mem:55.43 GiB
253 +
254 +This example uses the same buckets as in *Bucketing mechanism* section. Each output line corresponds to execution of a single bucket. When bucket is executed for the first time, its graph is compiled and can be reused later on, skipping further graph compilations.
255 +
256 +.. tip::
257 + Compiling all the buckets might take some time and can be turned off with ``VLLM_SKIP_WARMUP=true`` environment variable. Keep in mind that if you do that, you may face graph compilations once executing a given bucket for the first time. It is fine to disable warmup for development, but it's highly recommended to enable it in deployment.
258 +
259 +HPU Graph capture
260 +-----------------
261 +
262 +`HPU Graphs <https://docs.habana.ai/en/latest/PyTorch/Inference_on_PyTorch/Inference_Using_HPU_Graphs.html>`__ are currently the most performant execution method of vLLM on Intel Gaudi. When HPU Graphs are enabled, execution graphs will be traced (recorded) ahead of time (after performing warmup), to be later replayed during inference, significantly reducing host overheads. Recording can take large amounts of memory, which needs to be taken into account when allocating KV cache. Enabling HPU Graphs will impact the number of available KV cache blocks, but vLLM provides user-configurable variables to control memory management.
263 +
264 +
265 +When HPU Graphs are being used, they share the common memory pool ("usable memory") as KV cache, determined by ``gpu_memory_utilization`` flag (``0.9`` by default).
266 +Before KV cache gets allocated, model weights are loaded onto the device, and a forward pass of the model is executed on dummy data, to estimate memory usage.
267 +Only after that, ``gpu_memory_utilization`` flag is utilized - at its default value, will mark 90% of free device memory at that point as usable.
268 +Next, KV cache gets allocated, model is warmed up, and HPU Graphs are captured.
269 +Environment variable ``VLLM_GRAPH_RESERVED_MEM`` defines the ratio of memory reserved for HPU Graphs capture.
270 +With its default value (``VLLM_GRAPH_RESERVED_MEM=0.1``), 10% of usable memory will be reserved for graph capture (later referred to as "usable graph memory"), and the remaining 90% will be utilized for KV cache.
271 +Environment variable ``VLLM_GRAPH_PROMPT_RATIO`` determines the ratio of usable graph memory reserved for prefill and decode graphs. By default (``VLLM_GRAPH_PROMPT_RATIO=0.3``), both stages have equal memory constraints.
272 +Lower value corresponds to less usable graph memory reserved for prefill stage, e.g. ``VLLM_GRAPH_PROMPT_RATIO=0.2`` will reserve 20% of usable graph memory for prefill graphs, and 80% of usable graph memory for decode graphs.
273 +
274 +.. note::
275 + ``gpu_memory_utilization`` does not correspond to the absolute memory usage across HPU. It specifies the memory margin after loading the model and performing a profile run. If device has 100 GiB of total memory, and 50 GiB of free memory after loading model weights and executing profiling run, ``gpu_memory_utilization`` at its default value will mark 90% of 50 GiB as usable, leaving 5 GiB of margin, regardless of total device memory.
276 +
277 +User can also configure the strategy for capturing HPU Graphs for prompt and decode stages separately. Strategy affects the order of capturing graphs. There are two strategies implemented:
278 +- ``max_bs`` - graph capture queue will sorted in descending order by their batch sizes. Buckets with equal batch sizes are sorted by sequence length in ascending order (e.g. ``(64, 128)``, ``(64, 256)``, ``(32, 128)``, ``(32, 256)``, ``(1, 128)``, ``(1,256)``), default strategy for decode
279 +- ``min_tokens`` - graph capture queue will be sorted in ascending order by the number of tokens each graph processes (``batch_size*sequence_length``), default strategy for prompt
280 +
281 +When there's large amount of requests pending, vLLM scheduler will attempt to fill the maximum batch size for decode as soon as possible. When a request is finished, decode batch size decreases. When that happens, vLLM will attempt to schedule a prefill iteration for requests in the waiting queue, to fill the decode batch size to its previous state. This means that in a full load scenario, decode batch size is often at its maximum, which makes large batch size HPU Graphs crucial to capture, as reflected by ``max_bs`` strategy. On the other hand, prefills will be executed most frequently with very low batch sizes (1-4), which is reflected in ``min_tokens`` strategy.
282 +
283 +
284 +.. note::
285 + ``VLLM_GRAPH_PROMPT_RATIO`` does not set a hard limit on memory taken by graphs for each stage (prefill and decode). vLLM will first attempt to use up entirety of usable prefill graph memory (usable graph memory * ``VLLM_GRAPH_PROMPT_RATIO``) for capturing prefill HPU Graphs, next it will attempt do the same for decode graphs and usable decode graph memory pool. If one stage is fully captured, and there is unused memory left within usable graph memory pool, vLLM will attempt further graph capture for the other stage, until no more HPU Graphs can be captured without exceeding reserved memory pool. The behavior on that mechanism can be observed in the example below.
286 +
287 +
288 +Each described step is logged by vLLM server, as follows (negative values correspond to memory being released):
289 +
290 +.. code-block::
291 +
292 + INFO 08-02 17:37:44 hpu_model_runner.py:493] Prompt bucket config (min, step, max_warmup) bs:[1, 32, 4], seq:[128, 128, 1024]
293 + INFO 08-02 17:37:44 hpu_model_runner.py:499] Generated 24 prompt buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024)]
294 + INFO 08-02 17:37:44 hpu_model_runner.py:504] Decode bucket config (min, step, max_warmup) bs:[1, 128, 4], seq:[128, 128, 2048]
295 + INFO 08-02 17:37:44 hpu_model_runner.py:509] Generated 48 decode buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]
296 + INFO 08-02 17:37:52 hpu_model_runner.py:430] Pre-loading model weights on hpu:0 took 14.97 GiB of device memory (14.97 GiB/94.62 GiB used) and 2.95 GiB of host memory (475.2 GiB/1007 GiB used)
297 + INFO 08-02 17:37:52 hpu_model_runner.py:438] Wrapping in HPU Graph took 0 B of device memory (14.97 GiB/94.62 GiB used) and -252 KiB of host memory (475.2 GiB/1007 GiB used)
298 + INFO 08-02 17:37:52 hpu_model_runner.py:442] Loading model weights took in total 14.97 GiB of device memory (14.97 GiB/94.62 GiB used) and 2.95 GiB of host memory (475.2 GiB/1007 GiB used)
299 + INFO 08-02 17:37:54 hpu_worker.py:134] Model profiling run took 504 MiB of device memory (15.46 GiB/94.62 GiB used) and 180.9 MiB of host memory (475.4 GiB/1007 GiB used)
300 + INFO 08-02 17:37:54 hpu_worker.py:158] Free device memory: 79.16 GiB, 39.58 GiB usable (gpu_memory_utilization=0.5), 15.83 GiB reserved for HPUGraphs (VLLM_GRAPH_RESERVED_MEM=0.4), 23.75 GiB reserved for KV cache
301 + INFO 08-02 17:37:54 hpu_executor.py:85] # HPU blocks: 1519, # CPU blocks: 0
302 + INFO 08-02 17:37:54 hpu_worker.py:190] Initializing cache engine took 23.73 GiB of device memory (39.2 GiB/94.62 GiB used) and -1.238 MiB of host memory (475.4 GiB/1007 GiB used)
303 + INFO 08-02 17:37:54 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:55.43 GiB
304 + ...
305 + INFO 08-02 17:38:22 hpu_model_runner.py:1066] [Warmup][Decode][48/48] batch_size:1 seq_len:128 free_mem:55.43 GiB
306 + INFO 08-02 17:38:22 hpu_model_runner.py:1159] Using 15.85 GiB/55.43 GiB of free device memory for HPUGraphs, 7.923 GiB for prompt and 7.923 GiB for decode (VLLM_GRAPH_PROMPT_RATIO=0.3)
307 + INFO 08-02 17:38:22 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][1/24] batch_size:1 seq_len:128 free_mem:55.43 GiB
308 + ...
309 + INFO 08-02 17:38:26 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][11/24] batch_size:1 seq_len:896 free_mem:48.77 GiB
310 + INFO 08-02 17:38:27 hpu_model_runner.py:1066] [Warmup][Graph/Decode][1/48] batch_size:4 seq_len:128 free_mem:47.51 GiB
311 + ...
312 + INFO 08-02 17:38:41 hpu_model_runner.py:1066] [Warmup][Graph/Decode][48/48] batch_size:1 seq_len:2048 free_mem:47.35 GiB
313 + INFO 08-02 17:38:41 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][12/24] batch_size:4 seq_len:256 free_mem:47.35 GiB
314 + INFO 08-02 17:38:42 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][13/24] batch_size:2 seq_len:512 free_mem:45.91 GiB
315 + INFO 08-02 17:38:42 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][14/24] batch_size:1 seq_len:1024 free_mem:44.48 GiB
316 + INFO 08-02 17:38:43 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][15/24] batch_size:2 seq_len:640 free_mem:43.03 GiB
317 + INFO 08-02 17:38:43 hpu_model_runner.py:1128] Graph/Prompt captured:15 (62.5%) used_mem:14.03 GiB buckets:[(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (4, 128), (4, 256)]
318 + INFO 08-02 17:38:43 hpu_model_runner.py:1128] Graph/Decode captured:48 (100.0%) used_mem:161.9 MiB buckets:[(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]
319 + INFO 08-02 17:38:43 hpu_model_runner.py:1206] Warmup finished in 49 secs, allocated 14.19 GiB of device memory
320 + INFO 08-02 17:38:43 hpu_executor.py:91] init_cache_engine took 37.92 GiB of device memory (53.39 GiB/94.62 GiB used) and 57.86 MiB of host memory (475.4 GiB/1007 GiB used)
321 +
322 +
323 +Recommended vLLM Parameters
324 +---------------------------
325 +
326 +- We recommend running inference on Gaudi 2 with ``block_size`` of 128
327 + for BF16 data type. Using default values (16, 32) might lead to
328 + sub-optimal performance due to Matrix Multiplication Engine
329 + under-utilization (see `Gaudi
330 + Architecture <https://docs.habana.ai/en/latest/Gaudi_Overview/Gaudi_Architecture.html>`__).
331 +- For max throughput on Llama 7B, we recommend running with batch size
332 + of 128 or 256 and max context length of 2048 with HPU Graphs enabled.
333 + If you encounter out-of-memory issues, see troubleshooting section.
334 +
335 +Environment variables
336 +---------------------
337 +
338 +**Diagnostic and profiling knobs:**
339 +
340 +- ``VLLM_PROFILER_ENABLED``: if ``true``, high level profiler will be enabled. Resulting JSON traces can be viewed in `perfetto.habana.ai <https://perfetto.habana.ai/#!/viewer>`__. Disabled by default.
341 +- ``VLLM_HPU_LOG_STEP_GRAPH_COMPILATION``: if ``true``, will log graph compilations per each vLLM engine step, only when there was any - highly recommended to use alongside ``PT_HPU_METRICS_GC_DETAILS=1``. Disabled by default.
342 +- ``VLLM_HPU_LOG_STEP_GRAPH_COMPILATION_ALL``: if ``true``, will log graph compilations per each vLLM engine step, always, even if there were none. Disabled by default.
343 +- ``VLLM_HPU_LOG_STEP_CPU_FALLBACKS``: if ``true``, will log cpu fallbacks per each vLLM engine step, only when there was any. Disabled by default.
344 +- ``VLLM_HPU_LOG_STEP_CPU_FALLBACKS_ALL``: if ``true``, will log cpu fallbacks per each vLLM engine step, always, even if there were none. Disabled by default.
345 +
346 +**Performance tuning knobs:**
347 +
348 +- ``VLLM_SKIP_WARMUP``: if ``true``, warmup will be skipped, ``false`` by default
349 +- ``VLLM_GRAPH_RESERVED_MEM``: percentage of memory dedicated for HPUGraph capture, ``0.1`` by default
350 +- ``VLLM_GRAPH_PROMPT_RATIO``: percentage of reserved graph memory dedicated for prompt graphs, ``0.3`` by default
351 +- ``VLLM_GRAPH_PROMPT_STRATEGY``: strategy determining order of prompt graph capture, ``min_tokens`` or ``max_bs``, ``min_tokens`` by default
352 +- ``VLLM_GRAPH_DECODE_STRATEGY``: strategy determining order of decode graph capture, ``min_tokens`` or ``max_bs``, ``max_bs`` by default
353 +- ``VLLM_{phase}_{dim}_BUCKET_{param}`` - collection of 12 environment variables configuring ranges of bucketing mechanism
354 +
355 + - ``{phase}`` is either ``PROMPT`` or ``DECODE``
356 + - ``{dim}`` is either ``BS``, ``SEQ`` or ``BLOCK``
357 + - ``{param}`` is either ``MIN``, ``STEP`` or ``MAX``
358 + - Default values:
359 +
360 + - Prompt:
361 + - batch size min (``VLLM_PROMPT_BS_BUCKET_MIN``): ``1``
362 + - batch size step (``VLLM_PROMPT_BS_BUCKET_STEP``): ``min(max_num_seqs, 32)``
363 + - batch size max (``VLLM_PROMPT_BS_BUCKET_MAX``): ``min(max_num_seqs, 64)``
364 + - sequence length min (``VLLM_PROMPT_SEQ_BUCKET_MIN``): ``block_size``
365 + - sequence length step (``VLLM_PROMPT_SEQ_BUCKET_STEP``): ``block_size``
366 + - sequence length max (``VLLM_PROMPT_SEQ_BUCKET_MAX``): ``max_model_len``
367 +
368 + - Decode:
369 + - batch size min (``VLLM_DECODE_BS_BUCKET_MIN``): ``1``
370 + - batch size step (``VLLM_DECODE_BS_BUCKET_STEP``): ``min(max_num_seqs, 32)``
371 + - batch size max (``VLLM_DECODE_BS_BUCKET_MAX``): ``max_num_seqs``
372 + - sequence length min (``VLLM_DECODE_BLOCK_BUCKET_MIN``): ``block_size``
373 + - sequence length step (``VLLM_DECODE_BLOCK_BUCKET_STEP``): ``block_size``
374 + - sequence length max (``VLLM_DECODE_BLOCK_BUCKET_MAX``): ``max(128, (max_num_seqs*max_model_len)/block_size)``
375 +
376 +
377 +Additionally, there are HPU PyTorch Bridge environment variables impacting vLLM execution:
378 +
379 +- ``PT_HPU_LAZY_MODE``: if ``0``, PyTorch Eager backend for Gaudi will be used, if ``1`` PyTorch Lazy backend for Gaudi will be used, ``1`` is default
380 +- ``PT_HPU_ENABLE_LAZY_COLLECTIVES``: required to be ``true`` for tensor parallel inference with HPU Graphs
381 +
382 +Troubleshooting: Tweaking HPU Graphs
383 +====================================
384 +
385 +If you experience device out-of-memory issues or want to attempt
386 +inference at higher batch sizes, try tweaking HPU Graphs by following
387 +the below:
388 +
389 +- Tweak ``gpu_memory_utilization`` knob. It will decrease the
390 + allocation of KV cache, leaving some headroom for capturing graphs
391 + with larger batch size. By default ``gpu_memory_utilization`` is set
392 + to 0.9. It attempts to allocate ~90% of HBM left for KV cache after
393 + short profiling run. Note that decreasing reduces the number of KV
394 + cache blocks you have available, and therefore reduces the effective
395 + maximum number of tokens you can handle at a given time.
396 +
397 +- If this method is not efficient, you can disable ``HPUGraph``
398 + completely. With HPU Graphs disabled, you are trading latency and
399 + throughput at lower batches for potentially higher throughput on
400 + higher batches. You can do that by adding ``--enforce-eager`` flag to
401 + server (for online inference), or by passing ``enforce_eager=True``
402 + argument to LLM constructor (for offline inference).
docs/source/index.rst
+2 -1
@@ -43,7 +43,7 @@ vLLM is flexible and easy to use with:
43 * Tensor parallelism and pipeline parallelism support for distributed inference
44 * Streaming outputs
45 * OpenAI-compatible API server
46 -* Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Trainium and Inferentia Accelerators.
46 +* Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs, Gaudi® accelerators and GPUs, PowerPC CPUs, TPU, and AWS Trainium and Inferentia Accelerators.
47 * Prefix caching support
48 * Multi-lora support
49
@@ -66,6 +66,7 @@ Documentation
66 getting_started/amd-installation
67 getting_started/openvino-installation
68 getting_started/cpu-installation
69 + getting_started/gaudi-installation
70 getting_started/neuron-installation
71 getting_started/tpu-installation
72 getting_started/xpu-installation
requirements-hpu.txt new
+11
@@ -0,0 +1,11 @@
1 +# Common dependencies
2 +-r requirements-common.txt
3 +
4 +# Dependencies for HPU code
5 +ray
6 +triton
7 +pandas
8 +tabulate
9 +setuptools>=61
10 +setuptools-scm>=8
11 +vllm-hpu-extension @ git+https://github.com/HabanaAI/vllm-hpu-extension.git@fd7f2e6
setup.py
+45 -2
@@ -253,6 +253,24 @@ class cmake_build_ext(build_ext):
253 self.copy_file(file, dst_file)
254
255
256 +def _is_hpu() -> bool:
257 + is_hpu_available = True
258 + try:
259 + subprocess.run(["hl-smi"], capture_output=True, check=True)
260 + except (FileNotFoundError, PermissionError, subprocess.CalledProcessError):
261 + if not os.path.exists('/dev/accel/accel0') and not os.path.exists(
262 + '/dev/accel/accel_controlD0'):
263 + # last resort...
264 + try:
265 + output = subprocess.check_output(
266 + 'lsmod | grep habanalabs | wc -l', shell=True)
267 + is_hpu_available = int(output) > 0
268 + except (ValueError, FileNotFoundError, PermissionError,
269 + subprocess.CalledProcessError):
270 + is_hpu_available = False
271 + return is_hpu_available or VLLM_TARGET_DEVICE == "hpu"
272 +
273 +
274 def _no_device() -> bool:
275 return VLLM_TARGET_DEVICE == "empty"
276
@@ -260,7 +278,7 @@ def _no_device() -> bool:
278 def _is_cuda() -> bool:
279 has_cuda = torch.version.cuda is not None
280 return (VLLM_TARGET_DEVICE == "cuda" and has_cuda
263 - and not (_is_neuron() or _is_tpu()))
281 + and not (_is_neuron() or _is_tpu() or _is_hpu()))
282
283
284 def _is_hip() -> bool:
@@ -356,6 +374,23 @@ def get_path(*filepath) -> str:
374 return os.path.join(ROOT_DIR, *filepath)
375
376
377 +def get_gaudi_sw_version():
378 + """
379 + Returns the driver version.
380 + """
381 + # Enable console printing for `hl-smi` check
382 + output = subprocess.run("hl-smi",
383 + shell=True,
384 + text=True,
385 + stdout=subprocess.PIPE,
386 + stderr=subprocess.PIPE,
387 + env={"ENABLE_CONSOLE": "true"})
388 + if output.returncode == 0 and output.stdout:
389 + return output.stdout.split("\n")[2].replace(
390 + " ", "").split(":")[1][:-1].split("-")[0]
391 + return "0.0.0" # when hl-smi is not available
392 +
393 +
394 def get_vllm_version() -> str:
395 version = get_version(
396 write_to="vllm/_version.py", # TODO: move this to pyproject.toml
@@ -385,6 +420,12 @@ def get_vllm_version() -> str:
420 if neuron_version != MAIN_CUDA_VERSION:
421 neuron_version_str = neuron_version.replace(".", "")[:3]
422 version += f"{sep}neuron{neuron_version_str}"
423 + elif _is_hpu():
424 + # Get the Intel Gaudi Software Suite version
425 + gaudi_sw_version = str(get_gaudi_sw_version())
426 + if gaudi_sw_version != MAIN_CUDA_VERSION:
427 + gaudi_sw_version = gaudi_sw_version.replace(".", "")[:3]
428 + version += f"{sep}gaudi{gaudi_sw_version}"
429 elif _is_openvino():
430 version += f"{sep}openvino"
431 elif _is_tpu():
@@ -443,6 +484,8 @@ def get_requirements() -> List[str]:
484 requirements = _read_requirements("requirements-rocm.txt")
485 elif _is_neuron():
486 requirements = _read_requirements("requirements-neuron.txt")
487 + elif _is_hpu():
488 + requirements = _read_requirements("requirements-hpu.txt")
489 elif _is_openvino():
490 requirements = _read_requirements("requirements-openvino.txt")
491 elif _is_tpu():
@@ -453,7 +496,7 @@ def get_requirements() -> List[str]:
496 requirements = _read_requirements("requirements-xpu.txt")
497 else:
498 raise ValueError(
456 - "Unsupported platform, please use CUDA, ROCm, Neuron, "
499 + "Unsupported platform, please use CUDA, ROCm, Neuron, HPU, "
500 "OpenVINO, or CPU.")
501 return requirements
502
vllm/_custom_ops.py
+1 -1
@@ -12,7 +12,7 @@ from vllm.scalar_type import ScalarType
12
13 logger = init_logger(__name__)
14
15 -if not current_platform.is_tpu():
15 +if not current_platform.is_tpu() and not current_platform.is_hpu():
16 try:
17 import vllm._C
18 except ImportError as e:
vllm/attention/backends/hpu_attn.py new
+264
@@ -0,0 +1,264 @@
1 +###############################################################################
2 +# Copyright (C) 2024 Habana Labs, Ltd. an Intel Company
3 +###############################################################################
4 +
5 +import os
6 +from dataclasses import dataclass
7 +from typing import Any, Dict, List, Optional, Tuple, Type
8 +
9 +import torch
10 +import vllm_hpu_extension.ops as ops
11 +from vllm_hpu_extension.utils import Matmul, Softmax, VLLMKVCache
12 +
13 +from vllm.attention.backends.abstract import (AttentionBackend, AttentionImpl,
14 + AttentionMetadata, AttentionType)
15 +from vllm.attention.backends.utils import CommonAttentionState
16 +from vllm.attention.ops.hpu_paged_attn import (HPUPagedAttention,
17 + HPUPagedAttentionMetadata)
18 +from vllm.logger import init_logger
19 +
20 +logger = init_logger(__name__)
21 +
22 +
23 +class HPUAttentionBackend(AttentionBackend):
24 +
25 + @staticmethod
26 + def get_impl_cls() -> Type["HPUAttentionImpl"]:
27 + return HPUAttentionImpl
28 +
29 + @staticmethod
30 + def get_metadata_cls() -> Type["AttentionMetadata"]:
31 + return HPUAttentionMetadata
32 +
33 + @staticmethod
34 + def get_state_cls() -> Type["CommonAttentionState"]:
35 + return CommonAttentionState
36 +
37 + @staticmethod
38 + def get_kv_cache_shape(
39 + num_blocks: int,
40 + block_size: int,
41 + num_kv_heads: int,
42 + head_size: int,
43 + ) -> Tuple[int, ...]:
44 + return HPUPagedAttention.get_kv_cache_shape(num_blocks, block_size,
45 + num_kv_heads, head_size)
46 +
47 + @staticmethod
48 + def swap_blocks(
49 + src_kv_cache: torch.Tensor,
50 + dst_kv_cache: torch.Tensor,
51 + src_to_dst: Dict[int, int],
52 + ) -> None:
53 + HPUPagedAttention.swap_blocks(src_kv_cache, dst_kv_cache, src_to_dst)
54 +
55 + @staticmethod
56 + def copy_blocks(
57 + kv_caches: List[torch.Tensor],
58 + src_to_dists: Dict[int, List[int]],
59 + ) -> None:
60 + HPUPagedAttention.copy_blocks(kv_caches, src_to_dists)
61 +
62 +
63 +@dataclass
64 +class HPUAttentionMetadata(HPUPagedAttentionMetadata, AttentionMetadata):
65 + """Metadata for HPUAttentionbackend."""
66 + # Currently, input sequences can only contain all prompts
67 + # or all decoding. True if all sequences are prompts.
68 + is_prompt: bool
69 + attn_bias: Optional[torch.Tensor]
70 + seq_lens_tensor: Optional[torch.Tensor]
71 +
72 +
73 +class HPUAttentionImpl(AttentionImpl, torch.nn.Module):
74 + """
75 + If the input tensors contain prompt tokens, the layout is as follows:
76 + |<--------------- num_prefill_tokens ----------------->|
77 + |<--prefill_0-->|<--prefill_1-->|...|<--prefill_N-1--->|
78 +
79 + Otherwise, the layout is as follows:
80 + |<----------------- num_decode_tokens ------------------>|
81 + |<--decode_0-->|..........|<--decode_M-1-->|<--padding-->|
82 +
83 + Generation tokens can contain padding when cuda-graph is used.
84 + Currently, prompt tokens don't contain any padding.
85 +
86 + The prompts might have different lengths, while the generation tokens
87 + always have length 1.
88 + """
89 +
90 + def __init__(
91 + self,
92 + num_heads: int,
93 + head_size: int,
94 + scale: float,
95 + num_kv_heads: int,
96 + alibi_slopes: Optional[List[float]],
97 + sliding_window: Optional[int],
98 + kv_cache_dtype: str,
99 + blocksparse_params: Optional[Dict[str, Any]] = None,
100 + max_seq_len: int = 4096,
101 + ) -> None:
102 + super(AttentionImpl, self).__init__()
103 + self.kv_cache_dtype = kv_cache_dtype
104 + self.num_heads = num_heads
105 + self.head_size = head_size
106 + self.scale = float(scale)
107 + self.matmul_qk = Matmul()
108 + self.softmax = Softmax()
109 + self.matmul_av = Matmul()
110 + self.k_cache = VLLMKVCache()
111 + self.v_cache = VLLMKVCache()
112 + self.num_kv_heads = num_heads if num_kv_heads is None else num_kv_heads
113 + self.sliding_window = sliding_window
114 + self.alibi_slopes = alibi_slopes
115 + if alibi_slopes is not None:
116 + alibi_slopes_tensor = torch.tensor(alibi_slopes,
117 + dtype=torch.bfloat16)
118 + self.alibi_slopes = alibi_slopes_tensor
119 + assert self.num_heads % self.num_kv_heads == 0
120 + self.num_queries_per_kv = self.num_heads // self.num_kv_heads
121 +
122 + self.prefill_usefusedsdpa = os.getenv('VLLM_PROMPT_USE_FUSEDSDPA',
123 + '0').lower() in ['1', 'true']
124 + if self.prefill_usefusedsdpa:
125 + assert alibi_slopes is None, \
126 + 'Prefill with FusedSDPA not supported with alibi slopes!'
127 +
128 + suppored_head_sizes = HPUPagedAttention.get_supported_head_sizes()
129 + if head_size not in suppored_head_sizes:
130 + raise ValueError(
131 + f"Head size {head_size} is not supported by PagedAttention. "
132 + f"Supported head sizes are: {suppored_head_sizes}.")
133 +
134 + def forward(
135 + self,
136 + query: torch.Tensor,
137 + key: torch.Tensor,
138 + value: torch.Tensor,
139 + kv_cache: torch.Tensor,
140 + attn_metadata: HPUAttentionMetadata,
141 + k_scale: float = 1.0,
142 + v_scale: float = 1.0,
143 + attn_type: AttentionType = AttentionType.DECODER,
144 + ) -> torch.Tensor:
145 + """Forward pass with xFormers and PagedAttention.
146 +
147 + Args:
148 + query: shape = [num_tokens, num_heads * head_size]
149 + key: shape = [num_tokens, num_kv_heads * head_size]
150 + value: shape = [num_tokens, num_kv_heads * head_size]
151 + kv_cache = [2, num_blocks, block_size * num_kv_heads * head_size]
152 + attn_metadata: Metadata for attention.
153 + Returns:
154 + shape = [num_tokens, num_heads * head_size]
155 + """
156 + if attn_type != AttentionType.DECODER:
157 + raise NotImplementedError("Encoder self-attention and "
158 + "encoder/decoder cross-attention "
159 + "are not implemented for "
160 + "HPUAttentionImpl")
161 + batch_size, seq_len, hidden_size = query.shape
162 + _, seq_len_kv, _ = key.shape
163 +
164 + query = query.view(-1, self.num_heads, self.head_size)
165 + key = key.view(-1, self.num_kv_heads, self.head_size)
166 + value = value.view(-1, self.num_kv_heads, self.head_size)
167 + block_indices = attn_metadata.block_indices
168 + block_offsets = attn_metadata.block_offsets
169 + if attn_metadata.is_prompt:
170 + key = key.unflatten(0, (block_indices.size(0), -1))
171 + value = value.unflatten(0, (block_indices.size(0), -1))
172 + if kv_cache is not None:
173 + key_cache, value_cache = HPUPagedAttention.split_kv_cache(
174 + kv_cache, self.num_kv_heads, self.head_size)
175 +
176 + # Reshape the input keys and values and store them in the cache.
177 + # If kv_cache is not provided, the new key and value tensors are
178 + # not cached. This happens during the initial memory profiling run.
179 + key_cache = self.k_cache(key, key_cache, block_indices,
180 + block_offsets)
181 + value_cache = self.v_cache(value, value_cache, block_indices,
182 + block_offsets)
183 +
184 + if attn_metadata.is_prompt:
185 + # Prompt run.
186 + if not self.prefill_usefusedsdpa:
187 + # TODO: move this outside of model
188 + assert attn_metadata.attn_bias is not None, \
189 + 'attn_bias must be set before calling model.forward!'
190 + attn_bias = attn_metadata.attn_bias
191 + if self.alibi_slopes is not None:
192 + position_bias = _make_alibi_bias(self.alibi_slopes,
193 + self.num_kv_heads,
194 + attn_bias.dtype,
195 + attn_bias.shape[-1])
196 + attn_bias = attn_bias.tile((1, self.num_kv_heads, 1, 1))
197 + attn_bias.add_(position_bias)
198 + else:
199 + attn_bias = None
200 +
201 + query_shape = (batch_size, seq_len, self.num_heads, self.head_size)
202 + kv_shape = (batch_size, seq_len_kv, self.num_kv_heads,
203 + self.head_size)
204 + out = ops.prompt_attention(
205 + query.view(query_shape),
206 + key.view(kv_shape),
207 + value.view(kv_shape),
208 + attn_bias=attn_bias,
209 + p=0.0,
210 + scale=self.scale,
211 + matmul_qk_op=self.matmul_qk,
212 + softmax_op=self.softmax,
213 + matmul_av_op=self.matmul_av,
214 + )
215 + output = out.reshape(batch_size, seq_len, hidden_size)
216 + else:
217 + # Decoding run.
218 + output = HPUPagedAttention.forward_decode(
219 + query=query,
220 + key_cache=key_cache,
221 + value_cache=value_cache,
222 + block_list=attn_metadata.block_list,
223 + block_mapping=attn_metadata.block_mapping,
224 + block_bias=attn_metadata.attn_bias,
225 + block_scales=attn_metadata.block_scales,
226 + scale=self.scale,
227 + matmul_qk_op=self.matmul_qk,
228 + matmul_av_op=self.matmul_av,
229 + keys_fetch_func=self.k_cache.fetch_from_cache,
230 + values_fetch_func=self.v_cache.fetch_from_cache)
231 + # Reshape the output tensor.
232 + return output.view(batch_size, seq_len, hidden_size)
233 +
234 +
235 +def _make_alibi_bias(
236 + alibi_slopes: torch.Tensor,
237 + num_kv_heads: int,
238 + dtype: torch.dtype,
239 + seq_len: int,
240 +) -> torch.Tensor:
241 + bias = torch.arange(seq_len, dtype=dtype)
242 + # NOTE(zhuohan): HF uses
243 + # `bias = bias[None, :].repeat(seq_len, 1)`
244 + # here. We find that both biases give the same results, but
245 + # the bias below more accurately follows the original ALiBi
246 + # paper.
247 + # Calculate a matrix where each element represents ith element- jth
248 + # element.
249 + bias = bias[None, :] - bias[:, None]
250 +
251 + padded_len = (seq_len + 7) // 8 * 8
252 + num_heads = alibi_slopes.shape[0]
253 + bias = torch.empty(
254 + 1, # batch size
255 + num_heads,
256 + seq_len,
257 + padded_len,
258 + device=alibi_slopes.device,
259 + dtype=dtype,
260 + )[:, :, :, :seq_len].copy_(bias)
261 + bias.mul_(alibi_slopes[:, None, None])
262 + if num_heads != num_kv_heads:
263 + bias = bias.unflatten(1, (num_kv_heads, num_heads // num_kv_heads))
264 + return bias
vllm/attention/ops/hpu_paged_attn.py new
+103
@@ -0,0 +1,103 @@
1 +###############################################################################
2 +# Copyright (C) 2024 Habana Labs, Ltd. an Intel Company
3 +###############################################################################
4 +
5 +from dataclasses import dataclass
6 +from typing import Dict, List, Optional, Tuple
7 +
8 +import torch
9 +from vllm_hpu_extension import cache_ops, ops
10 +
11 +# Should be the same as PARTITION_SIZE in `paged_attention_v2_launcher`.
12 +_PARTITION_SIZE = 512
13 +
14 +
15 +@dataclass
16 +class HPUPagedAttentionMetadata:
17 + """Metadata for PagedAttention."""
18 + block_list: Optional[torch.Tensor]
19 + block_mapping: Optional[torch.Tensor]
20 + block_usage: Optional[torch.Tensor]
21 + block_indices: Optional[torch.Tensor]
22 + block_offsets: Optional[torch.Tensor]
23 + block_scales: Optional[torch.Tensor]
24 +
25 +
26 +class HPUPagedAttention:
27 +
28 + @staticmethod
29 + def get_supported_head_sizes() -> List[int]:
30 + return [64, 80, 96, 112, 128, 256]
31 +
32 + @staticmethod
33 + def get_kv_cache_shape(
34 + num_blocks: int,
35 + block_size: int,
36 + num_kv_heads: int,
37 + head_size: int,
38 + ) -> Tuple[int, ...]:
39 + return (num_blocks, block_size, num_kv_heads, head_size)
40 +
41 + @staticmethod
42 + def split_kv_cache(
43 + kv_cache: torch.Tensor,
44 + num_kv_heads: int,
45 + head_size: int,
46 + ) -> Tuple[torch.Tensor, torch.Tensor]:
47 + key_cache = kv_cache[0]
48 + value_cache = kv_cache[1]
49 + return key_cache, value_cache
50 +
51 + @staticmethod
52 + def write_to_paged_cache(key: torch.Tensor, value: torch.Tensor,
53 + key_cache: torch.Tensor,
54 + value_cache: torch.Tensor,
55 + slot_mapping: torch.Tensor, kv_cache_dtype: str,
56 + is_prompt: bool) -> None:
57 + cache_ops.reshape_and_cache(key, value, key_cache, value_cache,
58 + slot_mapping, kv_cache_dtype, is_prompt)
59 +
60 + @staticmethod
61 + def forward_decode(**kwargs) -> torch.Tensor:
62 + return ops.flat_pa(**kwargs)
63 +
64 + @staticmethod
65 + def forward_prefix(
66 + query: torch.Tensor,
67 + key: torch.Tensor,
68 + value: torch.Tensor,
69 + key_cache: torch.Tensor,
70 + value_cache: torch.Tensor,
71 + block_tables: torch.Tensor,
72 + subquery_start_loc: torch.Tensor,
73 + seq_lens_tensor: torch.Tensor,
74 + context_lens: torch.Tensor,
75 + max_query_len: int,
76 + alibi_slopes: Optional[torch.Tensor],
77 + sliding_window: Optional[int],
78 + ) -> torch.Tensor:
79 + raise NotImplementedError(
80 + "forward_prefix is not implemented for HPUPagedAttention")
81 +
82 + @staticmethod
83 + def swap_blocks(
84 + src_kv_cache: torch.Tensor,
85 + dst_kv_cache: torch.Tensor,
86 + src_to_dst: Dict[int, int],
87 + ) -> None:
88 + src_key_cache = src_kv_cache[0]
89 + dst_key_cache = dst_kv_cache[0]
90 + cache_ops.swap_blocks(src_key_cache, dst_key_cache, src_to_dst)
91 +
92 + src_value_cache = src_kv_cache[1]
93 + dst_value_cache = dst_kv_cache[1]
94 + cache_ops.swap_blocks(src_value_cache, dst_value_cache, src_to_dst)
95 +
96 + @staticmethod
97 + def copy_blocks(
98 + kv_caches: List[torch.Tensor],
99 + src_to_dists: Dict[int, List[int]],
100 + ) -> None:
101 + key_caches = [kv_cache[0] for kv_cache in kv_caches]
102 + value_caches = [kv_cache[1] for kv_cache in kv_caches]
103 + cache_ops.copy_blocks(key_caches, value_caches, src_to_dists)
vllm/attention/selector.py
+8
@@ -23,6 +23,7 @@ class _Backend(enum.Enum):
23 TORCH_SDPA = enum.auto()
24 OPENVINO = enum.auto()
25 FLASHINFER = enum.auto()
26 + HPU_ATTN = enum.auto()
27 PALLAS = enum.auto()
28 IPEX = enum.auto()
29 NO_ATTENTION = enum.auto()
@@ -145,6 +146,10 @@ def get_attn_backend(
146 logger.info("Using Flashinfer backend.")
147 from vllm.attention.backends.flashinfer import FlashInferBackend
148 return FlashInferBackend
149 + elif backend == _Backend.HPU_ATTN:
150 + logger.info("Using HPUAttention backend.")
151 + from vllm.attention.backends.hpu_attn import HPUAttentionBackend
152 + return HPUAttentionBackend
153 elif backend == _Backend.PALLAS:
154 logger.info("Using Pallas backend.")
155 from vllm.attention.backends.pallas import PallasAttentionBackend
@@ -220,6 +225,9 @@ def which_attn_to_use(
225 logger.info("%s is not supported in AMD GPUs.", selected_backend)
226 return _Backend.ROCM_FLASH
227
228 + if current_platform.is_hpu():
229 + return _Backend.HPU_ATTN
230 +
231 if envs.VLLM_USE_V1:
232 return _Backend.FLASH_ATTN_VLLM_V1
233
vllm/config.py
+19 -3
@@ -466,9 +466,10 @@ class ModelConfig:
466
467 # Reminder: Please update docs/source/serving/compatibility_matrix.rst
468 # If the feature combo become valid
469 - if device_config.device_type not in ("cuda", "tpu", "xpu"):
469 + if device_config.device_type not in ("cuda", "tpu", "xpu", "hpu"):
470 logger.warning(
471 - "Async output processing is only supported for CUDA, TPU, XPU. "
471 + "Async output processing is only supported for CUDA, TPU, XPU "
472 + "and HPU."
473 "Disabling it for other platforms.")
474 self.use_async_output_proc = False
475 return
@@ -860,7 +861,6 @@ class LoadConfig:
861 ignore_patterns: The list of patterns to ignore when loading the model.
862 Default to "original/**/*" to avoid repeated loading of llama's
863 checkpoints.
863 -
864 """
865
866 load_format: Union[str, LoadFormat, "BaseModelLoader"] = LoadFormat.AUTO
@@ -964,6 +964,13 @@ class ParallelConfig:
964 raise ValueError(
965 "TPU backend only supports Ray for distributed inference.")
966
967 + if current_platform.is_hpu() and self.world_size > 1:
968 + if self.distributed_executor_backend is None:
969 + self.distributed_executor_backend = "ray"
970 + if self.distributed_executor_backend != "ray":
971 + raise ValueError(
972 + "HPU backend only supports Ray for distributed inference.")
973 +
974 if self.distributed_executor_backend is None and self.world_size > 1:
975 # We use multiprocessing by default if world_size fits on the
976 # current node and we aren't in a ray placement group.
@@ -1166,6 +1173,8 @@ class DeviceConfig:
1173 self.device_type = "cuda"
1174 elif current_platform.is_neuron():
1175 self.device_type = "neuron"
1176 + elif current_platform.is_hpu():
1177 + self.device_type = "hpu"
1178 elif current_platform.is_openvino():
1179 self.device_type = "openvino"
1180 elif current_platform.is_tpu():
@@ -1745,6 +1754,13 @@ def _get_and_verify_dtype(
1754 torch_dtype = torch.float16
1755 else:
1756 torch_dtype = config_dtype
1757 +
1758 + if current_platform.is_hpu() and config_dtype == torch.float16:
1759 + logger.info(
1760 + "For HPU, we cast models to bfloat16 instead of"
1761 + "using float16 by default. Please specify `dtype` if you "
1762 + "want to use float16.")
1763 + torch_dtype = torch.bfloat16
1764 else:
1765 if dtype not in _STR_DTYPE_TO_TORCH_DTYPE:
1766 raise ValueError(f"Unknown dtype: {dtype}")
vllm/core/block/cpu_gpu_block_allocator.py
+6 -1
@@ -4,6 +4,7 @@ from vllm.core.block.interfaces import (Block, BlockAllocator, BlockId,
4 DeviceAwareBlockAllocator)
5 from vllm.core.block.naive_block import NaiveBlock, NaiveBlockAllocator
6 from vllm.core.block.prefix_caching_block import PrefixCachingBlockAllocator
7 +from vllm.platforms import current_platform
8 from vllm.utils import Device
9
10
@@ -52,7 +53,11 @@ class CpuGpuBlockAllocator(DeviceAwareBlockAllocator):
53 - The block IDs are assigned contiguously, with GPU block IDs coming
54 before CPU block IDs.
55 """
55 - block_ids = list(range(num_gpu_blocks + num_cpu_blocks))
56 + # For HPU, block id 0 is used only for padding
57 + reserved_blocks = 1 if current_platform.is_hpu() else 0
58 + block_ids = list(
59 + range(reserved_blocks, num_gpu_blocks + num_cpu_blocks))
60 + num_gpu_blocks -= reserved_blocks
61 gpu_block_ids = block_ids[:num_gpu_blocks]
62 cpu_block_ids = block_ids[num_gpu_blocks:]
63
vllm/distributed/device_communicators/hpu_communicator.py new
+48
@@ -0,0 +1,48 @@
1 +import torch
2 +import torch.distributed as dist
3 +from torch.distributed import ProcessGroup
4 +
5 +from vllm.platforms import current_platform
6 +
7 +if current_platform.is_hpu():
8 + import habana_frameworks.torch as htorch # noqa: F401
9 +
10 +
11 +class HpuCommunicator:
12 +
13 + def __init__(self, group: ProcessGroup):
14 + if not current_platform.is_hpu():
15 + self.disabled = True
16 + return
17 + self.disabled = False
18 + self.group = group
19 + self.world_size = dist.get_world_size(self.group)
20 +
21 + def all_reduce(self, x: torch.Tensor) -> torch.Tensor:
22 + # FIXME(kzawora): this is a workaround for a bug in Habana PT bridge
23 + # occurring when PT_HPU_ENABLE_LAZY_COLLECTIVES=true env var is used
24 + # (which is required for tensor parallel HPUGraph inference)
25 + htorch.core.mark_step()
26 + dist.all_reduce(x, group=self.group)
27 + return x
28 +
29 + def all_gather(self, x: torch.Tensor, dim: int = -1) -> torch.Tensor:
30 + world_size = self.world_size
31 + if dim < 0:
32 + # Convert negative dim to positive.
33 + dim += x.dim()
34 + input_size = x.size()
35 + # Allocate output tensor.
36 + output_tensor = torch.empty((world_size, ) + input_size,
37 + dtype=x.dtype,
38 + device=x.device)
39 + # All-gather.
40 + htorch.core.mark_step()
41 + dist.all_gather_into_tensor(output_tensor, x, group=self.group)
42 + # Reshape
43 + output_tensor = output_tensor.movedim(0, dim)
44 + output_tensor = output_tensor.reshape(input_size[:dim] +
45 + (world_size *
46 + input_size[dim], ) +
47 + input_size[dim + 1:])
48 + return output_tensor
vllm/distributed/parallel_state.py
+19
@@ -177,6 +177,7 @@ class GroupCoordinator:
177 use_pynccl: bool,
178 use_custom_allreduce: bool,
179 use_tpu_communicator: bool,
180 + use_hpu_communicator: bool,
181 use_message_queue_broadcaster: bool = False,
182 group_name: Optional[str] = None,
183 ):
@@ -213,6 +214,7 @@ class GroupCoordinator:
214 self.use_pynccl = use_pynccl
215 self.use_custom_allreduce = use_custom_allreduce
216 self.use_tpu_communicator = use_tpu_communicator
217 + self.use_hpu_communicator = use_hpu_communicator
218
219 # lazy import to avoid documentation build error
220 from vllm.distributed.device_communicators.custom_all_reduce import (
@@ -241,6 +243,12 @@ class GroupCoordinator:
243 if use_tpu_communicator and self.world_size > 1:
244 self.tpu_communicator = TpuCommunicator(group=self.cpu_group)
245
246 + from vllm.distributed.device_communicators.hpu_communicator import (
247 + HpuCommunicator)
248 + self.hpu_communicator: Optional[HpuCommunicator]
249 + if use_hpu_communicator and self.world_size > 1:
250 + self.hpu_communicator = HpuCommunicator(group=self.device_group)
251 +
252 from vllm.distributed.device_communicators.shm_broadcast import (
253 MessageQueue)
254 self.mq_broadcaster: Optional[MessageQueue] = None
@@ -362,6 +370,10 @@ class GroupCoordinator:
370 # TPU handles Dynamo with its own logic.
371 return self.tpu_communicator.all_reduce(input_)
372
373 + if self.hpu_communicator is not None and \
374 + not self.hpu_communicator.disabled:
375 + return self.hpu_communicator.all_reduce(input_)
376 +
377 if self.ca_comm is not None and \
378 not self.ca_comm.disabled and \
379 self.ca_comm.should_custom_ar(input_):
@@ -400,6 +412,11 @@ class GroupCoordinator:
412 if tpu_comm is not None and not tpu_comm.disabled:
413 return tpu_comm.all_gather(input_, dim)
414
415 + # For HPUs, use HPU communicator.
416 + hpu_comm = self.hpu_communicator
417 + if hpu_comm is not None and not hpu_comm.disabled:
418 + return hpu_comm.all_gather(input_, dim)
419 +
420 if dim < 0:
421 # Convert negative dim to positive.
422 dim += input_.dim()
@@ -879,6 +896,7 @@ def init_world_group(ranks: List[int], local_rank: int,
896 use_pynccl=False,
897 use_custom_allreduce=False,
898 use_tpu_communicator=False,
899 + use_hpu_communicator=False,
900 group_name="world",
901 )
902
@@ -900,6 +918,7 @@ def init_model_parallel_group(
918 use_pynccl=True,
919 use_custom_allreduce=use_custom_allreduce,
920 use_tpu_communicator=True,
921 + use_hpu_communicator=True,
922 use_message_queue_broadcaster=use_message_queue_broadcaster,
923 group_name=group_name,
924 )
vllm/engine/arg_utils.py
+7 -4
@@ -17,6 +17,7 @@ from vllm.config import (CacheConfig, ConfigFormat, DecodingConfig,
17 from vllm.executor.executor_base import ExecutorBase
18 from vllm.logger import init_logger
19 from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS
20 +from vllm.platforms import current_platform
21 from vllm.transformers_utils.config import (
22 maybe_register_config_serialize_by_value)
23 from vllm.transformers_utils.utils import check_gguf_file
@@ -37,6 +38,7 @@ DEVICE_OPTIONS = [
38 "openvino",
39 "tpu",
40 "xpu",
41 + "hpu",
42 ]
43
44
@@ -110,7 +112,9 @@ class EngineArgs:
112 pipeline_parallel_size: int = 1
113 tensor_parallel_size: int = 1
114 max_parallel_loading_workers: Optional[int] = None
113 - block_size: int = 16
115 + # NOTE(kzawora): default block size for Gaudi should be 128
116 + # smaller sizes still work, but very inefficiently
117 + block_size: int = 16 if not current_platform.is_hpu() else 128
118 enable_prefix_caching: bool = False
119 disable_sliding_window: bool = False
120 use_v2_block_manager: bool = True
@@ -397,7 +401,7 @@ class EngineArgs:
401 parser.add_argument('--block-size',
402 type=int,
403 default=EngineArgs.block_size,
400 - choices=[8, 16, 32],
404 + choices=[8, 16, 32, 64, 128],
405 help='Token block size for contiguous chunks of '
406 'tokens. This is ignored on neuron devices and '
407 'set to max-model-len')
@@ -1132,8 +1136,7 @@ class EngineArgs:
1136 multi_step_stream_outputs=self.multi_step_stream_outputs,
1137 send_delta_data=(envs.VLLM_USE_RAY_SPMD_WORKER
1138 and parallel_config.use_ray),
1135 - policy=self.scheduling_policy,
1136 - )
1139 + policy=self.scheduling_policy)
1140 lora_config = LoRAConfig(
1141 max_lora_rank=self.max_lora_rank,
1142 max_loras=self.max_loras,
vllm/engine/async_llm_engine.py
+8
@@ -627,6 +627,14 @@ class AsyncLLMEngine(EngineClient):
627 elif engine_config.device_config.device_type == "cpu":
628 from vllm.executor.cpu_executor import CPUExecutorAsync
629 executor_class = CPUExecutorAsync
630 + elif engine_config.device_config.device_type == "hpu":
631 + if distributed_executor_backend == "ray":
632 + initialize_ray_cluster(engine_config.parallel_config)
633 + from vllm.executor.ray_hpu_executor import RayHPUExecutorAsync
634 + executor_class = RayHPUExecutorAsync
635 + else:
636 + from vllm.executor.hpu_executor import HPUExecutorAsync
637 + executor_class = HPUExecutorAsync
638 elif engine_config.device_config.device_type == "openvino":
639 assert distributed_executor_backend is None, (
640 "Distributed execution is not supported with "
vllm/engine/llm_engine.py
+8
@@ -528,6 +528,14 @@ class LLMEngine:
528 elif engine_config.device_config.device_type == "cpu":
529 from vllm.executor.cpu_executor import CPUExecutor
530 executor_class = CPUExecutor
531 + elif engine_config.device_config.device_type == "hpu":
532 + if distributed_executor_backend == "ray":
533 + initialize_ray_cluster(engine_config.parallel_config)
534 + from vllm.executor.ray_hpu_executor import RayHPUExecutor
535 + executor_class = RayHPUExecutor
536 + else:
537 + from vllm.executor.hpu_executor import HPUExecutor
538 + executor_class = HPUExecutor
539 elif engine_config.device_config.device_type == "openvino":
540 from vllm.executor.openvino_executor import OpenVINOExecutor
541 executor_class = OpenVINOExecutor
vllm/executor/hpu_executor.py new
+205
@@ -0,0 +1,205 @@
1 +###############################################################################
2 +# Copyright (C) 2024 Habana Labs, Ltd. an Intel Company
3 +###############################################################################
4 +
5 +import contextlib
6 +import os
7 +from typing import Any, Dict, List, Optional, Set, Tuple
8 +
9 +from vllm.executor.executor_base import ExecutorAsyncBase, ExecutorBase
10 +from vllm.logger import init_logger
11 +from vllm.lora.request import LoRARequest
12 +from vllm.model_executor.layers.sampler import SamplerOutput
13 +from vllm.prompt_adapter.request import PromptAdapterRequest
14 +from vllm.sequence import ExecuteModelRequest
15 +from vllm.utils import (get_distributed_init_method, get_ip, get_open_port,
16 + make_async)
17 +from vllm.worker.worker_base import WorkerWrapperBase
18 +
19 +logger = init_logger(__name__)
20 +
21 +
22 +class HPUExecutor(ExecutorBase):
23 +
24 + uses_ray: bool = False
25 +
26 + def _init_executor(self) -> None:
27 + """Initialize the worker and load the model."""
28 + self._init_worker()
29 +
30 + def _get_worker_kwargs(
31 + self,
32 + local_rank: int = 0,
33 + rank: int = 0,
34 + distributed_init_method: Optional[str] = None) -> Dict[str, Any]:
35 + """Return worker init args for a given rank."""
36 + if distributed_init_method is None:
37 + distributed_init_method = get_distributed_init_method(
38 + get_ip(), get_open_port())
39 + return dict(
40 + vllm_config=self.vllm_config,
41 + local_rank=local_rank,
42 + rank=rank,
43 + distributed_init_method=distributed_init_method,
44 + is_driver_worker=rank == 0,
45 + )
46 +
47 + def _create_worker(self,
48 + local_rank: int = 0,
49 + rank: int = 0,
50 + distributed_init_method: Optional[str] = None):
51 + wrapper = WorkerWrapperBase(
52 + worker_module_name="vllm.worker.hpu_worker",
53 + worker_class_name="HPUWorker",
54 + )
55 + wrapper.init_worker(**self._get_worker_kwargs(local_rank, rank,
56 + distributed_init_method))
57 + return wrapper.worker
58 +
59 + def _init_worker(self):
60 + assert self.parallel_config.world_size == 1, (
61 + "GPUExecutor only supports single GPU.")
62 +
63 + self.driver_worker = self._create_worker()
64 + self.driver_worker.init_device()
65 + self.driver_worker.load_model()
66 +
67 + def determine_num_available_blocks(self) -> Tuple[int, int]:
68 + """Determine the number of available KV blocks by invoking the
69 + underlying worker.
70 + """
71 + return self.driver_worker.determine_num_available_blocks()
72 +
73 + def initialize_cache(self, num_gpu_blocks: int, num_cpu_blocks) -> None:
74 + """Initialize the KV cache by invoking the underlying worker.
75 + """
76 + # NOTE: This is logged in the executor because there can be >1 worker
77 + # with other executors. We could log in the engine level, but work
78 + # remains to abstract away the device for non-GPU configurations.
79 + logger.info("# HPU blocks: %d, # CPU blocks: %d", num_gpu_blocks,
80 + num_cpu_blocks)
81 + from vllm_hpu_extension.profiler import HabanaMemoryProfiler
82 + with HabanaMemoryProfiler() as cache_init_m:
83 + self.driver_worker.initialize_cache(num_gpu_blocks, num_cpu_blocks)
84 + msg = f"init_cache_engine took {cache_init_m.get_summary_string()}"
85 + logger.info(msg)
86 +
87 + def finish_measurements(self):
88 + self.driver_worker.finish_measurements()
89 +
90 + def execute_model(
91 + self,
92 + execute_model_req: ExecuteModelRequest) -> List[SamplerOutput]:
93 + # VLLM_HPU_LOG_STEP_GRAPH_COMPILATION - will log graph compilations per engine step, only when there was any - highly recommended to use alongside PT_HPU_METRICS_GC_DETAILS! # noqa:E501
94 + # VLLM_HPU_LOG_STEP_GRAPH_COMPILATION_ALL - will log graph compilations per engine step, always, even if there were none # noqa:E501
95 + # VLLM_HPU_LOG_STEP_CPU_FALLBACKS - will log cpu fallbacks per engine step, only when there was any # noqa:E501
96 + # VLLM_HPU_LOG_STEP_CPU_FALLBACKS_ALL - will log cpu fallbacks per engine step, always, even if there were none # noqa:E501
97 + log_graph_compilation_all = os.environ.get(
98 + 'VLLM_HPU_LOG_STEP_GRAPH_COMPILATION_ALL', '0') != '0'
99 + log_graph_compilation = os.environ.get(
100 + 'VLLM_HPU_LOG_STEP_GRAPH_COMPILATION',
101 + '0') != '0' or log_graph_compilation_all
102 + log_cpu_fallbacks_all = os.environ.get(
103 + 'VLLM_HPU_LOG_STEP_CPU_FALLBACKS_ALL', '0') != '0'
104 + log_cpu_fallbacks = os.environ.get('VLLM_HPU_LOG_STEP_CPU_FALLBACKS',
105 + '0') != '0' or log_cpu_fallbacks_all
106 + if log_graph_compilation or log_cpu_fallbacks:
107 + from habana_frameworks.torch.hpu.metrics import metric_localcontext
108 + seq_group_metadata_list = execute_model_req.seq_group_metadata_list
109 + is_prompt = any([
110 + seq_group_metadata.is_prompt
111 + for seq_group_metadata in seq_group_metadata_list
112 + ])
113 + max_context_len = max([
114 + max([
115 + len(v.prompt_token_ids) + len(v.output_token_ids)
116 + for v in seq_group_metadata.seq_data.values()
117 + ]) for seq_group_metadata in seq_group_metadata_list
118 + ]) # whoa, that's some spicy stuff right here
119 + max_num_blocks = (
120 + (max_context_len - 1) // self.cache_config.block_size) + 1
121 + input_stats = (f'is_prompt: {is_prompt}, '
122 + f'num_seqs: {len(seq_group_metadata_list)}, '
123 + f'max_context_len: {max_context_len}, '
124 + f'max_num_blocks {max_num_blocks}')
125 + gc_ctx = metric_localcontext(
126 + "graph_compilation"
127 + ) if log_graph_compilation else contextlib.nullcontext()
128 + cpu_fallback_ctx = metric_localcontext(
129 + "cpu_fallback"
130 + ) if log_cpu_fallbacks else contextlib.nullcontext()
131 + with gc_ctx as gc_local_metric, \
132 + cpu_fallback_ctx as cpu_fallback_local_metric:
133 + output = self.driver_worker.execute_model(execute_model_req)
134 + if (log_graph_compilation and gc_local_metric.stats()[0][1] > 0
135 + ) or log_graph_compilation_all:
136 + msg = ("VLLM_HPU_STEP_GRAPH_COMPILATION: "
137 + f"{gc_local_metric.stats()}, {input_stats}")
138 + logger.warning(msg)
139 + if (log_cpu_fallbacks and cpu_fallback_local_metric.stats()[0][1] >
140 + 0) or log_cpu_fallbacks_all:
141 + msg = ("VLLM_HPU_STEP_CPU_FALLBACK: "
142 + f"{cpu_fallback_local_metric.stats()}, {input_stats}")
143 + logger.warning(msg)
144 +
145 + return output
146 +
147 + output = self.driver_worker.execute_model(execute_model_req)
148 + return output
149 +
150 + def add_lora(self, lora_request: LoRARequest) -> bool:
151 + assert lora_request.lora_int_id > 0, "lora_id must be greater than 0."
152 + return self.driver_worker.add_lora(lora_request)
153 +
154 + def remove_lora(self, lora_id: int) -> bool:
155 + assert lora_id > 0, "lora_id must be greater than 0."
156 + return self.driver_worker.remove_lora(lora_id)
157 +
158 + def pin_lora(self, lora_id: int) -> bool:
159 + assert lora_id > 0, "lora_id must be greater than 0."
160 + return self.driver_worker.pin_lora(lora_id)
161 +
162 + def list_loras(self) -> Set[int]:
163 + return self.driver_worker.list_loras()
164 +
165 + def add_prompt_adapter(
166 + self, prompt_adapter_request: PromptAdapterRequest) -> bool:
167 + raise NotImplementedError(
168 + "Prompt Adapter is not implemented for HPU backend.")
169 +
170 + def remove_prompt_adapter(self, prompt_adapter_id: int) -> bool:
171 + raise NotImplementedError(
172 + "Prompt Adapter is not implemented for HPU backend.")
173 +
174 + def pin_prompt_adapter(self, prompt_adapter_id: int) -> bool:
175 + raise NotImplementedError(
176 + "Prompt Adapter is not implemented for HPU backend.")
177 +
178 + def list_prompt_adapters(self) -> Set[int]:
179 + raise NotImplementedError(
180 + "Prompt Adapter is not implemented for HPU backend.")
181 +
182 + def check_health(self) -> None:
183 + # GPUExecutor will always be healthy as long as
184 + # it's running.
185 + return
186 +
187 + def start_profile(self) -> None:
188 + self.driver_worker.start_profile()
189 +
190 + def stop_profile(self) -> None:
191 + self.driver_worker.stop_profile()
192 +
193 + def shutdown(self) -> None:
194 + self.driver_worker.shutdown_inc()
195 +
196 +
197 +class HPUExecutorAsync(HPUExecutor, ExecutorAsyncBase):
198 +
199 + async def execute_model_async(
200 + self,
201 + execute_model_req: ExecuteModelRequest,
202 + ) -> List[SamplerOutput]:
203 + output = await make_async(self.driver_worker.execute_model
204 + )(execute_model_req=execute_model_req, )
205 + return output
vllm/executor/ray_hpu_executor.py new
+554
@@ -0,0 +1,554 @@
1 +import asyncio
2 +import os
3 +from collections import defaultdict
4 +from itertools import islice, repeat
5 +from typing import (TYPE_CHECKING, Any, Callable, Dict, List, Optional, Tuple,
6 + Type)
7 +
8 +import msgspec
9 +
10 +import vllm.envs as envs
11 +from vllm.executor.distributed_gpu_executor import ( # yapf: disable
12 + DistributedGPUExecutor, DistributedGPUExecutorAsync)
13 +from vllm.executor.msgspec_utils import encode_hook
14 +from vllm.executor.ray_utils import RayWorkerWrapper, ray
15 +from vllm.logger import init_logger
16 +from vllm.model_executor.layers.sampler import SamplerOutput
17 +from vllm.sequence import ExecuteModelRequest
18 +from vllm.utils import (_run_task_with_lock, get_distributed_init_method,
19 + get_ip, get_open_port, get_vllm_instance_id,
20 + make_async)
21 +from vllm.worker.worker_base import WorkerBase
22 +
23 +if ray is not None:
24 + from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy
25 +
26 +if TYPE_CHECKING:
27 + from ray.util.placement_group import PlacementGroup
28 +
29 +logger = init_logger(__name__)
30 +
31 +
32 +class RayHPUExecutor(DistributedGPUExecutor):
33 +
34 + uses_ray: bool = True
35 +
36 + def _init_executor(self) -> None:
37 + self.forward_dag: Optional["ray.dag.CompiledDAG"] = None
38 + # If the env var is set, it uses the Ray's compiled DAG API
39 + # which optimizes the control plane overhead.
40 + # Run vLLM with VLLM_USE_RAY_COMPILED_DAG=1 to enable it.
41 + # Currently, this requires USE_RAY_SPMD_WORKER=True.
42 + self.use_ray_compiled_dag = envs.VLLM_USE_RAY_COMPILED_DAG
43 + # If the env var is set, then we do not distinguish between the
44 + # "driver worker" vs other workers. Also, the rank 0 worker will
45 + # be executed in a remote Ray worker. Currently this requires
46 + # USE_RAY_COMPILED_DAG=True.
47 + self.use_ray_spmd_worker = envs.VLLM_USE_RAY_SPMD_WORKER
48 + if self.use_ray_compiled_dag:
49 + assert self.use_ray_spmd_worker, (
50 + "VLLM_USE_RAY_COMPILED_DAG=1 requires "
51 + "VLLM_USE_RAY_SPMD_WORKER=1")
52 + if self.use_ray_spmd_worker:
53 + # TODO: Support SPMD worker for non-DAG Ray executor.
54 + assert self.use_ray_compiled_dag, (
55 + "VLLM_USE_RAY_SPMD_WORKER=1 requires "
56 + "VLLM_USE_RAY_COMPILED_DAG=1")
57 +
58 + assert self.uses_ray
59 + placement_group = self.parallel_config.placement_group
60 +
61 + # Disable Ray usage stats collection.
62 + ray_usage = os.environ.get("RAY_USAGE_STATS_ENABLED", "0")
63 + if ray_usage != "1":
64 + os.environ["RAY_USAGE_STATS_ENABLED"] = "0"
65 +
66 + # Create the parallel GPU workers.
67 + self._init_workers_ray(placement_group)
68 +
69 + self.input_encoder = msgspec.msgpack.Encoder(enc_hook=encode_hook)
70 + self.output_decoder = msgspec.msgpack.Decoder(
71 + Optional[List[SamplerOutput]])
72 +
73 + def shutdown(self) -> None:
74 + if hasattr(self, "forward_dag") and self.forward_dag is not None:
75 + self.forward_dag.teardown()
76 + import ray
77 + for worker in self.workers:
78 + ray.kill(worker)
79 + self.forward_dag = None
80 +
81 + def finish_measurements(self):
82 + self._run_workers("finish_measurements")
83 +
84 + def _get_worker_module_and_class(
85 + self
86 + ) -> Tuple[str, str, Optional[Callable[[],
87 + Type[WorkerBase]]]]: # noqa: F821
88 + worker_class_fn = None
89 + if self.scheduler_config.is_multi_step:
90 + raise NotImplementedError(
91 + "Multi-step execution is not implemented for HPU")
92 + elif self.speculative_config:
93 + raise NotImplementedError(
94 + "Speculative decoding is not implemented for HPU")
95 + else:
96 + worker_module_name = "vllm.worker.hpu_worker"
97 + worker_class_name = "HPUWorker"
98 + return (worker_module_name, worker_class_name, worker_class_fn)
99 +
100 + def _get_worker_wrapper_args(self) -> Dict[str, Any]:
101 + (worker_module_name, worker_class_name,
102 + worker_class_fn) = self._get_worker_module_and_class()
103 +
104 + return dict(
105 + worker_module_name=worker_module_name,
106 + worker_class_name=worker_class_name,
107 + worker_class_fn=worker_class_fn,
108 + trust_remote_code=self.model_config.trust_remote_code,
109 + )
110 +
111 + def _init_workers_ray(self, placement_group: "PlacementGroup",
112 + **ray_remote_kwargs):
113 + # Otherwise, the ray workers are allocated with a full GPU.
114 + num_gpus = 1
115 +
116 + # The driver dummy worker does not actually use any resources.
117 + # It holds the resource for the driver worker.
118 + self.driver_dummy_worker: Optional[RayWorkerWrapper] = None
119 + # The remaining workers are the actual ray actors.
120 + self.workers: List[RayWorkerWrapper] = []
121 +
122 + # Used in ray compiled DAG: indexed first by PP rank,
123 + # and then TP rank. In other words, the inner list is
124 + # the TP group of workers for a PP rank.
125 + self.pp_tp_workers: List[List[RayWorkerWrapper]] = []
126 +
127 + logger.info("use_ray_spmd_worker: %s", self.use_ray_spmd_worker)
128 +
129 + # Create the workers.
130 + driver_ip = get_ip()
131 + worker_wrapper_kwargs = self._get_worker_wrapper_args()
132 + for bundle_id, bundle in enumerate(placement_group.bundle_specs):
133 + if not bundle.get("HPU", 0):
134 + continue
135 + scheduling_strategy = PlacementGroupSchedulingStrategy(
136 + placement_group=placement_group,
137 + placement_group_capture_child_tasks=True,
138 + placement_group_bundle_index=bundle_id,
139 + )
140 +
141 + worker = ray.remote(
142 + num_cpus=0,
143 + num_gpus=0,
144 + resources={'HPU': num_gpus},
145 + scheduling_strategy=scheduling_strategy,
146 + **ray_remote_kwargs,
147 + )(RayWorkerWrapper).remote(**worker_wrapper_kwargs)
148 +
149 + if self.use_ray_spmd_worker:
150 + self.workers.append(worker)
151 + else:
152 + worker_ip = ray.get(worker.get_node_ip.remote())
153 + if worker_ip == driver_ip and self.driver_dummy_worker is None:
154 + # If the worker is on the same node as the driver, we use it
155 + # as the resource holder for the driver process.
156 + self.driver_dummy_worker = worker
157 + self.driver_worker = RayWorkerWrapper(
158 + **worker_wrapper_kwargs)
159 + else:
160 + # Else, added to the list of workers.
161 + self.workers.append(worker)
162 +
163 + logger.debug("workers: %s", self.workers)
164 + logger.debug("driver_dummy_worker: %s", self.driver_dummy_worker)
165 + if not self.use_ray_spmd_worker and self.driver_dummy_worker is None:
166 + raise ValueError(
167 + "Ray does not allocate any GPUs on the driver node. Consider "
168 + "adjusting the Ray placement group or running the driver on a "
169 + "GPU node.")
170 +
171 + worker_ips = [
172 + ray.get(worker.get_node_ip.remote()) # type: ignore[attr-defined]
173 + for worker in self.workers
174 + ]
175 + ip_counts: Dict[str, int] = {}
176 + for ip in worker_ips:
177 + ip_counts[ip] = ip_counts.get(ip, 0) + 1
178 +
179 + def sort_by_driver_then_worker_ip(worker):
180 + """
181 + Sort the workers based on 3 properties:
182 + 1. If the worker is on the same node as the driver (vllm engine),
183 + it should be placed first.
184 + 2. Then, if the worker is on a node with fewer workers, it should
185 + be placed first.
186 + 3. Finally, if the work is on a node with smaller IP address, it
187 + should be placed first.
188 + """
189 + ip = ray.get(worker.get_node_ip.remote())
190 + return (ip != driver_ip, ip_counts[ip], ip)
191 +
192 + # After sorting, the workers on the same node will be
193 + # close to each other, and the workers on the driver
194 + # node will be placed first.
195 + self.workers = sorted(self.workers, key=sort_by_driver_then_worker_ip)
196 +
197 + # Get the set of GPU IDs used on each node.
198 + worker_node_and_gpu_ids = self._run_workers("get_node_and_gpu_ids",
199 + use_dummy_driver=True)
200 +
201 + node_workers = defaultdict(list) # node id -> list of worker ranks
202 + node_gpus = defaultdict(list) # node id -> list of gpu ids
203 +
204 + for i, (node_id, gpu_ids) in enumerate(worker_node_and_gpu_ids):
205 + node_workers[node_id].append(i)
206 + # `gpu_ids` can be a list of strings or integers.
207 + # convert them to integers for consistency.
208 + # NOTE: gpu_ids can be larger than 9 (e.g. 16 GPUs),
209 + # string sorting is not sufficient.
210 + # see https://github.com/vllm-project/vllm/issues/5590
211 + gpu_ids = [int(x) for x in gpu_ids]
212 + node_gpus[node_id].extend(gpu_ids)
213 + for node_id, gpu_ids in node_gpus.items():
214 + node_gpus[node_id] = sorted(gpu_ids)
215 +
216 + all_ips = set(worker_ips + [driver_ip])
217 + n_ips = len(all_ips)
218 + n_nodes = len(node_workers)
219 +
220 + if n_nodes != n_ips:
221 + raise RuntimeError(
222 + f"Every node should have a unique IP address. Got {n_nodes}"
223 + f" nodes with node ids {list(node_workers.keys())} and "
224 + f"{n_ips} unique IP addresses {all_ips}. Please check your"
225 + " network configuration. If you set `VLLM_HOST_IP` or "
226 + "`HOST_IP` environment variable, make sure it is unique for"
227 + " each node.")
228 +
229 + VLLM_INSTANCE_ID = get_vllm_instance_id()
230 +
231 + # Set environment variables for the driver and workers.
232 + all_args_to_update_environment_variables = [({
233 + "VLLM_INSTANCE_ID":
234 + VLLM_INSTANCE_ID,
235 + "VLLM_TRACE_FUNCTION":
236 + str(envs.VLLM_TRACE_FUNCTION),
237 + }, ) for (node_id, _) in worker_node_and_gpu_ids]
238 + self._run_workers("update_environment_variables",
239 + all_args=all_args_to_update_environment_variables)
240 +
241 + if len(node_gpus) == 1:
242 + # in single node case, we don't need to get the IP address.
243 + # the loopback address is sufficient
244 + # NOTE: a node may have several IP addresses, one for each
245 + # network interface. `get_ip()` might return any of them,
246 + # while they might not work for communication inside the node
247 + # if the network setup is complicated. Using the loopback address
248 + # solves this issue, as it always works for communication inside
249 + # the node.
250 + driver_ip = "127.0.0.1"
251 + distributed_init_method = get_distributed_init_method(
252 + driver_ip, get_open_port())
253 +
254 + # Initialize the actual workers inside worker wrapper.
255 + init_worker_all_kwargs = [
256 + self._get_worker_kwargs(
257 + local_rank=node_workers[node_id].index(rank),
258 + rank=rank,
259 + distributed_init_method=distributed_init_method,
260 + ) for rank, (node_id, _) in enumerate(worker_node_and_gpu_ids)
261 + ]
262 + self._run_workers("init_worker", all_kwargs=init_worker_all_kwargs)
263 +
264 + self._run_workers("init_device")
265 + self._run_workers("load_model",
266 + max_concurrent_workers=self.parallel_config.
267 + max_parallel_loading_workers)
268 +
269 + if self.use_ray_spmd_worker:
270 + for pp_rank in range(self.parallel_config.pipeline_parallel_size):
271 + self.pp_tp_workers.append([])
272 + for tp_rank in range(
273 + self.parallel_config.tensor_parallel_size):
274 + # PP=2, TP=4
275 + # pp_tp_workers = [[0, 1, 2, 3], [4, 5, 6, 7]]
276 + rank = (pp_rank * self.parallel_config.tensor_parallel_size
277 + ) + tp_rank
278 + assert len(self.pp_tp_workers[pp_rank]) == tp_rank
279 + assert pp_rank < len(self.pp_tp_workers)
280 + self.pp_tp_workers[pp_rank].append(self.workers[rank])
281 +
282 + # This is the list of workers that are rank 0 of each TP group EXCEPT
283 + # global rank 0. These are the workers that will broadcast to the
284 + # rest of the workers.
285 + self.tp_driver_workers: List[RayWorkerWrapper] = []
286 + # This is the list of workers that are not drivers and not the first
287 + # worker in a TP group. These are the workers that will be
288 + # broadcasted to.
289 + self.non_driver_workers: List[RayWorkerWrapper] = []
290 +
291 + # Enforce rank order for correct rank to return final output.
292 + for index, worker in enumerate(self.workers):
293 + # The driver worker is rank 0 and not in self.workers.
294 + rank = index + 1
295 + if rank % self.parallel_config.tensor_parallel_size == 0:
296 + self.tp_driver_workers.append(worker)
297 + else:
298 + self.non_driver_workers.append(worker)
299 +
300 + def _driver_execute_model(
301 + self, execute_model_req: Optional[ExecuteModelRequest]
302 + ) -> Optional[List[SamplerOutput]]:
303 + """Run execute_model in the driver worker.
304 +
305 + Passing None will cause the driver to stop the model execution
306 + loop running in each of the remote workers.
307 + """
308 + assert not self.use_ray_spmd_worker, (
309 + "driver_worker does not exist for VLLM_USE_RAY_SPMD_WORKER=1")
310 + return self.driver_worker.execute_method("execute_model",
311 + execute_model_req)
312 +
313 + def execute_model(
314 + self,
315 + execute_model_req: ExecuteModelRequest) -> List[SamplerOutput]:
316 + if not self.use_ray_spmd_worker:
317 + return super().execute_model(execute_model_req)
318 +
319 + if self.forward_dag is None:
320 + self.forward_dag = self._compiled_ray_dag(enable_asyncio=False)
321 +
322 + serialized_data = self.input_encoder.encode(execute_model_req)
323 + outputs = ray.get(self.forward_dag.execute(serialized_data))
324 + output = self.output_decoder.decode(outputs[0])
325 + return output
326 +
327 + def _run_workers(
328 + self,
329 + method: str,
330 + *args,
331 + async_run_tensor_parallel_workers_only: bool = False,
332 + all_args: Optional[List[Tuple[Any, ...]]] = None,
333 + all_kwargs: Optional[List[Dict[str, Any]]] = None,
334 + use_dummy_driver: bool = False,
335 + max_concurrent_workers: Optional[int] = None,
336 + **kwargs,
337 + ) -> Any:
338 + """Runs the given method on all workers. Can be used in the following
339 + ways:
340 +
341 + Args:
342 + - async_run_tensor_parallel_workers_only: If True the method will be
343 + run only in the remote TP workers, not the driver worker.
344 + It will also be run asynchronously and return a list of futures
345 + rather than blocking on the results.
346 + - args/kwargs: All workers share the same args/kwargs
347 + - all_args/all_kwargs: args/kwargs for each worker are specified
348 + individually
349 + """
350 + if self.use_ray_spmd_worker:
351 + assert not async_run_tensor_parallel_workers_only, (
352 + "async_run_tensor_parallel_workers_only is not supported for "
353 + "spmd mode.")
354 +
355 + if max_concurrent_workers:
356 + raise NotImplementedError(
357 + "max_concurrent_workers is not supported yet.")
358 +
359 + count = len(self.workers) if not \
360 + async_run_tensor_parallel_workers_only \
361 + else len(self.non_driver_workers)
362 + # If using SPMD worker, all workers are the same, so we should execute
363 + # the args on all workers. Otherwise, we skip the first worker's args
364 + # because those args will go to the driver worker.
365 + first_worker_args_index: int = 0 if self.use_ray_spmd_worker else 1
366 + all_worker_args = repeat(args, count) if all_args is None \
367 + else islice(all_args, first_worker_args_index, None)
368 + all_worker_kwargs = repeat(kwargs, count) if all_kwargs is None \
369 + else islice(all_kwargs, first_worker_args_index, None)
370 +
371 + # Start the ray workers first.
372 + ray_workers = self.workers
373 + if async_run_tensor_parallel_workers_only:
374 + ray_workers = self.non_driver_workers
375 + ray_worker_outputs = [
376 + worker.execute_method.remote(method, *worker_args, **worker_kwargs)
377 + for (worker, worker_args, worker_kwargs
378 + ) in zip(ray_workers, all_worker_args, all_worker_kwargs)
379 + ]
380 +
381 + if async_run_tensor_parallel_workers_only:
382 + # Just return futures
383 + return ray_worker_outputs
384 +
385 + driver_worker_output = []
386 + # In SPMD mode, the driver worker is the same as any other worker,
387 + # so we only explicitly execute on the driver worker if using a
388 + # non-SPMD worker class.
389 + if not self.use_ray_spmd_worker:
390 + driver_args = args if all_args is None else all_args[0]
391 + driver_kwargs = kwargs if all_kwargs is None else all_kwargs[0]
392 +
393 + # Start the driver worker after all the ray workers.
394 + if not use_dummy_driver:
395 + driver_worker_output = [
396 + self.driver_worker.execute_method(method, *driver_args,
397 + **driver_kwargs)
398 + ]
399 + else:
400 + assert self.driver_dummy_worker is not None
401 + driver_worker_output = [
402 + ray.get(
403 + self.driver_dummy_worker.execute_method.remote(
404 + method, *driver_args, **driver_kwargs))
405 + ]
406 +
407 + # Get the results of the ray workers.
408 + if self.workers:
409 + ray_worker_outputs = ray.get(ray_worker_outputs)
410 +
411 + return driver_worker_output + ray_worker_outputs
412 +
413 + def _wait_for_tasks_completion(self, parallel_worker_tasks: Any) -> None:
414 + """Wait for futures returned from _run_workers() with
415 + async_run_remote_workers_only to complete."""
416 + ray.get(parallel_worker_tasks)
417 +
418 + def _check_ray_adag_installation(self):
419 + import pkg_resources
420 + from packaging import version
421 +
422 + required_version = version.parse("2.35")
423 + current_version = version.parse(
424 + pkg_resources.get_distribution("ray").version)
425 + # TODO: update the constraint once we adapt to the backward
426 + # incompatible API change from ray 2.36
427 + if current_version != required_version:
428 + raise ValueError(f"Ray version {required_version} is "
429 + f"required, but found {current_version}")
430 +
431 + import importlib.util
432 + adag_spec = importlib.util.find_spec(
433 + "ray.experimental.compiled_dag_ref")
434 + if adag_spec is None:
435 + raise ValueError("Ray accelerated DAG is not installed. "
436 + "Run `pip install ray[adag]` to install it.")
437 +
438 + def _compiled_ray_dag(self, enable_asyncio: bool):
439 + assert self.parallel_config.use_ray
440 + self._check_ray_adag_installation()
441 + from ray.dag import InputNode, MultiOutputNode
442 + from ray.experimental.channel.torch_tensor_type import TorchTensorType
443 +
444 + with InputNode() as input_data:
445 + # Example DAG: PP=2, TP=4
446 + # (ExecuteModelReq, None) -> 0 -> (ExecuteModelReq, IntermediateOutput) -> 4 -> SamplerOutput # noqa: E501
447 + # -> 1 -> (ExecuteModelReq, IntermediateOutput) -> 5 -> SamplerOutput # noqa: E501
448 + # -> 2 -> (ExecuteModelReq, IntermediateOutput) -> 6 -> SamplerOutput # noqa: E501
449 + # -> 3 -> (ExecuteModelReq, IntermediateOutput) -> 7 -> SamplerOutput # noqa: E501
450 +
451 + # All workers in the first TP group will take in the
452 + # ExecuteModelRequest as input.
453 + outputs = [input_data for _ in self.pp_tp_workers[0]]
454 + for pp_rank, tp_group in enumerate(self.pp_tp_workers):
455 + # Each PP worker takes in the output of the previous PP worker,
456 + # and the TP group executes in SPMD fashion.
457 + outputs = [
458 + worker.execute_model_spmd.
459 + bind( # type: ignore[attr-defined]
460 + outputs[i]) for i, worker in enumerate(tp_group)
461 + ]
462 +
463 + last_pp_rank = len(self.pp_tp_workers) - 1
464 + if pp_rank < last_pp_rank:
465 + # Specify how intermediate tensors should be passed
466 + # between pp stages, no need to specify for the last
467 + # pp stage.
468 + transport = "auto"
469 + outputs = [
470 + output.with_type_hint(
471 + TorchTensorType(transport=transport))
472 + for output in outputs
473 + ]
474 +
475 + forward_dag = MultiOutputNode(outputs)
476 +
477 + return forward_dag.experimental_compile(enable_asyncio=enable_asyncio)
478 +
479 + def __del__(self):
480 + self.shutdown()
481 +
482 +
483 +class RayHPUExecutorAsync(RayHPUExecutor, DistributedGPUExecutorAsync):
484 +
485 + def __init__(self, *args, **kwargs):
486 + super().__init__(*args, **kwargs)
487 + self.pp_locks: Optional[List[asyncio.Lock]] = None
488 + self.use_ray_spmd_worker = envs.VLLM_USE_RAY_SPMD_WORKER
489 + if not self.use_ray_compiled_dag:
490 + self.driver_exec_method = make_async(
491 + self.driver_worker.execute_method)
492 +
493 + async def execute_model_async(
494 + self,
495 + execute_model_req: ExecuteModelRequest) -> List[SamplerOutput]:
496 + if not self.use_ray_spmd_worker:
497 + return await super().execute_model_async(execute_model_req)
498 +
499 + if self.forward_dag is None:
500 + self.forward_dag = self._compiled_ray_dag(enable_asyncio=True)
501 +
502 + serialized_data = self.input_encoder.encode(execute_model_req)
503 + dag_future = await self.forward_dag.execute_async(serialized_data)
504 + outputs = await dag_future
505 + return self.output_decoder.decode(outputs[0])
506 +
507 + async def _driver_execute_model_async(
508 + self,
509 + execute_model_req: Optional[ExecuteModelRequest] = None
510 + ) -> List[SamplerOutput]:
511 + assert not self.use_ray_spmd_worker, (
512 + "driver_worker does not exist for VLLM_USE_RAY_SPMD_WORKER=1")
513 + if not self.tp_driver_workers:
514 + return await self.driver_exec_method("execute_model",
515 + execute_model_req)
516 + if self.pp_locks is None:
517 + # This locks each pipeline parallel stage so multiple virtual
518 + # engines can't execute on the same stage at the same time
519 + # We create the locks here to avoid creating them in the constructor
520 + # which uses a different asyncio loop.
521 + self.pp_locks = [
522 + asyncio.Lock()
523 + for _ in range(self.parallel_config.pipeline_parallel_size)
524 + ]
525 +
526 + tasks = [
527 + asyncio.create_task(
528 + _run_task_with_lock(self.driver_exec_method, self.pp_locks[0],
529 + "execute_model", execute_model_req))
530 + ]
531 + for pp_rank, driver_worker in enumerate(self.tp_driver_workers,
532 + start=1):
533 + tasks.append(
534 + asyncio.create_task(
535 + _run_task_with_lock(driver_worker.execute_method.remote,
536 + self.pp_locks[pp_rank],
537 + "execute_model", execute_model_req)))
538 +
539 + results = await asyncio.gather(*tasks)
540 +
541 + # Only the last PP stage has the final results.
542 + return results[-1]
543 +
544 + async def _start_worker_execution_loop(self):
545 + assert not self.use_ray_spmd_worker, (
546 + "worker loop is disabled for VLLM_USE_RAY_SPMD_WORKER=1")
547 + coros = [
548 + worker.execute_method.remote("start_worker_execution_loop")
549 + for worker in self.non_driver_workers
550 + ]
551 + return await asyncio.gather(*coros)
552 +
553 + def __del__(self):
554 + self.shutdown()
vllm/executor/ray_utils.py
+5 -1
@@ -249,7 +249,11 @@ def initialize_ray_cluster(
249 # Placement group is already set.
250 return
251
252 - device_str = "GPU" if not current_platform.is_tpu() else "TPU"
252 + device_str = "GPU"
253 + if current_platform.is_tpu():
254 + device_str = "TPU"
255 + elif current_platform.is_hpu():
256 + device_str = 'HPU'
257 # Create placement group for worker processes
258 current_placement_group = ray.util.get_current_placement_group()
259 if current_placement_group:
vllm/model_executor/custom_op.py
+3 -2
@@ -55,10 +55,9 @@ class CustomOp(nn.Module):
55 # NOTE(woosuk): This is a placeholder for future extensions.
56 return self.forward_native(*args, **kwargs)
57
58 - def forward_gaudi(self, *args, **kwargs):
58 + def forward_hpu(self, *args, **kwargs):
59 # By default, we assume that Gaudi ops are compatible with the
60 # PyTorch-native implementation.
61 - # NOTE(woosuk): This is a placeholder for future extensions.
61 return self.forward_native(*args, **kwargs)
62
63 def dispatch_forward(self):
@@ -76,6 +75,8 @@ class CustomOp(nn.Module):
75 return self.forward_hip
76 elif current_platform.is_cpu():
77 return self.forward_cpu
78 + elif current_platform.is_hpu():
79 + return self.forward_hpu
80 elif current_platform.is_tpu():
81 return self.forward_tpu
82 elif current_platform.is_xpu():
vllm/model_executor/layers/layernorm.py
+19
@@ -92,6 +92,25 @@ class RMSNorm(CustomOp):
92 )
93 return out
94
95 + def forward_hpu(
96 + self,
97 + x: torch.Tensor,
98 + residual: Optional[torch.Tensor] = None,
99 + ) -> Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]:
100 + from vllm_hpu_extension.ops import HPUFusedRMSNorm
101 + if HPUFusedRMSNorm is None:
102 + return self.forward_native(x, residual)
103 + if residual is not None:
104 + orig_shape = x.shape
105 + residual += x.view(residual.shape)
106 + # Note: HPUFusedRMSNorm requires 3D tensors as inputs
107 + x = HPUFusedRMSNorm.apply(residual, self.weight,
108 + self.variance_epsilon)
109 + return x.view(orig_shape), residual
110 +
111 + x = HPUFusedRMSNorm.apply(x, self.weight, self.variance_epsilon)
112 + return x
113 +
114 def forward_xpu(
115 self,
116 x: torch.Tensor,
vllm/model_executor/layers/logits_processor.py
+8 -2
@@ -111,8 +111,14 @@ def _prune_hidden_states(
111 hidden_states: torch.Tensor,
112 sampling_metadata: SamplingMetadata,
113 ) -> torch.Tensor:
114 - return hidden_states.index_select(0,
115 - sampling_metadata.selected_token_indices)
114 + # NOTE(kzawora): The if guard is needed for Gaudi - in some scenarios
115 + # (warmup, profile_run) we might not have selected_token_indices,
116 + # so we skip pruning.
117 + if sampling_metadata.selected_token_indices is not None:
118 + return hidden_states.index_select(
119 + 0, sampling_metadata.selected_token_indices)
120 + else:
121 + return hidden_states
122
123
124 def _apply_logits_processors(
vllm/model_executor/layers/rotary_embedding.py
+55
@@ -194,6 +194,61 @@ class RotaryEmbedding(CustomOp):
194 self.cos_sin_cache, self.is_neox_style)
195 return query, key
196
197 + def forward_hpu(
198 + self,
199 + positions: torch.Tensor,
200 + query: torch.Tensor,
201 + key: torch.Tensor,
202 + offsets: Optional[torch.Tensor] = None,
203 + ) -> Tuple[torch.Tensor, torch.Tensor]:
204 + from habana_frameworks.torch.hpex.kernels import (
205 + RotaryPosEmbeddingMode, apply_rotary_pos_emb)
206 + positions = positions.flatten()
207 + if offsets is not None:
208 + positions = positions + offsets
209 + num_tokens = positions.shape[0]
210 + cos_sin = self.cos_sin_cache.index_select(0, positions).view(
211 + num_tokens, 1, -1)
212 + cos, sin = cos_sin.chunk(2, dim=-1)
213 + # HPU RoPE kernel requires hidden dimension for cos and sin to be equal
214 + # to query hidden dimension, so the original tensors need to be
215 + # expanded
216 + # GPT-NeoX kernel requires position_ids = None, offset, mode = BLOCKWISE
217 + # and expansion of cos/sin tensors via concatenation
218 + # GPT-J kernel requires position_ids = None, offset = 0, mode = PAIRWISE
219 + # and expansion of cos/sin tensors via repeat_interleave
220 + rope_mode: RotaryPosEmbeddingMode
221 + if self.is_neox_style:
222 + rope_mode = RotaryPosEmbeddingMode.BLOCKWISE
223 + cos = torch.cat((cos, cos), dim=-1)
224 + sin = torch.cat((sin, sin), dim=-1)
225 + else:
226 + rope_mode = RotaryPosEmbeddingMode.PAIRWISE
227 + sin = torch.repeat_interleave(sin,
228 + 2,
229 + dim=-1,
230 + output_size=cos_sin.shape[-1])
231 + cos = torch.repeat_interleave(cos,
232 + 2,
233 + dim=-1,
234 + output_size=cos_sin.shape[-1])
235 +
236 + query_shape = query.shape
237 + query = query.view(num_tokens, -1, self.head_size)
238 + query_rot = query[..., :self.rotary_dim]
239 + query_pass = query[..., self.rotary_dim:]
240 + query_rot = apply_rotary_pos_emb(query_rot, cos, sin, None, 0,
241 + rope_mode)
242 + query = torch.cat((query_rot, query_pass), dim=-1).reshape(query_shape)
243 +
244 + key_shape = key.shape
245 + key = key.view(num_tokens, -1, self.head_size)
246 + key_rot = key[..., :self.rotary_dim]
247 + key_pass = key[..., self.rotary_dim:]
248 + key_rot = apply_rotary_pos_emb(key_rot, cos, sin, None, 0, rope_mode)
249 + key = torch.cat((key_rot, key_pass), dim=-1).reshape(key_shape)
250 + return query, key
251 +
252 def extra_repr(self) -> str:
253 s = f"head_size={self.head_size}, rotary_dim={self.rotary_dim}"
254 s += f", max_position_embeddings={self.max_position_embeddings}"
vllm/model_executor/layers/vocab_parallel_embedding.py
+15 -2
@@ -12,6 +12,7 @@ from vllm.model_executor.layers.quantization.base_config import (
12 QuantizationConfig, QuantizeMethodBase, method_has_implemented_embedding)
13 from vllm.model_executor.parameter import BasevLLMParameter
14 from vllm.model_executor.utils import set_weight_attrs
15 +from vllm.platforms import current_platform
16
17 DEFAULT_VOCAB_PADDING_SIZE = 64
18
@@ -382,8 +383,20 @@ class VocabParallelEmbedding(torch.nn.Module):
383
384 # Copy the data.
385 loaded_weight = loaded_weight.narrow(output_dim, start_idx, shard_size)
385 - param[:loaded_weight.shape[0]].data.copy_(loaded_weight)
386 - param[loaded_weight.shape[0]:].data.fill_(0)
386 +
387 + if current_platform.is_hpu():
388 + # FIXME(kzawora): Weight copy with slicing bugs out on Gaudi here,
389 + # so we're using a workaround. Remove this when fixed in
390 + # HPU PT bridge.
391 + padded_weight = torch.cat([
392 + loaded_weight,
393 + torch.zeros(param.shape[0] - loaded_weight.shape[0],
394 + *loaded_weight.shape[1:])
395 + ])
396 + param.data.copy_(padded_weight)
397 + else:
398 + param[:loaded_weight.shape[0]].data.copy_(loaded_weight)
399 + param[loaded_weight.shape[0]:].data.fill_(0)
400
401 def forward(self, input_):
402 if self.tp_size > 1:
vllm/model_executor/sampling_metadata.py
+2 -1
@@ -284,7 +284,8 @@ def _prepare_seq_groups(
284 else:
285 # Decode
286 prompt_logprob_len = 0
287 - query_len = query_lens[i] if query_lens is not None else 1
287 + query_len = query_lens[i] if query_lens is not None and len(
288 + query_lens) > 0 else 1
289 sample_len = len(seq_ids) * query_len if do_sample else 0
290
291 if sampling_params.seed is not None and generators is not None:
vllm/platforms/__init__.py
+10
@@ -42,6 +42,13 @@ try:
42 except Exception:
43 pass
44
45 +is_hpu = False
46 +try:
47 + from importlib import util
48 + is_hpu = util.find_spec('habana_frameworks') is not None
49 +except Exception:
50 + pass
51 +
52 is_xpu = False
53
54 try:
@@ -86,6 +93,9 @@ elif is_cuda:
93 elif is_rocm:
94 from .rocm import RocmPlatform
95 current_platform = RocmPlatform()
96 +elif is_hpu:
97 + from .hpu import HpuPlatform
98 + current_platform = HpuPlatform()
99 elif is_xpu:
100 from .xpu import XPUPlatform
101 current_platform = XPUPlatform()
vllm/platforms/hpu.py new
+11
@@ -0,0 +1,11 @@
1 +import torch
2 +
3 +from .interface import Platform, PlatformEnum
4 +
5 +
6 +class HpuPlatform(Platform):
7 + _enum = PlatformEnum.HPU
8 +
9 + @staticmethod
10 + def inference_mode():
11 + return torch.no_grad()
vllm/platforms/interface.py
+4
@@ -10,6 +10,7 @@ class PlatformEnum(enum.Enum):
10 CUDA = enum.auto()
11 ROCM = enum.auto()
12 TPU = enum.auto()
13 + HPU = enum.auto()
14 XPU = enum.auto()
15 CPU = enum.auto()
16 NEURON = enum.auto()
@@ -46,6 +47,9 @@ class Platform:
47 def is_tpu(self) -> bool:
48 return self._enum == PlatformEnum.TPU
49
50 + def is_hpu(self) -> bool:
51 + return self._enum == PlatformEnum.HPU
52 +
53 def is_xpu(self) -> bool:
54 return self._enum == PlatformEnum.XPU
55
vllm/utils.py
+3
@@ -728,6 +728,9 @@ def is_pin_memory_available() -> bool:
728 elif current_platform.is_neuron():
729 print_warning_once("Pin memory is not supported on Neuron.")
730 return False
731 + elif current_platform.is_hpu():
732 + print_warning_once("Pin memory is not supported on HPU.")
733 + return False
734 elif current_platform.is_cpu() or current_platform.is_openvino():
735 return False
736 return True
vllm/worker/hpu_model_runner.py new
+2008
@@ -0,0 +1,2008 @@
1 +###############################################################################
2 +# Copyright (C) 2024 Habana Labs, Ltd. an Intel Company
3 +###############################################################################
4 +
5 +import collections
6 +import contextlib
7 +import dataclasses
8 +import functools
9 +import gc
10 +import itertools
11 +import math
12 +import operator
13 +import os
14 +import time
15 +from array import array
16 +from dataclasses import dataclass, field
17 +from enum import IntEnum
18 +from typing import (TYPE_CHECKING, Any, Callable, Dict, List, NamedTuple,
19 + Optional, Set, Tuple, Type, TypeVar, Union)
20 +
21 +import habana_frameworks.torch as htorch
22 +import habana_frameworks.torch.internal.bridge_config as bc
23 +import torch
24 +from vllm_hpu_extension.ops import LoraMask as LoraMask
25 +from vllm_hpu_extension.profiler import (HabanaHighLevelProfiler,
26 + HabanaMemoryProfiler, format_bytes)
27 +
28 +from vllm.attention import AttentionMetadata, get_attn_backend
29 +from vllm.config import DeviceConfig, VllmConfig
30 +from vllm.distributed.parallel_state import get_world_group
31 +from vllm.logger import init_logger
32 +from vllm.lora.layers import LoRAMapping
33 +from vllm.lora.request import LoRARequest
34 +from vllm.lora.worker_manager import LRUCacheWorkerLoRAManager
35 +from vllm.model_executor import SamplingMetadata
36 +from vllm.model_executor.layers.sampler import SamplerOutput
37 +from vllm.model_executor.model_loader import get_model
38 +from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensorInputs,
39 + MultiModalInputs)
40 +from vllm.sampling_params import SamplingParams
41 +from vllm.sequence import (IntermediateTensors, SequenceData,
42 + SequenceGroupMetadata)
43 +from vllm.utils import is_pin_memory_available, make_tensor_with_pad
44 +from vllm.worker.model_runner_base import (
45 + ModelRunnerBase, ModelRunnerInputBase,
46 + _add_attn_metadata_broadcastable_dict,
47 + _add_sampling_metadata_broadcastable_dict,
48 + _init_attn_metadata_from_tensor_dict,
49 + _init_sampling_metadata_from_tensor_dict)
50 +
51 +if TYPE_CHECKING:
52 + from vllm.attention.backends.abstract import AttentionBackend
53 +
54 +logger = init_logger(__name__)
55 +
56 +_TYPE_CACHE = {}
57 +# These values are assumed to be zero in several places.
58 +# Use caution when updating them!
59 +_PAD_SLOT_ID = 0
60 +_PAD_BLOCK_ID = 0
61 +
62 +LORA_WARMUP_RANK = 8
63 +
64 +
65 +class Singleton(type):
66 + _instances: Dict[type, object] = {}
67 +
68 + def __call__(cls, *args, **kwargs):
69 + if cls not in cls._instances:
70 + cls._instances[cls] = super(Singleton,
71 + cls).__call__(*args, **kwargs)
72 + return cls._instances[cls]
73 +
74 +
75 +@dataclass
76 +class HPUBucketingGlobalState(metaclass=Singleton):
77 + prompt_bs_bucket_cfg: Tuple[int, int, int] = field(init=False)
78 + decode_bs_bucket_cfg: Tuple[int, int, int] = field(init=False)
79 + prompt_seq_bucket_cfg: Tuple[int, int, int] = field(init=False)
80 + decode_block_bucket_cfg: Tuple[int, int, int] = field(init=False)
81 + prompt_buckets: List[Tuple[int, int]] = field(init=False)
82 + decode_buckets: List[Tuple[int, int]] = field(init=False)
83 +
84 +
85 +def subtuple(obj: object,
86 + typename: str,
87 + to_copy: List[str],
88 + to_override: Optional[Dict[str, object]] = None):
89 + if obj is None:
90 + return None
91 + if to_override is None:
92 + to_override = {}
93 + fields = set(to_copy) | set(to_override.keys())
94 + values = {f: to_override.get(f, getattr(obj, f)) for f in fields}
95 + if typename not in _TYPE_CACHE:
96 + _TYPE_CACHE[typename] = collections.namedtuple(typename,
97 + ' '.join(fields))
98 + return _TYPE_CACHE[typename](**values)
99 +
100 +
101 +def read_bucket_settings(phase: str, dim: str, **defaults):
102 + """Read bucketing configuration from env variables.
103 +
104 + phase is either 'prompt' or 'decode'
105 + dim is either 'bs', 'seq' or 'block'
106 + param is either 'min', 'step' or 'max'
107 + example env variable: VLLM_DECODE_BS_BUCKET_STEP=128
108 + """
109 + params = ['min', 'step', 'max']
110 + env_vars = [f'VLLM_{phase}_{dim}_BUCKET_{p}'.upper() for p in params]
111 + default_values = [defaults[p] for p in params]
112 + values = [
113 + int(os.environ.get(e, d)) for e, d in zip(env_vars, default_values)
114 + ]
115 + for e, v, d in zip(env_vars, values, default_values):
116 + logger.info('%s=%s (default:%s)', e, v, d)
117 + return values
118 +
119 +
120 +def warmup_range(config: Tuple[int, int, int]):
121 + """Generate a warmup range.
122 +
123 + Start from bmin and multiply by 2 until you reach bstep.
124 + Then, increase the values in the range by the value of bstep until you
125 + reach bmax.
126 +
127 + Example:
128 + bmin = 2, bstep = 32, bmax = 64
129 + => ramp_up = (2, 4, 8, 16)
130 + => stable = (32, 64)
131 + => return ramp_up + stable => (2, 4, 8, 16, 32, 64)
132 + """
133 + bmin, bstep, bmax = config
134 + assert bmin <= bmax, ("Min. batch size cannot be greater than max. "
135 + "batch size. If you want to skip warmup, "
136 + "set VLLM_SKIP_WARMUP=true")
137 + base = itertools.repeat(2)
138 + ramp_up_acc = itertools.accumulate(base, func=operator.mul, initial=bmin)
139 + ramp_up_tw = itertools.takewhile(lambda x: x < bstep and x <= bmax, \
140 + ramp_up_acc)
141 + stable = range(bstep, bmax + 1, bstep)
142 + buckets = list(ramp_up_tw) + list(stable)
143 + return list(filter(lambda bucket: bucket >= bmin, buckets))
144 +
145 +
146 +def generate_prompt_buckets(bs_bucket_config,
147 + seq_bucket_config,
148 + max_num_batched_tokens=None):
149 + buckets = list(
150 + itertools.product(warmup_range(bs_bucket_config),
151 + warmup_range(seq_bucket_config)))
152 + if len(buckets) == 0:
153 + msg = ("No buckets could be captured with following config "
154 + f"(min, step, max_warmup): "
155 + f"bs:{bs_bucket_config}, "
156 + f"seq:{seq_bucket_config}")
157 + raise ValueError(msg)
158 +
159 + filtered_buckets = buckets
160 + if max_num_batched_tokens is not None:
161 + # Remove buckets exceeding batch token budget
162 + filtered_buckets = list(
163 + filter(
164 + lambda bucket: bucket[0] * bucket[1] <= max_num_batched_tokens,
165 + buckets))
166 +
167 + if len(filtered_buckets) == 0:
168 + # we can handle this if we ignore max_num_batched_tokens
169 + min_bucket_bs, min_bucket_seq = min(buckets,
170 + key=lambda b: (b[0] * b[1]))
171 + min_reqd_budget = min_bucket_bs * min_bucket_seq
172 + msg = (
173 + "The current bucketing configuration "
174 + f"(min, step, max_warmup): "
175 + f"bs:{bs_bucket_config}, "
176 + f"seq:{seq_bucket_config} cannot be used with specified "
177 + f"max_num_batched_tokens ({max_num_batched_tokens}), as the "
178 + f"smallest bucket ({min_reqd_budget}) would exceed token "
179 + "budget. Please increase max_num_batched_tokens or decrease "
180 + "bucket minimum Ignoring max_num_batched_tokens at risk of "
181 + "out-of-memory errors.")
182 + logger.error(msg)
183 + return list(
184 + sorted(buckets, key=lambda b: (b[0] * b[1], b[1], b[0]))), []
185 +
186 + captured_buckets = list(
187 + sorted(filtered_buckets, key=lambda b: (b[0] * b[1], b[1], b[0])))
188 + omitted_buckets = list(
189 + sorted([x for x in buckets if x not in filtered_buckets]))
190 + return captured_buckets, omitted_buckets
191 +
192 +
193 +def generate_decode_buckets(bs_bucket_config, blocks_bucket_config,
194 + max_blocks):
195 + buckets = []
196 + bs_buckets = warmup_range(bs_bucket_config)
197 + block_buckets = warmup_range(blocks_bucket_config)
198 + bmin, bstep, bmax = blocks_bucket_config
199 + last_bucket = round_up(max_blocks, bstep)
200 + for bs in bs_buckets:
201 + for blocks in block_buckets:
202 + if blocks < bs:
203 + continue
204 + if blocks > last_bucket:
205 + break
206 + buckets.append((bs, blocks))
207 + return list(sorted(buckets, key=lambda b: (b[0] * b[1], b[1], b[0])))
208 +
209 +
210 +def next_pow2(value: int, base: int):
211 + res = base
212 + while value > 1:
213 + value = (value + 1) // 2
214 + res *= 2
215 + return res
216 +
217 +
218 +def round_up(value: int, k: int):
219 + return (value + k - 1) // k * k
220 +
221 +
222 +def find_bucket(value: int, config: Tuple[int, int, int]):
223 + bmin, bstep, _ = config
224 + next_step = round_up(value, bstep)
225 + next_pow = next_pow2(value, bmin)
226 + return max(bmin, min(next_step, next_pow))
227 +
228 +
229 +def align_workers(value, op):
230 + group = get_world_group().cpu_group
231 + world_size = torch.distributed.get_world_size()
232 + if world_size <= 1:
233 + return value
234 + value_t = torch.tensor(value, device='cpu')
235 + torch.distributed.all_reduce(value_t, op=op, group=group)
236 + return value_t.item()
237 +
238 +
239 +def setup_profiler():
240 + schedule = torch.profiler.schedule(wait=0, warmup=2, active=1, repeat=1)
241 + DEVICE = 'hpu'
242 + activities = [torch.profiler.ProfilerActivity.CPU]
243 + activities.extend([torch.profiler.ProfilerActivity.HPU] if DEVICE ==
244 + 'hpu' else [])
245 + #from habana_frameworks.torch.activity_profiler import DebugActivity
246 + #debug_activities=[DebugActivity.BRIDGE_FUNCTION_CALLS]
247 +
248 + profiler = torch.profiler.profile(
249 + schedule=schedule,
250 + activities=activities,
251 + #debug_activities=debug_activities,
252 + on_trace_ready=torch.profiler.tensorboard_trace_handler('.',
253 + use_gzip=True),
254 + record_shapes=False,
255 + with_stack=True)
256 + return profiler
257 +
258 +
259 +def pad_list(list, k, v):
260 + target_len = round_up(len(list), k)
261 + padding = target_len - len(list)
262 + return list + [v] * padding
263 +
264 +
265 +def precompute_indices_and_offsets(block_size, slot_mapping, is_prompt):
266 + slot_mapping = slot_mapping.flatten()
267 + indices = torch.div(slot_mapping, block_size, rounding_mode="floor")
268 + if is_prompt:
269 + indices = indices.unflatten(0, (-1, block_size))[:, 0]
270 + offsets = None
271 + else:
272 + offsets = torch.fmod(slot_mapping, block_size)
273 + return indices, offsets
274 +
275 +
276 +class HpuModelAdapter():
277 +
278 + def __init__(self, model, block_size, dtype, enforce_eager):
279 + self.model = model
280 + self.prefill_use_fusedsdpa = os.getenv('VLLM_PROMPT_USE_FUSEDSDPA',
281 + '0').lower() in ['1', 'true']
282 + self.block_size = block_size
283 + self.dtype = dtype
284 + if not htorch.utils.internal.is_lazy() and not enforce_eager:
285 + self.model = torch.compile(self.model,
286 + backend='hpu_backend',
287 + dynamic=False)
288 +
289 + def _set_attn_bias(self, attn_metadata, batch_size, seq_len, device,
290 + dtype):
291 + prefill_metadata = attn_metadata
292 + if prefill_metadata is None or self.prefill_use_fusedsdpa:
293 + return attn_metadata
294 +
295 + seq_lens_t = prefill_metadata.seq_lens_tensor
296 + len_mask = (torch.arange(0, seq_len, device=device,
297 + dtype=torch.int32).view(1, seq_len).ge(
298 + seq_lens_t.unsqueeze(-1)).view(
299 + batch_size, 1, 1, seq_len))
300 + causal_mask = torch.triu(torch.ones((batch_size, 1, seq_len, seq_len),
301 + device=device,
302 + dtype=torch.bool),
303 + diagonal=1)
304 + mask = causal_mask.logical_or(len_mask)
305 + attn_bias = (torch.zeros_like(mask, dtype=dtype).masked_fill_(
306 + mask, -math.inf))
307 + attn_metadata = prefill_metadata._replace(attn_bias=attn_bias)
308 + return attn_metadata
309 +
310 + def _set_block_mapping(self, metadata, batch_size, device, dtype):
311 + mask = torch.arange(0,
312 + self.block_size,
313 + device=device,
314 + dtype=torch.int32).unsqueeze(0)
315 + mask = mask >= metadata.block_usage.unsqueeze(-1)
316 + attn_bias = (torch.zeros_like(mask, dtype=dtype).masked_fill_(
317 + mask, -math.inf))
318 + block_mapping = torch.nn.functional.one_hot(metadata.block_mapping,
319 + num_classes=batch_size)
320 + block_mapping = block_mapping.to(dtype)
321 + metadata = metadata._replace(block_mapping=block_mapping,
322 + attn_bias=attn_bias)
323 + return metadata
324 +
325 + def _update_metadata(self, attn_metadata, batch_size, seq_len, device,
326 + dtype):
327 + if attn_metadata.is_prompt:
328 + meta = attn_metadata
329 + attn_metadata = self._set_attn_bias(meta, batch_size, seq_len,
330 + device, dtype)
331 + else:
332 + meta = attn_metadata
333 + attn_metadata = self._set_block_mapping(meta, batch_size, device,
334 + dtype)
335 + return attn_metadata
336 +
337 + def forward(self, *args, **kwargs):
338 + kwargs = kwargs.copy()
339 + selected_token_indices = kwargs.pop('selected_token_indices')
340 + if 'warmup_mode' in kwargs:
341 + kwargs.pop('warmup_mode')
342 + input_ids = kwargs['input_ids']
343 + kwargs['attn_metadata'] = self._update_metadata(
344 + kwargs['attn_metadata'], input_ids.size(0), input_ids.size(1),
345 + input_ids.device, self.dtype)
346 + LoraMask.setLoraMask(kwargs.pop('lora_mask'))
347 + hidden_states = self.model(*args, **kwargs)
348 + hidden_states = hidden_states.view(-1, hidden_states.shape[-1])
349 + hidden_states = hidden_states.index_select(0, selected_token_indices)
350 + return hidden_states
351 +
352 + def compute_logits(self, *args, **kwargs):
353 + return self.model.compute_logits(*args, **kwargs)
354 +
355 + def sample(self, *args, **kwargs):
356 + return self.model.sample(*args, **kwargs)
357 +
358 +
359 +class PreparePromptMetadata(NamedTuple):
360 + input_tokens: torch.Tensor
361 + input_positions: List[List[int]]
362 + attn_metadata: Optional[AttentionMetadata]
363 + seq_lens: List[int]
364 + query_lens: List[int]
365 + lora_index_mapping: List[List[int]]
366 + lora_prompt_mapping: List[List[int]]
367 + lora_requests: Set[LoRARequest]
368 + multi_modal_kwargs: Optional[Dict[str, BatchedTensorInputs]]
369 + slot_mapping: List[List[int]]
370 + lora_ids: List[int]
371 +
372 + @classmethod
373 + def empty(cls):
374 + return PreparePromptMetadata(input_tokens=[],
375 + input_positions=[],
376 + attn_metadata=None,
377 + seq_lens=[],
378 + query_lens=[],
379 + lora_index_mapping=[],
380 + lora_prompt_mapping=[],
381 + lora_requests=set(),
382 + multi_modal_kwargs=None,
383 + slot_mapping=[],
384 + lora_ids=[])
385 +
386 +
387 +class PrepareDecodeMetadata(NamedTuple):
388 + input_tokens: torch.Tensor
389 + input_positions: List[List[int]]
390 + attn_metadata: Optional[AttentionMetadata]
391 + lora_index_mapping: List[List[int]]
392 + lora_prompt_mapping: List[List[int]]
393 + lora_requests: Set[LoRARequest]
394 + slot_mapping: List[List[int]]
395 + lora_ids: List[int]
396 +
397 + @classmethod
398 + def empty(cls):
399 + return PrepareDecodeMetadata(input_tokens=[],
400 + input_positions=[],
401 + attn_metadata=None,
402 + lora_index_mapping=[],
403 + lora_prompt_mapping=[],
404 + lora_requests=set(),
405 + slot_mapping=[],
406 + lora_ids=[])
407 +
408 +
409 +# How batches are constructed.
410 +class BatchType(IntEnum):
411 + # Every batch is prefill.
412 + PREFILL = 0
413 + # Every batch is decode.
414 + DECODE = 1
415 + # Batch is a mixture of prefill and decode.
416 + MIXED = 2
417 +
418 +
419 +TModelInputForHPU = TypeVar('TModelInputForHPU', bound="ModelInputForHPU")
420 +
421 +
422 +@dataclasses.dataclass(frozen=True)
423 +class ModelInputForHPU(ModelRunnerInputBase):
424 + """
425 + This base class contains metadata needed for the base model forward pass
426 + but not metadata for possible additional steps, e.g., sampling. Model
427 + runners that run additional steps should subclass this method to add
428 + additional fields.
429 + """
430 + input_tokens: Optional[torch.Tensor] = None
431 + input_positions: Optional[torch.Tensor] = None
432 + seq_lens: Optional[List[int]] = None
433 + query_lens: Optional[List[int]] = None
434 + lora_mapping: Optional["LoRAMapping"] = None
435 + lora_requests: Optional[Set[LoRARequest]] = None
436 + attn_metadata: Optional["AttentionMetadata"] = None
437 + multi_modal_kwargs: Optional[Dict[str, torch.Tensor]] = None
438 + real_batch_size: Optional[int] = None
439 + batch_size_padded: Optional[int] = None
440 + virtual_engine: int = 0
441 + lora_ids: Optional[List[int]] = None
442 + async_callback: Optional[Callable] = None
443 +
444 + def as_broadcastable_tensor_dict(self) -> Dict[str, Any]:
445 + tensor_dict = {
446 + "input_tokens": self.input_tokens,
447 + "input_positions": self.input_positions,
448 + "lora_requests": self.lora_requests,
449 + "lora_mapping": self.lora_mapping,
450 + "multi_modal_kwargs": self.multi_modal_kwargs,
451 + "real_batch_size": self.real_batch_size,
452 + "batch_size_padded": self.batch_size_padded,
453 + "virtual_engine": self.virtual_engine,
454 + "lora_ids": self.lora_ids,
455 + }
456 + _add_attn_metadata_broadcastable_dict(tensor_dict, self.attn_metadata)
457 + return tensor_dict
458 +
459 + @classmethod
460 + def from_broadcasted_tensor_dict(
461 + cls: Type[TModelInputForHPU],
462 + tensor_dict: Dict[str, Any],
463 + attn_backend: Optional["AttentionBackend"] = None,
464 + ) -> TModelInputForHPU:
465 + if attn_backend is not None:
466 + tensor_dict = _init_attn_metadata_from_tensor_dict(
467 + attn_backend, tensor_dict)
468 + return cls(**tensor_dict)
469 +
470 +
471 +@dataclasses.dataclass(frozen=True)
472 +class ModelInputForHPUWithSamplingMetadata(ModelInputForHPU):
473 + """
474 + Used by the ModelRunner.
475 + """
476 + sampling_metadata: Optional["SamplingMetadata"] = None
477 + # Used for speculative decoding. We do not broadcast it because it is only
478 + # used by the driver worker.
479 + is_prompt: Optional[bool] = None
480 +
481 + def as_broadcastable_tensor_dict(self) -> Dict[str, Any]:
482 + tensor_dict = {
483 + "input_tokens": self.input_tokens,
484 + "input_positions": self.input_positions,
485 + "lora_requests": self.lora_requests,
486 + "lora_mapping": self.lora_mapping,
487 + "multi_modal_kwargs": self.multi_modal_kwargs,
488 + "lora_ids": self.lora_ids,
489 + }
490 + _add_attn_metadata_broadcastable_dict(tensor_dict, self.attn_metadata)
491 + _add_sampling_metadata_broadcastable_dict(tensor_dict,
492 + self.sampling_metadata)
493 + return tensor_dict
494 +
495 + @classmethod
496 + def from_broadcasted_tensor_dict(
497 + cls,
498 + tensor_dict: Dict[str, Any],
499 + attn_backend: Optional["AttentionBackend"] = None,
500 + ) -> "ModelInputForHPUWithSamplingMetadata":
501 + tensor_dict = _init_sampling_metadata_from_tensor_dict(tensor_dict)
502 + # FIXME(kzawora): this fails for whatever reason - why?
503 + if attn_backend is not None:
504 + tensor_dict = _init_attn_metadata_from_tensor_dict(
505 + attn_backend, tensor_dict)
506 + return cls(**tensor_dict)
507 +
508 +
509 +class HPUModelRunnerBase(ModelRunnerBase[TModelInputForHPU]):
510 + """
511 + Helper class for shared methods between GPU model runners.
512 + """
513 + _model_input_cls: Type[TModelInputForHPU]
514 +
515 + def __init__(
516 + self,
517 + vllm_config: VllmConfig,
518 + is_driver_worker: bool = False,
519 + return_hidden_states: bool = False,
520 + ):
521 + ModelRunnerBase.__init__(self, vllm_config=vllm_config)
522 + self.is_driver_worker = is_driver_worker
523 + self.return_hidden_states = return_hidden_states
524 +
525 + self.sliding_window = (self.model_config.get_sliding_window()
526 + if self.model_config is not None else None)
527 + self.device_config = (self.device_config if self.device_config
528 + is not None else DeviceConfig())
529 + self.device = self.device_config.device
530 + self.enforce_eager = self.model_config.enforce_eager
531 + self.max_num_seqs = self.scheduler_config.max_num_seqs
532 + # NOTE(kzawora): Change that to scheduler_config.max_num_prefill_seqs
533 + # once padding-aware scheduling gets merged
534 + self.max_num_prefill_seqs = 64
535 + self.max_model_len = self.scheduler_config.max_model_len
536 + self.max_num_batched_tokens = \
537 + self.scheduler_config.max_num_batched_tokens
538 + self.block_size = self.cache_config.block_size
539 +
540 + self.pin_memory = is_pin_memory_available()
541 + self.kv_cache_dtype = self.cache_config.cache_dtype
542 +
543 + self.attn_backend = get_attn_backend(
544 + self.model_config.get_head_size(),
545 + self.model_config.dtype,
546 + self.kv_cache_dtype,
547 + self.block_size,
548 + self.model_config.is_attention_free,
549 + )
550 +
551 + # Lazy initialization
552 + self.lora_manager: LRUCacheWorkerLoRAManager = None
553 + self.model: torch.nn.Module = None
554 + self.inc_initialized_successfully = False
555 +
556 + # Profiler stats
557 + self.profiler = HabanaHighLevelProfiler()
558 + self.profiler_counter_helper = HabanaProfilerCounterHelper()
559 + self.seen_configs: set = set()
560 + self._mem_margin: Optional[int] = None
561 + self.bucketing_global_state = HPUBucketingGlobalState()
562 + self._setup_buckets()
563 + self._set_gc_threshold()
564 +
565 + def _set_gc_threshold(self) -> None:
566 + # Read https://docs.python.org/3/library/gc.html#gc.set_threshold
567 + # for comprehensive description of gc generations.
568 + # We can either use VLLM_GC_THR_GEN[0-2] (this has higher priority)
569 + # to set particular generation threshold or use simpler
570 + # VLLM_GC_THR_MULTIPLIER to multiply default values.
571 + default_gc_thrs = list(gc.get_threshold())
572 + requested_gc_thrs = [0] * len(default_gc_thrs)
573 + for i in range(len(default_gc_thrs)):
574 + requested_gc_thrs[i] = int(
575 + os.environ.get(f'VLLM_GC_THR_GEN{i}', default_gc_thrs[i]))
576 + if requested_gc_thrs == default_gc_thrs:
577 + gc_thr_multiplier = int(os.environ.get('VLLM_GC_THR_MULTIPLIER',
578 + 2))
579 + requested_gc_thrs = [
580 + t * gc_thr_multiplier for t in default_gc_thrs
581 + ]
582 + gc.set_threshold(*requested_gc_thrs)
583 +
584 + # Multi-modal data support
585 + self.multi_modal_input_mapper = MULTIMODAL_REGISTRY \
586 + .create_input_mapper(self.model_config)
587 +
588 + self.skip_warmup = os.environ.get('VLLM_SKIP_WARMUP',
589 + 'false').lower() == 'true'
590 +
591 + def load_model(self) -> None:
592 + import habana_frameworks.torch.core as htcore
593 + if self.model_config.quantization == 'inc' or \
594 + self.model_config.quantization == 'fp8':
595 + htcore.hpu_set_env()
596 + with HabanaMemoryProfiler() as m:
597 + with HabanaMemoryProfiler() as m_getmodel:
598 + self.model = get_model(vllm_config=self.vllm_config)
599 + msg = ("Pre-loading model weights on "
600 + f"{next(self.model.parameters()).device} "
601 + f"took {m_getmodel.get_summary_string()}")
602 + logger.info(msg)
603 +
604 + if self.lora_config:
605 + assert hasattr(self.model, "supported_lora_modules"
606 + ) and self.model.supported_lora_modules, (
607 + "Model does not support LoRA")
608 + assert hasattr(self.model, "embedding_modules"
609 + ), "Model does not have embedding_modules"
610 + assert hasattr(
611 + self.model, "embedding_padding_modules"
612 + ), "Model does not have embedding_padding_modules"
613 + self.lora_manager = LRUCacheWorkerLoRAManager(
614 + self.scheduler_config.max_num_seqs,
615 + self.scheduler_config.max_num_batched_tokens,
616 + self.vocab_size, self.lora_config, self.device,
617 + self.model.embedding_modules,
618 + self.model.embedding_padding_modules)
619 + self.model = self.lora_manager.create_lora_manager(self.model)
620 +
621 + if self.model_config.quantization == 'inc':
622 + logger.info("Preparing model with INC..")
623 + with HabanaMemoryProfiler() as m_inc:
624 + from neural_compressor.torch.quantization import (
625 + FP8Config, convert, prepare)
626 + config = FP8Config.from_json_file(
627 + os.getenv("QUANT_CONFIG", ""))
628 + if config.measure:
629 + self.model = prepare(self.model, config)
630 + elif config.quantize:
631 + self.model = convert(self.model, config)
632 + htcore.hpu_initialize(self.model,
633 + mark_only_scales_as_const=True)
634 + self.inc_initialized_successfully = True
635 + logger.info("Preparing model with INC took %s",
636 + m_inc.get_summary_string())
637 + else:
638 + self.model = self.model.to("hpu")
639 + htcore.mark_step()
640 + torch.hpu.synchronize()
641 +
642 + with HabanaMemoryProfiler() as m_wrap:
643 + self.model = _maybe_wrap_in_hpu_graph(
644 + self.model,
645 + self.block_size,
646 + dtype=self.model_config.dtype,
647 + enforce_eager=self.enforce_eager)
648 + msg = f"Wrapping in HPU Graph took {m_wrap.get_summary_string()}"
649 + logger.info(msg)
650 +
651 + self.model_memory_usage = m.consumed_device_memory
652 + msg = f"Loading model weights took in total {m.get_summary_string()}"
653 + logger.info(msg)
654 +
655 + def _use_graphs(self, batch_size, seq_len, is_prompt):
656 + if self.enforce_eager:
657 + return False
658 + if self.skip_warmup:
659 + return True
660 + return (batch_size, seq_len, is_prompt) in self.graphed_buckets
661 +
662 + def _is_valid_bucket(self, bucket):
663 + return bucket[0] * bucket[1] <= self.max_num_batched_tokens
664 +
665 + def _setup_buckets(self) -> None:
666 + align_bs = lambda x: min(self.max_num_seqs, x)
667 + #FIXME: The default values should be max_model_len
668 + max_prompt_seq = 1024
669 + max_decode_seq = 2048
670 + self.bucketing_global_state.prompt_bs_bucket_cfg = read_bucket_settings(
671 + 'prompt',
672 + 'bs',
673 + min=1,
674 + step=align_bs(32),
675 + max=self.max_num_prefill_seqs)
676 + self.bucketing_global_state.decode_bs_bucket_cfg = read_bucket_settings(
677 + 'decode', 'bs', min=1, step=align_bs(32), max=self.max_num_seqs)
678 + self.bucketing_global_state.prompt_seq_bucket_cfg = \
679 + read_bucket_settings(
680 + 'prompt',
681 + 'seq',
682 + min=self.block_size,
683 + step=self.block_size,
684 + max=max_prompt_seq)
685 + self.bucketing_global_state.decode_block_bucket_cfg = \
686 + read_bucket_settings(
687 + 'decode',
688 + 'block',
689 + min=self.block_size,
690 + step=self.block_size,
691 + max=max(self.block_size,
692 + self.max_num_seqs * max_decode_seq // self.block_size))
693 + self.graphed_buckets: Set[Any] = set()
694 +
695 + msg = ("Prompt bucket config (min, step, max_warmup) "
696 + f"bs:{self.bucketing_global_state.prompt_bs_bucket_cfg}, "
697 + f"seq:{self.bucketing_global_state.prompt_seq_bucket_cfg}")
698 + logger.info(msg)
699 +
700 + msg = ("Decode bucket config (min, step, max_warmup) "
701 + f"bs:{self.bucketing_global_state.decode_bs_bucket_cfg}, "
702 + f"block:{self.bucketing_global_state.decode_block_bucket_cfg}")
703 + logger.info(msg)
704 +
705 + def _prepare_prompt(
706 + self,
707 + seq_group_metadata_list: List[SequenceGroupMetadata],
708 + ) -> PreparePromptMetadata:
709 + input_tokens: List[List[int]] = []
710 + input_positions: List[List[int]] = []
711 + slot_mapping: List[List[int]] = []
712 + lora_index_mapping: List[List[int]] = []
713 + lora_prompt_mapping: List[List[int]] = []
714 + lora_requests: Set[LoRARequest] = set()
715 +
716 + seq_lens: List[int] = []
717 + context_lens: List[int] = []
718 + query_lens: List[int] = []
719 + prefix_block_tables: List[List[int]] = []
720 + multi_modal_inputs_list: List[MultiModalInputs] = []
721 +
722 + if len(seq_group_metadata_list) == 0:
723 + return PreparePromptMetadata.empty()
724 +
725 + for seq_group_metadata in seq_group_metadata_list:
726 + assert seq_group_metadata.is_prompt
727 + seq_ids = list(seq_group_metadata.seq_data.keys())
728 + assert len(seq_ids) == 1
729 + seq_id = seq_ids[0]
730 +
731 + computed_block_nums = seq_group_metadata.computed_block_nums
732 + if (self.scheduler_config is not None
733 + and self.scheduler_config.chunked_prefill_enabled
734 + and not (computed_block_nums is None
735 + or computed_block_nums == [])):
736 + raise RuntimeError(
737 + "chunked prefill cannot be used with prefix caching "
738 + "now.")
739 +
740 + token_chunk_size = seq_group_metadata.token_chunk_size
741 + seq_data = seq_group_metadata.seq_data[seq_id]
742 + context_len = seq_data.get_num_computed_tokens()
743 + # We should use get_len here because in case of preemption
744 + # it contains output tokens.
745 + seq_len = min(seq_data.get_len(), context_len + token_chunk_size)
746 + prompt_tokens = seq_data.get_token_ids()[context_len:seq_len]
747 + seq_lens.append(seq_len)
748 +
749 + # NOTE: This only works for oooooooxxx style attention.
750 + if computed_block_nums is not None and len(
751 + computed_block_nums) > 0 and self.sliding_window is None:
752 + # Prefix is not supported with sliding_window
753 + context_len = len(computed_block_nums) * self.block_size
754 + prompt_tokens = prompt_tokens[context_len:]
755 + prefix_block_tables.append(computed_block_nums)
756 + elif self.scheduler_config.chunked_prefill_enabled:
757 + if seq_group_metadata.block_tables is not None:
758 + # Prefill has chunked before.
759 + block_table = seq_group_metadata.block_tables[seq_id]
760 + prefix_block_tables.append(block_table)
761 + else:
762 + # The first prefill.
763 + prefix_block_tables.append([])
764 + else:
765 + prefix_block_tables.append([])
766 + # Right now, prefill start is always 0. However, this
767 + # assumption can be changed once chunked prefill is introduced.
768 + assert context_len == 0
769 +
770 + # actual prompt lens
771 + context_lens.append(context_len)
772 + query_lens.append(seq_len - context_len)
773 + input_tokens.append(prompt_tokens)
774 + # NOTE(woosuk): Here we assume that the first token in the prompt
775 + # is always the first token in the sequence.
776 + input_positions.append(list(range(context_len, seq_len)))
777 +
778 + mm_data = seq_group_metadata.multi_modal_data
779 + if mm_data:
780 + mm_kwargs = self.multi_modal_input_mapper(mm_data)
781 + multi_modal_inputs_list.append(mm_kwargs)
782 +
783 + if seq_group_metadata.block_tables is None:
784 + # During memory profiling, the block tables are not initialized
785 + # yet. In this case, we just use a dummy slot mapping.
786 + slot_mapping.append([_PAD_SLOT_ID] * seq_len)
787 + continue
788 +
789 + # Compute the slot mapping.
790 + slot_mapping.append([])
791 + block_table = seq_group_metadata.block_tables[seq_id]
792 +
793 + # Mask the [0, start_idx) tokens of the prompt with _PAD_SLOT_ID,
794 + # where start_idx is max(0, seq_len - sliding_window).
795 + # For example, if the prompt len is 10, sliding window is 8, and
796 + # block size is 4, the first two tokens are masked and the slot
797 + # mapping will be [-1, -1, 2, 3, 4, 5, 6, 7, 0, 1].
798 + start_idx = 0
799 + if self.sliding_window is not None:
800 + assert context_len == 0, (
801 + "Prefix caching is currently not supported with "
802 + "sliding window attention")
803 + start_idx = max(0, seq_len - self.sliding_window)
804 + for i in range(context_len, seq_len):
805 + if i < start_idx:
806 + slot_mapping[-1].append(_PAD_SLOT_ID)
807 + continue
808 +
809 + block_number = block_table[i // self.block_size]
810 + block_offset = i % self.block_size
811 + slot = block_number * self.block_size + block_offset
812 + slot_mapping[-1].append(slot)
813 +
814 + max_query_len = max(query_lens)
815 + sum_query_len = sum(query_lens)
816 + real_num_seqs = len(query_lens)
817 + assert max_query_len > 0
818 +
819 + max_prompt_len = max(
820 + find_bucket(max(seq_lens),
821 + self.bucketing_global_state.prompt_seq_bucket_cfg),
822 + self.block_size)
823 +
824 + lora_ids: List[int] = []
825 + for seq_group_metadata, context_len in zip(seq_group_metadata_list,
826 + context_lens):
827 + lora_id = seq_group_metadata.lora_int_id
828 + lora_ids.append(lora_id)
829 +
830 + if lora_id > 0:
831 + lora_requests.add(seq_group_metadata.lora_request)
832 +
833 + lora_index_mapping += [lora_id] * (max_prompt_len - context_len)
834 + lora_prompt_mapping.extend(
835 + [lora_id] *
836 + (max_prompt_len - context_len
837 + if seq_group_metadata.sampling_params.prompt_logprobs else 1))
838 +
839 + input_tokens = make_tensor_with_pad(input_tokens,
840 + max_len=max_prompt_len,
841 + pad=0,
842 + dtype=torch.long,
843 + device=self.device)
844 +
845 + input_positions = make_tensor_with_pad(input_positions,
846 + max_len=max_prompt_len,
847 + pad=0,
848 + dtype=torch.long,
849 + device=self.device)
850 +
851 + slot_mapping = make_tensor_with_pad(slot_mapping,
852 + max_len=max_prompt_len,
853 + pad=_PAD_SLOT_ID,
854 + dtype=torch.long,
855 + device=self.device)
856 +
857 + seq_lens_tensor = torch.tensor(seq_lens,
858 + dtype=torch.long,
859 + device=self.device)
860 +
861 + block_indices, block_offsets = precompute_indices_and_offsets(
862 + self.block_size, slot_mapping, True)
863 + attn_metadata = self.attn_backend.make_metadata(
864 + is_prompt=True,
865 + block_list=None,
866 + block_mapping=None,
867 + block_usage=None,
868 + block_indices=block_indices,
869 + block_offsets=block_offsets,
870 + block_scales=None,
871 + attn_bias=None,
872 + seq_lens_tensor=seq_lens_tensor,
873 + num_prefills=real_num_seqs,
874 + num_prefill_tokens=sum_query_len,
875 + num_decode_tokens=0,
876 + slot_mapping=slot_mapping,
877 + multi_modal_placeholder_index_maps=
878 + None # FIXME(kzawora): mutli-modality will not work here
879 + )
880 + multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list)
881 +
882 + return PreparePromptMetadata(input_tokens=input_tokens,
883 + input_positions=input_positions,
884 + attn_metadata=attn_metadata,
885 + seq_lens=seq_lens,
886 + query_lens=query_lens,
887 + lora_index_mapping=lora_index_mapping,
888 + lora_prompt_mapping=lora_prompt_mapping,
889 + lora_requests=lora_requests,
890 + multi_modal_kwargs=multi_modal_kwargs,
891 + slot_mapping=slot_mapping,
892 + lora_ids=lora_ids)
893 +
894 + def _prepare_decode(
895 + self,
896 + seq_group_metadata_list: List[SequenceGroupMetadata],
897 + ) -> PrepareDecodeMetadata:
898 + input_tokens: List[List[int]] = []
899 + input_positions: List[List[int]] = []
900 + slot_mapping: List[List[int]] = []
901 + seq_lens: List[int] = []
902 + block_tables: List[List[int]] = []
903 + lora_index_mapping: List[List[int]] = []
904 + lora_prompt_mapping: List[List[int]] = []
905 + lora_requests: Set[LoRARequest] = set()
906 +
907 + if len(seq_group_metadata_list) == 0:
908 + return PrepareDecodeMetadata.empty()
909 + lora_ids: List[int] = []
910 +
911 + dummy_slots = itertools.cycle(
912 + range(_PAD_SLOT_ID, _PAD_SLOT_ID + self.block_size))
913 +
914 + for seq_group_metadata in seq_group_metadata_list:
915 + assert not seq_group_metadata.is_prompt
916 + assert seq_group_metadata.token_chunk_size == 1
917 +
918 + seq_ids = list(seq_group_metadata.seq_data.keys())
919 + lora_id = seq_group_metadata.lora_int_id
920 + lora_ids.append(lora_id)
921 +
922 + if lora_id > 0:
923 + lora_requests.add(seq_group_metadata.lora_request)
924 +
925 + for seq_id in seq_ids:
926 + seq_data = seq_group_metadata.seq_data[seq_id]
927 + generation_token = seq_data.get_last_token_id()
928 + input_tokens.append([generation_token])
929 +
930 + seq_len = seq_data.get_len()
931 + position = seq_len - 1
932 + input_positions.append([position])
933 +
934 + seq_len = seq_len if self.sliding_window is None else min(
935 + seq_len, self.sliding_window)
936 + seq_lens.append(seq_len)
937 +
938 + block_table = seq_group_metadata.block_tables[seq_id]
939 + if len(block_table) == 0:
940 + block_number = _PAD_BLOCK_ID
941 + else:
942 + block_number = block_table[position // self.block_size]
943 + if block_number == _PAD_BLOCK_ID:
944 + slot = next(dummy_slots)
945 + else:
946 + block_offset = position % self.block_size
947 + slot = block_number * self.block_size + block_offset
948 + slot_mapping.append([slot])
949 + lora_index_mapping.append(lora_id)
950 + lora_prompt_mapping.append(lora_id)
951 +
952 + if self.sliding_window is not None:
953 + sliding_window_blocks = (self.sliding_window //
954 + self.block_size)
955 + block_table = block_table[-sliding_window_blocks:]
956 + block_tables.append(block_table)
957 +
958 + input_tokens = torch.tensor(input_tokens,
959 + dtype=torch.long,
960 + device=self.device)
961 + input_positions = torch.tensor(input_positions,
962 + dtype=torch.long,
963 + device=self.device)
964 +
965 + num_decode_tokens = sum(seq_lens)
966 +
967 + blocks_used = [len(bt) for bt in block_tables if bt]
968 + block_list = []
969 + block_scales = []
970 + for i, bt in enumerate(block_tables):
971 + block_list.extend(bt)
972 + blocks_in_group = len(bt)
973 + if blocks_in_group > 0:
974 + scale = 1.0 / blocks_in_group
975 + block_scales.extend([scale] * blocks_in_group)
976 +
977 + block_mapping_nested: List[List[int]] = [
978 + [i] * b_u for i, b_u in enumerate(blocks_used)
979 + ]
980 + block_mapping: List[int] = list(
981 + itertools.chain.from_iterable(block_mapping_nested))
982 +
983 + last_block = [
984 + sl % self.block_size + 1 for sl in itertools.chain(*slot_mapping)
985 + ]
986 + block_usage = [[self.block_size] * (b_u - 1) + [lb]
987 + for b_u, lb in zip(blocks_used, last_block)]
988 + block_usage = list(itertools.chain(*block_usage))
989 +
990 + block_bucket_size = find_bucket(
991 + len(block_list),
992 + self.bucketing_global_state.decode_block_bucket_cfg)
993 + block_list = pad_list(block_list, block_bucket_size, _PAD_BLOCK_ID)
994 + block_mapping = pad_list(block_mapping, block_bucket_size, -1)
995 + block_usage = pad_list(block_usage, block_bucket_size, 1)
996 + block_scales = pad_list(block_scales, block_bucket_size, 0.0)
997 +
998 + block_list = torch.tensor(block_list,
999 + dtype=torch.int,
1000 + device=self.device)
1001 + block_mapping = torch.tensor(block_mapping,
1002 + dtype=torch.long,
1003 + device=self.device)
1004 + block_usage = torch.tensor(block_usage,
1005 + dtype=self.model_config.dtype,
1006 + device=self.device)
1007 +
1008 + slot_mapping = torch.tensor(slot_mapping,
1009 + dtype=torch.long,
1010 + device=self.device)
1011 +
1012 + block_indices, block_offsets = precompute_indices_and_offsets(
1013 + self.block_size, slot_mapping, False)
1014 + block_scales = torch.tensor(block_scales,
1015 + dtype=self.model_config.dtype,
1016 + device=self.device)
1017 +
1018 + attn_metadata = self.attn_backend.make_metadata(
1019 + is_prompt=False,
1020 + block_list=block_list,
1021 + block_mapping=block_mapping,
1022 + block_usage=block_usage,
1023 + block_indices=block_indices,
1024 + block_offsets=block_offsets,
1025 + block_scales=block_scales,
1026 + attn_bias=None,
1027 + seq_lens_tensor=None,
1028 + num_prefills=0,
1029 + num_prefill_tokens=0,
1030 + num_decode_tokens=num_decode_tokens,
1031 + slot_mapping=slot_mapping,
1032 + multi_modal_placeholder_index_maps=None)
1033 + return PrepareDecodeMetadata(input_tokens=input_tokens,
1034 + input_positions=input_positions,
1035 + attn_metadata=attn_metadata,
1036 + lora_index_mapping=lora_index_mapping,
1037 + lora_prompt_mapping=lora_prompt_mapping,
1038 + lora_requests=lora_requests,
1039 + slot_mapping=slot_mapping,
1040 + lora_ids=lora_ids)
1041 +
1042 + def prepare_input_tensors(
1043 + self,
1044 + seq_group_metadata_list: List[SequenceGroupMetadata],
1045 + ) -> Tuple[TModelInputForHPU, SamplingMetadata]:
1046 + if len(seq_group_metadata_list) == 0:
1047 + return self._model_input_cls(), None
1048 +
1049 + input_tokens = None
1050 + input_positions = None
1051 + lora_mapping = None
1052 + lora_requests = None
1053 + multi_modal_kwargs = None
1054 + batch_type = None
1055 + seq_lens = None
1056 + query_lens = None
1057 + real_batch_size = None
1058 + batch_size_padded = None
1059 +
1060 + self.event_start = self.profiler.get_timestamp_us()
1061 + is_prompt = seq_group_metadata_list[0].is_prompt
1062 + base_event_name = 'prompt' if is_prompt else 'decode'
1063 + self.profiler.start('internal', base_event_name)
1064 +
1065 + real_batch_size = len(seq_group_metadata_list)
1066 + bucket_cfg = self.bucketing_global_state.prompt_bs_bucket_cfg \
1067 + if is_prompt else self.bucketing_global_state.decode_bs_bucket_cfg
1068 + batch_size_padded = find_bucket(real_batch_size, bucket_cfg)
1069 + batch_size_padding = batch_size_padded - real_batch_size
1070 + seq_group_metadata_list = seq_group_metadata_list.copy()
1071 + if batch_size_padding > 0:
1072 + dummy_seq_group_metadata = self.create_dummy_seq_group_metadata(
1073 + 0, 0, is_prompt)
1074 + seq_group_metadata_list.extend(dummy_seq_group_metadata
1075 + for _ in range(batch_size_padding))
1076 +
1077 + prefill_reqs = []
1078 + decode_reqs = []
1079 + for seq_group_meta in seq_group_metadata_list:
1080 + if seq_group_meta.is_prompt:
1081 + prefill_reqs.append(seq_group_meta)
1082 + else:
1083 + decode_reqs.append(seq_group_meta)
1084 +
1085 + # Prepare input tensors.
1086 + (
1087 + input_tokens,
1088 + input_positions,
1089 + prefill_attn_metadata,
1090 + seq_lens,
1091 + query_lens,
1092 + lora_index_mapping,
1093 + lora_prompt_mapping,
1094 + lora_requests,
1095 + multi_modal_kwargs,
1096 + slot_mapping,
1097 + lora_ids,
1098 + ) = self._prepare_prompt(prefill_reqs)
1099 + (
1100 + decode_input_tokens,
1101 + decode_input_positions,
1102 + decode_attn_metadata,
1103 + decode_lora_index_mapping,
1104 + decode_lora_prompt_mapping,
1105 + decode_lora_requests,
1106 + decode_slot_mapping,
1107 + decode_lora_ids,
1108 + ) = self._prepare_decode(decode_reqs)
1109 + sampling_metadata = SamplingMetadata.prepare(seq_group_metadata_list,
1110 + seq_lens, query_lens,
1111 + self.device,
1112 + self.pin_memory)
1113 +
1114 + if not self.scheduler_config.chunked_prefill_enabled:
1115 + assert (len(prefill_reqs) and len(decode_reqs)) == 0
1116 +
1117 + num_prefills = len(seq_lens)
1118 + num_prefill_tokens = len(input_tokens)
1119 + num_decode_tokens = len(decode_input_tokens)
1120 +
1121 + # NOTE(kzawora): Here we diverge from GPU code - we don't
1122 + # support mixed batches, so we either use decode or prefill
1123 + # inputs, without coalescing.
1124 + assert (num_prefills == 0 and num_decode_tokens > 0) or (
1125 + num_prefills > 0
1126 + and num_decode_tokens == 0), "HPU does not support mixed batches!"
1127 + if num_decode_tokens > 0:
1128 + input_tokens = decode_input_tokens
1129 + input_positions = decode_input_positions
1130 + slot_mapping = decode_slot_mapping
1131 + lora_index_mapping = decode_lora_index_mapping
1132 + lora_prompt_mapping = decode_lora_prompt_mapping
1133 + lora_requests = decode_lora_requests
1134 + lora_ids = decode_lora_ids
1135 +
1136 + # FIXME: We need to adjust selected_token_indices to accommodate
1137 + # for padding
1138 + max_len = input_tokens.size(1)
1139 + paddings = [max_len - s for s in seq_lens]
1140 + paddings = [0] + paddings[:-1]
1141 + paddings = list(itertools.accumulate(paddings))
1142 + paddings_prompt_logprobs = []
1143 + for i, seq_group_metadata in enumerate(seq_group_metadata_list):
1144 + if seq_group_metadata.sampling_params.prompt_logprobs is not None \
1145 + and seq_group_metadata.is_prompt:
1146 + paddings_prompt_logprobs += ([paddings[i]] * seq_lens[i])
1147 + paddings = torch.tensor(
1148 + paddings_prompt_logprobs if paddings_prompt_logprobs else paddings,
1149 + dtype=sampling_metadata.selected_token_indices.dtype,
1150 + device=sampling_metadata.selected_token_indices.device)
1151 + sampling_metadata.selected_token_indices.add_(paddings)
1152 +
1153 + if self.lora_config:
1154 + lora_mapping = LoRAMapping(
1155 + **dict(index_mapping=lora_index_mapping,
1156 + prompt_mapping=lora_prompt_mapping,
1157 + is_prefill=(num_prefills > 0)))
1158 + else:
1159 + lora_mapping = None
1160 +
1161 + if (prefill_attn_metadata is not None
1162 + and decode_attn_metadata is not None):
1163 + batch_type = BatchType.MIXED
1164 + raise NotImplementedError("Mixed batch is not supported on HPU")
1165 + elif prefill_attn_metadata is not None:
1166 + batch_type = BatchType.PREFILL
1167 + else:
1168 + batch_type = BatchType.DECODE
1169 +
1170 + metadata_dict = {
1171 + "input_tokens": input_tokens,
1172 + "input_positions": input_positions,
1173 + "selected_token_indices": sampling_metadata.selected_token_indices,
1174 + "lora_requests": lora_requests,
1175 + "lora_mapping": lora_mapping,
1176 + "multi_modal_kwargs": multi_modal_kwargs,
1177 + "num_prefill_tokens": num_prefill_tokens,
1178 + "num_decode_tokens": num_decode_tokens,
1179 + "slot_mapping": slot_mapping,
1180 + "num_prefills": num_prefills,
1181 + "batch_type": batch_type,
1182 + "seq_lens": seq_lens,
1183 + "query_lens": query_lens
1184 + }
1185 + if prefill_attn_metadata is not None:
1186 + metadata_dict.update(prefill_attn_metadata.asdict_zerocopy())
1187 + else:
1188 + assert decode_attn_metadata is not None
1189 + metadata_dict.update(decode_attn_metadata.asdict_zerocopy())
1190 +
1191 + attn_metadata = prefill_attn_metadata if \
1192 + prefill_attn_metadata is not None else decode_attn_metadata
1193 +
1194 + return self._model_input_cls(input_tokens=input_tokens,
1195 + seq_lens=seq_lens,
1196 + query_lens=query_lens,
1197 + input_positions=input_positions,
1198 + attn_metadata=attn_metadata,
1199 + lora_requests=lora_requests,
1200 + lora_mapping=lora_mapping,
1201 + multi_modal_kwargs=multi_modal_kwargs,
1202 + real_batch_size=real_batch_size,
1203 + batch_size_padded=batch_size_padded,
1204 + lora_ids=lora_ids), \
1205 + sampling_metadata
1206 +
1207 + def _seq_len(self, attn_metadata):
1208 + if attn_metadata.num_prefills != 0:
1209 + return attn_metadata.slot_mapping.size(1)
1210 + else:
1211 + return attn_metadata.block_list.numel()
1212 +
1213 + def trim_attn_metadata(self, metadata: AttentionMetadata) -> object:
1214 + # NOTE(kzawora): To anyone working on this in the future:
1215 + # Trimming metadata is required when using HPUGraphs.
1216 + # Attention metadata is going to be hashed by PT bridge, and
1217 + # appropriate HPUGraphs will be matched based on all inputs' hash.
1218 +
1219 + # Before you put more keys in here, make sure you know their
1220 + # value type and make sure you know how it's going to be hashed.
1221 + # You can find that information in input_hash function
1222 + # in habana_frameworks/torch/hpu/graphs.py. You can also hash
1223 + # it manually with torch.hpu.graphs.input_hash(attention_metadata)
1224 +
1225 + # If you use primitive types here - they will get hashed based
1226 + # on their value. You *will* get lots of excessive graph captures
1227 + # (and an OOM eventually) if you decide to put something like
1228 + # seq_len int here.
1229 + # If you absolutely need a scalar, put it in a tensor. Tensors
1230 + # get hashed using their metadata, not their values:
1231 + # input_hash(torch.tensor(123)) == input_hash(torch.tensor(321))
1232 + # input_hash(123) != input_hash(321)
1233 + # input_hash("abc") != input_hash("cba")
1234 + attention_metadata = subtuple(metadata, 'TrimmedAttentionMetadata', [
1235 + 'attn_bias', 'seq_lens_tensor', 'block_list', 'block_mapping',
1236 + 'block_usage', 'slot_mapping', 'is_prompt', 'block_indices',
1237 + 'block_offsets', 'block_scales'
1238 + ])
1239 + return attention_metadata
1240 +
1241 + def create_dummy_seq_group_metadata(self,
1242 + group_id,
1243 + seq_len,
1244 + is_prompt,
1245 + lora_request=None):
1246 + sampling_params = SamplingParams(temperature=0)
1247 + num_blocks = math.ceil(seq_len / self.block_size)
1248 + seq_len = max(seq_len, 1)
1249 + if is_prompt:
1250 + input_len = seq_len
1251 + output_len = 0
1252 + block_tables = None
1253 + else:
1254 + input_len = seq_len - 1
1255 + output_len = 1
1256 + block_tables = {group_id: [_PAD_BLOCK_ID] * num_blocks}
1257 + prompt_token_ids = [0] * input_len
1258 + output_token_ids = [1] * output_len
1259 + prompt_token_ids_array = array('l', prompt_token_ids) # noqa: F821
1260 + seq_data = SequenceData(prompt_token_ids_array)
1261 + seq_data.output_token_ids = output_token_ids
1262 + return SequenceGroupMetadata(request_id=str(group_id),
1263 + is_prompt=(output_len == 0),
1264 + seq_data={group_id: seq_data},
1265 + sampling_params=sampling_params,
1266 + block_tables=block_tables,
1267 + lora_request=lora_request)
1268 +
1269 + def profile_run(self) -> None:
1270 + num_layers = self.model_config.get_num_layers(self.parallel_config)
1271 + kv_caches = [None] * num_layers
1272 + max_batch_size = self.bucketing_global_state.prompt_bs_bucket_cfg[-1]
1273 + max_seq_len = min(
1274 + self.bucketing_global_state.prompt_seq_bucket_cfg[-1],
1275 + self.max_num_batched_tokens // max_batch_size)
1276 +
1277 + self.warmup_scenario(max_batch_size, max_seq_len, True, kv_caches,
1278 + False, True)
1279 + return
1280 +
1281 + def warmup_scenario(self,
1282 + batch_size,
1283 + seq_len,
1284 + is_prompt,
1285 + kv_caches,
1286 + is_pt_profiler_run=False,
1287 + is_lora_profile_run=False) -> None:
1288 + use_graphs = self._use_graphs(batch_size, seq_len, is_prompt)
1289 + scenario_name = ("warmup_"
1290 + f"{'prompt' if is_prompt else 'decode'}_"
1291 + f"bs{batch_size}_"
1292 + f"seq{seq_len}_"
1293 + f"graphs{'T' if use_graphs else 'F'}")
1294 + max_num_seqs = self.scheduler_config.max_num_seqs
1295 + # This represents the maximum number of different requests
1296 + # that will have unique loras, an therefore the max amount of memory
1297 + # consumption create dummy lora request copies from the lora request
1298 + # passed in, which contains a lora from the lora warmup path.
1299 + dummy_lora_requests: List[LoRARequest] = []
1300 + dummy_lora_requests_per_seq: List[LoRARequest] = []
1301 + if self.lora_config and is_lora_profile_run:
1302 + assert self.lora_manager is not None
1303 + with self.lora_manager.dummy_lora_cache():
1304 + for idx in range(self.lora_config.max_loras):
1305 + lora_id = idx + 1
1306 + dummy_lora_request = LoRARequest(
1307 + lora_name=f"warmup_{lora_id}",
1308 + lora_int_id=lora_id,
1309 + lora_local_path="/not/a/real/path",
1310 + )
1311 + self.lora_manager.add_dummy_lora(dummy_lora_request,
1312 + rank=LORA_WARMUP_RANK)
1313 + dummy_lora_requests.append(dummy_lora_request)
1314 + dummy_lora_requests_per_seq = [
1315 + dummy_lora_requests[idx % len(dummy_lora_requests)]
1316 + for idx in range(max_num_seqs)
1317 + ]
1318 + self.profiler.start('internal', scenario_name)
1319 + times = 3 if use_graphs or is_pt_profiler_run else 1
1320 + if self.lora_config and not is_lora_profile_run:
1321 + lora_mapping = LoRAMapping(
1322 + **dict(index_mapping=[0] * batch_size * seq_len,
1323 + prompt_mapping=[0] * batch_size * seq_len,
1324 + is_prefill=is_prompt))
1325 + self.set_active_loras(set(), lora_mapping)
1326 + if is_prompt:
1327 + seqs = [
1328 + self.create_dummy_seq_group_metadata(
1329 + i,
1330 + seq_len,
1331 + is_prompt,
1332 + lora_request=dummy_lora_requests_per_seq[i]
1333 + if dummy_lora_requests_per_seq else None)
1334 + for i in range(batch_size)
1335 + ]
1336 + else:
1337 + # FIXME: seq_len is actually number of blocks
1338 + blocks = [seq_len // batch_size for _ in range(batch_size)]
1339 + blocks[0] += seq_len % batch_size
1340 + seqs = [
1341 + self.create_dummy_seq_group_metadata(
1342 + i,
1343 + b * self.block_size - 1,
1344 + is_prompt,
1345 + lora_request=dummy_lora_requests_per_seq[i]
1346 + if dummy_lora_requests_per_seq else None)
1347 + for i, b in enumerate(blocks)
1348 + ]
1349 + torch.hpu.synchronize()
1350 + profiler = None
1351 + if is_pt_profiler_run and self.is_driver_worker:
1352 + profiler = setup_profiler()
1353 + profiler.start()
1354 + for _ in range(times):
1355 + inputs = self.prepare_model_input(seqs)
1356 + self.execute_model(inputs, kv_caches, warmup_mode=True)
1357 + torch.hpu.synchronize()
1358 + if profiler:
1359 + profiler.step()
1360 + if profiler:
1361 + profiler.stop()
1362 + self.profiler.end()
1363 + gc.collect()
1364 +
1365 + def remove_all_loras(self):
1366 + if not self.lora_manager:
1367 + raise RuntimeError("LoRA is not enabled.")
1368 + self.lora_manager.remove_all_adapters()
1369 +
1370 + def set_active_loras(self, lora_requests: Set[LoRARequest],
1371 + lora_mapping: LoRAMapping) -> None:
1372 + if not self.lora_manager:
1373 + raise RuntimeError("LoRA is not enabled.")
1374 + self.lora_manager.set_active_adapters(lora_requests, lora_mapping)
1375 +
1376 + def add_lora(self, lora_request: LoRARequest) -> bool:
1377 + if not self.lora_manager:
1378 + raise RuntimeError("LoRA is not enabled.")
1379 + return self.lora_manager.add_adapter(lora_request)
1380 +
1381 + def remove_lora(self, lora_id: int) -> bool:
1382 + if not self.lora_manager:
1383 + raise RuntimeError("LoRA is not enabled.")
1384 + return self.lora_manager.remove_adapter(lora_id)
1385 +
1386 + def pin_lora(self, lora_id: int) -> bool:
1387 + if not self.lora_manager:
1388 + raise RuntimeError("LoRA is not enabled.")
1389 + return self.lora_manager.pin_adapter(lora_id)
1390 +
1391 + def list_loras(self) -> Set[int]:
1392 + if not self.lora_manager:
1393 + raise RuntimeError("LoRA is not enabled.")
1394 + return self.lora_manager.list_adapters()
1395 +
1396 + def log_warmup(self, phase, i, max_i, batch_size, seq_len):
1397 + free_mem = format_bytes(
1398 + HabanaMemoryProfiler.current_free_device_memory())
1399 + dim = "num_blocks"
1400 + if phase == "Prompt":
1401 + dim = "seq_len"
1402 + msg = (f"[Warmup][{phase}][{i+1}/{max_i}] "
1403 + f"batch_size:{batch_size} "
1404 + f"{dim}:{seq_len} "
1405 + f"free_mem:{free_mem}")
1406 + logger.info(msg)
1407 +
1408 + def warmup_all_buckets(self, buckets, is_prompt, kv_caches):
1409 + for i, (batch_size, seq_len) in enumerate(reversed(buckets)):
1410 + self.log_warmup('Prompt' if is_prompt else 'Decode', i,
1411 + len(buckets), batch_size, seq_len)
1412 + self.warmup_scenario(batch_size, seq_len, is_prompt, kv_caches)
1413 +
1414 + def warmup_graphs(self,
1415 + strategy,
1416 + buckets,
1417 + is_prompt,
1418 + kv_caches,
1419 + available_mem,
1420 + starting_mem=0,
1421 + total_batch_seq=0.001):
1422 + total_mem = starting_mem
1423 + idx = 0
1424 + phase = f'Graph/{"Prompt" if is_prompt else "Decode"}'
1425 + num_candidates = len(buckets)
1426 + ordering : Union[Callable[[Any], Tuple[Any, Any]], \
1427 + Callable[[Any], Tuple[Any, Any, Any]]]
1428 + if strategy == 'min_tokens':
1429 + ordering = lambda b: (b[0] * b[1], b[1], b[0])
1430 + elif strategy == 'max_bs':
1431 + ordering = lambda b: (-b[0], b[1])
1432 + else:
1433 + raise NotImplementedError(
1434 + f'Unsupported graph allocation strategy: {strategy}')
1435 + buckets = list(sorted(buckets, key=ordering))
1436 + captured_all = True
1437 + for idx, (batch_size, seq_len) in enumerate(buckets):
1438 + # Graph memory usage is proportional to seq dimension in a batch
1439 + batch_seq = batch_size * seq_len if is_prompt else batch_size
1440 + mem_estimate = batch_seq / total_batch_seq * total_mem
1441 + if mem_estimate >= available_mem:
1442 + captured_all = False
1443 + continue
1444 + graphed_bucket = (batch_size, seq_len, is_prompt)
1445 + if graphed_bucket in self.graphed_buckets:
1446 + continue
1447 + self.graphed_buckets.add(graphed_bucket)
1448 + self.log_warmup(phase, idx, num_candidates, batch_size, seq_len)
1449 + with HabanaMemoryProfiler() as mem_prof:
1450 + self.warmup_scenario(batch_size, seq_len, is_prompt, kv_caches)
1451 + used_mem = align_workers(mem_prof.consumed_device_memory,
1452 + torch.distributed.ReduceOp.MAX)
1453 + available_mem -= used_mem
1454 + total_mem += used_mem
1455 + total_batch_seq += batch_seq
1456 +
1457 + return total_mem, total_batch_seq, captured_all
1458 +
1459 + def log_graph_warmup_summary(self, buckets, is_prompt, total_mem):
1460 + num_candidates = len(buckets)
1461 + phase = f'Graph/{"Prompt" if is_prompt else "Decode"}'
1462 + graphed = list(c[:2] for c in self.graphed_buckets
1463 + if c[2] == is_prompt)
1464 + if num_candidates == 0:
1465 + num_candidates = 1
1466 + msg = (f'{phase} captured:{len(graphed)} '
1467 + f'({100 * len(graphed) / num_candidates:.1f}%) '
1468 + f'used_mem:{format_bytes(total_mem)} '
1469 + f'buckets:{sorted(list(graphed))}')
1470 + logger.info(msg)
1471 +
1472 + @torch.inference_mode()
1473 + def warmup_model(self, kv_caches: List[torch.Tensor]) -> None:
1474 + if profile := os.environ.get('VLLM_PT_PROFILE', None):
1475 + phase, bs, seq_len, graph = profile.split('_')
1476 + is_prompt = phase == 'prompt'
1477 + graphs = graph == 't'
1478 + if graphs:
1479 + self.graphed_buckets.add((int(bs), int(seq_len), is_prompt))
1480 + self.warmup_scenario(int(bs), int(seq_len), is_prompt, kv_caches,
1481 + True)
1482 + raise AssertionError("Finished profiling")
1483 + if self.skip_warmup:
1484 + logger.info("Skipping warmup...")
1485 + return
1486 + self.profiler.start('internal', 'warmup')
1487 + max_blocks = kv_caches[0][0].size(0)
1488 +
1489 + self.bucketing_global_state.prompt_buckets, prompt_omitted_buckets = \
1490 + generate_prompt_buckets(
1491 + self.bucketing_global_state.prompt_bs_bucket_cfg,
1492 + self.bucketing_global_state.prompt_seq_bucket_cfg,
1493 + self.max_num_batched_tokens)
1494 +
1495 + msg = (f"Generated {len(self.bucketing_global_state.prompt_buckets)} "
1496 + f"prompt buckets [bs, seq]: \
1497 + {list(sorted(self.bucketing_global_state.prompt_buckets))}")
1498 + logger.info(msg)
1499 +
1500 + msg = (f"Omitted {len(prompt_omitted_buckets)} "
1501 + "prompt buckets due to exceeded token budget "
1502 + f"(max_num_batched_tokens={self.max_num_batched_tokens})")
1503 + logger.info(msg)
1504 +
1505 + msg = f"Omitted prompt buckets: {list(sorted(prompt_omitted_buckets))}"
1506 + logger.debug(msg)
1507 +
1508 + self.bucketing_global_state.decode_buckets = generate_decode_buckets(
1509 + self.bucketing_global_state.decode_bs_bucket_cfg,
1510 + self.bucketing_global_state.decode_block_bucket_cfg, max_blocks)
1511 + logger.info("Generated %d decode buckets [bs, total_blocks]: %s",
1512 + len(self.bucketing_global_state.decode_buckets),
1513 + list(sorted(self.bucketing_global_state.decode_buckets)))
1514 +
1515 + if not htorch.utils.internal.is_lazy() and not self.enforce_eager:
1516 + cache_size_limit = len(
1517 + self.bucketing_global_state.prompt_buckets) + len(
1518 + self.bucketing_global_state.decode_buckets) + 1
1519 + torch._dynamo.config.cache_size_limit = max(
1520 + cache_size_limit, torch._dynamo.config.cache_size_limit)
1521 + # Multiply by 8 to follow the original default ratio between
1522 + # the cache_size_limit and accumulated_cache_size_limit
1523 + torch._dynamo.config.accumulated_cache_size_limit = max(
1524 + cache_size_limit * 8,
1525 + torch._dynamo.config.accumulated_cache_size_limit)
1526 +
1527 + start_mem = HabanaMemoryProfiler.current_device_memory_usage()
1528 + start_time = time.perf_counter()
1529 +
1530 + compile_only_mode_context = functools.partial(bc.env_setting,
1531 + "PT_COMPILE_ONLY_MODE",
1532 + True)
1533 + can_use_compile_only_mode = True
1534 + try:
1535 + with compile_only_mode_context():
1536 + pass
1537 + logger.debug("Using PT_COMPILE_ONLY_MODE.")
1538 + except KeyError:
1539 + can_use_compile_only_mode = False
1540 + logger.warning('Cannot use PT_COMPILE_ONLY_MODE. '
1541 + 'Warmup time will be negatively impacted. '
1542 + 'Please update Gaudi Software Suite.')
1543 + with compile_only_mode_context(
1544 + ) if can_use_compile_only_mode else contextlib.nullcontext():
1545 + self.warmup_all_buckets(self.bucketing_global_state.prompt_buckets,
1546 + True, kv_caches)
1547 + self.warmup_all_buckets(self.bucketing_global_state.decode_buckets,
1548 + False, kv_caches)
1549 +
1550 + if not self.enforce_eager and htorch.utils.internal.is_lazy():
1551 + assert self.mem_margin is not None, \
1552 + ("HabanaWorker.determine_num_available_blocks needs "
1553 + "to be called before warming up the model.")
1554 + free_mem = HabanaMemoryProfiler.current_free_device_memory()
1555 + graph_free_mem = free_mem - self.mem_margin
1556 + graph_free_mem = align_workers(graph_free_mem,
1557 + torch.distributed.ReduceOp.MIN)
1558 + prompt_graph_mem_ratio = float(
1559 + os.environ.get('VLLM_GRAPH_PROMPT_RATIO', '0.3'))
1560 + prompt_available_memory = (prompt_graph_mem_ratio *
1561 + graph_free_mem)
1562 + decode_available_memory = (graph_free_mem -
1563 + prompt_available_memory)
1564 + msg = (
1565 + f"Using {format_bytes(graph_free_mem)}"
1566 + f"/{format_bytes(free_mem)} "
1567 + "of free device memory for HPUGraphs, "
1568 + f"{format_bytes(prompt_available_memory)} for prompt and "
1569 + f"{format_bytes(decode_available_memory)} for decode "
1570 + f"(VLLM_GRAPH_PROMPT_RATIO={prompt_graph_mem_ratio})")
1571 + logger.info(msg)
1572 + prompt_strategy = os.environ.get('VLLM_GRAPH_PROMPT_STRATEGY',
1573 + 'min_tokens')
1574 + decode_strategy = os.environ.get('VLLM_GRAPH_DECODE_STRATEGY',
1575 + 'max_bs')
1576 + mem_post_prompt, prompt_batch_seq, prompt_captured_all = \
1577 + self.warmup_graphs(
1578 + prompt_strategy, self.bucketing_global_state.prompt_buckets,
1579 + True, kv_caches, prompt_available_memory)
1580 + mem_post_decode, decode_batch_seq, decode_captured_all = \
1581 + self.warmup_graphs(
1582 + decode_strategy, self.bucketing_global_state.decode_buckets,
1583 + False, kv_caches, decode_available_memory)
1584 +
1585 + # Not all prompt buckets were captured, but all decode buckets
1586 + # were captured and we have some free graph-allocated space
1587 + # left. Let's try to use it for capturing more prompt buckets.
1588 + if (mem_post_decode + mem_post_prompt < graph_free_mem
1589 + and not prompt_captured_all and decode_captured_all):
1590 + mem_post_prompt, _, prompt_captured_all = (
1591 + self.warmup_graphs(
1592 + prompt_strategy,
1593 + self.bucketing_global_state.prompt_buckets, True,
1594 + kv_caches,
1595 + graph_free_mem - mem_post_prompt - mem_post_decode,
1596 + mem_post_prompt, prompt_batch_seq))
1597 +
1598 + # Not all decode buckets were captured, but all prompt buckets
1599 + # were captured and we have some free graph-allocated space
1600 + # left. Let's try to use it for capturing more decode buckets.
1601 + if mem_post_decode + mem_post_prompt < graph_free_mem \
1602 + and not decode_captured_all \
1603 + and prompt_captured_all:
1604 + mem_post_decode, _, _ = self.warmup_graphs(
1605 + decode_strategy,
1606 + self.bucketing_global_state.decode_buckets, False,
1607 + kv_caches,
1608 + graph_free_mem - mem_post_prompt - mem_post_decode,
1609 + mem_post_decode, decode_batch_seq)
1610 +
1611 + self.log_graph_warmup_summary(
1612 + self.bucketing_global_state.prompt_buckets, True,
1613 + mem_post_prompt)
1614 + self.log_graph_warmup_summary(
1615 + self.bucketing_global_state.decode_buckets, False,
1616 + mem_post_decode)
1617 +
1618 + end_time = time.perf_counter()
1619 + end_mem = HabanaMemoryProfiler.current_device_memory_usage()
1620 + elapsed_time = end_time - start_time
1621 + msg = (
1622 + f"Warmup finished in {elapsed_time:.0f} secs, "
1623 + f"allocated {format_bytes(end_mem - start_mem)} of device memory")
1624 + logger.info(msg)
1625 + self.profiler.end()
1626 +
1627 + @property
1628 + def vocab_size(self) -> int:
1629 + return self.model_config.get_vocab_size()
1630 +
1631 + @property
1632 + def mem_margin(self) -> Optional[int]:
1633 + return self._mem_margin
1634 +
1635 + @mem_margin.setter
1636 + def mem_margin(self, value):
1637 + self._mem_margin = value
1638 +
1639 +
1640 +def _maybe_wrap_in_hpu_graph(*args, **kwargs):
1641 + return htorch.hpu.wrap_in_hpu_graph(
1642 + HpuModelAdapter(*args, **kwargs), disable_tensor_cache=True
1643 + ) if htorch.utils.internal.is_lazy() else HpuModelAdapter(*args, **kwargs)
1644 +
1645 +
1646 +class HabanaProfilerCounterHelper():
1647 +
1648 + def __init__(self):
1649 + self.niter = 0
1650 + self.average_real_throughput = None
1651 + self.logged_once = False
1652 + self.real_seq_lens = []
1653 + self.prompt_seq_lens = []
1654 +
1655 + def capture_seq_group_metadata_stats(self, seq_group_metadata_list):
1656 + self.real_seq_lens = [
1657 + len(seq_data.prompt_token_ids) + len(seq_data.output_token_ids)
1658 + for seq_group_metadata in seq_group_metadata_list
1659 + for seq_data in seq_group_metadata.seq_data.values()
1660 + ]
1661 + self.prompt_seq_lens = [
1662 + len(seq_data.prompt_token_ids)
1663 + for seq_group_metadata in seq_group_metadata_list
1664 + for seq_data in seq_group_metadata.seq_data.values()
1665 + ]
1666 +
1667 + def get_counter_dict(self, cache_config, duration, seq_len,
1668 + batch_size_padded, real_batch_size, is_prompt):
1669 + throughput = batch_size_padded / (duration / 1e6)
1670 + throughput_effective = real_batch_size / (duration / 1e6)
1671 +
1672 + real_max_seq_len = max(self.real_seq_lens)
1673 + real_num_tokens = sum(self.real_seq_lens)
1674 + padded_num_tokens = batch_size_padded * seq_len
1675 + batch_token_utilization = real_num_tokens / padded_num_tokens
1676 + if self.average_real_throughput is None:
1677 + self.average_real_throughput = throughput_effective
1678 + else: # https://www.heikohoffmann.de/htmlthesis/node134.html
1679 + self.average_real_throughput = self.average_real_throughput + 1 / (
1680 + self.niter + 1) * (throughput_effective -
1681 + self.average_real_throughput)
1682 + phase = "prompt" if is_prompt else "decode"
1683 + counters = {
1684 + f'{phase}_bucket_batch_size': batch_size_padded,
1685 + f'{phase}_batch_size': real_batch_size,
1686 + f'{phase}_bucket_seq_len': seq_len,
1687 + f'{phase}_seq_len': real_max_seq_len,
1688 + f'{phase}_bucket_gen_throughput': throughput,
1689 + f'{phase}_real_gen_throughput': throughput_effective,
1690 + f'{phase}_batch_token_utilization': batch_token_utilization,
1691 + 'average_real_throughput': self.average_real_throughput,
1692 + 'engine_iteration': self.niter,
1693 + }
1694 + self.niter += 1
1695 + if is_prompt:
1696 + prompt_bucket_in_throughput = (seq_len * batch_size_padded) / (
1697 + duration / 1e6)
1698 + prompt_real_in_throughput = sum(
1699 + self.prompt_seq_lens) / (duration / 1e6)
1700 + counters[
1701 + f'{phase}_bucket_in_throughput'] = prompt_bucket_in_throughput
1702 + counters[f'{phase}_real_in_throughput'] = prompt_real_in_throughput
1703 +
1704 + # KV cache might not be created yet (e.g. for profiling run)
1705 + if cache_config.num_gpu_blocks is not None and \
1706 + cache_config.num_gpu_blocks != 0:
1707 + cache_num_blocks_used = [
1708 + math.ceil(sl / cache_config.block_size)
1709 + for sl in self.real_seq_lens
1710 + ]
1711 + cache_total_num_blocks_used = sum(cache_num_blocks_used)
1712 + num_cache_blocks = cache_config.num_gpu_blocks
1713 + cache_total_num_free_blocks = \
1714 + num_cache_blocks - cache_total_num_blocks_used
1715 + cache_computed_utilization = \
1716 + cache_total_num_blocks_used / num_cache_blocks
1717 + max_blocks_per_seq = math.ceil(seq_len / cache_config.block_size)
1718 + batch_block_utilization = cache_total_num_blocks_used / (
1719 + batch_size_padded * max_blocks_per_seq)
1720 + counters['cache_num_blocks_used'] = cache_total_num_blocks_used
1721 + counters['cache_num_free_blocks'] = cache_total_num_free_blocks
1722 + counters['cache_computed_utilization'] = cache_computed_utilization
1723 + counters[
1724 + f'{phase}_batch_block_utilization'] = batch_block_utilization
1725 + if not self.logged_once:
1726 + counters['const_cache_num_blocks'] = cache_config.num_gpu_blocks
1727 + counters[
1728 + 'const_gpu_memory_utilization'] = \
1729 + cache_config.gpu_memory_utilization
1730 + counters['const_block_size'] = cache_config.block_size
1731 + self.logged_once = True
1732 + return counters
1733 +
1734 +
1735 +def unwrap_model(model):
1736 + if isinstance(model, torch._dynamo.eval_frame.OptimizedModule):
1737 + return unwrap_model(model._orig_mod)
1738 + else:
1739 + model = list(vars(model)['_modules'].values())[0]
1740 + modules = list(vars(model)['_modules'].values())
1741 + return modules
1742 +
1743 +
1744 +class HPUModelRunner(HPUModelRunnerBase[ModelInputForHPUWithSamplingMetadata]):
1745 + """
1746 + GPU model runner with sampling step.
1747 + """
1748 + _model_input_cls: Type[ModelInputForHPUWithSamplingMetadata] = (
1749 + ModelInputForHPUWithSamplingMetadata)
1750 +
1751 + def make_model_input_from_broadcasted_tensor_dict(
1752 + self,
1753 + tensor_dict: Dict[str, Any],
1754 + ) -> ModelInputForHPUWithSamplingMetadata:
1755 + return (
1756 + ModelInputForHPUWithSamplingMetadata.from_broadcasted_tensor_dict(
1757 + tensor_dict,
1758 + attn_backend=self.attn_backend,
1759 + ))
1760 +
1761 + @torch.inference_mode()
1762 + def prepare_model_input(
1763 + self,
1764 + seq_group_metadata_list: List[SequenceGroupMetadata],
1765 + virtual_engine: int = 0,
1766 + finished_requests_ids: Optional[List[str]] = None
1767 + ) -> ModelInputForHPUWithSamplingMetadata:
1768 + """Prepare the model input based on a given sequence group, including
1769 + metadata for the sampling step.
1770 + The API assumes seq_group_metadata_list is sorted by prefill -> decode.
1771 + The result tensors and data structure also batches input in prefill
1772 + -> decode order. For example,
1773 + - input_tokens[:num_prefill_tokens] contains prefill tokens.
1774 + - input_tokens[num_prefill_tokens:] contains decode tokens.
1775 + If cuda graph is required, this API automatically pads inputs.
1776 + """
1777 + with self.profiler.record_event('internal', 'prepare_input_tensors'):
1778 + assert seq_group_metadata_list is not None
1779 + if self.profiler.enabled:
1780 + self.profiler_counter_helper.capture_seq_group_metadata_stats(
1781 + seq_group_metadata_list=seq_group_metadata_list)
1782 + model_input, sampling_metadata = self.prepare_input_tensors(
1783 + seq_group_metadata_list)
1784 + assert model_input.attn_metadata is not None
1785 + is_prompt = model_input.attn_metadata.is_prompt
1786 +
1787 + return dataclasses.replace(model_input,
1788 + sampling_metadata=sampling_metadata,
1789 + is_prompt=is_prompt,
1790 + virtual_engine=virtual_engine)
1791 +
1792 + def finish_measurements(self):
1793 + from neural_compressor.torch.quantization import finalize_calibration
1794 + finalize_calibration(self.model.model)
1795 +
1796 + def _check_config(self, batch_size, seq_len, is_prompt, warmup_mode):
1797 + cfg = (batch_size, seq_len, is_prompt)
1798 + seen = cfg in self.seen_configs
1799 + self.seen_configs.add(cfg)
1800 + if not seen and not warmup_mode:
1801 + phase = 'prompt' if is_prompt else 'decode'
1802 + logger.warning("Configuration: (%s, %s, %s) was not warmed-up!",
1803 + phase, batch_size, seq_len)
1804 +
1805 + def create_lora_mask(self, input_tokens: torch.Tensor, lora_ids: List[int],
1806 + is_prompt: bool):
1807 + '''
1808 + This is a helper function to create the mask for lora computations.
1809 + Lora Mask is needed to ensure we match the correct lora weights for the
1810 + for the request.
1811 + For Prompt phase we have
1812 + lora_mask with shape (batch_size * seq_len, max_loras * max_rank)
1813 + lora_logits_mask with shape (batch_size, max_loras * max_rank)
1814 + For Decode phase we have both
1815 + lora_mask and lora_logits_mask with shape
1816 + (batch_size, max_loras * max_rank)
1817 + '''
1818 + lora_mask: torch.Tensor = None
1819 + lora_logits_mask: torch.Tensor = None
1820 + lora_index = 0
1821 +
1822 + if self.lora_config:
1823 + if is_prompt:
1824 + lora_mask = torch.zeros(
1825 + input_tokens.shape[0] * input_tokens.shape[1],
1826 + (self.lora_config.max_loras) *\
1827 + self.lora_config.max_lora_rank,
1828 + dtype=self.lora_config.lora_dtype)
1829 + lora_logits_mask = torch.zeros(
1830 + input_tokens.shape[0], (self.lora_config.max_loras) *
1831 + self.lora_config.max_lora_rank,
1832 + dtype=self.lora_config.lora_dtype)
1833 +
1834 + ones = torch.ones(input_tokens.shape[1],
1835 + self.lora_config.max_lora_rank,
1836 + dtype=self.lora_config.lora_dtype)
1837 + logit_ones = torch.ones(1,
1838 + self.lora_config.max_lora_rank,
1839 + dtype=self.lora_config.lora_dtype)
1840 +
1841 + for i in range(len(lora_ids)):
1842 + if lora_ids[i] == 0:
1843 + continue
1844 + lora_index = self.lora_manager._adapter_manager.\
1845 + lora_index_to_id.index(lora_ids[i])
1846 + start_row = i * input_tokens.shape[1]
1847 + end_row = start_row + input_tokens.shape[1]
1848 + start_col = lora_index * self.lora_config.max_lora_rank
1849 + end_col = start_col + self.lora_config.max_lora_rank
1850 + lora_mask[start_row:end_row, start_col:end_col] = ones
1851 + lora_logits_mask[i, start_col:end_col] = logit_ones
1852 + lora_mask = lora_mask.to('hpu')
1853 + lora_logits_mask = lora_logits_mask.to('hpu')
1854 + else:
1855 + lora_mask = torch.zeros(input_tokens.shape[0],
1856 + (self.lora_config.max_loras) *
1857 + self.lora_config.max_lora_rank,
1858 + dtype=self.lora_config.lora_dtype)
1859 + ones = torch.ones(1,
1860 + self.lora_config.max_lora_rank,
1861 + dtype=self.lora_config.lora_dtype)
1862 + for i in range(len(lora_ids)):
1863 + if lora_ids[i] == 0:
1864 + continue
1865 + lora_index = self.lora_manager._adapter_manager.\
1866 + lora_index_to_id.index(lora_ids[i])
1867 + start_pos = lora_index * self.lora_config.max_lora_rank
1868 + end_pos = start_pos + self.lora_config.max_lora_rank
1869 + lora_mask[i, start_pos:end_pos] = ones
1870 + lora_mask = lora_mask.to('hpu')
1871 + lora_logits_mask = lora_mask
1872 +
1873 + return lora_mask, lora_logits_mask
1874 +
1875 + @torch.inference_mode()
1876 + def execute_model(
1877 + self,
1878 + model_input: ModelInputForHPUWithSamplingMetadata,
1879 + kv_caches: List[torch.Tensor],
1880 + intermediate_tensors: Optional[IntermediateTensors] = None,
1881 + num_steps: int = 1,
1882 + warmup_mode=False,
1883 + ) -> Optional[Union[List[SamplerOutput], IntermediateTensors]]:
1884 + if num_steps > 1:
1885 + raise ValueError(
1886 + "num_steps > 1 is not supported in HPUModelRunner")
1887 +
1888 + if self.lora_config:
1889 + assert model_input.lora_requests is not None
1890 + assert model_input.lora_mapping is not None
1891 + self.set_active_loras(model_input.lora_requests,
1892 + model_input.lora_mapping)
1893 + input_tokens = model_input.input_tokens
1894 + input_positions = model_input.input_positions
1895 + attn_metadata = model_input.attn_metadata
1896 + sampling_metadata = model_input.sampling_metadata
1897 + real_batch_size = model_input.real_batch_size
1898 + batch_size_padded = model_input.batch_size_padded
1899 + assert input_tokens is not None
1900 + assert input_positions is not None
1901 + assert sampling_metadata is not None
1902 + assert attn_metadata is not None
1903 + is_prompt = attn_metadata.is_prompt
1904 + assert is_prompt is not None
1905 + batch_size = input_tokens.size(0)
1906 + seq_len = self._seq_len(attn_metadata)
1907 + use_graphs = self._use_graphs(batch_size, seq_len, is_prompt)
1908 + self._check_config(batch_size, seq_len, is_prompt, warmup_mode)
1909 +
1910 + lora_mask: torch.Tensor = None
1911 + lora_logits_mask: torch.Tensor = None
1912 + if self.lora_config:
1913 + assert model_input.lora_ids is not None
1914 + lora_mask, lora_logits_mask = self.create_lora_mask(
1915 + input_tokens, model_input.lora_ids, attn_metadata.is_prompt)
1916 +
1917 + execute_model_kwargs = {
1918 + "input_ids": input_tokens,
1919 + "positions": input_positions,
1920 + "kv_caches": kv_caches,
1921 + "attn_metadata": self.trim_attn_metadata(attn_metadata),
1922 + "intermediate_tensors": intermediate_tensors,
1923 + "lora_mask": lora_mask,
1924 + **(model_input.multi_modal_kwargs or {}),
1925 + }
1926 + if htorch.utils.internal.is_lazy():
1927 + execute_model_kwargs.update({"bypass_hpu_graphs": not use_graphs})
1928 +
1929 + htorch.core.mark_step()
1930 + if self.is_driver_worker:
1931 + model_event_name = ("model_"
1932 + f"{'prompt' if is_prompt else 'decode'}_"
1933 + f"bs{batch_size}_"
1934 + f"seq{seq_len}_"
1935 + f"graphs{'T' if use_graphs else 'F'}")
1936 + else:
1937 + model_event_name = 'model_executable'
1938 + with self.profiler.record_event('internal', model_event_name):
1939 + hidden_states = self.model.forward(
1940 + **execute_model_kwargs,
1941 + selected_token_indices=sampling_metadata.selected_token_indices
1942 + )
1943 +
1944 + if self.lora_config:
1945 + LoraMask.setLoraMask(
1946 + lora_logits_mask.index_select(
1947 + 0, sampling_metadata.selected_token_indices))
1948 +
1949 + # Compute the logits.
1950 + with self.profiler.record_event(
1951 + 'internal', ('compute_logits_'
1952 + f'{"prompt" if is_prompt else "decode"}_bs'
1953 + f'{batch_size}_'
1954 + f'seq{seq_len}')):
1955 + sampling_metadata.selected_token_indices = None
1956 + logits = self.model.compute_logits(hidden_states,
1957 + sampling_metadata)
1958 + htorch.core.mark_step()
1959 + # Only perform sampling in the driver worker.
1960 + if not self.is_driver_worker:
1961 + return []
1962 +
1963 + if model_input.async_callback is not None:
1964 + model_input.async_callback()
1965 +
1966 + # Sample the next token.
1967 + with self.profiler.record_event(
1968 + 'internal', ('sample_'
1969 + f'{"prompt" if is_prompt else "decode"}_'
1970 + f'bs{batch_size}_'
1971 + f'seq{seq_len}')):
1972 + output = self.model.sample(
1973 + logits=logits,
1974 + sampling_metadata=sampling_metadata,
1975 + )
1976 + output.outputs = output.outputs[:real_batch_size]
1977 + htorch.core.mark_step()
1978 +
1979 + if self.is_driver_worker and self.profiler.enabled:
1980 + # Stop recording 'execute_model' event
1981 + self.profiler.end()
1982 + event_end = self.profiler.get_timestamp_us()
1983 + counters = self.profiler_counter_helper.get_counter_dict(
1984 + cache_config=self.cache_config,
1985 + duration=event_end - self.event_start,
1986 + seq_len=seq_len,
1987 + batch_size_padded=batch_size_padded,
1988 + real_batch_size=real_batch_size,
1989 + is_prompt=is_prompt)
1990 + self.profiler.record_counter(self.event_start, counters)
1991 + return [output]
1992 +
1993 + def shutdown_inc(self):
1994 + can_finalize_inc = False
1995 + from contextlib import suppress
1996 + with suppress(AttributeError):
1997 + can_finalize_inc = (self.model_config.quantization == 'inc') and \
1998 + (self.model.model is not None) and \
1999 + self.inc_initialized_successfully and \
2000 + not getattr(self, "_is_inc_finalized", False)
2001 + if can_finalize_inc:
2002 + from neural_compressor.torch.quantization import (
2003 + finalize_calibration)
2004 + finalize_calibration(self.model.model)
2005 + self._is_inc_finalized = True
2006 +
2007 + def __del__(self):
2008 + self.shutdown_inc()
vllm/worker/hpu_worker.py new
+410
@@ -0,0 +1,410 @@
1 +###############################################################################
2 +# Copyright (C) 2024 Habana Labs, Ltd. an Intel Company
3 +###############################################################################
4 +
5 +import gc
6 +import os
7 +from typing import List, Optional, Set, Tuple, Type
8 +
9 +import habana_frameworks.torch as htorch # noqa:F401
10 +import torch
11 +import torch.distributed
12 +from vllm_hpu_extension.profiler import HabanaMemoryProfiler, format_bytes
13 +
14 +import vllm.envs as envs
15 +from vllm.config import ParallelConfig, VllmConfig
16 +from vllm.distributed import (ensure_model_parallel_initialized,
17 + init_distributed_environment)
18 +from vllm.logger import init_logger
19 +from vllm.lora.request import LoRARequest
20 +from vllm.model_executor import set_random_seed
21 +from vllm.prompt_adapter.request import PromptAdapterRequest
22 +from vllm.sequence import ExecuteModelRequest
23 +from vllm.worker.cache_engine import CacheEngine
24 +from vllm.worker.hpu_model_runner import HPUModelRunner
25 +from vllm.worker.model_runner_base import ModelRunnerBase
26 +from vllm.worker.worker_base import (LocalOrDistributedWorkerBase, WorkerBase,
27 + WorkerInput)
28 +
29 +logger = init_logger(__name__)
30 +
31 +
32 +class HPUWorker(LocalOrDistributedWorkerBase):
33 + """A worker class that executes (a partition of) the model on a HPU.
34 +
35 + Each worker is associated with a single HPU. The worker is responsible for
36 + maintaining the KV cache and executing the model on the HPU. In case of
37 + distributed inference, each worker is assigned a partition of the model.
38 + """
39 +
40 + def __init__(
41 + self,
42 + vllm_config: VllmConfig,
43 + local_rank: int,
44 + rank: int,
45 + distributed_init_method: str,
46 + is_driver_worker: bool = False,
47 + model_runner_cls: Optional[Type[ModelRunnerBase]] = None,
48 + ) -> None:
49 + WorkerBase.__init__(self, vllm_config=vllm_config)
50 + self.parallel_config.rank = rank
51 + self.local_rank = local_rank
52 + self.rank = rank
53 + self.distributed_init_method = distributed_init_method
54 + self.is_driver_worker = is_driver_worker
55 + if self.is_driver_worker:
56 + assert self.rank == 0, "The driver worker must have rank 0."
57 +
58 + if self.model_config.trust_remote_code:
59 + # note: lazy import to avoid importing torch before initializing
60 + from vllm.utils import init_cached_hf_modules
61 + init_cached_hf_modules()
62 +
63 + self.model_runner: HPUModelRunner = HPUModelRunner(
64 + vllm_config=vllm_config, is_driver_worker=is_driver_worker)
65 + # Uninitialized cache engine. Will be initialized by
66 + # initialize_cache.
67 + self.cache_engine: List[HPUCacheEngine]
68 + # Initialize gpu_cache as embedding models don't initialize kv_caches
69 + self.hpu_cache: Optional[List[List[torch.tensor]]] = None
70 + # Torch profiler. Enabled and configured through env vars:
71 + # VLLM_TORCH_PROFILER_DIR=/path/to/save/trace
72 + if envs.VLLM_TORCH_PROFILER_DIR:
73 + torch_profiler_trace_dir = envs.VLLM_TORCH_PROFILER_DIR
74 + logger.info("Profiling enabled. Traces will be saved to: %s",
75 + torch_profiler_trace_dir)
76 + self.profiler = torch.profiler.profile(
77 + activities=[
78 + torch.profiler.ProfilerActivity.CPU,
79 + torch.profiler.ProfilerActivity.HPU,
80 + ],
81 + with_stack=True,
82 + on_trace_ready=torch.profiler.tensorboard_trace_handler(
83 + torch_profiler_trace_dir, use_gzip=True))
84 + else:
85 + self.profiler = None
86 +
87 + def start_profile(self):
88 + if self.profiler is None:
89 + raise RuntimeError("Profiler is not enabled.")
90 + self.profiler.start()
91 +
92 + def stop_profile(self):
93 + if self.profiler is None:
94 + raise RuntimeError("Profiler is not enabled.")
95 + self.profiler.stop()
96 +
97 + def _set_env_vars(self):
98 + local_rank = self.local_rank
99 + if self.parallel_config.world_size == 1:
100 + local_rank = -1
101 + import os
102 + os.environ["LOCAL_RANK"] = str(local_rank)
103 + os.environ["ID"] = str(local_rank)
104 + os.environ["WORLD_SIZE"] = str(self.parallel_config.world_size)
105 + os.environ["RANK"] = str(self.rank)
106 +
107 + def init_device(self) -> None:
108 + if self.device_config.device.type == "hpu":
109 + self.device = torch.device("hpu")
110 + torch.hpu.set_device(self.device)
111 + else:
112 + raise RuntimeError(
113 + f"Not support device type: {self.device_config.device}")
114 + # Initialize the distributed environment.
115 + if self.model_config.quantization == 'inc':
116 + self._set_env_vars()
117 + init_worker_distributed_environment(self.parallel_config, self.rank,
118 + self.distributed_init_method,
119 + self.local_rank)
120 + # Set random seed.
121 + set_random_seed(self.model_config.seed)
122 +
123 + def load_model(self):
124 + self.model_runner.load_model()
125 +
126 + @torch.inference_mode()
127 + def determine_num_available_blocks(self) -> Tuple[int, int]:
128 + """Profiles the peak memory usage of the model to determine how many
129 + KV blocks may be allocated without OOMs.
130 +
131 + The engine will first conduct a profiling of the existing memory usage.
132 + Then, it calculate the maximum possible number of GPU and CPU blocks
133 + that can be allocated with the remaining free memory.
134 +
135 + .. tip::
136 + You may limit the usage of GPU memory
137 + by adjusting the `gpu_memory_utilization` parameter.
138 + """
139 + # Profile the memory usage of the model and get the maximum number of
140 + # cache blocks that can be allocated with the remaining free memory.
141 +
142 + # Execute a forward pass with dummy inputs to profile the memory usage
143 + # of the model.
144 + with HabanaMemoryProfiler() as m:
145 + self.model_runner.profile_run()
146 + torch.hpu.synchronize()
147 + msg = ("Model profiling run "
148 + f"took {m.get_summary_string()}")
149 + logger.info(msg)
150 + # At this point we should've allocated the maximum workspace for all
151 + # recipes we will use the extra memory for graphs/blocks
152 + free_hpu_memory = torch.hpu.mem_get_info()[0]
153 +
154 + cache_block_size = self.get_cache_block_size_bytes()
155 + graph_reserved_mem = (float(
156 + os.environ.get('VLLM_GRAPH_RESERVED_MEM', '0.1'))
157 + if not self.model_config.enforce_eager else 0)
158 + graph_headroom = 1 - graph_reserved_mem
159 + available_hpu_memory = free_hpu_memory * \
160 + self.cache_config.gpu_memory_utilization
161 + hpu_memory_margin = free_hpu_memory * (
162 + 1 - self.cache_config.gpu_memory_utilization)
163 + self.model_runner.mem_margin = hpu_memory_margin
164 + cache_size_bytes = available_hpu_memory * graph_headroom
165 + graph_headroom_bytes = available_hpu_memory * (1 - graph_headroom)
166 + msg = (
167 + f"Free device memory: {format_bytes(free_hpu_memory)}, "
168 + f"{format_bytes(available_hpu_memory)} usable "
169 + f"(gpu_memory_utilization={self.cache_config.gpu_memory_utilization}),"
170 + f" {format_bytes(graph_headroom_bytes)} reserved for HPUGraphs "
171 + f"(VLLM_GRAPH_RESERVED_MEM={graph_reserved_mem}), "
172 + f"{format_bytes(cache_size_bytes)} reserved for KV cache")
173 + logger.info(msg)
174 + num_hpu_blocks = int(cache_size_bytes // cache_block_size)
175 + num_cpu_blocks = int(self.cache_config.swap_space_bytes //
176 + cache_block_size)
177 + num_hpu_blocks = max(num_hpu_blocks, 0)
178 + num_cpu_blocks = max(num_cpu_blocks, 0)
179 +
180 + if self.model_runner.lora_manager:
181 + self.model_runner.remove_all_loras()
182 +
183 + gc.collect()
184 + return num_hpu_blocks, num_cpu_blocks
185 +
186 + def initialize_cache(self, num_gpu_blocks: int,
187 + num_cpu_blocks: int) -> None:
188 + """Allocate GPU and CPU KV cache with the specified number of blocks.
189 +
190 + This also warms up the model, which may record CUDA graphs.
191 + """
192 + raise_if_cache_size_invalid(num_gpu_blocks,
193 + self.cache_config.block_size,
194 + self.model_config.max_model_len)
195 +
196 + self.cache_config.num_gpu_blocks = num_gpu_blocks
197 + self.cache_config.num_cpu_blocks = num_cpu_blocks
198 +
199 + with HabanaMemoryProfiler() as m:
200 + self._init_cache_engine()
201 + torch.hpu.synchronize()
202 + msg = ("Initializing cache engine "
203 + f"took {m.get_summary_string()}")
204 + logger.info(msg)
205 + self._warm_up_model()
206 +
207 + def _init_cache_engine(self):
208 + assert self.cache_config.num_gpu_blocks is not None
209 + self.cache_engine = [
210 + HPUCacheEngine(self.cache_config, self.model_config,
211 + self.parallel_config, self.device_config)
212 + for _ in range(self.parallel_config.pipeline_parallel_size)
213 + ]
214 + self.hpu_cache = [
215 + self.cache_engine[ve].gpu_cache
216 + for ve in range(self.parallel_config.pipeline_parallel_size)
217 + ]
218 +
219 + def _warm_up_model(self) -> None:
220 + # NOTE(kzawora): We should use virtual engine index here
221 + # for pipeline parallelism. Using 0 for now.
222 + assert self.hpu_cache is not None
223 + self.model_runner.warmup_model(self.hpu_cache[0])
224 + # Reset the seed to ensure that the random state is not affected by
225 + # the model initialization and profiling.
226 + set_random_seed(self.model_config.seed)
227 +
228 + def finish_measurements(self):
229 + self.model_runner.finish_measurements()
230 +
231 + @property
232 + def do_metadata_broadcast(self) -> bool:
233 + return self.parallel_config.tensor_parallel_size > 1
234 +
235 + @property
236 + def kv_cache(self) -> Optional[List[List[torch.Tensor]]]:
237 + return self.hpu_cache
238 +
239 + @torch.inference_mode()
240 + def prepare_worker_input(
241 + self, execute_model_req: ExecuteModelRequest) -> WorkerInput:
242 + virtual_engine = execute_model_req.virtual_engine
243 + num_seq_groups = len(execute_model_req.seq_group_metadata_list)
244 + # `blocks_to_swap_in` and `blocks_to_swap_out` are cpu tensors.
245 + # they contain parameters to launch cudamemcpyasync.
246 + blocks_to_swap_in = torch.tensor(execute_model_req.blocks_to_swap_in,
247 + device="cpu",
248 + dtype=torch.int64).view(-1, 2)
249 + blocks_to_swap_out = torch.tensor(execute_model_req.blocks_to_swap_out,
250 + device="cpu",
251 + dtype=torch.int64).view(-1, 2)
252 + # `blocks_to_copy` is a gpu tensor. The src and tgt of
253 + # blocks to copy are in the same device, and `blocks_to_copy`
254 + # can be used directly within cuda kernels.
255 + blocks_to_copy = torch.tensor(execute_model_req.blocks_to_copy,
256 + device=self.device,
257 + dtype=torch.int64).view(-1, 2)
258 +
259 + return WorkerInput(
260 + num_seq_groups=num_seq_groups,
261 + blocks_to_swap_in=blocks_to_swap_in,
262 + blocks_to_swap_out=blocks_to_swap_out,
263 + blocks_to_copy=blocks_to_copy,
264 + virtual_engine=virtual_engine,
265 + )
266 +
267 + @torch.inference_mode()
268 + def execute_worker(self, worker_input: WorkerInput) -> None:
269 + virtual_engine = worker_input.virtual_engine
270 + # Issue cache operations.
271 + if (worker_input.blocks_to_swap_in is not None
272 + and worker_input.blocks_to_swap_in.numel() > 0):
273 + self.cache_engine[virtual_engine].swap_in(
274 + worker_input.blocks_to_swap_in)
275 + if (worker_input.blocks_to_swap_out is not None
276 + and worker_input.blocks_to_swap_out.numel() > 0):
277 + self.cache_engine[virtual_engine].swap_out(
278 + worker_input.blocks_to_swap_out)
279 + if (worker_input.blocks_to_copy is not None
280 + and worker_input.blocks_to_copy.numel() > 0):
281 + self.cache_engine[virtual_engine].copy(worker_input.blocks_to_copy)
282 +
283 + def add_lora(self, lora_request: LoRARequest) -> bool:
284 + return self.model_runner.add_lora(lora_request)
285 +
286 + def remove_lora(self, lora_id: int) -> bool:
287 + return self.model_runner.remove_lora(lora_id)
288 +
289 + def pin_lora(self, lora_id: int) -> bool:
290 + return self.model_runner.pin_lora(lora_id)
291 +
292 + def list_loras(self) -> Set[int]:
293 + return self.model_runner.list_loras()
294 +
295 + def add_prompt_adapter(
296 + self, prompt_adapter_request: PromptAdapterRequest) -> bool:
297 + raise NotImplementedError(
298 + "Prompt Adapter is not implemented for HPU backend.")
299 +
300 + def remove_prompt_adapter(self, prompt_adapter_id: int) -> bool:
301 + raise NotImplementedError(
302 + "Prompt Adapter is not implemented for HPU backend.")
303 +
304 + def pin_prompt_adapter(self, prompt_adapter_id: int) -> bool:
305 + raise NotImplementedError(
306 + "Prompt Adapter is not implemented for HPU backend.")
307 +
308 + def list_prompt_adapters(self) -> Set[int]:
309 + raise NotImplementedError(
310 + "Prompt Adapter is not implemented for HPU backend.")
311 +
312 + def shutdown_inc(self):
313 + self.model_runner.shutdown_inc()
314 +
315 + @property
316 + def max_model_len(self) -> int:
317 + return self.model_config.max_model_len
318 +
319 + @property
320 + def vocab_size(self) -> int:
321 + return self.model_runner.vocab_size
322 +
323 + def get_cache_block_size_bytes(self) -> int:
324 + """Get the size of the KV cache block size in bytes.
325 + """
326 + return HPUCacheEngine.get_cache_block_size(self.cache_config,
327 + self.model_config,
328 + self.parallel_config)
329 +
330 +
331 +def init_worker_distributed_environment(
332 + parallel_config: ParallelConfig,
333 + rank: int,
334 + distributed_init_method: Optional[str] = None,
335 + local_rank: int = -1,
336 +) -> None:
337 + """Initialize the distributed environment."""
338 + init_distributed_environment(parallel_config.world_size,
339 + rank,
340 + distributed_init_method,
341 + local_rank,
342 + backend='hccl')
343 +
344 + ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
345 + parallel_config.pipeline_parallel_size)
346 +
347 + if torch.distributed.is_initialized():
348 + torch_world_size = torch.distributed.get_world_size()
349 + if torch_world_size != parallel_config.world_size:
350 + raise RuntimeError(
351 + "torch.distributed is already initialized but the torch world "
352 + "size does not match parallel_config.world_size "
353 + f"({torch_world_size} vs. {parallel_config.world_size}).")
354 + elif not distributed_init_method:
355 + raise ValueError(
356 + "distributed_init_method must be set if torch.distributed "
357 + "is not already initialized")
358 + else:
359 + torch.distributed.init_process_group(
360 + backend="hccl",
361 + world_size=parallel_config.world_size,
362 + rank=rank,
363 + init_method=distributed_init_method,
364 + )
365 +
366 + # A small all_reduce for warmup & checking conformance.
367 + dummy_tensor_hpu = torch.ones(1).to('hpu')
368 + torch.distributed.all_reduce(dummy_tensor_hpu)
369 + assert dummy_tensor_hpu.item() == parallel_config.world_size
370 + ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
371 + parallel_config.pipeline_parallel_size)
372 +
373 +
374 +def raise_if_cache_size_invalid(num_gpu_blocks, block_size,
375 + max_model_len) -> None:
376 + if num_gpu_blocks <= 0:
377 + raise ValueError("No available memory for the cache blocks. "
378 + "Try increasing `gpu_memory_utilization` when "
379 + "initializing the engine.")
380 + max_seq_len = block_size * num_gpu_blocks
381 + if max_model_len > max_seq_len:
382 + raise ValueError(
383 + f"The model's max seq len ({max_model_len}) "
384 + "is larger than the maximum number of tokens that can be "
385 + f"stored in KV cache ({max_seq_len}). Try increasing "
386 + "`gpu_memory_utilization` or decreasing `max_model_len` when "
387 + "initializing the engine.")
388 +
389 +
390 +class HPUCacheEngine(CacheEngine):
391 +
392 + def _allocate_kv_cache(
393 + self,
394 + num_blocks: int,
395 + device: str,
396 + ) -> List[Tuple[torch.Tensor, torch.Tensor]]:
397 + """Allocates KV cache on the specified device."""
398 + kv_cache_shape = self.attn_backend.get_kv_cache_shape(
399 + num_blocks, self.block_size, self.num_kv_heads, self.head_size)
400 + kv_cache: List[Tuple[torch.Tensor, torch.Tensor]] = []
401 + for _ in range(self.num_attention_layers):
402 + key_cache = torch.zeros(kv_cache_shape,
403 + dtype=self.dtype,
404 + device=device)
405 + value_cache = torch.zeros(kv_cache_shape,
406 + dtype=self.dtype,
407 + device=device)
408 + kv_layer = (key_cache, value_cache)
409 + kv_cache.append(kv_layer)
410 + return kv_cache