@tduncan / Nvidia-Optimization-Stack-VLLM / commits / d53b1c2efc

Remove native Hunyuan V1 and VL implementations (#53272)

Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com> Co-authored-by: OpenAI Codex <noreply@openai.com>

Tiezhen WANG committed Aug 22, 2026 at 00:07 UTC d53b1c2efc0ca7161c3844f51ca9acbcdb7129d5
7 files changed +9 -2382
docs/models/supported_models.md
+3 -3
@@ -409,8 +409,6 @@ th {
409 | `GraniteSWAForCausalLM` | Granite SWA | `ibm-granite/granite-swash-2b` | ✅︎ | ✅︎ |
410 | `GritLM` | GritLM | `parasail-ai/GritLM-7B-vllm`. | ✅︎ | ✅︎ |
411 | `HrmTextForCausalLM` | HRM-Text | `sapientinc/HRM-Text-1B`, etc. | | |
412 -| `HunYuanDenseV1ForCausalLM` | Hunyuan Dense | `tencent/Hunyuan-7B-Instruct` | ✅︎ | ✅︎ |
413 -| `HunYuanMoEV1ForCausalLM` | Hunyuan-A13B | `tencent/Hunyuan-A13B-Instruct`, `tencent/Hunyuan-A13B-Pretrain`, `tencent/Hunyuan-A13B-Instruct-FP8`, etc. | ✅︎ | ✅︎ |
412 | `HYV3ForCausalLM` | HY3 | `tencent/Hy3-preview-Base`, `tencent/Hy3-preview` | ✅︎ | ✅︎ |
413 | `HyperCLOVAXForCausalLM` | HyperCLOVAX-SEED-Think-14B | `naver-hyperclovax/HyperCLOVAX-SEED-Think-14B` | ✅︎ | ✅︎ |
414 | `InternLM2ForCausalLM` | InternLM2 | `internlm/internlm2-7b`, `internlm/internlm2-chat-7b`, etc. | ✅︎ | ✅︎ |
@@ -478,6 +476,8 @@ Some models are supported only via the [Transformers modeling backend](#transfor
476 | Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/parallelism_scaling.md) |
477 | ------------ | ------ | ----------------- | -------------------- | ------------------------- |
478 | `GPTBigCodeForCausalLM` | StarCoder, SantaCoder, WizardCoder | `bigcode/starcoder`, `bigcode/gpt_bigcode-santacoder`, `WizardLM/WizardCoder-15B-V1.0`, etc. | ✅︎ | |
479 +| `HunYuanDenseV1ForCausalLM` | Hunyuan Dense | `tencent/Hunyuan-7B-Instruct` | ✅︎ | ✅︎ |
480 +| `HunYuanMoEV1ForCausalLM` | Hunyuan-A13B | `tencent/Hunyuan-A13B-Instruct`, `tencent/Hunyuan-A13B-Pretrain`, `tencent/Hunyuan-A13B-Instruct-FP8`, etc. | ✅︎ | ✅︎ |
481 | `OlmoForCausalLM` | OLMo | `allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc. | ✅︎ | ✅︎ |
482 | `Olmo2ForCausalLM` | OLMo2 | `allenai/OLMo-2-0425-1B`, etc. | ✅︎ | ✅︎ |
483 | `SmolLM3ForCausalLM` | SmolLM3 | `HuggingFaceTB/SmolLM3-3B` | ✅︎ | ✅︎ |
@@ -552,7 +552,6 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
552 | `HCXVisionForCausalLM` | HyperCLOVAX-SEED-Vision-Instruct-3B | T + I<sup>+</sup> + V<sup>+</sup> | `naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B` | | |
553 | `HCXVisionV2ForCausalLM` | HyperCLOVAX-SEED-Think-32B | T + I<sup>+</sup> + V<sup>+</sup> | `naver-hyperclovax/HyperCLOVAX-SEED-Think-32B` | | |
554 | `H2OVLChatModel` | H2OVL | T + I<sup>E+</sup> | `h2oai/h2ovl-mississippi-800m`, `h2oai/h2ovl-mississippi-2b`, etc. | ✅︎ | ✅︎ |
555 -| `HunYuanVLForConditionalGeneration` | HunyuanOCR | T + I<sup>E+</sup> | `tencent/HunyuanOCR`, etc. | ✅︎ | ✅︎ |
555 | `Idefics3ForConditionalGeneration` | Idefics3 | T + I | `HuggingFaceM4/Idefics3-8B-Llama3`, etc. | ✅︎ | |
556 | `IsaacForConditionalGeneration` | Isaac | T + I<sup>+</sup> | `PerceptronAI/Isaac-0.1` | ✅︎ | ✅︎ |
557 | `InternS1ForConditionalGeneration` | Intern-S1 | T + I<sup>E+</sup> + V<sup>E+</sup> | `internlm/Intern-S1`, `internlm/Intern-S1-mini`, etc. | ✅︎ | ✅︎ |
@@ -629,6 +628,7 @@ Some models are supported only via the [Transformers modeling backend](#transfor
628 | Architecture | Models | Inputs | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/parallelism_scaling.md) |
629 | ------------ | ------ | ------ | ----------------- | --------------------------- | --------------------------------------- |
630 | `Emu3ForConditionalGeneration` | Emu3 | T + I<sup>+</sup> | `BAAI/Emu3-Chat-hf` | ✅︎ | ✅︎ |
631 +| `HunYuanVLForConditionalGeneration` | HunyuanOCR | T + I<sup>E+</sup> | `tencent/HunyuanOCR`, etc. | ✅︎ | ✅︎ |
632 | `VibeVoiceAsrForConditionalGeneration` | VibeVoice-ASR | T + A<sup>+</sup> | `microsoft/VibeVoice-ASR-HF` | ✅︎ | ✅︎ |
633
634 <sup>^</sup> You need to set the architecture name via `--hf-overrides` to match the one in vLLM.</br>
vllm/model_executor/models/hunyuan_v1.py deleted
-1038
@@ -1,1038 +0,0 @@
1 -# SPDX-License-Identifier: Apache-2.0
2 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 -
4 -# coding=utf-8
5 -# Copyright 2024 The HunYuan team.
6 -# Copyright 2023 The vLLM team.
7 -# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
8 -#
9 -# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
10 -# and OPT implementations in this library. It has been modified from its
11 -# original forms to accommodate minor architectural differences compared
12 -# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
13 -#
14 -# Licensed under the Apache License, Version 2.0 (the "License");
15 -# you may not use this file except in compliance with the License.
16 -# You may obtain a copy of the License at
17 -#
18 -# http://www.apache.org/licenses/LICENSE-2.0
19 -#
20 -# Unless required by applicable law or agreed to in writing, software
21 -# distributed under the License is distributed on an "AS IS" BASIS,
22 -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
23 -# See the License for the specific language governing permissions and
24 -# limitations under the License.
25 -"""Inference-only HunYuan model compatible with HuggingFace weights."""
26 -
27 -import typing
28 -from collections.abc import Callable, Iterable
29 -from itertools import islice
30 -
31 -import regex as re
32 -import torch
33 -from torch import nn
34 -from transformers import PretrainedConfig
35 -
36 -from vllm.compilation.decorators import support_torch_compile
37 -from vllm.config import CacheConfig, VllmConfig, get_current_vllm_config
38 -from vllm.distributed import (
39 - get_ep_group,
40 - get_pp_group,
41 - get_tensor_model_parallel_world_size,
42 -)
43 -from vllm.model_executor.layers.activation import SiluAndMul
44 -from vllm.model_executor.layers.attention import Attention
45 -from vllm.model_executor.layers.fused_moe import (
46 - FusedMoEFactory,
47 - fused_moe_make_expert_params_mapping,
48 -)
49 -from vllm.model_executor.layers.layernorm import RMSNorm
50 -from vllm.model_executor.layers.linear import (
51 - ColumnParallelLinear,
52 - MergedColumnParallelLinear,
53 - QKVParallelLinear,
54 - ReplicatedLinear,
55 - RowParallelLinear,
56 -)
57 -from vllm.model_executor.layers.logits_processor import LogitsProcessor
58 -from vllm.model_executor.layers.quantization import QuantizationConfig
59 -from vllm.model_executor.layers.rotary_embedding import get_rope
60 -from vllm.model_executor.layers.vocab_parallel_embedding import (
61 - ParallelLMHead,
62 - VocabParallelEmbedding,
63 -)
64 -from vllm.model_executor.model_loader.weight_utils import (
65 - default_weight_loader,
66 - maybe_remap_kv_scale_name,
67 -)
68 -from vllm.sequence import IntermediateTensors
69 -from vllm.v1.attention.backend import AttentionType
70 -
71 -from .interfaces import (
72 - EagleModelMixin,
73 - MixtureOfExperts,
74 - SupportsEagle,
75 - SupportsEagle3,
76 - SupportsLoRA,
77 - SupportsPP,
78 -)
79 -from .utils import (
80 - AutoWeightsLoader,
81 - PPMissingLayer,
82 - is_pp_missing_parameter,
83 - make_layers,
84 - maybe_prefix,
85 -)
86 -
87 -
88 -def _is_moe(config: PretrainedConfig) -> bool:
89 - num_experts = getattr(config, "num_experts", None)
90 - if isinstance(num_experts, int):
91 - return num_experts > 1
92 - if isinstance(num_experts, list) and num_experts:
93 - # Ensure all elements are integers before calling max.
94 - if all(isinstance(e, int) for e in num_experts):
95 - return max(num_experts) > 1
96 - else:
97 - return False
98 - return False
99 -
100 -
101 -def _get_cla_factor(config: PretrainedConfig) -> int:
102 - if not getattr(config, "use_cla", False):
103 - return 1
104 - return getattr(config, "cla_share_factor", 1)
105 -
106 -
107 -class HunYuanMLP(nn.Module):
108 - def __init__(
109 - self,
110 - hidden_size: int,
111 - intermediate_size: int,
112 - hidden_act: str,
113 - quant_config: QuantizationConfig | None = None,
114 - bias: bool = False,
115 - prefix: str = "",
116 - reduce_results: bool = True,
117 - ) -> None:
118 - super().__init__()
119 - self.gate_up_proj = MergedColumnParallelLinear(
120 - input_size=hidden_size,
121 - output_sizes=[intermediate_size] * 2,
122 - bias=bias,
123 - quant_config=quant_config,
124 - prefix=f"{prefix}.gate_up_proj",
125 - )
126 - self.down_proj = RowParallelLinear(
127 - input_size=intermediate_size,
128 - output_size=hidden_size,
129 - bias=bias,
130 - quant_config=quant_config,
131 - prefix=f"{prefix}.down_proj",
132 - reduce_results=reduce_results,
133 - )
134 - if hidden_act != "silu":
135 - raise ValueError(
136 - f"Unsupported activation: {hidden_act}. Only silu is supported for now."
137 - )
138 - self.act_fn = SiluAndMul()
139 -
140 - def forward(self, x):
141 - gate_up, _ = self.gate_up_proj(x)
142 - x = self.act_fn(gate_up)
143 - x, _ = self.down_proj(x)
144 - return x
145 -
146 -
147 -class HunYuanAttention(nn.Module):
148 - def __init__(
149 - self,
150 - config: PretrainedConfig,
151 - hidden_size: int,
152 - num_heads: int,
153 - num_kv_heads: int,
154 - max_position_embeddings: int = 8192,
155 - quant_config: QuantizationConfig | None = None,
156 - bias: bool = False,
157 - cache_config: CacheConfig | None = None,
158 - prefix: str = "",
159 - layer_id: int = -1,
160 - ) -> None:
161 - super().__init__()
162 - self.hidden_size = hidden_size
163 - tp_size = get_tensor_model_parallel_world_size()
164 - self.total_num_heads = num_heads
165 - assert self.total_num_heads % tp_size == 0
166 - self.num_heads = self.total_num_heads // tp_size
167 - self.total_num_kv_heads = num_kv_heads
168 - if self.total_num_kv_heads >= tp_size:
169 - # Number of KV heads is greater than TP size, so we partition
170 - # the KV heads across multiple tensor parallel GPUs.
171 - assert self.total_num_kv_heads % tp_size == 0
172 - else:
173 - # Number of KV heads is less than TP size, so we replicate
174 - # the KV heads across multiple tensor parallel GPUs.
175 - assert tp_size % self.total_num_kv_heads == 0
176 - self.num_kv_heads = max(1, self.total_num_kv_heads // tp_size)
177 -
178 - if hasattr(config, "head_dim") and config.head_dim:
179 - self.head_dim = config.head_dim
180 - elif hasattr(config, "attention_head_dim"):
181 - self.head_dim = config.attention_head_dim
182 - else:
183 - self.head_dim = self.hidden_size // self.total_num_heads
184 - self.q_size = self.num_heads * self.head_dim
185 - self.kv_size = self.num_kv_heads * self.head_dim
186 - self.scaling = self.head_dim**-0.5
187 - self.max_position_embeddings = max_position_embeddings
188 - self.use_qk_norm = getattr(config, "use_qk_norm", False)
189 - self.layer_id = layer_id
190 -
191 - self.qkv_proj = QKVParallelLinear(
192 - hidden_size=hidden_size,
193 - head_size=self.head_dim,
194 - total_num_heads=self.total_num_heads,
195 - total_num_kv_heads=self.total_num_kv_heads,
196 - bias=bias,
197 - quant_config=quant_config,
198 - prefix=f"{prefix}.qkv_proj",
199 - )
200 -
201 - self.o_proj = RowParallelLinear(
202 - input_size=self.total_num_heads * self.head_dim,
203 - output_size=hidden_size,
204 - bias=bias,
205 - quant_config=quant_config,
206 - prefix=f"{prefix}.o_proj",
207 - )
208 -
209 - self.rotary_emb = get_rope(
210 - self.head_dim,
211 - max_position=max_position_embeddings,
212 - rope_parameters=config.rope_parameters,
213 - is_neox_style=True,
214 - )
215 - self.attn = Attention(
216 - self.num_heads,
217 - self.head_dim,
218 - self.scaling,
219 - num_kv_heads=self.num_kv_heads,
220 - cache_config=cache_config,
221 - quant_config=quant_config,
222 - prefix=f"{prefix}.attn",
223 - )
224 -
225 - if self.use_qk_norm:
226 - self.query_layernorm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
227 - self.key_layernorm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
228 -
229 - def forward(
230 - self,
231 - positions: torch.Tensor,
232 - hidden_states: torch.Tensor,
233 - kv_states: tuple[torch.Tensor] | None = None,
234 - ) -> torch.Tensor:
235 - qkv, _ = self.qkv_proj(hidden_states)
236 - q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
237 - q, k = self.rotary_emb(positions, q, k)
238 - ori_k = k
239 - if self.use_qk_norm:
240 - q = self.query_layernorm(
241 - q.view(-1, self.num_heads, self.head_dim),
242 - )
243 - k = self.key_layernorm(
244 - k.view(-1, self.num_kv_heads, self.head_dim),
245 - )
246 -
247 - attn_output = self.attn(q, k, v)
248 - # For o_proj
249 - attn_output = attn_output.view(q.shape[0], -1)
250 - output, _ = self.o_proj(attn_output)
251 - return output, (ori_k, v)
252 -
253 -
254 -class HunYuanCrossAttention(nn.Module):
255 - def __init__(
256 - self,
257 - config: PretrainedConfig,
258 - hidden_size: int,
259 - num_heads: int,
260 - num_kv_heads: int,
261 - max_position_embeddings: int = 8192,
262 - quant_config: QuantizationConfig | None = None,
263 - bias: bool = False,
264 - cache_config: CacheConfig | None = None,
265 - prefix: str = "",
266 - layer_id: int = -1,
267 - ) -> None:
268 - super().__init__()
269 - self.hidden_size = hidden_size
270 - tp_size = get_tensor_model_parallel_world_size()
271 - self.total_num_heads = num_heads
272 - assert self.total_num_heads % tp_size == 0
273 - self.num_heads = self.total_num_heads // tp_size
274 - self.total_num_kv_heads = num_kv_heads
275 - if self.total_num_kv_heads >= tp_size:
276 - # Number of KV heads is greater than TP size, so we partition
277 - # the KV heads across multiple tensor parallel GPUs.
278 - assert self.total_num_kv_heads % tp_size == 0
279 - else:
280 - # Number of KV heads is less than TP size, so we replicate
281 - # the KV heads across multiple tensor parallel GPUs.
282 - assert tp_size % self.total_num_kv_heads == 0
283 - self.num_kv_heads = max(1, self.total_num_kv_heads // tp_size)
284 - # MistralConfig has an optional head_dim introduced by Mistral-Nemo
285 - if hasattr(config, "head_dim"):
286 - self.head_dim = config.head_dim
287 - elif hasattr(config, "attention_head_dim"):
288 - self.head_dim = config.attention_head_dim
289 - else:
290 - self.head_dim = self.hidden_size // self.total_num_heads
291 - self.q_size = self.num_heads * self.head_dim
292 - self.kv_size = self.num_kv_heads * self.head_dim
293 - self.scaling = self.head_dim**-0.5
294 - self.max_position_embeddings = max_position_embeddings
295 - self.use_qk_norm = getattr(config, "use_qk_norm", False)
296 - self.layer_id = layer_id
297 -
298 - self.q_proj = ColumnParallelLinear(
299 - hidden_size,
300 - hidden_size,
301 - bias=bias,
302 - quant_config=quant_config,
303 - prefix=f"{prefix}.q_proj",
304 - )
305 -
306 - self.o_proj = RowParallelLinear(
307 - input_size=self.total_num_heads * self.head_dim,
308 - output_size=hidden_size,
309 - bias=bias,
310 - quant_config=quant_config,
311 - prefix=f"{prefix}.o_proj",
312 - )
313 -
314 - self.rotary_emb = get_rope(
315 - self.head_dim,
316 - max_position=max_position_embeddings,
317 - rope_parameters=config.rope_parameters,
318 - is_neox_style=True,
319 - )
320 - self.attn = Attention(
321 - self.num_heads,
322 - self.head_dim,
323 - self.scaling,
324 - num_kv_heads=self.num_kv_heads,
325 - cache_config=cache_config,
326 - quant_config=quant_config,
327 - prefix=f"{prefix}.attn",
328 - attn_type=AttentionType.ENCODER_DECODER,
329 - )
330 -
331 - if self.use_qk_norm:
332 - self.query_layernorm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
333 - self.key_layernorm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
334 -
335 - def forward(
336 - self,
337 - positions: torch.Tensor,
338 - hidden_states: torch.Tensor,
339 - kv_states: tuple[torch.Tensor] | None = None,
340 - ) -> torch.Tensor:
341 - assert kv_states is not None
342 - ori_k, v = kv_states # use last layer kv,
343 - k = ori_k
344 - q, _ = self.q_proj(hidden_states)
345 - k_tmp = torch.empty_like(k) # Todo: reduant rotary embedding
346 - q, _ = self.rotary_emb(positions, q, k_tmp)
347 - if self.use_qk_norm:
348 - q = self.query_layernorm(
349 - q.view(-1, self.num_heads, self.head_dim),
350 - )
351 - k = self.key_layernorm(
352 - k.view(-1, self.num_kv_heads, self.head_dim),
353 - )
354 -
355 - attn_output = self.attn(q, k, v)
356 - # For o_proj
357 - attn_output = attn_output.view(q.shape[0], -1)
358 - output, _ = self.o_proj(attn_output)
359 - return output, (ori_k, v)
360 -
361 -
362 -class HunYuanSparseMoeBlock(nn.Module):
363 - def __init__(
364 - self,
365 - config: PretrainedConfig,
366 - quant_config: QuantizationConfig | None = None,
367 - layer_id: int = -1,
368 - prefix: str = "",
369 - enable_eplb: bool = False,
370 - ):
371 - super().__init__()
372 - self.tp_size = get_tensor_model_parallel_world_size()
373 -
374 - self.ep_group = get_ep_group().device_group
375 - self.ep_size = self.ep_group.size()
376 - self.n_routed_experts = config.num_experts
377 -
378 - if self.tp_size > config.num_experts:
379 - raise ValueError(
380 - f"Tensor parallel size {self.tp_size} is greater than "
381 - f"the number of experts {config.num_experts}."
382 - )
383 -
384 - # Get layer_id topk if config.moe_topk is a list
385 - if isinstance(config.moe_topk, list):
386 - assert layer_id >= 0
387 - assert len(config.moe_topk) > layer_id
388 - top_k = config.moe_topk[layer_id]
389 - else:
390 - top_k = config.moe_topk
391 -
392 - # If it is moe, moe_intermediate_size is preferred
393 - intermediate_size = config.intermediate_size
394 - if config.moe_intermediate_size is not None:
395 - intermediate_size = (
396 - config.moe_intermediate_size
397 - if isinstance(config.moe_intermediate_size, int)
398 - else config.moe_intermediate_size[layer_id]
399 - )
400 -
401 - # Load balancing settings.
402 - vllm_config = get_current_vllm_config()
403 - eplb_config = vllm_config.parallel_config.eplb_config
404 - self.enable_eplb = enable_eplb
405 -
406 - self.n_logical_experts = self.n_routed_experts
407 - self.n_redundant_experts = eplb_config.num_redundant_experts
408 - self.n_physical_experts = self.n_logical_experts + self.n_redundant_experts
409 - self.n_local_physical_experts = self.n_physical_experts // self.ep_size
410 - self.gate = ReplicatedLinear(
411 - config.hidden_size,
412 - config.num_experts,
413 - bias=False,
414 - quant_config=None,
415 - prefix=f"{prefix}.gate",
416 - )
417 - if config.use_mixed_mlp_moe > 0:
418 - # Get layer_id num_shared_expert if config.num_shared_expert is
419 - # a list.
420 - if isinstance(config.num_shared_expert, list):
421 - assert layer_id >= 0
422 - assert len(config.num_shared_expert) > layer_id
423 - num_shared_expert = config.num_shared_expert[layer_id]
424 - else:
425 - num_shared_expert = config.num_shared_expert
426 -
427 - self.shared_mlp = HunYuanMLP(
428 - hidden_size=config.hidden_size,
429 - intermediate_size=config.intermediate_size * num_shared_expert,
430 - hidden_act=config.hidden_act,
431 - quant_config=quant_config,
432 - reduce_results=False,
433 - prefix=f"{prefix}.shared_mlp",
434 - )
435 - else:
436 - self.shared_mlp = None
437 -
438 - self.experts = FusedMoEFactory(
439 - shared_experts=self.shared_mlp,
440 - num_experts=self.n_routed_experts,
441 - top_k=top_k,
442 - hidden_size=config.hidden_size,
443 - intermediate_size=intermediate_size,
444 - renormalize=top_k > 1,
445 - quant_config=quant_config,
446 - prefix=f"{prefix}.experts",
447 - enable_eplb=self.enable_eplb,
448 - num_redundant_experts=self.n_redundant_experts,
449 - )
450 -
451 - def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
452 - # NOTE: hidden_states can have either 1D or 2D shape.
453 - orig_shape = hidden_states.shape
454 - hidden_dim = hidden_states.shape[-1]
455 - hidden_states = hidden_states.view(-1, hidden_dim)
456 -
457 - # router_logits: (num_tokens, n_experts)
458 - router_logits, _ = self.gate(hidden_states)
459 - final_hidden_states = self.experts(
460 - hidden_states=hidden_states, router_logits=router_logits
461 - )
462 -
463 - return final_hidden_states.view(orig_shape)
464 -
465 -
466 -class HunYuanDecoderLayer(nn.Module):
467 - def __init__(
468 - self,
469 - config: PretrainedConfig,
470 - cache_config: CacheConfig | None = None,
471 - quant_config: QuantizationConfig | None = None,
472 - prefix: str = "",
473 - layer_id: int = -1,
474 - enable_eplb: bool = False,
475 - ) -> None:
476 - super().__init__()
477 - assert layer_id >= 0
478 - self.layer_id = layer_id
479 - self.hidden_size = config.hidden_size
480 - self.intermediate_size = (
481 - config.intermediate_size
482 - if isinstance(config.intermediate_size, int)
483 - else config.intermediate_size[layer_id]
484 - )
485 - max_position_embeddings = getattr(config, "max_position_embeddings", 8192)
486 - attention_bias = getattr(config, "attention_bias", False) or getattr(
487 - config, "bias", False
488 - )
489 - cla_factor = _get_cla_factor(config)
490 - attention_type = (
491 - AttentionType.ENCODER_DECODER
492 - if layer_id >= 0 and layer_id % cla_factor != 0
493 - else AttentionType.DECODER
494 - )
495 - if attention_type == AttentionType.DECODER:
496 - self.self_attn = HunYuanAttention(
497 - config=config,
498 - hidden_size=self.hidden_size,
499 - num_heads=config.num_attention_heads,
500 - num_kv_heads=getattr(
501 - config, "num_key_value_heads", config.num_attention_heads
502 - ),
503 - max_position_embeddings=max_position_embeddings,
504 - quant_config=quant_config,
505 - bias=attention_bias,
506 - cache_config=cache_config,
507 - prefix=f"{prefix}.self_attn",
508 - layer_id=layer_id,
509 - )
510 - elif attention_type == AttentionType.ENCODER_DECODER:
511 - self.self_attn = HunYuanCrossAttention(
512 - config=config,
513 - hidden_size=self.hidden_size,
514 - num_heads=config.num_attention_heads,
515 - num_kv_heads=getattr(
516 - config, "num_key_value_heads", config.num_attention_heads
517 - ),
518 - max_position_embeddings=max_position_embeddings,
519 - quant_config=quant_config,
520 - bias=attention_bias,
521 - cache_config=cache_config,
522 - prefix=f"{prefix}.self_attn",
523 - layer_id=layer_id,
524 - )
525 - else:
526 - raise RuntimeError(f"Unsupported attention type: {attention_type}")
527 -
528 - if _is_moe(config):
529 - self.mlp = HunYuanSparseMoeBlock(
530 - config=config,
531 - quant_config=quant_config,
532 - layer_id=layer_id,
533 - prefix=f"{prefix}.mlp",
534 - enable_eplb=enable_eplb,
535 - )
536 - else:
537 - self.mlp = HunYuanMLP(
538 - hidden_size=self.hidden_size,
539 - intermediate_size=self.intermediate_size,
540 - hidden_act=config.hidden_act,
541 - quant_config=quant_config,
542 - bias=getattr(config, "mlp_bias", False),
543 - prefix=f"{prefix}.mlp",
544 - )
545 -
546 - self.input_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
547 - self.post_attention_layernorm = RMSNorm(
548 - config.hidden_size, eps=config.rms_norm_eps
549 - )
550 -
551 - def forward(
552 - self,
553 - positions: torch.Tensor,
554 - hidden_states: torch.Tensor,
555 - residual: torch.Tensor | None,
556 - kv_states: tuple[torch.Tensor] | None = None,
557 - ) -> tuple[torch.Tensor, torch.Tensor]:
558 - # Self Attention
559 - if residual is None:
560 - residual = hidden_states
561 - hidden_states = self.input_layernorm(hidden_states)
562 - else:
563 - hidden_states, residual = self.input_layernorm(hidden_states, residual)
564 - hidden_states, ori_kv_states = self.self_attn(
565 - positions=positions,
566 - hidden_states=hidden_states,
567 - kv_states=kv_states,
568 - )
569 -
570 - # Fully Connected
571 - hidden_states, residual = self.post_attention_layernorm(hidden_states, residual)
572 - hidden_states = self.mlp(hidden_states)
573 - return hidden_states, residual, ori_kv_states
574 -
575 -
576 -@support_torch_compile(
577 - dynamic_arg_dims={
578 - "input_ids": 0,
579 - # positions is of shape (xd, seq_len) if xdrope is enabled for hunyuan-vl,
580 - # otherwise (seq_len, ).
581 - "positions": -1,
582 - "intermediate_tensors": 0,
583 - "inputs_embeds": 0,
584 - }
585 -)
586 -class HunYuanModel(nn.Module, EagleModelMixin):
587 - def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
588 - super().__init__()
589 -
590 - config = vllm_config.model_config.hf_config
591 - cache_config = vllm_config.cache_config
592 - quant_config = vllm_config.quant_config
593 -
594 - eplb_config = vllm_config.parallel_config.eplb_config
595 - enable_eplb = vllm_config.parallel_config.enable_eplb
596 - self.num_redundant_experts = eplb_config.num_redundant_experts
597 -
598 - self.config = config
599 - self.quant_config = quant_config
600 -
601 - self.vocab_size = config.vocab_size
602 -
603 - if get_pp_group().is_first_rank or (
604 - config.tie_word_embeddings and get_pp_group().is_last_rank
605 - ):
606 - self.embed_tokens = VocabParallelEmbedding(
607 - self.vocab_size,
608 - config.hidden_size,
609 - quant_config=quant_config,
610 - )
611 - else:
612 - self.embed_tokens = PPMissingLayer()
613 - self.start_layer, self.end_layer, self.layers = make_layers(
614 - config.num_hidden_layers,
615 - lambda prefix: HunYuanDecoderLayer(
616 - config=config,
617 - layer_id=int(prefix.split(".")[-1]),
618 - cache_config=cache_config,
619 - quant_config=quant_config,
620 - prefix=prefix,
621 - enable_eplb=enable_eplb,
622 - ),
623 - prefix=f"{prefix}.layers",
624 - )
625 - if get_pp_group().is_last_rank:
626 - self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
627 - else:
628 - self.norm = PPMissingLayer()
629 -
630 - def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
631 - return self.embed_tokens(input_ids)
632 -
633 - def forward(
634 - self,
635 - input_ids: torch.Tensor | None,
636 - positions: torch.Tensor,
637 - intermediate_tensors: IntermediateTensors | None,
638 - inputs_embeds: torch.Tensor | None = None,
639 - ) -> torch.Tensor | IntermediateTensors:
640 - if get_pp_group().is_first_rank:
641 - if inputs_embeds is not None:
642 - hidden_states = inputs_embeds
643 - else:
644 - hidden_states = self.embed_input_ids(input_ids)
645 - residual = None
646 - else:
647 - assert intermediate_tensors is not None
648 - hidden_states = intermediate_tensors["hidden_states"]
649 - residual = intermediate_tensors["residual"]
650 -
651 - cla_factor = _get_cla_factor(self.config)
652 - prev_kv_states = None
653 - aux_hidden_states = self._maybe_add_hidden_state([], 0, hidden_states, residual)
654 - for i, layer in enumerate(
655 - islice(self.layers, self.start_layer, self.end_layer)
656 - ):
657 - hidden_states, residual, kv_states = layer(
658 - positions,
659 - hidden_states,
660 - residual,
661 - prev_kv_states,
662 - )
663 -
664 - if getattr(self.config, "use_cla", False) and i % cla_factor == 0:
665 - prev_kv_states = kv_states
666 - else:
667 - prev_kv_states = None
668 -
669 - self._maybe_add_hidden_state(
670 - aux_hidden_states, i + 1, hidden_states, residual
671 - )
672 -
673 - if not get_pp_group().is_last_rank:
674 - return IntermediateTensors(
675 - {"hidden_states": hidden_states, "residual": residual}
676 - )
677 -
678 - hidden_states, _ = self.norm(hidden_states, residual)
679 -
680 - if len(aux_hidden_states) > 0:
681 - return hidden_states, aux_hidden_states
682 - return hidden_states
683 -
684 - def _split_qkv_weight(self, qkv: torch.Tensor):
685 - num_attention_heads = self.config.num_attention_heads
686 - num_kv_heads = getattr(
687 - self.config, "num_key_value_heads", self.config.num_attention_heads
688 - )
689 - num_key_value_groups = num_attention_heads // num_kv_heads
690 - hidden_size = self.config.hidden_size
691 -
692 - if hasattr(self.config, "head_dim"):
693 - attention_head_dim = self.config.head_dim
694 - elif hasattr(self.config, "attention_head_dim"):
695 - attention_head_dim = self.config.attention_head_dim
696 - else:
697 - attention_head_dim = self.config.hidden_size // num_attention_heads
698 -
699 - qkv = qkv.reshape(
700 - num_kv_heads, num_key_value_groups + 2, attention_head_dim, hidden_size
701 - )
702 - q, k, v = torch.split(qkv, (num_key_value_groups, 1, 1), dim=1)
703 - q = q.reshape(-1, hidden_size)
704 - k = k.reshape(-1, hidden_size)
705 - v = v.reshape(-1, hidden_size)
706 - return torch.concat((q, k, v))
707 -
708 - def get_expert_mapping(self) -> list[tuple[str, str, int, str]]:
709 - if _is_moe(self.config):
710 - # Params for weights, fp8 weight scales, fp8 activation scales
711 - # (param_name, weight_name, expert_id, shard_id)
712 - return fused_moe_make_expert_params_mapping(
713 - self,
714 - ckpt_gate_proj_name="gate_proj",
715 - ckpt_down_proj_name="down_proj",
716 - ckpt_up_proj_name="up_proj",
717 - num_experts=self.config.num_experts,
718 - num_redundant_experts=self.num_redundant_experts,
719 - )
720 - else:
721 - return []
722 -
723 - def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]):
724 - cla_factor = _get_cla_factor(self.config)
725 - stacked_params_mapping = [
726 - # (param_name, shard_name, shard_id)
727 - (".qkv_proj", ".q_proj", "q"),
728 - (".qkv_proj", ".k_proj", "k"),
729 - (".qkv_proj", ".v_proj", "v"),
730 - (".gate_up_proj", ".gate_proj", 0),
731 - (".gate_up_proj", ".up_proj", 1),
732 - ]
733 -
734 - num_attention_heads = self.config.num_attention_heads
735 - num_kv_heads = getattr(
736 - self.config, "num_key_value_heads", self.config.num_attention_heads
737 - )
738 - split_params_mapping = [
739 - (".gate_up_proj", ".gate_and_up_proj", 2, [(1, 1), (0, 1)], None),
740 - (
741 - ".qkv_proj",
742 - ".qkv_proj",
743 - num_attention_heads + num_kv_heads * 2,
744 - [("q", num_attention_heads), ("k", num_kv_heads), ("v", num_kv_heads)],
745 - self._split_qkv_weight,
746 - ),
747 - ]
748 -
749 - params_dict = dict(self.named_parameters())
750 - loaded_params: set[str] = set()
751 - expert_params_mapping = self.get_expert_mapping()
752 - for name, loaded_weight in weights:
753 - if "rotary_emb.inv_freq" in name:
754 - continue
755 - if "gate_proj_bias" in name:
756 - name = name.replace("gate_proj_bias", "gate_proj.bias")
757 - if "up_proj_bias" in name:
758 - name = name.replace("up_proj_bias", "up_proj.bias")
759 - if "rotary_emb.cos_cached" in name or "rotary_emb.sin_cached" in name:
760 - # Models trained using ColossalAI may include these tensors in
761 - # the checkpoint. Skip them.
762 - continue
763 - # With tie_word_embeddings, we can skip lm_head.weight
764 - # The weight might appear unnecessarily in the files if the model is
765 - # processed with quantization, LoRA, fine-tuning, etc.
766 - if self.config.tie_word_embeddings and "lm_head.weight" in name:
767 - continue
768 -
769 - is_found = False
770 - for param_name, weight_name, shard_id in stacked_params_mapping:
771 - if weight_name not in name:
772 - continue
773 - if "mlp.experts" in name:
774 - continue
775 - # cross layer only have q_proj, skip qkv pack
776 - if weight_name == ".q_proj":
777 - match = re.search(r"layers\.\d+", name)
778 - if match:
779 - layer_id = int(match.group(0).split(".")[-1])
780 - if cla_factor > 1 and layer_id % cla_factor != 0:
781 - continue
782 - name = name.replace(weight_name, param_name)
783 - # Skip loading extra bias for GPTQ models.
784 - if name.endswith(".bias") and name not in params_dict:
785 - continue
786 -
787 - if is_pp_missing_parameter(name, self):
788 - continue
789 -
790 - param = params_dict[name]
791 - weight_loader = param.weight_loader
792 - weight_loader(param, loaded_weight, shard_id)
793 - loaded_params.add(name)
794 - is_found = True
795 - break
796 - if is_found:
797 - continue
798 -
799 - for (
800 - param_name,
801 - weight_name,
802 - den,
803 - split_param,
804 - func,
805 - ) in split_params_mapping:
806 - if weight_name not in name:
807 - continue
808 - name = name.replace(weight_name, param_name)
809 - # Skip loading extra bias for GPTQ models.
810 - if name.endswith(".bias") and name not in params_dict:
811 - continue
812 -
813 - if is_pp_missing_parameter(name, self):
814 - continue
815 -
816 - assert loaded_weight.shape[0] % den == 0
817 - units = loaded_weight.shape[0] // den
818 -
819 - param = params_dict[name]
820 - weight_loader = param.weight_loader
821 - offset = 0
822 - for shard_id, num in split_param:
823 - new_offset = offset + num * units
824 - if func:
825 - weight_loader(
826 - param, func(loaded_weight)[offset:new_offset], shard_id
827 - )
828 - else:
829 - weight_loader(param, loaded_weight[offset:new_offset], shard_id)
830 - offset = new_offset
831 -
832 - break
833 - else:
834 - # Skip loading extra bias for GPTQ models.
835 - if name.endswith(".bias") and name not in params_dict:
836 - continue
837 - is_expert_weight = False
838 - for mapping in expert_params_mapping:
839 - param_name, weight_name, expert_id, shard_id = mapping
840 - if weight_name not in name:
841 - continue
842 - # this is an expert weight and should not be
843 - # attempted to load as other weights later
844 - is_expert_weight = True
845 -
846 - # Do not modify `name` since the loop may continue here
847 - # Instead, create a new variable
848 - name_mapped = name.replace(weight_name, param_name)
849 - if is_pp_missing_parameter(name_mapped, self):
850 - continue
851 - param = params_dict[name_mapped]
852 - # We should ask the weight loader to return success or not
853 - # here since otherwise we may skip experts with other
854 - # available replicas.
855 - weight_loader = typing.cast(
856 - Callable[..., bool], param.weight_loader
857 - )
858 - success = weight_loader(
859 - param,
860 - loaded_weight,
861 - name_mapped,
862 - shard_id=shard_id,
863 - expert_id=expert_id,
864 - return_success=True,
865 - )
866 - if success:
867 - name = name_mapped
868 - break
869 - else:
870 - if is_expert_weight:
871 - # We've checked that this is an expert weight
872 - # However it's not mapped locally to this rank
873 - # So we simply skip it
874 - continue
875 - # Remapping the name of FP8 kv-scale.
876 - name = maybe_remap_kv_scale_name(name, params_dict)
877 - if name is None:
878 - continue
879 -
880 - if is_pp_missing_parameter(name, self):
881 - continue
882 -
883 - if "mlp.gate.wg." in name:
884 - name = name.replace("wg.", "")
885 -
886 - param = params_dict[name]
887 - weight_loader = getattr(
888 - param, "weight_loader", default_weight_loader
889 - )
890 - weight_loader(param, loaded_weight)
891 - loaded_params.add(name)
892 - return loaded_params
893 -
894 -
895 -class HunyuanV1ModelBase(
896 - nn.Module, SupportsLoRA, SupportsPP, SupportsEagle, SupportsEagle3
897 -):
898 - packed_modules_mapping = {
899 - "qkv_proj": [
900 - "q_proj",
901 - "k_proj",
902 - "v_proj",
903 - ],
904 - "gate_up_proj": [
905 - "gate_proj",
906 - "up_proj",
907 - ],
908 - }
909 -
910 - def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
911 - super().__init__()
912 -
913 - config = vllm_config.model_config.hf_config
914 - quant_config = vllm_config.quant_config
915 - self.config = config
916 - self.quant_config = quant_config
917 -
918 - self.model = HunYuanModel(
919 - vllm_config=vllm_config,
920 - prefix=maybe_prefix(prefix, "model"),
921 - )
922 - if get_pp_group().is_last_rank:
923 - self.lm_head = ParallelLMHead(
924 - config.vocab_size,
925 - config.hidden_size,
926 - quant_config=quant_config,
927 - prefix=maybe_prefix(prefix, "lm_head"),
928 - )
929 - if config.tie_word_embeddings:
930 - self.lm_head = self.lm_head.tie_weights(self.model.embed_tokens)
931 -
932 - logit_scale = getattr(config, "logit_scale", 1.0)
933 - self.logits_processor = LogitsProcessor(
934 - config.vocab_size, scale=logit_scale
935 - )
936 - else:
937 - self.lm_head = PPMissingLayer()
938 -
939 - def forward(
940 - self,
941 - input_ids: torch.Tensor | None,
942 - positions: torch.Tensor,
943 - intermediate_tensors: IntermediateTensors | None = None,
944 - inputs_embeds: torch.Tensor | None = None,
945 - ) -> torch.Tensor | IntermediateTensors:
946 - model_output = self.model(
947 - input_ids, positions, intermediate_tensors, inputs_embeds
948 - )
949 - return model_output
950 -
951 - def compute_logits(
952 - self,
953 - hidden_states: torch.Tensor,
954 - ) -> torch.Tensor | None:
955 - logits = self.logits_processor(self.lm_head, hidden_states)
956 - return logits
957 -
958 - def make_empty_intermediate_tensors(
959 - self, batch_size: int, dtype: torch.dtype, device: torch.device
960 - ) -> IntermediateTensors:
961 - return IntermediateTensors(
962 - {
963 - "hidden_states": torch.zeros(
964 - (batch_size, self.config.hidden_size), dtype=dtype, device=device
965 - ),
966 - "residual": torch.zeros(
967 - (batch_size, self.config.hidden_size), dtype=dtype, device=device
968 - ),
969 - }
970 - )
971 -
972 - def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
973 - loader = AutoWeightsLoader(self)
974 - return loader.load_weights(weights)
975 -
976 - def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
977 - return self.model.embed_input_ids(input_ids)
978 -
979 -
980 -class HunYuanMoEV1Base(HunyuanV1ModelBase, MixtureOfExperts):
981 - def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
982 - super().__init__(vllm_config=vllm_config, prefix=prefix)
983 -
984 - # Set MoE hyperparameters
985 - self.num_expert_groups = 1
986 - self.moe_layers = []
987 - example_layer = None
988 - for layer in self.model.layers:
989 - if isinstance(layer, PPMissingLayer):
990 - continue
991 -
992 - assert isinstance(layer, HunYuanDecoderLayer)
993 - if isinstance(layer.mlp, HunYuanSparseMoeBlock):
994 - example_layer = layer.mlp
995 - self.moe_layers.append(layer.mlp.experts)
996 -
997 - if example_layer is None:
998 - raise RuntimeError("No HunYuanMoE layer found in model.layers.")
999 -
1000 - self.num_moe_layers = len(self.moe_layers)
1001 - self.num_logical_experts = example_layer.n_logical_experts
1002 - self.num_physical_experts = example_layer.n_physical_experts
1003 - self.num_local_physical_experts = example_layer.n_local_physical_experts
1004 - self.num_routed_experts = example_layer.n_routed_experts
1005 - self.num_redundant_experts = example_layer.n_redundant_experts
1006 -
1007 - def update_physical_experts_metadata(
1008 - self,
1009 - num_physical_experts: int,
1010 - num_local_physical_experts: int,
1011 - ) -> None:
1012 - assert self.num_local_physical_experts == num_local_physical_experts
1013 - self.num_physical_experts = num_physical_experts
1014 - self.num_local_physical_experts = num_local_physical_experts
1015 - self.num_redundant_experts = num_physical_experts - self.num_logical_experts
1016 - for layer in self.model.layers:
1017 - if isinstance(layer.mlp, HunYuanSparseMoeBlock):
1018 - moe = layer.mlp
1019 - moe.n_local_physical_experts = num_local_physical_experts
1020 - moe.n_physical_experts = num_physical_experts
1021 - moe.n_redundant_experts = self.num_redundant_experts
1022 - moe.experts.update_expert_map()
1023 -
1024 - def get_expert_mapping(self) -> list[tuple[str, str, int, str]]:
1025 - return self.model.get_expert_mapping()
1026 -
1027 -
1028 -class HunYuanDenseV1Base(HunyuanV1ModelBase):
1029 - def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
1030 - super().__init__(vllm_config=vllm_config, prefix=prefix)
1031 -
1032 -
1033 -class HunYuanDenseV1ForCausalLM(HunYuanDenseV1Base):
1034 - pass
1035 -
1036 -
1037 -class HunYuanMoEV1ForCausalLM(HunYuanMoEV1Base):
1038 - pass
vllm/model_executor/models/hunyuan_vision.py deleted
-1047
@@ -1,1047 +0,0 @@
1 -# SPDX-License-Identifier: Apache-2.0
2 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 -
4 -# coding=utf-8
5 -# Copyright 2025 The HunYuan team.
6 -# Copyright 2025 The vLLM team.
7 -# Copyright 2025 EleutherAI and the HuggingFace Inc. team. All rights reserved.
8 -#
9 -# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
10 -# and OPT implementations in this library. It has been modified from its
11 -# original forms to accommodate minor architectural differences compared
12 -# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
13 -#
14 -# Licensed under the Apache License, Version 2.0 (the "License");
15 -# you may not use this file except in compliance with the License.
16 -# You may obtain a copy of the License at
17 -#
18 -# http://www.apache.org/licenses/LICENSE-2.0
19 -#
20 -# Unless required by applicable law or agreed to in writing, software
21 -# distributed under the License is distributed on an "AS IS" BASIS,
22 -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
23 -# See the License for the specific language governing permissions and
24 -# limitations under the License.
25 -"""Inference-only HunYuan-VL model compatible with HuggingFace weights."""
26 -
27 -from collections.abc import Callable, Iterable, Mapping, Sequence
28 -from functools import partial
29 -from typing import Annotated, Any, Literal, TypeAlias
30 -
31 -import torch
32 -import torch.nn as nn
33 -import torch.nn.functional as F
34 -from transformers import BatchFeature, HunYuanVLProcessor
35 -from transformers.models.hunyuan_vl.image_processing_hunyuan_vl import (
36 - HunYuanVLImageProcessor,
37 - smart_resize,
38 -)
39 -
40 -from vllm.config import VllmConfig
41 -from vllm.config.multimodal import BaseDummyOptions
42 -from vllm.distributed import parallel_state
43 -from vllm.distributed import utils as dist_utils
44 -from vllm.inputs import ModalityData, MultiModalDataDict
45 -from vllm.logger import init_logger
46 -from vllm.model_executor.layers.activation import get_act_fn
47 -from vllm.model_executor.layers.attention import MMEncoderAttention
48 -from vllm.model_executor.layers.layernorm import RMSNorm
49 -from vllm.model_executor.layers.linear import (
50 - ColumnParallelLinear,
51 - QKVParallelLinear,
52 - RowParallelLinear,
53 -)
54 -from vllm.model_executor.layers.quantization import QuantizationConfig
55 -from vllm.model_executor.models.module_mapping import MultiModelKeys
56 -from vllm.multimodal import MULTIMODAL_REGISTRY
57 -from vllm.multimodal.inputs import (
58 - ImageItem,
59 - MultiModalFeatureSpec,
60 - MultiModalFieldConfig,
61 - MultiModalKwargsItems,
62 -)
63 -from vllm.multimodal.parse import (
64 - DictEmbeddingItems,
65 - ImageSize,
66 - ModalityDataItems,
67 - MultiModalDataItems,
68 - MultiModalDataParser,
69 -)
70 -from vllm.multimodal.processing import (
71 - BaseDummyInputsBuilder,
72 - BaseMultiModalProcessor,
73 - BaseProcessingInfo,
74 - PromptReplacement,
75 - PromptUpdate,
76 - PromptUpdateDetails,
77 -)
78 -from vllm.sequence import IntermediateTensors
79 -from vllm.transformers_utils.configs.hunyuan_vl import (
80 - HunYuanVLConfig,
81 - HunYuanVLVisionConfig,
82 -)
83 -from vllm.utils.tensor_schema import TensorSchema, TensorShape
84 -
85 -from .interfaces import (
86 - MultiModalEmbeddings,
87 - SupportsEagle,
88 - SupportsEagle3,
89 - SupportsLoRA,
90 - SupportsMultiModal,
91 - SupportsPP,
92 - SupportsQuant,
93 - SupportsXDRoPE,
94 -)
95 -from .utils import (
96 - AutoWeightsLoader,
97 - WeightsMapper,
98 - init_vllm_registered_model,
99 - maybe_prefix,
100 -)
101 -from .vision import is_vit_use_data_parallel
102 -
103 -logger = init_logger(__name__)
104 -
105 -# === Vision Inputs === #
106 -
107 -
108 -class HunYuanVLImagePixelInputs(TensorSchema):
109 - """
110 - Dimensions:
111 - - np: Number of patches
112 - - ni: Number of images
113 - - cps: Number of channels * patch_size * patch_size
114 - """
115 -
116 - type: Literal["pixel_values"]
117 -
118 - pixel_values: Annotated[
119 - torch.Tensor,
120 - TensorShape("np", "cps"),
121 - ]
122 -
123 - image_grid_thw: Annotated[
124 - torch.Tensor,
125 - TensorShape("ni", 3),
126 - ]
127 -
128 -
129 -class HunYuanVLImageEmbeddingInputs(TensorSchema):
130 - """
131 - Dimensions:
132 - - nf: Number of image features
133 - - hs: Hidden size
134 - - ni: Number of images
135 - """
136 -
137 - type: Literal["image_embeds"]
138 -
139 - image_embeds: Annotated[
140 - torch.Tensor,
141 - TensorShape("nf", "hs"),
142 - ]
143 -
144 - image_grid_thw: Annotated[
145 - torch.Tensor,
146 - TensorShape("ni", 3),
147 - ]
148 -
149 -
150 -HunYuanVLImageInputs: TypeAlias = (
151 - HunYuanVLImagePixelInputs | HunYuanVLImageEmbeddingInputs
152 -)
153 -
154 -# === Vision Encoder === #
155 -
156 -
157 -class HunYuanVisionMLP(nn.Module):
158 - def __init__(
159 - self,
160 - in_features: int,
161 - hidden_features: int,
162 - bias: bool = True,
163 - act_fn: Callable[[torch.Tensor], torch.Tensor] = F.gelu,
164 - quant_config: QuantizationConfig | None = None,
165 - prefix: str = "",
166 - ):
167 - super().__init__()
168 - use_data_parallel = is_vit_use_data_parallel()
169 - self.dense_h_to_4h = ColumnParallelLinear(
170 - in_features,
171 - hidden_features,
172 - bias=bias,
173 - quant_config=quant_config,
174 - prefix=f"{prefix}.dense_h_to_4h",
175 - disable_tp=use_data_parallel,
176 - )
177 - self.dense_4h_to_h = RowParallelLinear(
178 - hidden_features,
179 - in_features,
180 - bias=bias,
181 - quant_config=quant_config,
182 - prefix=f"{prefix}.dense_4h_to_h",
183 - disable_tp=use_data_parallel,
184 - )
185 - self.act_fn = act_fn
186 -
187 - def forward(self, x: torch.Tensor):
188 - x_up, _ = self.dense_h_to_4h(x)
189 - x_down, _ = self.dense_4h_to_h(self.act_fn(x_up))
190 - return x_down
191 -
192 -
193 -class HunYuanVisionAttention(nn.Module):
194 - def __init__(
195 - self,
196 - embed_dim: int,
197 - num_heads: int,
198 - projection_size: int,
199 - quant_config: QuantizationConfig | None = None,
200 - prefix: str = "",
201 - ) -> None:
202 - super().__init__()
203 - # Per attention head and per partition values.
204 - use_data_parallel = is_vit_use_data_parallel()
205 - self.tp_size = (
206 - 1
207 - if use_data_parallel
208 - else parallel_state.get_tensor_model_parallel_world_size()
209 - )
210 - self.hidden_size_per_attention_head = dist_utils.divide(
211 - projection_size, num_heads
212 - )
213 - self.num_attention_heads_per_partition = dist_utils.divide(
214 - num_heads, self.tp_size
215 - )
216 -
217 - self.qkv = QKVParallelLinear(
218 - hidden_size=embed_dim,
219 - head_size=self.hidden_size_per_attention_head,
220 - total_num_heads=num_heads,
221 - total_num_kv_heads=num_heads,
222 - bias=True,
223 - quant_config=quant_config,
224 - prefix=f"{prefix}.qkv",
225 - disable_tp=use_data_parallel,
226 - )
227 -
228 - self.o_proj = RowParallelLinear(
229 - input_size=projection_size,
230 - output_size=embed_dim,
231 - quant_config=quant_config,
232 - prefix=f"{prefix}.o_proj",
233 - disable_tp=use_data_parallel,
234 - )
235 -
236 - self.scale = self.hidden_size_per_attention_head**-0.5
237 - self.attn = MMEncoderAttention(
238 - self.num_attention_heads_per_partition,
239 - self.hidden_size_per_attention_head,
240 - self.scale,
241 - prefix=f"{prefix}.attn",
242 - )
243 -
244 - def forward(
245 - self,
246 - x: torch.Tensor,
247 - ) -> torch.Tensor:
248 - qkv, _ = self.qkv(x)
249 - q, k, v = qkv.chunk(3, dim=-1)
250 - out = self.attn(q, k, v)
251 - output, _ = self.o_proj(out)
252 - return output
253 -
254 -
255 -class HunYuanVisionBlock(nn.Module):
256 - def __init__(
257 - self,
258 - dim: int,
259 - num_heads: int,
260 - mlp_hidden_dim: int,
261 - act_fn: Callable[[torch.Tensor], torch.Tensor] = F.gelu,
262 - norm_layer: Callable[[int], nn.Module] | None = None,
263 - quant_config: QuantizationConfig | None = None,
264 - prefix: str = "",
265 - ) -> None:
266 - super().__init__()
267 - if norm_layer is None:
268 - norm_layer = partial(nn.LayerNorm, eps=1e-6)
269 - self.input_layernorm = norm_layer(dim)
270 - self.post_attention_layernorm = norm_layer(dim)
271 - self.self_attn = HunYuanVisionAttention(
272 - embed_dim=dim,
273 - num_heads=num_heads,
274 - projection_size=dim,
275 - quant_config=quant_config,
276 - prefix=f"{prefix}.self_attn",
277 - )
278 - self.mlp = HunYuanVisionMLP(
279 - dim,
280 - mlp_hidden_dim,
281 - act_fn=act_fn,
282 - bias=True,
283 - quant_config=quant_config,
284 - prefix=f"{prefix}.mlp",
285 - )
286 -
287 - def forward(
288 - self,
289 - x: torch.Tensor,
290 - ) -> torch.Tensor:
291 - x = x + self.self_attn(self.input_layernorm(x))
292 - x = x + self.mlp(self.post_attention_layernorm(x))
293 - return x
294 -
295 -
296 -class HunYuanVisionPatchEmbed(nn.Module):
297 - def __init__(self, config: HunYuanVLVisionConfig):
298 - super().__init__()
299 -
300 - self.config = config
301 - self.embed_dim = config.hidden_size
302 - self.patch_size = config.patch_size
303 - self.num_channels = config.num_channels
304 - self.spatial_merge_size = config.spatial_merge_size
305 - self.interpolate_mode = config.interpolate_mode
306 -
307 - self.patch_embedding = nn.Conv2d(
308 - in_channels=config.num_channels,
309 - out_channels=self.embed_dim,
310 - kernel_size=self.patch_size,
311 - stride=self.patch_size,
312 - bias=True,
313 - )
314 -
315 - self.max_num_patches = (config.max_image_size // self.patch_size) ** 2
316 -
317 - self.num_positions = self.max_num_patches + 1
318 - self.position_edge = int(self.num_positions**0.5)
319 - # first token is cls token, skip it
320 - self.position_embedding = nn.Embedding(self.num_positions, self.embed_dim)
321 -
322 - self.patch_pos_embed = None
323 -
324 - def forward(
325 - self, pixel_values: torch.Tensor, grid_thw: list[list[int]]
326 - ) -> torch.Tensor:
327 - num_patches = pixel_values.size(0)
328 - pixel_values = pixel_values.reshape(
329 - num_patches, self.num_channels, self.patch_size, self.patch_size
330 - )
331 -
332 - patch_embeds = self.patch_embedding(pixel_values)
333 - patch_embeds = patch_embeds.squeeze(-1).squeeze(-1).unsqueeze(0)
334 -
335 - if self.patch_pos_embed is None:
336 - patch_pos_shape = (
337 - 1,
338 - self.position_edge,
339 - self.position_edge,
340 - self.embed_dim,
341 - )
342 - self.patch_pos_embed = (
343 - self.position_embedding.weight[1:, :]
344 - .reshape(patch_pos_shape)
345 - .permute(0, 3, 1, 2)
346 - .float()
347 - )
348 -
349 - patch_pos_embed_list = []
350 - for grid in grid_thw:
351 - _, h0, w0 = grid
352 - # we add a small number to avoid floating point error in the interpolation
353 - # see discussion at https://github.com/facebookresearch/dino/issues/8
354 - h0, w0 = h0 + 0.1, w0 + 0.1
355 - patch_pos_embed = nn.functional.interpolate(
356 - self.patch_pos_embed,
357 - scale_factor=(h0 / self.position_edge, w0 / self.position_edge),
358 - mode=self.interpolate_mode,
359 - align_corners=False,
360 - )
361 -
362 - patch_pos_embed = (
363 - patch_pos_embed.reshape(self.embed_dim, -1)
364 - .transpose(0, 1)
365 - .unsqueeze(0)
366 - .to(patch_embeds.dtype)
367 - )
368 - patch_pos_embed_list.append(patch_pos_embed)
369 -
370 - patch_pos_embed = torch.cat(patch_pos_embed_list, dim=1)
371 - embeddings = patch_embeds + patch_pos_embed
372 -
373 - return embeddings
374 -
375 -
376 -class HunYuanVisionPatchMerger(nn.Module):
377 - def __init__(
378 - self,
379 - in_channels,
380 - out_channels,
381 - spatial_merge_size=2,
382 - rms_norm_eps=1e-5,
383 - prefix="",
384 - ):
385 - super().__init__()
386 - self.spatial_merge_size = spatial_merge_size
387 - embed_std = out_channels**-0.5
388 -
389 - self.proj = nn.Sequential(
390 - nn.Conv2d(
391 - in_channels,
392 - in_channels * 2,
393 - kernel_size=spatial_merge_size,
394 - stride=spatial_merge_size,
395 - ),
396 - nn.GELU(),
397 - nn.Conv2d(in_channels * 2, in_channels * 4, kernel_size=1),
398 - )
399 - self.mlp = nn.Linear(in_channels * 4, out_channels)
400 -
401 - self.image_newline = nn.Parameter(torch.randn(in_channels * 4) * embed_std)
402 - self.image_begin = nn.Parameter(torch.randn(out_channels) * embed_std)
403 - self.image_end = nn.Parameter(torch.randn(out_channels) * embed_std)
404 - self.image_sep = nn.Parameter(torch.randn(out_channels) * embed_std)
405 -
406 - self.before_rms = RMSNorm(in_channels, eps=rms_norm_eps)
407 - self.after_rms = RMSNorm(out_channels, eps=rms_norm_eps)
408 -
409 - def forward(self, x, size=(16, 16)):
410 - x = self.before_rms(x)
411 -
412 - h, w = size
413 - dtype = x.dtype
414 - x = x.permute(0, 2, 1).reshape(x.shape[0], -1, h, w)
415 -
416 - x = self.proj(x) # b,c,h,w
417 - b, c, h, w = x.shape
418 - x = torch.cat(
419 - [x, self.image_newline.reshape(1, c, 1, 1).expand(b, c, h, 1).to(dtype)],
420 - dim=-1,
421 - )
422 - x = x.reshape(b, c, -1).permute(0, 2, 1)
423 - x = self.mlp(x)
424 -
425 - begin = self.image_begin.reshape(1, 1, -1).expand(b, 1, x.shape[-1]).to(dtype)
426 - end = self.image_end.reshape(1, 1, -1).expand(b, 1, x.shape[-1]).to(dtype)
427 - x = torch.cat([begin, x, end], dim=1)
428 -
429 - return self.after_rms(x)
430 -
431 -
432 -class HunYuanVisionTransformer(nn.Module):
433 - hf_to_vllm_mapper = WeightsMapper(
434 - orig_to_new_stacked={
435 - ".q_proj": (".qkv", "q"),
436 - ".k_proj": (".qkv", "k"),
437 - ".v_proj": (".qkv", "v"),
438 - }
439 - )
440 -
441 - def __init__(
442 - self,
443 - vision_config: HunYuanVLVisionConfig,
444 - quant_config: QuantizationConfig | None = None,
445 - prefix: str = "",
446 - ) -> None:
447 - super().__init__()
448 -
449 - num_hidden_layers = vision_config.num_hidden_layers
450 - self.hidden_size = vision_config.hidden_size
451 - self.num_heads = vision_config.num_attention_heads
452 - self.spatial_merge_size = vision_config.spatial_merge_size
453 -
454 - from vllm.compilation.backends import set_model_tag
455 -
456 - with set_model_tag("HunYuanVisionPatchEmbed"):
457 - self.embeddings = HunYuanVisionPatchEmbed(vision_config)
458 -
459 - norm_layer = partial(nn.LayerNorm, eps=vision_config.rms_norm_eps)
460 -
461 - with set_model_tag("HunYuanVisionBlock"):
462 - self.layers = nn.ModuleList(
463 - [
464 - HunYuanVisionBlock(
465 - dim=vision_config.hidden_size,
466 - num_heads=vision_config.num_attention_heads,
467 - mlp_hidden_dim=vision_config.intermediate_size,
468 - act_fn=get_act_fn(vision_config.hidden_act),
469 - norm_layer=norm_layer,
470 - quant_config=quant_config,
471 - prefix=f"{prefix}.layers.{layer_idx}",
472 - )
473 - for layer_idx in range(num_hidden_layers)
474 - ]
475 - )
476 -
477 - with set_model_tag("HunYuanVisionPatchMerger"):
478 - self.perceive = HunYuanVisionPatchMerger(
479 - vision_config.hidden_size,
480 - vision_config.out_hidden_size,
481 - spatial_merge_size=vision_config.spatial_merge_size,
482 - rms_norm_eps=vision_config.rms_norm_eps,
483 - prefix=f"{prefix}.perceive",
484 - )
485 -
486 - @property
487 - def dtype(self) -> torch.dtype:
488 - return self.embeddings.patch_embedding.weight.dtype
489 -
490 - @property
491 - def device(self) -> torch.device:
492 - return self.embeddings.patch_embedding.weight.device
493 -
494 - def forward(
495 - self,
496 - x: torch.Tensor,
497 - grid_thw: list[list[int]],
498 - ) -> torch.Tensor:
499 - # patchify
500 - seq_len = x.size(0)
501 - cu_seqlens: list = [0]
502 -
503 - hidden_states = x.to(device=self.device, dtype=self.dtype)
504 - # embeddings = patch_embeds + patch_pos_embed
505 - hidden_states = self.embeddings(hidden_states, grid_thw)
506 -
507 - for t, h, w in grid_thw:
508 - t, h, w = int(t), int(h), int(w)
509 - cu_seqlens.append(h * w)
510 -
511 - cu_seqlens = torch.tensor(cu_seqlens, dtype=torch.int32)
512 - cu_seqlens = torch.cumsum(cu_seqlens, dim=0, dtype=torch.int32)
513 -
514 - cu_seqlens = cu_seqlens.to(device=self.device, non_blocking=True)
515 -
516 - hidden_states = hidden_states.reshape(seq_len, -1)
517 - hidden_states = hidden_states.unsqueeze(0)
518 -
519 - # build per-image lengths once
520 - split_lengths = [int(h) * int(w) for (_, h, w) in grid_thw]
521 - for layer in self.layers:
522 - # hidden_states: (1, T_total, D)
523 - parts = hidden_states.split(split_lengths, dim=1) # list of (1, L_i, D)
524 - parts = [layer(p) for p in parts]
525 - hidden_states = torch.cat(parts, dim=1)
526 -
527 - # adapter
528 - split_lengths = (cu_seqlens[1:] - cu_seqlens[:-1]).tolist()
529 - split_items = hidden_states.split(split_lengths, dim=1)
530 - image_embeds_list = []
531 - for grid, split_item in zip(grid_thw, split_items):
532 - image_embeds_list.append(
533 - self.perceive(split_item.contiguous(), size=grid[1:]).squeeze(0)
534 - )
535 -
536 - return image_embeds_list
537 -
538 - def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
539 - loader = AutoWeightsLoader(self)
540 - return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
541 -
542 -
543 -def _hunyuan_vl_field_config(hf_inputs: Mapping[str, torch.Tensor]):
544 - image_grid_thw = hf_inputs.get("image_grid_thw", torch.empty((0, 3)))
545 - image_grid_sizes = image_grid_thw.prod(-1)
546 - return dict(
547 - pixel_values=MultiModalFieldConfig.flat_from_sizes("image", image_grid_sizes),
548 - image_embeds=MultiModalFieldConfig.flat_from_sizes("image", image_grid_sizes),
549 - image_grid_thw=MultiModalFieldConfig.batched("image", keep_on_cpu=True),
550 - )
551 -
552 -
553 -class HunYuanVLMultiModalDataParser(MultiModalDataParser):
554 - # The patch grid is what sizes the placeholder range.
555 - embedding_fields = {
556 - "image": {"image_embeds": "values", "image_grid_thw": "metadata"},
557 - }
558 -
559 - def _parse_image_data(
560 - self,
561 - data: dict[str, torch.Tensor] | ModalityData[ImageItem],
562 - ) -> ModalityDataItems[Any, Any] | None:
563 - if isinstance(data, dict):
564 - required, optional = self.embedding_field_sets("image")
565 - return DictEmbeddingItems(
566 - data,
567 - modality="image",
568 - required_fields=required,
569 - optional_fields=optional,
570 - fields_factory=_hunyuan_vl_field_config,
571 - )
572 -
573 - return super()._parse_image_data(data)
574 -
575 -
576 -class HunYuanVLProcessingInfo(BaseProcessingInfo):
577 - def get_hf_config(self):
578 - return self.ctx.get_hf_config(HunYuanVLConfig)
579 -
580 - def get_hf_processor(
581 - self,
582 - **kwargs: object,
583 - ) -> HunYuanVLProcessor:
584 - # transformers>=5.13 replaced `use_fast` with `backend`; pin the
585 - # PIL backend to match the released HunyuanOCR checkpoint packing.
586 - kwargs.pop("use_fast", None)
587 - kwargs.setdefault("backend", "pil")
588 - return self.ctx.get_hf_processor(
589 - HunYuanVLProcessor,
590 - **kwargs,
591 - )
592 -
593 - def get_image_processor(
594 - self,
595 - **kwargs: object,
596 - ) -> HunYuanVLImageProcessor:
597 - return self.get_hf_processor(**kwargs).image_processor
598 -
599 - def get_data_parser(self):
600 - return HunYuanVLMultiModalDataParser(
601 - expected_hidden_size=self._get_expected_hidden_size(),
602 - allow_missing_mm_embeddings=self.allow_missing_mm_embeddings,
603 - )
604 -
605 - def get_supported_mm_limits(self) -> Mapping[str, int | None]:
606 - return {"image": None}
607 -
608 - def get_mm_max_tokens_per_item(
609 - self,
610 - seq_len: int,
611 - mm_counts: Mapping[str, int],
612 - ) -> Mapping[str, int]:
613 - max_image_tokens = self.get_max_image_tokens()
614 - # TODO: support video
615 - max_video_tokens = 0
616 - return {"image": max_image_tokens, "video": max_video_tokens}
617 -
618 - def _get_vision_info(
619 - self,
620 - *,
621 - image_width: int,
622 - image_height: int,
623 - num_frames: int = 1,
624 - do_resize: bool = True,
625 - image_processor: HunYuanVLImageProcessor,
626 - mm_kwargs: Mapping[str, object],
627 - ) -> tuple[ImageSize, int]:
628 - hf_config = self.get_hf_config()
629 - vision_config = hf_config.vision_config
630 - patch_size = vision_config.patch_size
631 - spatial_merge_size = vision_config.spatial_merge_size
632 -
633 - mm_kwargs = self.ctx.get_merged_mm_kwargs(mm_kwargs)
634 - size = image_processor.size
635 - if override_size := mm_kwargs.get("size"):
636 - size = size | override_size
637 - if (override_min_pixels := mm_kwargs.get("min_pixels")) is not None:
638 - size = size | {"shortest_edge": override_min_pixels}
639 - if (override_max_pixels := mm_kwargs.get("max_pixels")) is not None:
640 - size = size | {"longest_edge": override_max_pixels}
641 -
642 - if do_resize:
643 - resized_height, resized_width = smart_resize(
644 - height=image_height,
645 - width=image_width,
646 - factor=patch_size * spatial_merge_size,
647 - min_pixels=size["shortest_edge"],
648 - max_pixels=size["longest_edge"],
649 - )
650 - preprocessed_size = ImageSize(width=resized_width, height=resized_height)
651 - else:
652 - preprocessed_size = ImageSize(width=image_width, height=image_height)
653 -
654 - grid_t = 1
655 - grid_h = preprocessed_size.height // patch_size
656 - grid_w = preprocessed_size.width // patch_size
657 -
658 - num_vision_tokens = (
659 - grid_t * grid_h // spatial_merge_size * (grid_w // spatial_merge_size + 1)
660 - + 2
661 - )
662 -
663 - return preprocessed_size, num_vision_tokens
664 -
665 - def get_num_image_tokens(
666 - self,
667 - *,
668 - image_width: int,
669 - image_height: int,
670 - image_processor: HunYuanVLImageProcessor,
671 - mm_kwargs: Mapping[str, object],
672 - ) -> int:
673 - _, num_image_tokens = self._get_vision_info(
674 - image_width=image_width,
675 - image_height=image_height,
676 - image_processor=image_processor,
677 - mm_kwargs=mm_kwargs,
678 - )
679 - return num_image_tokens
680 -
681 - def get_image_size_with_most_features(self) -> ImageSize:
682 - image_processor = self.get_image_processor()
683 -
684 - max_image_size, _ = self._get_vision_info(
685 - image_width=512,
686 - image_height=8192,
687 - image_processor=image_processor,
688 - mm_kwargs={},
689 - )
690 - return max_image_size
691 -
692 - def get_max_image_tokens(self) -> int:
693 - image_processor = self.get_image_processor()
694 - target_width, target_height = self.get_image_size_with_most_features()
695 -
696 - return self.get_num_image_tokens(
697 - image_width=target_width,
698 - image_height=target_height,
699 - image_processor=image_processor,
700 - mm_kwargs={},
701 - )
702 -
703 -
704 -class HunYuanVLDummyInputsBuilder(BaseDummyInputsBuilder[HunYuanVLProcessingInfo]):
705 - def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
706 - num_images = mm_counts.get("image", 0)
707 -
708 - hf_processor = self.info.get_hf_processor(typ=HunYuanVLProcessor)
709 - image_placeholder = (
710 - f"{hf_processor.image_start_token}{hf_processor.image_token}"
711 - f"{hf_processor.image_end_token}"
712 - )
713 -
714 - return image_placeholder * num_images
715 -
716 - def get_dummy_mm_data(
717 - self,
718 - seq_len: int,
719 - mm_counts: Mapping[str, int],
720 - mm_options: Mapping[str, BaseDummyOptions],
721 - ) -> MultiModalDataDict:
722 - num_images = mm_counts.get("image", 1)
723 -
724 - target_width, target_height = self.info.get_image_size_with_most_features()
725 -
726 - return {
727 - "image": self._get_dummy_images(
728 - width=target_width, height=target_height, num_images=num_images
729 - ),
730 - }
731 -
732 -
733 -class HunYuanVLMultiModalProcessor(BaseMultiModalProcessor[HunYuanVLProcessingInfo]):
734 - def _call_hf_processor(
735 - self,
736 - prompt: str,
737 - mm_data: Mapping[str, object],
738 - mm_kwargs: Mapping[str, object],
739 - ) -> BatchFeature:
740 - hf_processor = self.info.get_hf_processor(**mm_kwargs)
741 - # HunYuanVLProcessor requires image placeholders wrapped with start/end tokens.
742 - if mm_data.get("images") is not None and prompt:
743 - img_tok = hf_processor.image_token
744 - wrapped = (
745 - f"{hf_processor.image_start_token}{img_tok}"
746 - f"{hf_processor.image_end_token}"
747 - )
748 - if img_tok in prompt and wrapped not in prompt:
749 - prompt = prompt.replace(img_tok, wrapped)
750 - return self.info.ctx.call_hf_processor(
751 - hf_processor,
752 - dict(text=prompt, **mm_data),
753 - mm_kwargs,
754 - )
755 -
756 - def _get_prompt_updates(
757 - self,
758 - mm_items: MultiModalDataItems,
759 - hf_processor_mm_kwargs: Mapping[str, Any],
760 - out_mm_kwargs: MultiModalKwargsItems,
761 - ) -> Sequence[PromptUpdate]:
762 - hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
763 - image_processor = self.info.get_image_processor(**hf_processor_mm_kwargs)
764 -
765 - token_ids = {
766 - "image": hf_processor.image_token_id,
767 - "image_start": hf_processor.image_start_token_id,
768 - "image_end": hf_processor.image_end_token_id,
769 - }
770 -
771 - merge_size = image_processor.merge_size
772 -
773 - def get_replacement_hunyuan_vl(item_idx: int, modality: str):
774 - out_item = out_mm_kwargs[modality][item_idx]
775 - grid_thw = out_item[f"{modality}_grid_thw"].data
776 - assert isinstance(grid_thw, torch.Tensor)
777 -
778 - _, grid_h, grid_w = grid_thw
779 - num_tokens = (int(grid_h) // merge_size) * (
780 - int(grid_w) // merge_size + 1
781 - ) + 2
782 - tokens = (
783 - [token_ids[f"{modality}_start"]]
784 - + [token_ids[modality]] * num_tokens
785 - + [token_ids[f"{modality}_end"]]
786 - )
787 - return PromptUpdateDetails.select_token_id(tokens, token_ids[modality])
788 -
789 - return [
790 - PromptReplacement(
791 - modality=modality,
792 - target=[
793 - token_ids[f"{modality}_start"],
794 - token_ids[modality],
795 - token_ids[f"{modality}_end"],
796 - ],
797 - replacement=partial(get_replacement_hunyuan_vl, modality=modality),
798 - )
799 - for modality in ("image",)
800 - ]
801 -
802 - def _get_mm_fields_config(
803 - self,
804 - hf_inputs: BatchFeature,
805 - hf_processor_mm_kwargs: Mapping[str, object],
806 - ) -> Mapping[str, MultiModalFieldConfig]:
807 - return _hunyuan_vl_field_config(hf_inputs)
808 -
809 -
810 -@MULTIMODAL_REGISTRY.register_processor(
811 - HunYuanVLMultiModalProcessor,
812 - info=HunYuanVLProcessingInfo,
813 - dummy_inputs=HunYuanVLDummyInputsBuilder,
814 -)
815 -class HunYuanVLForConditionalGeneration(
816 - nn.Module,
817 - SupportsMultiModal,
818 - SupportsLoRA,
819 - SupportsPP,
820 - SupportsQuant,
821 - SupportsXDRoPE,
822 - SupportsEagle,
823 - SupportsEagle3,
824 -):
825 - # To ensure correct weight loading and mapping.
826 - hf_to_vllm_mapper = WeightsMapper(
827 - orig_to_new_prefix={
828 - # mapping for new names in checkpoint saved after transformers v4.52
829 - "vit.vit.": "visual.",
830 - "vit.": "visual.",
831 - "model.": "language_model.model.",
832 - }
833 - )
834 -
835 - supports_encoder_tp_data = True
836 -
837 - def get_xdrope_input_positions(
838 - self,
839 - input_tokens: list[int],
840 - mm_features: list[MultiModalFeatureSpec],
841 - ) -> torch.Tensor:
842 - kwargs = MultiModalFeatureSpec.gather_kwargs(
843 - mm_features,
844 - {"image_grid_thw"},
845 - )
846 - image_grid_thw = [item.tolist() for item in kwargs.get("image_grid_thw", [])]
847 -
848 - hf_config = self.config
849 - image_start_token_id = hf_config.image_start_token_id
850 - spatial_merge_size = hf_config.vision_config.spatial_merge_size
851 - xd_num = len(hf_config.rope_scaling["xdrope_section"])
852 -
853 - input_tokens_tensor = torch.tensor(input_tokens)
854 - image_start_indices = torch.argwhere(
855 - input_tokens_tensor == image_start_token_id
856 - ).squeeze(1)
857 -
858 - p_index = torch.arange(len(input_tokens_tensor))
859 - w_index = torch.arange(len(input_tokens_tensor))
860 - h_index = torch.arange(len(input_tokens_tensor))
861 - t_index = torch.arange(len(input_tokens_tensor))
862 - for image_index in range(len(image_start_indices)):
863 - # +1 : first image_token, +2: for xdrope positions
864 - pos = image_start_indices[image_index] + 2
865 - t, h, w = image_grid_thw[image_index]
866 - _, llm_grid_h, llm_grid_w = (
867 - t,
868 - h // spatial_merge_size,
869 - w // spatial_merge_size,
870 - )
871 -
872 - token_num = (llm_grid_w + 1) * llm_grid_h
873 - w_index[pos : pos + token_num].copy_(
874 - torch.arange(0, llm_grid_w + 1)
875 - .reshape(1, -1)
876 - .expand(llm_grid_h, -1)
877 - .reshape(-1)
878 - )
879 - h_index[pos : pos + token_num].copy_(
880 - torch.arange(0, llm_grid_h)
881 - .reshape(-1, 1)
882 - .expand(-1, llm_grid_w + 1)
883 - .reshape(-1)
884 - )
885 - t_index[pos : pos + token_num] = image_index
886 -
887 - if xd_num == 4:
888 - llm_positions = torch.stack([p_index, w_index, h_index, t_index])
889 - elif xd_num == 3:
890 - llm_positions = torch.stack([w_index, h_index, t_index])
891 -
892 - return llm_positions
893 -
894 - @classmethod
895 - def get_placeholder_str(cls, modality: str, i: int) -> str | None:
896 - if modality.startswith("image"):
897 - return "<｜hy_place▁holder▁no▁100｜><｜hy_place▁holder▁no▁102｜><｜hy_place▁holder▁no▁101｜>" # noqa: E501
898 -
899 - raise ValueError("Only image modality is supported")
900 -
901 - def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
902 - super().__init__()
903 - config: HunYuanVLConfig = vllm_config.model_config.hf_config
904 -
905 - self.config = config
906 -
907 - with self._mark_tower_model(vllm_config, {"image"}):
908 - self.visual = HunYuanVisionTransformer(
909 - config.vision_config,
910 - quant_config=vllm_config.quant_config,
911 - prefix=maybe_prefix(prefix, "visual"),
912 - )
913 -
914 - with self._mark_language_model(vllm_config):
915 - self.language_model = init_vllm_registered_model(
916 - vllm_config=vllm_config,
917 - prefix=maybe_prefix(prefix, "language_model.model"),
918 - architectures=[
919 - "HunYuanDenseV1ForCausalLM",
920 - "HunYuanMoEV1ForCausalLM",
921 - ],
922 - )
923 -
924 - self.make_empty_intermediate_tensors = (
925 - self.language_model.make_empty_intermediate_tensors
926 - )
927 -
928 - def _parse_and_validate_image_input(
929 - self, **kwargs: object
930 - ) -> HunYuanVLImageInputs | None:
931 - pixel_values = kwargs.pop("pixel_values", None)
932 - image_embeds = kwargs.pop("image_embeds", None)
933 - image_grid_thw = kwargs.pop("image_grid_thw", None)
934 -
935 - if pixel_values is None and image_embeds is None:
936 - return None
937 -
938 - # TODO: refine
939 - if isinstance(pixel_values, list):
940 - pixel_values = torch.cat(pixel_values, dim=0)
941 - if len(pixel_values.shape) == 3:
942 - last_dim = pixel_values.shape[-1]
943 - pixel_values = pixel_values.reshape(-1, last_dim)
944 - image_grid_thw = image_grid_thw.reshape(-1, 3)
945 -
946 - if pixel_values is not None:
947 - return HunYuanVLImagePixelInputs(
948 - type="pixel_values",
949 - pixel_values=pixel_values,
950 - image_grid_thw=image_grid_thw,
951 - )
952 -
953 - if image_embeds is not None:
954 - return HunYuanVLImageEmbeddingInputs(
955 - type="image_embeds",
956 - image_embeds=image_embeds,
957 - image_grid_thw=image_grid_thw,
958 - )
959 -
960 - def _process_image_input(
961 - self, image_input: HunYuanVLImageInputs
962 - ) -> tuple[torch.Tensor, ...]:
963 - grid_thw = image_input["image_grid_thw"]
964 - assert grid_thw.ndim == 2
965 - grid_thw_list = grid_thw.tolist()
966 -
967 - if image_input["type"] == "image_embeds":
968 - image_embeds = image_input["image_embeds"].type(self.visual.dtype)
969 - else:
970 - pixel_values = image_input["pixel_values"]
971 -
972 - # TODO: use_data_parallel (split image_embeds in visual)
973 - image_embeds = self.visual(pixel_values, grid_thw=grid_thw_list)
974 -
975 - return image_embeds
976 -
977 - def _parse_and_validate_multimodal_inputs(self, **kwargs: object) -> dict:
978 - mm_input_by_modality = {}
979 -
980 - # Preserve the order of modalities if there are multiple of them
981 - # from the order of kwargs.
982 - for input_key in kwargs:
983 - if (
984 - input_key in ("pixel_values", "image_embeds")
985 - and "image" not in mm_input_by_modality
986 - ):
987 - mm_input_by_modality["image"] = self._parse_and_validate_image_input(
988 - **kwargs
989 - )
990 - return mm_input_by_modality
991 -
992 - def embed_multimodal(self, **kwargs: object) -> MultiModalEmbeddings:
993 - mm_input_by_modality = self._parse_and_validate_multimodal_inputs(**kwargs)
994 - if not mm_input_by_modality:
995 - return []
996 -
997 - # The result multimodal_embeddings is tuple of tensors, with each
998 - # tensor correspoending to a multimodal data item (image or video).
999 - multimodal_embeddings: tuple[torch.Tensor, ...] = ()
1000 -
1001 - # NOTE: It is important to iterate over the keys in this dictionary
1002 - # to preserve the order of the modalities.
1003 - for modality in mm_input_by_modality:
1004 - multimodal_input = mm_input_by_modality[modality]
1005 - if modality == "image":
1006 - image_embeddings = self._process_image_input(multimodal_input)
1007 - multimodal_embeddings += tuple(image_embeddings)
1008 - return multimodal_embeddings
1009 -
1010 - def forward(
1011 - self,
1012 - input_ids: torch.Tensor | None,
1013 - positions: torch.Tensor,
1014 - intermediate_tensors: IntermediateTensors | None,
1015 - inputs_embeds: torch.Tensor | None,
1016 - **kwargs: object,
1017 - ) -> torch.Tensor | IntermediateTensors:
1018 - if intermediate_tensors is not None:
1019 - inputs_embeds = None
1020 -
1021 - hidden_states = self.language_model(
1022 - input_ids=input_ids,
1023 - positions=positions,
1024 - intermediate_tensors=intermediate_tensors,
1025 - inputs_embeds=inputs_embeds,
1026 - )
1027 - return hidden_states
1028 -
1029 - def compute_logits(
1030 - self,
1031 - hidden_states: torch.Tensor,
1032 - ) -> torch.Tensor | None:
1033 - return self.language_model.compute_logits(hidden_states)
1034 -
1035 - def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
1036 - loader = AutoWeightsLoader(self)
1037 - return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
1038 -
1039 - def get_mm_mapping(self) -> MultiModelKeys:
1040 - """
1041 - Get the module prefix in multimodal models
1042 - """
1043 - return MultiModelKeys.from_string_field(
1044 - language_model="language_model.model",
1045 - connector="visual.perceive",
1046 - tower_model="visual",
1047 - )
vllm/model_executor/models/registry.py
+6 -6
@@ -131,8 +131,6 @@ _TEXT_GENERATION_MODELS = {
131 "GraniteSWAForCausalLM": ("granite", "GraniteForCausalLM"),
132 "GritLM": ("gritlm", "GritLM"),
133 "HrmTextForCausalLM": ("hrm_text", "HrmTextForCausalLM"),
134 - "HunYuanMoEV1ForCausalLM": ("hunyuan_v1", "HunYuanMoEV1ForCausalLM"),
135 - "HunYuanDenseV1ForCausalLM": ("hunyuan_v1", "HunYuanDenseV1ForCausalLM"),
134 "HYV3ForCausalLM": ("hy_v3", "HYV3ForCausalLM"),
135 "HCXVisionForCausalLM": ("hyperclovax_vision", "HCXVisionForCausalLM"),
136 "HCXVisionV2ForCausalLM": ("hyperclovax_vision_v2", "HCXVisionV2ForCausalLM"),
@@ -441,10 +439,6 @@ _MULTIMODAL_MODELS = {
439 "Granite4VisionForConditionalGeneration",
440 ),
441 "H2OVLChatModel": ("h2ovl", "H2OVLChatModel"),
444 - "HunYuanVLForConditionalGeneration": (
445 - "hunyuan_vision",
446 - "HunYuanVLForConditionalGeneration",
447 - ),
442 "InternVLChatModel": ("internvl", "InternVLChatModel"),
443 "InternS1ForConditionalGeneration": (
444 "interns1",
@@ -700,6 +694,8 @@ _SPECULATIVE_DECODING_MODELS = {
694 _TRANSFORMERS_SUPPORTED_MODELS = {
695 # Text generation models
696 "GPTBigCodeForCausalLM": ("transformers", "TransformersForCausalLM"),
697 + "HunYuanDenseV1ForCausalLM": ("transformers", "TransformersForCausalLM"),
698 + "HunYuanMoEV1ForCausalLM": ("transformers", "TransformersMoEForCausalLM"),
699 "OlmoForCausalLM": ("transformers", "TransformersForCausalLM"),
700 "Olmo2ForCausalLM": ("transformers", "TransformersForCausalLM"),
701 "SmolLM3ForCausalLM": ("transformers", "TransformersForCausalLM"),
@@ -710,6 +706,10 @@ _TRANSFORMERS_SUPPORTED_MODELS = {
706 "transformers",
707 "TransformersMultiModalForCausalLM",
708 ),
709 + "HunYuanVLForConditionalGeneration": (
710 + "transformers",
711 + "TransformersMultiModalForCausalLM",
712 + ),
713 "VibeVoiceAsrForConditionalGeneration": (
714 "transformers",
715 "TransformersMultiModalForCausalLM",
vllm/transformers_utils/config.py
-1
@@ -95,7 +95,6 @@ _CONFIG_REGISTRY: dict[str, type[PretrainedConfig]] = LazyConfigDict(
95 granite4_vision="Granite4VisionConfig",
96 hyperclovax="HyperCLOVAXConfig",
97 hyperclovax_vlm="HCXVisionConfig",
98 - hunyuan_vl="HunYuanVLConfig",
98 hy_v3="HYV3Config",
99 isaac="IsaacConfig",
100 kimi_k2="DeepseekV3Config", # Kimi K2 uses same architecture as DeepSeek V3
vllm/transformers_utils/configs/__init__.py
-6
@@ -44,9 +44,6 @@ _CLASS_TO_MODULE: dict[str, str] = {
44 "FunAudioChatConfig": "vllm.transformers_utils.configs.funaudiochat",
45 "FunAudioChatAudioEncoderConfig": "vllm.transformers_utils.configs.funaudiochat",
46 "Granite4VisionConfig": "vllm.transformers_utils.configs.granite4_vision",
47 - "HunYuanVLConfig": "vllm.transformers_utils.configs.hunyuan_vl",
48 - "HunYuanVLTextConfig": "vllm.transformers_utils.configs.hunyuan_vl",
49 - "HunYuanVLVisionConfig": "vllm.transformers_utils.configs.hunyuan_vl",
47 "HCXVisionConfig": "vllm.transformers_utils.configs.hyperclovax",
48 "HYV3Config": "vllm.transformers_utils.configs.hy_v3",
49 "HyperCLOVAXConfig": "vllm.transformers_utils.configs.hyperclovax",
@@ -141,9 +138,6 @@ __all__ = [
138 "FunAudioChatConfig",
139 "FunAudioChatAudioEncoderConfig",
140 "Granite4VisionConfig",
144 - "HunYuanVLConfig",
145 - "HunYuanVLTextConfig",
146 - "HunYuanVLVisionConfig",
141 "HCXVisionConfig",
142 "HYV3Config",
143 "HyperCLOVAXConfig",
vllm/transformers_utils/configs/hunyuan_vl.py deleted
-281
@@ -1,281 +0,0 @@
1 -# SPDX-License-Identifier: Apache-2.0
2 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 -
4 -# adapted from https://github.com/ManaEstras/transformers/blob/v4.57.1.hyvl/src/transformers/models/hunyuan_vl/configuration_hunyuan_vl.py
5 -
6 -from transformers import PretrainedConfig
7 -
8 -
9 -class HunYuanVLVisionConfig(PretrainedConfig):
10 - model_type = "hunyuan_vl"
11 - base_config_key = "vision_config"
12 -
13 - def __init__(
14 - self,
15 - hidden_act="gelu",
16 - hidden_size=1152,
17 - intermediate_size=4304,
18 - interpolate_mode="bilinear",
19 - rms_norm_eps=1e-05,
20 - learnable_mlp_pooling_size=0,
21 - num_attention_heads=16,
22 - num_key_value_heads=None,
23 - num_channels=3,
24 - num_hidden_layers=27,
25 - out_hidden_size=4096,
26 - patch_size=16,
27 - remove_prenorm=True,
28 - spatial_merge_size=2,
29 - temporal_patch_size=1,
30 - resize_resolution=2048,
31 - img_max_token_num=4096,
32 - max_image_size=2048,
33 - video_max_image_size=768,
34 - video_min_image_size=256,
35 - min_image_size=512,
36 - anyres_vit_max_image_size=2048,
37 - max_vit_seq_len=16384,
38 - text_hidden_size=3072,
39 - **kwargs,
40 - ):
41 - super().__init__(**kwargs)
42 -
43 - self.hidden_act = hidden_act
44 - self.hidden_size = hidden_size
45 - self.intermediate_size = intermediate_size
46 - self.interpolate_mode = interpolate_mode
47 - self.learnable_mlp_pooling_size = learnable_mlp_pooling_size
48 - self.num_attention_heads = num_attention_heads
49 - if not num_key_value_heads:
50 - self.num_key_value_heads = num_attention_heads
51 - else:
52 - self.num_key_value_heads = num_key_value_heads
53 - self.num_channels = num_channels
54 - self.num_hidden_layers = num_hidden_layers
55 - self.out_hidden_size = out_hidden_size
56 - self.patch_size = patch_size
57 - self.remove_prenorm = remove_prenorm
58 - self.spatial_merge_size = spatial_merge_size
59 - self.temporal_patch_size = temporal_patch_size
60 - self.rms_norm_eps = rms_norm_eps
61 -
62 - self.resize_resolution = resize_resolution
63 - self.img_max_token_num = img_max_token_num
64 - self.max_image_size = max_image_size
65 - self.min_image_size = min_image_size
66 - self.video_max_image_size = video_max_image_size
67 - self.video_min_image_size = video_min_image_size
68 - self.anyres_vit_max_image_size = anyres_vit_max_image_size
69 - self.max_vit_seq_len = max_vit_seq_len
70 - self.text_hidden_size = text_hidden_size
71 -
72 -
73 -class HunYuanVLTextConfig(PretrainedConfig):
74 - r"""
75 - This is the configuration class to store the configuration of a [`HunYuanVLTextConfig`]. It is used to instantiate an
76 - HunYuan model according to the specified arguments, defining the model architecture. Instantiating a configuration
77 - with the defaults will yield a similar configuration to that of the HunYuan-7B.
78 - Hunyuan-7B-Instruct [tencent/Hunyuan-7B-Instruct](https://huggingface.co/tencent/Hunyuan-7B-Instruct).
79 -
80 - Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
81 - documentation from [`PretrainedConfig`] for more information.
82 -
83 -
84 - Args:
85 - vocab_size (`int`, *optional*, defaults to 290943):
86 - Vocabulary size of the HunYuan model. Defines the number of different tokens that can be represented by the
87 - `inputs_ids` passed when calling [`HunYuanVLTextConfig`]
88 - hidden_size (`int`, *optional*, defaults to 4096):
89 - Dimension of the hidden representations.
90 - intermediate_size (`int`, *optional*, defaults to 11008):
91 - Dimension of the MLP representations or shared MLP representations.
92 - num_hidden_layers (`int`, *optional*, defaults to 32):
93 - Number of hidden layers in the Transformer decoder.
94 - num_attention_heads (`int`, *optional*, defaults to 32):
95 - Number of attention heads for each attention layer in the Transformer decoder.
96 - num_key_value_heads (`int`, *optional*):
97 - This is the number of key_value heads that should be used to implement Grouped Query Attention. If
98 - `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
99 - `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
100 - converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
101 - by meanpooling all the original heads within that group. For more details checkout [this
102 - paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to
103 - `num_attention_heads`.
104 - hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
105 - The non-linear activation function (function or string) in the decoder.
106 - max_position_embeddings (`int`, *optional*, defaults to 2048):
107 - The maximum sequence length that this model might ever be used with.
108 - initializer_range (`float`, *optional*, defaults to 0.02):
109 - The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
110 - rms_norm_eps (`float`, *optional*, defaults to 1e-05):
111 - The epsilon used by the rms normalization layers.
112 - use_cache (`bool`, *optional*, defaults to `True`):
113 - Whether or not the model should return the last key/values attentions (not used by all models). Only
114 - relevant if `config.is_decoder=True`.
115 - pad_token_id (`int`, *optional*, defaults to 0):
116 - Padding token id.
117 - bos_token_id (`int`, *optional*, defaults to 1):
118 - Beginning of stream token id.
119 - eos_token_id (`int`, *optional*, defaults to 2):
120 - End of stream token id.
121 - eod_token_id (int, *optional*, defaults to 3):
122 - Token ID representing the end-of-document marker. Used to indicate the termination of a text sequence.
123 - Example: In multi-document processing, this token helps the model distinguish between separate documents.
124 - pretraining_tp (`int`, *optional*, defaults to 1):
125 - Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this
126 - document](https://huggingface.co/docs/transformers/parallelism) to understand more about it. This value is
127 - necessary to ensure exact reproducibility of the pretraining results. Please refer to [this
128 - issue](https://github.com/pytorch/pytorch/issues/76232).
129 - tie_word_embeddings (`bool`, *optional*, defaults to `False`):
130 - Whether to tie weight embeddings
131 - rope_theta (`float`, *optional*, defaults to 10000.0):
132 - The base period of the RoPE embeddings.
133 - rope_scaling (`Dict`, *optional*):
134 - Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
135 - strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
136 - `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
137 - `max_position_embeddings` to the expected new maximum. See the following thread for more information on how
138 - these scaling strategies behave:
139 - https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/. This is an
140 - experimental feature, subject to breaking API changes in future versions.
141 - attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
142 - Whether to use a bias in the query, key, value and output projection layers during self-attention.
143 - attention_dropout (`float`, *optional*, defaults to 0.0):
144 - The dropout ratio for the attention probabilities.
145 - head_dim (`int`, *optional*, defaults to 128):
146 - The attention head dimension.
147 - """ # noqa: E501
148 -
149 - model_type = "hunyuan_vl_text"
150 - keys_to_ignore_at_inference = ["past_key_values"]
151 -
152 - def __init__(
153 - self,
154 - vocab_size=290943,
155 - hidden_size=4096,
156 - intermediate_size: int = 11008,
157 - num_hidden_layers=32,
158 - num_attention_heads=32,
159 - num_key_value_heads=None,
160 - hidden_act="silu",
161 - max_position_embeddings=2048,
162 - initializer_range=0.02,
163 - rms_norm_eps=1e-5,
164 - use_cache=True,
165 - pad_token_id=0,
166 - bos_token_id=1,
167 - eos_token_id=2,
168 - eod_token_id=3,
169 - pretraining_tp=1,
170 - tie_word_embeddings=False,
171 - rope_theta=10000.0,
172 - rope_scaling=None,
173 - attention_bias=False,
174 - attention_dropout=0.0,
175 - head_dim=None,
176 - **kwargs,
177 - ):
178 - self.vocab_size = vocab_size
179 - self.max_position_embeddings = max_position_embeddings
180 - self.hidden_size = hidden_size
181 - self.intermediate_size = intermediate_size
182 - self.num_hidden_layers = num_hidden_layers
183 - self.num_attention_heads = num_attention_heads
184 - self.head_dim = head_dim
185 - # for backward compatibility
186 - if num_key_value_heads is None:
187 - num_key_value_heads = num_attention_heads
188 -
189 - self.num_key_value_heads = num_key_value_heads
190 - self.hidden_act = hidden_act
191 - self.initializer_range = initializer_range
192 - self.rms_norm_eps = rms_norm_eps
193 - self.pretraining_tp = pretraining_tp
194 - self.use_cache = use_cache
195 - self.rope_theta = rope_theta
196 - self.rope_scaling = rope_scaling
197 - self.attention_bias = attention_bias
198 - self.attention_dropout = attention_dropout
199 -
200 - super().__init__(
201 - pad_token_id=pad_token_id,
202 - bos_token_id=bos_token_id,
203 - eos_token_id=eos_token_id,
204 - tie_word_embeddings=tie_word_embeddings,
205 - **kwargs,
206 - )
207 -
208 -
209 -class HunYuanVLConfig(PretrainedConfig):
210 - model_type = "hunyuan_vl"
211 - sub_configs = {
212 - "vision_config": HunYuanVLVisionConfig,
213 - "text_config": HunYuanVLTextConfig,
214 - }
215 - keys_to_ignore_at_inference = ["past_key_values"]
216 -
217 - def __init__(
218 - self,
219 - text_config=None,
220 - vision_config=None,
221 - im_start_id=120118,
222 - im_end_id=120119,
223 - image_token_id=120120,
224 - im_newline_id=120121,
225 - video_start_id=120122,
226 - video_end_id=120123,
227 - **kwargs,
228 - ):
229 - # We need to init super() here so that it does not reset values
230 - # that are in text config to the BaseClass defaults. The Base
231 - # config has many text related defaults and not all defaults are
232 - # same as for `HunYuanVLTextConfig`.
233 - super().__init__(**kwargs)
234 -
235 - if isinstance(vision_config, dict):
236 - self.vision_config = self.sub_configs["vision_config"](**vision_config)
237 - elif vision_config is None:
238 - self.vision_config = self.sub_configs["vision_config"]()
239 -
240 - if isinstance(text_config, dict):
241 - self.text_config = self.sub_configs["text_config"](**text_config)
242 - elif text_config is None:
243 - # For BC use all kwargs to init `TextConfig`
244 - self.text_config = self.sub_configs["text_config"](**kwargs)
245 -
246 - self.image_token_id = image_token_id
247 - self.im_start_id = im_start_id
248 - self.im_end_id = im_end_id
249 - self.im_newline_id = im_newline_id
250 - self.video_start_id = video_start_id
251 - self.video_end_id = video_end_id
252 -
253 - self.vision_config.text_hidden_size = self.text_config.hidden_size
254 -
255 - # Attention implementation to use. It sets it recursively on sub-configs
256 - # so we call it again in the end.
257 - self._attn_implementation = kwargs.pop("attn_implementation", None)
258 -
259 - def __setattr__(self, key, value):
260 - if (
261 - (text_config := super().__getattribute__("__dict__").get("text_config"))
262 - is not None
263 - and key not in ["dtype", "_attn_implementation_internal"]
264 - and key in text_config.__dict__
265 - ):
266 - setattr(text_config, key, value)
267 - else:
268 - super().__setattr__(key, value)
269 -
270 - def __getattribute__(self, key):
271 - if "text_config" in super().__getattribute__("__dict__") and key not in [
272 - "_name_or_path",
273 - "model_type",
274 - "dtype",
275 - "_attn_implementation_internal",
276 - ]:
277 - text_config = super().__getattribute__("text_config")
278 - if key in text_config.__dict__:
279 - return getattr(text_config, key)
280 -
281 - return super().__getattribute__(key)