[Model] Remove ten deprecated model architectures (#53608)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Xianbao QIAN <xianbao.qian@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Harry Mellor committed
Aug 24, 2026 at 17:06 UTC
7ca336929c169fee1210dd5293029d78811fba27
46 files changed
+49
-6620
docs/contributing/model/multimodal.md
-1
@@ -719,7 +719,6 @@ Examples:
719
720
Examples:
721
722
-- Chameleon (appends `sep_token`): [vllm/model_executor/models/chameleon.py](../../../vllm/model_executor/models/chameleon.py)
722
- Molmo2 (prepends `bos_token`): [vllm/model_executor/models/molmo2.py](../../../vllm/model_executor/models/molmo2.py)
723
- Molmo (applies chat template which is not defined elsewhere): [vllm/model_executor/models/molmo.py](../../../vllm/model_executor/models/molmo.py)
724
docs/models/pooling_models/embed.md
-1
@@ -42,7 +42,6 @@ You can compute pairwise similarity scores to build a similarity matrix using th
42
| `BgeM3EmbeddingModel` | BGE-M3 | `BAAI/bge-m3` | | |
43
| `Gemma2Model`<sup>C</sup> | Gemma 2-based | `BAAI/bge-multilingual-gemma2`, etc. | ✅︎ | ✅︎ |
44
| `Gemma3TextModel`<sup>C</sup> | Gemma 3-based | `google/embeddinggemma-300m`, etc. | ✅︎ | ✅︎ |
45
-| `GritLM` | GritLM | `parasail-ai/GritLM-7B-vllm`. | ✅︎ | ✅︎ |
45
| `GteModel` | Arctic-Embed-2.0-M | `Snowflake/snowflake-arctic-embed-m-v2.0`. | | |
46
| `GteNewModel` | mGTE-TRM (see note) | `Alibaba-NLP/gte-multilingual-base`, etc. | | |
47
| `JinaEmbeddingsV5Model`<sup>C</sup> | Qwen3-decoder or EuroBERT-encoder backbone with task-specific LoRA adapters | `jinaai/jina-embeddings-v5-text-small`, `jinaai/jina-embeddings-v5-text-nano` (see note) | ✅︎ | ✅︎ |
docs/models/supported_models.md
-10
@@ -358,7 +358,6 @@ th {
358
| `AfmoeForCausalLM` | Afmoe | TBA | ✅︎ | ✅︎ |
359
| `ApertusForCausalLM` | Apertus | `swiss-ai/Apertus-8B-2509`, `swiss-ai/Apertus-70B-Instruct-2509`, etc. | ✅︎ | ✅︎ |
360
| `ArceeForCausalLM` | Arcee (AFM) | `arcee-ai/AFM-4.5B-Base`, etc. | ✅︎ | ✅︎ |
361
-| `ArcticForCausalLM` | Arctic | `Snowflake/snowflake-arctic-base`, `Snowflake/snowflake-arctic-instruct`, etc. | | ✅︎ |
361
| `AXK1ForCausalLM` | A.X-K1 | `skt/A.X-K1`, etc. | | ✅︎ |
362
| `BailingMoeForCausalLM` | Ling | `inclusionAI/Ling-lite-1.5`, `inclusionAI/Ling-plus`, etc. | ✅︎ | ✅︎ |
363
| `BailingMoeV2ForCausalLM` | Ling | `inclusionAI/Ling-mini-2.0`, etc. | ✅︎ | ✅︎ |
@@ -382,7 +381,6 @@ th {
381
| `ExaoneForCausalLM` | EXAONE-3 | `LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc. | ✅︎ | ✅︎ |
382
| `ExaoneMoeForCausalLM` | K-EXAONE, K-EXAONE-2 | `LGAI-EXAONE/K-EXAONE-236B-A23B`, `LGAI-EXAONE/K-EXAONE-2.0-750B-A37B`, etc. | | |
383
| `Exaone4ForCausalLM` | EXAONE-4 | `LGAI-EXAONE/EXAONE-4.0-32B`, etc. | ✅︎ | ✅︎ |
385
-| `Fairseq2LlamaForCausalLM` | Llama (fairseq2 format) | `mgleize/fairseq2-dummy-Llama-3.2-1B`, etc. | ✅︎ | ✅︎ |
384
| `FalconForCausalLM` | Falcon | `tiiuae/falcon-7b`, `tiiuae/falcon-40b`, `tiiuae/falcon-rw-7b`, etc. | | ✅︎ |
385
| `FalconMambaForCausalLM` | FalconMamba | `tiiuae/falcon-mamba-7b`, `tiiuae/falcon-mamba-7b-instruct`, etc. | | ✅︎ |
386
| `FalconH1ForCausalLM` | Falcon-H1 | `tiiuae/Falcon-H1-34B-Base`, `tiiuae/Falcon-H1-34B-Instruct`, etc. | ✅︎ | ✅︎ |
@@ -407,7 +405,6 @@ th {
405
| `GraniteMoeSharedForCausalLM` | Granite MoE Shared | `ibm-research/moe-7b-1b-active-shared-experts` (test model) | ✅︎ | ✅︎ |
406
| `GraniteMoeSWAForCausalLM` | Granite MoE SWA | `ibm-granite/granite-swash-3b-a600m` | ✅︎ | ✅︎ |
407
| `GraniteSWAForCausalLM` | Granite SWA | `ibm-granite/granite-swash-2b` | ✅︎ | ✅︎ |
410
-| `GritLM` | GritLM | `parasail-ai/GritLM-7B-vllm`. | ✅︎ | ✅︎ |
408
| `HrmTextForCausalLM` | HRM-Text | `sapientinc/HRM-Text-1B`, etc. | | |
409
| `HunYuanDenseV1ForCausalLM` | Hunyuan Dense | `tencent/Hunyuan-7B-Instruct` | ✅︎ | ✅︎ |
410
| `HunYuanMoEV1ForCausalLM` | Hunyuan-A13B | `tencent/Hunyuan-A13B-Instruct`, `tencent/Hunyuan-A13B-Pretrain`, `tencent/Hunyuan-A13B-Instruct-FP8`, etc. | ✅︎ | ✅︎ |
@@ -437,7 +434,6 @@ th {
434
| `MistralForCausalLM` | Ministral-3, Mistral, Mistral-Instruct | `mistralai/Ministral-3-3B-Instruct-2512`, `mistralai/Mistral-7B-v0.1`, `mistralai/Mistral-7B-Instruct-v0.1`, etc. | ✅︎ | ✅︎ |
435
| `MistralLarge3ForCausalLM` | Mistral-Large-3-675B-Base-2512, Mistral-Large-3-675B-Instruct-2512 | `mistralai/Mistral-Large-3-675B-Base-2512`, `mistralai/Mistral-Large-3-675B-Instruct-2512`, etc. | ✅︎ | ✅︎ |
436
| `MixtralForCausalLM` | Mixtral-8x7B, Mixtral-8x7B-Instruct | `mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, `mistral-community/Mixtral-8x22B-v0.1`, etc. | ✅︎ | ✅︎ |
440
-| `MPTForCausalLM` | MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter | `mosaicml/mpt-7b`, `mosaicml/mpt-7b-storywriter`, `mosaicml/mpt-30b`, etc. | | ✅︎ |
437
| `NemotronForCausalLM` | Nemotron-3, Nemotron-4, Minitron | `nvidia/Minitron-8B-Base`, `mgoin/Nemotron-4-340B-Base-hf-FP8`, etc. | ✅︎ | ✅︎ |
438
| `NemotronHForCausalLM` | Nemotron-H | `nvidia/Nemotron-H-8B-Base-8K`, `nvidia/Nemotron-H-47B-Base-8K`, `nvidia/Nemotron-H-56B-Base-8K`, etc. | ✅︎ | ✅︎ |
439
| `Olmo3ForCausalLM` | OLMo3 | `allenai/Olmo-3-7B-Instruct`, `allenai/Olmo-3-32B-Think`, etc. | ✅︎ | ✅︎ |
@@ -458,14 +454,12 @@ th {
454
| `Qwen3ForCausalLM` | Qwen3 | `Qwen/Qwen3-8B`, etc. | ✅︎ | ✅︎ |
455
| `Qwen3MoeForCausalLM` | Qwen3MoE | `Qwen/Qwen3-30B-A3B`, etc. | ✅︎ | ✅︎ |
456
| `Qwen3NextForCausalLM` | Qwen3NextMoE | `Qwen/Qwen3-Next-80B-A3B-Instruct`, etc. | ✅︎ | ✅︎ |
461
-| `RWForCausalLM` | Falcon RW | `tiiuae/falcon-40b`, etc. | | ✅︎ |
457
| `Rnj1ForCausalLM` | Rnj1 | `EssentialAI/rnj-1-instruct`, etc. | | |
458
| `SarvamMoEForCausalLM` | Sarvam 2 | `sarvamai/sarvam2-30b-a3b`, etc. | ✅︎ | ✅︎ |
459
| `SarvamMLAForCausalLM` | Sarvam 2 | `sarvamai/sarvam2-105b-a9b`, etc. | | ✅︎ |
460
| `SeedOssForCausalLM` | SeedOss | `ByteDance-Seed/Seed-OSS-36B-Instruct`, etc. | ✅︎ | ✅︎ |
461
| `SolarForCausalLM` | Solar Pro | `upstage/solar-pro-preview-instruct`, etc. | ✅︎ | ✅︎ |
462
| `StableLmForCausalLM` | StableLM | `stabilityai/stablelm-3b-4e1t`, `stabilityai/stablelm-base-alpha-7b-v2`, etc. | | |
468
-| `StableLMEpochForCausalLM` | StableLM Epoch | `stabilityai/stablelm-zephyr-3b`, etc. | | ✅︎ |
463
| `Step1ForCausalLM` | Step-Audio | `stepfun-ai/Step-Audio-EditX`, etc. | ✅︎ | ✅︎ |
464
| `Step3p5ForCausalLM` | Step-3.5-flash | `stepfun-ai/Step-3.5-Flash`, etc. | | ✅︎ |
465
| `TeleChat2ForCausalLM` | TeleChat2 | `Tele-AI/TeleChat2-3B`, `Tele-AI/TeleChat2-7B`, `Tele-AI/TeleChat2-35B`, etc. | ✅︎ | ✅︎ |
@@ -527,8 +521,6 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
521
| `BagelForConditionalGeneration` | BAGEL | T + I<sup>+</sup> | `ByteDance-Seed/BAGEL-7B-MoT` | ✅︎ | ✅︎ |
522
| `BeeForConditionalGeneration` | Bee-8B | T + I<sup>E+</sup> | `Open-Bee/Bee-8B-RL`, `Open-Bee/Bee-8B-SFT` | | ✅︎ |
523
| `Blip2ForConditionalGeneration` | BLIP-2 | T + I<sup>E</sup> | `Salesforce/blip2-opt-2.7b`, `Salesforce/blip2-opt-6.7b`, etc. | ✅︎ | ✅︎ |
530
-| `ChameleonForConditionalGeneration` | Chameleon | T + I | `facebook/chameleon-7b`, etc. | | ✅︎ |
531
-| `CheersForConditionalGeneration` | Cheers | T + I | `ai9stars/Cheers` | | ✅︎ |
524
| `Cohere2VisionForConditionalGeneration` | Command A Vision, Command-A+ | T + I<sup>+</sup> | `CohereLabs/command-a-vision-07-2025`, `CohereLabs/command-a-plus-05-2026`, etc. | | ✅︎ |
525
| `Cosmos3ForConditionalGeneration` | Cosmos3 (understanding tower) | T + I<sup>E+</sup> + V<sup>E+</sup> | `nvidia/Cosmos3-Nano`, `nvidia/Cosmos3-Super` | | ✅︎ |
526
| `Cosmos3EdgeForConditionalGeneration` | Cosmos3-Edge (understanding tower) | T + I<sup>E+</sup> + V<sup>E+</sup> | `nvidia/Cosmos3-Edge` | | ✅︎ |
@@ -549,7 +541,6 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
541
| `Granite4VisionForConditionalGeneration` | Granite 4 Vision | T + I<sup>E+</sup> | `ibm-granite/granite-4.1-3b-vision`, etc. | ✅︎ | ✅︎ |
542
| `GraniteSpeechForConditionalGeneration` | Granite Speech | T + A | `ibm-granite/granite-speech-3.3-8b` | ✅︎ | ✅︎ |
543
| `GraniteSpeechPlusForConditionalGeneration` | Granite Speech Plus | T + A | `ibm-granite/granite-speech-4.1-2b-plus` | ✅︎ | ✅︎ |
552
-| `HCXVisionForCausalLM` | HyperCLOVAX-SEED-Vision-Instruct-3B | T + I<sup>+</sup> + V<sup>+</sup> | `naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B` | | |
544
| `HCXVisionV2ForCausalLM` | HyperCLOVAX-SEED-Think-32B | T + I<sup>+</sup> + V<sup>+</sup> | `naver-hyperclovax/HyperCLOVAX-SEED-Think-32B` | | |
545
| `H2OVLChatModel` | H2OVL | T + I<sup>E+</sup> | `h2oai/h2ovl-mississippi-800m`, `h2oai/h2ovl-mississippi-2b`, etc. | ✅︎ | ✅︎ |
546
| `HunYuanVLForConditionalGeneration` | HunyuanOCR | T + I<sup>E+</sup> | `tencent/HunyuanOCR`, etc. | ✅︎ | ✅︎ |
@@ -700,7 +691,6 @@ Speech2Text models trained specifically for Automatic Speech Recognition.
691
| ------------ | ------ | ----------------- | -------------------- | ------------------------- |
692
| `CohereAsrForConditionalGeneration` | Cohere-Transcribe | `CohereLabs/cohere-transcribe-03-2026` | | |
693
| `FireRedASR2ForConditionalGeneration` | FireRedASR2 | `allendou/FireRedASR2-LLM-vllm`, etc. | | |
703
-| `FireRedLIDForConditionalGeneration` | FireRedLID | `PatchyTisa/FireRedLID-vllm`, etc. | | |
694
| `FunASRForConditionalGeneration` | FunASR | `allendou/Fun-ASR-Nano-2512-vllm`, etc. | | |
695
| `Gemma3nForConditionalGeneration` | Gemma3n | `google/gemma-3n-E2B-it`, `google/gemma-3n-E4B-it`, etc. | | |
696
| `GlmAsrForConditionalGeneration` | GLM-ASR | `zai-org/GLM-ASR-Nano-2512` | ✅︎ | ✅︎ |
examples/generate/multimodal/audio_language_offline.py
-21
@@ -496,30 +496,9 @@ def run_whisper(question: str, audio_count: int) -> ModelRequestData:
496
)
497
498
499
-# FireRedLID
500
-def run_fireredlid(question: str, audio_count: int) -> ModelRequestData:
501
- assert audio_count == 1, "FireRedLID only supports single audio input per prompt"
502
- model_name = "PatchyTisa/FireRedLID-vllm"
503
-
504
- prompt = "<sos>"
505
-
506
- engine_args = EngineArgs(
507
- model=model_name,
508
- max_model_len=8,
509
- max_num_seqs=5,
510
- limit_mm_per_prompt={"audio": audio_count},
511
- )
512
-
513
- return ModelRequestData(
514
- engine_args=engine_args,
515
- prompt=prompt,
516
- )
517
-
518
-
499
model_example_map = {
500
"audioflamingo3": run_audioflamingo3,
501
"cohere_asr": run_cohere_asr,
522
- "fireredlid": run_fireredlid,
502
"funaudiochat": run_funaudiochat,
503
"gemma3n": run_gemma3n,
504
"glmasr": run_glmasr,
examples/generate/multimodal/encoder_decoder_multimodal_offline.py
-43
@@ -95,51 +95,8 @@ def run_fireredasr2():
95
)
96
97
98
-def run_fireredlid():
99
- """
100
- FireRedLID – Language Identification model.
101
-
102
- This encoder-decoder model identifies the spoken language of an audio
103
- clip. It outputs at most 2 tokens representing the detected language
104
- (e.g. "en", "zh mandarin").
105
- """
106
- engine_args = EngineArgs(
107
- model="PatchyTisa/FireRedLID-vllm",
108
- max_model_len=8,
109
- max_num_seqs=16,
110
- limit_mm_per_prompt={"audio": 1},
111
- )
112
-
113
- prompts = [
114
- { # Test explicit encoder/decoder prompt
115
- "encoder_prompt": {
116
- "prompt": "",
117
- "multi_modal_data": {
118
- "audio": AudioAsset("mary_had_lamb").audio_and_sample_rate,
119
- },
120
- },
121
- "decoder_prompt": "<sos>",
122
- },
123
- { # Another audio sample
124
- "encoder_prompt": {
125
- "prompt": "",
126
- "multi_modal_data": {
127
- "audio": AudioAsset("winning_call").audio_and_sample_rate,
128
- },
129
- },
130
- "decoder_prompt": "<sos>",
131
- },
132
- ]
133
-
134
- return ModelRequestData(
135
- engine_args=engine_args,
136
- prompts=prompts,
137
- )
138
-
139
-
98
model_example_map = {
99
"fireredasr2": run_fireredasr2,
142
- "fireredlid": run_fireredlid,
100
"whisper": run_whisper,
101
}
102
examples/generate/multimodal/vision_language_multi_image_offline.py
-48
@@ -365,53 +365,6 @@ def load_hunyuan_vl(question: str, image_urls: list[str]) -> ModelRequestData:
365
)
366
367
368
-def load_hyperclovax_seed_vision(
369
- question: str, image_urls: list[str]
370
-) -> ModelRequestData:
371
- model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B"
372
- tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
373
-
374
- engine_args = EngineArgs(
375
- model=model_name,
376
- trust_remote_code=True,
377
- max_model_len=16384,
378
- limit_mm_per_prompt={"image": len(image_urls)},
379
- )
380
-
381
- message = {"role": "user", "content": list()}
382
- for _image_url in image_urls:
383
- message["content"].append(
384
- {
385
- "type": "image",
386
- "image": _image_url,
387
- "ocr": "",
388
- "lens_keywords": "",
389
- "lens_local_keywords": "",
390
- }
391
- )
392
- message["content"].append(
393
- {
394
- "type": "text",
395
- "text": question,
396
- }
397
- )
398
-
399
- prompt = tokenizer.apply_chat_template(
400
- [
401
- message,
402
- ],
403
- tokenize=False,
404
- add_generation_prompt=True,
405
- )
406
-
407
- return ModelRequestData(
408
- engine_args=engine_args,
409
- prompt=prompt,
410
- stop_token_ids=None,
411
- image_data=[fetch_image(url) for url in image_urls],
412
- )
413
-
414
-
368
def load_idefics3(question: str, image_urls: list[str]) -> ModelRequestData:
369
model_name = "HuggingFaceM4/Idefics3-8B-Llama3"
370
@@ -1397,7 +1350,6 @@ model_example_map = {
1350
"granite4_vision": load_granite4_vision,
1351
"h2ovl_chat": load_h2ovl,
1352
"hunyuan_vl": load_hunyuan_vl,
1400
- "hyperclovax_seed_vision": load_hyperclovax_seed_vision,
1353
"idefics3": load_idefics3,
1354
"interns1": load_interns1,
1355
"internvl_chat": load_internvl,
examples/generate/multimodal/vision_language_offline.py
-151
@@ -140,51 +140,6 @@ def run_blip2(questions: list[str], modality: str) -> ModelRequestData:
140
)
141
142
143
-# Chameleon
144
-def run_chameleon(questions: list[str], modality: str) -> ModelRequestData:
145
- assert modality == "image"
146
-
147
- prompts = [f"{question}<image>" for question in questions]
148
- engine_args = EngineArgs(
149
- model="facebook/chameleon-7b",
150
- max_model_len=4096,
151
- max_num_seqs=2,
152
- limit_mm_per_prompt={modality: 1},
153
- )
154
-
155
- return ModelRequestData(
156
- engine_args=engine_args,
157
- prompts=prompts,
158
- )
159
-
160
-
161
-# Cheers
162
-def run_cheers(questions: list[str], modality: str) -> ModelRequestData:
163
- assert modality == "image"
164
- model_name = "ai9stars/Cheers"
165
-
166
- engine_args = EngineArgs(
167
- model=model_name,
168
- trust_remote_code=True,
169
- max_model_len=4096,
170
- limit_mm_per_prompt={modality: 1},
171
- )
172
-
173
- prompts = [
174
- (
175
- f"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
176
- f"<|im_start|>user\n<|image_pad|>{question}<|im_end|>\n"
177
- f"<|im_start|>assistant\n"
178
- )
179
- for question in questions
180
- ]
181
-
182
- return ModelRequestData(
183
- engine_args=engine_args,
184
- prompts=prompts,
185
- )
186
-
187
-
143
def run_command_a_vision(questions: list[str], modality: str) -> ModelRequestData:
144
assert modality == "image"
145
@@ -801,109 +756,6 @@ def run_hunyuan_vl(questions: list[str], modality: str) -> ModelRequestData:
756
)
757
758
804
-# naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B
805
-def run_hyperclovax_seed_vision(
806
- questions: list[str], modality: str
807
-) -> ModelRequestData:
808
- model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B"
809
- tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
810
-
811
- mm_limit = {"image": 1, "video": 1} if modality == "image+video" else {modality: 1}
812
- engine_args = EngineArgs(
813
- model=model_name,
814
- trust_remote_code=True,
815
- max_model_len=16384 if modality in ("video", "image+video") else 8192,
816
- limit_mm_per_prompt=mm_limit,
817
- )
818
-
819
- messages = list()
820
- for question in questions:
821
- if modality == "image":
822
- """
823
- ocr: List the words in the image in raster order.
824
- Even if the word order feels unnatural for reading,
825
- the model will handle it as long as it follows raster order.
826
- e.g. "Naver, CLOVA, bigshane"
827
- lens_keywords: List the entity names in the image.
828
- e.g. "iPhone"
829
- lens_local_keywords: List the entity names with quads in the image.
830
- e.g. "[0.07, 0.21, 0.92, 0.90] iPhone"
831
- """
832
- messages.append(
833
- [
834
- {
835
- "role": "user",
836
- "content": [
837
- {
838
- "type": "image",
839
- "ocr": "",
840
- "lens_keywords": "",
841
- "lens_local_keywords": "",
842
- },
843
- {
844
- "type": "text",
845
- "text": question,
846
- },
847
- ],
848
- }
849
- ]
850
- )
851
- elif modality == "video":
852
- messages.append(
853
- [
854
- {
855
- "role": "user",
856
- "content": [
857
- {
858
- "type": "video",
859
- },
860
- {
861
- "type": "text",
862
- "text": question,
863
- },
864
- ],
865
- }
866
- ]
867
- )
868
- elif modality == "image+video":
869
- messages.append(
870
- [
871
- {
872
- "role": "user",
873
- "content": [
874
- {
875
- "type": "image",
876
- "ocr": "",
877
- "lens_keywords": "",
878
- "lens_local_keywords": "",
879
- },
880
- {
881
- "type": "video",
882
- },
883
- {
884
- "type": "text",
885
- "text": question,
886
- },
887
- ],
888
- }
889
- ]
890
- )
891
- else:
892
- raise ValueError(f"Unsupported modality: {modality}")
893
-
894
- prompts = tokenizer.apply_chat_template(
895
- messages,
896
- tokenize=False,
897
- add_generation_prompt=True,
898
- )
899
-
900
- return ModelRequestData(
901
- engine_args=engine_args,
902
- prompts=prompts,
903
- stop_token_ids=None,
904
- )
905
-
906
-
759
# Idefics3-8B-Llama3
760
def run_idefics3(questions: list[str], modality: str) -> ModelRequestData:
761
assert modality == "image"
@@ -2328,10 +2180,8 @@ def run_step_vl(questions: list[str], modality: str) -> ModelRequestData:
2180
model_example_map = {
2181
"aria": run_aria,
2182
"bagel": run_bagel,
2331
- "cheers": run_cheers,
2183
"bee": run_bee,
2184
"blip-2": run_blip2,
2334
- "chameleon": run_chameleon,
2185
"command_a_vision": run_command_a_vision,
2186
"deepseek_vl_v2": run_deepseek_vl2,
2187
"deepseek_ocr": run_deepseek_ocr,
@@ -2350,7 +2200,6 @@ model_example_map = {
2200
"glm_ocr": run_glm_ocr,
2201
"h2ovl_chat": run_h2ovl,
2202
"hunyuan_vl": run_hunyuan_vl,
2353
- "hyperclovax_seed_vision": run_hyperclovax_seed_vision,
2203
"idefics3": run_idefics3,
2204
"interns1": run_interns1,
2205
"interns1_pro": run_interns1_pro,
examples/speech_to_text/lid/openai_lid_client.py
deleted
-193
@@ -1,193 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-"""
4
-Language Identification (LID) demo using the FireRedLID model on vLLM.
5
-
6
-FireRedLID is an audio encoder-decoder model that identifies the spoken
7
-language of an audio clip. Unlike ASR models that output full transcriptions,
8
-FireRedLID outputs at most 2 tokens representing the detected language
9
-(e.g. "en", "zh mandarin").
10
-
11
-Start the vLLM server:
12
-
13
- vllm serve PatchyTisa/FireRedLID-vllm
14
-
15
-Then run this script:
16
-
17
- # Use the built-in sample audio
18
- python examples/speech_to_text/lid/openai_lid_client.py
19
-
20
- # Use your own audio file(s)
21
- python examples/speech_to_text/lid/openai_lid_client.py \
22
- --audio_paths audio_en.wav audio_zh.wav audio_fr.wav
23
-
24
- # Batch-identify multiple files in one run
25
- python examples/speech_to_text/lid/openai_lid_client.py \
26
- --audio_paths /path/to/dir/*.wav
27
-
28
-Requirements:
29
-- vLLM with audio support
30
-- openai Python SDK
31
-- kaldi_native_fbank (pulled in by the model)
32
-"""
33
-
34
-import argparse
35
-import json
36
-import os
37
-
38
-from openai import OpenAI
39
-
40
-from vllm.assets.audio import AudioAsset
41
-
42
-# ──────────────────────────────────────────────────────────────────────
43
-# Helpers
44
-# ──────────────────────────────────────────────────────────────────────
45
-
46
-
47
-def identify_language(
48
- audio_path: str,
49
- client: OpenAI,
50
- model: str,
51
-) -> str:
52
- """
53
- Send a single audio file to the vLLM transcription endpoint and return
54
- the detected language tag.
55
-
56
- FireRedLID re-uses the OpenAI-compatible ``/v1/audio/transcriptions``
57
- endpoint. The "transcription" it returns is actually the language label
58
- (e.g. ``"en"`` or ``"zh mandarin"``).
59
- """
60
- with open(audio_path, "rb") as f:
61
- result = client.audio.transcriptions.create(
62
- file=f,
63
- model=model,
64
- response_format="json",
65
- temperature=0.0,
66
- )
67
- return result.text.strip()
68
-
69
-
70
-def identify_language_raw(
71
- audio_path: str,
72
- model: str,
73
- api_base: str,
74
-) -> str:
75
- """
76
- Same as :func:`identify_language` but uses raw HTTP so that the demo
77
- works without the ``openai`` SDK (useful for quick debugging).
78
- """
79
- import requests
80
-
81
- url = f"{api_base}/audio/transcriptions"
82
- with open(audio_path, "rb") as f:
83
- files = {"file": (os.path.basename(audio_path), f)}
84
- data = {
85
- "model": model,
86
- "response_format": "json",
87
- }
88
- resp = requests.post(url, files=files, data=data)
89
- resp.raise_for_status()
90
- return resp.json()["text"].strip()
91
-
92
-
93
-def identify_language_streaming(
94
- audio_path: str,
95
- model: str,
96
- api_base: str,
97
-) -> str:
98
- """
99
- Streaming variant – demonstrates the streaming transcription endpoint.
100
- For a 1-2 token output the stream finishes almost instantly, but this
101
- shows that the API path works end-to-end.
102
- """
103
- import requests
104
-
105
- url = f"{api_base}/audio/transcriptions"
106
- with open(audio_path, "rb") as f:
107
- files = {"file": (os.path.basename(audio_path), f)}
108
- data = {
109
- "stream": "true",
110
- "model": model,
111
- "response_format": "json",
112
- }
113
- response = requests.post(url, files=files, data=data, stream=True)
114
- response.raise_for_status()
115
-
116
- tokens: list[str] = []
117
- for chunk in response.iter_lines(
118
- chunk_size=8192, decode_unicode=False, delimiter=b"\n"
119
- ):
120
- if not chunk:
121
- continue
122
- payload = json.loads(chunk[len("data: ") :].decode("utf-8"))
123
- choice = payload["choices"][0]
124
- delta = choice.get("delta", {}).get("content", "")
125
- if delta:
126
- tokens.append(delta)
127
- if choice.get("finish_reason") is not None:
128
- break
129
-
130
- return "".join(tokens).strip()
131
-
132
-
133
-# ──────────────────────────────────────────────────────────────────────
134
-# Main
135
-# ──────────────────────────────────────────────────────────────────────
136
-
137
-
138
-def main(args: argparse.Namespace) -> None:
139
- api_base = args.api_base.rstrip("/")
140
- client = OpenAI(api_key="EMPTY", base_url=api_base)
141
- model = client.models.list().data[0].id
142
- print(f"Model : {model}")
143
- print(f"Server: {api_base}\n")
144
-
145
- # Resolve audio paths ------------------------------------------------
146
- if args.audio_paths:
147
- audio_paths = args.audio_paths
148
- else:
149
- # Fall back to the built-in vLLM sample audios (both are English).
150
- audio_paths = [
151
- str(AudioAsset("mary_had_lamb").get_local_path()),
152
- str(AudioAsset("winning_call").get_local_path()),
153
- ]
154
-
155
- # Run LID for each file ----------------------------------------------
156
- print(f"{'Audio File':<50} {'Language (sync)':<20} {'Language (stream)'}")
157
- print("-" * 90)
158
-
159
- for path in audio_paths:
160
- basename = os.path.basename(path)
161
-
162
- # 1) Synchronous via OpenAI SDK
163
- lang_sync = identify_language(path, client, model)
164
-
165
- # 2) Streaming via raw HTTP
166
- lang_stream = identify_language_streaming(path, model, api_base)
167
-
168
- print(f"{basename:<50} {lang_sync:<20} {lang_stream}")
169
-
170
- print()
171
-
172
-
173
-if __name__ == "__main__":
174
- parser = argparse.ArgumentParser(
175
- description="FireRedLID – Language Identification demo via vLLM",
176
- )
177
- parser.add_argument(
178
- "--audio_paths",
179
- nargs="+",
180
- default=None,
181
- help=(
182
- "One or more audio files to identify. "
183
- "If omitted, uses vLLM's built-in sample audios."
184
- ),
185
- )
186
- parser.add_argument(
187
- "--api_base",
188
- type=str,
189
- default="http://localhost:8000/v1",
190
- help="vLLM API base URL (default: http://localhost:8000/v1)",
191
- )
192
- args = parser.parse_args()
193
- main(args)
requirements/test/nightly-torch.txt
+1
-1
@@ -10,7 +10,7 @@ pytest-timeout
10
# testing utils
11
backoff # required for phi4mm test
12
blobfile # required for kimi-vl test
13
-einops # required for MPT, qwen-vl and Mamba
13
+einops # required for qwen-vl and Mamba
14
httpx
15
librosa # required for audio tests
16
vocos # required for minicpmo_26 test
tests/config/base_model_arch_groundtruth.json
-17
@@ -84,23 +84,6 @@
84
"is_multimodal_model": false,
85
"dtype": "torch.bfloat16"
86
},
87
- "mosaicml/mpt-7b": {
88
- "architectures": [
89
- "MPTForCausalLM"
90
- ],
91
- "model_type": "mpt",
92
- "text_model_type": "mpt",
93
- "hidden_size": 4096,
94
- "total_num_hidden_layers": 32,
95
- "total_num_attention_heads": 32,
96
- "head_size": 128,
97
- "vocab_size": 50432,
98
- "total_num_kv_heads": 32,
99
- "num_experts": 0,
100
- "is_deepseek_mla": false,
101
- "is_multimodal_model": false,
102
- "dtype": "torch.bfloat16"
103
- },
87
"databricks/dbrx-instruct": {
88
"architectures": [
89
"DbrxForCausalLM"
tests/config/test_model_arch_config.py
-2
@@ -38,8 +38,6 @@ BASE_MODELS_TO_TEST = [
38
# (NonGeoDataset import error). Tested in model initialization tests.
39
# "ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11",
40
"Zyphra/Zamba2-7B-instruct",
41
- # FIXME: mosaicml/mpt-7b has been deleted
42
- # "mosaicml/mpt-7b",
41
# FIXME: databricks/dbrx-instruct has been deleted
42
# "databricks/dbrx-instruct",
43
"tiiuae/falcon-7b",
tests/distributed/test_pipeline_parallel.py
-3
@@ -102,7 +102,6 @@ class PPTestSettings:
102
103
TEXT_GENERATION_MODELS = {
104
# [Decoder-only]
105
- "Snowflake/snowflake-arctic-instruct": PPTestSettings.fast(load_format="dummy"),
105
"bigscience/bloomz-1b1": PPTestSettings.fast(),
106
"zai-org/chatglm3-6b": PPTestSettings.fast(),
107
"CohereLabs/c4ai-command-r-v01": PPTestSettings.fast(load_format="dummy"),
@@ -131,7 +130,6 @@ TEXT_GENERATION_MODELS = {
130
# "mistralai/Mistral-7B-Instruct-v0.1": PPTestSettings.fast(),
131
"state-spaces/mamba-130m-hf": PPTestSettings.fast(),
132
"mistralai/Mixtral-8x7B-Instruct-v0.1": PPTestSettings.fast(load_format="dummy"),
134
- "mosaicml/mpt-7b": PPTestSettings.fast(),
133
"nvidia/Minitron-8B-Base": PPTestSettings.fast(),
134
"allenai/OLMo-1B-hf": PPTestSettings.fast(),
135
"allenai/OLMo-2-0425-1B": PPTestSettings.fast(),
@@ -165,7 +163,6 @@ EMBEDDING_MODELS = { # type: ignore[var-annotated]
163
MULTIMODAL_MODELS = {
164
# [Decoder-only]
165
"Salesforce/blip2-opt-6.7b": PPTestSettings.fast(),
168
- "facebook/chameleon-7b": PPTestSettings.fast(),
166
"zai-org/glm-4v-9b": PPTestSettings.fast(),
167
"OpenGVLab/InternVL3-1B": PPTestSettings.fast(),
168
"llava-hf/llava-1.5-7b-hf": PPTestSettings.fast(),
tests/models/language/pooling/test_gritlm.py
deleted
-188
@@ -1,188 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-import numpy as np
4
-import pytest
5
-from scipy.spatial.distance import cosine
6
-
7
-from vllm import LLM, SamplingParams
8
-from vllm.config import ModelConfig
9
-
10
-from ....utils import ROCM_EXTRA_ARGS, RemoteOpenAIServer
11
-from .embed_utils import run_client_embeddings
12
-
13
-MODEL_NAME = "parasail-ai/GritLM-7B-vllm"
14
-MAX_MODEL_LEN = 4000
15
-ATOL = 2.3e-3
16
-
17
-
18
-def _arr(arr):
19
- """
20
- Convert a list of integers to an array of integers.
21
- """
22
- return np.array(arr)
23
-
24
-
25
-def test_find_array():
26
- from vllm.model_executor.models.gritlm import GritLMMeanPool
27
-
28
- model_config = ModelConfig(
29
- MODEL_NAME,
30
- runner="pooling",
31
- dtype="bfloat16",
32
- seed=0,
33
- )
34
- pooling = GritLMMeanPool(model_config=model_config)
35
-
36
- arr = _arr([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
37
-
38
- assert pooling._find_array(arr, _arr([3, 4, 5]), start_idx=0) == 3
39
- assert pooling._find_array(arr, _arr([3, 4, 5]), start_idx=1) == 3
40
- assert pooling._find_array(arr, _arr([3, 4, 5]), start_idx=5) == -1
41
- assert pooling._find_array(arr, _arr([3, 4, 5]), end_idx=3) == -1
42
- assert pooling._find_array(arr, _arr([3, 4, 5]), end_idx=4) == 3
43
- assert pooling._find_array(arr, _arr([3, 5]), start_idx=0) == -1
44
-
45
- with pytest.raises(ValueError):
46
- pooling._find_array(arr, _arr([3, 4, 5]), start_idx=-1)
47
-
48
-
49
-def run_llm_encode(
50
- llm: LLM,
51
- queries: list[str],
52
- instruction: str,
53
-) -> list[list[float]]:
54
- outputs = llm.embed([instruction + q for q in queries])
55
- return [output.outputs.embedding for output in outputs]
56
-
57
-
58
-def gritlm_instruction(instruction):
59
- return (
60
- "<|user|>\n" + instruction + "\n<|embed|>\n" if instruction else "<|embed|>\n"
61
- )
62
-
63
-
64
-def get_test_data():
65
- """
66
- Grabbed this test data and the expected values from
67
- README.md in https://github.com/ContextualAI/gritlm
68
- """
69
- q_instruction = gritlm_instruction(
70
- "Given a scientific paper title, retrieve the paper's abstract",
71
- )
72
- queries = [
73
- "Bitcoin: A Peer-to-Peer Electronic Cash System",
74
- "Generative Representational Instruction Tuning",
75
- ]
76
-
77
- d_instruction = gritlm_instruction("")
78
- documents = [
79
- # ruff: noqa: E501
80
- "A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution. Digital signatures provide part of the solution, but the main benefits are lost if a trusted third party is still required to prevent double-spending. We propose a solution to the double-spending problem using a peer-to-peer network. The network timestamps transactions by hashing them into an ongoing chain of hash-based proof-of-work, forming a record that cannot be changed without redoing the proof-of-work. The longest chain not only serves as proof of the sequence of events witnessed, but proof that it came from the largest pool of CPU power. As long as a majority of CPU power is controlled by nodes that are not cooperating to attack the network, they'll generate the longest chain and outpace attackers. The network itself requires minimal structure. Messages are broadcast on a best effort basis, and nodes can leave and rejoin the network at will, accepting the longest proof-of-work chain as proof of what happened while they were gone.",
81
- "All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational instruction tuning (GRIT) whereby a large language model is trained to handle both generative and embedding tasks by distinguishing between them through instructions. Compared to other open models, our resulting GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) and outperforms all models up to its size on a range of generative tasks. By scaling up further, GritLM 8X7B outperforms all open generative language models that we tried while still being among the best embedding models. Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss. Among other benefits, the unification via GRIT speeds up Retrieval-Augmented Generation (RAG) by > 60% for long documents, by no longer requiring separate retrieval and generation models. Models, code, etc. are freely available at https://github.com/ContextualAI/gritlm.",
82
- ]
83
-
84
- return queries, q_instruction, documents, d_instruction
85
-
86
-
87
-def validate_embed_output(q_rep: list[list[float]], d_rep: list[list[float]]):
88
- cosine_sim_q0_d0 = 1 - cosine(q_rep[0], d_rep[0])
89
- assert cosine_sim_q0_d0 == pytest.approx(0.609, abs=ATOL)
90
-
91
- cosine_sim_q0_d1 = 1 - cosine(q_rep[0], d_rep[1])
92
- assert cosine_sim_q0_d1 == pytest.approx(0.101, abs=ATOL)
93
-
94
- cosine_sim_q1_d0 = 1 - cosine(q_rep[1], d_rep[0])
95
- assert cosine_sim_q1_d0 == pytest.approx(0.120, abs=ATOL)
96
-
97
- cosine_sim_q1_d1 = 1 - cosine(q_rep[1], d_rep[1])
98
- assert cosine_sim_q1_d1 == pytest.approx(0.534, abs=ATOL)
99
-
100
-
101
-def test_gritlm_offline_embedding(vllm_runner):
102
- queries, q_instruction, documents, d_instruction = get_test_data()
103
-
104
- with vllm_runner(
105
- MODEL_NAME,
106
- runner="pooling",
107
- max_model_len=MAX_MODEL_LEN,
108
- ) as vllm_model:
109
- llm = vllm_model.llm
110
-
111
- d_rep = run_llm_encode(
112
- llm,
113
- documents,
114
- d_instruction,
115
- )
116
- q_rep = run_llm_encode(
117
- llm,
118
- queries,
119
- q_instruction,
120
- )
121
-
122
- validate_embed_output(q_rep, d_rep)
123
-
124
-
125
-@pytest.mark.asyncio
126
-async def test_gritlm_api_server_embedding():
127
- queries, q_instruction, documents, d_instruction = get_test_data()
128
-
129
- args = [
130
- "--runner",
131
- "pooling",
132
- "--max_model_len",
133
- str(MAX_MODEL_LEN),
134
- *ROCM_EXTRA_ARGS,
135
- ]
136
-
137
- with RemoteOpenAIServer(MODEL_NAME, args) as server:
138
- client_embedding = server.get_async_client()
139
-
140
- d_rep = await run_client_embeddings(
141
- client_embedding,
142
- MODEL_NAME,
143
- documents,
144
- d_instruction,
145
- )
146
- q_rep = await run_client_embeddings(
147
- client_embedding,
148
- MODEL_NAME,
149
- queries,
150
- q_instruction,
151
- )
152
-
153
- validate_embed_output(q_rep, d_rep)
154
-
155
-
156
-def test_gritlm_offline_generate(monkeypatch: pytest.MonkeyPatch, vllm_runner):
157
- input = "<|user|>\nWhat is the capital of France?\n<|assistant|>\n"
158
-
159
- with vllm_runner(
160
- MODEL_NAME,
161
- runner="generate",
162
- max_model_len=MAX_MODEL_LEN,
163
- ) as vllm_model:
164
- llm = vllm_model.llm
165
-
166
- sampling_params = SamplingParams(temperature=0.0, max_tokens=256)
167
- outputs = llm.generate(input, sampling_params=sampling_params)
168
-
169
- assert outputs[0].outputs[0].text == "The capital of France is Paris."
170
-
171
-
172
-@pytest.mark.asyncio
173
-async def test_gritlm_api_server_generate():
174
- input = "<|user|>\nWhat is the capital of France?\n<|assistant|>\n"
175
-
176
- args = ["--runner", "generate", "--max_model_len", str(MAX_MODEL_LEN)]
177
-
178
- with RemoteOpenAIServer(MODEL_NAME, args) as server:
179
- client_generate = server.get_async_client()
180
-
181
- outputs = await client_generate.completions.create(
182
- model=MODEL_NAME,
183
- prompt=input,
184
- max_tokens=256,
185
- temperature=0.0,
186
- )
187
-
188
- assert outputs.choices[0].text == "The capital of France is Paris."
tests/models/multimodal/generation/test_common.py
-26
@@ -30,7 +30,6 @@ from ....conftest import (
30
VllmRunner,
31
)
32
from ....utils import create_new_process_for_each_test, large_gpu_mark, multi_gpu_marks
33
-from ...utils import check_outputs_equal
33
from .vlm_utils import custom_inputs, model_utils, runners
34
from .vlm_utils.case_filtering import get_parametrized_options
35
from .vlm_utils.types import (
@@ -321,20 +320,6 @@ VLM_TEST_SETTINGS = {
320
# FIXME: https://github.com/huggingface/transformers/pull/38510
321
marks=[pytest.mark.skip("Model is broken")],
322
),
324
- "chameleon": VLMTestInfo(
325
- models=["facebook/chameleon-7b"],
326
- test_type=VLMTestType.IMAGE,
327
- prompt_formatter=lambda img_prompt: f"USER: {img_prompt}\nASSISTANT:",
328
- max_model_len=4096,
329
- max_num_seqs=2,
330
- auto_cls=AutoModelForImageTextToText,
331
- # For chameleon, we only compare the sequences
332
- vllm_output_post_proc=lambda vllm_output, model: vllm_output[:2],
333
- hf_output_post_proc=lambda hf_output, model: hf_output[:2],
334
- comparator=check_outputs_equal,
335
- max_tokens=8,
336
- dtype="bfloat16",
337
- ),
323
"cosmos3": VLMTestInfo(
324
models=["nvidia/Cosmos3-Nano"],
325
test_type=(
@@ -935,17 +920,6 @@ VLM_TEST_SETTINGS = {
920
num_logprobs=10,
921
),
922
### Tensor parallel / multi-gpu broadcast tests
938
- "chameleon-broadcast": VLMTestInfo(
939
- models=["facebook/chameleon-7b"],
940
- prompt_formatter=lambda img_prompt: f"USER: {img_prompt}\nASSISTANT:",
941
- max_model_len=4096,
942
- auto_cls=AutoModelForImageTextToText,
943
- vllm_output_post_proc=lambda vllm_output, model: vllm_output[:2],
944
- hf_output_post_proc=lambda hf_output, model: hf_output[:2],
945
- comparator=check_outputs_equal,
946
- marks=multi_gpu_marks(num_gpus=2),
947
- **COMMON_BROADCAST_SETTINGS, # type: ignore
948
- ),
923
"llava-broadcast": VLMTestInfo(
924
models=["llava-hf/llava-1.5-7b-hf"],
925
prompt_formatter=lambda img_prompt: f"USER: {img_prompt}\nASSISTANT:",
tests/models/registry.py
-38
@@ -202,7 +202,6 @@ _TEXT_GENERATION_EXAMPLE_MODELS = {
202
"AfmoeForCausalLM": _HfExamplesInfo("arcee-ai/Trinity-Nano-Preview"),
203
"ApertusForCausalLM": _HfExamplesInfo("swiss-ai/Apertus-8B-Instruct-2509"),
204
"ArceeForCausalLM": _HfExamplesInfo("arcee-ai/AFM-4.5B-Base"),
205
- "ArcticForCausalLM": _HfExamplesInfo("Snowflake/snowflake-arctic-instruct"),
205
"AXK1ForCausalLM": _HfExamplesInfo("skt/A.X-K1", trust_remote_code=True),
206
"BailingMoeForCausalLM": _HfExamplesInfo(
207
"inclusionAI/Ling-lite-1.5", trust_remote_code=True
@@ -273,7 +272,6 @@ _TEXT_GENERATION_EXAMPLE_MODELS = {
272
"ExaoneMoeForCausalLM": _HfExamplesInfo(
273
"LGAI-EXAONE/K-EXAONE-236B-A23B", min_transformers_version="5.1.0"
274
),
276
- "Fairseq2LlamaForCausalLM": _HfExamplesInfo("mgleize/fairseq2-dummy-Llama-3.2-1B"),
275
"FalconForCausalLM": _HfExamplesInfo("tiiuae/falcon-7b"),
276
"FalconH1ForCausalLM": _HfExamplesInfo("tiiuae/Falcon-H1-0.5B-Base"),
277
"FlexOlmoForCausalLM": _HfExamplesInfo("allenai/Flex-reddit-2x7B-1T"),
@@ -448,9 +446,6 @@ _TEXT_GENERATION_EXAMPLE_MODELS = {
446
"mistralai/Mixtral-8x7B-Instruct-v0.1",
447
{"tiny": "TitanML/tiny-mixtral"},
448
),
451
- "MptForCausalLM": _HfExamplesInfo("mpt", is_available_online=False),
452
- # FIXME: mosaicml/mpt-7b has been deleted
453
- "MPTForCausalLM": _HfExamplesInfo("mosaicml/mpt-7b", is_available_online=False),
449
"NemotronForCausalLM": _HfExamplesInfo("nvidia/Minitron-8B-Base"),
450
"NemotronHForCausalLM": _HfExamplesInfo(
451
"nvidia/Nemotron-H-8B-Base-8K", trust_remote_code=True
@@ -523,7 +518,6 @@ _TEXT_GENERATION_EXAMPLE_MODELS = {
518
"EssentialAI/rnj-1-instruct",
519
is_available_online=False,
520
),
526
- "RWForCausalLM": _HfExamplesInfo("tiiuae/falcon-40b"),
521
"SarvamMoEForCausalLM": _HfExamplesInfo(
522
"sarvamai/sarvam-30b",
523
trust_remote_code=True,
@@ -548,7 +542,6 @@ _TEXT_GENERATION_EXAMPLE_MODELS = {
542
trust_remote_code=True,
543
),
544
"SmolLM3ForCausalLM": _HfExamplesInfo("HuggingFaceTB/SmolLM3-3B"),
551
- "StableLMEpochForCausalLM": _HfExamplesInfo("stabilityai/stablelm-zephyr-3b"),
545
"StableLmForCausalLM": _HfExamplesInfo("stabilityai/stablelm-3b-4e1t"),
546
"Starcoder2ForCausalLM": _HfExamplesInfo("bigcode/starcoder2-3b"),
547
"Step1ForCausalLM": _HfExamplesInfo(
@@ -600,7 +593,6 @@ _EMBEDDING_EXAMPLE_MODELS = {
593
),
594
"Gemma2Model": _HfExamplesInfo("BAAI/bge-multilingual-gemma2"),
595
"Gemma3TextModel": _HfExamplesInfo("google/embeddinggemma-300m"),
603
- "GritLM": _HfExamplesInfo("parasail-ai/GritLM-7B-vllm"),
596
"GteModel": _HfExamplesInfo(
597
"Snowflake/snowflake-arctic-embed-m-v2.0", trust_remote_code=True
598
),
@@ -642,14 +634,6 @@ _EMBEDDING_EXAMPLE_MODELS = {
634
),
635
"Qwen2VLForConditionalGeneration": _HfExamplesInfo("MrLight/dse-qwen2-2b-mrl-v1"),
636
"SiglipModel": _HfExamplesInfo("google/siglip-base-patch16-224"),
645
- "PrithviGeoSpatialMAE": _HfExamplesInfo(
646
- "ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11",
647
- dtype="float16",
648
- enforce_eager=True,
649
- require_embed_inputs=True,
650
- # This is to avoid the model going OOM in CI
651
- max_num_seqs=32,
652
- ),
637
"Terratorch": _HfExamplesInfo(
638
"ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11",
639
dtype="float16",
@@ -813,15 +797,6 @@ _MULTIMODAL_EXAMPLE_MODELS = {
797
"Salesforce/blip2-opt-2.7b",
798
extras={"6b": "Salesforce/blip2-opt-6.7b"},
799
),
816
- "ChameleonForConditionalGeneration": _HfExamplesInfo("facebook/chameleon-7b"),
817
- "Cheers": _HfExamplesInfo(
818
- "ai9stars/Cheers",
819
- trust_remote_code=True,
820
- ),
821
- "CheersForConditionalGeneration": _HfExamplesInfo(
822
- "ai9stars/Cheers",
823
- trust_remote_code=True,
824
- ),
800
"Cohere2VisionForConditionalGeneration": _HfExamplesInfo(
801
"CohereLabs/command-a-vision-07-2025",
802
extras={"command-a-plus": "CohereLabs/command-a-plus-05-2026-bf16"},
@@ -890,15 +865,6 @@ _MULTIMODAL_EXAMPLE_MODELS = {
865
"(dict object has no attribute '__name__').",
866
},
867
),
893
- "FireRedLIDForConditionalGeneration": _HfExamplesInfo(
894
- "PatchyTisa/FireRedLID-vllm",
895
- trust_remote_code=True,
896
- max_transformers_version="5.1",
897
- transformers_version_reason={
898
- "vllm": "Incompatible with transformers v5.2+ "
899
- "(dict object has no attribute '__name__').",
900
- },
901
- ),
868
"FunASRForConditionalGeneration": _HfExamplesInfo(
869
"allendou/Fun-ASR-Nano-2512-vllm",
870
trust_remote_code=True,
@@ -966,10 +932,6 @@ _MULTIMODAL_EXAMPLE_MODELS = {
932
max_transformers_version="4.48",
933
transformers_version_reason={"hf": "HF model is not compatible."},
934
),
969
- "HCXVisionForCausalLM": _HfExamplesInfo(
970
- "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B",
971
- trust_remote_code=True,
972
- ),
935
"HCXVisionV2ForCausalLM": _HfExamplesInfo(
936
"naver-hyperclovax/HyperCLOVAX-SEED-Think-32B",
937
trust_remote_code=True,
tests/models/test_initialization.py
+2
-2
@@ -39,7 +39,7 @@ MINIMAL_MODEL_ARCH_LIST = [
39
"InternVLChatModel",
40
"InternLM2ForRewardModel",
41
"TransformersMultiModalForCausalLM",
42
- "PrithviGeoSpatialMAE",
42
+ "Terratorch",
43
"UltravoxModel",
44
"DeepSeekMTPModel",
45
"XLMRobertaModel",
@@ -111,7 +111,7 @@ def can_initialize(
111
"which is not configured in test environment"
112
)
113
114
- if model_arch in ("PrithviGeoSpatialMAE", "Terratorch"):
114
+ if model_arch == "Terratorch":
115
import importlib.util
116
117
if importlib.util.find_spec("terratorch") is None:
tests/models/test_registry.py
+1
-1
@@ -38,7 +38,7 @@ def test_registry_imports(model_arch):
38
check_version_reason="vllm",
39
)
40
41
- if model_arch in ("PrithviGeoSpatialMAE", "Terratorch"):
41
+ if model_arch == "Terratorch":
42
import importlib.util
43
44
if importlib.util.find_spec("terratorch") is None:
tests/renderers/test_hf.py
-1
@@ -424,7 +424,6 @@ def test_resolve_content_format_hf_defined(model, expected_format):
424
("model", "expected_format"),
425
[
426
("Salesforce/blip2-opt-2.7b", "string"),
427
- ("facebook/chameleon-7b", "string"),
427
("deepseek-ai/deepseek-vl2-tiny", "string"),
428
("google/paligemma-3b-mix-224", "string"),
429
],
tests/tokenizers_/test_detokenize.py
-2
@@ -39,8 +39,6 @@ TOKENIZERS = [
39
"EleutherAI/gpt-j-6b",
40
"EleutherAI/pythia-70m",
41
"bigscience/bloom-560m",
42
- # FIXME: mosaicml/mpt-7b has been deleted
43
- # "mosaicml/mpt-7b",
42
"tiiuae/falcon-7b",
43
"meta-llama/Llama-3.2-1B-Instruct",
44
"codellama/CodeLlama-7b-hf",
tests/utils_/test_tensor_schema.py
+23
-2
@@ -1,13 +1,34 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
4
+from typing import Annotated, Literal
5
+
6
import pytest
7
import torch
8
9
from vllm.model_executor.models.glm4_1v import Glm4vImageEmbeddingInputs
10
from vllm.model_executor.models.granite_speech import GraniteSpeechAudioInputs
9
-from vllm.model_executor.models.hyperclovax_vision import HCXVisionVideoPixelInputs
11
from vllm.model_executor.models.phi3v import Phi3VImagePixelInputs
12
+from vllm.utils.tensor_schema import TensorSchema, TensorShape
13
+
14
+
15
+class _DoubleNestedVideoPixelInputs(TensorSchema):
16
+ """Minimal schema exercising doubly-nested list[list[torch.Tensor]] inputs.
17
+
18
+ Dimensions:
19
+ - n: Number of videos
20
+ - f: Number of frames
21
+ - g: Number of grids
22
+ - c: Number of channels (3)
23
+ - h: Height
24
+ - w: Width
25
+ """
26
+
27
+ type: Literal["pixel_values_videos"] = "pixel_values_videos"
28
+ pixel_values_videos: Annotated[
29
+ list[list[torch.Tensor]],
30
+ TensorShape("n", "f", "g", 3, "h", "w", dynamic_dims={"f", "g"}),
31
+ ]
32
33
34
def test_tensor_schema_valid_tensor():
@@ -100,7 +121,7 @@ def test_tensor_schema_double_nested_tensors():
121
x = torch.rand(4, 3, 32, 32)
122
y = torch.rand(2, 3, 32, 32)
123
103
- HCXVisionVideoPixelInputs(pixel_values_videos=([x, y, x], [y], [x, y]))
124
+ _DoubleNestedVideoPixelInputs(pixel_values_videos=([x, y, x], [y], [x, y]))
125
126
127
def test_tensor_schema_inconsistent_shapes_in_list():
tests/weight_loading/models.txt
+1
-2
@@ -16,5 +16,4 @@ compressed-tensors, neuralmagic/Phi-3-medium-128k-instruct-quantized.w4a16, main
16
#compressed-tensors, mgoin/DeepSeek-Coder-V2-Lite-Instruct-FP8, main
17
awq, casperhansen/mixtral-instruct-awq, main
18
awq_marlin, casperhansen/mixtral-instruct-awq, main
19
-fp8, neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV, main
20
-None, mgleize/fairseq2-dummy-Llama-3.2-1B, main
\ No newline at end of file
19
+fp8, neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV, main
\ No newline at end of file
vllm/model_executor/models/apertus.py
+1
-2
@@ -255,8 +255,7 @@ class ApertusDecoderLayer(nn.Module):
255
256
# Apertus defaults to causal attention as it is a decoder-only model.
257
# You can override the HF config with `is_causal=False` to enable
258
- # bidirectional attention, which is used in some embedding models
259
- # (e.g. parasail-ai/GritLM-7B-vllm)
258
+ # bidirectional attention, which is used in some embedding models.
259
if getattr(config, "is_causal", True):
260
attn_type = AttentionType.DECODER
261
else:
vllm/model_executor/models/arctic.py
deleted
-593
@@ -1,593 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-"""Inference-only Snowflake Arctic model."""
4
-
5
-from collections.abc import Iterable
6
-from itertools import islice
7
-
8
-import torch
9
-from torch import nn
10
-
11
-from vllm.compilation.decorators import support_torch_compile
12
-from vllm.config import CacheConfig, VllmConfig
13
-from vllm.distributed import (
14
- get_pp_group,
15
- get_tensor_model_parallel_rank,
16
- get_tensor_model_parallel_world_size,
17
- tensor_model_parallel_all_reduce,
18
-)
19
-from vllm.model_executor.layers.activation import SiluAndMul
20
-from vllm.model_executor.layers.attention import Attention
21
-from vllm.model_executor.layers.fused_moe import (
22
- fused_experts,
23
- fused_topk,
24
-)
25
-from vllm.model_executor.layers.layernorm import RMSNorm
26
-from vllm.model_executor.layers.linear import (
27
- MergedColumnParallelLinear,
28
- QKVParallelLinear,
29
- ReplicatedLinear,
30
- RowParallelLinear,
31
-)
32
-from vllm.model_executor.layers.logits_processor import LogitsProcessor
33
-from vllm.model_executor.layers.quantization import QuantizationConfig
34
-from vllm.model_executor.layers.rotary_embedding import get_rope
35
-from vllm.model_executor.layers.vocab_parallel_embedding import (
36
- ParallelLMHead,
37
- VocabParallelEmbedding,
38
-)
39
-from vllm.model_executor.model_loader.weight_utils import default_weight_loader
40
-from vllm.model_executor.utils import set_weight_attrs
41
-from vllm.platforms import current_platform
42
-from vllm.sequence import IntermediateTensors
43
-from vllm.transformers_utils.configs.arctic import ArcticConfig
44
-
45
-from .interfaces import SupportsPP, SupportsQuant
46
-from .utils import (
47
- AutoWeightsLoader,
48
- extract_layer_index,
49
- is_pp_missing_parameter,
50
- make_empty_intermediate_tensors_factory,
51
- make_layers,
52
- maybe_prefix,
53
-)
54
-
55
-
56
-class ArcticMLP(nn.Module):
57
- def __init__(
58
- self,
59
- config: ArcticConfig,
60
- expert_id: int = -1,
61
- is_residual_mlp: bool = False,
62
- quant_config: QuantizationConfig | None = None,
63
- reduce_results: bool = True,
64
- prefix: str = "",
65
- ):
66
- super().__init__()
67
- self.hidden_size = config.hidden_size
68
- self.expert_id = expert_id
69
-
70
- self.ffn_dim = (
71
- config.intermediate_size if not is_residual_mlp else self.hidden_size
72
- )
73
-
74
- self.w13 = MergedColumnParallelLinear(
75
- self.hidden_size,
76
- [self.ffn_dim] * 2,
77
- bias=False,
78
- quant_config=quant_config,
79
- prefix=f"{prefix}.w13",
80
- )
81
- self.w2 = RowParallelLinear(
82
- self.ffn_dim,
83
- self.hidden_size,
84
- bias=False,
85
- reduce_results=reduce_results,
86
- quant_config=quant_config,
87
- prefix=f"{prefix}.w2",
88
- )
89
- if config.hidden_act != "silu":
90
- raise ValueError(
91
- f"Unsupported activation: {config.hidden_act}. "
92
- "Only silu is supported for now."
93
- )
94
- self.act_fn = SiluAndMul()
95
-
96
- def forward(self, hidden_states):
97
- gate_up, _ = self.w13(hidden_states)
98
- hidden_states = self.act_fn(gate_up)
99
- hidden_states, _ = self.w2(hidden_states)
100
- return hidden_states
101
-
102
-
103
-class ArcticMoE(nn.Module):
104
- """
105
- Model-parallel implementation of Arctic MoE Layer.
106
- """
107
-
108
- def __init__(
109
- self,
110
- config: ArcticConfig,
111
- tp_size: int | None = None,
112
- params_dtype: torch.dtype | None = None,
113
- quant_config: QuantizationConfig | None = None,
114
- reduce_results: bool = True,
115
- prefix: str = "",
116
- ):
117
- super().__init__()
118
-
119
- layer_id = extract_layer_index(prefix)
120
- self.tp_size = tp_size or get_tensor_model_parallel_world_size()
121
- self.hidden_size = config.hidden_size
122
- self.num_experts = config.num_local_experts
123
- self.layer_id = layer_id
124
- self.top_k = config.num_experts_per_tok
125
- self.intermediate_size = config.intermediate_size // self.tp_size
126
-
127
- self.is_moe_layer = (layer_id + 1) % config.moe_layer_frequency == 0
128
- self.reduce_results = reduce_results
129
- # Some other parameters
130
- if params_dtype is None:
131
- params_dtype = torch.get_default_dtype()
132
- self.params_dtype = params_dtype
133
-
134
- if not self.is_moe_layer:
135
- self.mlp = ArcticMLP(
136
- config,
137
- quant_config=quant_config,
138
- reduce_results=reduce_results,
139
- prefix=f"{prefix}.mlp",
140
- )
141
- else:
142
- self.gate = ReplicatedLinear(
143
- self.hidden_size,
144
- self.num_experts,
145
- bias=False,
146
- params_dtype=self.params_dtype,
147
- quant_config=quant_config,
148
- prefix=f"{prefix}.gate",
149
- )
150
- self.ws = nn.Parameter(
151
- torch.empty(
152
- self.num_experts,
153
- 2 * self.intermediate_size,
154
- self.hidden_size,
155
- device=current_platform.device_type,
156
- dtype=self.params_dtype,
157
- )
158
- )
159
- self.w2s = nn.Parameter(
160
- torch.empty(
161
- self.num_experts,
162
- self.hidden_size,
163
- self.intermediate_size,
164
- device=current_platform.device_type,
165
- dtype=self.params_dtype,
166
- )
167
- )
168
- set_weight_attrs(
169
- self.ws,
170
- {
171
- "weight_loader": self.weight_loader,
172
- },
173
- )
174
- set_weight_attrs(
175
- self.w2s,
176
- {
177
- "weight_loader": self.weight_loader,
178
- },
179
- )
180
-
181
- def weight_loader(
182
- self,
183
- param: nn.Parameter,
184
- loaded_weight: torch.Tensor,
185
- weight_name: str,
186
- expert_id: int,
187
- ):
188
- tp_rank = get_tensor_model_parallel_rank()
189
- param_data = param.data
190
- shard_size = self.intermediate_size
191
- shard = slice(tp_rank * shard_size, (tp_rank + 1) * shard_size)
192
- if weight_name.endswith("w1.weight"):
193
- param_data[expert_id, 0:shard_size, :] = loaded_weight[shard, :]
194
- if weight_name.endswith("w3.weight"):
195
- param_data[expert_id, shard_size : 2 * shard_size, :] = loaded_weight[
196
- shard, :
197
- ]
198
- if weight_name.endswith("w2.weight"):
199
- param_data[expert_id, :, :] = loaded_weight[:, shard]
200
-
201
- def local_moe_fused(self, hidden_states: torch.Tensor) -> torch.Tensor:
202
- num_tokens, hidden_size = hidden_states.shape
203
- hidden_states = hidden_states.view(-1, self.hidden_size)
204
- # router_logits: (num_tokens, n_experts)
205
- router_logits, _ = self.gate(hidden_states)
206
- do_normalize = self.top_k > 1
207
- topk_weights, topk_ids, token_expert_indices = fused_topk(
208
- hidden_states, router_logits, self.top_k, renormalize=do_normalize
209
- )
210
- final_hidden_states = fused_experts(
211
- hidden_states,
212
- self.ws,
213
- self.w2s,
214
- topk_weights,
215
- topk_ids,
216
- )
217
- if self.reduce_results and self.tp_size > 1:
218
- final_hidden_states = tensor_model_parallel_all_reduce(final_hidden_states)
219
- return final_hidden_states.view(num_tokens, hidden_size)
220
-
221
- def forward(self, hidden_states: torch.Tensor):
222
- if self.is_moe_layer:
223
- final_hidden_states = self.local_moe_fused(hidden_states)
224
- else:
225
- final_hidden_states = self.mlp(hidden_states)
226
- return final_hidden_states
227
-
228
-
229
-class ArcticAttention(nn.Module):
230
- def __init__(
231
- self,
232
- config: ArcticConfig,
233
- cache_config: CacheConfig | None = None,
234
- quant_config: QuantizationConfig | None = None,
235
- prefix: str = "",
236
- ):
237
- super().__init__()
238
- self.config = config
239
- self.hidden_size = config.hidden_size
240
-
241
- tp_size = get_tensor_model_parallel_world_size()
242
- self.total_num_heads = config.num_attention_heads
243
- assert self.total_num_heads % tp_size == 0
244
- self.num_heads = self.total_num_heads // tp_size
245
- self.total_num_kv_heads = config.num_key_value_heads
246
- if self.total_num_kv_heads >= tp_size:
247
- assert self.total_num_kv_heads % tp_size == 0
248
- else:
249
- assert tp_size % self.total_num_kv_heads == 0
250
- self.num_kv_heads = max(1, self.total_num_kv_heads // tp_size)
251
- self.head_dim = self.hidden_size // self.total_num_heads
252
- self.q_size = self.num_heads * self.head_dim
253
- self.kv_size = self.num_kv_heads * self.head_dim
254
-
255
- self.max_position_embeddings = config.max_position_embeddings
256
- self.scaling = self.head_dim**-0.5
257
-
258
- self.qkv_proj = QKVParallelLinear(
259
- self.hidden_size,
260
- self.head_dim,
261
- self.total_num_heads,
262
- self.total_num_kv_heads,
263
- bias=False,
264
- quant_config=quant_config,
265
- prefix=f"{prefix}.qkv_proj",
266
- )
267
- self.o_proj = RowParallelLinear(
268
- self.total_num_heads * self.head_dim,
269
- self.hidden_size,
270
- bias=False,
271
- reduce_results=True,
272
- quant_config=quant_config,
273
- prefix=f"{prefix}.o_proj",
274
- )
275
-
276
- self.rotary_emb = get_rope(
277
- self.head_dim,
278
- max_position=self.max_position_embeddings,
279
- rope_parameters=config.rope_parameters,
280
- is_neox_style=True,
281
- )
282
-
283
- self.attn = Attention(
284
- self.num_heads,
285
- self.head_dim,
286
- self.scaling,
287
- num_kv_heads=self.num_kv_heads,
288
- cache_config=cache_config,
289
- quant_config=quant_config,
290
- prefix=f"{prefix}.attn",
291
- )
292
-
293
- def forward(
294
- self,
295
- positions: torch.Tensor,
296
- hidden_states: torch.Tensor,
297
- ) -> torch.Tensor:
298
- qkv, _ = self.qkv_proj(hidden_states)
299
- q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
300
- q, k = self.rotary_emb(positions, q, k)
301
- attn_output = self.attn(q, k, v)
302
- output, _ = self.o_proj(attn_output)
303
- return output
304
-
305
-
306
-class ArcticDecoderLayer(nn.Module):
307
- def __init__(
308
- self,
309
- config: ArcticConfig,
310
- cache_config: CacheConfig | None = None,
311
- quant_config: QuantizationConfig | None = None,
312
- prefix: str = "",
313
- ) -> None:
314
- super().__init__()
315
- self.hidden_size = config.hidden_size
316
- layer_idx = extract_layer_index(prefix)
317
- is_moe_layer = (layer_idx + 1) % config.moe_layer_frequency == 0
318
- self.use_residual = config.use_residual and is_moe_layer
319
- self.self_attn = ArcticAttention(
320
- config,
321
- cache_config,
322
- quant_config=quant_config,
323
- prefix=f"{prefix}.self_attn",
324
- )
325
- self.block_sparse_moe = ArcticMoE(
326
- config,
327
- quant_config=quant_config,
328
- reduce_results=(not self.use_residual),
329
- prefix=f"{prefix}.block_sparse_moe",
330
- )
331
-
332
- self.input_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
333
- self.post_attention_layernorm = RMSNorm(
334
- config.hidden_size, eps=config.rms_norm_eps
335
- )
336
-
337
- if self.use_residual:
338
- self.residual_layernorm = RMSNorm(
339
- config.hidden_size, eps=config.rms_norm_eps
340
- )
341
- self.residual_mlp = ArcticMLP(
342
- config,
343
- is_residual_mlp=True,
344
- reduce_results=False,
345
- prefix=f"{prefix}.residual_mlp",
346
- )
347
-
348
- def forward(
349
- self,
350
- positions: torch.Tensor,
351
- hidden_states: torch.Tensor,
352
- ) -> torch.Tensor:
353
- residual_input = hidden_states
354
- hidden_states = self.input_layernorm(hidden_states)
355
- hidden_states = self.self_attn(
356
- positions=positions,
357
- hidden_states=hidden_states,
358
- )
359
- hidden_states = residual_input + hidden_states
360
-
361
- residual_attn = hidden_states
362
- if self.use_residual:
363
- hidden_states = self.residual_layernorm(hidden_states)
364
- hidden_states = self.residual_mlp(hidden_states)
365
- residual_mlp = hidden_states
366
- hidden_states = self.post_attention_layernorm(residual_input)
367
- hidden_states = self.block_sparse_moe(hidden_states)
368
- hidden_states = residual_mlp + hidden_states
369
- hidden_states = tensor_model_parallel_all_reduce(hidden_states)
370
- hidden_states = residual_attn + hidden_states
371
- else:
372
- hidden_states = self.post_attention_layernorm(hidden_states)
373
- hidden_states = self.block_sparse_moe(hidden_states)
374
- hidden_states = residual_attn + hidden_states
375
- return hidden_states
376
-
377
-
378
-@support_torch_compile
379
-class ArcticModel(nn.Module):
380
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
381
- super().__init__()
382
-
383
- config = vllm_config.model_config.hf_config
384
- cache_config = vllm_config.cache_config
385
- quant_config = vllm_config.quant_config
386
-
387
- self.config = config
388
- self.vocab_size = config.vocab_size
389
- self.embed_tokens = VocabParallelEmbedding(
390
- self.vocab_size, config.hidden_size, org_num_embeddings=self.vocab_size
391
- )
392
- self.start_layer, self.end_layer, self.layers = make_layers(
393
- config.num_hidden_layers,
394
- lambda prefix: ArcticDecoderLayer(
395
- config, cache_config, quant_config, prefix=prefix
396
- ),
397
- prefix=f"{prefix}.layers",
398
- )
399
- self._attn_implementation = config._attn_implementation
400
- self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
401
- self.make_empty_intermediate_tensors = make_empty_intermediate_tensors_factory(
402
- ["hidden_states"], config.hidden_size
403
- )
404
-
405
- def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
406
- return self.embed_tokens(input_ids)
407
-
408
- def forward(
409
- self,
410
- input_ids: torch.Tensor | None,
411
- positions: torch.Tensor,
412
- intermediate_tensors: IntermediateTensors | None,
413
- inputs_embeds: torch.Tensor | None = None,
414
- ) -> torch.Tensor | IntermediateTensors:
415
- if get_pp_group().is_first_rank:
416
- if inputs_embeds is not None:
417
- hidden_states = inputs_embeds
418
- else:
419
- hidden_states = self.embed_input_ids(input_ids)
420
- else:
421
- assert intermediate_tensors is not None
422
- hidden_states = intermediate_tensors["hidden_states"]
423
- for layer in islice(self.layers, self.start_layer, self.end_layer):
424
- hidden_states = layer(positions, hidden_states)
425
- if not get_pp_group().is_last_rank:
426
- return IntermediateTensors({"hidden_states": hidden_states})
427
- hidden_states = self.norm(hidden_states)
428
- return hidden_states
429
-
430
- def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
431
- stacked_params_mapping = [
432
- # (param_name, shard_name, shard_id)
433
- ("qkv_proj", "q_proj", "q"),
434
- ("qkv_proj", "k_proj", "k"),
435
- ("qkv_proj", "v_proj", "v"),
436
- ]
437
-
438
- mlp_params_mapping: list[tuple[str, str, int]] = []
439
- expert_params_mapping: list[tuple[str, str, int]] = []
440
-
441
- for layer in range(self.config.num_hidden_layers):
442
- is_moe_layer = (layer + 1) % self.config.moe_layer_frequency == 0
443
- if is_moe_layer and self.config.use_residual:
444
- mlp_params_mapping.append(
445
- (
446
- f"layers.{layer}.residual_mlp.w13.weight",
447
- f"layers.{layer}.residual_mlp.w1.weight",
448
- 0,
449
- )
450
- )
451
- mlp_params_mapping.append(
452
- (
453
- f"layers.{layer}.residual_mlp.w13.weight",
454
- f"layers.{layer}.residual_mlp.w3.weight",
455
- 1,
456
- )
457
- )
458
-
459
- if is_moe_layer:
460
- for expert_id in range(self.config.num_local_experts):
461
- expert_params_mapping.append(
462
- ("ws", f"experts.{expert_id}.w1.weight", expert_id)
463
- )
464
- expert_params_mapping.append(
465
- ("w2s", f"experts.{expert_id}.w2.weight", expert_id)
466
- )
467
- expert_params_mapping.append(
468
- ("ws", f"experts.{expert_id}.w3.weight", expert_id)
469
- )
470
- else:
471
- mlp_params_mapping.append(
472
- (
473
- f"layers.{layer}.block_sparse_moe.mlp.w13.weight",
474
- f"layers.{layer}.block_sparse_moe.mlp.w1.weight",
475
- 0,
476
- )
477
- )
478
- mlp_params_mapping.append(
479
- (
480
- f"layers.{layer}.block_sparse_moe.mlp.w13.weight",
481
- f"layers.{layer}.block_sparse_moe.mlp.w3.weight",
482
- 1,
483
- )
484
- )
485
-
486
- params_dict = dict(self.named_parameters())
487
- loaded_params: set[str] = set()
488
-
489
- for name, loaded_weight in weights:
490
- for param_name, weight_name, shard_id in stacked_params_mapping:
491
- if weight_name not in name:
492
- continue
493
- name = name.replace(weight_name, param_name)
494
- # Skip loading extra bias for GPTQ models.
495
- if name.endswith(".bias") and name not in params_dict:
496
- continue
497
- if is_pp_missing_parameter(name, self):
498
- continue
499
- param = params_dict[name]
500
- weight_loader = param.weight_loader
501
- weight_loader(param, loaded_weight, shard_id)
502
- break
503
- else:
504
- for param_name, weight_name, mlp_shard_id in mlp_params_mapping:
505
- if weight_name not in name:
506
- continue
507
- name = name.replace(weight_name, param_name)
508
- if is_pp_missing_parameter(name, self):
509
- continue
510
- param = params_dict[name]
511
- weight_loader = param.weight_loader
512
- weight_loader(param, loaded_weight, mlp_shard_id)
513
- break
514
- else:
515
- for param_name, weight_name, expert_id in expert_params_mapping:
516
- if weight_name not in name:
517
- continue
518
- name = name.replace(weight_name, param_name)
519
- if is_pp_missing_parameter(name, self):
520
- continue
521
- param = params_dict[name]
522
- weight_loader = param.weight_loader
523
- weight_loader(
524
- param, loaded_weight, weight_name, expert_id=expert_id
525
- )
526
- break
527
- else:
528
- if name.endswith(".bias") and name not in params_dict:
529
- continue
530
- if is_pp_missing_parameter(name, self):
531
- continue
532
- param = params_dict[name]
533
- weight_loader = getattr(
534
- param, "weight_loader", default_weight_loader
535
- )
536
- weight_loader(param, loaded_weight)
537
- loaded_params.add(name)
538
- return loaded_params
539
-
540
-
541
-class ArcticForCausalLM(nn.Module, SupportsPP, SupportsQuant):
542
- packed_modules_mapping = {"qkv_proj": ["q_proj", "k_proj", "v_proj"]}
543
-
544
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
545
- super().__init__()
546
- config = vllm_config.model_config.hf_config
547
- quant_config = vllm_config.quant_config
548
- self.config = config
549
- self.model = ArcticModel(
550
- vllm_config=vllm_config, prefix=maybe_prefix(prefix, "model")
551
- )
552
- self.vocab_size = config.vocab_size
553
- self.lm_head = ParallelLMHead(
554
- self.vocab_size,
555
- config.hidden_size,
556
- quant_config=quant_config,
557
- prefix=maybe_prefix(prefix, "lm_head"),
558
- )
559
- if self.config.tie_word_embeddings:
560
- self.lm_head = self.lm_head.tie_weights(self.model.embed_tokens)
561
- self.num_experts = config.num_local_experts
562
- self.num_experts_per_tok = config.num_experts_per_tok
563
-
564
- self.logits_processor = LogitsProcessor(config.vocab_size)
565
- self.make_empty_intermediate_tensors = (
566
- self.model.make_empty_intermediate_tensors
567
- )
568
-
569
- def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
570
- return self.model.embed_input_ids(input_ids)
571
-
572
- def forward(
573
- self,
574
- input_ids: torch.Tensor | None,
575
- positions: torch.Tensor,
576
- intermediate_tensors: IntermediateTensors | None = None,
577
- inputs_embeds: torch.Tensor | None = None,
578
- ) -> torch.Tensor | IntermediateTensors:
579
- hidden_states = self.model(
580
- input_ids, positions, intermediate_tensors, inputs_embeds
581
- )
582
- return hidden_states
583
-
584
- def compute_logits(
585
- self,
586
- hidden_states: torch.Tensor,
587
- ) -> torch.Tensor | None:
588
- logits = self.logits_processor(self.lm_head, hidden_states)
589
- return logits
590
-
591
- def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
592
- loader = AutoWeightsLoader(self)
593
- return loader.load_weights(weights)
vllm/model_executor/models/chameleon.py
deleted
-1043
@@ -1,1043 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-from collections.abc import Iterable, Mapping, Sequence
5
-from functools import cached_property
6
-from itertools import islice
7
-from typing import Annotated, Any, Literal
8
-
9
-import torch
10
-import torch.nn as nn
11
-import torch.nn.functional as F
12
-from transformers import (
13
- BatchFeature,
14
- ChameleonConfig,
15
- ChameleonProcessor,
16
- ChameleonVQVAEConfig,
17
-)
18
-
19
-from vllm.config import CacheConfig, VllmConfig
20
-from vllm.config.multimodal import BaseDummyOptions
21
-from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
22
-from vllm.inputs import MultiModalDataDict
23
-from vllm.logger import init_logger
24
-from vllm.model_executor.layers.activation import SiluAndMul
25
-from vllm.model_executor.layers.attention import Attention
26
-from vllm.model_executor.layers.conv import Conv2dLayer
27
-from vllm.model_executor.layers.layernorm import RMSNorm
28
-from vllm.model_executor.layers.linear import (
29
- MergedColumnParallelLinear,
30
- QKVParallelLinear,
31
- RowParallelLinear,
32
-)
33
-from vllm.model_executor.layers.logits_processor import LogitsProcessor
34
-from vllm.model_executor.layers.quantization import QuantizationConfig
35
-from vllm.model_executor.layers.rotary_embedding import get_rope
36
-from vllm.model_executor.layers.vocab_parallel_embedding import (
37
- ParallelLMHead,
38
- VocabParallelEmbedding,
39
-)
40
-from vllm.model_executor.model_loader.weight_utils import row_parallel_weight_loader
41
-from vllm.model_executor.utils import set_weight_attrs
42
-from vllm.multimodal import MULTIMODAL_REGISTRY
43
-from vllm.multimodal.inputs import (
44
- MultiModalFieldConfig,
45
- MultiModalKwargsItems,
46
-)
47
-from vllm.multimodal.parse import MultiModalDataItems
48
-from vllm.multimodal.processing import (
49
- BaseDummyInputsBuilder,
50
- BaseMultiModalProcessor,
51
- BaseProcessingInfo,
52
- PromptReplacement,
53
- PromptUpdate,
54
- PromptUpdateDetails,
55
-)
56
-from vllm.sequence import IntermediateTensors
57
-from vllm.utils.tensor_schema import TensorSchema, TensorShape
58
-from vllm.utils.torch_utils import async_tensor_h2d
59
-
60
-from .interfaces import (
61
- MultiModalEmbeddings,
62
- SupportsMultiModal,
63
- SupportsPP,
64
- SupportsQuant,
65
-)
66
-from .utils import (
67
- AutoWeightsLoader,
68
- WeightsMapper,
69
- make_empty_intermediate_tensors_factory,
70
- make_layers,
71
- maybe_prefix,
72
-)
73
-
74
-logger = init_logger(__name__)
75
-
76
-
77
-class ChameleonImagePixelInputs(TensorSchema):
78
- """
79
- Dimensions:
80
- - bn: Batch size * number of images
81
- - c: Number of channels (3)
82
- - h: Height of each image
83
- - w: Width of each image
84
- """
85
-
86
- type: Literal["pixel_values"]
87
- data: Annotated[torch.Tensor, TensorShape("bn", 3, "h", "w")]
88
-
89
-
90
-class ChameleonProcessingInfo(BaseProcessingInfo):
91
- def get_hf_config(self):
92
- return self.ctx.get_hf_config(ChameleonConfig)
93
-
94
- def get_hf_processor(self, **kwargs: object):
95
- return self.ctx.get_hf_processor(ChameleonProcessor, **kwargs)
96
-
97
- def get_supported_mm_limits(self) -> Mapping[str, int | None]:
98
- return {"image": 1}
99
-
100
- def get_num_image_tokens(self) -> int:
101
- processor = self.get_hf_processor()
102
- return processor.image_seq_length
103
-
104
-
105
-class ChameleonDummyInputsBuilder(BaseDummyInputsBuilder[ChameleonProcessingInfo]):
106
- def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
107
- num_images = mm_counts.get("image", 0)
108
-
109
- processor = self.info.get_hf_processor()
110
- image_token = processor.image_token
111
-
112
- return image_token * num_images
113
-
114
- def get_dummy_mm_data(
115
- self,
116
- seq_len: int,
117
- mm_counts: Mapping[str, int],
118
- mm_options: Mapping[str, BaseDummyOptions],
119
- ) -> MultiModalDataDict:
120
- config = self.info.get_hf_config()
121
-
122
- width = height = config.vq_config.resolution
123
- num_images = mm_counts.get("image", 0)
124
-
125
- image_overrides = mm_options.get("image")
126
-
127
- return {
128
- "image": self._get_dummy_images(
129
- width=width,
130
- height=height,
131
- num_images=num_images,
132
- overrides=image_overrides,
133
- )
134
- }
135
-
136
-
137
-class ChameleonMultiModalProcessor(BaseMultiModalProcessor[ChameleonProcessingInfo]):
138
- def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
139
- # ChameleonProcessor requires text corresponding to the images
140
- return self.dummy_inputs.get_dummy_text(mm_counts)
141
-
142
- def _postprocess_prompt(self, prompt: list[int]) -> list[int]:
143
- # HF processor adds sep token for chat mode
144
- tokenizer = self.info.get_tokenizer()
145
- sep_token_id = tokenizer.get_vocab()[tokenizer.sep_token] # type: ignore
146
-
147
- return [*prompt, sep_token_id]
148
-
149
- def _get_mm_fields_config(
150
- self,
151
- hf_inputs: BatchFeature,
152
- hf_processor_mm_kwargs: Mapping[str, object],
153
- ) -> Mapping[str, MultiModalFieldConfig]:
154
- return dict(pixel_values=MultiModalFieldConfig.batched("image"))
155
-
156
- def _get_prompt_updates(
157
- self,
158
- mm_items: MultiModalDataItems,
159
- hf_processor_mm_kwargs: Mapping[str, object],
160
- out_mm_kwargs: MultiModalKwargsItems,
161
- ) -> Sequence[PromptUpdate]:
162
- processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
163
- tokenizer = self.info.get_tokenizer()
164
- vocab = tokenizer.get_vocab()
165
-
166
- image_start_id = vocab[processor.image_start_token]
167
- image_token_id = vocab[processor.image_token]
168
- image_end_id = vocab[processor.image_end_token]
169
-
170
- num_image_tokens = self.info.get_num_image_tokens()
171
- image_tokens = [image_token_id] * num_image_tokens
172
-
173
- return [
174
- PromptReplacement(
175
- modality="image",
176
- target=[image_token_id],
177
- replacement=PromptUpdateDetails.select_token_id(
178
- [image_start_id] + image_tokens + [image_end_id],
179
- embed_token_id=image_token_id,
180
- ),
181
- )
182
- ]
183
-
184
-
185
-class ChameleonLayerNorm(nn.LayerNorm):
186
- def __init__(self, hidden_size, *args, **kwargs):
187
- super().__init__(hidden_size, *args, **kwargs)
188
- self.normalized_shape = (hidden_size[-1],)
189
-
190
- set_weight_attrs(self.weight, {"weight_loader": row_parallel_weight_loader})
191
- set_weight_attrs(self.bias, {"weight_loader": row_parallel_weight_loader})
192
-
193
- def forward(self, hidden_states):
194
- hidden_states = F.layer_norm(
195
- hidden_states, self.normalized_shape, None, None, eps=1e-5
196
- )
197
- hidden_states = hidden_states * self.weight + self.bias
198
- return hidden_states
199
-
200
-
201
-# Copied from vllm.model_executor.models.llama.LlamaMLP -> ChameleonMLP
202
-class ChameleonMLP(nn.Module):
203
- def __init__(
204
- self,
205
- hidden_size: int,
206
- intermediate_size: int,
207
- hidden_act: str,
208
- quant_config: QuantizationConfig | None = None,
209
- bias: bool = False,
210
- prefix: str = "",
211
- ) -> None:
212
- super().__init__()
213
- self.gate_up_proj = MergedColumnParallelLinear(
214
- input_size=hidden_size,
215
- output_sizes=[intermediate_size] * 2,
216
- bias=bias,
217
- quant_config=quant_config,
218
- prefix=f"{prefix}.gate_up_proj",
219
- )
220
- self.down_proj = RowParallelLinear(
221
- input_size=intermediate_size,
222
- output_size=hidden_size,
223
- bias=bias,
224
- quant_config=quant_config,
225
- prefix=f"{prefix}.down_proj",
226
- )
227
- if hidden_act != "silu":
228
- raise ValueError(
229
- f"Unsupported activation: {hidden_act}. Only silu is supported for now."
230
- )
231
- self.act_fn = SiluAndMul()
232
-
233
- def forward(self, x):
234
- gate_up, _ = self.gate_up_proj(x)
235
- x = self.act_fn(gate_up)
236
- x, _ = self.down_proj(x)
237
- return x
238
-
239
-
240
-# Modified from vllm.model_executor.models.llama.LlamaAttention -> ChameleonAttention #noqa
241
-class ChameleonAttention(nn.Module):
242
- def __init__(
243
- self,
244
- hidden_size: int,
245
- num_heads: int,
246
- num_kv_heads: int,
247
- rope_parameters: dict[str, Any],
248
- max_position_embeddings: int = 4096,
249
- quant_config: QuantizationConfig | None = None,
250
- bias: bool = False,
251
- cache_config: CacheConfig | None = None,
252
- prefix: str = "",
253
- ) -> None:
254
- super().__init__()
255
- self.hidden_size = hidden_size
256
- tp_size = get_tensor_model_parallel_world_size()
257
- self.total_num_heads = num_heads
258
- assert self.total_num_heads % tp_size == 0
259
- self.num_heads = self.total_num_heads // tp_size
260
- self.total_num_kv_heads = num_kv_heads
261
- if self.total_num_kv_heads >= tp_size:
262
- # Number of KV heads is greater than TP size, so we partition
263
- # the KV heads across multiple tensor parallel GPUs.
264
- assert self.total_num_kv_heads % tp_size == 0
265
- else:
266
- # Number of KV heads is less than TP size, so we replicate
267
- # the KV heads across multiple tensor parallel GPUs.
268
- assert tp_size % self.total_num_kv_heads == 0
269
- self.num_kv_heads = max(1, self.total_num_kv_heads // tp_size)
270
- self.head_dim = hidden_size // self.total_num_heads
271
- self.q_size = self.num_heads * self.head_dim
272
- self.kv_size = self.num_kv_heads * self.head_dim
273
- self.scaling = self.head_dim**-0.5
274
- self.max_position_embeddings = max_position_embeddings
275
-
276
- self.qkv_proj = QKVParallelLinear(
277
- hidden_size=hidden_size,
278
- head_size=self.head_dim,
279
- total_num_heads=self.total_num_heads,
280
- total_num_kv_heads=self.total_num_kv_heads,
281
- bias=bias,
282
- quant_config=quant_config,
283
- prefix=f"{prefix}.qkv_proj",
284
- )
285
- self.o_proj = RowParallelLinear(
286
- input_size=self.total_num_heads * self.head_dim,
287
- output_size=hidden_size,
288
- bias=bias,
289
- quant_config=quant_config,
290
- prefix=f"{prefix}.o_proj",
291
- )
292
- self.q_norm = ChameleonLayerNorm((self.num_heads, self.head_dim))
293
- self.k_norm = ChameleonLayerNorm((self.num_kv_heads, self.head_dim))
294
- self.rotary_emb = get_rope(
295
- self.head_dim,
296
- max_position=max_position_embeddings,
297
- rope_parameters=rope_parameters,
298
- )
299
-
300
- self.attn = Attention(
301
- self.num_heads,
302
- self.head_dim,
303
- self.scaling,
304
- num_kv_heads=self.num_kv_heads,
305
- cache_config=cache_config,
306
- quant_config=quant_config,
307
- prefix=f"{prefix}.attn",
308
- )
309
-
310
- def _apply_qk_norm(
311
- self, q: torch.Tensor, k: torch.Tensor
312
- ) -> tuple[torch.Tensor, torch.Tensor]:
313
- # reshape for layernorm
314
- q = q.reshape(-1, self.num_heads, self.head_dim)
315
- k = k.reshape(-1, self.num_kv_heads, self.head_dim)
316
- q = self.q_norm(q)
317
- k = self.k_norm(k)
318
- q = q.view(*q.shape[:-2], -1)
319
- k = k.view(*k.shape[:-2], -1)
320
- return q, k
321
-
322
- def forward(
323
- self,
324
- positions: torch.Tensor,
325
- hidden_states: torch.Tensor,
326
- ) -> torch.Tensor:
327
- qkv, _ = self.qkv_proj(hidden_states)
328
- q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
329
- q, k = self._apply_qk_norm(q, k)
330
-
331
- q, k = self.rotary_emb(positions, q, k)
332
- attn_output = self.attn(q, k, v)
333
- output, _ = self.o_proj(attn_output)
334
- return output
335
-
336
-
337
-class ChameleonDecoderLayer(nn.Module):
338
- def __init__(
339
- self,
340
- config: ChameleonConfig,
341
- cache_config: CacheConfig | None = None,
342
- quant_config: QuantizationConfig | None = None,
343
- prefix: str = "",
344
- ) -> None:
345
- super().__init__()
346
- self.hidden_size = config.hidden_size
347
- max_position_embeddings = getattr(config, "max_position_embeddings", 4096)
348
-
349
- self.self_attn = ChameleonAttention(
350
- hidden_size=self.hidden_size,
351
- num_heads=config.num_attention_heads,
352
- num_kv_heads=getattr(
353
- config, "num_key_value_heads", config.num_attention_heads
354
- ),
355
- rope_parameters=config.rope_parameters,
356
- max_position_embeddings=max_position_embeddings,
357
- quant_config=quant_config,
358
- bias=False,
359
- cache_config=cache_config,
360
- prefix=f"{prefix}.self_attn",
361
- )
362
- self.mlp = ChameleonMLP(
363
- hidden_size=self.hidden_size,
364
- intermediate_size=config.intermediate_size,
365
- hidden_act=config.hidden_act,
366
- quant_config=quant_config,
367
- bias=getattr(config, "mlp_bias", False),
368
- prefix=f"{prefix}.mlp",
369
- )
370
- self.input_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
371
- self.post_attention_layernorm = RMSNorm(
372
- config.hidden_size, eps=config.rms_norm_eps
373
- )
374
-
375
- def forward(
376
- self,
377
- positions: torch.Tensor,
378
- hidden_states: torch.Tensor,
379
- residual: torch.Tensor | None,
380
- ) -> tuple[torch.Tensor, torch.Tensor | None]:
381
- if residual is None:
382
- residual = hidden_states
383
- hidden_states = self.input_layernorm(hidden_states)
384
- else:
385
- hidden_states, residual = self.input_layernorm(hidden_states, residual)
386
- hidden_states = self.self_attn(
387
- positions=positions,
388
- hidden_states=hidden_states,
389
- )
390
-
391
- # Fully Connected
392
- hidden_states, residual = self.post_attention_layernorm(hidden_states, residual)
393
- hidden_states = self.mlp(hidden_states)
394
-
395
- return hidden_states, residual
396
-
397
-
398
-class ChameleonSwinDecoderLayer(nn.Module):
399
- def __init__(
400
- self,
401
- config: ChameleonConfig,
402
- cache_config: CacheConfig | None = None,
403
- quant_config: QuantizationConfig | None = None,
404
- prefix: str = "",
405
- ) -> None:
406
- super().__init__()
407
- self.hidden_size = config.hidden_size
408
- max_position_embeddings = getattr(config, "max_position_embeddings", 4096)
409
-
410
- self.self_attn = ChameleonAttention(
411
- hidden_size=self.hidden_size,
412
- num_heads=config.num_attention_heads,
413
- num_kv_heads=getattr(
414
- config, "num_key_value_heads", config.num_attention_heads
415
- ),
416
- rope_parameters=config.rope_parameters,
417
- max_position_embeddings=max_position_embeddings,
418
- quant_config=quant_config,
419
- bias=False,
420
- cache_config=cache_config,
421
- prefix=f"{prefix}.self_attn",
422
- )
423
- self.mlp = ChameleonMLP(
424
- hidden_size=self.hidden_size,
425
- intermediate_size=config.intermediate_size,
426
- hidden_act=config.hidden_act,
427
- quant_config=quant_config,
428
- bias=getattr(config, "mlp_bias", False),
429
- prefix=f"{prefix}.mlp",
430
- )
431
- self.input_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
432
- self.post_attention_layernorm = RMSNorm(
433
- config.hidden_size, eps=config.rms_norm_eps
434
- )
435
-
436
- def forward(
437
- self,
438
- positions: torch.Tensor,
439
- hidden_states: torch.Tensor,
440
- residual: torch.Tensor | None,
441
- ) -> tuple[torch.Tensor, torch.Tensor]:
442
- residual = hidden_states
443
- hidden_states = self.self_attn(
444
- positions=positions,
445
- hidden_states=hidden_states,
446
- )
447
-
448
- hidden_states = self.input_layernorm(hidden_states)
449
- hidden_states = hidden_states + residual
450
-
451
- # Fully Connected
452
- residual = hidden_states
453
- hidden_states = self.mlp(hidden_states)
454
- hidden_states = self.post_attention_layernorm(hidden_states)
455
- hidden_states = residual + hidden_states
456
-
457
- return hidden_states, residual
458
-
459
-
460
-# Copied from transformers.models.chameleon.modeling_chameleon.ChameleonVQVAEVectorQuantizer #noqa
461
-class ChameleonVQVAEVectorQuantizer(nn.Module):
462
- def __init__(self, config: ChameleonVQVAEConfig):
463
- super().__init__()
464
- self.num_embeddings = config.num_embeddings
465
- self.embedding_dim = config.embed_dim
466
- self.beta = getattr(config, "beta", 0.25)
467
-
468
- self.embedding = nn.Embedding(self.num_embeddings, self.embedding_dim)
469
- self.re_embed = self.num_embeddings
470
-
471
- def forward(self, hidden_state: torch.Tensor):
472
- hidden_state = hidden_state.permute(0, 2, 3, 1).contiguous()
473
- hidden_state_flattened = hidden_state.view(-1, self.embedding_dim)
474
-
475
- # distances from z to embeddings e_j (z - e)^2 = z^2 + e^2 - 2 e * z
476
- distances = (
477
- torch.sum(hidden_state_flattened**2, dim=1, keepdim=True)
478
- + torch.sum(self.embedding.weight**2, dim=1)
479
- - 2
480
- * torch.einsum(
481
- "bd,dn->bn",
482
- hidden_state_flattened,
483
- self.embedding.weight.transpose(0, 1),
484
- )
485
- )
486
-
487
- min_encoding_indices = torch.argmin(distances, dim=1)
488
- hidden_state_quant = self.embedding(min_encoding_indices).view(
489
- hidden_state.shape
490
- )
491
-
492
- # compute loss for embedding
493
- loss = torch.mean(
494
- (hidden_state_quant.detach() - hidden_state) ** 2
495
- ) + self.beta * torch.mean((hidden_state_quant - hidden_state.detach()) ** 2)
496
-
497
- # preserve gradients
498
- hidden_state_quant = hidden_state + (hidden_state_quant - hidden_state).detach()
499
-
500
- # reshape back to match original input shape
501
- hidden_state_quant = hidden_state_quant.permute(0, 3, 1, 2).contiguous()
502
-
503
- return hidden_state_quant, loss, min_encoding_indices
504
-
505
-
506
-# Copied from transformers.models.chameleon.modeling_chameleon.ChameleonVQVAEEncoderConvDownsample #noqa
507
-class ChameleonVQVAEEncoderConvDownsample(nn.Module):
508
- def __init__(self, in_channels: int):
509
- super().__init__()
510
- self.conv = Conv2dLayer(
511
- in_channels, in_channels, kernel_size=3, stride=2, padding=0
512
- )
513
-
514
- def forward(self, hidden_states: torch.Tensor):
515
- # no asymmetric padding in torch conv, must do it ourselves
516
- hidden_states = F.pad(hidden_states, pad=(0, 1, 0, 1), mode="constant", value=0)
517
- hidden_states = self.conv(hidden_states)
518
- return hidden_states
519
-
520
-
521
-# Copied from transformers.models.chameleon.modeling_chameleon.ChameleonVQVAEEncoderResnetBlock #noqa
522
-class ChameleonVQVAEEncoderResnetBlock(nn.Module):
523
- def __init__(
524
- self,
525
- config: ChameleonVQVAEConfig,
526
- in_channels: int,
527
- out_channels=None,
528
- conv_shortcut=False,
529
- ):
530
- super().__init__()
531
- self.in_channels = in_channels
532
- self.out_channels = in_channels if out_channels is None else out_channels
533
- self.use_conv_shortcut = conv_shortcut
534
-
535
- self.norm1 = torch.nn.GroupNorm(
536
- num_groups=32, num_channels=in_channels, eps=1e-6, affine=True
537
- )
538
- self.conv1 = Conv2dLayer(
539
- in_channels, out_channels, kernel_size=3, stride=1, padding=1
540
- )
541
- self.norm2 = torch.nn.GroupNorm(
542
- num_groups=32, num_channels=out_channels, eps=1e-6, affine=True
543
- )
544
- self.dropout = torch.nn.Dropout(config.dropout)
545
- self.conv2 = Conv2dLayer(
546
- out_channels, out_channels, kernel_size=3, stride=1, padding=1
547
- )
548
- if self.in_channels != self.out_channels:
549
- if self.use_conv_shortcut:
550
- self.conv_shortcut = Conv2dLayer(
551
- in_channels, out_channels, kernel_size=3, stride=1, padding=1
552
- )
553
- else:
554
- self.nin_shortcut = Conv2dLayer(
555
- in_channels, out_channels, kernel_size=1, stride=1, padding=0
556
- )
557
-
558
- def forward(self, hidden_states: torch.Tensor):
559
- residual = hidden_states
560
- hidden_states = self.norm1(hidden_states)
561
- hidden_states *= torch.sigmoid(hidden_states)
562
- hidden_states = self.conv1(hidden_states)
563
-
564
- hidden_states = self.norm2(hidden_states)
565
- hidden_states *= torch.sigmoid(hidden_states)
566
- hidden_states = self.dropout(hidden_states)
567
- hidden_states = self.conv2(hidden_states)
568
-
569
- if self.in_channels != self.out_channels:
570
- if self.use_conv_shortcut:
571
- residual = self.conv_shortcut(residual)
572
- else:
573
- residual = self.nin_shortcut(residual)
574
-
575
- return residual + hidden_states
576
-
577
-
578
-# Copied from transformers.models.chameleon.modeling_chameleon.ChameleonVQVAEEncoderAttnBlock #noqa
579
-class ChameleonVQVAEEncoderAttnBlock(nn.Module):
580
- def __init__(self, in_channels: int):
581
- super().__init__()
582
- self.in_channels = in_channels
583
-
584
- self.norm = torch.nn.GroupNorm(
585
- num_groups=32, num_channels=in_channels, eps=1e-6, affine=True
586
- )
587
- self.q = Conv2dLayer(
588
- in_channels, in_channels, kernel_size=1, stride=1, padding=0
589
- )
590
- self.k = Conv2dLayer(
591
- in_channels, in_channels, kernel_size=1, stride=1, padding=0
592
- )
593
- self.v = Conv2dLayer(
594
- in_channels, in_channels, kernel_size=1, stride=1, padding=0
595
- )
596
- self.proj_out = Conv2dLayer(
597
- in_channels, in_channels, kernel_size=1, stride=1, padding=0
598
- )
599
-
600
- def forward(self, hidden_states: torch.Tensor):
601
- residual = hidden_states
602
- hidden_states = self.norm(hidden_states)
603
- query_states = self.q(hidden_states)
604
- key_states = self.k(hidden_states)
605
- value_states = self.v(hidden_states)
606
-
607
- # compute attention
608
- batch_size, channels, height, width = query_states.shape
609
- query_states = query_states.reshape(
610
- batch_size, channels, height * width
611
- ).permute(0, 2, 1)
612
- key_states = key_states.reshape(batch_size, channels, height * width)
613
- attn_weights = torch.bmm(query_states, key_states)
614
- attn_weights = attn_weights * (int(channels) ** (-0.5))
615
- attn_weights = F.softmax(attn_weights, dim=2)
616
-
617
- # attend to values
618
- value_states = value_states.reshape(batch_size, channels, height * width)
619
- attn_weights = attn_weights.permute(0, 2, 1)
620
- attn_output = torch.bmm(value_states, attn_weights).reshape(
621
- batch_size, channels, height, width
622
- )
623
-
624
- attn_output = self.proj_out(attn_output)
625
- return residual + attn_output
626
-
627
-
628
-# Copied from transformers.models.chameleon.modeling_chameleon.ChameleonVQVAEEncoder #noqa
629
-class ChameleonVQVAEEncoder(nn.Module):
630
- def __init__(self, config: ChameleonVQVAEConfig):
631
- super().__init__()
632
-
633
- self.num_resolutions = len(config.channel_multiplier)
634
- self.num_res_blocks = config.num_res_blocks
635
- base_channels = config.base_channels
636
- resolution = config.resolution
637
- in_channels = config.in_channels
638
- double_latent = config.double_latent
639
- latent_channels = config.latent_channels
640
- channel_multiplier = config.channel_multiplier
641
-
642
- self.conv_in = Conv2dLayer(
643
- in_channels, base_channels, kernel_size=3, stride=1, padding=1
644
- )
645
-
646
- curr_res = resolution
647
- in_channel_multiplier = (1,) + tuple(channel_multiplier)
648
- self.in_channel_multiplier = in_channel_multiplier
649
- self.down = nn.ModuleList()
650
- for i_level in range(self.num_resolutions):
651
- block = nn.ModuleList()
652
- attn = nn.ModuleList()
653
- block_in = base_channels * in_channel_multiplier[i_level]
654
- block_out = base_channels * channel_multiplier[i_level]
655
- for i_block in range(self.num_res_blocks):
656
- block.append(
657
- ChameleonVQVAEEncoderResnetBlock(
658
- config=config,
659
- in_channels=block_in,
660
- out_channels=block_out,
661
- )
662
- )
663
- block_in = block_out
664
- if (
665
- config.attn_resolutions is not None
666
- and curr_res in config.attn_resolutions
667
- and config.attn_type == "vanilla"
668
- ):
669
- attn.append(ChameleonVQVAEEncoderAttnBlock(block_in))
670
-
671
- down = nn.Module()
672
- down.block = block
673
- down.attn = attn
674
- if i_level != self.num_resolutions - 1:
675
- down.downsample = ChameleonVQVAEEncoderConvDownsample(block_in)
676
- curr_res = curr_res // 2
677
- self.down.append(down)
678
-
679
- self.mid = nn.Module()
680
- self.mid.block_1 = ChameleonVQVAEEncoderResnetBlock(
681
- config=config,
682
- in_channels=block_in,
683
- out_channels=block_in,
684
- )
685
- self.mid.attn_1 = (
686
- ChameleonVQVAEEncoderAttnBlock(block_in)
687
- if config.attn_type == "vanilla"
688
- else nn.Identity()
689
- )
690
- self.mid.block_2 = ChameleonVQVAEEncoderResnetBlock(
691
- config=config,
692
- in_channels=block_in,
693
- out_channels=block_in,
694
- )
695
-
696
- self.norm_out = torch.nn.GroupNorm(
697
- num_groups=32, num_channels=block_in, eps=1e-6, affine=True
698
- )
699
- self.conv_out = Conv2dLayer(
700
- block_in,
701
- 2 * latent_channels if double_latent else latent_channels,
702
- kernel_size=3,
703
- stride=1,
704
- padding=1,
705
- )
706
-
707
- def forward(self, pixel_values: torch.Tensor):
708
- pixel_values = pixel_values.to(self.conv_in.weight.dtype)
709
-
710
- # downsampling
711
- hidden_states = [self.conv_in(pixel_values)]
712
- for i_level in range(self.num_resolutions):
713
- for i_block in range(self.num_res_blocks):
714
- hidden_state = self.down[i_level].block[i_block](hidden_states[-1])
715
- if len(self.down[i_level].attn) > 0:
716
- hidden_state = self.down[i_level].attn[i_block](hidden_state)
717
- hidden_states.append(hidden_state)
718
- if i_level != self.num_resolutions - 1:
719
- hidden_states.append(self.down[i_level].downsample(hidden_states[-1]))
720
-
721
- # middle
722
- last_hidden_state = hidden_states[-1]
723
- last_hidden_state = self.mid.block_1(last_hidden_state)
724
- last_hidden_state = self.mid.attn_1(last_hidden_state)
725
- last_hidden_state = self.mid.block_2(last_hidden_state)
726
-
727
- # end
728
- last_hidden_state = self.norm_out(last_hidden_state)
729
- last_hidden_state *= torch.sigmoid(last_hidden_state)
730
- last_hidden_state = self.conv_out(last_hidden_state)
731
- return last_hidden_state
732
-
733
-
734
-# Adapted from transformers.models.chameleon.modeling_chameleon.ChameleonVQVAE #noqa
735
-class ChameleonVQVAE(nn.Module):
736
- def __init__(self, config: ChameleonVQVAEConfig):
737
- super().__init__()
738
- self.encoder = ChameleonVQVAEEncoder(config)
739
- self.quantize = ChameleonVQVAEVectorQuantizer(config)
740
- self.quant_conv = Conv2dLayer(config.latent_channels, config.embed_dim, 1)
741
- self.post_quant_conv = Conv2dLayer(config.embed_dim, config.latent_channels, 1)
742
- self.eval() # Chameleon's VQ model is frozen
743
-
744
- def encode(
745
- self, pixel_values: torch.Tensor
746
- ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
747
- hidden_states = self.encoder(pixel_values)
748
- hidden_states = self.quant_conv(hidden_states)
749
- quant, emb_loss, indices = self.quantize(hidden_states)
750
- return quant, emb_loss, indices
751
-
752
-
753
-# Copied from transformers.models.chameleon.modeling_chameleon.ChameleonImageVocabularyMapping #noqa
754
-class ChameleonImageVocabularyMapping:
755
- """
756
- A class for mapping discrete image tokens from VQGAN to BPE tokens.
757
- """
758
-
759
- def __init__(self, vocab_map: dict[str, int]):
760
- self.vocab_map = vocab_map
761
- self.image_token_id = vocab_map.get("<image>")
762
-
763
- @cached_property
764
- def val2name(self):
765
- return {v: k for k, v in self.vocab_map.items()}
766
-
767
- @cached_property
768
- def image_tokens(self):
769
- return sorted(
770
- [val for name, val in self.vocab_map.items() if name.startswith("IMGIMG")]
771
- )
772
-
773
- @cached_property
774
- def bpe2img(self):
775
- img_tkn_chr_mapping = {chr(ord("A") + i): str(i) for i in range(10)}
776
-
777
- def remap(old_name: str) -> str:
778
- return "".join(
779
- img_tkn_chr_mapping.get(c, c) for c in old_name[len("IMGIMG") : -1]
780
- )
781
-
782
- return {tok: int(remap(self.val2name[tok])) for tok in self.image_tokens}
783
-
784
- @cached_property
785
- def img2bpe(self):
786
- return {v: k for k, v in self.bpe2img.items()}
787
-
788
- @cached_property
789
- def bpe2img_search_tensors(self):
790
- return torch.tensor(sorted(self.bpe2img.keys())), torch.tensor(
791
- sorted(self.bpe2img.values())
792
- )
793
-
794
- @cached_property
795
- def img2bpe_mapping_tensor(self):
796
- mapping = torch.zeros(max(self.img2bpe.keys()) + 1, dtype=torch.int)
797
- for k, v in self.img2bpe.items():
798
- mapping[k] = v
799
- return mapping
800
-
801
- def convert_img2bpe(self, img_batch: torch.Tensor) -> torch.Tensor:
802
- device = img_batch.device
803
- # Cache a per-device copy of the (small, static) mapping tensor so we
804
- # can index entirely on `device` instead of forcing a D2H on
805
- # `img_batch` and an H2D on the result.
806
- cache = getattr(self, "_img2bpe_mapping_cache", None)
807
- if cache is None:
808
- cache = {}
809
- self._img2bpe_mapping_cache = cache
810
- mapping_on_device = cache.get(device)
811
- if mapping_on_device is None:
812
- mapping_on_device = async_tensor_h2d(
813
- self.img2bpe_mapping_tensor, device=device
814
- )
815
- cache[device] = mapping_on_device
816
- return mapping_on_device[img_batch]
817
-
818
-
819
-class ChameleonModel(nn.Module):
820
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
821
- super().__init__()
822
-
823
- config = vllm_config.model_config.hf_config
824
- cache_config = vllm_config.cache_config
825
- quant_config = vllm_config.quant_config
826
-
827
- self.config = config
828
- self.vocab_size = config.vocab_size
829
- self.embed_tokens = VocabParallelEmbedding(
830
- self.vocab_size,
831
- config.hidden_size,
832
- )
833
- self.vocabulary_mapping = ChameleonImageVocabularyMapping(config.vocabulary_map)
834
- decoder_layer = (
835
- ChameleonDecoderLayer
836
- if not self.config.swin_norm
837
- else ChameleonSwinDecoderLayer
838
- )
839
-
840
- self.start_layer, self.end_layer, self.layers = make_layers(
841
- config.num_hidden_layers,
842
- lambda prefix: decoder_layer(
843
- config=config,
844
- cache_config=cache_config,
845
- quant_config=quant_config,
846
- prefix=prefix,
847
- ),
848
- prefix=f"{prefix}.layers",
849
- )
850
-
851
- self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
852
- self.vqmodel = ChameleonVQVAE(config.vq_config)
853
- self.make_empty_intermediate_tensors = make_empty_intermediate_tensors_factory(
854
- ["hidden_states", "residual"], config.hidden_size
855
- )
856
-
857
- def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
858
- return self.embed_tokens(input_ids)
859
-
860
- def get_image_tokens(self, pixel_values: torch.Tensor) -> torch.Tensor:
861
- """
862
- Tokenizes images into discrete tokens with VQGAN module. Converts
863
- obtained image tokens into BPE tokens and wraps with "boi" and "eoi"
864
- special tokens.
865
- """
866
- batch_size = pixel_values.shape[0]
867
- _, _, image_toks = self.vqmodel.encode(pixel_values)
868
- bpe_toks = self.vocabulary_mapping.convert_img2bpe(image_toks)
869
- bpe_toks = bpe_toks.view(batch_size, -1)
870
- return bpe_toks
871
-
872
- def forward(
873
- self,
874
- input_ids: torch.Tensor | None,
875
- positions: torch.Tensor,
876
- intermediate_tensors: IntermediateTensors | None,
877
- inputs_embeds: torch.Tensor | None = None,
878
- ) -> torch.Tensor | IntermediateTensors:
879
- if get_pp_group().is_first_rank:
880
- if inputs_embeds is not None:
881
- hidden_states = inputs_embeds
882
- else:
883
- hidden_states = self.embed_input_ids(input_ids)
884
- residual = None
885
- else:
886
- assert intermediate_tensors is not None
887
- hidden_states = intermediate_tensors["hidden_states"]
888
- residual = intermediate_tensors["residual"]
889
- for layer in islice(self.layers, self.start_layer, self.end_layer):
890
- hidden_states, residual = layer(
891
- positions,
892
- hidden_states,
893
- residual,
894
- )
895
- if not get_pp_group().is_last_rank:
896
- return IntermediateTensors(
897
- {"hidden_states": hidden_states, "residual": residual}
898
- )
899
- hidden_states, _ = self.norm(hidden_states, residual)
900
- return hidden_states
901
-
902
-
903
-@MULTIMODAL_REGISTRY.register_processor(
904
- ChameleonMultiModalProcessor,
905
- info=ChameleonProcessingInfo,
906
- dummy_inputs=ChameleonDummyInputsBuilder,
907
-)
908
-class ChameleonForConditionalGeneration(
909
- nn.Module, SupportsMultiModal, SupportsPP, SupportsQuant
910
-):
911
- packed_modules_mapping = {
912
- "qkv_proj": ["q_proj", "k_proj", "v_proj"],
913
- "gate_up_proj": ["gate_proj", "up_proj"],
914
- }
915
-
916
- hf_to_vllm_mapper = WeightsMapper(
917
- orig_to_new_stacked={
918
- ".q_proj": (".qkv_proj", "q"),
919
- ".k_proj": (".qkv_proj", "k"),
920
- ".v_proj": (".qkv_proj", "v"),
921
- ".gate_proj": (".gate_up_proj", 0),
922
- ".up_proj": (".gate_up_proj", 1),
923
- },
924
- )
925
-
926
- @classmethod
927
- def get_placeholder_str(cls, modality: str, i: int) -> str | None:
928
- if modality.startswith("image"):
929
- return "<image>"
930
-
931
- raise ValueError("Only image modality is supported")
932
-
933
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
934
- super().__init__()
935
- config = vllm_config.model_config.hf_config
936
- multimodal_config = vllm_config.model_config.multimodal_config
937
- self.config = config
938
- self.multimodal_config = multimodal_config
939
-
940
- with self._mark_composite_model(
941
- vllm_config,
942
- language_targets=(
943
- ChameleonDecoderLayer
944
- if not self.config.swin_norm
945
- else ChameleonSwinDecoderLayer
946
- ),
947
- tower_targets={"image": ChameleonVQVAE},
948
- ):
949
- self.model = ChameleonModel(
950
- vllm_config=vllm_config,
951
- prefix=maybe_prefix(prefix, "model"),
952
- )
953
-
954
- self.lm_head = ParallelLMHead(
955
- config.vocab_size,
956
- config.hidden_size,
957
- prefix=maybe_prefix(prefix, "lm_head"),
958
- )
959
- if config.tie_word_embeddings:
960
- self.lm_head = self.lm_head.tie_weights(self.model.embed_tokens)
961
-
962
- logit_scale = getattr(config, "logit_scale", 1.0)
963
- self.logits_processor = LogitsProcessor(config.vocab_size, scale=logit_scale)
964
- self.make_empty_intermediate_tensors = (
965
- self.model.make_empty_intermediate_tensors
966
- )
967
-
968
- def _parse_and_validate_image_input(
969
- self, **kwargs: object
970
- ) -> ChameleonImagePixelInputs | None:
971
- pixel_values = kwargs.pop("pixel_values", None)
972
-
973
- if pixel_values is None:
974
- return None
975
-
976
- vq_config: ChameleonVQVAEConfig = self.config.vq_config
977
- expected_h = expected_w = vq_config.resolution
978
-
979
- return ChameleonImagePixelInputs(
980
- type="pixel_values",
981
- data=pixel_values,
982
- resolve_bindings={"h": expected_h, "w": expected_w},
983
- )
984
-
985
- def embed_multimodal(self, **kwargs: object) -> MultiModalEmbeddings:
986
- image_input = self._parse_and_validate_image_input(**kwargs)
987
- if image_input is None:
988
- return []
989
- assert self.model.vqmodel is not None
990
- image_tokens = self.model.get_image_tokens(
991
- image_input["data"].to(self.config.dtype)
992
- )
993
- vision_embeddings = self.model.embed_input_ids(image_tokens)
994
- return vision_embeddings
995
-
996
- def forward(
997
- self,
998
- input_ids: torch.Tensor | None,
999
- positions: torch.Tensor,
1000
- intermediate_tensors: IntermediateTensors | None = None,
1001
- inputs_embeds: torch.Tensor | None = None,
1002
- **kwargs,
1003
- ) -> torch.Tensor | IntermediateTensors:
1004
- if intermediate_tensors is not None:
1005
- inputs_embeds = None
1006
-
1007
- hidden_states = self.model(
1008
- input_ids, positions, intermediate_tensors, inputs_embeds=inputs_embeds
1009
- )
1010
- return hidden_states
1011
-
1012
- def compute_logits(
1013
- self,
1014
- hidden_states: torch.Tensor,
1015
- ) -> torch.Tensor | None:
1016
- logits = self.logits_processor(self.lm_head, hidden_states)
1017
-
1018
- # Disallow image tokens which does not include special
1019
- # begin-image and end-image tokens
1020
- if logits is not None:
1021
- # Cache a per-device index tensor for the (static) image-token
1022
- # set, and use `index_fill_` instead of advanced-index assign
1023
- # so the scatter runs entirely on device (no host roundtrip
1024
- # for the scalar fill value or for the Python-list indices).
1025
- cache = getattr(self, "_image_tokens_index_cache", None)
1026
- if cache is None:
1027
- cache = {}
1028
- self._image_tokens_index_cache = cache
1029
- image_tokens_idx = cache.get(logits.device)
1030
- if image_tokens_idx is None:
1031
- image_tokens_idx = async_tensor_h2d(
1032
- self.model.vocabulary_mapping.image_tokens,
1033
- dtype=torch.long,
1034
- device=logits.device,
1035
- )
1036
- cache[logits.device] = image_tokens_idx
1037
- logits.index_fill_(1, image_tokens_idx, torch.finfo(logits.dtype).min)
1038
-
1039
- return logits
1040
-
1041
- def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
1042
- loader = AutoWeightsLoader(self)
1043
- return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
vllm/model_executor/models/cheers.py
deleted
-734
@@ -1,734 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-"""Inference-only Cheers (UMM) model compatible with HuggingFace weights.
4
-
5
-Cheers is a unified multimodal model for image understanding and generation.
6
-For vLLM, we focus on the image understanding (vision-to-text) capabilities.
7
-The image generation part (gen_projector, hi_gate, etc.) is not supported,
8
-but the VAE encoder + decoder projector are required for image understanding.
9
-"""
10
-
11
-import math
12
-from collections.abc import Iterable, Mapping, Sequence
13
-from typing import Any, Literal, TypeAlias
14
-
15
-import torch
16
-import torch.nn as nn
17
-import torch.nn.functional as F
18
-from einops import rearrange
19
-
20
-from vllm.config import VllmConfig
21
-from vllm.config.multimodal import BaseDummyOptions
22
-from vllm.inputs import MultiModalDataDict
23
-from vllm.logger import init_logger
24
-from vllm.model_executor.layers.quantization import QuantizationConfig
25
-from vllm.multimodal import MULTIMODAL_REGISTRY
26
-from vllm.multimodal.inputs import (
27
- MultiModalFieldConfig,
28
- MultiModalKwargsItems,
29
-)
30
-from vllm.multimodal.parse import MultiModalDataItems
31
-from vllm.multimodal.processing import (
32
- BaseDummyInputsBuilder,
33
- BaseMultiModalProcessor,
34
- BaseProcessingInfo,
35
- PromptReplacement,
36
-)
37
-from vllm.sequence import IntermediateTensors
38
-from vllm.transformers_utils.processors.cheers import CheersProcessor
39
-from vllm.utils.tensor_schema import TensorSchema
40
-
41
-from .interfaces import (
42
- MultiModalEmbeddings,
43
- SupportsLoRA,
44
- SupportsMultiModal,
45
- SupportsPP,
46
-)
47
-from .siglip import SiglipVisionModel
48
-from .utils import (
49
- AutoWeightsLoader,
50
- WeightsMapper,
51
- init_vllm_registered_model,
52
- maybe_prefix,
53
-)
54
-
55
-logger = init_logger(__name__)
56
-
57
-
58
-# ── VAE components (needed for image understanding pipeline) ────────
59
-
60
-
61
-def _swish(x: torch.Tensor) -> torch.Tensor:
62
- return x * torch.sigmoid(x)
63
-
64
-
65
-class _AttnBlock(nn.Module):
66
- def __init__(self, in_channels: int):
67
- super().__init__()
68
- self.norm = nn.GroupNorm(32, in_channels, eps=1e-6, affine=True)
69
- self.q = nn.Conv2d(in_channels, in_channels, 1)
70
- self.k = nn.Conv2d(in_channels, in_channels, 1)
71
- self.v = nn.Conv2d(in_channels, in_channels, 1)
72
- self.proj_out = nn.Conv2d(in_channels, in_channels, 1)
73
-
74
- def forward(self, x: torch.Tensor) -> torch.Tensor:
75
- h_ = self.norm(x)
76
- q = self.q(h_)
77
- k = self.k(h_)
78
- v = self.v(h_)
79
- b, c, h, w = q.shape
80
- q = rearrange(q, "b c h w -> b 1 (h w) c").contiguous()
81
- k = rearrange(k, "b c h w -> b 1 (h w) c").contiguous()
82
- v = rearrange(v, "b c h w -> b 1 (h w) c").contiguous()
83
- h_ = F.scaled_dot_product_attention(q, k, v)
84
- h_ = rearrange(h_, "b 1 (h w) c -> b c h w", h=h, w=w, c=c, b=b)
85
- return x + self.proj_out(h_)
86
-
87
-
88
-class _ResnetBlock(nn.Module):
89
- def __init__(self, in_channels: int, out_channels: int):
90
- super().__init__()
91
- self.in_channels = in_channels
92
- self.out_channels = out_channels
93
- self.norm1 = nn.GroupNorm(32, in_channels, eps=1e-6, affine=True)
94
- self.conv1 = nn.Conv2d(in_channels, out_channels, 3, 1, 1)
95
- self.norm2 = nn.GroupNorm(32, out_channels, eps=1e-6, affine=True)
96
- self.conv2 = nn.Conv2d(out_channels, out_channels, 3, 1, 1)
97
- if in_channels != out_channels:
98
- self.nin_shortcut = nn.Conv2d(in_channels, out_channels, 1)
99
-
100
- def forward(self, x: torch.Tensor) -> torch.Tensor:
101
- h = _swish(self.norm1(x))
102
- h = self.conv1(h)
103
- h = _swish(self.norm2(h))
104
- h = self.conv2(h)
105
- if self.in_channels != self.out_channels:
106
- x = self.nin_shortcut(x)
107
- return x + h
108
-
109
-
110
-class _Downsample(nn.Module):
111
- def __init__(self, in_channels: int):
112
- super().__init__()
113
- self.conv = nn.Conv2d(in_channels, in_channels, 3, stride=2, padding=0)
114
-
115
- def forward(self, x: torch.Tensor) -> torch.Tensor:
116
- x = F.pad(x, (0, 1, 0, 1), mode="constant", value=0)
117
- return self.conv(x)
118
-
119
-
120
-class _Upsample(nn.Module):
121
- def __init__(self, in_channels: int):
122
- super().__init__()
123
- self.conv = nn.Conv2d(in_channels, in_channels, 3, 1, 1)
124
-
125
- def forward(self, x: torch.Tensor) -> torch.Tensor:
126
- x = F.interpolate(x, scale_factor=2.0, mode="nearest")
127
- return self.conv(x)
128
-
129
-
130
-_VAE_ENCODER_DEFAULTS = {
131
- "in_channels": 3,
132
- "ch": 128,
133
- "ch_mult": [1, 2, 4, 4],
134
- "num_res_blocks": 2,
135
- "z_channels": 32,
136
-}
137
-_VAE_DECODER_DEFAULTS = {
138
- "in_channels": 3,
139
- "out_ch": 3,
140
- "ch": 128,
141
- "ch_mult": [1, 2, 4, 4],
142
- "num_res_blocks": 2,
143
- "z_channels": 32,
144
-}
145
-
146
-
147
-def _cfg(config, key, defaults=None):
148
- """Access config attribute whether it's a dict or namespace object."""
149
- if isinstance(config, dict):
150
- if key in config:
151
- return config[key]
152
- if defaults and key in defaults:
153
- return defaults[key]
154
- raise KeyError(f"Key '{key}' not found in config dict: {list(config.keys())}")
155
- return getattr(config, key)
156
-
157
-
158
-class CheersVAEEncoder(nn.Module):
159
- """VAE encoder from the Cheers/UMM model."""
160
-
161
- def __init__(self, config):
162
- super().__init__()
163
- d = _VAE_ENCODER_DEFAULTS
164
- ch = _cfg(config, "ch", d)
165
- ch_mult = _cfg(config, "ch_mult", d)
166
- num_res_blocks = _cfg(config, "num_res_blocks", d)
167
- z_channels = _cfg(config, "z_channels", d)
168
- in_channels = _cfg(config, "in_channels", d)
169
- num_resolutions = len(ch_mult)
170
-
171
- self.quant_conv = nn.Conv2d(2 * z_channels, 2 * z_channels, 1)
172
- self.conv_in = nn.Conv2d(in_channels, ch, 3, 1, 1)
173
-
174
- in_ch_mult = (1,) + tuple(ch_mult)
175
- self.down = nn.ModuleList()
176
- block_in = ch
177
- for i_level in range(num_resolutions):
178
- block = nn.ModuleList()
179
- attn = nn.ModuleList()
180
- block_in = ch * in_ch_mult[i_level]
181
- block_out = ch * ch_mult[i_level]
182
- for _ in range(num_res_blocks):
183
- block.append(_ResnetBlock(block_in, block_out))
184
- block_in = block_out
185
- down = nn.Module()
186
- down.block = block
187
- down.attn = attn
188
- if i_level != num_resolutions - 1:
189
- down.downsample = _Downsample(block_in)
190
- self.down.append(down)
191
-
192
- self.mid = nn.Module()
193
- self.mid.block_1 = _ResnetBlock(block_in, block_in)
194
- self.mid.attn_1 = _AttnBlock(block_in)
195
- self.mid.block_2 = _ResnetBlock(block_in, block_in)
196
-
197
- self.norm_out = nn.GroupNorm(32, block_in, eps=1e-6, affine=True)
198
- self.conv_out = nn.Conv2d(block_in, 2 * z_channels, 3, 1, 1)
199
- self._num_resolutions = num_resolutions
200
- self._num_res_blocks = num_res_blocks
201
-
202
- def forward(self, x: torch.Tensor) -> torch.Tensor:
203
- hs = [self.conv_in(x)]
204
- for i_level in range(self._num_resolutions):
205
- for i_block in range(self._num_res_blocks):
206
- h = self.down[i_level].block[i_block](hs[-1])
207
- if len(self.down[i_level].attn) > 0:
208
- h = self.down[i_level].attn[i_block](h)
209
- hs.append(h)
210
- if hasattr(self.down[i_level], "downsample"):
211
- hs.append(self.down[i_level].downsample(hs[-1]))
212
- h = hs[-1]
213
- h = self.mid.block_1(h)
214
- h = self.mid.attn_1(h)
215
- h = self.mid.block_2(h)
216
- h = _swish(self.norm_out(h))
217
- h = self.conv_out(h)
218
- h = self.quant_conv(h)
219
- return h
220
-
221
-
222
-class CheersVAEDecoder(nn.Module):
223
- """VAE decoder (used inside VAEDecoderProjector)."""
224
-
225
- def __init__(self, config):
226
- super().__init__()
227
- d = _VAE_DECODER_DEFAULTS
228
- ch = _cfg(config, "ch", d)
229
- ch_mult = _cfg(config, "ch_mult", d)
230
- num_res_blocks = _cfg(config, "num_res_blocks", d)
231
- z_channels = _cfg(config, "z_channels", d)
232
- out_ch = _cfg(config, "out_ch", d)
233
- num_resolutions = len(ch_mult)
234
-
235
- self.post_quant_conv = nn.Conv2d(z_channels, z_channels, 1)
236
- block_in = ch * ch_mult[num_resolutions - 1]
237
- self.conv_in = nn.Conv2d(z_channels, block_in, 3, 1, 1)
238
-
239
- self.mid = nn.Module()
240
- self.mid.block_1 = _ResnetBlock(block_in, block_in)
241
- self.mid.attn_1 = _AttnBlock(block_in)
242
- self.mid.block_2 = _ResnetBlock(block_in, block_in)
243
-
244
- self.up = nn.ModuleList()
245
- for i_level in reversed(range(num_resolutions)):
246
- block = nn.ModuleList()
247
- attn = nn.ModuleList()
248
- block_out = ch * ch_mult[i_level]
249
- for _ in range(num_res_blocks + 1):
250
- block.append(_ResnetBlock(block_in, block_out))
251
- block_in = block_out
252
- up = nn.Module()
253
- up.block = block
254
- up.attn = attn
255
- if i_level != 0:
256
- up.upsample = _Upsample(block_in)
257
- self.up.insert(0, up)
258
-
259
- self.norm_out = nn.GroupNorm(32, block_in, eps=1e-6, affine=True)
260
- self.conv_out = nn.Conv2d(block_in, out_ch, 3, 1, 1)
261
- self._num_resolutions = num_resolutions
262
- self._num_res_blocks = num_res_blocks
263
-
264
- def forward(self, z: torch.Tensor) -> torch.Tensor:
265
- z = self.post_quant_conv(z)
266
- upscale_dtype = next(self.up.parameters()).dtype
267
- h = self.conv_in(z)
268
- h = self.mid.block_1(h)
269
- h = self.mid.attn_1(h)
270
- h = self.mid.block_2(h)
271
- h = h.to(upscale_dtype)
272
- for i_level in reversed(range(self._num_resolutions)):
273
- for i_block in range(self._num_res_blocks + 1):
274
- h = self.up[i_level].block[i_block](h)
275
- if len(self.up[i_level].attn) > 0:
276
- h = self.up[i_level].attn[i_block](h)
277
- if i_level != 0:
278
- h = self.up[i_level].upsample(h)
279
- h = _swish(self.norm_out(h))
280
- return self.conv_out(h)
281
-
282
-
283
-class CheersVAEModel(nn.Module):
284
- """VAE model with encoder only (for image understanding)."""
285
-
286
- def __init__(self, config):
287
- super().__init__()
288
- enc_cfg = _cfg(config, "vae_encoder_config")
289
- self.encoder = CheersVAEEncoder(enc_cfg)
290
- self.ps = [2, 2]
291
- z_ch = _cfg(enc_cfg, "z_channels", _VAE_ENCODER_DEFAULTS)
292
- self.bn = nn.BatchNorm2d(
293
- math.prod(self.ps) * z_ch,
294
- eps=1e-4,
295
- momentum=0.1,
296
- affine=False,
297
- track_running_stats=True,
298
- )
299
-
300
- def encode(self, x: torch.Tensor) -> torch.Tensor:
301
- self.bn.eval()
302
- moments = self.encoder(x)
303
- mean = torch.chunk(moments, 2, dim=1)[0]
304
- z = rearrange(
305
- mean,
306
- "... c (i pi) (j pj) -> ... (c pi pj) i j",
307
- pi=self.ps[0],
308
- pj=self.ps[1],
309
- )
310
- return self.bn(z)
311
-
312
-
313
-class CheersVAEDecoderProjector(nn.Module):
314
- """VAE decoder projector that converts latent back to pixel-like space."""
315
-
316
- def __init__(self, config):
317
- super().__init__()
318
- dec_cfg = _cfg(config, "vae_decoder_config")
319
- enc_cfg = _cfg(config, "vae_encoder_config")
320
- self.decoder = CheersVAEDecoder(dec_cfg)
321
- self.ps = [2, 2]
322
- z_ch = _cfg(enc_cfg, "z_channels", _VAE_ENCODER_DEFAULTS)
323
- self.bn = nn.BatchNorm2d(
324
- math.prod(self.ps) * z_ch,
325
- eps=1e-4,
326
- momentum=0.1,
327
- affine=False,
328
- track_running_stats=True,
329
- )
330
-
331
- def forward(self, z: torch.Tensor) -> torch.Tensor:
332
- self.bn.eval()
333
- s = torch.sqrt(self.bn.running_var.view(1, -1, 1, 1) + 1e-4)
334
- m = self.bn.running_mean.view(1, -1, 1, 1)
335
- z = z * s + m
336
- z = rearrange(
337
- z,
338
- "... (c pi pj) i j -> ... c (i pi) (j pj)",
339
- pi=self.ps[0],
340
- pj=self.ps[1],
341
- )
342
- return self.decoder(z)
343
-
344
-
345
-class CheersImagePixelInputs(TensorSchema):
346
- """
347
- Dimensions:
348
- - bn: Batch size * number of images
349
- - c: Number of channels (3)
350
- - h: Height of each image
351
- - w: Width of each image
352
- """
353
-
354
- type: Literal["pixel_values"]
355
- pixel_values: torch.Tensor # Shape: (bn, 3, h, w)
356
-
357
-
358
-CheersImageInputs: TypeAlias = CheersImagePixelInputs
359
-
360
-
361
-class CheersUndProjector(nn.Module):
362
- """Understanding projector that maps vision features to LLM dimension
363
- with 2x2 spatial compression (4x token reduction)."""
364
-
365
- def __init__(
366
- self,
367
- image_embed_dim: int,
368
- text_embed_dim: int,
369
- compression_factor: tuple[int, int] = (2, 2),
370
- quant_config: QuantizationConfig | None = None,
371
- prefix: str = "",
372
- ):
373
- super().__init__()
374
- self.image_embed_dim = image_embed_dim
375
- self.text_embed_dim = text_embed_dim
376
- self.compression_factor = compression_factor
377
- self.layernorm = nn.LayerNorm(image_embed_dim)
378
- hidden_size = image_embed_dim * (compression_factor[0] * compression_factor[1])
379
- self.mlp = nn.Sequential(
380
- nn.Linear(hidden_size, hidden_size),
381
- nn.GELU(),
382
- nn.Linear(hidden_size, text_embed_dim),
383
- )
384
-
385
- def forward(self, x: torch.Tensor) -> torch.Tensor:
386
- x = self.layernorm(x)
387
- height = width = int(x.size(1) ** 0.5)
388
- x = x.permute(0, 2, 1).unflatten(-1, (height, width))
389
- batch_size, dim, height, width = x.shape
390
- unfolded = x.unfold(
391
- 2, self.compression_factor[0], self.compression_factor[0]
392
- ).unfold(3, self.compression_factor[1], self.compression_factor[1])
393
- unfolded = unfolded.contiguous().view(
394
- batch_size,
395
- dim,
396
- -1,
397
- self.compression_factor[0] * self.compression_factor[1],
398
- )
399
- unfolded = (
400
- unfolded.permute(0, 2, 3, 1)
401
- .contiguous()
402
- .view(
403
- batch_size,
404
- -1,
405
- dim * self.compression_factor[0] * self.compression_factor[1],
406
- )
407
- )
408
- return self.mlp(unfolded)
409
-
410
-
411
-class CheersProcessingInfo(BaseProcessingInfo):
412
- """Processing information for Cheers model."""
413
-
414
- def get_hf_processor(self, **kwargs: object) -> CheersProcessor:
415
- from vllm.transformers_utils.processor import cached_get_image_processor
416
-
417
- image_processor = cached_get_image_processor(
418
- self.ctx.model_config.model,
419
- revision=self.ctx.model_config.revision,
420
- trust_remote_code=self.ctx.model_config.trust_remote_code,
421
- )
422
-
423
- tokenizer = self.get_tokenizer()
424
-
425
- return CheersProcessor(
426
- image_processor=image_processor,
427
- tokenizer=tokenizer,
428
- **kwargs,
429
- )
430
-
431
- def get_supported_mm_limits(self) -> Mapping[str, int | None]:
432
- return {"image": None}
433
-
434
- def get_mm_max_tokens_per_item(
435
- self,
436
- seq_len: int,
437
- mm_counts: Mapping[str, int],
438
- ) -> Mapping[str, int]:
439
- hf_config = self.get_hf_config()
440
- vit_config = hf_config.vision_representation_config
441
- patch_size = vit_config.patch_size
442
- image_size = vit_config.image_size
443
- num_patches = (image_size // patch_size) ** 2
444
- # After 2x2 compression, tokens reduce by 4x
445
- num_tokens = num_patches // 4
446
- return {"image": num_tokens}
447
-
448
- def get_num_image_tokens(
449
- self,
450
- *,
451
- image_width: int,
452
- image_height: int,
453
- ) -> int:
454
- hf_config = self.get_hf_config()
455
- vit_config = hf_config.vision_representation_config
456
- patch_size = vit_config.patch_size
457
- image_size = vit_config.image_size
458
- num_patches = (image_size // patch_size) ** 2
459
- return num_patches // 4
460
-
461
-
462
-class CheersDummyInputsBuilder(BaseDummyInputsBuilder[CheersProcessingInfo]):
463
- """Build dummy inputs for Cheers model profiling."""
464
-
465
- def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
466
- num_images = mm_counts.get("image", 0)
467
- return "<|image_pad|>" * num_images
468
-
469
- def get_dummy_mm_data(
470
- self,
471
- seq_len: int,
472
- mm_counts: Mapping[str, int],
473
- mm_options: Mapping[str, BaseDummyOptions] | None = None,
474
- ) -> MultiModalDataDict:
475
- num_images = mm_counts.get("image", 0)
476
- hf_config = self.info.get_hf_config()
477
- vit_config = hf_config.vision_representation_config
478
- image_size = vit_config.image_size
479
- image_overrides = mm_options.get("image") if mm_options else None
480
-
481
- return {
482
- "image": self._get_dummy_images(
483
- width=image_size,
484
- height=image_size,
485
- num_images=num_images,
486
- overrides=image_overrides,
487
- ),
488
- }
489
-
490
-
491
-class CheersMultiModalProcessor(BaseMultiModalProcessor[CheersProcessingInfo]):
492
- """Multimodal processor for Cheers model."""
493
-
494
- def _get_prompt_updates(
495
- self,
496
- mm_items: MultiModalDataItems,
497
- hf_processor_mm_kwargs: Mapping[str, Any],
498
- out_mm_kwargs: MultiModalKwargsItems,
499
- ) -> Sequence[PromptReplacement]:
500
- hf_config = self.info.get_hf_config()
501
- vit_config = hf_config.vision_representation_config
502
- patch_size = vit_config.patch_size
503
- image_size = vit_config.image_size
504
-
505
- tokenizer = self.info.get_tokenizer()
506
- image_token_id = tokenizer.get_vocab().get("<|image_pad|>")
507
- if image_token_id is None:
508
- raise ValueError(
509
- "Image token '<|image_pad|>' not found in tokenizer vocabulary"
510
- )
511
-
512
- def get_replacement_cheers(item_idx: int):
513
- num_patches = (image_size // patch_size) ** 2
514
- num_tokens = num_patches // 4
515
- return [image_token_id] * num_tokens
516
-
517
- return [
518
- PromptReplacement(
519
- modality="image",
520
- target=[image_token_id],
521
- replacement=get_replacement_cheers,
522
- )
523
- ]
524
-
525
- def _get_mm_fields_config(
526
- self,
527
- hf_inputs: Any,
528
- hf_processor_mm_kwargs: Mapping[str, object],
529
- ) -> Mapping[str, MultiModalFieldConfig]:
530
- return {
531
- "pixel_values": MultiModalFieldConfig.batched("image"),
532
- }
533
-
534
-
535
-@MULTIMODAL_REGISTRY.register_processor(
536
- CheersMultiModalProcessor,
537
- info=CheersProcessingInfo,
538
- dummy_inputs=CheersDummyInputsBuilder,
539
-)
540
-class CheersForConditionalGeneration(
541
- nn.Module, SupportsMultiModal, SupportsLoRA, SupportsPP
542
-):
543
- """
544
- Cheers: A unified multimodal model for image understanding and generation.
545
-
546
- For vLLM, we focus on the image understanding (vision-to-text) capabilities.
547
- The image generation part is not supported in vLLM.
548
- """
549
-
550
- requires_raw_input_tokens = True
551
-
552
- hf_to_vllm_mapper = WeightsMapper(
553
- orig_to_new_prefix={
554
- "model.language_model.": "language_model.model.",
555
- "model.vision_representation.": "vision_representation.vision_model.",
556
- "model.und_projector.": "und_projector.",
557
- "model.vae_model.": "vae_model.",
558
- "model.vae_decoder_projector.": "vae_decoder_projector.",
559
- "lm_head.": "language_model.lm_head.",
560
- }
561
- )
562
-
563
- @classmethod
564
- def get_placeholder_str(cls, modality: str, i: int) -> str | None:
565
- if modality.startswith("image"):
566
- return "<|image_pad|>"
567
- raise ValueError("Only image modality is supported")
568
-
569
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
570
- super().__init__()
571
-
572
- config = vllm_config.model_config.hf_config
573
- quant_config = vllm_config.quant_config
574
- multimodal_config = vllm_config.model_config.multimodal_config
575
-
576
- if type(config).__name__ not in ("CheersConfig", "UMMConfig"):
577
- raise ValueError(
578
- f"Expected CheersConfig or UMMConfig, got {type(config).__name__}."
579
- )
580
-
581
- self.config = config
582
- self.multimodal_config = multimodal_config
583
-
584
- # The Cheers model's custom Qwen2Config defaults rope_theta to
585
- # 1_000_000, but this isn't stored in the JSON. vLLM's standard
586
- # Qwen2Config defaults to 10_000, causing a 100× mismatch.
587
- # We must patch BOTH the attribute AND rope_parameters (which
588
- # patch_rope_parameters may have already populated from the wrong
589
- # default before __init__ runs).
590
- _CHEERS_ROPE_THETA = 1_000_000.0
591
- tc = config.text_config
592
- old_theta = getattr(tc, "rope_theta", None)
593
- if old_theta != _CHEERS_ROPE_THETA:
594
- logger.info(
595
- "Overriding text_config.rope_theta from %s to %s",
596
- old_theta,
597
- _CHEERS_ROPE_THETA,
598
- )
599
- tc.rope_theta = _CHEERS_ROPE_THETA
600
- rp = getattr(tc, "rope_parameters", None)
601
- if rp is not None and rp.get("rope_theta") != _CHEERS_ROPE_THETA:
602
- logger.info(
603
- "Overriding rope_parameters.rope_theta from %s to %s",
604
- rp.get("rope_theta"),
605
- _CHEERS_ROPE_THETA,
606
- )
607
- rp["rope_theta"] = _CHEERS_ROPE_THETA
608
-
609
- with self._mark_language_model(vllm_config):
610
- self.language_model = init_vllm_registered_model(
611
- vllm_config=vllm_config,
612
- hf_config=config.text_config,
613
- prefix=maybe_prefix(prefix, "language_model"),
614
- architectures=["Qwen2ForCausalLM"],
615
- )
616
-
617
- vit_config = config.vision_representation_config
618
-
619
- with self._mark_tower_model(vllm_config, "image"):
620
- self.vae_model = CheersVAEModel(config)
621
- self.vae_decoder_projector = CheersVAEDecoderProjector(config)
622
-
623
- self.vision_representation = SiglipVisionModel(
624
- config=vit_config,
625
- quant_config=quant_config,
626
- prefix=maybe_prefix(prefix, "vision_representation"),
627
- )
628
-
629
- vit_hidden_size = vit_config.hidden_size
630
- llm_hidden_size = config.text_config.hidden_size
631
-
632
- self.und_projector = CheersUndProjector(
633
- image_embed_dim=vit_hidden_size,
634
- text_embed_dim=llm_hidden_size,
635
- compression_factor=(2, 2),
636
- quant_config=quant_config,
637
- prefix=maybe_prefix(prefix, "und_projector"),
638
- )
639
-
640
- self.make_empty_intermediate_tensors = (
641
- self.language_model.make_empty_intermediate_tensors
642
- )
643
-
644
- def _parse_and_validate_image_input(
645
- self, **kwargs: object
646
- ) -> CheersImageInputs | None:
647
- pixel_values = kwargs.pop("pixel_values", None)
648
- if pixel_values is None:
649
- return None
650
- return CheersImagePixelInputs(
651
- type="pixel_values",
652
- pixel_values=pixel_values,
653
- )
654
-
655
- def _process_image_input(
656
- self, image_input: CheersImageInputs
657
- ) -> tuple[torch.Tensor, ...]:
658
- """Process image inputs through VAE → SigLIP → projector pipeline.
659
-
660
- HF native path: pixel_values → VAE.encode(t=1.0) → vae_decoder_projector
661
- → SigLIP → und_projector → text-space embeddings
662
- """
663
- pixel_values = image_input["pixel_values"]
664
-
665
- if pixel_values.ndim == 5:
666
- batch_size, num_images, channels, height, width = pixel_values.shape
667
- pixel_values = pixel_values.reshape(
668
- batch_size * num_images, channels, height, width
669
- )
670
-
671
- with torch.no_grad():
672
- vae_dtype = next(self.vae_model.parameters()).dtype
673
- image_latent = self.vae_model.encode(pixel_values.to(dtype=vae_dtype))
674
- image_pixel_hat = self.vae_decoder_projector(image_latent)
675
-
676
- vision_features = self.vision_representation(image_pixel_hat)
677
- vision_embeds = self.und_projector(vision_features)
678
-
679
- return tuple(vision_embeds)
680
-
681
- def embed_multimodal(self, **kwargs: object) -> MultiModalEmbeddings:
682
- image_input = self._parse_and_validate_image_input(**kwargs)
683
- if image_input is None:
684
- return []
685
- return self._process_image_input(image_input)
686
-
687
- def forward(
688
- self,
689
- input_ids: torch.Tensor | None,
690
- positions: torch.Tensor,
691
- intermediate_tensors: IntermediateTensors | None = None,
692
- inputs_embeds: torch.Tensor | None = None,
693
- **kwargs: object,
694
- ) -> torch.Tensor | IntermediateTensors:
695
- if intermediate_tensors is not None:
696
- inputs_embeds = None
697
-
698
- hidden_states = self.language_model.model(
699
- input_ids=input_ids,
700
- positions=positions,
701
- intermediate_tensors=intermediate_tensors,
702
- inputs_embeds=inputs_embeds,
703
- )
704
- return hidden_states
705
-
706
- def compute_logits(
707
- self,
708
- hidden_states: torch.Tensor,
709
- ) -> torch.Tensor | None:
710
- return self.language_model.compute_logits(hidden_states)
711
-
712
- def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
713
- """Load weights, keeping VAE encoder/decoder projector for understanding."""
714
- skip_prefixes = [
715
- "model.time_embed.",
716
- "model.gen_projector.",
717
- "model.hi_gate.",
718
- "model.hi_projector.",
719
- "model.vae_model.decoder.",
720
- ]
721
- skip_keywords = [
722
- "text_loss_fc",
723
- ]
724
-
725
- filtered_weights = []
726
- for name, tensor in weights:
727
- if any(name.startswith(p) for p in skip_prefixes):
728
- continue
729
- if any(kw in name for kw in skip_keywords):
730
- continue
731
- filtered_weights.append((name, tensor))
732
-
733
- loader = AutoWeightsLoader(self)
734
- return loader.load_weights(filtered_weights, mapper=self.hf_to_vllm_mapper)
vllm/model_executor/models/conformer_encoder.py
+3
-4
@@ -1,11 +1,10 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
"""
4
-Shared Conformer encoder components for FireRedASR2 and FireRedLID.
4
+Conformer encoder components for FireRedASR2.
5
6
-Both models use the same Conformer-based audio encoder architecture
6
+The audio encoder architecture is
7
(Conv2dSubsampling → RelPositionalEncoding → N × RelPosEmbConformerBlock).
8
-This module factors out the common building blocks to avoid duplication.
8
"""
9
10
import torch
@@ -288,7 +287,7 @@ class RelPosEmbConformerBlock(nn.Module):
287
288
class ConformerEncoder(nn.Module):
289
"""
291
- Conformer encoder shared by FireRedASR2 and FireRedLID.
290
+ Conformer encoder used by FireRedASR2.
291
"""
292
293
def __init__(
vllm/model_executor/models/fairseq2_llama.py
deleted
-149
@@ -1,149 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-# Copyright 2024 The vLLM team.
5
-# Copyright 2024 Meta Platforms, Inc. and affiliates. All rights reserved.
6
-#
7
-# Licensed under the Apache License, Version 2.0 (the "License");
8
-# you may not use this file except in compliance with the License.
9
-# You may obtain a copy of the License at
10
-#
11
-# http://www.apache.org/licenses/LICENSE-2.0
12
-#
13
-# Unless required by applicable law or agreed to in writing, software
14
-# distributed under the License is distributed on an "AS IS" BASIS,
15
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16
-# See the License for the specific language governing permissions and
17
-# limitations under the License.
18
-"""Llama model for fairseq2 weights."""
19
-
20
-from collections.abc import Iterable
21
-
22
-import torch
23
-from torch.nn import Parameter
24
-
25
-from vllm.config import VllmConfig
26
-from vllm.distributed import (
27
- get_tensor_model_parallel_rank,
28
- get_tensor_model_parallel_world_size,
29
-)
30
-from vllm.model_executor.layers.linear import set_weight_attrs
31
-from vllm.model_executor.models.llama import LlamaForCausalLM
32
-
33
-from .utils import AutoWeightsLoader, WeightsMapper
34
-
35
-
36
-class Fairseq2LlamaForCausalLM(LlamaForCausalLM):
37
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
38
- super().__init__(vllm_config=vllm_config, prefix=prefix)
39
- self.tp_rank = get_tensor_model_parallel_rank()
40
- self.tp_size = get_tensor_model_parallel_world_size()
41
- # For the model loader to read only the relevant checkpoint files
42
- self.allow_patterns_overrides = [
43
- # either the full checkpoint
44
- "model.pt",
45
- # or the tp-sharded checkpoint of the current rank
46
- f"model.{self.tp_rank}.pt",
47
- ]
48
-
49
- def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
50
- # fairseq2's serialization adds a wrapper to usual .pt state_dict's:
51
- # { "model_key": my_model_name, "my_model_name": state_dict }
52
- # which we first need to unpack
53
- weights_wrapped = dict(weights)
54
- weights = weights_wrapped[weights_wrapped["model_key"]].items() # type: ignore
55
-
56
- # remap keys
57
- fs2_to_vllm_mapper = WeightsMapper(
58
- orig_to_new_prefix={
59
- "decoder_frontend.embed.": "model.embed_tokens.",
60
- "decoder.": "model.",
61
- "final_proj.": "lm_head.",
62
- },
63
- orig_to_new_substr={
64
- ".self_attn_layer_norm.": ".input_layernorm.",
65
- ".ffn_layer_norm.": ".post_attention_layernorm.",
66
- ".self_attn.output_proj.": ".self_attn.o_proj.",
67
- ".ffn.gate_proj.": ".mlp.gate_proj.",
68
- ".ffn.inner_proj.": ".mlp.up_proj.",
69
- ".ffn.output_proj.": ".mlp.down_proj.",
70
- ".layer_norm.": ".norm.",
71
- },
72
- )
73
- weights = fs2_to_vllm_mapper.apply(weights)
74
-
75
- params = dict(self.named_parameters())
76
-
77
- loader = AutoWeightsLoader(self)
78
- return loader.load_weights(
79
- self.reshape_fairseq2_weights(name, loaded_weight, params)
80
- for name, loaded_weight in weights
81
- )
82
-
83
- def flag_sharded_weights(self, params: dict[str, Parameter]):
84
- """Sets the `is_sharded_weight` flag to True for all sharded weights"""
85
- for name, param in params.items():
86
- modules = name.split(".")
87
- if "norm" in name and len(param.size()) < 2:
88
- # layer norms are not sharded
89
- continue
90
- elif any(emb in modules for emb in ["embed_tokens", "lm_head"]):
91
- # for now we repeat embedding layers for compatibility
92
- continue
93
- else:
94
- # all other layers are sharded
95
- set_weight_attrs(param, {"is_sharded_weight": True})
96
-
97
- def reshape_fairseq2_weights(
98
- self,
99
- name: str,
100
- loaded_weight: torch.Tensor,
101
- params: dict[str, Parameter],
102
- ) -> tuple[str, torch.Tensor]:
103
- """Reshape fairseq2's weights."""
104
-
105
- def permute(w: torch.Tensor, n_heads: int) -> torch.Tensor:
106
- attn_in = self.config.head_dim * n_heads
107
- # check for a sharded weight on dim 0
108
- if attn_in // self.tp_size == w.size()[0]:
109
- attn_in //= self.tp_size
110
- n_heads //= self.tp_size
111
- attn_out = self.config.hidden_size
112
- return (
113
- w.view(n_heads, attn_in // n_heads // 2, 2, attn_out)
114
- .transpose(1, 2)
115
- .reshape(attn_in, attn_out)
116
- )
117
-
118
- modules = name.split(".")
119
-
120
- # rotary embeds should be sliced
121
- if "k_proj" in modules:
122
- loaded_weight = permute(loaded_weight, self.config.num_key_value_heads)
123
-
124
- elif "q_proj" in modules:
125
- loaded_weight = permute(loaded_weight, self.config.num_attention_heads)
126
-
127
- # We make the loaded weights compatible with both
128
- # full checkpoints and tp sharded checkpoints.
129
- # Embeddings are repeated to fit the vocab size.
130
- # Other weights are flagged for the weight_loader calls.
131
- if any(emb in modules for emb in ["embed_tokens", "lm_head"]):
132
- # Embeddings are sharded on dim 0
133
- dim = 0
134
- # In fairseq2, vocab size has to be divisible by tp_size
135
- # so we don't worry about padding
136
- if self.tp_size > 1 and loaded_weight.shape[dim] < self.config.vocab_size:
137
- assert (
138
- loaded_weight.shape[dim] * self.tp_size == self.config.vocab_size
139
- ), "vocab_size should be divisible by tp_size."
140
- repeats = [1] * len(loaded_weight.size())
141
- repeats[dim] = self.tp_size
142
- # repeat to match vocab size and to be easily 'narrow'able
143
- loaded_weight = loaded_weight.repeat(repeats)
144
- set_weight_attrs(params[name], {"is_sharded_weight": False})
145
- # if embeddings are sharded, the rest is too
146
- if "embed_tokens" in modules:
147
- self.flag_sharded_weights(params)
148
-
149
- return name, loaded_weight
vllm/model_executor/models/fireredlid.py
deleted
-794
@@ -1,794 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-"""
4
-FireRedLID – Language Identification model adapted for vLLM.
5
-
6
-Architecture: ConformerEncoder + TransformerDecoder (6-layer cross-attn)
7
-Vocabulary: 120 LID tokens (dict.txt)
8
-Output: Up to 2 tokens (e.g. "en", "zh mandarin")
9
-
10
-This implementation follows the Whisper-style encoder-decoder pattern:
11
- • Encoder processes audio features (Fbank + CMVN via FeatureExtractor)
12
- • Decoder performs single-step autoregressive forward
13
- • vLLM's generation loop handles beam search / sampling
14
-"""
15
-
16
-from __future__ import annotations
17
-
18
-from collections.abc import Iterable, Mapping, Sequence
19
-from typing import Annotated, Literal
20
-
21
-import torch
22
-from torch import nn
23
-from transformers import BatchFeature
24
-
25
-from vllm.config import ModelConfig, VllmConfig
26
-from vllm.config.multimodal import AudioDummyOptions, BaseDummyOptions
27
-from vllm.config.speech_to_text import SpeechToTextConfig, SpeechToTextParams
28
-from vllm.distributed import get_tensor_model_parallel_world_size
29
-from vllm.inputs import MultiModalDataDict, PromptType
30
-from vllm.logger import init_logger
31
-from vllm.model_executor.layers.attention import Attention, CrossAttention
32
-from vllm.model_executor.layers.linear import (
33
- ColumnParallelLinear,
34
- ReplicatedLinear,
35
- RowParallelLinear,
36
-)
37
-from vllm.model_executor.layers.logits_processor import LogitsProcessor
38
-from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead
39
-from vllm.multimodal import MULTIMODAL_REGISTRY
40
-from vllm.multimodal.inputs import (
41
- MultiModalFieldConfig,
42
- MultiModalKwargsItems,
43
-)
44
-from vllm.multimodal.parse import MultiModalDataItems, MultiModalDataParser
45
-from vllm.multimodal.processing import (
46
- BaseDummyInputsBuilder,
47
- BaseProcessingInfo,
48
- EncDecMultiModalProcessor,
49
- PromptReplacement,
50
- PromptUpdate,
51
-)
52
-from vllm.transformers_utils.processor import cached_processor_from_config
53
-from vllm.utils.tensor_schema import TensorSchema, TensorShape
54
-
55
-from .conformer_encoder import ConformerEncoder
56
-from .interfaces import (
57
- MultiModalEmbeddings,
58
- SupportsMultiModal,
59
- SupportsTranscription,
60
-)
61
-from .utils import (
62
- AutoWeightsLoader,
63
- WeightsMapper,
64
- maybe_prefix,
65
-)
66
-from .whisper_utils import ISO639_1_SUPPORTED_LANGS
67
-
68
-logger = init_logger(__name__)
69
-
70
-
71
-class FireRedLIDAudioInputs(TensorSchema):
72
- """
73
- Dimensions:
74
- - b: Batch size
75
- - t: Time frames (variable across utterances)
76
- - nmb: Number of mel bins (80)
77
- """
78
-
79
- input_features: Annotated[
80
- list[torch.Tensor] | None,
81
- TensorShape("b", "t", "nmb", dynamic_dims={"t"}),
82
- ]
83
- speech_lengths: Annotated[
84
- list[torch.Tensor] | None,
85
- TensorShape("b"),
86
- ]
87
- fake_token_lengths: Annotated[
88
- list[torch.Tensor] | None,
89
- TensorShape("b"),
90
- ]
91
-
92
-
93
-FireRedLIDEncoder = ConformerEncoder
94
-
95
-
96
-class FireRedLIDPositionalEmbedding(nn.Module):
97
- """Absolute sinusoidal positional embedding indexed by `positions`."""
98
-
99
- def __init__(self, d_model: int, max_len: int = 5000):
100
- super().__init__()
101
- assert d_model % 2 == 0
102
- pe = torch.zeros(max_len, d_model, requires_grad=False)
103
- position = torch.arange(0, max_len).unsqueeze(1).float()
104
- div_term = torch.exp(
105
- torch.arange(0, d_model, 2).float()
106
- * -(torch.log(torch.tensor(10000.0)).item() / d_model)
107
- )
108
- pe[:, 0::2] = torch.sin(position * div_term)
109
- pe[:, 1::2] = torch.cos(position * div_term)
110
- self.register_buffer("pe", pe, persistent=False)
111
-
112
- def forward(self, position_ids: torch.Tensor) -> torch.Tensor:
113
- return self.pe[position_ids]
114
-
115
-
116
-class FireRedLIDAttention(nn.Module):
117
- """Base attention with shared QKV/FC projections for the LID decoder."""
118
-
119
- def __init__(
120
- self,
121
- d_model: int,
122
- n_head: int,
123
- *,
124
- vllm_config: VllmConfig,
125
- prefix: str = "",
126
- ):
127
- super().__init__()
128
- tp_size = get_tensor_model_parallel_world_size()
129
- assert n_head % tp_size == 0
130
- self.total_num_heads = n_head
131
- self.num_heads = n_head // tp_size
132
- self.num_kv_heads = max(1, n_head // tp_size)
133
- self.head_dim = d_model // n_head
134
- self.scaling = self.head_dim**-0.5
135
-
136
- cache_config = vllm_config.cache_config
137
- quant_config = vllm_config.quant_config
138
-
139
- self.w_qs = ColumnParallelLinear(
140
- d_model,
141
- d_model,
142
- bias=True,
143
- quant_config=quant_config,
144
- prefix=f"{prefix}.w_qs",
145
- )
146
- self.w_ks = ColumnParallelLinear(
147
- d_model,
148
- d_model,
149
- bias=False,
150
- quant_config=quant_config,
151
- prefix=f"{prefix}.w_ks",
152
- )
153
- self.w_vs = ColumnParallelLinear(
154
- d_model,
155
- d_model,
156
- bias=True,
157
- quant_config=quant_config,
158
- prefix=f"{prefix}.w_vs",
159
- )
160
- self.fc = RowParallelLinear(
161
- d_model,
162
- d_model,
163
- bias=True,
164
- quant_config=quant_config,
165
- prefix=f"{prefix}.fc",
166
- )
167
- self._init_attn(cache_config, quant_config, prefix)
168
-
169
- def _init_attn(self, cache_config, quant_config, prefix: str) -> None:
170
- raise NotImplementedError
171
-
172
-
173
-class FireRedLIDSelfAttention(FireRedLIDAttention):
174
- def _init_attn(self, cache_config, quant_config, prefix: str) -> None:
175
- self.attn = Attention(
176
- self.num_heads,
177
- self.head_dim,
178
- self.scaling,
179
- num_kv_heads=self.num_kv_heads,
180
- cache_config=cache_config,
181
- quant_config=quant_config,
182
- prefix=f"{prefix}.attn",
183
- )
184
-
185
- def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
186
- q, _ = self.w_qs(hidden_states)
187
- k, _ = self.w_ks(hidden_states)
188
- v, _ = self.w_vs(hidden_states)
189
- attn_output = self.attn(q, k, v)
190
- output, _ = self.fc(attn_output)
191
- return output
192
-
193
-
194
-class FireRedLIDCrossAttention(FireRedLIDAttention):
195
- def _init_attn(self, cache_config, quant_config, prefix: str) -> None:
196
- self.attn = CrossAttention(
197
- self.num_heads,
198
- self.head_dim,
199
- self.scaling,
200
- num_kv_heads=self.num_kv_heads,
201
- cache_config=cache_config,
202
- quant_config=quant_config,
203
- prefix=f"{prefix}.attn",
204
- )
205
-
206
- def forward(
207
- self,
208
- hidden_states: torch.Tensor,
209
- encoder_hidden_states: torch.Tensor | None,
210
- ) -> torch.Tensor:
211
- q, _ = self.w_qs(hidden_states)
212
- if encoder_hidden_states is not None:
213
- k, _ = self.w_ks(encoder_hidden_states)
214
- v, _ = self.w_vs(encoder_hidden_states)
215
- else:
216
- k = v = None
217
-
218
- attn_output = self.attn(q, k, v)
219
- output, _ = self.fc(attn_output)
220
- return output
221
-
222
-
223
-class FireRedLIDFFN(nn.Module):
224
- def __init__(self, d_model: int, d_ff: int):
225
- super().__init__()
226
- self.w_1 = ReplicatedLinear(d_model, d_ff, bias=True)
227
- self.act = nn.GELU()
228
- self.w_2 = ReplicatedLinear(d_ff, d_model, bias=True)
229
-
230
- def forward(self, x: torch.Tensor) -> torch.Tensor:
231
- x, _ = self.w_1(x)
232
- x = self.act(x)
233
- x, _ = self.w_2(x)
234
- return x
235
-
236
-
237
-class FireRedLIDDecoderLayer(nn.Module):
238
- """vLLM-native decoder layer while preserving FireRedLID parameter names."""
239
-
240
- def __init__(
241
- self,
242
- d_model: int,
243
- n_head: int,
244
- *,
245
- vllm_config: VllmConfig,
246
- prefix: str = "",
247
- ):
248
- super().__init__()
249
- self.self_attn_norm = nn.LayerNorm(d_model)
250
- self.self_attn = FireRedLIDSelfAttention(
251
- d_model,
252
- n_head,
253
- vllm_config=vllm_config,
254
- prefix=f"{prefix}.self_attn",
255
- )
256
-
257
- self.cross_attn_norm = nn.LayerNorm(d_model)
258
- self.cross_attn = FireRedLIDCrossAttention(
259
- d_model,
260
- n_head,
261
- vllm_config=vllm_config,
262
- prefix=f"{prefix}.cross_attn",
263
- )
264
-
265
- self.mlp_norm = nn.LayerNorm(d_model)
266
- self.mlp = FireRedLIDFFN(d_model, d_model * 4)
267
-
268
- def forward(
269
- self,
270
- hidden_states: torch.Tensor,
271
- encoder_hidden_states: torch.Tensor | None,
272
- ) -> torch.Tensor:
273
- residual = hidden_states
274
- hidden_states = self.self_attn_norm(hidden_states)
275
- hidden_states = self.self_attn(hidden_states)
276
- hidden_states = residual + hidden_states
277
-
278
- residual = hidden_states
279
- hidden_states = self.cross_attn_norm(hidden_states)
280
- hidden_states = self.cross_attn(hidden_states, encoder_hidden_states)
281
- hidden_states = residual + hidden_states
282
-
283
- residual = hidden_states
284
- hidden_states = self.mlp_norm(hidden_states)
285
- hidden_states = residual + self.mlp(hidden_states)
286
-
287
- return hidden_states
288
-
289
-
290
-class FireRedLIDDecoder(nn.Module):
291
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
292
- super().__init__()
293
- config = vllm_config.model_config.hf_config
294
- self.pad_id = getattr(config, "pad_token_id", 2)
295
- self.n_layers = getattr(config, "n_layers_lid_dec", 6)
296
- self.d_model = getattr(config, "d_model", 1280)
297
- self.scale = self.d_model**0.5
298
-
299
- self.tgt_word_emb = nn.Embedding(
300
- getattr(config, "vocab_size", 120),
301
- self.d_model,
302
- padding_idx=self.pad_id,
303
- )
304
- self.positional_encoding = FireRedLIDPositionalEmbedding(
305
- self.d_model,
306
- max_len=getattr(config, "pe_maxlen", 5000),
307
- )
308
-
309
- self.layer_stack = nn.ModuleList(
310
- [
311
- FireRedLIDDecoderLayer(
312
- self.d_model,
313
- getattr(config, "n_head", 20),
314
- vllm_config=vllm_config,
315
- prefix=f"{prefix}.layer_stack.{idx}",
316
- )
317
- for idx in range(self.n_layers)
318
- ]
319
- )
320
- self.layer_norm_out = nn.LayerNorm(self.d_model)
321
-
322
- def forward(
323
- self,
324
- input_ids: torch.Tensor,
325
- positions: torch.Tensor,
326
- encoder_hidden_states: torch.Tensor | None,
327
- ) -> torch.Tensor:
328
- hidden_states = self.tgt_word_emb(input_ids) * self.scale
329
- hidden_states = hidden_states + self.positional_encoding(positions)
330
-
331
- for layer in self.layer_stack:
332
- hidden_states = layer(hidden_states, encoder_hidden_states)
333
-
334
- hidden_states = self.layer_norm_out(hidden_states)
335
- return hidden_states
336
-
337
- def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
338
- return self.tgt_word_emb(input_ids)
339
-
340
-
341
-class FireRedLIDModel(nn.Module):
342
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
343
- super().__init__()
344
- config = vllm_config.model_config.hf_config
345
-
346
- self.encoder = FireRedLIDEncoder(
347
- idim=getattr(config, "idim", 80),
348
- n_layers_enc=getattr(config, "n_layers_enc", 16),
349
- n_head=getattr(config, "n_head", 20),
350
- d_model=getattr(config, "d_model", 1280),
351
- kernel_size=getattr(config, "kernel_size", 33),
352
- pe_maxlen=getattr(config, "pe_maxlen", 5000),
353
- )
354
-
355
- self.decoder = FireRedLIDDecoder(
356
- vllm_config=vllm_config,
357
- prefix=maybe_prefix(prefix, "decoder"),
358
- )
359
-
360
- def forward(
361
- self,
362
- input_ids: torch.Tensor,
363
- positions: torch.Tensor,
364
- encoder_outputs: list[torch.Tensor] | None = None,
365
- ) -> torch.Tensor:
366
- enc_states = (
367
- torch.cat(encoder_outputs, dim=0)
368
- if encoder_outputs and len(encoder_outputs) > 0
369
- else None
370
- )
371
- decoder_outputs = self.decoder(
372
- input_ids=input_ids,
373
- positions=positions,
374
- encoder_hidden_states=enc_states,
375
- )
376
- return decoder_outputs
377
-
378
- def get_encoder_outputs(
379
- self,
380
- speech: torch.Tensor | list[torch.Tensor],
381
- speech_lengths: torch.Tensor | list[torch.Tensor],
382
- ) -> tuple[torch.Tensor, torch.Tensor]:
383
- """Run the encoder and return padded outputs plus true sequence lengths."""
384
- enc_output, enc_lengths, _ = self.encoder(speech, speech_lengths)
385
- return enc_output, enc_lengths
386
-
387
-
388
-class FireRedLIDProcessingInfo(BaseProcessingInfo):
389
- def get_hf_config(self):
390
- return self.ctx.get_hf_config()
391
-
392
- def get_supported_mm_limits(self) -> Mapping[str, int | None]:
393
- return {"audio": 1}
394
-
395
- def get_feature_extractor(self, **kwargs):
396
- hf_processor = self.get_hf_processor(**kwargs)
397
- feature_extractor = hf_processor.feature_extractor
398
- return feature_extractor
399
-
400
- def get_data_parser(self) -> MultiModalDataParser:
401
- feature_extractor = self.get_feature_extractor()
402
- return MultiModalDataParser(
403
- target_sr=feature_extractor.sampling_rate,
404
- target_channels=1,
405
- )
406
-
407
- @property
408
- def skip_prompt_length_check(self) -> bool:
409
- return True
410
-
411
- def get_num_audio_tokens(self) -> int:
412
- # For encoder profiling – return a reasonable dummy length.
413
- # This doesn't affect actual inference since encoder processes
414
- # variable-length features.
415
- return 1
416
-
417
-
418
-class FireRedLIDDummyInputsBuilder(BaseDummyInputsBuilder[FireRedLIDProcessingInfo]):
419
- def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
420
- return "<sos>"
421
-
422
- def get_dummy_mm_data(
423
- self,
424
- seq_len: int,
425
- mm_counts: Mapping[str, int],
426
- mm_options: Mapping[str, BaseDummyOptions],
427
- ) -> MultiModalDataDict:
428
- feature_extractor = self.info.get_feature_extractor()
429
- sampling_rate = feature_extractor.sampling_rate
430
- audio_len = feature_extractor.chunk_length * sampling_rate
431
- num_audios = mm_counts.get("audio", 0)
432
- audio_overrides = mm_options.get("audio")
433
- assert audio_overrides is None or isinstance(audio_overrides, AudioDummyOptions)
434
- return {
435
- "audio": self._get_dummy_audios(
436
- length=audio_len,
437
- num_audios=num_audios,
438
- overrides=audio_overrides,
439
- )
440
- }
441
-
442
-
443
-class FireRedLIDMultiModalProcessor(
444
- EncDecMultiModalProcessor[FireRedLIDProcessingInfo]
445
-):
446
- def create_encoder_prompt(
447
- self,
448
- prompt: list[int],
449
- mm_items: MultiModalDataItems,
450
- ) -> list[int]:
451
- # Dummy encoder prompt for profiling (encoder only processes audio).
452
- return [0]
453
-
454
- def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
455
- return self.dummy_inputs.get_dummy_text(mm_counts)
456
-
457
- def _preprocess_hf_mm_data(
458
- self,
459
- mm_data: Mapping[str, object],
460
- hf_processor_mm_kwargs: Mapping[str, object],
461
- ) -> tuple[Mapping[str, object], Mapping[str, object]]:
462
- feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)
463
-
464
- mm_data = dict(mm_data)
465
- mm_data["audio"] = mm_data.pop("audios")
466
-
467
- hf_processor_mm_kwargs = dict(
468
- **hf_processor_mm_kwargs,
469
- sampling_rate=feature_extractor.sampling_rate,
470
- )
471
-
472
- return mm_data, hf_processor_mm_kwargs
473
-
474
- def _postprocess_hf_mm_data(
475
- self,
476
- mm_data: Mapping[str, object],
477
- hf_processor_mm_kwargs: Mapping[str, object],
478
- processed_data: BatchFeature,
479
- ) -> BatchFeature:
480
- if "labels" in processed_data:
481
- processed_data["input_ids"] = processed_data.pop("labels")
482
-
483
- return processed_data
484
-
485
- def _get_mm_fields_config(
486
- self,
487
- hf_inputs: BatchFeature,
488
- hf_processor_mm_kwargs: Mapping[str, object],
489
- ) -> Mapping[str, MultiModalFieldConfig]:
490
- return dict(
491
- input_features=MultiModalFieldConfig.batched("audio"),
492
- speech_lengths=MultiModalFieldConfig.batched("audio"),
493
- fake_token_lengths=MultiModalFieldConfig.batched("audio", keep_on_cpu=True),
494
- )
495
-
496
- def _get_prompt_updates(
497
- self,
498
- mm_items: MultiModalDataItems,
499
- hf_processor_mm_kwargs: Mapping[str, object],
500
- out_mm_kwargs: MultiModalKwargsItems,
501
- ) -> Sequence[PromptUpdate]:
502
- out_mm_data = out_mm_kwargs.get_data()
503
- fake_token_lengths = out_mm_data.get("fake_token_lengths")
504
-
505
- if fake_token_lengths is None:
506
- # Fallback to max encoder output length if not available
507
- audio_output_lengths = []
508
- else:
509
- assert isinstance(fake_token_lengths, torch.Tensor)
510
- audio_output_lengths = fake_token_lengths.tolist()
511
-
512
- def get_replacement(item_idx: int):
513
- if audio_output_lengths:
514
- num_tokens = int(audio_output_lengths[item_idx])
515
- else:
516
- num_tokens = self.info.get_num_audio_tokens()
517
- return [0] * num_tokens
518
-
519
- return [
520
- PromptReplacement(
521
- modality="audio",
522
- target=[0],
523
- replacement=get_replacement,
524
- )
525
- ]
526
-
527
-
528
-# FireRedLID supports a wider set of languages than Whisper's shared list.
529
-# Only ISO 639-1 codes are listed; FireRedLID's dialect tokens (mandarin,
530
-# xinan, wu, …) are output tokens but not valid language *request* codes.
531
-_FIREREDLID_SUPPORTED_LANGUAGES: Mapping[str, str] = {
532
- **ISO639_1_SUPPORTED_LANGS,
533
- "am": "Amharic",
534
- "as": "Assamese",
535
- "ba": "Bashkir",
536
- "bn": "Bengali",
537
- "bo": "Tibetan",
538
- "br": "Breton",
539
- "eu": "Basque",
540
- "fo": "Faroese",
541
- "gu": "Gujarati",
542
- "ha": "Hausa",
543
- "haw": "Hawaiian",
544
- "ht": "Haitian Creole",
545
- "jw": "Javanese",
546
- "ka": "Georgian",
547
- "km": "Khmer",
548
- "la": "Latin",
549
- "lb": "Luxembourgish",
550
- "ln": "Lingala",
551
- "lo": "Lao",
552
- "mg": "Malagasy",
553
- "ml": "Malayalam",
554
- "mn": "Mongolian",
555
- "mt": "Maltese",
556
- "my": "Myanmar",
557
- "nn": "Nynorsk",
558
- "oc": "Occitan",
559
- "pa": "Panjabi",
560
- "ps": "Pashto",
561
- "sa": "Sanskrit",
562
- "sd": "Sindhi",
563
- "si": "Sinhala",
564
- "sn": "Shona",
565
- "so": "Somali",
566
- "sq": "Albanian",
567
- "su": "Sundanese",
568
- "te": "Telugu",
569
- "tg": "Tajik",
570
- "tk": "Turkmen",
571
- "tt": "Tatar",
572
- "uz": "Uzbek",
573
- "yi": "Yiddish",
574
- "yo": "Yoruba",
575
- "yue": "Cantonese",
576
-}
577
-
578
-
579
-@MULTIMODAL_REGISTRY.register_processor(
580
- FireRedLIDMultiModalProcessor,
581
- info=FireRedLIDProcessingInfo,
582
- dummy_inputs=FireRedLIDDummyInputsBuilder,
583
-)
584
-class FireRedLIDForConditionalGeneration(
585
- nn.Module, SupportsTranscription, SupportsMultiModal
586
-):
587
- # -- SupportsTranscription protocol attributes --
588
- supports_transcription_only = True
589
- supported_languages = _FIREREDLID_SUPPORTED_LANGUAGES
590
-
591
- hf_to_vllm_mapper = WeightsMapper(
592
- orig_to_new_substr={
593
- "encoder.": "model.encoder.",
594
- "lid_decoder.": "model.decoder.",
595
- # Encoder FFN: nn.Sequential indices → named children
596
- "net.0": "pre_layer_norm",
597
- "net.1": "linear_expand",
598
- "net.4": "linear_project",
599
- },
600
- orig_to_new_prefix={
601
- # Position encoding buffers are rebuilt at init, and the output
602
- # projection is tied to the embedding.
603
- "model.encoder.positional_encoding.pe": None,
604
- "model.decoder.positional_encoding.pe": None,
605
- "model.decoder.tgt_word_prj.weight": None,
606
- "proj_out.": None,
607
- },
608
- )
609
-
610
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
611
- super().__init__()
612
- config = vllm_config.model_config.hf_config
613
- self.config = config
614
- self.dtype = vllm_config.model_config.dtype
615
-
616
- with self._mark_composite_model(
617
- vllm_config,
618
- language_targets=FireRedLIDDecoder,
619
- tower_targets={"audio": FireRedLIDEncoder},
620
- ):
621
- self.model = FireRedLIDModel(
622
- vllm_config=vllm_config,
623
- prefix=maybe_prefix(prefix, "model"),
624
- )
625
-
626
- self.proj_out = ParallelLMHead(
627
- getattr(config, "vocab_size", 120),
628
- getattr(config, "d_model", 1280),
629
- quant_config=vllm_config.quant_config,
630
- prefix=maybe_prefix(prefix, "proj_out"),
631
- )
632
- self.proj_out = self.proj_out.tie_weights(self.model.decoder.tgt_word_emb)
633
-
634
- logit_scale = getattr(config, "logit_scale", 1.0)
635
- self.logits_processor = LogitsProcessor(
636
- getattr(config, "vocab_size", 120),
637
- scale=logit_scale,
638
- )
639
-
640
- def forward(
641
- self,
642
- input_ids: torch.Tensor,
643
- positions: torch.Tensor,
644
- encoder_outputs: list[torch.Tensor] | None = None,
645
- **kwargs,
646
- ) -> torch.Tensor:
647
- if encoder_outputs is None:
648
- encoder_outputs = []
649
- decoder_outputs = self.model(
650
- input_ids=input_ids,
651
- positions=positions,
652
- encoder_outputs=encoder_outputs,
653
- )
654
- return decoder_outputs
655
-
656
- def embed_multimodal(self, **kwargs: object) -> MultiModalEmbeddings:
657
- """Run encoder on audio features and return per-item embeddings."""
658
- audio_input = self._parse_and_validate_audio_input(**kwargs)
659
-
660
- speech = audio_input["input_features"]
661
- speech_lengths = audio_input["speech_lengths"]
662
- if speech is None or speech_lengths is None:
663
- return []
664
-
665
- # When audio items have different time lengths, vLLM's
666
- # MultiModalBatchedField._reduce_data returns a plain
667
- # list[Tensor] instead of a stacked Tensor. The encoder
668
- # expects a padded [B, Tmax, feat_dim] Tensor, so we
669
- # normalise both speech and speech_lengths here.
670
- if isinstance(speech, (list, tuple)):
671
- # Each element: [Ti, feat_dim] (or [1, Ti, feat_dim])
672
- tensors = [
673
- s.squeeze(0) if s.dim() == 3 and s.size(0) == 1 else s for s in speech
674
- ]
675
- device = tensors[0].device
676
- dtype = tensors[0].dtype
677
- feat_dim = tensors[0].shape[-1]
678
- lengths = torch.tensor(
679
- [t.size(0) for t in tensors],
680
- device=device,
681
- dtype=torch.int32,
682
- )
683
- t_max = int(lengths.max().item())
684
- # Pre-allocate zero-padded batch tensor
685
- speech = torch.zeros(
686
- (len(tensors), t_max, feat_dim),
687
- device=device,
688
- dtype=dtype,
689
- )
690
- for i, t in enumerate(tensors):
691
- speech[i, : t.size(0)] = t
692
- speech_lengths = lengths
693
- else:
694
- # Already a batched Tensor [B, T, feat_dim]
695
- if speech.dim() == 2:
696
- speech = speech.unsqueeze(0)
697
-
698
- speech_lengths = torch.as_tensor(
699
- speech_lengths, dtype=torch.int32, device=speech.device
700
- )
701
-
702
- enc_output, enc_lengths = self.model.get_encoder_outputs(
703
- speech=speech,
704
- speech_lengths=speech_lengths,
705
- )
706
-
707
- # vLLM expects one 2D tensor per multimodal item. Slice each batch entry
708
- # by the true encoder length so cross-attention never sees padded frames.
709
- return tuple(
710
- enc_output[i, : max(0, int(enc_lengths[i].item()))]
711
- for i in range(enc_output.size(0))
712
- )
713
-
714
- def embed_input_ids(
715
- self,
716
- input_ids: torch.Tensor,
717
- multimodal_embeddings: MultiModalEmbeddings | None = None,
718
- *,
719
- is_multimodal: torch.Tensor | None = None,
720
- ) -> torch.Tensor:
721
- return self.model.decoder.embed_input_ids(input_ids)
722
-
723
- def _parse_and_validate_audio_input(
724
- self, **kwargs: object
725
- ) -> FireRedLIDAudioInputs:
726
- input_features = kwargs.pop("input_features", None)
727
- speech_lengths = kwargs.pop("speech_lengths", None)
728
- fake_token_lengths = kwargs.pop("fake_token_lengths", None)
729
- return FireRedLIDAudioInputs(
730
- input_features=input_features,
731
- speech_lengths=speech_lengths,
732
- fake_token_lengths=fake_token_lengths,
733
- )
734
-
735
- def compute_logits(self, hidden_states: torch.Tensor) -> torch.Tensor:
736
- logits = self.logits_processor(self.proj_out, hidden_states)
737
- return logits
738
-
739
- @classmethod
740
- def validate_language(cls, language: str | None) -> str | None:
741
- # FireRedLID is a language *identification* model – the caller does
742
- # not need to specify a language up-front. Accept None silently.
743
- if language is None:
744
- return None
745
- return super().validate_language(language)
746
-
747
- @classmethod
748
- def get_generation_prompt(
749
- cls,
750
- stt_params: SpeechToTextParams,
751
- ) -> PromptType:
752
- """Build the prompt for the FireRedLID encoder-decoder model.
753
-
754
- The decoder receives a single <sos> token; the encoder processes
755
- the raw audio waveform via the multimodal pipeline.
756
- """
757
- prompt: PromptType = {
758
- "encoder_prompt": {
759
- "prompt": "",
760
- "multi_modal_data": {
761
- "audio": (
762
- stt_params.audio,
763
- int(stt_params.stt_config.sample_rate),
764
- ),
765
- },
766
- },
767
- "decoder_prompt": {
768
- "prompt": "<sos>",
769
- },
770
- }
771
- return prompt
772
-
773
- @classmethod
774
- def get_speech_to_text_config(
775
- cls,
776
- model_config: ModelConfig,
777
- task_type: Literal["transcribe", "translate"],
778
- ) -> SpeechToTextConfig:
779
- processor = cached_processor_from_config(model_config)
780
- return SpeechToTextConfig(
781
- max_audio_clip_s=processor.feature_extractor.chunk_length,
782
- sample_rate=processor.feature_extractor.sampling_rate,
783
- # LID output is at most 2 tokens – no chunking needed.
784
- min_energy_split_window_size=None,
785
- )
786
-
787
- @classmethod
788
- def post_process_output(cls, text: str) -> str:
789
- # Strip any leading/trailing whitespace from the raw LID output.
790
- return text.strip()
791
-
792
- def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
793
- loader = AutoWeightsLoader(self)
794
- return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
vllm/model_executor/models/gritlm.py
deleted
-243
@@ -1,243 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-from collections.abc import Set
4
-
5
-import numpy as np
6
-import torch
7
-
8
-from vllm.config import ModelConfig, VllmConfig
9
-from vllm.logger import init_logger
10
-from vllm.model_executor.layers.pooler import (
11
- DispatchPooler,
12
- PoolingParamsUpdate,
13
-)
14
-from vllm.model_executor.layers.pooler.activations import PoolerNormalize
15
-from vllm.model_executor.layers.pooler.seqwise import (
16
- EmbeddingPoolerHead,
17
- SequencePooler,
18
- SequencePoolingMethod,
19
- SequencePoolingMethodOutput,
20
- get_seq_pooling_method,
21
-)
22
-from vllm.model_executor.layers.pooler.tokwise import pooler_for_token_embed
23
-from vllm.model_executor.models.llama import LlamaForCausalLM
24
-from vllm.tasks import PoolingTask
25
-from vllm.tokenizers import cached_tokenizer_from_config
26
-from vllm.v1.pool.metadata import PoolingMetadata
27
-
28
-from .interfaces_base import default_pooling_type
29
-
30
-logger = init_logger(__name__)
31
-
32
-
33
-class GritLMMeanPool(SequencePoolingMethod):
34
- """As `MeanPool`, but only includes non-instruction tokens."""
35
-
36
- def __init__(self, model_config: ModelConfig):
37
- super().__init__()
38
-
39
- self.model_config = model_config
40
-
41
- tokenizer = cached_tokenizer_from_config(self.model_config)
42
-
43
- # Collect the tokens needed for pattern matching.
44
- # "▁<" is different from "_<". The former uses "▁" to indicate that
45
- # the next token is the start of a word.
46
- # "<0x0A>" is the newline token (i.e. "\n")."
47
- self.token_ids = {
48
- tok: tokenizer.convert_tokens_to_ids([tok])[0]
49
- for tok in ["<s>", "▁<", "<", "|", "embed", ">", "<0x0A>", "user"]
50
- }
51
-
52
- def tokens_to_ids(tokens: list[str]) -> np.ndarray:
53
- return np.array([self.token_ids[token] for token in tokens])
54
-
55
- self.user_pattern_ids = tokens_to_ids(["▁<", "|", "user", "|", ">", "<0x0A>"])
56
- self.embed_newline_pattern_ids = tokens_to_ids(
57
- ["<0x0A>", "<", "|", "embed", "|", ">", "<0x0A>"]
58
- )
59
- self.embed_pattern_ids = tokens_to_ids(["▁<", "|", "embed", "|", ">", "<0x0A>"])
60
-
61
- def _find_array(
62
- self,
63
- arr: np.ndarray,
64
- target: np.ndarray,
65
- start_idx: int = 0,
66
- end_idx: int | None = None,
67
- ) -> int:
68
- """
69
- Find the first occurrence of `target` in `arr` starting from
70
- `start_idx`.
71
-
72
- Args:
73
- arr: The array to search within.
74
- target: The consecutive subsequence to find.
75
- start_idx: The starting index to search from (inclusive).
76
- end_idx: The ending index to search from (exclusive).
77
-
78
- Returns:
79
- The index of the first occurrence of `target` in `arr`.
80
- """
81
- if start_idx < 0:
82
- raise ValueError("`start_idx` must be non-negative")
83
- if len(arr) == 0 or len(target) == 0:
84
- raise ValueError("Empty `arr` or `target` not allowed")
85
-
86
- arr_len = len(arr)
87
- target_len = len(target)
88
-
89
- if end_idx is None:
90
- end_idx = arr_len
91
-
92
- for i in range(start_idx, min(end_idx, arr_len - target_len + 1)):
93
- if (arr[i : i + target_len] == target).all():
94
- return i
95
-
96
- return -1
97
-
98
- def _get_instruction_len(self, prompt_token_ids: np.ndarray) -> int:
99
- """
100
- Get the length of the instruction in the prompt.
101
-
102
- We do a pattern matching to find the instruction in the prompt,
103
- and then return the length of the instruction.
104
-
105
- The pattern matching is done using integers instead of strings
106
- because the prompt is given as a list of token IDs.
107
- """
108
- instruction_len = 0
109
-
110
- # Return no instruction in case of missing BOS token.
111
- if prompt_token_ids[0] != self.token_ids["<s>"]:
112
- logger.warning(
113
- "BOS token not found in prompt, "
114
- "thus using empty string for instruction. "
115
- "GritLM requires BOS token in prompt."
116
- )
117
- return instruction_len
118
-
119
- # If user pattern is found in the prompt, that means there should be
120
- # a newline token before the embed pattern.
121
- embed_pattern_ids = self.embed_pattern_ids
122
- if (
123
- self._find_array(
124
- prompt_token_ids, self.user_pattern_ids, start_idx=1, end_idx=2
125
- )
126
- == 1
127
- ):
128
- embed_pattern_ids = self.embed_newline_pattern_ids
129
-
130
- # Find the embed pattern in the prompt.
131
- found_embed_pattern_idx = self._find_array(
132
- prompt_token_ids, embed_pattern_ids, start_idx=1
133
- )
134
-
135
- if found_embed_pattern_idx != -1:
136
- instruction_len = found_embed_pattern_idx + len(embed_pattern_ids)
137
- else:
138
- logger.warning(
139
- "Query instruction not found in prompt, "
140
- "thus using BOS token as instruction instead. "
141
- "GritLM requires query instruction in prompt."
142
- )
143
- instruction_len = 1
144
-
145
- return instruction_len
146
-
147
- def get_supported_tasks(self) -> Set[PoolingTask]:
148
- return {"embed"}
149
-
150
- def get_pooling_updates(self, task: PoolingTask) -> PoolingParamsUpdate:
151
- return PoolingParamsUpdate(requires_token_ids=True)
152
-
153
- def forward(
154
- self,
155
- hidden_states: torch.Tensor,
156
- pooling_metadata: PoolingMetadata,
157
- ) -> SequencePoolingMethodOutput:
158
- prompt_lens = pooling_metadata.prompt_lens
159
- prompt_token_ids = pooling_metadata.get_prompt_token_ids_cpu()
160
- instr_lens = torch.tensor(
161
- [
162
- self._get_instruction_len(token_ids.numpy())
163
- for token_ids in prompt_token_ids
164
- ],
165
- device="cpu",
166
- )
167
-
168
- offset = 0
169
- pooled_data = list[torch.Tensor]()
170
- for prompt_len, instr_len in zip(prompt_lens, instr_lens):
171
- pooled_data.append(
172
- hidden_states[offset + instr_len : offset + prompt_len].mean(
173
- dim=0, dtype=torch.float32
174
- )
175
- )
176
- offset += prompt_len
177
-
178
- return pooled_data
179
-
180
-
181
-class GritLMPooler(SequencePooler):
182
- def __init__(self, model_config: ModelConfig):
183
- pooler_config = model_config.pooler_config
184
- assert pooler_config is not None
185
-
186
- super().__init__(
187
- pooling=(
188
- GritLMMeanPool(model_config)
189
- if pooler_config.seq_pooling_type == "MEAN"
190
- else get_seq_pooling_method(pooler_config.seq_pooling_type)
191
- ),
192
- head=EmbeddingPoolerHead(
193
- head_dtype=model_config.head_dtype,
194
- activation=PoolerNormalize(),
195
- ),
196
- )
197
-
198
-
199
-@default_pooling_type(seq_pooling_type="MEAN")
200
-class GritLM(LlamaForCausalLM):
201
- """This class implements the embedding model for parasail-ai/GritLM-7B-vllm.
202
-
203
- The class inherits from LlamaForCausalLM and provides a custom pooling
204
- layer.
205
-
206
- The main difference between the pooling layer in GritLM and the one in
207
- LlamaForCausalLM is that GritLM ignores the query instruction in the prompt
208
- when pooling the hidden states.
209
-
210
- Embedding prompts should be in the following format:
211
- - With instruction: "<|user|>\nINSTRUCTION\n<|embed|>\nPROMPT".
212
- - Without instruction: "<|embed|>\nPROMPT".
213
-
214
- Generation prompts should be in the following format:
215
- - "<|user|>\nPROMPT\n<|assistant|>\n"
216
- """
217
-
218
- is_pooling_model = True
219
-
220
- def __init__(
221
- self,
222
- vllm_config: VllmConfig,
223
- prefix: str = "",
224
- **kwargs,
225
- ) -> None:
226
- if vllm_config.model_config.runner_type == "pooling":
227
- hf_config = vllm_config.model_config.hf_config
228
- hf_config.is_causal = False
229
-
230
- vllm_config.cache_config.sliding_window = None
231
-
232
- hf_config.sliding_window = None
233
-
234
- super().__init__(vllm_config=vllm_config, prefix=prefix, **kwargs)
235
-
236
- pooler_config = vllm_config.model_config.pooler_config
237
- if pooler_config is not None:
238
- self.pooler = DispatchPooler(
239
- {
240
- "token_embed": pooler_for_token_embed(pooler_config),
241
- "embed": GritLMPooler(vllm_config.model_config),
242
- }
243
- )
vllm/model_executor/models/hyperclovax_vision.py
deleted
-1101
@@ -1,1101 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-# copied from : https://github.com/huggingface/transformers
4
-import ast
5
-from collections.abc import Iterable, Mapping, Sequence
6
-from functools import partial
7
-from itertools import accumulate
8
-from typing import Annotated, Literal
9
-
10
-import numpy as np
11
-import torch
12
-import torch.nn as nn
13
-from einops import rearrange
14
-from timm.layers import LayerNorm, LayerNorm2d
15
-from timm.models.regnet import RegStage
16
-from transformers import BatchFeature, CLIPVisionConfig, SiglipVisionConfig
17
-
18
-from vllm.config import VllmConfig
19
-from vllm.config.multimodal import BaseDummyOptions
20
-from vllm.inputs import MultiModalDataDict
21
-from vllm.model_executor.layers.quantization import QuantizationConfig
22
-from vllm.multimodal import MULTIMODAL_REGISTRY
23
-from vllm.multimodal.inputs import (
24
- MultiModalFieldConfig,
25
- MultiModalKwargsItems,
26
-)
27
-from vllm.multimodal.parse import ImageSize, MultiModalDataItems
28
-from vllm.multimodal.processing import (
29
- BaseDummyInputsBuilder,
30
- BaseMultiModalProcessor,
31
- BaseProcessingInfo,
32
- PromptReplacement,
33
- PromptUpdate,
34
-)
35
-from vllm.sequence import IntermediateTensors
36
-from vllm.utils.tensor_schema import TensorSchema, TensorShape
37
-
38
-from .clip import CLIPVisionModel
39
-from .interfaces import MultiModalEmbeddings, SupportsMultiModal, SupportsPP
40
-from .siglip import SiglipVisionModel
41
-from .utils import (
42
- AutoWeightsLoader,
43
- flatten_bn,
44
- init_vllm_registered_model,
45
- maybe_prefix,
46
-)
47
-from .vision import get_vision_encoder_info
48
-
49
-IMAGE_TOKEN: str = "<|dummy3|>"
50
-VIDEO_TOKEN: str = "<|_unuse_missing_100270|>"
51
-
52
-
53
-# Based on combine_frames_into_images in
54
-# https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B/blob/main/processing_hyperclovax.py
55
-def get_num_combined_frames(
56
- num_frames: int,
57
- max_grid_shape: tuple[int, int] = (3, 3),
58
-) -> int:
59
- max_num_grids = max_grid_shape[0] * max_grid_shape[1]
60
-
61
- # Calculate the number of canvases needed.
62
- num_canvases = num_frames // max_num_grids
63
- leftover_frames = num_frames % max_num_grids
64
-
65
- return num_canvases + (leftover_frames > 0)
66
-
67
-
68
-class HCXVisionImagePixelInputs(TensorSchema):
69
- """
70
- Dimensions:
71
- - n: Number of images
72
- - g: Number of grids
73
- - c: Number of channels (3)
74
- - h: Height
75
- - w: Width
76
- """
77
-
78
- type: Literal["pixel_values"] = "pixel_values"
79
- pixel_values_images: Annotated[
80
- list[torch.Tensor], TensorShape("n", "g", 3, "h", "w", dynamic_dims={"g"})
81
- ]
82
- image_sizes_images: Annotated[torch.Tensor, TensorShape("n", 2)]
83
-
84
-
85
-HCXVisionImageInputs = HCXVisionImagePixelInputs
86
-
87
-
88
-class HCXVisionVideoPixelInputs(TensorSchema):
89
- """
90
- Dimensions:
91
- - n: Number of videos
92
- - f: Number of frames
93
- - g: Number of grids
94
- - c: Number of channels (3)
95
- - h: Height
96
- - w: Width
97
- """
98
-
99
- type: Literal["pixel_values_videos"] = "pixel_values_videos"
100
- pixel_values_videos: Annotated[
101
- list[list[torch.Tensor]],
102
- TensorShape("n", "f", "g", 3, "h", "w", dynamic_dims={"f", "g"}),
103
- ]
104
-
105
-
106
-HCXVisionVideoInputs = HCXVisionVideoPixelInputs
107
-
108
-
109
-class HCXVisionProcessingInfo(BaseProcessingInfo):
110
- def get_vision_encoder_info(self):
111
- return get_vision_encoder_info(self.get_hf_config())
112
-
113
- def get_supported_mm_limits(self) -> Mapping[str, int | None]:
114
- return {"image": None, "video": None}
115
-
116
- def get_num_image_tokens(
117
- self,
118
- *,
119
- vision_query_length: int | list[int],
120
- ) -> int:
121
- if isinstance(vision_query_length, int):
122
- return vision_query_length
123
- else:
124
- return sum(vision_query_length)
125
-
126
- def get_num_video_tokens(
127
- self,
128
- *,
129
- vision_query_length: int | list[int],
130
- ) -> int:
131
- if isinstance(vision_query_length, int):
132
- return vision_query_length
133
- else:
134
- return sum(vision_query_length)
135
-
136
- def get_image_size_with_most_features(self) -> ImageSize:
137
- vision_encoder_info = self.get_vision_encoder_info()
138
- width = height = vision_encoder_info.get_image_size()
139
- return ImageSize(width=width, height=height)
140
-
141
- def get_max_image_tokens(self) -> int:
142
- target_width, target_height = self.get_image_size_with_most_features()
143
-
144
- return self.get_num_image_tokens(
145
- image_width=target_width,
146
- image_height=target_height,
147
- )
148
-
149
-
150
-class HCXVisionDummyInputsBuilder(BaseDummyInputsBuilder[HCXVisionProcessingInfo]):
151
- def get_dummy_text(
152
- self,
153
- mm_counts: Mapping[str, int],
154
- ) -> str:
155
- dummy_text = IMAGE_TOKEN * mm_counts.get(
156
- "image", 0
157
- ) + VIDEO_TOKEN * mm_counts.get("video", 0)
158
- return dummy_text
159
-
160
- def get_dummy_mm_data(
161
- self,
162
- seq_len: int,
163
- mm_counts: Mapping[str, int],
164
- mm_options: Mapping[str, BaseDummyOptions],
165
- ) -> MultiModalDataDict:
166
- num_images = mm_counts.get("image", 0)
167
- num_videos = mm_counts.get("video", 0)
168
-
169
- target_width, target_height = self.info.get_image_size_with_most_features()
170
- target_num_frames = 32
171
-
172
- image_overrides = mm_options.get("image")
173
- video_overrides = mm_options.get("video")
174
-
175
- return {
176
- "image": self._get_dummy_images(
177
- width=target_width,
178
- height=target_height,
179
- num_images=num_images,
180
- overrides=image_overrides,
181
- ),
182
- "video": self._get_dummy_videos(
183
- width=target_width - 1,
184
- height=target_height - 1,
185
- num_frames=target_num_frames,
186
- num_videos=num_videos,
187
- overrides=video_overrides,
188
- ),
189
- }
190
-
191
-
192
-class HCXVisionMultiModalProcessor(BaseMultiModalProcessor[HCXVisionProcessingInfo]):
193
- def _apply_hf_processor_main(
194
- self,
195
- mm_items: MultiModalDataItems,
196
- hf_processor_mm_kwargs: Mapping[str, object],
197
- ) -> BatchFeature:
198
- valid_mm_items = mm_items.select(
199
- {k for k, c in mm_items.get_all_counts().items() if c > 0}
200
- )
201
- mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
202
-
203
- if not mm_data:
204
- return BatchFeature(dict(passthrough_data))
205
-
206
- for video_idx, video_arr in enumerate(mm_data.get("videos", [])):
207
- if video_arr.dtype != np.uint8:
208
- mm_data["videos"][video_idx] = video_arr.astype(np.uint8)
209
-
210
- images = mm_data.get("images")
211
- videos = mm_data.get("videos")
212
-
213
- # batchify input as a single item
214
- processed_data = self.info.ctx.call_hf_processor(
215
- hf_processor=self.info.get_hf_processor(**hf_processor_mm_kwargs),
216
- data=dict(
217
- images=None if images is None else [images],
218
- videos=None if videos is None else [videos],
219
- ),
220
- )
221
-
222
- for k, v in processed_data.items():
223
- if isinstance(v, list) and len(v) > 0:
224
- assert len(v) == 1
225
- processed_data[k] = v[0]
226
-
227
- if images:
228
- processed_data["image_sizes_images"] = torch.tensor(
229
- processed_data["image_sizes_images"]
230
- )
231
- processed_data["vision_query_lengths_images"] = torch.tensor(
232
- processed_data["vision_query_lengths_images"]
233
- )
234
-
235
- if videos:
236
- _idx_per_video = [
237
- 0,
238
- *accumulate(get_num_combined_frames(len(video)) for video in videos),
239
- ]
240
- processed_data["pixel_values_videos"] = [
241
- processed_data["pixel_values_videos"][
242
- _idx_per_video[i] : _idx_per_video[i + 1]
243
- ]
244
- for i in range(len(videos))
245
- ]
246
- processed_data["vision_query_lengths_videos"] = [
247
- torch.tensor(
248
- processed_data["vision_query_lengths_videos"][
249
- _idx_per_video[i] : _idx_per_video[i + 1]
250
- ]
251
- )
252
- for i in range(len(videos))
253
- ]
254
-
255
- processed_data.update(passthrough_data)
256
-
257
- return processed_data
258
-
259
- def _get_prompt_updates(
260
- self,
261
- mm_items: MultiModalDataItems,
262
- hf_processor_mm_kwargs: Mapping[str, object],
263
- out_mm_kwargs: MultiModalKwargsItems,
264
- ) -> Sequence[PromptUpdate]:
265
- hf_config = self.info.get_hf_config()
266
- placeholder = {
267
- "image": hf_config.image_token_id,
268
- "video": hf_config.video_token_id,
269
- }
270
-
271
- def get_replacement_hyperclovax(
272
- item_idx: int,
273
- modality: str,
274
- out_mm_kwargs: MultiModalKwargsItems,
275
- ):
276
- out_item = out_mm_kwargs[modality][item_idx]
277
-
278
- if modality == "image":
279
- lens = out_item["vision_query_lengths_images"].data.tolist()
280
- num_tokens = self.info.get_num_image_tokens(vision_query_length=lens)
281
- elif modality == "video":
282
- lens = out_item["vision_query_lengths_videos"].data.tolist()
283
- num_tokens = self.info.get_num_video_tokens(vision_query_length=lens)
284
- else:
285
- raise NotImplementedError(modality)
286
-
287
- return [placeholder[modality]] * num_tokens
288
-
289
- return [
290
- PromptReplacement(
291
- modality=modality,
292
- target=[
293
- placeholder[modality],
294
- ],
295
- replacement=partial(
296
- get_replacement_hyperclovax,
297
- modality=modality,
298
- out_mm_kwargs=out_mm_kwargs,
299
- ),
300
- )
301
- for modality in ("image", "video")
302
- ]
303
-
304
- def _get_mm_fields_config(
305
- self,
306
- hf_inputs: BatchFeature,
307
- hf_processor_mm_kwargs: Mapping[str, object],
308
- ) -> Mapping[str, MultiModalFieldConfig]:
309
- fields = dict(
310
- pixel_values_images=MultiModalFieldConfig.batched("image"),
311
- image_sizes_images=MultiModalFieldConfig.batched("image", keep_on_cpu=True),
312
- vision_query_lengths_images=MultiModalFieldConfig.batched(
313
- "image", keep_on_cpu=True
314
- ),
315
- pixel_values_videos=MultiModalFieldConfig.batched("video"),
316
- vision_query_lengths_videos=MultiModalFieldConfig.batched(
317
- "video", keep_on_cpu=True
318
- ),
319
- )
320
-
321
- return fields
322
-
323
-
324
-def init_vision_tower_for_hcxvision(
325
- vision_config,
326
- quant_config: QuantizationConfig | None,
327
- *,
328
- use_nth_layer: int | None = None,
329
- require_post_norm: bool | None = None,
330
- prefix: str = "",
331
-) -> CLIPVisionModel | SiglipVisionModel:
332
- num_hidden_layers = vision_config.num_hidden_layers
333
- if not isinstance(use_nth_layer, int):
334
- pass
335
- elif use_nth_layer >= 0:
336
- num_hidden_layers = use_nth_layer + 1
337
- else:
338
- num_hidden_layers = num_hidden_layers + use_nth_layer + 1
339
-
340
- if isinstance(vision_config, CLIPVisionConfig):
341
- return CLIPVisionModel(
342
- vision_config,
343
- quant_config=quant_config,
344
- num_hidden_layers_override=num_hidden_layers,
345
- require_post_norm=require_post_norm,
346
- prefix=prefix,
347
- )
348
- elif isinstance(vision_config, SiglipVisionConfig):
349
- return SiglipVisionModel(
350
- vision_config,
351
- quant_config=quant_config,
352
- num_hidden_layers_override=num_hidden_layers,
353
- require_post_norm=require_post_norm,
354
- prefix=prefix,
355
- )
356
-
357
- msg = f"Unsupported vision config: {type(vision_config)}"
358
- raise NotImplementedError(msg)
359
-
360
-
361
-class HCXVisionMlp(nn.Module):
362
- def __init__(
363
- self,
364
- mm_projector_type,
365
- in_features,
366
- hidden_features=None,
367
- out_features=None,
368
- act_layer=nn.GELU,
369
- ):
370
- super().__init__()
371
- out_features = out_features or in_features
372
- hidden_features = hidden_features or in_features
373
- self.mm_projector_type = mm_projector_type
374
- if self.mm_projector_type == "mlp":
375
- self.fc1 = nn.Linear(in_features, hidden_features)
376
- self.act = act_layer()
377
- self.fc2 = nn.Linear(hidden_features, out_features)
378
- elif self.mm_projector_type == "inverted_mlp":
379
- self.fc1 = nn.Linear(in_features, 2 * hidden_features)
380
- self.act = act_layer()
381
- self.fc2 = nn.Linear(2 * hidden_features, out_features)
382
- else:
383
- raise NotImplementedError(
384
- "{} is not implemented".format(self.mm_projector_type)
385
- )
386
-
387
- def forward(self, x):
388
- x = self.fc1(x)
389
- x = self.act(x)
390
- x = self.fc2(x)
391
- return x
392
-
393
-
394
-class HCXVisionCAbstractor(nn.Module):
395
- """
396
- This module is based on C-Abstractor, whose license is under apache-2.0.
397
- You can check the original code at
398
- https://github.com/khanrc/honeybee/blob/main/honeybee/projectors/projectors.py
399
- and we made necessary modifications.
400
- """
401
-
402
- def __init__(
403
- self,
404
- num_queries: int,
405
- num_input_tokens: int,
406
- encoder_hidden_size: int,
407
- hidden_size: int,
408
- output_hidden_size: int,
409
- pos_emb: bool = True,
410
- prenorm: bool = False,
411
- ):
412
- super().__init__()
413
- self.num_input_tokens = num_input_tokens
414
- self.output_hidden_size = output_hidden_size
415
-
416
- # Positional embedding
417
- if pos_emb:
418
- self.pos_emb = torch.nn.Parameter(
419
- torch.zeros(1, num_input_tokens, encoder_hidden_size)
420
- )
421
- self.pos_emb.data.normal_(mean=0.0, std=0.02)
422
- else:
423
- self.pos_emb = None
424
-
425
- # (Optional) Pre-normalization layer
426
- if prenorm:
427
- self.prenorm = LayerNorm(encoder_hidden_size)
428
- else:
429
- self.prenorm = None
430
-
431
- self.build_net(
432
- num_queries, encoder_hidden_size, hidden_size, output_hidden_size
433
- )
434
- self.dtype = next(self.parameters()).dtype
435
-
436
- def forward(
437
- self,
438
- x: torch.Tensor,
439
- num_queries_vis_abstractors: list[list[int]] | None = None,
440
- num_grids: list[int] | None = None,
441
- ) -> torch.Tensor:
442
- if self.prenorm is not None:
443
- x = self.prenorm(x)
444
-
445
- if self.pos_emb is not None:
446
- x = x + self.pos_emb
447
-
448
- x = self._forward(
449
- x,
450
- num_queries_vis_abstractors=num_queries_vis_abstractors,
451
- num_grids=num_grids,
452
- ) # (B, L, output_hidden_size)
453
-
454
- return x
455
-
456
- def _forward(
457
- self,
458
- x: torch.Tensor,
459
- num_queries_vis_abstractors: list[list[int]] | None = None,
460
- num_grids: list[int] | None = None,
461
- ) -> torch.Tensor:
462
- # x: [B, L, dim]
463
- B, L, dim = x.shape
464
- hw = int(L**0.5)
465
- x = rearrange(x, "b (h w) d -> b d h w", h=hw, w=hw)
466
-
467
- if num_queries_vis_abstractors is not None:
468
- assert num_grids is not None
469
- return self._forward_adaptive_num_query(
470
- x, num_queries_vis_abstractors, num_grids
471
- )
472
-
473
- x = self.net(x)
474
- x = rearrange(x, "b d h w -> b (h w) d")
475
- x = self.readout(x)
476
- return x
477
-
478
- def _forward_adaptive_num_query(
479
- self,
480
- x: torch.Tensor,
481
- num_queries_vis_abstractors: list[list[int]] | None = None,
482
- num_grids: list[int] | None = None,
483
- ) -> list[torch.Tensor]:
484
- # self.net is consisted by 3 layers (s1, sampler, s2)
485
- assert len(self.net) == 3
486
-
487
- x = self.net[0](x) # s1
488
- new_x = []
489
- for i, num_queries in enumerate(num_queries_vis_abstractors):
490
- hw = int(num_queries**0.5)
491
- sampler = nn.AdaptiveAvgPool2d((hw, hw))
492
- out = sampler(x[num_grids[i] : num_grids[i + 1], :])
493
- out = self.net[2](out) # s2
494
-
495
- out = rearrange(out, "b d h w -> b (h w) d")
496
- out = self.readout(out)
497
-
498
- new_x.append(out)
499
- return new_x
500
-
501
- def build_net(
502
- self,
503
- n_queries: int,
504
- encoder_hidden_size: int,
505
- hidden_size: int,
506
- output_hidden_size: int,
507
- depth: int = 3,
508
- mlp_depth: int = 2,
509
- ):
510
- assert (n_queries**0.5).is_integer(), (
511
- f"n_queries must be square number. n_queries: {n_queries}"
512
- )
513
- hw = int(n_queries**0.5)
514
-
515
- # RegBlock = ResBlock + SE
516
- RegBlock = partial(
517
- RegStage,
518
- stride=1,
519
- dilation=1,
520
- act_layer=nn.SiLU,
521
- norm_layer=LayerNorm2d,
522
- )
523
-
524
- s1 = RegBlock(
525
- depth,
526
- encoder_hidden_size,
527
- hidden_size,
528
- )
529
- sampler = nn.AdaptiveAvgPool2d((hw, hw))
530
- s2 = RegBlock(
531
- depth,
532
- hidden_size,
533
- hidden_size,
534
- )
535
-
536
- self.net = nn.Sequential(s1, sampler, s2)
537
- self.readout = self.build_mlp(mlp_depth, hidden_size, output_hidden_size)
538
-
539
- def build_mlp(
540
- self,
541
- depth: int,
542
- hidden_size: int,
543
- output_hidden_size: int,
544
- ):
545
- layers = [nn.Linear(hidden_size, output_hidden_size)]
546
- for _ in range(1, depth):
547
- layers.append(nn.SiLU())
548
- layers.append(nn.Linear(output_hidden_size, output_hidden_size))
549
- return nn.Sequential(*layers)
550
-
551
-
552
-@MULTIMODAL_REGISTRY.register_processor(
553
- HCXVisionMultiModalProcessor,
554
- info=HCXVisionProcessingInfo,
555
- dummy_inputs=HCXVisionDummyInputsBuilder,
556
-)
557
-class HCXVisionForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
558
- """
559
- HyperCLOVAX-SEED Vision-Language Model (V1 architecture).
560
-
561
- Supports:
562
- - HyperCLOVAX-SEED-Vision-Instruct-3B
563
-
564
- Uses CLIP/SigLIP as the vision encoder with C-Abstractor projector.
565
- """
566
-
567
- packed_modules_mapping = {
568
- "qkv_proj": ["q_proj", "k_proj", "v_proj"],
569
- "gate_up_proj": ["gate_proj", "up_proj"],
570
- }
571
-
572
- def __init__(
573
- self,
574
- *,
575
- vllm_config: VllmConfig,
576
- prefix: str = "",
577
- ) -> None:
578
- super().__init__()
579
-
580
- # init configs
581
- config = vllm_config.model_config.hf_config
582
- quant_config = vllm_config.quant_config
583
- # text_config
584
- text_config = config.text_config
585
- if text_config.model_type in ["gpt2", "hyperclovax", "llama"]:
586
- text_config._attn_implementation = "sdpa"
587
- if text_config.model_type != "hyperclovax":
588
- text_config.logits_scaling = 1.0
589
- # vision_config
590
- vision_config = config.vision_config
591
- vision_config.auto_map = {}
592
- vision_config.anyres = config.anyres
593
- vision_config.max_num_grids = config.max_num_grids
594
- self.dtype = vllm_config.model_config.dtype
595
-
596
- ## possible_resolution should be matched with preprocessor_config.json
597
- config.possible_resolutions = self._init_possible_resolutions(
598
- config, vision_config
599
- )
600
-
601
- with self._mark_tower_model(vllm_config, {"image", "video"}):
602
- self.vision_model = init_vision_tower_for_hcxvision(
603
- vision_config,
604
- quant_config=quant_config,
605
- use_nth_layer=getattr(config, "use_nth_layer", -1),
606
- require_post_norm=False,
607
- prefix=maybe_prefix(prefix, "vision_model"),
608
- )
609
- self.mm_projector = self._init_mm_projector(
610
- config, text_config, vision_config
611
- )
612
-
613
- if config.anyres:
614
- self.image_newline = nn.Parameter(
615
- torch.empty(text_config.hidden_size, dtype=self.dtype)
616
- )
617
-
618
- with self._mark_language_model(vllm_config):
619
- self.language_model = init_vllm_registered_model(
620
- vllm_config=vllm_config,
621
- hf_config=text_config,
622
- prefix=maybe_prefix(prefix, "language_model"),
623
- )
624
-
625
- self.config = config
626
- self.vision_config = vision_config
627
- self.text_config = text_config
628
-
629
- self.make_empty_intermediate_tensors = (
630
- self.language_model.make_empty_intermediate_tensors
631
- )
632
-
633
- @classmethod
634
- def get_placeholder_str(cls, modality: str, i: int) -> str | None:
635
- if modality.startswith("image"):
636
- return IMAGE_TOKEN
637
- if modality.startswith("video"):
638
- return VIDEO_TOKEN
639
-
640
- raise ValueError("Only image or video modality is supported")
641
-
642
- def _parse_and_validate_image_input(
643
- self,
644
- **kwargs: object,
645
- ) -> HCXVisionImageInputs | None:
646
- pixel_values_images = kwargs.pop("pixel_values_images", None)
647
-
648
- if pixel_values_images is None:
649
- return None
650
-
651
- image_sizes_images = kwargs.pop("image_sizes_images")
652
-
653
- return HCXVisionImagePixelInputs(
654
- pixel_values_images=pixel_values_images,
655
- image_sizes_images=image_sizes_images,
656
- )
657
-
658
- def _parse_and_validate_video_input(
659
- self,
660
- **kwargs: object,
661
- ) -> HCXVisionVideoInputs | None:
662
- pixel_values_videos = kwargs.pop("pixel_values_videos", None)
663
-
664
- if pixel_values_videos is None:
665
- return None
666
-
667
- return HCXVisionVideoPixelInputs(
668
- pixel_values_videos=pixel_values_videos,
669
- )
670
-
671
- def _process_image_input(
672
- self,
673
- image_input: HCXVisionImageInputs,
674
- ) -> tuple[torch.Tensor, ...]:
675
- return self.forward_images(
676
- pixel_values_images=image_input["pixel_values_images"],
677
- image_sizes_images=image_input["image_sizes_images"],
678
- )
679
-
680
- def _process_video_input(
681
- self,
682
- video_input: HCXVisionVideoInputs,
683
- ) -> tuple[torch.Tensor, ...]:
684
- return self.forward_videos(
685
- pixel_values_videos=video_input["pixel_values_videos"],
686
- )
687
-
688
- def _parse_and_validate_multimodal_inputs(self, **kwargs: object) -> dict:
689
- modalities = {}
690
-
691
- # Preserve the order of modalities if there are multiple of them
692
- # from the order of kwargs.
693
- for input_key in kwargs:
694
- if input_key == "pixel_values_images" and "images" not in modalities:
695
- modalities["images"] = self._parse_and_validate_image_input(**kwargs)
696
- if input_key == "pixel_values_videos" and "videos" not in modalities:
697
- modalities["videos"] = self._parse_and_validate_video_input(**kwargs)
698
-
699
- return modalities
700
-
701
- def embed_multimodal(
702
- self,
703
- **kwargs: object,
704
- ) -> MultiModalEmbeddings:
705
- modalities = self._parse_and_validate_multimodal_inputs(**kwargs)
706
- if not modalities:
707
- return []
708
-
709
- # The result multimodal_embeddings is tuple of tensors, with each
710
- # tensor correspoending to a multimodal data item (image or video).
711
- multimodal_embeddings: tuple[torch.Tensor, ...] = ()
712
-
713
- # NOTE: It is important to iterate over the keys in this dictionary
714
- # to preserve the order of the modalities.
715
- for modality in modalities:
716
- if modality == "images":
717
- image_input = modalities["images"]
718
- image_embeddings = self._process_image_input(image_input)
719
- multimodal_embeddings += tuple(image_embeddings)
720
- if modality == "videos":
721
- video_input = modalities["videos"]
722
- video_embeddings = self._process_video_input(video_input)
723
- multimodal_embeddings += tuple(video_embeddings)
724
-
725
- return multimodal_embeddings
726
-
727
- def forward(
728
- self,
729
- input_ids: torch.Tensor | None,
730
- positions: torch.Tensor,
731
- intermediate_tensors: IntermediateTensors | None = None,
732
- inputs_embeds: torch.Tensor | None = None,
733
- **kwargs: object,
734
- ) -> torch.Tensor | IntermediateTensors:
735
- if intermediate_tensors is not None:
736
- inputs_embeds = None
737
-
738
- hidden_states = self.language_model.model(
739
- input_ids, positions, intermediate_tensors, inputs_embeds=inputs_embeds
740
- )
741
- return hidden_states
742
-
743
- def forward_images(
744
- self,
745
- pixel_values_images: list[torch.Tensor],
746
- image_sizes_images: torch.Tensor,
747
- ) -> tuple[torch.Tensor, ...]:
748
- pixel_values_image_flat = flatten_bn(pixel_values_images, concat=True)
749
-
750
- visual_token_idx = 0 if "siglip" in self.vision_config.model_type else 1
751
- image_forward_outs = self.vision_model(pixel_values_image_flat)[
752
- :, visual_token_idx:
753
- ]
754
-
755
- image_forward_outs = image_forward_outs.to(dtype=self.mm_projector.dtype)
756
- image_forward_outs = self.mm_projector(image_forward_outs) # b (h w) d
757
-
758
- split_sizes = [len(item) for item in pixel_values_images]
759
- image_forward_outs = torch.split(image_forward_outs, split_sizes, dim=0)
760
-
761
- # newline for anyres postprocessing
762
- image_features = anyres_postprocessing(
763
- image_forward_outs=image_forward_outs,
764
- image_sizes=image_sizes_images.tolist(),
765
- num_queries_vis_abstractor=self.config.num_queries_vis_abstractor_image,
766
- unpad=self.config.unpad,
767
- patch_size=self.vision_config.patch_size,
768
- grid_size=self.vision_config.image_size,
769
- image_newline=self.image_newline,
770
- possible_resolutions=self.config.possible_resolutions,
771
- )
772
-
773
- return tuple(image_features)
774
-
775
- def forward_videos(
776
- self,
777
- pixel_values_videos: list[list[torch.Tensor]],
778
- ) -> tuple[torch.Tensor, ...]:
779
- pixel_values_videos_flat = flatten_bn(
780
- [frame for frames in pixel_values_videos for frame in frames],
781
- concat=True,
782
- )
783
-
784
- visual_token_idx = 0 if "siglip" in self.vision_config.model_type else 1
785
- video_forward_outs = self.vision_model(pixel_values_videos_flat)[
786
- :, visual_token_idx:
787
- ]
788
-
789
- video_forward_outs = video_forward_outs.to(dtype=self.mm_projector.dtype)
790
-
791
- # Run MM-Projector
792
- # len(num_grids) == len(num_queries_vis_abstractors) + 1
793
- grid_idx = 0
794
- # e.g. [0, 9, 18, 19, 27, 28, 36, 37, 45, 46, 54, 55, 56]
795
- num_grids = [grid_idx]
796
- # e.g. [81, 81, 81, 9, 81, 9, 81, 9, 81, 9, 81, 9]
797
- num_queries_vis_abstractors = []
798
- len_total_frames = video_forward_outs.shape[0]
799
-
800
- if self.config.first_last_frames_slow:
801
- # slowfast (first_last_frames_slow)
802
- assert len_total_frames != 0
803
- if len_total_frames <= 2:
804
- num_queries_vis_abstractors.append(
805
- self.config.num_queries_vis_abstractor_video_slow
806
- )
807
- grid_idx += len_total_frames
808
- num_grids.append(grid_idx)
809
- else:
810
- num_queries_vis_abstractors.append(
811
- self.config.num_queries_vis_abstractor_video_slow
812
- )
813
- grid_idx += 1
814
- num_grids.append(grid_idx)
815
-
816
- num_queries_vis_abstractors.append(
817
- self.config.num_queries_vis_abstractor_video_fast
818
- )
819
- grid_idx += len_total_frames - 2
820
- num_grids.append(grid_idx)
821
-
822
- num_queries_vis_abstractors.append(
823
- self.config.num_queries_vis_abstractor_video_slow
824
- )
825
- grid_idx += 1
826
- num_grids.append(grid_idx)
827
- else:
828
- # slowfast
829
- for pixel_values_frames in pixel_values_videos:
830
- for pixel_values_frame in pixel_values_frames:
831
- if len(pixel_values_frame) > 0:
832
- num_queries_vis_abstractors.append(
833
- self.config.num_queries_vis_abstractor_video_slow
834
- )
835
- grid_idx += 1
836
- num_grids.append(grid_idx)
837
- num_queries_vis_abstractors.append(
838
- self.config.num_queries_vis_abstractor_video_fast
839
- )
840
- grid_idx = grid_idx + len(pixel_values_frame) - 1
841
- num_grids.append(grid_idx)
842
-
843
- video_forward_outs = self.mm_projector(
844
- video_forward_outs, num_queries_vis_abstractors, num_grids
845
- )
846
-
847
- video_features = [] # what we want to return
848
- target_features = []
849
- target_group_size = 0
850
- group_counter = 0
851
- video_groups = [
852
- len(frame) for frames in pixel_values_videos for frame in frames
853
- ] # for concat video features after projector
854
-
855
- for forward_out in video_forward_outs:
856
- target_group_size += len(forward_out)
857
- target_features.append(forward_out.flatten(0, 1))
858
-
859
- video_group_size = video_groups[group_counter]
860
- if video_group_size == target_group_size:
861
- video_features.append(torch.cat(target_features, dim=0))
862
- target_features = []
863
- group_counter += 1
864
- target_group_size = 0
865
-
866
- elif video_group_size < target_group_size:
867
- raise RuntimeError(f"{video_group_size=} < {target_group_size=}")
868
-
869
- assert len(target_features) == 0, (
870
- f"target_features is not empty!! {target_features}"
871
- )
872
- assert len(video_groups) == len(video_features)
873
-
874
- feats_per_video = [len(video) for video in pixel_values_videos]
875
- idxs_per_video = [0, *accumulate(feats_per_video)]
876
- return tuple(
877
- torch.cat(video_features[idxs_per_video[i] : idxs_per_video[i + 1]])
878
- for i in range(len(feats_per_video))
879
- )
880
-
881
- def compute_logits(
882
- self,
883
- hidden_states: torch.Tensor,
884
- ) -> torch.Tensor | None:
885
- return self.language_model.compute_logits(hidden_states)
886
-
887
- def load_weights(
888
- self,
889
- weights: Iterable[tuple[str, torch.Tensor]],
890
- ) -> set[str]:
891
- loader = AutoWeightsLoader(self)
892
- return loader.load_weights(weights)
893
-
894
- def _init_possible_resolutions(
895
- self,
896
- config,
897
- vision_config,
898
- ):
899
- if not getattr(config, "possible_resolutions", []):
900
- possible_resolutions = []
901
- if config.anyres:
902
- assert config.max_num_grids > 0
903
- for i in range(1, config.max_num_grids + 1):
904
- for j in range(1, config.max_num_grids + 1):
905
- if i == 1 and j == 1 and not config.use_1x1_grid:
906
- continue
907
- if i * j <= config.max_num_grids:
908
- possible_resolutions.append([i, j])
909
-
910
- possible_resolutions = [
911
- [ys * vision_config.image_size, xs * vision_config.image_size]
912
- for ys, xs in possible_resolutions
913
- ]
914
- return possible_resolutions
915
- else:
916
- return config.possible_resolutions
917
-
918
- def _init_mm_projector(
919
- self,
920
- config,
921
- text_config,
922
- vision_config,
923
- ):
924
- input_hidden_size = vision_config.hidden_size
925
- if config.mm_projector_type == "linear":
926
- mm_projector = nn.Linear(input_hidden_size, text_config.hidden_size)
927
- mm_projector.dtype = next(mm_projector.parameters()).dtype
928
- elif config.mm_projector_type == "cabstractor":
929
- mm_projector = HCXVisionCAbstractor(
930
- num_queries=config.num_queries_vis_abstractor_image,
931
- num_input_tokens=(vision_config.image_size // vision_config.patch_size)
932
- ** 2,
933
- encoder_hidden_size=input_hidden_size,
934
- hidden_size=input_hidden_size,
935
- output_hidden_size=text_config.hidden_size,
936
- pos_emb=config.proj_pos_emb,
937
- prenorm=config.proj_prenorm,
938
- )
939
- else:
940
- mm_projector = HCXVisionMlp(
941
- config.mm_projector_type,
942
- input_hidden_size,
943
- hidden_features=input_hidden_size,
944
- out_features=self.text_config.hidden_size,
945
- )
946
- return mm_projector
947
-
948
-
949
-def unpad_image(tensor: torch.Tensor, original_size: tuple[int, int]) -> torch.Tensor:
950
- original_width, original_height = original_size
951
- current_height, current_width = tensor.shape[1:]
952
-
953
- original_aspect_ratio = original_width / original_height
954
- current_aspect_ratio = current_width / current_height
955
-
956
- if original_aspect_ratio > current_aspect_ratio:
957
- scale_factor = current_width / original_width
958
- new_height = int(original_height * scale_factor)
959
- padding = (current_height - new_height) // 2
960
- unpadded_tensor = tensor[:, padding : current_height - padding, :]
961
- else:
962
- scale_factor = current_height / original_height
963
- new_width = int(original_width * scale_factor)
964
- padding = (current_width - new_width) // 2
965
- unpadded_tensor = tensor[:, :, padding : current_width - padding]
966
-
967
- return unpadded_tensor
968
-
969
-
970
-def select_best_resolution(original_size: tuple, possible_resolutions: list) -> tuple:
971
- original_height, original_width = original_size
972
- best_fit = None
973
- max_effective_resolution = 0
974
- min_wasted_resolution = float("inf")
975
-
976
- for height, width in possible_resolutions:
977
- scale = min(width / original_width, height / original_height)
978
- downscaled_width, downscaled_height = (
979
- int(original_width * scale),
980
- int(original_height * scale),
981
- )
982
- effective_resolution = min(
983
- downscaled_width * downscaled_height, original_width * original_height
984
- )
985
- wasted_resolution = (width * height) - effective_resolution
986
-
987
- if effective_resolution > max_effective_resolution or (
988
- effective_resolution == max_effective_resolution
989
- and wasted_resolution < min_wasted_resolution
990
- ):
991
- max_effective_resolution = effective_resolution
992
- min_wasted_resolution = wasted_resolution
993
- best_fit = (height, width)
994
-
995
- return best_fit
996
-
997
-
998
-def get_anyres_image_grid_shape(
999
- image_size: tuple[int, int],
1000
- grid_pinpoints: str | list[tuple[int, int]],
1001
- patch_size: int,
1002
-) -> tuple[int, int]:
1003
- possible_resolutions = (
1004
- grid_pinpoints
1005
- if isinstance(grid_pinpoints, list)
1006
- else ast.literal_eval(grid_pinpoints)
1007
- )
1008
-
1009
- original_width, original_height = image_size
1010
- height, width = select_best_resolution(
1011
- (original_height, original_width), possible_resolutions
1012
- )
1013
- return width // patch_size, height // patch_size
1014
-
1015
-
1016
-def reshape_and_unpad_image_features(
1017
- image_feature: torch.Tensor,
1018
- height: int,
1019
- width: int,
1020
- image_size: tuple[int, int],
1021
- possible_resolutions: list[tuple[int, int]],
1022
- grid_size: int,
1023
- unpad: bool,
1024
- image_newline: torch.Tensor,
1025
-) -> torch.Tensor:
1026
- base_image_feature = image_feature[0]
1027
- image_feature = image_feature[1:]
1028
-
1029
- assert height * width == base_image_feature.shape[0], (
1030
- f"{height=} * {width=} != {base_image_feature.shape[0]=}"
1031
- )
1032
-
1033
- num_patch_width, num_patch_height = get_anyres_image_grid_shape(
1034
- image_size, possible_resolutions, grid_size
1035
- )
1036
- image_feature = image_feature.view(
1037
- num_patch_height, num_patch_width, height, width, -1
1038
- )
1039
-
1040
- if unpad:
1041
- image_feature = image_feature.permute(4, 0, 2, 1, 3).contiguous()
1042
- image_feature = image_feature.flatten(1, 2).flatten(2, 3)
1043
- image_feature = unpad_image(image_feature, image_size)
1044
- image_feature = torch.cat(
1045
- (
1046
- image_feature,
1047
- image_newline[:, None, None]
1048
- .expand(*image_feature.shape[:-1], 1)
1049
- .to(image_feature.device),
1050
- ),
1051
- dim=-1,
1052
- )
1053
- image_feature = image_feature.flatten(1, 2).transpose(0, 1)
1054
- else:
1055
- image_feature = image_feature.permute(0, 2, 1, 3, 4).contiguous()
1056
- image_feature = image_feature.flatten(0, 3)
1057
- image_feature = torch.cat((base_image_feature, image_feature), dim=0)
1058
-
1059
- return image_feature
1060
-
1061
-
1062
-def anyres_postprocessing(
1063
- image_forward_outs: list[torch.Tensor],
1064
- image_sizes: list[list[int]],
1065
- possible_resolutions: list[tuple[int, int]],
1066
- patch_size: int,
1067
- grid_size: int,
1068
- image_newline: torch.Tensor,
1069
- num_queries_vis_abstractor: int = -1,
1070
- unpad: bool = False,
1071
-) -> list[torch.Tensor]:
1072
- height = width = grid_size // patch_size
1073
-
1074
- if num_queries_vis_abstractor > 0:
1075
- assert (num_queries_vis_abstractor**0.5).is_integer(), (
1076
- "n_queries must be square number"
1077
- )
1078
- height = width = int(num_queries_vis_abstractor**0.5)
1079
-
1080
- # post-processing (unpad, add newline)
1081
- new_image_features = []
1082
- for image_idx, image_feature in enumerate(image_forward_outs):
1083
- if image_feature.shape[0] > 1:
1084
- image_feature = reshape_and_unpad_image_features(
1085
- image_feature=image_feature,
1086
- height=height,
1087
- width=width,
1088
- image_size=image_sizes[image_idx],
1089
- possible_resolutions=possible_resolutions,
1090
- grid_size=grid_size, # Pass grid info if needed by helper
1091
- unpad=unpad,
1092
- image_newline=image_newline,
1093
- )
1094
- else:
1095
- image_feature = image_feature[0]
1096
- image_feature = torch.cat(
1097
- (image_feature, image_newline[None].to(image_feature.device)), dim=0
1098
- )
1099
- new_image_features.append(image_feature)
1100
-
1101
- return new_image_features
vllm/model_executor/models/llama.py
+1
-1
@@ -273,7 +273,7 @@ class LlamaDecoderLayer(nn.Module):
273
# By default, Llama uses causal attention as it is a decoder-only model.
274
# You can override the HF config with `is_causal=False` to enable
275
# bidirectional attention, which is used in some embedding models
276
- # (e.g. parasail-ai/GritLM-7B-vllm)
276
+ # (e.g. nvidia/llama-nemotron-embed-1b-v2)
277
if getattr(config, "is_causal", True):
278
attn_type = AttentionType.DECODER
279
else:
vllm/model_executor/models/mpt.py
deleted
-317
@@ -1,317 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-# Adapted from https://huggingface.co/mosaicml/mpt-7b/tree/main
5
-import math
6
-from collections.abc import Iterable
7
-from itertools import islice
8
-
9
-import torch
10
-import torch.nn as nn
11
-from transformers import MptConfig
12
-
13
-from vllm.compilation.decorators import support_torch_compile
14
-from vllm.config import CacheConfig, VllmConfig
15
-from vllm.distributed import (
16
- get_pp_group,
17
- get_tensor_model_parallel_rank,
18
- get_tensor_model_parallel_world_size,
19
-)
20
-from vllm.model_executor.layers.activation import get_act_fn
21
-from vllm.model_executor.layers.attention import Attention
22
-from vllm.model_executor.layers.linear import (
23
- ColumnParallelLinear,
24
- QKVParallelLinear,
25
- RowParallelLinear,
26
-)
27
-from vllm.model_executor.layers.logits_processor import LogitsProcessor
28
-from vllm.model_executor.layers.quantization import QuantizationConfig
29
-from vllm.model_executor.layers.vocab_parallel_embedding import VocabParallelEmbedding
30
-from vllm.sequence import IntermediateTensors
31
-
32
-from .interfaces import SupportsPP
33
-from .utils import (
34
- AutoWeightsLoader,
35
- make_empty_intermediate_tensors_factory,
36
- make_layers,
37
- maybe_prefix,
38
-)
39
-
40
-
41
-def _get_alibi_slopes(
42
- total_num_heads: int,
43
- alibi_bias_max: int,
44
-) -> torch.Tensor:
45
- next_power_of_2 = 2 ** math.ceil(math.log2(total_num_heads))
46
- m = torch.arange(1, next_power_of_2 + 1, dtype=torch.float32)
47
- m = m.mul(alibi_bias_max / next_power_of_2)
48
- slopes = 1.0 / torch.pow(2, m)
49
- if next_power_of_2 != total_num_heads:
50
- slopes = torch.concat([slopes[1::2], slopes[::2]])[:total_num_heads]
51
- return slopes
52
-
53
-
54
-class MPTAttention(nn.Module):
55
- def __init__(
56
- self,
57
- config: MptConfig,
58
- cache_config: CacheConfig | None = None,
59
- quant_config: QuantizationConfig | None = None,
60
- prefix: str = "",
61
- ):
62
- super().__init__()
63
- self.d_model = config.d_model
64
- self.total_num_heads = config.n_heads
65
- self.head_dim = self.d_model // self.total_num_heads
66
- self.clip_qkv = config.attn_config.clip_qkv
67
- self.qk_ln = config.attn_config.qk_ln
68
- self.alibi_bias_max = config.attn_config.alibi_bias_max
69
- if "kv_n_heads" in config.attn_config:
70
- self.total_num_kv_heads = config.attn_config.kv_n_heads
71
- else:
72
- self.total_num_kv_heads = self.total_num_heads
73
- assert not config.attn_config.prefix_lm
74
- assert config.attn_config.alibi
75
-
76
- # pylint: disable=invalid-name
77
- self.Wqkv = QKVParallelLinear(
78
- self.d_model,
79
- self.d_model // self.total_num_heads,
80
- self.total_num_heads,
81
- self.total_num_kv_heads,
82
- bias=not config.no_bias,
83
- quant_config=quant_config,
84
- prefix=f"{prefix}.Wqkv",
85
- )
86
- if self.qk_ln:
87
- self.q_ln = nn.LayerNorm(self.d_model)
88
- self.k_ln = nn.LayerNorm(self.d_model)
89
- self.out_proj = RowParallelLinear(
90
- self.d_model,
91
- self.d_model,
92
- bias=not config.no_bias,
93
- quant_config=quant_config,
94
- prefix=f"{prefix}.out_proj",
95
- )
96
-
97
- tp_world_size = get_tensor_model_parallel_world_size()
98
- assert self.total_num_heads % tp_world_size == 0
99
- self.num_heads = self.total_num_heads // tp_world_size
100
-
101
- if self.total_num_kv_heads >= tp_world_size:
102
- # Number of KV heads is greater than TP size, so we partition
103
- # the KV heads across multiple tensor parallel GPUs.
104
- assert self.total_num_kv_heads % tp_world_size == 0
105
- else:
106
- # Number of KV heads is less than TP size, so we replicate
107
- # the KV heads across multiple tensor parallel GPUs.
108
- assert tp_world_size % self.total_num_kv_heads == 0
109
- self.num_kv_heads = max(1, self.total_num_kv_heads // tp_world_size)
110
- self.q_size = self.num_heads * self.head_dim
111
- self.kv_size = self.num_kv_heads * self.head_dim
112
- # Create the alibi slopes and slice them.
113
- tp_rank = get_tensor_model_parallel_rank()
114
- head_start = tp_rank * self.num_heads
115
- head_end = (tp_rank + 1) * self.num_heads
116
- alibi_slopes = _get_alibi_slopes(self.total_num_heads, self.alibi_bias_max)
117
- alibi_slopes = alibi_slopes[head_start:head_end].tolist()
118
-
119
- self.head_dim = self.d_model // self.total_num_heads
120
- scaling = self.head_dim**-0.5
121
- self.attn = Attention(
122
- self.num_heads,
123
- self.head_dim,
124
- scaling,
125
- alibi_slopes=alibi_slopes,
126
- num_kv_heads=self.num_kv_heads,
127
- cache_config=cache_config,
128
- quant_config=quant_config,
129
- prefix=f"{prefix}.attn",
130
- )
131
-
132
- def forward(
133
- self,
134
- position_ids: torch.Tensor,
135
- hidden_states: torch.Tensor,
136
- ) -> torch.Tensor:
137
- del position_ids # unused.
138
- qkv, _ = self.Wqkv(hidden_states)
139
- if self.clip_qkv is not None:
140
- qkv.clamp_(min=-self.clip_qkv, max=self.clip_qkv)
141
- q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
142
- if self.qk_ln:
143
- q = self.q_ln(q)
144
- k = self.k_ln(k)
145
- attn_output = self.attn(q, k, v)
146
- output, _ = self.out_proj(attn_output)
147
- return output
148
-
149
-
150
-class MPTMLP(nn.Module):
151
- def __init__(
152
- self,
153
- config: MptConfig,
154
- quant_config: QuantizationConfig | None = None,
155
- prefix: str = "",
156
- ):
157
- super().__init__()
158
- hidden_size = config.d_model
159
- expansion_ratio = config.expansion_ratio
160
- intermediate_size = expansion_ratio * hidden_size
161
- self.up_proj = ColumnParallelLinear(
162
- hidden_size,
163
- intermediate_size,
164
- bias=not config.no_bias,
165
- quant_config=quant_config,
166
- prefix=f"{prefix}.up_proj",
167
- )
168
- self.act = get_act_fn("gelu")
169
- self.down_proj = RowParallelLinear(
170
- intermediate_size,
171
- hidden_size,
172
- bias=not config.no_bias,
173
- quant_config=quant_config,
174
- prefix=f"{prefix}.down_proj",
175
- )
176
-
177
- def forward(self, x: torch.Tensor) -> torch.Tensor:
178
- x, _ = self.up_proj(x)
179
- x = self.act(x)
180
- x, _ = self.down_proj(x)
181
- return x
182
-
183
-
184
-class MPTBlock(nn.Module):
185
- def __init__(
186
- self,
187
- config: MptConfig,
188
- cache_config: CacheConfig | None = None,
189
- quant_config: QuantizationConfig | None = None,
190
- prefix: str = "",
191
- ):
192
- super().__init__()
193
- hidden_size = config.d_model
194
- self.norm_1 = nn.LayerNorm(hidden_size)
195
- self.attn = MPTAttention(
196
- config, cache_config, quant_config, prefix=f"{prefix}.attn"
197
- )
198
- self.norm_2 = nn.LayerNorm(hidden_size)
199
- self.ffn = MPTMLP(config, quant_config, prefix=f"{prefix}.ffn")
200
-
201
- def forward(
202
- self,
203
- position_ids: torch.Tensor,
204
- hidden_states: torch.Tensor,
205
- ) -> torch.Tensor:
206
- x = self.norm_1(hidden_states)
207
- x = self.attn(
208
- position_ids=position_ids,
209
- hidden_states=x,
210
- )
211
- hidden_states = hidden_states + x
212
- x = self.norm_2(hidden_states)
213
- x = self.ffn(x)
214
- hidden_states = hidden_states + x
215
- return hidden_states
216
-
217
-
218
-@support_torch_compile
219
-class MPTModel(nn.Module):
220
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
221
- super().__init__()
222
-
223
- config = vllm_config.model_config.hf_config
224
- cache_config = vllm_config.cache_config
225
- quant_config = vllm_config.quant_config
226
-
227
- assert config.embedding_fraction == 1.0
228
- assert config.norm_type == "low_precision_layernorm"
229
-
230
- self.wte = VocabParallelEmbedding(
231
- config.vocab_size,
232
- config.d_model,
233
- )
234
- self.start_layer, self.end_layer, self.blocks = make_layers(
235
- config.n_layers,
236
- lambda prefix: MPTBlock(config, cache_config, quant_config, prefix=prefix),
237
- prefix=f"{prefix}.blocks",
238
- )
239
- self.norm_f = nn.LayerNorm(config.d_model)
240
- if config.no_bias:
241
- for module in self.modules():
242
- if hasattr(module, "bias") and isinstance(module.bias, nn.Parameter):
243
- # Remove the bias term in Linear and LayerNorm.
244
- module.register_parameter("bias", None)
245
- self.make_empty_intermediate_tensors = make_empty_intermediate_tensors_factory(
246
- ["hidden_states"], config.d_model
247
- )
248
-
249
- def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
250
- return self.wte(input_ids)
251
-
252
- def forward(
253
- self,
254
- input_ids: torch.Tensor | None,
255
- position_ids: torch.Tensor,
256
- intermediate_tensors: IntermediateTensors | None,
257
- inputs_embeds: torch.Tensor | None = None,
258
- ) -> torch.Tensor | IntermediateTensors:
259
- if get_pp_group().is_first_rank:
260
- if inputs_embeds is not None:
261
- hidden_states = inputs_embeds
262
- else:
263
- hidden_states = self.embed_input_ids(input_ids)
264
- else:
265
- assert intermediate_tensors is not None
266
- hidden_states = intermediate_tensors["hidden_states"]
267
-
268
- for block in islice(self.blocks, self.start_layer, self.end_layer):
269
- hidden_states = block(position_ids, hidden_states)
270
- if not get_pp_group().is_last_rank:
271
- return IntermediateTensors({"hidden_states": hidden_states})
272
- hidden_states = self.norm_f(hidden_states)
273
- return hidden_states
274
-
275
-
276
-class MPTForCausalLM(nn.Module, SupportsPP):
277
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
278
- super().__init__()
279
- config = vllm_config.model_config.hf_config
280
- quant_config = vllm_config.quant_config
281
- self.config = config
282
- self.quant_config = quant_config
283
-
284
- self.transformer = MPTModel(
285
- vllm_config=vllm_config, prefix=maybe_prefix(prefix, "transformer")
286
- )
287
- self.lm_head = self.transformer.wte
288
- self.logits_processor = LogitsProcessor(config.vocab_size)
289
- self.make_empty_intermediate_tensors = (
290
- self.transformer.make_empty_intermediate_tensors
291
- )
292
-
293
- def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
294
- return self.transformer.embed_input_ids(input_ids)
295
-
296
- def forward(
297
- self,
298
- input_ids: torch.Tensor | None,
299
- positions: torch.Tensor,
300
- intermediate_tensors: IntermediateTensors | None = None,
301
- inputs_embeds: torch.Tensor | None = None,
302
- ) -> torch.Tensor | IntermediateTensors:
303
- hidden_states = self.transformer(
304
- input_ids, positions, intermediate_tensors, inputs_embeds
305
- )
306
- return hidden_states
307
-
308
- def compute_logits(
309
- self,
310
- hidden_states: torch.Tensor,
311
- ) -> torch.Tensor | None:
312
- logits = self.logits_processor(self.lm_head, hidden_states)
313
- return logits
314
-
315
- def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
316
- loader = AutoWeightsLoader(self)
317
- return loader.load_weights(weights)
vllm/model_executor/models/registry.py
+14
-21
@@ -74,7 +74,6 @@ _TEXT_GENERATION_MODELS = {
74
"AfmoeForCausalLM": ("afmoe", "AfmoeForCausalLM"),
75
"ApertusForCausalLM": ("apertus", "ApertusForCausalLM"),
76
"ArceeForCausalLM": ("arcee", "ArceeForCausalLM"),
77
- "ArcticForCausalLM": ("arctic", "ArcticForCausalLM"),
77
"AXK1ForCausalLM": ("AXK1", "AXK1ForCausalLM"),
78
"BailingMoeForCausalLM": ("bailing_moe", "BailingMoeForCausalLM"),
79
"BailingMoeV2ForCausalLM": ("bailing_moe", "BailingMoeV2ForCausalLM"),
@@ -102,7 +101,6 @@ _TEXT_GENERATION_MODELS = {
101
"ExaoneForCausalLM": ("exaone", "ExaoneForCausalLM"),
102
"Exaone4ForCausalLM": ("exaone4", "Exaone4ForCausalLM"),
103
"ExaoneMoeForCausalLM": ("exaone_moe", "ExaoneMoeForCausalLM"),
105
- "Fairseq2LlamaForCausalLM": ("fairseq2_llama", "Fairseq2LlamaForCausalLM"),
104
"FalconForCausalLM": ("falcon", "FalconForCausalLM"),
105
"FalconMambaForCausalLM": ("mamba", "MambaForCausalLM"),
106
"FalconH1ForCausalLM": ("falcon_h1", "FalconH1ForCausalLM"),
@@ -129,12 +127,10 @@ _TEXT_GENERATION_MODELS = {
127
"GraniteMoeSharedForCausalLM": ("granitemoeshared", "GraniteMoeSharedForCausalLM"),
128
"GraniteMoeSWAForCausalLM": ("granitemoeshared", "GraniteMoeSharedForCausalLM"),
129
"GraniteSWAForCausalLM": ("granite", "GraniteForCausalLM"),
132
- "GritLM": ("gritlm", "GritLM"),
130
"HrmTextForCausalLM": ("hrm_text", "HrmTextForCausalLM"),
131
"HunYuanMoEV1ForCausalLM": ("hunyuan_v1", "HunYuanMoEV1ForCausalLM"),
132
"HunYuanDenseV1ForCausalLM": ("hunyuan_v1", "HunYuanDenseV1ForCausalLM"),
133
"HYV3ForCausalLM": ("hy_v3", "HYV3ForCausalLM"),
137
- "HCXVisionForCausalLM": ("hyperclovax_vision", "HCXVisionForCausalLM"),
134
"HCXVisionV2ForCausalLM": ("hyperclovax_vision_v2", "HCXVisionV2ForCausalLM"),
135
"HyperCLOVAXForCausalLM": ("hyperclovax", "HyperCLOVAXForCausalLM"),
136
"InternLM2ForCausalLM": ("internlm2", "InternLM2ForCausalLM"),
@@ -178,9 +174,6 @@ _TEXT_GENERATION_MODELS = {
174
"MistralForCausalLM": ("mistral", "MistralForCausalLM"),
175
"MistralLarge3ForCausalLM": ("mistral_large_3", "MistralLarge3ForCausalLM"),
176
"MixtralForCausalLM": ("mixtral", "MixtralForCausalLM"),
181
- # transformers's mpt class has lower case
182
- "MptForCausalLM": ("mpt", "MPTForCausalLM"),
183
- "MPTForCausalLM": ("mpt", "MPTForCausalLM"),
177
"MiMoForCausalLM": ("mimo", "MiMoForCausalLM"),
178
"MiMoV2FlashForCausalLM": ("mimo_v2", "MiMoV2FlashForCausalLM"),
179
"MiMoV2ForCausalLM": ("mimo_v2", "MiMoV2ForCausalLM"),
@@ -207,14 +200,12 @@ _TEXT_GENERATION_MODELS = {
200
"Qwen3MoeForCausalLM": ("qwen3_moe", "Qwen3MoeForCausalLM"),
201
"Qwen3_5ForCausalLM": ("qwen3_5", "Qwen3_5ForCausalLM"),
202
"Qwen3_5MoeForCausalLM": ("qwen3_5", "Qwen3_5MoeForCausalLM"),
210
- "RWForCausalLM": ("falcon", "FalconForCausalLM"),
203
"SarvamMoEForCausalLM": ("sarvam", "SarvamMoEForCausalLM"),
204
"SarvamMLAForCausalLM": ("sarvam", "SarvamMLAForCausalLM"),
205
"SeedOssForCausalLM": ("seed_oss", "SeedOssForCausalLM"),
206
"Step1ForCausalLM": ("step1", "Step1ForCausalLM"),
207
"Step3TextForCausalLM": ("step3_text", "Step3TextForCausalLM"),
208
"Step3p5ForCausalLM": ("step3p5", "Step3p5ForCausalLM"),
217
- "StableLMEpochForCausalLM": ("stablelm", "StablelmForCausalLM"),
209
"StableLmForCausalLM": ("stablelm", "StablelmForCausalLM"),
210
"SolarForCausalLM": ("solar", "SolarForCausalLM"),
211
"TeleChat2ForCausalLM": ("telechat2", "TeleChat2ForCausalLM"),
@@ -234,7 +225,6 @@ _EMBEDDING_MODELS = {
225
"Gemma2Model": ("gemma2", "Gemma2ForCausalLM"),
226
"Gemma3TextModel": ("gemma3", "Gemma3Model"),
227
"GlmForCausalLM": ("glm", "GlmForCausalLM"),
237
- "GritLM": ("gritlm", "GritLM"),
228
"GteModel": ("bert_with_rope", "SnowflakeGteNewModel"),
229
"GteNewModel": ("bert_with_rope", "GteNewModel"),
230
"JinaEmbeddingsV5Model": ("jina", "JinaEmbeddingsV5Model"),
@@ -274,7 +264,6 @@ _EMBEDDING_MODELS = {
264
# Technically Terratorch models work on images, both in
265
# input and output. I am adding it here because it piggy-backs on embedding
266
# models for the time being.
277
- "PrithviGeoSpatialMAE": ("terratorch", "Terratorch"),
267
"Terratorch": ("terratorch", "Terratorch"),
268
}
269
@@ -360,12 +349,6 @@ _MULTIMODAL_MODELS = {
349
"BagelForConditionalGeneration": ("bagel", "BagelForConditionalGeneration"),
350
"BeeForConditionalGeneration": ("bee", "BeeForConditionalGeneration"),
351
"Blip2ForConditionalGeneration": ("blip2", "Blip2ForConditionalGeneration"),
363
- "ChameleonForConditionalGeneration": (
364
- "chameleon",
365
- "ChameleonForConditionalGeneration",
366
- ),
367
- "Cheers": ("cheers", "CheersForConditionalGeneration"),
368
- "CheersForConditionalGeneration": ("cheers", "CheersForConditionalGeneration"),
352
"Cohere2VisionForConditionalGeneration": (
353
"cohere2_vision",
354
"Cohere2VisionForConditionalGeneration",
@@ -401,10 +384,6 @@ _MULTIMODAL_MODELS = {
384
"FireRedASR2ForConditionalGeneration",
385
),
386
"FunASRForConditionalGeneration": ("funasr", "FunASRForConditionalGeneration"),
404
- "FireRedLIDForConditionalGeneration": (
405
- "fireredlid",
406
- "FireRedLIDForConditionalGeneration",
407
- ),
387
"FunAudioChatForConditionalGeneration": (
388
"funaudiochat",
389
"FunAudioChatForConditionalGeneration",
@@ -811,6 +790,20 @@ _PREVIOUSLY_SUPPORTED_MODELS = {
790
"FuyuForCausalLM": "0.25.0",
791
"Plamo2ForCausalLM": "0.26.0",
792
"OuroForCausalLM": "0.26.0",
793
+ "ArcticForCausalLM": "0.28.0",
794
+ "ChameleonForConditionalGeneration": "0.28.0",
795
+ "Cheers": "0.28.0",
796
+ "CheersForConditionalGeneration": "0.28.0",
797
+ "Fairseq2LlamaForCausalLM": "0.28.0",
798
+ "FireRedLIDForConditionalGeneration": "0.28.0",
799
+ "GritLM": "0.28.0",
800
+ "HCXVisionForCausalLM": "0.28.0",
801
+ "MPTForCausalLM": "0.28.0",
802
+ "MptForCausalLM": "0.28.0",
803
+ "RWForCausalLM": "0.28.0",
804
+ "StableLMEpochForCausalLM": "0.28.0",
805
+ # Superseded by the generic "Terratorch" architecture
806
+ "PrithviGeoSpatialMAE": "0.28.0",
807
}
808
809
_OOT_SUPPORTED_MODELS = {
vllm/transformers_utils/chat_templates/registry.py
-1
@@ -24,7 +24,6 @@ def _get_minicpmv_chat_template_fallback(tokenizer_name_or_path: str) -> Path |
24
25
_MODEL_TYPE_TO_CHAT_TEMPLATE_FALLBACK: dict[str, ChatTemplatePath] = {
26
"blip-2": CHAT_TEMPLATES_DIR / "template_blip2.jinja",
27
- "chameleon": CHAT_TEMPLATES_DIR / "template_basic.jinja",
27
"clip": CHAT_TEMPLATES_DIR / "template_basic.jinja",
28
"colpali": CHAT_TEMPLATES_DIR / "template_basic.jinja",
29
"deepseek_ocr": CHAT_TEMPLATES_DIR / "template_deepseek_ocr.jinja",
vllm/transformers_utils/config.py
-4
@@ -71,10 +71,8 @@ class LazyConfigDict(dict):
71
72
_CONFIG_REGISTRY: dict[str, type[PretrainedConfig]] = LazyConfigDict(
73
afmoe="AfmoeConfig",
74
- arctic="ArcticConfig",
74
axk1="AXK1Config",
75
bagel="BagelConfig",
77
- umm="CheersConfig",
76
chatglm="ChatGLMConfig",
77
modernvbert="ColModernVBertConfig",
78
colpali="ColPaliConfig",
@@ -90,11 +88,9 @@ _CONFIG_REGISTRY: dict[str, type[PretrainedConfig]] = LazyConfigDict(
88
dots3_note="Dots3NoteConfig",
89
k3_dspark="K3DSparkConfig",
90
flex_olmo="FlexOlmoConfig",
93
- fireredlid="FireRedLIDConfig",
91
funaudiochat="FunAudioChatConfig",
92
granite4_vision="Granite4VisionConfig",
93
hyperclovax="HyperCLOVAXConfig",
97
- hyperclovax_vlm="HCXVisionConfig",
94
hunyuan_vl="HunYuanVLConfig",
95
hy_v3="HYV3Config",
96
isaac="IsaacConfig",
vllm/transformers_utils/configs/__init__.py
-8
@@ -16,10 +16,8 @@ import importlib
16
17
_CLASS_TO_MODULE: dict[str, str] = {
18
"AfmoeConfig": "vllm.transformers_utils.configs.afmoe",
19
- "ArcticConfig": "vllm.transformers_utils.configs.arctic",
19
"AXK1Config": "vllm.transformers_utils.configs.AXK1",
20
"BagelConfig": "vllm.transformers_utils.configs.bagel",
22
- "CheersConfig": "vllm.transformers_utils.configs.cheers",
21
"ChatGLMConfig": "vllm.transformers_utils.configs.chatglm",
22
"ColModernVBertConfig": "vllm.transformers_utils.configs.colmodernvbert",
23
"ColPaliConfig": "vllm.transformers_utils.configs.colpali",
@@ -39,7 +37,6 @@ _CLASS_TO_MODULE: dict[str, str] = {
37
"K3DSparkConfig": "vllm.transformers_utils.configs.k3_dspark",
38
"DotsOCRConfig": "vllm.transformers_utils.configs.dotsocr",
39
"EAGLEConfig": "vllm.transformers_utils.configs.eagle",
42
- "FireRedLIDConfig": "vllm.transformers_utils.configs.fireredlid",
40
"FlexOlmoConfig": "vllm.transformers_utils.configs.flex_olmo",
41
"FunAudioChatConfig": "vllm.transformers_utils.configs.funaudiochat",
42
"FunAudioChatAudioEncoderConfig": "vllm.transformers_utils.configs.funaudiochat",
@@ -47,7 +44,6 @@ _CLASS_TO_MODULE: dict[str, str] = {
44
"HunYuanVLConfig": "vllm.transformers_utils.configs.hunyuan_vl",
45
"HunYuanVLTextConfig": "vllm.transformers_utils.configs.hunyuan_vl",
46
"HunYuanVLVisionConfig": "vllm.transformers_utils.configs.hunyuan_vl",
50
- "HCXVisionConfig": "vllm.transformers_utils.configs.hyperclovax",
47
"HYV3Config": "vllm.transformers_utils.configs.hy_v3",
48
"HyperCLOVAXConfig": "vllm.transformers_utils.configs.hyperclovax",
49
"IsaacConfig": "vllm.transformers_utils.configs.isaac",
@@ -112,10 +108,8 @@ _CLASS_TO_MODULE: dict[str, str] = {
108
109
__all__ = [
110
"AfmoeConfig",
115
- "ArcticConfig",
111
"AXK1Config",
112
"BagelConfig",
118
- "CheersConfig",
113
"ChatGLMConfig",
114
"ColModernVBertConfig",
115
"ColPaliConfig",
@@ -137,14 +131,12 @@ __all__ = [
131
"DotsOCRConfig",
132
"EAGLEConfig",
133
"FlexOlmoConfig",
140
- "FireRedLIDConfig",
134
"FunAudioChatConfig",
135
"FunAudioChatAudioEncoderConfig",
136
"Granite4VisionConfig",
137
"HunYuanVLConfig",
138
"HunYuanVLTextConfig",
139
"HunYuanVLVisionConfig",
147
- "HCXVisionConfig",
140
"HYV3Config",
141
"HyperCLOVAXConfig",
142
"IsaacConfig",
vllm/transformers_utils/configs/arctic.py
deleted
-216
@@ -1,216 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-# ruff: noqa: E501
5
-# coding=utf-8
6
-# Copied from
7
-# https://huggingface.co/Snowflake/snowflake-arctic-instruct/blob/main/configuration_arctic.py
8
-"""Arctic model configuration"""
9
-
10
-from dataclasses import asdict, dataclass
11
-from typing import Any
12
-
13
-from transformers.configuration_utils import PretrainedConfig
14
-from transformers.utils import logging
15
-
16
-logger = logging.get_logger(__name__)
17
-
18
-ARCTIC_PRETRAINED_CONFIG_ARCHIVE_MAP = {
19
- "arctic": "https://huggingface.co/Snowflake/snowflake-arctic-instruct/tree/main/config.json",
20
-}
21
-
22
-
23
-@dataclass
24
-class ArcticLoRAConfig:
25
- lora_r: int = 64
26
- lora_alpha: float = 16
27
- shard_base_weights: bool = False
28
-
29
-
30
-@dataclass
31
-class ArcticQuantizationConfig:
32
- q_bits: int = 8
33
- rounding: str = "nearest"
34
- mantissa_bits: int = 3
35
- group_size: int = 128
36
-
37
-
38
-class ArcticConfig(PretrainedConfig):
39
- r"""
40
- This is the configuration class to store the configuration of a [`ArcticModel`]. It is used to instantiate an
41
- Arctic model according to the specified arguments, defining the model architecture. Instantiating a configuration
42
- with the defaults will yield a similar configuration to that of the #TODO(rsamdani): add what model has the default config..
43
-
44
-
45
- Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
46
- documentation from [`PretrainedConfig`] for more information.
47
-
48
-
49
- Args:
50
- vocab_size (`int`, *optional*, defaults to 32000):
51
- Vocabulary size of the Arctic model. Defines the number of different tokens that can be represented by the
52
- `inputs_ids` passed when calling [`ArcticModel`]
53
- hidden_size (`int`, *optional*, defaults to 4096):
54
- Dimension of the hidden representations.
55
- intermediate_size (`int`, *optional*, defaults to 14336):
56
- Dimension of the MLP representations.
57
- num_hidden_layers (`int`, *optional*, defaults to 32):
58
- Number of hidden layers in the Transformer encoder.
59
- num_attention_heads (`int`, *optional*, defaults to 32):
60
- Number of attention heads for each attention layer in the Transformer encoder.
61
- num_key_value_heads (`int`, *optional*, defaults to 8):
62
- This is the number of key_value heads that should be used to implement Grouped Query Attention. If
63
- `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
64
- `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
65
- converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
66
- by meanpooling all the original heads within that group. For more details checkout [this
67
- paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `8`.
68
- hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
69
- The non-linear activation function (function or string) in the decoder.
70
- max_position_embeddings (`int`, *optional*, defaults to `4096*32`):
71
- The maximum sequence length that this model might ever be used with. Arctic's sliding window attention
72
- allows sequence of up to 4096*32 tokens.
73
- initializer_range (`float`, *optional*, defaults to 0.02):
74
- The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
75
- rms_norm_eps (`float`, *optional*, defaults to 1e-05):
76
- The epsilon used by the rms normalization layers.
77
- use_cache (`bool`, *optional*, defaults to `True`):
78
- Whether or not the model should return the last key/values attentions (not used by all models). Only
79
- relevant if `config.is_decoder=True`.
80
- pad_token_id (`int`, *optional*):
81
- The id of the padding token.
82
- bos_token_id (`int`, *optional*, defaults to 1):
83
- The id of the "beginning-of-sequence" token.
84
- eos_token_id (`int`, *optional*, defaults to 2):
85
- The id of the "end-of-sequence" token.
86
- tie_word_embeddings (`bool`, *optional*, defaults to `False`):
87
- Whether the model's input and output word embeddings should be tied.
88
- rope_parameters (`dict`, *optional*):
89
- Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
90
- and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
91
- accordingly.
92
- Expected contents:
93
- `rope_theta` (`float`): The base period of the RoPE embeddings.
94
- `rope_type` (`str`):
95
- The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
96
- 'llama3'], with 'default' being the original RoPE implementation.
97
- sliding_window (`int`, *optional*):
98
- Sliding window attention window size. If not specified, will default to `4096`.
99
- attention_dropout (`float`, *optional*, defaults to 0.0):
100
- The dropout ratio for the attention probabilities.
101
- num_experts_per_tok (`int`, *optional*, defaults to 2):
102
- The number of experts to root per-token, can be also interpreted as the `top-p` routing
103
- parameter
104
- num_local_experts (`int`, *optional*, defaults to 8):
105
- Number of experts per Sparse MLP layer.
106
- router_aux_loss_coef (`float`, *optional*, defaults to 0.001):
107
- The aux loss factor for the total loss.
108
-
109
- ```python
110
- >>> from transformers import ArcticModel, ArcticConfig
111
-
112
- >>> # Initializing a Arctic 7B style configuration TODO(rsamdani): verify which model does the default configuration correspond to.
113
- >>> configuration = ArcticConfig()
114
-
115
- >>> # Initializing a model from the Arctic 7B style configuration
116
- >>> model = ArcticModel(configuration)
117
-
118
- >>> # Accessing the model configuration
119
- >>> configuration = model.config
120
- ```"""
121
-
122
- model_type = "arctic"
123
- keys_to_ignore_at_inference = ["past_key_values"]
124
-
125
- def __init__(
126
- self,
127
- vocab_size=32000,
128
- hidden_size=4096,
129
- intermediate_size=14336,
130
- num_hidden_layers=32,
131
- num_attention_heads=32,
132
- num_key_value_heads=None,
133
- hidden_act="silu",
134
- max_position_embeddings=4096,
135
- initializer_range=0.02,
136
- rms_norm_eps=1e-5,
137
- use_cache=True,
138
- pad_token_id=None,
139
- bos_token_id=1,
140
- eos_token_id=2,
141
- tie_word_embeddings=False,
142
- rope_parameters: dict[str, Any] | None = None,
143
- sliding_window=None,
144
- attention_dropout=0.0,
145
- num_experts_per_tok=1,
146
- num_local_experts=8,
147
- router_aux_loss_coef=0.001,
148
- moe_layer_frequency=2,
149
- parallel_attn_mlp_res=False,
150
- moe_train_capacity_factor=1,
151
- moe_eval_capacity_factor=1,
152
- enable_expert_tensor_parallelism=False,
153
- moe_min_capacity=0,
154
- moe_token_dropping=True,
155
- quantization=None,
156
- **kwargs,
157
- ):
158
- self.vocab_size = vocab_size
159
- self.max_position_embeddings = max_position_embeddings
160
- self.hidden_size = hidden_size
161
- self.intermediate_size = intermediate_size
162
- self.num_hidden_layers = num_hidden_layers
163
- self.num_attention_heads = num_attention_heads
164
- self.sliding_window = sliding_window
165
-
166
- # for backward compatibility
167
- if num_key_value_heads is None:
168
- num_key_value_heads = num_attention_heads
169
-
170
- self.num_key_value_heads = num_key_value_heads
171
- self.hidden_act = hidden_act
172
- self.initializer_range = initializer_range
173
- self.rms_norm_eps = rms_norm_eps
174
- self.use_cache = use_cache
175
- rope_theta = kwargs.pop("rope_theta", 1e6)
176
- if rope_parameters is None:
177
- rope_parameters = {"rope_type": "default", "rope_theta": rope_theta}
178
- self.rope_parameters = rope_parameters
179
- self.attention_dropout = attention_dropout
180
-
181
- self.num_experts_per_tok = num_experts_per_tok
182
- self.num_local_experts = num_local_experts
183
- self.router_aux_loss_coef = router_aux_loss_coef
184
- self.moe_layer_frequency = moe_layer_frequency
185
- self.moe_train_capacity_factor = moe_train_capacity_factor
186
- self.moe_eval_capacity_factor = moe_eval_capacity_factor
187
- self.enable_expert_tensor_parallelism = enable_expert_tensor_parallelism
188
- self.moe_min_capacity = moe_min_capacity
189
- self.moe_token_dropping = moe_token_dropping
190
- self.parallel_attn_mlp_res = parallel_attn_mlp_res
191
- if isinstance(quantization, dict):
192
- self.quantization = ArcticQuantizationConfig(**quantization)
193
- else:
194
- self.quantization = quantization
195
-
196
- super().__init__(
197
- pad_token_id=pad_token_id,
198
- bos_token_id=bos_token_id,
199
- eos_token_id=eos_token_id,
200
- tie_word_embeddings=tie_word_embeddings,
201
- **kwargs,
202
- )
203
-
204
- @classmethod
205
- def from_dict(cls, config_dict: dict[str, Any], **kwargs) -> "ArcticConfig":
206
- result = super().from_dict(config_dict, **kwargs)
207
- config = result[0] if isinstance(result, tuple) else result
208
- if isinstance(config.quantization, dict):
209
- config.quantization = ArcticQuantizationConfig(**config.quantization)
210
- return result
211
-
212
- def to_dict(self) -> dict[str, Any]:
213
- ret = super().to_dict()
214
- if isinstance(ret["quantization"], ArcticQuantizationConfig):
215
- ret["quantization"] = asdict(ret["quantization"])
216
- return ret
vllm/transformers_utils/configs/cheers.py
deleted
-109
@@ -1,109 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-from transformers import PretrainedConfig, SiglipVisionConfig
4
-from transformers.modeling_rope_utils import rope_config_validation
5
-
6
-
7
-class CheersTextConfig(PretrainedConfig):
8
- """Qwen2-based text config with Cheers-specific defaults."""
9
-
10
- model_type = "umm"
11
- base_config_key = "text_config"
12
-
13
- def __init__(
14
- self,
15
- vocab_size=152064,
16
- hidden_size=3584,
17
- intermediate_size=18944,
18
- num_hidden_layers=28,
19
- num_attention_heads=28,
20
- num_key_value_heads=4,
21
- hidden_act="silu",
22
- max_position_embeddings=131072,
23
- initializer_range=0.02,
24
- rms_norm_eps=1e-6,
25
- use_cache=True,
26
- tie_word_embeddings=False,
27
- rope_theta=1000000.0,
28
- rope_scaling=None,
29
- use_sliding_window=False,
30
- sliding_window=131072,
31
- max_window_layers=28,
32
- layer_types=None,
33
- attention_dropout=0.0,
34
- **kwargs,
35
- ):
36
- self.vocab_size = vocab_size
37
- self.max_position_embeddings = max_position_embeddings
38
- self.hidden_size = hidden_size
39
- self.intermediate_size = intermediate_size
40
- self.num_hidden_layers = num_hidden_layers
41
- self.num_attention_heads = num_attention_heads
42
- self.use_sliding_window = use_sliding_window
43
- self.sliding_window = sliding_window if self.use_sliding_window else None
44
- self.max_window_layers = max_window_layers
45
- if num_key_value_heads is None:
46
- num_key_value_heads = num_attention_heads
47
- self.num_key_value_heads = num_key_value_heads
48
- self.hidden_act = hidden_act
49
- self.initializer_range = initializer_range
50
- self.rms_norm_eps = rms_norm_eps
51
- self.use_cache = use_cache
52
- self.rope_theta = rope_theta
53
- self.rope_scaling = rope_scaling
54
- self.attention_dropout = attention_dropout
55
- if self.rope_scaling is not None and "type" in self.rope_scaling:
56
- self.rope_scaling["rope_type"] = self.rope_scaling["type"]
57
- rope_config_validation(self)
58
-
59
- self.layer_types = layer_types
60
- if self.layer_types is None:
61
- self.layer_types = [
62
- "sliding_attention"
63
- if self.sliding_window is not None and i >= self.max_window_layers
64
- else "full_attention"
65
- for i in range(self.num_hidden_layers)
66
- ]
67
-
68
- super().__init__(
69
- tie_word_embeddings=tie_word_embeddings,
70
- **kwargs,
71
- )
72
-
73
-
74
-class CheersConfig(PretrainedConfig):
75
- """Configuration class for Cheers (UMM) model."""
76
-
77
- model_type = "umm"
78
-
79
- def __init__(
80
- self,
81
- text_config: dict | CheersTextConfig | None = None,
82
- vision_representation_config: dict | SiglipVisionConfig | None = None,
83
- vae_encoder_config: dict | None = None,
84
- vae_decoder_config: dict | None = None,
85
- **kwargs,
86
- ):
87
- super().__init__(**kwargs)
88
-
89
- if isinstance(text_config, dict):
90
- self.text_config = CheersTextConfig(**text_config)
91
- else:
92
- self.text_config = text_config or CheersTextConfig()
93
-
94
- if isinstance(vision_representation_config, dict):
95
- self.vision_representation_config = SiglipVisionConfig(
96
- **vision_representation_config
97
- )
98
- else:
99
- self.vision_representation_config = (
100
- vision_representation_config or SiglipVisionConfig()
101
- )
102
-
103
- self.vae_encoder_config = vae_encoder_config or {"resolution": 512}
104
- self.vae_decoder_config = vae_decoder_config or {"resolution": 512}
105
-
106
- @property
107
- def hidden_size(self) -> int:
108
- """Return the hidden size of the language model."""
109
- return self.text_config.hidden_size
vllm/transformers_utils/configs/fireredlid.py
deleted
-73
@@ -1,73 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-from __future__ import annotations
5
-
6
-import contextlib
7
-
8
-from transformers import AutoConfig
9
-from transformers.configuration_utils import PretrainedConfig
10
-
11
-
12
-class FireRedLIDConfig(PretrainedConfig):
13
- """Minimal config class for native vLLM FireRedLID support."""
14
-
15
- model_type = "fireredlid"
16
- keys_to_ignore_at_inference = ["past_key_values"]
17
-
18
- def __init__(
19
- self,
20
- vocab_size: int = 120,
21
- lid_odim: int = 120,
22
- idim: int = 80,
23
- d_model: int = 1280,
24
- n_head: int = 20,
25
- n_layers_enc: int = 16,
26
- n_layers_lid_dec: int = 6,
27
- kernel_size: int = 33,
28
- residual_dropout: float = 0.05,
29
- dropout_rate: float = 0.05,
30
- pe_maxlen: int = 5000,
31
- pad_token_id: int = 2,
32
- bos_token_id: int = 3,
33
- eos_token_id: int = 4,
34
- decoder_start_token_id: int = 3,
35
- tie_word_embeddings: bool = True,
36
- is_encoder_decoder: bool = True,
37
- architectures: list[str] | None = None,
38
- **kwargs,
39
- ):
40
- self.vocab_size = vocab_size
41
- self.lid_odim = lid_odim
42
- self.idim = idim
43
- self.d_model = d_model
44
- self.hidden_size = d_model
45
- self.n_head = n_head
46
- self.num_attention_heads = n_head
47
- self.n_layers_enc = n_layers_enc
48
- self.encoder_layers = n_layers_enc
49
- self.n_layers_lid_dec = n_layers_lid_dec
50
- self.decoder_layers = n_layers_lid_dec
51
- self.num_hidden_layers = n_layers_lid_dec
52
- self.kernel_size = kernel_size
53
- self.residual_dropout = residual_dropout
54
- self.dropout_rate = dropout_rate
55
- self.pe_maxlen = pe_maxlen
56
- self.tie_word_embeddings = tie_word_embeddings
57
- self.is_encoder_decoder = is_encoder_decoder
58
- self.architectures = architectures or ["FireRedLIDForConditionalGeneration"]
59
-
60
- super().__init__(
61
- pad_token_id=pad_token_id,
62
- bos_token_id=bos_token_id,
63
- eos_token_id=eos_token_id,
64
- decoder_start_token_id=decoder_start_token_id,
65
- tie_word_embeddings=tie_word_embeddings,
66
- is_encoder_decoder=is_encoder_decoder,
67
- architectures=self.architectures,
68
- **kwargs,
69
- )
70
-
71
-
72
-with contextlib.suppress(ValueError):
73
- AutoConfig.register(FireRedLIDConfig.model_type, FireRedLIDConfig)
vllm/transformers_utils/configs/hyperclovax.py
-72
@@ -17,7 +17,6 @@
17
# limitations under the License.
18
"""HyperCLOVA X model configuration."""
19
20
-from transformers import AutoConfig
20
from transformers.configuration_utils import PretrainedConfig
21
22
@@ -276,74 +275,3 @@ class HyperCLOVAXConfig(PretrainedConfig):
275
auto_map=auto_map,
276
**kwargs,
277
)
279
-
280
-
281
-class HCXVisionConfig(PretrainedConfig):
282
- """Vendored HyperCLOVAX Vision config with transformers v5 fix.
283
-
284
- The original remote code config does not handle empty initialization
285
- (text_config=None), which breaks transformers v5's @strict validation.
286
-
287
- TODO: Remove this class once HyperCLOVAX is upstreamed to transformers.
288
- Tracking PR: https://github.com/huggingface/transformers/pull/44956
289
- """
290
-
291
- model_type = "hyperclovax_vlm"
292
- keys_to_ignore_at_inference = ["past_key_values"]
293
-
294
- text_config_attribute_map = {
295
- "n_embd": "hidden_size",
296
- "n_positions": "max_position_embeddings",
297
- "n_head": "num_attention_heads",
298
- "n_layer": "num_hidden_layers",
299
- }
300
-
301
- def __init__(
302
- self,
303
- text_config=None,
304
- vision_config=None,
305
- use_nth_layer=-2,
306
- img_start_id=100009,
307
- decoder_max_length=4096,
308
- anyres=False,
309
- unpad=False,
310
- max_num_grids=-1,
311
- num_queries_vis_abstractor=-1,
312
- ignore_index=-100,
313
- proj_pos_emb=True,
314
- proj_prenorm=False,
315
- use_1x1_grid=False,
316
- **kwargs,
317
- ):
318
- for key, val in self.text_config_attribute_map.items():
319
- if text_config is not None and key in text_config:
320
- text_config[val] = text_config.pop(key)
321
-
322
- self.text_config = None
323
- if text_config is not None:
324
- _text_config = AutoConfig.for_model(text_config["model_type"])
325
- self.text_config = _text_config.from_dict(text_config)
326
- self.hidden_size = self.text_config.hidden_size
327
-
328
- self.vision_config = None
329
- if vision_config is not None:
330
- _vision_config = AutoConfig.for_model(vision_config["model_type"])
331
- self.vision_config = _vision_config.from_dict(vision_config)
332
-
333
- self.use_nth_layer = use_nth_layer
334
- self.decoder_max_length = decoder_max_length
335
- self.anyres = anyres
336
- self.unpad = unpad
337
- self.max_num_grids = max_num_grids
338
- self.num_queries_vis_abstractor = num_queries_vis_abstractor
339
- self.img_start_id = img_start_id
340
- self.ignore_index = ignore_index
341
- self.proj_pos_emb = proj_pos_emb
342
- self.proj_prenorm = proj_prenorm
343
- self.use_1x1_grid = use_1x1_grid
344
- super().__init__(**kwargs)
345
-
346
- def get_text_config(self, decoder=False):
347
- if self.text_config is not None:
348
- return self.text_config
349
- return self
vllm/transformers_utils/model_arch_config_convertor.py
-9
@@ -373,7 +373,6 @@ class ModelArchConfigConvertorBase:
373
"molmo2",
374
"moondream3",
375
"paligemma",
376
- "umm",
376
)
377
if not hasattr(self.hf_config, "model_type"):
378
return False
@@ -545,13 +544,6 @@ class FalconModelArchConfigConvertor(ModelArchConfigConvertorBase):
544
return super().get_total_num_kv_heads()
545
546
548
-class MPTModelArchConfigConvertor(ModelArchConfigConvertorBase):
549
- def get_total_num_kv_heads(self) -> int:
550
- if "kv_n_heads" in self.hf_text_config.attn_config:
551
- return self.hf_text_config.attn_config["kv_n_heads"]
552
- return self.hf_text_config.num_attention_heads
553
-
554
-
547
class DbrxModelArchConfigConvertor(ModelArchConfigConvertorBase):
548
def get_total_num_kv_heads(self) -> int:
549
return getattr(
@@ -792,7 +784,6 @@ MODEL_ARCH_CONFIG_CONVERTORS = {
784
"mimo_v2_mtp": MimoV2MTPModelArchConfigConvertor,
785
"mimo_v2_omni_mtp": MimoV2MTPModelArchConfigConvertor,
786
"moss_audio": MossAudioModelArchConfigConvertor,
795
- "mpt": MPTModelArchConfigConvertor,
787
"nemotron-nas": NemotronNasModelArchConfigConvertor,
788
"bailing_hybrid_v3_mtp": BailingHybridV3MTPModelArchConfigConvertor,
789
"pangu_ultra_moe_mtp": PanguUltraMoeMTPModelArchConfigConvertor,
vllm/transformers_utils/processor.py
+1
-3
@@ -37,9 +37,7 @@ def _transformers_v4_compatibility_import():
37
subset of `ProcessorChatTemplateKwargs` as defined in Transformers v4.
38
In Transformers v5 these were merged into `ProcessorChatTemplateKwargs` and
39
`ChatTemplateLoadKwargs` was removed. For backward compatibility, we add an alias
40
- for `ChatTemplateLoadKwargs` if it doesn't exist.
41
-
42
- This can be removed if `HCXVisionForCausalLM` is upstreamed to Transformers."""
40
+ for `ChatTemplateLoadKwargs` if it doesn't exist."""
41
old_import = getattr(processing_utils, "ChatTemplateLoadKwargs", None)
42
new_import = getattr(processing_utils, "ProcessorChatTemplateKwargs", None)
43
if old_import is None and new_import is not None:
vllm/transformers_utils/processors/__init__.py
-4
@@ -12,12 +12,10 @@ import importlib
12
13
__all__ = [
14
"BagelProcessor",
15
- "CheersProcessor",
15
"CohereASRProcessor",
16
"Cosmos3EdgeProcessor",
17
"DeepseekVLV2Processor",
18
"FireRedASR2Processor",
20
- "FireRedLIDProcessor",
19
"FunASRProcessor",
20
"GLM4VProcessor",
21
"Granite4VisionProcessor",
@@ -52,12 +50,10 @@ __all__ = [
50
51
_CLASS_TO_MODULE: dict[str, str] = {
52
"BagelProcessor": "vllm.transformers_utils.processors.bagel",
55
- "CheersProcessor": "vllm.transformers_utils.processors.cheers",
53
"CohereASRProcessor": "vllm.transformers_utils.processors.cohere_asr",
54
"Cosmos3EdgeProcessor": "vllm.transformers_utils.processors.cosmos3_edge",
55
"DeepseekVLV2Processor": "vllm.transformers_utils.processors.deepseek_vl2",
56
"FireRedASR2Processor": "vllm.transformers_utils.processors.fireredasr2",
60
- "FireRedLIDProcessor": "vllm.transformers_utils.processors.fireredlid",
57
"FunASRProcessor": "vllm.transformers_utils.processors.funasr",
58
"GLM4VProcessor": "vllm.transformers_utils.processors.glm4v",
59
"Granite4VisionProcessor": "vllm.transformers_utils.processors.granite4_vision",
vllm/transformers_utils/processors/cheers.py
deleted
-89
@@ -1,89 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-"""Cheers (UMM) processor for image and text inputs."""
4
-
5
-from transformers import AutoProcessor
6
-from transformers.feature_extraction_utils import BatchFeature
7
-from transformers.image_utils import ImageInput
8
-from transformers.processing_utils import ProcessingKwargs, ProcessorMixin, Unpack
9
-from transformers.tokenization_utils_base import PreTokenizedInput, TextInput
10
-
11
-
12
-class CheersProcessorKwargs(ProcessingKwargs, total=False): # type: ignore[call-arg]
13
- _defaults = {
14
- "images_kwargs": {
15
- "return_tensors": "pt",
16
- },
17
- }
18
-
19
-
20
-class CheersProcessor(ProcessorMixin):
21
- """
22
- Constructs a Cheers processor which wraps a
23
- SigLIP image processor and a Qwen2 tokenizer.
24
- """
25
-
26
- attributes = ["image_processor", "tokenizer"]
27
- image_processor_class = "AutoImageProcessor"
28
- tokenizer_class = "AutoTokenizer"
29
-
30
- def __call__(
31
- self,
32
- text: TextInput
33
- | PreTokenizedInput
34
- | list[TextInput]
35
- | list[PreTokenizedInput] = None,
36
- images: ImageInput = None,
37
- **kwargs: Unpack[CheersProcessorKwargs],
38
- ):
39
- output_kwargs = self._merge_kwargs(
40
- CheersProcessorKwargs,
41
- tokenizer_init_kwargs=self.tokenizer.init_kwargs,
42
- **kwargs,
43
- )
44
-
45
- if images is not None:
46
- import torch
47
-
48
- if isinstance(images, (list, tuple)):
49
- all_pv = []
50
- all_ghw = []
51
- for img in images:
52
- result = self.image_processor(img, **output_kwargs["images_kwargs"])
53
- all_pv.append(result["pixel_values"])
54
- if "grid_hws" in result:
55
- all_ghw.append(result["grid_hws"])
56
- pixel_values = {
57
- "pixel_values": torch.cat(all_pv, dim=0),
58
- }
59
- if all_ghw:
60
- pixel_values["grid_hws"] = torch.cat(all_ghw, dim=0)
61
- else:
62
- pixel_values = self.image_processor(
63
- images, **output_kwargs["images_kwargs"]
64
- )
65
- else:
66
- pixel_values = {}
67
-
68
- text_inputs = (
69
- self.tokenizer(text, **output_kwargs["text_kwargs"])
70
- if text is not None
71
- else {}
72
- )
73
-
74
- return BatchFeature(data={**pixel_values, **text_inputs})
75
-
76
- def batch_decode(self, *args, **kwargs):
77
- return self.tokenizer.batch_decode(*args, **kwargs)
78
-
79
- def decode(self, *args, **kwargs):
80
- return self.tokenizer.decode(*args, **kwargs)
81
-
82
- @property
83
- def model_input_names(self):
84
- tokenizer_input_names = self.tokenizer.model_input_names
85
- image_processor_input_names = self.image_processor.model_input_names
86
- return list(dict.fromkeys(tokenizer_input_names + image_processor_input_names))
87
-
88
-
89
-AutoProcessor.register("CheersProcessor", CheersProcessor)
vllm/transformers_utils/processors/fireredlid.py
deleted
-276
@@ -1,276 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-"""
4
-FireRedLID feature extractor and processor.
5
-
6
-The FeatureExtractor handles:
7
- - Raw waveform → 80-dim log-mel filterbank (via kaldi_native_fbank)
8
- - CMVN normalization (means / inverse_std_variences from preprocessor_config)
9
- - Padding + length tracking
10
-
11
-The Processor wraps the FeatureExtractor and a tokenizer.
12
-"""
13
-
14
-from __future__ import annotations
15
-
16
-from typing import TYPE_CHECKING
17
-
18
-import numpy as np
19
-import torch
20
-import torch.nn.functional as F
21
-from transformers import (
22
- AutoFeatureExtractor,
23
- BatchFeature,
24
-)
25
-from transformers.feature_extraction_sequence_utils import SequenceFeatureExtractor
26
-from transformers.processing_utils import ProcessorMixin
27
-from transformers.utils import TensorType
28
-
29
-from vllm.logger import init_logger
30
-from vllm.utils.import_utils import LazyLoader
31
-
32
-if TYPE_CHECKING:
33
- import kaldi_native_fbank as knf
34
-else:
35
- knf = LazyLoader("knf", globals(), "kaldi_native_fbank")
36
-
37
-
38
-logger = init_logger(__name__)
39
-
40
-
41
-# ---------------------------------------------------------------------------
42
-# Helpers (shared with FireRedASR2 processor)
43
-# ---------------------------------------------------------------------------
44
-
45
-
46
-class CMVN:
47
- def __init__(self, dim, means, inverse_std_variences):
48
- self.dim = dim
49
- self.means = np.array(means)
50
- self.inverse_std_variences = np.array(inverse_std_variences)
51
-
52
- def __call__(self, x):
53
- assert x.shape[-1] == self.dim, "CMVN dim mismatch"
54
- out = x - self.means
55
- out = out * self.inverse_std_variences
56
- return out
57
-
58
-
59
-class KaldifeatFbank:
60
- def __init__(
61
- self,
62
- num_mel_bins: int = 80,
63
- frame_length: int = 25,
64
- frame_shift: int = 10,
65
- dither: float = 0.0,
66
- ):
67
- self.dither = dither
68
- opts = knf.FbankOptions()
69
- opts.frame_opts.dither = dither
70
- opts.mel_opts.num_bins = num_mel_bins
71
- opts.frame_opts.snip_edges = True
72
- opts.mel_opts.debug_mel = False
73
- self.opts = opts
74
-
75
- def __call__(self, sample_rate, wav_np, is_train=False):
76
- dither = self.dither if is_train else 0.0
77
- self.opts.frame_opts.dither = dither
78
- fbank = knf.OnlineFbank(self.opts)
79
- fbank.accept_waveform(sample_rate, wav_np.tolist())
80
- feat = []
81
- for i in range(fbank.num_frames_ready):
82
- feat.append(fbank.get_frame(i))
83
- if len(feat) == 0:
84
- return np.zeros((0, self.opts.mel_opts.num_bins))
85
- return np.vstack(feat)
86
-
87
-
88
-# ---------------------------------------------------------------------------
89
-# Feature Extractor
90
-# ---------------------------------------------------------------------------
91
-
92
-
93
-class FireRedLIDFeatureExtractor(SequenceFeatureExtractor):
94
- """
95
- Extracts 80-dim log-mel filterbank features from raw waveforms,
96
- applies CMVN, and returns padded feature tensors with lengths.
97
-
98
- Also computes ``fake_token_lengths`` — the actual encoder output
99
- length for each audio — so that vLLM can allocate the correct
100
- number of cross-attention KV cache slots.
101
- """
102
-
103
- model_input_names = ["input_features"]
104
-
105
- def __init__(
106
- self,
107
- feature_size=80,
108
- sampling_rate=16000,
109
- chunk_length=30,
110
- padding_value=0.0,
111
- return_attention_mask=False,
112
- dim=80,
113
- means=None,
114
- inverse_std_variences=None,
115
- num_mel_bins=80,
116
- frame_length=25,
117
- frame_shift=10,
118
- dither=0.0,
119
- left_context=3,
120
- right_context=3,
121
- **kwargs,
122
- ):
123
- super().__init__(
124
- feature_size=feature_size,
125
- sampling_rate=sampling_rate,
126
- padding_value=padding_value,
127
- return_attention_mask=return_attention_mask,
128
- **kwargs,
129
- )
130
- self.chunk_length = chunk_length
131
- self.dim = dim
132
- self.means = means
133
- self.inverse_std_variences = inverse_std_variences
134
- self.num_mel_bins = num_mel_bins
135
- self.frame_length = frame_length
136
- self.frame_shift = frame_shift
137
- self.dither = dither
138
- self.sampling_rate = sampling_rate
139
- self.context = left_context + 1 + right_context
140
-
141
- def __call__(
142
- self,
143
- raw_speech: np.ndarray | list[float] | list[np.ndarray] | list[list[float]],
144
- truncation: bool = True,
145
- pad_to_multiple_of: int | None = None,
146
- return_tensors: str | TensorType | None = None,
147
- return_attention_mask: bool | None = None,
148
- padding: str | None = "max_length",
149
- max_length: int | None = None,
150
- sampling_rate: int | None = None,
151
- do_normalize: bool | None = None,
152
- **kwargs,
153
- ) -> BatchFeature:
154
- if sampling_rate is not None and sampling_rate != self.sampling_rate:
155
- raise ValueError(
156
- f"FireRedLIDFeatureExtractor expects sampling_rate="
157
- f"{self.sampling_rate}, got {sampling_rate}."
158
- )
159
-
160
- # Initialize helpers
161
- cmvn = CMVN(self.dim, self.means, self.inverse_std_variences)
162
- fbank = KaldifeatFbank(
163
- num_mel_bins=self.num_mel_bins,
164
- frame_length=self.frame_length,
165
- frame_shift=self.frame_shift,
166
- dither=self.dither,
167
- )
168
-
169
- def padding_position_is_0(padded_input, input_lengths):
170
- N, T = padded_input.size()[:2]
171
- mask = torch.ones((N, T)).to(padded_input.device)
172
- for i in range(N):
173
- mask[i, input_lengths[i] :] = 0
174
- mask = mask.unsqueeze(dim=1)
175
- return mask.to(torch.uint8)
176
-
177
- feats = []
178
- speech_lengths = []
179
- fake_token_lengths = []
180
-
181
- for speech in raw_speech:
182
- # vLLM loads audio via librosa (float32 in [-1,1]),
183
- # but kaldi_native_fbank expects int16-scale values.
184
- speech_scaled = speech * 32768
185
- feat = fbank(self.sampling_rate, speech_scaled)
186
- feat = cmvn(feat)
187
- feat = torch.from_numpy(feat).float()
188
- length = feat.size(0)
189
- feats.append(feat)
190
- speech_lengths.append(length)
191
-
192
- # Compute the actual Conv2dSubsampling output length.
193
- # This mirrors the mask logic in Conv2dSubsampling.forward:
194
- # pad context frames, then mask[:, :, :-2:2][:, :, :-2:2].sum()
195
- padded_input = F.pad(feat, (0, 0, 0, self.context - 1), "constant", 0.0)
196
- src_mask = padding_position_is_0(
197
- padded_input[None, :, :],
198
- torch.tensor([length], dtype=torch.int32),
199
- )
200
- mask = src_mask[:, :, :-2:2][:, :, :-2:2]
201
- enc_len = mask[:, -1, :].sum(dim=-1)
202
- fake_token_len = torch.clamp(enc_len, min=1)
203
- fake_token_lengths.append(fake_token_len)
204
-
205
- if len(feats) == 0:
206
- return BatchFeature()
207
-
208
- # Pad to uniform length
209
- max_feat_len = max(f.size(0) for f in feats)
210
- padded = feats[0].new_zeros(len(feats), max_feat_len, feats[0].size(1))
211
- for i, feat in enumerate(feats):
212
- padded[i, : feat.size(0)] = feat
213
-
214
- result = BatchFeature({"input_features": padded})
215
-
216
- if return_tensors is not None:
217
- result = result.convert_to_tensors(return_tensors)
218
-
219
- result["speech_lengths"] = torch.tensor(speech_lengths, dtype=torch.long)
220
- result["fake_token_lengths"] = torch.concat(fake_token_lengths)
221
- return result
222
-
223
-
224
-# ---------------------------------------------------------------------------
225
-# Processor
226
-# ---------------------------------------------------------------------------
227
-
228
-
229
-class FireRedLIDProcessor(ProcessorMixin):
230
- """
231
- Wraps FireRedLIDFeatureExtractor + a tokenizer.
232
- """
233
-
234
- feature_extractor_class = "FireRedLIDFeatureExtractor"
235
- tokenizer_class = ("PythonBackend", "TokenizersBackend")
236
-
237
- def __init__(self, feature_extractor, tokenizer):
238
- super().__init__(feature_extractor, tokenizer)
239
- self.current_processor = self.feature_extractor
240
- self._in_target_context_manager = False
241
-
242
- def __call__(self, *args, **kwargs):
243
- if self._in_target_context_manager:
244
- return self.current_processor(*args, **kwargs)
245
-
246
- audio = kwargs.pop("audio", None)
247
- sampling_rate = kwargs.pop("sampling_rate", None)
248
- text = kwargs.pop("text", None)
249
- if len(args) > 0:
250
- audio = args[0]
251
- args = args[1:]
252
-
253
- if audio is not None:
254
- inputs = self.feature_extractor(
255
- audio, *args, sampling_rate=sampling_rate, **kwargs
256
- )
257
- else:
258
- inputs = BatchFeature()
259
-
260
- if text is not None:
261
- if isinstance(text, str):
262
- text = [text]
263
- encodings = self.tokenizer(text, **kwargs)
264
- if audio is not None:
265
- inputs["labels"] = encodings["input_ids"]
266
- else:
267
- return encodings
268
-
269
- return inputs
270
-
271
-
272
-# ---------------------------------------------------------------------------
273
-# Registration
274
-# ---------------------------------------------------------------------------
275
-
276
-AutoFeatureExtractor.register("FireRedLIDFeatureExtractor", FireRedLIDFeatureExtractor)
vllm/v1/worker/gpu/model_states/encoder_decoder.py
+1
-1
@@ -43,7 +43,7 @@ class EncoderDecoderAttnMetadata(ModelSpecificAttnMetadata):
43
44
class EncoderDecoderModelState(ModelState):
45
"""ModelState for cross-attention encoder-decoder models
46
- (Whisper, CohereASR, NemotronParse, FireRedLID, ...)
46
+ (Whisper, CohereASR, NemotronParse, ...)
47
"""
48
49
def __init__(