@tduncan / Nvidia-Optimization-Stack-VLLM / commits / 412414d8e0

Remove PersimmonForCausalLM and FuyuForCausalLM model architectures (#48096)

Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Cyrus Leung <tlleungac@connect.ust.hk> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

Tiezhen WANG committed Jul 9, 2026 at 19:59 UTC 412414d8e081e7057768714d37670728dc8cb363

14 files changed +118 -1093

docs/configuration/conserving_memory.md

+1 -1

@@ -42,7 +42,7 @@ and the maximum batch size (`max_num_seqs` option).

42 ```python

43 from vllm import LLM

45 -llm = LLM(model="adept/fuyu-8b", max_model_len=2048, max_num_seqs=2)

45 +llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct", max_model_len=2048, max_num_seqs=2)

46 ```

48 ## Reduce CUDA Graphs

docs/contributing/model/multimodal.md

+115 -296

@@ -324,154 +324,44 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

324 return image_token * num_images

325 ```

326

327 -=== "No input placeholders: Fuyu"

327 +=== "No input placeholders: PaliGemma"

328

329 - Looking at the code of HF's `FuyuForCausalLM`:

329 + Unlike LLaVA, PaliGemma's HF processor does not expect image placeholder

330 + tokens in the input prompt; the placeholder feature tokens are instead

331 + inserted afterwards (see [Prompt updates](#prompt-updates)). So the dummy

332 + prompt text is empty regardless of the number of images:

333

331 - ??? code

332 -

333 - ```python

334 - # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/modeling_fuyu.py#L311-L322

335 - if image_patches is not None and past_key_values is None:

336 - patch_embeddings = [

337 - self.vision_embed_tokens(patch.to(self.vision_embed_tokens.weight.dtype))

338 - .squeeze(0)

339 - .to(inputs_embeds.device)

340 - for patch in image_patches

341 - ]

342 - inputs_embeds = self.gather_continuous_embeddings(

343 - word_embeddings=inputs_embeds,

344 - continuous_embeddings=patch_embeddings,

345 - image_patch_input_indices=image_patches_indices,

346 - )

347 - ```

348 -

349 - The number of placeholder feature tokens for the `i`th item in the batch is `patch_embeddings[i].shape[0]`,

350 - which is the same as `image_patches[i].shape[0]`, i.e. `num_total_patches`.

351 -

352 - Unlike LLaVA, Fuyu does not define the number of patches inside the modeling file. Where can we get more information?

353 - Considering that the model input comes from the output of `FuyuProcessor`, let's **look at the preprocessing files**.

354 -

355 - The image outputs are obtained by calling `FuyuImageProcessor.preprocess` and then

356 - `FuyuImageProcessor.preprocess_with_tokenizer_info` inside `FuyuProcessor`.

357 -

358 - In `FuyuImageProcessor.preprocess`, the images are resized and padded to the target `FuyuImageProcessor.size`,

359 - returning the dimensions after resizing (but before padding) as metadata.

360 -

361 - ??? code

362 -

363 - ```python

364 - # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L541-L544

365 - image_encoding = self.image_processor.preprocess(images, **output_kwargs["images_kwargs"])

366 - batch_images = image_encoding["images"]

367 - image_unpadded_heights = image_encoding["image_unpadded_heights"]

368 - image_unpadded_widths = image_encoding["image_unpadded_widths"]

369 -

370 - # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L480-L

371 - if do_resize:

372 - batch_images = [

373 - [self.resize(image, size=size, input_data_format=input_data_format) for image in images]

374 - for images in batch_images

375 - ]

376 -

377 - image_sizes = [get_image_size(images[0], channel_dim=input_data_format) for images in batch_images]

378 - image_unpadded_heights = [[image_size[0]] for image_size in image_sizes]

379 - image_unpadded_widths = [[image_size[1]] for image_size in image_sizes]

380 -

381 - if do_pad:

382 - batch_images = [

383 - [

384 - self.pad_image(

385 - image,

386 - size=size,

387 - mode=padding_mode,

388 - constant_values=padding_value,

389 - input_data_format=input_data_format,

390 - )

391 - for image in images

392 - ]

393 - for images in batch_images

394 - ]

395 - ```

396 -

397 - In `FuyuImageProcessor.preprocess_with_tokenizer_info`, the images are split into patches based on this metadata:

398 -

399 - ??? code

400 -

401 - ```python

402 - # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L425

403 - model_image_input = self.image_processor.preprocess_with_tokenizer_info(

404 - image_input=tensor_batch_images,

405 - image_present=image_present,

406 - image_unpadded_h=image_unpadded_heights,

407 - image_unpadded_w=image_unpadded_widths,

408 - image_placeholder_id=image_placeholder_id,

409 - image_newline_id=image_newline_id,

410 - variable_sized=True,

411 - )

412 -

413 - # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L638-L658

414 - image_height, image_width = image.shape[1], image.shape[2]

415 - if variable_sized: # variable_sized=True

416 - new_h = min(

417 - image_height,

418 - math.ceil(image_unpadded_h[batch_index, subseq_index] / patch_height) * patch_height,

419 - )

420 - new_w = min(

421 - image_width,

422 - math.ceil(image_unpadded_w[batch_index, subseq_index] / patch_width) * patch_width,

423 - )

424 - image = image[:, :new_h, :new_w]

425 - image_height, image_width = new_h, new_w

426 -

427 - num_patches = self.get_num_patches(image_height=image_height, image_width=image_width)

428 - tensor_of_image_ids = torch.full(

429 - [num_patches], image_placeholder_id, dtype=torch.int32, device=image_input.device

430 - )

431 - patches = self.patchify_image(image=image.unsqueeze(0)).squeeze(0)

432 - assert num_patches == patches.shape[0]

433 - ```

334 + ```python

335 + def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:

336 + return ""

337 + ```

338

435 - The number of patches is in turn defined by `FuyuImageProcessor.get_num_patches`:

339 + PaliGemma resizes every image to a square of `vision_config.image_size`, so

340 + the number of placeholder feature tokens per image is fixed at

341 + `(image_size // patch_size) ** 2`. This is computed by the SigLIP vision

342 + encoder that PaliGemma uses:

343

344 ??? code

345

346 ```python

440 - # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L552-L562

441 - patch_size = patch_size if patch_size is not None else self.patch_size

442 - patch_height, patch_width = self.patch_size["height"], self.patch_size["width"]

443 -

444 - if image_height % patch_height != 0:

445 - raise ValueError(f"{image_height=} must be divisible by {patch_height}")

446 - if image_width % patch_width != 0:

447 - raise ValueError(f"{image_width=} must be divisible by {patch_width}")

448 -

449 - num_patches_per_dim_h = image_height // patch_height

450 - num_patches_per_dim_w = image_width // patch_width

451 - num_patches = num_patches_per_dim_h * num_patches_per_dim_w

347 + # vllm/model_executor/models/siglip.py

348 + class SiglipEncoderInfo(VisionEncoderInfo[SiglipVisionConfig]):

349 + def get_num_image_tokens(

350 + self,

351 + *,

352 + image_width: int,

353 + image_height: int,

354 + ) -> int:

355 + return self.get_patch_grid_length() ** 2

356 +

357 + def get_patch_grid_length(self) -> int:

358 + image_size, patch_size = self.get_image_size(), self.get_patch_size()

359 + return image_size // patch_size

360 ```

361

454 - These image patches correspond to placeholder tokens (`|SPEAKER|`). So, we just need to maximize the number of image patches. Since input images are first resized

455 - to fit within `image_processor.size`, we can maximize the number of image patches by inputting an image with size equal to `image_processor.size`.

456 -

457 - ```python

458 - def get_image_size_with_most_features(self) -> ImageSize:

459 - image_processor = self.get_image_processor()

460 - return ImageSize(

461 - width=image_processor.size["width"],

462 - height=image_processor.size["height"],

463 - )

464 - ```

465 -

466 - Fuyu does not expect image placeholders in the inputs to HF processor, so

467 - the dummy prompt text is empty regardless of the number of images.

468 -

469 - ```python

470 - def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:

471 - return ""

472 - ```

473 -

474 - For the multimodal image profiling data, the logic is very similar to LLaVA:

362 + Since the number of image tokens doesn't depend on the input image dimensions,

363 + we can simply use a dummy image of the model's expected input size for the

364 + multimodal profiling data:

365

366 ??? code

367

@@ -482,16 +372,18 @@ Assuming that the memory usage increases with the number of tokens, the dummy in

372 mm_counts: Mapping[str, int],

373 mm_options: Mapping[str, BaseDummyOptions],

374 ) -> MultiModalDataDict:

485 - target_width, target_height = \

486 - self.info.get_image_size_with_most_features()

375 + hf_config = self.info.get_hf_config()

376 + vision_config = hf_config.vision_config

377 + max_image_size = vision_config.image_size

378 +

379 num_images = mm_counts.get("image", 0)

380

381 image_overrides = mm_options.get("image")

382

383 return {

384 "image": self._get_dummy_images(

493 - width=target_width,

494 - height=target_height,

385 + width=max_image_size,

386 + height=max_image_size,

387 num_images=num_images,

388 overrides=image_overrides,

389 )

@@ -545,28 +437,15 @@ return a schema of the tensors outputted by the HF processor that are related to

437 Our [actual code](../../../vllm/model_executor/models/llava.py) additionally supports

438 pre-computed image embeddings, which can be passed to be model via the `image_embeds` argument.

439

548 -=== "With postprocessing: Fuyu"

440 +=== "With postprocessing: Mistral3"

441

550 - The `image_patches` output of `FuyuImageProcessor.preprocess_with_tokenizer_info` concatenates

551 - the patches from each image belonging to an item in the batch:

442 + The `pixel_values` output of Mistral3's HF processor pads every image in the

443 + batch to a common size, so that they can be stacked into a single tensor.

444

553 - ```python

554 - # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L673-L679

555 - image_input_ids.append(tensor_of_image_ids)

556 - image_patches.append(patches)

557 - else:

558 - image_input_ids.append(torch.tensor([], dtype=torch.int32, device=image_input.device))

559 -

560 - batch_image_input_ids.append(image_input_ids)

561 - batch_image_patches.append(image_patches)

562 - ```

563 -

564 - The shape of `image_patches` outputted by `FuyuImageProcessor` is therefore

565 - `(1, num_images, num_patches, patch_width * patch_height * num_channels)`.

566 -

567 - In order to support the use of

568 - [MultiModalFieldConfig.batched][vllm.multimodal.inputs.MultiModalFieldConfig.batched]

569 - like in LLaVA, we remove the extra batch dimension by overriding

445 + To use [MultiModalFieldConfig.batched][vllm.multimodal.inputs.MultiModalFieldConfig.batched]

446 + like in LLaVA, each image's features must be independent of the others (which

447 + is also required for prefix caching to work correctly). So, we un-pad each image

448 + back to its own size by overriding

449 [BaseMultiModalProcessor._call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor]:

450

451 ??? code

@@ -586,33 +465,27 @@ return a schema of the tensors outputted by the HF processor that are related to

465 tok_kwargs=tok_kwargs,

466 )

467

589 - image_patches = processed_outputs.get("image_patches")

590 - if image_patches is not None:

591 - images = mm_data["images"]

592 - assert isinstance(images, list)

468 + pixel_values = processed_outputs.get("pixel_values")

469 + if pixel_values is not None:

470 + # Avoid padding since we need the output for each image to be

471 + # independent of other images for the cache to work correctly

472 + image_sizes = processed_outputs["image_sizes"]

473 + assert len(pixel_values) == len(image_sizes)

474

594 - # Original output: (1, num_images, Pn, Px * Py * C)

595 - # New output: (num_images, Pn, Px * Py * C)

596 - assert (isinstance(image_patches, list)

597 - and len(image_patches) == 1)

598 - assert (isinstance(image_patches[0], torch.Tensor)

599 - and len(image_patches[0]) == len(images))

600 -

601 - processed_outputs["image_patches"] = image_patches[0]

475 + processed_outputs["pixel_values"] = [

476 + p[:, :h, :w] for p, (h, w) in zip(pixel_values, image_sizes)

477 + ]

478

479 return processed_outputs

480 ```

481

606 - !!! note

607 - Our [actual code](../../../vllm/model_executor/models/fuyu.py) has special handling

608 - for text-only inputs to prevent unnecessary warnings from HF processor.

609 -

482 !!! note

483 The `_call_hf_processor` method specifies both `mm_kwargs` and `tok_kwargs` for

484 processing. `mm_kwargs` is used to both initialize and call the huggingface

485 processor, whereas `tok_kwargs` is only used to call the huggingface processor.

486

615 - This lets us override [_get_mm_fields_config][vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config] as follows:

487 + Since `pixel_values` is now a list with one tensor per image, we can override

488 + [_get_mm_fields_config][vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config] as follows:

489

490 ```python

491 def _get_mm_fields_config(

@@ -620,9 +493,15 @@ return a schema of the tensors outputted by the HF processor that are related to

493 hf_inputs: BatchFeature,

494 hf_processor_mm_kwargs: Mapping[str, object],

495 ) -> Mapping[str, MultiModalFieldConfig]:

623 - return dict(image_patches=MultiModalFieldConfig.batched("image"))

496 + return dict(

497 + pixel_values=MultiModalFieldConfig.batched("image"),

498 + image_embeds=MultiModalFieldConfig.batched("image"),

499 + )

500 ```

501

502 + !!! note

503 + See our [actual code](../../../vllm/model_executor/models/mistral3.py) for the full implementation.

504 +

505 ### Prompt updates

506

507 Override [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates] to

@@ -678,121 +557,53 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies

557 ]

558 ```

559

681 -=== "Handling additional tokens: Fuyu"

682 -

683 - Recall the layout of feature tokens from Step 2:

684 -

685 - ```

686 - |SPEAKER||SPEAKER|...|SPEAKER||NEWLINE|

687 - |SPEAKER||SPEAKER|...|SPEAKER||NEWLINE|

688 - ...

689 - |SPEAKER||SPEAKER|...|SPEAKER||NEWLINE|

690 - ```

691 -

692 - We define a helper function to return `ncols` and `nrows` directly:

693 -

694 - ??? code

695 -

696 - ```python

697 - def get_image_feature_grid_size(

698 - self,

699 - *,

700 - image_width: int,

701 - image_height: int,

702 - ) -> tuple[int, int]:

703 - image_processor = self.get_image_processor()

704 - target_width = image_processor.size["width"]

705 - target_height = image_processor.size["height"]

706 - patch_width = image_processor.patch_size["width"]

707 - patch_height = image_processor.patch_size["height"]

708 -

709 - if not (image_width <= target_width and image_height <= target_height):

710 - height_scale_factor = target_height / image_height

711 - width_scale_factor = target_width / image_width

712 - optimal_scale_factor = min(height_scale_factor, width_scale_factor)

713 -

714 - image_height = int(image_height * optimal_scale_factor)

715 - image_width = int(image_width * optimal_scale_factor)

716 -

717 - ncols = math.ceil(image_width / patch_width)

718 - nrows = math.ceil(image_height / patch_height)

719 - return ncols, nrows

720 - ```

560 +=== "Handling additional tokens: PaliGemma"

561

722 - Based on this, we can initially define our replacement tokens as:

562 + PaliGemma's HF processor inserts, after the prompt's leading `<bos>` token, a

563 + run of image tokens followed by a second `<bos>` token that marks the start of

564 + the text prompt. We start by building the run of image tokens, one per

565 + placeholder feature token:

566

567 ??? code

568

569 ```python

727 - def get_replacement(item_idx: int):

728 - images = mm_items.get_items("image", ImageProcessorItems)

729 - image_size = images.get_image_size(item_idx)

730 -

731 - ncols, nrows = self.info.get_image_feature_grid_size(

732 - image_width=image_size.width,

733 - image_height=image_size.height,

570 + def get_insertion(item_idx: int):

571 + images = mm_items.get_items(

572 + "image", (ImageEmbeddingItems, ImageProcessorItems)

573 )

574

736 - # `_IMAGE_TOKEN_ID` corresponds to `|SPEAKER|`

737 - # `_NEWLINE_TOKEN_ID` corresponds to `|NEWLINE|`

738 - return ([_IMAGE_TOKEN_ID] * ncols + [_NEWLINE_TOKEN_ID]) * nrows

739 - ```

740 -

741 - However, this is not entirely correct. After `FuyuImageProcessor.preprocess_with_tokenizer_info` is called,

742 - a BOS token (`<s>`) is also added to the prompt:

743 -

744 - ??? code

575 + if isinstance(images, ImageEmbeddingItems):

576 + num_image_tokens = images.get_feature_size(item_idx)

577 + else:

578 + image_size = images.get_image_size(item_idx)

579 + num_image_tokens = self.info.get_num_image_tokens(

580 + image_width=image_size.width,

581 + image_height=image_size.height,

582 + )

583

746 - ```python

747 - # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L435

748 - model_image_input = self.image_processor.preprocess_with_tokenizer_info(

749 - image_input=tensor_batch_images,

750 - image_present=image_present,

751 - image_unpadded_h=image_unpadded_heights,

752 - image_unpadded_w=image_unpadded_widths,

753 - image_placeholder_id=image_placeholder_id,

754 - image_newline_id=image_newline_id,

755 - variable_sized=True,

756 - )

757 - prompt_tokens, prompts_length = _tokenize_prompts_with_image_and_batch(

758 - tokenizer=self.tokenizer,

759 - prompts=prompts,

760 - scale_factors=scale_factors,

761 - max_tokens_to_generate=self.max_tokens_to_generate,

762 - max_position_embeddings=self.max_position_embeddings,

763 - add_BOS=True,

764 - add_beginning_of_answer_token=True,

765 - )

584 + image_tokens = [image_token_id] * num_image_tokens

585 + ...

586 ```

587

768 - To assign the vision embeddings to only the image tokens, instead of a string

769 - you can return an instance of [PromptUpdateDetails][vllm.multimodal.processing.PromptUpdateDetails]:

588 + The trailing `<bos>` token is an additional token that must **not** receive a

589 + vision embedding. To assign the vision embeddings to only the image tokens,

590 + instead of returning the token ids directly you can return an instance of

591 + [PromptUpdateDetails][vllm.multimodal.processing.PromptUpdateDetails] and mark

592 + the embedding tokens with `embed_token_id`:

593

594 ??? code

595

596 ```python

774 - hf_config = self.info.get_hf_config()

775 - bos_token_id = hf_config.bos_token_id # `<s>`

776 - assert isinstance(bos_token_id, int)

777 -

778 - def get_replacement_fuyu(item_idx: int):

779 - images = mm_items.get_items("image", ImageProcessorItems)

780 - image_size = images.get_image_size(item_idx)

781 -

782 - ncols, nrows = self.info.get_image_feature_grid_size(

783 - image_width=image_size.width,

784 - image_height=image_size.height,

785 - )

786 - image_tokens = ([_IMAGE_TOKEN_ID] * ncols + [_NEWLINE_TOKEN_ID]) * nrows

787 -

788 - return PromptUpdateDetails.select_token_id(

789 - image_tokens + [bos_token_id],

790 - embed_token_id=_IMAGE_TOKEN_ID,

791 - )

597 + return PromptUpdateDetails.select_token_id(

598 + image_tokens + [bos_token_id],

599 + embed_token_id=image_token_id,

600 + )

601 ```

602

794 - Finally, noticing that the HF processor removes the `|ENDOFTEXT|` token from the tokenized prompt,

795 - we can search for it to conduct the replacement at the start of the string:

603 + Putting it together, we override [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates].

604 + Since these tokens are inserted (rather than replacing an existing placeholder)

605 + after the prompt's leading `<bos>`, we use [PromptInsertion][vllm.multimodal.processing.PromptInsertion]

606 + with a prefix target:

607

608 ??? code

609

@@ -804,33 +615,41 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies

615 out_mm_kwargs: MultiModalKwargsItems,

616 ) -> Sequence[PromptUpdate]:

617 hf_config = self.info.get_hf_config()

807 - bos_token_id = hf_config.bos_token_id

808 - assert isinstance(bos_token_id, int)

618 + image_token_id = hf_config.image_token_index

619

620 tokenizer = self.info.get_tokenizer()

811 - eot_token_id = tokenizer.bos_token_id

812 - assert isinstance(eot_token_id, int)

621

814 - def get_replacement_fuyu(item_idx: int):

815 - images = mm_items.get_items("image", ImageProcessorItems)

816 - image_size = images.get_image_size(item_idx)

622 + bos_token_id = tokenizer.bos_token_id

623 + assert isinstance(bos_token_id, int)

624

818 - ncols, nrows = self.info.get_image_feature_grid_size(

819 - image_width=image_size.width,

820 - image_height=image_size.height,

625 + def get_insertion(item_idx: int):

626 + images = mm_items.get_items(

627 + "image", (ImageEmbeddingItems, ImageProcessorItems)

628 )

822 - image_tokens = ([_IMAGE_TOKEN_ID] * ncols + [_NEWLINE_TOKEN_ID]) * nrows

629 +

630 + if isinstance(images, ImageEmbeddingItems):

631 + num_image_tokens = images.get_feature_size(item_idx)

632 + else:

633 + image_size = images.get_image_size(item_idx)

634 + num_image_tokens = self.info.get_num_image_tokens(

635 + image_width=image_size.width,

636 + image_height=image_size.height,

637 + )

638 +

639 + image_tokens = [image_token_id] * num_image_tokens

640

641 return PromptUpdateDetails.select_token_id(

642 image_tokens + [bos_token_id],

826 - embed_token_id=_IMAGE_TOKEN_ID,

643 + embed_token_id=image_token_id,

644 )

645

646 return [

830 - PromptReplacement(

647 + PromptInsertion(

648 modality="image",

832 - target=[eot_token_id],

833 - replacement=get_replacement_fuyu,

649 + target=PromptIndexTargets.prefix(

650 + [bos_token_id] if tokenizer.add_bos_token else []

651 + ),

652 + insertion=get_insertion,

653 )

654 ]

655 ```

@@ -873,7 +692,7 @@ Examples:

692 Examples:

693

694 - Chameleon (appends `sep_token`): [vllm/model_executor/models/chameleon.py](../../../vllm/model_executor/models/chameleon.py)

876 -- Fuyu (appends `boa_token`): [vllm/model_executor/models/fuyu.py](../../../vllm/model_executor/models/fuyu.py)

695 +- Molmo2 (prepends `bos_token`): [vllm/model_executor/models/molmo2.py](../../../vllm/model_executor/models/molmo2.py)

696 - Molmo (applies chat template which is not defined elsewhere): [vllm/model_executor/models/molmo.py](../../../vllm/model_executor/models/molmo.py)

697

698 ### Custom HF processor

docs/models/supported_models.md

-2

@@ -436,7 +436,6 @@ th {

436 | `PhiForCausalLM` | Phi | `microsoft/phi-1_5`, `microsoft/phi-2`, etc. | ✅︎ | ✅︎ |

437 | `Phi3ForCausalLM` | Phi-4, Phi-3 | `microsoft/Phi-4-mini-instruct`, `microsoft/Phi-4`, `microsoft/Phi-3-mini-4k-instruct`, `microsoft/Phi-3-mini-128k-instruct`, `microsoft/Phi-3-medium-128k-instruct`, etc. | ✅︎ | ✅︎ |

438 | `PhiMoEForCausalLM` | Phi-3.5-MoE | `microsoft/Phi-3.5-MoE-instruct`, etc. | ✅︎ | ✅︎ |

439 | `Plamo2ForCausalLM` | PLaMo2 | `pfnet/plamo-2-1b`, `pfnet/plamo-2-8b`, etc. | ✅ | ✅︎ |

440 | `Plamo3ForCausalLM` | PLaMo3 | `pfnet/plamo-3-nict-2b-base`, `pfnet/plamo-3-nict-8b-base`, etc. | ✅ | ✅︎ |

441 | `Qwen2ForCausalLM` | QwQ, Qwen2 | `Qwen/QwQ-32B-Preview`, `Qwen/Qwen2-7B-Instruct`, `Qwen/Qwen2-7B`, etc. | ✅︎ | ✅︎ |

@@ -520,7 +519,6 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen

519 | `Eagle2_5_VLForConditionalGeneration` | Eagle2.5-VL | T + IE+ | `nvidia/Eagle2.5-8B`, etc. | ✅︎ | ✅︎ |

520 | `Ernie4_5_VLMoeForConditionalGeneration` | Ernie4.5-VL | T + I+/ V+ | `baidu/ERNIE-4.5-VL-28B-A3B-PT`, `baidu/ERNIE-4.5-VL-424B-A47B-PT` | | ✅︎ |

521 | `Exaone4_5_ForConditionalGeneration` | EXAONE-4.5 | T + IE+ | `LGAI-EXAONE/EXAONE-4.5-33B`, etc. | ✅︎ | ✅︎ |

523 -| `FuyuForCausalLM` | Fuyu | T + I | `adept/fuyu-8b`, etc. | | ✅︎ |

522 | `Gemma3ForConditionalGeneration` | Gemma 3 | T + IE+ | `google/gemma-3-4b-it`, `google/gemma-3-27b-it`, etc. | ✅︎ | ✅︎ |

examples/generate/multimodal/vision_language_offline.py

-19

@@ -448,24 +448,6 @@ def run_exaone4_5(questions: list[str], modality: str) -> ModelRequestData:

448 )

449

450

451 -# Fuyu

452 -def run_fuyu(questions: list[str], modality: str) -> ModelRequestData:

453 - assert modality == "image"

454 -

455 - prompts = [f"{question}\n" for question in questions]

456 - engine_args = EngineArgs(

457 - model="adept/fuyu-8b",

458 - max_model_len=2048,

459 - max_num_seqs=2,

460 - limit_mm_per_prompt={modality: 1},

461 - )

462 -

463 - return ModelRequestData(

464 - engine_args=engine_args,

465 - prompts=prompts,

466 - )

467 -

468 -

451 # Gemma 3

452 def run_gemma3(questions: list[str], modality: str) -> ModelRequestData:

453 assert modality == "image"

@@ -2318,7 +2300,6 @@ model_example_map = {

2300 "eagle2_5": run_eagle2_5,

2301 "ernie45_vl": run_ernie45_vl,

2302 "exaone4_5": run_exaone4_5,

2321 - "fuyu": run_fuyu,

2303 "gemma3": run_gemma3,

2304 "gemma3n": run_gemma3n,

2305 "glm4v": run_glm4v,

tests/distributed/test_pipeline_parallel.py

-2

@@ -139,7 +139,6 @@ TEXT_GENERATION_MODELS = {

139 "allenai/OLMoE-1B-7B-0924-Instruct": PPTestSettings.fast(),

140 "facebook/opt-iml-max-1.3b": PPTestSettings.fast(),

141 "OrionStarAI/Orion-14B-Chat": PPTestSettings.fast(),

142 - "adept/persimmon-8b-chat": PPTestSettings.fast(),

142 "microsoft/phi-2": PPTestSettings.fast(),

143 "microsoft/Phi-3-small-8k-instruct": PPTestSettings.fast(),

144 "microsoft/Phi-3.5-MoE-instruct": PPTestSettings.detailed(

@@ -168,7 +167,6 @@ MULTIMODAL_MODELS = {

167 # [Decoder-only]

168 "Salesforce/blip2-opt-6.7b": PPTestSettings.fast(),

169 "facebook/chameleon-7b": PPTestSettings.fast(),

171 - "adept/fuyu-8b": PPTestSettings.fast(),

170 "zai-org/glm-4v-9b": PPTestSettings.fast(),

171 "OpenGVLab/InternVL3-1B": PPTestSettings.fast(),

172 "llava-hf/llava-1.5-7b-hf": PPTestSettings.fast(),

tests/models/multimodal/generation/test_common.py

-14

@@ -388,20 +388,6 @@ VLM_TEST_SETTINGS = {

388 stop_str=["<｜end▁of▁sentence｜>", "<｜begin▁of▁sentence｜>"],

389 image_size_factors=[(1.0,), (1.0, 1.0, 1.0), (0.1, 0.5, 1.0)],

390 ),

391 - "fuyu": VLMTestInfo(

392 - models=["adept/fuyu-8b"],

393 - test_type=VLMTestType.IMAGE,

394 - prompt_formatter=lambda img_prompt: f"{img_prompt}\n",

395 - img_idx_to_prompt=lambda idx: "",

396 - max_model_len=2048,

397 - max_num_seqs=2,

398 - auto_cls=AutoModelForImageTextToText,

399 - use_tokenizer_eos=True,

400 - vllm_output_post_proc=model_utils.fuyu_vllm_to_hf_output,

401 - num_logprobs=10,

402 - image_size_factors=[(0.25,), (0.25, 0.25, 0.25), (0.25, 0.2, 0.15)],

403 - marks=[large_gpu_mark(min_gb=32)],

404 - ),

391 "gemma3": VLMTestInfo(

392 models=["google/gemma-3-4b-it"],

393 test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE),

tests/models/multimodal/generation/vlm_utils/model_utils.py

-9

@@ -52,15 +52,6 @@ def blip2_vllm_to_hf_output(vllm_output: RunnerOutput, model: str) -> RunnerOutp

52 return hf_output_ids, hf_output_str, out_logprobs

55 -def fuyu_vllm_to_hf_output(vllm_output: RunnerOutput, model: str) -> RunnerOutput:

56 - """Sanitize vllm output [fuyu models] to be comparable with hf output."""

57 - output_ids, output_str, out_logprobs = vllm_output

58 -

59 - hf_output_str = output_str.lstrip() + "|ENDOFTEXT|"

60 -

61 - return output_ids, hf_output_str, out_logprobs

62 -

63 -

55 def qwen_vllm_to_hf_output(

56 vllm_output: RunnerOutput, model: str

57 ) -> tuple[list[int], str, SampleLogprobs | None]:

tests/models/registry.py

-2

@@ -466,7 +466,6 @@ _TEXT_GENERATION_EXAMPLE_MODELS = {

466 "bharatgenai/Param2-17B-A2.4B-Thinking",

467 trust_remote_code=True,

468 ),

469 - "PersimmonForCausalLM": _HfExamplesInfo("adept/persimmon-8b-chat"),

469 "PhiForCausalLM": _HfExamplesInfo("microsoft/phi-2"),

470 "Phi3ForCausalLM": _HfExamplesInfo("microsoft/Phi-3-mini-4k-instruct"),

471 "PhiMoEForCausalLM": _HfExamplesInfo(

@@ -866,7 +865,6 @@ _MULTIMODAL_EXAMPLE_MODELS = {

865 "FunAudioChatForConditionalGeneration": _HfExamplesInfo(

866 "funaudiochat", is_available_online=False

867 ),

869 - "FuyuForCausalLM": _HfExamplesInfo("adept/fuyu-8b"),

868 "Gemma3ForConditionalGeneration": _HfExamplesInfo("google/gemma-3-4b-it"),

869 "DiffusionGemmaForBlockDiffusion": _HfExamplesInfo(

870 "google/diffusiongemma-26B-A4B-it",

tests/renderers/test_hf.py

-1

@@ -426,7 +426,6 @@ def test_resolve_content_format_hf_defined(model, expected_format):

426 ("Salesforce/blip2-opt-2.7b", "string"),

427 ("facebook/chameleon-7b", "string"),

428 ("deepseek-ai/deepseek-vl2-tiny", "string"),

429 - ("adept/fuyu-8b", "string"),

429 ("google/paligemma-3b-mix-224", "string"),

430 ],

431 )

vllm/model_executor/models/fuyu.py deleted

-368

@@ -1,368 +0,0 @@

1 -# SPDX-License-Identifier: Apache-2.0

2 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 -

4 -# adapted from https://github.com/huggingface/transformers/blob/v4.39.3/src/transformers/models/fuyu/modeling_fuyu.py

7 -#

8 -# Licensed under the Apache License, Version 2.0 (the "License");

9 -# you may not use this file except in compliance with the License.

10 -# You may obtain a copy of the License at

11 -#

12 -# http://www.apache.org/licenses/LICENSE-2.0

13 -#

14 -# Unless required by applicable law or agreed to in writing, software

15 -# distributed under the License is distributed on an "AS IS" BASIS,

16 -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

17 -# See the License for the specific language governing permissions and

18 -# limitations under the License.

19 -"""PyTorch Fuyu model."""

20 -

21 -import math

22 -from collections.abc import Iterable, Mapping, Sequence

23 -from typing import Annotated, Literal

24 -

25 -import torch

26 -import torch.nn as nn

27 -from transformers import BatchFeature, FuyuConfig, FuyuImageProcessor, FuyuProcessor

28 -

29 -from vllm.config import VllmConfig

30 -from vllm.config.multimodal import BaseDummyOptions

31 -from vllm.inputs import MultiModalDataDict

32 -from vllm.model_executor.layers.linear import ColumnParallelLinear

33 -from vllm.model_executor.models.persimmon import PersimmonForCausalLM

34 -from vllm.multimodal import MULTIMODAL_REGISTRY

35 -from vllm.multimodal.inputs import (

36 - MultiModalFieldConfig,

37 - MultiModalKwargsItems,

38 -)

39 -from vllm.multimodal.parse import ImageProcessorItems, ImageSize, MultiModalDataItems

40 -from vllm.multimodal.processing import (

41 - BaseDummyInputsBuilder,

42 - BaseMultiModalProcessor,

43 - BaseProcessingInfo,

44 - PromptReplacement,

45 - PromptUpdate,

46 - PromptUpdateDetails,

47 -)

48 -from vllm.sequence import IntermediateTensors

49 -from vllm.utils.tensor_schema import TensorSchema, TensorShape

50 -

51 -from .interfaces import MultiModalEmbeddings, SupportsMultiModal, SupportsPP

52 -from .utils import AutoWeightsLoader, WeightsMapper, flatten_bn, maybe_prefix

53 -

54 -# Cannot find the following 2 numbers from hf config.

55 -_IMAGE_TOKEN_ID = 71011

56 -_NEWLINE_TOKEN_ID = 71019

57 -

58 -

59 -class FuyuImagePatchInputs(TensorSchema):

60 - """

61 - Dimensions:

62 - - bn: Batch size * number of images

63 - - bnp: Batch size * number of images * number of patches

64 - - fn: patch_size_x * patch_size_y * num_channels

65 - """

66 -

67 - type: Literal["image_patches"] = "image_patches"

68 -

69 - image_patches_flat: Annotated[torch.Tensor, TensorShape("bnp", "fn")]

70 -

71 - patches_per_image: Annotated[list[int], TensorShape("bn")]

72 - """

73 - The number of total patches for each image in the batch.

74 -

75 - This is used to split the embeddings which has the first two dimensions

76 - flattened just like `image_patches_flat`.

77 - """

78 -

79 -

80 -class FuyuProcessingInfo(BaseProcessingInfo):

81 - def get_hf_config(self):

82 - return self.ctx.get_hf_config(FuyuConfig)

83 -

84 - def get_hf_processor(self, **kwargs: object):

85 - return self.ctx.get_hf_processor(FuyuProcessor, **kwargs)

86 -

87 - def get_image_processor(self, **kwargs: object) -> FuyuImageProcessor:

88 - return self.get_hf_processor(**kwargs).image_processor

89 -

90 - def get_supported_mm_limits(self) -> Mapping[str, int | None]:

91 - return {"image": 1}

92 -

93 - def get_image_feature_grid_size(

94 - self,

95 - *,

96 - image_width: int,

97 - image_height: int,

98 - ) -> tuple[int, int]:

99 - image_processor = self.get_image_processor()

100 - target_width = image_processor.size["width"]

101 - target_height = image_processor.size["height"]

102 - patch_width = image_processor.patch_size["width"]

103 - patch_height = image_processor.patch_size["height"]

104 -

105 - if not (image_width <= target_width and image_height <= target_height):

106 - height_scale_factor = target_height / image_height

107 - width_scale_factor = target_width / image_width

108 - optimal_scale_factor = min(height_scale_factor, width_scale_factor)

109 -

110 - image_height = int(image_height * optimal_scale_factor)

111 - image_width = int(image_width * optimal_scale_factor)

112 -

113 - ncols = math.ceil(image_width / patch_width)

114 - nrows = math.ceil(image_height / patch_height)

115 - return ncols, nrows

116 -

117 - def get_num_image_tokens(

118 - self,

119 - *,

120 - image_width: int,

121 - image_height: int,

122 - ) -> int:

123 - ncols, nrows = self.get_image_feature_grid_size(

124 - image_width=image_width,

125 - image_height=image_height,

126 - )

127 -

128 - return ncols * nrows

129 -

130 - def get_image_size_with_most_features(self) -> ImageSize:

131 - image_processor = self.get_image_processor()

132 - return ImageSize(

133 - width=image_processor.size["width"], height=image_processor.size["height"]

134 - )

135 -

136 -

137 -class FuyuDummyInputsBuilder(BaseDummyInputsBuilder[FuyuProcessingInfo]):

138 - def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:

139 - return ""

140 -

141 - def get_dummy_mm_data(

142 - self,

143 - seq_len: int,

144 - mm_counts: Mapping[str, int],

145 - mm_options: Mapping[str, BaseDummyOptions],

146 - ) -> MultiModalDataDict:

147 - target_width, target_height = self.info.get_image_size_with_most_features()

148 - num_images = mm_counts.get("image", 0)

149 -

150 - image_overrides = mm_options.get("image")

151 -

152 - return {

153 - "image": self._get_dummy_images(

154 - width=target_width,

155 - height=target_height,

156 - num_images=num_images,

157 - overrides=image_overrides,

158 - )

159 - }

160 -

161 -

162 -class FuyuMultiModalProcessor(BaseMultiModalProcessor[FuyuProcessingInfo]):

163 - def _call_hf_processor(

164 - self,

165 - prompt: str,

166 - mm_data: Mapping[str, object],

167 - mm_kwargs: Mapping[str, object],

168 - tok_kwargs: Mapping[str, object],

169 - ) -> BatchFeature:

170 - if not mm_data:

171 - # Avoid warning from HF logger for text-only input

172 - prompt_ids = self.info.get_tokenizer().encode(prompt)

173 - prompt_ids = self._apply_hf_processor_tokens_only(prompt_ids)

174 - return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")

175 -

176 - processed_outputs = super()._call_hf_processor(

177 - prompt=prompt,

178 - mm_data=mm_data,

179 - mm_kwargs=mm_kwargs,

180 - tok_kwargs=tok_kwargs,

181 - )

182 -

183 - image_patches = processed_outputs["image_patches"]

184 - processed_outputs["image_patches"] = flatten_bn(image_patches)

185 - processed_outputs["patches_per_image"] = torch.tensor(

186 - [len(p) for p in image_patches]

187 - )

188 -

189 - return processed_outputs

190 -

191 - def _apply_hf_processor_tokens_only(

192 - self,

193 - prompt_tokens: list[int],

194 - ) -> list[int]:

195 - # HF processor adds boa_token_id

196 - tokenizer = self.info.get_tokenizer()

197 - vocab = tokenizer.get_vocab()

198 -

199 - boa_token_id = vocab["<0x04>"]

200 - if prompt_tokens[-1] != boa_token_id:

201 - prompt_tokens.append(boa_token_id)

202 -

203 - return prompt_tokens

204 -

205 - def _get_mm_fields_config(

206 - self,

207 - hf_inputs: BatchFeature,

208 - hf_processor_mm_kwargs: Mapping[str, object],

209 - ) -> Mapping[str, MultiModalFieldConfig]:

210 - patches_per_image = hf_inputs.get("patches_per_image", torch.empty(0))

211 -

212 - return dict(

213 - image_patches=MultiModalFieldConfig.flat_from_sizes(

214 - "image", patches_per_image

215 - ),

216 - patches_per_image=MultiModalFieldConfig.batched("image"),

217 - )

218 -

219 - def _get_prompt_updates(

220 - self,

221 - mm_items: MultiModalDataItems,

222 - hf_processor_mm_kwargs: Mapping[str, object],

223 - out_mm_kwargs: MultiModalKwargsItems,

224 - ) -> Sequence[PromptUpdate]:

225 - hf_config = self.info.get_hf_config()

226 - bos_token_id = hf_config.bos_token_id

227 - assert isinstance(bos_token_id, int)

228 -

229 - tokenizer = self.info.get_tokenizer()

230 - eot_token_id = tokenizer.bos_token_id

231 - assert isinstance(eot_token_id, int)

232 -

233 - def get_replacement_fuyu(item_idx: int):

234 - images = mm_items.get_items("image", ImageProcessorItems)

235 - image_size = images.get_image_size(item_idx)

236 -

237 - ncols, nrows = self.info.get_image_feature_grid_size(

238 - image_width=image_size.width,

239 - image_height=image_size.height,

240 - )

241 - image_tokens = ([_IMAGE_TOKEN_ID] * ncols + [_NEWLINE_TOKEN_ID]) * nrows

242 -

243 - return PromptUpdateDetails.select_token_id(

244 - image_tokens + [bos_token_id],

245 - embed_token_id=_IMAGE_TOKEN_ID,

246 - )

247 -

248 - return [

249 - PromptReplacement(

250 - modality="image",

251 - target=[eot_token_id],

252 - replacement=get_replacement_fuyu,

253 - )

254 - ]

255 -

256 -

257 -@MULTIMODAL_REGISTRY.register_processor(

258 - FuyuMultiModalProcessor,

259 - info=FuyuProcessingInfo,

260 - dummy_inputs=FuyuDummyInputsBuilder,

261 -)

262 -class FuyuForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):

263 - hf_to_vllm_mapper = WeightsMapper(

264 - orig_to_new_prefix={

265 - "model.vision_embed_tokens.": "vision_embed_tokens.",

266 - "model.language_model.": "language_model.model.",

267 - "lm_head.": "language_model.lm_head.",

268 - }

269 - )

270 -

271 - @classmethod

272 - def get_placeholder_str(cls, modality: str, i: int) -> str | None:

273 - if modality.startswith("image"):

274 - return None

275 -

276 - raise ValueError("Only image modality is supported")

277 -

278 - def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):

279 - super().__init__()

280 - config = vllm_config.model_config.hf_config

281 - quant_config = vllm_config.quant_config

282 - multimodal_config = vllm_config.model_config.multimodal_config

283 - self.config = config

284 - self.multimodal_config = multimodal_config

285 -

286 - self.vocab_size = config.text_config.vocab_size

287 - self.image_token_id = _IMAGE_TOKEN_ID

288 - self.image_feature_size = config.patch_size**2 * config.num_channels

289 -

290 - with self._mark_tower_model(vllm_config, "image"):

291 - self.vision_embed_tokens = ColumnParallelLinear(

292 - self.image_feature_size,

293 - config.hidden_size,

294 - quant_config=quant_config,

295 - gather_output=True,

296 - )

297 -

298 - with self._mark_language_model(vllm_config):

299 - self.language_model = PersimmonForCausalLM(

300 - vllm_config=vllm_config.with_hf_config(config.text_config),

301 - prefix=maybe_prefix(prefix, "language_model"),

302 - )

303 -

304 - self.make_empty_intermediate_tensors = (

305 - self.language_model.make_empty_intermediate_tensors

306 - )

307 -

308 - def _parse_and_validate_image_input(

309 - self, **kwargs: object

310 - ) -> FuyuImagePatchInputs | None:

311 - image_patches = kwargs.pop("image_patches", None)

312 - patches_per_image = kwargs.pop("patches_per_image", None)

313 -

314 - if image_patches is None:

315 - return None

316 -

317 - return FuyuImagePatchInputs(

318 - type="image_patches",

319 - image_patches_flat=image_patches,

320 - patches_per_image=patches_per_image,

321 - resolve_bindings={"fn": self.image_feature_size},

322 - )

323 -

324 - def _process_image_input(

325 - self, image_input: FuyuImagePatchInputs

326 - ) -> MultiModalEmbeddings:

327 - image_patches_flat = image_input["image_patches_flat"]

328 - patches_per_image = image_input["patches_per_image"]

329 -

330 - vision_embeddings_flat, _ = self.vision_embed_tokens(image_patches_flat)

331 -

332 - return vision_embeddings_flat.split(patches_per_image.tolist(), dim=0)

333 -

334 - def embed_multimodal(self, **kwargs: object) -> MultiModalEmbeddings:

335 - image_input = self._parse_and_validate_image_input(**kwargs)

336 - if image_input is None:

337 - return []

338 -

339 - return self._process_image_input(image_input)

340 -

341 - def forward(

342 - self,

343 - input_ids: torch.Tensor | None,

344 - positions: torch.Tensor,

345 - intermediate_tensors: IntermediateTensors | None = None,

346 - inputs_embeds: torch.Tensor | None = None,

347 - **kwargs: object,

348 - ):

349 - if intermediate_tensors is not None:

350 - inputs_embeds = None

351 -

352 - hidden_states = self.language_model(

353 - input_ids=input_ids,

354 - positions=positions,

355 - intermediate_tensors=intermediate_tensors,

356 - inputs_embeds=inputs_embeds,

357 - )

358 - return hidden_states

359 -

360 - def compute_logits(

361 - self,

362 - hidden_states: torch.Tensor,

363 - ) -> torch.Tensor | None:

364 - return self.language_model.compute_logits(hidden_states)

365 -

366 - def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:

367 - loader = AutoWeightsLoader(self)

368 - return loader.load_weights(weights)

vllm/model_executor/models/persimmon.py deleted

-373

@@ -1,373 +0,0 @@

1 -# SPDX-License-Identifier: Apache-2.0

2 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 -

4 -# adapted from https://github.com/huggingface/transformers/blob/v4.39.3/src/transformers/models/persimmon/modeling_persimmon.py

7 -#

8 -# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX

9 -# and OPT implementations in this library. It has been modified from its

10 -# original forms to accommodate minor architectural differences compared

11 -# to GPT-NeoX and OPT used by the Meta AI team that trained the model.

12 -#

13 -# Licensed under the Apache License, Version 2.0 (the "License");

14 -# you may not use this file except in compliance with the License.

15 -# You may obtain a copy of the License at

16 -#

17 -# http://www.apache.org/licenses/LICENSE-2.0

18 -#

19 -# Unless required by applicable law or agreed to in writing, software

20 -# distributed under the License is distributed on an "AS IS" BASIS,

21 -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

22 -# See the License for the specific language governing permissions and

23 -# limitations under the License.

24 -"""Inference-only persimmon model compatible with HuggingFace weights."""

25 -

26 -from collections.abc import Iterable

27 -from itertools import islice

28 -

29 -import torch

30 -from torch import nn

31 -from transformers import PersimmonConfig

32 -

33 -from vllm.compilation.decorators import support_torch_compile

34 -from vllm.config import CacheConfig, VllmConfig

35 -from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size

36 -from vllm.model_executor.layers.activation import get_act_fn

37 -from vllm.model_executor.layers.attention import Attention

38 -from vllm.model_executor.layers.linear import (

39 - ColumnParallelLinear,

40 - QKVParallelLinear,

41 - RowParallelLinear,

42 -)

43 -from vllm.model_executor.layers.logits_processor import LogitsProcessor

44 -from vllm.model_executor.layers.quantization import QuantizationConfig

45 -from vllm.model_executor.layers.rotary_embedding import get_rope

46 -from vllm.model_executor.layers.vocab_parallel_embedding import (

47 - ParallelLMHead,

48 - VocabParallelEmbedding,

49 -)

50 -from vllm.model_executor.model_loader.weight_utils import default_weight_loader

51 -from vllm.sequence import IntermediateTensors

52 -

53 -from .interfaces import SupportsPP

54 -from .utils import (

55 - AutoWeightsLoader,

56 - is_pp_missing_parameter,

57 - make_empty_intermediate_tensors_factory,

58 - make_layers,

59 - maybe_prefix,

60 -)

61 -

62 -

63 -class PersimmonMLP(nn.Module):

64 - def __init__(

65 - self,

66 - config: PersimmonConfig,

67 - quant_config: QuantizationConfig | None = None,

68 - prefix: str = "",

69 - ):

70 - super().__init__()

71 - self.dense_h_to_4h = ColumnParallelLinear(

72 - config.hidden_size,

73 - config.intermediate_size,

74 - quant_config=quant_config,

75 - prefix=f"{prefix}.dense_h_to_4h",

76 - )

77 - self.dense_4h_to_h = RowParallelLinear(

78 - config.intermediate_size,

79 - config.hidden_size,

80 - quant_config=quant_config,

81 - prefix=f"{prefix}.dense_4h_to_h",

82 - )

83 - self.act = get_act_fn(config.hidden_act)

84 -

85 - def forward(self, hidden_states) -> torch.Tensor:

86 - hidden_states, _ = self.dense_h_to_4h(hidden_states)

87 - hidden_states = self.act(hidden_states)

88 - hidden_states, _ = self.dense_4h_to_h(hidden_states)

89 - return hidden_states

90 -

91 -

92 -class PersimmonAttention(nn.Module):

93 - def __init__(

94 - self,

95 - config: PersimmonConfig,

96 - cache_config: CacheConfig | None = None,

97 - quant_config: QuantizationConfig | None = None,

98 - prefix: str = "",

99 - ):

100 - super().__init__()

101 - self.config = config

102 - tensor_parallel_world_size = get_tensor_model_parallel_world_size()

103 -

104 - self.hidden_size = config.hidden_size

105 - self.total_num_heads = config.num_attention_heads

106 - self.num_heads = self.total_num_heads // tensor_parallel_world_size

107 - self.head_dim = self.hidden_size // self.total_num_heads

108 - self.max_position_embeddings = config.max_position_embeddings

109 - self.is_causal = True

110 -

111 - assert (self.head_dim * self.total_num_heads) == self.hidden_size

112 - assert self.total_num_heads % tensor_parallel_world_size == 0

113 -

114 - self.query_key_value = QKVParallelLinear(

115 - self.hidden_size,

116 - self.head_dim,

117 - self.total_num_heads,

118 - bias=True,

119 - quant_config=quant_config,

120 - prefix=f"{prefix}.query_key_value",

121 - )

122 - self.dense = RowParallelLinear(

123 - self.total_num_heads * self.head_dim,

124 - self.hidden_size,

125 - bias=True,

126 - quant_config=quant_config,

127 - prefix=f"{prefix}.dense",

128 - )

129 - self.is_qk_layernorm = config.qk_layernorm

130 -

131 - if self.is_qk_layernorm:

132 - self.q_layernorm = nn.LayerNorm(self.head_dim)

133 - self.k_layernorm = nn.LayerNorm(self.head_dim)

134 -

135 - self.rotary_emb = get_rope(

136 - self.head_dim,

137 - max_position=self.max_position_embeddings,

138 - rope_parameters=config.rope_parameters,

139 - )

140 - self.scaling = self.head_dim**-0.5

141 - self.attn = Attention(

142 - self.num_heads,

143 - self.head_dim,

144 - scale=self.scaling,

145 - cache_config=cache_config,

146 - quant_config=quant_config,

147 - prefix=f"{prefix}.attn",

148 - )

149 -

150 - def _split_heads(self, x: torch.Tensor) -> torch.Tensor:

151 - # [seq_length, hidden_size] -> [seq_length, num_heads, head_dim]

152 - seq_length = x.shape[0]

153 - return x.view(seq_length, self.num_heads, self.head_dim)

154 -

155 - def _merge_heads(self, x: torch.Tensor) -> torch.Tensor:

156 - # [seq_length, num_heads, head_dim] -> [seq_length, hidden_size]

157 - seq_length = x.shape[0]

158 - return x.view(seq_length, self.num_heads * self.head_dim)

159 -

160 - def forward(

161 - self,

162 - position_ids: torch.Tensor,

163 - hidden_states: torch.Tensor,

164 - ) -> torch.Tensor:

165 - # [seq_length, 3 x hidden_size]

166 - qkv, _ = self.query_key_value(hidden_states)

167 - q, k, v = qkv.chunk(chunks=3, dim=-1)

168 -

169 - if self.is_qk_layernorm:

170 - # [seq_length, num_heads, head_dim]

171 - q = self._split_heads(q)

172 - k = self._split_heads(k)

173 -

174 - q = self.q_layernorm(q)

175 - k = self.k_layernorm(k)

176 -

177 - q = self._merge_heads(q)

178 - k = self._merge_heads(k)

179 -

180 - q, k = self.rotary_emb(position_ids, q, k)

181 - attn_output = self.attn(q, k, v)

182 - output, _ = self.dense(attn_output)

183 - return output

184 -

185 -

186 -class PersimmonDecoderLayer(nn.Module):

187 - def __init__(

188 - self,

189 - config: PersimmonConfig,

190 - cache_config: CacheConfig | None = None,

191 - quant_config: QuantizationConfig | None = None,

192 - prefix: str = "",

193 - ):

194 - super().__init__()

195 - self.hidden_size = config.hidden_size

196 - self.self_attn = PersimmonAttention(

197 - config=config,

198 - cache_config=cache_config,

199 - quant_config=quant_config,

200 - prefix=f"{prefix}.self_attn",

201 - )

202 - self.mlp = PersimmonMLP(

203 - config,

204 - quant_config=quant_config,

205 - prefix=f"{prefix}.mlp",

206 - )

207 - self.input_layernorm = nn.LayerNorm(

208 - config.hidden_size, eps=config.layer_norm_eps

209 - )

210 - self.post_attention_layernorm = nn.LayerNorm(

211 - config.hidden_size, eps=config.layer_norm_eps

212 - )

213 -

214 - def forward(

215 - self,

216 - position_ids: torch.Tensor,

217 - hidden_states: torch.Tensor,

218 - ) -> torch.Tensor:

219 - residual = hidden_states

220 -

221 - hidden_states = self.input_layernorm(hidden_states)

222 -

223 - # Self Attention

224 - hidden_states = self.self_attn(

225 - position_ids=position_ids,

226 - hidden_states=hidden_states,

227 - )

228 - hidden_states = residual + hidden_states

229 -

230 - # Fully Connected

231 - residual = hidden_states

232 - hidden_states = self.post_attention_layernorm(hidden_states)

233 - hidden_states = self.mlp(hidden_states)

234 -

235 - hidden_states = hidden_states + residual

236 -

237 - outputs = hidden_states

238 - return outputs

239 -

240 -

241 -@support_torch_compile

242 -class PersimmonModel(nn.Module):

243 - def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):

244 - super().__init__()

245 -

246 - config = vllm_config.model_config.hf_config

247 - cache_config = vllm_config.cache_config

248 - quant_config = vllm_config.quant_config

249 -

250 - self.vocab_size = config.vocab_size

251 - self.config = config

252 - self.embed_tokens = VocabParallelEmbedding(

253 - config.vocab_size, config.hidden_size

254 - )

255 - self.start_layer, self.end_layer, self.layers = make_layers(

256 - config.num_hidden_layers,

257 - lambda prefix: PersimmonDecoderLayer(

258 - config, cache_config, quant_config, prefix=prefix

259 - ),

260 - prefix=f"{prefix}.layers",

261 - )

262 - self.final_layernorm = nn.LayerNorm(

263 - config.hidden_size, eps=config.layer_norm_eps

264 - )

265 - self.make_empty_intermediate_tensors = make_empty_intermediate_tensors_factory(

266 - ["hidden_states"], config.hidden_size

267 - )

268 -

269 - def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:

270 - return self.embed_tokens(input_ids)

271 -

272 - def forward(

273 - self,

274 - input_ids: torch.Tensor | None,

275 - positions: torch.Tensor,

276 - intermediate_tensors: IntermediateTensors | None,

277 - inputs_embeds: torch.Tensor | None = None,

278 - ) -> torch.Tensor | IntermediateTensors:

279 - if get_pp_group().is_first_rank:

280 - if inputs_embeds is not None:

281 - hidden_states = inputs_embeds

282 - else:

283 - hidden_states = self.embed_input_ids(input_ids)

284 - else:

285 - assert intermediate_tensors is not None

286 - hidden_states = intermediate_tensors["hidden_states"]

287 - for layer in islice(self.layers, self.start_layer, self.end_layer):

288 - hidden_states = layer(positions, hidden_states)

289 - if not get_pp_group().is_last_rank:

290 - return IntermediateTensors({"hidden_states": hidden_states})

291 - hidden_states = self.final_layernorm(hidden_states)

292 - return hidden_states

293 -

294 - def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:

295 - params_dict = dict(self.named_parameters(remove_duplicate=False))

296 - loaded_params: set[str] = set()

297 - for name, loaded_weight in weights:

298 - if is_pp_missing_parameter(name, self):

299 - continue

300 - param = params_dict[name]

301 -

302 - if "query_key_value" in name:

303 - # copy from vllm/model_executor/models/bloom.py

304 - # NOTE: Persimmon's fused QKV's output_dim has the shape of

305 - # (num_heads * 3 * head_size), while the

306 - # required shape is (3 * num_heads * head_size).

307 - # Thus, we need weight conversion.

308 - output_dim = getattr(param, "output_dim", None)

309 - num_heads = self.config.num_attention_heads

310 - if output_dim is not None:

311 - loaded_weight_shape = loaded_weight.shape

312 - loaded_weight = loaded_weight.view(

313 - loaded_weight_shape[:output_dim]

314 - + (num_heads, 3, -1)

315 - + loaded_weight_shape[output_dim + 1 :]

316 - )

317 - loaded_weight = loaded_weight.transpose(output_dim, output_dim + 1)

318 - loaded_weight = loaded_weight.reshape(loaded_weight_shape)

319 -

320 - weight_loader = getattr(param, "weight_loader", default_weight_loader)

321 - weight_loader(param, loaded_weight)

322 - loaded_params.add(name)

323 - return loaded_params

324 -

325 -

326 -class PersimmonForCausalLM(nn.Module, SupportsPP):

327 - def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):

328 - super().__init__()

329 - config = vllm_config.model_config.hf_config

330 - self.config = config

331 - self.vocab_size = config.vocab_size

332 - self.model = PersimmonModel(

333 - vllm_config=vllm_config, prefix=maybe_prefix(prefix, "model")

334 - )

335 - self.lm_head = ParallelLMHead(

336 - config.vocab_size,

337 - config.hidden_size,

338 - bias=False,

339 - prefix=maybe_prefix(prefix, "lm_head"),

340 - )

341 - self.logits_processor = LogitsProcessor(config.vocab_size)

342 - self.make_empty_intermediate_tensors = (

343 - self.model.make_empty_intermediate_tensors

344 - )

345 -

346 - def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor:

347 - return self.model.embed_input_ids(input_ids)

348 -

349 - def forward(

350 - self,

351 - input_ids: torch.Tensor | None,

352 - positions: torch.Tensor,

353 - intermediate_tensors: IntermediateTensors | None = None,

354 - inputs_embeds: torch.Tensor | None = None,

355 - ):

356 - hidden_states = self.model(

357 - input_ids=input_ids,

358 - positions=positions,

359 - intermediate_tensors=intermediate_tensors,

360 - inputs_embeds=inputs_embeds,

361 - )

362 - return hidden_states

363 -

364 - def compute_logits(

365 - self,

366 - hidden_states: torch.Tensor,

367 - ) -> torch.Tensor | None:

368 - logits = self.logits_processor(self.lm_head, hidden_states)

369 - return logits

370 -

371 - def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:

372 - loader = AutoWeightsLoader(self)

373 - return loader.load_weights(weights)

vllm/model_executor/models/registry.py

+2 -2

@@ -180,7 +180,6 @@ _TEXT_GENERATION_MODELS = {

180 "PanguProMoEV2ForCausalLM": ("openpangu", "PanguProMoEV2ForCausalLM"),

181 "PanguUltraMoEForCausalLM": ("openpangu", "PanguUltraMoEForCausalLM"),

182 "Param2MoEForCausalLM": ("param2moe", "Param2MoEForCausalLM"),

183 - "PersimmonForCausalLM": ("persimmon", "PersimmonForCausalLM"),

183 "PhiForCausalLM": ("phi", "PhiForCausalLM"),

184 "Phi3ForCausalLM": ("phi3", "Phi3ForCausalLM"),

185 "PhiMoEForCausalLM": ("phimoe", "PhiMoEForCausalLM"),

@@ -377,7 +376,6 @@ _MULTIMODAL_MODELS = {

376 "funaudiochat",

377 "FunAudioChatForConditionalGeneration",

378 ),

380 - "FuyuForCausalLM": ("fuyu", "FuyuForCausalLM"),

379 "Gemma3ForConditionalGeneration": ("gemma3_mm", "Gemma3ForConditionalGeneration"),

380 "Gemma3nForConditionalGeneration": (

381 "gemma3n_mm",

@@ -738,6 +736,8 @@ _PREVIOUSLY_SUPPORTED_MODELS = {

736 "MusicFlamingoForConditionalGeneration": "0.24.0",

737 "AyaVisionForConditionalGeneration": "0.24.0",

738 "TeleChatForCausalLM": "0.25.0",

739 + "PersimmonForCausalLM": "0.25.0",

740 + "FuyuForCausalLM": "0.25.0",

741 }

742

743 _OOT_SUPPORTED_MODELS = {

vllm/transformers_utils/chat_templates/registry.py

-1

@@ -31,7 +31,6 @@ _MODEL_TYPE_TO_CHAT_TEMPLATE_FALLBACK: dict[str, ChatTemplatePath] = {

31 "deepseek_ocr2": CHAT_TEMPLATES_DIR / "template_deepseek_ocr.jinja",

32 "unlimited-ocr": CHAT_TEMPLATES_DIR / "template_deepseek_ocr.jinja",

33 "deepseek_vl_v2": CHAT_TEMPLATES_DIR / "template_deepseek_vl2.jinja",

34 - "fuyu": CHAT_TEMPLATES_DIR / "template_fuyu.jinja",

34 "minicpmv": _get_minicpmv_chat_template_fallback,

35 "minicpmv4_6": _get_minicpmv_chat_template_fallback,

36 "paligemma": CHAT_TEMPLATES_DIR / "template_basic.jinja",

vllm/transformers_utils/chat_templates/template_fuyu.jinja deleted

-3

@@ -1,3 +0,0 @@

1 -{%- for message in messages -%}

2 - {{- message['content'] + '\n' -}}

3 -{%- endfor -%}