A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 9, 23:00 UTC
[Doc] Update docs for MM model development with context usage (#32691)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Jan 21, 2026 at 02:37 UTC
09194b90a5353bf51cabfeaa0ec93f139c2c4ac3
2 files changed
+20
-28
docs/contributing/model/multimodal.md
+19
-27
@@ -23,29 +23,32 @@ Further update the model as follows:
23
raise ValueError("Only image modality is supported")
24
```
25
26
-- Reserve a keyword parameter in [forward][torch.nn.Module.forward] for each input tensor that corresponds to a multi-modal input, as shown in the following example:
26
+- Inside `__init__` method, initialize the language components of the model inside [_mark_language_model][vllm.model_executor.models.interfaces.SupportsMultiModal._mark_language_model], and the multimodal components of the model inside [_mark_tower_model][vllm.model_executor.models.interfaces.SupportsMultiModal._mark_tower_model], e.g.:
27
28
- ```diff
29
- def forward(
30
- self,
31
- input_ids: torch.Tensor,
32
- positions: torch.Tensor,
33
- + pixel_values: torch.Tensor,
34
- ) -> SamplerOutput:
35
- ```
36
-
37
- More conveniently, you can simply pass `**kwargs` to the [forward][torch.nn.Module.forward] method and retrieve the keyword parameters for multimodal inputs from it.
28
+ ```python
29
+ def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
30
+ super().__init__()
31
+
32
+ config = vllm_config.model_config.hf_config
33
+
34
+ with self._mark_tower_model(vllm_config, "image"):
35
+ self.vision_encoder = ...
36
+ self.multi_modal_projector = ...
37
+
38
+ with self._mark_language_model(vllm_config):
39
+ self.language_model = init_vllm_registered_model(
40
+ vllm_config=vllm_config,
41
+ hf_config=config.text_config,
42
+ prefix=maybe_prefix(prefix, "language_model"),
43
+ )
44
+ ```
45
46
- Implement [embed_multimodal][vllm.model_executor.models.interfaces.SupportsMultiModal.embed_multimodal] that returns the embeddings from running the multimodal inputs through the multimodal tokenizer of the model. Below we provide a boilerplate of a typical implementation pattern, but feel free to adjust it to your own needs.
47
48
??? code
49
50
```python
44
- class YourModelForImage2Seq(nn.Module):
45
- ...
46
-
51
def _process_image_input(self, image_input: YourModelImageInputs) -> torch.Tensor:
48
- assert self.vision_encoder is not None
52
image_features = self.vision_encoder(image_input)
53
return self.multi_modal_projector(image_features)
54
@@ -71,18 +74,7 @@ Further update the model as follows:
74
[PlaceholderRange][vllm.multimodal.inputs.PlaceholderRange] from input processing.
75
This logic can be found at [embed_input_ids][vllm.model_executor.models.interfaces.SupportsMultiModal.embed_input_ids].
76
74
- You may override this method if additional logic is required for your model when merging embeddings.
75
-
76
-- Implement [get_language_model][vllm.model_executor.models.interfaces.SupportsMultiModal.get_language_model] getter to provide stable access to the underlying language model.
77
-
78
- ```python
79
- class YourModelForImage2Seq(nn.Module):
80
- ...
81
-
82
- def get_language_model(self) -> torch.nn.Module:
83
- # Change `language_model` according to your implementation.
84
- return self.language_model
85
- ```
77
+ You may override this method if additional logic is required for your model when merging embeddings.
78
79
- Once the above steps are done, update the model class with the [SupportsMultiModal][vllm.model_executor.models.interfaces.SupportsMultiModal] interface.
80
examples/online_serving/disaggregated_encoder/README.md
+1
-1
@@ -38,7 +38,7 @@ Encoder engines should be launched with the following flags:
38
39
- `--max-num-batched-tokens=<large value>` **(default: 2048)** – This flag controls the token scheduling budget per decoding step and is irrelevant to encoder-only instances. **Set it to a very high value (effectively unlimited) to bypass scheduler limitations.** The actual token budget is managed by the encoder cache manager.
40
41
-- `--mm-encoder-only` **(Optional)** - The language model is skipped during initialization to reduce device memory usage. **Models using this option must initialize the language component inside the context of `SupportsMultiModal._mark_language_model`.**
41
+- `--mm-encoder-only` **(Optional)** - If possible, skips the language model during initialization to reduce device memory usage.
42
43
## Local media inputs
44