A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
[Doc] Update LLaVA docs (#5437)
Co-authored-by: Roger Wang <ywang@roblox.com>
Cyrus Leung committed
Jun 14, 2024 at 02:22 UTC
0ce7b952f8eafdb13a7b6de3af53157c7aae98d4
3 files changed
+29
-38
docs/source/models/vlm.rst
+2
-2
@@ -20,9 +20,9 @@ The following :ref:`engine arguments <engine_args>` are specific to VLMs:
20
Currently, the support for vision language models on vLLM has the following limitations:
21
22
* Only single image input is supported per text prompt.
23
- * Dynamic ``image_input_shape`` is not supported: the input image will be resized to the static ``image_input_shape``. This means model output might not exactly match the HuggingFace implementation.
23
+ * Dynamic ``image_input_shape`` is not supported: the input image will be resized to the static ``image_input_shape``. This means our LLaVA-NeXT output may not exactly match the huggingface implementation.
24
25
- We are continuously improving user & developer experience for VLMs. Please raise an issue on GitHub if you have any feedback or feature requests.
25
+ We are continuously improving user & developer experience for VLMs. Please `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_ if you have any feedback or feature requests.
26
27
Offline Batched Inference
28
-------------------------
vllm/model_executor/models/llava.py
+16
-13
@@ -227,7 +227,7 @@ class LlavaForConditionalGeneration(VisionLanguageModelBase):
227
attn_metadata: AttentionMetadata,
228
**kwargs: object,
229
) -> SamplerOutput:
230
- """Run forward pass for Llava 1.5.
230
+ """Run forward pass for LLaVA-1.5.
231
232
One key thing to understand is the `input_ids` already accounts for the
233
positions of the to-be-inserted image embeddings.
@@ -247,22 +247,25 @@ class LlavaForConditionalGeneration(VisionLanguageModelBase):
247
This way, the `positions` and `attn_metadata` are consistent
248
with the `input_ids`.
249
250
- The model takes two types of image inputs:
251
- PIXEL_VALUES and IMAGE_FEATURES.
252
- The following shows how each maps to huggingface implementation.
253
- PIXEL_VALUES:
254
- - https://github.com/huggingface/transformers/blob/07bdbeb/src/transformers/models/llava/modeling_llava.py#L353
255
- IMAGE_FEATURES:
256
- - https://github.com/huggingface/transformers/blob/07bdbeb/src/transformers/models/llava/modeling_llava.py#L430
257
- before going through the multi modal projector.
250
+ This model has two modes of image inputs:
251
+ `PIXEL_VALUES` and `IMAGE_FEATURES`.
252
253
Args:
254
input_ids: Flattened (concatenated) input_ids corresponding to a
255
batch.
262
- pixel_values: For PIXEL_VALUES, expects a batch with shape
263
- [1, 3, 336, 336].
264
- image_features: For IMAGE_FEATURES, expects a batch with shape
265
- [1, 576, 1024].
256
+ pixel_values: The pixels in each input image.
257
+ Expects a batch with shape `[1, 3, 336, 336]`.
258
+ (Only applicable to `PIXEL_VALUES` mode)
259
+ image_features: The image features for each input image outputted by
260
+ the vision tower before passing to the multi-modal projector.
261
+ Expects a batch with shape `[1, 576, 1024]`.
262
+ (Only applicable to `IMAGE_FEATURES` mode)
263
+
264
+ See also:
265
+ Each input maps to huggingface implementation, as follows:
266
+
267
+ - `pixel_values`: https://github.com/huggingface/transformers/blob/v4.41.1/src/transformers/models/llava/modeling_llava.py#L360
268
+ - `image_features`: https://github.com/huggingface/transformers/blob/v4.41.1/src/transformers/models/llava/modeling_llava.py#L437
269
"""
270
image_input = self._parse_and_validate_image_input(**kwargs)
271
vllm/model_executor/models/llava_next.py
+11
-23
@@ -108,15 +108,6 @@ def _image_pixel_processor(
108
@MULTIMODAL_REGISTRY.register_image_pixel_input(_image_pixel_processor)
109
@MULTIMODAL_REGISTRY.register_dummy_data(_get_dummy_image_data)
110
class LlavaNextForConditionalGeneration(VisionLanguageModelBase):
111
- """
112
- Args to `forward()`:
113
- input_ids: Flattened (concatenated) input_ids corresponding to a
114
- batch.
115
- pixel_values: For PIXEL_VALUES, expects a batch with shape
116
- [1, num_patches, 3, 336, 336].
117
- image_features: For IMAGE_FEATURES, expects a batch with shape
118
- [1, num_patches, 1176, 1024].
119
- """
111
112
def __init__(self,
113
config: LlavaNextConfig,
@@ -355,7 +346,7 @@ class LlavaNextForConditionalGeneration(VisionLanguageModelBase):
346
attn_metadata: AttentionMetadata,
347
**kwargs: object,
348
) -> SamplerOutput:
358
- """Run forward pass for Llava 1.5.
349
+ """Run forward pass for LlaVA-NeXT.
350
351
One key thing to understand is the `input_ids` already accounts for the
352
positions of the to-be-inserted image embeddings.
@@ -375,22 +366,19 @@ class LlavaNextForConditionalGeneration(VisionLanguageModelBase):
366
This way, the `positions` and `attn_metadata` are consistent
367
with the `input_ids`.
368
378
- The model takes two types of image inputs:
379
- PIXEL_VALUES and IMAGE_FEATURES.
380
- The following shows how each maps to huggingface implementation.
381
- PIXEL_VALUES:
382
- - https://github.com/huggingface/transformers/blob/07bdbeb/src/transformers/models/llava/modeling_llava.py#L353
383
- IMAGE_FEATURES:
384
- - https://github.com/huggingface/transformers/blob/07bdbeb/src/transformers/models/llava/modeling_llava.py#L430
385
- before going through the multi modal projector.
386
-
369
Args:
370
input_ids: Flattened (concatenated) input_ids corresponding to a
371
batch.
390
- pixel_values: For PIXEL_VALUES, expects a batch with shape
391
- [1, 3, 336, 336].
392
- image_features: For IMAGE_FEATURES, expects a batch with shape
393
- [1, 576, 1024].
372
+ pixel_values: The pixels in each grid patch for each input image.
373
+ Expects a batch with shape `[1, num_patches, 3, 336, 336]`.
374
+ image_sizes: The original `(width, height)` for each input image.
375
+ Expects a batch with shape `[1, 2]`.
376
+
377
+ See also:
378
+ Each input maps to huggingface implementation, as follows:
379
+
380
+ - `pixel_values`: https://github.com/huggingface/transformers/blob/v4.41.1/src/transformers/models/llava_next/modeling_llava_next.py#L690
381
+ - `image_sizes`: https://github.com/huggingface/transformers/blob/v4.41.1/src/transformers/models/llava_next/modeling_llava_next.py#L691
382
"""
383
image_input = self._parse_and_validate_image_input(**kwargs)
384