@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 246598a6b1
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

[CI] docfix (#5410)

Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk> Co-authored-by: ywang96 <ywang@roblox.com>

SangBin Cho committed Jun 11, 2024 at 17:28 UTC 246598a6b1e22616630b7f1bf11bd9bcb31dc860
2 files changed +12 -7
docs/source/models/vlm.rst
+2 -1
@@ -20,7 +20,8 @@ The following :ref:`engine arguments <engine_args>` are specific to VLMs:
20 Currently, the support for vision language models on vLLM has the following limitations:
21
22 * Only single image input is supported per text prompt.
23 - * Dynamic ``image_input_shape`` is not supported: the input image will be resized to the static ``image_input_shape``. This means model output might not exactly match the huggingface implementation.
23 + * Dynamic ``image_input_shape`` is not supported: the input image will be resized to the static ``image_input_shape``. This means model output might not exactly match the HuggingFace implementation.
24 +
25 We are continuously improving user & developer experience for VLMs. Please raise an issue on GitHub if you have any feedback or feature requests.
26
27 Offline Batched Inference
docs/source/quantization/fp8.rst
+10 -6
@@ -13,7 +13,7 @@ The FP8 types typically supported in hardware have two distinct representations,
13 - **E5M2**: Consists of 1 sign bit, 5 exponent bits, and 2 bits of mantissa. It can store values up to +/-57344, +/- ``inf``, and ``nan``. The tradeoff for the increased dynamic range is lower precision of the stored values.
14
15 Quick Start with Online Dynamic Quantization
16 --------------------------------------
16 +--------------------------------------------
17
18 Dynamic quantization of an original precision BF16/FP16 model to FP8 can be achieved with vLLM without any calibration data required. You can enable the feature by specifying ``--quantization="fp8"`` in the command line or setting ``quantization="fp8"`` in the LLM constructor.
19
@@ -173,25 +173,28 @@ Here we detail the structure for the FP8 checkpoints.
173
174 The following is necessary to be present in the model's ``config.json``:
175
176 -.. code-block:: yaml
176 +.. code-block:: text
177 +
178 "quantization_config": {
179 "quant_method": "fp8",
180 "activation_scheme": "static" or "dynamic"
180 - },
181 + }
182
183
184 Each quantized layer in the state_dict will have these tensors:
185
185 -* If the config has `"activation_scheme": "static"`:
186 +* If the config has ``"activation_scheme": "static"``:
187
188 .. code-block:: text
189 +
190 model.layers.0.mlp.down_proj.weight < F8_E4M3
191 model.layers.0.mlp.down_proj.input_scale < F32
192 model.layers.0.mlp.down_proj.weight_scale < F32
193
192 -* If the config has `"activation_scheme": "dynamic"`:
194 +* If the config has ``"activation_scheme": "dynamic"``:
195
196 .. code-block:: text
197 +
198 model.layers.0.mlp.down_proj.weight < F8_E4M3
199 model.layers.0.mlp.down_proj.weight_scale < F32
200
@@ -199,4 +202,5 @@ Each quantized layer in the state_dict will have these tensors:
202 Additionally, there can be `FP8 kv-cache scaling factors <https://github.com/vllm-project/vllm/pull/4893>`_ contained within quantized checkpoints specified through the ``.kv_scale`` parameter present on the Attention Module, such as:
203
204 .. code-block:: text
202 - model.layers.0.self_attn.kv_scale < F32
\ No newline at end of file
205 +
206 + model.layers.0.self_attn.kv_scale < F32