A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 14:45 UTC
[Doc] Add more tips to avoid OOM (#16765)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Apr 17, 2025 at 17:54 UTC
61a44a0b22a9264ea4d123275981b79726f913e7
2 files changed
+33
docs/source/serving/offline_inference.md
+25
@@ -28,6 +28,8 @@ Please refer to the above pages for more details about each API.
28
[API Reference](/api/offline_inference/index)
29
:::
30
31
+(configuration-options)=
32
+
33
## Configuration Options
34
35
This section lists the most common options for running the vLLM engine.
@@ -184,6 +186,29 @@ llm = LLM(model="google/gemma-3-27b-it",
186
limit_mm_per_prompt={"image": 0})
187
```
188
189
+#### Multi-modal processor arguments
190
+
191
+For certain models, you can adjust the multi-modal processor arguments to
192
+reduce the size of the processed multi-modal inputs, which in turn saves memory.
193
+
194
+Here are some examples:
195
+
196
+```python
197
+from vllm import LLM
198
+
199
+# Available for Qwen2-VL series models
200
+llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
201
+ mm_processor_kwargs={
202
+ "max_pixels": 768 * 768, # Default is 1280 * 28 * 28
203
+ })
204
+
205
+# Available for InternVL series models
206
+llm = LLM(model="OpenGVLab/InternVL2-2B",
207
+ mm_processor_kwargs={
208
+ "max_dynamic_patch": 4, # Default is 12
209
+ })
210
+```
211
+
212
### Performance optimization and tuning
213
214
You can potentially improve the performance of vLLM by finetuning various options.
docs/source/serving/openai_compatible_server.md
+8
@@ -33,11 +33,13 @@ print(completion.choices[0].message)
33
vLLM supports some parameters that are not supported by OpenAI, `top_k` for example.
34
You can pass these parameters to vLLM using the OpenAI client in the `extra_body` parameter of your requests, i.e. `extra_body={"top_k": 50}` for `top_k`.
35
:::
36
+
37
:::{important}
38
By default, the server applies `generation_config.json` from the Hugging Face model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator.
39
40
To disable this behavior, please pass `--generation-config vllm` when launching the server.
41
:::
42
+
43
## Supported APIs
44
45
We currently support the following OpenAI APIs:
@@ -172,6 +174,12 @@ print(completion._request_id)
174
175
The `vllm serve` command is used to launch the OpenAI-compatible server.
176
177
+:::{tip}
178
+The vast majority of command-line arguments are based on those for offline inference.
179
+
180
+See [here](configuration-options) for some common options.
181
+:::
182
+
183
:::{argparse}
184
:module: vllm.entrypoints.openai.cli_args
185
:func: create_parser_for_docs