@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 185 lines 6.74 KB
Rendered Raw
1 # Conserving Memory
2
3 Large models might cause your machine to run out of memory (OOM). Here are some options that help alleviate this problem.
4
5 ## Tensor Parallelism (TP)
6
7 Tensor parallelism (`tensor_parallel_size` option) can be used to split the model across multiple GPUs.
8
9 The following code splits the model across 2 GPUs.
10
11 ```python
12 from vllm import LLM
13
14 llm = LLM(model="ibm-granite/granite-3.1-8b-instruct", tensor_parallel_size=2)
15 ```
16
17 !!! warning
18 To ensure that vLLM initializes CUDA correctly, you should avoid calling related functions (e.g. [torch.accelerator.set_device_index][])
19 before initializing vLLM. Otherwise, you may run into an error like `RuntimeError: Cannot re-initialize CUDA in forked subprocess`.
20
21 To control which devices are used, please instead set the `CUDA_VISIBLE_DEVICES` environment variable.
22
23 !!! note
24 With tensor parallelism enabled, each process will read the whole model and split it into chunks, which makes the disk reading time even longer (proportional to the size of tensor parallelism).
25
26 You can convert the model checkpoint to a sharded checkpoint using [examples/features/sharded_state/load_sharded_state_offline.py](../../examples/features/sharded_state/load_sharded_state_offline.py). The conversion process might take some time, but later you can load the sharded checkpoint much faster. The model loading time should remain constant regardless of the size of tensor parallelism.
27
28 ## Quantization
29
30 Quantized models take less memory at the cost of lower precision.
31
32 Statically quantized models can be downloaded from HF Hub (some popular ones are available at [Red Hat AI](https://huggingface.co/RedHatAI))
33 and used directly without extra configuration.
34
35 Dynamic quantization is also supported via the `quantization` option -- see [here](../features/quantization/README.md) for more details.
36
37 ## Context length and batch size
38
39 You can further reduce memory usage by limiting the context length of the model (`max_model_len` option)
40 and the maximum batch size (`max_num_seqs` option).
41
42 ```python
43 from vllm import LLM
44
45 llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct", max_model_len=2048, max_num_seqs=2)
46 ```
47
48 ## Reduce CUDA Graphs
49
50 By default, we optimize model inference using CUDA graphs which take up extra memory in the GPU.
51
52 You can adjust `compilation_config` to achieve a better balance between inference speed and memory usage:
53
54 ??? code
55
56 ```python
57 from vllm import LLM
58 from vllm.config import CompilationConfig, CompilationMode
59
60 llm = LLM(
61 model="meta-llama/Llama-3.1-8B-Instruct",
62 compilation_config=CompilationConfig(
63 mode=CompilationMode.VLLM_COMPILE,
64 # By default, it goes up to max_num_seqs
65 cudagraph_capture_sizes=[1, 2, 4, 8, 16],
66 ),
67 )
68 ```
69
70 You can disable graph capturing completely via the `enforce_eager` flag:
71
72 ```python
73 from vllm import LLM
74
75 llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", enforce_eager=True)
76 ```
77
78 ## Adjust cache size
79
80 If you run out of CPU RAM, try the following options:
81
82 - (Multi-modal models only) you can set the size of multi-modal cache by setting `mm_processor_cache_gb` engine argument (default 4 GiB).
83 - (CPU backend only) you can set the size of KV cache using `VLLM_CPU_KVCACHE_SPACE` environment variable (default 4 GiB).
84
85 ## Multi-modal input limits
86
87 You can allow a smaller number of multi-modal items per prompt to reduce the memory footprint of the model:
88
89 ```python
90 from vllm import LLM
91
92 # Accept up to 3 images and 1 video per prompt
93 llm = LLM(
94 model="Qwen/Qwen2.5-VL-3B-Instruct",
95 limit_mm_per_prompt={"image": 3, "video": 1},
96 )
97 ```
98
99 You can go a step further and disable unused modalities completely by setting its limit to zero.
100 For example, if your application only accepts image input, there is no need to allocate any memory for videos.
101
102 ```python
103 from vllm import LLM
104
105 # Accept any number of images but no videos
106 llm = LLM(
107 model="Qwen/Qwen2.5-VL-3B-Instruct",
108 limit_mm_per_prompt={"video": 0},
109 )
110 ```
111
112 You can even run a multi-modal model for text-only inference:
113
114 ```python
115 from vllm import LLM
116
117 # Don't accept images. Just text.
118 llm = LLM(
119 model="google/gemma-3-27b-it",
120 limit_mm_per_prompt={"image": 0},
121 )
122 ```
123
124 ### Configurable options
125
126 `limit_mm_per_prompt` also accepts configurable options per modality. In the configurable form, you still specify `count`, and you may optionally provide size hints that control how vLLM profiles and reserves memory for your multi‑modal inputs. This helps you tune memory for the actual media you expect, instead of the model’s absolute maxima.
127
128 Configurable options by modality:
129
130 - `image`: `{"count": int, "width": int, "height": int}`
131 - `video`: `{"count": int, "num_frames": int, "width": int, "height": int}`
132 - `audio`: `{"count": int, "length": int}`
133
134 Details could be found in [`ImageDummyOptions`][vllm.config.multimodal.ImageDummyOptions], [`VideoDummyOptions`][vllm.config.multimodal.VideoDummyOptions], and [`AudioDummyOptions`][vllm.config.multimodal.AudioDummyOptions].
135
136 Examples:
137
138 ```python
139 from vllm import LLM
140
141 # Up to 5 images per prompt, profile with 512x512.
142 # Up to 1 video per prompt, profile with 32 frames at 640x640.
143 llm = LLM(
144 model="Qwen/Qwen2.5-VL-3B-Instruct",
145 limit_mm_per_prompt={
146 "image": {"count": 5, "width": 512, "height": 512},
147 "video": {"count": 1, "num_frames": 32, "width": 640, "height": 640},
148 },
149 )
150 ```
151
152 For backward compatibility, passing an integer works as before and is interpreted as `{"count": <int>}`. For example:
153
154 - `limit_mm_per_prompt={"image": 5}` is equivalent to `limit_mm_per_prompt={"image": {"count": 5}}`
155 - You can mix formats: `limit_mm_per_prompt={"image": 5, "video": {"count": 1, "num_frames": 32, "width": 640, "height": 640}}`
156
157 !!! note
158 - The size hints affect memory profiling only. They shape the dummy inputs used to compute reserved activation sizes. They do not change how inputs are actually processed at inference time.
159 - If a hint exceeds what the model can accept, vLLM clamps it to the model's effective maximum and may log a warning.
160
161 !!! warning
162 These size hints currently only affect activation memory profiling. Encoder cache size is determined by the actual inputs at runtime and is not limited by these hints.
163
164 ## Multi-modal processor arguments
165
166 For certain models, you can adjust the multi-modal processor arguments to
167 reduce the size of the processed multi-modal inputs, which in turn saves memory.
168
169 Here are some examples:
170
171 ```python
172 from vllm import LLM
173
174 # Available for Qwen2-VL series models
175 llm = LLM(
176 model="Qwen/Qwen2.5-VL-3B-Instruct",
177 mm_processor_kwargs={"max_pixels": 768 * 768}, # Default is 1280 * 28 * 28
178 )
179
180 # Available for InternVL series models
181 llm = LLM(
182 model="OpenGVLab/InternVL2-2B",
183 mm_processor_kwargs={"max_dynamic_patch": 4}, # Default is 12
184 )
185 ```