A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 08:30 UTC
[Doc] Reorganize user guide (#18661)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
May 24, 2025 at 22:25 UTC
1cb194a0183db9b51cec6cb9ff473c276d8186de
27 files changed
+211
-216
.github/PULL_REQUEST_TEMPLATE.md
+1
-1
@@ -3,4 +3,4 @@ FILL IN THE PR DESCRIPTION HERE
3
FIX #xxxx (*link existing issues this PR will resolve*)
4
5
<!--- pyml disable-next-line no-emphasis-as-heading -->
6
-**BEFORE SUBMITTING, PLEASE READ <https://docs.vllm.ai/en/latest/contributing/overview.html>** (anything written below this line will be removed by GitHub Actions)
6
+**BEFORE SUBMITTING, PLEASE READ <https://docs.vllm.ai/en/latest/contributing>** (anything written below this line will be removed by GitHub Actions)
CONTRIBUTING.md
+1
-1
@@ -1,3 +1,3 @@
1
# Contributing to vLLM
2
3
-You may find information about contributing to vLLM on [docs.vllm.ai](https://docs.vllm.ai/en/latest/contributing/overview.html).
3
+You may find information about contributing to vLLM on [docs.vllm.ai](https://docs.vllm.ai/en/latest/contributing).
README.md
+1
-1
@@ -100,7 +100,7 @@ Visit our [documentation](https://docs.vllm.ai/en/latest/) to learn more.
100
## Contributing
101
102
We welcome and value any contributions and collaborations.
103
-Please check out [Contributing to vLLM](https://docs.vllm.ai/en/stable/contributing/overview.html) for how to get involved.
103
+Please check out [Contributing to vLLM](https://docs.vllm.ai/en/stable/contributing) for how to get involved.
104
105
## Sponsors
106
docs/.nav.yml
+18
-13
@@ -5,29 +5,35 @@ nav:
5
- getting_started/quickstart.md
6
- getting_started/installation
7
- Examples:
8
- - LMCache: getting_started/examples/lmcache
9
- - getting_started/examples/offline_inference
10
- - getting_started/examples/online_serving
11
- - getting_started/examples/other
8
+ - Offline Inference: getting_started/examples/offline_inference
9
+ - Online Serving: getting_started/examples/online_serving
10
+ - Others:
11
+ - LMCache: getting_started/examples/lmcache
12
+ - getting_started/examples/other/*
13
- Quick Links:
13
- - User Guide: serving/offline_inference.md
14
- - Developer Guide: contributing/overview.md
14
+ - User Guide: usage/README.md
15
+ - Developer Guide: contributing/README.md
16
- API Reference: api/README.md
17
- Timeline:
18
- Roadmap: https://roadmap.vllm.ai
19
- Releases: https://github.com/vllm-project/vllm/releases
20
- User Guide:
21
+ - usage/README.md
22
+ - General:
23
+ - usage/*
24
- Inference and Serving:
25
- serving/offline_inference.md
26
- serving/openai_compatible_server.md
27
- serving/*
28
- serving/integrations
25
- - Training: training
29
- Deployment:
30
- deployment/*
31
- deployment/frameworks
32
- deployment/integrations
30
- - Performance: performance
33
+ - Training: training
34
+ - Configuration:
35
+ - Summary: configuration/README.md
36
+ - configuration/*
37
- Models:
38
- models/supported_models.md
39
- models/generative_models.md
@@ -37,12 +43,11 @@ nav:
43
- features/compatibility_matrix.md
44
- features/*
45
- features/quantization
40
- - Other:
41
- - getting_started/*
46
- Developer Guide:
43
- - contributing/overview.md
44
- - glob: contributing/*
45
- flatten_single_child_sections: true
47
+ - contributing/README.md
48
+ - General:
49
+ - glob: contributing/*
50
+ flatten_single_child_sections: true
51
- Model Implementation: contributing/model
52
- Design Documents:
53
- V0: design
docs/configuration/README.md
new
+4
@@ -0,0 +1,4 @@
1
+# Configuration Options
2
+
3
+This section lists the most common options for running the vLLM engine.
4
+For a full list, refer to the [configuration][configuration] page.
docs/configuration/conserving_memory.md
new
+144
@@ -0,0 +1,144 @@
1
+# Conserving Memory
2
+
3
+Large models might cause your machine to run out of memory (OOM). Here are some options that help alleviate this problem.
4
+
5
+## Tensor Parallelism (TP)
6
+
7
+Tensor parallelism (`tensor_parallel_size` option) can be used to split the model across multiple GPUs.
8
+
9
+The following code splits the model across 2 GPUs.
10
+
11
+```python
12
+from vllm import LLM
13
+
14
+llm = LLM(model="ibm-granite/granite-3.1-8b-instruct",
15
+ tensor_parallel_size=2)
16
+```
17
+
18
+!!! warning
19
+ To ensure that vLLM initializes CUDA correctly, you should avoid calling related functions (e.g. [torch.cuda.set_device][])
20
+ before initializing vLLM. Otherwise, you may run into an error like `RuntimeError: Cannot re-initialize CUDA in forked subprocess`.
21
+
22
+ To control which devices are used, please instead set the `CUDA_VISIBLE_DEVICES` environment variable.
23
+
24
+!!! note
25
+ With tensor parallelism enabled, each process will read the whole model and split it into chunks, which makes the disk reading time even longer (proportional to the size of tensor parallelism).
26
+
27
+ You can convert the model checkpoint to a sharded checkpoint using <gh-file:examples/offline_inference/save_sharded_state.py>. The conversion process might take some time, but later you can load the sharded checkpoint much faster. The model loading time should remain constant regardless of the size of tensor parallelism.
28
+
29
+## Quantization
30
+
31
+Quantized models take less memory at the cost of lower precision.
32
+
33
+Statically quantized models can be downloaded from HF Hub (some popular ones are available at [Red Hat AI](https://huggingface.co/RedHatAI))
34
+and used directly without extra configuration.
35
+
36
+Dynamic quantization is also supported via the `quantization` option -- see [here][quantization-index] for more details.
37
+
38
+## Context length and batch size
39
+
40
+You can further reduce memory usage by limiting the context length of the model (`max_model_len` option)
41
+and the maximum batch size (`max_num_seqs` option).
42
+
43
+```python
44
+from vllm import LLM
45
+
46
+llm = LLM(model="adept/fuyu-8b",
47
+ max_model_len=2048,
48
+ max_num_seqs=2)
49
+```
50
+
51
+## Reduce CUDA Graphs
52
+
53
+By default, we optimize model inference using CUDA graphs which take up extra memory in the GPU.
54
+
55
+!!! warning
56
+ CUDA graph capture takes up more memory in V1 than in V0.
57
+
58
+You can adjust `compilation_config` to achieve a better balance between inference speed and memory usage:
59
+
60
+```python
61
+from vllm import LLM
62
+from vllm.config import CompilationConfig, CompilationLevel
63
+
64
+llm = LLM(
65
+ model="meta-llama/Llama-3.1-8B-Instruct",
66
+ compilation_config=CompilationConfig(
67
+ level=CompilationLevel.PIECEWISE,
68
+ # By default, it goes up to max_num_seqs
69
+ cudagraph_capture_sizes=[1, 2, 4, 8, 16],
70
+ ),
71
+)
72
+```
73
+
74
+You can disable graph capturing completely via the `enforce_eager` flag:
75
+
76
+```python
77
+from vllm import LLM
78
+
79
+llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct",
80
+ enforce_eager=True)
81
+```
82
+
83
+## Adjust cache size
84
+
85
+If you run out of CPU RAM, try the following options:
86
+
87
+- (Multi-modal models only) you can set the size of multi-modal input cache using `VLLM_MM_INPUT_CACHE_GIB` environment variable (default 4 GiB).
88
+- (CPU backend only) you can set the size of KV cache using `VLLM_CPU_KVCACHE_SPACE` environment variable (default 4 GiB).
89
+
90
+## Multi-modal input limits
91
+
92
+You can allow a smaller number of multi-modal items per prompt to reduce the memory footprint of the model:
93
+
94
+```python
95
+from vllm import LLM
96
+
97
+# Accept up to 3 images and 1 video per prompt
98
+llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
99
+ limit_mm_per_prompt={"image": 3, "video": 1})
100
+```
101
+
102
+You can go a step further and disable unused modalities completely by setting its limit to zero.
103
+For example, if your application only accepts image input, there is no need to allocate any memory for videos.
104
+
105
+```python
106
+from vllm import LLM
107
+
108
+# Accept any number of images but no videos
109
+llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
110
+ limit_mm_per_prompt={"video": 0})
111
+```
112
+
113
+You can even run a multi-modal model for text-only inference:
114
+
115
+```python
116
+from vllm import LLM
117
+
118
+# Don't accept images. Just text.
119
+llm = LLM(model="google/gemma-3-27b-it",
120
+ limit_mm_per_prompt={"image": 0})
121
+```
122
+
123
+## Multi-modal processor arguments
124
+
125
+For certain models, you can adjust the multi-modal processor arguments to
126
+reduce the size of the processed multi-modal inputs, which in turn saves memory.
127
+
128
+Here are some examples:
129
+
130
+```python
131
+from vllm import LLM
132
+
133
+# Available for Qwen2-VL series models
134
+llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
135
+ mm_processor_kwargs={
136
+ "max_pixels": 768 * 768, # Default is 1280 * 28 * 28
137
+ })
138
+
139
+# Available for InternVL series models
140
+llm = LLM(model="OpenGVLab/InternVL2-2B",
141
+ mm_processor_kwargs={
142
+ "max_dynamic_patch": 4, # Default is 12
143
+ })
144
+```
docs/configuration/engine_args.md
renamed
docs/configuration/model_resolution.md
new
+23
@@ -0,0 +1,23 @@
1
+# Model Resolution
2
+
3
+vLLM loads HuggingFace-compatible models by inspecting the `architectures` field in `config.json` of the model repository
4
+and finding the corresponding implementation that is registered to vLLM.
5
+Nevertheless, our model resolution may fail for the following reasons:
6
+
7
+- The `config.json` of the model repository lacks the `architectures` field.
8
+- Unofficial repositories refer to a model using alternative names which are not recorded in vLLM.
9
+- The same architecture name is used for multiple models, creating ambiguity as to which model should be loaded.
10
+
11
+To fix this, explicitly specify the model architecture by passing `config.json` overrides to the `hf_overrides` option.
12
+For example:
13
+
14
+```python
15
+from vllm import LLM
16
+
17
+model = LLM(
18
+ model="cerebras/Cerebras-GPT-1.3B",
19
+ hf_overrides={"architectures": ["GPT2LMHeadModel"]}, # GPT-2
20
+)
21
+```
22
+
23
+Our [list of supported models][supported-models] shows the model architectures that are recognized by vLLM.
docs/configuration/optimization.md
renamed
+1
-4
@@ -1,7 +1,4 @@
1
----
2
-title: Optimization and Tuning
3
----
4
-[](){ #optimization-and-tuning }
1
+# Optimization and Tuning
2
3
This guide covers optimization strategies and performance tuning for vLLM V1.
4
docs/configuration/serve_args.md
renamed
docs/contributing/README.md
renamed
docs/contributing/benchmarks.md
renamed
docs/design/multiprocessing.md
+1
-1
@@ -123,7 +123,7 @@ what is happening. First, a log message from vLLM:
123
WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
124
initialized. We must use the `spawn` multiprocessing start method. Setting
125
VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
126
- https://docs.vllm.ai/en/latest/getting_started/debugging.html#python-multiprocessing
126
+ https://docs.vllm.ai/en/latest/usage/debugging.html#python-multiprocessing
127
for more information.
128
```
129
docs/design/v1/metrics.md
+1
-1
@@ -57,7 +57,7 @@ In v0, the following metrics are exposed via a Prometheus-compatible `/metrics`
57
- `vllm:spec_decode_num_draft_tokens_total` (Counter)
58
- `vllm:spec_decode_num_emitted_tokens_total` (Counter)
59
60
-These are documented under [Inferencing and Serving -> Production Metrics](../../serving/metrics.md).
60
+These are documented under [Inferencing and Serving -> Production Metrics](../../usage/metrics.md).
61
62
### Grafana Dashboard
63
docs/features/tool_calling.md
+1
-1
@@ -93,7 +93,7 @@ specify the `name` of one of the tools in the `tool_choice` parameter of the cha
93
94
## Required Function Calling
95
96
-vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses guided decoding, so this is enabled by default and will work with any supported model. The required guided decoding features (JSON schema with `anyOf`) are currently only supported in the V0 engine with the guided decoding backend `outlines`. However, support for alternative decoding backends are on the [roadmap](https://docs.vllm.ai/en/latest/getting_started/v1_user_guide.html#feature-model) for the V1 engine.
96
+vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses guided decoding, so this is enabled by default and will work with any supported model. The required guided decoding features (JSON schema with `anyOf`) are currently only supported in the V0 engine with the guided decoding backend `outlines`. However, support for alternative decoding backends are on the [roadmap](https://docs.vllm.ai/en/latest/usage/v1_guide.html#feature-model) for the V1 engine.
97
98
When tool_choice='required' is set, the model is guaranteed to generate one or more tool calls based on the specified tool list in the `tools` parameter. The number of tool calls depends on the user's query. The output format strictly follows the schema defined in the `tools` parameter.
99
docs/serving/offline_inference.md
-185
@@ -27,188 +27,3 @@ Please refer to the above pages for more details about each API.
27
28
!!! info
29
[API Reference][offline-inference-api]
30
-
31
-[](){ #configuration-options }
32
-
33
-## Configuration Options
34
-
35
-This section lists the most common options for running the vLLM engine.
36
-For a full list, refer to the [configuration][configuration] page.
37
-
38
-[](){ #model-resolution }
39
-
40
-### Model resolution
41
-
42
-vLLM loads HuggingFace-compatible models by inspecting the `architectures` field in `config.json` of the model repository
43
-and finding the corresponding implementation that is registered to vLLM.
44
-Nevertheless, our model resolution may fail for the following reasons:
45
-
46
-- The `config.json` of the model repository lacks the `architectures` field.
47
-- Unofficial repositories refer to a model using alternative names which are not recorded in vLLM.
48
-- The same architecture name is used for multiple models, creating ambiguity as to which model should be loaded.
49
-
50
-To fix this, explicitly specify the model architecture by passing `config.json` overrides to the `hf_overrides` option.
51
-For example:
52
-
53
-```python
54
-from vllm import LLM
55
-
56
-model = LLM(
57
- model="cerebras/Cerebras-GPT-1.3B",
58
- hf_overrides={"architectures": ["GPT2LMHeadModel"]}, # GPT-2
59
-)
60
-```
61
-
62
-Our [list of supported models][supported-models] shows the model architectures that are recognized by vLLM.
63
-
64
-[](){ #reducing-memory-usage }
65
-
66
-### Reducing memory usage
67
-
68
-Large models might cause your machine to run out of memory (OOM). Here are some options that help alleviate this problem.
69
-
70
-#### Tensor Parallelism (TP)
71
-
72
-Tensor parallelism (`tensor_parallel_size` option) can be used to split the model across multiple GPUs.
73
-
74
-The following code splits the model across 2 GPUs.
75
-
76
-```python
77
-from vllm import LLM
78
-
79
-llm = LLM(model="ibm-granite/granite-3.1-8b-instruct",
80
- tensor_parallel_size=2)
81
-```
82
-
83
-!!! warning
84
- To ensure that vLLM initializes CUDA correctly, you should avoid calling related functions (e.g. [torch.cuda.set_device][])
85
- before initializing vLLM. Otherwise, you may run into an error like `RuntimeError: Cannot re-initialize CUDA in forked subprocess`.
86
-
87
- To control which devices are used, please instead set the `CUDA_VISIBLE_DEVICES` environment variable.
88
-
89
-!!! note
90
- With tensor parallelism enabled, each process will read the whole model and split it into chunks, which makes the disk reading time even longer (proportional to the size of tensor parallelism).
91
-
92
- You can convert the model checkpoint to a sharded checkpoint using <gh-file:examples/offline_inference/save_sharded_state.py>. The conversion process might take some time, but later you can load the sharded checkpoint much faster. The model loading time should remain constant regardless of the size of tensor parallelism.
93
-
94
-#### Quantization
95
-
96
-Quantized models take less memory at the cost of lower precision.
97
-
98
-Statically quantized models can be downloaded from HF Hub (some popular ones are available at [Red Hat AI](https://huggingface.co/RedHatAI))
99
-and used directly without extra configuration.
100
-
101
-Dynamic quantization is also supported via the `quantization` option -- see [here][quantization-index] for more details.
102
-
103
-#### Context length and batch size
104
-
105
-You can further reduce memory usage by limiting the context length of the model (`max_model_len` option)
106
-and the maximum batch size (`max_num_seqs` option).
107
-
108
-```python
109
-from vllm import LLM
110
-
111
-llm = LLM(model="adept/fuyu-8b",
112
- max_model_len=2048,
113
- max_num_seqs=2)
114
-```
115
-
116
-#### Reduce CUDA Graphs
117
-
118
-By default, we optimize model inference using CUDA graphs which take up extra memory in the GPU.
119
-
120
-!!! warning
121
- CUDA graph capture takes up more memory in V1 than in V0.
122
-
123
-You can adjust `compilation_config` to achieve a better balance between inference speed and memory usage:
124
-
125
-```python
126
-from vllm import LLM
127
-from vllm.config import CompilationConfig, CompilationLevel
128
-
129
-llm = LLM(
130
- model="meta-llama/Llama-3.1-8B-Instruct",
131
- compilation_config=CompilationConfig(
132
- level=CompilationLevel.PIECEWISE,
133
- # By default, it goes up to max_num_seqs
134
- cudagraph_capture_sizes=[1, 2, 4, 8, 16],
135
- ),
136
-)
137
-```
138
-
139
-You can disable graph capturing completely via the `enforce_eager` flag:
140
-
141
-```python
142
-from vllm import LLM
143
-
144
-llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct",
145
- enforce_eager=True)
146
-```
147
-
148
-#### Adjust cache size
149
-
150
-If you run out of CPU RAM, try the following options:
151
-
152
-- (Multi-modal models only) you can set the size of multi-modal input cache using `VLLM_MM_INPUT_CACHE_GIB` environment variable (default 4 GiB).
153
-- (CPU backend only) you can set the size of KV cache using `VLLM_CPU_KVCACHE_SPACE` environment variable (default 4 GiB).
154
-
155
-#### Multi-modal input limits
156
-
157
-You can allow a smaller number of multi-modal items per prompt to reduce the memory footprint of the model:
158
-
159
-```python
160
-from vllm import LLM
161
-
162
-# Accept up to 3 images and 1 video per prompt
163
-llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
164
- limit_mm_per_prompt={"image": 3, "video": 1})
165
-```
166
-
167
-You can go a step further and disable unused modalities completely by setting its limit to zero.
168
-For example, if your application only accepts image input, there is no need to allocate any memory for videos.
169
-
170
-```python
171
-from vllm import LLM
172
-
173
-# Accept any number of images but no videos
174
-llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
175
- limit_mm_per_prompt={"video": 0})
176
-```
177
-
178
-You can even run a multi-modal model for text-only inference:
179
-
180
-```python
181
-from vllm import LLM
182
-
183
-# Don't accept images. Just text.
184
-llm = LLM(model="google/gemma-3-27b-it",
185
- limit_mm_per_prompt={"image": 0})
186
-```
187
-
188
-#### Multi-modal processor arguments
189
-
190
-For certain models, you can adjust the multi-modal processor arguments to
191
-reduce the size of the processed multi-modal inputs, which in turn saves memory.
192
-
193
-Here are some examples:
194
-
195
-```python
196
-from vllm import LLM
197
-
198
-# Available for Qwen2-VL series models
199
-llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
200
- mm_processor_kwargs={
201
- "max_pixels": 768 * 768, # Default is 1280 * 28 * 28
202
- })
203
-
204
-# Available for InternVL series models
205
-llm = LLM(model="OpenGVLab/InternVL2-2B",
206
- mm_processor_kwargs={
207
- "max_dynamic_patch": 4, # Default is 12
208
- })
209
-```
210
-
211
-### Performance optimization and tuning
212
-
213
-You can potentially improve the performance of vLLM by finetuning various options.
214
-Please refer to [this guide][optimization-and-tuning] for more details.
docs/usage/README.md
new
+7
@@ -0,0 +1,7 @@
1
+# Using vLLM
2
+
3
+vLLM supports the following usage patterns:
4
+
5
+- [Inference and Serving](../serving/offline_inference.md): Run a single instance of a model.
6
+- [Deployment](../deployment/docker.md): Scale up model instances for production.
7
+- [Training](../training/rlhf.md): Train or fine-tune a model.
docs/usage/env_vars.md
renamed
docs/usage/faq.md
renamed
docs/usage/metrics.md
renamed
docs/usage/reproducibility.md
renamed
+1
-1
@@ -1,4 +1,4 @@
1
-# Seed Parameter Behavior
1
+# Reproducibility
2
3
## Overview
4
docs/usage/security.md
renamed
+1
-1
@@ -1,4 +1,4 @@
1
-# Security Guide
1
+# Security
2
3
## Inter-Node Communication
4
docs/usage/troubleshooting.md
renamed
+3
-3
@@ -23,7 +23,7 @@ It'd be better to store the model in a local disk. Additionally, have a look at
23
24
## Out of memory
25
26
-If the model is too large to fit in a single GPU, you will get an out-of-memory (OOM) error. Consider adopting [these options][reducing-memory-usage] to reduce the memory consumption.
26
+If the model is too large to fit in a single GPU, you will get an out-of-memory (OOM) error. Consider adopting [these options](../configuration/conserving_memory.md) to reduce the memory consumption.
27
28
## Generation quality changed
29
@@ -159,7 +159,7 @@ If you have seen a warning in your logs like this:
159
WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
160
initialized. We must use the `spawn` multiprocessing start method. Setting
161
VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
162
- https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html#python-multiprocessing
162
+ https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing
163
for more information.
164
```
165
@@ -258,7 +258,7 @@ or:
258
ValueError: Model architectures ['<arch>'] are not supported for now. Supported architectures: [...]
259
```
260
261
-But you are sure that the model is in the [list of supported models][supported-models], there may be some issue with vLLM's model resolution. In that case, please follow [these steps][model-resolution] to explicitly specify the vLLM implementation for the model.
261
+But you are sure that the model is in the [list of supported models][supported-models], there may be some issue with vLLM's model resolution. In that case, please follow [these steps](../configuration/model_resolution.md) to explicitly specify the vLLM implementation for the model.
262
263
## Failed to infer device type
264
docs/usage/usage_stats.md
renamed
docs/usage/v1_guide.md
renamed
+1
-1
@@ -1,4 +1,4 @@
1
-# vLLM V1 User Guide
1
+# vLLM V1
2
3
V1 is now enabled by default for all supported use cases, and we will gradually enable it for every use case we plan to support. Please share any feedback on [GitHub](https://github.com/vllm-project/vllm) or in the [vLLM Slack](https://inviter.co/vllm-slack).
4
vllm/envs.py
+1
-1
@@ -164,7 +164,7 @@ def get_vllm_port() -> Optional[int]:
164
raise ValueError(
165
f"VLLM_PORT '{port}' appears to be a URI. "
166
"This may be caused by a Kubernetes service discovery issue"
167
- "check the warning in: https://docs.vllm.ai/en/stable/serving/env_vars.html"
167
+ "check the warning in: https://docs.vllm.ai/en/stable/usage/env_vars.html"
168
)
169
except Exception:
170
pass
vllm/utils.py
+1
-1
@@ -2531,7 +2531,7 @@ def _maybe_force_spawn():
2531
logger.warning(
2532
"We must use the `spawn` multiprocessing start method. "
2533
"Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. "
2534
- "See https://docs.vllm.ai/en/latest/getting_started/"
2534
+ "See https://docs.vllm.ai/en/latest/usage/"
2535
"troubleshooting.html#python-multiprocessing "
2536
"for more information. Reason: %s", reason)
2537
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"