A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 02:00 UTC
[CI/Build] Add markdown linter (#11857)
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
Rafael Vasquez committed
Jan 12, 2025 at 03:17 UTC
43f3d9e6990811461ecb42bc50a17aad944d30f9
49 files changed
+596
-571
.github/workflows/doc-lint.yml
renamed
+2
-2
@@ -13,7 +13,7 @@ on:
13
- "docs/**"
14
15
jobs:
16
- sphinx-lint:
16
+ doc-lint:
17
runs-on: ubuntu-latest
18
strategy:
19
matrix:
@@ -29,4 +29,4 @@ jobs:
29
python -m pip install --upgrade pip
30
pip install -r requirements-lint.txt
31
- name: Linting docs
32
- run: tools/sphinx-lint.sh
32
+ run: tools/doc-lint.sh
docs/README.md
+1
@@ -16,4 +16,5 @@ make html
16
```bash
17
python -m http.server -d build/html/
18
```
19
+
20
Launch your browser and open localhost:8000.
docs/source/api/model/index.md
-1
@@ -9,4 +9,3 @@ interfaces_base
9
interfaces
10
adapters
11
```
12
-
docs/source/community/sponsors.md
+2
@@ -6,6 +6,7 @@ vLLM is a community project. Our compute resources for development and testing a
6
<!-- Note: Please keep these consistent with README.md. -->
7
8
Cash Donations:
9
+
10
- a16z
11
- Dropbox
12
- Sequoia Capital
@@ -13,6 +14,7 @@ Cash Donations:
14
- ZhenFund
15
16
Compute Resources:
17
+
18
- AMD
19
- Anyscale
20
- AWS
docs/source/contributing/model/multimodal.md
+4
@@ -200,6 +200,7 @@ def get_mm_max_tokens_per_item(self, seq_len: int) -> Mapping[str, int]:
200
```{note}
201
Our [actual code](gh-file:vllm/model_executor/models/llava.py) is more abstracted to support vision encoders other than CLIP.
202
```
203
+
204
:::
205
::::
206
@@ -248,6 +249,7 @@ def get_dummy_processor_inputs(
249
mm_data=mm_data,
250
)
251
```
252
+
253
:::
254
::::
255
@@ -312,6 +314,7 @@ def _get_mm_fields_config(
314
Our [actual code](gh-file:vllm/model_executor/models/llava.py) additionally supports
315
pre-computed image embeddings, which can be passed to be model via the `image_embeds` argument.
316
```
317
+
318
:::
319
::::
320
@@ -369,6 +372,7 @@ def _get_prompt_replacements(
372
),
373
]
374
```
375
+
376
:::
377
::::
378
docs/source/contributing/overview.md
-2
@@ -37,8 +37,6 @@ pytest tests/
37
Currently, the repository is not fully checked by `mypy`.
38
```
39
40
-# Contribution Guidelines
41
-
40
## Issues
41
42
If you encounter a bug or have a feature request, please [search existing issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue) first to see if it has already been reported. If not, please [file a new issue](https://github.com/vllm-project/vllm/issues/new/choose), providing as much relevant information as possible.
docs/source/deployment/docker.md
+2
-2
@@ -28,8 +28,8 @@ memory to share data between processes under the hood, particularly for tensor p
28
You can build and run vLLM from source via the provided <gh-file:Dockerfile>. To build vLLM:
29
30
```console
31
-$ # optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2
32
-$ DOCKER_BUILDKIT=1 docker build . --target vllm-openai --tag vllm/vllm-openai
31
+# optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2
32
+DOCKER_BUILDKIT=1 docker build . --target vllm-openai --tag vllm/vllm-openai
33
```
34
35
```{note}
docs/source/deployment/frameworks/cerebrium.md
+5
-5
@@ -13,14 +13,14 @@ vLLM can be run on a cloud based GPU machine with [Cerebrium](https://www.cerebr
13
To install the Cerebrium client, run:
14
15
```console
16
-$ pip install cerebrium
17
-$ cerebrium login
16
+pip install cerebrium
17
+cerebrium login
18
```
19
20
Next, create your Cerebrium project, run:
21
22
```console
23
-$ cerebrium init vllm-project
23
+cerebrium init vllm-project
24
```
25
26
Next, to install the required packages, add the following to your cerebrium.toml:
@@ -58,10 +58,10 @@ def run(prompts: list[str], temperature: float = 0.8, top_p: float = 0.95):
58
Then, run the following code to deploy it to the cloud:
59
60
```console
61
-$ cerebrium deploy
61
+cerebrium deploy
62
```
63
64
-If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case` /run`)
64
+If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case`/run`)
65
66
```python
67
curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \
docs/source/deployment/frameworks/dstack.md
+5
-5
@@ -13,16 +13,16 @@ vLLM can be run on a cloud based GPU machine with [dstack](https://dstack.ai/),
13
To install dstack client, run:
14
15
```console
16
-$ pip install "dstack[all]
17
-$ dstack server
16
+pip install "dstack[all]
17
+dstack server
18
```
19
20
Next, to configure your dstack project, run:
21
22
```console
23
-$ mkdir -p vllm-dstack
24
-$ cd vllm-dstack
25
-$ dstack init
23
+mkdir -p vllm-dstack
24
+cd vllm-dstack
25
+dstack init
26
```
27
28
Next, to provision a VM instance with LLM of your choice (`NousResearch/Llama-2-7b-chat-hf` for this example), create the following `serve.dstack.yml` file for the dstack `Service`:
docs/source/deployment/frameworks/skypilot.md
+6
-6
@@ -334,12 +334,12 @@ run: |
334
335
1. Start the chat web UI:
336
337
-```console
338
-sky launch -c gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint vllm)
339
-```
337
+ ```console
338
+ sky launch -c gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint vllm)
339
+ ```
340
341
2. Then, we can access the GUI at the returned gradio link:
342
343
-```console
344
-| INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live
345
-```
343
+ ```console
344
+ | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live
345
+ ```
docs/source/deployment/integrations/llamastack.md
+1
-1
@@ -7,7 +7,7 @@ vLLM is also available via [Llama Stack](https://github.com/meta-llama/llama-sta
7
To install Llama Stack, run
8
9
```console
10
-$ pip install llama-stack -q
10
+pip install llama-stack -q
11
```
12
13
## Inference using OpenAI Compatible API
docs/source/deployment/k8s.md
+226
-225
@@ -14,234 +14,235 @@ Before you begin, ensure that you have the following:
14
15
## Deployment Steps
16
17
-1. **Create a PVC , Secret and Deployment for vLLM**
18
-
19
-PVC is used to store the model cache and it is optional, you can use hostPath or other storage options
20
-
21
-```yaml
22
-apiVersion: v1
23
-kind: PersistentVolumeClaim
24
-metadata:
25
- name: mistral-7b
26
- namespace: default
27
-spec:
28
- accessModes:
29
- - ReadWriteOnce
30
- resources:
31
- requests:
32
- storage: 50Gi
33
- storageClassName: default
34
- volumeMode: Filesystem
35
-```
36
-
37
-Secret is optional and only required for accessing gated models, you can skip this step if you are not using gated models
38
-
39
-```yaml
40
-apiVersion: v1
41
-kind: Secret
42
-metadata:
43
- name: hf-token-secret
44
- namespace: default
45
-type: Opaque
46
-stringData:
47
- token: "REPLACE_WITH_TOKEN"
48
-```
49
-
50
-Next to create the deployment file for vLLM to run the model server. The following example deploys the `Mistral-7B-Instruct-v0.3` model.
51
-
52
-Here are two examples for using NVIDIA GPU and AMD GPU.
53
-
54
-- NVIDIA GPU
55
-
56
-```yaml
57
-apiVersion: apps/v1
58
-kind: Deployment
59
-metadata:
60
- name: mistral-7b
61
- namespace: default
62
- labels:
63
- app: mistral-7b
64
-spec:
65
- replicas: 1
66
- selector:
67
- matchLabels:
68
- app: mistral-7b
69
- template:
70
- metadata:
71
- labels:
72
- app: mistral-7b
73
- spec:
74
- volumes:
75
- - name: cache-volume
76
- persistentVolumeClaim:
77
- claimName: mistral-7b
78
- # vLLM needs to access the host's shared memory for tensor parallel inference.
79
- - name: shm
80
- emptyDir:
81
- medium: Memory
82
- sizeLimit: "2Gi"
83
- containers:
84
- - name: mistral-7b
85
- image: vllm/vllm-openai:latest
86
- command: ["/bin/sh", "-c"]
87
- args: [
88
- "vllm serve mistralai/Mistral-7B-Instruct-v0.3 --trust-remote-code --enable-chunked-prefill --max_num_batched_tokens 1024"
89
- ]
90
- env:
91
- - name: HUGGING_FACE_HUB_TOKEN
92
- valueFrom:
93
- secretKeyRef:
94
- name: hf-token-secret
95
- key: token
96
- ports:
97
- - containerPort: 8000
17
+1. Create a PVC, Secret and Deployment for vLLM
18
+
19
+ PVC is used to store the model cache and it is optional, you can use hostPath or other storage options
20
+
21
+ ```yaml
22
+ apiVersion: v1
23
+ kind: PersistentVolumeClaim
24
+ metadata:
25
+ name: mistral-7b
26
+ namespace: default
27
+ spec:
28
+ accessModes:
29
+ - ReadWriteOnce
30
resources:
99
- limits:
100
- cpu: "10"
101
- memory: 20G
102
- nvidia.com/gpu: "1"
31
requests:
104
- cpu: "2"
105
- memory: 6G
106
- nvidia.com/gpu: "1"
107
- volumeMounts:
108
- - mountPath: /root/.cache/huggingface
109
- name: cache-volume
110
- - name: shm
111
- mountPath: /dev/shm
112
- livenessProbe:
113
- httpGet:
114
- path: /health
115
- port: 8000
116
- initialDelaySeconds: 60
117
- periodSeconds: 10
118
- readinessProbe:
119
- httpGet:
120
- path: /health
121
- port: 8000
122
- initialDelaySeconds: 60
123
- periodSeconds: 5
124
-```
125
-
126
-- AMD GPU
127
-
128
-You can refer to the `deployment.yaml` below if using AMD ROCm GPU like MI300X.
129
-
130
-```yaml
131
-apiVersion: apps/v1
132
-kind: Deployment
133
-metadata:
134
- name: mistral-7b
135
- namespace: default
136
- labels:
137
- app: mistral-7b
138
-spec:
139
- replicas: 1
140
- selector:
141
- matchLabels:
142
- app: mistral-7b
143
- template:
144
- metadata:
145
- labels:
146
- app: mistral-7b
147
- spec:
148
- volumes:
149
- # PVC
150
- - name: cache-volume
151
- persistentVolumeClaim:
152
- claimName: mistral-7b
153
- # vLLM needs to access the host's shared memory for tensor parallel inference.
154
- - name: shm
155
- emptyDir:
156
- medium: Memory
157
- sizeLimit: "8Gi"
158
- hostNetwork: true
159
- hostIPC: true
160
- containers:
161
- - name: mistral-7b
162
- image: rocm/vllm:rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4
163
- securityContext:
164
- seccompProfile:
165
- type: Unconfined
166
- runAsGroup: 44
167
- capabilities:
168
- add:
169
- - SYS_PTRACE
170
- command: ["/bin/sh", "-c"]
171
- args: [
172
- "vllm serve mistralai/Mistral-7B-v0.3 --port 8000 --trust-remote-code --enable-chunked-prefill --max_num_batched_tokens 1024"
173
- ]
174
- env:
175
- - name: HUGGING_FACE_HUB_TOKEN
176
- valueFrom:
177
- secretKeyRef:
178
- name: hf-token-secret
179
- key: token
32
+ storage: 50Gi
33
+ storageClassName: default
34
+ volumeMode: Filesystem
35
+ ```
36
+
37
+ Secret is optional and only required for accessing gated models, you can skip this step if you are not using gated models
38
+
39
+ ```yaml
40
+ apiVersion: v1
41
+ kind: Secret
42
+ metadata:
43
+ name: hf-token-secret
44
+ namespace: default
45
+ type: Opaque
46
+ stringData:
47
+ token: "REPLACE_WITH_TOKEN"
48
+ ```
49
+
50
+ Next to create the deployment file for vLLM to run the model server. The following example deploys the `Mistral-7B-Instruct-v0.3` model.
51
+
52
+ Here are two examples for using NVIDIA GPU and AMD GPU.
53
+
54
+ NVIDIA GPU:
55
+
56
+ ```yaml
57
+ apiVersion: apps/v1
58
+ kind: Deployment
59
+ metadata:
60
+ name: mistral-7b
61
+ namespace: default
62
+ labels:
63
+ app: mistral-7b
64
+ spec:
65
+ replicas: 1
66
+ selector:
67
+ matchLabels:
68
+ app: mistral-7b
69
+ template:
70
+ metadata:
71
+ labels:
72
+ app: mistral-7b
73
+ spec:
74
+ volumes:
75
+ - name: cache-volume
76
+ persistentVolumeClaim:
77
+ claimName: mistral-7b
78
+ # vLLM needs to access the host's shared memory for tensor parallel inference.
79
+ - name: shm
80
+ emptyDir:
81
+ medium: Memory
82
+ sizeLimit: "2Gi"
83
+ containers:
84
+ - name: mistral-7b
85
+ image: vllm/vllm-openai:latest
86
+ command: ["/bin/sh", "-c"]
87
+ args: [
88
+ "vllm serve mistralai/Mistral-7B-Instruct-v0.3 --trust-remote-code --enable-chunked-prefill --max_num_batched_tokens 1024"
89
+ ]
90
+ env:
91
+ - name: HUGGING_FACE_HUB_TOKEN
92
+ valueFrom:
93
+ secretKeyRef:
94
+ name: hf-token-secret
95
+ key: token
96
+ ports:
97
+ - containerPort: 8000
98
+ resources:
99
+ limits:
100
+ cpu: "10"
101
+ memory: 20G
102
+ nvidia.com/gpu: "1"
103
+ requests:
104
+ cpu: "2"
105
+ memory: 6G
106
+ nvidia.com/gpu: "1"
107
+ volumeMounts:
108
+ - mountPath: /root/.cache/huggingface
109
+ name: cache-volume
110
+ - name: shm
111
+ mountPath: /dev/shm
112
+ livenessProbe:
113
+ httpGet:
114
+ path: /health
115
+ port: 8000
116
+ initialDelaySeconds: 60
117
+ periodSeconds: 10
118
+ readinessProbe:
119
+ httpGet:
120
+ path: /health
121
+ port: 8000
122
+ initialDelaySeconds: 60
123
+ periodSeconds: 5
124
+ ```
125
+
126
+ AMD GPU:
127
+
128
+ You can refer to the `deployment.yaml` below if using AMD ROCm GPU like MI300X.
129
+
130
+ ```yaml
131
+ apiVersion: apps/v1
132
+ kind: Deployment
133
+ metadata:
134
+ name: mistral-7b
135
+ namespace: default
136
+ labels:
137
+ app: mistral-7b
138
+ spec:
139
+ replicas: 1
140
+ selector:
141
+ matchLabels:
142
+ app: mistral-7b
143
+ template:
144
+ metadata:
145
+ labels:
146
+ app: mistral-7b
147
+ spec:
148
+ volumes:
149
+ # PVC
150
+ - name: cache-volume
151
+ persistentVolumeClaim:
152
+ claimName: mistral-7b
153
+ # vLLM needs to access the host's shared memory for tensor parallel inference.
154
+ - name: shm
155
+ emptyDir:
156
+ medium: Memory
157
+ sizeLimit: "8Gi"
158
+ hostNetwork: true
159
+ hostIPC: true
160
+ containers:
161
+ - name: mistral-7b
162
+ image: rocm/vllm:rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4
163
+ securityContext:
164
+ seccompProfile:
165
+ type: Unconfined
166
+ runAsGroup: 44
167
+ capabilities:
168
+ add:
169
+ - SYS_PTRACE
170
+ command: ["/bin/sh", "-c"]
171
+ args: [
172
+ "vllm serve mistralai/Mistral-7B-v0.3 --port 8000 --trust-remote-code --enable-chunked-prefill --max_num_batched_tokens 1024"
173
+ ]
174
+ env:
175
+ - name: HUGGING_FACE_HUB_TOKEN
176
+ valueFrom:
177
+ secretKeyRef:
178
+ name: hf-token-secret
179
+ key: token
180
+ ports:
181
+ - containerPort: 8000
182
+ resources:
183
+ limits:
184
+ cpu: "10"
185
+ memory: 20G
186
+ amd.com/gpu: "1"
187
+ requests:
188
+ cpu: "6"
189
+ memory: 6G
190
+ amd.com/gpu: "1"
191
+ volumeMounts:
192
+ - name: cache-volume
193
+ mountPath: /root/.cache/huggingface
194
+ - name: shm
195
+ mountPath: /dev/shm
196
+ ```
197
+
198
+ You can get the full example with steps and sample yaml files from <https://github.com/ROCm/k8s-device-plugin/tree/master/example/vllm-serve>.
199
+
200
+2. Create a Kubernetes Service for vLLM
201
+
202
+ Next, create a Kubernetes Service file to expose the `mistral-7b` deployment:
203
+
204
+ ```yaml
205
+ apiVersion: v1
206
+ kind: Service
207
+ metadata:
208
+ name: mistral-7b
209
+ namespace: default
210
+ spec:
211
ports:
181
- - containerPort: 8000
182
- resources:
183
- limits:
184
- cpu: "10"
185
- memory: 20G
186
- amd.com/gpu: "1"
187
- requests:
188
- cpu: "6"
189
- memory: 6G
190
- amd.com/gpu: "1"
191
- volumeMounts:
192
- - name: cache-volume
193
- mountPath: /root/.cache/huggingface
194
- - name: shm
195
- mountPath: /dev/shm
196
-```
197
-You can get the full example with steps and sample yaml files from <https://github.com/ROCm/k8s-device-plugin/tree/master/example/vllm-serve>.
198
-
199
-2. **Create a Kubernetes Service for vLLM**
200
-
201
-Next, create a Kubernetes Service file to expose the `mistral-7b` deployment:
202
-
203
-```yaml
204
-apiVersion: v1
205
-kind: Service
206
-metadata:
207
- name: mistral-7b
208
- namespace: default
209
-spec:
210
- ports:
211
- - name: http-mistral-7b
212
- port: 80
213
- protocol: TCP
214
- targetPort: 8000
215
- # The label selector should match the deployment labels & it is useful for prefix caching feature
216
- selector:
217
- app: mistral-7b
218
- sessionAffinity: None
219
- type: ClusterIP
220
-```
221
-
222
-3. **Deploy and Test**
223
-
224
-Apply the deployment and service configurations using `kubectl apply -f <filename>`:
225
-
226
-```console
227
-kubectl apply -f deployment.yaml
228
-kubectl apply -f service.yaml
229
-```
230
-
231
-To test the deployment, run the following `curl` command:
232
-
233
-```console
234
-curl http://mistral-7b.default.svc.cluster.local/v1/completions \
235
- -H "Content-Type: application/json" \
236
- -d '{
237
- "model": "mistralai/Mistral-7B-Instruct-v0.3",
238
- "prompt": "San Francisco is a",
239
- "max_tokens": 7,
240
- "temperature": 0
241
- }'
242
-```
243
-
244
-If the service is correctly deployed, you should receive a response from the vLLM model.
212
+ - name: http-mistral-7b
213
+ port: 80
214
+ protocol: TCP
215
+ targetPort: 8000
216
+ # The label selector should match the deployment labels & it is useful for prefix caching feature
217
+ selector:
218
+ app: mistral-7b
219
+ sessionAffinity: None
220
+ type: ClusterIP
221
+ ```
222
+
223
+3. Deploy and Test
224
+
225
+ Apply the deployment and service configurations using `kubectl apply -f <filename>`:
226
+
227
+ ```console
228
+ kubectl apply -f deployment.yaml
229
+ kubectl apply -f service.yaml
230
+ ```
231
+
232
+ To test the deployment, run the following `curl` command:
233
+
234
+ ```console
235
+ curl http://mistral-7b.default.svc.cluster.local/v1/completions \
236
+ -H "Content-Type: application/json" \
237
+ -d '{
238
+ "model": "mistralai/Mistral-7B-Instruct-v0.3",
239
+ "prompt": "San Francisco is a",
240
+ "max_tokens": 7,
241
+ "temperature": 0
242
+ }'
243
+ ```
244
+
245
+ If the service is correctly deployed, you should receive a response from the vLLM model.
246
247
## Conclusion
248
docs/source/design/automatic_prefix_caching.md
+4
-7
@@ -6,7 +6,7 @@ The core idea of [PagedAttention](https://blog.vllm.ai/2023/06/20/vllm.html) is
6
7
To automatically cache the KV cache, we utilize the following key observation: Each KV block can be uniquely identified by the tokens within the block and the tokens in the prefix before the block.
8
9
-```
9
+```text
10
Block 1 Block 2 Block 3
11
[A gentle breeze stirred] [the leaves as children] [laughed in the distance]
12
Block 1: |<--- block tokens ---->|
@@ -14,19 +14,16 @@ Block 2: |<------- prefix ------>| |<--- block tokens --->|
14
Block 3: |<------------------ prefix -------------------->| |<--- block tokens ---->|
15
```
16
17
-
17
In the example above, the KV cache in the first block can be uniquely identified with the tokens “A gentle breeze stirred”. The third block can be uniquely identified with the tokens in the block “laughed in the distance”, along with the prefix tokens “A gentle breeze stirred the leaves as children”. Therefore, we can build the following one-to-one mapping:
18
20
-```
19
+```text
20
hash(prefix tokens + block tokens) <--> KV Block
21
```
22
23
With this mapping, we can add another indirection in vLLM’s KV cache management. Previously, each sequence in vLLM maintained a mapping from their logical KV blocks to physical blocks. To achieve automatic caching of KV blocks, we map the logical KV blocks to their hash value and maintain a global hash table of all the physical blocks. In this way, all the KV blocks sharing the same hash value (e.g., shared prefix blocks across two requests) can be mapped to the same physical block and share the memory space.
24
26
-
25
This design achieves automatic prefix caching without the need of maintaining a tree structure among the KV blocks. More specifically, all of the blocks are independent of each other and can be allocated and freed by itself, which enables us to manages the KV cache as ordinary caches in operating system.
26
29
-
27
## Generalized Caching Policy
28
29
Keeping all the KV blocks in a hash table enables vLLM to cache KV blocks from earlier requests to save memory and accelerate the computation of future requests. For example, if a new request shares the system prompt with the previous request, the KV cache of the shared prompt can directly be used for the new request without recomputation. However, the total KV cache space is limited and we have to decide which KV blocks to keep or evict when the cache is full.
@@ -41,5 +38,5 @@ Note that this eviction policy effectively implements the exact policy as in [Ra
38
39
However, the hash-based KV cache management gives us the flexibility to handle more complicated serving scenarios and implement more complicated eviction policies beyond the policy above:
40
44
-- Multi-LoRA serving. When serving requests for multiple LoRA adapters, we can simply let the hash of each KV block to also include the LoRA ID the request is querying for to enable caching for all adapters. In this way, we can jointly manage the KV blocks for different adapters, which simplifies the system implementation and improves the global cache hit rate and efficiency.
45
-- Multi-modal models. When the user input includes more than just discrete tokens, we can use different hashing methods to handle the caching of inputs of different modalities. For example, perceptual hashing for images to cache similar input images.
41
+* Multi-LoRA serving. When serving requests for multiple LoRA adapters, we can simply let the hash of each KV block to also include the LoRA ID the request is querying for to enable caching for all adapters. In this way, we can jointly manage the KV blocks for different adapters, which simplifies the system implementation and improves the global cache hit rate and efficiency.
42
+* Multi-modal models. When the user input includes more than just discrete tokens, we can use different hashing methods to handle the caching of inputs of different modalities. For example, perceptual hashing for images to cache similar input images.
docs/source/features/quantization/auto_awq.md
+2
-2
@@ -15,7 +15,7 @@ The main benefits are lower latency and memory usage.
15
You can quantize your own models by installing AutoAWQ or picking one of the [400+ models on Huggingface](https://huggingface.co/models?sort=trending&search=awq).
16
17
```console
18
-$ pip install autoawq
18
+pip install autoawq
19
```
20
21
After installing AutoAWQ, you are ready to quantize a model. Here is an example of how to quantize `mistralai/Mistral-7B-Instruct-v0.2`:
@@ -47,7 +47,7 @@ print(f'Model is quantized and saved at "{quant_path}"')
47
To run an AWQ model with vLLM, you can use [TheBloke/Llama-2-7b-Chat-AWQ](https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ) with the following command:
48
49
```console
50
-$ python examples/offline_inference/llm_engine_example.py --model TheBloke/Llama-2-7b-Chat-AWQ --quantization awq
50
+python examples/offline_inference/llm_engine_example.py --model TheBloke/Llama-2-7b-Chat-AWQ --quantization awq
51
```
52
53
AWQ models are also supported directly through the LLM entrypoint:
docs/source/features/quantization/bnb.md
+4
-3
@@ -9,7 +9,7 @@ Compared to other quantization methods, BitsAndBytes eliminates the need for cal
9
Below are the steps to utilize BitsAndBytes with vLLM.
10
11
```console
12
-$ pip install bitsandbytes>=0.45.0
12
+pip install bitsandbytes>=0.45.0
13
```
14
15
vLLM reads the model's config file and supports both in-flight quantization and pre-quantized checkpoint.
@@ -17,7 +17,7 @@ vLLM reads the model's config file and supports both in-flight quantization and
17
You can find bitsandbytes quantized models on <https://huggingface.co/models?other=bitsandbytes>.
18
And usually, these repositories have a config.json file that includes a quantization_config section.
19
20
-## Read quantized checkpoint.
20
+## Read quantized checkpoint
21
22
```python
23
from vllm import LLM
@@ -37,10 +37,11 @@ model_id = "huggyllama/llama-7b"
37
llm = LLM(model=model_id, dtype=torch.bfloat16, trust_remote_code=True, \
38
quantization="bitsandbytes", load_format="bitsandbytes")
39
```
40
+
41
## OpenAI Compatible Server
42
43
Append the following to your 4bit model arguments:
44
44
-```
45
+```console
46
--quantization bitsandbytes --load-format bitsandbytes
47
```
docs/source/features/quantization/fp8.md
+2
-2
@@ -41,7 +41,7 @@ Currently, we load the model at original precision before quantizing down to 8-b
41
To produce performant FP8 quantized models with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library:
42
43
```console
44
-$ pip install llmcompressor
44
+pip install llmcompressor
45
```
46
47
## Quantization Process
@@ -98,7 +98,7 @@ tokenizer.save_pretrained(SAVE_DIR)
98
Install `vllm` and `lm-evaluation-harness`:
99
100
```console
101
-$ pip install vllm lm-eval==0.4.4
101
+pip install vllm lm-eval==0.4.4
102
```
103
104
Load and run the model in `vllm`:
docs/source/features/quantization/fp8_e4m3_kvcache.md
+1
-1
@@ -17,7 +17,7 @@ unquantized model through a quantizer tool (e.g. AMD quantizer or NVIDIA AMMO).
17
To install AMMO (AlgorithMic Model Optimization):
18
19
```console
20
-$ pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com nvidia-ammo
20
+pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com nvidia-ammo
21
```
22
23
Studies have shown that FP8 E4M3 quantization typically only minimally degrades inference accuracy. The most recent silicon
docs/source/features/quantization/gguf.md
+5
-5
@@ -13,16 +13,16 @@ Currently, vllm only supports loading single-file GGUF models. If you have a mul
13
To run a GGUF model with vLLM, you can download and use the local GGUF model from [TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF](https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF) with the following command:
14
15
```console
16
-$ wget https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf
17
-$ # We recommend using the tokenizer from base model to avoid long-time and buggy tokenizer conversion.
18
-$ vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0
16
+wget https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf
17
+# We recommend using the tokenizer from base model to avoid long-time and buggy tokenizer conversion.
18
+vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0
19
```
20
21
You can also add `--tensor-parallel-size 2` to enable tensor parallelism inference with 2 GPUs:
22
23
```console
24
-$ # We recommend using the tokenizer from base model to avoid long-time and buggy tokenizer conversion.
25
-$ vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0 --tensor-parallel-size 2
24
+# We recommend using the tokenizer from base model to avoid long-time and buggy tokenizer conversion.
25
+vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0 --tensor-parallel-size 2
26
```
27
28
```{warning}
docs/source/features/quantization/int8.md
+1
-1
@@ -16,7 +16,7 @@ INT8 computation is supported on NVIDIA GPUs with compute capability > 7.5 (Turi
16
To use INT8 quantization with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library:
17
18
```console
19
-$ pip install llmcompressor
19
+pip install llmcompressor
20
```
21
22
## Quantization Process
docs/source/features/spec_decode.md
+2
-8
@@ -192,11 +192,11 @@ A few important things to consider when using the EAGLE based draft models:
192
193
1. The EAGLE draft models available in the [HF repository for EAGLE models](https://huggingface.co/yuhuili) cannot be
194
used directly with vLLM due to differences in the expected layer names and model definition.
195
- To use these models with vLLM, use the [following script](https://gist.github.com/abhigoyal1997/1e7a4109ccb7704fbc67f625e86b2d6d)
195
+ To use these models with vLLM, use the [following script](https://gist.github.com/abhigoyal1997/1e7a4109ccb7704fbc67f625e86b2d6d)
196
to convert them. Note that this script does not modify the model's weights.
197
198
In the above example, use the script to first convert
199
- the [yuhuili/EAGLE-LLaMA3-Instruct-8B](https://huggingface.co/yuhuili/EAGLE-LLaMA3-Instruct-8B) model
199
+ the [yuhuili/EAGLE-LLaMA3-Instruct-8B](https://huggingface.co/yuhuili/EAGLE-LLaMA3-Instruct-8B) model
200
and then use the converted checkpoint as the draft model in vLLM.
201
202
2. The EAGLE based draft models need to be run without tensor parallelism
@@ -207,7 +207,6 @@ A few important things to consider when using the EAGLE based draft models:
207
reported in the reference implementation [here](https://github.com/SafeAILab/EAGLE). This issue is under
208
investigation and tracked here: [https://github.com/vllm-project/vllm/issues/9565](https://github.com/vllm-project/vllm/issues/9565).
209
210
-
210
A variety of EAGLE draft models are available on the Hugging Face hub:
211
212
| Base Model | EAGLE on Hugging Face | # EAGLE Parameters |
@@ -224,7 +223,6 @@ A variety of EAGLE draft models are available on the Hugging Face hub:
223
| Qwen2-7B-Instruct | yuhuili/EAGLE-Qwen2-7B-Instruct | 0.26B |
224
| Qwen2-72B-Instruct | yuhuili/EAGLE-Qwen2-72B-Instruct | 1.05B |
225
227
-
226
## Lossless guarantees of Speculative Decoding
227
228
In vLLM, speculative decoding aims to enhance inference efficiency while maintaining accuracy. This section addresses the lossless guarantees of
@@ -250,8 +248,6 @@ speculative decoding, breaking down the guarantees into three key areas:
248
same request across runs. For more details, see the FAQ section
249
titled *Can the output of a prompt vary across runs in vLLM?* in the [FAQs](#faq).
250
253
-**Conclusion**
254
-
251
While vLLM strives to ensure losslessness in speculative decoding, variations in generated outputs with and without speculative decoding
252
can occur due to following factors:
253
@@ -259,8 +255,6 @@ can occur due to following factors:
255
- **Batch Size and Numerical Stability**: Changes in batch size may cause variations in logprobs and output probabilities, potentially
256
due to non-deterministic behavior in batched operations or numerical instability.
257
262
-**Mitigation Strategies**
263
-
258
For mitigation strategies, please refer to the FAQ entry *Can the output of a prompt vary across runs in vLLM?* in the [FAQs](#faq).
259
260
## Resources for vLLM contributors
docs/source/features/tool_calling.md
+26
-18
@@ -55,21 +55,24 @@ print(f"Result: {get_weather(**json.loads(tool_call.arguments))}")
55
```
56
57
Example output:
58
-```
58
+
59
+```text
60
Function called: get_weather
61
Arguments: {"location": "San Francisco, CA", "unit": "fahrenheit"}
62
Result: Getting the weather for San Francisco, CA in fahrenheit...
63
```
64
65
This example demonstrates:
65
-- Setting up the server with tool calling enabled
66
-- Defining an actual function to handle tool calls
67
-- Making a request with `tool_choice="auto"`
68
-- Handling the structured response and executing the corresponding function
66
+
67
+* Setting up the server with tool calling enabled
68
+* Defining an actual function to handle tool calls
69
+* Making a request with `tool_choice="auto"`
70
+* Handling the structured response and executing the corresponding function
71
72
You can also specify a particular function using named function calling by setting `tool_choice={"type": "function", "function": {"name": "get_weather"}}`. Note that this will use the guided decoding backend - so the first time this is used, there will be several seconds of latency (or more) as the FSM is compiled for the first time before it is cached for subsequent requests.
73
74
Remember that it's the callers responsibility to:
75
+
76
1. Define appropriate tools in the request
77
2. Include relevant context in the chat messages
78
3. Handle the tool calls in your application logic
@@ -77,20 +80,21 @@ Remember that it's the callers responsibility to:
80
For more advanced usage, including parallel tool calls and different model-specific parsers, see the sections below.
81
82
## Named Function Calling
83
+
84
vLLM supports named function calling in the chat completion API by default. It does so using Outlines through guided decoding, so this is
85
enabled by default, and will work with any supported model. You are guaranteed a validly-parsable function call - not a
86
high-quality one.
87
84
-vLLM will use guided decoding to ensure the response matches the tool parameter object defined by the JSON schema in the `tools` parameter.
88
+vLLM will use guided decoding to ensure the response matches the tool parameter object defined by the JSON schema in the `tools` parameter.
89
For best results, we recommend ensuring that the expected output format / schema is specified in the prompt to ensure that the model's intended generation is aligned with the schema that it's being forced to generate by the guided decoding backend.
90
91
To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
92
specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
93
90
-
94
## Automatic Function Calling
95
96
To enable this feature, you should set the following flags:
97
+
98
* `--enable-auto-tool-choice` -- **mandatory** Auto tool choice. tells vLLM that you want to enable the model to generate its own tool calls when it
99
deems appropriate.
100
* `--tool-call-parser` -- select the tool parser to use (listed below). Additional tool parsers
@@ -104,28 +108,28 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
108
109
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
110
107
-
111
### Hermes Models (`hermes`)
112
113
All Nous Research Hermes-series models newer than Hermes 2 Pro should be supported.
114
+
115
* `NousResearch/Hermes-2-Pro-*`
116
* `NousResearch/Hermes-2-Theta-*`
117
* `NousResearch/Hermes-3-*`
118
115
-
119
_Note that the Hermes 2 **Theta** models are known to have degraded tool call quality & capabilities due to the merge
120
step in their creation_.
121
122
Flags: `--tool-call-parser hermes`
123
121
-
124
### Mistral Models (`mistral`)
125
126
Supported models:
127
+
128
* `mistralai/Mistral-7B-Instruct-v0.3` (confirmed)
129
* Additional mistral function-calling models are compatible as well.
130
131
Known issues:
132
+
133
1. Mistral 7B struggles to generate parallel tool calls correctly.
134
2. Mistral's `tokenizer_config.json` chat template requires tool call IDs that are exactly 9 digits, which is
135
much shorter than what vLLM generates. Since an exception is thrown when this condition
@@ -136,13 +140,12 @@ it works with vLLM's tool call IDs (provided `tool_call_id` fields are truncated
140
* `examples/tool_chat_template_mistral_parallel.jinja` - this is a "better" version that adds a tool-use system prompt
141
when tools are provided, that results in much better reliability when working with parallel tool calling.
142
139
-
143
Recommended flags: `--tool-call-parser mistral --chat-template examples/tool_chat_template_mistral_parallel.jinja`
144
142
-
145
### Llama Models (`llama3_json`)
146
147
Supported models:
148
+
149
* `meta-llama/Meta-Llama-3.1-8B-Instruct`
150
* `meta-llama/Meta-Llama-3.1-70B-Instruct`
151
* `meta-llama/Meta-Llama-3.1-405B-Instruct`
@@ -152,6 +155,7 @@ The tool calling that is supported is the [JSON based tool calling](https://llam
155
Other tool calling formats like the built in python tool calling or custom tool calling are not supported.
156
157
Known issues:
158
+
159
1. Parallel tool calls are not supported.
160
2. The model can generate parameters with a wrong format, such as generating
161
an array serialized as string instead of an array.
@@ -164,6 +168,7 @@ Recommended flags: `--tool-call-parser llama3_json --chat-template examples/tool
168
#### IBM Granite
169
170
Supported models:
171
+
172
* `ibm-granite/granite-3.0-8b-instruct`
173
174
Recommended flags: `--tool-call-parser granite --chat-template examples/tool_chat_template_granite.jinja`
@@ -182,42 +187,45 @@ Recommended flags: `--tool-call-parser granite-20b-fc --chat-template examples/t
187
188
`examples/tool_chat_template_granite_20b_fc.jinja`: this is a modified chat template from the original on Huggingface, which is not vLLM compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
189
185
-
190
### InternLM Models (`internlm`)
191
192
Supported models:
193
+
194
* `internlm/internlm2_5-7b-chat` (confirmed)
195
* Additional internlm2.5 function-calling models are compatible as well
196
197
Known issues:
198
+
199
* Although this implementation also supports InternLM2, the tool call results are not stable when testing with the `internlm/internlm2-chat-7b` model.
200
201
Recommended flags: `--tool-call-parser internlm --chat-template examples/tool_chat_template_internlm2_tool.jinja`
202
197
-
203
### Jamba Models (`jamba`)
204
+
205
AI21's Jamba-1.5 models are supported.
206
+
207
* `ai21labs/AI21-Jamba-1.5-Mini`
208
* `ai21labs/AI21-Jamba-1.5-Large`
209
203
-
210
Flags: `--tool-call-parser jamba`
211
206
-
212
### Models with Pythonic Tool Calls (`pythonic`)
213
214
A growing number of models output a python list to represent tool calls instead of using JSON. This has the advantage of inherently supporting parallel tool calls and removing ambiguity around the JSON schema required for tool calls. The `pythonic` tool parser can support such models.
215
216
As a concrete example, these models may look up the weather in San Francisco and Seattle by generating:
217
+
218
```python
219
[get_weather(city='San Francisco', metric='celsius'), get_weather(city='Seattle', metric='celsius')]
220
```
221
222
Limitations:
223
+
224
* The model must not generate both text and tool calls in the same generation. This may not be hard to change for a specific model, but the community currently lacks consensus on which tokens to emit when starting and ending tool calls. (In particular, the Llama 3.2 models emit no such tokens.)
225
* Llama's smaller models struggle to use tools effectively.
226
227
Example supported models:
228
+
229
* `meta-llama/Llama-3.2-1B-Instruct`\* (use with `examples/tool_chat_template_llama3.2_pythonic.jinja`)
230
* `meta-llama/Llama-3.2-3B-Instruct`\* (use with `examples/tool_chat_template_llama3.2_pythonic.jinja`)
231
* `Team-ACE/ToolACE-8B` (use with `examples/tool_chat_template_toolace.jinja`)
@@ -231,7 +239,6 @@ Llama's smaller models frequently fail to emit tool calls in the correct format.
239
240
---
241
234
-
242
## How to write a tool parser plugin
243
244
A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py.
@@ -284,7 +291,8 @@ class ExampleToolParser(ToolParser):
291
```
292
293
Then you can use this plugin in the command line like this.
287
-```
294
+
295
+```console
296
--enable-auto-tool-choice \
297
--tool-parser-plugin <absolute path of the plugin file>
298
--tool-call-parser example \
docs/source/getting_started/faq.md
+1
-1
@@ -30,7 +30,7 @@ changes in batch size, or batch expansion in speculative decoding. These batchin
30
can lead to slightly different logit/logprob values at each step. Such differences can accumulate, potentially resulting in
31
different tokens being sampled. Once a different token is sampled, further divergence is likely.
32
33
-**Mitigation Strategies**
33
+## Mitigation Strategies
34
35
- For improved stability and reduced variance, use `float32`. Note that this will require more memory.
36
- If using `bfloat16`, switching to `float16` can also help.
docs/source/getting_started/installation/cpu-apple.md
+7
-10
@@ -18,25 +18,23 @@ Currently the CPU implementation for macOS supports FP32 and FP16 datatypes.
18
19
After installation of XCode and the Command Line Tools, which include Apple Clang, execute the following commands to build and install vLLM from the source.
20
21
-```
22
-$ git clone https://github.com/vllm-project/vllm.git
23
-$ cd vllm
24
-$ pip install -r requirements-cpu.txt
25
-$ pip install -e .
21
+```console
22
+git clone https://github.com/vllm-project/vllm.git
23
+cd vllm
24
+pip install -r requirements-cpu.txt
25
+pip install -e .
26
```
27
28
```{note}
29
On macOS the `VLLM_TARGET_DEVICE` is automatically set to `cpu`, which currently is the only supported device.
30
```
31
32
-
33
-
32
## Troubleshooting
33
36
-If the build has error like the following snippet where standard C++ headers cannot be found, try to remove and reinstall your
34
+If the build has error like the following snippet where standard C++ headers cannot be found, try to remove and reinstall your
35
[Command Line Tools for Xcode](https://developer.apple.com/download/all/).
36
39
-```
37
+```text
38
[...] fatal error: 'map' file not found
39
1 | #include <map>
40
| ^~~~~
@@ -48,4 +46,3 @@ If the build has error like the following snippet where standard C++ headers can
46
| ^~~~~~~~~
47
1 error generated.
48
```
51
-
docs/source/getting_started/installation/cpu-x86.md
+22
-22
@@ -32,13 +32,13 @@ Table of contents:
32
## Quick start using Dockerfile
33
34
```console
35
-$ docker build -f Dockerfile.cpu -t vllm-cpu-env --shm-size=4g .
36
-$ docker run -it \
37
- --rm \
38
- --network=host \
39
- --cpuset-cpus=<cpu-id-list, optional> \
40
- --cpuset-mems=<memory-node, optional> \
41
- vllm-cpu-env
35
+docker build -f Dockerfile.cpu -t vllm-cpu-env --shm-size=4g .
36
+docker run -it \
37
+ --rm \
38
+ --network=host \
39
+ --cpuset-cpus=<cpu-id-list, optional> \
40
+ --cpuset-mems=<memory-node, optional> \
41
+ vllm-cpu-env
42
```
43
44
(build-cpu-backend-from-source)=
@@ -48,23 +48,23 @@ $ docker run -it \
48
- First, install recommended compiler. We recommend to use `gcc/g++ >= 12.3.0` as the default compiler to avoid potential problems. For example, on Ubuntu 22.4, you can run:
49
50
```console
51
-$ sudo apt-get update -y
52
-$ sudo apt-get install -y gcc-12 g++-12 libnuma-dev
53
-$ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12
51
+sudo apt-get update -y
52
+sudo apt-get install -y gcc-12 g++-12 libnuma-dev
53
+sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12
54
```
55
56
- Second, install Python packages for vLLM CPU backend building:
57
58
```console
59
-$ pip install --upgrade pip
60
-$ pip install cmake>=3.26 wheel packaging ninja "setuptools-scm>=8" numpy
61
-$ pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
59
+pip install --upgrade pip
60
+pip install cmake>=3.26 wheel packaging ninja "setuptools-scm>=8" numpy
61
+pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
62
```
63
64
- Finally, build and install vLLM CPU backend:
65
66
```console
67
-$ VLLM_TARGET_DEVICE=cpu python setup.py install
67
+VLLM_TARGET_DEVICE=cpu python setup.py install
68
```
69
70
```{note}
@@ -92,18 +92,18 @@ $ VLLM_TARGET_DEVICE=cpu python setup.py install
92
- We highly recommend to use TCMalloc for high performance memory allocation and better cache locality. For example, on Ubuntu 22.4, you can run:
93
94
```console
95
-$ sudo apt-get install libtcmalloc-minimal4 # install TCMalloc library
96
-$ find / -name *libtcmalloc* # find the dynamic link library path
97
-$ export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4:$LD_PRELOAD # prepend the library to LD_PRELOAD
98
-$ python examples/offline_inference/basic.py # run vLLM
95
+sudo apt-get install libtcmalloc-minimal4 # install TCMalloc library
96
+find / -name *libtcmalloc* # find the dynamic link library path
97
+export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4:$LD_PRELOAD # prepend the library to LD_PRELOAD
98
+python examples/offline_inference/basic.py # run vLLM
99
```
100
101
- When using the online serving, it is recommended to reserve 1-2 CPU cores for the serving framework to avoid CPU oversubscription. For example, on a platform with 32 physical CPU cores, reserving CPU 30 and 31 for the framework and using CPU 0-29 for OpenMP:
102
103
```console
104
-$ export VLLM_CPU_KVCACHE_SPACE=40
105
-$ export VLLM_CPU_OMP_THREADS_BIND=0-29
106
-$ vllm serve facebook/opt-125m
104
+export VLLM_CPU_KVCACHE_SPACE=40
105
+export VLLM_CPU_OMP_THREADS_BIND=0-29
106
+vllm serve facebook/opt-125m
107
```
108
109
- If using vLLM CPU backend on a machine with hyper-threading, it is recommended to bind only one OpenMP thread on each physical CPU core using `VLLM_CPU_OMP_THREADS_BIND`. On a hyper-threading enabled platform with 16 logical CPU cores / 8 physical CPU cores:
@@ -148,7 +148,7 @@ $ python examples/offline_inference/basic.py
148
- Using Tensor Parallel for a latency constraints deployment: following GPU backend design, a Megatron-LM's parallel algorithm will be used to shard the model, based on the number of NUMA nodes (e.g. TP = 2 for a two NUMA node system). With [TP feature on CPU](gh-pr:6125) merged, Tensor Parallel is supported for serving and offline inferencing. In general each NUMA node is treated as one GPU card. Below is the example script to enable Tensor Parallel = 2 for serving:
149
150
```console
151
- $ VLLM_CPU_KVCACHE_SPACE=40 VLLM_CPU_OMP_THREADS_BIND="0-31|32-63" vllm serve meta-llama/Llama-2-7b-chat-hf -tp=2 --distributed-executor-backend mp
151
+ VLLM_CPU_KVCACHE_SPACE=40 VLLM_CPU_OMP_THREADS_BIND="0-31|32-63" vllm serve meta-llama/Llama-2-7b-chat-hf -tp=2 --distributed-executor-backend mp
152
```
153
154
- Using Data Parallel for maximum throughput: to launch an LLM serving endpoint on each NUMA node along with one additional load balancer to dispatch the requests to those endpoints. Common solutions like [Nginx](#nginxloadbalancer) or HAProxy are recommended. Anyscale Ray project provides the feature on LLM [serving](https://docs.ray.io/en/latest/serve/index.html). Here is the example to setup a scalable LLM serving with [Ray Serve](https://github.com/intel/llm-on-ray/blob/main/docs/setup.md).
docs/source/getting_started/installation/gpu-cuda.md
+48
-48
@@ -17,9 +17,9 @@ vLLM is a Python library that also contains pre-compiled C++ and CUDA (12.1) bin
17
You can create a new Python environment using `conda`:
18
19
```console
20
-$ # (Recommended) Create a new conda environment.
21
-$ conda create -n myenv python=3.12 -y
22
-$ conda activate myenv
20
+# (Recommended) Create a new conda environment.
21
+conda create -n myenv python=3.12 -y
22
+conda activate myenv
23
```
24
25
```{note}
@@ -29,9 +29,9 @@ $ conda activate myenv
29
Or you can create a new Python environment using [uv](https://docs.astral.sh/uv/), a very fast Python environment manager. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment using the following command:
30
31
```console
32
-$ # (Recommended) Create a new uv environment. Use `--seed` to install `pip` and `setuptools` in the environment.
33
-$ uv venv myenv --python 3.12 --seed
34
-$ source myenv/bin/activate
32
+# (Recommended) Create a new uv environment. Use `--seed` to install `pip` and `setuptools` in the environment.
33
+uv venv myenv --python 3.12 --seed
34
+source myenv/bin/activate
35
```
36
37
In order to be performant, vLLM has to compile many cuda kernels. The compilation unfortunately introduces binary incompatibility with other CUDA versions and PyTorch versions, even for the same PyTorch version with different building configurations.
@@ -43,18 +43,18 @@ Therefore, it is recommended to install vLLM with a **fresh new** environment. I
43
You can install vLLM using either `pip` or `uv pip`:
44
45
```console
46
-$ # Install vLLM with CUDA 12.1.
47
-$ pip install vllm # If you are using pip.
48
-$ uv pip install vllm # If you are using uv.
46
+# Install vLLM with CUDA 12.1.
47
+pip install vllm # If you are using pip.
48
+uv pip install vllm # If you are using uv.
49
```
50
51
As of now, vLLM's binaries are compiled with CUDA 12.1 and public PyTorch release versions by default. We also provide vLLM binaries compiled with CUDA 11.8 and public PyTorch release versions:
52
53
```console
54
-$ # Install vLLM with CUDA 11.8.
55
-$ export VLLM_VERSION=0.6.1.post1
56
-$ export PYTHON_VERSION=310
57
-$ pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118
54
+# Install vLLM with CUDA 11.8.
55
+export VLLM_VERSION=0.6.1.post1
56
+export PYTHON_VERSION=310
57
+pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118
58
```
59
60
(install-the-latest-code)=
@@ -66,7 +66,7 @@ LLM inference is a fast-evolving field, and the latest code may contain bug fixe
66
### Install the latest code using `pip`
67
68
```console
69
-$ pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
69
+pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
70
```
71
72
`--pre` is required for `pip` to consider pre-released versions.
@@ -74,8 +74,8 @@ $ pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
74
If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), due to the limitation of `pip`, you have to specify the full URL of the wheel file by embedding the commit hash in the URL:
75
76
```console
77
-$ export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch
78
-$ pip install https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
77
+export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch
78
+pip install https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
79
```
80
81
Note that the wheels are built with Python 3.8 ABI (see [PEP 425](https://peps.python.org/pep-0425/) for more details about ABI), so **they are compatible with Python 3.8 and later**. The version string in the wheel file name (`1.0.0.dev`) is just a placeholder to have a unified URL for the wheels, the actual versions of wheels are contained in the wheel metadata (the wheels listed in the extra index url have correct versions). Although we don't support Python 3.8 any more (because PyTorch 2.5 dropped support for Python 3.8), the wheels are still built with Python 3.8 ABI to keep the same wheel name as before.
@@ -85,14 +85,14 @@ Note that the wheels are built with Python 3.8 ABI (see [PEP 425](https://peps.p
85
Another way to install the latest code is to use `uv`:
86
87
```console
88
-$ uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly
88
+uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly
89
```
90
91
If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL:
92
93
```console
94
-$ export VLLM_COMMIT=72d9c316d3f6ede485146fe5aabd4e61dbc59069 # use full commit hash from the main branch
95
-$ uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}
94
+export VLLM_COMMIT=72d9c316d3f6ede485146fe5aabd4e61dbc59069 # use full commit hash from the main branch
95
+uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}
96
```
97
98
The `uv` approach works for vLLM `v0.6.6` and later and offers an easy-to-remember command. A unique feature of `uv` is that packages in `--extra-index-url` have [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes). If the latest public release is `v0.6.6.post1`, `uv`'s behavior allows installing a commit before `v0.6.6.post1` by specifying the `--extra-index-url`. In contrast, `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install a development version prior to the released version.
@@ -102,8 +102,8 @@ The `uv` approach works for vLLM `v0.6.6` and later and offers an easy-to-rememb
102
Another way to access the latest code is to use the docker images:
103
104
```console
105
-$ export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch
106
-$ docker pull public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:${VLLM_COMMIT}
105
+export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch
106
+docker pull public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:${VLLM_COMMIT}
107
```
108
109
These docker images are used for CI and testing only, and they are not intended for production use. They will be expired after several days.
@@ -121,18 +121,18 @@ The latest code can contain bugs and may not be stable. Please use it with cauti
121
If you only need to change Python code, you can build and install vLLM without compilation. Using `pip`'s [`--editable` flag](https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs), changes you make to the code will be reflected when you run vLLM:
122
123
```console
124
-$ git clone https://github.com/vllm-project/vllm.git
125
-$ cd vllm
126
-$ VLLM_USE_PRECOMPILED=1 pip install --editable .
124
+git clone https://github.com/vllm-project/vllm.git
125
+cd vllm
126
+VLLM_USE_PRECOMPILED=1 pip install --editable .
127
```
128
129
-This will download the latest nightly wheel from https://wheels.vllm.ai/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl and use the compiled libraries from there in the installation.
129
+This will download the [latest nightly wheel](https://wheels.vllm.ai/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl) and use the compiled libraries from there in the installation.
130
131
The `VLLM_PRECOMPILED_WHEEL_LOCATION` environment variable can be used instead of `VLLM_USE_PRECOMPILED` to specify a custom path or URL to the wheel file. For example, to use the [0.6.1.post1 PyPi wheel](https://pypi.org/project/vllm/#files):
132
133
```console
134
-$ export VLLM_PRECOMPILED_WHEEL_LOCATION=https://files.pythonhosted.org/packages/4a/4c/ee65ba33467a4c0de350ce29fbae39b9d0e7fcd887cc756fa993654d1228/vllm-0.6.3.post1-cp38-abi3-manylinux1_x86_64.whl
135
-$ pip install --editable .
134
+export VLLM_PRECOMPILED_WHEEL_LOCATION=https://files.pythonhosted.org/packages/4a/4c/ee65ba33467a4c0de350ce29fbae39b9d0e7fcd887cc756fa993654d1228/vllm-0.6.3.post1-cp38-abi3-manylinux1_x86_64.whl
135
+pip install --editable .
136
```
137
138
You can find more information about vLLM's wheels [above](#install-the-latest-code).
@@ -147,9 +147,9 @@ It is recommended to use the same commit ID for the source code as the vLLM whee
147
If you want to modify C++ or CUDA code, you'll need to build vLLM from source. This can take several minutes:
148
149
```console
150
-$ git clone https://github.com/vllm-project/vllm.git
151
-$ cd vllm
152
-$ pip install -e .
150
+git clone https://github.com/vllm-project/vllm.git
151
+cd vllm
152
+pip install -e .
153
```
154
155
```{tip}
@@ -172,11 +172,11 @@ There are scenarios where the PyTorch dependency cannot be easily installed via
172
To build vLLM using an existing PyTorch installation:
173
174
```console
175
-$ git clone https://github.com/vllm-project/vllm.git
176
-$ cd vllm
177
-$ python use_existing_torch.py
178
-$ pip install -r requirements-build.txt
179
-$ pip install -e . --no-build-isolation
175
+git clone https://github.com/vllm-project/vllm.git
176
+cd vllm
177
+python use_existing_torch.py
178
+pip install -r requirements-build.txt
179
+pip install -e . --no-build-isolation
180
```
181
182
#### Use the local cutlass for compilation
@@ -185,9 +185,9 @@ Currently, before starting the build process, vLLM fetches cutlass code from Git
185
To achieve this, you can set the environment variable VLLM_CUTLASS_SRC_DIR to point to your local cutlass directory.
186
187
```console
188
-$ git clone https://github.com/vllm-project/vllm.git
189
-$ cd vllm
190
-$ VLLM_CUTLASS_SRC_DIR=/path/to/cutlass pip install -e .
188
+git clone https://github.com/vllm-project/vllm.git
189
+cd vllm
190
+VLLM_CUTLASS_SRC_DIR=/path/to/cutlass pip install -e .
191
```
192
193
#### Troubleshooting
@@ -196,8 +196,8 @@ To avoid your system being overloaded, you can limit the number of compilation j
196
to be run simultaneously, via the environment variable `MAX_JOBS`. For example:
197
198
```console
199
-$ export MAX_JOBS=6
200
-$ pip install -e .
199
+export MAX_JOBS=6
200
+pip install -e .
201
```
202
203
This is especially useful when you are building on less powerful machines. For example, when you use WSL it only [assigns 50% of the total memory by default](https://learn.microsoft.com/en-us/windows/wsl/wsl-config#main-wsl-settings), so using `export MAX_JOBS=1` can avoid compiling multiple files simultaneously and running out of memory.
@@ -206,22 +206,22 @@ A side effect is a much slower build process.
206
Additionally, if you have trouble building vLLM, we recommend using the NVIDIA PyTorch Docker image.
207
208
```console
209
-$ # Use `--ipc=host` to make sure the shared memory is large enough.
210
-$ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
209
+# Use `--ipc=host` to make sure the shared memory is large enough.
210
+docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
211
```
212
213
If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from [the official website](https://developer.nvidia.com/cuda-toolkit-archive). After installation, set the environment variable `CUDA_HOME` to the installation path of CUDA Toolkit, and make sure that the `nvcc` compiler is in your `PATH`, e.g.:
214
215
```console
216
-$ export CUDA_HOME=/usr/local/cuda
217
-$ export PATH="${CUDA_HOME}/bin:$PATH"
216
+export CUDA_HOME=/usr/local/cuda
217
+export PATH="${CUDA_HOME}/bin:$PATH"
218
```
219
220
Here is a sanity check to verify that the CUDA Toolkit is correctly installed:
221
222
```console
223
-$ nvcc --version # verify that nvcc is in your PATH
224
-$ ${CUDA_HOME}/bin/nvcc --version # verify that nvcc is in your CUDA_HOME
223
+nvcc --version # verify that nvcc is in your PATH
224
+${CUDA_HOME}/bin/nvcc --version # verify that nvcc is in your CUDA_HOME
225
```
226
227
### Unsupported OS build
@@ -231,6 +231,6 @@ vLLM can fully run only on Linux but for development purposes, you can still bui
231
Simply disable the `VLLM_TARGET_DEVICE` environment variable before installing:
232
233
```console
234
-$ export VLLM_TARGET_DEVICE=empty
235
-$ pip install -e .
234
+export VLLM_TARGET_DEVICE=empty
235
+pip install -e .
236
```
docs/source/getting_started/installation/gpu-rocm.md
+58
-58
@@ -47,13 +47,13 @@ Their values can be passed in when running `docker build` with `--build-arg` opt
47
To build vllm on ROCm 6.2 for MI200 and MI300 series, you can use the default:
48
49
```console
50
-$ DOCKER_BUILDKIT=1 docker build -f Dockerfile.rocm -t vllm-rocm .
50
+DOCKER_BUILDKIT=1 docker build -f Dockerfile.rocm -t vllm-rocm .
51
```
52
53
To build vllm on ROCm 6.2 for Radeon RX7900 series (gfx1100), you should specify `BUILD_FA` as below:
54
55
```console
56
-$ DOCKER_BUILDKIT=1 docker build --build-arg BUILD_FA="0" -f Dockerfile.rocm -t vllm-rocm .
56
+DOCKER_BUILDKIT=1 docker build --build-arg BUILD_FA="0" -f Dockerfile.rocm -t vllm-rocm .
57
```
58
59
To run the above docker image `vllm-rocm`, use the below command:
@@ -83,81 +83,81 @@ Where the `<path/to/model>` is the location where the model is stored, for examp
83
- [ROCm](https://rocm.docs.amd.com/en/latest/deploy/linux/index.html)
84
- [PyTorch](https://pytorch.org/)
85
86
-For installing PyTorch, you can start from a fresh docker image, e.g, `rocm/pytorch:rocm6.2_ubuntu20.04_py3.9_pytorch_release_2.3.0`, `rocm/pytorch-nightly`.
86
+ For installing PyTorch, you can start from a fresh docker image, e.g, `rocm/pytorch:rocm6.2_ubuntu20.04_py3.9_pytorch_release_2.3.0`, `rocm/pytorch-nightly`.
87
88
-Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/)
88
+ Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/)
89
90
1. Install [Triton flash attention for ROCm](https://github.com/ROCm/triton)
91
92
-Install ROCm's Triton flash attention (the default triton-mlir branch) following the instructions from [ROCm/triton](https://github.com/ROCm/triton/blob/triton-mlir/README.md)
92
+ Install ROCm's Triton flash attention (the default triton-mlir branch) following the instructions from [ROCm/triton](https://github.com/ROCm/triton/blob/triton-mlir/README.md)
93
94
-```console
95
-$ python3 -m pip install ninja cmake wheel pybind11
96
-$ pip uninstall -y triton
97
-$ git clone https://github.com/OpenAI/triton.git
98
-$ cd triton
99
-$ git checkout e192dba
100
-$ cd python
101
-$ pip3 install .
102
-$ cd ../..
103
-```
94
+ ```console
95
+ python3 -m pip install ninja cmake wheel pybind11
96
+ pip uninstall -y triton
97
+ git clone https://github.com/OpenAI/triton.git
98
+ cd triton
99
+ git checkout e192dba
100
+ cd python
101
+ pip3 install .
102
+ cd ../..
103
+ ```
104
105
-```{note}
106
-- If you see HTTP issue related to downloading packages during building triton, please try again as the HTTP error is intermittent.
107
-```
105
+ ```{note}
106
+ - If you see HTTP issue related to downloading packages during building triton, please try again as the HTTP error is intermittent.
107
+ ```
108
109
2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/ROCm/flash-attention/tree/ck_tile)
110
111
-Install ROCm's flash attention (v2.5.9.post1) following the instructions from [ROCm/flash-attention](https://github.com/ROCm/flash-attention/tree/ck_tile#amd-gpurocm-support)
112
-Alternatively, wheels intended for vLLM use can be accessed under the releases.
111
+ Install ROCm's flash attention (v2.5.9.post1) following the instructions from [ROCm/flash-attention](https://github.com/ROCm/flash-attention/tree/ck_tile#amd-gpurocm-support)
112
+ Alternatively, wheels intended for vLLM use can be accessed under the releases.
113
114
-For example, for ROCm 6.2, suppose your gfx arch is `gfx90a`. To get your gfx architecture, run `rocminfo |grep gfx`.
114
+ For example, for ROCm 6.2, suppose your gfx arch is `gfx90a`. To get your gfx architecture, run `rocminfo |grep gfx`.
115
116
-```console
117
-$ git clone https://github.com/ROCm/flash-attention.git
118
-$ cd flash-attention
119
-$ git checkout 3cea2fb
120
-$ git submodule update --init
121
-$ GPU_ARCHS="gfx90a" python3 setup.py install
122
-$ cd ..
123
-```
116
+ ```console
117
+ git clone https://github.com/ROCm/flash-attention.git
118
+ cd flash-attention
119
+ git checkout 3cea2fb
120
+ git submodule update --init
121
+ GPU_ARCHS="gfx90a" python3 setup.py install
122
+ cd ..
123
+ ```
124
125
-```{note}
126
-- You might need to downgrade the "ninja" version to 1.10 it is not used when compiling flash-attention-2 (e.g. `pip install ninja==1.10.2.4`)
127
-```
125
+ ```{note}
126
+ - You might need to downgrade the "ninja" version to 1.10 it is not used when compiling flash-attention-2 (e.g. `pip install ninja==1.10.2.4`)
127
+ ```
128
129
3. Build vLLM. For example, vLLM on ROCM 6.2 can be built with the following steps:
130
131
-```bash
132
-$ pip install --upgrade pip
131
+ ```bash
132
+ $ pip install --upgrade pip
133
134
-# Install PyTorch
135
-$ pip uninstall torch -y
136
-$ pip install --no-cache-dir --pre torch==2.6.0.dev20241024 --index-url https://download.pytorch.org/whl/nightly/rocm6.2
134
+ # Install PyTorch
135
+ $ pip uninstall torch -y
136
+ $ pip install --no-cache-dir --pre torch==2.6.0.dev20241024 --index-url https://download.pytorch.org/whl/nightly/rocm6.2
137
138
-# Build & install AMD SMI
139
-$ pip install /opt/rocm/share/amd_smi
138
+ # Build & install AMD SMI
139
+ $ pip install /opt/rocm/share/amd_smi
140
141
-# Install dependencies
142
-$ pip install --upgrade numba scipy huggingface-hub[cli]
143
-$ pip install "numpy<2"
144
-$ pip install -r requirements-rocm.txt
141
+ # Install dependencies
142
+ $ pip install --upgrade numba scipy huggingface-hub[cli]
143
+ $ pip install "numpy<2"
144
+ $ pip install -r requirements-rocm.txt
145
146
-# Build vLLM for MI210/MI250/MI300.
147
-$ export PYTORCH_ROCM_ARCH="gfx90a;gfx942"
148
-$ python3 setup.py develop
149
-```
146
+ # Build vLLM for MI210/MI250/MI300.
147
+ $ export PYTORCH_ROCM_ARCH="gfx90a;gfx942"
148
+ $ python3 setup.py develop
149
+ ```
150
151
-This may take 5-10 minutes. Currently, `pip install .` does not work for ROCm installation.
151
+ This may take 5-10 minutes. Currently, `pip install .` does not work for ROCm installation.
152
153
-```{tip}
154
-- Triton flash attention is used by default. For benchmarking purposes, it is recommended to run a warm up step before collecting perf numbers.
155
-- Triton flash attention does not currently support sliding window attention. If using half precision, please use CK flash-attention for sliding window support.
156
-- To use CK flash-attention or PyTorch naive attention, please use this flag `export VLLM_USE_TRITON_FLASH_ATTN=0` to turn off triton flash attention.
157
-- The ROCm version of PyTorch, ideally, should match the ROCm driver version.
158
-```
153
+ ```{tip}
154
+ - Triton flash attention is used by default. For benchmarking purposes, it is recommended to run a warm up step before collecting perf numbers.
155
+ - Triton flash attention does not currently support sliding window attention. If using half precision, please use CK flash-attention for sliding window support.
156
+ - To use CK flash-attention or PyTorch naive attention, please use this flag `export VLLM_USE_TRITON_FLASH_ATTN=0` to turn off triton flash attention.
157
+ - The ROCm version of PyTorch, ideally, should match the ROCm driver version.
158
+ ```
159
160
-```{tip}
161
-- For MI300x (gfx942) users, to achieve optimal performance, please refer to [MI300x tuning guide](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html) for performance optimization and tuning tips on system and workflow level.
162
- For vLLM, please refer to [vLLM performance optimization](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/workload.html#vllm-performance-optimization).
163
-```
160
+ ```{tip}
161
+ - For MI300x (gfx942) users, to achieve optimal performance, please refer to [MI300x tuning guide](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html) for performance optimization and tuning tips on system and workflow level.
162
+ For vLLM, please refer to [vLLM performance optimization](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/workload.html#vllm-performance-optimization).
163
+ ```
docs/source/getting_started/installation/hpu-gaudi.md
+32
-32
@@ -22,8 +22,8 @@ Guide](https://docs.habana.ai/en/latest/PyTorch/Model_Optimization_PyTorch/Optim
22
### Quick start using Dockerfile
23
24
```console
25
-$ docker build -f Dockerfile.hpu -t vllm-hpu-env .
26
-$ docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --rm vllm-hpu-env
25
+docker build -f Dockerfile.hpu -t vllm-hpu-env .
26
+docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --rm vllm-hpu-env
27
```
28
29
```{tip}
@@ -37,10 +37,10 @@ If you're observing the following error: `docker: Error response from daemon: Un
37
To verify that the Intel Gaudi software was correctly installed, run:
38
39
```console
40
-$ hl-smi # verify that hl-smi is in your PATH and each Gaudi accelerator is visible
41
-$ apt list --installed | grep habana # verify that habanalabs-firmware-tools, habanalabs-graph, habanalabs-rdma-core, habanalabs-thunk and habanalabs-container-runtime are installed
42
-$ pip list | grep habana # verify that habana-torch-plugin, habana-torch-dataloader, habana-pyhlml and habana-media-loader are installed
43
-$ pip list | grep neural # verify that neural_compressor is installed
40
+hl-smi # verify that hl-smi is in your PATH and each Gaudi accelerator is visible
41
+apt list --installed | grep habana # verify that habanalabs-firmware-tools, habanalabs-graph, habanalabs-rdma-core, habanalabs-thunk and habanalabs-container-runtime are installed
42
+pip list | grep habana # verify that habana-torch-plugin, habana-torch-dataloader, habana-pyhlml and habana-media-loader are installed
43
+pip list | grep neural # verify that neural_compressor is installed
44
```
45
46
Refer to [Intel Gaudi Software Stack
@@ -57,8 +57,8 @@ for more details.
57
Use the following commands to run a Docker image:
58
59
```console
60
-$ docker pull vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest
61
-$ docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --ipc=host vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest
60
+docker pull vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest
61
+docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --ipc=host vault.habana.ai/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0:latest
62
```
63
64
#### Build and Install vLLM
@@ -66,18 +66,18 @@ $ docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_
66
To build and install vLLM from source, run:
67
68
```console
69
-$ git clone https://github.com/vllm-project/vllm.git
70
-$ cd vllm
71
-$ python setup.py develop
69
+git clone https://github.com/vllm-project/vllm.git
70
+cd vllm
71
+python setup.py develop
72
```
73
74
Currently, the latest features and performance optimizations are developed in Gaudi's [vLLM-fork](https://github.com/HabanaAI/vllm-fork) and we periodically upstream them to vLLM main repo. To install latest [HabanaAI/vLLM-fork](https://github.com/HabanaAI/vllm-fork), run the following:
75
76
```console
77
-$ git clone https://github.com/HabanaAI/vllm-fork.git
78
-$ cd vllm-fork
79
-$ git checkout habana_main
80
-$ python setup.py develop
77
+git clone https://github.com/HabanaAI/vllm-fork.git
78
+cd vllm-fork
79
+git checkout habana_main
80
+python setup.py develop
81
```
82
83
## Supported Features
@@ -181,7 +181,7 @@ Bucketing allows us to reduce the number of required graphs significantly, but i
181
182
Bucketing ranges are determined with 3 parameters - `min`, `step` and `max`. They can be set separately for prompt and decode phase, and for batch size and sequence length dimension. These parameters can be observed in logs during vLLM startup:
183
184
-```
184
+```text
185
INFO 08-01 21:37:59 hpu_model_runner.py:493] Prompt bucket config (min, step, max_warmup) bs:[1, 32, 4], seq:[128, 128, 1024]
186
INFO 08-01 21:37:59 hpu_model_runner.py:499] Generated 24 prompt buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024)]
187
INFO 08-01 21:37:59 hpu_model_runner.py:504] Decode bucket config (min, step, max_warmup) bs:[1, 128, 4], seq:[128, 128, 2048]
@@ -192,7 +192,7 @@ INFO 08-01 21:37:59 hpu_model_runner.py:509] Generated 48 decode buckets: [(1, 1
192
193
Example (with ramp-up)
194
195
-```
195
+```text
196
min = 2, step = 32, max = 64
197
=> ramp_up = (2, 4, 8, 16)
198
=> stable = (32, 64)
@@ -201,7 +201,7 @@ min = 2, step = 32, max = 64
201
202
Example (without ramp-up)
203
204
-```
204
+```text
205
min = 128, step = 128, max = 512
206
=> ramp_up = ()
207
=> stable = (128, 256, 384, 512)
@@ -224,7 +224,7 @@ Bucketing is transparent to a client -- padding in sequence length dimension is
224
225
Warmup is an optional, but highly recommended step occurring before vLLM server starts listening. It executes a forward pass for each bucket with dummy data. The goal is to pre-compile all graphs and not incur any graph compilation overheads within bucket boundaries during server runtime. Each warmup step is logged during vLLM startup:
226
227
-```
227
+```text
228
INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:79.16 GiB
229
INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][2/24] batch_size:4 seq_len:896 free_mem:55.43 GiB
230
INFO 08-01 22:26:48 hpu_model_runner.py:1066] [Warmup][Prompt][3/24] batch_size:4 seq_len:768 free_mem:55.43 GiB
@@ -273,7 +273,7 @@ When there's large amount of requests pending, vLLM scheduler will attempt to fi
273
274
Each described step is logged by vLLM server, as follows (negative values correspond to memory being released):
275
276
-```
276
+```text
277
INFO 08-02 17:37:44 hpu_model_runner.py:493] Prompt bucket config (min, step, max_warmup) bs:[1, 32, 4], seq:[128, 128, 1024]
278
INFO 08-02 17:37:44 hpu_model_runner.py:499] Generated 24 prompt buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024)]
279
INFO 08-02 17:37:44 hpu_model_runner.py:504] Decode bucket config (min, step, max_warmup) bs:[1, 128, 4], seq:[128, 128, 2048]
@@ -349,19 +349,19 @@ INFO 08-02 17:38:43 hpu_executor.py:91] init_cache_engine took 37.92 GiB of devi
349
- Default values:
350
351
- Prompt:
352
- : - batch size min (`VLLM_PROMPT_BS_BUCKET_MIN`): `1`
353
- - batch size step (`VLLM_PROMPT_BS_BUCKET_STEP`): `min(max_num_seqs, 32)`
354
- - batch size max (`VLLM_PROMPT_BS_BUCKET_MAX`): `min(max_num_seqs, 64)`
355
- - sequence length min (`VLLM_PROMPT_SEQ_BUCKET_MIN`): `block_size`
356
- - sequence length step (`VLLM_PROMPT_SEQ_BUCKET_STEP`): `block_size`
357
- - sequence length max (`VLLM_PROMPT_SEQ_BUCKET_MAX`): `max_model_len`
352
+ - batch size min (`VLLM_PROMPT_BS_BUCKET_MIN`): `1`
353
+ - batch size step (`VLLM_PROMPT_BS_BUCKET_STEP`): `min(max_num_seqs, 32)`
354
+ - batch size max (`VLLM_PROMPT_BS_BUCKET_MAX`): `min(max_num_seqs, 64)`
355
+ - sequence length min (`VLLM_PROMPT_SEQ_BUCKET_MIN`): `block_size`
356
+ - sequence length step (`VLLM_PROMPT_SEQ_BUCKET_STEP`): `block_size`
357
+ - sequence length max (`VLLM_PROMPT_SEQ_BUCKET_MAX`): `max_model_len`
358
- Decode:
359
- : - batch size min (`VLLM_DECODE_BS_BUCKET_MIN`): `1`
360
- - batch size step (`VLLM_DECODE_BS_BUCKET_STEP`): `min(max_num_seqs, 32)`
361
- - batch size max (`VLLM_DECODE_BS_BUCKET_MAX`): `max_num_seqs`
362
- - sequence length min (`VLLM_DECODE_BLOCK_BUCKET_MIN`): `block_size`
363
- - sequence length step (`VLLM_DECODE_BLOCK_BUCKET_STEP`): `block_size`
364
- - sequence length max (`VLLM_DECODE_BLOCK_BUCKET_MAX`): `max(128, (max_num_seqs*max_model_len)/block_size)`
359
+ - batch size min (`VLLM_DECODE_BS_BUCKET_MIN`): `1`
360
+ - batch size step (`VLLM_DECODE_BS_BUCKET_STEP`): `min(max_num_seqs, 32)`
361
+ - batch size max (`VLLM_DECODE_BS_BUCKET_MAX`): `max_num_seqs`
362
+ - sequence length min (`VLLM_DECODE_BLOCK_BUCKET_MIN`): `block_size`
363
+ - sequence length step (`VLLM_DECODE_BLOCK_BUCKET_STEP`): `block_size`
364
+ - sequence length max (`VLLM_DECODE_BLOCK_BUCKET_MAX`): `max(128, (max_num_seqs*max_model_len)/block_size)`
365
366
Additionally, there are HPU PyTorch Bridge environment variables impacting vLLM execution:
367
docs/source/getting_started/installation/neuron.md
+4
-4
@@ -123,10 +123,10 @@ python -m pip install --upgrade neuronx-cc==2.* --pre torch-neuronx==2.1.* torch
123
Once neuronx-cc and transformers-neuronx packages are installed, we will be able to install vllm as follows:
124
125
```console
126
-$ git clone https://github.com/vllm-project/vllm.git
127
-$ cd vllm
128
-$ pip install -U -r requirements-neuron.txt
129
-$ VLLM_TARGET_DEVICE="neuron" pip install .
126
+git clone https://github.com/vllm-project/vllm.git
127
+cd vllm
128
+pip install -U -r requirements-neuron.txt
129
+VLLM_TARGET_DEVICE="neuron" pip install .
130
```
131
132
If neuron packages are detected correctly in the installation process, `vllm-0.3.0+neuron212` will be installed.
docs/source/getting_started/installation/openvino.md
+7
-7
@@ -27,8 +27,8 @@ vLLM powered by OpenVINO supports all LLM models from [vLLM supported models lis
27
## Quick start using Dockerfile
28
29
```console
30
-$ docker build -f Dockerfile.openvino -t vllm-openvino-env .
31
-$ docker run -it --rm vllm-openvino-env
30
+docker build -f Dockerfile.openvino -t vllm-openvino-env .
31
+docker run -it --rm vllm-openvino-env
32
```
33
34
(install-openvino-backend-from-source)=
@@ -38,21 +38,21 @@ $ docker run -it --rm vllm-openvino-env
38
- First, install Python. For example, on Ubuntu 22.04, you can run:
39
40
```console
41
- $ sudo apt-get update -y
42
- $ sudo apt-get install python3
41
+ sudo apt-get update -y
42
+ sudo apt-get install python3
43
```
44
45
- Second, install prerequisites vLLM OpenVINO backend installation:
46
47
```console
48
- $ pip install --upgrade pip
49
- $ pip install -r requirements-build.txt --extra-index-url https://download.pytorch.org/whl/cpu
48
+ pip install --upgrade pip
49
+ pip install -r requirements-build.txt --extra-index-url https://download.pytorch.org/whl/cpu
50
```
51
52
- Finally, install vLLM with OpenVINO backend:
53
54
```console
55
- $ PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu" VLLM_TARGET_DEVICE=openvino python -m pip install -v .
55
+ PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu" VLLM_TARGET_DEVICE=openvino python -m pip install -v .
56
```
57
58
- [Optional] To use vLLM OpenVINO backend with a GPU device, ensure your system is properly set up. Follow the instructions provided here: [https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html](https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-gpu.html).
docs/source/getting_started/installation/tpu.md
+3
-3
@@ -156,14 +156,14 @@ For more information about using TPUs with GKE, see
156
You can use <gh-file:Dockerfile.tpu> to build a Docker image with TPU support.
157
158
```console
159
-$ docker build -f Dockerfile.tpu -t vllm-tpu .
159
+docker build -f Dockerfile.tpu -t vllm-tpu .
160
```
161
162
Run the Docker image with the following command:
163
164
```console
165
-$ # Make sure to add `--privileged --net host --shm-size=16G`.
166
-$ docker run --privileged --net host --shm-size=16G -it vllm-tpu
165
+# Make sure to add `--privileged --net host --shm-size=16G`.
166
+docker run --privileged --net host --shm-size=16G -it vllm-tpu
167
```
168
169
```{note}
docs/source/getting_started/installation/xpu.md
+12
-12
@@ -40,15 +40,15 @@ $ docker run -it \
40
- Second, install Python packages for vLLM XPU backend building:
41
42
```console
43
-$ source /opt/intel/oneapi/setvars.sh
44
-$ pip install --upgrade pip
45
-$ pip install -v -r requirements-xpu.txt
43
+source /opt/intel/oneapi/setvars.sh
44
+pip install --upgrade pip
45
+pip install -v -r requirements-xpu.txt
46
```
47
48
- Finally, build and install vLLM XPU backend:
49
50
```console
51
-$ VLLM_TARGET_DEVICE=xpu python setup.py install
51
+VLLM_TARGET_DEVICE=xpu python setup.py install
52
```
53
54
```{note}
@@ -61,14 +61,14 @@ $ VLLM_TARGET_DEVICE=xpu python setup.py install
61
XPU platform supports tensor-parallel inference/serving and also supports pipeline parallel as a beta feature for online serving. We requires Ray as the distributed runtime backend. For example, a reference execution likes following:
62
63
```console
64
-$ python -m vllm.entrypoints.openai.api_server \
65
-$ --model=facebook/opt-13b \
66
-$ --dtype=bfloat16 \
67
-$ --device=xpu \
68
-$ --max_model_len=1024 \
69
-$ --distributed-executor-backend=ray \
70
-$ --pipeline-parallel-size=2 \
71
-$ -tp=8
64
+python -m vllm.entrypoints.openai.api_server \
65
+ --model=facebook/opt-13b \
66
+ --dtype=bfloat16 \
67
+ --device=xpu \
68
+ --max_model_len=1024 \
69
+ --distributed-executor-backend=ray \
70
+ --pipeline-parallel-size=2 \
71
+ -tp=8
72
```
73
74
By default, a ray instance will be launched automatically if no existing one is detected in system, with `num-gpus` equals to `parallel_config.world_size`. We recommend properly starting a ray cluster before execution, referring to the <gh-file:examples/online_serving/run_cluster.sh> helper script.
docs/source/getting_started/quickstart.md
+25
-25
@@ -19,17 +19,17 @@ If you are using NVIDIA GPUs, you can install vLLM using [pip](https://pypi.org/
19
It's recommended to use [uv](https://docs.astral.sh/uv/), a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment and install vLLM using the following commands:
20
21
```console
22
-$ uv venv myenv --python 3.12 --seed
23
-$ source myenv/bin/activate
24
-$ uv pip install vllm
22
+uv venv myenv --python 3.12 --seed
23
+source myenv/bin/activate
24
+uv pip install vllm
25
```
26
27
You can also use [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) to create and manage Python environments.
28
29
```console
30
-$ conda create -n myenv python=3.12 -y
31
-$ conda activate myenv
32
-$ pip install vllm
30
+conda create -n myenv python=3.12 -y
31
+conda activate myenv
32
+pip install vllm
33
```
34
35
```{note}
@@ -94,7 +94,7 @@ By default, it starts the server at `http://localhost:8000`. You can specify the
94
Run the following command to start the vLLM server with the [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) model:
95
96
```console
97
-$ vllm serve Qwen/Qwen2.5-1.5B-Instruct
97
+vllm serve Qwen/Qwen2.5-1.5B-Instruct
98
```
99
100
```{note}
@@ -105,7 +105,7 @@ You can learn about overriding it [here](#chat-template).
105
This server can be queried in the same format as OpenAI API. For example, to list the models:
106
107
```console
108
-$ curl http://localhost:8000/v1/models
108
+curl http://localhost:8000/v1/models
109
```
110
111
You can pass in the argument `--api-key` or environment variable `VLLM_API_KEY` to enable the server to check for API key in the header.
@@ -115,14 +115,14 @@ You can pass in the argument `--api-key` or environment variable `VLLM_API_KEY`
115
Once your server is started, you can query the model with input prompts:
116
117
```console
118
-$ curl http://localhost:8000/v1/completions \
119
-$ -H "Content-Type: application/json" \
120
-$ -d '{
121
-$ "model": "Qwen/Qwen2.5-1.5B-Instruct",
122
-$ "prompt": "San Francisco is a",
123
-$ "max_tokens": 7,
124
-$ "temperature": 0
125
-$ }'
118
+curl http://localhost:8000/v1/completions \
119
+ -H "Content-Type: application/json" \
120
+ -d '{
121
+ "model": "Qwen/Qwen2.5-1.5B-Instruct",
122
+ "prompt": "San Francisco is a",
123
+ "max_tokens": 7,
124
+ "temperature": 0
125
+ }'
126
```
127
128
Since this server is compatible with OpenAI API, you can use it as a drop-in replacement for any applications using OpenAI API. For example, another way to query the server is via the `openai` Python package:
@@ -151,15 +151,15 @@ vLLM is designed to also support the OpenAI Chat Completions API. The chat inter
151
You can use the [create chat completion](https://platform.openai.com/docs/api-reference/chat/completions/create) endpoint to interact with the model:
152
153
```console
154
-$ curl http://localhost:8000/v1/chat/completions \
155
-$ -H "Content-Type: application/json" \
156
-$ -d '{
157
-$ "model": "Qwen/Qwen2.5-1.5B-Instruct",
158
-$ "messages": [
159
-$ {"role": "system", "content": "You are a helpful assistant."},
160
-$ {"role": "user", "content": "Who won the world series in 2020?"}
161
-$ ]
162
-$ }'
154
+curl http://localhost:8000/v1/chat/completions \
155
+ -H "Content-Type: application/json" \
156
+ -d '{
157
+ "model": "Qwen/Qwen2.5-1.5B-Instruct",
158
+ "messages": [
159
+ {"role": "system", "content": "You are a helpful assistant."},
160
+ {"role": "user", "content": "Who won the world series in 2020?"}
161
+ ]
162
+ }'
163
```
164
165
Alternatively, you can use the `openai` Python package:
docs/source/getting_started/troubleshooting.md
+4
-2
@@ -48,6 +48,7 @@ If vLLM crashes and the error trace captures it somewhere around `self.graph.rep
48
To identify the particular CUDA operation that causes the error, you can add `--enforce-eager` to the command line, or `enforce_eager=True` to the {class}`~vllm.LLM` class to disable the CUDAGraph optimization and isolate the exact CUDA operation that causes the error.
49
50
(troubleshooting-incorrect-hardware-driver)=
51
+
52
## Incorrect hardware/driver
53
54
If GPU/CPU communication cannot be established, you can use the following Python script and follow the instructions below to confirm whether the GPU/CPU communication is working correctly.
@@ -118,13 +119,13 @@ dist.destroy_process_group()
119
If you are testing with a single node, adjust `--nproc-per-node` to the number of GPUs you want to use:
120
121
```console
121
-$ NCCL_DEBUG=TRACE torchrun --nproc-per-node=<number-of-GPUs> test.py
122
+NCCL_DEBUG=TRACE torchrun --nproc-per-node=<number-of-GPUs> test.py
123
```
124
125
If you are testing with multi-nodes, adjust `--nproc-per-node` and `--nnodes` according to your setup and set `MASTER_ADDR` to the correct IP address of the master node, reachable from all nodes. Then, run:
126
127
```console
127
-$ NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR test.py
128
+NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR test.py
129
```
130
131
If the script runs successfully, you should see the message `sanity check is successful!`.
@@ -141,6 +142,7 @@ Adjust `--nproc-per-node`, `--nnodes`, and `--node-rank` according to your setup
142
```
143
144
(troubleshooting-python-multiprocessing)=
145
+
146
## Python multiprocessing
147
148
### `RuntimeError` Exception
docs/source/index.md
+2
-2
@@ -1,4 +1,4 @@
1
-# Welcome to vLLM!
1
+# Welcome to vLLM
2
3
```{figure} ./assets/logos/vllm-logo-text-light.png
4
:align: center
@@ -186,7 +186,7 @@ community/meetups
186
community/sponsors
187
```
188
189
-# Indices and tables
189
+## Indices and tables
190
191
- {ref}`genindex`
192
- {ref}`modindex`
docs/source/models/extensions/runai_model_streamer.md
+6
-6
@@ -9,25 +9,25 @@ vLLM supports loading weights in Safetensors format using the Run:ai Model Strea
9
You first need to install vLLM RunAI optional dependency:
10
11
```console
12
-$ pip3 install vllm[runai]
12
+pip3 install vllm[runai]
13
```
14
15
To run it as an OpenAI-compatible server, add the `--load-format runai_streamer` flag:
16
17
```console
18
-$ vllm serve /home/meta-llama/Llama-3.2-3B-Instruct --load-format runai_streamer
18
+vllm serve /home/meta-llama/Llama-3.2-3B-Instruct --load-format runai_streamer
19
```
20
21
To run model from AWS S3 object store run:
22
23
```console
24
-$ vllm serve s3://core-llm/Llama-3-8b --load-format runai_streamer
24
+vllm serve s3://core-llm/Llama-3-8b --load-format runai_streamer
25
```
26
27
To run model from a S3 compatible object store run:
28
29
```console
30
-$ RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING=0 AWS_EC2_METADATA_DISABLED=true AWS_ENDPOINT_URL=https://storage.googleapis.com vllm serve s3://core-llm/Llama-3-8b --load-format runai_streamer
30
+RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING=0 AWS_EC2_METADATA_DISABLED=true AWS_ENDPOINT_URL=https://storage.googleapis.com vllm serve s3://core-llm/Llama-3-8b --load-format runai_streamer
31
```
32
33
## Tunable parameters
@@ -38,14 +38,14 @@ You can tune `concurrency` that controls the level of concurrency and number of
38
For reading from S3, it will be the number of client instances the host is opening to the S3 server.
39
40
```console
41
-$ vllm serve /home/meta-llama/Llama-3.2-3B-Instruct --load-format runai_streamer --model-loader-extra-config '{"concurrency":16}'
41
+vllm serve /home/meta-llama/Llama-3.2-3B-Instruct --load-format runai_streamer --model-loader-extra-config '{"concurrency":16}'
42
```
43
44
You can control the size of the CPU Memory buffer to which tensors are read from the file, and limit this size.
45
You can read further about CPU buffer memory limiting [here](https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/env-vars.md#runai_streamer_memory_limit).
46
47
```console
48
-$ vllm serve /home/meta-llama/Llama-3.2-3B-Instruct --load-format runai_streamer --model-loader-extra-config '{"memory_limit":5368709120}'
48
+vllm serve /home/meta-llama/Llama-3.2-3B-Instruct --load-format runai_streamer --model-loader-extra-config '{"memory_limit":5368709120}'
49
```
50
51
```{note}
docs/source/models/supported_models.md
+8
-5
@@ -45,7 +45,7 @@ Alternatively, you can [open an issue on GitHub](https://github.com/vllm-project
45
To use models from [ModelScope](https://www.modelscope.cn) instead of HuggingFace Hub, set an environment variable:
46
47
```shell
48
-$ export VLLM_USE_MODELSCOPE=True
48
+export VLLM_USE_MODELSCOPE=True
49
```
50
51
And use with `trust_remote_code=True`.
@@ -820,19 +820,22 @@ The following table lists those that are tested in vLLM.
820
821
_________________
822
823
-# Model Support Policy
823
+## Model Support Policy
824
825
At vLLM, we are committed to facilitating the integration and support of third-party models within our ecosystem. Our approach is designed to balance the need for robustness and the practical limitations of supporting a wide range of models. Here’s how we manage third-party model support:
826
827
1. **Community-Driven Support**: We encourage community contributions for adding new models. When a user requests support for a new model, we welcome pull requests (PRs) from the community. These contributions are evaluated primarily on the sensibility of the output they generate, rather than strict consistency with existing implementations such as those in transformers. **Call for contribution:** PRs coming directly from model vendors are greatly appreciated!
828
+
829
2. **Best-Effort Consistency**: While we aim to maintain a level of consistency between the models implemented in vLLM and other frameworks like transformers, complete alignment is not always feasible. Factors like acceleration techniques and the use of low-precision computations can introduce discrepancies. Our commitment is to ensure that the implemented models are functional and produce sensible results.
830
830
-```{tip}
831
-When comparing the output of `model.generate` from HuggingFace Transformers with the output of `llm.generate` from vLLM, note that the former reads the model's generation config file (i.e., [generation_config.json](https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/generation/utils.py#L1945)) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.
832
-```
831
+ ```{tip}
832
+ When comparing the output of `model.generate` from HuggingFace Transformers with the output of `llm.generate` from vLLM, note that the former reads the model's generation config file (i.e., [generation_config.json](https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/generation/utils.py#L1945)) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.
833
+ ```
834
835
3. **Issue Resolution and Model Updates**: Users are encouraged to report any bugs or issues they encounter with third-party models. Proposed fixes should be submitted via PRs, with a clear explanation of the problem and the rationale behind the proposed solution. If a fix for one model impacts another, we rely on the community to highlight and address these cross-model dependencies. Note: for bugfix PRs, it is good etiquette to inform the original author to seek their feedback.
836
+
837
4. **Monitoring and Updates**: Users interested in specific models should monitor the commit history for those models (e.g., by tracking changes in the main/vllm/model_executor/models directory). This proactive approach helps users stay informed about updates and changes that may affect the models they use.
838
+
839
5. **Selective Focus**: Our resources are primarily directed towards models with significant user interest and impact. Models that are less frequently used may receive less attention, and we rely on the community to play a more active role in their upkeep and improvement.
840
841
Through this approach, vLLM fosters a collaborative environment where both the core development team and the broader community contribute to the robustness and diversity of the third-party models supported in our ecosystem.
docs/source/performance/optimization.md
+1
-1
@@ -8,7 +8,7 @@ Due to the auto-regressive nature of transformer architecture, there are times w
8
The vLLM can preempt requests to free up KV cache space for other requests. Preempted requests are recomputed when sufficient KV cache space becomes
9
available again. When this occurs, the following warning is printed:
10
11
-```
11
+```text
12
WARNING 05-09 00:49:33 scheduler.py:1057 Sequence group 0 is preempted by PreemptionMode.SWAP mode because there is not enough KV cache space. This can affect the end-to-end performance. Increase gpu_memory_utilization or tensor_parallel_size to provide more KV cache memory. total_cumulative_preemption_cnt=1
13
```
14
docs/source/serving/distributed_serving.md
+20
-20
@@ -35,16 +35,16 @@ output = llm.generate("San Franciso is a")
35
To run multi-GPU serving, pass in the `--tensor-parallel-size` argument when starting the server. For example, to run API server on 4 GPUs:
36
37
```console
38
-$ vllm serve facebook/opt-13b \
39
-$ --tensor-parallel-size 4
38
+ vllm serve facebook/opt-13b \
39
+ --tensor-parallel-size 4
40
```
41
42
You can also additionally specify `--pipeline-parallel-size` to enable pipeline parallelism. For example, to run API server on 8 GPUs with pipeline parallelism and tensor parallelism:
43
44
```console
45
-$ vllm serve gpt2 \
46
-$ --tensor-parallel-size 4 \
47
-$ --pipeline-parallel-size 2
45
+ vllm serve gpt2 \
46
+ --tensor-parallel-size 4 \
47
+ --pipeline-parallel-size 2
48
```
49
50
## Running vLLM on multiple nodes
@@ -56,21 +56,21 @@ The first step, is to start containers and organize them into a cluster. We have
56
Pick a node as the head node, and run the following command:
57
58
```console
59
-$ bash run_cluster.sh \
60
-$ vllm/vllm-openai \
61
-$ ip_of_head_node \
62
-$ --head \
63
-$ /path/to/the/huggingface/home/in/this/node
59
+bash run_cluster.sh \
60
+ vllm/vllm-openai \
61
+ ip_of_head_node \
62
+ --head \
63
+ /path/to/the/huggingface/home/in/this/node
64
```
65
66
On the rest of the worker nodes, run the following command:
67
68
```console
69
-$ bash run_cluster.sh \
70
-$ vllm/vllm-openai \
71
-$ ip_of_head_node \
72
-$ --worker \
73
-$ /path/to/the/huggingface/home/in/this/node
69
+bash run_cluster.sh \
70
+ vllm/vllm-openai \
71
+ ip_of_head_node \
72
+ --worker \
73
+ /path/to/the/huggingface/home/in/this/node
74
```
75
76
Then you get a ray cluster of containers. Note that you need to keep the shells running these commands alive to hold the cluster. Any shell disconnect will terminate the cluster. In addition, please note that the argument `ip_of_head_node` should be the IP address of the head node, which is accessible by all the worker nodes. A common misunderstanding is to use the IP address of the worker node, which is not correct.
@@ -80,16 +80,16 @@ Then, on any node, use `docker exec -it node /bin/bash` to enter the container,
80
After that, on any node, you can use vLLM as usual, just as you have all the GPUs on one node. The common practice is to set the tensor parallel size to the number of GPUs in each node, and the pipeline parallel size to the number of nodes. For example, if you have 16 GPUs in 2 nodes (8 GPUs per node), you can set the tensor parallel size to 8 and the pipeline parallel size to 2:
81
82
```console
83
-$ vllm serve /path/to/the/model/in/the/container \
84
-$ --tensor-parallel-size 8 \
85
-$ --pipeline-parallel-size 2
83
+ vllm serve /path/to/the/model/in/the/container \
84
+ --tensor-parallel-size 8 \
85
+ --pipeline-parallel-size 2
86
```
87
88
You can also use tensor parallel without pipeline parallel, just set the tensor parallel size to the number of GPUs in the cluster. For example, if you have 16 GPUs in 2 nodes (8 GPUs per node), you can set the tensor parallel size to 16:
89
90
```console
91
-$ vllm serve /path/to/the/model/in/the/container \
92
-$ --tensor-parallel-size 16
91
+vllm serve /path/to/the/model/in/the/container \
92
+ --tensor-parallel-size 16
93
```
94
95
To make tensor parallel performant, you should make sure the communication between nodes is efficient, e.g. using high-speed network cards like Infiniband. To correctly set up the cluster to use Infiniband, append additional arguments like `--privileged -e NCCL_IB_HCA=mlx5` to the `run_cluster.sh` script. Please contact your system administrator for more information on how to set up the flags. One way to confirm if the Infiniband is working is to run vLLM with `NCCL_DEBUG=TRACE` environment variable set, e.g. `NCCL_DEBUG=TRACE vllm serve ...` and check the logs for the NCCL version and the network used. If you find `[send] via NET/Socket` in the logs, it means NCCL uses raw TCP Socket, which is not efficient for cross-node tensor parallel. If you find `[send] via NET/IB/GDRDMA` in the logs, it means NCCL uses Infiniband with GPU-Direct RDMA, which is efficient.
docs/source/serving/integrations/langchain.md
+1
-1
@@ -7,7 +7,7 @@ vLLM is also available via [LangChain](https://github.com/langchain-ai/langchain
7
To install LangChain, run
8
9
```console
10
-$ pip install langchain langchain_community -q
10
+pip install langchain langchain_community -q
11
```
12
13
To run inference on a single or multiple GPUs, use `VLLM` class from `langchain`.
docs/source/serving/integrations/llamaindex.md
+1
-1
@@ -7,7 +7,7 @@ vLLM is also available via [LlamaIndex](https://github.com/run-llama/llama_index
7
To install LlamaIndex, run
8
9
```console
10
-$ pip install llama-index-llms-vllm -q
10
+pip install llama-index-llms-vllm -q
11
```
12
13
To run inference on a single or multiple GPUs, use `Vllm` class from `llamaindex`.
docs/source/serving/metrics.md
+1
-1
@@ -7,7 +7,7 @@ OpenAI compatible API server.
7
You can start the server using Python, or using [Docker](#deployment-docker):
8
9
```console
10
-$ vllm serve unsloth/Llama-3.2-1B-Instruct
10
+vllm serve unsloth/Llama-3.2-1B-Instruct
11
```
12
13
Then query the endpoint to get the latest metrics from the server:
docs/source/serving/multimodal_inputs.md
+1
@@ -303,6 +303,7 @@ vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --task generate --max-model
303
```
304
305
Then, you can use the OpenAI client as follows:
306
+
307
```python
308
from openai import OpenAI
309
docs/source/serving/offline_inference.md
+1
-1
@@ -64,7 +64,7 @@ Dynamic quantization is also supported via the `quantization` option -- see [her
64
65
#### Context length and batch size
66
67
-You can further reduce memory usage by limit the context length of the model (`max_model_len` option)
67
+You can further reduce memory usage by limiting the context length of the model (`max_model_len` option)
68
and the maximum batch size (`max_num_seqs` option).
69
70
```python
docs/source/serving/openai_compatible_server.md
+15
-4
@@ -5,11 +5,13 @@
5
vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more!
6
7
You can start the server via the [`vllm serve`](#vllm-serve) command, or through [Docker](#deployment-docker):
8
+
9
```bash
10
vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
11
```
12
13
To call the server, you can use the [official OpenAI Python client](https://github.com/openai/openai-python), or any other HTTP client.
14
+
15
```python
16
from openai import OpenAI
17
client = OpenAI(
@@ -50,6 +52,7 @@ In addition, we have the following custom APIs:
52
- Only applicable to [cross-encoder models](../models/pooling_models.md) (`--task score`).
53
54
(chat-template)=
55
+
56
## Chat Template
57
58
In order for the language model to support chat protocol, vLLM requires the model to include
@@ -71,6 +74,7 @@ vLLM community provides a set of chat templates for popular models. You can find
74
75
With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies
76
both a `type` and a `text` field. An example is provided below:
77
+
78
```python
79
completion = client.chat.completions.create(
80
model="NousResearch/Meta-Llama-3-8B-Instruct",
@@ -80,7 +84,7 @@ completion = client.chat.completions.create(
84
)
85
```
86
83
-Most chat templates for LLMs expect the `content` field to be a string, but there are some newer models like
87
+Most chat templates for LLMs expect the `content` field to be a string, but there are some newer models like
88
`meta-llama/Llama-Guard-3-1B` that expect the content to be formatted according to the OpenAI schema in the
89
request. vLLM provides best-effort support to detect this automatically, which is logged as a string like
90
*"Detected the chat template content format to be..."*, and internally converts incoming requests to match
@@ -115,12 +119,12 @@ completion = client.chat.completions.create(
119
## Extra HTTP Headers
120
121
Only `X-Request-Id` HTTP request header is supported for now. It can be enabled
118
-with `--enable-request-id-headers`.
122
+with `--enable-request-id-headers`.
123
124
> Note that enablement of the headers can impact performance significantly at high QPS
125
> rates. We recommend implementing HTTP headers at the router level (e.g. via Istio),
126
> rather than within the vLLM layer for this reason.
123
-> See https://github.com/vllm-project/vllm/pull/11529 for more details.
127
+> See [this PR](https://github.com/vllm-project/vllm/pull/11529) for more details.
128
129
```python
130
completion = client.chat.completions.create(
@@ -147,6 +151,7 @@ print(completion._request_id)
151
## CLI Reference
152
153
(vllm-serve)=
154
+
155
### `vllm serve`
156
157
The `vllm serve` command is used to launch the OpenAI-compatible server.
@@ -175,7 +180,7 @@ uvicorn-log-level: "info"
180
To use the above config file:
181
182
```bash
178
-$ vllm serve SOME_MODEL --config config.yaml
183
+vllm serve SOME_MODEL --config config.yaml
184
```
185
186
```{note}
@@ -186,6 +191,7 @@ The order of priorities is `command line > config file values > defaults`.
191
## API Reference
192
193
(completions-api)=
194
+
195
### Completions API
196
197
Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions);
@@ -212,6 +218,7 @@ The following extra parameters are supported:
218
```
219
220
(chat-api)=
221
+
222
### Chat API
223
224
Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat);
@@ -243,6 +250,7 @@ The following extra parameters are supported:
250
```
251
252
(embeddings-api)=
253
+
254
### Embeddings API
255
256
Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);
@@ -284,6 +292,7 @@ For chat-like input (i.e. if `messages` is passed), these extra parameters are s
292
```
293
294
(tokenizer-api)=
295
+
296
### Tokenizer API
297
298
Our Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).
@@ -293,6 +302,7 @@ It consists of two endpoints:
302
- `/detokenize` corresponds to calling `tokenizer.decode()`.
303
304
(pooling-api)=
305
+
306
### Pooling API
307
308
Our Pooling API encodes input prompts using a [pooling model](../models/pooling_models.md) and returns the corresponding hidden states.
@@ -302,6 +312,7 @@ The input format is the same as [Embeddings API](#embeddings-api), but the outpu
312
Code example: <gh-file:examples/online_serving/openai_pooling_client.py>
313
314
(score-api)=
315
+
316
### Score API
317
318
Our Score API applies a cross-encoder model to predict scores for sentence pairs.
format.sh
+5
-5
@@ -41,7 +41,7 @@ MYPY_VERSION=$(mypy --version | awk '{print $2}')
41
CODESPELL_VERSION=$(codespell --version)
42
ISORT_VERSION=$(isort --vn)
43
CLANGFORMAT_VERSION=$(clang-format --version | awk '{print $3}')
44
-SPHINX_LINT_VERSION=$(sphinx-lint --version | awk '{print $2}')
44
+PYMARKDOWNLNT_VERSION=$(pymarkdownlnt version | awk '{print $1}')
45
46
# # params: tool name, tool version, required version
47
tool_version_check() {
@@ -58,7 +58,7 @@ tool_version_check "mypy" "$MYPY_VERSION"
58
tool_version_check "isort" "$ISORT_VERSION"
59
tool_version_check "codespell" "$CODESPELL_VERSION"
60
tool_version_check "clang-format" "$CLANGFORMAT_VERSION"
61
-tool_version_check "sphinx-lint" "$SPHINX_LINT_VERSION"
61
+tool_version_check "pymarkdownlnt" "$PYMARKDOWNLNT_VERSION"
62
63
YAPF_FLAGS=(
64
'--recursive'
@@ -316,6 +316,6 @@ else
316
echo "✨🎉 Format check passed! Congratulations! 🎉✨"
317
fi
318
319
-echo 'vLLM sphinx-lint:'
320
-tools/sphinx-lint.sh
321
-echo 'vLLM sphinx-lint: Done'
319
+echo 'vLLM doc-lint:'
320
+tools/doc-lint.sh
321
+echo 'vLLM doc-lint: Done'
pyproject.toml
+6
@@ -101,3 +101,9 @@ markers = [
101
"skip_v1: do not run this test with v1",
102
"optional: optional tests that are automatically skipped, include --optional to run them",
103
]
104
+
105
+[tool.pymarkdown]
106
+plugins.md013.enabled = false # line-length
107
+plugins.md041.enabled = false # first-line-h1
108
+plugins.md033.enabled = false # inline-html
109
+plugins.md024.allow_different_nesting = true # no-duplicate-headers
requirements-lint.txt
+1
-1
@@ -6,7 +6,7 @@ ruff==0.6.5
6
codespell==2.3.0
7
isort==5.13.2
8
clang-format==18.1.5
9
-sphinx-lint==1.0.0
9
+pymarkdownlnt==0.9.26
10
11
# type checking
12
mypy==1.11.1
tools/doc-lint.sh
new
+3
@@ -0,0 +1,3 @@
1
+#!/bin/bash
2
+
3
+pymarkdownlnt scan docs -r
tools/sphinx-lint.sh
deleted
-3
@@ -1,3 +0,0 @@
1
-#!/bin/bash
2
-
3
-sphinx-lint --disable trailing-whitespace,missing-final-newline docs