@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 447 lines 13.6 KB
Rendered Raw
1 # Using Kubernetes
2
3 Deploying vLLM on Kubernetes is a scalable and efficient way to serve machine learning models. This guide walks you through deploying vLLM using native Kubernetes.
4
5 - [Deployment with CPUs](#deployment-with-cpus)
6 - [Deployment with GPUs](#deployment-with-gpus)
7 - [Serving with gRPC](#serving-with-grpc)
8 - [Troubleshooting](#troubleshooting)
9 - [Startup Probe or Readiness Probe Failure, container log contains "KeyboardInterrupt: terminated"](#startup-probe-or-readiness-probe-failure-container-log-contains-keyboardinterrupt-terminated)
10 - [Conclusion](#conclusion)
11
12 Alternatively, you can deploy vLLM to Kubernetes using any of the following:
13
14 - [Helm](frameworks/helm.md)
15 - [NVIDIA Dynamo](integrations/dynamo.md)
16 - [InftyAI/llmaz](integrations/llmaz.md)
17 - [llm-d](integrations/llm-d.md)
18 - [KAITO](integrations/kaito.md)
19 - [KServe](integrations/kserve.md)
20 - [Kthena](integrations/kthena.md)
21 - [KubeRay](integrations/kuberay.md)
22 - [kubernetes-sigs/lws](frameworks/lws.md)
23 - [meta-llama/llama-stack](integrations/llamastack.md)
24 - [substratusai/kubeai](integrations/kubeai.md)
25 - [vllm-project/AIBrix](integrations/aibrix.md)
26 - [vllm-project/production-stack](integrations/production-stack.md)
27
28 ## Deployment with CPUs
29
30 !!! note
31 The use of CPUs here is for demonstration and testing purposes only and its performance will not be on par with GPUs.
32
33 First, create a Kubernetes PVC and Secret for downloading and storing Hugging Face model:
34
35 ??? console "Config"
36
37 ```bash
38 cat <<EOF |kubectl apply -f -
39 apiVersion: v1
40 kind: PersistentVolumeClaim
41 metadata:
42 name: vllm-models
43 spec:
44 accessModes:
45 - ReadWriteOnce
46 volumeMode: Filesystem
47 resources:
48 requests:
49 storage: 50Gi
50 ---
51 apiVersion: v1
52 kind: Secret
53 metadata:
54 name: hf-token-secret
55 type: Opaque
56 stringData:
57 token: "REPLACE_WITH_TOKEN"
58 EOF
59 ```
60
61 Here, the `token` field stores your **Hugging Face access token**. For details on how to generate a token,
62 see the [Hugging Face documentation](https://huggingface.co/docs/hub/en/security-tokens).
63
64 Next, start the vLLM server as a Kubernetes Deployment and Service.
65
66 Note that you will want to configure your vLLM image based on your processor arch:
67
68 ??? console "Config"
69
70 ```bash
71 VLLM_IMAGE=public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest # use this for x86_64
72 VLLM_IMAGE=public.ecr.aws/q9t5s3a7/vllm-arm64-cpu-release-repo:latest # use this for arm64
73 cat <<EOF |kubectl apply -f -
74 apiVersion: apps/v1
75 kind: Deployment
76 metadata:
77 name: vllm-server
78 spec:
79 replicas: 1
80 selector:
81 matchLabels:
82 app.kubernetes.io/name: vllm
83 template:
84 metadata:
85 labels:
86 app.kubernetes.io/name: vllm
87 spec:
88 containers:
89 - name: vllm
90 image: $VLLM_IMAGE
91 command: ["/bin/sh", "-c"]
92 args: [
93 "vllm serve meta-llama/Llama-3.2-1B-Instruct"
94 ]
95 env:
96 - name: HF_TOKEN
97 valueFrom:
98 secretKeyRef:
99 name: hf-token-secret
100 key: token
101 ports:
102 - containerPort: 8000
103 volumeMounts:
104 - name: llama-storage
105 mountPath: /root/.cache/huggingface
106 volumes:
107 - name: llama-storage
108 persistentVolumeClaim:
109 claimName: vllm-models
110 ---
111 apiVersion: v1
112 kind: Service
113 metadata:
114 name: vllm-server
115 spec:
116 selector:
117 app.kubernetes.io/name: vllm
118 ports:
119 - protocol: TCP
120 port: 8000
121 targetPort: 8000
122 type: ClusterIP
123 EOF
124 ```
125
126 We can verify that the vLLM server has started successfully via the logs (this might take a couple of minutes to download the model):
127
128 ```bash
129 kubectl logs -l app.kubernetes.io/name=vllm
130 ...
131 INFO: Started server process [1]
132 INFO: Waiting for application startup.
133 INFO: Application startup complete.
134 INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
135 ```
136
137 ## Deployment with GPUs
138
139 **Pre-requisite**: Ensure that you have a running [Kubernetes cluster with GPUs](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/).
140
141 1. Create a PVC, Secret and Deployment for vLLM
142
143 PVC is used to store the model cache and it is optional, you can use hostPath or other storage options
144
145 <details>
146 <summary>Yaml</summary>
147
148 ```yaml
149 apiVersion: v1
150 kind: PersistentVolumeClaim
151 metadata:
152 name: mistral-7b
153 namespace: default
154 spec:
155 accessModes:
156 - ReadWriteOnce
157 resources:
158 requests:
159 storage: 50Gi
160 storageClassName: default
161 volumeMode: Filesystem
162 ```
163
164 </details>
165
166 Secret is optional and only required for accessing gated models, you can skip this step if you are not using gated models
167
168 ```yaml
169 apiVersion: v1
170 kind: Secret
171 metadata:
172 name: hf-token-secret
173 namespace: default
174 type: Opaque
175 stringData:
176 token: "REPLACE_WITH_TOKEN"
177 ```
178
179 Next to create the deployment file for vLLM to run the model server. The following example deploys the `Mistral-7B-Instruct-v0.3` model.
180
181 Here are two examples for using NVIDIA GPU and AMD GPU.
182
183 NVIDIA GPU:
184
185 <details>
186 <summary>Yaml</summary>
187
188 ```yaml
189 apiVersion: apps/v1
190 kind: Deployment
191 metadata:
192 name: mistral-7b
193 namespace: default
194 labels:
195 app: mistral-7b
196 spec:
197 replicas: 1
198 selector:
199 matchLabels:
200 app: mistral-7b
201 template:
202 metadata:
203 labels:
204 app: mistral-7b
205 spec:
206 volumes:
207 - name: cache-volume
208 persistentVolumeClaim:
209 claimName: mistral-7b
210 # vLLM needs to access the host's shared memory for tensor parallel inference.
211 - name: shm
212 emptyDir:
213 medium: Memory
214 sizeLimit: "2Gi"
215 containers:
216 - name: mistral-7b
217 image: vllm/vllm-openai:latest
218 command: ["/bin/sh", "-c"]
219 args: [
220 "vllm serve mistralai/Mistral-7B-Instruct-v0.3 --trust-remote-code --enable-chunked-prefill --max-num-batched-tokens 1024"
221 ]
222 env:
223 - name: HF_TOKEN
224 valueFrom:
225 secretKeyRef:
226 name: hf-token-secret
227 key: token
228 ports:
229 - containerPort: 8000
230 resources:
231 limits:
232 cpu: "10"
233 memory: 20G
234 nvidia.com/gpu: "1"
235 requests:
236 cpu: "2"
237 memory: 6G
238 nvidia.com/gpu: "1"
239 volumeMounts:
240 - mountPath: /root/.cache/huggingface
241 name: cache-volume
242 - name: shm
243 mountPath: /dev/shm
244 livenessProbe:
245 httpGet:
246 path: /health
247 port: 8000
248 initialDelaySeconds: 60
249 periodSeconds: 10
250 readinessProbe:
251 httpGet:
252 path: /health
253 port: 8000
254 initialDelaySeconds: 60
255 periodSeconds: 5
256 ```
257
258 </details>
259
260 AMD GPU:
261
262 You can refer to the `deployment.yaml` below if using AMD ROCm GPU like MI300X.
263
264 <details>
265 <summary>Yaml</summary>
266
267 ```yaml
268 apiVersion: apps/v1
269 kind: Deployment
270 metadata:
271 name: mistral-7b
272 namespace: default
273 labels:
274 app: mistral-7b
275 spec:
276 replicas: 1
277 selector:
278 matchLabels:
279 app: mistral-7b
280 template:
281 metadata:
282 labels:
283 app: mistral-7b
284 spec:
285 volumes:
286 # PVC
287 - name: cache-volume
288 persistentVolumeClaim:
289 claimName: mistral-7b
290 # vLLM needs to access the host's shared memory for tensor parallel inference.
291 - name: shm
292 emptyDir:
293 medium: Memory
294 sizeLimit: "8Gi"
295 hostNetwork: true
296 hostIPC: true
297 containers:
298 - name: mistral-7b
299 image: rocm/vllm:rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4
300 securityContext:
301 seccompProfile:
302 type: Unconfined
303 runAsGroup: 44
304 capabilities:
305 add:
306 - SYS_PTRACE
307 command: ["/bin/sh", "-c"]
308 args: [
309 "vllm serve mistralai/Mistral-7B-v0.3 --port 8000 --trust-remote-code --enable-chunked-prefill --max-num-batched-tokens 1024"
310 ]
311 env:
312 - name: HF_TOKEN
313 valueFrom:
314 secretKeyRef:
315 name: hf-token-secret
316 key: token
317 ports:
318 - containerPort: 8000
319 resources:
320 limits:
321 cpu: "10"
322 memory: 20G
323 amd.com/gpu: "1"
324 requests:
325 cpu: "6"
326 memory: 6G
327 amd.com/gpu: "1"
328 volumeMounts:
329 - name: cache-volume
330 mountPath: /root/.cache/huggingface
331 - name: shm
332 mountPath: /dev/shm
333 ```
334
335 </details>
336
337 You can get the full example with steps and sample yaml files from <https://github.com/ROCm/k8s-device-plugin/tree/master/example/vllm-serve>.
338
339 2. Create a Kubernetes Service for vLLM
340
341 Next, create a Kubernetes Service file to expose the `mistral-7b` deployment:
342
343 <details>
344 <summary>Yaml</summary>
345
346 ```yaml
347 apiVersion: v1
348 kind: Service
349 metadata:
350 name: mistral-7b
351 namespace: default
352 spec:
353 ports:
354 - name: http-mistral-7b
355 port: 80
356 protocol: TCP
357 targetPort: 8000
358 # The label selector should match the deployment labels & it is useful for prefix caching feature
359 selector:
360 app: mistral-7b
361 sessionAffinity: None
362 type: ClusterIP
363 ```
364
365 </details>
366
367 3. Deploy and Test
368
369 Apply the deployment and service configurations using `kubectl apply -f <filename>`:
370
371 ```bash
372 kubectl apply -f deployment.yaml
373 kubectl apply -f service.yaml
374 ```
375
376 To test the deployment, run the following `curl` command:
377
378 ```bash
379 curl http://mistral-7b.default.svc.cluster.local/v1/completions \
380 -H "Content-Type: application/json" \
381 -d '{
382 "model": "mistralai/Mistral-7B-Instruct-v0.3",
383 "prompt": "San Francisco is a",
384 "max_tokens": 7,
385 "temperature": 0
386 }'
387 ```
388
389 If the service is correctly deployed, you should receive a response from the vLLM model.
390
391 ## Serving with gRPC
392
393 vLLM can serve models over gRPC instead of HTTP by passing the `--grpc` flag. This requires the optional gRPC dependencies:
394
395 ```bash
396 pip install vllm[grpc]
397 ```
398
399 When using `--grpc`, the server exposes the standard [gRPC Health Checking Protocol](https://github.com/grpc/grpc/blob/master/doc/health-checking.md) (`grpc.health.v1.Health`), which integrates with Kubernetes [native gRPC probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) (available since Kubernetes 1.24).
400
401 To deploy with gRPC, change the `vllm serve` command to include `--grpc` and replace `httpGet` probes with `grpc` probes:
402
403 ```yaml
404 containers:
405 - name: mistral-7b
406 image: vllm/vllm-openai:latest
407 command: ["/bin/sh", "-c"]
408 args: [
409 "pip install vllm[grpc] && vllm serve mistralai/Mistral-7B-Instruct-v0.3 --grpc --port 50051 --trust-remote-code"
410 ]
411 ports:
412 - containerPort: 50051
413 livenessProbe:
414 grpc:
415 port: 50051
416 initialDelaySeconds: 120
417 periodSeconds: 10
418 readinessProbe:
419 grpc:
420 port: 50051
421 initialDelaySeconds: 120
422 periodSeconds: 5
423 ```
424
425 !!! note
426 The gRPC health service checks the engine status on every probe. If the engine is unhealthy or the server is shutting down, the probe returns `NOT_SERVING`.
427
428 You can also verify the health service manually with `grpcurl`:
429
430 ```bash
431 grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check
432 ```
433
434 ## Troubleshooting
435
436 ### Startup Probe or Readiness Probe Failure, container log contains "KeyboardInterrupt: terminated"
437
438 If the startup or readiness probe failureThreshold is too low for the time needed to start up the server, Kubernetes scheduler will kill the container. A couple of indications that this has happened:
439
440 1. container log contains "KeyboardInterrupt: terminated"
441 2. `kubectl get events` shows message `Container $NAME failed startup probe, will be restarted`
442
443 To mitigate, increase the failureThreshold to allow more time for the model server to start serving. You can identify an ideal failureThreshold by removing the probes from the manifest and measuring how much time it takes for the model server to show it's ready to serve.
444
445 ## Conclusion
446
447 Deploying vLLM with Kubernetes allows for efficient scaling and management of ML models leveraging GPU resources. By following the steps outlined above, you should be able to set up and test a vLLM deployment within your Kubernetes cluster. If you encounter any issues or have suggestions, please feel free to contribute to the documentation.