A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
Update deploying_with_k8s.md with AMD ROCm GPU example (#11465)
Signed-off-by: Alex He <alehe@amd.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
AlexHe99 committed
Dec 27, 2024 at 18:00 UTC
d003f3ea391b4c879f6f848dd485dd3c04fa6ca9
1 file changed
+78
-1
docs/source/serving/deploying_with_k8s.md
+78
-1
@@ -47,7 +47,11 @@ data:
47
token: "REPLACE_WITH_TOKEN"
48
```
49
50
-Create a deployment file for vLLM to run the model server. The following example deploys the `Mistral-7B-Instruct-v0.3` model:
50
+Next to create the deployment file for vLLM to run the model server. The following example deploys the `Mistral-7B-Instruct-v0.3` model.
51
+
52
+Here are two examples for using NVIDIA GPU and AMD GPU.
53
+
54
+- NVIDIA GPU
55
56
```yaml
57
apiVersion: apps/v1
@@ -119,6 +123,79 @@ spec:
123
periodSeconds: 5
124
```
125
126
+- AMD GPU
127
+
128
+You can refer to the `deployment.yaml` below if using AMD ROCm GPU like MI300X.
129
+
130
+```yaml
131
+apiVersion: apps/v1
132
+kind: Deployment
133
+metadata:
134
+ name: mistral-7b
135
+ namespace: default
136
+ labels:
137
+ app: mistral-7b
138
+spec:
139
+ replicas: 1
140
+ selector:
141
+ matchLabels:
142
+ app: mistral-7b
143
+ template:
144
+ metadata:
145
+ labels:
146
+ app: mistral-7b
147
+ spec:
148
+ volumes:
149
+ # PVC
150
+ - name: cache-volume
151
+ persistentVolumeClaim:
152
+ claimName: mistral-7b
153
+ # vLLM needs to access the host's shared memory for tensor parallel inference.
154
+ - name: shm
155
+ emptyDir:
156
+ medium: Memory
157
+ sizeLimit: "8Gi"
158
+ hostNetwork: true
159
+ hostIPC: true
160
+ containers:
161
+ - name: mistral-7b
162
+ image: rocm/vllm:rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4
163
+ securityContext:
164
+ seccompProfile:
165
+ type: Unconfined
166
+ runAsGroup: 44
167
+ capabilities:
168
+ add:
169
+ - SYS_PTRACE
170
+ command: ["/bin/sh", "-c"]
171
+ args: [
172
+ "vllm serve mistralai/Mistral-7B-v0.3 --port 8000 --trust-remote-code --enable-chunked-prefill --max_num_batched_tokens 1024"
173
+ ]
174
+ env:
175
+ - name: HUGGING_FACE_HUB_TOKEN
176
+ valueFrom:
177
+ secretKeyRef:
178
+ name: hf-token-secret
179
+ key: token
180
+ ports:
181
+ - containerPort: 8000
182
+ resources:
183
+ limits:
184
+ cpu: "10"
185
+ memory: 20G
186
+ amd.com/gpu: "1"
187
+ requests:
188
+ cpu: "6"
189
+ memory: 6G
190
+ amd.com/gpu: "1"
191
+ volumeMounts:
192
+ - name: cache-volume
193
+ mountPath: /root/.cache/huggingface
194
+ - name: shm
195
+ mountPath: /dev/shm
196
+```
197
+You can get the full example with steps and sample yaml files from <https://github.com/ROCm/k8s-device-plugin/tree/master/example/vllm-serve>.
198
+
199
2. **Create a Kubernetes Service for vLLM**
200
201
Next, create a Kubernetes Service file to expose the `mistral-7b` deployment: