@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 312 lines 9.67 KB
Rendered Raw
1 # LWS
2
3 LeaderWorkerSet (LWS) is a Kubernetes API that aims to address common deployment patterns of AI/ML inference workloads.
4 A major use case is for multi-host/multi-node distributed inference.
5
6 vLLM can be deployed with [LWS](https://github.com/kubernetes-sigs/lws) on Kubernetes for distributed model serving.
7
8 ## Prerequisites
9
10 - At least two Kubernetes nodes, each with 8 GPUs, are required.
11 - Install LWS by following the instructions found [here](https://lws.sigs.k8s.io/docs/installation/).
12
13 ## Deploy and Serve
14
15 Deploy the following yaml file `lws.yaml` (we have examples that use multiprocessing or Ray):
16
17 ??? code "lws.yaml"
18 === "Multiprocessing (default)"
19 ```yaml
20 apiVersion: leaderworkerset.x-k8s.io/v1
21 kind: LeaderWorkerSet
22 metadata:
23 name: vllm
24 spec:
25 replicas: 1
26 leaderWorkerTemplate:
27 size: 2
28 restartPolicy: RecreateGroupOnPodRestart
29 leaderTemplate:
30 metadata:
31 labels:
32 role: leader
33 spec:
34 containers:
35 - name: vllm-leader
36 image: docker.io/vllm/vllm-openai:latest
37 env:
38 - name: HF_TOKEN
39 value: <your-hf-token>
40 command:
41 - sh
42 - -c
43 - "vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline-parallel-size $(LWS_GROUP_SIZE) --nnodes $(LWS_GROUP_SIZE) --node-rank $(LWS_WORKER_INDEX) --master-addr $(LWS_LEADER_ADDRESS) --port 8080"
44 resources:
45 limits:
46 nvidia.com/gpu: "8"
47 memory: 1124Gi
48 ephemeral-storage: 800Gi
49 requests:
50 ephemeral-storage: 800Gi
51 cpu: 125
52 ports:
53 - containerPort: 8080
54 readinessProbe:
55 tcpSocket:
56 port: 8080
57 initialDelaySeconds: 15
58 periodSeconds: 10
59 volumeMounts:
60 - mountPath: /dev/shm
61 name: dshm
62 volumes:
63 - name: dshm
64 emptyDir:
65 medium: Memory
66 sizeLimit: 15Gi
67 workerTemplate:
68 spec:
69 containers:
70 - name: vllm-worker
71 image: docker.io/vllm/vllm-openai:latest
72 command:
73 - sh
74 - -c
75 - "vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline-parallel-size $(LWS_GROUP_SIZE) --nnodes $(LWS_GROUP_SIZE) --node-rank $(LWS_WORKER_INDEX) --master-addr $(LWS_LEADER_ADDRESS) --headless"
76 resources:
77 limits:
78 nvidia.com/gpu: "8"
79 memory: 1124Gi
80 ephemeral-storage: 800Gi
81 requests:
82 ephemeral-storage: 800Gi
83 cpu: 125
84 env:
85 - name: HF_TOKEN
86 value: <your-hf-token>
87 volumeMounts:
88 - mountPath: /dev/shm
89 name: dshm
90 volumes:
91 - name: dshm
92 emptyDir:
93 medium: Memory
94 sizeLimit: 15Gi
95 ---
96 apiVersion: v1
97 kind: Service
98 metadata:
99 name: vllm-leader
100 spec:
101 ports:
102 - name: http
103 port: 8080
104 protocol: TCP
105 targetPort: 8080
106 selector:
107 leaderworkerset.sigs.k8s.io/name: vllm
108 role: leader
109 type: ClusterIP
110 ```
111
112 === "Ray"
113 ```yaml
114 apiVersion: leaderworkerset.x-k8s.io/v1
115 kind: LeaderWorkerSet
116 metadata:
117 name: vllm
118 spec:
119 replicas: 1
120 leaderWorkerTemplate:
121 size: 2
122 restartPolicy: RecreateGroupOnPodRestart
123 leaderTemplate:
124 metadata:
125 labels:
126 role: leader
127 spec:
128 containers:
129 - name: vllm-leader
130 image: docker.io/vllm/vllm-openai:latest
131 env:
132 - name: HF_TOKEN
133 value: <your-hf-token>
134 command:
135 - sh
136 - -c
137 - "bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);
138 vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct --port 8080 --tensor-parallel-size 8 --pipeline-parallel-size 2 --distributed-executor-backend ray"
139 resources:
140 limits:
141 nvidia.com/gpu: "8"
142 memory: 1124Gi
143 ephemeral-storage: 800Gi
144 requests:
145 ephemeral-storage: 800Gi
146 cpu: 125
147 ports:
148 - containerPort: 8080
149 readinessProbe:
150 tcpSocket:
151 port: 8080
152 initialDelaySeconds: 15
153 periodSeconds: 10
154 volumeMounts:
155 - mountPath: /dev/shm
156 name: dshm
157 volumes:
158 - name: dshm
159 emptyDir:
160 medium: Memory
161 sizeLimit: 15Gi
162 workerTemplate:
163 spec:
164 containers:
165 - name: vllm-worker
166 image: docker.io/vllm/vllm-openai:latest
167 command:
168 - sh
169 - -c
170 - "bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"
171 resources:
172 limits:
173 nvidia.com/gpu: "8"
174 memory: 1124Gi
175 ephemeral-storage: 800Gi
176 requests:
177 ephemeral-storage: 800Gi
178 cpu: 125
179 env:
180 - name: HF_TOKEN
181 value: <your-hf-token>
182 volumeMounts:
183 - mountPath: /dev/shm
184 name: dshm
185 volumes:
186 - name: dshm
187 emptyDir:
188 medium: Memory
189 sizeLimit: 15Gi
190 ---
191 apiVersion: v1
192 kind: Service
193 metadata:
194 name: vllm-leader
195 spec:
196 ports:
197 - name: http
198 port: 8080
199 protocol: TCP
200 targetPort: 8080
201 selector:
202 leaderworkerset.sigs.k8s.io/name: vllm
203 role: leader
204 type: ClusterIP
205 ```
206
207 ```bash
208 kubectl apply -f lws.yaml
209 ```
210
211 Verify the status of the pods:
212
213 ```bash
214 kubectl get pods
215 ```
216
217 Should get an output similar to this:
218
219 ```bash
220 NAME READY STATUS RESTARTS AGE
221 vllm-0 1/1 Running 0 2s
222 vllm-0-1 1/1 Running 0 2s
223 ```
224
225 Verify that the distributed tensor-parallel inference works:
226
227 === "Multiprocessing (default)"
228 ```bash
229 kubectl logs vllm-0 | grep -i "Model loading"
230 kubectl logs vllm-0-1 | grep -i "Model loading"
231 ```
232
233 Should get something similar to this:
234
235 POD 0 (PP Rank 0)
236
237 ```text
238 (Worker_PP0_TP0 pid=601) INFO 04-28 08:16:58 [gpu_model_runner.py:4820] Model loading took 3.82 GiB memory and 157.996399 seconds
239 ```
240
241 POD 1 (PP Rank 1)
242
243 ```text
244 (Worker_PP1_TP0 pid=396) INFO 04-28 08:17:09 [gpu_model_runner.py:4820] Model loading took 3.82 GiB memory and 168.878781 seconds
245 ```
246
247 === "Ray"
248 ```bash
249 kubectl logs vllm-0 | grep -i "Loading model weights took"
250 ```
251
252 Should get something similar to this:
253
254 ```text
255 INFO 05-08 03:20:24 model_runner.py:173] Loading model weights took 0.1189 GB
256 (RayWorkerWrapper pid=169, ip=10.20.0.197) INFO 05-08 03:20:28 model_runner.py:173] Loading model weights took 0.1189 GB
257 ```
258
259 ## Access ClusterIP service
260
261 ```bash
262 # Listen on port 8080 locally, forwarding to the targetPort of the service's port 8080 in a pod selected by the service
263 kubectl port-forward svc/vllm-leader 8080:8080
264 ```
265
266 The output should be similar to the following:
267
268 ```text
269 Forwarding from 127.0.0.1:8080 -> 8080
270 Forwarding from [::1]:8080 -> 8080
271 ```
272
273 ## Serve the model
274
275 Open another terminal and send a request
276
277 ```text
278 curl http://localhost:8080/v1/completions \
279 -H "Content-Type: application/json" \
280 -d '{
281 "model": "meta-llama/Meta-Llama-3.1-405B-Instruct",
282 "prompt": "San Francisco is a",
283 "max_tokens": 7,
284 "temperature": 0
285 }'
286 ```
287
288 The output should be similar to the following
289
290 ??? console "Output"
291 ```text
292 {
293 "id": "cmpl-1bb34faba88b43f9862cfbfb2200949d",
294 "object": "text_completion",
295 "created": 1715138766,
296 "model": "meta-llama/Meta-Llama-3.1-405B-Instruct",
297 "choices": [
298 {
299 "index": 0,
300 "text": " top destination for foodies, with",
301 "logprobs": null,
302 "finish_reason": "length",
303 "stop_reason": null
304 }
305 ],
306 "usage": {
307 "prompt_tokens": 5,
308 "total_tokens": 12,
309 "completion_tokens": 7
310 }
311 }
312 ```