@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 471 lines 14.8 KB
Rendered Raw
1 # Kthena
2
3 [**Kthena**](https://github.com/volcano-sh/kthena) is a Kubernetes-native LLM inference platform that transforms how organizations deploy and manage Large Language Models in production. Built with declarative model lifecycle management and intelligent request routing, it provides high performance and enterprise-grade scalability for LLM inference workloads.
4
5 Kthena manages inference workloads through `ModelServing` and provides model-aware request routing through `ModelServer` and `ModelRoute`.
6
7 This guide shows how to deploy a **multi-node vLLM** service on Kubernetes using `ModelServing` and expose it through a Kubernetes Service.
8
9 We’ll:
10
11 - Install the required components (Kthena + Volcano).
12 - Deploy a multi-node vLLM model via Kthena’s `ModelServing` CR.
13 - Validate the deployment.
14
15 ---
16
17 ## 1. Prerequisites
18
19 You need:
20
21 - A Kubernetes cluster with **GPU nodes**.
22 - `kubectl` access with cluster-admin or equivalent permissions.
23 - **Volcano 1.14 or later** for the role-level gang scheduling used in this example.
24 - **Kthena** installed with the `ModelServing` CRD available.
25 - A valid **Hugging Face token** if loading models from Hugging Face Hub.
26
27 ### 1.1 Install Volcano
28
29 ```bash
30 helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
31 helm repo update
32 helm install volcano volcano-sh/volcano -n volcano-system --create-namespace
33 ```
34
35 Volcano provides the gang scheduling used in this example. It is optional for Kthena deployments that do not use Volcano scheduling.
36
37 ### 1.2 Install Kthena
38
39 ```bash
40 helm install kthena oci://ghcr.io/volcano-sh/charts/kthena --version v1.0.0 --namespace kthena-system --create-namespace
41 ```
42
43 - The `kthena-system` namespace is created.
44 - Kthena controllers and CRDs, including `ModelServing`, are installed.
45
46 Validate:
47
48 ```bash
49 kubectl get pods -n kthena-system
50 kubectl get crd modelservings.workload.serving.volcano.sh
51 ```
52
53 You should see:
54
55 ```text
56 modelservings.workload.serving.volcano.sh ...
57 ```
58
59 ---
60
61 ## 2. The Multi-Node vLLM `ModelServing` Example
62
63 Kthena provides an example manifest to deploy a **multi-node vLLM cluster running Llama**. Each role replica contains an entry pod and its worker pods.
64
65 A simplified version of the example (`llama-multinode`) looks like:
66
67 - `spec.replicas: 1` – one `ServingGroup` (one logical model deployment).
68 - `roles`:
69 - `entryTemplate` – defines **leader** pods that run:
70 - vLLM’s **multi-node cluster bootstrap script**.
71 - vLLM **OpenAI-compatible API server**.
72 - `workerTemplate` – defines **worker** pods to join the leader’s Ray cluster (Ray backend) or to join same distributed process group (multiprocessing backend).
73
74 Key points from the example YAML:
75
76 Image: `vllm/vllm-openai:latest` (matches upstream vLLM images).
77 Commands:
78
79 ??? code "Yaml"
80 === "Multiprocessing (default)"
81 Leader:
82
83 ```yaml
84 command:
85 - sh
86 - -c
87 - >
88 vllm serve meta-llama/Llama-3.1-405B-Instruct
89 --tensor-parallel-size 8
90 --pipeline-parallel-size 2
91 --nnodes=2
92 --node-rank=0
93 --master-addr=$(ENTRY_ADDRESS)
94 --port 8080
95 ```
96
97 Worker:
98
99 ```yaml
100 command:
101 - sh
102 - -c
103 - >
104 vllm serve meta-llama/Llama-3.1-405B-Instruct
105 --tensor-parallel-size 8
106 --pipeline-parallel-size 2
107 --nnodes=2
108 --node-rank=1
109 --master-addr=$(ENTRY_ADDRESS)
110 --headless
111 ```
112
113 === "Ray"
114 Leader:
115
116 ```yaml
117 command:
118 - sh
119 - -c
120 - >
121 bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh
122 leader --ray_cluster_size=2;
123 vllm serve meta-llama/Llama-3.1-405B-Instruct --port 8080 --tensor-parallel-size 8
124 --pipeline-parallel-size 2 --distributed-executor-backend ray
125 ```
126
127 Worker:
128
129 ```yaml
130 command:
131 - sh
132 - -c
133 - >
134 bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh
135 worker --ray_address=$(ENTRY_ADDRESS)
136 ```
137
138 ---
139
140 ## 3. Deploying Multi-Node llama vLLM via Kthena
141
142 ### 3.1 Prepare the Manifest
143
144 **Recommended**: use a Secret instead of a raw env var:
145
146 ```bash
147 kubectl create secret generic hf-token \
148 -n default \
149 --from-literal=HUGGING_FACE_HUB_TOKEN='<your-token>'
150 ```
151
152 ### 3.2 Apply the `ModelServing`
153
154 Save one of the following manifests to `modelserving.yaml`:
155
156 ??? code "modelserving.yaml"
157 === "Multiprocessing (default)"
158 ```yaml
159 apiVersion: workload.serving.volcano.sh/v1alpha1
160 kind: ModelServing
161 metadata:
162 name: llama-multinode
163 namespace: default
164 spec:
165 schedulerName: volcano
166 replicas: 1 # group replicas
167 template:
168 restartGracePeriodSeconds: 60
169 gangPolicy:
170 minRoleReplicas:
171 llama-405b: 1
172 roles:
173 - name: llama-405b
174 replicas: 2
175 entryTemplate:
176 spec:
177 containers:
178 - name: leader
179 image: vllm/vllm-openai:latest
180 env:
181 - name: HUGGING_FACE_HUB_TOKEN
182 valueFrom:
183 secretKeyRef:
184 name: hf-token
185 key: HUGGING_FACE_HUB_TOKEN
186 command:
187 - sh
188 - -c
189 - "vllm serve meta-llama/Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline-parallel-size 2 --nnodes 2 --node-rank 0 --master-addr $(ENTRY_ADDRESS) --distributed-executor-backend mp --port 8080"
190 resources:
191 limits:
192 nvidia.com/gpu: "8"
193 memory: 1124Gi
194 ephemeral-storage: 800Gi
195 requests:
196 ephemeral-storage: 800Gi
197 cpu: 125
198 ports:
199 - containerPort: 8080
200 readinessProbe:
201 tcpSocket:
202 port: 8080
203 initialDelaySeconds: 15
204 periodSeconds: 10
205 volumeMounts:
206 - mountPath: /dev/shm
207 name: dshm
208 volumes:
209 - name: dshm
210 emptyDir:
211 medium: Memory
212 sizeLimit: 15Gi
213 workerReplicas: 1
214 workerTemplate:
215 spec:
216 containers:
217 - name: worker
218 image: vllm/vllm-openai:latest
219 command:
220 - sh
221 - -c
222 - "vllm serve meta-llama/Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline-parallel-size 2 --nnodes 2 --node-rank 1 --master-addr $(ENTRY_ADDRESS) --distributed-executor-backend mp --headless"
223 resources:
224 limits:
225 nvidia.com/gpu: "8"
226 memory: 1124Gi
227 ephemeral-storage: 800Gi
228 requests:
229 ephemeral-storage: 800Gi
230 cpu: 125
231 env:
232 - name: HUGGING_FACE_HUB_TOKEN
233 valueFrom:
234 secretKeyRef:
235 name: hf-token
236 key: HUGGING_FACE_HUB_TOKEN
237 volumeMounts:
238 - mountPath: /dev/shm
239 name: dshm
240 volumes:
241 - name: dshm
242 emptyDir:
243 medium: Memory
244 sizeLimit: 15Gi
245 ```
246
247 === "Ray"
248 ```yaml
249 apiVersion: workload.serving.volcano.sh/v1alpha1
250 kind: ModelServing
251 metadata:
252 name: llama-multinode
253 namespace: default
254 spec:
255 schedulerName: volcano
256 replicas: 1 # group replicas
257 template:
258 restartGracePeriodSeconds: 60
259 gangPolicy:
260 minRoleReplicas:
261 llama-405b: 1
262 roles:
263 - name: llama-405b
264 replicas: 2
265 entryTemplate:
266 spec:
267 containers:
268 - name: leader
269 image: vllm/vllm-openai:latest
270 env:
271 - name: HUGGING_FACE_HUB_TOKEN
272 valueFrom:
273 secretKeyRef:
274 name: hf-token
275 key: HUGGING_FACE_HUB_TOKEN
276 command:
277 - sh
278 - -c
279 - "bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh leader --ray_cluster_size=2;
280 vllm serve meta-llama/Llama-3.1-405B-Instruct --port 8080 --tensor-parallel-size 8 --pipeline-parallel-size 2 --distributed-executor-backend ray"
281 resources:
282 limits:
283 nvidia.com/gpu: "8"
284 memory: 1124Gi
285 ephemeral-storage: 800Gi
286 requests:
287 ephemeral-storage: 800Gi
288 cpu: 125
289 ports:
290 - containerPort: 8080
291 readinessProbe:
292 tcpSocket:
293 port: 8080
294 initialDelaySeconds: 15
295 periodSeconds: 10
296 volumeMounts:
297 - mountPath: /dev/shm
298 name: dshm
299 volumes:
300 - name: dshm
301 emptyDir:
302 medium: Memory
303 sizeLimit: 15Gi
304 workerReplicas: 1
305 workerTemplate:
306 spec:
307 containers:
308 - name: worker
309 image: vllm/vllm-openai:latest
310 command:
311 - sh
312 - -c
313 - "bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh worker --ray_address=$(ENTRY_ADDRESS)"
314 resources:
315 limits:
316 nvidia.com/gpu: "8"
317 memory: 1124Gi
318 ephemeral-storage: 800Gi
319 requests:
320 ephemeral-storage: 800Gi
321 cpu: 125
322 env:
323 - name: HUGGING_FACE_HUB_TOKEN
324 valueFrom:
325 secretKeyRef:
326 name: hf-token
327 key: HUGGING_FACE_HUB_TOKEN
328 volumeMounts:
329 - mountPath: /dev/shm
330 name: dshm
331 volumes:
332 - name: dshm
333 emptyDir:
334 medium: Memory
335 sizeLimit: 15Gi
336 ```
337
338 ```bash
339 kubectl apply -f modelserving.yaml
340 ```
341
342 Kthena will:
343
344 - Create a `ModelServing` object.
345 - Derive a `PodGroup` for Volcano gang scheduling.
346 - Create the leader and worker pods for each `ServingGroup` and `Role`.
347
348 ---
349
350 ## 4. Verifying the Deployment
351
352 ### 4.1 Check ModelServing Status
353
354 Use the snippet from the Kthena docs:
355
356 ```bash
357 kubectl get modelserving -oyaml | grep status -A 10
358 ```
359
360 You should see something like:
361
362 ```yaml
363 status:
364 availableReplicas: 1
365 conditions:
366 - type: Available
367 status: "True"
368 reason: AllGroupsReady
369 message: All Serving groups are ready
370 - type: Progressing
371 status: "False"
372 ...
373 replicas: 1
374 updatedReplicas: 1
375 ```
376
377 ### 4.2 Check Pods
378
379 List pods for your deployment:
380
381 ```bash
382 kubectl get pod -owide -l modelserving.volcano.sh/name=llama-multinode
383 ```
384
385 Example output (from docs):
386
387 ```text
388 NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE ...
389 default llama-multinode-0-llama-405b-0-0 1/1 Running 0 15m 10.244.0.56 192.168.5.12 ...
390 default llama-multinode-0-llama-405b-0-1 1/1 Running 0 15m 10.244.0.58 192.168.5.43 ...
391 default llama-multinode-0-llama-405b-1-0 1/1 Running 0 15m 10.244.0.57 192.168.5.58 ...
392 default llama-multinode-0-llama-405b-1-1 1/1 Running 0 15m 10.244.0.53 192.168.5.36 ...
393 ```
394
395 Pod name pattern:
396
397 - `llama-multinode-<group-idx>-<role-name>-<replica-idx>-<ordinal>`.
398
399 The first index identifies the `ServingGroup`, followed by the role name (`llama-405b`), role replica index, and pod index.
400
401 ---
402
403 ## 5. Accessing the vLLM OpenAI-Compatible API
404
405 Save the following Service as `service.yaml` to expose the entry pods:
406
407 ```yaml
408 apiVersion: v1
409 kind: Service
410 metadata:
411 name: llama-multinode-openai
412 namespace: default
413 spec:
414 selector:
415 modelserving.volcano.sh/name: llama-multinode
416 modelserving.volcano.sh/entry: "true"
417 ports:
418 - name: http
419 port: 80
420 targetPort: 8080
421 type: ClusterIP
422 ```
423
424 Apply the Service, then port-forward from your local machine:
425
426 ```bash
427 kubectl apply -f service.yaml
428 kubectl port-forward svc/llama-multinode-openai 30080:80 -n default
429 ```
430
431 Then:
432
433 - List models:
434
435 ```bash
436 curl -s http://localhost:30080/v1/models
437 ```
438
439 - Send a completion request (mirroring vLLM production stack docs):
440
441 ```bash
442 curl -X POST http://localhost:30080/v1/completions \
443 -H "Content-Type: application/json" \
444 -d '{
445 "model": "meta-llama/Llama-3.1-405B-Instruct",
446 "prompt": "Once upon a time,",
447 "max_tokens": 10
448 }'
449 ```
450
451 You should see an OpenAI-style response from vLLM.
452
453 ---
454
455 ## 6. Clean Up
456
457 To remove the deployment and its resources:
458
459 ```bash
460 kubectl delete -f service.yaml
461 kubectl delete modelserving llama-multinode -n default
462 ```
463
464 If you’re done with the entire stack:
465
466 ```bash
467 helm uninstall kthena -n kthena-system # or your Kthena release name
468 helm uninstall volcano -n volcano-system
469 ```
470
471 For model-aware routing and prefill-decode disaggregation, see the [Kthena documentation](https://kthena.volcano.sh/docs/user-guide/router-routing).