@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 60a66ea2dc
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 11, 23:45 UTC

[DOC]: Add kthena to integrations (#29931)

Signed-off-by: Zhonghu Xu <xuzhonghu@huawei.com>

Tiger Xu / Zhonghu Xu committed Dec 5, 2025 at 16:11 UTC 60a66ea2dc887d50a88f20a1ecffaee2ada8f9dc
2 files changed +334
docs/deployment/integrations/kthena.md new
+333
@@ -0,0 +1,333 @@
1 +# Kthena
2 +
3 +[**Kthena**](https://github.com/volcano-sh/kthena) is a Kubernetes-native LLM inference platform that transforms how organizations deploy and manage Large Language Models in production. Built with declarative model lifecycle management and intelligent request routing, it provides high performance and enterprise-grade scalability for LLM inference workloads.
4 +
5 +This guide shows how to deploy a production-grade, **multi-node vLLM** service on Kubernetes.
6 +
7 +We’ll:
8 +
9 +- Install the required components (Kthena + Volcano).
10 +- Deploy a multi-node vLLM model via Kthena’s `ModelServing` CR.
11 +- Validate the deployment.
12 +
13 +---
14 +
15 +## 1. Prerequisites
16 +
17 +You need:
18 +
19 +- A Kubernetes cluster with **GPU nodes**.
20 +- `kubectl` access with cluster-admin or equivalent permissions.
21 +- **Volcano** installed for gang scheduling.
22 +- **Kthena** installed with the `ModelServing` CRD available.
23 +- A valid **Hugging Face token** if loading models from Hugging Face Hub.
24 +
25 +### 1.1 Install Volcano
26 +
27 +```bash
28 +helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
29 +helm repo update
30 +helm install volcano volcano-sh/volcano -n volcano-system --create-namespace
31 +```
32 +
33 +This provides the gang-scheduling and network topology features used by Kthena.
34 +
35 +### 1.2 Install Kthena
36 +
37 +```bash
38 +helm install kthena oci://ghcr.io/volcano-sh/charts/kthena --version v0.1.0 --namespace kthena-system --create-namespace
39 +```
40 +
41 +- The `kthena-system` namespace is created.
42 +- Kthena controllers and CRDs, including `ModelServing`, are installed and healthy.
43 +
44 +Validate:
45 +
46 +```bash
47 +kubectl get crd | grep modelserving
48 +```
49 +
50 +You should see:
51 +
52 +```text
53 +modelservings.workload.serving.volcano.sh ...
54 +```
55 +
56 +---
57 +
58 +## 2. The Multi-Node vLLM `ModelServing` Example
59 +
60 +Kthena provides an example manifest to deploy a **multi-node vLLM cluster running Llama**. Conceptually this is equivalent to the vLLM production stack Helm deployment, but expressed with `ModelServing`.
61 +
62 +A simplified version of the example (`llama-multinode`) looks like:
63 +
64 +- `spec.replicas: 1` – one `ServingGroup` (one logical model deployment).
65 +- `roles`:
66 + - `entryTemplate` – defines **leader** pods that run:
67 + - vLLM’s **multi-node cluster bootstrap script** (Ray cluster).
68 + - vLLM **OpenAI-compatible API server**.
69 + - `workerTemplate` – defines **worker** pods that join the leader’s Ray cluster.
70 +
71 +Key points from the example YAML:
72 +
73 +- **Image**: `vllm/vllm-openai:latest` (matches upstream vLLM images).
74 +- **Command** (leader):
75 +
76 + ```yaml
77 + command:
78 + - sh
79 + - -c
80 + - >
81 + bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=2;
82 + python3 -m vllm.entrypoints.openai.api_server
83 + --port 8080
84 + --model meta-llama/Llama-3.1-405B-Instruct
85 + --tensor-parallel-size 8
86 + --pipeline-parallel-size 2
87 + ```
88 +
89 +- **Command** (worker):
90 +
91 + ```yaml
92 + command:
93 + - sh
94 + - -c
95 + - >
96 + bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(ENTRY_ADDRESS)
97 + ```
98 +
99 +---
100 +
101 +## 3. Deploying Multi-Node llama vLLM via Kthena
102 +
103 +### 3.1 Prepare the Manifest
104 +
105 +**Recommended**: use a Secret instead of a raw env var:
106 +
107 +```bash
108 +kubectl create secret generic hf-token \
109 + -n default \
110 + --from-literal=HUGGING_FACE_HUB_TOKEN='<your-token>'
111 +```
112 +
113 +### 3.2 Apply the `ModelServing`
114 +
115 +```bash
116 +cat <<EOF | kubectl apply -f -
117 +apiVersion: workload.serving.volcano.sh/v1alpha1
118 +kind: ModelServing
119 +metadata:
120 + name: llama-multinode
121 + namespace: default
122 +spec:
123 + schedulerName: volcano
124 + replicas: 1 # group replicas
125 + template:
126 + restartGracePeriodSeconds: 60
127 + gangPolicy:
128 + minRoleReplicas:
129 + 405b: 1
130 + roles:
131 + - name: 405b
132 + replicas: 2
133 + entryTemplate:
134 + spec:
135 + containers:
136 + - name: leader
137 + image: vllm/vllm-openai:latest
138 + env:
139 + - name: HUGGING_FACE_HUB_TOKEN
140 + valueFrom:
141 + secretKeyRef:
142 + name: hf-token
143 + key: HUGGING_FACE_HUB_TOKEN
144 + command:
145 + - sh
146 + - -c
147 + - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=2;
148 + python3 -m vllm.entrypoints.openai.api_server --port 8080 --model meta-llama/Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline-parallel-size 2"
149 + resources:
150 + limits:
151 + nvidia.com/gpu: "8"
152 + memory: 1124Gi
153 + ephemeral-storage: 800Gi
154 + requests:
155 + ephemeral-storage: 800Gi
156 + cpu: 125
157 + ports:
158 + - containerPort: 8080
159 + readinessProbe:
160 + tcpSocket:
161 + port: 8080
162 + initialDelaySeconds: 15
163 + periodSeconds: 10
164 + volumeMounts:
165 + - mountPath: /dev/shm
166 + name: dshm
167 + volumes:
168 + - name: dshm
169 + emptyDir:
170 + medium: Memory
171 + sizeLimit: 15Gi
172 + workerReplicas: 1
173 + workerTemplate:
174 + spec:
175 + containers:
176 + - name: worker
177 + image: vllm/vllm-openai:latest
178 + command:
179 + - sh
180 + - -c
181 + - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(ENTRY_ADDRESS)"
182 + resources:
183 + limits:
184 + nvidia.com/gpu: "8"
185 + memory: 1124Gi
186 + ephemeral-storage: 800Gi
187 + requests:
188 + ephemeral-storage: 800Gi
189 + cpu: 125
190 + env:
191 + - name: HUGGING_FACE_HUB_TOKEN
192 + valueFrom:
193 + secretKeyRef:
194 + name: hf-token
195 + key: HUGGING_FACE_HUB_TOKEN
196 + volumeMounts:
197 + - mountPath: /dev/shm
198 + name: dshm
199 + volumes:
200 + - name: dshm
201 + emptyDir:
202 + medium: Memory
203 + sizeLimit: 15Gi
204 +EOF
205 +```
206 +
207 +Kthena will:
208 +
209 +- Create a `ModelServing` object.
210 +- Derive a `PodGroup` for Volcano gang scheduling.
211 +- Create the leader and worker pods for each `ServingGroup` and `Role`.
212 +
213 +---
214 +
215 +## 4. Verifying the Deployment
216 +
217 +### 4.1 Check ModelServing Status
218 +
219 +Use the snippet from the Kthena docs:
220 +
221 +```bash
222 +kubectl get modelserving -oyaml | grep status -A 10
223 +```
224 +
225 +You should see something like:
226 +
227 +```yaml
228 +status:
229 + availableReplicas: 1
230 + conditions:
231 + - type: Available
232 + status: "True"
233 + reason: AllGroupsReady
234 + message: All Serving groups are ready
235 + - type: Progressing
236 + status: "False"
237 + ...
238 + replicas: 1
239 + updatedReplicas: 1
240 +```
241 +
242 +### 4.2 Check Pods
243 +
244 +List pods for your deployment:
245 +
246 +```bash
247 +kubectl get pod -owide -l modelserving.volcano.sh/name=llama-multinode
248 +```
249 +
250 +Example output (from docs):
251 +
252 +```text
253 +NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE ...
254 +default llama-multinode-0-405b-0-0 1/1 Running 0 15m 10.244.0.56 192.168.5.12 ...
255 +default llama-multinode-0-405b-0-1 1/1 Running 0 15m 10.244.0.58 192.168.5.43 ...
256 +default llama-multinode-0-405b-1-0 1/1 Running 0 15m 10.244.0.57 192.168.5.58 ...
257 +default llama-multinode-0-405b-1-1 1/1 Running 0 15m 10.244.0.53 192.168.5.36 ...
258 +```
259 +
260 +Pod name pattern:
261 +
262 +- `llama-multinode-<group-idx>-<role-name>-<replica-idx>-<ordinal>`.
263 +
264 +The first number indicates `ServingGroup`. The second (`405b`) is the `Role`. The remaining indices identify the pod within the role.
265 +
266 +---
267 +
268 +## 6. Accessing the vLLM OpenAI-Compatible API
269 +
270 +Expose the entry via a Service:
271 +
272 +```yaml
273 +apiVersion: v1
274 +kind: Service
275 +metadata:
276 + name: llama-multinode-openai
277 + namespace: default
278 +spec:
279 + selector:
280 + modelserving.volcano.sh/name: llama-multinode
281 + modelserving.volcano.sh/entry: "true"
282 + # optionally further narrow to leader role if you label it
283 + ports:
284 + - name: http
285 + port: 80
286 + targetPort: 8080
287 + type: ClusterIP
288 +```
289 +
290 +Port-forward from your local machine:
291 +
292 +```bash
293 +kubectl port-forward svc/llama-multinode-openai 30080:80 -n default
294 +```
295 +
296 +Then:
297 +
298 +- List models:
299 +
300 + ```bash
301 + curl -s http://localhost:30080/v1/models
302 + ```
303 +
304 +- Send a completion request (mirroring vLLM production stack docs):
305 +
306 + ```bash
307 + curl -X POST http://localhost:30080/v1/completions \
308 + -H "Content-Type: application/json" \
309 + -d '{
310 + "model": "meta-llama/Llama-3.1-405B-Instruct",
311 + "prompt": "Once upon a time,",
312 + "max_tokens": 10
313 + }'
314 + ```
315 +
316 +You should see an OpenAI-style response from vLLM.
317 +
318 +---
319 +
320 +## 7. Clean Up
321 +
322 +To remove the deployment and its resources:
323 +
324 +```bash
325 +kubectl delete modelserving llama-multinode -n default
326 +```
327 +
328 +If you’re done with the entire stack:
329 +
330 +```bash
331 +helm uninstall kthena -n kthena-system # or your Kthena release name
332 +helm uninstall volcano -n volcano-system
333 +```
docs/deployment/k8s.md
+1
@@ -14,6 +14,7 @@ Alternatively, you can deploy vLLM to Kubernetes using any of the following:
14 - [InftyAI/llmaz](integrations/llmaz.md)
15 - [KAITO](integrations/kaito.md)
16 - [KServe](integrations/kserve.md)
17 +- [Kthena](integrations/kthena.md)
18 - [KubeRay](integrations/kuberay.md)
19 - [kubernetes-sigs/lws](frameworks/lws.md)
20 - [meta-llama/llama-stack](integrations/llamastack.md)