3e67044972 · tduncan/Nvidia-Optimization-Stack-VLLM

@@ -2,7 +2,9 @@

2

3 [**Kthena**](https://github.com/volcano-sh/kthena) is a Kubernetes-native LLM inference platform that transforms how organizations deploy and manage Large Language Models in production. Built with declarative model lifecycle management and intelligent request routing, it provides high performance and enterprise-grade scalability for LLM inference workloads.

4

5 -This guide shows how to deploy a production-grade, **multi-node vLLM** service on Kubernetes.

5 +Kthena manages inference workloads through `ModelServing` and provides model-aware request routing through `ModelServer` and `ModelRoute`.

6 +

7 +This guide shows how to deploy a **multi-node vLLM** service on Kubernetes using `ModelServing` and expose it through a Kubernetes Service.

8

9 We’ll:

10

@@ -18,7 +20,7 @@ You need:

20

21 - A Kubernetes cluster with **GPU nodes**.

22 - `kubectl` access with cluster-admin or equivalent permissions.

21 -- **Volcano** installed for gang scheduling.

23 +- **Volcano 1.14 or later** for the role-level gang scheduling used in this example.

24 - **Kthena** installed with the `ModelServing` CRD available.

25 - A valid **Hugging Face token** if loading models from Hugging Face Hub.

26

@@ -30,21 +32,22 @@ helm repo update

32 helm install volcano volcano-sh/volcano -n volcano-system --create-namespace

33 ```

34

33 -This provides the gang-scheduling and network topology features used by Kthena.

35 +Volcano provides the gang scheduling used in this example. It is optional for Kthena deployments that do not use Volcano scheduling.

36

37 ### 1.2 Install Kthena

38

39 ```bash

38 -helm install kthena oci://ghcr.io/volcano-sh/charts/kthena --version v0.1.0 --namespace kthena-system --create-namespace

40 +helm install kthena oci://ghcr.io/volcano-sh/charts/kthena --version v1.0.0 --namespace kthena-system --create-namespace

41 ```

42

43 - The `kthena-system` namespace is created.

42 -- Kthena controllers and CRDs, including `ModelServing`, are installed and healthy.

44 +- Kthena controllers and CRDs, including `ModelServing`, are installed.

45

46 Validate:

47

48 ```bash

47 -kubectl get crd | grep modelserving

49 +kubectl get pods -n kthena-system

50 +kubectl get crd modelservings.workload.serving.volcano.sh

51 ```

52

53 You should see:

@@ -57,7 +60,7 @@ modelservings.workload.serving.volcano.sh ...

60

61 ## 2. The Multi-Node vLLM `ModelServing` Example

62

60 -Kthena provides an example manifest to deploy a **multi-node vLLM cluster running Llama**. Conceptually this is equivalent to the vLLM production stack Helm deployment, but expressed with `ModelServing`.

63 +Kthena provides an example manifest to deploy a **multi-node vLLM cluster running Llama**. Each role replica contains an entry pod and its worker pods.

64

65 A simplified version of the example (`llama-multinode`) looks like:

66

@@ -116,10 +119,9 @@ Commands:

119 - -c

120 - >

121 bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh

119 - leader --ray_cluster_size=2; python3 -m

120 - vllm.entrypoints.launchers.api_server.entry --port 8080 --model

121 - meta-llama/Llama-3.1-405B-Instruct --tensor-parallel-size 8

122 - --pipeline-parallel-size 2

122 + leader --ray_cluster_size=2;

123 + vllm serve meta-llama/Llama-3.1-405B-Instruct --port 8080 --tensor-parallel-size 8

124 + --pipeline-parallel-size 2 --distributed-executor-backend ray

125 ```

126

127 Worker:

@@ -166,9 +168,9 @@ Save one of the following manifests to `modelserving.yaml`:

168 restartGracePeriodSeconds: 60

169 gangPolicy:

170 minRoleReplicas:

169 - 405b: 1

171 + llama-405b: 1

172 roles:

171 - - name: 405b

173 + - name: llama-405b

174 replicas: 2

175 entryTemplate:

176 spec:

@@ -256,9 +258,9 @@ Save one of the following manifests to `modelserving.yaml`:

258 restartGracePeriodSeconds: 60

259 gangPolicy:

260 minRoleReplicas:

259 - 405b: 1

261 + llama-405b: 1

262 roles:

261 - - name: 405b

263 + - name: llama-405b

264 replicas: 2

265 entryTemplate:

266 spec:

@@ -275,7 +277,7 @@ Save one of the following manifests to `modelserving.yaml`:

277 - sh

278 - -c

279 - "bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh leader --ray_cluster_size=2;

278 - vllm serve meta-llama/Llama-3.1-405B-Instruct --port 8080 --tensor-parallel-size 8 --pipeline-parallel-size 2"

280 + vllm serve meta-llama/Llama-3.1-405B-Instruct --port 8080 --tensor-parallel-size 8 --pipeline-parallel-size 2 --distributed-executor-backend ray"

281 resources:

282 limits:

283 nvidia.com/gpu: "8"

@@ -383,24 +385,24 @@ kubectl get pod -owide -l modelserving.volcano.sh/name=llama-multinode

385 Example output (from docs):

386

387 ```text

386 -NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE ...

387 -default llama-multinode-0-405b-0-0 1/1 Running 0 15m 10.244.0.56 192.168.5.12 ...

388 -default llama-multinode-0-405b-0-1 1/1 Running 0 15m 10.244.0.58 192.168.5.43 ...

389 -default llama-multinode-0-405b-1-0 1/1 Running 0 15m 10.244.0.57 192.168.5.58 ...

390 -default llama-multinode-0-405b-1-1 1/1 Running 0 15m 10.244.0.53 192.168.5.36 ...

388 +NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE ...

389 +default llama-multinode-0-llama-405b-0-0 1/1 Running 0 15m 10.244.0.56 192.168.5.12 ...

390 +default llama-multinode-0-llama-405b-0-1 1/1 Running 0 15m 10.244.0.58 192.168.5.43 ...

391 +default llama-multinode-0-llama-405b-1-0 1/1 Running 0 15m 10.244.0.57 192.168.5.58 ...

392 +default llama-multinode-0-llama-405b-1-1 1/1 Running 0 15m 10.244.0.53 192.168.5.36 ...

393 ```

394

395 Pod name pattern:

396

397 - `llama-multinode-<group-idx>-<role-name>-<replica-idx>-<ordinal>`.

398

397 -The first number indicates `ServingGroup`. The second (`405b`) is the `Role`. The remaining indices identify the pod within the role.

399 +The first index identifies the `ServingGroup`, followed by the role name (`llama-405b`), role replica index, and pod index.

400

401 ---

402

401 -## 6. Accessing the vLLM OpenAI-Compatible API

403 +## 5. Accessing the vLLM OpenAI-Compatible API

404

403 -Expose the entry via a Service:

405 +Save the following Service as `service.yaml` to expose the entry pods:

406

407 ```yaml

408 apiVersion: v1

@@ -412,7 +414,6 @@ spec:

414 selector:

415 modelserving.volcano.sh/name: llama-multinode

416 modelserving.volcano.sh/entry: "true"

415 - # optionally further narrow to leader role if you label it

417 ports:

418 - name: http

419 port: 80

@@ -420,9 +421,10 @@ spec:

421 type: ClusterIP

422 ```

423

423 -Port-forward from your local machine:

424 +Apply the Service, then port-forward from your local machine:

425

426 ```bash

427 +kubectl apply -f service.yaml

428 kubectl port-forward svc/llama-multinode-openai 30080:80 -n default

429 ```

430

@@ -450,11 +452,12 @@ You should see an OpenAI-style response from vLLM.

452

453 ---

454

453 -## 7. Clean Up

455 +## 6. Clean Up

456

457 To remove the deployment and its resources:

458

459 ```bash

460 +kubectl delete -f service.yaml

461 kubectl delete modelserving llama-multinode -n default

462 ```

463

@@ -464,3 +467,5 @@ If you’re done with the entire stack:

467 helm uninstall kthena -n kthena-system # or your Kthena release name

468 helm uninstall volcano -n volcano-system

469 ```

470 +

471 +For model-aware routing and prefill-decode disaggregation, see the [Kthena documentation](https://kthena.volcano.sh/docs/user-guide/router-routing).

[Doc] Refresh Kthena integration guide (#57314)