@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 158 lines 5.8 KB
Rendered Raw
1 # Production stack
2
3 Deploying vLLM on Kubernetes is a scalable and efficient way to serve machine learning models. This guide walks you through deploying vLLM using the [vLLM production stack](https://github.com/vllm-project/production-stack). Born out of a Berkeley-UChicago collaboration, [vLLM production stack](https://github.com/vllm-project/production-stack) is an officially released, production-optimized codebase under the [vLLM project](https://github.com/vllm-project), designed for LLM deployment with:
4
5 * **Upstream vLLM compatibility** – It wraps around upstream vLLM without modifying its code.
6 * **Ease of use** – Simplified deployment via Helm charts and observability through Grafana dashboards.
7 * **High performance** – Optimized for LLM workloads with features like multimodel support, model-aware and prefix-aware routing, fast vLLM bootstrapping, and KV cache offloading with [LMCache](https://github.com/LMCache/LMCache) (wired up in vLLM via `--kv-offloading-backend lmcache`; see the [LMCache examples](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/lmcache) and [docs.lmcache.ai](https://docs.lmcache.ai)), among others.
8
9 If you are new to Kubernetes, don't worry: in the vLLM production stack [repo](https://github.com/vllm-project/production-stack), we provide a step-by-step [guide](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md) and a [short video](https://www.youtube.com/watch?v=EsTJbQtzj0g) to set up everything and get started in **4 minutes**!
10
11 ## Pre-requisite
12
13 Ensure that you have a running Kubernetes environment with GPU (you can follow [this tutorial](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md) to install a Kubernetes environment on a bare-metal GPU machine).
14
15 ## Deployment using vLLM production stack
16
17 The standard vLLM production stack is installed using a Helm chart. You can run this [bash script](https://github.com/vllm-project/production-stack/blob/main/utils/install-helm.sh) to install Helm on your GPU server.
18
19 To install the vLLM production stack, run the following commands on your desktop:
20
21 ```bash
22 sudo helm repo add vllm https://vllm-project.github.io/production-stack
23 sudo helm install vllm vllm/vllm-stack -f tutorials/assets/values-01-minimal-example.yaml
24 ```
25
26 This will instantiate a vLLM-production-stack-based deployment named `vllm` that runs a small LLM (Facebook opt-125M model).
27
28 ### Validate Installation
29
30 Monitor the deployment status using:
31
32 ```bash
33 sudo kubectl get pods
34 ```
35
36 And you will see that pods for the `vllm` deployment will transit to `Running` state.
37
38 ```text
39 NAME READY STATUS RESTARTS AGE
40 vllm-deployment-router-859d8fb668-2x2b7 1/1 Running 0 2m38s
41 vllm-opt125m-deployment-vllm-84dfc9bd7-vb9bs 1/1 Running 0 2m38s
42 ```
43
44 !!! note
45 It may take some time for the containers to download the Docker images and LLM weights.
46
47 ### Send a Query to the Stack
48
49 Forward the `vllm-router-service` port to the host machine:
50
51 ```bash
52 sudo kubectl port-forward svc/vllm-router-service 30080:80
53 ```
54
55 And then you can send out a query to the OpenAI-compatible API to check the available models:
56
57 ```bash
58 curl -o- http://localhost:30080/v1/models
59 ```
60
61 ??? console "Output"
62
63 ```json
64 {
65 "object": "list",
66 "data": [
67 {
68 "id": "facebook/opt-125m",
69 "object": "model",
70 "created": 1737428424,
71 "owned_by": "vllm",
72 "root": null
73 }
74 ]
75 }
76 ```
77
78 To send an actual chatting request, you can issue a curl request to the OpenAI `/completion` endpoint:
79
80 ```bash
81 curl -X POST http://localhost:30080/v1/completions \
82 -H "Content-Type: application/json" \
83 -d '{
84 "model": "facebook/opt-125m",
85 "prompt": "Once upon a time,",
86 "max_tokens": 10
87 }'
88 ```
89
90 ??? console "Output"
91
92 ```json
93 {
94 "id": "completion-id",
95 "object": "text_completion",
96 "created": 1737428424,
97 "model": "facebook/opt-125m",
98 "choices": [
99 {
100 "text": " there was a brave knight who...",
101 "index": 0,
102 "finish_reason": "length"
103 }
104 ]
105 }
106 ```
107
108 ### Uninstall
109
110 To remove the deployment, run:
111
112 ```bash
113 sudo helm uninstall vllm
114 ```
115
116 ---
117
118 ### (Advanced) Configuring vLLM production stack
119
120 The core vLLM production stack configuration is managed with YAML. Here is the example configuration used in the installation above:
121
122 ??? code "Yaml"
123
124 ```yaml
125 servingEngineSpec:
126 runtimeClassName: ""
127 modelSpec:
128 - name: "opt125m"
129 repository: "vllm/vllm-openai"
130 tag: "latest"
131 modelURL: "facebook/opt-125m"
132
133 replicaCount: 1
134
135 requestCPU: 6
136 requestMemory: "16Gi"
137 requestGPU: 1
138
139 pvcStorage: "10Gi"
140 ```
141
142 In this YAML configuration:
143
144 * **`modelSpec`** includes:
145 * `name`: A nickname that you prefer to call the model.
146 * `repository`: Docker repository of vLLM.
147 * `tag`: Docker image tag.
148 * `modelURL`: The LLM model that you want to use.
149 * **`replicaCount`**: Number of replicas.
150 * **`requestCPU` and `requestMemory`**: Specifies the CPU and memory resource requests for the pod.
151 * **`requestGPU`**: Specifies the number of GPUs required.
152 * **`pvcStorage`**: Allocates persistent storage for the model.
153
154 !!! note
155 If you intend to set up two pods, please refer to this [YAML file](https://github.com/vllm-project/production-stack/blob/main/tutorials/assets/values-01-2pods-minimal-example.yaml).
156
157 !!! tip
158 vLLM production stack offers many more features (*e.g.* CPU offloading and a wide range of routing algorithms). Please check out these [examples and tutorials](https://github.com/vllm-project/production-stack/tree/main/tutorials) and our [repo](https://github.com/vllm-project/production-stack) for more details!