@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 327 lines 9.35 KB
Rendered Raw
1 # SkyPilot
2
3 <p align="center">
4 <img src="https://imgur.com/yxtzPEu.png" alt="vLLM"/>
5 </p>
6
7 vLLM can be **run and scaled to multiple service replicas on clouds and Kubernetes** with [SkyPilot](https://github.com/skypilot-org/skypilot), an open-source framework for running LLMs on any cloud. More examples for various open models, such as Llama-3, Mixtral, etc., can be found in [SkyPilot AI gallery](https://skypilot.readthedocs.io/en/latest/gallery/index.html).
8
9 ## Prerequisites
10
11 - Go to the [HuggingFace model page](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and request access to the model `meta-llama/Meta-Llama-3-8B-Instruct`.
12 - Check that you have installed SkyPilot ([docs](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html)).
13 - Check that `sky check` shows clouds or Kubernetes are enabled.
14
15 ```bash
16 pip install skypilot-nightly
17 sky check
18 ```
19
20 ## Run on a single instance
21
22 See the vLLM SkyPilot YAML for serving, [serving.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/vllm/serve.yaml).
23
24 ??? code "Yaml"
25
26 ```yaml
27 resources:
28 accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
29 use_spot: True
30 disk_size: 512 # Ensure model checkpoints can fit.
31 disk_tier: best
32 ports: 8081 # Expose to internet traffic.
33
34 envs:
35 PYTHONUNBUFFERED: 1
36 MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
37 HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
38
39 setup: |
40 conda create -n vllm python=3.10 -y
41 conda activate vllm
42
43 pip install vllm==0.4.0.post1
44 # Install Gradio for web UI.
45 pip install gradio openai
46 pip install flash-attn==2.5.7
47
48 run: |
49 conda activate vllm
50 echo 'Starting vllm api server...'
51 vllm serve $MODEL_NAME \
52 --port 8081 \
53 --trust-remote-code \
54 --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
55 2>&1 | tee api_server.log &
56
57 echo 'Waiting for vllm api server to start...'
58 while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
59
60 echo 'Starting gradio server...'
61 git clone https://github.com/vllm-project/vllm.git || true
62 python vllm/examples/applications/chatbot/gradio_openai_chatbot_webserver.py \
63 -m $MODEL_NAME \
64 --port 8811 \
65 --model-url http://localhost:8081/v1 \
66 --stop-token-ids 128009,128001
67 ```
68
69 Start the serving the Llama-3 8B model on any of the candidate GPUs listed (L4, A10g, ...):
70
71 ```bash
72 HF_TOKEN="your-huggingface-token" sky launch serving.yaml --env HF_TOKEN
73 ```
74
75 Check the output of the command. There will be a shareable gradio link (like the last line of the following). Open it in your browser to use the LLaMA model to do the text completion.
76
77 ```console
78 (task, pid=7431) Running on public URL: https://<gradio-hash>.gradio.live
79 ```
80
81 **Optional**: Serve the 70B model instead of the default 8B and use more GPU:
82
83 ```bash
84 HF_TOKEN="your-huggingface-token" \
85 sky launch serving.yaml \
86 --gpus A100:8 \
87 --env HF_TOKEN \
88 --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct
89 ```
90
91 ## Scale up to multiple replicas
92
93 SkyPilot can scale up the service to multiple service replicas with built-in autoscaling, load-balancing and fault-tolerance. You can do it by adding a services section to the YAML file.
94
95 ??? code "Yaml"
96
97 ```yaml
98 service:
99 replicas: 2
100 # An actual request for readiness probe.
101 readiness_probe:
102 path: /v1/chat/completions
103 post_data:
104 model: $MODEL_NAME
105 messages:
106 - role: user
107 content: Hello! What is your name?
108 max_completion_tokens: 1
109 ```
110
111 ??? code "Yaml"
112
113 ```yaml
114 service:
115 replicas: 2
116 # An actual request for readiness probe.
117 readiness_probe:
118 path: /v1/chat/completions
119 post_data:
120 model: $MODEL_NAME
121 messages:
122 - role: user
123 content: Hello! What is your name?
124 max_completion_tokens: 1
125
126 resources:
127 accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
128 use_spot: True
129 disk_size: 512 # Ensure model checkpoints can fit.
130 disk_tier: best
131 ports: 8081 # Expose to internet traffic.
132
133 envs:
134 PYTHONUNBUFFERED: 1
135 MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
136 HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
137
138 setup: |
139 conda create -n vllm python=3.10 -y
140 conda activate vllm
141
142 pip install vllm==0.4.0.post1
143 # Install Gradio for web UI.
144 pip install gradio openai
145 pip install flash-attn==2.5.7
146
147 run: |
148 conda activate vllm
149 echo 'Starting vllm api server...'
150 vllm serve $MODEL_NAME \
151 --port 8081 \
152 --trust-remote-code \
153 --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
154 2>&1 | tee api_server.log
155 ```
156
157 Start the serving the Llama-3 8B model on multiple replicas:
158
159 ```bash
160 HF_TOKEN="your-huggingface-token" \
161 sky serve up -n vllm serving.yaml \
162 --env HF_TOKEN
163 ```
164
165 Wait until the service is ready:
166
167 ```bash
168 watch -n10 sky serve status vllm
169 ```
170
171 Example outputs:
172
173 ```console
174 Services
175 NAME VERSION UPTIME STATUS REPLICAS ENDPOINT
176 vllm 1 35s READY 2/2 xx.yy.zz.100:30001
177
178 Service Replicas
179 SERVICE_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION
180 vllm 1 1 xx.yy.zz.121 18 mins ago 1x GCP([Spot]{'L4': 1}) READY us-east4
181 vllm 2 1 xx.yy.zz.245 18 mins ago 1x GCP([Spot]{'L4': 1}) READY us-east4
182 ```
183
184 After the service is READY, you can find a single endpoint for the service and access the service with the endpoint:
185
186 ??? console "Commands"
187
188 ```bash
189 ENDPOINT=$(sky serve status --endpoint 8081 vllm)
190 curl -L http://$ENDPOINT/v1/chat/completions \
191 -H "Content-Type: application/json" \
192 -d '{
193 "model": "meta-llama/Meta-Llama-3-8B-Instruct",
194 "messages": [
195 {
196 "role": "system",
197 "content": "You are a helpful assistant."
198 },
199 {
200 "role": "user",
201 "content": "Who are you?"
202 }
203 ],
204 "stop_token_ids": [128009, 128001]
205 }'
206 ```
207
208 To enable autoscaling, you could replace the `replicas` with the following configs in `service`:
209
210 ```yaml
211 service:
212 replica_policy:
213 min_replicas: 2
214 max_replicas: 4
215 target_qps_per_replica: 2
216 ```
217
218 This will scale the service up to when the QPS exceeds 2 for each replica.
219
220 ??? code "Yaml"
221
222 ```yaml
223 service:
224 replica_policy:
225 min_replicas: 2
226 max_replicas: 4
227 target_qps_per_replica: 2
228 # An actual request for readiness probe.
229 readiness_probe:
230 path: /v1/chat/completions
231 post_data:
232 model: $MODEL_NAME
233 messages:
234 - role: user
235 content: Hello! What is your name?
236 max_completion_tokens: 1
237
238 resources:
239 accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
240 use_spot: True
241 disk_size: 512 # Ensure model checkpoints can fit.
242 disk_tier: best
243 ports: 8081 # Expose to internet traffic.
244
245 envs:
246 PYTHONUNBUFFERED: 1
247 MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
248 HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
249
250 setup: |
251 conda create -n vllm python=3.10 -y
252 conda activate vllm
253
254 pip install vllm==0.4.0.post1
255 # Install Gradio for web UI.
256 pip install gradio openai
257 pip install flash-attn==2.5.7
258
259 run: |
260 conda activate vllm
261 echo 'Starting vllm api server...'
262 vllm serve $MODEL_NAME \
263 --port 8081 \
264 --trust-remote-code \
265 --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
266 2>&1 | tee api_server.log
267 ```
268
269 To update the service with the new config:
270
271 ```bash
272 HF_TOKEN="your-huggingface-token" sky serve update vllm serving.yaml --env HF_TOKEN
273 ```
274
275 To stop the service:
276
277 ```bash
278 sky serve down vllm
279 ```
280
281 ### **Optional**: Connect a GUI to the endpoint
282
283 It is also possible to access the Llama-3 service with a separate GUI frontend, so the user requests send to the GUI will be load-balanced across replicas.
284
285 ??? code "Yaml"
286
287 ```yaml
288 envs:
289 MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
290 ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.
291
292 resources:
293 cpus: 2
294
295 setup: |
296 conda create -n vllm python=3.10 -y
297 conda activate vllm
298
299 # Install Gradio for web UI.
300 pip install gradio openai
301
302 run: |
303 conda activate vllm
304 export PATH=$PATH:/sbin
305
306 echo 'Starting gradio server...'
307 git clone https://github.com/vllm-project/vllm.git || true
308 python vllm/examples/applications/api_client/gradio_openai_chatbot_webserver.py \
309 -m $MODEL_NAME \
310 --port 8811 \
311 --model-url http://$ENDPOINT/v1 \
312 --stop-token-ids 128009,128001 | tee ~/gradio.log
313 ```
314
315 1. Start the chat web UI:
316
317 ```bash
318 sky launch \
319 -c gui ./gui.yaml \
320 --env ENDPOINT=$(sky serve status --endpoint vllm)
321 ```
322
323 2. Then, we can access the GUI at the returned gradio link:
324
325 ```console
326 | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live
327 ```