A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 20:45 UTC
[Doc] Update SkyPilot doc for wrong indents and instructions for update service (#4283)
Zhanghao Wu committed
Jul 26, 2024 at 17:39 UTC
150a1ffbfd3d0429d30fa5ab841f53903a0a8a62
1 file changed
+243
-187
docs/source/serving/run_on_sky.rst
+243
-187
@@ -5,9 +5,9 @@ Deploying and scaling up with SkyPilot
5
6
.. raw:: html
7
8
- <p align="center">
9
- <img src="https://imgur.com/yxtzPEu.png" alt="vLLM"/>
10
- </p>
8
+ <p align="center">
9
+ <img src="https://imgur.com/yxtzPEu.png" alt="vLLM"/>
10
+ </p>
11
12
vLLM can be **run and scaled to multiple service replicas on clouds and Kubernetes** with `SkyPilot <https://github.com/skypilot-org/skypilot>`__, an open-source framework for running LLMs on any cloud. More examples for various open models, such as Llama-3, Mixtral, etc, can be found in `SkyPilot AI gallery <https://skypilot.readthedocs.io/en/latest/gallery/index.html>`__.
13
@@ -21,8 +21,8 @@ Prerequisites
21
22
.. code-block:: console
23
24
- pip install skypilot-nightly
25
- sky check
24
+ pip install skypilot-nightly
25
+ sky check
26
27
28
Run on a single instance
@@ -32,64 +32,64 @@ See the vLLM SkyPilot YAML for serving, `serving.yaml <https://github.com/skypil
32
33
.. code-block:: yaml
34
35
- resources:
36
- accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
37
- use_spot: True
38
- disk_size: 512 # Ensure model checkpoints can fit.
39
- disk_tier: best
40
- ports: 8081 # Expose to internet traffic.
41
-
42
- envs:
43
- MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
44
- HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
45
-
46
- setup: |
47
- conda create -n vllm python=3.10 -y
48
- conda activate vllm
49
-
50
- pip install vllm==0.4.0.post1
51
- # Install Gradio for web UI.
52
- pip install gradio openai
53
- pip install flash-attn==2.5.7
54
-
55
- run: |
56
- conda activate vllm
57
- echo 'Starting vllm api server...'
58
- python -u -m vllm.entrypoints.openai.api_server \
59
- --port 8081 \
60
- --model $MODEL_NAME \
61
- --trust-remote-code \
62
- --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
63
- 2>&1 | tee api_server.log &
64
-
65
- echo 'Waiting for vllm api server to start...'
66
- while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
67
-
68
- echo 'Starting gradio server...'
69
- git clone https://github.com/vllm-project/vllm.git || true
70
- python vllm/examples/gradio_openai_chatbot_webserver.py \
71
- -m $MODEL_NAME \
72
- --port 8811 \
73
- --model-url http://localhost:8081/v1 \
74
- --stop-token-ids 128009,128001
35
+ resources:
36
+ accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
37
+ use_spot: True
38
+ disk_size: 512 # Ensure model checkpoints can fit.
39
+ disk_tier: best
40
+ ports: 8081 # Expose to internet traffic.
41
+
42
+ envs:
43
+ MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
44
+ HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
45
+
46
+ setup: |
47
+ conda create -n vllm python=3.10 -y
48
+ conda activate vllm
49
+
50
+ pip install vllm==0.4.0.post1
51
+ # Install Gradio for web UI.
52
+ pip install gradio openai
53
+ pip install flash-attn==2.5.7
54
+
55
+ run: |
56
+ conda activate vllm
57
+ echo 'Starting vllm api server...'
58
+ python -u -m vllm.entrypoints.openai.api_server \
59
+ --port 8081 \
60
+ --model $MODEL_NAME \
61
+ --trust-remote-code \
62
+ --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
63
+ 2>&1 | tee api_server.log &
64
+
65
+ echo 'Waiting for vllm api server to start...'
66
+ while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
67
+
68
+ echo 'Starting gradio server...'
69
+ git clone https://github.com/vllm-project/vllm.git || true
70
+ python vllm/examples/gradio_openai_chatbot_webserver.py \
71
+ -m $MODEL_NAME \
72
+ --port 8811 \
73
+ --model-url http://localhost:8081/v1 \
74
+ --stop-token-ids 128009,128001
75
76
Start the serving the Llama-3 8B model on any of the candidate GPUs listed (L4, A10g, ...):
77
78
.. code-block:: console
79
80
- HF_TOKEN="your-huggingface-token" sky launch serving.yaml --env HF_TOKEN
80
+ HF_TOKEN="your-huggingface-token" sky launch serving.yaml --env HF_TOKEN
81
82
Check the output of the command. There will be a shareable gradio link (like the last line of the following). Open it in your browser to use the LLaMA model to do the text completion.
83
84
.. code-block:: console
85
86
- (task, pid=7431) Running on public URL: https://<gradio-hash>.gradio.live
86
+ (task, pid=7431) Running on public URL: https://<gradio-hash>.gradio.live
87
88
**Optional**: Serve the 70B model instead of the default 8B and use more GPU:
89
90
.. code-block:: console
91
92
- HF_TOKEN="your-huggingface-token" sky launch serving.yaml --gpus A100:8 --env HF_TOKEN --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct
92
+ HF_TOKEN="your-huggingface-token" sky launch serving.yaml --gpus A100:8 --env HF_TOKEN --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct
93
94
95
Scale up to multiple replicas
@@ -99,151 +99,212 @@ SkyPilot can scale up the service to multiple service replicas with built-in aut
99
100
.. code-block:: yaml
101
102
- service:
103
- replicas: 2
104
- # An actual request for readiness probe.
105
- readiness_probe:
106
- path: /v1/chat/completions
107
- post_data:
108
- model: $MODEL_NAME
109
- messages:
110
- - role: user
111
- content: Hello! What is your name?
112
- max_tokens: 1
113
-
102
+ service:
103
+ replicas: 2
104
+ # An actual request for readiness probe.
105
+ readiness_probe:
106
+ path: /v1/chat/completions
107
+ post_data:
108
+ model: $MODEL_NAME
109
+ messages:
110
+ - role: user
111
+ content: Hello! What is your name?
112
+ max_tokens: 1
113
+
114
.. raw:: html
115
116
- <details>
117
- <summary>Click to see the full recipe YAML</summary>
116
+ <details>
117
+ <summary>Click to see the full recipe YAML</summary>
118
119
120
.. code-block:: yaml
121
122
- service:
123
- replicas: 2
124
- # An actual request for readiness probe.
125
- readiness_probe:
126
- path: /v1/chat/completions
127
- post_data:
128
- model: $MODEL_NAME
129
- messages:
130
- - role: user
131
- content: Hello! What is your name?
122
+ service:
123
+ replicas: 2
124
+ # An actual request for readiness probe.
125
+ readiness_probe:
126
+ path: /v1/chat/completions
127
+ post_data:
128
+ model: $MODEL_NAME
129
+ messages:
130
+ - role: user
131
+ content: Hello! What is your name?
132
max_tokens: 1
133
134
- resources:
135
- accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
136
- use_spot: True
137
- disk_size: 512 # Ensure model checkpoints can fit.
138
- disk_tier: best
139
- ports: 8081 # Expose to internet traffic.
140
-
141
- envs:
142
- MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
143
- HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
144
-
145
- setup: |
146
- conda create -n vllm python=3.10 -y
147
- conda activate vllm
148
-
149
- pip install vllm==0.4.0.post1
150
- # Install Gradio for web UI.
151
- pip install gradio openai
152
- pip install flash-attn==2.5.7
153
-
154
- run: |
155
- conda activate vllm
156
- echo 'Starting vllm api server...'
157
- python -u -m vllm.entrypoints.openai.api_server \
158
- --port 8081 \
159
- --model $MODEL_NAME \
160
- --trust-remote-code \
161
- --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
162
- 2>&1 | tee api_server.log &
163
-
164
- echo 'Waiting for vllm api server to start...'
165
- while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
166
-
167
- echo 'Starting gradio server...'
168
- git clone https://github.com/vllm-project/vllm.git || true
169
- python vllm/examples/gradio_openai_chatbot_webserver.py \
170
- -m $MODEL_NAME \
171
- --port 8811 \
172
- --model-url http://localhost:8081/v1 \
173
- --stop-token-ids 128009,128001
134
+ resources:
135
+ accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
136
+ use_spot: True
137
+ disk_size: 512 # Ensure model checkpoints can fit.
138
+ disk_tier: best
139
+ ports: 8081 # Expose to internet traffic.
140
+
141
+ envs:
142
+ MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
143
+ HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
144
+
145
+ setup: |
146
+ conda create -n vllm python=3.10 -y
147
+ conda activate vllm
148
+
149
+ pip install vllm==0.4.0.post1
150
+ # Install Gradio for web UI.
151
+ pip install gradio openai
152
+ pip install flash-attn==2.5.7
153
+
154
+ run: |
155
+ conda activate vllm
156
+ echo 'Starting vllm api server...'
157
+ python -u -m vllm.entrypoints.openai.api_server \
158
+ --port 8081 \
159
+ --model $MODEL_NAME \
160
+ --trust-remote-code \
161
+ --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
162
+ 2>&1 | tee api_server.log
163
164
.. raw:: html
165
177
- </details>
166
+ </details>
167
168
Start the serving the Llama-3 8B model on multiple replicas:
169
170
.. code-block:: console
171
183
- HF_TOKEN="your-huggingface-token" sky serve up -n vllm serving.yaml --env HF_TOKEN
172
+ HF_TOKEN="your-huggingface-token" sky serve up -n vllm serving.yaml --env HF_TOKEN
173
174
175
Wait until the service is ready:
176
177
.. code-block:: console
178
190
- watch -n10 sky serve status vllm
179
+ watch -n10 sky serve status vllm
180
181
182
.. raw:: html
183
195
- <details>
196
- <summary>Example outputs:</summary>
184
+ <details>
185
+ <summary>Example outputs:</summary>
186
187
.. code-block:: console
188
200
- Services
201
- NAME VERSION UPTIME STATUS REPLICAS ENDPOINT
202
- vllm 1 35s READY 2/2 xx.yy.zz.100:30001
189
+ Services
190
+ NAME VERSION UPTIME STATUS REPLICAS ENDPOINT
191
+ vllm 1 35s READY 2/2 xx.yy.zz.100:30001
192
204
- Service Replicas
205
- SERVICE_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION
206
- vllm 1 1 xx.yy.zz.121 18 mins ago 1x GCP({'L4': 1}) READY us-east4
207
- vllm 2 1 xx.yy.zz.245 18 mins ago 1x GCP({'L4': 1}) READY us-east4
193
+ Service Replicas
194
+ SERVICE_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION
195
+ vllm 1 1 xx.yy.zz.121 18 mins ago 1x GCP([Spot]{'L4': 1}) READY us-east4
196
+ vllm 2 1 xx.yy.zz.245 18 mins ago 1x GCP([Spot]{'L4': 1}) READY us-east4
197
198
.. raw:: html
210
-
211
- </details>
199
+
200
+ </details>
201
202
After the service is READY, you can find a single endpoint for the service and access the service with the endpoint:
203
204
.. code-block:: console
205
217
- ENDPOINT=$(sky serve status --endpoint 8081 vllm)
218
- curl -L http://$ENDPOINT/v1/chat/completions \
219
- -H "Content-Type: application/json" \
220
- -d '{
221
- "model": "meta-llama/Meta-Llama-3-8B-Instruct",
222
- "messages": [
223
- {
224
- "role": "system",
225
- "content": "You are a helpful assistant."
226
- },
227
- {
228
- "role": "user",
229
- "content": "Who are you?"
230
- }
231
- ],
232
- "stop_token_ids": [128009, 128001]
233
- }'
234
-
235
-To enable autoscaling, you could specify additional configs in `services`:
206
+ ENDPOINT=$(sky serve status --endpoint 8081 vllm)
207
+ curl -L http://$ENDPOINT/v1/chat/completions \
208
+ -H "Content-Type: application/json" \
209
+ -d '{
210
+ "model": "meta-llama/Meta-Llama-3-8B-Instruct",
211
+ "messages": [
212
+ {
213
+ "role": "system",
214
+ "content": "You are a helpful assistant."
215
+ },
216
+ {
217
+ "role": "user",
218
+ "content": "Who are you?"
219
+ }
220
+ ],
221
+ "stop_token_ids": [128009, 128001]
222
+ }'
223
+
224
+To enable autoscaling, you could replace the `replicas` with the following configs in `service`:
225
226
.. code-block:: yaml
227
239
- services:
240
- replica_policy:
241
- min_replicas: 0
242
- max_replicas: 3
243
- target_qps_per_replica: 2
228
+ service:
229
+ replica_policy:
230
+ min_replicas: 2
231
+ max_replicas: 4
232
+ target_qps_per_replica: 2
233
234
This will scale the service up to when the QPS exceeds 2 for each replica.
235
236
+
237
+.. raw:: html
238
+
239
+ <details>
240
+ <summary>Click to see the full recipe YAML</summary>
241
+
242
+
243
+.. code-block:: yaml
244
+
245
+ service:
246
+ replica_policy:
247
+ min_replicas: 2
248
+ max_replicas: 4
249
+ target_qps_per_replica: 2
250
+ # An actual request for readiness probe.
251
+ readiness_probe:
252
+ path: /v1/chat/completions
253
+ post_data:
254
+ model: $MODEL_NAME
255
+ messages:
256
+ - role: user
257
+ content: Hello! What is your name?
258
+ max_tokens: 1
259
+
260
+ resources:
261
+ accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
262
+ use_spot: True
263
+ disk_size: 512 # Ensure model checkpoints can fit.
264
+ disk_tier: best
265
+ ports: 8081 # Expose to internet traffic.
266
+
267
+ envs:
268
+ MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
269
+ HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
270
+
271
+ setup: |
272
+ conda create -n vllm python=3.10 -y
273
+ conda activate vllm
274
+
275
+ pip install vllm==0.4.0.post1
276
+ # Install Gradio for web UI.
277
+ pip install gradio openai
278
+ pip install flash-attn==2.5.7
279
+
280
+ run: |
281
+ conda activate vllm
282
+ echo 'Starting vllm api server...'
283
+ python -u -m vllm.entrypoints.openai.api_server \
284
+ --port 8081 \
285
+ --model $MODEL_NAME \
286
+ --trust-remote-code \
287
+ --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
288
+ 2>&1 | tee api_server.log
289
+
290
+
291
+.. raw:: html
292
+
293
+ </details>
294
+
295
+To update the service with the new config:
296
+
297
+.. code-block:: console
298
+
299
+ HF_TOKEN="your-huggingface-token" sky serve update vllm serving.yaml --env HF_TOKEN
300
+
301
+
302
+To stop the service:
303
+
304
+.. code-block:: console
305
+
306
+ sky serve down vllm
307
+
308
309
**Optional**: Connect a GUI to the endpoint
310
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -253,58 +314,53 @@ It is also possible to access the Llama-3 service with a separate GUI frontend,
314
315
.. raw:: html
316
256
- <details>
257
- <summary>Click to see the full GUI YAML</summary>
317
+ <details>
318
+ <summary>Click to see the full GUI YAML</summary>
319
320
.. code-block:: yaml
321
261
- envs:
262
- MODEL_NAME: meta-llama/Meta-Llama-3-70B-Instruct
263
- ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.
264
-
265
- resources:
266
- cpus: 2
267
-
268
- setup: |
269
- conda activate vllm
270
- if [ $? -ne 0 ]; then
271
- conda create -n vllm python=3.10 -y
272
- conda activate vllm
273
- fi
274
-
275
- # Install Gradio for web UI.
276
- pip install gradio openai
277
-
278
- run: |
279
- conda activate vllm
280
- export PATH=$PATH:/sbin
281
- WORKER_IP=$(hostname -I | cut -d' ' -f1)
282
- CONTROLLER_PORT=21001
283
- WORKER_PORT=21002
284
-
285
- echo 'Starting gradio server...'
286
- git clone https://github.com/vllm-project/vllm.git || true
287
- python vllm/examples/gradio_openai_chatbot_webserver.py \
288
- -m $MODEL_NAME \
289
- --port 8811 \
290
- --model-url http://$ENDPOINT/v1 \
291
- --stop-token-ids 128009,128001 | tee ~/gradio.log
322
+ envs:
323
+ MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
324
+ ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.
325
+
326
+ resources:
327
+ cpus: 2
328
+
329
+ setup: |
330
+ conda create -n vllm python=3.10 -y
331
+ conda activate vllm
332
+
333
+ # Install Gradio for web UI.
334
+ pip install gradio openai
335
+
336
+ run: |
337
+ conda activate vllm
338
+ export PATH=$PATH:/sbin
339
+
340
+ echo 'Starting gradio server...'
341
+ git clone https://github.com/vllm-project/vllm.git || true
342
+ python vllm/examples/gradio_openai_chatbot_webserver.py \
343
+ -m $MODEL_NAME \
344
+ --port 8811 \
345
+ --model-url http://$ENDPOINT/v1 \
346
+ --stop-token-ids 128009,128001 | tee ~/gradio.log
347
+
348
349
.. raw:: html
294
-
295
- </details>
350
+
351
+ </details>
352
353
1. Start the chat web UI:
354
355
.. code-block:: console
356
301
- sky launch -c gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint vllm)
357
+ sky launch -c gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint vllm)
358
359
360
2. Then, we can access the GUI at the returned gradio link:
361
362
.. code-block:: console
363
308
- | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live
364
+ | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live
365
366