A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 03:00 UTC
[Doc] Update the SkyPilot doc with serving and Llama-3 (#4276)
Zhanghao Wu committed
Apr 22, 2024 at 15:34 UTC
ceaf4ed0030c7816537359b8efc750474149ce0f
1 file changed
+264
-23
docs/source/serving/run_on_sky.rst
+264
-23
@@ -1,7 +1,7 @@
1
.. _on_cloud:
2
3
-Running on clouds with SkyPilot
4
-===============================
3
+Deploying and scaling up with SkyPilot
4
+================================================
5
6
.. raw:: html
7
@@ -9,51 +9,75 @@ Running on clouds with SkyPilot
9
<img src="https://imgur.com/yxtzPEu.png" alt="vLLM"/>
10
</p>
11
12
-vLLM can be run on the cloud to scale to multiple GPUs with `SkyPilot <https://github.com/skypilot-org/skypilot>`__, an open-source framework for running LLMs on any cloud.
12
+vLLM can be **run and scaled to multiple service replicas on clouds and Kubernetes** with `SkyPilot <https://github.com/skypilot-org/skypilot>`__, an open-source framework for running LLMs on any cloud. More examples for various open models, such as Llama-3, Mixtral, etc, can be found in `SkyPilot AI gallery <https://skypilot.readthedocs.io/en/latest/gallery/index.html>`__.
13
14
-To install SkyPilot and setup your cloud credentials, run:
14
+
15
+Prerequisites
16
+-------------
17
+
18
+- Go to the `HuggingFace model page <https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct>`__ and request access to the model :code:`meta-llama/Meta-Llama-3-8B-Instruct`.
19
+- Check that you have installed SkyPilot (`docs <https://skypilot.readthedocs.io/en/latest/getting-started/installation.html>`__).
20
+- Check that :code:`sky check` shows clouds or Kubernetes are enabled.
21
22
.. code-block:: console
23
18
- $ pip install skypilot
19
- $ sky check
24
+ pip install skypilot-nightly
25
+ sky check
26
+
27
+
28
+Run on a single instance
29
+------------------------
30
31
See the vLLM SkyPilot YAML for serving, `serving.yaml <https://github.com/skypilot-org/skypilot/blob/master/llm/vllm/serve.yaml>`__.
32
33
.. code-block:: yaml
34
35
resources:
26
- accelerators: A100
36
+ accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
37
+ use_spot: True
38
+ disk_size: 512 # Ensure model checkpoints can fit.
39
+ disk_tier: best
40
+ ports: 8081 # Expose to internet traffic.
41
42
envs:
29
- MODEL_NAME: decapoda-research/llama-13b-hf
30
- TOKENIZER: hf-internal-testing/llama-tokenizer
43
+ MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
44
+ HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
45
46
setup: |
33
- conda create -n vllm python=3.9 -y
47
+ conda create -n vllm python=3.10 -y
48
conda activate vllm
35
- git clone https://github.com/vllm-project/vllm.git
36
- cd vllm
37
- pip install .
38
- pip install gradio
49
+
50
+ pip install vllm==0.4.0.post1
51
+ # Install Gradio for web UI.
52
+ pip install gradio openai
53
+ pip install flash-attn==2.5.7
54
55
run: |
56
conda activate vllm
57
echo 'Starting vllm api server...'
43
- python -u -m vllm.entrypoints.api_server \
44
- --model $MODEL_NAME \
45
- --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
46
- --tokenizer $TOKENIZER 2>&1 | tee api_server.log &
58
+ python -u -m vllm.entrypoints.openai.api_server \
59
+ --port 8081 \
60
+ --model $MODEL_NAME \
61
+ --trust-remote-code \
62
+ --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
63
+ 2>&1 | tee api_server.log &
64
+
65
echo 'Waiting for vllm api server to start...'
66
while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
67
+
68
echo 'Starting gradio server...'
50
- python vllm/examples/gradio_webserver.py
69
+ git clone https://github.com/vllm-project/vllm.git || true
70
+ python vllm/examples/gradio_openai_chatbot_webserver.py \
71
+ -m $MODEL_NAME \
72
+ --port 8811 \
73
+ --model-url http://localhost:8081/v1 \
74
+ --stop-token-ids 128009,128001
75
52
-Start the serving the LLaMA-13B model on an A100 GPU:
76
+Start the serving the Llama-3 8B model on any of the candidate GPUs listed (L4, A10g, ...):
77
78
.. code-block:: console
79
56
- $ sky launch serving.yaml
80
+ HF_TOKEN="your-huggingface-token" sky launch serving.yaml --env HF_TOKEN
81
82
Check the output of the command. There will be a shareable gradio link (like the last line of the following). Open it in your browser to use the LLaMA model to do the text completion.
83
@@ -61,9 +85,226 @@ Check the output of the command. There will be a shareable gradio link (like the
85
86
(task, pid=7431) Running on public URL: https://<gradio-hash>.gradio.live
87
64
-**Optional**: Serve the 65B model instead of the default 13B and use more GPU:
88
+**Optional**: Serve the 70B model instead of the default 8B and use more GPU:
89
+
90
+.. code-block:: console
91
+
92
+ HF_TOKEN="your-huggingface-token" sky launch serving.yaml --gpus A100:8 --env HF_TOKEN --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct
93
+
94
+
95
+Scale up to multiple replicas
96
+-----------------------------
97
+
98
+SkyPilot can scale up the service to multiple service replicas with built-in autoscaling, load-balancing and fault-tolerance. You can do it by adding a services section to the YAML file.
99
+
100
+.. code-block:: yaml
101
+
102
+ service:
103
+ replicas: 2
104
+ # An actual request for readiness probe.
105
+ readiness_probe:
106
+ path: /v1/chat/completions
107
+ post_data:
108
+ model: $MODEL_NAME
109
+ messages:
110
+ - role: user
111
+ content: Hello! What is your name?
112
+ max_tokens: 1
113
+
114
+.. raw:: html
115
+
116
+ <details>
117
+ <summary>Click to see the full recipe YAML</summary>
118
+
119
+
120
+.. code-block:: yaml
121
+
122
+ service:
123
+ replicas: 2
124
+ # An actual request for readiness probe.
125
+ readiness_probe:
126
+ path: /v1/chat/completions
127
+ post_data:
128
+ model: $MODEL_NAME
129
+ messages:
130
+ - role: user
131
+ content: Hello! What is your name?
132
+ max_tokens: 1
133
+
134
+ resources:
135
+ accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
136
+ use_spot: True
137
+ disk_size: 512 # Ensure model checkpoints can fit.
138
+ disk_tier: best
139
+ ports: 8081 # Expose to internet traffic.
140
+
141
+ envs:
142
+ MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
143
+ HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
144
+
145
+ setup: |
146
+ conda create -n vllm python=3.10 -y
147
+ conda activate vllm
148
+
149
+ pip install vllm==0.4.0.post1
150
+ # Install Gradio for web UI.
151
+ pip install gradio openai
152
+ pip install flash-attn==2.5.7
153
+
154
+ run: |
155
+ conda activate vllm
156
+ echo 'Starting vllm api server...'
157
+ python -u -m vllm.entrypoints.openai.api_server \
158
+ --port 8081 \
159
+ --model $MODEL_NAME \
160
+ --trust-remote-code \
161
+ --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
162
+ 2>&1 | tee api_server.log &
163
+
164
+ echo 'Waiting for vllm api server to start...'
165
+ while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
166
+
167
+ echo 'Starting gradio server...'
168
+ git clone https://github.com/vllm-project/vllm.git || true
169
+ python vllm/examples/gradio_openai_chatbot_webserver.py \
170
+ -m $MODEL_NAME \
171
+ --port 8811 \
172
+ --model-url http://localhost:8081/v1 \
173
+ --stop-token-ids 128009,128001
174
+
175
+.. raw:: html
176
+
177
+ </details>
178
+
179
+Start the serving the Llama-3 8B model on multiple replicas:
180
+
181
+.. code-block:: console
182
+
183
+ HF_TOKEN="your-huggingface-token" sky serve up -n vllm serving.yaml --env HF_TOKEN
184
+
185
+
186
+Wait until the service is ready:
187
188
.. code-block:: console
189
68
- sky launch -c vllm-serve-new -s serve.yaml --gpus A100:8 --env MODEL_NAME=decapoda-research/llama-65b-hf
190
+ watch -n10 sky serve status vllm
191
+
192
+
193
+.. raw:: html
194
+
195
+ <details>
196
+ <summary>Example outputs:</summary>
197
+
198
+.. code-block:: console
199
+
200
+ Services
201
+ NAME VERSION UPTIME STATUS REPLICAS ENDPOINT
202
+ vllm 1 35s READY 2/2 xx.yy.zz.100:30001
203
+
204
+ Service Replicas
205
+ SERVICE_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION
206
+ vllm 1 1 xx.yy.zz.121 18 mins ago 1x GCP({'L4': 1}) READY us-east4
207
+ vllm 2 1 xx.yy.zz.245 18 mins ago 1x GCP({'L4': 1}) READY us-east4
208
+
209
+.. raw:: html
210
+
211
+ </details>
212
+
213
+After the service is READY, you can find a single endpoint for the service and access the service with the endpoint:
214
+
215
+.. code-block:: console
216
+
217
+ ENDPOINT=$(sky serve status --endpoint 8081 vllm)
218
+ curl -L http://$ENDPOINT/v1/chat/completions \
219
+ -H "Content-Type: application/json" \
220
+ -d '{
221
+ "model": "meta-llama/Meta-Llama-3-8B-Instruct",
222
+ "messages": [
223
+ {
224
+ "role": "system",
225
+ "content": "You are a helpful assistant."
226
+ },
227
+ {
228
+ "role": "user",
229
+ "content": "Who are you?"
230
+ }
231
+ ],
232
+ "stop_token_ids": [128009, 128001]
233
+ }'
234
+
235
+To enable autoscaling, you could specify additional configs in `services`:
236
+
237
+.. code-block:: yaml
238
+
239
+ services:
240
+ replica_policy:
241
+ min_replicas: 0
242
+ max_replicas: 3
243
+ target_qps_per_replica: 2
244
+
245
+This will scale the service up to when the QPS exceeds 2 for each replica.
246
+
247
+
248
+**Optional**: Connect a GUI to the endpoint
249
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
250
+
251
+
252
+It is also possible to access the Llama-3 service with a separate GUI frontend, so the user requests send to the GUI will be load-balanced across replicas.
253
+
254
+.. raw:: html
255
+
256
+ <details>
257
+ <summary>Click to see the full GUI YAML</summary>
258
+
259
+.. code-block:: yaml
260
+
261
+ envs:
262
+ MODEL_NAME: meta-llama/Meta-Llama-3-70B-Instruct
263
+ ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.
264
+
265
+ resources:
266
+ cpus: 2
267
+
268
+ setup: |
269
+ conda activate vllm
270
+ if [ $? -ne 0 ]; then
271
+ conda create -n vllm python=3.10 -y
272
+ conda activate vllm
273
+ fi
274
+
275
+ # Install Gradio for web UI.
276
+ pip install gradio openai
277
+
278
+ run: |
279
+ conda activate vllm
280
+ export PATH=$PATH:/sbin
281
+ WORKER_IP=$(hostname -I | cut -d' ' -f1)
282
+ CONTROLLER_PORT=21001
283
+ WORKER_PORT=21002
284
+
285
+ echo 'Starting gradio server...'
286
+ git clone https://github.com/vllm-project/vllm.git || true
287
+ python vllm/examples/gradio_openai_chatbot_webserver.py \
288
+ -m $MODEL_NAME \
289
+ --port 8811 \
290
+ --model-url http://$ENDPOINT/v1 \
291
+ --stop-token-ids 128009,128001 | tee ~/gradio.log
292
+
293
+.. raw:: html
294
+
295
+ </details>
296
+
297
+1. Start the chat web UI:
298
+
299
+.. code-block:: console
300
+
301
+ sky launch -c gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint vllm)
302
+
303
+
304
+2. Then, we can access the GUI at the returned gradio link:
305
+
306
+.. code-block:: console
307
+
308
+ | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live
309
+
310