@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 150a1ffbfd
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 20:45 UTC

[Doc] Update SkyPilot doc for wrong indents and instructions for update service (#4283)

Zhanghao Wu committed Jul 26, 2024 at 17:39 UTC 150a1ffbfd3d0429d30fa5ab841f53903a0a8a62
1 file changed +243 -187
docs/source/serving/run_on_sky.rst
+243 -187
@@ -5,9 +5,9 @@ Deploying and scaling up with SkyPilot
5
6 .. raw:: html
7
8 - <p align="center">
9 - <img src="https://imgur.com/yxtzPEu.png" alt="vLLM"/>
10 - </p>
8 + <p align="center">
9 + <img src="https://imgur.com/yxtzPEu.png" alt="vLLM"/>
10 + </p>
11
12 vLLM can be **run and scaled to multiple service replicas on clouds and Kubernetes** with `SkyPilot <https://github.com/skypilot-org/skypilot>`__, an open-source framework for running LLMs on any cloud. More examples for various open models, such as Llama-3, Mixtral, etc, can be found in `SkyPilot AI gallery <https://skypilot.readthedocs.io/en/latest/gallery/index.html>`__.
13
@@ -21,8 +21,8 @@ Prerequisites
21
22 .. code-block:: console
23
24 - pip install skypilot-nightly
25 - sky check
24 + pip install skypilot-nightly
25 + sky check
26
27
28 Run on a single instance
@@ -32,64 +32,64 @@ See the vLLM SkyPilot YAML for serving, `serving.yaml <https://github.com/skypil
32
33 .. code-block:: yaml
34
35 - resources:
36 - accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
37 - use_spot: True
38 - disk_size: 512 # Ensure model checkpoints can fit.
39 - disk_tier: best
40 - ports: 8081 # Expose to internet traffic.
41 -
42 - envs:
43 - MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
44 - HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
45 -
46 - setup: |
47 - conda create -n vllm python=3.10 -y
48 - conda activate vllm
49 -
50 - pip install vllm==0.4.0.post1
51 - # Install Gradio for web UI.
52 - pip install gradio openai
53 - pip install flash-attn==2.5.7
54 -
55 - run: |
56 - conda activate vllm
57 - echo 'Starting vllm api server...'
58 - python -u -m vllm.entrypoints.openai.api_server \
59 - --port 8081 \
60 - --model $MODEL_NAME \
61 - --trust-remote-code \
62 - --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
63 - 2>&1 | tee api_server.log &
64 -
65 - echo 'Waiting for vllm api server to start...'
66 - while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
67 -
68 - echo 'Starting gradio server...'
69 - git clone https://github.com/vllm-project/vllm.git || true
70 - python vllm/examples/gradio_openai_chatbot_webserver.py \
71 - -m $MODEL_NAME \
72 - --port 8811 \
73 - --model-url http://localhost:8081/v1 \
74 - --stop-token-ids 128009,128001
35 + resources:
36 + accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
37 + use_spot: True
38 + disk_size: 512 # Ensure model checkpoints can fit.
39 + disk_tier: best
40 + ports: 8081 # Expose to internet traffic.
41 +
42 + envs:
43 + MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
44 + HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
45 +
46 + setup: |
47 + conda create -n vllm python=3.10 -y
48 + conda activate vllm
49 +
50 + pip install vllm==0.4.0.post1
51 + # Install Gradio for web UI.
52 + pip install gradio openai
53 + pip install flash-attn==2.5.7
54 +
55 + run: |
56 + conda activate vllm
57 + echo 'Starting vllm api server...'
58 + python -u -m vllm.entrypoints.openai.api_server \
59 + --port 8081 \
60 + --model $MODEL_NAME \
61 + --trust-remote-code \
62 + --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
63 + 2>&1 | tee api_server.log &
64 +
65 + echo 'Waiting for vllm api server to start...'
66 + while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
67 +
68 + echo 'Starting gradio server...'
69 + git clone https://github.com/vllm-project/vllm.git || true
70 + python vllm/examples/gradio_openai_chatbot_webserver.py \
71 + -m $MODEL_NAME \
72 + --port 8811 \
73 + --model-url http://localhost:8081/v1 \
74 + --stop-token-ids 128009,128001
75
76 Start the serving the Llama-3 8B model on any of the candidate GPUs listed (L4, A10g, ...):
77
78 .. code-block:: console
79
80 - HF_TOKEN="your-huggingface-token" sky launch serving.yaml --env HF_TOKEN
80 + HF_TOKEN="your-huggingface-token" sky launch serving.yaml --env HF_TOKEN
81
82 Check the output of the command. There will be a shareable gradio link (like the last line of the following). Open it in your browser to use the LLaMA model to do the text completion.
83
84 .. code-block:: console
85
86 - (task, pid=7431) Running on public URL: https://<gradio-hash>.gradio.live
86 + (task, pid=7431) Running on public URL: https://<gradio-hash>.gradio.live
87
88 **Optional**: Serve the 70B model instead of the default 8B and use more GPU:
89
90 .. code-block:: console
91
92 - HF_TOKEN="your-huggingface-token" sky launch serving.yaml --gpus A100:8 --env HF_TOKEN --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct
92 + HF_TOKEN="your-huggingface-token" sky launch serving.yaml --gpus A100:8 --env HF_TOKEN --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct
93
94
95 Scale up to multiple replicas
@@ -99,151 +99,212 @@ SkyPilot can scale up the service to multiple service replicas with built-in aut
99
100 .. code-block:: yaml
101
102 - service:
103 - replicas: 2
104 - # An actual request for readiness probe.
105 - readiness_probe:
106 - path: /v1/chat/completions
107 - post_data:
108 - model: $MODEL_NAME
109 - messages:
110 - - role: user
111 - content: Hello! What is your name?
112 - max_tokens: 1
113 -
102 + service:
103 + replicas: 2
104 + # An actual request for readiness probe.
105 + readiness_probe:
106 + path: /v1/chat/completions
107 + post_data:
108 + model: $MODEL_NAME
109 + messages:
110 + - role: user
111 + content: Hello! What is your name?
112 + max_tokens: 1
113 +
114 .. raw:: html
115
116 - <details>
117 - <summary>Click to see the full recipe YAML</summary>
116 + <details>
117 + <summary>Click to see the full recipe YAML</summary>
118
119
120 .. code-block:: yaml
121
122 - service:
123 - replicas: 2
124 - # An actual request for readiness probe.
125 - readiness_probe:
126 - path: /v1/chat/completions
127 - post_data:
128 - model: $MODEL_NAME
129 - messages:
130 - - role: user
131 - content: Hello! What is your name?
122 + service:
123 + replicas: 2
124 + # An actual request for readiness probe.
125 + readiness_probe:
126 + path: /v1/chat/completions
127 + post_data:
128 + model: $MODEL_NAME
129 + messages:
130 + - role: user
131 + content: Hello! What is your name?
132 max_tokens: 1
133
134 - resources:
135 - accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
136 - use_spot: True
137 - disk_size: 512 # Ensure model checkpoints can fit.
138 - disk_tier: best
139 - ports: 8081 # Expose to internet traffic.
140 -
141 - envs:
142 - MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
143 - HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
144 -
145 - setup: |
146 - conda create -n vllm python=3.10 -y
147 - conda activate vllm
148 -
149 - pip install vllm==0.4.0.post1
150 - # Install Gradio for web UI.
151 - pip install gradio openai
152 - pip install flash-attn==2.5.7
153 -
154 - run: |
155 - conda activate vllm
156 - echo 'Starting vllm api server...'
157 - python -u -m vllm.entrypoints.openai.api_server \
158 - --port 8081 \
159 - --model $MODEL_NAME \
160 - --trust-remote-code \
161 - --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
162 - 2>&1 | tee api_server.log &
163 -
164 - echo 'Waiting for vllm api server to start...'
165 - while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
166 -
167 - echo 'Starting gradio server...'
168 - git clone https://github.com/vllm-project/vllm.git || true
169 - python vllm/examples/gradio_openai_chatbot_webserver.py \
170 - -m $MODEL_NAME \
171 - --port 8811 \
172 - --model-url http://localhost:8081/v1 \
173 - --stop-token-ids 128009,128001
134 + resources:
135 + accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
136 + use_spot: True
137 + disk_size: 512 # Ensure model checkpoints can fit.
138 + disk_tier: best
139 + ports: 8081 # Expose to internet traffic.
140 +
141 + envs:
142 + MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
143 + HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
144 +
145 + setup: |
146 + conda create -n vllm python=3.10 -y
147 + conda activate vllm
148 +
149 + pip install vllm==0.4.0.post1
150 + # Install Gradio for web UI.
151 + pip install gradio openai
152 + pip install flash-attn==2.5.7
153 +
154 + run: |
155 + conda activate vllm
156 + echo 'Starting vllm api server...'
157 + python -u -m vllm.entrypoints.openai.api_server \
158 + --port 8081 \
159 + --model $MODEL_NAME \
160 + --trust-remote-code \
161 + --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
162 + 2>&1 | tee api_server.log
163
164 .. raw:: html
165
177 - </details>
166 + </details>
167
168 Start the serving the Llama-3 8B model on multiple replicas:
169
170 .. code-block:: console
171
183 - HF_TOKEN="your-huggingface-token" sky serve up -n vllm serving.yaml --env HF_TOKEN
172 + HF_TOKEN="your-huggingface-token" sky serve up -n vllm serving.yaml --env HF_TOKEN
173
174
175 Wait until the service is ready:
176
177 .. code-block:: console
178
190 - watch -n10 sky serve status vllm
179 + watch -n10 sky serve status vllm
180
181
182 .. raw:: html
183
195 - <details>
196 - <summary>Example outputs:</summary>
184 + <details>
185 + <summary>Example outputs:</summary>
186
187 .. code-block:: console
188
200 - Services
201 - NAME VERSION UPTIME STATUS REPLICAS ENDPOINT
202 - vllm 1 35s READY 2/2 xx.yy.zz.100:30001
189 + Services
190 + NAME VERSION UPTIME STATUS REPLICAS ENDPOINT
191 + vllm 1 35s READY 2/2 xx.yy.zz.100:30001
192
204 - Service Replicas
205 - SERVICE_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION
206 - vllm 1 1 xx.yy.zz.121 18 mins ago 1x GCP({'L4': 1}) READY us-east4
207 - vllm 2 1 xx.yy.zz.245 18 mins ago 1x GCP({'L4': 1}) READY us-east4
193 + Service Replicas
194 + SERVICE_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION
195 + vllm 1 1 xx.yy.zz.121 18 mins ago 1x GCP([Spot]{'L4': 1}) READY us-east4
196 + vllm 2 1 xx.yy.zz.245 18 mins ago 1x GCP([Spot]{'L4': 1}) READY us-east4
197
198 .. raw:: html
210 -
211 - </details>
199 +
200 + </details>
201
202 After the service is READY, you can find a single endpoint for the service and access the service with the endpoint:
203
204 .. code-block:: console
205
217 - ENDPOINT=$(sky serve status --endpoint 8081 vllm)
218 - curl -L http://$ENDPOINT/v1/chat/completions \
219 - -H "Content-Type: application/json" \
220 - -d '{
221 - "model": "meta-llama/Meta-Llama-3-8B-Instruct",
222 - "messages": [
223 - {
224 - "role": "system",
225 - "content": "You are a helpful assistant."
226 - },
227 - {
228 - "role": "user",
229 - "content": "Who are you?"
230 - }
231 - ],
232 - "stop_token_ids": [128009, 128001]
233 - }'
234 -
235 -To enable autoscaling, you could specify additional configs in `services`:
206 + ENDPOINT=$(sky serve status --endpoint 8081 vllm)
207 + curl -L http://$ENDPOINT/v1/chat/completions \
208 + -H "Content-Type: application/json" \
209 + -d '{
210 + "model": "meta-llama/Meta-Llama-3-8B-Instruct",
211 + "messages": [
212 + {
213 + "role": "system",
214 + "content": "You are a helpful assistant."
215 + },
216 + {
217 + "role": "user",
218 + "content": "Who are you?"
219 + }
220 + ],
221 + "stop_token_ids": [128009, 128001]
222 + }'
223 +
224 +To enable autoscaling, you could replace the `replicas` with the following configs in `service`:
225
226 .. code-block:: yaml
227
239 - services:
240 - replica_policy:
241 - min_replicas: 0
242 - max_replicas: 3
243 - target_qps_per_replica: 2
228 + service:
229 + replica_policy:
230 + min_replicas: 2
231 + max_replicas: 4
232 + target_qps_per_replica: 2
233
234 This will scale the service up to when the QPS exceeds 2 for each replica.
235
236 +
237 +.. raw:: html
238 +
239 + <details>
240 + <summary>Click to see the full recipe YAML</summary>
241 +
242 +
243 +.. code-block:: yaml
244 +
245 + service:
246 + replica_policy:
247 + min_replicas: 2
248 + max_replicas: 4
249 + target_qps_per_replica: 2
250 + # An actual request for readiness probe.
251 + readiness_probe:
252 + path: /v1/chat/completions
253 + post_data:
254 + model: $MODEL_NAME
255 + messages:
256 + - role: user
257 + content: Hello! What is your name?
258 + max_tokens: 1
259 +
260 + resources:
261 + accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
262 + use_spot: True
263 + disk_size: 512 # Ensure model checkpoints can fit.
264 + disk_tier: best
265 + ports: 8081 # Expose to internet traffic.
266 +
267 + envs:
268 + MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
269 + HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
270 +
271 + setup: |
272 + conda create -n vllm python=3.10 -y
273 + conda activate vllm
274 +
275 + pip install vllm==0.4.0.post1
276 + # Install Gradio for web UI.
277 + pip install gradio openai
278 + pip install flash-attn==2.5.7
279 +
280 + run: |
281 + conda activate vllm
282 + echo 'Starting vllm api server...'
283 + python -u -m vllm.entrypoints.openai.api_server \
284 + --port 8081 \
285 + --model $MODEL_NAME \
286 + --trust-remote-code \
287 + --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
288 + 2>&1 | tee api_server.log
289 +
290 +
291 +.. raw:: html
292 +
293 + </details>
294 +
295 +To update the service with the new config:
296 +
297 +.. code-block:: console
298 +
299 + HF_TOKEN="your-huggingface-token" sky serve update vllm serving.yaml --env HF_TOKEN
300 +
301 +
302 +To stop the service:
303 +
304 +.. code-block:: console
305 +
306 + sky serve down vllm
307 +
308
309 **Optional**: Connect a GUI to the endpoint
310 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -253,58 +314,53 @@ It is also possible to access the Llama-3 service with a separate GUI frontend,
314
315 .. raw:: html
316
256 - <details>
257 - <summary>Click to see the full GUI YAML</summary>
317 + <details>
318 + <summary>Click to see the full GUI YAML</summary>
319
320 .. code-block:: yaml
321
261 - envs:
262 - MODEL_NAME: meta-llama/Meta-Llama-3-70B-Instruct
263 - ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.
264 -
265 - resources:
266 - cpus: 2
267 -
268 - setup: |
269 - conda activate vllm
270 - if [ $? -ne 0 ]; then
271 - conda create -n vllm python=3.10 -y
272 - conda activate vllm
273 - fi
274 -
275 - # Install Gradio for web UI.
276 - pip install gradio openai
277 -
278 - run: |
279 - conda activate vllm
280 - export PATH=$PATH:/sbin
281 - WORKER_IP=$(hostname -I | cut -d' ' -f1)
282 - CONTROLLER_PORT=21001
283 - WORKER_PORT=21002
284 -
285 - echo 'Starting gradio server...'
286 - git clone https://github.com/vllm-project/vllm.git || true
287 - python vllm/examples/gradio_openai_chatbot_webserver.py \
288 - -m $MODEL_NAME \
289 - --port 8811 \
290 - --model-url http://$ENDPOINT/v1 \
291 - --stop-token-ids 128009,128001 | tee ~/gradio.log
322 + envs:
323 + MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
324 + ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.
325 +
326 + resources:
327 + cpus: 2
328 +
329 + setup: |
330 + conda create -n vllm python=3.10 -y
331 + conda activate vllm
332 +
333 + # Install Gradio for web UI.
334 + pip install gradio openai
335 +
336 + run: |
337 + conda activate vllm
338 + export PATH=$PATH:/sbin
339 +
340 + echo 'Starting gradio server...'
341 + git clone https://github.com/vllm-project/vllm.git || true
342 + python vllm/examples/gradio_openai_chatbot_webserver.py \
343 + -m $MODEL_NAME \
344 + --port 8811 \
345 + --model-url http://$ENDPOINT/v1 \
346 + --stop-token-ids 128009,128001 | tee ~/gradio.log
347 +
348
349 .. raw:: html
294 -
295 - </details>
350 +
351 + </details>
352
353 1. Start the chat web UI:
354
355 .. code-block:: console
356
301 - sky launch -c gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint vllm)
357 + sky launch -c gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint vllm)
358
359
360 2. Then, we can access the GUI at the returned gradio link:
361
362 .. code-block:: console
363
308 - | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live
364 + | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live
365
366