A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 08:00 UTC
[Doc] Add doc for running vLLM on the cloud (#426)
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Zhanghao Wu committed
Jul 16, 2023 at 13:37 UTC
58df2883cb3d3813e1d09ba691744773d9dcae58
2 files changed
+70
docs/source/index.rst
+1
@@ -62,6 +62,7 @@ Documentation
62
:caption: Serving
63
64
serving/distributed_serving
65
+ serving/run_on_sky
66
67
.. toctree::
68
:maxdepth: 1
docs/source/serving/run_on_sky.rst
new
+69
@@ -0,0 +1,69 @@
1
+.. _on_cloud:
2
+
3
+Running on clouds with SkyPilot
4
+===============================
5
+
6
+.. raw:: html
7
+
8
+ <p align="center">
9
+ <img src="https://imgur.com/yxtzPEu.png" alt="vLLM"/>
10
+ </p>
11
+
12
+vLLM can be run on the cloud to scale to multiple GPUs with `SkyPilot <https://github.com/skypilot-org/skypilot>`__, an open-source framework for running LLMs on any cloud.
13
+
14
+To install SkyPilot and setup your cloud credentials, run:
15
+
16
+.. code-block:: console
17
+
18
+ $ pip install skypilot
19
+ $ sky check
20
+
21
+See the vLLM SkyPilot YAML for serving, `serving.yaml <https://github.com/skypilot-org/skypilot/blob/master/llm/vllm/serve.yaml>`__.
22
+
23
+.. code-block:: yaml
24
+
25
+ resources:
26
+ accelerators: A100
27
+
28
+ envs:
29
+ MODEL_NAME: decapoda-research/llama-13b-hf
30
+ TOKENIZER: hf-internal-testing/llama-tokenizer
31
+
32
+ setup: |
33
+ conda create -n vllm python=3.9 -y
34
+ conda activate vllm
35
+ git clone https://github.com/vllm-project/vllm.git
36
+ cd vllm
37
+ pip install .
38
+ pip install gradio
39
+
40
+ run: |
41
+ conda activate vllm
42
+ echo 'Starting vllm api server...'
43
+ python -u -m vllm.entrypoints.api_server \
44
+ --model $MODEL_NAME \
45
+ --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
46
+ --tokenizer $TOKENIZER 2>&1 | tee api_server.log &
47
+ echo 'Waiting for vllm api server to start...'
48
+ while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
49
+ echo 'Starting gradio server...'
50
+ python vllm/examples/gradio_webserver.py
51
+
52
+Start the serving the LLaMA-13B model on an A100 GPU:
53
+
54
+.. code-block:: console
55
+
56
+ $ sky launch serving.yaml
57
+
58
+Check the output of the command. There will be a sharable gradio link (like the last line of the following). Open it in your browser to use the LLaMA model to do the text completion.
59
+
60
+.. code-block:: console
61
+
62
+ (task, pid=7431) Running on public URL: https://<gradio-hash>.gradio.live
63
+
64
+**Optional**: Serve the 65B model instead of the default 13B and use more GPU:
65
+
66
+.. code-block:: console
67
+
68
+ sky launch -c vllm-serve-new -s serve.yaml --gpus A100:8 --env MODEL_NAME=decapoda-research/llama-65b-hf
69
+