@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 58df2883cb
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 08:00 UTC

[Doc] Add doc for running vLLM on the cloud (#426)

Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>

Zhanghao Wu committed Jul 16, 2023 at 13:37 UTC 58df2883cb3d3813e1d09ba691744773d9dcae58
2 files changed +70
docs/source/index.rst
+1
@@ -62,6 +62,7 @@ Documentation
62 :caption: Serving
63
64 serving/distributed_serving
65 + serving/run_on_sky
66
67 .. toctree::
68 :maxdepth: 1
docs/source/serving/run_on_sky.rst new
+69
@@ -0,0 +1,69 @@
1 +.. _on_cloud:
2 +
3 +Running on clouds with SkyPilot
4 +===============================
5 +
6 +.. raw:: html
7 +
8 + <p align="center">
9 + <img src="https://imgur.com/yxtzPEu.png" alt="vLLM"/>
10 + </p>
11 +
12 +vLLM can be run on the cloud to scale to multiple GPUs with `SkyPilot <https://github.com/skypilot-org/skypilot>`__, an open-source framework for running LLMs on any cloud.
13 +
14 +To install SkyPilot and setup your cloud credentials, run:
15 +
16 +.. code-block:: console
17 +
18 + $ pip install skypilot
19 + $ sky check
20 +
21 +See the vLLM SkyPilot YAML for serving, `serving.yaml <https://github.com/skypilot-org/skypilot/blob/master/llm/vllm/serve.yaml>`__.
22 +
23 +.. code-block:: yaml
24 +
25 + resources:
26 + accelerators: A100
27 +
28 + envs:
29 + MODEL_NAME: decapoda-research/llama-13b-hf
30 + TOKENIZER: hf-internal-testing/llama-tokenizer
31 +
32 + setup: |
33 + conda create -n vllm python=3.9 -y
34 + conda activate vllm
35 + git clone https://github.com/vllm-project/vllm.git
36 + cd vllm
37 + pip install .
38 + pip install gradio
39 +
40 + run: |
41 + conda activate vllm
42 + echo 'Starting vllm api server...'
43 + python -u -m vllm.entrypoints.api_server \
44 + --model $MODEL_NAME \
45 + --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
46 + --tokenizer $TOKENIZER 2>&1 | tee api_server.log &
47 + echo 'Waiting for vllm api server to start...'
48 + while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
49 + echo 'Starting gradio server...'
50 + python vllm/examples/gradio_webserver.py
51 +
52 +Start the serving the LLaMA-13B model on an A100 GPU:
53 +
54 +.. code-block:: console
55 +
56 + $ sky launch serving.yaml
57 +
58 +Check the output of the command. There will be a sharable gradio link (like the last line of the following). Open it in your browser to use the LLaMA model to do the text completion.
59 +
60 +.. code-block:: console
61 +
62 + (task, pid=7431) Running on public URL: https://<gradio-hash>.gradio.live
63 +
64 +**Optional**: Serve the 65B model instead of the default 13B and use more GPU:
65 +
66 +.. code-block:: console
67 +
68 + sky launch -c vllm-serve-new -s serve.yaml --gpus A100:8 --env MODEL_NAME=decapoda-research/llama-65b-hf
69 +