@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 429d89720e
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 00:30 UTC

add doc about serving option on dstack (#3074)

Co-authored-by: Roger Wang <ywang@roblox.com>

Chansung Park committed May 31, 2024 at 02:11 UTC 429d89720e41901c3c0499a8ed3ad5be693cc945
2 files changed +104
docs/source/serving/deploying_with_dstack.rst new
+103
@@ -0,0 +1,103 @@
1 +.. _deploying_with_dstack:
2 +
3 +Deploying with dstack
4 +============================
5 +
6 +.. raw:: html
7 +
8 + <p align="center">
9 + <img src="https://i.ibb.co/71kx6hW/vllm-dstack.png" alt="vLLM_plus_dstack"/>
10 + </p>
11 +
12 +vLLM can be run on a cloud based GPU machine with `dstack <https://dstack.ai/>`__, an open-source framework for running LLMs on any cloud. This tutorial assumes that you have already configured credentials, gateway, and GPU quotas on your cloud environment.
13 +
14 +To install dstack client, run:
15 +
16 +.. code-block:: console
17 +
18 + $ pip install "dstack[all]
19 + $ dstack server
20 +
21 +Next, to configure your dstack project, run:
22 +
23 +.. code-block:: console
24 +
25 + $ mkdir -p vllm-dstack
26 + $ cd vllm-dstack
27 + $ dstack init
28 +
29 +Next, to provision a VM instance with LLM of your choice(`NousResearch/Llama-2-7b-chat-hf` for this example), create the following `serve.dstack.yml` file for the dstack `Service`:
30 +
31 +.. code-block:: yaml
32 +
33 + type: service
34 +
35 + python: "3.11"
36 + env:
37 + - MODEL=NousResearch/Llama-2-7b-chat-hf
38 + port: 8000
39 + resources:
40 + gpu: 24GB
41 + commands:
42 + - pip install vllm
43 + - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000
44 + model:
45 + format: openai
46 + type: chat
47 + name: NousResearch/Llama-2-7b-chat-hf
48 +
49 +Then, run the following CLI for provisioning:
50 +
51 +.. code-block:: console
52 +
53 + $ dstack run . -f serve.dstack.yml
54 +
55 + ⠸ Getting run plan...
56 + Configuration serve.dstack.yml
57 + Project deep-diver-main
58 + User deep-diver
59 + Min resources 2..xCPU, 8GB.., 1xGPU (24GB)
60 + Max price -
61 + Max duration -
62 + Spot policy auto
63 + Retry policy no
64 +
65 + # BACKEND REGION INSTANCE RESOURCES SPOT PRICE
66 + 1 gcp us-central1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804
67 + 2 gcp us-east1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804
68 + 3 gcp us-west1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804
69 + ...
70 + Shown 3 of 193 offers, $5.876 max
71 +
72 + Continue? [y/n]: y
73 + ⠙ Submitting run...
74 + ⠏ Launching spicy-treefrog-1 (pulling)
75 + spicy-treefrog-1 provisioning completed (running)
76 + Service is published at ...
77 +
78 +After the provisioning, you can interact with the model by using the OpenAI SDK:
79 +
80 +.. code-block:: python
81 +
82 + from openai import OpenAI
83 +
84 + client = OpenAI(
85 + base_url="https://gateway.<gateway domain>",
86 + api_key="<YOUR-DSTACK-SERVER-ACCESS-TOKEN>"
87 + )
88 +
89 + completion = client.chat.completions.create(
90 + model="NousResearch/Llama-2-7b-chat-hf",
91 + messages=[
92 + {
93 + "role": "user",
94 + "content": "Compose a poem that explains the concept of recursion in programming.",
95 + }
96 + ]
97 + )
98 +
99 + print(completion.choices[0].message.content)
100 +
101 +.. note::
102 +
103 + dstack automatically handles authentication on the gateway using dstack's tokens. Meanwhile, if you don't want to configure a gateway, you can provision dstack `Task` instead of `Service`. The `Task` is for development purpose only. If you want to know more about hands-on materials how to serve vLLM using dstack, check out `this repository <https://github.com/dstackai/dstack-examples/tree/main/deployment/vllm>`__
docs/source/serving/integrations.rst
+1
@@ -9,4 +9,5 @@ Integrations
9 deploying_with_triton
10 deploying_with_bentoml
11 deploying_with_lws
12 + deploying_with_dstack
13 serving_with_langchain