| 1 | # dstack |
| 2 | |
| 3 | <p align="center"> |
| 4 | <img src="https://i.ibb.co/71kx6hW/vllm-dstack.png" alt="vLLM_plus_dstack"/> |
| 5 | </p> |
| 6 | |
| 7 | vLLM can be run on a cloud based GPU machine with [dstack](https://dstack.ai/), an open-source framework for running LLMs on any cloud. This tutorial assumes that you have already configured credentials, gateway, and GPU quotas on your cloud environment. |
| 8 | |
| 9 | To install dstack client, run: |
| 10 | |
| 11 | ```bash |
| 12 | pip install dstack[all] |
| 13 | dstack server |
| 14 | ``` |
| 15 | |
| 16 | Next, to configure your dstack project, run: |
| 17 | |
| 18 | ```bash |
| 19 | mkdir -p vllm-dstack |
| 20 | cd vllm-dstack |
| 21 | dstack init |
| 22 | ``` |
| 23 | |
| 24 | Next, to provision a VM instance with LLM of your choice (`NousResearch/Llama-2-7b-chat-hf` for this example), create the following `serve.dstack.yml` file for the dstack `Service`: |
| 25 | |
| 26 | ??? code "Config" |
| 27 | |
| 28 | ```yaml |
| 29 | type: service |
| 30 | |
| 31 | python: "3.11" |
| 32 | env: |
| 33 | - MODEL=NousResearch/Llama-2-7b-chat-hf |
| 34 | port: 8000 |
| 35 | resources: |
| 36 | gpu: 24GB |
| 37 | commands: |
| 38 | - pip install vllm |
| 39 | - vllm serve $MODEL --port 8000 |
| 40 | model: |
| 41 | format: openai |
| 42 | type: chat |
| 43 | name: NousResearch/Llama-2-7b-chat-hf |
| 44 | ``` |
| 45 | |
| 46 | Then, run the following CLI for provisioning: |
| 47 | |
| 48 | ??? console "Command" |
| 49 | |
| 50 | ```console |
| 51 | $ dstack run . -f serve.dstack.yml |
| 52 | |
| 53 | ⠸ Getting run plan... |
| 54 | Configuration serve.dstack.yml |
| 55 | Project deep-diver-main |
| 56 | User deep-diver |
| 57 | Min resources 2..xCPU, 8GB.., 1xGPU (24GB) |
| 58 | Max price - |
| 59 | Max duration - |
| 60 | Spot policy auto |
| 61 | Retry policy no |
| 62 | |
| 63 | # BACKEND REGION INSTANCE RESOURCES SPOT PRICE |
| 64 | 1 gcp us-central1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804 |
| 65 | 2 gcp us-east1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804 |
| 66 | 3 gcp us-west1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804 |
| 67 | ... |
| 68 | Shown 3 of 193 offers, $5.876 max |
| 69 | |
| 70 | Continue? [y/n]: y |
| 71 | ⠙ Submitting run... |
| 72 | ⠏ Launching spicy-treefrog-1 (pulling) |
| 73 | spicy-treefrog-1 provisioning completed (running) |
| 74 | Service is published at ... |
| 75 | ``` |
| 76 | |
| 77 | After the provisioning, you can interact with the model by using the OpenAI SDK: |
| 78 | |
| 79 | ??? code |
| 80 | |
| 81 | ```python |
| 82 | from openai import OpenAI |
| 83 | |
| 84 | client = OpenAI( |
| 85 | base_url="https://gateway.<gateway domain>", |
| 86 | api_key="<YOUR-DSTACK-SERVER-ACCESS-TOKEN>", |
| 87 | ) |
| 88 | |
| 89 | completion = client.chat.completions.create( |
| 90 | model="NousResearch/Llama-2-7b-chat-hf", |
| 91 | messages=[ |
| 92 | { |
| 93 | "role": "user", |
| 94 | "content": "Compose a poem that explains the concept of recursion in programming.", |
| 95 | } |
| 96 | ], |
| 97 | ) |
| 98 | |
| 99 | print(completion.choices[0].message.content) |
| 100 | ``` |
| 101 | |
| 102 | !!! note |
| 103 | dstack automatically handles authentication on the gateway using dstack's tokens. Meanwhile, if you don't want to configure a gateway, you can provision dstack `Task` instead of `Service`. The `Task` is for development purpose only. If you want to know more about hands-on materials how to serve vLLM using dstack, check out [this repository](https://github.com/dstackai/dstack-examples/tree/main/deployment/vllm) |