@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 137 lines 3.75 KB
Rendered Raw
1 # Using Nginx
2
3 This document shows how to launch multiple vLLM serving containers and use Nginx to act as a load balancer between the servers.
4
5 ## Build Nginx Container
6
7 This guide assumes that you have just cloned the vLLM project and you're currently in the vllm root directory.
8
9 ```bash
10 export vllm_root=`pwd`
11 ```
12
13 Create a file named `Dockerfile.nginx`:
14
15 ```dockerfile
16 FROM nginx:latest
17 RUN rm /etc/nginx/conf.d/default.conf
18 EXPOSE 80
19 CMD ["nginx", "-g", "daemon off;"]
20 ```
21
22 Build the container:
23
24 ```bash
25 docker build . -f Dockerfile.nginx --tag nginx-lb
26 ```
27
28 ## Create Simple Nginx Config file
29
30 Create a file named `nginx_conf/nginx.conf`. Note that you can add as many servers as you'd like. In the below example we'll start with two. To add more, add another `server vllmN:8000 max_fails=3 fail_timeout=10000s;` entry to `upstream backend`.
31
32 ??? console "Config"
33
34 ```console
35 upstream backend {
36 least_conn;
37 server vllm0:8000 max_fails=3 fail_timeout=10000s;
38 server vllm1:8000 max_fails=3 fail_timeout=10000s;
39 }
40 server {
41 listen 80;
42 location / {
43 proxy_pass http://backend;
44 proxy_set_header Host $host;
45 proxy_set_header X-Real-IP $remote_addr;
46 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
47 proxy_set_header X-Forwarded-Proto $scheme;
48 }
49 }
50 ```
51
52 ## Build vLLM Container
53
54 ```bash
55 cd $vllm_root
56 docker build -f docker/Dockerfile . --tag vllm
57 ```
58
59 If you are behind proxy, you can pass the proxy settings to the docker build command as shown below:
60
61 ```bash
62 cd $vllm_root
63 docker build \
64 -f docker/Dockerfile . \
65 --tag vllm \
66 --build-arg http_proxy=$http_proxy \
67 --build-arg https_proxy=$https_proxy
68 ```
69
70 ## Create Docker Network
71
72 ```bash
73 docker network create vllm_nginx
74 ```
75
76 ## Launch vLLM Containers
77
78 Notes:
79
80 - If you have your HuggingFace models cached somewhere else, update `hf_cache_dir` below.
81 - If you don't have an existing HuggingFace cache you will want to start `vllm0` and wait for the model to complete downloading and the server to be ready. This will ensure that `vllm1` can leverage the model you just downloaded and it won't have to be downloaded again.
82 - The below example assumes GPU backend used. If you are using CPU backend, remove `--gpus device=ID`, add `VLLM_CPU_KVCACHE_SPACE` and `VLLM_CPU_OMP_THREADS_BIND` environment variables to the docker run command.
83 - Adjust the model name that you want to use in your vLLM servers if you don't want to use `Llama-2-7b-chat-hf`.
84
85 ??? console "Commands"
86
87 ```console
88 mkdir -p ~/.cache/huggingface/hub/
89 hf_cache_dir=~/.cache/huggingface/
90 docker run \
91 -itd \
92 --ipc host \
93 --network vllm_nginx \
94 --gpus device=0 \
95 --shm-size=10.24gb \
96 -v $hf_cache_dir:/root/.cache/huggingface/ \
97 -p 8081:8000 \
98 --name vllm0 vllm \
99 --model meta-llama/Llama-2-7b-chat-hf
100 docker run \
101 -itd \
102 --ipc host \
103 --network vllm_nginx \
104 --gpus device=1 \
105 --shm-size=10.24gb \
106 -v $hf_cache_dir:/root/.cache/huggingface/ \
107 -p 8082:8000 \
108 --name vllm1 vllm \
109 --model meta-llama/Llama-2-7b-chat-hf
110 ```
111
112 !!! note
113 If you are behind proxy, you can pass the proxy settings to the docker run command via `-e http_proxy=$http_proxy -e https_proxy=$https_proxy`.
114
115 ## Launch Nginx
116
117 ```bash
118 docker run \
119 -itd \
120 -p 8000:80 \
121 --network vllm_nginx \
122 -v ./nginx_conf/:/etc/nginx/conf.d/ \
123 --name nginx-lb nginx-lb:latest
124 ```
125
126 ## Verify That vLLM Servers Are Ready
127
128 ```bash
129 docker logs vllm0 | grep Uvicorn
130 docker logs vllm1 | grep Uvicorn
131 ```
132
133 Both outputs should look like this:
134
135 ```console
136 INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
137 ```