A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 09:30 UTC
[Hardware][Intel CPU][DOC] Update docs for CPU backend (#6212)
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> Co-authored-by: Rafael Vasquez <rafvasq21@gmail.com> Co-authored-by: Gubrud, Aaron D <aaron.d.gubrud@intel.com> Co-authored-by: adgubrud <96072084+adgubrud@users.noreply.github.com>
Yuan committed
Oct 22, 2024 at 10:38 UTC
32a1ee74a0838e37e3b9dea2312ada925011c5ba
3 files changed
+165
-1
docs/source/getting_started/cpu-installation.rst
+22
-1
@@ -3,7 +3,13 @@
3
Installation with CPU
4
========================
5
6
-vLLM initially supports basic model inferencing and serving on x86 CPU platform, with data types FP32 and BF16.
6
+vLLM initially supports basic model inferencing and serving on x86 CPU platform, with data types FP32 and BF16. vLLM CPU backend supports the following vLLM features:
7
+
8
+- Tensor Parallel (``-tp = N``)
9
+- Quantization (``INT8 W8A8, AWQ``)
10
+
11
+.. note::
12
+ FP16 data type and more advanced features on `chunked-prefill`, `prefix-caching` and `FP8 KV cache` are under development and will be available soon.
13
14
Table of contents:
15
@@ -141,5 +147,20 @@ Performance tips
147
148
- If using vLLM CPU backend on a multi-socket machine with NUMA, be aware to set CPU cores using ``VLLM_CPU_OMP_THREADS_BIND`` to avoid cross NUMA node memory access.
149
150
+CPU Backend Considerations
151
+--------------------------
152
+
153
+- The CPU backend significantly differs from the GPU backend since the vLLM architecture was originally optimized for GPU use. A number of optimizations are needed to enhance its performance.
154
+
155
+- Decouple the HTTP serving components from the inference components. In a GPU backend configuration, the HTTP serving and tokenization tasks operate on the CPU, while inference runs on the GPU, which typically does not pose a problem. However, in a CPU-based setup, the HTTP serving and tokenization can cause significant context switching and reduced cache efficiency. Therefore, it is strongly recommended to segregate these two components for improved performance.
156
+
157
+- On CPU based setup with NUMA enabled, the memory access performance may be largely impacted by the `topology <https://github.com/intel/intel-extension-for-pytorch/blob/main/docs/tutorials/performance_tuning/tuning_guide.md#non-uniform-memory-access-numa>`_. For NUMA architecture, two optimizations are to recommended: Tensor Parallel or Data Parallel.
158
+
159
+ * Using Tensor Parallel for a latency constraints deployment: following GPU backend design, a Megatron-LM's parallel algorithm will be used to shard the model, based on the number of NUMA nodes (e.g. TP = 2 for a two NUMA node system). With `TP feature on CPU <https://github.com/vllm-project/vllm/pull/6125>`_ merged, Tensor Parallel is supported for serving and offline inferencing. In general each NUMA node is treated as one GPU card. Below is the example script to enable Tensor Parallel = 2 for serving:
160
+
161
+ .. code-block:: console
162
+
163
+ $ VLLM_CPU_KVCACHE_SPACE=40 VLLM_CPU_OMP_THREADS_BIND="0-31|32-63" vllm serve meta-llama/Llama-2-7b-chat-hf -tp=2 --distributed-executor-backend mp
164
165
166
+ * Using Data Parallel for maximum throughput: to launch an LLM serving endpoint on each NUMA node along with one additional load balancer to dispatch the requests to those endpoints. Common solutions like `Nginx <../serving/deploying_with_nginx.html>`_ or HAProxy are recommended. Anyscale Ray project provides the feature on LLM `serving <https://docs.ray.io/en/latest/serve/index.html>`_. Here is the example to setup a scalable LLM serving with `Ray Serve <https://github.com/intel/llm-on-ray/blob/main/docs/setup.md>`_.
\ No newline at end of file
docs/source/index.rst
+1
@@ -80,6 +80,7 @@ Documentation
80
serving/openai_compatible_server
81
serving/deploying_with_docker
82
serving/deploying_with_k8s
83
+ serving/deploying_with_nginx
84
serving/distributed_serving
85
serving/metrics
86
serving/env_vars
docs/source/serving/deploying_with_nginx.rst
new
+142
@@ -0,0 +1,142 @@
1
+.. _nginxloadbalancer:
2
+
3
+Deploying with Nginx Loadbalancer
4
+=================================
5
+
6
+This document shows how to launch multiple vLLM serving containers and use Nginx to act as a load balancer between the servers.
7
+
8
+Table of contents:
9
+
10
+#. :ref:`Build Nginx Container <nginxloadbalancer_nginx_build>`
11
+#. :ref:`Create Simple Nginx Config file <nginxloadbalancer_nginx_conf>`
12
+#. :ref:`Build vLLM Container <nginxloadbalancer_nginx_vllm_container>`
13
+#. :ref:`Create Docker Network <nginxloadbalancer_nginx_docker_network>`
14
+#. :ref:`Launch vLLM Containers <nginxloadbalancer_nginx_launch_container>`
15
+#. :ref:`Launch Nginx <nginxloadbalancer_nginx_launch_nginx>`
16
+#. :ref:`Verify That vLLM Servers Are Ready <nginxloadbalancer_nginx_verify_nginx>`
17
+
18
+.. _nginxloadbalancer_nginx_build:
19
+
20
+Build Nginx Container
21
+---------------------
22
+
23
+This guide assumes that you have just cloned the vLLM project and you're currently in the vllm root directory.
24
+
25
+.. code-block:: console
26
+
27
+ export vllm_root=`pwd`
28
+
29
+Create a file named ``Dockerfile.nginx``:
30
+
31
+.. code-block:: console
32
+
33
+ FROM nginx:latest
34
+ RUN rm /etc/nginx/conf.d/default.conf
35
+ EXPOSE 80
36
+ CMD ["nginx", "-g", "daemon off;"]
37
+
38
+Build the container:
39
+
40
+.. code-block:: console
41
+
42
+ docker build . -f Dockerfile.nginx --tag nginx-lb
43
+
44
+.. _nginxloadbalancer_nginx_conf:
45
+
46
+Create Simple Nginx Config file
47
+-------------------------------
48
+
49
+Create a file named ``nginx_conf/nginx.conf``. Note that you can add as many servers as you'd like. In the below example we'll start with two. To add more, add another ``server vllmN:8000 max_fails=3 fail_timeout=10000s;`` entry to ``upstream backend``.
50
+
51
+.. code-block:: console
52
+
53
+ upstream backend {
54
+ least_conn;
55
+ server vllm0:8000 max_fails=3 fail_timeout=10000s;
56
+ server vllm1:8000 max_fails=3 fail_timeout=10000s;
57
+ }
58
+ server {
59
+ listen 80;
60
+ location / {
61
+ proxy_pass http://backend;
62
+ proxy_set_header Host $host;
63
+ proxy_set_header X-Real-IP $remote_addr;
64
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
65
+ proxy_set_header X-Forwarded-Proto $scheme;
66
+ }
67
+ }
68
+
69
+.. _nginxloadbalancer_nginx_vllm_container:
70
+
71
+Build vLLM Container
72
+--------------------
73
+
74
+.. code-block:: console
75
+
76
+ cd $vllm_root
77
+ docker build -f Dockerfile . --tag vllm
78
+
79
+
80
+If you are behind proxy, you can pass the proxy settings to the docker build command as shown below:
81
+
82
+.. code-block:: console
83
+
84
+ cd $vllm_root
85
+ docker build -f Dockerfile . --tag vllm --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy
86
+
87
+.. _nginxloadbalancer_nginx_docker_network:
88
+
89
+Create Docker Network
90
+---------------------
91
+
92
+.. code-block:: console
93
+
94
+ docker network create vllm_nginx
95
+
96
+
97
+.. _nginxloadbalancer_nginx_launch_container:
98
+
99
+Launch vLLM Containers
100
+----------------------
101
+
102
+Notes:
103
+
104
+* If you have your HuggingFace models cached somewhere else, update ``hf_cache_dir`` below.
105
+* If you don't have an existing HuggingFace cache you will want to start ``vllm0`` and wait for the model to complete downloading and the server to be ready. This will ensure that ``vllm1`` can leverage the model you just downloaded and it won't have to be downloaded again.
106
+* The below example assumes GPU backend used. If you are using CPU backend, remove ``--gpus all``, add ``VLLM_CPU_KVCACHE_SPACE`` and ``VLLM_CPU_OMP_THREADS_BIND`` environment variables to the docker run command.
107
+* Adjust the model name that you want to use in your vLLM servers if you don't want to use ``Llama-2-7b-chat-hf``.
108
+
109
+.. code-block:: console
110
+
111
+ mkdir -p ~/.cache/huggingface/hub/
112
+ hf_cache_dir=~/.cache/huggingface/
113
+ docker run -itd --ipc host --privileged --network vllm_nginx --gpus all --shm-size=10.24gb -v $hf_cache_dir:/root/.cache/huggingface/ -p 8081:8000 --name vllm0 vllm --model meta-llama/Llama-2-7b-chat-hf
114
+ docker run -itd --ipc host --privileged --network vllm_nginx --gpus all --shm-size=10.24gb -v $hf_cache_dir:/root/.cache/huggingface/ -p 8082:8000 --name vllm1 vllm --model meta-llama/Llama-2-7b-chat-hf
115
+
116
+.. note::
117
+ If you are behind proxy, you can pass the proxy settings to the docker run command via ``-e http_proxy=$http_proxy -e https_proxy=$https_proxy``.
118
+
119
+.. _nginxloadbalancer_nginx_launch_nginx:
120
+
121
+Launch Nginx
122
+------------
123
+
124
+.. code-block:: console
125
+
126
+ docker run -itd -p 8000:80 --network vllm_nginx -v ./nginx_conf/:/etc/nginx/conf.d/ --name nginx-lb nginx-lb:latest
127
+
128
+.. _nginxloadbalancer_nginx_verify_nginx:
129
+
130
+Verify That vLLM Servers Are Ready
131
+----------------------------------
132
+
133
+.. code-block:: console
134
+
135
+ docker logs vllm0 | grep Uvicorn
136
+ docker logs vllm1 | grep Uvicorn
137
+
138
+Both outputs should look like this:
139
+
140
+.. code-block:: console
141
+
142
+ INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)