@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 32a1ee74a0
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 09:30 UTC

[Hardware][Intel CPU][DOC] Update docs for CPU backend (#6212)

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> Co-authored-by: Rafael Vasquez <rafvasq21@gmail.com> Co-authored-by: Gubrud, Aaron D <aaron.d.gubrud@intel.com> Co-authored-by: adgubrud <96072084+adgubrud@users.noreply.github.com>

Yuan committed Oct 22, 2024 at 10:38 UTC 32a1ee74a0838e37e3b9dea2312ada925011c5ba
3 files changed +165 -1
docs/source/getting_started/cpu-installation.rst
+22 -1
@@ -3,7 +3,13 @@
3 Installation with CPU
4 ========================
5
6 -vLLM initially supports basic model inferencing and serving on x86 CPU platform, with data types FP32 and BF16.
6 +vLLM initially supports basic model inferencing and serving on x86 CPU platform, with data types FP32 and BF16. vLLM CPU backend supports the following vLLM features:
7 +
8 +- Tensor Parallel (``-tp = N``)
9 +- Quantization (``INT8 W8A8, AWQ``)
10 +
11 +.. note::
12 + FP16 data type and more advanced features on `chunked-prefill`, `prefix-caching` and `FP8 KV cache` are under development and will be available soon.
13
14 Table of contents:
15
@@ -141,5 +147,20 @@ Performance tips
147
148 - If using vLLM CPU backend on a multi-socket machine with NUMA, be aware to set CPU cores using ``VLLM_CPU_OMP_THREADS_BIND`` to avoid cross NUMA node memory access.
149
150 +CPU Backend Considerations
151 +--------------------------
152 +
153 +- The CPU backend significantly differs from the GPU backend since the vLLM architecture was originally optimized for GPU use. A number of optimizations are needed to enhance its performance.
154 +
155 +- Decouple the HTTP serving components from the inference components. In a GPU backend configuration, the HTTP serving and tokenization tasks operate on the CPU, while inference runs on the GPU, which typically does not pose a problem. However, in a CPU-based setup, the HTTP serving and tokenization can cause significant context switching and reduced cache efficiency. Therefore, it is strongly recommended to segregate these two components for improved performance.
156 +
157 +- On CPU based setup with NUMA enabled, the memory access performance may be largely impacted by the `topology <https://github.com/intel/intel-extension-for-pytorch/blob/main/docs/tutorials/performance_tuning/tuning_guide.md#non-uniform-memory-access-numa>`_. For NUMA architecture, two optimizations are to recommended: Tensor Parallel or Data Parallel.
158 +
159 + * Using Tensor Parallel for a latency constraints deployment: following GPU backend design, a Megatron-LM's parallel algorithm will be used to shard the model, based on the number of NUMA nodes (e.g. TP = 2 for a two NUMA node system). With `TP feature on CPU <https://github.com/vllm-project/vllm/pull/6125>`_ merged, Tensor Parallel is supported for serving and offline inferencing. In general each NUMA node is treated as one GPU card. Below is the example script to enable Tensor Parallel = 2 for serving:
160 +
161 + .. code-block:: console
162 +
163 + $ VLLM_CPU_KVCACHE_SPACE=40 VLLM_CPU_OMP_THREADS_BIND="0-31|32-63" vllm serve meta-llama/Llama-2-7b-chat-hf -tp=2 --distributed-executor-backend mp
164
165
166 + * Using Data Parallel for maximum throughput: to launch an LLM serving endpoint on each NUMA node along with one additional load balancer to dispatch the requests to those endpoints. Common solutions like `Nginx <../serving/deploying_with_nginx.html>`_ or HAProxy are recommended. Anyscale Ray project provides the feature on LLM `serving <https://docs.ray.io/en/latest/serve/index.html>`_. Here is the example to setup a scalable LLM serving with `Ray Serve <https://github.com/intel/llm-on-ray/blob/main/docs/setup.md>`_.
\ No newline at end of file
docs/source/index.rst
+1
@@ -80,6 +80,7 @@ Documentation
80 serving/openai_compatible_server
81 serving/deploying_with_docker
82 serving/deploying_with_k8s
83 + serving/deploying_with_nginx
84 serving/distributed_serving
85 serving/metrics
86 serving/env_vars
docs/source/serving/deploying_with_nginx.rst new
+142
@@ -0,0 +1,142 @@
1 +.. _nginxloadbalancer:
2 +
3 +Deploying with Nginx Loadbalancer
4 +=================================
5 +
6 +This document shows how to launch multiple vLLM serving containers and use Nginx to act as a load balancer between the servers.
7 +
8 +Table of contents:
9 +
10 +#. :ref:`Build Nginx Container <nginxloadbalancer_nginx_build>`
11 +#. :ref:`Create Simple Nginx Config file <nginxloadbalancer_nginx_conf>`
12 +#. :ref:`Build vLLM Container <nginxloadbalancer_nginx_vllm_container>`
13 +#. :ref:`Create Docker Network <nginxloadbalancer_nginx_docker_network>`
14 +#. :ref:`Launch vLLM Containers <nginxloadbalancer_nginx_launch_container>`
15 +#. :ref:`Launch Nginx <nginxloadbalancer_nginx_launch_nginx>`
16 +#. :ref:`Verify That vLLM Servers Are Ready <nginxloadbalancer_nginx_verify_nginx>`
17 +
18 +.. _nginxloadbalancer_nginx_build:
19 +
20 +Build Nginx Container
21 +---------------------
22 +
23 +This guide assumes that you have just cloned the vLLM project and you're currently in the vllm root directory.
24 +
25 +.. code-block:: console
26 +
27 + export vllm_root=`pwd`
28 +
29 +Create a file named ``Dockerfile.nginx``:
30 +
31 +.. code-block:: console
32 +
33 + FROM nginx:latest
34 + RUN rm /etc/nginx/conf.d/default.conf
35 + EXPOSE 80
36 + CMD ["nginx", "-g", "daemon off;"]
37 +
38 +Build the container:
39 +
40 +.. code-block:: console
41 +
42 + docker build . -f Dockerfile.nginx --tag nginx-lb
43 +
44 +.. _nginxloadbalancer_nginx_conf:
45 +
46 +Create Simple Nginx Config file
47 +-------------------------------
48 +
49 +Create a file named ``nginx_conf/nginx.conf``. Note that you can add as many servers as you'd like. In the below example we'll start with two. To add more, add another ``server vllmN:8000 max_fails=3 fail_timeout=10000s;`` entry to ``upstream backend``.
50 +
51 +.. code-block:: console
52 +
53 + upstream backend {
54 + least_conn;
55 + server vllm0:8000 max_fails=3 fail_timeout=10000s;
56 + server vllm1:8000 max_fails=3 fail_timeout=10000s;
57 + }
58 + server {
59 + listen 80;
60 + location / {
61 + proxy_pass http://backend;
62 + proxy_set_header Host $host;
63 + proxy_set_header X-Real-IP $remote_addr;
64 + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
65 + proxy_set_header X-Forwarded-Proto $scheme;
66 + }
67 + }
68 +
69 +.. _nginxloadbalancer_nginx_vllm_container:
70 +
71 +Build vLLM Container
72 +--------------------
73 +
74 +.. code-block:: console
75 +
76 + cd $vllm_root
77 + docker build -f Dockerfile . --tag vllm
78 +
79 +
80 +If you are behind proxy, you can pass the proxy settings to the docker build command as shown below:
81 +
82 +.. code-block:: console
83 +
84 + cd $vllm_root
85 + docker build -f Dockerfile . --tag vllm --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy
86 +
87 +.. _nginxloadbalancer_nginx_docker_network:
88 +
89 +Create Docker Network
90 +---------------------
91 +
92 +.. code-block:: console
93 +
94 + docker network create vllm_nginx
95 +
96 +
97 +.. _nginxloadbalancer_nginx_launch_container:
98 +
99 +Launch vLLM Containers
100 +----------------------
101 +
102 +Notes:
103 +
104 +* If you have your HuggingFace models cached somewhere else, update ``hf_cache_dir`` below.
105 +* If you don't have an existing HuggingFace cache you will want to start ``vllm0`` and wait for the model to complete downloading and the server to be ready. This will ensure that ``vllm1`` can leverage the model you just downloaded and it won't have to be downloaded again.
106 +* The below example assumes GPU backend used. If you are using CPU backend, remove ``--gpus all``, add ``VLLM_CPU_KVCACHE_SPACE`` and ``VLLM_CPU_OMP_THREADS_BIND`` environment variables to the docker run command.
107 +* Adjust the model name that you want to use in your vLLM servers if you don't want to use ``Llama-2-7b-chat-hf``.
108 +
109 +.. code-block:: console
110 +
111 + mkdir -p ~/.cache/huggingface/hub/
112 + hf_cache_dir=~/.cache/huggingface/
113 + docker run -itd --ipc host --privileged --network vllm_nginx --gpus all --shm-size=10.24gb -v $hf_cache_dir:/root/.cache/huggingface/ -p 8081:8000 --name vllm0 vllm --model meta-llama/Llama-2-7b-chat-hf
114 + docker run -itd --ipc host --privileged --network vllm_nginx --gpus all --shm-size=10.24gb -v $hf_cache_dir:/root/.cache/huggingface/ -p 8082:8000 --name vllm1 vllm --model meta-llama/Llama-2-7b-chat-hf
115 +
116 +.. note::
117 + If you are behind proxy, you can pass the proxy settings to the docker run command via ``-e http_proxy=$http_proxy -e https_proxy=$https_proxy``.
118 +
119 +.. _nginxloadbalancer_nginx_launch_nginx:
120 +
121 +Launch Nginx
122 +------------
123 +
124 +.. code-block:: console
125 +
126 + docker run -itd -p 8000:80 --network vllm_nginx -v ./nginx_conf/:/etc/nginx/conf.d/ --name nginx-lb nginx-lb:latest
127 +
128 +.. _nginxloadbalancer_nginx_verify_nginx:
129 +
130 +Verify That vLLM Servers Are Ready
131 +----------------------------------
132 +
133 +.. code-block:: console
134 +
135 + docker logs vllm0 | grep Uvicorn
136 + docker logs vllm1 | grep Uvicorn
137 +
138 +Both outputs should look like this:
139 +
140 +.. code-block:: console
141 +
142 + INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)