@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / c85b80c2b6
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 06:45 UTC

[Docker] Add cuda arch list as build option (#1950)

Simon Mo committed Dec 8, 2023 at 09:53 UTC c85b80c2b64d0f420aaca59679e5f38f71a8a53e
2 files changed +13 -1
Dockerfile
+5 -1
@@ -30,11 +30,15 @@ COPY requirements.txt requirements.txt
30 COPY pyproject.toml pyproject.toml
31 COPY vllm/__init__.py vllm/__init__.py
32
33 +ARG torch_cuda_arch_list='7.0 7.5 8.0 8.6 8.9 9.0+PTX'
34 +ENV TORCH_CUDA_ARCH_LIST=${torch_cuda_arch_list}
35 # max jobs used by Ninja to build extensions
34 -ENV MAX_JOBS=$max_jobs
36 +ARG max_jobs=2
37 +ENV MAX_JOBS=${max_jobs}
38 # number of threads used by nvcc
39 ARG nvcc_threads=8
40 ENV NVCC_THREADS=$nvcc_threads
41 +
42 RUN python3 setup.py build_ext --inplace
43
44 # image to run unit testing suite
docs/source/serving/deploying_with_docker.rst
+8
@@ -31,6 +31,14 @@ You can build and run vLLM from source via the provided dockerfile. To build vLL
31
32 $ DOCKER_BUILDKIT=1 docker build . --target vllm-openai --tag vllm/vllm-openai # optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2
33
34 +
35 +.. note::
36 +
37 + By default vLLM will build for all GPU types for widest distribution. If you are just building for the
38 + current GPU type the machine is running on, you can add the argument ``--build-arg torch_cuda_arch_list=""``
39 + for vLLM to find the current GPU type and build for that.
40 +
41 +
42 To run vLLM:
43
44 .. code-block:: console