A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
Add documentation on how to do incremental builds (#2796)
Philipp Moritz committed
Feb 7, 2024 at 14:42 UTC
931746bc6d7c1c0ab40b2c4f58b51b855f0b2c94
2 files changed
+15
docs/source/getting_started/installation.rst
+10
@@ -67,3 +67,13 @@ You can also build and install vLLM from source:
67
68
$ # Use `--ipc=host` to make sure the shared memory is large enough.
69
$ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
70
+
71
+.. note::
72
+ If you are developing the C++ backend of vLLM, consider building vLLM with
73
+
74
+ .. code-block:: console
75
+
76
+ $ python setup.py develop
77
+
78
+ since it will give you incremental builds. The downside is that this method
79
+ is `deprecated by setuptools <https://github.com/pypa/setuptools/issues/917>`_.
setup.py
+5
@@ -15,6 +15,11 @@ from torch.utils.cpp_extension import BuildExtension, CUDAExtension, CUDA_HOME,
15
16
ROOT_DIR = os.path.dirname(__file__)
17
18
+# If you are developing the C++ backend of vLLM, consider building vLLM with
19
+# `python setup.py develop` since it will give you incremental builds.
20
+# The downside is that this method is deprecated, see
21
+# https://github.com/pypa/setuptools/issues/917
22
+
23
MAIN_CUDA_VERSION = "12.1"
24
25
# Supported NVIDIA GPU architectures.