A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 15:45 UTC
[build] enable existing pytorch (for GH200, aarch64, nightly) (#8713)
youkaichao committed
Sep 22, 2024 at 12:47 UTC
d4a2ac830291305f202a85e157bff3a07b58e616
3 files changed
+42
-1
docs/source/getting_started/installation.rst
+23
@@ -72,6 +72,29 @@ You can also build and install vLLM from source:
72
$ cd vllm
73
$ pip install -e . # This may take 5-10 minutes.
74
75
+.. note::
76
+
77
+ This will uninstall existing PyTorch, and install the version required by vLLM. If you want to use an existing PyTorch installation, there need to be some changes:
78
+
79
+ .. code-block:: console
80
+
81
+ $ git clone https://github.com/vllm-project/vllm.git
82
+ $ cd vllm
83
+ $ python use_existing_torch.py
84
+ $ pip install -r requirements-build.txt
85
+ $ pip install -e . --no-build-isolation
86
+
87
+ The differences are:
88
+
89
+ - ``python use_existing_torch.py``: This script will remove all the PyTorch versions in the requirements files, so that the existing PyTorch installation will be used.
90
+ - ``pip install -r requirements-build.txt``: You need to manually install the requirements for building vLLM.
91
+ - ``pip install -e . --no-build-isolation``: You need to disable build isolation, so that the build system can use the existing PyTorch installation.
92
+
93
+ This is especially useful when the PyTorch dependency cannot be easily installed via pip, e.g.:
94
+
95
+ - build vLLM with PyTorch nightly or a custom PyTorch build.
96
+ - build vLLM with aarch64 and cuda (GH200), where the PyTorch wheels are not available on PyPI. Currently, only PyTorch nightly has wheels for aarch64 with CUDA. You can run ``pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu124`` to install PyTorch nightly, and then build vLLM on top of it.
97
+
98
.. note::
99
100
vLLM can fully run only on Linux, but you can still build it on other systems (for example, macOS). This build is only for development purposes, allowing for imports and a more convenient dev environment. The binaries will not be compiled and not work on non-Linux systems. You can create such a build with the following commands:
requirements-common.txt
+1
-1
@@ -18,7 +18,7 @@ prometheus_client >= 0.18.0
18
prometheus-fastapi-instrumentator >= 7.0.0
19
tiktoken >= 0.6.0 # Required for DBRX tokenizer
20
lm-format-enforcer == 0.10.6
21
-outlines >= 0.0.43, < 0.1 # Requires torch >= 2.1.0
21
+outlines >= 0.0.43, < 0.1
22
typing_extensions >= 4.10
23
filelock >= 3.10.4 # filelock starts to support `mode` argument from 3.10.4
24
partial-json-parser # used for parsing partial JSON outputs
use_existing_torch.py
new
+18
@@ -0,0 +1,18 @@
1
+import glob
2
+
3
+requires_files = glob.glob('requirements*.txt')
4
+requires_files += ["pyproject.toml"]
5
+for file in requires_files:
6
+ print(f">>> cleaning {file}")
7
+ with open(file, 'r') as f:
8
+ lines = f.readlines()
9
+ if "torch" in "".join(lines).lower():
10
+ print("removed:")
11
+ with open(file, 'w') as f:
12
+ for line in lines:
13
+ if 'torch' not in line.lower():
14
+ f.write(line)
15
+ else:
16
+ print(line.strip())
17
+ print(f"<<< done cleaning {file}")
18
+ print()