A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 05:45 UTC
[CI/Build] improve python-only dev setup (#9621)
Signed-off-by: Daniele Trifirò <dtrifiro@redhat.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: youkaichao <youkaichao@gmail.com>
Daniele committed
Dec 4, 2024 at 22:48 UTC
e4c34c23de2a90ab837772ac182638ac3bc1636d
4 files changed
+102
-121
docs/source/getting_started/installation.rst
+12
-29
@@ -21,7 +21,7 @@ You can install vLLM using pip:
21
.. code-block:: console
22
23
$ # (Recommended) Create a new conda environment.
24
- $ conda create -n myenv python=3.10 -y
24
+ $ conda create -n myenv python=3.12 -y
25
$ conda activate myenv
26
27
$ # Install vLLM with CUDA 12.1.
@@ -89,45 +89,24 @@ Build from source
89
Python-only build (without compilation)
90
---------------------------------------
91
92
-If you only need to change Python code, you can simply build vLLM without compilation.
93
-
94
-The first step is to install the latest vLLM wheel:
95
-
96
-.. code-block:: console
97
-
98
- pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
99
-
100
-You can find more information about vLLM's wheels `above <#install-the-latest-code>`_.
101
-
102
-After verifying that the installation is successful, you can use `the following script <https://github.com/vllm-project/vllm/blob/main/python_only_dev.py>`_:
92
+If you only need to change Python code, you can build and install vLLM without compilation. Using `pip's ``--editable`` flag <https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs>`_, changes you make to the code will be reflected when you run vLLM:
93
94
.. code-block:: console
95
96
$ git clone https://github.com/vllm-project/vllm.git
97
$ cd vllm
108
- $ python python_only_dev.py
98
+ $ VLLM_USE_PRECOMPILED=1 pip install --editable .
99
110
-The script will:
100
+This will download the latest nightly wheel and use the compiled libraries from there in the install.
101
112
-* Find the installed vLLM package in the current environment.
113
-* Copy built files to the current directory.
114
-* Rename the installed vLLM package.
115
-* Symbolically link the current directory to the installed vLLM package.
116
-
117
-Now, you can edit the Python code in the current directory, and the changes will be reflected when you run vLLM.
118
-
119
-Once you have finished editing or want to install another vLLM wheel, you should exit the development environment using `the same script <https://github.com/vllm-project/vllm/blob/main/python_only_dev.py>`_ with the ``--quit-dev`` (or ``-q`` for short) flag:
102
+The ``VLLM_PRECOMPILED_WHEEL_LOCATION`` environment variable can be used instead of ``VLLM_USE_PRECOMPILED`` to specify a custom path or URL to the wheel file. For example, to use the `0.6.1.post1 PyPi wheel <https://pypi.org/project/vllm/#files>`_:
103
104
.. code-block:: console
105
123
- $ python python_only_dev.py --quit-dev
124
-
125
-The ``--quit-dev`` flag will:
126
-
127
-* Remove the symbolic link from the current directory to the vLLM package.
128
-* Restore the original vLLM package from the backup.
106
+ $ export VLLM_PRECOMPILED_WHEEL_LOCATION=https://files.pythonhosted.org/packages/4a/4c/ee65ba33467a4c0de350ce29fbae39b9d0e7fcd887cc756fa993654d1228/vllm-0.6.3.post1-cp38-abi3-manylinux1_x86_64.whl
107
+ $ pip install --editable .
108
130
-If you update the vLLM wheel and rebuild from the source to make further edits, you will need to repeat the `Python-only build <#python-only-build>`_ steps again.
109
+You can find more information about vLLM's wheels `above <#install-the-latest-code>`_.
110
111
.. note::
112
@@ -148,9 +127,13 @@ If you want to modify C++ or CUDA code, you'll need to build vLLM from source. T
127
.. tip::
128
129
Building from source requires a lot of compilation. If you are building from source repeatedly, it's more efficient to cache the compilation results.
130
+
131
For example, you can install `ccache <https://github.com/ccache/ccache>`_ using ``conda install ccache`` or ``apt install ccache`` .
132
As long as ``which ccache`` command can find the ``ccache`` binary, it will be used automatically by the build system. After the first build, subsequent builds will be much faster.
133
134
+ `sccache <https://github.com/mozilla/sccache>`_ works similarly to ``ccache``, but has the capability to utilize caching in remote storage environments.
135
+ The following environment variables can be set to configure the vLLM ``sccache`` remote: ``SCCACHE_BUCKET=vllm-build-sccache SCCACHE_REGION=us-west-2 SCCACHE_S3_NO_CREDENTIALS=1``. We also recommend setting ``SCCACHE_IDLE_TIMEOUT=0``.
136
+
137
138
Use an existing PyTorch installation
139
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
python_only_dev.py
+9
-87
@@ -1,92 +1,14 @@
1
-# enable python only development
2
-# copy compiled files to the current directory directly
1
+msg = """Old style python only build (without compilation) is deprecated, please check https://docs.vllm.ai/en/latest/getting_started/installation.html#python-only-build-without-compilation for the new way to do python only build (without compilation).
2
4
-import argparse
5
-import os
6
-import shutil
7
-import subprocess
8
-import sys
9
-import warnings
3
+TL;DR:
4
11
-parser = argparse.ArgumentParser(
12
- description="Development mode for python-only code")
13
-parser.add_argument('-q',
14
- '--quit-dev',
15
- action='store_true',
16
- help='Set the flag to quit development mode')
17
-args = parser.parse_args()
5
+VLLM_USE_PRECOMPILED=1 pip install -e .
6
19
-# cannot directly `import vllm` , because it will try to
20
-# import from the current directory
21
-output = subprocess.run([sys.executable, "-m", "pip", "show", "vllm"],
22
- capture_output=True)
7
+or
8
24
-assert output.returncode == 0, "vllm is not installed"
9
+export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch
10
+export VLLM_PRECOMPILED_WHEEL_LOCATION=https://vllm-wheels.s3.us-west-2.amazonaws.com/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
11
+pip install -e .
12
+""" # noqa
13
26
-text = output.stdout.decode("utf-8")
27
-
28
-package_path = None
29
-for line in text.split("\n"):
30
- if line.startswith("Location: "):
31
- package_path = line.split(": ")[1]
32
- break
33
-
34
-assert package_path is not None, "could not find package path"
35
-
36
-cwd = os.getcwd()
37
-
38
-assert cwd != package_path, "should not import from the current directory"
39
-
40
-files_to_copy = [
41
- "vllm/_C.abi3.so",
42
- "vllm/_moe_C.abi3.so",
43
- "vllm/vllm_flash_attn/vllm_flash_attn_c.abi3.so",
44
- "vllm/vllm_flash_attn/flash_attn_interface.py",
45
- "vllm/vllm_flash_attn/__init__.py",
46
- # "vllm/_version.py", # not available in nightly wheels yet
47
-]
48
-
49
-# Try to create _version.py to avoid version related warning
50
-# Refer to https://github.com/vllm-project/vllm/pull/8771
51
-try:
52
- from setuptools_scm import get_version
53
- get_version(write_to="vllm/_version.py")
54
-except ImportError:
55
- warnings.warn(
56
- "To avoid warnings related to vllm._version, "
57
- "you should install setuptools-scm by `pip install setuptools-scm`",
58
- stacklevel=2)
59
-
60
-if not args.quit_dev:
61
- for file in files_to_copy:
62
- src = os.path.join(package_path, file)
63
- dst = file
64
- print(f"Copying {src} to {dst}")
65
- shutil.copyfile(src, dst)
66
-
67
- pre_built_vllm_path = os.path.join(package_path, "vllm")
68
- tmp_path = os.path.join(package_path, "vllm_pre_built")
69
- current_vllm_path = os.path.join(cwd, "vllm")
70
-
71
- print(f"Renaming {pre_built_vllm_path} to {tmp_path} for backup")
72
- shutil.copytree(pre_built_vllm_path, tmp_path)
73
- shutil.rmtree(pre_built_vllm_path)
74
-
75
- print(f"Linking {current_vllm_path} to {pre_built_vllm_path}")
76
- os.symlink(current_vllm_path, pre_built_vllm_path)
77
-else:
78
- vllm_symlink_path = os.path.join(package_path, "vllm")
79
- vllm_backup_path = os.path.join(package_path, "vllm_pre_built")
80
- current_vllm_path = os.path.join(cwd, "vllm")
81
-
82
- print(f"Unlinking {current_vllm_path} to {vllm_symlink_path}")
83
- assert os.path.islink(
84
- vllm_symlink_path
85
- ), f"not in dev mode: {vllm_symlink_path} is not a symbolic link"
86
- assert current_vllm_path == os.readlink(
87
- vllm_symlink_path
88
- ), "current directory is not the source code of package"
89
- os.unlink(vllm_symlink_path)
90
-
91
- print(f"Recovering backup from {vllm_backup_path} to {vllm_symlink_path}")
92
- os.rename(vllm_backup_path, vllm_symlink_path)
14
+print(msg)
setup.py
+79
-4
@@ -249,6 +249,74 @@ class cmake_build_ext(build_ext):
249
self.copy_file(file, dst_file)
250
251
252
+class repackage_wheel(build_ext):
253
+ """Extracts libraries and other files from an existing wheel."""
254
+ default_wheel = "https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl"
255
+
256
+ def run(self) -> None:
257
+ wheel_location = os.getenv("VLLM_PRECOMPILED_WHEEL_LOCATION",
258
+ self.default_wheel)
259
+
260
+ assert _is_cuda(
261
+ ), "VLLM_USE_PRECOMPILED is only supported for CUDA builds"
262
+
263
+ import zipfile
264
+
265
+ if os.path.isfile(wheel_location):
266
+ wheel_path = wheel_location
267
+ print(f"Using existing wheel={wheel_path}")
268
+ else:
269
+ # Download the wheel from a given URL, assume
270
+ # the filename is the last part of the URL
271
+ wheel_filename = wheel_location.split("/")[-1]
272
+
273
+ import tempfile
274
+
275
+ # create a temporary directory to store the wheel
276
+ temp_dir = tempfile.mkdtemp(prefix="vllm-wheels")
277
+ wheel_path = os.path.join(temp_dir, wheel_filename)
278
+
279
+ print(f"Downloading wheel from {wheel_location} to {wheel_path}")
280
+
281
+ from urllib.request import urlretrieve
282
+
283
+ try:
284
+ urlretrieve(wheel_location, filename=wheel_path)
285
+ except Exception as e:
286
+ from setuptools.errors import SetupError
287
+
288
+ raise SetupError(
289
+ f"Failed to get vLLM wheel from {wheel_location}") from e
290
+
291
+ with zipfile.ZipFile(wheel_path) as wheel:
292
+ files_to_copy = [
293
+ "vllm/_C.abi3.so",
294
+ "vllm/_moe_C.abi3.so",
295
+ "vllm/vllm_flash_attn/vllm_flash_attn_c.abi3.so",
296
+ "vllm/vllm_flash_attn/flash_attn_interface.py",
297
+ "vllm/vllm_flash_attn/__init__.py",
298
+ # "vllm/_version.py", # not available in nightly wheels yet
299
+ ]
300
+ file_members = filter(lambda x: x.filename in files_to_copy,
301
+ wheel.filelist)
302
+
303
+ for file in file_members:
304
+ print(f"Extracting and including {file.filename} "
305
+ "from existing wheel")
306
+ package_name = os.path.dirname(file.filename).replace("/", ".")
307
+ file_name = os.path.basename(file.filename)
308
+
309
+ if package_name not in package_data:
310
+ package_data[package_name] = []
311
+
312
+ wheel.extract(file)
313
+ if file_name.endswith(".py"):
314
+ # python files shouldn't be added to package_data
315
+ continue
316
+
317
+ package_data[package_name].append(file_name)
318
+
319
+
320
def _is_hpu() -> bool:
321
is_hpu_available = True
322
try:
@@ -403,6 +471,8 @@ def get_vllm_version() -> str:
471
# skip this for source tarball, required for pypi
472
if "sdist" not in sys.argv:
473
version += f"{sep}cu{cuda_version_str}"
474
+ if envs.VLLM_USE_PRECOMPILED:
475
+ version += ".precompiled"
476
elif _is_hip():
477
# Get the HIP version
478
hipcc_version = get_hipcc_rocm_version()
@@ -514,13 +584,18 @@ if _build_custom_ops():
584
package_data = {
585
"vllm": ["py.typed", "model_executor/layers/fused_moe/configs/*.json"]
586
}
517
-if envs.VLLM_USE_PRECOMPILED:
518
- ext_modules = []
519
- package_data["vllm"].append("*.so")
587
588
if _no_device():
589
ext_modules = []
590
591
+if not ext_modules:
592
+ cmdclass = {}
593
+else:
594
+ cmdclass = {
595
+ "build_ext":
596
+ repackage_wheel if envs.VLLM_USE_PRECOMPILED else cmake_build_ext
597
+ }
598
+
599
setup(
600
name="vllm",
601
version=get_vllm_version(),
@@ -557,7 +632,7 @@ setup(
632
"audio": ["librosa", "soundfile"], # Required for audio processing
633
"video": ["decord"] # Required for video processing
634
},
560
- cmdclass={"build_ext": cmake_build_ext} if len(ext_modules) > 0 else {},
635
+ cmdclass=cmdclass,
636
package_data=package_data,
637
entry_points={
638
"console_scripts": [
vllm/envs.py
+2
-1
@@ -113,7 +113,8 @@ environment_variables: Dict[str, Callable[[], Any]] = {
113
114
# If set, vllm will use precompiled binaries (*.so)
115
"VLLM_USE_PRECOMPILED":
116
- lambda: bool(os.environ.get("VLLM_USE_PRECOMPILED")),
116
+ lambda: bool(os.environ.get("VLLM_USE_PRECOMPILED")) or bool(
117
+ os.environ.get("VLLM_PRECOMPILED_WHEEL_LOCATION")),
118
119
# CMake build type
120
# If not set, defaults to "Debug" or "RelWithDebInfo"