@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / e4c34c23de
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 05:45 UTC

[CI/Build] improve python-only dev setup (#9621)

Signed-off-by: Daniele Trifirò <dtrifiro@redhat.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: youkaichao <youkaichao@gmail.com>

Daniele committed Dec 4, 2024 at 22:48 UTC e4c34c23de2a90ab837772ac182638ac3bc1636d
4 files changed +102 -121
docs/source/getting_started/installation.rst
+12 -29
@@ -21,7 +21,7 @@ You can install vLLM using pip:
21 .. code-block:: console
22
23 $ # (Recommended) Create a new conda environment.
24 - $ conda create -n myenv python=3.10 -y
24 + $ conda create -n myenv python=3.12 -y
25 $ conda activate myenv
26
27 $ # Install vLLM with CUDA 12.1.
@@ -89,45 +89,24 @@ Build from source
89 Python-only build (without compilation)
90 ---------------------------------------
91
92 -If you only need to change Python code, you can simply build vLLM without compilation.
93 -
94 -The first step is to install the latest vLLM wheel:
95 -
96 -.. code-block:: console
97 -
98 - pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
99 -
100 -You can find more information about vLLM's wheels `above <#install-the-latest-code>`_.
101 -
102 -After verifying that the installation is successful, you can use `the following script <https://github.com/vllm-project/vllm/blob/main/python_only_dev.py>`_:
92 +If you only need to change Python code, you can build and install vLLM without compilation. Using `pip's ``--editable`` flag <https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs>`_, changes you make to the code will be reflected when you run vLLM:
93
94 .. code-block:: console
95
96 $ git clone https://github.com/vllm-project/vllm.git
97 $ cd vllm
108 - $ python python_only_dev.py
98 + $ VLLM_USE_PRECOMPILED=1 pip install --editable .
99
110 -The script will:
100 +This will download the latest nightly wheel and use the compiled libraries from there in the install.
101
112 -* Find the installed vLLM package in the current environment.
113 -* Copy built files to the current directory.
114 -* Rename the installed vLLM package.
115 -* Symbolically link the current directory to the installed vLLM package.
116 -
117 -Now, you can edit the Python code in the current directory, and the changes will be reflected when you run vLLM.
118 -
119 -Once you have finished editing or want to install another vLLM wheel, you should exit the development environment using `the same script <https://github.com/vllm-project/vllm/blob/main/python_only_dev.py>`_ with the ``--quit-dev`` (or ``-q`` for short) flag:
102 +The ``VLLM_PRECOMPILED_WHEEL_LOCATION`` environment variable can be used instead of ``VLLM_USE_PRECOMPILED`` to specify a custom path or URL to the wheel file. For example, to use the `0.6.1.post1 PyPi wheel <https://pypi.org/project/vllm/#files>`_:
103
104 .. code-block:: console
105
123 - $ python python_only_dev.py --quit-dev
124 -
125 -The ``--quit-dev`` flag will:
126 -
127 -* Remove the symbolic link from the current directory to the vLLM package.
128 -* Restore the original vLLM package from the backup.
106 + $ export VLLM_PRECOMPILED_WHEEL_LOCATION=https://files.pythonhosted.org/packages/4a/4c/ee65ba33467a4c0de350ce29fbae39b9d0e7fcd887cc756fa993654d1228/vllm-0.6.3.post1-cp38-abi3-manylinux1_x86_64.whl
107 + $ pip install --editable .
108
130 -If you update the vLLM wheel and rebuild from the source to make further edits, you will need to repeat the `Python-only build <#python-only-build>`_ steps again.
109 +You can find more information about vLLM's wheels `above <#install-the-latest-code>`_.
110
111 .. note::
112
@@ -148,9 +127,13 @@ If you want to modify C++ or CUDA code, you'll need to build vLLM from source. T
127 .. tip::
128
129 Building from source requires a lot of compilation. If you are building from source repeatedly, it's more efficient to cache the compilation results.
130 +
131 For example, you can install `ccache <https://github.com/ccache/ccache>`_ using ``conda install ccache`` or ``apt install ccache`` .
132 As long as ``which ccache`` command can find the ``ccache`` binary, it will be used automatically by the build system. After the first build, subsequent builds will be much faster.
133
134 + `sccache <https://github.com/mozilla/sccache>`_ works similarly to ``ccache``, but has the capability to utilize caching in remote storage environments.
135 + The following environment variables can be set to configure the vLLM ``sccache`` remote: ``SCCACHE_BUCKET=vllm-build-sccache SCCACHE_REGION=us-west-2 SCCACHE_S3_NO_CREDENTIALS=1``. We also recommend setting ``SCCACHE_IDLE_TIMEOUT=0``.
136 +
137
138 Use an existing PyTorch installation
139 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
python_only_dev.py
+9 -87
@@ -1,92 +1,14 @@
1 -# enable python only development
2 -# copy compiled files to the current directory directly
1 +msg = """Old style python only build (without compilation) is deprecated, please check https://docs.vllm.ai/en/latest/getting_started/installation.html#python-only-build-without-compilation for the new way to do python only build (without compilation).
2
4 -import argparse
5 -import os
6 -import shutil
7 -import subprocess
8 -import sys
9 -import warnings
3 +TL;DR:
4
11 -parser = argparse.ArgumentParser(
12 - description="Development mode for python-only code")
13 -parser.add_argument('-q',
14 - '--quit-dev',
15 - action='store_true',
16 - help='Set the flag to quit development mode')
17 -args = parser.parse_args()
5 +VLLM_USE_PRECOMPILED=1 pip install -e .
6
19 -# cannot directly `import vllm` , because it will try to
20 -# import from the current directory
21 -output = subprocess.run([sys.executable, "-m", "pip", "show", "vllm"],
22 - capture_output=True)
7 +or
8
24 -assert output.returncode == 0, "vllm is not installed"
9 +export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch
10 +export VLLM_PRECOMPILED_WHEEL_LOCATION=https://vllm-wheels.s3.us-west-2.amazonaws.com/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
11 +pip install -e .
12 +""" # noqa
13
26 -text = output.stdout.decode("utf-8")
27 -
28 -package_path = None
29 -for line in text.split("\n"):
30 - if line.startswith("Location: "):
31 - package_path = line.split(": ")[1]
32 - break
33 -
34 -assert package_path is not None, "could not find package path"
35 -
36 -cwd = os.getcwd()
37 -
38 -assert cwd != package_path, "should not import from the current directory"
39 -
40 -files_to_copy = [
41 - "vllm/_C.abi3.so",
42 - "vllm/_moe_C.abi3.so",
43 - "vllm/vllm_flash_attn/vllm_flash_attn_c.abi3.so",
44 - "vllm/vllm_flash_attn/flash_attn_interface.py",
45 - "vllm/vllm_flash_attn/__init__.py",
46 - # "vllm/_version.py", # not available in nightly wheels yet
47 -]
48 -
49 -# Try to create _version.py to avoid version related warning
50 -# Refer to https://github.com/vllm-project/vllm/pull/8771
51 -try:
52 - from setuptools_scm import get_version
53 - get_version(write_to="vllm/_version.py")
54 -except ImportError:
55 - warnings.warn(
56 - "To avoid warnings related to vllm._version, "
57 - "you should install setuptools-scm by `pip install setuptools-scm`",
58 - stacklevel=2)
59 -
60 -if not args.quit_dev:
61 - for file in files_to_copy:
62 - src = os.path.join(package_path, file)
63 - dst = file
64 - print(f"Copying {src} to {dst}")
65 - shutil.copyfile(src, dst)
66 -
67 - pre_built_vllm_path = os.path.join(package_path, "vllm")
68 - tmp_path = os.path.join(package_path, "vllm_pre_built")
69 - current_vllm_path = os.path.join(cwd, "vllm")
70 -
71 - print(f"Renaming {pre_built_vllm_path} to {tmp_path} for backup")
72 - shutil.copytree(pre_built_vllm_path, tmp_path)
73 - shutil.rmtree(pre_built_vllm_path)
74 -
75 - print(f"Linking {current_vllm_path} to {pre_built_vllm_path}")
76 - os.symlink(current_vllm_path, pre_built_vllm_path)
77 -else:
78 - vllm_symlink_path = os.path.join(package_path, "vllm")
79 - vllm_backup_path = os.path.join(package_path, "vllm_pre_built")
80 - current_vllm_path = os.path.join(cwd, "vllm")
81 -
82 - print(f"Unlinking {current_vllm_path} to {vllm_symlink_path}")
83 - assert os.path.islink(
84 - vllm_symlink_path
85 - ), f"not in dev mode: {vllm_symlink_path} is not a symbolic link"
86 - assert current_vllm_path == os.readlink(
87 - vllm_symlink_path
88 - ), "current directory is not the source code of package"
89 - os.unlink(vllm_symlink_path)
90 -
91 - print(f"Recovering backup from {vllm_backup_path} to {vllm_symlink_path}")
92 - os.rename(vllm_backup_path, vllm_symlink_path)
14 +print(msg)
setup.py
+79 -4
@@ -249,6 +249,74 @@ class cmake_build_ext(build_ext):
249 self.copy_file(file, dst_file)
250
251
252 +class repackage_wheel(build_ext):
253 + """Extracts libraries and other files from an existing wheel."""
254 + default_wheel = "https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl"
255 +
256 + def run(self) -> None:
257 + wheel_location = os.getenv("VLLM_PRECOMPILED_WHEEL_LOCATION",
258 + self.default_wheel)
259 +
260 + assert _is_cuda(
261 + ), "VLLM_USE_PRECOMPILED is only supported for CUDA builds"
262 +
263 + import zipfile
264 +
265 + if os.path.isfile(wheel_location):
266 + wheel_path = wheel_location
267 + print(f"Using existing wheel={wheel_path}")
268 + else:
269 + # Download the wheel from a given URL, assume
270 + # the filename is the last part of the URL
271 + wheel_filename = wheel_location.split("/")[-1]
272 +
273 + import tempfile
274 +
275 + # create a temporary directory to store the wheel
276 + temp_dir = tempfile.mkdtemp(prefix="vllm-wheels")
277 + wheel_path = os.path.join(temp_dir, wheel_filename)
278 +
279 + print(f"Downloading wheel from {wheel_location} to {wheel_path}")
280 +
281 + from urllib.request import urlretrieve
282 +
283 + try:
284 + urlretrieve(wheel_location, filename=wheel_path)
285 + except Exception as e:
286 + from setuptools.errors import SetupError
287 +
288 + raise SetupError(
289 + f"Failed to get vLLM wheel from {wheel_location}") from e
290 +
291 + with zipfile.ZipFile(wheel_path) as wheel:
292 + files_to_copy = [
293 + "vllm/_C.abi3.so",
294 + "vllm/_moe_C.abi3.so",
295 + "vllm/vllm_flash_attn/vllm_flash_attn_c.abi3.so",
296 + "vllm/vllm_flash_attn/flash_attn_interface.py",
297 + "vllm/vllm_flash_attn/__init__.py",
298 + # "vllm/_version.py", # not available in nightly wheels yet
299 + ]
300 + file_members = filter(lambda x: x.filename in files_to_copy,
301 + wheel.filelist)
302 +
303 + for file in file_members:
304 + print(f"Extracting and including {file.filename} "
305 + "from existing wheel")
306 + package_name = os.path.dirname(file.filename).replace("/", ".")
307 + file_name = os.path.basename(file.filename)
308 +
309 + if package_name not in package_data:
310 + package_data[package_name] = []
311 +
312 + wheel.extract(file)
313 + if file_name.endswith(".py"):
314 + # python files shouldn't be added to package_data
315 + continue
316 +
317 + package_data[package_name].append(file_name)
318 +
319 +
320 def _is_hpu() -> bool:
321 is_hpu_available = True
322 try:
@@ -403,6 +471,8 @@ def get_vllm_version() -> str:
471 # skip this for source tarball, required for pypi
472 if "sdist" not in sys.argv:
473 version += f"{sep}cu{cuda_version_str}"
474 + if envs.VLLM_USE_PRECOMPILED:
475 + version += ".precompiled"
476 elif _is_hip():
477 # Get the HIP version
478 hipcc_version = get_hipcc_rocm_version()
@@ -514,13 +584,18 @@ if _build_custom_ops():
584 package_data = {
585 "vllm": ["py.typed", "model_executor/layers/fused_moe/configs/*.json"]
586 }
517 -if envs.VLLM_USE_PRECOMPILED:
518 - ext_modules = []
519 - package_data["vllm"].append("*.so")
587
588 if _no_device():
589 ext_modules = []
590
591 +if not ext_modules:
592 + cmdclass = {}
593 +else:
594 + cmdclass = {
595 + "build_ext":
596 + repackage_wheel if envs.VLLM_USE_PRECOMPILED else cmake_build_ext
597 + }
598 +
599 setup(
600 name="vllm",
601 version=get_vllm_version(),
@@ -557,7 +632,7 @@ setup(
632 "audio": ["librosa", "soundfile"], # Required for audio processing
633 "video": ["decord"] # Required for video processing
634 },
560 - cmdclass={"build_ext": cmake_build_ext} if len(ext_modules) > 0 else {},
635 + cmdclass=cmdclass,
636 package_data=package_data,
637 entry_points={
638 "console_scripts": [
vllm/envs.py
+2 -1
@@ -113,7 +113,8 @@ environment_variables: Dict[str, Callable[[], Any]] = {
113
114 # If set, vllm will use precompiled binaries (*.so)
115 "VLLM_USE_PRECOMPILED":
116 - lambda: bool(os.environ.get("VLLM_USE_PRECOMPILED")),
116 + lambda: bool(os.environ.get("VLLM_USE_PRECOMPILED")) or bool(
117 + os.environ.get("VLLM_PRECOMPILED_WHEEL_LOCATION")),
118
119 # CMake build type
120 # If not set, defaults to "Debug" or "RelWithDebInfo"