@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / cab69a15e4
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 17:00 UTC

[doc] recommend pip instead of conda (#8446)

youkaichao committed Sep 12, 2024 at 23:52 UTC cab69a15e49aa592db7042f0dc675bbe9b684f83
1 file changed +9 -5
docs/source/getting_started/installation.rst
+9 -5
@@ -26,6 +26,10 @@ You can install vLLM using pip:
26 $ # Install vLLM with CUDA 12.1.
27 $ pip install vllm
28
29 +.. note::
30 +
31 + Although we recommend using ``conda`` to create and manage Python environments, it is highly recommended to use ``pip`` to install vLLM. This is because ``pip`` can install ``torch`` with separate library packages like ``NCCL``, while ``conda`` installs ``torch`` with statically linked ``NCCL``. This can cause issues when vLLM tries to use ``NCCL``. See `this issue <https://github.com/vllm-project/vllm/issues/8420>`_ for more details.
32 +
33 .. note::
34
35 As of now, vLLM's binaries are compiled with CUDA 12.1 and public PyTorch release versions by default.
@@ -34,7 +38,7 @@ You can install vLLM using pip:
38 .. code-block:: console
39
40 $ # Install vLLM with CUDA 11.8.
37 - $ export VLLM_VERSION=0.4.0
41 + $ export VLLM_VERSION=0.6.1.post1
42 $ export PYTHON_VERSION=310
43 $ pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118
44
@@ -48,7 +52,7 @@ You can install vLLM using pip:
52
53 .. code-block:: console
54
51 - $ export VLLM_VERSION=0.5.4 # vLLM's main branch version is currently set to latest released tag
55 + $ export VLLM_VERSION=0.6.1.post1 # vLLM's main branch version is currently set to latest released tag
56 $ pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-${VLLM_VERSION}-cp38-abi3-manylinux1_x86_64.whl
57 $ # You can also access a specific commit
58 $ # export VLLM_COMMIT=...
@@ -80,11 +84,11 @@ You can also build and install vLLM from source:
84
85 .. tip::
86
83 - Building from source requires quite a lot compilation. If you are building from source for multiple times, it is beneficial to cache the compilation results. For example, you can install `ccache <https://github.com/ccache/ccache>`_ via either `conda install ccache` or `apt install ccache` . As long as `which ccache` command can find the `ccache` binary, it will be used automatically by the build system. After the first build, the subsequent builds will be much faster.
87 + Building from source requires quite a lot compilation. If you are building from source for multiple times, it is beneficial to cache the compilation results. For example, you can install `ccache <https://github.com/ccache/ccache>`_ via either ``conda install ccache`` or ``apt install ccache`` . As long as ``which ccache`` command can find the ``ccache`` binary, it will be used automatically by the build system. After the first build, the subsequent builds will be much faster.
88
89 .. tip::
90 To avoid your system being overloaded, you can limit the number of compilation jobs
87 - to be run simultaneously, via the environment variable `MAX_JOBS`. For example:
91 + to be run simultaneously, via the environment variable ``MAX_JOBS``. For example:
92
93 .. code-block:: console
94
@@ -99,7 +103,7 @@ You can also build and install vLLM from source:
103 $ # Use `--ipc=host` to make sure the shared memory is large enough.
104 $ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
105
102 - If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from `the official website <https://developer.nvidia.com/cuda-toolkit-archive>`_. After installation, set the environment variable `CUDA_HOME` to the installation path of CUDA Toolkit, and make sure that the `nvcc` compiler is in your `PATH`, e.g.:
106 + If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from `the official website <https://developer.nvidia.com/cuda-toolkit-archive>`_. After installation, set the environment variable ``CUDA_HOME`` to the installation path of CUDA Toolkit, and make sure that the ``nvcc`` compiler is in your ``PATH``, e.g.:
107
108 .. code-block:: console
109