A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 06:45 UTC
Use `--ipc=host` in docker run for distributed inference (#1125)
Woosuk Kwon committed
Sep 21, 2023 at 18:26 UTC
7d7e3b78a3c265ab3c57eeff43af56f509907998
1 file changed
+2
-1
docs/source/getting_started/installation.rst
+2
-1
@@ -46,4 +46,5 @@ You can also build and install vLLM from source:
46
.. code-block:: console
47
48
$ # Pull the Docker image with CUDA 11.8.
49
- $ docker run --gpus all -it --rm --shm-size=8g nvcr.io/nvidia/pytorch:22.12-py3
49
+ $ # Use `--ipc=host` to make sure the shared memory is large enough.
50
+ $ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:22.12-py3