@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 7d7e3b78a3
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 06:45 UTC

Use `--ipc=host` in docker run for distributed inference (#1125)

Woosuk Kwon committed Sep 21, 2023 at 18:26 UTC 7d7e3b78a3c265ab3c57eeff43af56f509907998
1 file changed +2 -1
docs/source/getting_started/installation.rst
+2 -1
@@ -46,4 +46,5 @@ You can also build and install vLLM from source:
46 .. code-block:: console
47
48 $ # Pull the Docker image with CUDA 11.8.
49 - $ docker run --gpus all -it --rm --shm-size=8g nvcr.io/nvidia/pytorch:22.12-py3
49 + $ # Use `--ipc=host` to make sure the shared memory is large enough.
50 + $ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:22.12-py3