@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 2768884ac4
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 01:45 UTC

[Doc] Add note for docker user (#4340)

Co-authored-by: Simon Mo <simon.mo@hey.com>

youkaichao committed Apr 24, 2024 at 14:09 UTC 2768884ac4a026609efceef92edea55839af0c30
1 file changed +3
docs/source/serving/deploying_with_docker.rst
+3
@@ -49,3 +49,6 @@ To run vLLM:
49 --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
50 vllm/vllm-openai <args...>
51
52 +.. note::
53 +
54 + vLLM docker image is currently designed to be run under the root user (contribution welcomed for changing this!). It will try to load library at runtime under the root user's home directory, e.g. `/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` . If you are running the container under a different user, you may need to change the permissions of the library (and all the parent directories) to allow the user to access it. Then run vLLM with environment variable `VLLM_NCCL_SO_PATH=/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` .