@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 0f621c2c7d
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 06:45 UTC

[Docs] Add information about using shared memory in docker (#1845)

Simon Mo committed Nov 29, 2023 at 18:33 UTC 0f621c2c7dfe409b6e24e8810dc039745b9a8a7a
2 files changed +10 -2
docs/source/models/adding_model.rst
+1 -1
@@ -18,7 +18,7 @@ This document provides a high-level guide on integrating a `HuggingFace Transfor
18 0. Fork the vLLM repository
19 --------------------------------
20
21 -Start by forking our `GitHub <https://github.com/vllm-project/vllm/>`_ repository and then :ref:`build it from source <build_from_source>`.
21 +Start by forking our `GitHub`_ repository and then :ref:`build it from source <build_from_source>`.
22 This gives you the ability to modify the codebase and test your model.
23
24
docs/source/serving/deploying_with_docker.rst
+9 -1
@@ -11,12 +11,20 @@ The image is available on Docker Hub as `vllm/vllm-openai <https://hub.docker.co
11
12 $ docker run --runtime nvidia --gpus all \
13 -v ~/.cache/huggingface:/root/.cache/huggingface \
14 - -p 8000:8000 \
14 --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
15 + -p 8000:8000 \
16 + --ipc=host \
17 vllm/vllm-openai:latest \
18 --model mistralai/Mistral-7B-v0.1
19
20
21 +.. note::
22 +
23 + You can either use the ``ipc=host`` flag or ``--shm-size`` flag to allow the
24 + container to access the host's shared memory. vLLM uses PyTorch, which uses shared
25 + memory to share data between processes under the hood, particularly for tensor parallel inference.
26 +
27 +
28 You can build and run vLLM from source via the provided dockerfile. To build vLLM:
29
30 .. code-block:: console