A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 06:45 UTC
[Docs] Add information about using shared memory in docker (#1845)
Simon Mo committed
Nov 29, 2023 at 18:33 UTC
0f621c2c7dfe409b6e24e8810dc039745b9a8a7a
2 files changed
+10
-2
docs/source/models/adding_model.rst
+1
-1
@@ -18,7 +18,7 @@ This document provides a high-level guide on integrating a `HuggingFace Transfor
18
0. Fork the vLLM repository
19
--------------------------------
20
21
-Start by forking our `GitHub <https://github.com/vllm-project/vllm/>`_ repository and then :ref:`build it from source <build_from_source>`.
21
+Start by forking our `GitHub`_ repository and then :ref:`build it from source <build_from_source>`.
22
This gives you the ability to modify the codebase and test your model.
23
24
docs/source/serving/deploying_with_docker.rst
+9
-1
@@ -11,12 +11,20 @@ The image is available on Docker Hub as `vllm/vllm-openai <https://hub.docker.co
11
12
$ docker run --runtime nvidia --gpus all \
13
-v ~/.cache/huggingface:/root/.cache/huggingface \
14
- -p 8000:8000 \
14
--env "HUGGING_FACE_HUB_TOKEN=<secret>" \
15
+ -p 8000:8000 \
16
+ --ipc=host \
17
vllm/vllm-openai:latest \
18
--model mistralai/Mistral-7B-v0.1
19
20
21
+.. note::
22
+
23
+ You can either use the ``ipc=host`` flag or ``--shm-size`` flag to allow the
24
+ container to access the host's shared memory. vLLM uses PyTorch, which uses shared
25
+ memory to share data between processes under the hood, particularly for tensor parallel inference.
26
+
27
+
28
You can build and run vLLM from source via the provided dockerfile. To build vLLM:
29
30
.. code-block:: console