A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 06:45 UTC
Documentation about official docker image (#1709)
Simon Mo committed
Nov 19, 2023 at 20:56 UTC
37c1e3c218ed9987cb6e1a52a2efdeed2e3c304a
1 file changed
+16
-2
docs/source/serving/deploying_with_docker.rst
+16
-2
@@ -3,11 +3,25 @@
3
Deploying with Docker
4
============================
5
6
+vLLM offers official docker image for deployment.
7
+The image can be used to run OpenAI compatible server.
8
+The image is available on Docker Hub as `vllm/vllm-openai <https://hub.docker.com/r/vllm/vllm-openai/tags>`_.
9
+
10
+... code-block:: console
11
+
12
+ $ docker run --runtime nvidia --gpus all \
13
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
14
+ -p 8000:8000 \
15
+ --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
16
+ vllm/vllm-openai:latest \
17
+ --model mistralai/Mistral-7B-v0.1
18
+
19
+
20
You can build and run vLLM from source via the provided dockerfile. To build vLLM:
21
22
.. code-block:: console
23
10
- $ DOCKER_BUILDKIT=1 docker build . --target vllm --tag vllm --build-arg max_jobs=8
24
+ $ DOCKER_BUILDKIT=1 docker build . --target vllm-openai --tag vllm/vllm-openai --build-arg max_jobs=8
25
26
To run vLLM:
27
@@ -17,5 +31,5 @@ To run vLLM:
31
-v ~/.cache/huggingface:/root/.cache/huggingface \
32
-p 8000:8000 \
33
--env "HUGGING_FACE_HUB_TOKEN=<secret>" \
20
- vllm <args...>
34
+ vllm/vllm-openai <args...>
35