@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 37c1e3c218
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 06:45 UTC

Documentation about official docker image (#1709)

Simon Mo committed Nov 19, 2023 at 20:56 UTC 37c1e3c218ed9987cb6e1a52a2efdeed2e3c304a
1 file changed +16 -2
docs/source/serving/deploying_with_docker.rst
+16 -2
@@ -3,11 +3,25 @@
3 Deploying with Docker
4 ============================
5
6 +vLLM offers official docker image for deployment.
7 +The image can be used to run OpenAI compatible server.
8 +The image is available on Docker Hub as `vllm/vllm-openai <https://hub.docker.com/r/vllm/vllm-openai/tags>`_.
9 +
10 +... code-block:: console
11 +
12 + $ docker run --runtime nvidia --gpus all \
13 + -v ~/.cache/huggingface:/root/.cache/huggingface \
14 + -p 8000:8000 \
15 + --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
16 + vllm/vllm-openai:latest \
17 + --model mistralai/Mistral-7B-v0.1
18 +
19 +
20 You can build and run vLLM from source via the provided dockerfile. To build vLLM:
21
22 .. code-block:: console
23
10 - $ DOCKER_BUILDKIT=1 docker build . --target vllm --tag vllm --build-arg max_jobs=8
24 + $ DOCKER_BUILDKIT=1 docker build . --target vllm-openai --tag vllm/vllm-openai --build-arg max_jobs=8
25
26 To run vLLM:
27
@@ -17,5 +31,5 @@ To run vLLM:
31 -v ~/.cache/huggingface:/root/.cache/huggingface \
32 -p 8000:8000 \
33 --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
20 - vllm <args...>
34 + vllm/vllm-openai <args...>
35