A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 23:45 UTC
[Docs] Update docker.md with HF_TOKEN, new model, and podman fix (#21856)
Michael Goin committed
Jul 29, 2025 at 22:45 UTC
fb58e3a651f7321eb882ff28018a918b31726c82
1 file changed
+5
-5
docs/deployment/docker.md
+5
-5
@@ -10,23 +10,23 @@ The image can be used to run OpenAI compatible server and is available on Docker
10
```bash
11
docker run --runtime nvidia --gpus all \
12
-v ~/.cache/huggingface:/root/.cache/huggingface \
13
- --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
13
+ --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
14
-p 8000:8000 \
15
--ipc=host \
16
vllm/vllm-openai:latest \
17
- --model mistralai/Mistral-7B-v0.1
17
+ --model Qwen/Qwen3-0.6B
18
```
19
20
This image can also be used with other container engines such as [Podman](https://podman.io/).
21
22
```bash
23
-podman run --gpus all \
23
+podman run --device nvidia.com/gpu=all \
24
-v ~/.cache/huggingface:/root/.cache/huggingface \
25
--env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
26
-p 8000:8000 \
27
--ipc=host \
28
- vllm/vllm-openai:latest \
29
- --model mistralai/Mistral-7B-v0.1
28
+ docker.io/vllm/vllm-openai:latest \
29
+ --model Qwen/Qwen3-0.6B
30
```
31
32
You can add any other [engine-args](../configuration/engine_args.md) you need after the image tag (`vllm/vllm-openai:latest`).