A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 13:30 UTC
[Doc] add podman setup instructions for official image (#16796)
Signed-off-by: Nathan Weinberg <nweinber@redhat.com>
Nathan Weinberg committed
Apr 18, 2025 at 02:10 UTC
9c1d5b456d916c68f41f471bfe8ea13fc69b80fa
1 file changed
+12
docs/source/deployment/docker.md
+12
@@ -19,6 +19,18 @@ $ docker run --runtime nvidia --gpus all \
19
--model mistralai/Mistral-7B-v0.1
20
```
21
22
+This image can also be used with other container engines such as [Podman](https://podman.io/).
23
+
24
+```console
25
+$ podman run --gpus all \
26
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
27
+ --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
28
+ -p 8000:8000 \
29
+ --ipc=host \
30
+ vllm/vllm-openai:latest \
31
+ --model mistralai/Mistral-7B-v0.1
32
+```
33
+
34
You can add any other <project:#engine-args> you need after the image tag (`vllm/vllm-openai:latest`).
35
36
:::{note}