@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / c9d6ff530b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 02:00 UTC

Explain where the engine args go when using Docker (#12041)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Harry Mellor committed Jan 14, 2025 at 16:05 UTC c9d6ff530b32c526bedda3105dcbab3d2f6ce992
1 file changed +2
docs/source/deployment/docker.md
+2
@@ -19,6 +19,8 @@ $ docker run --runtime nvidia --gpus all \
19 --model mistralai/Mistral-7B-v0.1
20 ```
21
22 +You can add any other <project:#engine-args> you need after the image tag (`vllm/vllm-openai:latest`).
23 +
24 ```{note}
25 You can either use the `ipc=host` flag or `--shm-size` flag to allow the
26 container to access the host's shared memory. vLLM uses PyTorch, which uses shared