A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 02:00 UTC
Explain where the engine args go when using Docker (#12041)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Jan 14, 2025 at 16:05 UTC
c9d6ff530b32c526bedda3105dcbab3d2f6ce992
1 file changed
+2
docs/source/deployment/docker.md
+2
@@ -19,6 +19,8 @@ $ docker run --runtime nvidia --gpus all \
19
--model mistralai/Mistral-7B-v0.1
20
```
21
22
+You can add any other <project:#engine-args> you need after the image tag (`vllm/vllm-openai:latest`).
23
+
24
```{note}
25
You can either use the `ipc=host` flag or `--shm-size` flag to allow the
26
container to access the host's shared memory. vLLM uses PyTorch, which uses shared