A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 07:15 UTC
Update docker docs with ARM CUDA cross-compile (#19037)
Signed-off-by: mgoin <michael@neuralmagic.com>
Michael Goin committed
Jun 3, 2025 at 04:21 UTC
6d18ed2a2e858a8061dfe8c2e140c2c498d6a99a
1 file changed
+12
-1
docs/deployment/docker.md
+12
-1
@@ -107,10 +107,21 @@ DOCKER_BUILDKIT=1 docker build . \
107
-t vllm/vllm-gh200-openai:latest \
108
--build-arg max_jobs=66 \
109
--build-arg nvcc_threads=2 \
110
- --build-arg torch_cuda_arch_list="9.0+PTX" \
110
+ --build-arg torch_cuda_arch_list="9.0 10.0+PTX" \
111
--build-arg vllm_fa_cmake_gpu_arches="90-real"
112
```
113
114
+!!! note
115
+ If you are building the `linux/arm64` image on a non-ARM host (e.g., an x86_64 machine), you need to ensure your system is set up for cross-compilation using QEMU. This allows your host machine to emulate ARM64 execution.
116
+
117
+ Run the following command on your host machine to register QEMU user static handlers:
118
+
119
+ ```console
120
+ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
121
+ ```
122
+
123
+ After setting up QEMU, you can use the `--platform "linux/arm64"` flag in your `docker build` command.
124
+
125
## Use the custom-built vLLM Docker image
126
127
To run vLLM with the custom-built Docker image: