A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 04:30 UTC
fix gh200 tests on main (#11246)
Signed-off-by: youkaichao <youkaichao@gmail.com>
youkaichao committed
Dec 16, 2024 at 17:22 UTC
35bae114a89e03e3dc6a6d2f758378e58938bffa
2 files changed
+3
-6
.buildkite/run-gh200-test.sh
+2
-2
@@ -6,8 +6,8 @@ set -ex
6
7
# Try building the docker image
8
DOCKER_BUILDKIT=1 docker build . \
9
- --target test \
10
- -platform "linux/arm64" \
9
+ --target vllm-openai \
10
+ --platform "linux/arm64" \
11
-t gh200-test \
12
--build-arg max_jobs=66 \
13
--build-arg nvcc_threads=2 \
docs/source/serving/deploying_with_docker.rst
+1
-4
@@ -54,16 +54,13 @@ of PyTorch Nightly and should be considered **experimental**. Using the flag `--
54
# Example of building on Nvidia GH200 server. (Memory usage: ~12GB, Build time: ~1475s / ~25 min, Image size: 7.26GB)
55
$ DOCKER_BUILDKIT=1 sudo docker build . \
56
--target vllm-openai \
57
- -platform "linux/arm64" \
57
+ --platform "linux/arm64" \
58
-t vllm/vllm-gh200-openai:latest \
59
--build-arg max_jobs=66 \
60
--build-arg nvcc_threads=2 \
61
--build-arg torch_cuda_arch_list="9.0+PTX" \
62
--build-arg vllm_fa_cmake_gpu_arches="90-real"
63
64
-
65
-
66
-
64
To run vLLM:
65
66
.. code-block:: console