@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 35bae114a8
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 04:30 UTC

fix gh200 tests on main (#11246)

Signed-off-by: youkaichao <youkaichao@gmail.com>

youkaichao committed Dec 16, 2024 at 17:22 UTC 35bae114a89e03e3dc6a6d2f758378e58938bffa
2 files changed +3 -6
.buildkite/run-gh200-test.sh
+2 -2
@@ -6,8 +6,8 @@ set -ex
6
7 # Try building the docker image
8 DOCKER_BUILDKIT=1 docker build . \
9 - --target test \
10 - -platform "linux/arm64" \
9 + --target vllm-openai \
10 + --platform "linux/arm64" \
11 -t gh200-test \
12 --build-arg max_jobs=66 \
13 --build-arg nvcc_threads=2 \
docs/source/serving/deploying_with_docker.rst
+1 -4
@@ -54,16 +54,13 @@ of PyTorch Nightly and should be considered **experimental**. Using the flag `--
54 # Example of building on Nvidia GH200 server. (Memory usage: ~12GB, Build time: ~1475s / ~25 min, Image size: 7.26GB)
55 $ DOCKER_BUILDKIT=1 sudo docker build . \
56 --target vllm-openai \
57 - -platform "linux/arm64" \
57 + --platform "linux/arm64" \
58 -t vllm/vllm-gh200-openai:latest \
59 --build-arg max_jobs=66 \
60 --build-arg nvcc_threads=2 \
61 --build-arg torch_cuda_arch_list="9.0+PTX" \
62 --build-arg vllm_fa_cmake_gpu_arches="90-real"
63
64 -
65 -
66 -
64 To run vLLM:
65
66 .. code-block:: console