@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 09ccc9c8f7
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 02:00 UTC

[Documentation][AMD] Add information about prebuilt ROCm vLLM docker for perf validation purpose (#12281)

Signed-off-by: Hongxia Yang <hongxyan@amd.com>

Hongxia Yang committed Jan 21, 2025 at 18:49 UTC 09ccc9c8f7943220f0b211639cee0655527e857d
1 file changed +8
docs/source/getting_started/installation/gpu/rocm.inc.md
+8
@@ -13,6 +13,14 @@ vLLM supports AMD GPUs with ROCm 6.2.
13
14 Currently, there are no pre-built ROCm wheels.
15
16 +However, the [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offers a prebuilt, optimized
17 +docker image designed for validating inference performance on the AMD Instinct™ MI300X accelerator.
18 +
19 +```{tip}
20 +Please check [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/performance-validation/mi300x/vllm-benchmark.html)
21 +for instructions on how to use this prebuilt docker image.
22 +```
23 +
24 ### Build wheel from source
25
26 0. Install prerequisites (skip if you are already in an environment/docker with the following installed):