A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 19:45 UTC
docs: Add documentation for s390x cpu implementation (#14198)
Signed-off-by: Dilip Gowda Bhagavan <dilip.bhagavan@ibm.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Dilip Gowda Bhagavan committed
Mar 11, 2025 at 22:32 UTC
07964e2f30cdf3c608605648f7a31ead60565fcf
3 files changed
+97
docs/source/getting_started/installation.md
+1
@@ -21,6 +21,7 @@ installation/ai_accelerator
21
- Intel/AMD x86
22
- ARM AArch64
23
- Apple silicon
24
+ - IBM Z (S390X)
25
- <project:installation/ai_accelerator.md>
26
- Google TPU
27
- Intel Gaudi
docs/source/getting_started/installation/cpu.md
+34
@@ -36,6 +36,16 @@ vLLM is a Python library that supports the following CPU variants. Select your C
36
37
::::
38
39
+::::{tab-item} IBM Z (S390X)
40
+:sync: s390x
41
+
42
+:::{include} cpu/s390x.inc.md
43
+:start-after: "# Installation"
44
+:end-before: "## Requirements"
45
+:::
46
+
47
+::::
48
+
49
:::::
50
51
## Requirements
@@ -75,6 +85,16 @@ vLLM is a Python library that supports the following CPU variants. Select your C
85
86
::::
87
88
+::::{tab-item} IBM Z (S390X)
89
+:sync: s390x
90
+
91
+:::{include} cpu/s390x.inc.md
92
+:start-after: "## Requirements"
93
+:end-before: "## Set up using Python"
94
+:::
95
+
96
+::::
97
+
98
:::::
99
100
## Set up using Python
@@ -123,6 +143,16 @@ Currently, there are no pre-built CPU wheels.
143
144
::::
145
146
+::::{tab-item} IBM Z (s390x)
147
+:sync: s390x
148
+
149
+:::{include} cpu/s390x.inc.md
150
+:start-after: "### Build wheel from source"
151
+:end-before: "## Set up using Docker"
152
+:::
153
+
154
+::::
155
+
156
:::::
157
158
## Set up using Docker
@@ -147,6 +177,10 @@ $ docker run -it \
177
For ARM or Apple silicon, use `Dockerfile.arm`
178
::::
179
180
+::::{tip}
181
+For IBM Z (s390x), use `Dockerfile.s390x` and in `docker run` use flag `--dtype float`
182
+::::
183
+
184
## Supported features
185
186
vLLM CPU backend supports the following vLLM features:
docs/source/getting_started/installation/cpu/s390x.inc.md
new
+62
@@ -0,0 +1,62 @@
1
+# Installation
2
+
3
+vLLM has experimental support for s390x architecture on IBM Z platform. For now, users shall build from the vLLM source to natively run on IBM Z platform.
4
+
5
+Currently the CPU implementation for s390x architecture supports FP32 datatype only.
6
+
7
+:::{attention}
8
+There are no pre-built wheels or images for this device, so you must build vLLM from source.
9
+:::
10
+
11
+## Requirements
12
+
13
+- OS: `Linux`
14
+- SDK: `gcc/g++ >= 12.3.0` or later with Command Line Tools
15
+- Instruction Set Architecture (ISA): VXE support is required. Works with Z14 and above.
16
+- Build install python packages: `pyarrow`, `torch` and `torchvision`
17
+
18
+## Set up using Python
19
+
20
+### Pre-built wheels
21
+
22
+### Build wheel from source
23
+
24
+Install the following packages from the package manager before building the vLLM. For example on RHEL 9.4:
25
+
26
+```console
27
+dnf install -y \
28
+ which procps findutils tar vim git gcc g++ make patch make cython zlib-devel \
29
+ libjpeg-turbo-devel libtiff-devel libpng-devel libwebp-devel freetype-devel harfbuzz-devel \
30
+ openssl-devel openblas openblas-devel wget autoconf automake libtool cmake numactl-devel
31
+```
32
+
33
+Install rust>=1.80 which is needed for `outlines-core` and `uvloop` python packages installation.
34
+
35
+```console
36
+curl https://sh.rustup.rs -sSf | sh -s -- -y && \
37
+ . "$HOME/.cargo/env"
38
+```
39
+
40
+Execute the following commands to build and install vLLM from the source.
41
+
42
+::::{tip}
43
+Please build the following dependencies, `torchvision`, `pyarrow` from the source before building vLLM.
44
+::::
45
+
46
+```console
47
+ sed -i '/^torch/d' requirements-build.txt # remove torch from requirements-build.txt since we use nightly builds
48
+ pip install -v \
49
+ --extra-index-url https://download.pytorch.org/whl/nightly/cpu \
50
+ -r requirements-build.txt \
51
+ -r requirements-cpu.txt \
52
+ VLLM_TARGET_DEVICE=cpu python setup.py bdist_wheel && \
53
+ pip install dist/*.whl
54
+```
55
+
56
+## Set up using Docker
57
+
58
+### Pre-built images
59
+
60
+### Build image from source
61
+
62
+## Extra information