@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 166 lines 6.64 KB
Rendered Raw
1 <!-- markdownlint-disable MD041 -->
2 --8<-- [start:installation]
3
4 vLLM initially supports basic model inference and serving on Intel GPU platform.
5
6 --8<-- [end:installation]
7 --8<-- [start:requirements]
8
9 - Supported Hardware: Intel Data Center GPU, Intel ARC GPU
10 - Dependency: [vllm-xpu-kernels](https://github.com/vllm-project/vllm-xpu-kernels): a package provide all necessary vllm custom kernel when running vLLM on Intel GPU platform,
11 - Python: 3.12
12 !!! warning
13 The provided vllm-xpu-kernels whl is Python3.12 specific so this version is a MUST.
14
15 --8<-- [end:requirements]
16 --8<-- [start:set-up-using-python]
17
18 There is no extra information on creating a new Python environment for this device.
19
20 --8<-- [end:set-up-using-python]
21 --8<-- [start:pre-built-wheels]
22
23 Pre-built vLLM XPU wheels are published to `wheels.vllm.ai`. Each XPU wheel
24 index also contains the `triton==3.7.2+xpu` shim described below. PyTorch XPU
25 packages are served from the PyTorch XPU index, so both index URLs are needed.
26
27 #### Install the latest code
28
29 To install the wheel built from the latest main branch:
30
31 ```bash
32 uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match
33 ```
34
35 Note that after upgrading to PyTorch 2.14, xpu graph support requires specific oneAPI packages.
36
37 ```bash
38 uv pip install \
39 "intel-cmplr-lib-rt==2026.1.1" \
40 "intel-cmplr-lib-ur==2026.1.1" \
41 "intel-cmplr-lic-rt==2026.1.1" \
42 "intel-sycl-rt==2026.1.1" \
43 "oneccl-devel==2022.1.2; platform_system == 'Linux' and platform_machine == 'x86_64'" \
44 "oneccl==2022.1.2; platform_system == 'Linux' and platform_machine == 'x86_64'" \
45 "dpcpp-cpp-rt==2026.1.1" \
46 "intel-opencl-rt==2026.1.1" \
47 "intel-openmp==2026.1.1" \
48 "intel-pti==1.1.0"
49 ```
50
51 #### Install specific revisions
52
53 If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL:
54
55 ```bash
56 export VLLM_COMMIT=730bd35378bf2a5b56b6d3a45be28b3092d26519 # use full commit hash from the main branch
57 uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match
58 ```
59
60 --8<-- [end:pre-built-wheels]
61 --8<-- [start:build-wheel-from-source]
62
63 - First, install required [driver](https://dgpu-docs.intel.com/driver/installation.html#installing-gpu-drivers).
64 - Second, install Python packages for vLLM XPU backend building (Intel OneAPI dependencies are installed automatically as part of `torch-xpu`, see [PyTorch XPU get started](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html)):
65 - Start from vllm-xpu-kernels v0.1.10, we recommend user upgrade driver to [compute runtime 26.18](https://github.com/intel/compute-runtime/releases/tag/26.18.38308.1) release, to avoid potential compatibility issue.
66
67 ```bash
68 git clone https://github.com/vllm-project/vllm.git
69 cd vllm
70 pip install --upgrade pip
71 pip install -v -r requirements/xpu.txt
72 ```
73
74 - Then, install vLLM XPU backend:
75
76 ```bash
77 VLLM_TARGET_DEVICE=xpu pip install --no-build-isolation -e . -v
78 ```
79
80 !!! note
81 `requirements/xpu.txt` pins `triton==3.7.2+xpu`, a compatibility shim
82 hosted on `https://wheels.vllm.ai/xpu` that transparently resolves to
83 the real Intel XPU implementation (`triton-xpu`). This exists because
84 some transitive dependencies (e.g. `xgrammar`) unconditionally
85 require a distribution literally named `triton`, which otherwise
86 resolves to the NVIDIA-only PyPI `triton` package on XPU and can
87 cause correctness or runtime issues. No manual uninstall/reinstall of
88 `triton`/`triton-xpu` is needed; both `pip install` and `uv pip
89 install --index-strategy unsafe-best-match` resolve the correct
90 package automatically.
91
92 --8<-- [end:build-wheel-from-source]
93 --8<-- [start:pre-built-images]
94
95 vLLM offers official Docker images for deployment.
96 The images can be used to run OpenAI compatible server and are available on Docker Hub as [vllm/vllm-openai-xpu](https://hub.docker.com/r/vllm/vllm-openai-xpu/tags).
97
98 - `vllm/vllm-openai-xpu:latest` — stable release, available starting from v0.26.0
99 - `vllm/vllm-openai-xpu:nightly` — preview build from the latest development branch, use this if you want the latest features and fixes
100
101 ```bash
102 docker run --rm \
103 --network=host \
104 --device /dev/dri:/dev/dri \
105 -v /dev/dri/by-path:/dev/dri/by-path \
106 -v ~/.cache/huggingface:/root/.cache/huggingface \
107 --env "HF_TOKEN=$HF_TOKEN" \
108 --ipc=host \
109 --privileged \
110 vllm/vllm-openai-xpu:<tag> \
111 --model Qwen/Qwen3-0.6B
112 ```
113
114 To use the docker image as base for development, you can launch it in interactive session through overriding the entrypoint.
115
116 ???+ console "Commands"
117 ```bash
118 docker run --rm -it \
119 --network=host \
120 --device /dev/dri:/dev/dri \
121 -v /dev/dri/by-path:/dev/dri/by-path \
122 -v ~/.cache/huggingface:/root/.cache/huggingface \
123 --env "HF_TOKEN=$HF_TOKEN" \
124 --ipc=host \
125 --privileged \
126 --entrypoint /bin/bash \
127 vllm/vllm-openai-xpu:<tag>
128 ```
129
130 --8<-- [end:pre-built-images]
131 --8<-- [start:build-image-from-source]
132
133 ```bash
134 docker build -f docker/Dockerfile.xpu -t vllm-xpu-env --shm-size=4g .
135 docker run -it \
136 --rm \
137 --network=host \
138 --device /dev/dri:/dev/dri \
139 -v /dev/dri/by-path:/dev/dri/by-path \
140 --ipc=host \
141 --privileged \
142 vllm-xpu-env
143 ```
144
145 --8<-- [end:build-image-from-source]
146 --8<-- [start:supported-features]
147
148 XPU platform supports **tensor parallel** inference/serving and also supports **pipeline parallel** as a beta feature for online serving. For **pipeline parallel**, we support it on single node with mp as the backend. For example, a reference execution like following:
149
150 ```bash
151 vllm serve facebook/opt-13b \
152 --dtype=bfloat16 \
153 --max_model_len=1024 \
154 --distributed-executor-backend=mp \
155 --pipeline-parallel-size=2 \
156 -tp=8
157 ```
158
159 By default, a ray instance will be launched automatically if no existing one is detected in the system, with `num-gpus` equals to `parallel_config.world_size`. We recommend properly starting a ray cluster before execution, referring to the [examples/ray_serving/run_cluster.sh](https://github.com/vllm-project/vllm/blob/main/examples/ray_serving/run_cluster.sh) helper script.
160
161 --8<-- [end:supported-features]
162 --8<-- [start:distributed-backend]
163
164 XPU platform uses **torch-ccl** for torch<2.8 and **xccl** for torch>=2.8 as distributed backend, since torch 2.8 supports **xccl** as built-in backend for XPU.
165
166 --8<-- [end:distributed-backend]