A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 11, 00:00 UTC
[XPU] decrease IGC_ForceOCLSIMDWidth for speculative decoding triton-xpu kernel compilation (#30538)
Signed-off-by: Yan Ma <yan.ma@intel.com>
Yan Ma committed
Dec 23, 2025 at 13:22 UTC
f1c2c20136cca6ea8798a64855eaf52ee9a42210
3 files changed
+10
-3
docker/Dockerfile.xpu
+6
-1
@@ -2,7 +2,7 @@ FROM intel/deep-learning-essentials:2025.2.2-0-devel-ubuntu24.04 AS vllm-base
2
3
RUN wget -O- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB | gpg --dearmor | tee /usr/share/keyrings/oneapi-archive-keyring.gpg > /dev/null && \
4
echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" | tee /etc/apt/sources.list.d/oneAPI.list && \
5
- add-apt-repository -y ppa:kobuk-team/intel-graphics
5
+ add-apt-repository -y ppa:kobuk-team/intel-graphics-staging
6
7
RUN apt clean && apt-get update -y && \
8
apt-get install -y --no-install-recommends --fix-missing \
@@ -47,6 +47,11 @@ RUN --mount=type=cache,target=/root/.cache/pip \
47
pip install --no-cache-dir \
48
-r requirements/xpu.txt
49
50
+# arctic-inference is built from source which needs torch-xpu properly installed
51
+# used for suffix method speculative decoding
52
+RUN --mount=type=cache,target=/root/.cache/pip \
53
+ pip install --no-cache-dir arctic-inference==0.1.1
54
+
55
ENV LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib/"
56
57
COPY . .
docs/features/README.md
+1
-1
@@ -64,7 +64,7 @@ th:not(:first-child) {
64
| [CP](../configuration/optimization.md#chunked-prefill) | [❌](https://github.com/vllm-project/vllm/issues/2729) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
65
| [APC](automatic_prefix_caching.md) | [❌](https://github.com/vllm-project/vllm/issues/3687) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
66
| [LoRA](lora.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
67
-| [SD](spec_decode.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | [🟠](https://github.com/vllm-project/vllm/issues/26963) |
67
+| [SD](spec_decode.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ |
68
| CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | [❌](https://github.com/vllm-project/vllm/issues/26970) |
69
| [pooling](../models/pooling_models.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
70
| <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
vllm/platforms/xpu.py
+3
-1
@@ -156,7 +156,9 @@ class XPUPlatform(Platform):
156
157
if vllm_config.lora_config is not None:
158
compilation_config.mode = CompilationMode.NONE
159
-
159
+ # decrease triton kernel compilation scratch space for speculative decoding
160
+ if vllm_config.speculative_config is not None:
161
+ os.environ["IGC_ForceOCLSIMDWidth"] = "16" # noqa: SIM112
162
# check and update parallel config
163
parallel_config = vllm_config.parallel_config
164
# Only override worker_cls if it's still the default "auto"