A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 22:15 UTC
[Misc][Docs] Raise error when flashinfer is not installed and `VLLM_ATTENTION_BACKEND` is set (#12513)
Signed-off-by: NickLucche <nlucches@redhat.com>
Nicolò Lucchesi committed
Feb 24, 2025 at 16:43 UTC
444b0f0f62989246abe115f0b24d2ba289612b23
2 files changed
+19
docs/source/getting_started/quickstart.md
+10
@@ -184,3 +184,13 @@ chat_response = client.chat.completions.create(
184
)
185
print("Chat response:", chat_response)
186
```
187
+
188
+## On Attention Backends
189
+
190
+Currently, vLLM supports multiple backends for efficient Attention computation across different platforms and accelerator architectures. It automatically selects the most performant backend compatible with your system and model specifications.
191
+
192
+If desired, you can also manually set the backend of your choice by configuring the environment variable `VLLM_ATTENTION_BACKEND` to one of the following options: `FLASH_ATTN`, `FLASHINFER` or `XFORMERS`.
193
+
194
+```{attention}
195
+There are no pre-built vllm wheels containing Flash Infer, so you must install it in your environment first. Refer to the [Flash Infer official docs](https://docs.flashinfer.ai/) or see [Dockerfile](https://github.com/vllm-project/vllm/blob/main/Dockerfile) for instructions on how to install it.
196
+```
vllm/config.py
+9
@@ -9,6 +9,7 @@ import sys
9
import warnings
10
from contextlib import contextmanager
11
from dataclasses import dataclass, field, replace
12
+from importlib.util import find_spec
13
from pathlib import Path
14
from typing import (TYPE_CHECKING, Any, Callable, ClassVar, Counter, Dict,
15
Final, List, Literal, Mapping, Optional, Protocol, Set,
@@ -294,6 +295,14 @@ class ModelConfig:
295
296
self.maybe_pull_model_tokenizer_for_s3(model, tokenizer)
297
298
+ if (backend := envs.VLLM_ATTENTION_BACKEND
299
+ ) and backend == "FLASHINFER" and find_spec("flashinfer") is None:
300
+ raise ValueError(
301
+ "VLLM_ATTENTION_BACKEND is set to FLASHINFER, but flashinfer "
302
+ "module was not found."
303
+ "See https://github.com/vllm-project/vllm/blob/main/Dockerfile"
304
+ "for instructions on how to install it.")
305
+
306
# The tokenizer version is consistent with the model version by default.
307
if tokenizer_revision is None:
308
self.tokenizer_revision = revision