A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 13:30 UTC
[Doc] Add top anchor and a note to quantization/bitblas.md (#17042)
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
Michael Yao committed
Apr 23, 2025 at 22:32 UTC
f7912cba3d613afa8b96ce2e04dad671205050c6
1 file changed
+8
docs/source/features/quantization/bitblas.md
+8
@@ -1,7 +1,15 @@
1
+(bitblas)=
2
+
3
# BitBLAS
4
5
vLLM now supports [BitBLAS](https://github.com/microsoft/BitBLAS) for more efficient and flexible model inference. Compared to other quantization frameworks, BitBLAS provides more precision combinations.
6
7
+:::{note}
8
+Ensure your hardware supports the selected `dtype` (`torch.bfloat16` or `torch.float16`).
9
+Most recent NVIDIA GPUs support `float16`, while `bfloat16` is more common on newer architectures like Ampere or Hopper.
10
+For details see [supported hardware](https://docs.vllm.ai/en/latest/features/quantization/supported_hardware.html).
11
+:::
12
+
13
Below are the steps to utilize BitBLAS with vLLM.
14
15
```console