| 1 | # BitsAndBytes |
| 2 | |
| 3 | vLLM now supports [BitsAndBytes](https://github.com/TimDettmers/bitsandbytes) for more efficient model inference. |
| 4 | BitsAndBytes quantizes models to reduce memory usage and enhance performance without significantly sacrificing accuracy. |
| 5 | Compared to other quantization methods, BitsAndBytes eliminates the need for calibrating the quantized model with input data. |
| 6 | |
| 7 | BitsAndBytes support is provided by the out-of-tree |
| 8 | [`vllm-bnb-plugin`](https://github.com/vllm-project/vllm-bnb-plugin). |
| 9 | Install the plugin before using BitsAndBytes with vLLM: |
| 10 | |
| 11 | ```bash |
| 12 | uv pip install vllm-bnb-plugin |
| 13 | ``` |
| 14 | |
| 15 | vLLM reads the model's config file and supports both in-flight quantization and pre-quantized checkpoint. |
| 16 | |
| 17 | You can find bitsandbytes quantized models on [Hugging Face](https://huggingface.co/models?search=bitsandbytes). |
| 18 | And usually, these repositories have a config.json file that includes a quantization_config section. |
| 19 | |
| 20 | ## Read quantized checkpoint |
| 21 | |
| 22 | For pre-quantized checkpoints, vLLM will try to infer the quantization method from the config file, so you don't need to explicitly specify the quantization argument. |
| 23 | |
| 24 | ```python |
| 25 | from vllm import LLM |
| 26 | import torch |
| 27 | # unsloth/tinyllama-bnb-4bit is a pre-quantized checkpoint. |
| 28 | model_id = "unsloth/tinyllama-bnb-4bit" |
| 29 | llm = LLM( |
| 30 | model=model_id, |
| 31 | dtype=torch.bfloat16, |
| 32 | trust_remote_code=True, |
| 33 | ) |
| 34 | ``` |
| 35 | |
| 36 | ## Inflight quantization: load as 4bit quantization |
| 37 | |
| 38 | For inflight 4bit quantization with BitsAndBytes, you need to explicitly specify the quantization argument. |
| 39 | |
| 40 | ```python |
| 41 | from vllm import LLM |
| 42 | import torch |
| 43 | model_id = "huggyllama/llama-7b" |
| 44 | llm = LLM( |
| 45 | model=model_id, |
| 46 | dtype=torch.bfloat16, |
| 47 | trust_remote_code=True, |
| 48 | quantization="bitsandbytes", |
| 49 | ) |
| 50 | ``` |
| 51 | |
| 52 | ## OpenAI Compatible Server |
| 53 | |
| 54 | Append the following to your model arguments for 4bit inflight quantization: |
| 55 | |
| 56 | ```bash |
| 57 | --quantization bitsandbytes |
| 58 | ``` |