| 1 | # FP8 W8A8 |
| 2 | |
| 3 | vLLM supports FP8 (8-bit floating point) weight and activation quantization using hardware acceleration on GPUs such as Nvidia H100 and AMD MI300x. |
| 4 | Ada Lovelace, Hopper, and Blackwell GPUs are supported for W8A8. |
| 5 | Turing/Ampere GPUs are supported for W8A16 (weight-only FP8) utilizing Marlin kernels. |
| 6 | Quantization of models with FP8 allows for a 2x reduction in model memory requirements and up to a 1.6x improvement in throughput with minimal impact on accuracy. |
| 7 | |
| 8 | Please visit the HF collection of [quantized FP8 checkpoints of popular LLMs ready to use with vLLM](https://huggingface.co/collections/neuralmagic/fp8-llms-for-vllm-666742ed2b78b7ac8df13127). |
| 9 | |
| 10 | The FP8 types typically supported in hardware have two distinct representations, each useful in different scenarios: |
| 11 | |
| 12 | - **E4M3**: Consists of 1 sign bit, 4 exponent bits, and 3 bits of mantissa. It can store values up to +/-448 and `nan`. |
| 13 | - **E5M2**: Consists of 1 sign bit, 5 exponent bits, and 2 bits of mantissa. It can store values up to +/-57344, +/- `inf`, and `nan`. The tradeoff for the increased dynamic range is lower precision of the stored values. |
| 14 | |
| 15 | !!! note |
| 16 | FP8 computation is supported on NVIDIA GPUs with compute capability >= 8.9 (Ada Lovelace, Hopper, Blackwell). |
| 17 | FP8 models will run on compute capability >= 7.5 (Turing) as weight-only W8A16, utilizing FP8 Marlin. |
| 18 | |
| 19 | !!! note "GEMM kernel selection" |
| 20 | vLLM picks an FP8 GEMM kernel automatically at load time and logs a |
| 21 | `Selected <kernel> for <module>` line at startup. For block-quantized |
| 22 | checkpoints on CUDA it tries, in order: a FlashInfer/DeepGEMM hybrid |
| 23 | (Hopper only), DeepGEMM, CUTLASS, Marlin, Triton, Humming, then a |
| 24 | PyTorch fallback. GPUs without native FP8 support (e.g. Turing/Ampere) |
| 25 | land on weight-only (W8A16) Marlin. |
| 26 | |
| 27 | If inference hangs with no error, try `VLLM_USE_DEEP_GEMM=0` or |
| 28 | `--linear-backend cutlass`. |
| 29 | |
| 30 | `--linear-backend` only affects quantized linear layers; MoE experts |
| 31 | use the separate `--moe-backend`. An explicit backend that is not |
| 32 | supported on your hardware raises an error rather than falling back. |
| 33 | The full list is documented under |
| 34 | [`KernelConfig`](../../../cli/serve.md#kernelconfig) in the CLI |
| 35 | reference and shown by `vllm serve --help=KernelConfig`. |
| 36 | |
| 37 | ## Installation |
| 38 | |
| 39 | To produce performant FP8 quantized models with vLLM, you'll need to install the [llm-compressor](https://github.com/vllm-project/llm-compressor/) library: |
| 40 | |
| 41 | ```bash |
| 42 | (venv-llm-compressor) pip install llmcompressor |
| 43 | ``` |
| 44 | |
| 45 | Additionally, install `vllm` and `lm-evaluation-harness` for evaluation: |
| 46 | |
| 47 | ```bash |
| 48 | (venv-vllm) pip install vllm "lm-eval[api]>=0.4.12" |
| 49 | ``` |
| 50 | |
| 51 | Please use separate environments for vLLM and llm-compressor as they might not work together. |
| 52 | |
| 53 | ## Quantization Process |
| 54 | |
| 55 | The quantization process involves three main steps: |
| 56 | |
| 57 | 1. Loading the model |
| 58 | 2. Applying quantization |
| 59 | 3. Evaluating accuracy in vLLM |
| 60 | |
| 61 | ### 1. Loading the Model |
| 62 | |
| 63 | Load your model and tokenizer using the standard `transformers` AutoModel classes: |
| 64 | |
| 65 | ```python |
| 66 | from transformers import AutoTokenizer, AutoModelForCausalLM |
| 67 | |
| 68 | MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct" |
| 69 | model = AutoModelForCausalLM.from_pretrained( |
| 70 | MODEL_ID, |
| 71 | device_map="auto", |
| 72 | dtype="auto", |
| 73 | ) |
| 74 | tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) |
| 75 | ``` |
| 76 | |
| 77 | ### 2. Applying Quantization |
| 78 | |
| 79 | For FP8 quantization, we can recover accuracy with simple RTN quantization. We recommend targeting all `Linear` layers using the `FP8_DYNAMIC` scheme, which uses: |
| 80 | |
| 81 | - Static, per-channel quantization on the weights |
| 82 | - Dynamic, per-token quantization on the activations |
| 83 | |
| 84 | Since simple RTN does not require data for weight quantization and the activations are quantized dynamically, we do not need any calibration data for this quantization flow. |
| 85 | |
| 86 | ```python |
| 87 | from llmcompressor import oneshot |
| 88 | from llmcompressor.modifiers.quantization import QuantizationModifier |
| 89 | |
| 90 | # Configure the simple PTQ quantization |
| 91 | recipe = QuantizationModifier( |
| 92 | targets="Linear", |
| 93 | scheme="FP8_DYNAMIC", |
| 94 | ignore=["lm_head"], |
| 95 | ) |
| 96 | |
| 97 | # Apply the quantization algorithm. |
| 98 | oneshot(model=model, recipe=recipe) |
| 99 | |
| 100 | # Save the model: Meta-Llama-3-8B-Instruct-FP8-Dynamic |
| 101 | SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic" |
| 102 | model.save_pretrained(SAVE_DIR) |
| 103 | tokenizer.save_pretrained(SAVE_DIR) |
| 104 | ``` |
| 105 | |
| 106 | ### 3. Evaluating Accuracy |
| 107 | |
| 108 | Load and run the model in `vllm`: |
| 109 | |
| 110 | ```python |
| 111 | from vllm import LLM |
| 112 | |
| 113 | llm = LLM("./Meta-Llama-3-8B-Instruct-FP8-Dynamic") |
| 114 | result = llm.generate("Hello my name is") |
| 115 | print(result[0].outputs[0].text) |
| 116 | ``` |
| 117 | |
| 118 | Evaluate accuracy with `lm_eval` (for example on 250 samples of `gsm8k`): |
| 119 | |
| 120 | !!! note |
| 121 | Quantized models can be sensitive to the presence of the `bos` token. `lm_eval` does not add a `bos` token by default, so make sure to include the `add_bos_token=True` argument when running your evaluations. |
| 122 | |
| 123 | ```bash |
| 124 | MODEL=$PWD/Meta-Llama-3-8B-Instruct-FP8-Dynamic |
| 125 | lm_eval \ |
| 126 | --model vllm \ |
| 127 | --model_args pretrained=$MODEL,add_bos_token=True \ |
| 128 | --tasks gsm8k --num_fewshot 5 --batch_size auto --limit 250 |
| 129 | ``` |
| 130 | |
| 131 | Here's an example of the resulting scores: |
| 132 | |
| 133 | ```text |
| 134 | |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr| |
| 135 | | --- |------:| -------------- |-----:| --------- | - |----:| - |-----:| |
| 136 | |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.768|± |0.0268| |
| 137 | | | |strict-match | 5|exact_match|↑ |0.768|± |0.0268| |
| 138 | ``` |
| 139 | |
| 140 | ## Troubleshooting and Support |
| 141 | |
| 142 | If you encounter any issues or have feature requests, please open an issue on the [vllm-project/llm-compressor](https://github.com/vllm-project/llm-compressor/issues) GitHub repository. |
| 143 | |
| 144 | ## Online Dynamic Quantization |
| 145 | |
| 146 | Dynamic quantization of an original precision BF16/FP16 model to FP8 can be achieved with vLLM without any calibration data required. You can enable the feature by specifying `--quantization="fp8_per_tensor"` in the command line or setting `quantization="fp8_per_tensor"` in the LLM constructor. |
| 147 | |
| 148 | In this mode, all Linear modules (except for the final `lm_head`) have their weights quantized down to FP8_E4M3 precision with a per-tensor scale. Activations have their minimum and maximum values calculated during each forward pass to provide a dynamic per-tensor scale for high accuracy. As a result, latency improvements are limited in this mode. |
| 149 | |
| 150 | ```python |
| 151 | from vllm import LLM |
| 152 | |
| 153 | llm = LLM("facebook/opt-125m", quantization="fp8_per_tensor") |
| 154 | # INFO 06-10 17:55:42 model_runner.py:157] Loading model weights took 0.1550 GB |
| 155 | result = llm.generate("Hello, my name is") |
| 156 | print(result[0].outputs[0].text) |
| 157 | ``` |