@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 01a55941f5
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 02:00 UTC

[Docs] Update FP8 KV Cache documentation (#12238)

Signed-off-by: mgoin <michael@neuralmagic.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Michael Goin committed Jan 22, 2025 at 22:18 UTC 01a55941f5443c16a88889995f8149edcee2eec3
4 files changed +146 -77
docs/source/features/quantization/fp8_e4m3_kvcache.md deleted
-44
@@ -1,44 +0,0 @@
1 -(fp8-e4m3-kvcache)=
2 -
3 -# FP8 E4M3 KV Cache
4 -
5 -Quantizing the KV cache to FP8 reduces its memory footprint. This increases the number of tokens that can be stored in the cache,
6 -improving throughput. OCP (Open Compute Project www.opencompute.org) specifies two common 8-bit floating point data formats: E5M2
7 -(5 exponent bits and 2 mantissa bits) and E4M3FN (4 exponent bits and 3 mantissa bits), often shortened as E4M3. One benefit of
8 -the E4M3 format over E5M2 is that floating point numbers are represented in higher precision. However, the small dynamic range of
9 -FP8 E4M3 (±240.0 can be represented) typically necessitates the use of a higher-precision (typically FP32) scaling factor alongside
10 -each quantized tensor. For now, only per-tensor (scalar) scaling factors are supported. Development is ongoing to support scaling
11 -factors of a finer granularity (e.g. per-channel).
12 -
13 -These scaling factors can be specified by passing an optional quantization param JSON to the LLM engine at load time. If
14 -this JSON is not specified, scaling factors default to 1.0. These scaling factors are typically obtained when running an
15 -unquantized model through a quantizer tool (e.g. AMD quantizer or NVIDIA AMMO).
16 -
17 -To install AMMO (AlgorithMic Model Optimization):
18 -
19 -```console
20 -pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com nvidia-ammo
21 -```
22 -
23 -Studies have shown that FP8 E4M3 quantization typically only minimally degrades inference accuracy. The most recent silicon
24 -offerings e.g. AMD MI300, NVIDIA Hopper or later support native hardware conversion to and from fp32, fp16, bf16, etc.
25 -Thus, LLM inference is greatly accelerated with minimal accuracy loss.
26 -
27 -Here is an example of how to enable this feature:
28 -
29 -```python
30 -# two float8_e4m3fn kv cache scaling factor files are provided under tests/fp8_kv, please refer to
31 -# https://github.com/vllm-project/vllm/blob/main/examples/other/fp8/README.md to generate kv_cache_scales.json of your own.
32 -
33 -from vllm import LLM, SamplingParams
34 -sampling_params = SamplingParams(temperature=1.3, top_p=0.8)
35 -llm = LLM(model="meta-llama/Llama-2-7b-chat-hf",
36 - kv_cache_dtype="fp8",
37 - quantization_param_path="./tests/fp8_kv/llama2-7b-fp8-kv/kv_cache_scales.json")
38 -prompt = "London is the capital of"
39 -out = llm.generate(prompt, sampling_params)[0].outputs[0].text
40 -print(out)
41 -
42 -# output w/ scaling factors: England, the United Kingdom, and one of the world's leading financial,
43 -# output w/o scaling factors: England, located in the southeastern part of the country. It is known
44 -```
docs/source/features/quantization/fp8_e5m2_kvcache.md deleted
-31
@@ -1,31 +0,0 @@
1 -(fp8-kv-cache)=
2 -
3 -# FP8 E5M2 KV Cache
4 -
5 -The int8/int4 quantization scheme requires additional scale GPU memory storage, which reduces the expected GPU memory benefits.
6 -The FP8 data format retains 2~3 mantissa bits and can convert float/fp16/bfloat16 and fp8 to each other.
7 -
8 -Here is an example of how to enable this feature:
9 -
10 -```python
11 -from vllm import LLM, SamplingParams
12 -# Sample prompts.
13 -prompts = [
14 - "Hello, my name is",
15 - "The president of the United States is",
16 - "The capital of France is",
17 - "The future of AI is",
18 -]
19 -# Create a sampling params object.
20 -sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
21 -# Create an LLM.
22 -llm = LLM(model="facebook/opt-125m", kv_cache_dtype="fp8")
23 -# Generate texts from the prompts. The output is a list of RequestOutput objects
24 -# that contain the prompt, generated text, and other information.
25 -outputs = llm.generate(prompts, sampling_params)
26 -# Print the outputs.
27 -for output in outputs:
28 - prompt = output.prompt
29 - generated_text = output.outputs[0].text
30 - print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
31 -```
docs/source/features/quantization/index.md
+1 -2
@@ -14,6 +14,5 @@ bnb
14 gguf
15 int8
16 fp8
17 -fp8_e5m2_kvcache
18 -fp8_e4m3_kvcache
17 +quantized_kvcache
18 ```
docs/source/features/quantization/quantized_kvcache.md new
+145
@@ -0,0 +1,145 @@
1 +(quantized-kvcache)=
2 +
3 +# Quantized KV Cache
4 +
5 +## FP8 KV Cache
6 +
7 +Quantizing the KV cache to FP8 reduces its memory footprint. This increases the number of tokens that can be stored in the cache, improving throughput.
8 +
9 +### FP8 Formats
10 +
11 +[OCP (Open Compute Project)](https://www.opencompute.org) specifies two common 8-bit floating point data formats:
12 +
13 +- E5M2 (5 exponent bits and 2 mantissa bits)
14 +- E4M3FN (4 exponent bits and 3 mantissa bits, often shortened as E4M3)
15 +
16 +The E4M3 format offers higher precision compared to E5M2. However, due to its small dynamic range (±240.0), E4M3 typically requires a higher-precision (FP32) scaling factor alongside each quantized tensor.
17 +
18 +### Current Limitations
19 +
20 +For now, only per-tensor (scalar) scaling factors are supported. Development is ongoing to support scaling factors of a finer granularity (e.g. per-channel).
21 +
22 +### Performance Impact
23 +
24 +The current FP8 KV cache implementation primarily benefits throughput by allowing approximately double the amount of space for KV cache allocation. This enables either:
25 +
26 +- Processing longer context lengths for individual requests, or
27 +- Handling more concurrent request batches
28 +
29 +However, there are currently no latency improvements as the implementation does not yet include fused dequantization and attention operations. Future releases will support quantized attention with hardware acceleration, which should provide additional performance benefits. While the most recent silicon offerings (e.g. AMD MI300, NVIDIA Hopper or later) support native hardware conversion between FP8 and other formats (fp32, fp16, bf16), this benefit is not yet fully realized.
30 +
31 +Studies have shown that FP8 E4M3 quantization typically only minimally degrades inference accuracy, making it a practical choice for throughput optimization.
32 +
33 +## Usage Example
34 +
35 +Here is an example of how to enable FP8 quantization:
36 +
37 +```python
38 +from vllm import LLM, SamplingParams
39 +
40 +sampling_params = SamplingParams(temperature=0.7, top_p=0.8)
41 +llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", kv_cache_dtype="fp8")
42 +prompt = "London is the capital of"
43 +out = llm.generate(prompt, sampling_params)[0].outputs[0].text
44 +print(out)
45 +
46 +# output w/ scaling factors: England, the United Kingdom, and one of the world's leading financial,
47 +# output w/o scaling factors: England, located in the southeastern part of the country. It is known
48 +```
49 +
50 +The `kv_cache_dtype` argument specifies the data type for KV cache storage:
51 +- `"auto"`: Uses the model's default "unquantized" data type
52 +- `"fp8"` or `"fp8_e4m3"`: Supported on CUDA 11.8+ and ROCm (AMD GPU)
53 +- `"fp8_e5m2"`: Supported on CUDA 11.8+
54 +
55 +## Calibrated Scales for Better Accuracy
56 +
57 +For optimal model quality when using FP8 KV Cache, we recommend using calibrated scales tuned to representative inference data. [LLM Compressor](https://github.com/vllm-project/llm-compressor/) is the recommended tool for this process.
58 +
59 +### Installation
60 +
61 +First, install the required dependencies:
62 +
63 +```console
64 +pip install llmcompressor
65 +```
66 +
67 +### Example Usage
68 +
69 +Here's a complete example using `meta-llama/Llama-3.1-8B-Instruct` (most models can use this same pattern):
70 +
71 +```python
72 +from datasets import load_dataset
73 +from transformers import AutoModelForCausalLM, AutoTokenizer
74 +from llmcompressor.transformers import oneshot
75 +
76 +# Select model and load it
77 +MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
78 +model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto", torch_dtype="auto")
79 +tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
80 +
81 +# Select calibration dataset
82 +DATASET_ID = "HuggingFaceH4/ultrachat_200k"
83 +DATASET_SPLIT = "train_sft"
84 +
85 +# Configure calibration parameters
86 +NUM_CALIBRATION_SAMPLES = 512 # 512 samples is a good starting point
87 +MAX_SEQUENCE_LENGTH = 2048
88 +
89 +# Load and preprocess dataset
90 +ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
91 +ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
92 +
93 +def process_and_tokenize(example):
94 + text = tokenizer.apply_chat_template(example["messages"], tokenize=False)
95 + return tokenizer(
96 + text,
97 + padding=False,
98 + max_length=MAX_SEQUENCE_LENGTH,
99 + truncation=True,
100 + add_special_tokens=False,
101 + )
102 +
103 +ds = ds.map(process_and_tokenize, remove_columns=ds.column_names)
104 +
105 +# Configure quantization settings
106 +recipe = """
107 +quant_stage:
108 + quant_modifiers:
109 + QuantizationModifier:
110 + kv_cache_scheme:
111 + num_bits: 8
112 + type: float
113 + strategy: tensor
114 + dynamic: false
115 + symmetric: true
116 +"""
117 +
118 +# Apply quantization
119 +oneshot(
120 + model=model,
121 + dataset=ds,
122 + recipe=recipe,
123 + max_seq_length=MAX_SEQUENCE_LENGTH,
124 + num_calibration_samples=NUM_CALIBRATION_SAMPLES,
125 +)
126 +
127 +# Save quantized model
128 +SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-KV"
129 +model.save_pretrained(SAVE_DIR, save_compressed=True)
130 +tokenizer.save_pretrained(SAVE_DIR)
131 +```
132 +
133 +The above script will create a folder in your current directory containing your quantized model (e.g., `Llama-3.1-8B-Instruct-FP8-KV`) with calibrated scales.
134 +
135 +When running the model you must specify `kv_cache_dtype="fp8"` in order to enable the kv cache quantization and use the scales.
136 +
137 +```python
138 +from vllm import LLM, SamplingParams
139 +
140 +sampling_params = SamplingParams(temperature=0.7, top_p=0.8)
141 +llm = LLM(model="Llama-3.1-8B-Instruct-FP8-KV", kv_cache_dtype="fp8")
142 +prompt = "London is the capital of"
143 +out = llm.generate(prompt, sampling_params)[0].outputs[0].text
144 +print(out)
145 +```