A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 3, 09:15 UTC
[Docs] update attribution to reflect EDEN foundation (#41666)
Signed-off-by: amitport <1131991+amitport@users.noreply.github.com>
Amit Portnoy committed
May 18, 2026 at 21:22 UTC
ce88f01c9ac4fcde9dd43a983074d4e893cde65d
2 files changed
+31
-20
vllm/model_executor/layers/quantization/turboquant/__init__.py
+12
-7
@@ -5,15 +5,20 @@
5
Hadamard rotation + per-coordinate Lloyd-Max scalar quantization for
6
keys, uniform quantization for values.
7
8
-The technique implemented here consists of the scalar case of the HIGGS
8
+The core algorithmic pattern implemented for key quantization (Hadamard
9
+rotation followed by deterministic scalar quantization and
10
+re-normalization) was originally established in DRIVE (Vargaftik et al.,
11
+NeurIPS 2021) and EDEN (Vargaftik et al., ICML 2022). This formulation is
12
+also mathematically equivalent to the scalar case of the HIGGS
13
quantization method (Malinovskii et al., "Pushing the Limits of Large
14
Language Model Quantization via the Linearity Theorem", NAACL 2025;
11
-preprint arXiv:2411.17525): rotation + optimized grid + optional
12
-re-normalization, applied to KV cache compression. A first application
13
-of this approach to KV-cache compression is in "Cache Me If You Must:
14
-Adaptive Key-Value Quantization for Large Language Models" (Shutova
15
-et al., ICML 2025; preprint arXiv:2501.19392). Both these references
16
-pre-date the TurboQuant paper (Zandieh et al., ICLR 2026).
15
+preprint arXiv:2411.17525), which subsequently generalized these concepts.
16
+
17
+A first application of this approach to KV-cache compression is in "Cache
18
+Me If You Must: Adaptive Key-Value Quantization for Large Language Models"
19
+(Shutova et al., ICML 2025; preprint arXiv:2501.19392). All of these
20
+foundational and application references pre-date the TurboQuant paper
21
+(Zandieh et al., ICLR 2026).
22
"""
23
24
from vllm.model_executor.layers.quantization.turboquant.config import TurboQuantConfig
vllm/model_executor/layers/quantization/turboquant/config.py
+19
-13
@@ -48,19 +48,25 @@ class TurboQuantConfig:
48
Applies Hadamard rotation followed by per-coordinate Lloyd-Max scalar
49
quantization for keys, and uniform quantization for values.
50
51
- Historical note: this is the scalar case of the HIGGS quantization
52
- method (Malinovskii et al., "Pushing the Limits of Large Language Model
53
- Quantization via the Linearity Theorem", NAACL 2025; preprint
54
- arXiv:2411.17525): rotation + optimized grid + optional re-normalization,
55
- applied to KV cache compression. A first application of this approach to
56
- KV-cache compression is in "Cache Me If You Must: Adaptive Key-Value
57
- Quantization for Large Language Models" (Shutova et al., ICML 2025;
58
- preprint arXiv:2501.19392). Both these references pre-date the
59
- TurboQuant paper.
60
-
61
- QJL is intentionally omitted — community consensus (5+ independent
62
- groups) found it hurts attention quality by amplifying variance through
63
- softmax.
51
+ Historical note: the core algorithmic pattern implemented for key
52
+ quantization (Hadamard rotation followed by deterministic scalar
53
+ quantization and re-normalization) was originally established in DRIVE
54
+ (Vargaftik et al., NeurIPS 2021) and EDEN (Vargaftik et al., ICML
55
+ 2022). This formulation is also mathematically equivalent to the
56
+ scalar case of the HIGGS quantization method (Malinovskii et al.,
57
+ "Pushing the Limits of Large Language Model Quantization via the
58
+ Linearity Theorem", NAACL 2025; preprint arXiv:2411.17525), which
59
+ subsequently generalized these concepts.
60
+
61
+ A first application of this approach to KV-cache compression is in
62
+ "Cache Me If You Must: Adaptive Key-Value Quantization for Large
63
+ Language Models" (Shutova et al., ICML 2025; preprint
64
+ arXiv:2501.19392). All of these foundational and application
65
+ references pre-date the TurboQuant paper (Zandieh et al., ICLR 2026).
66
+
67
+ QJL is intentionally omitted: community consensus (5+ independent
68
+ groups) found it hurts attention quality by amplifying variance
69
+ through softmax.
70
71
Named presets (use via --kv-cache-dtype):
72
turboquant_k8v4: FP8 keys + 4-bit values, 2.6x, +1.17% PPL