A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 6, 05:45 UTC
update doc for online fp8 quantization (#37851)
Signed-off-by: Yan Ma <yan.ma@intel.com>
Yan Ma committed
Mar 23, 2026 at 13:19 UTC
d3fe85713508a69144ec610aa13253de34896338
1 file changed
-3
docs/features/quantization/fp8.md
-3
@@ -137,6 +137,3 @@ llm = LLM("facebook/opt-125m", quantization="fp8")
137
result = llm.generate("Hello, my name is")
138
print(result[0].outputs[0].text)
139
```
140
-
141
-!!! warning
142
- Currently, we load the model at original precision before quantizing down to 8-bits, so you need enough memory to load the whole model.