@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 30299a41fa
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

[MISC] Remove FP8 warning (#5472)

Co-authored-by: Philipp Moritz <pcmoritz@gmail.com>

Cody Yu committed Jun 13, 2024 at 11:22 UTC 30299a41fa78c7bf485aca7ef8ad584ca340a64d
1 file changed +1 -1
vllm/config.py
+1 -1
@@ -212,7 +212,7 @@ class ModelConfig:
212 f"{self.quantization} quantization is currently not "
213 f"supported in ROCm.")
214 if (self.quantization
215 - not in ["marlin", "gptq_marlin_24", "gptq_marlin"]):
215 + not in ("fp8", "marlin", "gptq_marlin_24", "gptq_marlin")):
216 logger.warning(
217 "%s quantization is not fully "
218 "optimized yet. The speed can be slower than "