A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
Update bnb.md with example for OpenAI (#11718)
Alberto Ferrer committed
Jan 4, 2025 at 00:29 UTC
d1d49397e7f8d1ac472d763dae395b67fdda1ef8
1 file changed
+7
docs/source/quantization/bnb.md
+7
@@ -37,3 +37,10 @@ model_id = "huggyllama/llama-7b"
37
llm = LLM(model=model_id, dtype=torch.bfloat16, trust_remote_code=True, \
38
quantization="bitsandbytes", load_format="bitsandbytes")
39
```
40
+## OpenAI Compatible Server
41
+
42
+Append the following to your 4bit model arguments:
43
+
44
+```
45
+--quantization bitsandbytes --load-format bitsandbytes
46
+```