@bartowski / Qwen2.5-3B-Instruct-GGUF

GGUF quants of Qwen2.5-3B-Instruct for llama.cpp.

Text Generation gguf License: apache-2.0 text-generation gguf quantized qwen2.5

Qwen2.5-3B-Instruct-GGUF

GGUF quantizations of Qwen/Qwen2.5-3B-Instruct, produced with llama.cpp. Pick a quant that fits your hardware — higher bits mean better quality, lower bits mean a smaller file and faster inference.

Quant table

File Quant Size Notes
Qwen2.5-3B-Instruct-Q8_0.gguf Q8_0 3.3 GB Near-lossless, recommended for quality
Qwen2.5-3B-Instruct-Q5_K_M.gguf Q5_K_M 2.2 GB Balanced quality / size
Qwen2.5-3B-Instruct-Q4_K_M.gguf Q4_K_M 1.9 GB Good default for most setups
Qwen2.5-3B-Instruct-Q3_K_M.gguf Q3_K_M 1.6 GB Smallest, some quality loss

Run it

llama-cli -hf bartowski/Qwen2.5-3B-Instruct-GGUF -p "Hello!"

Prompt format

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant