A high-throughput and memory-efficient inference and serving engine for LLMs
main
Promote `prefix_cache_retention_interval` to an argument and change the default to 0 (#52216)
Tyler Michael Smith committed 2 months ago
qli88 committed 2 months ago
[K3] support recoverssm for K3 (#51855)
Jiangyun Zhu committed 2 months ago
[Bugfix][CPU] Take an attention group's query head count from its layers (#51852)
Ganesh R committed 2 months ago
fix(pooling): validate BGE-M3 combined task ownership (#51823)
Ola committed 2 months ago
[XPU] Enable Kimi K3 KDA kernel tests on XPU (#51809)
pmanczak committed 2 months ago
[CI] Shard Humming A100 eval (#52325)
Kevin H. Luu committed 2 months ago
[Doc] [ROCm] Update installation documentation (#52303)
TJian committed 2 months ago
docs: fix incorrect --custom-skip-chat-template flag reference (#52588)
Amal Sebastian committed 2 months ago
[CI/Build] Fix accident pre-commit breakage due to concurrent merge (#52578)
Isotr0py committed 2 months ago
[Bugfix][Mamba] Fix overlapping state copy race (#50729)
Andreas Karatzas committed 2 months ago
[Frontend] Consolidate entrypoint middleware (#52309)
wang.yuqi committed 2 months ago
[CI/Build] Reduce more duplicate runner startup in tests (#52570)
Isotr0py committed 2 months ago
[ROCm][CI] Avoid forcing FlashAttention in the ColPali pooling test (#52565)
Andreas Karatzas committed 2 months ago
[ROCm][CI] Use the same-build wheel in Python-only CI (#49514)
Andreas Karatzas committed 2 months ago
[Hardware][NVIDIA] Add GB10 fused-MoE fp8 tuning configs (E=256, E=512) (#52502)
pavelzak committed 2 months ago
[Bugfix][DSv4] Keep indexer scoring in breakable graphs (#52492)
Lucas Wilkinson committed 2 months ago
[Performance][MRV2] Cache logits-processing request state (#52329)
Cheng Rui committed 2 months ago
[Doc] Add MatrixHub as a model loading source (#50492)
Yiting Jiang committed 2 months ago
[Bugfix][Frontend] Guard remaining before-validators against non-object JSON bodies (#52528)
Kaif Kohari committed 2 months ago