A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Bugfix][MiniCPM-o] Fix cuda/cpu device mismatch in Resampler2_5 pos_embed (#43844)
Parth Ashwin Jain committed 5 months ago
Andreas Karatzas committed 5 months ago
fix: prevent MM cache hang from stale LRU order keys (#43595)
jeff.ye committed 5 months ago
[Bugfix][Rust Frontend] Set a structured-output backend so requests do not 500 (#44729)
Ting SUN committed 5 months ago
[ROCm][MLA][Bugfix] Reserve FP8 prefill workspace before lock for Kimi-K2.5 (#42978)
xaguilar-amd committed 5 months ago
[Bugfix] Propagate ImportError from load_audio_pyav when vllm[audio] … (#44750)
littlecircle0730 committed 5 months ago
[MM][Perf][CG] Support ViT full CUDA graph for glm4_1v image and video inference (#40576)
GuorongYe committed 5 months ago
[XPU][CI] fix test case path (#44940)
Kunshang Ji committed 5 months ago
[Bugfix][Model] Qwen3-Omni: move cu_seqlens to GPU before VIT attention (#44264)
Lanze Liu committed 5 months ago
[Docs] Remove broken link to deleted disaggregated_prefill.sh (#44929)
Lanze Liu committed 5 months ago
[WideEP] Integrate DeepEP v2 (#41183)
Tyler Michael Smith committed 5 months ago
[ROCm][CI] Re-route NixlConnector jobs (#44809)
Andreas Karatzas committed 5 months ago
[Misc] usage_stats: report more engine, spec-decode, and EP config (#44595)
Zachary Xi committed 5 months ago
[Rust Frontend] [Refactor] Refine utility call interfaces (#44856)
Bugen Zhao committed 5 months ago
[Bugfix] Canonicalize FP8 weight layout to (K, N) at the source (#44735)
Michael Goin committed 5 months ago
[Bugfix][MoE] Fix fused MoE expert mapping helper call sites (#44897)
Mohammad Miadh Angkad committed 5 months ago
[Model Runner V2] Fix mrv2 mm lora issue (#44450)
Wentao Ye committed 5 months ago
[Attention] Extract KV-cache update from CPU attention backend (#40470)
Diego Maniloff committed 5 months ago
[MoE Refactor] FusedMoE/MoERunner inversion refactor (#41184)
bnellnm committed 5 months ago
[Quantization] add online fp8 ptpc (#44132)
Walter Beller-Morales committed 5 months ago