A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Model][Engram] Share host tables across co-located DP replicas by default (#57651)
Juntian Liu committed about 1 month ago
[Perf][ROCm] Add a ROCm path for Hy4 and compile the backbone (#57526)
akii96 committed about 1 month ago
[Bugfix][DSV4.1] Restrict mHC overlap to full CUDA graphs (#57874)
Woosuk Kwon committed about 1 month ago
[Perf][Attention] Avoid redundant sparse attention metadata operations (#57885)
Woosuk Kwon committed about 1 month ago
[Bugfix] Fix xgrammar feature gate bypass when JSON Schema type is a list (#48416)
CaiJohn committed about 1 month ago
[Bugfix][Model] Mistral3: align placeholder grid with public processor (#53758)
Oliver Holworthy committed about 1 month ago
[Bugfix] Fix Whisper engine crash on audio clips longer than 30s (#57769)
Tianyao Wu committed about 1 month ago
[Tests] Release HF embedding weights after prompt extraction (#57868)
Andreas Karatzas committed about 1 month ago
[MRV2][XPU] use xpu sample kernel in mrv2 sampler (#57277)
liuzhenwei committed about 1 month ago
[ROCm][Tests] Avoid pooling cleanup waits against live shared engines (#57864)
Andreas Karatzas committed about 1 month ago
[XPU][CI] fallback to 7-args of moe_align_block_size (#57855)
liuzhenwei committed about 1 month ago
[Core][KDA] Generalize Mamba prefill checkpoint builder and exporter (#57783)
Chauncey committed about 1 month ago
[Bugfix] DiffusionGemma: fix multimodal support (#57589)
Matt Mastracci committed about 1 month ago
[Perf][DSV4.1] Fuse TP all-reduce with mHC input preparation (#57643)
Woosuk Kwon committed about 1 month ago
[Build] Fix DeepGEMM CUDA 12.9 release builds (#57554)
Kevin H. Luu committed about 1 month ago
[BUGFIX][HY4] Record indexer completion event for full CUDA graph capture (#57811)
Jee Jee Li committed about 1 month ago
[Profiler] Unify platform-aware torch profiling (#57460)
czhu-cohere committed about 1 month ago
[ROCm][DSv4.1] Keep the Engram tables in host memory on ROCm (#57491)
JohnQinAMD committed about 1 month ago
[Doc] Refresh Kthena integration guide (#57314)
Zhou Zihang committed about 1 month ago
[Bugfix][KV Connector] Propagate cache reset failure during sleep (#54581)
CrazyCoder committed about 1 month ago