A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Perf] Avoid more h2d copies from non-pinned tensors (#54660)
Nick Hill committed about 2 months ago
[Kimi-K3] Overlap low-M TP8 KDA projections (#54697)
Yongye Zhu committed about 2 months ago
[Bugfix][KV Offloading] Fix eager SimpleCPUOffload cache registration and final flush (#53532)
maithilijoshi20 committed about 2 months ago
[Bugfix] Fix ColQwen3.5 pooler projector initialization (#54847)
Divakar Verma committed about 2 months ago
[Bugfix] Handle padded routes in CUTLASS MoE permutations (#54747)
Kevin H. Luu committed about 2 months ago
[Qwen3.8-Flash-Next] Separate prefill and decode paths for QSA indexer (#54513)
Thien Tran committed about 2 months ago
[CI/Build] Gate PR title check on ready PRs & use slim runners (#54827)
Bugen Zhao committed about 2 months ago
[ROCm][CI] Handle tied experts in softplus sqrt top-k test (#52679)
Andreas Karatzas committed about 2 months ago
[CI] Shard long kernel test groups (#54754)
Kevin H. Luu committed about 2 months ago
[CI] Shard distributed model jobs above the 24h P90 threshold (#54752)
Kevin H. Luu committed about 2 months ago
[CI] Split nightly MTP acceptance tests (#52353)
Kevin H. Luu committed about 2 months ago
[Rust Frontend] Attribute decoded text to tokens (#52910)
Bugen Zhao committed about 2 months ago
[Kimi-K3][Perf] Prefetch ll_bf16 router weights for M=1 (#53524)
mingg26 committed about 2 months ago
[CI] Remove MRV2-specific tests (#54823)
Nick Hill committed about 2 months ago
[Kimi Bug] Fix `cannot access local variable 'active_non_spec_mask_cpu'` (#54781)
Wentao Ye committed about 2 months ago
Micah Williamson committed about 2 months ago
[Bugfix][KV Connector] Fix DecodeBench DCP block selection (#54679)
Jeff (Junze) Ma committed about 2 months ago
[CI] Read the CRCR report token from a Buildkite secret (#54605)
Andrey Talman committed about 2 months ago
[ModelOpt] Redesign the LinearMethod classes using the generic QuantKey-driven method (#49381)
Juhi Mittal committed about 2 months ago
[CI] Disable CUDA graphs for GLM PCP evals (#54745)
Kevin H. Luu committed about 2 months ago