A high-throughput and memory-efficient inference and serving engine for LLMs
main
[ROCm][Perf] Insert MiniMax-M3 sparse-PA K/V without a contiguous copy (#56849)
akii96 committed about 1 month ago
[Bugfix] Use DP index for dense DP weight updates and EC CPU region (#56950)
Alexander Lee committed about 1 month ago
[Bugfix][Model Runner V2] Route dummy tokens to MoE experts during profiling (#57270)
Robert Shaw committed about 1 month ago
[Bugfix][MoE] Convert FlashInfer BF16 weights in place (#54699)
Yuchen Wang committed about 1 month ago
[HARDWARE][POWER] Enable W8A8 INT8 MoE on POWER (#55316)
Rukhaiya2004 committed about 1 month ago
Shantipriya Parida committed about 1 month ago
[BugFxi] Fix DeepGEMM FP8 workspace over allocation (#53914)
Lucas Wilkinson committed about 1 month ago
[Bugfix] Escape control characters in xgrammar choice grammar (#48115)
CaiJohn committed about 1 month ago
[Frontend] Only show the summary line of config docstrings in `--help` (#57357)
Harry Mellor committed about 1 month ago
[ROCm][CI] Fix AMD CI pipeline upload rejected by an invalid block-step key (#57375)
Matt committed about 1 month ago
[CI] Make ci-clean-log.sh portable to macOS/BSD sed (#57367)
Kevin H. Luu committed about 1 month ago
[Bugfix] Fall back to native sampling when FlashInfer cannot target the GPU (#48956)
Martín el Cheikh committed about 1 month ago
Fix Laguna patch mutating flat RoPE parameters (#57189)
Laura Lopez-Real committed about 1 month ago
[Compile] Fix compile warning #177-D (#56609)
Wentao Ye committed about 1 month ago
[Bugfix][ROCm][GLM-5.3-Flash] Apply deferred tilelang.jit already on attribute access (#57192)
Simon Danielsson committed about 1 month ago
[XPU][CI] skip test_hybrid_prefix_cache_hit_rate (#57301)
Yan Ma committed about 1 month ago
[Bugfix] Honor skip_reading_prefix_cache for KV connector hits (#57269)
aoshen02 committed about 1 month ago
[Perf][EPD] Batch image requests per encoder (#57095)
Tianyu Guo committed about 1 month ago
[CI] Raise DSv4-Flash disaggregated engine readiness timeout to 1800s (#57334)
Kevin H. Luu committed about 1 month ago
[CI] Raise H200 LM Eval Large Models timeout to 120 min (#57335)
Kevin H. Luu committed about 1 month ago