A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Perf] Kimi K3 nvfp4 Align in_proj weights by 128 to avoid elementwise copy (#55242)
Wei Zhao committed about 2 months ago
[Bugfix][Quantization] Register Quark per-block FP8 scales as weight_scale (#54770)
jimmy-adams committed about 2 months ago
[Perf][GDN] Build cudagraph-capture metadata without a device sync (#55404)
Nick Hill committed about 2 months ago
[Perf] Ensure async h2d copies are pinned in more places (#55202)
Nick Hill committed about 2 months ago
[Bugfix] Preserve Mamba state for padded prompt tails (#55178)
Nathan Sala committed about 2 months ago
[Bugfix][NIXL] Don't assert when a failed transfer is cleaned up twice (#54518)
Yizheng Jiao committed about 2 months ago
[Bugfix][V2] Warm up kernels before capturing CUDA graphs (#55341)
aoshen02 committed about 2 months ago
Misha Goin committed about 2 months ago
[1/2][Model Runner V2] DBO support, eager mode (#50945)
Anjie Hou committed about 2 months ago
[Mypy] Fix mypy typing for P models (#54169)
Taneem Ibrahim committed about 2 months ago
Zihan Zhang committed about 2 months ago
[Perf] Read VidCom2 frame budgets once (#55331)
Sun committed about 2 months ago
[Bugfix] Reject 0 or non-positive max concurrency (#54887)
Taneem Ibrahim committed about 2 months ago
[Bugfix][Spec Decode] Honour the draft's attention_backend on Model Runner V2 (#54826)
Stefano Castagnetta committed about 2 months ago
[Quantization] Support online quantization with partially pre-quantized checkpoints (#51392)
fxmarty-amd committed about 2 months ago
[CI] Surface why the CRCR nightly report cannot read its Buildkite secret (#54860)
Andrey Talman committed about 2 months ago
[CI] Raise AMD Spec Decode Eagle 1 job timeout to 35min (#55136)
Jared Wen committed about 2 months ago
[Bugfix][MLA] Restore DSpark cache-group capability under optimized Python (#55234)
Summer Yang committed about 2 months ago
[Bug-fix] Fix MoE fused sum row offsets (#50220)
yzyyzyhhh committed about 2 months ago
[Rust Frontend][gRPC] Preserve multimodal metadata for remote-prefill decode (#54814)
Connor Carpenter committed about 2 months ago