A high-throughput and memory-efficient inference and serving engine for LLMs
main
[CI/Build] Add agents auto-label rule (#59641)
Misha Goin committed 20 days ago
[Bugfix][HiSparse] Fix a chunked-prefill preemption livelock (#59494)
Matthew Bonanni committed 20 days ago
[ROCm][Perf] Parallelise AITER MLA page-index expansion over token chunks (#57978)
fululi12 committed 20 days ago
[Agents] Expose PR checklist skill to Claude (#59638)
Misha Goin committed 20 days ago
[Bugfix][ROCm] Use a zero default for masked scales in the MXFP8 GEMM (#59454)
Mehmet Cagri committed 20 days ago
[Bugfix][Frontend] Check reused prompt token ids against the vocab before streaming (#59555)
Shijie Lyu committed 20 days ago
[Bugfix] Tie lm_head.weight for Nemotron Parse when checkpoint omits it (#53020)
aniskumar-nv committed 20 days ago
[ROCm][CI] Drop four no-GPU CPU groups from the legacy AMD pipeline (#59595)
stefankoncarevic committed 20 days ago
[Bugfix][MRV2] Keep GDN prefill checkpoint metadata local to each cache group (#59536)
ai-jz committed 20 days ago
Maroon Ayoub committed 20 days ago
[Rust][Benchmark] Warn when temperature is left to the server default (#59247)
Idder Ghanbaja committed 20 days ago
[CI/Build] Add mooncake auto-label rule and assign topic owners (#59192)
Nicolò Lucchesi committed 20 days ago
[Core] Remove deprecated mamba_cache_mode "all" (#58997)
Jiangyun Zhu committed 20 days ago
[Bugfix] Gate Kimi-K3 KDA warmup on sys.modules to skip Kimi import for non-Kimi models (#59257)
YueshenZ committed 20 days ago
[Docs] Remove references to removed env vars (#59530)
Sundri Lai committed 20 days ago
[Rust Frontend] Add `hf` parser for response templates (#59005)
Bugen Zhao committed 20 days ago
[Core][Logging] Add built-in JSON formatter (#58739)
Mark McLoughlin committed 20 days ago
[Bugfix][Spec Decode] Per-module LM heads for multi-layer MTP on Model Runner V2 (#58921)
Jiangyun Zhu committed 20 days ago
[Model Runner V2] Support randomized dummy inputs (#58411)
Robert Shaw committed 20 days ago
[CPU][Zen] Pass f32 weight scales to the zentorch INT8 MoE (#59434)
Ganesh R committed 20 days ago