A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Bugfix][NIXL] Fix multiple handles xfer race (#56104)
Nicolò Lucchesi committed about 1 month ago
[CI] Pass the scale-out endpoint flag in EC E2E (#56819)
Kevin H. Luu committed about 1 month ago
[CI] Drop root privileges for the OTel GPU sampler (#56941)
Kevin H. Luu committed about 1 month ago
[CI/Build] Retry CPU image stream interruptions (#56597)
Kevin H. Luu committed about 1 month ago
[Kernel][MoE] DeepEP v2: async finalize to overlap shared experts with combine (#52781)
404mario committed about 1 month ago
[Bugfix] Avoid nested rerank cancellation handlers (#55602)
Miguel committed about 1 month ago
[Bugfix][NIXL] Report a full prefix cache hit as finished (#56640)
Yizheng Jiao committed about 1 month ago
[Model][DSv4.1] Store the whole KV in MXFP8 (FlashMLA V4.1 record) (#56893)
Yongye Zhu committed about 1 month ago
[Bugfix] Handle bare and malformed tool call openers in Gemma4 parser (#53444)
wuhangxian committed about 1 month ago
[Bugfix][KV Offload] Ignore pending chunks in invalid sliding windows (#56486)
Liqian Lin committed about 1 month ago
[Bugfix] Export weight-cache IPC tensors separately for each client (#56472)
shaohuaxi committed about 1 month ago
[KVConnector][P2P] Configurable unbound-store timeout and one-RTT rejection of a late fetch (#53453)
liranschour committed about 1 month ago
[XPU][CI] update gpt-oss package version (#56783)
liuzhenwei committed about 1 month ago
[ROCm][CI] Prepare TheRock image for CI (#56921)
Matt committed about 1 month ago
[Minor] Invert MRV1 specdec method fallback logic (#56899)
Nick Hill committed about 1 month ago
[Bugfix] Validate routed-expert prompt offsets before engine submission (#56844)
aoshen02 committed about 1 month ago
[ROCm][DSv4.1][Perf] Dequantize the MXFP8 weight once when dot_scaled cannot be used (#56560)
JohnQinAMD committed about 1 month ago
[Bugfix][Rust] Fix HF multi-turn dataset integration (#56915)
Kevin H. Luu committed about 1 month ago
[ROCm][Bugfix] Initialize Ray NIXL agents for sharded RDT (#56687)
Andreas Karatzas committed about 1 month ago
[Bugfix] Carry over queued work when materializing the dedicated stream (#56382)
zhaoguochun1995 committed about 1 month ago