A high-throughput and memory-efficient inference and serving engine for LLMs
main
[XPU] Honor device_ids for worker device placement (#56015)
Chaojun Zhang committed about 1 month ago
[Bugfix][Rust Frontend] Tolerate NaN-corrupted logprobs in engine-core (#51026)
Islam committed about 1 month ago
[XPU] Route to fused_qk_rmsnorm_rope_gate triton kernel (#56096)
Marceli Fylcek committed about 1 month ago
[XPU] fix_test_worker_memory_snapshot_for_xpu (#50097)
Qiming Zhang committed about 1 month ago
[CI/Build][Rust Frontend] Retire Buf schema publishing (#57046)
JulienDarve committed about 1 month ago
[Bugfix][ROCm] Fix MiniMax-M3 fused MXFP8 block scale (#53674)
maithilijoshi20 committed about 1 month ago
[Rust Frontend] Support HF config overrides `--hf-overrides` (#56931)
Bugen Zhao committed about 1 month ago
[Rust Frontend][Metrics] Add per-request preemption histogram (#57033)
Reid committed about 1 month ago
[Doc] Show how to get Responses prompt token IDs (#55912)
Francisco Javier Arceo committed about 1 month ago
[Misc] Allow install empty package from pip (#41074)
wangxiyuan committed about 1 month ago
[ROCm][Kimi-K3] Enable a4w4 flydsl kernels for KimiK3 (#53940)
ppalanga committed about 1 month ago
[ROCm] [Bugfix] Enable Load and Inference of GLM-5.3-Flash Quark MXFP4 Checkpoint (#56176)
Colin Z committed about 1 month ago
[Pooling] Cap max-length padding for chunked embeddings (#56505)
Taneem Ibrahim committed about 1 month ago
Giancarlo Delfin committed about 1 month ago
[Build][NVIDIA] Update public Rubin dependencies and MSA compatibility (#56545)
Shang Wang committed about 1 month ago
[KVConnector][NIXL] Support attention-HMA layouts in pipeline-parallel push prefill (#50494)
qizixi committed about 1 month ago
[Perf][Kernel] Add sampled filtering for persistent top-k (#56346)
Misha Goin committed about 1 month ago
[DSA] Wire DeepGEMM sparse MQA logits into the DeepSeek V4.1 indexer (#56254)
Jared Wen committed about 1 month ago
[MRV2] Buffer util simplifications (#56888)
Nick Hill committed about 1 month ago
[Bugfix] Format kernel-import errors eagerly so warning_once does not retain them (#54098)
zwang86 committed about 1 month ago