A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Test][Qwen3-VL] Cover compiled DeepStack input contract (#53529)
maithilijoshi20 committed about 2 months ago
[KV Offload] Preserve KV event metadata until final residency removal (#52068)
Moein Khazraee committed about 2 months ago
[Frontend] Move engine/protocol.py out openai folder (#54492)
wang.yuqi committed about 2 months ago
[Bugfix][KVOffload] P2P tier declares REQUEST_LEVEL on the producer leg (#52912)
liranschour committed about 2 months ago
[Bugfix][SM120] DSv4: pass contiguous C128A decode topk indices on SM120 (#53574)
Gabriel Wu committed about 2 months ago
[Bugfix][Frontend] Truncate prompt_is_token_ids with the prompt (#54509)
Hotragn Pettugani committed about 2 months ago
[Bugfix][MM] Fix MiniCPM-o image processor reuse on Transformers v5 (#54501)
Andreas Karatzas committed about 2 months ago
[Renderer] Shutdown the renderer properly. (#52124)
wang.yuqi committed about 2 months ago
[Quantization][Autoround][XPU] Support AutoRound MXFP8 MoE models (#51248)
jl9876 committed about 2 months ago
[CPU] Support FP16/BF16 persisted GDN state on AMX (#52191)
Tianmu Li committed about 2 months ago
[XPU] Ensure unquantized linear weight is N-contiguous (#53536)
zofia committed about 2 months ago
[Bugfix][CPU] Fix several bugs (#54042)
Li, Jiang committed about 2 months ago
[Bugfix][DCP] Fix NVIDIA DeepSeek-V3.2 / GLM-5.2 fused attention (#50005)
Yunxiao Ning committed about 2 months ago
[KV Offload] Forward ownership in KV cache events (#52067)
Moein Khazraee committed about 2 months ago
[Kernel] Retire the DSv3 router GEMM CUDA kernel (#54040)
Jee Jee Li committed about 2 months ago
[Performance] Optimize Dots3 NOTE runtime (#53517)
范裕达 committed about 2 months ago
[Bugfix][Multimodal] Honor modality-scoped mm_processor_kwargs (#53808)
Prudhvi Vuda committed about 2 months ago
[Frontend] Forward cache salt for content parts (#54315)
eligotts committed about 2 months ago
[Bugfix][Frontend] Truncate prompt_token_offsets with the prompt (#54407)
Hotragn Pettugani committed about 2 months ago
[Model] Support Qwen3.8-Flash-Next (#53896)
Harry Huang committed about 2 months ago