A high-throughput and memory-efficient inference and serving engine for LLMs
main
set AttentionCGSupport.UNIFORM_BATCH for fa2 on xpu (#46555)
Xinyu Chen committed 4 months ago
[Perf][Multimodal] Avoid building a full timestamps list in video frame sampling (#46543)
Lynn committed 4 months ago
[Attention] Add FLASH_ATTN_MLA_SPARSE backend for Hopper sparse MLA (#46189)
Dakai An committed 4 months ago
[Model][MiniMax-M3] Add pipeline parallelism support (#45810)
soaringk committed 4 months ago
[Model] Remove BaiChuanForCausalLM and BaichuanForCausalLM (#46362)
Tiezhen WANG committed 4 months ago
[CI/Build] Fix topk histogram build on SM75 (#46550)
Mohammad Miadh Angkad committed 4 months ago
[ROCm][CI] Skip the MoE Marlin tile-padding helper assertion (#46580)
Andreas Karatzas committed 4 months ago
feat: support to OpenMOSS-Team (#44124)
hurukawa committed 4 months ago
[MoE] Free unused MXFP4 scales in OAI Triton Backend (#46549)
Woosuk Kwon committed 4 months ago
[Kernel] GLM5 Router GEMM (#46385)
Jee Jee Li committed 4 months ago
[Bugfix][Spec Decode] Fix probabilistic sampling for parallel drafting (#45956)
Benjamin Chislett committed 4 months ago
[Bugfix] Fix NemotronLayerNorm1P hardcoded cuda device type (#46495)
Michał Ganczarenko committed 4 months ago
Upgrade tpu-inference to v0.23.0 (#46568)
Sting Lin committed 4 months ago
[Mooncake] Optimize lookup pool key string construction (#46188)
Wei Zhao committed 4 months ago
[Spec Decode] Reject placeholder (-1) draft tokens in rejection sampler (#46533)
Nick Hill committed 4 months ago
[Rust Frontend] Pass effective `reasoning_parser_kwargs` for structured output (#46360)
Bugen Zhao committed 4 months ago
Bugen Zhao committed 4 months ago
[Core][DP] Throttle prefills based on local prefill work (#46532)
Nick Hill committed 4 months ago
[ROCm][Bugfix] Fix `use_v2_model_runner` inside Ray driver thread (#45998)
Micah Williamson committed 4 months ago
[ROCm][CI] Stage C-II of gating additional test groups (#46537)
Andreas Karatzas committed 4 months ago