A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Bugfix][Frontend] Truncate pooling prompts before padding them (#54364)
Hotragn Pettugani committed about 2 months ago
[CPU] add CPU support for Voxtral (#53921)
LironKesem committed about 2 months ago
[CI/Build] Fix Kimi K3 Eagle3 test fixture (#54482)
Jee Jee Li committed about 2 months ago
[Frontend] Add video embeds input support (#54242)
Isotr0py committed about 2 months ago
[Bugfix][MLA] Fix BLHNC addressing for FlashInfer sparse MLA (#54465)
Lucas Wilkinson committed about 2 months ago
[Bugfix] Make metadata send non-blocking in GroupCoordinator.isend_tensor_dict (#49274)
AI-Infra Rookie committed about 2 months ago
[Flashinfer] Upgrade Flashinfer version to 0.6.18 (#54313)
Wei Zhao committed about 2 months ago
[CI] Restore gpu_1_queue routing for torch-abi audit (#54468)
Kevin H. Luu committed about 2 months ago
[Bugfix][Spec Decode] Keep default CUDA graph sizes memory-safe (#54418)
Kevin H. Luu committed about 2 months ago
[Bugfix][Quantization][MoE] Route weight only NVFP4 checkpoints through W4A16 (#54427)
Aditya Jha committed about 2 months ago
[codeowners] Add jperezdealgaba to security file ownership (#54358)
Juan Pérez de Algaba committed about 2 months ago
[Bugfix][Multimodal] Avoid caching full prompts in fallback (#54439)
waizuichougou committed about 2 months ago
[Bugfix][Kernel] Keep packed GDN decode beta in FP32 (#53877)
Yongji Wu committed about 2 months ago
[Test][VLM] Add batch-invariance tests for Qwen3-VL (#53531)
machero committed about 2 months ago
[Bugfix] Validate scale-out transfer params (#54324)
Taneem Ibrahim committed about 2 months ago
[Doc] Fix griffe warnings in HYV4 tool parser (#54412)
Harry Mellor committed about 2 months ago
[CI] Mark L4 GPU test steps with device: l4 for EKS migration (#54326)
Kevin H. Luu committed about 2 months ago
ci: add MIG slice size to H200 job labels (#54420)
Kevin H. Luu committed about 2 months ago
[Bugfix] Avoid global config lookup in sparse indexer forward (#54400)
Jiangyun Zhu committed about 2 months ago
[Perf][ROCm] Dual-stream decode with hipgraphs (#52033)
Simon Danielsson committed about 2 months ago