A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Model] Honor cap_pixels_per_frame in Qwen3-VL memory profiling (#54380)
Devon Krisman committed about 2 months ago
[Bugfix] Reset cached Mamba align metadata on profiling teardown (#54044)
Canlin Guo committed about 2 months ago
[CI] Add explicit step keys to 18 hardware test steps (#54330)
Kevin H. Luu committed about 2 months ago
[Bugfix] Bound cache_salt length to prevent DoS via scheduler CPU exhaustion (#54353)
Juan Pérez de Algaba committed about 2 months ago
[Fix] Improve ROCm detection in WSL environments (#38434)
Yizhou committed about 2 months ago
[Rust Frontend][gRPC] Add audio and video media inputs (#53760)
Connor Carpenter committed about 2 months ago
[Bugfix][Multimodal] Release Qwen2.5-VL and Qwen3-VL RoPE caches with the model (#54346)
waizuichougou committed about 2 months ago
[ROCm][MLA] Reach FULL cudagraphs for AITER MLA speculative decoding (#51171)
Matt Qin committed about 2 months ago
[Bugfix][MoE] Enable cuBLAS out_dtype router GEMM on all CUDA archs (fixes family-120/GB10) (#54048)
hcl committed about 2 months ago
[CI][Ray] Fix flaky multi-node assignment test after placement-group teardown (#53621)
yzong-rh committed about 2 months ago
[CI][Test] Deflake test_mem.py sleep-mode asserts via allocator bookeeping (#54312)
Oxana Korzh committed about 2 months ago
Giancarlo Delfin committed about 2 months ago
[Bugfix][Frontend] Only echo the assistant turn in batched chat completions (#52529)
Kaif Kohari committed about 2 months ago
[Bugfix][Frontend] Validate stop_token_ids against vocab size (#54196)
jack committed about 2 months ago
[Perf] Reuse topk SparseMatrix routing metadata in GPT-OSS MoE forward (#45457)
Shenglei Fu committed about 2 months ago
[Nixl][PD] DCP support for MLA models (#50611)
Nicolò Lucchesi committed about 2 months ago
[Multimodal] Deprecate PyAV video decoder backend (#54231)
Isotr0py committed about 2 months ago
[Mypy] Fix typing for J models (#54130)
Taneem Ibrahim committed about 2 months ago
[ROCm][CI] Stage E gating (#50920)
Andreas Karatzas committed about 2 months ago
[Bugfix][Spec Decode] Capture the widest uniform decode batch by default (#50488)
Rahul Chalamala committed about 2 months ago