A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Bugfix] Fix LFM2 ShortConv prefix breaking quant ignore list (#51495)
ylsun committed 3 months ago
Add torch compile for qwen3_vl encoder (#40116)
Tianyu Guo committed 3 months ago
[ROCm][CI] Baseline legacy extensions in the Torch ABI audit (#50805)
Andreas Karatzas committed 3 months ago
[Bugfix][MM] Avoid device sync in FusedInputNorm initialization (#51435)
Andreas Karatzas committed 3 months ago
[CI] Refresh hybrid Model Runner V2 coverage (#51410)
Jiangyun Zhu committed 3 months ago
[CPU] Optimize routed FP8/MXFP4 MoE GEMM dispatch (#50949)
Tianmu Li committed 3 months ago
[Bugfix][Multimodal] Fix PyNvVideoCodec video backend returning NCHW instead of NHWC (#51076)
Junpu Yu committed 3 months ago
[BugFix] Reject invalid data-parallel RPC ports (#51469)
aoshen02 committed 3 months ago
[Bugfix][models_multimodal] Remote HF python code misses importing class (#51427)
Gyula Chinoradszki committed 3 months ago
[Bugfix][multi_modal] Fix pos_ids being unitialized for minicpmv2.6 in hf runner (#51432)
music-dino committed 3 months ago
[CI] Guard remote-code Transformers compatibility (#51451)
Andreas Karatzas committed 3 months ago
[Bugfix][Quantization] Fix INT8 W8A8 MoE crash in TritonExperts (#51411)
djramic committed 3 months ago
[Bugfix] Fix ZMQ port TOCTOU race in shm_broadcast MessageQueue (#50960)
aoshen02 committed 3 months ago
[Kimi][MM] disable kimi_vit's dynamic torch.compile for TPU (#51196)
Linkun committed 3 months ago
[Bugfix] Close usage telemetry HTTP sessions (#51219)
Nils Matteson committed 3 months ago
[Perf] Optimize DeepSeek V3.2 sequence parallelism (#51434)
Woosuk Kwon committed 3 months ago
[CI Test] Add specific unit test for mrv2 offloading (#51440)
Wentao Ye committed 3 months ago
[Perf] Improve `--linear-backend` filtering (#48735)
Andrii Skliar committed 3 months ago
[Perf] Narrow DeepSeek V3.2 eager CUDA graph region (#51425)
Woosuk Kwon committed 3 months ago
[Spec Decode] Register Qwen3.6 dSpark acceptance coverage (#51310)
Michael Goin committed 3 months ago