A high-throughput and memory-efficient inference and serving engine for LLMs
main
[ROCm][CI] Adding metadata (#47477)
Andreas Karatzas committed 4 months ago
Fix Transformers modeling backend usage stats (#47472)
Harry Mellor committed 4 months ago
Dan Blanaru committed 4 months ago
Delete PagedAttention (#47361)
Michael Goin committed 4 months ago
[Bugfix][Tool Parser] poolside_v1: accept tool calls without newline after function name (#47311)
Joe Rowell committed 4 months ago
[BugFix][MRV2] Ensure all req slots are accounted for when scheduling (#46974)
Nick Hill committed 4 months ago
[Bugfix] Fix pooled Whisper encoder sliding-window kernel size (#47437)
Nick Hill committed 4 months ago
[Model Runner V2][Perf] Warm up GLM-5.2 DSA indexer prefill metadata kernel (#47285)
Chauncey committed 4 months ago
[Rust Frontend] Improve scheduler stats logging parity (#47435)
Bugen Zhao committed 4 months ago
support GLM-5.2 gate use FP32 (#47410)
Yuxuan Zhang committed 4 months ago
Woosuk Kwon committed 4 months ago
[ModelRunner V2] Fix Mamba2 crash on non-spec-decode (#47428)
Nick Hill committed 4 months ago
[ModelRunner V2] Enable by default for all dense models (#44443)
Wentao Ye committed 4 months ago
[ROCm] [PyTorch] Move to stable abi since ROCm upgraded to torch 2.11 (#47128)
TJian committed 4 months ago
Update DeepGEMM tag to point to latest nv-dev branch for sm120 support (#47304)
Michael Goin committed 4 months ago
[Rust Frontend] Use enum-backed domain types for engine outputs and structured outputs (#47283)
Bugen Zhao committed 4 months ago
feat(rust): expose profiler control routes in Rust frontend (#46306)
Pranav Thakur committed 4 months ago
[Model] Add LLaVA-OneVision-2 (LlavaOnevision2ForConditionalGeneration) (#44785)
chengzheng345 committed 4 months ago
[Feature] Universal speculative decoding for heterogeneous vocabularies (TLI) (#38174)
Wonderful committed 4 months ago
[XPU][CI] Split test_punica_ops into separate pytest invocations for stability (#47376)
Chaojun Zhang committed 4 months ago