A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Model Runner V2] Enable encoder token embedding (#50574)
Taneem Ibrahim committed 3 months ago
(feat): optionally disable lookup on PD decode (#50498)
Jeff (Junze) Ma committed 3 months ago
[Frontend] Add cache_salt support to Anthropic Messages API (#49498)
Cao Qian committed 3 months ago
[ROCm][MLA] Mask the AITER MLA small-head verify flatten causally (#50476)
Matt Qin committed 3 months ago
Add Understanding the Latency Metrics docs (#50600)
Michael Goin committed 3 months ago
[Frontend] Cohere chat v2 api support (#47189)
andrewbcohere committed 3 months ago
[Bugfix][Parser] Forward model_config to nested reasoning parsers (#50642)
Chauncey committed 3 months ago
Add @shen-shanshan to CODEOWNERS (#50655)
Shanshan Shen committed 3 months ago
[Bugfix][CI] Prevent common ops imports from initializing CUDA (#50639)
Andreas Karatzas committed 3 months ago
[Rust][Benchmark] Prevent invalid token IDs in random benchmarks (#50058)
Reid committed 3 months ago
[Build] Update pin to build ABI stable FA2 (#50474)
Jane (Yuan) Xu committed 3 months ago
[Bugfix][Test] Fix monolithic routing replay test buffer capacity (#50640)
Amir Samani committed 3 months ago
[CI] Organize speculative decoding E2E tests by coverage (#50330)
Michael Goin committed 3 months ago
[Kernel][Helion] Add numerics checks to benchmark script (#48968)
Shangdi Yu committed 3 months ago
Enable ModelOpt FP8 emulation on SM80 (#50019)
Mike G committed 3 months ago
bnellnm committed 3 months ago
[graceful shutdown] fix http server start firstly before app signal handler register (#49668)
Ning Xie committed 3 months ago
[UX] Reduce startup log noise (#50590)
Michael Goin committed 3 months ago
[Bugfix][Responses] Add tests for Chat Completions Responses API Render Parity (#50334)
yzong-rh committed 3 months ago
[ROCm][ViT] Detect Triton-AMD kernels at their new aiter location (#40289)
Adrian committed 3 months ago