A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Core][BugFix] Tag prefix-cache extra keys by source (#51899)
Clinton Thomas committed 22 days ago
[XPU][CI]Skip test_abort_timeout_on_prefiller in nightly (#58307)
xiangdong committed 22 days ago
[Bugfix][Responses API] Build streamed final response from streamed items (#59307)
Flora Feng committed 22 days ago
[Kernel][Perf] Add TP=2/4/8 per-rank shapes to the sm_120 batch-invariant matmul table (#58495)
Lio Einaudi committed 22 days ago
[Bugfix] Reject a prefix_match_unit that a single KV cache group cannot honor (#58021)
Harshil committed 22 days ago
[Security] Bump nltk, aiohttp, pillow, and datamodel-code-generator (#59249)
Juan Pérez de Algaba committed 22 days ago
[Bugfix][Responses API] Preserve built-in tool output call IDs (#55596)
RongJie G committed 22 days ago
drakosha committed 22 days ago
[WideEP] Change DeepEPv2 to auto select hybrid mode by default (#57991)
Tyler Michael Smith committed 22 days ago
[Bugfix][Frontend] Honor parallel_tool_calls=false in the Responses API (#59298)
Flora Feng committed 22 days ago
[Core] Combine per-engine utility results like the Rust client (#59240)
aoshen02 committed 22 days ago
[Bugfix] Route Step3p5 forced tool choices through XML parser (#51810)
cherry77-cloud committed 22 days ago
[Bugfix][Frontend] Enforce parallel_tool_calls=false in the required-tool grammar (#50502)
Ostring24 committed 22 days ago
[Perf] Reduce redundant Triton sampler warmup specializations (#58605)
Misha Goin committed 22 days ago
[Bugfix][Mamba] Keep the prompt-end prefill checkpoint under sparse retention (#59146)
Jared Wen committed 22 days ago
[Bugfix] Suppress HarmonyError Unexpected token while expecting start token 200006 (#59254)
yzong-rh committed 22 days ago
[Perf][KV Connector][Mooncake] Pack hybrid/MLA KV into coalesced transfer regions (#57952)
Stary committed 22 days ago
[MyPy][2/N] Fix mypy errors in small tests/ dirs (#51043)
Martin Hickey committed 22 days ago
[MRV2] Add DRY as a custom logits processor example (#50584)
sashko-zakharchuk committed 22 days ago
[Bugfix] Avoid JSON constraints for native tool parsers (#47512)
hubunt committed 22 days ago