A high-throughput and memory-efficient inference and serving engine for LLMs
main
Use std::bit_cast for type punning in CPU kernels (#45089)
Yuanyuan Chen committed 5 months ago
[Rust Frontend] Add Python bridge for Rust tool parsers (#44624)
Bugen Zhao committed 5 months ago
[Build] fix self-contradictory precompiled-flag orthogonality test (#44942)
Prajjwal Chittori committed 5 months ago
[Refactor][Parser] Unify Response API to use parser.parse() like Chat Completion API (#45190)
Flora Feng committed 5 months ago
[KV Connector] Mooncake store: prefix-cache retention interval for sparse attention (#44774)
Yifan Qiao committed 5 months ago
[CPU][RISC-V] Enable oneDNN W8A8 INT8 to run on RISC-V (#44478)
velonica0 committed 5 months ago
[Bugfix][Responses API] Set id on function_call item in streaming done event (#44608)
ankrovv committed 5 months ago
[Rust Frontend] Populate `cached_token_count` in responses (#44887)
Bugen Zhao committed 5 months ago
[CI Bug] Remove qwen test `ValueError: No example model defined for Qwen/Qwen-7B-Chat` (#45194)
Wentao Ye committed 5 months ago
Deprecate Transformers v4 support (#45161)
Harry Mellor committed 5 months ago
[CI][Bugfix] Update Dockerfile dependency graph PNG (#45209)
Flora Feng committed 5 months ago
[Bugfix][Rust Frontend] Stop unescaping XML-style tool-call parameter values (#45025)
Ting SUN committed 5 months ago
[Bugfix]: Fix Quark gpt-oss weight loading broken by FusedMoe refactor (#45067)
Rohan Potdar committed 5 months ago
[Chore] Add Github notification for MRv2 for @yewentao256 (#45191)
Wentao Ye committed 5 months ago
qizixi committed 5 months ago
[CI/Build] Skip test_use_trtllm_attention on non-CUDA platforms (#44687)
Dan Blanaru committed 5 months ago
[ROCm][CI] Moving MI300 tests to MI325 until cluster is stabilized (#45170)
Andreas Karatzas committed 5 months ago
[ROCm][Bugfix] Make intermediate_pad TP-aware in rocm_aiter_fused_experts (#44679)
Rohan Potdar committed 5 months ago
feat(qwen3-asr): support prompt parameter in v1/audio/transcriptions (#35415)
Nathan Price committed 5 months ago
[EPLB] Reject NCCL-based EPLB communicators with async EPLB (#44978)
Ilya Markov committed 5 months ago