A high-throughput and memory-efficient inference and serving engine for LLMs
main
Add Triton recompile detection (#45631)
Thien Tran committed 5 months ago
[Misc]Clean up useless test (#45792)
wangxiyuan committed 5 months ago
[XPU][CI] add intel xpu cases for nightly CI (#44372)
wenjun liu committed 5 months ago
[Bugfix] Fix trtllm fused allreduce+rms_norm for transformers backend (#45307)
Thomas Parnell committed 5 months ago
[Frontend] Remove AsyncMicrobatchTokenizer. (#45759)
wang.yuqi committed 5 months ago
[Bugfix] Prevent cuMemcpyBatchAsync segfault with MTP and KV offloading (#44784)
joshua abraham committed 5 months ago
[CPU] Support Gemma Diffusion (#45690)
Li, Jiang committed 5 months ago
[Bug Fix] Allow pinned memory for WSL2 (#41496)
Jimmy Lee committed 5 months ago
[Cleanup] Remove dead env (#45777)
Cyrus Leung committed 5 months ago
[Misc] Added validation for Cohere /v2/embed input field exclusivity (#45640)
Taneem Ibrahim committed 5 months ago
Register parsed config classes before tokenizer init (#40299)
Andrew Barnes committed 5 months ago
[Core] Use fastsafetensors ParallelLoader for weight loading (#40183)
gitbisector committed 5 months ago
[Frontend] [Parser] Migrate Nemotron V3 to streaming parser engine (#45755)
Ben Browning committed 5 months ago
[Rust Frontend] Require `ModelConfig.vocab_size` to be present (#45696)
Bugen Zhao committed 5 months ago
Luciano Martins committed 5 months ago
[XPU] Fix Triton attn fp8/bf16 check failing (#45758)
liuzhenwei committed 5 months ago
[BugFix] Support async scheduling with prompt embeds for multimodal models (#45673)
Ruinan Ma committed 5 months ago
[Multimodal] Add Qwen3-VL video loader (#44412)
Isotr0py committed 5 months ago
[Rust Frontend] Lower out-of-vocab validation to `text` layer (#45685)
Bugen Zhao committed 5 months ago
[Rust Frontend] Support `max_logprobs` validation (#45674)
Bugen Zhao committed 5 months ago