A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 13, 14:30 UTC
main
Enhance run_cluster.sh for multi-NIC support (#28328)
Ev Lacey committed about 1 year ago
[DCP] Support dcp kv_cache interleave size > 1 (#26696)
zhangsicheng5 committed about 1 year ago
[Bugfix] Spec decode + structured output + spec model max len edge case (#28298)
Andy Lo committed about 1 year ago
[Bugfix] Update device name for H200 detection (#28349)
Robert Shaw committed about 1 year ago
`reasoning_content` -> `reasoning` (#27752)
Harry Mellor committed about 1 year ago
[Attention] Remove max cudagraph size limit of 992 (#27840)
22quinn committed about 1 year ago
[flashinfer][fix] do not check nvcc availability when using pre-downloaded cubins (#27990)
Xiaozhu Meng committed about 1 year ago
Refactor CPU/GPU extension targets for CMake build (#28026)
Abolfazl Shahbazi committed about 1 year ago
[Model] Consolidate Deepseek-MoE implementation with DeepSeek-v2 (#28101)
Isotr0py committed about 1 year ago
[bugfix] support eagle with lora cudagraph specialization (#28318)
gnovack committed about 1 year ago
Bump arctic-inference requirement (#28174)
Aurick Qiao committed about 1 year ago
[Perf][DeepSeek] Add sigmoid+bias fusion to fused_grouped_topk from TRTLLM (#28124)
Michael Goin committed about 1 year ago
Hamid Mukhtar committed about 1 year ago
[Bugfix][LoRA][Spec Decode] Support LoRA with speculative decoding (#21068)
Xiaohong (Sean) Chen committed about 1 year ago
remove resolve_op_overloads and use splitting_ops directly (#28081)
Boyuan Feng committed about 1 year ago
[XPU] Enable Expert parallel for MoE models (#28263)
Kunshang Ji committed about 1 year ago
Update Flashinfer from `v0.4.1` to `v0.5.2` (#27952)
Harry Mellor committed about 1 year ago
[PerfFix] Avoid separate thread for MP executor shm spin (take 2) (#28319)
Nick Hill committed about 1 year ago
[Core] Rework handling of async scheduling config (#28250)
Nick Hill committed about 1 year ago
[Bugfix] Fix and add tests for GptOss reasoning parser (#28000)
Benjamin Chislett committed about 1 year ago