A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 10:45 UTC
main
[torch.compile] improve allreduce registration (#9061)
youkaichao committed about 2 years ago
[Hardware][Neuron] Add on-device sampling support for Neuron (#8746)
Chongming Ni committed about 2 years ago
Cody Yu committed about 2 years ago
[Misc] Add random seed for prefix cache benchmark (#9081)
Andy Dai committed about 2 years ago
[Misc] Improved prefix cache example (#9077)
Andy Dai committed about 2 years ago
[CI] Update performance benchmark: upgrade trt-llm to r24.07, and add SGLang (#7412)
Kuntai Du committed about 2 years ago
[Kernel] Zero point support in fused MarlinMoE kernel + AWQ Fused MoE (#8973)
ElizaWszola committed about 2 years ago
Adds truncate_prompt_tokens param for embeddings creation (#8999)
Flávia Béo committed about 2 years ago
[Core][VLM] Test registration for OOT multimodal models (#8717)
Roger Wang committed about 2 years ago
[Hardware][PowerPC] Make oneDNN dependency optional for Power (#9039)
Varad Ahirwadkar committed about 2 years ago
Remove AMD Ray Summit Banner (#9075)
Simon Mo committed about 2 years ago
[Model] add a bunch of supported lora modules for mixtral (#9008)
Prashant Gupta committed about 2 years ago
[Bugfix] Flash attention arches not getting set properly (#9062)
Lucas Wilkinson committed about 2 years ago
[Bugfix] Reshape the dimensions of the input image embeddings in Qwen2VL (#9071)
whyiug committed about 2 years ago
[Misc] Move registry to its own file (#9064)
Cyrus Leung committed about 2 years ago
[Models] Add remaining model PP support (#7168)
Murali Andoorveedu committed about 2 years ago
[Misc] Enable multi-step output streaming by default (#9047)
Michael Goin committed about 2 years ago
[CI/Build] Per file CUDA Archs (improve wheel size and dev build times) (#8845)
Lucas Wilkinson committed about 2 years ago
[Frontend][Feature] support tool calling for internlm/internlm2_5-7b-chat model (#8405)
代君 committed about 2 years ago
[Bugfix] Weight loading fix for OPT model (#9042)
Domen Vreš committed about 2 years ago