A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 03:00 UTC
main
[Model] Fix and clean commandr (#3671)
Roy committed over 2 years ago
[Docs] Add Command-R to supported models (#3669)
Woosuk Kwon committed over 2 years ago
Add support for Cohere's Command-R model (#3433)
zeppombal committed over 2 years ago
feat(benchmarks): Add Prefix Caching Benchmark to Serving Benchmark (#3277)
Roger Wang committed over 2 years ago
[Misc] add the "download-dir" option to the latency/throughput benchmarks (#3621)
AmadeusChan committed over 2 years ago
[Model] Add support for DBRX (#3660)
Megha Agarwal committed over 2 years ago
[Bugfix] [Hotfix] fix nccl library name (#3661)
youkaichao committed over 2 years ago
[Bugfix] More faithful implementation of Gemma (#3653)
Woosuk Kwon committed over 2 years ago
[Core] remove cupy dependency (#3625)
youkaichao committed over 2 years ago
[Misc] Minor fix in KVCache type (#3652)
Woosuk Kwon committed over 2 years ago
Jee Li committed over 2 years ago
[Kernel] support non-zero cuda devices in punica kernels (#3636)
Jee Li committed over 2 years ago
[BugFix] Fix ipv4 address parsing regression (#3645)
Nick Hill committed over 2 years ago
[Bugfix] Fix ipv6 address parsing bug (#3641)
liiliiliil committed over 2 years ago
Enable more models to inference based on LoRA (#3382)
Jee Li committed over 2 years ago
[Misc] Include matched stop string/token in responses (#2976)
Nick Hill committed over 2 years ago
Optimize `_get_ranks` in Sampler (#3623)
Antoni Baum committed over 2 years ago
[Feature] Add vision language model support. (#3042)
xwjiang2010 committed over 2 years ago
hotfix isort on logprobs ranks pr (#3622)
Simon Mo committed over 2 years ago
[Bugfix] API stream returning two stops (#3450)
Dylan Hawk committed over 2 years ago