A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 03:00 UTC
main
[Bugfix] Fix Llava inference with Tensor Parallelism. (#3883)
Isotr0py committed over 2 years ago
[Core] latency optimization (#3890)
youkaichao committed over 2 years ago
[Core] enable out-of-tree model register (#3871)
youkaichao committed over 2 years ago
[CI/Benchmark] add more iteration and use median for robust latency benchmark (#3889)
youkaichao committed over 2 years ago
[Bugfix] Fix incorrect output on OLMo models in Tensor Parallelism (#3869)
Isotr0py committed over 2 years ago
[Chunked Prefill][4/n] Chunked prefill scheduler. (#3853)
SangBin Cho committed over 2 years ago
Add option to completion API to truncate prompt tokens (#3144)
Thomas Parnell committed over 2 years ago
[Misc] Define common requirements (#3841)
Woosuk Kwon committed over 2 years ago
[Bugfix] Fixing requirements.txt (#3865)
Noam Gat committed over 2 years ago
[Misc] Fix linter issues in examples/fp8/quantizer/quantize.py (#3864)
Cade Daniel committed over 2 years ago
[Misc] Add pytest marker to opt-out of global test cleanup (#3863)
Cade Daniel committed over 2 years ago
[CI/Build] refactor dockerfile & fix pip cache
youkaichao committed over 2 years ago
[Doc]Add asynchronous engine arguments to documentation. (#3810)
Sean Gallen committed over 2 years ago
[Core] improve robustness of pynccl (#3860)
youkaichao committed over 2 years ago
[Model] Cohere CommandR+ (#3829)
Saurabh Dash committed over 2 years ago
[Hardware][CPU] Update cpu torch to match default of 2.2.1 (#3854)
Michael Goin committed over 2 years ago
[Core] manage nccl via a pypi package & upgrade to pt 2.2.1 (#3805)
youkaichao committed over 2 years ago
[Benchmark] Refactor sample_requests in benchmark_throughput (#3613)
TianYu GUO committed over 2 years ago
[Bugfix] Fix args in benchmark_serving (#3836)
Chang Su committed over 2 years ago
[Core] [Frontend] Make detokenization optional (#3749)
Matthias Gerstgrasser committed over 2 years ago