A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 01:45 UTC
main
[Core][Distributed] use cpu/gloo to initialize pynccl (#4248)
youkaichao committed over 2 years ago
[BUG] fixed fp8 conflict with aqlm (#4307)
Robert Shaw committed over 2 years ago
[Kernel] FP8 support for MoE kernel / Mixtral (#4244)
Philipp Moritz committed over 2 years ago
[Bugfix][Frontend] Raise exception when file-like chat template fails to be opened (#4292)
Cyrus Leung committed over 2 years ago
James Fleming committed over 2 years ago
[CI] Add ccache for wheel builds job (#4281)
Simon Mo committed over 2 years ago
[Core][Logging] Add last frame information for better debugging (#4278)
youkaichao committed over 2 years ago
[Bugfix] Add init_cached_hf_modules to RayWorkerWrapper (#4286)
DefTruth committed over 2 years ago
[Bugfix] Fixing max token error message for openai compatible server (#4016)
Jack Gordley committed over 2 years ago
[Speculative decoding 7/9] Speculative decoding end-to-end correctness tests. (#3951)
Cade Daniel committed over 2 years ago
[Core] Scheduling optimization 2 (#4280)
SangBin Cho committed over 2 years ago
[Core] Some simplification of WorkerWrapper changes (#4183)
Nick Hill committed over 2 years ago
[Mypy] Part 3 fix typing for nested directories for most of directory (#4161)
SangBin Cho committed over 2 years ago
Fix `autodoc` directives (#4272)
Harry Mellor committed over 2 years ago
[Core][Distributed] use absolute path for library file (#4271)
youkaichao committed over 2 years ago
[Doc] Update the SkyPilot doc with serving and Llama-3 (#4276)
Zhanghao Wu committed over 2 years ago
[Core] Scheduler perf fix (#4270)
SangBin Cho committed over 2 years ago
Add example scripts to documentation (#4225)
Harry Mellor committed over 2 years ago
[Bugfix] Ensure download_weights_from_hf(..) inside loader is using the revision parameter (#4217)
alexm-nm committed over 2 years ago
[Frontend] Enable support for CPU backend in AsyncLLMEngine. (#3993)
Tao He committed over 2 years ago