A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 01:45 UTC
main
[Kernel] Use flash-attn for decoding (#3648)
Stephen Krider committed over 2 years ago
[Speculative decoding] Improve n-gram efficiency (#4724)
Cody Yu committed over 2 years ago
Sanger Steel committed over 2 years ago
[Misc] Enhance attention selector (#4751)
Woosuk Kwon committed over 2 years ago
[Scheduler] Warning upon preemption and Swapping (#4647)
SangBin Cho committed over 2 years ago
[CI/Build] Move `test_utils.py` to `tests/utils.py` (#4425)
Cyrus Leung committed over 2 years ago
[Core][Distributed] refactor custom allreduce to support multiple tp groups (#4754)
youkaichao committed over 2 years ago
[CORE] Improvement in ranks code (#4718)
Swapnil Parekh committed over 2 years ago
[CI/Build] Tweak Marlin Nondeterminism Issues (#4713)
Robert Shaw committed over 2 years ago
[Model] Add support for IBM Granite Code models (#4636)
Yikang Shen committed over 2 years ago
[Model][Misc] Add e5-mistral-7b-instruct and Embedding API (#3734)
Chang Su committed over 2 years ago
[Core][Test] fix function name typo in custom allreduce (#4750)
youkaichao committed over 2 years ago
[CI] Nits for bad initialization of SeqGroup in testing (#4748)
Robert Shaw committed over 2 years ago
[Speculative decoding] CUDA graph support (#4295)
heeju-kim2 committed over 2 years ago
[Bugfix] Fix CLI arguments in OpenAI server docs (#4729)
Allen.Dou committed over 2 years ago
[Core] Fix circular reference which leaked llm instance in local dev env (#4737)
SangBin Cho committed over 2 years ago
[Misc] Apply a couple g++ cleanups (#4719)
Steve Grubb committed over 2 years ago
[Core]fix type annotation for `swap_blocks` (#4726)
Kunshang Ji committed over 2 years ago
chunked-prefill-doc-syntax (#4603)
Simon Mo committed over 2 years ago
[Misc] Keep only one implementation of the create_dummy_prompt function. (#4716)
Allen.Dou committed over 2 years ago