A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 20:45 UTC
main
Noam Gat committed over 2 years ago
[BugFix][Frontend] Use LoRA tokenizer in OpenAI APIs (#6227)
Nick Hill committed over 2 years ago
[Misc] Minor patch for draft model runner (#6523)
Cody Yu committed over 2 years ago
[Core] Introduce SPMD worker execution using Ray accelerated DAG (#6032)
Rui Qiao committed over 2 years ago
[BugFix] Avoid secondary error in ShmRingBuffer destructor (#6530)
Nick Hill committed over 2 years ago
[core][model] yet another cpu offload implementation (#6496)
youkaichao committed over 2 years ago
[ Kernel ] Fp8 Channelwise Weight Support (#6487)
Robert Shaw committed over 2 years ago
[Model] Pipeline parallel support for Mixtral (#6516)
Cody Yu committed over 2 years ago
[ Kernel ] FP8 Dynamic-Per-Token Quant Kernel (#6511)
Varun Sundar Rabindranath committed over 2 years ago
[Core] draft_model_runner: Implement prepare_inputs on GPU for advance_step (#6338)
Alexander Matveev committed over 2 years ago
[Bugfix] Fix Ray Metrics API usage (#6354)
Antoni Baum committed over 2 years ago
[DOC] - Add docker image to Cerebrium Integration (#6510)
milo157 committed over 2 years ago
[Core] Refactor _prepare_model_input_tensors - take 2 (#6164)
Cody Yu committed over 2 years ago
[Misc] Use `torch.Tensor` for type annotation (#6505)
Woosuk Kwon committed over 2 years ago
[TPU] Remove multi-modal args in TPU backend (#6504)
Woosuk Kwon committed over 2 years ago
[Bugfix] Fix for multinode crash on 4 PP (#6495)
Murali Andoorveedu committed over 2 years ago
[Doc][CI/Build] Update docs and tests to use `vllm serve` (#6431)
Cyrus Leung committed over 2 years ago
[Misc][Speculative decoding] Typos and typing fixes (#6467)
shangmingc committed over 2 years ago
[ROCm] Cleanup Dockerfile and remove outdated patch (#6482)
Hongxia Yang committed over 2 years ago
[Distributed][PP] only create embedding & lm head when necessary (#6455)
Wushi Dong committed over 2 years ago