A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 02:00 UTC
main
[core] LLM.collective_rpc interface and RLHF example (#12084)
youkaichao committed almost 2 years ago
Support torchrun and SPMD-style offline inference (#12071)
youkaichao committed almost 2 years ago
[Bugfix] Remove hardcoded `head_size=256` for Deepseek v2 and v3 (#12067)
Isotr0py committed almost 2 years ago
Various cosmetic/comment fixes (#12089)
Michael Goin committed almost 2 years ago
[Doc] Add documentation for specifying model architecture (#12105)
Cyrus Leung committed almost 2 years ago
[Core] Default to using per_token quantization for fp8 when cutlass is supported. (#8651)
Elfie Guo committed almost 2 years ago
Allow hip sources to be directly included when compiling for rocm. (#12087)
tvirolai-amd committed almost 2 years ago
[Bugfix] Fix _get_lora_device for HQQ marlin (#12090)
Varun Sundar Rabindranath committed almost 2 years ago
[V1][Core] Autotune encoder cache budget (#11895)
Roger Wang committed almost 2 years ago
[Bugfix] use right truncation for non-generative tasks (#12050)
Joe Runde committed almost 2 years ago
[Doc]: Update `OpenAI-Compatible Server` documents (#12082)
maang-h committed almost 2 years ago
[Misc][Quark] Upstream Quark format to VLLM (#10765)
kewang-xlnx committed almost 2 years ago
Misc: allow to use proxy in `HTTPConnection` (#12042)
Yuan committed almost 2 years ago
[Model]: Support internlm3 (#12037)
RunningLeon committed almost 2 years ago
[Platform] Do not raise error if _Backend is not found (#12023)
wangxiyuan committed almost 2 years ago
Type-fix: make execute_model output type optional (#12020)
Keyun Tong committed almost 2 years ago
Fix: cases with empty sparsity config (#12057)
Rahul Tuli committed almost 2 years ago
[V1][Prefix Cache] Move the logic of num_computed_tokens into KVCacheManager (#12003)
Chen Zhang committed almost 2 years ago
[Doc] Update examples to remove SparseAutoModelForCausalLM (#12062)
Kyle Sayers committed almost 2 years ago
[core] platform agnostic executor via collective_rpc (#11256)
youkaichao committed almost 2 years ago