A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 17:00 UTC
main
[Bugfix] Offline mode fix (#8376)
Joe Runde committed about 2 years ago
[BugFix] Fix Duplicate Assignment in Hermes2ProToolParser (#8423)
Luis Vega committed about 2 years ago
[BugFix] lazy init _copy_stream to avoid torch init wrong gpu instance (#8403)
WANGWEI committed about 2 years ago
[Model] Support multiple images for qwen-vl (#8247)
Alex Brooks committed about 2 years ago
[Bugfix] Fix InternVL2 inference with various num_patches (#8375)
Isotr0py committed about 2 years ago
[Hotfix][VLM] Fixing max position embeddings for Pixtral (#8399)
Roger Wang committed about 2 years ago
[torch.compile] hide slicing under custom op for inductor (#8384)
youkaichao committed about 2 years ago
[Misc] Use RoPE cache for MRoPE (#8396)
Woosuk Kwon committed about 2 years ago
[Misc] Raise error when using encoder/decoder model with cpu backend (#8355)
Kevin Lin committed about 2 years ago
[Gemma2] add bitsandbytes support for Gemma2 (#8338)
Blueyo0 committed about 2 years ago
[Bugfix]: Fix the logic for deciding if tool parsing is used (#8366)
tomeras91 committed about 2 years ago
Fix the AMD weight loading tests (#8390)
Michael Goin committed about 2 years ago
[TPU] Use Ray for default distributed backend (#8389)
Woosuk Kwon committed about 2 years ago
[misc] remove engine_use_ray (#8126)
youkaichao committed about 2 years ago
[MISC] Dump model runner inputs when crashing (#8305)
Cody Yu committed about 2 years ago
Bump version to v0.6.1 (#8379)
Simon Mo committed about 2 years ago
Patrick von Platen committed about 2 years ago
[Speculative Decoding] Test refactor (#8317)
Lily Liu committed about 2 years ago
[Misc] Move device options to a single place (#8322)
Aarni Koskela committed about 2 years ago
[Kernel][Misc] register ops to prevent graph breaks (#6917)
bnellnm committed about 2 years ago