A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 01:45 UTC
main
[Core][Distributed] refactor pynccl (#4591)
youkaichao committed over 2 years ago
[Kernel] Refactor FP8 kv-cache with NVIDIA float8_e4m3 support (#4535)
Cody Yu committed over 2 years ago
[Kernel] [FP8] Improve FP8 linear layer performance (#4691)
Philipp Moritz committed over 2 years ago
[Model] Snowflake arctic model implementation (#4652)
Hao Zhang committed over 2 years ago
[Bugfix] Add logs for all model dtype casting (#4717)
Michael Goin committed over 2 years ago
[Bugfix] Update grafana.json (#4711)
Robert Shaw committed over 2 years ago
[Bugfix] Fix CLI arguments in OpenAI server docs (#4709)
Cyrus Leung committed over 2 years ago
[ROCm] Add support for Punica kernels on AMD GPUs (#3140)
kliuae committed over 2 years ago
[Misc] Set block size at initialization & Fix test_model_runner (#4705)
Woosuk Kwon committed over 2 years ago
[Misc] Remove unnecessary ModelRunner imports (#4703)
Woosuk Kwon committed over 2 years ago
[Frontend] Move async logic outside of constructor (#4674)
Cyrus Leung committed over 2 years ago
[Frontend] add tok/s speed metric to llm class when using tqdm (#4400)
Mahmoud Ashraf committed over 2 years ago
[Bugfix] Fine-tune gptq_marlin configs to be more similar to marlin (#4626)
alexm-nm committed over 2 years ago
[Speculative decoding] [Bugfix] Fix overallocation in ngram + spec logprobs (#4672)
Cade Daniel committed over 2 years ago
[Dynamic Spec Decoding] Auto-disable by the running queue size (#4592)
Cody Yu committed over 2 years ago
[Misc] Use vllm-flash-attn instead of flash-attn (#4686)
Woosuk Kwon committed over 2 years ago
[CI/Test] fix swap test for multi gpu (#4689)
youkaichao committed over 2 years ago
[Core][Optimization] change python dict to pytorch tensor for blocks to swap (#4659)
youkaichao committed over 2 years ago
[Core] Faster startup for LoRA enabled models (#4634)
Antoni Baum committed over 2 years ago
[Misc] Add `get_name` method to attention backends (#4685)
Woosuk Kwon committed over 2 years ago