A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 00:30 UTC
main
[Bugfix][Model] Add base class for vision-language models (#4809)
Cyrus Leung committed over 2 years ago
[Lora] Support long context lora (#4787)
SangBin Cho committed over 2 years ago
[ROCm][Hardware][AMD] Adding Navi21 to fallback to naive attention if Triton is not used (#4658)
alexeykondrat committed over 2 years ago
[Bugfix] Relax tiktoken to >= 0.6.0 (#4890)
Michael Goin committed over 2 years ago
[Doc] Update Ray Data distributed offline inference example (#4871)
Antoni Baum committed over 2 years ago
Sync huggingface modifications of qwen Moe model (#4774)
eigenLiu committed over 2 years ago
[Bugfix] fix rope error when load models with different dtypes (#4835)
Jinzhen Lin committed over 2 years ago
Alexei-V-Ivanov-AMD committed over 2 years ago
[Frontend] OpenAI API server: Do not add bos token by default when encoding (#4688)
bofeng huang committed over 2 years ago
Support to serve vLLM on Kubernetes with LWS (#4829)
Kante Yin committed over 2 years ago
[Bugfix] Fix FP8 KV cache support (#4869)
Woosuk Kwon committed over 2 years ago
[Kernel] Add w8a8 CUTLASS kernels (#4749)
Tyler Michael Smith committed over 2 years ago
[Kernel] Add punica dimension for Qwen1.5-32B LoRA (#4850)
Silencio committed over 2 years ago
[Misc] remove old comments (#4866)
youkaichao committed over 2 years ago
[Core][Distributed] remove graph mode function (#4818)
youkaichao committed over 2 years ago
[ROCm][AMD][Bugfix] adding a missing triton autotune config (#4845)
Hongxia Yang committed over 2 years ago
Add JSON output support for benchmark_latency and benchmark_throughput (#4848)
Simon Mo committed over 2 years ago
Add GPTQ Marlin 2:4 sparse structured support (#4790)
Alexander Matveev committed over 2 years ago
[Bugfix] Bypass authorization API token for preflight requests (#4862)
Pierre Dulac committed over 2 years ago
[Frontend] Separate OpenAI Batch Runner usage from API Server (#4851)
Alex Wu committed over 2 years ago