A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 00:45 UTC
main
Set weights_only=True when using torch.load() (#12366)
Russell Bryant committed almost 2 years ago
[V1][Frontend] Coalesce bunched `RequestOutput`s (#12298)
Nick Hill committed almost 2 years ago
[Docs] Update spec decode + structured output in compat matrix (#12373)
Russell Bryant committed almost 2 years ago
[Docs] Add meetup slides (#12345)
Woosuk Kwon committed almost 2 years ago
[Misc] Fix OpenAI API Compatibility Issues in Benchmark Script (#12357)
Junichi Sato committed almost 2 years ago
Dipika Sikka committed almost 2 years ago
[Docs] Document Phi-4 support (#12362)
Isotr0py committed almost 2 years ago
[TPU] Update TPU CI to use torchxla nightly on 20250122 (#12334)
Siyuan Liu committed almost 2 years ago
[FP8][Kernel] Dynamic kv cache scaling factors computation (#11906)
Gregory Shtrasberg committed almost 2 years ago
[torch.compile] decouple compile sizes and cudagraph sizes (#12243)
youkaichao committed almost 2 years ago
[core] add wake_up doc and some sanity check (#12361)
youkaichao committed almost 2 years ago
[Bugfix] Fix broken internvl2 inference with v1 (#12360)
Isotr0py committed almost 2 years ago
Roger Wang committed almost 2 years ago
[Doc] Troubleshooting errors during model inspection (#12351)
Cyrus Leung committed almost 2 years ago
[Kernel] Flash Attention 3 Support (#12093)
Lucas Wilkinson committed almost 2 years ago
[Bugfix] Fix k_proj's bias for whisper self attention (#12342)
Isotr0py committed almost 2 years ago
[Hardware][Gaudi][BugFix] Fix dataclass error due to triton package update (#12338)
liuzhenwei committed almost 2 years ago
[doc] explain common errors around torch.compile (#12340)
youkaichao committed almost 2 years ago
[V1] Add `uncache_blocks` (#12333)
Cody Yu committed almost 2 years ago
[Docs] Document vulnerability disclosure process (#12326)
Russell Bryant committed almost 2 years ago