A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 22:00 UTC
main
[Kernel] Flashinfer for prefill & decode, with Cudagraph support for decode (#4628)
Lily Liu committed over 2 years ago
[Bugfix] Fix Engine Failing After Invalid Request - AsyncEngineDeadError (#5963)
Robert Shaw committed over 2 years ago
Unmark more files as executable (#5962)
Tyler Michael Smith committed over 2 years ago
[Bugfix] Only add `Attention.kv_scale` if kv cache quantization is enabled (#5936)
Michael Goin committed over 2 years ago
wangding zeng committed over 2 years ago
[ Bugfix ] Enabling Loading Models With Fused QKV/MLP on Disk with FP8 (#5921)
Robert Shaw committed over 2 years ago
[ Misc ] Remove `fp8_shard_indexer` from Col/Row Parallel Linear (Simplify Weight Loading) (#5928)
Robert Shaw committed over 2 years ago
[Bugfix] Fix compute datatype for cutlass 3.x epilogues (#5931)
Tyler Michael Smith committed over 2 years ago
[Spec Decode] Introduce DraftModelRunner (#5799)
Cody Yu committed over 2 years ago
[Distributed] Make it clear that % should not be in tensor dict keys. (#5927)
xwjiang2010 committed over 2 years ago
[CI/Build] [2/3] Reorganize entrypoints tests (#5904)
Cyrus Leung committed over 2 years ago
Thomas Parnell committed over 2 years ago
[Hardware][Intel] OpenVINO vLLM backend (#5379)
Ilya Lavrenov committed over 2 years ago
Unmark fused_moe config json file as executable (#5960)
Tyler Michael Smith committed over 2 years ago
[Core] Registry for processing model inputs (#5214)
Cyrus Leung committed over 2 years ago
[Bugfix][Hardware][Intel CPU] Fix unpassed multi_modal_kwargs for CPU runner (#5956)
Isotr0py committed over 2 years ago
[VLM][BugFix] Make sure that `multi_modal_kwargs` can broadcast properly with ring buffer. (#5905)
xwjiang2010 committed over 2 years ago
[Hardware][TPU] Optimize KV cache swapping (#5878)
Woosuk Kwon committed over 2 years ago
[Kernel][ROCm][AMD] fused_moe Triton configs v2 for mi300X (#5932)
Divakar Verma committed over 2 years ago
[core][misc] remove logical block (#5882)
youkaichao committed over 2 years ago