A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 02:15 UTC
main
[Quantization][1/N] MoE support BNB-Inflight Quantization (#20061)
Jee Jee Li committed over 1 year ago
[Bugfix] Upgrade depyf to 0.19 and streamline custom pass logging (#20777)
Luka Govedič committed over 1 year ago
[doc] fold long code block (#20795)
Reid committed over 1 year ago
[V1] Enable Mamba2 layers other than MambaMixer2 in the v1 engine (#20660)
nopperl committed over 1 year ago
[Perf][fp8] Use CustomOp abstraction for fp8 quant for better perf (#19830)
Luka Govedič committed over 1 year ago
[XPU] XCCL support enabled in torch 2.8.0.dev nightly builds (#20705)
Ratnam Parikh committed over 1 year ago
[Feature] Integrate SM100 DeepGEMM support (#20087)
Wentao Ye committed over 1 year ago
[Attention] MLA - Flashinfer Ragged Prefill (#20034)
Alexander Matveev committed over 1 year ago
[Model] Support HF format of minimax (#20211)
Michael Goin committed over 1 year ago
[fix]: disable cutlass block scaled group gemm for EP (#20781)
Duncan Moss committed over 1 year ago
Add kimi-k2 tool parser (#20789)
bigmoyan committed over 1 year ago
[CI Bugfix] Specify same TORCH_CUDA_ARCH_LIST for flashinfer aot and install (#20772)
Michael Goin committed over 1 year ago
[Docs] Lazy import gguf (#20785)
Simon Mo committed over 1 year ago
[Bugfix][Benchmark] Make sure the output length > 0 when testing prefill workload. (#20786)
Kuntai Du committed over 1 year ago
[Misc] MoE ModularKernel : Introduce TopKWeightAndReduce (#20648)
Varun Sundar Rabindranath committed over 1 year ago
[KVConnector] Always call connector `clear_metadata()` at end of step (#20756)
Nick Hill committed over 1 year ago
[Bugfix] Fused MoE Modular Kernel chunking loop (#20392)
Varun Sundar Rabindranath committed over 1 year ago
[Core] Add Support for Default Modality Specific LoRAs [generate / chat completions] (#19126)
Alex Brooks committed over 1 year ago
[Bugfix] Remove assertion of expert_map being None (#20714)
Ming Yang committed over 1 year ago
[CI] Fix pre commit issue (#20782)
Wentao Ye committed over 1 year ago