A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 19:30 UTC
main
PP comm optimization: replace send with partial send + allgather (#6695)
Aurick Qiao committed over 2 years ago
[Bugfix][Model] Skip loading lm_head weights if using tie_word_embeddings (#6758)
Travis Johnson committed over 2 years ago
[Bugfix][TPU] Do not use torch.Generator for TPUs (#6981)
Woosuk Kwon committed over 2 years ago
[Model] Pipeline parallel support for Qwen2 (#6924)
xuyi committed over 2 years ago
[Kernel][RFC] Refactor the punica kernel based on Triton (#5036)
Jee Jee Li committed over 2 years ago
Revert "[Frontend] Factor out code for running uvicorn" (#7012)
Simon Mo committed over 2 years ago
[Misc] Add compressed-tensors to optimized quant list (#7006)
Michael Goin committed over 2 years ago
[Kernel] Tuned int8 Cutlass Kernels for SM75 (T4) (#6996)
Varun Sundar Rabindranath committed over 2 years ago
[Kernel] Enable FP8 Cutlass for Ada Lovelace (#6950)
Varun Sundar Rabindranath committed over 2 years ago
[Bugfix] Support cpu offloading with fp8 quantization (#6960)
Michael Goin committed over 2 years ago
[MISC] Introduce pipeline parallelism partition strategies (#6920)
Cody Yu committed over 2 years ago
[Model] use FusedMoE layer in Jamba (#6935)
Avshalom Manevich committed over 2 years ago
[Bugfix] Fix feature size calculation for LLaVA-NeXT (#6982)
Cyrus Leung committed over 2 years ago
[Bugfix] Clean up MiniCPM-V (#6939)
Alphi committed over 2 years ago
Support W4A8 quantization for vllm (#5218)
HandH1998 committed over 2 years ago
[Bugfix] fix logit processor excceed vocab size issue (#6927)
Fei committed over 2 years ago
[Bugfix][TPU] Set readonly=True for non-root devices (#6980)
Woosuk Kwon committed over 2 years ago
[CI/Build] Fix mypy errors (#6968)
Cyrus Leung committed over 2 years ago
[Bugfix] Fix broadcasting logic for `multi_modal_kwargs` (#6836)
Cyrus Leung committed over 2 years ago
[mypy] Enable following imports for some directories (#6681)
Cyrus Leung committed over 2 years ago