A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 19:30 UTC
main
[Speculative decoding] Add serving benchmark for llama3 70b + speculative decoding (#6964)
Cade Daniel committed over 2 years ago
[CI] [nightly benchmark] Do not re-download sharegpt dataset if exists (#6706)
Cade Daniel committed over 2 years ago
[Nightly benchmarking suite] Remove pkill python from run benchmark suite (#6965)
Cade Daniel committed over 2 years ago
[Build] Temporarily Disable Kernels and LoRA tests (#6961)
Simon Mo committed over 2 years ago
[core][misc] improve free_finished_seq_groups (#6865)
youkaichao committed over 2 years ago
[Kernel] Remove scaled_fp8_quant kernel padding footgun (#6842)
Tyler Michael Smith committed over 2 years ago
[Bugfix] Fix tensorizer memory profiling bug during testing (#6881)
Sanger Steel committed over 2 years ago
[OpenVINO] Updated OpenVINO requirements and build docs (#6948)
Ilya Lavrenov committed over 2 years ago
[Kernel] Squash a few more warnings (#6914)
Tyler Michael Smith committed over 2 years ago
[BugFix] Fix use of per-request seed with pipeline parallel (#6698)
Nick Hill committed over 2 years ago
[Doc] Super tiny fix doc typo (#6949)
fzyzcjy committed over 2 years ago
[Bugfix] Fix PaliGemma MMP (#6930)
Roger Wang committed over 2 years ago
[TPU] Fix greedy decoding (#6933)
Woosuk Kwon committed over 2 years ago
[Kernel] Tuned int8 kernels for Ada Lovelace (#6848)
Varun Sundar Rabindranath committed over 2 years ago
[Kernel] Fix marlin divide-by-zero warnings (#6904)
Tyler Michael Smith committed over 2 years ago
[ci] GHA workflow to remove ready label upon "/notready" comment (#6921)
Kevin H. Luu committed over 2 years ago
[Kernel] Remove unused variables in awq/gemm_kernels.cu (#6908)
Tyler Michael Smith committed over 2 years ago
[Frontend] New `allowed_token_ids` decoding request parameter (#6753)
Nick Hill committed over 2 years ago
[Bugfix] Allow vllm to still work if triton is not installed. (#6786)
Thomas Parnell committed over 2 years ago
[TPU] Support tensor parallelism in async llm engine (#6891)
Earthwalker committed over 2 years ago