A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 17:00 UTC
main
improve chunked prefill performance
wang.yuqi committed about 2 years ago
[Bugfix] Fix internlm2 tensor parallel inference (#8055)
Isotr0py committed about 2 years ago
[Core][Bugfix] Accept GGUF model without .gguf extension (#8056)
Isotr0py committed about 2 years ago
[TPU] Align worker index with node boundary (#7932)
Woosuk Kwon committed about 2 years ago
[SpecDecode][Kernel] Flashinfer Rejection Sampling (#7244)
Lily Liu committed about 2 years ago
[Model] Add Granite model (#7436)
Shawn Tan committed about 2 years ago
[Misc] Optional installation of audio related packages (#8063)
Roger Wang committed about 2 years ago
[Frontend][VLM] Add support for multiple multi-modal items (#8049)
Roger Wang committed about 2 years ago
[BugFix][Core] Multistep Fix Crash on Request Cancellation (#8059)
Robert Shaw committed about 2 years ago
[Bugfix] Fix ModelScope models in v0.5.5 (#8037)
Nicolò Lucchesi committed about 2 years ago
[Bugfix] Fix import error in Phi-3.5-MoE (#8052)
Cyrus Leung committed about 2 years ago
[Bugfix] bugfix and add model test for flashinfer fp8 kv cache. (#8013)
Pavani Majety committed about 2 years ago
[Model] Adding support for MSFT Phi-3.5-MoE (#7729)
Wenxiang committed about 2 years ago
[TPU][Bugfix] Fix tpu type api (#8035)
Woosuk Kwon committed about 2 years ago
[Frontend]-config-cli-args (#7737)
Kaunil Dhruv committed about 2 years ago
[Core] Increase default `max_num_batched_tokens` for multimodal models (#8028)
Cyrus Leung committed about 2 years ago
[VLM][Model] TP support for ViTs (#7186)
Jungho Christopher Cho committed about 2 years ago
[Bugfix] Fix import error in Exaone model (#8034)
Cyrus Leung committed about 2 years ago
[TPU] Support single and multi-host TPUs on GKE (#7613)
Richard Liu committed about 2 years ago
[MODEL] add Exaone model support (#7819)
Yohan Na committed about 2 years ago