A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 04:30 UTC
main
[torch.compile] allow tracking forward time (#11081)
youkaichao committed almost 2 years ago
[[Misc]Upgrade bitsandbytes to the latest version 0.45.0 (#11201)
Jee Jee Li committed almost 2 years ago
Sungjae Lee committed almost 2 years ago
Enable mypy checking on V1 code (#11105)
Mark McLoughlin committed almost 2 years ago
[VLM] Fully dynamic prompt replacement in merged input processor (#11199)
Cyrus Leung committed almost 2 years ago
[Frontend] Add `logits_processors` as an extra completion argument (#11150)
Brad Hilton committed almost 2 years ago
[Misc] Minor improvements to the readability of PunicaWrapperBase (#11200)
Jee Jee Li committed almost 2 years ago
[V1][Bugfix] Fix V1 TP trust-remote-code (#11182)
Tyler Michael Smith committed almost 2 years ago
[Core] Update outlines and increase its threadpool size (#11140)
Russell Bryant committed almost 2 years ago
update compressed-tensors to latest version (#11183)
dhuangnm committed almost 2 years ago
[Bugfix][V1] Re-compute an entire block when fully cache hit (#11186)
Cody Yu committed almost 2 years ago
[V1][Bugfix] Fix EngineCoreProc profile (#11185)
Tyler Michael Smith committed almost 2 years ago
[Core] V1: Use multiprocessing by default (#11074)
Russell Bryant committed almost 2 years ago
[Distributed] Allow the placement group more time to wait for resources to be ready (#11138)
Jiaxin Shan committed almost 2 years ago
[Bugfix][Hardware][CPU] Enable Gemma2 with SDPA on CPU backend (#11169)
Jani Monoses committed almost 2 years ago
[Doc] Reorganize online pooling APIs (#11172)
Cyrus Leung committed almost 2 years ago
[Misc] Add tokenizer_mode param to benchmark_serving.py (#11174)
Alexander Matveev committed almost 2 years ago
zhangjf committed almost 2 years ago
Sungjae Lee committed almost 2 years ago
[Refactor]A simple device-related refactor (#11163)
Chenguang Li committed almost 2 years ago