A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 04:30 UTC
main
[V1] Use more persistent buffers to optimize input preparation overheads (#11111)
Woosuk Kwon committed almost 2 years ago
[Model] Add support for embedding model GritLM (#10816)
Pooya Davoodi committed almost 2 years ago
[Core] cleanup zmq ipc sockets on exit (#11115)
Russell Bryant committed almost 2 years ago
Update link to LlamaStack remote vLLM guide in serving_with_llamastack.rst (#11112)
Yuan Tang committed almost 2 years ago
Simon Mo committed almost 2 years ago
[Bugfix] Multiple fixes to tool streaming with hermes and mistral (#10979)
Clayton committed almost 2 years ago
[V1] VLM preprocessor hashing (#11020)
Alexander Matveev committed almost 2 years ago
[V1][Core] Remove should_shutdown to simplify core process termination (#11113)
Tyler Michael Smith committed almost 2 years ago
[CI/Build] Split up VLM tests (#11083)
Cyrus Leung committed almost 2 years ago
[core] Bump ray to use _overlap_gpu_communication in compiled graph tests (#10410)
Rui Qiao committed almost 2 years ago
[torch.compile] remove graph logging in ci (#11110)
youkaichao committed almost 2 years ago
[V1] Use input_ids as input for text-only models (#11032)
Woosuk Kwon committed almost 2 years ago
[torch.compile] use depyf to dump torch.compile internals (#10972)
youkaichao committed almost 2 years ago
[Doc] Installed version of llmcompressor for int8/fp8 quantization (#11103)
bingps committed almost 2 years ago
[CI/Build] Enable prefix caching test for AMD (#11098)
hissu-hyvarinen committed almost 2 years ago
[Doc] Update docs to refer to pooling models (#11093)
Cyrus Leung committed almost 2 years ago
[Misc] Split up pooling tasks (#10820)
Cyrus Leung committed almost 2 years ago
[Bugfix]: Clamp `-inf` logprob values in prompt_logprobs (#11073)
Rafael Vasquez committed almost 2 years ago
[Bugfix] Fix Idefics3 fails during multi-image inference (#11080)
B-201 committed almost 2 years ago
[Core] v1: Use atexit to handle engine core client shutdown (#11076)
Russell Bryant committed almost 2 years ago