A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 08:00 UTC
main
Wen Sun committed about 3 years ago
Initialize AsyncLLMEngine bg loop correctly (#943)
Antoni Baum committed about 3 years ago
Align vLLM's beam search implementation with HF generate (#857)
Zhuohan Li committed about 3 years ago
Only emit warning about internal tokenizer if it isn't being used (#939)
Nelson Liu committed about 3 years ago
Refactor AsyncLLMEngine (#880)
Antoni Baum committed about 3 years ago
[BugFix] Fix NaN errors in paged attention kernel (#936)
Woosuk Kwon committed about 3 years ago
Avoid compiling kernels for double data type (#933)
Woosuk Kwon committed about 3 years ago
Woosuk Kwon committed about 3 years ago
Zhuohan Li committed about 3 years ago
[Docs] Minor fixes in supported models (#920)
Woosuk Kwon committed about 3 years ago
fix: bug fix when penalties are negative (#913)
Dong-Yong Lee committed about 3 years ago
Improve _prune_hidden_states micro-benchmark (#707)
Aman Gupta Karmani committed about 3 years ago
Add acknowledgement to a16z grant
Zhuohan Li committed about 3 years ago
Accelerate LLaMA model loading (#234)
JFDuan committed about 3 years ago
Enable request body OpenAPI spec for OpenAI endpoints (#865)
lplcor committed about 3 years ago
use flash-attn via xformers (#877)
Aman Gupta Karmani committed about 3 years ago
[Fix] Fix a condition for ignored sequences (#867)
Zhuohan Li committed about 3 years ago
Add support for CodeLlama (#854)
Antoni Baum committed about 3 years ago
Bump up the version to v0.1.4 (#846)
Woosuk Kwon committed about 3 years ago
Set replacement=True in torch.multinomial (#858)
Woosuk Kwon committed about 3 years ago