A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 09:15 UTC
main
Implement custom kernel for LLaMA rotary embedding (#14)
Woosuk Kwon committed over 3 years ago
Woosuk Kwon committed over 3 years ago
Refactor the test code for attention kernels (#13)
Woosuk Kwon committed over 3 years ago
Add cache watermark to avoid frequent cache eviction (#11)
Woosuk Kwon committed over 3 years ago
FastAPI-based working frontend (#10)
Zhuohan Li committed over 3 years ago
Woosuk Kwon committed over 3 years ago
Zhuohan Li committed over 3 years ago
Add miscellaneous updates (#8)
Woosuk Kwon committed over 3 years ago
Add memory analyzer & utomatically configure KV cache size (#6)
Woosuk Kwon committed over 3 years ago
Support beam search & parallel generation (#7)
Woosuk Kwon committed over 3 years ago
Fix a bug in 1D input shape (#5)
Woosuk Kwon committed over 3 years ago
Use FlashAttention for `multi_query_kv_attention` (#4)
Woosuk Kwon committed over 3 years ago
Implement `single_query_cached_kv_attention` kernel (#3)
Woosuk Kwon committed over 3 years ago
Fix a bug in tying OPT embeddings (#1)
Woosuk Kwon committed over 3 years ago
Woosuk Kwon committed over 3 years ago
Woosuk Kwon committed over 3 years ago
Woosuk Kwon committed over 3 years ago
Woosuk Kwon committed over 3 years ago
Add max_num_steps to SamplingParams
Woosuk Kwon committed over 3 years ago
Woosuk Kwon committed over 3 years ago