A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 09:15 UTC
main
Add support for GPT-NeoX (Pythia) (#50)
Woosuk Kwon committed over 3 years ago
Woosuk Kwon committed over 3 years ago
Support various block sizes & Change default block size to 16 (#38)
Woosuk Kwon committed over 3 years ago
Collect system stats in scheduler & Add scripts for experiments (#30)
Woosuk Kwon committed over 3 years ago
Memcpy kernel for flash attention (#29)
Siyuan (Ryans) Zhuang committed over 3 years ago
Woosuk Kwon committed over 3 years ago
Add an option to use dummy model weights (#33)
Woosuk Kwon committed over 3 years ago
Add query stride to multi_query_cached_kv_attention & Add kernel benchmark script (#27)
Woosuk Kwon committed over 3 years ago
Implement block copy kernel to optimize beam search (#32)
Woosuk Kwon committed over 3 years ago
Fix potential bugs in FastAPI frontend and add comments (#28)
Zhuohan Li committed over 3 years ago
Add CUDA graph-based all reduce launcher (#26)
Woosuk Kwon committed over 3 years ago
Basic attention kernel that supports cached KV + (multi-)prompts (#24)
Siyuan (Ryans) Zhuang committed over 3 years ago
Woosuk Kwon committed over 3 years ago
Merge QKV into one linear layer (#15)
Zhuohan Li committed over 3 years ago
Woosuk Kwon committed over 3 years ago
Use FP32 for log probabilities (#19)
Woosuk Kwon committed over 3 years ago
Modify README to include info on loading LLaMA (#18)
Zhuohan Li committed over 3 years ago
Add custom kernel for RMS normalization (#16)
Woosuk Kwon committed over 3 years ago
Optimize tensor parallel execution speed (#17)
Zhuohan Li committed over 3 years ago
Implement preemption via recomputation & Refactor scheduling logic (#12)
Woosuk Kwon committed over 3 years ago