A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
main
Woosuk Kwon committed almost 3 years ago
Optimize Mixtral with expert parallelism (#2090)
Antoni Baum committed almost 3 years ago
[BugFix] Fix input positions for long context with sliding window (#2088)
Woosuk Kwon committed almost 3 years ago
[Docs] Add notes on ROCm-supported models (#2087)
Woosuk Kwon committed almost 3 years ago
Update installation instruction for vLLM + CUDA 11.8 (#2086)
Woosuk Kwon committed almost 3 years ago
[ROCm] Upgrade xformers version for ROCm & update doc (#2079)
TJian committed almost 3 years ago
Implement lazy model loader (#2044)
Woosuk Kwon committed almost 3 years ago
Fix peak memory profiling (#2031)
Woosuk Kwon committed almost 3 years ago
Update Dockerfile to build Megablocks (#2042)
Simon Mo committed almost 3 years ago
Megha Agarwal committed almost 3 years ago
Remove einops from requirements (#2049)
Woosuk Kwon committed almost 3 years ago
Upgrade transformers version to 4.36.0 (#2046)
Woosuk Kwon committed almost 3 years ago
[CI/CD] Upgrade PyTorch version to v2.1.1 (#2045)
Woosuk Kwon committed almost 3 years ago
Remove python 3.10 requirement (#2040)
Woosuk Kwon committed almost 3 years ago
Update Dockerfile to support Mixtral (#2027)
Simon Mo committed almost 3 years ago
Woosuk Kwon committed almost 3 years ago
Fix import error msg for megablocks (#2038)
Woosuk Kwon committed almost 3 years ago
Update README.md to add megablocks requirement for mixtral (#2033)
Ram committed almost 3 years ago
[Minor] Fix type annotation in Mixtral (#2036)
Woosuk Kwon committed almost 3 years ago
Fix latency benchmark script (#2035)
Woosuk Kwon committed almost 3 years ago