A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 04:15 UTC
main
[Testing] Fix core tests (#3224)
Cade Daniel committed over 2 years ago
[Tests] Add block manager and scheduler tests (#3108)
SangBin Cho committed over 2 years ago
[Fix] Avoid pickling entire LLMEngine for Ray workers (#3207)
Nick Hill committed over 2 years ago
Store `eos_token_id` in `Sequence` for easy access (#3166)
Nick Hill committed over 2 years ago
[ROCm] enable cupy in order to enable cudagraph mode for AMD GPUs (#3123)
Hongxia Yang committed over 2 years ago
Fix the openai benchmarking requests to work with latest OpenAI apis (#2992)
Chen Wang committed over 2 years ago
Add health check, make async Engine more robust (#3015)
Antoni Baum committed over 2 years ago
Push logprob generation to LLMEngine (#3065)
Antoni Baum committed over 2 years ago
[Minor fix] The domain dns.google may cause a socket.gaierror exception (#3176)
ttbachyinsda committed over 2 years ago
enable --gpu-memory-utilization in benchmark_throughput.py (#3175)
Allen.Dou committed over 2 years ago
Add document for vllm paged attention kernel. (#2978)
Jialun Lyu committed over 2 years ago
[Minor Fix] Remove unused code in benchmark_prefix_caching.py (#3171)
TianYu GUO committed over 2 years ago
[DOC] add setup document to support neuron backend (#2777)
Liangfu Chen committed over 2 years ago
Make it easy to profile workers with nsight (#3162)
Philipp Moritz committed over 2 years ago
[FIX] Fix styles in automatic prefix caching & add a automatic prefix caching benchmark (#3158)
Zhuohan Li committed over 2 years ago
Add vLLM version info to logs and openai API server (#3161)
Jason Cox committed over 2 years ago
Add Automatic Prefix Caching (#2762)
Sage Moore committed over 2 years ago
Reorder kv dtype check to avoid nvcc not found error on AMD platform (#3104)
cloudhan committed over 2 years ago
allow user chose log level by --log-level instead of fixed 'info'. (#3109)
Allen.Dou committed over 2 years ago
Woosuk Kwon committed over 2 years ago