A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 04:15 UTC
main
Re-enable the 80 char line width limit (#3305)
Zhuohan Li committed over 2 years ago
[Fix] Fix best_of behavior when n=1 (#3298)
Nick Hill committed over 2 years ago
[BugFix] Fix get tokenizer when using ray (#3301)
Roy committed over 2 years ago
[ROCM] Fix blockReduceSum to use correct warp counts for ROCm and CUDA (#3262)
Douglas Lehr committed over 2 years ago
Enhance lora tests with more layer and rank variations (#3243)
Terry committed over 2 years ago
[Speculative decoding 3/9] Worker which speculates, scores, and applies rejection sampling (#3103)
Cade Daniel committed over 2 years ago
[FIX] Fix prefix test error on main (#3286)
Zhuohan Li committed over 2 years ago
Move model filelocks from `/tmp/` to `~/.cache/vllm/locks/` dir (#3241)
Michael Goin committed over 2 years ago
[FIX] Make `flash_attn` optional (#3269)
Woosuk Kwon committed over 2 years ago
[Docs] Fix Unmocked Imports (#3275)
Roger Wang committed over 2 years ago
[Minor Fix] Fix comments in benchmark_serving (#3252)
TianYu GUO committed over 2 years ago
Feature add lora support for Qwen2 (#3177)
whyiug committed over 2 years ago
Connect engine healthcheck to openai server (#3260)
Nick Hill committed over 2 years ago
ElizaWszola committed over 2 years ago
jacobthebanana committed over 2 years ago
Measure model memory usage (#3120)
Michael Goin committed over 2 years ago
Separate attention backends (#3005)
Woosuk Kwon committed over 2 years ago
Update requirements-dev.txt to include package for benchmarking scripts. (#3181)
Chen Wang committed over 2 years ago
Add GPTQ support for Gemma (#3200)
TechxGenus committed over 2 years ago
Add tqdm `dynamic_ncols=True` (#3242)
Chujie Zheng committed over 2 years ago