A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
main
[BUGFIX] Do not return ignored sentences twice in async llm engine (#2258)
Zhuohan Li committed almost 3 years ago
[Docs] Add "About" Heading to README.md (#2260)
blueceiling committed almost 3 years ago
[Docs] Update installation instructions to include CUDA 11.8 xFormers (#2246)
Shivam Thakkar committed almost 3 years ago
[BugFix] Fix recovery logic for sequence group (#2186)
Woosuk Kwon committed almost 3 years ago
Disable Ray usage stats collection (#2206)
Woosuk Kwon committed almost 3 years ago
[Docs] Fix broken links (#2222)
Ronen Schaffer committed almost 3 years ago
Remove Sampler copy stream (#2209)
Antoni Baum committed almost 3 years ago
Make _prepare_sample non-blocking and use pinned memory for input buffers (#2207)
Hanzhi Zhou committed almost 3 years ago
[BugFix] Fix weight loading for Mixtral with TP (#2208)
Woosuk Kwon committed almost 3 years ago
Added DeciLM-7b and DeciLM-7b-instruct (#2062)
avideci committed almost 3 years ago
Add warning on CUDA graph memory usage (#2182)
Woosuk Kwon committed almost 3 years ago
Update Help Text for --gpu-memory-utilization Argument (#2183)
Suhong Moon committed almost 3 years ago
[ROCm] Fixes for GPTQ on ROCm (#2180)
kliuae committed almost 3 years ago
JohnSaxon committed almost 3 years ago
Add SSL arguments to API servers (#2109)
Harry Mellor committed almost 3 years ago
[BugFix] Fix RoPE kernel on long sequences(#2164)
Woosuk Kwon committed almost 3 years ago
[BugFix] Raise error when max_model_len is larger than KV cache (#2163)
Woosuk Kwon committed almost 3 years ago
[Docs] Update documentation for gpu-memory-utilization option (#2162)
Suhong Moon committed almost 3 years ago
Woosuk Kwon committed almost 3 years ago
Disable CUDA graph for SqueezeLLM (#2161)
Woosuk Kwon committed almost 3 years ago