A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 20:45 UTC
main
[misc][distributed] fix pp missing layer condition (#6446)
youkaichao committed over 2 years ago
bump version to v0.5.2 (#6433)
Simon Mo committed over 2 years ago
[Bugfix] use float32 precision in samplers/test_logprobs.py for comparing with HF (#6409)
Thomas Parnell committed over 2 years ago
[Bugfix] Add custom Triton cache manager to resolve MoE MP issue (#6140)
Thomas Parnell committed over 2 years ago
[Misc] Use 0.0.9 version for flashinfer (#6447)
Pernekhan Utemuratov committed over 2 years ago
[Kernel] Use CUTLASS kernels for the FP8 layers with Bias (#6270)
Tyler Michael Smith committed over 2 years ago
[doc][distributed] add suggestion for distributed inference (#6418)
youkaichao committed over 2 years ago
[VLM] Minor space optimization for `ClipVisionModel` (#6436)
Roger Wang committed over 2 years ago
[doc][misc] doc update (#6439)
youkaichao committed over 2 years ago
[Bugfix] Convert image to RGB by default (#6430)
Cyrus Leung committed over 2 years ago
[core][distributed] simplify code to support pipeline parallel (#6406)
youkaichao committed over 2 years ago
[Doc] add env docs for flashinfer backend (#6437)
DefTruth committed over 2 years ago
[BugFix] BatchResponseData body should be optional (#6345)
zifeitong committed over 2 years ago
Simon Mo committed over 2 years ago
Report usage for beam search (#6404)
Simon Mo committed over 2 years ago
Fish committed over 2 years ago
[CI/Build] Cross python wheel (#6394)
Robert Shaw committed over 2 years ago
[Doc] xpu backend requires running setvars.sh (#6393)
Robert Cohn committed over 2 years ago
Ethan Xu committed over 2 years ago
[ Misc ] Enable Quantizing All Layers of DeekSeekv2 (#6423)
Robert Shaw committed over 2 years ago