A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 01:45 UTC
main
[Doc] Chunked Prefill Documentation (#4580)
SangBin Cho committed over 2 years ago
[Misc][Refactor] Introduce ExecuteModelData (#4540)
Cody Yu committed over 2 years ago
[Misc] add installation time env vars (#4574)
youkaichao committed over 2 years ago
[Speculative decoding] Support target-model logprobs (#4378)
Cade Daniel committed over 2 years ago
[Kernel] Use flashinfer for decoding (#4353)
Lily Liu committed over 2 years ago
Fix/async chat serving (#2727)
Sebastian Schoennenbeck committed over 2 years ago
[Bugfix] Allow "None" or "" to be passed to CLI for string args that default to None (#4586)
Michael Goin committed over 2 years ago
[Core][Model runner refactoring 1/N] Refactor attn metadata term (#4518)
SangBin Cho committed over 2 years ago
[Doc] add env vars to the doc (#4572)
youkaichao committed over 2 years ago
[Misc] remove chunk detected debug logs (#4571)
DefTruth committed over 2 years ago
[BugFix] Prevent the task of `_force_log` from being garbage collected (#4567)
Yang, Bo committed over 2 years ago
[Core][Distributed] enable allreduce for multiple tp groups (#4566)
youkaichao committed over 2 years ago
[Core] Ignore infeasible swap requests. (#4557)
SangBin Cho committed over 2 years ago
[CI/Build] AMD CI pipeline with extended set of tests. (#4267)
Alexei-V-Ivanov-AMD committed over 2 years ago
[kernel] fix sliding window in prefix prefill Triton kernel (#4405)
Michał Moskal committed over 2 years ago
[Misc] centralize all usage of environment variables (#4548)
youkaichao committed over 2 years ago
[BugFix] Include target-device specific requirements.txt in sdist (#4559)
Mark McLoughlin committed over 2 years ago
[Misc] Exclude the `tests` directory from being packaged (#4552)
Hu Dong committed over 2 years ago
[mypy][7/N] Cover all directories (#4555)
SangBin Cho committed over 2 years ago
[Kernel] Support running GPTQ 8-bit models in Marlin (#4533)
alexm-nm committed over 2 years ago