A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 16:00 UTC
main
[ROCm][Bugfix] Bring back fallback to eager mode removed in #14917, but for ROCm only (#15413)
Gregory Shtrasberg committed over 1 year ago
[ROCm][Bugfix] Use platform specific FP8 dtype (#15717)
Gregory Shtrasberg committed over 1 year ago
[Distributed] [ROCM] Fix custom allreduce enable checks (#16010)
Ilya Markov committed over 1 year ago
[CPU] Change default block_size for CPU backend (#16002)
Li, Jiang committed over 1 year ago
[Benchmark][Doc] Update throughput benchmark and README (#15998)
Ziji Shi (Steven) committed over 1 year ago
[Bugfix][kernels] Fix half2float conversion in gguf kernels (#15995)
Isotr0py committed over 1 year ago
[Hardware][Gaudi][BugFix] fix arguments of hpu fused moe (#15945)
liuzhenwei committed over 1 year ago
[Model] use AutoWeightsLoader for baichuan, gpt-neox, mpt (#15939)
Jonghyun Choe committed over 1 year ago
[V1][Spec Decode] Avoid logging useless nan metrics (#16023)
Mark McLoughlin committed over 1 year ago
doc: add info for macos clang errors (#16049)
yihong committed over 1 year ago
[TPU][V1] Remove ragged attention kernel parameter hard coding (#16041)
Chengji Yao committed over 1 year ago
[Misc] improve gguf check (#15974)
Reid committed over 1 year ago
[ROCm] Tweak the benchmark script to run on ROCm (#14252)
Huy Do committed over 1 year ago
[Bugfix] Fix function names in test_block_fp8.py (#16033)
bnellnm committed over 1 year ago
[TPU] Switch Test to Non-Sliding Window (#15981)
Robert Shaw committed over 1 year ago
[TPU] Support sliding window and logit soft capping in the paged attention kernel for TPU. (#15732)
iefgnoix committed over 1 year ago
Re-enable the AMD Testing for the passing tests. (#15586)
Alexei-V-Ivanov-AMD committed over 1 year ago
[Misc][Performance] Advance tpu.txt to the most recent nightly torch … (#16024)
yarongmu-google committed over 1 year ago
[Minor] Fused experts refactor (#15914)
bnellnm committed over 1 year ago
[Neuron][kernel] Fuse kv cache into a single tensor (#15911)
Liangfu Chen committed over 1 year ago