A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
main
Disable custom all reduce by default (#2808)
Woosuk Kwon committed almost 3 years ago
[Ray] Integration compiled DAG off by default (#2471)
SangBin Cho committed almost 3 years ago
Add documentation on how to do incremental builds (#2796)
Philipp Moritz committed almost 3 years ago
[ROCm] Fix build problem resulted from previous commit related to FP8 kv-cache support (#2790)
Hongxia Yang committed almost 3 years ago
[Minor] More fix of test_cache.py CI test failure (#2750)
Lily Liu committed almost 3 years ago
modelscope: fix issue when model parameter is not a model id but path of the model. (#2489)
liuyhwangyh committed almost 3 years ago
Add fused top-K softmax kernel for MoE (#2769)
Woosuk Kwon committed almost 3 years ago
[ROCm] Fixup arch checks for ROCM (#2627)
Douglas Lehr committed almost 3 years ago
Set local logging level via env variable (#2774)
Lukas committed almost 3 years ago
[ROCm] Fix some kernels failed unit tests (#2498)
Hongxia Yang committed almost 3 years ago
[Minor] Fix benchmark_latency script (#2765)
Woosuk Kwon committed almost 3 years ago
Require triton >= 2.1.0 (#2746)
whyiug committed almost 3 years ago
Remove eos tokens from output by default (#2611)
Rex committed almost 3 years ago
set&get llm internal tokenizer instead of the TokenizerGroup (#2741)
dancingpipi committed almost 3 years ago
Massimiliano Pronesti committed almost 3 years ago
Add one example to run batch inference distributed on Ray (#2696)
Cheng Su committed almost 3 years ago
Fix default length_penalty to 1.0 (#2667)
zspo committed almost 3 years ago
Remove hardcoded `device="cuda" ` to support more devices (#2503)
Kunshang Ji committed almost 3 years ago
Use revision when downloading the quantization config file (#2697)
Pernekhan Utemuratov committed almost 3 years ago
Update README for meetup slides (#2718)
Simon Mo committed almost 3 years ago