A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 00:30 UTC
main
[Docs] Add Sequoia as sponsors (#5287)
Simon Mo committed over 2 years ago
[Kernel] Re-tune Mixtral MoE configurations for FP8 on H100 (#5238)
Philipp Moritz committed over 2 years ago
[BugFix] Apply get_cached_tokenizer to the tokenizer setter of LLM (#5207)
DriverSong committed over 2 years ago
[Model] Correct Mixtral FP8 checkpoint loading (#5231)
Cody Yu committed over 2 years ago
[Kernel] Add GPU architecture guards to the CUTLASS w8a8 kernels to reduce binary size (#5157)
Tyler Michael Smith committed over 2 years ago
[misc] benchmark_serving.py -- add ITL results and tweak TPOT results (#5263)
Tyler Michael Smith committed over 2 years ago
[CI] Add nightly benchmarks (#5260)
Simon Mo committed over 2 years ago
tomeras91 committed over 2 years ago
[Misc] Fix docstring of get_attn_backend (#5271)
Woosuk Kwon committed over 2 years ago
[Misc] Add CustomOp interface for device portability (#5255)
Woosuk Kwon committed over 2 years ago
[Bugfix] Fix prompt_logprobs when SamplingParams.detokenize is set to True (#5226)
zifeitong committed over 2 years ago
[Misc] update collect env (#5261)
youkaichao committed over 2 years ago
[Misc] Add transformers version to collect_env.py (#5259)
Michael Goin committed over 2 years ago
[CI] mark AMD test as softfail to prevent blockage (#5256)
Simon Mo committed over 2 years ago
[CI/Build] Reducing CPU CI execution time (#5241)
Li, Jiang committed over 2 years ago
[CI/Build] Simplify model loading for `HfRunner` (#5251)
Cyrus Leung committed over 2 years ago
[Kernel] Add back batch size 1536 and 3072 to MoE tuning (#5242)
Woosuk Kwon committed over 2 years ago
[Bugfix] Fix a bug caused by pip install setuptools>=49.4.0 for CPU backend (#5249)
Jie Fu (傅杰) committed over 2 years ago
[CI/Build] Add inputs tests (#5215)
Cyrus Leung committed over 2 years ago
[Bugfix] Fix torch.compile() error when using MultiprocessingGPUExecutor (#5229)
zifeitong committed over 2 years ago