A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 18:15 UTC
main
[Bugfix] Fix custom_ar support check (#7617)
bnellnm committed over 2 years ago
[Model] Align nemotron config with final HF state and fix lm-eval-small (#7611)
Michael Goin committed over 2 years ago
[Kernel] fix types used in aqlm and ggml kernels to support dynamo (#7596)
bnellnm committed over 2 years ago
[Kernel][Misc] dynamo support for ScalarType (#7594)
bnellnm committed over 2 years ago
[Kernel] register punica functions as torch ops (#7591)
bnellnm committed over 2 years ago
[Doc] Update quantization supported hardware table (#7595)
Michael Goin committed over 2 years ago
[Doc] Add docs for llmcompressor INT8 and FP8 checkpoints (#7444)
Michael Goin committed over 2 years ago
[Core] Fix tracking of model forward time in case of PP>1 (#7440)
Mahesh Keralapura committed over 2 years ago
[spec decode] [4/N] Move update_flash_attn_metadata to attn backend (#7571)
William Lin committed over 2 years ago
[Kernel] Add tuned triton configs for ExpertsInt8 (#7601)
Michael Goin committed over 2 years ago
[Kernel] W8A16 Int8 inside FusedMoE (#7415)
Mor Zusman committed over 2 years ago
[Feature][Hardware][Amd] Add fp8 Linear Layer for Rocm (#7210)
Charlie Fu committed over 2 years ago
support tqdm in notebooks (#7510)
fzyzcjy committed over 2 years ago
[Bugfix][Hardware][AMD][Frontend] add quantization param to embedding checking method (#7513)
Gordon Wong committed over 2 years ago
[CI] Fix crashes of performance benchmark (#7500)
Kuntai Du committed over 2 years ago
[Core] Use uvloop with zmq-decoupled front-end (#7570)
Nick Hill committed over 2 years ago
register custom op for flash attn and use from torch.ops (#7536)
youkaichao committed over 2 years ago
[Misc/Testing] Use `torch.testing.assert_close` (#7324)
jon-chuang committed over 2 years ago
[CI] Move quantization cpu offload tests out of fastcheck (#7574)
Michael Goin committed over 2 years ago
Chat method for offline llm (#5049)
nunjunj committed over 2 years ago