A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 05:45 UTC
main
[Bugfix][Mamba] Fix Multistep on Mamba-like models (#10705)
Mor Zusman committed almost 2 years ago
[Bugfix] Fix GGUF inference with FP16 unquantized checkpoint (#10675)
Isotr0py committed almost 2 years ago
[torch.compile] fix shape specialization (#10722)
youkaichao committed almost 2 years ago
youkaichao committed almost 2 years ago
[V1] Update interface for mistral-format Pixtral (#10703)
Roger Wang committed almost 2 years ago
[Bugfix][Hardware][CPU] Fix intel-omp version to avoid segfault (#10700)
Li, Jiang committed almost 2 years ago
[Model] Support telechat2 (#10311)
shunxing12345 committed almost 2 years ago
[Kernel] Remove if-else with identical branches in marlin 2:4 (#10687)
Tyler Michael Smith committed almost 2 years ago
[Misc]Further reduce BNB static variable (#10597)
Jee Jee Li committed almost 2 years ago
[Hardware][Gaudi]add get_name method for HPUAttentionBackend (#10667)
Kunshang Ji committed almost 2 years ago
[bugfix] fix the default value of llm_int8_threshold in BitsAndBytesConfig (#10657)
yansh97 committed almost 2 years ago
jeongin601 committed almost 2 years ago
[V1] Update interface for idefics3 (#10680)
Roger Wang committed almost 2 years ago
Remove hard-dependencies of Speculative decode to CUDA workers (#10587)
Chendi.Xue committed almost 2 years ago
[V1] Refactor model executable interface for multimodal models (#10570)
Roger Wang committed almost 2 years ago
[Bugfix] Check bnb_4bit_quant_storage for bitsandbytes (#10642)
Michael Goin committed almost 2 years ago
[Bugfix] Fix using `-O[0,3]` with LLM entrypoint (#10677)
Michael Goin committed almost 2 years ago
[Hardware][NVIDIA] Add non-NVML CUDA mode for Jetson (#9735)
Conroy Cheers committed almost 2 years ago
[Bugfix] Fix for Spec model TP + Chunked Prefill (#10232)
Murali Andoorveedu committed almost 2 years ago
[V1] Enable profile for LLMEngine (#10665)
Kunshang Ji committed almost 2 years ago