A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 00:45 UTC
main
[ROCM][AMD][TRITON] Halving warps number for fw_prefill to reduce spilling (#12713)
Aleksandr Malyshev committed almost 2 years ago
[Core] add and implement `VLLM_LOGITS_PROCESSOR_THREADS` (#12368)
Aviv Keshet committed almost 2 years ago
[Perf] Mem align KV caches for CUDA devices (MLA perf improvement) (#12676)
Lucas Wilkinson committed almost 2 years ago
[V1][Metrics] Add request_success_total counter, labelled with finish reason (#12579)
Mark McLoughlin committed almost 2 years ago
[Bugfix] Fix CI failures for InternVL and Mantis models (#12728)
Cyrus Leung committed almost 2 years ago
[Build] update requirements of no-device for plugin usage (#12630)
Sophie du Couédic committed almost 2 years ago
Avoid unnecessary multi-modal input data copy when len(batch) == 1 (#12722)
Kero Liang committed almost 2 years ago
[Bugfix] Fix loading of fine-tuned models based on Phi-3-Small (#12689)
Michael Greenbaum committed almost 2 years ago
[VLM] merged multimodal processor and V1 support for idefics3 (#12660)
Isotr0py committed almost 2 years ago
[V1] Remove scheduling constraint on partial requests (#12674)
Woosuk Kwon committed almost 2 years ago
[VLM] Merged multi-modal processor for InternVL-based models (#12553)
Cyrus Leung committed almost 2 years ago
[Misc] Add BNB quantization for Whisper (#12381)
Jee Jee Li committed almost 2 years ago
[AMD][ROCm] Enable DeepSeek model on ROCm (#12662)
Hongxia Yang committed almost 2 years ago
[Quant] Fix use_mla TypeError and support loading pure-sparsity Compressed Tensors configs (#12711)
Kyle Sayers committed almost 2 years ago
[Doc] Replace ibm-fms with ibm-ai-platform (#12709)
Thomas Parnell committed almost 2 years ago
Support Pixtral-Large HF by using llava multimodal_projector_bias config (#12710)
Michael Goin committed almost 2 years ago
[Core] Improve hash collision avoidance in prefix caching (#12621)
Russell Bryant committed almost 2 years ago
[V1] Revert `uncache_blocks` and support recaching full blocks (#12415)
Cody Yu committed almost 2 years ago
[MISC] Remove model input dumping when exception (#12582)
Cody Yu committed almost 2 years ago
[Model] Add Deepseek V3 fp8_w8a8 configs for B200 (#12707)
kushanam committed almost 2 years ago