A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 19:30 UTC
main
[TPU] Use mark_dynamic to reduce compilation time (#7340)
Woosuk Kwon committed over 2 years ago
[Bugfix] Fix phi3v batch inference when images have different aspect ratio (#7392)
Isotr0py committed over 2 years ago
[Core] Fix edge case in chunked prefill + block manager v2 (#7380)
Cade Daniel committed over 2 years ago
[Misc] Add numpy implementation of `compute_slot_mapping` (#7377)
Antoni Baum committed over 2 years ago
[Bugfix] Fix `PerTensorScaleParameter` weight loading for fused models (#7376)
Dipika Sikka committed over 2 years ago
[Core] Add span metrics for model_forward, scheduler and sampler time (#7089)
Mahesh Keralapura committed over 2 years ago
[Bugfix] Fix ITL recording in serving benchmark (#7372)
Roger Wang committed over 2 years ago
[Frontend] Support embeddings in the run_batch API (#7132)
Pooya Davoodi committed over 2 years ago
[Bugfix] Fix reinit procedure in ModelInputForGPUBuilder (#7360)
Alexander Matveev committed over 2 years ago
[Performance] e2e overheads reduction: Small followup diff (#7364)
Alexander Matveev committed over 2 years ago
[VLM][Doc] Add `stop_token_ids` to InternVL example (#7354)
Isotr0py committed over 2 years ago
[Model][Jamba] Mamba cache single buffer (#6739)
Mor Zusman committed over 2 years ago
[Core] Streamline stream termination in `AsyncLLMEngine` (#7336)
Nick Hill committed over 2 years ago
[Speculative decoding] [Multi-Step] decouple should_modify_greedy_probs_inplace (#6971)
William Lin committed over 2 years ago
[Bugfix] Fix speculative decoding with MLPSpeculator with padded vocabulary (#7218)
Travis Johnson committed over 2 years ago
[Performance] Optimize e2e overheads: Reduce python allocations (#7162)
Alexander Matveev committed over 2 years ago
[TPU] Fix dockerfile.tpu (#7331)
Woosuk Kwon committed over 2 years ago
[Core] Support serving encoder/decoder models (#7258)
Cyrus Leung committed over 2 years ago
[TPU] Add Load-time W8A16 quantization for TPU Backend (#7005)
Siyuan Liu committed over 2 years ago
Add Skywork AI as Sponsor (#7314)
Simon Mo committed over 2 years ago