A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 18:15 UTC
main
[Misc] Enhance prefix-caching benchmark tool (#6568)
Jiaxin Shan committed about 2 years ago
[Bugfix] spec decode handle None entries in topk args in create_sequence_group_output (#7232)
Travis Johnson committed about 2 years ago
[Misc] update fp8 to use `vLLMParameter` (#7437)
Dipika Sikka committed about 2 years ago
[Bugfix] Don't build machete on cuda <12.0 (#7757)
Lucas Wilkinson committed about 2 years ago
Fix ShardedStateLoader for vllm fp8 quantization (#7708)
Flex Wang committed about 2 years ago
[Speculative Decoding] EAGLE Implementation with Top-1 proposer (#6830)
Abhinav Goyal committed about 2 years ago
[Misc] Use torch.compile for GemmaRMSNorm (#7642)
Woosuk Kwon committed about 2 years ago
[ci] refine dependency for distributed tests (#7776)
youkaichao committed about 2 years ago
[TPU] Avoid initializing TPU runtime in is_tpu (#7763)
Woosuk Kwon committed about 2 years ago
Revert "[Kernel] Expand MoE weight loading + Add Fused Marlin MoE Kernel (#7527)" (#7764)
Michael Goin committed about 2 years ago
[Bug][Frontend] Improve ZMQ client robustness (#7443)
Joe Runde committed about 2 years ago
[Model] Fix Phi-3.5-vision-instruct 'num_crops' issue (#7710)
zifeitong committed about 2 years ago
[Kernel] Replaced `blockReduce[...]` functions with `cub::BlockReduce` (#7233)
Luka Govedič committed about 2 years ago
[AMD][CI/Build] Disambiguation of the function call for ROCm 6.2 headers compatibility (#7477)
Gregory Shtrasberg committed about 2 years ago
[distributed][misc] error on same VLLM_HOST_IP setting (#7756)
youkaichao committed about 2 years ago
[Kernel] Expand MoE weight loading + Add Fused Marlin MoE Kernel (#7527)
Dipika Sikka committed about 2 years ago
[ci] [multi-step] narrow multi-step test dependency paths (#7760)
William Lin committed about 2 years ago
[Model] Add UltravoxModel and UltravoxConfig (#7615)
Peter Salas committed about 2 years ago
[misc] Add Torch profiler support (#7451)
William Lin committed about 2 years ago
[Frontend] Improve Startup Failure UX (#7716)
Robert Shaw committed about 2 years ago