A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 15:45 UTC
main
[misc] soft drop beam search (#8763)
youkaichao committed about 2 years ago
[CI/Build] fix setuptools-scm usage (#8771)
Daniele committed about 2 years ago
[Bugfix] Fix torch dynamo fixes caused by `replace_parameters` (#8748)
Lucas Wilkinson committed about 2 years ago
[Frontend] Batch inference for llm.chat() API (#8648)
Andy committed about 2 years ago
[Kernel] Split Marlin MoE kernels into multiple files (#8661)
ElizaWszola committed about 2 years ago
[Bugfix] Fix potentially unsafe custom allreduce synchronization (#8558)
Hanzhi Zhou committed about 2 years ago
[Model] Expose Phi3v num_crops as a mm_processor_kwarg (#8658)
Alex Brooks committed about 2 years ago
[Core][Model] Support loading weights by ID within models (#7931)
Peter Salas committed about 2 years ago
[MISC] Skip dumping inputs when unpicklable (#8744)
Cody Yu committed about 2 years ago
Revert "[Core] Rename `PromptInputs` to `PromptType`, and `inputs` to `prompt`" (#8750)
Simon Mo committed about 2 years ago
re-implement beam search on top of vllm core (#8726)
youkaichao committed about 2 years ago
Fix tests in test_scheduler.py that fail with BlockManager V2 (#8728)
sroy745 committed about 2 years ago
[Hardware][AMD] ROCm6.2 upgrade (#8674)
Hongxia Yang committed about 2 years ago
Add output streaming support to multi-step + async while ensuring RequestOutput obj reuse (#8335)
Alexander Matveev committed about 2 years ago
Fix typical acceptance sampler with correct recovered token ids (#8562)
jiqing-feng committed about 2 years ago
[Core] Allow IPv6 in VLLM_HOST_IP with zmq (#8575)
Russell Bryant committed about 2 years ago
[Kernel][LoRA] Add assertion for punica sgmv kernels (#7585)
Jee Jee Li committed about 2 years ago
[Kernel] (2/N) Machete - Integrate into CompressedTensorsWNA16 and GPTQMarlin (#7701)
Lucas Wilkinson committed about 2 years ago
[CI/Build] use setuptools-scm to set __version__ (#4738)
Daniele committed about 2 years ago
Jani Monoses committed about 2 years ago