A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 22:00 UTC
main
[Bugfix]fix and needs_scalar_to_array logic check (#6238)
Baoyuan Qi committed over 2 years ago
[Docs] Docs update for Pipeline Parallel (#6222)
Murali Andoorveedu committed over 2 years ago
[CORE] Adding support for insertion of soft-tuned prompts (#4645)
Swapnil Parekh committed over 2 years ago
Add support for multi-node on CI (#5955)
Kevin H. Luu committed over 2 years ago
[Bugfix][TPU] Fix outlines installation in TPU Dockerfile (#6256)
Woosuk Kwon committed over 2 years ago
[Bugfix][TPU] Add missing None to model input (#6245)
Woosuk Kwon committed over 2 years ago
[core][distributed] fix ray worker rank assignment (#6235)
youkaichao committed over 2 years ago
[hardware][cuda] use device id under CUDA_VISIBLE_DEVICES for get_device_capability (#6216)
youkaichao committed over 2 years ago
Add FlashInfer to default Dockerfile (#6172)
Simon Mo committed over 2 years ago
[Bugfix] Mamba cache Cuda Graph padding (#6214)
tomeras91 committed over 2 years ago
[Bugfix] use diskcache in outlines _get_guide #5436 (#6203)
Eric committed over 2 years ago
afeldman-nm committed over 2 years ago
[Kernel] reloading fused_moe config on the last chunk (#6210)
Avshalom Manevich committed over 2 years ago
Feature/add benchmark testing (#5947)
Haichuan committed over 2 years ago
do not exclude `object` field in CompletionStreamResponse (#6196)
kczimm committed over 2 years ago
[misc][frontend] log all available endpoints (#6195)
youkaichao committed over 2 years ago
[ Misc ] Support Fp8 via `llm-compressor` (#6110)
Robert Shaw committed over 2 years ago
add benchmark for fix length input and output (#5857)
Haichuan committed over 2 years ago
Roger Wang committed over 2 years ago
[Doc] Move guide for multimodal model and other improvements (#6168)
Cyrus Leung committed over 2 years ago