A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 20:45 UTC
main
[Core] Fix ray forward_dag error mssg (#6792)
SangBin Cho committed over 2 years ago
[Docs] Publish 5th meetup slides (#6799)
Woosuk Kwon committed over 2 years ago
[doc][distributed] improve multinode serving doc (#6804)
youkaichao committed over 2 years ago
[Bugfix] Fix empty (nullptr) channelwise scales when loading wNa16 using compressed tensors (#6798)
Lucas Wilkinson committed over 2 years ago
[Doc] Add documentations for nightly benchmarks (#6412)
Kuntai Du committed over 2 years ago
[Bugfix] Add synchronize to prevent possible data race (#6788)
Tyler Michael Smith committed over 2 years ago
[Bugfix] Fix `kv_cache_dtype=fp8` without scales for FP8 checkpoints (#6761)
Michael Goin committed over 2 years ago
[ Misc ] `fp8-marlin` channelwise via `compressed-tensors` (#6524)
Robert Shaw committed over 2 years ago
[Bugfix] Add image placeholder for OpenAI Compatible Server of MiniCPM-V (#6787)
Alphi committed over 2 years ago
[Bugfix] Fix encoding_format in examples/openai_embedding_client.py (#6755)
Chang Su committed over 2 years ago
[Bugfix] Fix awq_marlin and gptq_marlin flags (#6745)
Alexander Matveev committed over 2 years ago
[Bugfix] Fix decode tokens w. CUDA graph (#6757)
Cody Yu committed over 2 years ago
[Model] Adding support for MiniCPM-V (#4087)
Alphi committed over 2 years ago
[Frontend] Represent tokens with identifiable strings (#6626)
Evan Z. Liu committed over 2 years ago
[core][distributed] fix zmq hang (#6759)
youkaichao committed over 2 years ago
[Doc][AMD][ROCm]Added tips to refer to mi300x tuning guide for mi300x users (#6754)
Hongxia Yang committed over 2 years ago
[Bugfix] Bump transformers to 4.43.2 (#6752)
Michael Goin committed over 2 years ago
[Core] Tweaks to model runner/input builder developer APIs (#6712)
Antoni Baum committed over 2 years ago
Add fp8 support to `reshape_and_cache_flash` (#6667)
Antoni Baum committed over 2 years ago
[Frontend] split run_server into build_server and run_server (#6740)
Daniele committed over 2 years ago