A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 3, 11:45 UTC
main
Support bf16 for mamba ssm cache (#41680)
Qi Zhou committed 6 months ago
Fix Weight loading for Qwen3.5-MTP and Qwen3-VL using runai_streamer (#42716)
weizhoublue committed 6 months ago
[CI/Build] Bump flashinfer to v0.6.11.post2 (#41711)
Artem Perevedentsev committed 6 months ago
[KV Connector] Support disk offloading in MooncakeStoreConnector (#42689)
Zhewen Li committed 6 months ago
Add unit tests for pooler activation functions (#42824)
Taneem Ibrahim committed 6 months ago
Fix: Propagate pinned model revisions into Ultravox secondary weight loading (#42830)
weizhoublue committed 6 months ago
[Experimental] Breakable CUDA graph (#42304)
Jiangyun Zhu committed 6 months ago
[ROCm][CI] Removed problematic command override mechanism (#42807)
Andreas Karatzas committed 6 months ago
[Docker][KVConnector] Build mooncake-transfer-engine from source (#42114)
Zhewen Li committed 6 months ago
Jee Jee Li committed 6 months ago
fix: add API key authorization to /v2 endpoints (#42594)
DustHunter committed 6 months ago
[Misc] Add common random prefix option to structured-output serving benchmark (#41632)
Viktor Pus committed 6 months ago
[Bugfix] Respect explicit --kv-cache-dtype over checkpoint kv_cache_scheme (#42782)
Michael Goin committed 6 months ago
[Kernel][UX] Add `--linear-backend` arg for linear kernel selection (#39538)
Michael Goin committed 6 months ago
[Bugfix] Unwrap VLM wrappers for EPLB on Model Runner V2 (#42706)
Keyi Li committed 6 months ago
[Bugfix] Fix layerwise reload alias-buffer corruption (#42481)
rasdani committed 6 months ago
[ROCm][Bugfix] Fix fused_mla_dual_rms_norm for AITER API rename _fused_qk_rmsnorm (#42606)
Rita Brugarolas committed 6 months ago
[Build] Switch CUDA 12.9 wheel builds to PyTorch manylinux_2_28 base (#41668)
Michael Goin committed 6 months ago
Add HumanEval and GSM8K benchmarks to datasets (#42648)
Sergei Skvortsov committed 6 months ago
[FlashAttn] Fix supports_kv_cache_dtype() accepting unhandled fp8 kv-cache dtype variants (#42685)
Lanze Liu committed 6 months ago