A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 06:00 UTC
main
[Docs] Note that alternative structured output backends are supported (#19426)
Russell Bryant committed over 1 year ago
[Misc] Slight improvement of the BNB (#19418)
Jee Jee Li committed over 1 year ago
Simplify ep kernels installation (#19412)
youkaichao committed over 1 year ago
[BugFix][CPU] Fix CPU CI by ignore collecting test_pixtral (#19411)
Li, Jiang committed over 1 year ago
Rachel Guo committed over 1 year ago
Revert "[v1] Add fp32 support to v1 engine through flex attn" (#19404)
Isotr0py committed over 1 year ago
Automatically bind CPU OMP Threads of a rank to CPU ids of a NUMA node. (#17930)
Louie Tsai committed over 1 year ago
Add clear documentation around the impact of debugging flag (#19369)
Anna Pendleton committed over 1 year ago
[Misc] refactor neuron_multimodal and profiling (#19397)
Reid committed over 1 year ago
Add security warning to bug report template (#19365)
Russell Bryant committed over 1 year ago
[Core] Batch multi modal input using pinned memory (#19169)
Lukas Geiger committed over 1 year ago
[Bugfix] Fix modelscope token passed in (#19389)
Li Wang committed over 1 year ago
[Core] Use tuple for kv cache group block ids (#19175)
Nick Hill committed over 1 year ago
[Frontend] Add tqdm_leave_pbar to control progress bar visibility (#19357)
Reid committed over 1 year ago
Fix docs/mkdocs/hooks/remove_announcement.py (#19382)
Harry Mellor committed over 1 year ago
Use xla flag to improve the quantized model performance (#19303)
XiongfeiWei committed over 1 year ago
[Bugfix] Fix benchmark_moe.py (#19016)
Tianyu Guo committed over 1 year ago
[V1] Reuse V0's memory_profiling util for gpu worker memory profiling (#19312)
Ye (Charlotte) Qi committed over 1 year ago
[Misc] Fix a config typo in disable_hybrid_kv_cache_manager configuration (#19383)
Siyuan Liu committed over 1 year ago
[HOT-FIX] Add `kv_sharing_target_layer_name` argument to cutlass_mla backend (#19374)
Pavani Majety committed over 1 year ago