A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
main
[Model] Initialize Phi-3-vision support (#4986)
Isotr0py committed over 2 years ago
sroy745 committed over 2 years ago
[Fix] Use utf-8 encoding in entrypoints/openai/run_batch.py (#5606)
zifeitong committed over 2 years ago
[Bugfix] Fix KV head calculation for MPT models when using GQA (#5142)
Bruce Fontaine committed over 2 years ago
[Optimization] use a pool to reuse LogicalTokenBlock.token_ids (#5584)
youkaichao committed over 2 years ago
[bugfix][distributed] fix 16 gpus local rank arrangement (#5604)
youkaichao committed over 2 years ago
[CI] the readability of benchmarking and prepare for dashboard (#5571)
Kuntai Du committed over 2 years ago
[CI/BUILD] Support non-AVX512 vLLM building and testing (#5574)
Jie Fu (傅杰) committed over 2 years ago
[Hardware][Intel GPU] Add Intel GPU(XPU) inference backend (#3814)
Kunshang Ji committed over 2 years ago
[Misc] use AutoTokenizer for benchmark serving when vLLM not installed (#5588)
zhyncs committed over 2 years ago
[Kernel] `compressed-tensors` marlin 24 support (#5435)
Dipika Sikka committed over 2 years ago
Correct alignment in the seq_len diagram. (#5592)
Charles Riggins committed over 2 years ago
[Model] Rename Phi3 rope scaling type (#5595)
Amit Garg committed over 2 years ago
Fix w8a8 benchmark and add Llama-3-8B (#5562)
Cody Yu committed over 2 years ago
[Doc] add debugging tips for crash and multi-node debugging (#5581)
youkaichao committed over 2 years ago
[build][misc] limit numpy version (#5582)
youkaichao committed over 2 years ago
[CI][BugFix] Flip is_quant_method_supported condition (#5577)
Michael Goin committed over 2 years ago
Add basic correctness 2 GPU tests to 4 GPU pipeline (#5518)
Antoni Baum committed over 2 years ago
[Fix] Correct OpenAI batch response format (#5554)
zifeitong committed over 2 years ago
[BugFix] Don't start a Ray cluster when not using Ray (#5570)
Nick Hill committed over 2 years ago