A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 00:45 UTC
main
Update `pre-commit` hooks (#12475)
Harry Mellor committed almost 2 years ago
Nicolò Lucchesi committed almost 2 years ago
[FlashInfer] Upgrade to 0.2.0 (#11194)
Bowen Wang committed almost 2 years ago
[V1][CI/Test] Do basic test for top-p & top-k sampling (#12469)
Woosuk Kwon committed almost 2 years ago
[V1][Metrics] Add initial Prometheus logger (#12416)
Mark McLoughlin committed almost 2 years ago
[Build] Only build 9.0a for scaled_mm and sparse kernels (#12339)
Lucas Wilkinson committed almost 2 years ago
[Bugfix] Fix gpt2 GGUF inference (#12467)
Isotr0py committed almost 2 years ago
[V1][Minor] Minor optimizations for update_from_output (#12454)
Woosuk Kwon committed almost 2 years ago
[Bugfix] Fix missing seq_start_loc in xformers prefill metadata (#12464)
Isotr0py committed almost 2 years ago
[Bugfix] Fix Granite 3.0 MoE model loading (#12446)
Cyrus Leung committed almost 2 years ago
[Frontend] Support scores endpoint in run_batch (#12430)
Pooya Davoodi committed almost 2 years ago
[V1] Avoid list creation in input preparation (#12457)
Woosuk Kwon committed almost 2 years ago
[DOC] Add link to vLLM blog (#12460)
Yuan Tang committed almost 2 years ago
[Frontend] Rerank API (Jina- and Cohere-compatible API) (#12376)
Kyle Mistele committed almost 2 years ago
[Build/CI] Fix libcuda.so linkage (#12424)
Tyler Michael Smith committed almost 2 years ago
[Bugfix][Kernel] Fix perf regression caused by PR #12405 (#12434)
Lucas Wilkinson committed almost 2 years ago
[Bugfix/CI] Fix broken kernels/test_mha.py (#12450)
Tyler Michael Smith committed almost 2 years ago
[Bugfix] Disable w16a16 2of4 sparse CompressedTensors24 (#12417)
Tyler Michael Smith committed almost 2 years ago
[Frontend] generation_config.json for maximum tokens(#12242)
Matthew Hendrey committed almost 2 years ago
[Misc] Revert FA on ViT #12355 and #12435 (#12445)
Roger Wang committed almost 2 years ago