A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 09:30 UTC
main
[CI/Build] Add error matching for ruff output (#9513)
Russell Bryant committed about 2 years ago
[Doc] update gpu-memory-utilization flag docs (#9507)
Joe Runde committed about 2 years ago
[Frontend] Avoid creating guided decoding LogitsProcessor unnecessarily (#9521)
Nick Hill committed about 2 years ago
:bug: fix torch memory profiling (#9516)
Joe Runde committed about 2 years ago
[Bugfix] Fix offline mode when using `mistral_common` (#9457)
sasha0552 committed about 2 years ago
[Kernel] Add env variable to force flashinfer backend to enable tensor cores (#9497)
Thomas Parnell committed about 2 years ago
[MISC] Consolidate cleanup() and refactor offline_inference_with_prefix.py (#9510)
Cody Yu committed about 2 years ago
[MISC] Add lora requests to metrics (#9477)
Kunjan committed about 2 years ago
[Model] Support Pixtral models in the HF Transformers format (#9036)
Michael Goin committed about 2 years ago
[CI/Build] Add error matching config for mypy (#9512)
Russell Bryant committed about 2 years ago
[Model] Add user-configurable task for models that support both generation and embedding (#9424)
Cyrus Leung committed about 2 years ago
[Misc] benchmark: Add option to set max concurrency (#9390)
Russell Bryant committed about 2 years ago
[Bugfix] Fix offline_inference_with_prefix.py (#9505)
Tyler Michael Smith committed about 2 years ago
[CI/Build] Fix lint errors in mistral tokenizer (#9504)
Cyrus Leung committed about 2 years ago
[BugFix] Fix and simplify completion API usage streaming (#9475)
Nick Hill committed about 2 years ago
[Frontend][Feature] Add jamba tool parser (#9154)
tomeras91 committed about 2 years ago
[BugFix] Typing fixes to RequestOutput.prompt and beam search (#9473)
Nick Hill committed about 2 years ago
[CI/Build] Use commit hash references for github actions (#9430)
Russell Bryant committed about 2 years ago
[Qwen2.5] Support bnb quant for Qwen2.5 (#9467)
Haoyu Wang committed about 2 years ago
[Bugfix][Core] Use torch.cuda.memory_stats() to profile peak memory usage (#9352)
Joe Runde committed about 2 years ago