A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 09:30 UTC
main
[BugFix] Stop silent failures on compressed-tensors parsing (#9381)
Dipika Sikka committed about 2 years ago
Support `BERTModel` (first `encoder-only` embedding model) (#9056)
Robert Shaw committed about 2 years ago
[Hardwware][Neuron] Simplify model load for transformers-neuronx library (#9380)
Shashwat Srijan committed about 2 years ago
[Bugfix] Print warnings related to `mistral_common` tokenizer only once (#9468)
sasha0552 committed about 2 years ago
[Bugfix] Add random_seed to sample_hf_requests in benchmark_serving script (#9013)
Kai Wu committed about 2 years ago
[Bugfix] Fix support for dimension like integers and ScalarType (#9299)
bnellnm committed about 2 years ago
[torch.compile] Fine-grained CustomOp enabling mechanism (#9300)
Luka Govedič committed about 2 years ago
[Misc] Remove commit id file (#9470)
Cyrus Leung committed about 2 years ago
[CI/Build] remove .github from .dockerignore, add dirty repo check (#9375)
Daniele committed about 2 years ago
[Core] Deprecating block manager v1 and make block manager v2 default (#8704)
Kuntai Du committed about 2 years ago
[Hardware][CPU] compressed-tensor INT8 W8A8 AZP support (#9344)
Li, Jiang committed about 2 years ago
[TPU] Call torch._sync(param) during weight loading (#9437)
Woosuk Kwon committed about 2 years ago
[Bugfix] Allow prefill of assistant response when using `mistral_common` (#9446)
sasha0552 committed about 2 years ago
[misc] CUDA Time Layerwise Profiler (#8337)
Lucas Wilkinson committed about 2 years ago
[Misc] Print stack trace using `logger.exception` (#9461)
Cyrus Leung committed about 2 years ago
[Kernel] Add Exllama as a backend for compressed-tensors (#9395)
Lucas Wilkinson committed about 2 years ago
Add notes on the use of Slack (#9442)
Yuan Tang committed about 2 years ago
[BugFix] [Kernel] Fix GPU SEGV occurring in int8 kernels (#9391)
rasmith committed about 2 years ago
[Bugfix][Kernel] Prevent integer overflow in fp8 dynamic per-token quantize kernel (#9425)
Tyler Michael Smith committed about 2 years ago
[CI/Build] mypy: Resolve some errors from checking vllm/engine (#9267)
Russell Bryant committed about 2 years ago