A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 01:45 UTC
main
[CI] clean docker cache for neuron (#4441)
Simon Mo committed over 2 years ago
Add more Prometheus metrics (#2764)
Ronen Schaffer committed over 2 years ago
[Misc] fix typo in llm_engine init logging (#4428)
DefTruth committed over 2 years ago
[BugFix] Resolved Issues For LinearMethod --> QuantConfig (#4418)
Robert Shaw committed over 2 years ago
[BugFix] Fix return type of executor execute_model methods (#4402)
Nick Hill committed over 2 years ago
[Core] Support offline use of local cache for models (#4374)
Prashant Gupta committed over 2 years ago
[BugFix] Fix `min_tokens` when `eos_token_id` is None (#4389)
Nick Hill committed over 2 years ago
[Bugfix] Abort requests when the connection to /v1/completions is interrupted (#4363)
Ruoyu Qin committed over 2 years ago
[Bugfix][Core] Fix get decoding config from ray (#4335)
Roy committed over 2 years ago
[Model] Phi-3 4k sliding window temp. fix (#4380)
Caio Mendes committed over 2 years ago
[Kernel] Full Tensor Parallelism for LoRA Layers (#3524)
Austin Veselka committed over 2 years ago
[ROCm][Hardware][AMD] Enable group query attention for triton FA (#4406)
Hongxia Yang committed over 2 years ago
[Misc] Fix logger format typo (#4396)
Roy committed over 2 years ago
[Frontend][Bugfix] Disallow extra fields in OpenAI API (#4355)
Cyrus Leung committed over 2 years ago
[Kernel] Optimize FP8 support for MoE kernel / Mixtral via static scales (#4343)
Philipp Moritz committed over 2 years ago
[Core] Introduce `DistributedGPUExecutor` abstract class (#4348)
Nick Hill committed over 2 years ago
[Misc] add RFC issue template (#4401)
youkaichao committed over 2 years ago
[Misc][Refactor] Generalize linear_method to be quant_method (#4373)
Cody Yu committed over 2 years ago
[Core] Refactoring sampler and support prompt logprob for chunked prefill (#4309)
SangBin Cho committed over 2 years ago
[CI] Disable non-lazy string operation on logging (#4326)
SangBin Cho committed over 2 years ago