A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 15:45 UTC
main
Simon Mo committed about 2 years ago
[Doc] Add documentation for GGUF quantization (#8618)
Isotr0py committed about 2 years ago
[Core] simplify logits resort in _apply_top_k_top_p (#8619)
盏一 committed about 2 years ago
[Kernel][Amd] Add fp8 kv cache support for rocm custom paged attention (#8577)
Charlie Fu committed about 2 years ago
[Frontend] Use MQLLMEngine for embeddings models too (#8584)
Nick Hill committed about 2 years ago
[MISC] remove engine_use_ray in benchmark_throughput.py (#8615)
Kunshang Ji committed about 2 years ago
[Bugfix] fixing sonnet benchmark bug in benchmark_serving.py (#8616)
Kuntai Du committed about 2 years ago
Revert "[Misc][Bugfix] Disable guided decoding for mistral tokenizer" (#8593)
Roger Wang committed about 2 years ago
sroy745 committed about 2 years ago
[Kernel] Remove marlin moe templating on thread_m_blocks (#8573)
Tyler Michael Smith committed about 2 years ago
[Bugfix] add `dead_error` property to engine client (#8574)
Joe Runde committed about 2 years ago
[BugFix] Nonzero exit code if MQLLMEngine startup fails (#8572)
Nick Hill committed about 2 years ago
[Kernel] Change interface to Mamba selective_state_update for continuous batching (#8039)
Tyler Michael Smith committed about 2 years ago
[AMD][ROCm]Quantization methods on ROCm; Fix _scaled_mm call (#8380)
Gregory Shtrasberg committed about 2 years ago
[Model] Support Solar Model (#8386)
Geun, Lim committed about 2 years ago
[Core] zmq: bind only to 127.0.0.1 for local-only usage (#8543)
Russell Bryant committed about 2 years ago
[Core] *Prompt* logprobs support in Multi-step (#8199)
afeldman-nm committed about 2 years ago
[Core][Bugfix][Perf] Introduce `MQLLMEngine` to avoid `asyncio` OH (#8157)
Alexander Matveev committed about 2 years ago
[CI/Build] Update Ruff version (#8469)
Aaron Pham committed about 2 years ago
[CI/Build] Avoid CUDA initialization (#8534)
Cyrus Leung committed about 2 years ago