A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 04:15 UTC
main
Integrate Marlin Kernels for Int4 GPTQ inference (#2497)
Robert Shaw committed over 2 years ago
fix relative import path of protocol.py (#3134)
Huarong committed over 2 years ago
docs: Add tutorial on deploying vLLM model with KServe (#2586)
Yuan Tang committed over 2 years ago
Remove exclude_unset in streaming response (#3143)
Seonghyeon committed over 2 years ago
Fix: Output text is always truncated in some models (#3016)
Sherry committed over 2 years ago
Add guided decoding for OpenAI API server (#2819)
felixzhu555 committed over 2 years ago
[Fix] Don't deep-copy LogitsProcessors when copying SamplingParams (#3099)
Nick Hill committed over 2 years ago
Fix building from source on WSL (#3112)
Billy Cao committed over 2 years ago
Support starcoder2 architecture (#3089)
Seonghyeon committed over 2 years ago
add cache_config's info to prometheus metrics. (#3100)
Allen.Dou committed over 2 years ago
Fix: `AttributeError` in OpenAI-compatible server (#3018)
Jae-Won Chung committed over 2 years ago
Add Support for 2/3/8-bit GPTQ Quantization Models (#2330)
CHU Tianxiang committed over 2 years ago
Add LoRA support for Gemma (#3050)
Woosuk Kwon committed over 2 years ago
[Neuron] Support inference with transformers-neuronx (#2569)
Liangfu Chen committed over 2 years ago
Restrict prometheus_client >= 0.18.0 to prevent errors when importing pkgs (#3070)
Allen.Dou committed over 2 years ago
multi-lora documentation fix (#3064)
Ganesh Jagadeesan committed over 2 years ago
Enable GQA support in the prefix prefill kernels (#3007)
Tao He committed over 2 years ago
[Minor] Fix StableLMEpochForCausalLM -> StableLmForCausalLM (#3046)
Woosuk Kwon committed over 2 years ago
Support logit bias for OpenAI API (#3027)
Dylan Hawk committed over 2 years ago
[Minor] Fix type annotation in fused moe (#3045)
Woosuk Kwon committed over 2 years ago