A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 04:15 UTC
main
Add args for mTLS support (#3410)
Dan Clark committed over 2 years ago
Install `flash_attn` in Docker image (#3396)
Thomas Parnell committed over 2 years ago
youkaichao committed over 2 years ago
allow user to chose which vllm's merics to display in grafana (#3393)
Allen.Dou committed over 2 years ago
[Hotfix] [Debug] test_openai_server.py::test_guided_regex_completion (#3383)
Simon Mo committed over 2 years ago
[FIX] Simpler fix for async engine running on ray (#3371)
Zhuohan Li committed over 2 years ago
Antoni Baum committed over 2 years ago
Add batched RoPE kernel (#3095)
Terry committed over 2 years ago
Or Sharir committed over 2 years ago
[Minor Fix] Use cupy-cuda11x in CUDA 11.8 build (#3256)
陈序 committed over 2 years ago
[Minor] Fix bias in if to remove ambiguity (#3259)
Hui Liu committed over 2 years ago
add hf_transfer to requirements.txt (#3031)
Ronan McGovern committed over 2 years ago
[Fix] Fix quantization="gptq" when using Marlin (#3319)
Bo-Wen Wang committed over 2 years ago
Add kernel for GeGLU with approximate GELU (#3337)
Woosuk Kwon committed over 2 years ago
Fixes #1556 double free (#3347)
Breno Faria committed over 2 years ago
docs: Add BentoML deployment doc (#3336)
Sherlock Xu committed over 2 years ago
Support Mistral Model Inference with transformers-neuronx (#3153)
DAIZHENWEI committed over 2 years ago
[ROCm] Fix warp and lane calculation in blockReduceSum (#3321)
kliuae committed over 2 years ago
Add distributed model executor abstraction (#3191)
Zhuohan Li committed over 2 years ago
[docs] Add LoRA support information for models (#3299)
Philipp Moritz committed over 2 years ago