A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 06:45 UTC
main
Use `quantization_config` in hf config (#1695)
Woosuk Kwon committed almost 3 years ago
Roy committed almost 3 years ago
Zhuofan committed almost 3 years ago
Update the adding-model doc according to the new refactor (#1692)
Zhuohan Li committed about 3 years ago
Zhuohan Li committed about 3 years ago
Return usage for openai streaming requests (#1663)
Iskren Ivov Chernev committed about 3 years ago
[Fix] Update Supported Models List (#1690)
Zhuohan Li committed about 3 years ago
Support Microsoft Phi 1.5 (#1664)
maximzubkov committed about 3 years ago
[Minor] Fix duplication of ignored seq group in engine step (#1666)
Simon Mo committed about 3 years ago
Fix loading error when safetensors contains empty tensor (#1687)
twaka committed about 3 years ago
feat(config): support parsing torch.dtype (#1641)
Aaron Pham committed about 3 years ago
Revert `MptConfig` to `MPTConfig` (#1668)
Megha Agarwal committed about 3 years ago
Zhuohan Li committed about 3 years ago
Add DeepSpeed MII backend to benchmark script (#1649)
Woosuk Kwon committed about 3 years ago
[Minor] Move RoPE selection logic to `get_rope` (#1633)
Woosuk Kwon committed about 3 years ago
Fix #1474 - AssertionError:assert param_slice.shape == loaded_weight.shape (#1631)
lirui committed about 3 years ago
Run default _AsyncLLMEngine._run_workers_async in threadpool (#1628)
Dominik Schwabe committed about 3 years ago
config parser: add ChatGLM2 seq_length to `_get_and_verify_max_len` (#1617)
Sin committed about 3 years ago
Dockerfile: Upgrade Cuda to 12.1 (#1609)
GhaziSyed committed about 3 years ago
Add Yi model to quantization support (#1600)
forpanyang committed about 3 years ago