A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
main
Simon Mo committed almost 3 years ago
Fix compile error when using rocm (#2648)
zhaoyang-star committed almost 3 years ago
Fengzhe Zhou committed almost 3 years ago
Refactor Prometheus and Add Request Level Metrics (#2316)
Robert Shaw committed almost 3 years ago
Add unit test for Mixtral MoE layer (#2677)
Philipp Moritz committed almost 3 years ago
[Minor] Fix test_cache.py CI test failure (#2684)
Philipp Moritz committed almost 3 years ago
zspo committed almost 3 years ago
Fixes assertion failure in prefix caching: the lora index mapping should respect prefix_len (#2688)
Tao He committed almost 3 years ago
Bump up version to v0.3.0 (#2656)
Zhuohan Li committed almost 3 years ago
Add quantized mixtral support (#2673)
Woosuk Kwon committed almost 3 years ago
[Minor] Fix false warning when TP=1 (#2674)
Woosuk Kwon committed almost 3 years ago
[Minor] Fix a small typo (#2672)
Philipp Moritz committed almost 3 years ago
Add swap_blocks unit tests (#2616)
Vladimir committed almost 3 years ago
Fix 'Actor methods cannot be called directly' when using `--engine-use-ray` (#2664)
Wen Sun committed almost 3 years ago
Philipp Moritz committed almost 3 years ago
DeepseekMoE support with Fused MoE kernel (#2453)
wangding zeng committed almost 3 years ago
ROCm: Allow setting compilation target (#2581)
Rasmus Larsen committed almost 3 years ago
Hanzhi Zhou committed almost 3 years ago
zhaoyang-star committed almost 3 years ago
Support FP8-E5M2 KV Cache (#2279)
zhaoyang-star committed almost 3 years ago