A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 20:45 UTC
[Docs] Announce llama3.1 support (#6688)
Woosuk Kwon committed
Jul 23, 2024 at 08:18 UTC
cb1362a8892d444294117b848547c76b80d6dc65
2 files changed
+3
-2
README.md
+1
@@ -26,6 +26,7 @@ Register now [here](https://lu.ma/lp0gyjqr) and be part of the event!
26
---
27
28
*Latest News* 🔥
29
+- [2024/07] In partnership with Meta, vLLM officially supports Llama 3.1 with FP8 quantization and pipeline parallelism! Please check out our blog post [here](https://blog.vllm.ai/2024/07/23/llama31.html).
30
- [2024/06] We hosted [the fourth vLLM meetup](https://lu.ma/agivllm) with Cloudflare and BentoML! Please find the meetup slides [here](https://docs.google.com/presentation/d/1iJ8o7V2bQEi0BFEljLTwc5G1S10_Rhv3beed5oB0NJ4/edit?usp=sharing).
31
- [2024/04] We hosted [the third vLLM meetup](https://robloxandvllmmeetup2024.splashthat.com/) with Roblox! Please find the meetup slides [here](https://docs.google.com/presentation/d/1A--47JAK4BJ39t954HyTkvtfwn0fkqtsL8NGFuslReM/edit?usp=sharing).
32
- [2024/01] We hosted [the second vLLM meetup](https://lu.ma/ygxbpzhl) with IBM! Please find the meetup slides [here](https://docs.google.com/presentation/d/12mI2sKABnUw5RBWXDYY-HtHth4iMSNcEoQ10jDQbxgA/edit?usp=sharing).
docs/source/models/supported_models.rst
+2
-2
@@ -94,8 +94,8 @@ Decoder-only Language Models
94
- :code:`ai21labs/Jamba-v0.1`, etc.
95
- ✅︎
96
* - :code:`LlamaForCausalLM`
97
- - LLaMA, Llama 2, Meta Llama 3, Vicuna, Alpaca, Yi
98
- - :code:`meta-llama/Meta-Llama-3-8B-Instruct`, :code:`meta-llama/Meta-Llama-3-70B-Instruct`, :code:`meta-llama/Llama-2-13b-hf`, :code:`meta-llama/Llama-2-70b-hf`, :code:`openlm-research/open_llama_13b`, :code:`lmsys/vicuna-13b-v1.3`, :code:`01-ai/Yi-6B`, :code:`01-ai/Yi-34B`, etc.
97
+ - Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
98
+ - :code:`meta-llama/Meta-Llama-3.1-405B-Instruct`, :code:`meta-llama/Meta-Llama-3.1-70B`, :code:`meta-llama/Meta-Llama-3-70B-Instruct`, :code:`meta-llama/Llama-2-70b-hf`, :code:`01-ai/Yi-34B`, etc.
99
- ✅︎
100
* - :code:`MiniCPMForCausalLM`
101
- MiniCPM