A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
Bump up version to v0.3.0 (#2656)
Zhuohan Li committed
Jan 31, 2024 at 00:07 UTC
1af090b57d0e23d268e79941f8084bf0a8ad8621
3 files changed
+7
-3
README.md
+3
-1
@@ -46,7 +46,7 @@ vLLM is fast with:
46
- Efficient management of attention key and value memory with **PagedAttention**
47
- Continuous batching of incoming requests
48
- Fast model execution with CUDA/HIP graph
49
-- Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629)
49
+- Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629), FP8 KV Cache
50
- Optimized CUDA kernels
51
52
vLLM is flexible and easy to use with:
@@ -57,6 +57,8 @@ vLLM is flexible and easy to use with:
57
- Streaming outputs
58
- OpenAI-compatible API server
59
- Support NVIDIA GPUs and AMD GPUs
60
+- (Experimental) Prefix caching support
61
+- (Experimental) Multi-lora support
62
63
vLLM seamlessly supports many Hugging Face models, including the following architectures:
64
docs/source/index.rst
+3
-1
@@ -31,7 +31,7 @@ vLLM is fast with:
31
* Efficient management of attention key and value memory with **PagedAttention**
32
* Continuous batching of incoming requests
33
* Fast model execution with CUDA/HIP graph
34
-* Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, `SqueezeLLM <https://arxiv.org/abs/2306.07629>`_
34
+* Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, `SqueezeLLM <https://arxiv.org/abs/2306.07629>`_, FP8 KV Cache
35
* Optimized CUDA kernels
36
37
vLLM is flexible and easy to use with:
@@ -42,6 +42,8 @@ vLLM is flexible and easy to use with:
42
* Streaming outputs
43
* OpenAI-compatible API server
44
* Support NVIDIA GPUs and AMD GPUs
45
+* (Experimental) Prefix caching support
46
+* (Experimental) Multi-lora support
47
48
For more information, check out the following:
49
vllm/__init__.py
+1
-1
@@ -8,7 +8,7 @@ from vllm.entrypoints.llm import LLM
8
from vllm.outputs import CompletionOutput, RequestOutput
9
from vllm.sampling_params import SamplingParams
10
11
-__version__ = "0.2.7"
11
+__version__ = "0.3.0"
12
13
__all__ = [
14
"LLM",