@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 1af090b57d
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 05:30 UTC

Bump up version to v0.3.0 (#2656)

Zhuohan Li committed Jan 31, 2024 at 00:07 UTC 1af090b57d0e23d268e79941f8084bf0a8ad8621
3 files changed +7 -3
README.md
+3 -1
@@ -46,7 +46,7 @@ vLLM is fast with:
46 - Efficient management of attention key and value memory with **PagedAttention**
47 - Continuous batching of incoming requests
48 - Fast model execution with CUDA/HIP graph
49 -- Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629)
49 +- Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629), FP8 KV Cache
50 - Optimized CUDA kernels
51
52 vLLM is flexible and easy to use with:
@@ -57,6 +57,8 @@ vLLM is flexible and easy to use with:
57 - Streaming outputs
58 - OpenAI-compatible API server
59 - Support NVIDIA GPUs and AMD GPUs
60 +- (Experimental) Prefix caching support
61 +- (Experimental) Multi-lora support
62
63 vLLM seamlessly supports many Hugging Face models, including the following architectures:
64
docs/source/index.rst
+3 -1
@@ -31,7 +31,7 @@ vLLM is fast with:
31 * Efficient management of attention key and value memory with **PagedAttention**
32 * Continuous batching of incoming requests
33 * Fast model execution with CUDA/HIP graph
34 -* Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, `SqueezeLLM <https://arxiv.org/abs/2306.07629>`_
34 +* Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, `SqueezeLLM <https://arxiv.org/abs/2306.07629>`_, FP8 KV Cache
35 * Optimized CUDA kernels
36
37 vLLM is flexible and easy to use with:
@@ -42,6 +42,8 @@ vLLM is flexible and easy to use with:
42 * Streaming outputs
43 * OpenAI-compatible API server
44 * Support NVIDIA GPUs and AMD GPUs
45 +* (Experimental) Prefix caching support
46 +* (Experimental) Multi-lora support
47
48 For more information, check out the following:
49
vllm/__init__.py
+1 -1
@@ -8,7 +8,7 @@ from vllm.entrypoints.llm import LLM
8 from vllm.outputs import CompletionOutput, RequestOutput
9 from vllm.sampling_params import SamplingParams
10
11 -__version__ = "0.2.7"
11 +__version__ = "0.3.0"
12
13 __all__ = [
14 "LLM",