A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 15:45 UTC
[Doc] Add documentation for GGUF quantization (#8618)
Isotr0py committed
Sep 20, 2024 at 03:15 UTC
ea4647b7d77c4738c5ed2ab77a2c9f5ad335f6fb
2 files changed
+74
docs/source/index.rst
+1
@@ -107,6 +107,7 @@ Documentation
107
quantization/supported_hardware
108
quantization/auto_awq
109
quantization/bnb
110
+ quantization/gguf
111
quantization/int8
112
quantization/fp8
113
quantization/fp8_e5m2_kvcache
docs/source/quantization/gguf.rst
new
+73
@@ -0,0 +1,73 @@
1
+.. _gguf:
2
+
3
+GGUF
4
+==================
5
+
6
+.. warning::
7
+
8
+ Please note that GGUF support in vLLM is highly experimental and under-optimized at the moment, it might be incompatible with other features. Currently, you can use GGUF as a way to reduce memory footprint. If you encounter any issues, please report them to the vLLM team.
9
+
10
+.. warning::
11
+
12
+ Currently, vllm only supports loading single-file GGUF models. If you have a multi-files GGUF model, you can use `gguf-split <https://github.com/ggerganov/llama.cpp/pull/6135>`_ tool to merge them to a single-file model.
13
+
14
+To run a GGUF model with vLLM, you can download and use the local GGUF model from `TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF <https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF>`_ with the following command:
15
+
16
+.. code-block:: console
17
+
18
+ $ wget https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf
19
+ $ # We recommend using the tokenizer from base model to avoid long-time and buggy tokenizer conversion.
20
+ $ vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0
21
+
22
+You can also add ``--tensor-parallel-size 2`` to enable tensor parallelism inference with 2 GPUs:
23
+
24
+.. code-block:: console
25
+
26
+ $ # We recommend using the tokenizer from base model to avoid long-time and buggy tokenizer conversion.
27
+ $ vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0 --tensor-parallel-size 2
28
+
29
+.. warning::
30
+
31
+ We recommend using the tokenizer from base model instead of GGUF model. Because the tokenizer conversion from GGUF is time-consuming and unstable, especially for some models with large vocab size.
32
+
33
+You can also use the GGUF model directly through the LLM entrypoint:
34
+
35
+.. code-block:: python
36
+
37
+ from vllm import LLM, SamplingParams
38
+
39
+ # In this script, we demonstrate how to pass input to the chat method:
40
+ conversation = [
41
+ {
42
+ "role": "system",
43
+ "content": "You are a helpful assistant"
44
+ },
45
+ {
46
+ "role": "user",
47
+ "content": "Hello"
48
+ },
49
+ {
50
+ "role": "assistant",
51
+ "content": "Hello! How can I assist you today?"
52
+ },
53
+ {
54
+ "role": "user",
55
+ "content": "Write an essay about the importance of higher education.",
56
+ },
57
+ ]
58
+
59
+ # Create a sampling params object.
60
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
61
+
62
+ # Create an LLM.
63
+ llm = LLM(model="./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf",
64
+ tokenizer="TinyLlama/TinyLlama-1.1B-Chat-v1.0")
65
+ # Generate texts from the prompts. The output is a list of RequestOutput objects
66
+ # that contain the prompt, generated text, and other information.
67
+ outputs = llm.chat(conversation, sampling_params)
68
+
69
+ # Print the outputs.
70
+ for output in outputs:
71
+ prompt = output.prompt
72
+ generated_text = output.outputs[0].text
73
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")