A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 12:15 UTC
[doc] add install tips (#17373)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
May 1, 2025 at 01:02 UTC
2ac74d098ef7b8748db0cdaa255eeceb5cdd5366
5 files changed
+29
-10
docs/source/features/quantization/fp8.md
+7
-7
@@ -44,6 +44,12 @@ To produce performant FP8 quantized models with vLLM, you'll need to install the
44
pip install llmcompressor
45
```
46
47
+Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:
48
+
49
+```console
50
+pip install vllm lm-eval==0.4.4
51
+```
52
+
53
## Quantization Process
54
55
The quantization process involves three main steps:
@@ -86,7 +92,7 @@ recipe = QuantizationModifier(
92
# Apply the quantization algorithm.
93
oneshot(model=model, recipe=recipe)
94
89
-# Save the model.
95
+# Save the model: Meta-Llama-3-8B-Instruct-FP8-Dynamic
96
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
97
model.save_pretrained(SAVE_DIR)
98
tokenizer.save_pretrained(SAVE_DIR)
@@ -94,12 +100,6 @@ tokenizer.save_pretrained(SAVE_DIR)
100
101
### 3. Evaluating Accuracy
102
97
-Install `vllm` and `lm-evaluation-harness`:
98
-
99
-```console
100
-pip install vllm lm-eval==0.4.4
101
-```
102
-
103
Load and run the model in `vllm`:
104
105
```python
docs/source/features/quantization/int4.md
+7
-1
@@ -18,6 +18,12 @@ To use INT4 quantization with vLLM, you'll need to install the [llm-compressor](
18
pip install llmcompressor
19
```
20
21
+Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:
22
+
23
+```console
24
+pip install vllm lm-eval==0.4.4
25
+```
26
+
27
## Quantization Process
28
29
The quantization process involves four main steps:
@@ -87,7 +93,7 @@ oneshot(
93
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
94
)
95
90
-# Save the compressed model
96
+# Save the compressed model: Meta-Llama-3-8B-Instruct-W4A16-G128
97
SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A16-G128"
98
model.save_pretrained(SAVE_DIR, save_compressed=True)
99
tokenizer.save_pretrained(SAVE_DIR)
docs/source/features/quantization/int8.md
+7
-1
@@ -19,6 +19,12 @@ To use INT8 quantization with vLLM, you'll need to install the [llm-compressor](
19
pip install llmcompressor
20
```
21
22
+Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:
23
+
24
+```console
25
+pip install vllm lm-eval==0.4.4
26
+```
27
+
28
## Quantization Process
29
30
The quantization process involves four main steps:
@@ -91,7 +97,7 @@ oneshot(
97
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
98
)
99
94
-# Save the compressed model
100
+# Save the compressed model: Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token
101
SAVE_DIR = MODEL_ID.split("/")[1] + "-W8A8-Dynamic-Per-Token"
102
model.save_pretrained(SAVE_DIR, save_compressed=True)
103
tokenizer.save_pretrained(SAVE_DIR)
docs/source/features/quantization/quantized_kvcache.md
+1
-1
@@ -126,7 +126,7 @@ oneshot(
126
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
127
)
128
129
-# Save quantized model
129
+# Save quantized model: Llama-3.1-8B-Instruct-FP8-KV
130
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-KV"
131
model.save_pretrained(SAVE_DIR, save_compressed=True)
132
tokenizer.save_pretrained(SAVE_DIR)
docs/source/features/quantization/quark.md
+7
@@ -19,6 +19,12 @@ pip install amd-quark
19
You can refer to [Quark installation guide](https://quark.docs.amd.com/latest/install.html)
20
for more installation details.
21
22
+Additionally, install `vllm` and `lm-evaluation-harness` for evaluation:
23
+
24
+```console
25
+pip install vllm lm-eval==0.4.4
26
+```
27
+
28
## Quantization Process
29
30
After installing Quark, we will use an example to illustrate how to use Quark.
@@ -150,6 +156,7 @@ LLAMA_KV_CACHE_GROUP = ["*k_proj", "*v_proj"]
156
export_config = ExporterConfig(json_export_config=JsonExporterConfig())
157
export_config.json_export_config.kv_cache_group = LLAMA_KV_CACHE_GROUP
158
159
+# Model: Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant
160
EXPORT_DIR = MODEL_ID.split("/")[1] + "-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant"
161
exporter = ModelExporter(config=export_config, export_dir=EXPORT_DIR)
162
with torch.no_grad():