A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 02:00 UTC
[Doc] Update examples to remove SparseAutoModelForCausalLM (#12062)
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Kyle Sayers committed
Jan 15, 2025 at 01:36 UTC
3f9b7ab9f59f83ab0551a6a2f1894e30bc0cb41c
2 files changed
+8
-10
docs/source/features/quantization/fp8.md
+5
-6
@@ -54,16 +54,15 @@ The quantization process involves three main steps:
54
55
### 1. Loading the Model
56
57
-Use `SparseAutoModelForCausalLM`, which wraps `AutoModelForCausalLM`, for saving and loading quantized models:
57
+Load your model and tokenizer using the standard `transformers` AutoModel classes:
58
59
```python
60
-from llmcompressor.transformers import SparseAutoModelForCausalLM
61
-from transformers import AutoTokenizer
60
+from transformers import AutoTokenizer, AutoModelForCausalLM
61
62
MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
64
-
65
-model = SparseAutoModelForCausalLM.from_pretrained(
66
- MODEL_ID, device_map="auto", torch_dtype="auto")
63
+model = AutoModelForCausalLM.from_pretrained(
64
+ MODEL_ID, device_map="auto", torch_dtype="auto",
65
+)
66
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
67
```
68
docs/source/features/quantization/int8.md
+3
-4
@@ -30,14 +30,13 @@ The quantization process involves four main steps:
30
31
### 1. Loading the Model
32
33
-Use `SparseAutoModelForCausalLM`, which wraps `AutoModelForCausalLM`, for saving and loading quantized models:
33
+Load your model and tokenizer using the standard `transformers` AutoModel classes:
34
35
```python
36
-from llmcompressor.transformers import SparseAutoModelForCausalLM
37
-from transformers import AutoTokenizer
36
+from transformers import AutoTokenizer, AutoModelForCausalLM
37
38
MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
40
-model = SparseAutoModelForCausalLM.from_pretrained(
39
+model = AutoModelForCausalLM.from_pretrained(
40
MODEL_ID, device_map="auto", torch_dtype="auto",
41
)
42
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)