@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 3f9b7ab9f5
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 02:00 UTC

[Doc] Update examples to remove SparseAutoModelForCausalLM (#12062)

Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

Kyle Sayers committed Jan 15, 2025 at 01:36 UTC 3f9b7ab9f59f83ab0551a6a2f1894e30bc0cb41c
2 files changed +8 -10
docs/source/features/quantization/fp8.md
+5 -6
@@ -54,16 +54,15 @@ The quantization process involves three main steps:
54
55 ### 1. Loading the Model
56
57 -Use `SparseAutoModelForCausalLM`, which wraps `AutoModelForCausalLM`, for saving and loading quantized models:
57 +Load your model and tokenizer using the standard `transformers` AutoModel classes:
58
59 ```python
60 -from llmcompressor.transformers import SparseAutoModelForCausalLM
61 -from transformers import AutoTokenizer
60 +from transformers import AutoTokenizer, AutoModelForCausalLM
61
62 MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
64 -
65 -model = SparseAutoModelForCausalLM.from_pretrained(
66 - MODEL_ID, device_map="auto", torch_dtype="auto")
63 +model = AutoModelForCausalLM.from_pretrained(
64 + MODEL_ID, device_map="auto", torch_dtype="auto",
65 +)
66 tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
67 ```
68
docs/source/features/quantization/int8.md
+3 -4
@@ -30,14 +30,13 @@ The quantization process involves four main steps:
30
31 ### 1. Loading the Model
32
33 -Use `SparseAutoModelForCausalLM`, which wraps `AutoModelForCausalLM`, for saving and loading quantized models:
33 +Load your model and tokenizer using the standard `transformers` AutoModel classes:
34
35 ```python
36 -from llmcompressor.transformers import SparseAutoModelForCausalLM
37 -from transformers import AutoTokenizer
36 +from transformers import AutoTokenizer, AutoModelForCausalLM
37
38 MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
40 -model = SparseAutoModelForCausalLM.from_pretrained(
39 +model = AutoModelForCausalLM.from_pretrained(
40 MODEL_ID, device_map="auto", torch_dtype="auto",
41 )
42 tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)