A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 13:30 UTC
[doc] update wrong hf model links (#17184)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
Apr 26, 2025 at 00:40 UTC
df5c87952781253dc2011413a0e5e78e9d127eb6
5 files changed
+6
-7
docs/source/features/quantization/auto_awq.md
+1
-1
@@ -6,7 +6,7 @@ To create a new 4-bit quantized model, you can leverage [AutoAWQ](https://github
6
Quantization reduces the model's precision from BF16/FP16 to INT4 which effectively reduces the total model memory footprint.
7
The main benefits are lower latency and memory usage.
8
9
-You can quantize your own models by installing AutoAWQ or picking one of the [6500+ models on Huggingface](https://huggingface.co/models?sort=trending&search=awq).
9
+You can quantize your own models by installing AutoAWQ or picking one of the [6500+ models on Huggingface](https://huggingface.co/models?search=awq).
10
11
```console
12
pip install autoawq
docs/source/features/quantization/bitblas.md
+2
-2
@@ -20,8 +20,8 @@ vLLM reads the model's config file and supports pre-quantized checkpoints.
20
21
You can find pre-quantized models on:
22
23
-- [Hugging Face (BitBLAS)](https://huggingface.co/models?other=bitblas)
24
-- [Hugging Face (GPTQ)](https://huggingface.co/models?other=gptq)
23
+- [Hugging Face (BitBLAS)](https://huggingface.co/models?search=bitblas)
24
+- [Hugging Face (GPTQ)](https://huggingface.co/models?search=gptq)
25
26
Usually, these repositories have a `quantize_config.json` file that includes a `quantization_config` section.
27
docs/source/features/quantization/bnb.md
+1
-1
@@ -14,7 +14,7 @@ pip install bitsandbytes>=0.45.3
14
15
vLLM reads the model's config file and supports both in-flight quantization and pre-quantized checkpoint.
16
17
-You can find bitsandbytes quantized models on <https://huggingface.co/models?other=bitsandbytes>.
17
+You can find bitsandbytes quantized models on <https://huggingface.co/models?search=bitsandbytes>.
18
And usually, these repositories have a config.json file that includes a quantization_config section.
19
20
## Read quantized checkpoint
docs/source/features/quantization/gptqmodel.md
+1
-1
@@ -18,7 +18,7 @@ for more details on this and other advanced features.
18
19
## Installation
20
21
-You can quantize your own models by installing [GPTQModel](https://github.com/ModelCloud/GPTQModel) or picking one of the [5000+ models on Huggingface](https://huggingface.co/models?sort=trending&search=gptq).
21
+You can quantize your own models by installing [GPTQModel](https://github.com/ModelCloud/GPTQModel) or picking one of the [5000+ models on Huggingface](https://huggingface.co/models?search=gptq).
22
23
```console
24
pip install -U gptqmodel --no-build-isolation -v
docs/source/features/quantization/torchao.md
+1
-2
@@ -30,5 +30,4 @@ tokenizer.push_to_hub(hub_repo)
30
quantized_model.push_to_hub(hub_repo, safe_serialization=False)
31
```
32
33
-Alternatively, you can use the TorchAO Quantization space for quantizing models with a simple UI.
34
-See: https://huggingface.co/spaces/medmekk/TorchAO_Quantization
33
+Alternatively, you can use the [TorchAO Quantization space](https://huggingface.co/spaces/medmekk/TorchAO_Quantization) for quantizing models with a simple UI.