@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / fd22220687
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 04:30 UTC

[Doc] Installed version of llmcompressor for int8/fp8 quantization (#11103)

Signed-off-by: Guangda Liu <bingps@users.noreply.github.com> Co-authored-by: Guangda Liu <bingps@users.noreply.github.com>

bingps committed Dec 11, 2024 at 23:43 UTC fd22220687af5ccd89d9f8f2812069ef0422244c
2 files changed +3 -3
docs/source/quantization/fp8.rst
+1 -1
@@ -45,7 +45,7 @@ To produce performant FP8 quantized models with vLLM, you'll need to install the
45
46 .. code-block:: console
47
48 - $ pip install llmcompressor==0.1.0
48 + $ pip install llmcompressor
49
50 Quantization Process
51 --------------------
docs/source/quantization/int8.rst
+2 -2
@@ -19,7 +19,7 @@ To use INT8 quantization with vLLM, you'll need to install the `llm-compressor <
19
20 .. code-block:: console
21
22 - $ pip install llmcompressor==0.1.0
22 + $ pip install llmcompressor
23
24 Quantization Process
25 --------------------
@@ -142,4 +142,4 @@ Best Practices
142 Troubleshooting and Support
143 ---------------------------
144
145 -If you encounter any issues or have feature requests, please open an issue on the ``vllm-project/llm-compressor`` GitHub repository.
\ No newline at end of file
145 +If you encounter any issues or have feature requests, please open an issue on the ``vllm-project/llm-compressor`` GitHub repository.