A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 04:30 UTC
[Doc] Installed version of llmcompressor for int8/fp8 quantization (#11103)
Signed-off-by: Guangda Liu <bingps@users.noreply.github.com> Co-authored-by: Guangda Liu <bingps@users.noreply.github.com>
bingps committed
Dec 11, 2024 at 23:43 UTC
fd22220687af5ccd89d9f8f2812069ef0422244c
2 files changed
+3
-3
docs/source/quantization/fp8.rst
+1
-1
@@ -45,7 +45,7 @@ To produce performant FP8 quantized models with vLLM, you'll need to install the
45
46
.. code-block:: console
47
48
- $ pip install llmcompressor==0.1.0
48
+ $ pip install llmcompressor
49
50
Quantization Process
51
--------------------
docs/source/quantization/int8.rst
+2
-2
@@ -19,7 +19,7 @@ To use INT8 quantization with vLLM, you'll need to install the `llm-compressor <
19
20
.. code-block:: console
21
22
- $ pip install llmcompressor==0.1.0
22
+ $ pip install llmcompressor
23
24
Quantization Process
25
--------------------
@@ -142,4 +142,4 @@ Best Practices
142
Troubleshooting and Support
143
---------------------------
144
145
-If you encounter any issues or have feature requests, please open an issue on the ``vllm-project/llm-compressor`` GitHub repository.
\ No newline at end of file
145
+If you encounter any issues or have feature requests, please open an issue on the ``vllm-project/llm-compressor`` GitHub repository.