@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 208cb34c81
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 09:30 UTC

[Doc]: Update tensorizer docs to include vllm[tensorizer] (#7889)

Co-authored-by: Kaunil Dhruv <dhruv.kaunil@gmail.com>

Seth Kimmel committed Oct 22, 2024 at 15:43 UTC 208cb34c812585ce387d7aff82678a3776a66756
1 file changed +4 -1
docs/source/serving/tensorizer.rst
+4 -1
@@ -9,4 +9,7 @@ shorter Pod startup times and CPU memory usage. Tensor encryption is also suppor
9
10 For more information on CoreWeave's Tensorizer, please refer to
11 `CoreWeave's Tensorizer documentation <https://github.com/coreweave/tensorizer>`_. For more information on serializing a vLLM model, as well a general usage guide to using Tensorizer with vLLM, see
12 -the `vLLM example script <https://docs.vllm.ai/en/stable/getting_started/examples/tensorize_vllm_model.html>`_.
\ No newline at end of file
12 +the `vLLM example script <https://docs.vllm.ai/en/stable/getting_started/examples/tensorize_vllm_model.html>`_.
13 +
14 +.. note::
15 + Note that to use this feature you will need to install `tensorizer` by running `pip install vllm[tensorizer]`.