@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 6e2527a7cb
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

[Doc] Update documentation on Tensorizer (#5471)

Sanger Steel committed Jun 14, 2024 at 14:27 UTC 6e2527a7cb94fa9154e34a42b95c1e4eb9a83e01
3 files changed +14 -1
docs/source/index.rst
+1
@@ -81,6 +81,7 @@ Documentation
81 serving/env_vars
82 serving/usage_stats
83 serving/integrations
84 + serving/tensorizer
85
86 .. toctree::
87 :maxdepth: 1
docs/source/serving/tensorizer.rst new
+12
@@ -0,0 +1,12 @@
1 +.. _tensorizer:
2 +
3 +Loading Models with CoreWeave's Tensorizer
4 +==========================================
5 +vLLM supports loading models with `CoreWeave's Tensorizer <https://docs.coreweave.com/coreweave-machine-learning-and-ai/inference/tensorizer>`_.
6 +vLLM model tensors that have been serialized to disk, an HTTP/HTTPS endpoint, or S3 endpoint can be deserialized
7 +at runtime extremely quickly directly to the GPU, resulting in significantly
8 +shorter Pod startup times and CPU memory usage. Tensor encryption is also supported.
9 +
10 +For more information on CoreWeave's Tensorizer, please refer to
11 +`CoreWeave's Tensorizer documentation <https://github.com/coreweave/tensorizer>`_. For more information on serializing a vLLM model, as well a general usage guide to using Tensorizer with vLLM, see
12 +the `vLLM example script <https://docs.vllm.ai/en/stable/getting_started/examples/tensorize_vllm_model.html>`_.
\ No newline at end of file
vllm/engine/arg_utils.py
+1 -1
@@ -230,7 +230,7 @@ class EngineArgs:
230 '* "dummy" will initialize the weights with random values, '
231 'which is mainly for profiling.\n'
232 '* "tensorizer" will load the weights using tensorizer from '
233 - 'CoreWeave. See the Tensorize vLLM Model script in the Examples'
233 + 'CoreWeave. See the Tensorize vLLM Model script in the Examples '
234 'section for more information.\n'
235 '* "bitsandbytes" will load the weights using bitsandbytes '
236 'quantization.\n')