A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
[Doc] Update documentation on Tensorizer (#5471)
Sanger Steel committed
Jun 14, 2024 at 14:27 UTC
6e2527a7cb94fa9154e34a42b95c1e4eb9a83e01
3 files changed
+14
-1
docs/source/index.rst
+1
@@ -81,6 +81,7 @@ Documentation
81
serving/env_vars
82
serving/usage_stats
83
serving/integrations
84
+ serving/tensorizer
85
86
.. toctree::
87
:maxdepth: 1
docs/source/serving/tensorizer.rst
new
+12
@@ -0,0 +1,12 @@
1
+.. _tensorizer:
2
+
3
+Loading Models with CoreWeave's Tensorizer
4
+==========================================
5
+vLLM supports loading models with `CoreWeave's Tensorizer <https://docs.coreweave.com/coreweave-machine-learning-and-ai/inference/tensorizer>`_.
6
+vLLM model tensors that have been serialized to disk, an HTTP/HTTPS endpoint, or S3 endpoint can be deserialized
7
+at runtime extremely quickly directly to the GPU, resulting in significantly
8
+shorter Pod startup times and CPU memory usage. Tensor encryption is also supported.
9
+
10
+For more information on CoreWeave's Tensorizer, please refer to
11
+`CoreWeave's Tensorizer documentation <https://github.com/coreweave/tensorizer>`_. For more information on serializing a vLLM model, as well a general usage guide to using Tensorizer with vLLM, see
12
+the `vLLM example script <https://docs.vllm.ai/en/stable/getting_started/examples/tensorize_vllm_model.html>`_.
\ No newline at end of file
vllm/engine/arg_utils.py
+1
-1
@@ -230,7 +230,7 @@ class EngineArgs:
230
'* "dummy" will initialize the weights with random values, '
231
'which is mainly for profiling.\n'
232
'* "tensorizer" will load the weights using tensorizer from '
233
- 'CoreWeave. See the Tensorize vLLM Model script in the Examples'
233
+ 'CoreWeave. See the Tensorize vLLM Model script in the Examples '
234
'section for more information.\n'
235
'* "bitsandbytes" will load the weights using bitsandbytes '
236
'quantization.\n')