A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 22:15 UTC
[Doc] fix the incorrect module path of tensorize_vllm_model (#13863)
Albert committed
Feb 26, 2025 at 14:56 UTC
e656f638de122095bab21a5f5f9c21d2e4974b07
1 file changed
+5
-5
examples/other/tensorize_vllm_model.py
+5
-5
@@ -27,7 +27,7 @@ https://github.com/coreweave/tensorizer
27
To serialize a model, install vLLM from source, then run something
28
like this from the root level of this repository:
29
30
-python -m examples.offline_inference.tensorize_vllm_model \
30
+python -m examples.other.tensorize_vllm_model \
31
--model facebook/opt-125m \
32
serialize \
33
--serialized-directory s3://my-bucket \
@@ -47,7 +47,7 @@ providing a `--keyfile` argument.
47
To deserialize a model, you can run something like this from the root
48
level of this repository:
49
50
-python -m examples.offline_inference.tensorize_vllm_model \
50
+python -m examples.other.tensorize_vllm_model \
51
--model EleutherAI/gpt-j-6B \
52
--dtype float16 \
53
deserialize \
@@ -65,11 +65,11 @@ shard's rank. Sharded models serialized with this script will be named as
65
model-rank-%03d.tensors
66
67
For more information on the available arguments for serializing, run
68
-`python -m examples.offline_inference.tensorize_vllm_model serialize --help`.
68
+`python -m examples.other.tensorize_vllm_model serialize --help`.
69
70
Or for deserializing:
71
72
-`python -m examples.offline_inference.tensorize_vllm_model deserialize --help`.
72
+`python -m examples.other.tensorize_vllm_model deserialize --help`.
73
74
Once a model is serialized, tensorizer can be invoked with the `LLM` class
75
directly to load models:
@@ -90,7 +90,7 @@ TensorizerConfig arguments desired.
90
In order to see all of the available arguments usable to configure
91
loading with tensorizer that are given to `TensorizerConfig`, run:
92
93
-`python -m examples.offline_inference.tensorize_vllm_model deserialize --help`
93
+`python -m examples.other.tensorize_vllm_model deserialize --help`
94
95
under the `tensorizer options` section. These can also be used for
96
deserialization in this example script, although `--tensorizer-uri` and