| 1 | # Loading Model Weights with InstantTensor |
| 2 | |
| 3 | InstantTensor accelerates loading Safetensors weights on CUDA devices through distributed loading, pipelined prefetching, and direct I/O. InstantTensor also supports GDS (GPUDirect Storage) when available. |
| 4 | For more details, see the [InstantTensor GitHub repository](https://github.com/scitix/InstantTensor). |
| 5 | |
| 6 | ## Installation |
| 7 | |
| 8 | ```bash |
| 9 | pip install instanttensor |
| 10 | ``` |
| 11 | |
| 12 | ## Use InstantTensor in vLLM |
| 13 | |
| 14 | Add `--load-format instanttensor` as a command-line argument. |
| 15 | |
| 16 | For example: |
| 17 | |
| 18 | ```bash |
| 19 | vllm serve Qwen/Qwen2.5-0.5B --load-format instanttensor |
| 20 | ``` |
| 21 | |
| 22 | ## Benchmarks |
| 23 | |
| 24 | | Model | GPU | Backend | Load Time (s) | Throughput (GB/s) | Speedup | |
| 25 | | --- | ---: | --- | ---: | ---: | --- | |
| 26 | | Qwen3-30B-A3B | 1*H200 | Safetensors | 57.4 | 1.1 | 1x | |
| 27 | | Qwen3-30B-A3B | 1*H200 | InstantTensor | 1.77 | 35 | <span style="color: green">**32.4x**</span> | |
| 28 | | DeepSeek-R1 | 8*H200 | Safetensors | 160 | 4.3 | 1x | |
| 29 | | DeepSeek-R1 | 8*H200 | InstantTensor | 15.3 | 45 | <span style="color: green">**10.5x**</span> | |
| 30 | |
| 31 | For the full benchmark results, see <https://github.com/scitix/InstantTensor/blob/main/docs/benchmark.md>. |