@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 115 lines 4.84 KB
Rendered Raw
1 # Loading models with Run:ai Model Streamer
2
3 Run:ai Model Streamer is a library to read tensors in concurrency, while streaming it to GPU memory.
4 Further reading can be found in [Run:ai Model Streamer Documentation](https://github.com/run-ai/runai-model-streamer/blob/master/docs/README.md).
5
6 vLLM supports loading weights in Safetensors format using the Run:ai Model Streamer.
7 You first need to install vLLM RunAI optional dependency:
8
9 ```bash
10 pip3 install vllm[runai]
11 ```
12
13 To run it as an OpenAI-compatible server, add the `--load-format runai_streamer` flag:
14
15 ```bash
16 vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \
17 --load-format runai_streamer
18 ```
19
20 To run model from AWS S3 object store run:
21
22 ```bash
23 vllm serve s3://core-llm/Llama-3-8b \
24 --load-format runai_streamer
25 ```
26
27 To run model from Google Cloud Storage run:
28
29 ```bash
30 vllm serve gs://core-llm/Llama-3-8b \
31 --load-format runai_streamer
32 ```
33
34 To run model from Azure Blob Storage run:
35
36 ```bash
37 AZURE_STORAGE_ACCOUNT_NAME=<account> \
38 vllm serve az://<container>/<model-path> \
39 --load-format runai_streamer
40 ```
41
42 Authentication uses `DefaultAzureCredential`, which supports `az login`, managed identity, environment variables (`AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_CLIENT_SECRET`), and other methods.
43
44 To run model from a S3 compatible object store run:
45
46 ```bash
47 RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING=0 \
48 AWS_EC2_METADATA_DISABLED=true \
49 AWS_ENDPOINT_URL=https://storage.googleapis.com \
50 vllm serve s3://core-llm/Llama-3-8b \
51 --load-format runai_streamer
52 ```
53
54 ## Tunable parameters
55
56 You can tune parameters using `--model-loader-extra-config`:
57
58 You can tune `distributed` that controls whether distributed streaming should be used. This is currently only possible on CUDA and ROCM devices. This can significantly improve loading times from object storage or high-throughput network fileshares.
59 You can read further about Distributed streaming [here](https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/usage.md#distributed-streaming)
60
61 ```bash
62 vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \
63 --load-format runai_streamer \
64 --model-loader-extra-config '{"distributed":true}'
65 ```
66
67 You can tune `concurrency` that controls the level of concurrency and number of OS threads reading tensors from the file to the CPU buffer.
68 For reading from S3, it will be the number of client instances the host is opening to the S3 server.
69
70 ```bash
71 vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \
72 --load-format runai_streamer \
73 --model-loader-extra-config '{"concurrency":16}'
74 ```
75
76 You can control the size of the CPU Memory buffer to which tensors are read from the file, and limit this size.
77 You can read further about CPU buffer memory limiting [here](https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/env-vars.md#runai_streamer_memory_limit).
78
79 ```bash
80 vllm serve /home/meta-llama/Llama-3.2-3B-Instruct \
81 --load-format runai_streamer \
82 --model-loader-extra-config '{"memory_limit":5368709120}'
83 ```
84
85 !!! note
86 For further instructions about tunable parameters and additional parameters configurable through environment variables, read the [Environment Variables Documentation](https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/env-vars.md).
87
88 ## Sharded Model Loading
89
90 vLLM also supports loading sharded models using Run:ai Model Streamer. This is particularly useful for large models that are split across multiple files. To use this feature, use the `--load-format runai_streamer_sharded` flag:
91
92 ```bash
93 vllm serve /path/to/sharded/model --load-format runai_streamer_sharded
94 ```
95
96 The sharded loader expects model files to follow the same naming pattern as the regular sharded state loader: `model-rank-{rank}-part-{part}.safetensors`. You can customize this pattern using the `pattern` parameter in `--model-loader-extra-config`:
97
98 ```bash
99 vllm serve /path/to/sharded/model \
100 --load-format runai_streamer_sharded \
101 --model-loader-extra-config '{"pattern":"custom-model-rank-{rank}-part-{part}.safetensors"}'
102 ```
103
104 To create sharded model files, you can use the script provided in [examples/features/sharded_state/save_sharded_state_offline.py](../../../examples/features/sharded_state/save_sharded_state_offline.py). This script demonstrates how to save a model in the sharded format that is compatible with the Run:ai Model Streamer sharded loader.
105
106 The sharded loader supports all the same tunable parameters as the regular Run:ai Model Streamer, including `concurrency` and `memory_limit`. These can be configured in the same way:
107
108 ```bash
109 vllm serve /path/to/sharded/model \
110 --load-format runai_streamer_sharded \
111 --model-loader-extra-config '{"concurrency":16, "memory_limit":5368709120}'
112 ```
113
114 !!! note
115 The sharded loader is particularly efficient for tensor or pipeline parallel models where each worker only needs to read its own shard rather than the entire checkpoint.