@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 203 lines 8.84 KB
Rendered Raw
1 # Preload
2
3 vLLM's preload feature keeps model artifacts resident in GPU memory across
4 engine restarts, so a restarting engine reuses them instead of rebuilding them
5 from scratch. The design is extensible to different kinds of artifacts; it
6 currently supports **model weights** via the weight cache daemon.
7
8 With weight preloading, a daemon process per GPU holds its rank's
9 post-quantized, TP-sharded weights and serves CUDA IPC handles to vLLM engines
10 over a Unix domain socket, so a restarting engine maps the weights via
11 zero-copy IPC instead of reloading them from disk.
12
13 Key benefits:
14
15 - **Fast engine restarts**: weight loading reduces to mapping existing GPU
16 memory, cutting restart time from minutes (disk load + quantization
17 processing) to seconds.
18 - **Zero-copy sharing**: in the default `zero_copy` mode the engine directly
19 shares the daemon's GPU memory, so a restart adds no extra GPU memory
20 footprint.
21 - **Safe fallback**: if the daemon is unavailable or its cached weights don't
22 match the engine's configuration, the engine falls back to loading from
23 disk.
24
25 !!! note
26 Weight preloading is only supported on CUDA and ROCm platforms. Tensor,
27 expert and data parallelism are supported, including across nodes;
28 pipeline parallelism is rejected when launching the daemon.
29
30 ## Quick start
31
32 Launch one weight cache daemon per GPU with a single command:
33
34 ```bash
35 vllm preload --model meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 4
36 ```
37
38 The daemon process loads and post-processes the weights once, then waits for
39 engines. Each rank binds its Unix socket only after its shard is fully cached,
40 so engines connecting before that simply fall back to disk loading.
41
42 Then start (or restart) engines with the `ipc_cache` load format:
43
44 ```bash
45 vllm serve meta-llama/Llama-3.1-8B-Instruct \
46 --tensor-parallel-size 4 \
47 --load-format ipc_cache
48 ```
49
50 The daemon itself must load from disk; passing `--load-format ipc_cache` to
51 `vllm preload` is an error.
52
53 ## How it works
54
55 1. `vllm preload` spawns one daemon process per GPU. Each daemon loads its
56 shard from disk using the configured loader, runs quantization
57 post-processing, and exports every parameter/buffer as a CUDA IPC handle.
58 2. An engine started with `--load-format ipc_cache` builds its model on the
59 meta device (no storage) and requests the tensors from the daemon on its
60 GPU.
61 3. Before serving anything, the engine and daemon compare a fingerprint of the
62 cached weights: checkpoint content (hashed from safetensors metadata, so
63 identical weights in different directories still match), model
64 architecture, TP/DP size/rank, dtype, quantization method and config, model
65 revision, and vLLM version. On any mismatch the engine falls back to disk
66 loading (unless `fallback` is disabled).
67 4. The engine also verifies the daemon's GPU UUID matches its own device, so a
68 stale socket cannot serve weights for the wrong GPU.
69
70 Tied weights (e.g. `lm_head.weight` sharing storage with
71 `embed_tokens.weight`) are exported once and re-established as aliases in the
72 engine, preserving parameter identity.
73
74 ## Multi-node and data parallelism
75
76 CUDA IPC handles are node-local, so each node serves only its local GPUs'
77 shards. For multi-node tensor parallelism, run one `vllm preload` launcher per
78 node with a shared rendezvous: reuse the `--nnodes` / `--node-rank` /
79 `--master-addr` flags you pass the engine, plus a `--weight-cache-master-port`
80 distinct from the engine's `--master-port`:
81
82 ```bash
83 # node 0 (8 local GPUs)
84 vllm preload --model /path/to/model --tensor-parallel-size 16 \
85 --nnodes 2 --node-rank 0 --master-addr 10.0.0.1 \
86 --weight-cache-master-port 29600
87 # node 1 (8 local GPUs)
88 vllm preload --model /path/to/model --tensor-parallel-size 16 \
89 --nnodes 2 --node-rank 1 --master-addr 10.0.0.1 \
90 --weight-cache-master-port 29600
91 ```
92
93 For data parallelism (e.g. a TP1 x DP16 x EP decode fleet), run one launcher
94 per node with the engine's DP placement flags. Local GPU `i` serves DP rank
95 `start_rank + i // tp_size` and TP rank `i % tp_size`, and all
96 `dp_size * tp_size` daemons form one world group on `--data-parallel-address`
97 / `--weight-cache-master-port` so the expert shards are laid out exactly as in
98 the engine:
99
100 ```bash
101 # node r (4 local GPUs)
102 vllm preload --model /path/to/model --tensor-parallel-size 1 \
103 --enable-expert-parallel \
104 --data-parallel-size 16 --data-parallel-size-local 4 \
105 --data-parallel-start-rank 4r --data-parallel-address 10.0.0.1 \
106 --weight-cache-master-port 29600
107 ```
108
109 Data parallelism also combines with multi-node tensor parallelism: pass both
110 flag sets. Each node then serves a contiguous block of the
111 `dp_size * tp_size` global ranks.
112
113 ## Speculative decoding
114
115 With MTP, EAGLE or EAGLE3 speculative decoding, `vllm preload` additionally
116 starts a draft daemon group that caches the draft model. It uses its own cache
117 key, Unix sockets (`*_draft.sock`) and rendezvous port
118 (`--weight-cache-draft-master-port`, default `--weight-cache-master-port + 1`),
119 so each daemon process serves exactly one model role. Other draft types are
120 not cached and keep loading from disk in the engine.
121
122 ## Cache modes
123
124 The loader supports two modes, selected via `--model-loader-extra-config`:
125
126 - `zero_copy` (default): the engine maps the daemon's CUDA allocations
127 directly. The daemon must stay alive for the engine's lifetime, and each
128 GPU carries only one copy of the weights.
129 - `copy`: the engine clones every tensor into its own GPU memory and then
130 asks the daemon to release its cache. Use this when the daemon should free
131 GPU memory after handing off, at the cost of a full copy per restart.
132
133 !!! warning
134 In `zero_copy` mode the weights live in the daemon's CUDA IPC allocations,
135 so [sleep mode](sleep_mode.md) (CuMemAllocator weight offloading) must not
136 be used with this loader.
137
138 ## Loader configuration
139
140 The `ipc_cache` loader accepts extra keys via `--model-loader-extra-config`:
141
142 ```bash
143 vllm serve meta-llama/Llama-3.1-8B-Instruct \
144 --load-format ipc_cache \
145 --model-loader-extra-config '{"mode": "copy", "fallback": false}'
146 ```
147
148 | Key | Default | Description |
149 | --- | ------- | ----------- |
150 | `socket_path` | per-GPU path derived from the GPU UUID | Explicit daemon socket path. |
151 | `socket_dir` | per-user private dir under the temp dir | Directory containing the daemon sockets. |
152 | `mode` | `zero_copy` | `zero_copy` or `copy` (see above). |
153 | `fallback` | `true` | Fall back to disk loading when the daemon is unavailable or the fingerprints mismatch. |
154 | `connect_timeout_s` | `5.0` | Socket connect timeout in seconds. |
155 | `state_timeout_s` | `300.0` | Timeout for the weight-transfer request in seconds. |
156
157 The daemon's `--weight-cache-socket-dir` and the loader's `socket_dir` /
158 `socket_path` must agree when the default per-user directory is not used.
159 Socket paths are derived from the GPU UUID, so they are stable regardless of
160 `CUDA_VISIBLE_DEVICES` index remapping.
161
162 ## Limitations
163
164 - **Platform**: CUDA and ROCm only; other platforms raise
165 `UnsupportedPlatformForIPCError` even when `fallback` is enabled, since it
166 is a permanent misconfiguration rather than a transient daemon outage.
167 - **Parallelism**: tensor, expert and data parallelism are supported;
168 launching the daemon with pipeline parallelism is rejected.
169 - **Quantization**: every quantization method in the model must declare
170 support for pre-processed weights (the daemon transfers weights *after*
171 quantization post-processing). Unsupported methods raise
172 `UnsupportedQuantForIPCError`.
173 - **Consistency**: the daemon and engine must run the same vLLM version and
174 agree on model, dtype, quantization, and TP/DP layout, otherwise the
175 fingerprint mismatch triggers the disk fallback.
176 - **Localhost only**: the daemon serves over a Unix domain socket; both the
177 daemon and the engines must run on the same node as the same user.
178
179 ## Security
180
181 The socket protocol uses pickle and is intended only for trusted local
182 processes owned by the same user:
183
184 - Daemon sockets live in a per-user private directory (mode `0700`) and the
185 socket files are restricted to the owner (`0600`).
186 - Both sides reject symlinked or non-owned socket paths; the auto-derived
187 directory is additionally rejected if it is group/world accessible.
188 - On Linux the daemon verifies the connecting peer's UID via `SO_PEERCRED`.
189
190 See the [security documentation](../usage/security.md) for vLLM's general
191 threat model.
192
193 ## Daemon lifecycle
194
195 - Each GPU's socket path is guarded by an exclusive lock file, so a second
196 daemon for the same GPU fails fast instead of clobbering the live socket.
197 - `vllm preload` reports readiness once every rank is serving; if any rank
198 dies during startup, the remaining ranks are terminated and the command
199 exits with that rank's exit code.
200 - `SIGINT`/`SIGTERM` to the `vllm preload` process terminates all daemon
201 ranks, which unlinks their sockets and releases the GPU memory.
202
203 See [vllm preload](../cli/preload.md) for the full CLI reference.