@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 152 lines 7.82 KB
Rendered Raw
1 # Initialized engine snapshots
2
3 Initialized engine snapshots are an experimental way to trade local disk space
4 and host privileges for a faster vLLM activation. Snapshot creation initializes
5 the engine and records deterministic generation output before CRIU captures the
6 process tree and CUDA state. Restore validates the saved environment, restores
7 the engine, binds the HTTP server, and reproduces the recorded token and sampled
8 token log probability before returning.
9
10 This path is intended for repeatedly activating the same model and engine
11 configuration on the same machine. It is not a portable model artifact.
12
13 ## Requirements
14
15 Snapshots currently require:
16
17 - Linux on x86-64 with one NVIDIA GPU.
18 - TP1 with one unauthenticated plaintext HTTP server. Other parallel sizes,
19 TLS, middleware, Unix sockets, and speculative decoding are unsupported.
20 - Because snapshot mode requires an unauthenticated plaintext HTTP server, use
21 a trusted host or network boundary, or external controls. See the vLLM
22 [Security guide](../usage/security.md).
23 - [CRIU](https://github.com/checkpoint-restore/criu), its CUDA plugin, a
24 `cuda-checkpoint`-compatible helper, and `nvidia-smi` on `PATH`.
25 - Root or passwordless `sudo` for CRIU.
26 - `io_uring` disabled before launch because CRIU cannot dump it. Use
27 `kernel.io_uring_disabled=1` for an unprivileged process. A process running
28 as root, including the `docker exec` flow below, bypasses `=1`, so use `=2`
29 host-wide.
30 - No established TCP connection to a peer outside the captured process tree.
31 Current Hugging Face hub clients hold their connections for the process
32 lifetime, so download the model in a separate step and run create with
33 `HF_HUB_OFFLINE=1`, as the quickstart below does.
34 - A remote model ID and an immutable 40-character `--revision`. Local model
35 directories and mutable revisions are not supported.
36 - Enough disk for the artifact, with the same installed vLLM package, model
37 files, container filesystem, and generated-cache paths available at restore.
38 Generated-cache files are not copied into the artifact. The manifest
39 fingerprints the ones the captured tree holds open, so restore fails early
40 and names the file when one is removed or replaced. Only open descriptors
41 are recorded. A library the tree mapped and then closed is not fingerprinted,
42 and CRIU still reopens it by path, so replacing that file permanently
43 invalidates the artifact without an early error.
44
45 The official CUDA 13 `vllm/vllm-openai` Linux x86-64 images include the
46 snapshot runtime. CUDA 12.x and Arm64 images omit it. A compatible host
47 driver, kernel, and privileges are still required. Source installs must set
48 `CRIU_CUDA_PLUGIN_DIR` to the directory containing `cuda_plugin.so`.
49
50 Run snapshot commands with `docker exec` inside a long-lived container. Restore
51 hands the API server off as a detached process, so a one-shot container would
52 stop that server when its PID 1 exits. Snapshot preflight requires every
53 component of the artifact path to be owned by the invoking user or root, with
54 the directory itself at mode 0700, and the commands below run as root inside
55 the official image, so the bind-mounted host directory is created with `sudo`
56 and root ownership. The model downloads in its own step so the captured tree
57 holds no hub connection, and create runs offline. This example also keeps the
58 container filesystem and `/dev/shm` namespace stable for the lifetime of the
59 artifact:
60
61 ```bash
62 sudo sysctl kernel.io_uring_disabled=2
63
64 snapshot_root="$(pwd)/vllm-snapshots"
65 sudo install -d -m 0700 -o root -g root "${snapshot_root}"
66
67 docker run --detach --name vllm-snapshot \
68 --gpus all \
69 --privileged \
70 --pid=host \
71 --ipc=host \
72 --network=host \
73 --mount "type=bind,source=${snapshot_root},target=/snapshots" \
74 --entrypoint sleep \
75 vllm/vllm-openai:latest infinity
76
77 docker exec vllm-snapshot hf download Qwen/Qwen3-0.6B \
78 --revision c1899de289a04d12100db370d81485cdf75e47ca
79
80 docker exec -e HF_HUB_OFFLINE=1 vllm-snapshot vllm snapshot create Qwen/Qwen3-0.6B \
81 --snapshot-dir /snapshots/qwen3-0.6b \
82 --revision c1899de289a04d12100db370d81485cdf75e47ca \
83 --dtype float16 \
84 --max-model-len 512
85
86 docker exec vllm-snapshot vllm snapshot inspect /snapshots/qwen3-0.6b
87
88 docker exec vllm-snapshot vllm snapshot restore \
89 /snapshots/qwen3-0.6b --host 0.0.0.0 --port 8000
90 ```
91
92 Keep that container and its mounts available while the snapshot is in use.
93 Stop and remove it only after the restored API server is no longer needed.
94
95 Create initializes the engine, records a one-token canary, releases and reloads
96 weights and KV cache to rehearse restore, then releases them again for capture.
97 The manifest is published only after CRIU completes and the source tree stops.
98 Creation is offline preparation and is not part of restore latency.
99
100 The private `0700` artifact contains process memory, CUDA state, engine
101 arguments, compatibility identity, and canary output; its manifest is `0600`.
102 Literal API keys and Hugging Face tokens are redacted from the manifest's engine
103 arguments. The manifest records selected environment names plus deterministic
104 name-bound fingerprints, not their values. The protected CRIU artifact can
105 still contain process secrets, so treat it as sensitive data. Restore reloads
106 model files and KV cache before binding HTTP, then reproduces the canary or
107 tears down the restored tree. The inspect command prints that identity and
108 canary without executing the saved process.
109
110 ## Restore behavior
111
112 Restore fails before CRIU runs if the saved identity does not match the current
113 host. It does not silently fall back to ordinary startup. After CRIU restores
114 the process tree, vLLM releases the saved engine to bind the requested HTTP
115 address and checks the first generated token and sampled-token log probability
116 against the snapshot canary. The command returns only after that check passes.
117 The restored API server continues to run as a detached process.
118
119 The pre-release port probe is best-effort only. It neither reserves the port
120 nor authenticates the listener that appears afterward.
121
122 Rollback terminates and waits for the restored tree after process identity
123 verification succeeds. If verification fails, vLLM writes an abort marker for
124 the snapshot server instead of signaling unverified PIDs. If that server has
125 already exited, surviving engine processes can require operator cleanup.
126 Verify that the previous tree has stopped before retrying; do not kill a process
127 solely because its PID appears in the manifest, since PIDs can be reused.
128
129 The artifact is reusable after its previous restored tree stops. Only one
130 snapshot or external CRIU operation may use a shared `/dev/shm` mount at a time.
131
132 ## Tradeoffs and limitations
133
134 - Creation has its own latency and briefly requires the full engine. Artifact
135 size can approach the captured process and GPU memory.
136 - Artifacts are not guaranteed to survive a power loss. The manifest is
137 fsynced, but publication does not explicitly flush all captured payload files
138 and directories. Recreate the artifact after an unclean host shutdown.
139 - Restore currently requires the same host, GPU, driver, kernel, Python,
140 PyTorch, installed vLLM version, model revision, engine arguments,
141 selected environment variables, and CRIU plugin binaries.
142 - Only dense float16 TP1 has been validated. Other model formats depend on their
143 existing sleep level 2 reload support; NCCL state is not restored.
144 - CRIU support varies by kernel and driver. Preserve package, library, model,
145 and generated-cache paths for the artifact lifetime.
146 - A snapshot can include application secrets or request state present in the
147 process. Build it before traffic and protect it like process memory.
148 - Any feature that opens an external connection must close it before capture.
149
150 For a lower-complexity option that retains a live process, see
151 [Sleep mode](sleep_mode.md). Sleep mode and initialized snapshots retain
152 different amounts of state and have different idle resource costs.