| 1 | # VM Templates and Clones |
| 2 | |
| 3 | :::danger |
| 4 | |
| 5 | **Destructive Operations - Data Loss Warning** |
| 6 | |
| 7 | The commands in this guide **permanently delete**: |
| 8 | - All historical metrics |
| 9 | - [Node identity](/docs/learn/node-identities.md#agent-self-identity) |
| 10 | - [Cloud connection](/docs/learn/node-identities.md#agent-cloud-link-aclk-identity) |
| 11 | - Alert history |
| 12 | |
| 13 | **This is irreversible. There is no undo.** |
| 14 | |
| 15 | Only run these commands on VMs you intend to convert to templates. |
| 16 | Running these on a production system will destroy your monitoring data. |
| 17 | |
| 18 | ::: |
| 19 | |
| 20 | :::tip |
| 21 | |
| 22 | **What You'll Learn** |
| 23 | |
| 24 | How to prepare a VM template so each clone gets a unique Netdata identity and automatically connects to Netdata Cloud. |
| 25 | |
| 26 | ::: |
| 27 | |
| 28 | ## Prerequisites |
| 29 | |
| 30 | - **Read first**: [Node Identities](/docs/learn/node-identities.md) - understand what you're deleting |
| 31 | - Netdata installed on a VM |
| 32 | - Hypervisor that supports templates or golden images |
| 33 | - (Optional) `/etc/netdata/claim.conf` configured for auto-claiming to Cloud |
| 34 | |
| 35 | ## Overview |
| 36 | |
| 37 | To prepare a VM template: |
| 38 | |
| 39 | 1. **Stop Netdata** - Prevent file regeneration |
| 40 | 2. **Delete identity and data files** - Force new identity on clone boot |
| 41 | 3. **Keep claim.conf** - Enable auto-claiming (optional) |
| 42 | 4. **Convert to template** - Without starting Netdata |
| 43 | |
| 44 | ## Node Types: Ephemeral vs Permanent |
| 45 | |
| 46 | VMs cloned from templates can be configured as **ephemeral** or **permanent** nodes. This affects how Netdata handles disconnections, alerts, and cleanup. |
| 47 | |
| 48 | | Type | Behavior | Use Case | |
| 49 | |------|-----------|----------| |
| 50 | | **Ephemeral** | No alerts on disconnect, auto-cleanup after 24h | Auto-scaling instances, spot VMs, short-lived workloads | |
| 51 | | **Permanent** | Alerts trigger on disconnect | Long-running production systems | |
| 52 | |
| 53 | ### Configuring Ephemeral Nodes in Templates |
| 54 | |
| 55 | To make cloned VMs ephemeral by default, add to `/etc/netdata/netdata.conf` **in the template**: |
| 56 | |
| 57 | ```ini |
| 58 | [global] |
| 59 | is ephemeral node = yes |
| 60 | ``` |
| 61 | |
| 62 | ### When to Use Each Type |
| 63 | |
| 64 | - **Ephemeral**: Auto-scaling groups, spot instances, Kubernetes pod templates, CI/CD build agents |
| 65 | - **Permanent**: Production database servers, core infrastructure, stable monitoring targets |
| 66 | |
| 67 | See [Node Ephemerality](/docs/nodes-ephemerality.md) for full documentation. |
| 68 | |
| 69 | ## Files to Delete |
| 70 | |
| 71 | :::danger |
| 72 | |
| 73 | **Verify you are on the correct VM before running these commands.** |
| 74 | |
| 75 | ::: |
| 76 | |
| 77 | | Category | Files | What's Lost | |
| 78 | |----------|-------|-------------| |
| 79 | | **[Agent Identity](/docs/learn/node-identities.md#agent-self-identity)** | [GUID file](/docs/learn/node-identities.md#agent-self-identity), [status backups](/docs/learn/node-identities.md#status-file-backups) | Node identity | |
| 80 | | **[ACLK Auth](/docs/learn/node-identities.md#agent-cloud-link-aclk-identity)** | [`cloud.d/`](/docs/learn/node-identities.md#agent-cloud-link-aclk-identity) directory | Cloud connection, must re-claim | |
| 81 | | **[Node Metadata](/docs/learn/node-identities.md#parent-children-identities)** | `netdata-meta.db*`, `context-meta.db*` | Node metadata, metric mappings | |
| 82 | | **Metrics** | `dbengine*` directories (all tiers) | All historical metrics | |
| 83 | |
| 84 | **Keep**: `/etc/netdata/claim.conf` - enables auto-claiming on clones |
| 85 | |
| 86 | ## Step-by-Step |
| 87 | |
| 88 | ### 1. Stop Netdata |
| 89 | |
| 90 | ```bash |
| 91 | sudo systemctl stop netdata |
| 92 | ``` |
| 93 | |
| 94 | ### 2. Delete All Identity and Data Files |
| 95 | |
| 96 | :::danger |
| 97 | |
| 98 | **Point of No Return** |
| 99 | |
| 100 | The following commands permanently delete Netdata data. Verify you are on the template VM. |
| 101 | |
| 102 | ::: |
| 103 | |
| 104 | ```bash |
| 105 | # Machine GUID (Agent Self Identity) |
| 106 | sudo rm -f /var/lib/netdata/registry/netdata.public.unique.id |
| 107 | |
| 108 | # Status file backups (GUID recovery locations) |
| 109 | sudo rm -f /var/lib/netdata/status-netdata.json |
| 110 | sudo rm -f /var/cache/netdata/status-netdata.json |
| 111 | sudo rm -f /tmp/status-netdata.json |
| 112 | sudo rm -f /run/status-netdata.json |
| 113 | sudo rm -f /var/run/status-netdata.json |
| 114 | |
| 115 | # ACLK authentication (Claimed ID, RSA keys) |
| 116 | sudo rm -rf /var/lib/netdata/cloud.d/ |
| 117 | |
| 118 | # Databases and metrics (metadata, all dbengine tiers) |
| 119 | sudo rm -f /var/cache/netdata/netdata-meta.db* |
| 120 | sudo rm -f /var/cache/netdata/context-meta.db* |
| 121 | sudo rm -rf /var/cache/netdata/dbengine* |
| 122 | ``` |
| 123 | |
| 124 | ### 3. Configure Auto-Claiming (Optional) |
| 125 | |
| 126 | To have clones automatically claim to Netdata Cloud on first boot, ensure `/etc/netdata/claim.conf` exists: |
| 127 | |
| 128 | ```bash |
| 129 | cat /etc/netdata/claim.conf |
| 130 | ``` |
| 131 | |
| 132 | Should contain: |
| 133 | ```ini |
| 134 | [global] |
| 135 | url = https://app.netdata.cloud |
| 136 | token = YOUR_SPACE_TOKEN |
| 137 | rooms = ROOM_ID |
| 138 | ``` |
| 139 | |
| 140 | ### 4. Convert to Template |
| 141 | |
| 142 | **Do not start Netdata.** Convert the VM to a template using your hypervisor. |
| 143 | |
| 144 | ## When Clones Boot |
| 145 | |
| 146 | 1. Netdata starts, no [GUID](/docs/learn/node-identities.md#agent-self-identity) found, generates new unique identity |
| 147 | 2. If `claim.conf` exists, auto-claims to Cloud |
| 148 | 3. Cloud assigns [Node ID](/docs/learn/node-identities.md#cloud-node-identity), new node appears in your Space |
| 149 | |
| 150 | Each clone is a unique, independent node. |
| 151 | |
| 152 | ## Hypervisor Notes |
| 153 | |
| 154 | The Netdata cleanup commands are the same for all hypervisors. The difference is **when** and **how** to run them. |
| 155 | |
| 156 | | Hypervisor | Template Support | When to Clean | Automation | |
| 157 | |------------|------------------|---------------|------------| |
| 158 | | **Proxmox** | Convert to Template | Before conversion | cloud-init scripts | |
| 159 | | **VMware/vSphere** | VM Templates | Before conversion | Guest customization | |
| 160 | | **libvirt/KVM** | virt-sysprep | During sysprep | `--delete` flags | |
| 161 | | **AWS** | AMI | Before image creation | user-data scripts | |
| 162 | | **Azure** | Managed Image | Before capture | cloud-init | |
| 163 | | **GCP** | Machine Image | Before creation | startup scripts | |
| 164 | | **Vagrant** | Box packaging | Before `vagrant package` | Vagrantfile provisioner | |
| 165 | |
| 166 | <details> |
| 167 | <summary><strong>libvirt/KVM: virt-sysprep example</strong></summary> |
| 168 | |
| 169 | ```bash |
| 170 | virt-sysprep -a myvm.qcow2 \ |
| 171 | --delete /var/lib/netdata/registry/netdata.public.unique.id \ |
| 172 | --delete /var/lib/netdata/status-netdata.json \ |
| 173 | --delete /var/cache/netdata/status-netdata.json \ |
| 174 | --delete /tmp/status-netdata.json \ |
| 175 | --delete /run/status-netdata.json \ |
| 176 | --delete /var/run/status-netdata.json \ |
| 177 | --delete /var/lib/netdata/cloud.d \ |
| 178 | --delete '/var/cache/netdata/netdata-meta.db*' \ |
| 179 | --delete '/var/cache/netdata/context-meta.db*' \ |
| 180 | --delete '/var/cache/netdata/dbengine*' |
| 181 | ``` |
| 182 | |
| 183 | </details> |
| 184 | |
| 185 | <details> |
| 186 | <summary><strong>Cloud-init: Fresh install approach</strong></summary> |
| 187 | |
| 188 | Alternative: Install Netdata on first boot instead of templating: |
| 189 | |
| 190 | ```yaml |
| 191 | # cloud-init user-data |
| 192 | runcmd: |
| 193 | - curl -fsSL https://get.netdata.cloud/kickstart.sh -o /tmp/kickstart.sh |
| 194 | - bash /tmp/kickstart.sh --claim-token TOKEN --claim-rooms ROOM_ID |
| 195 | ``` |
| 196 | |
| 197 | Each instance installs fresh with unique identity. |
| 198 | |
| 199 | </details> |
| 200 | |
| 201 | ## Troubleshooting |
| 202 | |
| 203 | ### Clones share the same identity |
| 204 | |
| 205 | Cause: [GUID recovered from status backup](/docs/learn/node-identities.md#status-file-backups). Netdata checks multiple backup locations before generating a new GUID. |
| 206 | |
| 207 | Solution: Delete **all** status file locations, not just the primary GUID file. See the cleanup commands in [Step 2](#2-delete-all-identity-and-data-files). |
| 208 | |
| 209 | ### Clones don't connect to Parent |
| 210 | |
| 211 | Cause: Either clones share the same [Machine GUID](/docs/learn/node-identities.md#agent-self-identity) (only one can connect at a time), or `stream.conf` wasn't configured in the template. |
| 212 | |
| 213 | Solution: |
| 214 | - Verify each clone has a unique GUID: `cat /var/lib/netdata/registry/netdata.public.unique.id` |
| 215 | - Verify `stream.conf` exists and has the correct Parent destination and API key |
| 216 | - If GUIDs are duplicated, run the cleanup on each clone (loses metrics) |
| 217 | |
| 218 | ### Stale "template" node appears in Cloud |
| 219 | |
| 220 | Cause: [Database files kept](/docs/learn/node-identities.md#multiple-node-identities-in-database) from the template. The template's node identity persists in the metadata. |
| 221 | |
| 222 | Solution: Delete databases on all clones. This loses historical metrics but removes the stale node reference. |
| 223 | |
| 224 | ### Clones using Parent profile unexpectedly |
| 225 | |
| 226 | Cause: Template had `stream.conf` with an enabled API key section (configured to receive streams, as Parent). |
| 227 | |
| 228 | Solution: Reset `stream.conf` on clones or delete the API key sections that enable receiving. |
| 229 | |
| 230 | ### Unstable Cloud connections (flapping) |
| 231 | |
| 232 | Cause: Two agents have the same [Machine GUID](/docs/learn/node-identities.md#agent-self-identity). Cloud kicks the older connection offline when the second connects. |
| 233 | |
| 234 | Solution: Each agent needs a unique GUID. Run the cleanup procedure on affected clones. |
| 235 | |
| 236 | ### Clone doesn't auto-claim to Cloud |
| 237 | |
| 238 | Cause: Missing `claim.conf` or environment variables not set. |
| 239 | |
| 240 | Solution: Create `/etc/netdata/claim.conf` with your Space token. |
| 241 | |
| 242 | ### Fixing Already-Deployed Clones |
| 243 | |
| 244 | If clones were deployed with identity files: |
| 245 | |
| 246 | ```bash |
| 247 | # On each affected clone |
| 248 | sudo systemctl stop netdata |
| 249 | |
| 250 | # Machine GUID |
| 251 | sudo rm -f /var/lib/netdata/registry/netdata.public.unique.id |
| 252 | |
| 253 | # Status file backups (all locations) |
| 254 | sudo rm -f /var/lib/netdata/status-netdata.json |
| 255 | sudo rm -f /var/cache/netdata/status-netdata.json |
| 256 | sudo rm -f /tmp/status-netdata.json |
| 257 | sudo rm -f /run/status-netdata.json |
| 258 | sudo rm -f /var/run/status-netdata.json |
| 259 | |
| 260 | # ACLK authentication (if re-claiming to Cloud) |
| 261 | sudo rm -rf /var/lib/netdata/cloud.d/ |
| 262 | |
| 263 | # Databases and metrics |
| 264 | sudo rm -f /var/cache/netdata/netdata-meta.db* |
| 265 | sudo rm -f /var/cache/netdata/context-meta.db* |
| 266 | sudo rm -rf /var/cache/netdata/dbengine* |
| 267 | |
| 268 | sudo systemctl start netdata |
| 269 | ``` |
| 270 | |
| 271 | :::warning |
| 272 | |
| 273 | This deletes all historical metrics on the clone. If you skip deleting `cloud.d/`, you must re-claim to Cloud manually. |
| 274 | |
| 275 | ::: |
| 276 | |
| 277 | ## FAQ |
| 278 | |
| 279 | <details> |
| 280 | <summary>What if I reboot a clone?</summary> |
| 281 | |
| 282 | Identity persists. Netdata only generates a new [GUID](/docs/learn/node-identities.md#agent-self-identity) when the file AND all [backups](/docs/learn/node-identities.md#status-file-backups) are missing. |
| 283 | |
| 284 | </details> |
| 285 | |
| 286 | <details> |
| 287 | <summary>Can multiple clones use the same claim token?</summary> |
| 288 | |
| 289 | Yes. Each clone gets a unique [Machine GUID](/docs/learn/node-identities.md#agent-self-identity) and [Claimed ID](/docs/learn/node-identities.md#agent-cloud-link-aclk-identity). They authenticate with the same token but appear as separate nodes. |
| 290 | |
| 291 | </details> |
| 292 | |
| 293 | <details> |
| 294 | <summary>Do containers need this?</summary> |
| 295 | |
| 296 | No. Containers start with empty volumes, so each gets a unique identity automatically. |
| 297 | |
| 298 | </details> |
| 299 | |
| 300 | <details> |
| 301 | <summary>Is my claim token secure in the template?</summary> |
| 302 | |
| 303 | The token only allows claiming to your Space. It cannot read data or modify other nodes. Treat it like an API key - don't expose publicly, but it's safe in private templates. |
| 304 | |
| 305 | </details> |
| 306 | |
| 307 | <details> |
| 308 | <summary>Should I make my template VMs ephemeral or permanent?</summary> |
| 309 | |
| 310 | Use **ephemeral** for auto-scaling groups, spot instances, or any VMs that may terminate at any time. Use **permanent** for long-running production systems where disconnections indicate problems. |
| 311 | |
| 312 | Configure in the template before conversion: |
| 313 | |
| 314 | ```ini |
| 315 | # For ephemeral (auto-scaling, spot instances) |
| 316 | [global] |
| 317 | is ephemeral node = yes |
| 318 | |
| 319 | # For permanent (default - production systems) |
| 320 | [global] |
| 321 | is ephemeral node = no |
| 322 | ``` |
| 323 | |
| 324 | See [Node Types](/docs/learn/vm-templates.md#node-types-ephemeral-vs-permanent) for full guidance. |
| 325 | |
| 326 | </details> |
| 327 | |
| 328 | <details> |
| 329 | <summary>Does ephemerality affect cleanup behavior?</summary> |
| 330 | |
| 331 | Yes. Ephemeral nodes are automatically cleaned up after 24 hours of disconnection (configurable via `cleanup ephemeral hosts after`). Permanent nodes never auto-cleanup - they remain visible until manually removed, or their metrics are fully rotated due to retention. |
| 332 | |
| 333 | This prevents dashboards from cluttering with stale auto-scaled instances while preserving long-term monitoring data for permanent infrastructure. |
| 334 | |
| 335 | </details> |