@cryptotaxi247 / netdata
Code Commits Issues Pulls Sessions CI/CD
master
master
md 175 lines 7.74 KB
Rendered Raw
1 # Node Types and Lifecycle Strategies
2
3 Netdata categorizes nodes as **ephemeral** or **permanent** to help you tailor alerting, cleanup, and monitoring strategies for dynamic or static infrastructures.
4
5 ## Node Types
6
7 | Type | Description | Common Use Cases |
8 |---------------|------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
9 | **Ephemeral** | Expected to disconnect or reconnect frequently | • Auto-scaling cloud instances<br />• Dynamic containers and VMs<br />• IoT devices with intermittent connectivity<br />• Test environments |
10 | **Permanent** | Expected to maintain continuous connectivity | • Production servers<br />• Core infrastructure nodes<br />• Critical monitoring systems<br />• Stable database servers |
11
12 :::note
13
14 Disconnections in **permanent nodes** may indicate system failures and require immediate attention.
15
16 :::
17
18 ### Key Benefits of Ephemeral Nodes
19
20 1. **Reduced Alert Noise**: Disconnection alerts apply only to permanent nodes.
21 2. **Support for Dynamic Infrastructure**: Designate temporary resources as ephemeral to avoid false alarms.
22 3. **Automated Cleanup**: Configure retention policies for ephemeral nodes to keep dashboards uncluttered.
23
24 ## Configuring Ephemeral Nodes
25
26 By default, Netdata treats all nodes as permanent. To mark a node as ephemeral:
27
28 1. Open the `netdata.conf` file on the target node.
29 2. Add the following configuration:
30 ```ini
31 [global]
32 is ephemeral node = yes
33 ```
34 3. Restart the Netdata Agent.
35
36 Once restarted, Netdata automatically applies the `_is_ephemeral` host label to the node. This label propagates to your Parent nodes and appears in Netdata Cloud, allowing the system to identify and handle the node as ephemeral.
37
38 ## Alerts for Parent Nodes
39
40 Netdata v2.3.0 introduces two alerts specific to permanent nodes:
41
42 | Alert | Trigger Condition |
43 |-----------------------------|---------------------------------------------------------|
44 | `streaming_never_connected` | A permanent node has never connected to a Parent. |
45 | `streaming_disconnected` | A previously connected permanent node has disconnected. |
46
47 ## Automatic Node Instance Cleanup in Netdata Cloud
48
49 Netdata Cloud automatically removes inactive nodes to keep your dashboards clean and organized.
50
51 ### Cleanup Rules
52
53 | Node Type | Offline Duration | Description |
54 |------------------------------|------------------|------------------------------------------------------------------------------------------------------------------------------------|
55 | **Child nodes** | 48 hours | Nodes that connect through a Parent node. Deleted when:<br />• The child node goes offline, OR<br />• The Parent node goes offline |
56 | **Directly connected nodes** | 7 days | Nodes claimed directly to Netdata Cloud without going through a Parent. |
57 | **Ephemeral nodes** | Custom | Temporary nodes (containers, auto-scaling VMs) with configurable cleanup periods. See configuration below. |
58 | **Unseen nodes** | 48 hours | Nodes that were claimed but have never connected to Netdata Cloud. |
59
60 :::tip
61
62 **Stale nodes are NOT automatically deleted.** Only Offline and Unseen nodes are subject to automatic cleanup. A Stale node has historical data available via a Parent, so it's preserved until the Parent goes offline or the data expires.
63
64 :::
65
66 :::info Important
67
68 If a child node is deleted and later reconnects, it's automatically recreated in Netdata Cloud. Any historical data retained on the Parent node remains accessible.
69
70 :::
71
72 ## Monitoring and Managing Node Status
73
74 ### Mark Permanently Offline Nodes as Ephemeral
75
76 To mark nodes (including virtual ones) as ephemeral:
77
78 ```bash
79 netdatacli mark-stale-nodes-ephemeral <node_id | machine_guid | hostname | ALL_NODES>
80 ```
81
82 This keeps historical data queryable and clears active alerts.
83
84 ```mermaid
85 flowchart TD
86 A("**Offline Node Detected**") -->|Run CLI Command| B("**Use netdatacli mark-stale-nodes-ephemeral**")
87 B --> C("**Node Marked as Ephemeral**")
88 C --> D("**Metrics Remain Available**")
89 C --> E("**Active Alerts Cleared**")
90 C --> F{"**Node Reconnects?**"}
91 F -->|Yes - no config| G("**Reverts to Permanent**")
92 F -->|No| H("**Remains Ephemeral**")
93 classDef step fill: #4caf50, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 16px
94 classDef alert fill: #ffeb3b, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 16px
95 class A step
96 class B step
97 class C step
98 class D step
99 class E step
100 class F alert
101 class G alert
102 class H alert
103 ```
104
105 ### Removing Offline Nodes
106
107 To fully remove permanently offline nodes:
108
109 ```bash
110 netdatacli remove-stale-node <node_id | machine_guid | hostname | ALL_NODES>
111 ```
112
113 :::note
114
115 For detailed instructions on removing nodes from Netdata Cloud (including **offline** and **stale** nodes, bulk operations, and UI-based removal), see the [Remove Node Guide](https://github.com/netdata/netdata/edit/master/docs/learn/remove-node.md). This covers scenarios where UI removal is disabled due to parent-child configured relationships.
116
117 :::
118
119 ```mermaid
120 flowchart TD
121 A("**Offline Node Detected**") -->|Run CLI Tool| B("**Execute remove-stale-node Command**")
122 B --> C("**Node Removed from System**")
123 C --> D("**Node No Longer Queryable**")
124 C --> E("**Alerts for Node Cleared**")
125 classDef step fill: #4caf50, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 16px
126 class A step
127 class B step
128 class C step
129 class D step
130 class E step
131 ```
132
133 ## Automatically Removing Ephemeral Nodes
134
135 To enable automatic cleanup of ephemeral nodes:
136
137 1. Open the `netdata.conf` file on Netdata Parent nodes.
138 2. Add the following configuration:
139
140 ```ini
141 [db]
142 cleanup ephemeral hosts after = 1d
143 ```
144
145 3. Restart the Netdata Agent.
146
147 This removes ephemeral nodes after 24 hours of disconnection. Once all Parents purge the node, it is automatically removed from Netdata Cloud.
148
149 ```mermaid
150 flowchart TD
151 A("**Configure Auto-Removal in netdata.conf**") --> B("**Restart Parent Nodes**")
152 B --> C("**Ephemeral Node Disconnects**")
153 C --> D{"**Wait Period Elapsed?**"}
154 D -->|Yes| E("**Node Automatically Removed**")
155 D -->|No| F("**Node Remains in System**")
156 E --> G{"**All Parents Removed Node?**"}
157 G -->|Yes| H("**Node Removed from Cloud**")
158 classDef step fill: #4caf50, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 16px
159 classDef alert fill: #ffeb3b, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 16px
160 classDef database fill: #2196F3, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 16px
161 class A step
162 class B step
163 class C step
164 class D alert
165 class E alert
166 class F alert
167 class G database
168 class H database
169 ```
170
171 ## See Also
172
173 - [Node States and Transitions](/docs/netdata-cloud/node-states-and-transitions.md) - Comprehensive reference for node states (Live, Stale, Offline, Unseen) and transition triggers
174 - [Node Identities](/docs/learn/node-identities.md) - Understand how node identity works alongside ephemerality
175 - [VM Templates](/docs/learn/vm-templates.md) - Configure ephemerality in VM templates for auto-scaling groups