1
+# Node States
2
+
3
+Netdata provides dashboards at multiple levels of your infrastructure. Each level displays node states based on what it can observe. This page explains what each state means, when transitions happen, and how to configure the behavior.
4
+
5
+## Dashboard Levels
6
+
7
+Netdata's distributed architecture provides three observation points:
8
+
9
+| Dashboard | What You See | How to Access |
10
+|-----------|--------------|---------------|
11
+| **Agent** | Local node only | `http://node-ip:19999` |
12
+| **Parent** | All nodes streaming to this Parent | `http://parent-ip:19999` |
13
+| **Netdata Cloud** | All nodes claimed to your Space | `https://app.netdata.cloud` |
14
+
15
+**How data flows:**
16
+
17
+```
18
+Agent → (streaming) → Parent → (ACLK) → Netdata Cloud
19
+Agent → (ACLK) → Netdata Cloud (standalone, no Parent)
20
+```
21
+
22
+Node states reflect this flow: if a link breaks, states change based on where data is still available.
23
+
24
+## States on Netdata Cloud
25
+
26
+| State | Meaning |
27
+|-------|---------|
28
+| **Live** | Node is connected to Netdata Cloud (directly or via Parents) and providing live metrics |
29
+| **Stale** | Node disconnected, but a Parent connected to Netdata Cloud has its historical data |
30
+| **Offline** | Node is disconnected and no data is available |
31
+| **Unseen** | Node was claimed but has never connected |
32
+
33
+### Stale vs Offline
34
+
35
+The difference is **data availability**:
36
+
37
+| Scenario | State | Can Query Data? |
38
+|----------|-------|-----------------|
39
+| Child disconnected, Parent connected to Cloud | **Stale** | Yes, via Parent |
40
+| Standalone Agent disconnected | **Offline** | No |
41
+| Child disconnected, all Parents disconnected from Cloud | **Offline** | No |
42
+
43
+Stale nodes remain queryable because the Parent serves as a data cache. This is why you cannot delete Stale nodes from the UI—they still have accessible data.
44
+
45
+## States on Parent Dashboards
46
+
47
+Parents display nodes that stream (or have streamed) to them:
48
+
49
+| State | Meaning |
50
+|-------|---------|
51
+| **Live** | Node is actively streaming metrics |
52
+| **Stale** | Node stopped streaming, historical data retained |
53
+
54
+Parents don't show Offline or Unseen states. When a node's retention expires or cleanup runs, it disappears from the Parent's view entirely.
55
+
56
+## State Mapping: Parent → Cloud
57
+
58
+When a Parent connects to Netdata Cloud, it reports the state of all its children:
59
+
60
+| Parent Sees | Cloud Shows | Why |
61
+|-------------|-------------|-----|
62
+| Live | **Live** | Data flowing through Parent |
63
+| Stale | **Stale** | Parent connected to Cloud has historical data |
64
+| (removed) | **Offline** | No data source available |
65
+
66
+**High-availability setups (recommended):**
67
+With two Parents (Child → P1 → P2), children stream to one Parent, which replicates to the other. Both Parents connect to Cloud.
68
+
69
+If the child connects to Cloud only via Parents:
70
+
71
+| Event | Result | Why |
72
+|-------|--------|-----|
73
+| P1 disconnects from Cloud | No change (Live) | P1 still runs, replicates to P2, P2 reports to Cloud |
74
+| P1 stops | Brief Stale, then Live | Child fails over to P2 |
75
+| P2 disconnects from Cloud | No change (Live) | P1 still connected to Cloud |
76
+| P2 stops | No change (Live) | Child still streams to P1, P1 reports to Cloud |
77
+| Both disconnect from Cloud | **Offline** | No Parent can report to Cloud |
78
+
79
+If the child also connects directly to Cloud, it remains Live regardless of Parent status.
80
+
81
+**Single Parent setups:**
82
+When the only Parent disconnects from Cloud, all its children become **Offline** because Cloud can no longer query their data.
83
+
84
+**When a Parent reconnects:**
85
+Children with retained data appear as **Stale** (or **Live** if actively streaming).
86
+
87
+## Transition Timings
88
+
89
+### Detection Speed
90
+
91
+**Netdata Cloud detects agent/parent disconnection:**
92
+
93
+| Event | Detection Time | Mechanism |
94
+|-------|----------------|-----------|
95
+| Agent or Parent loses Cloud connection | **~60 seconds** | MQTT keepalive (60s interval) |
96
+| UI reflects state change | **1-2 minutes** | Cloud processing + UI refresh |
97
+
98
+**Parent detects child disconnection:**
99
+
100
+| Event | Detection Time | Mechanism |
101
+|-------|----------------|-----------|
102
+| Child shuts down gracefully | **Immediate** | Socket close detected |
103
+| Child crashes or network drops | **~60 seconds** | TCP keepalive probes (30s idle + 3×10s probes) |
104
+| Child silently stops sending data | **10 minutes** | Idle activity timeout |
105
+
106
+These timings are hardcoded and not user-configurable.
107
+
108
+### Standalone Agent Transitions
109
+
110
+A standalone Agent connects directly to Cloud without a Parent.
111
+
112
+| Event | From | To | Timing |
113
+|-------|------|-----|--------|
114
+| Agent starts, connects to Cloud | Unseen/Offline | **Live** | Immediate on connection |
115
+| Agent stops or loses network | Live | **Offline** | Immediate to ~60 seconds |
116
+| Agent restarts | Offline | **Live** | Immediate on reconnection |
117
+
118
+**No Stale state**: Standalone agents go directly to Offline because there's no Parent holding their data.
119
+
120
+### Child Node Transitions
121
+
122
+A child streams metrics to a Parent, which connects to Cloud.
123
+
124
+| Event | From | To | Timing |
125
+|-------|------|-----|--------|
126
+| Child connects to Parent | Unseen/Offline | **Live** | Immediate |
127
+| Child stops streaming | Live | **Stale** | Immediate to ~60 seconds (see Detection Speed) |
128
+| Child restarts streaming | Stale | **Live** | Immediate |
129
+| All Parents go offline | Live/Stale | **Offline** | Immediate to ~60 seconds |
130
+| Parent reconnects (child still down) | Offline | **Stale** | Immediate (if data retained) |
131
+
132
+### First Connection
133
+
134
+| Event | From | To | Timing |
135
+|-------|------|-----|--------|
136
+| Node claimed to Space | - | **Unseen** | Immediate |
137
+| Node connects for first time | Unseen | **Live** | Immediate on connection |
138
+
139
+## Automatic Cleanup
140
+
141
+### Netdata Cloud Cleanup
142
+
143
+Cloud automatically removes nodes that remain Offline or Unseen:
144
+
145
+| Node Type | Cleanup After | Notes |
146
+|-----------|---------------|-------|
147
+| Standalone agents (0 hops) | **7 days** | Direct Cloud connection |
148
+| Child nodes (1+ hops) | **48 hours** | Connected via Parent |
149
+| Unseen nodes | **48 hours** | Claimed but never connected |
150
+
151
+**Stale nodes are never automatically removed.** They have queryable data via their Parent.
152
+
153
+These thresholds are managed by Netdata Cloud infrastructure and are not user-configurable.
154
+
155
+## Configuration Options
156
+
157
+### Ephemeral Nodes
158
+
159
+For dynamic infrastructure (auto-scaling groups, containers, spot instances), mark nodes as ephemeral:
160
+
161
+```ini
162
+# On the child node's netdata.conf
163
+[global]
164
+ is ephemeral node = yes
165
+```
166
+
167
+**Effects:**
168
+- No disconnection alerts for this node
169
+- Node label `_is_ephemeral=true` propagates to Parents and Cloud
170
+
171
+### Marking Existing Nodes as Ephemeral
172
+
173
+To mark already-offline nodes as ephemeral (clears alerts, keeps data queryable):
174
+
175
+```bash
176
+netdatacli mark-stale-nodes-ephemeral <node-id | hostname | ALL_NODES>
177
+```
178
+
179
+### Removing Nodes
180
+
181
+To force-remove a Stale node:
182
+
183
+```bash
184
+netdatacli remove-stale-node <node-id | hostname | ALL_NODES>
185
+```
186
+
187
+This sends an offline signal to Cloud. The node transitions to Offline and becomes eligible for cleanup (or immediate UI deletion).
188
+
189
+See [Remove Node](/docs/learn/remove-node.md) for detailed instructions.
190
+
191
+### Connection Hops
192
+
193
+Check how a node connects to Cloud:
194
+
195
+| Hops | Meaning |
196
+|------|---------|
197
+| 0 | Direct Cloud connection (standalone) |
198
+| 1 | Connected via one Parent |
199
+| 2+ | Connected via chained Parents |
200
+
201
+View hops in Netdata Cloud by clicking the node info button.
202
+
203
+Nodes with more hops have more potential failure points, but also benefit from Parent data caching (Stale state instead of Offline).
204
+
205
+## Troubleshooting
206
+
207
+### Log Filtering with MESSAGE_ID
208
+
209
+Netdata logs include MESSAGE_IDs for filtering specific events. Use `journalctl` to view relevant logs:
210
+
211
+```bash
212
+# Cloud connection events (ACLK)
213
+journalctl -u netdata MESSAGE_ID=acb33cb9-5778-476b-aac7-02eb7e4e151d
214
+
215
+# Streaming from children (on Parent)
216
+journalctl -u netdata MESSAGE_ID=ed4cdb8f-1beb-4ad3-b57c-b3cae2d162fa
217
+
218
+# Streaming to parent (on Child)
219
+journalctl -u netdata MESSAGE_ID=6e2e3839-0676-4896-8b64-6045dbf28d66
220
+```
221
+
222
+### Node shows Stale, expected Live
223
+
224
+**Cause:** Node stopped streaming to its Parent.
225
+
226
+**Check:**
227
+1. Is the node's Netdata Agent running? `systemctl status netdata`
228
+2. Can the node reach the Parent? Check network/firewall
229
+3. Check streaming config: `cat /etc/netdata/stream.conf`
230
+4. Check agent logs: `journalctl -u netdata | grep -i stream`
231
+
232
+### Node shows Offline, expected Stale
233
+
234
+**Cause:** Either it's a standalone Agent, or all its Parents are disconnected from Cloud.
235
+
236
+**Check:**
237
+1. Is this a standalone Agent or does it stream to a Parent?
238
+2. If streaming: Is the Parent online and connected to Cloud?
239
+3. Check Parent's dashboard—does it show the child?
240
+
241
+### Node shows Unseen
242
+
243
+**Cause:** Node was claimed but never successfully connected to Cloud.
244
+
245
+**Check:**
246
+1. Is the Netdata Agent running?
247
+2. Can the agent reach `app.netdata.cloud`? Check firewall/proxy
248
+3. Is the claiming token correct?
249
+4. Check agent logs: `journalctl -u netdata | grep -i aclk`
250
+
251
+### All children went Offline simultaneously
252
+
253
+**Cause:** All Parents lost their Cloud connection. With HA setups (two Parents), this only happens if both disconnect.
254
+
255
+**Check:**
256
+1. Are the Parents online? `systemctl status netdata`
257
+2. Can they reach Cloud? Check network
258
+3. Check Parent logs: `journalctl -u netdata | grep -i aclk`
259
+
260
+### Can't delete node from UI
261
+
262
+**Cause:** Node is Stale (has data via Parent). UI prevents deletion to protect queryable data.
263
+
264
+**Solution:** Use CLI to remove:
265
+```bash
266
+netdatacli remove-stale-node <node-id>
267
+```
268
+
269
+### Node reappears after deletion
270
+
271
+**Cause:** Agent is still running and configured to reconnect.
272
+
273
+**Solution:**
274
+1. Stop the agent: `systemctl stop netdata`
275
+2. Remove claim: `rm /var/lib/netdata/cloud.d/claimed_id`
276
+3. Clear environment variables if set
277
+
278
+## See Also
279
+
280
+- [Node Types and Lifecycle](/docs/nodes-ephemerality.md) - Ephemeral vs permanent nodes
281
+- [Remove Node](/docs/learn/remove-node.md) - Detailed removal instructions
282
+- [Streaming Configuration](/docs/observability-centralization-points/metrics-centralization-points/README.md) - Parent-child setup