Improve ephemerality docs, adding `remove-stale-node` (#20057)
* Improve ephemerality docs, adding `remove-stale-node` * Fix links * Update docs/nodes-ephemerality.md Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * Update docs/nodes-ephemerality.md Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * Update docs/nodes-ephemerality.md Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * Update src/cli/README.md Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> * format --------- Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud>
Ralph Meijer committed
Apr 7, 2025 at 12:16 UTC
fedc7fee64adaa807018cec902347830c768c31f
2 files changed
+40
-17
docs/nodes-ephemerality.md
+21
-3
@@ -4,8 +4,8 @@
4
5
Netdata categorizes nodes into two types:
6
7
-| Type | Description | Common Use Cases |
8
-|---------------|------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
7
+| Type | Description | Common Use Cases |
8
+|---------------|------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
9
| **Ephemeral** | Expected to disconnect or reconnect frequently | - Auto-scaling cloud instances<br/>- Dynamic containers and VMs<br/>- IoT devices with intermittent connectivity<br/>- Development/test environments with frequent restarts |
10
| **Permanent** | Expected to maintain continuous connectivity | - Production servers<br/>- Core infrastructure nodes<br/>- Critical monitoring systems<br/>- Stable database servers |
11
@@ -56,16 +56,34 @@ To investigate an alert:
56
57
## Managing Offline Nodes
58
59
-To clear alerts for permanently offline nodes, run:
59
+The [Netdata CLI](/src/cli/README.md) tool has two commands for working with archived nodes.
60
+
61
+### mark-stale-nodes-ephemeral
62
+
63
+To mark a permanently offline nodes, including virtual nodes, as ephemeral:
64
65
```bash
66
netdatacli mark-stale-nodes-ephemeral <node_id | machine_guid | hostname | ALL_NODES>
67
```
68
69
+This keeps the previously collected metrics data available for querying and clears any active alerts.
70
+
71
> **Note:** Nodes will revert to permanent status if they reconnect unless explicitly configured as ephemeral in `netdata.conf`.
72
73
+### remove-stale-node
74
+
75
+To fully remove permanently offline nodes:
76
+
77
+```bash
78
+netdatacli remove-stale-node <node_id | machine_guid | hostname | ALL_NODES>
79
+```
80
+
81
+This is like the `mark-stale-nodes-ephemeral` subcommand, but it also removes the nodes so they are no longer available for querying.
82
+
83
## Cloud Integration
84
85
+In Netdata Cloud, ephemeral nodes remain visible but marked as 'stale' as long as at least one Agent reports having queryable metrics data for that node. Once all Agents report the node as offline, ephemeral nodes are automatically removed from Cloud.
86
+
87
From v2.3.0 onward, Netdata Cloud sends unreachable-node notifications **only for permanent nodes**, reducing unnecessary alerts.
88
89
## Automatically Removing Ephemeral Nodes
src/cli/README.md
+19
-14
@@ -6,19 +6,24 @@ It is located in the same directory as the `netdata` binary.
6
7
Available commands:
8
9
-| Command | Description |
10
-|------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
11
-| `help` | Display usage information and exit. |
12
-| `reload-health` | Reloads the Netdata health configuration, updating alerts based on changes made to configuration files. |
13
-| `reload-labels` | Reloads [host labels](/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts.md#custom-labels) from netdata.conf. |
14
-| `reopen-logs` | Close and reopen log files. |
15
-| `shutdown-agent` | Gracefully shut down the Netdata Agent. |
16
-| `fatal-agent` | Log the current state and forcefully halt the Netdata Agent. |
17
-| `reload-claiming-state` | Reload the Agent's claiming state from disk. |
18
-| `ping` | Checks the Agent's status. If the Agent is alive, it exits with status code 0 and prints 'pong' to standard output. Exits with status code 255 otherwise. |
19
-| `aclk-state [json]` | Return the current state of ACLK and Cloud connection. Optionally in JSON. |
20
-| `dumpconfig` | Display the current netdata.conf configuration. |
21
-| `remove-stale-node <node_id \| machine_guid \| hostname \| ALL_NODES>` | Un-registers a stale child Node, removing it from the parent Node's UI and Netdata Cloud. This is useful for ephemeral Nodes that may stop streaming and remain visible as stale. |
22
-| `version` | Display the Netdata Agent version. |
9
+| Command | Description |
10
+|---------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
11
+| `help` | Display usage information and exit. |
12
+| `reload-health` | Reloads the Netdata health configuration, updating alerts based on changes made to configuration files. |
13
+| `reload-labels` | Reloads [host labels](/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts.md#custom-labels) from netdata.conf. |
14
+| `reopen-logs` | Close and reopen log files. |
15
+| `shutdown-agent` | Gracefully shut down the Netdata Agent. |
16
+| `fatal-agent` | Log the current state and forcefully halt the Netdata Agent. |
17
+| `reload-claiming-state` | Reload the Agent's claiming state from disk. |
18
+| `ping` | Checks the Agent's status. If the Agent is alive, it exits with status code 0 and prints 'pong' to standard output. Exits with status code 255 otherwise. |
19
+| `aclk-state [json]` | Return the current state of ACLK and Cloud connection. Optionally in JSON. |
20
+| `dumpconfig` | Display the current netdata.conf configuration. |
21
+| `mark-stale-nodes-ephemeral <node_id \| machine_guid \| hostname \| ALL_NODES>` | Marks one or all disconnected nodes, including virtual nodes, as [ephemeral](/docs/nodes-ephemerality.md), while keeping their previously collected metrics data available for queries on both this Netdata Agent dashboard and Netdata Cloud.[^1][^2] |
22
+| `remove-stale-node <node_id \| machine_guid \| hostname \| ALL_NODES>` | Marks one or all disconnected nodes, including virtual nodes, as [ephemeral](/docs/nodes-ephemerality.md), and removes them so that they are no longer available for queries, from both this Netdata Agent dashboard and Netdata Cloud. This is useful to remove decommissioned nodes that have stopped streaming, but for which the Agent still holds metric data.[^1][^3] |
23
+| `version` | Display the Netdata Agent version. |
24
25
See also the Netdata daemon [command line options](/src/daemon/README.md#command-line-options).
26
+
27
+[^1]: This also clears any active alerts.
28
+[^2]: The Agent may be configured to [automatically clean up ephemeral nodes](/docs/nodes-ephemerality.md#automatic-ephemeral-nodes-cleanup) when they have been offline for longer than the configured time interval.
29
+[^3]: If a node is represented by multiple Parent Agents in a HA setup, this command must be executed on each Parent Agent.