@cryptotaxi247 / netdata-1 / commits / f38fbdc4b

Improve metrics centralization points documentation (#20412)

Co-authored-by: ilyam8 <ilya@netdata.cloud>

Kanela committed Jun 5, 2025 at 14:15 UTC f38fbdc4b69b952b010940a2309e6d34889b9a20
5 files changed +385 -100
docs/observability-centralization-points/metrics-centralization-points/README.md
+47 -15
@@ -1,28 +1,60 @@
1 -# Metrics Centralization Points (Netdata Parents)
1 +# Configure Netdata Parents to centralize metrics from multiple production systems with streaming, replication, and high availability configurations.
2 +
3 +:::
4 +
5 +## How Metrics Centralization Works
6
7 ```mermaid
8 flowchart BT
5 - C1["Netdata Child 1"]
6 - C2["Netdata Child 2"]
7 - C3["Netdata Child N"]
8 - P1["Netdata Parent 1"]
9 + C1("**Netdata Child 1**<br/><br/>Production System")
10 + C2("**Netdata Child 2**<br/><br/>Production System")
11 + C3("**Netdata Child N**<br/><br/>Production System")
12 + P1("**Netdata Parent 1**<br/><br/>Centralization Point")
13 C1 -->|stream| P1
14 C2 -->|stream| P1
15 C3 -->|stream| P1
16 +
17 + %% Style definitions
18 + classDef child fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:16px
19 + classDef parent fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:16px
20 +
21 + %% Apply styles
22 + class C1,C2,C3 child
23 + class P1 parent
24 ```
25
14 -- **Netdata Streaming and Replication**:
15 - - Copies **recent past samples** (replication) and **real-time new samples** (streaming) from production systems (**Netdata Children**) to **metrics centralization points** (**Netdata Parents**).
16 - - **Netdata Parents** store the database for these metrics based on **retention settings**.
26 +## Key Features
27 +
28 +:::important
29 +
30 +**Netdata Streaming and Replication**
31 +
32 +Copies **recent past samples** (replication) and **real-time new samples** (streaming) from production systems (**Netdata Children**) to **metrics centralization points** (**Netdata Parents**). **Netdata Parents** store the database for these metrics based on **retention settings**.
33 +
34 +:::
35 +
36 +### How Your Systems Connect
37 +
38 +<details>
39 +<summary><strong>Netdata Child Behavior</strong></summary><br/>
40 +
41 +- Each **Netdata Child** can stream to **only one** Netdata Parent at a time.
42 +- Multiple **Netdata Parents** can be configured for **high availability**, but only the **first working one** will be used.
43 +
44 +<br/>
45 +</details>
46 +
47 +<details>
48 +<summary><strong>Netdata Parent Capabilities</strong></summary><br/>
49 +
50 +- Receives metric samples **from multiple Netdata Children**.
51 +- Can **re-stream** received metrics to another **Netdata Parent**, forming an **infinite hierarchy** of Parents.
52 +- Supports **Netdata Parents Clusters** for **high availability**.
53
18 -- **Netdata Child Behavior**:
19 - - Each **Netdata Child** can stream to **only one** Netdata Parent at a time.
20 - - Multiple **Netdata Parents** can be configured for **high availability**, but only the **first working one** will be used.
54 +<br/>
55 +</details>
56
22 -- **Netdata Parent Capabilities**:
23 - - Receives metric samples **from multiple Netdata Children**.
24 - - Can **re-stream** received metrics to another **Netdata Parent**, forming an **infinite hierarchy** of Parents.
25 - - Supports **Netdata Parents Clusters** for **high availability**.
57 +## Feature Comparison
58
59 | Feature | Netdata Child (Production System) | Netdata Parent (Centralization Point) |
60 |----------------------------|---------------------------------------------------|-------------------------------------------------------|
docs/observability-centralization-points/metrics-centralization-points/clustering-and-high-availability-of-netdata-parents.md
+89 -26
@@ -1,50 +1,113 @@
1 # Clustering and High Availability of Netdata Parents
2
3 +:::tip
4 +
5 +**What You'll Learn**
6 +
7 +How to set up clustered Netdata Parents for high availability, including failover mechanisms, maintenance procedures, and parent duplication strategies.
8 +
9 +:::
10 +
11 +## How Clustering Works
12 +
13 ```mermaid
14 flowchart BT
5 - C1["Netdata Child 1"]
6 - C2["Netdata Child 2"]
7 - C3["Netdata Child N"]
8 - P1["Netdata Parent 1"]
9 - P2["Netdata Parent 2"]
15 + C1("**Netdata Child 1**<br/><br/>Production System")
16 + C2("**Netdata Child 2**<br/><br/>Production System")
17 + C3("**Netdata Child N**<br/><br/>Production System")
18 + P1("**Netdata Parent 1**<br/><br/>Cluster Node")
19 + P2("**Netdata Parent 2**<br/><br/>Cluster Node")
20 C1 & C2 & C3 -->|stream| P1
21 P1 -->|stream| P2
12 - C1 & C2 & C3 .->|failover| P2
13 - P2 .->|failover| P1
22 + C1 & C2 & C3 -.->|failover| P2
23 + P2 -.->|failover| P1
24 +
25 + %% Style definitions
26 + classDef child fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:16px
27 + classDef parent fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:16px
28 +
29 + %% Apply styles
30 + class C1,C2,C3 child
31 + class P1,P2 parent
32 ```
33
34 Netdata supports building Parent clusters of 2+ nodes. Clustering and high availability work like this:
35
18 -1. All Netdata Children are configured to stream to all Netdata Parents. The first one found working will be used by each Netdata Child, and the others will be automatically used if and when this connection is interrupted.
19 -2. The Netdata Parents are configured to stream to all other Netdata Parents. For each of them, the first-found working will be used and the others will be automatically used if and when this connection is interrupted.
36 +## Cluster Architecture
37 +
38 +:::important
39 +
40 +**How Failover Works**
41 +
42 +1. **Configure your Children to connect to all Parents**.
43 + Each Child will use the first working Parent it finds, and automatically switch to backup Parents if the connection fails.
44 +
45 +2. **Configure your Parents to sync with each other**.
46 + Each Parent connects to other Parents in the cluster, using the first working connection and failing over to others when needed.
47 +
48 +:::
49 +
50 +:::note
51 +
52 +### Data Distribution
53 +
54 +**When you set up a cluster, every Parent receives all metrics from every Child**, no matter which Parent the Child is connected to. Parents also share their data with each other, so you never lose visibility into your infrastructure.
55 +
56 +**When a Parent goes down, your Children automatically reconnect to another available Parent**. That Parent then shares the metrics with all other Parents in your cluster, keeping everything in sync.
57 +
58 +### Netdata Cloud Visibility
59 +
60 +Your cluster ensures continuous visibility in Netdata Cloud. All Parents register your Children with the Cloud, so as long as at least one Parent is online, you'll see all your systems in Netdata Cloud.
61 +
62 +### Smart Data Retention
63 +
64 +Children only need to keep data long enough to handle Parent switchovers. When a Child connects to a Parent, they work together to fill in any missing data gaps, ensuring complete historical visibility.
65 +
66 +:::
67 +
68 +## Maintenance Procedures
69 +
70 +<details>
71 +<summary><strong>Bringing a Parent back online after maintenance</strong></summary><br/>
72 +
73 +When you bring a Parent back online after extended maintenance, you need to be careful about data gaps. Check the [replication limitations](/docs/observability-centralization-points/metrics-centralization-points/replication-of-past-samples.md#understanding-limitations) to understand the constraints.
74 +
75 +If your Children don't have enough historical data to fill gaps on the restored Parent, temporarily block Children from connecting to it until it syncs with other Parents in your cluster.
76
21 -All the Netdata Parents in such a cluster will receive all the metrics of all Netdata Children connected to any of them. They will also receive the metrics all the other Netdata Parents have.
77 +### Temporarily Blocking Children
78
23 -If a Parent node fails, its Child nodes automatically connect to another available Parent node, which then re-streams metrics to all other Parent nodes.
79 +You have two options to block Children while allowing Parent-to-Parent sync:
80
25 -Netdata Cloud will receive registrations for all Netdata Children from all the Netdata Parents. As long as at least one of the Netdata Parents is connected to Netdata Cloud, all the Netdata Children will be available on Netdata Cloud.
81 +1. **Use firewall rules**.
82 + Block port 19999 access from your Children's IP ranges to the restored Parent
83 +2. **Use separate API keys**.
84 + Configure different streaming keys in `stream.conf` for Children vs Parents, then disable the Children's key until sync completes
85
27 -Netdata Children need to maintain retention only for the time required to switch Netdata Parents. When Netdata Children connect to a Netdata Parent, they negotiate the available retention and any missing data on the Netdata Parent are replicated from the Netdata Children.
86 +<br/>
87 +</details>
88
29 -## Restoring a Netdata Parent after maintenance
89 +<details>
90 +<summary><strong>Creating a new Parent from an existing one</strong></summary><br/>
91
31 -Given the [replication limitations](/docs/observability-centralization-points/metrics-centralization-points/replication-of-past-samples.md#replication-limitations), special care is needed when restoring a Netdata Parent after some long maintenance work on it.
92 +The fastest way to add a new Parent to your cluster is copying data from an existing Parent using `rsync` to transfer `/var/cache/netdata`.
93
33 -If the Netdata Children do not have enough retention to replicate the missing data on this Netdata Parent, it is preferable to block access to this Netdata Parent from the Netdata Children, until it replicates the missing data from the other Netdata Parents.
94 +:::warning
95
35 -To block access from Netdata Children, and still allow access from other Netdata Parent siblings:
96 +**Configure Before Starting**
97
37 -1. Use `iptables` to block access to port 19999 from Netdata Children to the restored Netdata Parent, or
38 -2. Use separate streaming API keys (in `stream.conf`) for Netdata Children and Netdata Parents, and disable the API key used by Netdata Children, until the restored Netdata Parent has been synchronized.
98 +Don't start your new Parent with default settings because it might delete the copied data to apply default size limits. Set your retention configuration first, then start the service.
99
40 -## Duplicating a Parent
100 +:::
101
42 -The easiest way is to `rsync` the directory `/var/cache/netdata` from the existing Netdata Parent to the new Netdata Parent.
102 +### Setting Up Retention
103
44 -> Important: Starting the new Netdata Parent with default settings, may delete the new files in `/var/cache/netdata` to apply the default disk size constraints. Therefore, it is important to set the right retention settings in the new Netdata Parent before starting it up with the copied files.
104 +Before starting your new Parent, configure these settings in `netdata.conf` to match your existing Parent:
105
46 -To configure retention at the new Netdata Parent, set in `netdata.conf` the following to at least the values the old Netdata Parent has:
106 +| Section | Setting | Description | Default |
107 +|---------|----------------------------------|-----------------------------------------------|---------|
108 +| `[db]` | `dbengine tier 0 retention size` | Maximum disk space for high-resolution data | 1GiB |
109 +| `[db]` | `dbengine tier 1 retention size` | Maximum disk space for medium-resolution data | 1GiB |
110 +| `[db]` | `dbengine tier 2 retention size` | Maximum disk space for low-resolution data | 1GiB |
111
48 -- `[db].dbengine tier 0 retention size`, this is the max disk size for `tier0`. The default is 1GiB.
49 -- `[db].dbengine tier 1 retention size`, this is the max disk space for `tier1`. The default is 1GiB.
50 -- `[db].dbengine tier 2 retention size`, this is the max disk space for `tier2`. The default is 1GiB.
112 +<br/>
113 +</details>
docs/observability-centralization-points/metrics-centralization-points/configuration.md
+65 -12
@@ -1,15 +1,31 @@
1 # Configuring Metrics Centralization Points
2
3 +:::tip
4 +
5 +**What You'll Learn**
6 +
7 +How to configure streaming between Netdata Children and Parents using stream.conf, including basic setup, TLS/SSL encryption, and troubleshooting connection issues.
8 +
9 +:::
10 +
11 Metrics streaming configuration for both Netdata Children and Parents is done via `stream.conf`.
12
13 +:::important
14 +
15 +**Configuration File Security**
16 +
17 `netdata.conf` and `stream.conf` have the same `ini` format, but `netdata.conf` is considered a non-sensitive file, while `stream.conf` contains API keys, IPs and other sensitive information that enable communication between Netdata Agents.
18
19 +:::
20 +
21 +## Understanding stream.conf Structure
22 +
23 `stream.conf` has two main sections:
24
25 - The `[stream]` section includes options for the **sending Netdata** (i.e., Netdata Children, or Netdata Parents that stream to Grand Parents, or to other sibling Netdata Parents in a cluster).
26 - The rest includes multiple sections that define API keys for the **receiving Netdata** (i.e., Netdata Parents).
27
12 -## Edit `stream.conf`
28 +## Edit stream.conf
29
30 To edit `stream.conf`, run this on your terminal:
31
@@ -20,7 +36,10 @@ sudo ./edit-config stream.conf
36
37 Your editor will open, with defaults and commented `stream.conf` options.
38
23 -## Configuring a Netdata Parent
39 +## Configuration Steps
40 +
41 +<details>
42 +<summary><strong>Configuring a Netdata Parent</strong></summary><br/>
43
44 To enable the reception of metrics from Netdata Children, generate a random API key with this command:
45
@@ -38,7 +57,11 @@ Then, copy the UUID generated, [edit `stream.conf`](#edit-streamconf), find the
57
58 Save the file and restart Netdata.
59
41 -## Configuring Netdata Children
60 +<br/>
61 +</details>
62 +
63 +<details>
64 +<summary><strong>Configuring Netdata Children</strong></summary><br/>
65
66 To enable streaming metrics to a Netdata Parent, [edit `stream.conf`](#edit-streamconf), and at the `[stream]` section at the top, set:
67
@@ -54,15 +77,25 @@ To enable streaming metrics to a Netdata Parent, [edit `stream.conf`](#edit-stre
77
78 Save the file and restart Netdata.
79
57 -## Enable TLS/SSL Communication
80 +<br/>
81 +</details>
82 +
83 +<details>
84 +<summary><strong>Enable TLS/SSL Communication</strong></summary><br/>
85
86 While encrypting the connection between your parent and child nodes is recommended for security, it's not required to get started.
87
88 This example uses self-signed certificates.
89
63 -> **Note**
64 -> This section assumes you have read the documentation on [how to edit the Netdata configuration files](/docs/netdata-agent/configuration/README.md).
65 -<!-- here we need a link to the section that will contain the restarting instructions -->
90 +:::note
91 +
92 +**Prerequisites**
93 +
94 +This section assumes you have read the documentation on [how to edit the Netdata configuration files](/docs/netdata-agent/configuration/README.md).
95 +
96 +:::
97 +
98 +### SSL Setup Process
99
100 1. **Parent node**
101 To generate an SSL key and certificate using `openssl`, take a look at the related section around [Securing Netdata Agents](/src/web/server/README.md#enable-httpstls-support) in our Documentation.
@@ -78,26 +111,46 @@ This example uses self-signed certificates.
111 api key = 11111111-2222-3333-4444-555555555555
112 ```
113
81 -3. Restart the Netdata Agent on both the parent and child nodes to stream encrypted metrics using TLS/SSL.
114 +3. **Restart both nodes**
115 + Restart the Netdata Agent on both the parent and child nodes to stream encrypted metrics using TLS/SSL.
116 +
117 +<br/>
118 +</details>
119
120 ## Troubleshooting Streaming Connections
121
122 You can find any issues related to streaming at Netdata logs.
123
87 -### From the UI
124 +<details>
125 +<summary><strong>From the UI</strong></summary><br/>
126
127 Netdata logs to systemd-journald by default, and its logs are available at the `Logs` tab of the UI. At the `MESSAGE_ID` field look for `Netdata connection from child` and `Netdata connection to parent`.
128
91 -### From the terminal
129 +<br/>
130 +</details>
131 +
132 +<details>
133 +<summary><strong>From the terminal</strong></summary><br/>
134
93 -On the Parents:
135 +**On the Parents:**
136
137 ```bash
138 journalctl -r --namespace=netdata MESSAGE_ID=ed4cdb8f1beb4ad3b57cb3cae2d162fa
139 ```
140
99 -On the Children:
141 +**On the Children:**
142
143 ```bash
144 journalctl -r --namespace=netdata MESSAGE_ID=6e2e3839067648968b646045dbf28d66
145 ```
146 +
147 +<br/>
148 +</details>
149 +
150 +## Next Steps
151 +
152 +### Sizing Netdata Parents
153 +
154 +After configuring your metrics centralization setup, you'll want to properly size your Netdata Parents to handle the expected load from your Children nodes.
155 +
156 +For detailed information on resource requirements and optimization strategies, see [Resource Utilization](https://learn.netdata.cloud/docs/netdata-agent/resource-utilization/).
docs/observability-centralization-points/metrics-centralization-points/faq.md
+70 -14
@@ -1,51 +1,96 @@
1 # FAQ on Metrics Centralization Points
2
3 -## How much can a Netdata Parent node scale?
3 +:::tip
4
5 -Netdata Parents generally scale well. According [to our tests](https://blog.netdata.cloud/netdata-vs-prometheus-performance-analysis/) Netdata Parents scale better than Prometheus for the same workload: -35% CPU utilization, -49% Memory Consumption, -12% Network Bandwidth, -98% Disk I/O, -75% Disk footprint.
5 +**What You'll Learn**
6 +
7 +Answers to common questions about scaling Netdata Parents, clustering, retention, encryption, and troubleshooting centralized monitoring setups.
8 +
9 +:::
10 +
11 +<details>
12 +<summary><strong>How much can a Netdata Parent node scale?</strong></summary><br/>
13 +
14 +Netdata Parents generally scale well. According [to our tests](https://blog.netdata.cloud/netdata-vs-prometheus-performance-analysis/), Netdata Parents scale better than Prometheus for the same workload: -35% CPU utilization, -49% Memory Consumption, -12% Network Bandwidth, -98% Disk I/O, -75% Disk footprint.
15
16 For more information, Check [Sizing Netdata Parents](/docs/observability-centralization-points/metrics-centralization-points/sizing-netdata-parents.md).
17
9 -## If I set up a parents cluster, will I be able to have more Child nodes stream to them?
18 +<br/>
19 +</details>
20 +
21 +<details>
22 +<summary><strong>If I set up a Parent cluster, will I be able to have more Child nodes stream to them?</strong></summary><br/>
23
24 No. When you set up an active-active cluster, even if child nodes connect randomly to one or the other, all the parent nodes receive all the metrics of all the child nodes. So, all of them do all the work.
25
13 -## How much retention do the child nodes need?
26 +<br/>
27 +</details>
28 +
29 +<details>
30 +<summary><strong>How much retention do the child nodes need?</strong></summary><br/>
31
32 Child nodes need to have only the retention required to connect to another Parent if one fails or stops for maintenance.
33
34 - If you have a cluster of parents, 5 to 10 minutes in `alloc` mode is usually enough.
18 -- If you have only one parent, it would be better to run the child nodes with `dbengine` so that they will have enough retention to back-fill the parent node if it stops for maintenance.
35 +- If you have only one parent, it would be better to run the child nodes with `dbengine` so that they will have enough retention to backfill the parent node if it stops for maintenance.
36
20 -## Does streaming between child nodes and parents support encryption?
37 +<br/>
38 +</details>
39 +
40 +<details>
41 +<summary><strong>Does streaming between child nodes and parents support encryption?</strong></summary><br/>
42
43 Yes. You can configure your parent nodes to enable TLS at their web server and configure the child nodes to connect with TLS to it. The streaming connection is also compressed, on top of TLS.
44
24 -## Can I have an HTTP proxy between parent and child nodes?
45 +<br/>
46 +</details>
47 +
48 +<details>
49 +<summary><strong>Can I have an HTTP proxy between parent and child nodes?</strong></summary><br/>
50
51 No. The streaming protocol works on the same port as the internal web server of Netdata Agents, but the protocol is not HTTP-friendly and cannot be understood by HTTP proxy servers.
52
28 -## Should I load balance multiple parents with a TCP load balancer?
53 +<br/>
54 +</details>
55 +
56 +<details>
57 +<summary><strong>Should I load balance multiple parents with a TCP load balancer?</strong></summary><br/>
58
59 Although this can be done and for streaming between child and parent nodes it could work, we recommend not doing it. It can lead to several kinds of problems.
60
61 It is better to configure all the parent nodes directly in the child nodes `stream.conf`. The child nodes will do everything in their power to find a parent node to connect, and they will never give up.
62
34 -## When I have multiple parents for the same children, will I receive alert notifications from all of them?
63 +<br/>
64 +</details>
65 +
66 +<details>
67 +<summary><strong>When I have multiple parents for the same children, will I receive alert notifications from all of them?</strong></summary><br/>
68
69 If all parents are configured to run health checks and trigger alerts, yes.
70
71 We recommend using Netdata Cloud to avoid receiving duplicate alert notifications. Netdata Cloud deduplicates alert notifications so that you will receive them only once.
72
40 -## When I have only Parents connected to Netdata Cloud, will I be able to use the Functions feature on my child nodes?
73 +<br/>
74 +</details>
75 +
76 +<details>
77 +<summary><strong>When I have only Parents connected to Netdata Cloud, will I be able to use the Functions feature on my child nodes?</strong></summary><br/>
78
79 Yes. Function requests will be received by the Parents and forwarded to the Child via their streaming connection. Function requests are propagated between parents, so this will work even if multiple levels of Netdata Parents are involved.
80
44 -## If I have a cluster of parents and get one out for maintenance for a few hours, will it have missing data when it returns online?
81 +<br/>
82 +</details>
83 +
84 +<details>
85 +<summary><strong>If I have a cluster of parents and get one out for maintenance for a few hours, will it have missing data when it returns online?</strong></summary><br/>
86
87 Check [Restoring a Netdata Parent after maintenance](/docs/observability-centralization-points/metrics-centralization-points/clustering-and-high-availability-of-netdata-parents.md).
88
48 -## I have a cluster of parents. Which one is used by Netdata Cloud?
89 +<br/>
90 +</details>
91 +
92 +<details>
93 +<summary><strong>I have a cluster of parents. Which one is used by Netdata Cloud?</strong></summary><br/>
94
95 When there are multiple data sources for the same node, Netdata Cloud follows this strategy:
96
@@ -53,11 +98,19 @@ When there are multiple data sources for the same node, Netdata Cloud follows th
98 2. For time-series queries, when multiple Netdata Agents have the retention required to answer the query, Netdata Cloud prefers the one that is further away from production systems.
99 3. For Functions, Netdata Cloud prefers Netdata Agents that are closer to the production systems.
100
56 -## Is there a way to balance child nodes to the parent nodes of a cluster?
101 +<br/>
102 +</details>
103 +
104 +<details>
105 +<summary><strong>Is there a way to balance child nodes to the parent nodes of a cluster?</strong></summary><br/>
106
107 Yes. When configuring the Parents at the Children `stream.conf`, configure them in different order. Children get connected to the first Parent they find available, so if the order given to them is different, they will spread the connections to the Parents available.
108
60 -## Is there a way to get notified when a child gets disconnected?
109 +<br/>
110 +</details>
111 +
112 +<details>
113 +<summary><strong>Is there a way to get notified when a child gets disconnected?</strong></summary><br/>
114
115 It depends on the ephemerality setting of each Netdata Child.
116
@@ -76,3 +129,6 @@ netdatacli remove-stale-node <node_id | machine_guid | hostname | ALL_NODES>
129 ```
130
131 When using Netdata Cloud (via a parent or directly), and a permanent node gets disconnected, Netdata Cloud sends node disconnection notifications.
132 +
133 +<br/>
134 +</details>
docs/observability-centralization-points/metrics-centralization-points/replication-of-past-samples.md
+114 -33
@@ -1,60 +1,141 @@
1 # Replication of Past Samples
2
3 -Replication is triggered when a Netdata Child connects to a Netdata Parent. It replicates the latest samples of collected metrics a Netdata Parent may be missing. The goal of replication is to back-fill samples that were collected between disconnects and reconnects, so that the Netdata Parent does not have gaps on the charts for the time Netdata Children were disconnected.
3 +:::tip
4
5 -The same replication mechanism is used between Netdata Parents (the sending Netdata is treated as a Child and the receiving Netdata as a Parent).
5 +**What You'll Learn**
6
7 -## Replication Limitations
7 +How Netdata automatically fills data gaps when Children reconnect to Parents, including replication limitations, configuration options, and monitoring progress.
8
9 -The current implementation is optimized to replicate small durations and have minimal impact during reconnecting. As a result, it has the following limitations:
9 +:::
10
11 -1. Replication can only append samples to metrics. Only missing samples at the end of each time-series are replicated.
11 +When your Netdata Child reconnects to a Parent after being offline, replication automatically kicks in. Your Parent gets the metric samples it missed while the Child was disconnected, ensuring you don't see gaps in your charts.
12
13 -2. Only `tier0` samples are replicated. Samples of higher tiers in Netdata are derived from `tier0` samples, and therefore there is no mechanism for ingesting them directly. This means that the maximum retention that can be replicated across Netdata is limited by the samples available in `tier0` of the sending Netdata.
13 +:::note
14
15 -3. Only samples of metrics that are currently being collected are replicated. Archived metrics (or even archived nodes) will be replicated when and if they are collected again. Netdata archives metrics 1 hour after they stop being collected, so Netdata Parents may miss data only if Netdata Children are disconnected for more than an hour from their Parents.
15 +This same process works between Parents too. When Parents sync with each other, one acts as the sender and the other as the receiver.
16
17 -When multiple Netdata Parents are available, the replication happens in sequence, like in the following diagram.
17 +:::
18 +
19 +## How Replication Works
20 +
21 +When multiple Netdata Parents are available, the replication happens in sequence, like in the following diagram:
22
23 ```mermaid
24 +%%{init: {
25 + "theme": "base",
26 + "themeVariables": {
27 + "actorBkg": "#ffeb3b",
28 + "actorBorder": "#000000",
29 + "actorTextColor": "#000000",
30 + "actorLineColor": "#000000",
31 + "signalColor": "#000000",
32 + "signalTextColor": "#000000",
33 + "labelBoxBkgColor": "#f9f9f9",
34 + "labelBoxBorderColor": "#000000",
35 + "labelTextColor": "#000000",
36 + "loopTextColor": "#000000",
37 + "activationBkgColor": "#4caf50",
38 + "activationBorderColor": "#000000",
39 + "fontFamily": "Arial, sans-serif",
40 + "fontSize": "16px"
41 + }
42 +}}%%
43 sequenceDiagram
21 - Child-->>Parent1: Connect
22 - Parent1-->>Child: OK
23 - Parent1-->>Parent2: Connect
24 - Parent2-->>Parent1: OK
25 - Child-->>Parent1: Metric M1 with retention up to Now
26 - Parent1-->>Child: M1 stopped at -60sec, replicate up to Now
27 - Child-->>Parent1: replicate M1 samples -60sec to Now
28 - Child-->>Parent1: streaming M1
29 - Parent1-->>Parent2: Metric M1 with retention up to Now
30 - Parent2-->>Parent1: M1 stopped at -63sec, replicate up to Now
31 - Parent1-->>Parent2: replicate M1 samples -63sec to Now
32 - Parent1-->>Parent2: streaming M1
44 + participant C as Child
45 + participant P1 as Parent 1
46 + participant P2 as Parent 2
47 +
48 + C->>P1: Connect
49 + P1->>C: OK
50 + P1->>P2: Connect
51 + P2->>P1: OK
52 + C->>P1: Metric M1 with retention up to Now
53 + P1->>C: M1 stopped at -60sec, replicate up to Now
54 + C->>P1: replicate M1 samples -60sec to Now
55 + C->>P1: streaming M1
56 + P1->>P2: Metric M1 with retention up to Now
57 + P2->>P1: M1 stopped at -63sec, replicate up to Now
58 + P1->>P2: replicate M1 samples -63sec to Now
59 + P1->>P2: streaming M1
60 ```
61
62 +### Replication Process
63 +
64 As shown in the diagram:
65
37 -1. All connections are established immediately after a Netdata child connects to any of the Netdata Parents.
38 -2. Each pair of connections (Child->Parent1, Parent1->Parent2) complete replication on the receiving side and then initiate replication on the sending side.
39 -3. Replication pushes data up to Now, and the sending side immediately enters streaming mode, without leaving any gaps on the samples of the receiving side.
40 -4. On every pair of connections, replication negotiates the retention of the receiving party to back-fill as much data as necessary.
66 +1. **Connections establish immediately** after a Netdata child connects to any of the Netdata Parents.
67 +2. **Each connection pair completes replication** (Child→Parent1, Parent1→Parent2) on the receiving side and then initiates replication on the sending side.
68 +3. **Replication fills gaps up to now**, and the sending side immediately enters streaming mode, without leaving any gaps on the samples of the receiving side.
69 +4. **Each connection negotiates retention** to back-fill as much data as necessary.
70 +
71 +## Understanding Limitations
72 +
73 +:::important
74 +
75 +**Key Replication Constraints**
76 +
77 +The current implementation is optimized to replicate small durations and have minimal impact during reconnecting. Understanding these limitations helps you plan your monitoring setup effectively.
78 +
79 +:::
80 +
81 +<details>
82 +<summary><strong>What Can and Can’t Be Replicated</strong></summary><br/>
83 +
84 +1. **Append-only replication**.
85 + Replication can only append samples to metrics. Only missing samples at the end of each time-series are replicated.
86 +
87 +2. **Tier0 samples only**.
88 + Only `tier0` samples are replicated. Samples of higher tiers in Netdata are derived from `tier0` samples, and therefore there is no mechanism for ingesting them directly. This means that the maximum retention that can be replicated across Netdata is limited by the samples available in `tier0` of the sending Netdata.
89 +
90 +3. **Active metrics only**.
91 + Only samples of metrics that are currently being collected are replicated. Archived metrics (or even archived nodes) will be replicated when and if they are collected again.
92
42 -## Configuration options for Replication
93 +:::note
94
44 -The following `netdata.conf` configuration parameters affect replication.
95 +Netdata archives metrics 1 hour after they stop being collected, so Netdata Parents may miss data only if Netdata Children are disconnected for more than an hour from their Parents.
96
46 -On the receiving side (Netdata Parent):
97 +:::
98
48 -- `[db].replication period` limits the maximum time to be replicated. The default is 1 day. Keep in mind that replication is also limited by the `tier0` retention the sending side has.
99 +<br/>
100 +</details>
101
50 -On the sending side (Netdata Children, or Netdata Parent when parents are clustered):
102 +## Configuration Options
103
52 -- `[db].replication threads` controls how many concurrent threads will be replicating metrics. The default is 1. Usually the performance is about two million samples per second per thread, so increasing this number may allow replication to progress faster between Netdata Parents.
104 +Configure these options in `netdata.conf` on the respective systems.
105
54 -- `[db].cleanup obsolete charts after` controls for how much time after metrics stop being collected will not be available for replication. The default is 1 hour (3600 seconds). If you plan to have scheduled maintenance on Netdata Parents of more than 1 hour, we recommend increasing this setting. Keep in mind, however, that increasing this duration in highly ephemeral environments can have an impact on RAM utilization, since metrics will be considered as collected for longer durations.
106 +<details>
107 +<summary><strong>Receiving Side Configuration (Netdata Parent)</strong></summary><br/>
108 +
109 +| Setting | Description | Default |
110 +|---------------------------|----------------------------------------------------------------------------------------------------------------------------------|---------|
111 +| `[db].replication period` | Sets the maximum time window for replication. Remember, you're also limited by how much tier0 data your Child systems have kept. | 1 day |
112 +
113 +</details>
114 +
115 +<details>
116 +<summary><strong>Sending Side Configuration (Netdata Children or clustered Parents)</strong></summary><br/>
117 +
118 +| Setting | Description | Default |
119 +|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------|
120 +| `[db].replication threads` | Controls how many parallel threads handle replication. Each thread can handle about two million samples per second, so more threads can speed up replication between Parents with lots of data. | 1 thread |
121 +| `[db].cleanup obsolete charts after` | Controls how long metrics remain available for replication after collection stops. If you expect Parent maintenance to last longer than 1 hour, increase this setting. Just be aware that in dynamic environments with lots of short-lived metrics, this can increase RAM usage since metrics stay "active" longer. | 1 hour<br/>(3600 seconds) |
122 +
123 +</details>
124
125 ## Monitoring Replication Progress
126
58 -Inbound and outbound replication progress is reported at the dashboard using the Netdata Function `Streaming`, under the `Top` tab.
127 +:::note
128 +
129 +**Where to Check Replication Status**
130 +
131 +You can monitor how replication is progressing through both your dashboard and API endpoints to make sure your data synchronization is working correctly.
132 +
133 +:::
134 +
135 +### Dashboard Monitoring
136 +
137 +Check your replication progress right in your dashboard using the Netdata Function `Netdata-streaming`, under the `Top` tab.
138 +
139 +### API Monitoring
140
60 -The same information is exposed via the API endpoint `http://agent-ip:19999/api/v2/node_instances` of both Netdata Parents and Children.
141 +You can also get the same information via the API endpoint `http://agent-ip:19999/api/v2/node_instances` on both your Parents and Children.