| 1 | # Notes on Netdata Active-Active Parent Clusters |
| 2 | |
| 3 | #### **Streaming Connection Overview** |
| 4 | |
| 5 | Each Netdata child node specifies its parent nodes through the |
| 6 | `[stream].destination` configuration in `stream.conf`. While a child can list |
| 7 | multiple parents, it will connect to only one at a time. If the connection to |
| 8 | the first parent fails, the child will try the next parent in the list, |
| 9 | continuing this process until a successful connection is established. If no |
| 10 | connection can be made, the child will retry the list in order. |
| 11 | |
| 12 | Once a Netdata parent receives data from its child nodes, it can also act as a |
| 13 | child to another parent (or "grandparent") to propagate the data further up the |
| 14 | hierarchy. |
| 15 | |
| 16 | #### **Active-Active Parent Clusters Overview** |
| 17 | |
| 18 | Active-active parent clusters involve circular data propagation among parent |
| 19 | nodes. For example, parent A streams its data to parent B (its grandparent), |
| 20 | while parent B streams back to parent A, creating redundancy. This configuration |
| 21 | ensures that each parent node has the same data, allowing child nodes to connect |
| 22 | to any available parent. |
| 23 | |
| 24 | This setup can be expanded to more than two parents by configuring all parents |
| 25 | as grandparents of each other. |
| 26 | |
| 27 | --- |
| 28 | |
| 29 | ### **Data Replication** |
| 30 | |
| 31 | When a child node connects to a parent, it enters a negotiation phase to |
| 32 | announce the metrics it will stream, including their retention period. The |
| 33 | parent checks its database for missing data. If data gaps exist, the parent |
| 34 | requests replication of the missing metrics from the child before transitioning |
| 35 | to streaming fresh data. |
| 36 | |
| 37 | Replication occurs at the instance (metric group) level, meaning some metrics |
| 38 | may replicate historical data while others stream in real time. Only high- |
| 39 | resolution (`tier0`) data is replicated since higher-tier data can be derived |
| 40 | from `tier0`. Therefore, maintaining sufficient `tier0` retention on the child |
| 41 | is crucial to prevent gaps in the parent’s database. |
| 42 | |
| 43 | --- |
| 44 | |
| 45 | ### **Challenges in Active-Active Clusters** |
| 46 | |
| 47 | #### **Adding a New Parent** |
| 48 | |
| 49 | Introducing a new parent to an active-active cluster involves two major |
| 50 | challenges: |
| 51 | |
| 52 | 1. **Replicating Existing Data** |
| 53 | Since Netdata replication only propagates currently collected metrics, |
| 54 | archived data such as metrics from stopped containers or disconnected devices |
| 55 | will not be replicated. To ensure the new parent has complete historical |
| 56 | data: |
| 57 | - Copy the existing database from another parent (`/var/cache/netdata`) using |
| 58 | tools like `rsync` for dbengine files (safe for hot-copy) and `sqlite3` for |
| 59 | SQLite databases. |
| 60 | - Perform multiple copies and start the new parent promptly to minimize the |
| 61 | data gap. |
| 62 | |
| 63 | 2. **Preventing Premature Connections** |
| 64 | Child nodes should not connect to the new parent until it has completed data |
| 65 | replication. Premature connections could lead to data gaps, as the child may |
| 66 | lack the necessary historical data. |
| 67 | |
| 68 | In Netdata v2.1+, a balancing feature allows children to query parent |
| 69 | retention and prioritize connections to parents with the most recent data. |
| 70 | However, children will still connect to the first available parent, |
| 71 | potentially introducing gaps to the new parent's database. |
| 72 | **Solution:** Keep the new parent isolated from children until its |
| 73 | replication process is complete. Only then should children be configured to |
| 74 | include the new parent. |
| 75 | |
| 76 | --- |
| 77 | |
| 78 | ### **Resource Management in Clusters** |
| 79 | |
| 80 | Resource usage on parent nodes depends on three key factors: |
| 81 | |
| 82 | 1. **Ingestion Rate** |
| 83 | All parent nodes ingest all data of all children (not just their own). The |
| 84 | resource load is the same across all parents. |
| 85 | |
| 86 | 2. **Machine Learning** |
| 87 | Machine learning is CPU-intensive and affects memory usage. |
| 88 | - **Before Netdata 2.1:** Every node in a cluster independently trained |
| 89 | machine learning models for all children, increasing resource consumption |
| 90 | exponentially. |
| 91 | - **Netdata 2.1+:** The first node (child or parent) to train ML models |
| 92 | propagates the trained data to other nodes, significantly reducing resource |
| 93 | requirements. This allows flexibility: machine learning can either run at |
| 94 | the edge (child nodes) or on the first parent receiving the data. |
| 95 | |
| 96 | 3. **Re-Streaming Rate** |
| 97 | Propagating data to other parents consumes CPU and bandwidth for formatting, |
| 98 | compressing, and transmitting data. Each parent except the last grandparent |
| 99 | in the chain contributes to this workload. |
| 100 | |
| 101 | --- |
| 102 | |
| 103 | ### **Parent Balancing** |
| 104 | |
| 105 | Netdata v2.1 introduces a balancing algorithm for child nodes to optimize parent |
| 106 | connections. This feature is designed to ensure that child nodes connect to the |
| 107 | most suitable parent, balancing the load across the cluster and reducing the |
| 108 | likelihood of data gaps or resource bottlenecks. |
| 109 | |
| 110 | #### **Initial Balancing** |
| 111 | |
| 112 | Before establishing a connection, each child node queries its candidate parents |
| 113 | to retrieve their retention details. Based on this information, the child |
| 114 | evaluates the parents and prioritizes those with the most recent data. |
| 115 | |
| 116 | - **Retention Difference Threshold**: |
| 117 | Parents are considered equivalent if their retention times differ by less than |
| 118 | two minutes. In such cases, the child selects a parent randomly to avoid |
| 119 | overloading a single node. This randomness ensures an even distribution of |
| 120 | connections when all candidate parents are equally suitable. |
| 121 | |
| 122 | - **Disconnection Handling**: |
| 123 | To prevent overloading a parent during network disruptions, children |
| 124 | temporarily block a recently disconnected parent for a randomized duration |
| 125 | before attempting to reconnect. This cooldown period reduces the risk of |
| 126 | repeated disconnections and ensures smoother reconnections. |
| 127 | |
| 128 | #### **Re-balancing After Cluster Changes** |
| 129 | |
| 130 | Currently, the only way to re-balance the cluster (i.e. to break the existing |
| 131 | connections so that the children nodes will connect to both parents), is to |
| 132 | restart the children. |
| 133 | |
| 134 | --- |
| 135 | |
| 136 | ### **Summary** |
| 137 | |
| 138 | Active-active parent clusters in Netdata provide robust data redundancy and |
| 139 | flexibility. Properly configuring replication, balancing resources, and managing |
| 140 | parent-child connections ensures optimal performance and data integrity. With |
| 141 | improvements in v2.1, including machine learning propagation and connection |
| 142 | balancing, Netdata clusters are more efficient and scalable, catering to complex |
| 143 | and dynamic environments. |