@cryptotaxi247 / netdata-1
Code Commits Issues Pulls Sessions CI/CD
master
master
md 143 lines 6.17 KB
Rendered Raw
1 # Notes on Netdata Active-Active Parent Clusters
2
3 #### **Streaming Connection Overview**
4
5 Each Netdata child node specifies its parent nodes through the
6 `[stream].destination` configuration in `stream.conf`. While a child can list
7 multiple parents, it will connect to only one at a time. If the connection to
8 the first parent fails, the child will try the next parent in the list,
9 continuing this process until a successful connection is established. If no
10 connection can be made, the child will retry the list in order.
11
12 Once a Netdata parent receives data from its child nodes, it can also act as a
13 child to another parent (or "grandparent") to propagate the data further up the
14 hierarchy.
15
16 #### **Active-Active Parent Clusters Overview**
17
18 Active-active parent clusters involve circular data propagation among parent
19 nodes. For example, parent A streams its data to parent B (its grandparent),
20 while parent B streams back to parent A, creating redundancy. This configuration
21 ensures that each parent node has the same data, allowing child nodes to connect
22 to any available parent.
23
24 This setup can be expanded to more than two parents by configuring all parents
25 as grandparents of each other.
26
27 ---
28
29 ### **Data Replication**
30
31 When a child node connects to a parent, it enters a negotiation phase to
32 announce the metrics it will stream, including their retention period. The
33 parent checks its database for missing data. If data gaps exist, the parent
34 requests replication of the missing metrics from the child before transitioning
35 to streaming fresh data.
36
37 Replication occurs at the instance (metric group) level, meaning some metrics
38 may replicate historical data while others stream in real time. Only high-
39 resolution (`tier0`) data is replicated since higher-tier data can be derived
40 from `tier0`. Therefore, maintaining sufficient `tier0` retention on the child
41 is crucial to prevent gaps in the parent’s database.
42
43 ---
44
45 ### **Challenges in Active-Active Clusters**
46
47 #### **Adding a New Parent**
48
49 Introducing a new parent to an active-active cluster involves two major
50 challenges:
51
52 1. **Replicating Existing Data**
53 Since Netdata replication only propagates currently collected metrics,
54 archived data such as metrics from stopped containers or disconnected devices
55 will not be replicated. To ensure the new parent has complete historical
56 data:
57 - Copy the existing database from another parent (`/var/cache/netdata`) using
58 tools like `rsync` for dbengine files (safe for hot-copy) and `sqlite3` for
59 SQLite databases.
60 - Perform multiple copies and start the new parent promptly to minimize the
61 data gap.
62
63 2. **Preventing Premature Connections**
64 Child nodes should not connect to the new parent until it has completed data
65 replication. Premature connections could lead to data gaps, as the child may
66 lack the necessary historical data.
67
68 In Netdata v2.1+, a balancing feature allows children to query parent
69 retention and prioritize connections to parents with the most recent data.
70 However, children will still connect to the first available parent,
71 potentially introducing gaps to the new parent's database.
72 **Solution:** Keep the new parent isolated from children until its
73 replication process is complete. Only then should children be configured to
74 include the new parent.
75
76 ---
77
78 ### **Resource Management in Clusters**
79
80 Resource usage on parent nodes depends on three key factors:
81
82 1. **Ingestion Rate**
83 All parent nodes ingest all data of all children (not just their own). The
84 resource load is the same across all parents.
85
86 2. **Machine Learning**
87 Machine learning is CPU-intensive and affects memory usage.
88 - **Before Netdata 2.1:** Every node in a cluster independently trained
89 machine learning models for all children, increasing resource consumption
90 exponentially.
91 - **Netdata 2.1+:** The first node (child or parent) to train ML models
92 propagates the trained data to other nodes, significantly reducing resource
93 requirements. This allows flexibility: machine learning can either run at
94 the edge (child nodes) or on the first parent receiving the data.
95
96 3. **Re-Streaming Rate**
97 Propagating data to other parents consumes CPU and bandwidth for formatting,
98 compressing, and transmitting data. Each parent except the last grandparent
99 in the chain contributes to this workload.
100
101 ---
102
103 ### **Parent Balancing**
104
105 Netdata v2.1 introduces a balancing algorithm for child nodes to optimize parent
106 connections. This feature is designed to ensure that child nodes connect to the
107 most suitable parent, balancing the load across the cluster and reducing the
108 likelihood of data gaps or resource bottlenecks.
109
110 #### **Initial Balancing**
111
112 Before establishing a connection, each child node queries its candidate parents
113 to retrieve their retention details. Based on this information, the child
114 evaluates the parents and prioritizes those with the most recent data.
115
116 - **Retention Difference Threshold**:
117 Parents are considered equivalent if their retention times differ by less than
118 two minutes. In such cases, the child selects a parent randomly to avoid
119 overloading a single node. This randomness ensures an even distribution of
120 connections when all candidate parents are equally suitable.
121
122 - **Disconnection Handling**:
123 To prevent overloading a parent during network disruptions, children
124 temporarily block a recently disconnected parent for a randomized duration
125 before attempting to reconnect. This cooldown period reduces the risk of
126 repeated disconnections and ensures smoother reconnections.
127
128 #### **Re-balancing After Cluster Changes**
129
130 Currently, the only way to re-balance the cluster (i.e. to break the existing
131 connections so that the children nodes will connect to both parents), is to
132 restart the children.
133
134 ---
135
136 ### **Summary**
137
138 Active-active parent clusters in Netdata provide robust data redundancy and
139 flexibility. Properly configuring replication, balancing resources, and managing
140 parent-child connections ensures optimal performance and data integrity. With
141 improvements in v2.1, including machine learning propagation and connection
142 balancing, Netdata clusters are more efficient and scalable, catering to complex
143 and dynamic environments.