@cryptotaxi247 / netdata-1
Code Commits Issues Pulls Sessions CI/CD
master
master
md 113 lines 4.52 KB
Rendered Raw
1 # Clustering and High Availability of Netdata Parents
2
3 :::tip
4
5 **What You'll Learn**
6
7 How to set up clustered Netdata Parents for high availability, including failover mechanisms, maintenance procedures, and parent duplication strategies.
8
9 :::
10
11 ## How Clustering Works
12
13 ```mermaid
14 flowchart BT
15 C1("**Netdata Child 1**<br/><br/>Production System")
16 C2("**Netdata Child 2**<br/><br/>Production System")
17 C3("**Netdata Child N**<br/><br/>Production System")
18 P1("**Netdata Parent 1**<br/><br/>Cluster Node")
19 P2("**Netdata Parent 2**<br/><br/>Cluster Node")
20 C1 & C2 & C3 -->|stream| P1
21 P1 -->|stream| P2
22 C1 & C2 & C3 -.->|failover| P2
23 P2 -.->|failover| P1
24
25 %% Style definitions
26 classDef child fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:16px
27 classDef parent fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:16px
28
29 %% Apply styles
30 class C1,C2,C3 child
31 class P1,P2 parent
32 ```
33
34 Netdata supports building Parent clusters of 2+ nodes. Clustering and high availability work like this:
35
36 ## Cluster Architecture
37
38 :::important
39
40 **How Failover Works**
41
42 1. **Configure your Children to connect to all Parents**.
43 Each Child will use the first working Parent it finds, and automatically switch to backup Parents if the connection fails.
44
45 2. **Configure your Parents to sync with each other**.
46 Each Parent connects to other Parents in the cluster, using the first working connection and failing over to others when needed.
47
48 :::
49
50 :::note
51
52 ### Data Distribution
53
54 **When you set up a cluster, every Parent receives all metrics from every Child**, no matter which Parent the Child is connected to. Parents also share their data with each other, so you never lose visibility into your infrastructure.
55
56 **When a Parent goes down, your Children automatically reconnect to another available Parent**. That Parent then shares the metrics with all other Parents in your cluster, keeping everything in sync.
57
58 ### Netdata Cloud Visibility
59
60 Your cluster ensures continuous visibility in Netdata Cloud. All Parents register your Children with the Cloud, so as long as at least one Parent is online, you'll see all your systems in Netdata Cloud.
61
62 ### Smart Data Retention
63
64 Children only need to keep data long enough to handle Parent switchovers. When a Child connects to a Parent, they work together to fill in any missing data gaps, ensuring complete historical visibility.
65
66 :::
67
68 ## Maintenance Procedures
69
70 <details>
71 <summary><strong>Bringing a Parent back online after maintenance</strong></summary><br/>
72
73 When you bring a Parent back online after extended maintenance, you need to be careful about data gaps. Check the [replication limitations](/docs/observability-centralization-points/metrics-centralization-points/replication-of-past-samples.md#understanding-limitations) to understand the constraints.
74
75 If your Children don't have enough historical data to fill gaps on the restored Parent, temporarily block Children from connecting to it until it syncs with other Parents in your cluster.
76
77 ### Temporarily Blocking Children
78
79 You have two options to block Children while allowing Parent-to-Parent sync:
80
81 1. **Use firewall rules**.
82 Block port 19999 access from your Children's IP ranges to the restored Parent
83 2. **Use separate API keys**.
84 Configure different streaming keys in `stream.conf` for Children vs Parents, then disable the Children's key until sync completes
85
86 <br/>
87 </details>
88
89 <details>
90 <summary><strong>Creating a new Parent from an existing one</strong></summary><br/>
91
92 The fastest way to add a new Parent to your cluster is copying data from an existing Parent using `rsync` to transfer `/var/cache/netdata`.
93
94 :::warning
95
96 **Configure Before Starting**
97
98 Don't start your new Parent with default settings because it might delete the copied data to apply default size limits. Set your retention configuration first, then start the service.
99
100 :::
101
102 ### Setting Up Retention
103
104 Before starting your new Parent, configure these settings in `netdata.conf` to match your existing Parent:
105
106 | Section | Setting | Description | Default |
107 |---------|----------------------------------|-----------------------------------------------|---------|
108 | `[db]` | `dbengine tier 0 retention size` | Maximum disk space for high-resolution data | 1GiB |
109 | `[db]` | `dbengine tier 1 retention size` | Maximum disk space for medium-resolution data | 1GiB |
110 | `[db]` | `dbengine tier 2 retention size` | Maximum disk space for low-resolution data | 1GiB |
111
112 <br/>
113 </details>