| 1 | # Replication of Past Samples |
| 2 | |
| 3 | :::tip |
| 4 | |
| 5 | **What You'll Learn** |
| 6 | |
| 7 | How Netdata automatically fills data gaps when Children reconnect to Parents, including replication limitations, configuration options, and monitoring progress. |
| 8 | |
| 9 | ::: |
| 10 | |
| 11 | When your Netdata Child reconnects to a Parent after being offline, replication automatically kicks in. Your Parent gets the metric samples it missed while the Child was disconnected, ensuring you don't see gaps in your charts. |
| 12 | |
| 13 | :::note |
| 14 | |
| 15 | This same process works between Parents too. When Parents sync with each other, one acts as the sender and the other as the receiver. |
| 16 | |
| 17 | ::: |
| 18 | |
| 19 | ## How Replication Works |
| 20 | |
| 21 | When multiple Netdata Parents are available, the replication happens in sequence, like in the following diagram: |
| 22 | |
| 23 | ```mermaid |
| 24 | flowchart TD |
| 25 | C("Child") |
| 26 | P1("Parent 1") |
| 27 | P2("Parent 2") |
| 28 | |
| 29 | C --> P1 |
| 30 | P1 --> P2 |
| 31 | P1 --> C |
| 32 | P2 --> P1 |
| 33 | |
| 34 | %% Style definitions |
| 35 | classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 36 | classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 37 | classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 38 | classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 39 | |
| 40 | %% Apply styles |
| 41 | class C alert |
| 42 | class P1,P2 complete |
| 43 | ``` |
| 44 | |
| 45 | ### Replication Process |
| 46 | |
| 47 | As shown in the diagram: |
| 48 | |
| 49 | 1. **Connections establish immediately** after a Netdata child connects to any of the Netdata Parents. |
| 50 | 2. **Each connection pair completes replication** (Child→Parent1, Parent1→Parent2) on the receiving side and then initiates replication on the sending side. |
| 51 | 3. **Replication fills gaps up to now**, and the sending side immediately enters streaming mode, without leaving any gaps on the samples of the receiving side. |
| 52 | 4. **Each connection negotiates retention** to back-fill as much data as necessary. |
| 53 | |
| 54 | ## Understanding Limitations |
| 55 | |
| 56 | :::important |
| 57 | |
| 58 | **Key Replication Constraints** |
| 59 | |
| 60 | The current implementation is optimized to replicate small durations and have minimal impact during reconnecting. Understanding these limitations helps you plan your monitoring setup effectively. |
| 61 | |
| 62 | ::: |
| 63 | |
| 64 | <details> |
| 65 | <summary><strong>What Can and Can’t Be Replicated</strong></summary><br/> |
| 66 | |
| 67 | 1. **Append-only replication**. |
| 68 | Replication can only append samples to metrics. Only missing samples at the end of each time-series are replicated. |
| 69 | |
| 70 | 2. **Tier0 samples only**. |
| 71 | Only `tier0` samples are replicated. Samples of higher tiers in Netdata are derived from `tier0` samples, and therefore there is no mechanism for ingesting them directly. This means that the maximum retention that can be replicated across Netdata is limited by the samples available in `tier0` of the sending Netdata. |
| 72 | |
| 73 | 3. **Active metrics only**. |
| 74 | Only samples of metrics that are currently being collected are replicated. Archived metrics (or even archived nodes) will be replicated when and if they are collected again. |
| 75 | |
| 76 | :::note |
| 77 | |
| 78 | Netdata archives metrics 1 hour after they stop being collected, so Netdata Parents may miss data only if Netdata Children are disconnected for more than an hour from their Parents. |
| 79 | |
| 80 | ::: |
| 81 | |
| 82 | <br/> |
| 83 | </details> |
| 84 | |
| 85 | ## Configuration Options |
| 86 | |
| 87 | Configure these options in `netdata.conf` on the respective systems. |
| 88 | |
| 89 | <details> |
| 90 | <summary><strong>Receiving Side Configuration (Netdata Parent)</strong></summary><br/> |
| 91 | |
| 92 | | Setting | Description | Default | |
| 93 | |---------------------------|----------------------------------------------------------------------------------------------------------------------------------|---------| |
| 94 | | `[db].replication period` | Sets the maximum time window for replication. Remember, you're also limited by how much tier0 data your Child systems have kept. | 1 day | |
| 95 | |
| 96 | </details> |
| 97 | |
| 98 | <details> |
| 99 | <summary><strong>Sending Side Configuration (Netdata Children or clustered Parents)</strong></summary><br/> |
| 100 | |
| 101 | | Setting | Description | Default | |
| 102 | |--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------| |
| 103 | | `[db].replication threads` | Controls how many parallel threads handle replication. Each thread can handle about two million samples per second, so more threads can speed up replication between Parents with lots of data. | 1 thread | |
| 104 | | `[db].cleanup obsolete charts after` | Controls how long metrics remain available for replication after collection stops. If you expect Parent maintenance to last longer than 1 hour, increase this setting. Just be aware that in dynamic environments with lots of short-lived metrics, this can increase RAM usage since metrics stay "active" longer. | 1 hour<br/>(3600 seconds) | |
| 105 | |
| 106 | </details> |
| 107 | |
| 108 | ## Monitoring Replication Progress |
| 109 | |
| 110 | :::note |
| 111 | |
| 112 | **Where to Check Replication Status** |
| 113 | |
| 114 | You can monitor how replication is progressing through both your dashboard and API endpoints to make sure your data synchronization is working correctly. |
| 115 | |
| 116 | ::: |
| 117 | |
| 118 | ### Dashboard Monitoring |
| 119 | |
| 120 | Check your replication progress right in your dashboard using the Netdata Function `Netdata-streaming`, under the `Live` tab. |
| 121 | |
| 122 | ### API Monitoring |
| 123 | |
| 124 | You can also get the same information via the API endpoint `http://agent-ip:19999/api/v2/node_instances` on both your Parents and Children. |