docs: Observability centralization points (#20292)
Co-authored-by: ilyam8 <ilya@netdata.cloud>
kanelatechnical committed
May 19, 2025 at 14:11 UTC
3683fc4cfdace2e396d09e0f82d3699785c80416
4 files changed
+867
-147
docs/nodes-ephemerality.md
+99
-28
@@ -6,19 +6,41 @@ Netdata categorizes nodes into two types:
6
7
| Type | Description | Common Use Cases |
8
|---------------|------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
9
-| **Ephemeral** | Expected to disconnect or reconnect frequently | - Auto-scaling cloud instances<br/>- Dynamic containers and VMs<br/>- IoT devices with intermittent connectivity<br/>- Development/test environments with frequent restarts |
10
-| **Permanent** | Expected to maintain continuous connectivity | - Production servers<br/>- Core infrastructure nodes<br/>- Critical monitoring systems<br/>- Stable database servers |
9
+| **Ephemeral** | Expected to disconnect or reconnect frequently | • Auto-scaling cloud instances<br\>• Dynamic containers and VMs<br\>• IoT devices with intermittent connectivity<br\>• Development/test environments with frequent restarts |
10
+| **Permanent** | Expected to maintain continuous connectivity | • Production servers<br\>• Core infrastructure nodes<br\>• Critical monitoring systems<br\>• Stable database servers |
11
12
-> **Note:** Disconnections in permanent nodes indicate potential system failures and require immediate attention.
12
+:::note
13
+
14
+Disconnections in permanent nodes indicate potential system failures and require immediate attention.
15
+
16
+:::
17
18
### Key Benefits
19
16
-1. **Reduced Alert Noise**: Disconnection alerts now apply only to permanent nodes, helping teams focus on actual issues.
17
-2. **Improved Dynamic Infrastructure Support**: Auto-scaling cloud instances, containers, and other temporary resources can be designated as ephemeral to prevent unnecessary alerts.
18
-3. **Automated Node Cleanup**: Ephemeral nodes are removed based on configurable retention periods, keeping dashboards relevant and uncluttered.
20
+1. **Reduced Alert Noise**: Disconnection alerts now apply only to permanent nodes, helping you focus on actual issues.
21
+2. **Improved Dynamic Infrastructure Support**: You can designate auto-scaling cloud instances, containers, and other temporary resources as ephemeral to prevent unnecessary alerts.
22
+3. **Automated Node Cleanup**: You can configure ephemeral nodes to be automatically removed based on your preferred retention periods, keeping your dashboards relevant and uncluttered.
23
24
## Configuring Ephemeral Nodes
25
26
+```mermaid
27
+flowchart TD
28
+ A[Start:<br> Node Permanent by Default] -->|Step 1| B[Open netdata.conf<br> on Target Node]
29
+ B -->|Step 2| C["Add Configuration"]
30
+ C -->|Step 3| D[Restart the Node]
31
+ D --> E[Node Now Marked<br>Ephemeral]
32
+ E --> F[_is_ephemeral<br>Label Applied]
33
+ F --> G[Label Propagates to Parents<br>and Cloud]
34
+
35
+ style A fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
36
+ style B fill:#4caf50,stroke:#333,stroke-width:1px,color:white
37
+ style C fill:#4caf50,stroke:#333,stroke-width:1px,color:white
38
+ style D fill:#4caf50,stroke:#333,stroke-width:1px,color:white
39
+ style E fill:#ffeb3b,stroke:#333,stroke-width:1px,color:black
40
+ style F fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
41
+ style G fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
42
+```
43
+
44
By default, Netdata treats all nodes as permanent. To mark a node as ephemeral:
45
46
1. Open the `netdata.conf` file on the target node.
@@ -29,7 +51,7 @@ By default, Netdata treats all nodes as permanent. To mark a node as ephemeral:
51
```
52
3. Restart the node.
53
32
-This setting applies the `_is_ephemeral` host label, which propagates to Netdata Parents and Netdata Cloud.
54
+This setting applies the `_is_ephemeral` host label, which propagates to your Netdata Parents and Netdata Cloud.
55
56
## Alerts for Parent Nodes
57
@@ -42,25 +64,29 @@ Netdata v2.3.0 introduces two new alerts specifically for permanent nodes:
64
65
## Monitoring Child Node Status
66
45
-To investigate an alert:
46
-
47
-1. Open the `Top` tab in your Netdata dashboard.
48
-2. Select the `Netdata-streaming` function.
49
-3. Review the node status table:
50
- - **Red lines**: Connection issues when nodes attempt to connect to a Parent.
51
- - **Yellow lines**: Restreaming issues when a Parent streams data to another Parent.
52
- - **Color highlighting applies only to permanent nodes**.
53
- - Use the `Ephemerality` filter to view only permanent nodes.
54
- - Check `InStatus`, `InReason`, and `InAge` for incoming connection status.
55
- - Check `OutStatus`, `OutReason`, and `OutAge` for outgoing streaming status.
56
-
57
-## Managing Offline Nodes
58
-
59
-The [Netdata CLI](/src/cli/README.md) tool has two commands for working with archived nodes.
60
-
61
-### mark-stale-nodes-ephemeral
67
+```mermaid
68
+flowchart TD
69
+ A[Start:<br> Permanently Offline Node] -->|Run CLI Command| B[Use netdatacli Utility]
70
+ B -->|Specify Target| C[Specify Node<br> to Mark as Ephemeral]
71
+ C --> D[Node Marked as Ephemeral]
72
+ D --> E[Metrics Data Remains<br> Available]
73
+ D --> F[Active Alerts Cleared]
74
+ D --> G{Node Reconnects?}
75
+ G -->|Yes, without config| H[Reverts to<br> Permanent Status]
76
+ G -->|No| I[Remains Ephemeral]
77
+
78
+ style A fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
79
+ style B fill:#4caf50,stroke:#333,stroke-width:1px,color:white
80
+ style C fill:#4caf50,stroke:#333,stroke-width:1px,color:white
81
+ style D fill:#ffeb3b,stroke:#333,stroke-width:1px,color:black
82
+ style E fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
83
+ style F fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
84
+ style G fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
85
+ style H fill:#f44336,stroke:#333,stroke-width:1px,color:white
86
+ style I fill:#ffeb3b,stroke:#333,stroke-width:1px,color:black
87
+```
88
63
-To mark a permanently offline nodes, including virtual nodes, as ephemeral:
89
+To mark permanently offline nodes, including virtual nodes, as ephemeral:
90
91
```bash
92
netdatacli mark-stale-nodes-ephemeral <node_id | machine_guid | hostname | ALL_NODES>
@@ -68,10 +94,30 @@ netdatacli mark-stale-nodes-ephemeral <node_id | machine_guid | hostname | ALL_N
94
95
This keeps the previously collected metrics data available for querying and clears any active alerts.
96
71
-> **Note:** Nodes will revert to permanent status if they reconnect unless explicitly configured as ephemeral in `netdata.conf`.
97
+:::note
98
+
99
+Nodes will revert to permanent status if they reconnect unless explicitly configured as ephemeral in `netdata.conf`.
100
+
101
+:::
102
103
### remove-stale-node
104
105
+```mermaid
106
+flowchart TD
107
+ A[Start:<br>Offline Node Detected] -->|Use CLI Tool| B[Run Node Removal<br> Command]
108
+ B -->|Specify Target| C[Select Node to Remove]
109
+ C --> D[Node Removed from System]
110
+ D --> E[Node No Longer Queryable]
111
+ D --> F[Alerts for Node Cleared]
112
+
113
+ style A fill:#4caf50,stroke:#333,stroke-width:1px,color:white
114
+ style B fill:#4caf50,stroke:#333,stroke-width:1px,color:white
115
+ style C fill:#4caf50,stroke:#333,stroke-width:1px,color:white
116
+ style D fill:#f44336,stroke:#333,stroke-width:1px,color:white
117
+ style E fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
118
+ style F fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
119
+```
120
+
121
To fully remove permanently offline nodes:
122
123
```bash
@@ -82,13 +128,35 @@ This is like the `mark-stale-nodes-ephemeral` subcommand, but it also removes th
128
129
## Cloud Integration
130
85
-In Netdata Cloud, ephemeral nodes remain visible but marked as 'stale' as long as at least one Agent reports having queryable metrics data for that node. Once all Agents report the node as offline, ephemeral nodes are automatically removed from Cloud.
131
+In Netdata Cloud, ephemeral nodes remain visible but marked as 'stale' as long as at least one Agent reports having queryable metrics data for that node. Once all Agents report the node as offline, ephemeral nodes are automatically removed from the Cloud.
132
133
From v2.3.0 onward, Netdata Cloud sends unreachable-node notifications **only for permanent nodes**, reducing unnecessary alerts.
134
135
## Automatically Removing Ephemeral Nodes
136
91
-By default, Netdata does not automatically remove disconnected ephemeral nodes. To enable automatic cleanup:
137
+```mermaid
138
+flowchart TD
139
+ A[Start:<br> Configure Auto-Removal] -->|Edit Configuration| B[Open netdata.conf<br> on Parent Nodes]
140
+ B -->|Add Configuration| C["Add: [db]\ncleanup<br> ephemeral hosts after = 1d"]
141
+ C -->|Restart Node| D[Restart Netdata]
142
+ D --> E[Ephemeral Node<br> Disconnects]
143
+ E -->|Wait Period| F{24 Hours Passed?}
144
+ F -->|Yes| G[Node<br> Automatically Removed]
145
+ F -->|No| H[Node Remains in System]
146
+ G -->|If All Parents Remove Node| I[Node Removed from Cloud]
147
+
148
+ style A fill:#4caf50,stroke:#333,stroke-width:1px,color:white
149
+ style B fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
150
+ style C fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
151
+ style D fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
152
+ style E fill:#ffeb3b,stroke:#333,stroke-width:1px,color:black
153
+ style F fill:#f9f9f9,stroke:#333,stroke-width:1px,color:black
154
+ style G fill:#f44336,stroke:#333,stroke-width:1px,color:white
155
+ style H fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
156
+ style I fill:#f44336,stroke:#333,stroke-width:1px,color:white
157
+```
158
+
159
+By default, Netdata does not automatically remove disconnected ephemeral nodes. **To enable automatic cleanup**:
160
161
1. Open the `netdata.conf` file on Netdata Parent nodes.
162
2. Add the following configuration:
@@ -99,3 +167,6 @@ By default, Netdata does not automatically remove disconnected ephemeral nodes.
167
3. Restart the node.
168
169
This setting removes ephemeral nodes from queries after 24 hours of disconnection. Once all parent nodes remove a node, Netdata Cloud automatically deletes it as well.
170
+
171
+
172
+
docs/observability-centralization-points/README.md
+72
-18
@@ -1,26 +1,80 @@
1
-# **Observability Centralization Points**
1
+# Observability Centralization Points
2
3
-Netdata allows you to set up multiple **Observability Centralization Points** to aggregate metrics, logs, and metadata across your infrastructure.
3
+## What Are Centralization Points?
4
5
-## **Why Use Centralization Points?**
5
+Observability Centralization Points are specialized Netdata installations that you can configure to **receive, store, and process** observability data (metrics and logs) from multiple other systems in your infrastructure.
6
7
-- **Ephemeral Systems**:
8
- - Ideal for **Kubernetes nodes or temporary VMs** that frequently go offline.
9
- - Ensures metrics and logs remain available for analysis and troubleshooting.
7
+These centralization points give you several core functions:
8
11
-- **Limited Resources**:
12
- - Offloads observability tasks from systems with **low disk space, CPU, RAM, or I/O bandwidth**.
13
- - Keeps production systems running efficiently without performance trade-offs.
9
+* **Receiving and storing** metrics and logs from multiple systems
10
+* **Processing and analyzing** your collected data
11
+* **Running health checks and alerts**
12
+* Providing **unified dashboards** across all your systems
13
+* **Replicating data** for your historical analysis
14
15
-- **Multi-Node Dashboards Without Netdata Cloud**:
16
- - Aggregates data from multiple nodes for **centralized dashboards**, similar to Netdata Cloud.
15
+This **distributed yet centralized** approach gives you the benefits of both decentralized collection and centralized analysis.
16
18
-- **Restricted Netdata Cloud Access**:
19
- - Acts as a **bridge** when monitored systems can’t connect to Netdata Cloud due to **firewall restrictions**.
17
+## Why Use Centralization Points?
18
21
-## **How Multiple Centralization Points Work**
19
+| Use Case | Description | Benefits |
20
+|-------------------------------------------------|------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------|
21
+| **Ephemeral Systems** | Ideal for your Kubernetes nodes or temporary VMs that frequently go offline | You retain metrics and logs for analysis and troubleshooting even after node termination |
22
+| **Limited Resources** | Offloads observability tasks from your systems with low disk space, CPU, RAM, or I/O bandwidth | Your production systems run efficiently without performance trade-offs |
23
+| **Multi-Node Dashboards Without Netdata Cloud** | Aggregates data from all your nodes for centralized dashboards | You get Cloud-like functionality in environments that prefer or require on-premises solutions |
24
+| **Restricted Netdata Cloud Access** | Acts as a bridge when your monitored systems can't connect to Netdata Cloud | You can still use Cloud features despite firewall restrictions or security policies |
25
23
-- **With Netdata Cloud**:
24
- - Queries all centralization points in parallel for a unified view of the infrastructure.
25
-- **Without Netdata Cloud**:
26
- - Parent nodes consolidate data from connected systems, providing a local view of metrics and logs.
26
+## How Multiple Centralization Points Work
27
+
28
+| Scenario | Operation | Advantages |
29
+|-----------------------------|--------------------------------------------------------------------------|--------------------------------------------------------------------------|
30
+| **With Netdata Cloud** | Queries all your centralization points in parallel for a unified view | You get a seamless experience regardless of your underlying architecture |
31
+| **Without Netdata Cloud** | Your centralization points consolidate data from connected systems | You have a local view of metrics and logs without external dependencies |
32
+| **High Availability Setup** | Your centralization points share data with each other, forming a cluster | You won't lose data if one centralization point fails |
33
+
34
+```mermaid
35
+graph TD
36
+ A[Centralization Points<br>Architecture] --> B[Single Centralization Point<br> Setup]
37
+ A --> C[Multiple Independent<br>Centralization Points]
38
+ A --> D[High Availability Cluster]
39
+
40
+ B --> B1[All systems stream<br>to one centralization point]
41
+ C --> C1[Systems divided<br>by region/service/team]
42
+ D --> D1[Centralization points<br>share data with each other]
43
+
44
+classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333;
45
+classDef green fill:#4caf50,stroke:#333,stroke-width:1px,color:black;
46
+class A default;
47
+class B,C,D,B1,C1,D1 green;
48
+```
49
+
50
+## Technical Implementation
51
+
52
+Observability Centralization Points consist of two major components you can deploy:
53
+
54
+1. **Metrics Centralization** - Uses Netdata's streaming and replication features to centralize your metrics data
55
+2. **Logs Centralization** - Uses systemd-journald methodologies to centralize your log data
56
+
57
+You can configure your systems to connect to **multiple centralization points** for redundancy. If a connection fails, they automatically switch to an available alternative.
58
+
59
+In a **high-availability setup**, your centralization points can form a cluster by sharing data with each other, ensuring all points have a complete copy of all your metrics and logs.
60
+
61
+```mermaid
62
+graph TD
63
+ CP1[Centralization Point 1] --- CP2[Centralization Point 2]
64
+
65
+ S1[System 1] --> CP1
66
+ S2[System 2] --> CP1
67
+ S3[System 3] --> CP2
68
+ S4[System 4] --> CP2
69
+
70
+ S1 -.-> CP2
71
+ S2 -.-> CP2
72
+ S3 -.-> CP1
73
+ S4 -.-> CP1
74
+
75
+classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333;
76
+classDef green fill:#4caf50,stroke:#333,stroke-width:1px,color:black;
77
+classDef blue fill:#2196F3,stroke:#333,stroke-width:1px,color:white;
78
+class CP1,CP2 default;
79
+class S1,S2,S3,S4 blue;
80
+```
docs/observability-centralization-points/best-practices.md
+77
-18
@@ -4,32 +4,91 @@
4
5
When setting up Observability Centralization Points, consider the following:
6
7
-1. **System Volume**: The number of monitored systems impacts scaling. Larger infrastructures may need multiple centralization points to maintain performance.
8
-2. **Data Transfer Costs**: In multi-cloud or hybrid environments, placing centralization points strategically reduces egress bandwidth costs.
9
-3. **Usability Without Netdata Cloud**: Using fewer centralization points simplifies access and management when Netdata Cloud is not in use.
10
-4. **Optimized Deployment with Netdata Cloud**: Netdata Cloud provides a complete infrastructure view, allowing you to optimize based on:
11
- - **Security** (internet access controls)
12
- - **Cost** (bandwidth and resource allocation)
13
- - **Operational needs** (regional, service, or team-based isolation)
7
+| Factor | Description | Impact |
8
+|---------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------|
9
+| **System Volume** | The number of monitored systems | Larger infrastructures may need multiple centralization points to maintain performance |
10
+| **Data Transfer Costs** | Bandwidth usage between environments | Strategic placement reduces egress bandwidth costs in multi-cloud or hybrid environments |
11
+| **Usability Without Netdata Cloud** | Standalone operation considerations | Fewer centralization points simplifies access and management |
12
+| **Optimized Deployment with Netdata Cloud** | Cloud integration benefits | Provides complete infrastructure view with optimized security, cost, and operational controls |
13
+
14
+```mermaid
15
+graph TD
16
+ A[Optimized Deployment<br>with Netdata Cloud] --> B[Security]
17
+ A --> C[Cost]
18
+ A --> D[Operational Needs]
19
+
20
+ B --> B1[Internet access controls]
21
+ C --> C1[Bandwidth and<br>resource allocation]
22
+ D --> D1[Regional, service, or<br>team-based isolation]
23
+
24
+classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333;
25
+classDef green fill:#4caf50,stroke:#333,stroke-width:1px,color:black;
26
+class A default;
27
+class B,C,D,B1,C1,D1 green;
28
+```
29
30
## Cost Optimization Strategies
31
32
Netdata is designed to keep observability efficient and cost-effective. To manage costs:
33
19
-- **Scale Out**: Use multiple smaller centralization points to improve efficiency and performance.
20
-- **Use Existing Resources**: Leverage spare capacity before dedicating new resources to observability.
21
-- **Centralized or Separate Logs & Metrics**: Choose whether to store logs and metrics together or separately based on access needs, retention policies, and compliance.
22
-- **Flexible Configuration Management**: Each centralization point can have unique retention and alert settings, helping to control costs and tailor observability for different teams or services.
34
+| Strategy | Description | Benefit |
35
+|--------------------------------------------|--------------------------------------------|-------------------------------------------------------------------------------------------------|
36
+| **Scale Out** | Use multiple smaller centralization points | Improves efficiency and performance across distributed systems |
37
+| **Use Existing Resources** | Leverage spare capacity | Minimize additional hardware costs by using available resources |
38
+| **Centralized or Separate Logs & Metrics** | Choose storage approach based on needs | Optimize based on access patterns, retention policies, and compliance requirements |
39
+| **Flexible Configuration Management** | Customize each centralization point | Control costs with unique retention and alert settings tailored for different teams or services |
40
+
41
+```mermaid
42
+graph TD
43
+ A[Cost Optimization<br>Strategies] --> B[Scale Out]
44
+ A --> C[Use Existing<br>Resources]
45
+ A --> D[Centralized or<br>Separate Logs & Metrics]
46
+ A --> E[Flexible<br>Configuration Management]
47
+
48
+ B --> B1[Multiple smaller<br>centralization points]
49
+ C --> C1[Leverage spare capacity]
50
+ D --> D1[Based on access needs,<br>retention policies,<br>and compliance]
51
+ E --> E1[Unique settings for<br>different teams or services]
52
+
53
+classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333;
54
+classDef green fill:#4caf50,stroke:#333,stroke-width:1px,color:black;
55
+class A default;
56
+class B,C,D,E,B1,C1,D1,E1 green;
57
+```
58
59
## Advantages of Netdata's Approach
60
61
Netdata provides several benefits over other observability solutions:
62
28
-- **Scalability & Flexibility**: Multiple independent centralization points allow for customized observability by region, service, or team.
29
-- **Resilience & Reliability**: Built-in replication ensures that observability continues even if a centralization point fails.
30
-- **Optimized Cost & Performance**: Distributing workloads prevents bottlenecks and improves resource efficiency.
31
-- **Ease of Use**: Netdata Agents require minimal setup and maintenance, reducing complexity.
32
-- **On-Prem Control**: Centralization points remain on-prem even when using Netdata Cloud, keeping data within your infrastructure.
33
-- **Comprehensive Observability**: Netdata enables deep visibility by segmenting infrastructure into independent observability points with tailored retention, alerts, and machine learning, while Netdata Cloud provides a unified view.
63
+| Advantage | Description | Value |
64
+|----------------------------------|--------------------------------------------|-----------------------------------------------------------------------|
65
+| **Scalability & Flexibility** | Multiple independent centralization points | Customized observability by region, service, or team |
66
+| **Resilience & Reliability** | Built-in replication | Observability continues even if a centralization point fails |
67
+| **Optimized Cost & Performance** | Distributed workloads | Prevents bottlenecks and improves resource efficiency |
68
+| **Ease of Use** | Minimal setup and maintenance | Reduces complexity and operational overhead |
69
+| **On-Prem Control** | Data remains within your infrastructure | Enhanced security and compliance, even when using Netdata Cloud |
70
+| **Comprehensive Observability** | Segmented infrastructure with unified view | Deep visibility with tailored retention, alerts, and machine learning |
71
+
72
+```mermaid
73
+graph TD
74
+ A[Advantages of<br>Netdata's Approach] --> B[Scalability & Flexibility]
75
+ A --> C[Resilience & Reliability]
76
+ A --> D[Optimized Cost &<br>Performance]
77
+ A --> E[Ease of Use]
78
+ A --> F[On-Prem Control]
79
+ A --> G[Comprehensive<br>Observability]
80
+
81
+ B --> B1[Customized observability<br>by region, service, or team]
82
+ C --> C1[Observability continues<br> even if a centralization<br> point fails]
83
+ D --> D1[Prevents bottlenecks<br>and improves<br>resource efficiency]
84
+ E --> E1[Minimal setup and maintenance]
85
+ F --> F1[Data remains within<br>your infrastructure]
86
+ G --> G1[Unified view with<br>tailored segments]
87
+
88
+classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333;
89
+classDef green fill:#4caf50,stroke:#333,stroke-width:1px,color:black;
90
+class A default;
91
+class B,C,D,E,F,G,B1,C1,D1,E1,F1,G1 green;
92
+```
93
35
-Following these best practices helps maintain a **cost-effective**, **high-performance** observability setup with Netdata.
94
+Following these best practices helps you maintain a **cost-effective**, **high-performance** observability setup with Netdata.
src/streaming/README.md
+619
-83
@@ -1,6 +1,67 @@
1
# Streaming and Replication Reference
2
3
-This guide covers advanced streaming options and recommended deployment strategies for production environments. If you're new to Netdata streaming, start with the [quick introduction to streaming](/docs/observability-centralization-points/README.md) to set up a basic parent-child configuration.
3
+## Introduction
4
+
5
+This guide covers Netdata's advanced streaming and replication capabilities, which allow you to build centralized observability points across your infrastructure.
6
+
7
+**[Streaming and replication](https://learn.netdata.cloud/docs/developer-and-contributor-corner/glossary#r)** work together to send metrics data from one Netdata Agent (child) to another Netdata Agent (parent). Streaming sends metrics in real-time, while replication ensures historical data is copied, as well, maintaining complete data integrity even after connection interruptions.
8
+
9
+::: tip
10
+
11
+If you're new to Netdata streaming or prefer a guided approach, [jump to our step-by-step guide](#step-by-step-setup-guide) at the end of this document. The guide will walk you through setting up a basic streaming configuration and then connecting to the comprehensive reference sections as needed.
12
+
13
+For a quick reference on setting up the parent-child relationship, see the [example configurations](#complete-configuration-examples) or refer to our comprehensive [Centralized Deployment Guide](https://learn.netdata.cloud/docs/deployment-guides/centralized) for more details.
14
+
15
+:::
16
+
17
+## Understanding Streaming Architecture
18
+
19
+Before diving into configuration details, it's important to understand the key concepts behind Netdata's streaming architecture:
20
+
21
+```mermaid
22
+graph TD
23
+ A[Child Node] -->|Streams real-time metrics| B[Parent Node]
24
+ A -->|Replicates historical data| B
25
+ C[Child Node] -->|Streams real-time metrics| B
26
+ C -->|Replicates historical data| B
27
+ B -->|Presents unified dashboard| D[Monitoring User]
28
+
29
+classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333;
30
+classDef green fill:#4caf50,stroke:#333,stroke-width:1px,color:black;
31
+class A,C green;
32
+class B,D default;
33
+```
34
+
35
+### Parent-Child Relationship
36
+
37
+- **Child nodes** (data sources) collect metrics directly from systems they monitor
38
+- **Parent nodes** (data centralization points) receive, store, and visualize metrics from multiple child nodes
39
+- **A single parent can receive data from many children**, creating a centralized monitoring hub
40
+- **Child nodes maintain independence and continue collecting metrics** even if the connection to a parent is lost
41
+
42
+### Data Flow
43
+
44
+1. **Collection**: Child nodes collect metrics from their local systems
45
+2. **Streaming**: Child nodes send real-time metrics to parent nodes
46
+3. **Replication**: When a connection is established or restored, child nodes replicate historical data to ensure parents have complete history
47
+4. **Storage**: Parent nodes store metrics based on their configured retention policies
48
+5. **Visualization**: Users access the parent node's dashboard to view metrics from all connected child nodes
49
+
50
+### Benefits of This Architecture
51
+
52
+- **Efficiency**: Distribute collection workload across multiple nodes while centralizing visualization
53
+- **Resilience**: Maintain data collection even during network disruptions or parent node failures
54
+- **Scalability**: Add more child nodes or additional parent nodes as your infrastructure grows
55
+- **Flexibility**: Configure retention, alerts, and dashboards according to your specific needs
56
+
57
+## Quick Reference
58
+
59
+| Task | Configuration | Example |
60
+|------------------------------------------|-------------------------------------------|----------------------------------------------------------------|
61
+| Enable streaming on a child | Set `enabled = yes` in `[stream]` section | `[stream]`<br\>`enabled = yes`<br\>`destination = 192.168.1.5` |
62
+| Configure a parent to accept connections | Create an `[API_KEY]` section | `[API_KEY]`<br\>`enabled = yes`<br\>`allow from = *` |
63
+| Set up high availability | Configure multiple destinations on child | `[stream]`<br\>`destination = parent1:19999 parent2:19999` |
64
+| Filter which metrics to send | Use `send charts matching` setting | `send charts matching = system.* !system.uptime` |
65
66
## Configuration Overview
67
@@ -12,7 +73,10 @@ Netdata's streaming capabilities are configured through two key files:
73
To edit these files, navigate to your Netdata configuration directory (typically `/etc/netdata`) and run:
74
75
```sh
76
+# Edit streaming configuration
77
sudo ./edit-config stream.conf
78
+
79
+# Edit global Netdata settings
80
sudo ./edit-config netdata.conf
81
```
82
@@ -20,9 +84,9 @@ sudo ./edit-config netdata.conf
84
85
The `stream.conf` file has three main sections:
86
23
-- **`[stream]`** – Configures child nodes.
24
-- **`[API_KEY]`** – Defines settings for all child nodes using the same API key.
25
-- **`[MACHINE_GUID]`** – Sets configurations for a specific child node matching the given GUID.
87
+1. **`[stream]`** – With these settings, you can configure how child nodes send metrics.
88
+2. **`[API_KEY]`** – Here you can define settings for authentication and access control between parents and children.
89
+3. **`[MACHINE_GUID]`** – This area lets you customize settings for specific child nodes by their unique ID.
90
91
### Identifying a Node's GUID
92
@@ -38,24 +102,67 @@ This file is generated automatically the first time Netdata starts and remains u
102
103
For a production-ready streaming setup, consider the following best practices:
104
41
-- **Use Multiple Parent Nodes** – Ensures redundancy and improves resilience.
42
-- **Optimize Data Retention** – Configure retention periods to balance storage costs and data availability.
43
-- **Secure Communications** – Enable encryption and authentication to protect data streams.
44
-- **Monitor Performance** – Regularly review logs and metrics to ensure efficient streaming operations.
105
+```mermaid
106
+graph TD
107
+ A[Recommended Strategies] --> B[Multiple Parent Nodes]
108
+ A --> C[Optimized Data Retention]
109
+ A --> D[Secure Communications]
110
+ A --> E[Performance Monitoring]
111
+
112
+ B --> B1[Improved redundancy<br>and resilience]
113
+ C --> C1[Balance storage costs<br>and data availability]
114
+ D --> D1[Enable encryption<br>and authentication]
115
+ E --> E1[Regular log and<br>metric reviews]
116
+
117
+classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333;
118
+classDef green fill:#4caf50,stroke:#333,stroke-width:1px,color:black;
119
+class A default;
120
+class B,C,D,E,B1,C1,D1,E1 green;
121
+```
122
+
123
+:::tip
124
+
125
+### Multiple Parent Nodes
126
+
127
+Setting up multiple parent nodes creates redundancy in your monitoring infrastructure. **If one parent fails, child nodes can automatically switch to another available parent.** This approach:
128
+
129
+- **Prevents single points of failure** in your monitoring system
130
+- **Allows for maintenance** on parent nodes without monitoring interruptions
131
+- Can be **geographically distributed to reduce latency** for global deployments
132
+
133
+### Optimized Data Retention
134
+
135
+Configure data retention settings based on your specific monitoring needs:
136
46
-By following these guidelines, you can set up a scalable and reliable Netdata streaming environment.
137
+- **Use longer retention periods** for critical systems and metrics
138
+- **Implement tiered storage strategies** with different retention policies
139
+- **Balance storage requirements with data availability** for historical analysis
140
48
-## `stream.conf`
141
+### Secure Communications
142
50
-The `stream.conf` file consists of three main sections:
143
+**Protect your metrics data** during transmission:
144
52
-1. **`[stream]`** – Configures child nodes (data senders).
53
-2. **`[API_KEY]`** – Defines API keys for parent nodes (data receivers).
54
-3. **`[MACHINE_GUID]`** – Customizes settings for specific child nodes.
145
+- **Enable SSL/TLS encryption** for all streaming connections
146
+- Implement **proper API key management** and rotation
147
+- Use **IP-based restrictions** to control which nodes can connect
148
+
149
+### Performance Monitoring
150
+
151
+Regularly evaluate the **health of your** streaming **setup**:
152
+
153
+- **Monitor network traffic** between child and parent nodes
154
+- **Check for buffer overflows** or connection issues
155
+- **Adjust settings** like buffer size based on actual performance
156
+
157
+**By following these guidelines, you can set up a scalable and reliable Netdata streaming environment.**
158
+
159
+:::
160
+
161
+## `stream.conf` Detailed Reference
162
163
### `[stream]` Section (Child Node Settings)
164
58
-This section configures a child node to send metrics to a parent.
165
+With these settings, you can configure how your child nodes send metrics to parent nodes.
166
167
| Setting | Default | Description |
168
|-------------------------------------------------|---------------------------|---------------------------------------------------------------------|
@@ -75,7 +182,7 @@ This section configures a child node to send metrics to a parent.
182
183
### `[API_KEY]` Section (Parent Node Authentication)
184
78
-This section allows parent nodes to accept streaming data from child nodes using an API key.
185
+Here you can define settings for authentication and access control between parents and children.
186
187
| Setting | Default | Description |
188
|------------------------------|------------|-------------------------------------------------------------|
@@ -99,11 +206,11 @@ This section allows parent nodes to accept streaming data from child nodes using
206
207
### `[MACHINE_GUID]` Section (Per-Node Customization)
208
102
-This section customizes settings for specific child nodes using their unique Machine GUID.
209
+This area lets you customize settings for specific child nodes by their unique ID.
210
211
| Setting | Default | Description |
212
|------------------------------|------------|----------------------------------------------------------|
106
-| `enabled` | `no` | Enables or disables this specific node’s configuration. |
213
+| `enabled` | `no` | Enables or disables this specific node's configuration. |
214
| `type` | `machine` | Defines the section as a machine-specific configuration. |
215
| [`allow from`](#allow-from) | `*` | Lists IP addresses allowed to stream metrics. |
216
| `retention` | `3600` | Retention period for child metrics in RAM-based storage. |
@@ -138,6 +245,7 @@ Example (TCP connection with SSL to `203.0.113.0` on port `20000`):
245
246
```ini
247
[stream]
248
+ # Send metrics securely to the parent at 203.0.113.0:20000
249
destination = tcp:203.0.113.0:20000:SSL
250
```
251
@@ -150,6 +258,7 @@ Controls which charts are streamed.
258
259
```ini
260
[stream]
261
+ # Only send CPU application charts and all system charts
262
send charts matching = apps.cpu system.*
263
```
264
@@ -157,6 +266,7 @@ Controls which charts are streamed.
266
267
```ini
268
[stream]
269
+ # Send all charts except CPU application charts
270
send charts matching = !apps.cpu *
271
```
272
@@ -168,6 +278,7 @@ Defines which child nodes (by IP) can connect.
278
279
```ini
280
[API_KEY]
281
+ # Only allow connections from 203.0.113.10
282
allow from = 203.0.113.10
283
```
284
@@ -175,6 +286,7 @@ Defines which child nodes (by IP) can connect.
286
287
```ini
288
[API_KEY]
289
+ # Allow all 10.*.*.* addresses except 10.1.2.3
290
allow from = !10.1.2.3 10.*
291
```
292
@@ -188,18 +300,15 @@ Defines the database mode:
300
301
```ini
302
[API_KEY]
303
+ # Use disk-based database for all metrics
304
db = dbengine
305
```
306
194
-Here’s an optimized version of the `netdata.conf` structure with clearer, more direct language. I've broken down the key sections for readability and made the content less dense:
195
-
196
-## `netdata.conf`
307
+## `netdata.conf` Settings Affecting Streaming
308
198
-The `netdata.conf` file is the primary configuration file for the Netdata agent. It controls the agent’s settings, including networking, data collection, logging, and resource usage.
309
+The `netdata.conf` file is the primary configuration file for the Netdata agent. The following sections can impact streaming:
310
200
-### Sections
201
-
202
-#### [global]
311
+### [global]
312
313
This section defines global settings for the Netdata agent.
314
@@ -207,7 +316,7 @@ This section defines global settings for the Netdata agent.
316
- **memory mode**: Choose the memory mode for data collection (e.g., `ram` or `swap`).
317
- **error log file**: Path to the file where error logs are saved.
318
210
-#### [web]
319
+### [web]
320
321
Configure the web interface settings here.
322
@@ -215,14 +324,7 @@ Configure the web interface settings here.
324
- **port**: Set the port for the web interface (default: 19999).
325
- **disable SSL**: Set to `yes` to disable SSL support.
326
218
-#### [plugin]
219
-
220
-This section configures individual plugins for data collection.
221
-
222
-- **enabled**: Enable or disable the plugin.
223
-- **update every**: Define the update frequency (in seconds).
224
-
225
-#### [database]
327
+### [database]
328
329
Manage database settings for data storage and retention.
330
@@ -230,108 +332,542 @@ Manage database settings for data storage and retention.
332
- **data retention**: Set how long to keep historical data.
333
- **compression**: Enable or disable data compression.
334
233
-#### [logging]
335
+## Complete Configuration Examples
336
235
-Configure logging behavior for Netdata.
337
+### Basic Parent-Child Setup
338
237
-- **log file**: Define the log file location.
238
-- **log level**: Set the verbosity of logs (e.g., `info`, `debug`).
339
+**Parent node configuration (stream.conf):**
340
240
-## Troubleshooting
341
+```ini
342
+# Generate a random UUID first: uuidgen
343
+[11111111-2222-3333-4444-555555555555]
344
+ # Enable this API key
345
+ enabled = yes
346
+ # Allow all IPs to connect with this key
347
+ allow from = *
348
+ # Store data using dbengine for persistence
349
+ db = dbengine
350
+```
351
242
-Both parent and child nodes log information in `/var/log/netdata/error.log`.
352
+**Child node configuration (stream.conf):**
353
+
354
+```ini
355
+[stream]
356
+ # Enable streaming on this node
357
+ enabled = yes
358
+ # Connect to parent at 192.168.1.5 port 19999
359
+ destination = 192.168.1.5
360
+ # Use the same API key defined on the parent
361
+ api key = 11111111-2222-3333-4444-555555555555
362
+```
363
+
364
+### High Availability Setup with Multiple Parents
365
+
366
+**Parent nodes configuration (stream.conf on both parents):**
367
+
368
+```ini
369
+# Configuration for accepting metrics from children
370
+[11111111-2222-3333-4444-555555555555]
371
+ enabled = yes
372
+ allow from = *
373
+ db = dbengine
374
+
375
+# Configuration for accepting metrics from other parents
376
+[22222222-3333-4444-5555-666666666666]
377
+ enabled = yes
378
+ # Only allow the other parent's IP
379
+ allow from = 192.168.1.5 192.168.1.6
380
+ db = dbengine
381
+```
382
+
383
+**First parent node's configuration for streaming to the second parent:**
384
+
385
+```ini
386
+[stream]
387
+ enabled = yes
388
+ destination = 192.168.1.6
389
+ api key = 22222222-3333-4444-5555-666666666666
390
+```
391
+
392
+**Second parent node's configuration for streaming to the first parent:**
393
+
394
+```ini
395
+[stream]
396
+ enabled = yes
397
+ destination = 192.168.1.5
398
+ api key = 22222222-3333-4444-5555-666666666666
399
+```
400
+
401
+**Child node configuration:**
402
244
-If the child successfully connects to the parent, you’ll see logs similar to the following on the parent:
403
+```ini
404
+[stream]
405
+ enabled = yes
406
+ # List both parents for failover
407
+ destination = 192.168.1.5 192.168.1.6
408
+ api key = 11111111-2222-3333-4444-555555555555
409
+```
410
+
411
+## Verifying Successful Connections
412
+
413
+If the streaming configuration is working correctly, you'll see logs similar to the following.
414
+
415
+On the parent side:
416
417
```
418
2017-03-09 09:38:52: netdata: INFO : STREAM [receive from [10.11.12.86]:38564]: new client connection.
419
2017-03-09 09:38:52: netdata: INFO : STREAM xxx [10.11.12.86]:38564: receive thread created (task id 27721)
249
-2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: client willing to stream metrics for host 'xxx' with machine_guid '1234567-1976-11e6-ae19-7cdd9077342a': update every = 1, history = 3600, memory mode = ram, health auto
250
-2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: initializing communication...
251
-2017-03-09 09:38:52: netdata: INFO : STREAM xxx [receive from [10.11.12.86]:38564]: receiving metrics...
420
```
421
254
-On the child side, you might see:
422
+On the child side:
423
424
```
425
2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: connecting...
258
-2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: initializing communication...
259
-2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: waiting response from remote netdata...
426
2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: established communication - sending metrics...
427
```
428
263
-The following sections cover common issues when connecting parent and child nodes.
429
+Both parent and child nodes log information in `/var/log/netdata/error.log`.
430
265
-### Slow Connections Between Parent and Child
431
+## Troubleshooting
432
+
433
+<details>
434
+<summary><strong>Slow Connection Issues</strong></summary>
435
+<br/>
436
+
437
+**Symptoms:**
438
267
-Slow connections may lead to several errors, mainly logged in the child’s `error.log`:
439
+- Buffer overflow errors
440
+- Connection resets
441
+- Partial message errors
442
269
-```bash
443
+**Child logs:**
444
+
445
+```
446
netdata ERROR : STREAM_SENDER[CHILD HOSTNAME] : STREAM CHILD HOSTNAME [send to PARENT IP:PARENT PORT]: too many data pending - buffer is X bytes long, Y unsent - we have sent Z bytes in total, W on this connection. Closing connection to flush the data.
447
```
448
273
-On the parent side, you might see:
449
+**Parent logs:**
450
451
```
452
netdata ERROR : STREAM_PARENT[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : read failed: end of file
453
```
454
279
-Another issue in slow connections is the child sending partial messages to the parent. In this case, the parent will log:
455
+**What's happening:**
456
+Slow network connections or high-latency links can cause the streaming buffer to fill up faster than it can be transmitted. When the buffer reaches its maximum size, Netdata closes the connection to flush the pending data, then re-establishes the connection. This can lead to data gaps or inconsistencies if it happens frequently.
457
281
-```
282
-ERROR : STREAM_RECEIVER[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : sent command 'B' which is not known by netdata, for host 'HOSTNAME'. Disabling it.
283
-```
458
+**Solutions:**
459
285
-Slow connections can also cause the parent to miss a message. For example, if the parent misses a message about the child’s charts and then receives a `SET` command for a chart, the parent might log:
460
+- Increase buffer size in `stream.conf`: `buffer size bytes = 20971520` (20MB)
461
+- Check network bandwidth and latency between nodes
462
+- Consider reducing the collection frequency on high-volume metrics
463
+- If possible, place parent nodes closer (network-wise) to child nodes
464
287
-```
288
-ERROR : STREAM_RECEIVER[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : requested a SET on chart 'CHART NAME' of host 'HOSTNAME', without a dimension. Disabling it.
289
-```
465
+</details>
466
+
467
+<details>
468
+<summary><strong>Connection Issues</strong></summary>
469
+<br/>
470
291
-### Child Can’t Connect to Parent
471
+**Symptoms:**
472
293
-If the child can't connect to the parent (due to misconfiguration, networking issues, firewalls, or the parent being down), the child will log:
473
+- Child can't establish a connection to a parent
474
+- Repeated reconnection attempts
475
+
476
+**Child logs:**
477
478
```
479
ERROR : STREAM_SENDER[HOSTNAME] : Failed to connect to 'PARENT IP', port 'PARENT PORT' (errno 113, No route to host)
480
```
481
299
-### 'Is This a Netdata?'
482
+**What's happening:**
483
+This error indicates network connectivity problems between the child and parent nodes. It could be due to firewall rules, incorrect IP addresses, or the parent node not running.
484
+
485
+**Solutions:**
486
+
487
+- Verify firewalls allow traffic on port 19999 (or your custom port)
488
+- Check that the parent node is running and listening on the correct interface
489
+- Verify that the IP address/hostname is correct in child's configuration
490
+- Test basic connectivity with tools like `ping` or `telnet`
491
+- Check network routing between the nodes
492
301
-This error typically occurs when the parent is using SSL and the child attempts a plain-text connection, or if the child tries to connect to a non-Netdata server. The error message looks like this:
493
+</details>
494
+
495
+<details>
496
+<summary><strong>Authentication and Permission Issues</strong></summary>
497
+<br/>
498
+
499
+**Symptoms:**
500
+
501
+- Connection established but immediately rejected
502
+- "Forbidding access" errors
503
+
504
+**Parent logs:**
505
506
```
304
-ERROR : STREAM_SENDER[CHILD HOSTNAME] : STREAM child HOSTNAME [send to PARENT HOSTNAME:PARENT PORT]: server is not replying properly (is it a netdata?).
507
+STREAM [receive from [child HOSTNAME]:child IP]: `API key 'VALUE' is not allowed`. Forbidding access.
508
```
509
307
-### Stream Charts Wrong
510
+**What's happening:**
511
+The parent node is rejecting the connection because the API key doesn't match or the child's IP address is not allowed by the `allow from` setting.
512
+
513
+**Solutions:**
514
309
-If chart data is inconsistent between the parent and child (e.g., gaps in metrics collection), it likely indicates a mismatch in the `[db].db` settings between the parent and child. Refer to our [db documentation](/src/database/README.md) for more information on how Netdata stores metrics to ensure data consistency.
515
+- Verify if the API key matches exactly between parent and child
516
+- Check that the `allow from` setting permits the child's IP address
517
+- Ensure GUID formats are valid
518
+- Check for whitespace or invisible characters in the API key
519
+- Remember that API keys are case-sensitive
520
311
-### Forbidding Access
521
+</details>
522
313
-Access might be forbidden for several reasons, such as slow connections or other failures. Look for the following errors in the parent’s `error.log`:
523
+<details>
524
+<summary><strong>'Is This a Netdata?' Error</strong></summary>
525
+<br/>
526
+
527
+**Symptoms:**
528
+
529
+- Child tries to connect but receives an unexpected response
530
+
531
+**Child logs:**
532
533
```
316
-STREAM [receive from [child HOSTNAME]:child IP]: `MESSAGE`. Forbidding access."
534
+ERROR : STREAM_SENDER[CHILD HOSTNAME] : STREAM child HOSTNAME [send to PARENT HOSTNAME:PARENT PORT]: server is not replying properly (is it a netdata?).
535
```
536
319
-Possible causes for this error include:
537
+**What's happening:**
538
+The child node is connecting to the destination, but the server is not responding with the expected Netdata streaming protocol. This commonly occurs when there's a mismatch in SSL/TLS settings or when the destination is not a Netdata server.
539
321
-- `request without KEY`: Incomplete message, missing API key, hostname, or machine GUID.
322
-- `API key 'VALUE' is not valid GUID`: Invalid UUID format.
323
-- `machine GUID 'VALUE' is not GUID.`: Invalid machine GUID.
324
-- `API key 'VALUE' is not allowed`: Invalid API key.
325
-- `API key 'VALUE' is not permitted from this IP`: IP not allowed to use STREAM with this parent.
326
-- `machine GUID 'VALUE' is not allowed.`: GUID not permitted.
327
-- `Machine GUID 'VALUE' is not permitted from this IP.`: IP not matching the allowed pattern.
540
+**Solutions:**
541
329
-### Netdata Could Not Create a Stream
542
+- Check SSL settings in the destination URL (add or remove `:SSL` as needed)
543
+- Verify that you're connecting to a Netdata server and not another service
544
+- Ensure both nodes are running compatible Netdata versions
545
+- Check if a proxy or firewall is altering the connection
546
331
-If the parent can’t convert the initial connection into a stream, it will log the following error:
547
+</details>
548
549
+<details>
550
+<summary><strong>Stream Charts Wrong</strong></summary>
551
+<br/>
552
+
553
+**Symptoms:**
554
+
555
+- Data inconsistencies between parent and child
556
+- Gaps in metrics collection
557
+
558
+**What's happening:**
559
+When the database settings between parent and child nodes don't match, it can cause inconsistencies in how data is stored and displayed. The most common cause is different memory modes or retention settings.
560
+
561
+**Solutions:**
562
+
563
+- Check for mismatch in the `[db].db` settings between the parent and child
564
+- Ensure database retention settings are compatible
565
+- Verify replication is enabled and properly configured
566
+- Make sure both nodes are using the same (or compatible) database engine
567
+- Check that clocks are synchronized between nodes
568
+
569
+</details>
570
+
571
+## FAQ
572
+
573
+<details>
574
+<summary><strong>Can I stream to multiple parents simultaneously?</strong></summary>
575
+<br/>
576
+
577
+No, you can't stream to multiple parents at the same time. However, you can configure multiple destinations for failover. Your child node will connect to the first available parent in the list.
578
+</details>
579
+
580
+<details>
581
+<summary><strong>How does replication work with interrupted connections?</strong></summary>
582
+<br/>
583
+
584
+When you re-establish a connection, your child node will replicate historical data based on the `replication period` setting. This ensures your parent has a complete history even after interruptions.
585
+</details>
586
+
587
+<details>
588
+<summary><strong>How much bandwidth does streaming use?</strong></summary>
589
+<br/>
590
+
591
+Your streaming setup will be very efficient, especially with compression enabled. Typically, it uses about 10–20 KB/s for a moderately active node. The actual bandwidth depends on the number of metrics and collection frequency you've configured.
592
+</details>
593
+
594
+<details>
595
+<summary><strong>Can I filter which metrics are sent to the parent?</strong></summary>
596
+<br/>
597
+
598
+Yes, you can use the `send charts matching` setting to include or exclude specific metrics from streaming. This works with wildcard patterns, giving you precise control over what metrics are transferred.
599
+</details>
600
+
601
+<details>
602
+<summary><strong>How do I secure the streaming connection?</strong></summary>
603
+<br/>
604
+
605
+You can enable SSL in the destination setting by adding `:SSL` at the end. Configure proper certificates using the `CAfile` and `CApath` settings for production environments to ensure your metric data is protected in transit.
606
+</details>
607
+
608
+<details>
609
+<summary><strong>Do I need to configure streaming on every child node?</strong></summary>
610
+<br/>
611
+
612
+Yes, you need to configure each child node with its own streaming configuration. However, you can use configuration management tools to deploy a standard configuration across your infrastructure, making this process more efficient.
613
+</details>
614
+
615
+## Step-by-Step Setup Guide
616
+
617
+This guide will walk you through setting up Netdata streaming between nodes. **By following these sequential steps, you'll create a basic streaming configuration** that you can later customize based on your needs.
618
+
619
+<details>
620
+<summary><strong>Step 1: Prepare Your Environment</strong></summary>
621
+<br/>
622
+
623
+Before configuring streaming, ensure you have:
624
+
625
+1. At least two Netdata instances are installed (one to act as parent, one as child)
626
+2. Network connectivity between the instances
627
+3. Administrative access to edit configuration files on both systems
628
+
629
+</details>
630
+
631
+<details>
632
+<summary><strong>Step 2: Generate an API Key</strong></summary>
633
+<br/>
634
+
635
+The API key is used to authenticate the connection between parent and child nodes.
636
+
637
+1. On the parent node, generate a UUID to use as your API key:
638
+
639
+ ```bash
640
+ uuidgen
641
+ ```
642
+
643
+2. If the command isn't available, you can use an online UUID generator or create one with:
644
+
645
+ ```bash
646
+ cat /proc/sys/kernel/random/uuid
647
+ ```
648
+
649
+3. Copy the generated UUID (it should look like `11111111-2222-3333-4444-555555555555`)
650
+
651
+</details>
652
+
653
+<details>
654
+<summary><strong>Step 3: Configure the Parent Node</strong></summary>
655
+<br/>
656
+
657
+The parent node receives and stores metrics from child nodes.
658
+
659
+1. Open the stream configuration file for editing:
660
+
661
+ ```bash
662
+ cd /etc/netdata
663
+ sudo ./edit-config stream.conf
664
+ ```
665
+
666
+2. Add a section for your API key (replace with your actual UUID):
667
+
668
+ ```ini
669
+ [11111111-2222-3333-4444-555555555555]
670
+ enabled = yes
671
+ allow from = *
672
+ ```
673
+
674
+3. Save and close the file
675
+
676
+4. Restart Netdata to apply changes:
677
+
678
+ ```bash
679
+ sudo systemctl restart netdata
680
+ ```
681
+
682
+:::tip
683
+
684
+**Deployment Strategy**
685
+For critical environments, consider setting up at least two parent nodes for redundancy. Each parent should have enough disk space for your required retention period.
686
+
687
+:::
688
+
689
+</details>
690
+
691
+<details>
692
+<summary><strong>Step 4: Configure the Child Node</strong></summary>
693
+<br/>
694
+
695
+The child node streams its metrics to the parent node.
696
+
697
+1. Open the stream configuration file on the child node:
698
+
699
+ ```bash
700
+ cd /etc/netdata
701
+ sudo ./edit-config stream.conf
702
+ ```
703
+
704
+2. Find the `[stream]` section and update it (replace PARENT_IP with your parent's actual IP address):
705
+
706
+ ```ini
707
+ [stream]
708
+ enabled = yes
709
+ destination = PARENT_IP:19999
710
+ api key = 11111111-2222-3333-4444-555555555555
711
+ ```
712
+
713
+3. Save and close the file
714
+
715
+4. Restart Netdata on the child node:
716
+
717
+ ```bash
718
+ sudo systemctl restart netdata
719
+ ```
720
+
721
+:::tip
722
+
723
+**Security**
724
+
725
+For production environments, enable SSL by adding `:SSL` to your destination. This encrypts the metric data in transit.
726
+
727
+:::
728
+
729
+</details>
730
+
731
+<details>
732
+<summary><strong>Step 5: Verify the Connection</strong></summary>
733
+<br/>
734
+
735
+Check that streaming is working properly between your nodes.
736
+
737
+1. Check the Netdata logs on the parent node:
738
+
739
+ ```bash
740
+ tail -f /var/log/netdata/error.log | grep STREAM
741
+ ```
742
+
743
+2. You should see connection messages similar to:
744
+
745
+ ```
746
+ STREAM [receive from [CHILD_IP]]: new client connection.
747
+ STREAM xxx [CHILD_IP]: receive thread created (task id xxxxx)
748
+ ```
749
+
750
+3. On the child node, you should see:
751
+
752
+ ```
753
+ STREAM xxx [send to PARENT_IP:19999]: connecting...
754
+ STREAM xxx [send to PARENT_IP:19999]: established communication - sending metrics...
755
+ ```
756
+
757
+4. Open the Netdata dashboard on the parent node (http://PARENT_IP:19999) and look for the child node's hostname in the menu
758
+
759
+:::tip
760
+
761
+**Performance**
762
+Monitor the connection logs for the first few hours to ensure there are no buffer overflow issues or frequent disconnections.
763
+
764
+:::
765
+
766
+</details>
767
+
768
+<details>
769
+<summary><strong>Step 6: Customize Your Setup (Optional)</strong></summary>
770
+<br/>
771
+
772
+Now that you have a working basic setup, you can customize it based on your deployment strategy:
773
+
774
+### To Filter Which Metrics Are Streamed (Optimize Performance)
775
+
776
+Add the following to the child's `[stream]` section:
777
+
778
+```ini
779
+[stream]
780
+ # Only send system and disk metrics, but not uptime
781
+ send charts matching = system.* disk.* !system.uptime
782
```
334
-file descriptor given is not a valid stream
335
-```
783
337
-After logging this error, Netdata will close the stream.
784
+### To Enable SSL Encryption (Security Enhancement)
785
+
786
+1. On the child node, update the destination to include SSL:
787
+
788
+ ```ini
789
+ [stream]
790
+ destination = PARENT_IP:19999:SSL
791
+ ```
792
+
793
+2. If using self-signed certificates, you may need to add:
794
+
795
+ ```ini
796
+ [stream]
797
+ ssl skip certificate verification = yes
798
+ ```
799
+
800
+### To Set Up Multiple Parents for High Availability (Redundancy Strategy)
801
+
802
+1. Configure multiple destinations on the child:
803
+
804
+ ```ini
805
+ [stream]
806
+ destination = PARENT1_IP:19999 PARENT2_IP:19999
807
+ ```
808
+
809
+2. The child will connect to the first available parent and automatically switch if that connection fails
810
+
811
+### Optimizing Data Retention (Storage Strategy)
812
+
813
+On the parent node, you can [configure retention settings](/src/database/CONFIGURATION.md#retention-settings) to control how long metrics are stored.
814
+
815
+:::tip
816
+
817
+**Advanced**
818
+For large-scale deployments, consider setting up parent-to-parent streaming to create a hierarchical architecture that balances local responsiveness with centralized monitoring.
819
+
820
+:::
821
+
822
+</details>
823
+
824
+## Quick Troubleshooting Tips
825
+
826
+<details>
827
+<summary><strong>Connection Problems</strong></summary>
828
+<br/>
829
+
830
+Verify that:
831
+
832
+- Firewalls allow traffic on port 19999
833
+- Both Netdata instances are running
834
+- The API key matches exactly on both systems
835
+- The parent IP address is correct
836
+
837
+</details>
838
+
839
+<details>
840
+<summary><strong>Metrics Not Appearing</strong></summary>
841
+<br/>
842
+
843
+Verify that:
844
+
845
+- The connection is established (check logs)
846
+- The child node hasn't been excluded with `allow from` settings
847
+- The metrics aren't being filtered out with `send charts matching`
848
+
849
+</details>
850
+
851
+<details>
852
+<summary><strong>SSL/TLS Issues</strong></summary>
853
+<br/>
854
+
855
+If you're using SSL encryption:
856
+
857
+- Make sure both `:SSL` is added to the destination on the child node
858
+- Set `ssl skip certificate verification = yes` if using self-signed certificates
859
+- Check that certificate paths are correct if using custom certificates
860
+
861
+</details>
862
+
863
+<details>
864
+<summary><strong>Performance Problems</strong></summary>
865
+<br/>
866
+
867
+If streaming is slow or unstable:
868
+
869
+- Increase the buffer size: `buffer size bytes = 20971520` (20MB)
870
+- Check network quality between nodes
871
+- Consider streaming fewer metrics with `send charts matching`
872
+
873
+</details>