streaming routing documentation (#20743)
Co-authored-by: Fotis Voutsas <fotis@netdata.cloud> Co-authored-by: ilyam8 <ilya@netdata.cloud>
Costa Tsaousis committed
Aug 1, 2025 at 11:26 UTC
b2e118d0230acbe0b76cda353bba8be05b8113ce
2 files changed
+218
.gitignore
+3
@@ -200,3 +200,6 @@ packaging/tools/agent-events/go.mod
200
packaging/tools/agent-events/go.sum
201
202
**/.claude/settings.local.json
203
+.mcp.json
204
+.users_cache.json
205
+.channels_cache_v2.json
docs/streaming-routing.md
new
+215
@@ -0,0 +1,215 @@
1
+# Netdata Streaming Routing
2
+
3
+Streaming routing controls how Netdata child nodes connect to parent nodes when multiple parents are available. It handles three key operations: initial parent selection, connection management, and failover.
4
+
5
+:::info Prerequisites
6
+
7
+This feature requires configuring streaming in `netdata.conf`. See [Streaming Configuration](/src/streaming/README.md) for setup instructions.
8
+
9
+:::
10
+
11
+## How Streaming Routing Works
12
+
13
+### 1. Initial Parent Selection
14
+
15
+When a child node starts, it queries all configured parents simultaneously to determine the best connection:
16
+
17
+```mermaid
18
+graph LR
19
+ A[Start] --> B[Query all parents]
20
+ B --> C{Data recency<br/>delta < 1min?}
21
+ C -->|Multiple| D[Random select]
22
+ C -->|One best| E[Connect to<br/>most recent]
23
+ C -->|No data| D
24
+ D --> F[Connect]
25
+ E --> F
26
+```
27
+
28
+**How it works:**
29
+
30
+1. Child sends HTTP requests to all parents in parallel
31
+2. Each parent responds with:
32
+
33
+- Last timestamp of this child's data (if any)
34
+- Random seed for load balancing
35
+
36
+3. Child calculates time delta for each parent
37
+4. Selection based on data recency (not data amount)
38
+
39
+**Example:**
40
+
41
+```ini
42
+# In child's stream.conf
43
+[stream]
44
+ enabled = yes
45
+ destination = parent-a:19999 parent-b:19999 parent-c:19999
46
+ api key = YOUR_API_KEY
47
+```
48
+
49
+With this configuration:
50
+
51
+```
52
+Child Node startup:
53
+ │
54
+ ├─→ Parent A (has historical data) ✓ Selected (random between A & B)
55
+ ├─→ Parent B (has historical data)
56
+ └─→ Parent C (no historical data) ← Lower priority
57
+```
58
+
59
+### 2. Connection Management
60
+
61
+Once connected, the child maintains a persistent connection:
62
+
63
+- **Connection timeout**: 60 seconds (default)
64
+- **Keepalive**: Continuous streaming maintains connection
65
+- **No automatic rebalancing**: Child stays connected until failure
66
+- **Data integrity**: Historical metrics are replicated automatically after reconnection
67
+
68
+:::info Data Recovery
69
+
70
+Netdata automatically replicates missing historical data when reconnection occurs. Data is only lost if:
71
+
72
+- Child restarts during disconnection AND
73
+- Child uses `memory mode = ram` (metrics stored in memory) AND
74
+- Disconnection exceeds retention period (default: 1 hour for RAM mode)
75
+
76
+For persistent data, use `memory mode = dbengine`.
77
+
78
+:::
79
+
80
+:::warning Important
81
+
82
+Children do not automatically reconnect to their original parent after failover. This prevents connection flapping but requires manual intervention for load redistribution.
83
+
84
+:::
85
+
86
+### 3. Failover and Reconnection
87
+
88
+When the active connection fails, the child repeats the parent selection process:
89
+
90
+```mermaid
91
+graph LR
92
+ A[Failed] --> B[Wait 5-X sec]
93
+ B --> C[Query all parents]
94
+ C --> D{Select best<br/>by recency}
95
+ D --> E[Try connect]
96
+ E --> F{OK?}
97
+ F -->|No| B
98
+ F -->|Yes| G[Stream]
99
+```
100
+
101
+:::note Smart Failover
102
+
103
+Unlike traditional round-robin failover, Netdata re-evaluates all parents on each attempt. This means a child might connect to a different parent than expected if data states have changed.
104
+
105
+:::
106
+
107
+**Failover Example:**
108
+
109
+```
110
+Normal: Child → Parent A
111
+
112
+Failure: Child ✗ Parent A (connection lost)
113
+ Child → Parent B (immediate failover)
114
+
115
+Recovery: Parent A comes back online
116
+ Child → Parent B (stays connected - no automatic switch)
117
+```
118
+
119
+## Key Routing Behaviors
120
+
121
+| Behavior | Description | Impact |
122
+|----------------------------|-----------------------------------------------------------|-------------------------------------------------------------|
123
+| **Data Recency Priority** | Selects parent with most recent data (lowest time delta) | Minimizes gap in historical data |
124
+| **Parallel Parent Query** | Queries all parents simultaneously via HTTP | Fast parent selection, no sequential delays |
125
+| **Sticky Connections** | No automatic rebalancing after failover | Requires manual intervention to redistribute load |
126
+| **Smart Failover** | Re-evaluates all parents on each connection attempt | May connect to different parent based on current data state |
127
+| **Connection Persistence** | Maintains connection until failure occurs | Prevents unnecessary reconnections and data gaps |
128
+| **No Health Checks** | Doesn't proactively test parent availability | Discovers failures only when connection breaks |
129
+| **Randomized Delays** | Reconnection waits random time (5s to configured maximum) | Prevents thundering herd during mass reconnections |
130
+
131
+## Configuration Reference
132
+
133
+### Essential Parameters
134
+
135
+```ini
136
+[stream]
137
+ # Streaming targets (space-separated list)
138
+ # Order doesn't matter - selection is based on data recency
139
+ destination = parent1:19999 parent2:19999 parent3:19999
140
+
141
+ # Reconnection delay - randomized between 5 and this value (seconds)
142
+ # Default: 5, Minimum: 5
143
+ reconnect delay seconds = 5
144
+
145
+ # Initial connection timeout
146
+ timeout seconds = 60
147
+```
148
+
149
+### Multi-Tier Setup
150
+
151
+For larger deployments:
152
+
153
+```
154
+Child Nodes ──→ Parent Proxies ──→ Ultimate Parents
155
+ (forward only) (store & analyze)
156
+```
157
+
158
+Configure intermediate parents as proxies to distribute load without storage overhead.
159
+
160
+## Monitoring Streaming Status
161
+
162
+### Check Connection Status
163
+
164
+#### Using the UI
165
+
166
+The **Netdata Streaming** function (under the "Functions" tab) provides:
167
+
168
+- Comprehensive overview of all streaming connections
169
+- Status, replication completion time, and connection details
170
+- Works on both parent and child nodes:
171
+ - **On child**: Shows outgoing connections
172
+ - **On parent**: Shows incoming connections (InHops = 1 for direct children, >1 for proxied connections)
173
+
174
+#### Viewing Logs
175
+
176
+```bash
177
+# Check journal for streaming-related messages
178
+journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep stream
179
+```
180
+
181
+### Verify Parent Connectivity
182
+
183
+```bash
184
+# Test each parent
185
+nc -zv parent-a 19999
186
+nc -zv parent-b 19999
187
+```
188
+
189
+:::note Troubleshooting
190
+
191
+If a child connects to an unexpected parent, check the data retention on all parents. The child prefers parents that already have its historical data.
192
+
193
+:::
194
+
195
+## Common Scenarios
196
+
197
+| Scenario | What Happens | Why |
198
+|-------------------|-----------------------------------------|--------------------------|
199
+| Parent A fails | Child switches to Parent B | Automatic failover |
200
+| All parents fail | Child cycles through list every second | Continuous retry |
201
+| Parent A recovers | Child stays on Parent B | No automatic rebalancing |
202
+| New child starts | Randomly selects from parents with data | Load distribution |
203
+
204
+:::caution Maintenance Planning
205
+
206
+When taking a parent offline for maintenance, its children will failover to other parents and won't automatically return. Plan capacity accordingly.
207
+
208
+:::
209
+
210
+## Best Practices
211
+
212
+1. **List parents in priority order** - First parent is preferred if all equal
213
+2. **Configure at least 3 parents** - Ensures availability during maintenance
214
+3. **Monitor parent data completeness** - Affects routing decisions
215
+4. **Plan maintenance carefully** - Children won't automatically return