| 1 | # Netdata Streaming Routing |
| 2 | |
| 3 | Streaming routing controls how Netdata child nodes connect to parent nodes when multiple parents are available. It handles three key operations: initial parent selection, connection management, and failover. |
| 4 | |
| 5 | :::info |
| 6 | |
| 7 | This feature requires configuring streaming in `netdata.conf`. See [Streaming Configuration](/src/streaming/README.md) for setup instructions. |
| 8 | |
| 9 | ::: |
| 10 | |
| 11 | ## How Streaming Routing Works |
| 12 | |
| 13 | ### 1. Initial Parent Selection |
| 14 | |
| 15 | When a child node starts, it queries all configured parents simultaneously to determine the best connection: |
| 16 | |
| 17 | ```mermaid |
| 18 | flowchart LR |
| 19 | A("**Start**<br/>Child node startup") --> B("**Query all parents**<br/>Parallel HTTP requests") |
| 20 | B --> C("**Data recency<br/>delta < 1min?**<br/>Compare timestamps") |
| 21 | C -->|Multiple| D("**Random select**<br/>Load balancing") |
| 22 | C -->|One best| E("**Connect to<br/>most recent**<br/>Data continuity") |
| 23 | C -->|No data| D |
| 24 | D --> F("**Connect**<br/>Streaming active") |
| 25 | E --> F |
| 26 | %% Style definitions matching the reference |
| 27 | classDef alert fill: #ffeb3b, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 18px |
| 28 | classDef neutral fill: #f9f9f9, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 18px |
| 29 | classDef complete fill: #4caf50, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 18px |
| 30 | classDef database fill: #2196F3, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 18px |
| 31 | %% Apply styles |
| 32 | class A alert |
| 33 | class B neutral |
| 34 | class C neutral |
| 35 | class D database |
| 36 | class E database |
| 37 | class F complete |
| 38 | ``` |
| 39 | |
| 40 | **How it works:** |
| 41 | |
| 42 | 1. Child sends HTTP requests to all parents in parallel |
| 43 | 2. Each parent responds with: |
| 44 | - Last timestamp of this child's data (if any) |
| 45 | - Random seed for load balancing |
| 46 | 3. Child calculates time delta for each parent |
| 47 | 4. Selection based on data recency (not data amount) |
| 48 | |
| 49 | **Example:** |
| 50 | |
| 51 | ```ini |
| 52 | # In child's stream.conf |
| 53 | [stream] |
| 54 | enabled = yes |
| 55 | destination = parent-a:19999 parent-b:19999 parent-c:19999 |
| 56 | api key = YOUR_API_KEY |
| 57 | ``` |
| 58 | |
| 59 | With this configuration: |
| 60 | |
| 61 | ``` |
| 62 | Child Node startup: |
| 63 | │ |
| 64 | ├─→ Parent A (has historical data) ✓ Selected (random between A & B) |
| 65 | ├─→ Parent B (has historical data) |
| 66 | └─→ Parent C (no historical data) ← Lower priority |
| 67 | ``` |
| 68 | |
| 69 | ### 2. Connection Management |
| 70 | |
| 71 | Once connected, the child maintains a persistent connection: |
| 72 | |
| 73 | - **Connection timeout**: 60 seconds (default) |
| 74 | - **Keepalive**: Continuous streaming maintains connection |
| 75 | - **No automatic rebalancing**: Child stays connected until failure |
| 76 | - **Data integrity**: Historical metrics are replicated automatically after reconnection |
| 77 | |
| 78 | :::info |
| 79 | |
| 80 | Netdata automatically replicates missing historical data when reconnection occurs. Data is only lost if: |
| 81 | |
| 82 | - Child restarts during disconnection AND |
| 83 | - Child uses `memory mode = ram` (metrics stored in memory) AND |
| 84 | - Disconnection exceeds retention period (default: 1 hour for RAM mode) |
| 85 | |
| 86 | For persistent data, use `memory mode = dbengine`. |
| 87 | |
| 88 | ::: |
| 89 | |
| 90 | :::warning |
| 91 | |
| 92 | Children do not automatically reconnect to their original parent after failover. This prevents connection flapping but requires manual intervention for load redistribution. |
| 93 | |
| 94 | ::: |
| 95 | |
| 96 | ### 3. Failover and Reconnection |
| 97 | |
| 98 | When the active connection fails, the child repeats the parent selection process: |
| 99 | |
| 100 | ```mermaid |
| 101 | flowchart LR |
| 102 | A("**Failed**<br/>Connection lost") --> B("**Wait 5-X sec**<br/>Randomized delay") |
| 103 | B --> C("**Query all parents**<br/>Re-evaluate options") |
| 104 | C --> D("**Select best<br/>by recency**<br/>Data continuity priority") |
| 105 | D --> E("**Try connect**<br/>Attempt connection") |
| 106 | E --> F("**OK?**<br/>Connection test") |
| 107 | F -->|No| B |
| 108 | F -->|Yes| G("**Stream**<br/>Active monitoring") |
| 109 | %% Style definitions matching the reference |
| 110 | classDef alert fill: #ffeb3b, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 18px |
| 111 | classDef neutral fill: #f9f9f9, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 18px |
| 112 | classDef complete fill: #4caf50, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 18px |
| 113 | classDef database fill: #2196F3, stroke: #000000, stroke-width: 3px, color: #000000, font-size: 18px |
| 114 | %% Apply styles |
| 115 | class A alert |
| 116 | class B neutral |
| 117 | class C neutral |
| 118 | class E neutral |
| 119 | class D database |
| 120 | class F database |
| 121 | class G complete |
| 122 | ``` |
| 123 | |
| 124 | :::note |
| 125 | |
| 126 | Unlike traditional round-robin failover, Netdata re-evaluates all parents on each attempt. This means a child might connect to a different parent than expected if data states have changed. |
| 127 | |
| 128 | ::: |
| 129 | |
| 130 | **Failover Example:** |
| 131 | |
| 132 | ``` |
| 133 | Normal: Child → Parent A |
| 134 | |
| 135 | Failure: Child ✗ Parent A (connection lost) |
| 136 | Child → Parent B (immediate failover) |
| 137 | |
| 138 | Recovery: Parent A comes back online |
| 139 | Child → Parent B (stays connected - no automatic switch) |
| 140 | ``` |
| 141 | |
| 142 | ## Key Routing Behaviors |
| 143 | |
| 144 | | Behavior | Description | Impact | |
| 145 | |----------------------------|-----------------------------------------------------------|-------------------------------------------------------------| |
| 146 | | **Data Recency Priority** | Selects parent with most recent data (lowest time delta) | Minimizes gap in historical data | |
| 147 | | **Parallel Parent Query** | Queries all parents simultaneously via HTTP | Fast parent selection, no sequential delays | |
| 148 | | **Sticky Connections** | No automatic rebalancing after failover | Requires manual intervention to redistribute load | |
| 149 | | **Smart Failover** | Re-evaluates all parents on each connection attempt | May connect to different parent based on current data state | |
| 150 | | **Connection Persistence** | Maintains connection until failure occurs | Prevents unnecessary reconnections and data gaps | |
| 151 | | **No Health Checks** | Doesn't proactively test parent availability | Discovers failures only when connection breaks | |
| 152 | | **Randomized Delays** | Reconnection waits random time (5s to configured maximum) | Prevents thundering herd during mass reconnections | |
| 153 | |
| 154 | ## Configuration Reference |
| 155 | |
| 156 | ### Essential Parameters |
| 157 | |
| 158 | ```ini |
| 159 | [stream] |
| 160 | # Streaming targets (space-separated list) |
| 161 | # Order doesn't matter - selection is based on data recency |
| 162 | destination = parent1:19999 parent2:19999 parent3:19999 |
| 163 | |
| 164 | # Reconnection delay - randomized between 5 and this value (seconds) |
| 165 | # Default: 5, Minimum: 5 |
| 166 | reconnect delay seconds = 5 |
| 167 | |
| 168 | # Initial connection timeout |
| 169 | timeout seconds = 60 |
| 170 | ``` |
| 171 | |
| 172 | ### Multi-Tier Setup |
| 173 | |
| 174 | For larger deployments: |
| 175 | |
| 176 | ``` |
| 177 | Child Nodes ──→ Parent Proxies ──→ Ultimate Parents |
| 178 | (forward only) (store & analyze) |
| 179 | ``` |
| 180 | |
| 181 | Configure intermediate parents as proxies to distribute load without storage overhead. |
| 182 | |
| 183 | ## Monitoring Streaming Status |
| 184 | |
| 185 | ### Check Connection Status |
| 186 | |
| 187 | #### Using the UI |
| 188 | |
| 189 | The **Netdata Streaming** function (under the "Functions" tab) provides: |
| 190 | |
| 191 | - Comprehensive overview of all streaming connections |
| 192 | - Status, replication completion time, and connection details |
| 193 | - Works on both parent and child nodes: |
| 194 | - **On child**: Shows outgoing connections |
| 195 | - **On parent**: Shows incoming connections (InHops = 1 for direct children, >1 for proxied connections) |
| 196 | |
| 197 | #### Viewing Logs |
| 198 | |
| 199 | ```bash |
| 200 | # Check journal for streaming-related messages |
| 201 | journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep stream |
| 202 | ``` |
| 203 | |
| 204 | ### Verify Parent Connectivity |
| 205 | |
| 206 | ```bash |
| 207 | # Test each parent |
| 208 | nc -zv parent-a 19999 |
| 209 | nc -zv parent-b 19999 |
| 210 | ``` |
| 211 | |
| 212 | :::note Troubleshooting |
| 213 | |
| 214 | If a child connects to an unexpected parent, check the data retention on all parents. The child prefers parents that already have its historical data. |
| 215 | |
| 216 | ::: |
| 217 | |
| 218 | ## Common Scenarios |
| 219 | |
| 220 | | Scenario | What Happens | Why | |
| 221 | |-------------------|-----------------------------------------|--------------------------| |
| 222 | | Parent A fails | Child switches to Parent B | Automatic failover | |
| 223 | | All parents fail | Child cycles through list every second | Continuous retry | |
| 224 | | Parent A recovers | Child stays on Parent B | No automatic rebalancing | |
| 225 | | New child starts | Randomly selects from parents with data | Load distribution | |
| 226 | |
| 227 | :::caution Maintenance Planning |
| 228 | |
| 229 | When taking a parent offline for maintenance, its children will fail over to other parents and won't automatically return. Plan capacity accordingly. |
| 230 | |
| 231 | ::: |
| 232 | |
| 233 | ## Best Practices |
| 234 | |
| 235 | 1. **List parents in priority order** - First parent is preferred if all equal |
| 236 | 2. **Configure at least 3 parents** - Ensures availability during maintenance |
| 237 | 3. **Monitor parent data completeness** - Affects routing decisions |
| 238 | 4. **Plan maintenance carefully** - Children won't automatically return |