| 1 | # Parent-Child Configuration Reference |
| 2 | |
| 3 | ## Introduction |
| 4 | |
| 5 | This guide covers Netdata's advanced streaming and replication capabilities, which allow you to build centralized observability points across your infrastructure. |
| 6 | |
| 7 | **[Streaming and replication](https://learn.netdata.cloud/docs/developer-and-contributor-corner/glossary#r)** work together to send metrics data from one Netdata Agent (Child) to another Netdata Agent (Parent). Streaming sends metrics in real-time, while replication ensures historical data is copied, as well, maintaining complete data integrity even after connection interruptions. |
| 8 | |
| 9 | :::tip |
| 10 | |
| 11 | If you're new to Netdata streaming or prefer a guided approach, [jump to our step-by-step guide](#step-by-step-setup-guide) at the end of this document. The guide will walk you through setting up a basic streaming configuration and then connecting to the comprehensive reference sections as needed. |
| 12 | |
| 13 | For a quick reference on setting up the Parent-Child relationship, see the [Configuration Examples](#complete-configuration-examples) or refer to our comprehensive [Parents: Your Centralization Points](https://learn.netdata.cloud/docs/deployment-guides/parents-your-centralization-points) documentation for more details. |
| 14 | |
| 15 | ::: |
| 16 | |
| 17 | ## Understanding Streaming Architecture |
| 18 | |
| 19 | Before diving into configuration details, it's important to understand the key concepts behind Netdata's streaming architecture: |
| 20 | |
| 21 | <details> |
| 22 | <summary><strong>Click to see how streaming and replication work</strong></summary><br/> |
| 23 | |
| 24 | ```mermaid |
| 25 | flowchart TB |
| 26 | subgraph infrastructure["Your Infrastructure"] |
| 27 | direction TB |
| 28 | C1[C1] |
| 29 | C2[C2] |
| 30 | P[P] |
| 31 | C1("**Child 1**<br/>Collects metrics") |
| 32 | C2("**Child 2**<br/>Collects metrics") |
| 33 | P("**Parent**<br/>Stores all metrics") |
| 34 | C1 -->|Streams real - time metrics| P |
| 35 | C1 -.->|Replicates historical data| P |
| 36 | C2 -->|Streams real - time metrics| P |
| 37 | C2 -.->|Replicates historical data| P |
| 38 | end |
| 39 | |
| 40 | U[U] |
| 41 | U("**You**<br/>Access unified dashboard") |
| 42 | P -->|Presents all data| U |
| 43 | classDef child fill: #e8f5e8, stroke: #27ae60, stroke-width: 2px, color: #2c3e50, rx: 10, ry: 10 |
| 44 | classDef parent fill: #f3e8ff, stroke: #9b59b6, stroke-width: 2px, color: #2c3e50, rx: 10, ry: 10 |
| 45 | classDef user fill: #fff2e8, stroke: #f39c12, stroke-width: 2px, color: #2c3e50, rx: 10, ry: 10 |
| 46 | classDef subgraphStyle fill: #f8f9fa, stroke: #6c757d, stroke-width: 2px, color: #2c3e50, rx: 15, ry: 15 |
| 47 | class C1 child |
| 48 | class C2 child |
| 49 | class P parent |
| 50 | class U user |
| 51 | class infrastructure subgraphStyle |
| 52 | ``` |
| 53 | |
| 54 | </details><br/> |
| 55 | |
| 56 | ### Parent-Child Relationship |
| 57 | |
| 58 | - **Child nodes** (data sources) collect metrics directly from systems they monitor |
| 59 | - **Parent nodes** (data centralization points) receive, store, and visualize metrics from multiple Child nodes |
| 60 | - **A single Parent can receive data from many Children**, creating a centralized monitoring hub |
| 61 | - **Child nodes maintain independence and continue collecting metrics** even if the connection to a Parent is lost |
| 62 | |
| 63 | ### Data Flow |
| 64 | |
| 65 | 1. **Collection**: Child nodes collect metrics from their local systems |
| 66 | 2. **Streaming**: Child nodes send real-time metrics to Parent nodes |
| 67 | 3. **Replication**: When a connection is established or restored, Child nodes replicate historical data to ensure Parents have complete history |
| 68 | 4. **Storage**: Parent nodes store metrics based on their configured retention policies |
| 69 | 5. **Visualization**: Users access the Parent node's dashboard to view metrics from all connected Child nodes |
| 70 | |
| 71 | ### Benefits of This Architecture |
| 72 | |
| 73 | - **Efficiency**: Distribute collection workload across multiple nodes while centralizing visualization |
| 74 | - **Resilience**: Maintain data collection even during network disruptions or Parent node failures |
| 75 | - **Scalability**: Add more Child nodes or additional Parent nodes as your infrastructure grows |
| 76 | - **Flexibility**: Configure retention, alerts, and dashboards according to your specific needs |
| 77 | |
| 78 | ### Protocol Architecture |
| 79 | |
| 80 | Netdata streaming uses a **custom binary protocol over TCP**, not HTTP/HTTPS. This is an important distinction: |
| 81 | |
| 82 | - **Custom binary protocol**: Streaming uses Netdata's own protocol designed for efficient metrics transfer, not HTTP |
| 83 | - **TLS encryption**: When you enable `:SSL` in the destination, it adds TLS encryption as a security layer on top of the custom streaming protocol (this is not HTTPS) |
| 84 | - **Port multiplexing**: The same port (19999 by default) handles both web API requests and streaming connections—the server automatically detects which protocol is being used based on the initial handshake |
| 85 | - **Destination requirement**: This is why streaming requires a Netdata server at the destination, not just any HTTP server |
| 86 | |
| 87 | ## Quick Reference |
| 88 | |
| 89 | | Task | Configuration | Example | |
| 90 | |------------------------------------------|-------------------------------------------|----------------------------------------------------------------| |
| 91 | | Enable streaming on a Child | Set `enabled = yes` in `[stream]` section | `[stream]`<br/>`enabled = yes`<br/>`destination = 192.168.1.5` | |
| 92 | | Configure a Parent to accept connections | Create an `[API_KEY]` section | `[API_KEY]`<br/>`type = api`<br/>`enabled = yes`<br/>`allow from = *` | |
| 93 | | Set up high availability | Configure multiple destinations on Child | `[stream]`<br/>`destination = parent1:19999 parent2:19999` | |
| 94 | | Filter which metrics to send | Use `send charts matching` setting | `send charts matching = system.* !system.uptime` | |
| 95 | |
| 96 | ## Configuration Overview |
| 97 | |
| 98 | Netdata's streaming capabilities are configured through two key files: |
| 99 | |
| 100 | - **`stream.conf`** – Controls streaming behavior, including Parent and Child configurations. |
| 101 | - **`netdata.conf`** – Contains global settings that can impact streaming. |
| 102 | |
| 103 | To edit these files, navigate to your Netdata configuration directory (typically `/etc/netdata`) and run: |
| 104 | |
| 105 | ```sh |
| 106 | # Edit streaming configuration |
| 107 | sudo ./edit-config stream.conf |
| 108 | |
| 109 | # Edit global Netdata settings |
| 110 | sudo ./edit-config netdata.conf |
| 111 | ``` |
| 112 | |
| 113 | ## Configuring `stream.conf` |
| 114 | |
| 115 | The `stream.conf` file has three main sections: |
| 116 | |
| 117 | 1. **`[stream]`** – With these settings, you can configure how Child nodes send metrics. |
| 118 | 2. **`[API_KEY]`** – Here you can define settings for authentication and access control between Parents and Children. |
| 119 | 3. **`[MACHINE_GUID]`** – This area lets you customize settings for specific Child nodes by their unique ID. |
| 120 | |
| 121 | ### Identifying a Node's GUID |
| 122 | |
| 123 | Each Netdata node has a unique identifier stored in: |
| 124 | |
| 125 | ```sh |
| 126 | /var/lib/netdata/registry/netdata.public.unique.id |
| 127 | ``` |
| 128 | |
| 129 | This file is generated automatically the first time Netdata starts and remains unchanged. |
| 130 | |
| 131 | ## Recommended Deployment Strategies |
| 132 | |
| 133 | For a production-ready streaming setup, consider the following best practices: |
| 134 | |
| 135 | <details> |
| 136 | <summary><strong>Click to see deployment best practices</strong></summary><br/> |
| 137 | |
| 138 | ```mermaid |
| 139 | flowchart TB |
| 140 | A[A] |
| 141 | B[B] |
| 142 | C[C] |
| 143 | D[D] |
| 144 | E[E] |
| 145 | B1[B1] |
| 146 | C1[C1] |
| 147 | D1[D1] |
| 148 | E1[E1] |
| 149 | A("**Recommended Strategies**") |
| 150 | B("Multiple Parent Nodes") |
| 151 | C("Optimized Data Retention") |
| 152 | D("Secure Communications") |
| 153 | E("Performance Monitoring") |
| 154 | B1("Improved redundancy<br/>and resilience") |
| 155 | C1("Balance storage costs<br/>and data availability") |
| 156 | D1("Enable encryption<br/>and authentication") |
| 157 | E1("Regular log and<br/>metric reviews") |
| 158 | A --> B |
| 159 | A --> C |
| 160 | A --> D |
| 161 | A --> E |
| 162 | B --> B1 |
| 163 | C --> C1 |
| 164 | D --> D1 |
| 165 | E --> E1 |
| 166 | classDef default fill: #f9f9f9, stroke: #333, stroke-width: 2px, color: #2c3e50, rx: 10, ry: 10 |
| 167 | classDef strategies fill: #e8f5e8, stroke: #27ae60, stroke-width: 2px, color: #2c3e50, rx: 10, ry: 10 |
| 168 | class A default |
| 169 | class B strategies |
| 170 | class C strategies |
| 171 | class D strategies |
| 172 | class E strategies |
| 173 | class B1 strategies |
| 174 | class C1 strategies |
| 175 | class D1 strategies |
| 176 | class E1 strategies |
| 177 | ``` |
| 178 | |
| 179 | </details><br/> |
| 180 | |
| 181 | :::tip |
| 182 | |
| 183 | ### Multiple Parent Nodes |
| 184 | |
| 185 | Setting up multiple Parent nodes creates redundancy in your monitoring infrastructure. **If one Parent fails, Child nodes can automatically switch to another available Parent.** This approach: |
| 186 | |
| 187 | - **Prevents single points of failure** in your monitoring system |
| 188 | - **Allows for maintenance** on Parent nodes without monitoring interruptions |
| 189 | - Can be **geographically distributed to reduce latency** for global deployments |
| 190 | |
| 191 | ### Optimized Data Retention |
| 192 | |
| 193 | Configure data retention settings based on your specific monitoring needs: |
| 194 | |
| 195 | - **Use longer retention periods** for critical systems and metrics |
| 196 | - **Implement tiered storage strategies** with different retention policies |
| 197 | - **Balance storage requirements with data availability** for historical analysis |
| 198 | |
| 199 | ### Secure Communications |
| 200 | |
| 201 | **Protect your metrics data** during transmission: |
| 202 | |
| 203 | - **Enable SSL/TLS encryption** for all streaming connections |
| 204 | - Implement **proper API key management** and rotation |
| 205 | - Use **IP-based restrictions** to control which nodes can connect |
| 206 | |
| 207 | ### Performance Monitoring |
| 208 | |
| 209 | Regularly evaluate the **health of your** streaming **setup**: |
| 210 | |
| 211 | - **Monitor network traffic** between Child and Parent nodes |
| 212 | - **Check for buffer overflows** or connection issues |
| 213 | - **Adjust settings** like buffer size based on actual performance |
| 214 | |
| 215 | **By following these guidelines, you can set up a scalable and reliable Netdata streaming environment.** |
| 216 | |
| 217 | ::: |
| 218 | |
| 219 | ## `stream.conf` Detailed Reference |
| 220 | |
| 221 | ### `[stream]` Section (Child Node Settings) |
| 222 | |
| 223 | With these settings, you can configure how your Child nodes send metrics to Parent nodes. |
| 224 | |
| 225 | | Setting | Default | Description | |
| 226 | |-------------------------------------------------|---------------------------|---------------------------------------------------------------------| |
| 227 | | `enabled` | `no` | Enables streaming. Set to `yes` to allow this node to send metrics. | |
| 228 | | [`destination`](#destination) | (empty) | Defines one or more Parent nodes to send data to. | |
| 229 | | `ssl skip certificate verification` | `yes` | Accepts self-signed or expired SSL certificates. | |
| 230 | | `CApath` | `/etc/ssl/certs/` | Directory for trusted SSL certificates. | |
| 231 | | `CAfile` | `/etc/ssl/certs/cert.pem` | File containing trusted certificates. | |
| 232 | | `api key` | (empty) | API key used by the Child to authenticate with the Parent. | |
| 233 | | `timeout` | `1m` | Connection timeout duration. | |
| 234 | | `default port` | `19999` | Default port for streaming if not specified in `destination`. | |
| 235 | | [`send charts matching`](#send-charts-matching) | `*` | Filters which charts are streamed. | |
| 236 | | `buffer size bytes` | `10485760` | Buffer size (10MB by default). Increase for higher latencies. | |
| 237 | | `reconnect delay` | `5s` | Time before retrying connection to the Parent. | |
| 238 | | `initial clock resync iterations` | `60` | Syncs chart clocks during startup. | |
| 239 | | `parent using h2o` | `no` | Set to `yes` if connecting to a Parent using the H2O web server. | |
| 240 | |
| 241 | ### `[API_KEY]` Section (Parent Node Authentication) |
| 242 | |
| 243 | Here you can define settings for authentication and access control between Parents and Children. |
| 244 | |
| 245 | | Setting | Default | Description | |
| 246 | |------------------------------|------------|-------------------------------------------------------------| |
| 247 | | `enabled` | `no` | Enables or disables this API key. | |
| 248 | | `type` | `api` | Defines the section as an API key configuration. | |
| 249 | | [`allow from`](#allow-from) | `*` | Specifies which Child nodes (IP addresses) can connect. | |
| 250 | | `retention` | `1h` | How long to keep Child node metrics in RAM-based storage. | |
| 251 | | [`db`](#db) | `dbengine` | Specifies the database type for this API key. | |
| 252 | | `health enabled` | `auto` | Controls alerts and notifications (`auto`, `yes`, or `no`). | |
| 253 | | `postpone alerts on connect` | `1m` | Delay alerts for a period after the Child connects. | |
| 254 | | `health log retention` | `5d` | Duration (in seconds) to keep health log events. | |
| 255 | | `proxy enabled` | (empty) | Enables routing metrics through a proxy. | |
| 256 | | `proxy destination` | (empty) | IP and port of the proxy server. | |
| 257 | | `proxy api key` | (empty) | API key for the proxy server. | |
| 258 | | `send charts matching` | `*` | Defines which charts to stream. | |
| 259 | | `enable compression` | `yes` | Enables or disables data compression. | |
| 260 | | `enable replication` | `yes` | Enables or disables data replication. | |
| 261 | | `replication period` | `1d` | Maximum time window replicated from each Child. | |
| 262 | | `replication step` | `10m` | Time interval for each replication step. | |
| 263 | | `is ephemeral node` | `no` | Marks the Child as ephemeral (removes it after inactivity). | |
| 264 | |
| 265 | ### `[MACHINE_GUID]` Section (Per-Node Customization) |
| 266 | |
| 267 | This area lets you customize settings for specific Child nodes by their unique ID. |
| 268 | |
| 269 | | Setting | Default | Description | |
| 270 | |------------------------------|------------|----------------------------------------------------------| |
| 271 | | `enabled` | `no` | Enables or disables this specific node's configuration. | |
| 272 | | `type` | `machine` | Defines the section as a machine-specific configuration. | |
| 273 | | [`allow from`](#allow-from) | `*` | Lists IP addresses allowed to stream metrics. | |
| 274 | | `retention` | `3600` | Retention period for Child metrics in RAM-based storage. | |
| 275 | | [`db`](#db) | `dbengine` | Database type for this node. | |
| 276 | | `health enabled` | `auto` | Controls alerts (`auto`, `yes`, `no`). | |
| 277 | | `postpone alerts on connect` | `1m` | Delay alerts for a period after connection. | |
| 278 | | `health log retention` | `5d` | Duration to keep health log events. | |
| 279 | | `proxy enabled` | (empty) | Routes metrics through a proxy if enabled. | |
| 280 | | `proxy destination` | (empty) | Proxy server IP and port. | |
| 281 | | `proxy api key` | (empty) | API key for the proxy. | |
| 282 | | `send charts matching` | `*` | Filters streamed charts. | |
| 283 | | `enable compression` | `yes` | Enables or disables compression. | |
| 284 | | `enable replication` | `yes` | Enables or disables replication. | |
| 285 | | `replication period` | `1d` | Maximum replication window. | |
| 286 | | `replication step` | `10m` | Time interval for each replication step. | |
| 287 | | `is ephemeral node` | `no` | Marks the node as ephemeral (removes after inactivity). | |
| 288 | |
| 289 | ### Additional Settings |
| 290 | |
| 291 | #### `destination` |
| 292 | |
| 293 | Defines Parent nodes for streaming using the format: |
| 294 | `[PROTOCOL:]HOST[%INTERFACE][:PORT][:SSL]` |
| 295 | |
| 296 | - **PROTOCOL**: `tcp`, `udp`, or `unix` (only `tcp` and `unix` are supported for Parents). |
| 297 | - **HOST**: IPv4, IPv6 (in brackets `[ ]`), hostname, or Unix domain socket path. |
| 298 | - **INTERFACE** (IPv6 only): Network interface to use. |
| 299 | - **PORT**: Port number or service name. |
| 300 | - **SSL**: Enables TLS/SSL encryption. |
| 301 | |
| 302 | Example (TCP connection with SSL to `203.0.113.0` on port `20000`): |
| 303 | |
| 304 | ```ini |
| 305 | [stream] |
| 306 | # Send metrics securely to the Parent at 203.0.113.0:20000 |
| 307 | destination = tcp:203.0.113.0:20000:SSL |
| 308 | ``` |
| 309 | |
| 310 | #### `send charts matching` |
| 311 | |
| 312 | Controls which charts are streamed. |
| 313 | |
| 314 | - `*` (default) – Streams all charts. |
| 315 | - Specific charts: |
| 316 | |
| 317 | ```ini |
| 318 | [stream] |
| 319 | # Only send CPU application charts and all system charts |
| 320 | send charts matching = apps.cpu system.* |
| 321 | ``` |
| 322 | |
| 323 | - Exclude charts using `!`: |
| 324 | |
| 325 | ```ini |
| 326 | [stream] |
| 327 | # Send all charts except CPU application charts |
| 328 | send charts matching = !apps.cpu * |
| 329 | ``` |
| 330 | |
| 331 | #### `allow from` |
| 332 | |
| 333 | Defines which Child nodes (by IP) can connect. |
| 334 | |
| 335 | - Allow a single IP: |
| 336 | |
| 337 | ```ini |
| 338 | [API_KEY] |
| 339 | # Only allow connections from 203.0.113.10 |
| 340 | allow from = 203.0.113.10 |
| 341 | ``` |
| 342 | |
| 343 | - Allow a range but exclude one: |
| 344 | |
| 345 | ```ini |
| 346 | [API_KEY] |
| 347 | # Allow all 10.*.*.* addresses except 10.1.2.3 |
| 348 | allow from = !10.1.2.3 10.* |
| 349 | ``` |
| 350 | |
| 351 | #### `db` |
| 352 | |
| 353 | Defines the database mode: |
| 354 | |
| 355 | - `dbengine` – Stores recent metrics in RAM and writes older data to disk. |
| 356 | - `ram` – Stores metrics only in RAM (lost on restart). |
| 357 | - `none` – No database. |
| 358 | |
| 359 | ```ini |
| 360 | [API_KEY] |
| 361 | # Use disk-based database for all metrics |
| 362 | db = dbengine |
| 363 | ``` |
| 364 | |
| 365 | ## `netdata.conf` Settings Affecting Streaming |
| 366 | |
| 367 | The `netdata.conf` file is the primary configuration file for the Netdata agent. The following sections can impact streaming: |
| 368 | |
| 369 | ### [global] |
| 370 | |
| 371 | This section defines global settings for the Netdata agent. |
| 372 | |
| 373 | - **hostname**: The hostname used by the agent. |
| 374 | - **memory mode**: Choose the memory mode for data collection (e.g., `ram` or `swap`). |
| 375 | - **error log file**: Path to the file where error logs are saved. |
| 376 | |
| 377 | ### [web] |
| 378 | |
| 379 | Configure the web interface settings here. |
| 380 | |
| 381 | - **bind to**: Define the network address to which Netdata binds. |
| 382 | - **port**: Set the port for the web interface (default: 19999). |
| 383 | - **disable SSL**: Set to `yes` to disable SSL support. |
| 384 | |
| 385 | ### [database] |
| 386 | |
| 387 | Manage database settings for data storage and retention. |
| 388 | |
| 389 | - **memory mode**: Choose between in-memory or disk-based storage. |
| 390 | - **data retention**: Set how long to keep historical data. |
| 391 | - **compression**: Enable or disable data compression. |
| 392 | |
| 393 | ## Complete Configuration Examples |
| 394 | |
| 395 | ### Basic Parent-Child Setup |
| 396 | |
| 397 | **Parent node configuration (stream.conf):** |
| 398 | |
| 399 | ```ini |
| 400 | # Generate a random UUID first: uuidgen |
| 401 | [11111111-2222-3333-4444-555555555555] |
| 402 | type = api |
| 403 | # Enable this API key |
| 404 | enabled = yes |
| 405 | # Allow all IPs to connect with this key |
| 406 | allow from = * |
| 407 | # Store data using dbengine for persistence |
| 408 | db = dbengine |
| 409 | ``` |
| 410 | |
| 411 | **Child node configuration (stream.conf):** |
| 412 | |
| 413 | ```ini |
| 414 | [stream] |
| 415 | # Enable streaming on this node |
| 416 | enabled = yes |
| 417 | # Connect to Parent at 192.168.1.5 port 19999 |
| 418 | destination = 192.168.1.5 |
| 419 | # Use the same API key defined on the Parent |
| 420 | api key = 11111111-2222-3333-4444-555555555555 |
| 421 | ``` |
| 422 | |
| 423 | ### High Availability Setup with Multiple Parents |
| 424 | |
| 425 | **Parent nodes configuration (stream.conf on both Parents):** |
| 426 | |
| 427 | ```ini |
| 428 | # Configuration for accepting metrics from Children |
| 429 | [11111111-2222-3333-4444-555555555555] |
| 430 | type = api |
| 431 | enabled = yes |
| 432 | allow from = * |
| 433 | db = dbengine |
| 434 | |
| 435 | # Configuration for accepting metrics from other Parents |
| 436 | [22222222-3333-4444-5555-666666666666] |
| 437 | type = api |
| 438 | enabled = yes |
| 439 | # Only allow the other Parent's IP |
| 440 | allow from = 192.168.1.5 192.168.1.6 |
| 441 | db = dbengine |
| 442 | ``` |
| 443 | |
| 444 | **First Parent node's configuration for streaming to the second Parent:** |
| 445 | |
| 446 | ```ini |
| 447 | [stream] |
| 448 | enabled = yes |
| 449 | destination = 192.168.1.6 |
| 450 | api key = 22222222-3333-4444-5555-666666666666 |
| 451 | ``` |
| 452 | |
| 453 | **Second Parent node's configuration for streaming to the first Parent:** |
| 454 | |
| 455 | ```ini |
| 456 | [stream] |
| 457 | enabled = yes |
| 458 | destination = 192.168.1.5 |
| 459 | api key = 22222222-3333-4444-5555-666666666666 |
| 460 | ``` |
| 461 | |
| 462 | **Child node configuration:** |
| 463 | |
| 464 | ```ini |
| 465 | [stream] |
| 466 | enabled = yes |
| 467 | # List both Parents for failover |
| 468 | destination = 192.168.1.5 192.168.1.6 |
| 469 | api key = 11111111-2222-3333-4444-555555555555 |
| 470 | ``` |
| 471 | |
| 472 | ## Verifying Successful Connections |
| 473 | |
| 474 | If the streaming configuration is working correctly, you'll see logs similar to the following. |
| 475 | |
| 476 | On the Parent side: |
| 477 | |
| 478 | ``` |
| 479 | 2017-03-09 09:38:52: netdata: INFO : STREAM [receive from [10.11.12.86]:38564]: new client connection. |
| 480 | 2017-03-09 09:38:52: netdata: INFO : STREAM xxx [10.11.12.86]:38564: receive thread created (task id 27721) |
| 481 | ``` |
| 482 | |
| 483 | On the Child side: |
| 484 | |
| 485 | ``` |
| 486 | 2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: connecting... |
| 487 | 2017-03-09 09:38:28: netdata: INFO : STREAM xxx [send to box:19999]: established communication - sending metrics... |
| 488 | ``` |
| 489 | |
| 490 | Both Parent and Child nodes log information in `/var/log/netdata/error.log`. |
| 491 | |
| 492 | ## Troubleshooting |
| 493 | |
| 494 | <details> |
| 495 | <summary><strong>Slow Connection Issues</strong></summary> |
| 496 | <br/> |
| 497 | |
| 498 | **Symptoms:** |
| 499 | |
| 500 | - Buffer overflow errors |
| 501 | - Connection resets |
| 502 | - Partial message errors |
| 503 | |
| 504 | **Child logs:** |
| 505 | |
| 506 | ``` |
| 507 | netdata ERROR : STREAM_SENDER[CHILD HOSTNAME] : STREAM CHILD HOSTNAME [send to PARENT IP:PARENT PORT]: too many data pending - buffer is X bytes long, Y unsent - we have sent Z bytes in total, W on this connection. Closing connection to flush the data. |
| 508 | ``` |
| 509 | |
| 510 | **Parent logs:** |
| 511 | |
| 512 | ``` |
| 513 | netdata ERROR : STREAM_PARENT[CHILD HOSTNAME,[CHILD IP]:CHILD PORT] : read failed: end of file |
| 514 | ``` |
| 515 | |
| 516 | **What's happening:** |
| 517 | Slow network connections or high-latency links can cause the streaming buffer to fill up faster than it can be transmitted. When the buffer reaches its maximum size, Netdata closes the connection to flush the pending data, then re-establishes the connection. This can lead to data gaps or inconsistencies if it happens frequently. |
| 518 | |
| 519 | **Solutions:** |
| 520 | |
| 521 | - Increase buffer size in `stream.conf`: `buffer size bytes = 20971520` (20MB) |
| 522 | - Check network bandwidth and latency between nodes |
| 523 | - Consider reducing the collection frequency on high-volume metrics |
| 524 | - If possible, place Parent nodes closer (network-wise) to Child nodes |
| 525 | |
| 526 | </details> |
| 527 | |
| 528 | <details> |
| 529 | <summary><strong>Connection Issues</strong></summary> |
| 530 | <br/> |
| 531 | |
| 532 | **Symptoms:** |
| 533 | |
| 534 | - Child can't establish a connection to a Parent |
| 535 | - Repeated reconnection attempts |
| 536 | |
| 537 | **Child logs:** |
| 538 | |
| 539 | ``` |
| 540 | ERROR : STREAM_SENDER[HOSTNAME] : Failed to connect to 'PARENT IP', port 'PARENT PORT' (errno 113, No route to host) |
| 541 | ``` |
| 542 | |
| 543 | **What's happening:** |
| 544 | This error indicates network connectivity problems between the Child and Parent nodes. It could be due to firewall rules, incorrect IP addresses, or the Parent node not running. |
| 545 | |
| 546 | **Solutions:** |
| 547 | |
| 548 | - Verify firewalls allow traffic on port 19999 (or your custom port) |
| 549 | - Check that the Parent node is running and listening on the correct interface |
| 550 | - Verify that the IP address/hostname is correct in Child's configuration |
| 551 | - Test basic connectivity with tools like `ping` or `telnet` |
| 552 | - Check network routing between the nodes |
| 553 | |
| 554 | </details> |
| 555 | |
| 556 | <details> |
| 557 | <summary><strong>Authentication and Permission Issues</strong></summary> |
| 558 | <br/> |
| 559 | |
| 560 | **Symptoms:** |
| 561 | |
| 562 | - Connection established but immediately rejected |
| 563 | - "Forbidding access" errors |
| 564 | |
| 565 | **Parent logs:** |
| 566 | |
| 567 | ``` |
| 568 | STREAM [receive from [child HOSTNAME]:child IP]: `API key 'VALUE' is not allowed`. Forbidding access. |
| 569 | ``` |
| 570 | |
| 571 | **What's happening:** |
| 572 | The Parent node is rejecting the connection because the API key doesn't match or the Child's IP address is not allowed by the `allow from` setting. |
| 573 | |
| 574 | **Solutions:** |
| 575 | |
| 576 | - Verify if the API key matches exactly between Parent and Child |
| 577 | - Check that the `allow from` setting permits the Child's IP address |
| 578 | - Ensure GUID formats are valid |
| 579 | - Check for whitespace or invisible characters in the API key |
| 580 | - Remember that API keys are case-sensitive |
| 581 | |
| 582 | </details> |
| 583 | |
| 584 | <details> |
| 585 | <summary><strong>'Is This a Netdata?' Error</strong></summary> |
| 586 | <br/> |
| 587 | |
| 588 | **Symptoms:** |
| 589 | |
| 590 | - Child tries to connect but receives an unexpected response |
| 591 | |
| 592 | **Child logs:** |
| 593 | |
| 594 | ``` |
| 595 | ERROR : STREAM_SENDER[CHILD HOSTNAME] : STREAM child HOSTNAME [send to PARENT HOSTNAME:PARENT PORT]: server is not replying properly (is it a netdata?). |
| 596 | ``` |
| 597 | |
| 598 | **What's happening:** |
| 599 | The Child node is connecting to the destination, but the server is not responding with the expected Netdata streaming protocol. This commonly occurs when there's a mismatch in SSL/TLS settings or when the destination is not a Netdata server. |
| 600 | |
| 601 | **Solutions:** |
| 602 | |
| 603 | - Check SSL settings in the destination URL (add or remove `:SSL` as needed) |
| 604 | - Verify that you're connecting to a Netdata server and not another service |
| 605 | - Ensure both nodes are running compatible Netdata versions |
| 606 | - Check if a proxy or firewall is altering the connection |
| 607 | |
| 608 | </details> |
| 609 | |
| 610 | <details> |
| 611 | <summary><strong>Stream Charts Wrong</strong></summary> |
| 612 | <br/> |
| 613 | |
| 614 | **Symptoms:** |
| 615 | |
| 616 | - Data inconsistencies between Parent and Child |
| 617 | - Gaps in metrics collection |
| 618 | |
| 619 | **What's happening:** |
| 620 | When the database settings between Parent and Child nodes don't match, it can cause inconsistencies in how data is stored and displayed. The most common cause is different memory modes or retention settings. |
| 621 | |
| 622 | **Solutions:** |
| 623 | |
| 624 | - Check for mismatch in the `[db].db` settings between the Parent and Child |
| 625 | - Ensure database retention settings are compatible |
| 626 | - Verify replication is enabled and properly configured |
| 627 | - Make sure both nodes are using the same (or compatible) database engine |
| 628 | - Check that clocks are synchronized between nodes |
| 629 | |
| 630 | </details> |
| 631 | |
| 632 | ## FAQ |
| 633 | |
| 634 | <details> |
| 635 | <summary><strong>Can I stream to multiple Parents simultaneously?</strong></summary> |
| 636 | <br/> |
| 637 | |
| 638 | No, you can't stream to multiple Parents at the same time. However, you can configure multiple destinations for failover. Your Child node will connect to the first available Parent in the list. |
| 639 | </details> |
| 640 | |
| 641 | <details> |
| 642 | <summary><strong>How does replication work with interrupted connections?</strong></summary> |
| 643 | <br/> |
| 644 | |
| 645 | When you re-establish a connection, your Child node will replicate historical data based on the `replication period` setting. This ensures your Parent has a complete history even after interruptions. |
| 646 | </details> |
| 647 | |
| 648 | <details> |
| 649 | <summary><strong>How much bandwidth does streaming use?</strong></summary> |
| 650 | <br/> |
| 651 | |
| 652 | Your streaming setup will be very efficient, especially with compression enabled. Typically, it uses about 10–20 KB/s for a moderately active node. The actual bandwidth depends on the number of metrics and collection frequency you've configured. |
| 653 | </details> |
| 654 | |
| 655 | <details> |
| 656 | <summary><strong>Can I filter which metrics are sent to the Parent?</strong></summary> |
| 657 | <br/> |
| 658 | |
| 659 | Yes, you can use the `send charts matching` setting to include or exclude specific metrics from streaming. This works with wildcard patterns, giving you precise control over what metrics are transferred. |
| 660 | </details> |
| 661 | |
| 662 | <details> |
| 663 | <summary><strong>How do I secure the streaming connection?</strong></summary> |
| 664 | <br/> |
| 665 | |
| 666 | You can enable SSL in the destination setting by adding `:SSL` at the end. Configure proper certificates using the `CAfile` and `CApath` settings for production environments to ensure your metric data is protected in transit. |
| 667 | </details> |
| 668 | |
| 669 | <details> |
| 670 | <summary><strong>Do I need to configure streaming on every Child node?</strong></summary> |
| 671 | <br/> |
| 672 | |
| 673 | Yes, you need to configure each Child node with its own streaming configuration. However, you can use configuration management tools to deploy a standard configuration across your infrastructure, making this process more efficient. |
| 674 | </details> |
| 675 | |
| 676 | ## Step-by-Step Setup Guide |
| 677 | |
| 678 | This guide will walk you through setting up Netdata streaming between nodes. **By following these sequential steps, you'll create a basic streaming configuration** that you can later customize based on your needs. |
| 679 | |
| 680 | <details> |
| 681 | <summary><strong>Step 1: Prepare Your Environment</strong></summary> |
| 682 | <br/> |
| 683 | |
| 684 | Before configuring streaming, ensure you have: |
| 685 | |
| 686 | 1. At least two Netdata instances are installed (one to act as Parent, one as Child) |
| 687 | 2. Network connectivity between the instances |
| 688 | 3. Administrative access to edit configuration files on both systems |
| 689 | |
| 690 | </details> |
| 691 | |
| 692 | <details> |
| 693 | <summary><strong>Step 2: Generate an API Key</strong></summary> |
| 694 | <br/> |
| 695 | |
| 696 | The API key is used to authenticate the connection between Parent and Child nodes. |
| 697 | |
| 698 | 1. On the Parent node, generate a UUID to use as your API key: |
| 699 | |
| 700 | ```bash |
| 701 | uuidgen |
| 702 | ``` |
| 703 | |
| 704 | 2. If the command isn't available, you can use an online UUID generator or create one with: |
| 705 | |
| 706 | ```bash |
| 707 | cat /proc/sys/kernel/random/uuid |
| 708 | ``` |
| 709 | |
| 710 | 3. Copy the generated UUID (it should look like `11111111-2222-3333-4444-555555555555`) |
| 711 | |
| 712 | </details> |
| 713 | |
| 714 | <details> |
| 715 | <summary><strong>Step 3: Configure the Parent Node</strong></summary> |
| 716 | <br/> |
| 717 | |
| 718 | The Parent node receives and stores metrics from Child nodes. |
| 719 | |
| 720 | 1. Open the stream configuration file for editing: |
| 721 | |
| 722 | ```bash |
| 723 | cd /etc/netdata |
| 724 | sudo ./edit-config stream.conf |
| 725 | ``` |
| 726 | |
| 727 | 2. Add a section for your API key (replace with your actual UUID): |
| 728 | |
| 729 | ```ini |
| 730 | [11111111-2222-3333-4444-555555555555] |
| 731 | type = api |
| 732 | enabled = yes |
| 733 | allow from = * |
| 734 | ``` |
| 735 | |
| 736 | 3. Save and close the file |
| 737 | |
| 738 | 4. Restart Netdata to apply changes: |
| 739 | |
| 740 | ```bash |
| 741 | sudo systemctl restart netdata |
| 742 | ``` |
| 743 | |
| 744 | :::tip |
| 745 | |
| 746 | **Deployment Strategy** |
| 747 | For critical environments, consider setting up at least two Parent nodes for redundancy. Each Parent should have enough disk space for your required retention period. |
| 748 | |
| 749 | ::: |
| 750 | |
| 751 | </details> |
| 752 | |
| 753 | <details> |
| 754 | <summary><strong>Step 4: Configure the Child Node</strong></summary> |
| 755 | <br/> |
| 756 | |
| 757 | The Child node streams its metrics to the Parent node. |
| 758 | |
| 759 | 1. Open the stream configuration file on the Child node: |
| 760 | |
| 761 | ```bash |
| 762 | cd /etc/netdata |
| 763 | sudo ./edit-config stream.conf |
| 764 | ``` |
| 765 | |
| 766 | 2. Find the `[stream]` section and update it (replace PARENT_IP with your Parent's actual IP address): |
| 767 | |
| 768 | ```ini |
| 769 | [stream] |
| 770 | enabled = yes |
| 771 | destination = PARENT_IP:19999 |
| 772 | api key = 11111111-2222-3333-4444-555555555555 |
| 773 | ``` |
| 774 | |
| 775 | 3. Save and close the file |
| 776 | |
| 777 | 4. Restart Netdata on the Child node: |
| 778 | |
| 779 | ```bash |
| 780 | sudo systemctl restart netdata |
| 781 | ``` |
| 782 | |
| 783 | :::tip |
| 784 | |
| 785 | **Security** |
| 786 | |
| 787 | For production environments, enable SSL by adding `:SSL` to your destination. This encrypts the metric data in transit. |
| 788 | |
| 789 | ::: |
| 790 | |
| 791 | </details> |
| 792 | |
| 793 | <details> |
| 794 | <summary><strong>Step 5: Verify the Connection</strong></summary> |
| 795 | <br/> |
| 796 | |
| 797 | Check that streaming is working properly between your nodes. |
| 798 | |
| 799 | 1. Check the Netdata logs on the Parent node: |
| 800 | |
| 801 | ```bash |
| 802 | tail -f /var/log/netdata/error.log | grep STREAM |
| 803 | ``` |
| 804 | |
| 805 | 2. You should see connection messages similar to: |
| 806 | |
| 807 | ``` |
| 808 | STREAM [receive from [CHILD_IP]]: new client connection. |
| 809 | STREAM xxx [CHILD_IP]: receive thread created (task id xxxxx) |
| 810 | ``` |
| 811 | |
| 812 | 3. On the Child node, you should see: |
| 813 | |
| 814 | ``` |
| 815 | STREAM xxx [send to PARENT_IP:19999]: connecting... |
| 816 | STREAM xxx [send to PARENT_IP:19999]: established communication - sending metrics... |
| 817 | ``` |
| 818 | |
| 819 | 4. Open the Netdata dashboard on the Parent node (http://PARENT_IP:19999) and look for the Child node's hostname in the menu |
| 820 | |
| 821 | :::tip |
| 822 | |
| 823 | **Performance** |
| 824 | Monitor the connection logs for the first few hours to ensure there are no buffer overflow issues or frequent disconnections. |
| 825 | |
| 826 | ::: |
| 827 | |
| 828 | </details> |
| 829 | |
| 830 | <details> |
| 831 | <summary><strong>Step 6: Customize Your Setup (Optional)</strong></summary> |
| 832 | <br/> |
| 833 | |
| 834 | Now that you have a working basic setup, you can customize it based on your deployment strategy: |
| 835 | |
| 836 | ### To Filter Which Metrics Are Streamed (Optimize Performance) |
| 837 | |
| 838 | Add the following to the Child's `[stream]` section: |
| 839 | |
| 840 | ```ini |
| 841 | [stream] |
| 842 | # Only send system and disk metrics, but not uptime |
| 843 | send charts matching = system.* disk.* !system.uptime |
| 844 | ``` |
| 845 | |
| 846 | ### To Enable SSL Encryption (Security Enhancement) |
| 847 | |
| 848 | 1. On the Child node, update the destination to include SSL: |
| 849 | |
| 850 | ```ini |
| 851 | [stream] |
| 852 | destination = PARENT_IP:19999:SSL |
| 853 | ``` |
| 854 | |
| 855 | 2. If using self-signed certificates, you may need to add: |
| 856 | |
| 857 | ```ini |
| 858 | [stream] |
| 859 | ssl skip certificate verification = yes |
| 860 | ``` |
| 861 | |
| 862 | ### To Set Up Multiple Parents for High Availability (Redundancy Strategy) |
| 863 | |
| 864 | 1. Configure multiple destinations on the Child: |
| 865 | |
| 866 | ```ini |
| 867 | [stream] |
| 868 | destination = PARENT1_IP:19999 PARENT2_IP:19999 |
| 869 | ``` |
| 870 | |
| 871 | 2. The Child will connect to the first available Parent and automatically switch if that connection fails |
| 872 | |
| 873 | ### Optimizing Data Retention (Storage Strategy) |
| 874 | |
| 875 | On the Parent node, you can [configure retention settings](/src/database/CONFIGURATION.md#retention-settings) to control how long metrics are stored. |
| 876 | |
| 877 | :::tip |
| 878 | |
| 879 | **Advanced** |
| 880 | For large-scale deployments, consider setting up Parent-to-Parent streaming to create a hierarchical architecture that balances local responsiveness with centralized monitoring. |
| 881 | |
| 882 | ::: |
| 883 | |
| 884 | </details> |
| 885 | |
| 886 | ## Quick Troubleshooting Tips |
| 887 | |
| 888 | <details> |
| 889 | <summary><strong>Connection Problems</strong></summary> |
| 890 | <br/> |
| 891 | |
| 892 | Verify that: |
| 893 | |
| 894 | - Firewalls allow traffic on port 19999 |
| 895 | - Both Netdata instances are running |
| 896 | - The API key matches exactly on both systems |
| 897 | - The Parent IP address is correct |
| 898 | |
| 899 | </details> |
| 900 | |
| 901 | <details> |
| 902 | <summary><strong>Metrics Not Appearing</strong></summary> |
| 903 | <br/> |
| 904 | |
| 905 | Verify that: |
| 906 | |
| 907 | - The connection is established (check logs) |
| 908 | - The Child node hasn't been excluded with `allow from` settings |
| 909 | - The metrics aren't being filtered out with `send charts matching` |
| 910 | |
| 911 | </details> |
| 912 | |
| 913 | <details> |
| 914 | <summary><strong>SSL/TLS Issues</strong></summary> |
| 915 | <br/> |
| 916 | |
| 917 | If you're using SSL encryption: |
| 918 | |
| 919 | - Make sure `:SSL` is added to the destination on the Child node |
| 920 | - Set `ssl skip certificate verification = yes` if using self-signed certificates |
| 921 | - Check that certificate paths are correct if using custom certificates |
| 922 | |
| 923 | </details> |
| 924 | |
| 925 | <details> |
| 926 | <summary><strong>Performance Problems</strong></summary> |
| 927 | <br/> |
| 928 | |
| 929 | If streaming is slow or unstable: |
| 930 | |
| 931 | - Increase the buffer size: `buffer size bytes = 20971520` (20MB) |
| 932 | - Check network quality between nodes |
| 933 | - Consider streaming fewer metrics with `send charts matching` |
| 934 | |
| 935 | </details> |