Update replication documentation (#16778)
Co-authored-by: ilyam8 <ilya@netdata.cloud>
thiagoftsm committed
Jan 17, 2024 at 13:41 UTC
aae9a7ea6365aa596ef7e95721ed1d85ec337c3f
2 files changed
+41
-12
streaming/README.md
+37
-8
@@ -52,8 +52,8 @@ node**. This file is automatically generated by Netdata the first time it is sta
52
|-----------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
53
| `enabled` | `no` | Whether this API KEY enabled or disabled. |
54
| [`allow from`](#allow-from) | `*` | A space-separated list of [Netdata simple patterns](https://github.com/netdata/netdata/blob/master/libnetdata/simple_pattern/README.md) matching the IPs of nodes that will stream metrics using this API key. [Read more →](#allow-from) |
55
-| `default history` | `3600` | The default amount of child metrics history to retain when using the `ram` memory mode. |
56
-| [`default memory mode`](#default-memory-mode) | `ram` | The [database](https://github.com/netdata/netdata/blob/master/database/README.md) to use for all nodes using this `API_KEY`. Valid settings are `dbengine`, `ram`, or `none`. [Read more →](#default-memory-mode) |
55
+| `default history` | `3600` | The default amount of child metrics history to retain when using the `ram` memory mode. |
56
+| [`default memory mode`](#default-memory-mode) | `ram` | The [database](https://github.com/netdata/netdata/blob/master/database/README.md) to use for all nodes using this `API_KEY`. Valid settings are `dbengine`, `ram`, or `none`. [Read more →](#default-memory-mode) |
57
| `health enabled by default` | `auto` | Whether alerts and notifications should be enabled for nodes using this `API_KEY`. `auto` enables alerts when the child is connected. `yes` enables alerts always, and `no` disables alerts. |
58
| `default postpone alarms on connect seconds` | `60` | Postpone alerts and notifications for a period of time after the child connects. |
59
| `default health log history` | `432000` | History of health log events (in seconds) kept in the database. |
@@ -61,6 +61,11 @@ node**. This file is automatically generated by Netdata the first time it is sta
61
| `default proxy destination` | | Space-separated list of `IP:PORT` for proxies. |
62
| `default proxy api key` | | The `API_KEY` of the proxy. |
63
| `default send charts matching` | `*` | See [`send charts matching`](#send-charts-matching). |
64
+| `enable compression` | `yes` | Enable/disable stream compression. |
65
+| `enable replication` | `yes` | Enable/disable replication. |
66
+| `seconds to replicate` | `86400` | How many seconds of data to replicate from each child at a time |
67
+| `seconds per replication step` | `600` | The duration we want to replicate per each replication step. |
68
+| `is ephemeral node` | `no` | Indicate whether this child is an ephemeral node. An ephemeral node will become unavailable after the specified duration of "cleanup ephemeral hosts after secs" from the time of the node's last connection. |
69
70
#### `destination`
71
@@ -143,13 +148,13 @@ cache size` and `dbengine multihost disk space` settings in the `[global]` secti
148
149
### `netdata.conf`
150
146
-| Setting | Default | Description |
147
-|--------------------------------------------|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
148
-| `[global]` section | | |
151
+| Setting | Default | Description |
152
+|--------------------------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
153
+| `[global]` section | | |
154
| `memory mode` | `dbengine` | Determines the [database type](https://github.com/netdata/netdata/blob/master/database/README.md) to be used on that node. Other options settings include `none`, and `ram`. `none` disables the database at this host. This also disables alerts and notifications, as those can't run without a database. |
150
-| `[web]` section | | |
151
-| `mode` | `static-threaded` | Determines the [web server](https://github.com/netdata/netdata/blob/master/web/server/README.md) type. The other option is `none`, which disables the dashboard, API, and registry. |
152
-| `accept a streaming request every seconds` | `0` | Set a limit on how often a parent node accepts streaming requests from child nodes. `0` equals no limit. If this is set, you may see `... too busy to accept new streaming request. Will be allowed in X secs` in Netdata's `error.log`. |
155
+| `[web]` section | | |
156
+| `mode` | `static-threaded` | Determines the [web server](https://github.com/netdata/netdata/blob/master/web/server/README.md) type. The other option is `none`, which disables the dashboard, API, and registry. |
157
+| `accept a streaming request every seconds` | `0` | Set a limit on how often a parent node accepts streaming requests from child nodes. `0` equals no limit. If this is set, you may see `... too busy to accept new streaming request. Will be allowed in X secs` in Netdata's `error.log`. |
158
159
### Basic use cases
160
@@ -459,6 +464,30 @@ In addition, edit `netdata.conf` on each child node to disable the database and
464
enabled = no
465
```
466
467
+## Replication
468
+
469
+Netdata streaming automatically replicates data from child nodes to parent nodes, ensuring that the parent node has a complete and up-to-date view of all metrics.
470
+This replication process ensures data continuity even if child nodes temporarily disconnect.
471
+
472
+Replication is enabled by default in Netdata, but you can customize the replication behavior by modifying the `[API_KEY]` section of the `stream.conf` file. Here's an example configuration:
473
+
474
+```conf
475
+[11111111-2222-3333-4444-555555555555]
476
+ # Enable replication for all hosts using this api key. Default: yes.
477
+ enable replication = yes
478
+
479
+ # How many seconds of data to replicate from each child at a time. Default: a day (86400 seconds).
480
+ seconds to replicate = 86400
481
+
482
+ # The duration we want to replicate per each replication step. Default: 600 seconds (10 minutes).
483
+ seconds per replication step = 600
484
+```
485
+
486
+You can monitor the replication process in two ways:
487
+
488
+1. **Netdata Monitoring**: access the Netdata Monitoring section and look for the Replication charts.
489
+2. **Streaming Function**: use the Streaming function (Top) to see the replication status of children nodes. This function provides real-time insights into the replication status of each child node.
490
+
491
## Troubleshooting
492
493
Both parent and child nodes log information at `/var/log/netdata/error.log`.
streaming/stream.conf
+4
-4
@@ -181,12 +181,12 @@
181
#seconds to replicate = 86400
182
183
# The duration we want to replicate per each step.
184
- #replication_step = 600
184
+ #seconds per replication step = 600
185
186
# Indicate whether this child is an ephemeral node. An ephemeral node will become unavailable
187
# after the specified duration of "cleanup ephemeral hosts after secs" (as defined in the db section of netdata.conf)
188
# from the time of the node's last connection.
189
- #is ephemeral node = false
189
+ #is ephemeral node = no
190
191
# -----------------------------------------------------------------------------
192
# 3. PER SENDING HOST SETTINGS, ON PARENT NETDATA
@@ -257,9 +257,9 @@
257
#seconds to replicate = 86400
258
259
# The duration we want to replicate per each step.
260
- #replication_step = 600
260
+ #seconds per replication step = 600
261
262
# Indicate whether this child is an ephemeral node. An ephemeral node will become unavailable
263
# after the specified duration of "cleanup ephemeral hosts after secs" (as defined in the db section of netdata.conf)
264
# from the time of the node's last connection.
265
- #is ephemeral node = false
265
+ #is ephemeral node = no