Update documentation (Replication DB) (#16816)
Co-authored-by: ilyam8 <ilya@netdata.cloud>
thiagoftsm committed
Feb 6, 2024 at 09:42 UTC
4d4459e87876e03b27cf9550ea892050afa9bc3c
1 file changed
+66
-37
src/streaming/README.md
+66
-37
@@ -30,42 +30,42 @@ node**. This file is automatically generated by Netdata the first time it is sta
30
31
#### `[stream]` section
32
33
-| Setting | Default | Description |
34
-|-------------------------------------------------|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
35
-| `enabled` | `no` | Whether this node streams metrics to any parent. Change to `yes` to enable streaming. |
36
-| [`destination`](#destination) | | A space-separated list of parent nodes to attempt to stream to, with the first available parent receiving metrics, using the following format: `[PROTOCOL:]HOST[%INTERFACE][:PORT][:SSL]`. [Read more →](#destination) |
37
-| `ssl skip certificate verification` | `yes` | If you want to accept self-signed or expired certificates, set to `yes` and uncomment. |
38
-| `CApath` | `/etc/ssl/certs/` | The directory where known certificates are found. Defaults to OpenSSL's default path. |
39
-| `CAfile` | `/etc/ssl/certs/cert.pem` | Add a parent node certificate to the list of known certificates in `CAPath`. |
40
-| `api key` | | The `API_KEY` to use as the child node. |
41
-| `timeout seconds` | `60` | The timeout to connect and send metrics to a parent. |
42
-| `default port` | `19999` | The port to use if `destination` does not specify one. |
43
-| [`send charts matching`](#send-charts-matching) | `*` | A space-separated list of [Netdata simple patterns](https://github.com/netdata/netdata/blob/master/src/libnetdata/simple_pattern/README.md) to filter which charts are streamed. [Read more →](#send-charts-matching) |
44
-| `buffer size bytes` | `10485760` | The size of the buffer to use when sending metrics. The default `10485760` equals a buffer of 10MB, which is good for 60 seconds of data. Increase this if you expect latencies higher than that. The buffer is flushed on reconnect. |
45
-| `reconnect delay seconds` | `5` | How long to wait until retrying to connect to the parent node. |
46
-| `initial clock resync iterations` | `60` | Sync the clock of charts for how many seconds when starting. |
47
-| `parent using h2o` | `no` | Set to yes if you are connecting to parent trough it's h2o webserver/port. Currently there is no reason to set this to `yes` unless you are testing the new h2o based netdata webserver. When production ready this will be set to `yes` as default. |
33
+| Setting | Default | Description |
34
+|-------------------------------------------------|---------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
35
+| `enabled` | `no` | Whether this node streams metrics to any parent. Change to `yes` to enable streaming. |
36
+| [`destination`](#destination) | | A space-separated list of parent nodes to attempt to stream to, with the first available parent receiving metrics, using the following format: `[PROTOCOL:]HOST[%INTERFACE][:PORT][:SSL]`. [Read more →](#destination) |
37
+| `ssl skip certificate verification` | `yes` | If you want to accept self-signed or expired certificates, set to `yes` and uncomment. |
38
+| `CApath` | `/etc/ssl/certs/` | The directory where known certificates are found. Defaults to OpenSSL's default path. |
39
+| `CAfile` | `/etc/ssl/certs/cert.pem` | Add a parent node certificate to the list of known certificates in `CAPath`. |
40
+| `api key` | | The `API_KEY` to use as the child node. |
41
+| `timeout seconds` | `60` | The timeout to connect and send metrics to a parent. |
42
+| `default port` | `19999` | The port to use if `destination` does not specify one. |
43
+| [`send charts matching`](#send-charts-matching) | `*` | A space-separated list of [Netdata simple patterns](https://github.com/netdata/netdata/blob/master/src/libnetdata/simple_pattern/README.md) to filter which charts are streamed. [Read more →](#send-charts-matching) |
44
+| `buffer size bytes` | `10485760` | The size of the buffer to use when sending metrics. The default `10485760` equals a buffer of 10MB, which is good for 60 seconds of data. Increase this if you expect latencies higher than that. The buffer is flushed on reconnect. |
45
+| `reconnect delay seconds` | `5` | How long to wait until retrying to connect to the parent node. |
46
+| `initial clock resync iterations` | `60` | Sync the clock of charts for how many seconds when starting. |
47
+| `parent using h2o` | `no` | Set to yes if you are connecting to parent trough it's h2o webserver/port. Currently there is no reason to set this to `yes` unless you are testing the new h2o based netdata webserver. When production ready this will be set to `yes` as default. |
48
49
### `[API_KEY]` and `[MACHINE_GUID]` sections
50
51
-| Setting | Default | Description |
52
-|-----------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
53
-| `enabled` | `no` | Whether this API KEY enabled or disabled. |
51
+| Setting | Default | Description |
52
+|-----------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
53
+| `enabled` | `no` | Whether this API KEY enabled or disabled. |
54
| [`allow from`](#allow-from) | `*` | A space-separated list of [Netdata simple patterns](https://github.com/netdata/netdata/blob/master/src/libnetdata/simple_pattern/README.md) matching the IPs of nodes that will stream metrics using this API key. [Read more →](#allow-from) |
55
-| `default history` | `3600` | The default amount of child metrics history to retain when using the `ram` memory mode. |
55
+| `default history` | `3600` | The default amount of child metrics history to retain when using the `ram` memory mode. |
56
| [`default memory mode`](#default-memory-mode) | `ram` | The [database](https://github.com/netdata/netdata/blob/master/src/database/README.md) to use for all nodes using this `API_KEY`. Valid settings are `dbengine`, `ram`, or `none`. [Read more →](#default-memory-mode) |
57
-| `health enabled by default` | `auto` | Whether alerts and notifications should be enabled for nodes using this `API_KEY`. `auto` enables alerts when the child is connected. `yes` enables alerts always, and `no` disables alerts. |
58
-| `default postpone alarms on connect seconds` | `60` | Postpone alerts and notifications for a period of time after the child connects. |
59
-| `default health log history` | `432000` | History of health log events (in seconds) kept in the database. |
60
-| `default proxy enabled` | | Route metrics through a proxy. |
61
-| `default proxy destination` | | Space-separated list of `IP:PORT` for proxies. |
62
-| `default proxy api key` | | The `API_KEY` of the proxy. |
63
-| `default send charts matching` | `*` | See [`send charts matching`](#send-charts-matching). |
64
-| `enable compression` | `yes` | Enable/disable stream compression. |
65
-| `enable replication` | `yes` | Enable/disable replication. |
66
-| `seconds to replicate` | `86400` | How many seconds of data to replicate from each child at a time |
67
-| `seconds per replication step` | `600` | The duration we want to replicate per each replication step. |
68
-| `is ephemeral node` | `no` | Indicate whether this child is an ephemeral node. An ephemeral node will become unavailable after the specified duration of "cleanup ephemeral hosts after secs" from the time of the node's last connection. |
57
+| `health enabled by default` | `auto` | Whether alerts and notifications should be enabled for nodes using this `API_KEY`. `auto` enables alerts when the child is connected. `yes` enables alerts always, and `no` disables alerts. |
58
+| `default postpone alarms on connect seconds` | `60` | Postpone alerts and notifications for a period of time after the child connects. |
59
+| `default health log history` | `432000` | History of health log events (in seconds) kept in the database. |
60
+| `default proxy enabled` | | Route metrics through a proxy. |
61
+| `default proxy destination` | | Space-separated list of `IP:PORT` for proxies. |
62
+| `default proxy api key` | | The `API_KEY` of the proxy. |
63
+| `default send charts matching` | `*` | See [`send charts matching`](#send-charts-matching). |
64
+| `enable compression` | `yes` | Enable/disable stream compression. |
65
+| `enable replication` | `yes` | Enable/disable replication. |
66
+| `seconds to replicate` | `86400` | How many seconds of data to replicate from each child at a time |
67
+| `seconds per replication step` | `600` | The duration we want to replicate per each replication step. |
68
+| `is ephemeral node` | `no` | Indicate whether this child is an ephemeral node. An ephemeral node will become unavailable after the specified duration of "cleanup ephemeral hosts after secs" from the time of the node's last connection. |
69
70
#### `destination`
71
@@ -148,13 +148,13 @@ cache size` and `dbengine multihost disk space` settings in the `[global]` secti
148
149
### `netdata.conf`
150
151
-| Setting | Default | Description |
152
-|--------------------------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
153
-| `[global]` section | | |
151
+| Setting | Default | Description |
152
+|--------------------------------------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
153
+| `[global]` section | | |
154
| `memory mode` | `dbengine` | Determines the [database type](https://github.com/netdata/netdata/blob/master/src/database/README.md) to be used on that node. Other options settings include `none`, and `ram`. `none` disables the database at this host. This also disables alerts and notifications, as those can't run without a database. |
155
-| `[web]` section | | |
156
-| `mode` | `static-threaded` | Determines the [web server](https://github.com/netdata/netdata/blob/master/web/server/README.md) type. The other option is `none`, which disables the dashboard, API, and registry. |
157
-| `accept a streaming request every seconds` | `0` | Set a limit on how often a parent node accepts streaming requests from child nodes. `0` equals no limit. If this is set, you may see `... too busy to accept new streaming request. Will be allowed in X secs` in Netdata's `error.log`. |
155
+| `[web]` section | | |
156
+| `mode` | `static-threaded` | Determines the [web server](https://github.com/netdata/netdata/blob/master/web/server/README.md) type. The other option is `none`, which disables the dashboard, API, and registry. |
157
+| `accept a streaming request every seconds` | `0` | Set a limit on how often a parent node accepts streaming requests from child nodes. `0` equals no limit. If this is set, you may see `... too busy to accept new streaming request. Will be allowed in X secs` in Netdata's `error.log`. |
158
159
### Basic use cases
160
@@ -488,6 +488,35 @@ You can monitor the replication process in two ways:
488
1. **Netdata Monitoring**: access the Netdata Monitoring section and look for the Replication charts.
489
2. **Streaming Function**: use the Streaming function (Top) to see the replication status of children nodes. This function provides real-time insights into the replication status of each child node.
490
491
+### Replication history
492
+
493
+Replication history in [dbengine](https://github.com/netdata/netdata/blob/master/src/database/README.md) mode is limited
494
+by [Tier 0 retention](https://learn.netdata.cloud/docs/configuring/optimizing-metrics-database/change-how-long-netdata-stores-metrics#effect-of-storage-tiers-and-disk-space-on-retention):
495
+
496
+- Child instances replicate only Tier 0 data.
497
+- Parent instance calculates higher-level tiers using Tier 0 as the basis.
498
+
499
+Extend replication history by increasing Tier 0 retention.
500
+
501
+Checking Tier 0 retention:
502
+
503
+- Using a web browser:
504
+ - Navigate to `http://{CHILD_IP}:19999/api/v2/node_instances`.
505
+ - Locate the `expected_retention` value for Tier 0 of your Agent.
506
+ - Convert the value from seconds to days for a more meaningful representation.
507
+- Using `curl` and `jq`:
508
+ - Execute the following command:
509
+ ```bash
510
+ $ curl -s "http://{CHILD_IP}:19999/api/v2/node_instances" | jq '.agents[] | {nm, retention: (.db_size[0].retention / 86400 | .*100 | round/100) }'
511
+ ```
512
+ - Example output:
513
+ ```json
514
+ {
515
+ "nm": "myhost",
516
+ "retention": 12.73
517
+ }
518
+ ```
519
+
520
## Troubleshooting
521
522
Both parent and child nodes log information at `/var/log/netdata/error.log`.