Split database overview and configuration reference (#19077)
Co-authored-by: ilyam8 <ilya@netdata.cloud>
Fotis Voutsas committed
Nov 25, 2024 at 16:35 UTC
f3ac2fea1a5a3f25b983d5b484985c926c398efb
14 files changed
+181
-170
docs/dashboards-and-charts/visualization-date-and-time-controls.md
+1
-1
@@ -81,7 +81,7 @@ beyond stored historical metrics, you'll see this message:
81
82
83
84
-At any time, [configure the internal TSDB's storage capacity](/src/database/README.md) to expand your
84
+At any time, [configure the internal TSDB's storage capacity](/src/database/CONFIGURATION.md) to expand your
85
depth of historical metrics.
86
87
### Timezone selector
docs/deployment-guides/deployment-strategies.md
+1
-1
@@ -193,7 +193,7 @@ We also suggest that you:
193
194
For increased security, user management and access to our latest features, tools and troubleshooting solutions.
195
196
-2. [Change how long Netdata stores metrics](/src/database/README.md#modes)
196
+2. [Change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers)
197
198
To control Netdata's memory use, when you have a lot of ephemeral metrics.
199
docs/netdata-agent/configuration/common-configuration-changes.md
+1
-1
@@ -19,7 +19,7 @@ changes reflected in those visualizations due to the way Netdata Cloud proxies m
19
20
### Increase the long-term metrics retention period
21
22
-Read our doc on [increasing long-term metrics storage](/src/database/README.md#tiers) for details.
22
+Read our doc on [increasing long-term metrics storage](/src/database/CONFIGURATION.md#tiers) for details.
23
24
## Modify alerts and notifications
25
docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md
+9
-9
@@ -22,8 +22,8 @@ The following table summarizes the effect of each optimization on the CPU, RAM a
22
| [Use streaming and replication](#use-streaming-and-replication) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
23
| [Disable unneeded plugins or collectors](#disable-unneeded-plugins-or-collectors) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
24
| [Reduce data collection frequency](#reduce-collection-frequency) | :heavy_check_mark: | | :heavy_check_mark: |
25
-| [Change how long Netdata stores metrics](/src/database/README.md#tiers) | | :heavy_check_mark: | :heavy_check_mark: |
26
-| [Use a different metric storage database](/src/database/README.md) | | :heavy_check_mark: | :heavy_check_mark: |
25
+| [Change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers) | | :heavy_check_mark: | :heavy_check_mark: |
26
+| [Use a different metric storage database](/src/database/CONFIGURATION.md) | | :heavy_check_mark: | :heavy_check_mark: |
27
| [Disable machine learning](#disable-machine-learning) | :heavy_check_mark: | | |
28
| [Use a reverse proxy](#run-netdata-behind-a-proxy) | :heavy_check_mark: | | |
29
| [Disable/lower gzip compression for the Agent dashboard](#disablelower-gzip-compression-for-the-dashboard) | :heavy_check_mark: | | |
@@ -72,8 +72,8 @@ The memory footprint of Netdata is mainly influenced by the number of metrics co
72
To estimate and control memory consumption, you can (either one or a combination of the following actions):
73
74
1. [Disable unneeded plugins or collectors](#disable-unneeded-plugins-or-collectors)
75
-2. [Change how long Netdata stores metrics](/src/database/README.md#tiers)
76
-3. [Use a different metric storage database](/src/database/README.md).
75
+2. [Change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers)
76
+3. [Use a different metric storage database](/src/database/CONFIGURATION.md#modes).
77
78
### Disk footprint and I/O
79
@@ -90,11 +90,11 @@ To optimize your disk footprint in any aspect described below, you can:
90
91
To configure retention, you can:
92
93
-1. [Change how long Netdata stores metrics](/src/database/README.md#tiers).
93
+1. [Change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers).
94
95
To control disk I/O:
96
97
-1. [Use a different metric storage database](/src/database/README.md),
97
+1. [Use a different metric storage database](/src/database/CONFIGURATION.md),
98
99
Minimize deployment impact on the production system by optimizing disk footprint:
100
@@ -123,7 +123,7 @@ in `stream.conf`). On the child nodes you should add to `netdata.conf` the follo
123
124
### Use memory mode ram for the child nodes
125
126
-See [using a different metric storage database](/src/database/README.md).
126
+See [using a different metric storage database](/src/database/README.md#modes).
127
128
## Disable unneeded plugins or collectors
129
@@ -211,11 +211,11 @@ update_every: 10
211
## Lower memory usage for metrics retention
212
213
See how
214
-to [change how long Netdata stores metrics](/src/database/README.md#tiers).
214
+to [change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers).
215
216
## Use a different metric storage database
217
218
-Consider [using a different metric storage database](/src/database/README.md)
218
+Consider [using a different metric storage database](/src/database/README.md#modes)
219
when running Netdata on IoT devices, and for children in a parent-child set up based
220
on [streaming and replication](/docs/observability-centralization-points/README.md).
221
docs/netdata-agent/configuration/optimizing-metrics-database/README.md
deleted
-3
@@ -1,3 +0,0 @@
1
-# Optimizing Metrics Database Overview
2
-
3
-This section contains documentation to help you understand how the metrics DB works, understand the key features and configure them to suit your needs.
docs/netdata-agent/sizing-netdata-agents/disk-requirements-and-retention.md
+1
-1
@@ -36,7 +36,7 @@ gantt
36
**Configuring dbengine mode and retention**:
37
38
- Enable dbengine mode: The dbengine mode is already the default, so no configuration change is necessary. For reference, the dbengine mode can be configured by setting `[db].mode` to `dbengine` in `netdata.conf`.
39
-- Adjust retention (optional): see [Change how long Netdata stores metrics](/src/database/README.md#tiers).
39
+- Adjust retention (optional): see [Change how long Netdata stores metrics](/src/database/CONFIGURATION.md#tiers).
40
41
## `ram`
42
src/daemon/config/README.md
+50
-50
@@ -64,10 +64,10 @@ Please note that your data history will be lost if you have modified `history` p
64
65
| setting | default | info |
66
|:----------------------------------:|:-------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
67
-| process scheduling policy | `keep` | See [Netdata process scheduling policy](/src/daemon/README.md#netdata-process-scheduling-policy) |
67
+| process scheduling policy | `keep` | See [Netdata process scheduling policy](/src/daemon/README.md#netdata-process-scheduling-policy) |
68
| OOM score | `0` | |
69
-| glibc malloc arena max for plugins | `1` | See [Virtual memory](/src/daemon/README.md#virtual-memory). |
70
-| glibc malloc arena max for Netdata | `1` | See [Virtual memory](/src/daemon/README.md#virtual-memory). |
69
+| glibc malloc arena max for plugins | `1` | See [Virtual memory](/src/daemon/README.md#virtual-memory). |
70
+| glibc malloc arena max for Netdata | `1` | See [Virtual memory](/src/daemon/README.md#virtual-memory). |
71
| hostname | auto-detected | The hostname of the computer running Netdata. |
72
| host access prefix | empty | This is used in docker environments where /proc, /sys, etc have to be accessed via another path. You may also have to set SYS_PTRACE capability on the docker for this work. Check [issue 43](https://github.com/netdata/netdata/issues/43). |
73
| timezone | auto-detected | The timezone retrieved from the environment variable |
@@ -76,22 +76,22 @@ Please note that your data history will be lost if you have modified `history` p
76
77
### [db] section options
78
79
-| setting | default | info |
80
-|:---------------------------------------------:|:-------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
81
-| mode | `dbengine` | `dbengine`: The default for long-term metrics storage with efficient RAM and disk usage. Can be extended with `dbengine page cache size` and `dbengine tier X retention size`. <br />`ram`: The round-robin database will be temporary and it will be lost when Netdata exits. <br />`alloc`: Similar to `ram`, but can significantly reduce memory usage, when combined with a low retention and does not support KSM. <br />`none`: Disables the database at this host, and disables health monitoring entirely, as that requires a database of metrics. Not to be used together with streaming. |
82
-| retention | `3600` | Used with `mode = ram/alloc`, not the default `mode = dbengine`. This number reflects the number of entries the `netdata` daemon will by default keep in memory for each chart dimension. Check [Memory Requirements](/src/database/README.md) for more information. |
83
-| storage tiers | `3` | The number of storage tiers you want to have in your dbengine. Check the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiering). You can have up to 5 tiers of data (including the _Tier 0_). This number ranges between 1 and 5. |
84
-| dbengine page cache size | `32MiB` | Determines the amount of RAM in MiB that is dedicated to caching for _Tier 0_ Netdata metric values. |
85
-| dbengine tier **`N`** retention size | `1GiB` | The disk space dedicated to metrics storage, per tier. Can be used in single-node environments as well. <br /> `N belongs to [1..4]` |
86
-| dbengine tier **`N`** retention time | `14d`, `3mo`, `1y`, `1y`, `1y` | The database retention, expressed in time. Can be used in single-node environments as well. <br /> `N belongs to [1..4]` |
87
-| update every | `1` | The frequency in seconds, for data collection. For more information see the [performance guide](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md). These metrics stored as _Tier 0_ data. Explore the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiering). |
88
-| dbengine tier **`N`** update every iterations | `60` | The down sampling value of each tier from the previous one. For each Tier, the greater by one Tier has N (equal to 60 by default) less data points of any metric it collects. This setting can take values from `2` up to `255`. <br /> `N belongs to [1..4]` |
89
-| dbengine tier back fill | `new` | Specifies the strategy of recreating missing data on higher database Tiers.<br /> `new`: Sees the latest point on each Tier and save new points to it only if the exact lower Tier has available points for it's observation window (`dbengine tier N update every iterations` window). <br /> `none`: No back filling is applied. <br /> `N belongs to [1..4]` |
90
-| memory deduplication (ksm) | `yes` | When set to `yes`, Netdata will offer its in-memory round robin database and the dbengine page cache to kernel same page merging (KSM) for deduplication. For more information check [Memory Deduplication - Kernel Same Page Merging - KSM](/src/database/README.md#ksm) |
91
-| cleanup obsolete charts after | `1h` | See [monitoring ephemeral containers](/src/collectors/cgroups.plugin/README.md#monitoring-ephemeral-containers), also sets the timeout for cleaning up obsolete dimensions |
92
-| gap when lost iterations above | `1` | |
93
-| cleanup orphan hosts after | `1h` | How long to wait until automatically removing from the DB a remote Netdata host (child) that is no longer sending data. |
94
-| enable zero metrics | `no` | Set to `yes` to show charts when all their metrics are zero. |
79
+| setting | default | info |
80
+|:---------------------------------------------:|:------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
81
+| mode | `dbengine` | `dbengine`: The default for long-term metrics storage with efficient RAM and disk usage. Can be extended with `dbengine page cache size` and `dbengine tier X retention size`. <br />`ram`: The round-robin database will be temporary and it will be lost when Netdata exits. <br />`alloc`: Similar to `ram`, but can significantly reduce memory usage, when combined with a low retention and does not support KSM. <br />`none`: Disables the database at this host, and disables health monitoring entirely, as that requires a database of metrics. Not to be used together with streaming. |
82
+| retention | `3600` | Used with `mode = ram/alloc`, not the default `mode = dbengine`. This number reflects the number of entries the `netdata` daemon will by default keep in memory for each chart dimension. Check [Memory Requirements](/docs/netdata-agent/sizing-netdata-agents/disk-requirements-and-retention.md) for more information. |
83
+| storage tiers | `3` | The number of storage tiers you want to have in your dbengine. Check the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiering). You can have up to 5 tiers of data (including the _Tier 0_). This number ranges between 1 and 5. |
84
+| dbengine page cache size | `32MiB` | Determines the amount of RAM in MiB that is dedicated to caching for _Tier 0_ Netdata metric values. |
85
+| dbengine tier **`N`** retention size | `1GiB` | The disk space dedicated to metrics storage, per tier. Can be used in single-node environments as well. <br /> `N belongs to [1..4]` |
86
+| dbengine tier **`N`** retention time | `14d`, `3mo`, `1y`, `1y`, `1y` | The database retention, expressed in time. Can be used in single-node environments as well. <br /> `N belongs to [1..4]` |
87
+| update every | `1` | The frequency in seconds, for data collection. For more information see the [performance guide](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md). These metrics stored as _Tier 0_ data. Explore the tiering mechanism in the [dbengine's reference](/src/database/engine/README.md#tiering). |
88
+| dbengine tier **`N`** update every iterations | `60` | The down sampling value of each tier from the previous one. For each Tier, the greater by one Tier has N (equal to 60 by default) less data points of any metric it collects. This setting can take values from `2` up to `255`. <br /> `N belongs to [1..4]` |
89
+| dbengine tier back fill | `new` | Specifies the strategy of recreating missing data on higher database Tiers.<br /> `new`: Sees the latest point on each Tier and save new points to it only if the exact lower Tier has available points for it's observation window (`dbengine tier N update every iterations` window). <br /> `none`: No back filling is applied. <br /> `N belongs to [1..4]` |
90
+| memory deduplication (ksm) | `yes` | When set to `yes`, Netdata will offer its in-memory round robin database and the dbengine page cache to kernel same page merging (KSM) for deduplication. |
91
+| cleanup obsolete charts after | `1h` | See [monitoring ephemeral containers](/src/collectors/cgroups.plugin/README.md#monitoring-ephemeral-containers), also sets the timeout for cleaning up obsolete dimensions |
92
+| gap when lost iterations above | `1` | |
93
+| cleanup orphan hosts after | `1h` | How long to wait until automatically removing from the DB a remote Netdata host (child) that is no longer sending data. |
94
+| enable zero metrics | `no` | Set to `yes` to show charts when all their metrics are zero. |
95
96
> ### Info
97
>
@@ -103,7 +103,7 @@ Please note that your data history will be lost if you have modified `history` p
103
|:-------------------:|:------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
104
| config | `/etc/netdata` | The directory configuration files are kept. |
105
| stock config | `/usr/lib/netdata/conf.d` | |
106
-| log | `/var/log/netdata` | The directory in which the [log files](/src/daemon/README.md#log-files) are kept. |
106
+| log | `/var/log/netdata` | The directory in which the [log files](/src/daemon/README.md#log-files) are kept. |
107
| web | `/usr/share/netdata/web` | The directory the web static files are kept. |
108
| cache | `/var/cache/netdata` | The directory the memory database will be stored if and when Netdata exits. Netdata will re-read the database when it will start again, to continue from the same point. |
109
| lib | `/var/lib/netdata` | Contains the alert log and the Netdata instance GUID. |
@@ -112,25 +112,25 @@ Please note that your data history will be lost if you have modified `history` p
112
| plugins | `"/usr/libexec/netdata/plugins.d" "/etc/netdata/custom-plugins.d"` | The directory plugin programs are kept. This setting supports multiple directories, space separated. If any directory path contains spaces, enclose it in single or double quotes. |
113
| health config | `/etc/netdata/health.d` | The directory containing the user alert configuration files, to override the stock configurations |
114
| stock health config | `/usr/lib/netdata/conf.d/health.d` | Contains the stock alert configuration files for each collector |
115
-| registry | `/opt/netdata/var/lib/netdata/registry` | Contains the [registry](/src/registry/README.md) database and GUID that uniquely identifies each Netdata Agent |
115
+| registry | `/opt/netdata/var/lib/netdata/registry` | Contains the [registry](/src/registry/README.md) database and GUID that uniquely identifies each Netdata Agent |
116
117
### [logs] section options
118
119
There are additional configuration options for the logs. For more info, see [Netdata Logging](/src/libnetdata/log/README.md).
120
121
-| setting | default | info |
122
-|:----------------------------------:|:-----------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
123
-| debug flags | `0x0000000000000000` | Bitmap of debug options to enable. For more information check [Tracing Options](/src/daemon/README.md#debugging). |
124
-| debug | `/var/log/netdata/debug.log` | The filename to save debug information. This file will not be created if debugging is not enabled. You can also set it to `syslog` to send the debug messages to syslog, or `off` to disable this log. For more information check [Tracing Options](/src/daemon/README.md#debugging). |
125
-| error | `/var/log/netdata/error.log` | The filename to save error messages for Netdata daemon and all plugins (`stderr` is sent here for all Netdata programs, including the plugins). You can also set it to `syslog` to send the errors to syslog, or `off` to disable this log. |
126
-| access | `/var/log/netdata/access.log` | The filename to save the log of web clients accessing Netdata charts. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. |
127
-| collector | `journal` | The filename to save the log of Netdata collectors. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
128
-| health | `journal` | The filename to save the log of Netdata health collectors. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
129
-| daemon | `journal` | The filename to save the log of Netdata daemon. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
130
-| facility | `daemon` | A facility keyword is used to specify the type of system that is logging the message. |
131
-| logs flood protection period | `1m` | Length of period during which the number of errors should not exceed the `errors to trigger flood protection`. |
132
-| logs to trigger flood protection | `1000` | Number of errors written to the log in `errors flood protection period` sec before flood protection is activated. |
133
-| level | `info` | Controls which log messages are logged, with error being the most important. Supported values: `info` and `error`. |
121
+| setting | default | info |
122
+|:--------------------------------:|:-----------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
123
+| debug flags | `0x0000000000000000` | Bitmap of debug options to enable. For more information check [Tracing Options](/src/daemon/README.md#debugging). |
124
+| debug | `/var/log/netdata/debug.log` | The filename to save debug information. This file will not be created if debugging is not enabled. You can also set it to `syslog` to send the debug messages to syslog, or `off` to disable this log. For more information check [Tracing Options](/src/daemon/README.md#debugging). |
125
+| error | `/var/log/netdata/error.log` | The filename to save error messages for Netdata daemon and all plugins (`stderr` is sent here for all Netdata programs, including the plugins). You can also set it to `syslog` to send the errors to syslog, or `off` to disable this log. |
126
+| access | `/var/log/netdata/access.log` | The filename to save the log of web clients accessing Netdata charts. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. |
127
+| collector | `journal` | The filename to save the log of Netdata collectors. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
128
+| health | `journal` | The filename to save the log of Netdata health collectors. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
129
+| daemon | `journal` | The filename to save the log of Netdata daemon. You can also set it to `syslog` to send the access log to syslog, or `off` to disable this log. Defaults to `Journal` if using systemd. |
130
+| facility | `daemon` | A facility keyword is used to specify the type of system that is logging the message. |
131
+| logs flood protection period | `1m` | Length of period during which the number of errors should not exceed the `errors to trigger flood protection`. |
132
+| logs to trigger flood protection | `1000` | Number of errors written to the log in `errors flood protection period` sec before flood protection is activated. |
133
+| level | `info` | Controls which log messages are logged, with error being the most important. Supported values: `info` and `error`. |
134
135
### [environment variables] section options
136
@@ -160,14 +160,14 @@ monitoring](/src/health/README.md).
160
161
[Alert notifications](/src/health/notifications/README.md) are configured in `health_alarm_notify.conf`.
162
163
-| setting | default | info |
164
-|:--------------------------------------:|:------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
165
-| enabled | `yes` | Set to `no` to disable all alerts and notifications |
166
-| in memory max health log entries | 1000 | Size of the alert history held in RAM |
167
-| script to execute on alarm | `/usr/libexec/netdata/plugins.d/alarm-notify.sh` | The script that sends alert notifications. Note that in versions before 1.16, the plugins.d directory may be installed in a different location in certain OSs (e.g. under `/usr/lib/netdata`). |
168
-| run at least every | `10s` | Controls how often all alert conditions should be evaluated. |
169
-| postpone alarms during hibernation for | `1m` | Prevents false alerts. May need to be increased if you get alerts during hibernation. |
170
-| health log retention | `5d` | Specifies the history of alert events (in seconds) kept in the Agent's sqlite database. |
163
+| setting | default | info |
164
+|:--------------------------------------:|:------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
165
+| enabled | `yes` | Set to `no` to disable all alerts and notifications |
166
+| in memory max health log entries | 1000 | Size of the alert history held in RAM |
167
+| script to execute on alarm | `/usr/libexec/netdata/plugins.d/alarm-notify.sh` | The script that sends alert notifications. Note that in versions before 1.16, the plugins.d directory may be installed in a different location in certain OSs (e.g. under `/usr/lib/netdata`). |
168
+| run at least every | `10s` | Controls how often all alert conditions should be evaluated. |
169
+| postpone alarms during hibernation for | `1m` | Prevents false alerts. May need to be increased if you get alerts during hibernation. |
170
+| health log retention | `5d` | Specifies the history of alert events (in seconds) kept in the Agent's sqlite database. |
171
| enabled alarms | * | Defines which alerts to load from both user and stock directories. This is a [simple pattern](/src/libnetdata/simple_pattern/README.md) list of alert or template names. Can be used to disable specific alerts. For example, `enabled alarms = !oom_kill *` will load all alerts except `oom_kill`. |
172
173
### [web] section options
@@ -183,11 +183,11 @@ under `collectors/python.d.plugin` will be disabled.
183
184
Additionally, there will be the following options:
185
186
-| setting | default | info |
187
-|:-------------------------------:|:---------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
188
-| enable running new plugins | `yes` | When set to `yes`, Netdata will enable detected plugins, even if they are not configured explicitly. Setting this to `no` will only enable plugins explicitly configured in this file with a `yes` |
189
-| check for new plugins every | 60 | The time in seconds to check for new plugins in the plugins directory. This allows having other applications dynamically creating plugins for Netdata. |
190
-| checks | `no` | This is a debugging plugin for the internal latency |
186
+| setting | default | info |
187
+|:---------------------------:|:-------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
188
+| enable running new plugins | `yes` | When set to `yes`, Netdata will enable detected plugins, even if they are not configured explicitly. Setting this to `no` will only enable plugins explicitly configured in this file with a `yes` |
189
+| check for new plugins every | 60 | The time in seconds to check for new plugins in the plugins directory. This allows having other applications dynamically creating plugins for Netdata. |
190
+| checks | `no` | This is a debugging plugin for the internal latency |
191
192
### [registry] section options
193
@@ -214,10 +214,10 @@ for all internal Netdata plugins.
214
215
External plugins will have only 2 options at `netdata.conf`:
216
217
-| setting | default | info |
218
-|:---------------:|:--------------------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
217
+| setting | default | info |
218
+|:---------------:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
219
| update every | the value of `[global].update every` setting | The frequency in seconds the plugin should collect values. For more information check the [performance guide](/docs/netdata-agent/configuration/optimize-the-netdata-agents-performance.md). |
220
-| command options | - | Additional command line options to pass to the plugin. |
220
+| command options | - | Additional command line options to pass to the plugin. |
221
222
External plugins that need additional configuration may support a dedicated file in `/etc/netdata`. Check their
223
documentation.
src/database/CONFIGURATION.md
new
+108
@@ -0,0 +1,108 @@
1
+# Database Configuration Reference
2
+
3
+This document provides details on configuring the Agent's Database. For a deeper understanding of the Database components, see the [Database overview](/src/database/README.md).
4
+
5
+## Modes
6
+
7
+Use [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) to open `netdata.conf` and set your preferred mode:
8
+
9
+```text
10
+[db]
11
+ # dbengine, ram, none
12
+ mode = dbengine
13
+```
14
+
15
+## Tiers
16
+
17
+### Retention Settings
18
+
19
+> **Important**
20
+>
21
+> In a Parent-Child setup, these settings manage the entire storage space used by the Parent for storing metrics collected both by itself and its Children.
22
+
23
+You can fine-tune retention for each tier by setting a time limit or size limit. Setting a limit to 0 disables it. This enables the following retention strategies:
24
+
25
+| Setting | Retention Behavior |
26
+|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
27
+| Size Limit = 0, Time Limit > 0 | **Time based:** data is stored for a specific duration regardless of disk usage |
28
+| Time Limit = 0, Size Limit > 0 | **Space based:** data is stored with a disk space limit, regardless of time |
29
+| Time Limit > 0, Size Limit > 0 | **Combined time and space limits:** data is deleted once it reaches either the time limit or the disk space limit, whichever comes first |
30
+
31
+You can change these limits using [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) to open `netdata.conf`:
32
+
33
+```text
34
+[db]
35
+ mode = dbengine
36
+ storage tiers = 3
37
+
38
+ # Tier 0, per second data. Set to 0 for no limit.
39
+ dbengine tier 0 retention size = 1GiB
40
+ dbengine tier 0 retention time = 14d
41
+
42
+ # Tier 1, per minute data. Set to 0 for no limit.
43
+ dbengine tier 1 retention size = 1GiB
44
+ dbengine tier 1 retention time = 3mo
45
+
46
+ # Tier 2, per hour data. Set to 0 for no limit.
47
+ dbengine tier 2 retention size = 1GiB
48
+ dbengine tier 2 retention time = 2y
49
+```
50
+
51
+### Legacy configuration
52
+
53
+<details><summary>v1.99.0 and prior</summary>
54
+
55
+Netdata prior to v2 supports the following configuration options in `netdata.conf`.
56
+They have the same defaults as the latest v2, but the unit of each value is given in the option name, not at the value.
57
+
58
+```text
59
+storage tiers = 3
60
+# Tier 0, per second data. Set to 0 for no limit.
61
+dbengine tier 0 disk space MB = 1024
62
+dbengine tier 0 retention days = 14
63
+# Tier 1, per minute data. Set to 0 for no limit.
64
+dbengine tier 1 disk space MB = 1024
65
+dbengine tier 1 retention days = 90
66
+# Tier 2, per hour data. Set to 0 for no limit.
67
+dbengine tier 2 disk space MB = 1024
68
+dbengine tier 2 retention days = 730
69
+```
70
+
71
+</details>
72
+
73
+<details><summary>v1.45.6 and prior</summary>
74
+
75
+Netdata versions prior to v1.46.0 relied on disk space-based retention.
76
+
77
+**Default Retention Limits**:
78
+
79
+| Tier | Resolution | Size Limit |
80
+|:----:|:-------------------:|:----------:|
81
+| 0 | high (per second) | 256 MB |
82
+| 1 | middle (per minute) | 128 MB |
83
+| 2 | low (per hour) | 64 GiB |
84
+
85
+You can change these limits in `netdata.conf`:
86
+
87
+```text
88
+[db]
89
+ mode = dbengine
90
+ storage tiers = 3
91
+ # Tier 0, per second data
92
+ dbengine multihost disk space MB = 256
93
+ # Tier 1, per minute data
94
+ dbengine tier 1 multihost disk space MB = 1024
95
+ # Tier 2, per hour data
96
+ dbengine tier 2 multihost disk space MB = 1024
97
+```
98
+
99
+</details>
100
+
101
+## Cache sizes
102
+
103
+There are two cache sizes that can be configured in `netdata.conf` to better optimize the Database:
104
+
105
+1. `[db].dbengine page cache size`: controls the size of the cache that keeps metric data on memory.
106
+2. `[db].dbengine extent cache size`: controls the size of the cache that keeps in memory compressed data blocks.
107
+
108
+Both of them are dynamically adjusted to use some of the total memory computed above. The configuration in `netdata.conf` allows providing additional memory to them, increasing their caching efficiency.
src/database/README.md
+4
-98
@@ -1,6 +1,6 @@
1
# Database
2
3
-Netdata stores detailed metrics at one-second granularity using its Database engine.
3
+Netdata stores detailed metrics at one-second granularity using its Database engine. This document provides an overview of the various elements of the Database, if you want to configure it, check the [configuration reference page](/src/database/CONFIGURATION.md)
4
5
## Modes
6
@@ -17,14 +17,6 @@ The default `dbengine` mode is optimized for:
17
18
For resource-constrained environments, particularly Child nodes in Centralization setups, consider using `ram`.
19
20
-Use [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) to open `netdata.conf` and set your preferred mode:
21
-
22
-```text
23
-[db]
24
- # dbengine, ram, none
25
- mode = dbengine
26
-```
27
-
20
## Tiers
21
22
Netdata offers a granular approach to data retention, allowing you to manage storage based on both **time** and **disk space**. This provides greater control and helps you optimize storage usage for your specific needs.
@@ -43,99 +35,13 @@ Netdata offers a granular approach to data retention, allowing you to manage sto
35
36
With these defaults, Netdata requires approximately 4 GiB of storage space (including metadata).
37
46
-### Retention Settings
47
-
48
-> **Important**
49
->
50
-> In a Parent-Child setup, these settings manage the entire storage space used by the Parent for storing metrics collected both by itself and its Children.
51
-
52
-You can fine-tune retention for each tier by setting a time limit or size limit. Setting a limit to 0 disables it. This enables the following retention strategies:
53
-
54
-| Setting | Retention Behavior |
55
-|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
56
-| Size Limit = 0, Time Limit > 0 | **Time based:** data is stored for a specific duration regardless of disk usage |
57
-| Time Limit = 0, Size Limit > 0 | **Space based:** data is stored with a disk space limit, regardless of time |
58
-| Time Limit > 0, Size Limit > 0 | **Combined time and space limits:** data is deleted once it reaches either the time limit or the disk space limit, whichever comes first |
59
-
60
-You can change these limits using [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config) to open `netdata.conf`:
61
-
62
-```text
63
-[db]
64
- mode = dbengine
65
- storage tiers = 3
66
-
67
- # Tier 0, per second data. Set to 0 for no limit.
68
- dbengine tier 0 retention size = 1GiB
69
- dbengine tier 0 retention time = 14d
70
-
71
- # Tier 1, per minute data. Set to 0 for no limit.
72
- dbengine tier 1 retention size = 1GiB
73
- dbengine tier 1 retention time = 3mo
74
-
75
- # Tier 2, per hour data. Set to 0 for no limit.
76
- dbengine tier 2 retention size = 1GiB
77
- dbengine tier 2 retention time = 2y
78
-```
79
-
38
### Monitoring Retention Utilization
39
40
Netdata provides a visual representation of storage utilization for both the time and space limits across all Tiers under "Netdata" -> "dbengine retention" on the dashboard. This chart shows exactly how your storage space (disk space limits) and time (time limits) are used for metric retention.
41
84
-### Legacy configuration
85
-
86
-<details><summary>v1.99.0 and prior</summary>
87
-
88
-Netdata prior to v2 supports the following configuration options in `netdata.conf`.
89
-They have the same defaults as the latest v2, but the unit of each value is given in the option name, not at the value.
90
-
91
-```text
92
-storage tiers = 3
93
-# Tier 0, per second data. Set to 0 for no limit.
94
-dbengine tier 0 disk space MB = 1024
95
-dbengine tier 0 retention days = 14
96
-# Tier 1, per minute data. Set to 0 for no limit.
97
-dbengine tier 1 disk space MB = 1024
98
-dbengine tier 1 retention days = 90
99
-# Tier 2, per hour data. Set to 0 for no limit.
100
-dbengine tier 2 disk space MB = 1024
101
-dbengine tier 2 retention days = 730
102
-```
103
-
104
-</details>
105
-
106
-<details><summary>v1.45.6 and prior</summary>
107
-
108
-Netdata versions prior to v1.46.0 relied on disk space-based retention.
109
-
110
-**Default Retention Limits**:
111
-
112
-| Tier | Resolution | Size Limit |
113
-|:----:|:-------------------:|:----------:|
114
-| 0 | high (per second) | 256 MB |
115
-| 1 | middle (per minute) | 128 MB |
116
-| 2 | low (per hour) | 64 GiB |
117
-
118
-You can change these limits in `netdata.conf`:
119
-
120
-```text
121
-[db]
122
- mode = dbengine
123
- storage tiers = 3
124
- # Tier 0, per second data
125
- dbengine multihost disk space MB = 256
126
- # Tier 1, per minute data
127
- dbengine tier 1 multihost disk space MB = 1024
128
- # Tier 2, per hour data
129
- dbengine tier 2 multihost disk space MB = 1024
130
-```
131
-
132
-</details>
133
-
42
## Cache sizes
43
136
-There are two cache sizes that can be configured in `netdata.conf` to better optimize the Database:
137
-
138
-1. `[db].dbengine page cache size`: this is the main cache that keeps metrics data into memory. When data is not found in it, the extent cache is consulted, and if not found in that too, they are loaded from the disk.
139
-2. `[db].dbengine extent cache size`: this is the compressed extent cache. It keeps in memory compressed data blocks, as they appear on disk, to avoid reading them again. Data found in the extent cache but not in the main cache have to be uncompressed to be queried.
44
+There are two cache sizes that can be used to optimize the Database:
45
141
-Both of them are dynamically adjusted to use some of the total memory computed above. The configuration in `netdata.conf` allows providing additional memory to them, increasing their caching efficiency.
46
+1. **Page cache size**: The main cache that keeps metrics data into memory. When data is not found in it, the extent cache is consulted, and if not found in that too, they are loaded from the disk.
47
+2. **Extent cache size**: The compressed extent cache. It keeps in memory compressed data blocks, as they appear on disk, to avoid reading them again. Data found in the extent cache but not in the main cache have to be uncompressed to be queried.
src/database/engine/README.md
+2
-2
@@ -115,7 +115,7 @@ When the Netdata Agent starts, during the first data collection of each metric,
115
data from lower tiers, so that the aggregation they provide will be accurate.
116
117
Configuring how the number of tiers and the disk space allocated to each tier is how you can
118
-[change how long netdata stores metrics](/src/database/README.md#tiers).
118
+[change how long netdata stores metrics](/src/database/CONFIGURATION.md#tiers).
119
120
### Data loss
121
@@ -127,7 +127,7 @@ multiple other Netdata Agents.
127
128
## Memory requirements and retention
129
130
-See [change how long netdata stores metrics](/src/database/README.md#tiers)
130
+See [change how long netdata stores metrics](/src/database/CONFIGURATION.md#tiers)
131
132
#### Exceptions
133
src/health/guides/dbengine/10min_dbengine_global_flushing_errors.md
+1
-1
@@ -9,5 +9,5 @@ faster disks. This alert is triggered in critical state when the number deleted
9
10
### Useful resources
11
12
-[Read more about Netdata DB engine](/src/database/README.md/engine)
12
+[Read more about Netdata DB engine](/src/database/engine/README.md)
13
src/health/guides/dbengine/10min_dbengine_global_flushing_warnings.md
+1
-1
@@ -11,5 +11,5 @@ This alert is triggered in warn state when the number of `dbengine` dirty pages
11
12
### Useful resources
13
14
-[Read more about Netdata DB engine](/src/database/README.md/engine)
14
+[Read more about Netdata DB engine](/src/database/engine/README.md)
15
src/health/guides/dbengine/10min_dbengine_global_fs_errors.md
+1
-1
@@ -10,5 +10,5 @@ This alert is triggered in warning state when the number of filesystem errors is
10
11
### Useful resources
12
13
-[Read more about Netdata DB engine](/src/database/README.md/engine)
13
+[Read more about Netdata DB engine](/src/database/engine/README.md)
14
src/health/guides/dbengine/10min_dbengine_global_io_errors.md
+1
-1
@@ -10,5 +10,5 @@ This alert is triggered in critical state when the number of IO errors is greate
10
11
### Useful resources
12
13
-[Read more about Netdata DB engine](/src/database/README.md/engine)
13
+[Read more about Netdata DB engine](/src/database/engine/README.md)
14