Update main metric retention docs (#14530)
* Update main metric retention docs * Fix link and typo
Chris Akritidis committed
Feb 14, 2023 at 10:33 UTC
b48ecea209154fe4b6c8ee951a487958ee1ed28b
2 files changed
+71
-108
database/engine/README.md
+4
-44
@@ -118,51 +118,11 @@ Tiers are supported in Netdata Agents with version `netdata-1.35.0.138.nightly`
118
119
Updating the higher **tiers** is automated, and it happens in real-time while data are being collected for **tier 0**.
120
121
-When the Netdata Agent starts, during the first data collection of each metric, higher tiers are automatically **backfilled** with data from lower tiers, so that the aggregation they provide will be accurate.
121
+When the Netdata Agent starts, during the first data collection of each metric, higher tiers are automatically **backfilled** with
122
+data from lower tiers, so that the aggregation they provide will be accurate.
123
123
-3 tiers are enabled by default in Netdata, with the following configuration:
124
-
125
-```
126
-[db]
127
- mode = dbengine
128
-
129
- # per second data collection
130
- update every = 1
131
-
132
- # number of tiers used (1 to 5, 3 being default)
133
- storage tiers = 3
134
-
135
- # Tier 0, per second data
136
- dbengine multihost disk space MB = 256
137
-
138
- # Tier 1, per minute data
139
- dbengine tier 1 multihost disk space MB = 128
140
-
141
- # Tier 2, per hour data
142
- dbengine tier 2 multihost disk space MB = 64
143
-```
144
-
145
-The exact retention that can be achieved by each tier depends on the number of metrics collected. The more the metrics, the smaller the retention that will fit in a given size. The general rule is that Netdata needs about **1 byte per data point on disk for tier 0**, and **4 bytes per data point on disk for tier 1 and above**.
146
-
147
-So, for 1000 metrics collected per second and 256 MB for tier 0, Netdata will store about:
148
-
149
-```
150
-256MB on disk / 1 byte per point / 1000 metrics => 256k points per metric / 86400 seconds per day = about 3 days
151
-```
152
-
153
-At tier 1 (per minute):
154
-
155
-```
156
-128MB on disk / 4 bytes per point / 1000 metrics => 32k points per metric / (24 hours * 60 minutes) = about 22 days
157
-```
158
-
159
-At tier 2 (per hour):
160
-
161
-```
162
-64MB on disk / 4 bytes per point / 1000 metrics => 16k points per metric / 24 hours per day = about 2 years
163
-```
164
-
165
-Of course double the metrics, half the retention. There are more factors that affect retention. The number of ephemeral metrics (i.e. metrics that are collected for part of the time). The number of metrics that are usually constant over time (affecting compression efficiency). The number of restarts a Netdata Agents gets through time (because it has to break pages prematurely, increasing the metadata overhead). But the actual numbers should not deviate significantly from the above.
124
+Configuring how the number of tiers and the disk space allocated to each tier is how you can
125
+[change how long netdata stores metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md).
126
127
### Data Loss
128
docs/store/change-metrics-storage.md
+67
-64
@@ -10,93 +10,96 @@ learn_rel_path: "Setup"
10
11
# Change how long Netdata stores metrics
12
13
-The Netdata Agent uses a custom made time-series database (TSDB), named the [`dbengine`](https://github.com/netdata/netdata/blob/master/database/engine/README.md), to store metrics.
13
+The Netdata Agent uses a custom made time-series database (TSDB), named the
14
+[`dbengine`](https://github.com/netdata/netdata/blob/master/database/engine/README.md), to store metrics.
15
15
-The default settings retain approximately two day's worth of metrics on a system collecting 2,000 metrics every second,
16
-but the Netdata Agent is highly configurable if you want your nodes to store days, weeks, or months worth of per-second
17
-data.
16
+To increase or decrease the metric retention time, you just [configure](#configure-metric-retention)
17
+the number of storage tiers and the space allocated to each one. The effect of these two parameters
18
+on the maximum retention and the memory used by Netdata is described in detail, below.
19
19
-The Netdata Agent uses the following three fundamental settings in `netdata.conf` to change the behavior of the database engine:
20
+## Calculate the system resources (RAM, disk space) needed to store metrics
21
+
22
+### Disk space allocated to each tier
23
+
24
+3 tiers are enabled by default in Netdata, with the following configuration:
25
21
-```conf
22
-[global]
23
- dbengine page cache size = 32
24
- dbengine multihost disk space = 256
25
- storage tiers = 1
26
+```
27
+[db]
28
+ mode = dbengine
29
+
30
+ # per second data collection
31
+ update every = 1
32
+
33
+ # number of tiers used (1 to 5, 3 being default)
34
+ storage tiers = 3
35
+
36
+ # Tier 0, per second data
37
+ dbengine multihost disk space MB = 256
38
+
39
+ # Tier 1, per minute data
40
+ dbengine tier 1 multihost disk space MB = 128
41
+
42
+ # Tier 2, per hour data
43
+ dbengine tier 2 multihost disk space MB = 64
44
```
45
28
-`dbengine page cache size` sets the maximum amount of RAM (in MiB) the database engine uses to cache and index recent
29
-metrics.
30
-`dbengine multihost disk space` sets the maximum disk space (again, in MiB) the database engine uses to store
31
-historical, compressed metrics and `storage tiers` specifies the number of storage tiers you want to have in
32
-your `dbengine`. When the size of stored metrics exceeds the allocated disk space, the database engine removes the
33
-oldest metrics on a rolling basis.
46
+The exact retention that can be achieved by each tier depends on the number of metrics collected. The more
47
+the metrics, the smaller the retention that will fit in a given size. The general rule is that Netdata needs
48
+about **1 byte per data point on disk for tier 0**, and **4 bytes per data point on disk for tier 1 and above**.
49
35
-## Calculate the system resources (RAM, disk space) needed to store metrics
50
+So, for 1000 metrics collected per second and 256 MB for tier 0, Netdata will store about:
51
37
-You can store more or less metrics using the database engine by changing the allocated disk space. Use the calculator
38
-below to find the appropriate value for the `dbengine` based on how many metrics your node(s) collect, whether you are
39
-streaming metrics to a parent node, and more.
52
+```
53
+256MB on disk / 1 byte per point / 1000 metrics => 256k points per metric / 86400 seconds per day = about 3 days
54
+```
55
41
-You do not need to edit the `dbengine page cache size` setting to store more metrics using the database engine. However,
42
-if you want to store more metrics _specifically in memory_, you can increase the cache size.
56
+At tier 1 (per minute):
57
44
-:::tip
58
+```
59
+128MB on disk / 4 bytes per point / 1000 metrics => 32k points per metric / (24 hours * 60 minutes) = about 22 days
60
+```
61
46
-We advise you to visit the [tiering mechanism](https://github.com/netdata/netdata/blob/master/database/engine/README.md#tiering) reference. This will help you
47
-configure the Agent to retain metrics for longer periods.
62
+At tier 2 (per hour):
63
49
-:::
64
+```
65
+64MB on disk / 4 bytes per point / 1000 metrics => 16k points per metric / 24 hours per day = about 2 years
66
+```
67
51
-:::caution
68
+Of course double the metrics, half the retention. There are more factors that affect retention. The number
69
+of ephemeral metrics (i.e. metrics that are collected for part of the time). The number of metrics that are
70
+usually constant over time (affecting compression efficiency). The number of restarts a Netdata Agents gets
71
+through time (because it has to break pages prematurely, increasing the metadata overhead). But the actual
72
+numbers should not deviate significantly from the above.
73
53
-This calculator provides an estimation of disk and RAM usage for **metrics usage**. Real-life usage may vary based on
54
-the accuracy of the values you enter below, changes in the compression ratio, and the types of metrics stored.
74
56
-:::
75
+### Memory for concurrently collected metrics
76
58
-Visit the [Netdata Storage Calculator](https://netdata-storage-calculator.herokuapp.com/) app to customize
59
-data retention according to your preferences.
77
+DBENGINE memory is related to the number of metrics concurrently being collected, the retention of the metrics
78
+on disk in relation with the queries running, and the number of metrics for which retention is maintained.
79
61
-## Edit `netdata.conf` with recommended database engine settings
80
+The precise analysis of how much memory will be used is described in
81
+[dbengine memory requirements](https://github.com/netdata/netdata/blob/master/database/engine/README.md#memory-requirements).
82
63
-Now that you have a recommended setting for your Agent's `dbengine`, open `netdata.conf` with
64
-[`edit-config`](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) and look for the `[db]`
65
-subsection. Change it to the recommended values you calculated from the calculator. For example:
83
+The quick rule of thumb for a high level estimation is
84
67
-```conf
68
-[db]
69
- mode = dbengine
70
- storage tiers = 3
71
- update every = 1
72
- dbengine multihost disk space MB = 1024
73
- dbengine page cache size MB = 32
74
- dbengine tier 1 update every iterations = 60
75
- dbengine tier 1 multihost disk space MB = 384
76
- dbengine tier 1 page cache size MB = 32
77
- dbengine tier 2 update every iterations = 60
78
- dbengine tier 2 multihost disk space MB = 16
79
- dbengine tier 2 page cache size MB = 32
85
+```
86
+memory in KiB = METRICS x (TIERS - 1) x 4KiB x 2 + 32768 KiB
87
```
88
82
-Save the file and restart the Agent with `sudo systemctl restart netdata`, or
83
-the [appropriate method](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md) for your system, to change the database engine's size.
84
-
85
-## What's next?
89
+So, for 2000 metrics (dimensions) in 3 storage tiers:
90
87
-If you have multiple nodes with the Netdata Agent installed, you
88
-can [stream metrics](https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/how-streaming-works.mdx) from any number of _child_ nodes to a _
89
-parent_ node and store metrics using a centralized time-series database. Streaming allows you to centralize your data,
90
-run Agents as headless collectors, replicate data, and more.
91
+```
92
+memory for 2k metrics = 2000 x (3 - 1) x 4 KiB x 2 + 32768 KiB = 64 MiB
93
+```
94
92
-Storing metrics with the database engine is completely interoperable
93
-with [exporting to other time-series databases](https://github.com/netdata/netdata/blob/master/docs/export/external-databases.md). With exporting, you can use the
94
-node's resources to surface metrics when [viewing dashboards](https://github.com/netdata/netdata/blob/master/docs/visualize/interact-dashboards-charts.md), while also
95
-archiving metrics elsewhere for further analysis, visualization, or correlation with other tools.
95
+## Configure metric retention
96
97
-### Related reference documentation
97
+Once you have decided how to size each tier, open `netdata.conf` with
98
+[`edit-config`](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files)
99
+and make your changes in the `[db]` subsection.
100
99
-- [Netdata Agent · Database engine](https://github.com/netdata/netdata/blob/master/database/engine/README.md)
100
-- [Netdata Agent · Database engine configuration option](https://github.com/netdata/netdata/blob/master/daemon/config/README.md#[db]-section-options)
101
+Save the file and restart the Agent with `sudo systemctl restart netdata`, or
102
+the [appropriate method](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md)
103
+for your system, to change the database engine's size.
104
105