Update change-metrics-storage.md (#14726)
Provide better calculation of RAM usage
Chris Akritidis committed
Mar 14, 2023 at 11:05 UTC
4263a234d679122cbbc5b7eb1049276c6b7fbfd4
1 file changed
+51
-18
docs/store/change-metrics-storage.md
+51
-18
@@ -1,13 +1,3 @@
1
-<!--
2
-title: "Change how long Netdata stores metrics"
3
-description: "With a single configuration change, the Netdata Agent can store days, weeks, or months of metrics at its famous per-second granularity."
4
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/store/change-metrics-storage.md"
5
-sidebar_label: "Change how long Netdata stores metrics"
6
-learn_status: "Published"
7
-learn_topic_type: "Tasks"
8
-learn_rel_path: "Configuration"
9
--->
10
-
1
# Change how long Netdata stores metrics
2
3
The Netdata Agent uses a custom made time-series database (TSDB), named the
@@ -86,23 +76,67 @@ numbers should not deviate significantly from the above.
76
77
### Memory for concurrently collected metrics
78
89
-DBENGINE memory is related to the number of metrics concurrently being collected, the retention of the metrics
79
+The total memory Netdata uses is heavily influenced by the memory consumed by the DBENGINE.
80
+The DBENGINE memory is related to the number of metrics concurrently being collected, the retention of the metrics
81
on disk in relation with the queries running, and the number of metrics for which retention is maintained.
82
92
-The precise analysis of how much memory will be used is described in
93
-[dbengine memory requirements](https://github.com/netdata/netdata/blob/master/database/engine/README.md#memory-requirements).
83
+The precise analysis of how much memory will be used by the DBENGINE itself is described in
84
+[DBENGINE memory requirements](https://github.com/netdata/netdata/blob/master/database/engine/README.md#memory-requirements).
85
+
86
+In addition to the DBENGINE, Netdata uses memory for contexts, metric labels (e.g. in a Kubernetes setup),
87
+other Netdata structures/processes (e.g. Health) and system overhead.
88
95
-The quick rule of thumb for a high level estimation is
89
+The quick rule of thumb, for a high level estimation is
90
91
```
98
-memory in KiB = METRICS x (TIERS - 1) x 4KiB x 2 + 32768 KiB
92
+DBENGINE memory in MiB = METRICS x (TIERS - 1) x 8 / 1024 MiB
93
+Total Netdata memory in MiB = Metric cardinality factor x DBENGINE memory in MiB + "dbengine page cache size MB" from netdata.conf
94
```
95
+The cardinality factor is usually between 3 or 4 and depends mainly on the ephemerality of the collected metrics. The more ephemeral
96
+the infrastructure, the higher the factor. If the cardinality is extremely high with a lot of extremely short lived containers
97
+(hundreds started every minute), the multiplication factor can get really high. In such cases, we recommend splitting the load across
98
+multiple Netdata parents, until we can provide a way to lower the cardinality by aggregating similar metrics.
99
+
100
+#### Small agent RAM usage
101
101
-So, for 2000 metrics (dimensions) in 3 storage tiers:
102
+For 2000 metrics (dimensions) in 3 storage tiers and the default cache size:
103
104
```
104
-memory for 2k metrics = 2000 x (3 - 1) x 4 KiB x 2 + 32768 KiB = 64 MiB
105
+DBENGINE memory for 2k metrics = 2000 x (3 - 1) x 8 / 1024 MiB = 32 MiB
106
+dbengine page cache size MB = 32 MiB
107
+Total Netdata memory in MiB = Between 2*32 + 32 = 96 MiB and 3*32 + 32 = 196 MiB, for low to average cardinality
108
```
109
+#### Large parent RAM usage
110
+
111
+The Netdata parent in our production infrastructure at the time of writing:
112
+ - Collects 206k metrics per second, most from children streaming data
113
+ - The metrics include moderately ephemeral Kubernetes containers (average ephemerality), leading to a cardinality factor of about 4
114
+ - 3 tiers are used for retention
115
+ - The `dbengine page cache size MB` in `netdata.conf` is configured to be 4GB
116
+
117
+The rule of thumb calculation for this set up gives us
118
+```
119
+DBENGINE memory = 206,000 x 16 / 1024 = 3 GiB
120
+Extra cache = 4 GiB
121
+Metric cardinality factor = 4
122
+Estimated total Netdata memory = 3 * 4 + 4 = 16 GiB
123
+```
124
+
125
+The actual measurement during a low usage time was the following:
126
+
127
+Purpose|RAM|Note
128
+:--- | ---: | :---
129
+DBENGINE usage | 5.9 GiB | Out of 7GB max
130
+Cardinality related memory (k8s contexts, labels, strings) | 3.4 GiB
131
+Buffer for queries | 0 GiB | Out of 0.5 GiB max, when heavily queried
132
+Other | 0.5 GiB |
133
+System overhead | 4.4 GiB | Calculated by subtracting all of the above from the total
134
+**Total Netdata memory usage** | 14.2 GiB |
135
+
136
+All the figures above except for the system memory management overhead were retrieved from Netdata itself.
137
+The overhead can't be directly calculated, so we subtracted all the other figures from the total Netdata memory usage to get it.
138
+This overhead is usually around 50% of the memory actually useable by Netdata, but could range from 20% in small
139
+setups, all the way to 100% in some edge cases.
140
141
## Configure metric retention
142
@@ -114,4 +148,3 @@ Save the file and restart the Agent with `sudo systemctl restart netdata`, or
148
the [appropriate method](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md)
149
for your system, to change the database engine's size.
150
117
-