e418f938c · cryptotaxi247/netdata-1

@@ -1,180 +1,88 @@

1 # Change how long Netdata stores metrics

2

3 -The Netdata Agent uses a custom made time-series database (TSDB), named the

4 -[`dbengine`](/src/database/engine/README.md), to store metrics.

3 +Netdata offers a granular approach to data retention, allowing you to manage storage based on both **time** and **disk

4 +space**. This provides greater control and helps you optimize storage usage for your specific needs.

5

6 -To see the number of metrics stored and the retention in days per tier, use the `/api/v1/dbengine_stats` endpoint.

6 +**Default Retention Limits**:

7

8 -To increase or decrease the metric retention time, you just [configure](#configure-metric-retention)

9 -the number of storage tiers and the space allocated to each one. The effect of these two parameters

10 -on the maximum retention and the memory used by Netdata is described in detail, below.

9 +|:----:|:-------------------:|:----------:|:----------:|

10 +| 0 | high (per second) | 14 days | 1 GiB |

11 +| 1 | middle (per minute) | 3 months | 1 GiB |

12 +| 2 | low (per hour) | 2 years | 1 GiB |

13

12 -## Calculate the system resources (RAM, disk space) needed to store metrics

14 +With these defaults, Netdata requires approximately 4 GiB of storage space (including metadata).

15

14 -### Effect of storage tiers and disk space on retention

16 +## Retention Settings

17

16 -3 tiers are enabled by default in Netdata, with the following configuration:

18 +> **In a parent-child setup**, these settings manage the shared storage space utilized by the Netdata parent agent for storing metrics collected by both the parent and its child nodes.

19

18 -```

19 -[db]

20 - mode = dbengine

21 -

22 - # per second data collection

23 - update every = 1

24 -

25 - # number of tiers used (1 to 5, 3 being default)

26 - storage tiers = 3

27 -

28 - # Tier 0, per second data

29 - dbengine multihost disk space MB = 256

30 -

31 - # Tier 1, per minute data

32 - dbengine tier 1 multihost disk space MB = 128

33 - dbengine tier 1 update every iterations = 60

34 -

35 - # Tier 2, per hour data

36 - dbengine tier 2 multihost disk space MB = 64

37 - dbengine tier 2 update every iterations = 60

38 -```

39 -

40 -The default "update every iterations" of 60 means that if a metric is collected per second in Tier 0, then

41 -we will have a data point every minute in tier 1 and every hour in tier 2.

20 +You can fine-tune retention for each tier by setting either the time limit or size limit to zero. This enables various

21 +retention strategies as shown in the table below:

22

43 -Up to 5 tiers are supported. You may add, or remove tiers and/or modify these multipliers, as long as the

44 -product of all the "update every iterations" does not exceed 65535 (number of points for each tier0 point).

23 +| Setting | Retention Behavior |

24 +|--------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|

25 +| Size Limit = 0, Time Limit > 0 | **Time-based only:** data is stored for a specific duration regardless of disk usage. |

26 +| Time Limit = 0, Size Limit > 0 | **Space-based only:** data is stored until it reaches a certain amount of disk space, regardless of time. |

27 +| Time Limit > 0, Size Limit > 0 | **Combined time and space limits:** data is deleted once it reaches either the time limit or the disk space limit, whichever comes first. |

28

46 -e.g. If you simply add a fourth tier by setting `storage tiers = 4` and define the disk space for the new tier,

47 -the product of the "update every iterations" will be 60 \* 60 \* 60 = 216,000, which is > 65535. So you'd need to reduce

48 -the `update every iterations` of the tiers, to stay under the limit.

29 +You can change these limits in `netdata.conf`:

30

50 -The exact retention that can be achieved by each tier depends on the number of metrics collected. The more

51 -the metrics, the smaller the retention that will fit in a given size. The general rule is that Netdata needs

52 -about **1 byte per data point on disk for tier 0**, and **4 bytes per data point on disk for tier 1 and above**.

53 -

54 -So, for 1000 metrics collected per second and 256 MB for tier 0, Netdata will store about:

55 -

56 -```

57 -256MB on disk / 1 byte per point / 1000 metrics => 256k points per metric / 86400 sec per day ~= 3 days

31 ```

32 +[db]

33 + mode = dbengine

34 + storage tiers = 3

35

60 -At tier 1 (per minute):

61 -

62 -```

63 -128MB on disk / 4 bytes per point / 1000 metrics => 32k points per metric / (24 hr * 60 min) ~= 22 days

64 -```

36 + # Tier 0, per second data. Set to 0 for no limit.

37 + dbengine tier 0 disk space MB = 1024

38 + dbengine tier 0 retention days = 14

39

66 -At tier 2 (per hour):

40 + # Tier 1, per minute data. Set to 0 for no limit.

41 + dbengine tier 1 disk space MB = 1024

42 + dbengine tier 1 retention days = 90

43

68 -```

69 -64MB on disk / 4 bytes per point / 1000 metrics => 16k points per metric / 24 hr per day ~= 2 years

44 + # Tier 2, per hour data. Set to 0 for no limit.

45 + dbengine tier 2 disk space MB = 1024

46 + dbengine tier 2 retention days = 730

47 ```

48

72 -Of course double the metrics, half the retention. There are more factors that affect retention. The number

73 -of ephemeral metrics (i.e. metrics that are collected for part of the time). The number of metrics that are

74 -usually constant over time (affecting compression efficiency). The number of restarts a Netdata Agents gets

75 -through time (because it has to break pages prematurely, increasing the metadata overhead). But the actual

76 -numbers should not deviate significantly from the above.

49 +## Monitoring Retention Utilization

50

78 -To see the number of metrics stored and the retention in days per tier, use the `/api/v1/dbengine_stats` endpoint.

51 +Netdata provides a visual representation of storage utilization for both time and space limits across all tiers within

52 +the 'dbengine retention' subsection of the 'Netdata Monitoring' section on the dashboard. This chart shows exactly how

53 +your storage space (disk space limits) and time (time limits) are used for metric retention.

54

80 -### Effect of storage tiers and retention on memory usage

81 -

82 -The total memory Netdata uses is heavily influenced by the memory consumed by the DBENGINE.

83 -The DBENGINE memory is related to the number of metrics concurrently being collected, the retention of the metrics

84 -on disk in relation with the queries running, and the number of metrics for which retention is maintained.

85 -

86 -The precise analysis of how much memory will be used by the DBENGINE itself is described in

87 -[DBENGINE memory requirements](/src/database/engine/README.md#memory-requirements).

88 -

89 -In addition to the DBENGINE, Netdata uses memory for contexts, metric labels (e.g. in a Kubernetes setup),

90 -other Netdata structures/processes (e.g. Health) and system overhead.

91 -

92 -The quick rule of thumb, for a high level estimation is

93 -

94 -```

95 -DBENGINE memory in MiB = METRICS x (TIERS - 1) x 8 / 1024 MiB

96 -Total Netdata memory in MiB = Metric ephemerality factor x DBENGINE memory in MiB + "dbengine page cache size MB" from netdata.conf

97 -```

98 -

99 -You can get the currently collected **METRICS** from the "dbengine metrics" chart of the Netdata dashboard. You just need to divide the

100 -value of the "collected" dimension with the number of tiers. For example, at the specific point highlighted in the chart below, 608k metrics

101 -were being collected across all 3 tiers, which means that `METRICS = 608k / 3 = 203667`.

55 +## Legacy configuration

56

103 -<img width="988" alt="image" src="https://user-images.githubusercontent.com/43294513/225335899-a9216ba7-a09e-469e-89f6-4690aada69a4.png" />

57 +### v1.45.6 and prior

58

59 +Netdata versions prior to v1.46.0 relied on a disk space-based retention.

60

106 -The **ephemerality factor** is usually between 3 or 4 and depends on how frequently the identifiers of the collected metrics change, increasing their

107 -cardinality. The more ephemeral the infrastructure, the more short-lived metrics you have, increasing the ephemerality factor. If the metric cardinality is

108 -extremely high due for example to a lot of extremely short lived containers (hundreds started every minute), the ephemerality factor can be much higher than 4.

109 -In such cases, we recommend splitting the load across multiple Netdata parents, until we can provide a way to lower the metric cardinality,

110 -by aggregating similar metrics.

61 +**Default Retention Limits**:

62

112 -#### Small agent RAM usage

63 +| Tier | Resolution | Size Limit |

64 +|:----:|:-------------------:|:----------:|

65 +| 0 | high (per second) | 256 MB |

66 +| 1 | middle (per minute) | 128 MB |

67 +| 2 | low (per hour) | 64 GiB |

68

114 -For 2000 metrics (dimensions) in 3 storage tiers and the default cache size:

69 +You can change these limits in `netdata.conf`:

70

71 ```

117 -DBENGINE memory for 2k metrics = 2000 x (3 - 1) x 8 / 1024 MiB = 32 MiB

118 -dbengine page cache size MB = 32 MiB

119 -Total Netdata memory in MiB = 3*32 + 32 = 128 MiB (low ephemerality)

120 -```

121 -

122 -#### Large parent RAM usage

72 +[db]

73 + mode = dbengine

74 + storage tiers = 3

75

124 -The Netdata parent in our production infrastructure at the time of writing:

125 - - Collects 206k metrics per second, most from children streaming data

126 - - The metrics include moderately ephemeral Kubernetes containers, leading to an ephemerality factor of about 4

127 - - 3 tiers are used for retention

128 - - The `dbengine page cache size MB` in `netdata.conf` is configured to be 4GB

76 + # Tier 0, per second data

77 + dbengine multihost disk space MB = 256

78

130 -Netdata parents can end up collecting millions of metrics per second. See also [scaling dedicated parent nodes](#scaling-dedicated-parent-nodes).

79 + # Tier 1, per minute data

80 + dbengine tier 1 multihost disk space MB = 1024

81

132 -The rule of thumb calculation for this set up gives us

133 -```

134 -DBENGINE memory = 206,000 x 16 / 1024 MiB = 3,217 MiB = about 3 GiB

135 -Extra cache = 4 GiB

136 -Metric ephemerality factor = 4

137 -Estimated total Netdata memory = 3 * 4 + 4 = 16 GiB

82 + # Tier 2, per hour data

83 + dbengine tier 2 multihost disk space MB = 1024

84 ```

85

140 -The actual measurement during a low usage time was the following:

141 -

142 -| Purpose | RAM | Note |

143 -|:------------------------------------------------------------------------|---------:|:----------------------------------------------------------|

144 -| DBENGINE usage | 5.9 GiB | Out of 7GB max |

145 -| Cardinality/ephemerality related memory (k8s contexts, labels, strings) | 3.4 GiB | |

146 -| Buffer for queries | 0 GiB | Out of 0.5 GiB max, when heavily queried |

147 -| Other | 0.5 GiB | |

148 -| System overhead | 4.4 GiB | Calculated by subtracting all of the above from the total |

149 -| **Total Netdata memory usage** | 14.2 GiB | |

150 -

151 -All the figures above except for the system memory management overhead were retrieved from Netdata itself.

152 -The overhead can't be directly calculated, so we subtracted all the other figures from the total Netdata memory usage to get it.

153 -This overhead is usually around 50% of the memory actually useable by Netdata, but could range from 20% in small

154 -setups, all the way to 100% in some edge cases.

155 -

156 -## Configure metric retention

157 -

158 -Once you have decided how to size each tier, open `netdata.conf` with

159 -[`edit-config`](/docs/netdata-agent/configuration/README.md#edit-netdataconf)

160 -and make your changes in the `[db]` subsection.

161 -

162 -Save the file and restart the Agent with `sudo systemctl restart netdata`, or

163 -the [appropriate method](/packaging/installer/README.md#maintaining-a-netdata-agent-installation)

164 -for your system, to change the database engine's size.

165 -

166 -## Scaling dedicated parent nodes

167 -

168 -When you use streaming in medium to large infrastructures, you can have potentially millions of metrics per second reaching each parent node.

169 -In the lab we have reliably collected 1 million metrics/sec with 16cores and 32GB RAM.

170 -

171 -Our suggestion for scaling parents is to have them running on dedicated VMs, using a maximum of 50% of cpu, and ensuring you have enough RAM

172 -for the desired retention. When your infrastructure can lead a parent to exceed these characteristics, split the load to multiple parents that

173 -do not communicate with each other. With each child sending data to only one of the parents, you can still have replication, high availability,

174 -and infrastructure level observability via the Netdata Cloud UI.

175 -

176 -## Legacy configuration

177 -

86 ### v1.35.1 and prior

87

88 These versions of the Agent do not support tiers. You could change the metric retention for the parent and

@@ -192,9 +100,14 @@ the `[db]` section of your `netdata.conf`.

100

101 ### v1.23.2 and prior

102

195 -_For Netdata Agents earlier than v1.23.2_, the Agent on the parent node uses one dbengine instance for itself, and another instance for every child node it receives metrics from. If you had four streaming nodes, you would have five instances in total (`1 parent + 4 child nodes = 5 instances`).

103 +_For Netdata Agents earlier than v1.23.2_, the Agent on the parent node uses one dbengine instance for itself, and

104 +another instance for every child node it receives metrics from. If you had four streaming nodes, you would have five

105 +instances in total (`1 parent + 4 child nodes = 5 instances`).

106

197 -The Agent allocates resources for each instance separately using the `dbengine disk space MB` (**deprecated**) setting. If `dbengine disk space MB`(**deprecated**) is set to the default `256`, each instance is given 256 MiB in disk space, which means the total disk space required to store all instances is, roughly, `256 MiB * 1 parent * 4 child nodes = 1280 MiB`.

107 +The Agent allocates resources for each instance separately using the `dbengine disk space MB` (**deprecated**) setting.

108 +If `dbengine disk space MB`(**deprecated**) is set to the default `256`, each instance is given 256 MiB in disk space,

109 +which means the total disk space required to store all instances is,

110 +roughly, `256 MiB * 1 parent * 4 child nodes = 1280 MiB`.

111

112 #### Backward compatibility

113

docs: add time-based retention to "Change how long Netdata stores metrics" (#17957)