@cryptotaxi247 / netdata / commits / c5f079388

docs: clarify dbengine tier 0 retention size enforcement and disk usage overflow (#22521)

* docs: update docs/netdata-dbengine-retention-size-tier0-disk-usage-overflow * docs: Execution completed * docs: fix inaccurate retention enforcement description and overshoot guidance - README.md line 47: correct timer callback flow (retention_timer_cb -> check_and_schedule_db_rotation -> rrdeng_ctx_tier_cap_exceeded) and describe rrdeng_get_used_disk_space as a heuristic (not a sum of datafile sizes), noting it may differ from `du` output - README.md line 49: clarify that after_database_rotate calls check_and_schedule_db_rotation again so rotation jobs chain; usage can exceed the limit by more than one datafile before enforcement catches up - best-practices.md line 75: replace the hard "provision at least 2x" rule with a workload-dependent note that overshoot can exceed 2x in practice, and advise provisioning additional headroom * docs: Rewrite the flagged PR documentation section to remove all code-level * Apply suggestion from @Ancairon * Apply suggestion from @Ancairon * docs: close combined retention code fence * docs: replace fixed-cadence enforcement language with asynchronous description - README.md: rename "Periodic checks" to "Periodic and asynchronous checks"; clarify that enforcement fires after extent writes and rotation completions in addition to a background timer; note that multiple datafiles may be deleted across one or more rotation passes (not just one file) - CONFIGURATION.md: replace "approximately every 60 seconds" with "periodically and asynchronously"; mention quota checks occur both on the background timer and after normal dbengine activity - best-practices.md: remove the "approximately every 60 seconds" claim; describe asynchronous quota evaluation and whole-datafile deletion loop - disk-requirements-and-retention.md: replace "approximately every 60 seconds" with the same asynchronous enforcement description used in the other files * docs: Fix code quote fabrications and inaccuracies in the PR documentation * pr review * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: nedi-app[bot] <nedi-app[bot]@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Fotis Voutsas <fotis@netdata.cloud> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

nedi-app[bot] committed Jun 4, 2026 at 09:33 UTC c5f079388ef086f439694b118d5a669728f3ef97

5 files changed +48 -20

docs/netdata-agent/sizing-netdata-agents/disk-requirements-and-retention.md

+4 -4

@@ -37,13 +37,13 @@ gantt

38 Netdata Agent metrics storage is limited to 3 GiB by default (configurable), using 1 GiB per tier × 3 tiers. In total, with SQLite databases, alert transitions, and other metadata, expect about 4 GiB of disk usage under normal conditions. The default retention limits are:

41 |:-------:|:----------:|:----------:|:----------:|

46 -Data is deleted when it reaches **either** the size limit or the time limit, whichever comes first. The number of metrics collected determines how far back in time retention extends within the size limit.

46 +Data is deleted when retention enforcement detects that **either** the size limit or the time limit has been reached, whichever comes first. Retention is enforced asynchronously — dbengine evaluates quotas and schedules rotation both on a background timer and after normal activity (such as extent writes), deleting whole datafiles until the retention check no longer reports the tier over its limit. Actual disk usage may temporarily exceed the configured size limit. The number of metrics collected determines how far back in time retention extends within the size limit.

48 In practice, with default settings and an ingestion rate of about 4,000 metrics per second, Netdata provides about 14 days of high resolution (per-second) data, 3 months of medium resolution (per-minute) data, and more than 1 year of low resolution (per-hour) data.

docs/observability-centralization-points/best-practices.md

+13 -4

@@ -60,6 +60,7 @@ Netdata supports three retention strategies. Choose the one that best fits your

60 1. **Time-based retention** (recommended for predictable retention periods):

62 Guarantees data is kept for a fixed time, regardless of disk usage (assuming you have enough disk space)

63 +

64 ```ini

65 [db]

66 dbengine tier 0 retention time = 30d

@@ -72,7 +73,8 @@ Netdata supports three retention strategies. Choose the one that best fits your

74 2. **Space-based retention** (recommended for predictable disk usage):

75 - Guarantees storage usage stays within defined limits, at the cost of variable retention duration.

76 + Targets keeping storage usage within defined limits, at the cost of variable retention duration. Retention size is enforced asynchronously, not in real-time at write time — dbengine evaluates quotas and schedules rotation both on a background timer and after normal activity (such as extent writes), deleting whole datafiles until the retention check no longer reports the tier over its limit. Actual disk usage may temporarily exceed the configured limit — especially on tier 0 with high ingestion rates from many streaming children. The amount of overshoot is workload-dependent (ingestion rate, compression variance, rotation throughput); provision additional disk headroom beyond the configured limit to reduce the risk of disk-full conditions.

77 +

78 ```ini

79 [db]

80 dbengine tier 0 retention size = 500GB

@@ -86,6 +88,7 @@ Netdata supports three retention strategies. Choose the one that best fits your

88 3. **Combined retention** (use with caution):

90 Uses both time and space limits. Data is dropped as soon as either limit is reached.

91 +

92 ```ini

93 [db]

94 dbengine tier 0 retention time = 30d

@@ -96,6 +99,12 @@ Netdata supports three retention strategies. Choose the one that best fits your

99 dbengine tier 2 retention size = 100GB # Must be large enough to hold 5 years of data!

100 ```

101

102 +:::warning

103 +

104 +Retention size limits are soft targets, not hard caps. Enforcement is periodic and asynchronous — quota checks and rotation scheduling occur both on a background timer and after normal dbengine activity (such as extent writes). Actual disk usage can exceed the configured limit between enforcement cycles. This is most noticeable on tier 0 with high metric volumes (e.g., parent nodes receiving streams from many children). Always provision more disk space than your configured limit to avoid unexpected disk-full conditions.

105 +

106 +:::

107 +

108 :::tip

109

110 - For Parent nodes with millions of metrics, expect to allocate 100GB-1TB+ per tier.

@@ -118,9 +127,9 @@ Parent nodes are the central long-term storage layer in a Netdata infrastructure

127

128 Assume a Parent configured with:

129

121 -* **Tier 0:** 30 days retention (per-second resolution)

122 -* **Tier 1:** 6 months retention (per-minute resolution)

123 -* **Tier 2:** 5 years retention (per-hour resolution)

130 +- **Tier 0:** 30 days retention (per-second resolution)

131 +- **Tier 1:** 6 months retention (per-minute resolution)

132 +- **Tier 2:** 5 years retention (per-hour resolution)

133

134 One metric would consume approximately **3.7 MB** across tiers.

135 For **1,000,000 metrics streamed to the Parent**, this equals **≈ 3.7 TB**.

src/database/CONFIGURATION.md

@@ -30,6 +30,12 @@ You can fine-tune retention for each tier by setting a time limit or size limit.

30 | Time Limit = 0, Size Limit > 0 | **Space based:** data is stored with a disk space limit, regardless of time |

31 | Time Limit > 0, Size Limit > 0 | **Combined time and space limits:** data is deleted once it reaches either the time limit or the disk space limit, whichever comes first |

33 +:::note

34 +

35 +Retention size limits are soft targets enforced periodically and asynchronously — not hard caps enforced at write time. Quota checks and rotation scheduling occur both on a background timer and after normal dbengine activity (such as extent writes). Actual disk usage can temporarily exceed the configured limit. This is most noticeable on tier 0 with high metric volumes, such as parent nodes receiving streams from many children. Provision more disk space than your configured limit to accommodate temporary overshoot.

36 +

37 +:::

38 +

39 You can change these limits using [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-configuration-files) to open `netdata.conf`:

41 ```text

src/database/README.md

+22 -5

@@ -4,11 +4,11 @@ Netdata stores detailed metrics at one-second granularity using its Database eng

5 ## Modes

7 -| Mode | Description |

8 -|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

7 +| Mode | Description |

8 +|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

9 | `dbengine` | The high performance multi-tiered time-series database of Netdata, providing superior storage efficiency (~0.5 bytes per sample on disk for high resolution per-second data), and fast long term data queries (typically 20+ times faster) by transparently utilizing all available database tiers. For details, see [Database Engine](/src/database/engine/README.md). |

10 -| `ram` | Stores data entirely in memory without disk persistence. This is typically used in IoT deviced or children that stream their metrics to Netdata parents, to avoid having any disk dependency on Netdata |

11 -| `none` | Operates without storage (metrics can only be streamed to a Netdata parent). |

10 +| `ram` | Stores data entirely in memory without disk persistence. This is typically used in IoT devices or children that stream their metrics to Netdata parents, to avoid having any disk dependency on Netdata |

11 +| `none` | Operates without storage (metrics can only be streamed to a Netdata parent). |

13 ## Tiers

@@ -38,10 +38,27 @@ These limits are fully configurable. See [Changing how long Netdata stores metri

39 Netdata provides a visual representation of storage utilization for both the time and space limits across all Tiers under "Netdata" -> "dbengine retention" on the dashboard. This chart shows exactly how your storage space (disk space limits) and time (time limits) are used for metric retention.

41 +### Retention Size Enforcement

42 +

43 +Retention size limits are **soft caps**, not hard caps. Netdata writes data unconditionally and checks limits afterwards — it does not block or reject writes when a cap is approaching.

44 +

45 +How enforcement works:

46 +

47 +1. **Periodic and asynchronous checks**: Netdata checks whether a tier has exceeded its configured size limit periodically and also after normal database activity (such as extent writes and rotation completions). Data continues to be written to disk without restriction between these checks.

48 +

49 +2. **Whole-file deletion**: When the configured size limit is exceeded, Netdata schedules deletion of the oldest complete datafiles until the retention check no longer reports the tier over its limit. Multiple files may be deleted across one or more rotation passes. Datafile size is determined automatically (see [Database Engine datafiles](/src/database/engine/README.md#datafiles)). Because entire files are removed and cannot be partially deleted, actual disk usage can overshoot the configured limit before enforcement catches up.

50 +

51 +3. **Why tier 0 overshoots more**: Tier 0 collects per-second data, producing the highest write volume. More data accumulates between enforcement checks, and data files fill faster. Tier 1 and tier 2 have lower write volumes and their disk usage grows more predictably.

52 +

53 +**Practical guidance**:

54 +

55 +- Provision storage with workload-specific headroom, especially on parent nodes streaming from many children. The required headroom depends on ingestion rate, compression, storage throughput, and how quickly rotation catches up.

56 +- Setting both **retention size** and **retention time** for the same tier can reduce retained history when either threshold is reached, but it does not create a hard disk cap.

57 +- There is currently no mechanism to enforce a true hard cap on dbengine disk usage. To reduce the risk of disk-full conditions, validate retention sizing under expected production load and ensure adequate storage headroom.

58 +

59 ## Cache sizes

61 There are two cache sizes that can be used to optimize the Database:

63 1. **Page cache size**: The main cache that keeps metrics data into memory. When data is not found in it, the extent cache is consulted, and if not found in that too, they are loaded from the disk.

64 2. **Extent cache size**: The compressed extent cache. It keeps in memory compressed data blocks, as they appear on disk, to avoid reading them again. Data found in the extent cache but not in the main cache have to be uncompressed to be queried.

47 -

src/database/engine/README.md

+3 -7

@@ -21,7 +21,7 @@ Using the **timestamp** and **duration**, Netdata calculates for each point its

22 For incremental metrics (counters), Netdata interpolates the collected values to align them to the expected **end time** at the microsecond level, absorbing data collection micro-latencies.

24 -When data points are stored in higher tiers (time aggregations - see [Tiers](#Tiers) below), each data point has:

24 +When data points are stored in higher tiers (time aggregations - see [Tiers](#tiers) below), each data point has:

26 1. The **sum** of the original values that have been aggregated

27 2. The **count** of all the original values aggregated,

@@ -83,7 +83,7 @@ This collection of 64 pages that is packed and compressed together is called an

84 #### Datafiles

86 -Multiple **extents** are appended to **datafiles** (filename suffix `.ndf`), until these **datafiles** become full. The size of each **datafile** is determined automatically by Netdata. The minimum for each **datafile** is 4MB and the maximum 512MB. Depending on the amount of disk space configured for each tier, Netdata will decide a **datafile** size trying to maintain about 50 datafiles for the whole database, within the limits mentioned (4MB min, 512MB max per file). The maximum number of datafiles supported is 65536, and therefore the maximum database size (per tier) that Netdata can support is 32TB.

86 +Multiple **extents** are appended to **datafiles** (filename suffix `.ndf`) until these **datafiles** become full. The size of each **datafile** is determined automatically by Netdata. Depending on the amount of disk space configured for each tier, Netdata chooses a target **datafile** size and clamps it between the compiled minimum and maximum datafile sizes. The current target is to keep about 100 datafiles for the tier.

88 #### Journal Files

@@ -169,7 +169,7 @@ Its primary use is to index information about the open datafile, the one that st

169

170 The clean queue is an LRU for reducing the journal v2 scans during querying.

171

172 -Open cache uses memory ballooning too, like the main cache, based on its own hot pages. Open cache hot size is mainly controlled by the size of the open datafile. This is why on netdata versions with journal files v2, we decreased the maximum datafile size from 1GB to 512MB, and we increased the target number of datafiles from 20 to 50.

172 +Open cache uses memory ballooning too, like the main cache, based on its own hot pages. Open cache hot size is mainly controlled by the size of the open datafile. This is why Netdata keeps datafile sizes bounded and targets many datafiles per tier.

173

174 On bigger setups open cache will get a bigger LRU by automatically sizing it (the whole open cache) to 5% to the size of (the whole) main cache.

175

@@ -186,7 +186,3 @@ The time-ranges of the queries running control the amount of shared memory requi

186 ## Metrics Registry

187

188 DBENGINE uses 150 bytes of memory for every metric for which retention is maintained but is not currently being collected.

189 -

190 -

191 -

192 -