@cryptotaxi247 / netdata-1 / commits / 8fa7b2f61

protection against extreme cardinality (#19486)

* protection against extreme cardinality * do not cleanup everything, but only what is required to reach 50% * check for collection flags and settings to prevent deletion * 1000 to 10000 * 1000 old instances * log extreme cardinality cleanup * add configuration for extreme cardinality * add documentation about extreme cardinality protection * allow retention to be configured in durations less than a day * min db size 32MiB * extreme cardinality working * fixed cleanup * check for extreme cardinality on rotation, not on startup * added option to enable/disable extreme cardinality protection * enable it by default * make unknown UUID errors and debug log * extreme cardinality logs more information about the current status * enable protection conditions * never delete more than requested

Costa Tsaousis committed Feb 6, 2025 at 12:28 UTC 8fa7b2f6185fc19b98a937f983ade52fb353e55d

27 files changed +412 -89

docs/extreme-cardinality-protection.md new

+114

@@ -0,0 +1,114 @@

1 +# Extreme Cardinality Protection in Netdata

2 +

3 +Netdata’s tiered storage is designed to efficiently retain metric data and metadata for long periods. However, when extreme cardinality occurs—often unintentionally through misconfigurations or inadvertent practices (e.g., spawning many short-lived docker containers or using unbounded label values)—the long-term retention of metadata can lead to excessive resource consumption.

4 +

5 +To protect Netdata from extreme cardinality, Netdata has an automated protection. This document explains **why** this protection is needed, **how** it works, **how** to configure it, and **how** to verify its operation.

6 +

7 +## Why Extreme Cardinality Protection is Needed

8 +

9 +Extreme cardinality refers to the explosion in the number of unique time series generated when metrics are combined with a wide range of labels or dimensions. In modern observability platforms like Netdata, metrics aren’t just simple numeric values—they come with metadata (labels, tags, dimensions) that help contextualize the data. When these labels are overly dynamic or unbounded (for example, when using unique identifiers such as session IDs, user IDs, or ephemeral container names), combined with a very long retention, like the one provided by Netdata, the system ends up tracking an enormous number of unique series.

10 +

11 +Despite the fact that Netdata performs better than most other observability solution, extreme cardinality has a few implications:

12 +

13 +- **Resource Consumption:** The system needs to remember and index vast amounts of metadata, increasing its memory footprint.

14 +- **Performance Degradation:** When performing long-term queries (days, weeks, months), the system needs to query a vast number of time-series leading to slower query responses.

15 +- **Operational Complexity:** High cardinality makes it harder to manage, visualize, and analyze data. Dashboards can become cluttered.

16 +- **Scalability Challenges:** As the number of time series grows, the resources required for maintaining aggregation points (Netdata parents) increase too.

17 +

18 +## Definition of Metrics Ephemerality

19 +

20 +Metrics ephemerality is the percentage of metrics that is no longer actively collected (old) compared to the total metrics available (sum of currently collected metrics and old metrics).

21 +

22 +- **Old Metrics** = The number of unique time-series that were once collected, but not currently.

23 +- **Current Metrics** = The number of unique time-series actively being collected.

24 +

25 +High Ephemerality (close to 100%): The system frequently generates new unique metrics for a short period, indicating a high turnover in metrics.

26 +Low Ephemerality (close to 0%): The system maintains a stable set of metrics over time, with little change in the total number of unique series.

27 +

28 +## How The Netdata Protection Works

29 +

30 +The mechanism kicks in during tier0 (high-resolution) database rotations (i.e., when the oldest tier0 samples are deleted) and proceeds as follows:

31 +

32 +1. **Counting Instances with Zero Tier0 Retention:**

33 + - For each context (e.g., containers, disks, network interfaces, etc), Netdata counts the number of instances that have **ZERO** retention in tier0.

34 +

35 +2. **Threshold Verification:**

36 + - If the number of instances with zero tier0 retention is **greater than or equal to 1000** (the default threshold) **and** these instances make up more than **50%** (the default Ephemerality threshold) of the total instances in that context, further action is taken.

37 +

38 +3. **Forceful Clearing in Long-Term Storage:**

39 + - The system forcefully clears the retention of the excess time-series. This action automatically triggers the deletion of the associated metadata. So, Netdata "forgets" them. Their samples are still on disk, but they are no longer accessible.

40 +

41 +4. **Retention Rules:**

42 + - **Protected Data:**

43 + - Metrics that are actively collected (and thus present in tier0) are never deleted.

44 + - A context with fewer than 1000 instances (as presented in the Netdata dashboards at the NIDL bar of the charts) is considered safe and is not modified.

45 + - **Clean-up Trigger:**

46 + - Only metrics that have lost their tier0 retention in a context that meets the thresholds (≥1000 instances and >50% ephemerality) will have their long-term retention cleared.

47 +

48 +## Configuration

49 +

50 +You can control the protection mechanism via the following settings in the `netdata.conf` file under the `[db]` section:

51 +

52 +```ini

53 +[db]

54 + extreme cardinality protection = yes

55 + extreme cardinality keep instances = 1000

56 + extreme cardinality min ephemerality = 50

57 +```

58 +

59 +- **extreme cardinality keep instances:**

60 + The minimum number of instances per context that should be kept. The default value is **1000**.

61 +

62 +- **extreme cardinality min ephemerality:**

63 + The minimum percentage (in percent) of instances in a context that have zero tier0 retention to trigger the cleanup. The default value is **50%**.

64 +

65 +

66 +**Recommendations:**

67 +

68 +- If you have samples in tier0, you also have their corresponding long-term data and metadata. Ensure that tier0 retention is configured properly.

69 +- If you expect to have more than 1000 instances per context per node (for example, more than 1000 containers, disks, network interfaces, database tables, etc.), adjust these settings to suit your specific environment.

70 +

71 +## How to Check Its Work

72 +

73 +When the protection mechanism is activated, Netdata logs a detailed message. The log entry includes:

74 +

75 +- The host name.

76 +- The context affected.

77 +- The number of metrics and instances that had their retention forcefully cleared.

78 +- The time range for which the non-tier0 retention was deleted.

79 +

80 +### Example Log Message

81 +

82 +```

83 +EXTREME CARDINALITY PROTECTION: on host '<HOST>', for context '<CONTEXT>': forcefully cleared the retention of <METRICS_COUNT> metrics and <INSTANCES_COUNT> instances, having non-tier0 retention from <START_TIME> to <END_TIME>.

84 +```

85 +

86 +This log message is tagged with the following message ID for easy identification:

87 +

88 +```

89 +MESSAGE_ID=d1f59606dd4d41e3b217a0cfcae8e632

90 +```

91 +

92 +### Verification Steps

93 +

94 +1. **Using System Logs:**

95 +

96 + You can use `journalctl` (or your system’s log viewer) to search for the message ID:

97 +

98 +```

99 +journalctl --namespace=netdata MESSAGE_ID=d1f59606dd4d41e3b217a0cfcae8e632

100 +```

101 +

102 +2. **Netdata Logs Dashboard:**

103 +

104 + Navigate to the Netdata Logs dashboard. On the right side under `MESSAGE_ID`, select **"Netdata extreme cardinality"** to filter only those messages.

105 +

106 +## Summary

107 +

108 +The extreme cardinality protection mechanism in Netdata is designed to automatically safeguard your system against the potential issues caused by excessive metric metadata retention. It does so by:

109 +

110 +- Automatically counting instances without tier0 retention.

111 +- Checking against configurable thresholds.

112 +- Forcefully clearing long-term retention (and metadata) when thresholds are exceeded.

113 +

114 +By properly configuring tier0 and adjusting the `extreme cardinality` settings in `netdata.conf`, you can ensure that your system remains both efficient and protected, even when extreme cardinality issues occur.

src/collectors/systemd-journal.plugin/systemd-journal-annotations.c

@@ -618,6 +618,7 @@ static void netdata_systemd_journal_message_ids_init(void) {

618 msgid_into_dict("8ddaf5ba33a74078b609250db1e951f3", "Sensor state transition");

619 msgid_into_dict("ec87a56120d5431bace51e2fb8bba243", "Netdata log flood protection");

620 msgid_into_dict("acb33cb95778476baac702eb7e4e151d", "Netdata Cloud connection");

621 + msgid_into_dict("d1f59606dd4d41e3b217a0cfcae8e632", "Netdata extreme cardinality");

622 }

623

624 void netdata_systemd_journal_transform_message_id(FACETS *facets __maybe_unused, BUFFER *wb, FACETS_TRANSFORMATION_SCOPE scope __maybe_unused, void *data __maybe_unused) {

src/daemon/config/netdata-conf-db.c

+6 -4

@@ -3,12 +3,13 @@

3 #include "netdata-conf-db.h"

4 #include "daemon/common.h"

6 +#define DAYS 86400

7 int default_rrd_history_entries = RRD_DEFAULT_HISTORY_ENTRIES;

9 bool dbengine_enabled = false; // will become true if and when dbengine is initialized

10 bool dbengine_use_direct_io = true;

11 static size_t storage_tiers_grouping_iterations[RRD_STORAGE_TIERS] = {1, 60, 60, 60, 60};

11 -static double storage_tiers_retention_days[RRD_STORAGE_TIERS] = {14, 90, 2 * 365, 2 * 365, 2 * 365};

12 +static time_t storage_tiers_retention_time_s[RRD_STORAGE_TIERS] = {14 * DAYS, 90 * DAYS, 2 * 365 * DAYS, 2 * 365 * DAYS, 2 * 365 * DAYS};

14 time_t rrdset_free_obsolete_time_s = 3600;

15 time_t rrdhost_free_orphan_time_s = 3600;

@@ -275,12 +276,13 @@ void netdata_conf_dbengine_init(const char *hostname) {

276 disk_space_mb = inicfg_get_size_mb(&netdata_config, CONFIG_SECTION_DB, dbengineconfig, disk_space_mb);

277

278 snprintfz(dbengineconfig, sizeof(dbengineconfig) - 1, "dbengine tier %zu retention time", tier);

278 - storage_tiers_retention_days[tier] = inicfg_get_duration_days(&netdata_config,

279 - CONFIG_SECTION_DB, dbengineconfig, new_dbengine_defaults ? storage_tiers_retention_days[tier] : 0);

279 + storage_tiers_retention_time_s[tier] = inicfg_get_duration_days_to_seconds(

280 + &netdata_config, CONFIG_SECTION_DB,

281 + dbengineconfig, new_dbengine_defaults ? storage_tiers_retention_time_s[tier] : 0);

282

283 tiers_init[tier].disk_space_mb = (int) disk_space_mb;

284 tiers_init[tier].tier = tier;

283 - tiers_init[tier].retention_seconds = (size_t) (86400.0 * storage_tiers_retention_days[tier]);

285 + tiers_init[tier].retention_seconds = (size_t) storage_tiers_retention_time_s[tier];

286 strncpyz(tiers_init[tier].path, dbenginepath, FILENAME_MAX);

287 tiers_init[tier].ret = 0;

288

src/database/contexts/api_v2_contexts_agents.c

+24 -20

@@ -5,6 +5,17 @@

6 void build_info_to_json_object(BUFFER *b);

8 +static time_t round_retention(time_t retention_seconds) {

9 + if(retention_seconds > 60 * 86400)

10 + retention_seconds = HOWMANY(retention_seconds, 86400) * 86400;

11 + else if(retention_seconds > 86400)

12 + retention_seconds = HOWMANY(retention_seconds, 3600) * 3600;

13 + else

14 + retention_seconds = HOWMANY(retention_seconds, 60) * 60;

15 +

16 + return retention_seconds;

17 +}

18 +

19 void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now_s, bool info, bool array) {

20 if(!now_s)

21 now_s = now_realtime_sec();

@@ -128,9 +139,9 @@ void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now

139

140 buffer_json_add_array_item_object(wb);

141 buffer_json_member_add_uint64(wb, "tier", tier);

131 - char human_retention[128];

132 - duration_snprintf_time_t(human_retention, sizeof(human_retention), (stime_t)group_seconds);

133 - buffer_json_member_add_string(wb, "granularity", human_retention);

142 + char human_duration[128];

143 + duration_snprintf_time_t(human_duration, sizeof(human_duration), (stime_t)group_seconds);

144 + buffer_json_member_add_string(wb, "granularity", human_duration);

145

146 buffer_json_member_add_uint64(wb, "metrics", storage_engine_metrics(eng->seb, localhost->db[tier].si));

147 buffer_json_member_add_uint64(wb, "samples", storage_engine_samples(eng->seb, localhost->db[tier].si));

@@ -148,18 +159,10 @@ void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now

159 buffer_json_member_add_time_t(wb, "to", now_s);

160 buffer_json_member_add_time_t(wb, "retention", retention);

161

151 - if(retention < 60)

152 - duration_snprintf_time_t(human_retention, sizeof(human_retention), retention);

153 - else if(retention < 24 * 60 * 60) {

154 - int64_t rounded_retention_mins = duration_round_to_resolution(retention, 60);

155 - duration_snprintf_mins(human_retention, sizeof(human_retention), rounded_retention_mins);

156 - }

157 - else {

158 - int64_t rounded_retention_hours = duration_round_to_resolution(retention, 3600);

159 - duration_snprintf_hours(human_retention, sizeof(human_retention), rounded_retention_hours);

160 - }

162 + duration_snprintf(human_duration, sizeof(human_duration),

163 + round_retention(retention), "s", false);

164

162 - buffer_json_member_add_string(wb, "retention_human", human_retention);

165 + buffer_json_member_add_string(wb, "retention_human", human_duration);

166

167 if(used || max) { // we have disk space information

168 time_t time_retention = 0;

@@ -169,17 +172,18 @@ void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now

172 time_t space_retention = (time_t)((NETDATA_DOUBLE)(now_s - first_time_s) * 100.0 / percent);

173 time_t actual_retention = MIN(space_retention, time_retention ? time_retention : space_retention);

174

172 - duration_snprintf_hours(human_retention, sizeof(human_retention),

173 - (int)duration_round_to_resolution(time_retention, 3600));

175 + duration_snprintf(

176 + human_duration, sizeof(human_duration),

177 + (int)time_retention, "s", false);

178

179 buffer_json_member_add_time_t(wb, "requested_retention", time_retention);

176 - buffer_json_member_add_string(wb, "requested_retention_human", human_retention);

180 + buffer_json_member_add_string(wb, "requested_retention_human", human_duration);

181

178 - duration_snprintf_hours(human_retention, sizeof(human_retention),

179 - (int)duration_round_to_resolution(actual_retention, 3600));

182 + duration_snprintf(human_duration, sizeof(human_duration),

183 + (int)round_retention(actual_retention), "s", false);

184

185 buffer_json_member_add_time_t(wb, "expected_retention", actual_retention);

182 - buffer_json_member_add_string(wb, "expected_retention_human", human_retention);

186 + buffer_json_member_add_string(wb, "expected_retention_human", human_duration);

187 }

188 }

189 buffer_json_object_close(wb);

src/database/contexts/contexts-loading.c

+1 -1

@@ -18,7 +18,7 @@ static void rrdinstance_load_dimension_callback(SQL_DIMENSION_DATA *sd, void *da

19 UUIDMAP_ID id = uuidmap_create(sd->dim_id);

20 time_t min_first_time_t = LONG_MAX, max_last_time_t = 0;

21 - get_metric_retention_by_id(host, id, &min_first_time_t, &max_last_time_t);

21 + get_metric_retention_by_id(host, id, &min_first_time_t, &max_last_time_t, NULL);

22 if((!min_first_time_t || min_first_time_t == LONG_MAX) && !max_last_time_t) {

23 uuidmap_free(id);

24 th_zero_retention_metrics++;

src/database/contexts/internal.h

+2 -1

@@ -60,6 +60,7 @@ typedef enum __attribute__ ((__packed__)) {

60 RRD_FLAG_UPDATE_REASON_UNUSED = (1 << 22), // this context is not used anymore

61 RRD_FLAG_UPDATE_REASON_DB_ROTATION = (1 << 23), // this context changed because of a db rotation

63 + RRD_FLAG_NO_TIER0_RETENTION = (1 << 28),

64 RRD_FLAG_MERGED_COLLECTED_RI_TO_RC = (1 << 29),

66 // action to perform on an object

@@ -475,7 +476,7 @@ void rrdcontext_update_from_collected_rrdinstance(RRDINSTANCE *ri);

476

477 void rrdcontext_garbage_collect_single_host(RRDHOST *host, bool worker_jobs);

478

478 -void get_metric_retention_by_id(RRDHOST *host, UUIDMAP_ID id, time_t *min_first_time_t, time_t *max_last_time_t);

479 +void get_metric_retention_by_id(RRDHOST *host, UUIDMAP_ID id, time_t *min_first_time_t, time_t *max_last_time_t, bool *tier0_retention);

480

481 void rrdcontext_delete_after_loading(RRDHOST *host, RRDCONTEXT *rc);

482 void rrdcontext_initial_processing_after_loading(RRDCONTEXT *rc);

src/database/contexts/metric.c

+4 -3

@@ -111,6 +111,7 @@ static void rrdmetric_delete_callback(const DICTIONARY_ITEM *item __maybe_unused

111 static bool rrdmetric_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *old_value, void *new_value, void *rrdinstance __maybe_unused) {

112 RRDMETRIC *rm = old_value;

113 RRDMETRIC *rm_new = new_value;

114 + rm_new->ri = rm->ri;

115

116 internal_error(rm->id != rm_new->id,

117 "RRDMETRIC: '%s' cannot change id to '%s'",

@@ -131,9 +132,9 @@ static bool rrdmetric_conflict_callback(const DICTIONARY_ITEM *item __maybe_unus

132

133 time_t new_first_time_s = 0;

134 time_t new_last_time_s = 0;

134 - if(rrdmetric_update_retention(rm)) {

135 - new_first_time_s = rm->first_time_s;

136 - new_last_time_s = rm->last_time_s;

135 + if(rrdmetric_update_retention(rm_new)) {

136 + new_first_time_s = rm_new->first_time_s;

137 + new_last_time_s = rm_new->last_time_s;

138 }

139

140 internal_error(true,

src/database/contexts/worker.c

+171 -13

@@ -2,6 +2,18 @@

3 #include "internal.h"

5 +static struct {

6 + bool enabled;

7 + size_t db_rotations;

8 + size_t instances_count;

9 + size_t active_vs_archived_percentage;

10 +} extreme_cardinality = {

11 + .enabled = true, // this value is ignored - there is a dynamic condition to enable it

12 + .db_rotations = 0,

13 + .instances_count = 1000,

14 + .active_vs_archived_percentage = 50,

15 +};

16 +

17 static uint64_t rrdcontext_get_next_version(RRDCONTEXT *rc);

19 static bool check_if_cloud_version_changed_unsafe(RRDCONTEXT *rc, bool sending __maybe_unused);

@@ -9,7 +21,7 @@ static bool check_if_cloud_version_changed_unsafe(RRDCONTEXT *rc, bool sending _

21 static void rrdcontext_delete_from_sql_unsafe(RRDCONTEXT *rc);

23 static void rrdcontext_dequeue_from_post_processing(RRDCONTEXT *rc);

12 -static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAGS reason, bool worker_jobs);

24 +static bool rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAGS reason, bool worker_jobs);

26 static void rrdcontext_garbage_collect_for_all_hosts(void);

@@ -100,7 +112,10 @@ static void rrdhost_update_cached_retention(RRDHOST *host, time_t first_time_s,

112 }

113

114 void rrdcontext_recalculate_context_retention(RRDCONTEXT *rc, RRD_FLAGS reason, bool worker_jobs) {

103 - rrdcontext_post_process_updates(rc, true, reason, worker_jobs);

115 + bool forcefully_removed_instances = false;

116 + do {

117 + forcefully_removed_instances = rrdcontext_post_process_updates(rc, true, reason, worker_jobs);

118 + } while(forcefully_removed_instances);

119 }

120

121 void rrdcontext_recalculate_host_retention(RRDHOST *host, RRD_FLAGS reason, bool worker_jobs) {

@@ -137,7 +152,7 @@ static void rrdcontext_recalculate_retention_all_hosts(void) {

152 // ----------------------------------------------------------------------------

153 // garbage collector

154

140 -void get_metric_retention_by_id(RRDHOST *host, UUIDMAP_ID id, time_t *min_first_time_t, time_t *max_last_time_t) {

155 +void get_metric_retention_by_id(RRDHOST *host, UUIDMAP_ID id, time_t *min_first_time_t, time_t *max_last_time_t, bool *tier0_retention) {

156 *min_first_time_t = LONG_MAX;

157 *max_last_time_t = 0;

158

@@ -152,6 +167,9 @@ void get_metric_retention_by_id(RRDHOST *host, UUIDMAP_ID id, time_t *min_first_

167 if (last_time_t > *max_last_time_t)

168 *max_last_time_t = last_time_t;

169 }

170 +

171 + if(tier == 0 && tier0_retention)

172 + *tier0_retention = first_time_t || last_time_t;

173 }

174 }

175

@@ -161,21 +179,24 @@ bool rrdmetric_update_retention(RRDMETRIC *rm) {

179 if(rm->rrddim) {

180 min_first_time_t = rrddim_first_entry_s(rm->rrddim);

181 max_last_time_t = rrddim_last_entry_s(rm->rrddim);

182 + rrd_flag_clear(rm, RRD_FLAG_NO_TIER0_RETENTION);

183 }

165 - else

166 - get_metric_retention_by_id(rm->ri->rc->rrdhost, rm->uuid, &min_first_time_t, &max_last_time_t);

184 + else {

185 + bool tier0_retention;

186 + get_metric_retention_by_id(rm->ri->rc->rrdhost, rm->uuid, &min_first_time_t, &max_last_time_t, &tier0_retention);

187

168 - if((min_first_time_t == LONG_MAX || min_first_time_t == 0) && max_last_time_t == 0)

169 - return false;

188 + if(tier0_retention)

189 + rrd_flag_clear(rm, RRD_FLAG_NO_TIER0_RETENTION);

190 + else

191 + rrd_flag_set(rm, RRD_FLAG_NO_TIER0_RETENTION);

192 + }

193

194 if(min_first_time_t == LONG_MAX)

195 min_first_time_t = 0;

196

197 if(min_first_time_t > max_last_time_t) {

198 internal_error(true, "RRDMETRIC: retention of '%s' is flipped, first_time_t = %ld, last_time_t = %ld", string2str(rm->id), min_first_time_t, max_last_time_t);

176 - time_t tmp = min_first_time_t;

177 - min_first_time_t = max_last_time_t;

178 - max_last_time_t = tmp;

199 + SWAP(min_first_time_t, max_last_time_t);

200 }

201

202 // check if retention changed

@@ -402,7 +423,7 @@ static void rrdinstance_post_process_updates(RRDINSTANCE *ri, bool force, RRD_FL

423 worker_is_busy(WORKER_JOB_PP_INSTANCE);

424

425 time_t min_first_time_t = LONG_MAX, max_last_time_t = 0;

405 - size_t metrics_active = 0, metrics_deleted = 0;

426 + size_t metrics_active = 0, metrics_deleted = 0, metrics_no_tier0 = 0;

427 bool live_retention = true, currently_collected = false;

428 if(dictionary_entries(ri->rrdmetrics) > 0) {

429 RRDMETRIC *rm;

@@ -418,6 +439,9 @@ static void rrdinstance_post_process_updates(RRDINSTANCE *ri, bool force, RRD_FL

439 if(unlikely(!rrd_flag_check(rm, RRD_FLAG_LIVE_RETENTION)))

440 live_retention = false;

441

442 + if(unlikely(rrd_flag_check(rm, RRD_FLAG_NO_TIER0_RETENTION)))

443 + metrics_no_tier0++;

444 +

445 if (unlikely((rrdmetric_should_be_deleted(rm)))) {

446 metrics_deleted++;

447 continue;

@@ -437,6 +461,11 @@ static void rrdinstance_post_process_updates(RRDINSTANCE *ri, bool force, RRD_FL

461 dfe_done(rm);

462 }

463

464 + if(metrics_no_tier0 && metrics_no_tier0 == metrics_active)

465 + rrd_flag_set(ri, RRD_FLAG_NO_TIER0_RETENTION);

466 + else

467 + rrd_flag_clear(ri, RRD_FLAG_NO_TIER0_RETENTION);

468 +

469 if(unlikely(live_retention && !rrd_flag_check(ri, RRD_FLAG_LIVE_RETENTION)))

470 rrd_flag_set(ri, RRD_FLAG_LIVE_RETENTION);

471 else if(unlikely(!live_retention && rrd_flag_check(ri, RRD_FLAG_LIVE_RETENTION)))

@@ -500,7 +529,98 @@ static void rrdinstance_post_process_updates(RRDINSTANCE *ri, bool force, RRD_FL

529 rrd_flag_unset_updated(ri);

530 }

531

503 -static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAGS reason, bool worker_jobs) {

532 +static bool rrdinstance_forcefully_clear_retention(RRDCONTEXT *rc, size_t count, const char *descr) {

533 + if(!count) return false;

534 +

535 + RRDHOST *host = rc->rrdhost;

536 +

537 + time_t from_s = LONG_MAX;

538 + time_t to_s = 0;

539 +

540 + size_t instances_deleted = 0;

541 + size_t metrics_deleted = 0;

542 + RRDINSTANCE *ri;

543 + dfe_start_read(rc->rrdinstances, ri) {

544 + if(!rrd_flag_check(ri, RRD_FLAG_NO_TIER0_RETENTION) || rrd_flag_is_collected(ri) || ri->rrdset)

545 + continue;

546 +

547 + size_t metrics_cleared = 0;

548 + RRDMETRIC *rm;

549 + dfe_start_read(ri->rrdmetrics, rm) {

550 + if(!rrd_flag_check(rm, RRD_FLAG_NO_TIER0_RETENTION) || rrd_flag_is_collected(rm) || rm->rrddim)

551 + continue;

552 +

553 + rrdmetric_update_retention(rm);

554 +

555 + if(rm->first_time_s < from_s)

556 + from_s = rm->first_time_s;

557 +

558 + if(rm->last_time_s > to_s)

559 + to_s = rm->last_time_s;

560 +

561 + for (size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {

562 + STORAGE_ENGINE *eng = host->db[tier].eng;

563 + eng->api.metric_retention_delete_by_id(host->db[tier].si, rm->uuid);

564 + }

565 +

566 + metrics_cleared++;

567 + metrics_deleted++;

568 + rrdmetric_update_retention(rm);

569 + rrdmetric_trigger_updates(rm, __FUNCTION__ );

570 + }

571 + dfe_done(rm);

572 +

573 + if(metrics_cleared) {

574 + rrdinstance_trigger_updates(ri, __FUNCTION__ );

575 + instances_deleted++;

576 +

577 + if(--count == 0)

578 + break;

579 + }

580 + }

581 + dfe_done(ri);

582 +

583 + if(metrics_deleted) {

584 + char from_txt[128], to_txt[128];

585 +

586 + if(!from_s || from_s == LONG_MAX)

587 + snprintfz(from_txt, sizeof(from_txt), "%s", "NONE");

588 + else

589 + rfc3339_datetime_ut(from_txt, sizeof(from_txt), from_s * USEC_PER_SEC, 0, true);

590 +

591 + if(!to_s)

592 + snprintfz(to_txt, sizeof(to_txt), "%s", "NONE");

593 + else

594 + rfc3339_datetime_ut(to_txt, sizeof(to_txt), to_s * USEC_PER_SEC, 0, true);

595 +

596 + ND_LOG_STACK lgs[] = {

597 + ND_LOG_FIELD_TXT(NDF_MODULE, "extreme cardinality protection"),

598 + ND_LOG_FIELD_STR(NDF_NIDL_NODE, rc->rrdhost->hostname),

599 + ND_LOG_FIELD_STR(NDF_NIDL_CONTEXT, rc->id),

600 + ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &extreme_cardinality_msgid),

601 + ND_LOG_FIELD_END(),

602 + };

603 + ND_LOG_STACK_PUSH(lgs);

604 +

605 + nd_log(NDLS_DAEMON, NDLP_NOTICE,

606 + "EXTREME CARDINALITY PROTECTION: host '%s', context '%s', %s: "

607 + "forcefully cleared the retention of %zu metrics and %zu instances, "

608 + "having non-tier0 retention from %s to %s.",

609 + rrdhost_hostname(rc->rrdhost),

610 + string2str(rc->id),

611 + descr,

612 + metrics_deleted, instances_deleted,

613 + from_txt, to_txt);

614 +

615 + return true;

616 + }

617 +

618 + return false;

619 +}

620 +

621 +static bool rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAGS reason, bool worker_jobs) {

622 + bool ret = false;

623 +

624 if(reason != RRD_FLAG_NONE)

625 rrd_flag_set_updated(rc, reason);

626

@@ -511,7 +631,7 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG

631 size_t min_priority_not_collected = LONG_MAX;

632 size_t min_priority = LONG_MAX;

633 time_t min_first_time_t = LONG_MAX, max_last_time_t = 0;

514 - size_t instances_active = 0, instances_deleted = 0;

634 + size_t instances_active = 0, instances_deleted = 0, instances_no_tier0 = 0;

635 bool live_retention = true, currently_collected = false, hidden = true;

636 if(dictionary_entries(rc->rrdinstances) > 0) {

637 RRDINSTANCE *ri;

@@ -535,6 +655,9 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG

655 continue;

656 }

657

658 + if(unlikely(rrd_flag_check(ri, RRD_FLAG_NO_TIER0_RETENTION)))

659 + instances_no_tier0++;

660 +

661 bool ri_collected = rrd_flag_is_collected(ri);

662

663 if(ri_collected && !rrd_flag_check(ri, RRD_FLAG_MERGED_COLLECTED_RI_TO_RC)) {

@@ -571,6 +694,25 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG

694 }

695 dfe_done(ri);

696

697 + if(extreme_cardinality.enabled &&

698 + extreme_cardinality.db_rotations &&

699 + instances_no_tier0 >= extreme_cardinality.instances_count) {

700 + size_t percent = (100 * instances_no_tier0 / instances_active);

701 + if(percent >= extreme_cardinality.active_vs_archived_percentage) {

702 + size_t to_keep = extreme_cardinality.active_vs_archived_percentage * instances_active / 100;

703 + to_keep = MAX(to_keep, extreme_cardinality.instances_count);

704 + size_t to_remove = instances_no_tier0 > to_keep ? instances_no_tier0 - to_keep : 0;

705 +

706 + if(to_remove) {

707 + char buf[256];

708 + snprintfz(buf, sizeof(buf),

709 + "total active instances %zu, not in tier0 %zu, ephemerality %zu%%",

710 + instances_active, instances_no_tier0, percent);

711 + ret = rrdinstance_forcefully_clear_retention(rc, to_remove, buf);

712 + }

713 + }

714 + }

715 +

716 if(min_priority_collected != LONG_MAX)

717 // use the collected priority

718 min_priority = min_priority_collected;

@@ -669,6 +811,8 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG

811

812 rrd_flag_unset_updated(rc);

813 rrdcontext_unlock(rc);

814 +

815 + return ret;

816 }

817

818 void rrdcontext_queue_for_post_processing(RRDCONTEXT *rc, const char *function __maybe_unused, RRD_FLAGS flags __maybe_unused) {

@@ -1016,6 +1160,19 @@ void *rrdcontext_main(void *ptr) {

1160 heartbeat_t hb;

1161 heartbeat_init(&hb, RRDCONTEXT_WORKER_THREAD_HEARTBEAT_USEC);

1162

1163 + extreme_cardinality.enabled = inicfg_get_boolean(

1164 + &netdata_config, CONFIG_SECTION_DB, "extreme cardinality protection",

1165 + nd_profile.storage_tiers > 1 && default_rrd_memory_mode == RRD_DB_MODE_DBENGINE

1166 + );

1167 +

1168 + extreme_cardinality.instances_count = inicfg_get_number_range(

1169 + &netdata_config, CONFIG_SECTION_DB, "extreme cardinality keep instances",

1170 + (long long)extreme_cardinality.instances_count, 1, 1000000);

1171 +

1172 + extreme_cardinality.active_vs_archived_percentage = inicfg_get_number_range(

1173 + &netdata_config, CONFIG_SECTION_DB, "extreme cardinality min ephemerality",

1174 + (long long)extreme_cardinality.active_vs_archived_percentage, 0, 100);

1175 +

1176 while (service_running(SERVICE_CONTEXT)) {

1177 worker_is_idle();

1178 heartbeat_next(&hb);

@@ -1025,6 +1182,7 @@ void *rrdcontext_main(void *ptr) {

1182 usec_t now_ut = now_realtime_usec();

1183

1184 if(rrdcontext_next_db_rotation_ut && now_ut > rrdcontext_next_db_rotation_ut) {

1185 + extreme_cardinality.db_rotations++;

1186 rrdcontext_recalculate_retention_all_hosts();

1187 rrdcontext_garbage_collect_for_all_hosts();

1188 rrdcontext_next_db_rotation_ut = 0;

src/database/engine/datafile.h

+1 -1

@@ -20,7 +20,7 @@ struct rrdengine_instance;

20 #error MIN_DATAFILE_SIZE > MAX_DATAFILE_SIZE

21 #endif

23 -#define MIN_DATAFILE_SIZE (4LU * 1024LU * 1024LU)

23 +#define MIN_DATAFILE_SIZE (512LU * 1024LU)

24 #define MAX_DATAFILES (65536 * 4) /* Supports up to 64TiB for now */

25 #define TARGET_DATAFILES (100)

src/database/engine/metric.c

+7 -1

@@ -459,6 +459,12 @@ ALWAYS_INLINE time_t mrg_metric_get_first_time_s(MRG *mrg __maybe_unused, METRIC

459 return mrg_metric_get_first_time_s_smart(mrg, metric);

460 }

461

462 +void mrg_metric_clear_retention(MRG *mrg __maybe_unused, METRIC *metric) {

463 + __atomic_store_n(&metric->first_time_s, 0, __ATOMIC_RELAXED);

464 + __atomic_store_n(&metric->latest_time_s_clean, 0, __ATOMIC_RELAXED);

465 + __atomic_store_n(&metric->latest_time_s_hot, 0, __ATOMIC_RELAXED);

466 +}

467 +

468 ALWAYS_INLINE_HOT void mrg_metric_get_retention(MRG *mrg __maybe_unused, METRIC *metric, time_t *first_time_s, time_t *last_time_s, uint32_t *update_every_s) {

469 time_t clean = __atomic_load_n(&metric->latest_time_s_clean, __ATOMIC_RELAXED);

470 time_t hot = __atomic_load_n(&metric->latest_time_s_hot, __ATOMIC_RELAXED);

@@ -490,7 +496,7 @@ ALWAYS_INLINE bool mrg_metric_set_clean_latest_time_s(MRG *mrg __maybe_unused, M

496 }

497

498 // returns true when metric still has retention

493 -ALWAYS_INLINE bool mrg_metric_zero_disk_retention(MRG *mrg __maybe_unused, METRIC *metric) {

499 +ALWAYS_INLINE bool mrg_metric_has_zero_disk_retention(MRG *mrg __maybe_unused, METRIC *metric) {

500 Word_t section = mrg_metric_section(mrg, metric);

501 bool do_again = false;

502 size_t countdown = 5;

src/database/engine/metric.h

+3 -2

@@ -53,7 +53,7 @@ bool mrg_metric_release_and_delete(MRG *mrg, METRIC *metric);

54 Word_t mrg_metric_id(MRG *mrg, METRIC *metric);

55 nd_uuid_t *mrg_metric_uuid(MRG *mrg, METRIC *metric);

56 -UUIDMAP_ID mrg_metric_uuidmap_id_dup(MRG *mrg __maybe_unused, METRIC *metric);

56 +UUIDMAP_ID mrg_metric_uuidmap_id_dup(MRG *mrg, METRIC *metric);

57 Word_t mrg_metric_section(MRG *mrg, METRIC *metric);

59 bool mrg_metric_set_first_time_s(MRG *mrg, METRIC *metric, time_t first_time_s);

@@ -71,7 +71,8 @@ uint32_t mrg_metric_get_update_every_s(MRG *mrg, METRIC *metric);

72 void mrg_metric_expand_retention(MRG *mrg, METRIC *metric, time_t first_time_s, time_t last_time_s, uint32_t update_every_s);

73 void mrg_metric_get_retention(MRG *mrg, METRIC *metric, time_t *first_time_s, time_t *last_time_s, uint32_t *update_every_s);

74 -bool mrg_metric_zero_disk_retention(MRG *mrg __maybe_unused, METRIC *metric);

74 +bool mrg_metric_has_zero_disk_retention(MRG *mrg, METRIC *metric);

75 +void mrg_metric_clear_retention(MRG *mrg, METRIC *metric);

77 #ifdef NETDATA_INTERNAL_CHECKS

78 bool mrg_metric_set_writer(MRG *mrg, METRIC *metric);

src/database/engine/pdc.c

+7 -7

@@ -895,7 +895,7 @@ static ALWAYS_INLINE struct page_details *epdl_get_pd_load_link_list_from_metric

895 return pd_list;

896 }

897

898 -static void epdl_extent_loading_error_log(struct rrdengine_instance *ctx, EPDL *epdl, struct rrdeng_extent_page_descr *descr, const char *msg) {

898 +static void epdl_extent_loading_error_log(struct rrdengine_instance *ctx, EPDL *epdl, struct rrdeng_extent_page_descr *descr, const char *msg, ND_LOG_FIELD_PRIORITY priority) {

899 char uuid[UUID_STR_LEN] = "";

900 time_t start_time_s = 0;

901 time_t end_time_s = 0;

@@ -950,7 +950,7 @@ static void epdl_extent_loading_error_log(struct rrdengine_instance *ctx, EPDL *

950 log_date(end_time_str, LOG_DATE_LENGTH, end_time_s);

951

952 nd_log_limit_static_global_var(erl, 1, 0);

953 - nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR,

953 + nd_log_limit(&erl, NDLS_DAEMON, priority,

954 "DBENGINE: error while reading extent from datafile %u of tier %d, at offset %" PRIu64 " (%u bytes) "

955 "%s from %ld (%s) to %ld (%s) %s%s: "

956 "%s",

@@ -1011,7 +1011,7 @@ static bool epdl_populate_pages_from_extent_data(

1011 (payload_length != trailer_offset - payload_offset) ||

1012 (data_length != payload_offset + payload_length + sizeof(*trailer))

1013 ) {

1014 - epdl_extent_loading_error_log(ctx, epdl, NULL, "header is INVALID");

1014 + epdl_extent_loading_error_log(ctx, epdl, NULL, "header is INVALID", NDLP_ERR);

1015 return false;

1016 }

1017

@@ -1020,7 +1020,7 @@ static bool epdl_populate_pages_from_extent_data(

1020 if (unlikely(crc32cmp(trailer->checksum, crc))) {

1021 ctx_io_error(ctx);

1022 have_read_error = true;

1023 - epdl_extent_loading_error_log(ctx, epdl, NULL, "CRC32 checksum FAILED");

1023 + epdl_extent_loading_error_log(ctx, epdl, NULL, "CRC32 checksum FAILED", NDLP_ERR);

1024 }

1025

1026 if(worker)

@@ -1081,7 +1081,7 @@ static bool epdl_populate_pages_from_extent_data(

1081 if(!page_length || !start_time_s) {

1082 char log[200 + 1];

1083 snprintfz(log, sizeof(log) - 1, "page %u (out of %u) is EMPTY", i, count);

1084 - epdl_extent_loading_error_log(ctx, epdl, &header->descr[i], log);

1084 + epdl_extent_loading_error_log(ctx, epdl, &header->descr[i], log, NDLP_ERR);

1085 continue;

1086 }

1087

@@ -1090,7 +1090,7 @@ static bool epdl_populate_pages_from_extent_data(

1090 if(!metric) {

1091 char log[200 + 1];

1092 snprintfz(log, sizeof(log) - 1, "page %u (out of %u) has unknown UUID", i, count);

1093 - epdl_extent_loading_error_log(ctx, epdl, &header->descr[i], log);

1093 + epdl_extent_loading_error_log(ctx, epdl, &header->descr[i], log, NDLP_DEBUG);

1094 continue;

1095 }

1096 mrg_metric_release(main_mrg, metric);

@@ -1126,7 +1126,7 @@ static bool epdl_populate_pages_from_extent_data(

1126 snprintfz(log, sizeof(log) - 1, "page %u (out of %u) offset %u + page length %zu, "

1127 "exceeds the uncompressed buffer size %u",

1128 i, count, page_offset, vd.page_length, uncompressed_payload_length);

1129 - epdl_extent_loading_error_log(ctx, epdl, &header->descr[i], log);

1129 + epdl_extent_loading_error_log(ctx, epdl, &header->descr[i], log, NDLP_ERR);

1130

1131 pgd = PGD_EMPTY;

1132 stats_load_invalid_page++;

src/database/engine/rrdengine.c

+1 -1

@@ -1181,7 +1181,7 @@ static void update_metrics_first_time_s(struct rrdengine_instance *ctx, struct r

1181 zero_disk_retention++;

1182

1183 // there is no retention for this metric

1184 - bool has_retention = mrg_metric_zero_disk_retention(main_mrg, uuid_first_t_entry->metric);

1184 + bool has_retention = mrg_metric_has_zero_disk_retention(main_mrg, uuid_first_t_entry->metric);

1185 if (!has_retention) {

1186 time_t first_time_s = mrg_metric_get_first_time_s(main_mrg, uuid_first_t_entry->metric);

1187 time_t last_time_s = mrg_metric_get_latest_time_s(main_mrg, uuid_first_t_entry->metric);

src/database/engine/rrdengineapi.c

+15

@@ -1005,6 +1005,21 @@ bool rrdeng_metric_retention_by_id(STORAGE_INSTANCE *si, UUIDMAP_ID id, time_t *

1005 return true;

1006 }

1007

1008 +void rrdeng_metric_retention_delete_by_id(STORAGE_INSTANCE *si, UUIDMAP_ID id) {

1009 + struct rrdengine_instance *ctx = (struct rrdengine_instance *)si;

1010 + if (unlikely(!ctx)) {

1011 + netdata_log_error("DBENGINE: invalid STORAGE INSTANCE to %s()", __FUNCTION__);

1012 + return;

1013 + }

1014 +

1015 + METRIC *metric = mrg_metric_get_and_acquire_by_id(main_mrg, id, (Word_t)ctx);

1016 + if (unlikely(!metric))

1017 + return;

1018 +

1019 + mrg_metric_clear_retention(main_mrg, metric);

1020 + mrg_metric_release(main_mrg, metric);

1021 +}

1022 +

1023 uint64_t rrdeng_disk_space_max(STORAGE_INSTANCE *si) {

1024 struct rrdengine_instance *ctx = (struct rrdengine_instance *)si;

1025 return ctx->config.max_disk_space;

src/database/engine/rrdengineapi.h

+2 -1

@@ -6,7 +6,7 @@

6 #include "rrdengine.h"

8 #define RRDENG_MIN_PAGE_CACHE_SIZE_MB (8)

9 -#define RRDENG_MIN_DISK_SPACE_MB (256)

9 +#define RRDENG_MIN_DISK_SPACE_MB (25)

10 #define RRDENG_DEFAULT_TIER_DISK_SPACE_MB (1024)

12 #define RRDENG_NR_STATS (38)

@@ -78,6 +78,7 @@ void rrdeng_quiesce(struct rrdengine_instance *ctx);

79 bool rrdeng_metric_retention_by_id(STORAGE_INSTANCE *si, UUIDMAP_ID id, time_t *first_entry_s, time_t *last_entry_s);

80 bool rrdeng_metric_retention_by_uuid(STORAGE_INSTANCE *si, nd_uuid_t *dim_uuid, time_t *first_entry_s, time_t *last_entry_s);

81 +void rrdeng_metric_retention_delete_by_id(STORAGE_INSTANCE *si, UUIDMAP_ID id);

83 extern STORAGE_METRICS_GROUP *rrdeng_metrics_group_get(STORAGE_INSTANCE *si, nd_uuid_t *uuid);

84 extern void rrdeng_metrics_group_release(STORAGE_INSTANCE *si, STORAGE_METRICS_GROUP *smg);

src/database/ram/rrddim_mem.c

@@ -155,6 +155,10 @@ bool rrddim_metric_retention_by_id(STORAGE_INSTANCE *si __maybe_unused, UUIDMAP_

155 return true;

156 }

157

158 +void rrddim_retention_delete_by_id(STORAGE_INSTANCE *si __maybe_unused, UUIDMAP_ID id __maybe_unused) {

159 + ;

160 +}

161 +

162 void rrddim_store_metric_change_collection_frequency(STORAGE_COLLECT_HANDLE *sch, int update_every) {

163 struct mem_collect_handle *ch = (struct mem_collect_handle *)sch;

164 struct mem_metric_handle *mh = (struct mem_metric_handle *)ch->smh;

src/database/ram/rrddim_mem.h

@@ -30,6 +30,7 @@ void rrddim_metric_release(STORAGE_METRIC_HANDLE *smh);

31 bool rrddim_metric_retention_by_id(STORAGE_INSTANCE *si, UUIDMAP_ID id, time_t *first_entry_s, time_t *last_entry_s);

32 bool rrddim_metric_retention_by_uuid(STORAGE_INSTANCE *si, nd_uuid_t *uuid, time_t *first_entry_s, time_t *last_entry_s);

33 +void rrddim_retention_delete_by_id(STORAGE_INSTANCE *si, UUIDMAP_ID id);

35 STORAGE_METRICS_GROUP *rrddim_metrics_group_get(STORAGE_INSTANCE *si, nd_uuid_t *uuid);

36 void rrddim_metrics_group_release(STORAGE_INSTANCE *si, STORAGE_METRICS_GROUP *smg);

src/database/storage-engine.c

@@ -19,6 +19,7 @@ static STORAGE_ENGINE engines[] = {

19 .metric_release = rrddim_metric_release,

20 .metric_retention_by_id = rrddim_metric_retention_by_id,

21 .metric_retention_by_uuid = rrddim_metric_retention_by_uuid,

22 + .metric_retention_delete_by_id = rrddim_retention_delete_by_id,

23 }

24 },

25 {

@@ -33,6 +34,7 @@ static STORAGE_ENGINE engines[] = {

34 .metric_release = rrddim_metric_release,

35 .metric_retention_by_id = rrddim_metric_retention_by_id,

36 .metric_retention_by_uuid = rrddim_metric_retention_by_uuid,

37 + .metric_retention_delete_by_id = rrddim_retention_delete_by_id,

38 }

39 },

40 {

@@ -47,6 +49,7 @@ static STORAGE_ENGINE engines[] = {

49 .metric_release = rrddim_metric_release,

50 .metric_retention_by_id = rrddim_metric_retention_by_id,

51 .metric_retention_by_uuid = rrddim_metric_retention_by_uuid,

52 + .metric_retention_delete_by_id = rrddim_retention_delete_by_id,

53 }

54 },

55 #ifdef ENABLE_DBENGINE

@@ -62,6 +65,7 @@ static STORAGE_ENGINE engines[] = {

65 .metric_release = rrdeng_metric_release,

66 .metric_retention_by_id = rrdeng_metric_retention_by_id,

67 .metric_retention_by_uuid = rrdeng_metric_retention_by_uuid,

68 + .metric_retention_delete_by_id = rrdeng_metric_retention_delete_by_id,

69 }

70 },

71 #endif

src/database/storage-engine.h

@@ -68,6 +68,7 @@ typedef struct storage_engine_api {

68 STORAGE_METRIC_HANDLE *(*metric_dup)(STORAGE_METRIC_HANDLE *);

69 bool (*metric_retention_by_id)(STORAGE_INSTANCE *si, UUIDMAP_ID id, time_t *first_entry_s, time_t *last_entry_s);

70 bool (*metric_retention_by_uuid)(STORAGE_INSTANCE *si, nd_uuid_t *uuid, time_t *first_entry_s, time_t *last_entry_s);

71 + void (*metric_retention_delete_by_id)(STORAGE_INSTANCE *si, UUIDMAP_ID id);

72 } STORAGE_ENGINE_API;

74 typedef struct storage {

src/libnetdata/inicfg/inicfg.h

+1 -2

@@ -208,7 +208,6 @@ msec_t inicfg_set_duration_ms(struct config *root, const char *section, const ch

208 time_t inicfg_get_duration_seconds(struct config *root, const char *section, const char *name, time_t default_value);

209 time_t inicfg_set_duration_seconds(struct config *root, const char *section, const char *name, time_t value);

210

211 -unsigned inicfg_get_duration_days(struct config *root, const char *section, const char *name, unsigned default_value);

212 -unsigned inicfg_set_duration_days(struct config *root, const char *section, const char *name, unsigned value);

211 +time_t inicfg_get_duration_days_to_seconds(struct config *root, const char *section, const char *name, unsigned default_value_seconds);

212

213 #endif // LIBNETDATA_INICFG_H

src/libnetdata/inicfg/inicfg_api.c

+13 -17

@@ -166,13 +166,13 @@ msec_t inicfg_set_duration_ms(struct config *root, const char *section, const ch

166 return value;

167 }

168

169 -static STRING *reformat_duration_days(STRING *value) {

169 +static STRING *reformat_duration_days_to_seconds(STRING *value) {

170 int64_t result = 0;

171 - if(!duration_parse_days(string2str(value), &result))

171 + if(!duration_parse(string2str(value), &result, "d", "s"))

172 return value;

173

174 char buf[128];

175 - if(duration_snprintf_days(buf, sizeof(buf), result) > 0 && string_strcmp(value, buf) != 0) {

175 + if(duration_snprintf(buf, sizeof(buf), result, "s", false) > 0 && string_strcmp(value, buf) != 0) {

176 string_freez(value);

177 return string_strdupz(buf);

178 }

@@ -180,34 +180,30 @@ static STRING *reformat_duration_days(STRING *value) {

180 return value;

181 }

182

183 -unsigned inicfg_get_duration_days(struct config *root, const char *section, const char *name, unsigned default_value) {

183 +time_t inicfg_get_duration_days_to_seconds(struct config *root, const char *section, const char *name, unsigned default_value_seconds) {

184 char default_str[128];

185 - duration_snprintf_days(default_str, sizeof(default_str), (int)default_value);

185 + duration_snprintf(default_str, sizeof(default_str), (int)default_value_seconds, "s", false);

186

187 struct config_option *opt = inicfg_get_raw_value(

188 - root, section, name, default_str, CONFIG_VALUE_TYPE_DURATION_IN_DAYS, reformat_duration_days);

188 + root, section, name, default_str,

189 + CONFIG_VALUE_TYPE_DURATION_IN_DAYS_TO_SECONDS,

190 + reformat_duration_days_to_seconds);

191 +

192 if(!opt)

190 - return default_value;

193 + return default_value_seconds;

194

195 const char *s = string2str(opt->value);

196

197 int64_t result = 0;

195 - if(!duration_parse_days(s, &result)) {

196 - inicfg_set_raw_value(root, section, name, default_str, CONFIG_VALUE_TYPE_DURATION_IN_DAYS);

198 + if(!duration_parse(s, &result, "d", "s")) {

199 + inicfg_set_raw_value(root, section, name, default_str, CONFIG_VALUE_TYPE_DURATION_IN_DAYS_TO_SECONDS);

200 netdata_log_error("config option '[%s].%s = %s' is configured with an invalid duration", section, name, s);

198 - return default_value;

201 + return default_value_seconds;

202 }

203

204 return (unsigned)ABS(result);

205 }

206

204 -unsigned inicfg_set_duration_days(struct config *root, const char *section, const char *name, unsigned value) {

205 - char str[128];

206 - duration_snprintf_days(str, sizeof(str), value);

207 - inicfg_set_raw_value(root, section, name, str, CONFIG_VALUE_TYPE_DURATION_IN_DAYS);

208 - return value;

209 -}

210 -

207 long long inicfg_get_number(struct config *root, const char *section, const char *name, long long value) {

208 char buffer[100];

209 sprintf(buffer, "%lld", value);

src/libnetdata/inicfg/inicfg_conf_file.c

+1 -1

@@ -19,7 +19,7 @@ ENUM_STR_MAP_DEFINE(CONFIG_VALUE_TYPES) = {

19 { .id = CONFIG_VALUE_TYPE_BOOLEAN_ONDEMAND, .name ="yes, no, or auto", },

20 { .id = CONFIG_VALUE_TYPE_DURATION_IN_SECS, .name ="duration (seconds)", },

21 { .id = CONFIG_VALUE_TYPE_DURATION_IN_MS, .name ="duration (ms)", },

22 - { .id = CONFIG_VALUE_TYPE_DURATION_IN_DAYS, .name ="duration (days)", },

22 + { .id = CONFIG_VALUE_TYPE_DURATION_IN_DAYS_TO_SECONDS, .name ="duration (days)", },

23 { .id = CONFIG_VALUE_TYPE_SIZE_IN_BYTES, .name ="size (bytes)", },

24 { .id = CONFIG_VALUE_TYPE_SIZE_IN_MB, .name ="size (MiB)", },

25 };

src/libnetdata/inicfg/inicfg_internals.h

+1 -1

@@ -22,7 +22,7 @@ typedef enum __attribute__((packed)) {

22 CONFIG_VALUE_TYPE_BOOLEAN_ONDEMAND,

23 CONFIG_VALUE_TYPE_DURATION_IN_SECS,

24 CONFIG_VALUE_TYPE_DURATION_IN_MS,

25 - CONFIG_VALUE_TYPE_DURATION_IN_DAYS,

25 + CONFIG_VALUE_TYPE_DURATION_IN_DAYS_TO_SECONDS,

26 CONFIG_VALUE_TYPE_SIZE_IN_BYTES,

27 CONFIG_VALUE_TYPE_SIZE_IN_MB,

28 } CONFIG_VALUE_TYPES;

src/libnetdata/parsers/duration.c

+16 -4

@@ -82,7 +82,7 @@ inline int64_t duration_round_to_resolution(int64_t value, int64_t resolution) {

82 // -------------------------------------------------------------------------------------------------------------------

83 // parse a duration string

85 -bool duration_parse(const char *duration, int64_t *result, const char *default_unit) {

85 +bool duration_parse(const char *duration, int64_t *result, const char *default_unit, const char *output_unit) {

86 if (!duration || !*duration) {

87 *result = 0;

88 return false;

@@ -94,6 +94,12 @@ bool duration_parse(const char *duration, int64_t *result, const char *default_u

94 return false;

95 }

97 + const struct duration_unit *du_out = duration_find_unit(output_unit);

98 + if(!du_out) {

99 + *result = 0;

100 + return false;

101 + }

102 +

103 int64_t sign = 1;

104 const char *s = duration;

105 while (isspace((uint8_t)*s)) s++;

@@ -155,10 +161,16 @@ bool duration_parse(const char *duration, int64_t *result, const char *default_u

161

162 v *= sign;

163

158 - if(du_def->multiplier == 1)

164 + // Convert the final value from nanoseconds to the desired output unit

165 + // and apply appropriate rounding

166 + if(du_out->multiplier == 1)

167 *result = v;

160 - else

161 - *result = duration_round_to_resolution(v, du_def->multiplier);

168 + else {

169 + // First convert to the output unit

170 + NETDATA_DOUBLE converted = (NETDATA_DOUBLE)v / (NETDATA_DOUBLE)du_out->multiplier;

171 + // Then round to nearest integer in the output unit

172 + *result = (int64_t)round(converted);

173 + }

174

175 return true;

176 }

src/libnetdata/parsers/duration.h

+8 -8

@@ -8,14 +8,14 @@

8 int64_t duration_round_to_resolution(int64_t value, int64_t resolution);

10 // duration (string to number)

11 -bool duration_parse(const char *duration, int64_t *result, const char *default_unit);

12 -#define duration_parse_nsec_t(duration, ns_ptr) duration_parse(duration, ns_ptr, "ns")

13 -#define duration_parse_usec_t(duration, us_ptr) duration_parse(duration, us_ptr, "us")

14 -#define duration_parse_msec_t(duration, ms_ptr) duration_parse(duration, ms_ptr, "ms")

15 -#define duration_parse_time_t(duration, secs_ptr) duration_parse(duration, secs_ptr, "s")

16 -#define duration_parse_mins(duration, mins_ptr) duration_parse(duration, mins_ptr, "m")

17 -#define duration_parse_hours(duration, hours_ptr) duration_parse(duration, hours_ptr, "h")

18 -#define duration_parse_days(duration, days_ptr) duration_parse(duration, days_ptr, "d")

11 +bool duration_parse(const char *duration, int64_t *result, const char *default_unit, const char *output_unit);

12 +#define duration_parse_nsec_t(duration, ns_ptr) duration_parse(duration, ns_ptr, "ns", "ns")

13 +#define duration_parse_usec_t(duration, us_ptr) duration_parse(duration, us_ptr, "us", "us")

14 +#define duration_parse_msec_t(duration, ms_ptr) duration_parse(duration, ms_ptr, "ms", "ms")

15 +#define duration_parse_time_t(duration, secs_ptr) duration_parse(duration, secs_ptr, "s", "s")

16 +#define duration_parse_mins(duration, mins_ptr) duration_parse(duration, mins_ptr, "m", "m")

17 +#define duration_parse_hours(duration, hours_ptr) duration_parse(duration, hours_ptr, "h", "h")

18 +#define duration_parse_days(duration, days_ptr) duration_parse(duration, days_ptr, "d", "d")

20 // duration (number to string)

21 ssize_t duration_snprintf(char *dst, size_t dst_size, int64_t value, const char *unit, bool add_spaces);

src/libnetdata/uuid/uuid.h

@@ -36,6 +36,7 @@ ND_UUID_DEFINE(sensors_state_transition_msgid, 0x8d, 0xda, 0xf5, 0xba, 0x33, 0xa

36 ND_UUID_DEFINE(log_flood_protection_msgid, 0xec, 0x87, 0xa5, 0x61, 0x20, 0xd5, 0x43, 0x1b, 0xac, 0xe5, 0x1e, 0x2f, 0xb8, 0xbb, 0xa2, 0x43);

37 ND_UUID_DEFINE(netdata_startup_msgid, 0x1e, 0x60, 0x61, 0xa9, 0xfb, 0xd4, 0x45, 0x01, 0xb3, 0xcc, 0xc3, 0x68, 0x11, 0x9f, 0x2b, 0x69);

38 ND_UUID_DEFINE(aclk_connection_msgid, 0xac, 0xb3, 0x3c, 0xb9, 0x57, 0x78, 0x47, 0x6b, 0xaa, 0xc7, 0x02, 0xeb, 0x7e, 0x4e, 0x15, 0x1d);

39 +ND_UUID_DEFINE(extreme_cardinality_msgid, 0xd1, 0xf5, 0x96, 0x06, 0xdd, 0x4d, 0x41, 0xe3, 0xb2, 0x17, 0xa0, 0xcf, 0xca, 0xe8, 0xe6, 0x32);

41 ND_UUID UUID_generate_from_hash(const void *payload, size_t payload_len);

src/registry/registry_init.c

+2 -1

@@ -93,7 +93,8 @@ int registry_init(void) {

94 // configuration options

95 registry.save_registry_every_entries = (unsigned long long)inicfg_get_number(&netdata_config, CONFIG_SECTION_REGISTRY, "registry save db every new entries", 1000000);

96 - registry.persons_expiration = inicfg_get_duration_days(&netdata_config, CONFIG_SECTION_REGISTRY, "registry expire idle persons", 365) * 86400;

96 + registry.persons_expiration = inicfg_get_duration_days_to_seconds(

97 + &netdata_config, CONFIG_SECTION_REGISTRY, "registry expire idle persons", 365 * 86400);

98 registry.registry_domain = inicfg_get(&netdata_config, CONFIG_SECTION_REGISTRY, "registry domain", "");

99 registry.registry_to_announce = inicfg_get(&netdata_config, CONFIG_SECTION_REGISTRY, "registry to announce", "https://registry.my-netdata.io");

100 registry.hostname = inicfg_get(&netdata_config, CONFIG_SECTION_REGISTRY, "registry hostname", netdata_configured_hostname);