@cryptotaxi247 / netdata-1 / commits / cb7af25c0

RRD structures managed by dictionaries (#13646)

* rrdset - in progress * rrdset optimal constructor; rrdset conflict * rrdset final touches * re-organization of rrdset object members * prevent use-after-free * dictionary dfe supports also counting of iterations * rrddim managed by dictionary * rrd.h cleanup * DICTIONARY_ITEM now is referencing actual dictionary items in the code * removed rrdset linked list * Revert "removed rrdset linked list" This reverts commit 690d6a588b4b99619c2c5e10f84e8f868ae6def5. * removed rrdset linked list * added comments * Switch chart uuid to static allocation in rrdset Remove unused functions * rrdset_archive() and friends... * always create rrdfamily * enable ml_free_dimension * rrddim_foreach done with dfe * most custom rrddim loops replaced with rrddim_foreach * removed accesses to rrddim->dimensions * removed locks that are no longer needed * rrdsetvar is now managed by the dictionary * set rrdset is rrdsetvar, fixes https://github.com/netdata/netdata/pull/13646#issuecomment-1242574853 * conflict callback of rrdsetvar now properly checks if it has to reset the variable * dictionary registered callbacks accept as first parameter the DICTIONARY_ITEM * dictionary dfe now uses internal counter to report; avoided excess variables defined with dfe * dictionary walkthrough callbacks get dictionary acquired items * dictionary reference counters that can be dupped from zero * added advanced functions for get and del * rrdvar managed by dictionaries * thread safety for rrdsetvar * faster rrdvar initialization * rrdvar string lengths should match in all add, del, get functions * rrdvar internals hidden from the rest of the world * rrdvar is now acquired throughout netdata * hide the internal structures of rrdsetvar * rrdsetvar is now acquired through out netdata * rrddimvar managed by dictionary; rrddimvar linked list removed; rrddimvar structures hidden from the rest of netdata * better error handling * dont create variables if not initialized for health * dont create variables if not initialized for health again * rrdfamily is now managed by dictionaries; references of it are acquired dictionary items * type checking on acquired objects * rrdcalc renaming of functions * type checking for rrdfamily_acquired * rrdcalc managed by dictionaries * rrdcalc double free fix * host rrdvars is always needed * attempt to fix deadlock 1 * attempt to fix deadlock 2 * Remove unused variable * attempt to fix deadlock 3 * snprintfz * rrdcalc index in rrdset fix * Stop storing active charts and computing chart hashes * Remove store active chart function * Remove compute chart hash function * Remove sql_store_chart_hash function * Remove store_active_dimension function * dictionary delayed destruction * formatting and cleanup * zero dictionary base on rrdsetvar * added internal error to log delayed destructions of dictionaries * typo in rrddimvar * added debugging info to dictionary * debug info * fix for rrdcalc keys being empty * remove forgotten unlock * remove deadlock * Switch to metadata version 5 and drop chart_hash chart_hash_map chart_active dimension_active v_chart_hash * SQL cosmetic changes * do not busy wait while destroying a referenced dictionary * remove deadlock * code cleanup; re-organization; * fast cleanup and flushing of dictionaries * number formatting fixes * do not delete configured alerts when archiving a chart * rrddim obsolete linked list management outside dictionaries * removed duplicate contexts call * fix crash when rrdfamily is not initialized * dont keep rrddimvar referenced * properly cleanup rrdvar * removed some locks * Do not attempt to cleanup chart_hash / chart_hash_map * rrdcalctemplate managed by dictionary * register callbacks on the right dictionary * removed some more locks * rrdcalc secondary index replaced with linked-list; rrdcalc labels updates are now executed by health thread * when looking up for an alarm look using both chart id and chart name * host initialization a bit more modular * init rrdlabels on host update * preparation for dictionary views * improved comment * unused variables without internal checks * service threads isolation and worker info * more worker info in service thread * thread cancelability debugging with internal checks * strings data races addressed; fixes https://github.com/netdata/netdata/issues/13647 * dictionary modularization * Remove unused SQL statement definition * unit-tested thread safety of dictionaries; removed data race conditions on dictionaries and strings; dictionaries now can detect if the caller is holds a write lock and automatically all the calls become their unsafe versions; all direct calls to unsafe version is eliminated * remove worker_is_idle() from the exit of service functions, because we lose the lock time between loops * rewritten dictionary to have 2 separate locks, one for indexing and another for traversal * Update collectors/cgroups.plugin/sys_fs_cgroup.c Co-authored-by: Vladimir Kobal <vlad@prokk.net> * Update collectors/cgroups.plugin/sys_fs_cgroup.c Co-authored-by: Vladimir Kobal <vlad@prokk.net> * Update collectors/proc.plugin/proc_net_dev.c Co-authored-by: Vladimir Kobal <vlad@prokk.net> * fix memory leak in rrdset cache_dir * minor dictionary changes * dont use index locks in single threaded * obsolete dict option * rrddim options and flags separation; rrdset_done() optimization to keep array of reference pointers to rrddim; * fix jump on uninitialized value in dictionary; remove double free of cache_dir * addressed codacy findings * removed debugging code * use the private refcount on dictionaries * make dictionary item desctructors work on dictionary destruction; strictier control on dictionary API; proper cleanup sequence on rrddim; * more dictionary statistics * global statistics about dictionary operations, memory, items, callbacks * dictionary support for views - missing the public API * removed warning about unused parameter * chart and context name for cloud * chart and context name for cloud, again * dictionary statistics fixed; first implementation of dictionary views - not currently used * only the master can globally delete an item * context needs netdata prefix * fix context and chart it of spins * fix for host variables when health is not enabled * run garbage collector on item insert too * Fix info message; remove extra "using" * update dict unittest for new placement of garbage collector * we need RRDHOST->rrdvars for maintaining custom host variables * Health initialization needs the host->host_uuid * split STRING to its own files; no code changes other than that * initialize health unconditionally * unit tests do not pollute the global scope with their variables * Skip initialization when creating archived hosts on startup. When a child connects it will initialize properly Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com> Co-authored-by: Vladimir Kobal <vlad@prokk.net>

Costa Tsaousis committed Sep 19, 2022 at 23:46 UTC cb7af25c09d8775d1967cb0553268075cda868d4
106 files changed +7032 -5206
CMakeLists.txt
+19 -12
@@ -220,7 +220,11 @@ pkg_check_modules(CURL libcurl)
220 # -----------------------------------------------------------------------------
221 # Detect libcups
222
223 -pkg_check_modules(CURL libcups)
223 +pkg_check_modules(CUPS libcups)
224 +IF(NOT CUPS_LIBRARIES)
225 + pkg_check_modules(CUPS cups)
226 +ENDIF()
227 +
228 # later we use:
229 # ${CUPS_LIBRARIES}
230 # ${CUPS_CFLAGS_OTHER}
@@ -446,6 +450,8 @@ set(LIBNETDATA_FILES
450 libnetdata/avl/avl.h
451 libnetdata/buffer/buffer.c
452 libnetdata/buffer/buffer.h
453 + libnetdata/circular_buffer/circular_buffer.c
454 + libnetdata/circular_buffer/circular_buffer.h
455 libnetdata/clocks/clocks.c
456 libnetdata/clocks/clocks.h
457 libnetdata/completion/completion.c
@@ -454,10 +460,15 @@ set(LIBNETDATA_FILES
460 libnetdata/dictionary/dictionary.h
461 libnetdata/eval/eval.c
462 libnetdata/eval/eval.h
463 + libnetdata/health/health.c
464 + libnetdata/health/health.h
465 libnetdata/inlined.h
466 + libnetdata/json/json.c
467 + libnetdata/json/json.h
468 + libnetdata/json/jsmn.c
469 + libnetdata/json/jsmn.h
470 libnetdata/libnetdata.c
471 libnetdata/libnetdata.h
460 - libnetdata/required_dummies.h
472 libnetdata/locks/locks.c
473 libnetdata/locks/locks.h
474 libnetdata/log/log.c
@@ -470,6 +481,9 @@ set(LIBNETDATA_FILES
481 libnetdata/popen/popen.h
482 libnetdata/procfile/procfile.c
483 libnetdata/procfile/procfile.h
484 + libnetdata/required_dummies.h
485 + libnetdata/socket/security.c
486 + libnetdata/socket/security.h
487 libnetdata/simple_pattern/simple_pattern.c
488 libnetdata/simple_pattern/simple_pattern.h
489 libnetdata/socket/socket.c
@@ -478,23 +492,16 @@ set(LIBNETDATA_FILES
492 libnetdata/statistical/statistical.h
493 libnetdata/storage_number/storage_number.c
494 libnetdata/storage_number/storage_number.h
495 + libnetdata/string/string.c
496 + libnetdata/string/string.h
497 libnetdata/threads/threads.c
498 libnetdata/threads/threads.h
499 libnetdata/url/url.c
500 libnetdata/url/url.h
485 - libnetdata/json/json.c
486 - libnetdata/json/json.h
487 - libnetdata/json/jsmn.c
488 - libnetdata/json/jsmn.h
489 - libnetdata/health/health.c
490 - libnetdata/health/health.h
501 libnetdata/string/utf8.h
492 - libnetdata/socket/security.c
493 - libnetdata/socket/security.h
502 libnetdata/worker_utilization/worker_utilization.c
503 libnetdata/worker_utilization/worker_utilization.h
496 - libnetdata/circular_buffer/circular_buffer.c
497 - libnetdata/circular_buffer/circular_buffer.h)
504 + )
505
506 IF(ENABLE_PLUGIN_EBPF)
507 list(APPEND LIBNETDATA_FILES
Makefile.am
+2
@@ -173,6 +173,8 @@ LIBNETDATA_FILES = \
173 libnetdata/socket/security.h \
174 libnetdata/statistical/statistical.c \
175 libnetdata/statistical/statistical.h \
176 + libnetdata/string/string.c \
177 + libnetdata/string/string.h \
178 libnetdata/storage_number/storage_number.c \
179 libnetdata/storage_number/storage_number.h \
180 libnetdata/threads/threads.c \
collectors/cgroups.plugin/sys_fs_cgroup.c
+11 -12
@@ -857,16 +857,16 @@ struct cgroup {
857 char *filename_cpu_cfs_quota;
858 unsigned long long cpu_cfs_quota;
859
860 - RRDSETVAR *chart_var_cpu_limit;
860 + const RRDSETVAR_ACQUIRED *chart_var_cpu_limit;
861 NETDATA_DOUBLE prev_cpu_usage;
862
863 char *filename_memory_limit;
864 unsigned long long memory_limit;
865 - RRDSETVAR *chart_var_memory_limit;
865 + const RRDSETVAR_ACQUIRED *chart_var_memory_limit;
866
867 char *filename_memoryswap_limit;
868 unsigned long long memoryswap_limit;
869 - RRDSETVAR *chart_var_memoryswap_limit;
869 + const RRDSETVAR_ACQUIRED *chart_var_memoryswap_limit;
870
871 // services
872 RRDDIM *rd_cpu;
@@ -3708,10 +3708,10 @@ cpu_limits2_err:
3708 }
3709 }
3710
3711 -static inline int update_memory_limits(char **filename, RRDSETVAR **chart_var, unsigned long long *value, const char *chart_var_name, struct cgroup *cg) {
3711 +static inline int update_memory_limits(char **filename, const RRDSETVAR_ACQUIRED **chart_var, unsigned long long *value, const char *chart_var_name, struct cgroup *cg) {
3712 if(*filename) {
3713 if(unlikely(!*chart_var)) {
3714 - *chart_var = rrdsetvar_custom_chart_variable_create(cg->st_mem_usage, chart_var_name);
3714 + *chart_var = rrdsetvar_custom_chart_variable_add_and_acquire(cg->st_mem_usage, chart_var_name);
3715 if(!*chart_var) {
3716 error("Cannot create cgroup %s chart variable '%s'. Will not update its limit anymore.", cg->id, chart_var_name);
3717 freez(*filename);
@@ -3727,7 +3727,7 @@ static inline int update_memory_limits(char **filename, RRDSETVAR **chart_var, u
3727 *filename = NULL;
3728 }
3729 else {
3730 - rrdsetvar_custom_chart_variable_set(*chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
3730 + rrdsetvar_custom_chart_variable_set(cg->st_mem_usage, *chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
3731 return 1;
3732 }
3733 } else {
@@ -3742,11 +3742,11 @@ static inline int update_memory_limits(char **filename, RRDSETVAR **chart_var, u
3742 char *s = "max\n\0";
3743 if(strcmp(s, buffer) == 0){
3744 *value = UINT64_MAX;
3745 - rrdsetvar_custom_chart_variable_set(*chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
3745 + rrdsetvar_custom_chart_variable_set(cg->st_mem_usage, *chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
3746 return 1;
3747 }
3748 *value = str2ull(buffer);
3749 - rrdsetvar_custom_chart_variable_set(*chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
3749 + rrdsetvar_custom_chart_variable_set(cg->st_mem_usage, *chart_var, (NETDATA_DOUBLE)(*value / (1024 * 1024)));
3750 return 1;
3751 }
3752 }
@@ -3843,7 +3843,7 @@ void update_cgroup_charts(int update_every) {
3843 }
3844
3845 if(unlikely(!cg->chart_var_cpu_limit)) {
3846 - cg->chart_var_cpu_limit = rrdsetvar_custom_chart_variable_create(cg->st_cpu, "cpu_limit");
3846 + cg->chart_var_cpu_limit = rrdsetvar_custom_chart_variable_add_and_acquire(cg->st_cpu, "cpu_limit");
3847 if(!cg->chart_var_cpu_limit) {
3848 error("Cannot create cgroup %s chart variable 'cpu_limit'. Will not update its limit anymore.", cg->id);
3849 if(cg->filename_cpuset_cpus) freez(cg->filename_cpuset_cpus);
@@ -3868,8 +3868,6 @@ void update_cgroup_charts(int update_every) {
3868 value = (NETDATA_DOUBLE)cg->cpuset_cpus * 100;
3869 }
3870 if(likely(value)) {
3871 - rrdsetvar_custom_chart_variable_set(cg->chart_var_cpu_limit, value);
3872 -
3871 if(unlikely(!cg->st_cpu_limit)) {
3872 snprintfz(title, CHART_TITLE_MAX, "CPU Usage within the limits");
3873
@@ -3909,14 +3907,15 @@ void update_cgroup_charts(int update_every) {
3907
3908 cg->prev_cpu_usage = cpu_usage;
3909
3910 + rrdsetvar_custom_chart_variable_set(cg->st_cpu, cg->chart_var_cpu_limit, value);
3911 rrdset_done(cg->st_cpu_limit);
3912 }
3913 else {
3915 - rrdsetvar_custom_chart_variable_set(cg->chart_var_cpu_limit, NAN);
3914 if(unlikely(cg->st_cpu_limit)) {
3915 rrdset_is_obsolete(cg->st_cpu_limit);
3916 cg->st_cpu_limit = NULL;
3917 }
3918 + rrdsetvar_custom_chart_variable_set(cg->st_cpu, cg->chart_var_cpu_limit, NAN);
3919 }
3920 }
3921 }
collectors/cgroups.plugin/tests/test_doubles.c
-8
@@ -146,14 +146,6 @@ void netdev_rename_device_del(const char *host_device)
146 UNUSED(host_device);
147 }
148
149 -void sql_store_chart_label(uuid_t *chart_uuid, int source_type, char *label, char *value)
150 -{
151 - UNUSED(chart_uuid);
152 - UNUSED(source_type);
153 - UNUSED(label);
154 - UNUSED(value);
155 -}
156 -
149 void rrdcalc_update_rrdlabels(RRDSET *st) {
150 (void)st;
151 }
collectors/cups.plugin/cups_plugin.c
+5 -8
@@ -137,10 +137,7 @@ getIntegerOption(
137 return ((int)intvalue);
138 }
139
140 -static int reset_job_metrics(const char *name, void *entry, void *data) {
141 - (void)name;
142 - (void)data;
143 -
140 +static int reset_job_metrics(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data __maybe_unused) {
141 struct job_metrics *jm = (struct job_metrics *)entry;
142
143 jm->is_collected = 0;
@@ -175,8 +172,8 @@ struct job_metrics *get_job_metrics(char *dest) {
172 return jm;
173 }
174
178 -int collect_job_metrics(const char *name, void *entry, void *data) {
179 - (void)data;
175 +int collect_job_metrics(const DICTIONARY_ITEM *item, void *entry, void *data __maybe_unused) {
176 + const char *name = dictionary_acquired_item_name(item);
177
178 struct job_metrics *jm = (struct job_metrics *)entry;
179
@@ -205,7 +202,7 @@ int collect_job_metrics(const char *name, void *entry, void *data) {
202 printf("DIMENSION pending '' absolute 1 1\n");
203 printf("DIMENSION held '' absolute 1 1\n");
204 printf("DIMENSION processing '' absolute 1 1\n");
208 - dictionary_del_having_write_lock(dict_dest_job_metrics, name);
205 + dictionary_del(dict_dest_job_metrics, name);
206 }
207
208 return 0;
@@ -243,7 +240,7 @@ int main(int argc, char **argv) {
240
241 errno = 0;
242
246 - dict_dest_job_metrics = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
243 + dict_dest_job_metrics = dictionary_create(DICT_OPTION_SINGLE_THREADED);
244
245 // ------------------------------------------------------------------------
246 // the main loop
collectors/diskspace.plugin/plugin_diskspace.c
+3 -3
@@ -95,8 +95,8 @@ int mount_point_cleanup(const char *name, void *entry, int slow) {
95 return 0;
96 }
97
98 -int mount_point_cleanup_cb(const char *name, void *entry, void *data) {
99 - UNUSED(data);
98 +int mount_point_cleanup_cb(const DICTIONARY_ITEM *item, void *entry, void *data __maybe_unused) {
99 + const char *name = dictionary_acquired_item_name(item);
100
101 return mount_point_cleanup(name, (struct mount_point_metadata *)entry, 0);
102 }
@@ -330,7 +330,7 @@ static inline void do_disk_space_stats(struct mountinfo *mi, int update_every) {
330 , SIMPLE_PATTERN_EXACT
331 );
332
333 - dict_mountpoints = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
333 + dict_mountpoints = dictionary_create(DICT_OPTION_SINGLE_THREADED);
334 }
335
336 struct mount_point_metadata *m = dictionary_get(dict_mountpoints, mi->mount_point);
collectors/plugins.d/pluginsd_parser.c
+15 -11
@@ -101,15 +101,19 @@ PARSER_RC pluginsd_variable_action(void *user, RRDHOST *host, RRDSET *st, char *
101 UNUSED(user);
102
103 if (global) {
104 - RRDVAR *rv = rrdvar_custom_host_variable_create(host, name);
105 - if (rv)
106 - rrdvar_custom_host_variable_set(host, rv, value);
104 + const RRDVAR_ACQUIRED *rva = rrdvar_custom_host_variable_add_and_acquire(host, name);
105 + if (rva) {
106 + rrdvar_custom_host_variable_set(host, rva, value);
107 + rrdvar_custom_host_variable_release(host, rva);
108 + }
109 else
110 error("cannot find/create HOST VARIABLE '%s' on host '%s'", name, rrdhost_hostname(host));
111 } else {
110 - RRDSETVAR *rs = rrdsetvar_custom_chart_variable_create(st, name);
111 - if (rs)
112 - rrdsetvar_custom_chart_variable_set(rs, value);
112 + const RRDSETVAR_ACQUIRED *rsa = rrdsetvar_custom_chart_variable_add_and_acquire(st, name);
113 + if (rsa) {
114 + rrdsetvar_custom_chart_variable_set(st, rsa, value);
115 + rrdsetvar_custom_chart_variable_release(st, rsa);
116 + }
117 else
118 error("cannot find/create CHART VARIABLE '%s' on host '%s', chart '%s'", name, rrdhost_hostname(host), rrdset_id(st));
119 }
@@ -127,7 +131,7 @@ PARSER_RC pluginsd_dimension_action(void *user, RRDSET *st, char *id, char *name
131 RRDDIM *rd = rrddim_add(st, id, name, multiplier, divisor, algorithm_type);
132 int unhide_dimension = 1;
133
130 - rrddim_flag_clear(rd, RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS);
134 + rrddim_option_clear(rd, RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS);
135 if (options && *options) {
136 if (strstr(options, "obsolete") != NULL)
137 rrddim_is_obsolete(st, rd);
@@ -137,20 +141,20 @@ PARSER_RC pluginsd_dimension_action(void *user, RRDSET *st, char *id, char *name
141 unhide_dimension = !strstr(options, "hidden");
142
143 if (strstr(options, "noreset") != NULL)
140 - rrddim_flag_set(rd, RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS);
144 + rrddim_option_set(rd, RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS);
145 if (strstr(options, "nooverflow") != NULL)
142 - rrddim_flag_set(rd, RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS);
146 + rrddim_option_set(rd, RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS);
147 } else
148 rrddim_isnot_obsolete(st, rd);
149
150 if (likely(unhide_dimension)) {
147 - rrddim_flag_clear(rd, RRDDIM_FLAG_HIDDEN);
151 + rrddim_option_clear(rd, RRDDIM_OPTION_HIDDEN);
152 if (rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN)) {
153 (void)sql_set_dimension_option(&rd->metric_uuid, NULL);
154 rrddim_flag_clear(rd, RRDDIM_FLAG_META_HIDDEN);
155 }
156 } else {
153 - rrddim_flag_set(rd, RRDDIM_FLAG_HIDDEN);
157 + rrddim_option_set(rd, RRDDIM_OPTION_HIDDEN);
158 if (!rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN)) {
159 (void)sql_set_dimension_option(&rd->metric_uuid, "hidden");
160 rrddim_flag_set(rd, RRDDIM_FLAG_META_HIDDEN);
collectors/proc.plugin/ipc.c
+4 -8
@@ -281,7 +281,7 @@ int do_ipc(int update_every, usec_t dt) {
281 static int read_limits_next = -1;
282 static struct ipc_limits limits;
283 static struct ipc_status status;
284 - static RRDVAR *arrays_max = NULL, *semaphores_max = NULL;
284 + static const RRDVAR_ACQUIRED *arrays_max = NULL, *semaphores_max = NULL;
285 static RRDSET *st_semaphores = NULL, *st_arrays = NULL;
286 static RRDDIM *rd_semaphores = NULL, *rd_arrays = NULL;
287 static char *msg_filename = NULL;
@@ -352,8 +352,8 @@ int do_ipc(int update_every, usec_t dt) {
352 }
353
354 // variables
355 - semaphores_max = rrdvar_custom_host_variable_create(localhost, "ipc_semaphores_max");
356 - arrays_max = rrdvar_custom_host_variable_create(localhost, "ipc_semaphores_arrays_max");
355 + semaphores_max = rrdvar_custom_host_variable_add_and_acquire(localhost, "ipc_semaphores_max");
356 + arrays_max = rrdvar_custom_host_variable_add_and_acquire(localhost, "ipc_semaphores_arrays_max");
357 }
358
359 struct stat stbuf;
@@ -483,11 +483,7 @@ int do_ipc(int update_every, usec_t dt) {
483 rrdset_done(st_msq_messages);
484 rrdset_done(st_msq_bytes);
485
486 - long long dimensions_num = 0;
487 - RRDDIM *rd;
488 - rrdset_rdlock(st_msq_messages);
489 - rrddim_foreach_read(rd, st_msq_messages) dimensions_num++;
490 - rrdset_unlock(st_msq_messages);
486 + long long dimensions_num = rrdset_number_of_dimensions(st_msq_messages);
487
488 if(unlikely(dimensions_num > dimensions_limit)) {
489 info("Message queue statistics has been disabled");
collectors/proc.plugin/proc_diskstats.c
-2
@@ -874,8 +874,6 @@ static void add_labels_to_disk(struct disk *d, RRDSET *st) {
874 rrdlabels_add(st->rrdlabels, "device_type", "virtual", RRDLABEL_SRC_AUTO);
875 break;
876 }
877 -
878 - rrdcalc_update_rrdlabels(st);
877 }
878
879 int do_proc_diskstats(int update_every, usec_t dt) {
collectors/proc.plugin/proc_interrupts.c
+2 -2
@@ -175,7 +175,7 @@ int do_proc_interrupts(int update_every, usec_t dt) {
175 // calls of this function.
176 if(unlikely(!irr->rd || strncmp(rrddim_name(irr->rd), irr->name, MAX_INTERRUPT_NAME) != 0)) {
177 irr->rd = rrddim_add(st_system_interrupts, irr->id, irr->name, 1, 1, RRD_ALGORITHM_INCREMENTAL);
178 - rrddim_set_name(st_system_interrupts, irr->rd, irr->name);
178 + rrddim_reset_name(st_system_interrupts, irr->rd, irr->name);
179
180 // also reset per cpu RRDDIMs to avoid repeating strncmp() in the per core loop
181 if(likely(do_per_core != CONFIG_BOOLEAN_NO)) {
@@ -237,7 +237,7 @@ int do_proc_interrupts(int update_every, usec_t dt) {
237 if(irr->used && (do_per_core == CONFIG_BOOLEAN_YES || irr->cpu[c].value)) {
238 if(unlikely(!irr->cpu[c].rd)) {
239 irr->cpu[c].rd = rrddim_add(core_st[c], irr->id, irr->name, 1, 1, RRD_ALGORITHM_INCREMENTAL);
240 - rrddim_set_name(core_st[c], irr->cpu[c].rd, irr->name);
240 + rrddim_reset_name(core_st[c], irr->cpu[c].rd, irr->name);
241 }
242
243 rrddim_set_by_pointer(core_st[c], irr->cpu[c].rd, irr->cpu[c].value);
collectors/proc.plugin/proc_loadavg.c
+3 -3
@@ -99,7 +99,7 @@ int do_proc_loadavg(int update_every, usec_t dt) {
99 if(likely(do_all_processes)) {
100 static RRDSET *processes_chart = NULL;
101 static RRDDIM *rd_active = NULL;
102 - static RRDSETVAR *rd_pidmax;
102 + static const RRDSETVAR_ACQUIRED *rd_pidmax;
103
104 if(unlikely(!processes_chart)) {
105 processes_chart = rrdset_create_localhost(
@@ -118,12 +118,12 @@ int do_proc_loadavg(int update_every, usec_t dt) {
118 );
119
120 rd_active = rrddim_add(processes_chart, "active", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
121 - rd_pidmax = rrdsetvar_custom_chart_variable_create(processes_chart, "pidmax");
121 + rd_pidmax = rrdsetvar_custom_chart_variable_add_and_acquire(processes_chart, "pidmax");
122 }
123 else rrdset_next(processes_chart);
124
125 rrddim_set_by_pointer(processes_chart, rd_active, active_processes);
126 - rrdsetvar_custom_chart_variable_set(rd_pidmax, max_processes);
126 + rrdsetvar_custom_chart_variable_set(processes_chart, rd_pidmax, max_processes);
127 rrdset_done(processes_chart);
128 }
129
collectors/proc.plugin/proc_mdstat.c
-1
@@ -80,7 +80,6 @@ static inline void make_chart_obsolete(char *name, const char *id_modifier)
80 static void add_labels_to_mdstat(struct raid *raid, RRDSET *st) {
81 rrdlabels_add(st->rrdlabels, "device", raid->name, RRDLABEL_SRC_AUTO);
82 rrdlabels_add(st->rrdlabels, "raid_level", raid->level, RRDLABEL_SRC_AUTO);
83 - rrdcalc_update_rrdlabels(st);
83 }
84
85 int do_proc_mdstat(int update_every, usec_t dt)
collectors/proc.plugin/proc_net_dev.c
+5 -4
@@ -188,7 +188,7 @@ static struct netdev {
188 RRDDIM *rd_mtu;
189
190 char *filename_speed;
191 - RRDSETVAR *chart_var_speed;
191 + const RRDSETVAR_ACQUIRED *chart_var_speed;
192
193 char *filename_duplex;
194 char *filename_operstate;
@@ -967,7 +967,8 @@ int do_proc_net_dev(int update_every, usec_t dt) {
967 // update the interface speed
968 if(d->filename_speed) {
969 if(unlikely(!d->chart_var_speed)) {
970 - d->chart_var_speed = rrdsetvar_custom_chart_variable_create(d->st_bandwidth, "nic_speed_max");
970 + d->chart_var_speed =
971 + rrdsetvar_custom_chart_variable_add_and_acquire(d->st_bandwidth, "nic_speed_max");
972 if(!d->chart_var_speed) {
973 error("Cannot create interface %s chart variable 'nic_speed_max'. Will not update its speed anymore.", d->name);
974 freez(d->filename_speed);
@@ -990,8 +991,6 @@ int do_proc_net_dev(int update_every, usec_t dt) {
991 d->filename_speed = NULL;
992 }
993 else {
993 - rrdsetvar_custom_chart_variable_set(d->chart_var_speed, (NETDATA_DOUBLE) d->speed * KILOBITS_IN_A_MEGABIT);
994 -
994 if(d->do_speed != CONFIG_BOOLEAN_NO) {
995 if(unlikely(!d->st_speed)) {
996 d->st_speed = rrdset_create_localhost(
@@ -1020,6 +1019,8 @@ int do_proc_net_dev(int update_every, usec_t dt) {
1019 rrddim_set_by_pointer(d->st_speed, d->rd_speed, (collected_number)d->speed * KILOBITS_IN_A_MEGABIT);
1020 rrdset_done(d->st_speed);
1021 }
1022 +
1023 + rrdsetvar_custom_chart_variable_set(d->st_bandwidth, d->chart_var_speed, (NETDATA_DOUBLE) d->speed * KILOBITS_IN_A_MEGABIT);
1024 }
1025 }
1026 }
collectors/proc.plugin/proc_net_snmp.c
+2 -2
@@ -104,7 +104,7 @@ int do_proc_net_snmp(int update_every, usec_t dt) {
104 *arl_udp = NULL,
105 *arl_udplite = NULL;
106
107 - static RRDVAR *tcp_max_connections_var = NULL;
107 + static const RRDVAR_ACQUIRED *tcp_max_connections_var = NULL;
108
109 if(unlikely(!arl_ip)) {
110 do_ip_packets = config_get_boolean_ondemand("plugin:proc:/proc/net/snmp", "ipv4 packets", CONFIG_BOOLEAN_AUTO);
@@ -216,7 +216,7 @@ int do_proc_net_snmp(int update_every, usec_t dt) {
216 arl_expect(arl_udplite, "InCsumErrors", &snmp_root.udplite_InCsumErrors);
217 arl_expect(arl_udplite, "IgnoredMulti", &snmp_root.udplite_IgnoredMulti);
218
219 - tcp_max_connections_var = rrdvar_custom_host_variable_create(localhost, "tcp_max_connections");
219 + tcp_max_connections_var = rrdvar_custom_host_variable_add_and_acquire(localhost, "tcp_max_connections");
220 }
221
222 if(unlikely(!ff)) {
collectors/proc.plugin/proc_net_sockstat.c
+6 -6
@@ -27,14 +27,14 @@ static struct proc_net_sockstat {
27
28 static int read_tcp_mem(void) {
29 static char *filename = NULL;
30 - static RRDVAR *tcp_mem_low_threshold = NULL,
30 + static const RRDVAR_ACQUIRED *tcp_mem_low_threshold = NULL,
31 *tcp_mem_pressure_threshold = NULL,
32 *tcp_mem_high_threshold = NULL;
33
34 if(unlikely(!tcp_mem_low_threshold)) {
35 - tcp_mem_low_threshold = rrdvar_custom_host_variable_create(localhost, "tcp_mem_low");
36 - tcp_mem_pressure_threshold = rrdvar_custom_host_variable_create(localhost, "tcp_mem_pressure");
37 - tcp_mem_high_threshold = rrdvar_custom_host_variable_create(localhost, "tcp_mem_high");
35 + tcp_mem_low_threshold = rrdvar_custom_host_variable_add_and_acquire(localhost, "tcp_mem_low");
36 + tcp_mem_pressure_threshold = rrdvar_custom_host_variable_add_and_acquire(localhost, "tcp_mem_pressure");
37 + tcp_mem_high_threshold = rrdvar_custom_host_variable_add_and_acquire(localhost, "tcp_mem_high");
38 }
39
40 if(unlikely(!filename)) {
@@ -69,7 +69,7 @@ static int read_tcp_mem(void) {
69
70 static kernel_uint_t read_tcp_max_orphans(void) {
71 static char *filename = NULL;
72 - static RRDVAR *tcp_max_orphans_var = NULL;
72 + static const RRDVAR_ACQUIRED *tcp_max_orphans_var = NULL;
73
74 if(unlikely(!filename)) {
75 char buffer[FILENAME_MAX + 1];
@@ -81,7 +81,7 @@ static kernel_uint_t read_tcp_max_orphans(void) {
81 if(read_single_number_file(filename, &tcp_max_orphans) == 0) {
82
83 if(unlikely(!tcp_max_orphans_var))
84 - tcp_max_orphans_var = rrdvar_custom_host_variable_create(localhost, "tcp_max_orphans");
84 + tcp_max_orphans_var = rrdvar_custom_host_variable_add_and_acquire(localhost, "tcp_max_orphans");
85
86 rrdvar_custom_host_variable_set(localhost, tcp_max_orphans_var, tcp_max_orphans);
87 return tcp_max_orphans;
collectors/proc.plugin/proc_net_stat_conntrack.c
+2 -2
@@ -12,7 +12,7 @@ int do_proc_net_stat_conntrack(int update_every, usec_t dt) {
12 static usec_t get_max_every = 10 * USEC_PER_SEC, usec_since_last_max = 0;
13 static int read_full = 1;
14 static char *nf_conntrack_filename, *nf_conntrack_count_filename, *nf_conntrack_max_filename;
15 - static RRDVAR *rrdvar_max = NULL;
15 + static const RRDVAR_ACQUIRED *rrdvar_max = NULL;
16
17 unsigned long long aentries = 0, asearched = 0, afound = 0, anew = 0, ainvalid = 0, aignore = 0, adelete = 0, adelete_list = 0,
18 ainsert = 0, ainsert_failed = 0, adrop = 0, aearly_drop = 0, aicmp_error = 0, aexpect_new = 0, aexpect_create = 0, aexpect_delete = 0, asearch_restart = 0;
@@ -50,7 +50,7 @@ int do_proc_net_stat_conntrack(int update_every, usec_t dt) {
50 if(!do_sockets && !read_full)
51 return 1;
52
53 - rrdvar_max = rrdvar_custom_host_variable_create(localhost, "netfilter_conntrack_max");
53 + rrdvar_max = rrdvar_custom_host_variable_add_and_acquire(localhost, "netfilter_conntrack_max");
54 }
55
56 if(likely(read_full)) {
collectors/proc.plugin/proc_net_wireless.c
-1
@@ -200,7 +200,6 @@ static void configure_device(int do_status, int do_quality, int do_discarded_pac
200
201 static void add_labels_to_wireless(struct netwireless *w, RRDSET *st) {
202 rrdlabels_add(st->rrdlabels, "device", w->name, RRDLABEL_SRC_AUTO);
203 - rrdcalc_update_rrdlabels(st);
203 }
204
205 int do_proc_net_wireless(int update_every, usec_t dt)
collectors/proc.plugin/proc_self_mountinfo.c
+2 -1
@@ -227,7 +227,8 @@ struct mountinfo *mountinfo_read(int do_statvfs) {
227 struct mountinfo *root = NULL, *last = NULL, *mi = NULL;
228
229 // create a dictionary to track uniqueness
230 - DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE|DICTIONARY_FLAG_NAME_LINK_DONT_CLONE);
230 + DICTIONARY *dict = dictionary_create(
231 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_NAME_LINK_DONT_CLONE);
232
233 unsigned long l, lines = procfile_lines(ff);
234 for(l = 0; l < lines ;l++) {
collectors/proc.plugin/proc_softirqs.c
+2 -2
@@ -155,7 +155,7 @@ int do_proc_softirqs(int update_every, usec_t dt) {
155 // calls of this function.
156 if(unlikely(!irr->rd || strncmp(irr->name, rrddim_name(irr->rd), MAX_INTERRUPT_NAME) != 0)) {
157 irr->rd = rrddim_add(st_system_softirqs, irr->id, irr->name, 1, 1, RRD_ALGORITHM_INCREMENTAL);
158 - rrddim_set_name(st_system_softirqs, irr->rd, irr->name);
158 + rrddim_reset_name(st_system_softirqs, irr->rd, irr->name);
159
160 // also reset per cpu RRDDIMs to avoid repeating strncmp() in the per core loop
161 if(likely(do_per_core != CONFIG_BOOLEAN_NO)) {
@@ -231,7 +231,7 @@ int do_proc_softirqs(int update_every, usec_t dt) {
231 if(irr->used && (do_per_core == CONFIG_BOOLEAN_YES || irr->cpu[c].value)) {
232 if(unlikely(!irr->cpu[c].rd)) {
233 irr->cpu[c].rd = rrddim_add(core_st[c], irr->id, irr->name, 1, 1, RRD_ALGORITHM_INCREMENTAL);
234 - rrddim_set_name(core_st[c], irr->cpu[c].rd, irr->name);
234 + rrddim_reset_name(core_st[c], irr->cpu[c].rd, irr->name);
235 }
236
237 rrddim_set_by_pointer(core_st[c], irr->cpu[c].rd, irr->cpu[c].value);
collectors/proc.plugin/proc_spl_kstat_zfs.c
+3 -3
@@ -252,8 +252,8 @@ void disable_zfs_pool_state(struct zfs_pool *pool)
252 pool->disabled = 1;
253 }
254
255 -int update_zfs_pool_state_chart(const char *name, void *pool_p, void *update_every_p)
256 -{
255 +int update_zfs_pool_state_chart(const DICTIONARY_ITEM *item, void *pool_p, void *update_every_p) {
256 + const char *name = dictionary_acquired_item_name(item);
257 struct zfs_pool *pool = (struct zfs_pool *)pool_p;
258 int update_every = *(int *)update_every_p;
259
@@ -321,7 +321,7 @@ int do_proc_spl_kstat_zfs_pool_state(int update_every, usec_t dt)
321 snprintfz(filename, FILENAME_MAX, "%s%s", netdata_configured_host_prefix, "/proc/spl/kstat/zfs");
322 dirname = config_get("plugin:proc:" ZFS_PROC_POOLS, "directory to monitor", filename);
323
324 - zfs_pools = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
324 + zfs_pools = dictionary_create(DICT_OPTION_SINGLE_THREADED);
325
326 do_zfs_pool_state = 1;
327 }
collectors/proc.plugin/proc_stat.c
+2 -2
@@ -481,7 +481,7 @@ int do_proc_stat(int update_every, usec_t dt) {
481 static uint32_t hash_intr, hash_ctxt, hash_processes, hash_procs_running, hash_procs_blocked;
482 static char *core_throttle_count_filename = NULL, *package_throttle_count_filename = NULL, *scaling_cur_freq_filename = NULL,
483 *time_in_state_filename = NULL, *schedstat_filename = NULL, *cpuidle_name_filename = NULL, *cpuidle_time_filename = NULL;
484 - static RRDVAR *cpus_var = NULL;
484 + static const RRDVAR_ACQUIRED *cpus_var = NULL;
485 static int accurate_freq_avail = 0, accurate_freq_is_used = 0;
486 size_t cores_found = (size_t)processors;
487
@@ -713,7 +713,7 @@ int do_proc_stat(int update_every, usec_t dt) {
713 rrddim_hide(cpu_chart->st, "idle");
714
715 if(unlikely(core == 0 && cpus_var == NULL))
716 - cpus_var = rrdvar_custom_host_variable_create(localhost, "active_processors");
716 + cpus_var = rrdvar_custom_host_variable_add_and_acquire(localhost, "active_processors");
717 }
718 else rrdset_next(cpu_chart->st);
719
collectors/proc.plugin/sys_block_zram.c
+8 -5
@@ -177,7 +177,7 @@ static void free_device(DICTIONARY *dict, const char *name)
177 rrdset_obsolete_and_pointer_null(d->st_savings);
178 rrdset_obsolete_and_pointer_null(d->st_alloc_efficiency);
179 rrdset_obsolete_and_pointer_null(d->st_comp_ratio);
180 - dictionary_del_having_write_lock(dict, name);
180 + dictionary_del(dict, name);
181 }
182 // --------------------------------------------------------------------
183
@@ -239,13 +239,16 @@ static inline int _collect_zram_metrics(const char* name, ZRAM_DEVICE *d, int ad
239 return 0;
240 }
241
242 -static int collect_first_zram_metrics(const char *name, void *entry, void *data) {
242 +static int collect_first_zram_metrics(const DICTIONARY_ITEM *item, void *entry, void *data) {
243 + const char *name = dictionary_acquired_item_name(item);
244 +
245 // collect without calling rrdset_next (init only)
246 return _collect_zram_metrics(name, (ZRAM_DEVICE *)entry, 0, (DICTIONARY *)data);
247 }
248
247 -static int collect_zram_metrics(const char *name, void *entry, void *data) {
248 - (void)name;
249 +static int collect_zram_metrics(const DICTIONARY_ITEM *item, void *entry, void *data) {
250 + const char *name = dictionary_acquired_item_name(item);
251 +
252 // collect with calling rrdset_next
253 return _collect_zram_metrics(name, (ZRAM_DEVICE *)entry, 1, (DICTIONARY *)data);
254 }
@@ -280,7 +283,7 @@ int do_sys_block_zram(int update_every, usec_t dt) {
283 }
284 procfile_close(ff);
285
283 - devices = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
286 + devices = dictionary_create(DICT_OPTION_SINGLE_THREADED);
287 device_count = init_devices(devices, (unsigned int)zram_id, update_every);
288 if (device_count < 1)
289 return 1;
collectors/proc.plugin/sys_class_infiniband.c
+3 -3
@@ -184,7 +184,7 @@ static struct ibport {
184 RRDSET *st_hwpackets;
185 RRDSET *st_hwerrors;
186
187 - RRDSETVAR *stv_speed;
187 + const RRDSETVAR_ACQUIRED *stv_speed;
188
189 usec_t speed_last_collected_usec;
190
@@ -543,7 +543,7 @@ int do_sys_class_infiniband(int update_every, usec_t dt)
543 // x4 lanes multiplier as per Documentation/ABI/stable/sysfs-class-infiniband
544 FOREACH_COUNTER_BYTES(GEN_RRD_DIM_ADD_CUSTOM, port, 4 * 8 * port->width, 1024, RRD_ALGORITHM_INCREMENTAL)
545
546 - port->stv_speed = rrdsetvar_custom_chart_variable_create(port->st_bytes, "link_speed");
546 + port->stv_speed = rrdsetvar_custom_chart_variable_add_and_acquire(port->st_bytes, "link_speed");
547 } else
548 rrdset_next(port->st_bytes);
549
@@ -551,7 +551,7 @@ int do_sys_class_infiniband(int update_every, usec_t dt)
551 FOREACH_COUNTER_BYTES(GEN_RRD_DIM_SETP, port)
552
553 // For link speed set only variable
554 - rrdsetvar_custom_chart_variable_set(port->stv_speed, port->speed);
554 + rrdsetvar_custom_chart_variable_set(port->st_bytes, port->stv_speed, port->speed);
555
556 rrdset_done(port->st_bytes);
557 }
collectors/proc.plugin/sys_class_power_supply.c
-1
@@ -114,7 +114,6 @@ void power_supply_free(struct power_supply *ps) {
114
115 static void add_labels_to_power_supply(struct power_supply *ps, RRDSET *st) {
116 rrdlabels_add(st->rrdlabels, "device", ps->name, RRDLABEL_SRC_AUTO);
117 - rrdcalc_update_rrdlabels(st);
117 }
118
119 int do_sys_class_power_supply(int update_every, usec_t dt) {
collectors/proc.plugin/sys_fs_btrfs.c
-1
@@ -451,7 +451,6 @@ static inline int find_all_btrfs_pools(const char *path) {
451 static void add_labels_to_btrfs(BTRFS_NODE *n, RRDSET *st) {
452 rrdlabels_add(st->rrdlabels, "device", n->id, RRDLABEL_SRC_AUTO);
453 rrdlabels_add(st->rrdlabels, "device_label", n->label, RRDLABEL_SRC_AUTO);
454 - rrdcalc_update_rrdlabels(st);
454 }
455
456 int do_sys_fs_btrfs(int update_every, usec_t dt) {
collectors/statsd.plugin/statsd.c
+31 -22
@@ -24,9 +24,9 @@
24
25 #ifdef STATSD_MULTITHREADED
26 // DO NOT ENABLE MULTITHREADING - IT IS NOT WELL TESTED
27 -#define STATSD_DICTIONARY_OPTIONS DICTIONARY_FLAG_DONT_OVERWRITE_VALUE|DICTIONARY_FLAG_ADD_IN_FRONT
27 +#define STATSD_DICTIONARY_OPTIONS (DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_ADD_IN_FRONT)
28 #else
29 -#define STATSD_DICTIONARY_OPTIONS DICTIONARY_FLAG_DONT_OVERWRITE_VALUE|DICTIONARY_FLAG_ADD_IN_FRONT|DICTIONARY_FLAG_SINGLE_THREADED
29 +#define STATSD_DICTIONARY_OPTIONS (DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_ADD_IN_FRONT | DICT_OPTION_SINGLE_THREADED)
30 #endif
31
32 #define STATSD_DECIMAL_DETAIL 1000 // floating point values get multiplied by this, with the same divisor
@@ -192,6 +192,7 @@ typedef struct statsd_app_chart_dimension {
192 collected_number multiplier; // the multiplier of the dimension
193 collected_number divisor; // the divisor of the dimension
194 RRDDIM_FLAGS flags; // the RRDDIM flags for this dimension
195 + RRDDIM_OPTIONS options; // the RRDDIM options for this dimension
196
197 STATSD_APP_CHART_DIM_VALUE_TYPE value_type; // which value to use of the source metric
198
@@ -371,9 +372,10 @@ static struct statsd {
372 // --------------------------------------------------------------------------------------------------------------------
373 // statsd index management - add/find metrics
374
374 -static void dictionary_metric_insert_callback(const char *name, void *value, void *data) {
375 +static void dictionary_metric_insert_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
376 STATSD_INDEX *index = (STATSD_INDEX *)data;
377 STATSD_METRIC *m = (STATSD_METRIC *)value;
378 + const char *name = dictionary_acquired_item_name(item);
379
380 debug(D_STATSD, "Creating new %s metric '%s'", index->name, name);
381
@@ -390,9 +392,9 @@ static void dictionary_metric_insert_callback(const char *name, void *value, voi
392 __atomic_fetch_add(&index->metrics, 1, __ATOMIC_RELAXED);
393 }
394
393 -static void dictionary_metric_delete_callback(const char *name, void *value, void *data) {
395 +static void dictionary_metric_delete_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
396 (void)data; // STATSD_INDEX *index = (STATSD_INDEX *)data;
395 - (void)name;
397 + (void)item;
398 STATSD_METRIC *m = (STATSD_METRIC *)value;
399
400 if(m->type == STATSD_METRIC_TYPE_HISTOGRAM || m->type == STATSD_METRIC_TYPE_TIMER) {
@@ -416,7 +418,7 @@ static inline STATSD_METRIC *statsd_find_or_add_metric(STATSD_INDEX *index, cons
418 // no locks here, go faster
419 // this will call the dictionary_metric_insert_callback() if an item
420 // is inserted, otherwise it will return the existing one.
419 - // We used the flag DICTIONARY_FLAG_DONT_OVERWRITE_VALUE to support this.
421 + // We used the flag DICT_OPTION_DONT_OVERWRITE_VALUE to support this.
422 STATSD_METRIC *m = dictionary_set(index->dict, name, NULL, sizeof(STATSD_METRIC));
423 #endif
424
@@ -572,8 +574,8 @@ static inline void statsd_process_histogram_or_timer(STATSD_METRIC *m, const cha
574 #define statsd_process_timer(m, value, sampling) statsd_process_histogram_or_timer(m, value, sampling, "timer")
575 #define statsd_process_histogram(m, value, sampling) statsd_process_histogram_or_timer(m, value, sampling, "histogram")
576
575 -static void dictionary_metric_set_value_insert_callback(const char *name, void *value, void *data) {
576 - (void)name;
577 +static void dictionary_metric_set_value_insert_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
578 + (void)item;
579 (void)value;
580 STATSD_METRIC *m = (STATSD_METRIC *)data;
581 m->set.unique++;
@@ -617,8 +619,8 @@ static inline void statsd_process_set(STATSD_METRIC *m, const char *value) {
619 }
620 }
621
620 -static void dictionary_metric_dict_value_insert_callback(const char *name, void *value, void *data) {
621 - (void)name;
622 +static void dictionary_metric_dict_value_insert_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
623 + (void)item;
624 (void)value;
625 STATSD_METRIC *m = (STATSD_METRIC *)data;
626 m->dictionary.unique++;
@@ -1209,6 +1211,7 @@ static STATSD_APP_CHART_DIM *add_dimension_to_app_chart(
1211 , collected_number multiplier
1212 , collected_number divisor
1213 , RRDDIM_FLAGS flags
1214 + , RRDDIM_OPTIONS options
1215 , STATSD_APP_CHART_DIM_VALUE_TYPE value_type
1216 ) {
1217 STATSD_APP_CHART_DIM *dim = callocz(sizeof(STATSD_APP_CHART_DIM), 1);
@@ -1221,6 +1224,7 @@ static STATSD_APP_CHART_DIM *add_dimension_to_app_chart(
1224 dim->divisor = divisor;
1225 dim->value_type = value_type;
1226 dim->flags = flags;
1227 + dim->options = options;
1228
1229 if(!dim->multiplier)
1230 dim->multiplier = 1;
@@ -1323,7 +1327,7 @@ static int statsd_readfile(const char *filename, STATSD_APP *app, STATSD_APP_CHA
1327 else if(app) {
1328 if(!strcmp(s, "dictionary")) {
1329 if(!app->dict)
1326 - app->dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
1330 + app->dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
1331
1332 dict = app->dict;
1333 }
@@ -1474,17 +1478,18 @@ static int statsd_readfile(const char *filename, STATSD_APP *app, STATSD_APP_CHA
1478 pattern = 1;
1479 }
1480
1477 - char *dim_name = words[i++];
1478 - char *type = words[i++];
1479 - char *multiplier = words[i++];
1480 - char *divisor = words[i++];
1481 - char *options = words[i++];
1481 + char *dim_name = words[i++];
1482 + char *type = words[i++];
1483 + char *multiplier = words[i++];
1484 + char *divisor = words[i++];
1485 + char *opts = words[i++];
1486
1487 RRDDIM_FLAGS flags = RRDDIM_FLAG_NONE;
1484 - if(options && *options) {
1485 - if(strstr(options, "hidden") != NULL) flags |= RRDDIM_FLAG_HIDDEN;
1486 - if(strstr(options, "noreset") != NULL) flags |= RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS;
1487 - if(strstr(options, "nooverflow") != NULL) flags |= RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS;
1488 + RRDDIM_OPTIONS options = RRDDIM_OPTION_NONE;
1489 + if(opts && *opts) {
1490 + if(strstr(opts, "hidden") != NULL) options |= RRDDIM_OPTION_HIDDEN;
1491 + if(strstr(opts, "noreset") != NULL) options |= RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS;
1492 + if(strstr(opts, "nooverflow") != NULL) options |= RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS;
1493 }
1494
1495 if(!pattern) {
@@ -1510,7 +1515,8 @@ static int statsd_readfile(const char *filename, STATSD_APP *app, STATSD_APP_CHA
1515 , (multiplier && *multiplier)?str2l(multiplier):1
1516 , (divisor && *divisor)?str2l(divisor):1
1517 , flags
1513 - , string2valuetype(type, line, filename)
1518 + ,
1519 + options, string2valuetype(type, line, filename)
1520 );
1521
1522 if(pattern)
@@ -1775,7 +1781,7 @@ static inline void statsd_private_chart_dictionary(STATSD_METRIC *m) {
1781
1782 STATSD_METRIC_DICTIONARY_ITEM *t;
1783 dfe_start_read(m->dictionary.dict, t) {
1778 - if (!t->rd) t->rd = rrddim_add(m->st, t_name, NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1784 + if (!t->rd) t->rd = rrddim_add(m->st, t_dfe.name, NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1785 rrddim_set_by_pointer(m->st, t->rd, (collected_number)t->count);
1786 }
1787 dfe_done(t);
@@ -2130,6 +2136,7 @@ static inline void check_if_metric_is_for_app(STATSD_INDEX *index, STATSD_METRIC
2136 , dim->multiplier
2137 , dim->divisor
2138 , dim->flags
2139 + , dim->options
2140 , dim->value_type
2141 );
2142
@@ -2186,11 +2193,13 @@ static inline RRDDIM *statsd_add_dim_to_app_chart(STATSD_APP *app, STATSD_APP_CH
2193
2194 dim->rd = rrddim_add(chart->st, metric, dim->name, dim->multiplier, dim->divisor, dim->algorithm);
2195 if(dim->flags != RRDDIM_FLAG_NONE) dim->rd->flags |= dim->flags;
2196 + if(dim->options != RRDDIM_OPTION_NONE) dim->rd->options |= dim->options;
2197 return dim->rd;
2198 }
2199
2200 dim->rd = rrddim_add(chart->st, dim->metric, dim->name, dim->multiplier, dim->divisor, dim->algorithm);
2201 if(dim->flags != RRDDIM_FLAG_NONE) dim->rd->flags |= dim->flags;
2202 + if(dim->options != RRDDIM_OPTION_NONE) dim->rd->options |= dim->options;
2203 return dim->rd;
2204 }
2205
collectors/tc.plugin/plugin_tc.c
+36 -39
@@ -74,7 +74,7 @@ struct tc_device {
74 // ----------------------------------------------------------------------------
75 // tc_class index
76
77 -static void tc_class_free_callback(const char *name __maybe_unused, void *value, void *data __maybe_unused) {
77 +static void tc_class_free_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
78 // struct tc_device *d = data;
79 struct tc_class *c = value;
80
@@ -84,22 +84,21 @@ static void tc_class_free_callback(const char *name __maybe_unused, void *value,
84 string_freez(c->parentid);
85 }
86
87 -static void tc_class_conflict_callback(const char *name __maybe_unused, void *old_value, void *new_value, void *data __maybe_unused) {
87 +static bool tc_class_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *old_value, void *new_value, void *data __maybe_unused) {
88 struct tc_device *d = data; (void)d;
89 struct tc_class *c = old_value; (void)c;
90 struct tc_class *new_c = new_value; (void)new_c;
91
92 - error("TC: class '%s' is already in device '%s'. Ignoring duplicate.", name, string2str(d->id));
92 + error("TC: class '%s' is already in device '%s'. Ignoring duplicate.", dictionary_acquired_item_name(item), string2str(d->id));
93
94 - tc_class_free_callback(name, new_value, data);
94 + tc_class_free_callback(item, new_value, data);
95 +
96 + return true;
97 }
98
99 static void tc_class_index_init(struct tc_device *d) {
100 if(!d->classes) {
99 - d->classes = dictionary_create(
100 - DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
101 - |DICTIONARY_FLAG_SINGLE_THREADED
102 - );
101 + d->classes = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_SINGLE_THREADED);
102
103 dictionary_register_delete_callback(d->classes, tc_class_free_callback, d);
104 dictionary_register_conflict_callback(d->classes, tc_class_conflict_callback, d);
@@ -128,12 +127,12 @@ static inline struct tc_class *tc_class_index_find(struct tc_device *d, const ch
127
128 static DICTIONARY *tc_device_root_index = NULL;
129
131 -static void tc_device_add_callback(const char *name __maybe_unused, void *value, void *data __maybe_unused) {
130 +static void tc_device_add_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
131 struct tc_device *d = value;
132 tc_class_index_init(d);
133 }
134
136 -static void tc_device_free_callback(const char *name __maybe_unused, void *value, void *data __maybe_unused) {
135 +static void tc_device_free_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
136 struct tc_device *d = value;
137
138 tc_class_index_destroy(d);
@@ -146,10 +145,7 @@ static void tc_device_free_callback(const char *name __maybe_unused, void *value
145 static void tc_device_index_init() {
146 if(!tc_device_root_index) {
147 tc_device_root_index = dictionary_create(
149 - DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
150 - |DICTIONARY_FLAG_SINGLE_THREADED
151 - |DICTIONARY_FLAG_ADD_IN_FRONT
152 - );
148 + DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_SINGLE_THREADED | DICT_OPTION_ADD_IN_FRONT);
149
150 dictionary_register_insert_callback(tc_device_root_index, tc_device_add_callback, NULL);
151 dictionary_register_delete_callback(tc_device_root_index, tc_device_free_callback, NULL);
@@ -196,7 +192,7 @@ static inline void tc_device_classes_cleanup(struct tc_device *d) {
192 d->family_updated = false;
193
194 struct tc_class *c;
199 - dfe_start_unsafe(d->classes, c) {
195 + dfe_start_write(d->classes, c) {
196 if(unlikely(cleanup_every && c->unupdated >= cleanup_every))
197 tc_class_free(d, c);
198
@@ -254,7 +250,7 @@ static inline void tc_device_commit(struct tc_device *d) {
250 // prepare all classes
251 // we set reasonable defaults for the rest of the code below
252
257 - dfe_start_unsafe(d->classes, c) {
253 + dfe_start_read(d->classes, c) {
254 c->render = false; // do not render this class
255 c->isleaf = true; // this is a leaf class
256 c->hasparent = false; // without a parent
@@ -283,7 +279,7 @@ static inline void tc_device_commit(struct tc_device *d) {
279 error("TC: device '%s' has active both classes (%d) and qdiscs (%d). Will render only qdiscs.", string2str(d->id), updated_classes, updated_qdiscs);
280
281 // set all classes to !updated
286 - dfe_start_unsafe(d->classes, c) {
282 + dfe_start_read(d->classes, c) {
283 if (unlikely(!c->isqdisc && c->updated))
284 c->updated = false;
285 }
@@ -307,7 +303,7 @@ static inline void tc_device_commit(struct tc_device *d) {
303 // so, here we remove the isleaf flag from nodes in the middle
304 // and we add the hasparent flag to leaf nodes we found their parent
305 if(likely(!d->enabled_all_classes_qdiscs)) {
310 - dfe_start_unsafe(d->classes, c) {
306 + dfe_start_read(d->classes, c) {
307 if(unlikely(!c->updated))
308 continue;
309
@@ -319,7 +315,7 @@ static inline void tc_device_commit(struct tc_device *d) {
315 // c->parentid?c->parentid:"NULL");
316
317 // find if c is leaf or not
322 - dfe_start_unsafe(d->classes, x) {
318 + dfe_start_read(d->classes, x) {
319 if(unlikely(!x->updated || c == x || !x->parentid))
320 continue;
321
@@ -339,7 +335,7 @@ static inline void tc_device_commit(struct tc_device *d) {
335 dfe_done(c);
336 }
337
342 - dfe_start_unsafe(d->classes, c) {
338 + dfe_start_read(d->classes, c) {
339 if(unlikely(!c->updated))
340 continue;
341
@@ -367,7 +363,7 @@ static inline void tc_device_commit(struct tc_device *d) {
363 // dump all the list to see what we know
364
365 if(unlikely(debug_flags & D_TC_LOOP)) {
370 - dfe_start_unsafe(d->classes, c) {
366 + dfe_start_read(d->classes, c) {
367 if(c->render) debug(D_TC_LOOP, "TC: final nodes dump for '%s': class %s, OK", string2str(d->name), string2str(c->id));
368 else debug(D_TC_LOOP, "TC: final nodes dump for '%s': class '%s', IGNORE (updated: %d, isleaf: %d, hasparent: %d, parent: '%s')",
369 string2str(d->name?d->name:d->id), string2str(c->id), c->updated, c->isleaf, c->hasparent, string2str(c->parentid));
@@ -426,7 +422,8 @@ static inline void tc_device_commit(struct tc_device *d) {
422 }
423 else {
424 rrdset_next(d->st_bytes);
429 - if(unlikely(d->name_updated)) rrdset_set_name(d->st_bytes, string2str(d->name));
425 + if(unlikely(d->name_updated))
426 + rrdset_reset_name(d->st_bytes, string2str(d->name));
427
428 if(d->name && d->name_updated)
429 rrdlabels_add(d->st_bytes->rrdlabels, "name", string2str(d->name), RRDLABEL_SRC_AUTO);
@@ -438,13 +435,13 @@ static inline void tc_device_commit(struct tc_device *d) {
435 // update the family
436 }
437
441 - dfe_start_unsafe(d->classes, c) {
438 + dfe_start_read(d->classes, c) {
439 if(unlikely(!c->render)) continue;
440
441 if(unlikely(!c->rd_bytes))
442 c->rd_bytes = rrddim_add(d->st_bytes, string2str(c->id), string2str(c->name?c->name:c->id), 8, BITS_IN_A_KILOBIT, RRD_ALGORITHM_INCREMENTAL);
443 else if(unlikely(c->name_updated))
447 - rrddim_set_name(d->st_bytes, c->rd_bytes, string2str(c->name));
444 + rrddim_reset_name(d->st_bytes, c->rd_bytes, string2str(c->name));
445
446 rrddim_set_by_pointer(d->st_bytes, c->rd_bytes, c->bytes);
447 }
@@ -491,7 +488,7 @@ static inline void tc_device_commit(struct tc_device *d) {
488 if(unlikely(d->name_updated)) {
489 char name[RRD_ID_LENGTH_MAX + 1];
490 snprintfz(name, RRD_ID_LENGTH_MAX, "%s_packets", string2str(d->name?d->name:d->id));
494 - rrdset_set_name(d->st_packets, name);
491 + rrdset_reset_name(d->st_packets, name);
492 }
493
494 if(d->name && d->name_updated)
@@ -504,13 +501,13 @@ static inline void tc_device_commit(struct tc_device *d) {
501 // update the family
502 }
503
507 - dfe_start_unsafe(d->classes, c) {
504 + dfe_start_read(d->classes, c) {
505 if(unlikely(!c->render)) continue;
506
507 if(unlikely(!c->rd_packets))
508 c->rd_packets = rrddim_add(d->st_packets, string2str(c->id), string2str(c->name?c->name:c->id), 1, 1, RRD_ALGORITHM_INCREMENTAL);
509 else if(unlikely(c->name_updated))
513 - rrddim_set_name(d->st_packets, c->rd_packets, string2str(c->name));
510 + rrddim_reset_name(d->st_packets, c->rd_packets, string2str(c->name));
511
512 rrddim_set_by_pointer(d->st_packets, c->rd_packets, c->packets);
513 }
@@ -557,7 +554,7 @@ static inline void tc_device_commit(struct tc_device *d) {
554 if(unlikely(d->name_updated)) {
555 char name[RRD_ID_LENGTH_MAX + 1];
556 snprintfz(name, RRD_ID_LENGTH_MAX, "%s_dropped", string2str(d->name?d->name:d->id));
560 - rrdset_set_name(d->st_dropped, name);
557 + rrdset_reset_name(d->st_dropped, name);
558 }
559
560 if(d->name && d->name_updated)
@@ -570,13 +567,13 @@ static inline void tc_device_commit(struct tc_device *d) {
567 // update the family
568 }
569
573 - dfe_start_unsafe(d->classes, c) {
570 + dfe_start_read(d->classes, c) {
571 if(unlikely(!c->render)) continue;
572
573 if(unlikely(!c->rd_dropped))
574 c->rd_dropped = rrddim_add(d->st_dropped, string2str(c->id), string2str(c->name?c->name:c->id), 1, 1, RRD_ALGORITHM_INCREMENTAL);
575 else if(unlikely(c->name_updated))
579 - rrddim_set_name(d->st_dropped, c->rd_dropped, string2str(c->name));
576 + rrddim_reset_name(d->st_dropped, c->rd_dropped, string2str(c->name));
577
578 rrddim_set_by_pointer(d->st_dropped, c->rd_dropped, c->dropped);
579 }
@@ -623,7 +620,7 @@ static inline void tc_device_commit(struct tc_device *d) {
620 if(unlikely(d->name_updated)) {
621 char name[RRD_ID_LENGTH_MAX + 1];
622 snprintfz(name, RRD_ID_LENGTH_MAX, "%s_tokens", string2str(d->name?d->name:d->id));
626 - rrdset_set_name(d->st_tokens, name);
623 + rrdset_reset_name(d->st_tokens, name);
624 }
625
626 if(d->name && d->name_updated)
@@ -636,14 +633,14 @@ static inline void tc_device_commit(struct tc_device *d) {
633 // update the family
634 }
635
639 - dfe_start_unsafe(d->classes, c) {
636 + dfe_start_read(d->classes, c) {
637 if(unlikely(!c->render)) continue;
638
639 if(unlikely(!c->rd_tokens)) {
640 c->rd_tokens = rrddim_add(d->st_tokens, string2str(c->id), string2str(c->name?c->name:c->id), 1, 1, RRD_ALGORITHM_ABSOLUTE);
641 }
642 else if(unlikely(c->name_updated))
646 - rrddim_set_name(d->st_tokens, c->rd_tokens, string2str(c->name));
643 + rrddim_reset_name(d->st_tokens, c->rd_tokens, string2str(c->name));
644
645 rrddim_set_by_pointer(d->st_tokens, c->rd_tokens, c->tokens);
646 }
@@ -691,7 +688,7 @@ static inline void tc_device_commit(struct tc_device *d) {
688 if(unlikely(d->name_updated)) {
689 char name[RRD_ID_LENGTH_MAX + 1];
690 snprintfz(name, RRD_ID_LENGTH_MAX, "%s_ctokens", string2str(d->name?d->name:d->id));
694 - rrdset_set_name(d->st_ctokens, name);
691 + rrdset_reset_name(d->st_ctokens, name);
692 }
693
694 if(d->name && d->name_updated)
@@ -704,13 +701,13 @@ static inline void tc_device_commit(struct tc_device *d) {
701 // update the family
702 }
703
707 - dfe_start_unsafe(d->classes, c) {
704 + dfe_start_read(d->classes, c) {
705 if(unlikely(!c->render)) continue;
706
707 if(unlikely(!c->rd_ctokens))
708 c->rd_ctokens = rrddim_add(d->st_ctokens, string2str(c->id), string2str(c->name?c->name:c->id), 1, 1, RRD_ALGORITHM_ABSOLUTE);
709 else if(unlikely(c->name_updated))
713 - rrddim_set_name(d->st_ctokens, c->rd_ctokens, string2str(c->name));
710 + rrddim_reset_name(d->st_ctokens, c->rd_ctokens, string2str(c->name));
711
712 rrddim_set_by_pointer(d->st_ctokens, c->rd_ctokens, c->ctokens);
713 }
@@ -1146,12 +1143,12 @@ void *tc_main(void *ptr) {
1143 worker_is_busy(WORKER_TC_WORKTIME);
1144 worker_set_metric(WORKER_TC_PLUGIN_TIME, str2ll(words[1], NULL));
1145
1149 - size_t number_of_devices = dictionary_stats_entries(tc_device_root_index);
1146 + size_t number_of_devices = dictionary_entries(tc_device_root_index);
1147 size_t number_of_classes = 0;
1148
1149 struct tc_device *d;
1153 - dfe_start_unsafe(tc_device_root_index, d) {
1154 - number_of_classes += dictionary_stats_entries(d->classes);
1150 + dfe_start_read(tc_device_root_index, d) {
1151 + number_of_classes += dictionary_entries(d->classes);
1152 }
1153 dfe_done(d);
1154
configure.ac
+1
@@ -1718,6 +1718,7 @@ AC_CONFIG_FILES([
1718 libnetdata/simple_pattern/Makefile
1719 libnetdata/socket/Makefile
1720 libnetdata/statistical/Makefile
1721 + libnetdata/string/Makefile
1722 libnetdata/storage_number/Makefile
1723 libnetdata/storage_number/tests/Makefile
1724 libnetdata/threads/Makefile
daemon/analytics.c
+26 -34
@@ -249,8 +249,7 @@ void analytics_exporters(void)
249 buffer_free(bi);
250 }
251
252 -int collector_counter_callb(const char *name, void *entry, void *data) {
253 - (void)name;
252 +int collector_counter_callb(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
253
254 struct array_printer *ap = (struct array_printer *)data;
255 struct collector *col = (struct collector *)entry;
@@ -279,21 +278,22 @@ int collector_counter_callb(const char *name, void *entry, void *data) {
278 void analytics_collectors(void)
279 {
280 RRDSET *st;
282 - DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
281 + DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
282 char name[500];
283 BUFFER *bt = buffer_create(1000);
284
286 - rrdset_foreach_read(st, localhost)
287 - {
288 - if (rrdset_is_available_for_viewers(st)) {
289 - struct collector col = {
290 - .plugin = rrdset_plugin_name(st),
291 - .module = rrdset_module_name(st)
292 - };
293 - snprintfz(name, 499, "%s:%s", col.plugin, col.module);
294 - dictionary_set(dict, name, &col, sizeof(struct collector));
295 - }
285 + rrdset_foreach_read(st, localhost) {
286 + if(!rrdset_is_available_for_viewers(st))
287 + continue;
288 +
289 + struct collector col = {
290 + .plugin = rrdset_plugin_name(st),
291 + .module = rrdset_module_name(st)
292 + };
293 + snprintfz(name, 499, "%s:%s", col.plugin, col.module);
294 + dictionary_set(dict, name, &col, sizeof(struct collector));
295 }
296 + rrdset_foreach_done(st);
297
298 struct array_printer ap;
299 ap.c = 0;
@@ -398,12 +398,11 @@ void analytics_charts(void)
398 {
399 RRDSET *st;
400 int c = 0;
401 +
402 rrdset_foreach_read(st, localhost)
402 - {
403 - if (rrdset_is_available_for_viewers(st)) {
404 - c++;
405 - }
406 - }
403 + if(rrdset_is_available_for_viewers(st)) c++;
404 + rrdset_foreach_done(st);
405 +
406 {
407 char b[7];
408 snprintfz(b, 6, "%d", c);
@@ -415,22 +414,19 @@ void analytics_metrics(void)
414 {
415 RRDSET *st;
416 long int dimensions = 0;
418 - RRDDIM *rd;
419 - rrdset_foreach_read(st, localhost)
420 - {
421 - rrdset_rdlock(st);
422 -
417 + rrdset_foreach_read(st, localhost) {
418 if (rrdset_is_available_for_viewers(st)) {
424 - rrddim_foreach_read(rd, st)
425 - {
426 - if (rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN) || rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE))
419 + RRDDIM *rd;
420 + rrddim_foreach_read(rd, st) {
421 + if (rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN) || rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE))
422 continue;
423 dimensions++;
424 }
425 + rrddim_foreach_done(rd);
426 }
431 -
432 - rrdset_unlock(st);
427 }
428 + rrdset_foreach_done(st);
429 +
430 {
431 char b[7];
432 snprintfz(b, 6, "%ld", dimensions);
@@ -443,7 +439,7 @@ void analytics_alarms(void)
439 int alarm_warn = 0, alarm_crit = 0, alarm_normal = 0;
440 char b[10];
441 RRDCALC *rc;
446 - foreach_rrdcalc_in_rrdhost(localhost, rc) {
442 + foreach_rrdcalc_in_rrdhost_read(localhost, rc) {
443 if (unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
444 continue;
445
@@ -458,6 +454,7 @@ void analytics_alarms(void)
454 alarm_normal++;
455 }
456 }
457 + foreach_rrdcalc_in_rrdhost_done(rc);
458
459 snprintfz(b, 9, "%d", alarm_normal);
460 analytics_set_data(&analytics_data.netdata_alarms_normal, b);
@@ -527,16 +524,11 @@ void analytics_gather_immutable_meta_data(void)
524 */
525 void analytics_gather_mutable_meta_data(void)
526 {
530 - rrdhost_rdlock(localhost);
531 -
527 analytics_collectors();
528 analytics_alarms();
529 analytics_charts();
530 analytics_metrics();
531 analytics_aclk();
537 -
538 - rrdhost_unlock(localhost);
539 -
532 analytics_mirrored_hosts();
533 analytics_alarms_notifications();
534
daemon/global_statistics.c
+368 -1
@@ -10,8 +10,9 @@
10 #define WORKER_JOB_DBENGINE 3
11 #define WORKER_JOB_HEARTBEAT 4
12 #define WORKER_JOB_STRINGS 5
13 +#define WORKER_JOB_DICTIONARIES 6
14
14 -#if WORKER_UTILIZATION_MAX_JOB_TYPES < 6
15 +#if WORKER_UTILIZATION_MAX_JOB_TYPES < 7
16 #error WORKER_UTILIZATION_MAX_JOB_TYPES has to be at least 5
17 #endif
18
@@ -1215,6 +1216,367 @@ static void update_heartbeat_charts() {
1216 rrdset_done(st_heartbeat);
1217 }
1218
1219 +// ---------------------------------------------------------------------------------------------------------------------
1220 +// dictionary statistics
1221 +
1222 +struct dictionary_categories {
1223 + struct dictionary_stats *stats;
1224 + const char *family;
1225 + const char *context_prefix;
1226 + int priority;
1227 +
1228 + RRDSET *st_dicts;
1229 + RRDDIM *rd_dicts_active;
1230 + RRDDIM *rd_dicts_deleted;
1231 +
1232 + RRDSET *st_items;
1233 + RRDDIM *rd_items_entries;
1234 + RRDDIM *rd_items_referenced;
1235 + RRDDIM *rd_items_pending_deletion;
1236 +
1237 + RRDSET *st_ops;
1238 + RRDDIM *rd_ops_creations;
1239 + RRDDIM *rd_ops_destructions;
1240 + RRDDIM *rd_ops_flushes;
1241 + RRDDIM *rd_ops_traversals;
1242 + RRDDIM *rd_ops_walkthroughs;
1243 + RRDDIM *rd_ops_garbage_collections;
1244 + RRDDIM *rd_ops_searches;
1245 + RRDDIM *rd_ops_inserts;
1246 + RRDDIM *rd_ops_resets;
1247 + RRDDIM *rd_ops_deletes;
1248 +
1249 + RRDSET *st_callbacks;
1250 + RRDDIM *rd_callbacks_inserts;
1251 + RRDDIM *rd_callbacks_conflicts;
1252 + RRDDIM *rd_callbacks_reacts;
1253 + RRDDIM *rd_callbacks_deletes;
1254 +
1255 + RRDSET *st_memory;
1256 + RRDDIM *rd_memory_indexed;
1257 + RRDDIM *rd_memory_values;
1258 + RRDDIM *rd_memory_dict;
1259 +
1260 + RRDSET *st_spins;
1261 + RRDDIM *rd_spins_use;
1262 + RRDDIM *rd_spins_search;
1263 + RRDDIM *rd_spins_insert;
1264 +
1265 +} dictionary_categories[] = {
1266 + { .stats = &dictionary_stats_category_other, "dictionaries", "dictionaries", 900000 },
1267 +
1268 + // terminator
1269 + { .stats = NULL, NULL, NULL, 0 },
1270 +};
1271 +
1272 +#define load_dictionary_stats_entry(x) total += (size_t)(stats.x = __atomic_load_n(&c->stats->x, __ATOMIC_RELAXED))
1273 +
1274 +static void update_dictionary_category_charts(struct dictionary_categories *c) {
1275 + struct dictionary_stats stats;
1276 + stats.name = c->stats->name;
1277 +
1278 + // ------------------------------------------------------------------------
1279 +
1280 + size_t total = 0;
1281 + load_dictionary_stats_entry(dictionaries.active);
1282 + load_dictionary_stats_entry(dictionaries.deleted);
1283 +
1284 + if(c->st_dicts || total != 0) {
1285 + if (unlikely(!c->st_dicts)) {
1286 + char id[RRD_ID_LENGTH_MAX + 1];
1287 + snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.dictionaries", c->context_prefix, stats.name);
1288 +
1289 + char context[RRD_ID_LENGTH_MAX + 1];
1290 + snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.dictionaries", c->context_prefix);
1291 +
1292 + c->st_dicts = rrdset_create_localhost(
1293 + "netdata"
1294 + , id
1295 + , NULL
1296 + , c->family
1297 + , context
1298 + , "Dictionaries"
1299 + , "dictionaries"
1300 + , "netdata"
1301 + , "stats"
1302 + , c->priority + 0
1303 + , localhost->rrd_update_every
1304 + , RRDSET_TYPE_LINE
1305 + );
1306 +
1307 + c->rd_dicts_active = rrddim_add(c->st_dicts, "active", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1308 + c->rd_dicts_deleted = rrddim_add(c->st_dicts, "deleted", NULL, -1, 1, RRD_ALGORITHM_ABSOLUTE);
1309 +
1310 + rrdlabels_add(c->st_dicts->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
1311 + }
1312 + else
1313 + rrdset_next(c->st_dicts);
1314 +
1315 + rrddim_set_by_pointer(c->st_dicts, c->rd_dicts_active, (collected_number)stats.dictionaries.active);
1316 + rrddim_set_by_pointer(c->st_dicts, c->rd_dicts_deleted, (collected_number)stats.dictionaries.deleted);
1317 + rrdset_done(c->st_dicts);
1318 + }
1319 +
1320 + // ------------------------------------------------------------------------
1321 +
1322 + total = 0;
1323 + load_dictionary_stats_entry(items.entries);
1324 + load_dictionary_stats_entry(items.referenced);
1325 + load_dictionary_stats_entry(items.pending_deletion);
1326 +
1327 + if(c->st_items || total != 0) {
1328 + if (unlikely(!c->st_items)) {
1329 + char id[RRD_ID_LENGTH_MAX + 1];
1330 + snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.items", c->context_prefix, stats.name);
1331 +
1332 + char context[RRD_ID_LENGTH_MAX + 1];
1333 + snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.items", c->context_prefix);
1334 +
1335 + c->st_items = rrdset_create_localhost(
1336 + "netdata"
1337 + , id
1338 + , NULL
1339 + , c->family
1340 + , context
1341 + , "Dictionary Items"
1342 + , "items"
1343 + , "netdata"
1344 + , "stats"
1345 + , c->priority + 1
1346 + , localhost->rrd_update_every
1347 + , RRDSET_TYPE_LINE
1348 + );
1349 +
1350 + c->rd_items_entries = rrddim_add(c->st_items, "active", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1351 + c->rd_items_pending_deletion = rrddim_add(c->st_items, "deleted", NULL, -1, 1, RRD_ALGORITHM_ABSOLUTE);
1352 + c->rd_items_referenced = rrddim_add(c->st_items, "referenced", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1353 +
1354 + rrdlabels_add(c->st_items->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
1355 + }
1356 + else
1357 + rrdset_next(c->st_items);
1358 +
1359 + rrddim_set_by_pointer(c->st_items, c->rd_items_entries, stats.items.entries);
1360 + rrddim_set_by_pointer(c->st_items, c->rd_items_pending_deletion, stats.items.pending_deletion);
1361 + rrddim_set_by_pointer(c->st_items, c->rd_items_referenced, stats.items.referenced);
1362 + rrdset_done(c->st_items);
1363 + }
1364 +
1365 + // ------------------------------------------------------------------------
1366 +
1367 + total = 0;
1368 + load_dictionary_stats_entry(ops.creations);
1369 + load_dictionary_stats_entry(ops.destructions);
1370 + load_dictionary_stats_entry(ops.flushes);
1371 + load_dictionary_stats_entry(ops.traversals);
1372 + load_dictionary_stats_entry(ops.walkthroughs);
1373 + load_dictionary_stats_entry(ops.garbage_collections);
1374 + load_dictionary_stats_entry(ops.searches);
1375 + load_dictionary_stats_entry(ops.inserts);
1376 + load_dictionary_stats_entry(ops.resets);
1377 + load_dictionary_stats_entry(ops.deletes);
1378 +
1379 + if(c->st_ops || total != 0) {
1380 + if (unlikely(!c->st_ops)) {
1381 + char id[RRD_ID_LENGTH_MAX + 1];
1382 + snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.ops", c->context_prefix, stats.name);
1383 +
1384 + char context[RRD_ID_LENGTH_MAX + 1];
1385 + snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.ops", c->context_prefix);
1386 +
1387 + c->st_ops = rrdset_create_localhost(
1388 + "netdata"
1389 + , id
1390 + , NULL
1391 + , c->family
1392 + , context
1393 + , "Dictionary Operations"
1394 + , "ops/s"
1395 + , "netdata"
1396 + , "stats"
1397 + , c->priority + 2
1398 + , localhost->rrd_update_every
1399 + , RRDSET_TYPE_LINE
1400 + );
1401 +
1402 + c->rd_ops_creations = rrddim_add(c->st_ops, "creations", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1403 + c->rd_ops_destructions = rrddim_add(c->st_ops, "destructions", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1404 + c->rd_ops_flushes = rrddim_add(c->st_ops, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1405 + c->rd_ops_traversals = rrddim_add(c->st_ops, "traversals", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1406 + c->rd_ops_walkthroughs = rrddim_add(c->st_ops, "walkthroughs", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1407 + c->rd_ops_garbage_collections = rrddim_add(c->st_ops, "garbage_collections", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1408 + c->rd_ops_searches = rrddim_add(c->st_ops, "searches", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1409 + c->rd_ops_inserts = rrddim_add(c->st_ops, "inserts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1410 + c->rd_ops_resets = rrddim_add(c->st_ops, "resets", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1411 + c->rd_ops_deletes = rrddim_add(c->st_ops, "deletes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1412 +
1413 + rrdlabels_add(c->st_ops->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
1414 + }
1415 + else
1416 + rrdset_next(c->st_ops);
1417 +
1418 + rrddim_set_by_pointer(c->st_ops, c->rd_ops_creations, (collected_number)stats.ops.creations);
1419 + rrddim_set_by_pointer(c->st_ops, c->rd_ops_destructions, (collected_number)stats.ops.destructions);
1420 + rrddim_set_by_pointer(c->st_ops, c->rd_ops_flushes, (collected_number)stats.ops.flushes);
1421 + rrddim_set_by_pointer(c->st_ops, c->rd_ops_traversals, (collected_number)stats.ops.traversals);
1422 + rrddim_set_by_pointer(c->st_ops, c->rd_ops_walkthroughs, (collected_number)stats.ops.walkthroughs);
1423 + rrddim_set_by_pointer(c->st_ops, c->rd_ops_garbage_collections, (collected_number)stats.ops.garbage_collections);
1424 + rrddim_set_by_pointer(c->st_ops, c->rd_ops_searches, (collected_number)stats.ops.searches);
1425 + rrddim_set_by_pointer(c->st_ops, c->rd_ops_inserts, (collected_number)stats.ops.inserts);
1426 + rrddim_set_by_pointer(c->st_ops, c->rd_ops_resets, (collected_number)stats.ops.resets);
1427 + rrddim_set_by_pointer(c->st_ops, c->rd_ops_deletes, (collected_number)stats.ops.deletes);
1428 +
1429 + rrdset_done(c->st_ops);
1430 + }
1431 +
1432 + // ------------------------------------------------------------------------
1433 +
1434 + total = 0;
1435 + load_dictionary_stats_entry(callbacks.inserts);
1436 + load_dictionary_stats_entry(callbacks.conflicts);
1437 + load_dictionary_stats_entry(callbacks.reacts);
1438 + load_dictionary_stats_entry(callbacks.deletes);
1439 +
1440 + if(c->st_callbacks || total != 0) {
1441 + if (unlikely(!c->st_callbacks)) {
1442 + char id[RRD_ID_LENGTH_MAX + 1];
1443 + snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.callbacks", c->context_prefix, stats.name);
1444 +
1445 + char context[RRD_ID_LENGTH_MAX + 1];
1446 + snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.callbacks", c->context_prefix);
1447 +
1448 + c->st_callbacks = rrdset_create_localhost(
1449 + "netdata"
1450 + , id
1451 + , NULL
1452 + , c->family
1453 + , context
1454 + , "Dictionary Callbacks"
1455 + , "callbacks/s"
1456 + , "netdata"
1457 + , "stats"
1458 + , c->priority + 3
1459 + , localhost->rrd_update_every
1460 + , RRDSET_TYPE_LINE
1461 + );
1462 +
1463 + c->rd_callbacks_inserts = rrddim_add(c->st_callbacks, "inserts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1464 + c->rd_callbacks_deletes = rrddim_add(c->st_callbacks, "deletes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1465 + c->rd_callbacks_conflicts = rrddim_add(c->st_callbacks, "conflicts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1466 + c->rd_callbacks_reacts = rrddim_add(c->st_callbacks, "reacts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1467 +
1468 + rrdlabels_add(c->st_callbacks->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
1469 + }
1470 + else
1471 + rrdset_next(c->st_callbacks);
1472 +
1473 + rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_inserts, (collected_number)stats.callbacks.inserts);
1474 + rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_conflicts, (collected_number)stats.callbacks.conflicts);
1475 + rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_reacts, (collected_number)stats.callbacks.reacts);
1476 + rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_deletes, (collected_number)stats.callbacks.deletes);
1477 +
1478 + rrdset_done(c->st_callbacks);
1479 + }
1480 +
1481 + // ------------------------------------------------------------------------
1482 +
1483 + total = 0;
1484 + load_dictionary_stats_entry(memory.indexed);
1485 + load_dictionary_stats_entry(memory.values);
1486 + load_dictionary_stats_entry(memory.dict);
1487 +
1488 + if(c->st_memory || total != 0) {
1489 + if (unlikely(!c->st_memory)) {
1490 + char id[RRD_ID_LENGTH_MAX + 1];
1491 + snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.memory", c->context_prefix, stats.name);
1492 +
1493 + char context[RRD_ID_LENGTH_MAX + 1];
1494 + snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.memory", c->context_prefix);
1495 +
1496 + c->st_memory = rrdset_create_localhost(
1497 + "netdata"
1498 + , id
1499 + , NULL
1500 + , c->family
1501 + , context
1502 + , "Dictionary Memory"
1503 + , "bytes"
1504 + , "netdata"
1505 + , "stats"
1506 + , c->priority + 4
1507 + , localhost->rrd_update_every
1508 + , RRDSET_TYPE_STACKED
1509 + );
1510 +
1511 + c->rd_memory_indexed = rrddim_add(c->st_memory, "index", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1512 + c->rd_memory_values = rrddim_add(c->st_memory, "data", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1513 + c->rd_memory_dict = rrddim_add(c->st_memory, "structures", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1514 +
1515 + rrdlabels_add(c->st_memory->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
1516 + }
1517 + else
1518 + rrdset_next(c->st_memory);
1519 +
1520 + rrddim_set_by_pointer(c->st_memory, c->rd_memory_indexed, (collected_number)stats.memory.indexed);
1521 + rrddim_set_by_pointer(c->st_memory, c->rd_memory_values, (collected_number)stats.memory.values);
1522 + rrddim_set_by_pointer(c->st_memory, c->rd_memory_dict, (collected_number)stats.memory.dict);
1523 +
1524 + rrdset_done(c->st_memory);
1525 + }
1526 +
1527 + // ------------------------------------------------------------------------
1528 +
1529 + total = 0;
1530 + load_dictionary_stats_entry(spin_locks.use);
1531 + load_dictionary_stats_entry(spin_locks.search);
1532 + load_dictionary_stats_entry(spin_locks.insert);
1533 +
1534 + if(c->st_spins || total != 0) {
1535 + if (unlikely(!c->st_spins)) {
1536 + char id[RRD_ID_LENGTH_MAX + 1];
1537 + snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.spins", c->context_prefix, stats.name);
1538 +
1539 + char context[RRD_ID_LENGTH_MAX + 1];
1540 + snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.spins", c->context_prefix);
1541 +
1542 + c->st_spins = rrdset_create_localhost(
1543 + "netdata"
1544 + , id
1545 + , NULL
1546 + , c->family
1547 + , context
1548 + , "Dictionary Spins"
1549 + , "count"
1550 + , "netdata"
1551 + , "stats"
1552 + , c->priority + 5
1553 + , localhost->rrd_update_every
1554 + , RRDSET_TYPE_LINE
1555 + );
1556 +
1557 + c->rd_spins_use = rrddim_add(c->st_spins, "use", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1558 + c->rd_spins_search = rrddim_add(c->st_spins, "search", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1559 + c->rd_spins_insert = rrddim_add(c->st_spins, "insert", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1560 +
1561 + rrdlabels_add(c->st_spins->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
1562 + }
1563 + else
1564 + rrdset_next(c->st_spins);
1565 +
1566 + rrddim_set_by_pointer(c->st_spins, c->rd_spins_use, (collected_number)stats.spin_locks.use);
1567 + rrddim_set_by_pointer(c->st_spins, c->rd_spins_search, (collected_number)stats.spin_locks.search);
1568 + rrddim_set_by_pointer(c->st_spins, c->rd_spins_insert, (collected_number)stats.spin_locks.insert);
1569 +
1570 + rrdset_done(c->st_spins);
1571 + }
1572 +}
1573 +
1574 +static void dictionary_statistics(void) {
1575 + for(int i = 0; dictionary_categories[i].stats ;i++) {
1576 + update_dictionary_category_charts(&dictionary_categories[i]);
1577 + }
1578 +}
1579 +
1580 // ---------------------------------------------------------------------------------------------------------------------
1581 // worker utilization
1582
@@ -1334,6 +1696,7 @@ static struct worker_utilization all_workers_utilization[] = {
1696 { .name = "TIMEX", .family = "workers plugin timex", .priority = 1000000 },
1697 { .name = "IDLEJITTER", .family = "workers plugin idlejitter", .priority = 1000000 },
1698 { .name = "RRDCONTEXT", .family = "workers contexts", .priority = 1000000 },
1699 + { .name = "SERVICE", .family = "workers service", .priority = 1000000 },
1700
1701 // has to be terminated with a NULL
1702 { .name = NULL, .family = NULL }
@@ -2011,6 +2374,7 @@ void *global_statistics_main(void *ptr)
2374 worker_register_job_name(WORKER_JOB_WORKERS, "workers");
2375 worker_register_job_name(WORKER_JOB_DBENGINE, "dbengine");
2376 worker_register_job_name(WORKER_JOB_STRINGS, "strings");
2377 + worker_register_job_name(WORKER_JOB_DICTIONARIES, "dictionaries");
2378
2379 netdata_thread_cleanup_push(global_statistics_cleanup, ptr);
2380
@@ -2048,6 +2412,9 @@ void *global_statistics_main(void *ptr)
2412
2413 worker_is_busy(WORKER_JOB_STRINGS);
2414 update_strings_charts();
2415 +
2416 + worker_is_busy(WORKER_JOB_DICTIONARIES);
2417 + dictionary_statistics();
2418 }
2419
2420 netdata_thread_cleanup_pop(1);
daemon/main.c
+4
@@ -1004,6 +1004,7 @@ int main(int argc, char **argv) {
1004 if(test_dbengine()) return 1;
1005 #endif
1006 if(test_sqlite()) return 1;
1007 + if(string_unittest(10000)) return 1;
1008 if (dictionary_unittest(10000))
1009 return 1;
1010 if (rrdlabels_unittest())
@@ -1028,6 +1029,9 @@ int main(int argc, char **argv) {
1029 else if(strcmp(optarg, "dicttest") == 0) {
1030 return dictionary_unittest(10000);
1031 }
1032 + else if(strcmp(optarg, "stringtest") == 0) {
1033 + return string_unittest(10000);
1034 + }
1035 else if(strcmp(optarg, "rrdlabelstest") == 0) {
1036 return rrdlabels_unittest();
1037 }
daemon/service.c
+245 -6
@@ -5,12 +5,238 @@
5 /* Run service jobs every X seconds */
6 #define SERVICE_HEARTBEAT 10
7
8 +#define WORKER_JOB_CHILD_CHART_OBSOLETION_CHECK 1
9 +#define WORKER_JOB_CLEANUP_OBSOLETE_CHARTS 2
10 +#define WORKER_JOB_ARCHIVE_CHART 3
11 +#define WORKER_JOB_ARCHIVE_CHART_DIMENSIONS 4
12 +#define WORKER_JOB_ARCHIVE_DIMENSION 5
13 +#define WORKER_JOB_CLEANUP_ORPHAN_HOSTS 6
14 +#define WORKER_JOB_CLEANUP_OBSOLETE_CHARTS_ON_HOSTS 7
15 +#define WORKER_JOB_FREE_HOST 9
16 +#define WORKER_JOB_SAVE_HOST_CHARTS 10
17 +#define WORKER_JOB_DELETE_HOST_CHARTS 11
18 +#define WORKER_JOB_FREE_CHART 12
19 +#define WORKER_JOB_SAVE_CHART 13
20 +#define WORKER_JOB_DELETE_CHART 14
21 +#define WORKER_JOB_FREE_DIMENSION 15
22 +
23 +static void svc_rrddim_obsolete_to_archive(RRDDIM *rd) {
24 + RRDSET *st = rd->rrdset;
25 +
26 + if(rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED | RRDDIM_FLAG_ACLK) || !rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE))
27 + return;
28 +
29 + worker_is_busy(WORKER_JOB_ARCHIVE_DIMENSION);
30 +
31 + rrddim_flag_set(rd, RRDDIM_FLAG_ARCHIVED);
32 + rrddim_flag_clear(rd, RRDDIM_FLAG_OBSOLETE);
33 +
34 + const char *cache_filename = rrddim_cache_filename(rd);
35 + if(cache_filename) {
36 + info("Deleting dimension file '%s'.", cache_filename);
37 + if (unlikely(unlink(cache_filename) == -1))
38 + error("Cannot delete dimension file '%s'", cache_filename);
39 + }
40 +
41 + if (rd->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
42 + rrddimvar_delete_all(rd);
43 +
44 + /* only a collector can mark a chart as obsolete, so we must remove the reference */
45 +
46 + size_t tiers_available = 0, tiers_said_yes = 0;
47 + for(int tier = 0; tier < storage_tiers ;tier++) {
48 + if(rd->tiers[tier]) {
49 + tiers_available++;
50 +
51 + if(rd->tiers[tier]->collect_ops.finalize(rd->tiers[tier]->db_collection_handle))
52 + tiers_said_yes++;
53 +
54 + rd->tiers[tier]->db_collection_handle = NULL;
55 + }
56 + }
57 +
58 + if (tiers_available == tiers_said_yes && tiers_said_yes) {
59 + /* This metric has no data and no references */
60 + delete_dimension_uuid(&rd->metric_uuid);
61 + }
62 + else {
63 + /* Do not delete this dimension */
64 +#ifdef ENABLE_ACLK
65 + queue_dimension_to_aclk(rd, calc_dimension_liveness(rd, now_realtime_sec()));
66 +#endif
67 + return;
68 + }
69 + }
70 +
71 + worker_is_busy(WORKER_JOB_FREE_DIMENSION);
72 + rrddim_free(st, rd);
73 +}
74 +
75 +static void svc_rrdset_archive_obsolete_dimensions(RRDSET *st, bool all_dimensions) {
76 + worker_is_busy(WORKER_JOB_ARCHIVE_CHART_DIMENSIONS);
77 +
78 + RRDDIM *rd;
79 + time_t now = now_realtime_sec();
80 +
81 + dfe_start_reentrant(st->rrddim_root_index, rd) {
82 + if(unlikely(
83 + all_dimensions ||
84 + (rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE) && (rd->last_collected_time.tv_sec + rrdset_free_obsolete_time < now))
85 + )) {
86 +
87 + info("Removing obsolete dimension '%s' (%s) of '%s' (%s).", rrddim_name(rd), rrddim_id(rd), rrdset_name(st), rrdset_id(st));
88 + svc_rrddim_obsolete_to_archive(rd);
89 +
90 + }
91 + }
92 + dfe_done(rd);
93 +}
94 +
95 +static void svc_rrdset_obsolete_to_archive(RRDSET *st) {
96 + worker_is_busy(WORKER_JOB_ARCHIVE_CHART);
97 +
98 + rrdset_flag_set(st, RRDSET_FLAG_ARCHIVED);
99 + rrdset_flag_clear(st, RRDSET_FLAG_OBSOLETE);
100 +
101 + rrdcalc_unlink_all_rrdset_alerts(st);
102 +
103 + svc_rrdset_archive_obsolete_dimensions(st, true);
104 +
105 + rrdsetvar_release_and_delete_all(st);
106 +
107 + // has to be run after all dimensions are archived - or use-after-free will occur
108 + rrdvar_delete_all(st->rrdvars);
109 +
110 + if(st->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE) {
111 + if(rrdhost_flag_check(st->rrdhost, RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS)) {
112 + worker_is_busy(WORKER_JOB_DELETE_CHART);
113 + rrdset_delete_files(st);
114 + }
115 + else {
116 + worker_is_busy(WORKER_JOB_SAVE_CHART);
117 + rrdset_save(st);
118 + }
119 +
120 + worker_is_busy(WORKER_JOB_FREE_CHART);
121 + rrdset_free(st);
122 + }
123 +}
124 +
125 +static void svc_rrdhost_cleanup_obsolete_charts(RRDHOST *host) {
126 + worker_is_busy(WORKER_JOB_CLEANUP_OBSOLETE_CHARTS);
127 +
128 + time_t now = now_realtime_sec();
129 + RRDSET *st;
130 + rrdset_foreach_reentrant(st, host) {
131 + if(unlikely(rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE)
132 + && st->last_accessed_time + rrdset_free_obsolete_time < now
133 + && st->last_updated.tv_sec + rrdset_free_obsolete_time < now
134 + && st->last_collected_time.tv_sec + rrdset_free_obsolete_time < now
135 + )) {
136 + svc_rrdset_obsolete_to_archive(st);
137 + }
138 + else if(rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS)) {
139 + rrdset_flag_clear(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS);
140 + svc_rrdset_archive_obsolete_dimensions(st, false);
141 + }
142 +#ifdef ENABLE_ACLK
143 + else
144 + sql_check_chart_liveness(st);
145 +#endif
146 + }
147 + rrdset_foreach_done(st);
148 +}
149 +
150 +static void svc_rrdset_check_obsoletion(RRDHOST *host) {
151 + worker_is_busy(WORKER_JOB_CHILD_CHART_OBSOLETION_CHECK);
152 +
153 + time_t last_entry_t;
154 + RRDSET *st;
155 + rrdset_foreach_read(st, host) {
156 + last_entry_t = rrdset_last_entry_t(st);
157 +
158 + if(last_entry_t && last_entry_t < host->senders_connect_time)
159 + rrdset_is_obsolete(st);
160 +
161 + }
162 + rrdset_foreach_done(st);
163 +}
164 +
165 +static void svc_rrd_cleanup_obsolete_charts_from_all_hosts() {
166 + worker_is_busy(WORKER_JOB_CLEANUP_OBSOLETE_CHARTS_ON_HOSTS);
167 +
168 + rrd_rdlock();
169 +
170 + RRDHOST *host;
171 + rrdhost_foreach_read(host) {
172 +
173 + if(rrdhost_flag_check(host, RRDHOST_FLAG_PENDING_OBSOLETE_CHARTS|RRDHOST_FLAG_PENDING_OBSOLETE_DIMENSIONS)) {
174 + rrdhost_flag_clear(host, RRDHOST_FLAG_PENDING_OBSOLETE_CHARTS|RRDHOST_FLAG_PENDING_OBSOLETE_DIMENSIONS);
175 + svc_rrdhost_cleanup_obsolete_charts(host);
176 + }
177 +
178 + if(host != localhost
179 + && host->trigger_chart_obsoletion_check
180 + && (
181 + (
182 + host->senders_last_chart_command
183 + && host->senders_last_chart_command + host->health_delay_up_to < now_realtime_sec()
184 + )
185 + || (host->senders_connect_time + 300 < now_realtime_sec())
186 + )
187 + ) {
188 + svc_rrdset_check_obsoletion(host);
189 + host->trigger_chart_obsoletion_check = 0;
190 +
191 + }
192 + }
193 +
194 + rrd_unlock();
195 +}
196 +
197 +static void svc_rrdhost_cleanup_orphan_hosts(RRDHOST *protected_host) {
198 + worker_is_busy(WORKER_JOB_CLEANUP_ORPHAN_HOSTS);
199 + rrd_wrlock();
200 +
201 + time_t now = now_realtime_sec();
202 +
203 + RRDHOST *host;
204 +
205 +restart_after_removal:
206 + rrdhost_foreach_write(host) {
207 + if(rrdhost_should_be_removed(host, protected_host, now)) {
208 + info("Host '%s' with machine guid '%s' is obsolete - cleaning up.", rrdhost_hostname(host), host->machine_guid);
209 +
210 + if (rrdhost_flag_check(host, RRDHOST_FLAG_DELETE_ORPHAN_HOST)
211 +#ifdef ENABLE_DBENGINE
212 + /* don't delete multi-host DB host files */
213 + && !(host->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE && is_storage_engine_shared(host->storage_instance[0]))
214 +#endif
215 + ) {
216 + worker_is_busy(WORKER_JOB_DELETE_HOST_CHARTS);
217 + rrdhost_delete_charts(host);
218 + }
219 + else {
220 + worker_is_busy(WORKER_JOB_SAVE_HOST_CHARTS);
221 + rrdhost_save_charts(host);
222 + }
223 +
224 + worker_is_busy(WORKER_JOB_FREE_HOST);
225 + rrdhost_free(host, 0);
226 + goto restart_after_removal;
227 + }
228 + }
229 +
230 + rrd_unlock();
231 +}
232 +
233 static void service_main_cleanup(void *ptr)
234 {
235 struct netdata_static_thread *static_thread = (struct netdata_static_thread *)ptr;
236 static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
237
238 debug(D_SYSTEM, "Cleaning up...");
239 + worker_unregister();
240
241 static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
242 }
@@ -20,6 +246,22 @@ static void service_main_cleanup(void *ptr)
246 */
247 void *service_main(void *ptr)
248 {
249 + worker_register("SERVICE");
250 + worker_register_job_name(WORKER_JOB_CHILD_CHART_OBSOLETION_CHECK, "child chart obsoletion check");
251 + worker_register_job_name(WORKER_JOB_CLEANUP_OBSOLETE_CHARTS, "cleanup obsolete charts");
252 + worker_register_job_name(WORKER_JOB_ARCHIVE_CHART, "archive chart");
253 + worker_register_job_name(WORKER_JOB_ARCHIVE_CHART_DIMENSIONS, "archive chart dimensions");
254 + worker_register_job_name(WORKER_JOB_ARCHIVE_DIMENSION, "archive dimension");
255 + worker_register_job_name(WORKER_JOB_CLEANUP_ORPHAN_HOSTS, "cleanup orphan hosts");
256 + worker_register_job_name(WORKER_JOB_CLEANUP_OBSOLETE_CHARTS_ON_HOSTS, "cleanup obsolete charts on all hosts");
257 + worker_register_job_name(WORKER_JOB_FREE_HOST, "free host");
258 + worker_register_job_name(WORKER_JOB_SAVE_HOST_CHARTS, "save host charts");
259 + worker_register_job_name(WORKER_JOB_DELETE_HOST_CHARTS, "delete host charts");
260 + worker_register_job_name(WORKER_JOB_FREE_CHART, "free chart");
261 + worker_register_job_name(WORKER_JOB_SAVE_CHART, "save chart");
262 + worker_register_job_name(WORKER_JOB_DELETE_CHART, "delete chart");
263 + worker_register_job_name(WORKER_JOB_FREE_DIMENSION, "free dimension");
264 +
265 netdata_thread_cleanup_push(service_main_cleanup, ptr);
266 heartbeat_t hb;
267 heartbeat_init(&hb);
@@ -28,14 +270,11 @@ void *service_main(void *ptr)
270 debug(D_SYSTEM, "Service thread starts");
271
272 while (!netdata_exit) {
273 + worker_is_idle();
274 heartbeat_next(&hb, step);
275
33 - rrd_cleanup_obsolete_charts();
34 -
35 - rrd_wrlock();
36 - rrdhost_cleanup_orphan_hosts_nolock(localhost);
37 - rrd_unlock();
38 -
276 + svc_rrd_cleanup_obsolete_charts_from_all_hosts();
277 + svc_rrdhost_cleanup_orphan_hosts(localhost);
278 }
279
280 netdata_thread_cleanup_pop(1);
daemon/unit_test.c
+24 -12
@@ -1269,27 +1269,27 @@ static int test_variable_renames(void) {
1269 fprintf(stderr, "Created dimension with id '%s', name '%s'\n", rrddim_id(rd2), rrddim_name(rd2));
1270
1271 fprintf(stderr, "Renaming chart to CHARTNAME1\n");
1272 - rrdset_set_name(st, "CHARTNAME1");
1272 + rrdset_reset_name(st, "CHARTNAME1");
1273 fprintf(stderr, "Renamed chart with id '%s' to name '%s'\n", rrdset_id(st), rrdset_name(st));
1274
1275 fprintf(stderr, "Renaming chart to CHARTNAME2\n");
1276 - rrdset_set_name(st, "CHARTNAME2");
1276 + rrdset_reset_name(st, "CHARTNAME2");
1277 fprintf(stderr, "Renamed chart with id '%s' to name '%s'\n", rrdset_id(st), rrdset_name(st));
1278
1279 fprintf(stderr, "Renaming dimension DIM1 to DIM1NAME1\n");
1280 - rrddim_set_name(st, rd1, "DIM1NAME1");
1280 + rrddim_reset_name(st, rd1, "DIM1NAME1");
1281 fprintf(stderr, "Renamed dimension with id '%s' to name '%s'\n", rrddim_id(rd1), rrddim_name(rd1));
1282
1283 fprintf(stderr, "Renaming dimension DIM1 to DIM1NAME2\n");
1284 - rrddim_set_name(st, rd1, "DIM1NAME2");
1284 + rrddim_reset_name(st, rd1, "DIM1NAME2");
1285 fprintf(stderr, "Renamed dimension with id '%s' to name '%s'\n", rrddim_id(rd1), rrddim_name(rd1));
1286
1287 fprintf(stderr, "Renaming dimension DIM2 to DIM2NAME1\n");
1288 - rrddim_set_name(st, rd2, "DIM2NAME1");
1288 + rrddim_reset_name(st, rd2, "DIM2NAME1");
1289 fprintf(stderr, "Renamed dimension with id '%s' to name '%s'\n", rrddim_id(rd2), rrddim_name(rd2));
1290
1291 fprintf(stderr, "Renaming dimension DIM2 to DIM2NAME2\n");
1292 - rrddim_set_name(st, rd2, "DIM2NAME2");
1292 + rrddim_reset_name(st, rd2, "DIM2NAME2");
1293 fprintf(stderr, "Renamed dimension with id '%s' to name '%s'\n", rrddim_id(rd2), rrddim_name(rd2));
1294
1295 BUFFER *buf = buffer_create(1);
@@ -1447,9 +1447,8 @@ int unit_test(long delay, long shift)
1447 long increment = 1000;
1448 collected_number i = 0;
1449
1450 - unsigned long c, dimensions = 0;
1450 + unsigned long c, dimensions = rrdset_number_of_dimensions(st);
1451 RRDDIM *rd;
1452 - for(rd = st->dimensions; rd ; rd = rd->next) dimensions++;
1452
1453 for(c = 0; c < 20 ;c++) {
1454 i += increment;
@@ -1470,8 +1469,10 @@ int unit_test(long delay, long shift)
1469 }
1470
1471 // prevent it from deleting the dimensions
1473 - for(rd = st->dimensions; rd ; rd = rd->next)
1472 + rrddim_foreach_read(rd, st) {
1473 rd->last_collected_time.tv_sec = st->last_collected_time.tv_sec;
1474 + }
1475 + rrddim_foreach_done(rd);
1476
1477 rrdset_done(st);
1478 }
@@ -1486,7 +1487,7 @@ int unit_test(long delay, long shift)
1487 for(c = 0 ; c < st->counter ; c++) {
1488 fprintf(stderr, "\nPOSITION: c = %lu, EXPECTED VALUE %lu\n", c, (oincrement + c * increment + increment * (1000000 - shift) / 1000000 )* 10);
1489
1489 - for(rd = st->dimensions; rd ; rd = rd->next) {
1490 + rrddim_foreach_read(rd, st) {
1491 sn = rd->db[c];
1492 cn = unpack_storage_number(sn);
1493 fprintf(stderr, "\t %s " NETDATA_DOUBLE_FORMAT " (PACKED AS " STORAGE_NUMBER_FORMAT ") -> ", rrddim_id(rd), cn, sn);
@@ -1508,6 +1509,7 @@ int unit_test(long delay, long shift)
1509 ret = 1;
1510 }
1511 }
1512 + rrddim_foreach_done(rd);
1513 }
1514
1515 if(ret)
@@ -1967,7 +1969,11 @@ static int test_dbengine_check_rrdr(RRDSET *st[CHARTS], RRDDIM *rd[CHARTS][DIMS]
1969 time_retrieved = r->t[c];
1970
1971 // for each dimension
1970 - for (j = 0, d = r->st->dimensions; d && j < r->d ; ++j, d = d->next) {
1972 + rrddim_foreach_read(d, r->st) {
1973 + if(unlikely((int)d_dfe.counter >= r->d)) break; // d_counter is provided by the dictionary dfe
1974 +
1975 + j = (int)d_dfe.counter;
1976 +
1977 NETDATA_DOUBLE *cn = &r->v[ c * r->d ];
1978 value = cn[j];
1979 assert(rd[i][j] == d);
@@ -1990,6 +1996,7 @@ static int test_dbengine_check_rrdr(RRDSET *st[CHARTS], RRDDIM *rd[CHARTS][DIMS]
1996 time_errors++;
1997 }
1998 }
1999 + rrddim_foreach_done(d);
2000 }
2001 rrdr_free(owa, r);
2002 }
@@ -2103,7 +2110,11 @@ int test_dbengine(void)
2110 time_t time_retrieved = r->t[c];
2111
2112 // for each dimension
2106 - for(j = 0, d = r->st->dimensions; d && j < r->d ; ++j, d = d->next) {
2113 + rrddim_foreach_read(d, r->st) {
2114 + if(unlikely((int)d_dfe.counter >= r->d)) break; // d_counter is provided by the dictionary dfe
2115 +
2116 + j = (int)d_dfe.counter;
2117 +
2118 NETDATA_DOUBLE *cn = &r->v[ c * r->d ];
2119 NETDATA_DOUBLE value = cn[j];
2120 assert(rd[i][j] == d);
@@ -2126,6 +2137,7 @@ int test_dbengine(void)
2137 time_errors++;
2138 }
2139 }
2140 + rrddim_foreach_done(d);
2141 }
2142 rrdr_free(owa, r);
2143 }
database/engine/journalfile.c
+1 -1
@@ -519,7 +519,7 @@ int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfi
519 journalfile->file = file;
520 journalfile->pos = file_size;
521 journalfile->data = netdata_mmap(path, file_size, MAP_SHARED, 0);
522 - info("Loading journal file \"%s\" using %s.", path, journalfile->data?"using MMAP":"using uv_fs_read");
522 + info("Loading journal file \"%s\" using %s.", path, journalfile->data?"MMAP":"uv_fs_read");
523
524 max_id = iterate_transactions(ctx, journalfile);
525
database/engine/rrdengineapi.c
+1 -1
@@ -138,7 +138,7 @@ STORAGE_METRIC_HANDLE *rrdeng_metric_init(RRDDIM *rd, STORAGE_INSTANCE *db_insta
138 uuid_copy(rd->metric_uuid, multihost_legacy_uuid);
139
140 if (unlikely(need_to_store && !ctx->tier))
141 - (void)sql_store_dimension(&rd->metric_uuid, rd->rrdset->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor, rd->algorithm);
141 + (void)sql_store_dimension(&rd->metric_uuid, &rd->rrdset->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor, rd->algorithm);
142 }
143
144 struct rrdeng_metric_handle *mh = mallocz(sizeof(struct rrdeng_metric_handle));
database/ram/rrddim_mem.c
+1 -1
@@ -42,7 +42,7 @@ void rrddim_collect_store_metric(STORAGE_COLLECT_HANDLE *collection_handle, usec
42 void rrddim_store_metric_flush(STORAGE_COLLECT_HANDLE *collection_handle) {
43 struct mem_collect_handle *ch = (struct mem_collect_handle *)collection_handle;
44 RRDDIM *rd = ch->rd;
45 - memset(rd->db, 0, rd->entries * sizeof(storage_number));
45 + memset(rd->db, 0, rd->rrdset->entries * sizeof(storage_number));
46 }
47
48 int rrddim_collect_finalize(STORAGE_COLLECT_HANDLE *collection_handle) {
database/rrd.h
+214 -235
@@ -16,20 +16,21 @@ typedef struct storage_metric_handle STORAGE_METRIC_HANDLE;
16 typedef struct rrdhost RRDHOST;
17 typedef struct rrddim RRDDIM;
18 typedef struct rrdset RRDSET;
19 -typedef struct rrdvar RRDVAR;
20 -typedef struct rrdsetvar RRDSETVAR;
21 -typedef struct rrddimvar RRDDIMVAR;
19 typedef struct rrdcalc RRDCALC;
20 typedef struct rrdcalctemplate RRDCALCTEMPLATE;
21 typedef struct alarm_entry ALARM_ENTRY;
22 typedef struct context_param CONTEXT_PARAM;
23
24 +typedef struct rrdfamily_acquired RRDFAMILY_ACQUIRED;
25 +typedef struct rrdvar_acquired RRDVAR_ACQUIRED;
26 +typedef struct rrdsetvar_acquired RRDSETVAR_ACQUIRED;
27 +typedef struct rrdcalc_acquired RRDCALC_ACQUIRED;
28 +
29 typedef void *ml_host_t;
30 typedef void *ml_dimension_t;
31
32 // forward declarations
33 struct rrddim_tier;
32 -struct rrdset_volatile;
34 struct context_param;
35
36 #ifdef ENABLE_DBENGINE
@@ -75,11 +76,6 @@ struct context_param {
76 uint8_t flags;
77 };
78
78 -#define META_CHART_UPDATED 1
79 -#define META_PLUGIN_UPDATED 2
80 -#define META_MODULE_UPDATED 4
81 -#define META_CHART_ACTIVATED 8
82 -
79 #define UPDATE_EVERY 1
80 #define UPDATE_EVERY_MAX 3600
81
@@ -122,7 +118,9 @@ typedef enum rrd_memory_mode {
118 RRD_MEMORY_MODE_MAP = 2,
119 RRD_MEMORY_MODE_SAVE = 3,
120 RRD_MEMORY_MODE_ALLOC = 4,
125 - RRD_MEMORY_MODE_DBENGINE = 5
121 + RRD_MEMORY_MODE_DBENGINE = 5,
122 +
123 + // this is 8-bit
124 } RRD_MEMORY_MODE;
125
126 #define RRD_MEMORY_MODE_NONE_NAME "none"
@@ -145,7 +143,9 @@ typedef enum rrd_algorithm {
143 RRD_ALGORITHM_ABSOLUTE = 0,
144 RRD_ALGORITHM_INCREMENTAL = 1,
145 RRD_ALGORITHM_PCENT_OVER_DIFF_TOTAL = 2,
148 - RRD_ALGORITHM_PCENT_OVER_ROW_TOTAL = 3
146 + RRD_ALGORITHM_PCENT_OVER_ROW_TOTAL = 3,
147 +
148 + // this is 8-bit
149 } RRD_ALGORITHM;
150
151 #define RRD_ALGORITHM_ABSOLUTE_NAME "absolute"
@@ -159,34 +159,42 @@ extern const char *rrd_algorithm_name(RRD_ALGORITHM algorithm);
159 // ----------------------------------------------------------------------------
160 // RRD FAMILY
161
162 -struct rrdfamily {
163 - STRING *family;
164 - DICTIONARY *rrdvar_root_index;
165 -
166 - size_t use_count;
167 -};
168 -typedef struct rrdfamily RRDFAMILY;
162 +extern const RRDFAMILY_ACQUIRED *rrdfamily_add_and_acquire(RRDHOST *host, const char *id);
163 +extern void rrdfamily_release(RRDHOST *host, const RRDFAMILY_ACQUIRED *rfa);
164 +extern void rrdfamily_index_init(RRDHOST *host);
165 +extern void rrdfamily_index_destroy(RRDHOST *host);
166 +extern DICTIONARY *rrdfamily_rrdvars_dict(const RRDFAMILY_ACQUIRED *rf);
167
168
169 // ----------------------------------------------------------------------------
172 -// flags
173 -// use this for configuration flags, not for state control
174 -// flags are set/unset in a manner that is not thread safe
175 -// and may lead to missing information.
170 +// flags & options
171
172 +// options are permanent configuration options (no atomics to alter/access them)
173 +typedef enum rrddim_options {
174 + RRDDIM_OPTION_NONE = 0,
175 + RRDDIM_OPTION_HIDDEN = (1 << 0), // this dimension will not be offered to callers
176 + RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS = (1 << 1), // do not offer RESET or OVERFLOW info to callers
177 +
178 + // this is 8-bit
179 +} RRDDIM_OPTIONS;
180 +
181 +#define rrddim_option_check(rd, flag) ((rd)->flags & (flag))
182 +#define rrddim_option_set(rd, flag) (rd)->flags |= (flag)
183 +#define rrddim_option_clear(rd, flag) (rd)->flags &= ~(flag)
184 +
185 +// flags are runtime changing status flags (atomics are required to alter/access them)
186 typedef enum rrddim_flags {
187 RRDDIM_FLAG_NONE = 0,
179 - RRDDIM_FLAG_HIDDEN = (1 << 0), // this dimension will not be offered to callers
180 - RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS = (1 << 1), // do not offer RESET or OVERFLOW info to callers
188 RRDDIM_FLAG_OBSOLETE = (1 << 2), // this is marked by the collector/module as obsolete
182 - // No new values have been collected for this dimension since agent start or it was marked RRDDIM_FLAG_OBSOLETE at
189 + // No new values have been collected for this dimension since agent start, or it was marked RRDDIM_FLAG_OBSOLETE at
190 // least rrdset_free_obsolete_time seconds ago.
191 RRDDIM_FLAG_ARCHIVED = (1 << 3),
192 RRDDIM_FLAG_ACLK = (1 << 4),
193
187 - RRDDIM_FLAG_PENDING_FOREACH_ALARM = (1 << 5), // set when foreach alarm has not been initialized yet
194 + RRDDIM_FLAG_PENDING_FOREACH_ALARMS = (1 << 5), // set when foreach alarm has not been initialized yet
195 RRDDIM_FLAG_META_HIDDEN = (1 << 6), // Status of hidden option in the metadata database
189 - RRDDIM_FLAG_INDEXED_ID = (1 << 7),
196 +
197 + // this is 8 bit
198 } RRDDIM_FLAGS;
199
200 #define rrddim_flag_check(rd, flag) (__atomic_load_n(&((rd)->flags), __ATOMIC_SEQ_CST) & (flag))
@@ -231,6 +239,7 @@ extern void rrdlabels_copy(DICTIONARY *dst, DICTIONARY *src);
239
240 void reload_host_labels(void);
241 extern void rrdset_update_rrdlabels(RRDSET *st, DICTIONARY *new_rrdlabels);
242 +extern void rrdset_save_rrdlabels_to_sql(RRDSET *st);
243
244 extern int rrdlabels_unittest(void);
245
@@ -244,13 +253,15 @@ struct rrddim {
253 uuid_t metric_uuid; // global UUID for this metric (unique_across hosts)
254
255 // ------------------------------------------------------------------------
247 - // the dimension definition
256 + // dimension definition
257
258 STRING *id; // the id of this dimension (for internal identification)
259 STRING *name; // the name of this dimension (as presented to user)
251 - RRD_ALGORITHM algorithm; // the algorithm that is applied to add new collected values
252 - RRD_MEMORY_MODE rrd_memory_mode; // the memory mode for this dimension
253 - RRDDIM_FLAGS flags; // configuration flags for the dimension
260 +
261 + RRD_ALGORITHM algorithm:8; // the algorithm that is applied to add new collected values
262 + RRDDIM_OPTIONS options:8; // permanent configuration options
263 + RRD_MEMORY_MODE rrd_memory_mode:8; // the memory mode for this dimension
264 + /*RRDDIM_FLAGS*/ uint8_t flags; // run time changing status flags
265
266 bool updated; // 1 when the dimension has been updated since the last processing
267 bool exposed; // 1 when set what have sent this dimension to the central netdata
@@ -258,57 +269,54 @@ struct rrddim {
269 collected_number multiplier; // the multiplier of the collected values
270 collected_number divisor; // the divider of the collected values
271
261 - // ------------------------------------------------------------------------
262 - // members for temporary data we need for calculations
272 + int update_every; // every how many seconds is this updated
273 + // TODO - remove update_every from rrddim
274 + // it is always the same in rrdset
275
264 - struct timeval last_collected_time; // when was this dimension last updated
265 - // this is actual date time we updated the last_collected_value
266 - // THIS IS DIFFERENT FROM THE SAME MEMBER OF RRDSET
276 + // ------------------------------------------------------------------------
277 + // operational state members
278
279 #ifdef ENABLE_ACLK
280 int aclk_live_status;
281 #endif
271 - ml_dimension_t ml_dimension;
272 -
273 - struct rrddim_tier *tiers[RRD_STORAGE_TIERS]; // our tiers of databases
274 -
275 - size_t collections_counter; // the number of times we added values to this rrddim
276 - collected_number collected_value_max; // the absolute maximum of the collected value
277 -
278 - NETDATA_DOUBLE calculated_value; // the current calculated value, after applying the algorithm - resets to zero after being used
279 - NETDATA_DOUBLE last_calculated_value; // the last calculated value processed
280 - NETDATA_DOUBLE last_stored_value; // the last value as stored in the database (after interpolation)
282
282 - collected_number collected_value; // the current value, as collected - resets to 0 after being used
283 - collected_number last_collected_value; // the last value that was collected, after being processed
283 + ml_dimension_t ml_dimension; // machine learning data about this dimension
284
285 - // the *_volume members are used to calculate the accuracy of the rounding done by the
286 - // storage number - they are printed to debug.log when debug is enabled for a set.
287 - NETDATA_DOUBLE collected_volume; // the sum of all collected values so far
288 - NETDATA_DOUBLE stored_volume; // the sum of all stored values so far
285 + // ------------------------------------------------------------------------
286 + // linking to siblings and parents
287
288 struct rrddim *next; // linking of dimensions within the same data set
289 struct rrddim *prev; // linking of dimensions within the same data set
290
291 struct rrdset *rrdset;
292 +
293 RRDMETRIC_ACQUIRED *rrdmetric; // the rrdmetric of this dimension
294
295 // ------------------------------------------------------------------------
297 - // members for checking the data when loading from disk
296 + // data collection members
297
299 - long entries; // how many entries this dimension has in ram
300 - // this is the same to the entries of the data set
301 - // we set it here, to check the data when we load it from disk.
298 + struct rrddim_tier *tiers[RRD_STORAGE_TIERS]; // our tiers of databases
299
303 - int update_every; // every how many seconds is this updated
300 + struct timeval last_collected_time; // when was this dimension last updated
301 + // this is actual date time we updated the last_collected_value
302 + // THIS IS DIFFERENT FROM THE SAME MEMBER OF RRDSET
303
305 - size_t memsize; // the memory allocated for this dimension (without RRDDIM)
304 + size_t collections_counter; // the number of times we added values to this rrddim
305 + collected_number collected_value_max; // the absolute maximum of the collected value
306 +
307 + NETDATA_DOUBLE calculated_value; // the current calculated value, after applying the algorithm - resets to zero after being used
308 + NETDATA_DOUBLE last_calculated_value; // the last calculated value processed
309 + NETDATA_DOUBLE last_stored_value; // the last value as stored in the database (after interpolation)
310
307 - struct rrddimvar *variables;
311 + collected_number collected_value; // the current value, as collected - resets to 0 after being used
312 + collected_number last_collected_value; // the last value that was collected, after being processed
313
314 // ------------------------------------------------------------------------
310 - // the values stored in this dimension, using our floating point numbers
315 + // db mode RAM, SAVE, MAP, ALLOC, NONE specifics
316 + // TODO - they should be managed by storage engine
317 + // (RRDDIM_DB_STATE ptr to an undefined structure, and a call to clean this up during destruction)
318
319 + size_t memsize; // the memory allocated for this dimension (without RRDDIM)
320 void *rd_on_file; // pointer to the header written on disk
321 storage_number *db; // the array of values
322 };
@@ -346,7 +354,6 @@ struct rrddim_query_handle {
354 RRDDIM *rd;
355 time_t start_time;
356 time_t end_time;
349 - TIER_QUERY_FETCH tier_query_fetch_type;
357 STORAGE_QUERY_HANDLE* handle;
358 };
359
@@ -399,7 +406,7 @@ struct rrddim_collect_ops {
406 // run this to flush / reset the current data collection sequence
407 void (*flush)(STORAGE_COLLECT_HANDLE *collection_handle);
408
402 - // an finalization function to run after collection is over
409 + // a finalization function to run after collection is over
410 // returns 1 if it's safe to delete the dimension
411 int (*finalize)(STORAGE_COLLECT_HANDLE *collection_handle);
412 };
@@ -432,7 +439,6 @@ struct rrddim_query_ops {
439 struct rrddim_tier {
440 int tier_grouping;
441 RRD_MEMORY_MODE mode; // the memory mode of this tier
435 - RRD_BACKFILL backfill; // backfilling configuration
442 STORAGE_METRIC_HANDLE *db_metric_handle; // the metric handle inside the database
443 STORAGE_COLLECT_HANDLE *db_collection_handle; // the data collection handle
444 STORAGE_POINT virtual_point;
@@ -448,11 +454,16 @@ extern void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, int tier, time_t n
454 // these loop macros make sure the linked list is accessed with the right lock
455
456 #define rrddim_foreach_read(rd, st) \
451 - for((rd) = (st)->dimensions, rrdset_check_rdlock(st); (rd) ; (rd) = (rd)->next)
457 + dfe_start_read((st)->rrddim_root_index, rd)
458
459 #define rrddim_foreach_write(rd, st) \
454 - for((rd) = (st)->dimensions, rrdset_check_wrlock(st); (rd) ; (rd) = (rd)->next)
460 + dfe_start_write((st)->rrddim_root_index, rd)
461 +
462 +#define rrddim_foreach_reentrant(rd, st) \
463 + dfe_start_reentrant((st)->rrddim_root_index, rd)
464
465 +#define rrddim_foreach_done(rd) \
466 + dfe_done(rd)
467
468 // ----------------------------------------------------------------------------
469 // RRDSET - this is a chart
@@ -477,7 +488,7 @@ typedef enum rrdset_flags {
488 RRDSET_FLAG_HIDDEN = (1 << 12), // if set, do not show this chart on the dashboard, but use it for exporting
489 RRDSET_FLAG_SYNC_CLOCK = (1 << 13), // if set, microseconds on next data collection will be ignored (the chart will be synced to now)
490 RRDSET_FLAG_OBSOLETE_DIMENSIONS = (1 << 14), // this is marked by the collector/module when a chart has obsolete dimensions
480 - // No new values have been collected for this chart since agent start or it was marked RRDSET_FLAG_OBSOLETE at
491 + // No new values have been collected for this chart since agent start, or it was marked RRDSET_FLAG_OBSOLETE at
492 // least rrdset_free_obsolete_time seconds ago.
493 RRDSET_FLAG_ARCHIVED = (1 << 15),
494 RRDSET_FLAG_ACLK = (1 << 16),
@@ -496,14 +507,19 @@ typedef enum rrdset_flags {
507 #define rrdset_is_ar_chart(st) rrdset_flag_check(st, RRDSET_FLAG_ANOMALY_RATE_CHART)
508
509 struct rrdset {
499 - uuid_t uuid;
510 + uuid_t chart_uuid; // the global UUID for this chart
511
512 // ------------------------------------------------------------------------
502 - // the set configuration
513 + // chart configuration
514
504 - STRING *id; // the ID of the data set
505 - STRING *name; // the name of this dimension (as presented to user)
506 - STRING *type; // the type of graph RRD_TYPE_* (a category, for determining graphing options)
515 + struct {
516 + STRING *type; // the type of {type}.{id}
517 + STRING *id; // the id of {type}.{id}
518 + STRING *name; // the name of {type}.{name}
519 + } parts;
520 +
521 + STRING *id; // the unique ID of the rrdset as {type}.{id}
522 + STRING *name; // the unique name of the rrdset as {type}.{name}
523 STRING *family; // grouping sets under the same family
524 STRING *title; // title shown to user
525 STRING *units; // units of measurement
@@ -511,44 +527,51 @@ struct rrdset {
527 STRING *plugin_name; // the name of the plugin that generated this
528 STRING *module_name; // the name of the plugin module that generated this
529
514 - RRDINSTANCE_ACQUIRED *rrdinstance; // the rrdinstance of this chart
515 - RRDCONTEXT_ACQUIRED *rrdcontext; // the rrdcontext this chart belongs to
530 + RRDSET_TYPE chart_type; // line, area, stacked
531 +
532 + long priority; // the sorting priority of this chart
533 +
534 + int update_every; // data collection frequency
535 +
536 + DICTIONARY *rrdlabels; // chart labels
537 + DICTIONARY *rrdsetvar_root_index; // chart variables
538 + DICTIONARY *rrddimvar_root_index; // dimension variables
539 + // we use this dictionary to manage their allocation
540 +
541 + // TODO - dimensions linked list and lock to be removed
542 + netdata_rwlock_t rrdset_rwlock; // protects the dimensions linked list
543 + RRDDIM *dimensions; // chart metrics
544
545 + // ------------------------------------------------------------------------
546 + // operational state members
547 +
548 + RRDSET_FLAGS flags; // flags
549 RRD_MEMORY_MODE rrd_memory_mode; // the db mode of this rrdset
518 - RRDSET_TYPE chart_type; // line, area, stacked
519 - RRDSET_FLAGS flags; // configuration flags
520 - RRDSET_FLAGS *exporting_flags; // array of flags for exporting connector instances
550
522 - int update_every; // every how many seconds is this updated?
551 + uuid_t hash_uuid; // hash_id for syncing with cloud
552 + // TODO - obsolete now - cleanup
553 +
554 + DICTIONARY *rrddim_root_index; // dimensions index
555
556 int gap_when_lost_iterations_above; // after how many lost iterations a gap should be stored
557 // netdata will interpolate values for gaps lower than this
558 + // TODO - use the global - all charts have the same value
559
527 - long entries; // total number of entries in the data set
528 -
529 - long current_entry; // the entry that is currently being updated
530 - // it goes around in a round-robin fashion
560 + // ------------------------------------------------------------------------
561 + // linking to siblings and parents
562
532 - long priority; // the sorting priority of this chart
563 + RRDHOST *rrdhost; // pointer to RRDHOST this chart belongs to
564
565 + RRDINSTANCE_ACQUIRED *rrdinstance; // the rrdinstance of this chart
566 + RRDCONTEXT_ACQUIRED *rrdcontext; // the rrdcontext this chart belongs to
567
568 // ------------------------------------------------------------------------
536 - // members for temporary data we need for calculations
537 -
538 - char *cache_dir; // the directory to store dimensions
569 + // data collection members
570
571 size_t counter; // the number of times we added values to this database
572 size_t counter_done; // the number of times rrdset_done() has been called
573
543 - union {
544 - time_t last_accessed_time; // the last time this RRDSET has been accessed
545 - time_t last_entry_t; // the last_entry_t computed for transient RRDSET
546 - };
547 - time_t upstream_resync_time; // the timestamp up to which we should resync clock upstream
548 -
549 - uuid_t *chart_uuid; // Store the global GUID for this chart
550 - // this object.
551 - size_t rrddim_page_alignment; // keeps metric pages in alignment when using dbengine
574 + time_t last_accessed_time; // the last time this RRDSET has been accessed
575
576 usec_t usec_since_last_update; // the time in microseconds since the last collection of data
577
@@ -558,46 +581,73 @@ struct rrdset {
581 total_number collected_total; // used internally to calculate percentages
582 total_number last_collected_total; // used internally to calculate percentages
583
561 - RRDFAMILY *rrdfamily; // pointer to RRDFAMILY this chart belongs to
562 - RRDHOST *rrdhost; // pointer to RRDHOST this chart belongs to
584 + size_t rrdlabels_last_saved_version;
585
564 - struct rrdset *next; // linking of rrdsets
565 - struct rrdset *prev; // linking of rrdsets
586 + // ------------------------------------------------------------------------
587 + // data collection - streaming to parents, temp variables
588 +
589 + time_t upstream_resync_time; // the timestamp up to which we should resync clock upstream
590
591 // ------------------------------------------------------------------------
568 - // local variables
592 + // context queries temp variables
593 + // TODO - eliminate these
594
570 - NETDATA_DOUBLE green; // green threshold for this chart
571 - NETDATA_DOUBLE red; // red threshold for this chart
595 + time_t last_entry_t; // the last_entry_t computed for transient RRDSET
596
573 - DICTIONARY *rrdvar_root_index; // RRDVAR index for this chart
574 - RRDSETVAR *variables; // RRDSETVAR linked list for this chart (one RRDSETVAR, many RRDVARs)
575 - RRDCALC *alarms; // RRDCALC linked list for this chart
597 + // ------------------------------------------------------------------------
598 + // dbengine specifics
599 + // TODO - they should be managed by storage engine
600 + // (RRDSET_DB_STATE ptr to an undefined structure, and a call to clean this up during destruction)
601 +
602 + size_t rrddim_page_alignment; // keeps metric pages in alignment when using dbengine
603
604 // ------------------------------------------------------------------------
578 - // members for checking the data when loading from disk
605 + // db mode SAVE, MAP specifics
606 + // TODO - they should be managed by storage engine
607 + // (RRDSET_DB_STATE ptr to an undefined structure, and a call to clean this up during destruction)
608
609 + char *cache_dir; // the directory to store dimensions
610 unsigned long memsize; // how much mem we have allocated for this (without dimensions)
611 void *st_on_file; // compatibility with V019 RRDSET files
612
613 // ------------------------------------------------------------------------
584 - // chart labels
614 + // db mode RAM, SAVE, MAP, ALLOC, NONE specifics
615 + // TODO - they should be managed by storage engine
616 + // (RRDSET_DB_STATE ptr to an undefined structure, and a call to clean this up during destruction)
617
586 - DICTIONARY *rrdlabels;
618 + long entries; // total number of entries in the data set
619 +
620 + long current_entry; // the entry that is currently being updated
621 + // it goes around in a round-robin fashion
622
623 // ------------------------------------------------------------------------
589 - // the dimensions
624 + // exporting to 3rd party time-series members
625 + // TODO - they should be managed by exporting engine
626 + // (RRDSET_EXPORTING_STATE ptr to an undefined structure, and a call to clean this up during destruction)
627
591 - DICTIONARY *rrddim_root_index; // the root of the dimensions index
628 + RRDSET_FLAGS *exporting_flags; // array of flags for exporting connector instances
629 +
630 + // ------------------------------------------------------------------------
631 + // health monitoring members
632 + // TODO - they should be managed by health
633 + // (RRDSET_HEALTH_STATE ptr to an undefined structure, and a call to clean this up during destruction)
634
593 - netdata_rwlock_t rrdset_rwlock; // protects dimensions linked list
594 - RRDDIM *dimensions; // the actual data for every dimension
635 + NETDATA_DOUBLE green; // green threshold for this chart
636 + NETDATA_DOUBLE red; // red threshold for this chart
637 +
638 + DICTIONARY *rrdvars; // RRDVAR index for this chart
639 + const RRDFAMILY_ACQUIRED *rrdfamily; // pointer to RRDFAMILY dictionary item, this chart belongs to
640 +
641 + struct {
642 + netdata_rwlock_t rwlock; // protection for RRDCALC *base
643 + RRDCALC *base; // double linked list of RRDCALC related to this RRDSET
644 + } alerts;
645 };
646
647 #define rrdset_plugin_name(st) string2str((st)->plugin_name)
648 #define rrdset_module_name(st) string2str((st)->module_name)
649 #define rrdset_units(st) string2str((st)->units)
600 -#define rrdset_type(st) string2str((st)->type)
650 +#define rrdset_parts_type(st) string2str((st)->parts.type)
651 #define rrdset_family(st) string2str((st)->family)
652 #define rrdset_title(st) string2str((st)->title)
653 #define rrdset_context(st) string2str((st)->context)
@@ -614,11 +664,19 @@ extern STRING *rrd_string_strdupz(const char *s);
664 // these loop macros make sure the linked list is accessed with the right lock
665
666 #define rrdset_foreach_read(st, host) \
617 - for((st) = (host)->rrdset_root, rrdhost_check_rdlock(host); st ; (st) = (st)->next)
667 + dfe_start_read((host)->rrdset_root_index, st)
668
669 #define rrdset_foreach_write(st, host) \
620 - for((st) = (host)->rrdset_root, rrdhost_check_wrlock(host); st ; (st) = (st)->next)
670 + dfe_start_write((host)->rrdset_root_index, st)
671 +
672 +#define rrdset_foreach_reentrant(st, host) \
673 + dfe_start_reentrant((host)->rrdset_root_index, st)
674
675 +#define rrdset_foreach_done(st) \
676 + dfe_done(st)
677 +
678 +#define rrdset_number_of_dimensions(st) \
679 + dictionary_entries((st)->rrddim_root_index)
680
681 extern void rrdset_memory_file_save(RRDSET *st);
682 extern void rrdset_memory_file_free(RRDSET *st);
@@ -633,19 +691,23 @@ extern bool rrdset_memory_load_or_create_map_save(RRDSET *st_on_file, RRD_MEMORY
691 // and may lead to missing information.
692
693 typedef enum rrdhost_flags {
636 - RRDHOST_FLAG_ORPHAN = (1 << 0), // this host is orphan (not receiving data)
637 - RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS = (1 << 1), // delete files of obsolete charts
638 - RRDHOST_FLAG_DELETE_ORPHAN_HOST = (1 << 2), // delete the entire host when orphan
639 - RRDHOST_FLAG_EXPORTING_SEND = (1 << 3), // send it to external databases
640 - RRDHOST_FLAG_EXPORTING_DONT_SEND = (1 << 4), // don't send it to external databases
641 - RRDHOST_FLAG_ARCHIVED = (1 << 5), // The host is archived, no collected charts yet
642 - RRDHOST_FLAG_PENDING_FOREACH_ALARMS = (1 << 7), // contains dims with uninitialized foreach alarms
643 - RRDHOST_FLAG_STREAM_LABELS_UPDATE = (1 << 8),
644 - RRDHOST_FLAG_STREAM_LABELS_STOP = (1 << 9),
645 - RRDHOST_FLAG_ACLK_STREAM_CONTEXTS = (1 << 10), // when set, we should send ACLK stream context updates
646 - RRDHOST_FLAG_INDEXED_MACHINE_GUID = (1 << 11), // when set, we have indexed its machine guid
647 - RRDHOST_FLAG_INDEXED_HOSTNAME = (1 << 12), // when set, we have indexed its hostname
648 - RRDHOST_FLAG_STREAM_COLLECTED_METRICS = (1 << 13), // when set, rrdset_done() should push metrics to parent
694 + RRDHOST_FLAG_ORPHAN = (1 << 0), // this host is orphan (not receiving data)
695 + RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS = (1 << 1), // delete files of obsolete charts
696 + RRDHOST_FLAG_DELETE_ORPHAN_HOST = (1 << 2), // delete the entire host when orphan
697 + RRDHOST_FLAG_EXPORTING_SEND = (1 << 3), // send it to external databases
698 + RRDHOST_FLAG_EXPORTING_DONT_SEND = (1 << 4), // don't send it to external databases
699 + RRDHOST_FLAG_ARCHIVED = (1 << 5), // The host is archived, no collected charts yet
700 + RRDHOST_FLAG_PENDING_FOREACH_ALARMS = (1 << 7), // contains dims with uninitialized foreach alarms
701 + RRDHOST_FLAG_STREAM_LABELS_UPDATE = (1 << 8),
702 + RRDHOST_FLAG_STREAM_LABELS_STOP = (1 << 9),
703 + RRDHOST_FLAG_ACLK_STREAM_CONTEXTS = (1 << 10), // when set, we should send ACLK stream context updates
704 + RRDHOST_FLAG_INDEXED_MACHINE_GUID = (1 << 11), // when set, we have indexed its machine guid
705 + RRDHOST_FLAG_INDEXED_HOSTNAME = (1 << 12), // when set, we have indexed its hostname
706 + RRDHOST_FLAG_STREAM_COLLECTED_METRICS = (1 << 13), // when set, rrdset_done() should push metrics to parent
707 + RRDHOST_FLAG_INITIALIZED_HEALTH = (1 << 14), // the host has initialized health structures
708 + RRDHOST_FLAG_INITIALIZED_RRDPUSH = (1 << 15), // the host has initialized rrdpush structures
709 + RRDHOST_FLAG_PENDING_OBSOLETE_CHARTS = (1 << 16), // the host has pending chart obsoletions
710 + RRDHOST_FLAG_PENDING_OBSOLETE_DIMENSIONS = (1 << 17), // the host has pending dimension obsoletions
711 } RRDHOST_FLAGS;
712
713 #define rrdhost_flag_check(host, flag) (__atomic_load_n(&((host)->flags), __ATOMIC_SEQ_CST) & (flag))
@@ -868,26 +930,17 @@ struct rrdhost {
930 // all RRDCALCs are primarily allocated and linked here
931 // RRDCALCs may be linked to charts at any point
932 // (charts may or may not exist when these are loaded)
871 - RRDCALC *host_alarms;
872 - RRDCALC *alarms_with_foreach;
873 -
874 - ALARM_LOG health_log; // alarms historical events (event log)
875 - uint32_t health_last_processed_id; // the last processed health id from the log
876 - uint32_t health_max_unique_id; // the max alarm log unique id given for the host
877 - uint32_t health_max_alarm_id; // the max alarm id given for the host
933 + DICTIONARY *rrdcalc_root_index;
934
935 // templates of alarms
936 // these are used to create alarms when charts
937 // are created or renamed, that match them
882 - RRDCALCTEMPLATE *alarms_templates;
883 -
884 - // ------------------------------------------------------------------------
885 - // the charts of the host
886 -
887 - RRDSET *rrdset_root; // the host charts
888 -
889 - unsigned int obsolete_charts_count;
938 + DICTIONARY *rrdcalctemplate_root_index;
939
940 + ALARM_LOG health_log; // alarms historical events (event log)
941 + uint32_t health_last_processed_id; // the last processed health id from the log
942 + uint32_t health_max_unique_id; // the max alarm log unique id given for the host
943 + uint32_t health_max_alarm_id; // the max alarm id given for the host
944
945 // ------------------------------------------------------------------------
946 // locks
@@ -909,7 +962,8 @@ struct rrdhost {
962 DICTIONARY *rrdset_root_index_name; // the host's charts index (by name)
963
964 DICTIONARY *rrdfamily_root_index; // the host's chart families index
912 - DICTIONARY *rrdvar_root_index; // the host's chart variables index
965 + DICTIONARY *rrdvars; // the host's chart variables index
966 + // this includes custom host variables
967
968 STORAGE_INSTANCE *storage_instance[RRD_STORAGE_TIERS]; // the database instances of the storage tiers
969
@@ -973,6 +1027,11 @@ extern netdata_rwlock_t rrd_rwlock;
1027 // ----------------------------------------------------------------------------
1028
1029 extern bool is_storage_engine_shared(STORAGE_INSTANCE *engine);
1030 +extern void rrdset_index_init(RRDHOST *host);
1031 +extern void rrdset_index_destroy(RRDHOST *host);
1032 +
1033 +extern void rrddim_index_init(RRDSET *st);
1034 +extern void rrddim_index_destroy(RRDSET *st);
1035
1036 // ----------------------------------------------------------------------------
1037
@@ -1058,7 +1117,7 @@ extern void __rrd_check_wrlock(const char *file, const char *function, const uns
1117 // ----------------------------------------------------------------------------
1118 // RRDSET functions
1119
1061 -extern int rrdset_set_name(RRDSET *st, const char *name);
1120 +extern int rrdset_reset_name(RRDSET *st, const char *name);
1121
1122 extern RRDSET *rrdset_create_custom(RRDHOST *host
1123 , const char *type
@@ -1086,12 +1145,10 @@ extern void rrdhost_free_all(void);
1145 extern void rrdhost_save_all(void);
1146 extern void rrdhost_cleanup_all(void);
1147
1089 -extern void rrdhost_cleanup_orphan_hosts_nolock(RRDHOST *protected_host);
1148 extern void rrdhost_system_info_free(struct rrdhost_system_info *system_info);
1149 extern void rrdhost_free(RRDHOST *host, bool force);
1150 extern void rrdhost_save_charts(RRDHOST *host);
1151 extern void rrdhost_delete_charts(RRDHOST *host);
1094 -extern void rrd_cleanup_obsolete_charts();
1152
1153 extern int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, time_t now);
1154
@@ -1140,95 +1197,19 @@ extern void rrdset_is_obsolete(RRDSET *st);
1197 extern void rrdset_isnot_obsolete(RRDSET *st);
1198
1199 // checks if the RRDSET should be offered to viewers
1143 -#define rrdset_is_available_for_viewers(st) (!rrdset_flag_check(st, RRDSET_FLAG_HIDDEN) && !rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE) && !rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED) && (st)->dimensions && (st)->rrd_memory_mode != RRD_MEMORY_MODE_NONE)
1200 +#define rrdset_is_available_for_viewers(st) (!rrdset_flag_check(st, RRDSET_FLAG_HIDDEN) && !rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE) && !rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED) && rrdset_number_of_dimensions(st) && (st)->rrd_memory_mode != RRD_MEMORY_MODE_NONE)
1201 #define rrdset_is_available_for_exporting_and_alarms(st) (!rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE) && !rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED) && (st)->dimensions)
1202 #define rrdset_is_archived(st) (rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED) && (st)->dimensions)
1203
1147 -// get the timestamp of the last entry in the round robin database
1148 -static inline time_t rrddim_last_entry_t(RRDDIM *rd) {
1149 - time_t latest = rd->tiers[0]->query_ops.latest_time(rd->tiers[0]->db_metric_handle);
1150 -
1151 - for(int tier = 1; tier < storage_tiers ;tier++) {
1152 - if(unlikely(!rd->tiers[tier])) continue;
1153 -
1154 - time_t t = rd->tiers[tier]->query_ops.latest_time(rd->tiers[tier]->db_metric_handle);
1155 - if(t > latest)
1156 - latest = t;
1157 - }
1158 -
1159 - return latest;
1160 -}
1161 -
1162 -static inline time_t rrddim_first_entry_t(RRDDIM *rd) {
1163 - time_t oldest = 0;
1164 -
1165 - for(int tier = 0; tier < storage_tiers ;tier++) {
1166 - if(unlikely(!rd->tiers[tier])) continue;
1167 -
1168 - time_t t = rd->tiers[tier]->query_ops.oldest_time(rd->tiers[tier]->db_metric_handle);
1169 - if(t != 0 && (oldest == 0 || t < oldest))
1170 - oldest = t;
1171 - }
1172 -
1173 - return oldest;
1174 -}
1175 -
1176 -// get the timestamp of the last entry in the round robin database
1177 -static inline time_t rrdset_last_entry_t_nolock(RRDSET *st) {
1178 - RRDDIM *rd;
1179 - time_t last_entry_t = 0;
1180 -
1181 - rrddim_foreach_read(rd, st) {
1182 - time_t t = rrddim_last_entry_t(rd);
1183 - if(t > last_entry_t) last_entry_t = t;
1184 - }
1185 -
1186 - return last_entry_t;
1187 -}
1188 -
1189 -static inline time_t rrdset_last_entry_t(RRDSET *st) {
1190 - time_t last_entry_t;
1191 -
1192 - netdata_rwlock_rdlock(&st->rrdset_rwlock);
1193 - last_entry_t = rrdset_last_entry_t_nolock(st);
1194 - netdata_rwlock_unlock(&st->rrdset_rwlock);
1195 -
1196 - return last_entry_t;
1197 -}
1198 -
1199 -// get the timestamp of first entry in the round robin database
1200 -static inline time_t rrdset_first_entry_t_nolock(RRDSET *st) {
1201 - RRDDIM *rd;
1202 - time_t first_entry_t = LONG_MAX;
1203 -
1204 - rrddim_foreach_read(rd, st) {
1205 - time_t t = rrddim_first_entry_t(rd);
1206 - if(t < first_entry_t)
1207 - first_entry_t = t;
1208 - }
1209 -
1210 - if (unlikely(LONG_MAX == first_entry_t)) return 0;
1211 - return first_entry_t;
1212 -}
1213 -
1214 -static inline time_t rrdset_first_entry_t(RRDSET *st)
1215 -{
1216 - time_t first_entry_t;
1217 -
1218 - netdata_rwlock_rdlock(&st->rrdset_rwlock);
1219 - first_entry_t = rrdset_first_entry_t_nolock(st);
1220 - netdata_rwlock_unlock(&st->rrdset_rwlock);
1221 -
1222 - return first_entry_t;
1223 -}
1224 -
1225 -time_t rrdhost_last_entry_t(RRDHOST *h);
1204 +extern time_t rrddim_first_entry_t(RRDDIM *rd);
1205 +extern time_t rrddim_last_entry_t(RRDDIM *rd);
1206 +extern time_t rrdset_last_entry_t(RRDSET *st);
1207 +extern time_t rrdset_first_entry_t(RRDSET *st);
1208 +extern time_t rrdhost_last_entry_t(RRDHOST *h);
1209
1210 // ----------------------------------------------------------------------------
1211 // RRD DIMENSION functions
1212
1230 -extern void rrdcalc_link_to_rrddim(RRDDIM *rd, RRDSET *st, RRDHOST *host);
1231 -
1213 extern RRDDIM *rrddim_add_custom(RRDSET *st
1214 , const char *id
1215 , const char *name
@@ -1241,7 +1222,7 @@ extern RRDDIM *rrddim_add_custom(RRDSET *st
1222 #define rrddim_add(st, id, name, multiplier, divisor, algorithm) \
1223 rrddim_add_custom(st, id, name, multiplier, divisor, algorithm, (st)->rrd_memory_mode)
1224
1244 -extern int rrddim_set_name(RRDSET *st, RRDDIM *rd, const char *name);
1225 +extern int rrddim_reset_name(RRDSET *st, RRDDIM *rd, const char *name);
1226 extern int rrddim_set_algorithm(RRDSET *st, RRDDIM *rd, RRD_ALGORITHM algorithm);
1227 extern int rrddim_set_multiplier(RRDSET *st, RRDDIM *rd, collected_number multiplier);
1228 extern int rrddim_set_divisor(RRDSET *st, RRDDIM *rd, collected_number divisor);
@@ -1270,19 +1251,17 @@ extern char *rrdset_strncpyz_name(char *to, const char *from, size_t length);
1251 // ----------------------------------------------------------------------------
1252 // RRD internal functions
1253
1254 +extern void rrdset_delete_files(RRDSET *st);
1255 +extern void rrdset_save(RRDSET *st);
1256 +extern void rrdset_free(RRDSET *st);
1257 +
1258 #ifdef NETDATA_RRD_INTERNALS
1259
1260 extern char *rrdset_cache_dir(RRDHOST *host, const char *id);
1261
1262 extern void rrddim_free(RRDSET *st, RRDDIM *rd);
1263
1279 -extern RRDFAMILY *rrdfamily_create(RRDHOST *host, const char *id);
1280 -extern void rrdfamily_free(RRDHOST *host, RRDFAMILY *rc);
1281 -
1282 -extern void rrdset_free(RRDSET *st);
1264 extern void rrdset_reset(RRDSET *st);
1284 -extern void rrdset_save(RRDSET *st);
1285 -extern void rrdset_delete_files(RRDSET *st);
1265 extern void rrdset_delete_obsolete_dimensions(RRDSET *st);
1266
1267 extern RRDHOST *rrdhost_create(
database/rrdcalc.c
+468 -401
@@ -1,10 +1,9 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 -#define NETDATA_HEALTH_INTERNALS
3 #include "rrd.h"
4
5 // ----------------------------------------------------------------------------
7 -// RRDCALC management
6 +// RRDCALC helpers
7
8 inline const char *rrdcalc_status2string(RRDCALC_STATUS status) {
9 switch(status) {
@@ -35,7 +34,38 @@ inline const char *rrdcalc_status2string(RRDCALC_STATUS status) {
34 }
35 }
36
38 -static STRING *rrdcalc_replace_variables(const char *line, RRDCALC *rc) {
37 +uint32_t rrdcalc_get_unique_id(RRDHOST *host, STRING *chart, STRING *name, uint32_t *next_event_id) {
38 + netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
39 +
40 + // re-use old IDs, by looking them up in the alarm log
41 + ALARM_ENTRY *ae = NULL;
42 + for(ae = host->health_log.alarms; ae ;ae = ae->next) {
43 + if(unlikely(name == ae->name && chart == ae->chart)) {
44 + if(next_event_id) *next_event_id = ae->alarm_event_id + 1;
45 + break;
46 + }
47 + }
48 +
49 + uint32_t alarm_id;
50 +
51 + if(ae)
52 + alarm_id = ae->alarm_id;
53 +
54 + else {
55 + if (unlikely(!host->health_log.next_alarm_id))
56 + host->health_log.next_alarm_id = (uint32_t)now_realtime_sec();
57 +
58 + alarm_id = host->health_log.next_alarm_id++;
59 + }
60 +
61 + netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
62 + return alarm_id;
63 +}
64 +
65 +// ----------------------------------------------------------------------------
66 +// RRDCALC replacing info text variables with RRDSET labels
67 +
68 +static STRING *rrdcalc_replace_variables_with_rrdset_labels(const char *line, RRDCALC *rc) {
69 if (!line || !*line)
70 return NULL;
71
@@ -85,19 +115,62 @@ static STRING *rrdcalc_replace_variables(const char *line, RRDCALC *rc) {
115 return ret;
116 }
117
88 -void rrdcalc_update_rrdlabels(RRDSET *st) {
89 - RRDCALC *rc;
90 - foreach_rrdcalc_in_rrdset(st, rc) {
91 - if (rc->original_info) {
92 - if (rc->info)
93 - string_freez(rc->info);
118 +void rrdcalc_update_info_using_rrdset_labels(RRDCALC *rc) {
119 + if(!rc->rrdset || !rc->original_info || !rc->rrdset->rrdlabels) return;
120
95 - rc->info = rrdcalc_replace_variables(rrdcalc_original_info(rc), rc);
96 - }
121 + size_t labels_version = dictionary_version(rc->rrdset->rrdlabels);
122 + if(rc->labels_version != labels_version) {
123 +
124 + STRING *old = rc->info;
125 + rc->info = rrdcalc_replace_variables_with_rrdset_labels(rrdcalc_original_info(rc), rc);
126 + string_freez(old);
127 +
128 + rc->labels_version = labels_version;
129 }
130 }
131
100 -static void rrdsetcalc_link(RRDSET *st, RRDCALC *rc) {
132 +// ----------------------------------------------------------------------------
133 +// RRDCALC index management for RRDSET
134 +
135 +// the dictionary requires a unique key for every item
136 +// we use {chart id}.{alert name} for both the RRDHOST and RRDSET alert indexes.
137 +
138 +#define RRDCALC_MAX_KEY_SIZE 1024
139 +static size_t rrdcalc_key(char *dst, size_t dst_len, const char *chart, const char *alert) {
140 + return snprintfz(dst, dst_len, "%s/%s", chart, alert);
141 +}
142 +
143 +const RRDCALC_ACQUIRED *rrdcalc_from_rrdset_get(RRDSET *st, const char *alert_name) {
144 + char key[RRDCALC_MAX_KEY_SIZE + 1];
145 + size_t key_len = rrdcalc_key(key, RRDCALC_MAX_KEY_SIZE, rrdset_id(st), alert_name);
146 +
147 + const RRDCALC_ACQUIRED *rca = (const RRDCALC_ACQUIRED *)dictionary_get_and_acquire_item_advanced(st->rrdhost->rrdcalc_root_index, key, (ssize_t)(key_len + 1));
148 +
149 + if(!rca) {
150 + key_len = rrdcalc_key(key, RRDCALC_MAX_KEY_SIZE, rrdset_name(st), alert_name);
151 + rca = (const RRDCALC_ACQUIRED *)dictionary_get_and_acquire_item_advanced(st->rrdhost->rrdcalc_root_index, key, (ssize_t)(key_len + 1));
152 + }
153 +
154 + return rca;
155 +}
156 +
157 +void rrdcalc_from_rrdset_release(RRDSET *st, const RRDCALC_ACQUIRED *rca) {
158 + if(!rca) return;
159 +
160 + dictionary_acquired_item_release(st->rrdhost->rrdcalc_root_index, (const DICTIONARY_ITEM *)rca);
161 +}
162 +
163 +RRDCALC *rrdcalc_acquired_to_rrdcalc(const RRDCALC_ACQUIRED *rca) {
164 + if(rca)
165 + return dictionary_acquired_item_value((const DICTIONARY_ITEM *)rca);
166 +
167 + return NULL;
168 +}
169 +
170 +// ----------------------------------------------------------------------------
171 +// RRDCALC managing the linking with RRDSET
172 +
173 +static void rrdcalc_link_to_rrdset(RRDSET *st, RRDCALC *rc) {
174 RRDHOST *host = st->rrdhost;
175
176 debug(D_HEALTH, "Health linking alarm '%s.%s' to chart '%s' of host '%s'", rrdcalc_chart_name(rc), rrdcalc_name(rc), rrdset_id(st), rrdhost_hostname(host));
@@ -105,7 +178,9 @@ static void rrdsetcalc_link(RRDSET *st, RRDCALC *rc) {
178 rc->last_status_change = now_realtime_sec();
179 rc->rrdset = st;
180
108 - DOUBLE_LINKED_LIST_PREPEND_UNSAFE(st->alarms, rc, rrdset_prev, rrdset_next);
181 + netdata_rwlock_wrlock(&st->alerts.rwlock);
182 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(st->alerts.base, rc, prev, next);
183 + netdata_rwlock_unlock(&st->alerts.rwlock);
184
185 if(rc->update_every < rc->rrdset->update_every) {
186 error("Health alarm '%s.%s' has update every %d, less than chart update every %d. Setting alarm update frequency to %d.", rrdset_id(rc->rrdset), rrdcalc_name(rc), rc->update_every, rc->rrdset->update_every, rc->rrdset->update_every);
@@ -114,40 +189,61 @@ static void rrdsetcalc_link(RRDSET *st, RRDCALC *rc) {
189
190 if(!isnan(rc->green) && isnan(st->green)) {
191 debug(D_HEALTH, "Health alarm '%s.%s' green threshold set from " NETDATA_DOUBLE_FORMAT_AUTO
117 - " to " NETDATA_DOUBLE_FORMAT_AUTO ".", rrdset_id(rc->rrdset), rrdcalc_name(rc), rc->rrdset->green, rc->green);
192 + " to " NETDATA_DOUBLE_FORMAT_AUTO ".", rrdset_id(rc->rrdset), rrdcalc_name(rc), rc->rrdset->green, rc->green);
193 st->green = rc->green;
194 }
195
196 if(!isnan(rc->red) && isnan(st->red)) {
197 debug(D_HEALTH, "Health alarm '%s.%s' red threshold set from " NETDATA_DOUBLE_FORMAT_AUTO " to " NETDATA_DOUBLE_FORMAT_AUTO
123 - ".", rrdset_id(rc->rrdset), rrdcalc_name(rc), rc->rrdset->red, rc->red);
198 + ".", rrdset_id(rc->rrdset), rrdcalc_name(rc), rc->rrdset->red, rc->red);
199 st->red = rc->red;
200 }
201
127 - rc->local = rrdvar_create_and_index("local", st->rrdvar_root_index, rc->name, RRDVAR_TYPE_CALCULATED, RRDVAR_OPTION_RRDCALC_LOCAL_VAR, &rc->value);
128 - rc->family = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rc->name, RRDVAR_TYPE_CALCULATED, RRDVAR_OPTION_RRDCALC_FAMILY_VAR, &rc->value);
202 + char buf[RRDVAR_MAX_LENGTH + 1];
203 + snprintfz(buf, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_name(st), rrdcalc_name(rc));
204 + STRING *rrdset_name_rrdcalc_name = string_strdupz(buf);
205 + snprintfz(buf, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_id(st), rrdcalc_name(rc));
206 + STRING *rrdset_id_rrdcalc_name = string_strdupz(buf);
207
130 - char fullname[RRDVAR_MAX_LENGTH + 1];
131 - snprintfz(fullname, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_id(st), rrdcalc_name(rc));
132 - STRING *fullname_string = string_strdupz(fullname);
133 - rc->hostid = rrdvar_create_and_index("host", host->rrdvar_root_index, fullname_string, RRDVAR_TYPE_CALCULATED, RRDVAR_OPTION_RRDCALC_HOST_CHARTID_VAR, &rc->value);
134 -
135 - snprintfz(fullname, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_name(st), rrdcalc_name(rc));
136 - rc->hostname = rrdvar_create_and_index("host", host->rrdvar_root_index, fullname_string, RRDVAR_TYPE_CALCULATED, RRDVAR_OPTION_RRDCALC_HOST_CHARTNAME_VAR, &rc->value);
137 -
138 - string_freez(fullname_string);
139 -
140 - if(rc->hostid && !rc->hostname)
141 - rc->hostid->options |= RRDVAR_OPTION_RRDCALC_HOST_CHARTNAME_VAR;
142 -
143 - if(!rc->units) rc->units = string_dup(st->units);
144 -
145 - if (rc->original_info) {
146 - if (rc->info)
147 - string_freez(rc->info);
208 + rc->rrdvar_local = rrdvar_add_and_acquire(
209 + "local",
210 + st->rrdvars,
211 + rc->name,
212 + RRDVAR_TYPE_CALCULATED,
213 + RRDVAR_FLAG_RRDCALC_LOCAL_VAR,
214 + &rc->value);
215
149 - rc->info = rrdcalc_replace_variables(rrdcalc_original_info(rc), rc);
150 - }
216 + rc->rrdvar_family = rrdvar_add_and_acquire(
217 + "family",
218 + rrdfamily_rrdvars_dict(st->rrdfamily),
219 + rc->name,
220 + RRDVAR_TYPE_CALCULATED,
221 + RRDVAR_FLAG_RRDCALC_FAMILY_VAR,
222 + &rc->value);
223 +
224 + rc->rrdvar_host_chart_name = rrdvar_add_and_acquire(
225 + "host",
226 + host->rrdvars,
227 + rrdset_name_rrdcalc_name,
228 + RRDVAR_TYPE_CALCULATED,
229 + RRDVAR_FLAG_RRDCALC_HOST_CHARTNAME_VAR,
230 + &rc->value);
231 +
232 + rc->rrdvar_host_chart_id = rrdvar_add_and_acquire(
233 + "host",
234 + host->rrdvars,
235 + rrdset_id_rrdcalc_name,
236 + RRDVAR_TYPE_CALCULATED,
237 + RRDVAR_FLAG_RRDCALC_HOST_CHARTID_VAR | ((rc->rrdvar_host_chart_name) ? 0 : RRDVAR_FLAG_RRDCALC_HOST_CHARTNAME_VAR),
238 + &rc->value);
239 +
240 + string_freez(rrdset_id_rrdcalc_name);
241 + string_freez(rrdset_name_rrdcalc_name);
242 +
243 + if(!rc->units)
244 + rc->units = string_dup(st->units);
245 +
246 + rrdcalc_update_info_using_rrdset_labels(rc);
247
248 time_t now = now_realtime_sec();
249
@@ -177,43 +273,10 @@ static void rrdsetcalc_link(RRDSET *st, RRDCALC *rc) {
273 0,
274 rrdcalc_isrepeating(rc)?HEALTH_ENTRY_FLAG_IS_REPEATING:0);
275
180 - health_alarm_log(host, ae);
181 -}
182 -
183 -static inline int rrdcalc_is_matching_rrdset(RRDCALC *rc, RRDSET *st) {
184 - if ( (rc->chart != st->id)
185 - && (rc->chart != st->name))
186 - return 0;
187 -
188 - if (rc->module_pattern && !simple_pattern_matches(rc->module_pattern, rrdset_module_name(st)))
189 - return 0;
190 -
191 - if (rc->plugin_pattern && !simple_pattern_matches(rc->plugin_pattern, rrdset_plugin_name(st)))
192 - return 0;
193 -
194 - if (st->rrdhost->rrdlabels && rc->host_labels_pattern && !rrdlabels_match_simple_pattern_parsed(st->rrdhost->rrdlabels, rc->host_labels_pattern, '='))
195 - return 0;
196 -
197 - return 1;
198 -}
199 -
200 -// this has to be called while the RRDHOST is locked
201 -inline void rrdsetcalc_link_matching(RRDSET *st) {
202 - RRDHOST *host = st->rrdhost;
203 - // debug(D_HEALTH, "find matching alarms for chart '%s'", st->id);
204 -
205 - RRDCALC *rc;
206 - foreach_rrdcalc_in_rrdhost(host, rc) {
207 - if(unlikely(rc->rrdset))
208 - continue;
209 -
210 - if(unlikely(rrdcalc_is_matching_rrdset(rc, st)))
211 - rrdsetcalc_link(st, rc);
212 - }
276 + health_alarm_log_add_entry(host, ae);
277 }
278
215 -// this has to be called while the RRDHOST is locked
216 -inline void rrdsetcalc_unlink(RRDCALC *rc) {
279 +static void rrdcalc_unlink_from_rrdset(RRDCALC *rc, bool having_ll_wrlock) {
280 RRDSET *st = rc->rrdset;
281
282 if(!st) {
@@ -252,139 +315,176 @@ inline void rrdsetcalc_unlink(RRDCALC *rc) {
315 0,
316 0);
317
255 - health_alarm_log(host, ae);
318 + health_alarm_log_add_entry(host, ae);
319
320 debug(D_HEALTH, "Health unlinking alarm '%s.%s' from chart '%s' of host '%s'", rrdcalc_chart_name(rc), rrdcalc_name(rc), rrdset_id(st), rrdhost_hostname(host));
321
322 // unlink it
260 - DOUBLE_LINKED_LIST_REMOVE_UNSAFE(st->alarms, rc, rrdset_prev, rrdset_next);
261 -
262 - rrdvar_free(host, st->rrdvar_root_index, rc->local);
263 - rc->local = NULL;
323
265 - rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rc->family);
266 - rc->family = NULL;
324 + if(!having_ll_wrlock)
325 + netdata_rwlock_wrlock(&st->alerts.rwlock);
326
268 - rrdvar_free(host, host->rrdvar_root_index, rc->hostid);
269 - rc->hostid = NULL;
327 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(st->alerts.base, rc, prev, next);
328
271 - rrdvar_free(host, host->rrdvar_root_index, rc->hostname);
272 - rc->hostname = NULL;
329 + if(!having_ll_wrlock)
330 + netdata_rwlock_unlock(&st->alerts.rwlock);
331
332 rc->rrdset = NULL;
333
334 + rrdvar_release_and_del(st->rrdvars, rc->rrdvar_local);
335 + rc->rrdvar_local = NULL;
336 +
337 + rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rc->rrdvar_family);
338 + rc->rrdvar_family = NULL;
339 +
340 + rrdvar_release_and_del(host->rrdvars, rc->rrdvar_host_chart_id);
341 + rc->rrdvar_host_chart_id = NULL;
342 +
343 + rrdvar_release_and_del(host->rrdvars, rc->rrdvar_host_chart_name);
344 + rc->rrdvar_host_chart_name = NULL;
345 +
346 // RRDCALC will remain in RRDHOST
347 // so that if the matching chart is found in the future
348 // it will be applied automatically
349 }
350
281 -RRDCALC *rrdcalc_find(RRDSET *st, const char *name) {
282 - RRDCALC *rc = NULL;
351 +static inline bool rrdcalc_check_if_it_matches_rrdset(RRDCALC *rc, RRDSET *st) {
352 + if ( (rc->chart != st->id)
353 + && (rc->chart != st->name))
354 + return false;
355
284 - STRING *name_string = string_strdupz(name);
356 + if (rc->module_pattern && !simple_pattern_matches(rc->module_pattern, rrdset_module_name(st)))
357 + return false;
358
286 - foreach_rrdcalc_in_rrdset(st, rc) {
287 - if(unlikely(rc->name == name_string))
288 - break;
289 - }
359 + if (rc->plugin_pattern && !simple_pattern_matches(rc->plugin_pattern, rrdset_plugin_name(st)))
360 + return false;
361
291 - string_freez(name_string);
362 + if (st->rrdhost->rrdlabels && rc->host_labels_pattern && !rrdlabels_match_simple_pattern_parsed(st->rrdhost->rrdlabels, rc->host_labels_pattern, '='))
363 + return false;
364
293 - return rc;
365 + return true;
366 }
367
296 -inline int rrdcalc_exists(RRDHOST *host, const char *chart, const char *name) {
368 +void rrdcalc_link_matching_alerts_to_rrdset(RRDSET *st) {
369 + RRDHOST *host = st->rrdhost;
370 + // debug(D_HEALTH, "find matching alarms for chart '%s'", st->id);
371 +
372 RRDCALC *rc;
373 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
374 + if(rc->rrdset)
375 + continue;
376
299 - if(unlikely(!chart)) {
300 - error("attempt to find RRDCALC '%s' without giving a chart name", name);
301 - return 1;
377 + if(unlikely(rrdcalc_check_if_it_matches_rrdset(rc, st)))
378 + rrdcalc_link_to_rrdset(st, rc);
379 }
380 + foreach_rrdcalc_in_rrdhost_done(rc);
381 +}
382
304 - STRING *name_string = string_strdupz(name);
305 - STRING *chart_string = string_strdupz(chart);
306 - int ret = 0;
383 +static inline int rrdcalc_check_and_link_rrdset_callback(RRDSET *st, void *rrdcalc) {
384 + RRDCALC *rc = rrdcalc;
385
308 - // make sure it does not already exist
309 - foreach_rrdcalc_in_rrdhost(host, rc) {
310 - if (unlikely(rc->chart == chart_string && rc->name == name_string)) {
311 - debug(D_HEALTH, "Health alarm '%s/%s' already exists in host '%s'.", chart, name, rrdhost_hostname(host));
312 - info("Health alarm '%s/%s' already exists in host '%s'.", chart, name, rrdhost_hostname(host));
313 - ret = 1;
314 - break;
315 - }
386 + if(unlikely(rrdcalc_check_if_it_matches_rrdset(rc, st))) {
387 + rrdcalc_link_to_rrdset(st, rc);
388 + return -1;
389 }
390
318 - string_freez(name_string);
319 - string_freez(chart_string);
320 -
321 - return ret;
391 + return 0;
392 }
393
324 -inline uint32_t rrdcalc_get_unique_id(RRDHOST *host, STRING *chart, STRING *name, uint32_t *next_event_id) {
325 - // re-use old IDs, by looking them up in the alarm log
326 - ALARM_ENTRY *ae = NULL;
327 - for(ae = host->health_log.alarms; ae ;ae = ae->next) {
328 - if(unlikely(name == ae->name && chart == ae->chart)) {
329 - if(next_event_id) *next_event_id = ae->alarm_event_id + 1;
330 - break;
394 +// ----------------------------------------------------------------------------
395 +// RRDCALC rrdhost index management - constructor
396 +
397 +struct rrdcalc_constructor {
398 + RRDHOST *rrdhost; // the host we operate upon
399 + RRDCALC *from_config; // points to the original RRDCALC, as loaded from the config
400 + RRDCALCTEMPLATE *from_rrdcalctemplate; // the template this alert is generated from
401 + RRDSET *rrdset; // when this comes from rrdcalctemplate, we have a matching rrdset
402 + const char *overwrite_alert_name; // when we have a dimension foreach, the alert is renamed
403 + const char *overwrite_dimensions; // when we have a dimension foreach, the dimensions filter is renamed
404 +
405 + enum {
406 + RRDCALC_REACT_NONE,
407 + RRDCALC_REACT_NEW,
408 + } react_action;
409 +};
410 +
411 +static void rrdcalc_rrdhost_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdcalc, void *constructor_data) {
412 + RRDCALC *rc = rrdcalc;
413 + struct rrdcalc_constructor *ctr = constructor_data;
414 + RRDHOST *host = ctr->rrdhost;
415 +
416 + rc->key = string_strdupz(dictionary_acquired_item_name(item));
417 +
418 + if(ctr->from_rrdcalctemplate) {
419 + rc->run_flags |= RRDCALC_FLAG_FROM_TEMPLATE;
420 +
421 + RRDCALCTEMPLATE *rt = ctr->from_rrdcalctemplate;
422 + RRDSET *st = ctr->rrdset;
423 +
424 + rc->next_event_id = 1;
425 + rc->name = (ctr->overwrite_alert_name) ? string_strdupz(ctr->overwrite_alert_name) : string_dup(rt->name);
426 + rc->chart = string_dup(st->id);
427 + uuid_copy(rc->config_hash_id, rt->config_hash_id);
428 +
429 + rc->dimensions = (ctr->overwrite_dimensions) ? string_strdupz(ctr->overwrite_dimensions) : string_dup(rt->dimensions);
430 + rc->foreach_dimension = NULL;
431 + rc->foreach_dimension_pattern = NULL;
432 +
433 + rc->green = rt->green;
434 + rc->red = rt->red;
435 + rc->value = NAN;
436 + rc->old_value = NAN;
437 +
438 + rc->delay_up_duration = rt->delay_up_duration;
439 + rc->delay_down_duration = rt->delay_down_duration;
440 + rc->delay_max_duration = rt->delay_max_duration;
441 + rc->delay_multiplier = rt->delay_multiplier;
442 +
443 + rc->last_repeat = 0;
444 + rc->times_repeat = 0;
445 + rc->warn_repeat_every = rt->warn_repeat_every;
446 + rc->crit_repeat_every = rt->crit_repeat_every;
447 +
448 + rc->group = rt->group;
449 + rc->after = rt->after;
450 + rc->before = rt->before;
451 + rc->update_every = rt->update_every;
452 + rc->options = rt->options;
453 +
454 + rc->exec = string_dup(rt->exec);
455 + rc->recipient = string_dup(rt->recipient);
456 + rc->source = string_dup(rt->source);
457 + rc->units = string_dup(rt->units);
458 + rc->info = string_dup(rt->info);
459 + rc->original_info = string_dup(rt->info);
460 +
461 + rc->classification = string_dup(rt->classification);
462 + rc->component = string_dup(rt->component);
463 + rc->type = string_dup(rt->type);
464 +
465 + if(rt->calculation) {
466 + rc->calculation = expression_parse(rt->calculation->source, NULL, NULL);
467 + if(!rc->calculation)
468 + error("Health alarm '%s.%s': failed to parse calculation expression '%s'", rrdset_id(st), rrdcalctemplate_name(rt), rt->calculation->source);
469 + }
470 + if(rt->warning) {
471 + rc->warning = expression_parse(rt->warning->source, NULL, NULL);
472 + if(!rc->warning)
473 + error("Health alarm '%s.%s': failed to re-parse warning expression '%s'", rrdset_id(st), rrdcalctemplate_name(rt), rt->warning->source);
474 + }
475 + if(rt->critical) {
476 + rc->critical = expression_parse(rt->critical->source, NULL, NULL);
477 + if(!rc->critical)
478 + error("Health alarm '%s.%s': failed to re-parse critical expression '%s'", rrdset_id(st), rrdcalctemplate_name(rt), rt->critical->source);
479 }
480 }
333 -
334 - if(ae)
335 - return ae->alarm_id;
336 -
337 - if (unlikely(!host->health_log.next_alarm_id))
338 - host->health_log.next_alarm_id = (uint32_t)now_realtime_sec();
339 -
340 - return host->health_log.next_alarm_id++;
341 -}
342 -
343 -/**
344 - * Alarm name with dimension
345 - *
346 - * Change the name of the current alarm appending a new diagram.
347 - *
348 - * @param name the alarm name
349 - * @param namelen is the length of the previous vector.
350 - * @param dim the dimension of the chart.
351 - * @param dimlen is the length of the previous vector.
352 - *
353 - * @return It returns the new name on success and the old otherwise
354 - */
355 -char *alarm_name_with_dim(const char *name, size_t namelen, const char *dim, size_t dimlen) {
356 - char *newname,*move;
357 -
358 - newname = mallocz(namelen + dimlen + 2);
359 - move = newname;
360 - memcpy(move, name, namelen);
361 - move += namelen;
362 -
363 - *move++ = '_';
364 - memcpy(move, dim, dimlen);
365 - move += dimlen;
366 - *move = '\0';
367 -
368 - return newname;
369 -}
370 -
371 -/**
372 - * Remove pipe comma
373 - *
374 - * Remove the pipes and commas converting to space.
375 - *
376 - * @param str the string to change.
377 - */
378 -void dimension_remove_pipe_comma(char *str) {
379 - while(*str) {
380 - if(*str == '|' || *str == ',') *str = ' ';
381 -
382 - str++;
481 + else if(ctr->from_config) {
482 + // dictionary has already copied all the members values and pointers
483 + // no need for additional work in this case
484 + ;
485 }
384 -}
486
386 -inline void rrdcalc_add_to_host(RRDHOST *host, RRDCALC *rc) {
387 - rrdhost_check_rdlock(host);
487 + rc->id = rrdcalc_get_unique_id(host, rc->chart, rc->name, &rc->next_event_id);
488
489 if(rc->calculation) {
490 rc->calculation->status = &rc->status;
@@ -410,218 +510,66 @@ inline void rrdcalc_add_to_host(RRDHOST *host, RRDCALC *rc) {
510 rc->critical->rrdcalc = rc;
511 }
512
413 - if(!rc->foreachdim) {
414 - // link it to the host alarms list
415 - DOUBLE_LINKED_LIST_APPEND_UNSAFE(host->host_alarms, rc, prev, next);
416 -
417 - // link it to its chart
418 - RRDSET *st;
419 - rrdset_foreach_read(st, host) {
420 - if(rrdcalc_is_matching_rrdset(rc, st)) {
421 - rrdsetcalc_link(st, rc);
422 - break;
423 - }
424 - }
425 - }
426 - else {
427 - DOUBLE_LINKED_LIST_APPEND_UNSAFE(host->alarms_with_foreach, rc, prev, next);
428 - // We are not linking this alarm directly to the host here
429 - // It will eventually be done if it matches the dimensions
430 - }
431 -}
432 -
433 -inline RRDCALC *rrdcalc_create_from_template(RRDHOST *host, RRDCALCTEMPLATE *rt, const char *chart) {
434 - debug(D_HEALTH, "Health creating dynamic alarm (from template) '%s.%s'", chart, rrdcalctemplate_name(rt));
435 -
436 - if(rrdcalc_exists(host, chart, rrdcalctemplate_name(rt)))
437 - return NULL;
438 -
439 - RRDCALC *rc = callocz(1, sizeof(RRDCALC));
440 - rc->next_event_id = 1;
441 - rc->name = string_dup(rt->name);
442 - rc->chart = string_strdupz(chart);
443 - uuid_copy(rc->config_hash_id, rt->config_hash_id);
444 -
445 - rc->id = rrdcalc_get_unique_id(host, rc->chart, rc->name, &rc->next_event_id);
446 -
447 - rc->dimensions = string_dup(rt->dimensions);
448 - rc->foreachdim = string_dup(rt->foreachdim);
449 - if(rt->foreachdim)
450 - rc->spdim = health_pattern_from_foreach(rrdcalc_foreachdim(rc));
451 -
452 - rc->foreachcounter = rt->foreachcounter;
453 -
454 - rc->green = rt->green;
455 - rc->red = rt->red;
456 - rc->value = NAN;
457 - rc->old_value = NAN;
458 -
459 - rc->delay_up_duration = rt->delay_up_duration;
460 - rc->delay_down_duration = rt->delay_down_duration;
461 - rc->delay_max_duration = rt->delay_max_duration;
462 - rc->delay_multiplier = rt->delay_multiplier;
463 -
464 - rc->last_repeat = 0;
465 - rc->times_repeat = 0;
466 - rc->warn_repeat_every = rt->warn_repeat_every;
467 - rc->crit_repeat_every = rt->crit_repeat_every;
468 -
469 - rc->group = rt->group;
470 - rc->after = rt->after;
471 - rc->before = rt->before;
472 - rc->update_every = rt->update_every;
473 - rc->options = rt->options;
474 -
475 - rc->exec = string_dup(rt->exec);
476 - rc->recipient = string_dup(rt->recipient);
477 - rc->source = string_dup(rt->source);
478 - rc->units = string_dup(rt->units);
479 - rc->info = string_dup(rt->info);
480 - rc->original_info = string_dup(rt->info);
481 -
482 - rc->classification = string_dup(rt->classification);
483 - rc->component = string_dup(rt->component);
484 - rc->type = string_dup(rt->type);
485 -
486 - if(rt->calculation) {
487 - rc->calculation = expression_parse(rt->calculation->source, NULL, NULL);
488 - if(!rc->calculation)
489 - error("Health alarm '%s.%s': failed to parse calculation expression '%s'", chart, rrdcalctemplate_name(rt), rt->calculation->source);
490 - }
491 - if(rt->warning) {
492 - rc->warning = expression_parse(rt->warning->source, NULL, NULL);
493 - if(!rc->warning)
494 - error("Health alarm '%s.%s': failed to re-parse warning expression '%s'", chart, rrdcalctemplate_name(rt), rt->warning->source);
495 - }
496 - if(rt->critical) {
497 - rc->critical = expression_parse(rt->critical->source, NULL, NULL);
498 - if(!rc->critical)
499 - error("Health alarm '%s.%s': failed to re-parse critical expression '%s'", chart, rrdcalctemplate_name(rt), rt->critical->source);
500 - }
501 -
502 - debug(D_HEALTH, "Health runtime added alarm '%s.%s': exec '%s', recipient '%s', green " NETDATA_DOUBLE_FORMAT_AUTO
503 - ", red " NETDATA_DOUBLE_FORMAT_AUTO
504 - ", lookup: group %d, after %d, before %d, options %u, dimensions '%s', for each dimension '%s', update every %d, calculation '%s', warning '%s', critical '%s', source '%s', delay up %d, delay down %d, delay max %d, delay_multiplier %f, warn_repeat_every %u, crit_repeat_every %u",
505 - rrdcalc_chart_name(rc),
506 - rrdcalc_name(rc),
507 - (rc->exec)?rrdcalc_exec(rc):"DEFAULT",
508 - (rc->recipient)?rrdcalc_recipient(rc):"DEFAULT",
509 - rc->green,
510 - rc->red,
511 - (int)rc->group,
512 - rc->after,
513 - rc->before,
514 - rc->options,
515 - (rc->dimensions)?rrdcalc_dimensions(rc):"NONE",
516 - (rc->foreachdim)?rrdcalc_foreachdim(rc):"NONE",
517 - rc->update_every,
518 - (rc->calculation)?rc->calculation->parsed_as:"NONE",
519 - (rc->warning)?rc->warning->parsed_as:"NONE",
520 - (rc->critical)?rc->critical->parsed_as:"NONE",
521 - rrdcalc_source(rc),
522 - rc->delay_up_duration,
523 - rc->delay_down_duration,
524 - rc->delay_max_duration,
525 - rc->delay_multiplier,
526 - rc->warn_repeat_every,
527 - rc->crit_repeat_every
513 + debug(D_HEALTH, "Health added alarm '%s.%s': exec '%s', recipient '%s', green " NETDATA_DOUBLE_FORMAT_AUTO
514 + ", red " NETDATA_DOUBLE_FORMAT_AUTO
515 + ", lookup: group %d, after %d, before %d, options %u, dimensions '%s', for each dimension '%s', update every %d, calculation '%s', warning '%s', critical '%s', source '%s', delay up %d, delay down %d, delay max %d, delay_multiplier %f, warn_repeat_every %u, crit_repeat_every %u",
516 + rrdcalc_chart_name(rc),
517 + rrdcalc_name(rc),
518 + (rc->exec)?rrdcalc_exec(rc):"DEFAULT",
519 + (rc->recipient)?rrdcalc_recipient(rc):"DEFAULT",
520 + rc->green,
521 + rc->red,
522 + (int)rc->group,
523 + rc->after,
524 + rc->before,
525 + rc->options,
526 + (rc->dimensions)?rrdcalc_dimensions(rc):"NONE",
527 + (rc->foreach_dimension)?rrdcalc_foreachdim(rc):"NONE",
528 + rc->update_every,
529 + (rc->calculation)?rc->calculation->parsed_as:"NONE",
530 + (rc->warning)?rc->warning->parsed_as:"NONE",
531 + (rc->critical)?rc->critical->parsed_as:"NONE",
532 + rrdcalc_source(rc),
533 + rc->delay_up_duration,
534 + rc->delay_down_duration,
535 + rc->delay_max_duration,
536 + rc->delay_multiplier,
537 + rc->warn_repeat_every,
538 + rc->crit_repeat_every
539 );
540
530 - rrdcalc_add_to_host(host, rc);
531 - return rc;
541 + ctr->react_action = RRDCALC_REACT_NEW;
542 }
543
534 -/**
535 - * Create from RRDCALC
536 - *
537 - * Create a new alarm using another alarm as template.
538 - *
539 - * @param rc is the alarm that will be used as source
540 - * @param host is the host structure.
541 - * @param name is the newest chart name.
542 - * @param dimension is the current dimension
543 - * @param foreachdim the whole list of dimension
544 - *
545 - * @return it returns the new alarm changed.
546 - */
547 -inline RRDCALC *rrdcalc_create_from_rrdcalc(RRDCALC *rc, RRDHOST *host, const char *name, const char *dimension) {
548 - RRDCALC *newrc = callocz(1, sizeof(RRDCALC));
549 -
550 - newrc->next_event_id = 1;
551 - newrc->name = string_strdupz(name);
552 - newrc->chart = string_dup(rc->chart);
553 - newrc->id = rrdcalc_get_unique_id(host, newrc->chart, newrc->name, &rc->next_event_id);
554 - uuid_copy(newrc->config_hash_id, *((uuid_t *) &rc->config_hash_id));
555 -
556 - newrc->dimensions = string_strdupz(dimension);
557 - newrc->foreachdim = NULL;
558 - rc->foreachcounter++;
559 - newrc->foreachcounter = rc->foreachcounter;
560 -
561 - newrc->green = rc->green;
562 - newrc->red = rc->red;
563 - newrc->value = NAN;
564 - newrc->old_value = NAN;
565 -
566 - newrc->delay_up_duration = rc->delay_up_duration;
567 - newrc->delay_down_duration = rc->delay_down_duration;
568 - newrc->delay_max_duration = rc->delay_max_duration;
569 - newrc->delay_multiplier = rc->delay_multiplier;
570 -
571 - newrc->last_repeat = 0;
572 - newrc->times_repeat = 0;
573 - newrc->warn_repeat_every = rc->warn_repeat_every;
574 - newrc->crit_repeat_every = rc->crit_repeat_every;
575 -
576 - newrc->group = rc->group;
577 - newrc->after = rc->after;
578 - newrc->before = rc->before;
579 - newrc->update_every = rc->update_every;
580 - newrc->options = rc->options;
581 -
582 - newrc->exec = string_dup(rc->exec);
583 - newrc->recipient = string_dup(rc->recipient);
584 - newrc->source = string_dup(rc->source);
585 - newrc->units = string_dup(rc->units);
586 - newrc->info = string_dup(rc->info);
587 - newrc->original_info = string_dup(rc->original_info);
588 -
589 - newrc->classification = string_dup(rc->classification);
590 - newrc->component = string_dup(rc->component);
591 - newrc->type = string_dup(rc->type);
544 +static void rrdcalc_rrdhost_react_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdcalc, void *constructor_data) {
545 + RRDCALC *rc = rrdcalc;
546 + struct rrdcalc_constructor *ctr = constructor_data;
547 + RRDHOST *host = ctr->rrdhost;
548
593 - if(rc->calculation) {
594 - newrc->calculation = expression_parse(rc->calculation->source, NULL, NULL);
595 - if(!newrc->calculation)
596 - error("Health alarm '%s.%s': failed to parse calculation expression '%s'", rrdcalc_chart_name(rc), rrdcalc_name(rc), rc->calculation->source);
597 - }
598 -
599 - if(rc->warning) {
600 - newrc->warning = expression_parse(rc->warning->source, NULL, NULL);
601 - if(!newrc->warning)
602 - error("Health alarm '%s.%s': failed to re-parse warning expression '%s'", rrdcalc_chart_name(rc), rrdcalc_name(rc), rc->warning->source);
603 - }
549 + if(ctr->react_action == RRDCALC_REACT_NEW) {
550 + if(ctr->rrdset)
551 + rrdcalc_link_to_rrdset(ctr->rrdset, rc);
552
605 - if(rc->critical) {
606 - newrc->critical = expression_parse(rc->critical->source, NULL, NULL);
607 - if(!newrc->critical)
608 - error("Health alarm '%s.%s': failed to re-parse critical expression '%s'", rrdcalc_chart_name(rc), rrdcalc_name(rc), rc->critical->source);
553 + else if (ctr->from_rrdcalctemplate)
554 + rrdcontext_foreach_instance_with_rrdset_in_context(host, string2str(ctr->from_rrdcalctemplate->context), rrdcalc_check_and_link_rrdset_callback, rc);
555 }
610 -
611 - return newrc;
556 }
557
614 -void rrdcalc_free(RRDCALC *rc) {
558 +// ----------------------------------------------------------------------------
559 +// RRDCALC rrdhost index management - destructor
560 +
561 +static void rrdcalc_free_internals(RRDCALC *rc) {
562 if(unlikely(!rc)) return;
563
564 expression_free(rc->calculation);
565 expression_free(rc->warning);
566 expression_free(rc->critical);
567
568 + string_freez(rc->key);
569 string_freez(rc->name);
570 string_freez(rc->chart);
571 string_freez(rc->dimensions);
624 - string_freez(rc->foreachdim);
572 + string_freez(rc->foreach_dimension);
573 string_freez(rc->exec);
574 string_freez(rc->recipient);
575 string_freez(rc->source);
@@ -635,42 +583,138 @@ void rrdcalc_free(RRDCALC *rc) {
583 string_freez(rc->module_match);
584 string_freez(rc->plugin_match);
585
638 - simple_pattern_free(rc->spdim);
586 + simple_pattern_free(rc->foreach_dimension_pattern);
587 simple_pattern_free(rc->host_labels_pattern);
588 simple_pattern_free(rc->module_pattern);
589 simple_pattern_free(rc->plugin_pattern);
590 +}
591
643 - freez(rc->family);
644 - freez(rc);
592 +static void rrdcalc_rrdhost_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdcalc, void *rrdhost __maybe_unused) {
593 + RRDCALC *rc = rrdcalc;
594 + //RRDHOST *host = rrdhost;
595 +
596 + if(unlikely(rc->rrdset))
597 + rrdcalc_unlink_from_rrdset(rc, false);
598 +
599 + // any destruction actions that require other locks
600 + // have to be placed in rrdcalc_del(), because the object is actually locked for deletion
601 +
602 + rrdcalc_free_internals(rc);
603 }
604
647 -void rrdcalc_unlink_and_free(RRDHOST *host, RRDCALC *rc) {
648 - if(unlikely(!rc)) return;
605 +// ----------------------------------------------------------------------------
606 +// RRDCALC rrdhost index management - index API
607 +
608 +void rrdcalc_rrdhost_index_init(RRDHOST *host) {
609 + if(!host->rrdcalc_root_index) {
610 + host->rrdcalc_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
611 +
612 + dictionary_register_insert_callback(host->rrdcalc_root_index, rrdcalc_rrdhost_insert_callback, NULL);
613 + dictionary_register_react_callback(host->rrdcalc_root_index, rrdcalc_rrdhost_react_callback, NULL);
614 + dictionary_register_delete_callback(host->rrdcalc_root_index, rrdcalc_rrdhost_delete_callback, host);
615 + }
616 +}
617
650 - debug(D_HEALTH, "Health removing alarm '%s.%s' of host '%s'", rrdcalc_chart_name(rc), rrdcalc_name(rc), rrdhost_hostname(host));
618 +void rrdcalc_rrdhost_index_destroy(RRDHOST *host) {
619 + dictionary_destroy(host->rrdcalc_root_index);
620 + host->rrdcalc_root_index = NULL;
621 +}
622 +
623 +void rrdcalc_add_from_rrdcalctemplate(RRDHOST *host, RRDCALCTEMPLATE *rt, RRDSET *st, const char *overwrite_alert_name, const char *overwrite_dimensions) {
624 + char key[RRDCALC_MAX_KEY_SIZE + 1];
625 + size_t key_len = rrdcalc_key(key, RRDCALC_MAX_KEY_SIZE, rrdset_id(st),
626 + overwrite_alert_name?overwrite_alert_name:string2str(rt->name));
627 +
628 + struct rrdcalc_constructor tmp = {
629 + .rrdhost = host,
630 + .from_config = NULL,
631 + .from_rrdcalctemplate = rt,
632 + .rrdset = st,
633 + .overwrite_alert_name = overwrite_alert_name,
634 + .overwrite_dimensions = overwrite_dimensions,
635 + .react_action = RRDCALC_REACT_NONE,
636 + };
637 +
638 + dictionary_set_advanced(host->rrdcalc_root_index, key, (ssize_t)(key_len + 1), NULL, sizeof(RRDCALC), &tmp);
639 + if(tmp.react_action != RRDCALC_REACT_NEW)
640 + error("RRDCALC: from template '%s' on chart '%s' with key '%s', failed to be added to host '%s'. It already exists.",
641 + string2str(rt->name), rrdset_id(st), key, rrdhost_hostname(host));
642 +}
643 +
644 +int rrdcalc_add_from_config(RRDHOST *host, RRDCALC *rc) {
645 + if(!rc->chart) {
646 + error("Health configuration for alarm '%s' does not have a chart", rrdcalc_name(rc));
647 + return 0;
648 + }
649 +
650 + if(!rc->update_every) {
651 + error("Health configuration for alarm '%s.%s' has no frequency (parameter 'every'). Ignoring it.", rrdcalc_chart_name(rc), rrdcalc_name(rc));
652 + return 0;
653 + }
654 +
655 + if(!RRDCALC_HAS_DB_LOOKUP(rc) && !rc->calculation && !rc->warning && !rc->critical) {
656 + error("Health configuration for alarm '%s.%s' is useless (no db lookup, no calculation, no warning and no critical expressions)", rrdcalc_chart_name(rc), rrdcalc_name(rc));
657 + return 0;
658 + }
659 +
660 + char key[RRDCALC_MAX_KEY_SIZE + 1];
661 + size_t key_len = rrdcalc_key(key, RRDCALC_MAX_KEY_SIZE, string2str(rc->chart), string2str(rc->name));
662 +
663 + struct rrdcalc_constructor tmp = {
664 + .rrdhost = host,
665 + .from_config = rc,
666 + .from_rrdcalctemplate = NULL,
667 + .rrdset = NULL,
668 + .react_action = RRDCALC_REACT_NONE,
669 + };
670 +
671 + int ret = 1;
672 + RRDCALC *t = dictionary_set_advanced(host->rrdcalc_root_index, key, (ssize_t)(key_len + 1), rc, sizeof(RRDCALC), &tmp);
673 + if(tmp.react_action == RRDCALC_REACT_NEW) {
674 + // we copied rc into the dictionary, so we have to free the container here
675 + freez(rc);
676 + rc = t;
677 +
678 + // since we loaded this config from configuration, we need to check if we can link it to alarms
679 + RRDSET *st;
680 + rrdset_foreach_read(st, host) {
681 + if (unlikely(rrdcalc_check_and_link_rrdset_callback(st, rc) == -1))
682 + break;
683 + }
684 + rrdset_foreach_done(st);
685 + }
686 + else {
687 + error(
688 + "RRDCALC: from config '%s' on chart '%s' failed to be added to host '%s'. It already exists.",
689 + string2str(rc->name),
690 + string2str(rc->chart),
691 + rrdhost_hostname(host));
692
652 - // unlink it from RRDSET
653 - if(rc->rrdset) rrdsetcalc_unlink(rc);
693 + ret = 0;
694
655 - // unlink it from RRDHOST
656 - DOUBLE_LINKED_LIST_REMOVE_UNSAFE(host->host_alarms, rc, prev, next);
695 + // free all of it, internals and the container
696 + rrdcalc_free_unused_rrdcalc_loaded_from_config(rc);
697 + }
698
658 - rrdcalc_free(rc);
699 + return ret;
700 }
701
661 -void rrdcalc_foreach_unlink_and_free(RRDHOST *host, RRDCALC *rc) {
662 - DOUBLE_LINKED_LIST_REMOVE_UNSAFE(host->alarms_with_foreach, rc, prev, next);
663 - rrdcalc_free(rc);
702 +static void rrdcalc_unlink_and_delete(RRDHOST *host, RRDCALC *rc, bool having_ll_wrlock) {
703 + if(rc->rrdset)
704 + rrdcalc_unlink_from_rrdset(rc, having_ll_wrlock);
705 +
706 + dictionary_del_advanced(host->rrdcalc_root_index, string2str(rc->key), (ssize_t)string_strlen(rc->key) + 1);
707 }
708
666 -static void rrdcalc_labels_unlink_alarm_loop(RRDHOST *host, RRDCALC *alarms) {
667 - for(RRDCALC *rc = alarms ; rc ; ) {
668 - RRDCALC *rc_next = rc->next;
709
670 - if (!rc->host_labels) {
671 - rc = rc_next;
710 +// ----------------------------------------------------------------------------
711 +// RRDCALC cleanup API functions
712 +
713 +void rrdcalc_delete_alerts_not_matching_host_labels_from_this_host(RRDHOST *host) {
714 + RRDCALC *rc;
715 + foreach_rrdcalc_in_rrdhost_reentrant(host, rc) {
716 + if (!rc->host_labels)
717 continue;
673 - }
718
719 if(!rrdlabels_match_simple_pattern_parsed(host->rrdlabels, rc->host_labels_pattern, '=')) {
720 info("Health configuration for alarm '%s' cannot be applied, because the host %s does not have the label(s) '%s'",
@@ -678,21 +722,13 @@ static void rrdcalc_labels_unlink_alarm_loop(RRDHOST *host, RRDCALC *alarms) {
722 rrdhost_hostname(host),
723 rrdcalc_host_labels(rc));
724
681 - if(host->host_alarms == alarms)
682 - rrdcalc_unlink_and_free(host, rc);
683 - else
684 - rrdcalc_foreach_unlink_and_free(host, rc);
725 + rrdcalc_unlink_and_delete(host, rc, false);
726 }
686 - rc = rc_next;
727 }
728 + foreach_rrdcalc_in_rrdhost_done(rc);
729 }
730
690 -void rrdcalc_labels_unlink_alarm_from_host(RRDHOST *host) {
691 - rrdcalc_labels_unlink_alarm_loop(host, host->host_alarms);
692 - rrdcalc_labels_unlink_alarm_loop(host, host->alarms_with_foreach);
693 -}
694 -
695 -void rrdcalc_labels_unlink() {
731 +void rrdcalc_delete_alerts_not_matching_host_labels_from_all_hosts() {
732 rrd_rdlock();
733
734 RRDHOST *host;
@@ -700,14 +736,45 @@ void rrdcalc_labels_unlink() {
736 if (unlikely(!host->health_enabled))
737 continue;
738
703 - if (host->rrdlabels) {
704 - rrdhost_wrlock(host);
739 + if (host->rrdlabels)
740 + rrdcalc_delete_alerts_not_matching_host_labels_from_this_host(host);
741 + }
742
706 - rrdcalc_labels_unlink_alarm_from_host(host);
743 + rrd_unlock();
744 +}
745 +
746 +void rrdcalc_unlink_all_rrdset_alerts(RRDSET *st) {
747 + RRDCALC *rc, *last = NULL;
748 + netdata_rwlock_wrlock(&st->alerts.rwlock);
749 + while((rc = st->alerts.base)) {
750 + if(last == rc) {
751 + error("RRDCALC: malformed list of alerts linked to chart - cannot cleanup - giving up.");
752 + break;
753 + }
754 + last = rc;
755
708 - rrdhost_unlock(host);
756 + if(rc->run_flags & RRDCALC_FLAG_FROM_TEMPLATE) {
757 + // if the alert comes from a template we can just delete it
758 + rrdcalc_unlink_and_delete(st->rrdhost, rc, true);
759 + }
760 + else {
761 + // this is a configuration for a specific chart
762 + // it should stay in the list
763 + rrdcalc_unlink_from_rrdset(rc, true);
764 }
765 +
766 }
767 + netdata_rwlock_unlock(&st->alerts.rwlock);
768 +}
769
712 - rrd_unlock();
770 +void rrdcalc_delete_all(RRDHOST *host) {
771 + dictionary_flush(host->rrdcalc_root_index);
772 +}
773 +
774 +void rrdcalc_free_unused_rrdcalc_loaded_from_config(RRDCALC *rc) {
775 + if(rc->rrdset)
776 + rrdcalc_unlink_from_rrdset(rc, false);
777 +
778 + rrdcalc_free_internals(rc);
779 + freez(rc);
780 }
database/rrdcalc.h
+62 -53
@@ -10,28 +10,35 @@
10 // (defined in their update_every member below)
11 // They increase the overhead of netdata.
12 //
13 -// These calculations are allocated and linked (->next)
14 -// under RRDHOST.
15 -// Then are also linked to RRDSET (of course only when the
16 -// chart is found, via ->rrdset_next and ->rrdset_prev).
17 -// This double-linked list is maintained sorted at all times
18 -// having as RRDSET.calculations the RRDCALC to be processed
19 -// next.
20 -
21 -#define RRDCALC_FLAG_DB_ERROR 0x00000001
22 -#define RRDCALC_FLAG_DB_NAN 0x00000002
23 -/* #define RRDCALC_FLAG_DB_STALE 0x00000004 */
24 -#define RRDCALC_FLAG_CALC_ERROR 0x00000008
25 -#define RRDCALC_FLAG_WARN_ERROR 0x00000010
26 -#define RRDCALC_FLAG_CRIT_ERROR 0x00000020
27 -#define RRDCALC_FLAG_RUNNABLE 0x00000040
28 -#define RRDCALC_FLAG_DISABLED 0x00000080
29 -#define RRDCALC_FLAG_SILENCED 0x00000100
30 -#define RRDCALC_FLAG_RUN_ONCE 0x00000200
31 -#define RRDCALC_FLAG_NO_CLEAR_NOTIFICATION 0x80000000
32 -
13 +// These calculations are stored under RRDHOST.
14 +// Then are also linked to RRDSET (of course only when a
15 +// matching chart is found).
16 +
17 +typedef enum {
18 + RRDCALC_FLAG_DB_ERROR = (1 << 0),
19 + RRDCALC_FLAG_DB_NAN = (1 << 1),
20 + // RRDCALC_FLAG_DB_STALE = (1 << 2),
21 + RRDCALC_FLAG_CALC_ERROR = (1 << 3),
22 + RRDCALC_FLAG_WARN_ERROR = (1 << 4),
23 + RRDCALC_FLAG_CRIT_ERROR = (1 << 5),
24 + RRDCALC_FLAG_RUNNABLE = (1 << 6),
25 + RRDCALC_FLAG_DISABLED = (1 << 7),
26 + RRDCALC_FLAG_SILENCED = (1 << 8),
27 + RRDCALC_FLAG_RUN_ONCE = (1 << 9),
28 + RRDCALC_FLAG_FROM_TEMPLATE = (1 << 10), // the rrdcalc has been created from a template
29 +} RRDCALC_FLAGS;
30 +
31 +typedef enum {
32 + // This list uses several other options from RRDR_OPTIONS for db lookups.
33 + // To add an item here, you need to reserve a bit in RRDR_OPTIONS.
34 + RRDCALC_OPTION_NO_CLEAR_NOTIFICATION = 0x80000000,
35 +} RRDCALC_OPTIONS;
36 +
37 +#define RRDCALC_ALL_OPTIONS_EXCLUDING_THE_RRDR_ONES (RRDCALC_OPTION_NO_CLEAR_NOTIFICATION)
38
39 struct rrdcalc {
40 + STRING *key; // the unique key in the host's rrdcalc_root_index
41 +
42 uint32_t id; // the unique id of this alarm
43 uint32_t next_event_id; // the next event id that will be used for this alarm
44
@@ -68,14 +75,12 @@ struct rrdcalc {
75 // database lookup settings
76
77 STRING *dimensions; // the chart dimensions
71 - STRING *foreachdim; // the group of dimensions that the `foreach` will be applied.
72 - SIMPLE_PATTERN *spdim; // used if and only if there is a simple pattern for the chart.
73 - int foreachcounter; // the number of alarms created with foreachdim, this also works as an id of the
74 - // children
78 + STRING *foreach_dimension; // the group of dimensions that the `foreach` will be applied.
79 + SIMPLE_PATTERN *foreach_dimension_pattern; // used if and only if there is a simple pattern for the chart.
80 RRDR_GROUPING group; // grouping method: average, max, etc.
81 int before; // ending point in time-series
82 int after; // starting point in time-series
78 - uint32_t options; // calculation options
83 + RRDCALC_OPTIONS options; // configuration options
84
85 // ------------------------------------------------------------------------
86 // expressions related to the alarm
@@ -113,7 +118,7 @@ struct rrdcalc {
118 NETDATA_DOUBLE value; // the current value of the alarm
119 NETDATA_DOUBLE old_value; // the previous value of the alarm
120
116 - uint32_t rrdcalc_flags; // check RRDCALC_FLAG_*
121 + RRDCALC_FLAGS run_flags; // check RRDCALC_FLAG_*
122
123 time_t last_updated; // the last update timestamp of the alarm
124 time_t next_update; // the next update timestamp of the alarm
@@ -132,20 +137,17 @@ struct rrdcalc {
137 // ------------------------------------------------------------------------
138 // variables this alarm exposes to the rest of the alarms
139
135 - RRDVAR *local;
136 - RRDVAR *family;
137 - RRDVAR *hostid;
138 - RRDVAR *hostname;
140 + const RRDVAR_ACQUIRED *rrdvar_local;
141 + const RRDVAR_ACQUIRED *rrdvar_family;
142 + const RRDVAR_ACQUIRED *rrdvar_host_chart_id;
143 + const RRDVAR_ACQUIRED *rrdvar_host_chart_name;
144
145 // ------------------------------------------------------------------------
146 // the chart this alarm it is linked to
147
148 + size_t labels_version;
149 struct rrdset *rrdset;
150
145 - // linking of this alarm on its chart
146 - struct rrdcalc *rrdset_next;
147 - struct rrdcalc *rrdset_prev;
148 -
151 struct rrdcalc *next;
152 struct rrdcalc *prev;
153 };
@@ -164,14 +166,17 @@ struct rrdcalc {
166 #define rrdcalc_original_info(rc) string2str((rc)->original_info)
167 #define rrdcalc_info(rc) string2str((rc)->info)
168 #define rrdcalc_dimensions(rc) string2str((rc)->dimensions)
167 -#define rrdcalc_foreachdim(rc) string2str((rc)->foreachdim)
169 +#define rrdcalc_foreachdim(rc) string2str((rc)->foreach_dimension)
170 #define rrdcalc_host_labels(rc) string2str((rc)->host_labels)
171
170 -#define foreach_rrdcalc_in_rrdset(st, rc) \
171 - DOUBLE_LINKED_LIST_FOREACH_FORWARD((st)->alarms, rc, rrdset_prev, rrdset_next)
172 +#define foreach_rrdcalc_in_rrdhost_read(host, rc) \
173 + dfe_start_read((host)->rrdcalc_root_index, rc) \
174 +
175 +#define foreach_rrdcalc_in_rrdhost_reentrant(host, rc) \
176 + dfe_start_reentrant((host)->rrdcalc_root_index, rc)
177
173 -#define foreach_rrdcalc_in_rrdhost(host, rc) \
174 - DOUBLE_LINKED_LIST_FOREACH_FORWARD((host)->host_alarms, rc, prev, next)
178 +#define foreach_rrdcalc_in_rrdhost_done(rc) \
179 + dfe_done(rc)
180
181 struct alert_config {
182 STRING *alarm;
@@ -213,26 +218,24 @@ struct alert_config {
218
219 #define RRDCALC_HAS_DB_LOOKUP(rc) ((rc)->after)
220
216 -extern void rrdsetcalc_link_matching(RRDSET *st);
217 -extern void rrdsetcalc_unlink(RRDCALC *rc);
218 -extern RRDCALC *rrdcalc_find(RRDSET *st, const char *name);
221 +extern void rrdcalc_update_info_using_rrdset_labels(RRDCALC *rc);
222 +
223 +extern void rrdcalc_link_matching_alerts_to_rrdset(RRDSET *st);
224 +
225 +extern const RRDCALC_ACQUIRED *rrdcalc_from_rrdset_get(RRDSET *st, const char *alert_name);
226 +extern void rrdcalc_from_rrdset_release(RRDSET *st, const RRDCALC_ACQUIRED *rca);
227 +extern RRDCALC *rrdcalc_acquired_to_rrdcalc(const RRDCALC_ACQUIRED *rca);
228
229 extern const char *rrdcalc_status2string(RRDCALC_STATUS status);
230
222 -extern void rrdcalc_free(RRDCALC *rc);
223 -extern void rrdcalc_unlink_and_free(RRDHOST *host, RRDCALC *rc);
231 +extern void rrdcalc_free_unused_rrdcalc_loaded_from_config(RRDCALC *rc);
232
225 -extern int rrdcalc_exists(RRDHOST *host, const char *chart, const char *name);
233 extern uint32_t rrdcalc_get_unique_id(RRDHOST *host, STRING *chart, STRING *name, uint32_t *next_event_id);
227 -extern RRDCALC *rrdcalc_create_from_template(RRDHOST *host, RRDCALCTEMPLATE *rt, const char *chart);
228 -extern RRDCALC *rrdcalc_create_from_rrdcalc(RRDCALC *rc, RRDHOST *host, const char *name, const char *dimension);
229 -extern void rrdcalc_add_to_host(RRDHOST *host, RRDCALC *rc);
230 -extern void dimension_remove_pipe_comma(char *str);
231 -extern char *alarm_name_with_dim(const char *name, size_t namelen, const char *dim, size_t dimlen);
232 -extern void rrdcalc_update_rrdlabels(RRDSET *st);
234 +extern void rrdcalc_add_from_rrdcalctemplate(RRDHOST *host, RRDCALCTEMPLATE *rt, RRDSET *st, const char *overwrite_alert_name, const char *overwrite_dimensions);
235 +extern int rrdcalc_add_from_config(RRDHOST *host, RRDCALC *rc);
236
234 -extern void rrdcalc_labels_unlink();
235 -extern void rrdcalc_labels_unlink_alarm_from_host(RRDHOST *host);
237 +extern void rrdcalc_delete_alerts_not_matching_host_labels_from_all_hosts();
238 +extern void rrdcalc_delete_alerts_not_matching_host_labels_from_this_host(RRDHOST *host);
239
240 static inline int rrdcalc_isrepeating(RRDCALC *rc) {
241 if (unlikely(rc->warn_repeat_every > 0 || rc->crit_repeat_every > 0)) {
@@ -241,6 +244,12 @@ static inline int rrdcalc_isrepeating(RRDCALC *rc) {
244 return 0;
245 }
246
247 +extern void rrdcalc_unlink_all_rrdset_alerts(RRDSET *st);
248 +extern void rrdcalc_delete_all(RRDHOST *host);
249 +
250 +extern void rrdcalc_rrdhost_index_init(RRDHOST *host);
251 +extern void rrdcalc_rrdhost_index_destroy(RRDHOST *host);
252 +
253 #define RRDCALC_VAR_MAX 100
254 #define RRDCALC_VAR_FAMILY "$family"
255 #define RRDCALC_VAR_LABEL "$label:"
database/rrdcalctemplate.c
+174 -28
@@ -1,6 +1,5 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 -#define NETDATA_HEALTH_INTERNALS
3 #include "rrd.h"
4
5 // ----------------------------------------------------------------------------
@@ -11,45 +10,84 @@
10 * @param rt is the template used to create the chart.
11 * @param st is the chart where the alarm will be attached.
12 */
14 -void rrdcalctemplate_check_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host) {
13 +
14 +static char *rrdcalc_alert_name_with_dimension(const char *name, size_t namelen, const char *dim, size_t dimlen) {
15 + char *newname,*move;
16 +
17 + newname = mallocz(namelen + dimlen + 2);
18 + move = newname;
19 + memcpy(move, name, namelen);
20 + move += namelen;
21 +
22 + *move++ = '_';
23 + memcpy(move, dim, dimlen);
24 + move += dimlen;
25 + *move = '\0';
26 +
27 + return newname;
28 +}
29 +
30 +bool rrdcalctemplate_check_rrdset_conditions(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host) {
31 if(rt->context != st->context)
16 - return;
32 + return false;
33
18 - if (rt->charts_pattern && !simple_pattern_matches(rt->charts_pattern, rrdset_name(st)))
19 - return;
34 + if(rt->foreach_dimension_pattern && !rrdset_number_of_dimensions(st))
35 + return false;
36 +
37 + if (rt->charts_pattern && !simple_pattern_matches(rt->charts_pattern, rrdset_name(st)) && !simple_pattern_matches(rt->charts_pattern, rrdset_id(st)))
38 + return false;
39
40 if (rt->family_pattern && !simple_pattern_matches(rt->family_pattern, rrdset_family(st)))
22 - return;
41 + return false;
42
43 if (rt->module_pattern && !simple_pattern_matches(rt->module_pattern, rrdset_module_name(st)))
25 - return;
44 + return false;
45
46 if (rt->plugin_pattern && !simple_pattern_matches(rt->plugin_pattern, rrdset_plugin_name(st)))
28 - return;
47 + return false;
48
49 if(host->rrdlabels && rt->host_labels_pattern && !rrdlabels_match_simple_pattern_parsed(host->rrdlabels, rt->host_labels_pattern, '='))
50 + return false;
51 +
52 + return true;
53 +}
54 +
55 +void rrdcalctemplate_check_rrddim_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDDIM *rd, RRDHOST *host) {
56 + if (simple_pattern_matches(rt->foreach_dimension_pattern, rrddim_id(rd)) || simple_pattern_matches(rt->foreach_dimension_pattern, rrddim_name(rd))) {
57 + char *overwrite_alert_name = rrdcalc_alert_name_with_dimension(
58 + rrdcalctemplate_name(rt), string_strlen(rt->name), rrddim_name(rd), string_strlen(rd->name));
59 + rrdcalc_add_from_rrdcalctemplate(host, rt, st, overwrite_alert_name, rrddim_name(rd));
60 + freez(overwrite_alert_name);
61 + }
62 +}
63 +
64 +void rrdcalctemplate_check_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host) {
65 + if(!rrdcalctemplate_check_rrdset_conditions(rt, st, host))
66 + return;
67 +
68 + if(!rt->foreach_dimension_pattern) {
69 + rrdcalc_add_from_rrdcalctemplate(host, rt, st, NULL, NULL);
70 return;
71 + }
72
33 - RRDCALC *rc = rrdcalc_create_from_template(host, rt, rrdset_id(st));
34 - if (unlikely(!rc))
35 - info("Health tried to create alarm from template '%s' on chart '%s' of host '%s', but it failed", rrdcalctemplate_name(rt), rrdset_id(st), rrdhost_hostname(host));
36 -#ifdef NETDATA_INTERNAL_CHECKS
37 - else if (rc->rrdset != st && !rc->foreachdim) //When we have a template with foreachdim, the child will be added to the index late
38 - error("Health alarm '%s.%s' should be linked to chart '%s', but it is not", rrdcalc_chart_name(rc), rrdcalc_name(rc), rrdset_id(st));
39 -#endif
73 + RRDDIM *rd;
74 + rrddim_foreach_read(rd, st) {
75 + rrdcalctemplate_check_rrddim_conditions_and_link(rt, st, rd, host);
76 + }
77 + rrddim_foreach_done(rd);
78 }
79
42 -void rrdcalctemplate_link_matching(RRDSET *st) {
80 +void rrdcalctemplate_link_matching_templates_to_rrdset(RRDSET *st) {
81 RRDHOST *host = st->rrdhost;
44 - RRDCALCTEMPLATE *rt;
82
46 - foreach_rrdcalctemplate_in_rrdhost(host, rt)
83 + RRDCALCTEMPLATE *rt;
84 + foreach_rrdcalctemplate_read(host, rt) {
85 rrdcalctemplate_check_conditions_and_link(rt, st, host);
86 + }
87 + foreach_rrdcalctemplate_done(rt);
88 }
89
50 -inline void rrdcalctemplate_free(RRDCALCTEMPLATE *rt) {
51 - if(unlikely(!rt)) return;
52 -
90 +static void rrdcalctemplate_free_internals(RRDCALCTEMPLATE *rt) {
91 expression_free(rt->calculation);
92 expression_free(rt->warning);
93 expression_free(rt->critical);
@@ -77,19 +115,127 @@ inline void rrdcalctemplate_free(RRDCALCTEMPLATE *rt) {
115 string_freez(rt->units);
116 string_freez(rt->info);
117 string_freez(rt->dimensions);
80 - string_freez(rt->foreachdim);
118 + string_freez(rt->foreach_dimension);
119 string_freez(rt->host_labels);
82 - simple_pattern_free(rt->spdim);
120 + simple_pattern_free(rt->foreach_dimension_pattern);
121 simple_pattern_free(rt->host_labels_pattern);
84 - freez(rt);
122 }
123
87 -inline void rrdcalctemplate_unlink_and_free(RRDHOST *host, RRDCALCTEMPLATE *rt) {
124 +void rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(RRDCALCTEMPLATE *rt) {
125 if(unlikely(!rt)) return;
126
90 - debug(D_HEALTH, "Health removing template '%s' of host '%s'", rrdcalctemplate_name(rt), rrdhost_hostname(host));
127 + rrdcalctemplate_free_internals(rt);
128 + freez(rt);
129 +}
130 +static void rrdcalctemplate_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdcalctemplate, void *added_bool) {
131 + RRDCALCTEMPLATE *rt = rrdcalctemplate; (void)rt;
132 +
133 + bool *added = added_bool;
134 + *added = true;
135 +
136 + debug(D_HEALTH, "Health configuration adding template '%s'"
137 + ": context '%s'"
138 + ", exec '%s'"
139 + ", recipient '%s'"
140 + ", green " NETDATA_DOUBLE_FORMAT_AUTO
141 + ", red " NETDATA_DOUBLE_FORMAT_AUTO
142 + ", lookup: group %d"
143 + ", after %d"
144 + ", before %d"
145 + ", options %u"
146 + ", dimensions '%s'"
147 + ", for each dimension '%s'"
148 + ", update every %d"
149 + ", calculation '%s'"
150 + ", warning '%s'"
151 + ", critical '%s'"
152 + ", source '%s'"
153 + ", delay up %d"
154 + ", delay down %d"
155 + ", delay max %d"
156 + ", delay_multiplier %f"
157 + ", warn_repeat_every %u"
158 + ", crit_repeat_every %u",
159 + rrdcalctemplate_name(rt),
160 + (rt->context)?string2str(rt->context):"NONE",
161 + (rt->exec)?rrdcalctemplate_exec(rt):"DEFAULT",
162 + (rt->recipient)?rrdcalctemplate_recipient(rt):"DEFAULT",
163 + rt->green,
164 + rt->red,
165 + (int)rt->group,
166 + rt->after,
167 + rt->before,
168 + rt->options,
169 + (rt->dimensions)?rrdcalctemplate_dimensions(rt):"NONE",
170 + (rt->foreach_dimension)?rrdcalctemplate_foreachdim(rt):"NONE",
171 + rt->update_every,
172 + (rt->calculation)?rt->calculation->parsed_as:"NONE",
173 + (rt->warning)?rt->warning->parsed_as:"NONE",
174 + (rt->critical)?rt->critical->parsed_as:"NONE",
175 + rrdcalctemplate_source(rt),
176 + rt->delay_up_duration,
177 + rt->delay_down_duration,
178 + rt->delay_max_duration,
179 + rt->delay_multiplier,
180 + rt->warn_repeat_every,
181 + rt->crit_repeat_every
182 + );
183 +}
184 +
185 +static void rrdcalctemplate_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdcalctemplate, void *rrdhost __maybe_unused) {
186 + RRDCALCTEMPLATE *rt = rrdcalctemplate;
187 + rrdcalctemplate_free_internals(rt);
188 +}
189 +
190 +void rrdcalctemplate_index_init(RRDHOST *host) {
191 + if(!host->rrdcalctemplate_root_index) {
192 + host->rrdcalctemplate_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
193 +
194 + dictionary_register_insert_callback(host->rrdcalctemplate_root_index, rrdcalctemplate_insert_callback, NULL);
195 + dictionary_register_delete_callback(host->rrdcalctemplate_root_index, rrdcalctemplate_delete_callback, host);
196 + }
197 +}
198 +
199 +void rrdcalctemplate_index_destroy(RRDHOST *host) {
200 + dictionary_destroy(host->rrdcalctemplate_root_index);
201 + host->rrdcalctemplate_root_index = NULL;
202 +}
203 +
204 +inline void rrdcalctemplate_delete_all(RRDHOST *host) {
205 + dictionary_flush(host->rrdcalctemplate_root_index);
206 +}
207 +
208 +#define RRDCALCTEMPLATE_MAX_KEY_SIZE 1024
209 +static size_t rrdcalctemplate_key(char *dst, size_t dst_len, const char *name, const char *family_match) {
210 + return snprintfz(dst, dst_len, "%s/%s", name, (family_match && *family_match)?family_match:"*");
211 +}
212 +
213 +void rrdcalctemplate_add_from_config(RRDHOST *host, RRDCALCTEMPLATE *rt) {
214 + if(unlikely(!rt->context)) {
215 + error("Health configuration for template '%s' does not have a context", rrdcalctemplate_name(rt));
216 + return;
217 + }
218 +
219 + if(unlikely(!rt->update_every)) {
220 + error("Health configuration for template '%s' has no frequency (parameter 'every'). Ignoring it.", rrdcalctemplate_name(rt));
221 + return;
222 + }
223 +
224 + if(unlikely(!RRDCALCTEMPLATE_HAS_DB_LOOKUP(rt) && !rt->calculation && !rt->warning && !rt->critical)) {
225 + error("Health configuration for template '%s' is useless (no calculation, no warning and no critical evaluation)", rrdcalctemplate_name(rt));
226 + return;
227 + }
228 +
229 + char key[RRDCALCTEMPLATE_MAX_KEY_SIZE + 1];
230 + size_t key_len = rrdcalctemplate_key(key, RRDCALCTEMPLATE_MAX_KEY_SIZE, rrdcalctemplate_name(rt), rrdcalctemplate_family_match(rt));
231
92 - DOUBLE_LINKED_LIST_REMOVE_UNSAFE(host->alarms_templates, rt, prev, next);
232 + bool added = false;
233 + dictionary_set_advanced(host->rrdcalctemplate_root_index, key, (ssize_t)(key_len + 1), rt, sizeof(*rt), &added);
234
94 - rrdcalctemplate_free(rt);
235 + if(added)
236 + freez(rt);
237 + else {
238 + info("Health configuration template '%s' already exists for host '%s'.", rrdcalctemplate_name(rt), rrdhost_hostname(host));
239 + rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(rt);
240 + }
241 }
database/rrdcalctemplate.h
+23 -12
@@ -48,14 +48,12 @@ struct rrdcalctemplate {
48 // database lookup settings
49
50 STRING *dimensions; // the chart dimensions
51 - STRING *foreachdim; // the group of dimensions that the lookup will be applied.
52 - SIMPLE_PATTERN *spdim; // used if and only if there is a simple pattern for the chart.
53 - int foreachcounter; // the number of alarms created with foreachdim, this also works as an id of the
54 - // children
51 + STRING *foreach_dimension; // the group of dimensions that the lookup will be applied.
52 + SIMPLE_PATTERN *foreach_dimension_pattern; // used if and only if there is a simple pattern for the chart.
53 RRDR_GROUPING group; // grouping method: average, max, etc.
54 int before; // ending point in time-series
55 int after; // starting point in time-series
58 - uint32_t options; // calculation options
56 + RRDCALC_OPTIONS options; // configuration options
57
58 // ------------------------------------------------------------------------
59 // notification delay settings
@@ -87,8 +85,11 @@ struct rrdcalctemplate {
85 struct rrdcalctemplate *prev;
86 };
87
90 -#define foreach_rrdcalctemplate_in_rrdhost(host, rt) \
91 - DOUBLE_LINKED_LIST_FOREACH_FORWARD((host)->alarms_templates, rt, prev, next)
88 +#define foreach_rrdcalctemplate_read(host, rt) \
89 + dfe_start_read((host)->rrdcalctemplate_root_index, rt)
90 +
91 +#define foreach_rrdcalctemplate_done(rt) \
92 + dfe_done(rt)
93
94 #define rrdcalctemplate_name(rt) string2str((rt)->name)
95 #define rrdcalctemplate_exec(rt) string2str((rt)->exec)
@@ -104,14 +105,24 @@ struct rrdcalctemplate {
105 #define rrdcalctemplate_info(rt) string2str((rt)->info)
106 #define rrdcalctemplate_source(rt) string2str((rt)->source)
107 #define rrdcalctemplate_dimensions(rt) string2str((rt)->dimensions)
107 -#define rrdcalctemplate_foreachdim(rt) string2str((rt)->foreachdim)
108 +#define rrdcalctemplate_foreachdim(rt) string2str((rt)->foreach_dimension)
109 #define rrdcalctemplate_host_labels(rt) string2str((rt)->host_labels)
110
111 #define RRDCALCTEMPLATE_HAS_DB_LOOKUP(rt) ((rt)->after)
112
112 -extern void rrdcalctemplate_link_matching(RRDSET *st);
113 +extern void rrdcalctemplate_link_matching_templates_to_rrdset(RRDSET *st);
114 +
115 +extern void rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(RRDCALCTEMPLATE *rt);
116 +extern void rrdcalctemplate_delete_all(RRDHOST *host);
117 +extern void rrdcalctemplate_add_from_config(RRDHOST *host, RRDCALCTEMPLATE *rt);
118 +
119 +extern void rrdcalctemplate_check_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host);
120 +
121 +extern bool rrdcalctemplate_check_rrdset_conditions(RRDCALCTEMPLATE *rt, RRDSET *st, RRDHOST *host);
122 +extern void rrdcalctemplate_check_rrddim_conditions_and_link(RRDCALCTEMPLATE *rt, RRDSET *st, RRDDIM *rd, RRDHOST *host);
123 +
124 +
125 +extern void rrdcalctemplate_index_init(RRDHOST *host);
126 +extern void rrdcalctemplate_index_destroy(RRDHOST *host);
127
114 -extern void rrdcalctemplate_free(RRDCALCTEMPLATE *rt);
115 -extern void rrdcalctemplate_unlink_and_free(RRDHOST *host, RRDCALCTEMPLATE *rt);
116 -extern void rrdcalctemplate_create_alarms(RRDHOST *host, RRDCALCTEMPLATE *rt, RRDSET *st);
128 #endif //NETDATA_RRDCALCTEMPLATE_H
database/rrdcontext.c
+108 -80
@@ -8,6 +8,8 @@
8
9 int rrdcontext_enabled = CONFIG_BOOLEAN_YES;
10
11 +// #define LOG_POST_PROCESSING_QUEUE_INSERTIONS 1
12 +
13 #define MESSAGES_PER_BUNDLE_TO_SEND_TO_HUB_PER_HOST 5000
14 #define FULL_RETENTION_SCAN_DELAY_AFTER_DB_ROTATION_SECS 120
15 #define RRDCONTEXT_WORKER_THREAD_HEARTBEAT_USEC (1000 * USEC_PER_MS)
@@ -432,11 +434,11 @@ static void rrdmetric_free(RRDMETRIC *rm) {
434
435 // called when this rrdmetric is inserted to the rrdmetrics dictionary of a rrdinstance
436 // the constructor of the rrdmetric object
435 -static void rrdmetric_insert_callback(const char *id __maybe_unused, void *value, void *data) {
437 +static void rrdmetric_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdinstance) {
438 RRDMETRIC *rm = value;
439
440 // link it to its parent
439 - rm->ri = data;
441 + rm->ri = rrdinstance;
442
443 // remove flags that we need to figure out at runtime
444 rm->flags = rm->flags & RRD_FLAGS_ALLOWED_EXTERNALLY_ON_NEW_OBJECTS; // no need for atomics
@@ -447,7 +449,7 @@ static void rrdmetric_insert_callback(const char *id __maybe_unused, void *value
449
450 // called when this rrdmetric is deleted from the rrdmetrics dictionary of a rrdinstance
451 // the destructor of the rrdmetric object
450 -static void rrdmetric_delete_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
452 +static void rrdmetric_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdinstance __maybe_unused) {
453 RRDMETRIC *rm = value;
454
455 internal_error(rm->rrddim, "RRDMETRIC: '%s' is freed but there is a RRDDIM linked to it.", string2str(rm->id));
@@ -458,7 +460,7 @@ static void rrdmetric_delete_callback(const char *id __maybe_unused, void *value
460
461 // called when the same rrdmetric is inserted again to the rrdmetrics dictionary of a rrdinstance
462 // while this is called, the dictionary is write locked, but there may be other users of the object
461 -static void rrdmetric_conflict_callback(const char *id __maybe_unused, void *oldv, void *newv, void *data __maybe_unused) {
463 +static bool rrdmetric_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *oldv, void *newv, void *rrdinstance __maybe_unused) {
464 RRDMETRIC *rm = oldv;
465 RRDMETRIC *rm_new = newv;
466
@@ -518,11 +520,12 @@ static void rrdmetric_conflict_callback(const char *id __maybe_unused, void *old
520 rrdmetric_free(rm_new);
521
522 // the react callback will continue from here
523 + return rrd_flag_is_updated(rm);
524 }
525
526 // this is called after the insert or the conflict callbacks,
527 // but the dictionary is now unlocked
525 -static void rrdmetric_react_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
528 +static void rrdmetric_react_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdinstance __maybe_unused) {
529 RRDMETRIC *rm = value;
530 rrdmetric_trigger_updates(rm, __FUNCTION__ );
531 }
@@ -531,11 +534,11 @@ static void rrdmetrics_create_in_rrdinstance(RRDINSTANCE *ri) {
534 if(unlikely(!ri)) return;
535 if(likely(ri->rrdmetrics)) return;
536
534 - ri->rrdmetrics = dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
535 - dictionary_register_insert_callback(ri->rrdmetrics, rrdmetric_insert_callback, (void *)ri);
536 - dictionary_register_delete_callback(ri->rrdmetrics, rrdmetric_delete_callback, (void *)ri);
537 - dictionary_register_conflict_callback(ri->rrdmetrics, rrdmetric_conflict_callback, (void *)ri);
538 - dictionary_register_react_callback(ri->rrdmetrics, rrdmetric_react_callback, (void *)ri);
537 + ri->rrdmetrics = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
538 + dictionary_register_insert_callback(ri->rrdmetrics, rrdmetric_insert_callback, ri);
539 + dictionary_register_delete_callback(ri->rrdmetrics, rrdmetric_delete_callback, ri);
540 + dictionary_register_conflict_callback(ri->rrdmetrics, rrdmetric_conflict_callback, ri);
541 + dictionary_register_react_callback(ri->rrdmetrics, rrdmetric_react_callback, ri);
542 }
543
544 static void rrdmetrics_destroy_from_rrdinstance(RRDINSTANCE *ri) {
@@ -669,7 +672,7 @@ static void rrdinstance_free(RRDINSTANCE *ri) {
672 ri->rrdset = NULL;
673 }
674
672 -static void rrdinstance_insert_callback(const char *id __maybe_unused, void *value, void *data) {
675 +static void rrdinstance_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdcontext) {
676 static STRING *ml_anomaly_rates_id = NULL;
677
678 if(unlikely(!ml_anomaly_rates_id))
@@ -678,7 +681,7 @@ static void rrdinstance_insert_callback(const char *id __maybe_unused, void *val
681 RRDINSTANCE *ri = value;
682
683 // link it to its parent
681 - ri->rc = data;
684 + ri->rc = rrdcontext;
685
686 ri->flags = ri->flags & RRD_FLAGS_ALLOWED_EXTERNALLY_ON_NEW_OBJECTS; // no need for atomics
687
@@ -711,9 +714,7 @@ static void rrdinstance_insert_callback(const char *id __maybe_unused, void *val
714 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_NEW_OBJECT);
715 }
716
714 -static void rrdinstance_delete_callback(const char *id, void *value, void *data) {
715 - (void)id;
716 - RRDCONTEXT *rc = data; (void)rc;
717 +static void rrdinstance_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdcontext __maybe_unused) {
718 RRDINSTANCE *ri = (RRDINSTANCE *)value;
719
720 internal_error(ri->rrdset, "RRDINSTANCE: '%s' is freed but there is a RRDSET linked to it.", string2str(ri->id));
@@ -721,7 +722,7 @@ static void rrdinstance_delete_callback(const char *id, void *value, void *data)
722 rrdinstance_free(ri);
723 }
724
724 -static void rrdinstance_conflict_callback(const char *id __maybe_unused, void *oldv, void *newv, void *data __maybe_unused) {
725 +static bool rrdinstance_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *oldv, void *newv, void *rrdcontext __maybe_unused) {
726 RRDINSTANCE *ri = (RRDINSTANCE *)oldv;
727 RRDINSTANCE *ri_new = (RRDINSTANCE *)newv;
728
@@ -739,10 +740,10 @@ static void rrdinstance_conflict_callback(const char *id __maybe_unused, void *o
740 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_LINKING);
741 }
742
742 - if(ri->rrdset && ri->rrdset->chart_uuid && uuid_compare(ri->uuid, *ri->rrdset->chart_uuid) != 0) {
743 + if(ri->rrdset && uuid_compare(ri->uuid, ri->rrdset->chart_uuid) != 0) {
744 char uuid1[UUID_STR_LEN], uuid2[UUID_STR_LEN];
745 uuid_unparse(ri->uuid, uuid1);
745 - uuid_unparse(*ri->rrdset->chart_uuid, uuid2);
746 + uuid_unparse(ri->rrdset->chart_uuid, uuid2);
747 internal_error(true, "RRDINSTANCE: '%s' is linked to RRDSET '%s' but they have different UUIDs. RRDINSTANCE has '%s', RRDSET has '%s'", string2str(ri->id), rrdset_id(ri->rrdset), uuid1, uuid2);
748 }
749
@@ -823,9 +824,10 @@ static void rrdinstance_conflict_callback(const char *id __maybe_unused, void *o
824 rrdinstance_free(ri_new);
825
826 // the react callback will continue from here
827 + return rrd_flag_is_updated(ri);
828 }
829
828 -static void rrdinstance_react_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
830 +static void rrdinstance_react_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdcontext __maybe_unused) {
831 RRDINSTANCE *ri = value;
832
833 rrdinstance_trigger_updates(ri, __FUNCTION__ );
@@ -837,11 +839,11 @@ void rrdinstances_create_in_rrdcontext(RRDCONTEXT *rc) {
839
840 if(unlikely(!rc || rc->rrdinstances)) return;
841
840 - rc->rrdinstances = dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
841 - dictionary_register_insert_callback(rc->rrdinstances, rrdinstance_insert_callback, (void *)rc);
842 - dictionary_register_delete_callback(rc->rrdinstances, rrdinstance_delete_callback, (void *)rc);
843 - dictionary_register_conflict_callback(rc->rrdinstances, rrdinstance_conflict_callback, (void *)rc);
844 - dictionary_register_react_callback(rc->rrdinstances, rrdinstance_react_callback, (void *)rc);
842 + rc->rrdinstances = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
843 + dictionary_register_insert_callback(rc->rrdinstances, rrdinstance_insert_callback, rc);
844 + dictionary_register_delete_callback(rc->rrdinstances, rrdinstance_delete_callback, rc);
845 + dictionary_register_conflict_callback(rc->rrdinstances, rrdinstance_conflict_callback, rc);
846 + dictionary_register_react_callback(rc->rrdinstances, rrdinstance_react_callback, rc);
847 }
848
849 void rrdinstances_destroy_from_rrdcontext(RRDCONTEXT *rc) {
@@ -907,7 +909,7 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
909 .flags = RRD_FLAG_NONE, // no need for atomics
910 .rrdset = st,
911 };
910 - uuid_copy(tri.uuid, *st->chart_uuid);
912 + uuid_copy(tri.uuid, st->chart_uuid);
913
914 RRDINSTANCE_ACQUIRED *ria = (RRDINSTANCE_ACQUIRED *)dictionary_set_and_acquire_item(rc->rrdinstances, string2str(tri.id), &tri, sizeof(tri));
915
@@ -934,7 +936,6 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
936 RRDINSTANCE *ri_old = rrdinstance_acquired_value(ria_old);
937
938 // migrate all dimensions to the new metrics
937 - rrdset_rdlock(st);
939 RRDDIM *rd;
940 rrddim_foreach_read(rd, st) {
941 if (!rd->rrdmetric) continue;
@@ -950,7 +951,7 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
951
952 rrdmetric_from_rrddim(rd);
953 }
953 - rrdset_unlock(st);
954 + rrddim_foreach_done(rd);
955
956 // mark the old instance, ready to be deleted
957 if(!rrd_flag_check(ri_old, RRD_FLAG_OWN_LABELS))
@@ -966,7 +967,7 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
967
968 /*
969 // trigger updates on the old context
969 - if(!dictionary_stats_entries(rc_old->rrdinstances) && !dictionary_stats_referenced_items(rc_old->rrdinstances)) {
970 + if(!dictionary_entries(rc_old->rrdinstances) && !dictionary_stats_referenced_items(rc_old->rrdinstances)) {
971 rrdcontext_lock(rc_old);
972 rc_old->flags = ((rc_old->flags & RRD_FLAG_QUEUED)?RRD_FLAG_QUEUED:RRD_FLAG_NONE)|RRD_FLAG_DELETED|RRD_FLAG_UPDATED|RRD_FLAG_LIVE_RETENTION|RRD_FLAG_UPDATE_REASON_UNUSED|RRD_FLAG_UPDATE_REASON_ZERO_RETENTION;
973 rc_old->first_time_t = 0;
@@ -1102,9 +1103,8 @@ static void rrdcontext_freez(RRDCONTEXT *rc) {
1103 string_freez(rc->family);
1104 }
1105
1105 -static void rrdcontext_insert_callback(const char *id, void *value, void *data) {
1106 - (void)id;
1107 - RRDHOST *host = (RRDHOST *)data;
1106 +static void rrdcontext_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdhost) {
1107 + RRDHOST *host = (RRDHOST *)rrdhost;
1108 RRDCONTEXT *rc = (RRDCONTEXT *)value;
1109
1110 rc->rrdhost = host;
@@ -1167,10 +1167,7 @@ static void rrdcontext_insert_callback(const char *id, void *value, void *data)
1167 rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_NEW_OBJECT);
1168 }
1169
1170 -static void rrdcontext_delete_callback(const char *id, void *value, void *data) {
1171 - (void)id;
1172 - RRDHOST *host = (RRDHOST *)data;
1173 - (void)host;
1170 +static void rrdcontext_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdhost __maybe_unused) {
1171
1172 RRDCONTEXT *rc = (RRDCONTEXT *)value;
1173
@@ -1179,18 +1176,14 @@ static void rrdcontext_delete_callback(const char *id, void *value, void *data)
1176 rrdcontext_freez(rc);
1177 }
1178
1182 -static void rrdcontext_conflict_callback(const char *id, void *oldv, void *newv, void *data) {
1183 - (void)id;
1184 - RRDHOST *host = (RRDHOST *)data;
1185 - (void)host;
1186 -
1179 +static bool rrdcontext_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *oldv, void *newv, void *rrdhost __maybe_unused) {
1180 RRDCONTEXT *rc = (RRDCONTEXT *)oldv;
1181 RRDCONTEXT *rc_new = (RRDCONTEXT *)newv;
1182
1183 //current rc is not archived, new_rc is archived, dont merge
1184 if (!rrd_flag_is_archived(rc) && rrd_flag_is_archived(rc_new)) {
1185 rrdcontext_freez(rc_new);
1193 - return;
1186 + return false;
1187 }
1188
1189 rrdcontext_lock(rc);
@@ -1246,11 +1239,11 @@ static void rrdcontext_conflict_callback(const char *id, void *oldv, void *newv,
1239 rrdcontext_freez(rc_new);
1240
1241 // the react callback will continue from here
1242 + return rrd_flag_is_updated(rc);
1243 }
1244
1251 -static void rrdcontext_react_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
1245 +static void rrdcontext_react_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *rrdhost __maybe_unused) {
1246 RRDCONTEXT *rc = (RRDCONTEXT *)value;
1253 -
1247 rrdcontext_trigger_updates(rc, __FUNCTION__ );
1248 }
1249
@@ -1259,33 +1252,35 @@ static void rrdcontext_trigger_updates(RRDCONTEXT *rc, const char *function) {
1252 rrdcontext_queue_for_post_processing(rc, function, rc->flags);
1253 }
1254
1262 -static void rrdcontext_hub_queue_insert_callback(const char *name __maybe_unused, void *context, void *data __maybe_unused) {
1255 +static void rrdcontext_hub_queue_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *context, void *nothing __maybe_unused) {
1256 RRDCONTEXT *rc = context;
1257 rrd_flag_set(rc, RRD_FLAG_QUEUED_FOR_HUB);
1258 rc->queue.queued_ut = now_realtime_usec();
1259 rc->queue.queued_flags = rrd_flags_get(rc);
1260 }
1261
1269 -static void rrdcontext_hub_queue_delete_callback(const char *name __maybe_unused, void *context, void *data __maybe_unused) {
1262 +static void rrdcontext_hub_queue_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *context, void *nothing __maybe_unused) {
1263 RRDCONTEXT *rc = context;
1264 rrd_flag_clear(rc, RRD_FLAG_QUEUED_FOR_HUB);
1265 }
1266
1274 -static void rrdcontext_hub_queue_conflict_callback(const char *name __maybe_unused, void *context, void *new_context __maybe_unused, void *data __maybe_unused) {
1267 +static bool rrdcontext_hub_queue_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *context, void *new_context __maybe_unused, void *nothing __maybe_unused) {
1268 // context and new_context are the same
1269 // we just need to update the timings
1270 RRDCONTEXT *rc = context;
1271 rrd_flag_set(rc, RRD_FLAG_QUEUED_FOR_HUB);
1272 rc->queue.queued_ut = now_realtime_usec();
1273 rc->queue.queued_flags |= rrd_flags_get(rc);
1274 +
1275 + return true;
1276 }
1277
1283 -static void rrdcontext_post_processing_queue_insert_callback(const char *name __maybe_unused, void *context, void *data __maybe_unused) {
1278 +static void rrdcontext_post_processing_queue_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *context, void *nothing __maybe_unused) {
1279 RRDCONTEXT *rc = context;
1280 rrd_flag_set(rc, RRD_FLAG_QUEUED_FOR_POST_PROCESSING);
1281 }
1282
1288 -static void rrdcontext_post_processing_queue_delete_callback(const char *name __maybe_unused, void *context, void *data __maybe_unused) {
1283 +static void rrdcontext_post_processing_queue_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *context, void *nothing __maybe_unused) {
1284 RRDCONTEXT *rc = context;
1285 rrd_flag_clear(rc, RRD_FLAG_QUEUED_FOR_POST_PROCESSING);
1286 }
@@ -1297,23 +1292,19 @@ void rrdhost_create_rrdcontexts(RRDHOST *host) {
1292 if(unlikely(!host)) return;
1293 if(likely(host->rrdctx)) return;
1294
1300 - host->rrdctx = (RRDCONTEXTS *)dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
1301 - dictionary_register_insert_callback((DICTIONARY *)host->rrdctx, rrdcontext_insert_callback, (void *)host);
1302 - dictionary_register_delete_callback((DICTIONARY *)host->rrdctx, rrdcontext_delete_callback, (void *)host);
1303 - dictionary_register_conflict_callback((DICTIONARY *)host->rrdctx, rrdcontext_conflict_callback, (void *)host);
1304 - dictionary_register_react_callback((DICTIONARY *)host->rrdctx, rrdcontext_react_callback, (void *)host);
1295 + host->rrdctx = (RRDCONTEXTS *)dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
1296 + dictionary_register_insert_callback((DICTIONARY *)host->rrdctx, rrdcontext_insert_callback, host);
1297 + dictionary_register_delete_callback((DICTIONARY *)host->rrdctx, rrdcontext_delete_callback, host);
1298 + dictionary_register_conflict_callback((DICTIONARY *)host->rrdctx, rrdcontext_conflict_callback, host);
1299 + dictionary_register_react_callback((DICTIONARY *)host->rrdctx, rrdcontext_react_callback, host);
1300
1306 - host->rrdctx_hub_queue = (RRDCONTEXTS *)dictionary_create(
1307 - DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
1308 - |DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE);
1301 + host->rrdctx_hub_queue = (RRDCONTEXTS *)dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_VALUE_LINK_DONT_CLONE);
1302
1303 dictionary_register_insert_callback((DICTIONARY *)host->rrdctx_hub_queue, rrdcontext_hub_queue_insert_callback, NULL);
1304 dictionary_register_delete_callback((DICTIONARY *)host->rrdctx_hub_queue, rrdcontext_hub_queue_delete_callback, NULL);
1305 dictionary_register_conflict_callback((DICTIONARY *)host->rrdctx_hub_queue, rrdcontext_hub_queue_conflict_callback, NULL);
1306
1314 - host->rrdctx_post_processing_queue = (RRDCONTEXTS *)dictionary_create(
1315 - DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
1316 - |DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE);
1307 + host->rrdctx_post_processing_queue = (RRDCONTEXTS *)dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_VALUE_LINK_DONT_CLONE);
1308
1309 dictionary_register_insert_callback((DICTIONARY *)host->rrdctx_hub_queue, rrdcontext_post_processing_queue_insert_callback, NULL);
1310 dictionary_register_delete_callback((DICTIONARY *)host->rrdctx_hub_queue, rrdcontext_post_processing_queue_delete_callback, NULL);
@@ -1331,7 +1322,7 @@ void rrdhost_destroy_rrdcontexts(RRDHOST *host) {
1322
1323 RRDCONTEXT *rc;
1324 dfe_start_write(old, rc) {
1334 - dictionary_del_having_write_lock(old, string2str(rc->id));
1325 + dictionary_del(old, string2str(rc->id));
1326 }
1327 dfe_done(rc);
1328 dictionary_destroy(old);
@@ -1343,7 +1334,7 @@ void rrdhost_destroy_rrdcontexts(RRDHOST *host) {
1334
1335 RRDCONTEXT *rc;
1336 dfe_start_write(old, rc) {
1346 - dictionary_del_having_write_lock(old, string2str(rc->id));
1337 + dictionary_del(old, string2str(rc->id));
1338 }
1339 dfe_done(rc);
1340 dictionary_destroy(old);
@@ -1464,6 +1455,35 @@ void rrdcontext_db_rotation(void) {
1455 rrdcontext_next_db_rotation_ut = now_realtime_usec() + FULL_RETENTION_SCAN_DELAY_AFTER_DB_ROTATION_SECS * USEC_PER_SEC;
1456 }
1457
1458 +int rrdcontext_foreach_instance_with_rrdset_in_context(RRDHOST *host, const char *context, int (*callback)(RRDSET *st, void *data), void *data) {
1459 + if(unlikely(!host || !context || !*context || !callback))
1460 + return -1;
1461 +
1462 + RRDCONTEXT_ACQUIRED *rca = (RRDCONTEXT_ACQUIRED *)dictionary_get_and_acquire_item((DICTIONARY *)host->rrdctx, context);
1463 + if(unlikely(!rca)) return -1;
1464 +
1465 + RRDCONTEXT *rc = rrdcontext_acquired_value(rca);
1466 + if(unlikely(!rc)) return -1;
1467 +
1468 + int ret = 0;
1469 + RRDINSTANCE *ri;
1470 + dfe_start_read(rc->rrdinstances, ri) {
1471 + if(ri->rrdset) {
1472 + int r = callback(ri->rrdset, data);
1473 + if(r >= 0) ret += r;
1474 + else {
1475 + ret = r;
1476 + break;
1477 + }
1478 + }
1479 + }
1480 + dfe_done(ri);
1481 +
1482 + rrdcontext_release(rca);
1483 +
1484 + return ret;
1485 +}
1486 +
1487 // ----------------------------------------------------------------------------
1488 // ACLK interface
1489
@@ -1601,7 +1621,8 @@ struct rrdcontext_to_json {
1621 RRD_FLAGS combined_flags;
1622 };
1623
1604 -static inline int rrdmetric_to_json_callback(const char *id, void *value, void *data) {
1624 +static inline int rrdmetric_to_json_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
1625 + const char *id = dictionary_acquired_item_name(item);
1626 struct rrdcontext_to_json * t = data;
1627 RRDMETRIC *rm = value;
1628 BUFFER *wb = t->wb;
@@ -1673,7 +1694,9 @@ static inline int rrdmetric_to_json_callback(const char *id, void *value, void *
1694 return 1;
1695 }
1696
1676 -static inline int rrdinstance_to_json_callback(const char *id, void *value, void *data) {
1697 +static inline int rrdinstance_to_json_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
1698 + const char *id = dictionary_acquired_item_name(item);
1699 +
1700 struct rrdcontext_to_json *t_parent = data;
1701 RRDINSTANCE *ri = value;
1702 BUFFER *wb = t_parent->wb;
@@ -1788,7 +1811,7 @@ static inline int rrdinstance_to_json_callback(const char *id, void *value, void
1811 buffer_strcat(wb, "\"");
1812 }
1813
1791 - if(options & RRDCONTEXT_OPTION_SHOW_LABELS && ri->rrdlabels && dictionary_stats_entries(ri->rrdlabels)) {
1814 + if(options & RRDCONTEXT_OPTION_SHOW_LABELS && ri->rrdlabels && dictionary_entries(ri->rrdlabels)) {
1815 buffer_sprintf(wb, ",\n\t\t\t\t\t\"labels\": {\n");
1816 rrdlabels_to_buffer(ri->rrdlabels, wb, "\t\t\t\t\t\t", ":", "\"", ",\n", NULL, NULL, NULL, NULL);
1817 buffer_strcat(wb, "\n\t\t\t\t\t}");
@@ -1807,7 +1830,8 @@ static inline int rrdinstance_to_json_callback(const char *id, void *value, void
1830 return 1;
1831 }
1832
1810 -static inline int rrdcontext_to_json_callback(const char *id, void *value, void *data) {
1833 +static inline int rrdcontext_to_json_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
1834 + const char *id = dictionary_acquired_item_name(item);
1835 struct rrdcontext_to_json *t_parent = data;
1836 RRDCONTEXT *rc = value;
1837 BUFFER *wb = t_parent->wb;
@@ -1974,7 +1998,7 @@ int rrdcontext_to_json(RRDHOST *host, BUFFER *wb, time_t after, time_t before, R
1998 .written = 0,
1999 .now = now_realtime_sec(),
2000 };
1977 - rrdcontext_to_json_callback(context, rc, &t_contexts);
2001 + rrdcontext_to_json_callback((DICTIONARY_ITEM *)rca, rc, &t_contexts);
2002
2003 rrdcontext_release(rca);
2004
@@ -2300,10 +2324,10 @@ static inline bool rrdinstance_should_be_deleted(RRDINSTANCE *ri) {
2324 if(likely(ri->rrdset))
2325 return false;
2326
2303 - if(unlikely(dictionary_stats_referenced_items(ri->rrdmetrics) != 0))
2327 + if(unlikely(dictionary_referenced_items(ri->rrdmetrics) != 0))
2328 return false;
2329
2306 - if(unlikely(dictionary_stats_entries(ri->rrdmetrics) != 0))
2330 + if(unlikely(dictionary_entries(ri->rrdmetrics) != 0))
2331 return false;
2332
2333 if(ri->first_time_t || ri->last_time_t)
@@ -2319,10 +2343,10 @@ static inline bool rrdcontext_should_be_deleted(RRDCONTEXT *rc) {
2343 if(likely(rrd_flag_check(rc, RRD_FLAGS_PREVENTING_DELETIONS)))
2344 return false;
2345
2322 - if(unlikely(dictionary_stats_referenced_items(rc->rrdinstances) != 0))
2346 + if(unlikely(dictionary_referenced_items(rc->rrdinstances) != 0))
2347 return false;
2348
2325 - if(unlikely(dictionary_stats_entries(rc->rrdinstances) != 0))
2349 + if(unlikely(dictionary_entries(rc->rrdinstances) != 0))
2350 return false;
2351
2352 if(unlikely(rc->first_time_t || rc->last_time_t))
@@ -2364,7 +2388,7 @@ static void rrdcontext_garbage_collect_single_host(RRDHOST *host, bool worker_jo
2388 dfe_start_write(ri->rrdmetrics, rm) {
2389 if(rrdmetric_should_be_deleted(rm)) {
2390 if(worker_jobs) worker_is_busy(WORKER_JOB_CLEANUP_DELETE);
2367 - if(dictionary_del_having_write_lock(ri->rrdmetrics, string2str(rm->id)) != 0)
2391 + if(!dictionary_del(ri->rrdmetrics, string2str(rm->id)))
2392 error("RRDCONTEXT: metric '%s' of instance '%s' of context '%s' of host '%s', failed to be deleted from rrdmetrics dictionary.",
2393 string2str(rm->id),
2394 string2str(ri->id),
@@ -2384,7 +2408,7 @@ static void rrdcontext_garbage_collect_single_host(RRDHOST *host, bool worker_jo
2408
2409 if(rrdinstance_should_be_deleted(ri)) {
2410 if(worker_jobs) worker_is_busy(WORKER_JOB_CLEANUP_DELETE);
2387 - if(dictionary_del(rc->rrdinstances, string2str(ri->id)) != 0)
2411 + if(!dictionary_del(rc->rrdinstances, string2str(ri->id)))
2412 error("RRDCONTEXT: instance '%s' of context '%s' of host '%s', failed to be deleted from rrdmetrics dictionary.",
2413 string2str(ri->id),
2414 string2str(rc->id),
@@ -2405,7 +2429,7 @@ static void rrdcontext_garbage_collect_single_host(RRDHOST *host, bool worker_jo
2429 rrdcontext_dequeue_from_post_processing(rc);
2430 rrdcontext_delete_from_sql_unsafe(rc);
2431
2408 - if(dictionary_del((DICTIONARY *)host->rrdctx, string2str(rc->id)) != 0)
2432 + if(!dictionary_del((DICTIONARY *)host->rrdctx, string2str(rc->id)))
2433 error("RRDCONTEXT: context '%s' of host '%s', failed to be deleted from rrdmetrics dictionary.",
2434 string2str(rc->id),
2435 rrdhost_hostname(host));
@@ -2471,7 +2495,7 @@ static void rrdinstance_post_process_updates(RRDINSTANCE *ri, bool force, RRD_FL
2495 time_t min_first_time_t = LONG_MAX, max_last_time_t = 0;
2496 size_t metrics_active = 0, metrics_deleted = 0;
2497 bool live_retention = true, currently_collected = false;
2474 - if(dictionary_stats_entries(ri->rrdmetrics) > 0) {
2498 + if(dictionary_entries(ri->rrdmetrics) > 0) {
2499 RRDMETRIC *rm;
2500 dfe_start_read((DICTIONARY *)ri->rrdmetrics, rm) {
2501 if(unlikely(netdata_exit)) break;
@@ -2574,7 +2598,7 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG
2598 time_t min_first_time_t = LONG_MAX, max_last_time_t = 0;
2599 size_t instances_active = 0, instances_deleted = 0;
2600 bool live_retention = true, currently_collected = false, hidden = true;
2577 - if(dictionary_stats_entries(rc->rrdinstances) > 0) {
2601 + if(dictionary_entries(rc->rrdinstances) > 0) {
2602 RRDINSTANCE *ri;
2603 dfe_start_reentrant(rc->rrdinstances, ri) {
2604 if(unlikely(netdata_exit)) break;
@@ -2714,7 +2738,7 @@ static void rrdcontext_queue_for_post_processing(RRDCONTEXT *rc, const char *fun
2738 rc,
2739 sizeof(*rc));
2740
2717 -#ifdef NETDATA_INTERNAL_CHECKS
2741 +#if(defined(NETDATA_INTERNAL_CHECKS) && defined(LOG_POST_PROCESSING_QUEUE_INSERTIONS))
2742 {
2743 BUFFER *wb_flags = buffer_create(1000);
2744 rrd_flags_to_buffer(flags, wb_flags);
@@ -2912,7 +2936,7 @@ static void rrdcontext_dispatch_queued_contexts_to_hub(RRDHOST *host, usec_t now
2936 return;
2937
2938 // check if there are queued items to send
2915 - if(!dictionary_stats_entries((DICTIONARY *)host->rrdctx_hub_queue))
2939 + if(!dictionary_entries((DICTIONARY *)host->rrdctx_hub_queue))
2940 return;
2941
2942 if(!host->node_id)
@@ -2975,7 +2999,7 @@ static void rrdcontext_dispatch_queued_contexts_to_hub(RRDHOST *host, usec_t now
2999 rrdcontext_unlock(rc);
3000
3001 // delete it from the master dictionary
2978 - if(dictionary_del((DICTIONARY *)host->rrdctx, string2str(rc->id)) != 0)
3002 + if(!dictionary_del((DICTIONARY *)host->rrdctx, string2str(rc->id)))
3003 error("RRDCONTEXT: '%s' of host '%s' failed to be deleted from rrdcontext dictionary.",
3004 string2str(id), rrdhost_hostname(host));
3005
@@ -3010,7 +3034,10 @@ static void rrdcontext_dispatch_queued_contexts_to_hub(RRDHOST *host, usec_t now
3034 static void rrdcontext_main_cleanup(void *ptr) {
3035 struct netdata_static_thread *static_thread = (struct netdata_static_thread *)ptr;
3036 static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
3037 +
3038 // custom code
3039 + worker_unregister();
3040 +
3041 static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
3042 }
3043
@@ -3065,12 +3092,13 @@ void *rrdcontext_main(void *ptr) {
3092 worker_is_busy(WORKER_JOB_HOSTS);
3093
3094 if(host->rrdctx_post_processing_queue) {
3068 - pp_queued_contexts_for_all_hosts += dictionary_stats_entries((DICTIONARY *)host->rrdctx_post_processing_queue);
3095 + pp_queued_contexts_for_all_hosts +=
3096 + dictionary_entries((DICTIONARY *)host->rrdctx_post_processing_queue);
3097 rrdcontext_post_process_queued_contexts(host);
3098 }
3099
3100 if(host->rrdctx_hub_queue) {
3073 - hub_queued_contexts_for_all_hosts += dictionary_stats_entries((DICTIONARY *)host->rrdctx_hub_queue);
3101 + hub_queued_contexts_for_all_hosts += dictionary_entries((DICTIONARY *)host->rrdctx_hub_queue);
3102 rrdcontext_dispatch_queued_contexts_to_hub(host, now_ut);
3103 }
3104 }
database/rrdcontext.h
+2
@@ -34,6 +34,8 @@ extern void rrdhost_destroy_rrdcontexts(RRDHOST *host);
34 extern void rrdcontext_host_child_connected(RRDHOST *host);
35 extern void rrdcontext_host_child_disconnected(RRDHOST *host);
36
37 +extern int rrdcontext_foreach_instance_with_rrdset_in_context(RRDHOST *host, const char *context, int (*callback)(RRDSET *st, void *data), void *data);
38 +
39 typedef enum {
40 RRDCONTEXT_OPTION_NONE = 0,
41 RRDCONTEXT_OPTION_SHOW_METRICS = (1 << 0),
database/rrddim.c
+325 -264
@@ -10,255 +10,80 @@
10 // ----------------------------------------------------------------------------
11 // RRDDIM index
12
13 -static inline void rrddim_index_add(RRDSET *st, RRDDIM *rd) {
14 - if(likely(dictionary_set(st->rrddim_root_index, string2str(rd->id), rd, sizeof(RRDDIM)) == rd)) {
15 - rrddim_flag_set(rd, RRDDIM_FLAG_INDEXED_ID);
16 - }
17 - else {
18 - rrddim_flag_clear(rd, RRDDIM_FLAG_INDEXED_ID);
19 - error("RRDDIM: %s() attempted to index duplicate dimension with key '%s' of chart '%s' of host '%s'", __FUNCTION__, rrddim_id(rd), rrdset_id(st), rrdhost_hostname(st->rrdhost));
20 - }
21 -}
22 -
23 -static inline void rrddim_index_del(RRDSET *st, RRDDIM *rd) {
24 - if(rrddim_flag_check(rd, RRDDIM_FLAG_INDEXED_ID)) {
25 - if (likely(dictionary_del(st->rrddim_root_index, string2str(rd->id)) == 0))
26 - rrddim_flag_clear(rd, RRDDIM_FLAG_INDEXED_ID);
27 - else
28 - error("RRDDIM: %s() attempted to delete non-indexed dimension with key '%s' of chart '%s' of host '%s'", __FUNCTION__, rrddim_id(rd), rrdset_id(st), rrdhost_hostname(st->rrdhost));
29 - }
30 -}
31 -
32 -static inline RRDDIM *rrddim_index_find(RRDSET *st, const char *id) {
33 - return dictionary_get(st->rrddim_root_index, id);
34 -}
35 -
36 -// ----------------------------------------------------------------------------
37 -// RRDDIM - find a dimension
38 -
39 -inline RRDDIM *rrddim_find(RRDSET *st, const char *id) {
40 - debug(D_RRD_CALLS, "rrddim_find() for chart %s, dimension %s", rrdset_name(st), id);
41 -
42 - return rrddim_index_find(st, id);
43 -}
44 -
45 -// This will not return dimensions that are archived
46 -RRDDIM *rrddim_find_active(RRDSET *st, const char *id) {
47 - RRDDIM *rd = rrddim_find(st, id);
48 -
49 - if (unlikely(rd && rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)))
50 - return NULL;
13 +struct rrddim_constructor {
14 + RRDSET *st;
15 + const char *id;
16 + const char *name;
17 + collected_number multiplier;
18 + collected_number divisor;
19 + RRD_ALGORITHM algorithm;
20 + RRD_MEMORY_MODE memory_mode;
21 +
22 + enum {
23 + RRDDIM_REACT_NONE = 0,
24 + RRDDIM_REACT_NEW = (1 << 0),
25 + RRDDIM_REACT_UPDATED = (1 << 2),
26 + } react_action;
27
52 - return rd;
53 -}
54 -
55 -// ----------------------------------------------------------------------------
56 -// RRDDIM rename a dimension
57 -
58 -inline int rrddim_set_name(RRDSET *st, RRDDIM *rd, const char *name) {
59 - if(unlikely(!name || !*name || !strcmp(rrddim_name(rd), name)))
60 - return 0;
61 -
62 - debug(D_RRD_CALLS, "rrddim_set_name() from %s.%s to %s.%s", rrdset_name(st), rrddim_name(rd), rrdset_name(st), name);
63 -
64 - string_freez(rd->name);
65 - rd->name = rrd_string_strdupz(name);
66 -
67 - if (!rrdset_is_ar_chart(st))
68 - rrddimvar_rename_all(rd);
69 -
70 - rd->exposed = 0;
71 - rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
72 -
73 - ml_dimension_update_name(st, rd, name);
74 -
75 - return 1;
76 -}
77 -
78 -inline int rrddim_set_algorithm(RRDSET *st, RRDDIM *rd, RRD_ALGORITHM algorithm) {
79 - if(unlikely(rd->algorithm == algorithm))
80 - return 0;
81 -
82 - debug(D_RRD_CALLS, "Updating algorithm of dimension '%s/%s' from %s to %s", rrdset_id(st), rrddim_name(rd), rrd_algorithm_name(rd->algorithm), rrd_algorithm_name(algorithm));
83 - rd->algorithm = algorithm;
84 - rd->exposed = 0;
85 - rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
86 - rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
87 - rrdcontext_updated_rrddim_algorithm(rd);
88 - return 1;
89 -}
90 -
91 -inline int rrddim_set_multiplier(RRDSET *st, RRDDIM *rd, collected_number multiplier) {
92 - if(unlikely(rd->multiplier == multiplier))
93 - return 0;
94 -
95 - debug(D_RRD_CALLS, "Updating multiplier of dimension '%s/%s' from " COLLECTED_NUMBER_FORMAT " to " COLLECTED_NUMBER_FORMAT, rrdset_id(st), rrddim_name(rd), rd->multiplier, multiplier);
96 - rd->multiplier = multiplier;
97 - rd->exposed = 0;
98 - rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
99 - rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
100 - rrdcontext_updated_rrddim_multiplier(rd);
101 - return 1;
102 -}
103 -
104 -inline int rrddim_set_divisor(RRDSET *st, RRDDIM *rd, collected_number divisor) {
105 - if(unlikely(rd->divisor == divisor))
106 - return 0;
28 +};
29
108 - debug(D_RRD_CALLS, "Updating divisor of dimension '%s/%s' from " COLLECTED_NUMBER_FORMAT " to " COLLECTED_NUMBER_FORMAT, rrdset_id(st), rrddim_name(rd), rd->divisor, divisor);
109 - rd->divisor = divisor;
110 - rd->exposed = 0;
111 - rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
112 - rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
113 - rrdcontext_updated_rrddim_divisor(rd);
114 - return 1;
115 -}
30 +static void rrddim_update_rrddimvars_unsafe(RRDDIM *rd) {
31 + RRDSET *st = rd->rrdset;
32 + RRDHOST *host = st->rrdhost;
33
117 -// ----------------------------------------------------------------------------
118 -// RRDDIM create a dimension
119 -
120 -void rrdcalc_link_to_rrddim(RRDDIM *rd, RRDSET *st, RRDHOST *host) {
121 - RRDCALC *rc;
122 -
123 - for (rc = host->alarms_with_foreach; rc; rc = rc->next) {
124 - if (simple_pattern_matches(rc->spdim, rrddim_id(rd)) || simple_pattern_matches(rc->spdim, rrddim_name(rd))) {
125 - if (rc->chart == st->name || rc->chart == st->id) {
126 - char *name = alarm_name_with_dim(rrdcalc_name(rc), string_strlen(rc->name), rrddim_name(rd), string_strlen(rd->name));
127 - if(rrdcalc_exists(host, rrdset_name(st), name)) {
128 - freez(name);
129 - continue;
130 - }
131 -
132 - netdata_rwlock_wrlock(&host->health_log.alarm_log_rwlock);
133 - RRDCALC *child = rrdcalc_create_from_rrdcalc(rc, host, name, rrddim_name(rd));
134 - netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
135 -
136 - if (child)
137 - rrdcalc_add_to_host(host, child);
138 -
139 - else {
140 - error("Cannot allocate a new alarm.");
141 - rc->foreachcounter--;
142 - }
143 - }
144 - }
34 + if(host->health_enabled && !rrdset_is_ar_chart(st)) {
35 + rrddimvar_add_and_leave_released(rd, RRDVAR_TYPE_CALCULATED, NULL, NULL, &rd->last_stored_value, RRDVAR_FLAG_NONE);
36 + rrddimvar_add_and_leave_released(rd, RRDVAR_TYPE_COLLECTED, NULL, "_raw", &rd->last_collected_value, RRDVAR_FLAG_NONE);
37 + rrddimvar_add_and_leave_released(rd, RRDVAR_TYPE_TIME_T, NULL, "_last_collected_t", &rd->last_collected_time.tv_sec, RRDVAR_FLAG_NONE);
38 + rrddim_flag_set(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARMS);
39 + rrdset_flag_set(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS);
40 + rrdhost_flag_set(host, RRDHOST_FLAG_PENDING_FOREACH_ALARMS);
41 }
42 }
43
148 -// Return either
149 -// 0 : Dimension is live
150 -// last collected time : Dimension is not live
151 -
152 -#ifdef ENABLE_ACLK
153 -time_t calc_dimension_liveness(RRDDIM *rd, time_t now)
154 -{
155 - time_t last_updated = rd->last_collected_time.tv_sec;
156 - int live;
157 - if (rd->aclk_live_status == 1)
158 - live =
159 - ((now - last_updated) <
160 - MIN(rrdset_free_obsolete_time, RRDSET_MINIMUM_DIM_OFFLINE_MULTIPLIER * rd->update_every));
161 - else
162 - live = ((now - last_updated) < RRDSET_MINIMUM_DIM_LIVE_MULTIPLIER * rd->update_every);
163 - return live ? 0 : last_updated;
164 -}
165 -#endif
166 -
167 -RRDDIM *rrddim_add_custom(RRDSET *st
168 - , const char *id
169 - , const char *name
170 - , collected_number multiplier
171 - , collected_number divisor
172 - , RRD_ALGORITHM algorithm
173 - , RRD_MEMORY_MODE memory_mode
174 - ) {
44 +static void rrddim_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddim, void *constructor_data) {
45 + struct rrddim_constructor *ctr = constructor_data;
46 + RRDDIM *rd = rrddim;
47 + RRDSET *st = ctr->st;
48 RRDHOST *host = st->rrdhost;
176 - rrdset_wrlock(st);
177 -
178 - RRDDIM *rd = rrddim_find(st, id);
179 - if(unlikely(rd)) {
180 - debug(D_RRD_CALLS, "Cannot create rrd dimension '%s/%s', it already exists.", rrdset_id(st), name?name:"<NONAME>");
181 -
182 - int rc = rrddim_set_name(st, rd, name);
183 - rc += rrddim_set_algorithm(st, rd, algorithm);
184 - rc += rrddim_set_multiplier(st, rd, multiplier);
185 - rc += rrddim_set_divisor(st, rd, divisor);
186 -
187 - if (rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)) {
188 - store_active_dimension(&rd->metric_uuid);
189 -
190 - for(int tier = 0; tier < storage_tiers ;tier++) {
191 - if (rd->tiers[tier])
192 - rd->tiers[tier]->db_collection_handle =
193 - rd->tiers[tier]->collect_ops.init(rd->tiers[tier]->db_metric_handle);
194 - }
49
196 - rrddim_flag_clear(rd, RRDDIM_FLAG_ARCHIVED);
197 - rrddimvar_create(rd, RRDVAR_TYPE_CALCULATED, NULL, NULL, &rd->last_stored_value, RRDVAR_OPTION_DEFAULT);
198 - rrddimvar_create(rd, RRDVAR_TYPE_COLLECTED, NULL, "_raw", &rd->last_collected_value, RRDVAR_OPTION_DEFAULT);
199 - rrddimvar_create(rd, RRDVAR_TYPE_TIME_T, NULL, "_last_collected_t", &rd->last_collected_time.tv_sec, RRDVAR_OPTION_DEFAULT);
50 + rd->flags = RRDDIM_FLAG_NONE;
51
201 - rrddim_flag_set(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARM);
202 - rrdset_flag_set(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS);
203 - rrdhost_flag_set(host, RRDHOST_FLAG_PENDING_FOREACH_ALARMS);
204 - }
205 -
206 - if (unlikely(rc)) {
207 - debug(D_METADATALOG, "DIMENSION [%s] metadata updated", rrddim_id(rd));
208 - (void)sql_store_dimension(&rd->metric_uuid, rd->rrdset->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor,
209 - rd->algorithm);
210 -#ifdef ENABLE_ACLK
211 - queue_dimension_to_aclk(rd, calc_dimension_liveness(rd, now_realtime_sec()));
212 -#endif
213 - rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
214 - rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
215 - }
216 - rrdset_unlock(st);
217 - rrdcontext_updated_rrddim(rd);
218 - return rd;
219 - }
220 -
221 - rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
222 - rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
223 -
224 - rd = callocz(1, sizeof(RRDDIM));
225 - rd->id = string_strdupz(id);
52 + rd->id = string_strdupz(ctr->id);
53 + rd->name = (ctr->name && *ctr->name)?rrd_string_strdupz(ctr->name):string_dup(rd->id);
54
227 - rd->name = (name && *name)?rrd_string_strdupz(name):string_dup(rd->id);
228 -
229 - rd->algorithm = algorithm;
230 - rd->multiplier = multiplier;
231 - rd->divisor = divisor;
55 + rd->algorithm = ctr->algorithm;
56 + rd->multiplier = ctr->multiplier;
57 + rd->divisor = ctr->divisor;
58 if(!rd->divisor) rd->divisor = 1;
59
234 - rd->entries = st->entries;
60 rd->update_every = st->update_every;
61
62 + rd->rrdset = st;
63 +
64 if(rrdset_flag_check(st, RRDSET_FLAG_STORE_FIRST))
65 rd->collections_counter = 1;
66
240 - rd->rrdset = st;
241 -
242 - if(memory_mode == RRD_MEMORY_MODE_MAP || memory_mode == RRD_MEMORY_MODE_SAVE) {
243 - if(!rrddim_memory_load_or_create_map_save(st, rd, memory_mode)) {
244 - info("Failed to use memory mode %s for chart '%s', dimension '%s', falling back to ram", (memory_mode == RRD_MEMORY_MODE_MAP)?"map":"save", rrdset_name(st), rrddim_name(rd));
245 - memory_mode = RRD_MEMORY_MODE_RAM;
67 + if(ctr->memory_mode == RRD_MEMORY_MODE_MAP || ctr->memory_mode == RRD_MEMORY_MODE_SAVE) {
68 + if(!rrddim_memory_load_or_create_map_save(st, rd, ctr->memory_mode)) {
69 + info("Failed to use memory mode %s for chart '%s', dimension '%s', falling back to ram", (ctr->memory_mode == RRD_MEMORY_MODE_MAP)?"map":"save", rrdset_name(st), rrddim_name(rd));
70 + ctr->memory_mode = RRD_MEMORY_MODE_RAM;
71 }
72 }
73
249 - if(memory_mode == RRD_MEMORY_MODE_RAM) {
74 + if(ctr->memory_mode == RRD_MEMORY_MODE_RAM) {
75 size_t entries = st->entries;
76 if(!entries) entries = 5;
77
78 rd->db = netdata_mmap(NULL, entries * sizeof(storage_number), MAP_PRIVATE, 1);
79 if(!rd->db) {
80 info("Failed to use memory mode ram for chart '%s', dimension '%s', falling back to alloc", rrdset_name(st), rrddim_name(rd));
256 - memory_mode = RRD_MEMORY_MODE_ALLOC;
81 + ctr->memory_mode = RRD_MEMORY_MODE_ALLOC;
82 }
83 else rd->memsize = entries * sizeof(storage_number);
84 }
85
261 - if(memory_mode == RRD_MEMORY_MODE_ALLOC || memory_mode == RRD_MEMORY_MODE_NONE) {
86 + if(ctr->memory_mode == RRD_MEMORY_MODE_ALLOC || ctr->memory_mode == RRD_MEMORY_MODE_NONE) {
87 size_t entries = st->entries;
88 if(entries < 5) entries = 5;
89
@@ -266,7 +91,7 @@ RRDDIM *rrddim_add_custom(RRDSET *st
91 rd->memsize = entries * sizeof(storage_number);
92 }
93
269 - rd->rrd_memory_mode = memory_mode;
94 + rd->rrd_memory_mode = ctr->memory_mode;
95
96 #ifdef ENABLE_ACLK
97 rd->aclk_live_status = -1;
@@ -277,7 +102,7 @@ RRDDIM *rrddim_add_custom(RRDSET *st
102 // initialize the db tiers
103 {
104 size_t initialized = 0;
280 - RRD_MEMORY_MODE wanted_mode = memory_mode;
105 + RRD_MEMORY_MODE wanted_mode = ctr->memory_mode;
106 for(int tier = 0; tier < storage_tiers ; tier++, wanted_mode = RRD_MEMORY_MODE_DBENGINE) {
107 STORAGE_ENGINE *eng = storage_engine_get(wanted_mode);
108 if(!eng) continue;
@@ -301,8 +126,6 @@ RRDDIM *rrddim_add_custom(RRDSET *st
126 error("Failed to initialize the first db tier for chart '%s', dimension '%s", rrdset_name(st), rrddim_name(rd));
127 }
128
304 - store_active_dimension(&rd->metric_uuid);
305 -
129 // initialize data collection for all tiers
130 {
131 size_t initialized = 0;
@@ -322,51 +145,47 @@ RRDDIM *rrddim_add_custom(RRDSET *st
145
146 if(td->algorithm != rd->algorithm || ABS(td->multiplier) != ABS(rd->multiplier) || ABS(td->divisor) != ABS(rd->divisor)) {
147 if(!rrdset_flag_check(st, RRDSET_FLAG_HETEROGENEOUS)) {
325 - #ifdef NETDATA_INTERNAL_CHECKS
148 +#ifdef NETDATA_INTERNAL_CHECKS
149 info("Dimension '%s' added on chart '%s' of host '%s' is not homogeneous to other dimensions already present (algorithm is '%s' vs '%s', multiplier is " COLLECTED_NUMBER_FORMAT " vs " COLLECTED_NUMBER_FORMAT ", divisor is " COLLECTED_NUMBER_FORMAT " vs " COLLECTED_NUMBER_FORMAT ").",
327 - rrddim_name(rd),
328 - rrdset_name(st),
329 - rrdhost_hostname(host),
330 - rrd_algorithm_name(rd->algorithm), rrd_algorithm_name(td->algorithm),
331 - rd->multiplier, td->multiplier,
332 - rd->divisor, td->divisor
150 + rrddim_name(rd),
151 + rrdset_name(st),
152 + rrdhost_hostname(host),
153 + rrd_algorithm_name(rd->algorithm), rrd_algorithm_name(td->algorithm),
154 + rd->multiplier, td->multiplier,
155 + rd->divisor, td->divisor
156 );
334 - #endif
157 +#endif
158 rrdset_flag_set(st, RRDSET_FLAG_HETEROGENEOUS);
159 }
160 }
161 }
162
340 - // append this dimension
341 - DOUBLE_LINKED_LIST_APPEND_UNSAFE(st->dimensions, rd, prev, next);
163 + rrddim_update_rrddimvars_unsafe(rd);
164
343 - if(host->health_enabled && !rrdset_is_ar_chart(st)) {
344 - rrddimvar_create(rd, RRDVAR_TYPE_CALCULATED, NULL, NULL, &rd->last_stored_value, RRDVAR_OPTION_DEFAULT);
345 - rrddimvar_create(rd, RRDVAR_TYPE_COLLECTED, NULL, "_raw", &rd->last_collected_value, RRDVAR_OPTION_DEFAULT);
346 - rrddimvar_create(rd, RRDVAR_TYPE_TIME_T, NULL, "_last_collected_t", &rd->last_collected_time.tv_sec, RRDVAR_OPTION_DEFAULT);
347 - }
165 + // let the chart resync
166 + rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
167 + rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
168
349 - rrddim_index_add(st, rd);
169 + ml_new_dimension(rd);
170
351 - rrddim_flag_set(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARM);
352 - rrdset_flag_set(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS);
353 - rrdhost_flag_set(host, RRDHOST_FLAG_PENDING_FOREACH_ALARMS);
171 + ctr->react_action = RRDDIM_REACT_NEW;
172
355 - ml_new_dimension(rd);
173 + internal_error(false, "RRDDIM: inserted dimension '%s' of chart '%s' of host '%s'",
174 + rrddim_name(rd), rrdset_name(st), rrdhost_hostname(st->rrdhost));
175
357 - rrdset_unlock(st);
358 - rrdcontext_updated_rrddim(rd);
359 - return(rd);
176 }
177
362 -// ----------------------------------------------------------------------------
363 -// RRDDIM remove / free a dimension
178 +static void rrddim_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddim, void *rrdset) {
179 + RRDDIM *rd = rrddim;
180 + RRDSET *st = rrdset; (void)st;
181 +
182 + internal_error(false, "RRDDIM: deleting dimension '%s' of chart '%s' of host '%s'",
183 + rrddim_name(rd), rrdset_name(st), rrdhost_hostname(st->rrdhost));
184
365 -void rrddim_free(RRDSET *st, RRDDIM *rd)
366 -{
185 rrdcontext_removed_rrddim(rd);
186 +
187 ml_delete_dimension(rd);
369 -
188 +
189 debug(D_RRD_CALLS, "rrddim_free() %s.%s", rrdset_name(st), rrddim_name(rd));
190
191 if (!rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)) {
@@ -389,18 +208,13 @@ void rrddim_free(RRDSET *st, RRDDIM *rd)
208 }
209 }
210
392 - DOUBLE_LINKED_LIST_REMOVE_UNSAFE(st->dimensions, rd, prev, next);
393 -
394 - while(rd->variables)
395 - rrddimvar_free(rd->variables);
396 -
397 - rrddim_index_del(st, rd);
211 + rrddimvar_delete_all(rd);
212
213 // free(rd->annotations);
400 -//#ifdef ENABLE_ACLK
401 -// if (!netdata_exit)
402 -// aclk_send_dimension_update(rd);
403 -//#endif
214 + //#ifdef ENABLE_ACLK
215 + // if (!netdata_exit)
216 + // aclk_send_dimension_update(rd);
217 + //#endif
218
219 // this will free MEMORY_MODE_SAVE and MEMORY_MODE_MAP structures
220 rrddim_memory_file_free(rd);
@@ -425,7 +239,253 @@ void rrddim_free(RRDSET *st, RRDDIM *rd)
239
240 string_freez(rd->id);
241 string_freez(rd->name);
428 - freez(rd);
242 +}
243 +
244 +static bool rrddim_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddim, void *new_rrddim, void *constructor_data) {
245 + (void)new_rrddim; // it is NULL
246 +
247 + struct rrddim_constructor *ctr = constructor_data;
248 + RRDDIM *rd = rrddim;
249 + RRDSET *st = ctr->st;
250 +
251 + ctr->react_action = RRDDIM_REACT_NONE;
252 +
253 + int rc = rrddim_reset_name(st, rd, ctr->name);
254 + rc += rrddim_set_algorithm(st, rd, ctr->algorithm);
255 + rc += rrddim_set_multiplier(st, rd, ctr->multiplier);
256 + rc += rrddim_set_divisor(st, rd, ctr->divisor);
257 +
258 + if(rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)) {
259 +
260 + for(int tier = 0; tier < storage_tiers ;tier++) {
261 + if (rd->tiers[tier])
262 + rd->tiers[tier]->db_collection_handle =
263 + rd->tiers[tier]->collect_ops.init(rd->tiers[tier]->db_metric_handle);
264 + }
265 +
266 + rrddim_flag_clear(rd, RRDDIM_FLAG_ARCHIVED);
267 + rrddim_update_rrddimvars_unsafe(rd);
268 + }
269 +
270 + if(unlikely(rc))
271 + ctr->react_action = RRDDIM_REACT_UPDATED;
272 +
273 + return ctr->react_action == RRDDIM_REACT_UPDATED;
274 +}
275 +
276 +static void rrddim_react_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddim, void *constructor_data) {
277 + struct rrddim_constructor *ctr = constructor_data;
278 + RRDDIM *rd = rrddim;
279 + RRDSET *st = ctr->st;
280 +
281 + if(ctr->react_action == RRDDIM_REACT_UPDATED) {
282 + debug(D_METADATALOG, "DIMENSION [%s] metadata updated", rrddim_id(rd));
283 + (void)sql_store_dimension(&rd->metric_uuid, &rd->rrdset->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor, rd->algorithm);
284 +#ifdef ENABLE_ACLK
285 + queue_dimension_to_aclk(rd, calc_dimension_liveness(rd, now_realtime_sec()));
286 +#endif
287 +
288 + // the chart needs to be updated to the parent
289 + rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
290 + rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
291 + }
292 +
293 + rrdcontext_updated_rrddim(rd);
294 +}
295 +
296 +void rrddim_index_init(RRDSET *st) {
297 + if(!st->rrddim_root_index) {
298 + st->rrddim_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
299 +
300 + dictionary_register_insert_callback(st->rrddim_root_index, rrddim_insert_callback, NULL);
301 + dictionary_register_conflict_callback(st->rrddim_root_index, rrddim_conflict_callback, NULL);
302 + dictionary_register_delete_callback(st->rrddim_root_index, rrddim_delete_callback, st);
303 + dictionary_register_react_callback(st->rrddim_root_index, rrddim_react_callback, st);
304 + }
305 +}
306 +
307 +void rrddim_index_destroy(RRDSET *st) {
308 + dictionary_destroy(st->rrddim_root_index);
309 + st->rrddim_root_index = NULL;
310 +}
311 +
312 +static inline RRDDIM *rrddim_index_find(RRDSET *st, const char *id) {
313 + return dictionary_get(st->rrddim_root_index, id);
314 +}
315 +
316 +// ----------------------------------------------------------------------------
317 +// RRDDIM - find a dimension
318 +
319 +inline RRDDIM *rrddim_find(RRDSET *st, const char *id) {
320 + debug(D_RRD_CALLS, "rrddim_find() for chart %s, dimension %s", rrdset_name(st), id);
321 +
322 + return rrddim_index_find(st, id);
323 +}
324 +
325 +// This will not return dimensions that are archived
326 +RRDDIM *rrddim_find_active(RRDSET *st, const char *id) {
327 + RRDDIM *rd = rrddim_find(st, id);
328 +
329 + if (unlikely(rd && rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)))
330 + return NULL;
331 +
332 + return rd;
333 +}
334 +
335 +// ----------------------------------------------------------------------------
336 +// RRDDIM rename a dimension
337 +
338 +inline int rrddim_reset_name(RRDSET *st, RRDDIM *rd, const char *name) {
339 + if(unlikely(!name || !*name || !strcmp(rrddim_name(rd), name)))
340 + return 0;
341 +
342 + debug(D_RRD_CALLS, "rrddim_reset_name() from %s.%s to %s.%s", rrdset_name(st), rrddim_name(rd), rrdset_name(st), name);
343 +
344 + STRING *old = rd->name;
345 + rd->name = rrd_string_strdupz(name);
346 + string_freez(old);
347 +
348 + if (!rrdset_is_ar_chart(st))
349 + rrddimvar_rename_all(rd);
350 +
351 + rd->exposed = 0;
352 + rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
353 +
354 + ml_dimension_update_name(st, rd, name);
355 +
356 + return 1;
357 +}
358 +
359 +inline int rrddim_set_algorithm(RRDSET *st, RRDDIM *rd, RRD_ALGORITHM algorithm) {
360 + if(unlikely(rd->algorithm == algorithm))
361 + return 0;
362 +
363 + debug(D_RRD_CALLS, "Updating algorithm of dimension '%s/%s' from %s to %s", rrdset_id(st), rrddim_name(rd), rrd_algorithm_name(rd->algorithm), rrd_algorithm_name(algorithm));
364 + rd->algorithm = algorithm;
365 + rd->exposed = 0;
366 + rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
367 + rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
368 + rrdcontext_updated_rrddim_algorithm(rd);
369 + return 1;
370 +}
371 +
372 +inline int rrddim_set_multiplier(RRDSET *st, RRDDIM *rd, collected_number multiplier) {
373 + if(unlikely(rd->multiplier == multiplier))
374 + return 0;
375 +
376 + debug(D_RRD_CALLS, "Updating multiplier of dimension '%s/%s' from " COLLECTED_NUMBER_FORMAT " to " COLLECTED_NUMBER_FORMAT, rrdset_id(st), rrddim_name(rd), rd->multiplier, multiplier);
377 + rd->multiplier = multiplier;
378 + rd->exposed = 0;
379 + rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
380 + rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
381 + rrdcontext_updated_rrddim_multiplier(rd);
382 + return 1;
383 +}
384 +
385 +inline int rrddim_set_divisor(RRDSET *st, RRDDIM *rd, collected_number divisor) {
386 + if(unlikely(rd->divisor == divisor))
387 + return 0;
388 +
389 + debug(D_RRD_CALLS, "Updating divisor of dimension '%s/%s' from " COLLECTED_NUMBER_FORMAT " to " COLLECTED_NUMBER_FORMAT, rrdset_id(st), rrddim_name(rd), rd->divisor, divisor);
390 + rd->divisor = divisor;
391 + rd->exposed = 0;
392 + rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
393 + rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
394 + rrdcontext_updated_rrddim_divisor(rd);
395 + return 1;
396 +}
397 +
398 +// ----------------------------------------------------------------------------
399 +
400 +// get the timestamp of the last entry in the round-robin database
401 +time_t rrddim_last_entry_t(RRDDIM *rd) {
402 + time_t latest = rd->tiers[0]->query_ops.latest_time(rd->tiers[0]->db_metric_handle);
403 +
404 + for(int tier = 1; tier < storage_tiers ;tier++) {
405 + if(unlikely(!rd->tiers[tier])) continue;
406 +
407 + time_t t = rd->tiers[tier]->query_ops.latest_time(rd->tiers[tier]->db_metric_handle);
408 + if(t > latest)
409 + latest = t;
410 + }
411 +
412 + return latest;
413 +}
414 +
415 +time_t rrddim_first_entry_t(RRDDIM *rd) {
416 + time_t oldest = 0;
417 +
418 + for(int tier = 0; tier < storage_tiers ;tier++) {
419 + if(unlikely(!rd->tiers[tier])) continue;
420 +
421 + time_t t = rd->tiers[tier]->query_ops.oldest_time(rd->tiers[tier]->db_metric_handle);
422 + if(t != 0 && (oldest == 0 || t < oldest))
423 + oldest = t;
424 + }
425 +
426 + return oldest;
427 +}
428 +
429 +// Return either
430 +// 0 : Dimension is live
431 +// last collected time : Dimension is not live
432 +
433 +#ifdef ENABLE_ACLK
434 +time_t calc_dimension_liveness(RRDDIM *rd, time_t now)
435 +{
436 + time_t last_updated = rd->last_collected_time.tv_sec;
437 + int live;
438 +
439 + if(rd->aclk_live_status == 1) {
440 + int offline_time = RRDSET_MINIMUM_DIM_OFFLINE_MULTIPLIER * rd->update_every;
441 + live = ((now - last_updated) < MIN(rrdset_free_obsolete_time, offline_time));
442 + }
443 + else
444 + live = ((now - last_updated) < RRDSET_MINIMUM_DIM_LIVE_MULTIPLIER * rd->update_every);
445 +
446 + return live ? 0 : last_updated;
447 +}
448 +#endif
449 +
450 +RRDDIM *rrddim_add_custom(RRDSET *st
451 + , const char *id
452 + , const char *name
453 + , collected_number multiplier
454 + , collected_number divisor
455 + , RRD_ALGORITHM algorithm
456 + , RRD_MEMORY_MODE memory_mode
457 + ) {
458 + struct rrddim_constructor tmp = {
459 + .st = st,
460 + .id = id,
461 + .name = name,
462 + .multiplier = multiplier,
463 + .divisor = divisor,
464 + .algorithm = algorithm,
465 + .memory_mode = memory_mode,
466 + };
467 +
468 + RRDDIM *rd = dictionary_set_advanced(st->rrddim_root_index, tmp.id, -1, NULL, sizeof(RRDDIM), &tmp);
469 +
470 + if(tmp.react_action == RRDDIM_REACT_NEW) {
471 + // append this dimension
472 + rrdset_wrlock(st);
473 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(st->dimensions, rd, prev, next);
474 + rrdset_unlock(st);
475 + }
476 +
477 + return(rd);
478 +}
479 +
480 +// ----------------------------------------------------------------------------
481 +// RRDDIM remove / free a dimension
482 +
483 +void rrddim_free(RRDSET *st, RRDDIM *rd) {
484 + rrdset_wrlock(st);
485 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(st->dimensions, rd, prev, next);
486 + rrdset_unlock(st);
487 +
488 + dictionary_del(st->rrddim_root_index, string2str(rd->id));
489 }
490
491
@@ -445,7 +505,7 @@ int rrddim_hide(RRDSET *st, const char *id) {
505 if (!rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN))
506 (void)sql_set_dimension_option(&rd->metric_uuid, "hidden");
507
448 - rrddim_flag_set(rd, RRDDIM_FLAG_HIDDEN);
508 + rrddim_option_set(rd, RRDDIM_OPTION_HIDDEN);
509 rrddim_flag_set(rd, RRDDIM_FLAG_META_HIDDEN);
510 rrdcontext_updated_rrddim_flags(rd);
511 return 0;
@@ -463,7 +523,7 @@ int rrddim_unhide(RRDSET *st, const char *id) {
523 if (rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN))
524 (void)sql_set_dimension_option(&rd->metric_uuid, NULL);
525
466 - rrddim_flag_clear(rd, RRDDIM_FLAG_HIDDEN);
526 + rrddim_option_clear(rd, RRDDIM_OPTION_HIDDEN);
527 rrddim_flag_clear(rd, RRDDIM_FLAG_META_HIDDEN);
528 rrdcontext_updated_rrddim_flags(rd);
529 return 0;
@@ -478,6 +538,7 @@ inline void rrddim_is_obsolete(RRDSET *st, RRDDIM *rd) {
538 }
539 rrddim_flag_set(rd, RRDDIM_FLAG_OBSOLETE);
540 rrdset_flag_set(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS);
541 + rrdhost_flag_set(st->rrdhost, RRDHOST_FLAG_PENDING_OBSOLETE_DIMENSIONS);
542 rrdcontext_updated_rrddim_flags(rd);
543 }
544
database/rrddimvar.c
+173 -105
@@ -1,84 +1,87 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 -#define NETDATA_HEALTH_INTERNALS
3 #include "rrd.h"
4
5 +typedef struct rrddimvar {
6 + struct rrddim *rrddim;
7 +
8 + STRING *prefix;
9 + STRING *suffix;
10 + void *value;
11 +
12 + const RRDVAR_ACQUIRED *rrdvar_local_dim_id;
13 + const RRDVAR_ACQUIRED *rrdvar_local_dim_name;
14 +
15 + const RRDVAR_ACQUIRED *rrdvar_family_id;
16 + const RRDVAR_ACQUIRED *rrdvar_family_name;
17 + const RRDVAR_ACQUIRED *rrdvar_family_context_dim_id;
18 + const RRDVAR_ACQUIRED *rrdvar_family_context_dim_name;
19 +
20 + const RRDVAR_ACQUIRED *rrdvar_host_chart_id_dim_id;
21 + const RRDVAR_ACQUIRED *rrdvar_host_chart_id_dim_name;
22 + const RRDVAR_ACQUIRED *rrdvar_host_chart_name_dim_id;
23 + const RRDVAR_ACQUIRED *rrdvar_host_chart_name_dim_name;
24 +
25 + RRDVAR_FLAGS flags:24;
26 + RRDVAR_TYPE type:8;
27 +} RRDDIMVAR;
28 +
29 // ----------------------------------------------------------------------------
30 // RRDDIMVAR management
31 // DIMENSION VARIABLES
32
33 #define RRDDIMVAR_ID_MAX 1024
34
12 -static inline void rrddimvar_free_variables(RRDDIMVAR *rs) {
35 +static inline void rrddimvar_free_variables_unsafe(RRDDIMVAR *rs) {
36 RRDDIM *rd = rs->rrddim;
37 RRDSET *st = rd->rrdset;
38 RRDHOST *host = st->rrdhost;
39
40 // CHART VARIABLES FOR THIS DIMENSION
41
19 - rrdvar_free(host, st->rrdvar_root_index, rs->var_local_id);
20 - rs->var_local_id = NULL;
42 + if(st->rrdvars) {
43 + rrdvar_release_and_del(st->rrdvars, rs->rrdvar_local_dim_id);
44 + rs->rrdvar_local_dim_id = NULL;
45
22 - rrdvar_free(host, st->rrdvar_root_index, rs->var_local_name);
23 - rs->var_local_name = NULL;
46 + rrdvar_release_and_del(st->rrdvars, rs->rrdvar_local_dim_name);
47 + rs->rrdvar_local_dim_name = NULL;
48 + }
49
50 // FAMILY VARIABLES FOR THIS DIMENSION
51
27 - rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_id);
28 - rs->var_family_id = NULL;
52 + if(st->rrdfamily) {
53 + rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_id);
54 + rs->rrdvar_family_id = NULL;
55
30 - rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_name);
31 - rs->var_family_name = NULL;
56 + rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_name);
57 + rs->rrdvar_family_name = NULL;
58
33 - rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_contextid);
34 - rs->var_family_contextid = NULL;
59 + rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_context_dim_id);
60 + rs->rrdvar_family_context_dim_id = NULL;
61
36 - rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_contextname);
37 - rs->var_family_contextname = NULL;
62 + rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_context_dim_name);
63 + rs->rrdvar_family_context_dim_name = NULL;
64 + }
65
66 // HOST VARIABLES FOR THIS DIMENSION
67
41 - rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartidid);
42 - rs->var_host_chartidid = NULL;
43 -
44 - rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartidname);
45 - rs->var_host_chartidname = NULL;
46 -
47 - rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartnameid);
48 - rs->var_host_chartnameid = NULL;
49 -
50 - rrdvar_free(host, host->rrdvar_root_index, rs->var_host_chartnamename);
51 - rs->var_host_chartnamename = NULL;
52 -
53 - // KEYS
54 -
55 - string_freez(rs->key_id);
56 - rs->key_id = NULL;
57 -
58 - string_freez(rs->key_name);
59 - rs->key_name = NULL;
60 -
61 - string_freez(rs->key_fullidid);
62 - rs->key_fullidid = NULL;
63 -
64 - string_freez(rs->key_fullidname);
65 - rs->key_fullidname = NULL;
66 -
67 - string_freez(rs->key_contextid);
68 - rs->key_contextid = NULL;
68 + if(host->rrdvars && host->health_enabled) {
69 + rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_id_dim_id);
70 + rs->rrdvar_host_chart_id_dim_id = NULL;
71
70 - string_freez(rs->key_contextname);
71 - rs->key_contextname = NULL;
72 + rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_id_dim_name);
73 + rs->rrdvar_host_chart_id_dim_name = NULL;
74
73 - string_freez(rs->key_fullnameid);
74 - rs->key_fullnameid = NULL;
75 + rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_name_dim_id);
76 + rs->rrdvar_host_chart_name_dim_id = NULL;
77
76 - string_freez(rs->key_fullnamename);
77 - rs->key_fullnamename = NULL;
78 + rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_name_dim_name);
79 + rs->rrdvar_host_chart_name_dim_name = NULL;
80 + }
81 }
82
80 -static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
81 - rrddimvar_free_variables(rs);
83 +static inline void rrddimvar_update_variables_unsafe(RRDDIMVAR *rs) {
84 + rrddimvar_free_variables_unsafe(rs);
85
86 RRDDIM *rd = rs->rrddim;
87 RRDSET *st = rd->rrdset;
@@ -89,28 +92,28 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
92 // KEYS
93
94 snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s%s%s", string2str(rs->prefix), rrddim_id(rd), string2str(rs->suffix));
92 - rs->key_id = string_strdupz(buffer);
95 + STRING *key_dim_id = string_strdupz(buffer);
96
97 snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s%s%s", string2str(rs->prefix), rrddim_name(rd), string2str(rs->suffix));
95 - rs->key_name = string_strdupz(buffer);
98 + STRING *key_dim_name = string_strdupz(buffer);
99
97 - snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_id(st), string2str(rs->key_id));
98 - rs->key_fullidid = string_strdupz(buffer);
100 + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_id(st), string2str(key_dim_id));
101 + STRING *key_chart_id_dim_id = string_strdupz(buffer);
102
100 - snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_id(st), string2str(rs->key_name));
101 - rs->key_fullidname = string_strdupz(buffer);
103 + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_id(st), string2str(key_dim_name));
104 + STRING *key_chart_id_dim_name = string_strdupz(buffer);
105
103 - snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_context(st), string2str(rs->key_id));
104 - rs->key_contextid = string_strdupz(buffer);
106 + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_context(st), string2str(key_dim_id));
107 + STRING *key_context_dim_id = string_strdupz(buffer);
108
106 - snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_context(st), string2str(rs->key_name));
107 - rs->key_contextname = string_strdupz(buffer);
109 + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_context(st), string2str(key_dim_name));
110 + STRING *key_context_dim_name = string_strdupz(buffer);
111
109 - snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_name(st), string2str(rs->key_id));
110 - rs->key_fullnameid = string_strdupz(buffer);
112 + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_name(st), string2str(key_dim_id));
113 + STRING *key_chart_name_dim_id = string_strdupz(buffer);
114
112 - snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_name(st), string2str(rs->key_name));
113 - rs->key_fullnamename = string_strdupz(buffer);
115 + snprintfz(buffer, RRDDIMVAR_ID_MAX, "%s.%s", rrdset_name(st), string2str(key_dim_name));
116 + STRING *key_chart_name_dim_name = string_strdupz(buffer);
117
118 // CHART VARIABLES FOR THIS DIMENSION
119 // -----------------------------------
@@ -119,8 +122,10 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
122 // - $id
123 // - $name
124
122 - rs->var_local_id = rrdvar_create_and_index("local", st->rrdvar_root_index, rs->key_id, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
123 - rs->var_local_name = rrdvar_create_and_index("local", st->rrdvar_root_index, rs->key_name, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
125 + if(st->rrdvars) {
126 + rs->rrdvar_local_dim_id = rrdvar_add_and_acquire("local", st->rrdvars, key_dim_id, rs->type, RRDVAR_FLAG_NONE, rs->value);
127 + rs->rrdvar_local_dim_name = rrdvar_add_and_acquire("local", st->rrdvars, key_dim_name, rs->type, RRDVAR_FLAG_NONE, rs->value);
128 + }
129
130 // FAMILY VARIABLES FOR THIS DIMENSION
131 // -----------------------------------
@@ -131,10 +136,12 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
136 // - $chart-context.id
137 // - $chart-context.name
138
134 - rs->var_family_id = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_id, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
135 - rs->var_family_name = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_name, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
136 - rs->var_family_contextid = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_contextid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
137 - rs->var_family_contextname = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_contextname, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
139 + if(st->rrdfamily) {
140 + rs->rrdvar_family_id = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_dim_id, rs->type, RRDVAR_FLAG_NONE, rs->value);
141 + rs->rrdvar_family_name = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_dim_name, rs->type, RRDVAR_FLAG_NONE, rs->value);
142 + rs->rrdvar_family_context_dim_id = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_context_dim_id, rs->type, RRDVAR_FLAG_NONE, rs->value);
143 + rs->rrdvar_family_context_dim_name = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_context_dim_name, rs->type, RRDVAR_FLAG_NONE, rs->value);
144 + }
145
146 // HOST VARIABLES FOR THIS DIMENSION
147 // -----------------------------------
@@ -145,60 +152,121 @@ static inline void rrddimvar_create_variables(RRDDIMVAR *rs) {
152 // - $chart-name.id
153 // - $chart-name.name
154
148 - rs->var_host_chartidid = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullidid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
149 - rs->var_host_chartidname = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullidname, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
150 - rs->var_host_chartnameid = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullnameid, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
151 - rs->var_host_chartnamename = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullnamename, rs->type, RRDVAR_OPTION_DEFAULT, rs->value);
155 + if(host->rrdvars && host->health_enabled) {
156 + rs->rrdvar_host_chart_id_dim_id = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_id_dim_id, rs->type, RRDVAR_FLAG_NONE, rs->value);
157 + rs->rrdvar_host_chart_id_dim_name = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_id_dim_name, rs->type, RRDVAR_FLAG_NONE, rs->value);
158 + rs->rrdvar_host_chart_name_dim_id = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_name_dim_id, rs->type, RRDVAR_FLAG_NONE, rs->value);
159 + rs->rrdvar_host_chart_name_dim_name = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_name_dim_name, rs->type, RRDVAR_FLAG_NONE, rs->value);
160 + }
161 +
162 + // free the keys
163 +
164 + string_freez(key_dim_id);
165 + string_freez(key_dim_name);
166 + string_freez(key_chart_id_dim_id);
167 + string_freez(key_chart_id_dim_name);
168 + string_freez(key_context_dim_id);
169 + string_freez(key_context_dim_name);
170 + string_freez(key_chart_name_dim_id);
171 + string_freez(key_chart_name_dim_name);
172 }
173
154 -RRDDIMVAR *rrddimvar_create(RRDDIM *rd, RRDVAR_TYPE type, const char *prefix, const char *suffix, void *value, RRDVAR_OPTIONS options) {
155 - RRDSET *st = rd->rrdset;
156 - (void)st;
174 +struct rrddimvar_constructor {
175 + RRDDIM *rrddim;
176 + const char *prefix;
177 + const char *suffix;
178 + void *value;
179 + RRDVAR_FLAGS flags :16;
180 + RRDVAR_TYPE type:8;
181 +};
182
158 - debug(D_VARIABLES, "RRDDIMSET create for chart id '%s' name '%s', dimension id '%s', name '%s%s%s'", rrdset_id(st), rrdset_name(st), rrddim_id(rd), (prefix)?prefix:"", rrddim_name(rd), (suffix)?suffix:"");
183 +static void rrddimvar_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddimvar, void *constructor_data) {
184 + RRDDIMVAR *rs = rrddimvar;
185 + struct rrddimvar_constructor *ctr = constructor_data;
186
160 - if(!prefix) prefix = "";
161 - if(!suffix) suffix = "";
187 + if(!ctr->prefix) ctr->prefix = "";
188 + if(!ctr->suffix) ctr->suffix = "";
189 +
190 + rs->prefix = string_strdupz(ctr->prefix);
191 + rs->suffix = string_strdupz(ctr->suffix);
192 +
193 + rs->type = ctr->type;
194 + rs->value = ctr->value;
195 + rs->flags = ctr->flags;
196 + rs->rrddim = ctr->rrddim;
197 +
198 + rrddimvar_update_variables_unsafe(rs);
199 +}
200 +
201 +static bool rrddimvar_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddimvar, void *new_rrddimvar __maybe_unused, void *constructor_data __maybe_unused) {
202 + RRDDIMVAR *rs = rrddimvar;
203 + rrddimvar_update_variables_unsafe(rs);
204 +
205 + return true;
206 +}
207
163 - RRDDIMVAR *rs = (RRDDIMVAR *)callocz(1, sizeof(RRDDIMVAR));
208 +static void rrddimvar_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrddimvar, void *rrdset __maybe_unused) {
209 + RRDDIMVAR *rs = rrddimvar;
210 + rrddimvar_free_variables_unsafe(rs);
211 + string_freez(rs->prefix);
212 + string_freez(rs->suffix);
213 +}
214
165 - rs->prefix = string_strdupz(prefix);
166 - rs->suffix = string_strdupz(suffix);
215 +void rrddimvar_index_init(RRDSET *st) {
216 + if(!st->rrddimvar_root_index) {
217 + st->rrddimvar_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
218
168 - rs->type = type;
169 - rs->value = value;
170 - rs->options = options;
171 - rs->rrddim = rd;
219 + dictionary_register_insert_callback(st->rrddimvar_root_index, rrddimvar_insert_callback, NULL);
220 + dictionary_register_conflict_callback(st->rrddimvar_root_index, rrddimvar_conflict_callback, NULL);
221 + dictionary_register_delete_callback(st->rrddimvar_root_index, rrddimvar_delete_callback, st);
222 + }
223 +}
224
173 - DOUBLE_LINKED_LIST_PREPEND_UNSAFE(rd->variables, rs, prev, next);
225 +void rrddimvar_index_destroy(RRDSET *st) {
226 + dictionary_destroy(st->rrddimvar_root_index);
227 + st->rrddimvar_root_index = NULL;
228 +}
229
175 - rrddimvar_create_variables(rs);
230 +void rrddimvar_add_and_leave_released(RRDDIM *rd, RRDVAR_TYPE type, const char *prefix, const char *suffix, void *value, RRDVAR_FLAGS flags) {
231 + if(!prefix) prefix = "";
232 + if(!suffix) suffix = "";
233
177 - return rs;
234 + char key[RRDDIMVAR_ID_MAX + 1];
235 + size_t key_len = snprintfz(key, RRDDIMVAR_ID_MAX, "%s_%s_%s", prefix, rrddim_id(rd), suffix);
236 +
237 + struct rrddimvar_constructor tmp = {
238 + .suffix = suffix,
239 + .prefix = prefix,
240 + .type = type,
241 + .flags = flags,
242 + .value = value,
243 + .rrddim = rd
244 + };
245 + dictionary_set_advanced(rd->rrdset->rrddimvar_root_index, key, (ssize_t)(key_len + 1), NULL, sizeof(RRDDIMVAR), &tmp);
246 }
247
248 void rrddimvar_rename_all(RRDDIM *rd) {
249 RRDSET *st = rd->rrdset;
182 - (void)st;
250
184 - debug(D_VARIABLES, "RRDDIMSET rename for chart id '%s' name '%s', dimension id '%s', name '%s'", rrdset_id(st), rrdset_name(st), rrddim_id(rd), rrddim_name(rd));
251 + debug(D_VARIABLES, "RRDDIMVAR rename for chart id '%s' name '%s', dimension id '%s', name '%s'", rrdset_id(st), rrdset_name(st), rrddim_id(rd), rrddim_name(rd));
252
186 - RRDDIMVAR *rs, *next = rd->variables;
187 - while((rs = next)) {
188 - next = rs->next;
189 - rrddimvar_create_variables(rs);
253 + RRDDIMVAR *rs;
254 + dfe_start_write(st->rrddimvar_root_index, rs) {
255 + if(unlikely(rs->rrddim == rd))
256 + rrddimvar_update_variables_unsafe(rs);
257 }
258 + dfe_done(rs);
259 }
260
193 -void rrddimvar_free(RRDDIMVAR *rs) {
194 - RRDDIM *rd = rs->rrddim;
195 - debug(D_VARIABLES, "RRDDIMSET free for chart id '%s' name '%s', dimension id '%s', name '%s', prefix='%s', suffix='%s'", rrdset_id(rd->rrdset), rrdset_name(rd->rrdset), rrddim_id(rd), rrddim_name(rd), string2str(rs->prefix), string2str(rs->suffix));
196 -
197 - rrddimvar_free_variables(rs);
261 +void rrddimvar_delete_all(RRDDIM *rd) {
262 + RRDSET *st = rd->rrdset;
263
199 - DOUBLE_LINKED_LIST_REMOVE_UNSAFE(rd->variables, rs, prev, next);
264 + debug(D_VARIABLES, "RRDDIMVAR delete for chart id '%s' name '%s', dimension id '%s', name '%s'", rrdset_id(st), rrdset_name(st), rrddim_id(rd), rrddim_name(rd));
265
201 - string_freez(rs->prefix);
202 - string_freez(rs->suffix);
203 - freez(rs);
266 + RRDDIMVAR *rs;
267 + dfe_start_write(st->rrddimvar_root_index, rs) {
268 + if(unlikely(rs->rrddim == rd))
269 + dictionary_del(st->rrddimvar_root_index, rs_dfe.name);
270 + }
271 + dfe_done(rs);
272 }
database/rrddimvar.h
+4 -40
@@ -10,48 +10,12 @@
10 // calculated / processed by the normal data collection process
11 // This means, there will be no speed penalty for using
12 // these variables
13 -struct rrddimvar {
14 - STRING *prefix;
15 - STRING *suffix;
16 -
17 - STRING *key_id; // dimension id
18 - STRING *key_name; // dimension name
19 - STRING *key_contextid; // context + dimension id
20 - STRING *key_contextname; // context + dimension name
21 - STRING *key_fullidid; // chart type.chart id + dimension id
22 - STRING *key_fullidname; // chart type.chart id + dimension name
23 - STRING *key_fullnameid; // chart type.chart name + dimension id
24 - STRING *key_fullnamename; // chart type.chart name + dimension name
25 -
26 - RRDVAR_TYPE type;
27 - void *value;
28 -
29 - RRDVAR_OPTIONS options;
30 -
31 - RRDVAR *var_local_id;
32 - RRDVAR *var_local_name;
33 -
34 - RRDVAR *var_family_id;
35 - RRDVAR *var_family_name;
36 - RRDVAR *var_family_contextid;
37 - RRDVAR *var_family_contextname;
38 -
39 - RRDVAR *var_host_chartidid;
40 - RRDVAR *var_host_chartidname;
41 - RRDVAR *var_host_chartnameid;
42 - RRDVAR *var_host_chartnamename;
43 -
44 - struct rrddim *rrddim;
45 -
46 - struct rrddimvar *next;
47 - struct rrddimvar *prev;
48 -};
49 -
13
14 extern void rrddimvar_rename_all(RRDDIM *rd);
52 -extern RRDDIMVAR *rrddimvar_create(RRDDIM *rd, RRDVAR_TYPE type, const char *prefix, const char *suffix, void *value, RRDVAR_OPTIONS options);
53 -extern void rrddimvar_free(RRDDIMVAR *rs);
54 -
15 +extern void rrddimvar_add_and_leave_released(RRDDIM *rd, RRDVAR_TYPE type, const char *prefix, const char *suffix, void *value, RRDVAR_FLAGS flags);
16 +extern void rrddimvar_delete_all(RRDDIM *rd);
17
18 +extern void rrddimvar_index_init(RRDSET *st);
19 +extern void rrddimvar_index_destroy(RRDSET *st);
20
21 #endif //NETDATA_RRDDIMVAR_H
database/rrdfamily.c
+46 -41
@@ -3,61 +3,66 @@
3 #define NETDATA_RRD_INTERNALS
4 #include "rrd.h"
5
6 +typedef struct rrdfamily {
7 + STRING *family;
8 + DICTIONARY *rrdvars;
9 +} RRDFAMILY;
10 +
11 // ----------------------------------------------------------------------------
12 // RRDFAMILY index
13
9 -static inline RRDFAMILY *rrdfamily_index_add(RRDHOST *host, RRDFAMILY *rc) {
10 - return dictionary_set(host->rrdfamily_root_index, string2str(rc->family), rc, sizeof(RRDFAMILY));
11 -}
14 +struct rrdfamily_constructor {
15 + const char *family;
16 +};
17
13 -static inline RRDFAMILY *rrdfamily_index_del(RRDHOST *host, RRDFAMILY *rc) {
14 - dictionary_del(host->rrdfamily_root_index, string2str(rc->family));
15 - return rc;
18 +static void rrdfamily_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdfamily, void *constructor_data) {
19 + RRDFAMILY *rf = rrdfamily;
20 + struct rrdfamily_constructor *ctr = constructor_data;
21 +
22 + rf->family = string_strdupz(ctr->family);
23 + rf->rrdvars = rrdvariables_create();
24 }
25
18 -static inline RRDFAMILY *rrdfamily_index_find(RRDHOST *host, const char *id) {
19 - return dictionary_get(host->rrdfamily_root_index, id);
26 +static void rrdfamily_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdfamily, void *rrdhost __maybe_unused) {
27 + RRDFAMILY *rf = rrdfamily;
28 + string_freez(rf->family);
29 + rrdvariables_destroy(rf->rrdvars);
30 + rf->family = NULL;
31 + rf->rrdvars = NULL;
32 }
33
22 -// ----------------------------------------------------------------------------
23 -// RRDFAMILY management
34 +void rrdfamily_index_init(RRDHOST *host) {
35 + if(!host->rrdfamily_root_index) {
36 + host->rrdfamily_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
37
25 -RRDFAMILY *rrdfamily_create(RRDHOST *host, const char *id) {
26 - RRDFAMILY *rc = rrdfamily_index_find(host, id);
27 - if(!rc) {
28 - rc = callocz(1, sizeof(RRDFAMILY));
38 + dictionary_register_insert_callback(host->rrdfamily_root_index, rrdfamily_insert_callback, NULL);
39 + dictionary_register_delete_callback(host->rrdfamily_root_index, rrdfamily_delete_callback, host);
40 + }
41 +}
42
30 - rc->family = string_strdupz(id);
43 +void rrdfamily_index_destroy(RRDHOST *host) {
44 + dictionary_destroy(host->rrdfamily_root_index);
45 + host->rrdfamily_root_index = NULL;
46 +}
47
32 - rc->rrdvar_root_index = dictionary_create(
33 - DICTIONARY_FLAG_NAME_LINK_DONT_CLONE
34 - |DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE
35 - |DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
36 - );
48
38 - RRDFAMILY *ret = rrdfamily_index_add(host, rc);
39 - if(ret != rc)
40 - error("RRDFAMILY: INTERNAL ERROR: Expected to INSERT RRDFAMILY '%s' into index, but inserted '%s'.", string2str(rc->family), (ret)?string2str(ret->family):"NONE");
41 - }
49 +// ----------------------------------------------------------------------------
50 +// RRDFAMILY management
51
43 - rc->use_count++;
44 - return rc;
52 +const RRDFAMILY_ACQUIRED *rrdfamily_add_and_acquire(RRDHOST *host, const char *id) {
53 + struct rrdfamily_constructor tmp = {
54 + .family = id,
55 + };
56 + return (const RRDFAMILY_ACQUIRED *)dictionary_set_and_acquire_item_advanced(host->rrdfamily_root_index, id, -1, NULL, sizeof(RRDFAMILY), &tmp);
57 }
58
47 -void rrdfamily_free(RRDHOST *host, RRDFAMILY *rc) {
48 - rc->use_count--;
49 - if(!rc->use_count) {
50 - RRDFAMILY *ret = rrdfamily_index_del(host, rc);
51 - if(ret != rc)
52 - error("RRDFAMILY: INTERNAL ERROR: Expected to DELETE RRDFAMILY '%s' from index, but deleted '%s'.", string2str(rc->family), (ret)?string2str(ret->family):"NONE");
53 - else {
54 - debug(D_RRD_CALLS, "RRDFAMILY: Cleaning up remaining family variables for host '%s', family '%s'", rrdhost_hostname(host), string2str(rc->family));
55 - rrdvar_free_remaining_variables(host, rc->rrdvar_root_index);
56 -
57 - dictionary_destroy(rc->rrdvar_root_index);
58 - string_freez(rc->family);
59 - freez(rc);
60 - }
61 - }
59 +void rrdfamily_release(RRDHOST *host, const RRDFAMILY_ACQUIRED *rfa) {
60 + if(unlikely(!rfa)) return;
61 + dictionary_acquired_item_release(host->rrdfamily_root_index, (const DICTIONARY_ITEM *)rfa);
62 }
63
64 +DICTIONARY *rrdfamily_rrdvars_dict(const RRDFAMILY_ACQUIRED *rfa) {
65 + if(unlikely(!rfa)) return NULL;
66 + RRDFAMILY *rf = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rfa);
67 + return(rf->rrdvars);
68 +}
database/rrdhost.c
+170 -374
@@ -51,18 +51,12 @@ static DICTIONARY *rrdhost_root_index_hostname = NULL;
51 static inline void rrdhost_init() {
52 if(unlikely(!rrdhost_root_index)) {
53 rrdhost_root_index = dictionary_create(
54 - DICTIONARY_FLAG_NAME_LINK_DONT_CLONE
55 - | DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE
56 - | DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
57 - );
54 + DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
55 }
56
57 if(unlikely(!rrdhost_root_index_hostname)) {
58 rrdhost_root_index_hostname = dictionary_create(
62 - DICTIONARY_FLAG_NAME_LINK_DONT_CLONE
63 - | DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE
64 - | DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
65 - );
59 + DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
60 }
61 }
62
@@ -70,7 +64,7 @@ static inline void rrdhost_init() {
64 // RRDHOST index by UUID
65
66 inline long rrdhost_hosts_available(void) {
73 - return dictionary_stats_entries(rrdhost_root_index);
67 + return dictionary_entries(rrdhost_root_index);
68 }
69
70 inline RRDHOST *rrdhost_find_by_guid(const char *guid) {
@@ -91,7 +85,7 @@ static inline RRDHOST *rrdhost_index_add_by_guid(RRDHOST *host) {
85
86 static void rrdhost_index_del_by_guid(RRDHOST *host) {
87 if(rrdhost_flag_check(host, RRDHOST_FLAG_INDEXED_MACHINE_GUID)) {
94 - if(dictionary_del(rrdhost_root_index, host->machine_guid) != 0)
88 + if(!dictionary_del(rrdhost_root_index, host->machine_guid))
89 error("RRDHOST: %s() failed to delete machine guid '%s' from index", __FUNCTION__, host->machine_guid);
90
91 rrdhost_flag_clear(host, RRDHOST_FLAG_INDEXED_MACHINE_GUID);
@@ -126,7 +120,7 @@ static inline void rrdhost_index_del_hostname(RRDHOST *host) {
120 if(unlikely(!host->hostname)) return;
121
122 if(rrdhost_flag_check(host, RRDHOST_FLAG_INDEXED_HOSTNAME)) {
129 - if(dictionary_del(rrdhost_root_index_hostname, rrdhost_hostname(host)) != 0)
123 + if(!dictionary_del(rrdhost_root_index_hostname, rrdhost_hostname(host)))
124 error("RRDHOST: %s() failed to delete hostname '%s' from index", __FUNCTION__, rrdhost_hostname(host));
125
126 rrdhost_flag_clear(host, RRDHOST_FLAG_INDEXED_HOSTNAME);
@@ -206,6 +200,114 @@ void set_host_properties(RRDHOST *host, int update_every, RRD_MEMORY_MODE memory
200 // ----------------------------------------------------------------------------
201 // RRDHOST - add a host
202
203 +static void rrdhost_initialize_rrdpush(RRDHOST *host,
204 + unsigned int rrdpush_enabled,
205 + char *rrdpush_destination,
206 + char *rrdpush_api_key,
207 + char *rrdpush_send_charts_matching
208 +) {
209 + if(rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_RRDPUSH)) return;
210 + rrdhost_flag_set(host, RRDHOST_FLAG_INITIALIZED_RRDPUSH);
211 +
212 + sender_init(host);
213 + netdata_mutex_init(&host->receiver_lock);
214 +
215 + host->rrdpush_send_enabled = (rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) ? 1 : 0;
216 + host->rrdpush_send_destination = (host->rrdpush_send_enabled)?strdupz(rrdpush_destination):NULL;
217 +
218 + if (host->rrdpush_send_destination)
219 + host->destinations = destinations_init(host->rrdpush_send_destination);
220 +
221 + host->rrdpush_send_api_key = (host->rrdpush_send_enabled)?strdupz(rrdpush_api_key):NULL;
222 + host->rrdpush_send_charts_matching = simple_pattern_create(rrdpush_send_charts_matching, NULL, SIMPLE_PATTERN_EXACT);
223 +
224 + host->rrdpush_sender_pipe[0] = -1;
225 + host->rrdpush_sender_pipe[1] = -1;
226 + host->rrdpush_sender_socket = -1;
227 +
228 + //host->stream_version = STREAMING_PROTOCOL_CURRENT_VERSION; Unused?
229 +#ifdef ENABLE_HTTPS
230 + host->ssl.conn = NULL;
231 + host->ssl.flags = NETDATA_SSL_START;
232 + host->stream_ssl.conn = NULL;
233 + host->stream_ssl.flags = NETDATA_SSL_START;
234 +#endif
235 +}
236 +
237 +static void rrdhost_initialize_health(RRDHOST *host,
238 + int is_localhost
239 + ) {
240 + if(!host->health_enabled || rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH)) return;
241 + rrdhost_flag_set(host, RRDHOST_FLAG_INITIALIZED_HEALTH);
242 +
243 + rrdfamily_index_init(host);
244 + rrdcalctemplate_index_init(host);
245 + rrdcalc_rrdhost_index_init(host);
246 +
247 + host->health_default_warn_repeat_every = config_get_duration(CONFIG_SECTION_HEALTH, "default repeat warning", "never");
248 + host->health_default_crit_repeat_every = config_get_duration(CONFIG_SECTION_HEALTH, "default repeat critical", "never");
249 +
250 + host->health_log.next_log_id = 1;
251 + host->health_log.next_alarm_id = 1;
252 + host->health_log.max = 1000;
253 + host->health_log.next_log_id = (uint32_t)now_realtime_sec();
254 + host->health_log.next_alarm_id = 0;
255 +
256 + long n = config_get_number(CONFIG_SECTION_HEALTH, "in memory max health log entries", host->health_log.max);
257 + if(n < 10) {
258 + error("Host '%s': health configuration has invalid max log entries %ld. Using default %u", rrdhost_hostname(host), n, host->health_log.max);
259 + config_set_number(CONFIG_SECTION_HEALTH, "in memory max health log entries", (long)host->health_log.max);
260 + }
261 + else
262 + host->health_log.max = (unsigned int)n;
263 +
264 + netdata_rwlock_init(&host->health_log.alarm_log_rwlock);
265 +
266 + char filename[FILENAME_MAX + 1];
267 +
268 + if(!is_localhost) {
269 + int r = mkdir(host->varlib_dir, 0775);
270 + if (r != 0 && errno != EEXIST)
271 + error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), host->varlib_dir);
272 + }
273 +
274 + {
275 + snprintfz(filename, FILENAME_MAX, "%s/health", host->varlib_dir);
276 + int r = mkdir(filename, 0775);
277 + if(r != 0 && errno != EEXIST)
278 + error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), filename);
279 + }
280 +
281 + snprintfz(filename, FILENAME_MAX, "%s/health/health-log.db", host->varlib_dir);
282 + host->health_log_filename = strdupz(filename);
283 +
284 + snprintfz(filename, FILENAME_MAX, "%s/alarm-notify.sh", netdata_configured_primary_plugins_dir);
285 + host->health_default_exec = string_strdupz(config_get(CONFIG_SECTION_HEALTH, "script to execute on alarm", filename));
286 + host->health_default_recipient = string_strdupz("root");
287 +
288 + // ------------------------------------------------------------------------
289 + // load health configuration
290 +
291 + health_readdir(host, health_user_config_dir(), health_stock_config_dir(), NULL);
292 +
293 + if (!file_is_migrated(host->health_log_filename)) {
294 + int rc = sql_create_health_log_table(host);
295 + if (unlikely(rc)) {
296 + error_report("Failed to create health log table in the database");
297 + health_alarm_log_load(host);
298 + health_alarm_log_open(host);
299 + }
300 + else {
301 + health_alarm_log_load(host);
302 + add_migrated_file(host->health_log_filename, 0);
303 + }
304 + } else {
305 + sql_create_health_log_table(host);
306 + sql_health_alarm_log_load(host);
307 + }
308 +}
309 +
310 +
311 RRDHOST *rrdhost_create(const char *hostname,
312 const char *registry_hostname,
313 const char *guid,
@@ -251,40 +353,18 @@ RRDHOST *rrdhost_create(const char *hostname,
353 host->rrd_history_entries = align_entries_to_pagesize(memory_mode, entries);
354 host->health_enabled = ((memory_mode == RRD_MEMORY_MODE_NONE)) ? 0 : health_enabled;
355
254 - sender_init(host);
255 - netdata_mutex_init(&host->receiver_lock);
256 -
257 - host->rrdpush_send_enabled = (rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) ? 1 : 0;
258 - host->rrdpush_send_destination = (host->rrdpush_send_enabled)?strdupz(rrdpush_destination):NULL;
259 - if (host->rrdpush_send_destination)
260 - host->destinations = destinations_init(host->rrdpush_send_destination);
261 - host->rrdpush_send_api_key = (host->rrdpush_send_enabled)?strdupz(rrdpush_api_key):NULL;
262 - host->rrdpush_send_charts_matching = simple_pattern_create(rrdpush_send_charts_matching, NULL, SIMPLE_PATTERN_EXACT);
263 -
264 - host->rrdpush_sender_pipe[0] = -1;
265 - host->rrdpush_sender_pipe[1] = -1;
266 - host->rrdpush_sender_socket = -1;
267 -
268 - //host->stream_version = STREAMING_PROTOCOL_CURRENT_VERSION; Unused?
269 -#ifdef ENABLE_HTTPS
270 - host->ssl.conn = NULL;
271 - host->ssl.flags = NETDATA_SSL_START;
272 - host->stream_ssl.conn = NULL;
273 - host->stream_ssl.flags = NETDATA_SSL_START;
274 -#endif
275 -
276 - netdata_rwlock_init(&host->rrdhost_rwlock);
277 - if (likely(!archived))
356 + if (likely(!archived)) {
357 host->rrdlabels = rrdlabels_create();
358 + rrdhost_initialize_rrdpush(
359 + host, rrdpush_enabled, rrdpush_destination, rrdpush_api_key, rrdpush_send_charts_matching);
360 + }
361
362 + netdata_rwlock_init(&host->rrdhost_rwlock);
363 netdata_mutex_init(&host->aclk_state_lock);
364
365 host->system_info = system_info;
366
284 - host->rrdset_root_index = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
285 - host->rrdset_root_index_name = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
286 - host->rrdfamily_root_index = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
287 - host->rrdvar_root_index = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
367 + rrdset_index_init(host);
368
369 if(config_get_boolean(CONFIG_SECTION_DB, "delete obsolete charts files", 1))
370 rrdhost_flag_set(host, RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS);
@@ -292,41 +372,17 @@ RRDHOST *rrdhost_create(const char *hostname,
372 if(config_get_boolean(CONFIG_SECTION_DB, "delete orphan hosts files", 1) && !is_localhost)
373 rrdhost_flag_set(host, RRDHOST_FLAG_DELETE_ORPHAN_HOST);
374
295 - host->health_default_warn_repeat_every = config_get_duration(CONFIG_SECTION_HEALTH, "default repeat warning", "never");
296 - host->health_default_crit_repeat_every = config_get_duration(CONFIG_SECTION_HEALTH, "default repeat critical", "never");
297 -
298 - // ------------------------------------------------------------------------
299 - // initialize health variables
300 -
301 - host->health_log.next_log_id = 1;
302 - host->health_log.next_alarm_id = 1;
303 - host->health_log.max = 1000;
304 - host->health_log.next_log_id = (uint32_t)now_realtime_sec();
305 - host->health_log.next_alarm_id = 0;
306 -
307 - long n = config_get_number(CONFIG_SECTION_HEALTH, "in memory max health log entries", host->health_log.max);
308 - if(n < 10) {
309 - error("Host '%s': health configuration has invalid max log entries %ld. Using default %u", rrdhost_hostname(host), n, host->health_log.max);
310 - config_set_number(CONFIG_SECTION_HEALTH, "in memory max health log entries", (long)host->health_log.max);
311 - }
312 - else
313 - host->health_log.max = (unsigned int)n;
314 -
315 - netdata_rwlock_init(&host->health_log.alarm_log_rwlock);
316 -
375 char filename[FILENAME_MAX + 1];
318 -
376 if(is_localhost) {
320 -
377 host->cache_dir = strdupz(netdata_configured_cache_dir);
378 host->varlib_dir = strdupz(netdata_configured_varlib_dir);
323 -
379 }
380 else {
381 // this is not localhost - append our GUID to localhost path
382 if (is_in_multihost) { // don't append to cache dir in multihost
383 host->cache_dir = strdupz(netdata_configured_cache_dir);
329 - } else {
384 + }
385 + else {
386 snprintfz(filename, FILENAME_MAX, "%s/%s", netdata_configured_cache_dir, host->machine_guid);
387 host->cache_dir = strdupz(filename);
388 }
@@ -340,38 +396,11 @@ RRDHOST *rrdhost_create(const char *hostname,
396
397 snprintfz(filename, FILENAME_MAX, "%s/%s", netdata_configured_varlib_dir, host->machine_guid);
398 host->varlib_dir = strdupz(filename);
343 -
344 - if(host->health_enabled) {
345 - int r = mkdir(host->varlib_dir, 0775);
346 - if(r != 0 && errno != EEXIST)
347 - error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), host->varlib_dir);
348 - }
349 -
350 - }
351 -
352 - if(host->health_enabled) {
353 - snprintfz(filename, FILENAME_MAX, "%s/health", host->varlib_dir);
354 - int r = mkdir(filename, 0775);
355 - if(r != 0 && errno != EEXIST)
356 - error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), filename);
399 }
400
359 - snprintfz(filename, FILENAME_MAX, "%s/health/health-log.db", host->varlib_dir);
360 - host->health_log_filename = strdupz(filename);
361 -
362 - snprintfz(filename, FILENAME_MAX, "%s/alarm-notify.sh", netdata_configured_primary_plugins_dir);
363 - host->health_default_exec = string_strdupz(config_get(CONFIG_SECTION_HEALTH, "script to execute on alarm", filename));
364 - host->health_default_recipient = string_strdupz("root");
365 -
366 -
367 - // ------------------------------------------------------------------------
368 - // load health configuration
369 -
370 - if(host->health_enabled) {
371 - rrdhost_wrlock(host);
372 - health_readdir(host, health_user_config_dir(), health_stock_config_dir(), NULL);
373 - rrdhost_unlock(host);
374 - }
401 + // this is also needed for custom host variables - not only health
402 + if(!host->rrdvars)
403 + host->rrdvars = rrdvariables_create();
404
405 RRDHOST *t = rrdhost_index_add_by_guid(host);
406 if(t != host) {
@@ -382,33 +411,20 @@ RRDHOST *rrdhost_create(const char *hostname,
411
412 if (likely(!uuid_parse(host->machine_guid, host->host_uuid))) {
413 int rc;
385 - if (!archived) {
414 +
415 + if(!archived) {
416 rc = sql_store_host_info(host);
417 if (unlikely(rc))
418 error_report("Failed to store machine GUID to the database");
419 }
420 +
421 sql_load_node_id(host);
391 - if (host->health_enabled) {
392 - if (!file_is_migrated(host->health_log_filename)) {
393 - rc = sql_create_health_log_table(host);
394 - if (unlikely(rc)) {
395 - error_report("Failed to create health log table in the database");
396 - health_alarm_log_load(host);
397 - health_alarm_log_open(host);
398 - }
399 - else {
400 - health_alarm_log_load(host);
401 - add_migrated_file(host->health_log_filename, 0);
402 - }
403 - } else {
404 - sql_create_health_log_table(host);
405 - sql_health_alarm_log_load(host);
406 - }
407 - }
422 }
423 else
424 error_report("Host machine GUID %s is not valid", host->machine_guid);
425
426 + rrdhost_initialize_health(host, is_localhost);
427 +
428 if (host->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
429 #ifdef ENABLE_DBENGINE
430 char dbenginepath[FILENAME_MAX + 1];
@@ -530,6 +546,8 @@ RRDHOST *rrdhost_create(const char *hostname,
546 ml_new_host(host);
547 else
548 rrdhost_flag_set(host, RRDHOST_FLAG_ARCHIVED);
549 +
550 +
551 return host;
552 }
553
@@ -605,50 +623,24 @@ void rrdhost_update(RRDHOST *host
623 // update host tags
624 rrdhost_init_tags(host, tags);
625
626 + if(!host->rrdvars)
627 + host->rrdvars = rrdvariables_create();
628 +
629 if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)) {
630 rrdhost_flag_clear(host, RRDHOST_FLAG_ARCHIVED);
631
611 - host->rrdpush_send_enabled = (rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) ? 1 : 0;
612 - host->rrdpush_send_destination = (host->rrdpush_send_enabled)?strdupz(rrdpush_destination):NULL;
613 - if (host->rrdpush_send_destination)
614 - host->destinations = destinations_init(host->rrdpush_send_destination);
615 - host->rrdpush_send_api_key = (host->rrdpush_send_enabled)?strdupz(rrdpush_api_key):NULL;
616 - host->rrdpush_send_charts_matching = simple_pattern_create(rrdpush_send_charts_matching, NULL, SIMPLE_PATTERN_EXACT);
617 -
618 - if(host->health_enabled) {
619 - int r;
620 - char filename[FILENAME_MAX + 1];
621 -
622 - if (host != localhost) {
623 - r = mkdir(host->varlib_dir, 0775);
624 - if (r != 0 && errno != EEXIST)
625 - error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), host->varlib_dir);
626 - }
627 - snprintfz(filename, FILENAME_MAX, "%s/health", host->varlib_dir);
628 - r = mkdir(filename, 0775);
629 - if(r != 0 && errno != EEXIST)
630 - error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), filename);
631 -
632 - rrdhost_wrlock(host);
633 - health_readdir(host, health_user_config_dir(), health_stock_config_dir(), NULL);
634 - rrdhost_unlock(host);
635 -
636 - if (!file_is_migrated(host->health_log_filename)) {
637 - int rc = sql_create_health_log_table(host);
638 - if (unlikely(rc)) {
639 - error_report("Failed to create health log table in the database");
640 -
641 - health_alarm_log_load(host);
642 - health_alarm_log_open(host);
643 - } else {
644 - health_alarm_log_load(host);
645 - add_migrated_file(host->health_log_filename, 0);
646 - }
647 - } else {
648 - sql_create_health_log_table(host);
649 - sql_health_alarm_log_load(host);
650 - }
651 - }
632 + if(!host->rrdlabels)
633 + host->rrdlabels = rrdlabels_create();
634 +
635 + rrdhost_initialize_rrdpush(host,
636 + rrdpush_enabled,
637 + rrdpush_destination,
638 + rrdpush_api_key,
639 + rrdpush_send_charts_matching);
640 +
641 + rrdhost_initialize_health(host,
642 + host == localhost);
643 +
644 rrd_hosts_available++;
645 ml_new_host(host);
646 rrdhost_load_rrdcontext_data(host);
@@ -762,32 +754,6 @@ inline int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, tim
754 return 0;
755 }
756
765 -void rrdhost_cleanup_orphan_hosts_nolock(RRDHOST *protected_host) {
766 - time_t now = now_realtime_sec();
767 -
768 - RRDHOST *host;
769 -
770 -restart_after_removal:
771 - rrdhost_foreach_write(host) {
772 - if(rrdhost_should_be_removed(host, protected_host, now)) {
773 - info("Host '%s' with machine guid '%s' is obsolete - cleaning up.", rrdhost_hostname(host), host->machine_guid);
774 -
775 - if (rrdhost_flag_check(host, RRDHOST_FLAG_DELETE_ORPHAN_HOST)
776 -#ifdef ENABLE_DBENGINE
777 - /* don't delete multi-host DB host files */
778 - && !(host->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE && is_storage_engine_shared(host->storage_instance[0]))
779 -#endif
780 - )
781 - rrdhost_delete_charts(host);
782 - else
783 - rrdhost_save_charts(host);
784 -
785 - rrdhost_free(host, 0);
786 - goto restart_after_removal;
787 - }
788 - }
789 -}
790 -
757 // ----------------------------------------------------------------------------
758 // RRDHOST global / startup initialization
759
@@ -1091,12 +1057,21 @@ void rrdhost_free(RRDHOST *host, bool force) {
1057
1058 // ------------------------------------------------------------------------
1059 // clean up streaming
1060 +
1061 stop_streaming_sender(host);
1062
1063 if (netdata_exit || force)
1064 stop_streaming_receiver(host);
1065
1066 +
1067 + // ------------------------------------------------------------------------
1068 + // clean up alarms
1069 +
1070 + rrdcalc_delete_all(host);
1071 +
1072 +
1073 rrdhost_wrlock(host); // lock this RRDHOST
1074 +
1075 // ------------------------------------------------------------------------
1076 // release its children resources
1077
@@ -1109,27 +1084,13 @@ void rrdhost_free(RRDHOST *host, bool force) {
1084 }
1085 #endif
1086
1112 - while(host->rrdset_root)
1113 - rrdset_free(host->rrdset_root);
1087 + // delete all the RRDSETs of the host
1088 + rrdset_index_destroy(host);
1089 + rrdcalc_rrdhost_index_destroy(host);
1090 + rrdcalctemplate_index_destroy(host);
1091
1092 freez(host->exporting_flags);
1093
1117 - while(host->host_alarms)
1118 - rrdcalc_unlink_and_free(host, host->host_alarms);
1119 -
1120 - RRDCALC *rc,*nc;
1121 - for(rc = host->alarms_with_foreach; rc ; rc = nc) {
1122 - nc = rc->next;
1123 - rrdcalc_free(rc);
1124 - }
1125 - host->alarms_with_foreach = NULL;
1126 -
1127 - while(host->alarms_templates)
1128 - rrdcalctemplate_unlink_and_free(host, host->alarms_templates);
1129 -
1130 - debug(D_RRD_CALLS, "RRDHOST: Cleaning up remaining host variables for host '%s'", rrdhost_hostname(host));
1131 - rrdvar_free_remaining_variables(host, host->rrdvar_root_index);
1132 -
1094 health_alarm_log_free(host);
1095
1096 #ifdef ENABLE_DBENGINE
@@ -1208,10 +1169,8 @@ void rrdhost_free(RRDHOST *host, bool force) {
1169 netdata_rwlock_destroy(&host->rrdhost_rwlock);
1170 freez(host->node_id);
1171
1211 - dictionary_destroy(host->rrdset_root_index);
1212 - dictionary_destroy(host->rrdset_root_index_name);
1213 - dictionary_destroy(host->rrdfamily_root_index);
1214 - dictionary_destroy(host->rrdvar_root_index);
1172 + rrdfamily_index_destroy(host);
1173 + rrdvariables_destroy(host->rrdvars);
1174
1175 rrdhost_destroy_rrdcontexts(host);
1176
@@ -1249,15 +1208,10 @@ void rrdhost_save_charts(RRDHOST *host) {
1208
1209 // we get a write lock
1210 // to ensure only one thread is saving the database
1252 - rrdhost_wrlock(host);
1253 -
1211 rrdset_foreach_write(st, host) {
1255 - rrdset_rdlock(st);
1212 rrdset_save(st);
1257 - rrdset_unlock(st);
1213 }
1259 -
1260 - rrdhost_unlock(host);
1214 + rrdset_foreach_done(st);
1215 }
1216
1217 static void rrdhost_load_auto_labels(void) {
@@ -1411,17 +1365,12 @@ void rrdhost_delete_charts(RRDHOST *host) {
1365
1366 // we get a write lock
1367 // to ensure only one thread is saving the database
1414 - rrdhost_wrlock(host);
1415 -
1368 rrdset_foreach_write(st, host) {
1417 - rrdset_rdlock(st);
1418 - rrdset_delete_files(st);
1419 - rrdset_unlock(st);
1369 + rrdset_delete_files(st);
1370 }
1371 + rrdset_foreach_done(st);
1372
1373 recursively_delete_dir(host->cache_dir, "left over host");
1423 -
1424 - rrdhost_unlock(host);
1374 }
1375
1376 // ----------------------------------------------------------------------------
@@ -1437,22 +1386,19 @@ void rrdhost_cleanup_charts(RRDHOST *host) {
1386
1387 // we get a write lock
1388 // to ensure only one thread is saving the database
1440 - rrdhost_wrlock(host);
1441 -
1389 rrdset_foreach_write(st, host) {
1443 - rrdset_rdlock(st);
1390
1391 if(rrdhost_delete_obsolete_charts && rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE))
1392 rrdset_delete_files(st);
1393 +
1394 else if(rrdhost_delete_obsolete_charts && rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS))
1395 rrdset_delete_obsolete_dimensions(st);
1396 +
1397 else
1398 rrdset_save(st);
1399
1452 - rrdset_unlock(st);
1400 }
1454 -
1455 - rrdhost_unlock(host);
1401 + rrdset_foreach_done(st);
1402 }
1403
1404
@@ -1496,157 +1442,6 @@ void rrdhost_cleanup_all(void) {
1442 }
1443
1444
1499 -// ----------------------------------------------------------------------------
1500 -// RRDHOST - save or delete all the host charts from disk
1501 -
1502 -void rrdhost_cleanup_obsolete_charts(RRDHOST *host) {
1503 - time_t now = now_realtime_sec();
1504 -
1505 - RRDSET *st;
1506 -
1507 - uint32_t rrdhost_delete_obsolete_charts = rrdhost_flag_check(host, RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS);
1508 -
1509 -restart_after_removal:
1510 - rrdset_foreach_write(st, host) {
1511 - if(unlikely(rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE)
1512 - && st->last_accessed_time + rrdset_free_obsolete_time < now
1513 - && st->last_updated.tv_sec + rrdset_free_obsolete_time < now
1514 - && st->last_collected_time.tv_sec + rrdset_free_obsolete_time < now
1515 - )) {
1516 - st->rrdhost->obsolete_charts_count--;
1517 -#ifdef ENABLE_DBENGINE
1518 - if(st->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
1519 - RRDDIM *rd, *last;
1520 -
1521 - rrdset_flag_set(st, RRDSET_FLAG_ARCHIVED);
1522 - while (st->variables) rrdsetvar_free(st->variables);
1523 - while (st->alarms) rrdsetcalc_unlink(st->alarms);
1524 - rrdset_wrlock(st);
1525 - for (rd = st->dimensions, last = NULL ; likely(rd) ; ) {
1526 - if (rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)) {
1527 - last = rd;
1528 - rd = rd->next;
1529 - continue;
1530 - }
1531 -
1532 - if (rrddim_flag_check(rd, RRDDIM_FLAG_ACLK)) {
1533 - last = rd;
1534 - rd = rd->next;
1535 - continue;
1536 - }
1537 - rrddim_flag_set(rd, RRDDIM_FLAG_ARCHIVED);
1538 - while (rd->variables)
1539 - rrddimvar_free(rd->variables);
1540 -
1541 - if (rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)) {
1542 - rrddim_flag_clear(rd, RRDDIM_FLAG_OBSOLETE);
1543 -
1544 - /* only a collector can mark a chart as obsolete, so we must remove the reference */
1545 -
1546 - size_t tiers_available = 0, tiers_said_yes = 0;
1547 - for(int tier = 0; tier < storage_tiers ;tier++) {
1548 - if(rd->tiers[tier]) {
1549 - tiers_available++;
1550 -
1551 - if(rd->tiers[tier]->collect_ops.finalize(rd->tiers[tier]->db_collection_handle))
1552 - tiers_said_yes++;
1553 -
1554 - rd->tiers[tier]->db_collection_handle = NULL;
1555 - }
1556 - }
1557 -
1558 - if (tiers_available == tiers_said_yes && tiers_said_yes) {
1559 - /* This metric has no data and no references */
1560 - delete_dimension_uuid(&rd->metric_uuid);
1561 - rrddim_free(st, rd);
1562 - if (unlikely(!last)) {
1563 - rd = st->dimensions;
1564 - }
1565 - else {
1566 - rd = last->next;
1567 - }
1568 - continue;
1569 - }
1570 -#ifdef ENABLE_ACLK
1571 - else
1572 - queue_dimension_to_aclk(rd, rd->last_collected_time.tv_sec);
1573 -#endif
1574 - }
1575 - last = rd;
1576 - rd = rd->next;
1577 - }
1578 - rrdset_unlock(st);
1579 -
1580 - debug(D_RRD_CALLS, "RRDSET: Cleaning up remaining chart variables for host '%s', chart '%s'", rrdhost_hostname(host), rrdset_id(st));
1581 - rrdvar_free_remaining_variables(host, st->rrdvar_root_index);
1582 -
1583 - rrdset_flag_clear(st, RRDSET_FLAG_OBSOLETE);
1584 -
1585 - if (st->dimensions) {
1586 - /* If the chart still has dimensions don't delete it from the metadata log */
1587 - continue;
1588 - }
1589 - }
1590 -#endif
1591 - rrdset_rdlock(st);
1592 -
1593 - if(rrdhost_delete_obsolete_charts)
1594 - rrdset_delete_files(st);
1595 - else
1596 - rrdset_save(st);
1597 -
1598 - rrdset_unlock(st);
1599 -
1600 - rrdset_free(st);
1601 - goto restart_after_removal;
1602 - }
1603 -#ifdef ENABLE_ACLK
1604 - else
1605 - sql_check_chart_liveness(st);
1606 -#endif
1607 - }
1608 -}
1609 -
1610 -void rrdset_check_obsoletion(RRDHOST *host)
1611 -{
1612 - RRDSET *st;
1613 - time_t last_entry_t;
1614 - rrdset_foreach_read(st, host) {
1615 - last_entry_t = rrdset_last_entry_t(st);
1616 - if (last_entry_t && last_entry_t < host->senders_connect_time) {
1617 - rrdset_is_obsolete(st);
1618 - }
1619 - }
1620 -}
1621 -
1622 -void rrd_cleanup_obsolete_charts()
1623 -{
1624 - rrd_rdlock();
1625 -
1626 - RRDHOST *host;
1627 - rrdhost_foreach_read(host)
1628 - {
1629 - if (host->obsolete_charts_count) {
1630 - rrdhost_wrlock(host);
1631 - rrdhost_cleanup_obsolete_charts(host);
1632 - rrdhost_unlock(host);
1633 - }
1634 -
1635 - if ( host != localhost &&
1636 - host->trigger_chart_obsoletion_check &&
1637 - ((host->senders_last_chart_command &&
1638 - host->senders_last_chart_command + host->health_delay_up_to < now_realtime_sec())
1639 - || (host->senders_connect_time + 300 < now_realtime_sec())) ) {
1640 - rrdhost_rdlock(host);
1641 - rrdset_check_obsoletion(host);
1642 - rrdhost_unlock(host);
1643 - host->trigger_chart_obsoletion_check = 0;
1644 - }
1645 - }
1646 -
1647 - rrd_unlock();
1648 -}
1649 -
1445 // ----------------------------------------------------------------------------
1446 // RRDHOST - set system info from environment variables
1447 // system_info fields must be heap allocated or NULL
@@ -1786,14 +1581,15 @@ int rrdhost_set_system_info_variable(struct rrdhost_system_info *system_info, ch
1581 // Added for gap-filling, if this proves to be a bottleneck in large-scale systems then we will need to cache
1582 // the last entry times as the metric updates, but let's see if it is a problem first.
1583 time_t rrdhost_last_entry_t(RRDHOST *h) {
1789 - rrdhost_rdlock(h);
1584 RRDSET *st;
1585 time_t result = 0;
1586 +
1587 rrdset_foreach_read(st, h) {
1588 time_t st_last = rrdset_last_entry_t(st);
1589 +
1590 if (st_last > result)
1591 result = st_last;
1592 }
1797 - rrdhost_unlock(h);
1593 + rrdset_foreach_done(st);
1594 return result;
1595 }
database/rrdlabels.c
+57 -40
@@ -478,9 +478,7 @@ typedef struct rrdlabel {
478 RRDLABEL_SRC label_source;
479 } RRDLABEL;
480
481 -static void rrdlabel_insert_callback(const char *name, void *value, void *data) {
482 - (void)name;
483 - DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
481 +static void rrdlabel_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *dict_ptr __maybe_unused) {
482 RRDLABEL *lb = (RRDLABEL *)value;
483
484 // label_value is already allocated by the STRING
@@ -488,42 +486,43 @@ static void rrdlabel_insert_callback(const char *name, void *value, void *data)
486 lb->label_source &= ~RRDLABEL_FLAG_OLD;
487 }
488
491 -static void rrdlabel_delete_callback(const char *name, void *value, void *data) {
492 - (void)name;
493 - DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
489 +static void rrdlabel_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *dict_ptr __maybe_unused) {
490 RRDLABEL *lb = (RRDLABEL *)value;
491
492 string_freez(lb->label_value);
493 lb->label_value = NULL;
494 }
495
500 -static void rrdlabel_conflict_callback(const char *name, void *oldvalue, void *newvalue, void *data) {
501 - (void)name;
502 - DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
496 +static bool rrdlabel_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *oldvalue, void *newvalue, void *dict_ptr __maybe_unused) {
497 RRDLABEL *lbold = (RRDLABEL *)oldvalue;
498 RRDLABEL *lbnew = (RRDLABEL *)newvalue;
499
500 if(lbold->label_value == lbnew->label_value) {
501 // they are the same
502 +
503 lbold->label_source |= lbnew->label_source;
504 lbold->label_source |= RRDLABEL_FLAG_OLD;
505 lbold->label_source &= ~RRDLABEL_FLAG_NEW;
506
507 // free the new one
508 string_freez(lbnew->label_value);
509 +
510 + return false;
511 }
515 - else {
516 - // they are different
517 - string_freez(lbold->label_value);
518 - lbold->label_value = lbnew->label_value;
519 - lbold->label_source = lbnew->label_source;
520 - lbold->label_source |= RRDLABEL_FLAG_NEW;
521 - lbold->label_source &= ~RRDLABEL_FLAG_OLD;
522 - }
512 +
513 + // they are different
514 +
515 + string_freez(lbold->label_value);
516 + lbold->label_value = lbnew->label_value;
517 + lbold->label_source = lbnew->label_source;
518 + lbold->label_source |= RRDLABEL_FLAG_NEW;
519 + lbold->label_source &= ~RRDLABEL_FLAG_OLD;
520 +
521 + return true;
522 }
523
524 DICTIONARY *rrdlabels_create(void) {
526 - DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
525 + DICTIONARY *dict = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
526 dictionary_register_insert_callback(dict, rrdlabel_insert_callback, dict);
527 dictionary_register_delete_callback(dict, rrdlabel_delete_callback, dict);
528 dictionary_register_conflict_callback(dict, rrdlabel_conflict_callback, dict);
@@ -623,7 +622,7 @@ void rrdlabels_add_pair(DICTIONARY *dict, const char *string, RRDLABEL_SRC ls) {
622 // rrdlabels_get_value_to_buffer_or_null()
623
624 void rrdlabels_get_value_to_buffer_or_null(DICTIONARY *labels, BUFFER *wb, const char *key, const char *quote, const char *null) {
626 - DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
625 + const DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
626 RRDLABEL *lb = dictionary_acquired_item_value(acquired_item);
627
628 if(lb && lb->label_value)
@@ -638,7 +637,7 @@ void rrdlabels_get_value_to_buffer_or_null(DICTIONARY *labels, BUFFER *wb, const
637 // rrdlabels_get_value_to_char_or_null()
638
639 void rrdlabels_get_value_to_char_or_null(DICTIONARY *labels, char **value, const char *key) {
641 - DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
640 + const DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
641 RRDLABEL *lb = dictionary_acquired_item_value(acquired_item);
642
643 *value = (lb && lb->label_value) ? strdupz(string2str(lb->label_value)) : NULL;
@@ -650,10 +649,7 @@ void rrdlabels_get_value_to_char_or_null(DICTIONARY *labels, char **value, const
649 // rrdlabels_unmark_all()
650 // remove labels RRDLABEL_FLAG_OLD and RRDLABEL_FLAG_NEW from all dictionary items
651
653 -static int remove_flags_old_new(const char *name, void *value, void *data) {
654 - (void)name;
655 - (void)data;
656 -
652 +static int remove_flags_old_new(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
653 RRDLABEL *lb = (RRDLABEL *)value;
654
655 if(lb->label_source & RRDLABEL_FLAG_OLD) lb->label_source &= ~RRDLABEL_FLAG_OLD;
@@ -671,12 +667,13 @@ void rrdlabels_unmark_all(DICTIONARY *labels) {
667 // rrdlabels_remove_all_unmarked()
668 // remove dictionary items that are neither old, nor new
669
674 -static int remove_not_old_not_new_callback(const char *name, void *value, void *data) {
670 +static int remove_not_old_not_new_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
671 + const char *name = dictionary_acquired_item_name(item);
672 DICTIONARY *dict = (DICTIONARY *)data;
673 RRDLABEL *lb = (RRDLABEL *)value;
674
675 if(!(lb->label_source & (RRDLABEL_FLAG_OLD | RRDLABEL_FLAG_NEW | RRDLABEL_FLAG_PERMANENT))) {
679 - dictionary_del_having_write_lock(dict, name);
676 + dictionary_del(dict, name);
677 return 1;
678 }
679
@@ -696,7 +693,8 @@ struct labels_walkthrough {
693 void *data;
694 };
695
699 -static int labels_walkthrough_callback(const char *name, void *value, void *data) {
696 +static int labels_walkthrough_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
697 + const char *name = dictionary_acquired_item_name(item);
698 struct labels_walkthrough *d = (struct labels_walkthrough *)data;
699 RRDLABEL *lb = (RRDLABEL *)value;
700
@@ -728,7 +726,8 @@ int rrdlabels_sorted_walkthrough_read(DICTIONARY *labels, int (*callback)(const
726 // rrdlabels_migrate_to_these()
727 // migrate an existing label list to a new list, INPLACE
728
731 -static int copy_label_to_dictionary_callback(const char *name, void *value, void *data) {
729 +static int copy_label_to_dictionary_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
730 + const char *name = dictionary_acquired_item_name(item);
731 DICTIONARY *dst = (DICTIONARY *)data;
732 RRDLABEL *lb = (RRDLABEL *)value;
733 labels_add_already_sanitized(dst, name, string2str(lb->label_value), lb->label_source);
@@ -765,7 +764,8 @@ struct simple_pattern_match_name_value {
764 char equal;
765 };
766
768 -static int simple_pattern_match_name_only_callback(const char *name, void *value, void *data) {
767 +static int simple_pattern_match_name_only_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
768 + const char *name = dictionary_acquired_item_name(item);
769 struct simple_pattern_match_name_value *t = (struct simple_pattern_match_name_value *)data;
770 (void)value;
771
@@ -775,7 +775,8 @@ static int simple_pattern_match_name_only_callback(const char *name, void *value
775 return 0;
776 }
777
778 -static int simple_pattern_match_name_and_value_callback(const char *name, void *value, void *data) {
778 +static int simple_pattern_match_name_and_value_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
779 + const char *name = dictionary_acquired_item_name(item);
780 struct simple_pattern_match_name_value *t = (struct simple_pattern_match_name_value *)data;
781 RRDLABEL *lb = (RRDLABEL *)value;
782
@@ -841,7 +842,9 @@ bool rrdlabels_match_simple_pattern(DICTIONARY *labels, const char *simple_patte
842 // ----------------------------------------------------------------------------
843 // Log all labels
844
844 -static int rrdlabels_log_label_to_buffer_callback(const char *name, void *value, void *data) {
845 +static int rrdlabels_log_label_to_buffer_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
846 + const char *name = dictionary_acquired_item_name(item);
847 +
848 BUFFER *wb = (BUFFER *)data;
849 RRDLABEL *lb = (RRDLABEL *)value;
850
@@ -891,7 +894,8 @@ struct labels_to_buffer {
894 size_t count;
895 };
896
894 -static int label_to_buffer_callback(const char *name, void *value, void *data) {
897 +static int label_to_buffer_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
898 + const char *name = dictionary_acquired_item_name(item);
899 struct labels_to_buffer *t = (struct labels_to_buffer *)data;
900 RRDLABEL *lb = (RRDLABEL *)value;
901
@@ -961,17 +965,30 @@ void rrdset_update_rrdlabels(RRDSET *st, DICTIONARY *new_rrdlabels) {
965 if (new_rrdlabels)
966 rrdlabels_migrate_to_these(st->rrdlabels, new_rrdlabels);
967
964 - rrdcalc_update_rrdlabels(st);
968 + rrdset_save_rrdlabels_to_sql(st);
969 +}
970 +
971 +void rrdset_save_rrdlabels_to_sql(RRDSET *st) {
972 + if(!st->rrdlabels) return;
973 +
974 + size_t old_version = st->rrdlabels_last_saved_version;
975 + size_t new_version = dictionary_version(st->rrdlabels);
976 +
977 + if(new_version != old_version) {
978 + // TODO - we should also cleanup sqlite from old new_rrdlabels that have been removed
979
966 - // TODO - we should also cleanup sqlite from old new_rrdlabels that have been removed
967 - BUFFER *sql_buf = buffer_create(1024);
968 - struct label_str tmp = {.sql = sql_buf, .count = 0 };
969 - uuid_unparse_lower(*st->chart_uuid, tmp.uuid_str);
970 - rrdlabels_walkthrough_read(st->rrdlabels, chart_label_store_to_sql_callback, &tmp);
971 - db_execute(buffer_tostring(sql_buf));
972 - buffer_free(sql_buf);
980 + BUFFER *sql_buf = buffer_create(1024);
981 + struct label_str tmp = {.sql = sql_buf, .count = 0};
982 + uuid_unparse_lower(st->chart_uuid, tmp.uuid_str);
983 + rrdlabels_walkthrough_read(st->rrdlabels, chart_label_store_to_sql_callback, &tmp);
984 + db_execute(buffer_tostring(sql_buf));
985 + buffer_free(sql_buf);
986 +
987 + st->rrdlabels_last_saved_version = new_version;
988 + }
989 }
990
991 +
992 // ----------------------------------------------------------------------------
993 // rrdlabels unit test
994
database/rrdset.c
+564 -521
@@ -21,56 +21,395 @@ void __rrdset_check_wrlock(RRDSET *st, const char *file, const char *function, c
21 }
22
23
24 +// ----------------------------------------------------------------------------
25 +// RRDSET name index
26 +
27 +static void rrdset_name_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdset, void *rrdhost __maybe_unused) {
28 + RRDSET *st = rrdset;
29 + rrdset_flag_set(st, RRDSET_FLAG_INDEXED_NAME);
30 +}
31 +static void rrdset_name_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdset, void *rrdhost __maybe_unused) {
32 + RRDSET *st = rrdset;
33 + rrdset_flag_clear(st, RRDSET_FLAG_INDEXED_NAME);
34 +}
35 +
36 +static inline void rrdset_index_add_name(RRDHOST *host, RRDSET *st) {
37 + if(!st->name) return;
38 + dictionary_set(host->rrdset_root_index_name, rrdset_name(st), st, sizeof(RRDSET));
39 +}
40 +
41 +static inline void rrdset_index_del_name(RRDHOST *host, RRDSET *st) {
42 + if(rrdset_flag_check(st, RRDSET_FLAG_INDEXED_NAME))
43 + dictionary_del(host->rrdset_root_index_name, rrdset_name(st));
44 +}
45 +
46 +static inline RRDSET *rrdset_index_find_name(RRDHOST *host, const char *name) {
47 + return dictionary_get(host->rrdset_root_index_name, name);
48 +}
49 +
50 // ----------------------------------------------------------------------------
51 // RRDSET index
52
27 -static inline void rrdset_index_add(RRDHOST *host, RRDSET *st) {
28 - if(likely(dictionary_set(host->rrdset_root_index, rrdset_id(st), st, sizeof(RRDSET)) == st)) {
29 - rrdset_flag_set(st, RRDSET_FLAG_INDEXED_ID);
30 - }
31 - else {
32 - rrdset_flag_clear(st, RRDSET_FLAG_INDEXED_ID);
33 - error("RRDSET: %s() attempted to index duplicate object with key '%s'", __FUNCTION__, rrdset_id(st));
34 - }
53 +static inline void rrdset_update_permanent_labels(RRDSET *st) {
54 + if(!st->rrdlabels) return;
55 +
56 + rrdlabels_add(st->rrdlabels, "_collect_plugin", rrdset_plugin_name(st), RRDLABEL_SRC_AUTO| RRDLABEL_FLAG_PERMANENT);
57 + rrdlabels_add(st->rrdlabels, "_collect_module", rrdset_module_name(st), RRDLABEL_SRC_AUTO| RRDLABEL_FLAG_PERMANENT);
58 }
59
37 -static inline void rrdset_index_del(RRDHOST *host, RRDSET *st) {
38 - if(rrdset_flag_check(st, RRDSET_FLAG_INDEXED_ID)) {
39 - if(likely(dictionary_del(host->rrdset_root_index, rrdset_id(st)) == 0))
40 - rrdset_flag_clear(st, RRDSET_FLAG_INDEXED_ID);
60 +static STRING *rrdset_fix_name(RRDHOST *host, const char *chart_full_id, const char *type, const char *current_name, const char *name) {
61 + if(!name || !*name) return NULL;
62 +
63 + char full_name[RRD_ID_LENGTH_MAX + 1];
64 + char sanitized_name[CONFIG_MAX_VALUE + 1];
65 + char new_name[CONFIG_MAX_VALUE + 1];
66 +
67 + snprintfz(full_name, RRD_ID_LENGTH_MAX, "%s.%s", type, name);
68 + rrdset_strncpyz_name(sanitized_name, full_name, CONFIG_MAX_VALUE);
69 + strncpyz(new_name, sanitized_name, CONFIG_MAX_VALUE);
70 +
71 + if(rrdset_index_find_name(host, new_name)) {
72 + debug(D_RRD_CALLS, "RRDSET: chart name '%s' on host '%s' already exists.", new_name, rrdhost_hostname(host));
73 + if(!strcmp(chart_full_id, full_name) && (!current_name || !*current_name)) {
74 + unsigned i = 1;
75 +
76 + do {
77 + snprintfz(new_name, CONFIG_MAX_VALUE, "%s_%u", sanitized_name, i);
78 + i++;
79 + } while (rrdset_index_find_name(host, new_name));
80 +
81 + info("RRDSET: using name '%s' for chart '%s' on host '%s'.", new_name, full_name, rrdhost_hostname(host));
82 + }
83 else
42 - error("RRDSET: %s() attempted to delete non-indexed object with key '%s'", __FUNCTION__, rrdset_id(st));
84 + return NULL;
85 }
86 +
87 + return string_strdupz(new_name);
88 }
89
46 -static RRDSET *rrdset_index_find(RRDHOST *host, const char *id) {
47 - return dictionary_get(host->rrdset_root_index, id);
90 +struct rrdset_constructor {
91 + RRDHOST *host;
92 + const char *type;
93 + const char *id;
94 + const char *name;
95 + const char *family;
96 + const char *context;
97 + const char *title;
98 + const char *units;
99 + const char *plugin;
100 + const char *module;
101 + long priority;
102 + int update_every;
103 + RRDSET_TYPE chart_type;
104 + RRD_MEMORY_MODE memory_mode;
105 + long history_entries;
106 +
107 + enum {
108 + RRDSET_REACT_NONE = 0,
109 + RRDSET_REACT_NEW = (1 << 0),
110 + RRDSET_REACT_CHART_ARCHIVED_TO_LIVE = (1 << 1),
111 + RRDSET_REACT_PLUGIN_UPDATED = (1 << 2),
112 + RRDSET_REACT_MODULE_UPDATED = (1 << 3),
113 + RRDSET_REACT_CHART_ACTIVATED = (1 << 4),
114 + } react_action;
115 +};
116 +
117 +// the constructor - the dictionary is write locked while this runs
118 +static void rrdset_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdset, void *constructor_data) {
119 + static STRING *anomaly_rates_chart = NULL;
120 +
121 + if(!unlikely(!anomaly_rates_chart))
122 + anomaly_rates_chart = string_strdupz(ML_ANOMALY_RATES_CHART_ID);
123 +
124 + struct rrdset_constructor *ctr = constructor_data;
125 + RRDHOST *host = ctr->host;
126 + RRDSET *st = rrdset;
127 +
128 + const char *chart_full_id = dictionary_acquired_item_name(item);
129 +
130 + st->id = string_strdupz(chart_full_id);
131 +
132 + st->name = rrdset_fix_name(host, chart_full_id, ctr->type, NULL, ctr->name);
133 + if(!st->name)
134 + st->name = rrdset_fix_name(host, chart_full_id, ctr->type, NULL, ctr->id);
135 + rrdset_index_add_name(host, st);
136 +
137 + st->parts.id = string_strdupz(ctr->id);
138 + st->parts.type = string_strdupz(ctr->type);
139 + st->parts.name = string_strdupz(ctr->name);
140 +
141 + st->family = (ctr->family && *ctr->family) ? rrd_string_strdupz(ctr->family) : rrd_string_strdupz(ctr->type);
142 + st->context = (ctr->context && *ctr->context) ? rrd_string_strdupz(ctr->context) : rrd_string_strdupz(chart_full_id);
143 +
144 + st->units = rrd_string_strdupz(ctr->units);
145 + st->title = rrd_string_strdupz(ctr->title);
146 + st->plugin_name = rrd_string_strdupz(ctr->plugin);
147 + st->module_name = rrd_string_strdupz(ctr->module);
148 + st->priority = ctr->priority;
149 +
150 + st->cache_dir = rrdset_cache_dir(host, chart_full_id);
151 + st->entries = (ctr->memory_mode != RRD_MEMORY_MODE_DBENGINE) ? align_entries_to_pagesize(ctr->memory_mode, ctr->history_entries) : 5;
152 + st->update_every = ctr->update_every;
153 + st->rrd_memory_mode = ctr->memory_mode;
154 +
155 + st->chart_type = ctr->chart_type;
156 + st->gap_when_lost_iterations_above = (int) (gap_when_lost_iterations_above + 2);
157 + st->rrdhost = host;
158 +
159 + st->flags = RRDSET_FLAG_SYNC_CLOCK | RRDSET_FLAG_INDEXED_ID;
160 + if(unlikely(st->id == anomaly_rates_chart))
161 + st->flags |= RRDSET_FLAG_ANOMALY_RATE_CHART;
162 +
163 + netdata_rwlock_init(&st->rrdset_rwlock);
164 + netdata_rwlock_init(&st->alerts.rwlock);
165 +
166 + if(st->rrd_memory_mode == RRD_MEMORY_MODE_SAVE || st->rrd_memory_mode == RRD_MEMORY_MODE_MAP) {
167 + if(!rrdset_memory_load_or_create_map_save(st, st->rrd_memory_mode)) {
168 + info("Failed to use db mode %s for chart '%s', falling back to ram mode.", (st->rrd_memory_mode == RRD_MEMORY_MODE_MAP)?"map":"save", rrdset_name(st));
169 + st->rrd_memory_mode = RRD_MEMORY_MODE_RAM;
170 + }
171 + }
172 +
173 + if (find_chart_uuid(host, string2str(st->parts.type), string2str(st->parts.id), string2str(st->parts.name), &st->chart_uuid))
174 + uuid_generate(st->chart_uuid);
175 + update_chart_metadata(&st->chart_uuid, st, string2str(st->parts.id), string2str(st->parts.name));
176 +
177 + rrddim_index_init(st);
178 +
179 + // chart variables - we need this for data collection to work (collector given chart variables) - not only health
180 + rrdsetvar_index_init(st);
181 +
182 + if(host->health_enabled) {
183 + st->green = NAN;
184 + st->red = NAN;
185 + st->rrdfamily = rrdfamily_add_and_acquire(host, rrdset_family(st));
186 + st->rrdvars = rrdvariables_create();
187 + rrddimvar_index_init(st);
188 + }
189 +
190 + st->rrdlabels = rrdlabels_create();
191 + rrdset_update_permanent_labels(st);
192 +
193 + ctr->react_action = RRDSET_REACT_NEW;
194 }
195
50 -// ----------------------------------------------------------------------------
51 -// RRDSET name index
196 +// the destructor - the dictionary is write locked while this runs
197 +static void rrdset_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdset, void *rrdhost) {
198 + RRDHOST *host = rrdhost;
199 + RRDSET *st = rrdset;
200
53 -static inline void rrdset_index_add_name(RRDHOST *host, RRDSET *st) {
54 - if(likely(dictionary_set(host->rrdset_root_index_name, rrdset_name(st), st, sizeof(RRDSET)) == st)) {
55 - rrdset_flag_set(st, RRDSET_FLAG_INDEXED_NAME);
201 + rrdset_flag_clear(st, RRDSET_FLAG_INDEXED_ID);
202 +
203 + // remove it from the name index
204 + rrdset_index_del_name(host, st);
205 +
206 + rrdcalc_unlink_all_rrdset_alerts(st);
207 +
208 + // ------------------------------------------------------------------------
209 + // the order of destruction is important here
210 +
211 + // 1. delete RRDDIMVAR index - this will speed up the destruction of RRDDIMs
212 + // because each dimension loops to find its own variables in this index.
213 + // There are no references to the items on this index from the dimensions.
214 + // To find their own, they have to walk-through the dictionary.
215 + rrddimvar_index_destroy(st); // destroy the rrddimvar index
216 +
217 + // 2. delete RRDSETVAR index
218 + rrdsetvar_index_destroy(st); // destroy the rrdsetvar index
219 +
220 + // 3. delete RRDVAR index after the above, to avoid triggering its garbage collector (they have references on this)
221 + rrdvariables_destroy(st->rrdvars); // free all variables and destroy the rrdvar dictionary
222 +
223 + // 4. delete RRDFAMILY - this has to be last, because RRDDIMVAR and RRDSETVAR need the reference counter
224 + rrdfamily_release(host, st->rrdfamily); // release the acquired rrdfamily -- has to be after all variables
225 +
226 + // 5. delete RRDDIMs, now their variables are not existing, so this is fast
227 + rrddim_index_destroy(st); // free all the dimensions and destroy the dimensions index
228 +
229 + // 6. this has to be after the dimensions are freed, but before labels are freed (contexts need the labels)
230 + rrdcontext_removed_rrdset(st); // let contexts know
231 +
232 + // 7. destroy the chart labels
233 + rrdlabels_destroy(st->rrdlabels); // destroy the labels, after letting the contexts know
234 +
235 + rrdset_memory_file_free(st); // remove files of db mode save and map
236 +
237 + // ------------------------------------------------------------------------
238 + // free it
239 +
240 + netdata_rwlock_destroy(&st->rrdset_rwlock);
241 + netdata_rwlock_destroy(&st->alerts.rwlock);
242 +
243 + string_freez(st->id);
244 + string_freez(st->name);
245 + string_freez(st->parts.id);
246 + string_freez(st->parts.type);
247 + string_freez(st->parts.name);
248 + string_freez(st->family);
249 + string_freez(st->title);
250 + string_freez(st->units);
251 + string_freez(st->context);
252 + string_freez(st->plugin_name);
253 + string_freez(st->module_name);
254 +
255 + freez(st->exporting_flags);
256 + freez(st->cache_dir);
257 +}
258 +
259 +// the item to be inserted, is already in the dictionary
260 +// this callback deals with the situation, migrating the existing object to the new values
261 +// the dictionary is write locked while this runs
262 +static bool rrdset_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdset, void *new_rrdset, void *constructor_data) {
263 + (void)new_rrdset; // it is NULL
264 +
265 + struct rrdset_constructor *ctr = constructor_data;
266 + RRDSET *st = rrdset;
267 +
268 + rrdset_isnot_obsolete(st);
269 +
270 + ctr->react_action = RRDSET_REACT_NONE;
271 +
272 + if (rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED)) {
273 + rrdset_flag_clear(st, RRDSET_FLAG_ARCHIVED);
274 + ctr->react_action |= RRDSET_REACT_CHART_ACTIVATED;
275 }
57 - else {
58 - rrdset_flag_clear(st, RRDSET_FLAG_INDEXED_NAME);
59 - error("RRDSET: %s() attempted to index duplicate object with key '%s'", __FUNCTION__, rrdset_name(st));
276 +
277 + if (rrdset_reset_name(st, (ctr->name && *ctr->name) ? ctr->name : ctr->id) == 2)
278 + ctr->react_action |= RRDSET_REACT_CHART_ARCHIVED_TO_LIVE;
279 +
280 + if (unlikely(st->priority != ctr->priority)) {
281 + st->priority = ctr->priority;
282 + ctr->react_action |= RRDSET_REACT_CHART_ARCHIVED_TO_LIVE;
283 + }
284 + if (unlikely(st->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE && st->update_every != ctr->update_every)) {
285 + st->update_every = ctr->update_every;
286 + ctr->react_action |= RRDSET_REACT_CHART_ARCHIVED_TO_LIVE;
287 + }
288 +
289 + if(ctr->plugin && *ctr->plugin) {
290 + STRING *old_plugin = st->plugin_name;
291 + st->plugin_name = rrd_string_strdupz(ctr->plugin);
292 + if (old_plugin != st->plugin_name)
293 + ctr->react_action |= RRDSET_REACT_PLUGIN_UPDATED;
294 + string_freez(old_plugin);
295 + }
296 +
297 + if(ctr->module && *ctr->module) {
298 + STRING *old_module = st->module_name;
299 + st->module_name = rrd_string_strdupz(ctr->module);
300 + if (old_module != st->module_name)
301 + ctr->react_action |= RRDSET_REACT_MODULE_UPDATED;
302 + string_freez(old_module);
303 + }
304 +
305 + if(ctr->title && *ctr->title) {
306 + STRING *old_title = st->title;
307 + st->title = rrd_string_strdupz(ctr->title);
308 + if(old_title != st->title)
309 + ctr->react_action |= RRDSET_REACT_CHART_ARCHIVED_TO_LIVE;
310 + string_freez(old_title);
311 + }
312 +
313 + if(ctr->units && *ctr->units) {
314 + STRING *old_units = st->units;
315 + st->units = rrd_string_strdupz(ctr->units);
316 + if(old_units != st->units)
317 + ctr->react_action |= RRDSET_REACT_CHART_ARCHIVED_TO_LIVE;
318 + string_freez(old_units);
319 + }
320 +
321 + if(ctr->context && *ctr->context) {
322 + STRING *old_context = st->context;
323 + st->context = rrd_string_strdupz(ctr->context);
324 + if(old_context != st->context)
325 + ctr->react_action |= RRDSET_REACT_CHART_ARCHIVED_TO_LIVE;
326 + string_freez(old_context);
327 }
328 +
329 + if(st->chart_type != ctr->chart_type) {
330 + st->chart_type = ctr->chart_type;
331 + ctr->react_action |= RRDSET_REACT_CHART_ARCHIVED_TO_LIVE;
332 + }
333 +
334 + if(ctr->react_action)
335 + rrdset_flag_clear(st, RRDSET_FLAG_ACLK);
336 +
337 + rrdset_update_permanent_labels(st);
338 +
339 + rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
340 + rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
341 +
342 + return ctr->react_action != RRDSET_REACT_NONE;
343 }
344
63 -static inline void rrdset_index_del_name(RRDHOST *host, RRDSET *st) {
64 - if(rrdset_flag_check(st, RRDSET_FLAG_INDEXED_ID)) {
65 - if(likely(dictionary_del(host->rrdset_root_index_name, rrdset_name(st)) != 0))
66 - rrdset_flag_clear(st, RRDSET_FLAG_INDEXED_NAME);
67 - else
68 - error("RRDSET: %s() attempted to delete non-index object with key '%s'", __FUNCTION__, rrdset_name(st));
345 +// this is called after all insertions/conflicts, with the dictionary unlocked, with a reference to RRDSET
346 +// so, any actions requiring locks on other objects, should be placed here
347 +static void rrdset_react_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdset, void *constructor_data) {
348 + struct rrdset_constructor *ctr = constructor_data;
349 + RRDSET *st = rrdset;
350 + RRDHOST *host = st->rrdhost;
351 +
352 + if(host->health_enabled && (ctr->react_action & (RRDSET_REACT_NEW | RRDSET_REACT_CHART_ACTIVATED))) {
353 + rrdsetvar_add_and_leave_released(st, "last_collected_t", RRDVAR_TYPE_TIME_T, &st->last_collected_time.tv_sec, RRDVAR_FLAG_NONE);
354 + rrdsetvar_add_and_leave_released(st, "collected_total_raw", RRDVAR_TYPE_TOTAL, &st->last_collected_total, RRDVAR_FLAG_NONE);
355 + rrdsetvar_add_and_leave_released(st, "green", RRDVAR_TYPE_CALCULATED, &st->green, RRDVAR_FLAG_NONE);
356 + rrdsetvar_add_and_leave_released(st, "red", RRDVAR_TYPE_CALCULATED, &st->red, RRDVAR_FLAG_NONE);
357 + rrdsetvar_add_and_leave_released(st, "update_every", RRDVAR_TYPE_INT, &st->update_every, RRDVAR_FLAG_NONE);
358 +
359 + rrdcalc_link_matching_alerts_to_rrdset(st);
360 + rrdcalctemplate_link_matching_templates_to_rrdset(st);
361 }
362 +
363 + if(ctr->react_action & (RRDSET_REACT_CHART_ARCHIVED_TO_LIVE | RRDSET_REACT_PLUGIN_UPDATED | RRDSET_REACT_MODULE_UPDATED)) {
364 + debug(D_METADATALOG, "CHART [%s] metadata updated", rrdset_id(st));
365 + if(unlikely(update_chart_metadata(&st->chart_uuid, st, ctr->id, ctr->name)))
366 + error_report("Failed to update chart metadata in the database");
367 + }
368 +
369 + rrdcontext_updated_rrdset(st);
370 }
371
72 -static inline RRDSET *rrdset_index_find_name(RRDHOST *host, const char *name) {
73 - return dictionary_get(host->rrdset_root_index_name, name);
372 +void rrdset_index_init(RRDHOST *host) {
373 + if(!host->rrdset_root_index) {
374 + host->rrdset_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
375 +
376 + dictionary_register_insert_callback(host->rrdset_root_index, rrdset_insert_callback, NULL);
377 + dictionary_register_conflict_callback(host->rrdset_root_index, rrdset_conflict_callback, NULL);
378 + dictionary_register_react_callback(host->rrdset_root_index, rrdset_react_callback, NULL);
379 + dictionary_register_delete_callback(host->rrdset_root_index, rrdset_delete_callback, host);
380 + }
381 +
382 + if(!host->rrdset_root_index_name) {
383 + host->rrdset_root_index_name = dictionary_create(
384 + DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
385 +
386 + dictionary_register_insert_callback(host->rrdset_root_index_name, rrdset_name_insert_callback, host);
387 + dictionary_register_delete_callback(host->rrdset_root_index_name, rrdset_name_delete_callback, host);
388 + }
389 +}
390 +
391 +void rrdset_index_destroy(RRDHOST *host) {
392 + // destroy the name index first
393 + dictionary_destroy(host->rrdset_root_index_name);
394 + host->rrdset_root_index_name = NULL;
395 +
396 + // destroy the id index last
397 + dictionary_destroy(host->rrdset_root_index);
398 + host->rrdset_root_index = NULL;
399 +}
400 +
401 +static inline RRDSET *rrdset_index_add(RRDHOST *host, const char *id, struct rrdset_constructor *st_ctr) {
402 + return dictionary_set_advanced(host->rrdset_root_index, id, -1, NULL, sizeof(RRDSET), st_ctr);
403 +}
404 +
405 +static inline void rrdset_index_del(RRDHOST *host, RRDSET *st) {
406 + if(rrdset_flag_check(st, RRDSET_FLAG_INDEXED_ID))
407 + dictionary_del(host->rrdset_root_index, rrdset_id(st));
408 +}
409 +
410 +static RRDSET *rrdset_index_find(RRDHOST *host, const char *id) {
411 + // TODO - the name index should have an acquired dictionary item, not just a pointer to RRDSET
412 + return dictionary_get(host->rrdset_root_index, id);
413 }
414
415 // ----------------------------------------------------------------------------
@@ -118,53 +457,30 @@ char *rrdset_strncpyz_name(char *to, const char *from, size_t length) {
457 return to;
458 }
459
121 -int rrdset_set_name(RRDSET *st, const char *name) {
460 +int rrdset_reset_name(RRDSET *st, const char *name) {
461 if(unlikely(!strcmp(rrdset_name(st), name)))
462 return 1;
463
464 RRDHOST *host = st->rrdhost;
465
127 - debug(D_RRD_CALLS, "rrdset_set_name() old: '%s', new: '%s'", rrdset_name(st), name);
466 + debug(D_RRD_CALLS, "rrdset_reset_name() old: '%s', new: '%s'", rrdset_name(st), name);
467
129 - char full_name[RRD_ID_LENGTH_MAX + 1];
130 - char sanitized_name[CONFIG_MAX_VALUE + 1];
131 - char new_name[CONFIG_MAX_VALUE + 1];
132 -
133 - snprintfz(full_name, RRD_ID_LENGTH_MAX, "%s.%s", rrdset_type(st), name);
134 - rrdset_strncpyz_name(sanitized_name, full_name, CONFIG_MAX_VALUE);
135 - strncpyz(new_name, sanitized_name, CONFIG_MAX_VALUE);
136 -
137 - if(rrdset_index_find_name(host, new_name)) {
138 - debug(D_RRD_CALLS, "RRDSET: chart name '%s' on host '%s' already exists.", new_name, rrdhost_hostname(host));
139 - if(!strcmp(rrdset_id(st), full_name) && !st->name) {
140 - unsigned i = 1;
141 -
142 - do {
143 - snprintfz(new_name, CONFIG_MAX_VALUE, "%s_%u", sanitized_name, i);
144 - i++;
145 - } while (rrdset_index_find_name(host, new_name));
146 -
147 - info("RRDSET: using name '%s' for chart '%s' on host '%s'.", new_name, full_name, rrdhost_hostname(host));
148 - } else {
149 - return 0;
150 - }
151 - }
468 + STRING *name_string = rrdset_fix_name(host, rrdset_id(st), rrdset_parts_type(st), string2str(st->name), name);
469 + if(!name_string) return 0;
470
471 if(st->name) {
472 rrdset_index_del_name(host, st);
473 string_freez(st->name);
156 - st->name = string_strdupz(new_name);
474 + st->name = name_string;
475 rrdsetvar_rename_all(st);
476 }
159 - else {
160 - st->name = string_strdupz(new_name);
161 - }
477 + else
478 + st->name = name_string;
479
163 - rrdset_wrlock(st);
480 RRDDIM *rd;
165 - rrddim_foreach_write(rd, st)
481 + rrddim_foreach_read(rd, st)
482 rrddimvar_rename_all(rd);
167 - rrdset_unlock(st);
483 + rrddim_foreach_done(rd);
484
485 rrdset_index_add_name(host, st);
486
@@ -178,6 +494,36 @@ int rrdset_set_name(RRDSET *st, const char *name) {
494 return 2;
495 }
496
497 +// get the timestamp of the last entry in the round-robin database
498 +time_t rrdset_last_entry_t(RRDSET *st) {
499 + RRDDIM *rd;
500 + time_t last_entry_t = 0;
501 +
502 + rrddim_foreach_read(rd, st) {
503 + time_t t = rrddim_last_entry_t(rd);
504 + if(t > last_entry_t) last_entry_t = t;
505 + }
506 + rrddim_foreach_done(rd);
507 +
508 + return last_entry_t;
509 +}
510 +
511 +// get the timestamp of first entry in the round-robin database
512 +time_t rrdset_first_entry_t(RRDSET *st) {
513 + RRDDIM *rd;
514 + time_t first_entry_t = LONG_MAX;
515 +
516 + rrddim_foreach_read(rd, st) {
517 + time_t t = rrddim_first_entry_t(rd);
518 + if(t < first_entry_t)
519 + first_entry_t = t;
520 + }
521 + rrddim_foreach_done(rd);
522 +
523 + if (unlikely(LONG_MAX == first_entry_t)) return 0;
524 + return first_entry_t;
525 +}
526 +
527 inline void rrdset_is_obsolete(RRDSET *st) {
528 if(unlikely(rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED))) {
529 info("Cannot obsolete already archived chart %s", rrdset_name(st));
@@ -186,8 +532,9 @@ inline void rrdset_is_obsolete(RRDSET *st) {
532
533 if(unlikely(!(rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE)))) {
534 rrdset_flag_set(st, RRDSET_FLAG_OBSOLETE);
535 + rrdhost_flag_set(st->rrdhost, RRDHOST_FLAG_PENDING_OBSOLETE_CHARTS);
536 +
537 st->last_accessed_time = now_realtime_sec();
190 - st->rrdhost->obsolete_charts_count++;
538
539 rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
540
@@ -202,7 +549,6 @@ inline void rrdset_isnot_obsolete(RRDSET *st) {
549 if(unlikely((rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE)))) {
550 rrdset_flag_clear(st, RRDSET_FLAG_OBSOLETE);
551 st->last_accessed_time = now_realtime_sec();
205 - st->rrdhost->obsolete_charts_count--;
552
553 rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
554
@@ -220,11 +566,20 @@ inline void rrdset_update_heterogeneous_flag(RRDSET *st) {
566
567 rrdset_flag_clear(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
568
223 - RRD_ALGORITHM algorithm = st->dimensions->algorithm;
224 - collected_number multiplier = ABS(st->dimensions->multiplier);
225 - collected_number divisor = ABS(st->dimensions->divisor);
569 + bool init = false, is_heterogeneous = false;
570 + RRD_ALGORITHM algorithm;
571 + collected_number multiplier;
572 + collected_number divisor;
573
574 rrddim_foreach_read(rd, st) {
575 + if(!init) {
576 + algorithm = rd->algorithm;
577 + multiplier = rd->multiplier;
578 + divisor = ABS(rd->divisor);
579 + init = true;
580 + continue;
581 + }
582 +
583 if(algorithm != rd->algorithm || multiplier != ABS(rd->multiplier) || divisor != ABS(rd->divisor)) {
584 if(!rrdset_flag_check(st, RRDSET_FLAG_HETEROGENEOUS)) {
585 #ifdef NETDATA_INTERNAL_CHECKS
@@ -239,12 +594,17 @@ inline void rrdset_update_heterogeneous_flag(RRDSET *st) {
594 #endif
595 rrdset_flag_set(st, RRDSET_FLAG_HETEROGENEOUS);
596 }
242 - return;
597 +
598 + is_heterogeneous = true;
599 + break;
600 }
601 }
602 + rrddim_foreach_done(rd);
603
246 - rrdset_flag_clear(st, RRDSET_FLAG_HETEROGENEOUS);
247 - rrdcontext_updated_rrdset_flags(st);
604 + if(!is_heterogeneous) {
605 + rrdset_flag_clear(st, RRDSET_FLAG_HETEROGENEOUS);
606 + rrdcontext_updated_rrdset_flags(st);
607 + }
608 }
609
610 // ----------------------------------------------------------------------------
@@ -275,6 +635,7 @@ void rrdset_reset(RRDSET *st) {
635 }
636 }
637 }
638 + rrddim_foreach_done(rd);
639 }
640
641 // ----------------------------------------------------------------------------
@@ -326,89 +687,20 @@ static inline void last_updated_time_align(RRDSET *st) {
687
688 void rrdset_free(RRDSET *st) {
689 if(unlikely(!st)) return;
329 -
330 - RRDHOST *host = st->rrdhost;
331 -
332 - rrdhost_check_wrlock(host); // make sure we have a write lock on the host
333 - rrdset_wrlock(st); // lock this RRDSET
334 - // info("Removing chart '%s' ('%s')", st->id, st->name);
335 -
336 - // ------------------------------------------------------------------------
337 - // remove it from the indexes
338 -
339 - rrdset_index_del(host, st);
340 - rrdset_index_del_name(host, st);
341 -
342 - // ------------------------------------------------------------------------
343 - // free its children structures
344 -
345 - freez(st->exporting_flags);
346 -
347 - while(st->variables) rrdsetvar_free(st->variables);
348 -// while(st->alarms) rrdsetcalc_unlink(st->alarms);
349 - /* We must free all connected alarms here in case this has been an ephemeral chart whose alarm was
350 - * created by a template. This leads to an effective memory leak, which cannot be detected since the
351 - * alarms will still be connected to the host, and freed during shutdown. */
352 - while(st->alarms) rrdcalc_unlink_and_free(st->rrdhost, st->alarms);
353 - while(st->dimensions) rrddim_free(st, st->dimensions);
354 -
355 - rrdfamily_free(host, st->rrdfamily);
356 -
357 - debug(D_RRD_CALLS, "RRDSET: Cleaning up remaining chart variables for host '%s', chart '%s'", rrdhost_hostname(host), rrdset_id(st));
358 - rrdvar_free_remaining_variables(host, st->rrdvar_root_index);
359 -
360 - // ------------------------------------------------------------------------
361 - // unlink it from the host
362 -
363 - DOUBLE_LINKED_LIST_REMOVE_UNSAFE(host->rrdset_root, st, prev, next);
364 -
365 - rrdset_unlock(st);
366 -
367 - // this has to be after the dimensions are freed
368 - rrdcontext_removed_rrdset(st);
369 -
370 - // ------------------------------------------------------------------------
371 - // free it
372 -
373 - netdata_rwlock_destroy(&st->rrdset_rwlock);
374 -
375 - rrdset_memory_file_free(st);
376 - rrdlabels_destroy(st->rrdlabels);
377 -
378 - // free directly allocated members
379 -
380 - dictionary_destroy(st->rrddim_root_index);
381 - dictionary_destroy(st->rrdvar_root_index);
382 -
383 - string_freez(st->id);
384 - string_freez(st->name);
385 - string_freez(st->type);
386 - string_freez(st->family);
387 - string_freez(st->title);
388 - string_freez(st->units);
389 - string_freez(st->context);
390 - string_freez(st->plugin_name);
391 - string_freez(st->module_name);
392 -
393 - freez(st->cache_dir);
394 - freez(st->chart_uuid);
395 -
396 - freez(st);
690 + rrdset_index_del(st->rrdhost, st);
691 }
692
693 void rrdset_save(RRDSET *st) {
400 - rrdset_check_rdlock(st);
401 -
694 rrdset_memory_file_save(st);
695
696 RRDDIM *rd;
697 rrddim_foreach_read(rd, st)
698 rrddim_memory_file_save(rd);
699 + rrddim_foreach_done(rd);
700 }
701
702 void rrdset_delete_files(RRDSET *st) {
703 RRDDIM *rd;
411 - rrdset_check_rdlock(st);
704
705 info("Deleting chart '%s' ('%s') from disk...", rrdset_id(st), rrdset_name(st));
706
@@ -431,6 +723,7 @@ void rrdset_delete_files(RRDSET *st) {
723 if(unlikely(unlink(cache_filename) == -1))
724 error("Cannot delete dimension file '%s'", cache_filename);
725 }
726 + rrddim_foreach_done(rd);
727
728 recursively_delete_dir(st->cache_dir, "left-over chart");
729 }
@@ -438,8 +731,6 @@ void rrdset_delete_files(RRDSET *st) {
731 void rrdset_delete_obsolete_dimensions(RRDSET *st) {
732 RRDDIM *rd;
733
441 - rrdset_check_rdlock(st);
442 -
734 info("Deleting dimensions of chart '%s' ('%s') from disk...", rrdset_id(st), rrdset_name(st));
735
736 rrddim_foreach_read(rd, st) {
@@ -451,29 +742,12 @@ void rrdset_delete_obsolete_dimensions(RRDSET *st) {
742 error("Cannot delete dimension file '%s'", cache_filename);
743 }
744 }
745 + rrddim_foreach_done(rd);
746 }
747
748 // ----------------------------------------------------------------------------
749 // RRDSET - create a chart
750
459 -static inline RRDSET *rrdset_find_on_create(RRDHOST *host, const char *fullid) {
460 - RRDSET *st = rrdset_find(host, fullid);
461 - if(unlikely(st)) {
462 - rrdset_isnot_obsolete(st);
463 - debug(D_RRD_CALLS, "RRDSET '%s', already exists.", fullid);
464 - return st;
465 - }
466 -
467 - return NULL;
468 -}
469 -
470 -static inline void rrdset_update_permanent_labels(RRDSET *st) {
471 - if(!st->rrdlabels) return;
472 -
473 - rrdlabels_add(st->rrdlabels, "_collect_plugin", rrdset_plugin_name(st), RRDLABEL_SRC_AUTO| RRDLABEL_FLAG_PERMANENT);
474 - rrdlabels_add(st->rrdlabels, "_collect_module", rrdset_module_name(st), RRDLABEL_SRC_AUTO| RRDLABEL_FLAG_PERMANENT);
475 -}
476 -
751 RRDSET *rrdset_create_custom(
752 RRDHOST *host
753 , const char *type
@@ -491,7 +765,10 @@ RRDSET *rrdset_create_custom(
765 , RRD_MEMORY_MODE memory_mode
766 , long history_entries
767 ) {
494 - if(!type || !type[0]) {
768 + if (host != localhost)
769 + host->senders_last_chart_command = now_realtime_sec();
770 +
771 + if(!type || !type[0])
772 fatal("Cannot create rrd stats without a type: id '%s', name '%s', family '%s', context '%s', title '%s', units '%s', plugin '%s', module '%s'."
773 , (id && *id)?id:"<unset>"
774 , (name && *name)?name:"<unset>"
@@ -502,10 +779,8 @@ RRDSET *rrdset_create_custom(
779 , (plugin && *plugin)?plugin:"<unset>"
780 , (module && *module)?module:"<unset>"
781 );
505 - return NULL;
506 - }
782
508 - if(!id || !id[0]) {
783 + if(!id || !id[0])
784 fatal("Cannot create rrd stats without an id: type '%s', name '%s', family '%s', context '%s', title '%s', units '%s', plugin '%s', module '%s'."
785 , type
786 , (name && *name)?name:"<unset>"
@@ -516,249 +791,40 @@ RRDSET *rrdset_create_custom(
791 , (plugin && *plugin)?plugin:"<unset>"
792 , (module && *module)?module:"<unset>"
793 );
519 - return NULL;
520 - }
521 -
522 - if (host != localhost) {
523 - host->senders_last_chart_command = now_realtime_sec();
524 - }
794
795 // ------------------------------------------------------------------------
796 // check if it already exists
797
529 - char fullid[RRD_ID_LENGTH_MAX + 1];
530 - snprintfz(fullid, RRD_ID_LENGTH_MAX, "%s.%s", type, id);
531 - json_fix_string(fullid);
532 -
533 - int changed_from_archived_to_active = 0;
534 - RRDSET *st = rrdset_find_on_create(host, fullid);
535 - if (st) {
536 - int mark_rebuild = 0;
537 - if (rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED)) {
538 - rrdset_flag_clear(st, RRDSET_FLAG_ARCHIVED);
539 - changed_from_archived_to_active = 1;
540 - mark_rebuild |= META_CHART_ACTIVATED;
541 - }
542 -
543 - int rc;
544 - if(unlikely(name && *name))
545 - rc = rrdset_set_name(st, name);
546 - else
547 - rc = rrdset_set_name(st, id);
548 -
549 - if (rc == 2)
550 - mark_rebuild |= META_CHART_UPDATED;
551 -
552 - if (unlikely(st->priority != priority)) {
553 - st->priority = priority;
554 - mark_rebuild |= META_CHART_UPDATED;
555 - }
556 - if (unlikely(st->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE && st->update_every != update_every)) {
557 - st->update_every = update_every;
558 - mark_rebuild |= META_CHART_UPDATED;
559 - }
560 -
561 - if(plugin && *plugin) {
562 - STRING *old_plugin = st->plugin_name;
563 - st->plugin_name = rrd_string_strdupz(plugin);
564 - if (old_plugin != st->plugin_name)
565 - mark_rebuild |= META_PLUGIN_UPDATED;
566 - string_freez(old_plugin);
567 - }
568 -
569 - if(module && *module) {
570 - STRING *old_module = st->module_name;
571 - st->module_name = rrd_string_strdupz(module);
572 - if (old_module != st->module_name)
573 - mark_rebuild |= META_MODULE_UPDATED;
574 - string_freez(old_module);
575 - }
576 -
577 - if(title && *title) {
578 - STRING *old_title = st->title;
579 - st->title = rrd_string_strdupz(title);
580 - if(old_title != st->title)
581 - mark_rebuild |= META_CHART_UPDATED;
582 - string_freez(old_title);
583 - }
584 -
585 - if(units && *units) {
586 - STRING *old_units = st->units;
587 - st->units = rrd_string_strdupz(units);
588 - if(old_units != st->units)
589 - mark_rebuild |= META_CHART_UPDATED;
590 - string_freez(old_units);
591 - }
592 -
593 - if(context && *context) {
594 - STRING *old_context = st->context;
595 - st->context = rrd_string_strdupz(context);
596 - if(old_context != st->context)
597 - mark_rebuild |= META_CHART_UPDATED;
598 - string_freez(old_context);
599 - }
600 -
601 - if (st->chart_type != chart_type) {
602 - st->chart_type = chart_type;
603 - mark_rebuild |= META_CHART_UPDATED;
604 - }
605 -
606 - if (mark_rebuild) {
607 - rrdset_flag_clear(st, RRDSET_FLAG_ACLK);
608 - if (mark_rebuild != META_CHART_ACTIVATED) {
609 - info("Collector updated metadata for chart %s", rrdset_id(st));
610 - sched_yield();
611 - }
612 - }
613 - if (mark_rebuild & (META_CHART_UPDATED | META_PLUGIN_UPDATED | META_MODULE_UPDATED)) {
614 - debug(D_METADATALOG, "CHART [%s] metadata updated", rrdset_id(st));
615 - rc = update_chart_metadata(st->chart_uuid, st, id, name);
616 - if (unlikely(rc))
617 - error_report("Failed to update chart metadata in the database");
618 -
619 - if (!changed_from_archived_to_active) {
620 - rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
621 - rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
622 - }
623 - }
624 - /* Fall-through during switch from archived to active so that the host lock is taken and health is linked */
625 - if (!changed_from_archived_to_active) {
626 - rrdset_update_permanent_labels(st);
627 - rrdcontext_updated_rrdset(st);
628 - return st;
629 - }
630 - }
631 -
632 - rrdhost_wrlock(host);
633 -
634 - st = rrdset_find_on_create(host, fullid);
635 - if(st) {
636 - if (changed_from_archived_to_active) {
637 - rrdset_flag_clear(st, RRDSET_FLAG_ARCHIVED);
638 - rrdsetvar_create(st, "last_collected_t", RRDVAR_TYPE_TIME_T, &st->last_collected_time.tv_sec, RRDVAR_OPTION_DEFAULT);
639 - rrdsetvar_create(st, "collected_total_raw", RRDVAR_TYPE_TOTAL, &st->last_collected_total, RRDVAR_OPTION_DEFAULT);
640 - rrdsetvar_create(st, "green", RRDVAR_TYPE_CALCULATED, &st->green, RRDVAR_OPTION_DEFAULT);
641 - rrdsetvar_create(st, "red", RRDVAR_TYPE_CALCULATED, &st->red, RRDVAR_OPTION_DEFAULT);
642 - rrdsetvar_create(st, "update_every", RRDVAR_TYPE_INT, &st->update_every, RRDVAR_OPTION_DEFAULT);
643 - rrdsetcalc_link_matching(st);
644 - rrdcalctemplate_link_matching(st);
645 - }
646 - rrdhost_unlock(host);
647 - rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
648 - rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
649 - rrdcontext_updated_rrdset(st);
650 - return st;
651 - }
652 -
653 - // ------------------------------------------------------------------------
654 - // get the options from the config, we need to create it
655 -
656 - long entries = 5;
657 - if (memory_mode != RRD_MEMORY_MODE_DBENGINE)
658 - entries = align_entries_to_pagesize(memory_mode, history_entries);
659 -
660 - char *cache_dir = rrdset_cache_dir(host, fullid);
798 + char full_id[RRD_ID_LENGTH_MAX + 1];
799 + snprintfz(full_id, RRD_ID_LENGTH_MAX, "%s.%s", type, id);
800
801 // ------------------------------------------------------------------------
663 - // load it or allocate it
802 + // allocate it
803
804 debug(D_RRD_CALLS, "Creating RRD_STATS for '%s.%s'.", type, id);
805
667 - st = callocz(1, sizeof(RRDSET));
668 -
669 - st->id = string_strdupz(fullid); // fullid is already json_fix'ed
670 -
671 - st->rrdhost = host;
672 - st->cache_dir = cache_dir;
673 - st->entries = entries;
674 - st->update_every = update_every;
675 -
676 - if(memory_mode == RRD_MEMORY_MODE_SAVE || memory_mode == RRD_MEMORY_MODE_MAP) {
677 - if(!rrdset_memory_load_or_create_map_save(st, memory_mode)) {
678 - info("Failed to use memory mode %s for chart '%s', falling back to ram", (memory_mode == RRD_MEMORY_MODE_MAP)?"map":"save", rrdset_name(st));
679 - memory_mode = RRD_MEMORY_MODE_RAM;
680 - }
681 - }
682 - st->rrd_memory_mode = memory_mode;
683 -
684 - st->plugin_name = rrd_string_strdupz(plugin);
685 - st->module_name = rrd_string_strdupz(module);
686 - st->chart_type = chart_type;
687 - st->type = rrd_string_strdupz(type);
688 - st->family = family ? rrd_string_strdupz(family) : string_dup(st->type);
689 -
690 - if(strcmp(rrdset_id(st), ML_ANOMALY_RATES_CHART_ID) == 0)
691 - rrdset_flag_set(st, RRDSET_FLAG_ANOMALY_RATE_CHART);
692 -
693 - st->units = rrd_string_strdupz(units);
694 -
695 - st->context = (context && *context) ? rrd_string_strdupz(context) : string_dup(st->id);
696 -
697 - st->priority = priority;
698 -
699 - rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
700 -
701 - st->green = NAN;
702 - st->red = NAN;
703 -
704 - st->gap_when_lost_iterations_above = (int) (gap_when_lost_iterations_above + 2);
705 -
706 - st->rrddim_root_index = dictionary_create(
707 - DICTIONARY_FLAG_NAME_LINK_DONT_CLONE
708 - | DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE
709 - | DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
710 - );
711 -
712 - st->rrdvar_root_index = dictionary_create(
713 - DICTIONARY_FLAG_NAME_LINK_DONT_CLONE
714 - | DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE
715 - | DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
716 - );
717 -
718 - netdata_rwlock_init(&st->rrdset_rwlock);
719 - st->rrdlabels = rrdlabels_create();
720 - rrdset_update_permanent_labels(st);
721 -
722 - if(name && *name && rrdset_set_name(st, name))
723 - // we did set the name
724 - ;
725 - else
726 - // could not use the name, use the id
727 - rrdset_set_name(st, id);
728 -
729 - st->title = rrd_string_strdupz(title);
730 -
731 - st->rrdfamily = rrdfamily_create(host, rrdset_family(st));
732 -
733 - DOUBLE_LINKED_LIST_APPEND_UNSAFE(host->rrdset_root, st, prev, next);
734 -
735 - if(host->health_enabled) {
736 - rrdsetvar_create(st, "last_collected_t", RRDVAR_TYPE_TIME_T, &st->last_collected_time.tv_sec, RRDVAR_OPTION_DEFAULT);
737 - rrdsetvar_create(st, "collected_total_raw", RRDVAR_TYPE_TOTAL, &st->last_collected_total, RRDVAR_OPTION_DEFAULT);
738 - rrdsetvar_create(st, "green", RRDVAR_TYPE_CALCULATED, &st->green, RRDVAR_OPTION_DEFAULT);
739 - rrdsetvar_create(st, "red", RRDVAR_TYPE_CALCULATED, &st->red, RRDVAR_OPTION_DEFAULT);
740 - rrdsetvar_create(st, "update_every", RRDVAR_TYPE_INT, &st->update_every, RRDVAR_OPTION_DEFAULT);
741 - }
742 -
743 - rrdset_index_add(host, st);
744 - rrdsetcalc_link_matching(st);
745 - rrdcalctemplate_link_matching(st);
746 -
747 - st->chart_uuid = find_chart_uuid(host, type, id, name);
748 - if (unlikely(!st->chart_uuid))
749 - st->chart_uuid = create_chart_uuid(st, id, name);
750 - else
751 - update_chart_metadata(st->chart_uuid, st, id, name);
752 -
753 - store_active_chart(st->chart_uuid);
754 - compute_chart_hash(st);
755 -
756 - rrdhost_unlock(host);
757 - rrdcontext_updated_rrdset(st);
806 + struct rrdset_constructor tmp = {
807 + .host = host,
808 + .type = type,
809 + .id = id,
810 + .name = name,
811 + .family = family,
812 + .context = context,
813 + .title = title,
814 + .units = units,
815 + .plugin = plugin,
816 + .module = module,
817 + .priority = priority,
818 + .update_every = update_every,
819 + .chart_type = chart_type,
820 + .memory_mode = memory_mode,
821 + .history_entries = history_entries,
822 + };
823 +
824 + RRDSET *st = rrdset_index_add(host, full_id, &tmp);
825 return(st);
826 }
827
761 -
828 // ----------------------------------------------------------------------------
829 // RRDSET - data collection iteration control
830
@@ -1030,8 +1096,36 @@ static void store_metric(RRDDIM *rd, usec_t point_end_time_ut, NETDATA_DOUBLE n,
1096 rrdcontext_collected_rrddim(rd);
1097 }
1098
1099 +struct rda_item {
1100 + const DICTIONARY_ITEM *item;
1101 + RRDDIM *rd;
1102 +};
1103 +
1104 +static __thread struct rda_item *thread_rda = NULL;
1105 +static __thread size_t thread_rda_entries = 0;
1106 +
1107 +struct rda_item *rrdset_thread_rda(size_t *dimensions) {
1108 +
1109 + if(unlikely(!thread_rda || (*dimensions) > thread_rda_entries)) {
1110 + freez(thread_rda);
1111 + thread_rda = mallocz((*dimensions) * sizeof(struct rda_item));
1112 + thread_rda_entries = *dimensions;
1113 + }
1114 +
1115 + *dimensions = thread_rda_entries;
1116 + return thread_rda;
1117 +}
1118 +
1119 +void rrdset_thread_rda_free(void) {
1120 + freez(thread_rda);
1121 + thread_rda = NULL;
1122 + thread_rda_entries = 0;
1123 +}
1124 +
1125 static inline size_t rrdset_done_interpolate(
1126 RRDSET *st
1127 + , struct rda_item *rda_base
1128 + , size_t rda_slots
1129 , usec_t update_every_ut
1130 , usec_t last_stored_ut
1131 , usec_t next_store_ut
@@ -1068,9 +1162,11 @@ static inline size_t rrdset_done_interpolate(
1162
1163 last_ut = next_store_ut;
1164
1071 - rrddim_foreach_read(rd, st) {
1072 - if (rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED))
1073 - continue;
1165 + struct rda_item *rda;
1166 + size_t dim_id;
1167 + for(dim_id = 0, rda = rda_base, rd = rda->rd ; dim_id < rda_slots ; ++dim_id, ++rda) {
1168 + rd = rda->rd;
1169 + if(unlikely(!rd)) continue;
1170
1171 NETDATA_DOUBLE new_value;
1172
@@ -1226,6 +1322,7 @@ static inline void rrdset_done_fill_the_gap(RRDSET *st) {
1322 #endif
1323 }
1324 }
1325 + rrddim_foreach_done(rd);
1326
1327 if(c > 0) {
1328 c--;
@@ -1258,13 +1355,14 @@ void rrdset_done(RRDSET *st) {
1355
1356 netdata_thread_disable_cancelability();
1357
1261 - // a read lock is OK here
1262 - rrdset_rdlock(st);
1263 -
1358 #ifdef ENABLE_ACLK
1265 - if (likely(!rrdset_is_ar_chart(st))) {
1266 - if (unlikely(!rrdset_flag_check(st, RRDSET_FLAG_ACLK))) {
1267 - if (likely(st->dimensions && st->counter_done && !queue_chart_to_aclk(st))) {
1359 + time_t mark = now_realtime_sec();
1360 + bool rrdset_flag_aclk = rrdset_flag_check(st, RRDSET_FLAG_ACLK);
1361 + bool rrdset_is_ar = rrdset_is_ar_chart(st);
1362 +
1363 + if (likely(!rrdset_is_ar)) {
1364 + if (unlikely(!rrdset_flag_aclk)) {
1365 + if (likely(rrdset_number_of_dimensions(st) && st->counter_done && !queue_chart_to_aclk(st))) {
1366 rrdset_flag_set(st, RRDSET_FLAG_ACLK);
1367 }
1368 }
@@ -1414,6 +1512,42 @@ after_first_database_work:
1512 if(unlikely(st->rrdhost->rrdpush_send_enabled))
1513 rrdset_done_push(st);
1514
1515 + size_t rda_slots = dictionary_entries(st->rrddim_root_index);
1516 + struct rda_item *rda_base = rrdset_thread_rda(&rda_slots);
1517 +
1518 + size_t dim_id;
1519 + size_t dimensions = 0;
1520 + struct rda_item *rda = rda_base;
1521 + st->collected_total = 0;
1522 + rrddim_foreach_read(rd, st) {
1523 + if(rd_dfe.counter >= rda_slots)
1524 + break;
1525 +
1526 + rda = &rda_base[dimensions++];
1527 +
1528 + if(rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED)) {
1529 + rda->item = NULL;
1530 + rda->rd = NULL;
1531 + continue;
1532 + }
1533 +
1534 + // store the dimension in the array
1535 + rda->item = dictionary_acquired_item_dup(st->rrddim_root_index, rd_dfe.item);
1536 + rda->rd = dictionary_acquired_item_value(rda->item);
1537 +
1538 + // calculate totals
1539 + if(likely(rd->updated)) {
1540 + st->collected_total += rd->collected_value;
1541 +
1542 + if(unlikely(rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE))) {
1543 + error("Dimension %s in chart '%s' has the OBSOLETE flag set, but it is collected.", rrddim_name(rd), rrdset_id(st));
1544 + rrddim_isnot_obsolete(st, rd);
1545 + }
1546 + }
1547 + }
1548 + rrddim_foreach_done(rd);
1549 + rda_slots = dimensions;
1550 +
1551 if (unlikely(st->rrd_memory_mode == RRD_MEMORY_MODE_NONE))
1552 goto after_second_database_work;
1553
@@ -1424,38 +1558,20 @@ after_first_database_work:
1558 rrdset_debug(st, "next_store_ut = %0.3" NETDATA_DOUBLE_MODIFIER " (next interpolation point)", (NETDATA_DOUBLE)next_store_ut/USEC_PER_SEC);
1559 #endif
1560
1427 - // calculate totals and count the dimensions
1428 - int dimensions = 0;
1429 - st->collected_total = 0;
1430 - rrddim_foreach_read(rd, st) {
1431 - if (rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED))
1432 - continue;
1433 -
1434 - dimensions++;
1435 -
1436 - if(likely(rd->updated))
1437 - st->collected_total += rd->collected_value;
1438 - }
1439 -
1561 uint32_t has_reset_value = 0;
1562
1563 // process all dimensions to calculate their values
1564 // based on the collected figures only
1565 // at this stage we do not interpolate anything
1445 - rrddim_foreach_read(rd, st) {
1446 - if (rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED))
1447 - continue;
1566 + for(dim_id = 0, rda = rda_base, rd = rda->rd ; dim_id < rda_slots ; ++dim_id, ++rda) {
1567 + rd = rda->rd;
1568 + if(unlikely(!rd)) continue;
1569
1570 if(unlikely(!rd->updated)) {
1571 rd->calculated_value = 0;
1572 continue;
1573 }
1574
1454 - if(unlikely(rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE))) {
1455 - error("Dimension %s in chart '%s' has the OBSOLETE flag set, but it is collected.", rrddim_name(rd), rrdset_id(st));
1456 - rrddim_isnot_obsolete(st, rd);
1457 - }
1458 -
1575 #ifdef NETDATA_INTERNAL_CHECKS
1576 rrdset_debug(st, "%s: START "
1577 " last_collected_value = " COLLECTED_NUMBER_FORMAT
@@ -1530,7 +1646,7 @@ after_first_database_work:
1646 , rd->last_collected_value
1647 , rd->collected_value);
1648
1533 - if(!(rrddim_flag_check(rd, RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS)))
1649 + if(!(rrddim_option_check(rd, RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS)))
1650 has_reset_value = 1;
1651
1652 uint64_t last = (uint64_t)rd->last_collected_value;
@@ -1599,7 +1715,7 @@ after_first_database_work:
1715 , rd->collected_value
1716 );
1717
1602 - if(!(rrddim_flag_check(rd, RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS)))
1718 + if(!(rrddim_option_check(rd, RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS)))
1719 has_reset_value = 1;
1720
1721 rd->last_collected_value = rd->collected_value;
@@ -1669,7 +1785,10 @@ after_first_database_work:
1785 // }
1786 // #endif
1787
1672 - rrdset_done_interpolate(st
1788 + rrdset_done_interpolate(
1789 + st
1790 + , rda_base
1791 + ,rda_slots
1792 , update_every_ut
1793 , last_stored_ut
1794 , next_store_ut
@@ -1682,17 +1801,13 @@ after_first_database_work:
1801 after_second_database_work:
1802 st->last_collected_total = st->collected_total;
1803
1685 -#ifdef ENABLE_ACLK
1686 - time_t mark = now_realtime_sec();
1687 -#endif
1688 -
1689 - rrddim_foreach_read(rd, st) {
1690 - if (rrddim_flag_check(rd, RRDDIM_FLAG_ARCHIVED))
1691 - continue;
1804 + for(dim_id = 0, rda = rda_base, rd = rda->rd ; dim_id < rda_slots ; ++dim_id, ++rda) {
1805 + rd = rda->rd;
1806 + if(unlikely(!rd)) continue;
1807
1808 #ifdef ENABLE_ACLK
1694 - if (likely(!rrdset_is_ar_chart(st))) {
1695 - if (!rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN) && likely(rrdset_flag_check(st, RRDSET_FLAG_ACLK)))
1809 + if (likely(!rrdset_is_ar)) {
1810 + if (!rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN) && likely(rrdset_flag_aclk))
1811 queue_dimension_to_aclk(rd, calc_dimension_liveness(rd, mark));
1812 }
1813 #endif
@@ -1760,97 +1875,25 @@ after_second_database_work:
1875 // update the memory mapped files with the latest values
1876
1877 rrdset_memory_file_update(st);
1763 - rrddim_foreach_read(rd, st) {
1878 +
1879 + for(dim_id = 0, rda = rda_base, rd = rda->rd ; dim_id < rda_slots ; ++dim_id, ++rda) {
1880 + rd = rda->rd;
1881 + if(unlikely(!rd)) continue;
1882 rrddim_memory_file_update(rd);
1883 }
1884 }
1885
1768 - // find if there are any obsolete dimensions
1769 - if(unlikely(rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS))) {
1770 - rrddim_foreach_read(rd, st)
1771 - if(unlikely(rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)))
1772 - break;
1773 -
1774 - if(unlikely(rd)) {
1775 - time_t now = now_realtime_sec();
1776 -
1777 - RRDDIM *last;
1778 - // there is a dimension to free
1779 - // upgrade our read lock to a write lock
1780 - rrdset_unlock(st);
1781 - rrdset_wrlock(st);
1782 -
1783 - for( rd = st->dimensions, last = NULL ; likely(rd) ; ) {
1784 - if(unlikely(rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE) && !rrddim_flag_check(rd, RRDDIM_FLAG_ACLK)
1785 - && (rd->last_collected_time.tv_sec + rrdset_free_obsolete_time < now))) {
1786 - info("Removing obsolete dimension '%s' (%s) of '%s' (%s).", rrddim_name(rd), rrddim_id(rd), rrdset_name(st), rrdset_id(st));
1787 -
1788 - const char *cache_filename = rrddim_cache_filename(rd);
1789 - if(cache_filename) {
1790 - info("Deleting dimension file '%s'.", cache_filename);
1791 - if (unlikely(unlink(cache_filename) == -1))
1792 - error("Cannot delete dimension file '%s'", cache_filename);
1793 - }
1886 + for(dim_id = 0, rda = rda_base, rd = rda->rd ; dim_id < rda_slots ; ++dim_id, ++rda) {
1887 + rd = rda->rd;
1888 + if(unlikely(!rd)) continue;
1889
1795 -#ifdef ENABLE_DBENGINE
1796 - if (rd->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
1797 - rrddim_flag_set(rd, RRDDIM_FLAG_ARCHIVED);
1798 - while(rd->variables)
1799 - rrddimvar_free(rd->variables);
1800 -
1801 - rrddim_flag_clear(rd, RRDDIM_FLAG_OBSOLETE);
1802 - /* only a collector can mark a chart as obsolete, so we must remove the reference */
1803 -
1804 - size_t tiers_available = 0, tiers_said_yes = 0;
1805 - for(int tier = 0; tier < storage_tiers ;tier++) {
1806 - if(rd->tiers[tier]) {
1807 - tiers_available++;
1808 -
1809 - if(rd->tiers[tier]->collect_ops.finalize(rd->tiers[tier]->db_collection_handle))
1810 - tiers_said_yes++;
1811 -
1812 - rd->tiers[tier]->db_collection_handle = NULL;
1813 - }
1814 - }
1815 -
1816 - if (tiers_available == tiers_said_yes && tiers_said_yes) {
1817 - /* This metric has no data and no references */
1818 - delete_dimension_uuid(&rd->metric_uuid);
1819 - } else {
1820 - /* Do not delete this dimension */
1821 -#ifdef ENABLE_ACLK
1822 - queue_dimension_to_aclk(rd, calc_dimension_liveness(rd, mark));
1823 -#endif
1824 - last = rd;
1825 - rd = rd->next;
1826 - continue;
1827 - }
1828 - }
1829 -#endif
1830 - if(unlikely(!last)) {
1831 - rrddim_free(st, rd);
1832 - rd = st->dimensions;
1833 - continue;
1834 - }
1835 - else {
1836 - rrddim_free(st, rd);
1837 - rd = last->next;
1838 - continue;
1839 - }
1840 - }
1841 -
1842 - last = rd;
1843 - rd = rd->next;
1844 - }
1845 - }
1846 - else {
1847 - rrdset_flag_clear(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS);
1848 - }
1890 + dictionary_acquired_item_release(st->rrddim_root_index, rda->item);
1891 + rda->item = NULL;
1892 + rda->rd = NULL;
1893 }
1894
1895 rrdcontext_collected_rrdset(st);
1896
1853 - rrdset_unlock(st);
1897 netdata_thread_enable_cancelability();
1898 }
1899
database/rrdsetvar.c
+210 -95
@@ -1,170 +1,268 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 -#define NETDATA_HEALTH_INTERNALS
3 #include "rrd.h"
4
6 -// ----------------------------------------------------------------------------
7 -// RRDSETVAR management
8 -// CHART VARIABLES
5 +typedef struct rrdsetvar {
6 + STRING *name; // variable name
7 + void *value; // we need this to maintain the allocation for custom chart variables
8
10 -static inline void rrdsetvar_free_variables(RRDSETVAR *rs) {
11 - RRDSET *st = rs->rrdset;
9 + const RRDVAR_ACQUIRED *rrdvar_local;
10 + const RRDVAR_ACQUIRED *rrdvar_family_chart_id;
11 + const RRDVAR_ACQUIRED *rrdvar_family_chart_name;
12 + const RRDVAR_ACQUIRED *rrdvar_host_chart_id;
13 + const RRDVAR_ACQUIRED *rrdvar_host_chart_name;
14 +
15 + RRDVAR_FLAGS flags:24;
16 + RRDVAR_TYPE type:8;
17 +} RRDSETVAR;
18 +
19 +// should only be called while the rrdsetvar dict is write locked
20 +// otherwise, 2+ threads may be setting the same variables at the same time
21 +static inline void rrdsetvar_free_rrdvars_unsafe(RRDSET *st, RRDSETVAR *rs) {
22 RRDHOST *host = st->rrdhost;
23
24 // ------------------------------------------------------------------------
25 // CHART
16 - rrdvar_free(host, st->rrdvar_root_index, rs->var_local);
17 - rs->var_local = NULL;
26 +
27 + if(st->rrdvars) {
28 + rrdvar_release_and_del(st->rrdvars, rs->rrdvar_local);
29 + rs->rrdvar_local = NULL;
30 + }
31
32 // ------------------------------------------------------------------------
33 // FAMILY
21 - rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family);
22 - rs->var_family = NULL;
34
24 - rrdvar_free(host, st->rrdfamily->rrdvar_root_index, rs->var_family_name);
25 - rs->var_family_name = NULL;
35 + if(st->rrdfamily) {
36 + rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_chart_id);
37 + rs->rrdvar_family_chart_id = NULL;
38 +
39 + rrdvar_release_and_del(rrdfamily_rrdvars_dict(st->rrdfamily), rs->rrdvar_family_chart_name);
40 + rs->rrdvar_family_chart_name = NULL;
41 + }
42
43 // ------------------------------------------------------------------------
44 // HOST
29 - rrdvar_free(host, host->rrdvar_root_index, rs->var_host);
30 - rs->var_host = NULL;
45
32 - rrdvar_free(host, host->rrdvar_root_index, rs->var_host_name);
33 - rs->var_host_name = NULL;
46 + if(host->rrdvars && host->health_enabled) {
47 + rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_id);
48 + rs->rrdvar_host_chart_id = NULL;
49
35 - // ------------------------------------------------------------------------
36 - // KEYS
37 - string_freez(rs->key_fullid);
38 - rs->key_fullid = NULL;
39 -
40 - string_freez(rs->key_fullname);
41 - rs->key_fullname = NULL;
50 + rrdvar_release_and_del(host->rrdvars, rs->rrdvar_host_chart_name);
51 + rs->rrdvar_host_chart_name = NULL;
52 + }
53 }
54
44 -static inline void rrdsetvar_create_variables(RRDSETVAR *rs) {
45 - RRDSET *st = rs->rrdset;
55 +// should only be called while the rrdsetvar dict is write locked
56 +// otherwise, 2+ threads may be setting the same variables at the same time
57 +static inline void rrdsetvar_update_rrdvars_unsafe(RRDSET *st, RRDSETVAR *rs) {
58 RRDHOST *host = st->rrdhost;
59
48 - RRDVAR_OPTIONS options = rs->options;
49 - if(rs->options & RRDVAR_OPTION_ALLOCATED)
50 - options &= ~ RRDVAR_OPTION_ALLOCATED;
60 + RRDVAR_FLAGS options = rs->flags;
61 + options &= ~RRDVAR_OPTIONS_REMOVED_WHEN_PROPAGATING_TO_RRDVAR;
62
63 // ------------------------------------------------------------------------
64 // free the old ones (if any)
65
55 - rrdsetvar_free_variables(rs);
66 + rrdsetvar_free_rrdvars_unsafe(st, rs);
67
68 // ------------------------------------------------------------------------
69 // KEYS
70
71 char buffer[RRDVAR_MAX_LENGTH + 1];
61 - snprintfz(buffer, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_id(st), string2str(rs->variable));
62 - rs->key_fullid = string_strdupz(buffer);
72 + snprintfz(buffer, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_id(st), string2str(rs->name));
73 + STRING *key_chart_id = string_strdupz(buffer);
74
64 - snprintfz(buffer, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_name(st), string2str(rs->variable));
65 - rs->key_fullname = string_strdupz(buffer);
75 + snprintfz(buffer, RRDVAR_MAX_LENGTH, "%s.%s", rrdset_name(st), string2str(rs->name));
76 + STRING *key_chart_name = string_strdupz(buffer);
77
78 // ------------------------------------------------------------------------
79 // CHART
69 - rs->var_local = rrdvar_create_and_index("local", st->rrdvar_root_index, rs->variable, rs->type, options, rs->value);
80 +
81 + if(st->rrdvars) {
82 + rs->rrdvar_local = rrdvar_add_and_acquire("local", st->rrdvars, rs->name, rs->type, options, rs->value);
83 + }
84
85 // ------------------------------------------------------------------------
86 // FAMILY
73 - rs->var_family = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_fullid, rs->type, options, rs->value);
74 - rs->var_family_name = rrdvar_create_and_index("family", st->rrdfamily->rrdvar_root_index, rs->key_fullname, rs->type, options, rs->value);
87 +
88 + if(st->rrdfamily) {
89 + rs->rrdvar_family_chart_id = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_chart_id, rs->type, options, rs->value);
90 + rs->rrdvar_family_chart_name = rrdvar_add_and_acquire("family", rrdfamily_rrdvars_dict(st->rrdfamily), key_chart_name, rs->type, options, rs->value);
91 + }
92
93 // ------------------------------------------------------------------------
94 // HOST
78 - rs->var_host = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullid, rs->type, options, rs->value);
79 - rs->var_host_name = rrdvar_create_and_index("host", host->rrdvar_root_index, rs->key_fullname, rs->type, options, rs->value);
80 -}
95
82 -RRDSETVAR *rrdsetvar_create(RRDSET *st, const char *variable, RRDVAR_TYPE type, void *value, RRDVAR_OPTIONS options) {
83 - debug(D_VARIABLES, "RRDVARSET create for chart id '%s' name '%s' with variable name '%s'", rrdset_id(st), rrdset_name(st), variable);
84 - RRDSETVAR *rs = (RRDSETVAR *)callocz(1, sizeof(RRDSETVAR));
96 + if(host->rrdvars && host->health_enabled) {
97 + rs->rrdvar_host_chart_id = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_id, rs->type, options, rs->value);
98 + rs->rrdvar_host_chart_name = rrdvar_add_and_acquire("host", host->rrdvars, key_chart_name, rs->type, options, rs->value);
99 + }
100
86 - rs->variable = string_strdupz(variable);
87 - rs->type = type;
88 - rs->value = value;
89 - rs->options = options;
90 - rs->rrdset = st;
101 + // free the keys
102 + string_freez(key_chart_id);
103 + string_freez(key_chart_name);
104 +}
105
92 - DOUBLE_LINKED_LIST_PREPEND_UNSAFE(st->variables, rs, prev, next);
106 +static void rrdsetvar_free_value_unsafe(RRDSETVAR *rs) {
107 + if(rs->flags & RRDVAR_FLAG_ALLOCATED) {
108 + void *old = rs->value;
109 + rs->value = NULL;
110 + rs->flags &= ~RRDVAR_FLAG_ALLOCATED;
111 + freez(old);
112 + }
113 +}
114
94 - rrdsetvar_create_variables(rs);
115 +static void rrdsetvar_set_value_unsafe(RRDSETVAR *rs, void *new_value) {
116 + rrdsetvar_free_value_unsafe(rs);
117
96 - return rs;
118 + if(new_value)
119 + rs->value = new_value;
120 + else {
121 + NETDATA_DOUBLE *n = mallocz(sizeof(NETDATA_DOUBLE));
122 + *n = NAN;
123 + rs->value = n;
124 + rs->flags |= RRDVAR_FLAG_ALLOCATED;
125 + }
126 }
127
99 -void rrdsetvar_rename_all(RRDSET *st) {
100 - debug(D_VARIABLES, "RRDSETVAR rename for chart id '%s' name '%s'", rrdset_id(st), rrdset_name(st));
128 +struct rrdsetvar_constructor {
129 + RRDSET *rrdset;
130 + const char *name;
131 + void *value;
132 + RRDVAR_FLAGS flags :16;
133 + RRDVAR_TYPE type:8;
134 +};
135
102 - RRDSETVAR *rs;
103 - for(rs = st->variables; rs ; rs = rs->next)
104 - rrdsetvar_create_variables(rs);
136 +static void rrdsetvar_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdsetvar, void *constructor_data) {
137 + RRDSETVAR *rs = rrdsetvar;
138 + struct rrdsetvar_constructor *ctr = constructor_data;
139 +
140 + ctr->flags &= ~RRDVAR_OPTIONS_REMOVED_ON_NEW_OBJECTS;
141 +
142 + rs->name = string_strdupz(ctr->name);
143 + rs->type = ctr->type;
144 + rs->flags = ctr->flags;
145 + rrdsetvar_set_value_unsafe(rs, ctr->value);
146
106 - rrdsetcalc_link_matching(st);
147 + // create the rrdvariables while we are having a write lock to the dictionary
148 + rrdsetvar_update_rrdvars_unsafe(ctr->rrdset, rs);
149 }
150
109 -void rrdsetvar_free(RRDSETVAR *rs) {
110 - RRDSET *st = rs->rrdset;
111 - debug(D_VARIABLES, "RRDSETVAR free for chart id '%s' name '%s', variable '%s'", rrdset_id(st), rrdset_name(st), string2str(rs->variable));
151 +static bool rrdsetvar_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdsetvar, void *new_rrdsetvar __maybe_unused, void *constructor_data) {
152 + RRDSETVAR *rs = rrdsetvar;
153 + struct rrdsetvar_constructor *ctr = constructor_data;
154
113 - DOUBLE_LINKED_LIST_REMOVE_UNSAFE(st->variables, rs, prev, next);
155 + ctr->flags &= ~RRDVAR_OPTIONS_REMOVED_ON_NEW_OBJECTS;
156
115 - rrdsetvar_free_variables(rs);
157 + RRDVAR_FLAGS options = rs->flags;
158 + options &= ~RRDVAR_OPTIONS_REMOVED_ON_NEW_OBJECTS;
159
117 - string_freez(rs->variable);
160 + if(((ctr->value == NULL && rs->value != NULL && rs->flags & RRDVAR_FLAG_ALLOCATED) || (rs->value == ctr->value))
161 + && ctr->flags == options && rs->type == ctr->type) {
162 + // don't reset it - everything is the same, or as it should...
163 + return false;
164 + }
165 +
166 + internal_error(true, "RRDSETVAR: resetting variable '%s' of chart '%s' of host '%s', options from 0x%x to 0x%x, type from %d to %d",
167 + string2str(rs->name), rrdset_id(ctr->rrdset), rrdhost_hostname(ctr->rrdset->rrdhost),
168 + options, ctr->flags, rs->type, ctr->type);
169
119 - if(rs->options & RRDVAR_OPTION_ALLOCATED)
120 - freez(rs->value);
170 + rrdsetvar_free_value_unsafe(rs); // we are going to change the options, so free it before setting it
171 + rs->flags = ctr->flags;
172 + rs->type = ctr->type;
173 + rrdsetvar_set_value_unsafe(rs, ctr->value);
174
122 - freez(rs);
175 + // recreate the rrdvariables while we are having a write lock to the dictionary
176 + rrdsetvar_update_rrdvars_unsafe(ctr->rrdset, rs);
177 + return true;
178 }
179
125 -// --------------------------------------------------------------------------------------------------------------------
126 -// custom chart variables
180 +static void rrdsetvar_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdsetvar, void *rrdset __maybe_unused) {
181 + RRDSET *st = rrdset;
182 + RRDSETVAR *rs = rrdsetvar;
183
128 -RRDSETVAR *rrdsetvar_custom_chart_variable_create(RRDSET *st, const char *name) {
129 - RRDHOST *host = st->rrdhost;
184 + rrdsetvar_free_rrdvars_unsafe(st, rs);
185 + rrdsetvar_free_value_unsafe(rs);
186 + string_freez(rs->name);
187 + rs->name = NULL;
188 +}
189 +
190 +void rrdsetvar_index_init(RRDSET *st) {
191 + if(!st->rrdsetvar_root_index) {
192 + st->rrdsetvar_root_index = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
193
131 - STRING *n = rrdvar_name_to_string(name);
194 + dictionary_register_insert_callback(st->rrdsetvar_root_index, rrdsetvar_insert_callback, NULL);
195 + dictionary_register_conflict_callback(st->rrdsetvar_root_index, rrdsetvar_conflict_callback, NULL);
196 + dictionary_register_delete_callback(st->rrdsetvar_root_index, rrdsetvar_delete_callback, st);
197 + }
198 +}
199
133 - rrdset_wrlock(st);
200 +void rrdsetvar_index_destroy(RRDSET *st) {
201 + dictionary_destroy(st->rrdsetvar_root_index);
202 + st->rrdsetvar_root_index = NULL;
203 +}
204 +
205 +const RRDSETVAR_ACQUIRED *rrdsetvar_add_and_acquire(RRDSET *st, const char *name, RRDVAR_TYPE type, void *value, RRDVAR_FLAGS flags) {
206 + struct rrdsetvar_constructor tmp = {
207 + .name = name,
208 + .type = type,
209 + .value = value,
210 + .flags = flags,
211 + .rrdset = st,
212 + };
213 +
214 + const RRDSETVAR_ACQUIRED *rsa = (const RRDSETVAR_ACQUIRED *)dictionary_set_and_acquire_item_advanced(st->rrdsetvar_root_index, name, -1, NULL, sizeof(RRDSETVAR), &tmp);
215 + return rsa;
216 +}
217 +
218 +void rrdsetvar_add_and_leave_released(RRDSET *st, const char *name, RRDVAR_TYPE type, void *value, RRDVAR_FLAGS flags) {
219 + const RRDSETVAR_ACQUIRED *rsa = rrdsetvar_add_and_acquire(st, name, type, value, flags);
220 + dictionary_acquired_item_release(st->rrdsetvar_root_index, (const DICTIONARY_ITEM *)rsa);
221 +}
222 +
223 +void rrdsetvar_rename_all(RRDSET *st) {
224 + debug(D_VARIABLES, "RRDSETVAR rename for chart id '%s' name '%s'", rrdset_id(st), rrdset_name(st));
225
135 - // find it
226 RRDSETVAR *rs;
137 - for(rs = st->variables; rs ; rs = rs->next) {
138 - if(rs->variable == n) {
139 - rrdset_unlock(st);
140 - if(rs->options & RRDVAR_OPTION_CUSTOM_CHART_VAR) {
141 - string_freez(n);
142 - return rs;
143 - }
144 - else {
145 - error("RRDSETVAR: custom variable '%s' on chart '%s' of host '%s', conflicts with an internal chart variable", string2str(n), rrdset_id(st), rrdhost_hostname(host));
146 - string_freez(n);
147 - return NULL;
148 - }
149 - }
227 + dfe_start_write(st->rrdsetvar_root_index, rs) {
228 + // should only be called while the rrdsetvar dict is write locked
229 + rrdsetvar_update_rrdvars_unsafe(st, rs);
230 }
231 + dfe_done(rs);
232
152 - // not found, allocate one
233 + rrdcalc_link_matching_alerts_to_rrdset(st);
234 +}
235 +
236 +void rrdsetvar_release_and_delete_all(RRDSET *st) {
237 + RRDSETVAR *rs;
238 + dfe_start_write(st->rrdsetvar_root_index, rs) {
239 + dictionary_del_advanced(st->rrdsetvar_root_index, string2str(rs->name), (ssize_t)string_strlen(rs->name) + 1);
240 + }
241 + dfe_done(rs);
242 +}
243
154 - NETDATA_DOUBLE *v = mallocz(sizeof(NETDATA_DOUBLE));
155 - *v = NAN;
244 +void rrdsetvar_release(DICTIONARY *dict, const RRDSETVAR_ACQUIRED *rsa) {
245 + dictionary_acquired_item_release(dict, (const DICTIONARY_ITEM *)rsa);
246 +}
247
157 - rs = rrdsetvar_create(st, string2str(n), RRDVAR_TYPE_CALCULATED, v, RRDVAR_OPTION_ALLOCATED|RRDVAR_OPTION_CUSTOM_CHART_VAR);
158 - rrdset_unlock(st);
248 +// --------------------------------------------------------------------------------------------------------------------
249 +// custom chart variables
250
160 - string_freez(n);
251 +const RRDSETVAR_ACQUIRED *rrdsetvar_custom_chart_variable_add_and_acquire(RRDSET *st, const char *name) {
252 + STRING *name_string = rrdvar_name_to_string(name);
253 + const RRDSETVAR_ACQUIRED *rs = rrdsetvar_add_and_acquire(st, string2str(name_string), RRDVAR_TYPE_CALCULATED, NULL, RRDVAR_FLAG_CUSTOM_CHART_VAR);
254 + string_freez(name_string);
255 return rs;
256 }
257
164 -void rrdsetvar_custom_chart_variable_set(RRDSETVAR *rs, NETDATA_DOUBLE value) {
165 - if(rs->type != RRDVAR_TYPE_CALCULATED || !(rs->options & RRDVAR_OPTION_CUSTOM_CHART_VAR) || !(rs->options & RRDVAR_OPTION_ALLOCATED)) {
258 +void rrdsetvar_custom_chart_variable_set(RRDSET *st, const RRDSETVAR_ACQUIRED *rsa, NETDATA_DOUBLE value) {
259 + if(!rsa) return;
260 +
261 + RRDSETVAR *rs = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rsa);
262 +
263 + if(rs->type != RRDVAR_TYPE_CALCULATED || !(rs->flags & RRDVAR_FLAG_CUSTOM_CHART_VAR) || !(rs->flags & RRDVAR_FLAG_ALLOCATED)) {
264 error("RRDSETVAR: requested to set variable '%s' of chart '%s' on host '%s' to value " NETDATA_DOUBLE_FORMAT
167 - " but the variable is not a custom chart one.", string2str(rs->variable), rrdset_id(rs->rrdset), rrdhost_hostname(rs->rrdset->rrdhost), value);
265 + " but the variable is not a custom chart one (it has options 0x%x, value pointer %p). Ignoring request.", string2str(rs->name), rrdset_id(st), rrdhost_hostname(st->rrdhost), value, (uint32_t)rs->flags, rs->value);
266 }
267 else {
268 NETDATA_DOUBLE *v = rs->value;
@@ -172,7 +270,24 @@ void rrdsetvar_custom_chart_variable_set(RRDSETVAR *rs, NETDATA_DOUBLE value) {
270 *v = value;
271
272 // mark the chart to be sent upstream
175 - rrdset_flag_clear(rs->rrdset, RRDSET_FLAG_UPSTREAM_EXPOSED);
273 + rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
274 + }
275 + }
276 +}
277 +
278 +void rrdsetvar_print_to_streaming_custom_chart_variables(RRDSET *st, BUFFER *wb) {
279 + // send the chart local custom variables
280 + RRDSETVAR *rs;
281 + dfe_start_read(st->rrdsetvar_root_index, rs) {
282 + if(unlikely(rs->type == RRDVAR_TYPE_CALCULATED && rs->flags & RRDVAR_FLAG_CUSTOM_CHART_VAR)) {
283 + NETDATA_DOUBLE *value = (NETDATA_DOUBLE *) rs->value;
284 +
285 + buffer_sprintf(wb
286 + , "VARIABLE CHART %s = " NETDATA_DOUBLE_FORMAT "\n"
287 + , string2str(rs->name)
288 + , *value
289 + );
290 }
291 }
292 + dfe_done(rs);
293 }
database/rrdsetvar.h
+11 -25
@@ -11,34 +11,20 @@
11 // This means, there will be no speed penalty for using
12 // these variables
13
14 -struct rrdsetvar {
15 - STRING *variable; // variable name
14 +extern void rrdsetvar_index_init(RRDSET *st);
15 +extern void rrdsetvar_index_destroy(RRDSET *st);
16 +extern void rrdsetvar_release_and_delete_all(RRDSET *st);
17
17 - STRING *key_fullid; // chart type.chart id.variable
18 - STRING *key_fullname; // chart type.chart name.variable
18 +#define rrdsetvar_custom_chart_variable_release(st, rsa) rrdsetvar_release((st)->rrdsetvar_root_index, rsa)
19 +extern void rrdsetvar_release(DICTIONARY *dict, const RRDSETVAR_ACQUIRED *rsa);
20
20 - RRDVAR_TYPE type;
21 - void *value;
22 -
23 - RRDVAR_OPTIONS options;
24 -
25 - RRDVAR *var_local;
26 - RRDVAR *var_family;
27 - RRDVAR *var_host;
28 - RRDVAR *var_family_name;
29 - RRDVAR *var_host_name;
30 -
31 - struct rrdset *rrdset;
32 -
33 - struct rrdsetvar *next;
34 - struct rrdsetvar *prev;
35 -};
36 -
37 -extern RRDSETVAR *rrdsetvar_custom_chart_variable_create(RRDSET *st, const char *name);
38 -extern void rrdsetvar_custom_chart_variable_set(RRDSETVAR *rv, NETDATA_DOUBLE value);
21 +extern const RRDSETVAR_ACQUIRED *rrdsetvar_custom_chart_variable_add_and_acquire(RRDSET *st, const char *name);
22 +extern void rrdsetvar_custom_chart_variable_set(RRDSET *st, const RRDSETVAR_ACQUIRED *rsa, NETDATA_DOUBLE value);
23
24 extern void rrdsetvar_rename_all(RRDSET *st);
41 -extern RRDSETVAR *rrdsetvar_create(RRDSET *st, const char *variable, RRDVAR_TYPE type, void *value, RRDVAR_OPTIONS options);
42 -extern void rrdsetvar_free(RRDSETVAR *rs);
25 +extern const RRDSETVAR_ACQUIRED *rrdsetvar_add_and_acquire(RRDSET *st, const char *name, RRDVAR_TYPE type, void *value, RRDVAR_FLAGS flags);
26 +extern void rrdsetvar_add_and_leave_released(RRDSET *st, const char *name, RRDVAR_TYPE type, void *value, RRDVAR_FLAGS flags);
27 +
28 +extern void rrdsetvar_print_to_streaming_custom_chart_variables(RRDSET *st, BUFFER *wb);
29
30 #endif //NETDATA_RRDSETVAR_H
database/rrdvar.c
+155 -117
@@ -1,8 +1,19 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 -#define NETDATA_HEALTH_INTERNALS
3 #include "rrd.h"
4
5 +// the variables as stored in the variables indexes
6 +// there are 3 indexes:
7 +// 1. at each chart (RRDSET.rrdvar_root_index)
8 +// 2. at each context (RRDFAMILY.rrdvar_root_index)
9 +// 3. at each host (RRDHOST.rrdvar_root_index)
10 +typedef struct rrdvar {
11 + STRING *name;
12 + void *value;
13 + RRDVAR_FLAGS flags:24;
14 + RRDVAR_TYPE type:8;
15 +} RRDVAR;
16 +
17 // ----------------------------------------------------------------------------
18 // RRDVAR management
19
@@ -20,142 +31,153 @@ inline int rrdvar_fix_name(char *variable) {
31 return fixed;
32 }
33
23 -static inline RRDVAR *rrdvar_index_add(DICTIONARY *dict, RRDVAR *rv) {
24 - return dictionary_set(dict, rrdvar_name(rv), rv, sizeof(RRDVAR));
34 +inline STRING *rrdvar_name_to_string(const char *name) {
35 + char *variable = strdupz(name);
36 + rrdvar_fix_name(variable);
37 + STRING *name_string = string_strdupz(variable);
38 + freez(variable);
39 + return name_string;
40 }
41
27 -static inline RRDVAR *rrdvar_index_del(DICTIONARY *dict, RRDVAR *rv) {
28 - if(dictionary_del(dict, rrdvar_name(rv)) != 0) {
29 - error("Request to remove RRDVAR '%s' from index failed. Not Found.", rrdvar_name(rv));
30 - return NULL;
31 - }
42 +struct rrdvar_constructor {
43 + STRING *name;
44 + void *value;
45 + RRDVAR_FLAGS options:16;
46 + RRDVAR_TYPE type:8;
47
33 - return rv;
34 -}
48 + enum {
49 + RRDVAR_REACT_NONE = 0,
50 + RRDVAR_REACT_NEW = (1 << 0),
51 + } react_action;
52 +};
53
36 -static inline RRDVAR *rrdvar_index_find(DICTIONARY *dict, STRING *name) {
37 - return dictionary_get(dict, string2str(name));
38 -}
54 +static void rrdvar_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdvar, void *constructor_data) {
55 + RRDVAR *rv = rrdvar;
56 + struct rrdvar_constructor *ctr = constructor_data;
57
40 -inline void rrdvar_free(RRDHOST *host, DICTIONARY *dict, RRDVAR *rv) {
41 - (void)host;
58 + ctr->options &= ~RRDVAR_OPTIONS_REMOVED_ON_NEW_OBJECTS;
59
43 - if(!rv) return;
60 + rv->name = string_dup(ctr->name);
61 + rv->type = ctr->type;
62 + rv->flags = ctr->options;
63
45 - if(dict) {
46 - debug(D_VARIABLES, "Deleting variable '%s'", rrdvar_name(rv));
47 - if(unlikely(!rrdvar_index_del(dict, rv)))
48 - error("RRDVAR: Attempted to delete variable '%s' from host '%s', but it is not found.", rrdvar_name(rv), rrdhost_hostname(host));
64 + if(!ctr->value) {
65 + NETDATA_DOUBLE *v = mallocz(sizeof(NETDATA_DOUBLE));
66 + *v = NAN;
67 + rv->value = v;
68 + rv->flags |= RRDVAR_FLAG_ALLOCATED;
69 }
70 + else
71 + rv->value = ctr->value;
72
51 - if(rv->options & RRDVAR_OPTION_ALLOCATED)
73 + ctr->react_action = RRDVAR_REACT_NEW;
74 +}
75 +
76 +static void rrdvar_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdvar, void *nothing __maybe_unused) {
77 + RRDVAR *rv = rrdvar;
78 +
79 + if(rv->flags & RRDVAR_FLAG_ALLOCATED)
80 freez(rv->value);
81
82 string_freez(rv->name);
55 - freez(rv);
83 + rv->name = NULL;
84 }
85
58 -inline STRING *rrdvar_name_to_string(const char *name) {
59 - char *variable = strdupz(name);
60 - rrdvar_fix_name(variable);
61 - STRING *name_string = string_strdupz(variable);
62 - freez(variable);
63 - return name_string;
86 +DICTIONARY *rrdvariables_create(void) {
87 + DICTIONARY *dict = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
88 +
89 + dictionary_register_insert_callback(dict, rrdvar_insert_callback, NULL);
90 + dictionary_register_delete_callback(dict, rrdvar_delete_callback, NULL);
91 +
92 + return dict;
93 }
94
66 -inline RRDVAR *rrdvar_create_and_index(const char *scope __maybe_unused, DICTIONARY *dict, STRING *name,
67 - RRDVAR_TYPE type, RRDVAR_OPTIONS options, void *value) {
68 -
69 - RRDVAR *rv = rrdvar_index_find(dict, name);
70 - if(unlikely(!rv)) {
71 - debug(D_VARIABLES, "Variable '%s' not found in scope '%s'. Creating a new one.", string2str(name), scope);
72 -
73 - rv = callocz(1, sizeof(RRDVAR));
74 - rv->name = string_dup(name);
75 - rv->type = type;
76 - rv->options = options;
77 - rv->value = value;
78 - rv->last_updated = now_realtime_sec();
79 -
80 - RRDVAR *ret = rrdvar_index_add(dict, rv);
81 - if(unlikely(ret != rv)) {
82 - debug(D_VARIABLES, "Variable '%s' in scope '%s' already exists", string2str(name), scope);
83 - freez(rv);
84 - rv = NULL;
85 - }
86 - else
87 - debug(D_VARIABLES, "Variable '%s' created in scope '%s'", string2str(name), scope);
88 - }
89 - else {
90 - debug(D_VARIABLES, "Variable '%s' is already found in scope '%s'.", string2str(name), scope);
95 +void rrdvariables_destroy(DICTIONARY *dict) {
96 + dictionary_destroy(dict);
97 +}
98
92 - // this is important
93 - // it must return NULL - not the existing variable - or double-free will happen
94 - rv = NULL;
95 - }
99 +static inline const RRDVAR_ACQUIRED *rrdvar_get_and_acquire(DICTIONARY *dict, STRING *name) {
100 + return (const RRDVAR_ACQUIRED *)dictionary_get_and_acquire_item_advanced(dict, string2str(name), (ssize_t)string_strlen(name) + 1);
101 +}
102
97 - return rv;
103 +inline void rrdvar_release_and_del(DICTIONARY *dict, const RRDVAR_ACQUIRED *rva) {
104 + if(unlikely(!dict || !rva)) return;
105 +
106 + RRDVAR *rv = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rva);
107 +
108 + dictionary_del_advanced(dict, string2str(rv->name), (ssize_t)string_strlen(rv->name) + 1);
109 +
110 + dictionary_acquired_item_release(dict, (const DICTIONARY_ITEM *)rva);
111 }
112
100 -void rrdvar_free_remaining_variables(RRDHOST *host, DICTIONARY *dict) {
101 - RRDVAR *rv;
102 - dfe_start_reentrant(dict, rv) {
103 - rrdvar_free(host, dict, rv);
104 - }
105 - dfe_done(rv);
113 +inline const RRDVAR_ACQUIRED *rrdvar_add_and_acquire(const char *scope __maybe_unused, DICTIONARY *dict, STRING *name, RRDVAR_TYPE type, RRDVAR_FLAGS options, void *value) {
114 + if(unlikely(!dict || !name)) return NULL;
115 +
116 + struct rrdvar_constructor tmp = {
117 + .name = name,
118 + .value = value,
119 + .type = type,
120 + .options = options,
121 + .react_action = RRDVAR_REACT_NONE,
122 + };
123 + return (const RRDVAR_ACQUIRED *)dictionary_set_and_acquire_item_advanced(dict, string2str(name), (ssize_t)string_strlen(name) + 1, NULL, sizeof(RRDVAR), &tmp);
124 +}
125 +
126 +void rrdvar_delete_all(DICTIONARY *dict) {
127 + dictionary_flush(dict);
128 }
129
130 +
131 // ----------------------------------------------------------------------------
132 // CUSTOM HOST VARIABLES
133
111 -inline int rrdvar_walkthrough_read(DICTIONARY *dict, int (*callback)(const char *name, void *rrdvar, void *data), void *data) {
134 +inline int rrdvar_walkthrough_read(DICTIONARY *dict, int (*callback)(const DICTIONARY_ITEM *item, void *rrdvar, void *data), void *data) {
135 + if(unlikely(!dict)) return 0; // when health is not enabled
136 return dictionary_walkthrough_read(dict, callback, data);
137 }
138
115 -static RRDVAR *rrdvar_custom_variable_create(const char *scope, DICTIONARY *dict, const char *name) {
116 - NETDATA_DOUBLE *v = callocz(1, sizeof(NETDATA_DOUBLE));
117 - *v = NAN;
139 +const RRDVAR_ACQUIRED *rrdvar_custom_host_variable_add_and_acquire(RRDHOST *host, const char *name) {
140 + DICTIONARY *dict = host->rrdvars;
141 + if(unlikely(!dict)) return NULL; // when health is not enabled
142
143 STRING *name_string = rrdvar_name_to_string(name);
144
121 - RRDVAR *rv = rrdvar_create_and_index(scope, dict, name_string, RRDVAR_TYPE_CALCULATED, RRDVAR_OPTION_CUSTOM_HOST_VAR|RRDVAR_OPTION_ALLOCATED, v);
122 - if(unlikely(!rv)) {
123 - freez(v);
124 - debug(D_VARIABLES, "Requested variable '%s' already exists - possibly 2 plugins are updating it at the same time.", string2str(name_string));
125 -
126 - // find the existing one to return it
127 - rv = rrdvar_index_find(dict, name_string);
128 - }
145 + const RRDVAR_ACQUIRED *rva = rrdvar_add_and_acquire("host", dict, name_string, RRDVAR_TYPE_CALCULATED, RRDVAR_FLAG_CUSTOM_HOST_VAR, NULL);
146
147 string_freez(name_string);
131 -
132 - return rv;
148 + return rva;
149 }
150
135 -RRDVAR *rrdvar_custom_host_variable_create(RRDHOST *host, const char *name) {
136 - return rrdvar_custom_variable_create("host", host->rrdvar_root_index, name);
137 -}
151 +void rrdvar_custom_host_variable_set(RRDHOST *host, const RRDVAR_ACQUIRED *rva, NETDATA_DOUBLE value) {
152 + if(unlikely(!host->rrdvars || !rva)) return; // when health is not enabled
153
139 -void rrdvar_custom_host_variable_set(RRDHOST *host, RRDVAR *rv, NETDATA_DOUBLE value) {
140 - if(rv->type != RRDVAR_TYPE_CALCULATED || !(rv->options & RRDVAR_OPTION_CUSTOM_HOST_VAR) || !(rv->options & RRDVAR_OPTION_ALLOCATED))
141 - error("requested to set variable '%s' to value " NETDATA_DOUBLE_FORMAT " but the variable is not a custom one.", rrdvar_name(rv), value);
154 + if(rrdvar_type(rva) != RRDVAR_TYPE_CALCULATED || !(rrdvar_flags(rva) & (RRDVAR_FLAG_CUSTOM_HOST_VAR | RRDVAR_FLAG_ALLOCATED)))
155 + error("requested to set variable '%s' to value " NETDATA_DOUBLE_FORMAT " but the variable is not a custom one.", rrdvar_name(rva), value);
156 else {
157 + RRDVAR *rv = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rva);
158 NETDATA_DOUBLE *v = rv->value;
159 if(*v != value) {
160 *v = value;
161
147 - rv->last_updated = now_realtime_sec();
148 -
162 // if the host is streaming, send this variable upstream immediately
150 - rrdpush_sender_send_this_host_variable_now(host, rv);
163 + rrdpush_sender_send_this_host_variable_now(host, rva);
164 }
165 }
166 }
167
168 +void rrdvar_release(DICTIONARY *dict, const RRDVAR_ACQUIRED *rva) {
169 + if(unlikely(!dict || !rva)) return; // when health is not enabled
170 + dictionary_acquired_item_release(dict, (const DICTIONARY_ITEM *)rva);
171 +}
172 +
173 // ----------------------------------------------------------------------------
174 // RRDVAR lookup
175
158 -NETDATA_DOUBLE rrdvar2number(RRDVAR *rv) {
176 +NETDATA_DOUBLE rrdvar2number(const RRDVAR_ACQUIRED *rva) {
177 + if(unlikely(!rva)) return NAN;
178 +
179 + RRDVAR *rv = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rva);
180 +
181 switch(rv->type) {
182 case RRDVAR_TYPE_CALCULATED: {
183 NETDATA_DOUBLE *n = (NETDATA_DOUBLE *)rv->value;
@@ -164,17 +186,17 @@ NETDATA_DOUBLE rrdvar2number(RRDVAR *rv) {
186
187 case RRDVAR_TYPE_TIME_T: {
188 time_t *n = (time_t *)rv->value;
167 - return *n;
189 + return (NETDATA_DOUBLE)*n;
190 }
191
192 case RRDVAR_TYPE_COLLECTED: {
193 collected_number *n = (collected_number *)rv->value;
172 - return *n;
194 + return (NETDATA_DOUBLE)*n;
195 }
196
197 case RRDVAR_TYPE_TOTAL: {
198 total_number *n = (total_number *)rv->value;
177 - return *n;
199 + return (NETDATA_DOUBLE)*n;
200 }
201
202 case RRDVAR_TYPE_INT: {
@@ -193,23 +215,26 @@ int health_variable_lookup(STRING *variable, RRDCALC *rc, NETDATA_DOUBLE *result
215 if(!st) return 0;
216
217 RRDHOST *host = st->rrdhost;
196 - RRDVAR *rv;
218 + const RRDVAR_ACQUIRED *rva;
219
198 - rv = rrdvar_index_find(st->rrdvar_root_index, variable);
199 - if(rv) {
200 - *result = rrdvar2number(rv);
220 + rva = rrdvar_get_and_acquire(st->rrdvars, variable);
221 + if(rva) {
222 + *result = rrdvar2number(rva);
223 + dictionary_acquired_item_release(st->rrdvars, (const DICTIONARY_ITEM *)rva);
224 return 1;
225 }
226
204 - rv = rrdvar_index_find(st->rrdfamily->rrdvar_root_index, variable);
205 - if(rv) {
206 - *result = rrdvar2number(rv);
227 + rva = rrdvar_get_and_acquire(rrdfamily_rrdvars_dict(st->rrdfamily), variable);
228 + if(rva) {
229 + *result = rrdvar2number(rva);
230 + dictionary_acquired_item_release(rrdfamily_rrdvars_dict(st->rrdfamily), (const DICTIONARY_ITEM *)rva);
231 return 1;
232 }
233
210 - rv = rrdvar_index_find(host->rrdvar_root_index, variable);
211 - if(rv) {
212 - *result = rrdvar2number(rv);
234 + rva = rrdvar_get_and_acquire(host->rrdvars, variable);
235 + if(rva) {
236 + *result = rrdvar2number(rva);
237 + dictionary_acquired_item_release(host->rrdvars, (const DICTIONARY_ITEM *)rva);
238 return 1;
239 }
240
@@ -222,19 +247,19 @@ int health_variable_lookup(STRING *variable, RRDCALC *rc, NETDATA_DOUBLE *result
247 struct variable2json_helper {
248 BUFFER *buf;
249 size_t counter;
225 - RRDVAR_OPTIONS options;
250 + RRDVAR_FLAGS options;
251 };
252
228 -static int single_variable2json(const char *name __maybe_unused, void *entry, void *data) {
229 - struct variable2json_helper *helper = (struct variable2json_helper *)data;
230 - RRDVAR *rv = (RRDVAR *)entry;
231 - NETDATA_DOUBLE value = rrdvar2number(rv);
253 +static int single_variable2json_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry __maybe_unused, void *helper_data) {
254 + struct variable2json_helper *helper = (struct variable2json_helper *)helper_data;
255 + const RRDVAR_ACQUIRED *rva = (const RRDVAR_ACQUIRED *)item;
256 + NETDATA_DOUBLE value = rrdvar2number(rva);
257
233 - if (helper->options == RRDVAR_OPTION_DEFAULT || rv->options & helper->options) {
258 + if (helper->options == RRDVAR_FLAG_NONE || rrdvar_flags(rva) & helper->options) {
259 if(unlikely(isnan(value) || isinf(value)))
235 - buffer_sprintf(helper->buf, "%s\n\t\t\"%s\": null", helper->counter?",":"", rrdvar_name(rv));
260 + buffer_sprintf(helper->buf, "%s\n\t\t\"%s\": null", helper->counter?",":"", rrdvar_name(rva));
261 else
237 - buffer_sprintf(helper->buf, "%s\n\t\t\"%s\": %0.5" NETDATA_DOUBLE_MODIFIER, helper->counter?",":"", rrdvar_name(rv), (NETDATA_DOUBLE)value);
262 + buffer_sprintf(helper->buf, "%s\n\t\t\"%s\": %0.5" NETDATA_DOUBLE_MODIFIER, helper->counter?",":"", rrdvar_name(rva), (NETDATA_DOUBLE)value);
263
264 helper->counter++;
265 }
@@ -246,11 +271,10 @@ void health_api_v1_chart_custom_variables2json(RRDSET *st, BUFFER *buf) {
271 struct variable2json_helper helper = {
272 .buf = buf,
273 .counter = 0,
249 - .options = RRDVAR_OPTION_CUSTOM_CHART_VAR
250 - };
274 + .options = RRDVAR_FLAG_CUSTOM_CHART_VAR};
275
276 buffer_sprintf(buf, "{");
253 - rrdvar_walkthrough_read(st->rrdvar_root_index, single_variable2json, &helper);
277 + rrdvar_walkthrough_read(st->rrdvars, single_variable2json_callback, &helper);
278 buffer_strcat(buf, "\n\t\t\t}");
279 }
280
@@ -260,20 +284,34 @@ void health_api_v1_chart_variables2json(RRDSET *st, BUFFER *buf) {
284 struct variable2json_helper helper = {
285 .buf = buf,
286 .counter = 0,
263 - .options = RRDVAR_OPTION_DEFAULT
264 - };
287 + .options = RRDVAR_FLAG_NONE};
288
289 buffer_sprintf(buf, "{\n\t\"chart\": \"%s\",\n\t\"chart_name\": \"%s\",\n\t\"chart_context\": \"%s\",\n\t\"chart_variables\": {", rrdset_id(st), rrdset_name(st), rrdset_context(st));
267 - rrdvar_walkthrough_read(st->rrdvar_root_index, single_variable2json, &helper);
290 + rrdvar_walkthrough_read(st->rrdvars, single_variable2json_callback, &helper);
291
292 buffer_sprintf(buf, "\n\t},\n\t\"family\": \"%s\",\n\t\"family_variables\": {", rrdset_family(st));
293 helper.counter = 0;
271 - rrdvar_walkthrough_read(st->rrdfamily->rrdvar_root_index, single_variable2json, &helper);
294 + rrdvar_walkthrough_read(rrdfamily_rrdvars_dict(st->rrdfamily), single_variable2json_callback, &helper);
295
296 buffer_sprintf(buf, "\n\t},\n\t\"host\": \"%s\",\n\t\"host_variables\": {", rrdhost_hostname(host));
297 helper.counter = 0;
275 - rrdvar_walkthrough_read(host->rrdvar_root_index, single_variable2json, &helper);
298 + rrdvar_walkthrough_read(host->rrdvars, single_variable2json_callback, &helper);
299
300 buffer_strcat(buf, "\n\t}\n}\n");
301 }
302
303 +// ----------------------------------------------------------------------------
304 +// RRDVAR private members examination
305 +
306 +const char *rrdvar_name(const RRDVAR_ACQUIRED *rva) {
307 + return dictionary_acquired_item_name((const DICTIONARY_ITEM *)rva);
308 +}
309 +
310 +RRDVAR_FLAGS rrdvar_flags(const RRDVAR_ACQUIRED *rva) {
311 + RRDVAR *rv = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rva);
312 + return rv->flags;
313 +}
314 +RRDVAR_TYPE rrdvar_type(const RRDVAR_ACQUIRED *rva) {
315 + RRDVAR *rv = dictionary_acquired_item_value((const DICTIONARY_ITEM *)rva);
316 + return rv->type;
317 +}
database/rrdvar.h
+41 -33
@@ -11,35 +11,32 @@ typedef enum rrdvar_type {
11 RRDVAR_TYPE_COLLECTED = 3,
12 RRDVAR_TYPE_TOTAL = 4,
13 RRDVAR_TYPE_INT = 5
14 +
15 + // this is 8 bit
16 + // to increase it you have to set change the bitfield in
17 + // rrdvar, rrdsetvar, rrddimvar
18 } RRDVAR_TYPE;
19
20 typedef enum rrdvar_options {
17 - RRDVAR_OPTION_DEFAULT = 0,
18 - RRDVAR_OPTION_ALLOCATED = (1 << 0), // the value ptr is allocated (not a reference)
19 - RRDVAR_OPTION_CUSTOM_HOST_VAR = (1 << 1), // this is a custom host variable, not associated with a dimension
20 - RRDVAR_OPTION_CUSTOM_CHART_VAR = (1 << 2), // this is a custom chart variable, not associated with a dimension
21 - RRDVAR_OPTION_RRDCALC_LOCAL_VAR = (1 << 3), // this is a an alarm variable, attached to a chart
22 - RRDVAR_OPTION_RRDCALC_FAMILY_VAR = (1 << 4), // this is a an alarm variable, attached to a family
23 - RRDVAR_OPTION_RRDCALC_HOST_CHARTID_VAR = (1 << 5), // this is a an alarm variable, attached to the host, using the chart id
24 - RRDVAR_OPTION_RRDCALC_HOST_CHARTNAME_VAR = (1 << 6), // this is a an alarm variable, attached to the host, using the chart name
25 -} RRDVAR_OPTIONS;
26 -
27 -// the variables as stored in the variables indexes
28 -// there are 3 indexes:
29 -// 1. at each chart (RRDSET.rrdvar_root_index)
30 -// 2. at each context (RRDFAMILY.rrdvar_root_index)
31 -// 3. at each host (RRDHOST.rrdvar_root_index)
32 -struct rrdvar {
33 - STRING *name;
34 -
35 - void *value;
36 - time_t last_updated;
37 -
38 - RRDVAR_OPTIONS options:16;
39 - RRDVAR_TYPE type:8;
40 -};
41 -
42 -#define rrdvar_name(rv) string2str((rv)->name)
21 + RRDVAR_FLAG_NONE = 0,
22 + RRDVAR_FLAG_ALLOCATED = (1 << 0), // the value ptr is allocated (not a reference)
23 + RRDVAR_FLAG_CUSTOM_HOST_VAR = (1 << 1), // this is a custom host variable, not associated with a dimension
24 + RRDVAR_FLAG_CUSTOM_CHART_VAR = (1 << 2), // this is a custom chart variable, not associated with a dimension
25 + RRDVAR_FLAG_RRDCALC_LOCAL_VAR = (1 << 3), // this is a an alarm variable, attached to a chart
26 + RRDVAR_FLAG_RRDCALC_FAMILY_VAR = (1 << 4), // this is a an alarm variable, attached to a family
27 + RRDVAR_FLAG_RRDCALC_HOST_CHARTID_VAR = (1 << 5), // this is a an alarm variable, attached to the host, using the chart id
28 + RRDVAR_FLAG_RRDCALC_HOST_CHARTNAME_VAR = (1 << 6), // this is a an alarm variable, attached to the host, using the chart name
29 +
30 + // this is 24 bit
31 + // to increase it you have to set change the bitfield in
32 + // rrdvar, rrdsetvar, rrddimvar
33 +} RRDVAR_FLAGS;
34 +
35 +#define RRDVAR_OPTIONS_REMOVED_ON_NEW_OBJECTS \
36 + (RRDVAR_FLAG_ALLOCATED)
37 +
38 +#define RRDVAR_OPTIONS_REMOVED_WHEN_PROPAGATING_TO_RRDVAR \
39 + (RRDVAR_FLAG_ALLOCATED)
40
41 #define RRDVAR_MAX_LENGTH 1024
42
@@ -49,15 +46,26 @@ extern int rrdvar_fix_name(char *variable);
46
47 extern STRING *rrdvar_name_to_string(const char *name);
48
52 -extern RRDVAR *rrdvar_custom_host_variable_create(RRDHOST *host, const char *name);
53 -extern void rrdvar_custom_host_variable_set(RRDHOST *host, RRDVAR *rv, NETDATA_DOUBLE value);
54 -extern void rrdvar_free_remaining_variables(RRDHOST *host, DICTIONARY *dict);
49 +extern const RRDVAR_ACQUIRED *rrdvar_custom_host_variable_add_and_acquire(RRDHOST *host, const char *name);
50 +extern void rrdvar_custom_host_variable_set(RRDHOST *host, const RRDVAR_ACQUIRED *rva, NETDATA_DOUBLE value);
51 +
52 +extern int rrdvar_walkthrough_read(DICTIONARY *dict, int (*callback)(const DICTIONARY_ITEM *item, void *rrdvar, void *data), void *data);
53 +
54 +#define rrdvar_custom_host_variable_release(host, rva) rrdvar_release((host)->rrdvars, rva)
55 +extern void rrdvar_release(DICTIONARY *dict, const RRDVAR_ACQUIRED *rva);
56 +
57 +extern NETDATA_DOUBLE rrdvar2number(const RRDVAR_ACQUIRED *rva);
58 +
59 +extern const RRDVAR_ACQUIRED *rrdvar_add_and_acquire(const char *scope, DICTIONARY *dict, STRING *name, RRDVAR_TYPE type, RRDVAR_FLAGS options, void *value);
60 +extern void rrdvar_release_and_del(DICTIONARY *dict, const RRDVAR_ACQUIRED *rva);
61
56 -extern int rrdvar_walkthrough_read(DICTIONARY *dict, int (*callback)(const char *name, void *rrdvar, void *data), void *data);
62 +extern DICTIONARY *rrdvariables_create(void);
63 +extern void rrdvariables_destroy(DICTIONARY *dict);
64
58 -extern NETDATA_DOUBLE rrdvar2number(RRDVAR *rv);
65 +extern void rrdvar_delete_all(DICTIONARY *dict);
66
60 -extern RRDVAR *rrdvar_create_and_index(const char *scope, DICTIONARY *dict, STRING *name, RRDVAR_TYPE type, RRDVAR_OPTIONS options, void *value);
61 -extern void rrdvar_free(RRDHOST *host, DICTIONARY *dict, RRDVAR *rv);
67 +extern const char *rrdvar_name(const RRDVAR_ACQUIRED *rva);
68 +extern RRDVAR_FLAGS rrdvar_flags(const RRDVAR_ACQUIRED *rva);
69 +extern RRDVAR_TYPE rrdvar_type(const RRDVAR_ACQUIRED *rva);
70
71 #endif //NETDATA_RRDVAR_H
database/sqlite/sqlite_aclk.c
-3
@@ -1023,9 +1023,6 @@ void sql_check_aclk_table_list(struct aclk_database_worker_config *wc)
1023 db_execute("DELETE FROM dimension_delete WHERE host_id NOT IN (SELECT host_id FROM host) "
1024 " OR unixepoch() - date_created > 604800;");
1025
1026 - db_execute("DELETE FROM chart_hash WHERE CAST(last_used AS INT) < unixepoch() - 604800;");
1027 - db_execute("DELETE FROM chart_hash_map WHERE hash_id NOT IN (SELECT hash_id FROM chart_hash);");
1028 -
1026 return;
1027 }
1028
database/sqlite/sqlite_aclk_chart.c
+16 -20
@@ -150,7 +150,7 @@ int aclk_add_chart_event(struct aclk_database_worker_config *wc, struct aclk_dat
150 if (likely(claim_id)) {
151 struct chart_instance_updated chart_payload;
152 memset(&chart_payload, 0, sizeof(chart_payload));
153 - chart_payload.config_hash = get_str_from_uuid(&st->uuid);
153 + chart_payload.config_hash = get_str_from_uuid(&st->hash_uuid);
154 chart_payload.update_every = st->update_every;
155 chart_payload.memory_mode = st->rrd_memory_mode;
156 chart_payload.name = (char *)rrdset_name(st);
@@ -164,7 +164,7 @@ int aclk_add_chart_event(struct aclk_database_worker_config *wc, struct aclk_dat
164 size_t size;
165 char *payload = generate_chart_instance_updated(&size, &chart_payload);
166 if (likely(payload))
167 - rc = aclk_add_chart_payload(wc, st->chart_uuid, claim_id, ACLK_PAYLOAD_CHART, (void *) payload, size, NULL, 1);
167 + rc = aclk_add_chart_payload(wc, &st->chart_uuid, claim_id, ACLK_PAYLOAD_CHART, (void *) payload, size, NULL, 1);
168 freez(payload);
169 chart_instance_updated_destroy(&chart_payload);
170 }
@@ -588,24 +588,23 @@ void aclk_receive_chart_reset(struct aclk_database_worker_config *wc, struct acl
588
589 RRDHOST *host = wc->host;
590 if (likely(host)) {
591 - rrdhost_rdlock(host);
591 RRDSET *st;
593 - rrdset_foreach_read(st, host)
594 - {
595 - rrdset_rdlock(st);
592 + rrdset_foreach_read(st, host) {
593 rrdset_flag_clear(st, RRDSET_FLAG_ACLK);
594 RRDDIM *rd;
598 - rrddim_foreach_read(rd, st)
599 - {
595 + rrddim_foreach_read(rd, st) {
596 rrddim_flag_clear(rd, RRDDIM_FLAG_ACLK);
597 rd->aclk_live_status = (rd->aclk_live_status == 0);
598 }
603 - rrdset_unlock(st);
599 + rrddim_foreach_done(rd);
600 }
605 - rrdhost_unlock(host);
606 - } else
601 + rrdset_foreach_done(st);
602 + }
603 + else
604 error_report("ACLK synchronization thread for %s is not linked to HOST", wc->host_guid);
608 - } else {
605 +
606 + }
607 + else {
608 log_access(
609 "ACLK STA [%s (%s)]: RESTARTING CHART SYNC FROM SEQUENCE %" PRIu64,
610 wc->node_id,
@@ -1268,29 +1267,26 @@ void sql_check_chart_liveness(RRDSET *st) {
1267 if (unlikely(rrdset_is_ar_chart(st)))
1268 return;
1269
1271 - rrdset_rdlock(st);
1272 -
1273 - if (unlikely(!rrdset_flag_check(st, RRDSET_FLAG_ACLK))) {
1274 - rrdset_unlock(st);
1270 + if (unlikely(!rrdset_flag_check(st, RRDSET_FLAG_ACLK)))
1271 return;
1276 - }
1272
1273 if (unlikely(!rrdset_flag_check(st, RRDSET_FLAG_ACLK))) {
1279 - if (likely(st->dimensions && st->counter_done && !queue_chart_to_aclk(st))) {
1274 + if (likely(rrdset_number_of_dimensions(st) && st->counter_done && !queue_chart_to_aclk(st))) {
1275 debug(D_ACLK_SYNC,"Check chart liveness [%s] submit chart definition", rrdset_name(st));
1276 rrdset_flag_set(st, RRDSET_FLAG_ACLK);
1277 }
1278 }
1279 else
1280 debug(D_ACLK_SYNC,"Check chart liveness [%s] chart definition already submitted", rrdset_name(st));
1281 +
1282 time_t mark = now_realtime_sec();
1283
1284 debug(D_ACLK_SYNC,"Check chart liveness [%s] scanning dimensions", rrdset_name(st));
1285 rrddim_foreach_read(rd, st) {
1290 - if (!rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN))
1286 + if (!rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN))
1287 queue_dimension_to_aclk(rd, calc_dimension_liveness(rd, mark));
1288 }
1293 - rrdset_unlock(st);
1289 + rrddim_foreach_done(rd);
1290 }
1291
1292 // ST is read locked
database/sqlite/sqlite_aclk_node.c
+2 -3
@@ -12,7 +12,6 @@ DICTIONARY *collectors_from_charts(RRDHOST *host, DICTIONARY *dict) {
12 RRDSET *st;
13 char name[500];
14
15 - rrdhost_rdlock(host);
15 rrdset_foreach_read(st, host) {
16 if (rrdset_is_available_for_viewers(st)) {
17 struct collector_info col = {
@@ -23,7 +22,7 @@ DICTIONARY *collectors_from_charts(RRDHOST *host, DICTIONARY *dict) {
22 dictionary_set(dict, name, &col, sizeof(struct collector_info));
23 }
24 }
26 - rrdhost_unlock(host);
25 + rrdset_foreach_done(st);
26
27 return dict;
28 }
@@ -36,7 +35,7 @@ void sql_build_node_collectors(struct aclk_database_worker_config *wc)
35 return;
36
37 struct update_node_collectors upd_node_collectors;
39 - DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
38 + DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
39
40 upd_node_collectors.node_id = wc->node_id;
41 upd_node_collectors.claim_id = get_agent_claimid();
database/sqlite/sqlite_db_migration.c
+18
@@ -64,6 +64,15 @@ const char *database_migrate_v2_v3[] = {
64 NULL
65 };
66
67 +const char *database_migrate_v4_v5[] = {
68 + "DROP TABLE IF EXISTS chart_active;",
69 + "DROP TABLE IF EXISTS dimension_active;",
70 + "DROP TABLE IF EXISTS chart_hash;",
71 + "DROP TABLE IF EXISTS chart_hash_map;",
72 + "DROP VIEW IF EXISTS v_chart_hash;",
73 + NULL
74 +};
75 +
76 static int do_migration_v1_v2(sqlite3 *database, const char *name)
77 {
78 UNUSED(name);
@@ -116,6 +125,14 @@ static int do_migration_v3_v4(sqlite3 *database, const char *name)
125 return 0;
126 }
127
128 +static int do_migration_v4_v5(sqlite3 *database, const char *name)
129 +{
130 + UNUSED(name);
131 + info("Running \"%s\" database migration", name);
132 +
133 + return init_database_batch(database, DB_CHECK_NONE, 0, &database_migrate_v4_v5[0]);
134 +}
135 +
136 static int do_migration_noop(sqlite3 *database, const char *name)
137 {
138 UNUSED(database);
@@ -163,6 +180,7 @@ DATABASE_FUNC_MIGRATION_LIST migration_action[] = {
180 {.name = "v1 to v2", .func = do_migration_v1_v2},
181 {.name = "v2 to v3", .func = do_migration_v2_v3},
182 {.name = "v3 to v4", .func = do_migration_v3_v4},
183 + {.name = "v4 to v5", .func = do_migration_v4_v5},
184 // the terminator of this array
185 {.name = NULL, .func = NULL}
186 };
database/sqlite/sqlite_functions.c
+18 -363
@@ -3,7 +3,7 @@
3 #include "sqlite_functions.h"
4 #include "sqlite_db_migration.h"
5
6 -#define DB_METADATA_VERSION 4
6 +#define DB_METADATA_VERSION 5
7
8 const char *database_config[] = {
9 "CREATE TABLE IF NOT EXISTS host(host_id BLOB PRIMARY KEY, hostname TEXT NOT NULL, "
@@ -21,11 +21,6 @@ const char *database_config[] = {
21 "CREATE TABLE IF NOT EXISTS dimension(dim_id blob PRIMARY KEY, chart_id blob, id text, name text, "
22 "multiplier int, divisor int , algorithm int, options text);",
23
24 - "DROP TABLE IF EXISTS chart_active;",
25 - "DROP TABLE IF EXISTS dimension_active;",
26 -
27 - "CREATE TABLE IF NOT EXISTS chart_active(chart_id blob PRIMARY KEY, date_created int);",
28 - "CREATE TABLE IF NOT EXISTS dimension_active(dim_id blob primary key, date_created int);",
24 "CREATE TABLE IF NOT EXISTS metadata_migration(filename text, file_size, date_created int);",
25 "CREATE INDEX IF NOT EXISTS ind_d1 on dimension (chart_id, id, name);",
26 "CREATE INDEX IF NOT EXISTS ind_c1 on chart (host_id, id, type, name);",
@@ -46,36 +41,16 @@ const char *database_config[] = {
41 "CREATE TABLE IF NOT EXISTS host_label(host_id blob, source_type int, label_key text NOT NULL, "
42 "label_value text NOT NULL, date_created INT, PRIMARY KEY (host_id, label_key));",
43
49 - "CREATE TABLE IF NOT EXISTS chart_hash_map(chart_id blob , hash_id blob, UNIQUE (chart_id, hash_id));",
50 -
51 - "CREATE TABLE IF NOT EXISTS chart_hash(hash_id blob PRIMARY KEY,type text, id text, name text, "
52 - "family text, context text, title text, unit text, plugin text, "
53 - "module text, priority integer, chart_type, last_used);",
54 -
55 - "CREATE VIEW IF NOT EXISTS v_chart_hash as SELECT ch.*, chm.chart_id FROM chart_hash ch, chart_hash_map chm "
56 - "WHERE ch.hash_id = chm.hash_id;",
57 -
44 "CREATE TRIGGER IF NOT EXISTS ins_host AFTER INSERT ON host BEGIN INSERT INTO node_instance (host_id, date_created)"
45 " SELECT new.host_id, unixepoch() WHERE new.host_id NOT IN (SELECT host_id FROM node_instance); END;",
46
61 - "CREATE TRIGGER IF NOT EXISTS tr_v_chart_hash INSTEAD OF INSERT on v_chart_hash BEGIN "
62 - "INSERT INTO chart_hash (hash_id, type, id, name, family, context, title, unit, plugin, "
63 - "module, priority, chart_type, last_used) "
64 - "values (new.hash_id, new.type, new.id, new.name, new.family, new.context, new.title, new.unit, new.plugin, "
65 - "new.module, new.priority, new.chart_type, unixepoch()) "
66 - "ON CONFLICT (hash_id) DO UPDATE SET last_used = unixepoch(); "
67 - "INSERT INTO chart_hash_map (chart_id, hash_id) values (new.chart_id, new.hash_id) "
68 - "on conflict (chart_id, hash_id) do nothing; END; ",
69 -
47 NULL
48 };
49
50 const char *database_cleanup[] = {
74 - "delete from chart where chart_id not in (select chart_id from dimension);",
75 - "delete from host where host_id not in (select host_id from chart);",
76 - "delete from chart_label where chart_id not in (select chart_id from chart);",
77 - "DELETE FROM chart_hash_map WHERE chart_id NOT IN (SELECT chart_id FROM chart);",
78 - "DELETE FROM chart_hash WHERE hash_id NOT IN (SELECT hash_id FROM chart_hash_map);",
51 + "DELETE FROM chart WHERE chart_id NOT IN (SELECT chart_id FROM dimension);",
52 + "DELETE FROM host WHERE host_id NOT IN (SELECT host_id FROM chart);",
53 + "DELETE FROM chart_label WHERE chart_id NOT IN (SELECT chart_id FROM chart);",
54 "DELETE FROM node_instance WHERE host_id NOT IN (SELECT host_id FROM host);",
55 "DELETE FROM host_info WHERE host_id NOT IN (SELECT host_id FROM host);",
56 "DELETE FROM host_label WHERE host_id NOT IN (SELECT host_id FROM host);",
@@ -190,88 +165,6 @@ int prepare_statement(sqlite3 *database, char *query, sqlite3_stmt **statement)
165 return rc;
166 }
167
193 -/*
194 - * Store a chart or dimension UUID in chart_active or dimension_active
195 - * The statement that will be prepared determines that
196 - */
197 -
198 -static int store_active_uuid_object(sqlite3_stmt **res, char *statement, uuid_t *uuid)
199 -{
200 - int rc;
201 -
202 - // Check if we should need to prepare the statement
203 - if (!*res) {
204 - rc = prepare_statement(db_meta, statement, res);
205 - if (unlikely(rc != SQLITE_OK)) {
206 - error_report("Failed to prepare statement to store active object, rc = %d", rc);
207 - return rc;
208 - }
209 - }
210 -
211 - rc = sqlite3_bind_blob(*res, 1, uuid, sizeof(*uuid), SQLITE_STATIC);
212 - if (unlikely(rc != SQLITE_OK))
213 - error_report("Failed to bind input parameter to store active object, rc = %d", rc);
214 - else
215 - rc = execute_insert(*res);
216 - return rc;
217 -}
218 -
219 -/*
220 - * Marks a chart with UUID as active
221 - * Input: UUID
222 - */
223 -void store_active_chart(uuid_t *chart_uuid)
224 -{
225 - static __thread sqlite3_stmt *res = NULL;
226 - int rc;
227 -
228 - if (unlikely(!db_meta)) {
229 - if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE)
230 - error_report("Database has not been initialized");
231 - return;
232 - }
233 -
234 - if (unlikely(!chart_uuid))
235 - return;
236 -
237 - rc = store_active_uuid_object(&res, SQL_STORE_ACTIVE_CHART, chart_uuid);
238 - if (rc != SQLITE_DONE)
239 - error_report("Failed to store active chart, rc = %d", rc);
240 -
241 - rc = sqlite3_reset(res);
242 - if (unlikely(rc != SQLITE_OK))
243 - error_report("Failed to finalize statement in store active chart, rc = %d", rc);
244 - return;
245 -}
246 -
247 -/*
248 - * Marks a dimension with UUID as active
249 - * Input: UUID
250 - */
251 -void store_active_dimension(uuid_t *dimension_uuid)
252 -{
253 - static __thread sqlite3_stmt *res = NULL;
254 - int rc;
255 -
256 - if (unlikely(!db_meta)) {
257 - if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE)
258 - error_report("Database has not been initialized");
259 - return;
260 - }
261 -
262 - if (unlikely(!dimension_uuid))
263 - return;
264 -
265 - rc = store_active_uuid_object(&res, SQL_STORE_ACTIVE_DIMENSION, dimension_uuid);
266 - if (rc != SQLITE_DONE)
267 - error_report("Failed to store active dimension, rc = %d", rc);
268 -
269 - rc = sqlite3_reset(res);
270 - if (unlikely(rc != SQLITE_OK))
271 - error_report("Failed to finalize statement in store active dimension, rc = %d", rc);
272 - return;
273 -}
274 -
168 static int check_table_integrity_cb(void *data, int argc, char **argv, char **column)
169 {
170 int *status = data;
@@ -623,7 +516,7 @@ int find_dimension_uuid(RRDSET *st, RRDDIM *rd, uuid_t *store_uuid)
516 }
517 }
518
626 - rc = sqlite3_bind_blob(res, 1, st->chart_uuid, sizeof(*st->chart_uuid), SQLITE_STATIC);
519 + rc = sqlite3_bind_blob(res, 1, &st->chart_uuid, sizeof(st->chart_uuid), SQLITE_STATIC);
520 if (unlikely(rc != SQLITE_OK))
521 goto bind_fail;
522
@@ -642,7 +535,7 @@ int find_dimension_uuid(RRDSET *st, RRDDIM *rd, uuid_t *store_uuid)
535 }
536 else {
537 uuid_generate(*store_uuid);
645 - status = sql_store_dimension(store_uuid, st->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor, rd->algorithm);
538 + status = sql_store_dimension(store_uuid, &st->chart_uuid, rrddim_id(rd), rrddim_name(rd), rd->multiplier, rd->divisor, rd->algorithm);
539 if (unlikely(status))
540 error_report("Failed to store dimension metadata in the database");
541 }
@@ -697,20 +590,20 @@ bind_fail:
590 * Do a database lookup to find the UUID of a chart
591 *
592 */
700 -uuid_t *find_chart_uuid(RRDHOST *host, const char *type, const char *id, const char *name)
593 +int find_chart_uuid(RRDHOST *host, const char *type, const char *id, const char *name, uuid_t *store_uuid)
594 {
595 static __thread sqlite3_stmt *res = NULL;
703 - uuid_t *uuid = NULL;
596 int rc;
597 + int status = 1;
598
599 if (unlikely(!db_meta) && default_rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE)
707 - return NULL;
600 + return 1;
601
602 if (unlikely(!res)) {
603 rc = prepare_statement(db_meta, SQL_FIND_CHART_UUID, &res);
604 if (rc != SQLITE_OK) {
605 error_report("Failed to prepare statement to lookup chart UUID in the database");
713 - return NULL;
606 + return 1;
607 }
608 }
609
@@ -731,32 +624,23 @@ uuid_t *find_chart_uuid(RRDHOST *host, const char *type, const char *id, const c
624 goto bind_fail;
625
626 rc = sqlite3_step_monitored(res);
734 - if (likely(rc == SQLITE_ROW)) {
735 - uuid = mallocz(sizeof(uuid_t));
736 - uuid_copy(*uuid, sqlite3_column_blob(res, 0));
627 + if (likely(rc == SQLITE_ROW && sqlite3_column_bytes(res,0) == sizeof(*store_uuid))) {
628 + uuid_copy(*store_uuid, *((uuid_t *) sqlite3_column_blob(res, 0)));
629 + status = 0;
630 }
631
632 rc = sqlite3_reset(res);
633 if (unlikely(rc != SQLITE_OK))
634 error_report("Failed to reset statement when searching for a chart UUID, rc = %d", rc);
635
743 -#ifdef NETDATA_INTERNAL_CHECKS
744 - char uuid_str[GUID_LEN + 1];
745 - if (likely(uuid)) {
746 - uuid_unparse_lower(*uuid, uuid_str);
747 - debug(D_METADATALOG, "Found UUID %s for chart %s.%s", uuid_str, type, name ? name : id);
748 - }
749 - else
750 - debug(D_METADATALOG, "UUID not found for chart %s.%s", type, name ? name : id);
751 -#endif
752 - return uuid;
636 + return status;
637
638 bind_fail:
639 error_report("Failed to bind input parameter to perform chart UUID database lookup, rc = %d", rc);
640 rc = sqlite3_reset(res);
641 if (unlikely(rc != SQLITE_OK))
642 error_report("Failed to reset statement when searching for a chart UUID, rc = %d", rc);
759 - return NULL;
643 + return 1;
644 }
645
646 int update_chart_metadata(uuid_t *chart_uuid, RRDSET *st, const char *id, const char *name)
@@ -767,7 +651,8 @@ int update_chart_metadata(uuid_t *chart_uuid, RRDSET *st, const char *id, const
651 return 0;
652
653 rc = sql_store_chart(
770 - chart_uuid, &st->rrdhost->host_uuid, rrdset_type(st), id, name,
654 + chart_uuid, &st->rrdhost->host_uuid,
655 + rrdset_parts_type(st), id, name,
656 rrdset_family(st), rrdset_context(st), rrdset_title(st), rrdset_units(st),
657 rrdset_plugin_name(st), rrdset_module_name(st),
658 st->priority, st->update_every, st->chart_type,
@@ -776,28 +661,6 @@ int update_chart_metadata(uuid_t *chart_uuid, RRDSET *st, const char *id, const
661 return rc;
662 }
663
779 -uuid_t *create_chart_uuid(RRDSET *st, const char *id, const char *name)
780 -{
781 - uuid_t *uuid = NULL;
782 - int rc;
783 -
784 - uuid = mallocz(sizeof(uuid_t));
785 - uuid_generate(*uuid);
786 -
787 -#ifdef NETDATA_INTERNAL_CHECKS
788 - char uuid_str[GUID_LEN + 1];
789 - uuid_unparse_lower(*uuid, uuid_str);
790 - debug(D_METADATALOG,"Generating uuid [%s] for chart %s under host %s", uuid_str, rrdset_id(st), rrdhost_hostname(st->rrdhost));
791 -#endif
792 -
793 - rc = update_chart_metadata(uuid, st, id, name);
794 -
795 - if (unlikely(rc))
796 - error_report("Failed to store chart metadata in the database");
797 -
798 - return uuid;
799 -}
800 -
664 static int exec_statement_with_uuid(const char *sql, uuid_t *uuid)
665 {
666 int rc, result = 1;
@@ -1695,34 +1558,6 @@ skip_store:
1558 return rc != SQLITE_DONE;
1559 }
1560
1698 -#define SQL_INS_CHART_LABEL "insert or replace into chart_label " \
1699 - "(chart_id, source_type, label_key, label_value, date_created) " \
1700 - "values (@chart, @source, @label, @value, unixepoch());"
1701 -
1702 -void sql_store_chart_label(uuid_t *chart_uuid, int source_type, char *label, char *value)
1703 -{
1704 - static __thread sqlite3_stmt *res = NULL;
1705 - int rc;
1706 -
1707 - if (unlikely(!db_meta)) {
1708 - if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE)
1709 - error_report("Database has not been initialized");
1710 - return;
1711 - }
1712 -
1713 - if (unlikely(!res)) {
1714 - rc = prepare_statement(db_meta, SQL_INS_CHART_LABEL, &res);
1715 - if (unlikely(rc != SQLITE_OK)) {
1716 - error_report("Failed to prepare statement store chart labels");
1717 - return;
1718 - }
1719 - }
1720 -
1721 - sql_store_label(res, chart_uuid, source_type, label, value);
1722 -
1723 - return;
1724 -}
1725 -
1561 #define SQL_INS_HOST_LABEL "INSERT OR REPLACE INTO host_label " \
1562 "(host_id, source_type, label_key, label_value, date_created) " \
1563 "values (@chart, @source, @label, @value, unixepoch());"
@@ -1934,7 +1769,7 @@ void sql_build_context_param_list(ONEWAYALLOC *owa, struct context_param **para
1769 if (unlikely(!rd))
1770 continue;
1771 if (sqlite3_column_int(res, 9) == 1)
1937 - rrddim_flag_set(rd, RRDDIM_FLAG_HIDDEN);
1772 + rrddim_option_set(rd, RRDDIM_OPTION_HIDDEN);
1773 rd->next = (*param_list)->rd;
1774 (*param_list)->rd = rd;
1775 }
@@ -1964,186 +1799,6 @@ failed:
1799 }
1800
1801
1967 -/*
1968 - * Store a chart hash in the database
1969 - */
1970 -
1971 -#define SQL_STORE_CHART_HASH "insert into v_chart_hash (hash_id, type, id, " \
1972 - "name, family, context, title, unit, plugin, module, priority, chart_type, last_used, chart_id) " \
1973 - "values (?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11, ?12, unixepoch(), ?13);"
1974 -
1975 -int sql_store_chart_hash(
1976 - uuid_t *hash_id, uuid_t *chart_id, const char *type, const char *id, const char *name, const char *family,
1977 - const char *context, const char *title, const char *units, const char *plugin, const char *module, long priority,
1978 - RRDSET_TYPE chart_type)
1979 -{
1980 - static __thread sqlite3_stmt *res = NULL;
1981 - int rc, param = 0;
1982 -
1983 - if (unlikely(!db_meta)) {
1984 - if (default_rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE)
1985 - return 0;
1986 - error_report("Database has not been initialized");
1987 - return 1;
1988 - }
1989 -
1990 - if (unlikely(!res)) {
1991 - rc = prepare_statement(db_meta, SQL_STORE_CHART_HASH, &res);
1992 - if (unlikely(rc != SQLITE_OK)) {
1993 - error_report("Failed to prepare statement to store chart, rc = %d", rc);
1994 - return 1;
1995 - }
1996 - }
1997 -
1998 - param++;
1999 - rc = sqlite3_bind_blob(res, 1, hash_id, sizeof(*hash_id), SQLITE_STATIC);
2000 - if (unlikely(rc != SQLITE_OK))
2001 - goto bind_fail;
2002 -
2003 - param++;
2004 - rc = sqlite3_bind_text(res, 2, type, -1, SQLITE_STATIC);
2005 - if (unlikely(rc != SQLITE_OK))
2006 - goto bind_fail;
2007 -
2008 - param++;
2009 - rc = sqlite3_bind_text(res, 3, id, -1, SQLITE_STATIC);
2010 - if (unlikely(rc != SQLITE_OK))
2011 - goto bind_fail;
2012 -
2013 - param++;
2014 - if (name && *name)
2015 - rc = sqlite3_bind_text(res, 4, name, -1, SQLITE_STATIC);
2016 - else
2017 - rc = sqlite3_bind_null(res, 4);
2018 - if (unlikely(rc != SQLITE_OK))
2019 - goto bind_fail;
2020 -
2021 - param++;
2022 - rc = sqlite3_bind_text(res, 5, family, -1, SQLITE_STATIC);
2023 - if (unlikely(rc != SQLITE_OK))
2024 - goto bind_fail;
2025 -
2026 - param++;
2027 - rc = sqlite3_bind_text(res, 6, context, -1, SQLITE_STATIC);
2028 - if (unlikely(rc != SQLITE_OK))
2029 - goto bind_fail;
2030 -
2031 - param++;
2032 - rc = sqlite3_bind_text(res, 7, title, -1, SQLITE_STATIC);
2033 - if (unlikely(rc != SQLITE_OK))
2034 - goto bind_fail;
2035 -
2036 - param++;
2037 - rc = sqlite3_bind_text(res, 8, units, -1, SQLITE_STATIC);
2038 - if (unlikely(rc != SQLITE_OK))
2039 - goto bind_fail;
2040 -
2041 - param++;
2042 - rc = sqlite3_bind_text(res, 9, plugin, -1, SQLITE_STATIC);
2043 - if (unlikely(rc != SQLITE_OK))
2044 - goto bind_fail;
2045 -
2046 - param++;
2047 - rc = sqlite3_bind_text(res, 10, module, -1, SQLITE_STATIC);
2048 - if (unlikely(rc != SQLITE_OK))
2049 - goto bind_fail;
2050 -
2051 - param++;
2052 - rc = sqlite3_bind_int(res, 11, (int) priority);
2053 - if (unlikely(rc != SQLITE_OK))
2054 - goto bind_fail;
2055 -
2056 - param++;
2057 - rc = sqlite3_bind_int(res, 12, chart_type);
2058 - if (unlikely(rc != SQLITE_OK))
2059 - goto bind_fail;
2060 -
2061 - param++;
2062 - rc = sqlite3_bind_blob(res, 13, chart_id, sizeof(*chart_id), SQLITE_STATIC);
2063 - if (unlikely(rc != SQLITE_OK))
2064 - goto bind_fail;
2065 -
2066 - rc = execute_insert(res);
2067 - if (unlikely(rc != SQLITE_DONE))
2068 - error_report("Failed to store chart hash_id, rc = %d", rc);
2069 -
2070 - rc = sqlite3_reset(res);
2071 - if (unlikely(rc != SQLITE_OK))
2072 - error_report("Failed to reset statement in chart hash_id store function, rc = %d", rc);
2073 -
2074 - return 0;
2075 -
2076 - bind_fail:
2077 - error_report("Failed to bind parameter %d to store chart hash_id, rc = %d", param, rc);
2078 - rc = sqlite3_reset(res);
2079 - if (unlikely(rc != SQLITE_OK))
2080 - error_report("Failed to reset statement in chart hash_id store function, rc = %d", rc);
2081 - return 1;
2082 -}
2083 -
2084 -/*
2085 - chart hashes are used for cloud communication.
2086 - if cloud is disabled or openssl is not available (which will prevent cloud connectivity)
2087 - skip hash calculations
2088 -*/
2089 -void compute_chart_hash(RRDSET *st)
2090 -{
2091 -#if !defined DISABLE_CLOUD && defined ENABLE_HTTPS
2092 - EVP_MD_CTX *evpctx;
2093 - unsigned char hash_value[EVP_MAX_MD_SIZE];
2094 - unsigned int hash_len;
2095 - char priority_str[32];
2096 -
2097 - if (rrdhost_flag_check(st->rrdhost, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)) {
2098 - internal_error(true, "Skipping compute_chart_hash for host %s because context streaming is enabled", rrdhost_hostname(st->rrdhost));
2099 - return;
2100 - }
2101 -
2102 - sprintf(priority_str, "%ld", st->priority);
2103 -
2104 - evpctx = EVP_MD_CTX_create();
2105 - EVP_DigestInit_ex(evpctx, EVP_sha256(), NULL);
2106 - //EVP_DigestUpdate(evpctx, st->type, strlen(st->type));
2107 - EVP_DigestUpdate(evpctx, rrdset_id(st), string_strlen(st->id));
2108 - EVP_DigestUpdate(evpctx, rrdset_name(st), string_strlen(st->name));
2109 - EVP_DigestUpdate(evpctx, rrdset_family(st), string_strlen(st->family));
2110 - EVP_DigestUpdate(evpctx, rrdset_context(st), string_strlen(st->context));
2111 - EVP_DigestUpdate(evpctx, rrdset_title(st), string_strlen(st->title));
2112 - EVP_DigestUpdate(evpctx, rrdset_units(st), string_strlen(st->units));
2113 - EVP_DigestUpdate(evpctx, rrdset_plugin_name(st), string_strlen(st->plugin_name));
2114 - EVP_DigestUpdate(evpctx, rrdset_module_name(st), string_strlen(st->module_name));
2115 -// EVP_DigestUpdate(evpctx, priority_str, strlen(priority_str));
2116 - EVP_DigestUpdate(evpctx, &st->priority, sizeof(st->priority));
2117 - EVP_DigestUpdate(evpctx, &st->chart_type, sizeof(st->chart_type));
2118 - EVP_DigestFinal_ex(evpctx, hash_value, &hash_len);
2119 - EVP_MD_CTX_destroy(evpctx);
2120 - fatal_assert(hash_len > sizeof(uuid_t));
2121 -
2122 - char uuid_str[GUID_LEN + 1];
2123 - uuid_unparse_lower(*((uuid_t *) &hash_value), uuid_str);
2124 - //info("Calculating HASH %s for chart %s", uuid_str, st->name);
2125 - uuid_copy(st->uuid, *((uuid_t *) &hash_value));
2126 -
2127 - (void)sql_store_chart_hash(
2128 - (uuid_t *)&hash_value,
2129 - st->chart_uuid,
2130 - rrdset_type(st),
2131 - rrdset_id(st),
2132 - rrdset_name(st),
2133 - rrdset_family(st),
2134 - rrdset_context(st),
2135 - rrdset_title(st),
2136 - rrdset_units(st),
2137 - rrdset_plugin_name(st),
2138 - rrdset_module_name(st),
2139 - st->priority,
2140 - st->chart_type);
2141 -#else
2142 - UNUSED(st);
2143 -#endif
2144 - return;
2145 -}
2146 -
1802 #define SQL_STORE_CLAIM_ID "insert into node_instance " \
1803 "(host_id, claim_id, date_created) values (@host_id, @claim_id, unixepoch()) " \
1804 "on conflict(host_id) do update set claim_id = excluded.claim_id;"
database/sqlite/sqlite_functions.h
+1 -12
@@ -37,18 +37,12 @@ typedef enum db_check_action_type {
37 #define SQL_FIND_CHART_UUID \
38 "select chart_id from chart where host_id = @host and type=@type and id=@id and (name is null or name=@name) and chart_id is not null;"
39
40 -#define SQL_STORE_ACTIVE_CHART \
41 - "insert or replace into chart_active (chart_id, date_created) values (@id, unixepoch());"
42 -
40 #define SQL_STORE_DIMENSION \
41 "INSERT OR REPLACE into dimension (dim_id, chart_id, id, name, multiplier, divisor , algorithm) values (?0001,?0002,?0003,?0004,?0005,?0006,?0007);"
42
43 #define SQL_FIND_DIMENSION_UUID \
44 "select dim_id from dimension where chart_id=@chart and id=@id and name=@name and length(dim_id)=16;"
45
49 -#define SQL_STORE_ACTIVE_DIMENSION \
50 - "insert or replace into dimension_active (dim_id, date_created) values (@id, unixepoch());"
51 -
46 #define CHECK_SQLITE_CONNECTION(db_meta) \
47 if (unlikely(!db_meta)) { \
48 if (default_rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE) { \
@@ -83,12 +77,9 @@ extern int sql_store_dimension(uuid_t *dim_uuid, uuid_t *chart_uuid, const char
77 collected_number divisor, int algorithm);
78
79 extern int find_dimension_uuid(RRDSET *st, RRDDIM *rd, uuid_t *store_uuid);
86 -extern void store_active_dimension(uuid_t *dimension_uuid);
80
88 -extern uuid_t *find_chart_uuid(RRDHOST *host, const char *type, const char *id, const char *name);
89 -extern uuid_t *create_chart_uuid(RRDSET *st, const char *id, const char *name);
81 +extern int find_chart_uuid(RRDHOST *host, const char *type, const char *id, const char *name, uuid_t *store_uuid);
82 extern int update_chart_metadata(uuid_t *chart_uuid, RRDSET *st, const char *id, const char *name);
91 -extern void store_active_chart(uuid_t *dimension_uuid);
83
84 extern int find_uuid_type(uuid_t *uuid);
85
@@ -103,7 +94,6 @@ extern void add_migrated_file(char *path, uint64_t file_size);
94 extern void db_unlock(void);
95 extern void db_lock(void);
96 extern void delete_dimension_uuid(uuid_t *dimension_uuid);
106 -extern void sql_store_chart_label(uuid_t *chart_uuid, int source_type, char *label, char *value);
97 extern void sql_build_context_param_list(ONEWAYALLOC *owa, struct context_param **param_list, RRDHOST *host, char *context, char *chart);
98 extern void store_claim_id(uuid_t *host_id, uuid_t *claim_id);
99 extern int update_node_id(uuid_t *host_id, uuid_t *node_id);
@@ -112,7 +102,6 @@ extern int get_host_id(uuid_t *node_id, uuid_t *host_id);
102 extern void invalidate_node_instances(uuid_t *host_id, uuid_t *claim_id);
103 extern struct node_instance_list *get_node_list(void);
104 extern void sql_load_node_id(RRDHOST *host);
115 -extern void compute_chart_hash(RRDSET *st);
105 extern int sql_set_dimension_option(uuid_t *dim_uuid, char *option);
106 char *get_hostname_by_node_id(char *node_id);
107 void free_temporary_host(RRDHOST *host);
database/sqlite/sqlite_health.c
+9 -6
@@ -632,12 +632,15 @@ void sql_health_alarm_log_load(RRDHOST *host) {
632 return;
633 }
634
635 - netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
636 -
637 - DICTIONARY *all_rrdcalcs = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
635 + DICTIONARY *all_rrdcalcs = dictionary_create(
636 + DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
637 RRDCALC *rc;
639 - foreach_rrdcalc_in_rrdhost(host, rc)
638 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
639 dictionary_set(all_rrdcalcs, rrdcalc_name(rc), rc, sizeof(*rc));
640 + }
641 + foreach_rrdcalc_in_rrdhost_done(rc);
642 +
643 + netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
644
645 while (sqlite3_step_monitored(res) == SQLITE_ROW) {
646 ALARM_ENTRY *ae = NULL;
@@ -790,11 +793,11 @@ void sql_health_alarm_log_load(RRDHOST *host) {
793 loaded++;
794 }
795
796 + netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
797 +
798 dictionary_destroy(all_rrdcalcs);
799 all_rrdcalcs = NULL;
800
796 - netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
797 -
801 if(!host->health_max_unique_id) host->health_max_unique_id = (uint32_t)now_realtime_sec();
802 if(!host->health_max_alarm_id) host->health_max_alarm_id = (uint32_t)now_realtime_sec();
803
exporting/json/json.c
+2 -2
@@ -197,7 +197,7 @@ int format_dimension_collected_json_plaintext(struct instance *instance, RRDDIM
197 rrdset_name(st),
198 rrdset_family(st),
199 rrdset_context(st),
200 - rrdset_type(st),
200 + rrdset_parts_type(st),
201 rrdset_units(st),
202 rrddim_id(rd),
203 rrddim_name(rd),
@@ -281,7 +281,7 @@ int format_dimension_stored_json_plaintext(struct instance *instance, RRDDIM *rd
281 rrdset_name(st),
282 rrdset_family(st),
283 rrdset_context(st),
284 - rrdset_type(st),
284 + rrdset_parts_type(st),
285 rrdset_units(st),
286 rrddim_id(rd),
287 rrddim_name(rd),
exporting/process_data.c
+4 -8
@@ -338,26 +338,22 @@ void prepare_buffers(struct engine *engine)
338
339 rrd_rdlock();
340 RRDHOST *host;
341 - rrdhost_foreach_read(host)
342 - {
343 - rrdhost_rdlock(host);
341 + rrdhost_foreach_read(host) {
342 start_host_formatting(engine, host);
343 RRDSET *st;
346 - rrdset_foreach_read(st, host)
347 - {
348 - rrdset_rdlock(st);
344 + rrdset_foreach_read(st, host) {
345 start_chart_formatting(engine, st);
346
347 RRDDIM *rd;
348 rrddim_foreach_read(rd, st)
349 metric_formatting(engine, rd);
350 + rrddim_foreach_done(rd);
351
352 end_chart_formatting(engine, st);
356 - rrdset_unlock(st);
353 }
354 + rrdset_foreach_done(st);
355 variables_formatting(engine, host);
356 end_host_formatting(engine, host);
360 - rrdhost_unlock(host);
357 }
358 rrd_unlock();
359 netdata_thread_enable_cancelability();
exporting/prometheus/prometheus.c
+12 -16
@@ -349,12 +349,12 @@ struct host_variables_callback_options {
349 * @param data callback options.
350 * @return Returns 1 if the chart can be sent, 0 otherwise.
351 */
352 -static int print_host_variables(const char *name __maybe_unused, void *rv_ptr, void *data) {
353 - RRDVAR *rv = rv_ptr;
352 +static int print_host_variables_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rv_ptr __maybe_unused, void *data) {
353 + const RRDVAR_ACQUIRED *rv = (const RRDVAR_ACQUIRED *)item;
354
355 struct host_variables_callback_options *opts = data;
356
357 - if (rv->options & (RRDVAR_OPTION_CUSTOM_HOST_VAR | RRDVAR_OPTION_CUSTOM_CHART_VAR)) {
357 + if (rrdvar_flags(rv) & (RRDVAR_FLAG_CUSTOM_HOST_VAR | RRDVAR_FLAG_CUSTOM_CHART_VAR)) {
358 if (!opts->host_header_printed) {
359 opts->host_header_printed = 1;
360
@@ -519,7 +519,6 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
519 PROMETHEUS_OUTPUT_OPTIONS output_options)
520 {
521 SIMPLE_PATTERN *filter = simple_pattern_create(filter_string, NULL, SIMPLE_PATTERN_EXACT);
522 - rrdhost_rdlock(host);
522
523 char hostname[PROMETHEUS_ELEMENT_MAX + 1];
524 prometheus_label_copy(hostname, rrdhost_hostname(host), PROMETHEUS_ELEMENT_MAX);
@@ -564,17 +563,14 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
563 .host_header_printed = 0
564 };
565
567 - rrdvar_walkthrough_read(host->rrdvar_root_index, print_host_variables, &opts);
566 + rrdvar_walkthrough_read(host->rrdvars, print_host_variables_callback, &opts);
567 }
568
569 // for each chart
570 RRDSET *st;
572 - rrdset_foreach_read(st, host)
573 - {
571 + rrdset_foreach_read(st, host) {
572
573 if (likely(can_send_rrdset(instance, st, filter))) {
576 - rrdset_rdlock(st);
577 -
574 char chart[PROMETHEUS_ELEMENT_MAX + 1];
575 char context[PROMETHEUS_ELEMENT_MAX + 1];
576 char family[PROMETHEUS_ELEMENT_MAX + 1];
@@ -615,8 +611,7 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
611
612 // for each dimension
613 RRDDIM *rd;
618 - rrddim_foreach_read(rd, st)
619 - {
614 + rrddim_foreach_read(rd, st) {
615 if (rd->collections_counter && !rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)) {
616 char dimension[PROMETHEUS_ELEMENT_MAX + 1];
617 char *suffix = "";
@@ -665,7 +660,8 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
660 buffer_sprintf(wb, "# TYPE %s_%s%s %s\n", prefix, context, suffix, p.type);
661
662 generate_as_collected_prom_metric(wb, &p, homogeneous, prometheus_collector);
668 - } else {
663 + }
664 + else {
665 // the dimensions of the chart, do not have the same algorithm, multiplier or divisor
666 // we create a metric per dimension
667
@@ -683,7 +679,8 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
679
680 generate_as_collected_prom_metric(wb, &p, homogeneous, prometheus_collector);
681 }
686 - } else {
682 + }
683 + else {
684 // we need average or sum of the data
685
686 time_t first_time = instance->after;
@@ -750,12 +747,11 @@ static void rrd_stats_api_v1_charts_allmetrics_prometheus(
747 }
748 }
749 }
753 -
754 - rrdset_unlock(st);
750 + rrddim_foreach_done(rd);
751 }
752 }
753 + rrdset_foreach_done(st);
754
758 - rrdhost_unlock(host);
755 simple_pattern_free(filter);
756 }
757
exporting/prometheus/remote_write/remote_write.c
+4 -4
@@ -321,12 +321,12 @@ int format_dimension_prometheus_remote_write(struct instance *instance, RRDDIM *
321 return 0;
322 }
323
324 -static int format_variable_prometheus_remote_write_callback(const char *name_txt __maybe_unused, void *rv_ptr, void *data) {
325 - RRDVAR *rv = rv_ptr;
324 +static int format_variable_prometheus_remote_write_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rv_ptr __maybe_unused, void *data) {
325 + const RRDVAR_ACQUIRED *rv = (const RRDVAR_ACQUIRED *)item;
326
327 struct prometheus_remote_write_variables_callback_options *opts = data;
328
329 - if (rv->options & (RRDVAR_OPTION_CUSTOM_HOST_VAR | RRDVAR_OPTION_CUSTOM_CHART_VAR)) {
329 + if (rrdvar_flags(rv) & (RRDVAR_FLAG_CUSTOM_HOST_VAR | RRDVAR_FLAG_CUSTOM_CHART_VAR)) {
330 RRDHOST *host = opts->host;
331 struct instance *instance = opts->instance;
332 struct simple_connector_data *simple_connector_data =
@@ -363,7 +363,7 @@ int format_variables_prometheus_remote_write(struct instance *instance, RRDHOST
363 .now = now_realtime_usec(),
364 };
365
366 - return rrdvar_walkthrough_read(host->rrdvar_root_index, format_variable_prometheus_remote_write_callback, &opt);
366 + return rrdvar_walkthrough_read(host->rrdvars, format_variable_prometheus_remote_write_callback, &opt);
367 }
368
369 /**
exporting/tests/netdata_doubles.c
-8
@@ -242,14 +242,6 @@ void __mock_rrddim_query_finalize(struct rrddim_query_handle *handle)
242 function_called();
243 }
244
245 -void sql_store_chart_label(uuid_t *chart_uuid, int source_type, char *label, char *value)
246 -{
247 - (void)chart_uuid;
248 - (void)source_type;
249 - (void)label;
250 - (void)value;
251 -}
252 -
245 void rrdcalc_update_rrdlabels(RRDSET *st)
246 {
247 (void)st;
health/health.c
+78 -106
@@ -153,63 +153,42 @@ static void health_reload_host(RRDHOST *host) {
153 char *stock_path = health_stock_config_dir();
154
155 // free all running alarms
156 - rrdhost_wrlock(host);
157 -
158 - while(host->alarms_templates)
159 - rrdcalctemplate_unlink_and_free(host, host->alarms_templates);
160 -
161 - while(host->host_alarms)
162 - rrdcalc_unlink_and_free(host, host->host_alarms);
163 -
164 - RRDCALC *rc,*nc;
165 - for(rc = host->alarms_with_foreach; rc ; rc = nc) {
166 - nc = rc->next;
167 - rrdcalc_free(rc);
168 - }
169 - host->alarms_with_foreach = NULL;
170 -
171 - rrdhost_unlock(host);
156 + rrdcalc_delete_all(host);
157 + rrdcalctemplate_delete_all(host);
158
159 // invalidate all previous entries in the alarm log
160 + netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
161 ALARM_ENTRY *t;
162 for(t = host->health_log.alarms ; t ; t = t->next) {
163 if(t->new_status != RRDCALC_STATUS_REMOVED)
164 t->flags |= HEALTH_ENTRY_FLAG_UPDATED;
165 }
166 + netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
167
180 - rrdhost_rdlock(host);
168 // reset all thresholds to all charts
169 RRDSET *st;
170 rrdset_foreach_read(st, host) {
171 st->green = NAN;
172 st->red = NAN;
173 }
187 - rrdhost_unlock(host);
174 + rrdset_foreach_done(st);
175
176 // load the new alarms
190 - rrdhost_wrlock(host);
177 health_readdir(host, user_path, stock_path, NULL);
178
179 //Discard alarms with labels that do not apply to host
194 - rrdcalc_labels_unlink_alarm_from_host(host);
180 + rrdcalc_delete_alerts_not_matching_host_labels_from_this_host(host);
181
182 // link the loaded alarms to their charts
197 - RRDDIM *rd;
183 rrdset_foreach_write(st, host) {
184 if (rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED))
185 continue;
201 - rrdsetcalc_link_matching(st);
202 - rrdcalctemplate_link_matching(st);
186
204 - //This loop must be the last, because ` rrdcalctemplate_link_matching` will create alarms related to it.
205 - rrdset_rdlock(st);
206 - rrddim_foreach_read(rd, st) {
207 - rrdcalc_link_to_rrddim(rd, st, host);
208 - }
209 - rrdset_unlock(st);
187 + rrdcalc_link_matching_alerts_to_rrdset(st);
188 + rrdcalctemplate_link_matching_templates_to_rrdset(st);
189 }
190 + rrdset_foreach_done(st);
191
212 - rrdhost_unlock(host);
192 }
193
194 /**
@@ -323,7 +302,7 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
302 warn_alarms = buffer_create(NETDATA_WEB_RESPONSE_INITIAL_SIZE);
303 crit_alarms = buffer_create(NETDATA_WEB_RESPONSE_INITIAL_SIZE);
304
326 - foreach_rrdcalc_in_rrdhost(host, rc) {
305 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
306 if(unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
307 continue;
308
@@ -351,6 +330,7 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
330 expr = rc->warning;
331 }
332 }
333 + foreach_rrdcalc_in_rrdhost_done(rc);
334
335 if (n_warn+n_crit>1)
336 qsort (active_alerts, n_warn+n_crit, sizeof(active_alerts_t), compare_active_alerts);
@@ -477,13 +457,13 @@ static inline void health_alarm_log_process(RRDHOST *host) {
457 }
458 }
459
460 + netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
461 +
462 // remember this for the next iteration
463 host->health_last_processed_id = first_waiting;
464
465 bool cleanup_excess_log_entries = host->health_log.count > host->health_log.max;
466
485 - netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
486 -
467 if (!cleanup_excess_log_entries)
468 return;
469
@@ -554,10 +534,8 @@ static inline int rrdcalc_isrunnable(RRDCALC *rc, time_t now, time_t *next_run)
534 }
535
536 int update_every = rc->rrdset->update_every;
557 - rrdset_rdlock(rc->rrdset);
558 - time_t first = rrdset_first_entry_t_nolock(rc->rrdset);
559 - time_t last = rrdset_last_entry_t_nolock(rc->rrdset);
560 - rrdset_unlock(rc->rrdset);
537 + time_t first = rrdset_first_entry_t(rc->rrdset);
538 + time_t last = rrdset_last_entry_t(rc->rrdset);
539
540 if(unlikely(now + update_every < first /* || now - update_every > last */)) {
541 debug(D_HEALTH
@@ -653,29 +631,29 @@ static SILENCE_TYPE check_silenced(RRDCALC *rc, const char *host, SILENCERS *sil
631 * @return It returns 1 case rrdcalc_flags is DISABLED or 0 otherwise
632 */
633 static int update_disabled_silenced(RRDHOST *host, RRDCALC *rc) {
656 - uint32_t rrdcalc_flags_old = rc->rrdcalc_flags;
634 + uint32_t rrdcalc_flags_old = rc->run_flags;
635 // Clear the flags
658 - rc->rrdcalc_flags &= ~(RRDCALC_FLAG_DISABLED | RRDCALC_FLAG_SILENCED);
636 + rc->run_flags &= ~(RRDCALC_FLAG_DISABLED | RRDCALC_FLAG_SILENCED);
637 if (unlikely(silencers->all_alarms)) {
660 - if (silencers->stype == STYPE_DISABLE_ALARMS) rc->rrdcalc_flags |= RRDCALC_FLAG_DISABLED;
661 - else if (silencers->stype == STYPE_SILENCE_NOTIFICATIONS) rc->rrdcalc_flags |= RRDCALC_FLAG_SILENCED;
638 + if (silencers->stype == STYPE_DISABLE_ALARMS) rc->run_flags |= RRDCALC_FLAG_DISABLED;
639 + else if (silencers->stype == STYPE_SILENCE_NOTIFICATIONS) rc->run_flags |= RRDCALC_FLAG_SILENCED;
640 } else {
641 SILENCE_TYPE st = check_silenced(rc, rrdhost_hostname(host), silencers);
664 - if (st == STYPE_DISABLE_ALARMS) rc->rrdcalc_flags |= RRDCALC_FLAG_DISABLED;
665 - else if (st == STYPE_SILENCE_NOTIFICATIONS) rc->rrdcalc_flags |= RRDCALC_FLAG_SILENCED;
642 + if (st == STYPE_DISABLE_ALARMS) rc->run_flags |= RRDCALC_FLAG_DISABLED;
643 + else if (st == STYPE_SILENCE_NOTIFICATIONS) rc->run_flags |= RRDCALC_FLAG_SILENCED;
644 }
645
668 - if (rrdcalc_flags_old != rc->rrdcalc_flags) {
646 + if (rrdcalc_flags_old != rc->run_flags) {
647 info("Alarm silencing changed for host '%s' alarm '%s': Disabled %s->%s Silenced %s->%s",
648 rrdhost_hostname(host),
649 rrdcalc_name(rc),
650 (rrdcalc_flags_old & RRDCALC_FLAG_DISABLED)?"true":"false",
673 - (rc->rrdcalc_flags & RRDCALC_FLAG_DISABLED)?"true":"false",
651 + (rc->run_flags & RRDCALC_FLAG_DISABLED)?"true":"false",
652 (rrdcalc_flags_old & RRDCALC_FLAG_SILENCED)?"true":"false",
675 - (rc->rrdcalc_flags & RRDCALC_FLAG_SILENCED)?"true":"false"
653 + (rc->run_flags & RRDCALC_FLAG_SILENCED)?"true":"false"
654 );
655 }
678 - if (rc->rrdcalc_flags & RRDCALC_FLAG_DISABLED)
656 + if (rc->run_flags & RRDCALC_FLAG_DISABLED)
657 return 1;
658 else
659 return 0;
@@ -683,36 +661,38 @@ static int update_disabled_silenced(RRDHOST *host, RRDCALC *rc) {
661
662 // Create alarms for dimensions that have been added to charts
663 // since the previous iteration.
686 -static void init_pending_foreach_alarms(RRDHOST *host) {
664 +static void health_execute_pending_updates(RRDHOST *host) {
665 RRDSET *st;
688 - RRDDIM *rd;
666
667 if (!rrdhost_flag_check(host, RRDHOST_FLAG_PENDING_FOREACH_ALARMS))
668 return;
669
693 - rrdhost_wrlock(host);
694 -
695 - rrdset_foreach_write(st, host) {
696 - if (!rrdset_flag_check(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS))
670 + rrdset_foreach_reentrant(st, host) {
671 + if(!rrdset_flag_check(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS))
672 continue;
673
699 - rrdset_rdlock(st);
700 -
674 + RRDDIM *rd;
675 rrddim_foreach_read(rd, st) {
702 - if (!rrddim_flag_check(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARM))
676 + if(!rrddim_flag_check(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARMS))
677 continue;
678
705 - rrdcalc_link_to_rrddim(rd, st, host);
679 + RRDCALCTEMPLATE *rt;
680 + foreach_rrdcalctemplate_read(host, rt) {
681 + if(!rt->foreach_dimension_pattern)
682 + continue;
683
707 - rrddim_flag_clear(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARM);
708 - }
684 + if(rrdcalctemplate_check_rrdset_conditions(rt, st, host))
685 + rrdcalctemplate_check_rrddim_conditions_and_link(rt, st, rd, host);
686 + }
687 + foreach_rrdcalctemplate_done(rt);
688
689 + rrddim_flag_clear(rd, RRDDIM_FLAG_PENDING_FOREACH_ALARMS);
690 + }
691 + rrddim_foreach_done(rd);
692 rrdset_flag_clear(st, RRDSET_FLAG_PENDING_FOREACH_ALARMS);
711 - rrdset_unlock(st);
693 }
713 -
694 + rrdset_foreach_done(st);
695 rrdhost_flag_clear(host, RRDHOST_FLAG_PENDING_FOREACH_ALARMS);
715 - rrdhost_unlock(host);
696 }
697
698 /**
@@ -759,7 +739,7 @@ void *health_main(void *ptr) {
739 time_t now = now_realtime_sec();
740 time_t hibernation_delay = config_get_number(CONFIG_SECTION_HEALTH, "postpone alarms during hibernation for seconds", 60);
741
762 - rrdcalc_labels_unlink();
742 + rrdcalc_delete_alerts_not_matching_host_labels_from_all_hosts();
743
744 unsigned int loop = 0;
745 #ifdef ENABLE_ACLK
@@ -829,13 +809,14 @@ void *health_main(void *ptr) {
809 if(likely(!host->health_log_fp) && (loop == 1 || loop % cleanup_sql_every_loop == 0))
810 sql_health_alarm_log_cleanup(host);
811
832 - init_pending_foreach_alarms(host);
812 + health_execute_pending_updates(host);
813
814 worker_is_busy(WORKER_HEALTH_JOB_HOST_LOCK);
835 - rrdhost_rdlock(host);
815
816 // the first loop is to lookup values from the db
838 - foreach_rrdcalc_in_rrdhost(host, rc) {
817 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
818 +
819 + rrdcalc_update_info_using_rrdset_labels(rc);
820
821 if (update_disabled_silenced(host, rc))
822 continue;
@@ -876,7 +857,7 @@ void *health_main(void *ptr) {
857 rrdcalc_isrepeating(rc)?HEALTH_ENTRY_FLAG_IS_REPEATING:0);
858
859 if (ae) {
879 - health_alarm_log(host, ae);
860 + health_alarm_log_add_entry(host, ae);
861 rc->old_status = rc->status;
862 rc->status = RRDCALC_STATUS_REMOVED;
863 rc->last_status_change = now;
@@ -891,14 +872,14 @@ void *health_main(void *ptr) {
872 }
873
874 if (unlikely(!rrdcalc_isrunnable(rc, now, &next_run))) {
894 - if (unlikely(rc->rrdcalc_flags & RRDCALC_FLAG_RUNNABLE))
895 - rc->rrdcalc_flags &= ~RRDCALC_FLAG_RUNNABLE;
875 + if (unlikely(rc->run_flags & RRDCALC_FLAG_RUNNABLE))
876 + rc->run_flags &= ~RRDCALC_FLAG_RUNNABLE;
877 continue;
878 }
879
880 runnable++;
881 rc->old_value = rc->value;
901 - rc->rrdcalc_flags |= RRDCALC_FLAG_RUNNABLE;
882 + rc->run_flags |= RRDCALC_FLAG_RUNNABLE;
883
884 // ------------------------------------------------------------
885 // if there is database lookup, do it
@@ -919,13 +900,13 @@ void *health_main(void *ptr) {
900 if (unlikely(ret != 200)) {
901 // database lookup failed
902 rc->value = NAN;
922 - rc->rrdcalc_flags |= RRDCALC_FLAG_DB_ERROR;
903 + rc->run_flags |= RRDCALC_FLAG_DB_ERROR;
904
905 debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': database lookup returned error %d",
906 rrdhost_hostname(host), rrdcalc_chart_name(rc), rrdcalc_name(rc), ret
907 );
908 } else
928 - rc->rrdcalc_flags &= ~RRDCALC_FLAG_DB_ERROR;
909 + rc->run_flags &= ~RRDCALC_FLAG_DB_ERROR;
910
911 /* - RRDCALC_FLAG_DB_STALE not currently used
912 if (unlikely(old_db_timestamp == rc->db_before)) {
@@ -945,14 +926,14 @@ void *health_main(void *ptr) {
926 if (unlikely(value_is_null)) {
927 // collected value is null
928 rc->value = NAN;
948 - rc->rrdcalc_flags |= RRDCALC_FLAG_DB_NAN;
929 + rc->run_flags |= RRDCALC_FLAG_DB_NAN;
930
931 debug(D_HEALTH,
932 "Health on host '%s', alarm '%s.%s': database lookup returned empty value (possibly value is not collected yet)",
933 rrdhost_hostname(host), rrdcalc_chart_name(rc), rrdcalc_name(rc)
934 );
935 } else
955 - rc->rrdcalc_flags &= ~RRDCALC_FLAG_DB_NAN;
936 + rc->run_flags &= ~RRDCALC_FLAG_DB_NAN;
937
938 debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': database lookup gave value " NETDATA_DOUBLE_FORMAT,
939 rrdhost_hostname(host), rrdcalc_chart_name(rc), rrdcalc_name(rc), rc->value
@@ -968,14 +949,14 @@ void *health_main(void *ptr) {
949 if (unlikely(!expression_evaluate(rc->calculation))) {
950 // calculation failed
951 rc->value = NAN;
971 - rc->rrdcalc_flags |= RRDCALC_FLAG_CALC_ERROR;
952 + rc->run_flags |= RRDCALC_FLAG_CALC_ERROR;
953
954 debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': expression '%s' failed: %s",
955 rrdhost_hostname(host), rrdcalc_chart_name(rc), rrdcalc_name(rc),
956 rc->calculation->parsed_as, buffer_tostring(rc->calculation->error_msg)
957 );
958 } else {
978 - rc->rrdcalc_flags &= ~RRDCALC_FLAG_CALC_ERROR;
959 + rc->run_flags &= ~RRDCALC_FLAG_CALC_ERROR;
960
961 debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': expression '%s' gave value "
962 NETDATA_DOUBLE_FORMAT
@@ -985,25 +966,17 @@ void *health_main(void *ptr) {
966 );
967
968 rc->value = rc->calculation->result;
988 -
989 - if (rc->local) rc->local->last_updated = now;
990 - if (rc->family) rc->family->last_updated = now;
991 - if (rc->hostid) rc->hostid->last_updated = now;
992 - if (rc->hostname) rc->hostname->last_updated = now;
969 }
970 }
971 }
996 -
997 - rrdhost_unlock(host);
972 + foreach_rrdcalc_in_rrdhost_done(rc);
973
974 if (unlikely(runnable && !netdata_exit)) {
1000 - rrdhost_rdlock(host);
1001 -
1002 - foreach_rrdcalc_in_rrdhost(host, rc) {
1003 - if (unlikely(!(rc->rrdcalc_flags & RRDCALC_FLAG_RUNNABLE)))
975 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
976 + if (unlikely(!(rc->run_flags & RRDCALC_FLAG_RUNNABLE)))
977 continue;
978
1006 - if (rc->rrdcalc_flags & RRDCALC_FLAG_DISABLED) {
979 + if (rc->run_flags & RRDCALC_FLAG_DISABLED) {
980 continue;
981 }
982 RRDCALC_STATUS warning_status = RRDCALC_STATUS_UNDEFINED;
@@ -1017,7 +990,7 @@ void *health_main(void *ptr) {
990
991 if (unlikely(!expression_evaluate(rc->warning))) {
992 // calculation failed
1020 - rc->rrdcalc_flags |= RRDCALC_FLAG_WARN_ERROR;
993 + rc->run_flags |= RRDCALC_FLAG_WARN_ERROR;
994
995 debug(D_HEALTH,
996 "Health on host '%s', alarm '%s.%s': warning expression failed with error: %s",
@@ -1025,7 +998,7 @@ void *health_main(void *ptr) {
998 buffer_tostring(rc->warning->error_msg)
999 );
1000 } else {
1028 - rc->rrdcalc_flags &= ~RRDCALC_FLAG_WARN_ERROR;
1001 + rc->run_flags &= ~RRDCALC_FLAG_WARN_ERROR;
1002 debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': warning expression gave value "
1003 NETDATA_DOUBLE_FORMAT
1004 ": %s (source: %s)", rrdhost_hostname(host), rrdcalc_chart_name(rc),
@@ -1043,7 +1016,7 @@ void *health_main(void *ptr) {
1016
1017 if (unlikely(!expression_evaluate(rc->critical))) {
1018 // calculation failed
1046 - rc->rrdcalc_flags |= RRDCALC_FLAG_CRIT_ERROR;
1019 + rc->run_flags |= RRDCALC_FLAG_CRIT_ERROR;
1020
1021 debug(D_HEALTH,
1022 "Health on host '%s', alarm '%s.%s': critical expression failed with error: %s",
@@ -1051,7 +1024,7 @@ void *health_main(void *ptr) {
1024 buffer_tostring(rc->critical->error_msg)
1025 );
1026 } else {
1054 - rc->rrdcalc_flags &= ~RRDCALC_FLAG_CRIT_ERROR;
1027 + rc->run_flags &= ~RRDCALC_FLAG_CRIT_ERROR;
1028 debug(D_HEALTH, "Health on host '%s', alarm '%s.%s': critical expression gave value "
1029 NETDATA_DOUBLE_FORMAT
1030 ": %s (source: %s)", rrdhost_hostname(host), rrdcalc_chart_name(rc),
@@ -1156,13 +1129,13 @@ void *health_main(void *ptr) {
1129 rc->info,
1130 rc->delay_last,
1131 (
1159 - ((rc->options & RRDCALC_FLAG_NO_CLEAR_NOTIFICATION)? HEALTH_ENTRY_FLAG_NO_CLEAR_NOTIFICATION : 0) |
1160 - ((rc->rrdcalc_flags & RRDCALC_FLAG_SILENCED)? HEALTH_ENTRY_FLAG_SILENCED : 0) |
1132 + ((rc->options & RRDCALC_OPTION_NO_CLEAR_NOTIFICATION)? HEALTH_ENTRY_FLAG_NO_CLEAR_NOTIFICATION : 0) |
1133 + ((rc->run_flags & RRDCALC_FLAG_SILENCED)? HEALTH_ENTRY_FLAG_SILENCED : 0) |
1134 (rrdcalc_isrepeating(rc)?HEALTH_ENTRY_FLAG_IS_REPEATING:0)
1135 )
1136 );
1137
1165 - health_alarm_log(host, ae);
1138 + health_alarm_log_add_entry(host, ae);
1139
1140 rc->last_status_change = now;
1141 rc->old_status = rc->status;
@@ -1175,20 +1148,20 @@ void *health_main(void *ptr) {
1148 if (next_run > rc->next_update)
1149 next_run = rc->next_update;
1150 }
1151 + foreach_rrdcalc_in_rrdhost_done(rc);
1152
1153 // process repeating alarms
1180 - RRDCALC *rc;
1181 - foreach_rrdcalc_in_rrdhost(host, rc) {
1154 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
1155 int repeat_every = 0;
1156 if(unlikely(rrdcalc_isrepeating(rc) && rc->delay_up_to_timestamp <= now)) {
1157 if(unlikely(rc->status == RRDCALC_STATUS_WARNING)) {
1185 - rc->rrdcalc_flags &= ~RRDCALC_FLAG_RUN_ONCE;
1158 + rc->run_flags &= ~RRDCALC_FLAG_RUN_ONCE;
1159 repeat_every = rc->warn_repeat_every;
1160 } else if(unlikely(rc->status == RRDCALC_STATUS_CRITICAL)) {
1188 - rc->rrdcalc_flags &= ~RRDCALC_FLAG_RUN_ONCE;
1161 + rc->run_flags &= ~RRDCALC_FLAG_RUN_ONCE;
1162 repeat_every = rc->crit_repeat_every;
1163 } else if(unlikely(rc->status == RRDCALC_STATUS_CLEAR)) {
1191 - if(!(rc->rrdcalc_flags & RRDCALC_FLAG_RUN_ONCE)) {
1164 + if(!(rc->run_flags & RRDCALC_FLAG_RUN_ONCE)) {
1165 if(rc->old_status == RRDCALC_STATUS_CRITICAL) {
1166 repeat_every = 1;
1167 } else if (rc->old_status == RRDCALC_STATUS_WARNING) {
@@ -1230,25 +1203,24 @@ void *health_main(void *ptr) {
1203 rc->info,
1204 rc->delay_last,
1205 (
1233 - ((rc->options & RRDCALC_FLAG_NO_CLEAR_NOTIFICATION)? HEALTH_ENTRY_FLAG_NO_CLEAR_NOTIFICATION : 0) |
1234 - ((rc->rrdcalc_flags & RRDCALC_FLAG_SILENCED)? HEALTH_ENTRY_FLAG_SILENCED : 0) |
1206 + ((rc->options & RRDCALC_OPTION_NO_CLEAR_NOTIFICATION)? HEALTH_ENTRY_FLAG_NO_CLEAR_NOTIFICATION : 0) |
1207 + ((rc->run_flags & RRDCALC_FLAG_SILENCED)? HEALTH_ENTRY_FLAG_SILENCED : 0) |
1208 (rrdcalc_isrepeating(rc)?HEALTH_ENTRY_FLAG_IS_REPEATING:0)
1209 )
1210 );
1211
1212 ae->last_repeat = rc->last_repeat;
1240 - if (!(rc->rrdcalc_flags & RRDCALC_FLAG_RUN_ONCE) && rc->status == RRDCALC_STATUS_CLEAR) {
1213 + if (!(rc->run_flags & RRDCALC_FLAG_RUN_ONCE) && rc->status == RRDCALC_STATUS_CLEAR) {
1214 ae->flags |= HEALTH_ENTRY_RUN_ONCE;
1215 }
1243 - rc->rrdcalc_flags |= RRDCALC_FLAG_RUN_ONCE;
1216 + rc->run_flags |= RRDCALC_FLAG_RUN_ONCE;
1217 health_process_notifications(host, ae);
1218 debug(D_HEALTH, "Notification sent for the repeating alarm %u.", ae->alarm_id);
1219 health_alarm_wait_for_execution(ae);
1220 health_alarm_log_free_one_nochecks_nounlink(ae);
1221 }
1222 }
1250 -
1251 - rrdhost_unlock(host);
1223 + foreach_rrdcalc_in_rrdhost_done(rc);
1224 }
1225
1226 if (unlikely(netdata_exit))
health/health.h
+1 -3
@@ -74,7 +74,7 @@ extern ALARM_ENTRY* health_create_alarm_entry(
74 int delay,
75 uint32_t flags);
76
77 -extern void health_alarm_log(RRDHOST *host, ALARM_ENTRY *ae);
77 +extern void health_alarm_log_add_entry(RRDHOST *host, ALARM_ENTRY *ae);
78
79 extern void health_readdir(RRDHOST *host, const char *user_path, const char *stock_path, const char *subpath);
80 extern char *health_user_config_dir(void);
@@ -90,6 +90,4 @@ extern void health_label_log_save(RRDHOST *host);
90 extern char *health_edit_command_from_source(const char *source);
91 extern void sql_refresh_hashes(void);
92
93 -extern SIMPLE_PATTERN *health_pattern_from_foreach(const char *s);
94 -
93 #endif //NETDATA_HEALTH_H
health/health_config.c
+45 -143
@@ -33,122 +33,6 @@
33 #define HEALTH_HOST_LABEL_KEY "host labels"
34 #define HEALTH_FOREACH_KEY "foreach"
35
36 -static inline int rrdcalc_add_alarm_from_config(RRDHOST *host, RRDCALC *rc) {
37 - if(!rc->chart) {
38 - error("Health configuration for alarm '%s' does not have a chart", rrdcalc_name(rc));
39 - return 0;
40 - }
41 -
42 - if(!rc->update_every) {
43 - error("Health configuration for alarm '%s.%s' has no frequency (parameter 'every'). Ignoring it.", rrdcalc_chart_name(rc), rrdcalc_name(rc));
44 - return 0;
45 - }
46 -
47 - if(!RRDCALC_HAS_DB_LOOKUP(rc) && !rc->calculation && !rc->warning && !rc->critical) {
48 - error("Health configuration for alarm '%s.%s' is useless (no db lookup, no calculation, no warning and no critical expressions)", rrdcalc_chart_name(rc), rrdcalc_name(rc));
49 - return 0;
50 - }
51 -
52 - if (rrdcalc_exists(host, rrdcalc_chart_name(rc), rrdcalc_name(rc)))
53 - return 0;
54 -
55 - rc->id = rrdcalc_get_unique_id(host, rc->chart, rc->name, &rc->next_event_id);
56 -
57 - debug(D_HEALTH, "Health configuration adding alarm '%s.%s' (%u): exec '%s', recipient '%s', green " NETDATA_DOUBLE_FORMAT_AUTO
58 - ", red " NETDATA_DOUBLE_FORMAT_AUTO
59 - ", lookup: group %d, after %d, before %d, options %u, dimensions '%s', for each dimension '%s', update every %d, calculation '%s', warning '%s', critical '%s', source '%s', delay up %d, delay down %d, delay max %d, delay_multiplier %f, warn_repeat_every %u, crit_repeat_every %u",
60 - rrdcalc_chart_name(rc),
61 - rrdcalc_name(rc),
62 - rc->id,
63 - (rc->exec)?rrdcalc_exec(rc):"DEFAULT",
64 - (rc->recipient)?rrdcalc_recipient(rc):"DEFAULT",
65 - rc->green,
66 - rc->red,
67 - (int)rc->group,
68 - rc->after,
69 - rc->before,
70 - rc->options,
71 - (rc->dimensions)?rrdcalc_dimensions(rc):"NONE",
72 - (rc->foreachdim)?rrdcalc_foreachdim(rc):"NONE",
73 - rc->update_every,
74 - (rc->calculation)?rc->calculation->parsed_as:"NONE",
75 - (rc->warning)?rc->warning->parsed_as:"NONE",
76 - (rc->critical)?rc->critical->parsed_as:"NONE",
77 - rrdcalc_source(rc),
78 - rc->delay_up_duration,
79 - rc->delay_down_duration,
80 - rc->delay_max_duration,
81 - rc->delay_multiplier,
82 - rc->warn_repeat_every,
83 - rc->crit_repeat_every
84 - );
85 -
86 - rrdcalc_add_to_host(host, rc);
87 -
88 - return 1;
89 -}
90 -
91 -static inline int rrdcalctemplate_add_template_from_config(RRDHOST *host, RRDCALCTEMPLATE *rt) {
92 - if(unlikely(!rt->context)) {
93 - error("Health configuration for template '%s' does not have a context", rrdcalctemplate_name(rt));
94 - return 0;
95 - }
96 -
97 - if(unlikely(!rt->update_every)) {
98 - error("Health configuration for template '%s' has no frequency (parameter 'every'). Ignoring it.", rrdcalctemplate_name(rt));
99 - return 0;
100 - }
101 -
102 - if(unlikely(!RRDCALCTEMPLATE_HAS_DB_LOOKUP(rt) && !rt->calculation && !rt->warning && !rt->critical)) {
103 - error("Health configuration for template '%s' is useless (no calculation, no warning and no critical evaluation)", rrdcalctemplate_name(rt));
104 - return 0;
105 - }
106 -
107 - RRDCALCTEMPLATE *t;
108 - foreach_rrdcalctemplate_in_rrdhost(host, t) {
109 - if(unlikely(t->name == rt->name && !strcmp(t->family_match?rrdcalctemplate_family_match(t):"*", rt->family_match?rrdcalctemplate_family_match(rt):"*"))) {
110 - info("Health configuration template '%s' already exists for host '%s'.", rrdcalctemplate_name(rt), rrdhost_hostname(host));
111 - return 0;
112 - }
113 - }
114 -
115 - if(rt->foreachdim)
116 - DOUBLE_LINKED_LIST_PREPEND_UNSAFE(host->alarms_templates, rt, prev, next);
117 - else
118 - DOUBLE_LINKED_LIST_APPEND_UNSAFE(host->alarms_templates, rt, prev, next);
119 -
120 - debug(D_HEALTH, "Health configuration adding template '%s': context '%s', exec '%s', recipient '%s', green " NETDATA_DOUBLE_FORMAT_AUTO
121 - ", red " NETDATA_DOUBLE_FORMAT_AUTO
122 - ", lookup: group %d, after %d, before %d, options %u, dimensions '%s', for each dimension '%s', update every %d, calculation '%s', warning '%s', critical '%s', source '%s', delay up %d, delay down %d, delay max %d, delay_multiplier %f, warn_repeat_every %u, crit_repeat_every %u",
123 - rrdcalctemplate_name(rt),
124 - (rt->context)?string2str(rt->context):"NONE",
125 - (rt->exec)?rrdcalctemplate_exec(rt):"DEFAULT",
126 - (rt->recipient)?rrdcalctemplate_recipient(rt):"DEFAULT",
127 - rt->green,
128 - rt->red,
129 - (int)rt->group,
130 - rt->after,
131 - rt->before,
132 - rt->options,
133 - (rt->dimensions)?rrdcalctemplate_dimensions(rt):"NONE",
134 - (rt->foreachdim)?rrdcalctemplate_foreachdim(rt):"NONE",
135 - rt->update_every,
136 - (rt->calculation)?rt->calculation->parsed_as:"NONE",
137 - (rt->warning)?rt->warning->parsed_as:"NONE",
138 - (rt->critical)?rt->critical->parsed_as:"NONE",
139 - rrdcalctemplate_source(rt),
140 - rt->delay_up_duration,
141 - rt->delay_down_duration,
142 - rt->delay_max_duration,
143 - rt->delay_multiplier,
144 - rt->warn_repeat_every,
145 - rt->crit_repeat_every
146 - );
147 -
148 -
149 - return 1;
150 -}
151 -
36 static inline int health_parse_delay(
37 size_t line, const char *filename, char *string,
38 int *delay_up_duration,
@@ -249,7 +133,7 @@ static inline uint32_t health_parse_options(const char *s) {
133 buf[count] = '\0';
134
135 if(!strcasecmp(buf, "no-clear-notification") || !strcasecmp(buf, "no-clear"))
252 - options |= RRDCALC_FLAG_NO_CLEAR_NOTIFICATION;
136 + options |= RRDCALC_OPTION_NO_CLEAR_NOTIFICATION;
137 else
138 error("Ignoring unknown alarm option '%s'", buf);
139 }
@@ -308,13 +192,21 @@ static inline int health_parse_repeat(
192 *
193 * @param s the string that will be used to create the simple pattern.
194 */
311 -SIMPLE_PATTERN *health_pattern_from_foreach(const char *s) {
195 +
196 +static void dimension_remove_pipe_comma(char *str) {
197 + while(*str) {
198 + if(*str == '|' || *str == ',') *str = ' ';
199 + str++;
200 + }
201 +}
202 +
203 +static SIMPLE_PATTERN *health_pattern_from_foreach(const char *s) {
204 char *convert= strdupz(s);
205 SIMPLE_PATTERN *val = NULL;
206 +
207 if(convert) {
208 dimension_remove_pipe_comma(convert);
209 val = simple_pattern_create(convert, NULL, SIMPLE_PATTERN_EXACT);
317 -
210 freez(convert);
211 }
212
@@ -324,7 +216,7 @@ SIMPLE_PATTERN *health_pattern_from_foreach(const char *s) {
216 static inline int health_parse_db_lookup(
217 size_t line, const char *filename, char *string,
218 RRDR_GROUPING *group_method, int *after, int *before, int *every,
327 - uint32_t *options, STRING **dimensions, STRING **foreachdim
219 + RRDCALC_OPTIONS *options, STRING **dimensions, STRING **foreachdim
220 ) {
221 debug(D_HEALTH, "Health configuration parsing database lookup %zu@%s: %s", line, filename, string);
222
@@ -335,7 +227,7 @@ static inline int health_parse_db_lookup(
227 *after = 0;
228 *before = 0;
229 *every = 0;
338 - *options = 0;
230 + *options = (*options) & RRDCALC_ALL_OPTIONS_EXCLUDING_THE_RRDR_ONES; // preserve rrdcalc options
231
232 char *s = string, *key;
233
@@ -644,16 +536,20 @@ static int health_readfile(const char *filename, void *data) {
536
537 if(hash == hash_alarm && !strcasecmp(key, HEALTH_ALARM_KEY)) {
538 if(rc) {
647 - if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalc_add_alarm_from_config(host, rc)) {
648 - rrdcalc_free(rc);
649 - }
539 + if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
540 + rrdcalc_free_unused_rrdcalc_loaded_from_config(rc);
541 + else
542 + rrdcalc_add_from_config(host, rc);
543 +
544 // health_add_alarms_loop(host, rc, ignore_this) ;
545 }
546
547 if(rt) {
654 - if (!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalctemplate_add_template_from_config(host, rt)) {
655 - rrdcalctemplate_free(rt);
656 - }
548 + if(!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
549 + rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(rt);
550 + else
551 + rrdcalctemplate_add_from_config(host, rt);
552 +
553 rt = NULL;
554 }
555
@@ -688,15 +584,19 @@ static int health_readfile(const char *filename, void *data) {
584 else if(hash == hash_template && !strcasecmp(key, HEALTH_TEMPLATE_KEY)) {
585 if(rc) {
586 // health_add_alarms_loop(host, rc, ignore_this) ;
691 - if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalc_add_alarm_from_config(host, rc))
692 - rrdcalc_free(rc);
587 + if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
588 + rrdcalc_free_unused_rrdcalc_loaded_from_config(rc);
589 + else
590 + rrdcalc_add_from_config(host, rc);
591
592 rc = NULL;
593 }
594
595 if(rt) {
698 - if(!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalctemplate_add_template_from_config(host, rt))
699 - rrdcalctemplate_free(rt);
596 + if(!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
597 + rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(rt);
598 + else
599 + rrdcalctemplate_add_from_config(host, rt);
600 }
601
602 rt = callocz(1, sizeof(RRDCALCTEMPLATE));
@@ -811,10 +711,10 @@ static int health_readfile(const char *filename, void *data) {
711 else if(hash == hash_lookup && !strcasecmp(key, HEALTH_LOOKUP_KEY)) {
712 alert_cfg->lookup = string_strdupz(value);
713 health_parse_db_lookup(line, filename, value, &rc->group, &rc->after, &rc->before,
814 - &rc->update_every, &rc->options, &rc->dimensions, &rc->foreachdim);
714 + &rc->update_every, &rc->options, &rc->dimensions, &rc->foreach_dimension);
715
816 - if(rc->foreachdim)
817 - rc->spdim = health_pattern_from_foreach(rrdcalc_foreachdim(rc));
716 + if(rc->foreach_dimension)
717 + rc->foreach_dimension_pattern = health_pattern_from_foreach(rrdcalc_foreachdim(rc));
718
719 if (rc->after) {
720 if (rc->dimensions)
@@ -1071,10 +971,10 @@ static int health_readfile(const char *filename, void *data) {
971 else if(hash == hash_lookup && !strcasecmp(key, HEALTH_LOOKUP_KEY)) {
972 alert_cfg->lookup = string_strdupz(value);
973 health_parse_db_lookup(line, filename, value, &rt->group, &rt->after, &rt->before,
1074 - &rt->update_every, &rt->options, &rt->dimensions, &rt->foreachdim);
974 + &rt->update_every, &rt->options, &rt->dimensions, &rt->foreach_dimension);
975
1076 - if(rt->foreachdim)
1077 - rt->spdim = health_pattern_from_foreach(rrdcalctemplate_foreachdim(rt));
976 + if(rt->foreach_dimension)
977 + rt->foreach_dimension_pattern = health_pattern_from_foreach(rrdcalctemplate_foreachdim(rt));
978
979 if (rt->after) {
980 if (rt->dimensions)
@@ -1237,15 +1137,17 @@ static int health_readfile(const char *filename, void *data) {
1137
1138 if(rc) {
1139 //health_add_alarms_loop(host, rc, ignore_this) ;
1240 - if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalc_add_alarm_from_config(host, rc)) {
1241 - rrdcalc_free(rc);
1242 - }
1140 + if(!alert_hash_and_store_config(rc->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
1141 + rrdcalc_free_unused_rrdcalc_loaded_from_config(rc);
1142 + else
1143 + rrdcalc_add_from_config(host, rc);
1144 }
1145
1146 if(rt) {
1246 - if(!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this || !rrdcalctemplate_add_template_from_config(host, rt)) {
1247 - rrdcalctemplate_free(rt);
1248 - }
1147 + if(!alert_hash_and_store_config(rt->config_hash_id, alert_cfg, sql_store_hashes) || ignore_this)
1148 + rrdcalctemplate_free_unused_rrdcalctemplate_loaded_from_config(rt);
1149 + else
1150 + rrdcalctemplate_add_from_config(host, rt);
1151 }
1152
1153 if (alert_cfg)
health/health_json.c
+15 -16
@@ -116,7 +116,6 @@ void health_alarm_entry2json_nolock(BUFFER *wb, ALARM_ENTRY *ae, RRDHOST *host)
116 }
117
118 void health_alarm_log2json(RRDHOST *host, BUFFER *wb, uint32_t after, char *chart) {
119 - netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
119
120 buffer_strcat(wb, "[");
121
@@ -125,6 +124,8 @@ void health_alarm_log2json(RRDHOST *host, BUFFER *wb, uint32_t after, char *char
124
125 STRING *chart_string = string_strdupz(chart);
126
127 + netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
128 +
129 ALARM_ENTRY *ae;
130 for (ae = host->health_log.alarms; ae && count < max; ae = ae->next) {
131 if ((ae->unique_id > after) && (!chart || chart_string == ae->chart)) {
@@ -135,11 +136,11 @@ void health_alarm_log2json(RRDHOST *host, BUFFER *wb, uint32_t after, char *char
136 }
137 }
138
139 + netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
140 +
141 string_freez(chart_string);
142
143 buffer_strcat(wb, "\n]\n");
141 -
142 - netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
144 }
145
146 static inline void health_rrdcalc_values2json_nolock(RRDHOST *host, BUFFER *wb, RRDCALC *rc) {
@@ -216,8 +217,8 @@ static inline void health_rrdcalc2json_nolock(RRDHOST *host, BUFFER *wb, RRDCALC
217 , rc->component?rrdcalc_component(rc):"Unknown"
218 , rc->type?rrdcalc_type(rc):"Unknown"
219 , (rc->rrdset)?"true":"false"
219 - , (rc->rrdcalc_flags & RRDCALC_FLAG_DISABLED)?"true":"false"
220 - , (rc->rrdcalc_flags & RRDCALC_FLAG_SILENCED)?"true":"false"
220 + , (rc->run_flags & RRDCALC_FLAG_DISABLED)?"true":"false"
221 + , (rc->run_flags & RRDCALC_FLAG_SILENCED)?"true":"false"
222 , rc->exec?rrdcalc_exec(rc):string2str(host->health_default_exec)
223 , rc->recipient?rrdcalc_recipient(rc):string2str(host->health_default_recipient)
224 , rrdcalc_source(rc)
@@ -241,7 +242,7 @@ static inline void health_rrdcalc2json_nolock(RRDHOST *host, BUFFER *wb, RRDCALC
242 , (unsigned long)rc->times_repeat
243 );
244
244 - if(unlikely(rc->options & RRDCALC_FLAG_NO_CLEAR_NOTIFICATION)) {
245 + if(unlikely(rc->options & RRDCALC_OPTION_NO_CLEAR_NOTIFICATION)) {
246 buffer_strcat(wb, "\t\t\t\"no_clear_notification\": true,\n");
247 }
248
@@ -306,8 +307,6 @@ void health_aggregate_alarms(RRDHOST *host, BUFFER *wb, BUFFER* contexts, RRDCAL
307 char *tok = NULL;
308 char *p = NULL;
309
309 - rrdhost_rdlock(host);
310 -
310 if (contexts) {
311 p = (char*)buffer_tostring(contexts);
312 while(p && *p && (tok = mystrsep(&p, ", |"))) {
@@ -315,7 +314,7 @@ void health_aggregate_alarms(RRDHOST *host, BUFFER *wb, BUFFER* contexts, RRDCAL
314
315 STRING *tok_string = string_strdupz(tok);
316
318 - foreach_rrdcalc_in_rrdhost(host, rc) {
317 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
318 if(unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
319 continue;
320 if (unlikely(!rrdset_is_available_for_exporting_and_alarms(rc->rrdset)))
@@ -325,12 +324,13 @@ void health_aggregate_alarms(RRDHOST *host, BUFFER *wb, BUFFER* contexts, RRDCAL
324 && ((status==RRDCALC_STATUS_RAISED)?(rc->status >= RRDCALC_STATUS_WARNING):rc->status == status)))
325 numberOfAlarms++;
326 }
327 + foreach_rrdcalc_in_rrdhost_done(rc);
328
329 string_freez(tok_string);
330 }
331 }
332 else {
333 - foreach_rrdcalc_in_rrdhost(host, rc) {
333 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
334 if(unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
335 continue;
336 if (unlikely(!rrdset_is_available_for_exporting_and_alarms(rc->rrdset)))
@@ -338,16 +338,16 @@ void health_aggregate_alarms(RRDHOST *host, BUFFER *wb, BUFFER* contexts, RRDCAL
338 if(unlikely((status==RRDCALC_STATUS_RAISED)?(rc->status >= RRDCALC_STATUS_WARNING):rc->status == status))
339 numberOfAlarms++;
340 }
341 + foreach_rrdcalc_in_rrdhost_done(rc);
342 }
343
344 buffer_sprintf(wb, "%d", numberOfAlarms);
344 - rrdhost_unlock(host);
345 }
346
347 static void health_alarms2json_fill_alarms(RRDHOST *host, BUFFER *wb, int all, void (*fp)(RRDHOST *, BUFFER *, RRDCALC *)) {
348 RRDCALC *rc;
349 int i = 0;
350 - foreach_rrdcalc_in_rrdhost(host, rc) {
350 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
351 if(unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
352 continue;
353
@@ -361,10 +361,10 @@ static void health_alarms2json_fill_alarms(RRDHOST *host, BUFFER *wb, int all, v
361 fp(host, wb, rc);
362 i++;
363 }
364 + foreach_rrdcalc_in_rrdhost_done(rc);
365 }
366
367 void health_alarms2json(RRDHOST *host, BUFFER *wb, int all) {
367 - rrdhost_rdlock(host);
368 buffer_sprintf(wb, "{\n\t\"hostname\": \"%s\","
369 "\n\t\"latest_alarm_log_unique_id\": %u,"
370 "\n\t\"status\": %s,"
@@ -377,17 +377,17 @@ void health_alarms2json(RRDHOST *host, BUFFER *wb, int all) {
377
378 health_alarms2json_fill_alarms(host, wb, all, health_rrdcalc2json_nolock);
379
380 +// rrdhost_rdlock(host);
381 // buffer_strcat(wb, "\n\t},\n\t\"templates\": {");
382 // RRDCALCTEMPLATE *rt;
383 // for(rt = host->templates; rt ; rt = rt->next)
384 // health_rrdcalctemplate2json_nolock(wb, rt);
385 +// rrdhost_unlock(host);
386
387 buffer_strcat(wb, "\n\t}\n}\n");
386 - rrdhost_unlock(host);
388 }
389
390 void health_alarms_values2json(RRDHOST *host, BUFFER *wb, int all) {
390 - rrdhost_rdlock(host);
391 buffer_sprintf(wb, "{\n\t\"hostname\": \"%s\","
392 "\n\t\"alarms\": {\n",
393 rrdhost_hostname(host));
@@ -395,7 +395,6 @@ void health_alarms_values2json(RRDHOST *host, BUFFER *wb, int all) {
395 health_alarms2json_fill_alarms(host, wb, all, health_rrdcalc_values2json_nolock);
396
397 buffer_strcat(wb, "\n\t}\n}\n");
398 - rrdhost_unlock(host);
398 }
399
400 static int have_recent_alarm(RRDHOST *host, uint32_t alarm_id, time_t mark)
health/health_log.c
+10 -9
@@ -182,12 +182,15 @@ static inline ssize_t health_alarm_log_read(RRDHOST *host, FILE *fp, const char
182 size_t line = 0, len = 0;
183 ssize_t loaded = 0, updated = 0, errored = 0, duplicate = 0;
184
185 - netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
186 -
187 - DICTIONARY *all_rrdcalcs = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
185 + DICTIONARY *all_rrdcalcs = dictionary_create(
186 + DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
187 RRDCALC *rc;
189 - foreach_rrdcalc_in_rrdhost(host, rc)
188 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
189 dictionary_set(all_rrdcalcs, rrdcalc_name(rc), rc, sizeof(*rc));
190 + }
191 + foreach_rrdcalc_in_rrdhost_done(rc);
192 +
193 + netdata_rwlock_rdlock(&host->health_log.alarm_log_rwlock);
194
195 while((s = fgets_trim_len(buf, 65536, fp, &len))) {
196 host->health_log_entries_written++;
@@ -394,11 +397,11 @@ static inline ssize_t health_alarm_log_read(RRDHOST *host, FILE *fp, const char
397 }
398 }
399
400 + netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
401 +
402 dictionary_destroy(all_rrdcalcs);
403 all_rrdcalcs = NULL;
404
400 - netdata_rwlock_unlock(&host->health_log.alarm_log_rwlock);
401 -
405 freez(buf);
406
407 if(!host->health_max_unique_id) host->health_max_unique_id = (uint32_t)now_realtime_sec();
@@ -510,7 +513,7 @@ inline ALARM_ENTRY* health_create_alarm_entry(
513 return ae;
514 }
515
513 -inline void health_alarm_log(
516 +inline void health_alarm_log_add_entry(
517 RRDHOST *host,
518 ALARM_ENTRY *ae
519 ) {
@@ -568,8 +571,6 @@ inline void health_alarm_log_free_one_nochecks_nounlink(ALARM_ENTRY *ae) {
571 }
572
573 inline void health_alarm_log_free(RRDHOST *host) {
571 - rrdhost_check_wrlock(host);
572 -
574 netdata_rwlock_wrlock(&host->health_log.alarm_log_rwlock);
575
576 ALARM_ENTRY *ae;
libnetdata/Makefile.am
+1
@@ -25,6 +25,7 @@ SUBDIRS = \
25 socket \
26 statistical \
27 storage_number \
28 + string \
29 threads \
30 url \
31 worker_utilization \
libnetdata/dictionary/README.md
+9 -9
@@ -18,7 +18,7 @@ Dictionaries provide an interface to:
18 - **Delete** an item from the dictionary (provided its `name`)
19 - **Traverse** the list of items in the dictionary
20
21 -Dictionaries are **ordered**, meaning that the order they have been added is preserved while traversing them. The caller may reverse this order by passing the flag `DICTIONARY_FLAG_ADD_IN_FRONT` when creating the dictionary.
21 +Dictionaries are **ordered**, meaning that the order they have been added is preserved while traversing them. The caller may reverse this order by passing the flag `DICT_OPTION_ADD_IN_FRONT` when creating the dictionary.
22
23 Dictionaries guarantee **uniqueness** of all items added to them, meaning that only one item with a given name can exist in the dictionary at any given time.
24
@@ -35,21 +35,21 @@ In **clone** mode, the dictionary guarantees that all operations on the dictiona
35
36 1.`dictionary_register_insert_callback()` that will be called just after the insertion of an item to the dictionary, or after the replacement of the value of a dictionary item (but while the dictionary is write-locked - if locking is enabled).
37 2. `dictionary_register_delete_callback()` that will be called just prior to the deletion of an item from the dictionary, or prior to the replacement of the value of a dictionary item (but while the dictionary is write-locked - if locking is enabled).
38 - 3. `dictionary_register_conflict_callback()` that will be called when `DICTIONARY_FLAG_DONT_OVERWRITE_VALUE` is set and another value is attempted to be inserted for the same key.
38 + 3. `dictionary_register_conflict_callback()` that will be called when `DICT_OPTION_DONT_OVERWRITE_VALUE` is set and another value is attempted to be inserted for the same key.
39
40 In **link** mode, the name and/or the value are just linked to the dictionary item, and it is the user's responsibility to free the memory they use after an item is deleted from the dictionary or when the dictionary is destroyed.
41
42 By default, **clone** mode is used for both the name and the value.
43
44 -To use **link** mode for names, add `DICTIONARY_FLAG_NAME_LINK_DONT_CLONE` to the flags when creating the dictionary.
44 +To use **link** mode for names, add `DICT_OPTION_NAME_LINK_DONT_CLONE` to the flags when creating the dictionary.
45
46 -To use **link** mode for values, add `DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE` to the flags when creating the dictionary.
46 +To use **link** mode for values, add `DICT_OPTION_VALUE_LINK_DONT_CLONE` to the flags when creating the dictionary.
47
48 ## Locks
49
50 The dictionary allows both **single-threaded** operation (no locks - faster) and **multi-threaded** operation utilizing a read-write lock.
51
52 -The default is **multi-threaded**. To enable **single-threaded** add `DICTIONARY_FLAG_SINGLE_THREADED` to the flags when creating the dictionary.
52 +The default is **multi-threaded**. To enable **single-threaded** add `DICT_OPTION_SINGLE_THREADED` to the flags when creating the dictionary.
53
54 ## Hash table operations
55
@@ -72,7 +72,7 @@ This call is used to:
72 - **add** an item to the dictionary.
73 - **reset** the value of an existing item in the dictionary.
74
75 -If **resetting** is not desired, add `DICTIONARY_FLAG_DONT_OVERWRITE_VALUE` to the flags when creating the dictionary. In this case, `dictionary_set()` will return the value of the original item found in the dictionary instead of resetting it and the value passed to the call will be ignored. Optionally a conflict callback function can be registered, to manipulate (probably merge or extend) the original value, based on the new value attempted to be added to the dictionary.
75 +If **resetting** is not desired, add `DICT_OPTION_DONT_OVERWRITE_VALUE` to the flags when creating the dictionary. In this case, `dictionary_set()` will return the value of the original item found in the dictionary instead of resetting it and the value passed to the call will be ignored. Optionally a conflict callback function can be registered, to manipulate (probably merge or extend) the original value, based on the new value attempted to be added to the dictionary.
76
77 For **multi-threaded** operation, the `dictionary_set()` calls get an exclusive write lock on the dictionary.
78
@@ -143,7 +143,7 @@ Example:
143
144 ```c
145 // create the dictionary
146 -DICTIONARY *dict = dictionary_create(DICTIONARY_FLAGS_NONE);
146 +DICTIONARY *dict = dictionary_create(DICT_OPTION_NONE);
147
148 // add an item to it
149 dictionary_set(dict, "name", "value", 6);
@@ -189,7 +189,7 @@ There are 4 calls:
189 - `dictionary_walkthrough_read()` and `dictionary_sorted_walkthrough_read()` that acquire a shared read lock, and they call a callback function for every item of the dictionary. The callback function may use the unsafe versions of the `dictionary_get()` calls to lookup other items in the dictionary, but it should not attempt to add or remove items to/from the dictionary.
190 - `dictionary_walkthrough_write()` and `dictionary_sorted_walkthrough_write()` that acquire an exclusive write lock, and they call a callback function for every item of the dictionary. This is to be used when items need to be added to or removed from the dictionary. The `write` versions can be used to delete any or all the items from the dictionary, including the currently working one. For the `sorted` version, all items in the dictionary maintain a reference counter, so all deletions are deferred until the sorted walkthrough finishes.**
191
192 -The non sorted versions traverse the items in the same order they have been added to the dictionary (or the reverse order if the flag `DICTIONARY_FLAG_ADD_IN_FRONT` is set during dictionary creation). The sorted versions sort alphabetically the items based on their name, and then they traverse them in the sorted order.
192 +The non sorted versions traverse the items in the same order they have been added to the dictionary (or the reverse order if the flag `DICT_OPTION_ADD_IN_FRONT` is set during dictionary creation). The sorted versions sort alphabetically the items based on their name, and then they traverse them in the sorted order.
193
194 The callback function returns an `int`. If this value is negative, traversal of the dictionary is stopped immediately and the negative value is returned to the caller. If the returned value of all callback calls is zero or positive, the walkthrough functions return the sum of the return values of all callbacks. So, if you are just interested to know how many items fall into some condition, write a callback function that returns 1 when the item satisfies that condition and 0 when it does not and the walkthrough function will return how many tested positive.
195
@@ -241,5 +241,5 @@ Since the dictionary uses a hash table and a double linked list, if the contract
241
242 This is currently used in statsd:
243
244 -- the data collection thread uses only `get` and `set`. It never uses `del`. New items are added at the front of the linked list (`DICTIONARY_FLAG_ADD_IN_FRONT`).
244 +- the data collection thread uses only `get` and `set`. It never uses `del`. New items are added at the front of the linked list (`DICT_OPTION_ADD_IN_FRONT`).
245 - the flushing thread is only traversing the dictionary up to the point it last traversed it (it uses a flag for that to know where it stopped last time). It never uses `get`, `set` or `del`.
libnetdata/dictionary/dictionary.c
+2144 -1481
@@ -1,383 +1,708 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 -// NOT TO BE USED BY USERS
4 -#define DICTIONARY_FLAG_EXCLUSIVE_ACCESS (1 << 28) // there is only one thread accessing the dictionary
5 -#define DICTIONARY_FLAG_DESTROYED (1 << 29) // this dictionary has been destroyed
6 -#define DICTIONARY_FLAG_DEFER_ALL_DELETIONS (1 << 30) // defer all deletions of items in the dictionary
7 -
8 -// our reserved flags that cannot be set by users
9 -#define DICTIONARY_FLAGS_RESERVED (DICTIONARY_FLAG_EXCLUSIVE_ACCESS|DICTIONARY_FLAG_DESTROYED|DICTIONARY_FLAG_DEFER_ALL_DELETIONS)
10 -
3 #define DICTIONARY_INTERNALS
4
5 #include "../libnetdata.h"
6 #include <Judy.h>
7
16 -typedef enum name_value_flags {
17 - NAME_VALUE_FLAG_NONE = 0,
18 - NAME_VALUE_FLAG_NAME_IS_ALLOCATED = (1 << 0), // the name pointer is a STRING
19 - NAME_VALUE_FLAG_DELETED = (1 << 1), // this item is deleted, so it is not available for traversal
20 - NAME_VALUE_FLAG_NEW_OR_UPDATED = (1 << 2), // this item is new or just updated (used by the react callback)
8 +// runtime flags of the dictionary - must be checked with atomics
9 +typedef enum {
10 + DICT_FLAG_NONE = 0,
11 + DICT_FLAG_DESTROYED = (1 << 0), // this dictionary has been destroyed
12 +} DICT_FLAGS;
13 +
14 +#define dict_flag_check(dict, flag) (__atomic_load_n(&((dict)->flags), __ATOMIC_SEQ_CST) & (flag))
15 +#define dict_flag_set(dict, flag) __atomic_or_fetch(&((dict)->flags), flag, __ATOMIC_SEQ_CST)
16 +#define dict_flag_clear(dict, flag) __atomic_and_fetch(&((dict)->flags), ~(flag), __ATOMIC_SEQ_CST)
17 +
18 +// flags macros
19 +#define is_dictionary_destroyed(dict) dict_flag_check(dict, DICT_FLAG_DESTROYED)
20 +
21 +// configuration options macros
22 +#define is_dictionary_single_threaded(dict) ((dict)->options & DICT_OPTION_SINGLE_THREADED)
23 +#define is_view_dictionary(dict) ((dict)->master)
24 +#define is_master_dictionary(dict) (!is_view_dictionary(dict))
25 +
26 +typedef enum item_options {
27 + ITEM_OPTION_NONE = 0,
28 + ITEM_OPTION_ALLOCATED_NAME = (1 << 0), // the name pointer is a STRING
29 +
30 + // IMPORTANT: This is 1-bit - to add more change ITEM_OPTIONS_BITS
31 +} ITEM_OPTIONS;
32 +
33 +typedef enum item_flags {
34 + ITEM_FLAG_NONE = 0,
35 + ITEM_FLAG_DELETED = (1 << 0), // this item is deleted, so it is not available for traversal
36 +
37 + // IMPORTANT: This is 8-bit
38 +} ITEM_FLAGS;
39 +
40 +#define item_flag_check(item, flag) (__atomic_load_n(&((item)->flags), __ATOMIC_SEQ_CST) & (flag))
41 +#define item_flag_set(item, flag) __atomic_or_fetch(&((item)->flags), flag, __ATOMIC_SEQ_CST)
42 +#define item_flag_clear(item, flag) __atomic_and_fetch(&((item)->flags), ~(flag), __ATOMIC_SEQ_CST)
43 +
44 +#define item_shared_flag_check(item, flag) (__atomic_load_n(&((item)->shared->flags), __ATOMIC_SEQ_CST) & (flag))
45 +#define item_shared_flag_set(item, flag) __atomic_or_fetch(&((item)->shared->flags), flag, __ATOMIC_SEQ_CST)
46 +#define item_shared_flag_clear(item, flag) __atomic_and_fetch(&((item)->shared->flags), ~(flag), __ATOMIC_SEQ_CST)
47 +
48 +#define REFCOUNT_DELETING (-100)
49 +
50 +#define ITEM_FLAGS_TYPE uint8_t
51 +#define KEY_LEN_TYPE uint32_t
52 +#define VALUE_LEN_TYPE uint32_t
53 +
54 +#define ITEM_OPTIONS_BITS 1
55 +#define KEY_LEN_BITS ((sizeof(KEY_LEN_TYPE) * 8) - (sizeof(ITEM_FLAGS_TYPE) * 8) - ITEM_OPTIONS_BITS)
56 +#define KEY_LEN_MAX ((1 << KEY_LEN_BITS) - 1)
57 +
58 +#define VALUE_LEN_BITS ((sizeof(VALUE_LEN_TYPE) * 8) - (sizeof(ITEM_FLAGS_TYPE) * 8))
59 +#define VALUE_LEN_MAX ((1 << VALUE_LEN_BITS) - 1)
60
22 - // IMPORTANT: IF YOU ADD ANOTHER FLAG, YOU NEED TO ALLOCATE ANOTHER BIT TO FLAGS IN NAME_VALUE !!!
23 -} NAME_VALUE_FLAGS;
61
62 /*
63 * Every item in the dictionary has the following structure.
64 */
65
29 -typedef struct name_value {
66 +typedef int32_t REFCOUNT;
67 +
68 +typedef struct dictionary_item_shared {
69 + void *value; // the value of the dictionary item
70 +
71 + // the order of the following items is important!
72 + // The total of their storage should be 64-bits
73 +
74 + REFCOUNT links; // how many links this item has
75 + VALUE_LEN_TYPE value_len:VALUE_LEN_BITS; // the size of the value
76 + ITEM_FLAGS_TYPE flags; // shared flags
77 +} DICTIONARY_ITEM_SHARED;
78 +
79 +struct dictionary_item {
80 #ifdef NETDATA_INTERNAL_CHECKS
81 DICTIONARY *dict;
82 #endif
83
34 - struct name_value *next; // a double linked list to allow fast insertions and deletions
35 - struct name_value *prev;
84 + DICTIONARY_ITEM_SHARED *shared;
85
37 - uint32_t refcount; // the reference counter
38 - uint32_t value_len:29; // the size of the value (assumed binary)
39 - uint8_t flags:3; // the flags for this item
86 + struct dictionary_item *next; // a double linked list to allow fast insertions and deletions
87 + struct dictionary_item *prev;
88
41 - void *value; // the value of the dictionary item
89 union {
43 - STRING *string_name; // the name of the dictionary item
44 - char *caller_name; // the user supplied string pointer
90 + STRING *string_name; // the name of the dictionary item
91 + char *caller_name; // the user supplied string pointer
92 +// void *key_ptr; // binary key pointer
93 };
46 -} NAME_VALUE;
94
48 -struct dictionary {
49 -#ifdef NETDATA_INTERNAL_CHECKS
50 - const char *creation_function;
51 - const char *creation_file;
52 - size_t creation_line;
53 -#endif
95 + // the order of the following items is important!
96 + // The total of their storage should be 64-bits
97 +
98 + REFCOUNT refcount; // the private reference counter
99
55 - DICTIONARY_FLAGS flags; // the flags of the dictionary
100 + KEY_LEN_TYPE key_len:KEY_LEN_BITS; // the size of key indexed (for strings, including the null terminator)
101 + // this is (2^23 - 1) = 8.388.607 bytes max key length.
102
57 - NAME_VALUE *first_item; // the double linked list base pointers
58 - NAME_VALUE *last_item;
103 + ITEM_OPTIONS options:ITEM_OPTIONS_BITS; // permanent configuration options
104 + // (no atomic operations on this - they never change)
105
60 - Pvoid_t JudyHSArray; // the hash table
106 + ITEM_FLAGS_TYPE flags; // runtime changing flags for this item (atomic operations on this)
107 + // cannot be a bit field because of atomics.
108 +};
109
62 - netdata_rwlock_t rwlock; // the r/w lock when DICTIONARY_FLAG_SINGLE_THREADED is not set
110 +struct dictionary_hooks {
111 + REFCOUNT links;
112 + usec_t last_master_deletion_us;
113
64 - void (*ins_callback)(const char *name, void *value, void *data);
114 + void (*ins_callback)(const DICTIONARY_ITEM *item, void *value, void *data);
115 void *ins_callback_data;
116
67 - void (*react_callback)(const char *name, void *value, void *data);
117 + bool (*conflict_callback)(const DICTIONARY_ITEM *item, void *old_value, void *new_value, void *data);
118 + void *conflict_callback_data;
119 +
120 + void (*react_callback)(const DICTIONARY_ITEM *item, void *value, void *data);
121 void *react_callback_data;
122
70 - void (*del_callback)(const char *name, void *value, void *data);
123 + void (*del_callback)(const DICTIONARY_ITEM *item, void *value, void *data);
124 void *del_callback_data;
125 +};
126
73 - void (*conflict_callback)(const char *name, void *old_value, void *new_value, void *data);
74 - void *conflict_callback_data;
127 +struct dictionary_stats dictionary_stats_category_other = {
128 + .name = "other",
129 +};
130
76 - size_t version; // the current version of the dictionary
77 - size_t inserts; // how many index insertions have been performed
78 - size_t deletes; // how many index deletions have been performed
79 - size_t searches; // how many index searches have been performed
80 - size_t resets; // how many times items have reset their values
81 - size_t walkthroughs; // how many walkthroughs have been done
82 - long int memory; // how much memory the dictionary has currently allocated
83 - long int entries; // how many items are currently in the index (the linked list may have more)
84 - long int referenced_items; // how many items of the dictionary are currently being used by 3rd parties
85 - long int pending_deletion_items; // how many items of the dictionary have been deleted, but have not been removed yet
86 - int readers; // how many readers are currently using the dictionary
87 - int writers; // how many writers are currently using the dictionary
88 -
89 - size_t scratchpad_size; // the size of the scratchpad in bytes
90 - uint8_t scratchpad[]; // variable size scratchpad requested by the caller
131 +struct dictionary {
132 +#ifdef NETDATA_INTERNAL_CHECKS
133 + const char *creation_function;
134 + const char *creation_file;
135 + size_t creation_line;
136 +#endif
137 +
138 + usec_t last_gc_run_us;
139 + DICT_OPTIONS options; // the configuration flags of the dictionary (they never change - no atomics)
140 + DICT_FLAGS flags; // run time flags for the dictionary (they change all the time - atomics needed)
141 +
142 + struct { // support for multiple indexing engines
143 + Pvoid_t JudyHSArray; // the hash table
144 + netdata_rwlock_t rwlock; // protect the index
145 + } index;
146 +
147 + struct {
148 + DICTIONARY_ITEM *list; // the double linked list of all items in the dictionary
149 + netdata_rwlock_t rwlock; // protect the linked-list
150 + pid_t writer_pid; // the gettid() of the writer
151 + size_t writer_depth; // nesting of write locks
152 + } items;
153 +
154 + struct dictionary_hooks *hooks; // pointer to external function callbacks to be called at certain points
155 + struct dictionary_stats *stats; // statistics data, when DICT_OPTION_STATS is set
156 +
157 + DICTIONARY *master; // the master dictionary
158 + DICTIONARY *next; // linked list for delayed destruction (garbage collection of whole dictionaries)
159 +
160 + size_t version; // the current version of the dictionary
161 + // it is incremented when:
162 + // - item added
163 + // - item removed
164 + // - item value reset
165 + // - conflict callback returns true
166 + // - function dictionary_version_increment() is called
167 +
168 + long int entries; // how many items are currently in the index (the linked list may have more)
169 + long int referenced_items; // how many items of the dictionary are currently being used by 3rd parties
170 + long int pending_deletion_items; // how many items of the dictionary have been deleted, but have not been removed yet
171 };
172
93 -static inline void linkedlist_namevalue_unlink_unsafe(DICTIONARY *dict, NAME_VALUE *nv);
94 -static size_t namevalue_destroy_unsafe(DICTIONARY *dict, NAME_VALUE *nv);
95 -static inline const char *namevalue_get_name(NAME_VALUE *nv);
173 +// forward definitions of functions used in reverse order in the code
174 +static void garbage_collect_pending_deletes(DICTIONARY *dict);
175 +static inline void item_linked_list_remove(DICTIONARY *dict, DICTIONARY_ITEM *item);
176 +static size_t item_free_with_hooks(DICTIONARY *dict, DICTIONARY_ITEM *item);
177 +static inline const char *item_get_name(const DICTIONARY_ITEM *item);
178 +static bool item_is_not_referenced_and_can_be_removed(DICTIONARY *dict, DICTIONARY_ITEM *item);
179 +static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, size_t name_len, void *item);
180 +static void item_release(DICTIONARY *dict, DICTIONARY_ITEM *item);
181 +
182 +#define ITEM_OK 0
183 +#define ITEM_MARKED_FOR_DELETION (-1) // the item is marked for deletion
184 +#define ITEM_IS_CURRENTLY_BEING_DELETED (-2) // the item is currently being deleted
185 +#define item_check_and_acquire(dict, item) (item_check_and_acquire_advanced(dict, item, false) == ITEM_OK)
186 +static int item_check_and_acquire_advanced(DICTIONARY *dict, DICTIONARY_ITEM *item, bool having_index_lock);
187
188 // ----------------------------------------------------------------------------
98 -// callbacks registration
189 +// memory statistics
190
100 -void dictionary_register_insert_callback(DICTIONARY *dict, void (*ins_callback)(const char *name, void *value, void *data), void *data) {
101 - dict->ins_callback = ins_callback;
102 - dict->ins_callback_data = data;
191 +static inline void DICTIONARY_STATS_PLUS_MEMORY(DICTIONARY *dict, size_t key_size, size_t item_size, size_t value_size) {
192 + if(key_size)
193 + __atomic_fetch_add(&dict->stats->memory.indexed, (long)key_size, __ATOMIC_RELAXED);
194 +
195 + if(item_size)
196 + __atomic_fetch_add(&dict->stats->memory.dict, (long)item_size, __ATOMIC_RELAXED);
197 +
198 + if(value_size)
199 + __atomic_fetch_add(&dict->stats->memory.values, (long)value_size, __ATOMIC_RELAXED);
200 }
201 +static inline void DICTIONARY_STATS_MINUS_MEMORY(DICTIONARY *dict, size_t key_size, size_t item_size, size_t value_size) {
202 + if(key_size)
203 + __atomic_fetch_sub(&dict->stats->memory.indexed, (long)key_size, __ATOMIC_RELAXED);
204
105 -void dictionary_register_delete_callback(DICTIONARY *dict, void (*del_callback)(const char *name, void *value, void *data), void *data) {
106 - dict->del_callback = del_callback;
107 - dict->del_callback_data = data;
205 + if(item_size)
206 + __atomic_fetch_sub(&dict->stats->memory.dict, (long)item_size, __ATOMIC_RELAXED);
207 +
208 + if(value_size)
209 + __atomic_fetch_sub(&dict->stats->memory.values, (long)value_size, __ATOMIC_RELAXED);
210 }
211
110 -void dictionary_register_conflict_callback(DICTIONARY *dict, void (*conflict_callback)(const char *name, void *old_value, void *new_value, void *data), void *data) {
111 - dict->conflict_callback = conflict_callback;
112 - dict->conflict_callback_data = data;
212 +// ----------------------------------------------------------------------------
213 +// callbacks registration
214 +
215 +static inline void dictionary_hooks_allocate(DICTIONARY *dict) {
216 + if(dict->hooks) return;
217 +
218 + dict->hooks = callocz(1, sizeof(struct dictionary_hooks));
219 + dict->hooks->links = 1;
220 +
221 + DICTIONARY_STATS_PLUS_MEMORY(dict, 0, sizeof(struct dictionary_hooks), 0);
222 }
223
115 -void dictionary_register_react_callback(DICTIONARY *dict, void (*react_callback)(const char *name, void *value, void *data), void *data) {
116 - dict->react_callback = react_callback;
117 - dict->react_callback_data = data;
224 +static inline size_t dictionary_hooks_free(DICTIONARY *dict) {
225 + if(!dict->hooks) return 0;
226 +
227 + REFCOUNT links = __atomic_sub_fetch(&dict->hooks->links, 1, __ATOMIC_SEQ_CST);
228 + if(links == 0) {
229 + freez(dict->hooks);
230 + dict->hooks = NULL;
231 +
232 + DICTIONARY_STATS_MINUS_MEMORY(dict, 0, sizeof(struct dictionary_hooks), 0);
233 + return sizeof(struct dictionary_hooks);
234 + }
235 +
236 + return 0;
237 }
238
120 -// ----------------------------------------------------------------------------
121 -// dictionary statistics maintenance
239 +void dictionary_register_insert_callback(DICTIONARY *dict, void (*ins_callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data) {
240 + if(unlikely(is_view_dictionary(dict)))
241 + fatal("DICTIONARY: called %s() on a view.", __FUNCTION__ );
242
123 -long int dictionary_stats_allocated_memory(DICTIONARY *dict) {
124 - if(unlikely(!dict)) return 0;
125 - return dict->memory;
243 + dictionary_hooks_allocate(dict);
244 + dict->hooks->ins_callback = ins_callback;
245 + dict->hooks->ins_callback_data = data;
246 }
127 -long int dictionary_stats_entries(DICTIONARY *dict) {
128 - if(unlikely(!dict)) return 0;
129 - return dict->entries;
247 +
248 +void dictionary_register_conflict_callback(DICTIONARY *dict, bool (*conflict_callback)(const DICTIONARY_ITEM *item, void *old_value, void *new_value, void *data), void *data) {
249 + if(unlikely(is_view_dictionary(dict)))
250 + fatal("DICTIONARY: called %s() on a view.", __FUNCTION__ );
251 +
252 + dictionary_hooks_allocate(dict);
253 + dict->hooks->conflict_callback = conflict_callback;
254 + dict->hooks->conflict_callback_data = data;
255 }
131 -size_t dictionary_stats_version(DICTIONARY *dict) {
132 - if(unlikely(!dict)) return 0;
133 - return dict->version;
256 +
257 +void dictionary_register_react_callback(DICTIONARY *dict, void (*react_callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data) {
258 + if(unlikely(is_view_dictionary(dict)))
259 + fatal("DICTIONARY: called %s() on a view.", __FUNCTION__ );
260 +
261 + dictionary_hooks_allocate(dict);
262 + dict->hooks->react_callback = react_callback;
263 + dict->hooks->react_callback_data = data;
264 }
135 -size_t dictionary_stats_searches(DICTIONARY *dict) {
136 - if(unlikely(!dict)) return 0;
137 - return dict->searches;
265 +
266 +void dictionary_register_delete_callback(DICTIONARY *dict, void (*del_callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data) {
267 + if(unlikely(is_view_dictionary(dict)))
268 + fatal("DICTIONARY: called %s() on a view.", __FUNCTION__ );
269 +
270 + dictionary_hooks_allocate(dict);
271 + dict->hooks->del_callback = del_callback;
272 + dict->hooks->del_callback_data = data;
273 }
139 -size_t dictionary_stats_inserts(DICTIONARY *dict) {
274 +
275 +// ----------------------------------------------------------------------------
276 +// dictionary statistics API
277 +
278 +size_t dictionary_version(DICTIONARY *dict) {
279 if(unlikely(!dict)) return 0;
141 - return dict->inserts;
280 +
281 + // this is required for views to return the right number
282 + garbage_collect_pending_deletes(dict);
283 +
284 + return __atomic_load_n(&dict->version, __ATOMIC_SEQ_CST);
285 }
143 -size_t dictionary_stats_deletes(DICTIONARY *dict) {
286 +size_t dictionary_entries(DICTIONARY *dict) {
287 if(unlikely(!dict)) return 0;
145 - return dict->deletes;
288 +
289 + // this is required for views to return the right number
290 + garbage_collect_pending_deletes(dict);
291 +
292 + long int entries = __atomic_load_n(&dict->entries, __ATOMIC_SEQ_CST);
293 + if(entries < 0)
294 + fatal("DICTIONARY: entries is negative: %ld", entries);
295 +
296 + return entries;
297 }
147 -size_t dictionary_stats_resets(DICTIONARY *dict) {
298 +size_t dictionary_referenced_items(DICTIONARY *dict) {
299 if(unlikely(!dict)) return 0;
149 - return dict->resets;
300 +
301 + long int referenced_items = __atomic_load_n(&dict->referenced_items, __ATOMIC_SEQ_CST);
302 + if(referenced_items < 0)
303 + fatal("DICTIONARY: referenced items is negative: %ld", referenced_items);
304 +
305 + return referenced_items;
306 }
151 -size_t dictionary_stats_walkthroughs(DICTIONARY *dict) {
307 +
308 +long int dictionary_stats_for_registry(DICTIONARY *dict) {
309 if(unlikely(!dict)) return 0;
153 - return dict->walkthroughs;
310 + return (dict->stats->memory.indexed + dict->stats->memory.dict);
311 }
155 -size_t dictionary_stats_referenced_items(DICTIONARY *dict) {
156 - if(unlikely(!dict)) return 0;
157 - return __atomic_load_n(&dict->referenced_items, __ATOMIC_SEQ_CST);
312 +void dictionary_version_increment(DICTIONARY *dict) {
313 + __atomic_fetch_add(&dict->version, 1, __ATOMIC_SEQ_CST);
314 }
315
316 +// ----------------------------------------------------------------------------
317 +// internal statistics API
318 +
319 static inline void DICTIONARY_STATS_SEARCHES_PLUS1(DICTIONARY *dict) {
161 - if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
162 - dict->searches++;
163 - }
164 - else {
165 - __atomic_fetch_add(&dict->searches, 1, __ATOMIC_RELAXED);
166 - }
320 + __atomic_fetch_add(&dict->stats->ops.searches, 1, __ATOMIC_RELAXED);
321 }
168 -static inline void DICTIONARY_STATS_ENTRIES_PLUS1(DICTIONARY *dict, size_t size) {
169 - if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
322 +static inline void DICTIONARY_ENTRIES_PLUS1(DICTIONARY *dict) {
323 + // statistics
324 + __atomic_fetch_add(&dict->stats->items.entries, 1, __ATOMIC_RELAXED);
325 + __atomic_fetch_add(&dict->stats->items.referenced, 1, __ATOMIC_RELAXED);
326 + __atomic_fetch_add(&dict->stats->ops.inserts, 1, __ATOMIC_RELAXED);
327 +
328 + if(unlikely(is_dictionary_single_threaded(dict))) {
329 dict->version++;
171 - dict->inserts++;
330 dict->entries++;
173 - dict->memory += (long)size;
331 + dict->referenced_items++;
332 +
333 }
334 else {
335 __atomic_fetch_add(&dict->version, 1, __ATOMIC_SEQ_CST);
177 - __atomic_fetch_add(&dict->inserts, 1, __ATOMIC_RELAXED);
178 - __atomic_fetch_add(&dict->entries, 1, __ATOMIC_RELAXED);
179 - __atomic_fetch_add(&dict->memory, (long)size, __ATOMIC_RELAXED);
336 + __atomic_fetch_add(&dict->entries, 1, __ATOMIC_SEQ_CST);
337 + __atomic_fetch_add(&dict->referenced_items, 1, __ATOMIC_SEQ_CST);
338 }
339 }
182 -static inline void DICTIONARY_STATS_ENTRIES_MINUS1(DICTIONARY *dict) {
183 - if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
340 +static inline void DICTIONARY_ENTRIES_MINUS1(DICTIONARY *dict) {
341 + // statistics
342 + __atomic_fetch_add(&dict->stats->ops.deletes, 1, __ATOMIC_RELAXED);
343 + __atomic_fetch_sub(&dict->stats->items.entries, 1, __ATOMIC_RELAXED);
344 +
345 + if(unlikely(is_dictionary_single_threaded(dict))) {
346 dict->version++;
185 - dict->deletes++;
347 dict->entries--;
348 }
349 else {
350 __atomic_fetch_add(&dict->version, 1, __ATOMIC_SEQ_CST);
190 - __atomic_fetch_add(&dict->deletes, 1, __ATOMIC_RELAXED);
191 - __atomic_fetch_sub(&dict->entries, 1, __ATOMIC_RELAXED);
192 - }
193 -}
194 -static inline void DICTIONARY_STATS_ENTRIES_MINUS_MEMORY(DICTIONARY *dict, size_t size) {
195 - if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
196 - dict->memory -= (long)size;
197 - }
198 - else {
199 - __atomic_fetch_sub(&dict->memory, (long)size, __ATOMIC_RELAXED);
351 + __atomic_fetch_sub(&dict->entries, 1, __ATOMIC_SEQ_CST);
352 }
353 }
202 -static inline void DICTIONARY_STATS_VALUE_RESETS_PLUS1(DICTIONARY *dict, size_t oldsize, size_t newsize) {
203 - if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
354 +static inline void DICTIONARY_VALUE_RESETS_PLUS1(DICTIONARY *dict) {
355 + __atomic_fetch_add(&dict->stats->ops.resets, 1, __ATOMIC_RELAXED);
356 +
357 + if(unlikely(is_dictionary_single_threaded(dict)))
358 dict->version++;
205 - dict->resets++;
206 - dict->memory += (long)newsize;
207 - dict->memory -= (long)oldsize;
208 - }
209 - else {
359 + else
360 __atomic_fetch_add(&dict->version, 1, __ATOMIC_SEQ_CST);
211 - __atomic_fetch_add(&dict->resets, 1, __ATOMIC_RELAXED);
212 - __atomic_fetch_add(&dict->memory, (long)newsize, __ATOMIC_RELAXED);
213 - __atomic_fetch_sub(&dict->memory, (long)oldsize, __ATOMIC_RELAXED);
214 - }
361 }
216 -
362 +static inline void DICTIONARY_STATS_TRAVERSALS_PLUS1(DICTIONARY *dict) {
363 + __atomic_fetch_add(&dict->stats->ops.traversals, 1, __ATOMIC_RELAXED);
364 +}
365 static inline void DICTIONARY_STATS_WALKTHROUGHS_PLUS1(DICTIONARY *dict) {
218 - if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
219 - dict->walkthroughs++;
220 - }
221 - else {
222 - __atomic_fetch_add(&dict->walkthroughs, 1, __ATOMIC_RELAXED);
223 - }
366 + __atomic_fetch_add(&dict->stats->ops.walkthroughs, 1, __ATOMIC_RELAXED);
367 +}
368 +static inline void DICTIONARY_STATS_CHECK_SPINS_PLUS(DICTIONARY *dict, size_t count) {
369 + __atomic_fetch_add(&dict->stats->spin_locks.use, count, __ATOMIC_RELAXED);
370 +}
371 +static inline void DICTIONARY_STATS_INSERT_SPINS_PLUS(DICTIONARY *dict, size_t count) {
372 + __atomic_fetch_add(&dict->stats->spin_locks.insert, count, __ATOMIC_RELAXED);
373 +}
374 +static inline void DICTIONARY_STATS_SEARCH_IGNORES_PLUS1(DICTIONARY *dict) {
375 + __atomic_fetch_add(&dict->stats->spin_locks.search, 1, __ATOMIC_RELAXED);
376 +}
377 +static inline void DICTIONARY_STATS_CALLBACK_INSERTS_PLUS1(DICTIONARY *dict) {
378 + __atomic_fetch_add(&dict->stats->callbacks.inserts, 1, __ATOMIC_RELAXED);
379 +}
380 +static inline void DICTIONARY_STATS_CALLBACK_CONFLICTS_PLUS1(DICTIONARY *dict) {
381 + __atomic_fetch_add(&dict->stats->callbacks.conflicts, 1, __ATOMIC_RELAXED);
382 +}
383 +static inline void DICTIONARY_STATS_CALLBACK_REACTS_PLUS1(DICTIONARY *dict) {
384 + __atomic_fetch_add(&dict->stats->callbacks.reacts, 1, __ATOMIC_RELAXED);
385 +}
386 +static inline void DICTIONARY_STATS_CALLBACK_DELETES_PLUS1(DICTIONARY *dict) {
387 + __atomic_fetch_add(&dict->stats->callbacks.deletes, 1, __ATOMIC_RELAXED);
388 +}
389 +static inline void DICTIONARY_STATS_GARBAGE_COLLECTIONS_PLUS1(DICTIONARY *dict) {
390 + __atomic_fetch_add(&dict->stats->ops.garbage_collections, 1, __ATOMIC_RELAXED);
391 +}
392 +static inline void DICTIONARY_STATS_DICT_CREATIONS_PLUS1(DICTIONARY *dict) {
393 + __atomic_fetch_add(&dict->stats->dictionaries.active, 1, __ATOMIC_RELAXED);
394 + __atomic_fetch_add(&dict->stats->ops.creations, 1, __ATOMIC_RELAXED);
395 +}
396 +static inline void DICTIONARY_STATS_DICT_DESTRUCTIONS_PLUS1(DICTIONARY *dict) {
397 + __atomic_fetch_sub(&dict->stats->dictionaries.active, 1, __ATOMIC_RELAXED);
398 + __atomic_fetch_add(&dict->stats->ops.destructions, 1, __ATOMIC_RELAXED);
399 +}
400 +static inline void DICTIONARY_STATS_DICT_DESTROY_QUEUED_PLUS1(DICTIONARY *dict) {
401 + __atomic_fetch_add(&dict->stats->dictionaries.deleted, 1, __ATOMIC_RELAXED);
402 +}
403 +static inline void DICTIONARY_STATS_DICT_DESTROY_QUEUED_MINUS1(DICTIONARY *dict) {
404 + __atomic_fetch_sub(&dict->stats->dictionaries.deleted, 1, __ATOMIC_RELAXED);
405 }
406 +static inline void DICTIONARY_STATS_DICT_FLUSHES_PLUS1(DICTIONARY *dict) {
407 + __atomic_fetch_add(&dict->stats->ops.flushes, 1, __ATOMIC_RELAXED);
408 +}
409 +
410 +static inline long int DICTIONARY_REFERENCED_ITEMS_PLUS1(DICTIONARY *dict) {
411 + __atomic_fetch_add(&dict->stats->items.referenced, 1, __ATOMIC_RELAXED);
412
226 -static inline size_t DICTIONARY_STATS_REFERENCED_ITEMS_PLUS1(DICTIONARY *dict) {
227 - return __atomic_add_fetch(&dict->referenced_items, 1, __ATOMIC_SEQ_CST);
413 + if(unlikely(is_dictionary_single_threaded(dict)))
414 + return ++dict->referenced_items;
415 + else
416 + return __atomic_add_fetch(&dict->referenced_items, 1, __ATOMIC_SEQ_CST);
417 }
418
230 -static inline size_t DICTIONARY_STATS_REFERENCED_ITEMS_MINUS1(DICTIONARY *dict) {
231 - return __atomic_sub_fetch(&dict->referenced_items, 1, __ATOMIC_SEQ_CST);
419 +static inline long int DICTIONARY_REFERENCED_ITEMS_MINUS1(DICTIONARY *dict) {
420 + __atomic_fetch_sub(&dict->stats->items.referenced, 1, __ATOMIC_RELAXED);
421 +
422 + if(unlikely(is_dictionary_single_threaded(dict)))
423 + return --dict->referenced_items;
424 + else
425 + return __atomic_sub_fetch(&dict->referenced_items, 1, __ATOMIC_SEQ_CST);
426 }
427
234 -static inline size_t DICTIONARY_STATS_PENDING_DELETES_PLUS1(DICTIONARY *dict) {
235 - return __atomic_add_fetch(&dict->pending_deletion_items, 1, __ATOMIC_SEQ_CST);
428 +static inline long int DICTIONARY_PENDING_DELETES_PLUS1(DICTIONARY *dict) {
429 + __atomic_fetch_add(&dict->stats->items.pending_deletion, 1, __ATOMIC_RELAXED);
430 +
431 + if(unlikely(is_dictionary_single_threaded(dict)))
432 + return ++dict->pending_deletion_items;
433 + else
434 + return __atomic_add_fetch(&dict->pending_deletion_items, 1, __ATOMIC_SEQ_CST);
435 }
436
238 -static inline size_t DICTIONARY_STATS_PENDING_DELETES_MINUS1(DICTIONARY *dict) {
239 - return __atomic_sub_fetch(&dict->pending_deletion_items, 1, __ATOMIC_SEQ_CST);
437 +static inline long int DICTIONARY_PENDING_DELETES_MINUS1(DICTIONARY *dict) {
438 + __atomic_fetch_sub(&dict->stats->items.pending_deletion, 1, __ATOMIC_RELAXED);
439 +
440 + if(unlikely(is_dictionary_single_threaded(dict)))
441 + return --dict->pending_deletion_items;
442 + else
443 + return __atomic_sub_fetch(&dict->pending_deletion_items, 1, __ATOMIC_SEQ_CST);
444 }
445
242 -static inline size_t DICTIONARY_STATS_PENDING_DELETES_GET(DICTIONARY *dict) {
243 - return __atomic_load_n(&dict->pending_deletion_items, __ATOMIC_SEQ_CST);
446 +static inline long int DICTIONARY_PENDING_DELETES_GET(DICTIONARY *dict) {
447 + if(unlikely(is_dictionary_single_threaded(dict)))
448 + return dict->pending_deletion_items;
449 + else
450 + return __atomic_load_n(&dict->pending_deletion_items, __ATOMIC_SEQ_CST);
451 }
452
246 -static inline int DICTIONARY_NAME_VALUE_REFCOUNT_GET(NAME_VALUE *nv) {
247 - return __atomic_load_n(&nv->refcount, __ATOMIC_SEQ_CST);
453 +static inline REFCOUNT DICTIONARY_ITEM_REFCOUNT_GET(DICTIONARY *dict, DICTIONARY_ITEM *item) {
454 + if(unlikely(dict && is_dictionary_single_threaded(dict))) // this is an exception, dict can be null
455 + return item->refcount;
456 + else
457 + return (REFCOUNT)__atomic_load_n(&item->refcount, __ATOMIC_SEQ_CST);
458 }
459
460 // ----------------------------------------------------------------------------
251 -// garbage collector
252 -// it is called every time someone gets a write lock to the dictionary
461 +// callbacks execution
462
254 -static void garbage_collect_pending_deletes_unsafe(DICTIONARY *dict) {
255 - if(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS)) return;
463 +static void dictionary_execute_insert_callback(DICTIONARY *dict, DICTIONARY_ITEM *item, void *constructor_data) {
464 + if(likely(!dict->hooks || !dict->hooks->ins_callback))
465 + return;
466
257 - if(likely(!DICTIONARY_STATS_PENDING_DELETES_GET(dict))) return;
467 + if(unlikely(is_view_dictionary(dict)))
468 + fatal("DICTIONARY: called %s() on a view.", __FUNCTION__ );
469
259 - NAME_VALUE *nv = dict->first_item;
260 - while(nv) {
261 - if((nv->flags & NAME_VALUE_FLAG_DELETED) && DICTIONARY_NAME_VALUE_REFCOUNT_GET(nv) == 0) {
262 - NAME_VALUE *nv_next = nv->next;
470 + internal_error(false,
471 + "DICTIONARY: Running insert callback on item '%s' of dictionary created from %s() %zu@%s.",
472 + item_get_name(item),
473 + dict->creation_function,
474 + dict->creation_line,
475 + dict->creation_file);
476
264 - linkedlist_namevalue_unlink_unsafe(dict, nv);
265 - namevalue_destroy_unsafe(dict, nv);
477 + DICTIONARY_STATS_CALLBACK_INSERTS_PLUS1(dict);
478 + dict->hooks->ins_callback(item, item->shared->value, constructor_data?constructor_data:dict->hooks->ins_callback_data);
479 +}
480
267 - size_t pending = DICTIONARY_STATS_PENDING_DELETES_MINUS1(dict);
268 - if(!pending) break;
481 +static bool dictionary_execute_conflict_callback(DICTIONARY *dict, DICTIONARY_ITEM *item, void *new_value, void *constructor_data) {
482 + if(likely(!dict->hooks || !dict->hooks->conflict_callback))
483 + return false;
484
270 - nv = nv_next;
271 - }
272 - else
273 - nv = nv->next;
274 - }
485 + if(unlikely(is_view_dictionary(dict)))
486 + fatal("DICTIONARY: called %s() on a view.", __FUNCTION__ );
487 +
488 + internal_error(false,
489 + "DICTIONARY: Running conflict callback on item '%s' of dictionary created from %s() %zu@%s.",
490 + item_get_name(item),
491 + dict->creation_function,
492 + dict->creation_line,
493 + dict->creation_file);
494 +
495 + DICTIONARY_STATS_CALLBACK_CONFLICTS_PLUS1(dict);
496 + return dict->hooks->conflict_callback(
497 + item, item->shared->value, new_value,
498 + constructor_data ? constructor_data : dict->hooks->conflict_callback_data);
499 }
500
277 -// ----------------------------------------------------------------------------
278 -// dictionary locks
501 +static void dictionary_execute_react_callback(DICTIONARY *dict, DICTIONARY_ITEM *item, void *constructor_data) {
502 + if(likely(!dict->hooks || !dict->hooks->react_callback))
503 + return;
504 +
505 + if(unlikely(is_view_dictionary(dict)))
506 + fatal("DICTIONARY: called %s() on a view.", __FUNCTION__ );
507 +
508 + internal_error(false,
509 + "DICTIONARY: Running react callback on item '%s' of dictionary created from %s() %zu@%s.",
510 + item_get_name(item),
511 + dict->creation_function,
512 + dict->creation_line,
513 + dict->creation_file);
514
280 -static inline size_t dictionary_lock_init(DICTIONARY *dict) {
281 - if(likely(!(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))) {
282 - netdata_rwlock_init(&dict->rwlock);
515 + DICTIONARY_STATS_CALLBACK_REACTS_PLUS1(dict);
516 + dict->hooks->react_callback(item, item->shared->value,
517 + constructor_data?constructor_data:dict->hooks->react_callback_data);
518 +}
519 +
520 +static void dictionary_execute_delete_callback(DICTIONARY *dict, DICTIONARY_ITEM *item) {
521 + if(likely(!dict->hooks || !dict->hooks->del_callback))
522 + return;
523 +
524 + // We may execute the delete callback on items deleted from a view,
525 + // because we may have references to it, after the master is gone
526 + // so, the shared structure will remain until the last reference is released.
527
284 - if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS)
285 - dict->flags &= ~DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
528 + internal_error(false,
529 + "DICTIONARY: Running delete callback on item '%s' of dictionary created from %s() %zu@%s.",
530 + item_get_name(item),
531 + dict->creation_function,
532 + dict->creation_line,
533 + dict->creation_file);
534 +
535 + DICTIONARY_STATS_CALLBACK_DELETES_PLUS1(dict);
536 + dict->hooks->del_callback(item, item->shared->value, dict->hooks->del_callback_data);
537 +}
538 +
539 +// ----------------------------------------------------------------------------
540 +// dictionary locks
541
542 +static inline size_t dictionary_locks_init(DICTIONARY *dict) {
543 + if(likely(!is_dictionary_single_threaded(dict))) {
544 + netdata_rwlock_init(&dict->index.rwlock);
545 + netdata_rwlock_init(&dict->items.rwlock);
546 return 0;
547 }
289 -
290 - // we are single threaded
291 - dict->flags |= DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
548 return 0;
549 }
550
295 -static inline size_t dictionary_lock_free(DICTIONARY *dict) {
296 - if(likely(!(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))) {
297 - netdata_rwlock_destroy(&dict->rwlock);
551 +static inline size_t dictionary_locks_destroy(DICTIONARY *dict) {
552 + if(likely(!is_dictionary_single_threaded(dict))) {
553 + netdata_rwlock_destroy(&dict->index.rwlock);
554 + netdata_rwlock_destroy(&dict->items.rwlock);
555 return 0;
556 }
557 return 0;
558 }
559
303 -static void dictionary_lock(DICTIONARY *dict, char rw) {
304 - if(rw == DICTIONARY_LOCK_NONE || rw == 'U') return;
560 +static inline void ll_recursive_lock_set_thread_as_writer(DICTIONARY *dict) {
561 + pid_t expected = 0, desired = gettid();
562 + if(!__atomic_compare_exchange_n(&dict->items.writer_pid, &expected, desired, false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST))
563 + fatal("DICTIONARY: Cannot set thread %d as exclusive writer, expected %d, desired %d, found %d.", gettid(), expected, desired, __atomic_load_n(&dict->items.writer_pid, __ATOMIC_SEQ_CST));
564 +}
565
306 - if(rw == DICTIONARY_LOCK_READ || rw == DICTIONARY_LOCK_REENTRANT || rw == 'R') {
307 - // read lock
308 - __atomic_add_fetch(&dict->readers, 1, __ATOMIC_RELAXED);
309 - }
310 - else {
311 - // write lock
312 - __atomic_add_fetch(&dict->writers, 1, __ATOMIC_RELAXED);
313 - }
566 +static inline void ll_recursive_unlock_unset_thread_writer(DICTIONARY *dict) {
567 + pid_t expected = gettid(), desired = 0;
568 + if(!__atomic_compare_exchange_n(&dict->items.writer_pid, &expected, desired, false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST))
569 + fatal("DICTIONARY: Cannot unset thread %d as exclusive writer, expected %d, desired %d, found %d.", gettid(), expected, desired, __atomic_load_n(&dict->items.writer_pid, __ATOMIC_SEQ_CST));
570 +}
571 +
572 +static inline bool ll_recursive_lock_is_thread_the_writer(DICTIONARY *dict) {
573 + pid_t tid = gettid();
574 + return tid > 0 && tid == __atomic_load_n(&dict->items.writer_pid, __ATOMIC_SEQ_CST);
575 +}
576
315 - if(likely(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))
577 +static void ll_recursive_lock(DICTIONARY *dict, char rw) {
578 + if(unlikely(is_dictionary_single_threaded(dict)))
579 return;
580
581 + if(ll_recursive_lock_is_thread_the_writer(dict)) {
582 + dict->items.writer_depth++;
583 + return;
584 + }
585 +
586 if(rw == DICTIONARY_LOCK_READ || rw == DICTIONARY_LOCK_REENTRANT || rw == 'R') {
587 // read lock
320 - netdata_rwlock_rdlock(&dict->rwlock);
321 -
322 - if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
323 - internal_error(true, "DICTIONARY: left-over exclusive access to dictionary created by %s (%zu@%s) found", dict->creation_function, dict->creation_line, dict->creation_file);
324 - dict->flags &= ~DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
325 - }
588 + netdata_rwlock_rdlock(&dict->items.rwlock);
589 }
590 else {
591 // write lock
329 - netdata_rwlock_wrlock(&dict->rwlock);
330 -
331 - dict->flags |= DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
592 + netdata_rwlock_wrlock(&dict->items.rwlock);
593 + ll_recursive_lock_set_thread_as_writer(dict);
594 }
595 }
596
335 -static void dictionary_unlock(DICTIONARY *dict, char rw) {
336 - if(rw == DICTIONARY_LOCK_NONE || rw == 'U') return;
597 +static void ll_recursive_unlock(DICTIONARY *dict, char rw) {
598 + if(unlikely(is_dictionary_single_threaded(dict)))
599 + return;
600 +
601 + if(ll_recursive_lock_is_thread_the_writer(dict) && dict->items.writer_depth > 0) {
602 + dict->items.writer_depth--;
603 + return;
604 + }
605
606 if(rw == DICTIONARY_LOCK_READ || rw == DICTIONARY_LOCK_REENTRANT || rw == 'R') {
607 // read unlock
340 - __atomic_sub_fetch(&dict->readers, 1, __ATOMIC_RELAXED);
608 +
609 + netdata_rwlock_unlock(&dict->items.rwlock);
610 }
611 else {
612 // write unlock
344 - garbage_collect_pending_deletes_unsafe(dict);
345 - __atomic_sub_fetch(&dict->writers, 1, __ATOMIC_RELAXED);
613 +
614 + ll_recursive_unlock_unset_thread_writer(dict);
615 +
616 + netdata_rwlock_unlock(&dict->items.rwlock);
617 }
618 +}
619 +
620 +
621 +static inline void dictionary_index_lock_rdlock(DICTIONARY *dict) {
622 + if(unlikely(is_dictionary_single_threaded(dict)))
623 + return;
624
348 - if(likely(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))
625 + netdata_rwlock_rdlock(&dict->index.rwlock);
626 +}
627 +static inline void dictionary_index_lock_wrlock(DICTIONARY *dict) {
628 + if(unlikely(is_dictionary_single_threaded(dict)))
629 return;
630
351 - if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS)
352 - dict->flags &= ~DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
631 + netdata_rwlock_wrlock(&dict->index.rwlock);
632 +}
633 +static inline void dictionary_index_lock_unlock(DICTIONARY *dict) {
634 + if(unlikely(is_dictionary_single_threaded(dict)))
635 + return;
636
354 - netdata_rwlock_unlock(&dict->rwlock);
637 + netdata_rwlock_unlock(&dict->index.rwlock);
638 }
639
640 // ----------------------------------------------------------------------------
358 -// deferred deletions
641 +// items garbage collector
642
360 -void dictionary_defer_all_deletions_unsafe(DICTIONARY *dict, char rw) {
361 - if(rw == 'r' || rw == 'R') {
362 - // read locked - no need to defer deletions
363 - ;
364 - }
365 - else {
366 - // write locked - defer deletions
367 - dict->flags |= DICTIONARY_FLAG_DEFER_ALL_DELETIONS;
368 - }
369 -}
643 +static void garbage_collect_pending_deletes(DICTIONARY *dict) {
644 + usec_t last_master_deletion_us = dict->hooks?__atomic_load_n(&dict->hooks->last_master_deletion_us, __ATOMIC_SEQ_CST):0;
645 + usec_t last_gc_run_us = __atomic_load_n(&dict->last_gc_run_us, __ATOMIC_SEQ_CST);
646
371 -void dictionary_restore_all_deletions_unsafe(DICTIONARY *dict, char rw) {
372 - if(rw == 'r' || rw == 'R') {
373 - // read locked - no need to defer deletions
374 - internal_error(dict->flags & DICTIONARY_FLAG_DEFER_ALL_DELETIONS, "DICTIONARY: deletions are deferred on a read lock");
375 - }
376 - else {
377 - // write locked - defer deletions
378 - if(dict->flags & DICTIONARY_FLAG_DEFER_ALL_DELETIONS)
379 - dict->flags &= ~DICTIONARY_FLAG_DEFER_ALL_DELETIONS;
647 + bool is_view = is_view_dictionary(dict);
648 +
649 + if(likely(!(
650 + DICTIONARY_PENDING_DELETES_GET(dict) > 0 ||
651 + (is_view && last_master_deletion_us > last_gc_run_us)
652 + )))
653 + return;
654 +
655 + ll_recursive_lock(dict, DICTIONARY_LOCK_WRITE);
656 +
657 + __atomic_store_n(&dict->last_gc_run_us, now_realtime_usec(), __ATOMIC_SEQ_CST);
658 +
659 + if(is_view)
660 + dictionary_index_lock_wrlock(dict);
661 +
662 + DICTIONARY_STATS_GARBAGE_COLLECTIONS_PLUS1(dict);
663 +
664 + size_t deleted = 0, pending = 0, examined = 0;
665 + DICTIONARY_ITEM *item = dict->items.list, *item_next;
666 + while(item) {
667 + examined++;
668 +
669 + // this will cleanup
670 + item_next = item->next;
671 + int rc = item_check_and_acquire_advanced(dict, item, is_view);
672 +
673 + if(rc == ITEM_MARKED_FOR_DELETION) {
674 + // we don't have got a reference
675 +
676 + if(item_is_not_referenced_and_can_be_removed(dict, item)) {
677 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(dict->items.list, item, prev, next);
678 + item_free_with_hooks(dict, item);
679 + deleted++;
680 +
681 + pending = DICTIONARY_PENDING_DELETES_MINUS1(dict);
682 + if (!pending)
683 + break;
684 + }
685 + }
686 + else if(rc == ITEM_IS_CURRENTLY_BEING_DELETED)
687 + ; // do not touch this item (we haven't got a reference)
688 +
689 + else if(rc == ITEM_OK)
690 + item_release(dict, item);
691 +
692 + item = item_next;
693 }
694 +
695 + if(is_view)
696 + dictionary_index_lock_unlock(dict);
697 +
698 + ll_recursive_unlock(dict, DICTIONARY_LOCK_WRITE);
699 +
700 + (void)deleted;
701 + (void)examined;
702 +
703 + internal_error(false, "DICTIONARY: garbage collected dictionary created by %s (%zu@%s), examined %zu items, deleted %zu items, still pending %zu items",
704 + dict->creation_function, dict->creation_line, dict->creation_file, examined, deleted, pending);
705 +
706 }
707
708 // ----------------------------------------------------------------------------
@@ -399,105 +724,229 @@ static inline size_t reference_counter_free(DICTIONARY *dict) {
724 return 0;
725 }
726
402 -static int reference_counter_increase(NAME_VALUE *nv) {
403 - int refcount = __atomic_add_fetch(&nv->refcount, 1, __ATOMIC_SEQ_CST);
404 - if(refcount == 1)
405 - fatal("DICTIONARY: request to dup item '%s' but its reference counter was zero", namevalue_get_name(nv));
406 - return refcount;
407 -}
727 +static void item_acquire(DICTIONARY *dict, DICTIONARY_ITEM *item) {
728 + REFCOUNT refcount;
729
409 -static int reference_counter_acquire(DICTIONARY *dict, NAME_VALUE *nv) {
410 - int refcount;
411 - if(likely(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))
412 - refcount = ++nv->refcount;
413 - else
414 - refcount = __atomic_add_fetch(&nv->refcount, 1, __ATOMIC_SEQ_CST);
730 + if(unlikely(is_dictionary_single_threaded(dict))) {
731 + refcount = ++item->refcount;
732 + }
733 + else {
734 + // increment the refcount
735 + refcount = __atomic_add_fetch(&item->refcount, 1, __ATOMIC_SEQ_CST);
736 + }
737 +
738 + if(refcount <= 0) {
739 + internal_error(
740 + true,
741 + "DICTIONARY: attempted to acquire item which is deleted (refcount = %d): "
742 + "'%s' on dictionary created by %s() (%zu@%s)",
743 + refcount - 1,
744 + item_get_name(item),
745 + dict->creation_function,
746 + dict->creation_line,
747 + dict->creation_file);
748 +
749 + fatal(
750 + "DICTIONARY: request to acquire item '%s', which is deleted (refcount = %d)!",
751 + item_get_name(item),
752 + refcount - 1);
753 + }
754
755 if(refcount == 1) {
756 // referenced items counts number of unique items referenced
757 // so, we increase it only when refcount == 1
419 - DICTIONARY_STATS_REFERENCED_ITEMS_PLUS1(dict);
758 + DICTIONARY_REFERENCED_ITEMS_PLUS1(dict);
759
760 // if this is a deleted item, but the counter increased to 1
761 // we need to remove it from the pending items to delete
423 - if (nv->flags & NAME_VALUE_FLAG_DELETED)
424 - DICTIONARY_STATS_PENDING_DELETES_MINUS1(dict);
762 + if(item_flag_check(item, ITEM_FLAG_DELETED))
763 + DICTIONARY_PENDING_DELETES_MINUS1(dict);
764 }
426 -
427 - return refcount;
765 }
766
430 -static uint32_t reference_counter_release(DICTIONARY *dict, NAME_VALUE *nv, bool can_get_write_lock) {
767 +static void item_release(DICTIONARY *dict, DICTIONARY_ITEM *item) {
768 // this function may be called without any lock on the dictionary
432 - // or even when someone else has a write lock on the dictionary
433 - // so, we cannot check for EXCLUSIVE ACCESS
769 + // or even when someone else has write lock on the dictionary
770
435 - uint32_t refcount;
436 - if(likely(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))
437 - refcount = nv->refcount--;
438 - else
439 - refcount = __atomic_fetch_sub(&nv->refcount, 1, __ATOMIC_SEQ_CST);
771 + bool is_deleted;
772 + REFCOUNT refcount;
773
441 - if(refcount == 0) {
442 - internal_error(true, "DICTIONARY: attempted to release item without references: '%s' on dictionary created by %s() (%zu@%s)", namevalue_get_name(nv), dict->creation_function, dict->creation_line, dict->creation_file);
443 - fatal("DICTIONARY: attempted to release item without references: '%s'", namevalue_get_name(nv));
774 + if(unlikely(is_dictionary_single_threaded(dict))) {
775 + is_deleted = item->flags & ITEM_FLAG_DELETED;
776 + refcount = --item->refcount;
777 }
778 + else {
779 + // get the flags before decrementing any reference counters
780 + // (the other way around may lead to use-after-free)
781 + is_deleted = item_flag_check(item, ITEM_FLAG_DELETED);
782
446 - if(refcount == 1) {
447 - if((nv->flags & NAME_VALUE_FLAG_DELETED))
448 - DICTIONARY_STATS_PENDING_DELETES_PLUS1(dict);
783 + // decrement the refcount
784 + refcount = __atomic_sub_fetch(&item->refcount, 1, __ATOMIC_SEQ_CST);
785 + }
786 +
787 + if(refcount < 0) {
788 + internal_error(
789 + true,
790 + "DICTIONARY: attempted to release item without references (refcount = %d): "
791 + "'%s' on dictionary created by %s() (%zu@%s)",
792 + refcount + 1,
793 + item_get_name(item),
794 + dict->creation_function,
795 + dict->creation_line,
796 + dict->creation_file);
797 +
798 + fatal(
799 + "DICTIONARY: attempted to release item '%s' without references (refcount = %d)",
800 + item_get_name(item),
801 + refcount + 1);
802 + }
803 +
804 + if(refcount == 0) {
805 +
806 + if(is_deleted)
807 + DICTIONARY_PENDING_DELETES_PLUS1(dict);
808
809 // referenced items counts number of unique items referenced
810 // so, we decrease it only when refcount == 0
452 - DICTIONARY_STATS_REFERENCED_ITEMS_MINUS1(dict);
811 + DICTIONARY_REFERENCED_ITEMS_MINUS1(dict);
812 }
813 +}
814
455 - if(can_get_write_lock && DICTIONARY_STATS_PENDING_DELETES_GET(dict)) {
456 - // we can garbage collect now
815 +static int item_check_and_acquire_advanced(DICTIONARY *dict, DICTIONARY_ITEM *item, bool having_index_lock) {
816 + size_t spins = 0;
817 + REFCOUNT refcount, desired;
818
458 - dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
459 - garbage_collect_pending_deletes_unsafe(dict);
460 - dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
461 - }
819 + do {
820 + spins++;
821
463 - return refcount;
464 -}
822 + refcount = DICTIONARY_ITEM_REFCOUNT_GET(dict, item);
823
466 -// ----------------------------------------------------------------------------
467 -// hash table
824 + if(refcount < 0) {
825 + // we can't use this item
826 + return ITEM_IS_CURRENTLY_BEING_DELETED;
827 + }
828
469 -static void hashtable_init_unsafe(DICTIONARY *dict) {
470 - dict->JudyHSArray = NULL;
471 -}
829 + if(item_flag_check(item, ITEM_FLAG_DELETED)) {
830 + // we can't use this item
831 + return ITEM_MARKED_FOR_DELETION;
832 + }
833
473 -static size_t hashtable_destroy_unsafe(DICTIONARY *dict) {
474 - if(unlikely(!dict->JudyHSArray)) return 0;
834 + desired = refcount + 1;
835
476 - JError_t J_Error;
477 - Word_t ret = JudyHSFreeArray(&dict->JudyHSArray, &J_Error);
478 - if(unlikely(ret == (Word_t) JERR)) {
479 - error("DICTIONARY: Cannot destroy JudyHS, JU_ERRNO_* == %u, ID == %d",
480 - JU_ERRNO(&J_Error), JU_ERRID(&J_Error));
481 - }
836 + } while(!__atomic_compare_exchange_n(&item->refcount, &refcount, desired,
837 + false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST));
838
483 - debug(D_DICTIONARY, "Dictionary: hash table freed %lu bytes", ret);
839 + // we acquired the item
840
485 - dict->JudyHSArray = NULL;
486 - return (size_t)ret;
487 -}
841 + if(is_view_dictionary(dict) && item_shared_flag_check(item, ITEM_FLAG_DELETED) && !item_flag_check(item, ITEM_FLAG_DELETED)) {
842 + // but, we can't use this item
843
489 -static inline void **hashtable_insert_unsafe(DICTIONARY *dict, const char *name, size_t name_len) {
490 - internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: inserting item from the index without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
844 + if(having_index_lock) {
845 + // delete it from the hashtable
846 + hashtable_delete_unsafe(dict, item_get_name(item), item->key_len, item);
847
492 - JError_t J_Error;
493 - Pvoid_t *Rc = JudyHSIns(&dict->JudyHSArray, (void *)name, name_len, &J_Error);
494 - if (unlikely(Rc == PJERR)) {
495 - fatal("DICTIONARY: Cannot insert entry with name '%s' to JudyHS, JU_ERRNO_* == %u, ID == %d",
496 - name, JU_ERRNO(&J_Error), JU_ERRID(&J_Error));
497 - }
848 + // mark it in our dictionary as deleted too
849 + // this is safe to be done here, because we have got
850 + // a reference counter on item
851 + item_flag_set(item, ITEM_FLAG_DELETED);
852
499 - // if *Rc == 0, new item added to the array
500 - // otherwise the existing item value is returned in *Rc
853 + DICTIONARY_ENTRIES_MINUS1(dict);
854 +
855 + // decrement the refcount we incremented above
856 + if (__atomic_sub_fetch(&item->refcount, 1, __ATOMIC_SEQ_CST) == 0) {
857 + // this is a deleted item, and we are the last one
858 + DICTIONARY_PENDING_DELETES_PLUS1(dict);
859 + }
860 +
861 + // do not touch the item below this point
862 + }
863 + else {
864 + // this is traversal / walkthrough
865 + // decrement the refcount we incremented above
866 + __atomic_sub_fetch(&item->refcount, 1, __ATOMIC_SEQ_CST);
867 + }
868 +
869 + return ITEM_MARKED_FOR_DELETION;
870 + }
871 +
872 + if(desired == 1)
873 + DICTIONARY_REFERENCED_ITEMS_PLUS1(dict);
874 +
875 + if(unlikely(spins > 2 && dict->stats))
876 + DICTIONARY_STATS_CHECK_SPINS_PLUS(dict, spins - 2);
877 +
878 + return ITEM_OK; // we can use this item
879 +}
880 +
881 +// if a dictionary item can be deleted, return true, otherwise return false
882 +// we use the private reference counter
883 +static inline bool item_is_not_referenced_and_can_be_removed(DICTIONARY *dict, DICTIONARY_ITEM *item) {
884 + // if we can set refcount to REFCOUNT_DELETING, we can delete this item
885 +
886 + REFCOUNT expected = DICTIONARY_ITEM_REFCOUNT_GET(dict, item);
887 + if(expected == 0 && __atomic_compare_exchange_n(&item->refcount, &expected, REFCOUNT_DELETING,
888 + false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)) {
889 +
890 + // we are going to delete it
891 + return true;
892 + }
893 +
894 + // we can't delete this
895 + return false;
896 +}
897 +
898 +// if a dictionary item can be freed, return true, otherwise return false
899 +// we use the shared reference counter
900 +static inline bool item_shared_release_and_check_if_it_can_be_freed(DICTIONARY *dict __maybe_unused, DICTIONARY_ITEM *item) {
901 + // if we can set refcount to REFCOUNT_DELETING, we can delete this item
902 +
903 + REFCOUNT links = __atomic_sub_fetch(&item->shared->links, 1, __ATOMIC_SEQ_CST);
904 + if(links == 0 && __atomic_compare_exchange_n(&item->shared->links, &links, REFCOUNT_DELETING,
905 + false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)) {
906 +
907 + // we can delete it
908 + return true;
909 + }
910 +
911 + // we can't delete it
912 + return false;
913 +}
914 +
915 +
916 +// ----------------------------------------------------------------------------
917 +// hash table operations
918 +
919 +static size_t hashtable_init_unsafe(DICTIONARY *dict) {
920 + dict->index.JudyHSArray = NULL;
921 + return 0;
922 +}
923 +
924 +static size_t hashtable_destroy_unsafe(DICTIONARY *dict) {
925 + if(unlikely(!dict->index.JudyHSArray)) return 0;
926 +
927 + JError_t J_Error;
928 + Word_t ret = JudyHSFreeArray(&dict->index.JudyHSArray, &J_Error);
929 + if(unlikely(ret == (Word_t) JERR)) {
930 + error("DICTIONARY: Cannot destroy JudyHS, JU_ERRNO_* == %u, ID == %d",
931 + JU_ERRNO(&J_Error), JU_ERRID(&J_Error));
932 + }
933 +
934 + debug(D_DICTIONARY, "Dictionary: hash table freed %lu bytes", ret);
935 +
936 + dict->index.JudyHSArray = NULL;
937 + return (size_t)ret;
938 +}
939 +
940 +static inline void **hashtable_insert_unsafe(DICTIONARY *dict, const char *name, size_t name_len) {
941 + JError_t J_Error;
942 + Pvoid_t *Rc = JudyHSIns(&dict->index.JudyHSArray, (void *)name, name_len, &J_Error);
943 + if (unlikely(Rc == PJERR)) {
944 + fatal("DICTIONARY: Cannot insert entry with name '%s' to JudyHS, JU_ERRNO_* == %u, ID == %d",
945 + name, JU_ERRNO(&J_Error), JU_ERRID(&J_Error));
946 + }
947 +
948 + // if *Rc == 0, new item added to the array
949 + // otherwise the existing item value is returned in *Rc
950
951 // we return a pointer to a pointer, so that the caller can
952 // put anything needed at the value of the index.
@@ -506,14 +955,12 @@ static inline void **hashtable_insert_unsafe(DICTIONARY *dict, const char *name,
955 return Rc;
956 }
957
509 -static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, size_t name_len, void *nv) {
510 - internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: deleting item from the index without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
511 -
512 - (void)nv;
513 - if(unlikely(!dict->JudyHSArray)) return 0;
958 +static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, size_t name_len, void *item) {
959 + (void)item;
960 + if(unlikely(!dict->index.JudyHSArray)) return 0;
961
962 JError_t J_Error;
516 - int ret = JudyHSDel(&dict->JudyHSArray, (void *)name, name_len, &J_Error);
963 + int ret = JudyHSDel(&dict->index.JudyHSArray, (void *)name, name_len, &J_Error);
964 if(unlikely(ret == JERR)) {
965 error("DICTIONARY: Cannot delete entry with name '%s' from JudyHS, JU_ERRNO_* == %u, ID == %d", name,
966 JU_ERRNO(&J_Error), JU_ERRID(&J_Error));
@@ -533,16 +980,16 @@ static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, si
980 }
981 }
982
536 -static inline NAME_VALUE *hashtable_get_unsafe(DICTIONARY *dict, const char *name, size_t name_len) {
537 - if(unlikely(!dict->JudyHSArray)) return NULL;
983 +static inline DICTIONARY_ITEM *hashtable_get_unsafe(DICTIONARY *dict, const char *name, size_t name_len) {
984 + if(unlikely(!dict->index.JudyHSArray)) return NULL;
985
986 DICTIONARY_STATS_SEARCHES_PLUS1(dict);
987
988 Pvoid_t *Rc;
542 - Rc = JudyHSGet(dict->JudyHSArray, (void *)name, name_len);
989 + Rc = JudyHSGet(dict->index.JudyHSArray, (void *)name, name_len);
990 if(likely(Rc)) {
991 // found in the hash table
545 - return (NAME_VALUE *)*Rc;
992 + return (DICTIONARY_ITEM *)*Rc;
993 }
994 else {
995 // not found in the hash table
@@ -550,343 +997,352 @@ static inline NAME_VALUE *hashtable_get_unsafe(DICTIONARY *dict, const char *nam
997 }
998 }
999
553 -static inline void hashtable_inserted_name_value_unsafe(DICTIONARY *dict, void *nv) {
1000 +static inline void hashtable_inserted_item_unsafe(DICTIONARY *dict, void *item) {
1001 (void)dict;
555 - (void)nv;
1002 + (void)item;
1003 +
1004 + // this is called just after an item is successfully inserted to the hashtable
1005 + // we don't need this for judy, but we may need it if we integrate more hash tables
1006 +
1007 ;
1008 }
1009
1010 // ----------------------------------------------------------------------------
1011 // linked list management
1012
562 -static inline void linkedlist_namevalue_link_unsafe(DICTIONARY *dict, NAME_VALUE *nv) {
563 - internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: adding item to the linked-list without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
564 -
565 - if (unlikely(!dict->first_item)) {
566 - // we are the only ones here
567 - nv->next = NULL;
568 - nv->prev = NULL;
569 - dict->first_item = dict->last_item = nv;
570 - return;
571 - }
1013 +static inline void item_linked_list_add(DICTIONARY *dict, DICTIONARY_ITEM *item) {
1014 + ll_recursive_lock(dict, DICTIONARY_LOCK_WRITE);
1015
573 - if(dict->flags & DICTIONARY_FLAG_ADD_IN_FRONT) {
574 - // add it at the beginning
575 - nv->prev = NULL;
576 - nv->next = dict->first_item;
577 -
578 - if (likely(nv->next)) nv->next->prev = nv;
579 - dict->first_item = nv;
580 - }
581 - else {
582 - // add it at the end
583 - nv->next = NULL;
584 - nv->prev = dict->last_item;
1016 + if(dict->options & DICT_OPTION_ADD_IN_FRONT)
1017 + DOUBLE_LINKED_LIST_PREPEND_UNSAFE(dict->items.list, item, prev, next);
1018 + else
1019 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(dict->items.list, item, prev, next);
1020
586 - if (likely(nv->prev)) nv->prev->next = nv;
587 - dict->last_item = nv;
588 - }
1021 + garbage_collect_pending_deletes(dict);
1022 + ll_recursive_unlock(dict, DICTIONARY_LOCK_WRITE);
1023 }
1024
591 -static inline void linkedlist_namevalue_unlink_unsafe(DICTIONARY *dict, NAME_VALUE *nv) {
592 - internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: removing item from the linked-list without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
1025 +static inline void item_linked_list_remove(DICTIONARY *dict, DICTIONARY_ITEM *item) {
1026 + ll_recursive_lock(dict, DICTIONARY_LOCK_WRITE);
1027 +
1028 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(dict->items.list, item, prev, next);
1029
594 - if(nv->next) nv->next->prev = nv->prev;
595 - if(nv->prev) nv->prev->next = nv->next;
596 - if(dict->first_item == nv) dict->first_item = nv->next;
597 - if(dict->last_item == nv) dict->last_item = nv->prev;
1030 + garbage_collect_pending_deletes(dict);
1031 + ll_recursive_unlock(dict, DICTIONARY_LOCK_WRITE);
1032 }
1033
1034 // ----------------------------------------------------------------------------
601 -// NAME_VALUE methods
1035 +// ITEM initialization and updates
1036
603 -static inline size_t namevalue_set_name(DICTIONARY *dict, NAME_VALUE *nv, const char *name, size_t name_len) {
604 - if(likely(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE)) {
605 - nv->caller_name = (char *)name;
606 - return 0;
1037 +static inline size_t item_set_name(DICTIONARY *dict, DICTIONARY_ITEM *item, const char *name, size_t name_len) {
1038 + if(likely(dict->options & DICT_OPTION_NAME_LINK_DONT_CLONE)) {
1039 + item->caller_name = (char *)name;
1040 + item->key_len = name_len;
1041 + }
1042 + else {
1043 + item->string_name = string_strdupz(name);
1044 + item->key_len = string_strlen(item->string_name) + 1;
1045 + item->options |= ITEM_OPTION_ALLOCATED_NAME;
1046 }
1047
609 - nv->string_name = string_strdupz(name);
610 - nv->flags |= NAME_VALUE_FLAG_NAME_IS_ALLOCATED;
611 - return name_len;
1048 + return item->key_len;
1049 }
1050
614 -static inline size_t namevalue_free_name(DICTIONARY *dict, NAME_VALUE *nv) {
615 - if(unlikely(!(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE)))
616 - string_freez(nv->string_name);
1051 +static inline size_t item_free_name(DICTIONARY *dict, DICTIONARY_ITEM *item) {
1052 + if(likely(!(dict->options & DICT_OPTION_NAME_LINK_DONT_CLONE)))
1053 + string_freez(item->string_name);
1054
618 - return 0;
1055 + return item->key_len;
1056 }
1057
621 -static inline const char *namevalue_get_name(NAME_VALUE *nv) {
622 - if(nv->flags & NAME_VALUE_FLAG_NAME_IS_ALLOCATED)
623 - return string2str(nv->string_name);
1058 +static inline const char *item_get_name(const DICTIONARY_ITEM *item) {
1059 + if(item->options & ITEM_OPTION_ALLOCATED_NAME)
1060 + return string2str(item->string_name);
1061 else
625 - return nv->caller_name;
1062 + return item->caller_name;
1063 }
1064
628 -static NAME_VALUE *namevalue_create_unsafe(DICTIONARY *dict, const char *name, size_t name_len, void *value, size_t value_len) {
629 - debug(D_DICTIONARY, "Creating name value entry for name '%s'.", name);
1065 +static DICTIONARY_ITEM *item_allocate(DICTIONARY *dict __maybe_unused, size_t *allocated_bytes, DICTIONARY_ITEM *master_item) {
1066 + DICTIONARY_ITEM *item;
1067
631 - size_t size = sizeof(NAME_VALUE);
632 - NAME_VALUE *nv = mallocz(size);
633 - size_t allocated = size;
1068 + size_t size = sizeof(DICTIONARY_ITEM);
1069 + item = callocz(1, size);
1070 + item->refcount = 1;
1071 + *allocated_bytes += size;
1072
1073 + if(master_item) {
1074 + item->shared = master_item->shared;
1075 +
1076 + if(unlikely(__atomic_add_fetch(&item->shared->links, 1, __ATOMIC_SEQ_CST) <= 1))
1077 + fatal("DICTIONARY: attempted to link to a shared item structure that had zero references");
1078 + }
1079 + else {
1080 + size = sizeof(DICTIONARY_ITEM_SHARED);
1081 + item->shared = callocz(1, size);
1082 + item->shared->links = 1;
1083 + *allocated_bytes += size;
1084 + }
1085 +
1086 +#ifdef NETDATA_INTERNAL_CHECKS
1087 + item->dict = dict;
1088 +#endif
1089 + return item;
1090 +}
1091 +
1092 +static DICTIONARY_ITEM *item_create_with_hooks(DICTIONARY *dict, const char *name, size_t name_len, void *value, size_t value_len, void *constructor_data, DICTIONARY_ITEM *master_item) {
1093 #ifdef NETDATA_INTERNAL_CHECKS
636 - nv->dict = dict;
1094 + if(unlikely(name_len > KEY_LEN_MAX))
1095 + fatal("DICTIONARY: tried to index a key of size %zu, but the maximum acceptable is %zu", name_len, (size_t)KEY_LEN_MAX);
1096 +
1097 + if(unlikely(value_len > VALUE_LEN_MAX))
1098 + fatal("DICTIONARY: tried to add an item of size %zu, but the maximum acceptable is %zu", value_len, (size_t)VALUE_LEN_MAX);
1099 #endif
1100
639 - nv->refcount = 0;
640 - nv->flags = NAME_VALUE_FLAG_NONE;
641 - nv->value_len = value_len;
1101 + size_t item_size = 0, key_size = 0, value_size = 0;
1102 +
1103 + DICTIONARY_ITEM *item = item_allocate(dict, &item_size, master_item);
1104 + key_size += item_set_name(dict, item, name, name_len);
1105
643 - allocated += namevalue_set_name(dict, nv, name, name_len);
1106 + if(unlikely(is_view_dictionary(dict))) {
1107 + // we are on a view dictionary
1108 + // do not touch the value
1109 + ;
1110
645 - if(likely(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE))
646 - nv->value = value;
1111 +#ifdef NETDATA_INTERNAL_CHECKS
1112 + if(unlikely(!master_item))
1113 + fatal("DICTIONARY: cannot add an item to a view without a master item.");
1114 +#endif
1115 + }
1116 else {
648 - if(likely(value_len)) {
649 - if (value) {
650 - // a value has been supplied
651 - // copy it
652 - nv->value = mallocz(value_len);
653 - memcpy(nv->value, value, value_len);
1117 + // we are on the master dictionary
1118 +
1119 + if(likely(dict->options & DICT_OPTION_VALUE_LINK_DONT_CLONE))
1120 + item->shared->value = value;
1121 + else {
1122 + if(likely(value_len)) {
1123 + if(value) {
1124 + // a value has been supplied
1125 + // copy it
1126 + item->shared->value = mallocz(value_len);
1127 + memcpy(item->shared->value, value, value_len);
1128 + }
1129 + else {
1130 + // no value has been supplied
1131 + // allocate a clear memory block
1132 + item->shared->value = callocz(1, value_len);
1133 + }
1134 +
1135 }
1136 else {
656 - // no value has been supplied
657 - // allocate a clear memory block
658 - nv->value = callocz(1, value_len);
1137 + // the caller wants an item without any value
1138 + item->shared->value = NULL;
1139 }
1140 }
661 - else {
662 - // the caller wants an item without any value
663 - nv->value = NULL;
664 - }
1141 + item->shared->value_len = value_len;
1142 + value_size += value_len;
1143
666 - allocated += value_len;
1144 + dictionary_execute_insert_callback(dict, item, constructor_data);
1145 }
1146
669 - DICTIONARY_STATS_ENTRIES_PLUS1(dict, allocated);
1147 + DICTIONARY_ENTRIES_PLUS1(dict);
1148 + DICTIONARY_STATS_PLUS_MEMORY(dict, key_size, item_size, value_size);
1149
671 - if(dict->ins_callback)
672 - dict->ins_callback(namevalue_get_name(nv), nv->value, dict->ins_callback_data);
673 -
674 - return nv;
1150 + return item;
1151 }
1152
677 -static void namevalue_reset_unsafe(DICTIONARY *dict, NAME_VALUE *nv, void *value, size_t value_len) {
678 - debug(D_DICTIONARY, "Dictionary entry with name '%s' found. Changing its value.", namevalue_get_name(nv));
1153 +static void item_reset_value_with_hooks(DICTIONARY *dict, DICTIONARY_ITEM *item, void *value, size_t value_len, void *constructor_data) {
1154 + if(unlikely(is_view_dictionary(dict)))
1155 + fatal("DICTIONARY: %s() should never be called on views.", __FUNCTION__ );
1156
680 - DICTIONARY_STATS_VALUE_RESETS_PLUS1(dict, nv->value_len, value_len);
1157 + debug(D_DICTIONARY, "Dictionary entry with name '%s' found. Changing its value.", item_get_name(item));
1158
682 - if(dict->del_callback)
683 - dict->del_callback(namevalue_get_name(nv), nv->value, dict->del_callback_data);
1159 + DICTIONARY_VALUE_RESETS_PLUS1(dict);
1160
685 - if(likely(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE)) {
686 - debug(D_DICTIONARY, "Dictionary: linking value to '%s'", namevalue_get_name(nv));
687 - nv->value = value;
688 - nv->value_len = value_len;
1161 + if(item->shared->value_len != value_len) {
1162 + DICTIONARY_STATS_PLUS_MEMORY(dict, 0, 0, value_len);
1163 + DICTIONARY_STATS_MINUS_MEMORY(dict, 0, 0, item->shared->value_len);
1164 + }
1165 +
1166 + dictionary_execute_delete_callback(dict, item);
1167 +
1168 + if(likely(dict->options & DICT_OPTION_VALUE_LINK_DONT_CLONE)) {
1169 + debug(D_DICTIONARY, "Dictionary: linking value to '%s'", item_get_name(item));
1170 + item->shared->value = value;
1171 + item->shared->value_len = value_len;
1172 }
1173 else {
691 - debug(D_DICTIONARY, "Dictionary: cloning value to '%s'", namevalue_get_name(nv));
1174 + debug(D_DICTIONARY, "Dictionary: cloning value to '%s'", item_get_name(item));
1175
693 - void *oldvalue = nv->value;
694 - void *newvalue = NULL;
1176 + void *old_value = item->shared->value;
1177 + void *new_value = NULL;
1178 if(value_len) {
696 - newvalue = mallocz(value_len);
697 - if(value) memcpy(newvalue, value, value_len);
698 - else memset(newvalue, 0, value_len);
1179 + new_value = mallocz(value_len);
1180 + if(value) memcpy(new_value, value, value_len);
1181 + else memset(new_value, 0, value_len);
1182 }
700 - nv->value = newvalue;
701 - nv->value_len = value_len;
1183 + item->shared->value = new_value;
1184 + item->shared->value_len = value_len;
1185
703 - debug(D_DICTIONARY, "Dictionary: freeing old value of '%s'", namevalue_get_name(nv));
704 - freez(oldvalue);
1186 + debug(D_DICTIONARY, "Dictionary: freeing old value of '%s'", item_get_name(item));
1187 + freez(old_value);
1188 }
1189
707 - if(dict->ins_callback)
708 - dict->ins_callback(namevalue_get_name(nv), nv->value, dict->ins_callback_data);
1190 + dictionary_execute_insert_callback(dict, item, constructor_data);
1191 }
1192
711 -static size_t namevalue_destroy_unsafe(DICTIONARY *dict, NAME_VALUE *nv) {
712 - debug(D_DICTIONARY, "Destroying name value entry for name '%s'.", namevalue_get_name(nv));
713 -
714 - if(dict->del_callback)
715 - dict->del_callback(namevalue_get_name(nv), nv->value, dict->del_callback_data);
1193 +static size_t item_free_with_hooks(DICTIONARY *dict, DICTIONARY_ITEM *item) {
1194 + debug(D_DICTIONARY, "Destroying name value entry for name '%s'.", item_get_name(item));
1195
717 - size_t freed = 0;
1196 + size_t item_size = 0, key_size = 0, value_size = 0;
1197
719 - if(unlikely(!(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE))) {
720 - debug(D_DICTIONARY, "Dictionary freeing value of '%s'", namevalue_get_name(nv));
721 - freez(nv->value);
722 - freed += nv->value_len;
723 - }
1198 + key_size += item->key_len;
1199 + if(unlikely(!(dict->options & DICT_OPTION_NAME_LINK_DONT_CLONE)))
1200 + item_free_name(dict, item);
1201
725 - if(unlikely(!(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE))) {
726 - debug(D_DICTIONARY, "Dictionary freeing name '%s'", namevalue_get_name(nv));
727 - freed += namevalue_free_name(dict, nv);
728 - }
1202 + if(item_shared_release_and_check_if_it_can_be_freed(dict, item)) {
1203 + dictionary_execute_delete_callback(dict, item);
1204
730 - freez(nv);
731 - freed += sizeof(NAME_VALUE);
732 -
733 - DICTIONARY_STATS_ENTRIES_MINUS_MEMORY(dict, freed);
1205 + if(unlikely(!(dict->options & DICT_OPTION_VALUE_LINK_DONT_CLONE))) {
1206 + debug(D_DICTIONARY, "Dictionary freeing value of '%s'", item_get_name(item));
1207 + freez(item->shared->value);
1208 + item->shared->value = NULL;
1209 + }
1210 + value_size += item->shared->value_len;
1211
735 - return freed;
736 -}
1212 + freez(item->shared);
1213 + item->shared = NULL;
1214 + item_size += sizeof(DICTIONARY_ITEM_SHARED);
1215 + }
1216
738 -// if a dictionary item can be deleted, return true, otherwise return false
739 -static bool name_value_can_be_deleted(DICTIONARY *dict, NAME_VALUE *nv) {
740 - if(unlikely(dict->flags & DICTIONARY_FLAG_DEFER_ALL_DELETIONS))
741 - return false;
1217 + freez(item);
1218 + item_size += sizeof(DICTIONARY_ITEM);
1219
743 - if(unlikely(DICTIONARY_NAME_VALUE_REFCOUNT_GET(nv) > 0))
744 - return false;
1220 + DICTIONARY_STATS_MINUS_MEMORY(dict, key_size, item_size, value_size);
1221
746 - return true;
1222 + // we return the memory we actually freed
1223 + return item_size + (dict->options & DICT_OPTION_VALUE_LINK_DONT_CLONE)?0:value_size;
1224 }
1225
1226 // ----------------------------------------------------------------------------
750 -// API - dictionary management
751 -#ifdef NETDATA_INTERNAL_CHECKS
752 -DICTIONARY *dictionary_create_advanced_with_trace(DICTIONARY_FLAGS flags, size_t scratchpad_size, const char *function, size_t line, const char *file) {
753 -#else
754 -DICTIONARY *dictionary_create_advanced(DICTIONARY_FLAGS flags, size_t scratchpad_size) {
755 -#endif
756 - debug(D_DICTIONARY, "Creating dictionary.");
757 -
758 - if(unlikely(flags & DICTIONARY_FLAGS_RESERVED))
759 - flags &= ~DICTIONARY_FLAGS_RESERVED;
760 -
761 - DICTIONARY *dict = callocz(1, sizeof(DICTIONARY) + scratchpad_size);
762 - size_t allocated = sizeof(DICTIONARY) + scratchpad_size;
763 -
764 - dict->scratchpad_size = scratchpad_size;
765 - dict->flags = flags;
766 - dict->first_item = dict->last_item = NULL;
767 -
768 - allocated += dictionary_lock_init(dict);
769 - allocated += reference_counter_init(dict);
770 - dict->memory = (long)allocated;
1227 +// item operations
1228
772 - hashtable_init_unsafe(dict);
1229 +static void item_shared_set_deleted(DICTIONARY *dict, DICTIONARY_ITEM *item) {
1230 + if(is_master_dictionary(dict)) {
1231 + item_shared_flag_set(item, ITEM_FLAG_DELETED);
1232
774 -#ifdef NETDATA_INTERNAL_CHECKS
775 - dict->creation_function = function;
776 - dict->creation_file = file;
777 - dict->creation_line = line;
778 -#endif
779 -
780 - return (DICTIONARY *)dict;
1233 + if(dict->hooks)
1234 + __atomic_store_n(&dict->hooks->last_master_deletion_us, now_realtime_usec(), __ATOMIC_SEQ_CST);
1235 + }
1236 }
1237
783 -void *dictionary_scratchpad(DICTIONARY *dict) {
784 - return &dict->scratchpad;
1238 +static inline void item_free_or_mark_deleted(DICTIONARY *dict, DICTIONARY_ITEM *item) {
1239 + if(item_is_not_referenced_and_can_be_removed(dict, item)) {
1240 + item_shared_set_deleted(dict, item);
1241 + item_linked_list_remove(dict, item);
1242 + item_free_with_hooks(dict, item);
1243 + }
1244 + else {
1245 + item_shared_set_deleted(dict, item);
1246 + item_flag_set(item, ITEM_FLAG_DELETED);
1247 + // after this point do not touch the item
1248 + }
1249 +
1250 + // the item is not available anymore
1251 + DICTIONARY_ENTRIES_MINUS1(dict);
1252 }
1253
787 -size_t dictionary_destroy(DICTIONARY *dict) {
788 - if(!dict) return 0;
1254 +// this is used by traversal functions to remove the current item
1255 +// if it is deleted and it has zero references. This will eliminate
1256 +// the need for the garbage collector to kick-in later.
1257 +// Most deletions happen during traversal, so this is a nice hack
1258 +// to speed up everything!
1259 +static inline void item_release_and_check_if_it_is_deleted_and_can_be_removed_under_this_lock_mode(DICTIONARY *dict, DICTIONARY_ITEM *item, char rw) {
1260 + if(rw == DICTIONARY_LOCK_WRITE) {
1261 + bool should_be_deleted = item_flag_check(item, ITEM_FLAG_DELETED);
1262
790 - NAME_VALUE *nv;
1263 + item_release(dict, item);
1264
792 - debug(D_DICTIONARY, "Destroying dictionary.");
1265 + if(should_be_deleted && item_is_not_referenced_and_can_be_removed(dict, item)) {
1266 + // this has to be before removing from the linked list,
1267 + // otherwise the garbage collector will also kick in!
1268 + DICTIONARY_PENDING_DELETES_MINUS1(dict);
1269
794 - long referenced_items = 0;
795 - size_t retries = 0;
796 - do {
797 - referenced_items = __atomic_load_n(&dict->referenced_items, __ATOMIC_SEQ_CST);
798 - if (referenced_items) {
799 - dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
800 -
801 - // there are referenced items
802 - // delete all items individually, so that only the referenced will remain
803 - NAME_VALUE *nv_next;
804 - for (nv = dict->first_item; nv; nv = nv_next) {
805 - nv_next = nv->next;
806 - size_t refcount = DICTIONARY_NAME_VALUE_REFCOUNT_GET(nv);
807 - if (!refcount && !(nv->flags & NAME_VALUE_FLAG_DELETED))
808 - dictionary_del_unsafe(dict, namevalue_get_name(nv));
809 - }
810 -
811 - internal_error(
812 - retries == 0,
813 - "DICTIONARY: waiting (try %zu) for destruction of dictionary created from %s() %zu@%s, because it has %ld referenced items in it (%ld total).",
814 - retries + 1,
815 - dict->creation_function,
816 - dict->creation_line,
817 - dict->creation_file,
818 - referenced_items,
819 - dict->entries);
820 -
821 - dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
822 - sleep_usec(10000);
1270 + item_linked_list_remove(dict, item);
1271 + item_free_with_hooks(dict, item);
1272 }
824 - } while(referenced_items > 0 && ++retries < 10);
825 -
826 - if(referenced_items) {
827 - dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
1273 + }
1274 + else {
1275 + // we can't do anything under this mode
1276 + item_release(dict, item);
1277 + }
1278 +}
1279
829 - dict->flags |= DICTIONARY_FLAG_DESTROYED;
1280 +static bool item_del(DICTIONARY *dict, const char *name, ssize_t name_len) {
1281 + if(unlikely(!name || !*name)) {
1282 internal_error(
1283 true,
832 - "DICTIONARY: delaying destruction of dictionary created from %s() %zu@%s after %zu retries, because it has %ld referenced items in it (%ld total).",
1284 + "DICTIONARY: attempted to %s() without a name on a dictionary created from %s() %zu@%s.",
1285 + __FUNCTION__,
1286 dict->creation_function,
1287 dict->creation_line,
835 - dict->creation_file,
836 - retries,
837 - referenced_items,
838 - dict->entries);
1288 + dict->creation_file);
1289 + return false;
1290 + }
1291
840 - dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
841 - return 0;
1292 + if(unlikely(is_dictionary_destroyed(dict))) {
1293 + internal_error(true, "DICTIONARY: attempted to dictionary_del() on a destroyed dictionary");
1294 + return false;
1295 }
1296
844 - dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
1297 + if(name_len == -1)
1298 + name_len = (ssize_t)strlen(name) + 1; // we need the terminating null too
1299
846 - size_t freed = 0;
847 - nv = dict->first_item;
848 - while (nv) {
849 - // cache nv->next
850 - // because we are going to free nv
851 - NAME_VALUE *nv_next = nv->next;
852 - freed += namevalue_destroy_unsafe(dict, nv);
853 - nv = nv_next;
854 - // to speed up destruction, we don't
855 - // unlink nv from the linked-list here
856 - }
1300 + debug(D_DICTIONARY, "DEL dictionary entry with name '%s'.", name);
1301
858 - dict->first_item = NULL;
859 - dict->last_item = NULL;
1302 + // Unfortunately, the JudyHSDel() does not return the value of the
1303 + // item that was deleted, so we have to find it before we delete it,
1304 + // since we need to release our structures too.
1305
861 - // destroy the dictionary
862 - freed += hashtable_destroy_unsafe(dict);
1306 + dictionary_index_lock_wrlock(dict);
1307
864 - dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
865 - freed += dictionary_lock_free(dict);
866 - freed += reference_counter_free(dict);
867 - freed += sizeof(DICTIONARY) + dict->scratchpad_size;
868 - freez(dict);
1308 + int ret;
1309 + DICTIONARY_ITEM *item = hashtable_get_unsafe(dict, name, name_len);
1310 + if(unlikely(!item)) {
1311 + dictionary_index_lock_unlock(dict);
1312 + ret = false;
1313 + }
1314 + else {
1315 + if(hashtable_delete_unsafe(dict, name, name_len, item) == 0)
1316 + error("DICTIONARY: INTERNAL ERROR: tried to delete item with name '%s' that is not in the index", name);
1317
870 - return freed;
871 -}
1318 + dictionary_index_lock_unlock(dict);
1319
873 -// ----------------------------------------------------------------------------
874 -// helpers
1320 + item_free_or_mark_deleted(dict, item);
1321 + ret = true;
1322 + }
1323
876 -static NAME_VALUE *dictionary_set_name_value_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
877 - if(unlikely(!name)) {
878 - internal_error(true, "DICTIONARY: attempted to dictionary_set() a dictionary item without a name");
1324 + return ret;
1325 +}
1326 +
1327 +static DICTIONARY_ITEM *item_add_or_reset_value_and_acquire(DICTIONARY *dict, const char *name, ssize_t name_len, void *value, size_t value_len, void *constructor_data, DICTIONARY_ITEM *master_item) {
1328 + if(unlikely(!name || !*name)) {
1329 + internal_error(
1330 + true,
1331 + "DICTIONARY: attempted to %s() without a name on a dictionary created from %s() %zu@%s.",
1332 + __FUNCTION__,
1333 + dict->creation_function,
1334 + dict->creation_line,
1335 + dict->creation_file);
1336 return NULL;
1337 }
1338
882 - if(unlikely(dict->flags & DICTIONARY_FLAG_DESTROYED)) {
1339 + if(unlikely(is_dictionary_destroyed(dict))) {
1340 internal_error(true, "DICTIONARY: attempted to dictionary_set() on a destroyed dictionary");
1341 return NULL;
1342 }
1343
887 - internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: inserting dictionary item '%s' without exclusive access to dictionary", name);
888 -
889 - size_t name_len = strlen(name) + 1; // we need the terminating null too
1344 + if(name_len == -1)
1345 + name_len = (ssize_t)strlen(name) + 1; // we need the terminating null too
1346
1347 debug(D_DICTIONARY, "SET dictionary entry with name '%s'.", name);
1348
@@ -901,858 +1357,851 @@ static NAME_VALUE *dictionary_set_name_value_unsafe(DICTIONARY *dict, const char
1357 // But the caller has the option to do this on his/her own.
1358 // So, let's do the fastest here and let the caller decide the flow of calls.
1359
904 - NAME_VALUE *nv, **pnv = (NAME_VALUE **)hashtable_insert_unsafe(dict, name, name_len);
905 - if(likely(*pnv == 0)) {
906 - // a new item added to the index
907 - nv = *pnv = namevalue_create_unsafe(dict, name, name_len, value, value_len);
908 - hashtable_inserted_name_value_unsafe(dict, nv);
909 - linkedlist_namevalue_link_unsafe(dict, nv);
910 - nv->flags |= NAME_VALUE_FLAG_NEW_OR_UPDATED;
911 - }
912 - else {
913 - // the item is already in the index
914 - // so, either we will return the old one
915 - // or overwrite the value, depending on dictionary flags
1360 + dictionary_index_lock_wrlock(dict);
1361
917 - // We should not compare the values here!
918 - // even if they are the same, we have to do the whole job
919 - // so that the callbacks will be called.
1362 + bool added_or_updated = false;
1363 + size_t spins = 0;
1364 + DICTIONARY_ITEM *item = NULL;
1365 + do {
1366 + DICTIONARY_ITEM **item_pptr = (DICTIONARY_ITEM **)hashtable_insert_unsafe(dict, name, name_len);
1367 + if (likely(*item_pptr == 0)) {
1368 + // a new item added to the index
1369
921 - nv = *pnv;
1370 + // create the dictionary item
1371 + item = *item_pptr = item_create_with_hooks(dict, name, name_len, value, value_len, constructor_data, master_item);
1372
923 - if(!(dict->flags & DICTIONARY_FLAG_DONT_OVERWRITE_VALUE)) {
924 - namevalue_reset_unsafe(dict, nv, value, value_len);
925 - nv->flags |= NAME_VALUE_FLAG_NEW_OR_UPDATED;
926 - }
1373 + // call the hashtable react
1374 + hashtable_inserted_item_unsafe(dict, item);
1375
928 - else if(dict->conflict_callback) {
929 - dict->conflict_callback(namevalue_get_name(nv), nv->value, value, dict->conflict_callback_data);
930 - nv->flags |= NAME_VALUE_FLAG_NEW_OR_UPDATED;
931 - }
1376 + // unlock the index lock, before we add it to the linked list
1377 + // DONT DO IT THE OTHER WAY AROUND - DO NOT CROSS THE LOCKS!
1378 + dictionary_index_lock_unlock(dict);
1379
1380 + item_linked_list_add(dict, item);
1381 + added_or_updated = true;
1382 + }
1383 else {
934 - // we did really nothing!
935 - // make sure this flag is not set.
936 - nv->flags &= ~NAME_VALUE_FLAG_NEW_OR_UPDATED;
1384 + if(item_check_and_acquire_advanced(dict, *item_pptr, true) != ITEM_OK) {
1385 + spins++;
1386 + continue;
1387 + }
1388 +
1389 + // the item is already in the index
1390 + // so, either we will return the old one
1391 + // or overwrite the value, depending on dictionary flags
1392 +
1393 + // We should not compare the values here!
1394 + // even if they are the same, we have to do the whole job
1395 + // so that the callbacks will be called.
1396 +
1397 + item = *item_pptr;
1398 +
1399 + if(is_view_dictionary(dict)) {
1400 + // view dictionary
1401 + // the item is already there and can be used
1402 + if(item->shared != master_item->shared)
1403 + error("DICTIONARY: changing the master item on a view is not supported. The previous item will remain. To change the key of an item in a view, delete it and add it again.");
1404 + }
1405 + else {
1406 + // master dictionary
1407 + // the user wants to reset its value
1408 +
1409 + if (!(dict->options & DICT_OPTION_DONT_OVERWRITE_VALUE)) {
1410 + item_reset_value_with_hooks(dict, item, value, value_len, constructor_data);
1411 + added_or_updated = true;
1412 + }
1413 +
1414 + else if (dictionary_execute_conflict_callback(dict, item, value, constructor_data)) {
1415 + dictionary_version_increment(dict);
1416 + added_or_updated = true;
1417 + }
1418 +
1419 + else {
1420 + // we did really nothing!
1421 + ;
1422 + }
1423 + }
1424 +
1425 + dictionary_index_lock_unlock(dict);
1426 }
938 - }
1427 + } while(!item);
1428 +
1429
940 - return nv;
1430 + if(unlikely(spins > 0 && dict->stats))
1431 + DICTIONARY_STATS_INSERT_SPINS_PLUS(dict, spins);
1432 +
1433 + if(is_master_dictionary(dict) && added_or_updated)
1434 + dictionary_execute_react_callback(dict, item, constructor_data);
1435 +
1436 + return item;
1437 }
1438
943 -static NAME_VALUE *dictionary_get_name_value_unsafe(DICTIONARY *dict, const char *name) {
944 - if(unlikely(!name)) {
945 - internal_error(true, "attempted to dictionary_get() without a name");
1439 +static DICTIONARY_ITEM *item_find_and_acquire(DICTIONARY *dict, const char *name, ssize_t name_len) {
1440 + if(unlikely(!name || !*name)) {
1441 + internal_error(
1442 + true,
1443 + "DICTIONARY: attempted to %s() without a name on a dictionary created from %s() %zu@%s.",
1444 + __FUNCTION__,
1445 + dict->creation_function,
1446 + dict->creation_line,
1447 + dict->creation_file);
1448 return NULL;
1449 }
1450
949 - if(unlikely(dict->flags & DICTIONARY_FLAG_DESTROYED)) {
1451 + if(unlikely(is_dictionary_destroyed(dict))) {
1452 internal_error(true, "DICTIONARY: attempted to dictionary_get() on a destroyed dictionary");
1453 return NULL;
1454 }
1455
954 - size_t name_len = strlen(name) + 1; // we need the terminating null too
1456 + if(name_len == -1)
1457 + name_len = (ssize_t)strlen(name) + 1; // we need the terminating null too
1458
1459 debug(D_DICTIONARY, "GET dictionary entry with name '%s'.", name);
1460
958 - NAME_VALUE *nv = hashtable_get_unsafe(dict, name, name_len);
959 - if(unlikely(!nv)) {
960 - debug(D_DICTIONARY, "Not found dictionary entry with name '%s'.", name);
961 - return NULL;
1461 + dictionary_index_lock_rdlock(dict);
1462 +
1463 + DICTIONARY_ITEM *item = hashtable_get_unsafe(dict, name, name_len);
1464 + if(unlikely(item && !item_check_and_acquire(dict, item))) {
1465 + item = NULL;
1466 + DICTIONARY_STATS_SEARCH_IGNORES_PLUS1(dict);
1467 }
1468
964 - debug(D_DICTIONARY, "Found dictionary entry with name '%s'.", name);
965 - return nv;
1469 + dictionary_index_lock_unlock(dict);
1470 +
1471 + return item;
1472 }
1473
1474 // ----------------------------------------------------------------------------
969 -// API - items management
1475 +// delayed destruction of dictionaries
1476
971 -void *dictionary_set_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
972 - NAME_VALUE *nv = dictionary_set_name_value_unsafe(dict, name, value, value_len);
1477 +static bool dictionary_free_all_resources(DICTIONARY *dict, size_t *mem, bool force) {
1478 + if(mem)
1479 + *mem = 0;
1480
974 - if(unlikely(dict->react_callback && nv && (nv->flags & NAME_VALUE_FLAG_NEW_OR_UPDATED))) {
975 - // we need to call the react callback with a reference counter on nv
976 - reference_counter_acquire(dict, nv);
977 - dict->react_callback(namevalue_get_name(nv), nv->value, dict->react_callback_data);
978 - reference_counter_release(dict, nv, false);
979 - }
980 -
981 - return nv ? nv->value : NULL;
982 -}
983 -
984 -void *dictionary_set(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
985 - dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
986 - NAME_VALUE *nv = dictionary_set_name_value_unsafe(dict, name, value, value_len);
987 -
988 - // we need to get a reference counter for the react callback
989 - // before we unlock the dictionary
990 - if(unlikely(dict->react_callback && nv && (nv->flags & NAME_VALUE_FLAG_NEW_OR_UPDATED)))
991 - reference_counter_acquire(dict, nv);
1481 + if(!force && dictionary_referenced_items(dict))
1482 + return false;
1483
993 - dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
1484 + size_t dict_size = 0, counted_items = 0, item_size = 0, index_size = 0;
1485 + (void)counted_items;
1486
995 - if(unlikely(dict->react_callback && nv && (nv->flags & NAME_VALUE_FLAG_NEW_OR_UPDATED))) {
996 - // we got the reference counter we need, above
997 - dict->react_callback(namevalue_get_name(nv), nv->value, dict->react_callback_data);
998 - reference_counter_release(dict, nv, false);
999 - }
1487 +#ifdef NETDATA_INTERNAL_CHECKS
1488 + long int entries = dict->entries;
1489 + long int referenced_items = dict->referenced_items;
1490 + long int pending_deletion_items = dict->pending_deletion_items;
1491 + const char *creation_function = dict->creation_function;
1492 + const char *creation_file = dict->creation_file;
1493 + size_t creation_line = dict->creation_line;
1494 +#endif
1495
1001 - return nv ? nv->value : NULL;
1002 -}
1496 + // destroy the index
1497 + index_size += hashtable_destroy_unsafe(dict);
1498
1004 -DICTIONARY_ITEM *dictionary_set_and_acquire_item_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
1005 - NAME_VALUE *nv = dictionary_set_name_value_unsafe(dict, name, value, value_len);
1499 + ll_recursive_lock(dict, DICTIONARY_LOCK_WRITE);
1500 + DICTIONARY_ITEM *item = dict->items.list;
1501 + while (item) {
1502 + // cache item->next
1503 + // because we are going to free item
1504 + DICTIONARY_ITEM *item_next = item->next;
1505 + item_size += item_free_with_hooks(dict, item);
1506 + item = item_next;
1507
1007 - if(unlikely(!nv))
1008 - return NULL;
1508 + DICTIONARY_ENTRIES_MINUS1(dict);
1509
1010 - reference_counter_acquire(dict, nv);
1510 + // to speed up destruction, we don't
1511 + // unlink item from the linked-list here
1512
1012 - if(unlikely(dict->react_callback && (nv->flags & NAME_VALUE_FLAG_NEW_OR_UPDATED))) {
1013 - dict->react_callback(namevalue_get_name(nv), nv->value, dict->react_callback_data);
1513 + counted_items++;
1514 }
1515 + dict->items.list = NULL;
1516 + ll_recursive_unlock(dict, DICTIONARY_LOCK_WRITE);
1517
1016 - return (DICTIONARY_ITEM *)nv;
1017 -}
1018 -
1019 -DICTIONARY_ITEM *dictionary_set_and_acquire_item(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
1020 - dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
1021 - NAME_VALUE *nv = dictionary_set_name_value_unsafe(dict, name, value, value_len);
1518 + dict_size += dictionary_locks_destroy(dict);
1519 + dict_size += reference_counter_free(dict);
1520 + dict_size += dictionary_hooks_free(dict);
1521 + dict_size += sizeof(DICTIONARY);
1522 + DICTIONARY_STATS_MINUS_MEMORY(dict, 0, sizeof(DICTIONARY), 0);
1523
1023 - // we need to get the reference counter before we unlock
1024 - if(nv) reference_counter_acquire(dict, nv);
1524 + freez(dict);
1525
1026 - dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
1526 + internal_error(
1527 + true,
1528 + "DICTIONARY: Freed dictionary created from %s() %zu@%s, having %ld (counted %zu) entries, %ld referenced, %ld pending deletion, total freed memory: %zu bytes (sizeof(dict) = %zu, sizeof(item) = %zu).",
1529 + creation_function,
1530 + creation_line,
1531 + creation_file,
1532 + entries, counted_items, referenced_items, pending_deletion_items,
1533 + dict_size, sizeof(DICTIONARY), sizeof(DICTIONARY_ITEM) + sizeof(DICTIONARY_ITEM_SHARED));
1534
1028 - if(unlikely(dict->react_callback && nv && (nv->flags & NAME_VALUE_FLAG_NEW_OR_UPDATED))) {
1029 - // we already have a reference counter, for the caller, no need for another one
1030 - dict->react_callback(namevalue_get_name(nv), nv->value, dict->react_callback_data);
1031 - }
1535 + if(mem)
1536 + *mem = dict_size + item_size + index_size;
1537
1033 - return (DICTIONARY_ITEM *)nv;
1538 + return true;
1539 }
1540
1036 -void *dictionary_get_unsafe(DICTIONARY *dict, const char *name) {
1037 - NAME_VALUE *nv = dictionary_get_name_value_unsafe(dict, name);
1038 -
1039 - if(unlikely(!nv))
1040 - return NULL;
1541 +netdata_mutex_t dictionaries_waiting_to_be_destroyed_mutex = NETDATA_MUTEX_INITIALIZER;
1542 +static DICTIONARY *dictionaries_waiting_to_be_destroyed = NULL;
1543
1042 - return nv->value;
1043 -}
1044 -
1045 -void *dictionary_get(DICTIONARY *dict, const char *name) {
1046 - dictionary_lock(dict, DICTIONARY_LOCK_READ);
1047 - void *ret = dictionary_get_unsafe(dict, name);
1048 - dictionary_unlock(dict, DICTIONARY_LOCK_READ);
1049 - return ret;
1050 -}
1544 +void dictionary_queue_for_destruction(DICTIONARY *dict) {
1545 + if(is_dictionary_destroyed(dict))
1546 + return;
1547
1052 -DICTIONARY_ITEM *dictionary_get_and_acquire_item_unsafe(DICTIONARY *dict, const char *name) {
1053 - NAME_VALUE *nv = dictionary_get_name_value_unsafe(dict, name);
1548 + DICTIONARY_STATS_DICT_DESTROY_QUEUED_PLUS1(dict);
1549 + dict_flag_set(dict, DICT_FLAG_DESTROYED);
1550
1055 - if(unlikely(!nv))
1056 - return NULL;
1551 + netdata_mutex_lock(&dictionaries_waiting_to_be_destroyed_mutex);
1552
1058 - reference_counter_acquire(dict, nv);
1059 - return (DICTIONARY_ITEM *)nv;
1060 -}
1553 + dict->next = dictionaries_waiting_to_be_destroyed;
1554 + dictionaries_waiting_to_be_destroyed = dict;
1555
1062 -DICTIONARY_ITEM *dictionary_get_and_acquire_item(DICTIONARY *dict, const char *name) {
1063 - dictionary_lock(dict, DICTIONARY_LOCK_READ);
1064 - void *ret = dictionary_get_and_acquire_item_unsafe(dict, name);
1065 - dictionary_unlock(dict, DICTIONARY_LOCK_READ);
1066 - return ret;
1556 + netdata_mutex_unlock(&dictionaries_waiting_to_be_destroyed_mutex);
1557 }
1558
1069 -DICTIONARY_ITEM *dictionary_acquired_item_dup(DICTIONARY_ITEM *item) {
1070 - if(unlikely(!item)) return NULL;
1071 - reference_counter_increase((NAME_VALUE *)item);
1072 - return item;
1073 -}
1074 -
1075 -const char *dictionary_acquired_item_name(DICTIONARY_ITEM *item) {
1076 - if(unlikely(!item)) return NULL;
1077 - return namevalue_get_name((NAME_VALUE *)item);
1078 -}
1559 +void cleanup_destroyed_dictionaries(void) {
1560 + if(!dictionaries_waiting_to_be_destroyed)
1561 + return;
1562
1080 -void *dictionary_acquired_item_value(DICTIONARY_ITEM *item) {
1081 - if(unlikely(!item)) return NULL;
1082 - return ((NAME_VALUE *)item)->value;
1083 -}
1563 + netdata_mutex_lock(&dictionaries_waiting_to_be_destroyed_mutex);
1564
1085 -void dictionary_acquired_item_release_unsafe(DICTIONARY *dict, DICTIONARY_ITEM *item) {
1086 - if(unlikely(!item)) return;
1565 + DICTIONARY *dict, *last = NULL, *next = NULL;
1566 + for(dict = dictionaries_waiting_to_be_destroyed; dict ; dict = next) {
1567 + next = dict->next;
1568
1569 #ifdef NETDATA_INTERNAL_CHECKS
1089 - if(((NAME_VALUE *)item)->dict != dict)
1090 - fatal("DICTIONARY: %s(): name_value item with name '%s' does not belong to this dictionary", __FUNCTION__, namevalue_get_name((NAME_VALUE *)item));
1570 + size_t line = dict->creation_line;
1571 + const char *file = dict->creation_file;
1572 + const char *function = dict->creation_function;
1573 #endif
1574
1093 - reference_counter_release(dict, (NAME_VALUE *)item, false);
1094 -}
1095 -
1096 -void dictionary_acquired_item_release(DICTIONARY *dict, DICTIONARY_ITEM *item) {
1097 - if(unlikely(!item)) return;
1575 + DICTIONARY_STATS_DICT_DESTROY_QUEUED_MINUS1(dict);
1576 + if(dictionary_free_all_resources(dict, NULL, false)) {
1577
1099 -#ifdef NETDATA_INTERNAL_CHECKS
1100 - if(((NAME_VALUE *)item)->dict != dict)
1101 - fatal("DICTIONARY: %s(): name_value item with name '%s' does not belong to this dictionary", __FUNCTION__, namevalue_get_name((NAME_VALUE *)item));
1102 -#endif
1103 -
1104 - // no need to get a lock here
1105 - // we pass the last parameter to reference_counter_release() as true
1106 - // so that the release may get a write-lock if required to clean up
1578 + internal_error(
1579 + true,
1580 + "DICTIONARY: freed dictionary with delayed destruction, created from %s() %zu@%s.",
1581 + function, line, file);
1582
1108 - reference_counter_release(dict, (NAME_VALUE *)item, true);
1583 + if(last) last->next = next;
1584 + else dictionaries_waiting_to_be_destroyed = next;
1585 + }
1586 + else {
1587 + DICTIONARY_STATS_DICT_DESTROY_QUEUED_PLUS1(dict);
1588 + last = dict;
1589 + }
1590 + }
1591
1110 - if(unlikely(dict->flags & DICTIONARY_FLAG_DESTROYED))
1111 - dictionary_destroy(dict);
1592 + netdata_mutex_unlock(&dictionaries_waiting_to_be_destroyed_mutex);
1593 }
1594
1114 -int dictionary_del_unsafe(DICTIONARY *dict, const char *name) {
1115 - if(unlikely(dict->flags & DICTIONARY_FLAG_DESTROYED)) {
1116 - internal_error(true, "DICTIONARY: attempted to dictionary_del() on a destroyed dictionary");
1117 - return -1;
1118 - }
1595 +// ----------------------------------------------------------------------------
1596 +// API internal checks
1597
1120 - if(unlikely(!name || !*name)) {
1121 - internal_error(true, "DICTIONARY: attempted to dictionary_del() without a name");
1122 - return -1;
1598 +#ifdef NETDATA_INTERNAL_CHECKS
1599 +#define api_internal_check(dict, item, allow_null_dict, allow_null_item) api_internal_check_with_trace(dict, item, __FUNCTION__, allow_null_dict, allow_null_item)
1600 +static inline void api_internal_check_with_trace(DICTIONARY *dict, DICTIONARY_ITEM *item, const char *function, bool allow_null_dict, bool allow_null_item) {
1601 + if(!allow_null_dict && !dict) {
1602 + internal_error(
1603 + item,
1604 + "DICTIONARY: attempted to %s() with a NULL dictionary, passing an item created from %s() %zu@%s.",
1605 + function,
1606 + item->dict->creation_function,
1607 + item->dict->creation_line,
1608 + item->dict->creation_file);
1609 + fatal("DICTIONARY: attempted to %s() but item is NULL", function);
1610 }
1611
1125 - internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: INTERNAL ERROR: deleting dictionary item '%s' without exclusive access to dictionary", name);
1126 -
1127 - size_t name_len = strlen(name) + 1; // we need the terminating null too
1128 -
1129 - debug(D_DICTIONARY, "DEL dictionary entry with name '%s'.", name);
1130 -
1131 - // Unfortunately, the JudyHSDel() does not return the value of the
1132 - // item that was deleted, so we have to find it before we delete it,
1133 - // since we need to release our structures too.
1134 -
1135 - int ret;
1136 - NAME_VALUE *nv = hashtable_get_unsafe(dict, name, name_len);
1137 - if(unlikely(!nv)) {
1138 - debug(D_DICTIONARY, "Not found dictionary entry with name '%s'.", name);
1139 - ret = -1;
1612 + if(!allow_null_item && !item) {
1613 + internal_error(
1614 + true,
1615 + "DICTIONARY: attempted to %s() without an item on a dictionary created from %s() %zu@%s.",
1616 + function,
1617 + dict?dict->creation_function:"unknown",
1618 + dict?dict->creation_line:0,
1619 + dict?dict->creation_file:"unknown");
1620 + fatal("DICTIONARY: attempted to %s() but item is NULL", function);
1621 }
1141 - else {
1142 - debug(D_DICTIONARY, "Found dictionary entry with name '%s'.", name);
1622
1144 - if(hashtable_delete_unsafe(dict, name, name_len, nv) == 0)
1145 - error("DICTIONARY: INTERNAL ERROR: tried to delete item with name '%s' that is not in the index", name);
1623 + if(dict && item && dict != item->dict) {
1624 + internal_error(
1625 + true,
1626 + "DICTIONARY: attempted to %s() an item on a dictionary created from %s() %zu@%s, but the item belongs to the dictionary created from %s() %zu@%s.",
1627 + function,
1628 + dict->creation_function,
1629 + dict->creation_line,
1630 + dict->creation_file,
1631 + item->dict->creation_function,
1632 + item->dict->creation_line,
1633 + item->dict->creation_file
1634 + );
1635 + fatal("DICTIONARY: %s(): item does not belong to this dictionary.", function);
1636 + }
1637
1147 - if(name_value_can_be_deleted(dict, nv)) {
1148 - linkedlist_namevalue_unlink_unsafe(dict, nv);
1149 - namevalue_destroy_unsafe(dict, nv);
1638 + if(item) {
1639 + REFCOUNT refcount = DICTIONARY_ITEM_REFCOUNT_GET(dict, item);
1640 + if (unlikely(refcount <= 0)) {
1641 + internal_error(
1642 + true,
1643 + "DICTIONARY: attempted to %s() of an item with reference counter = %d on a dictionary created from %s() %zu@%s",
1644 + function,
1645 + refcount,
1646 + item->dict->creation_function,
1647 + item->dict->creation_line,
1648 + item->dict->creation_file);
1649 + fatal("DICTIONARY: attempted to %s but item is having refcount = %d", function, refcount);
1650 }
1151 - else
1152 - nv->flags |= NAME_VALUE_FLAG_DELETED;
1153 -
1154 - ret = 0;
1155 -
1156 - DICTIONARY_STATS_ENTRIES_MINUS1(dict);
1157 -
1651 }
1159 - return ret;
1160 -}
1161 -
1162 -int dictionary_del(DICTIONARY *dict, const char *name) {
1163 - dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
1164 - int ret = dictionary_del_unsafe(dict, name);
1165 - dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
1166 - return ret;
1652 }
1653 +#else
1654 +#define api_internal_check(dict, item, allow_null_dict, allow_null_item) debug_dummy()
1655 +#endif
1656
1169 -// ----------------------------------------------------------------------------
1170 -// traversal with loop
1171 -
1172 -void *dictionary_foreach_start_rw(DICTFE *dfe, DICTIONARY *dict, char rw) {
1173 - if(unlikely(!dfe || !dict)) return NULL;
1174 -
1175 - if(unlikely(dict->flags & DICTIONARY_FLAG_DESTROYED)) {
1176 - internal_error(true, "DICTIONARY: attempted to dictionary_foreach_start_rw() on a destroyed dictionary");
1177 - dfe->last_item = NULL;
1178 - dfe->name = NULL;
1179 - dfe->value = NULL;
1180 - return NULL;
1657 +#define api_is_name_good(dict, name, name_len) api_is_name_good_with_trace(dict, name, name_len, __FUNCTION__)
1658 +static bool api_is_name_good_with_trace(DICTIONARY *dict __maybe_unused, const char *name, ssize_t name_len __maybe_unused, const char *function __maybe_unused) {
1659 + if(unlikely(!name)) {
1660 + internal_error(
1661 + true,
1662 + "DICTIONARY: attempted to %s() with name = NULL on a dictionary created from %s() %zu@%s.",
1663 + function,
1664 + dict?dict->creation_function:"unknown",
1665 + dict?dict->creation_line:0,
1666 + dict?dict->creation_file:"unknown");
1667 + return false;
1668 }
1669
1183 - dfe->dict = dict;
1184 - dfe->rw = rw;
1185 - dfe->started_ut = now_realtime_usec();
1186 -
1187 - dictionary_lock(dict, dfe->rw);
1188 -
1189 - DICTIONARY_STATS_WALKTHROUGHS_PLUS1(dict);
1190 -
1191 - // get the first item from the list
1192 - NAME_VALUE *nv = dict->first_item;
1193 -
1194 - // skip all the deleted items
1195 - while(nv && (nv->flags & NAME_VALUE_FLAG_DELETED))
1196 - nv = nv->next;
1197 -
1198 - if(likely(nv)) {
1199 - dfe->last_item = nv;
1200 - dfe->name = (char *)namevalue_get_name(nv);
1201 - dfe->value = nv->value;
1202 - reference_counter_acquire(dict, nv);
1203 - }
1204 - else {
1205 - dfe->last_item = NULL;
1206 - dfe->name = NULL;
1207 - dfe->value = NULL;
1670 + if(unlikely(!*name)) {
1671 + internal_error(
1672 + true,
1673 + "DICTIONARY: attempted to %s() with empty name on a dictionary created from %s() %zu@%s.",
1674 + function,
1675 + dict?dict->creation_function:"unknown",
1676 + dict?dict->creation_line:0,
1677 + dict?dict->creation_file:"unknown");
1678 + return false;
1679 }
1680
1210 - if(unlikely(dfe->rw == DICTIONARY_LOCK_REENTRANT))
1211 - dictionary_unlock(dfe->dict, dfe->rw);
1681 + internal_error(
1682 + name_len > 0 && name_len != (ssize_t)(strlen(name) + 1),
1683 + "DICTIONARY: attempted to %s() with a name of '%s', having length of %zu (incl. '\\0'), but the supplied name_len = %ld, on a dictionary created from %s() %zu@%s.",
1684 + function,
1685 + name,
1686 + strlen(name) + 1,
1687 + name_len,
1688 + dict?dict->creation_function:"unknown",
1689 + dict?dict->creation_line:0,
1690 + dict?dict->creation_file:"unknown");
1691 +
1692 + internal_error(
1693 + name_len <= 0 && name_len != -1,
1694 + "DICTIONARY: attempted to %s() with a name of '%s', having length of %zu (incl. '\\0'), but the supplied name_len = %ld, on a dictionary created from %s() %zu@%s.",
1695 + function,
1696 + name,
1697 + strlen(name) + 1,
1698 + name_len,
1699 + dict?dict->creation_function:"unknown",
1700 + dict?dict->creation_line:0,
1701 + dict?dict->creation_file:"unknown");
1702
1213 - return dfe->value;
1703 + return true;
1704 }
1705
1216 -void *dictionary_foreach_next(DICTFE *dfe) {
1217 - if(unlikely(!dfe || !dfe->dict)) return NULL;
1706 +// ----------------------------------------------------------------------------
1707 +// API - dictionary management
1708
1219 - if(unlikely(dfe->dict->flags & DICTIONARY_FLAG_DESTROYED)) {
1220 - internal_error(true, "DICTIONARY: attempted to dictionary_foreach_next() on a destroyed dictionary");
1221 - dfe->last_item = NULL;
1222 - dfe->name = NULL;
1223 - dfe->value = NULL;
1224 - return NULL;
1225 - }
1709 +static DICTIONARY *dictionary_create_internal(DICT_OPTIONS options, struct dictionary_stats *stats) {
1710 + cleanup_destroyed_dictionaries();
1711
1227 - if(unlikely(dfe->rw == DICTIONARY_LOCK_REENTRANT))
1228 - dictionary_lock(dfe->dict, dfe->rw);
1712 + DICTIONARY *dict = callocz(1, sizeof(DICTIONARY));
1713 + dict->options = options;
1714 + dict->stats = stats;
1715
1230 - // the item we just did
1231 - NAME_VALUE *nv = (NAME_VALUE *)dfe->last_item;
1716 + size_t dict_size = 0;
1717 + dict_size += sizeof(DICTIONARY);
1718 + dict_size += dictionary_locks_init(dict);
1719 + dict_size += reference_counter_init(dict);
1720 + dict_size += hashtable_init_unsafe(dict);
1721
1233 - // get the next item from the list
1234 - NAME_VALUE *nv_next = (nv) ? nv->next : NULL;
1722 + DICTIONARY_STATS_PLUS_MEMORY(dict, 0, dict_size, 0);
1723
1236 - // skip all the deleted items
1237 - while(nv_next && (nv_next->flags & NAME_VALUE_FLAG_DELETED))
1238 - nv_next = nv_next->next;
1724 + return dict;
1725 +}
1726
1240 - // release the old, so that it can possibly be deleted
1241 - if(likely(nv))
1242 - reference_counter_release(dfe->dict, nv, false);
1727 +#ifdef NETDATA_INTERNAL_CHECKS
1728 +DICTIONARY *dictionary_create_advanced_with_trace(DICT_OPTIONS options, struct dictionary_stats *stats, const char *function, size_t line, const char *file) {
1729 +#else
1730 +DICTIONARY *dictionary_create_advanced(DICT_OPTIONS options, struct dictionary_stats *stats) {
1731 +#endif
1732
1244 - if(likely(nv = nv_next)) {
1245 - dfe->last_item = nv;
1246 - dfe->name = (char *)namevalue_get_name(nv);
1247 - dfe->value = nv->value;
1248 - reference_counter_acquire(dfe->dict, nv);
1249 - }
1250 - else {
1251 - dfe->last_item = NULL;
1252 - dfe->name = NULL;
1253 - dfe->value = NULL;
1254 - }
1733 + DICTIONARY *dict = dictionary_create_internal(options, stats?stats:&dictionary_stats_category_other);
1734
1256 - if(unlikely(dfe->rw == DICTIONARY_LOCK_REENTRANT))
1257 - dictionary_unlock(dfe->dict, dfe->rw);
1735 +#ifdef NETDATA_INTERNAL_CHECKS
1736 + dict->creation_function = function;
1737 + dict->creation_file = file;
1738 + dict->creation_line = line;
1739 +#endif
1740
1259 - return dfe->value;
1741 + DICTIONARY_STATS_DICT_CREATIONS_PLUS1(dict);
1742 + return dict;
1743 }
1744
1262 -usec_t dictionary_foreach_done(DICTFE *dfe) {
1263 - if(unlikely(!dfe || !dfe->dict)) return 0;
1264 -
1265 - if(unlikely(dfe->dict->flags & DICTIONARY_FLAG_DESTROYED)) {
1266 - internal_error(true, "DICTIONARY: attempted to dictionary_foreach_next() on a destroyed dictionary");
1267 - return 0;
1268 - }
1745 +#ifdef NETDATA_INTERNAL_CHECKS
1746 +DICTIONARY *dictionary_create_view_with_trace(DICTIONARY *master, const char *function, size_t line, const char *file) {
1747 +#else
1748 +DICTIONARY *dictionary_create_view(DICTIONARY *master) {
1749 +#endif
1750
1270 - // the item we just did
1271 - NAME_VALUE *nv = (NAME_VALUE *)dfe->last_item;
1751 + DICTIONARY *dict = dictionary_create_internal(master->options, master->stats);
1752 + dict->master = master;
1753
1273 - // release it, so that it can possibly be deleted
1274 - if(likely(nv))
1275 - reference_counter_release(dfe->dict, nv, false);
1754 + dictionary_hooks_allocate(master);
1755
1277 - if(likely(dfe->rw != DICTIONARY_LOCK_REENTRANT))
1278 - dictionary_unlock(dfe->dict, dfe->rw);
1756 + if(unlikely(__atomic_load_n(&master->hooks->links, __ATOMIC_SEQ_CST)) < 1)
1757 + fatal("DICTIONARY: attempted to create a view that has %d links", master->hooks->links);
1758
1280 - dfe->dict = NULL;
1281 - dfe->last_item = NULL;
1282 - dfe->name = NULL;
1283 - dfe->value = NULL;
1759 + dict->hooks = master->hooks;
1760 + __atomic_add_fetch(&master->hooks->links, 1, __ATOMIC_SEQ_CST);
1761
1285 - usec_t usec = now_realtime_usec() - dfe->started_ut;
1286 - dfe->started_ut = 0;
1762 +#ifdef NETDATA_INTERNAL_CHECKS
1763 + dict->creation_function = function;
1764 + dict->creation_file = file;
1765 + dict->creation_line = line;
1766 +#endif
1767
1288 - return usec;
1768 + DICTIONARY_STATS_DICT_CREATIONS_PLUS1(dict);
1769 + return dict;
1770 }
1771
1291 -// ----------------------------------------------------------------------------
1292 -// API - walk through the dictionary
1293 -// the dictionary is locked for reading while this happens
1294 -// do not use other dictionary calls while walking the dictionary - deadlock!
1772 +void dictionary_flush(DICTIONARY *dict) {
1773 + if(unlikely(!dict))
1774 + return;
1775
1296 -int dictionary_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const char *name, void *entry, void *data), void *data) {
1297 - if(unlikely(!dict)) return 0;
1776 + // delete the index
1777 + dictionary_index_lock_wrlock(dict);
1778 + hashtable_destroy_unsafe(dict);
1779 + dictionary_index_lock_unlock(dict);
1780
1299 - if(unlikely(dict->flags & DICTIONARY_FLAG_DESTROYED)) {
1300 - internal_error(true, "DICTIONARY: attempted to dictionary_walkthrough_rw() on a destroyed dictionary");
1301 - return 0;
1302 - }
1781 + // delete all items
1782 + ll_recursive_lock(dict, DICTIONARY_LOCK_WRITE); // get write lock here, to speed it up (it is recursive)
1783 + DICTIONARY_ITEM *item, *item_next;
1784 + for (item = dict->items.list; item; item = item_next) {
1785 + item_next = item->next;
1786
1304 - dictionary_lock(dict, rw);
1787 + if(!item_flag_check(item, ITEM_FLAG_DELETED))
1788 + item_free_or_mark_deleted(dict, item);
1789 + }
1790 + ll_recursive_unlock(dict, DICTIONARY_LOCK_WRITE);
1791
1306 - DICTIONARY_STATS_WALKTHROUGHS_PLUS1(dict);
1792 + DICTIONARY_STATS_DICT_FLUSHES_PLUS1(dict);
1793 +}
1794
1308 - // written in such a way, that the callback can delete the active element
1795 +size_t dictionary_destroy(DICTIONARY *dict) {
1796 + cleanup_destroyed_dictionaries();
1797
1310 - int ret = 0;
1311 - NAME_VALUE *nv = dict->first_item, *nv_next;
1312 - while(nv) {
1798 + if(!dict) return 0;
1799
1314 - // skip the deleted items
1315 - if(unlikely(nv->flags & NAME_VALUE_FLAG_DELETED)) {
1316 - nv = nv->next;
1317 - continue;
1318 - }
1800 + DICTIONARY_STATS_DICT_DESTRUCTIONS_PLUS1(dict);
1801
1320 - // get a reference counter, so that our item will not be deleted
1321 - // while we are using it
1322 - reference_counter_acquire(dict, nv);
1802 + size_t referenced_items = dictionary_referenced_items(dict);
1803 + if(referenced_items) {
1804 + dictionary_flush(dict);
1805 + dictionary_queue_for_destruction(dict);
1806
1324 - if(unlikely(rw == DICTIONARY_LOCK_REENTRANT))
1325 - dictionary_unlock(dict, rw);
1807 + internal_error(
1808 + true,
1809 + "DICTIONARY: delaying destruction of dictionary created from %s() %zu@%s, because it has %ld referenced items in it (%ld total).",
1810 + dict->creation_function,
1811 + dict->creation_line,
1812 + dict->creation_file,
1813 + dict->referenced_items,
1814 + dict->entries);
1815
1327 - int r = callback(namevalue_get_name(nv), nv->value, data);
1816 + return 0;
1817 + }
1818
1329 - if(unlikely(rw == DICTIONARY_LOCK_REENTRANT))
1330 - dictionary_lock(dict, rw);
1819 + size_t freed;
1820 + dictionary_free_all_resources(dict, &freed, true);
1821
1332 - // since we have a reference counter, this item cannot be deleted
1333 - // until we release the reference counter, so the pointers are there
1334 - nv_next = nv->next;
1335 - reference_counter_release(dict, nv, false);
1822 + return freed;
1823 +}
1824
1337 - if(unlikely(r < 0)) {
1338 - ret = r;
1339 - break;
1340 - }
1825 +// ----------------------------------------------------------------------------
1826 +// SET an item to the dictionary
1827
1342 - ret += r;
1828 +DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_set_and_acquire_item_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, void *value, size_t value_len, void *constructor_data) {
1829 + if(unlikely(!api_is_name_good(dict, name, name_len)))
1830 + return NULL;
1831
1344 - nv = nv_next;
1345 - }
1832 + api_internal_check(dict, NULL, false, true);
1833
1347 - dictionary_unlock(dict, rw);
1834 + if(unlikely(is_view_dictionary(dict)))
1835 + fatal("DICTIONARY: this dictionary is a view, you cannot add items other than the ones from the master dictionary.");
1836
1349 - return ret;
1837 + DICTIONARY_ITEM *item = item_add_or_reset_value_and_acquire(dict, name, name_len, value, value_len, constructor_data, NULL);
1838 + api_internal_check(dict, item, false, false);
1839 + return item;
1840 }
1841
1352 -// ----------------------------------------------------------------------------
1353 -// sorted walkthrough
1842 +void *dictionary_set_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, void *value, size_t value_len, void *constructor_data) {
1843 + DICTIONARY_ITEM *item = dictionary_set_and_acquire_item_advanced(dict, name, name_len, value, value_len, constructor_data);
1844
1355 -static int dictionary_sort_compar(const void *nv1, const void *nv2) {
1356 - return strcmp(namevalue_get_name((*(NAME_VALUE **)nv1)), namevalue_get_name((*(NAME_VALUE **)nv2)));
1357 -}
1845 + if(likely(item)) {
1846 + void *v = item->shared->value;
1847 + item_release(dict, item);
1848 + return v;
1849 + }
1850
1359 -int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const char *name, void *entry, void *data), void *data) {
1360 - if(unlikely(!dict || !dict->entries)) return 0;
1851 + return NULL;
1852 +}
1853
1362 - if(unlikely(dict->flags & DICTIONARY_FLAG_DESTROYED)) {
1363 - internal_error(true, "DICTIONARY: attempted to dictionary_sorted_walkthrough_rw() on a destroyed dictionary");
1364 - return 0;
1365 - }
1854 +DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_view_set_and_acquire_item_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, DICTIONARY_ITEM *master_item) {
1855 + if(unlikely(!api_is_name_good(dict, name, name_len)))
1856 + return NULL;
1857
1367 - dictionary_lock(dict, rw);
1368 - dictionary_defer_all_deletions_unsafe(dict, rw);
1858 + api_internal_check(dict, NULL, false, true);
1859
1370 - DICTIONARY_STATS_WALKTHROUGHS_PLUS1(dict);
1860 + if(unlikely(is_master_dictionary(dict)))
1861 + fatal("DICTIONARY: this dictionary is a master, you cannot add items from other dictionaries.");
1862
1372 - size_t count = dict->entries;
1373 - NAME_VALUE **array = mallocz(sizeof(NAME_VALUE *) * count);
1863 + dictionary_acquired_item_dup(dict->master, master_item);
1864 + DICTIONARY_ITEM *item = item_add_or_reset_value_and_acquire(dict, name, name_len, NULL, 0, NULL, master_item);
1865 + dictionary_acquired_item_release(dict->master, master_item);
1866
1375 - size_t i;
1376 - NAME_VALUE *nv;
1377 - for(nv = dict->first_item, i = 0; nv && i < count ;nv = nv->next) {
1378 - if(likely(!(nv->flags & NAME_VALUE_FLAG_DELETED)))
1379 - array[i++] = nv;
1380 - }
1867 + api_internal_check(dict, item, false, false);
1868 + return item;
1869 +}
1870
1382 - internal_error(nv != NULL, "DICTIONARY: during sorting expected to have %zu items in dictionary, but there are more. Sorted results may be incomplete. Dictionary fails to maintain an accurate number of the number of entries it has.", count);
1871 +void *dictionary_view_set_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, DICTIONARY_ITEM *master_item) {
1872 + DICTIONARY_ITEM *item = dictionary_view_set_and_acquire_item_advanced(dict, name, name_len, master_item);
1873
1384 - if(unlikely(i != count)) {
1385 - internal_error(true, "DICTIONARY: during sorting expected to have %zu items in dictionary, but there are %zu. Sorted results may be incomplete. Dictionary fails to maintain an accurate number of the number of entries it has.", count, i);
1386 - count = i;
1874 + if(likely(item)) {
1875 + void *v = item->shared->value;
1876 + item_release(dict, item);
1877 + return v;
1878 }
1879
1389 - qsort(array, count, sizeof(NAME_VALUE *), dictionary_sort_compar);
1880 + return NULL;
1881 +}
1882
1391 - int ret = 0;
1392 - for(i = 0; i < count ;i++) {
1393 - nv = array[i];
1394 - if(likely(!(nv->flags & NAME_VALUE_FLAG_DELETED))) {
1395 - reference_counter_acquire(dict, nv);
1883 +// ----------------------------------------------------------------------------
1884 +// GET an item from the dictionary
1885
1397 - if(unlikely(rw == DICTIONARY_LOCK_REENTRANT))
1398 - dictionary_unlock(dict, rw);
1886 +DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_get_and_acquire_item_advanced(DICTIONARY *dict, const char *name, ssize_t name_len) {
1887 + if(unlikely(!api_is_name_good(dict, name, name_len)))
1888 + return NULL;
1889
1400 - int r = callback(namevalue_get_name(nv), nv->value, data);
1890 + api_internal_check(dict, NULL, false, true);
1891 + DICTIONARY_ITEM *item = item_find_and_acquire(dict, name, name_len);
1892 + api_internal_check(dict, item, false, true);
1893 + return item;
1894 +}
1895
1402 - if(unlikely(rw == DICTIONARY_LOCK_REENTRANT))
1403 - dictionary_lock(dict, rw);
1896 +void *dictionary_get_advanced(DICTIONARY *dict, const char *name, ssize_t name_len) {
1897 + DICTIONARY_ITEM *item = dictionary_get_and_acquire_item_advanced(dict, name, name_len);
1898
1405 - reference_counter_release(dict, nv, false);
1406 - if (r < 0) {
1407 - ret = r;
1408 - break;
1409 - }
1410 - ret += r;
1411 - }
1899 + if(likely(item)) {
1900 + void *v = item->shared->value;
1901 + item_release(dict, item);
1902 + return v;
1903 }
1904
1414 - dictionary_restore_all_deletions_unsafe(dict, rw);
1415 - dictionary_unlock(dict, rw);
1416 - freez(array);
1417 -
1418 - return ret;
1905 + return NULL;
1906 }
1907
1908 // ----------------------------------------------------------------------------
1422 -// STRING implementation - dedup all STRINGs
1909 +// DUP/REL an item (increase/decrease its reference counter)
1910
1424 -struct netdata_string {
1425 - uint32_t length; // the string length including the terminating '\0'
1911 +DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_acquired_item_dup(DICTIONARY *dict, DICT_ITEM_CONST DICTIONARY_ITEM *item) {
1912 + // we allow the item to be NULL here
1913 + api_internal_check(dict, item, false, true);
1914
1427 - int32_t refcount; // how many times this string is used
1428 - // We use a signed number to be able to detect duplicate frees of a string.
1429 - // If at any point this goes below zero, we have a duplicate free.
1915 + if(likely(item)) {
1916 + item_acquire(dict, item);
1917 + api_internal_check(dict, item, false, false);
1918 + }
1919
1431 - const char str[]; // the string itself, is appended to this structure
1432 -};
1920 + return item;
1921 +}
1922
1434 -static struct string_hashtable {
1435 - Pvoid_t JudyHSArray; // the Judy array - hashtable
1436 - netdata_rwlock_t rwlock; // the R/W lock to protect the Judy array
1923 +void dictionary_acquired_item_release(DICTIONARY *dict, DICT_ITEM_CONST DICTIONARY_ITEM *item) {
1924 + // we allow the item to be NULL here
1925 + api_internal_check(dict, item, false, true);
1926
1438 - long int entries; // the number of entries in the index
1439 - long int active_references; // the number of active references alive
1440 - long int memory; // the memory used, without the JudyHS index
1927 + // no need to get a lock here
1928 + // we pass the last parameter to reference_counter_release() as true
1929 + // so that the release may get a write-lock if required to clean up
1930
1442 - size_t inserts; // the number of successful inserts to the index
1443 - size_t deletes; // the number of successful deleted from the index
1444 - size_t searches; // the number of successful searches in the index
1445 - size_t duplications; // when a string is referenced
1446 - size_t releases; // when a string is unreferenced
1931 + if(likely(item))
1932 + item_release(dict, item);
1933 +}
1934
1448 -#ifdef NETDATA_INTERNAL_CHECKS
1449 - // internal statistics
1450 - size_t found_deleted_on_search;
1451 - size_t found_available_on_search;
1452 - size_t found_deleted_on_insert;
1453 - size_t found_available_on_insert;
1454 - size_t spins;
1455 -#endif
1935 +// ----------------------------------------------------------------------------
1936 +// get the name/value of an item
1937
1457 -} string_base = {
1458 - .JudyHSArray = NULL,
1459 - .rwlock = NETDATA_RWLOCK_INITIALIZER,
1460 -};
1938 +const char *dictionary_acquired_item_name(DICT_ITEM_CONST DICTIONARY_ITEM *item) {
1939 + api_internal_check(NULL, item, true, false);
1940 + return item_get_name(item);
1941 +}
1942
1462 -#ifdef NETDATA_INTERNAL_CHECKS
1463 -#define string_internal_stats_add(var, val) __atomic_add_fetch(&string_base.var, val, __ATOMIC_RELAXED)
1464 -#else
1465 -#define string_internal_stats_add(var, val) do {;} while(0)
1466 -#endif
1943 +void *dictionary_acquired_item_value(DICT_ITEM_CONST DICTIONARY_ITEM *item) {
1944 + // we allow the item to be NULL here
1945 + api_internal_check(NULL, item, true, true);
1946
1468 -#define string_stats_atomic_increment(var) __atomic_add_fetch(&string_base.var, 1, __ATOMIC_RELAXED)
1469 -#define string_stats_atomic_decrement(var) __atomic_sub_fetch(&string_base.var, 1, __ATOMIC_RELAXED)
1947 + if(likely(item))
1948 + return item->shared->value;
1949
1471 -void string_statistics(size_t *inserts, size_t *deletes, size_t *searches, size_t *entries, size_t *references, size_t *memory, size_t *duplications, size_t *releases) {
1472 - *inserts = string_base.inserts;
1473 - *deletes = string_base.deletes;
1474 - *searches = string_base.searches;
1475 - *entries = (size_t)string_base.entries;
1476 - *references = (size_t)string_base.active_references;
1477 - *memory = (size_t)string_base.memory;
1478 - *duplications = string_base.duplications;
1479 - *releases = string_base.releases;
1950 + return NULL;
1951 }
1952
1482 -#define string_entry_acquire(se) __atomic_add_fetch(&((se)->refcount), 1, __ATOMIC_SEQ_CST);
1483 -#define string_entry_release(se) __atomic_sub_fetch(&((se)->refcount), 1, __ATOMIC_SEQ_CST);
1953 +// ----------------------------------------------------------------------------
1954 +// DEL an item
1955
1485 -static inline bool string_entry_check_and_acquire(STRING *se) {
1486 - int32_t expected, desired, count = 0;
1487 - do {
1488 - count++;
1956 +bool dictionary_del_advanced(DICTIONARY *dict, const char *name, ssize_t name_len) {
1957 + if(unlikely(!api_is_name_good(dict, name, name_len)))
1958 + return false;
1959
1490 - expected = se->refcount;
1960 + api_internal_check(dict, NULL, false, true);
1961 + return item_del(dict, name, name_len);
1962 +}
1963
1492 - if(expected <= 0) {
1493 - // We cannot use this.
1494 - // The reference counter reached value zero,
1495 - // so another thread is deleting this.
1496 - string_internal_stats_add(spins, count - 1);
1497 - return false;
1498 - }
1964 +// ----------------------------------------------------------------------------
1965 +// traversal with loop
1966 +
1967 +void *dictionary_foreach_start_rw(DICTFE *dfe, DICTIONARY *dict, char rw) {
1968 + if(unlikely(!dfe || !dict)) return NULL;
1969
1500 - desired = expected + 1;
1970 + if(unlikely(is_dictionary_destroyed(dict))) {
1971 + internal_error(true, "DICTIONARY: attempted to dictionary_foreach_start_rw() on a destroyed dictionary");
1972 + dfe->counter = 0;
1973 + dfe->item = NULL;
1974 + dfe->name = NULL;
1975 + dfe->value = NULL;
1976 + return NULL;
1977 }
1502 - while(!__atomic_compare_exchange_n(&se->refcount, &expected, desired, false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST));
1978
1504 - string_internal_stats_add(spins, count - 1);
1979 + dfe->counter = 0;
1980 + dfe->dict = dict;
1981 + dfe->rw = rw;
1982
1506 - // statistics
1507 - // string_base.active_references is altered at the in string_strdupz() and string_freez()
1508 - string_stats_atomic_increment(duplications);
1983 + ll_recursive_lock(dict, dfe->rw);
1984
1510 - return true;
1511 -}
1985 + DICTIONARY_STATS_TRAVERSALS_PLUS1(dict);
1986
1513 -STRING *string_dup(STRING *string) {
1514 - if(unlikely(!string)) return NULL;
1987 + // get the first item from the list
1988 + DICTIONARY_ITEM *item = dict->items.list;
1989
1516 -#ifdef NETDATA_INTERNAL_CHECKS
1517 - if(unlikely(__atomic_load_n(&string->refcount, __ATOMIC_SEQ_CST) <= 0))
1518 - fatal("STRING: tried to %s() a string that is freed (it has %d references).", __FUNCTION__, string->refcount);
1519 -#endif
1990 + // skip all the deleted items
1991 + while(item && !item_check_and_acquire(dict, item))
1992 + item = item->next;
1993
1521 - string_entry_acquire(string);
1994 + if(likely(item)) {
1995 + dfe->item = item;
1996 + dfe->name = (char *)item_get_name(item);
1997 + dfe->value = item->shared->value;
1998 + }
1999 + else {
2000 + dfe->item = NULL;
2001 + dfe->name = NULL;
2002 + dfe->value = NULL;
2003 + }
2004
1523 - // statistics
1524 - string_stats_atomic_increment(active_references);
1525 - string_stats_atomic_increment(duplications);
2005 + if(unlikely(dfe->rw == DICTIONARY_LOCK_REENTRANT))
2006 + ll_recursive_unlock(dfe->dict, dfe->rw);
2007
1527 - return string;
2008 + return dfe->value;
2009 }
2010
1530 -// Search the index and return an ACQUIRED string entry, or NULL
1531 -static inline STRING *string_index_search(const char *str, size_t length) {
1532 - if(unlikely(!string_base.JudyHSArray))
2011 +void *dictionary_foreach_next(DICTFE *dfe) {
2012 + if(unlikely(!dfe || !dfe->dict)) return NULL;
2013 +
2014 + if(unlikely(is_dictionary_destroyed(dfe->dict))) {
2015 + internal_error(true, "DICTIONARY: attempted to dictionary_foreach_next() on a destroyed dictionary");
2016 + dfe->item = NULL;
2017 + dfe->name = NULL;
2018 + dfe->value = NULL;
2019 return NULL;
2020 + }
2021
1535 - STRING *string;
2022 + if(unlikely(dfe->rw == DICTIONARY_LOCK_REENTRANT))
2023 + ll_recursive_lock(dfe->dict, dfe->rw);
2024
1537 - // Find the string in the index
1538 - // With a read-lock so that multiple readers can use the index concurrently.
2025 + // the item we just did
2026 + DICTIONARY_ITEM *item = dfe->item;
2027
1540 - netdata_rwlock_rdlock(&string_base.rwlock);
2028 + // get the next item from the list
2029 + DICTIONARY_ITEM *item_next = (item) ? item->next : NULL;
2030
1542 - Pvoid_t *Rc;
1543 - Rc = JudyHSGet(string_base.JudyHSArray, (void *)str, length);
1544 - if(likely(Rc)) {
1545 - // found in the hash table
1546 - string = *Rc;
2031 + // skip all the deleted items until one that can be acquired is found
2032 + while(item_next && !item_check_and_acquire(dfe->dict, item_next))
2033 + item_next = item_next->next;
2034
1548 - if(string_entry_check_and_acquire(string)) {
1549 - // we can use this entry
1550 - string_internal_stats_add(found_available_on_search, 1);
1551 - }
1552 - else {
1553 - // this entry is about to be deleted by another thread
1554 - // do not touch it, let it go...
1555 - string = NULL;
1556 - string_internal_stats_add(found_deleted_on_search, 1);
1557 - }
2035 + if(likely(item)) {
2036 + item_release_and_check_if_it_is_deleted_and_can_be_removed_under_this_lock_mode(dfe->dict, item, dfe->rw);
2037 + // item_release(dfe->dict, item);
2038 + }
2039 +
2040 + item = item_next;
2041 + if(likely(item)) {
2042 + dfe->item = item;
2043 + dfe->name = (char *)item_get_name(item);
2044 + dfe->value = item->shared->value;
2045 + dfe->counter++;
2046 }
2047 else {
1560 - // not found in the hash table
1561 - string = NULL;
2048 + dfe->item = NULL;
2049 + dfe->name = NULL;
2050 + dfe->value = NULL;
2051 }
2052
1564 - string_stats_atomic_increment(searches);
1565 - netdata_rwlock_unlock(&string_base.rwlock);
2053 + if(unlikely(dfe->rw == DICTIONARY_LOCK_REENTRANT))
2054 + ll_recursive_unlock(dfe->dict, dfe->rw);
2055
1567 - return string;
2056 + return dfe->value;
2057 }
2058
1570 -// Insert a string to the index and return an ACQUIRED string entry,
1571 -// or NULL if the call needs to be retried (a deleted entry with the same key is still in the index)
1572 -// The returned entry is ACQUIRED and it can either be:
1573 -// 1. a new item inserted, or
1574 -// 2. an item found in the index that is not currently deleted
1575 -static inline STRING *string_index_insert(const char *str, size_t length) {
1576 - STRING *string;
1577 -
1578 - netdata_rwlock_wrlock(&string_base.rwlock);
1579 -
1580 - STRING **ptr;
1581 - {
1582 - JError_t J_Error;
1583 - Pvoid_t *Rc = JudyHSIns(&string_base.JudyHSArray, (void *)str, length, &J_Error);
1584 - if (unlikely(Rc == PJERR)) {
1585 - fatal(
1586 - "STRING: Cannot insert entry with name '%s' to JudyHS, JU_ERRNO_* == %u, ID == %d",
1587 - str,
1588 - JU_ERRNO(&J_Error),
1589 - JU_ERRID(&J_Error));
1590 - }
1591 - ptr = (STRING **)Rc;
1592 - }
2059 +void dictionary_foreach_done(DICTFE *dfe) {
2060 + if(unlikely(!dfe || !dfe->dict)) return;
2061
1594 - if (likely(*ptr == 0)) {
1595 - // a new item added to the index
1596 - size_t mem_size = sizeof(STRING) + length;
1597 - string = mallocz(mem_size);
1598 - strcpy((char *)string->str, str);
1599 - string->length = length;
1600 - string->refcount = 1;
1601 - *ptr = string;
1602 - string_base.inserts++;
1603 - string_base.entries++;
1604 - string_base.memory += (long)mem_size;
2062 + if(unlikely(is_dictionary_destroyed(dfe->dict))) {
2063 + internal_error(true, "DICTIONARY: attempted to dictionary_foreach_next() on a destroyed dictionary");
2064 + return;
2065 }
1606 - else {
1607 - // the item is already in the index
1608 - string = *ptr;
2066
1610 - if(string_entry_check_and_acquire(string)) {
1611 - // we can use this entry
1612 - string_internal_stats_add(found_available_on_insert, 1);
1613 - }
1614 - else {
1615 - // this entry is about to be deleted by another thread
1616 - // do not touch it, let it go...
1617 - string = NULL;
1618 - string_internal_stats_add(found_deleted_on_insert, 1);
1619 - }
2067 + // the item we just did
2068 + DICTIONARY_ITEM *item = dfe->item;
2069
1621 - string_stats_atomic_increment(searches);
2070 + // release it, so that it can possibly be deleted
2071 + if(likely(item)) {
2072 + item_release_and_check_if_it_is_deleted_and_can_be_removed_under_this_lock_mode(dfe->dict, item, dfe->rw);
2073 + // item_release(dfe->dict, item);
2074 }
2075
1624 - netdata_rwlock_unlock(&string_base.rwlock);
1625 - return string;
1626 -}
2076 + if(likely(dfe->rw != DICTIONARY_LOCK_REENTRANT))
2077 + ll_recursive_unlock(dfe->dict, dfe->rw);
2078
1628 -// delete an entry from the index
1629 -static inline void string_index_delete(STRING *string) {
1630 - netdata_rwlock_wrlock(&string_base.rwlock);
2079 + dfe->dict = NULL;
2080 + dfe->item = NULL;
2081 + dfe->name = NULL;
2082 + dfe->value = NULL;
2083 + dfe->counter = 0;
2084 +}
2085
1632 -#ifdef NETDATA_INTERNAL_CHECKS
1633 - if(unlikely(__atomic_load_n(&string->refcount, __ATOMIC_SEQ_CST) != 0))
1634 - fatal("STRING: tried to delete a string at %s() that is already freed (it has %d references).", __FUNCTION__, string->refcount);
1635 -#endif
2086 +// ----------------------------------------------------------------------------
2087 +// API - walk through the dictionary.
2088 +// The dictionary is locked for reading while this happens
2089 +// do not use other dictionary calls while walking the dictionary - deadlock!
2090
1637 - bool deleted = false;
2091 +int dictionary_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const DICTIONARY_ITEM *item, void *entry, void *data), void *data) {
2092 + if(unlikely(!dict || !callback)) return 0;
2093
1639 - if (likely(string_base.JudyHSArray)) {
1640 - JError_t J_Error;
1641 - int ret = JudyHSDel(&string_base.JudyHSArray, (void *)string->str, string->length, &J_Error);
1642 - if (unlikely(ret == JERR)) {
1643 - error(
1644 - "STRING: Cannot delete entry with name '%s' from JudyHS, JU_ERRNO_* == %u, ID == %d",
1645 - string->str,
1646 - JU_ERRNO(&J_Error),
1647 - JU_ERRID(&J_Error));
1648 - } else
1649 - deleted = true;
2094 + if(unlikely(is_dictionary_destroyed(dict))) {
2095 + internal_error(true, "DICTIONARY: attempted to dictionary_walkthrough_rw() on a destroyed dictionary");
2096 + return 0;
2097 }
2098
1652 - if (unlikely(!deleted))
1653 - error("STRING: tried to delete '%s' that is not in the index. Ignoring it.", string->str);
1654 - else {
1655 - size_t mem_size = sizeof(STRING) + string->length;
1656 - string_base.deletes++;
1657 - string_base.entries--;
1658 - string_base.memory -= (long)mem_size;
1659 - freez(string);
1660 - }
2099 + ll_recursive_lock(dict, rw);
2100
1662 - netdata_rwlock_unlock(&string_base.rwlock);
1663 -}
2101 + DICTIONARY_STATS_WALKTHROUGHS_PLUS1(dict);
2102
1665 -STRING *string_strdupz(const char *str) {
1666 - if(unlikely(!str || !*str)) return NULL;
2103 + // written in such a way, that the callback can delete the active element
2104
1668 - size_t length = strlen(str) + 1;
1669 - STRING *string = string_index_search(str, length);
2105 + int ret = 0;
2106 + DICTIONARY_ITEM *item = dict->items.list, *item_next;
2107 + while(item) {
2108
1671 - while(!string) {
1672 - // The search above did not find anything,
1673 - // We loop here, because during insert we may find an entry that is being deleted by another thread.
1674 - // So, we have to let it go and retry to insert it again.
2109 + // skip the deleted items
2110 + if(unlikely(!item_check_and_acquire(dict, item))) {
2111 + item = item->next;
2112 + continue;
2113 + }
2114
1676 - string = string_index_insert(str, length);
1677 - }
2115 + if(unlikely(rw == DICTIONARY_LOCK_REENTRANT))
2116 + ll_recursive_unlock(dict, rw);
2117
1679 - // statistics
1680 - string_stats_atomic_increment(active_references);
2118 + int r = callback(item, item->shared->value, data);
2119
1682 - return string;
1683 -}
2120 + if(unlikely(rw == DICTIONARY_LOCK_REENTRANT))
2121 + ll_recursive_lock(dict, rw);
2122
1685 -void string_freez(STRING *string) {
1686 - if(unlikely(!string)) return;
2123 + // since we have a reference counter, this item cannot be deleted
2124 + // until we release the reference counter, so the pointers are there
2125 + item_next = item->next;
2126
1688 - int32_t refcount = string_entry_release(string);
2127 + item_release_and_check_if_it_is_deleted_and_can_be_removed_under_this_lock_mode(dict, item, rw);
2128 + // item_release(dict, item);
2129
1690 -#ifdef NETDATA_INTERNAL_CHECKS
1691 - if(unlikely(refcount < 0))
1692 - fatal("STRING: tried to %s() a string that is already freed (it has %d references).", __FUNCTION__, string->refcount);
1693 -#endif
2130 + if(unlikely(r < 0)) {
2131 + ret = r;
2132 + break;
2133 + }
2134 +
2135 + ret += r;
2136
1695 - if(unlikely(refcount == 0))
1696 - string_index_delete(string);
2137 + item = item_next;
2138 + }
2139
1698 - // statistics
1699 - string_stats_atomic_decrement(active_references);
1700 - string_stats_atomic_increment(releases);
1701 -}
2140 + ll_recursive_unlock(dict, rw);
2141
1703 -size_t string_strlen(STRING *string) {
1704 - if(unlikely(!string)) return 0;
1705 - return string->length - 1;
2142 + return ret;
2143 }
2144
1708 -const char *string2str(STRING *string) {
1709 - if(unlikely(!string)) return "";
1710 - return string->str;
2145 +// ----------------------------------------------------------------------------
2146 +// sorted walkthrough
2147 +
2148 +static int dictionary_sort_compar(const void *item1, const void *item2) {
2149 + return strcmp(item_get_name((*(DICTIONARY_ITEM **)item1)), item_get_name((*(DICTIONARY_ITEM **)item2)));
2150 }
2151
1713 -STRING *string_2way_merge(STRING *a, STRING *b) {
1714 - static STRING *X = NULL;
2152 +int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const DICTIONARY_ITEM *item, void *entry, void *data), void *data) {
2153 + if(unlikely(!dict || !callback)) return 0;
2154 +
2155 + if(unlikely(is_dictionary_destroyed(dict))) {
2156 + internal_error(true, "DICTIONARY: attempted to dictionary_sorted_walkthrough_rw() on a destroyed dictionary");
2157 + return 0;
2158 + }
2159 +
2160 + DICTIONARY_STATS_WALKTHROUGHS_PLUS1(dict);
2161 +
2162 + ll_recursive_lock(dict, rw);
2163 + size_t entries = __atomic_load_n(&dict->entries, __ATOMIC_SEQ_CST);
2164 + DICTIONARY_ITEM **array = mallocz(sizeof(DICTIONARY_ITEM *) * entries);
2165
1716 - if(unlikely(!X)) {
1717 - X = string_strdupz("[x]");
2166 + size_t i;
2167 + DICTIONARY_ITEM *item;
2168 + for(item = dict->items.list, i = 0; item && i < entries; item = item->next) {
2169 + if(likely(item_check_and_acquire(dict, item)))
2170 + array[i++] = item;
2171 }
2172 + ll_recursive_unlock(dict, rw);
2173
1720 - if(unlikely(a == b)) return string_dup(a);
1721 - if(unlikely(a == X)) return string_dup(a);
1722 - if(unlikely(b == X)) return string_dup(b);
1723 - if(unlikely(!a)) return string_dup(X);
1724 - if(unlikely(!b)) return string_dup(X);
2174 + if(unlikely(i != entries))
2175 + entries = i;
2176
1726 - size_t alen = string_strlen(a);
1727 - size_t blen = string_strlen(b);
1728 - size_t length = alen + blen + string_strlen(X) + 1;
1729 - char buf1[length + 1], buf2[length + 1], *dst1;
1730 - const char *s1, *s2;
2177 + qsort(array, entries, sizeof(DICTIONARY_ITEM *), dictionary_sort_compar);
2178
1732 - s1 = string2str(a);
1733 - s2 = string2str(b);
1734 - dst1 = buf1;
1735 - for( ; *s1 && *s2 && *s1 == *s2 ;s1++, s2++)
1736 - *dst1++ = *s1;
2179 + bool callit = true;
2180 + int ret = 0, r;
2181 + for(i = 0; i < entries ;i++) {
2182 + item = array[i];
2183
1738 - *dst1 = '\0';
2184 + if(callit)
2185 + r = callback(item, item->shared->value, data);
2186
1740 - if(*s1 != '\0' || *s2 != '\0') {
1741 - *dst1++ = '[';
1742 - *dst1++ = 'x';
1743 - *dst1++ = ']';
2187 + item_release_and_check_if_it_is_deleted_and_can_be_removed_under_this_lock_mode(dict, item, rw);
2188 + // item_release(dict, item);
2189
1745 - s1 = &(string2str(a))[alen - 1];
1746 - s2 = &(string2str(b))[blen - 1];
1747 - char *dst2 = &buf2[length];
1748 - *dst2 = '\0';
1749 - for (; *s1 && *s2 && *s1 == *s2; s1--, s2--)
1750 - *(--dst2) = *s1;
2190 + if(r < 0) {
2191 + ret = r;
2192 + r = 0;
2193
1752 - strcpy(dst1, dst2);
2194 + // stop calling the callback,
2195 + // but we have to continue, to release all the reference counters
2196 + callit = false;
2197 + }
2198 + else
2199 + ret += r;
2200 }
2201
1755 - return string_strdupz(buf1);
2202 + freez(array);
2203 +
2204 + return ret;
2205 }
2206
2207 // ----------------------------------------------------------------------------
@@ -1845,12 +2294,12 @@ static size_t dictionary_unittest_set_clone(DICTIONARY *dict, char **names, char
2294 static size_t dictionary_unittest_set_null(DICTIONARY *dict, char **names, char **values, size_t entries) {
2295 (void)values;
2296 size_t errors = 0;
1848 - long i = 0;
1849 - for(; i < (long)entries ;i++) {
2297 + size_t i = 0;
2298 + for(; i < entries ;i++) {
2299 void *val = dictionary_set(dict, names[i], NULL, 0);
2300 if(val != NULL) { fprintf(stderr, ">>> %s() returns a non NULL value\n", __FUNCTION__); errors++; }
2301 }
1853 - if(dictionary_stats_entries(dict) != i) {
2302 + if(dictionary_entries(dict) != i) {
2303 fprintf(stderr, ">>> %s() dictionary items do not match\n", __FUNCTION__);
2304 errors++;
2305 }
@@ -1902,8 +2351,8 @@ static size_t dictionary_unittest_del_nonexisting(DICTIONARY *dict, char **names
2351 (void)names;
2352 size_t errors = 0;
2353 for(size_t i = 0; i < entries ;i++) {
1905 - int ret = dictionary_del(dict, values[i]);
1906 - if(ret != -1) { fprintf(stderr, ">>> %s() deleted non-existing item\n", __FUNCTION__); errors++; }
2354 + bool ret = dictionary_del(dict, values[i]);
2355 + if(ret) { fprintf(stderr, ">>> %s() deleted non-existing item\n", __FUNCTION__); errors++; }
2356 }
2357 return errors;
2358 }
@@ -1917,18 +2366,18 @@ static size_t dictionary_unittest_del_existing(DICTIONARY *dict, char **names, c
2366 size_t backward_from = middle_to, backward_to = entries;
2367
2368 for(size_t i = forward_from; i < forward_to ;i++) {
1920 - int ret = dictionary_del(dict, names[i]);
1921 - if(ret == -1) { fprintf(stderr, ">>> %s() didn't delete (forward) existing item\n", __FUNCTION__); errors++; }
2369 + bool ret = dictionary_del(dict, names[i]);
2370 + if(!ret) { fprintf(stderr, ">>> %s() didn't delete (forward) existing item\n", __FUNCTION__); errors++; }
2371 }
2372
2373 for(size_t i = middle_to - 1; i >= middle_from ;i--) {
1925 - int ret = dictionary_del(dict, names[i]);
1926 - if(ret == -1) { fprintf(stderr, ">>> %s() didn't delete (middle) existing item\n", __FUNCTION__); errors++; }
2374 + bool ret = dictionary_del(dict, names[i]);
2375 + if(!ret) { fprintf(stderr, ">>> %s() didn't delete (middle) existing item\n", __FUNCTION__); errors++; }
2376 }
2377
2378 for(size_t i = backward_to - 1; i >= backward_from ;i--) {
1930 - int ret = dictionary_del(dict, names[i]);
1931 - if(ret == -1) { fprintf(stderr, ">>> %s() didn't delete (backward) existing item\n", __FUNCTION__); errors++; }
2379 + bool ret = dictionary_del(dict, names[i]);
2380 + if(!ret) { fprintf(stderr, ">>> %s() didn't delete (backward) existing item\n", __FUNCTION__); errors++; }
2381 }
2382
2383 return errors;
@@ -1971,10 +2420,7 @@ static size_t dictionary_unittest_reset_dont_overwrite_nonclone(DICTIONARY *dict
2420 return errors;
2421 }
2422
1974 -static int dictionary_unittest_walkthrough_callback(const char *name, void *value, void *data) {
1975 - (void)name;
1976 - (void)value;
1977 - (void)data;
2423 +static int dictionary_unittest_walkthrough_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value __maybe_unused, void *data __maybe_unused) {
2424 return 1;
2425 }
2426
@@ -1986,10 +2432,10 @@ static size_t dictionary_unittest_walkthrough(DICTIONARY *dict, char **names, ch
2432 else return sum - entries;
2433 }
2434
1989 -static int dictionary_unittest_walkthrough_delete_this_callback(const char *name, void *value, void *data) {
1990 - (void)value;
2435 +static int dictionary_unittest_walkthrough_delete_this_callback(const DICTIONARY_ITEM *item, void *value __maybe_unused, void *data) {
2436 + const char *name = dictionary_acquired_item_name((DICTIONARY_ITEM *)item);
2437
1992 - if(dictionary_del_having_write_lock((DICTIONARY *)data, name) == -1)
2438 + if(!dictionary_del((DICTIONARY *)data, name))
2439 return 0;
2440
2441 return 1;
@@ -2003,10 +2449,7 @@ static size_t dictionary_unittest_walkthrough_delete_this(DICTIONARY *dict, char
2449 else return sum - entries;
2450 }
2451
2006 -static int dictionary_unittest_walkthrough_stop_callback(const char *name, void *value, void *data) {
2007 - (void)name;
2008 - (void)value;
2009 - (void)data;
2452 +static int dictionary_unittest_walkthrough_stop_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value __maybe_unused, void *data __maybe_unused) {
2453 return -1;
2454 }
2455
@@ -2040,7 +2483,7 @@ static size_t dictionary_unittest_foreach_delete_this(DICTIONARY *dict, char **n
2483 size_t count = 0;
2484 char *item;
2485 dfe_start_write(dict, item)
2043 - if(dictionary_del_having_write_lock(dict, item_name) != -1) count++;
2486 + if(dictionary_del(dict, item_dfe.name)) count++;
2487 dfe_done(item);
2488
2489 if(count > entries) return count - entries;
@@ -2066,7 +2509,22 @@ static usec_t dictionary_unittest_run_and_measure_time(DICTIONARY *dict, char *m
2509
2510 if(callback == dictionary_unittest_destroy) dict = NULL;
2511
2069 - fprintf(stderr, " %zu errors, %ld items in dictionary, %llu usec \n", errs, dict? dictionary_stats_entries(dict):0, dt);
2512 + long int found_ok = 0, found_deleted = 0, found_referenced = 0;
2513 + if(dict) {
2514 + DICTIONARY_ITEM *item;
2515 + DOUBLE_LINKED_LIST_FOREACH_FORWARD(dict->items.list, item, prev, next) {
2516 + if(item->refcount >= 0 && !(item ->flags & ITEM_FLAG_DELETED))
2517 + found_ok++;
2518 + else
2519 + found_deleted++;
2520 +
2521 + if(item->refcount > 0)
2522 + found_referenced++;
2523 + }
2524 + }
2525 +
2526 + fprintf(stderr, " %zu errors, %ld (found %ld) items in dictionary, %ld (found %ld) referenced, %ld (found %ld) deleted, %llu usec \n",
2527 + errs, dict?dict->entries:0, found_ok, dict?dict->referenced_items:0, found_referenced, dict?dict->pending_deletion_items:0, found_deleted, dt);
2528 *errors += errs;
2529 return dt;
2530 }
@@ -2102,23 +2560,24 @@ static void dictionary_unittest_nonclone(DICTIONARY *dict, char **names, char **
2560 }
2561
2562 struct dictionary_unittest_sorting {
2105 - const char *oldname;
2106 - const char *oldvalue;
2563 + const char *old_name;
2564 + const char *old_value;
2565 size_t count;
2566 };
2567
2110 -static int dictionary_unittest_sorting_callback(const char *name, void *value, void *data) {
2568 +static int dictionary_unittest_sorting_callback(const DICTIONARY_ITEM *item, void *value, void *data) {
2569 + const char *name = dictionary_acquired_item_name((DICTIONARY_ITEM *)item);
2570 struct dictionary_unittest_sorting *t = (struct dictionary_unittest_sorting *)data;
2571 const char *v = (const char *)value;
2572
2573 int ret = 0;
2115 - if(t->oldname && strcmp(t->oldname, name) > 0) {
2116 - fprintf(stderr, "name '%s' should be after '%s'\n", t->oldname, name);
2574 + if(t->old_name && strcmp(t->old_name, name) > 0) {
2575 + fprintf(stderr, "name '%s' should be after '%s'\n", t->old_name, name);
2576 ret = 1;
2577 }
2578 t->count++;
2120 - t->oldname = name;
2121 - t->oldvalue = v;
2579 + t->old_name = name;
2580 + t->old_value = v;
2581
2582 return ret;
2583 }
@@ -2126,7 +2585,7 @@ static int dictionary_unittest_sorting_callback(const char *name, void *value, v
2585 static size_t dictionary_unittest_sorted_walkthrough(DICTIONARY *dict, char **names, char **values, size_t entries) {
2586 (void)names;
2587 (void)values;
2129 - struct dictionary_unittest_sorting tmp = { .oldname = NULL, .oldvalue = NULL, .count = 0 };
2588 + struct dictionary_unittest_sorting tmp = { .old_name = NULL, .old_value = NULL, .count = 0 };
2589 size_t errors;
2590 errors = dictionary_sorted_walkthrough_read(dict, dictionary_unittest_sorting_callback, &tmp);
2591
@@ -2148,62 +2607,94 @@ static void dictionary_unittest_null_dfe(DICTIONARY *dict, char **names, char **
2607 }
2608
2609
2151 -static int check_dictionary_callback(const char *name, void *value, void *data) {
2152 - (void)name;
2153 - (void)value;
2154 - (void)data;
2610 +static int unittest_check_dictionary_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value __maybe_unused, void *data __maybe_unused) {
2611 return 1;
2612 }
2613
2158 -static size_t check_dictionary(DICTIONARY *dict, size_t entries, size_t linked_list_members) {
2614 +static size_t unittest_check_dictionary(const char *label, DICTIONARY *dict, size_t traversable, size_t active_items, size_t deleted_items, size_t referenced_items, size_t pending_deletion) {
2615 size_t errors = 0;
2616
2161 - fprintf(stderr, "dictionary entries %ld, expected %zu...\t\t\t\t\t", dictionary_stats_entries(dict), entries);
2162 - if (dictionary_stats_entries(dict) != (long)entries) {
2617 + size_t ll = 0;
2618 + void *t;
2619 + dfe_start_read(dict, t)
2620 + ll++;
2621 + dfe_done(t);
2622 +
2623 + fprintf(stderr, "DICT %-20s: dictionary foreach entries %zu, expected %zu...\t\t\t\t\t",
2624 + label, ll, traversable);
2625 + if(ll != traversable) {
2626 fprintf(stderr, "FAILED\n");
2627 errors++;
2628 }
2629 else
2630 fprintf(stderr, "OK\n");
2631
2169 - size_t ll = 0;
2170 - void *t;
2171 - dfe_start_read(dict, t)
2172 - ll++;
2173 - dfe_done(t);
2632 + ll = dictionary_walkthrough_read(dict, unittest_check_dictionary_callback, NULL);
2633 + fprintf(stderr, "DICT %-20s: dictionary walkthrough entries %zu, expected %zu...\t\t\t\t",
2634 + label, ll, traversable);
2635 + if(ll != traversable) {
2636 + fprintf(stderr, "FAILED\n");
2637 + errors++;
2638 + }
2639 + else
2640 + fprintf(stderr, "OK\n");
2641
2175 - fprintf(stderr, "dictionary foreach entries %zu, expected %zu...\t\t\t\t", ll, entries);
2176 - if(ll != entries) {
2642 + ll = dictionary_sorted_walkthrough_read(dict, unittest_check_dictionary_callback, NULL);
2643 + fprintf(stderr, "DICT %-20s: dictionary sorted walkthrough entries %zu, expected %zu...\t\t\t",
2644 + label, ll, traversable);
2645 + if(ll != traversable) {
2646 fprintf(stderr, "FAILED\n");
2647 errors++;
2648 }
2649 else
2650 fprintf(stderr, "OK\n");
2651
2183 - ll = dictionary_walkthrough_read(dict, check_dictionary_callback, NULL);
2184 - fprintf(stderr, "dictionary walkthrough entries %zu, expected %zu...\t\t\t\t", ll, entries);
2185 - if(ll != entries) {
2652 + DICTIONARY_ITEM *item;
2653 + size_t active = 0, deleted = 0, referenced = 0, pending = 0;
2654 + for(item = dict->items.list; item; item = item->next) {
2655 + if(!(item->flags & ITEM_FLAG_DELETED) && !(item->shared->flags & ITEM_FLAG_DELETED))
2656 + active++;
2657 + else {
2658 + deleted++;
2659 +
2660 + if(item->refcount == 0)
2661 + pending++;
2662 + }
2663 +
2664 + if(item->refcount > 0)
2665 + referenced++;
2666 + }
2667 +
2668 + fprintf(stderr, "DICT %-20s: dictionary active items reported %ld, counted %zu, expected %zu...\t\t\t",
2669 + label, dict->entries, active, active_items);
2670 + if(active != active_items || active != (size_t)dict->entries) {
2671 fprintf(stderr, "FAILED\n");
2672 errors++;
2673 }
2674 else
2675 fprintf(stderr, "OK\n");
2676
2192 - ll = dictionary_sorted_walkthrough_read(dict, check_dictionary_callback, NULL);
2193 - fprintf(stderr, "dictionary sorted walkthrough entries %zu, expected %zu...\t\t\t", ll, entries);
2194 - if(ll != entries) {
2677 + fprintf(stderr, "DICT %-20s: dictionary deleted items counted %zu, expected %zu...\t\t\t\t",
2678 + label, deleted, deleted_items);
2679 + if(deleted != deleted_items) {
2680 fprintf(stderr, "FAILED\n");
2681 errors++;
2682 }
2683 else
2684 fprintf(stderr, "OK\n");
2685
2201 - NAME_VALUE *nv;
2202 - for(ll = 0, nv = dict->first_item; nv ;nv = nv->next)
2203 - ll++;
2686 + fprintf(stderr, "DICT %-20s: dictionary referenced items reported %ld, counted %zu, expected %zu...\t\t",
2687 + label, dict->referenced_items, referenced, referenced_items);
2688 + if(referenced != referenced_items || dict->referenced_items != (long int)referenced) {
2689 + fprintf(stderr, "FAILED\n");
2690 + errors++;
2691 + }
2692 + else
2693 + fprintf(stderr, "OK\n");
2694
2205 - fprintf(stderr, "dictionary linked list entries %zu, expected %zu...\t\t\t\t", ll, linked_list_members);
2206 - if(ll != linked_list_members) {
2695 + fprintf(stderr, "DICT %-20s: dictionary pending deletion items reported %ld, counted %zu, expected %zu...\t",
2696 + label, dict->pending_deletion_items, pending, pending_deletion);
2697 + if(pending != pending_deletion || pending != (size_t)dict->pending_deletion_items) {
2698 fprintf(stderr, "FAILED\n");
2699 errors++;
2700 }
@@ -2213,40 +2704,44 @@ static size_t check_dictionary(DICTIONARY *dict, size_t entries, size_t linked_l
2704 return errors;
2705 }
2706
2216 -static int check_name_value_callback(const char *name, void *value, void *data) {
2217 - (void)name;
2707 +static int check_item_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data) {
2708 return value == data;
2709 }
2710
2221 -static size_t check_name_value_deleted_flag(DICTIONARY *dict, NAME_VALUE *nv, const char *name, const char *value, unsigned refcount, NAME_VALUE_FLAGS deleted_flags, bool searchable, bool browsable, bool linked) {
2711 +static size_t unittest_check_item(const char *label, DICTIONARY *dict,
2712 + DICTIONARY_ITEM *item, const char *name, const char *value, int refcount,
2713 + ITEM_FLAGS deleted_flags, bool searchable, bool browsable, bool linked) {
2714 size_t errors = 0;
2715
2224 - fprintf(stderr, "NAME_VALUE name is '%s', expected '%s'...\t\t\t\t", namevalue_get_name(nv), name);
2225 - if(strcmp(namevalue_get_name(nv), name) != 0) {
2716 + fprintf(stderr, "ITEM %-20s: name is '%s', expected '%s'...\t\t\t\t\t\t", label, item_get_name(item), name);
2717 + if(strcmp(item_get_name(item), name) != 0) {
2718 fprintf(stderr, "FAILED\n");
2719 errors++;
2720 }
2721 else
2722 fprintf(stderr, "OK\n");
2723
2232 - fprintf(stderr, "NAME_VALUE value is '%s', expected '%s'...\t\t\t", (const char *)nv->value, value);
2233 - if(strcmp((const char *)nv->value, value) != 0) {
2724 + fprintf(stderr, "ITEM %-20s: value is '%s', expected '%s'...\t\t\t\t\t", label, (const char *)item->shared->value, value);
2725 + if(strcmp((const char *)item->shared->value, value) != 0) {
2726 fprintf(stderr, "FAILED\n");
2727 errors++;
2728 }
2729 else
2730 fprintf(stderr, "OK\n");
2731
2240 - fprintf(stderr, "NAME_VALUE refcount is %u, expected %u...\t\t\t\t\t", nv->refcount, refcount);
2241 - if (nv->refcount != refcount) {
2732 + fprintf(stderr, "ITEM %-20s: refcount is %d, expected %d...\t\t\t\t\t\t\t", label, item->refcount, refcount);
2733 + if (item->refcount != refcount) {
2734 fprintf(stderr, "FAILED\n");
2735 errors++;
2736 }
2737 else
2738 fprintf(stderr, "OK\n");
2739
2248 - fprintf(stderr, "NAME_VALUE deleted flag is %s, expected %s...\t\t\t", (nv->flags & NAME_VALUE_FLAG_DELETED)?"TRUE":"FALSE", (deleted_flags & NAME_VALUE_FLAG_DELETED)?"TRUE":"FALSE");
2249 - if ((nv->flags & NAME_VALUE_FLAG_DELETED) != (deleted_flags & NAME_VALUE_FLAG_DELETED)) {
2740 + fprintf(stderr, "ITEM %-20s: deleted flag is %s, expected %s...\t\t\t\t\t", label,
2741 + (item->flags & ITEM_FLAG_DELETED || item->shared->flags & ITEM_FLAG_DELETED)?"true":"false",
2742 + (deleted_flags & ITEM_FLAG_DELETED)?"true":"false");
2743 +
2744 + if ((item->flags & ITEM_FLAG_DELETED || item->shared->flags & ITEM_FLAG_DELETED) != (deleted_flags & ITEM_FLAG_DELETED)) {
2745 fprintf(stderr, "FAILED\n");
2746 errors++;
2747 }
@@ -2254,8 +2749,9 @@ static size_t check_name_value_deleted_flag(DICTIONARY *dict, NAME_VALUE *nv, co
2749 fprintf(stderr, "OK\n");
2750
2751 void *v = dictionary_get(dict, name);
2257 - bool found = v == nv->value;
2258 - fprintf(stderr, "NAME_VALUE searchable %5s, expected %5s...\t\t\t\t", found?"true":"false", searchable?"true":"false");
2752 + bool found = v == item->shared->value;
2753 + fprintf(stderr, "ITEM %-20s: searchable %5s, expected %5s...\t\t\t\t\t\t", label,
2754 + found?"true":"false", searchable?"true":"false");
2755 if(found != searchable) {
2756 fprintf(stderr, "FAILED\n");
2757 errors++;
@@ -2266,11 +2762,12 @@ static size_t check_name_value_deleted_flag(DICTIONARY *dict, NAME_VALUE *nv, co
2762 found = false;
2763 void *t;
2764 dfe_start_read(dict, t) {
2269 - if(t == nv->value) found = true;
2765 + if(t == item->shared->value) found = true;
2766 }
2767 dfe_done(t);
2768
2273 - fprintf(stderr, "NAME_VALUE dfe browsable %5s, expected %5s...\t\t\t", found?"true":"false", browsable?"true":"false");
2769 + fprintf(stderr, "ITEM %-20s: dfe browsable %5s, expected %5s...\t\t\t\t\t", label,
2770 + found?"true":"false", browsable?"true":"false");
2771 if(found != browsable) {
2772 fprintf(stderr, "FAILED\n");
2773 errors++;
@@ -2278,8 +2775,9 @@ static size_t check_name_value_deleted_flag(DICTIONARY *dict, NAME_VALUE *nv, co
2775 else
2776 fprintf(stderr, "OK\n");
2777
2281 - found = dictionary_walkthrough_read(dict, check_name_value_callback, nv->value);
2282 - fprintf(stderr, "NAME_VALUE walkthrough browsable %5s, expected %5s...\t\t", found?"true":"false", browsable?"true":"false");
2778 + found = dictionary_walkthrough_read(dict, check_item_callback, item->shared->value);
2779 + fprintf(stderr, "ITEM %-20s: walkthrough browsable %5s, expected %5s...\t\t\t\t", label,
2780 + found?"true":"false", browsable?"true":"false");
2781 if(found != browsable) {
2782 fprintf(stderr, "FAILED\n");
2783 errors++;
@@ -2287,8 +2785,9 @@ static size_t check_name_value_deleted_flag(DICTIONARY *dict, NAME_VALUE *nv, co
2785 else
2786 fprintf(stderr, "OK\n");
2787
2290 - found = dictionary_sorted_walkthrough_read(dict, check_name_value_callback, nv->value);
2291 - fprintf(stderr, "NAME_VALUE sorted walkthrough browsable %5s, expected %5s...\t", found?"true":"false", browsable?"true":"false");
2788 + found = dictionary_sorted_walkthrough_read(dict, check_item_callback, item->shared->value);
2789 + fprintf(stderr, "ITEM %-20s: sorted walkthrough browsable %5s, expected %5s...\t\t\t", label,
2790 + found?"true":"false", browsable?"true":"false");
2791 if(found != browsable) {
2792 fprintf(stderr, "FAILED\n");
2793 errors++;
@@ -2297,11 +2796,12 @@ static size_t check_name_value_deleted_flag(DICTIONARY *dict, NAME_VALUE *nv, co
2796 fprintf(stderr, "OK\n");
2797
2798 found = false;
2300 - NAME_VALUE *n;
2301 - for(n = dict->first_item; n ;n = n->next)
2302 - if(n == nv) found = true;
2799 + DICTIONARY_ITEM *n;
2800 + for(n = dict->items.list; n ;n = n->next)
2801 + if(n == item) found = true;
2802
2304 - fprintf(stderr, "NAME_VALUE linked %5s, expected %5s...\t\t\t\t", found?"true":"false", linked?"true":"false");
2803 + fprintf(stderr, "ITEM %-20s: linked %5s, expected %5s...\t\t\t\t\t\t", label,
2804 + found?"true":"false", linked?"true":"false");
2805 if(found != linked) {
2806 fprintf(stderr, "FAILED\n");
2807 errors++;
@@ -2312,27 +2812,349 @@ static size_t check_name_value_deleted_flag(DICTIONARY *dict, NAME_VALUE *nv, co
2812 return errors;
2813 }
2814
2315 -static int string_threads_join = 0;
2316 -static void *string_thread(void *arg __maybe_unused) {
2317 - int dups = 1; //(gettid() % 10);
2815 +struct thread_unittest {
2816 + int join;
2817 + DICTIONARY *dict;
2818 + int dups;
2819 +};
2820 +
2821 +static void *unittest_dict_thread(void *arg) {
2822 + struct thread_unittest *tu = arg;
2823 for(; 1 ;) {
2319 - if(string_threads_join)
2824 + if(__atomic_load_n(&tu->join, __ATOMIC_RELAXED))
2825 break;
2826
2322 - STRING *s = string_strdupz("string thread checking 1234567890");
2827 + DICT_ITEM_CONST DICTIONARY_ITEM *item =
2828 + dictionary_set_and_acquire_item_advanced(tu->dict, "dict thread checking 1234567890",
2829 + -1, NULL, 0, NULL);
2830 +
2831 +
2832 + dictionary_get(tu->dict, dictionary_acquired_item_name(item));
2833 +
2834 + void *t1;
2835 + dfe_start_write(tu->dict, t1) {
2836 +
2837 + // this should delete the referenced item
2838 + dictionary_del(tu->dict, t1_dfe.name);
2839 +
2840 + void *t2;
2841 + dfe_start_write(tu->dict, t2) {
2842 + // this should add another
2843 + dictionary_set(tu->dict, t2_dfe.name, NULL, 0);
2844 +
2845 + dictionary_get(tu->dict, dictionary_acquired_item_name(item));
2846 +
2847 + // and this should delete it again
2848 + dictionary_del(tu->dict, t2_dfe.name);
2849 + }
2850 + dfe_done(t2);
2851 +
2852 + // this should fail to add it
2853 + dictionary_set(tu->dict, t1_dfe.name, NULL, 0);
2854 + dictionary_del(tu->dict, t1_dfe.name);
2855 + }
2856 + dfe_done(t1);
2857 +
2858 + for(int i = 0; i < tu->dups ; i++) {
2859 + dictionary_acquired_item_dup(tu->dict, item);
2860 + dictionary_get(tu->dict, dictionary_acquired_item_name(item));
2861 + }
2862 +
2863 + for(int i = 0; i < tu->dups ; i++) {
2864 + dictionary_acquired_item_release(tu->dict, item);
2865 + dictionary_del(tu->dict, dictionary_acquired_item_name(item));
2866 + }
2867 +
2868 + dictionary_acquired_item_release(tu->dict, item);
2869 +
2870 + dictionary_del(tu->dict, "dict thread checking 1234567890");
2871 + }
2872 +
2873 + return arg;
2874 +}
2875 +
2876 +static int dictionary_unittest_threads() {
2877 +
2878 + struct thread_unittest tu = {
2879 + .join = 0,
2880 + .dict = NULL,
2881 + .dups = 1,
2882 + };
2883 +
2884 + // threads testing of dictionary
2885 + tu.dict = dictionary_create(DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
2886 + time_t seconds_to_run = 5;
2887 + int threads_to_create = 2;
2888 + fprintf(
2889 + stderr,
2890 + "\nChecking dictionary concurrency with %d threads for %ld seconds...\n",
2891 + threads_to_create,
2892 + seconds_to_run);
2893 +
2894 + netdata_thread_t threads[threads_to_create];
2895 + tu.join = 0;
2896 + for (int i = 0; i < threads_to_create; i++) {
2897 + char buf[100 + 1];
2898 + snprintf(buf, 100, "dict%d", i);
2899 + netdata_thread_create(
2900 + &threads[i],
2901 + buf,
2902 + NETDATA_THREAD_OPTION_DONT_LOG | NETDATA_THREAD_OPTION_JOINABLE,
2903 + unittest_dict_thread,
2904 + &tu);
2905 + }
2906 + sleep_usec(seconds_to_run * USEC_PER_SEC);
2907 +
2908 + __atomic_store_n(&tu.join, 1, __ATOMIC_RELAXED);
2909 + for (int i = 0; i < threads_to_create; i++) {
2910 + void *retval;
2911 + netdata_thread_join(threads[i], &retval);
2912 + }
2913 +
2914 + fprintf(stderr,
2915 + "inserts %zu"
2916 + ", deletes %zu"
2917 + ", searches %zu"
2918 + ", resets %zu"
2919 + ", entries %ld"
2920 + ", referenced_items %ld"
2921 + ", pending deletions %ld"
2922 + ", check spins %zu"
2923 + ", insert spins %zu"
2924 + ", search ignores %zu"
2925 + "\n",
2926 + tu.dict->stats->ops.inserts,
2927 + tu.dict->stats->ops.deletes,
2928 + tu.dict->stats->ops.searches,
2929 + tu.dict->stats->ops.resets,
2930 + tu.dict->entries,
2931 + tu.dict->referenced_items,
2932 + tu.dict->pending_deletion_items,
2933 + tu.dict->stats->spin_locks.use,
2934 + tu.dict->stats->spin_locks.insert,
2935 + tu.dict->stats->spin_locks.search
2936 + );
2937 + dictionary_destroy(tu.dict);
2938 + tu.dict = NULL;
2939 +
2940 + return 0;
2941 +}
2942 +
2943 +struct thread_view_unittest {
2944 + int join;
2945 + DICTIONARY *master;
2946 + DICTIONARY *view;
2947 + DICTIONARY_ITEM *item_master;
2948 + int dups;
2949 +};
2950 +
2951 +static void *unittest_dict_master_thread(void *arg) {
2952 + struct thread_view_unittest *tv = arg;
2953 +
2954 + while(!__atomic_load_n(&tv->join, __ATOMIC_SEQ_CST)) {
2955 + if(__atomic_load_n(&tv->item_master, __ATOMIC_SEQ_CST) != NULL)
2956 + continue;
2957 +
2958 + DICTIONARY_ITEM *item = dictionary_set_and_acquire_item(tv->master, "ITEM1", "123", strlen("123") + 1);
2959 + dictionary_acquired_item_dup(tv->master, item);
2960 + dictionary_del(tv->master, "ITEM1");
2961 +
2962 + __atomic_store_n(&tv->item_master, item, __ATOMIC_SEQ_CST);
2963 +
2964 + for(int i = 0; i < tv->dups ; i++) {
2965 + dictionary_acquired_item_dup(tv->master, item);
2966 + }
2967 +
2968 + for(int i = 0; i < tv->dups ; i++) {
2969 + dictionary_acquired_item_release(tv->master, item);
2970 + }
2971 +
2972 + dictionary_acquired_item_release(tv->master, item);
2973 + }
2974 +
2975 + return arg;
2976 +}
2977 +
2978 +static void *unittest_dict_view_thread(void *arg) {
2979 + struct thread_view_unittest *tv = arg;
2980 +
2981 + while(!__atomic_load_n(&tv->join, __ATOMIC_SEQ_CST)) {
2982 + DICTIONARY_ITEM *m_item = __atomic_load_n(&tv->item_master, __ATOMIC_SEQ_CST);
2983 + if(!m_item) continue;
2984 +
2985 + DICTIONARY_ITEM *v_item = dictionary_view_set_and_acquire_item(tv->view, "ITEM2", m_item);
2986 + dictionary_acquired_item_release(tv->master, m_item);
2987 + __atomic_store_n(&tv->item_master, NULL, __ATOMIC_SEQ_CST);
2988
2324 - for(int i = 0; i < dups ; i++)
2325 - string_dup(s);
2989 + for(int i = 0; i < tv->dups ; i++) {
2990 + dictionary_acquired_item_dup(tv->view, v_item);
2991 + }
2992 +
2993 + for(int i = 0; i < tv->dups ; i++) {
2994 + dictionary_acquired_item_release(tv->view, v_item);
2995 + }
2996
2327 - for(int i = 0; i < dups ; i++)
2328 - string_freez(s);
2997 + dictionary_del(tv->view, "ITEM2");
2998
2330 - string_freez(s);
2999 + dictionary_acquired_item_release(tv->view, v_item);
3000 }
3001
3002 return arg;
3003 }
3004
3005 +static int dictionary_unittest_view_threads() {
3006 +
3007 + struct thread_view_unittest tv = {
3008 + .join = 0,
3009 + .master = NULL,
3010 + .view = NULL,
3011 + .item_master = NULL,
3012 + .dups = 1,
3013 + };
3014 +
3015 + // threads testing of dictionary
3016 + struct dictionary_stats stats = {};
3017 + tv.master = dictionary_create_advanced(DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE, &stats);
3018 + tv.view = dictionary_create_view(tv.master);
3019 +
3020 + time_t seconds_to_run = 5;
3021 + fprintf(
3022 + stderr,
3023 + "\nChecking dictionary concurrency with 1 master and 1 view threads for %ld seconds...\n",
3024 + seconds_to_run);
3025 +
3026 + netdata_thread_t master_thread, view_thread;
3027 + tv.join = 0;
3028 +
3029 + netdata_thread_create(
3030 + &master_thread,
3031 + "master",
3032 + NETDATA_THREAD_OPTION_DONT_LOG | NETDATA_THREAD_OPTION_JOINABLE,
3033 + unittest_dict_master_thread,
3034 + &tv);
3035 +
3036 + netdata_thread_create(
3037 + &view_thread,
3038 + "view",
3039 + NETDATA_THREAD_OPTION_DONT_LOG | NETDATA_THREAD_OPTION_JOINABLE,
3040 + unittest_dict_view_thread,
3041 + &tv);
3042 +
3043 + sleep_usec(seconds_to_run * USEC_PER_SEC);
3044 +
3045 + __atomic_store_n(&tv.join, 1, __ATOMIC_RELAXED);
3046 + void *retval;
3047 + netdata_thread_join(view_thread, &retval);
3048 + netdata_thread_join(master_thread, &retval);
3049 +
3050 + fprintf(stderr,
3051 + "inserts %zu"
3052 + ", deletes %zu"
3053 + ", searches %zu"
3054 + ", resets %zu"
3055 + ", entries %ld (%ld on view)"
3056 + ", referenced_items %ld (%ld on view)"
3057 + ", pending deletions %ld (%ld on view)"
3058 + ", check spins %zu"
3059 + ", insert spins %zu"
3060 + ", search ignores %zu"
3061 + "\n",
3062 + stats.ops.inserts,
3063 + stats.ops.deletes,
3064 + stats.ops.searches,
3065 + stats.ops.resets,
3066 + tv.master->entries, tv.view->entries,
3067 + tv.master->referenced_items, tv.view->referenced_items,
3068 + tv.master->pending_deletion_items, tv.view->pending_deletion_items,
3069 + stats.spin_locks.use,
3070 + stats.spin_locks.insert,
3071 + stats.spin_locks.search
3072 + );
3073 + dictionary_destroy(tv.master);
3074 + dictionary_destroy(tv.view);
3075 +
3076 + return 0;
3077 +}
3078 +
3079 +size_t dictionary_unittest_views(void) {
3080 + size_t errors = 0;
3081 + struct dictionary_stats stats = {};
3082 + DICTIONARY *master = dictionary_create_advanced(DICT_OPTION_NONE, &stats);
3083 + DICTIONARY *view = dictionary_create_view(master);
3084 +
3085 + fprintf(stderr, "\n\nChecking dictionary views...\n");
3086 +
3087 + // Add an item to both master and view, then remove the view first and the master second
3088 + fprintf(stderr, "\nPASS 1: Adding 1 item to master:\n");
3089 + DICTIONARY_ITEM *item1_on_master = dictionary_set_and_acquire_item(master, "KEY 1", "VALUE1", strlen("VALUE1") + 1);
3090 + errors += unittest_check_dictionary("master", master, 1, 1, 0, 1, 0);
3091 + errors += unittest_check_item("master", master, item1_on_master, "KEY 1", item1_on_master->shared->value, 1, ITEM_FLAG_NONE, true, true, true);
3092 +
3093 + fprintf(stderr, "\nPASS 1: Adding master item to view:\n");
3094 + DICTIONARY_ITEM *item1_on_view = dictionary_view_set_and_acquire_item(view, "KEY 1 ON VIEW", item1_on_master);
3095 + errors += unittest_check_dictionary("view", view, 1, 1, 0, 1, 0);
3096 + errors += unittest_check_item("view", view, item1_on_view, "KEY 1 ON VIEW", item1_on_master->shared->value, 1, ITEM_FLAG_NONE, true, true, true);
3097 +
3098 + fprintf(stderr, "\nPASS 1: Deleting view item:\n");
3099 + dictionary_del(view, "KEY 1 ON VIEW");
3100 + errors += unittest_check_dictionary("master", master, 1, 1, 0, 1, 0);
3101 + errors += unittest_check_dictionary("view", view, 0, 0, 1, 1, 0);
3102 + errors += unittest_check_item("master", master, item1_on_master, "KEY 1", item1_on_master->shared->value, 1, ITEM_FLAG_NONE, true, true, true);
3103 + errors += unittest_check_item("view", view, item1_on_view, "KEY 1 ON VIEW", item1_on_master->shared->value, 1, ITEM_FLAG_DELETED, false, false, true);
3104 +
3105 + fprintf(stderr, "\nPASS 1: Releasing the deleted view item:\n");
3106 + dictionary_acquired_item_release(view, item1_on_view);
3107 + errors += unittest_check_dictionary("master", master, 1, 1, 0, 1, 0);
3108 + errors += unittest_check_dictionary("view", view, 0, 0, 1, 0, 1);
3109 + errors += unittest_check_item("master", master, item1_on_master, "KEY 1", item1_on_master->shared->value, 1, ITEM_FLAG_NONE, true, true, true);
3110 +
3111 + fprintf(stderr, "\nPASS 1: Releasing the acquired master item:\n");
3112 + dictionary_acquired_item_release(master, item1_on_master);
3113 + errors += unittest_check_dictionary("master", master, 1, 1, 0, 0, 0);
3114 + errors += unittest_check_dictionary("view", view, 0, 0, 1, 0, 1);
3115 + errors += unittest_check_item("master", master, item1_on_master, "KEY 1", item1_on_master->shared->value, 0, ITEM_FLAG_NONE, true, true, true);
3116 +
3117 + fprintf(stderr, "\nPASS 1: Deleting the released master item:\n");
3118 + dictionary_del(master, "KEY 1");
3119 + errors += unittest_check_dictionary("master", master, 0, 0, 0, 0, 0);
3120 + errors += unittest_check_dictionary("view", view, 0, 0, 1, 0, 1);
3121 +
3122 + // The other way now:
3123 + // Add an item to both master and view, then remove the master first and verify it is deleted on the view also
3124 + fprintf(stderr, "\nPASS 2: Adding 1 item to master:\n");
3125 + item1_on_master = dictionary_set_and_acquire_item(master, "KEY 1", "VALUE1", strlen("VALUE1") + 1);
3126 + errors += unittest_check_dictionary("master", master, 1, 1, 0, 1, 0);
3127 + errors += unittest_check_item("master", master, item1_on_master, "KEY 1", item1_on_master->shared->value, 1, ITEM_FLAG_NONE, true, true, true);
3128 +
3129 + fprintf(stderr, "\nPASS 2: Adding master item to view:\n");
3130 + item1_on_view = dictionary_view_set_and_acquire_item(view, "KEY 1 ON VIEW", item1_on_master);
3131 + errors += unittest_check_dictionary("view", view, 1, 1, 0, 1, 0);
3132 + errors += unittest_check_item("view", view, item1_on_view, "KEY 1 ON VIEW", item1_on_master->shared->value, 1, ITEM_FLAG_NONE, true, true, true);
3133 +
3134 + fprintf(stderr, "\nPASS 2: Deleting master item:\n");
3135 + dictionary_del(master, "KEY 1");
3136 + dictionary_version(view);
3137 + errors += unittest_check_dictionary("master", master, 0, 0, 1, 1, 0);
3138 + errors += unittest_check_dictionary("view", view, 0, 0, 1, 1, 0);
3139 + errors += unittest_check_item("master", master, item1_on_master, "KEY 1", item1_on_master->shared->value, 1, ITEM_FLAG_DELETED, false, false, true);
3140 + errors += unittest_check_item("view", view, item1_on_view, "KEY 1 ON VIEW", item1_on_master->shared->value, 1, ITEM_FLAG_DELETED, false, false, true);
3141 +
3142 + fprintf(stderr, "\nPASS 2: Releasing the acquired master item:\n");
3143 + dictionary_acquired_item_release(master, item1_on_master);
3144 + errors += unittest_check_dictionary("master", master, 0, 0, 1, 0, 1);
3145 + errors += unittest_check_dictionary("view", view, 0, 0, 1, 1, 0);
3146 + errors += unittest_check_item("view", view, item1_on_view, "KEY 1 ON VIEW", item1_on_master->shared->value, 1, ITEM_FLAG_DELETED, false, false, true);
3147 +
3148 + fprintf(stderr, "\nPASS 2: Releasing the deleted view item:\n");
3149 + dictionary_acquired_item_release(view, item1_on_view);
3150 + errors += unittest_check_dictionary("master", master, 0, 0, 1, 0, 1);
3151 + errors += unittest_check_dictionary("view", view, 0, 0, 1, 0, 1);
3152 +
3153 + dictionary_destroy(master);
3154 + dictionary_destroy(view);
3155 + return errors;
3156 +}
3157 +
3158 int dictionary_unittest(size_t entries) {
3159 if(entries < 10) entries = 10;
3160
@@ -2344,23 +3166,28 @@ int dictionary_unittest(size_t entries) {
3166 char **values = dictionary_unittest_generate_values(entries);
3167
3168 fprintf(stderr, "\nCreating dictionary single threaded, clone, %zu items\n", entries);
2347 - dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
3169 + dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
3170 dictionary_unittest_clone(dict, names, values, entries, &errors);
3171
3172 fprintf(stderr, "\nCreating dictionary multi threaded, clone, %zu items\n", entries);
2351 - dict = dictionary_create(DICTIONARY_FLAG_NONE);
3173 + dict = dictionary_create(DICT_OPTION_NONE);
3174 dictionary_unittest_clone(dict, names, values, entries, &errors);
3175
3176 fprintf(stderr, "\nCreating dictionary single threaded, non-clone, add-in-front options, %zu items\n", entries);
2355 - dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED|DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_ADD_IN_FRONT);
3177 + dict = dictionary_create(
3178 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE |
3179 + DICT_OPTION_ADD_IN_FRONT);
3180 dictionary_unittest_nonclone(dict, names, values, entries, &errors);
3181
3182 fprintf(stderr, "\nCreating dictionary multi threaded, non-clone, add-in-front options, %zu items\n", entries);
2359 - dict = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_ADD_IN_FRONT);
3183 + dict = dictionary_create(
3184 + DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_ADD_IN_FRONT);
3185 dictionary_unittest_nonclone(dict, names, values, entries, &errors);
3186
3187 fprintf(stderr, "\nCreating dictionary single-threaded, non-clone, don't overwrite options, %zu items\n", entries);
2363 - dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED|DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
3188 + dict = dictionary_create(
3189 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE |
3190 + DICT_OPTION_DONT_OVERWRITE_VALUE);
3191 dictionary_unittest_run_and_measure_time(dict, "adding entries", names, values, entries, &errors, dictionary_unittest_set_nonclone);
3192 dictionary_unittest_run_and_measure_time(dict, "resetting non-overwrite entries", names, values, entries, &errors, dictionary_unittest_reset_dont_overwrite_nonclone);
3193 dictionary_unittest_run_and_measure_time(dict, "traverse foreach read loop", names, values, entries, &errors, dictionary_unittest_foreach);
@@ -2369,13 +3196,15 @@ int dictionary_unittest(size_t entries) {
3196 dictionary_unittest_run_and_measure_time(dict, "destroying full dictionary", names, values, entries, &errors, dictionary_unittest_destroy);
3197
3198 fprintf(stderr, "\nCreating dictionary multi-threaded, non-clone, don't overwrite options, %zu items\n", entries);
2372 - dict = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
3199 + dict = dictionary_create(
3200 + DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
3201 dictionary_unittest_run_and_measure_time(dict, "adding entries", names, values, entries, &errors, dictionary_unittest_set_nonclone);
3202 dictionary_unittest_run_and_measure_time(dict, "walkthrough write delete this", names, values, entries, &errors, dictionary_unittest_walkthrough_delete_this);
3203 dictionary_unittest_run_and_measure_time(dict, "destroying empty dictionary", names, values, entries, &errors, dictionary_unittest_destroy);
3204
3205 fprintf(stderr, "\nCreating dictionary multi-threaded, non-clone, don't overwrite options, %zu items\n", entries);
2378 - dict = dictionary_create(DICTIONARY_FLAG_NAME_LINK_DONT_CLONE|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE|DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
3206 + dict = dictionary_create(
3207 + DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
3208 dictionary_unittest_run_and_measure_time(dict, "adding entries", names, values, entries, &errors, dictionary_unittest_set_nonclone);
3209 dictionary_unittest_run_and_measure_time(dict, "foreach write delete this", names, values, entries, &errors, dictionary_unittest_foreach_delete_this);
3210 dictionary_unittest_run_and_measure_time(dict, "traverse foreach read loop empty", names, values, 0, &errors, dictionary_unittest_foreach);
@@ -2383,265 +3212,99 @@ int dictionary_unittest(size_t entries) {
3212 dictionary_unittest_run_and_measure_time(dict, "destroying empty dictionary", names, values, entries, &errors, dictionary_unittest_destroy);
3213
3214 fprintf(stderr, "\nCreating dictionary single threaded, clone, %zu items\n", entries);
2386 - dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
3215 + dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
3216 dictionary_unittest_sorting(dict, names, values, entries, &errors);
3217 dictionary_unittest_run_and_measure_time(dict, "destroying full dictionary", names, values, entries, &errors, dictionary_unittest_destroy);
3218
3219 fprintf(stderr, "\nCreating dictionary single threaded, clone, %zu items\n", entries);
2391 - dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
3220 + dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
3221 dictionary_unittest_null_dfe(dict, names, values, entries, &errors);
3222 dictionary_unittest_run_and_measure_time(dict, "destroying full dictionary", names, values, entries, &errors, dictionary_unittest_destroy);
3223
3224 fprintf(stderr, "\nCreating dictionary single threaded, noclone, %zu items\n", entries);
2396 - dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE);
3225 + dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_VALUE_LINK_DONT_CLONE);
3226 dictionary_unittest_null_dfe(dict, names, values, entries, &errors);
3227 dictionary_unittest_run_and_measure_time(dict, "destroying full dictionary", names, values, entries, &errors, dictionary_unittest_destroy);
3228
3229 // check reference counters
3230 {
3231 fprintf(stderr, "\nTesting reference counters:\n");
2403 - dict = dictionary_create(DICTIONARY_FLAG_NONE|DICTIONARY_FLAG_NAME_LINK_DONT_CLONE);
2404 - errors += check_dictionary(dict, 0, 0);
3232 + dict = dictionary_create(DICT_OPTION_NONE | DICT_OPTION_NAME_LINK_DONT_CLONE);
3233 + errors += unittest_check_dictionary("", dict, 0, 0, 0, 0, 0);
3234
3235 fprintf(stderr, "\nAdding test item to dictionary and acquiring it\n");
3236 dictionary_set(dict, "test", "ITEM1", 6);
2408 - NAME_VALUE *nv = (NAME_VALUE *)dictionary_get_and_acquire_item(dict, "test");
3237 + DICTIONARY_ITEM *item = (DICTIONARY_ITEM *)dictionary_get_and_acquire_item(dict, "test");
3238
2410 - errors += check_dictionary(dict, 1, 1);
2411 - errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_NONE, true, true, true);
3239 + errors += unittest_check_dictionary("", dict, 1, 1, 0, 1, 0);
3240 + errors += unittest_check_item("ACQUIRED", dict, item, "test", "ITEM1", 1, ITEM_FLAG_NONE, true, true, true);
3241
3242 fprintf(stderr, "\nChecking that reference counters are increased:\n");
3243 void *t;
3244 dfe_start_read(dict, t) {
2416 - errors += check_dictionary(dict, 1, 1);
2417 - errors +=
2418 - check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 2, NAME_VALUE_FLAG_NONE, true, true, true);
3245 + errors += unittest_check_dictionary("", dict, 1, 1, 0, 1, 0);
3246 + errors += unittest_check_item("ACQUIRED TRAVERSAL", dict, item, "test", "ITEM1", 2, ITEM_FLAG_NONE, true, true, true);
3247 }
3248 dfe_done(t);
3249
3250 fprintf(stderr, "\nChecking that reference counters are decreased:\n");
2423 - errors += check_dictionary(dict, 1, 1);
2424 - errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_NONE, true, true, true);
3251 + errors += unittest_check_dictionary("", dict, 1, 1, 0, 1, 0);
3252 + errors += unittest_check_item("ACQUIRED TRAVERSAL 2", dict, item, "test", "ITEM1", 1, ITEM_FLAG_NONE, true, true, true);
3253
3254 fprintf(stderr, "\nDeleting the item we have acquired:\n");
3255 dictionary_del(dict, "test");
3256
2429 - errors += check_dictionary(dict, 0, 1);
2430 - errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
3257 + errors += unittest_check_dictionary("", dict, 0, 0, 1, 1, 0);
3258 + errors += unittest_check_item("DELETED", dict, item, "test", "ITEM1", 1, ITEM_FLAG_DELETED, false, false, true);
3259
3260 fprintf(stderr, "\nAdding another item with the same name of the item we deleted, while being acquired:\n");
3261 dictionary_set(dict, "test", "ITEM2", 6);
2434 - errors += check_dictionary(dict, 1, 2);
3262 + errors += unittest_check_dictionary("", dict, 1, 1, 1, 1, 0);
3263
3264 fprintf(stderr, "\nAcquiring the second item:\n");
2437 - NAME_VALUE *nv2 = (NAME_VALUE *)dictionary_get_and_acquire_item(dict, "test");
2438 - errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
2439 - errors += check_name_value_deleted_flag(dict, nv2, "test", "ITEM2", 1, NAME_VALUE_FLAG_NONE, true, true, true);
3265 + DICTIONARY_ITEM *item2 = (DICTIONARY_ITEM *)dictionary_get_and_acquire_item(dict, "test");
3266 + errors += unittest_check_item("FIRST", dict, item, "test", "ITEM1", 1, ITEM_FLAG_DELETED, false, false, true);
3267 + errors += unittest_check_item("SECOND", dict, item2, "test", "ITEM2", 1, ITEM_FLAG_NONE, true, true, true);
3268 + errors += unittest_check_dictionary("", dict, 1, 1, 1, 2, 0);
3269
3270 fprintf(stderr, "\nReleasing the second item (the first is still acquired):\n");
2442 - dictionary_acquired_item_release(dict, (DICTIONARY_ITEM *)nv2);
2443 - errors += check_dictionary(dict, 1, 2);
2444 - errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
2445 - errors += check_name_value_deleted_flag(dict, nv2, "test", "ITEM2", 0, NAME_VALUE_FLAG_NONE, true, true, true);
3271 + dictionary_acquired_item_release(dict, (DICTIONARY_ITEM *)item2);
3272 + errors += unittest_check_dictionary("", dict, 1, 1, 1, 1, 0);
3273 + errors += unittest_check_item("FIRST", dict, item, "test", "ITEM1", 1, ITEM_FLAG_DELETED, false, false, true);
3274 + errors += unittest_check_item("SECOND RELEASED", dict, item2, "test", "ITEM2", 0, ITEM_FLAG_NONE, true, true, true);
3275
3276 fprintf(stderr, "\nDeleting the second item (the first is still acquired):\n");
3277 dictionary_del(dict, "test");
2449 - errors += check_dictionary(dict, 0, 1);
2450 - errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
3278 + errors += unittest_check_dictionary("", dict, 0, 0, 1, 1, 0);
3279 + errors += unittest_check_item("ACQUIRED DELETED", dict, item, "test", "ITEM1", 1, ITEM_FLAG_DELETED, false, false, true);
3280
3281 fprintf(stderr, "\nReleasing the first item (which we have already deleted):\n");
2453 - dictionary_acquired_item_release(dict, (DICTIONARY_ITEM *)nv);
2454 - errors += check_dictionary(dict, 0, 0);
3282 + dictionary_acquired_item_release(dict, (DICTIONARY_ITEM *)item);
3283 + dfe_start_write(dict, item) ; dfe_done(item);
3284 + errors += unittest_check_dictionary("", dict, 0, 0, 1, 0, 1);
3285
3286 fprintf(stderr, "\nAdding again the test item to dictionary and acquiring it\n");
3287 dictionary_set(dict, "test", "ITEM1", 6);
2458 - nv = (NAME_VALUE *)dictionary_get_and_acquire_item(dict, "test");
3288 + item = (DICTIONARY_ITEM *)dictionary_get_and_acquire_item(dict, "test");
3289
2460 - errors += check_dictionary(dict, 1, 1);
2461 - errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_NONE, true, true, true);
3290 + errors += unittest_check_dictionary("", dict, 1, 1, 0, 1, 0);
3291 + errors += unittest_check_item("RE-ADDITION", dict, item, "test", "ITEM1", 1, ITEM_FLAG_NONE, true, true, true);
3292
3293 fprintf(stderr, "\nDestroying the dictionary while we have acquired an item\n");
3294 dictionary_destroy(dict);
3295
3296 fprintf(stderr, "Releasing the item (on a destroyed dictionary)\n");
2467 - dictionary_acquired_item_release(dict, (DICTIONARY_ITEM *)nv);
2468 - nv = NULL;
3297 + dictionary_acquired_item_release(dict, (DICTIONARY_ITEM *)item);
3298 + item = NULL;
3299 dict = NULL;
3300 }
3301
2472 - // check string
2473 - {
2474 - long int string_entries_starting = string_base.entries;
2475 -
2476 - fprintf(stderr, "\nChecking strings...\n");
2477 -
2478 - STRING *s1 = string_strdupz("hello unittest");
2479 - STRING *s2 = string_strdupz("hello unittest");
2480 - if(s1 != s2) {
2481 - errors++;
2482 - fprintf(stderr, "ERROR: duplicating strings are not deduplicated\n");
2483 - }
2484 - else
2485 - fprintf(stderr, "OK: duplicating string are deduplicated\n");
2486 -
2487 - STRING *s3 = string_dup(s1);
2488 - if(s3 != s1) {
2489 - errors++;
2490 - fprintf(stderr, "ERROR: cloning strings are not deduplicated\n");
2491 - }
2492 - else
2493 - fprintf(stderr, "OK: cloning string are deduplicated\n");
2494 -
2495 - if(s1->refcount != 3) {
2496 - errors++;
2497 - fprintf(stderr, "ERROR: string refcount is not 3\n");
2498 - }
2499 - else
2500 - fprintf(stderr, "OK: string refcount is 3\n");
2501 -
2502 - STRING *s4 = string_strdupz("world unittest");
2503 - if(s4 == s1) {
2504 - errors++;
2505 - fprintf(stderr, "ERROR: string is sharing pointers on different strings\n");
2506 - }
2507 - else
2508 - fprintf(stderr, "OK: string is properly handling different strings\n");
2509 -
2510 - usec_t start_ut, end_ut;
2511 - STRING **strings = mallocz(entries * sizeof(STRING *));
2512 -
2513 - start_ut = now_realtime_usec();
2514 - for(size_t i = 0; i < entries ;i++) {
2515 - strings[i] = string_strdupz(names[i]);
2516 - }
2517 - end_ut = now_realtime_usec();
2518 - fprintf(stderr, "Created %zu strings in %llu usecs\n", entries, end_ut - start_ut);
2519 -
2520 - start_ut = now_realtime_usec();
2521 - for(size_t i = 0; i < entries ;i++) {
2522 - strings[i] = string_dup(strings[i]);
2523 - }
2524 - end_ut = now_realtime_usec();
2525 - fprintf(stderr, "Cloned %zu strings in %llu usecs\n", entries, end_ut - start_ut);
2526 -
2527 - start_ut = now_realtime_usec();
2528 - for(size_t i = 0; i < entries ;i++) {
2529 - string_freez(strings[i]);
2530 - string_freez(strings[i]);
2531 - }
2532 - end_ut = now_realtime_usec();
2533 - fprintf(stderr, "Freed %zu strings in %llu usecs\n", entries, end_ut - start_ut);
2534 -
2535 - freez(strings);
2536 -
2537 - if(string_base.entries != string_entries_starting + 2) {
2538 - errors++;
2539 - fprintf(stderr, "ERROR: strings dictionary should have %ld items but it has %ld\n", string_entries_starting + 2, string_base.entries);
2540 - }
2541 - else
2542 - fprintf(stderr, "OK: strings dictionary has 2 items\n");
2543 - }
2544 -
2545 - // check 2-way merge
2546 - {
2547 - struct testcase {
2548 - char *src1; char *src2; char *expected;
2549 - } tests[] = {
2550 - { "", "", ""},
2551 - { "a", "", "[x]"},
2552 - { "", "a", "[x]"},
2553 - { "a", "a", "a"},
2554 - { "abcd", "abcd", "abcd"},
2555 - { "foo_cs", "bar_cs", "[x]_cs"},
2556 - { "cp_UNIQUE_INFIX_cs", "cp_unique_infix_cs", "cp_[x]_cs"},
2557 - { "cp_UNIQUE_INFIX_ci_unique_infix_cs", "cp_unique_infix_ci_UNIQUE_INFIX_cs", "cp_[x]_cs"},
2558 - { "foo[1234]", "foo[4321]", "foo[[x]]"},
2559 - { NULL, NULL, NULL },
2560 - };
2561 -
2562 - for (struct testcase *tc = &tests[0]; tc->expected != NULL; tc++) {
2563 - STRING *src1 = string_strdupz(tc->src1);
2564 - STRING *src2 = string_strdupz(tc->src2);
2565 - STRING *expected = string_strdupz(tc->expected);
2566 -
2567 - STRING *result = string_2way_merge(src1, src2);
2568 - if (string_cmp(result, expected) != 0) {
2569 - fprintf(stderr, "string_2way_merge(\"%s\", \"%s\") -> \"%s\" (expected=\"%s\")\n",
2570 - string2str(src1),
2571 - string2str(src2),
2572 - string2str(result),
2573 - string2str(expected));
2574 - errors++;
2575 - }
2576 -
2577 - string_freez(src1);
2578 - string_freez(src2);
2579 - string_freez(expected);
2580 - string_freez(result);
2581 - }
2582 - }
2583 -
3302 dictionary_unittest_free_char_pp(names, entries);
3303 dictionary_unittest_free_char_pp(values, entries);
3304
2587 - {
2588 -#ifdef NETDATA_INTERNAL_CHECKS
2589 - size_t ofound_deleted_on_search = string_base.found_deleted_on_search,
2590 - ofound_available_on_search = string_base.found_available_on_search,
2591 - ofound_deleted_on_insert = string_base.found_deleted_on_insert,
2592 - ofound_available_on_insert = string_base.found_available_on_insert,
2593 - ospins = string_base.spins;
2594 -#endif
2595 -
2596 - size_t oinserts, odeletes, osearches, oentries, oreferences, omemory, oduplications, oreleases;
2597 - string_statistics(&oinserts, &odeletes, &osearches, &oentries, &oreferences, &omemory, &oduplications, &oreleases);
2598 -
2599 - time_t seconds_to_run = 5;
2600 - int threads_to_create = 2;
2601 - fprintf(
2602 - stderr,
2603 - "Checking string concurrency with %d threads for %ld seconds...\n",
2604 - threads_to_create,
2605 - seconds_to_run);
2606 - // check string concurrency
2607 - netdata_thread_t threads[threads_to_create];
2608 - string_threads_join = 0;
2609 - for (int i = 0; i < threads_to_create; i++) {
2610 - char buf[100 + 1];
2611 - snprintf(buf, 100, "string%d", i);
2612 - netdata_thread_create(
2613 - &threads[i], buf, NETDATA_THREAD_OPTION_DONT_LOG | NETDATA_THREAD_OPTION_JOINABLE, string_thread, NULL);
2614 - }
2615 - sleep_usec(seconds_to_run * USEC_PER_SEC);
2616 -
2617 - string_threads_join = 1;
2618 - for (int i = 0; i < threads_to_create; i++) {
2619 - void *retval;
2620 - netdata_thread_join(threads[i], &retval);
2621 - }
2622 -
2623 - size_t inserts, deletes, searches, sentries, references, memory, duplications, releases;
2624 - string_statistics(&inserts, &deletes, &searches, &sentries, &references, &memory, &duplications, &releases);
2625 -
2626 - fprintf(stderr, "inserts %zu, deletes %zu, searches %zu, entries %zu, references %zu, memory %zu, duplications %zu, releases %zu\n",
2627 - inserts - oinserts, deletes - odeletes, searches - osearches, sentries - oentries, references - oreferences, memory - omemory, duplications - oduplications, releases - oreleases);
2628 -
2629 -#ifdef NETDATA_INTERNAL_CHECKS
2630 - size_t found_deleted_on_search = string_base.found_deleted_on_search,
2631 - found_available_on_search = string_base.found_available_on_search,
2632 - found_deleted_on_insert = string_base.found_deleted_on_insert,
2633 - found_available_on_insert = string_base.found_available_on_insert,
2634 - spins = string_base.spins;
2635 -
2636 - fprintf(stderr, "on insert: %zu ok + %zu deleted\non search: %zu ok + %zu deleted\nspins: %zu\n",
2637 - found_available_on_insert - ofound_available_on_insert,
2638 - found_deleted_on_insert - ofound_deleted_on_insert,
2639 - found_available_on_search - ofound_available_on_search,
2640 - found_deleted_on_search - ofound_deleted_on_search,
2641 - spins - ospins
2642 - );
2643 -#endif
2644 - }
3305 + errors += dictionary_unittest_views();
3306 + errors += dictionary_unittest_threads();
3307 + errors += dictionary_unittest_view_threads();
3308
3309 fprintf(stderr, "\n%zu errors found\n", errors);
3310 return errors ? 1 : 0;
libnetdata/dictionary/dictionary.h
+167 -116
@@ -13,19 +13,19 @@
13 * Names and Values in the dictionary can be cloned or linked.
14 * In clone mode, the dictionary does all the memory management.
15 * The default is clone for both names and values.
16 - * Set DICTIONARY_FLAG_NAME_LINK_DONT_CLONE to link names.
17 - * Set DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE to link names.
16 + * Set DICT_OPTION_NAME_LINK_DONT_CLONE to link names.
17 + * Set DICT_OPTION_VALUE_LINK_DONT_CLONE to link names.
18 *
19 * ORDERED
20 * Items are ordered in the order they are added (new items are appended at the end).
21 - * You may reverse the order by setting the flag DICTIONARY_FLAG_ADD_IN_FRONT.
21 + * You may reverse the order by setting the flag DICT_OPTION_ADD_IN_FRONT.
22 *
23 * LOOKUP
24 * The dictionary uses JudyHS to maintain a very fast randomly accessible hash table.
25 *
26 * MULTI-THREADED and SINGLE-THREADED
27 * Each dictionary may be single threaded (no locks), or multi-threaded (multiple readers or one writer).
28 - * The default is multi-threaded. Add the flag DICTIONARY_FLAG_SINGLE_THREADED for single-threaded.
28 + * The default is multi-threaded. Add the flag DICT_OPTION_SINGLE_THREADED for single-threaded.
29 *
30 * WALK-THROUGH and FOREACH traversal
31 * The dictionary can be traversed on read or write mode, either with a callback (walkthrough) or with
@@ -35,110 +35,184 @@
35 *
36 */
37
38 +#ifdef DICTIONARY_INTERNALS
39 +#define DICTFE_CONST
40 +#define DICT_ITEM_CONST
41 +#else
42 +#define DICTFE_CONST const
43 +#define DICT_ITEM_CONST const
44 +#endif
45 +
46 typedef struct dictionary DICTIONARY;
47 typedef struct dictionary_item DICTIONARY_ITEM;
48
41 -typedef enum dictionary_flags {
42 - DICTIONARY_FLAG_NONE = 0, // the default is the opposite of all below
43 - DICTIONARY_FLAG_SINGLE_THREADED = (1 << 0), // don't use any locks (default: use locks)
44 - DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE = (1 << 1), // don't copy the value, just point to the one provided (default: copy)
45 - DICTIONARY_FLAG_NAME_LINK_DONT_CLONE = (1 << 2), // don't copy the name, just point to the one provided (default: copy)
46 - DICTIONARY_FLAG_DONT_OVERWRITE_VALUE = (1 << 3), // don't overwrite values of dictionary items (default: overwrite)
47 - DICTIONARY_FLAG_ADD_IN_FRONT = (1 << 4), // add dictionary items at the front of the linked list (default: at the end)
48 -
49 - // to change the value of the following, you also need to change the corresponding #defines in dictionary.c
50 - DICTIONARY_FLAG_RESERVED1 = (1 << 28), // reserved for DICTIONARY_FLAG_EXCLUSIVE_ACCESS
51 - DICTIONARY_FLAG_RESERVED2 = (1 << 29), // reserved for DICTIONARY_FLAG_DESTROYED
52 - DICTIONARY_FLAG_RESERVED3 = (1 << 30), // reserved for DICTIONARY_FLAG_DEFER_ALL_DELETIONS
53 -} DICTIONARY_FLAGS;
49 +typedef enum dictionary_options {
50 + DICT_OPTION_NONE = 0, // the default is the opposite of all below
51 + DICT_OPTION_SINGLE_THREADED = (1 << 0), // don't use any locks (default: use locks)
52 + DICT_OPTION_VALUE_LINK_DONT_CLONE = (1 << 1), // don't copy the value, just point to the one provided (default: copy)
53 + DICT_OPTION_NAME_LINK_DONT_CLONE = (1 << 2), // don't copy the name, just point to the one provided (default: copy)
54 + DICT_OPTION_DONT_OVERWRITE_VALUE = (1 << 3), // don't overwrite values of dictionary items (default: overwrite)
55 + DICT_OPTION_ADD_IN_FRONT = (1 << 4), // add dictionary items at the front of the linked list (default: at the end)
56 +} DICT_OPTIONS;
57 +
58 +struct dictionary_stats {
59 + const char *name; // the name of the category
60 +
61 + struct {
62 + size_t active; // the number of active dictionaries
63 + size_t deleted; // the number of dictionaries queued for destruction
64 + } dictionaries;
65 +
66 + struct {
67 + long entries; // active items in the dictionary
68 + long pending_deletion; // pending deletion items in the dictionary
69 + long referenced; // referenced items in the dictionary
70 + } items;
71 +
72 + struct {
73 + size_t creations; // dictionary creations
74 + size_t destructions; // dictionary destructions
75 + size_t flushes; // dictionary flushes
76 + size_t traversals; // dictionary foreach
77 + size_t walkthroughs; // dictionary walkthrough
78 + size_t garbage_collections; // dictionary garbage collections
79 + size_t searches; // item searches
80 + size_t inserts; // item inserts
81 + size_t resets; // item resets
82 + size_t deletes; // item deletes
83 + } ops;
84 +
85 + struct {
86 + size_t inserts; // number of times the insert callback is called
87 + size_t conflicts; // number of times the insert conflict is called
88 + size_t reacts; // number of times the insert react is called
89 + size_t deletes; // number of times the insert delete is called
90 + } callbacks;
91 +
92 + // memory
93 + struct {
94 + long indexed; // bytes of keys indexed (indication of the index size)
95 + long values; // bytes of caller structures
96 + long dict; // bytes of the structures dictionary needs
97 + } memory;
98 +
99 + // spin locks
100 + struct {
101 + size_t use; // number of times a reference to item had to spin to acquire it or ignore it
102 + size_t search; // number of times a successful search result had to be thrown away
103 + size_t insert; // number of times an insertion to the hash table had to be repeated
104 + } spin_locks;
105 +};
106
107 // Create a dictionary
108 #ifdef NETDATA_INTERNAL_CHECKS
57 -#define dictionary_create(flags) dictionary_create_advanced_with_trace(flags, 0, __FUNCTION__, __LINE__, __FILE__);
58 -#define dictionary_create_advanced(flags) dictionary_create_advanced_with_trace(flags, 0, __FUNCTION__, __LINE__, __FILE__);
59 -extern DICTIONARY *dictionary_create_advanced_with_trace(DICTIONARY_FLAGS flags, size_t scratchpad_size, const char *function, size_t line, const char *file);
109 +#define dictionary_create(options) dictionary_create_advanced_with_trace(options, NULL, __FUNCTION__, __LINE__, __FILE__)
110 +#define dictionary_create_advanced(options, stats) dictionary_create_advanced_with_trace(options, stats, __FUNCTION__, __LINE__, __FILE__)
111 +extern DICTIONARY *dictionary_create_advanced_with_trace(DICT_OPTIONS options, struct dictionary_stats *stats, const char *function, size_t line, const char *file);
112 #else
61 -#define dictionary_create(flags) dictionary_create_advanced(flags, 0);
62 -extern DICTIONARY *dictionary_create_advanced(DICTIONARY_FLAGS flags, size_t scratchpad_size);
113 +#define dictionary_create(options) dictionary_create_advanced(options, NULL);
114 +extern DICTIONARY *dictionary_create_advanced(DICT_OPTIONS options, struct dictionary_stats *stats);
115 #endif
116
65 -extern void *dictionary_scratchpad(DICTIONARY *dict);
117 +// Create a view on a dictionary
118 +#ifdef NETDATA_INTERNAL_CHECKS
119 +#define dictionary_create_view(master) dictionary_create_view_with_trace(master, __FUNCTION__, __LINE__, __FILE__)
120 +extern DICTIONARY *dictionary_create_view_with_trace(DICTIONARY *master, const char *function, size_t line, const char *file);
121 +#else
122 +extern DICTIONARY *dictionary_create_view(DICTIONARY *master);
123 +#endif
124
125 // an insert callback to be called just after an item is added to the dictionary
126 // this callback is called while the dictionary is write locked!
69 -extern void dictionary_register_insert_callback(DICTIONARY *dict, void (*ins_callback)(const char *name, void *value, void *data), void *data);
127 +extern void dictionary_register_insert_callback(DICTIONARY *dict, void (*ins_callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data);
128
129 // a delete callback to be called just before an item is deleted forever
130 // this callback is called while the dictionary is write locked!
73 -extern void dictionary_register_delete_callback(DICTIONARY *dict, void (*del_callback)(const char *name, void *value, void *data), void *data);
131 +extern void dictionary_register_delete_callback(DICTIONARY *dict, void (*del_callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data);
132
75 -// a merge callback to be called when DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
133 +// a merge callback to be called when DICT_OPTION_DONT_OVERWRITE_VALUE
134 // and an item is already found in the dictionary - the dictionary does nothing else in this case
135 // the old_value will remain in the dictionary - the new_value is ignored
78 -extern void dictionary_register_conflict_callback(DICTIONARY *dict, void (*conflict_callback)(const char *name, void *old_value, void *new_value, void *data), void *data);
136 +// The callback should return true if the value has been updated (it increases the dictionary version).
137 +extern void dictionary_register_conflict_callback(DICTIONARY *dict, bool (*conflict_callback)(const DICTIONARY_ITEM *item, void *old_value, void *new_value, void *data), void *data);
138
139 // a reaction callback to be called after every item insertion or conflict
140 // after the constructors have finished and the items are fully available for use
141 // and the dictionary is not write locked anymore
83 -extern void dictionary_register_react_callback(DICTIONARY *dict, void (*react_callback)(const char *name, void *value, void *data), void *data);
142 +extern void dictionary_register_react_callback(DICTIONARY *dict, void (*react_callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data);
143
144 // Destroy a dictionary
86 -// returns the number of bytes freed
87 -// the returned value will not include name and value sizes if DICTIONARY_FLAG_WITH_STATISTICS is not set
145 +// Returns the number of bytes freed
146 +// The returned value will not include name/key sizes
147 +// Registered delete callbacks will be run for each item in the dictionary.
148 extern size_t dictionary_destroy(DICTIONARY *dict);
149
150 +// Empties a dictionary
151 +// Referenced items will survive, but are not offered anymore.
152 +// Registered delete callbacks will be run for each item in the dictionary.
153 +extern void dictionary_flush(DICTIONARY *dict);
154 +
155 +extern void dictionary_version_increment(DICTIONARY *dict);
156 +
157 +// ----------------------------------------------------------------------------
158 // Set an item in the dictionary
159 +//
160 // - if an item with the same name does not exist, create one
161 // - if an item with the same name exists, then:
93 -// a) if DICTIONARY_FLAG_DONT_OVERWRITE_VALUE is set, just return the existing value (ignore the new value)
162 +// a) if DICT_OPTION_DONT_OVERWRITE_VALUE is set, just return the existing value (ignore the new value)
163 // else b) reset the value to the new value passed at the call
164 //
96 -// When DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE is set, the value is linked, otherwise it is copied
97 -// When DICTIONARY_FLAG_NAME_LINK_DONT_CLONE is set, the name is linked, otherwise it is copied
165 +// When DICT_OPTION_VALUE_LINK_DONT_CLONE is set, the value is linked, otherwise it is copied
166 +// When DICT_OPTION_NAME_LINK_DONT_CLONE is set, the name is linked, otherwise it is copied
167 //
99 -// When neither DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE nor DICTIONARY_FLAG_NAME_LINK_DONT_CLONE are set, all the
168 +// When neither DICT_OPTION_VALUE_LINK_DONT_CLONE nor DICT_OPTION_NAME_LINK_DONT_CLONE are set, all the
169 // memory management for names and values is done by the dictionary.
170 //
171 // Passing NULL as value, the dictionary will callocz() the newly allocated value, otherwise it will copy it.
172 // Passing 0 as value_len, the dictionary will set the value to NULL (no allocations for value will be made).
104 -extern void *dictionary_set(DICTIONARY *dict, const char *name, void *value, size_t value_len);
173 +#define dictionary_set(dict, name, value, value_len) dictionary_set_advanced(dict, name, -1, value, value_len, NULL)
174 +extern void *dictionary_set_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, void *value, size_t value_len, void *constructor_data);
175
176 +#define dictionary_set_and_acquire_item(dict, name, value, value_len) dictionary_set_and_acquire_item_advanced(dict, name, -1, value, value_len, NULL)
177 +extern DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_set_and_acquire_item_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, void *value, size_t value_len, void *constructor_data);
178 +
179 +// set an item in a dictionary view
180 +#define dictionary_view_set_and_acquire_item(dict, name, master_item) dictionary_view_set_and_acquire_item_advanced(dict, name, -1, master_item)
181 +extern DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_view_set_and_acquire_item_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, DICTIONARY_ITEM *master_item);
182 +#define dictionary_view_set(dict, name, master_item) dictionary_view_set_advanced(dict, name, -1, master_item)
183 +extern void *dictionary_view_set_advanced(DICTIONARY *dict, const char *name, ssize_t name_len, DICTIONARY_ITEM *master_item);
184 +
185 +// ----------------------------------------------------------------------------
186 // Get an item from the dictionary
187 // If it returns NULL, the item is not found
108 -extern void *dictionary_get(DICTIONARY *dict, const char *name);
188
189 +#define dictionary_get(dict, name) dictionary_get_advanced(dict, name, -1)
190 +extern void *dictionary_get_advanced(DICTIONARY *dict, const char *name, ssize_t name_len);
191 +
192 +#define dictionary_get_and_acquire_item(dict, name) dictionary_get_and_acquire_item_advanced(dict, name, -1)
193 +extern DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_get_and_acquire_item_advanced(DICTIONARY *dict, const char *name, ssize_t name_len);
194 +
195 +
196 +// ----------------------------------------------------------------------------
197 // Delete an item from the dictionary
111 -// returns 0 if the item was found and has been deleted
112 -// returns -1 if the item was not found in the index
113 -extern int dictionary_del(DICTIONARY *dict, const char *name);
198 +// returns true if the item was found and has been deleted
199 +// returns false if the item was not found in the index
200
115 -extern DICTIONARY_ITEM *dictionary_get_and_acquire_item_unsafe(DICTIONARY *dict, const char *name);
116 -extern DICTIONARY_ITEM *dictionary_get_and_acquire_item(DICTIONARY *dict, const char *name);
201 +#define dictionary_del(dict, name) dictionary_del_advanced(dict, name, -1)
202 +extern bool dictionary_del_advanced(DICTIONARY *dict, const char *name, ssize_t name_len);
203
118 -extern DICTIONARY_ITEM *dictionary_set_and_acquire_item_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len);
119 -extern DICTIONARY_ITEM *dictionary_set_and_acquire_item(DICTIONARY *dict, const char *name, void *value, size_t value_len);
204 +// ----------------------------------------------------------------------------
205 +// reference counters management
206
121 -extern void dictionary_acquired_item_release_unsafe(DICTIONARY *dict, DICTIONARY_ITEM *item);
122 -extern void dictionary_acquired_item_release(DICTIONARY *dict, DICTIONARY_ITEM *item);
207 +extern void dictionary_acquired_item_release(DICTIONARY *dict, DICT_ITEM_CONST DICTIONARY_ITEM *item);
208
124 -extern DICTIONARY_ITEM *dictionary_acquired_item_dup(DICTIONARY_ITEM *item);
125 -extern const char *dictionary_acquired_item_name(DICTIONARY_ITEM *item);
126 -extern void *dictionary_acquired_item_value(DICTIONARY_ITEM *item);
209 +extern DICT_ITEM_CONST DICTIONARY_ITEM *dictionary_acquired_item_dup(DICTIONARY *dict, DICT_ITEM_CONST DICTIONARY_ITEM *item);
210
128 -// UNSAFE functions, without locks
129 -// to be used when the user is traversing with the right lock type
130 -// Read lock is acquired by dictionary_walktrhough_read() and dfe_start_read()
131 -// Write lock is acquired by dictionary_walktrhough_write() and dfe_start_write()
132 -// For code readability, please use these macros:
133 -#define dictionary_get_having_read_lock(dict, name) dictionary_get_unsafe(dict, name)
134 -#define dictionary_get_having_write_lock(dict, name) dictionary_get_unsafe(dict, name)
135 -#define dictionary_set_having_write_lock(dict, name, value, value_len) dictionary_set_unsafe(dict, name, value, value_len)
136 -#define dictionary_del_having_write_lock(dict, name) dictionary_del_unsafe(dict, name)
211 +extern const char *dictionary_acquired_item_name(DICT_ITEM_CONST DICTIONARY_ITEM *item);
212 +extern void *dictionary_acquired_item_value(DICT_ITEM_CONST DICTIONARY_ITEM *item);
213
138 -extern void *dictionary_get_unsafe(DICTIONARY *dict, const char *name);
139 -extern void *dictionary_set_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len);
140 -extern int dictionary_del_unsafe(DICTIONARY *dict, const char *name);
214
215 +// ----------------------------------------------------------------------------
216 // Traverse (walk through) the items of the dictionary.
217 // The order of traversal is currently the order of insertion.
218 //
@@ -153,12 +227,13 @@ extern int dictionary_del_unsafe(DICTIONARY *dict, const char *name);
227 //
228 #define dictionary_walkthrough_read(dict, callback, data) dictionary_walkthrough_rw(dict, 'r', callback, data)
229 #define dictionary_walkthrough_write(dict, callback, data) dictionary_walkthrough_rw(dict, 'w', callback, data)
156 -extern int dictionary_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const char *name, void *value, void *data), void *data);
230 +extern int dictionary_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data);
231
232 #define dictionary_sorted_walkthrough_read(dict, callback, data) dictionary_sorted_walkthrough_rw(dict, 'r', callback, data)
233 #define dictionary_sorted_walkthrough_write(dict, callback, data) dictionary_sorted_walkthrough_rw(dict, 'w', callback, data)
160 -int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const char *name, void *entry, void *data), void *data);
234 +int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const DICTIONARY_ITEM *item, void *entry, void *data), void *data);
235
236 +// ----------------------------------------------------------------------------
237 // Traverse with foreach
238 //
239 // Use like this:
@@ -173,84 +248,60 @@ int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(
248 // You can only delete the current item from inside a dfe_start_write() - you can add as many as you want.
249 //
250
176 -#ifdef DICTIONARY_INTERNALS
177 -#define DICTFE_CONST
178 -#else
179 -#define DICTFE_CONST const
180 -#endif
181 -
251 #define DICTIONARY_LOCK_READ 'r'
252 #define DICTIONARY_LOCK_WRITE 'w'
253 #define DICTIONARY_LOCK_REENTRANT 'z'
185 -#define DICTIONARY_LOCK_NONE 'u'
254
255 typedef DICTFE_CONST struct dictionary_foreach {
256 + DICTIONARY *dict; // the dictionary upon we work
257 +
258 + DICTIONARY_ITEM *item; // the item we work on, to remember the position we are at
259 + // this can be used with dictionary_acquired_item_dup() to
260 + // acquire the currently working item.
261 +
262 DICTFE_CONST char *name; // the dictionary name of the last item used
263 void *value; // the dictionary value of the last item used
264 // same as the return value of dictfe_start() and dictfe_next()
265
192 - // the following are for internal use only - to keep track of the point we are
266 + size_t counter; // counts the number of iterations made, starting from zero
267 +
268 char rw; // the lock mode 'r' or 'w'
194 - usec_t started_ut; // the time the caller started iterating (now_realtime_usec())
195 - DICTIONARY *dict; // the dictionary upon we work
196 - void *last_item; // the item we work on, to remember the position we are at
269 } DICTFE;
270
271 #define dfe_start_read(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_READ)
272 #define dfe_start_write(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_WRITE)
273 #define dfe_start_reentrant(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_REENTRANT)
202 -#define dfe_start_unsafe(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_NONE)
203 -#define dfe_start_rw(dict, value, mode) \
204 - do { \
205 - DICTFE value ## _dfe = {}; \
206 - const char *value ## _name; (void)(value ## _name); (void)(value); \
207 - for((value) = dictionary_foreach_start_rw(&value ## _dfe, (dict), (mode)), ( value ## _name ) = value ## _dfe.name; \
208 - (value ## _dfe.name) ;\
209 - (value) = dictionary_foreach_next(&value ## _dfe), ( value ## _name ) = value ## _dfe.name) \
274 +
275 +#define dfe_start_rw(dict, value, mode) \
276 + do { \
277 + DICTFE value ## _dfe = {}; \
278 + (void)(value); /* needed to avoid warning when looping without using this */ \
279 + for((value) = dictionary_foreach_start_rw(&value ## _dfe, (dict), (mode)); \
280 + (value ## _dfe.item) ; \
281 + (value) = dictionary_foreach_next(&value ## _dfe)) \
282 {
283
212 -#define dfe_done(value) \
213 - } \
214 - dictionary_foreach_done(&value ## _dfe); \
284 +#define dfe_done(value) \
285 + } \
286 + dictionary_foreach_done(&value ## _dfe); \
287 } while(0)
288
217 -extern void * dictionary_foreach_start_rw(DICTFE *dfe, DICTIONARY *dict, char rw);
218 -extern void * dictionary_foreach_next(DICTFE *dfe);
219 -extern usec_t dictionary_foreach_done(DICTFE *dfe);
220 -
221 -// Get statistics about the dictionary
222 -extern long int dictionary_stats_allocated_memory(DICTIONARY *dict);
223 -extern long int dictionary_stats_entries(DICTIONARY *dict);
224 -extern size_t dictionary_stats_version(DICTIONARY *dict);
225 -extern size_t dictionary_stats_inserts(DICTIONARY *dict);
226 -extern size_t dictionary_stats_searches(DICTIONARY *dict);
227 -extern size_t dictionary_stats_deletes(DICTIONARY *dict);
228 -extern size_t dictionary_stats_resets(DICTIONARY *dict);
229 -extern size_t dictionary_stats_walkthroughs(DICTIONARY *dict);
230 -extern size_t dictionary_stats_referenced_items(DICTIONARY *dict);
231 -
232 -extern int dictionary_unittest(size_t entries);
289 +extern void *dictionary_foreach_start_rw(DICTFE *dfe, DICTIONARY *dict, char rw);
290 +extern void *dictionary_foreach_next(DICTFE *dfe);
291 +extern void dictionary_foreach_done(DICTFE *dfe);
292
293 // ----------------------------------------------------------------------------
235 -// STRING implementation
236 -
237 -typedef struct netdata_string STRING;
238 -extern STRING *string_strdupz(const char *str);
239 -extern STRING *string_dup(STRING *string);
240 -extern void string_freez(STRING *string);
241 -extern size_t string_strlen(STRING *string);
242 -extern const char *string2str(STRING *string) NEVERNULL;
294 +// Get statistics about the dictionary
295
244 -// keep common prefix/suffix and replace everything else with [x]
245 -extern STRING *string_2way_merge(STRING *a, STRING *b);
296 +extern size_t dictionary_version(DICTIONARY *dict);
297 +extern size_t dictionary_entries(DICTIONARY *dict);
298 +extern size_t dictionary_referenced_items(DICTIONARY *dict);
299 +extern long int dictionary_stats_for_registry(DICTIONARY *dict);
300
247 -static inline int string_cmp(STRING *s1, STRING *s2) {
248 - // STRINGs are deduplicated, so the same strings have the same pointer
249 - // when they differ, we do the typical strcmp() comparison
250 - return (s1 == s2)?0:strcmp(string2str(s1), string2str(s2));
251 -}
301 +// for all cases that the caller does not provide a stats structure, this is where they are accumulated.
302 +extern struct dictionary_stats dictionary_stats_category_other;
303
253 -extern void string_statistics(size_t *inserts, size_t *deletes, size_t *searches, size_t *entries, size_t *references, size_t *memory, size_t *duplications, size_t *releases);
304 +extern int dictionary_unittest(size_t entries);
305
306 // ----------------------------------------------------------------------------
307 // THREAD CACHE
libnetdata/libnetdata.h
+1
@@ -410,6 +410,7 @@ extern char *netdata_configured_host_prefix;
410 #include "config/appconfig.h"
411 #include "log/log.h"
412 #include "procfile/procfile.h"
413 +#include "string/string.h"
414 #include "dictionary/dictionary.h"
415 #if defined(HAVE_LIBBPF) && !defined(__cplusplus)
416 #include "ebpf/ebpf.h"
libnetdata/locks/README.md
+3 -3
@@ -49,7 +49,7 @@ The library maintains a linked-list of all the lock holders (one entry per threa
49 If any call is expected to pause the caller (ie the caller is attempting a read lock while there is a write lock in place and vice versa), the library will log something like this:
50
51 ```
52 -RW_LOCK ON LOCK 0x0x5651c9fcce20: 4190039 'HEALTH' (function init_pending_foreach_alarms() 661@health/health.c) WANTS a 'W' lock (while holding 1 rwlocks and 1 mutexes).
52 +RW_LOCK ON LOCK 0x0x5651c9fcce20: 4190039 'HEALTH' (function health_execute_pending_updates() 661@health/health.c) WANTS a 'W' lock (while holding 1 rwlocks and 1 mutexes).
53 There are 7 readers and 0 writers are holding the lock:
54 => 1: RW_LOCK: process 4190091 'WEB_SERVER[static14]' (function web_client_api_request_v1_data() 526@web/api/web_api_v1.c) is having 1 'R' lock for 709847 usec.
55 => 2: RW_LOCK: process 4190079 'WEB_SERVER[static6]' (function web_client_api_request_v1_data() 526@web/api/web_api_v1.c) is having 1 'R' lock for 709869 usec.
@@ -63,9 +63,9 @@ There are 7 readers and 0 writers are holding the lock:
63 And each of the above is paired with a `GOT` log, like this:
64
65 ```
66 -RW_LOCK ON LOCK 0x0x5651c9fcce20: 4190039 'HEALTH' (function init_pending_foreach_alarms() 661@health/health.c) GOT a 'W' lock (while holding 2 rwlocks and 1 mutexes).
66 +RW_LOCK ON LOCK 0x0x5651c9fcce20: 4190039 'HEALTH' (function health_execute_pending_updates() 661@health/health.c) GOT a 'W' lock (while holding 2 rwlocks and 1 mutexes).
67 There are 0 readers and 1 writers are holding the lock:
68 - => 1: RW_LOCK: process 4190039 'HEALTH' (function init_pending_foreach_alarms() 661@health/health.c) is having 1 'W' lock for 36 usec.
68 + => 1: RW_LOCK: process 4190039 'HEALTH' (function health_execute_pending_updates() 661@health/health.c) is having 1 'W' lock for 36 usec.
69 ```
70
71 Keep in mind that the lock and log are not atomic. The list of callers is indicative (and sometimes just empty because the original holders of the lock, unlocked it until we had the chance to print their names).
libnetdata/log/log.c
+11
@@ -850,6 +850,13 @@ void error_int( const char *prefix, const char *file __maybe_unused, const char
850 log_unlock();
851 }
852
853 +#ifdef NETDATA_INTERNAL_CHECKS
854 +static void crash_netdata(void) {
855 + // make Netdata core dump
856 + abort();
857 +}
858 +#endif
859 +
860 void fatal_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) {
861 // save a copy of errno - just in case this function generates a new error
862 int __errno = errno;
@@ -897,6 +904,10 @@ void fatal_int( const char *file, const char *function, const unsigned long line
904 snprintfz(action_result, 60, "%s:%s", program_name, strncmp(thread_tag, "STREAM_RECEIVER", strlen("STREAM_RECEIVER")) ? thread_tag : "[x]");
905 send_statistics("FATAL", action_result, action_data);
906
907 +#ifdef NETDATA_INTERNAL_CHECKS
908 + crash_netdata();
909 +#endif
910 +
911 netdata_cleanup_and_exit(1);
912 }
913
libnetdata/required_dummies.h
+2
@@ -33,6 +33,8 @@ int health_variable_lookup(STRING *variable, struct rrdcalc *rc, NETDATA_DOUBLE
33 };
34 #endif
35
36 +void rrdset_thread_rda_free(void){};
37 +
38 // required by get_system_cpus()
39 char *netdata_configured_host_prefix = "";
40
libnetdata/string/Makefile.am new
+8
@@ -0,0 +1,8 @@
1 +# SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +AUTOMAKE_OPTIONS = subdir-objects
4 +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
5 +
6 +dist_noinst_DATA = \
7 + README.md \
8 + $(NULL)
libnetdata/string/README.md new
+20
@@ -0,0 +1,20 @@
1 +<!--
2 +custom_edit_url: https://github.com/netdata/netdata/edit/master/libnetdata/string/README.md
3 +-->
4 +
5 +# STRING
6 +
7 +STRING provides a way to allocate and free text strings, while de-duplicating them.
8 +
9 +It can be used similarly to libc string functions:
10 +
11 + - `strdup()` and `strdupz()` become `string_strdupz()`.
12 + - `strlen()` becomes `string_strlen()` (and it does not walkthrough the bytes of the string).
13 + - `free()` and `freez()` become `string_freez()`.
14 +
15 +There is also a special `string_dup()` function that increases the reference counter of a STRING, avoiding the
16 +index lookup to find it.
17 +
18 +Once there is a `STRING *`, the actual `const char *` can be accessed with `string2str()`.
19 +
20 +All STRING should be constant. Changing the contents of a `const char *` that has been acquired by `string2str()` should never happen.
\ No newline at end of file
libnetdata/string/string.c new
+595
@@ -0,0 +1,595 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
4 +#include <Judy.h>
5 +
6 +typedef int32_t REFCOUNT;
7 +
8 +// ----------------------------------------------------------------------------
9 +// STRING implementation - dedup all STRING
10 +
11 +struct netdata_string {
12 + uint32_t length; // the string length including the terminating '\0'
13 +
14 + REFCOUNT refcount; // how many times this string is used
15 + // We use a signed number to be able to detect duplicate frees of a string.
16 + // If at any point this goes below zero, we have a duplicate free.
17 +
18 + const char str[]; // the string itself, is appended to this structure
19 +};
20 +
21 +static struct string_hashtable {
22 + Pvoid_t JudyHSArray; // the Judy array - hashtable
23 + netdata_rwlock_t rwlock; // the R/W lock to protect the Judy array
24 +
25 + long int entries; // the number of entries in the index
26 + long int active_references; // the number of active references alive
27 + long int memory; // the memory used, without the JudyHS index
28 +
29 + size_t inserts; // the number of successful inserts to the index
30 + size_t deletes; // the number of successful deleted from the index
31 + size_t searches; // the number of successful searches in the index
32 + size_t duplications; // when a string is referenced
33 + size_t releases; // when a string is unreferenced
34 +
35 +#ifdef NETDATA_INTERNAL_CHECKS
36 + // internal statistics
37 + size_t found_deleted_on_search;
38 + size_t found_available_on_search;
39 + size_t found_deleted_on_insert;
40 + size_t found_available_on_insert;
41 + size_t spins;
42 +#endif
43 +
44 +} string_base = {
45 + .JudyHSArray = NULL,
46 + .rwlock = NETDATA_RWLOCK_INITIALIZER,
47 +};
48 +
49 +#ifdef NETDATA_INTERNAL_CHECKS
50 +#define string_internal_stats_add(var, val) __atomic_add_fetch(&string_base.var, val, __ATOMIC_RELAXED)
51 +#else
52 +#define string_internal_stats_add(var, val) do {;} while(0)
53 +#endif
54 +
55 +#define string_stats_atomic_increment(var) __atomic_add_fetch(&string_base.var, 1, __ATOMIC_RELAXED)
56 +#define string_stats_atomic_decrement(var) __atomic_sub_fetch(&string_base.var, 1, __ATOMIC_RELAXED)
57 +
58 +void string_statistics(size_t *inserts, size_t *deletes, size_t *searches, size_t *entries, size_t *references, size_t *memory, size_t *duplications, size_t *releases) {
59 + *inserts = string_base.inserts;
60 + *deletes = string_base.deletes;
61 + *searches = string_base.searches;
62 + *entries = (size_t)string_base.entries;
63 + *references = (size_t)string_base.active_references;
64 + *memory = (size_t)string_base.memory;
65 + *duplications = string_base.duplications;
66 + *releases = string_base.releases;
67 +}
68 +
69 +#define string_entry_acquire(se) __atomic_add_fetch(&((se)->refcount), 1, __ATOMIC_SEQ_CST);
70 +#define string_entry_release(se) __atomic_sub_fetch(&((se)->refcount), 1, __ATOMIC_SEQ_CST);
71 +
72 +static inline bool string_entry_check_and_acquire(STRING *se) {
73 + REFCOUNT expected, desired, count = 0;
74 + do {
75 + count++;
76 +
77 + expected = __atomic_load_n(&se->refcount, __ATOMIC_SEQ_CST);
78 +
79 + if(expected <= 0) {
80 + // We cannot use this.
81 + // The reference counter reached value zero,
82 + // so another thread is deleting this.
83 + string_internal_stats_add(spins, count - 1);
84 + return false;
85 + }
86 +
87 + desired = expected + 1;
88 + }
89 + while(!__atomic_compare_exchange_n(&se->refcount, &expected, desired, false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST));
90 +
91 + string_internal_stats_add(spins, count - 1);
92 +
93 + // statistics
94 + // string_base.active_references is altered at the in string_strdupz() and string_freez()
95 + string_stats_atomic_increment(duplications);
96 +
97 + return true;
98 +}
99 +
100 +STRING *string_dup(STRING *string) {
101 + if(unlikely(!string)) return NULL;
102 +
103 +#ifdef NETDATA_INTERNAL_CHECKS
104 + if(unlikely(__atomic_load_n(&string->refcount, __ATOMIC_SEQ_CST) <= 0))
105 + fatal("STRING: tried to %s() a string that is freed (it has %d references).", __FUNCTION__, string->refcount);
106 +#endif
107 +
108 + string_entry_acquire(string);
109 +
110 + // statistics
111 + string_stats_atomic_increment(active_references);
112 + string_stats_atomic_increment(duplications);
113 +
114 + return string;
115 +}
116 +
117 +// Search the index and return an ACQUIRED string entry, or NULL
118 +static inline STRING *string_index_search(const char *str, size_t length) {
119 + STRING *string;
120 +
121 + // Find the string in the index
122 + // With a read-lock so that multiple readers can use the index concurrently.
123 +
124 + netdata_rwlock_rdlock(&string_base.rwlock);
125 +
126 + Pvoid_t *Rc;
127 + Rc = JudyHSGet(string_base.JudyHSArray, (void *)str, length);
128 + if(likely(Rc)) {
129 + // found in the hash table
130 + string = *Rc;
131 +
132 + if(string_entry_check_and_acquire(string)) {
133 + // we can use this entry
134 + string_internal_stats_add(found_available_on_search, 1);
135 + }
136 + else {
137 + // this entry is about to be deleted by another thread
138 + // do not touch it, let it go...
139 + string = NULL;
140 + string_internal_stats_add(found_deleted_on_search, 1);
141 + }
142 + }
143 + else {
144 + // not found in the hash table
145 + string = NULL;
146 + }
147 +
148 + string_stats_atomic_increment(searches);
149 + netdata_rwlock_unlock(&string_base.rwlock);
150 +
151 + return string;
152 +}
153 +
154 +// Insert a string to the index and return an ACQUIRED string entry,
155 +// or NULL if the call needs to be retried (a deleted entry with the same key is still in the index)
156 +// The returned entry is ACQUIRED, and it can either be:
157 +// 1. a new item inserted, or
158 +// 2. an item found in the index that is not currently deleted
159 +static inline STRING *string_index_insert(const char *str, size_t length) {
160 + STRING *string;
161 +
162 + netdata_rwlock_wrlock(&string_base.rwlock);
163 +
164 + STRING **ptr;
165 + {
166 + JError_t J_Error;
167 + Pvoid_t *Rc = JudyHSIns(&string_base.JudyHSArray, (void *)str, length, &J_Error);
168 + if (unlikely(Rc == PJERR)) {
169 + fatal(
170 + "STRING: Cannot insert entry with name '%s' to JudyHS, JU_ERRNO_* == %u, ID == %d",
171 + str,
172 + JU_ERRNO(&J_Error),
173 + JU_ERRID(&J_Error));
174 + }
175 + ptr = (STRING **)Rc;
176 + }
177 +
178 + if (likely(*ptr == 0)) {
179 + // a new item added to the index
180 + size_t mem_size = sizeof(STRING) + length;
181 + string = mallocz(mem_size);
182 + strcpy((char *)string->str, str);
183 + string->length = length;
184 + string->refcount = 1;
185 + *ptr = string;
186 + string_base.inserts++;
187 + string_base.entries++;
188 + string_base.memory += (long)mem_size;
189 + }
190 + else {
191 + // the item is already in the index
192 + string = *ptr;
193 +
194 + if(string_entry_check_and_acquire(string)) {
195 + // we can use this entry
196 + string_internal_stats_add(found_available_on_insert, 1);
197 + }
198 + else {
199 + // this entry is about to be deleted by another thread
200 + // do not touch it, let it go...
201 + string = NULL;
202 + string_internal_stats_add(found_deleted_on_insert, 1);
203 + }
204 +
205 + string_stats_atomic_increment(searches);
206 + }
207 +
208 + netdata_rwlock_unlock(&string_base.rwlock);
209 + return string;
210 +}
211 +
212 +// delete an entry from the index
213 +static inline void string_index_delete(STRING *string) {
214 + netdata_rwlock_wrlock(&string_base.rwlock);
215 +
216 +#ifdef NETDATA_INTERNAL_CHECKS
217 + if(unlikely(__atomic_load_n(&string->refcount, __ATOMIC_SEQ_CST) != 0))
218 + fatal("STRING: tried to delete a string at %s() that is already freed (it has %d references).", __FUNCTION__, string->refcount);
219 +#endif
220 +
221 + bool deleted = false;
222 +
223 + if (likely(string_base.JudyHSArray)) {
224 + JError_t J_Error;
225 + int ret = JudyHSDel(&string_base.JudyHSArray, (void *)string->str, string->length, &J_Error);
226 + if (unlikely(ret == JERR)) {
227 + error(
228 + "STRING: Cannot delete entry with name '%s' from JudyHS, JU_ERRNO_* == %u, ID == %d",
229 + string->str,
230 + JU_ERRNO(&J_Error),
231 + JU_ERRID(&J_Error));
232 + } else
233 + deleted = true;
234 + }
235 +
236 + if (unlikely(!deleted))
237 + error("STRING: tried to delete '%s' that is not in the index. Ignoring it.", string->str);
238 + else {
239 + size_t mem_size = sizeof(STRING) + string->length;
240 + string_base.deletes++;
241 + string_base.entries--;
242 + string_base.memory -= (long)mem_size;
243 + freez(string);
244 + }
245 +
246 + netdata_rwlock_unlock(&string_base.rwlock);
247 +}
248 +
249 +STRING *string_strdupz(const char *str) {
250 + if(unlikely(!str || !*str)) return NULL;
251 +
252 + size_t length = strlen(str) + 1;
253 + STRING *string = string_index_search(str, length);
254 +
255 + while(!string) {
256 + // The search above did not find anything,
257 + // We loop here, because during insert we may find an entry that is being deleted by another thread.
258 + // So, we have to let it go and retry to insert it again.
259 +
260 + string = string_index_insert(str, length);
261 + }
262 +
263 + // statistics
264 + string_stats_atomic_increment(active_references);
265 +
266 + return string;
267 +}
268 +
269 +void string_freez(STRING *string) {
270 + if(unlikely(!string)) return;
271 +
272 + REFCOUNT refcount = string_entry_release(string);
273 +
274 +#ifdef NETDATA_INTERNAL_CHECKS
275 + if(unlikely(refcount < 0))
276 + fatal("STRING: tried to %s() a string that is already freed (it has %d references).", __FUNCTION__, string->refcount);
277 +#endif
278 +
279 + if(unlikely(refcount == 0))
280 + string_index_delete(string);
281 +
282 + // statistics
283 + string_stats_atomic_decrement(active_references);
284 + string_stats_atomic_increment(releases);
285 +}
286 +
287 +size_t string_strlen(STRING *string) {
288 + if(unlikely(!string)) return 0;
289 + return string->length - 1;
290 +}
291 +
292 +const char *string2str(STRING *string) {
293 + if(unlikely(!string)) return "";
294 + return string->str;
295 +}
296 +
297 +STRING *string_2way_merge(STRING *a, STRING *b) {
298 + static STRING *X = NULL;
299 +
300 + if(unlikely(!X)) {
301 + X = string_strdupz("[x]");
302 + }
303 +
304 + if(unlikely(a == b)) return string_dup(a);
305 + if(unlikely(a == X)) return string_dup(a);
306 + if(unlikely(b == X)) return string_dup(b);
307 + if(unlikely(!a)) return string_dup(X);
308 + if(unlikely(!b)) return string_dup(X);
309 +
310 + size_t alen = string_strlen(a);
311 + size_t blen = string_strlen(b);
312 + size_t length = alen + blen + string_strlen(X) + 1;
313 + char buf1[length + 1], buf2[length + 1], *dst1;
314 + const char *s1, *s2;
315 +
316 + s1 = string2str(a);
317 + s2 = string2str(b);
318 + dst1 = buf1;
319 + for( ; *s1 && *s2 && *s1 == *s2 ;s1++, s2++)
320 + *dst1++ = *s1;
321 +
322 + *dst1 = '\0';
323 +
324 + if(*s1 != '\0' || *s2 != '\0') {
325 + *dst1++ = '[';
326 + *dst1++ = 'x';
327 + *dst1++ = ']';
328 +
329 + s1 = &(string2str(a))[alen - 1];
330 + s2 = &(string2str(b))[blen - 1];
331 + char *dst2 = &buf2[length];
332 + *dst2 = '\0';
333 + for (; *s1 && *s2 && *s1 == *s2; s1--, s2--)
334 + *(--dst2) = *s1;
335 +
336 + strcpy(dst1, dst2);
337 + }
338 +
339 + return string_strdupz(buf1);
340 +}
341 +
342 +// ----------------------------------------------------------------------------
343 +// STRING unit test
344 +
345 +struct thread_unittest {
346 + int join;
347 + int dups;
348 +};
349 +
350 +static void *string_thread(void *arg) {
351 + struct thread_unittest *tu = arg;
352 +
353 + for(; 1 ;) {
354 + if(__atomic_load_n(&tu->join, __ATOMIC_RELAXED))
355 + break;
356 +
357 + STRING *s = string_strdupz("string thread checking 1234567890");
358 +
359 + for(int i = 0; i < tu->dups ; i++)
360 + string_dup(s);
361 +
362 + for(int i = 0; i < tu->dups ; i++)
363 + string_freez(s);
364 +
365 + string_freez(s);
366 + }
367 +
368 + return arg;
369 +}
370 +
371 +static char **string_unittest_generate_names(size_t entries) {
372 + char **names = mallocz(sizeof(char *) * entries);
373 + for(size_t i = 0; i < entries ;i++) {
374 + char buf[25 + 1] = "";
375 + snprintfz(buf, 25, "name.%zu.0123456789.%zu \t !@#$%%^&*(),./[]{}\\|~`", i, entries / 2 + i);
376 + names[i] = strdupz(buf);
377 + }
378 + return names;
379 +}
380 +
381 +static void string_unittest_free_char_pp(char **pp, size_t entries) {
382 + for(size_t i = 0; i < entries ;i++)
383 + freez(pp[i]);
384 +
385 + freez(pp);
386 +}
387 +
388 +int string_unittest(size_t entries) {
389 + size_t errors = 0;
390 +
391 + fprintf(stderr, "Generating %zu names and values...\n", entries);
392 + char **names = string_unittest_generate_names(entries);
393 +
394 + // check string
395 + {
396 + long int string_entries_starting = string_base.entries;
397 +
398 + fprintf(stderr, "\nChecking strings...\n");
399 +
400 + STRING *s1 = string_strdupz("hello unittest");
401 + STRING *s2 = string_strdupz("hello unittest");
402 + if(s1 != s2) {
403 + errors++;
404 + fprintf(stderr, "ERROR: duplicating strings are not deduplicated\n");
405 + }
406 + else
407 + fprintf(stderr, "OK: duplicating string are deduplicated\n");
408 +
409 + STRING *s3 = string_dup(s1);
410 + if(s3 != s1) {
411 + errors++;
412 + fprintf(stderr, "ERROR: cloning strings are not deduplicated\n");
413 + }
414 + else
415 + fprintf(stderr, "OK: cloning string are deduplicated\n");
416 +
417 + if(s1->refcount != 3) {
418 + errors++;
419 + fprintf(stderr, "ERROR: string refcount is not 3\n");
420 + }
421 + else
422 + fprintf(stderr, "OK: string refcount is 3\n");
423 +
424 + STRING *s4 = string_strdupz("world unittest");
425 + if(s4 == s1) {
426 + errors++;
427 + fprintf(stderr, "ERROR: string is sharing pointers on different strings\n");
428 + }
429 + else
430 + fprintf(stderr, "OK: string is properly handling different strings\n");
431 +
432 + usec_t start_ut, end_ut;
433 + STRING **strings = mallocz(entries * sizeof(STRING *));
434 +
435 + start_ut = now_realtime_usec();
436 + for(size_t i = 0; i < entries ;i++) {
437 + strings[i] = string_strdupz(names[i]);
438 + }
439 + end_ut = now_realtime_usec();
440 + fprintf(stderr, "Created %zu strings in %llu usecs\n", entries, end_ut - start_ut);
441 +
442 + start_ut = now_realtime_usec();
443 + for(size_t i = 0; i < entries ;i++) {
444 + strings[i] = string_dup(strings[i]);
445 + }
446 + end_ut = now_realtime_usec();
447 + fprintf(stderr, "Cloned %zu strings in %llu usecs\n", entries, end_ut - start_ut);
448 +
449 + start_ut = now_realtime_usec();
450 + for(size_t i = 0; i < entries ;i++) {
451 + strings[i] = string_strdupz(string2str(strings[i]));
452 + }
453 + end_ut = now_realtime_usec();
454 + fprintf(stderr, "Found %zu existing strings in %llu usecs\n", entries, end_ut - start_ut);
455 +
456 + start_ut = now_realtime_usec();
457 + for(size_t i = 0; i < entries ;i++) {
458 + string_freez(strings[i]);
459 + }
460 + end_ut = now_realtime_usec();
461 + fprintf(stderr, "Released %zu referenced strings in %llu usecs\n", entries, end_ut - start_ut);
462 +
463 + start_ut = now_realtime_usec();
464 + for(size_t i = 0; i < entries ;i++) {
465 + string_freez(strings[i]);
466 + }
467 + end_ut = now_realtime_usec();
468 + fprintf(stderr, "Released (again) %zu referenced strings in %llu usecs\n", entries, end_ut - start_ut);
469 +
470 + start_ut = now_realtime_usec();
471 + for(size_t i = 0; i < entries ;i++) {
472 + string_freez(strings[i]);
473 + }
474 + end_ut = now_realtime_usec();
475 + fprintf(stderr, "Freed %zu strings in %llu usecs\n", entries, end_ut - start_ut);
476 +
477 + freez(strings);
478 +
479 + if(string_base.entries != string_entries_starting + 2) {
480 + errors++;
481 + fprintf(stderr, "ERROR: strings dictionary should have %ld items but it has %ld\n", string_entries_starting + 2, string_base.entries);
482 + }
483 + else
484 + fprintf(stderr, "OK: strings dictionary has 2 items\n");
485 + }
486 +
487 + // check 2-way merge
488 + {
489 + struct testcase {
490 + char *src1; char *src2; char *expected;
491 + } tests[] = {
492 + { "", "", ""},
493 + { "a", "", "[x]"},
494 + { "", "a", "[x]"},
495 + { "a", "a", "a"},
496 + { "abcd", "abcd", "abcd"},
497 + { "foo_cs", "bar_cs", "[x]_cs"},
498 + { "cp_UNIQUE_INFIX_cs", "cp_unique_infix_cs", "cp_[x]_cs"},
499 + { "cp_UNIQUE_INFIX_ci_unique_infix_cs", "cp_unique_infix_ci_UNIQUE_INFIX_cs", "cp_[x]_cs"},
500 + { "foo[1234]", "foo[4321]", "foo[[x]]"},
501 + { NULL, NULL, NULL },
502 + };
503 +
504 + for (struct testcase *tc = &tests[0]; tc->expected != NULL; tc++) {
505 + STRING *src1 = string_strdupz(tc->src1);
506 + STRING *src2 = string_strdupz(tc->src2);
507 + STRING *expected = string_strdupz(tc->expected);
508 +
509 + STRING *result = string_2way_merge(src1, src2);
510 + if (string_cmp(result, expected) != 0) {
511 + fprintf(stderr, "string_2way_merge(\"%s\", \"%s\") -> \"%s\" (expected=\"%s\")\n",
512 + string2str(src1),
513 + string2str(src2),
514 + string2str(result),
515 + string2str(expected));
516 + errors++;
517 + }
518 +
519 + string_freez(src1);
520 + string_freez(src2);
521 + string_freez(expected);
522 + string_freez(result);
523 + }
524 + }
525 +
526 + // threads testing of string
527 + {
528 + struct thread_unittest tu = {
529 + .dups = 1,
530 + .join = 0,
531 + };
532 +
533 +#ifdef NETDATA_INTERNAL_CHECKS
534 + size_t ofound_deleted_on_search = string_base.found_deleted_on_search,
535 + ofound_available_on_search = string_base.found_available_on_search,
536 + ofound_deleted_on_insert = string_base.found_deleted_on_insert,
537 + ofound_available_on_insert = string_base.found_available_on_insert,
538 + ospins = string_base.spins;
539 +#endif
540 +
541 + size_t oinserts, odeletes, osearches, oentries, oreferences, omemory, oduplications, oreleases;
542 + string_statistics(&oinserts, &odeletes, &osearches, &oentries, &oreferences, &omemory, &oduplications, &oreleases);
543 +
544 + time_t seconds_to_run = 5;
545 + int threads_to_create = 2;
546 + fprintf(
547 + stderr,
548 + "Checking string concurrency with %d threads for %ld seconds...\n",
549 + threads_to_create,
550 + seconds_to_run);
551 + // check string concurrency
552 + netdata_thread_t threads[threads_to_create];
553 + tu.join = 0;
554 + for (int i = 0; i < threads_to_create; i++) {
555 + char buf[100 + 1];
556 + snprintf(buf, 100, "string%d", i);
557 + netdata_thread_create(
558 + &threads[i], buf, NETDATA_THREAD_OPTION_DONT_LOG | NETDATA_THREAD_OPTION_JOINABLE, string_thread, &tu);
559 + }
560 + sleep_usec(seconds_to_run * USEC_PER_SEC);
561 +
562 + __atomic_store_n(&tu.join, 1, __ATOMIC_RELAXED);
563 + for (int i = 0; i < threads_to_create; i++) {
564 + void *retval;
565 + netdata_thread_join(threads[i], &retval);
566 + }
567 +
568 + size_t inserts, deletes, searches, sentries, references, memory, duplications, releases;
569 + string_statistics(&inserts, &deletes, &searches, &sentries, &references, &memory, &duplications, &releases);
570 +
571 + fprintf(stderr, "inserts %zu, deletes %zu, searches %zu, entries %zu, references %zu, memory %zu, duplications %zu, releases %zu\n",
572 + inserts - oinserts, deletes - odeletes, searches - osearches, sentries - oentries, references - oreferences, memory - omemory, duplications - oduplications, releases - oreleases);
573 +
574 +#ifdef NETDATA_INTERNAL_CHECKS
575 + size_t found_deleted_on_search = string_base.found_deleted_on_search,
576 + found_available_on_search = string_base.found_available_on_search,
577 + found_deleted_on_insert = string_base.found_deleted_on_insert,
578 + found_available_on_insert = string_base.found_available_on_insert,
579 + spins = string_base.spins;
580 +
581 + fprintf(stderr, "on insert: %zu ok + %zu deleted\non search: %zu ok + %zu deleted\nspins: %zu\n",
582 + found_available_on_insert - ofound_available_on_insert,
583 + found_deleted_on_insert - ofound_deleted_on_insert,
584 + found_available_on_search - ofound_available_on_search,
585 + found_deleted_on_search - ofound_deleted_on_search,
586 + spins - ospins
587 + );
588 +#endif
589 + }
590 +
591 + string_unittest_free_char_pp(names, entries);
592 +
593 + fprintf(stderr, "\n%zu errors found\n", errors);
594 + return errors ? 1 : 0;
595 +}
libnetdata/string/string.h new
+30
@@ -0,0 +1,30 @@
1 +
2 +#ifndef NETDATA_STRING_H
3 +#define NETDATA_STRING_H 1
4 +
5 +#include "../libnetdata.h"
6 +
7 +// ----------------------------------------------------------------------------
8 +// STRING implementation
9 +
10 +typedef struct netdata_string STRING;
11 +extern STRING *string_strdupz(const char *str);
12 +extern STRING *string_dup(STRING *string);
13 +extern void string_freez(STRING *string);
14 +extern size_t string_strlen(STRING *string);
15 +extern const char *string2str(STRING *string) NEVERNULL;
16 +
17 +// keep common prefix/suffix and replace everything else with [x]
18 +extern STRING *string_2way_merge(STRING *a, STRING *b);
19 +
20 +static inline int string_cmp(STRING *s1, STRING *s2) {
21 + // STRINGs are deduplicated, so the same strings have the same pointer
22 + // when they differ, we do the typical strcmp() comparison
23 + return (s1 == s2)?0:strcmp(string2str(s1), string2str(s2));
24 +}
25 +
26 +extern void string_statistics(size_t *inserts, size_t *deletes, size_t *searches, size_t *entries, size_t *references, size_t *memory, size_t *duplications, size_t *releases);
27 +
28 +extern int string_unittest(size_t entries);
29 +
30 +#endif
libnetdata/threads/threads.c
+24 -5
@@ -29,26 +29,35 @@ const char *netdata_thread_tag(void) {
29 // ----------------------------------------------------------------------------
30 // compatibility library functions
31
32 +static __thread pid_t gettid_cached_tid = 0;
33 pid_t gettid(void) {
34 + pid_t tid = 0;
35 +
36 + if(likely(gettid_cached_tid > 0))
37 + return gettid_cached_tid;
38 +
39 #ifdef __FreeBSD__
40
35 - return (pid_t)pthread_getthreadid_np();
41 + tid = (pid_t)pthread_getthreadid_np();
42
43 #elif defined(__APPLE__)
44
45 #if (defined __MAC_OS_X_VERSION_MIN_REQUIRED && __MAC_OS_X_VERSION_MIN_REQUIRED >= 1060)
46 uint64_t curthreadid;
47 pthread_threadid_np(NULL, &curthreadid);
42 - return (pid_t)curthreadid;
48 + tid = (pid_t)curthreadid;
49 #else /* __MAC_OS_X_VERSION_MIN_REQUIRED */
44 - return (pid_t)pthread_self;
50 + tid = (pid_t)pthread_self;
51 #endif /* __MAC_OS_X_VERSION_MIN_REQUIRED */
52
53 #else /* __APPLE__*/
54
49 - return (pid_t)syscall(SYS_gettid);
55 + tid = (pid_t)syscall(SYS_gettid);
56
57 #endif /* __FreeBSD__, __APPLE__*/
58 +
59 + gettid_cached_tid = tid;
60 + return tid;
61 }
62
63 // ----------------------------------------------------------------------------
@@ -97,6 +106,8 @@ void netdata_threads_init_after_fork(size_t stacksize) {
106 // ----------------------------------------------------------------------------
107 // netdata_thread_create
108
109 +extern void rrdset_thread_rda_free(void);
110 +
111 static void thread_cleanup(void *ptr) {
112 if(netdata_thread != ptr) {
113 NETDATA_THREAD *info = (NETDATA_THREAD *)ptr;
@@ -106,6 +117,7 @@ static void thread_cleanup(void *ptr) {
117 if(!(netdata_thread->options & NETDATA_THREAD_OPTION_DONT_LOG_CLEANUP))
118 info("thread with task id %d finished", gettid());
119
120 + rrdset_thread_rda_free();
121 thread_cache_destroy();
122
123 freez((void *)netdata_thread->tag);
@@ -215,11 +227,18 @@ int netdata_thread_create(netdata_thread_t *thread, const char *tag, NETDATA_THR
227
228 // ----------------------------------------------------------------------------
229 // netdata_thread_cancel
218 -
230 +#ifdef NETDATA_INTERNAL_CHECKS
231 +int netdata_thread_cancel_with_trace(netdata_thread_t thread, int line, const char *file, const char *function) {
232 +#else
233 int netdata_thread_cancel(netdata_thread_t thread) {
234 +#endif
235 int ret = pthread_cancel(thread);
236 if(ret != 0)
237 +#ifdef NETDATA_INTERNAL_CHECKS
238 + error("cannot cancel thread. pthread_cancel() failed with code %d at %d@%s, function %s()", ret, line, file, function);
239 +#else
240 error("cannot cancel thread. pthread_cancel() failed with code %d.", ret);
241 +#endif
242
243 return ret;
244 }
libnetdata/threads/threads.h
+7
@@ -28,7 +28,14 @@ extern size_t netdata_threads_init(void);
28 extern void netdata_threads_init_after_fork(size_t stacksize);
29
30 extern int netdata_thread_create(netdata_thread_t *thread, const char *tag, NETDATA_THREAD_OPTIONS options, void *(*start_routine) (void *), void *arg);
31 +
32 +#ifdef NETDATA_INTERNAL_CHECKS
33 +#define netdata_thread_cancel(thread) netdata_thread_cancel_with_trace(thread, __LINE__, __FILE__, __FUNCTION__)
34 +extern int netdata_thread_cancel_with_trace(netdata_thread_t thread, int line, const char *file, const char *function);
35 +#else
36 extern int netdata_thread_cancel(netdata_thread_t thread);
37 +#endif
38 +
39 extern int netdata_thread_join(netdata_thread_t thread, void **retval);
40 extern int netdata_thread_detach(pthread_t thread);
41
ml/Dimension.h
+1 -1
@@ -44,7 +44,7 @@ public:
44 RRDDIM *getAnomalyRateRD() const { return AnomalyRateRD; }
45
46 void setAnomalyRateRDName(const char *Name) const {
47 - rrddim_set_name(AnomalyRateRD->rrdset, AnomalyRateRD, Name);
47 + rrddim_reset_name(AnomalyRateRD->rrdset, AnomalyRateRD, Name);
48 }
49
50 virtual ~RrdDimension() {}
registry/registry.c
+3 -5
@@ -108,9 +108,7 @@ static int registry_json_person_url_callback(void *entry, void *data) {
108 }
109
110 // callback for rendering MACHINE_URLs
111 -static int registry_json_machine_url_callback(const char *name, void *entry, void *data) {
112 - (void)name;
113 -
111 +static int registry_json_machine_url_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
112 REGISTRY_MACHINE_URL *mu = (REGISTRY_MACHINE_URL *)entry;
113 struct registry_json_walk_person_urls_callback *c = (struct registry_json_walk_person_urls_callback *)data;
114 struct web_client *w = c->w;
@@ -443,8 +441,8 @@ void registry_statistics(void) {
441 }
442 else rrdset_next(stm);
443
446 - rrddim_set(stm, "persons", registry.persons_memory + dictionary_stats_allocated_memory(registry.persons));
447 - rrddim_set(stm, "machines", registry.machines_memory + dictionary_stats_allocated_memory(registry.machines));
444 + rrddim_set(stm, "persons", registry.persons_memory + dictionary_stats_for_registry(registry.persons));
445 + rrddim_set(stm, "machines", registry.machines_memory + dictionary_stats_for_registry(registry.machines));
446 rrddim_set(stm, "urls", registry.urls_memory);
447 rrddim_set(stm, "persons_urls", registry.persons_urls_memory);
448 rrddim_set(stm, "machines_urls", registry.machines_urls_memory);
registry/registry_db.c
+3 -8
@@ -11,9 +11,7 @@ int registry_db_should_be_saved(void) {
11 // ----------------------------------------------------------------------------
12 // INTERNAL FUNCTIONS FOR SAVING REGISTRY OBJECTS
13
14 -static int registry_machine_save_url(const char *name, void *entry, void *file) {
15 - (void)name;
16 -
14 +static int registry_machine_save_url(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *file) {
15 REGISTRY_MACHINE_URL *mu = entry;
16 FILE *fp = file;
17
@@ -32,8 +30,7 @@ static int registry_machine_save_url(const char *name, void *entry, void *file)
30 return ret;
31 }
32
35 -static int registry_machine_save(const char *name, void *entry, void *file) {
36 - (void)name;
33 +static int registry_machine_save(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *file) {
34
35 REGISTRY_MACHINE *m = entry;
36 FILE *fp = file;
@@ -79,9 +76,7 @@ static inline int registry_person_save_url(void *entry, void *file) {
76 return ret;
77 }
78
82 -static inline int registry_person_save(const char *name, void *entry, void *file) {
83 - (void)name;
84 -
79 +static inline int registry_person_save(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *file) {
80 REGISTRY_PERSON *p = entry;
81 FILE *fp = file;
82
registry/registry_init.c
+5 -13
@@ -76,8 +76,8 @@ int registry_init(void) {
76 netdata_mutex_init(&registry.lock);
77
78 // create dictionaries
79 - registry.persons = dictionary_create(REGISTRY_DICTIONARY_FLAGS);
80 - registry.machines = dictionary_create(REGISTRY_DICTIONARY_FLAGS);
79 + registry.persons = dictionary_create(REGISTRY_DICTIONARY_OPTIONS);
80 + registry.machines = dictionary_create(REGISTRY_DICTIONARY_OPTIONS);
81 avl_init(&registry.registry_urls_root_index, registry_url_compare);
82
83 // load the registry database
@@ -93,9 +93,7 @@ int registry_init(void) {
93 return 0;
94 }
95
96 -static int machine_urls_delete_callback(const char *name, void *entry, void *data) {
97 - (void)name;
98 -
96 +static int machine_urls_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
97 REGISTRY_MACHINE *m = (REGISTRY_MACHINE *)data;
98 (void)m;
99
@@ -110,10 +108,7 @@ static int machine_urls_delete_callback(const char *name, void *entry, void *dat
108 return 1;
109 }
110
113 -static int machine_delete_callback(const char *name, void *entry, void *data) {
114 - (void)name;
115 - (void)data;
116 -
111 +static int machine_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data __maybe_unused) {
112 REGISTRY_MACHINE *m = (REGISTRY_MACHINE *)entry;
113 int ret = dictionary_walkthrough_read(m->machine_urls, machine_urls_delete_callback, m);
114
@@ -122,10 +117,7 @@ static int machine_delete_callback(const char *name, void *entry, void *data) {
117
118 return ret + 1;
119 }
125 -static int registry_person_del_callback(const char *name, void *entry, void *d) {
126 - (void)name;
127 - (void)d;
128 -
120 +static int registry_person_del_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *d __maybe_unused) {
121 REGISTRY_PERSON *p = (REGISTRY_PERSON *)entry;
122
123 debug(D_REGISTRY, "Registry: registry_person_del('%s'): deleting person", p->guid);
registry/registry_internals.h
+1 -1
@@ -8,7 +8,7 @@
8 #define REGISTRY_URL_FLAGS_DEFAULT 0x00
9 #define REGISTRY_URL_FLAGS_EXPIRED 0x01
10
11 -#define REGISTRY_DICTIONARY_FLAGS (DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE | DICTIONARY_FLAG_NAME_LINK_DONT_CLONE | DICTIONARY_FLAG_SINGLE_THREADED)
11 +#define REGISTRY_DICTIONARY_OPTIONS (DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_SINGLE_THREADED)
12
13 // ----------------------------------------------------------------------------
14 // COMMON structures
registry/registry_machine.c
+5 -5
@@ -25,9 +25,9 @@ REGISTRY_MACHINE_URL *registry_machine_url_allocate(REGISTRY_MACHINE *m, REGISTR
25
26 debug(D_REGISTRY, "registry_machine_url_allocate('%s', '%s'): indexing URL in machine", m->guid, u->url);
27
28 - registry.machines_urls_memory -= dictionary_stats_allocated_memory(m->machine_urls);
28 + registry.machines_urls_memory -= dictionary_stats_for_registry(m->machine_urls);
29 dictionary_set(m->machine_urls, u->url, mu, sizeof(REGISTRY_MACHINE_URL));
30 - registry.machines_urls_memory += dictionary_stats_allocated_memory(m->machine_urls);
30 + registry.machines_urls_memory += dictionary_stats_for_registry(m->machine_urls);
31
32 registry_url_link(u);
33
@@ -42,7 +42,7 @@ REGISTRY_MACHINE *registry_machine_allocate(const char *machine_guid, time_t whe
42 strncpyz(m->guid, machine_guid, GUID_LEN);
43
44 debug(D_REGISTRY, "Registry: registry_machine_allocate('%s'): creating dictionary of urls", machine_guid);
45 - m->machine_urls = dictionary_create(REGISTRY_DICTIONARY_FLAGS);
45 + m->machine_urls = dictionary_create(REGISTRY_DICTIONARY_OPTIONS);
46
47 m->first_t = m->last_t = (uint32_t)when;
48 m->usages = 0;
@@ -50,9 +50,9 @@ REGISTRY_MACHINE *registry_machine_allocate(const char *machine_guid, time_t whe
50 registry.machines_memory += sizeof(REGISTRY_MACHINE);
51 registry.machines_count++;
52
53 - registry.machines_urls_memory -= dictionary_stats_allocated_memory(m->machine_urls);
53 + registry.machines_urls_memory -= dictionary_stats_for_registry(m->machine_urls);
54 dictionary_set(registry.machines, m->guid, m, sizeof(REGISTRY_MACHINE));
55 - registry.machines_urls_memory += dictionary_stats_allocated_memory(m->machine_urls);
55 + registry.machines_urls_memory += dictionary_stats_for_registry(m->machine_urls);
56
57 return m;
58 }
streaming/rrdpush.c
+9 -21
@@ -205,7 +205,7 @@ void rrdpush_send_clabels(RRDHOST *host, RRDSET *st) {
205
206 // Send the current chart definition.
207 // Assumes that collector thread has already called sender_start for mutex / buffer state.
208 -static inline void rrdpush_send_chart_definition_nolock(RRDSET *st) {
208 +static inline void rrdpush_send_chart_definition(RRDSET *st) {
209 RRDHOST *host = st->rrdhost;
210
211 rrdset_flag_set(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
@@ -260,26 +260,15 @@ static inline void rrdpush_send_chart_definition_nolock(RRDSET *st) {
260 , rd->multiplier
261 , rd->divisor
262 , rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)?"obsolete":""
263 - , rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN)?"hidden":""
264 - , rrddim_flag_check(rd, RRDDIM_FLAG_DONT_DETECT_RESETS_OR_OVERFLOWS)?"noreset":""
263 + , rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN)?"hidden":""
264 + , rrddim_option_check(rd, RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS)?"noreset":""
265 );
266 rd->exposed = 1;
267 }
268 + rrddim_foreach_done(rd);
269
270 // send the chart local custom variables
270 - RRDSETVAR *rs;
271 - for(rs = st->variables; rs ;rs = rs->next) {
272 - if(unlikely(rs->type == RRDVAR_TYPE_CALCULATED && rs->options & RRDVAR_OPTION_CUSTOM_CHART_VAR)) {
273 - NETDATA_DOUBLE *value = (NETDATA_DOUBLE *) rs->value;
274 -
275 - buffer_sprintf(
276 - host->sender->build
277 - , "VARIABLE CHART %s = " NETDATA_DOUBLE_FORMAT "\n"
278 - , string2str(rs->variable)
279 - , *value
280 - );
281 - }
282 - }
271 + rrdsetvar_print_to_streaming_custom_chart_variables(st, host->sender->build);
272
273 st->upstream_resync_time = st->last_collected_time.tv_sec + (remote_clock_resync_iterations * st->update_every);
274 }
@@ -301,6 +290,7 @@ static inline bool rrdpush_send_chart_metrics_nolock(RRDSET *st, struct sender_s
290 count_of_dimensions_written++;
291 }
292 }
293 + rrddim_foreach_done(rd);
294 buffer_strcat(host->sender->build, "END\n");
295
296 return count_of_dimensions_written != 0;
@@ -315,11 +305,9 @@ bool rrdset_push_chart_definition_now(RRDSET *st) {
305 if(unlikely(!host->rrdpush_send_enabled || !should_send_chart_matching(st)))
306 return false;
307
318 - rrdset_rdlock(st);
308 sender_start(host->sender);
320 - rrdpush_send_chart_definition_nolock(st);
309 + rrdpush_send_chart_definition(st);
310 sender_commit(host->sender);
322 - rrdset_unlock(st);
311
312 return true;
313 }
@@ -380,13 +368,13 @@ void rrdset_done_push(RRDSET *st) {
368 host->rrdpush_sender_error_shown = 0;
369 }
370
383 - if(dictionary_stats_entries(st->rrddim_root_index) == 0)
371 + if(dictionary_entries(st->rrddim_root_index) == 0)
372 return;
373
374 sender_start(host->sender);
375
376 if(need_to_send_chart_definition(st))
389 - rrdpush_send_chart_definition_nolock(st);
377 + rrdpush_send_chart_definition(st);
378
379 if(rrdpush_send_chart_metrics_nolock(st, host->sender)) {
380 // signal the sender there are more data
streaming/rrdpush.h
+1 -1
@@ -178,7 +178,7 @@ extern void rrdpush_claimed_id(RRDHOST *host);
178 extern int rrdpush_receiver_thread_spawn(struct web_client *w, char *url);
179 extern void rrdpush_sender_thread_stop(RRDHOST *host);
180
181 -extern void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, RRDVAR *rv);
181 +extern void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, const RRDVAR_ACQUIRED *rva);
182 extern void log_stream_connection(const char *client_ip, const char *client_port, const char *api_key, const char *machine_guid, const char *host, const char *msg);
183 extern int connect_to_one_of_destinations(
184 struct rrdpush_destinations *destinations,
streaming/sender.c
+12 -20
@@ -124,33 +124,31 @@ static inline void rrdpush_sender_thread_close_socket(RRDHOST *host) {
124 }
125 }
126
127 -static inline void rrdpush_sender_add_host_variable_to_buffer_nolock(RRDHOST *host, RRDVAR *rv) {
128 - NETDATA_DOUBLE *value = (NETDATA_DOUBLE *)rv->value;
129 -
127 +static inline void rrdpush_sender_add_host_variable_to_buffer_nolock(RRDHOST *host, const RRDVAR_ACQUIRED *rva) {
128 buffer_sprintf(
129 host->sender->build
130 , "VARIABLE HOST %s = " NETDATA_DOUBLE_FORMAT "\n"
133 - , rrdvar_name(rv)
134 - , *value
131 + , rrdvar_name(rva)
132 + , rrdvar2number(rva)
133 );
134
137 - debug(D_STREAM, "RRDVAR pushed HOST VARIABLE %s = " NETDATA_DOUBLE_FORMAT, rrdvar_name(rv), *value);
135 + debug(D_STREAM, "RRDVAR pushed HOST VARIABLE %s = " NETDATA_DOUBLE_FORMAT, rrdvar_name(rva), rrdvar2number(rva));
136 }
137
140 -void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, RRDVAR *rv) {
138 +void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, const RRDVAR_ACQUIRED *rva) {
139 if(host->rrdpush_send_enabled && host->rrdpush_sender_spawn && __atomic_load_n(&host->rrdpush_sender_connected, __ATOMIC_SEQ_CST)) {
140 sender_start(host->sender);
143 - rrdpush_sender_add_host_variable_to_buffer_nolock(host, rv);
141 + rrdpush_sender_add_host_variable_to_buffer_nolock(host, rva);
142 sender_commit(host->sender);
143 }
144 }
145
146
149 -static int rrdpush_sender_thread_custom_host_variables_callback(const char *name __maybe_unused, void *rrdvar_ptr, void *host_ptr) {
150 - RRDVAR *rv = (RRDVAR *)rrdvar_ptr;
147 +static int rrdpush_sender_thread_custom_host_variables_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdvar_ptr __maybe_unused, void *host_ptr) {
148 + const RRDVAR_ACQUIRED *rv = (const RRDVAR_ACQUIRED *)item;
149 RRDHOST *host = (RRDHOST *)host_ptr;
150
153 - if(unlikely(rv->options & RRDVAR_OPTION_CUSTOM_HOST_VAR && rv->type == RRDVAR_TYPE_CALCULATED)) {
151 + if(unlikely(rrdvar_flags(rv) & RRDVAR_FLAG_CUSTOM_HOST_VAR && rrdvar_type(rv) == RRDVAR_TYPE_CALCULATED)) {
152 rrdpush_sender_add_host_variable_to_buffer_nolock(host, rv);
153
154 // return 1, so that the traversal will return the number of variables sent
@@ -163,7 +161,7 @@ static int rrdpush_sender_thread_custom_host_variables_callback(const char *name
161
162 static void rrdpush_sender_thread_send_custom_host_variables(RRDHOST *host) {
163 sender_start(host->sender);
166 - int ret = rrdvar_walkthrough_read(host->rrdvar_root_index, rrdpush_sender_thread_custom_host_variables_callback, host);
164 + int ret = rrdvar_walkthrough_read(host->rrdvars, rrdpush_sender_thread_custom_host_variables_callback, host);
165 (void)ret;
166 sender_commit(host->sender);
167
@@ -173,24 +171,18 @@ static void rrdpush_sender_thread_send_custom_host_variables(RRDHOST *host) {
171 // resets all the chart, so that their definitions
172 // will be resent to the central netdata
173 static void rrdpush_sender_thread_reset_all_charts(RRDHOST *host) {
176 - rrdhost_rdlock(host);
177 -
174 RRDSET *st;
175 rrdset_foreach_read(st, host) {
176 rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
177
178 st->upstream_resync_time = 0;
179
184 - rrdset_rdlock(st);
185 -
180 RRDDIM *rd;
181 rrddim_foreach_read(rd, st)
182 rd->exposed = 0;
189 -
190 - rrdset_unlock(st);
183 + rrddim_foreach_done(rd);
184 }
192 -
193 - rrdhost_unlock(host);
185 + rrdset_foreach_done(st);
186 }
187
188 static inline void rrdpush_sender_thread_data_flush(RRDHOST *host) {
tests/profile/benchmark-dictionary.c
+2 -2
@@ -19,8 +19,8 @@ void netdata_cleanup_and_exit(int ret) { exit(ret); }
19 int main(int argc, char **argv) {
20 if(argc || argv) {;}
21
22 -// DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED|DICTIONARY_FLAG_WITH_STATISTICS);
23 - DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_WITH_STATISTICS);
22 +// DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED|DICT_OPTION_WITH_STATISTICS);
23 + DICTIONARY *dict = dictionary_create(DICT_OPTION_STATS);
24 if(!dict) fatal("Cannot create dictionary.");
25
26 struct rusage start, end;
web/api/badges/web_buffer_svg.c
+10 -4
@@ -893,6 +893,10 @@ int web_client_api_request_v1_badge(RRDHOST *host, struct web_client *w, char *u
893 int group = RRDR_GROUPING_AVERAGE;
894 uint32_t options = 0x00000000;
895
896 + const RRDCALC_ACQUIRED *rca = NULL;
897 + RRDCALC *rc = NULL;
898 + RRDSET *st = NULL;
899 +
900 while(url) {
901 char *value = mystrsep(&url, "&");
902 if(!value || !*value) continue;
@@ -957,7 +961,7 @@ int web_client_api_request_v1_badge(RRDHOST *host, struct web_client *w, char *u
961
962 int scale = (scale_str && *scale_str)?str2i(scale_str):100;
963
960 - RRDSET *st = rrdset_find(host, chart);
964 + st = rrdset_find(host, chart);
965 if(!st) st = rrdset_find_byname(host, chart);
966 if(!st) {
967 buffer_no_cacheable(w->response.data);
@@ -967,9 +971,10 @@ int web_client_api_request_v1_badge(RRDHOST *host, struct web_client *w, char *u
971 }
972 st->last_accessed_time = now_realtime_sec();
973
970 - RRDCALC *rc = NULL;
974 if(alarm) {
972 - rc = rrdcalc_find(st, alarm);
975 + rca = rrdcalc_from_rrdset_get(st, alarm);
976 + rc = rrdcalc_acquired_to_rrdcalc(rca);
977 +
978 if (!rc) {
979 buffer_no_cacheable(w->response.data);
980 buffer_svg(w->response.data, "alarm not found", NAN, "", NULL, NULL, -1, scale, 0, -1, -1, NULL, NULL);
@@ -1143,7 +1148,8 @@ int web_client_api_request_v1_badge(RRDHOST *host, struct web_client *w, char *u
1148 );
1149 }
1150
1146 - cleanup:
1151 +cleanup:
1152 + rrdcalc_from_rrdset_release(st, rca);
1153 buffer_free(dimensions);
1154 return ret;
1155 }
web/api/exporters/shell/allmetrics_shell.c
+8 -13
@@ -25,7 +25,6 @@ static inline size_t shell_name_copy(char *d, const char *s, size_t usable) {
25 void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, const char *filter_string, BUFFER *wb) {
26 analytics_log_shell();
27 SIMPLE_PATTERN *filter = simple_pattern_create(filter_string, NULL, SIMPLE_PATTERN_EXACT);
28 - rrdhost_rdlock(host);
28
29 // for each chart
30 RRDSET *st;
@@ -39,8 +38,6 @@ void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, const char *filter_
38
39 buffer_sprintf(wb, "\n# chart: %s (name: %s)\n", rrdset_id(st), rrdset_name(st));
40 if(rrdset_is_available_for_viewers(st)) {
42 - rrdset_rdlock(st);
43 -
41 // for each dimension
42 RRDDIM *rd;
43 rrddim_foreach_read(rd, st) {
@@ -55,22 +52,23 @@ void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, const char *filter_
52 else {
53 if(rd->multiplier < 0 || rd->divisor < 0) n = -n;
54 n = roundndd(n);
58 - if(!rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN)) total += n;
55 + if(!rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN)) total += n;
56 buffer_sprintf(wb, "NETDATA_%s_%s=\"" NETDATA_DOUBLE_FORMAT_ZERO "\" # %s\n", chart, dimension, n, rrdset_units(st));
57 }
58 }
59 }
60 + rrddim_foreach_done(rd);
61
62 total = roundndd(total);
63 buffer_sprintf(wb, "NETDATA_%s_VISIBLETOTAL=\"" NETDATA_DOUBLE_FORMAT_ZERO "\" # %s\n", chart, total, rrdset_units(st));
66 - rrdset_unlock(st);
64 }
65 }
66 + rrdset_foreach_done(st);
67
68 buffer_strcat(wb, "\n# NETDATA ALARMS RUNNING\n");
69
70 RRDCALC *rc;
73 - foreach_rrdcalc_in_rrdhost(host, rc) {
71 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
72 if(!rc->rrdset) continue;
73
74 char chart[SHELL_ELEMENT_MAX + 1];
@@ -90,8 +88,8 @@ void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, const char *filter_
88
89 buffer_sprintf(wb, "NETDATA_ALARM_%s_%s_STATUS=\"%s\"\n", chart, alarm, rrdcalc_status2string(rc->status));
90 }
91 + foreach_rrdcalc_in_rrdhost_done(rc);
92
94 - rrdhost_unlock(host);
93 simple_pattern_free(filter);
94 }
95
@@ -100,7 +98,6 @@ void rrd_stats_api_v1_charts_allmetrics_shell(RRDHOST *host, const char *filter_
98 void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, const char *filter_string, BUFFER *wb) {
99 analytics_log_json();
100 SIMPLE_PATTERN *filter = simple_pattern_create(filter_string, NULL, SIMPLE_PATTERN_EXACT);
103 - rrdhost_rdlock(host);
101
102 buffer_strcat(wb, "{");
103
@@ -114,8 +111,6 @@ void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, const char *filter_s
111 continue;
112
113 if(rrdset_is_available_for_viewers(st)) {
117 - rrdset_rdlock(st);
118 -
114 buffer_sprintf(
115 wb,
116 "%s\n"
@@ -132,7 +127,7 @@ void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, const char *filter_s
127 rrdset_family(st),
128 rrdset_context(st),
129 rrdset_units(st),
135 - (int64_t)rrdset_last_entry_t_nolock(st));
130 + (int64_t)rrdset_last_entry_t(st));
131
132 chart_counter++;
133 dimension_counter = 0;
@@ -161,14 +156,14 @@ void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, const char *filter_s
156 dimension_counter++;
157 }
158 }
159 + rrddim_foreach_done(rd);
160
161 buffer_strcat(wb, "\n\t\t}\n\t}");
166 - rrdset_unlock(st);
162 }
163 }
164 + rrdset_foreach_done(st);
165
166 buffer_strcat(wb, "\n}");
171 - rrdhost_unlock(host);
167 simple_pattern_free(filter);
168 }
169
web/api/formatters/charts2json.c
+6 -9
@@ -69,7 +69,6 @@ void charts2json(RRDHOST *host, BUFFER *wb, int skip_volatile, int show_archived
69 );
70
71 c = 0;
72 - rrdhost_rdlock(host);
72 rrdset_foreach_read(st, host) {
73 if ((!show_archived && rrdset_is_available_for_viewers(st)) || (show_archived && rrdset_is_archived(st))) {
74 if(c) buffer_strcat(wb, ",");
@@ -82,13 +81,14 @@ void charts2json(RRDHOST *host, BUFFER *wb, int skip_volatile, int show_archived
81 st->last_accessed_time = now;
82 }
83 }
84 + rrdset_foreach_done(st);
85
86 RRDCALC *rc;
87 - foreach_rrdcalc_in_rrdhost(host, rc) {
87 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
88 if(rc->rrdset)
89 alarms++;
90 }
91 - rrdhost_unlock(host);
91 + foreach_rrdcalc_in_rrdhost_done(rc);
92
93 buffer_sprintf(wb
94 , "\n\t}"
@@ -150,9 +150,7 @@ struct array_printer {
150 BUFFER *wb;
151 };
152
153 -static int print_collector_callback(const char *name, void *entry, void *data) {
154 - (void)name;
155 -
153 +static int print_collector_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
154 struct array_printer *ap = (struct array_printer *)data;
155 BUFFER *wb = ap->wb;
156 struct collector *col=(struct collector *) entry;
@@ -167,12 +165,11 @@ static int print_collector_callback(const char *name, void *entry, void *data) {
165 }
166
167 void chartcollectors2json(RRDHOST *host, BUFFER *wb) {
170 - DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
168 + DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
169 RRDSET *st;
170 char name[500];
171
172 time_t now = now_realtime_sec();
175 - rrdhost_rdlock(host);
173 rrdset_foreach_read(st, host) {
174 if (rrdset_is_available_for_viewers(st)) {
175 struct collector col = {
@@ -184,7 +181,7 @@ void chartcollectors2json(RRDHOST *host, BUFFER *wb) {
181 st->last_accessed_time = now;
182 }
183 }
187 - rrdhost_unlock(host);
184 + rrdset_foreach_done(st);
185 struct array_printer ap = {
186 .c = 0,
187 .wb = wb
web/api/formatters/json_wrapper.c
+9 -16
@@ -7,9 +7,7 @@ struct value_output {
7 BUFFER *wb;
8 };
9
10 -static int value_list_output(const char *name, void *entry, void *data) {
11 - (void)name;
12 -
10 +static int value_list_output_callback(const DICTIONARY_ITEM *item __maybe_unused, void *entry, void *data) {
11 struct value_output *ap = (struct value_output *)data;
12 BUFFER *wb = ap->wb;
13 char *output = (char *) entry;
@@ -64,8 +62,6 @@ void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS
62 sq[0] = '"';
63 }
64
67 - if (should_lock)
68 - rrdset_rdlock(r->st);
65 buffer_sprintf(wb, "{\n"
66 " %sapi%s: 1,\n"
67 " %sid%s: %s%s%s,\n"
@@ -83,8 +79,8 @@ void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS
79 , kq, kq, sq, context_mode && temp_rd?rrdset_context(r->st):rrdset_name(r->st), sq
80 , kq, kq, r->update_every
81 , kq, kq, r->st->update_every
86 - , kq, kq, (uint32_t) (context_param_list ? context_param_list->first_entry_t : rrdset_first_entry_t_nolock(r->st))
87 - , kq, kq, (uint32_t) (context_param_list ? context_param_list->last_entry_t : rrdset_last_entry_t_nolock(r->st))
82 + , kq, kq, (uint32_t) (context_param_list ? context_param_list->first_entry_t : rrdset_first_entry_t(r->st))
83 + , kq, kq, (uint32_t) (context_param_list ? context_param_list->last_entry_t : rrdset_last_entry_t(r->st))
84 , kq, kq, (uint32_t)r->before
85 , kq, kq, (uint32_t)r->after
86 , kq, kq, sq, web_client_api_request_v1_data_group_to_string(group_method), sq
@@ -94,9 +90,6 @@ void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS
90
91 buffer_sprintf(wb, "%s,\n %sdimension_names%s: [", sq, kq, kq);
92
97 - if (should_lock)
98 - rrdset_unlock(r->st);
99 -
93 for(c = 0, i = 0, rd = temp_rd?temp_rd:r->st->dimensions; rd && c < r->d ;c++, rd = rd->next) {
94 if(unlikely(r->od[c] & RRDR_DIMENSION_HIDDEN)) continue;
95 if(unlikely((options & RRDR_OPTION_NONZERO) && !(r->od[c] & RRDR_DIMENSION_NONZERO))) continue;
@@ -147,37 +140,37 @@ void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS
140
141 struct value_output co = {.c = 0, .wb = wb};
142
150 - DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
143 + DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
144 for (i = 0, rd = temp_rd ? temp_rd : r->st->dimensions; rd; rd = rd->next) {
145 snprintfz(name, RRD_ID_LENGTH_MAX * 2, "%s:%s", rrddim_id(rd), rrddim_name(rd));
146 int len = snprintfz(output, RRD_ID_LENGTH_MAX * 2 + 7, "[\"%s\",\"%s\"]", rrddim_id(rd), rrddim_name(rd));
147 dictionary_set(dict, name, output, len+1);
148 }
156 - dictionary_walkthrough_read(dict, value_list_output, &co);
149 + dictionary_walkthrough_read(dict, value_list_output_callback, &co);
150 dictionary_destroy(dict);
151
152 co.c = 0;
153 buffer_sprintf(wb, "],\n %sfull_chart_list%s: [", kq, kq);
161 - dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
154 + dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
155 for (i = 0, rd = temp_rd ? temp_rd : r->st->dimensions; rd; rd = rd->next) {
156 int len = snprintfz(output, RRD_ID_LENGTH_MAX * 2 + 7, "[\"%s\",\"%s\"]", rrdset_id(rd->rrdset), rrdset_name(rd->rrdset));
157 snprintfz(name, RRD_ID_LENGTH_MAX * 2, "%s:%s", rrdset_id(rd->rrdset), rrdset_name(rd->rrdset));
158 dictionary_set(dict, name, output, len + 1);
159 }
160
168 - dictionary_walkthrough_read(dict, value_list_output, &co);
161 + dictionary_walkthrough_read(dict, value_list_output_callback, &co);
162 dictionary_destroy(dict);
163
164 RRDSET *st;
165 co.c = 0;
166 buffer_sprintf(wb, "],\n %sfull_chart_labels%s: [", kq, kq);
174 - dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
167 + dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
168 for (i = 0, rd = temp_rd ? temp_rd : r->st->dimensions; rd; rd = rd->next) {
169 st = rd->rrdset;
170 if (st->rrdlabels)
171 rrdlabels_walkthrough_read(st->rrdlabels, fill_formatted_callback, dict);
172 }
180 - dictionary_walkthrough_read(dict, value_list_output, &co);
173 + dictionary_walkthrough_read(dict, value_list_output_callback, &co);
174 dictionary_destroy(dict);
175 buffer_strcat(wb, "],\n");
176 }
web/api/formatters/rrd2json.c
+4 -7
@@ -104,13 +104,11 @@ void build_context_param_list(ONEWAYALLOC *owa, struct context_param **param_lis
104 (*param_list)->rd = NULL;
105 }
106
107 - RRDDIM *rd1;
107 st->last_accessed_time = now_realtime_sec();
109 - rrdset_rdlock(st);
110 -
111 - (*param_list)->first_entry_t = MIN((*param_list)->first_entry_t, rrdset_first_entry_t_nolock(st));
112 - (*param_list)->last_entry_t = MAX((*param_list)->last_entry_t, rrdset_last_entry_t_nolock(st));
108 + (*param_list)->first_entry_t = MIN((*param_list)->first_entry_t, rrdset_first_entry_t(st));
109 + (*param_list)->last_entry_t = MAX((*param_list)->last_entry_t, rrdset_last_entry_t(st));
110
111 + RRDDIM *rd1;
112 rrddim_foreach_read(rd1, st) {
113 RRDDIM *rd = onewayalloc_memdupz(owa, rd1, sizeof(RRDDIM));
114 rd->id = string_dup(rd1->id);
@@ -124,8 +122,7 @@ void build_context_param_list(ONEWAYALLOC *owa, struct context_param **param_lis
122 rd->next = (*param_list)->rd;
123 (*param_list)->rd = rd;
124 }
127 -
128 - rrdset_unlock(st);
125 + rrddim_foreach_done(rd1);
126 }
127
128 void rrd_stats_api_v1_chart(RRDSET *st, BUFFER *wb) {
web/api/formatters/rrdset2json.c
+8 -9
@@ -25,10 +25,8 @@ void chart_labels2json(RRDSET *st, BUFFER *wb, size_t indentation)
25 // generate JSON for the /api/v1/chart API call
26
27 void rrdset2json(RRDSET *st, BUFFER *wb, size_t *dimensions_count, size_t *memory_used, int skip_volatile) {
28 - rrdset_rdlock(st);
29 -
30 - time_t first_entry_t = rrdset_first_entry_t_nolock(st);
31 - time_t last_entry_t = rrdset_last_entry_t_nolock(st);
28 + time_t first_entry_t = rrdset_first_entry_t(st);
29 + time_t last_entry_t = rrdset_last_entry_t(st);
30
31 buffer_sprintf(
32 wb,
@@ -47,7 +45,7 @@ void rrdset2json(RRDSET *st, BUFFER *wb, size_t *dimensions_count, size_t *memor
45 "\t\t\t\"chart_type\": \"%s\",\n",
46 rrdset_id(st),
47 rrdset_name(st),
50 - rrdset_type(st),
48 + rrdset_parts_type(st),
49 rrdset_family(st),
50 rrdset_context(st),
51 rrdset_title(st),
@@ -90,7 +88,7 @@ void rrdset2json(RRDSET *st, BUFFER *wb, size_t *dimensions_count, size_t *memor
88 size_t dimensions = 0;
89 RRDDIM *rd;
90 rrddim_foreach_read(rd, st) {
93 - if(rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN) || rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)) continue;
91 + if(rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN) || rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE)) continue;
92
93 memory += sizeof(RRDDIM) + rd->memsize;
94
@@ -105,6 +103,7 @@ void rrdset2json(RRDSET *st, BUFFER *wb, size_t *dimensions_count, size_t *memor
103
104 dimensions++;
105 }
106 + rrddim_foreach_done(rd);
107
108 if(dimensions_count) *dimensions_count += dimensions;
109 if(memory_used) *memory_used += memory;
@@ -121,7 +120,8 @@ void rrdset2json(RRDSET *st, BUFFER *wb, size_t *dimensions_count, size_t *memor
120 buffer_strcat(wb, ",\n\t\t\t\"alarms\": {\n");
121 size_t alarms = 0;
122 RRDCALC *rc;
124 - foreach_rrdcalc_in_rrdset(st, rc) {
123 + netdata_rwlock_rdlock(&st->alerts.rwlock);
124 + DOUBLE_LINKED_LIST_FOREACH_FORWARD(st->alerts.base, rc, prev, next) {
125 buffer_sprintf(
126 wb,
127 "%s"
@@ -136,6 +136,7 @@ void rrdset2json(RRDSET *st, BUFFER *wb, size_t *dimensions_count, size_t *memor
136
137 alarms++;
138 }
139 + netdata_rwlock_unlock(&st->alerts.rwlock);
140 buffer_sprintf(wb,
141 "\n\t\t\t}"
142 );
@@ -148,6 +149,4 @@ void rrdset2json(RRDSET *st, BUFFER *wb, size_t *dimensions_count, size_t *memor
149 buffer_sprintf(wb,
150 "\n\t\t}"
151 );
151 -
152 - rrdset_unlock(st);
152 }
web/api/queries/query.c
+21 -19
@@ -837,36 +837,38 @@ static int rrddim_find_best_tier_for_timeframe(RRDDIM *rd, time_t after_wanted,
837 static int rrdset_find_natural_update_every_for_timeframe(RRDSET *st, time_t after_wanted, time_t before_wanted, long points_wanted, RRDR_OPTIONS options, int tier) {
838 int ret = st->update_every;
839
840 - if(unlikely(!st->dimensions))
840 + if(unlikely(!rrdset_number_of_dimensions(st)))
841 return ret;
842
843 - rrdset_rdlock(st);
844 - int best_tier;
843 + RRDDIM *first_rd = NULL;
844 + rrddim_foreach_read(first_rd, st) break; rrddim_foreach_done(first_rd);
845 + if(!first_rd)
846 + return ret;
847
848 + int best_tier;
849 if(options & RRDR_OPTION_SELECTED_TIER && tier >= 0 && tier < storage_tiers)
850 best_tier = tier;
848 - else
849 - best_tier = rrddim_find_best_tier_for_timeframe(st->dimensions, after_wanted, before_wanted, points_wanted);
851 + else {
852 + best_tier = rrddim_find_best_tier_for_timeframe(first_rd, after_wanted, before_wanted, points_wanted);
853 + }
854
851 - if(!st->dimensions->tiers[best_tier]) {
855 + if(!first_rd->tiers[best_tier]) {
856 internal_error(
857 true,
858 "QUERY: tier %d on chart '%s', is not initialized", best_tier, rrdset_name(st));
859 }
860 else {
857 - ret = (int)st->dimensions->tiers[best_tier]->tier_grouping * (int)st->update_every;
861 + ret = (int)first_rd->tiers[best_tier]->tier_grouping * (int)st->update_every;
862 if(unlikely(!ret)) {
863 internal_error(
864 true,
865 "QUERY: update_every calculated to be zero on chart '%s', tier_grouping %d, update_every %d",
862 - rrdset_name(st), st->dimensions->tiers[best_tier]->tier_grouping, st->update_every);
866 + rrdset_name(st), first_rd->tiers[best_tier]->tier_grouping, st->update_every);
867
868 ret = st->update_every;
869 }
870 }
871
868 - rrdset_unlock(st);
869 -
872 return ret;
873 }
874
@@ -1510,7 +1512,10 @@ static void rrd2rrdr_log_request_response_metadata(RRDR *r
1512 //, size_t before_slot
1513 , const char *msg
1514 ) {
1513 - netdata_rwlock_rdlock(&r->st->rrdset_rwlock);
1515 +
1516 + time_t first_entry_t = rrdset_first_entry_t(r->st);
1517 + time_t last_entry_t = rrdset_last_entry_t(r->st);
1518 +
1519 info("INTERNAL ERROR: rrd2rrdr() on %s update every %d with %s grouping %s (group: %ld, resampling_time: %ld, resampling_group: %ld), "
1520 "after (got: %zu, want: %zu, req: %ld, db: %zu), "
1521 "before (got: %zu, want: %zu, req: %ld, db: %zu), "
@@ -1532,19 +1537,19 @@ static void rrd2rrdr_log_request_response_metadata(RRDR *r
1537 , (size_t)r->after
1538 , (size_t)after_wanted
1539 , after_requested
1535 - , (size_t)rrdset_first_entry_t_nolock(r->st)
1540 + , (size_t)first_entry_t
1541
1542 // before
1543 , (size_t)r->before
1544 , (size_t)before_wanted
1545 , before_requested
1541 - , (size_t)rrdset_last_entry_t_nolock(r->st)
1546 + , (size_t)last_entry_t
1547
1548 // duration
1549 , (size_t)(r->before - r->after + r->st->update_every)
1550 , (size_t)(before_wanted - after_wanted + r->st->update_every)
1551 , before_requested - after_requested
1547 - , (size_t)((rrdset_last_entry_t_nolock(r->st) - rrdset_first_entry_t_nolock(r->st)) + r->st->update_every)
1552 + , (size_t)((last_entry_t - first_entry_t) + r->st->update_every)
1553
1554 // slot
1555 /*
@@ -1562,7 +1567,6 @@ static void rrd2rrdr_log_request_response_metadata(RRDR *r
1567 // message
1568 , msg
1569 );
1565 - netdata_rwlock_unlock(&r->st->rrdset_rwlock);
1570 }
1571 #endif // NETDATA_INTERNAL_CHECKS
1572
@@ -1727,10 +1731,8 @@ RRDR *rrd2rrdr(
1731 if(!context_param_list) {
1732 relative_period_requested = true;
1733
1730 - rrdset_rdlock(st);
1731 - time_t first_entry_t = rrdset_first_entry_t_nolock(st);
1732 - time_t last_entry_t = rrdset_last_entry_t_nolock(st);
1733 - rrdset_unlock(st);
1734 + time_t first_entry_t = rrdset_first_entry_t(st);
1735 + time_t last_entry_t = rrdset_last_entry_t(st);
1736
1737 if(first_entry_t == 0 || last_entry_t == 0) {
1738 internal_error(true, "QUERY: chart without data detected on '%s'", rrdset_name(st));
web/api/queries/rrdr.c
+2 -2
@@ -111,7 +111,7 @@ RRDR *rrdr_create(ONEWAYALLOC *owa, struct rrdset *st, long n, struct context_pa
111 t = t->next;
112 }
113 } else
114 - rrddim_foreach_read(rd, st) dimensions++;
114 + dimensions = rrdset_number_of_dimensions(st);
115
116 // create the rrdr
117 RRDR *r = rrdr_create_for_x_dimensions(owa, dimensions, n);
@@ -121,7 +121,7 @@ RRDR *rrdr_create(ONEWAYALLOC *owa, struct rrdset *st, long n, struct context_pa
121 // set the hidden flag on hidden dimensions
122 int c;
123 for (c = 0, rd = temp_rd ? temp_rd : st->dimensions; rd; c++, rd = rd->next) {
124 - if (unlikely(rrddim_flag_check(rd, RRDDIM_FLAG_HIDDEN)))
124 + if (unlikely(rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN)))
125 r->od[c] = RRDR_DIMENSION_HIDDEN;
126 else
127 r->od[c] = RRDR_DIMENSION_DEFAULT;
web/api/queries/rrdr.h
+1
@@ -44,6 +44,7 @@ typedef enum rrdr_options {
44
45 // internal ones - not to be exposed to the API
46 RRDR_OPTION_INTERNAL_AR = 0x10000000, // internal use only, to let the formatters we want to render the anomaly rate
47 + RRDR_OPTION_HEALTH_RSRVD1 = 0x80000000, // reserved for RRDCALC_OPTION_NO_CLEAR_NOTIFICATION
48 } RRDR_OPTIONS;
49
50 typedef enum rrdr_value_flag {
web/api/queries/weights.c
+17 -25
@@ -65,10 +65,7 @@ struct register_result {
65 struct register_result *next; // used to link contexts together
66 };
67
68 -static void register_result_insert_callback(const char *name, void *value, void *data) {
69 - (void)name;
70 - (void)data;
71 -
68 +static void register_result_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
69 struct register_result *t = (struct register_result *)value;
70
71 if(t->chart_id) t->chart_id = strdupz(t->chart_id);
@@ -76,9 +73,7 @@ static void register_result_insert_callback(const char *name, void *value, void
73 if(t->dim_name) t->dim_name = strdupz(t->dim_name);
74 }
75
79 -static void register_result_delete_callback(const char *name, void *value, void *data) {
80 - (void)name;
81 - (void)data;
76 +static void register_result_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
77 struct register_result *t = (struct register_result *)value;
78
79 freez((void *)t->chart_id);
@@ -87,7 +82,7 @@ static void register_result_delete_callback(const char *name, void *value, void
82 }
83
84 static DICTIONARY *register_result_init() {
90 - DICTIONARY *results = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
85 + DICTIONARY *results = dictionary_create(DICT_OPTION_SINGLE_THREADED);
86 dictionary_register_insert_callback(results, register_result_insert_callback, results);
87 dictionary_register_delete_callback(results, register_result_delete_callback, results);
88 return results;
@@ -261,11 +256,8 @@ static size_t registered_results_to_json_contexts(DICTIONARY *results, BUFFER *w
256 points, method, group, options, shifts, examined_dimensions, duration, stats);
257
258 DICTIONARY *context_results = dictionary_create(
264 - DICTIONARY_FLAG_SINGLE_THREADED
265 - |DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE
266 - |DICTIONARY_FLAG_NAME_LINK_DONT_CLONE
267 - |DICTIONARY_FLAG_DONT_OVERWRITE_VALUE
268 - );
259 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_NAME_LINK_DONT_CLONE |
260 + DICT_OPTION_DONT_OVERWRITE_VALUE);
261
262 struct register_result *t;
263 dfe_start_read(results, t) {
@@ -605,7 +597,9 @@ static int rrdset_metric_correlations_ks2(RRDSET *st, DICTIONARY *results,
597 // for each dimension
598 RRDDIM *d;
599 int i;
608 - for(i = 0, d = base_rrdr->st->dimensions; d && i < base_rrdr->d; i++, d = d->next) {
600 + rrddim_foreach_read(d, base_rrdr->st) {
601 + if(unlikely((int)d_dfe.counter >= base_rrdr->d)) break;
602 + i = (int)d_dfe.counter; // d_counter is provided by the dictionary
603
604 // skip the not evaluated ones
605 if(unlikely(base_rrdr->od[i] & RRDR_DIMENSION_HIDDEN) || (high_rrdr->od[i] & RRDR_DIMENSION_HIDDEN))
@@ -650,6 +644,7 @@ static int rrdset_metric_correlations_ks2(RRDSET *st, DICTIONARY *results,
644 register_result(results, base_rrdr->st, d, 1.0 - prob, RESULT_IS_BASE_HIGH_RATIO, stats, register_zero);
645 }
646 }
647 + rrddim_foreach_done(d);
648
649 cleanup:
650 rrdr_free(owa, high_rrdr);
@@ -676,7 +671,7 @@ static int rrdset_metric_correlations_volume(RRDSET *st, DICTIONARY *results,
671 usec_t started_usec = now_realtime_usec();
672
673 RRDDIM *d;
679 - for(d = st->dimensions; d ; d = d->next) {
674 + rrddim_foreach_read(d, st) {
675 usec_t now_usec = now_realtime_usec();
676 if(now_usec - started_usec > timeout * USEC_PER_MS)
677 return examined_dimensions;
@@ -766,6 +761,7 @@ static int rrdset_metric_correlations_volume(RRDSET *st, DICTIONARY *results,
761
762 register_result(results, st, d, pcent, flags, stats, register_zero);
763 }
764 + rrddim_foreach_done(d);
765
766 return examined_dimensions;
767 }
@@ -787,7 +783,7 @@ static int rrdset_weights_anomaly_rate(RRDSET *st, DICTIONARY *results,
783 usec_t started_usec = now_realtime_usec();
784
785 RRDDIM *d;
790 - for(d = st->dimensions; d ; d = d->next) {
786 + rrddim_foreach_read(d, st) {
787 usec_t now_usec = now_realtime_usec();
788 if(now_usec - started_usec > timeout * USEC_PER_MS)
789 return examined_dimensions;
@@ -814,6 +810,7 @@ static int rrdset_weights_anomaly_rate(RRDSET *st, DICTIONARY *results,
810 if(ret == HTTP_RESP_OK || !value_is_null || netdata_double_isnumber(average))
811 register_result(results, st, d, average, 0, stats, register_zero);
812 }
813 + rrddim_foreach_done(d);
814
815 return examined_dimensions;
816 }
@@ -853,7 +850,7 @@ static size_t spread_results_evenly(DICTIONARY *results, WEIGHTS_STATS *stats) {
850 struct register_result *t;
851
852 // count the dimensions
856 - size_t dimensions = dictionary_stats_entries(results);
853 + size_t dimensions = dictionary_entries(results);
854 if(!dimensions) return 0;
855
856 if(stats->max_base_high_ratio == 0.0)
@@ -911,7 +908,7 @@ int web_api_v1_weights(RRDHOST *host, BUFFER *wb, WEIGHTS_METHOD method, WEIGHTS
908 WEIGHTS_STATS stats = {};
909
910 DICTIONARY *results = register_result_init();
914 - DICTIONARY *charts = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED|DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE);;
911 + DICTIONARY *charts = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_VALUE_LINK_DONT_CLONE);;
912 char *error = NULL;
913 int resp = HTTP_RESP_OK;
914
@@ -1003,14 +1000,13 @@ int web_api_v1_weights(RRDHOST *host, BUFFER *wb, WEIGHTS_METHOD method, WEIGHTS
1000 // dont lock here and wait for results
1001 // get the charts and run mc after
1002 RRDSET *st;
1006 - rrdhost_rdlock(host);
1003 rrdset_foreach_read(st, host) {
1004 if (rrdset_is_available_for_viewers(st)) {
1005 if(!contexts || simple_pattern_matches(contexts, rrdset_context(st)))
1006 dictionary_set(charts, rrdset_name(st), NULL, 0);
1007 }
1008 }
1013 - rrdhost_unlock(host);
1009 + rrdset_foreach_done(st);
1010
1011 size_t examined_dimensions = 0;
1012 void *ptr;
@@ -1030,11 +1026,9 @@ int web_api_v1_weights(RRDHOST *host, BUFFER *wb, WEIGHTS_METHOD method, WEIGHTS
1026 goto cleanup;
1027 }
1028
1033 - st = rrdset_find_byname(host, ptr_name);
1029 + st = rrdset_find_byname(host, ptr_dfe.name); // ptr_dfe.name is provided by dictionary
1030 if(!st) continue;
1031
1036 - rrdset_rdlock(st);
1037 -
1032 switch(method) {
1033 case WEIGHTS_METHOD_ANOMALY_RATE:
1034 options |= RRDR_OPTION_ANOMALY_BIT;
@@ -1066,8 +1060,6 @@ int web_api_v1_weights(RRDHOST *host, BUFFER *wb, WEIGHTS_METHOD method, WEIGHTS
1060 &stats, register_zero);
1061 break;
1062 }
1069 -
1070 - rrdset_unlock(st);
1063 }
1064 dfe_done(ptr);
1065
web/api/web_api_v1.c
+3 -5
@@ -710,14 +710,13 @@ inline int web_client_api_request_v1_data(RRDHOST *host, struct web_client *w, c
710 chart_labels_filter_pattern = simple_pattern_create(chart_labels_filter, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
711
712 STRING *context_string = string_strdupz(context);
713 - rrdhost_rdlock(host);
713 rrdset_foreach_read(st1, host) {
714 if (st1->context == context_string &&
715 (!chart_label_key_pattern || rrdlabels_match_simple_pattern_parsed(st1->rrdlabels, chart_label_key_pattern, ':')) &&
716 (!chart_labels_filter_pattern || rrdlabels_match_simple_pattern_parsed(st1->rrdlabels, chart_labels_filter_pattern, ':')))
717 build_context_param_list(owa, &context_param_list, st1);
718 }
720 - rrdhost_unlock(host);
719 + rrdset_foreach_done(st1);
720 string_freez(context_string);
721
722 if (likely(context_param_list && context_param_list->rd)) // Just set the first one
@@ -1054,8 +1053,7 @@ inline int web_client_api_request_v1_registry(RRDHOST *host, struct web_client *
1053 static inline void web_client_api_request_v1_info_summary_alarm_statuses(RRDHOST *host, BUFFER *wb) {
1054 int alarm_normal = 0, alarm_warn = 0, alarm_crit = 0;
1055 RRDCALC *rc;
1057 - rrdhost_rdlock(host);
1058 - foreach_rrdcalc_in_rrdhost(host, rc) {
1056 + foreach_rrdcalc_in_rrdhost_read(host, rc) {
1057 if(unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
1058 continue;
1059
@@ -1070,7 +1068,7 @@ static inline void web_client_api_request_v1_info_summary_alarm_statuses(RRDHOST
1068 alarm_normal++;
1069 }
1070 }
1073 - rrdhost_unlock(host);
1071 + foreach_rrdcalc_in_rrdhost_done(rc);
1072 buffer_sprintf(wb, "\t\t\"normal\": %d,\n", alarm_normal);
1073 buffer_sprintf(wb, "\t\t\"warning\": %d,\n", alarm_warn);
1074 buffer_sprintf(wb, "\t\t\"critical\": %d\n", alarm_crit);