Rrdcontext (#13335)
* type checking on dictionary return values * first STRING implementation, used by DICTIONARY and RRDLABEL * enable AVL compilation of STRING * Initial functions to store context info * Call simple test functions * Add host_id when getting charts * Allow host to be null and in this case it will process the localhost * Simplify init Do not use strdupz - link directly to sqlite result set * Init the database during startup * make it compile - no functionality yet * intermediate commit * intermidiate * first interface to sql * loading instances * check if we need to update cloud * comparison of rrdcontext on conflict * merge context titles * rrdcontext public interface; statistics on STRING; scratchpad on DICTIONARY * dictionaries maintain version numbers; rrdcontext api * cascading changes * first operational cleanup * string unittest * proper cleanup of referenced dictionaries * added rrdmetrics * rrdmetric starting retention * Add fields to context Adjuct context creation and delete * Memory cleanup * Fix get context list Fix memory double free in tests Store context with two hosts * calculated retention * rrdcontext retention with collection * Persist database and shutdown * loading all from sql * Get chart list and dimension list changes * fully working attempt 1 * fully working attempt 2 * missing archived flag from log * fixed archived / collected * operational * proper cleanup * cleanup - implemented all interface functions - dictionary react callback triggers after the dictionary is unlocked * track all reasons for changes * proper tracking of reasons of changes * fully working thread * better versioning of contexts * fix string indexing with AVL * running version per context vs hub version; ifdef dbengine * added option to disable rrdmetrics * release old context when a chart changes context * cleanup properly * renamed config * cleanup contexts; general cleanup; * deletion inline with dequeue; lots of cleanup; child connected/disconnected * ml should start after rrdcontext * added missing NULL to ri->rrdset; rrdcontext flags are now only changed under a mutex lock * fix buggy STRING under AVL * Rework database initialization Add migration logic to the context database * fix data race conditions during context deletion * added version hash algorithm * fix string over AVL * update aclk-schemas * compile new ctx related protos * add ctx stream message utils * add context messages * add dummy rx message handlers * add the new topics * add ctx capability * add helper functions to send the new messages * update cmake build to not fail * update topic names * handle rrdcontext_enabled * add more functions * fatal on OOM cases instead of return NULL * silence unknown query type error * fully working attempt 1 * fully working attempt 2 * allow compiling without ACLK * added family to the context * removed excess character in UUID * smarter merging of titles and families * Database migration code to add family Add family to SQL_CHART_DATA and VERSIONED_CONTEXT_DATA * add family to context message * enable ctx in communication * hardcoded enabled contexts * Add hard code for CTX * add update node collectors to json * add context message log * fix log about last_time_t * fix collected flags for queued items * prevent crash on charts cleanup * fix bug in AVL indexing of dictionaries; make sure react callback of dictionaries has a reference counter, which is acquired while the dictionary is locked * fixed dictionary unittest * strict policy to cleanup and garbage collector * fix db rotation and garbage collection timings * remove deadlock * proper garbage collection - a lot faster retention recalculation * Added not NULL in database columns Remove migration code for context -- we will ship with version 1 of the table schema Added define for query in tests to detect localhost * Use UUID_STR_LEN instead of GUID_LEN + 1 Use realistic timestamps when adding test data in the database * Add NULL checks for passed parameters * Log deleted context when compiled with NETDATA_INTERNAL_CHECKS * Error checking for null host id * add missing ContextsCheckpoint log convertor * Fix spelling in VACCUM * Hold additional information for host -- prepare to load archived hosts on startup * Make sure claim id is valid * is_get_claimed is actually get the current claim id * Simplify ctx get chart list query * remove env negotiation * fix string unittest when there are some strings already in the index * propagate live-retention flag upstream; cleanup all update reasons; updated instances logging; automated attaching started/stopped collecting flags; * first implementation of /api/v1/contexts * full contexts API; updated swagger * disabled debugging; rrdcontext enabled by default * final cleanup and renaming of global variables * return current time on currently collected contexts, charts and dimensions * added option "deepscan" to the API to have the server refresh the retention and recalculate the contexts on the fly * fixed identation of yaml * Add constrains to the host table * host->node_id may not be available * new capabilities * lock the context while rendering json * update aclk-schemas * added permanent labels to all charts about plugin, module and family; added labels to all proc plugin modules * always add the labels * allow merging of families down to [x] * dont show uuids by default, added option to enable them; response is now accepting after,before to show only data for a specific timeframe; deleted items are only shown when "deleted" is requested; hub version is now shown when "queue" is requested * Use the localhost claim id * Fix to handle host constrains better * cgroups: add "k8s." prefix to chart context in k8s * Improve sqlite metadata version migration check * empty values set to "[none]"; fix labels unit test to reflect that * Check if we reached the version we want first (address CODACY report re: Array index 'i' is used before limits check) * Rewrite condition to address CODACY report (Redundant condition: t->filter_callback. '!A || (A && B)' is equivalent to '!A || B') * Properly unlock context * fixed memory leak on rrdcontexts - it was not freeing all dictionaries in rrdhost; added wait of up to 100ms on dictionary_destroy() to give time to dictionaries to release their items before destroying them * fixed memory leak on rrdlabels not freed on rrdinstances * fixed leak when dimensions and charts are redefined * Mark entries for charts and dimensions as submitted to the cloud 3600 seconds after their creation Mark entries for charts and dimensions as updated (confirmed by the cloud) 1800 seconds after their submission * renamed struct string * update cgroups alarms * fixed codacy suggestions * update dashboard info * fix k8s_cgroup_10s_received_packets_storm alarm * added filtering options to /api/v1/contexts and /api/v1/context * fix eslint * fix eslint * Fix pointer binding for host / chart uuids * Fix cgroups unit tests * fixed non-retention updates not propagated upstream * removed non-fatal fatals * Remove context from 2 way string merge. * Move string_2way_merge to dictionary.c * Add 2-way string merge tests. * split long lines * fix indentation in netdata-swagger.yaml * update netdata-swagger.json * yamllint please * remove the deleted flag when a context is collected * fix yaml warning in swagger * removed non-fatal fatals * charts should now be able to switch contexts * allow deletion of unused metrics, instances and contexts * keep the queued flag * cleanup old rrdinstance labels * dont hide objects when there is no filter; mark objects as deleted when there are no sub-objects * delete old instances once they changed context * delete all instances and contexts that do not have sub-objects * more precise transitions * Load archived hosts on startup (part 1) * update the queued time every time * disable by default; dedup deleted dimensions after snapshot * Load archived hosts on startup (part 2) * delayed processing of events until charts are being collected * remove dont-trigger flag when object is collected * polish all triggers given the new dont_process flag * Remove always true condition Enums for readbility / create_host_callback only if ACLK is enabled (for now) * Skip retention message if context streaming is enabled Add messages in the access log if context streaming is enabled * Check for node id being a UUID that can be parsed Improve error check / reporting when loading archived hosts and creating ACLK sync threads * collected, archived, deleted are now mutually exclusive * Enable the "orphan" handling for now Remove dead code Fix memory leak on free host * Queue charts and dimensions will be no-op if host is set to stream contexts * removed unused parameter and made sure flags are set on rrdcontext insert * make the rrdcontext thread abort mid-work when exiting * Skip chart hash computation and storage if contexts streaming is enabled Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com> Co-authored-by: Timo <timotej@netdata.cloud> Co-authored-by: ilyam8 <ilya@netdata.cloud> Co-authored-by: Vladimir Kobal <vlad@prokk.net> Co-authored-by: Vasilis Kalintiris <vasilis@netdata.cloud>
Costa Tsaousis committed
Jul 24, 2022 at 22:33 UTC
291b978282e1066a43a80658bd1b49be0fbb2eaf
82 files changed
+6978
-779
CMakeLists.txt
+12
@@ -710,6 +710,8 @@ set(RRD_PLUGIN_FILES
710
database/rrdcalc.h
711
database/rrdcalctemplate.c
712
database/rrdcalctemplate.h
713
+ database/rrdcontext.c
714
+ database/rrdcontext.h
715
database/rrddim.c
716
database/rrddimvar.c
717
database/rrddimvar.h
@@ -729,6 +731,8 @@ set(RRD_PLUGIN_FILES
731
database/ram/rrddim_mem.h
732
database/sqlite/sqlite_functions.c
733
database/sqlite/sqlite_functions.h
734
+ database/sqlite/sqlite_context.c
735
+ database/sqlite/sqlite_context.h
736
database/sqlite/sqlite_db_migration.c
737
database/sqlite/sqlite_db_migration.h
738
database/sqlite/sqlite_aclk.c
@@ -881,6 +885,8 @@ set(ACLK_FILES
885
aclk/aclk_charts_api.h
886
aclk/aclk_alarm_api.c
887
aclk/aclk_alarm_api.h
888
+ aclk/aclk_contexts_api.c
889
+ aclk/aclk_contexts_api.h
890
mqtt_websockets/src/mqtt_wss_client.c
891
mqtt_websockets/src/include/mqtt_wss_client.h
892
mqtt_websockets/src/mqtt_wss_log.c
@@ -917,6 +923,10 @@ set(ACLK_FILES
923
aclk/schema-wrappers/capability.h
924
aclk/schema-wrappers/proto_2_json.cc
925
aclk/schema-wrappers/proto_2_json.h
926
+ aclk/schema-wrappers/context_stream.cc
927
+ aclk/schema-wrappers/context_stream.h
928
+ aclk/schema-wrappers/context.cc
929
+ aclk/schema-wrappers/context.h
930
aclk/schema-wrappers/schema_wrappers.h
931
aclk/schema-wrappers/schema_wrapper_utils.cc
932
aclk/schema-wrappers/schema_wrapper_utils.h
@@ -1231,6 +1241,8 @@ set(ACLK_PROTO_DEFS
1241
aclk/aclk-schemas/proto/nodeinstance/connection/v1/connection.proto
1242
aclk/aclk-schemas/proto/nodeinstance/create/v1/creation.proto
1243
aclk/aclk-schemas/proto/nodeinstance/info/v1/info.proto
1244
+ aclk/aclk-schemas/proto/context/v1/context.proto
1245
+ aclk/aclk-schemas/proto/context/v1/stream.proto
1246
)
1247
PROTOBUF_ACLK_GENERATE_CPP(ACLK_PROTO_BUILT_SRCS ACLK_PROTO_BUILT_HDRS ${ACLK_PROTO_DEFS})
1248
Makefile.am
+24
@@ -434,6 +434,8 @@ RRD_PLUGIN_FILES = \
434
database/rrdcalc.h \
435
database/rrdcalctemplate.c \
436
database/rrdcalctemplate.h \
437
+ database/rrdcontext.c \
438
+ database/rrdcontext.h \
439
database/rrddim.c \
440
database/rrddimvar.c \
441
database/rrddimvar.h \
@@ -453,6 +455,8 @@ RRD_PLUGIN_FILES = \
455
database/ram/rrddim_mem.h \
456
database/sqlite/sqlite_functions.c \
457
database/sqlite/sqlite_functions.h \
458
+ database/sqlite/sqlite_context.c \
459
+ database/sqlite/sqlite_context.h \
460
database/sqlite/sqlite_db_migration.c \
461
database/sqlite/sqlite_db_migration.h \
462
database/sqlite/sqlite_aclk.c \
@@ -706,6 +710,8 @@ ACLK_FILES = \
710
aclk/aclk_charts_api.h \
711
aclk/aclk_alarm_api.c \
712
aclk/aclk_alarm_api.h \
713
+ aclk/aclk_contexts_api.c \
714
+ aclk/aclk_contexts_api.h \
715
aclk/schema-wrappers/connection.cc \
716
aclk/schema-wrappers/connection.h \
717
aclk/schema-wrappers/node_connection.cc \
@@ -729,6 +735,10 @@ ACLK_FILES = \
735
aclk/schema-wrappers/schema_wrappers.h \
736
aclk/schema-wrappers/schema_wrapper_utils.cc \
737
aclk/schema-wrappers/schema_wrapper_utils.h \
738
+ aclk/schema-wrappers/context_stream.cc \
739
+ aclk/schema-wrappers/context_stream.h \
740
+ aclk/schema-wrappers/context.cc \
741
+ aclk/schema-wrappers/context.h \
742
$(NULL)
743
744
mqtt_websockets/src/mqtt_wss_client.$(OBJEXT) : CFLAGS += -Wno-unused-result
@@ -746,6 +756,8 @@ ACLK_PROTO_DEFINITIONS = \
756
aclk/aclk-schemas/proto/nodeinstance/connection/v1/connection.proto \
757
aclk/aclk-schemas/proto/nodeinstance/create/v1/creation.proto \
758
aclk/aclk-schemas/proto/nodeinstance/info/v1/info.proto \
759
+ aclk/aclk-schemas/proto/context/v1/context.proto \
760
+ aclk/aclk-schemas/proto/context/v1/stream.proto \
761
$(NULL)
762
763
dist_noinst_DATA += $(ACLK_PROTO_DEFINITIONS)
@@ -774,6 +786,10 @@ ACLK_PROTO_BUILT_FILES = aclk/aclk-schemas/proto/agent/v1/connection.pb.cc \
786
aclk/aclk-schemas/proto/alarm/v1/stream.pb.h \
787
aclk/aclk-schemas/proto/nodeinstance/info/v1/info.pb.cc \
788
aclk/aclk-schemas/proto/nodeinstance/info/v1/info.pb.h \
789
+ aclk/aclk-schemas/proto/context/v1/context.pb.cc \
790
+ aclk/aclk-schemas/proto/context/v1/context.pb.h \
791
+ aclk/aclk-schemas/proto/context/v1/stream.pb.cc \
792
+ aclk/aclk-schemas/proto/context/v1/stream.pb.h \
793
$(NULL)
794
795
BUILT_SOURCES += $(ACLK_PROTO_BUILT_FILES)
@@ -828,6 +844,14 @@ aclk/aclk-schemas/proto/nodeinstance/info/v1/info.pb.cc \
844
aclk/aclk-schemas/proto/nodeinstance/info/v1/info.pb.h: aclk/aclk-schemas/proto/nodeinstance/info/v1/info.proto
845
$(PROTOC) -I=aclk/aclk-schemas --cpp_out=$(builddir)/aclk/aclk-schemas $^
846
847
+aclk/aclk-schemas/proto/context/v1/context.pb.cc \
848
+aclk/aclk-schemas/proto/context/v1/context.pb.h: aclk/aclk-schemas/proto/context/v1/context.proto
849
+ $(PROTOC) -I=aclk/aclk-schemas --cpp_out=$(builddir)/aclk/aclk-schemas $^
850
+
851
+aclk/aclk-schemas/proto/context/v1/stream.pb.cc \
852
+aclk/aclk-schemas/proto/context/v1/stream.pb.h: aclk/aclk-schemas/proto/context/v1/stream.proto
853
+ $(PROTOC) -I=aclk/aclk-schemas --cpp_out=$(builddir)/aclk/aclk-schemas $^
854
+
855
endif #ENABLE_ACLK
856
857
ACLK_ALWAYS_BUILD_FILES = \
aclk/aclk-schemas
+1
-1
@@ -1 +1 @@
1
-Subproject commit fa46ccca237a9bdb613b3b1f2809a25b7b45c7c4
1
+Subproject commit 3252118bd547640251356629f0df05eaf952ac39
aclk/aclk.c
+28
-4
@@ -141,12 +141,12 @@ static int wait_till_cloud_enabled()
141
static int wait_till_agent_claimed(void)
142
{
143
//TODO prevent malloc and freez
144
- char *agent_id = is_agent_claimed();
144
+ char *agent_id = get_agent_claimid();
145
while (likely(!agent_id)) {
146
sleep_usec(USEC_PER_SEC * 1);
147
if (netdata_exit)
148
return 1;
149
- agent_id = is_agent_claimed();
149
+ agent_id = get_agent_claimid();
150
}
151
freez(agent_id);
152
return 0;
@@ -769,6 +769,16 @@ void aclk_host_state_update(RRDHOST *host, int cmd)
769
};
770
node_state_update.node_id = mallocz(UUID_STR_LEN);
771
uuid_unparse_lower(node_id, (char*)node_state_update.node_id);
772
+
773
+ struct capability caps[] = {
774
+ { .name = "proto", .version = 1, .enabled = 1 },
775
+ { .name = "ml", .version = ml_capable(localhost), .enabled = ml_enabled(host) },
776
+ { .name = "mc", .version = enable_metric_correlations ? metric_correlations_version : 0, .enabled = enable_metric_correlations },
777
+ { .name = "ctx", .version = 1, .enabled = rrdcontext_enabled },
778
+ { .name = NULL, .version = 0, .enabled = 0 }
779
+ };
780
+ node_state_update.capabilities = caps;
781
+
782
rrdhost_aclk_state_lock(localhost);
783
node_state_update.claim_id = localhost->aclk_state.claimed_id;
784
query->data.bin_payload.payload = generate_node_instance_connection(&query->data.bin_payload.size, &node_state_update);
@@ -801,6 +811,20 @@ void aclk_send_node_instances()
811
};
812
node_state_update.node_id = mallocz(UUID_STR_LEN);
813
uuid_unparse_lower(list->node_id, (char*)node_state_update.node_id);
814
+
815
+ char host_id[UUID_STR_LEN];
816
+ uuid_unparse_lower(list->host_id, host_id);
817
+
818
+ RRDHOST *host = rrdhost_find_by_guid(host_id, 0);
819
+ struct capability caps[] = {
820
+ { .name = "proto", .version = 1, .enabled = 1 },
821
+ { .name = "ml", .version = ml_capable(localhost), .enabled = host ? ml_enabled(host) : 0 },
822
+ { .name = "mc", .version = enable_metric_correlations ? metric_correlations_version : 0, .enabled = enable_metric_correlations },
823
+ { .name = "ctx", .version = 1, .enabled = rrdcontext_enabled },
824
+ { .name = NULL, .version = 0, .enabled = 0 }
825
+ };
826
+ node_state_update.capabilities = caps;
827
+
828
rrdhost_aclk_state_lock(localhost);
829
node_state_update.claim_id = localhost->aclk_state.claimed_id;
830
query->data.bin_payload.payload = generate_node_instance_connection(&query->data.bin_payload.size, &node_state_update);
@@ -913,7 +937,7 @@ char *ng_aclk_state(void)
937
);
938
buffer_sprintf(wb, "Protocol Used: Protobuf\nMQTT Version: %d\nClaimed: ", use_mqtt_5 ? 5 : 3);
939
916
- char *agent_id = is_agent_claimed();
940
+ char *agent_id = get_agent_claimid();
941
if (agent_id == NULL)
942
buffer_strcat(wb, "No\n");
943
else {
@@ -1079,7 +1103,7 @@ char *ng_aclk_state_json(void)
1103
json_object_array_add(grp, tmp);
1104
json_object_object_add(msg, "protocols-supported", grp);
1105
1082
- char *agent_id = is_agent_claimed();
1106
+ char *agent_id = get_agent_claimid();
1107
tmp = json_object_new_boolean(agent_id != NULL);
1108
json_object_object_add(msg, "agent-claimed", tmp);
1109
aclk/aclk_api.c
+1
@@ -16,6 +16,7 @@ int aclk_disable_runtime = 0;
16
17
int aclk_stats_enabled;
18
int use_mqtt_5 = 0;
19
+int aclk_ctx_based = 0;
20
21
#define ACLK_IMPL_KEY_NAME "aclk implementation"
22
aclk/aclk_api.h
+1
@@ -20,6 +20,7 @@ extern int aclk_stats_enabled;
20
extern int aclk_alert_reloaded;
21
22
extern int use_mqtt_5;
23
+extern int aclk_ctx_based;
24
25
#ifdef ENABLE_ACLK
26
void *aclk_starter(void *ptr);
aclk/aclk_contexts_api.c
new
+23
@@ -0,0 +1,23 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "aclk_query_queue.h"
4
+
5
+#include "aclk_contexts_api.h"
6
+
7
+void aclk_send_contexts_snapshot(contexts_snapshot_t data)
8
+{
9
+ aclk_query_t query = aclk_query_new(PROTO_BIN_MESSAGE);
10
+ query->data.bin_payload.topic = ACLK_TOPICID_CTXS_SNAPSHOT;
11
+ query->data.bin_payload.payload = contexts_snapshot_2bin(data, &query->data.bin_payload.size);
12
+ query->data.bin_payload.msg_name = "ContextsSnapshot";
13
+ QUEUE_IF_PAYLOAD_PRESENT(query);
14
+}
15
+
16
+void aclk_send_contexts_updated(contexts_updated_t data)
17
+{
18
+ aclk_query_t query = aclk_query_new(PROTO_BIN_MESSAGE);
19
+ query->data.bin_payload.topic = ACLK_TOPICID_CTXS_UPDATED;
20
+ query->data.bin_payload.payload = contexts_updated_2bin(data, &query->data.bin_payload.size);
21
+ query->data.bin_payload.msg_name = "ContextsUpdated";
22
+ QUEUE_IF_PAYLOAD_PRESENT(query);
23
+}
aclk/aclk_contexts_api.h
new
+12
@@ -0,0 +1,12 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+#ifndef ACLK_CONTEXTS_API_H
3
+#define ACLK_CONTEXTS_API_H
4
+
5
+#include "schema-wrappers/schema_wrappers.h"
6
+
7
+
8
+void aclk_send_contexts_snapshot(contexts_snapshot_t data);
9
+void aclk_send_contexts_updated(contexts_updated_t data);
10
+
11
+#endif /* ACLK_CONTEXTS_API_H */
12
+
aclk/aclk_otp.c
+6
-3
@@ -493,7 +493,7 @@ int aclk_get_mqtt_otp(RSA *p_key, char **mqtt_id, char **mqtt_usr, char **mqtt_p
493
unsigned char *challenge;
494
int challenge_bytes;
495
496
- char *agent_id = is_agent_claimed();
496
+ char *agent_id = get_agent_claimid();
497
if (agent_id == NULL) {
498
error("Agent was not claimed - cannot perform challenge/response");
499
return 1;
@@ -836,7 +836,7 @@ int aclk_get_env(aclk_env_t *env, const char* aclk_hostname, int aclk_port) {
836
837
req.request_type = HTTP_REQ_GET;
838
839
- char *agent_id = is_agent_claimed();
839
+ char *agent_id = get_agent_claimid();
840
if (agent_id == NULL)
841
{
842
error("Agent was not claimed - cannot perform challenge/response");
@@ -844,7 +844,10 @@ int aclk_get_env(aclk_env_t *env, const char* aclk_hostname, int aclk_port) {
844
return 1;
845
}
846
847
- buffer_sprintf(buf, "/api/v1/env?v=%s&cap=proto&claim_id=%s", &(VERSION[1]) /* skip 'v' at beginning */, agent_id);
847
+ if (rrdcontext_enabled)
848
+ buffer_sprintf(buf, "/api/v1/env?v=%s&cap=proto,ctx&claim_id=%s", &(VERSION[1]) /* skip 'v' at beginning */, agent_id);
849
+ else
850
+ buffer_sprintf(buf, "/api/v1/env?v=%s&cap=proto&claim_id=%s", &(VERSION[1]) /* skip 'v' at beginning */, agent_id);
851
852
freez(agent_id);
853
aclk/aclk_query.c
+1
@@ -278,6 +278,7 @@ const char *aclk_query_get_name(aclk_query_type_t qt)
278
case ALARM_PROVIDE_CFG: return "provide_alarm_config";
279
case ALARM_SNAPSHOT: return "alarm_snapshot";
280
case UPDATE_NODE_COLLECTORS: return "update_node_collectors";
281
+ case PROTO_BIN_MESSAGE: return "generic_binary_proto_message";
282
default:
283
error_report("Unknown query type used %d", (int) qt);
284
return "unknown";
aclk/aclk_query_queue.c
+1
@@ -122,6 +122,7 @@ void aclk_query_free(aclk_query_t query)
122
case ALARM_PROVIDE_CFG:
123
case ALARM_SNAPSHOT:
124
case UPDATE_NODE_COLLECTORS:
125
+ case PROTO_BIN_MESSAGE:
126
if (!use_mqtt_5)
127
freez(query->data.bin_payload.payload);
128
break;
aclk/aclk_query_queue.h
+1
@@ -23,6 +23,7 @@ typedef enum {
23
ALARM_PROVIDE_CFG,
24
ALARM_SNAPSHOT,
25
UPDATE_NODE_COLLECTORS,
26
+ PROTO_BIN_MESSAGE,
27
ACLK_QUERY_TYPE_COUNT // always keep this as last
28
} aclk_query_type_t;
29
aclk/aclk_rx_msgs.c
+47
@@ -289,6 +289,15 @@ int create_node_instance_result(const char *msg, size_t msg_len)
289
}
290
}
291
292
+ struct capability caps[] = {
293
+ { .name = "proto", .version = 1, .enabled = 1 },
294
+ { .name = "ml", .version = ml_capable(localhost), .enabled = host ? ml_enabled(host) : 0 },
295
+ { .name = "mc", .version = enable_metric_correlations ? metric_correlations_version : 0, .enabled = enable_metric_correlations },
296
+ { .name = "ctx", .version = 1, .enabled = rrdcontext_enabled },
297
+ { .name = NULL, .version = 0, .enabled = 0 }
298
+ };
299
+ node_state_update.capabilities = caps;
300
+
301
rrdhost_aclk_state_lock(localhost);
302
node_state_update.claim_id = localhost->aclk_state.claimed_id;
303
query->data.bin_payload.payload = generate_node_instance_connection(&query->data.bin_payload.size, &node_state_update);
@@ -313,6 +322,7 @@ int send_node_instances(const char *msg, size_t msg_len)
322
323
int stream_charts_and_dimensions(const char *msg, size_t msg_len)
324
{
325
+ aclk_ctx_based = 0;
326
stream_charts_and_dims_t res = parse_stream_charts_and_dims(msg, msg_len);
327
if (!res.claim_id || !res.node_id) {
328
error("Error parsing StreamChartsAndDimensions msg");
@@ -426,6 +436,41 @@ int handle_disconnect_req(const char *msg, size_t msg_len)
436
return 0;
437
}
438
439
+int contexts_checkpoint(const char *msg, size_t msg_len)
440
+{
441
+ aclk_ctx_based = 1;
442
+
443
+ struct ctxs_checkpoint *cmd = parse_ctxs_checkpoint(msg, msg_len);
444
+ if (!cmd)
445
+ return 1;
446
+
447
+ rrdcontext_hub_checkpoint_command(cmd);
448
+
449
+ freez(cmd->claim_id);
450
+ freez(cmd->node_id);
451
+ freez(cmd);
452
+ return 0;
453
+}
454
+
455
+int stop_streaming_contexts(const char *msg, size_t msg_len)
456
+{
457
+ if (!aclk_ctx_based) {
458
+ error_report("Received StopStreamingContexts message but context based communication was not enabled (Cloud violated the protocol). Ignoring message");
459
+ return 1;
460
+ }
461
+
462
+ struct stop_streaming_ctxs *cmd = parse_stop_streaming_ctxs(msg, msg_len);
463
+ if (!cmd)
464
+ return 1;
465
+
466
+ rrdcontext_hub_stop_streaming_command(cmd);
467
+
468
+ freez(cmd->claim_id);
469
+ freez(cmd->node_id);
470
+ freez(cmd);
471
+ return 0;
472
+}
473
+
474
typedef struct {
475
const char *name;
476
simple_hash_t name_hash;
@@ -444,6 +489,8 @@ new_cloud_rx_msg_t rx_msgs[] = {
489
{ .name = "SendAlarmConfiguration", .name_hash = 0, .fnc = send_alarm_configuration },
490
{ .name = "SendAlarmSnapshot", .name_hash = 0, .fnc = send_alarm_snapshot },
491
{ .name = "DisconnectReq", .name_hash = 0, .fnc = handle_disconnect_req },
492
+ { .name = "ContextsCheckpoint", .name_hash = 0, .fnc = contexts_checkpoint },
493
+ { .name = "StopStreamingContexts", .name_hash = 0, .fnc = stop_streaming_contexts },
494
{ .name = NULL, .name_hash = 0, .fnc = NULL },
495
};
496
aclk/aclk_tx_msgs.c
+1
@@ -248,6 +248,7 @@ uint16_t aclk_send_agent_connection_update(mqtt_wss_client client, int reachable
248
{ .name = "ml", .version = 1, .enabled = ml_enabled(localhost) },
249
#endif
250
{ .name = "mc", .version = enable_metric_correlations ? metric_correlations_version : 0, .enabled = enable_metric_correlations },
251
+ { .name = "ctx", .version = 1, .enabled = rrdcontext_enabled },
252
{ .name = NULL, .version = 0, .enabled = 0 }
253
};
254
aclk/aclk_util.c
+4
@@ -124,6 +124,8 @@ struct topic_name {
124
{ .id = ACLK_TOPICID_ALARM_CONFIG, .name = "alarm-config" },
125
{ .id = ACLK_TOPICID_ALARM_SNAPSHOT, .name = "alarm-snapshot" },
126
{ .id = ACLK_TOPICID_NODE_COLLECTORS, .name = "node-instance-collectors" },
127
+ { .id = ACLK_TOPICID_CTXS_SNAPSHOT, .name = "contexts-snapshot" },
128
+ { .id = ACLK_TOPICID_CTXS_UPDATED, .name = "contexts-updated" },
129
{ .id = ACLK_TOPICID_UNKNOWN, .name = NULL }
130
};
131
@@ -147,6 +149,8 @@ enum aclk_topics compulsory_topics[] = {
149
ACLK_TOPICID_ALARM_CONFIG,
150
ACLK_TOPICID_ALARM_SNAPSHOT,
151
ACLK_TOPICID_NODE_COLLECTORS,
152
+ ACLK_TOPICID_CTXS_SNAPSHOT,
153
+ ACLK_TOPICID_CTXS_UPDATED,
154
ACLK_TOPICID_UNKNOWN
155
};
156
aclk/aclk_util.h
+3
-1
@@ -88,7 +88,9 @@ enum aclk_topics {
88
ACLK_TOPICID_ALARM_HEALTH = 15,
89
ACLK_TOPICID_ALARM_CONFIG = 16,
90
ACLK_TOPICID_ALARM_SNAPSHOT = 17,
91
- ACLK_TOPICID_NODE_COLLECTORS = 18
91
+ ACLK_TOPICID_NODE_COLLECTORS = 18,
92
+ ACLK_TOPICID_CTXS_SNAPSHOT = 19,
93
+ ACLK_TOPICID_CTXS_UPDATED = 20
94
};
95
96
const char *aclk_get_topic(enum aclk_topics topic);
aclk/schema-wrappers/context.cc
new
+125
@@ -0,0 +1,125 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "proto/context/v1/context.pb.h"
4
+
5
+#include "libnetdata/libnetdata.h"
6
+
7
+#include "schema_wrapper_utils.h"
8
+
9
+#include "context.h"
10
+
11
+using namespace context::v1;
12
+
13
+// ContextsSnapshot
14
+contexts_snapshot_t contexts_snapshot_new(const char *claim_id, const char *node_id, uint64_t version)
15
+{
16
+ ContextsSnapshot *ctxs_snap = new ContextsSnapshot;
17
+
18
+ if (ctxs_snap == NULL)
19
+ fatal("Cannot allocate ContextsSnapshot object. OOM");
20
+
21
+ ctxs_snap->set_claim_id(claim_id);
22
+ ctxs_snap->set_node_id(node_id);
23
+ ctxs_snap->set_version(version);
24
+
25
+ return ctxs_snap;
26
+}
27
+
28
+void contexts_snapshot_delete(contexts_snapshot_t snapshot)
29
+{
30
+ delete (ContextsSnapshot *)snapshot;
31
+}
32
+
33
+void contexts_snapshot_set_version(contexts_snapshot_t ctxs_snapshot, uint64_t version)
34
+{
35
+ ((ContextsSnapshot *)ctxs_snapshot)->set_version(version);
36
+}
37
+
38
+static void fill_ctx_updated(ContextUpdated *ctx, struct context_updated *c_ctx)
39
+{
40
+ ctx->set_id(c_ctx->id);
41
+ ctx->set_version(c_ctx->version);
42
+ ctx->set_first_entry(c_ctx->first_entry);
43
+ ctx->set_last_entry(c_ctx->last_entry);
44
+ ctx->set_deleted(c_ctx->deleted);
45
+ ctx->set_title(c_ctx->title);
46
+ ctx->set_priority(c_ctx->priority);
47
+ ctx->set_chart_type(c_ctx->chart_type);
48
+ ctx->set_units(c_ctx->units);
49
+ ctx->set_family(c_ctx->family);
50
+}
51
+
52
+void contexts_snapshot_add_ctx_update(contexts_snapshot_t ctxs_snapshot, struct context_updated *ctx_update)
53
+{
54
+ ContextsSnapshot *ctxs_snap = (ContextsSnapshot *)ctxs_snapshot;
55
+ ContextUpdated *ctx = ctxs_snap->add_contexts();
56
+
57
+ fill_ctx_updated(ctx, ctx_update);
58
+}
59
+
60
+char *contexts_snapshot_2bin(contexts_snapshot_t ctxs_snapshot, size_t *len)
61
+{
62
+ ContextsSnapshot *ctxs_snap = (ContextsSnapshot *)ctxs_snapshot;
63
+ *len = PROTO_COMPAT_MSG_SIZE_PTR(ctxs_snap);
64
+ char *bin = (char*)mallocz(*len);
65
+ if (!ctxs_snap->SerializeToArray(bin, *len)) {
66
+ freez(bin);
67
+ delete ctxs_snap;
68
+ return NULL;
69
+ }
70
+
71
+ delete ctxs_snap;
72
+ return bin;
73
+}
74
+
75
+// ContextsUpdated
76
+contexts_updated_t contexts_updated_new(const char *claim_id, const char *node_id, uint64_t version_hash, uint64_t created_at)
77
+{
78
+ ContextsUpdated *ctxs_updated = new ContextsUpdated;
79
+
80
+ if (ctxs_updated == NULL)
81
+ fatal("Cannot allocate ContextsUpdated object. OOM");
82
+
83
+ ctxs_updated->set_claim_id(claim_id);
84
+ ctxs_updated->set_node_id(node_id);
85
+ ctxs_updated->set_version_hash(version_hash);
86
+ ctxs_updated->set_created_at(created_at);
87
+
88
+ return ctxs_updated;
89
+}
90
+
91
+void contexts_updated_delete(contexts_updated_t ctxs_updated)
92
+{
93
+ delete (ContextsUpdated *)ctxs_updated;
94
+}
95
+
96
+void contexts_updated_update_version_hash(contexts_updated_t ctxs_updated, uint64_t version_hash)
97
+{
98
+ ((ContextsUpdated *)ctxs_updated)->set_version_hash(version_hash);
99
+}
100
+
101
+void contexts_updated_add_ctx_update(contexts_updated_t ctxs_updated, struct context_updated *ctx_update)
102
+{
103
+ ContextsUpdated *ctxs_update = (ContextsUpdated *)ctxs_updated;
104
+ ContextUpdated *ctx = ctxs_update->add_contextupdates();
105
+
106
+ if (ctx == NULL)
107
+ fatal("Cannot allocate ContextUpdated object. OOM");
108
+
109
+ fill_ctx_updated(ctx, ctx_update);
110
+}
111
+
112
+char *contexts_updated_2bin(contexts_updated_t ctxs_updated, size_t *len)
113
+{
114
+ ContextsUpdated *ctxs_update = (ContextsUpdated *)ctxs_updated;
115
+ *len = PROTO_COMPAT_MSG_SIZE_PTR(ctxs_update);
116
+ char *bin = (char*)mallocz(*len);
117
+ if (!ctxs_update->SerializeToArray(bin, *len)) {
118
+ freez(bin);
119
+ delete ctxs_update;
120
+ return NULL;
121
+ }
122
+
123
+ delete ctxs_update;
124
+ return bin;
125
+}
aclk/schema-wrappers/context.h
new
+53
@@ -0,0 +1,53 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef ACLK_SCHEMA_WRAPPER_CONTEXT_H
4
+#define ACLK_SCHEMA_WRAPPER_CONTEXT_H
5
+
6
+#include <stdint.h>
7
+#include <sys/types.h>
8
+
9
+#ifdef __cplusplus
10
+extern "C" {
11
+#endif
12
+
13
+typedef void* contexts_updated_t;
14
+typedef void* contexts_snapshot_t;
15
+
16
+struct context_updated {
17
+ // context id
18
+ const char *id;
19
+
20
+ uint64_t version;
21
+
22
+ uint64_t first_entry;
23
+ uint64_t last_entry;
24
+
25
+ int deleted;
26
+
27
+ const char *title;
28
+ uint64_t priority;
29
+ const char *chart_type;
30
+ const char *units;
31
+ const char *family;
32
+};
33
+
34
+// ContextS Snapshot related
35
+contexts_snapshot_t contexts_snapshot_new(const char *claim_id, const char *node_id, uint64_t version);
36
+void contexts_snapshot_delete(contexts_snapshot_t ctxs_snapshot);
37
+void contexts_snapshot_set_version(contexts_snapshot_t ctxs_snapshot, uint64_t version);
38
+void contexts_snapshot_add_ctx_update(contexts_snapshot_t ctxs_snapshot, struct context_updated *ctx_update);
39
+char *contexts_snapshot_2bin(contexts_snapshot_t ctxs_snapshot, size_t *len);
40
+
41
+// ContextS Updated related
42
+contexts_updated_t contexts_updated_new(const char *claim_id, const char *node_id, uint64_t version_hash, uint64_t created_at);
43
+void contexts_updated_delete(contexts_updated_t ctxs_updated);
44
+void contexts_updated_update_version_hash(contexts_updated_t ctxs_updated, uint64_t version_hash);
45
+void contexts_updated_add_ctx_update(contexts_updated_t ctxs_updated, struct context_updated *ctx_update);
46
+char *contexts_updated_2bin(contexts_updated_t ctxs_updated, size_t *len);
47
+
48
+
49
+#ifdef __cplusplus
50
+}
51
+#endif
52
+
53
+#endif /* ACLK_SCHEMA_WRAPPER_CONTEXT_H */
aclk/schema-wrappers/context_stream.cc
new
+42
@@ -0,0 +1,42 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "proto/context/v1/stream.pb.h"
4
+
5
+#include "context_stream.h"
6
+
7
+#include "libnetdata/libnetdata.h"
8
+
9
+struct stop_streaming_ctxs *parse_stop_streaming_ctxs(const char *data, size_t len)
10
+{
11
+ context::v1::StopStreamingContexts msg;
12
+
13
+ struct stop_streaming_ctxs *res;
14
+
15
+ if (!msg.ParseFromArray(data, len))
16
+ return NULL;
17
+
18
+ res = (struct stop_streaming_ctxs *)callocz(1, sizeof(struct stop_streaming_ctxs));
19
+
20
+ res->claim_id = strdupz(msg.claim_id().c_str());
21
+ res->node_id = strdupz(msg.node_id().c_str());
22
+
23
+ return res;
24
+}
25
+
26
+struct ctxs_checkpoint *parse_ctxs_checkpoint(const char *data, size_t len)
27
+{
28
+ context::v1::ContextsCheckpoint msg;
29
+
30
+ struct ctxs_checkpoint *res;
31
+
32
+ if (!msg.ParseFromArray(data, len))
33
+ return NULL;
34
+
35
+ res = (struct ctxs_checkpoint *)callocz(1, sizeof(struct ctxs_checkpoint));
36
+
37
+ res->claim_id = strdupz(msg.claim_id().c_str());
38
+ res->node_id = strdupz(msg.node_id().c_str());
39
+ res->version_hash = msg.version_hash();
40
+
41
+ return res;
42
+}
aclk/schema-wrappers/context_stream.h
new
+36
@@ -0,0 +1,36 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef ACLK_SCHEMA_WRAPPER_CONTEXT_STREAM_H
4
+#define ACLK_SCHEMA_WRAPPER_CONTEXT_STREAM_H
5
+
6
+#ifdef __cplusplus
7
+extern "C" {
8
+#endif
9
+
10
+struct stop_streaming_ctxs {
11
+ char *claim_id;
12
+ char *node_id;
13
+ // we omit reason as there is only one defined at this point
14
+ // as soon as there is more than one defined in StopStreaminContextsReason
15
+ // we should add it
16
+ // 0 - RATE_LIMIT_EXCEEDED
17
+};
18
+
19
+struct stop_streaming_ctxs *parse_stop_streaming_ctxs(const char *data, size_t len);
20
+
21
+struct ctxs_checkpoint {
22
+ char *claim_id;
23
+ char *node_id;
24
+
25
+ uint64_t version_hash;
26
+};
27
+
28
+struct ctxs_checkpoint *parse_ctxs_checkpoint(const char *data, size_t len);
29
+
30
+
31
+
32
+#ifdef __cplusplus
33
+}
34
+#endif
35
+
36
+#endif /* ACLK_SCHEMA_WRAPPER_CONTEXT_STREAM_H */
aclk/schema-wrappers/node_connection.cc
+9
@@ -28,6 +28,15 @@ char *generate_node_instance_connection(size_t *len, const node_instance_connect
28
timestamp->set_seconds(tv.tv_sec);
29
timestamp->set_nanos(tv.tv_usec * 1000);
30
31
+ if (data->capabilities) {
32
+ struct capability *capa = data->capabilities;
33
+ while (capa->name) {
34
+ aclk_lib::v1::Capability *proto_capa = msg.add_capabilities();
35
+ capability_set(proto_capa, capa);
36
+ capa++;
37
+ }
38
+ }
39
+
40
*len = PROTO_COMPAT_MSG_SIZE(msg);
41
char *bin = (char*)malloc(*len);
42
if (bin)
aclk/schema-wrappers/node_connection.h
+3
@@ -3,6 +3,8 @@
3
#ifndef ACLK_SCHEMA_WRAPPER_NODE_CONNECTION_H
4
#define ACLK_SCHEMA_WRAPPER_NODE_CONNECTION_H
5
6
+#include "capability.h"
7
+
8
#ifdef __cplusplus
9
extern "C" {
10
#endif
@@ -17,6 +19,7 @@ typedef struct {
19
int64_t session_id;
20
21
int32_t hops;
22
+ struct capability *capabilities;
23
} node_instance_connection_t;
24
25
char *generate_node_instance_connection(size_t *len, const node_instance_connection_t *data);
aclk/schema-wrappers/proto_2_json.cc
+12
@@ -11,6 +11,8 @@
11
#include "proto/nodeinstance/connection/v1/connection.pb.h"
12
#include "proto/nodeinstance/create/v1/creation.pb.h"
13
#include "proto/nodeinstance/info/v1/info.pb.h"
14
+#include "proto/context/v1/stream.pb.h"
15
+#include "proto/context/v1/context.pb.h"
16
17
#include "libnetdata/libnetdata.h"
18
@@ -45,6 +47,12 @@ static google::protobuf::Message *msg_name_to_protomsg(const char *msgname)
47
return new alarms::v1::AlarmSnapshot;
48
if (!strcmp(msgname, "AlarmLogEntry"))
49
return new alarms::v1::AlarmLogEntry;
50
+ if (!strcmp(msgname, "UpdateNodeCollectors"))
51
+ return new nodeinstance::info::v1::UpdateNodeCollectors;
52
+ if (!strcmp(msgname, "ContextsUpdated"))
53
+ return new context::v1::ContextsUpdated;
54
+ if (!strcmp(msgname, "ContextsSnapshot"))
55
+ return new context::v1::ContextsSnapshot;
56
57
//rx side
58
if (!strcmp(msgname, "CreateNodeInstanceResult"))
@@ -67,6 +75,10 @@ static google::protobuf::Message *msg_name_to_protomsg(const char *msgname)
75
return new alarms::v1::SendAlarmSnapshot;
76
if (!strcmp(msgname, "DisconnectReq"))
77
return new agent::v1::DisconnectReq;
78
+ if (!strcmp(msgname, "ContextsCheckpoint"))
79
+ return new context::v1::ContextsCheckpoint;
80
+ if (!strcmp(msgname, "StopStreamingContexts"))
81
+ return new context::v1::StopStreamingContexts;
82
83
return NULL;
84
}
aclk/schema-wrappers/schema_wrappers.h
+2
@@ -14,5 +14,7 @@
14
#include "alarm_stream.h"
15
#include "node_info.h"
16
#include "capability.h"
17
+#include "context_stream.h"
18
+#include "context.h"
19
20
#endif /* SCHEMA_WRAPPERS_H */
claim/claim.c
+1
-1
@@ -31,7 +31,7 @@ static char *claiming_errors[] = {
31
/* Retrieve the claim id for the agent.
32
* Caller owns the string.
33
*/
34
-char *is_agent_claimed()
34
+char *get_agent_claimid()
35
{
36
char *result;
37
rrdhost_aclk_state_lock(localhost);
claim/claim.h
+1
-1
@@ -9,7 +9,7 @@ extern char *claiming_pending_arguments;
9
extern struct config cloud_config;
10
11
void claim_agent(char *claiming_arguments);
12
-char *is_agent_claimed(void);
12
+char *get_agent_claimid(void);
13
void load_claiming_state(void);
14
void load_cloud_conf(int silent);
15
collectors/cgroups.plugin/sys_fs_cgroup.c
+54
-33
@@ -768,6 +768,12 @@ struct cgroup_network_interface {
768
struct cgroup_network_interface *next;
769
};
770
771
+enum cgroups_container_orchestrator {
772
+ CGROUPS_ORCHESTRATOR_UNSET,
773
+ CGROUPS_ORCHESTRATOR_UNKNOWN,
774
+ CGROUPS_ORCHESTRATOR_K8S
775
+};
776
+
777
// *** WARNING *** The fields are not thread safe. Take care of safe usage.
778
struct cgroup {
779
uint32_t options;
@@ -791,6 +797,8 @@ struct cgroup {
797
798
DICTIONARY *chart_labels;
799
800
+ int container_orchestrator;
801
+
802
struct cpuacct_stat cpuacct_stat;
803
struct cpuacct_usage cpuacct_usage;
804
struct cpuacct_cpu_throttling cpuacct_cpu_throttling;
@@ -935,6 +943,10 @@ static inline int is_cgroup_systemd_service(struct cgroup *cg) {
943
}
944
945
// ---------------------------------------------------------------------------------------------
946
+static int k8s_is_kubepod(struct cgroup *cg) {
947
+ return cg->container_orchestrator == CGROUPS_ORCHESTRATOR_K8S;
948
+}
949
+
950
static int k8s_is_container(const char *id) {
951
// examples:
952
// https://github.com/netdata/netdata/blob/0fc101679dcd12f1cb8acdd07bb4c85d8e553e53/collectors/cgroups.plugin/cgroup-name.sh#L121-L147
@@ -1690,7 +1702,8 @@ static inline void read_cgroup_network_interfaces(struct cgroup *cg) {
1702
info("CGROUP: cgroup '%s' has network interface '%s' as '%s'", cg->id, i->host_device, i->container_device);
1703
1704
// register a device rename to proc_net_dev.c
1693
- netdev_rename_device_add(i->host_device, i->container_device, cg->chart_id, cg->chart_labels);
1705
+ netdev_rename_device_add(
1706
+ i->host_device, i->container_device, cg->chart_id, cg->chart_labels, k8s_is_kubepod(cg) ? "k8s." : "");
1707
}
1708
}
1709
@@ -2589,6 +2602,14 @@ static inline void discovery_process_first_time_seen_cgroup(struct cgroup *cg) {
2602
2603
char comm[TASK_COMM_LEN];
2604
2605
+ if (cg->container_orchestrator == CGROUPS_ORCHESTRATOR_UNSET) {
2606
+ if (strstr(cg->id, "kubepods")) {
2607
+ cg->container_orchestrator = CGROUPS_ORCHESTRATOR_K8S;
2608
+ } else {
2609
+ cg->container_orchestrator = CGROUPS_ORCHESTRATOR_UNKNOWN;
2610
+ }
2611
+ }
2612
+
2613
if (is_inside_k8s && !k8s_get_container_first_proc_comm(cg->id, comm)) {
2614
// container initialization may take some time when CPU % is high
2615
// seen on GKE: comm is '6' before 'runc:[2:INIT]' (dunno if it could be another number)
@@ -3782,7 +3803,7 @@ void update_cgroup_charts(int update_every) {
3803
, "cpu"
3804
, NULL
3805
, "cpu"
3785
- , "cgroup.cpu"
3806
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.cpu" : "cgroup.cpu"
3807
, title
3808
, "percentage"
3809
, PLUGIN_CGROUPS_NAME
@@ -3855,7 +3876,7 @@ void update_cgroup_charts(int update_every) {
3876
, "cpu_limit"
3877
, NULL
3878
, "cpu"
3858
- , "cgroup.cpu_limit"
3879
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_limit" : "cgroup.cpu_limit"
3880
, title
3881
, "percentage"
3882
, PLUGIN_CGROUPS_NAME
@@ -3908,7 +3929,7 @@ void update_cgroup_charts(int update_every) {
3929
, "throttled"
3930
, NULL
3931
, "cpu"
3911
- , "cgroup.throttled"
3932
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.throttled" : "cgroup.throttled"
3933
, title
3934
, "percentage"
3935
, PLUGIN_CGROUPS_NAME
@@ -3934,7 +3955,7 @@ void update_cgroup_charts(int update_every) {
3955
, "throttled_duration"
3956
, NULL
3957
, "cpu"
3937
- , "cgroup.throttled_duration"
3958
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.throttled_duration" : "cgroup.throttled_duration"
3959
, title
3960
, "ms"
3961
, PLUGIN_CGROUPS_NAME
@@ -3962,7 +3983,7 @@ void update_cgroup_charts(int update_every) {
3983
, "cpu_shares"
3984
, NULL
3985
, "cpu"
3965
- , "cgroup.cpu_shares"
3986
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_shares" : "cgroup.cpu_shares"
3987
, title
3988
, "shares"
3989
, PLUGIN_CGROUPS_NAME
@@ -3993,7 +4014,7 @@ void update_cgroup_charts(int update_every) {
4014
, "cpu_per_core"
4015
, NULL
4016
, "cpu"
3996
- , "cgroup.cpu_per_core"
4017
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_per_core" : "cgroup.cpu_per_core"
4018
, title
4019
, "percentage"
4020
, PLUGIN_CGROUPS_NAME
@@ -4029,7 +4050,7 @@ void update_cgroup_charts(int update_every) {
4050
, "mem"
4051
, NULL
4052
, "mem"
4032
- , "cgroup.mem"
4053
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.mem" : "cgroup.mem"
4054
, title
4055
, "MiB"
4056
, PLUGIN_CGROUPS_NAME
@@ -4089,7 +4110,7 @@ void update_cgroup_charts(int update_every) {
4110
, "writeback"
4111
, NULL
4112
, "mem"
4092
- , "cgroup.writeback"
4113
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.writeback" : "cgroup.writeback"
4114
, title
4115
, "MiB"
4116
, PLUGIN_CGROUPS_NAME
@@ -4124,7 +4145,7 @@ void update_cgroup_charts(int update_every) {
4145
, "mem_activity"
4146
, NULL
4147
, "mem"
4127
- , "cgroup.mem_activity"
4148
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.mem_activity" : "cgroup.mem_activity"
4149
, title
4150
, "MiB/s"
4151
, PLUGIN_CGROUPS_NAME
@@ -4155,7 +4176,7 @@ void update_cgroup_charts(int update_every) {
4176
, "pgfaults"
4177
, NULL
4178
, "mem"
4158
- , "cgroup.pgfaults"
4179
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.pgfaults" : "cgroup.pgfaults"
4180
, title
4181
, "MiB/s"
4182
, PLUGIN_CGROUPS_NAME
@@ -4187,7 +4208,7 @@ void update_cgroup_charts(int update_every) {
4208
, "mem_usage"
4209
, NULL
4210
, "mem"
4190
- , "cgroup.mem_usage"
4211
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.mem_usage" : "cgroup.mem_usage"
4212
, title
4213
, "MiB"
4214
, PLUGIN_CGROUPS_NAME
@@ -4254,7 +4275,7 @@ void update_cgroup_charts(int update_every) {
4275
, "mem_usage_limit"
4276
, NULL
4277
, "mem"
4257
- , "cgroup.mem_usage_limit"
4278
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.mem_usage_limit": "cgroup.mem_usage_limit"
4279
, title
4280
, "MiB"
4281
, PLUGIN_CGROUPS_NAME
@@ -4286,7 +4307,7 @@ void update_cgroup_charts(int update_every) {
4307
, "mem_utilization"
4308
, NULL
4309
, "mem"
4289
- , "cgroup.mem_utilization"
4310
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.mem_utilization" : "cgroup.mem_utilization"
4311
, title
4312
, "percentage"
4313
, PLUGIN_CGROUPS_NAME
@@ -4335,7 +4356,7 @@ void update_cgroup_charts(int update_every) {
4356
, "mem_failcnt"
4357
, NULL
4358
, "mem"
4338
- , "cgroup.mem_failcnt"
4359
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.mem_failcnt" : "cgroup.mem_failcnt"
4360
, title
4361
, "count"
4362
, PLUGIN_CGROUPS_NAME
@@ -4365,7 +4386,7 @@ void update_cgroup_charts(int update_every) {
4386
, "io"
4387
, NULL
4388
, "disk"
4368
- , "cgroup.io"
4389
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.io" : "cgroup.io"
4390
, title
4391
, "KiB/s"
4392
, PLUGIN_CGROUPS_NAME
@@ -4397,7 +4418,7 @@ void update_cgroup_charts(int update_every) {
4418
, "serviced_ops"
4419
, NULL
4420
, "disk"
4400
- , "cgroup.serviced_ops"
4421
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.serviced_ops" : "cgroup.serviced_ops"
4422
, title
4423
, "operations/s"
4424
, PLUGIN_CGROUPS_NAME
@@ -4429,7 +4450,7 @@ void update_cgroup_charts(int update_every) {
4450
, "throttle_io"
4451
, NULL
4452
, "disk"
4432
- , "cgroup.throttle_io"
4453
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.throttle_io" : "cgroup.throttle_io"
4454
, title
4455
, "KiB/s"
4456
, PLUGIN_CGROUPS_NAME
@@ -4461,7 +4482,7 @@ void update_cgroup_charts(int update_every) {
4482
, "throttle_serviced_ops"
4483
, NULL
4484
, "disk"
4464
- , "cgroup.throttle_serviced_ops"
4485
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.throttle_serviced_ops" : "cgroup.throttle_serviced_ops"
4486
, title
4487
, "operations/s"
4488
, PLUGIN_CGROUPS_NAME
@@ -4493,7 +4514,7 @@ void update_cgroup_charts(int update_every) {
4514
, "queued_ops"
4515
, NULL
4516
, "disk"
4496
- , "cgroup.queued_ops"
4517
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.queued_ops" : "cgroup.queued_ops"
4518
, title
4519
, "operations"
4520
, PLUGIN_CGROUPS_NAME
@@ -4525,7 +4546,7 @@ void update_cgroup_charts(int update_every) {
4546
, "merged_ops"
4547
, NULL
4548
, "disk"
4528
- , "cgroup.merged_ops"
4549
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.merged_ops" : "cgroup.merged_ops"
4550
, title
4551
, "operations/s"
4552
, PLUGIN_CGROUPS_NAME
@@ -4563,7 +4584,7 @@ void update_cgroup_charts(int update_every) {
4584
, "cpu_some_pressure"
4585
, NULL
4586
, "cpu"
4566
- , "cgroup.cpu_some_pressure"
4587
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_some_pressure" : "cgroup.cpu_some_pressure"
4588
, title
4589
, "percentage"
4590
, PLUGIN_CGROUPS_NAME
@@ -4587,7 +4608,7 @@ void update_cgroup_charts(int update_every) {
4608
, "cpu_some_pressure_stall_time"
4609
, NULL
4610
, "cpu"
4590
- , "cgroup.cpu_some_pressure_stall_time"
4611
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_some_pressure_stall_time" : "cgroup.cpu_some_pressure_stall_time"
4612
, title
4613
, "ms"
4614
, PLUGIN_CGROUPS_NAME
@@ -4615,7 +4636,7 @@ void update_cgroup_charts(int update_every) {
4636
, "cpu_full_pressure"
4637
, NULL
4638
, "cpu"
4618
- , "cgroup.cpu_full_pressure"
4639
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_full_pressure" : "cgroup.cpu_full_pressure"
4640
, title
4641
, "percentage"
4642
, PLUGIN_CGROUPS_NAME
@@ -4639,7 +4660,7 @@ void update_cgroup_charts(int update_every) {
4660
, "cpu_full_pressure_stall_time"
4661
, NULL
4662
, "cpu"
4642
- , "cgroup.cpu_full_pressure_stall_time"
4663
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.cpu_full_pressure_stall_time" : "cgroup.cpu_full_pressure_stall_time"
4664
, title
4665
, "ms"
4666
, PLUGIN_CGROUPS_NAME
@@ -4670,7 +4691,7 @@ void update_cgroup_charts(int update_every) {
4691
, "mem_some_pressure"
4692
, NULL
4693
, "mem"
4673
- , "cgroup.memory_some_pressure"
4694
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.memory_some_pressure" : "cgroup.memory_some_pressure"
4695
, title
4696
, "percentage"
4697
, PLUGIN_CGROUPS_NAME
@@ -4694,7 +4715,7 @@ void update_cgroup_charts(int update_every) {
4715
, "memory_some_pressure_stall_time"
4716
, NULL
4717
, "mem"
4697
- , "cgroup.memory_some_pressure_stall_time"
4718
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.memory_some_pressure_stall_time" : "cgroup.memory_some_pressure_stall_time"
4719
, title
4720
, "ms"
4721
, PLUGIN_CGROUPS_NAME
@@ -4724,7 +4745,7 @@ void update_cgroup_charts(int update_every) {
4745
, "mem_full_pressure"
4746
, NULL
4747
, "mem"
4727
- , "cgroup.memory_full_pressure"
4748
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.memory_full_pressure" : "cgroup.memory_full_pressure"
4749
, title
4750
, "percentage"
4751
, PLUGIN_CGROUPS_NAME
@@ -4749,7 +4770,7 @@ void update_cgroup_charts(int update_every) {
4770
, "memory_full_pressure_stall_time"
4771
, NULL
4772
, "mem"
4752
- , "cgroup.memory_full_pressure_stall_time"
4773
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.memory_full_pressure_stall_time" : "cgroup.memory_full_pressure_stall_time"
4774
, title
4775
, "ms"
4776
, PLUGIN_CGROUPS_NAME
@@ -4780,7 +4801,7 @@ void update_cgroup_charts(int update_every) {
4801
, "io_some_pressure"
4802
, NULL
4803
, "disk"
4783
- , "cgroup.io_some_pressure"
4804
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.io_some_pressure" : "cgroup.io_some_pressure"
4805
, title
4806
, "percentage"
4807
, PLUGIN_CGROUPS_NAME
@@ -4804,7 +4825,7 @@ void update_cgroup_charts(int update_every) {
4825
, "io_some_pressure_stall_time"
4826
, NULL
4827
, "disk"
4807
- , "cgroup.io_some_pressure_stall_time"
4828
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.io_some_pressure_stall_time" : "cgroup.io_some_pressure_stall_time"
4829
, title
4830
, "ms"
4831
, PLUGIN_CGROUPS_NAME
@@ -4833,7 +4854,7 @@ void update_cgroup_charts(int update_every) {
4854
, "io_full_pressure"
4855
, NULL
4856
, "disk"
4836
- , "cgroup.io_full_pressure"
4857
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.io_full_pressure" : "cgroup.io_full_pressure"
4858
, title
4859
, "percentage"
4860
, PLUGIN_CGROUPS_NAME
@@ -4857,7 +4878,7 @@ void update_cgroup_charts(int update_every) {
4878
, "io_full_pressure_stall_time"
4879
, NULL
4880
, "disk"
4860
- , "cgroup.io_full_pressure_stall_time"
4881
+ , k8s_is_kubepod(cg) ? "k8s.cgroup.io_full_pressure_stall_time" : "cgroup.io_full_pressure_stall_time"
4882
, title
4883
, "ms"
4884
, PLUGIN_CGROUPS_NAME
collectors/cgroups.plugin/tests/test_doubles.c
+2
-1
@@ -132,12 +132,13 @@ void update_pressure_charts(struct pressure_charts *charts)
132
}
133
134
void netdev_rename_device_add(
135
- const char *host_device, const char *container_device, const char *container_name, DICTIONARY *labels)
135
+ const char *host_device, const char *container_device, const char *container_name, DICTIONARY *labels, const char *ctx_prefix)
136
{
137
UNUSED(host_device);
138
UNUSED(container_device);
139
UNUSED(container_name);
140
UNUSED(labels);
141
+ UNUSED(ctx_prefix);
142
}
143
144
void netdev_rename_device_del(const char *host_device)
collectors/cups.plugin/cups_plugin.c
+6
-6
@@ -162,12 +162,12 @@ struct job_metrics *get_job_metrics(char *dest) {
162
reset_job_metrics(NULL, &new_job_metrics, NULL);
163
jm = dictionary_set(dict_dest_job_metrics, dest, &new_job_metrics, sizeof(struct job_metrics));
164
165
- printf("CHART cups.job_num_%s '' 'Active job number of destination %s' jobs '%s' cups.job_num stacked %i %i\n", dest, dest, dest, netdata_priority++, netdata_update_every);
165
+ printf("CHART cups.job_num_%s '' 'Active jobs of %s' jobs '%s' cups.job_num stacked %i %i\n", dest, dest, dest, netdata_priority++, netdata_update_every);
166
printf("DIMENSION pending '' absolute 1 1\n");
167
printf("DIMENSION held '' absolute 1 1\n");
168
printf("DIMENSION processing '' absolute 1 1\n");
169
170
- printf("CHART cups.job_size_%s '' 'Active job size of destination %s' KB '%s' cups.job_size stacked %i %i\n", dest, dest, dest, netdata_priority++, netdata_update_every);
170
+ printf("CHART cups.job_size_%s '' 'Active jobs size of %s' KB '%s' cups.job_size stacked %i %i\n", dest, dest, dest, netdata_priority++, netdata_update_every);
171
printf("DIMENSION pending '' absolute 1 1\n");
172
printf("DIMENSION held '' absolute 1 1\n");
173
printf("DIMENSION processing '' absolute 1 1\n");
@@ -196,12 +196,12 @@ int collect_job_metrics(const char *name, void *entry, void *data) {
196
"END\n",
197
name, jm->size_pending, jm->size_held, jm->size_processing);
198
} else {
199
- printf("CHART cups.job_num_%s '' 'Active job number of destination %s' jobs '%s' cups.job_num stacked 1 %i 'obsolete'\n", name, name, name, netdata_update_every);
199
+ printf("CHART cups.job_num_%s '' 'Active jobs of %s' jobs '%s' cups.job_num stacked 1 %i 'obsolete'\n", name, name, name, netdata_update_every);
200
printf("DIMENSION pending '' absolute 1 1\n");
201
printf("DIMENSION held '' absolute 1 1\n");
202
printf("DIMENSION processing '' absolute 1 1\n");
203
204
- printf("CHART cups.job_size_%s '' 'Active job size of destination %s' KB '%s' cups.job_size stacked 1 %i 'obsolete'\n", name, name, name, netdata_update_every);
204
+ printf("CHART cups.job_size_%s '' 'Active jobs size of %s' KB '%s' cups.job_size stacked 1 %i 'obsolete'\n", name, name, name, netdata_update_every);
205
printf("DIMENSION pending '' absolute 1 1\n");
206
printf("DIMENSION held '' absolute 1 1\n");
207
printf("DIMENSION processing '' absolute 1 1\n");
@@ -387,12 +387,12 @@ int main(int argc, char **argv) {
387
printf("DIMENSION acceptingjobs '' absolute 1 1\n");
388
printf("DIMENSION shared '' absolute 1 1\n");
389
390
- printf("CHART cups.job_num '' 'Total active job number' jobs overview cups.job_num stacked 100002 %i\n", netdata_update_every);
390
+ printf("CHART cups.job_num '' 'Active jobs' jobs overview cups.job_num stacked 100002 %i\n", netdata_update_every);
391
printf("DIMENSION pending '' absolute 1 1\n");
392
printf("DIMENSION held '' absolute 1 1\n");
393
printf("DIMENSION processing '' absolute 1 1\n");
394
395
- printf("CHART cups.job_size '' 'Total active job size' KB overview cups.job_size stacked 100003 %i\n", netdata_update_every);
395
+ printf("CHART cups.job_size '' 'Active jobs size' KB overview cups.job_size stacked 100003 %i\n", netdata_update_every);
396
printf("DIMENSION pending '' absolute 1 1\n");
397
printf("DIMENSION held '' absolute 1 1\n");
398
printf("DIMENSION processing '' absolute 1 1\n");
collectors/diskspace.plugin/plugin_diskspace.c
+3
-1
@@ -431,7 +431,9 @@ static inline void do_disk_space_stats(struct mountinfo *mi, int update_every) {
431
};
432
433
mp.chart_labels = rrdlabels_create();
434
- rrdlabels_add(mp.chart_labels, "fstype", mi->filesystem, RRDLABEL_SRC_AUTO);
434
+ rrdlabels_add(mp.chart_labels, "mount_point", mi->mount_point, RRDLABEL_SRC_AUTO);
435
+ rrdlabels_add(mp.chart_labels, "filesystem", mi->filesystem, RRDLABEL_SRC_AUTO);
436
+ rrdlabels_add(mp.chart_labels, "mount_root", mi->root, RRDLABEL_SRC_AUTO);
437
438
m = dictionary_set(dict_mountpoints, mi->mount_point, &mp, sizeof(struct mount_point_metadata));
439
collectors/proc.plugin/plugin_proc.h
+6
-1
@@ -54,7 +54,12 @@ extern unsigned long long zfs_arcstats_shrinkable_cache_size_bytes;
54
55
// netdev renames
56
extern void netdev_rename_device_add(
57
- const char *host_device, const char *container_device, const char *container_name, DICTIONARY *labels);
57
+ const char *host_device,
58
+ const char *container_device,
59
+ const char *container_name,
60
+ DICTIONARY *labels,
61
+ const char *ctx_prefix);
62
+
63
extern void netdev_rename_device_del(const char *host_device);
64
65
#include "proc_self_mountinfo.h"
collectors/proc.plugin/proc_diskstats.c
+103
-22
@@ -830,6 +830,30 @@ static struct disk *get_disk(unsigned long major, unsigned long minor, char *dis
830
return d;
831
}
832
833
+static void add_labels_to_disk(struct disk *d, RRDSET *st) {
834
+ rrdlabels_add(st->state->chart_labels, "device", d->disk, RRDLABEL_SRC_AUTO);
835
+ rrdlabels_add(st->state->chart_labels, "mount_point", d->mount_point, RRDLABEL_SRC_AUTO);
836
+
837
+ switch (d->type) {
838
+ default:
839
+ case DISK_TYPE_UNKNOWN:
840
+ rrdlabels_add(st->state->chart_labels, "device_type", "unknown", RRDLABEL_SRC_AUTO);
841
+ break;
842
+
843
+ case DISK_TYPE_PHYSICAL:
844
+ rrdlabels_add(st->state->chart_labels, "device_type", "physical", RRDLABEL_SRC_AUTO);
845
+ break;
846
+
847
+ case DISK_TYPE_PARTITION:
848
+ rrdlabels_add(st->state->chart_labels, "device_type", "partition", RRDLABEL_SRC_AUTO);
849
+ break;
850
+
851
+ case DISK_TYPE_VIRTUAL:
852
+ rrdlabels_add(st->state->chart_labels, "device_type", "virtual", RRDLABEL_SRC_AUTO);
853
+ break;
854
+ }
855
+}
856
+
857
int do_proc_diskstats(int update_every, usec_t dt) {
858
static procfile *ff = NULL;
859
@@ -1067,6 +1091,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1091
1092
d->rd_io_reads = rrddim_add(d->st_io, "reads", NULL, d->sector_size, 1024, RRD_ALGORITHM_INCREMENTAL);
1093
d->rd_io_writes = rrddim_add(d->st_io, "writes", NULL, d->sector_size * -1, 1024, RRD_ALGORITHM_INCREMENTAL);
1094
+
1095
+ add_labels_to_disk(d, d->st_io);
1096
}
1097
else rrdset_next(d->st_io);
1098
@@ -1094,8 +1120,9 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1120
, RRDSET_TYPE_AREA
1121
);
1122
1097
- d->rd_io_discards =
1098
- rrddim_add(d->st_ext_io, "discards", NULL, d->sector_size, 1024, RRD_ALGORITHM_INCREMENTAL);
1123
+ d->rd_io_discards = rrddim_add(d->st_ext_io, "discards", NULL, d->sector_size, 1024, RRD_ALGORITHM_INCREMENTAL);
1124
+
1125
+ add_labels_to_disk(d, d->st_ext_io);
1126
} else
1127
rrdset_next(d->st_ext_io);
1128
@@ -1130,6 +1157,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1157
1158
d->rd_ops_reads = rrddim_add(d->st_ops, "reads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1159
d->rd_ops_writes = rrddim_add(d->st_ops, "writes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1160
+
1161
+ add_labels_to_disk(d, d->st_ops);
1162
}
1163
else rrdset_next(d->st_ops);
1164
@@ -1162,7 +1191,10 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1191
d->rd_ops_discards = rrddim_add(d->st_ext_ops, "discards", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1192
if (do_fl_stats)
1193
d->rd_ops_flushes = rrddim_add(d->st_ext_ops, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1165
- } else
1194
+
1195
+ add_labels_to_disk(d, d->st_ext_ops);
1196
+ }
1197
+ else
1198
rrdset_next(d->st_ext_ops);
1199
1200
last_discards = rrddim_set_by_pointer(d->st_ext_ops, d->rd_ops_discards, discards);
@@ -1196,6 +1228,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1228
rrdset_flag_set(d->st_qops, RRDSET_FLAG_DETAIL);
1229
1230
d->rd_qops_operations = rrddim_add(d->st_qops, "operations", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1231
+
1232
+ add_labels_to_disk(d, d->st_qops);
1233
}
1234
else rrdset_next(d->st_qops);
1235
@@ -1228,6 +1262,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1262
rrdset_flag_set(d->st_backlog, RRDSET_FLAG_DETAIL);
1263
1264
d->rd_backlog_backlog = rrddim_add(d->st_backlog, "backlog", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1265
+
1266
+ add_labels_to_disk(d, d->st_backlog);
1267
}
1268
else rrdset_next(d->st_backlog);
1269
@@ -1259,8 +1295,9 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1295
1296
rrdset_flag_set(d->st_busy, RRDSET_FLAG_DETAIL);
1297
1262
- d->rd_busy_busy =
1263
- rrddim_add(d->st_busy, "busy", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1298
+ d->rd_busy_busy = rrddim_add(d->st_busy, "busy", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1299
+
1300
+ add_labels_to_disk(d, d->st_busy);
1301
}
1302
else rrdset_next(d->st_busy);
1303
@@ -1288,6 +1325,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1325
rrdset_flag_set(d->st_util, RRDSET_FLAG_DETAIL);
1326
1327
d->rd_util_utilization = rrddim_add(d->st_util, "utilization", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1328
+
1329
+ add_labels_to_disk(d, d->st_util);
1330
}
1331
else rrdset_next(d->st_util);
1332
@@ -1326,6 +1365,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1365
1366
d->rd_mops_reads = rrddim_add(d->st_mops, "reads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1367
d->rd_mops_writes = rrddim_add(d->st_mops, "writes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1368
+
1369
+ add_labels_to_disk(d, d->st_mops);
1370
}
1371
else rrdset_next(d->st_mops);
1372
@@ -1358,7 +1399,10 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1399
rrdset_flag_set(d->st_ext_mops, RRDSET_FLAG_DETAIL);
1400
1401
d->rd_mops_discards = rrddim_add(d->st_ext_mops, "discards", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1361
- } else
1402
+
1403
+ add_labels_to_disk(d, d->st_ext_mops);
1404
+ }
1405
+ else
1406
rrdset_next(d->st_ext_mops);
1407
1408
rrddim_set_by_pointer(d->st_ext_mops, d->rd_mops_discards, mdiscards);
@@ -1391,6 +1435,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1435
1436
d->rd_iotime_reads = rrddim_add(d->st_iotime, "reads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1437
d->rd_iotime_writes = rrddim_add(d->st_iotime, "writes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1438
+
1439
+ add_labels_to_disk(d, d->st_iotime);
1440
}
1441
else rrdset_next(d->st_iotime);
1442
@@ -1422,9 +1468,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1468
1469
d->rd_iotime_discards = rrddim_add(d->st_ext_iotime, "discards", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1470
if (do_fl_stats)
1425
- d->rd_iotime_flushes =
1426
- rrddim_add(d->st_ext_iotime, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1427
- } else
1471
+ d->rd_iotime_flushes = rrddim_add(d->st_ext_iotime, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1472
+
1473
+ add_labels_to_disk(d, d->st_ext_iotime);
1474
+ }
1475
+ else
1476
rrdset_next(d->st_ext_iotime);
1477
1478
last_discardms = rrddim_set_by_pointer(d->st_ext_iotime, d->rd_iotime_discards, discardms);
@@ -1465,6 +1513,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1513
1514
d->rd_await_reads = rrddim_add(d->st_await, "reads", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1515
d->rd_await_writes = rrddim_add(d->st_await, "writes", NULL, -1, 1, RRD_ALGORITHM_ABSOLUTE);
1516
+
1517
+ add_labels_to_disk(d, d->st_await);
1518
}
1519
else rrdset_next(d->st_await);
1520
@@ -1494,18 +1544,22 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1544
1545
d->rd_await_discards = rrddim_add(d->st_ext_await, "discards", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1546
if (do_fl_stats)
1497
- d->rd_await_flushes =
1498
- rrddim_add(d->st_ext_await, "flushes", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1499
- } else
1547
+ d->rd_await_flushes = rrddim_add(d->st_ext_await, "flushes", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1548
+
1549
+ add_labels_to_disk(d, d->st_ext_await);
1550
+ }
1551
+ else
1552
rrdset_next(d->st_ext_await);
1553
1554
rrddim_set_by_pointer(
1555
d->st_ext_await, d->rd_await_discards,
1556
(discards - last_discards) ? (discardms - last_discardms) / (discards - last_discards) : 0);
1557
+
1558
if (do_fl_stats)
1559
rrddim_set_by_pointer(
1560
d->st_ext_await, d->rd_await_flushes,
1561
(flushes - last_flushes) ? (flushms - last_flushms) / (flushes - last_flushes) : 0);
1562
+
1563
rrdset_done(d->st_ext_await);
1564
}
1565
@@ -1534,6 +1588,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1588
1589
d->rd_avgsz_reads = rrddim_add(d->st_avgsz, "reads", NULL, d->sector_size, 1024, RRD_ALGORITHM_ABSOLUTE);
1590
d->rd_avgsz_writes = rrddim_add(d->st_avgsz, "writes", NULL, d->sector_size * -1, 1024, RRD_ALGORITHM_ABSOLUTE);
1591
+
1592
+ add_labels_to_disk(d, d->st_avgsz);
1593
}
1594
else rrdset_next(d->st_avgsz);
1595
@@ -1561,9 +1617,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1617
1618
rrdset_flag_set(d->st_ext_avgsz, RRDSET_FLAG_DETAIL);
1619
1564
- d->rd_avgsz_discards =
1565
- rrddim_add(d->st_ext_avgsz, "discards", NULL, d->sector_size, 1024, RRD_ALGORITHM_ABSOLUTE);
1566
- } else
1620
+ d->rd_avgsz_discards = rrddim_add(d->st_ext_avgsz, "discards", NULL, d->sector_size, 1024, RRD_ALGORITHM_ABSOLUTE);
1621
+
1622
+ add_labels_to_disk(d, d->st_ext_avgsz);
1623
+ }
1624
+ else
1625
rrdset_next(d->st_ext_avgsz);
1626
1627
rrddim_set_by_pointer(
@@ -1599,8 +1657,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1657
rrdset_flag_set(d->st_svctm, RRDSET_FLAG_DETAIL);
1658
1659
d->rd_svctm_svctm = rrddim_add(d->st_svctm, "svctm", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1660
+
1661
+ add_labels_to_disk(d, d->st_svctm);
1662
}
1603
- else rrdset_next(d->st_svctm);
1663
+ else
1664
+ rrdset_next(d->st_svctm);
1665
1666
rrddim_set_by_pointer(d->st_svctm, d->rd_svctm_svctm, ((reads - last_reads) + (writes - last_writes)) ? (busy_ms - last_busy_ms) / ((reads - last_reads) + (writes - last_writes)) : 0);
1667
rrdset_done(d->st_svctm);
@@ -1705,8 +1766,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1766
d->rd_bcache_hit_ratio_1hour = rrddim_add(d->st_bcache_hit_ratio, "1hour", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1767
d->rd_bcache_hit_ratio_1day = rrddim_add(d->st_bcache_hit_ratio, "1day", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1768
d->rd_bcache_hit_ratio_total = rrddim_add(d->st_bcache_hit_ratio, "ever", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1769
+
1770
+ add_labels_to_disk(d, d->st_bcache_hit_ratio);
1771
}
1709
- else rrdset_next(d->st_bcache_hit_ratio);
1772
+ else
1773
+ rrdset_next(d->st_bcache_hit_ratio);
1774
1775
rrddim_set_by_pointer(d->st_bcache_hit_ratio, d->rd_bcache_hit_ratio_5min, stats_five_minute_cache_hit_ratio);
1776
rrddim_set_by_pointer(d->st_bcache_hit_ratio, d->rd_bcache_hit_ratio_1hour, stats_hour_cache_hit_ratio);
@@ -1735,8 +1799,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1799
1800
d->rd_bcache_rate_congested = rrddim_add(d->st_bcache_rates, "congested", NULL, 1, 1024, RRD_ALGORITHM_ABSOLUTE);
1801
d->rd_bcache_rate_writeback = rrddim_add(d->st_bcache_rates, "writeback", NULL, -1, 1024, RRD_ALGORITHM_ABSOLUTE);
1802
+
1803
+ add_labels_to_disk(d, d->st_bcache_rates);
1804
}
1739
- else rrdset_next(d->st_bcache_rates);
1805
+ else
1806
+ rrdset_next(d->st_bcache_rates);
1807
1808
rrddim_set_by_pointer(d->st_bcache_rates, d->rd_bcache_rate_writeback, writeback_rate);
1809
rrddim_set_by_pointer(d->st_bcache_rates, d->rd_bcache_rate_congested, cache_congested);
@@ -1761,8 +1828,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1828
);
1829
1830
d->rd_bcache_dirty_size = rrddim_add(d->st_bcache_size, "dirty", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
1831
+
1832
+ add_labels_to_disk(d, d->st_bcache_size);
1833
}
1765
- else rrdset_next(d->st_bcache_size);
1834
+ else
1835
+ rrdset_next(d->st_bcache_size);
1836
1837
rrddim_set_by_pointer(d->st_bcache_size, d->rd_bcache_dirty_size, dirty_data);
1838
rrdset_done(d->st_bcache_size);
@@ -1786,8 +1856,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1856
);
1857
1858
d->rd_bcache_available_percent = rrddim_add(d->st_bcache_usage, "avail", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1859
+
1860
+ add_labels_to_disk(d, d->st_bcache_usage);
1861
}
1790
- else rrdset_next(d->st_bcache_usage);
1862
+ else
1863
+ rrdset_next(d->st_bcache_usage);
1864
1865
rrddim_set_by_pointer(d->st_bcache_usage, d->rd_bcache_available_percent, cache_available_percent);
1866
rrdset_done(d->st_bcache_usage);
@@ -1813,8 +1886,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1886
1887
d->rd_bcache_cache_read_races = rrddim_add(d->st_bcache_cache_read_races, "races", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1888
d->rd_bcache_cache_io_errors = rrddim_add(d->st_bcache_cache_read_races, "errors", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1889
+
1890
+ add_labels_to_disk(d, d->st_bcache_cache_read_races);
1891
}
1817
- else rrdset_next(d->st_bcache_cache_read_races);
1892
+ else
1893
+ rrdset_next(d->st_bcache_cache_read_races);
1894
1895
rrddim_set_by_pointer(d->st_bcache_cache_read_races, d->rd_bcache_cache_read_races, cache_read_races);
1896
rrddim_set_by_pointer(d->st_bcache_cache_read_races, d->rd_bcache_cache_io_errors, cache_io_errors);
@@ -1849,8 +1925,11 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1925
d->rd_bcache_misses = rrddim_add(d->st_bcache, "misses", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1926
d->rd_bcache_miss_collisions = rrddim_add(d->st_bcache, "collisions", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1927
d->rd_bcache_readaheads = rrddim_add(d->st_bcache, "readaheads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1928
+
1929
+ add_labels_to_disk(d, d->st_bcache);
1930
}
1853
- else rrdset_next(d->st_bcache);
1931
+ else
1932
+ rrdset_next(d->st_bcache);
1933
1934
rrddim_set_by_pointer(d->st_bcache, d->rd_bcache_hits, stats_total_cache_hits);
1935
rrddim_set_by_pointer(d->st_bcache, d->rd_bcache_misses, stats_total_cache_misses);
@@ -1884,6 +1963,8 @@ int do_proc_diskstats(int update_every, usec_t dt) {
1963
1964
d->rd_bcache_bypass_hits = rrddim_add(d->st_bcache_bypass, "hits", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1965
d->rd_bcache_bypass_misses = rrddim_add(d->st_bcache_bypass, "misses", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1966
+
1967
+ add_labels_to_disk(d, d->st_bcache_bypass);
1968
}
1969
else rrdset_next(d->st_bcache_bypass);
1970
collectors/proc.plugin/proc_interrupts.c
+4
@@ -225,6 +225,10 @@ int do_proc_interrupts(int update_every, usec_t dt) {
225
, update_every
226
, RRDSET_TYPE_STACKED
227
);
228
+
229
+ char core[50+1];
230
+ snprintfz(core, 50, "cpu%d", c);
231
+ rrdlabels_add(core_st[c]->state->chart_labels, "cpu", core, RRDLABEL_SRC_AUTO);
232
}
233
else rrdset_next(core_st[c]);
234
collectors/proc.plugin/proc_mdstat.c
+31
-7
@@ -77,6 +77,11 @@ static inline void make_chart_obsolete(char *name, const char *id_modifier)
77
}
78
}
79
80
+static void add_labels_to_mdstat(struct raid *raid, RRDSET *st) {
81
+ rrdlabels_add(st->state->chart_labels, "device", raid->name, RRDLABEL_SRC_AUTO);
82
+ rrdlabels_add(st->state->chart_labels, "raid_level", raid->level, RRDLABEL_SRC_AUTO);
83
+}
84
+
85
int do_proc_mdstat(int update_every, usec_t dt)
86
{
87
(void)dt;
@@ -407,7 +412,8 @@ int do_proc_mdstat(int update_every, usec_t dt)
412
RRDSET_TYPE_LINE);
413
414
rrdset_isnot_obsolete(st_mdstat_health);
410
- } else
415
+ }
416
+ else
417
rrdset_next(st_mdstat_health);
418
419
if (!redundant_num) {
@@ -458,7 +464,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
464
RRDSET_TYPE_STACKED);
465
466
rrdset_isnot_obsolete(raid->st_disks);
461
- } else
467
+
468
+ add_labels_to_mdstat(raid, raid->st_disks);
469
+ }
470
+ else
471
rrdset_next(raid->st_disks);
472
473
if (unlikely(!raid->rd_inuse && !(raid->rd_inuse = rrddim_find_active(raid->st_disks, "inuse"))))
@@ -495,7 +504,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
504
RRDSET_TYPE_LINE);
505
506
rrdset_isnot_obsolete(raid->st_mismatch_cnt);
498
- } else
507
+
508
+ add_labels_to_mdstat(raid, raid->st_mismatch_cnt);
509
+ }
510
+ else
511
rrdset_next(raid->st_mismatch_cnt);
512
513
if (unlikely(!raid->rd_mismatch_cnt && !(raid->rd_mismatch_cnt = rrddim_find_active(raid->st_mismatch_cnt, "count"))))
@@ -529,7 +541,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
541
RRDSET_TYPE_LINE);
542
543
rrdset_isnot_obsolete(raid->st_operation);
532
- } else
544
+
545
+ add_labels_to_mdstat(raid, raid->st_operation);
546
+ }
547
+ else
548
rrdset_next(raid->st_operation);
549
550
if(unlikely(!raid->rd_check && !(raid->rd_check = rrddim_find_active(raid->st_operation, "check"))))
@@ -569,7 +584,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
584
update_every, RRDSET_TYPE_LINE);
585
586
rrdset_isnot_obsolete(raid->st_finish);
572
- } else
587
+
588
+ add_labels_to_mdstat(raid, raid->st_finish);
589
+ }
590
+ else
591
rrdset_next(raid->st_finish);
592
593
if(unlikely(!raid->rd_finish_in && !(raid->rd_finish_in = rrddim_find_active(raid->st_finish, "finish_in"))))
@@ -601,7 +619,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
619
RRDSET_TYPE_LINE);
620
621
rrdset_isnot_obsolete(raid->st_speed);
604
- } else
622
+
623
+ add_labels_to_mdstat(raid, raid->st_speed);
624
+ }
625
+ else
626
rrdset_next(raid->st_speed);
627
628
if (unlikely(!raid->rd_speed && !(raid->rd_speed = rrddim_find_active(raid->st_speed, "speed"))))
@@ -635,7 +656,10 @@ int do_proc_mdstat(int update_every, usec_t dt)
656
RRDSET_TYPE_LINE);
657
658
rrdset_isnot_obsolete(raid->st_nonredundant);
638
- } else
659
+
660
+ add_labels_to_mdstat(raid, raid->st_nonredundant);
661
+ }
662
+ else
663
rrdset_next(raid->st_nonredundant);
664
665
if (unlikely(!raid->rd_nonredundant && !(raid->rd_nonredundant = rrddim_find_active(raid->st_nonredundant, "available"))))
collectors/proc.plugin/proc_net_dev.c
+39
-16
@@ -333,6 +333,7 @@ static struct netdev_rename {
333
334
const char *container_device;
335
const char *container_name;
336
+ const char *ctx_prefix;
337
338
DICTIONARY *chart_labels;
339
@@ -355,7 +356,13 @@ static struct netdev_rename *netdev_rename_find(const char *host_device, uint32_
356
}
357
358
// other threads can call this function to register a rename to a netdev
358
-void netdev_rename_device_add(const char *host_device, const char *container_device, const char *container_name, DICTIONARY *labels) {
359
+void netdev_rename_device_add(
360
+ const char *host_device,
361
+ const char *container_device,
362
+ const char *container_name,
363
+ DICTIONARY *labels,
364
+ const char *ctx_prefix)
365
+{
366
netdata_mutex_lock(&netdev_rename_mutex);
367
368
uint32_t hash = simple_hash(host_device);
@@ -365,6 +372,7 @@ void netdev_rename_device_add(const char *host_device, const char *container_dev
372
r->host_device = strdupz(host_device);
373
r->container_device = strdupz(container_device);
374
r->container_name = strdupz(container_name);
375
+ r->ctx_prefix = strdupz(ctx_prefix);
376
r->chart_labels = rrdlabels_create();
377
rrdlabels_migrate_to_these(r->chart_labels, labels);
378
r->hash = hash;
@@ -415,6 +423,7 @@ void netdev_rename_device_del(const char *host_device) {
423
freez((void *) r->host_device);
424
freez((void *) r->container_name);
425
freez((void *) r->container_device);
426
+ freez((void *) r->ctx_prefix);
427
rrdlabels_destroy(r->chart_labels);
428
freez((void *) r);
429
break;
@@ -471,18 +480,30 @@ static inline void netdev_rename_cgroup(struct netdev *d, struct netdev_rename *
480
snprintfz(buffer, RRD_ID_LENGTH_MAX, "net_mtu_%s", r->container_device);
481
d->chart_id_net_mtu = strdupz(buffer);
482
474
- d->chart_ctx_net_bytes = strdupz("cgroup.net_net");
475
- d->chart_ctx_net_compressed = strdupz("cgroup.net_compressed");
476
- d->chart_ctx_net_drops = strdupz("cgroup.net_drops");
477
- d->chart_ctx_net_errors = strdupz("cgroup.net_errors");
478
- d->chart_ctx_net_events = strdupz("cgroup.net_events");
479
- d->chart_ctx_net_fifo = strdupz("cgroup.net_fifo");
480
- d->chart_ctx_net_packets = strdupz("cgroup.net_packets");
481
- d->chart_ctx_net_speed = strdupz("cgroup.net_speed");
482
- d->chart_ctx_net_duplex = strdupz("cgroup.net_duplex");
483
- d->chart_ctx_net_operstate = strdupz("cgroup.net_operstate");
484
- d->chart_ctx_net_carrier = strdupz("cgroup.net_carrier");
485
- d->chart_ctx_net_mtu = strdupz("cgroup.net_mtu");
483
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_net", r->ctx_prefix);
484
+ d->chart_ctx_net_bytes = strdupz(buffer);
485
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_compressed", r->ctx_prefix);
486
+ d->chart_ctx_net_compressed = strdupz(buffer);
487
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_drops", r->ctx_prefix);
488
+ d->chart_ctx_net_drops = strdupz(buffer);
489
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_errors", r->ctx_prefix);
490
+ d->chart_ctx_net_errors = strdupz(buffer);
491
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_events", r->ctx_prefix);
492
+ d->chart_ctx_net_events = strdupz(buffer);
493
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_fifo", r->ctx_prefix);
494
+ d->chart_ctx_net_fifo = strdupz(buffer);
495
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_packets", r->ctx_prefix);
496
+ d->chart_ctx_net_packets = strdupz(buffer);
497
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_speed", r->ctx_prefix);
498
+ d->chart_ctx_net_speed = strdupz(buffer);
499
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_duplex", r->ctx_prefix);
500
+ d->chart_ctx_net_duplex = strdupz(buffer);
501
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_operstate", r->ctx_prefix);
502
+ d->chart_ctx_net_operstate = strdupz(buffer);
503
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_carrier", r->ctx_prefix);
504
+ d->chart_ctx_net_carrier = strdupz(buffer);
505
+ snprintfz(buffer, RRD_ID_LENGTH_MAX, "%scgroup.net_mtu", r->ctx_prefix);
506
+ d->chart_ctx_net_mtu = strdupz(buffer);
507
508
snprintfz(buffer, RRD_ID_LENGTH_MAX, "net %s", r->container_device);
509
d->chart_family = strdupz(buffer);
@@ -743,11 +764,13 @@ int do_proc_net_dev(int update_every, usec_t dt) {
764
snprintfz(buffer, FILENAME_MAX, path_to_sys_devices_virtual_net, d->name);
765
if (likely(access(buffer, R_OK) == 0)) {
766
d->virtual = 1;
746
- rrdlabels_add(d->chart_labels, "type", "virtual", RRDLABEL_SRC_AUTO);
747
- } else {
767
+ rrdlabels_add(d->chart_labels, "interface_type", "virtual", RRDLABEL_SRC_AUTO|RRDLABEL_FLAG_PERMANENT);
768
+ }
769
+ else {
770
d->virtual = 0;
749
- rrdlabels_add(d->chart_labels, "type", "real", RRDLABEL_SRC_AUTO);
771
+ rrdlabels_add(d->chart_labels, "interface_type", "real", RRDLABEL_SRC_AUTO|RRDLABEL_FLAG_PERMANENT);
772
}
773
+ rrdlabels_add(d->chart_labels, "device", name, RRDLABEL_SRC_AUTO|RRDLABEL_FLAG_PERMANENT);
774
775
if(likely(!d->virtual)) {
776
// set the filename to get the interface speed
collectors/proc.plugin/proc_net_wireless.c
+133
-141
@@ -198,6 +198,10 @@ static void configure_device(int do_status, int do_quality, int do_discarded_pac
198
wireless_dev->chart_id_net_missed_beacon = strdupz(buffer);
199
}
200
201
+static void add_labels_to_wireless(struct netwireless *w, RRDSET *st) {
202
+ rrdlabels_add(st->state->chart_labels, "device", w->name, RRDLABEL_SRC_AUTO);
203
+}
204
+
205
int do_proc_net_wireless(int update_every, usec_t dt)
206
{
207
UNUSED(dt);
@@ -209,21 +213,11 @@ int do_proc_net_wireless(int update_every, usec_t dt)
213
char filename[FILENAME_MAX + 1];
214
snprintfz(filename, FILENAME_MAX, "%s%s", netdata_configured_host_prefix, "/proc/net/wireless");
215
212
- proc_net_wireless_filename = config_get(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,"filename to monitor",
213
- filename);
214
-
215
- do_status = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,
216
- "status for all interfaces", CONFIG_BOOLEAN_AUTO);
217
-
218
- do_quality = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,
219
- "quality for all interfaces", CONFIG_BOOLEAN_AUTO);
220
-
221
- do_discarded_packets = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,
222
- "discarded packets for all interfaces",
223
- CONFIG_BOOLEAN_AUTO);
224
-
225
- do_beacon = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,
226
- "missed beacon for all interface", CONFIG_BOOLEAN_AUTO);
216
+ proc_net_wireless_filename = config_get(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS,"filename to monitor", filename);
217
+ do_status = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS, "status for all interfaces", CONFIG_BOOLEAN_AUTO);
218
+ do_quality = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS, "quality for all interfaces", CONFIG_BOOLEAN_AUTO);
219
+ do_discarded_packets = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS, "discarded packets for all interfaces", CONFIG_BOOLEAN_AUTO);
220
+ do_beacon = config_get_boolean_ondemand(CONFIG_SECTION_PLUGIN_PROC_NETWIRELESS, "missed beacon for all interface", CONFIG_BOOLEAN_AUTO);
221
}
222
223
if (unlikely(!ff)) {
@@ -255,25 +249,28 @@ int do_proc_net_wireless(int update_every, usec_t dt)
249
wireless_dev->status = str2kernel_uint_t(procfile_lineword(ff, l, 1));
250
251
if (unlikely(!wireless_dev->st_status)) {
258
- wireless_dev->st_status = rrdset_create_localhost("wireless",
259
- wireless_dev->chart_id_net_status,
260
- NULL,
261
- wireless_dev->name,
262
- "wireless.status",
263
- "Internal status reported by interface.",
264
- "status",
265
- PLUGIN_PROC_NAME,
266
- PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
267
- NETDATA_CHART_PRIO_WIRELESS_IFACE,
268
- update_every,
269
- RRDSET_TYPE_LINE);
252
+ wireless_dev->st_status = rrdset_create_localhost(
253
+ "wireless",
254
+ wireless_dev->chart_id_net_status,
255
+ NULL,
256
+ wireless_dev->name,
257
+ "wireless.status",
258
+ "Internal status reported by interface.",
259
+ "status",
260
+ PLUGIN_PROC_NAME,
261
+ PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
262
+ NETDATA_CHART_PRIO_WIRELESS_IFACE,
263
+ update_every,
264
+ RRDSET_TYPE_LINE);
265
+
266
rrdset_flag_set(wireless_dev->st_status, RRDSET_FLAG_DETAIL);
267
272
- wireless_dev->rd_status = rrddim_add(wireless_dev->st_status, "status", NULL, 1,
273
- 1, RRD_ALGORITHM_ABSOLUTE);
274
- } else {
275
- rrdset_next(wireless_dev->st_status);
268
+ wireless_dev->rd_status = rrddim_add(wireless_dev->st_status, "status", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
269
+
270
+ add_labels_to_wireless(wireless_dev, wireless_dev->st_status);
271
}
272
+ else
273
+ rrdset_next(wireless_dev->st_status);
274
275
rrddim_set_by_pointer(wireless_dev->st_status, wireless_dev->rd_status,
276
(collected_number)wireless_dev->status);
@@ -286,78 +283,81 @@ int do_proc_net_wireless(int update_every, usec_t dt)
283
wireless_dev->noise = str2ndd(procfile_lineword(ff, l, 4), NULL);
284
285
if (unlikely(!wireless_dev->st_link)) {
289
- wireless_dev->st_link = rrdset_create_localhost("wireless",
290
- wireless_dev->chart_id_net_link,
291
- NULL,
292
- wireless_dev->name,
293
- "wireless.link_quality",
294
- "Overall quality of the link. This is an aggregate value, and depends on the driver and hardware.",
295
- "value",
296
- PLUGIN_PROC_NAME,
297
- PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
298
- NETDATA_CHART_PRIO_WIRELESS_IFACE + 1,
299
- update_every,
300
- RRDSET_TYPE_LINE);
286
+ wireless_dev->st_link = rrdset_create_localhost(
287
+ "wireless",
288
+ wireless_dev->chart_id_net_link,
289
+ NULL,
290
+ wireless_dev->name,
291
+ "wireless.link_quality",
292
+ "Overall quality of the link. This is an aggregate value, and depends on the driver and hardware.",
293
+ "value",
294
+ PLUGIN_PROC_NAME,
295
+ PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
296
+ NETDATA_CHART_PRIO_WIRELESS_IFACE + 1,
297
+ update_every,
298
+ RRDSET_TYPE_LINE);
299
rrdset_flag_set(wireless_dev->st_link, RRDSET_FLAG_DETAIL);
300
303
- wireless_dev->rd_link = rrddim_add(wireless_dev->st_link, "link_quality", NULL, 1, 1,
304
- RRD_ALGORITHM_ABSOLUTE);
305
- } else {
306
- rrdset_next(wireless_dev->st_link);
301
+ wireless_dev->rd_link = rrddim_add(wireless_dev->st_link, "link_quality", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
302
+
303
+ add_labels_to_wireless(wireless_dev, wireless_dev->st_link);
304
}
305
+ else
306
+ rrdset_next(wireless_dev->st_link);
307
308
if (unlikely(!wireless_dev->st_level)) {
310
- wireless_dev->st_level = rrdset_create_localhost("wireless",
311
- wireless_dev->chart_id_net_level,
312
- NULL,
313
- wireless_dev->name,
314
- "wireless.signal_level",
315
- "The signal level is the wireless signal power level received by the wireless client. The closer the value is to 0, the stronger the signal.",
316
- "dBm",
317
- PLUGIN_PROC_NAME,
318
- PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
319
- NETDATA_CHART_PRIO_WIRELESS_IFACE + 2,
320
- update_every,
321
- RRDSET_TYPE_LINE);
309
+ wireless_dev->st_level = rrdset_create_localhost(
310
+ "wireless",
311
+ wireless_dev->chart_id_net_level,
312
+ NULL,
313
+ wireless_dev->name,
314
+ "wireless.signal_level",
315
+ "The signal level is the wireless signal power level received by the wireless client. The closer the value is to 0, the stronger the signal.",
316
+ "dBm",
317
+ PLUGIN_PROC_NAME,
318
+ PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
319
+ NETDATA_CHART_PRIO_WIRELESS_IFACE + 2,
320
+ update_every,
321
+ RRDSET_TYPE_LINE);
322
rrdset_flag_set(wireless_dev->st_level, RRDSET_FLAG_DETAIL);
323
324
- wireless_dev->rd_level = rrddim_add(wireless_dev->st_level, "signal_level", NULL, 1, 1,
325
- RRD_ALGORITHM_ABSOLUTE);
326
- } else {
327
- rrdset_next(wireless_dev->st_level);
324
+ wireless_dev->rd_level = rrddim_add(wireless_dev->st_level, "signal_level", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
325
+
326
+ add_labels_to_wireless(wireless_dev, wireless_dev->st_level);
327
}
328
+ else
329
+ rrdset_next(wireless_dev->st_level);
330
331
if (unlikely(!wireless_dev->st_noise)) {
331
- wireless_dev->st_noise = rrdset_create_localhost("wireless",
332
- wireless_dev->chart_id_net_noise,
333
- NULL,
334
- wireless_dev->name,
335
- "wireless.noise_level",
336
- "The noise level indicates the amount of background noise in your environment. The closer the value to 0, the greater the noise level.",
337
- "dBm",
338
- PLUGIN_PROC_NAME,
339
- PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
340
- NETDATA_CHART_PRIO_WIRELESS_IFACE + 3,
341
- update_every,
342
- RRDSET_TYPE_LINE);
332
+ wireless_dev->st_noise = rrdset_create_localhost(
333
+ "wireless",
334
+ wireless_dev->chart_id_net_noise,
335
+ NULL,
336
+ wireless_dev->name,
337
+ "wireless.noise_level",
338
+ "The noise level indicates the amount of background noise in your environment. The closer the value to 0, the greater the noise level.",
339
+ "dBm",
340
+ PLUGIN_PROC_NAME,
341
+ PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
342
+ NETDATA_CHART_PRIO_WIRELESS_IFACE + 3,
343
+ update_every,
344
+ RRDSET_TYPE_LINE);
345
rrdset_flag_set(wireless_dev->st_noise, RRDSET_FLAG_DETAIL);
346
345
- wireless_dev->rd_noise = rrddim_add(wireless_dev->st_noise, "noise_level", NULL, 1, 1,
346
- RRD_ALGORITHM_ABSOLUTE);
347
- } else {
348
- rrdset_next(wireless_dev->st_noise);
347
+ wireless_dev->rd_noise = rrddim_add(wireless_dev->st_noise, "noise_level", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
348
+
349
+ add_labels_to_wireless(wireless_dev, wireless_dev->st_noise);
350
}
351
+ else
352
+ rrdset_next(wireless_dev->st_noise);
353
351
- rrddim_set_by_pointer(wireless_dev->st_link, wireless_dev->rd_link,
352
- (collected_number)wireless_dev->link);
354
+ rrddim_set_by_pointer(wireless_dev->st_link, wireless_dev->rd_link, (collected_number)wireless_dev->link);
355
rrdset_done(wireless_dev->st_link);
356
355
- rrddim_set_by_pointer(wireless_dev->st_level, wireless_dev->rd_level,
356
- (collected_number)wireless_dev->level);
357
+ rrddim_set_by_pointer(wireless_dev->st_level, wireless_dev->rd_level, (collected_number)wireless_dev->level);
358
rrdset_done(wireless_dev->st_level);
359
359
- rrddim_set_by_pointer(wireless_dev->st_noise, wireless_dev->rd_noise,
360
- (collected_number)wireless_dev->noise);
360
+ rrddim_set_by_pointer(wireless_dev->st_noise, wireless_dev->rd_noise, (collected_number)wireless_dev->noise);
361
rrdset_done(wireless_dev->st_noise);
362
}
363
@@ -369,49 +369,38 @@ int do_proc_net_wireless(int update_every, usec_t dt)
369
wireless_dev->misc = str2kernel_uint_t(procfile_lineword(ff, l, 9));
370
371
if (unlikely(!wireless_dev->st_discarded_packets)) {
372
- wireless_dev->st_discarded_packets = rrdset_create_localhost("wireless",
373
- wireless_dev->chart_id_net_discarded_packets,
374
- NULL,
375
- wireless_dev->name,
376
- "wireless.discarded_packets",
377
- "Packet discarded in the wireless adapter due to \"wireless\" specific problems.",
378
- "packets/s",
379
- PLUGIN_PROC_NAME,
380
- PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
381
- NETDATA_CHART_PRIO_WIRELESS_IFACE + 4,
382
- update_every,
383
- RRDSET_TYPE_LINE);
372
+ wireless_dev->st_discarded_packets = rrdset_create_localhost(
373
+ "wireless",
374
+ wireless_dev->chart_id_net_discarded_packets,
375
+ NULL,
376
+ wireless_dev->name,
377
+ "wireless.discarded_packets",
378
+ "Packet discarded in the wireless adapter due to \"wireless\" specific problems.",
379
+ "packets/s",
380
+ PLUGIN_PROC_NAME,
381
+ PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
382
+ NETDATA_CHART_PRIO_WIRELESS_IFACE + 4,
383
+ update_every,
384
+ RRDSET_TYPE_LINE);
385
386
rrdset_flag_set(wireless_dev->st_discarded_packets, RRDSET_FLAG_DETAIL);
387
387
- wireless_dev->rd_nwid = rrddim_add(wireless_dev->st_discarded_packets, "nwid", NULL, 1,
388
- 1, RRD_ALGORITHM_INCREMENTAL);
389
- wireless_dev->rd_crypt = rrddim_add(wireless_dev->st_discarded_packets, "crypt", NULL, 1,
390
- 1, RRD_ALGORITHM_INCREMENTAL);
391
- wireless_dev->rd_frag = rrddim_add(wireless_dev->st_discarded_packets, "frag", NULL, 1,
392
- 1, RRD_ALGORITHM_INCREMENTAL);
393
- wireless_dev->rd_retry = rrddim_add(wireless_dev->st_discarded_packets, "retry", NULL, 1,
394
- 1, RRD_ALGORITHM_INCREMENTAL);
395
- wireless_dev->rd_misc = rrddim_add(wireless_dev->st_discarded_packets, "misc", NULL, 1,
396
- 1, RRD_ALGORITHM_INCREMENTAL);
397
- } else {
398
- rrdset_next(wireless_dev->st_discarded_packets);
399
- }
400
-
401
- rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_nwid,
402
- (collected_number)wireless_dev->nwid);
403
-
404
- rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_crypt,
405
- (collected_number)wireless_dev->crypt);
388
+ wireless_dev->rd_nwid = rrddim_add(wireless_dev->st_discarded_packets, "nwid", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
389
+ wireless_dev->rd_crypt = rrddim_add(wireless_dev->st_discarded_packets, "crypt", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
390
+ wireless_dev->rd_frag = rrddim_add(wireless_dev->st_discarded_packets, "frag", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
391
+ wireless_dev->rd_retry = rrddim_add(wireless_dev->st_discarded_packets, "retry", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
392
+ wireless_dev->rd_misc = rrddim_add(wireless_dev->st_discarded_packets, "misc", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
393
407
- rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_frag,
408
- (collected_number)wireless_dev->frag);
409
-
410
- rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_retry,
411
- (collected_number)wireless_dev->retry);
394
+ add_labels_to_wireless(wireless_dev, wireless_dev->st_discarded_packets);
395
+ }
396
+ else
397
+ rrdset_next(wireless_dev->st_discarded_packets);
398
413
- rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_misc,
414
- (collected_number)wireless_dev->misc);
399
+ rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_nwid, (collected_number)wireless_dev->nwid);
400
+ rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_crypt, (collected_number)wireless_dev->crypt);
401
+ rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_frag, (collected_number)wireless_dev->frag);
402
+ rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_retry, (collected_number)wireless_dev->retry);
403
+ rrddim_set_by_pointer(wireless_dev->st_discarded_packets, wireless_dev->rd_misc, (collected_number)wireless_dev->misc);
404
405
rrdset_done(wireless_dev->st_discarded_packets);
406
}
@@ -420,28 +409,31 @@ int do_proc_net_wireless(int update_every, usec_t dt)
409
wireless_dev->missed_beacon = str2kernel_uint_t(procfile_lineword(ff, l, 10));
410
411
if (unlikely(!wireless_dev->st_missed_beacon)) {
423
- wireless_dev->st_missed_beacon = rrdset_create_localhost("wireless",
424
- wireless_dev->chart_id_net_missed_beacon,
425
- NULL,
426
- wireless_dev->name,
427
- "wireless.missed_beacons",
428
- "Number of missed beacons.",
429
- "frames/s",
430
- PLUGIN_PROC_NAME,
431
- PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
432
- NETDATA_CHART_PRIO_WIRELESS_IFACE + 5,
433
- update_every,
434
- RRDSET_TYPE_LINE);
412
+ wireless_dev->st_missed_beacon = rrdset_create_localhost(
413
+ "wireless",
414
+ wireless_dev->chart_id_net_missed_beacon,
415
+ NULL,
416
+ wireless_dev->name,
417
+ "wireless.missed_beacons",
418
+ "Number of missed beacons.",
419
+ "frames/s",
420
+ PLUGIN_PROC_NAME,
421
+ PLUGIN_PROC_MODULE_NETWIRELESS_NAME,
422
+ NETDATA_CHART_PRIO_WIRELESS_IFACE + 5,
423
+ update_every,
424
+ RRDSET_TYPE_LINE);
425
+
426
rrdset_flag_set(wireless_dev->st_missed_beacon, RRDSET_FLAG_DETAIL);
427
437
- wireless_dev->rd_missed_beacon = rrddim_add(wireless_dev->st_missed_beacon, "missed_beacons",
438
- NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
439
- } else {
440
- rrdset_next(wireless_dev->st_missed_beacon);
428
+ wireless_dev->rd_missed_beacon = rrddim_add(wireless_dev->st_missed_beacon, "missed_beacons", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
429
+
430
+ add_labels_to_wireless(wireless_dev, wireless_dev->st_missed_beacon);
431
}
432
+ else
433
+ rrdset_next(wireless_dev->st_missed_beacon);
434
+
435
+ rrddim_set_by_pointer(wireless_dev->st_missed_beacon, wireless_dev->rd_missed_beacon, (collected_number)wireless_dev->missed_beacon);
436
443
- rrddim_set_by_pointer(wireless_dev->st_missed_beacon, wireless_dev->rd_missed_beacon,
444
- (collected_number)wireless_dev->missed_beacon);
437
rrdset_done(wireless_dev->st_missed_beacon);
438
}
439
collectors/proc.plugin/proc_pagetypeinfo.c
+9
-3
@@ -242,15 +242,14 @@ int do_proc_pagetypeinfo(int update_every, usec_t dt) {
242
243
// "Node" + NUMA-NodeID + ZoneName + TypeName
244
char setname[4+1+MAX_ZONETYPE_NAME+1+MAX_PAGETYPE_NAME +1];
245
- snprintfz(setname, MAX_ZONETYPE_NAME + MAX_PAGETYPE_NAME, "Node %d %s %s",
246
- pgl->node, pgl->zone, pgl->type);
245
+ snprintfz(setname, MAX_ZONETYPE_NAME + MAX_PAGETYPE_NAME, "Node %d %s %s", pgl->node, pgl->zone, pgl->type);
246
247
st_nodezonetype[p] = rrdset_create_localhost(
248
"mem"
249
, setid
250
, NULL
251
, "pagetype"
253
- , NULL
252
+ , "mem.pagetype"
253
, setname
254
, "B"
255
, PLUGIN_PROC_NAME
@@ -259,6 +258,13 @@ int do_proc_pagetypeinfo(int update_every, usec_t dt) {
258
, update_every
259
, RRDSET_TYPE_STACKED
260
);
261
+
262
+ char node[50+1];
263
+ snprintfz(node, 50, "node%d", pgl->node);
264
+ rrdlabels_add(st_nodezonetype[p]->state->chart_labels, "node_id", node, RRDLABEL_SRC_AUTO);
265
+ rrdlabels_add(st_nodezonetype[p]->state->chart_labels, "node_zone", pgl->zone, RRDLABEL_SRC_AUTO);
266
+ rrdlabels_add(st_nodezonetype[p]->state->chart_labels, "node_type", pgl->type, RRDLABEL_SRC_AUTO);
267
+
268
for (o = 0; o < pageorders_cnt; o++) {
269
char dimid[3+1];
270
snprintfz(dimid, 3, "%lu", o);
collectors/proc.plugin/proc_softirqs.c
+4
@@ -217,6 +217,10 @@ int do_proc_softirqs(int update_every, usec_t dt) {
217
, update_every
218
, RRDSET_TYPE_STACKED
219
);
220
+
221
+ char core[50+1];
222
+ snprintfz(core, 50, "cpu%d", c);
223
+ rrdlabels_add(core_st[c]->state->chart_labels, "cpu", core, RRDLABEL_SRC_AUTO);
224
}
225
else
226
rrdset_next(core_st[c]);
collectors/proc.plugin/proc_stat.c
+4
@@ -1039,6 +1039,10 @@ int do_proc_stat(int update_every, usec_t dt) {
1039
, RRDSET_TYPE_STACKED
1040
);
1041
1042
+ char corebuf[50+1];
1043
+ snprintfz(corebuf, 50, "cpu%zu", core);
1044
+ rrdlabels_add(cpuidle_charts[core].st->state->chart_labels, "cpu", corebuf, RRDLABEL_SRC_AUTO);
1045
+
1046
char cpuidle_dim_id[RRD_ID_LENGTH_MAX + 1];
1047
cpuidle_charts[core].active_time_rd = rrddim_add(cpuidle_charts[core].st, "active", "C0 (active)", 1, 1, RRD_ALGORITHM_PCENT_OVER_DIFF_TOTAL);
1048
for(state = 0; state < cpuidle_charts[core].cpuidle_state_len; state++) {
collectors/proc.plugin/sys_block_zram.c
+4
@@ -75,6 +75,7 @@ static inline void init_rrd(const char *name, ZRAM_DEVICE *d, int update_every)
75
, RRDSET_TYPE_AREA);
76
d->rd_compr_data_size = rrddim_add(d->st_usage, "compressed", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
77
d->rd_metadata_size = rrddim_add(d->st_usage, "metadata", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
78
+ rrdlabels_add(d->st_usage->state->chart_labels, "device", name, RRDLABEL_SRC_AUTO);
79
80
snprintfz(chart_name, RRD_ID_LENGTH_MAX, "zram_savings.%s", name);
81
d->st_savings = rrdset_create_localhost(
@@ -92,6 +93,7 @@ static inline void init_rrd(const char *name, ZRAM_DEVICE *d, int update_every)
93
, RRDSET_TYPE_AREA);
94
d->rd_savings_size = rrddim_add(d->st_savings, "savings", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
95
d->rd_original_size = rrddim_add(d->st_savings, "original", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
96
+ rrdlabels_add(d->st_savings->state->chart_labels, "device", name, RRDLABEL_SRC_AUTO);
97
98
snprintfz(chart_name, RRD_ID_LENGTH_MAX, "zram_ratio.%s", name);
99
d->st_comp_ratio = rrdset_create_localhost(
@@ -108,6 +110,7 @@ static inline void init_rrd(const char *name, ZRAM_DEVICE *d, int update_every)
110
, update_every
111
, RRDSET_TYPE_LINE);
112
d->rd_comp_ratio = rrddim_add(d->st_comp_ratio, "ratio", NULL, 1, 100, RRD_ALGORITHM_ABSOLUTE);
113
+ rrdlabels_add(d->st_comp_ratio->state->chart_labels, "device", name, RRDLABEL_SRC_AUTO);
114
115
snprintfz(chart_name, RRD_ID_LENGTH_MAX, "zram_efficiency.%s", name);
116
d->st_alloc_efficiency = rrdset_create_localhost(
@@ -124,6 +127,7 @@ static inline void init_rrd(const char *name, ZRAM_DEVICE *d, int update_every)
127
, update_every
128
, RRDSET_TYPE_LINE);
129
d->rd_alloc_efficiency = rrddim_add(d->st_alloc_efficiency, "percent", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
130
+ rrdlabels_add(d->st_alloc_efficiency->state->chart_labels, "device", name, RRDLABEL_SRC_AUTO);
131
}
132
133
static int init_devices(DICTIONARY *devices, unsigned int zram_id, int update_every) {
collectors/proc.plugin/sys_class_power_supply.c
+8
@@ -112,6 +112,10 @@ void power_supply_free(struct power_supply *ps) {
112
}
113
}
114
115
+static void add_labels_to_power_supply(struct power_supply *ps, RRDSET *st) {
116
+ rrdlabels_add(st->state->chart_labels, "device", ps->name, RRDLABEL_SRC_AUTO);
117
+}
118
+
119
int do_sys_class_power_supply(int update_every, usec_t dt) {
120
(void)dt;
121
static int do_capacity = -1, do_property[3] = {-1};
@@ -358,6 +362,8 @@ int do_sys_class_power_supply(int update_every, usec_t dt) {
362
, update_every
363
, RRDSET_TYPE_LINE
364
);
365
+
366
+ add_labels_to_power_supply(ps, ps->capacity->st);
367
}
368
else
369
rrdset_next(ps->capacity->st);
@@ -389,6 +395,8 @@ int do_sys_class_power_supply(int update_every, usec_t dt) {
395
, update_every
396
, RRDSET_TYPE_LINE
397
);
398
+
399
+ add_labels_to_power_supply(ps, pr->st);
400
}
401
else
402
rrdset_next(pr->st);
collectors/proc.plugin/sys_devices_system_node.c
+2
@@ -115,6 +115,8 @@ int do_proc_sys_devices_system_node(int update_every, usec_t dt) {
115
, RRDSET_TYPE_LINE
116
);
117
118
+ rrdlabels_add(m->numastat_st->state->chart_labels, "numa_node", m->name, RRDLABEL_SRC_AUTO);
119
+
120
rrdset_flag_set(m->numastat_st, RRDSET_FLAG_DETAIL);
121
122
rrddim_add(m->numastat_st, "numa_hit", "hit", 1, 1, RRD_ALGORITHM_INCREMENTAL);
collectors/proc.plugin/sys_fs_btrfs.c
+13
@@ -448,6 +448,11 @@ static inline int find_all_btrfs_pools(const char *path) {
448
return 0;
449
}
450
451
+static void add_labels_to_btrfs(BTRFS_NODE *n, RRDSET *st) {
452
+ rrdlabels_add(st->state->chart_labels, "device", n->id, RRDLABEL_SRC_AUTO);
453
+ rrdlabels_add(st->state->chart_labels, "device_label", n->label, RRDLABEL_SRC_AUTO);
454
+}
455
+
456
int do_sys_fs_btrfs(int update_every, usec_t dt) {
457
static int initialized = 0
458
, do_allocation_disks = CONFIG_BOOLEAN_AUTO
@@ -579,6 +584,8 @@ int do_sys_fs_btrfs(int update_every, usec_t dt) {
584
node->rd_allocation_disks_metadata_used = rrddim_add(node->st_allocation_disks, "meta_used", "meta used", 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
585
node->rd_allocation_disks_system_free = rrddim_add(node->st_allocation_disks, "sys_free", "sys free", 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
586
node->rd_allocation_disks_system_used = rrddim_add(node->st_allocation_disks, "sys_used", "sys used", 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
587
+
588
+ add_labels_to_btrfs(node, node->st_allocation_disks);
589
}
590
else rrdset_next(node->st_allocation_disks);
591
@@ -632,6 +639,8 @@ int do_sys_fs_btrfs(int update_every, usec_t dt) {
639
640
node->rd_allocation_data_free = rrddim_add(node->st_allocation_data, "free", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
641
node->rd_allocation_data_used = rrddim_add(node->st_allocation_data, "used", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
642
+
643
+ add_labels_to_btrfs(node, node->st_allocation_data);
644
}
645
else rrdset_next(node->st_allocation_data);
646
@@ -676,6 +685,8 @@ int do_sys_fs_btrfs(int update_every, usec_t dt) {
685
node->rd_allocation_metadata_free = rrddim_add(node->st_allocation_metadata, "free", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
686
node->rd_allocation_metadata_used = rrddim_add(node->st_allocation_metadata, "used", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
687
node->rd_allocation_metadata_reserved = rrddim_add(node->st_allocation_metadata, "reserved", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
688
+
689
+ add_labels_to_btrfs(node, node->st_allocation_metadata);
690
}
691
else rrdset_next(node->st_allocation_metadata);
692
@@ -720,6 +731,8 @@ int do_sys_fs_btrfs(int update_every, usec_t dt) {
731
732
node->rd_allocation_system_free = rrddim_add(node->st_allocation_system, "free", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
733
node->rd_allocation_system_used = rrddim_add(node->st_allocation_system, "used", NULL, 1, 1024 * 1024, RRD_ALGORITHM_ABSOLUTE);
734
+
735
+ add_labels_to_btrfs(node, node->st_allocation_system);
736
}
737
else rrdset_next(node->st_allocation_system);
738
daemon/analytics.c
+1
-1
@@ -541,7 +541,7 @@ void analytics_gather_mutable_meta_data(void)
541
analytics_set_data(
542
&analytics_data.netdata_config_is_parent, (localhost->next || configured_as_parent()) ? "true" : "false");
543
544
- char *claim_id = is_agent_claimed();
544
+ char *claim_id = get_agent_claimid();
545
analytics_set_data(&analytics_data.netdata_host_agent_claimed, claim_id ? "true" : "false");
546
freez(claim_id);
547
daemon/global_statistics.c
+1
@@ -996,6 +996,7 @@ static struct worker_utilization all_workers_utilization[] = {
996
{ .name = "TC", .family = "workers plugin tc", .priority = 1000000 },
997
{ .name = "TIMEX", .family = "workers plugin timex", .priority = 1000000 },
998
{ .name = "IDLEJITTER", .family = "workers plugin idlejitter", .priority = 1000000 },
999
+ { .name = "RRDCONTEXT", .family = "workers aclk contexts", .priority = 1000000 },
1000
1001
// has to be terminated with a NULL
1002
{ .name = NULL, .family = NULL }
daemon/main.c
+11
@@ -64,6 +64,7 @@ void netdata_cleanup_and_exit(int ret) {
64
rrdeng_exit(multidb_ctx[tier]);
65
#endif
66
}
67
+ sql_close_context_database();
68
sql_close_database();
69
70
// unlink the pid
@@ -732,6 +733,12 @@ static void get_netdata_configured_variables() {
733
config_set_number(CONFIG_SECTION_DB, "gap when lost iterations above", gap_when_lost_iterations_above);
734
}
735
736
+ // --------------------------------------------------------------------
737
+ // rrdcontext
738
+
739
+ rrdcontext_enabled = config_get_boolean(CONFIG_SECTION_CLOUD, "rrdcontexts", rrdcontext_enabled);
740
+
741
+
742
// --------------------------------------------------------------------
743
// get various system parameters
744
@@ -978,6 +985,7 @@ int main(int argc, char **argv) {
985
// No call to load the config file on this code-path
986
post_conf_load(&user);
987
get_netdata_configured_variables();
988
+ rrdcontext_enabled = CONFIG_BOOLEAN_NO;
989
default_rrd_update_every = 1;
990
default_rrd_memory_mode = RRD_MEMORY_MODE_RAM;
991
default_health_enabled = 0;
@@ -1010,6 +1018,9 @@ int main(int argc, char **argv) {
1018
else if(strcmp(optarg, "mctest") == 0) {
1019
return mc_unittest();
1020
}
1021
+ else if(strcmp(optarg, "ctxtest") == 0) {
1022
+ return ctx_unittest();
1023
+ }
1024
else if(strcmp(optarg, "dicttest") == 0) {
1025
return dictionary_unittest(10000);
1026
}
daemon/static_threads.c
+10
@@ -135,6 +135,16 @@ const struct netdata_static_thread static_threads_common[] = {
135
},
136
#endif
137
138
+ {
139
+ .name = "rrdcontext",
140
+ .config_section = NULL,
141
+ .config_name = NULL,
142
+ .enabled = 1,
143
+ .thread = NULL,
144
+ .init_routine = NULL,
145
+ .start_routine = rrdcontext_main
146
+ },
147
+
148
{NULL, NULL, NULL, 0, NULL, NULL, NULL}
149
};
150
daemon/unit_test.c
+2
-1
@@ -1772,6 +1772,7 @@ static RRDHOST *dbengine_rrdhost_find_or_create(char *name)
1772
, default_rrdpush_api_key
1773
, default_rrdpush_send_charts_matching
1774
, NULL
1775
+ , 0
1776
);
1777
}
1778
@@ -2279,7 +2280,7 @@ void generate_dbengine_dataset(unsigned history_seconds)
2280
}
2281
freez(thread_info);
2282
rrd_wrlock();
2282
- rrdhost_free(host);
2283
+ rrdhost_free(host, 1);
2284
rrd_unlock();
2285
}
2286
database/engine/rrdengine.c
+1
@@ -887,6 +887,7 @@ static void after_delete_old_data(struct rrdengine_worker_config* wc)
887
888
wc->cleanup_thread_deleting_files = 0;
889
aclk_data_rotated();
890
+ rrdcontext_db_rotation();
891
892
/* interrupt event loop */
893
uv_stop(wc->loop);
database/engine/rrdengineapi.c
+30
@@ -623,6 +623,36 @@ int rrdeng_metric_latest_time_by_uuid(uuid_t *dim_uuid, time_t *first_entry_t, t
623
return 1;
624
}
625
626
+int rrdeng_metric_retention_by_uuid(STORAGE_INSTANCE *si, uuid_t *dim_uuid, time_t *first_entry_t, time_t *last_entry_t)
627
+{
628
+ struct page_cache *pg_cache;
629
+ struct rrdengine_instance *ctx;
630
+ Pvoid_t *PValue;
631
+ struct pg_cache_page_index *page_index = NULL;
632
+
633
+ ctx = (struct rrdengine_instance *)si;
634
+ if (unlikely(!ctx)) {
635
+ error("DBENGINE: invalid STORAGE INSTANCE to %s()", __FUNCTION__);
636
+ return 1;
637
+ }
638
+ pg_cache = &ctx->pg_cache;
639
+
640
+ uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
641
+ PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, dim_uuid, sizeof(uuid_t));
642
+ if (likely(NULL != PValue)) {
643
+ page_index = *PValue;
644
+ }
645
+ uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
646
+
647
+ if (likely(page_index)) {
648
+ *first_entry_t = page_index->oldest_time / USEC_PER_SEC;
649
+ *last_entry_t = page_index->latest_time / USEC_PER_SEC;
650
+ return 0;
651
+ }
652
+
653
+ return 1;
654
+}
655
+
656
/* Also gets a reference for the page */
657
void *rrdeng_create_page(struct rrdengine_instance *ctx, uuid_t *id, struct rrdeng_page_descr **ret_descr)
658
{
database/engine/rrdengineapi.h
+1
@@ -76,5 +76,6 @@ extern int rrdeng_init(RRDHOST *host, struct rrdengine_instance **ctxp, char *db
76
extern int rrdeng_exit(struct rrdengine_instance *ctx);
77
extern void rrdeng_prepare_exit(struct rrdengine_instance *ctx);
78
extern int rrdeng_metric_latest_time_by_uuid(uuid_t *dim_uuid, time_t *first_entry_t, time_t *last_entry_t, int tier);
79
+extern int rrdeng_metric_retention_by_uuid(STORAGE_INSTANCE *si, uuid_t *dim_uuid, time_t *first_entry_t, time_t *last_entry_t);
80
81
#endif /* NETDATA_RRDENGINEAPI_H */
database/rrd.h
+35
-24
@@ -49,6 +49,7 @@ struct pg_cache_page_index;
49
#include "streaming/rrdpush.h"
50
#include "aclk/aclk_rrdhost_state.h"
51
#include "sqlite/sqlite_health.h"
52
+#include "rrdcontext.h"
53
54
extern int storage_tiers;
55
extern int storage_tiers_grouping_iterations[RRD_STORAGE_TIERS];
@@ -196,15 +197,16 @@ typedef enum rrddim_flags {
197
#define rrddim_flag_clear(rd, flag) __atomic_and_fetch(&((rd)->flags), ~(flag), __ATOMIC_SEQ_CST)
198
199
typedef enum rrdlabel_source {
199
- RRDLABEL_SRC_AUTO = (1 << 0), // set when Netdata found the label by some automation
200
- RRDLABEL_SRC_CONFIG = (1 << 1), // set when the user configured the label
201
- RRDLABEL_SRC_K8S = (1 << 2), // set when this label is found from k8s (RRDLABEL_SRC_AUTO should also be set)
202
- RRDLABEL_SRC_ACLK = (1 << 3), // set when this label is found from ACLK (RRDLABEL_SRC_AUTO should also be set)
200
+ RRDLABEL_SRC_AUTO = (1 << 0), // set when Netdata found the label by some automation
201
+ RRDLABEL_SRC_CONFIG = (1 << 1), // set when the user configured the label
202
+ RRDLABEL_SRC_K8S = (1 << 2), // set when this label is found from k8s (RRDLABEL_SRC_AUTO should also be set)
203
+ RRDLABEL_SRC_ACLK = (1 << 3), // set when this label is found from ACLK (RRDLABEL_SRC_AUTO should also be set)
204
205
// more sources can be added here
206
206
- RRDLABEL_FLAG_OLD = (1 << 30), // marks set for rrdlabels internal use - they are not exposed outside rrdlabels
207
- RRDLABEL_FLAG_NEW = (1 << 31) //
207
+ RRDLABEL_FLAG_PERMANENT = (1 << 29), // set when this label should never be removed (can be overwritten though)
208
+ RRDLABEL_FLAG_OLD = (1 << 30), // marks for rrdlabels internal use - they are not exposed outside rrdlabels
209
+ RRDLABEL_FLAG_NEW = (1 << 31) // marks for rrdlabels internal use - they are not exposed outside rrdlabels
210
} RRDLABEL_SRC;
211
212
extern DICTIONARY *rrdlabels_create(void);
@@ -222,7 +224,7 @@ extern int rrdlabels_sorted_walkthrough_read(DICTIONARY *labels, int (*callback)
224
extern void rrdlabels_log_to_buffer(DICTIONARY *labels, BUFFER *wb);
225
extern bool rrdlabels_match_simple_pattern(DICTIONARY *labels, const char *simple_pattern_txt);
226
extern bool rrdlabels_match_simple_pattern_parsed(DICTIONARY *labels, SIMPLE_PATTERN *pattern, char equal);
225
-extern void rrdlabels_to_buffer(DICTIONARY *labels, BUFFER *wb, const char *before_each, const char *equal, const char *quote, const char *between_them, bool (*filter_callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *filter_data, void (*name_sanitizer)(char *dst, const char *src, size_t dst_size), void (*value_sanitizer)(char *dst, const char *src, size_t dst_size));
227
+extern int rrdlabels_to_buffer(DICTIONARY *labels, BUFFER *wb, const char *before_each, const char *equal, const char *quote, const char *between_them, bool (*filter_callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *filter_data, void (*name_sanitizer)(char *dst, const char *src, size_t dst_size), void (*value_sanitizer)(char *dst, const char *src, size_t dst_size));
228
229
extern void rrdlabels_migrate_to_these(DICTIONARY *dst, DICTIONARY *src);
230
extern void rrdlabels_copy(DICTIONARY *dst, DICTIONARY *src);
@@ -284,7 +286,7 @@ struct rrddim {
286
287
struct rrddim_tier *tiers[RRD_STORAGE_TIERS]; // our tiers of databases
288
287
- size_t collections_counter; // the number of times we added values to this rrdim
289
+ size_t collections_counter; // the number of times we added values to this rrddim
290
collected_number collected_value_max; // the absolute maximum of the collected value
291
292
NETDATA_DOUBLE calculated_value; // the current calculated value, after applying the algorithm - resets to zero after being used
@@ -296,11 +298,12 @@ struct rrddim {
298
299
// the *_volume members are used to calculate the accuracy of the rounding done by the
300
// storage number - they are printed to debug.log when debug is enabled for a set.
299
- NETDATA_DOUBLE collected_volume; // the sum of all collected values so far
300
- NETDATA_DOUBLE stored_volume; // the sum of all stored values so far
301
+ NETDATA_DOUBLE collected_volume; // the sum of all collected values so far
302
+ NETDATA_DOUBLE stored_volume; // the sum of all stored values so far
303
304
struct rrddim *next; // linking of dimensions within the same data set
305
struct rrdset *rrdset;
306
+ RRDMETRIC_ACQUIRED *rrdmetric; // the rrdmetric of this dimension
307
308
// ------------------------------------------------------------------------
309
// members for checking the data when loading from disk
@@ -531,6 +534,9 @@ struct rrdset {
534
char *context; // the template of this data set
535
uint32_t hash_context; // the hash of the chart's context
536
537
+ RRDINSTANCE_ACQUIRED *rrdinstance; // the rrdinstance of this chart
538
+ RRDCONTEXT_ACQUIRED *rrdcontext; // the rrdcontext this chart belongs to
539
+
540
RRDSET_TYPE chart_type; // line, area, stacked
541
542
int update_every; // every how many seconds is this updated?
@@ -644,17 +650,17 @@ extern bool rrdset_memory_load_or_create_map_save(RRDSET *st_on_file, RRD_MEMORY
650
// and may lead to missing information.
651
652
typedef enum rrdhost_flags {
647
- RRDHOST_FLAG_ORPHAN = 1 << 0, // this host is orphan (not receiving data)
648
- RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS = 1 << 1, // delete files of obsolete charts
649
- RRDHOST_FLAG_DELETE_ORPHAN_HOST = 1 << 2, // delete the entire host when orphan
650
- RRDHOST_FLAG_EXPORTING_SEND = 1 << 3, // send it to external databases
651
- RRDHOST_FLAG_EXPORTING_DONT_SEND = 1 << 4, // don't send it to external databases
652
- RRDHOST_FLAG_ARCHIVED = 1 << 5, // The host is archived, no collected charts yet
653
- RRDHOST_FLAG_MULTIHOST = 1 << 6, // Host belongs to localhost/megadb
654
- RRDHOST_FLAG_PENDING_FOREACH_ALARMS = 1 << 7, // contains dims with uninitialized foreach alarms
655
- RRDHOST_FLAG_STREAM_LABELS_UPDATE = 1 << 8,
656
- RRDHOST_FLAG_STREAM_LABELS_STOP = 1 << 9,
657
-
653
+ RRDHOST_FLAG_ORPHAN = (1 << 0), // this host is orphan (not receiving data)
654
+ RRDHOST_FLAG_DELETE_OBSOLETE_CHARTS = (1 << 1), // delete files of obsolete charts
655
+ RRDHOST_FLAG_DELETE_ORPHAN_HOST = (1 << 2), // delete the entire host when orphan
656
+ RRDHOST_FLAG_EXPORTING_SEND = (1 << 3), // send it to external databases
657
+ RRDHOST_FLAG_EXPORTING_DONT_SEND = (1 << 4), // don't send it to external databases
658
+ RRDHOST_FLAG_ARCHIVED = (1 << 5), // The host is archived, no collected charts yet
659
+ RRDHOST_FLAG_MULTIHOST = (1 << 6), // Host belongs to localhost/megadb
660
+ RRDHOST_FLAG_PENDING_FOREACH_ALARMS = (1 << 7), // contains dims with uninitialized foreach alarms
661
+ RRDHOST_FLAG_STREAM_LABELS_UPDATE = (1 << 8),
662
+ RRDHOST_FLAG_STREAM_LABELS_STOP = (1 << 9),
663
+ RRDHOST_FLAG_ACLK_STREAM_CONTEXTS = (1 << 10), // when set, we should send ACLK stream context updates
664
} RRDHOST_FLAGS;
665
666
#define rrdhost_flag_check(host, flag) (__atomic_load_n(&((host)->flags), __ATOMIC_SEQ_CST) & (flag))
@@ -925,6 +931,9 @@ struct rrdhost {
931
932
STORAGE_INSTANCE *storage_instance[RRD_STORAGE_TIERS]; // the database instances of the storage tiers
933
934
+ RRDCONTEXTS *rrdctx_queue;
935
+ RRDCONTEXTS *rrdctx;
936
+
937
uuid_t host_uuid; // Global GUID for this host
938
uuid_t *node_id; // Cloud node_id
939
@@ -1000,6 +1009,7 @@ extern RRDHOST *rrdhost_find_or_create(
1009
, char *rrdpush_api_key
1010
, char *rrdpush_send_charts_matching
1011
, struct rrdhost_system_info *system_info
1012
+ , bool is_archived
1013
);
1014
1015
extern void rrdhost_update(RRDHOST *host
@@ -1083,7 +1093,7 @@ extern void rrdhost_cleanup_all(void);
1093
1094
extern void rrdhost_cleanup_orphan_hosts_nolock(RRDHOST *protected_host);
1095
extern void rrdhost_system_info_free(struct rrdhost_system_info *system_info);
1086
-extern void rrdhost_free(RRDHOST *host);
1096
+extern void rrdhost_free(RRDHOST *host, bool force);
1097
extern void rrdhost_save_charts(RRDHOST *host);
1098
extern void rrdhost_delete_charts(RRDHOST *host);
1099
extern void rrd_cleanup_obsolete_charts();
@@ -1291,7 +1301,7 @@ extern RRDSET *rrdset_index_del_name(RRDHOST *host, RRDSET *st);
1301
extern void rrdset_free(RRDSET *st);
1302
extern void rrdset_reset(RRDSET *st);
1303
extern void rrdset_save(RRDSET *st);
1294
-extern void rrdset_delete(RRDSET *st);
1304
+extern void rrdset_delete_files(RRDSET *st);
1305
extern void rrdset_delete_obsolete_dimensions(RRDSET *st);
1306
1307
extern RRDHOST *rrdhost_create(
@@ -1299,7 +1309,7 @@ extern RRDHOST *rrdhost_create(
1309
const char *abbrev_timezone, int32_t utc_offset,const char *tags, const char *program_name, const char *program_version,
1310
int update_every, long entries, RRD_MEMORY_MODE memory_mode, unsigned int health_enabled, unsigned int rrdpush_enabled,
1311
char *rrdpush_destination, char *rrdpush_api_key, char *rrdpush_send_charts_matching, struct rrdhost_system_info *system_info,
1302
- int is_localhost); //TODO: Remove , int is_archived);
1312
+ int is_localhost, bool is_archived);
1313
1314
#endif /* NETDATA_RRD_INTERNALS */
1315
@@ -1317,6 +1327,7 @@ extern int get_tier_grouping(int tier);
1327
#include "database/engine/rrdengineapi.h"
1328
#endif
1329
#include "sqlite/sqlite_functions.h"
1330
+#include "sqlite/sqlite_context.h"
1331
#include "sqlite/sqlite_aclk.h"
1332
#include "sqlite/sqlite_aclk_chart.h"
1333
#include "sqlite/sqlite_aclk_alert.h"
database/rrdcontext.c
new
+2830
@@ -0,0 +1,2830 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "rrdcontext.h"
4
+#include "sqlite/sqlite_context.h"
5
+#include "aclk/schema-wrappers/context.h"
6
+#include "aclk/aclk_contexts_api.h"
7
+#include "aclk/aclk_api.h"
8
+
9
+int rrdcontext_enabled = CONFIG_BOOLEAN_NO;
10
+
11
+#define MESSAGES_PER_BUNDLE_TO_SEND_TO_HUB_PER_HOST 5000
12
+#define FULL_RETENTION_SCAN_DELAY_AFTER_DB_ROTATION_SECS 120
13
+#define RRDCONTEXT_WORKER_THREAD_HEARTBEAT_SECS 1
14
+#define RRDCONTEXT_MINIMUM_ALLOWED_PRIORITY 10
15
+
16
+// #define LOG_TRANSITIONS 1
17
+// #define LOG_RRDINSTANCES 1
18
+
19
+typedef enum {
20
+ RRD_FLAG_NONE = 0,
21
+ RRD_FLAG_DELETED = (1 << 0), // this is a deleted object (metrics, instances, contexts)
22
+ RRD_FLAG_COLLECTED = (1 << 1), // this object is currently being collected
23
+ RRD_FLAG_UPDATED = (1 << 2), // this object has updates to propagate
24
+ RRD_FLAG_ARCHIVED = (1 << 3), // this object is not currently being collected
25
+ RRD_FLAG_OWN_LABELS = (1 << 4), // this instance has its own labels - not linked to an RRDSET
26
+ RRD_FLAG_LIVE_RETENTION = (1 << 5), // we have got live retention from the database
27
+ RRD_FLAG_QUEUED = (1 << 6), // this context is currently queued to be dispatched to hub
28
+ RRD_FLAG_DONT_PROCESS = (1 << 7), // don't process updates for this object
29
+
30
+ RRD_FLAG_UPDATE_REASON_LOAD_SQL = (1 << 10), // this object has just been loaded from SQL
31
+ RRD_FLAG_UPDATE_REASON_NEW_OBJECT = (1 << 11), // this object has just been created
32
+ RRD_FLAG_UPDATE_REASON_UPDATED_OBJECT = (1 << 12), // we received an update on this object
33
+ RRD_FLAG_UPDATE_REASON_CHANGED_LINKING = (1 << 13), // an instance or a metric switched RRDSET or RRDDIM
34
+ RRD_FLAG_UPDATE_REASON_CHANGED_UUID = (1 << 14), // an instance or a metric changed UUID
35
+ RRD_FLAG_UPDATE_REASON_CHANGED_NAME = (1 << 15), // an instance or a metric changed name
36
+ RRD_FLAG_UPDATE_REASON_CHANGED_UNITS = (1 << 16), // this context or instance changed units
37
+ RRD_FLAG_UPDATE_REASON_CHANGED_TITLE = (1 << 17), // this context or instance changed title
38
+ RRD_FLAG_UPDATE_REASON_CHANGED_FAMILY = (1 << 18), // the context or the instance changed family
39
+ RRD_FLAG_UPDATE_REASON_CHANGED_CHART_TYPE = (1 << 19), // this context or instance changed chart type
40
+ RRD_FLAG_UPDATE_REASON_CHANGED_PRIORITY = (1 << 20), // this context or instance changed its priority
41
+ RRD_FLAG_UPDATE_REASON_CHANGED_UPDATE_EVERY = (1 << 21), // the instance or the metric changed update frequency
42
+ RRD_FLAG_UPDATE_REASON_ZERO_RETENTION = (1 << 22), // this object has not retention
43
+ RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T = (1 << 23), // this object changed its oldest time in the db
44
+ RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T = (1 << 24), // this object change its latest time in the db
45
+ RRD_FLAG_UPDATE_REASON_STOPPED_BEING_COLLECTED = (1 << 25), // this object has stopped being collected
46
+ RRD_FLAG_UPDATE_REASON_STARTED_BEING_COLLECTED = (1 << 26), // this object has started being collected
47
+ RRD_FLAG_UPDATE_REASON_DISCONNECTED_CHILD = (1 << 27), // this context belongs to a host that just disconnected
48
+ RRD_FLAG_UPDATE_REASON_DB_ROTATION = (1 << 28), // this context changed because of a db rotation
49
+ RRD_FLAG_UPDATE_REASON_UNUSED = (1 << 29), // this context is not used anymore
50
+} RRD_FLAGS;
51
+
52
+#define RRD_FLAG_ALL_UPDATE_REASONS ( \
53
+ RRD_FLAG_UPDATE_REASON_LOAD_SQL \
54
+ |RRD_FLAG_UPDATE_REASON_NEW_OBJECT \
55
+ |RRD_FLAG_UPDATE_REASON_UPDATED_OBJECT \
56
+ |RRD_FLAG_UPDATE_REASON_CHANGED_LINKING \
57
+ |RRD_FLAG_UPDATE_REASON_CHANGED_UUID \
58
+ |RRD_FLAG_UPDATE_REASON_CHANGED_NAME \
59
+ |RRD_FLAG_UPDATE_REASON_CHANGED_UNITS \
60
+ |RRD_FLAG_UPDATE_REASON_CHANGED_TITLE \
61
+ |RRD_FLAG_UPDATE_REASON_CHANGED_FAMILY \
62
+ |RRD_FLAG_UPDATE_REASON_CHANGED_CHART_TYPE \
63
+ |RRD_FLAG_UPDATE_REASON_CHANGED_PRIORITY \
64
+ |RRD_FLAG_UPDATE_REASON_CHANGED_UPDATE_EVERY \
65
+ |RRD_FLAG_UPDATE_REASON_ZERO_RETENTION \
66
+ |RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T \
67
+ |RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T \
68
+ |RRD_FLAG_UPDATE_REASON_STOPPED_BEING_COLLECTED \
69
+ |RRD_FLAG_UPDATE_REASON_STARTED_BEING_COLLECTED \
70
+ |RRD_FLAG_UPDATE_REASON_DISCONNECTED_CHILD \
71
+ |RRD_FLAG_UPDATE_REASON_DB_ROTATION \
72
+ |RRD_FLAG_UPDATE_REASON_UNUSED \
73
+)
74
+
75
+#define RRD_FLAGS_ALLOWED_EXTERNALLY_ON_NEW_OBJECTS ( \
76
+ RRD_FLAG_ARCHIVED \
77
+ |RRD_FLAG_DONT_PROCESS \
78
+ |RRD_FLAG_ALL_UPDATE_REASONS \
79
+ )
80
+
81
+#define RRD_FLAGS_PREVENTING_DELETIONS ( \
82
+ RRD_FLAG_QUEUED \
83
+ |RRD_FLAG_COLLECTED \
84
+ |RRD_FLAG_UPDATE_REASON_LOAD_SQL \
85
+ |RRD_FLAG_UPDATE_REASON_NEW_OBJECT \
86
+ |RRD_FLAG_UPDATE_REASON_UPDATED_OBJECT \
87
+ |RRD_FLAG_UPDATE_REASON_CHANGED_LINKING \
88
+)
89
+
90
+#define rrd_flag_set_updated(obj, reason) (obj)->flags |= (RRD_FLAG_UPDATED | (reason))
91
+#define rrd_flag_unset_updated(obj) (obj)->flags &= ~(RRD_FLAG_UPDATED | RRD_FLAG_ALL_UPDATE_REASONS)
92
+
93
+#define rrd_flag_set_collected(obj) do { \
94
+ if(likely( !((obj)->flags & RRD_FLAG_COLLECTED))) \
95
+ (obj)->flags |= (RRD_FLAG_COLLECTED | RRD_FLAG_UPDATE_REASON_STARTED_BEING_COLLECTED | RRD_FLAG_UPDATED); \
96
+ if(likely( ((obj)->flags & (RRD_FLAG_ARCHIVED | RRD_FLAG_UPDATE_REASON_STOPPED_BEING_COLLECTED)))) \
97
+ (obj)->flags &= ~(RRD_FLAG_ARCHIVED | RRD_FLAG_UPDATE_REASON_STOPPED_BEING_COLLECTED); \
98
+ if(unlikely(((obj)->flags & (RRD_FLAG_DELETED | RRD_FLAG_UPDATE_REASON_ZERO_RETENTION)))) \
99
+ (obj)->flags &= ~(RRD_FLAG_DELETED | RRD_FLAG_UPDATE_REASON_ZERO_RETENTION); \
100
+ if(unlikely(((obj)->flags & RRD_FLAG_DONT_PROCESS))) \
101
+ (obj)->flags &= ~RRD_FLAG_DONT_PROCESS; \
102
+} while(0)
103
+
104
+#define rrd_flag_set_archived(obj) do { \
105
+ if(likely( !((obj)->flags & RRD_FLAG_ARCHIVED))) \
106
+ (obj)->flags |= (RRD_FLAG_ARCHIVED | RRD_FLAG_UPDATE_REASON_STOPPED_BEING_COLLECTED | RRD_FLAG_UPDATED); \
107
+ if(likely( ((obj)->flags & (RRD_FLAG_COLLECTED | RRD_FLAG_UPDATE_REASON_STARTED_BEING_COLLECTED)))) \
108
+ (obj)->flags &= ~(RRD_FLAG_COLLECTED | RRD_FLAG_UPDATE_REASON_STARTED_BEING_COLLECTED); \
109
+ if(unlikely(((obj)->flags & (RRD_FLAG_DELETED | RRD_FLAG_UPDATE_REASON_ZERO_RETENTION)))) \
110
+ (obj)->flags &= ~(RRD_FLAG_DELETED | RRD_FLAG_UPDATE_REASON_ZERO_RETENTION); \
111
+} while(0)
112
+
113
+#define rrd_flag_set_deleted(obj, reason) do { \
114
+ if(likely( !((obj)->flags & RRD_FLAG_DELETED))) \
115
+ (obj)->flags |= (RRD_FLAG_DELETED | RRD_FLAG_UPDATE_REASON_ZERO_RETENTION | RRD_FLAG_UPDATED | (reason)); \
116
+ if(unlikely(((obj)->flags & RRD_FLAG_ARCHIVED))) \
117
+ (obj)->flags &= ~RRD_FLAG_ARCHIVED; \
118
+ if(likely( ((obj)->flags & RRD_FLAG_COLLECTED))) \
119
+ (obj)->flags &= ~RRD_FLAG_COLLECTED; \
120
+} while(0)
121
+
122
+
123
+#define rrd_flag_is_collected(obj) ((obj)->flags & RRD_FLAG_COLLECTED)
124
+#define rrd_flag_is_archived(obj) ((obj)->flags & RRD_FLAG_ARCHIVED)
125
+
126
+static struct rrdcontext_reason {
127
+ RRD_FLAGS flag;
128
+ const char *name;
129
+ usec_t delay_ut;
130
+} rrdcontext_reasons[] = {
131
+ // context related
132
+ { RRD_FLAG_UPDATE_REASON_NEW_OBJECT, "object created", 60 * USEC_PER_SEC },
133
+ { RRD_FLAG_UPDATE_REASON_UPDATED_OBJECT, "object updated", 60 * USEC_PER_SEC },
134
+ { RRD_FLAG_UPDATE_REASON_LOAD_SQL, "loaded from sql", 60 * USEC_PER_SEC },
135
+ { RRD_FLAG_UPDATE_REASON_CHANGED_TITLE, "changed title", 30 * USEC_PER_SEC },
136
+ { RRD_FLAG_UPDATE_REASON_CHANGED_UNITS, "changed units", 30 * USEC_PER_SEC },
137
+ { RRD_FLAG_UPDATE_REASON_CHANGED_FAMILY, "changed family", 30 * USEC_PER_SEC },
138
+ { RRD_FLAG_UPDATE_REASON_CHANGED_PRIORITY, "changed priority", 30 * USEC_PER_SEC },
139
+ { RRD_FLAG_UPDATE_REASON_ZERO_RETENTION, "has no retention", 60 * USEC_PER_SEC },
140
+ { RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T, "updated first_time_t", 30 * USEC_PER_SEC },
141
+ { RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T, "updated last_time_t", 60 * USEC_PER_SEC },
142
+ { RRD_FLAG_UPDATE_REASON_CHANGED_CHART_TYPE, "changed chart type", 30 * USEC_PER_SEC },
143
+ { RRD_FLAG_UPDATE_REASON_STOPPED_BEING_COLLECTED, "stopped collected", 60 * USEC_PER_SEC },
144
+ { RRD_FLAG_UPDATE_REASON_STARTED_BEING_COLLECTED, "started collected", 0 * USEC_PER_SEC },
145
+ { RRD_FLAG_UPDATE_REASON_UNUSED, "unused", 0 * USEC_PER_SEC },
146
+
147
+ // not context related
148
+ { RRD_FLAG_UPDATE_REASON_CHANGED_UUID, "changed uuid", 60 * USEC_PER_SEC },
149
+ { RRD_FLAG_UPDATE_REASON_CHANGED_UPDATE_EVERY, "changed updated every",60 * USEC_PER_SEC },
150
+ { RRD_FLAG_UPDATE_REASON_CHANGED_LINKING, "changed rrd link", 60 * USEC_PER_SEC },
151
+ { RRD_FLAG_UPDATE_REASON_CHANGED_NAME, "changed name", 60 * USEC_PER_SEC },
152
+ { RRD_FLAG_UPDATE_REASON_DISCONNECTED_CHILD, "child disconnected", 30 * USEC_PER_SEC },
153
+ { RRD_FLAG_UPDATE_REASON_DB_ROTATION, "db rotation", 60 * USEC_PER_SEC },
154
+
155
+ // terminator
156
+ { 0, NULL, 0 },
157
+};
158
+
159
+
160
+typedef struct rrdmetric {
161
+ uuid_t uuid;
162
+
163
+ STRING *id;
164
+ STRING *name;
165
+
166
+ RRDDIM *rrddim;
167
+
168
+ time_t first_time_t;
169
+ time_t last_time_t;
170
+ RRD_FLAGS flags;
171
+
172
+ struct rrdinstance *ri;
173
+
174
+ usec_t created_ut; // the time this object was created
175
+} RRDMETRIC;
176
+
177
+typedef struct rrdinstance {
178
+ uuid_t uuid;
179
+
180
+ STRING *id;
181
+ STRING *name;
182
+ STRING *title;
183
+ STRING *units;
184
+ STRING *family;
185
+ uint32_t priority;
186
+ RRDSET_TYPE chart_type;
187
+
188
+ RRD_FLAGS flags; // flags related to this instance
189
+ time_t first_time_t;
190
+ time_t last_time_t;
191
+
192
+ int update_every; // data collection frequency
193
+ RRDSET *rrdset; // pointer to RRDSET when collected, or NULL
194
+
195
+ DICTIONARY *rrdlabels; // linked to RRDSET->state->chart_labels or own version
196
+
197
+ struct rrdcontext *rc;
198
+ DICTIONARY *rrdmetrics;
199
+} RRDINSTANCE;
200
+
201
+typedef struct rrdcontext {
202
+ uint64_t version;
203
+
204
+ STRING *id;
205
+ STRING *title;
206
+ STRING *units;
207
+ STRING *family;
208
+ uint32_t priority;
209
+ RRDSET_TYPE chart_type;
210
+
211
+ RRD_FLAGS flags;
212
+ time_t first_time_t;
213
+ time_t last_time_t;
214
+
215
+ VERSIONED_CONTEXT_DATA hub;
216
+
217
+ DICTIONARY *rrdinstances;
218
+ RRDHOST *rrdhost;
219
+
220
+ struct {
221
+ RRD_FLAGS queued_flags; // the last flags that triggered the queueing
222
+ usec_t queued_ut; // the last time this was queued
223
+ usec_t delay_calc_ut; // the last time we calculated the scheduled_dispatched_ut
224
+ usec_t scheduled_dispatch_ut; // the time it was/is scheduled to be sent
225
+ usec_t dequeued_ut; // the last time we sent (or deduped) this context
226
+ } queue;
227
+
228
+ netdata_mutex_t mutex;
229
+} RRDCONTEXT;
230
+
231
+// ----------------------------------------------------------------------------
232
+// helper one-liners for RRDMETRIC
233
+
234
+static inline RRDMETRIC *rrdmetric_acquired_value(RRDMETRIC_ACQUIRED *rma) {
235
+ return dictionary_acquired_item_value((DICTIONARY_ITEM *)rma);
236
+}
237
+
238
+static inline void rrdmetric_release(RRDMETRIC_ACQUIRED *rma) {
239
+ RRDMETRIC *rm = rrdmetric_acquired_value(rma);
240
+ dictionary_acquired_item_release(rm->ri->rrdmetrics, (DICTIONARY_ITEM *)rma);
241
+}
242
+
243
+// ----------------------------------------------------------------------------
244
+// helper one-liners for RRDINSTANCE
245
+
246
+static inline RRDINSTANCE_ACQUIRED *rrdinstance_dup(RRDINSTANCE_ACQUIRED *ria) {
247
+ return (RRDINSTANCE_ACQUIRED *)dictionary_acquired_item_dup((DICTIONARY_ITEM *)ria);
248
+}
249
+
250
+static inline RRDINSTANCE *rrdinstance_acquired_value(RRDINSTANCE_ACQUIRED *ria) {
251
+ return dictionary_acquired_item_value((DICTIONARY_ITEM *)ria);
252
+}
253
+
254
+static inline const char *rrdinstance_acquired_name(RRDINSTANCE_ACQUIRED *ria) {
255
+ return dictionary_acquired_item_name((DICTIONARY_ITEM *)ria);
256
+}
257
+
258
+static inline void rrdinstance_release(RRDINSTANCE_ACQUIRED *ria) {
259
+ RRDINSTANCE *ri = rrdinstance_acquired_value(ria);
260
+ dictionary_acquired_item_release(ri->rc->rrdinstances, (DICTIONARY_ITEM *)ria);
261
+}
262
+
263
+// ----------------------------------------------------------------------------
264
+// helper one-liners for RRDCONTEXT
265
+
266
+static inline RRDCONTEXT_ACQUIRED *rrdcontext_dup(RRDCONTEXT_ACQUIRED *rca) {
267
+ return (RRDCONTEXT_ACQUIRED *)dictionary_acquired_item_dup((DICTIONARY_ITEM *)rca);
268
+}
269
+
270
+static inline const char *rrdcontext_acquired_name(RRDCONTEXT_ACQUIRED *rca) {
271
+ return dictionary_acquired_item_name((DICTIONARY_ITEM *)rca);
272
+}
273
+
274
+static inline RRDCONTEXT *rrdcontext_acquired_value(RRDCONTEXT_ACQUIRED *rca) {
275
+ return dictionary_acquired_item_value((DICTIONARY_ITEM *)rca);
276
+}
277
+
278
+static inline RRDCONTEXT_ACQUIRED *rrdcontext_acquire(RRDHOST *host, const char *name) {
279
+ return (RRDCONTEXT_ACQUIRED *)dictionary_get_and_acquire_item((DICTIONARY *)host->rrdctx, name);
280
+}
281
+
282
+static inline void rrdcontext_release(RRDCONTEXT_ACQUIRED *rca) {
283
+ RRDCONTEXT *rc = rrdcontext_acquired_value(rca);
284
+ dictionary_acquired_item_release((DICTIONARY *)rc->rrdhost->rrdctx, (DICTIONARY_ITEM *)rca);
285
+}
286
+
287
+static void rrdcontext_recalculate_context_retention(RRDCONTEXT *rc, RRD_FLAGS reason, int job_id);
288
+static void rrdcontext_recalculate_host_retention(RRDHOST *host, RRD_FLAGS reason, int job_id);
289
+
290
+#define rrdcontext_version_hash(host) rrdcontext_version_hash_with_callback(host, NULL, false, NULL)
291
+static uint64_t rrdcontext_version_hash_with_callback(RRDHOST *host, void (*callback)(RRDCONTEXT *, bool, void *), bool snapshot, void *bundle);
292
+
293
+void rrdcontext_delete_from_sql_unsafe(RRDCONTEXT *rc);
294
+
295
+#define rrdcontext_lock(rc) netdata_mutex_lock(&((rc)->mutex))
296
+#define rrdcontext_unlock(rc) netdata_mutex_unlock(&((rc)->mutex))
297
+
298
+// ----------------------------------------------------------------------------
299
+// Updates triggers
300
+
301
+static void rrdmetric_trigger_updates(RRDMETRIC *rm, bool force, bool escalate);
302
+static void rrdinstance_trigger_updates(RRDINSTANCE *ri, bool force, bool escalate);
303
+static void rrdcontext_trigger_updates(RRDCONTEXT *rc, bool force);
304
+
305
+// ----------------------------------------------------------------------------
306
+// visualizing flags
307
+
308
+static void rrd_flags_to_buffer(RRD_FLAGS flags, BUFFER *wb) {
309
+ if(flags & RRD_FLAG_QUEUED)
310
+ buffer_strcat(wb, "QUEUED ");
311
+
312
+ if(flags & RRD_FLAG_DELETED)
313
+ buffer_strcat(wb, "DELETED ");
314
+
315
+ if(flags & RRD_FLAG_COLLECTED)
316
+ buffer_strcat(wb, "COLLECTED ");
317
+
318
+ if(flags & RRD_FLAG_UPDATED)
319
+ buffer_strcat(wb, "UPDATED ");
320
+
321
+ if(flags & RRD_FLAG_ARCHIVED)
322
+ buffer_strcat(wb, "ARCHIVED ");
323
+
324
+ if(flags & RRD_FLAG_OWN_LABELS)
325
+ buffer_strcat(wb, "OWN_LABELS ");
326
+
327
+ if(flags & RRD_FLAG_LIVE_RETENTION)
328
+ buffer_strcat(wb, "LIVE_RETENTION ");
329
+
330
+ if(flags & RRD_FLAG_DONT_PROCESS)
331
+ buffer_strcat(wb, "DONT_PROCESS ");
332
+}
333
+
334
+static void rrd_reasons_to_buffer(RRD_FLAGS flags, BUFFER *wb) {
335
+ for(int i = 0, added = 0; rrdcontext_reasons[i].name ; i++) {
336
+ if (flags & rrdcontext_reasons[i].flag) {
337
+ if (added)
338
+ buffer_strcat(wb, ", ");
339
+ buffer_strcat(wb, rrdcontext_reasons[i].name);
340
+ added++;
341
+ }
342
+ }
343
+}
344
+
345
+// ----------------------------------------------------------------------------
346
+// logging of all data collected
347
+
348
+#ifdef LOG_TRANSITIONS
349
+static void log_transition(STRING *metric, STRING *instance, STRING *context, RRD_FLAGS flags, const char *msg) {
350
+ BUFFER *wb = buffer_create(1000);
351
+
352
+ buffer_sprintf(wb, "RRD TRANSITION: context '%s'", string2str(context));
353
+
354
+ if(instance)
355
+ buffer_sprintf(wb, ", instance '%s'", string2str(instance));
356
+
357
+ if(metric)
358
+ buffer_sprintf(wb, ", metric '%s'", string2str(metric));
359
+
360
+ buffer_sprintf(wb, ", triggered by %s: ", msg);
361
+
362
+ rrd_flags_to_buffer(flags, wb);
363
+
364
+ buffer_strcat(wb, ", reasons: ");
365
+
366
+ rrd_reasons_to_buffer(flags, wb);
367
+
368
+ internal_error(true, "%s", buffer_tostring(wb));
369
+ buffer_free(wb);
370
+}
371
+#else
372
+#define log_transition(metric, instance, context, flags, msg) debug_dummy()
373
+#endif
374
+
375
+#ifdef LOG_RRDINSTANCES
376
+static void rrdinstance_log(RRDINSTANCE *ri, const char *msg) {
377
+ char uuid[UUID_STR_LEN];
378
+
379
+ uuid_unparse(ri->uuid, uuid);
380
+
381
+ BUFFER *wb = buffer_create(1000);
382
+
383
+ buffer_sprintf(wb,
384
+ "RRDINSTANCE: %s id '%s' (host '%s'), uuid '%s', name '%s', context '%s', title '%s', units '%s', family '%s', priority %zu, chart type '%s', update every %d, rrdset '%s', flags %s%s%s%s%s%s%s%s, first_time_t %ld, last_time_t %ld",
385
+ msg,
386
+ string2str(ri->id),
387
+ ri->rc->rrdhost->hostname,
388
+ uuid,
389
+ string2str(ri->name),
390
+ string2str(ri->rc->id),
391
+ string2str(ri->title),
392
+ string2str(ri->units),
393
+ string2str(ri->family),
394
+ ri->priority,
395
+ rrdset_type_name(ri->chart_type),
396
+ ri->update_every,
397
+ ri->rrdset?ri->rrdset->id:"NONE",
398
+ ri->flags & RRD_FLAG_DELETED ?"DELETED ":"",
399
+ ri->flags & RRD_FLAG_UPDATED ?"UPDATED ":"",
400
+ rrd_flag_is_collected(ri) ?"COLLECTED ":"",
401
+ rrd_flag_is_archived(ri) ?"ARCHIVED ":"",
402
+ ri->flags & RRD_FLAG_OWNLABELS ?"OWNLABELS ":"",
403
+ ri->flags & RRD_FLAG_LIVE_RETENTION ?"LIVE ":"",
404
+ ri->flags & RRD_FLAG_QUEUED ?"QUEUED ":"",
405
+ ri->flags & RRD_FLAG_DONT_TRIGGER ?"BLOCKED ":"",
406
+ ri->first_time_t,
407
+ ri->last_time_t
408
+ );
409
+
410
+ buffer_strcat(wb, ", update reasons: { ");
411
+ for(int i = 0, added = 0; rrdcontext_reasons[i].name ;i++)
412
+ if(ri->flags & rrdcontext_reasons[i].flag) {
413
+ if(added) buffer_strcat(wb, ", ");
414
+ buffer_strcat(wb, rrdcontext_reasons[i].name);
415
+ added++;
416
+ }
417
+ buffer_strcat(wb, " }");
418
+
419
+ buffer_strcat(wb, ", labels: { ");
420
+ if(ri->rrdlabels) {
421
+ if(!rrdlabels_to_buffer(ri->rrdlabels, wb, "", "=", "'", ", ", NULL, NULL, NULL, NULL))
422
+ buffer_strcat(wb, "EMPTY }");
423
+ else
424
+ buffer_strcat(wb, " }");
425
+ }
426
+ else
427
+ buffer_strcat(wb, "NONE }");
428
+
429
+ buffer_strcat(wb, ", metrics: { ");
430
+ if(ri->rrdmetrics) {
431
+ RRDMETRIC *v;
432
+ int i = 0;
433
+ dfe_start_read((DICTIONARY *)ri->rrdmetrics, v) {
434
+ buffer_sprintf(wb, "%s%s", i?",":"", v_name);
435
+ i++;
436
+ }
437
+ dfe_done(v);
438
+
439
+ if(!i)
440
+ buffer_strcat(wb, "EMPTY }");
441
+ else
442
+ buffer_strcat(wb, " }");
443
+ }
444
+ else
445
+ buffer_strcat(wb, "NONE }");
446
+
447
+ internal_error(true, "%s", buffer_tostring(wb));
448
+ buffer_free(wb);
449
+}
450
+#else
451
+#define rrdinstance_log(ir, msg) debug_dummy()
452
+#endif
453
+
454
+// ----------------------------------------------------------------------------
455
+// RRDMETRIC
456
+
457
+static void rrdmetric_free(RRDMETRIC *rm) {
458
+ string_freez(rm->id);
459
+ string_freez(rm->name);
460
+
461
+ rm->id = NULL;
462
+ rm->name = NULL;
463
+ rm->ri = NULL;
464
+}
465
+
466
+static void rrdmetric_update_retention(RRDMETRIC *rm) {
467
+ time_t min_first_time_t = LONG_MAX, max_last_time_t = 0;
468
+
469
+ if(rm->rrddim) {
470
+ min_first_time_t = rrddim_first_entry_t(rm->rrddim);
471
+ max_last_time_t = rrddim_last_entry_t(rm->rrddim);
472
+ }
473
+#ifdef ENABLE_DBENGINE
474
+ else {
475
+ RRDHOST *rrdhost = rm->ri->rc->rrdhost;
476
+ for (int tier = 0; tier < storage_tiers; tier++) {
477
+ if(!rrdhost->storage_instance[tier]) continue;
478
+
479
+ time_t first_time_t, last_time_t;
480
+ if (rrdeng_metric_retention_by_uuid(rrdhost->storage_instance[tier], &rm->uuid, &first_time_t, &last_time_t) == 0) {
481
+ if (first_time_t < min_first_time_t)
482
+ min_first_time_t = first_time_t;
483
+
484
+ if (last_time_t > max_last_time_t)
485
+ max_last_time_t = last_time_t;
486
+ }
487
+ }
488
+ }
489
+#endif
490
+
491
+ if(min_first_time_t == LONG_MAX)
492
+ min_first_time_t = 0;
493
+
494
+ if(min_first_time_t > max_last_time_t) {
495
+ internal_error(true, "RRDMETRIC: retention of '%s' is flipped", string2str(rm->id));
496
+ time_t tmp = min_first_time_t;
497
+ min_first_time_t = max_last_time_t;
498
+ max_last_time_t = tmp;
499
+ }
500
+
501
+ // check if retention changed
502
+
503
+ if (min_first_time_t != rm->first_time_t) {
504
+ rm->first_time_t = min_first_time_t;
505
+ rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
506
+ }
507
+
508
+ if (max_last_time_t != rm->last_time_t) {
509
+ rm->last_time_t = max_last_time_t;
510
+ rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
511
+ }
512
+
513
+ if(unlikely(!rm->first_time_t && !rm->last_time_t))
514
+ rrd_flag_set_deleted(rm, RRD_FLAG_UPDATE_REASON_ZERO_RETENTION);
515
+
516
+ rm->flags |= RRD_FLAG_LIVE_RETENTION;
517
+}
518
+
519
+// called when this rrdmetric is inserted to the rrdmetrics dictionary of a rrdinstance
520
+static void rrdmetric_insert_callback(const char *id __maybe_unused, void *value, void *data) {
521
+ RRDMETRIC *rm = value;
522
+
523
+ // link it to its parent
524
+ rm->ri = data;
525
+
526
+ // remove flags that we need to figure out at runtime
527
+ rm->flags = rm->flags & RRD_FLAGS_ALLOWED_EXTERNALLY_ON_NEW_OBJECTS;
528
+
529
+ rm->created_ut = now_realtime_usec();
530
+
531
+ // signal the react callback to do the job
532
+ rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_NEW_OBJECT);
533
+}
534
+
535
+// called when this rrdmetric is deleted from the rrdmetrics dictionary of a rrdinstance
536
+static void rrdmetric_delete_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
537
+ RRDMETRIC *rm = value;
538
+
539
+ internal_error(rm->rrddim, "RRDMETRIC: '%s' is freed but there is a RRDDIM linked to it.", string2str(rm->id));
540
+
541
+ // free the resources
542
+ rrdmetric_free(rm);
543
+}
544
+
545
+// called when the same rrdmetric is inserted again to the rrdmetrics dictionary of a rrdinstance
546
+static void rrdmetric_conflict_callback(const char *id __maybe_unused, void *oldv, void *newv, void *data __maybe_unused) {
547
+ RRDMETRIC *rm = oldv;
548
+ RRDMETRIC *rm_new = newv;
549
+
550
+ internal_error(rm->id != rm_new->id,
551
+ "RRDMETRIC: '%s' cannot change id to '%s'",
552
+ string2str(rm->id), string2str(rm_new->id));
553
+
554
+ if(uuid_compare(rm->uuid, rm_new->uuid) != 0) {
555
+ char uuid1[UUID_STR_LEN], uuid2[UUID_STR_LEN];
556
+ uuid_unparse(rm->uuid, uuid1);
557
+ uuid_unparse(rm_new->uuid, uuid2);
558
+ internal_error(true, "RRDMETRIC: '%s' of instance '%s' changed uuid from '%s' to '%s'", string2str(rm->id), string2str(rm->ri->id), uuid1, uuid2);
559
+ uuid_copy(rm->uuid, rm_new->uuid);
560
+ rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_UUID);
561
+ }
562
+
563
+ if(rm->rrddim && rm_new->rrddim && rm->rrddim != rm_new->rrddim) {
564
+ rm->rrddim = rm_new->rrddim;
565
+ rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_LINKING);
566
+ }
567
+
568
+ if(rm->rrddim && uuid_compare(rm->uuid, rm->rrddim->metric_uuid) != 0) {
569
+ char uuid1[UUID_STR_LEN], uuid2[UUID_STR_LEN];
570
+ uuid_unparse(rm->uuid, uuid1);
571
+ uuid_unparse(rm_new->uuid, uuid2);
572
+ internal_error(true, "RRDMETRIC: '%s' is linked to RRDDIM '%s' but they have different UUIDs. RRDMETRIC has '%s', RRDDIM has '%s'", string2str(rm->id), rm->rrddim->id, uuid1, uuid2);
573
+ }
574
+
575
+ if(rm->rrddim != rm_new->rrddim)
576
+ rm->rrddim = rm_new->rrddim;
577
+
578
+ if(rm->name != rm_new->name) {
579
+ STRING *old = rm->name;
580
+ rm->name = string_dup(rm_new->name);
581
+ string_freez(old);
582
+ rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_NAME);
583
+ }
584
+
585
+ if(!rm->first_time_t || (rm_new->first_time_t && rm_new->first_time_t < rm->first_time_t)) {
586
+ rm->first_time_t = rm_new->first_time_t;
587
+ rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
588
+ }
589
+
590
+ if(!rm->last_time_t || (rm_new->last_time_t && rm_new->last_time_t > rm->last_time_t)) {
591
+ rm->last_time_t = rm_new->last_time_t;
592
+ rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
593
+ }
594
+
595
+ rm->flags |= (rm_new->flags & RRD_FLAGS_ALLOWED_EXTERNALLY_ON_NEW_OBJECTS);
596
+
597
+ if(rrd_flag_is_collected(rm) && rrd_flag_is_archived(rm))
598
+ rrd_flag_set_collected(rm);
599
+
600
+ if(rm->flags & RRD_FLAG_UPDATED)
601
+ rm->flags |= RRD_FLAG_UPDATE_REASON_UPDATED_OBJECT;
602
+
603
+ rrdmetric_free(rm_new);
604
+
605
+ // the react callback will continue from here
606
+}
607
+
608
+static void rrdmetric_react_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
609
+ RRDMETRIC *rm = value;
610
+
611
+ rrdmetric_trigger_updates(rm, false, true);
612
+}
613
+
614
+static void rrdmetrics_create(RRDINSTANCE *ri) {
615
+ if(unlikely(!ri)) return;
616
+ if(likely(ri->rrdmetrics)) return;
617
+
618
+ ri->rrdmetrics = dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
619
+ dictionary_register_insert_callback(ri->rrdmetrics, rrdmetric_insert_callback, (void *)ri);
620
+ dictionary_register_delete_callback(ri->rrdmetrics, rrdmetric_delete_callback, (void *)ri);
621
+ dictionary_register_conflict_callback(ri->rrdmetrics, rrdmetric_conflict_callback, (void *)ri);
622
+ dictionary_register_react_callback(ri->rrdmetrics, rrdmetric_react_callback, (void *)ri);
623
+}
624
+
625
+static void rrdmetrics_destroy(RRDINSTANCE *ri) {
626
+ if(unlikely(!ri || !ri->rrdmetrics)) return;
627
+ dictionary_destroy(ri->rrdmetrics);
628
+ ri->rrdmetrics = NULL;
629
+}
630
+
631
+static inline bool rrdmetric_should_be_deleted(RRDMETRIC *rm) {
632
+ if(likely(!(rm->flags & RRD_FLAG_DELETED)))
633
+ return false;
634
+
635
+ if(likely(!(rm->flags & RRD_FLAG_LIVE_RETENTION)))
636
+ return false;
637
+
638
+ if(unlikely(rm->flags & RRD_FLAGS_PREVENTING_DELETIONS))
639
+ return false;
640
+
641
+ if(likely(rm->rrddim))
642
+ return false;
643
+
644
+ if((now_realtime_usec() - rm->created_ut) < 600 * USEC_PER_SEC)
645
+ return false;
646
+
647
+ rrdmetric_update_retention(rm);
648
+ if(rm->first_time_t || rm->last_time_t)
649
+ return false;
650
+
651
+ return true;
652
+}
653
+
654
+static void rrdmetric_trigger_updates(RRDMETRIC *rm, bool force, bool escalate) {
655
+ if(likely(!force && !(rm->flags & RRD_FLAG_UPDATED))) return;
656
+
657
+ if(unlikely(rrd_flag_is_collected(rm) && !rm->rrddim))
658
+ rrd_flag_set_archived(rm);
659
+
660
+ if(unlikely((rm->flags & RRD_FLAG_UPDATE_REASON_DISCONNECTED_CHILD) && rrd_flag_is_collected(rm)))
661
+ rrd_flag_set_archived(rm);
662
+
663
+ rrdmetric_update_retention(rm);
664
+
665
+ if(unlikely(escalate && rm->flags & RRD_FLAG_UPDATED && !(rm->ri->flags & RRD_FLAG_DONT_PROCESS))) {
666
+ log_transition(rm->id, rm->ri->id, rm->ri->rc->id, rm->flags, "RRDMETRIC");
667
+ rrdinstance_trigger_updates(rm->ri, true, true);
668
+ }
669
+}
670
+
671
+static inline void rrdmetric_from_rrddim(RRDDIM *rd) {
672
+ if(unlikely(!rd->rrdset))
673
+ fatal("RRDMETRIC: rrddim '%s' does not have a rrdset.", rd->id);
674
+
675
+ if(unlikely(!rd->rrdset->rrdhost))
676
+ fatal("RRDMETRIC: rrdset '%s' does not have a rrdhost", rd->rrdset->id);
677
+
678
+ if(unlikely(!rd->rrdset->rrdinstance))
679
+ fatal("RRDMETRIC: rrdset '%s' does not have a rrdinstance", rd->rrdset->id);
680
+
681
+ RRDINSTANCE *ri = rrdinstance_acquired_value(rd->rrdset->rrdinstance);
682
+
683
+ RRDMETRIC trm = {
684
+ .id = string_strdupz(rd->id),
685
+ .name = string_strdupz(rd->name),
686
+ .flags = RRD_FLAG_NONE,
687
+ .rrddim = rd,
688
+ };
689
+ uuid_copy(trm.uuid, rd->metric_uuid);
690
+
691
+ RRDMETRIC_ACQUIRED *rma = (RRDMETRIC_ACQUIRED *)dictionary_set_and_acquire_item(ri->rrdmetrics, string2str(trm.id), &trm, sizeof(trm));
692
+
693
+ if(rd->rrdmetric)
694
+ rrdmetric_release(rd->rrdmetric);
695
+
696
+ rd->rrdmetric = rma;
697
+}
698
+
699
+#define rrddim_get_rrdmetric(rd) rrddim_get_rrdmetric_with_trace(rd, __FUNCTION__)
700
+static inline RRDMETRIC *rrddim_get_rrdmetric_with_trace(RRDDIM *rd, const char *function) {
701
+ if(unlikely(!rd->rrdmetric))
702
+ fatal("RRDMETRIC: RRDDIM '%s' is not linked to an RRDMETRIC at %s()", rd->id, function);
703
+
704
+ RRDMETRIC *rm = rrdmetric_acquired_value(rd->rrdmetric);
705
+
706
+ if(unlikely(rm->rrddim != rd))
707
+ fatal("RRDMETRIC: '%s' is not linked to RRDDIM '%s' at %s()", string2str(rm->id), rd->id, function);
708
+
709
+ return rm;
710
+}
711
+
712
+static inline void rrdmetric_rrddim_is_freed(RRDDIM *rd) {
713
+ RRDMETRIC *rm = rrddim_get_rrdmetric(rd);
714
+
715
+ if(unlikely(rrd_flag_is_collected(rm)))
716
+ rrd_flag_set_archived(rm);
717
+
718
+ rm->rrddim = NULL;
719
+ rrdmetric_trigger_updates(rm, false, true);
720
+ rrdmetric_release(rd->rrdmetric);
721
+ rd->rrdmetric = NULL;
722
+}
723
+
724
+static inline void rrdmetric_updated_rrddim_flags(RRDDIM *rd) {
725
+ RRDMETRIC *rm = rrddim_get_rrdmetric(rd);
726
+
727
+ if(unlikely(rd->flags & (RRDDIM_FLAG_ARCHIVED | RRDDIM_FLAG_OBSOLETE))) {
728
+ if(unlikely(rrd_flag_is_collected(rm)))
729
+ rrd_flag_set_archived(rm);
730
+ }
731
+
732
+ rrdmetric_trigger_updates(rm, false, true);
733
+}
734
+
735
+static inline void rrdmetric_collected_rrddim(RRDDIM *rd) {
736
+ RRDMETRIC *rm = rrddim_get_rrdmetric(rd);
737
+
738
+ if(unlikely(!rrd_flag_is_collected(rm)))
739
+ rrd_flag_set_collected(rm);
740
+
741
+ rrdmetric_trigger_updates(rm, false, true);
742
+}
743
+
744
+// ----------------------------------------------------------------------------
745
+// RRDINSTANCE
746
+
747
+static void rrdinstance_free(RRDINSTANCE *ri) {
748
+
749
+ if(ri->flags & RRD_FLAG_OWN_LABELS)
750
+ dictionary_destroy(ri->rrdlabels);
751
+
752
+ rrdmetrics_destroy(ri);
753
+ string_freez(ri->id);
754
+ string_freez(ri->name);
755
+ string_freez(ri->title);
756
+ string_freez(ri->units);
757
+ string_freez(ri->family);
758
+
759
+ ri->id = NULL;
760
+ ri->name = NULL;
761
+ ri->title = NULL;
762
+ ri->units = NULL;
763
+ ri->family = NULL;
764
+ ri->rc = NULL;
765
+ ri->rrdlabels = NULL;
766
+ ri->rrdmetrics = NULL;
767
+ ri->rrdset = NULL;
768
+}
769
+
770
+static void rrdinstance_insert_callback(const char *id __maybe_unused, void *value, void *data) {
771
+ RRDINSTANCE *ri = value;
772
+
773
+ // link it to its parent
774
+ ri->rc = data;
775
+
776
+ ri->flags = ri->flags & RRD_FLAGS_ALLOWED_EXTERNALLY_ON_NEW_OBJECTS;
777
+
778
+ if(!ri->name)
779
+ ri->name = string_dup(ri->id);
780
+
781
+ if(ri->rrdset && ri->rrdset->state) {
782
+ ri->rrdlabels = ri->rrdset->state->chart_labels;
783
+ if(ri->flags & RRD_FLAG_OWN_LABELS)
784
+ ri->flags &= ~RRD_FLAG_OWN_LABELS;
785
+ }
786
+ else {
787
+ ri->rrdlabels = rrdlabels_create();
788
+ ri->flags |= RRD_FLAG_OWN_LABELS;
789
+ }
790
+
791
+ rrdmetrics_create(ri);
792
+ rrdinstance_log(ri, "INSERT");
793
+
794
+ // signal the react callback to do the job
795
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_NEW_OBJECT);
796
+}
797
+
798
+static void rrdinstance_delete_callback(const char *id, void *value, void *data) {
799
+ (void)id;
800
+ RRDCONTEXT *rc = data; (void)rc;
801
+ RRDINSTANCE *ri = (RRDINSTANCE *)value;
802
+
803
+ rrdinstance_log(ri, "DELETE");
804
+
805
+ internal_error(ri->rrdset, "RRDINSTANCE: '%s' is freed but there is a RRDSET linked to it.", string2str(ri->id));
806
+
807
+ rrdinstance_free(ri);
808
+}
809
+
810
+static void rrdinstance_conflict_callback(const char *id __maybe_unused, void *oldv, void *newv, void *data __maybe_unused) {
811
+ RRDINSTANCE *ri = (RRDINSTANCE *)oldv;
812
+ RRDINSTANCE *ri_new = (RRDINSTANCE *)newv;
813
+
814
+ internal_error(ri->id != ri_new->id,
815
+ "RRDINSTANCE: '%s' cannot change id to '%s'",
816
+ string2str(ri->id), string2str(ri_new->id));
817
+
818
+ if(uuid_compare(ri->uuid, ri_new->uuid) != 0) {
819
+ uuid_copy(ri->uuid, ri_new->uuid);
820
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_UUID);
821
+ }
822
+
823
+ if(ri->rrdset && ri_new->rrdset && ri->rrdset != ri_new->rrdset) {
824
+ ri->rrdset = ri_new->rrdset;
825
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_LINKING);
826
+ }
827
+
828
+ if(ri->rrdset && ri->rrdset->chart_uuid && uuid_compare(ri->uuid, *ri->rrdset->chart_uuid) != 0) {
829
+ char uuid1[UUID_STR_LEN], uuid2[UUID_STR_LEN];
830
+ uuid_unparse(ri->uuid, uuid1);
831
+ uuid_unparse(*ri->rrdset->chart_uuid, uuid2);
832
+ internal_error(true, "RRDINSTANCE: '%s' is linked to RRDSET '%s' but they have different UUIDs. RRDINSTANCE has '%s', RRDSET has '%s'", string2str(ri->id), ri->rrdset->id, uuid1, uuid2);
833
+ }
834
+
835
+ if(ri->name != ri_new->name) {
836
+ STRING *old = ri->name;
837
+ ri->name = string_dup(ri_new->name);
838
+ string_freez(old);
839
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_NAME);
840
+ }
841
+
842
+ if(ri->title != ri_new->title) {
843
+ STRING *old = ri->title;
844
+ ri->title = string_dup(ri_new->title);
845
+ string_freez(old);
846
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_TITLE);
847
+ }
848
+
849
+ if(ri->units != ri_new->units) {
850
+ STRING *old = ri->units;
851
+ ri->units = string_dup(ri_new->units);
852
+ string_freez(old);
853
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_UNITS);
854
+ }
855
+
856
+ if(ri->family != ri_new->family) {
857
+ STRING *old = ri->family;
858
+ ri->family = string_dup(ri_new->family);
859
+ string_freez(old);
860
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_FAMILY);
861
+ }
862
+
863
+ if(ri->chart_type != ri_new->chart_type) {
864
+ ri->chart_type = ri_new->chart_type;
865
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_CHART_TYPE);
866
+ }
867
+
868
+ if(ri->priority != ri_new->priority) {
869
+ ri->priority = ri_new->priority;
870
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_PRIORITY);
871
+ }
872
+
873
+ if(ri->update_every != ri_new->update_every) {
874
+ ri->update_every = ri_new->update_every;
875
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_UPDATE_EVERY);
876
+ }
877
+
878
+ if(ri->rrdset != ri_new->rrdset) {
879
+ ri->rrdset = ri_new->rrdset;
880
+
881
+ if(ri->flags & RRD_FLAG_OWN_LABELS) {
882
+ DICTIONARY *old = ri->rrdlabels;
883
+ ri->rrdlabels = ri->rrdset->state->chart_labels;
884
+ ri->flags &= ~RRD_FLAG_OWN_LABELS;
885
+ rrdlabels_destroy(old);
886
+ }
887
+ }
888
+
889
+ ri->flags |= (ri_new->flags & RRD_FLAGS_ALLOWED_EXTERNALLY_ON_NEW_OBJECTS);
890
+
891
+ if(rrd_flag_is_collected(ri) && rrd_flag_is_archived(ri))
892
+ rrd_flag_set_collected(ri);
893
+
894
+ if(ri->flags & RRD_FLAG_UPDATED)
895
+ ri->flags |= RRD_FLAG_UPDATE_REASON_UPDATED_OBJECT;
896
+
897
+ rrdinstance_log(ri, "CONFLICT");
898
+
899
+ // free the new one
900
+ rrdinstance_free(ri_new);
901
+
902
+ // the react callback will continue from here
903
+}
904
+
905
+static void rrdinstance_react_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
906
+ RRDINSTANCE *ri = value;
907
+
908
+ rrdinstance_trigger_updates(ri, false, true);
909
+}
910
+
911
+void rrdinstances_create(RRDCONTEXT *rc) {
912
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
913
+ return;
914
+
915
+ if(unlikely(!rc || rc->rrdinstances)) return;
916
+
917
+ rc->rrdinstances = dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
918
+ dictionary_register_insert_callback(rc->rrdinstances, rrdinstance_insert_callback, (void *)rc);
919
+ dictionary_register_delete_callback(rc->rrdinstances, rrdinstance_delete_callback, (void *)rc);
920
+ dictionary_register_conflict_callback(rc->rrdinstances, rrdinstance_conflict_callback, (void *)rc);
921
+ dictionary_register_react_callback(rc->rrdinstances, rrdinstance_react_callback, (void *)rc);
922
+}
923
+
924
+void rrdinstances_destroy(RRDCONTEXT *rc) {
925
+ if(unlikely(!rc || !rc->rrdinstances)) return;
926
+
927
+ dictionary_destroy(rc->rrdinstances);
928
+ rc->rrdinstances = NULL;
929
+}
930
+
931
+static inline bool rrdinstance_should_be_deleted(RRDINSTANCE *ri) {
932
+ if(likely(!(ri->flags & RRD_FLAG_DELETED)))
933
+ return false;
934
+
935
+ if(likely(!(ri->flags & RRD_FLAG_LIVE_RETENTION)))
936
+ return false;
937
+
938
+ if(unlikely(ri->flags & RRD_FLAGS_PREVENTING_DELETIONS))
939
+ return false;
940
+
941
+ if(likely(ri->rrdset))
942
+ return false;
943
+
944
+ if(unlikely(dictionary_stats_referenced_items(ri->rrdmetrics) != 0))
945
+ return false;
946
+
947
+ if(unlikely(dictionary_stats_entries(ri->rrdmetrics) != 0))
948
+ return false;
949
+
950
+ if(ri->first_time_t || ri->last_time_t)
951
+ return false;
952
+
953
+ return true;
954
+}
955
+
956
+static void rrdinstance_trigger_updates(RRDINSTANCE *ri, bool force, bool escalate) {
957
+ if(unlikely(ri->flags & RRD_FLAG_DONT_PROCESS)) return;
958
+ if(unlikely(!force && !(ri->flags & RRD_FLAG_UPDATED))) return;
959
+
960
+ if(likely(ri->rrdset)) {
961
+ if(unlikely(ri->rrdset->priority != ri->priority)) {
962
+ ri->priority = ri->rrdset->priority;
963
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_PRIORITY);
964
+ }
965
+ if(unlikely(ri->rrdset->update_every != ri->update_every)) {
966
+ ri->update_every = ri->rrdset->update_every;
967
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_UPDATE_EVERY);
968
+ }
969
+ }
970
+ else if(unlikely(rrd_flag_is_collected(ri))) {
971
+ rrd_flag_set_archived(ri);
972
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_LINKING);
973
+ }
974
+
975
+ time_t min_first_time_t = LONG_MAX, max_last_time_t = 0;
976
+ size_t metrics_active = 0, metrics_deleted = 0;
977
+ bool live_retention = true, currently_collected = false;
978
+ {
979
+ RRDMETRIC *rm;
980
+ dfe_start_read((DICTIONARY *)ri->rrdmetrics, rm) {
981
+ if(!(rm->flags & RRD_FLAG_LIVE_RETENTION))
982
+ live_retention = false;
983
+
984
+ if (unlikely((rrdmetric_should_be_deleted(rm)))) {
985
+ metrics_deleted++;
986
+ rrd_flag_unset_updated(rm);
987
+ continue;
988
+ }
989
+
990
+ if(rm->flags & RRD_FLAG_COLLECTED)
991
+ currently_collected = true;
992
+
993
+ metrics_active++;
994
+
995
+ if (rm->first_time_t && rm->first_time_t < min_first_time_t)
996
+ min_first_time_t = rm->first_time_t;
997
+
998
+ if (rm->last_time_t && rm->last_time_t > max_last_time_t)
999
+ max_last_time_t = rm->last_time_t;
1000
+
1001
+ rrd_flag_unset_updated(rm);
1002
+ }
1003
+ dfe_done(rm);
1004
+ }
1005
+
1006
+ if(live_retention && !(ri->flags & RRD_FLAG_LIVE_RETENTION))
1007
+ ri->flags |= RRD_FLAG_LIVE_RETENTION;
1008
+ else if(!live_retention && (ri->flags & RRD_FLAG_LIVE_RETENTION))
1009
+ ri->flags &= ~RRD_FLAG_LIVE_RETENTION;
1010
+
1011
+ if(unlikely(!metrics_active)) {
1012
+ // no metrics available
1013
+
1014
+ if(ri->first_time_t) {
1015
+ ri->first_time_t = 0;
1016
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
1017
+ }
1018
+
1019
+ if(ri->last_time_t) {
1020
+ ri->last_time_t = 0;
1021
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
1022
+ }
1023
+
1024
+ rrd_flag_set_deleted(ri, RRD_FLAG_UPDATE_REASON_ZERO_RETENTION);
1025
+ }
1026
+ else {
1027
+ // we have active metrics...
1028
+
1029
+ if (unlikely(min_first_time_t == LONG_MAX))
1030
+ min_first_time_t = 0;
1031
+
1032
+ if (unlikely(min_first_time_t == 0 || max_last_time_t == 0)) {
1033
+ if(ri->first_time_t) {
1034
+ ri->first_time_t = 0;
1035
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
1036
+ }
1037
+
1038
+ if(ri->last_time_t) {
1039
+ ri->last_time_t = 0;
1040
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
1041
+ }
1042
+
1043
+ if(unlikely(live_retention))
1044
+ rrd_flag_set_deleted(ri, RRD_FLAG_UPDATE_REASON_ZERO_RETENTION);
1045
+ }
1046
+ else {
1047
+ ri->flags &= ~RRD_FLAG_UPDATE_REASON_ZERO_RETENTION;
1048
+
1049
+ if (unlikely(ri->first_time_t != min_first_time_t)) {
1050
+ ri->first_time_t = min_first_time_t;
1051
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
1052
+ }
1053
+
1054
+ if (unlikely(ri->last_time_t != max_last_time_t)) {
1055
+ ri->last_time_t = max_last_time_t;
1056
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
1057
+ }
1058
+
1059
+ if(likely(currently_collected))
1060
+ rrd_flag_set_collected(ri);
1061
+ else
1062
+ rrd_flag_set_archived(ri);
1063
+ }
1064
+ }
1065
+
1066
+ if(unlikely(escalate && ri->flags & RRD_FLAG_UPDATED && !(ri->rc->flags & RRD_FLAG_DONT_PROCESS))) {
1067
+ log_transition(NULL, ri->id, ri->rc->id, ri->flags, "RRDINSTANCE");
1068
+ rrdcontext_trigger_updates(ri->rc, true);
1069
+ }
1070
+}
1071
+
1072
+static inline void rrdinstance_from_rrdset(RRDSET *st) {
1073
+ RRDCONTEXT trc = {
1074
+ .id = string_strdupz(st->context),
1075
+ .title = string_strdupz(st->title),
1076
+ .units = string_strdupz(st->units),
1077
+ .family = string_strdupz(st->family),
1078
+ .priority = st->priority,
1079
+ .chart_type = st->chart_type,
1080
+ .flags = RRD_FLAG_NONE,
1081
+ .rrdhost = st->rrdhost,
1082
+ };
1083
+
1084
+ RRDCONTEXT_ACQUIRED *rca = (RRDCONTEXT_ACQUIRED *)dictionary_set_and_acquire_item((DICTIONARY *)st->rrdhost->rrdctx, string2str(trc.id), &trc, sizeof(trc));
1085
+ RRDCONTEXT *rc = rrdcontext_acquired_value(rca);
1086
+
1087
+ RRDINSTANCE tri = {
1088
+ .id = string_strdupz(st->id),
1089
+ .name = string_strdupz(st->name),
1090
+ .units = string_strdupz(st->units),
1091
+ .family = string_strdupz(st->family),
1092
+ .title = string_strdupz(st->title),
1093
+ .chart_type = st->chart_type,
1094
+ .priority = st->priority,
1095
+ .update_every = st->update_every,
1096
+ .flags = RRD_FLAG_DONT_PROCESS,
1097
+ .rrdset = st,
1098
+ };
1099
+ uuid_copy(tri.uuid, *st->chart_uuid);
1100
+
1101
+ RRDINSTANCE_ACQUIRED *ria = (RRDINSTANCE_ACQUIRED *)dictionary_set_and_acquire_item(rc->rrdinstances, string2str(tri.id), &tri, sizeof(tri));
1102
+
1103
+ RRDCONTEXT_ACQUIRED *rca_old = st->rrdcontext;
1104
+ RRDINSTANCE_ACQUIRED *ria_old = st->rrdinstance;
1105
+
1106
+ st->rrdcontext = rca;
1107
+ st->rrdinstance = ria;
1108
+
1109
+ if(rca == rca_old) {
1110
+ rrdcontext_release(rca_old);
1111
+ rca_old = NULL;
1112
+ }
1113
+
1114
+ if(ria == ria_old) {
1115
+ rrdinstance_release(ria_old);
1116
+ ria_old = NULL;
1117
+ }
1118
+
1119
+ if(rca_old && ria_old) {
1120
+ // the chart changed context
1121
+ RRDCONTEXT *rc_old = rrdcontext_acquired_value(rca_old);
1122
+ RRDINSTANCE *ri_old = rrdinstance_acquired_value(ria_old);
1123
+
1124
+ // migrate all dimensions to the new metrics
1125
+ rrdset_rdlock(st);
1126
+ RRDDIM *rd;
1127
+ rrddim_foreach_read(rd, st) {
1128
+ if (!rd->rrdmetric) continue;
1129
+
1130
+ RRDMETRIC *rm_old = rrdmetric_acquired_value(rd->rrdmetric);
1131
+ rm_old->flags = RRD_FLAG_DELETED|RRD_FLAG_UPDATED|RRD_FLAG_LIVE_RETENTION|RRD_FLAG_UPDATE_REASON_UNUSED|RRD_FLAG_UPDATE_REASON_ZERO_RETENTION;
1132
+ rm_old->rrddim = NULL;
1133
+ rm_old->first_time_t = 0;
1134
+ rm_old->last_time_t = 0;
1135
+
1136
+ rrdmetric_release(rd->rrdmetric);
1137
+ rd->rrdmetric = NULL;
1138
+
1139
+ rrdmetric_from_rrddim(rd);
1140
+ }
1141
+ rrdset_unlock(st);
1142
+
1143
+ // mark the old instance, ready to be deleted
1144
+ if(!(ri_old->flags & RRD_FLAG_OWN_LABELS))
1145
+ ri_old->rrdlabels = rrdlabels_create();
1146
+
1147
+ ri_old->flags = RRD_FLAG_OWN_LABELS|RRD_FLAG_DELETED|RRD_FLAG_UPDATED|RRD_FLAG_LIVE_RETENTION|RRD_FLAG_UPDATE_REASON_UNUSED|RRD_FLAG_UPDATE_REASON_ZERO_RETENTION;
1148
+ ri_old->rrdset = NULL;
1149
+ ri_old->first_time_t = 0;
1150
+ ri_old->last_time_t = 0;
1151
+
1152
+ ri_old->flags &= ~RRD_FLAG_DONT_PROCESS;
1153
+ rc_old->flags &= ~RRD_FLAG_DONT_PROCESS;
1154
+
1155
+ rrdinstance_trigger_updates(ri_old, true, true);
1156
+
1157
+ ri_old->flags |= RRD_FLAG_DONT_PROCESS;
1158
+ rrdinstance_release(ria_old);
1159
+
1160
+ /*
1161
+ // trigger updates on the old context
1162
+ if(!dictionary_stats_entries(rc_old->rrdinstances) && !dictionary_stats_referenced_items(rc_old->rrdinstances)) {
1163
+ rrdcontext_lock(rc_old);
1164
+ rc_old->flags = ((rc_old->flags & RRD_FLAG_QUEUED)?RRD_FLAG_QUEUED:RRD_FLAG_NONE)|RRD_FLAG_DELETED|RRD_FLAG_UPDATED|RRD_FLAG_LIVE_RETENTION|RRD_FLAG_UPDATE_REASON_UNUSED|RRD_FLAG_UPDATE_REASON_ZERO_RETENTION;
1165
+ rc_old->first_time_t = 0;
1166
+ rc_old->last_time_t = 0;
1167
+ rrdcontext_unlock(rc_old);
1168
+ rrdcontext_trigger_updates(rc_old, true);
1169
+ }
1170
+ else
1171
+ rrdcontext_trigger_updates(rc_old, true);
1172
+ */
1173
+
1174
+ rrdcontext_release(rca_old);
1175
+ rca_old = NULL;
1176
+ ria_old = NULL;
1177
+ }
1178
+
1179
+ if(rca_old || ria_old)
1180
+ fatal("RRDCONTEXT: cannot switch rrdcontext without switching rrdinstance too");
1181
+}
1182
+
1183
+#define rrdset_get_rrdinstance(st) rrdset_get_rrdinstance_with_trace(st, __FUNCTION__);
1184
+static inline RRDINSTANCE *rrdset_get_rrdinstance_with_trace(RRDSET *st, const char *function) {
1185
+ if(unlikely(!st->rrdinstance))
1186
+ fatal("RRDINSTANCE: RRDSET '%s' is not linked to an RRDINSTANCE at %s()", st->id, function);
1187
+
1188
+ RRDINSTANCE *ri = rrdinstance_acquired_value(st->rrdinstance);
1189
+
1190
+ if(unlikely(ri->rrdset != st))
1191
+ fatal("RRDINSTANCE: '%s' is not linked to RRDSET '%s' at %s()", string2str(ri->id), st->id, function);
1192
+
1193
+ return ri;
1194
+}
1195
+
1196
+static inline void rrdinstance_rrdset_is_freed(RRDSET *st) {
1197
+ RRDINSTANCE *ri = rrdset_get_rrdinstance(st);
1198
+
1199
+ rrd_flag_set_archived(ri);
1200
+
1201
+ if(!(ri->flags & RRD_FLAG_OWN_LABELS)) {
1202
+ ri->flags |= RRD_FLAG_OWN_LABELS;
1203
+ ri->rrdlabels = rrdlabels_create();
1204
+ rrdlabels_copy(ri->rrdlabels, st->state->chart_labels);
1205
+ }
1206
+
1207
+ ri->rrdset = NULL;
1208
+
1209
+ ri->flags &= ~RRD_FLAG_DONT_PROCESS;
1210
+ rrdinstance_trigger_updates(ri, false, true);
1211
+ ri->flags |= RRD_FLAG_DONT_PROCESS;
1212
+
1213
+ rrdinstance_release(st->rrdinstance);
1214
+ st->rrdinstance = NULL;
1215
+
1216
+ rrdcontext_release(st->rrdcontext);
1217
+ st->rrdcontext = NULL;
1218
+}
1219
+
1220
+static inline void rrdinstance_updated_rrdset_name(RRDSET *st) {
1221
+ // the chart may not be initialized when this is called
1222
+ if(unlikely(!st->rrdinstance)) return;
1223
+
1224
+ RRDINSTANCE *ri = rrdset_get_rrdinstance(st);
1225
+
1226
+ STRING *old = ri->name;
1227
+ ri->name = string_strdupz(st->name);
1228
+
1229
+ if(ri->name != old)
1230
+ rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_NAME);
1231
+
1232
+ string_freez(old);
1233
+
1234
+ rrdinstance_trigger_updates(ri, false, true);
1235
+}
1236
+
1237
+static inline void rrdinstance_updated_rrdset_flags(RRDSET *st) {
1238
+ RRDINSTANCE *ri = rrdset_get_rrdinstance(st);
1239
+
1240
+ if(unlikely(st->flags & (RRDSET_FLAG_ARCHIVED | RRDSET_FLAG_OBSOLETE)))
1241
+ rrd_flag_set_archived(ri);
1242
+
1243
+ ri->flags &= ~RRD_FLAG_DONT_PROCESS;
1244
+ rrdinstance_trigger_updates(ri, false, true);
1245
+ ri->flags |= RRD_FLAG_DONT_PROCESS;
1246
+}
1247
+
1248
+static inline void rrdinstance_collected_rrdset(RRDSET *st) {
1249
+ RRDINSTANCE *ri = rrdset_get_rrdinstance(st);
1250
+
1251
+ if(unlikely(!rrd_flag_is_collected(ri)))
1252
+ rrd_flag_set_collected(ri);
1253
+
1254
+ // the chart is collected, let's process it now
1255
+ if(unlikely(ri->flags & RRD_FLAG_DONT_PROCESS))
1256
+ ri->flags &= ~RRD_FLAG_DONT_PROCESS;
1257
+
1258
+ rrdinstance_trigger_updates(ri, false, true);
1259
+}
1260
+
1261
+// ----------------------------------------------------------------------------
1262
+// RRDCONTEXT
1263
+
1264
+static void rrdcontext_freez(RRDCONTEXT *rc) {
1265
+ string_freez(rc->id);
1266
+ string_freez(rc->title);
1267
+ string_freez(rc->units);
1268
+ string_freez(rc->family);
1269
+}
1270
+
1271
+static uint64_t rrdcontext_get_next_version(RRDCONTEXT *rc) {
1272
+ time_t now = now_realtime_sec();
1273
+ uint64_t version = MAX(rc->version, rc->hub.version);
1274
+ version = MAX((uint64_t)now, version);
1275
+ version++;
1276
+ return version;
1277
+}
1278
+
1279
+static void rrdcontext_message_send_unsafe(RRDCONTEXT *rc, bool snapshot __maybe_unused, void *bundle __maybe_unused) {
1280
+
1281
+ // save it, so that we know the last version we sent to hub
1282
+ rc->version = rc->hub.version = rrdcontext_get_next_version(rc);
1283
+ rc->hub.id = string2str(rc->id);
1284
+ rc->hub.title = string2str(rc->title);
1285
+ rc->hub.units = string2str(rc->units);
1286
+ rc->hub.family = string2str(rc->family);
1287
+ rc->hub.chart_type = rrdset_type_name(rc->chart_type);
1288
+ rc->hub.priority = rc->priority;
1289
+ rc->hub.first_time_t = rc->first_time_t;
1290
+ rc->hub.last_time_t = rrd_flag_is_collected(rc) ? 0 : rc->last_time_t;
1291
+ rc->hub.deleted = (rc->flags & RRD_FLAG_DELETED) ? true : false;
1292
+
1293
+#ifdef ENABLE_ACLK
1294
+ struct context_updated message = {
1295
+ .id = rc->hub.id,
1296
+ .version = rc->hub.version,
1297
+ .title = rc->hub.title,
1298
+ .units = rc->hub.units,
1299
+ .family = rc->hub.family,
1300
+ .chart_type = rc->hub.chart_type,
1301
+ .priority = rc->hub.priority,
1302
+ .first_entry = rc->hub.first_time_t,
1303
+ .last_entry = rc->hub.last_time_t,
1304
+ .deleted = rc->hub.deleted,
1305
+ };
1306
+
1307
+ if(snapshot) {
1308
+ if(!rc->hub.deleted)
1309
+ contexts_snapshot_add_ctx_update(bundle, &message);
1310
+ }
1311
+ else
1312
+ contexts_updated_add_ctx_update(bundle, &message);
1313
+#endif
1314
+
1315
+ // store it to SQL
1316
+
1317
+ if(rc->flags & RRD_FLAG_DELETED) {
1318
+ rrdcontext_delete_from_sql_unsafe(rc);
1319
+ }
1320
+ else {
1321
+ if (ctx_store_context(&rc->rrdhost->host_uuid, &rc->hub) != 0)
1322
+ error("RRDCONTEXT: failed to save context '%s' version %lu to SQL.", rc->hub.id, rc->hub.version);
1323
+ }
1324
+}
1325
+
1326
+static bool check_if_cloud_version_changed_unsafe(RRDCONTEXT *rc, bool sending __maybe_unused) {
1327
+ bool id_changed = false,
1328
+ title_changed = false,
1329
+ units_changed = false,
1330
+ family_changed = false,
1331
+ chart_type_changed = false,
1332
+ priority_changed = false,
1333
+ first_time_changed = false,
1334
+ last_time_changed = false,
1335
+ deleted_changed = false;
1336
+
1337
+ if(unlikely(string2str(rc->id) != rc->hub.id))
1338
+ id_changed = true;
1339
+
1340
+ if(unlikely(string2str(rc->title) != rc->hub.title))
1341
+ title_changed = true;
1342
+
1343
+ if(unlikely(string2str(rc->units) != rc->hub.units))
1344
+ units_changed = true;
1345
+
1346
+ if(unlikely(string2str(rc->family) != rc->hub.family))
1347
+ family_changed = true;
1348
+
1349
+ if(unlikely(rrdset_type_name(rc->chart_type) != rc->hub.chart_type))
1350
+ chart_type_changed = true;
1351
+
1352
+ if(unlikely(rc->priority != rc->hub.priority))
1353
+ priority_changed = true;
1354
+
1355
+ if(unlikely((uint64_t)rc->first_time_t != rc->hub.first_time_t))
1356
+ first_time_changed = true;
1357
+
1358
+ if(unlikely((uint64_t)(rrd_flag_is_collected(rc) ? 0 : rc->last_time_t) != rc->hub.last_time_t))
1359
+ last_time_changed = true;
1360
+
1361
+ if(unlikely(((rc->flags & RRD_FLAG_DELETED) ? true : false) != rc->hub.deleted))
1362
+ deleted_changed = true;
1363
+
1364
+ if(unlikely(id_changed || title_changed || units_changed || family_changed || chart_type_changed || priority_changed || first_time_changed || last_time_changed || deleted_changed)) {
1365
+
1366
+ internal_error(true, "RRDCONTEXT: %s NEW VERSION '%s'%s, version %zu, title '%s'%s, units '%s'%s, family '%s'%s, chart type '%s'%s, priority %u%s, first_time_t %ld%s, last_time_t %ld%s, deleted '%s'%s, (queued for %llu ms, expected %llu ms)",
1367
+ sending?"SENDING":"QUEUE",
1368
+ string2str(rc->id), id_changed ? " (CHANGED)" : "",
1369
+ rc->version,
1370
+ string2str(rc->title), title_changed ? " (CHANGED)" : "",
1371
+ string2str(rc->units), units_changed ? " (CHANGED)" : "",
1372
+ string2str(rc->family), family_changed ? " (CHANGED)" : "",
1373
+ rrdset_type_name(rc->chart_type), chart_type_changed ? " (CHANGED)" : "",
1374
+ rc->priority, priority_changed ? " (CHANGED)" : "",
1375
+ rc->first_time_t, first_time_changed ? " (CHANGED)" : "",
1376
+ rrd_flag_is_collected(rc) ? 0 : rc->last_time_t, last_time_changed ? " (CHANGED)" : "",
1377
+ (rc->flags & RRD_FLAG_DELETED) ? "true" : "false", deleted_changed ? " (CHANGED)" : "",
1378
+ sending ? (now_realtime_usec() - rc->queue.queued_ut) / USEC_PER_MS : 0,
1379
+ sending ? (rc->queue.scheduled_dispatch_ut - rc->queue.queued_ut) / USEC_PER_SEC : 0
1380
+ );
1381
+ return true;
1382
+ }
1383
+
1384
+ return false;
1385
+}
1386
+
1387
+static void rrdcontext_insert_callback(const char *id, void *value, void *data) {
1388
+ (void)id;
1389
+ RRDHOST *host = (RRDHOST *)data;
1390
+ RRDCONTEXT *rc = (RRDCONTEXT *)value;
1391
+
1392
+ rc->rrdhost = host;
1393
+ rc->flags = rc->flags & RRD_FLAGS_ALLOWED_EXTERNALLY_ON_NEW_OBJECTS;
1394
+
1395
+ if(rc->hub.version) {
1396
+ // we are loading data from the SQL database
1397
+
1398
+ if(rc->version)
1399
+ error("RRDCONTEXT: context '%s' is already initialized with version %lu, but it is loaded again from SQL with version %lu", string2str(rc->id), rc->version, rc->hub.version);
1400
+
1401
+ // IMPORTANT
1402
+ // replace all string pointers in rc->hub with our own versions
1403
+ // the originals are coming from a tmp allocation of sqlite
1404
+
1405
+ string_freez(rc->id);
1406
+ rc->id = string_strdupz(rc->hub.id);
1407
+ rc->hub.id = string2str(rc->id);
1408
+
1409
+ string_freez(rc->title);
1410
+ rc->title = string_strdupz(rc->hub.title);
1411
+ rc->hub.title = string2str(rc->title);
1412
+
1413
+ string_freez(rc->units);
1414
+ rc->units = string_strdupz(rc->hub.units);
1415
+ rc->hub.units = string2str(rc->units);
1416
+
1417
+ string_freez(rc->family);
1418
+ rc->family = string_strdupz(rc->hub.family);
1419
+ rc->hub.family = string2str(rc->family);
1420
+
1421
+ rc->chart_type = rrdset_type_id(rc->hub.chart_type);
1422
+ rc->hub.chart_type = rrdset_type_name(rc->chart_type);
1423
+
1424
+ rc->version = rc->hub.version;
1425
+ rc->priority = rc->hub.priority;
1426
+ rc->first_time_t = rc->hub.first_time_t;
1427
+ rc->last_time_t = rc->hub.last_time_t;
1428
+
1429
+ if(rc->hub.deleted || !rc->hub.first_time_t)
1430
+ rrd_flag_set_deleted(rc, 0);
1431
+ else {
1432
+ if (rc->last_time_t == 0)
1433
+ rrd_flag_set_collected(rc);
1434
+ else
1435
+ rrd_flag_set_archived(rc);
1436
+ }
1437
+
1438
+ rc->flags |= RRD_FLAG_UPDATE_REASON_LOAD_SQL;
1439
+ }
1440
+ else {
1441
+ // we are adding this context now for the first time
1442
+ rc->version = now_realtime_sec();
1443
+ }
1444
+
1445
+ rrdinstances_create(rc);
1446
+ netdata_mutex_init(&rc->mutex);
1447
+
1448
+ // signal the react callback to do the job
1449
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_NEW_OBJECT);
1450
+}
1451
+
1452
+static void rrdcontext_delete_callback(const char *id, void *value, void *data) {
1453
+ (void)id;
1454
+ RRDHOST *host = (RRDHOST *)data;
1455
+ (void)host;
1456
+
1457
+ RRDCONTEXT *rc = (RRDCONTEXT *)value;
1458
+
1459
+ rrdinstances_destroy(rc);
1460
+ netdata_mutex_destroy(&rc->mutex);
1461
+ rrdcontext_freez(rc);
1462
+}
1463
+
1464
+static void rrdcontext_conflict_callback(const char *id, void *oldv, void *newv, void *data) {
1465
+ (void)id;
1466
+ RRDHOST *host = (RRDHOST *)data;
1467
+ (void)host;
1468
+
1469
+ RRDCONTEXT *rc = (RRDCONTEXT *)oldv;
1470
+ RRDCONTEXT *rc_new = (RRDCONTEXT *)newv;
1471
+
1472
+ rrdcontext_lock(rc);
1473
+
1474
+ if(rc->title != rc_new->title) {
1475
+ STRING *old_title = rc->title;
1476
+ rc->title = string_2way_merge(rc->title, rc_new->title);
1477
+ string_freez(old_title);
1478
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_TITLE);
1479
+ }
1480
+
1481
+ if(rc->units != rc_new->units) {
1482
+ STRING *old_units = rc->units;
1483
+ rc->units = string_dup(rc_new->units);
1484
+ string_freez(old_units);
1485
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_UNITS);
1486
+ }
1487
+
1488
+ if(rc->family != rc_new->family) {
1489
+ STRING *old_family = rc->family;
1490
+ rc->family = string_2way_merge(rc->family, rc_new->family);
1491
+ string_freez(old_family);
1492
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_FAMILY);
1493
+ }
1494
+
1495
+ if(rc->chart_type != rc_new->chart_type) {
1496
+ rc->chart_type = rc_new->chart_type;
1497
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_CHART_TYPE);
1498
+ }
1499
+
1500
+ if(rc->priority != rc_new->priority) {
1501
+ rc->priority = rc_new->priority;
1502
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_PRIORITY);
1503
+ }
1504
+
1505
+ rc->flags |= (rc_new->flags & RRD_FLAGS_ALLOWED_EXTERNALLY_ON_NEW_OBJECTS);
1506
+
1507
+ if(rrd_flag_is_collected(rc) && rrd_flag_is_archived(rc))
1508
+ rrd_flag_set_collected(rc);
1509
+
1510
+ if(rc->flags & RRD_FLAG_UPDATED)
1511
+ rc->flags |= RRD_FLAG_UPDATE_REASON_UPDATED_OBJECT;
1512
+
1513
+ rrdcontext_unlock(rc);
1514
+
1515
+ // free the resources of the new one
1516
+ rrdcontext_freez(rc_new);
1517
+
1518
+ // the react callback will continue from here
1519
+}
1520
+
1521
+static void rrdcontext_react_callback(const char *id __maybe_unused, void *value, void *data __maybe_unused) {
1522
+ RRDCONTEXT *rc = (RRDCONTEXT *)value;
1523
+
1524
+ rrdcontext_trigger_updates(rc, false);
1525
+}
1526
+
1527
+void rrdhost_create_rrdcontexts(RRDHOST *host) {
1528
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1529
+ return;
1530
+
1531
+ if(unlikely(!host)) return;
1532
+ if(likely(host->rrdctx)) return;
1533
+
1534
+ host->rrdctx = (RRDCONTEXTS *)dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE);
1535
+ dictionary_register_insert_callback((DICTIONARY *)host->rrdctx, rrdcontext_insert_callback, (void *)host);
1536
+ dictionary_register_delete_callback((DICTIONARY *)host->rrdctx, rrdcontext_delete_callback, (void *)host);
1537
+ dictionary_register_conflict_callback((DICTIONARY *)host->rrdctx, rrdcontext_conflict_callback, (void *)host);
1538
+ dictionary_register_react_callback((DICTIONARY *)host->rrdctx, rrdcontext_react_callback, (void *)host);
1539
+
1540
+ host->rrdctx_queue = (RRDCONTEXTS *)dictionary_create(DICTIONARY_FLAG_DONT_OVERWRITE_VALUE | DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE);
1541
+}
1542
+
1543
+void rrdhost_destroy_rrdcontexts(RRDHOST *host) {
1544
+ if(unlikely(!host)) return;
1545
+ if(unlikely(!host->rrdctx)) return;
1546
+
1547
+ if(host->rrdctx_queue) {
1548
+ dictionary_destroy((DICTIONARY *)host->rrdctx_queue);
1549
+ host->rrdctx_queue = NULL;
1550
+ }
1551
+
1552
+ dictionary_destroy((DICTIONARY *)host->rrdctx);
1553
+ host->rrdctx = NULL;
1554
+}
1555
+
1556
+static inline bool rrdcontext_should_be_deleted(RRDCONTEXT *rc) {
1557
+ if(likely(!(rc->flags & RRD_FLAG_DELETED)))
1558
+ return false;
1559
+
1560
+ if(likely(!(rc->flags & RRD_FLAG_LIVE_RETENTION)))
1561
+ return false;
1562
+
1563
+ if(unlikely(rc->flags & RRD_FLAGS_PREVENTING_DELETIONS))
1564
+ return false;
1565
+
1566
+ if(unlikely(dictionary_stats_referenced_items(rc->rrdinstances) != 0))
1567
+ return false;
1568
+
1569
+ if(unlikely(dictionary_stats_entries(rc->rrdinstances) != 0))
1570
+ return false;
1571
+
1572
+ if(unlikely(rc->first_time_t || rc->last_time_t))
1573
+ return false;
1574
+
1575
+ return true;
1576
+}
1577
+
1578
+static void rrdcontext_trigger_updates(RRDCONTEXT *rc, bool force) {
1579
+ if(unlikely(rc->flags & RRD_FLAG_DONT_PROCESS)) return;
1580
+ if(unlikely(!force && !(rc->flags & RRD_FLAG_UPDATED))) return;
1581
+
1582
+ rrdcontext_lock(rc);
1583
+
1584
+ size_t min_priority = LONG_MAX;
1585
+ time_t min_first_time_t = LONG_MAX, max_last_time_t = 0;
1586
+ size_t instances_active = 0, instances_deleted = 0;
1587
+ bool live_retention = true, currently_collected = false;
1588
+ {
1589
+ RRDINSTANCE *ri;
1590
+ dfe_start_read(rc->rrdinstances, ri) {
1591
+ if(!(ri->flags & RRD_FLAG_LIVE_RETENTION))
1592
+ live_retention = false;
1593
+
1594
+ if (unlikely(rrdinstance_should_be_deleted(ri))) {
1595
+ instances_deleted++;
1596
+ rrd_flag_unset_updated(ri);
1597
+ continue;
1598
+ }
1599
+
1600
+ if(ri->flags & RRD_FLAG_COLLECTED)
1601
+ currently_collected = true;
1602
+
1603
+ internal_error(rc->units != ri->units,
1604
+ "RRDCONTEXT: '%s' rrdinstance '%s' has different units, context '%s', instance '%s'",
1605
+ string2str(rc->id), string2str(ri->id),
1606
+ string2str(rc->units), string2str(ri->units));
1607
+
1608
+ instances_active++;
1609
+
1610
+ if (ri->priority >= RRDCONTEXT_MINIMUM_ALLOWED_PRIORITY && ri->priority < min_priority)
1611
+ min_priority = ri->priority;
1612
+
1613
+ if (ri->first_time_t && ri->first_time_t < min_first_time_t)
1614
+ min_first_time_t = ri->first_time_t;
1615
+
1616
+ if (ri->last_time_t && ri->last_time_t > max_last_time_t)
1617
+ max_last_time_t = ri->last_time_t;
1618
+
1619
+ rrd_flag_unset_updated(ri);
1620
+ }
1621
+ dfe_done(ri);
1622
+ }
1623
+
1624
+ if(live_retention && !(rc->flags & RRD_FLAG_LIVE_RETENTION))
1625
+ rc->flags |= RRD_FLAG_LIVE_RETENTION;
1626
+ else if(!live_retention && (rc->flags & RRD_FLAG_LIVE_RETENTION))
1627
+ rc->flags &= ~RRD_FLAG_LIVE_RETENTION;
1628
+
1629
+ if(unlikely(!instances_active)) {
1630
+ // we had some instances, but they are gone now...
1631
+
1632
+ if(rc->first_time_t) {
1633
+ rc->first_time_t = 0;
1634
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
1635
+ }
1636
+
1637
+ if(rc->last_time_t) {
1638
+ rc->last_time_t = 0;
1639
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
1640
+ }
1641
+
1642
+ rrd_flag_set_deleted(rc, RRD_FLAG_UPDATE_REASON_ZERO_RETENTION);
1643
+ }
1644
+ else {
1645
+ // we have some active instances...
1646
+
1647
+ if (unlikely(min_first_time_t == LONG_MAX))
1648
+ min_first_time_t = 0;
1649
+
1650
+ if (unlikely(min_first_time_t == 0 && max_last_time_t == 0)) {
1651
+ if(rc->first_time_t) {
1652
+ rc->first_time_t = 0;
1653
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
1654
+ }
1655
+
1656
+ if(rc->last_time_t) {
1657
+ rc->last_time_t = 0;
1658
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
1659
+ }
1660
+
1661
+ rrd_flag_set_deleted(rc, RRD_FLAG_UPDATE_REASON_ZERO_RETENTION);
1662
+ }
1663
+ else {
1664
+ rc->flags &= ~RRD_FLAG_UPDATE_REASON_ZERO_RETENTION;
1665
+
1666
+ if (unlikely(rc->first_time_t != min_first_time_t)) {
1667
+ rc->first_time_t = min_first_time_t;
1668
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
1669
+ }
1670
+
1671
+ if (rc->last_time_t != max_last_time_t) {
1672
+ rc->last_time_t = max_last_time_t;
1673
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
1674
+ }
1675
+
1676
+ if(likely(currently_collected))
1677
+ rrd_flag_set_collected(rc);
1678
+ else
1679
+ rrd_flag_set_archived(rc);
1680
+ }
1681
+
1682
+ if (min_priority != LONG_MAX && rc->priority != min_priority) {
1683
+ rc->priority = min_priority;
1684
+ rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_PRIORITY);
1685
+ }
1686
+ }
1687
+
1688
+ if(unlikely(rc->flags & RRD_FLAG_UPDATED)) {
1689
+ log_transition(NULL, NULL, rc->id, rc->flags, "RRDCONTEXT");
1690
+
1691
+ if(check_if_cloud_version_changed_unsafe(rc, false)) {
1692
+ rc->version = rrdcontext_get_next_version(rc);
1693
+
1694
+ if(rc->flags & RRD_FLAG_QUEUED) {
1695
+ rc->queue.queued_ut = now_realtime_usec();
1696
+ rc->queue.queued_flags |= rc->flags;
1697
+ }
1698
+ else {
1699
+ rc->queue.queued_ut = now_realtime_usec();
1700
+ rc->queue.queued_flags = rc->flags;
1701
+
1702
+ rc->flags |= RRD_FLAG_QUEUED;
1703
+ dictionary_set((DICTIONARY *)rc->rrdhost->rrdctx_queue, string2str(rc->id), rc, sizeof(*rc));
1704
+ }
1705
+ }
1706
+
1707
+ rrd_flag_unset_updated(rc);
1708
+ }
1709
+
1710
+ rrdcontext_unlock(rc);
1711
+}
1712
+
1713
+// ----------------------------------------------------------------------------
1714
+// public API
1715
+
1716
+void rrdcontext_updated_rrddim(RRDDIM *rd) {
1717
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1718
+ return;
1719
+
1720
+ rrdmetric_from_rrddim(rd);
1721
+}
1722
+
1723
+void rrdcontext_removed_rrddim(RRDDIM *rd) {
1724
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1725
+ return;
1726
+
1727
+ rrdmetric_rrddim_is_freed(rd);
1728
+}
1729
+
1730
+void rrdcontext_updated_rrddim_algorithm(RRDDIM *rd) {
1731
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1732
+ return;
1733
+
1734
+ rrdmetric_updated_rrddim_flags(rd);
1735
+}
1736
+
1737
+void rrdcontext_updated_rrddim_multiplier(RRDDIM *rd) {
1738
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1739
+ return;
1740
+
1741
+ rrdmetric_updated_rrddim_flags(rd);
1742
+}
1743
+
1744
+void rrdcontext_updated_rrddim_divisor(RRDDIM *rd) {
1745
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1746
+ return;
1747
+
1748
+ rrdmetric_updated_rrddim_flags(rd);
1749
+}
1750
+
1751
+void rrdcontext_updated_rrddim_flags(RRDDIM *rd) {
1752
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1753
+ return;
1754
+
1755
+ rrdmetric_updated_rrddim_flags(rd);
1756
+}
1757
+
1758
+void rrdcontext_collected_rrddim(RRDDIM *rd) {
1759
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1760
+ return;
1761
+
1762
+ rrdmetric_collected_rrddim(rd);
1763
+}
1764
+
1765
+void rrdcontext_updated_rrdset(RRDSET *st) {
1766
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1767
+ return;
1768
+
1769
+ rrdinstance_from_rrdset(st);
1770
+}
1771
+
1772
+void rrdcontext_removed_rrdset(RRDSET *st) {
1773
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1774
+ return;
1775
+
1776
+ rrdinstance_rrdset_is_freed(st);
1777
+}
1778
+
1779
+void rrdcontext_updated_rrdset_name(RRDSET *st) {
1780
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1781
+ return;
1782
+
1783
+ rrdinstance_updated_rrdset_name(st);
1784
+}
1785
+
1786
+void rrdcontext_updated_rrdset_flags(RRDSET *st) {
1787
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1788
+ return;
1789
+
1790
+ rrdinstance_updated_rrdset_flags(st);
1791
+}
1792
+
1793
+void rrdcontext_collected_rrdset(RRDSET *st) {
1794
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1795
+ return;
1796
+
1797
+ rrdinstance_collected_rrdset(st);
1798
+}
1799
+
1800
+void rrdcontext_host_child_connected(RRDHOST *host) {
1801
+ (void)host;
1802
+
1803
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1804
+ return;
1805
+
1806
+ // no need to do anything here
1807
+ ;
1808
+}
1809
+
1810
+void rrdcontext_host_child_disconnected(RRDHOST *host) {
1811
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
1812
+ return;
1813
+
1814
+ rrdcontext_recalculate_host_retention(host, RRD_FLAG_UPDATE_REASON_DISCONNECTED_CHILD, -1);
1815
+}
1816
+
1817
+// ----------------------------------------------------------------------------
1818
+// ACLK interface
1819
+
1820
+static bool rrdhost_check_our_claim_id(const char *claim_id) {
1821
+ if(!localhost->aclk_state.claimed_id) return false;
1822
+ return (strcasecmp(claim_id, localhost->aclk_state.claimed_id) == 0) ? true : false;
1823
+}
1824
+
1825
+static RRDHOST *rrdhost_find_by_node_id(const char *node_id) {
1826
+ uuid_t uuid;
1827
+ if (uuid_parse(node_id, uuid))
1828
+ return NULL;
1829
+
1830
+ RRDHOST *host = NULL;
1831
+
1832
+ rrd_rdlock();
1833
+ rrdhost_foreach_read(host) {
1834
+ if(!host->node_id) continue;
1835
+
1836
+ if(uuid_compare(uuid, *host->node_id) == 0)
1837
+ break;
1838
+ }
1839
+ rrd_unlock();
1840
+
1841
+ return host;
1842
+}
1843
+
1844
+void rrdcontext_hub_checkpoint_command(void *ptr) {
1845
+ struct ctxs_checkpoint *cmd = ptr;
1846
+
1847
+ if(!rrdhost_check_our_claim_id(cmd->claim_id)) {
1848
+ error("RRDCONTEXT: received checkpoint command for claim_id '%s', node id '%s', but this is not our claim id. Ours '%s', received '%s'. Ignoring command.",
1849
+ cmd->claim_id, cmd->node_id,
1850
+ localhost->aclk_state.claimed_id?localhost->aclk_state.claimed_id:"NOT SET",
1851
+ cmd->claim_id);
1852
+
1853
+ return;
1854
+ }
1855
+
1856
+ RRDHOST *host = rrdhost_find_by_node_id(cmd->node_id);
1857
+ if(!host) {
1858
+ error("RRDCONTEXT: received checkpoint command for claim id '%s', node id '%s', but there is no node with such node id here. Ignoring command.",
1859
+ cmd->claim_id, cmd->node_id);
1860
+
1861
+ return;
1862
+ }
1863
+
1864
+ if(rrdhost_flag_check(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)) {
1865
+ info("RRDCONTEXT: received checkpoint command for claim id '%s', node id '%s', while node '%s' has an active context streaming.",
1866
+ cmd->claim_id, cmd->node_id, host->hostname);
1867
+
1868
+ // disable it temporarily, so that our worker will not attempt to send messages in parallel
1869
+ rrdhost_flag_clear(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS);
1870
+ }
1871
+
1872
+ uint64_t our_version_hash = rrdcontext_version_hash(host);
1873
+
1874
+ if(cmd->version_hash != our_version_hash) {
1875
+ error("RRDCONTEXT: received version hash %lu for host '%s', does not match our version hash %lu. Sending snapshot of all contexts.",
1876
+ cmd->version_hash, host->hostname, our_version_hash);
1877
+
1878
+#ifdef ENABLE_ACLK
1879
+ // prepare the snapshot
1880
+ char uuid[UUID_STR_LEN];
1881
+ uuid_unparse_lower(*host->node_id, uuid);
1882
+ contexts_snapshot_t bundle = contexts_snapshot_new(cmd->claim_id, uuid, our_version_hash);
1883
+
1884
+ // do a deep scan on every metric of the host to make sure all our data are updated
1885
+ rrdcontext_recalculate_host_retention(host, RRD_FLAG_NONE, -1);
1886
+
1887
+ // calculate version hash and pack all the messages together in one go
1888
+ our_version_hash = rrdcontext_version_hash_with_callback(host, rrdcontext_message_send_unsafe, true, bundle);
1889
+
1890
+ // update the version
1891
+ contexts_snapshot_set_version(bundle, our_version_hash);
1892
+
1893
+ // send it
1894
+ aclk_send_contexts_snapshot(bundle);
1895
+#endif
1896
+ }
1897
+
1898
+ internal_error(true, "RRDCONTEXT: host '%s' enabling streaming of contexts", host->hostname);
1899
+ rrdhost_flag_set(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS);
1900
+ char node_str[UUID_STR_LEN];
1901
+ uuid_unparse_lower(*host->node_id, node_str);
1902
+ log_access("ACLK REQ [%s (%s)]: STREAM CONTEXTS ENABLED", node_str, host->hostname);
1903
+}
1904
+
1905
+void rrdcontext_hub_stop_streaming_command(void *ptr) {
1906
+ struct stop_streaming_ctxs *cmd = ptr;
1907
+
1908
+ if(!rrdhost_check_our_claim_id(cmd->claim_id)) {
1909
+ error("RRDCONTEXT: received stop streaming command for claim_id '%s', node id '%s', but this is not our claim id. Ours '%s', received '%s'. Ignoring command.",
1910
+ cmd->claim_id, cmd->node_id,
1911
+ localhost->aclk_state.claimed_id?localhost->aclk_state.claimed_id:"NOT SET",
1912
+ cmd->claim_id);
1913
+
1914
+ return;
1915
+ }
1916
+
1917
+ RRDHOST *host = rrdhost_find_by_node_id(cmd->node_id);
1918
+ if(!host) {
1919
+ error("RRDCONTEXT: received stop streaming command for claim id '%s', node id '%s', but there is no node with such node id here. Ignoring command.",
1920
+ cmd->claim_id, cmd->node_id);
1921
+
1922
+ return;
1923
+ }
1924
+
1925
+ if(!rrdhost_flag_check(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)) {
1926
+ error("RRDCONTEXT: received stop streaming command for claim id '%s', node id '%s', but node '%s' does not have active context streaming. Ignoring command.",
1927
+ cmd->claim_id, cmd->node_id, host->hostname);
1928
+
1929
+ return;
1930
+ }
1931
+
1932
+ internal_error(true, "RRDCONTEXT: host '%s' disabling streaming of contexts", host->hostname);
1933
+ rrdhost_flag_clear(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS);
1934
+}
1935
+
1936
+// ----------------------------------------------------------------------------
1937
+// web API
1938
+
1939
+struct rrdcontext_to_json {
1940
+ BUFFER *wb;
1941
+ RRDCONTEXT_TO_JSON_OPTIONS options;
1942
+ time_t after;
1943
+ time_t before;
1944
+ SIMPLE_PATTERN *chart_label_key;
1945
+ SIMPLE_PATTERN *chart_labels_filter;
1946
+ SIMPLE_PATTERN *chart_dimensions;
1947
+ size_t written;
1948
+ time_t now;
1949
+ time_t combined_first_time_t;
1950
+ time_t combined_last_time_t;
1951
+ RRD_FLAGS combined_flags;
1952
+};
1953
+
1954
+static inline int rrdmetric_to_json_callback(const char *id, void *value, void *data) {
1955
+ struct rrdcontext_to_json * t = data;
1956
+ RRDMETRIC *rm = value;
1957
+ BUFFER *wb = t->wb;
1958
+ RRDCONTEXT_TO_JSON_OPTIONS options = t->options;
1959
+ time_t after = t->after;
1960
+ time_t before = t->before;
1961
+
1962
+ if((rm->flags & RRD_FLAG_DELETED) && !(options & RRDCONTEXT_OPTION_SHOW_DELETED))
1963
+ return 0;
1964
+
1965
+ if(after && (!rm->last_time_t || after > rm->last_time_t))
1966
+ return 0;
1967
+
1968
+ if(before && (!rm->first_time_t || before < rm->first_time_t))
1969
+ return 0;
1970
+
1971
+ if(t->chart_dimensions
1972
+ && !simple_pattern_matches(t->chart_dimensions, string2str(rm->id))
1973
+ && !simple_pattern_matches(t->chart_dimensions, string2str(rm->name)))
1974
+ return 0;
1975
+
1976
+ if(t->written) {
1977
+ buffer_strcat(wb, ",\n");
1978
+ t->combined_first_time_t = MIN(t->combined_first_time_t, rm->first_time_t);
1979
+ t->combined_last_time_t = MAX(t->combined_last_time_t, rm->last_time_t);
1980
+ t->combined_flags |= rm->flags;
1981
+ }
1982
+ else {
1983
+ buffer_strcat(wb, "\n");
1984
+ t->combined_first_time_t = rm->first_time_t;
1985
+ t->combined_last_time_t = rm->last_time_t;
1986
+ t->combined_flags = rm->flags;
1987
+ }
1988
+
1989
+ buffer_sprintf(wb, "\t\t\t\t\t\t\"%s\": {", id);
1990
+
1991
+ if(options & RRDCONTEXT_OPTION_SHOW_UUIDS) {
1992
+ char uuid[UUID_STR_LEN];
1993
+ uuid_unparse(rm->uuid, uuid);
1994
+ buffer_sprintf(wb, "\n\t\t\t\t\t\t\t\"uuid\":\"%s\",", uuid);
1995
+ }
1996
+
1997
+ buffer_sprintf(wb,
1998
+ "\n\t\t\t\t\t\t\t\"name\":\"%s\""
1999
+ ",\n\t\t\t\t\t\t\t\"first_time_t\":%ld"
2000
+ ",\n\t\t\t\t\t\t\t\"last_time_t\":%ld"
2001
+ ",\n\t\t\t\t\t\t\t\"collected\":%s"
2002
+ , string2str(rm->name)
2003
+ , rm->first_time_t
2004
+ , rrd_flag_is_collected(rm) ? t->now : rm->last_time_t
2005
+ , rm->flags & RRD_FLAG_COLLECTED ? "true" : "false"
2006
+ );
2007
+
2008
+ if(options & RRDCONTEXT_OPTION_SHOW_DELETED) {
2009
+ buffer_sprintf(wb,
2010
+ ",\n\t\t\t\t\t\t\t\"deleted\":%s"
2011
+ , rm->flags & RRD_FLAG_DELETED ? "true" : "false"
2012
+ );
2013
+ }
2014
+
2015
+ if(options & RRDCONTEXT_OPTION_SHOW_FLAGS) {
2016
+ buffer_strcat(wb, ",\n\t\t\t\t\t\t\t\"flags\":\"");
2017
+ rrd_flags_to_buffer(rm->flags, wb);
2018
+ buffer_strcat(wb, "\"");
2019
+ }
2020
+
2021
+ buffer_strcat(wb, "\n\t\t\t\t\t\t}");
2022
+ t->written++;
2023
+ return 1;
2024
+}
2025
+
2026
+static inline int rrdinstance_to_json_callback(const char *id, void *value, void *data) {
2027
+ struct rrdcontext_to_json *t_parent = data;
2028
+ RRDINSTANCE *ri = value;
2029
+ BUFFER *wb = t_parent->wb;
2030
+ RRDCONTEXT_TO_JSON_OPTIONS options = t_parent->options;
2031
+ time_t after = t_parent->after;
2032
+ time_t before = t_parent->before;
2033
+ bool has_filter = t_parent->chart_label_key || t_parent->chart_labels_filter || t_parent->chart_dimensions;
2034
+
2035
+ if((ri->flags & RRD_FLAG_DELETED) && !(options & RRDCONTEXT_OPTION_SHOW_DELETED))
2036
+ return 0;
2037
+
2038
+ if(after && (!ri->last_time_t || after > ri->last_time_t))
2039
+ return 0;
2040
+
2041
+ if(before && (!ri->first_time_t || before < ri->first_time_t))
2042
+ return 0;
2043
+
2044
+ if(t_parent->chart_label_key && !rrdlabels_match_simple_pattern_parsed(ri->rrdlabels, t_parent->chart_label_key, '\0'))
2045
+ return 0;
2046
+
2047
+ if(t_parent->chart_labels_filter && !rrdlabels_match_simple_pattern_parsed(ri->rrdlabels, t_parent->chart_labels_filter, ':'))
2048
+ return 0;
2049
+
2050
+ time_t first_time_t = ri->first_time_t;
2051
+ time_t last_time_t = ri->last_time_t;
2052
+ RRD_FLAGS flags = ri->flags;
2053
+
2054
+ BUFFER *wb_metrics = NULL;
2055
+ if(options & RRDCONTEXT_OPTION_SHOW_METRICS || t_parent->chart_dimensions) {
2056
+
2057
+ wb_metrics = buffer_create(4096);
2058
+
2059
+ struct rrdcontext_to_json t_metrics = {
2060
+ .wb = wb_metrics,
2061
+ .options = options,
2062
+ .chart_label_key = t_parent->chart_label_key,
2063
+ .chart_labels_filter = t_parent->chart_labels_filter,
2064
+ .chart_dimensions = t_parent->chart_dimensions,
2065
+ .after = after,
2066
+ .before = before,
2067
+ .written = 0,
2068
+ .now = t_parent->now,
2069
+ };
2070
+ dictionary_walkthrough_read(ri->rrdmetrics, rrdmetric_to_json_callback, &t_metrics);
2071
+
2072
+ if(has_filter && !t_metrics.written) {
2073
+ buffer_free(wb_metrics);
2074
+ return 0;
2075
+ }
2076
+
2077
+ first_time_t = t_metrics.combined_first_time_t;
2078
+ last_time_t = t_metrics.combined_last_time_t;
2079
+ flags = t_metrics.combined_flags;
2080
+ }
2081
+
2082
+ if(t_parent->written) {
2083
+ buffer_strcat(wb, ",\n");
2084
+ t_parent->combined_first_time_t = MIN(t_parent->combined_first_time_t, first_time_t);
2085
+ t_parent->combined_last_time_t = MAX(t_parent->combined_last_time_t, last_time_t);
2086
+ t_parent->combined_flags |= flags;
2087
+ }
2088
+ else {
2089
+ buffer_strcat(wb, "\n");
2090
+ t_parent->combined_first_time_t = first_time_t;
2091
+ t_parent->combined_last_time_t = last_time_t;
2092
+ t_parent->combined_flags = flags;
2093
+ }
2094
+
2095
+ buffer_sprintf(wb, "\t\t\t\t\"%s\": {", id);
2096
+
2097
+ if(options & RRDCONTEXT_OPTION_SHOW_UUIDS) {
2098
+ char uuid[UUID_STR_LEN];
2099
+ uuid_unparse(ri->uuid, uuid);
2100
+ buffer_sprintf(wb,"\n\t\t\t\t\t\"uuid\":\"%s\",", uuid);
2101
+ }
2102
+
2103
+ buffer_sprintf(wb,
2104
+ "\n\t\t\t\t\t\"name\":\"%s\""
2105
+ ",\n\t\t\t\t\t\"context\":\"%s\""
2106
+ ",\n\t\t\t\t\t\"title\":\"%s\""
2107
+ ",\n\t\t\t\t\t\"units\":\"%s\""
2108
+ ",\n\t\t\t\t\t\"family\":\"%s\""
2109
+ ",\n\t\t\t\t\t\"chart_type\":\"%s\""
2110
+ ",\n\t\t\t\t\t\"priority\":%u"
2111
+ ",\n\t\t\t\t\t\"update_every\":%d"
2112
+ ",\n\t\t\t\t\t\"first_time_t\":%ld"
2113
+ ",\n\t\t\t\t\t\"last_time_t\":%ld"
2114
+ ",\n\t\t\t\t\t\"collected\":%s"
2115
+ , string2str(ri->name)
2116
+ , string2str(ri->rc->id)
2117
+ , string2str(ri->title)
2118
+ , string2str(ri->units)
2119
+ , string2str(ri->family)
2120
+ , rrdset_type_name(ri->chart_type)
2121
+ , ri->priority
2122
+ , ri->update_every
2123
+ , first_time_t
2124
+ , (flags & RRD_FLAG_COLLECTED) ? t_parent->now : last_time_t
2125
+ , (flags & RRD_FLAG_COLLECTED) ? "true" : "false"
2126
+ );
2127
+
2128
+ if(options & RRDCONTEXT_OPTION_SHOW_DELETED) {
2129
+ buffer_sprintf(wb,
2130
+ ",\n\t\t\t\t\t\"deleted\":%s"
2131
+ , (ri->flags & RRD_FLAG_DELETED) ? "true" : "false"
2132
+ );
2133
+ }
2134
+
2135
+ if(options & RRDCONTEXT_OPTION_SHOW_FLAGS) {
2136
+ buffer_strcat(wb, ",\n\t\t\t\t\t\"flags\":\"");
2137
+ rrd_flags_to_buffer(ri->flags, wb);
2138
+ buffer_strcat(wb, "\"");
2139
+ }
2140
+
2141
+ if(options & RRDCONTEXT_OPTION_SHOW_LABELS && ri->rrdlabels && dictionary_stats_entries(ri->rrdlabels)) {
2142
+ buffer_sprintf(wb, ",\n\t\t\t\t\t\"labels\": {\n");
2143
+ rrdlabels_to_buffer(ri->rrdlabels, wb, "\t\t\t\t\t\t", ":", "\"", ",\n", NULL, NULL, NULL, NULL);
2144
+ buffer_strcat(wb, "\n\t\t\t\t\t}");
2145
+ }
2146
+
2147
+ if(wb_metrics) {
2148
+ buffer_sprintf(wb, ",\n\t\t\t\t\t\"dimensions\": {");
2149
+ buffer_fast_strcat(wb, buffer_tostring(wb_metrics), buffer_strlen(wb_metrics));
2150
+ buffer_strcat(wb, "\n\t\t\t\t\t}");
2151
+
2152
+ buffer_free(wb_metrics);
2153
+ }
2154
+
2155
+ buffer_strcat(wb, "\n\t\t\t\t}");
2156
+ t_parent->written++;
2157
+ return 1;
2158
+}
2159
+
2160
+static inline int rrdcontext_to_json_callback(const char *id, void *value, void *data) {
2161
+ struct rrdcontext_to_json *t_parent = data;
2162
+ RRDCONTEXT *rc = value;
2163
+ BUFFER *wb = t_parent->wb;
2164
+ RRDCONTEXT_TO_JSON_OPTIONS options = t_parent->options;
2165
+ time_t after = t_parent->after;
2166
+ time_t before = t_parent->before;
2167
+ bool has_filter = t_parent->chart_label_key || t_parent->chart_labels_filter || t_parent->chart_dimensions;
2168
+
2169
+ if((rc->flags & RRD_FLAG_DELETED) && !(options & RRDCONTEXT_OPTION_SHOW_DELETED))
2170
+ return 0;
2171
+
2172
+ if(options & RRDCONTEXT_OPTION_DEEPSCAN)
2173
+ rrdcontext_recalculate_context_retention(rc, RRD_FLAG_NONE, -1);
2174
+
2175
+ if(after && (!rc->last_time_t || after > rc->last_time_t))
2176
+ return 0;
2177
+
2178
+ if(before && (!rc->first_time_t || before < rc->first_time_t))
2179
+ return 0;
2180
+
2181
+ time_t first_time_t = rc->first_time_t;
2182
+ time_t last_time_t = rc->last_time_t;
2183
+ RRD_FLAGS flags = rc->flags;
2184
+
2185
+ BUFFER *wb_instances = NULL;
2186
+ if((options & (RRDCONTEXT_OPTION_SHOW_LABELS|RRDCONTEXT_OPTION_SHOW_INSTANCES|RRDCONTEXT_OPTION_SHOW_METRICS))
2187
+ || t_parent->chart_label_key
2188
+ || t_parent->chart_labels_filter
2189
+ || t_parent->chart_dimensions) {
2190
+
2191
+ wb_instances = buffer_create(4096);
2192
+
2193
+ struct rrdcontext_to_json t_instances = {
2194
+ .wb = wb_instances,
2195
+ .options = options,
2196
+ .chart_label_key = t_parent->chart_label_key,
2197
+ .chart_labels_filter = t_parent->chart_labels_filter,
2198
+ .chart_dimensions = t_parent->chart_dimensions,
2199
+ .after = after,
2200
+ .before = before,
2201
+ .written = 0,
2202
+ .now = t_parent->now,
2203
+ };
2204
+ dictionary_walkthrough_read(rc->rrdinstances, rrdinstance_to_json_callback, &t_instances);
2205
+
2206
+ if(has_filter && !t_instances.written) {
2207
+ buffer_free(wb_instances);
2208
+ return 0;
2209
+ }
2210
+
2211
+ first_time_t = t_instances.combined_first_time_t;
2212
+ last_time_t = t_instances.combined_last_time_t;
2213
+ flags = t_instances.combined_flags;
2214
+ }
2215
+
2216
+ if(t_parent->written)
2217
+ buffer_strcat(wb, ",\n");
2218
+ else
2219
+ buffer_strcat(wb, "\n");
2220
+
2221
+ if(options & RRDCONTEXT_OPTION_SKIP_ID)
2222
+ buffer_sprintf(wb, "\t\t\{");
2223
+ else
2224
+ buffer_sprintf(wb, "\t\t\"%s\": {", id);
2225
+
2226
+ rrdcontext_lock(rc);
2227
+
2228
+ buffer_sprintf(wb,
2229
+ "\n\t\t\t\"title\":\"%s\""
2230
+ ",\n\t\t\t\"units\":\"%s\""
2231
+ ",\n\t\t\t\"family\":\"%s\""
2232
+ ",\n\t\t\t\"chart_type\":\"%s\""
2233
+ ",\n\t\t\t\"priority\":%u"
2234
+ ",\n\t\t\t\"first_time_t\":%ld"
2235
+ ",\n\t\t\t\"last_time_t\":%ld"
2236
+ ",\n\t\t\t\"collected\":%s"
2237
+ , string2str(rc->title)
2238
+ , string2str(rc->units)
2239
+ , string2str(rc->family)
2240
+ , rrdset_type_name(rc->chart_type)
2241
+ , rc->priority
2242
+ , first_time_t
2243
+ , (flags & RRD_FLAG_COLLECTED) ? t_parent->now : last_time_t
2244
+ , (flags & RRD_FLAG_COLLECTED) ? "true" : "false"
2245
+ );
2246
+
2247
+ if(options & RRDCONTEXT_OPTION_SHOW_DELETED) {
2248
+ buffer_sprintf(wb,
2249
+ ",\n\t\t\t\"deleted\":%s"
2250
+ , (rc->flags & RRD_FLAG_DELETED) ? "true" : "false"
2251
+ );
2252
+ }
2253
+
2254
+ if(options & RRDCONTEXT_OPTION_SHOW_FLAGS) {
2255
+ buffer_strcat(wb, ",\n\t\t\t\"flags\":\"");
2256
+ rrd_flags_to_buffer(rc->flags, wb);
2257
+ buffer_strcat(wb, "\"");
2258
+ }
2259
+
2260
+ if(options & RRDCONTEXT_OPTION_SHOW_QUEUED) {
2261
+ buffer_strcat(wb, ",\n\t\t\t\"queued_reasons\":\"");
2262
+ rrd_reasons_to_buffer(rc->queue.queued_flags, wb);
2263
+ buffer_strcat(wb, "\"");
2264
+
2265
+ buffer_sprintf(wb,
2266
+ ",\n\t\t\t\"last_queued\":%llu"
2267
+ ",\n\t\t\t\"scheduled_dispatch\":%llu"
2268
+ ",\n\t\t\t\"last_dequeued\":%llu"
2269
+ ",\n\t\t\t\"hub_version\":%lu"
2270
+ ",\n\t\t\t\"version\":%lu"
2271
+ , rc->queue.queued_ut / USEC_PER_SEC
2272
+ , rc->queue.scheduled_dispatch_ut / USEC_PER_SEC
2273
+ , rc->queue.dequeued_ut / USEC_PER_SEC
2274
+ , rc->hub.version
2275
+ , rc->version
2276
+ );
2277
+ }
2278
+
2279
+ rrdcontext_unlock(rc);
2280
+
2281
+ if(wb_instances) {
2282
+ buffer_sprintf(wb, ",\n\t\t\t\"charts\": {");
2283
+ buffer_fast_strcat(wb, buffer_tostring(wb_instances), buffer_strlen(wb_instances));
2284
+ buffer_strcat(wb, "\n\t\t\t}");
2285
+
2286
+ buffer_free(wb_instances);
2287
+ }
2288
+
2289
+ buffer_strcat(wb, "\n\t\t}");
2290
+ t_parent->written++;
2291
+ return 1;
2292
+}
2293
+
2294
+int rrdcontext_to_json(RRDHOST *host, BUFFER *wb, time_t after, time_t before, RRDCONTEXT_TO_JSON_OPTIONS options, const char *context, SIMPLE_PATTERN *chart_label_key, SIMPLE_PATTERN *chart_labels_filter, SIMPLE_PATTERN *chart_dimensions) {
2295
+ RRDCONTEXT_ACQUIRED *rca = (RRDCONTEXT_ACQUIRED *)dictionary_get_and_acquire_item((DICTIONARY *)host->rrdctx, context);
2296
+ if(!rca) return HTTP_RESP_NOT_FOUND;
2297
+
2298
+ RRDCONTEXT *rc = rrdcontext_acquired_value(rca);
2299
+
2300
+ if(after != 0 && before != 0) {
2301
+ long long after_wanted = after;
2302
+ long long before_wanted = before;
2303
+ rrdr_relative_window_to_absolute(&after_wanted, &before_wanted);
2304
+ after = after_wanted;
2305
+ before = before_wanted;
2306
+ }
2307
+
2308
+ struct rrdcontext_to_json t_contexts = {
2309
+ .wb = wb,
2310
+ .options = options|RRDCONTEXT_OPTION_SKIP_ID,
2311
+ .chart_label_key = chart_label_key,
2312
+ .chart_labels_filter = chart_labels_filter,
2313
+ .chart_dimensions = chart_dimensions,
2314
+ .after = after,
2315
+ .before = before,
2316
+ .written = 0,
2317
+ .now = now_realtime_sec(),
2318
+ };
2319
+ rrdcontext_to_json_callback(context, rc, &t_contexts);
2320
+
2321
+ rrdcontext_release(rca);
2322
+
2323
+ if(!t_contexts.written)
2324
+ return HTTP_RESP_NOT_FOUND;
2325
+
2326
+ return HTTP_RESP_OK;
2327
+}
2328
+
2329
+int rrdcontexts_to_json(RRDHOST *host, BUFFER *wb, time_t after, time_t before, RRDCONTEXT_TO_JSON_OPTIONS options, SIMPLE_PATTERN *chart_label_key, SIMPLE_PATTERN *chart_labels_filter, SIMPLE_PATTERN *chart_dimensions) {
2330
+ char node_uuid[UUID_STR_LEN] = "";
2331
+
2332
+ if(host->node_id)
2333
+ uuid_unparse(*host->node_id, node_uuid);
2334
+
2335
+ if(after != 0 && before != 0) {
2336
+ long long after_wanted = after;
2337
+ long long before_wanted = before;
2338
+ rrdr_relative_window_to_absolute(&after_wanted, &before_wanted);
2339
+ after = after_wanted;
2340
+ before = before_wanted;
2341
+ }
2342
+
2343
+ buffer_sprintf(wb, "{\n"
2344
+ "\t\"hostname\": \"%s\""
2345
+ ",\n\t\"machine_guid\": \"%s\""
2346
+ ",\n\t\"node_id\": \"%s\""
2347
+ ",\n\t\"claim_id\": \"%s\""
2348
+ , host->hostname
2349
+ , host->machine_guid
2350
+ , node_uuid
2351
+ , host->aclk_state.claimed_id ? host->aclk_state.claimed_id : ""
2352
+ );
2353
+
2354
+ if(options & RRDCONTEXT_OPTION_SHOW_LABELS) {
2355
+ buffer_sprintf(wb, ",\n\t\"host_labels\": {\n");
2356
+ rrdlabels_to_buffer(host->host_labels, wb, "\t\t", ":", "\"", ",\n", NULL, NULL, NULL, NULL);
2357
+ buffer_strcat(wb, "\n\t}");
2358
+ }
2359
+
2360
+ buffer_sprintf(wb, ",\n\t\"contexts\": {");
2361
+ struct rrdcontext_to_json t_contexts = {
2362
+ .wb = wb,
2363
+ .options = options,
2364
+ .chart_label_key = chart_label_key,
2365
+ .chart_labels_filter = chart_labels_filter,
2366
+ .chart_dimensions = chart_dimensions,
2367
+ .after = after,
2368
+ .before = before,
2369
+ .written = 0,
2370
+ .now = now_realtime_sec(),
2371
+ };
2372
+ dictionary_walkthrough_read((DICTIONARY *)host->rrdctx, rrdcontext_to_json_callback, &t_contexts);
2373
+
2374
+ // close contexts, close main
2375
+ buffer_strcat(wb, "\n\t}\n}");
2376
+
2377
+ return HTTP_RESP_OK;
2378
+}
2379
+
2380
+// ----------------------------------------------------------------------------
2381
+// load from SQL
2382
+
2383
+static void rrdinstance_load_clabel(SQL_CLABEL_DATA *sld, void *data) {
2384
+ RRDINSTANCE *ri = data;
2385
+ rrdlabels_add(ri->rrdlabels, sld->label_key, sld->label_value, sld->label_source);
2386
+}
2387
+
2388
+static void rrdinstance_load_dimension(SQL_DIMENSION_DATA *sd, void *data) {
2389
+ RRDINSTANCE *ri = data;
2390
+
2391
+ RRDMETRIC trm = {
2392
+ .id = string_strdupz(sd->id),
2393
+ .name = string_strdupz(sd->name),
2394
+ .flags = RRD_FLAG_ARCHIVED | RRD_FLAG_UPDATE_REASON_LOAD_SQL,
2395
+ };
2396
+ uuid_copy(trm.uuid, sd->dim_id);
2397
+
2398
+ dictionary_set(ri->rrdmetrics, string2str(trm.id), &trm, sizeof(trm));
2399
+}
2400
+
2401
+static void rrdinstance_load_chart_callback(SQL_CHART_DATA *sc, void *data) {
2402
+ RRDHOST *host = data;
2403
+
2404
+ RRDCONTEXT tc = {
2405
+ .id = string_strdupz(sc->context),
2406
+ .title = string_strdupz(sc->title),
2407
+ .units = string_strdupz(sc->units),
2408
+ .family = string_strdupz(sc->family),
2409
+ .priority = sc->priority,
2410
+ .chart_type = sc->chart_type,
2411
+ .flags = RRD_FLAG_ARCHIVED | RRD_FLAG_DONT_PROCESS | RRD_FLAG_UPDATE_REASON_LOAD_SQL,
2412
+ .rrdhost = host,
2413
+ };
2414
+
2415
+ RRDCONTEXT_ACQUIRED *rca = (RRDCONTEXT_ACQUIRED *)dictionary_set_and_acquire_item((DICTIONARY *)host->rrdctx, string2str(tc.id), &tc, sizeof(tc));
2416
+ RRDCONTEXT *rc = rrdcontext_acquired_value(rca);
2417
+
2418
+ RRDINSTANCE tri = {
2419
+ .id = string_strdupz(sc->id),
2420
+ .name = string_strdupz(sc->name),
2421
+ .title = string_strdupz(sc->title),
2422
+ .units = string_strdupz(sc->units),
2423
+ .family = string_strdupz(sc->family),
2424
+ .chart_type = sc->chart_type,
2425
+ .priority = sc->priority,
2426
+ .update_every = sc->update_every,
2427
+ .flags = RRD_FLAG_ARCHIVED | RRD_FLAG_DONT_PROCESS | RRD_FLAG_UPDATE_REASON_LOAD_SQL,
2428
+ };
2429
+ uuid_copy(tri.uuid, sc->chart_id);
2430
+
2431
+ RRDINSTANCE_ACQUIRED *ria = (RRDINSTANCE_ACQUIRED *)dictionary_set_and_acquire_item(rc->rrdinstances, sc->id, &tri, sizeof(tri));
2432
+ RRDINSTANCE *ri = rrdinstance_acquired_value(ria);
2433
+
2434
+ ctx_get_dimension_list(&ri->uuid, rrdinstance_load_dimension, ri);
2435
+ ctx_get_label_list(&ri->uuid, rrdinstance_load_clabel, ri);
2436
+ ri->flags &= ~RRD_FLAG_DONT_PROCESS;
2437
+ rrdinstance_trigger_updates(ri, true, true);
2438
+
2439
+ // let the instance be in "don't process" mode
2440
+ // so that we process it once, when it is collected
2441
+ ri->flags |= RRD_FLAG_DONT_PROCESS;
2442
+
2443
+ rrdinstance_release(ria);
2444
+ rrdcontext_release(rca);
2445
+}
2446
+
2447
+static void rrdcontext_load_context_callback(VERSIONED_CONTEXT_DATA *ctx_data, void *data) {
2448
+ RRDHOST *host = data;
2449
+ (void)host;
2450
+
2451
+ RRDCONTEXT trc = {
2452
+ .id = string_strdupz(ctx_data->id),
2453
+ .flags = RRD_FLAG_ARCHIVED | RRD_FLAG_DONT_PROCESS | RRD_FLAG_UPDATE_REASON_LOAD_SQL,
2454
+
2455
+ // no need to set more data here
2456
+ // we only need the hub data
2457
+
2458
+ .hub = *ctx_data,
2459
+ };
2460
+ dictionary_set((DICTIONARY *)host->rrdctx, string2str(trc.id), &trc, sizeof(trc));
2461
+}
2462
+
2463
+void rrdhost_load_rrdcontext_data(RRDHOST *host) {
2464
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
2465
+ return;
2466
+
2467
+ if(host->rrdctx) return;
2468
+
2469
+ rrdhost_create_rrdcontexts(host);
2470
+ ctx_get_context_list(&host->host_uuid, rrdcontext_load_context_callback, host);
2471
+ ctx_get_chart_list(&host->host_uuid, rrdinstance_load_chart_callback, host);
2472
+
2473
+ RRDCONTEXT *rc;
2474
+ dfe_start_read((DICTIONARY *)host->rrdctx, rc) {
2475
+ rc->flags &= ~RRD_FLAG_DONT_PROCESS;
2476
+ rrdcontext_trigger_updates(rc, true);
2477
+ }
2478
+ dfe_done(rc);
2479
+}
2480
+
2481
+// ----------------------------------------------------------------------------
2482
+// the worker thread
2483
+
2484
+static inline usec_t rrdcontext_calculate_queued_dispatch_time_ut(RRDCONTEXT *rc, usec_t now_ut) {
2485
+
2486
+ if(likely(rc->queue.delay_calc_ut >= rc->queue.queued_ut))
2487
+ return rc->queue.scheduled_dispatch_ut;
2488
+
2489
+ RRD_FLAGS flags = rc->queue.queued_flags;
2490
+
2491
+ usec_t delay = LONG_MAX;
2492
+ int i;
2493
+ struct rrdcontext_reason *reason;
2494
+ for(i = 0, reason = &rrdcontext_reasons[i]; reason->name ; reason = &rrdcontext_reasons[++i]) {
2495
+ if(unlikely(flags & reason->flag)) {
2496
+ if(reason->delay_ut < delay)
2497
+ delay = reason->delay_ut;
2498
+ }
2499
+ }
2500
+
2501
+ if(unlikely(delay == LONG_MAX)) {
2502
+ internal_error(true, "RRDCONTEXT: '%s', cannot find minimum delay of flags %x", string2str(rc->id), (unsigned int)flags);
2503
+ delay = 60 * USEC_PER_SEC;
2504
+ }
2505
+
2506
+ rc->queue.delay_calc_ut = now_ut;
2507
+ usec_t dispatch_ut = rc->queue.scheduled_dispatch_ut = rc->queue.queued_ut + delay;
2508
+ return dispatch_ut;
2509
+}
2510
+
2511
+#define WORKER_JOB_HOSTS 1
2512
+#define WORKER_JOB_CHECK 2
2513
+#define WORKER_JOB_SEND 3
2514
+#define WORKER_JOB_DEQUEUE 4
2515
+#define WORKER_JOB_RETENTION 5
2516
+#define WORKER_JOB_QUEUED 6
2517
+#define WORKER_JOB_CLEANUP 7
2518
+#define WORKER_JOB_CLEANUP_DELETE 8
2519
+
2520
+static usec_t rrdcontext_next_db_rotation_ut = 0;
2521
+void rrdcontext_db_rotation(void) {
2522
+ // called when the db rotates its database
2523
+ rrdcontext_next_db_rotation_ut = now_realtime_usec() + FULL_RETENTION_SCAN_DELAY_AFTER_DB_ROTATION_SECS * USEC_PER_SEC;
2524
+}
2525
+
2526
+static uint64_t rrdcontext_version_hash_with_callback(
2527
+ RRDHOST *host,
2528
+ void (*callback)(RRDCONTEXT *, bool, void *),
2529
+ bool snapshot,
2530
+ void *bundle) {
2531
+
2532
+ if(unlikely(!host || !host->rrdctx)) return 0;
2533
+
2534
+ RRDCONTEXT *rc;
2535
+ uint64_t hash = 0;
2536
+
2537
+ // loop through all contexts of the host
2538
+ dfe_start_read((DICTIONARY *)host->rrdctx, rc) {
2539
+
2540
+ rrdcontext_lock(rc);
2541
+
2542
+ if(unlikely(callback))
2543
+ callback(rc, snapshot, bundle);
2544
+
2545
+ // skip any deleted contexts
2546
+ if(unlikely(rc->flags & RRD_FLAG_DELETED)) {
2547
+ rrdcontext_unlock(rc);
2548
+ continue;
2549
+ }
2550
+
2551
+ // we use rc->hub.* which has the latest
2552
+ // metadata we have sent to the hub
2553
+
2554
+ // if a context is currently queued, rc->hub.* does NOT
2555
+ // reflect the queued changes. rc->hub.* is updated with
2556
+ // their metadata, after messages are dispatched to hub.
2557
+
2558
+ // when the context is being collected,
2559
+ // rc->hub.last_time_t is already zero
2560
+
2561
+ hash += rc->hub.version + rc->hub.last_time_t - rc->hub.first_time_t;
2562
+
2563
+ rrdcontext_unlock(rc);
2564
+
2565
+ }
2566
+ dfe_done(rc);
2567
+
2568
+ return hash;
2569
+}
2570
+
2571
+static void rrdcontext_recalculate_context_retention(RRDCONTEXT *rc, RRD_FLAGS reason, int job_id) {
2572
+ RRDINSTANCE *ri;
2573
+ dfe_start_read(rc->rrdinstances, ri) {
2574
+ RRDMETRIC *rm;
2575
+ dfe_start_read(ri->rrdmetrics, rm) {
2576
+
2577
+ if(job_id >= 0)
2578
+ worker_is_busy(job_id);
2579
+
2580
+ rrd_flag_set_updated(rm, reason);
2581
+
2582
+ rm->flags &= ~RRD_FLAG_DONT_PROCESS;
2583
+ rrdmetric_trigger_updates(rm, true, false);
2584
+ }
2585
+ dfe_done(rm);
2586
+
2587
+ ri->flags &= ~RRD_FLAG_DONT_PROCESS;
2588
+ rrdinstance_trigger_updates(ri, true, false);
2589
+ ri->flags |= RRD_FLAG_DONT_PROCESS;
2590
+ }
2591
+ dfe_done(ri);
2592
+
2593
+ rc->flags &= ~RRD_FLAG_DONT_PROCESS;
2594
+ rrdcontext_trigger_updates(rc, true);
2595
+}
2596
+
2597
+static void rrdcontext_recalculate_host_retention(RRDHOST *host, RRD_FLAGS reason, int job_id) {
2598
+ if(unlikely(!host || !host->rrdctx)) return;
2599
+
2600
+ RRDCONTEXT *rc;
2601
+ dfe_start_read((DICTIONARY *)host->rrdctx, rc) {
2602
+ rrdcontext_recalculate_context_retention(rc, reason, job_id);
2603
+ }
2604
+ dfe_done(rc);
2605
+}
2606
+
2607
+static void rrdcontext_recalculate_retention(int job_id) {
2608
+ rrdcontext_next_db_rotation_ut = 0;
2609
+ rrd_rdlock();
2610
+ RRDHOST *host;
2611
+ rrdhost_foreach_read(host) {
2612
+ rrdcontext_recalculate_host_retention(host, RRD_FLAG_UPDATE_REASON_DB_ROTATION, job_id);
2613
+ }
2614
+ rrd_unlock();
2615
+}
2616
+
2617
+void rrdcontext_delete_from_sql_unsafe(RRDCONTEXT *rc) {
2618
+ // we need to refresh the string pointers in rc->hub
2619
+ // in case the context changed values
2620
+ rc->hub.id = string2str(rc->id);
2621
+ rc->hub.title = string2str(rc->title);
2622
+ rc->hub.units = string2str(rc->units);
2623
+ rc->hub.family = string2str(rc->family);
2624
+
2625
+ // delete it from SQL
2626
+ if(ctx_delete_context(&rc->rrdhost->host_uuid, &rc->hub) != 0)
2627
+ error("RRDCONTEXT: failed to delete context '%s' version %lu from SQL.", rc->hub.id, rc->hub.version);
2628
+}
2629
+
2630
+static void rrdcontext_garbage_collect(void) {
2631
+ rrd_rdlock();
2632
+ RRDHOST *host;
2633
+ rrdhost_foreach_read(host) {
2634
+ RRDCONTEXT *rc;
2635
+ dfe_start_write((DICTIONARY *)host->rrdctx, rc) {
2636
+ worker_is_busy(WORKER_JOB_CLEANUP);
2637
+
2638
+ rrdcontext_lock(rc);
2639
+
2640
+ if(unlikely(rrdcontext_should_be_deleted(rc))) {
2641
+ worker_is_busy(WORKER_JOB_CLEANUP_DELETE);
2642
+ rrdcontext_delete_from_sql_unsafe(rc);
2643
+
2644
+ if(dictionary_del_having_write_lock((DICTIONARY *)host->rrdctx, string2str(rc->id)) != 0)
2645
+ error("RRDCONTEXT: '%s' of host '%s' failed to be deleted from rrdcontext dictionary.",
2646
+ string2str(rc->id), host->hostname);
2647
+ }
2648
+ else {
2649
+ RRDINSTANCE *ri;
2650
+ dfe_start_write(rc->rrdinstances, ri) {
2651
+ if(rrdinstance_should_be_deleted(ri)) {
2652
+ worker_is_busy(WORKER_JOB_CLEANUP_DELETE);
2653
+ dictionary_del_having_write_lock(rc->rrdinstances, string2str(ri->id));
2654
+ }
2655
+ else {
2656
+ RRDMETRIC *rm;
2657
+ dfe_start_write(ri->rrdmetrics, rm) {
2658
+ if(rrdmetric_should_be_deleted(rm)) {
2659
+ worker_is_busy(WORKER_JOB_CLEANUP_DELETE);
2660
+ dictionary_del_having_write_lock(ri->rrdmetrics, string2str(rm->id));
2661
+ }
2662
+ }
2663
+ dfe_done(rm);
2664
+ }
2665
+ }
2666
+ dfe_done(ri);
2667
+ }
2668
+
2669
+ // the item is referenced in the dictionary
2670
+ // so, it is still here to unlock, even if we have deleted it
2671
+ rrdcontext_unlock(rc);
2672
+ }
2673
+ dfe_done(rc);
2674
+ }
2675
+ rrd_unlock();
2676
+}
2677
+
2678
+static void rrdcontext_main_cleanup(void *ptr) {
2679
+ struct netdata_static_thread *static_thread = (struct netdata_static_thread *)ptr;
2680
+ static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
2681
+ // custom code
2682
+ static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
2683
+}
2684
+
2685
+void *rrdcontext_main(void *ptr) {
2686
+ netdata_thread_cleanup_push(rrdcontext_main_cleanup, ptr);
2687
+
2688
+ if(unlikely(rrdcontext_enabled == CONFIG_BOOLEAN_NO))
2689
+ goto exit;
2690
+
2691
+ worker_register("RRDCONTEXT");
2692
+ worker_register_job_name(WORKER_JOB_HOSTS, "hosts");
2693
+ worker_register_job_name(WORKER_JOB_CHECK, "dedup checks");
2694
+ worker_register_job_name(WORKER_JOB_SEND, "sent contexts");
2695
+ worker_register_job_name(WORKER_JOB_DEQUEUE, "deduped contexts");
2696
+ worker_register_job_name(WORKER_JOB_RETENTION, "metrics retention");
2697
+ worker_register_job_name(WORKER_JOB_QUEUED, "queued contexts");
2698
+ worker_register_job_name(WORKER_JOB_CLEANUP, "cleanups");
2699
+ worker_register_job_name(WORKER_JOB_CLEANUP_DELETE, "deletes");
2700
+
2701
+ heartbeat_t hb;
2702
+ heartbeat_init(&hb);
2703
+ usec_t step = USEC_PER_SEC * RRDCONTEXT_WORKER_THREAD_HEARTBEAT_SECS;
2704
+
2705
+ while (!netdata_exit) {
2706
+ worker_is_idle();
2707
+ heartbeat_next(&hb, step);
2708
+
2709
+ if(unlikely(netdata_exit)) break;
2710
+
2711
+ if(!aclk_connected) continue;
2712
+
2713
+ usec_t now_ut = now_realtime_usec();
2714
+
2715
+ if(rrdcontext_next_db_rotation_ut && now_ut > rrdcontext_next_db_rotation_ut) {
2716
+ rrdcontext_recalculate_retention(WORKER_JOB_RETENTION);
2717
+ rrdcontext_garbage_collect();
2718
+ rrdcontext_next_db_rotation_ut = 0;
2719
+ }
2720
+
2721
+ rrd_rdlock();
2722
+ RRDHOST *host;
2723
+ rrdhost_foreach_read(host) {
2724
+ if(unlikely(netdata_exit)) break;
2725
+
2726
+ worker_is_busy(WORKER_JOB_HOSTS);
2727
+
2728
+ // check if we have received a streaming command for this host
2729
+ if(!rrdhost_flag_check(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS))
2730
+ continue;
2731
+
2732
+ // check if there are queued items to send
2733
+ if(!dictionary_stats_entries((DICTIONARY *)host->rrdctx_queue))
2734
+ continue;
2735
+
2736
+ if(!host->node_id)
2737
+ continue;
2738
+
2739
+ size_t messages_added = 0;
2740
+ contexts_updated_t bundle = NULL;
2741
+
2742
+ RRDCONTEXT *rc;
2743
+ dfe_start_write((DICTIONARY *)host->rrdctx_queue, rc) {
2744
+ if(unlikely(netdata_exit)) break;
2745
+
2746
+ if(unlikely(messages_added >= MESSAGES_PER_BUNDLE_TO_SEND_TO_HUB_PER_HOST))
2747
+ break;
2748
+
2749
+ worker_is_busy(WORKER_JOB_QUEUED);
2750
+ usec_t dispatch_ut = rrdcontext_calculate_queued_dispatch_time_ut(rc, now_ut);
2751
+ char *claim_id = get_agent_claimid();
2752
+ if(unlikely(now_ut >= dispatch_ut) && claim_id) {
2753
+ worker_is_busy(WORKER_JOB_CHECK);
2754
+
2755
+ rrdcontext_lock(rc);
2756
+
2757
+ if(check_if_cloud_version_changed_unsafe(rc, true)) {
2758
+ worker_is_busy(WORKER_JOB_SEND);
2759
+
2760
+#ifdef ENABLE_ACLK
2761
+ if(!bundle) {
2762
+ // prepare the bundle to send the messages
2763
+ char uuid[UUID_STR_LEN];
2764
+ uuid_unparse_lower(*host->node_id, uuid);
2765
+
2766
+ bundle = contexts_updated_new(claim_id, uuid, 0, now_ut);
2767
+ }
2768
+#endif
2769
+ // update the hub data of the context, give a new version, pack the message
2770
+ // and save an update to SQL
2771
+ rrdcontext_message_send_unsafe(rc, false, bundle);
2772
+ messages_added++;
2773
+
2774
+ rc->queue.dequeued_ut = now_ut;
2775
+ }
2776
+ else
2777
+ rc->version = rc->hub.version;
2778
+
2779
+ // remove the queued flag, so that it can be queued again
2780
+ rc->flags &= ~RRD_FLAG_QUEUED;
2781
+
2782
+ // remove it from the queue
2783
+ worker_is_busy(WORKER_JOB_DEQUEUE);
2784
+ dictionary_del_having_write_lock((DICTIONARY *)host->rrdctx_queue, string2str(rc->id));
2785
+
2786
+ if(unlikely(rrdcontext_should_be_deleted(rc))) {
2787
+ // this is a deleted context - delete it forever...
2788
+
2789
+ worker_is_busy(WORKER_JOB_CLEANUP_DELETE);
2790
+ rrdcontext_delete_from_sql_unsafe(rc);
2791
+
2792
+ STRING *id = string_dup(rc->id);
2793
+ rrdcontext_unlock(rc);
2794
+
2795
+ // delete it from the master dictionary
2796
+ if(dictionary_del((DICTIONARY *)host->rrdctx, string2str(rc->id)) != 0)
2797
+ error("RRDCONTEXT: '%s' of host '%s' failed to be deleted from rrdcontext dictionary.",
2798
+ string2str(id), host->hostname);
2799
+
2800
+ string_freez(id);
2801
+ }
2802
+ else
2803
+ rrdcontext_unlock(rc);
2804
+ }
2805
+ freez(claim_id);
2806
+ }
2807
+ dfe_done(rc);
2808
+
2809
+#ifdef ENABLE_ACLK
2810
+ if(!netdata_exit && bundle) {
2811
+ // we have a bundle to send messages
2812
+
2813
+ // update the version hash
2814
+ contexts_updated_update_version_hash(bundle, rrdcontext_version_hash(host));
2815
+
2816
+ // send it
2817
+ aclk_send_contexts_updated(bundle);
2818
+ }
2819
+ else if(bundle)
2820
+ contexts_updated_delete(bundle);
2821
+#endif
2822
+ }
2823
+ rrd_unlock();
2824
+
2825
+ }
2826
+
2827
+exit:
2828
+ netdata_thread_cleanup_pop(1);
2829
+ return NULL;
2830
+}
database/rrdcontext.h
new
+91
@@ -0,0 +1,91 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_RRDCONTEXT_H
4
+#define NETDATA_RRDCONTEXT_H 1
5
+
6
+// ----------------------------------------------------------------------------
7
+// RRDMETRIC
8
+
9
+typedef struct rrdmetric_acquired RRDMETRIC_ACQUIRED;
10
+
11
+
12
+// ----------------------------------------------------------------------------
13
+// RRDINSTANCE
14
+
15
+typedef struct rrdinstance_acquired RRDINSTANCE_ACQUIRED;
16
+
17
+// ----------------------------------------------------------------------------
18
+// RRDCONTEXT
19
+
20
+typedef struct rrdcontexts_dictionary RRDCONTEXTS;
21
+typedef struct rrdcontext_acquired RRDCONTEXT_ACQUIRED;
22
+
23
+// ----------------------------------------------------------------------------
24
+
25
+#include "rrd.h"
26
+
27
+// ----------------------------------------------------------------------------
28
+// public API for rrdhost
29
+
30
+extern void rrdhost_load_rrdcontext_data(RRDHOST *host);
31
+extern void rrdhost_create_rrdcontexts(RRDHOST *host);
32
+extern void rrdhost_destroy_rrdcontexts(RRDHOST *host);
33
+
34
+extern void rrdcontext_host_child_connected(RRDHOST *host);
35
+extern void rrdcontext_host_child_disconnected(RRDHOST *host);
36
+
37
+typedef enum {
38
+ RRDCONTEXT_OPTION_NONE = 0,
39
+ RRDCONTEXT_OPTION_SHOW_METRICS = (1 << 0),
40
+ RRDCONTEXT_OPTION_SHOW_INSTANCES = (1 << 1),
41
+ RRDCONTEXT_OPTION_SHOW_LABELS = (1 << 2),
42
+ RRDCONTEXT_OPTION_SHOW_QUEUED = (1 << 3),
43
+ RRDCONTEXT_OPTION_SHOW_FLAGS = (1 << 4),
44
+ RRDCONTEXT_OPTION_SHOW_DELETED = (1 << 5),
45
+ RRDCONTEXT_OPTION_DEEPSCAN = (1 << 6),
46
+ RRDCONTEXT_OPTION_SHOW_UUIDS = (1 << 7),
47
+ RRDCONTEXT_OPTION_SKIP_ID = (1 << 31), // internal use
48
+} RRDCONTEXT_TO_JSON_OPTIONS;
49
+
50
+#define RRDCONTEXT_OPTIONS_ALL (RRDCONTEXT_OPTION_SHOW_METRICS|RRDCONTEXT_OPTION_SHOW_INSTANCES|RRDCONTEXT_OPTION_SHOW_LABELS|RRDCONTEXT_OPTION_SHOW_QUEUED|RRDCONTEXT_OPTION_SHOW_FLAGS|RRDCONTEXT_OPTION_SHOW_DELETED|RRDCONTEXT_OPTION_SHOW_UUIDS)
51
+
52
+extern int rrdcontext_to_json(RRDHOST *host, BUFFER *wb, time_t after, time_t before, RRDCONTEXT_TO_JSON_OPTIONS options, const char *context, SIMPLE_PATTERN *chart_label_key, SIMPLE_PATTERN *chart_labels_filter, SIMPLE_PATTERN *chart_dimensions);
53
+extern int rrdcontexts_to_json(RRDHOST *host, BUFFER *wb, time_t after, time_t before, RRDCONTEXT_TO_JSON_OPTIONS options, SIMPLE_PATTERN *chart_label_key, SIMPLE_PATTERN *chart_labels_filter, SIMPLE_PATTERN *chart_dimensions);
54
+
55
+// ----------------------------------------------------------------------------
56
+// public API for rrddims
57
+
58
+extern void rrdcontext_updated_rrddim(RRDDIM *rd);
59
+extern void rrdcontext_removed_rrddim(RRDDIM *rd);
60
+extern void rrdcontext_updated_rrddim_algorithm(RRDDIM *rd);
61
+extern void rrdcontext_updated_rrddim_multiplier(RRDDIM *rd);
62
+extern void rrdcontext_updated_rrddim_divisor(RRDDIM *rd);
63
+extern void rrdcontext_updated_rrddim_flags(RRDDIM *rd);
64
+extern void rrdcontext_collected_rrddim(RRDDIM *rd);
65
+
66
+// ----------------------------------------------------------------------------
67
+// public API for rrdsets
68
+
69
+extern void rrdcontext_updated_rrdset(RRDSET *st);
70
+extern void rrdcontext_removed_rrdset(RRDSET *st);
71
+extern void rrdcontext_updated_rrdset_name(RRDSET *st);
72
+extern void rrdcontext_updated_rrdset_flags(RRDSET *st);
73
+extern void rrdcontext_collected_rrdset(RRDSET *st);
74
+
75
+// ----------------------------------------------------------------------------
76
+// public API for ACLK
77
+
78
+extern void rrdcontext_hub_checkpoint_command(void *cmd);
79
+extern void rrdcontext_hub_stop_streaming_command(void *cmd);
80
+
81
+
82
+// ----------------------------------------------------------------------------
83
+// public API for threads
84
+
85
+extern int rrdcontext_enabled;
86
+
87
+extern void rrdcontext_db_rotation(void);
88
+extern void *rrdcontext_main(void *);
89
+
90
+#endif // NETDATA_RRDCONTEXT_H
91
+
database/rrddim.c
+12
@@ -73,6 +73,7 @@ inline int rrddim_set_algorithm(RRDSET *st, RRDDIM *rd, RRD_ALGORITHM algorithm)
73
rd->exposed = 0;
74
rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
75
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
76
+ rrdcontext_updated_rrddim_algorithm(rd);
77
return 1;
78
}
79
@@ -85,6 +86,7 @@ inline int rrddim_set_multiplier(RRDSET *st, RRDDIM *rd, collected_number multip
86
rd->exposed = 0;
87
rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
88
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
89
+ rrdcontext_updated_rrddim_multiplier(rd);
90
return 1;
91
}
92
@@ -97,6 +99,7 @@ inline int rrddim_set_divisor(RRDSET *st, RRDDIM *rd, collected_number divisor)
99
rd->exposed = 0;
100
rrdset_flag_set(st, RRDSET_FLAG_HOMOGENEOUS_CHECK);
101
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
102
+ rrdcontext_updated_rrddim_divisor(rd);
103
return 1;
104
}
105
@@ -199,6 +202,7 @@ RRDDIM *rrddim_add_custom(RRDSET *st, const char *id, const char *name, collecte
202
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
203
}
204
rrdset_unlock(st);
205
+ rrdcontext_updated_rrddim(rd);
206
return rd;
207
}
208
@@ -344,6 +348,7 @@ RRDDIM *rrddim_add_custom(RRDSET *st, const char *id, const char *name, collecte
348
ml_new_dimension(rd);
349
350
rrdset_unlock(st);
351
+ rrdcontext_updated_rrddim(rd);
352
return(rd);
353
}
354
@@ -352,6 +357,7 @@ RRDDIM *rrddim_add_custom(RRDSET *st, const char *id, const char *name, collecte
357
358
void rrddim_free(RRDSET *st, RRDDIM *rd)
359
{
360
+ rrdcontext_removed_rrddim(rd);
361
ml_delete_dimension(rd);
362
363
debug(D_RRD_CALLS, "rrddim_free() %s.%s", st->name, rd->name);
@@ -446,6 +452,7 @@ int rrddim_hide(RRDSET *st, const char *id) {
452
453
rrddim_flag_set(rd, RRDDIM_FLAG_HIDDEN);
454
rrddim_flag_set(rd, RRDDIM_FLAG_META_HIDDEN);
455
+ rrdcontext_updated_rrddim_flags(rd);
456
return 0;
457
}
458
@@ -463,6 +470,7 @@ int rrddim_unhide(RRDSET *st, const char *id) {
470
471
rrddim_flag_clear(rd, RRDDIM_FLAG_HIDDEN);
472
rrddim_flag_clear(rd, RRDDIM_FLAG_META_HIDDEN);
473
+ rrdcontext_updated_rrddim_flags(rd);
474
return 0;
475
}
476
@@ -475,12 +483,14 @@ inline void rrddim_is_obsolete(RRDSET *st, RRDDIM *rd) {
483
}
484
rrddim_flag_set(rd, RRDDIM_FLAG_OBSOLETE);
485
rrdset_flag_set(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS);
486
+ rrdcontext_updated_rrddim_flags(rd);
487
}
488
489
inline void rrddim_isnot_obsolete(RRDSET *st __maybe_unused, RRDDIM *rd) {
490
debug(D_RRD_CALLS, "rrddim_isnot_obsolete() for chart %s, dimension %s", st->name, rd->name);
491
492
rrddim_flag_clear(rd, RRDDIM_FLAG_OBSOLETE);
493
+ rrdcontext_updated_rrddim_flags(rd);
494
}
495
496
// ----------------------------------------------------------------------------
@@ -489,6 +499,8 @@ inline void rrddim_isnot_obsolete(RRDSET *st __maybe_unused, RRDDIM *rd) {
499
inline collected_number rrddim_set_by_pointer(RRDSET *st __maybe_unused, RRDDIM *rd, collected_number value) {
500
debug(D_RRD_CALLS, "rrddim_set_by_pointer() for chart %s, dimension %s, value " COLLECTED_NUMBER_FORMAT, st->name, rd->name, value);
501
502
+ rrdcontext_collected_rrddim(rd);
503
+
504
now_realtime_timeval(&rd->last_collected_time);
505
rd->collected_value = value;
506
rd->updated = 1;
database/rrdhost.c
+112
-64
@@ -187,7 +187,8 @@ RRDHOST *rrdhost_create(const char *hostname,
187
char *rrdpush_api_key,
188
char *rrdpush_send_charts_matching,
189
struct rrdhost_system_info *system_info,
190
- int is_localhost
190
+ int is_localhost,
191
+ bool archived
192
) {
193
debug(D_RRDHOST, "Host '%s': adding with guid '%s'", hostname, guid);
194
@@ -207,8 +208,7 @@ RRDHOST *rrdhost_create(const char *hostname,
208
host->rrd_history_entries = align_entries_to_pagesize(memory_mode, entries);
209
host->health_enabled = ((memory_mode == RRD_MEMORY_MODE_NONE)) ? 0 : health_enabled;
210
210
- host->sender = mallocz(sizeof(*host->sender));
211
- sender_init(host->sender, host);
211
+ sender_init(host);
212
netdata_mutex_init(&host->receiver_lock);
213
214
host->rrdpush_send_enabled = (rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) ? 1 : 0;
@@ -335,19 +335,21 @@ RRDHOST *rrdhost_create(const char *hostname,
335
336
if(t != host) {
337
error("Host '%s': cannot add host with machine guid '%s' to index. It already exists as host '%s' with machine guid '%s'.", host->hostname, host->machine_guid, t->hostname, t->machine_guid);
338
- rrdhost_free(host);
338
+ rrdhost_free(host, 1);
339
return NULL;
340
}
341
342
if (likely(!uuid_parse(host->machine_guid, host->host_uuid))) {
343
- int rc = sql_store_host(&host->host_uuid, hostname, registry_hostname, update_every, os, timezone, tags,
344
- host->system_info ? host->system_info->hops : 0);
345
- if (unlikely(rc))
346
- error_report("Failed to store machine GUID to the database");
343
+ int rc;
344
+ if (!archived) {
345
+ rc = sql_store_host_info(host);
346
+ if (unlikely(rc))
347
+ error_report("Failed to store machine GUID to the database");
348
+ }
349
sql_load_node_id(host);
350
if (host->health_enabled) {
351
if (!file_is_migrated(host->health_log_filename)) {
350
- int rc = sql_create_health_log_table(host);
352
+ rc = sql_create_health_log_table(host);
353
if (unlikely(rc)) {
354
error_report("Failed to create health log table in the database");
355
health_alarm_log_load(host);
@@ -402,7 +404,7 @@ RRDHOST *rrdhost_create(const char *hostname,
404
error(
405
"Host '%s': cannot initialize host with machine guid '%s'. Failed to initialize DB engine at '%s'.",
406
host->hostname, host->machine_guid, host->cache_dir);
405
- rrdhost_free(host);
407
+ rrdhost_free(host, 1);
408
host = NULL;
409
//rrd_hosts_available++; //TODO: maybe we want this?
410
@@ -447,8 +449,6 @@ RRDHOST *rrdhost_create(const char *hostname,
449
host->system_info->mc_version = enable_metric_correlations ? metric_correlations_version : 0;
450
}
451
450
- ml_new_host(host);
451
-
452
info("Host '%s' (at registry as '%s') with guid '%s' initialized"
453
", os '%s'"
454
", timezone '%s'"
@@ -491,6 +491,11 @@ RRDHOST *rrdhost_create(const char *hostname,
491
492
rrd_hosts_available++;
493
494
+ rrdhost_load_rrdcontext_data(host);
495
+ if (!archived)
496
+ ml_new_host(host);
497
+ else
498
+ rrdhost_flag_set(host, RRDHOST_FLAG_ARCHIVED);
499
return host;
500
}
501
@@ -571,6 +576,14 @@ void rrdhost_update(RRDHOST *host
576
577
if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)) {
578
rrdhost_flag_clear(host, RRDHOST_FLAG_ARCHIVED);
579
+
580
+ host->rrdpush_send_enabled = (rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) ? 1 : 0;
581
+ host->rrdpush_send_destination = (host->rrdpush_send_enabled)?strdupz(rrdpush_destination):NULL;
582
+ if (host->rrdpush_send_destination)
583
+ host->destinations = destinations_init(host->rrdpush_send_destination);
584
+ host->rrdpush_send_api_key = (host->rrdpush_send_enabled)?strdupz(rrdpush_api_key):NULL;
585
+ host->rrdpush_send_charts_matching = simple_pattern_create(rrdpush_send_charts_matching, NULL, SIMPLE_PATTERN_EXACT);
586
+
587
if(host->health_enabled) {
588
int r;
589
char filename[FILENAME_MAX + 1];
@@ -606,6 +619,8 @@ void rrdhost_update(RRDHOST *host
619
}
620
}
621
rrd_hosts_available++;
622
+ ml_new_host(host);
623
+ rrdhost_load_rrdcontext_data(host);
624
info("Host %s is not in archived mode anymore", host->hostname);
625
}
626
@@ -632,6 +647,7 @@ RRDHOST *rrdhost_find_or_create(
647
, char *rrdpush_api_key
648
, char *rrdpush_send_charts_matching
649
, struct rrdhost_system_info *system_info
650
+ , bool archived
651
) {
652
debug(D_RRDHOST, "Searching for host '%s' with guid '%s'", hostname, guid);
653
@@ -641,7 +657,7 @@ RRDHOST *rrdhost_find_or_create(
657
/* If a legacy memory mode instantiates all dbengine state must be discarded to avoid inconsistencies */
658
error("Archived host '%s' has memory mode '%s', but the wanted one is '%s'. Discarding archived state.",
659
host->hostname, rrd_memory_mode_name(host->rrd_memory_mode), rrd_memory_mode_name(mode));
644
- rrdhost_free(host);
660
+ rrdhost_free(host, 1);
661
host = NULL;
662
}
663
if(!host) {
@@ -666,6 +682,7 @@ RRDHOST *rrdhost_find_or_create(
682
, rrdpush_send_charts_matching
683
, system_info
684
, 0
685
+ , archived
686
);
687
}
688
else {
@@ -705,6 +722,7 @@ inline int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, tim
722
if(host != protected_host
723
&& host != localhost
724
&& rrdhost_flag_check(host, RRDHOST_FLAG_ORPHAN)
725
+ && !rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)
726
&& !host->receiver
727
&& host->senders_disconnected_time
728
&& host->senders_disconnected_time + rrdhost_free_orphan_time < now)
@@ -733,7 +751,7 @@ restart_after_removal:
751
else
752
rrdhost_save_charts(host);
753
736
- rrdhost_free(host);
754
+ rrdhost_free(host, 0);
755
goto restart_after_removal;
756
}
757
}
@@ -750,6 +768,10 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info) {
768
info("Skipping SQLITE metadata initialization since memory mode is not dbengine");
769
}
770
771
+ if (unlikely(sql_init_context_database(0))) {
772
+ error_report("Failed to initialize context metadata database");
773
+ }
774
+
775
#ifdef ENABLE_DBENGINE
776
storage_tiers = config_get_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
777
if(storage_tiers < 1) {
@@ -895,6 +917,7 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info) {
917
, default_rrdpush_send_charts_matching
918
, system_info
919
, 1
920
+ , 0
921
);
922
if (unlikely(!localhost)) {
923
rrd_unlock();
@@ -903,8 +926,8 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info) {
926
927
if (likely(system_info))
928
migrate_localhost(&localhost->host_uuid);
906
- sql_aclk_sync_init();
929
rrd_unlock();
930
+ sql_aclk_sync_init();
931
932
web_client_api_v1_management_init();
933
return localhost==NULL;
@@ -988,19 +1011,12 @@ void rrdhost_system_info_free(struct rrdhost_system_info *system_info) {
1011
}
1012
1013
void destroy_receiver_state(struct receiver_state *rpt);
991
-void rrdhost_free(RRDHOST *host) {
992
- if(!host) return;
993
-
994
- info("Freeing all memory for host '%s'...", host->hostname);
1014
996
- rrd_check_wrlock(); // make sure the RRDs are write locked
997
-
998
- rrdhost_wrlock(host);
999
- ml_delete_host(host);
1000
- rrdhost_unlock(host);
1015
+void stop_streaming_sender(RRDHOST *host)
1016
+{
1017
+ if (unlikely(!host->sender))
1018
+ return;
1019
1002
- // ------------------------------------------------------------------------
1003
- // clean up streaming
1020
rrdpush_sender_thread_stop(host); // stop a possibly running thread
1021
cbuffer_free(host->sender->buffer);
1022
buffer_free(host->sender->build);
@@ -1010,40 +1026,45 @@ void rrdhost_free(RRDHOST *host) {
1026
#endif
1027
freez(host->sender);
1028
host->sender = NULL;
1013
- if (netdata_exit) {
1014
- netdata_mutex_lock(&host->receiver_lock);
1015
- if (host->receiver) {
1016
- if (!host->receiver->exited)
1017
- netdata_thread_cancel(host->receiver->thread);
1018
- netdata_mutex_unlock(&host->receiver_lock);
1019
- struct receiver_state *rpt = host->receiver;
1020
- while (host->receiver && !rpt->exited)
1021
- sleep_usec(50 * USEC_PER_MS);
1022
- // If the receiver detached from the host then its thread will destroy the state
1023
- if (host->receiver == rpt)
1024
- destroy_receiver_state(host->receiver);
1025
- }
1026
- else
1027
- netdata_mutex_unlock(&host->receiver_lock);
1028
- }
1029
+}
1030
+
1031
+void stop_streaming_receiver(RRDHOST *host)
1032
+{
1033
+ netdata_mutex_lock(&host->receiver_lock);
1034
+ if (host->receiver) {
1035
+ if (!host->receiver->exited)
1036
+ netdata_thread_cancel(host->receiver->thread);
1037
+ netdata_mutex_unlock(&host->receiver_lock);
1038
+ struct receiver_state *rpt = host->receiver;
1039
+ while (host->receiver && !rpt->exited)
1040
+ sleep_usec(50 * USEC_PER_MS);
1041
+ // If the receiver detached from the host then its thread will destroy the state
1042
+ if (host->receiver == rpt)
1043
+ destroy_receiver_state(host->receiver);
1044
+ } else
1045
+ netdata_mutex_unlock(&host->receiver_lock);
1046
+}
1047
+
1048
+void rrdhost_free(RRDHOST *host, bool force) {
1049
+ if(!host) return;
1050
1051
+ if (netdata_exit || force)
1052
+ info("Freeing all memory for host '%s'...", host->hostname);
1053
+
1054
+ rrd_check_wrlock(); // make sure the RRDs are write locked
1055
+
1056
+ rrdhost_wrlock(host);
1057
+ ml_delete_host(host);
1058
+ rrdhost_unlock(host);
1059
+
1060
+ // ------------------------------------------------------------------------
1061
+ // clean up streaming
1062
+ stop_streaming_sender(host);
1063
1064
+ if (netdata_exit || force)
1065
+ stop_streaming_receiver(host);
1066
1067
rrdhost_wrlock(host); // lock this RRDHOST
1033
-#ifdef ENABLE_ACLK
1034
- struct aclk_database_worker_config *wc = host->dbsync_worker;
1035
- if (wc && !netdata_exit) {
1036
- struct aclk_database_cmd cmd;
1037
- memset(&cmd, 0, sizeof(cmd));
1038
- cmd.opcode = ACLK_DATABASE_ORPHAN_HOST;
1039
- struct aclk_completion compl ;
1040
- init_aclk_completion(&compl );
1041
- cmd.completion = &compl ;
1042
- aclk_database_enq_cmd(wc, &cmd);
1043
- wait_for_aclk_completion(&compl );
1044
- destroy_aclk_completion(&compl );
1045
- }
1046
-#endif
1068
// ------------------------------------------------------------------------
1069
// release its children resources
1070
@@ -1095,13 +1116,34 @@ void rrdhost_free(RRDHOST *host) {
1116
}
1117
#endif
1118
1119
+ if (!netdata_exit && !force) {
1120
+ info("Setting archive mode for host '%s'...", host->hostname);
1121
+ rrdhost_flag_set(host, RRDHOST_FLAG_ARCHIVED);
1122
+ rrdhost_unlock(host);
1123
+ return;
1124
+ }
1125
+
1126
+#ifdef ENABLE_ACLK
1127
+ struct aclk_database_worker_config *wc = host->dbsync_worker;
1128
+ if (wc && !netdata_exit) {
1129
+ struct aclk_database_cmd cmd;
1130
+ memset(&cmd, 0, sizeof(cmd));
1131
+ cmd.opcode = ACLK_DATABASE_ORPHAN_HOST;
1132
+ struct aclk_completion compl ;
1133
+ init_aclk_completion(&compl );
1134
+ cmd.completion = &compl ;
1135
+ aclk_database_enq_cmd(wc, &cmd);
1136
+ wait_for_aclk_completion(&compl );
1137
+ destroy_aclk_completion(&compl );
1138
+ }
1139
+#endif
1140
+
1141
// ------------------------------------------------------------------------
1142
// remove it from the indexes
1143
1144
if(rrdhost_index_del(host) != host)
1145
error("RRDHOST '%s' removed from index, deleted the wrong entry.", host->hostname);
1146
1104
-
1147
// ------------------------------------------------------------------------
1148
// unlink it from the host
1149
@@ -1118,8 +1160,6 @@ void rrdhost_free(RRDHOST *host) {
1160
else error("Request to free RRDHOST '%s': cannot find it", host->hostname);
1161
}
1162
1121
-
1122
-
1163
// ------------------------------------------------------------------------
1164
// free it
1165
@@ -1138,6 +1178,12 @@ void rrdhost_free(RRDHOST *host) {
1178
freez(host->varlib_dir);
1179
freez(host->rrdpush_send_api_key);
1180
freez(host->rrdpush_send_destination);
1181
+ struct rrdpush_destinations *tmp_destination;
1182
+ while (host->destinations) {
1183
+ tmp_destination = host->destinations->next;
1184
+ freez(host->destinations);
1185
+ host->destinations = tmp_destination;
1186
+ }
1187
freez(host->health_default_exec);
1188
freez(host->health_default_recipient);
1189
freez(host->health_log_filename);
@@ -1149,6 +1195,8 @@ void rrdhost_free(RRDHOST *host) {
1195
netdata_rwlock_destroy(&host->rrdhost_rwlock);
1196
freez(host->node_id);
1197
1198
+ rrdhost_destroy_rrdcontexts(host);
1199
+
1200
freez(host);
1201
#ifdef ENABLE_ACLK
1202
if (wc)
@@ -1160,8 +1208,8 @@ void rrdhost_free(RRDHOST *host) {
1208
void rrdhost_free_all(void) {
1209
rrd_wrlock();
1210
/* Make sure child-hosts are released before the localhost. */
1163
- while(localhost->next) rrdhost_free(localhost->next);
1164
- rrdhost_free(localhost);
1211
+ while(localhost->next) rrdhost_free(localhost->next, 1);
1212
+ rrdhost_free(localhost, 1);
1213
rrd_unlock();
1214
}
1215
@@ -1342,7 +1390,7 @@ void rrdhost_delete_charts(RRDHOST *host) {
1390
1391
rrdset_foreach_write(st, host) {
1392
rrdset_rdlock(st);
1345
- rrdset_delete(st);
1393
+ rrdset_delete_files(st);
1394
rrdset_unlock(st);
1395
}
1396
@@ -1370,7 +1418,7 @@ void rrdhost_cleanup_charts(RRDHOST *host) {
1418
rrdset_rdlock(st);
1419
1420
if(rrdhost_delete_obsolete_charts && rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE))
1373
- rrdset_delete(st);
1421
+ rrdset_delete_files(st);
1422
else if(rrdhost_delete_obsolete_charts && rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE_DIMENSIONS))
1423
rrdset_delete_obsolete_dimensions(st);
1424
else
@@ -1518,7 +1566,7 @@ restart_after_removal:
1566
rrdset_rdlock(st);
1567
1568
if(rrdhost_delete_obsolete_charts)
1521
- rrdset_delete(st);
1569
+ rrdset_delete_files(st);
1570
else
1571
rrdset_save(st);
1572
database/rrdlabels.c
+53
-41
@@ -369,11 +369,12 @@ __attribute__((constructor)) void initialize_labels_keys_char_map(void) {
369
370
}
371
372
-static size_t rrdlabels_sanitize(unsigned char *dst, const unsigned char *src, size_t dst_size, unsigned char *char_map, bool utf) {
372
+static size_t rrdlabels_sanitize(unsigned char *dst, const unsigned char *src, size_t dst_size, unsigned char *char_map, bool utf, const char *empty) {
373
if(unlikely(!dst_size)) return 0;
374
if(unlikely(!src || !*src)) {
375
- *dst = '\0';
376
- return 0;
375
+ strncpyz((char *)dst, empty, dst_size);
376
+ dst[dst_size - 1] = '\0';
377
+ return strlen((char *)dst);
378
}
379
380
unsigned char *d = dst;
@@ -447,27 +448,33 @@ static size_t rrdlabels_sanitize(unsigned char *dst, const unsigned char *src, s
448
// check if dst is all underscores and empty it if it is
449
d = dst;
450
while(*d == '_') d++;
450
- if(!*d) {
451
+ if(unlikely(*d == '\0')) {
452
*dst = '\0';
453
mblen = 0;
454
}
455
456
+ if(unlikely(*dst == '\0')) {
457
+ strncpyz((char *)dst, empty, dst_size);
458
+ dst[dst_size - 1] = '\0';
459
+ return strlen((char *)dst);
460
+ }
461
+
462
return mblen;
463
}
464
465
static inline size_t rrdlabels_sanitize_name(char *dst, const char *src, size_t dst_size) {
459
- return rrdlabels_sanitize((unsigned char *)dst, (const unsigned char *)src, dst_size, label_names_char_map, 0);
466
+ return rrdlabels_sanitize((unsigned char *)dst, (const unsigned char *)src, dst_size, label_names_char_map, 0, "");
467
}
468
469
static inline size_t rrdlabels_sanitize_value(char *dst, const char *src, size_t dst_size) {
463
- return rrdlabels_sanitize((unsigned char *)dst, (const unsigned char *)src, dst_size, label_values_char_map, 1);
470
+ return rrdlabels_sanitize((unsigned char *)dst, (const unsigned char *)src, dst_size, label_values_char_map, 1, "[none]");
471
}
472
473
// ----------------------------------------------------------------------------
474
// rrdlabels_create()
475
476
typedef struct rrdlabel {
470
- const char *value;
477
+ STRING *label_value;
478
RRDLABEL_SRC label_source;
479
} RRDLABEL;
480
@@ -476,9 +483,9 @@ static void rrdlabel_insert_callback(const char *name, void *value, void *data)
483
DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
484
RRDLABEL *lb = (RRDLABEL *)value;
485
479
- // allocate our own memory for the value
480
- lb->value = strdupz(lb->value);
481
- lb->label_source |= RRDLABEL_FLAG_NEW;
486
+ // label_value is already allocated by the STRING
487
+ lb->label_source |= RRDLABEL_FLAG_NEW;
488
+ lb->label_source &= ~RRDLABEL_FLAG_OLD;
489
}
490
491
static void rrdlabel_delete_callback(const char *name, void *value, void *data) {
@@ -486,8 +493,8 @@ static void rrdlabel_delete_callback(const char *name, void *value, void *data)
493
DICTIONARY *dict = (DICTIONARY *)data; (void)dict;
494
RRDLABEL *lb = (RRDLABEL *)value;
495
489
- freez((void *)lb->value);
490
- lb->value = NULL;
496
+ string_freez(lb->label_value);
497
+ lb->label_value = NULL;
498
}
499
500
static void rrdlabel_conflict_callback(const char *name, void *oldvalue, void *newvalue, void *data) {
@@ -496,17 +503,22 @@ static void rrdlabel_conflict_callback(const char *name, void *oldvalue, void *n
503
RRDLABEL *lbold = (RRDLABEL *)oldvalue;
504
RRDLABEL *lbnew = (RRDLABEL *)newvalue;
505
499
- if(strcmp(lbold->value, lbnew->value) == 0) {
506
+ if(lbold->label_value == lbnew->label_value || strcmp(string2str(lbold->label_value), string2str(lbnew->label_value)) == 0) {
507
// they are the same
501
- lbold->label_source |= lbnew->label_source;
502
- lbold->label_source |= RRDLABEL_FLAG_OLD;
508
+ lbold->label_source |= lbnew->label_source;
509
+ lbold->label_source |= RRDLABEL_FLAG_OLD;
510
+ lbold->label_source &= ~RRDLABEL_FLAG_NEW;
511
+
512
+ // free the new one
513
+ string_freez(lbnew->label_value);
514
}
515
else {
516
// they are different
506
- freez((void *)lbold->value);
507
- lbold->value = strdupz(lbnew->value);
508
- lbold->label_source = lbnew->label_source;
509
- lbold->label_source |= RRDLABEL_FLAG_NEW;
517
+ string_freez(lbold->label_value);
518
+ lbold->label_value = lbnew->label_value;
519
+ lbold->label_source = lbnew->label_source;
520
+ lbold->label_source |= RRDLABEL_FLAG_NEW;
521
+ lbold->label_source &= ~RRDLABEL_FLAG_OLD;
522
}
523
}
524
@@ -536,7 +548,7 @@ static void labels_add_already_sanitized(DICTIONARY *dict, const char *key, cons
548
549
RRDLABEL tmp = {
550
.label_source = ls,
539
- .value = value
551
+ .label_value = string_strdupz(value)
552
};
553
dictionary_set(dict, key, &tmp, sizeof(RRDLABEL));
554
}
@@ -611,11 +623,11 @@ void rrdlabels_add_pair(DICTIONARY *dict, const char *string, RRDLABEL_SRC ls) {
623
// rrdlabels_get_to_buffer_or_null()
624
625
void rrdlabels_get_value_to_buffer_or_null(DICTIONARY *labels, BUFFER *wb, const char *key, const char *quote, const char *null) {
614
- void *acquired_item = dictionary_acquire_item(labels, key);
615
- RRDLABEL *lb = dictionary_acquired_item_value(labels, acquired_item);
626
+ DICTIONARY_ITEM *acquired_item = dictionary_get_and_acquire_item(labels, key);
627
+ RRDLABEL *lb = dictionary_acquired_item_value(acquired_item);
628
617
- if(lb && lb->value)
618
- buffer_sprintf(wb, "%s%s%s", quote, lb->value, quote);
629
+ if(lb && lb->label_value)
630
+ buffer_sprintf(wb, "%s%s%s", quote, string2str(lb->label_value), quote);
631
else
632
buffer_strcat(wb, null);
633
@@ -652,7 +664,7 @@ static int remove_not_old_not_new_callback(const char *name, void *value, void *
664
DICTIONARY *dict = (DICTIONARY *)data;
665
RRDLABEL *lb = (RRDLABEL *)value;
666
655
- if(!(lb->label_source & RRDLABEL_FLAG_OLD) && !(lb->label_source & RRDLABEL_FLAG_NEW)) {
667
+ if(!(lb->label_source & (RRDLABEL_FLAG_OLD | RRDLABEL_FLAG_NEW | RRDLABEL_FLAG_PERMANENT))) {
668
dictionary_del_having_write_lock(dict, name);
669
return 1;
670
}
@@ -681,7 +693,7 @@ static int labels_walkthrough_callback(const char *name, void *value, void *data
693
if(ls & RRDLABEL_FLAG_NEW) ls &= ~RRDLABEL_FLAG_NEW;
694
if(ls & RRDLABEL_FLAG_OLD) ls &= ~RRDLABEL_FLAG_OLD;
695
684
- return d->callback(name, lb->value, ls, d->data);
696
+ return d->callback(name, string2str(lb->label_value), ls, d->data);
697
}
698
699
int rrdlabels_walkthrough_read(DICTIONARY *labels, int (*callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *data) {
@@ -708,7 +720,7 @@ int rrdlabels_sorted_walkthrough_read(DICTIONARY *labels, int (*callback)(const
720
static int copy_label_to_dictionary_callback(const char *name, void *value, void *data) {
721
DICTIONARY *dst = (DICTIONARY *)data;
722
RRDLABEL *lb = (RRDLABEL *)value;
711
- labels_add_already_sanitized(dst, name, lb->value, lb->label_source);
723
+ labels_add_already_sanitized(dst, name, string2str(lb->label_value), lb->label_source);
724
return 1;
725
}
726
@@ -761,7 +773,7 @@ static int simple_pattern_match_name_and_value_callback(const char *name, void *
773
774
size_t len = RRDLABELS_MAX_NAME_LENGTH + RRDLABELS_MAX_VALUE_LENGTH + 2; // +1 for =, +1 for \0
775
char tmp[len], *dst = &tmp[0];
764
- const char *v = lb->value;
776
+ const char *v = string2str(lb->label_value);
777
778
// copy the name
779
while(*name) *dst++ = *name++;
@@ -822,7 +834,7 @@ static int rrdlabels_log_label_to_buffer_callback(const char *name, void *value,
834
BUFFER *wb = (BUFFER *)data;
835
RRDLABEL *lb = (RRDLABEL *)value;
836
825
- buffer_sprintf(wb, "Label: %s: \"%s\" (", name, lb->value);
837
+ buffer_sprintf(wb, "Label: %s: \"%s\" (", name, string2str(lb->label_value));
838
839
size_t sources = 0;
840
if(lb->label_source & RRDLABEL_SRC_AUTO) {
@@ -878,7 +890,7 @@ static int label_to_buffer_callback(const char *name, void *value, void *data) {
890
char n[n_size];
891
char v[v_size];
892
881
- const char *nn = name, *vv = lb->value;
893
+ const char *nn = name, *vv = string2str(lb->label_value);
894
895
if(t->name_sanitizer) {
896
t->name_sanitizer(n, name, n_size);
@@ -886,11 +898,11 @@ static int label_to_buffer_callback(const char *name, void *value, void *data) {
898
}
899
900
if(t->value_sanitizer) {
889
- t->value_sanitizer(v, lb->value, v_size);
901
+ t->value_sanitizer(v, string2str(lb->label_value), v_size);
902
vv = v;
903
}
904
893
- if(!t->filter_callback || (t->filter_callback && t->filter_callback(name, lb->value, lb->label_source, t->filter_data))) {
905
+ if(!t->filter_callback || t->filter_callback(name, string2str(lb->label_value), lb->label_source, t->filter_data)) {
906
buffer_sprintf(t->wb, "%s%s%s%s%s%s%s%s%s", t->count++?t->between_them:"", t->before_each, t->quote, nn, t->quote, t->equal, t->quote, vv, t->quote);
907
return 1;
908
}
@@ -898,7 +910,7 @@ static int label_to_buffer_callback(const char *name, void *value, void *data) {
910
return 0;
911
}
912
901
-void rrdlabels_to_buffer(DICTIONARY *labels, BUFFER *wb, const char *before_each, const char *equal, const char *quote, const char *between_them, bool (*filter_callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *filter_data, void (*name_sanitizer)(char *dst, const char *src, size_t dst_size), void (*value_sanitizer)(char *dst, const char *src, size_t dst_size)) {
913
+int rrdlabels_to_buffer(DICTIONARY *labels, BUFFER *wb, const char *before_each, const char *equal, const char *quote, const char *between_them, bool (*filter_callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *filter_data, void (*name_sanitizer)(char *dst, const char *src, size_t dst_size), void (*value_sanitizer)(char *dst, const char *src, size_t dst_size)) {
914
struct labels_to_buffer tmp = {
915
.wb = wb,
916
.filter_callback = filter_callback,
@@ -911,7 +923,7 @@ void rrdlabels_to_buffer(DICTIONARY *labels, BUFFER *wb, const char *before_each
923
.between_them = between_them,
924
.count = 0
925
};
914
- dictionary_sorted_walkthrough_read(labels, label_to_buffer_callback, (void *)&tmp);
926
+ return dictionary_walkthrough_read(labels, label_to_buffer_callback, (void *)&tmp);
927
}
928
929
static int chart_label_store_to_sql_callback(const char *name, const char *value, RRDLABEL_SRC ls, void *data) {
@@ -1024,12 +1036,12 @@ int rrdlabels_unittest_add_pairs() {
1036
errors += rrdlabels_unittest_add_a_pair("tag:a", "tag", "a");
1037
1038
// test empty values
1027
- errors += rrdlabels_unittest_add_a_pair("tag", "tag", "");
1028
- errors += rrdlabels_unittest_add_a_pair("tag:", "tag", "");
1029
- errors += rrdlabels_unittest_add_a_pair("tag:\"\"", "tag", "");
1030
- errors += rrdlabels_unittest_add_a_pair("tag:''", "tag", "");
1031
- errors += rrdlabels_unittest_add_a_pair("tag:\r\n", "tag", "");
1032
- errors += rrdlabels_unittest_add_a_pair("tag\r\n", "tag", "");
1039
+ errors += rrdlabels_unittest_add_a_pair("tag", "tag", "[none]");
1040
+ errors += rrdlabels_unittest_add_a_pair("tag:", "tag", "[none]");
1041
+ errors += rrdlabels_unittest_add_a_pair("tag:\"\"", "tag", "[none]");
1042
+ errors += rrdlabels_unittest_add_a_pair("tag:''", "tag", "[none]");
1043
+ errors += rrdlabels_unittest_add_a_pair("tag:\r\n", "tag", "[none]");
1044
+ errors += rrdlabels_unittest_add_a_pair("tag\r\n", "tag", "[none]");
1045
1046
// test UTF-8 in values
1047
errors += rrdlabels_unittest_add_a_pair("tag: country:Ελλάδα", "tag", "country:Ελλάδα");
@@ -1107,7 +1119,7 @@ int rrdlabels_unittest_sanitize_value(const char *src, const char *expected) {
1119
int rrdlabels_unittest_sanitization() {
1120
int errors = 0;
1121
1110
- errors += rrdlabels_unittest_sanitize_value("", "");
1122
+ errors += rrdlabels_unittest_sanitize_value("", "[none]");
1123
errors += rrdlabels_unittest_sanitize_value("1", "1");
1124
errors += rrdlabels_unittest_sanitize_value(" hello world ", "hello world");
1125
database/rrdset.c
+24
-2
@@ -186,6 +186,7 @@ int rrdset_set_name(RRDSET *st, const char *name) {
186
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_IGNORE);
187
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
188
189
+ rrdcontext_updated_rrdset_name(st);
190
return 2;
191
}
192
@@ -204,6 +205,7 @@ inline void rrdset_is_obsolete(RRDSET *st) {
205
// the chart will not get more updates (data collection)
206
// so, we have to push its definition now
207
rrdset_push_chart_definition_now(st);
208
+ rrdcontext_updated_rrdset_flags(st);
209
}
210
}
211
@@ -216,6 +218,7 @@ inline void rrdset_isnot_obsolete(RRDSET *st) {
218
219
// the chart will be pushed upstream automatically
220
// due to data collection
221
+ rrdcontext_updated_rrdset_flags(st);
222
}
223
}
224
@@ -251,6 +254,7 @@ inline void rrdset_update_heterogeneous_flag(RRDSET *st) {
254
}
255
256
rrdset_flag_clear(st, RRDSET_FLAG_HETEROGENEOUS);
257
+ rrdcontext_updated_rrdset_flags(st);
258
}
259
260
// ----------------------------------------------------------------------------
@@ -383,6 +387,9 @@ void rrdset_free(RRDSET *st) {
387
388
rrdset_unlock(st);
389
390
+ // this has to be after the dimensions are freed
391
+ rrdcontext_removed_rrdset(st);
392
+
393
// ------------------------------------------------------------------------
394
// free it
395
@@ -419,7 +426,7 @@ void rrdset_save(RRDSET *st) {
426
rrddim_memory_file_save(rd);
427
}
428
422
-void rrdset_delete(RRDSET *st) {
429
+void rrdset_delete_files(RRDSET *st) {
430
RRDDIM *rd;
431
rrdset_check_rdlock(st);
432
@@ -480,6 +487,14 @@ static inline RRDSET *rrdset_find_on_create(RRDHOST *host, const char *fullid) {
487
return NULL;
488
}
489
490
+static inline void rrdset_update_permanent_labels(RRDSET *st) {
491
+ if(!st->state || !st->state->chart_labels) return;
492
+
493
+ rrdlabels_add(st->state->chart_labels, "_collect_plugin", st->plugin_name, RRDLABEL_SRC_AUTO| RRDLABEL_FLAG_PERMANENT);
494
+ rrdlabels_add(st->state->chart_labels, "_collect_module", st->module_name, RRDLABEL_SRC_AUTO| RRDLABEL_FLAG_PERMANENT);
495
+ rrdlabels_add(st->state->chart_labels, "_instance_family", st->family, RRDLABEL_SRC_AUTO| RRDLABEL_FLAG_PERMANENT);
496
+}
497
+
498
RRDSET *rrdset_create_custom(
499
RRDHOST *host
500
, const char *type
@@ -659,8 +674,11 @@ RRDSET *rrdset_create_custom(
674
}
675
}
676
/* Fall-through during switch from archived to active so that the host lock is taken and health is linked */
662
- if (!changed_from_archived_to_active)
677
+ if (!changed_from_archived_to_active) {
678
+ rrdset_update_permanent_labels(st);
679
+ rrdcontext_updated_rrdset(st);
680
return st;
681
+ }
682
}
683
684
rrdhost_wrlock(host);
@@ -680,6 +698,7 @@ RRDSET *rrdset_create_custom(
698
rrdhost_unlock(host);
699
rrdset_flag_set(st, RRDSET_FLAG_SYNC_CLOCK);
700
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
701
+ rrdcontext_updated_rrdset(st);
702
return st;
703
}
704
@@ -748,6 +767,7 @@ RRDSET *rrdset_create_custom(
767
768
netdata_rwlock_init(&st->rrdset_rwlock);
769
st->state->chart_labels = rrdlabels_create();
770
+ rrdset_update_permanent_labels(st);
771
772
if(name && *name && rrdset_set_name(st, name))
773
// we did set the name
@@ -789,6 +809,7 @@ RRDSET *rrdset_create_custom(
809
compute_chart_hash(st);
810
811
rrdhost_unlock(host);
812
+ rrdcontext_updated_rrdset(st);
813
return(st);
814
}
815
@@ -1280,6 +1301,7 @@ void rrdset_done(RRDSET *st) {
1301
if(unlikely(netdata_exit)) return;
1302
1303
debug(D_RRD_CALLS, "rrdset_done() for chart %s", st->name);
1304
+ rrdcontext_collected_rrdset(st);
1305
1306
RRDDIM *rd;
1307
database/sqlite/sqlite_aclk.c
+92
-3
@@ -258,6 +258,68 @@ void aclk_sync_exit_all()
258
uv_mutex_unlock(&aclk_async_lock);
259
}
260
261
+#ifdef ENABLE_ACLK
262
+enum {
263
+ IDX_HOST_ID,
264
+ IDX_HOSTNAME,
265
+ IDX_REGISTRY,
266
+ IDX_UPDATE_EVERY,
267
+ IDX_OS,
268
+ IDX_TIMEZONE,
269
+ IDX_TAGS,
270
+ IDX_HOPS,
271
+ IDX_MEMORY_MODE,
272
+ IDX_ABBREV_TIMEZONE,
273
+ IDX_UTC_OFFSET,
274
+ IDX_PROGRAM_NAME,
275
+ IDX_PROGRAM_VERSION,
276
+ IDX_ENTRIES,
277
+ IDX_HEALTH_ENABLED,
278
+};
279
+
280
+static int create_host_callback(void *data, int argc, char **argv, char **column)
281
+{
282
+ UNUSED(data);
283
+ UNUSED(argc);
284
+ UNUSED(column);
285
+
286
+ char guid[UUID_STR_LEN];
287
+ uuid_unparse_lower(*(uuid_t *)argv[IDX_HOST_ID], guid);
288
+
289
+ struct rrdhost_system_info *system_info = callocz(1, sizeof(struct rrdhost_system_info));
290
+ system_info->hops = str2i((const char *) argv[IDX_HOPS]);
291
+
292
+ sql_build_host_system_info((uuid_t *)argv[IDX_HOST_ID], system_info);
293
+
294
+ RRDHOST *host = rrdhost_find_or_create(
295
+ (const char *) argv[IDX_HOSTNAME]
296
+ , (const char *) argv[IDX_REGISTRY]
297
+ , guid
298
+ , (const char *) argv[IDX_OS]
299
+ , (const char *) argv[IDX_TIMEZONE]
300
+ , (const char *) argv[IDX_ABBREV_TIMEZONE]
301
+ , argv[IDX_UTC_OFFSET] ? str2uint32_t(argv[IDX_UTC_OFFSET]) : 0
302
+ , (const char *) argv[IDX_TAGS]
303
+ , (const char *) (argv[IDX_PROGRAM_NAME] ? argv[IDX_PROGRAM_NAME] : "unknown")
304
+ , (const char *) (argv[IDX_PROGRAM_VERSION] ? argv[IDX_PROGRAM_VERSION] : "unknown")
305
+ , argv[3] ? str2i(argv[IDX_UPDATE_EVERY]) : 1
306
+ , argv[13] ? str2i(argv[IDX_ENTRIES]) : 0
307
+ , RRD_MEMORY_MODE_DBENGINE
308
+ , 0 // health
309
+ , 0 // rrdpush enabled
310
+ , NULL //destination
311
+ , NULL // api key
312
+ , NULL // send charts matching
313
+ , system_info
314
+ , 1
315
+ );
316
+ char node_str[UUID_STR_LEN] = "<none>";
317
+ uuid_unparse_lower(*host->node_id, node_str);
318
+ internal_error(true, "Adding archived host \"%s\" with GUID \"%s\" node id = \"%s\"", host->hostname, host->machine_guid, node_str);
319
+ return 0;
320
+}
321
+#endif
322
+
323
int aclk_start_sync_thread(void *data, int argc, char **argv, char **column)
324
{
325
char uuid_str[GUID_LEN + 1];
@@ -267,10 +329,11 @@ int aclk_start_sync_thread(void *data, int argc, char **argv, char **column)
329
330
uuid_unparse_lower(*((uuid_t *) argv[0]), uuid_str);
331
270
- if (rrdhost_find_by_guid(uuid_str, 0) == localhost)
332
+ RRDHOST *host = rrdhost_find_by_guid(uuid_str, 0);
333
+ if (host == localhost)
334
return 0;
335
273
- sql_create_aclk_table(NULL, (uuid_t *) argv[0], (uuid_t *) argv[1]);
336
+ sql_create_aclk_table(host, (uuid_t *) argv[0], (uuid_t *) argv[1]);
337
return 0;
338
}
339
@@ -303,8 +366,23 @@ void sql_aclk_sync_init(void)
366
info("SQLite aclk sync initialization completed");
367
fatal_assert(0 == uv_mutex_init(&aclk_async_lock));
368
369
+ if (likely(rrdcontext_enabled == CONFIG_BOOLEAN_YES)) {
370
+ rc = sqlite3_exec(db_meta, "SELECT host_id, hostname, registry_hostname, update_every, os, "
371
+ "timezone, tags, hops, memory_mode, abbrev_timezone, utc_offset, program_name, "
372
+ "program_version, entries, health_enabled FROM host WHERE hops >0;",
373
+ create_host_callback, NULL, &err_msg);
374
+ if (rc != SQLITE_OK) {
375
+ error_report("SQLite error when loading archived hosts, rc = %d (%s)", rc, err_msg);
376
+ sqlite3_free(err_msg);
377
+ }
378
+ }
379
+
380
rc = sqlite3_exec(db_meta, "SELECT ni.host_id, ni.node_id FROM host h, node_instance ni WHERE "
307
- "h.host_id = ni.host_id AND ni.node_id IS NOT NULL;", aclk_start_sync_thread, NULL, NULL);
381
+ "h.host_id = ni.host_id AND ni.node_id IS NOT NULL;", aclk_start_sync_thread, NULL, &err_msg);
382
+ if (rc != SQLITE_OK) {
383
+ error_report("SQLite error when starting ACLK sync threads, rc = %d (%s)", rc, err_msg);
384
+ sqlite3_free(err_msg);
385
+ }
386
#endif
387
return;
388
}
@@ -803,6 +881,17 @@ void sql_maint_aclk_sync_database(struct aclk_database_worker_config *wc, struct
881
buffer_sprintf(sql,"DELETE FROM aclk_alert_%s WHERE date_submitted IS NOT NULL AND "
882
"date_cloud_ack < unixepoch()-%d;", wc->uuid_str, ACLK_DELETE_ACK_ALERTS_INTERNAL);
883
db_execute(buffer_tostring(sql));
884
+ buffer_flush(sql);
885
+
886
+ buffer_sprintf(sql,"UPDATE aclk_chart_%s SET status = NULL, date_submitted=unixepoch() WHERE "
887
+ "date_submitted IS NULL AND date_created < unixepoch()-%d;", wc->uuid_str, ACLK_AUTO_MARK_SUBMIT_INTERVAL);
888
+ db_execute(buffer_tostring(sql));
889
+ buffer_flush(sql);
890
+
891
+ buffer_sprintf(sql,"UPDATE aclk_chart_%s SET date_updated = unixepoch() WHERE date_updated IS NULL"
892
+ " AND date_submitted IS NOT NULL AND date_submitted < unixepoch()-%d;",
893
+ wc->uuid_str, ACLK_AUTO_MARK_UPDATED_INTERVAL);
894
+ db_execute(buffer_tostring(sql));
895
896
buffer_free(sql);
897
return;
database/sqlite/sqlite_aclk.h
+2
@@ -22,6 +22,8 @@
22
#define ACLK_DATABASE_ROTATION_INTERVAL (3600)
23
#define ACLK_DELETE_ACK_INTERNAL (600)
24
#define ACLK_DELETE_ACK_ALERTS_INTERNAL (86400)
25
+#define ACLK_AUTO_MARK_SUBMIT_INTERVAL (3600)
26
+#define ACLK_AUTO_MARK_UPDATED_INTERVAL (1800)
27
#define ACLK_SYNC_QUERY_SIZE 512
28
29
struct aclk_completion {
database/sqlite/sqlite_aclk_alert.c
+3
-3
@@ -224,7 +224,7 @@ void aclk_push_alert_event(struct aclk_database_worker_config *wc, struct aclk_d
224
return;
225
}
226
227
- char *claim_id = is_agent_claimed();
227
+ char *claim_id = get_agent_claimid();
228
if (unlikely(!claim_id))
229
return;
230
@@ -449,7 +449,7 @@ void aclk_push_alarm_health_log(struct aclk_database_worker_config *wc, struct a
449
#else
450
int rc;
451
452
- char *claim_id = is_agent_claimed();
452
+ char *claim_id = get_agent_claimid();
453
if (unlikely(!claim_id))
454
return;
455
@@ -911,7 +911,7 @@ void aclk_push_alert_snapshot_event(struct aclk_database_worker_config *wc, stru
911
if (unlikely(!wc->alerts_snapshot_id))
912
return;
913
914
- char *claim_id = is_agent_claimed();
914
+ char *claim_id = get_agent_claimid();
915
if (unlikely(!claim_id))
916
return;
917
database/sqlite/sqlite_aclk_chart.c
+22
-8
@@ -143,7 +143,7 @@ int aclk_add_chart_event(struct aclk_database_worker_config *wc, struct aclk_dat
143
int rc = 0;
144
CHECK_SQLITE_CONNECTION(db_meta);
145
146
- char *claim_id = is_agent_claimed();
146
+ char *claim_id = get_agent_claimid();
147
148
RRDSET *st = cmd.data;
149
@@ -222,7 +222,7 @@ void aclk_process_dimension_deletion(struct aclk_database_worker_config *wc, str
222
if (uuid_parse(wc->host_guid, host_id))
223
return;
224
225
- char *claim_id = is_agent_claimed();
225
+ char *claim_id = get_agent_claimid();
226
if (!claim_id)
227
return;
228
@@ -281,7 +281,7 @@ int aclk_add_dimension_event(struct aclk_database_worker_config *wc, struct aclk
281
282
struct aclk_chart_dimension_data *aclk_cd_data = cmd.data;
283
284
- char *claim_id = is_agent_claimed();
284
+ char *claim_id = get_agent_claimid();
285
if (!claim_id)
286
goto cleanup;
287
@@ -308,7 +308,7 @@ void aclk_send_chart_event(struct aclk_database_worker_config *wc, struct aclk_d
308
return;
309
}
310
311
- char *claim_id = is_agent_claimed();
311
+ char *claim_id = get_agent_claimid();
312
if (unlikely(!claim_id))
313
return;
314
@@ -736,7 +736,7 @@ void aclk_start_streaming(char *node_id, uint64_t sequence_id, time_t created_at
736
wc->chart_reset_count);
737
738
chart_reset_t chart_reset;
739
- chart_reset.claim_id = is_agent_claimed();
739
+ chart_reset.claim_id = get_agent_claimid();
740
if (chart_reset.claim_id) {
741
chart_reset.node_id = node_id;
742
chart_reset.reason = SEQ_ID_NOT_EXISTS;
@@ -825,7 +825,12 @@ void aclk_update_retention(struct aclk_database_worker_config *wc)
825
if (!aclk_connected)
826
return;
827
828
- char *claim_id = is_agent_claimed();
828
+ if (wc->host && rrdhost_flag_check(wc->host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)) {
829
+ internal_error(true, "Skipping aclk_update_retention for host %s because context streaming is enabled", wc->host->hostname);
830
+ return;
831
+ }
832
+
833
+ char *claim_id = get_agent_claimid();
834
if (unlikely(!claim_id))
835
return;
836
@@ -1060,6 +1065,10 @@ void sql_get_last_chart_sequence(struct aclk_database_worker_config *wc)
1065
1066
void queue_dimension_to_aclk(RRDDIM *rd, time_t last_updated)
1067
{
1068
+ RRDHOST *host = rd->rrdset->rrdhost;
1069
+ if (likely(rrdhost_flag_check(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)))
1070
+ return;
1071
+
1072
int live = !last_updated;
1073
1074
if (likely(rd->aclk_live_status == live))
@@ -1076,7 +1085,7 @@ void queue_dimension_to_aclk(RRDDIM *rd, time_t last_updated)
1085
if (unlikely(!wc))
1086
return;
1087
1079
- char *claim_id = is_agent_claimed();
1088
+ char *claim_id = get_agent_claimid();
1089
if (unlikely(!claim_id))
1090
return;
1091
@@ -1120,7 +1129,7 @@ void queue_dimension_to_aclk(RRDDIM *rd, time_t last_updated)
1129
1130
void aclk_send_dimension_update(RRDDIM *rd)
1131
{
1123
- char *claim_id = is_agent_claimed();
1132
+ char *claim_id = get_agent_claimid();
1133
if (unlikely(!claim_id))
1134
return;
1135
@@ -1290,6 +1299,11 @@ void sql_check_chart_liveness(RRDSET *st) {
1299
// ST is read locked
1300
int queue_chart_to_aclk(RRDSET *st)
1301
{
1302
+ RRDHOST *host = st->rrdhost;
1303
+
1304
+ if (likely(rrdhost_flag_check(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)))
1305
+ return 0;
1306
+
1307
return sql_queue_chart_payload((struct aclk_database_worker_config *) st->rrdhost->dbsync_worker,
1308
st, ACLK_DATABASE_ADD_CHART);
1309
}
database/sqlite/sqlite_aclk_node.c
+3
-2
@@ -39,7 +39,7 @@ void sql_build_node_collectors(struct aclk_database_worker_config *wc)
39
DICTIONARY *dict = dictionary_create(DICTIONARY_FLAG_SINGLE_THREADED);
40
41
upd_node_collectors.node_id = wc->node_id;
42
- upd_node_collectors.claim_id = is_agent_claimed();
42
+ upd_node_collectors.claim_id = get_agent_claimid();
43
44
upd_node_collectors.node_collectors = collectors_from_charts(wc->host, dict);
45
aclk_update_node_collectors(&upd_node_collectors);
@@ -68,7 +68,7 @@ void sql_build_node_info(struct aclk_database_worker_config *wc, struct aclk_dat
68
69
rrd_rdlock();
70
node_info.node_id = wc->node_id;
71
- node_info.claim_id = is_agent_claimed();
71
+ node_info.claim_id = get_agent_claimid();
72
node_info.machine_guid = wc->host_guid;
73
node_info.child = (wc->host != localhost);
74
node_info.ml_info.ml_capable = ml_capable(localhost);
@@ -78,6 +78,7 @@ void sql_build_node_info(struct aclk_database_worker_config *wc, struct aclk_dat
78
{ .name = "proto", .version = 1, .enabled = 1 },
79
{ .name = "ml", .version = ml_capable(localhost), .enabled = ml_enabled(wc->host) },
80
{ .name = "mc", .version = enable_metric_correlations ? metric_correlations_version : 0, .enabled = enable_metric_correlations },
81
+ { .name = "ctx", .version = 1, .enabled = rrdcontext_enabled},
82
{ .name = NULL, .version = 0, .enabled = 0 }
83
};
84
node_info.node_instance_capabilities = instance_caps;
database/sqlite/sqlite_context.c
new
+598
@@ -0,0 +1,598 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "sqlite_functions.h"
4
+#include "sqlite_context.h"
5
+#include "sqlite_db_migration.h"
6
+
7
+#define DB_CONTEXT_METADATA_VERSION 1
8
+
9
+const char *database_context_config[] = {
10
+ "CREATE TABLE IF NOT EXISTS context (host_id BLOB, id TEXT NOT NULL, version INT NOT NULL, title TEXT NOT NULL, " \
11
+ "chart_type TEXT NOT NULL, unit TEXT NOT NULL, priority INT NOT NULL, first_time_t INT NOT NULL, "
12
+ "last_time_t INT NOT NULL, deleted INT NOT NULL, "
13
+ "family TEXT, PRIMARY KEY (host_id, id));",
14
+
15
+ NULL
16
+};
17
+
18
+const char *database_context_cleanup[] = {
19
+ "VACUUM;",
20
+ NULL
21
+};
22
+
23
+sqlite3 *db_context_meta = NULL;
24
+
25
+/*
26
+ * Initialize the SQLite database
27
+ * Return 0 on success
28
+ */
29
+int sql_init_context_database(int memory)
30
+{
31
+ char sqlite_database[FILENAME_MAX + 1];
32
+ int rc;
33
+
34
+ if (likely(!memory))
35
+ snprintfz(sqlite_database, FILENAME_MAX, "%s/context-meta.db", netdata_configured_cache_dir);
36
+ else
37
+ strcpy(sqlite_database, ":memory:");
38
+
39
+ rc = sqlite3_open(sqlite_database, &db_context_meta);
40
+ if (rc != SQLITE_OK) {
41
+ error_report("Failed to initialize database at %s, due to \"%s\"", sqlite_database, sqlite3_errstr(rc));
42
+ sqlite3_close(db_context_meta);
43
+ db_context_meta = NULL;
44
+ return 1;
45
+ }
46
+
47
+ info("SQLite database %s initialization", sqlite_database);
48
+
49
+ char buf[1024 + 1] = "";
50
+ const char *list[2] = { buf, NULL };
51
+
52
+ int target_version = DB_CONTEXT_METADATA_VERSION;
53
+ if (likely(!memory))
54
+ target_version = perform_context_database_migration(db_context_meta, DB_CONTEXT_METADATA_VERSION);
55
+
56
+ // https://www.sqlite.org/pragma.html#pragma_auto_vacuum
57
+ // PRAGMA schema.auto_vacuum = 0 | NONE | 1 | FULL | 2 | INCREMENTAL;
58
+ snprintfz(buf, 1024, "PRAGMA auto_vacuum=%s;", config_get(CONFIG_SECTION_SQLITE, "auto vacuum", "INCREMENTAL"));
59
+ if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
60
+
61
+ // https://www.sqlite.org/pragma.html#pragma_synchronous
62
+ // PRAGMA schema.synchronous = 0 | OFF | 1 | NORMAL | 2 | FULL | 3 | EXTRA;
63
+ snprintfz(buf, 1024, "PRAGMA synchronous=%s;", config_get(CONFIG_SECTION_SQLITE, "synchronous", "NORMAL"));
64
+ if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
65
+
66
+ // https://www.sqlite.org/pragma.html#pragma_journal_mode
67
+ // PRAGMA schema.journal_mode = DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF
68
+ snprintfz(buf, 1024, "PRAGMA journal_mode=%s;", config_get(CONFIG_SECTION_SQLITE, "journal mode", "WAL"));
69
+ if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
70
+
71
+ // https://www.sqlite.org/pragma.html#pragma_temp_store
72
+ // PRAGMA temp_store = 0 | DEFAULT | 1 | FILE | 2 | MEMORY;
73
+ snprintfz(buf, 1024, "PRAGMA temp_store=%s;", config_get(CONFIG_SECTION_SQLITE, "temp store", "MEMORY"));
74
+ if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
75
+
76
+ // https://www.sqlite.org/pragma.html#pragma_journal_size_limit
77
+ // PRAGMA schema.journal_size_limit = N ;
78
+ snprintfz(buf, 1024, "PRAGMA journal_size_limit=%lld;", config_get_number(CONFIG_SECTION_SQLITE, "journal size limit", 16777216));
79
+ if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
80
+
81
+ // https://www.sqlite.org/pragma.html#pragma_cache_size
82
+ // PRAGMA schema.cache_size = pages;
83
+ // PRAGMA schema.cache_size = -kibibytes;
84
+ snprintfz(buf, 1024, "PRAGMA cache_size=%lld;", config_get_number(CONFIG_SECTION_SQLITE, "cache size", -2000));
85
+ if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
86
+
87
+ snprintfz(buf, 1024, "PRAGMA user_version=%d;", target_version);
88
+ if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
89
+
90
+ snprintfz(buf, 1024, "ATTACH DATABASE \"%s/netdata-meta.db\" as meta;", netdata_configured_cache_dir);
91
+ if(init_database_batch(db_context_meta, DB_CHECK_NONE, 0, list)) return 1;
92
+
93
+ if (init_database_batch(db_context_meta, DB_CHECK_NONE, 0, &database_context_config[0]))
94
+ return 1;
95
+
96
+ if (init_database_batch(db_context_meta, DB_CHECK_NONE, 0, &database_context_cleanup[0]))
97
+ return 1;
98
+
99
+ return 0;
100
+}
101
+
102
+/*
103
+ * Close the sqlite database
104
+ */
105
+
106
+void sql_close_context_database(void)
107
+{
108
+ int rc;
109
+ if (unlikely(!db_context_meta))
110
+ return;
111
+
112
+ info("Closing context SQLite database");
113
+
114
+ rc = sqlite3_close_v2(db_context_meta);
115
+ if (unlikely(rc != SQLITE_OK))
116
+ error_report("Error %d while closing the context SQLite database, %s", rc, sqlite3_errstr(rc));
117
+ return;
118
+}
119
+
120
+//
121
+// Fetching data
122
+//
123
+#define CTX_GET_CHART_LIST "SELECT c.chart_id, c.type||'.'||c.id, c.name, c.context, c.title, c.unit, c.priority, " \
124
+ "c.update_every, c.chart_type, c.family FROM meta.chart c WHERE c.host_id = @host_id; "
125
+
126
+void ctx_get_chart_list(uuid_t *host_uuid, void (*dict_cb)(SQL_CHART_DATA *, void *), void *data)
127
+{
128
+ int rc;
129
+ sqlite3_stmt *res = NULL;
130
+
131
+ if (unlikely(!host_uuid)) {
132
+ internal_error(true, "Requesting context chart list without host_id");
133
+ return;
134
+ }
135
+
136
+ rc = sqlite3_prepare_v2(db_context_meta, CTX_GET_CHART_LIST, -1, &res, 0);
137
+ if (unlikely(rc != SQLITE_OK)) {
138
+ error_report("Failed to prepare statement to fetch chart list");
139
+ return;
140
+ }
141
+
142
+ rc = sqlite3_bind_blob(res, 1, host_uuid, sizeof(*host_uuid), SQLITE_STATIC);
143
+ if (unlikely(rc != SQLITE_OK)) {
144
+ error_report("Failed to bind host_id to fetch the chart list");
145
+ goto skip_load;
146
+ }
147
+
148
+ SQL_CHART_DATA chart_data = { 0 };
149
+ while (sqlite3_step(res) == SQLITE_ROW) {
150
+ uuid_copy(chart_data.chart_id, *((uuid_t *)sqlite3_column_blob(res, 0)));
151
+ chart_data.id = (char *) sqlite3_column_text(res, 1);
152
+ chart_data.name = (char *) sqlite3_column_text(res, 2);
153
+ chart_data.context = (char *) sqlite3_column_text(res, 3);
154
+ chart_data.title = (char *) sqlite3_column_text(res, 4);
155
+ chart_data.units = (char *) sqlite3_column_text(res, 5);
156
+ chart_data.priority = sqlite3_column_int(res, 6);
157
+ chart_data.update_every = sqlite3_column_int(res, 7);
158
+ chart_data.chart_type = sqlite3_column_int(res, 8);
159
+ chart_data.family = (char *) sqlite3_column_text(res, 9);
160
+ dict_cb(&chart_data, data);
161
+ }
162
+
163
+skip_load:
164
+ rc = sqlite3_finalize(res);
165
+ if (rc != SQLITE_OK)
166
+ error_report("Failed to finalize statement that fetches chart label data, rc = %d", rc);
167
+}
168
+
169
+// Dimension list
170
+#define CTX_GET_DIMENSION_LIST "SELECT d.dim_id, d.id, d.name FROM meta.dimension d WHERE d.chart_id = @id;"
171
+void ctx_get_dimension_list(uuid_t *chart_uuid, void (*dict_cb)(SQL_DIMENSION_DATA *, void *), void *data)
172
+{
173
+ int rc;
174
+ sqlite3_stmt *res = NULL;
175
+
176
+ rc = sqlite3_prepare_v2(db_context_meta, CTX_GET_DIMENSION_LIST, -1, &res, 0);
177
+ if (unlikely(rc != SQLITE_OK)) {
178
+ error_report("Failed to prepare statement to fetch chart dimension data");
179
+ return;
180
+ }
181
+
182
+ rc = sqlite3_bind_blob(res, 1, chart_uuid, sizeof(*chart_uuid), SQLITE_STATIC);
183
+ if (unlikely(rc != SQLITE_OK)) {
184
+ error_report("Failed to bind chart_id to fetch dimension list");
185
+ goto failed;
186
+ }
187
+
188
+ SQL_DIMENSION_DATA dimension_data;
189
+
190
+ while (sqlite3_step(res) == SQLITE_ROW) {
191
+ uuid_copy(dimension_data.dim_id, *((uuid_t *)sqlite3_column_blob(res, 0)));
192
+ dimension_data.id = (char *) sqlite3_column_text(res, 1);
193
+ dimension_data.name = (char *) sqlite3_column_text(res, 2);
194
+ dict_cb(&dimension_data, data);
195
+ }
196
+
197
+failed:
198
+ rc = sqlite3_finalize(res);
199
+ if (rc != SQLITE_OK)
200
+ error_report("Failed to finalize statement that fetches the chart dimension list, rc = %d", rc);
201
+}
202
+
203
+// LABEL LIST
204
+#define CTX_GET_LABEL_LIST "SELECT l.label_key, l.label_value, l.source_type FROM meta.chart_label l WHERE l.chart_id = @id;"
205
+void ctx_get_label_list(uuid_t *chart_uuid, void (*dict_cb)(SQL_CLABEL_DATA *, void *), void *data)
206
+{
207
+ int rc;
208
+ sqlite3_stmt *res = NULL;
209
+
210
+ rc = sqlite3_prepare_v2(db_context_meta, CTX_GET_LABEL_LIST, -1, &res, 0);
211
+ if (unlikely(rc != SQLITE_OK)) {
212
+ error_report("Failed to prepare statement to fetch chart lanbels");
213
+ return;
214
+ }
215
+
216
+ rc = sqlite3_bind_blob(res, 1, chart_uuid, sizeof(*chart_uuid), SQLITE_STATIC);
217
+ if (unlikely(rc != SQLITE_OK)) {
218
+ error_report("Failed to bind chart_id to fetch chart labels");
219
+ goto failed;
220
+ }
221
+
222
+ SQL_CLABEL_DATA label_data;
223
+
224
+ while (sqlite3_step(res) == SQLITE_ROW) {
225
+ label_data.label_key = (char *) sqlite3_column_text(res, 0);
226
+ label_data.label_value = (char *) sqlite3_column_text(res, 1);
227
+ label_data.label_source = sqlite3_column_int(res, 2);
228
+ dict_cb(&label_data, data);
229
+ }
230
+
231
+failed:
232
+ rc = sqlite3_finalize(res);
233
+ if (rc != SQLITE_OK)
234
+ error_report("Failed to finalize statement that fetches chart label data, rc = %d", rc);
235
+
236
+ return;
237
+}
238
+
239
+// CONTEXT LIST
240
+#define CTX_GET_CONTEXT_LIST "SELECT id, version, title, chart_type, unit, priority, first_time_t, " \
241
+ "last_time_t, deleted, family FROM context c WHERE c.host_id = @host_id;"
242
+void ctx_get_context_list(uuid_t *host_uuid, void (*dict_cb)(VERSIONED_CONTEXT_DATA *, void *), void *data)
243
+{
244
+
245
+ if (unlikely(!host_uuid))
246
+ return;
247
+
248
+ int rc;
249
+ sqlite3_stmt *res = NULL;
250
+
251
+ rc = sqlite3_prepare_v2(db_context_meta, CTX_GET_CONTEXT_LIST, -1, &res, 0);
252
+ if (unlikely(rc != SQLITE_OK)) {
253
+ error_report("Failed to prepare statement to fetch stored context list");
254
+ return;
255
+ }
256
+
257
+ VERSIONED_CONTEXT_DATA context_data = {0};
258
+
259
+ rc = sqlite3_bind_blob(res, 1, host_uuid, sizeof(*host_uuid), SQLITE_STATIC);
260
+
261
+ if (unlikely(rc != SQLITE_OK)) {
262
+ error_report("Failed to bind host_id to fetch versioned context data");
263
+ goto failed;
264
+ }
265
+
266
+ while (sqlite3_step(res) == SQLITE_ROW) {
267
+ context_data.id = (char *) sqlite3_column_text(res, 0);
268
+ context_data.version = sqlite3_column_int64(res, 1);
269
+ context_data.title = (char *) sqlite3_column_text(res, 2);
270
+ context_data.chart_type = (char *) sqlite3_column_text(res, 3);
271
+ context_data.units = (char *) sqlite3_column_text(res, 4);
272
+ context_data.priority = sqlite3_column_int64(res, 5);
273
+ context_data.first_time_t = sqlite3_column_int64(res, 6);
274
+ context_data.last_time_t = sqlite3_column_int64(res, 7);
275
+ context_data.deleted = sqlite3_column_int(res, 8);
276
+ context_data.family = (char *) sqlite3_column_text(res, 9);
277
+ dict_cb(&context_data, data);
278
+ }
279
+
280
+failed:
281
+ rc = sqlite3_finalize(res);
282
+ if (rc != SQLITE_OK)
283
+ error_report("Failed to finalize statement that fetches stored context versioned data, rc = %d", rc);
284
+}
285
+
286
+
287
+//
288
+// Storing Data
289
+//
290
+#define CTX_STORE_CONTEXT "INSERT OR REPLACE INTO context " \
291
+ "(host_id, id, version, title, chart_type, unit, priority, first_time_t, last_time_t, deleted, family) " \
292
+ "VALUES (@host_id, @context, @version, @title, @chart_type, @unit, @priority, @first_time_t, @last_time_t, @deleted, @family);"
293
+
294
+int ctx_store_context(uuid_t *host_uuid, VERSIONED_CONTEXT_DATA *context_data)
295
+{
296
+ int rc, rc_stored = 1;
297
+ sqlite3_stmt *res = NULL;
298
+
299
+ if (unlikely(!host_uuid || !context_data || !context_data->id))
300
+ return 0;
301
+
302
+ rc = sqlite3_prepare_v2(db_context_meta, CTX_STORE_CONTEXT, -1, &res, 0);
303
+ if (unlikely(rc != SQLITE_OK)) {
304
+ error_report("Failed to prepare statement to store context");
305
+ return 1;
306
+ }
307
+
308
+ rc = sqlite3_bind_blob(res, 1, host_uuid, sizeof(*host_uuid), SQLITE_STATIC);
309
+ if (unlikely(rc != SQLITE_OK)) {
310
+ error_report("Failed to bind host_uuid to store context details");
311
+ goto skip_store;
312
+ }
313
+
314
+ rc = bind_text_null(res, 2, context_data->id, 0);
315
+ if (unlikely(rc != SQLITE_OK)) {
316
+ error_report("Failed to bind context to store details");
317
+ goto skip_store;
318
+ }
319
+
320
+ rc = sqlite3_bind_int64(res, 3, (time_t) context_data->version);
321
+ if (unlikely(rc != SQLITE_OK)) {
322
+ error_report("Failed to bind first_time_t to store context details");
323
+ goto skip_store;
324
+ }
325
+
326
+ rc = bind_text_null(res, 4, context_data->title, 0);
327
+ if (unlikely(rc != SQLITE_OK)) {
328
+ error_report("Failed to bind context to store details");
329
+ goto skip_store;
330
+ }
331
+
332
+ rc = bind_text_null(res, 5, context_data->chart_type, 0);
333
+ if (unlikely(rc != SQLITE_OK)) {
334
+ error_report("Failed to bind context to store details");
335
+ goto skip_store;
336
+ }
337
+
338
+ rc = bind_text_null(res, 6, context_data->units, 0);
339
+ if (unlikely(rc != SQLITE_OK)) {
340
+ error_report("Failed to bind context to store details");
341
+ goto skip_store;
342
+ }
343
+
344
+ rc = sqlite3_bind_int64(res, 7, (time_t) context_data->priority);
345
+ if (unlikely(rc != SQLITE_OK)) {
346
+ error_report("Failed to bind first_time_t to store context details");
347
+ goto skip_store;
348
+ }
349
+
350
+ rc = sqlite3_bind_int64(res, 8, (time_t) context_data->first_time_t);
351
+ if (unlikely(rc != SQLITE_OK)) {
352
+ error_report("Failed to bind first_time_t to store context details");
353
+ goto skip_store;
354
+ }
355
+
356
+ rc = sqlite3_bind_int64(res, 9, (time_t) context_data->last_time_t);
357
+ if (unlikely(rc != SQLITE_OK)) {
358
+ error_report("Failed to bind last_time_t to store context details");
359
+ goto skip_store;
360
+ }
361
+
362
+ rc = sqlite3_bind_int(res, 10, (time_t) context_data->deleted);
363
+ if (unlikely(rc != SQLITE_OK)) {
364
+ error_report("Failed to bind last_time_t to store context details");
365
+ goto skip_store;
366
+ }
367
+
368
+ rc = bind_text_null(res, 11, context_data->family, 1);
369
+ if (unlikely(rc != SQLITE_OK)) {
370
+ error_report("Failed to bind context to store details");
371
+ goto skip_store;
372
+ }
373
+
374
+ rc_stored = execute_insert(res);
375
+
376
+ if (rc_stored != SQLITE_DONE)
377
+ error_report("Failed store context details for context %s, rc = %d", context_data->id, rc_stored);
378
+
379
+skip_store:
380
+ rc = sqlite3_finalize(res);
381
+ if (rc != SQLITE_OK)
382
+ error_report("Failed to finalize statement that stores context details, rc = %d", rc);
383
+
384
+ return (rc_stored != SQLITE_DONE);
385
+}
386
+
387
+// Delete a context
388
+
389
+#define CTX_DELETE_CONTEXT "DELETE FROM context WHERE host_id = @host_id AND id = @context;"
390
+int ctx_delete_context(uuid_t *host_uuid, VERSIONED_CONTEXT_DATA *context_data)
391
+{
392
+ int rc, rc_stored = 1;
393
+ sqlite3_stmt *res = NULL;
394
+
395
+ if (unlikely(!context_data || !context_data->id))
396
+ return 0;
397
+
398
+ rc = sqlite3_prepare_v2(db_context_meta, CTX_DELETE_CONTEXT, -1, &res, 0);
399
+ if (unlikely(rc != SQLITE_OK)) {
400
+ error_report("Failed to prepare statement to delete context");
401
+ return 1;
402
+ }
403
+
404
+ rc = sqlite3_bind_blob(res, 1, host_uuid, sizeof(*host_uuid), SQLITE_STATIC);
405
+ if (unlikely(rc != SQLITE_OK)) {
406
+ error_report("Failed to bind host_id to delete context data");
407
+ goto skip_delete;
408
+ }
409
+
410
+ rc = sqlite3_bind_text(res, 2, context_data->id, -1, SQLITE_STATIC);
411
+ if (unlikely(rc != SQLITE_OK)) {
412
+ error_report("Failed to bind context id for data deletion");
413
+ goto skip_delete;
414
+ }
415
+
416
+ rc_stored = execute_insert(res);
417
+
418
+ if (rc_stored != SQLITE_DONE)
419
+ error_report("Failed to delete context %s, rc = %d", context_data->id, rc_stored);
420
+#ifdef NETDATA_INTERNAL_CHECKS
421
+ else {
422
+ char host_uuid_str[UUID_STR_LEN];
423
+ uuid_unparse_lower(*host_uuid, host_uuid_str);
424
+ info("%s: Deleted context %s under host %s", __FUNCTION__ , context_data->id, host_uuid_str);
425
+ }
426
+#endif
427
+
428
+skip_delete:
429
+ rc = sqlite3_finalize(res);
430
+ if (rc != SQLITE_OK)
431
+ error_report("Failed to finalize statement where deleting a context, rc = %d", rc);
432
+
433
+ return (rc_stored != SQLITE_DONE);
434
+}
435
+
436
+//
437
+// TESTING FUNCTIONS
438
+//
439
+static void dict_ctx_get_label_list_cb(SQL_CLABEL_DATA *label_data_ptr, void *data)
440
+{
441
+ (void)data;
442
+ SQL_CLABEL_DATA *label_data = label_data_ptr;
443
+
444
+ info(" LABEL %d %s = %s", label_data->label_source, label_data->label_key, label_data->label_value);
445
+}
446
+
447
+static void dict_ctx_get_dimension_list_cb(SQL_DIMENSION_DATA *dimension_data_ptr, void *data)
448
+{
449
+ (void)data;
450
+
451
+ SQL_DIMENSION_DATA *dimension_data = dimension_data_ptr;
452
+
453
+ char uuid_str[UUID_STR_LEN];
454
+ uuid_unparse_lower(dimension_data->dim_id, uuid_str);
455
+
456
+ info(" Dimension %s = %s", uuid_str, dimension_data->id);
457
+}
458
+
459
+
460
+static void dict_ctx_get_chart_list_cb(SQL_CHART_DATA *chart_data, void *data)
461
+{
462
+ (void)data;
463
+
464
+ char uuid_str[UUID_STR_LEN];
465
+ uuid_unparse_lower(chart_data->chart_id, uuid_str);
466
+ info("OK GOT %s ID = %s NAME = %s CONTEXT = %s", uuid_str, chart_data->id, chart_data->name, chart_data->context);
467
+ ctx_get_label_list(&chart_data->chart_id, dict_ctx_get_label_list_cb, NULL);
468
+ ctx_get_dimension_list(&chart_data->chart_id, dict_ctx_get_dimension_list_cb, NULL);
469
+}
470
+
471
+static void dict_ctx_get_context_list_cb(VERSIONED_CONTEXT_DATA *context_data, void *data)
472
+{
473
+ (void)data;
474
+ info(" Context id = %s "
475
+ "version = %lu "
476
+ "title = %s "
477
+ "chart_type = %s "
478
+ "units = %s "
479
+ "priority = %lu "
480
+ "first time = %lu "
481
+ "last time = %lu "
482
+ "deleted = %d"
483
+ "family = %s",
484
+ context_data->id,
485
+ context_data->version,
486
+ context_data->title,
487
+ context_data->chart_type,
488
+ context_data->units,
489
+ context_data->priority,
490
+ context_data->first_time_t,
491
+ context_data->last_time_t,
492
+ context_data->deleted,
493
+ context_data->family);
494
+}
495
+
496
+static int localhost_uuid_cb(void *data, int argc, char **argv, char **column)
497
+{
498
+ uuid_t *uuid = data;
499
+ UNUSED(argc);
500
+ UNUSED(column);
501
+ uuid_copy(*uuid, * (uuid_t *) argv[0]);
502
+ return 0;
503
+}
504
+
505
+
506
+#define SQL_FIND_LOCALHOST "SELECT host_id FROM meta.host WHERE hops = 0;"
507
+int ctx_unittest(void)
508
+{
509
+ uuid_t host_uuid;
510
+ uuid_t host_uuid1;
511
+
512
+ uuid_generate(host_uuid1);
513
+
514
+ char *err_msg;
515
+
516
+ sql_init_context_database(1);
517
+
518
+ int rc = sqlite3_exec(db_context_meta, SQL_FIND_LOCALHOST, localhost_uuid_cb, (void *) &host_uuid, &err_msg);
519
+ if (rc != SQLITE_OK) {
520
+ info("Failed to discover localhost UUID rc = %d -- %s", rc, err_msg);
521
+ sqlite3_free(err_msg);
522
+ }
523
+
524
+ ctx_get_chart_list(&host_uuid, dict_ctx_get_chart_list_cb, NULL);
525
+
526
+ // Store a context
527
+ VERSIONED_CONTEXT_DATA context_data;
528
+
529
+ context_data.id = strdupz("cpu.cpu");
530
+ context_data.title = strdupz("TestContextTitle");
531
+ context_data.units= strdupz("TestContextUnits");
532
+ context_data.chart_type = strdupz("TestContextChartType");
533
+ context_data.family = strdupz("TestContextFamily");
534
+ context_data.priority = 50000;
535
+ context_data.deleted = 0;
536
+ context_data.first_time_t = 1657781000;
537
+ context_data.last_time_t = 1657781100;
538
+ context_data.version = now_realtime_usec();
539
+
540
+ if (likely(!ctx_store_context(&host_uuid, &context_data)))
541
+ info("Entry %s inserted", context_data.id);
542
+ else
543
+ info("Entry %s not inserted", context_data.id);
544
+
545
+ if (likely(!ctx_store_context(&host_uuid1, &context_data)))
546
+ info("Entry %s inserted", context_data.id);
547
+ else
548
+ info("Entry %s not inserted", context_data.id);
549
+
550
+ // This will change end time
551
+ context_data.first_time_t = 1657781000;
552
+ context_data.last_time_t = 1657782001;
553
+ if (likely(!ctx_update_context(&host_uuid, &context_data)))
554
+ info("Entry %s updated", context_data.id);
555
+ else
556
+ info("Entry %s not updated", context_data.id);
557
+ info("List context start after insert");
558
+ ctx_get_context_list(&host_uuid, dict_ctx_get_context_list_cb, NULL);
559
+ info("List context end after insert");
560
+
561
+ // This will change start time
562
+ context_data.first_time_t = 1657782000;
563
+ context_data.last_time_t = 1657782001;
564
+ if (likely(!ctx_update_context(&host_uuid, &context_data)))
565
+ info("Entry %s updated", context_data.id);
566
+ else
567
+ info("Entry %s not updated", context_data.id);
568
+
569
+ // This will list one entry
570
+ info("List context start after insert");
571
+ ctx_get_context_list(&host_uuid, dict_ctx_get_context_list_cb, NULL);
572
+ info("List context end after insert");
573
+
574
+ info("List context start after insert");
575
+ ctx_get_context_list(&host_uuid1, dict_ctx_get_context_list_cb, NULL);
576
+ info("List context end after insert");
577
+
578
+ // This will delete the entry
579
+ if (likely(!ctx_delete_context(&host_uuid, &context_data)))
580
+ info("Entry %s deleted", context_data.id);
581
+ else
582
+ info("Entry %s not deleted", context_data.id);
583
+
584
+ freez((void *)context_data.id);
585
+ freez((void *)context_data.title);
586
+ freez((void *)context_data.chart_type);
587
+ freez((void *)context_data.family);
588
+
589
+ // The list should be empty
590
+ info("List context start after delete");
591
+ ctx_get_context_list(&host_uuid, dict_ctx_get_context_list_cb, NULL);
592
+ info("List context end after delete");
593
+
594
+ sql_close_context_database();
595
+
596
+ return 0;
597
+}
598
+
database/sqlite/sqlite_context.h
new
+68
@@ -0,0 +1,68 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_SQLITE_CONTEXT_H
4
+#define NETDATA_SQLITE_CONTEXT_H
5
+
6
+#include "daemon/common.h"
7
+#include "sqlite3.h"
8
+
9
+typedef struct ctx_chart {
10
+ uuid_t chart_id;
11
+ const char *id;
12
+ const char *name;
13
+ const char *context;
14
+ const char *title;
15
+ const char *units;
16
+ const char *family;
17
+ int chart_type;
18
+ int priority;
19
+ int update_every;
20
+} SQL_CHART_DATA;
21
+
22
+typedef struct ctx_dimension {
23
+ uuid_t dim_id;
24
+ char *id;
25
+ char *name;
26
+} SQL_DIMENSION_DATA;
27
+
28
+typedef struct ctx_label {
29
+ char *label_key;
30
+ char *label_value;
31
+ int label_source;
32
+} SQL_CLABEL_DATA;
33
+
34
+// Structure to store or delete
35
+typedef struct versioned_context_data {
36
+ uint64_t version; // the version of this context as EPOCH in seconds
37
+
38
+ const char *id; // the id of the context
39
+ const char *title; // the title of the context
40
+ const char *chart_type; // the chart_type of the context
41
+ const char *units; // the units of the context
42
+ const char *family; // the family of the context
43
+
44
+ uint64_t priority; // the chart priority of the context
45
+
46
+ uint64_t first_time_t; // the first entry in the database, in seconds
47
+ uint64_t last_time_t; // the last point in the database, in seconds
48
+
49
+ bool deleted; // true when this is deleted
50
+
51
+} VERSIONED_CONTEXT_DATA;
52
+
53
+extern void ctx_get_context_list(uuid_t *host_uuid, void (*dict_cb)(VERSIONED_CONTEXT_DATA *, void *), void *data);
54
+
55
+extern void ctx_get_chart_list(uuid_t *host_uuid, void (*dict_cb)(SQL_CHART_DATA *, void *), void *data);
56
+extern void ctx_get_label_list(uuid_t *chart_uuid, void (*dict_cb)(SQL_CLABEL_DATA *, void *), void *data);
57
+extern void ctx_get_dimension_list(uuid_t *chart_uuid, void (*dict_cb)(SQL_DIMENSION_DATA *, void *), void *data);
58
+
59
+extern int ctx_store_context(uuid_t *host_uuid, VERSIONED_CONTEXT_DATA *context_data);
60
+
61
+#define ctx_update_context(host_uuid, context_data) ctx_store_context(host_uuid, context_data)
62
+
63
+extern int ctx_delete_context(uuid_t *host_id, VERSIONED_CONTEXT_DATA *context_data);
64
+
65
+extern int sql_init_context_database(int memory);
66
+extern void sql_close_context_database(void);
67
+extern int ctx_unittest(void);
68
+#endif //NETDATA_SQLITE_CONTEXT_H
database/sqlite/sqlite_db_migration.c
+79
-18
@@ -11,6 +11,25 @@ static int return_int_cb(void *data, int argc, char **argv, char **column)
11
return 0;
12
}
13
14
+
15
+static int table_exists_in_database(const char *table)
16
+{
17
+ char *err_msg = NULL;
18
+ char sql[128];
19
+
20
+ int exists = 0;
21
+
22
+ snprintf(sql, 127, "select 1 from sqlite_schema where type = 'table' and name = '%s';", table);
23
+
24
+ int rc = sqlite3_exec(db_meta, sql, return_int_cb, (void *) &exists, &err_msg);
25
+ if (rc != SQLITE_OK) {
26
+ info("Error checking table existence; %s", err_msg);
27
+ sqlite3_free(err_msg);
28
+ }
29
+
30
+ return exists;
31
+}
32
+
33
static int column_exists_in_table(const char *table, const char *column)
34
{
35
char *err_msg = NULL;
@@ -34,14 +53,34 @@ const char *database_migrate_v1_v2[] = {
53
NULL
54
};
55
56
+const char *database_migrate_v2_v3[] = {
57
+ "ALTER TABLE host ADD memory_mode INT NOT NULL DEFAULT 0;",
58
+ "ALTER TABLE host ADD abbrev_timezone TEXT NOT NULL DEFAULT '';",
59
+ "ALTER TABLE host ADD utc_offset INT NOT NULL DEFAULT 0;",
60
+ "ALTER TABLE host ADD program_name TEXT NOT NULL DEFAULT 'unknown';",
61
+ "ALTER TABLE host ADD program_version TEXT NOT NULL DEFAULT 'unknown';",
62
+ "ALTER TABLE host ADD entries INT NOT NULL DEFAULT 0;",
63
+ "ALTER TABLE host ADD health_enabled INT NOT NULL DEFAULT 0;",
64
+ NULL
65
+};
66
+
67
static int do_migration_v1_v2(sqlite3 *database, const char *name)
68
{
39
- UNUSED(database);
69
UNUSED(name);
41
- info("Running database migration %s", name);
70
+ info("Running \"%s\" database migration", name);
71
+
72
+ if (table_exists_in_database("host") && !column_exists_in_table("host", "hops"))
73
+ return init_database_batch(database, DB_CHECK_NONE, 0, &database_migrate_v1_v2[0]);
74
+ return 0;
75
+}
76
+
77
+static int do_migration_v2_v3(sqlite3 *database, const char *name)
78
+{
79
+ UNUSED(name);
80
+ info("Running \"%s\" database migration", name);
81
43
- if (!column_exists_in_table("host", "hops"))
44
- return init_database_batch(DB_CHECK_NONE, 0, &database_migrate_v1_v2[0]);
82
+ if (table_exists_in_database("host") && !column_exists_in_table("host", "memory_mode"))
83
+ return init_database_batch(database, DB_CHECK_NONE, 0, &database_migrate_v2_v3[0]);
84
return 0;
85
}
86
@@ -54,40 +93,62 @@ static int do_migration_noop(sqlite3 *database, const char *name)
93
return 0;
94
}
95
57
-static struct database_func_migration_list {
96
+typedef struct database_func_migration_list {
97
char *name;
98
int (*func)(sqlite3 *database, const char *name);
60
-} migration_action[] = {
61
- {.name = "v0 to v1", .func = do_migration_noop},
62
- {.name = "v1 to v2", .func = do_migration_v1_v2},
63
- // the terminator of this array
64
- {.name = NULL, .func = NULL}
65
-};
99
+} DATABASE_FUNC_MIGRATION_LIST;
100
101
68
-int perform_database_migration(sqlite3 *database, int target_version)
102
+static int migrate_database(sqlite3 *database, int target_version, char *db_name, DATABASE_FUNC_MIGRATION_LIST *migration_list)
103
{
104
int user_version = 0;
105
char *err_msg = NULL;
106
107
int rc = sqlite3_exec(database, "PRAGMA user_version;", return_int_cb, (void *) &user_version, &err_msg);
108
if (rc != SQLITE_OK) {
75
- info("Error checking the database version; %s", err_msg);
109
+ info("Error checking the %s database version; %s", db_name, err_msg);
110
sqlite3_free(err_msg);
111
}
112
113
if (likely(user_version == target_version)) {
80
- info("Metadata database version is %d", target_version);
114
+ info("%s database version is %d (no migration needed)", db_name, target_version);
115
return target_version;
116
}
117
84
- info("Database version is %d, current version is %d. Running migration ...", user_version, target_version);
85
- for (int i = user_version; migration_action[i].func && i < target_version; i++) {
86
- rc = (migration_action[i].func)(database, migration_action[i].name);
118
+ info("Database version is %d, current version is %d. Running migration for %s ...", user_version, target_version, db_name);
119
+ for (int i = user_version; i < target_version && migration_list[i].func; i++) {
120
+ rc = (migration_list[i].func)(database, migration_list[i].name);
121
if (unlikely(rc)) {
88
- error_report("Database migration from version %d to version %d failed", i, i + 1);
122
+ error_report("Database %s migration from version %d to version %d failed", db_name, i, i + 1);
123
return i;
124
}
125
}
126
return target_version;
127
+
128
+}
129
+
130
+DATABASE_FUNC_MIGRATION_LIST migration_action[] = {
131
+ {.name = "v0 to v1", .func = do_migration_noop},
132
+ {.name = "v1 to v2", .func = do_migration_v1_v2},
133
+ {.name = "v2 to v3", .func = do_migration_v2_v3},
134
+ // the terminator of this array
135
+ {.name = NULL, .func = NULL}
136
+};
137
+
138
+DATABASE_FUNC_MIGRATION_LIST context_migration_action[] = {
139
+ {.name = "v0 to v1", .func = do_migration_noop},
140
+
141
+ // the terminator of this array
142
+ {.name = NULL, .func = NULL}
143
+};
144
+
145
+
146
+int perform_database_migration(sqlite3 *database, int target_version)
147
+{
148
+ return migrate_database(database, target_version, "metadata", migration_action);
149
+}
150
+
151
+int perform_context_database_migration(sqlite3 *database, int target_version)
152
+{
153
+ return migrate_database(database, target_version, "context", context_migration_action);
154
}
database/sqlite/sqlite_db_migration.h
+1
@@ -7,5 +7,6 @@
7
8
9
int perform_database_migration(sqlite3 *database, int target_version);
10
+int perform_context_database_migration(sqlite3 *database, int target_version);
11
12
#endif //NETDATA_SQLITE_DB_MIGRATION_H
database/sqlite/sqlite_functions.c
+144
-15
@@ -3,11 +3,18 @@
3
#include "sqlite_functions.h"
4
#include "sqlite_db_migration.h"
5
6
-#define DB_METADATA_VERSION 2
6
+#define DB_METADATA_VERSION 3
7
8
const char *database_config[] = {
9
- "CREATE TABLE IF NOT EXISTS host(host_id blob PRIMARY KEY, hostname text, "
10
- "registry_hostname text, update_every int, os text, timezone text, tags text, hops INT);",
9
+ "CREATE TABLE IF NOT EXISTS host(host_id BLOB PRIMARY KEY, hostname TEXT NOT NULL, "
10
+ "registry_hostname TEXT NOT NULL default 'unknown', update_every INT NOT NULL default 1, "
11
+ "os TEXT NOT NULL default 'unknown', timezone TEXT NOT NULL default 'unknown', tags TEXT NOT NULL default '',"
12
+ "hops INT NOT NULL DEFAULT 0,"
13
+ "memory_mode INT DEFAULT 0, abbrev_timezone TEXT DEFAULT '', utc_offset INT NOT NULL DEFAULT 0,"
14
+ "program_name TEXT NOT NULL DEFAULT 'unknown', program_version TEXT NOT NULL DEFAULT 'unknown', "
15
+ "entries INT NOT NULL DEFAULT 0,"
16
+ "health_enabled INT NOT NULL DEFAULT 0);",
17
+
18
"CREATE TABLE IF NOT EXISTS chart(chart_id blob PRIMARY KEY, host_id blob, type text, id text, name text, "
19
"family text, context text, title text, unit text, plugin text, module text, priority int, update_every int, "
20
"chart_type int, memory_mode int, history_entries);",
@@ -349,13 +356,13 @@ static int attempt_database_fix()
356
return sql_init_database(DB_CHECK_FIX_DB | DB_CHECK_CONT, 0);
357
}
358
352
-int init_database_batch(int rebuild, int init_type, const char *batch[])
359
+int init_database_batch(sqlite3 *database, int rebuild, int init_type, const char *batch[])
360
{
361
int rc;
362
char *err_msg = NULL;
363
for (int i = 0; batch[i]; i++) {
364
debug(D_METADATALOG, "Executing %s", batch[i]);
358
- rc = sqlite3_exec(db_meta, batch[i], 0, 0, &err_msg);
365
+ rc = sqlite3_exec(database, batch[i], 0, 0, &err_msg);
366
if (rc != SQLITE_OK) {
367
error_report("SQLite error during database %s, rc = %d (%s)", init_type ? "cleanup" : "setup", rc, err_msg);
368
error_report("SQLite failed statement %s", batch[i]);
@@ -449,41 +456,41 @@ int sql_init_database(db_check_action_type_t rebuild, int memory)
456
// https://www.sqlite.org/pragma.html#pragma_auto_vacuum
457
// PRAGMA schema.auto_vacuum = 0 | NONE | 1 | FULL | 2 | INCREMENTAL;
458
snprintfz(buf, 1024, "PRAGMA auto_vacuum=%s;", config_get(CONFIG_SECTION_SQLITE, "auto vacuum", "INCREMENTAL"));
452
- if(init_database_batch(rebuild, 0, list)) return 1;
459
+ if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
460
461
// https://www.sqlite.org/pragma.html#pragma_synchronous
462
// PRAGMA schema.synchronous = 0 | OFF | 1 | NORMAL | 2 | FULL | 3 | EXTRA;
463
snprintfz(buf, 1024, "PRAGMA synchronous=%s;", config_get(CONFIG_SECTION_SQLITE, "synchronous", "NORMAL"));
457
- if(init_database_batch(rebuild, 0, list)) return 1;
464
+ if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
465
466
// https://www.sqlite.org/pragma.html#pragma_journal_mode
467
// PRAGMA schema.journal_mode = DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF
468
snprintfz(buf, 1024, "PRAGMA journal_mode=%s;", config_get(CONFIG_SECTION_SQLITE, "journal mode", "WAL"));
462
- if(init_database_batch(rebuild, 0, list)) return 1;
469
+ if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
470
471
// https://www.sqlite.org/pragma.html#pragma_temp_store
472
// PRAGMA temp_store = 0 | DEFAULT | 1 | FILE | 2 | MEMORY;
473
snprintfz(buf, 1024, "PRAGMA temp_store=%s;", config_get(CONFIG_SECTION_SQLITE, "temp store", "MEMORY"));
467
- if(init_database_batch(rebuild, 0, list)) return 1;
468
-
474
+ if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
475
+
476
// https://www.sqlite.org/pragma.html#pragma_journal_size_limit
477
// PRAGMA schema.journal_size_limit = N ;
478
snprintfz(buf, 1024, "PRAGMA journal_size_limit=%lld;", config_get_number(CONFIG_SECTION_SQLITE, "journal size limit", 16777216));
472
- if(init_database_batch(rebuild, 0, list)) return 1;
479
+ if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
480
481
// https://www.sqlite.org/pragma.html#pragma_cache_size
482
// PRAGMA schema.cache_size = pages;
483
// PRAGMA schema.cache_size = -kibibytes;
484
snprintfz(buf, 1024, "PRAGMA cache_size=%lld;", config_get_number(CONFIG_SECTION_SQLITE, "cache size", -2000));
478
- if(init_database_batch(rebuild, 0, list)) return 1;
485
+ if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
486
487
snprintfz(buf, 1024, "PRAGMA user_version=%d;", target_version);
481
- if(init_database_batch(rebuild, 0, list)) return 1;
488
+ if(init_database_batch(db_meta, rebuild, 0, list)) return 1;
489
483
- if (init_database_batch(rebuild, 0, &database_config[0]))
490
+ if (init_database_batch(db_meta, rebuild, 0, &database_config[0]))
491
return 1;
492
486
- if (init_database_batch(rebuild, 0, &database_cleanup[0]))
493
+ if (init_database_batch(db_meta, rebuild, 0, &database_cleanup[0]))
494
return 1;
495
496
fatal_assert(0 == uv_mutex_init(&sqlite_transaction_lock));
@@ -853,6 +860,113 @@ bind_fail:
860
return 1;
861
}
862
863
+//
864
+// Store host and host system info information in the database
865
+#define SQL_STORE_HOST_INFO "INSERT OR REPLACE INTO host " \
866
+ "(host_id, hostname, registry_hostname, update_every, os, timezone," \
867
+ "tags, hops, memory_mode, abbrev_timezone, utc_offset, program_name, program_version," \
868
+ "entries, health_enabled) " \
869
+ "values (@host_id, @hostname, @registry_hostname, @update_every, @os, @timezone, @tags, @hops, @memory_mode, " \
870
+ "@abbrev_timezone, @utc_offset, @program_name, @program_version, " \
871
+ "@entries, @health_enabled);"
872
+
873
+int sql_store_host_info(RRDHOST *host)
874
+{
875
+ static __thread sqlite3_stmt *res = NULL;
876
+ int rc;
877
+
878
+ if (unlikely(!db_meta)) {
879
+ if (default_rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE)
880
+ return 0;
881
+ error_report("Database has not been initialized");
882
+ return 1;
883
+ }
884
+
885
+ if (unlikely((!res))) {
886
+ rc = prepare_statement(db_meta, SQL_STORE_HOST_INFO, &res);
887
+ if (unlikely(rc != SQLITE_OK)) {
888
+ error_report("Failed to prepare statement to store host, rc = %d", rc);
889
+ return 1;
890
+ }
891
+ }
892
+
893
+ rc = sqlite3_bind_blob(res, 1, &host->host_uuid, sizeof(host->host_uuid), SQLITE_STATIC);
894
+ if (unlikely(rc != SQLITE_OK))
895
+ goto bind_fail;
896
+
897
+ rc = bind_text_null(res, 2, host->hostname, 0);
898
+ if (unlikely(rc != SQLITE_OK))
899
+ goto bind_fail;
900
+
901
+ rc = bind_text_null(res, 3, host->registry_hostname, 1);
902
+ if (unlikely(rc != SQLITE_OK))
903
+ goto bind_fail;
904
+
905
+ rc = sqlite3_bind_int(res, 4, host->rrd_update_every);
906
+ if (unlikely(rc != SQLITE_OK))
907
+ goto bind_fail;
908
+
909
+ rc = bind_text_null(res, 5, host->os, 1);
910
+ if (unlikely(rc != SQLITE_OK))
911
+ goto bind_fail;
912
+
913
+ rc = bind_text_null(res, 6, host->timezone, 1);
914
+ if (unlikely(rc != SQLITE_OK))
915
+ goto bind_fail;
916
+
917
+ rc = bind_text_null(res, 7, host->tags, 1);
918
+ if (unlikely(rc != SQLITE_OK))
919
+ goto bind_fail;
920
+
921
+ rc = sqlite3_bind_int(res, 8, host->system_info ? host->system_info->hops : 0);
922
+ if (unlikely(rc != SQLITE_OK))
923
+ goto bind_fail;
924
+
925
+ rc = sqlite3_bind_int(res, 9, host->rrd_memory_mode);
926
+ if (unlikely(rc != SQLITE_OK))
927
+ goto bind_fail;
928
+
929
+ rc = bind_text_null(res, 10, host->abbrev_timezone, 1);
930
+ if (unlikely(rc != SQLITE_OK))
931
+ goto bind_fail;
932
+
933
+ rc = sqlite3_bind_int(res, 11, host->utc_offset);
934
+ if (unlikely(rc != SQLITE_OK))
935
+ goto bind_fail;
936
+
937
+ rc = bind_text_null(res, 12, host->program_name, 1);
938
+ if (unlikely(rc != SQLITE_OK))
939
+ goto bind_fail;
940
+
941
+ rc = bind_text_null(res, 13, host->program_version, 1);
942
+ if (unlikely(rc != SQLITE_OK))
943
+ goto bind_fail;
944
+
945
+ rc = sqlite3_bind_int64(res, 14, host->rrd_history_entries);
946
+ if (unlikely(rc != SQLITE_OK))
947
+ goto bind_fail;
948
+
949
+ rc = sqlite3_bind_int(res, 15, host->health_enabled);
950
+ if (unlikely(rc != SQLITE_OK))
951
+ goto bind_fail;
952
+
953
+ int store_rc = sqlite3_step(res);
954
+ if (unlikely(store_rc != SQLITE_DONE))
955
+ error_report("Failed to store host %s, rc = %d", host->hostname, rc);
956
+
957
+ rc = sqlite3_reset(res);
958
+ if (unlikely(rc != SQLITE_OK))
959
+ error_report("Failed to reset statement to store host %s, rc = %d", host->hostname, rc);
960
+
961
+ return !(store_rc == SQLITE_DONE);
962
+bind_fail:
963
+ error_report("Failed to bind parameter to store host %s, rc = %d", host->hostname, rc);
964
+ rc = sqlite3_reset(res);
965
+ if (unlikely(rc != SQLITE_OK))
966
+ error_report("Failed to reset statement to store host %s, rc = %d", host->hostname, rc);
967
+ return 1;
968
+}
969
+
970
/*
971
* Store a chart in the database
972
*/
@@ -1884,6 +1998,11 @@ void compute_chart_hash(RRDSET *st)
1998
unsigned int hash_len;
1999
char priority_str[32];
2000
2001
+ if (rrdhost_flag_check(st->rrdhost, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)) {
2002
+ internal_error(true, "Skipping compute_chart_hash for host %s because context streaming is enabled", st->rrdhost->hostname);
2003
+ return;
2004
+ }
2005
+
2006
sprintf(priority_str, "%ld", st->priority);
2007
2008
evpctx = EVP_MD_CTX_create();
@@ -2318,6 +2437,7 @@ failed:
2437
return;
2438
};
2439
2440
+
2441
#define SELECT_HOST_INFO "SELECT system_key, system_value FROM host_info WHERE host_id = @host_id;"
2442
2443
void sql_build_host_system_info(uuid_t *host_id, struct rrdhost_system_info *system_info)
@@ -2481,3 +2601,12 @@ void sql_store_host_system_info(uuid_t *host_id, const struct rrdhost_system_inf
2601
return;
2602
}
2603
2604
+// Utils
2605
+int bind_text_null(sqlite3_stmt *res, int position, const char *text, bool can_be_null)
2606
+{
2607
+ if (likely(text))
2608
+ return sqlite3_bind_text(res, position, text, -1, SQLITE_STATIC);
2609
+ if (!can_be_null)
2610
+ return 1;
2611
+ return sqlite3_bind_null(res, position);
2612
+}
database/sqlite/sqlite_functions.h
+6
-2
@@ -60,9 +60,12 @@ typedef enum db_check_action_type {
60
61
extern int sql_init_database(db_check_action_type_t rebuild, int memory);
62
extern void sql_close_database(void);
63
-
63
+extern int bind_text_null(sqlite3_stmt *res, int position, const char *text, bool can_be_null);
64
extern int sql_store_host(uuid_t *guid, const char *hostname, const char *registry_hostname, int update_every, const char *os,
65
const char *timezone, const char *tags, int hops);
66
+
67
+extern int sql_store_host_info(RRDHOST *host);
68
+
69
extern int sql_store_chart(
70
uuid_t *chart_uuid, uuid_t *host_uuid, const char *type, const char *id, const char *name, const char *family,
71
const char *context, const char *title, const char *units, const char *plugin, const char *module, long priority,
@@ -104,7 +107,8 @@ extern void compute_chart_hash(RRDSET *st);
107
extern int sql_set_dimension_option(uuid_t *dim_uuid, char *option);
108
char *get_hostname_by_node_id(char *node_id);
109
void free_temporary_host(RRDHOST *host);
107
-int init_database_batch(int rebuild, int init_type, const char *batch[]);
110
+int init_database_batch(sqlite3 *database, int rebuild, int init_type, const char *batch[]);
111
void migrate_localhost(uuid_t *host_uuid);
112
extern void sql_store_host_system_info(uuid_t *host_id, const struct rrdhost_system_info *system_info);
113
+extern void sql_build_host_system_info(uuid_t *host_id, struct rrdhost_system_info *system_info);
114
#endif //NETDATA_SQLITE_FUNCTIONS_H
health/health.d/cgroups.conf
+70
@@ -69,3 +69,73 @@ component: Network
69
info: ratio of average number of received packets for the network interface $family over the last 10 seconds, \
70
compared to the rate over the last minute
71
to: sysadmin
72
+
73
+# ---------------------------------K8s containers--------------------------------------------
74
+
75
+ template: k8s_cgroup_10min_cpu_usage
76
+ on: k8s.cgroup.cpu_limit
77
+ class: Utilization
78
+ type: Cgroups
79
+component: CPU
80
+ os: linux
81
+ hosts: *
82
+ lookup: average -10m unaligned
83
+ units: %
84
+ every: 1m
85
+ warn: $this > (($status >= $WARNING) ? (75) : (85))
86
+ crit: $this > (($status == $CRITICAL) ? (85) : (95))
87
+ delay: down 15m multiplier 1.5 max 1h
88
+ info: average cgroup CPU utilization over the last 10 minutes
89
+ to: sysadmin
90
+
91
+ template: k8s_cgroup_ram_in_use
92
+ on: k8s.cgroup.mem_usage
93
+ class: Utilization
94
+ type: Cgroups
95
+component: Memory
96
+ os: linux
97
+ hosts: *
98
+ calc: ($ram) * 100 / $memory_limit
99
+ units: %
100
+ every: 10s
101
+ warn: $this > (($status >= $WARNING) ? (80) : (90))
102
+ crit: $this > (($status == $CRITICAL) ? (90) : (98))
103
+ delay: down 15m multiplier 1.5 max 1h
104
+ info: cgroup memory utilization
105
+ to: sysadmin
106
+
107
+# check for packet storms
108
+
109
+# 1. calculate the rate packets are received in 1m: 1m_received_packets_rate
110
+# 2. do the same for the last 10s
111
+# 3. raise an alarm if the later is 10x or 20x the first
112
+# we assume the minimum packet storm should at least have
113
+# 10000 packets/s, average of the last 10 seconds
114
+
115
+ template: k8s_cgroup_1m_received_packets_rate
116
+ on: k8s.cgroup.net_packets
117
+ class: Workload
118
+ type: Cgroups
119
+component: Network
120
+ hosts: *
121
+ lookup: average -1m unaligned of received
122
+ units: packets
123
+ every: 10s
124
+ info: average number of packets received by the network interface $family over the last minute
125
+
126
+ template: k8s_cgroup_10s_received_packets_storm
127
+ on: k8s.cgroup.net_packets
128
+ class: Workload
129
+ type: Cgroups
130
+component: Network
131
+ hosts: *
132
+ lookup: average -10s unaligned of received
133
+ calc: $this * 100 / (($k8s_cgroup_10s_received_packets_storm < 1000)?(1000):($k8s_cgroup_10s_received_packets_storm))
134
+ every: 10s
135
+ units: %
136
+ warn: $this > (($status >= $WARNING)?(200):(5000))
137
+ crit: $this > (($status == $CRITICAL)?(5000):(6000))
138
+ options: no-clear-notification
139
+ info: ratio of average number of received packets for the network interface $family over the last 10 seconds, \
140
+ compared to the rate over the last minute
141
+ to: sysadmin
libnetdata/dictionary/README.md
+3
-3
@@ -97,7 +97,7 @@ This call is used to get the value of an item, given its name. It utilizes the `
97
98
For **multi-threaded** operation, the `dictionary_get()` call gets a shared read lock on the dictionary.
99
100
-In clone mode, the value returned is not guaranteed to be valid, as any other thread may delete the item from the dictionary at any time. To ensure the value will be available, use `dictionary_acquire_item()`, which uses a reference counter to defer deletes until the item is released.
100
+In clone mode, the value returned is not guaranteed to be valid, as any other thread may delete the item from the dictionary at any time. To ensure the value will be available, use `dictionary_get_and_acquire_item()`, which uses a reference counter to defer deletes until the item is released.
101
102
The format is:
103
@@ -133,7 +133,7 @@ Where:
133
134
> **IMPORTANT**<br/>There is also an **unsafe** version (without locks) of this call. This is to be used when traversing the dictionary, to delete the current item. It should never be called without an active lock on the dictionary, which can only be acquired while traversing.
135
136
-### dictionary_acquire_item()
136
+### dictionary_get_and_acquire_item()
137
138
This call can be used the search and get a dictionary item, while ensuring that it will be available for use, until `dictionary_acquired_item_release()` is called.
139
@@ -149,7 +149,7 @@ DICTIONARY *dict = dictionary_create(DICTIONARY_FLAGS_NONE);
149
dictionary_set(dict, "name", "value", 6);
150
151
// find the item we added and acquire it
152
-void *item = dictionary_acquire_item(dict, "name");
152
+void *item = dictionary_get_and_acquire_item(dict, "name");
153
154
// extract its value
155
char *value = (char *)dictionary_acquired_item_value(dict, item);
libnetdata/dictionary/dictionary.c
+662
-138
@@ -26,7 +26,11 @@ typedef struct dictionary DICTIONARY;
26
27
typedef enum name_value_flags {
28
NAME_VALUE_FLAG_NONE = 0,
29
- NAME_VALUE_FLAG_DELETED = (1 << 0), // this item is deleted
29
+ NAME_VALUE_FLAG_NAME_IS_ALLOCATED = (1 << 0), // the name pointer is a STRING
30
+ NAME_VALUE_FLAG_DELETED = (1 << 1), // this item is deleted, so it is not available for traversal
31
+ NAME_VALUE_FLAG_NEW_OR_UPDATED = (1 << 2), // this item is new or just updated (used by the react callback)
32
+
33
+ // IMPORTANT: IF YOU ADD ANOTHER FLAG, YOU NEED TO ALLOCATE ANOTHER BIT TO FLAGS IN NAME_VALUE !!!
34
} NAME_VALUE_FLAGS;
35
36
/*
@@ -38,20 +42,31 @@ typedef struct name_value {
42
avl_t avl_node;
43
#endif
44
45
+#ifdef NETDATA_INTERNAL_CHECKS
46
+ DICTIONARY *dict;
47
+#endif
48
+
49
struct name_value *next; // a double linked list to allow fast insertions and deletions
50
struct name_value *prev;
51
44
- size_t name_len; // the size of the name, including the terminating zero
45
- size_t value_len; // the size of the value (assumed binary)
52
+ uint32_t refcount; // the reference counter
53
+ uint32_t value_len:29; // the size of the value (assumed binary)
54
+ uint8_t flags:3; // the flags for this item
55
56
void *value; // the value of the dictionary item
48
- char *name; // the name of the dictionary item
49
-
50
- int refcount; // the reference counter
51
- NAME_VALUE_FLAGS flags; // the flags for this item
57
+ union {
58
+ STRING *string_name; // the name of the dictionary item
59
+ char *caller_name; // the user supplied string pointer
60
+ };
61
} NAME_VALUE;
62
63
struct dictionary {
64
+#ifdef NETDATA_INTERNAL_CHECKS
65
+ const char *creation_function;
66
+ const char *creation_file;
67
+ size_t creation_line;
68
+#endif
69
+
70
DICTIONARY_FLAGS flags; // the flags of the dictionary
71
72
NAME_VALUE *first_item; // the double linked list base pointers
@@ -60,23 +75,28 @@ struct dictionary {
75
#ifdef DICTIONARY_WITH_AVL
76
avl_tree_type values_index;
77
NAME_VALUE *hash_base;
78
+ void *(*get_thread_static_name_value)(const char *name);
79
#endif
80
81
#ifdef DICTIONARY_WITH_JUDYHS
82
Pvoid_t JudyHSArray; // the hash table
83
#endif
84
69
- netdata_rwlock_t *rwlock; // the r/w lock when DICTIONARY_FLAG_SINGLE_THREADED is not set
85
+ netdata_rwlock_t rwlock; // the r/w lock when DICTIONARY_FLAG_SINGLE_THREADED is not set
86
87
void (*ins_callback)(const char *name, void *value, void *data);
88
void *ins_callback_data;
89
90
+ void (*react_callback)(const char *name, void *value, void *data);
91
+ void *react_callback_data;
92
+
93
void (*del_callback)(const char *name, void *value, void *data);
94
void *del_callback_data;
95
96
void (*conflict_callback)(const char *name, void *old_value, void *new_value, void *data);
97
void *conflict_callback_data;
98
99
+ size_t version; // the current version of the dictionary
100
size_t inserts; // how many index insertions have been performed
101
size_t deletes; // how many index deletions have been performed
102
size_t searches; // how many index searches have been performed
@@ -88,10 +108,14 @@ struct dictionary {
108
long int pending_deletion_items; // how many items of the dictionary have been deleted, but have not been removed yet
109
int readers; // how many readers are currently using the dictionary
110
int writers; // how many writers are currently using the dictionary
111
+
112
+ size_t scratchpad_size; // the size of the scratchpad in bytes
113
+ uint8_t scratchpad[]; // variable size scratchpad requested by the caller
114
};
115
116
static inline void linkedlist_namevalue_unlink_unsafe(DICTIONARY *dict, NAME_VALUE *nv);
117
static size_t namevalue_destroy_unsafe(DICTIONARY *dict, NAME_VALUE *nv);
118
+static inline const char *namevalue_get_name(NAME_VALUE *nv);
119
120
// ----------------------------------------------------------------------------
121
// callbacks registration
@@ -111,6 +135,11 @@ void dictionary_register_conflict_callback(DICTIONARY *dict, void (*conflict_cal
135
dict->conflict_callback_data = data;
136
}
137
138
+void dictionary_register_react_callback(DICTIONARY *dict, void (*react_callback)(const char *name, void *value, void *data), void *data) {
139
+ dict->react_callback = react_callback;
140
+ dict->react_callback_data = data;
141
+}
142
+
143
// ----------------------------------------------------------------------------
144
// dictionary statistics maintenance
145
@@ -120,6 +149,9 @@ long int dictionary_stats_allocated_memory(DICTIONARY *dict) {
149
long int dictionary_stats_entries(DICTIONARY *dict) {
150
return dict->entries;
151
}
152
+size_t dictionary_stats_version(DICTIONARY *dict) {
153
+ return dict->version;
154
+}
155
size_t dictionary_stats_searches(DICTIONARY *dict) {
156
return dict->searches;
157
}
@@ -135,6 +167,9 @@ size_t dictionary_stats_resets(DICTIONARY *dict) {
167
size_t dictionary_stats_walkthroughs(DICTIONARY *dict) {
168
return dict->walkthroughs;
169
}
170
+size_t dictionary_stats_referenced_items(DICTIONARY *dict) {
171
+ return __atomic_load_n(&dict->referenced_items, __ATOMIC_SEQ_CST);
172
+}
173
174
static inline void DICTIONARY_STATS_SEARCHES_PLUS1(DICTIONARY *dict) {
175
if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
@@ -146,11 +181,13 @@ static inline void DICTIONARY_STATS_SEARCHES_PLUS1(DICTIONARY *dict) {
181
}
182
static inline void DICTIONARY_STATS_ENTRIES_PLUS1(DICTIONARY *dict, size_t size) {
183
if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
184
+ dict->version++;
185
dict->inserts++;
186
dict->entries++;
187
dict->memory += (long)size;
188
}
189
else {
190
+ __atomic_fetch_add(&dict->version, 1, __ATOMIC_SEQ_CST);
191
__atomic_fetch_add(&dict->inserts, 1, __ATOMIC_RELAXED);
192
__atomic_fetch_add(&dict->entries, 1, __ATOMIC_RELAXED);
193
__atomic_fetch_add(&dict->memory, (long)size, __ATOMIC_RELAXED);
@@ -158,10 +195,12 @@ static inline void DICTIONARY_STATS_ENTRIES_PLUS1(DICTIONARY *dict, size_t size)
195
}
196
static inline void DICTIONARY_STATS_ENTRIES_MINUS1(DICTIONARY *dict) {
197
if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
198
+ dict->version++;
199
dict->deletes++;
200
dict->entries--;
201
}
202
else {
203
+ __atomic_fetch_add(&dict->version, 1, __ATOMIC_SEQ_CST);
204
__atomic_fetch_add(&dict->deletes, 1, __ATOMIC_RELAXED);
205
__atomic_fetch_sub(&dict->entries, 1, __ATOMIC_RELAXED);
206
}
@@ -176,11 +215,13 @@ static inline void DICTIONARY_STATS_ENTRIES_MINUS_MEMORY(DICTIONARY *dict, size_
215
}
216
static inline void DICTIONARY_STATS_VALUE_RESETS_PLUS1(DICTIONARY *dict, size_t oldsize, size_t newsize) {
217
if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
218
+ dict->version++;
219
dict->resets++;
220
dict->memory += (long)newsize;
221
dict->memory -= (long)oldsize;
222
}
223
else {
224
+ __atomic_fetch_add(&dict->version, 1, __ATOMIC_SEQ_CST);
225
__atomic_fetch_add(&dict->resets, 1, __ATOMIC_RELAXED);
226
__atomic_fetch_add(&dict->memory, (long)newsize, __ATOMIC_RELAXED);
227
__atomic_fetch_sub(&dict->memory, (long)oldsize, __ATOMIC_RELAXED);
@@ -231,7 +272,7 @@ static void garbage_collect_pending_deletes_unsafe(DICTIONARY *dict) {
272
273
NAME_VALUE *nv = dict->first_item;
274
while(nv) {
234
- if(nv->flags & NAME_VALUE_FLAG_DELETED && DICTIONARY_NAME_VALUE_REFCOUNT_GET(nv) == 0) {
275
+ if((nv->flags & NAME_VALUE_FLAG_DELETED) && DICTIONARY_NAME_VALUE_REFCOUNT_GET(nv) == 0) {
276
NAME_VALUE *nv_next = nv->next;
277
278
linkedlist_namevalue_unlink_unsafe(dict, nv);
@@ -252,31 +293,30 @@ static void garbage_collect_pending_deletes_unsafe(DICTIONARY *dict) {
293
294
static inline size_t dictionary_lock_init(DICTIONARY *dict) {
295
if(likely(!(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))) {
255
- dict->rwlock = mallocz(sizeof(netdata_rwlock_t));
256
- netdata_rwlock_init(dict->rwlock);
296
+ netdata_rwlock_init(&dict->rwlock);
297
258
- if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS)
259
- dict->flags &= ~DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
298
+ if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS)
299
+ dict->flags &= ~DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
300
261
- return sizeof(netdata_rwlock_t);
301
+ return 0;
302
}
303
304
// we are single threaded
305
dict->flags |= DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
266
- dict->rwlock = NULL;
306
return 0;
307
}
308
309
static inline size_t dictionary_lock_free(DICTIONARY *dict) {
310
if(likely(!(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))) {
272
- netdata_rwlock_destroy(dict->rwlock);
273
- freez(dict->rwlock);
274
- return sizeof(netdata_rwlock_t);
311
+ netdata_rwlock_destroy(&dict->rwlock);
312
+ return 0;
313
}
314
return 0;
315
}
316
317
static void dictionary_lock(DICTIONARY *dict, char rw) {
318
+ if(rw == 'u' || rw == 'U') return;
319
+
320
if(rw == 'r' || rw == 'R') {
321
// read lock
322
__atomic_add_fetch(&dict->readers, 1, __ATOMIC_RELAXED);
@@ -291,22 +331,24 @@ static void dictionary_lock(DICTIONARY *dict, char rw) {
331
332
if(rw == 'r' || rw == 'R') {
333
// read lock
294
- netdata_rwlock_rdlock(dict->rwlock);
334
+ netdata_rwlock_rdlock(&dict->rwlock);
335
336
if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS) {
297
- internal_error(true, "DICTIONARY: left-over exclusive access to dictionary found");
337
+ internal_error(true, "DICTIONARY: left-over exclusive access to dictionary created by %s (%zu@%s) found", dict->creation_function, dict->creation_line, dict->creation_file);
338
dict->flags &= ~DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
339
}
340
}
341
else {
342
// write lock
303
- netdata_rwlock_wrlock(dict->rwlock);
343
+ netdata_rwlock_wrlock(&dict->rwlock);
344
345
dict->flags |= DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
346
}
347
}
348
349
static void dictionary_unlock(DICTIONARY *dict, char rw) {
350
+ if(rw == 'u' || rw == 'U') return;
351
+
352
if(rw == 'r' || rw == 'R') {
353
// read unlock
354
__atomic_sub_fetch(&dict->readers, 1, __ATOMIC_RELAXED);
@@ -323,7 +365,7 @@ static void dictionary_unlock(DICTIONARY *dict, char rw) {
365
if(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS)
366
dict->flags &= ~DICTIONARY_FLAG_EXCLUSIVE_ACCESS;
367
326
- netdata_rwlock_unlock(dict->rwlock);
368
+ netdata_rwlock_unlock(&dict->rwlock);
369
}
370
371
// ----------------------------------------------------------------------------
@@ -371,6 +413,13 @@ static inline size_t reference_counter_free(DICTIONARY *dict) {
413
return 0;
414
}
415
416
+static int reference_counter_increase(NAME_VALUE *nv) {
417
+ int refcount = __atomic_add_fetch(&nv->refcount, 1, __ATOMIC_SEQ_CST);
418
+ if(refcount == 1)
419
+ fatal("DICTIONARY: request to dup item '%s' but its reference counter was zero", namevalue_get_name(nv));
420
+ return refcount;
421
+}
422
+
423
static int reference_counter_acquire(DICTIONARY *dict, NAME_VALUE *nv) {
424
int refcount;
425
if(likely(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))
@@ -392,18 +441,23 @@ static int reference_counter_acquire(DICTIONARY *dict, NAME_VALUE *nv) {
441
return refcount;
442
}
443
395
-static int reference_counter_release(DICTIONARY *dict, NAME_VALUE *nv, bool can_get_write_lock) {
444
+static uint32_t reference_counter_release(DICTIONARY *dict, NAME_VALUE *nv, bool can_get_write_lock) {
445
// this function may be called without any lock on the dictionary
446
// or even when someone else has a write lock on the dictionary
447
// so, we cannot check for EXCLUSIVE ACCESS
448
400
- int refcount;
449
+ uint32_t refcount;
450
if(likely(dict->flags & DICTIONARY_FLAG_SINGLE_THREADED))
402
- refcount = --nv->refcount;
451
+ refcount = nv->refcount--;
452
else
404
- refcount = __atomic_sub_fetch(&nv->refcount, 1, __ATOMIC_SEQ_CST);
453
+ refcount = __atomic_fetch_sub(&nv->refcount, 1, __ATOMIC_SEQ_CST);
454
455
if(refcount == 0) {
456
+ internal_error(true, "DICTIONARY: attempted to release item without references: '%s' on dictionary created by %s() (%zu@%s)", namevalue_get_name(nv), dict->creation_function, dict->creation_line, dict->creation_file);
457
+ fatal("DICTIONARY: attempted to release item without references: '%s'", namevalue_get_name(nv));
458
+ }
459
+
460
+ if(refcount == 1) {
461
if((nv->flags & NAME_VALUE_FLAG_DELETED))
462
DICTIONARY_STATS_PENDING_DELETES_PLUS1(dict);
463
@@ -415,9 +469,9 @@ static int reference_counter_release(DICTIONARY *dict, NAME_VALUE *nv, bool can_
469
if(can_get_write_lock && DICTIONARY_STATS_PENDING_DELETES_GET(dict)) {
470
// we can garbage collect now
471
418
- dictionary_lock(dict, 'w');
472
+ dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
473
garbage_collect_pending_deletes_unsafe(dict);
420
- dictionary_unlock(dict, 'w');
474
+ dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
475
}
476
477
return refcount;
@@ -427,12 +481,22 @@ static int reference_counter_release(DICTIONARY *dict, NAME_VALUE *nv, bool can_
481
// hash table
482
483
#ifdef DICTIONARY_WITH_AVL
484
+static inline const char *namevalue_get_name(NAME_VALUE *nv);
485
+
486
static int name_value_compare(void* a, void* b) {
431
- return strcmp(((NAME_VALUE *)a)->name, ((NAME_VALUE *)b)->name);
487
+ return strcmp(namevalue_get_name((NAME_VALUE *)a), namevalue_get_name((NAME_VALUE *)b));
488
+}
489
+
490
+static void *get_thread_static_name_value(const char *name) {
491
+ static __thread NAME_VALUE tmp = { 0 };
492
+ tmp.flags = NAME_VALUE_FLAG_NONE;
493
+ tmp.caller_name = (char *)name;
494
+ return &tmp;
495
}
496
497
static void hashtable_init_unsafe(DICTIONARY *dict) {
498
avl_init(&dict->values_index, name_value_compare);
499
+ dict->get_thread_static_name_value = get_thread_static_name_value;
500
}
501
502
static size_t hashtable_destroy_unsafe(DICTIONARY *dict) {
@@ -440,7 +504,7 @@ static size_t hashtable_destroy_unsafe(DICTIONARY *dict) {
504
return 0;
505
}
506
443
-static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, size_t name_len, NAME_VALUE *nv) {
507
+static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, size_t name_len, void *nv) {
508
(void)name;
509
(void)name_len;
510
@@ -453,9 +517,8 @@ static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, si
517
static inline NAME_VALUE *hashtable_get_unsafe(DICTIONARY *dict, const char *name, size_t name_len) {
518
(void)name_len;
519
456
- NAME_VALUE tmp;
457
- tmp.name = (char *)name;
458
- return (NAME_VALUE *)avl_search(&(dict->values_index), (avl_t *) &tmp);
520
+ void *tmp = dict->get_thread_static_name_value(name);
521
+ return (NAME_VALUE *)avl_search(&(dict->values_index), (avl_t *)tmp);
522
}
523
524
static inline NAME_VALUE **hashtable_insert_unsafe(DICTIONARY *dict, const char *name, size_t name_len) {
@@ -469,13 +532,10 @@ static inline NAME_VALUE **hashtable_insert_unsafe(DICTIONARY *dict, const char
532
return &dict->hash_base;
533
}
534
472
-static inline void hashtable_inserted_name_value_unsafe(DICTIONARY *dict, const char *name, size_t name_len, NAME_VALUE *nv) {
535
+static inline void hashtable_inserted_name_value_unsafe(DICTIONARY *dict, void *nv) {
536
// we have our new NAME_VALUE object.
537
// Let's index it.
538
476
- (void)name;
477
- (void)name_len;
478
-
539
if(unlikely(avl_insert(&((dict)->values_index), (avl_t *)(nv)) != (avl_t *)nv))
540
error("dictionary: INTERNAL ERROR: duplicate insertion to dictionary.");
541
}
@@ -503,7 +563,7 @@ static size_t hashtable_destroy_unsafe(DICTIONARY *dict) {
563
}
564
565
static inline NAME_VALUE **hashtable_insert_unsafe(DICTIONARY *dict, const char *name, size_t name_len) {
506
- internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: inserting to the index without exclusive access to the dictionary.");
566
+ internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: inserting item from the index without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
567
568
JError_t J_Error;
569
Pvoid_t *Rc = JudyHSIns(&dict->JudyHSArray, (void *)name, name_len, &J_Error);
@@ -522,8 +582,8 @@ static inline NAME_VALUE **hashtable_insert_unsafe(DICTIONARY *dict, const char
582
return (NAME_VALUE **)Rc;
583
}
584
525
-static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, size_t name_len, NAME_VALUE *nv) {
526
- internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: deleting from the index without exclusive access to the dictionary.");
585
+static inline int hashtable_delete_unsafe(DICTIONARY *dict, const char *name, size_t name_len, void *nv) {
586
+ internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: deleting item from the index without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
587
588
(void)nv;
589
if(unlikely(!dict->JudyHSArray)) return 0;
@@ -566,10 +626,8 @@ static inline NAME_VALUE *hashtable_get_unsafe(DICTIONARY *dict, const char *nam
626
}
627
}
628
569
-static inline void hashtable_inserted_name_value_unsafe(DICTIONARY *dict, const char *name, size_t name_len, NAME_VALUE *nv) {
629
+static inline void hashtable_inserted_name_value_unsafe(DICTIONARY *dict, void *nv) {
630
(void)dict;
571
- (void)name;
572
- (void)name_len;
631
(void)nv;
632
;
633
}
@@ -580,7 +638,7 @@ static inline void hashtable_inserted_name_value_unsafe(DICTIONARY *dict, const
638
// linked list management
639
640
static inline void linkedlist_namevalue_link_unsafe(DICTIONARY *dict, NAME_VALUE *nv) {
583
- internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: adding item to the linked-list without exclusive access to the dictionary.");
641
+ internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: adding item to the linked-list without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
642
643
if (unlikely(!dict->first_item)) {
644
// we are the only ones here
@@ -609,7 +667,7 @@ static inline void linkedlist_namevalue_link_unsafe(DICTIONARY *dict, NAME_VALUE
667
}
668
669
static inline void linkedlist_namevalue_unlink_unsafe(DICTIONARY *dict, NAME_VALUE *nv) {
612
- internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: removing item from the linked-list without exclusive access to the dictionary.");
670
+ internal_error(!(dict->flags & DICTIONARY_FLAG_EXCLUSIVE_ACCESS), "DICTIONARY: removing item from the linked-list without exclusive access to the dictionary created by %s() (%zu@%s)", dict->creation_function, dict->creation_line, dict->creation_file);
671
672
if(nv->next) nv->next->prev = nv->prev;
673
if(nv->prev) nv->prev->next = nv->next;
@@ -620,6 +678,31 @@ static inline void linkedlist_namevalue_unlink_unsafe(DICTIONARY *dict, NAME_VAL
678
// ----------------------------------------------------------------------------
679
// NAME_VALUE methods
680
681
+static inline size_t namevalue_set_name(DICTIONARY *dict, NAME_VALUE *nv, const char *name, size_t name_len) {
682
+ if(likely(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE)) {
683
+ nv->caller_name = (char *)name;
684
+ return 0;
685
+ }
686
+
687
+ nv->string_name = string_strdupz(name);
688
+ nv->flags |= NAME_VALUE_FLAG_NAME_IS_ALLOCATED;
689
+ return name_len;
690
+}
691
+
692
+static inline size_t namevalue_free_name(DICTIONARY *dict, NAME_VALUE *nv) {
693
+ if(unlikely(!(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE)))
694
+ string_freez(nv->string_name);
695
+
696
+ return 0;
697
+}
698
+
699
+static inline const char *namevalue_get_name(NAME_VALUE *nv) {
700
+ if(nv->flags & NAME_VALUE_FLAG_NAME_IS_ALLOCATED)
701
+ return string2str(nv->string_name);
702
+ else
703
+ return nv->caller_name;
704
+}
705
+
706
static NAME_VALUE *namevalue_create_unsafe(DICTIONARY *dict, const char *name, size_t name_len, void *value, size_t value_len) {
707
debug(D_DICTIONARY, "Creating name value entry for name '%s'.", name);
708
@@ -627,18 +710,15 @@ static NAME_VALUE *namevalue_create_unsafe(DICTIONARY *dict, const char *name, s
710
NAME_VALUE *nv = mallocz(size);
711
size_t allocated = size;
712
713
+#ifdef NETDATA_INTERNAL_CHECKS
714
+ nv->dict = dict;
715
+#endif
716
+
717
nv->refcount = 0;
718
nv->flags = NAME_VALUE_FLAG_NONE;
632
- nv->name_len = name_len;
719
nv->value_len = value_len;
720
635
- if(likely(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE))
636
- nv->name = (char *)name;
637
- else {
638
- nv->name = mallocz(name_len);
639
- memcpy(nv->name, name, name_len);
640
- allocated += name_len;
641
- }
721
+ allocated += namevalue_set_name(dict, nv, name, name_len);
722
723
if(likely(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE))
724
nv->value = value;
@@ -667,26 +747,26 @@ static NAME_VALUE *namevalue_create_unsafe(DICTIONARY *dict, const char *name, s
747
DICTIONARY_STATS_ENTRIES_PLUS1(dict, allocated);
748
749
if(dict->ins_callback)
670
- dict->ins_callback(nv->name, nv->value, dict->ins_callback_data);
750
+ dict->ins_callback(namevalue_get_name(nv), nv->value, dict->ins_callback_data);
751
752
return nv;
753
}
754
755
static void namevalue_reset_unsafe(DICTIONARY *dict, NAME_VALUE *nv, void *value, size_t value_len) {
676
- debug(D_DICTIONARY, "Dictionary entry with name '%s' found. Changing its value.", nv->name);
756
+ debug(D_DICTIONARY, "Dictionary entry with name '%s' found. Changing its value.", namevalue_get_name(nv));
757
758
DICTIONARY_STATS_VALUE_RESETS_PLUS1(dict, nv->value_len, value_len);
759
760
if(dict->del_callback)
681
- dict->del_callback(nv->name, nv->value, dict->del_callback_data);
761
+ dict->del_callback(namevalue_get_name(nv), nv->value, dict->del_callback_data);
762
763
if(likely(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE)) {
684
- debug(D_DICTIONARY, "Dictionary: linking value to '%s'", nv->name);
764
+ debug(D_DICTIONARY, "Dictionary: linking value to '%s'", namevalue_get_name(nv));
765
nv->value = value;
766
nv->value_len = value_len;
767
}
768
else {
689
- debug(D_DICTIONARY, "Dictionary: cloning value to '%s'", nv->name);
769
+ debug(D_DICTIONARY, "Dictionary: cloning value to '%s'", namevalue_get_name(nv));
770
771
void *oldvalue = nv->value;
772
void *newvalue = NULL;
@@ -698,32 +778,31 @@ static void namevalue_reset_unsafe(DICTIONARY *dict, NAME_VALUE *nv, void *value
778
nv->value = newvalue;
779
nv->value_len = value_len;
780
701
- debug(D_DICTIONARY, "Dictionary: freeing old value of '%s'", nv->name);
781
+ debug(D_DICTIONARY, "Dictionary: freeing old value of '%s'", namevalue_get_name(nv));
782
freez(oldvalue);
783
}
784
785
if(dict->ins_callback)
706
- dict->ins_callback(nv->name, nv->value, dict->ins_callback_data);
786
+ dict->ins_callback(namevalue_get_name(nv), nv->value, dict->ins_callback_data);
787
}
788
789
static size_t namevalue_destroy_unsafe(DICTIONARY *dict, NAME_VALUE *nv) {
710
- debug(D_DICTIONARY, "Destroying name value entry for name '%s'.", nv->name);
790
+ debug(D_DICTIONARY, "Destroying name value entry for name '%s'.", namevalue_get_name(nv));
791
792
if(dict->del_callback)
713
- dict->del_callback(nv->name, nv->value, dict->del_callback_data);
793
+ dict->del_callback(namevalue_get_name(nv), nv->value, dict->del_callback_data);
794
795
size_t freed = 0;
796
797
if(unlikely(!(dict->flags & DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE))) {
718
- debug(D_DICTIONARY, "Dictionary freeing value of '%s'", nv->name);
798
+ debug(D_DICTIONARY, "Dictionary freeing value of '%s'", namevalue_get_name(nv));
799
freez(nv->value);
800
freed += nv->value_len;
801
}
802
803
if(unlikely(!(dict->flags & DICTIONARY_FLAG_NAME_LINK_DONT_CLONE))) {
724
- debug(D_DICTIONARY, "Dictionary freeing name '%s'", nv->name);
725
- freez(nv->name);
726
- freed += nv->name_len;
804
+ debug(D_DICTIONARY, "Dictionary freeing name '%s'", namevalue_get_name(nv));
805
+ freed += namevalue_free_name(dict, nv);
806
}
807
808
freez(nv);
@@ -747,16 +826,20 @@ static bool name_value_can_be_deleted(DICTIONARY *dict, NAME_VALUE *nv) {
826
827
// ----------------------------------------------------------------------------
828
// API - dictionary management
750
-
751
-DICTIONARY *dictionary_create(DICTIONARY_FLAGS flags) {
829
+#ifdef NETDATA_INTERNAL_CHECKS
830
+DICTIONARY *dictionary_create_advanced_with_trace(DICTIONARY_FLAGS flags, size_t scratchpad_size, const char *function, size_t line, const char *file) {
831
+#else
832
+DICTIONARY *dictionary_create_advanced(DICTIONARY_FLAGS flags, size_t scratchpad_size) {
833
+#endif
834
debug(D_DICTIONARY, "Creating dictionary.");
835
836
if(unlikely(flags & DICTIONARY_FLAGS_RESERVED))
837
flags &= ~DICTIONARY_FLAGS_RESERVED;
838
757
- DICTIONARY *dict = callocz(1, sizeof(DICTIONARY));
758
- size_t allocated = sizeof(DICTIONARY);
839
+ DICTIONARY *dict = callocz(1, sizeof(DICTIONARY) + scratchpad_size);
840
+ size_t allocated = sizeof(DICTIONARY) + scratchpad_size;
841
842
+ dict->scratchpad_size = scratchpad_size;
843
dict->flags = flags;
844
dict->first_item = dict->last_item = NULL;
845
@@ -765,29 +848,87 @@ DICTIONARY *dictionary_create(DICTIONARY_FLAGS flags) {
848
dict->memory = (long)allocated;
849
850
hashtable_init_unsafe(dict);
851
+
852
+#ifdef NETDATA_INTERNAL_CHECKS
853
+ dict->creation_function = function;
854
+ dict->creation_file = file;
855
+ dict->creation_line = line;
856
+#endif
857
+
858
return (DICTIONARY *)dict;
859
}
860
861
+void *dictionary_scratchpad(DICTIONARY *dict) {
862
+ return &dict->scratchpad;
863
+}
864
+
865
size_t dictionary_destroy(DICTIONARY *dict) {
866
if(!dict) return 0;
867
774
- if(dict->referenced_items) {
868
+ NAME_VALUE *nv;
869
+
870
+ debug(D_DICTIONARY, "Destroying dictionary.");
871
+
872
+ long referenced_items = 0;
873
+ size_t retries = 0;
874
+ do {
875
+ referenced_items = __atomic_load_n(&dict->referenced_items, __ATOMIC_SEQ_CST);
876
+ if (referenced_items) {
877
+ dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
878
+
879
+ // there are referenced items
880
+ // delete all items individually, so that only the referenced will remain
881
+ NAME_VALUE *nv_next;
882
+ for (nv = dict->first_item; nv; nv = nv_next) {
883
+ nv_next = nv->next;
884
+ size_t refcount = DICTIONARY_NAME_VALUE_REFCOUNT_GET(nv);
885
+ if (!refcount && !(nv->flags & NAME_VALUE_FLAG_DELETED))
886
+ dictionary_del_unsafe(dict, namevalue_get_name(nv));
887
+ }
888
+
889
+ internal_error(
890
+ retries == 0,
891
+ "DICTIONARY: waiting (try %zu) for destruction of dictionary created from %s() %zu@%s, because it has %ld referenced items in it (%ld total).",
892
+ retries + 1,
893
+ dict->creation_function,
894
+ dict->creation_line,
895
+ dict->creation_file,
896
+ referenced_items,
897
+ dict->entries);
898
+
899
+ dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
900
+ sleep_usec(10000);
901
+ }
902
+ } while(referenced_items > 0 && ++retries < 10);
903
+
904
+ if(referenced_items) {
905
+ dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
906
+
907
dict->flags |= DICTIONARY_FLAG_DESTROYED;
908
+ internal_error(
909
+ true,
910
+ "DICTIONARY: delaying destruction of dictionary created from %s() %zu@%s after %zu retries, because it has %ld referenced items in it (%ld total).",
911
+ dict->creation_function,
912
+ dict->creation_line,
913
+ dict->creation_file,
914
+ retries,
915
+ referenced_items,
916
+ dict->entries);
917
+
918
+ dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
919
return 0;
920
}
921
779
- debug(D_DICTIONARY, "Destroying dictionary.");
780
-
781
- dictionary_lock(dict, 'w');
922
+ dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
923
924
size_t freed = 0;
784
- NAME_VALUE *nv = dict->first_item;
925
+ nv = dict->first_item;
926
while (nv) {
927
// cache nv->next
928
// because we are going to free nv
788
- NAME_VALUE *nvnext = nv->next;
929
+ NAME_VALUE *nv_next = nv->next;
930
freed += namevalue_destroy_unsafe(dict, nv);
790
- nv = nvnext;
931
+ nv = nv_next;
932
// to speed up destruction, we don't
933
// unlink nv from the linked-list here
934
}
@@ -798,21 +939,20 @@ size_t dictionary_destroy(DICTIONARY *dict) {
939
// destroy the dictionary
940
freed += hashtable_destroy_unsafe(dict);
941
801
- dictionary_unlock(dict, 'w');
942
+ dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
943
freed += dictionary_lock_free(dict);
944
freed += reference_counter_free(dict);
804
-
945
+ freed += sizeof(DICTIONARY) + dict->scratchpad_size;
946
freez(dict);
806
- freed += sizeof(DICTIONARY);
947
948
return freed;
949
}
950
951
// ----------------------------------------------------------------------------
812
-// API - items management
952
+// helpers
953
814
-void *dictionary_set_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
815
- if(unlikely(!name || !*name)) {
954
+static NAME_VALUE *dictionary_set_name_value_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
955
+ if(unlikely(!name)) {
956
internal_error(true, "DICTIONARY: attempted to dictionary_set() a dictionary item without a name");
957
return NULL;
958
}
@@ -843,8 +983,9 @@ void *dictionary_set_unsafe(DICTIONARY *dict, const char *name, void *value, siz
983
if(likely(*pnv == 0)) {
984
// a new item added to the index
985
nv = *pnv = namevalue_create_unsafe(dict, name, name_len, value, value_len);
846
- hashtable_inserted_name_value_unsafe(dict, name, name_len, nv);
986
+ hashtable_inserted_name_value_unsafe(dict, nv);
987
linkedlist_namevalue_link_unsafe(dict, nv);
988
+ nv->flags |= NAME_VALUE_FLAG_NEW_OR_UPDATED;
989
}
990
else {
991
// the item is already in the index
@@ -852,25 +993,28 @@ void *dictionary_set_unsafe(DICTIONARY *dict, const char *name, void *value, siz
993
// or overwrite the value, depending on dictionary flags
994
995
nv = *pnv;
855
- if(!(dict->flags & DICTIONARY_FLAG_DONT_OVERWRITE_VALUE))
996
+
997
+ if(!(dict->flags & DICTIONARY_FLAG_DONT_OVERWRITE_VALUE)) {
998
namevalue_reset_unsafe(dict, nv, value, value_len);
999
+ nv->flags |= NAME_VALUE_FLAG_NEW_OR_UPDATED;
1000
+ }
1001
858
- else if(dict->conflict_callback)
859
- dict->conflict_callback(nv->name, nv->value, value, dict->conflict_callback_data);
860
- }
1002
+ else if(dict->conflict_callback) {
1003
+ dict->conflict_callback(namevalue_get_name(nv), nv->value, value, dict->conflict_callback_data);
1004
+ nv->flags |= NAME_VALUE_FLAG_NEW_OR_UPDATED;
1005
+ }
1006
862
- return nv->value;
863
-}
1007
+ else {
1008
+ // make sure this flag is not set
1009
+ nv->flags &= ~NAME_VALUE_FLAG_NEW_OR_UPDATED;
1010
+ }
1011
+ }
1012
865
-void *dictionary_set(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
866
- dictionary_lock(dict, 'w');
867
- void *ret = dictionary_set_unsafe(dict, name, value, value_len);
868
- dictionary_unlock(dict, 'w');
869
- return ret;
1013
+ return nv;
1014
}
1015
1016
static NAME_VALUE *dictionary_get_name_value_unsafe(DICTIONARY *dict, const char *name) {
873
- if(unlikely(!name || !*name)) {
1017
+ if(unlikely(!name)) {
1018
internal_error(true, "attempted to dictionary_get() without a name");
1019
return NULL;
1020
}
@@ -894,6 +1038,74 @@ static NAME_VALUE *dictionary_get_name_value_unsafe(DICTIONARY *dict, const char
1038
return nv;
1039
}
1040
1041
+// ----------------------------------------------------------------------------
1042
+// API - items management
1043
+
1044
+void *dictionary_set_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
1045
+ NAME_VALUE *nv = dictionary_set_name_value_unsafe(dict, name, value, value_len);
1046
+
1047
+ if(unlikely(dict->react_callback && nv && (nv->flags & NAME_VALUE_FLAG_NEW_OR_UPDATED))) {
1048
+ // we need to call the react callback with a reference counter on nv
1049
+ reference_counter_acquire(dict, nv);
1050
+ dict->react_callback(namevalue_get_name(nv), nv->value, dict->react_callback_data);
1051
+ reference_counter_release(dict, nv, false);
1052
+ }
1053
+
1054
+ return nv ? nv->value : NULL;
1055
+}
1056
+
1057
+void *dictionary_set(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
1058
+ dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
1059
+ NAME_VALUE *nv = dictionary_set_name_value_unsafe(dict, name, value, value_len);
1060
+
1061
+ // we need to get a reference counter for the react callback
1062
+ // before we unlock the dictionary
1063
+ if(unlikely(dict->react_callback && nv && (nv->flags & NAME_VALUE_FLAG_NEW_OR_UPDATED)))
1064
+ reference_counter_acquire(dict, nv);
1065
+
1066
+ dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
1067
+
1068
+ if(unlikely(dict->react_callback && nv && (nv->flags & NAME_VALUE_FLAG_NEW_OR_UPDATED))) {
1069
+ // we got the reference counter we need, above
1070
+ dict->react_callback(namevalue_get_name(nv), nv->value, dict->react_callback_data);
1071
+ reference_counter_release(dict, nv, false);
1072
+ }
1073
+
1074
+ return nv ? nv->value : NULL;
1075
+}
1076
+
1077
+DICTIONARY_ITEM *dictionary_set_and_acquire_item_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
1078
+ NAME_VALUE *nv = dictionary_set_name_value_unsafe(dict, name, value, value_len);
1079
+
1080
+ if(unlikely(!nv))
1081
+ return NULL;
1082
+
1083
+ reference_counter_acquire(dict, nv);
1084
+
1085
+ if(unlikely(dict->react_callback && (nv->flags & NAME_VALUE_FLAG_NEW_OR_UPDATED))) {
1086
+ dict->react_callback(namevalue_get_name(nv), nv->value, dict->react_callback_data);
1087
+ }
1088
+
1089
+ return (DICTIONARY_ITEM *)nv;
1090
+}
1091
+
1092
+DICTIONARY_ITEM *dictionary_set_and_acquire_item(DICTIONARY *dict, const char *name, void *value, size_t value_len) {
1093
+ dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
1094
+ NAME_VALUE *nv = dictionary_set_name_value_unsafe(dict, name, value, value_len);
1095
+
1096
+ // we need to get the reference counter before we unlock
1097
+ if(nv) reference_counter_acquire(dict, nv);
1098
+
1099
+ dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
1100
+
1101
+ if(unlikely(dict->react_callback && nv && (nv->flags & NAME_VALUE_FLAG_NEW_OR_UPDATED))) {
1102
+ // we already have a reference counter, for the caller, no need for another one
1103
+ dict->react_callback(namevalue_get_name(nv), nv->value, dict->react_callback_data);
1104
+ }
1105
+
1106
+ return (DICTIONARY_ITEM *)nv;
1107
+}
1108
+
1109
void *dictionary_get_unsafe(DICTIONARY *dict, const char *name) {
1110
NAME_VALUE *nv = dictionary_get_name_value_unsafe(dict, name);
1111
@@ -904,42 +1116,64 @@ void *dictionary_get_unsafe(DICTIONARY *dict, const char *name) {
1116
}
1117
1118
void *dictionary_get(DICTIONARY *dict, const char *name) {
907
- dictionary_lock(dict, 'r');
1119
+ dictionary_lock(dict, DICTIONARY_LOCK_READ);
1120
void *ret = dictionary_get_unsafe(dict, name);
909
- dictionary_unlock(dict, 'r');
1121
+ dictionary_unlock(dict, DICTIONARY_LOCK_READ);
1122
return ret;
1123
}
1124
913
-void *dictionary_acquire_item_unsafe(DICTIONARY *dict, const char *name) {
1125
+DICTIONARY_ITEM *dictionary_get_and_acquire_item_unsafe(DICTIONARY *dict, const char *name) {
1126
NAME_VALUE *nv = dictionary_get_name_value_unsafe(dict, name);
1127
1128
if(unlikely(!nv))
1129
return NULL;
1130
1131
reference_counter_acquire(dict, nv);
920
- return nv;
1132
+ return (DICTIONARY_ITEM *)nv;
1133
}
1134
923
-void *dictionary_acquire_item(DICTIONARY *dict, const char *name) {
924
- dictionary_lock(dict, 'r');
925
- void *ret = dictionary_acquire_item_unsafe(dict, name);
926
- dictionary_unlock(dict, 'r');
1135
+DICTIONARY_ITEM *dictionary_get_and_acquire_item(DICTIONARY *dict, const char *name) {
1136
+ dictionary_lock(dict, DICTIONARY_LOCK_READ);
1137
+ void *ret = dictionary_get_and_acquire_item_unsafe(dict, name);
1138
+ dictionary_unlock(dict, DICTIONARY_LOCK_READ);
1139
return ret;
1140
}
1141
930
-void *dictionary_acquired_item_value(DICTIONARY *dict __maybe_unused, void *item) {
1142
+DICTIONARY_ITEM *dictionary_acquired_item_dup(DICTIONARY_ITEM *item) {
1143
+ if(unlikely(!item)) return NULL;
1144
+ reference_counter_increase((NAME_VALUE *)item);
1145
+ return item;
1146
+}
1147
+
1148
+const char *dictionary_acquired_item_name(DICTIONARY_ITEM *item) {
1149
+ if(unlikely(!item)) return NULL;
1150
+ return namevalue_get_name((NAME_VALUE *)item);
1151
+}
1152
+
1153
+void *dictionary_acquired_item_value(DICTIONARY_ITEM *item) {
1154
if(unlikely(!item)) return NULL;
1155
return ((NAME_VALUE *)item)->value;
1156
}
1157
935
-void dictionary_acquired_item_release_unsafe(DICTIONARY *dict, void *item) {
1158
+void dictionary_acquired_item_release_unsafe(DICTIONARY *dict, DICTIONARY_ITEM *item) {
1159
if(unlikely(!item)) return;
1160
+
1161
+#ifdef NETDATA_INTERNAL_CHECKS
1162
+ if(((NAME_VALUE *)item)->dict != dict)
1163
+ fatal("DICTIONARY: %s(): name_value item with name '%s' does not belong to this dictionary", __FUNCTION__, namevalue_get_name((NAME_VALUE *)item));
1164
+#endif
1165
+
1166
reference_counter_release(dict, (NAME_VALUE *)item, false);
1167
}
1168
940
-void dictionary_acquired_item_release(DICTIONARY *dict, void *item) {
1169
+void dictionary_acquired_item_release(DICTIONARY *dict, DICTIONARY_ITEM *item) {
1170
if(unlikely(!item)) return;
1171
1172
+#ifdef NETDATA_INTERNAL_CHECKS
1173
+ if(((NAME_VALUE *)item)->dict != dict)
1174
+ fatal("DICTIONARY: %s(): name_value item with name '%s' does not belong to this dictionary", __FUNCTION__, namevalue_get_name((NAME_VALUE *)item));
1175
+#endif
1176
+
1177
// no need to get a lock here
1178
// we pass the last parameter to reference_counter_release() as true
1179
// so that the release may get a write-lock if required to clean up
@@ -999,9 +1233,9 @@ int dictionary_del_unsafe(DICTIONARY *dict, const char *name) {
1233
}
1234
1235
int dictionary_del(DICTIONARY *dict, const char *name) {
1002
- dictionary_lock(dict, 'w');
1236
+ dictionary_lock(dict, DICTIONARY_LOCK_WRITE);
1237
int ret = dictionary_del_unsafe(dict, name);
1004
- dictionary_unlock(dict, 'w');
1238
+ dictionary_unlock(dict, DICTIONARY_LOCK_WRITE);
1239
return ret;
1240
}
1241
@@ -1036,7 +1270,7 @@ void *dictionary_foreach_start_rw(DICTFE *dfe, DICTIONARY *dict, char rw) {
1270
1271
if(likely(nv)) {
1272
dfe->last_item = nv;
1039
- dfe->name = nv->name;
1273
+ dfe->name = (char *)namevalue_get_name(nv);
1274
dfe->value = nv->value;
1275
reference_counter_acquire(dict, nv);
1276
}
@@ -1076,7 +1310,7 @@ void *dictionary_foreach_next(DICTFE *dfe) {
1310
1311
if(likely(nv = nv_next)) {
1312
dfe->last_item = nv;
1079
- dfe->name = nv->name;
1313
+ dfe->name = (char *)namevalue_get_name(nv);
1314
dfe->value = nv->value;
1315
reference_counter_acquire(dfe->dict, nv);
1316
}
@@ -1149,7 +1383,7 @@ int dictionary_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const c
1383
// while we are using it
1384
reference_counter_acquire(dict, nv);
1385
1152
- int r = callback(nv->name, nv->value, data);
1386
+ int r = callback(namevalue_get_name(nv), nv->value, data);
1387
1388
// since we have a reference counter, this item cannot be deleted
1389
// until we release the reference counter, so the pointers are there
@@ -1175,7 +1409,7 @@ int dictionary_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const c
1409
// sorted walkthrough
1410
1411
static int dictionary_sort_compar(const void *nv1, const void *nv2) {
1178
- return strcmp((*(NAME_VALUE **)nv1)->name, (*(NAME_VALUE **)nv2)->name);
1412
+ return strcmp(namevalue_get_name((*(NAME_VALUE **)nv1)), namevalue_get_name((*(NAME_VALUE **)nv2)));
1413
}
1414
1415
int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const char *name, void *entry, void *data), void *data) {
@@ -1215,7 +1449,7 @@ int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(
1449
nv = array[i];
1450
if(likely(!(nv->flags & NAME_VALUE_FLAG_DELETED))) {
1451
reference_counter_acquire(dict, nv);
1218
- int r = callback(nv->name, nv->value, data);
1452
+ int r = callback(namevalue_get_name(nv), nv->value, data);
1453
reference_counter_release(dict, nv, false);
1454
if (r < 0) {
1455
ret = r;
@@ -1232,6 +1466,182 @@ int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(
1466
return ret;
1467
}
1468
1469
+// ----------------------------------------------------------------------------
1470
+// STRING implementation - dedup all STRINGs
1471
+
1472
+typedef struct string_entry {
1473
+#ifdef DICTIONARY_WITH_AVL
1474
+ avl_t avl_node;
1475
+#endif
1476
+ uint32_t length; // the string length with the terminating '\0'
1477
+ uint32_t refcount; // how many times this string is used
1478
+ const char str[]; // the string itself
1479
+} STRING_ENTRY;
1480
+
1481
+#ifdef DICTIONARY_WITH_AVL
1482
+static int string_entry_compare(void* a, void* b) {
1483
+ return strcmp(((STRING_ENTRY *)a)->str, ((STRING_ENTRY *)b)->str);
1484
+}
1485
+
1486
+static void *get_thread_static_string_entry(const char *name) {
1487
+ static __thread size_t _length = 0;
1488
+ static __thread STRING_ENTRY *_tmp = NULL;
1489
+
1490
+ size_t size = sizeof(STRING_ENTRY) + strlen(name) + 1;
1491
+ if(likely(_tmp && _length < size)) {
1492
+ freez(_tmp);
1493
+ _tmp = NULL;
1494
+ _length = 0;
1495
+ }
1496
+
1497
+ if(unlikely(!_tmp)) {
1498
+ _tmp = callocz(1, size);
1499
+ _length = size;
1500
+ }
1501
+
1502
+ strcpy((char *)&_tmp->str[0], name);
1503
+ return _tmp;
1504
+}
1505
+#endif
1506
+
1507
+DICTIONARY string_dictionary = {
1508
+#ifdef DICTIONARY_WITH_AVL
1509
+ .values_index = {
1510
+ .root = NULL,
1511
+ .compar = string_entry_compare
1512
+ },
1513
+ .get_thread_static_name_value = get_thread_static_string_entry,
1514
+#endif
1515
+
1516
+ .flags = DICTIONARY_FLAG_EXCLUSIVE_ACCESS,
1517
+ .rwlock = NETDATA_RWLOCK_INITIALIZER
1518
+};
1519
+
1520
+static netdata_mutex_t string_mutex = NETDATA_MUTEX_INITIALIZER;
1521
+
1522
+STRING *string_dup(STRING *string) {
1523
+ if(unlikely(!string)) return NULL;
1524
+
1525
+ STRING_ENTRY *se = (STRING_ENTRY *)string;
1526
+ netdata_mutex_lock(&string_mutex);
1527
+ se->refcount++;
1528
+ netdata_mutex_unlock(&string_mutex);
1529
+ return string;
1530
+}
1531
+
1532
+STRING *string_strdupz(const char *str) {
1533
+ if(unlikely(!str || !*str)) return NULL;
1534
+
1535
+ netdata_mutex_lock(&string_mutex);
1536
+
1537
+ size_t length = strlen(str) + 1;
1538
+ STRING_ENTRY *se;
1539
+ STRING_ENTRY **ptr = (STRING_ENTRY **)hashtable_insert_unsafe(&string_dictionary, str, length);
1540
+ if(unlikely(*ptr == 0)) {
1541
+ // a new item added to the index
1542
+ size_t mem_size = sizeof(STRING_ENTRY) + length;
1543
+ se = mallocz(mem_size);
1544
+ strcpy((char *)se->str, str);
1545
+ se->length = length;
1546
+ se->refcount = 1;
1547
+ *ptr = se;
1548
+ hashtable_inserted_name_value_unsafe(&string_dictionary, se);
1549
+ string_dictionary.version++;
1550
+ string_dictionary.inserts++;
1551
+ string_dictionary.entries++;
1552
+ string_dictionary.memory += (long)mem_size;
1553
+ }
1554
+ else {
1555
+ // the item is already in the index
1556
+ se = *ptr;
1557
+ se->refcount++;
1558
+ string_dictionary.searches++;
1559
+ }
1560
+
1561
+ netdata_mutex_unlock(&string_mutex);
1562
+ return (STRING *)se;
1563
+}
1564
+
1565
+void string_freez(STRING *string) {
1566
+ if(unlikely(!string)) return;
1567
+ netdata_mutex_lock(&string_mutex);
1568
+
1569
+ STRING_ENTRY *se = (STRING_ENTRY *)string;
1570
+
1571
+ if(se->refcount == 0)
1572
+ fatal("STRING: tried to free string that has zero references.");
1573
+
1574
+ se->refcount--;
1575
+ if(unlikely(se->refcount == 0)) {
1576
+ if(hashtable_delete_unsafe(&string_dictionary, se->str, se->length, se) == 0)
1577
+ error("STRING: INTERNAL ERROR: tried to delete '%s' that is not in the index", se->str);
1578
+
1579
+ size_t mem_size = sizeof(STRING_ENTRY) + se->length;
1580
+ freez(se);
1581
+ string_dictionary.version++;
1582
+ string_dictionary.deletes++;
1583
+ string_dictionary.entries--;
1584
+ string_dictionary.memory -= (long)mem_size;
1585
+ }
1586
+
1587
+ netdata_mutex_unlock(&string_mutex);
1588
+}
1589
+
1590
+size_t string_length(STRING *string) {
1591
+ if(unlikely(!string)) return 0;
1592
+ return ((STRING_ENTRY *)string)->length - 1;
1593
+}
1594
+
1595
+const char *string2str(STRING *string) {
1596
+ if(unlikely(!string)) return "";
1597
+ return ((STRING_ENTRY *)string)->str;
1598
+}
1599
+
1600
+STRING *string_2way_merge(STRING *a, STRING *b) {
1601
+ static STRING *X = NULL;
1602
+
1603
+ if(unlikely(!X)) {
1604
+ X = string_strdupz("[x]");
1605
+ }
1606
+
1607
+ if(unlikely(a == b)) return string_dup(a);
1608
+ if(unlikely(a == X)) return string_dup(a);
1609
+ if(unlikely(b == X)) return string_dup(b);
1610
+ if(unlikely(!a)) return string_dup(X);
1611
+ if(unlikely(!b)) return string_dup(X);
1612
+
1613
+ size_t alen = string_length(a);
1614
+ size_t blen = string_length(b);
1615
+ size_t length = alen + blen + string_length(X) + 1;
1616
+ char buf1[length + 1], buf2[length + 1], *dst1;
1617
+ const char *s1, *s2;
1618
+
1619
+ s1 = string2str(a);
1620
+ s2 = string2str(b);
1621
+ dst1 = buf1;
1622
+ for( ; *s1 && *s2 && *s1 == *s2 ;s1++, s2++)
1623
+ *dst1++ = *s1;
1624
+
1625
+ *dst1 = '\0';
1626
+
1627
+ if(*s1 != '\0' || *s2 != '\0') {
1628
+ *dst1++ = '[';
1629
+ *dst1++ = 'x';
1630
+ *dst1++ = ']';
1631
+
1632
+ s1 = &(string2str(a))[alen - 1];
1633
+ s2 = &(string2str(b))[blen - 1];
1634
+ char *dst2 = &buf2[length];
1635
+ *dst2 = '\0';
1636
+ for (; *s1 && *s2 && *s1 == *s2; s1--, s2--)
1637
+ *(--dst2) = *s1;
1638
+
1639
+ strcpy(dst1, dst2);
1640
+ }
1641
+
1642
+ return string_strdupz(buf1);
1643
+}
1644
+
1645
// ----------------------------------------------------------------------------
1646
// unit test
1647
@@ -1649,11 +2059,11 @@ static int check_name_value_callback(const char *name, void *value, void *data)
2059
return value == data;
2060
}
2061
1652
-static size_t check_name_value(DICTIONARY *dict, NAME_VALUE *nv, const char *name, const char *value, int refcount, NAME_VALUE_FLAGS flags, bool searchable, bool browsable, bool linked) {
2062
+static size_t check_name_value_deleted_flag(DICTIONARY *dict, NAME_VALUE *nv, const char *name, const char *value, unsigned refcount, NAME_VALUE_FLAGS deleted_flags, bool searchable, bool browsable, bool linked) {
2063
size_t errors = 0;
2064
1655
- fprintf(stderr, "NAME_VALUE name is '%s', expected '%s'...\t\t\t\t", nv->name, name);
1656
- if(strcmp(nv->name, name) != 0) {
2065
+ fprintf(stderr, "NAME_VALUE name is '%s', expected '%s'...\t\t\t\t", namevalue_get_name(nv), name);
2066
+ if(strcmp(namevalue_get_name(nv), name) != 0) {
2067
fprintf(stderr, "FAILED\n");
2068
errors++;
2069
}
@@ -1668,7 +2078,7 @@ static size_t check_name_value(DICTIONARY *dict, NAME_VALUE *nv, const char *nam
2078
else
2079
fprintf(stderr, "OK\n");
2080
1671
- fprintf(stderr, "NAME_VALUE refcount is %d, expected %d...\t\t\t\t\t", nv->refcount, refcount);
2081
+ fprintf(stderr, "NAME_VALUE refcount is %u, expected %u...\t\t\t\t\t", nv->refcount, refcount);
2082
if (nv->refcount != refcount) {
2083
fprintf(stderr, "FAILED\n");
2084
errors++;
@@ -1676,8 +2086,8 @@ static size_t check_name_value(DICTIONARY *dict, NAME_VALUE *nv, const char *nam
2086
else
2087
fprintf(stderr, "OK\n");
2088
1679
- fprintf(stderr, "NAME_VALUE flags is %u, expected %u...\t\t\t\t\t", nv->flags, flags);
1680
- if (nv->flags != flags) {
2089
+ fprintf(stderr, "NAME_VALUE deleted flag is %s, expected %s...\t\t\t", (nv->flags & NAME_VALUE_FLAG_DELETED)?"TRUE":"FALSE", (deleted_flags & NAME_VALUE_FLAG_DELETED)?"TRUE":"FALSE");
2090
+ if ((nv->flags & NAME_VALUE_FLAG_DELETED) != (deleted_flags & NAME_VALUE_FLAG_DELETED)) {
2091
fprintf(stderr, "FAILED\n");
2092
errors++;
2093
}
@@ -1810,77 +2220,191 @@ int dictionary_unittest(size_t entries) {
2220
// check reference counters
2221
{
2222
fprintf(stderr, "\nTesting reference counters:\n");
1813
- dict = dictionary_create(DICTIONARY_FLAG_NONE);
2223
+ dict = dictionary_create(DICTIONARY_FLAG_NONE|DICTIONARY_FLAG_NAME_LINK_DONT_CLONE);
2224
errors += check_dictionary(dict, 0, 0);
2225
2226
fprintf(stderr, "\nAdding test item to dictionary and acquiring it\n");
2227
dictionary_set(dict, "test", "ITEM1", 6);
1818
- NAME_VALUE *nv = dictionary_acquire_item(dict, "test");
2228
+ NAME_VALUE *nv = (NAME_VALUE *)dictionary_get_and_acquire_item(dict, "test");
2229
2230
errors += check_dictionary(dict, 1, 1);
1821
- errors += check_name_value(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_NONE, true, true, true);
2231
+ errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_NONE, true, true, true);
2232
2233
fprintf(stderr, "\nChecking that reference counters are increased:\n");
2234
void *t;
2235
dfe_start_read(dict, t) {
2236
errors += check_dictionary(dict, 1, 1);
1827
- errors += check_name_value(dict, nv, "test", "ITEM1", 2, NAME_VALUE_FLAG_NONE, true, true, true);
2237
+ errors +=
2238
+ check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 2, NAME_VALUE_FLAG_NONE, true, true, true);
2239
}
2240
dfe_done(t);
2241
2242
fprintf(stderr, "\nChecking that reference counters are decreased:\n");
2243
errors += check_dictionary(dict, 1, 1);
1833
- errors += check_name_value(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_NONE, true, true, true);
2244
+ errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_NONE, true, true, true);
2245
2246
fprintf(stderr, "\nDeleting the item we have acquired:\n");
2247
dictionary_del(dict, "test");
2248
2249
errors += check_dictionary(dict, 0, 1);
1839
- errors += check_name_value(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
2250
+ errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
2251
2252
fprintf(stderr, "\nAdding another item with the same name of the item we deleted, while being acquired:\n");
2253
dictionary_set(dict, "test", "ITEM2", 6);
2254
errors += check_dictionary(dict, 1, 2);
2255
2256
fprintf(stderr, "\nAcquiring the second item:\n");
1846
- NAME_VALUE *nv2 = dictionary_acquire_item(dict, "test");
1847
- errors += check_name_value(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
1848
- errors += check_name_value(dict, nv2, "test", "ITEM2", 1, NAME_VALUE_FLAG_NONE, true, true, true);
2257
+ NAME_VALUE *nv2 = (NAME_VALUE *)dictionary_get_and_acquire_item(dict, "test");
2258
+ errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
2259
+ errors += check_name_value_deleted_flag(dict, nv2, "test", "ITEM2", 1, NAME_VALUE_FLAG_NONE, true, true, true);
2260
2261
fprintf(stderr, "\nReleasing the second item (the first is still acquired):\n");
1851
- dictionary_acquired_item_release(dict, nv2);
2262
+ dictionary_acquired_item_release(dict, (DICTIONARY_ITEM *)nv2);
2263
errors += check_dictionary(dict, 1, 2);
1853
- errors += check_name_value(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
1854
- errors += check_name_value(dict, nv2, "test", "ITEM2", 0, NAME_VALUE_FLAG_NONE, true, true, true);
2264
+ errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
2265
+ errors += check_name_value_deleted_flag(dict, nv2, "test", "ITEM2", 0, NAME_VALUE_FLAG_NONE, true, true, true);
2266
2267
fprintf(stderr, "\nDeleting the second item (the first is still acquired):\n");
2268
dictionary_del(dict, "test");
2269
errors += check_dictionary(dict, 0, 1);
1859
- errors += check_name_value(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
2270
+ errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_DELETED, false, false, true);
2271
2272
fprintf(stderr, "\nReleasing the first item (which we have already deleted):\n");
1862
- dictionary_acquired_item_release(dict, nv);
2273
+ dictionary_acquired_item_release(dict, (DICTIONARY_ITEM *)nv);
2274
errors += check_dictionary(dict, 0, 0);
2275
2276
fprintf(stderr, "\nAdding again the test item to dictionary and acquiring it\n");
2277
dictionary_set(dict, "test", "ITEM1", 6);
1867
- nv = dictionary_acquire_item(dict, "test");
2278
+ nv = (NAME_VALUE *)dictionary_get_and_acquire_item(dict, "test");
2279
2280
errors += check_dictionary(dict, 1, 1);
1870
- errors += check_name_value(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_NONE, true, true, true);
2281
+ errors += check_name_value_deleted_flag(dict, nv, "test", "ITEM1", 1, NAME_VALUE_FLAG_NONE, true, true, true);
2282
2283
fprintf(stderr, "\nDestroying the dictionary while we have acquired an item\n");
2284
dictionary_destroy(dict);
2285
2286
fprintf(stderr, "Releasing the item (on a destroyed dictionary)\n");
1876
- dictionary_acquired_item_release(dict, nv);
2287
+ dictionary_acquired_item_release(dict, (DICTIONARY_ITEM *)nv);
2288
nv = NULL;
2289
dict = NULL;
2290
}
2291
2292
+ // check string
2293
+ {
2294
+ long string_entries_starting = dictionary_stats_entries(&string_dictionary);
2295
+
2296
+ fprintf(stderr, "\nChecking strings...\n");
2297
+
2298
+ STRING *s1 = string_strdupz("hello unittest");
2299
+ STRING *s2 = string_strdupz("hello unittest");
2300
+ if(s1 != s2) {
2301
+ errors++;
2302
+ fprintf(stderr, "ERROR: duplicating strings are not deduplicated\n");
2303
+ }
2304
+ else
2305
+ fprintf(stderr, "OK: duplicating string are deduplicated\n");
2306
+
2307
+ STRING *s3 = string_dup(s1);
2308
+ if(s3 != s1) {
2309
+ errors++;
2310
+ fprintf(stderr, "ERROR: cloning strings are not deduplicated\n");
2311
+ }
2312
+ else
2313
+ fprintf(stderr, "OK: cloning string are deduplicated\n");
2314
+
2315
+ STRING_ENTRY *se = (STRING_ENTRY *)s1;
2316
+ if(se->refcount != 3) {
2317
+ errors++;
2318
+ fprintf(stderr, "ERROR: string refcount is not 3\n");
2319
+ }
2320
+ else
2321
+ fprintf(stderr, "OK: string refcount is 3\n");
2322
+
2323
+ STRING *s4 = string_strdupz("world unittest");
2324
+ if(s4 == s1) {
2325
+ errors++;
2326
+ fprintf(stderr, "ERROR: string is sharing pointers on different strings\n");
2327
+ }
2328
+ else
2329
+ fprintf(stderr, "OK: string is properly handling different strings\n");
2330
+
2331
+ usec_t start_ut, end_ut;
2332
+ STRING **strings = mallocz(entries * sizeof(STRING *));
2333
+
2334
+ start_ut = now_realtime_usec();
2335
+ for(size_t i = 0; i < entries ;i++) {
2336
+ strings[i] = string_strdupz(names[i]);
2337
+ }
2338
+ end_ut = now_realtime_usec();
2339
+ fprintf(stderr, "Created %zu strings in %llu usecs\n", entries, end_ut - start_ut);
2340
+
2341
+ start_ut = now_realtime_usec();
2342
+ for(size_t i = 0; i < entries ;i++) {
2343
+ strings[i] = string_dup(strings[i]);
2344
+ }
2345
+ end_ut = now_realtime_usec();
2346
+ fprintf(stderr, "Cloned %zu strings in %llu usecs\n", entries, end_ut - start_ut);
2347
+
2348
+ start_ut = now_realtime_usec();
2349
+ for(size_t i = 0; i < entries ;i++) {
2350
+ string_freez(strings[i]);
2351
+ string_freez(strings[i]);
2352
+ }
2353
+ end_ut = now_realtime_usec();
2354
+ fprintf(stderr, "Freed %zu strings in %llu usecs\n", entries, end_ut - start_ut);
2355
+
2356
+ freez(strings);
2357
+
2358
+ if(dictionary_stats_entries(&string_dictionary) != string_entries_starting + 2) {
2359
+ errors++;
2360
+ fprintf(stderr, "ERROR: strings dictionary should have %ld items but it has %ld\n", string_entries_starting + 2, dictionary_stats_entries(&string_dictionary));
2361
+ }
2362
+ else
2363
+ fprintf(stderr, "OK: strings dictionary has 2 items\n");
2364
+ }
2365
+
2366
+ // check 2-way merge
2367
+ {
2368
+ struct testcase {
2369
+ char *src1; char *src2; char *expected;
2370
+ } tests[] = {
2371
+ { "", "", ""},
2372
+ { "a", "", "[x]"},
2373
+ { "", "a", "[x]"},
2374
+ { "a", "a", "a"},
2375
+ { "abcd", "abcd", "abcd"},
2376
+ { "foo_cs", "bar_cs", "[x]_cs"},
2377
+ { "cp_UNIQUE_INFIX_cs", "cp_unique_infix_cs", "cp_[x]_cs"},
2378
+ { "cp_UNIQUE_INFIX_ci_unique_infix_cs", "cp_unique_infix_ci_UNIQUE_INFIX_cs", "cp_[x]_cs"},
2379
+ { "foo[1234]", "foo[4321]", "foo[[x]]"},
2380
+ { NULL, NULL, NULL },
2381
+ };
2382
+
2383
+ for (struct testcase *tc = &tests[0]; tc->expected != NULL; tc++) {
2384
+ STRING *src1 = string_strdupz(tc->src1);
2385
+ STRING *src2 = string_strdupz(tc->src2);
2386
+ STRING *expected = string_strdupz(tc->expected);
2387
+
2388
+ STRING *result = string_2way_merge(src1, src2);
2389
+ if (string_cmp(result, expected) != 0) {
2390
+ fprintf(stderr, "string_2way_merge(\"%s\", \"%s\") -> \"%s\" (expected=\"%s\")\n",
2391
+ string2str(src1),
2392
+ string2str(src2),
2393
+ string2str(result),
2394
+ string2str(expected));
2395
+ errors++;
2396
+ }
2397
+
2398
+ string_freez(src1);
2399
+ string_freez(src2);
2400
+ string_freez(expected);
2401
+ string_freez(result);
2402
+ }
2403
+ }
2404
+
2405
dictionary_unittest_free_char_pp(names, entries);
2406
dictionary_unittest_free_char_pp(values, entries);
2407
2408
fprintf(stderr, "\n%zu errors found\n", errors);
1885
- return (int)errors;
2409
+ return errors ? 1 : 0;
2410
}
libnetdata/dictionary/dictionary.h
+67
-20
@@ -35,26 +35,34 @@
35
*
36
*/
37
38
-#ifndef DICTIONARY_INTERNALS
39
-typedef void DICTIONARY;
40
-#endif
38
+typedef struct dictionary DICTIONARY;
39
+typedef struct dictionary_item DICTIONARY_ITEM;
40
41
typedef enum dictionary_flags {
43
- DICTIONARY_FLAG_NONE = 0, // the default is the opposite of all below
44
- DICTIONARY_FLAG_SINGLE_THREADED = (1 << 0), // don't use any locks (default: use locks)
45
- DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE = (1 << 1), // don't copy the value, just point to the one provided (default: copy)
46
- DICTIONARY_FLAG_NAME_LINK_DONT_CLONE = (1 << 2), // don't copy the name, just point to the one provided (default: copy)
47
- DICTIONARY_FLAG_DONT_OVERWRITE_VALUE = (1 << 3), // don't overwrite values of dictionary items (default: overwrite)
48
- DICTIONARY_FLAG_ADD_IN_FRONT = (1 << 4), // add dictionary items at the front of the linked list (default: at the end)
42
+ DICTIONARY_FLAG_NONE = 0, // the default is the opposite of all below
43
+ DICTIONARY_FLAG_SINGLE_THREADED = (1 << 0), // don't use any locks (default: use locks)
44
+ DICTIONARY_FLAG_VALUE_LINK_DONT_CLONE = (1 << 1), // don't copy the value, just point to the one provided (default: copy)
45
+ DICTIONARY_FLAG_NAME_LINK_DONT_CLONE = (1 << 2), // don't copy the name, just point to the one provided (default: copy)
46
+ DICTIONARY_FLAG_DONT_OVERWRITE_VALUE = (1 << 3), // don't overwrite values of dictionary items (default: overwrite)
47
+ DICTIONARY_FLAG_ADD_IN_FRONT = (1 << 4), // add dictionary items at the front of the linked list (default: at the end)
48
49
// to change the value of the following, you also need to change the corresponding #defines in dictionary.c
51
- DICTIONARY_FLAG_RESERVED1 = (1 << 29), // reserved for DICTIONARY_FLAG_EXCLUSIVE_ACCESS
52
- DICTIONARY_FLAG_RESERVED2 = (1 << 30), // reserved for DICTIONARY_FLAG_DESTROYED
53
- DICTIONARY_FLAG_RESERVED3 = (1 << 31), // reserved for DICTIONARY_FLAG_DEFER_ALL_DELETIONS
50
+ DICTIONARY_FLAG_RESERVED1 = (1 << 29), // reserved for DICTIONARY_FLAG_EXCLUSIVE_ACCESS
51
+ DICTIONARY_FLAG_RESERVED2 = (1 << 30), // reserved for DICTIONARY_FLAG_DESTROYED
52
+ DICTIONARY_FLAG_RESERVED3 = (1 << 31), // reserved for DICTIONARY_FLAG_DEFER_ALL_DELETIONS
53
} DICTIONARY_FLAGS;
54
55
// Create a dictionary
57
-extern DICTIONARY *dictionary_create(DICTIONARY_FLAGS flags);
56
+#ifdef NETDATA_INTERNAL_CHECKS
57
+#define dictionary_create(flags) dictionary_create_advanced_with_trace(flags, 0, __FUNCTION__, __LINE__, __FILE__);
58
+#define dictionary_create_advanced(flags) dictionary_create_advanced_with_trace(flags, 0, __FUNCTION__, __LINE__, __FILE__);
59
+extern DICTIONARY *dictionary_create_advanced_with_trace(DICTIONARY_FLAGS flags, size_t scratchpad_size, const char *function, size_t line, const char *file);
60
+#else
61
+#define dictionary_create(flags) dictionary_create_advanced(flags, 0);
62
+extern DICTIONARY *dictionary_create_advanced(DICTIONARY_FLAGS flags, size_t scratchpad_size);
63
+#endif
64
+
65
+extern void *dictionary_scratchpad(DICTIONARY *dict);
66
67
// an insert callback to be called just after an item is added to the dictionary
68
// this callback is called while the dictionary is write locked!
@@ -69,6 +77,11 @@ extern void dictionary_register_delete_callback(DICTIONARY *dict, void (*del_cal
77
// the old_value will remain in the dictionary - the new_value is ignored
78
extern void dictionary_register_conflict_callback(DICTIONARY *dict, void (*conflict_callback)(const char *name, void *old_value, void *new_value, void *data), void *data);
79
80
+// a reaction callback to be called after every item insertion or conflict
81
+// after the constructors have finished and the items are fully available for use
82
+// and the dictionary is not write locked anymore
83
+extern void dictionary_register_react_callback(DICTIONARY *dict, void (*react_callback)(const char *name, void *value, void *data), void *data);
84
+
85
// Destroy a dictionary
86
// returns the number of bytes freed
87
// the returned value will not include name and value sizes if DICTIONARY_FLAG_WITH_STATISTICS is not set
@@ -99,11 +112,18 @@ extern void *dictionary_get(DICTIONARY *dict, const char *name);
112
// returns -1 if the item was not found in the index
113
extern int dictionary_del(DICTIONARY *dict, const char *name);
114
102
-extern void *dictionary_acquire_item_unsafe(DICTIONARY *dict, const char *name);
103
-extern void *dictionary_acquire_item(DICTIONARY *dict, const char *name);
104
-extern void *dictionary_acquired_item_value(DICTIONARY *dict, void *item);
105
-extern void dictionary_acquired_item_release(DICTIONARY *dict, void *item);
106
-extern void dictionary_acquired_item_release_unsafe(DICTIONARY *dict, void *item);
115
+extern DICTIONARY_ITEM *dictionary_get_and_acquire_item_unsafe(DICTIONARY *dict, const char *name);
116
+extern DICTIONARY_ITEM *dictionary_get_and_acquire_item(DICTIONARY *dict, const char *name);
117
+
118
+extern DICTIONARY_ITEM *dictionary_set_and_acquire_item_unsafe(DICTIONARY *dict, const char *name, void *value, size_t value_len);
119
+extern DICTIONARY_ITEM *dictionary_set_and_acquire_item(DICTIONARY *dict, const char *name, void *value, size_t value_len);
120
+
121
+extern void dictionary_acquired_item_release_unsafe(DICTIONARY *dict, DICTIONARY_ITEM *item);
122
+extern void dictionary_acquired_item_release(DICTIONARY *dict, DICTIONARY_ITEM *item);
123
+
124
+extern DICTIONARY_ITEM *dictionary_acquired_item_dup(DICTIONARY_ITEM *item);
125
+extern const char *dictionary_acquired_item_name(DICTIONARY_ITEM *item);
126
+extern void *dictionary_acquired_item_value(DICTIONARY_ITEM *item);
127
128
// UNSAFE functions, without locks
129
// to be used when the user is traversing with the right lock type
@@ -159,6 +179,10 @@ int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(
179
#define DICTFE_CONST const
180
#endif
181
182
+#define DICTIONARY_LOCK_READ 'r'
183
+#define DICTIONARY_LOCK_WRITE 'w'
184
+#define DICTIONARY_LOCK_NONE 'u'
185
+
186
typedef DICTFE_CONST struct dictionary_foreach {
187
DICTFE_CONST char *name; // the dictionary name of the last item used
188
void *value; // the dictionary value of the last item used
@@ -171,8 +195,8 @@ typedef DICTFE_CONST struct dictionary_foreach {
195
void *last_item; // the item we work on, to remember the position we are at
196
} DICTFE;
197
174
-#define dfe_start_read(dict, value) dfe_start_rw(dict, value, 'r')
175
-#define dfe_start_write(dict, value) dfe_start_rw(dict, value, 'w')
198
+#define dfe_start_read(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_READ)
199
+#define dfe_start_write(dict, value) dfe_start_rw(dict, value, DICTIONARY_LOCK_WRITE)
200
#define dfe_start_rw(dict, value, mode) \
201
do { \
202
DICTFE value ## _dfe = {}; \
@@ -194,12 +218,35 @@ extern usec_t dictionary_foreach_done(DICTFE *dfe);
218
// Get statistics about the dictionary
219
extern long int dictionary_stats_allocated_memory(DICTIONARY *dict);
220
extern long int dictionary_stats_entries(DICTIONARY *dict);
221
+extern size_t dictionary_stats_version(DICTIONARY *dict);
222
extern size_t dictionary_stats_inserts(DICTIONARY *dict);
223
extern size_t dictionary_stats_searches(DICTIONARY *dict);
224
extern size_t dictionary_stats_deletes(DICTIONARY *dict);
225
extern size_t dictionary_stats_resets(DICTIONARY *dict);
226
extern size_t dictionary_stats_walkthroughs(DICTIONARY *dict);
227
+extern size_t dictionary_stats_referenced_items(DICTIONARY *dict);
228
229
extern int dictionary_unittest(size_t entries);
230
231
+// ----------------------------------------------------------------------------
232
+// STRING implementation
233
+
234
+typedef struct netdata_string STRING;
235
+extern STRING *string_strdupz(const char *str);
236
+extern STRING *string_dup(STRING *string);
237
+extern void string_freez(STRING *string);
238
+extern size_t string_length(STRING *string);
239
+extern const char *string2str(STRING *string) NEVERNULL;
240
+
241
+// keep common prefix/suffix and replace everything else with [x]
242
+extern STRING *string_2way_merge(STRING *a, STRING *b);
243
+
244
+static inline int string_cmp(STRING *s1, STRING *s2) {
245
+ // STRINGs are deduplicated, so the same strings have the same pointer
246
+ if(unlikely(s1 == s2)) return 0;
247
+
248
+ // they differ, do the typical comparison
249
+ return strcmp(string2str(s1), string2str(s2));
250
+}
251
+
252
#endif /* NETDATA_DICTIONARY_H */
streaming/receiver.c
+5
@@ -517,6 +517,7 @@ static int rrdpush_receive(struct receiver_state *rpt)
517
, rrdpush_api_key
518
, rrdpush_send_charts_matching
519
, rpt->system_info
520
+ , 0
521
);
522
523
if(!rpt->host) {
@@ -706,6 +707,8 @@ static int rrdpush_receive(struct receiver_state *rpt)
707
aclk_host_state_update(rpt->host, 1);
708
#endif
709
710
+ rrdcontext_host_child_connected(rpt->host);
711
+
712
size_t count = streaming_parser(rpt, &cd, fp);
713
714
log_stream_connection(rpt->client_ip, rpt->client_port, rpt->key, rpt->host->machine_guid, rpt->hostname,
@@ -713,6 +716,8 @@ static int rrdpush_receive(struct receiver_state *rpt)
716
error("STREAM %s [receive from [%s]:%s]: disconnected (completed %zu updates).", rpt->hostname, rpt->client_ip,
717
rpt->client_port, count);
718
719
+ rrdcontext_host_child_disconnected(rpt->host);
720
+
721
#ifdef ENABLE_ACLK
722
// in case we have cloud connection we inform cloud
723
// new child connected
streaming/rrdpush.c
+3
@@ -489,6 +489,9 @@ struct rrdpush_destinations *destinations_init(const char *dests) {
489
// The sender mutex guards thread creation, any spurious data is wiped on reconnection.
490
void rrdpush_sender_thread_stop(RRDHOST *host) {
491
492
+ if (!host->sender)
493
+ return;
494
+
495
netdata_mutex_lock(&host->sender->mutex);
496
netdata_thread_t thr = 0;
497
streaming/rrdpush.h
+1
-1
@@ -160,7 +160,7 @@ extern char *default_rrdpush_api_key;
160
extern char *default_rrdpush_send_charts_matching;
161
extern unsigned int remote_clock_resync_iterations;
162
163
-extern void sender_init(struct sender_state *s, RRDHOST *parent);
163
+extern void sender_init(RRDHOST *parent);
164
extern struct rrdpush_destinations *destinations_init(const char *destinations);
165
void sender_start(struct sender_state *s);
166
void sender_commit(struct sender_state *s);
streaming/sender.c
+12
-8
@@ -724,17 +724,21 @@ static void rrdpush_sender_thread_cleanup_callback(void *ptr) {
724
netdata_mutex_unlock(&host->sender->mutex);
725
}
726
727
-void sender_init(struct sender_state *s, RRDHOST *parent) {
728
- memset(s, 0, sizeof(*s));
729
- s->host = parent;
730
- s->buffer = cbuffer_new(1024, 1024*1024);
731
- s->build = buffer_create(1);
727
+void sender_init(RRDHOST *parent)
728
+{
729
+ if (parent->sender)
730
+ return;
731
+
732
+ parent->sender = callocz(1, sizeof(*parent->sender));
733
+ parent->sender->host = parent;
734
+ parent->sender->buffer = cbuffer_new(1024, 1024*1024);
735
+ parent->sender->build = buffer_create(1);
736
#ifdef ENABLE_COMPRESSION
733
- s->rrdpush_compression = default_compression_enabled;
737
+ parent->sender->rrdpush_compression = default_compression_enabled;
738
if (default_compression_enabled)
735
- s->compressor = create_compressor();
739
+ parent->sender->compressor = create_compressor();
740
#endif
737
- netdata_mutex_init(&s->mutex);
741
+ netdata_mutex_init(&parent->sender->mutex);
742
}
743
744
void *rrdpush_sender_thread(void *ptr) {
web/api/netdata-swagger.json
+299
-3
@@ -48,7 +48,7 @@
48
"/chart": {
49
"get": {
50
"summary": "Get info about a specific chart",
51
- "description": "The Chart endpoint returns detailed information about a chart.",
51
+ "description": "The chart endpoint returns detailed information about a chart.",
52
"parameters": [
53
{
54
"name": "chart",
@@ -82,6 +82,219 @@
82
}
83
}
84
},
85
+ "/contexts": {
86
+ "get": {
87
+ "summary": "Get a list of all contexts available at the server",
88
+ "description": "The contexts endpoint returns a summary about all contexts stored in the netdata server.",
89
+ "parameters": [
90
+ {
91
+ "name": "options",
92
+ "in": "query",
93
+ "description": "Options that affect data generation.",
94
+ "required": false,
95
+ "allowEmptyValue": true,
96
+ "schema": {
97
+ "type": "array",
98
+ "items": {
99
+ "type": "string",
100
+ "enum": [
101
+ "full",
102
+ "all",
103
+ "charts",
104
+ "dimensions",
105
+ "labels",
106
+ "uuids",
107
+ "queue",
108
+ "flags",
109
+ "deleted",
110
+ "deepscan"
111
+ ]
112
+ },
113
+ "default": [
114
+ "full"
115
+ ]
116
+ }
117
+ },
118
+ {
119
+ "name": "after",
120
+ "in": "query",
121
+ "description": "limit the results on context having data after this timestamp.",
122
+ "required": false,
123
+ "schema": {
124
+ "type": "number",
125
+ "format": "integer"
126
+ }
127
+ },
128
+ {
129
+ "name": "before",
130
+ "in": "query",
131
+ "description": "limit the results on context having data before this timestamp.",
132
+ "required": false,
133
+ "schema": {
134
+ "type": "number",
135
+ "format": "integer"
136
+ }
137
+ },
138
+ {
139
+ "name": "chart_label_key",
140
+ "in": "query",
141
+ "description": "a simple pattern matching charts label keys (use comma or pipe as separator)",
142
+ "required": false,
143
+ "allowEmptyValue": true,
144
+ "schema": {
145
+ "type": "string"
146
+ }
147
+ },
148
+ {
149
+ "name": "chart_labels_filter",
150
+ "in": "query",
151
+ "description": "a simple pattern matching charts label key and values (use colon for equality, comma or pipe as separator)",
152
+ "required": false,
153
+ "allowEmptyValue": true,
154
+ "schema": {
155
+ "type": "string"
156
+ }
157
+ },
158
+ {
159
+ "name": "dimensions",
160
+ "in": "query",
161
+ "description": "a simple pattern matching dimensions (use comma or pipe as separator)",
162
+ "required": false,
163
+ "allowEmptyValue": true,
164
+ "schema": {
165
+ "type": "string"
166
+ }
167
+ }
168
+ ],
169
+ "responses": {
170
+ "200": {
171
+ "description": "An array of contexts.",
172
+ "content": {
173
+ "application/json": {
174
+ "schema": {
175
+ "$ref": "#/components/schemas/context_summary"
176
+ }
177
+ }
178
+ }
179
+ }
180
+ }
181
+ }
182
+ },
183
+ "/context": {
184
+ "get": {
185
+ "summary": "Get info about a specific context",
186
+ "description": "The context endpoint returns detailed information about a given context.",
187
+ "parameters": [
188
+ {
189
+ "name": "context",
190
+ "in": "query",
191
+ "description": "The id of the context as returned by the /contexts call.",
192
+ "required": true,
193
+ "schema": {
194
+ "type": "string",
195
+ "format": "as returned by /contexts",
196
+ "default": "system.cpu"
197
+ }
198
+ },
199
+ {
200
+ "name": "options",
201
+ "in": "query",
202
+ "description": "Options that affect data generation.",
203
+ "required": false,
204
+ "allowEmptyValue": true,
205
+ "schema": {
206
+ "type": "array",
207
+ "items": {
208
+ "type": "string",
209
+ "enum": [
210
+ "full",
211
+ "all",
212
+ "charts",
213
+ "dimensions",
214
+ "labels",
215
+ "uuids",
216
+ "queue",
217
+ "flags",
218
+ "deleted",
219
+ "deepscan"
220
+ ]
221
+ },
222
+ "default": [
223
+ "full"
224
+ ]
225
+ }
226
+ },
227
+ {
228
+ "name": "after",
229
+ "in": "query",
230
+ "description": "limit the results on context having data after this timestamp.",
231
+ "required": false,
232
+ "schema": {
233
+ "type": "number",
234
+ "format": "integer"
235
+ }
236
+ },
237
+ {
238
+ "name": "before",
239
+ "in": "query",
240
+ "description": "limit the results on context having data before this timestamp.",
241
+ "required": false,
242
+ "schema": {
243
+ "type": "number",
244
+ "format": "integer"
245
+ }
246
+ },
247
+ {
248
+ "name": "chart_label_key",
249
+ "in": "query",
250
+ "description": "a simple pattern matching charts label keys (use comma or pipe as separator)",
251
+ "required": false,
252
+ "allowEmptyValue": true,
253
+ "schema": {
254
+ "type": "string"
255
+ }
256
+ },
257
+ {
258
+ "name": "chart_labels_filter",
259
+ "in": "query",
260
+ "description": "a simple pattern matching charts label key and values (use colon for equality, comma or pipe as separator)",
261
+ "required": false,
262
+ "allowEmptyValue": true,
263
+ "schema": {
264
+ "type": "string"
265
+ }
266
+ },
267
+ {
268
+ "name": "dimensions",
269
+ "in": "query",
270
+ "description": "a simple pattern matching dimensions (use comma or pipe as separator)",
271
+ "required": false,
272
+ "allowEmptyValue": true,
273
+ "schema": {
274
+ "type": "string"
275
+ }
276
+ }
277
+ ],
278
+ "responses": {
279
+ "200": {
280
+ "description": "A javascript object with detailed information about the context.",
281
+ "content": {
282
+ "application/json": {
283
+ "schema": {
284
+ "$ref": "#/components/schemas/context"
285
+ }
286
+ }
287
+ }
288
+ },
289
+ "400": {
290
+ "description": "No context id was supplied in the request."
291
+ },
292
+ "404": {
293
+ "description": "No context with the given id is found."
294
+ }
295
+ }
296
+ }
297
+ },
298
"/alarm_variables": {
299
"get": {
300
"summary": "List variables available to configure alarms for a chart",
@@ -924,7 +1137,7 @@
1137
"/alarms_values": {
1138
"get": {
1139
"summary": "Get a list of active or raised alarms on the server",
927
- "description": "The alarms_values endpoint returns the list of all raised or enabled alarms on the netdata server. Called without any parameters, the raised alarms in state WARNING or CRITICAL are returned. By passing \"?all\", all the enabled alarms are returned. This option output differs from `/alarms` in the number of variables delivered. This endpoint gives to user `id`, `value`, `last_updated` time, and alarm `status`.",
1140
+ "description": "The alarms_values endpoint returns the list of all raised or enabled alarms on the netdata server. Called without any parameters, the raised alarms in state WARNING or CRITICAL are returned. By passing '?all', all the enabled alarms are returned. This option output differs from `/alarms` in the number of variables delivered. This endpoint gives to user `id`, `value`, `last_updated` time, and alarm `status`.",
1141
"parameters": [
1142
{
1143
"name": "all",
@@ -1131,7 +1344,7 @@
1344
"/aclk": {
1345
"get": {
1346
"summary": "Get information about current ACLK state",
1134
- "description": "aclk endpoint returns detailed information about current state of ACLK (Agent to Cloud communication).",
1347
+ "description": "ACLK endpoint returns detailed information about current state of ACLK (Agent to Cloud communication).",
1348
"responses": {
1349
"200": {
1350
"description": "JSON object with ACLK information.",
@@ -1678,6 +1891,89 @@
1891
}
1892
}
1893
},
1894
+ "context_summary": {
1895
+ "type": "object",
1896
+ "properties": {
1897
+ "hostname": {
1898
+ "type": "string",
1899
+ "description": "The hostname of the netdata server."
1900
+ },
1901
+ "machine_guid": {
1902
+ "type": "string",
1903
+ "description": "The unique installation id of this netdata server."
1904
+ },
1905
+ "node_id": {
1906
+ "type": "string",
1907
+ "description": "The unique node id of this netdata server at the hub.",
1908
+ "example": "nightly"
1909
+ },
1910
+ "claim_id": {
1911
+ "type": "string",
1912
+ "description": "The unique handshake id of this netdata server and the hub."
1913
+ },
1914
+ "host_labels": {
1915
+ "type": "object",
1916
+ "description": "The host labels associated with this netdata server."
1917
+ },
1918
+ "context": {
1919
+ "type": "object",
1920
+ "description": "An object containing all the context objects available at the netdata server. This is used as an indexed array. The key of each context object is the id of the context.",
1921
+ "additionalProperties": {
1922
+ "$ref": "#/components/schemas/context"
1923
+ }
1924
+ }
1925
+ }
1926
+ },
1927
+ "context": {
1928
+ "type": "object",
1929
+ "properties": {
1930
+ "version": {
1931
+ "type": "string",
1932
+ "description": "The version of this context. The number are not sequential, but bigger numbers depict a newer object."
1933
+ },
1934
+ "hub_version": {
1935
+ "type": "string",
1936
+ "description": "The version of this context, as known by hub."
1937
+ },
1938
+ "family": {
1939
+ "type": "string",
1940
+ "description": "The family of the context. When multiple charts of a context have different families, the netdata server replaces the different parts with [x], so that the context can have only one family."
1941
+ },
1942
+ "title": {
1943
+ "type": "string",
1944
+ "description": "The title of the context. When multiple charts of a context have different titles, the netdata server replaces the different parts with [x], so that the context can have only one title."
1945
+ },
1946
+ "priority": {
1947
+ "type": "number",
1948
+ "description": "The relative priority of the context. When multiple contexts have different priorities, the minimum among them is selected as the priority of the context."
1949
+ },
1950
+ "units": {
1951
+ "type": "string",
1952
+ "description": "The unit of measurement for the values of all dimensions of the context. If multiple charts of context have different units, the latest collected is selected."
1953
+ },
1954
+ "chart_type": {
1955
+ "type": "string",
1956
+ "description": "The chart type.",
1957
+ "enum": [
1958
+ "line",
1959
+ "area",
1960
+ "stacked"
1961
+ ]
1962
+ },
1963
+ "first_time_t": {
1964
+ "type": "number",
1965
+ "description": "The UNIX timestamp of the first entry (the oldest) in the database."
1966
+ },
1967
+ "last_time_t": {
1968
+ "type": "number",
1969
+ "description": "The UNIX timestamp of the latest entry in the database."
1970
+ },
1971
+ "charts": {
1972
+ "type": "object",
1973
+ "description": "An object containing all the charts available for the chart. This is used as an indexed array. For each pair in the dictionary, the key is the id of the chart and the value provides all details about the chart."
1974
+ }
1975
+ }
1976
+ },
1977
"alarm_variables": {
1978
"type": "object",
1979
"properties": {
web/api/netdata-swagger.yaml
+261
-21
@@ -43,7 +43,7 @@ paths:
43
/chart:
44
get:
45
summary: Get info about a specific chart
46
- description: The Chart endpoint returns detailed information about a chart.
46
+ description: The chart endpoint returns detailed information about a chart.
47
parameters:
48
- name: chart
49
in: query
@@ -64,6 +64,159 @@ paths:
64
description: No chart id was supplied in the request.
65
"404":
66
description: No chart with the given id is found.
67
+ /contexts:
68
+ get:
69
+ summary: Get a list of all contexts available at the server
70
+ description: The contexts endpoint returns a summary about all contexts stored in the
71
+ netdata server.
72
+ parameters:
73
+ - name: options
74
+ in: query
75
+ description: Options that affect data generation.
76
+ required: false
77
+ allowEmptyValue: true
78
+ schema:
79
+ type: array
80
+ items:
81
+ type: string
82
+ enum:
83
+ - full
84
+ - all
85
+ - charts
86
+ - dimensions
87
+ - labels
88
+ - uuids
89
+ - queue
90
+ - flags
91
+ - deleted
92
+ - deepscan
93
+ default:
94
+ - full
95
+ - name: after
96
+ in: query
97
+ description: limit the results on context having data after this timestamp.
98
+ required: false
99
+ schema:
100
+ type: number
101
+ format: integer
102
+ - name: before
103
+ in: query
104
+ description: limit the results on context having data before this timestamp.
105
+ required: false
106
+ schema:
107
+ type: number
108
+ format: integer
109
+ - name: chart_label_key
110
+ in: query
111
+ description: a simple pattern matching charts label keys (use comma or pipe as separator)
112
+ required: false
113
+ allowEmptyValue: true
114
+ schema:
115
+ type: string
116
+ - name: chart_labels_filter
117
+ in: query
118
+ description: "a simple pattern matching charts label key and values (use colon for equality, comma or pipe
119
+ as separator)"
120
+ required: false
121
+ allowEmptyValue: true
122
+ schema:
123
+ type: string
124
+ - name: dimensions
125
+ in: query
126
+ description: a simple pattern matching dimensions (use comma or pipe as separator)
127
+ required: false
128
+ allowEmptyValue: true
129
+ schema:
130
+ type: string
131
+ responses:
132
+ "200":
133
+ description: An array of contexts.
134
+ content:
135
+ application/json:
136
+ schema:
137
+ $ref: "#/components/schemas/context_summary"
138
+ /context:
139
+ get:
140
+ summary: Get info about a specific context
141
+ description: The context endpoint returns detailed information about a given context.
142
+ parameters:
143
+ - name: context
144
+ in: query
145
+ description: The id of the context as returned by the /contexts call.
146
+ required: true
147
+ schema:
148
+ type: string
149
+ format: as returned by /contexts
150
+ default: system.cpu
151
+ - name: options
152
+ in: query
153
+ description: Options that affect data generation.
154
+ required: false
155
+ allowEmptyValue: true
156
+ schema:
157
+ type: array
158
+ items:
159
+ type: string
160
+ enum:
161
+ - full
162
+ - all
163
+ - charts
164
+ - dimensions
165
+ - labels
166
+ - uuids
167
+ - queue
168
+ - flags
169
+ - deleted
170
+ - deepscan
171
+ default:
172
+ - full
173
+ - name: after
174
+ in: query
175
+ description: limit the results on context having data after this timestamp.
176
+ required: false
177
+ schema:
178
+ type: number
179
+ format: integer
180
+ - name: before
181
+ in: query
182
+ description: limit the results on context having data before this timestamp.
183
+ required: false
184
+ schema:
185
+ type: number
186
+ format: integer
187
+ - name: chart_label_key
188
+ in: query
189
+ description: a simple pattern matching charts label keys (use comma or pipe as separator)
190
+ required: false
191
+ allowEmptyValue: true
192
+ schema:
193
+ type: string
194
+ - name: chart_labels_filter
195
+ in: query
196
+ description: "a simple pattern matching charts label key and values (use colon for equality, comma or pipe
197
+ as separator)"
198
+ required: false
199
+ allowEmptyValue: true
200
+ schema:
201
+ type: string
202
+ - name: dimensions
203
+ in: query
204
+ description: a simple pattern matching dimensions (use comma or pipe as separator)
205
+ required: false
206
+ allowEmptyValue: true
207
+ schema:
208
+ type: string
209
+ responses:
210
+ "200":
211
+ description: A javascript object with detailed information about the context.
212
+ content:
213
+ application/json:
214
+ schema:
215
+ $ref: "#/components/schemas/context"
216
+ "400":
217
+ description: No context id was supplied in the request.
218
+ "404":
219
+ description: No context with the given id is found.
220
/alarm_variables:
221
get:
222
summary: List variables available to configure alarms for a chart
@@ -461,7 +614,9 @@ paths:
614
format: any text
615
- name: label_color
616
in: query
464
- description: A color to be used for the background of the label side(left side) of the badge. One of predefined colors or specific color in hex `RGB` or `RRGGBB` format (without preceding `#` character). If value wrong or not given default color will be used.
617
+ description: "A color to be used for the background of the label side(left side) of the badge.
618
+ One of predefined colors or specific color in hex `RGB` or `RRGGBB` format (without preceding `#` character).
619
+ If value wrong or not given default color will be used."
620
required: false
621
allowEmptyValue: true
622
schema:
@@ -486,7 +641,8 @@ paths:
641
description: "A color to be used for the background of the value *(right)* part of badge. You can set
642
multiple using a pipe with a condition each, like this:
643
`color<value|color:null` The following operators are
489
- supported: >, <, >=, <=, =, :null (to check if no value exists). Each color can be specified in same manner as for `label_color` parameter.
644
+ supported: >, <, >=, <=, =, :null (to check if no value exists).
645
+ Each color can be specified in same manner as for `label_color` parameter.
646
Currently only integers are supported as values."
647
required: false
648
allowEmptyValue: true
@@ -495,7 +651,9 @@ paths:
651
format: any text
652
- name: text_color_lbl
653
in: query
498
- description: Font color for label *(left)* part of the badge. One of predefined colors or as HTML hexadecimal color without preceding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value given default color will be used.
654
+ description: "Font color for label *(left)* part of the badge. One of predefined colors or as HTML hexadecimal
655
+ color without preceding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value given default
656
+ color will be used."
657
required: false
658
allowEmptyValue: true
659
schema:
@@ -517,7 +675,9 @@ paths:
675
format: ^([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$
676
- name: text_color_val
677
in: query
520
- description: Font color for value *(right)* part of the badge. One of predefined colors or as HTML hexadecimal color without preceding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value given default color will be used.
678
+ description: "Font color for value *(right)* part of the badge. One of predefined colors or as HTML
679
+ hexadecimal color without preceding `#` character. Formats allowed `RGB` or `RRGGBB`. If no or wrong value
680
+ given default color will be used."
681
required: false
682
allowEmptyValue: true
683
schema:
@@ -565,7 +725,12 @@ paths:
725
format: integer
726
- name: fixed_width_lbl
727
in: query
568
- description: This parameter overrides auto-sizing of badge and creates it with fixed width. This parameter determines the size of the label's left side *(label/name)*. You must set this parameter together with `fixed_width_val` otherwise it will be ignored. You should set the label/value widths wide enough to provide space for all the possible values/contents of the badge you're requesting. In case the text cannot fit the space given it will be clipped. The `scale` parameter still applies on the values you give to `fixed_width_lbl` and `fixed_width_val`.
728
+ description: "This parameter overrides auto-sizing of badge and creates it with fixed width.
729
+ This parameter determines the size of the label's left side *(label/name)*.
730
+ You must set this parameter together with `fixed_width_val` otherwise it will be ignored.
731
+ You should set the label/value widths wide enough to provide space for all the possible values/contents of
732
+ the badge you're requesting. In case the text cannot fit the space given it will be clipped.
733
+ The `scale` parameter still applies on the values you give to `fixed_width_lbl` and `fixed_width_val`."
734
required: false
735
allowEmptyValue: false
736
schema:
@@ -573,7 +738,12 @@ paths:
738
format: integer
739
- name: fixed_width_val
740
in: query
576
- description: This parameter overrides auto-sizing of badge and creates it with fixed width. This parameter determines the size of the label's right side *(value)*. You must set this parameter together with `fixed_width_lbl` otherwise it will be ignored. You should set the label/value widths wide enough to provide space for all the possible values/contents of the badge you're requesting. In case the text cannot fit the space given it will be clipped. The `scale` parameter still applies on the values you give to `fixed_width_lbl` and `fixed_width_val`.
741
+ description: "This parameter overrides auto-sizing of badge and creates it with fixed width. This parameter
742
+ determines the size of the label's right side *(value)*. You must set this parameter together with
743
+ `fixed_width_lbl` otherwise it will be ignored. You should set the label/value widths wide enough to
744
+ provide space for all the possible values/contents of the badge you're requesting. In case the text cannot
745
+ fit the space given it will be clipped. The `scale` parameter still applies on the values you give to
746
+ `fixed_width_lbl` and `fixed_width_val`."
747
required: false
748
allowEmptyValue: false
749
schema:
@@ -754,12 +924,12 @@ paths:
924
/alarms_values:
925
get:
926
summary: Get a list of active or raised alarms on the server
757
- description: The alarms_values endpoint returns the list of all raised or enabled alarms on
927
+ description: "The alarms_values endpoint returns the list of all raised or enabled alarms on
928
the netdata server. Called without any parameters, the raised alarms in
759
- state WARNING or CRITICAL are returned. By passing "?all", all the
929
+ state WARNING or CRITICAL are returned. By passing '?all', all the
930
enabled alarms are returned.
931
This option output differs from `/alarms` in the number of variables delivered. This endpoint gives
762
- to user `id`, `value`, `last_updated` time, and alarm `status`.
932
+ to user `id`, `value`, `last_updated` time, and alarm `status`."
933
parameters:
934
- name: all
935
in: query
@@ -855,16 +1025,16 @@ paths:
1025
memory.
1026
/manage/health:
1027
get:
858
- summary: Accesses the health management API to control health checks and
859
- notifications at runtime.
860
- description: Available from Netdata v1.12 and above, protected via bearer
1028
+ summary: "Accesses the health management API to control health checks and
1029
+ notifications at runtime."
1030
+ description: "Available from Netdata v1.12 and above, protected via bearer
1031
authorization. Especially useful for maintenance periods, the API allows
1032
you to disable health checks completely, silence alarm notifications, or
1033
Disable/Silence specific alarms that match selectors on alarm/template
1034
name, chart, context, host and family. For the simple disable/silence
1035
all scenarios, only the cmd parameter is required. The other parameters
1036
are used to define alarm selectors. For more information and examples,
867
- refer to the netdata documentation.
1037
+ refer to the netdata documentation."
1038
parameters:
1039
- name: cmd
1040
in: query
@@ -918,8 +1088,8 @@ paths:
1088
/aclk:
1089
get:
1090
summary: Get information about current ACLK state
921
- description: aclk endpoint returns detailed information
922
- about current state of ACLK (Agent to Cloud communication).
1091
+ description: "ACLK endpoint returns detailed information
1092
+ about current state of ACLK (Agent to Cloud communication)."
1093
responses:
1094
"200":
1095
description: JSON object with ACLK information.
@@ -929,15 +1099,15 @@ paths:
1099
$ref: "#/components/schemas/aclk_state"
1100
/metric_correlations:
1101
get:
932
- summary: Analyze all the metrics to find their correlations
933
- description: Given two time-windows (baseline, highlight), it goes
1102
+ summary: "Analyze all the metrics to find their correlations"
1103
+ description: "Given two time-windows (baseline, highlight), it goes
1104
through all the available metrics, querying both windows and tries to find
1105
how these two windows relate to each other. It supports
1106
multiple algorithms to do so. The result is a list of all
1107
metrics evaluated, weighted for 0.0 (the two windows are
1108
more different) to 1.0 (the two windows are similar).
1109
The algorithm adjusts automatically the baseline window to be
940
- a power of two multiple of the highlighted (1, 2, 4, 8, etc).
1110
+ a power of two multiple of the highlighted (1, 2, 4, 8, etc)."
1111
parameters:
1112
- name: baseline_after
1113
in: query
@@ -1377,6 +1547,75 @@ components:
1547
type: number
1548
nullable: true
1549
description: Chart health red threshold.
1550
+ context_summary:
1551
+ type: object
1552
+ properties:
1553
+ hostname:
1554
+ type: string
1555
+ description: The hostname of the netdata server.
1556
+ machine_guid:
1557
+ type: string
1558
+ description: The unique installation id of this netdata server.
1559
+ node_id:
1560
+ type: string
1561
+ description: The unique node id of this netdata server at the hub.
1562
+ example: nightly
1563
+ claim_id:
1564
+ type: string
1565
+ description: The unique handshake id of this netdata server and the hub.
1566
+ host_labels:
1567
+ type: object
1568
+ description: The host labels associated with this netdata server.
1569
+ context:
1570
+ type: object
1571
+ description: "An object containing all the context objects available at the netdata server.
1572
+ This is used as an indexed array. The key of each context object is the id of the context."
1573
+ additionalProperties:
1574
+ $ref: "#/components/schemas/context"
1575
+ context:
1576
+ type: object
1577
+ properties:
1578
+ version:
1579
+ type: string
1580
+ description: "The version of this context.
1581
+ The number are not sequential, but bigger numbers depict a newer object."
1582
+ hub_version:
1583
+ type: string
1584
+ description: The version of this context, as known by hub.
1585
+ family:
1586
+ type: string
1587
+ description: "The family of the context. When multiple charts of a context have different families,
1588
+ the netdata server replaces the different parts with [x], so that the context can have only one family."
1589
+ title:
1590
+ type: string
1591
+ description: "The title of the context. When multiple charts of a context have different titles,
1592
+ the netdata server replaces the different parts with [x], so that the context can have only one title."
1593
+ priority:
1594
+ type: number
1595
+ description: "The relative priority of the context. When multiple contexts have different priorities,
1596
+ the minimum among them is selected as the priority of the context."
1597
+ units:
1598
+ type: string
1599
+ description: "The unit of measurement for the values of all dimensions of the context. If multiple charts
1600
+ of context have different units, the latest collected is selected."
1601
+ chart_type:
1602
+ type: string
1603
+ description: The chart type.
1604
+ enum:
1605
+ - line
1606
+ - area
1607
+ - stacked
1608
+ first_time_t:
1609
+ type: number
1610
+ description: The UNIX timestamp of the first entry (the oldest) in the database.
1611
+ last_time_t:
1612
+ type: number
1613
+ description: The UNIX timestamp of the latest entry in the database.
1614
+ charts:
1615
+ type: object
1616
+ description: "An object containing all the charts available for the chart. This is used as an indexed array.
1617
+ For each pair in the dictionary, the key is the id of the chart and the value provides all details about
1618
+ the chart."
1619
alarm_variables:
1620
type: object
1621
properties:
@@ -1848,8 +2087,9 @@ components:
2087
properties:
2088
aclk-available:
2089
type: string
1851
- description: Describes whether this agent is capable of connection to the Cloud.
1852
- False means agent has been built without ACLK component either on purpose (user choice) or due to missing dependency.
2090
+ description: "Describes whether this agent is capable of connection to the Cloud.
2091
+ False means agent has been built without ACLK component either on purpose (user choice)
2092
+ or due to missing dependency."
2093
aclk-version:
2094
type: integer
2095
description: Describes which ACLK version is currently used.
web/api/queries/query.c
+2
-2
@@ -1173,8 +1173,8 @@ void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, int tier, time_t now) {
1173
all_points_read += points;
1174
tmp->query_ops.finalize(&handle);
1175
1176
- internal_error(true, "DBENGINE: backfilled chart '%s', dimension '%s', tier %d, from %ld to %ld, with %zu points from tier %d",
1177
- rd->rrdset->name, rd->name, tier, after_wanted, before_wanted, points, tr);
1176
+ //internal_error(true, "DBENGINE: backfilled chart '%s', dimension '%s', tier %d, from %ld to %ld, with %zu points from tier %d",
1177
+ // rd->rrdset->name, rd->name, tier, after_wanted, before_wanted, points, tr);
1178
}
1179
1180
rrdr_query_completed(all_points_read, all_points_read);
web/api/web_api_v1.c
+152
-5
@@ -374,6 +374,155 @@ inline int web_client_api_request_v1_alarm_variables(RRDHOST *host, struct web_c
374
return web_client_api_request_single_chart(host, w, url, health_api_v1_chart_variables2json);
375
}
376
377
+static RRDCONTEXT_TO_JSON_OPTIONS rrdcontext_to_json_parse_options(char *o) {
378
+ RRDCONTEXT_TO_JSON_OPTIONS options = RRDCONTEXT_OPTION_NONE;
379
+ char *tok;
380
+
381
+ while(o && *o && (tok = mystrsep(&o, ", |"))) {
382
+ if(!*tok) continue;
383
+
384
+ if(!strcmp(tok, "full") || !strcmp(tok, "all"))
385
+ options |= RRDCONTEXT_OPTIONS_ALL;
386
+ else if(!strcmp(tok, "charts") || !strcmp(tok, "instances"))
387
+ options |= RRDCONTEXT_OPTION_SHOW_INSTANCES;
388
+ else if(!strcmp(tok, "dimensions") || !strcmp(tok, "metrics"))
389
+ options |= RRDCONTEXT_OPTION_SHOW_METRICS;
390
+ else if(!strcmp(tok, "queue"))
391
+ options |= RRDCONTEXT_OPTION_SHOW_QUEUED;
392
+ else if(!strcmp(tok, "flags"))
393
+ options |= RRDCONTEXT_OPTION_SHOW_FLAGS;
394
+ else if(!strcmp(tok, "uuids"))
395
+ options |= RRDCONTEXT_OPTION_SHOW_UUIDS;
396
+ else if(!strcmp(tok, "deleted"))
397
+ options |= RRDCONTEXT_OPTION_SHOW_DELETED;
398
+ else if(!strcmp(tok, "labels"))
399
+ options |= RRDCONTEXT_OPTION_SHOW_LABELS;
400
+ else if(!strcmp(tok, "deepscan"))
401
+ options |= RRDCONTEXT_OPTION_DEEPSCAN;
402
+ }
403
+
404
+ return options;
405
+}
406
+
407
+static int web_client_api_request_v1_context(RRDHOST *host, struct web_client *w, char *url) {
408
+ char *context = NULL;
409
+ RRDCONTEXT_TO_JSON_OPTIONS options = RRDCONTEXT_OPTION_NONE;
410
+ time_t after = 0, before = 0;
411
+ const char *chart_label_key = NULL, *chart_labels_filter = NULL;
412
+ BUFFER *dimensions = NULL;
413
+
414
+ buffer_flush(w->response.data);
415
+
416
+ while(url) {
417
+ char *value = mystrsep(&url, "&");
418
+ if(!value || !*value) continue;
419
+
420
+ char *name = mystrsep(&value, "=");
421
+ if(!name || !*name) continue;
422
+ if(!value || !*value) continue;
423
+
424
+ // name and value are now the parameters
425
+ // they are not null and not empty
426
+
427
+ if(!strcmp(name, "context") || !strcmp(name, "ctx")) context = value;
428
+ else if(!strcmp(name, "after")) after = str2l(value);
429
+ else if(!strcmp(name, "before")) before = str2l(value);
430
+ else if(!strcmp(name, "options")) options = rrdcontext_to_json_parse_options(value);
431
+ else if(!strcmp(name, "chart_label_key")) chart_label_key = value;
432
+ else if(!strcmp(name, "chart_labels_filter")) chart_labels_filter = value;
433
+ else if(!strcmp(name, "dimension") || !strcmp(name, "dim") || !strcmp(name, "dimensions") || !strcmp(name, "dims")) {
434
+ if(!dimensions) dimensions = buffer_create(100);
435
+ buffer_strcat(dimensions, "|");
436
+ buffer_strcat(dimensions, value);
437
+ }
438
+ }
439
+
440
+ if(!context || !*context) {
441
+ buffer_sprintf(w->response.data, "No context is given at the request.");
442
+ return HTTP_RESP_BAD_REQUEST;
443
+ }
444
+
445
+ SIMPLE_PATTERN *chart_label_key_pattern = NULL;
446
+ SIMPLE_PATTERN *chart_labels_filter_pattern = NULL;
447
+ SIMPLE_PATTERN *chart_dimensions_pattern = NULL;
448
+
449
+ if(chart_label_key)
450
+ chart_label_key_pattern = simple_pattern_create(chart_label_key, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
451
+
452
+ if(chart_labels_filter)
453
+ chart_labels_filter_pattern = simple_pattern_create(chart_labels_filter, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
454
+
455
+ if(dimensions) {
456
+ chart_dimensions_pattern = simple_pattern_create(buffer_tostring(dimensions), ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
457
+ buffer_free(dimensions);
458
+ }
459
+
460
+ w->response.data->contenttype = CT_APPLICATION_JSON;
461
+ int ret = rrdcontext_to_json(host, w->response.data, after, before, options, context, chart_label_key_pattern, chart_labels_filter_pattern, chart_dimensions_pattern);
462
+
463
+ simple_pattern_free(chart_label_key_pattern);
464
+ simple_pattern_free(chart_labels_filter_pattern);
465
+ simple_pattern_free(chart_dimensions_pattern);
466
+
467
+ return ret;
468
+}
469
+
470
+static int web_client_api_request_v1_contexts(RRDHOST *host, struct web_client *w, char *url) {
471
+ RRDCONTEXT_TO_JSON_OPTIONS options = RRDCONTEXT_OPTION_NONE;
472
+ time_t after = 0, before = 0;
473
+ const char *chart_label_key = NULL, *chart_labels_filter = NULL;
474
+ BUFFER *dimensions = NULL;
475
+
476
+ buffer_flush(w->response.data);
477
+
478
+ while(url) {
479
+ char *value = mystrsep(&url, "&");
480
+ if(!value || !*value) continue;
481
+
482
+ char *name = mystrsep(&value, "=");
483
+ if(!name || !*name) continue;
484
+ if(!value || !*value) continue;
485
+
486
+ // name and value are now the parameters
487
+ // they are not null and not empty
488
+
489
+ if(!strcmp(name, "after")) after = str2l(value);
490
+ else if(!strcmp(name, "before")) before = str2l(value);
491
+ else if(!strcmp(name, "options")) options = rrdcontext_to_json_parse_options(value);
492
+ else if(!strcmp(name, "chart_label_key")) chart_label_key = value;
493
+ else if(!strcmp(name, "chart_labels_filter")) chart_labels_filter = value;
494
+ else if(!strcmp(name, "dimension") || !strcmp(name, "dim") || !strcmp(name, "dimensions") || !strcmp(name, "dims")) {
495
+ if(!dimensions) dimensions = buffer_create(100);
496
+ buffer_strcat(dimensions, "|");
497
+ buffer_strcat(dimensions, value);
498
+ }
499
+ }
500
+
501
+ SIMPLE_PATTERN *chart_label_key_pattern = NULL;
502
+ SIMPLE_PATTERN *chart_labels_filter_pattern = NULL;
503
+ SIMPLE_PATTERN *chart_dimensions_pattern = NULL;
504
+
505
+ if(chart_label_key)
506
+ chart_label_key_pattern = simple_pattern_create(chart_label_key, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
507
+
508
+ if(chart_labels_filter)
509
+ chart_labels_filter_pattern = simple_pattern_create(chart_labels_filter, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
510
+
511
+ if(dimensions) {
512
+ chart_dimensions_pattern = simple_pattern_create(buffer_tostring(dimensions), ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT);
513
+ buffer_free(dimensions);
514
+ }
515
+
516
+ w->response.data->contenttype = CT_APPLICATION_JSON;
517
+ int ret = rrdcontexts_to_json(host, w->response.data, after, before, options, chart_label_key_pattern, chart_labels_filter_pattern, chart_dimensions_pattern);
518
+
519
+ simple_pattern_free(chart_label_key_pattern);
520
+ simple_pattern_free(chart_labels_filter_pattern);
521
+ simple_pattern_free(chart_dimensions_pattern);
522
+
523
+ return ret;
524
+}
525
+
526
inline int web_client_api_request_v1_charts(RRDHOST *host, struct web_client *w, char *url) {
527
(void)url;
528
@@ -932,8 +1081,6 @@ static inline void web_client_api_request_v1_info_mirrored_hosts(BUFFER *wb) {
1081
buffer_strcat(wb, "\t\"mirrored_hosts\": [\n");
1082
rrd_rdlock();
1083
rrdhost_foreach_read(host) {
935
- if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED))
936
- continue;
1084
if (count > 0)
1085
buffer_strcat(wb, ",\n");
1086
@@ -945,8 +1092,6 @@ static inline void web_client_api_request_v1_info_mirrored_hosts(BUFFER *wb) {
1092
count = 0;
1093
rrdhost_foreach_read(host)
1094
{
948
- if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED))
949
- continue;
1095
if (count > 0)
1096
buffer_strcat(wb, ",\n");
1097
@@ -1077,7 +1222,7 @@ inline int web_client_api_request_v1_info_fill_buffer(RRDHOST *host, BUFFER *wb)
1222
buffer_strcat(wb, "\t\"aclk-ng-available\": false,\n");
1223
buffer_strcat(wb, "\t\"aclk-legacy-available\": false,\n");
1224
#endif
1080
- char *agent_id = is_agent_claimed();
1225
+ char *agent_id = get_agent_claimid();
1226
if (agent_id == NULL)
1227
buffer_strcat(wb, "\t\"agent-claimed\": false,\n");
1228
else {
@@ -1398,6 +1543,8 @@ static struct api_command {
1543
{ "data", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_data },
1544
{ "chart", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_chart },
1545
{ "charts", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_charts },
1546
+ { "context", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_context },
1547
+ { "contexts", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_contexts },
1548
{ "archivedcharts", 0, WEB_CLIENT_ACL_DASHBOARD, web_client_api_request_v1_archivedcharts },
1549
1550
// registry checks the ACL by itself, so we allow everything
web/gui/dashboard_info.js
+334
-152
@@ -1284,6 +1284,41 @@ const ebpfUDPrecv = 'Number of calls to <a href="https://learn.netdata.cloud/doc
1284
'Netdata gives a summary for this chart in <a href="#ebpf_global_udp_bandwidth_call">Network Stack</a>. ' +
1285
'When the integration is <a href="https://learn.netdata.cloud/guides/troubleshoot/monitor-debug-applications-ebpf" target="_blank">enabled</a>, Netdata shows UDP calls per <a href="#ebpf_apps_udp_recv">application</a>.' + ebpfChartProvides
1286
1287
+const cgroupCPULimit = 'Total CPU utilization within the configured or system-wide (if not set) limits. When the CPU utilization of a cgroup exceeds the limit for the configured period, the tasks belonging to its hierarchy will be throttled and are not allowed to run again until the next period.'
1288
+const cgroupCPU = 'Total CPU utilization within the system-wide CPU resources (all cores). The amount of time spent by tasks of the cgroup in <a href="https://en.wikipedia.org/wiki/CPU_modes#Mode_types" target="_blank">user and kernel</a> modes.'
1289
+const cgroupThrottled = 'The percentage of runnable periods when tasks in a cgroup have been throttled. The tasks have not been allowed to run because they have exhausted all of the available time as specified by their CPU quota.'
1290
+const cgroupThrottledDuration = 'The total time duration for which tasks in a cgroup have been throttled. When an application has used its allotted CPU quota for a given period, it gets throttled until the next period.'
1291
+const cgroupCPUShared = '<p>The weight of each group living in the same hierarchy, that translates into the amount of CPU it is expected to get. The percentage of CPU assigned to the cgroup is the value of shares divided by the sum of all shares in all cgroups in the same level.</p> <p>For example, tasks in two cgroups that have <b>cpu.shares</b> set to 100 will receive equal CPU time, but tasks in a cgroup that has <b>cpu.shares</b> set to 200 receive twice the CPU time of tasks in a cgroup where <b>cpu.shares</b> is set to 100.</p>'
1292
+const cgroupCPUPerCore = 'Total CPU utilization per core within the system-wide CPU resources.'
1293
+const cgroupCPUSomePressure = 'CPU <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on CPU. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
1294
+const cgroupCPUSomePressureStallTime = 'The amount of time some processes have been waiting for CPU time.'
1295
+const cgroupCPUFullPressure = 'CPU <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Full</b> indicates the share of time in which <b>all non-idle tasks</b> are stalled on CPU resource simultaneously. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
1296
+const cgroupCPUFullPressureStallTime = 'The amount of time all non-idle processes have been stalled due to CPU congestion.'
1297
+
1298
+const cgroupMemUtilization = 'RAM utilization within the configured or system-wide (if not set) limits. When the RAM utilization of a cgroup exceeds the limit, OOM killer will start killing the tasks belonging to the cgroup.'
1299
+const cgroupMemUsageLimit = 'RAM usage within the configured or system-wide (if not set) limits. When the RAM usage of a cgroup exceeds the limit, OOM killer will start killing the tasks belonging to the cgroup.'
1300
+const cgroupMemUsage = 'The amount of used RAM and swap memory.'
1301
+const cgroupMem = 'Memory usage statistics. The individual metrics are described in the memory.stat section for <a href="https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/memory.html#per-memory-cgroup-local-status" target="_blank">cgroup-v1</a> and <a href="https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#memory-interface-files" target="_blank">cgroup-v2</a>.'
1302
+const cgroupMemFailCnt = 'The number of memory usage hits limits.'
1303
+const cgroupWriteback = '<b>Dirty</b> is the amount of memory waiting to be written to disk. <b>Writeback</b> is how much memory is actively being written to disk.'
1304
+const cgroupMemActivity = '<p>Memory accounting statistics.</p><p><b>In</b> - a page is accounted as either mapped anon page (RSS) or cache page (Page Cache) to the cgroup. <b>Out</b> - a page is unaccounted from the cgroup.</p>'
1305
+const cgroupPgFaults = '<p>Memory <a href="https://en.wikipedia.org/wiki/Page_fault" target="_blank">page fault</a> statistics.</p><p><b>Pgfault</b> - all page faults. <b>Swap</b> - major page faults.</p>'
1306
+const cgroupMemorySomePressure = 'Memory <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on memory. In this state the CPU is still doing productive work. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
1307
+const cgroupMemorySomePressureStallTime = 'The amount of time some processes have been waiting due to memory congestion.'
1308
+const cgroupMemoryFullPressure = 'Memory <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Full</b> indicates the share of time in which <b>all non-idle tasks</b> are stalled on memory resource simultaneously. In this state actual CPU cycles are going to waste, and a workload that spends extended time in this state is considered to be thrashing. This has severe impact on performance. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
1309
+const cgroupMemoryFullPressureStallTime = 'The amount of time all non-idle processes have been stalled due to memory congestion.'
1310
+
1311
+const cgroupIO = 'The amount of data transferred to and from specific devices as seen by the CFQ scheduler. It is not updated when the CFQ scheduler is operating on a request queue.'
1312
+const cgroupServicedOps = 'The number of I/O operations performed on specific devices as seen by the CFQ scheduler.'
1313
+const cgroupQueuedOps = 'The number of requests queued for I/O operations.'
1314
+const cgroupMergedOps = 'The number of BIOS requests merged into requests for I/O operations.'
1315
+const cgroupThrottleIO = 'The amount of data transferred to and from specific devices as seen by the throttling policy.'
1316
+const cgroupThrottleIOServicesOps = 'The number of I/O operations performed on specific devices as seen by the throttling policy.'
1317
+const cgroupIOSomePressure = 'I/O <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on I/O. In this state the CPU is still doing productive work. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
1318
+const cgroupIOSomePRessureStallTime = 'The amount of time some processes have been waiting due to I/O congestion.'
1319
+const cgroupIOFullPressure = 'I/O <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. <b>Full</b> line indicates the share of time in which <b>all non-idle tasks</b> are stalled on I/O resource simultaneously. In this state actual CPU cycles are going to waste, and a workload that spends extended time in this state is considered to be thrashing. This has severe impact on performance. The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
1320
+const cgroupIOFullPressureStallTime = 'The amount of time all non-idle processes have been stalled due to I/O congestion.'
1321
+
1322
netdataDashboard.context = {
1323
'system.cpu': {
1324
info: function (os) {
@@ -3140,6 +3175,64 @@ netdataDashboard.context = {
3175
info: netMTUInfo
3176
},
3177
3178
+ 'k8s.cgroup.net_net': {
3179
+ mainheads: [
3180
+ function (_, id) {
3181
+ var iface;
3182
+ try {
3183
+ iface = ' ' + id.substring(id.lastIndexOf('.net_') + 5, id.length);
3184
+ } catch (e) {
3185
+ iface = '';
3186
+ }
3187
+ return netdataDashboard.gaugeChart('Received' + iface, '12%', 'received');
3188
+
3189
+ },
3190
+ function (_, id) {
3191
+ var iface;
3192
+ try {
3193
+ iface = ' ' + id.substring(id.lastIndexOf('.net_') + 5, id.length);
3194
+ } catch (e) {
3195
+ iface = '';
3196
+ }
3197
+ return netdataDashboard.gaugeChart('Sent' + iface, '12%', 'sent');
3198
+ }
3199
+ ],
3200
+ info: netBytesInfo
3201
+ },
3202
+ 'k8s.cgroup.net_packets': {
3203
+ info: netPacketsInfo
3204
+ },
3205
+ 'k8s.cgroup.net_errors': {
3206
+ info: netErrorsInfo
3207
+ },
3208
+ 'k8s.cgroup.net_fifo': {
3209
+ info: netFIFOInfo
3210
+ },
3211
+ 'k8s.cgroup.net_drops': {
3212
+ info: netDropsInfo
3213
+ },
3214
+ 'k8s.cgroup.net_compressed': {
3215
+ info: netCompressedInfo
3216
+ },
3217
+ 'k8s.cgroup.net_events': {
3218
+ info: netEventsInfo
3219
+ },
3220
+ 'k8s.cgroup.net_operstate': {
3221
+ info: netOperstateInfo
3222
+ },
3223
+ 'k8s.cgroup.net_duplex': {
3224
+ info: netDuplexInfo
3225
+ },
3226
+ 'k8s.cgroup.net_carrier': {
3227
+ info: netCarrierInfo
3228
+ },
3229
+ 'k8s.cgroup.net_speed': {
3230
+ info: netSpeedInfo
3231
+ },
3232
+ 'k8s.cgroup.net_mtu': {
3233
+ info: netMTUInfo
3234
+ },
3235
+
3236
// ------------------------------------------------------------------------
3237
// WIRELESS NETWORK INTERFACES
3238
@@ -4008,8 +4101,7 @@ netdataDashboard.context = {
4101
'cgroup.cpu_limit': {
4102
valueRange: "[0, null]",
4103
mainheads: [
4011
- function (os, id) {
4012
- void (os);
4104
+ function (_, id) {
4105
cgroupCPULimitIsSet = 1;
4106
return '<div data-netdata="' + id + '"'
4107
+ ' data-dimensions="used"'
@@ -4026,15 +4118,11 @@ netdataDashboard.context = {
4118
+ ' role="application"></div>';
4119
}
4120
],
4029
- info: 'Total CPU utilization within the configured or system-wide (if not set) limits. '+
4030
- 'When the CPU utilization of a cgroup exceeds the limit for the configured period, '+
4031
- 'the tasks belonging to its hierarchy will be throttled and are not allowed to run again until the next period.'
4121
+ info: cgroupCPULimit
4122
},
4033
-
4123
'cgroup.cpu': {
4124
mainheads: [
4036
- function (os, id) {
4037
- void (os);
4125
+ function (_, id) {
4126
if (cgroupCPULimitIsSet === 0) {
4127
return '<div data-netdata="' + id + '"'
4128
+ ' data-chart-library="gauge"'
@@ -4051,60 +4139,107 @@ netdataDashboard.context = {
4139
return '';
4140
}
4141
],
4054
- info: 'Total CPU utilization within the system-wide CPU resources (all cores). '+
4055
- 'The amount of time spent by tasks of the cgroup in '+
4056
- '<a href="https://en.wikipedia.org/wiki/CPU_modes#Mode_types" target="_blank">user and kernel</a> modes.'
4142
+ info: cgroupCPU
4143
},
4058
-
4144
'cgroup.throttled': {
4060
- info: 'The percentage of runnable periods when tasks in a cgroup have been throttled. '+
4061
- 'The tasks have not been allowed to run because they have exhausted all of the available time as specified by their CPU quota.'
4145
+ info: cgroupThrottled
4146
},
4063
-
4147
'cgroup.throttled_duration': {
4065
- info: 'The total time duration for which tasks in a cgroup have been throttled. '+
4066
- 'When an application has used its allotted CPU quota for a given period, it gets throttled until the next period.'
4148
+ info: cgroupThrottledDuration
4149
},
4068
-
4150
'cgroup.cpu_shares': {
4070
- info: '<p>The weight of each group living in the same hierarchy, that translates into the amount of CPU it is expected to get. '+
4071
- 'The percentage of CPU assigned to the cgroup is the value of shares divided by the sum of all shares in all cgroups in the same level.</p>'+
4072
- '<p>For example, tasks in two cgroups that have <b>cpu.shares</b> set to 100 will receive equal CPU time, '+
4073
- 'but tasks in a cgroup that has <b>cpu.shares</b> set to 200 receive twice the CPU time of tasks in a cgroup where <b>cpu.shares</b> is set to 100.</p>'
4151
+ info: cgroupCPUShared
4152
},
4075
-
4153
'cgroup.cpu_per_core': {
4077
- info: 'Total CPU utilization per core within the system-wide CPU resources.'
4154
+ info: cgroupCPUPerCore
4155
},
4079
-
4156
'cgroup.cpu_some_pressure': {
4081
- info: 'CPU <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. '+
4082
- '<b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on CPU. ' +
4083
- 'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
4157
+ info: cgroupCPUSomePressure
4158
},
4159
'cgroup.cpu_some_pressure_stall_time': {
4086
- info: 'The amount of time some processes have been waiting for CPU time.'
4160
+ info: cgroupCPUSomePressureStallTime
4161
},
4088
-
4162
'cgroup.cpu_full_pressure': {
4090
- info: 'CPU <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. ' +
4091
- '<b>Full</b> indicates the share of time in which <b>all non-idle tasks</b> are stalled on CPU resource simultaneously. ' +
4092
- 'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
4163
+ info: cgroupCPUFullPressure
4164
},
4165
'cgroup.cpu_full_pressure_stall_time': {
4095
- info: 'The amount of time all non-idle processes have been stalled due to CPU congestion.'
4166
+ info: cgroupCPUFullPressureStallTime
4167
},
4168
4098
- 'cgroup.mem_utilization': {
4099
- info: 'RAM utilization within the configured or system-wide (if not set) limits. '+
4100
- 'When the RAM utilization of a cgroup exceeds the limit, '+
4101
- 'OOM killer will start killing the tasks belonging to the cgroup.'
4169
+ 'k8s.cgroup.cpu_limit': {
4170
+ valueRange: "[0, null]",
4171
+ mainheads: [
4172
+ function (_, id) {
4173
+ cgroupCPULimitIsSet = 1;
4174
+ return '<div data-netdata="' + id + '"'
4175
+ + ' data-dimensions="used"'
4176
+ + ' data-gauge-max-value="100"'
4177
+ + ' data-chart-library="gauge"'
4178
+ + ' data-title="CPU"'
4179
+ + ' data-units="%"'
4180
+ + ' data-gauge-adjust="width"'
4181
+ + ' data-width="12%"'
4182
+ + ' data-before="0"'
4183
+ + ' data-after="-CHART_DURATION"'
4184
+ + ' data-points="CHART_DURATION"'
4185
+ + ' data-colors="' + NETDATA.colors[4] + '"'
4186
+ + ' role="application"></div>';
4187
+ }
4188
+ ],
4189
+ info: cgroupCPULimit
4190
+ },
4191
+ 'k8s.cgroup.cpu': {
4192
+ mainheads: [
4193
+ function (_, id) {
4194
+ if (cgroupCPULimitIsSet === 0) {
4195
+ return '<div data-netdata="' + id + '"'
4196
+ + ' data-chart-library="gauge"'
4197
+ + ' data-title="CPU"'
4198
+ + ' data-units="%"'
4199
+ + ' data-gauge-adjust="width"'
4200
+ + ' data-width="12%"'
4201
+ + ' data-before="0"'
4202
+ + ' data-after="-CHART_DURATION"'
4203
+ + ' data-points="CHART_DURATION"'
4204
+ + ' data-colors="' + NETDATA.colors[4] + '"'
4205
+ + ' role="application"></div>';
4206
+ } else
4207
+ return '';
4208
+ }
4209
+ ],
4210
+ info: cgroupCPU
4211
+ },
4212
+ 'k8s.cgroup.throttled': {
4213
+ info: cgroupThrottled
4214
+ },
4215
+ 'k8s.cgroup.throttled_duration': {
4216
+ info: cgroupThrottledDuration
4217
+ },
4218
+ 'k8s.cgroup.cpu_shares': {
4219
+ info: cgroupCPUShared
4220
+ },
4221
+ 'k8s.cgroup.cpu_per_core': {
4222
+ info: cgroupCPUPerCore
4223
+ },
4224
+ 'k8s.cgroup.cpu_some_pressure': {
4225
+ info: cgroupCPUSomePressure
4226
+ },
4227
+ 'k8s.cgroup.cpu_some_pressure_stall_time': {
4228
+ info: cgroupCPUSomePressureStallTime
4229
+ },
4230
+ 'k8s.cgroup.cpu_full_pressure': {
4231
+ info: cgroupCPUFullPressure
4232
+ },
4233
+ 'k8s.cgroup.cpu_full_pressure_stall_time': {
4234
+ info: cgroupCPUFullPressureStallTime
4235
},
4236
4237
+ 'cgroup.mem_utilization': {
4238
+ info: cgroupMemUtilization
4239
+ },
4240
'cgroup.mem_usage_limit': {
4241
mainheads: [
4106
- function (os, id) {
4107
- void (os);
4242
+ function (_, id) {
4243
cgroupMemLimitIsSet = 1;
4244
return '<div data-netdata="' + id + '"'
4245
+ ' data-dimensions="used"'
@@ -4122,15 +4257,11 @@ netdataDashboard.context = {
4257
+ ' role="application"></div>';
4258
}
4259
],
4125
- info: 'RAM usage within the configured or system-wide (if not set) limits. '+
4126
- 'When the RAM usage of a cgroup exceeds the limit, '+
4127
- 'OOM killer will start killing the tasks belonging to the cgroup.'
4260
+ info: cgroupMemUsageLimit
4261
},
4129
-
4262
'cgroup.mem_usage': {
4263
mainheads: [
4132
- function (os, id) {
4133
- void (os);
4264
+ function (_, id) {
4265
if (cgroupMemLimitIsSet === 0) {
4266
return '<div data-netdata="' + id + '"'
4267
+ ' data-chart-library="gauge"'
@@ -4147,79 +4278,125 @@ netdataDashboard.context = {
4278
return '';
4279
}
4280
],
4150
- info: 'The amount of used RAM and swap memory.'
4281
+ info: cgroupMemUsage
4282
},
4152
-
4283
'cgroup.mem': {
4154
- info: 'Memory usage statistics. '+
4155
- 'The individual metrics are described in the memory.stat section for '+
4156
- '<a href="https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/memory.html#per-memory-cgroup-local-status" target="_blank">cgroup-v1 </a>'+
4157
- 'and '+
4158
- '<a href="https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#memory-interface-files" target="_blank">cgroup-v2</a>.'
4284
+ info: cgroupMem
4285
},
4160
-
4286
'cgroup.mem_failcnt': {
4162
- info: 'The number of memory usage hits limits.'
4287
+ info: cgroupMemFailCnt
4288
},
4164
-
4289
'cgroup.writeback': {
4166
- info: '<b>Dirty</b> is the amount of memory waiting to be written to disk. <b>Writeback</b> is how much memory is actively being written to disk.'
4290
+ info: cgroupWriteback
4291
},
4168
-
4292
'cgroup.mem_activity': {
4170
- info: '<p>Memory accounting statistics.</p>'+
4171
- '<p><b>In</b> - a page is accounted as either mapped anon page (RSS) or cache page (Page Cache) to the cgroup. '+
4172
- '<b>Out</b> - a page is unaccounted from the cgroup.</p>'
4293
+ info: cgroupMemActivity
4294
},
4174
-
4295
'cgroup.pgfaults': {
4176
- info: '<p>Memory <a href="https://en.wikipedia.org/wiki/Page_fault" target="_blank">page fault</a> statistics.</p>'+
4177
- '<p><b>Pgfault</b> - all page faults. '+
4178
- '<b>Swap</b> - major page faults.</p>'
4296
+ info: cgroupPgFaults
4297
},
4180
-
4298
'cgroup.memory_some_pressure': {
4182
- info: 'Memory <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. '+
4183
- '<b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on memory. ' +
4184
- 'In this state the CPU is still doing productive work. '+
4185
- 'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
4299
+ info: cgroupMemorySomePressure
4300
},
4301
'cgroup.memory_some_pressure_stall_time': {
4188
- info: 'The amount of time some processes have been waiting due to memory congestion.'
4302
+ info: cgroupMemorySomePressureStallTime
4303
},
4190
-
4304
'cgroup.memory_full_pressure': {
4192
- info: 'Memory <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. ' +
4193
- '<b>Full</b> indicates the share of time in which <b>all non-idle tasks</b> are stalled on memory resource simultaneously. ' +
4194
- 'In this state actual CPU cycles are going to waste, and a workload that spends extended time in this state is considered to be thrashing. '+
4195
- 'This has severe impact on performance. '+
4196
- 'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
4305
+ info: cgroupMemoryFullPressure
4306
},
4307
'cgroup.memory_full_pressure_stall_time': {
4199
- info: 'The amount of time all non-idle processes have been stalled due to memory congestion.'
4308
+ info: cgroupMemoryFullPressureStallTime
4309
},
4310
4202
- 'cgroup.io': {
4203
- info: 'The amount of data transferred to and from specific devices as seen by the CFQ scheduler. '+
4204
- 'It is not updated when the CFQ scheduler is operating on a request queue.'
4311
+ 'k8s.cgroup.mem_utilization': {
4312
+ info: cgroupMemUtilization
4313
+ },
4314
+ 'k8s.cgroup.mem_usage_limit': {
4315
+ mainheads: [
4316
+ function (_, id) {
4317
+ cgroupMemLimitIsSet = 1;
4318
+ return '<div data-netdata="' + id + '"'
4319
+ + ' data-dimensions="used"'
4320
+ + ' data-append-options="percentage"'
4321
+ + ' data-gauge-max-value="100"'
4322
+ + ' data-chart-library="gauge"'
4323
+ + ' data-title="Memory"'
4324
+ + ' data-units="%"'
4325
+ + ' data-gauge-adjust="width"'
4326
+ + ' data-width="12%"'
4327
+ + ' data-before="0"'
4328
+ + ' data-after="-CHART_DURATION"'
4329
+ + ' data-points="CHART_DURATION"'
4330
+ + ' data-colors="' + NETDATA.colors[1] + '"'
4331
+ + ' role="application"></div>';
4332
+ }
4333
+ ],
4334
+ info: cgroupMemUsageLimit
4335
+ },
4336
+ 'k8s.cgroup.mem_usage': {
4337
+ mainheads: [
4338
+ function (_, id) {
4339
+ if (cgroupMemLimitIsSet === 0) {
4340
+ return '<div data-netdata="' + id + '"'
4341
+ + ' data-chart-library="gauge"'
4342
+ + ' data-title="Memory"'
4343
+ + ' data-units="MB"'
4344
+ + ' data-gauge-adjust="width"'
4345
+ + ' data-width="12%"'
4346
+ + ' data-before="0"'
4347
+ + ' data-after="-CHART_DURATION"'
4348
+ + ' data-points="CHART_DURATION"'
4349
+ + ' data-colors="' + NETDATA.colors[1] + '"'
4350
+ + ' role="application"></div>';
4351
+ } else
4352
+ return '';
4353
+ }
4354
+ ],
4355
+ info: cgroupMemUsage
4356
+ },
4357
+ 'k8s.cgroup.mem': {
4358
+ info: cgroupMem
4359
+ },
4360
+ 'k8s.cgroup.mem_failcnt': {
4361
+ info: cgroupMemFailCnt
4362
+ },
4363
+ 'k8s.cgroup.writeback': {
4364
+ info: cgroupWriteback
4365
+ },
4366
+ 'k8s.cgroup.mem_activity': {
4367
+ info: cgroupMemActivity
4368
+ },
4369
+ 'k8s.cgroup.pgfaults': {
4370
+ info: cgroupPgFaults
4371
+ },
4372
+ 'k8s.cgroup.memory_some_pressure': {
4373
+ info: cgroupMemorySomePressure
4374
+ },
4375
+ 'k8s.cgroup.memory_some_pressure_stall_time': {
4376
+ info: cgroupMemorySomePressureStallTime
4377
+ },
4378
+ 'k8s.cgroup.memory_full_pressure': {
4379
+ info: cgroupMemoryFullPressure
4380
+ },
4381
+ 'k8s.cgroup.memory_full_pressure_stall_time': {
4382
+ info: cgroupMemoryFullPressureStallTime
4383
},
4384
4385
+ 'cgroup.io': {
4386
+ info: cgroupIO
4387
+ },
4388
'cgroup.serviced_ops': {
4208
- info: 'The number of I/O operations performed on specific devices as seen by the CFQ scheduler.'
4389
+ info: cgroupServicedOps
4390
},
4210
-
4391
'cgroup.queued_ops': {
4212
- info: 'The number of requests queued for I/O operations.'
4392
+ info: cgroupQueuedOps
4393
},
4214
-
4394
'cgroup.merged_ops': {
4216
- info: 'The number of BIOS requests merged into requests for I/O operations.'
4395
+ info: cgroupMergedOps
4396
},
4218
-
4397
'cgroup.throttle_io': {
4398
mainheads: [
4221
- function (os, id) {
4222
- void (os);
4399
+ function (_, id) {
4400
return '<div data-netdata="' + id + '"'
4401
+ ' data-dimensions="read"'
4402
+ ' data-chart-library="gauge"'
@@ -4233,8 +4410,7 @@ netdataDashboard.context = {
4410
+ ' data-colors="' + NETDATA.colors[2] + '"'
4411
+ ' role="application"></div>';
4412
},
4236
- function (os, id) {
4237
- void (os);
4413
+ function (_, id) {
4414
return '<div data-netdata="' + id + '"'
4415
+ ' data-dimensions="write"'
4416
+ ' data-chart-library="gauge"'
@@ -4249,214 +4425,220 @@ netdataDashboard.context = {
4425
+ ' role="application"></div>';
4426
}
4427
],
4252
- info: 'The amount of data transferred to and from specific devices as seen by the throttling policy.'
4428
+ info: cgroupThrottleIO
4429
},
4254
-
4430
'cgroup.throttle_serviced_ops': {
4256
- info: 'The number of I/O operations performed on specific devices as seen by the throttling policy.'
4431
+ info: cgroupThrottleIOServicesOps
4432
},
4258
-
4433
'cgroup.io_some_pressure': {
4260
- info: 'I/O <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. '+
4261
- '<b>Some</b> indicates the share of time in which at least <b>some tasks</b> are stalled on I/O. ' +
4262
- 'In this state the CPU is still doing productive work. '+
4263
- 'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
4434
+ info: cgroupIOSomePressure
4435
},
4436
'cgroup.io_some_pressure_stall_time': {
4266
- info: 'The amount of time some processes have been waiting due to I/O congestion.'
4437
+ info: cgroupIOSomePRessureStallTime
4438
},
4268
-
4439
'cgroup.io_full_pressure': {
4270
- info: 'I/O <a href="https://www.kernel.org/doc/html/latest/accounting/psi.html" target="_blank">Pressure Stall Information</a>. ' +
4271
- '<b>Full</b> line indicates the share of time in which <b>all non-idle tasks</b> are stalled on I/O resource simultaneously. ' +
4272
- 'In this state actual CPU cycles are going to waste, and a workload that spends extended time in this state is considered to be thrashing. '+
4273
- 'This has severe impact on performance. '+
4274
- 'The ratios are tracked as recent trends over 10-, 60-, and 300-second windows.'
4440
+ info: cgroupIOFullPressure
4441
},
4442
'cgroup.io_full_pressure_stall_time': {
4277
- info: 'The amount of time all non-idle processes have been stalled due to I/O congestion.'
4443
+ info: cgroupIOFullPressureStallTime
4444
+ },
4445
+
4446
+ 'k8s.cgroup.io': {
4447
+ info: cgroupIO
4448
+ },
4449
+ 'k8s.cgroup.serviced_ops': {
4450
+ info: cgroupServicedOps
4451
+ },
4452
+ 'k8s.cgroup.queued_ops': {
4453
+ info: cgroupQueuedOps
4454
+ },
4455
+ 'k8s.cgroup.merged_ops': {
4456
+ info: cgroupMergedOps
4457
+ },
4458
+ 'k8s.cgroup.throttle_io': {
4459
+ mainheads: [
4460
+ function (_, id) {
4461
+ return '<div data-netdata="' + id + '"'
4462
+ + ' data-dimensions="read"'
4463
+ + ' data-chart-library="gauge"'
4464
+ + ' data-title="Read Disk I/O"'
4465
+ + ' data-units="KB/s"'
4466
+ + ' data-gauge-adjust="width"'
4467
+ + ' data-width="12%"'
4468
+ + ' data-before="0"'
4469
+ + ' data-after="-CHART_DURATION"'
4470
+ + ' data-points="CHART_DURATION"'
4471
+ + ' data-colors="' + NETDATA.colors[2] + '"'
4472
+ + ' role="application"></div>';
4473
+ },
4474
+ function (_, id) {
4475
+ return '<div data-netdata="' + id + '"'
4476
+ + ' data-dimensions="write"'
4477
+ + ' data-chart-library="gauge"'
4478
+ + ' data-title="Write Disk I/O"'
4479
+ + ' data-units="KB/s"'
4480
+ + ' data-gauge-adjust="width"'
4481
+ + ' data-width="12%"'
4482
+ + ' data-before="0"'
4483
+ + ' data-after="-CHART_DURATION"'
4484
+ + ' data-points="CHART_DURATION"'
4485
+ + ' data-colors="' + NETDATA.colors[3] + '"'
4486
+ + ' role="application"></div>';
4487
+ }
4488
+ ],
4489
+ info: cgroupThrottleIO
4490
+ },
4491
+ 'k8s.cgroup.throttle_serviced_ops': {
4492
+ info: cgroupThrottleIOServicesOps
4493
+ },
4494
+ 'k8s.cgroup.io_some_pressure': {
4495
+ info: cgroupIOSomePressure
4496
+ },
4497
+ 'k8s.cgroup.io_some_pressure_stall_time': {
4498
+ info: cgroupIOSomePRessureStallTime
4499
+ },
4500
+ 'k8s.cgroup.io_full_pressure': {
4501
+ info: cgroupIOFullPressure
4502
+ },
4503
+ 'k8s.cgroup.io_full_pressure_stall_time': {
4504
+ info: cgroupIOFullPressureStallTime
4505
},
4506
4507
'cgroup.swap_read': {
4508
info: ebpfSwapRead
4509
},
4283
-
4510
'cgroup.swap_write': {
4511
info: ebpfSwapWrite
4512
},
4287
-
4513
'cgroup.fd_open': {
4514
info: ebpfFileOpen
4515
},
4291
-
4516
'cgroup.fd_open_error': {
4517
info: ebpfFileOpenError
4518
},
4295
-
4519
'cgroup.fd_close': {
4520
info: ebpfFileClosed
4521
},
4299
-
4522
'cgroup.fd_close_error': {
4523
info: ebpfFileCloseError
4524
},
4303
-
4525
'cgroup.vfs_unlink': {
4526
info: ebpfVFSUnlink
4527
},
4307
-
4528
'cgroup.vfs_write': {
4529
info: ebpfVFSWrite
4530
},
4311
-
4531
'cgroup.vfs_write_error': {
4532
info: ebpfVFSWriteError
4533
},
4315
-
4534
'cgroup.vfs_read': {
4535
info: ebpfVFSRead
4536
},
4319
-
4537
'cgroup.vfs_read_error': {
4538
info: ebpfVFSReadError
4539
},
4323
-
4540
'cgroup.vfs_write_bytes': {
4541
info: ebpfVFSWriteBytes
4542
},
4327
-
4543
'cgroup.vfs_read_bytes': {
4544
info: ebpfVFSReadBytes
4545
},
4331
-
4546
'cgroup.vfs_fsync': {
4547
info: ebpfVFSSync
4548
},
4335
-
4549
'cgroup.vfs_fsync_error': {
4550
info: ebpfVFSSyncError
4551
},
4339
-
4552
'cgroup.vfs_open': {
4553
info: ebpfVFSOpen
4554
},
4343
-
4555
'cgroup.vfs_open_error': {
4556
info: ebpfVFSOpenError
4557
},
4347
-
4558
'cgroup.vfs_create': {
4559
info: ebpfVFSCreate
4560
},
4351
-
4561
'cgroup.vfs_create_error': {
4562
info: ebpfVFSCreateError
4563
},
4355
-
4564
'cgroup.process_create': {
4565
info: ebpfProcessCreate
4566
},
4359
-
4567
'cgroup.thread_create': {
4568
info: ebpfThreadCreate
4569
},
4363
-
4570
'cgroup.task_exit': {
4571
info: ebpfTaskExit
4572
},
4367
-
4573
'cgroup.task_close': {
4574
info: ebpfTaskClose
4575
},
4371
-
4576
'cgroup.task_error': {
4577
info: ebpfTaskError
4578
},
4375
-
4579
'cgroup.dc_ratio': {
4580
info: 'Percentage of file accesses that were present in the directory cache. 100% means that every file that was accessed was present in the directory cache. If files are not present in the directory cache 1) they are not present in the file system, 2) the files were not accessed before. Read more about <a href="https://www.kernel.org/doc/htmldocs/filesystems/the_directory_cache.html" target="_blank">directory cache</a>. Netdata also gives a summary for these charts in <a href="#menu_filesystem_submenu_directory_cache__eBPF_">Filesystem submenu</a>.'
4581
},
4379
-
4582
'cgroup.shmget': {
4583
info: ebpfSHMget
4584
},
4383
-
4585
'cgroup.shmat': {
4586
info: ebpfSHMat
4587
},
4387
-
4588
'cgroup.shmdt': {
4589
info: ebpfSHMdt
4590
},
4391
-
4591
'cgroup.shmctl': {
4592
info: ebpfSHMctl
4593
},
4395
-
4594
'cgroup.outbound_conn_v4': {
4595
info: ebpfIPV4conn
4596
},
4399
-
4597
'cgroup.outbound_conn_v6': {
4598
info: ebpfIPV6conn
4599
},
4403
-
4600
'cgroup.net_bytes_send': {
4601
info: ebpfBandwidthSent
4602
},
4407
-
4603
'cgroup.net_bytes_recv': {
4604
info: ebpfBandwidthRecv
4605
},
4411
-
4606
'cgroup.net_tcp_send': {
4607
info: ebpfTCPSendCall
4608
},
4415
-
4609
'cgroup.net_tcp_recv': {
4610
info: ebpfTCPRecvCall
4611
},
4419
-
4612
'cgroup.net_retransmit': {
4613
info: ebpfTCPRetransmit
4614
},
4423
-
4615
'cgroup.net_udp_send': {
4616
info: ebpfUDPsend
4617
},
4427
-
4618
'cgroup.net_udp_recv': {
4619
info: ebpfUDPrecv
4620
},
4431
-
4621
'cgroup.dc_hit_ratio': {
4622
info: ebpfDCHit
4623
},
4435
-
4624
'cgroup.dc_reference': {
4625
info: ebpfDCReference
4626
},
4439
-
4627
'cgroup.dc_not_cache': {
4628
info: ebpfDCNotCache
4629
},
4443
-
4630
'cgroup.dc_not_found': {
4631
info: ebpfDCNotFound
4632
},
4447
-
4633
'cgroup.cachestat_ratio': {
4634
info: ebpfCachestatRatio
4635
},
4451
-
4636
'cgroup.cachestat_dirties': {
4637
info: ebpfCachestatDirties
4638
},
4455
-
4639
'cgroup.cachestat_hits': {
4640
info: ebpfCachestatHits
4641
},
4459
-
4642
'cgroup.cachestat_misses': {
4643
info: ebpfCachestatMisses
4644
},