Replication of metrics (gaps filling) during streaming (#13873)
* Revert "Use llvm's ar and ranlib when compiling with clang (#13854)" This reverts commit a9135f47bbb36e9cb437b18a7109607569580db7. * Profile plugin * Fix macos static thread * Add support for replication - Add a new capability for replication, when not supported the agent should behave as previously. - When replication is supported, the text protocol supports the following new commands: - CHART_DEFINITION_END: send the first/last entry of the child - REPLAY_RRDSET_BEGIN: sends the name of the chart we are replicating - REPLAY_RRDSET_HEADER: sends a line describing the columns of the following command (ie. start-time, end-time, dim1-name, ...) - REPLAY_RRDSET_DONE: sends values to push for a specific start/end time - REPLAY_RRDSET_END: send the (a) update every of the chart, (b) first/last entries in DB, (c) whether the child's been told to start streaming, (d) original after/before period to replicate. - REPLAY_CHART: Sent from a parent to a child, specifying (a) the chart name we want data for, (b) whether the child should start streaming once it has fullfilled the request with the aforementioned commands, (c) after/before of the data the parent wants - As a consequence of the new protocol, streaming is disabled for all charts on a new connection. It's enabled once replication is finished. - The configuration parameters are specified from within stream.conf: - "enable replication = yes|no" - "seconds to replicate = 3600" - "replication step = 600" (ie. how many seconds to fill per roundtrip request. * Minor fixes - quote set and dim ids - start streaming after writing replicated data to the buffer - write replicated data only when buffer is less than 50% full. - use reentrant iteration for charts * Do not send chart definitions on connection. * Track replication status through rrdset flags. * Add debug flag for noisy log messages. * Add license notice. * Iterate charts with reentrant loop * Set replication finished flag when streaming is disabled. * Revert "Profile plugin" This reverts commit 468fc9386e5283e0865fae56e9989b8ec83de14d. Used only for testing purposes. * Revert "Revert "Use llvm's ar and ranlib when compiling with clang (#13854)"" This reverts commit 27c955c58d95aed6c44d42e8b675f0cf3ca45c6d. Reapply commit that I had to revert in order to be able to build the agent on MacOS. * Build replication source files with CMake. * Pass number of words in plugind functions. * Use get_word instead of indexing words. * Use size_t instead of int. * Pay only what we use when splitting words. * no need to redefine PLUGINSD_MAX_WORDS * fix formatting warning * all usages of pluginsd_split_words() should use the return value to ensure non-cached results reuse; no need to lock the host to find a chart * keep a sender dictionary with all the replication commands received and remove replication commands from charts * do not replicate future data * use last_updated to find the end of the db * uniformity of replication logs * rewrite of the query logic * replication.c in C; debug info in human readable dates * update the chart on every replication row * update all chart members so that rrdset_done() can continue * update the protocol to push one dimension per line and transfer data collection state to parent * fix formatting * remove replication object from pluginsd * shorter communication * fix typo * support for replication proxies * proper use of flags * set receiver replication finished flag on charts created after the sender has been connected * clear RRDSET_FLAG_SYNC_CLOCK on replicated charts * log storing of nulls * log first store * log update every switches * test ignoring timestamps but sending a point just after replication end * replication should work on end_time * use replicated timestamps * at the final replication step, replicate all the remaining points * cleanup code from tests * print timestamps as unsigned long long * more formating changes; fix conflicting type of replicate_chart_response() * updated stream.conf * always respond to replication requests * in non-dbengine db modes, do not replicate more than the database size * advance the db pointer of legacy db modes * should be multiplied by update_every * fix buggy label parsing - identified by codacy * dont log error on history mismatches for db mode dbengine * allow SSL requests to streaming children * dont use ssl variable Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
vkalintiris committed
Oct 31, 2022 at 19:53 UTC
282e0dfaa97289cc6542742e9e389bd76b7e4164
31 files changed
+1442
-555
CMakeLists.txt
+2
@@ -860,6 +860,8 @@ set(STREAMING_PLUGIN_FILES
860
streaming/compression.c
861
streaming/receiver.c
862
streaming/sender.c
863
+ streaming/replication.c
864
+ streaming/replication.h
865
)
866
867
set(CLAIM_PLUGIN_FILES
Makefile.am
+2
@@ -622,6 +622,8 @@ STREAMING_PLUGIN_FILES = \
622
streaming/compression.c \
623
streaming/sender.c \
624
streaming/receiver.c \
625
+ streaming/replication.h \
626
+ streaming/replication.c \
627
streaming/rrdpush.h \
628
$(NULL)
629
collectors/apps.plugin/apps_plugin.c
+28
-25
@@ -4311,7 +4311,7 @@ static void apps_plugin_function_processes(const char *transaction, char *functi
4311
struct pid_stat *p;
4312
4313
char *words[PLUGINSD_MAX_WORDS] = { NULL };
4314
- pluginsd_split_words(function, words, PLUGINSD_MAX_WORDS, NULL, NULL, 0);
4314
+ size_t num_words = pluginsd_split_words(function, words, PLUGINSD_MAX_WORDS, NULL, NULL, 0);
4315
4316
struct target *category = NULL, *user = NULL, *group = NULL;
4317
const char *process_name = NULL;
@@ -4322,51 +4322,52 @@ static void apps_plugin_function_processes(const char *transaction, char *functi
4322
bool filter_pid = false, filter_uid = false, filter_gid = false;
4323
4324
for(int i = 1; i < PLUGINSD_MAX_WORDS ;i++) {
4325
- if(!words[i]) break;
4325
+ const char *keyword = get_word(words, num_words, i);
4326
+ if(!keyword) break;
4327
4327
- if(!category && strncmp(words[i], PROCESS_FILTER_CATEGORY, strlen(PROCESS_FILTER_CATEGORY)) == 0) {
4328
- category = find_target_by_name(apps_groups_root_target, &words[i][strlen(PROCESS_FILTER_CATEGORY)]);
4328
+ if(!category && strncmp(keyword, PROCESS_FILTER_CATEGORY, strlen(PROCESS_FILTER_CATEGORY)) == 0) {
4329
+ category = find_target_by_name(apps_groups_root_target, &keyword[strlen(PROCESS_FILTER_CATEGORY)]);
4330
if(!category) {
4331
apps_plugin_function_error(transaction, HTTP_RESP_BAD_REQUEST, "No category with that name found.");
4332
return;
4333
}
4334
}
4334
- else if(!user && strncmp(words[i], PROCESS_FILTER_USER, strlen(PROCESS_FILTER_USER)) == 0) {
4335
- user = find_target_by_name(users_root_target, &words[i][strlen(PROCESS_FILTER_USER)]);
4335
+ else if(!user && strncmp(keyword, PROCESS_FILTER_USER, strlen(PROCESS_FILTER_USER)) == 0) {
4336
+ user = find_target_by_name(users_root_target, &keyword[strlen(PROCESS_FILTER_USER)]);
4337
if(!user) {
4338
apps_plugin_function_error(transaction, HTTP_RESP_BAD_REQUEST, "No user with that name found.");
4339
return;
4340
}
4341
}
4341
- else if(strncmp(words[i], PROCESS_FILTER_GROUP, strlen(PROCESS_FILTER_GROUP)) == 0) {
4342
- group = find_target_by_name(groups_root_target, &words[i][strlen(PROCESS_FILTER_GROUP)]);
4342
+ else if(strncmp(keyword, PROCESS_FILTER_GROUP, strlen(PROCESS_FILTER_GROUP)) == 0) {
4343
+ group = find_target_by_name(groups_root_target, &keyword[strlen(PROCESS_FILTER_GROUP)]);
4344
if(!group) {
4345
apps_plugin_function_error(transaction, HTTP_RESP_BAD_REQUEST, "No group with that name found.");
4346
return;
4347
}
4348
}
4348
- else if(!process_name && strncmp(words[i], PROCESS_FILTER_PROCESS, strlen(PROCESS_FILTER_PROCESS)) == 0) {
4349
- process_name = &words[i][strlen(PROCESS_FILTER_PROCESS)];
4349
+ else if(!process_name && strncmp(keyword, PROCESS_FILTER_PROCESS, strlen(PROCESS_FILTER_PROCESS)) == 0) {
4350
+ process_name = &keyword[strlen(PROCESS_FILTER_PROCESS)];
4351
}
4351
- else if(!pid && strncmp(words[i], PROCESS_FILTER_PID, strlen(PROCESS_FILTER_PID)) == 0) {
4352
- pid = str2i(&words[i][strlen(PROCESS_FILTER_PID)]);
4352
+ else if(!pid && strncmp(keyword, PROCESS_FILTER_PID, strlen(PROCESS_FILTER_PID)) == 0) {
4353
+ pid = str2i(&keyword[strlen(PROCESS_FILTER_PID)]);
4354
filter_pid = true;
4355
}
4355
- else if(!uid && strncmp(words[i], PROCESS_FILTER_UID, strlen(PROCESS_FILTER_UID)) == 0) {
4356
- uid = str2i(&words[i][strlen(PROCESS_FILTER_UID)]);
4356
+ else if(!uid && strncmp(keyword, PROCESS_FILTER_UID, strlen(PROCESS_FILTER_UID)) == 0) {
4357
+ uid = str2i(&keyword[strlen(PROCESS_FILTER_UID)]);
4358
filter_uid = true;
4359
}
4359
- else if(!gid && strncmp(words[i], PROCESS_FILTER_GID, strlen(PROCESS_FILTER_GID)) == 0) {
4360
- gid = str2i(&words[i][strlen(PROCESS_FILTER_GID)]);
4360
+ else if(!gid && strncmp(keyword, PROCESS_FILTER_GID, strlen(PROCESS_FILTER_GID)) == 0) {
4361
+ gid = str2i(&keyword[strlen(PROCESS_FILTER_GID)]);
4362
filter_gid = true;
4363
}
4363
- else if(strcmp(words[i], "help") == 0) {
4364
+ else if(strcmp(keyword, "help") == 0) {
4365
apps_plugin_function_processes_help(transaction);
4366
return;
4367
}
4368
else {
4369
char msg[PLUGINSD_LINE_MAX];
4369
- snprintfz(msg, PLUGINSD_LINE_MAX, "Invalid parameter '%s'", words[i]);
4370
+ snprintfz(msg, PLUGINSD_LINE_MAX, "Invalid parameter '%s'", keyword);
4371
apps_plugin_function_error(transaction, HTTP_RESP_BAD_REQUEST, msg);
4372
return;
4373
}
@@ -4779,16 +4780,18 @@ void *reader_main(void *arg __maybe_unused) {
4780
while(!apps_plugin_exit && (s = fgets(buffer, PLUGINSD_LINE_MAX, stdin))) {
4781
4782
char *words[PLUGINSD_MAX_WORDS] = { NULL };
4782
- pluginsd_split_words(buffer, words, PLUGINSD_MAX_WORDS, NULL, NULL, 0);
4783
+ size_t num_words = pluginsd_split_words(buffer, words, PLUGINSD_MAX_WORDS, NULL, NULL, 0);
4784
4784
- if(words[0] && strcmp(words[0], PLUGINSD_KEYWORD_FUNCTION) == 0) {
4785
- char *transaction = words[1];
4786
- char *timeout_s = words[2];
4787
- char *function = words[3];
4785
+ const char *keyword = get_word(words, num_words, 0);
4786
+
4787
+ if(keyword && strcmp(keyword, PLUGINSD_KEYWORD_FUNCTION) == 0) {
4788
+ char *transaction = get_word(words, num_words, 1);
4789
+ char *timeout_s = get_word(words, num_words, 2);
4790
+ char *function = get_word(words, num_words, 3);
4791
4792
if(!transaction || !*transaction || !timeout_s || !*timeout_s || !function || !*function) {
4793
error("Received incomplete %s (transaction = '%s', timeout = '%s', function = '%s'). Ignoring it.",
4791
- words[0],
4794
+ keyword,
4795
transaction?transaction:"(unset)",
4796
timeout_s?timeout_s:"(unset)",
4797
function?function:"(unset)");
@@ -4813,7 +4816,7 @@ void *reader_main(void *arg __maybe_unused) {
4816
}
4817
}
4818
else
4816
- error("Received unknown command: %s", words[0]?words[0]:"(unset)");
4819
+ error("Received unknown command: %s", keyword?keyword:"(unset)");
4820
}
4821
4822
if(!s || feof(stdin) || ferror(stdin)) {
collectors/plugins.d/plugins_d.c
+1
-1
@@ -6,7 +6,7 @@
6
char *plugin_directories[PLUGINSD_MAX_DIRECTORIES] = { NULL };
7
struct plugind *pluginsd_root = NULL;
8
9
-inline int pluginsd_initialize_plugin_directories()
9
+inline size_t pluginsd_initialize_plugin_directories()
10
{
11
char plugins_dirs[(FILENAME_MAX * 2) + 1];
12
static char *plugins_dir_list = NULL;
collectors/plugins.d/plugins_d.h
+10
-3
@@ -11,6 +11,7 @@
11
#define PLUGINSD_STOCK_PLUGINS_DIRECTORY_PATH 0
12
13
#define PLUGINSD_KEYWORD_CHART "CHART"
14
+#define PLUGINSD_KEYWORD_CHART_DEFINITION_END "CHART_DEFINITION_END"
15
#define PLUGINSD_KEYWORD_DIMENSION "DIMENSION"
16
#define PLUGINSD_KEYWORD_BEGIN "BEGIN"
17
#define PLUGINSD_KEYWORD_SET "SET"
@@ -29,12 +30,18 @@
30
#define PLUGINSD_KEYWORD_CONTEXT "CONTEXT"
31
#define PLUGINSD_KEYWORD_TOMBSTONE "TOMBSTONE"
32
#define PLUGINSD_KEYWORD_HOST "HOST"
32
-//#define PLUGINSD_KEYWORD_GAPS_REQUEST "GAPS_REQUEST" // child -> parent
33
-//#define PLUGINSD_KEYWORD_CHART_GAP "CHART_GAP" // parent <- child
33
+
34
+#define PLUGINSD_KEYWORD_REPLAY_CHART "REPLAY_CHART"
35
+#define PLUGINSD_KEYWORD_REPLAY_BEGIN "RBEGIN"
36
+#define PLUGINSD_KEYWORD_REPLAY_SET "RSET"
37
+#define PLUGINSD_KEYWORD_REPLAY_RRDDIM_STATE "RDSTATE"
38
+#define PLUGINSD_KEYWORD_REPLAY_RRDSET_STATE "RSSTATE"
39
+#define PLUGINSD_KEYWORD_REPLAY_END "REND"
40
41
#define PLUGINS_FUNCTIONS_TIMEOUT_DEFAULT 10 // seconds
42
43
#define PLUGINSD_LINE_MAX_SSL_READ 512
44
+
45
#define PLUGINSD_MAX_WORDS 20
46
47
#define PLUGINSD_MAX_DIRECTORIES 20
@@ -69,7 +76,7 @@ extern struct plugind *pluginsd_root;
76
77
size_t pluginsd_process(RRDHOST *host, struct plugind *cd, FILE *fp_plugin_input, FILE *fp_plugin_output, int trust_durations);
78
72
-int pluginsd_initialize_plugin_directories();
79
+size_t pluginsd_initialize_plugin_directories();
80
81
82
collectors/plugins.d/pluginsd_parser.c
+457
-106
@@ -4,10 +4,35 @@
4
5
#define LOG_FUNCTIONS false
6
7
-PARSER_RC pluginsd_set(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
7
+static int send_to_plugin(const char *txt, void *data) {
8
+ PARSER *parser = data;
9
+
10
+ if(!txt || !*txt)
11
+ return 0;
12
+
13
+#ifdef ENABLE_HTTPS
14
+ struct netdata_ssl *ssl = parser->ssl_output;
15
+ if(ssl) {
16
+ if(ssl->conn && ssl->flags == NETDATA_SSL_HANDSHAKE_COMPLETE) {
17
+ size_t size = strlen(txt);
18
+ return SSL_write(ssl->conn, txt, (int)size);
19
+ }
20
+
21
+ error("cannot write to SSL connection - connection is not ready.");
22
+ return -1;
23
+ }
24
+#endif
25
+
26
+ FILE *fp = parser->output;
27
+ int ret = fprintf(fp, "%s", txt);
28
+ fflush(fp);
29
+ return ret;
30
+}
31
+
32
+PARSER_RC pluginsd_set(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
33
{
9
- char *dimension = words[1];
10
- char *value = words[2];
34
+ char *dimension = get_word(words, num_words, 1);
35
+ char *value = get_word(words, num_words, 2);
36
37
RRDSET *st = ((PARSER_USER_OBJECT *) user)->st;
38
RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
@@ -47,10 +72,10 @@ disable:
72
return PARSER_RC_ERROR;
73
}
74
50
-PARSER_RC pluginsd_begin(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
75
+PARSER_RC pluginsd_begin(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
76
{
52
- char *id = words[1];
53
- char *microseconds_txt = words[2];
77
+ char *id = get_word(words, num_words, 1);
78
+ char *microseconds_txt = get_word(words, num_words, 2);
79
80
RRDSET *st = NULL;
81
RRDHOST *host = ((PARSER_USER_OBJECT *)user)->host;
@@ -86,9 +111,11 @@ disable:
111
return PARSER_RC_ERROR;
112
}
113
89
-PARSER_RC pluginsd_end(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
114
+PARSER_RC pluginsd_end(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
115
{
116
UNUSED(words);
117
+ UNUSED(num_words);
118
+
119
RRDSET *st = ((PARSER_USER_OBJECT *) user)->st;
120
RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
121
@@ -107,7 +134,7 @@ PARSER_RC pluginsd_end(char **words, void *user, PLUGINSD_ACTION *plugins_actio
134
return PARSER_RC_OK;
135
}
136
110
-PARSER_RC pluginsd_chart(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
137
+PARSER_RC pluginsd_chart(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
138
{
139
RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
140
if (unlikely(!host && !((PARSER_USER_OBJECT *) user)->host_exists)) {
@@ -115,18 +142,18 @@ PARSER_RC pluginsd_chart(char **words, void *user, PLUGINSD_ACTION *plugins_act
142
return PARSER_RC_OK;
143
}
144
118
- char *type = words[1];
119
- char *name = words[2];
120
- char *title = words[3];
121
- char *units = words[4];
122
- char *family = words[5];
123
- char *context = words[6];
124
- char *chart = words[7];
125
- char *priority_s = words[8];
126
- char *update_every_s = words[9];
127
- char *options = words[10];
128
- char *plugin = words[11];
129
- char *module = words[12];
145
+ char *type = get_word(words, num_words, 1);
146
+ char *name = get_word(words, num_words, 2);
147
+ char *title = get_word(words, num_words, 3);
148
+ char *units = get_word(words, num_words, 4);
149
+ char *family = get_word(words, num_words, 5);
150
+ char *context = get_word(words, num_words, 6);
151
+ char *chart = get_word(words, num_words, 7);
152
+ char *priority_s = get_word(words, num_words, 8);
153
+ char *update_every_s = get_word(words, num_words, 9);
154
+ char *options = get_word(words, num_words, 10);
155
+ char *plugin = get_word(words, num_words, 11);
156
+ char *module = get_word(words, num_words, 12);
157
158
// parse the id from type
159
char *id = NULL;
@@ -231,14 +258,36 @@ PARSER_RC pluginsd_chart(char **words, void *user, PLUGINSD_ACTION *plugins_act
258
return PARSER_RC_OK;
259
}
260
234
-PARSER_RC pluginsd_dimension(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
261
+PARSER_RC pluginsd_chart_definition_end(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action)
262
{
236
- char *id = words[1];
237
- char *name = words[2];
238
- char *algorithm = words[3];
239
- char *multiplier_s = words[4];
240
- char *divisor_s = words[5];
241
- char *options = words[6];
263
+ UNUSED(plugins_action);
264
+
265
+ long first_entry_child = str2l(get_word(words, num_words, 1));
266
+ long last_entry_child = str2l(get_word(words, num_words, 2));
267
+
268
+ PARSER_USER_OBJECT *user_object = (PARSER_USER_OBJECT *) user;
269
+
270
+ RRDHOST *host = user_object->host;
271
+ RRDSET *st = user_object->st;
272
+ if(unlikely(!host || !st)) {
273
+ error("REPLAY: received " PLUGINSD_KEYWORD_CHART_DEFINITION_END " command without a chart. Disabling it.");
274
+ return PARSER_RC_ERROR;
275
+ }
276
+
277
+ rrdset_flag_clear(st, RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
278
+
279
+ bool ok = replicate_chart_request(send_to_plugin, user_object->parser, host, st, first_entry_child, last_entry_child, 0, 0);
280
+ return ok ? PARSER_RC_OK : PARSER_RC_ERROR;
281
+}
282
+
283
+PARSER_RC pluginsd_dimension(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
284
+{
285
+ char *id = get_word(words, num_words, 1);
286
+ char *name = get_word(words, num_words, 2);
287
+ char *algorithm = get_word(words, num_words, 3);
288
+ char *multiplier_s = get_word(words, num_words, 4);
289
+ char *divisor_s = get_word(words, num_words, 5);
290
+ char *options = get_word(words, num_words, 6);
291
292
RRDSET *st = ((PARSER_USER_OBJECT *) user)->st;
293
RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
@@ -341,16 +390,18 @@ static void inflight_functions_insert_callback(const DICTIONARY_ITEM *item, void
390
struct inflight_function *pf = func;
391
392
PARSER *parser = parser_ptr;
344
- FILE *fp = parser->output;
393
394
// leave this code as default, so that when the dictionary is destroyed this will be sent back to the caller
395
pf->code = HTTP_RESP_GATEWAY_TIMEOUT;
396
397
+ char buffer[2048 + 1];
398
+ snprintfz(buffer, 2048, "FUNCTION %s %d \"%s\"\n",
399
+ dictionary_acquired_item_name(item),
400
+ pf->timeout,
401
+ string2str(pf->function));
402
+
403
// send the command to the plugin
350
- int ret = fprintf(fp, "FUNCTION %s %d \"%s\"\n",
351
- dictionary_acquired_item_name(item),
352
- pf->timeout,
353
- string2str(pf->function));
404
+ int ret = send_to_plugin(buffer, parser);
405
406
pf->sent_ut = now_realtime_usec();
407
@@ -359,11 +410,9 @@ static void inflight_functions_insert_callback(const DICTIONARY_ITEM *item, void
410
rrd_call_function_error(pf->destination_wb, "Failed to communicate with collector", HTTP_RESP_BACKEND_FETCH_FAILED);
411
}
412
else {
362
- fflush(fp);
363
-
413
internal_error(LOG_FUNCTIONS,
365
- "FUNCTION '%s' with transaction '%s' sent to collector (%d bytes, fd %d, in %llu usec)",
366
- string2str(pf->function), dictionary_acquired_item_name(item), ret, fileno(fp),
414
+ "FUNCTION '%s' with transaction '%s' sent to collector (%d bytes, in %llu usec)",
415
+ string2str(pf->function), dictionary_acquired_item_name(item), ret,
416
pf->sent_ut - pf->started_ut);
417
}
418
}
@@ -461,18 +510,18 @@ static int pluginsd_execute_function_callback(BUFFER *destination_wb, int timeou
510
return HTTP_RESP_OK;
511
}
512
464
-PARSER_RC pluginsd_function(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
513
+PARSER_RC pluginsd_function(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
514
{
515
bool global = false;
467
- int i = 1;
468
- if(strcmp(words[i], "GLOBAL") == 0) {
516
+ size_t i = 1;
517
+ if(num_words >= 2 && strcmp(get_word(words, num_words, 1), "GLOBAL") == 0) {
518
i++;
519
global = true;
520
}
521
473
- char *name = words[i++];
474
- char *timeout_s = words[i++];
475
- char *help = words[i++];
522
+ char *name = get_word(words, num_words, i++);
523
+ char *timeout_s = get_word(words, num_words, i++);
524
+ char *help = get_word(words, num_words, i++);
525
526
RRDSET *st = (global)?NULL:((PARSER_USER_OBJECT *) user)->st;
527
RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
@@ -508,12 +557,12 @@ static void pluginsd_function_result_end(struct parser *parser, void *action_dat
557
string_freez(key);
558
}
559
511
-PARSER_RC pluginsd_function_result_begin(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
560
+PARSER_RC pluginsd_function_result_begin(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
561
{
513
- char *key = words[1];
514
- char *status = words[2];
515
- char *format = words[3];
516
- char *expires = words[4];
562
+ char *key = get_word(words, num_words, 1);
563
+ char *status = get_word(words, num_words, 2);
564
+ char *format = get_word(words, num_words, 3);
565
+ char *expires = get_word(words, num_words, 4);
566
567
if (unlikely(!key || !*key || !status || !*status || !format || !*format || !expires || !*expires)) {
568
error("got a " PLUGINSD_KEYWORD_FUNCTION_RESULT_BEGIN " without providing the required data (key = '%s', status = '%s', format = '%s', expires = '%s')."
@@ -564,10 +613,10 @@ PARSER_RC pluginsd_function_result_begin(char **words, void *user, PLUGINSD_ACTI
613
614
// ----------------------------------------------------------------------------
615
567
-PARSER_RC pluginsd_variable(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
616
+PARSER_RC pluginsd_variable(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
617
{
569
- char *name = words[1];
570
- char *value = words[2];
618
+ char *name = get_word(words, num_words, 1);
619
+ char *value = get_word(words, num_words, 2);
620
NETDATA_DOUBLE v;
621
622
RRDSET *st = ((PARSER_USER_OBJECT *) user)->st;
@@ -578,12 +627,12 @@ PARSER_RC pluginsd_variable(char **words, void *user, PLUGINSD_ACTION *plugins_
627
if (name && *name) {
628
if ((strcmp(name, "GLOBAL") == 0 || strcmp(name, "HOST") == 0)) {
629
global = 1;
581
- name = words[2];
582
- value = words[3];
630
+ name = get_word(words, num_words, 2);
631
+ value = get_word(words, num_words, 3);
632
} else if ((strcmp(name, "LOCAL") == 0 || strcmp(name, "CHART") == 0)) {
633
global = 0;
585
- name = words[2];
586
- value = words[3];
634
+ name = get_word(words, num_words, 2);
635
+ value = get_word(words, num_words, 3);
636
}
637
}
638
@@ -641,69 +690,78 @@ PARSER_RC pluginsd_variable(char **words, void *user, PLUGINSD_ACTION *plugins_
690
return PARSER_RC_OK;
691
}
692
644
-PARSER_RC pluginsd_flush(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
693
+PARSER_RC pluginsd_flush(char **words __maybe_unused, size_t num_words __maybe_unused, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
694
{
646
- UNUSED(words);
695
debug(D_PLUGINSD, "requested a FLUSH");
696
((PARSER_USER_OBJECT *) user)->st = NULL;
697
+ ((PARSER_USER_OBJECT *) user)->replay.start_time = 0;
698
+ ((PARSER_USER_OBJECT *) user)->replay.end_time = 0;
699
+ ((PARSER_USER_OBJECT *) user)->replay.start_time_ut = 0;
700
+ ((PARSER_USER_OBJECT *) user)->replay.end_time_ut = 0;
701
return PARSER_RC_OK;
702
}
703
652
-PARSER_RC pluginsd_disable(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
704
+PARSER_RC pluginsd_disable(char **words __maybe_unused, size_t num_words __maybe_unused, void *user __maybe_unused, PLUGINSD_ACTION *plugins_action __maybe_unused)
705
{
654
- UNUSED(user);
655
- UNUSED(words);
656
-
706
info("called DISABLE. Disabling it.");
707
((PARSER_USER_OBJECT *) user)->enabled = 0;
708
return PARSER_RC_ERROR;
709
}
710
662
-PARSER_RC pluginsd_label(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
711
+PARSER_RC pluginsd_label(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
712
{
664
- char *store;
713
+ const char *name = get_word(words, num_words, 1);
714
+ const char *label_source = get_word(words, num_words, 2);
715
+ const char *value = get_word(words, num_words, 3);
716
666
- if (!words[1] || !words[2] || !words[3]) {
717
+ if (!name || !label_source || !value) {
718
error("Ignoring malformed or empty LABEL command.");
719
return PARSER_RC_OK;
720
}
670
- if (!words[4])
671
- store = words[3];
672
- else {
673
- store = callocz(PLUGINSD_LINE_MAX + 1, sizeof(char));
721
+
722
+ char *store = (char *)value;
723
+ bool allocated_store = false;
724
+
725
+ if(unlikely(num_words > 4)) {
726
+ allocated_store = true;
727
+ store = mallocz(PLUGINSD_LINE_MAX + 1);
728
size_t remaining = PLUGINSD_LINE_MAX;
729
char *move = store;
676
- int i = 3;
677
- while (i < PLUGINSD_MAX_WORDS) {
678
- size_t length = strlen(words[i]);
679
- if ((length + 1) >= remaining)
680
- break;
681
-
682
- remaining -= (length + 1);
683
- memcpy(move, words[i], length);
730
+ char *word;
731
+ for(size_t i = 3; i < num_words && remaining > 2 && (word = get_word(words, num_words, i)) ;i++) {
732
+ if(i > 3) {
733
+ *move++ = ' ';
734
+ *move = '\0';
735
+ remaining--;
736
+ }
737
+
738
+ size_t length = strlen(word);
739
+ if (length > remaining)
740
+ length = remaining;
741
+
742
+ remaining -= length;
743
+ memcpy(move, word, length);
744
move += length;
685
- *move++ = ' ';
686
-
687
- i++;
688
- if (!words[i])
689
- break;
745
+ *move = '\0';
746
}
747
}
748
749
if(unlikely(!((PARSER_USER_OBJECT *) user)->new_host_labels))
750
((PARSER_USER_OBJECT *) user)->new_host_labels = rrdlabels_create();
751
696
- rrdlabels_add(((PARSER_USER_OBJECT *)user)->new_host_labels, words[1], store, strtol(words[2], NULL, 10));
752
+ rrdlabels_add(((PARSER_USER_OBJECT *)user)->new_host_labels,
753
+ name,
754
+ store,
755
+ str2l(label_source));
756
698
- if (store != words[3])
757
+ if (allocated_store)
758
freez(store);
759
+
760
return PARSER_RC_OK;
761
}
762
703
-PARSER_RC pluginsd_overwrite(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
763
+PARSER_RC pluginsd_overwrite(char **words __maybe_unused, size_t num_words __maybe_unused, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
764
{
705
- UNUSED(words);
706
-
765
RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
766
debug(D_PLUGINSD, "requested to OVERWRITE host labels");
767
@@ -719,9 +777,13 @@ PARSER_RC pluginsd_overwrite(char **words, void *user, PLUGINSD_ACTION *plugins
777
}
778
779
722
-PARSER_RC pluginsd_clabel(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
780
+PARSER_RC pluginsd_clabel(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
781
{
724
- if (!words[1] || !words[2] || !words[3]) {
782
+ const char *name = get_word(words, num_words, 1);
783
+ const char *value = get_word(words, num_words, 2);
784
+ const char *label_source = get_word(words, num_words, 3);
785
+
786
+ if (!name || !value || !*label_source) {
787
error("Ignoring malformed or empty CHART LABEL command.");
788
return PARSER_RC_OK;
789
}
@@ -731,15 +793,14 @@ PARSER_RC pluginsd_clabel(char **words, void *user, PLUGINSD_ACTION *plugins_ac
793
rrdlabels_unmark_all(((PARSER_USER_OBJECT *)user)->chart_rrdlabels_linked_temporarily);
794
}
795
734
- rrdlabels_add(((PARSER_USER_OBJECT *)user)->chart_rrdlabels_linked_temporarily, words[1], words[2], strtol(words[3], NULL, 10));
796
+ rrdlabels_add(((PARSER_USER_OBJECT *)user)->chart_rrdlabels_linked_temporarily,
797
+ name, value, str2l(label_source));
798
799
return PARSER_RC_OK;
800
}
801
739
-PARSER_RC pluginsd_clabel_commit(char **words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
802
+PARSER_RC pluginsd_clabel_commit(char **words __maybe_unused, size_t num_words __maybe_unused, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
803
{
741
- UNUSED(words);
742
-
804
RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
805
RRDSET *st = ((PARSER_USER_OBJECT *)user)->st;
806
@@ -762,9 +823,9 @@ PARSER_RC pluginsd_clabel_commit(char **words, void *user, PLUGINSD_ACTION *plu
823
return PARSER_RC_OK;
824
}
825
765
-PARSER_RC pluginsd_guid(char **words, void *user, PLUGINSD_ACTION *plugins_action)
826
+PARSER_RC pluginsd_guid(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action)
827
{
767
- char *uuid_str = words[1];
828
+ char *uuid_str = get_word(words, num_words, 1);
829
uuid_t uuid;
830
831
if (unlikely(!uuid_str)) {
@@ -784,9 +845,9 @@ PARSER_RC pluginsd_guid(char **words, void *user, PLUGINSD_ACTION *plugins_actio
845
return PARSER_RC_OK;
846
}
847
787
-PARSER_RC pluginsd_context(char **words, void *user, PLUGINSD_ACTION *plugins_action)
848
+PARSER_RC pluginsd_context(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action)
849
{
789
- char *uuid_str = words[1];
850
+ char *uuid_str = get_word(words, num_words, 1);
851
uuid_t uuid;
852
853
if (unlikely(!uuid_str)) {
@@ -806,9 +867,9 @@ PARSER_RC pluginsd_context(char **words, void *user, PLUGINSD_ACTION *plugins_ac
867
return PARSER_RC_OK;
868
}
869
809
-PARSER_RC pluginsd_tombstone(char **words, void *user, PLUGINSD_ACTION *plugins_action)
870
+PARSER_RC pluginsd_tombstone(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action)
871
{
811
- char *uuid_str = words[1];
872
+ char *uuid_str = get_word(words, num_words, 1);
873
uuid_t uuid;
874
875
if (unlikely(!uuid_str)) {
@@ -828,15 +889,15 @@ PARSER_RC pluginsd_tombstone(char **words, void *user, PLUGINSD_ACTION *plugins_
889
return PARSER_RC_OK;
890
}
891
831
-PARSER_RC metalog_pluginsd_host(char **words, void *user, PLUGINSD_ACTION *plugins_action)
892
+PARSER_RC metalog_pluginsd_host(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action)
893
{
833
- char *machine_guid = words[1];
834
- char *hostname = words[2];
835
- char *registry_hostname = words[3];
836
- char *update_every_s = words[4];
837
- char *os = words[5];
838
- char *timezone = words[6];
839
- char *tags = words[7];
894
+ char *machine_guid = get_word(words, num_words, 1);
895
+ char *hostname = get_word(words, num_words, 2);
896
+ char *registry_hostname = get_word(words, num_words, 3);
897
+ char *update_every_s = get_word(words, num_words, 4);
898
+ char *os = get_word(words, num_words, 5);
899
+ char *timezone = get_word(words, num_words, 6);
900
+ char *tags = get_word(words, num_words, 7);
901
902
int update_every = 1;
903
if (likely(update_every_s && *update_every_s))
@@ -855,6 +916,296 @@ PARSER_RC metalog_pluginsd_host(char **words, void *user, PLUGINSD_ACTION *plug
916
return PARSER_RC_OK;
917
}
918
919
+PARSER_RC pluginsd_replay_rrdset_begin(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
920
+{
921
+ char *id = get_word(words, num_words, 1);
922
+ char *start_time_str = get_word(words, num_words, 2);
923
+ char *end_time_str = get_word(words, num_words, 3);
924
+
925
+ RRDSET *st = ((PARSER_USER_OBJECT *) user)->st;
926
+ RRDHOST *host = ((PARSER_USER_OBJECT *)user)->host;
927
+
928
+ if (unlikely(!id || (!st && !*id))) {
929
+ error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_BEGIN " without a chart id for host '%s'. Disabling it.", rrdhost_hostname(host));
930
+ goto disable;
931
+ }
932
+
933
+ if(*id) {
934
+ st = rrdset_find(host, id);
935
+ if (unlikely(!st)) {
936
+ error("requested a " PLUGINSD_KEYWORD_REPLAY_BEGIN " on chart '%s', which does not exist on host '%s'. Disabling it.",
937
+ id, rrdhost_hostname(host));
938
+ goto disable;
939
+ }
940
+
941
+ ((PARSER_USER_OBJECT *) user)->st = st;
942
+ ((PARSER_USER_OBJECT *) user)->replay.start_time = 0;
943
+ ((PARSER_USER_OBJECT *) user)->replay.end_time = 0;
944
+ ((PARSER_USER_OBJECT *) user)->replay.start_time_ut = 0;
945
+ ((PARSER_USER_OBJECT *) user)->replay.end_time_ut = 0;
946
+ }
947
+
948
+ if(start_time_str && end_time_str) {
949
+ time_t start_time = strtol(start_time_str, NULL, 0);
950
+ time_t end_time = strtol(end_time_str, NULL, 0);
951
+
952
+ if(start_time && end_time) {
953
+ if (start_time > end_time) {
954
+ error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_BEGIN " on chart '%s' ('%s') on host '%s', but timings are invalid (%ld to %ld). Disabling it.",
955
+ rrdset_name(st), rrdset_id(st), rrdhost_hostname(st->rrdhost), start_time, end_time);
956
+ goto disable;
957
+ }
958
+
959
+ if (end_time - start_time != st->update_every)
960
+ rrdset_set_update_every(st, end_time - start_time);
961
+
962
+ st->last_collected_time.tv_sec = end_time;
963
+ st->last_collected_time.tv_usec = 0;
964
+
965
+ st->last_updated.tv_sec = end_time;
966
+ st->last_updated.tv_usec = 0;
967
+
968
+ ((PARSER_USER_OBJECT *) user)->replay.start_time = start_time;
969
+ ((PARSER_USER_OBJECT *) user)->replay.end_time = end_time;
970
+ ((PARSER_USER_OBJECT *) user)->replay.start_time_ut = (usec_t) start_time * USEC_PER_SEC;
971
+ ((PARSER_USER_OBJECT *) user)->replay.end_time_ut = (usec_t) end_time * USEC_PER_SEC;
972
+
973
+ st->counter++;
974
+ st->counter_done++;
975
+
976
+ // these are only needed for db mode RAM, SAVE, MAP, ALLOC
977
+ st->current_entry++;
978
+ if(st->current_entry >= st->entries)
979
+ st->current_entry -= st->entries;
980
+ }
981
+ }
982
+
983
+ return PARSER_RC_OK;
984
+
985
+disable:
986
+ ((PARSER_USER_OBJECT *)user)->enabled = 0;
987
+ return PARSER_RC_ERROR;
988
+}
989
+
990
+PARSER_RC pluginsd_replay_set(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
991
+{
992
+ char *dimension = get_word(words, num_words, 1);
993
+ char *value_str = get_word(words, num_words, 2);
994
+ char *flags_str = get_word(words, num_words, 3);
995
+
996
+ RRDSET *st = ((PARSER_USER_OBJECT *) user)->st;
997
+ RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
998
+
999
+ if (unlikely(!st)) {
1000
+ error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_SET " on dimension '%s' on host '%s', without a " PLUGINSD_KEYWORD_REPLAY_BEGIN ". Disabling it.",
1001
+ dimension, rrdhost_hostname(host));
1002
+ goto disable;
1003
+ }
1004
+
1005
+ if (unlikely(!dimension || !*dimension)) {
1006
+ error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_SET " on chart '%s' of host '%s', without a dimension. Disabling it.",
1007
+ rrdset_id(st), rrdhost_hostname(host));
1008
+ goto disable;
1009
+ }
1010
+
1011
+ if (unlikely(!((PARSER_USER_OBJECT *) user)->replay.start_time || !((PARSER_USER_OBJECT *) user)->replay.end_time)) {
1012
+ error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_SET " on dimension '%s' on host '%s', without timings from a " PLUGINSD_KEYWORD_REPLAY_BEGIN ". Disabling it.",
1013
+ dimension, rrdhost_hostname(host));
1014
+ goto disable;
1015
+ }
1016
+
1017
+ if (unlikely(!value_str || !*value_str))
1018
+ value_str = "nan";
1019
+
1020
+ if(unlikely(!flags_str))
1021
+ flags_str = "";
1022
+
1023
+ if (unlikely(rrdset_flag_check(st, RRDSET_FLAG_DEBUG)))
1024
+ debug(D_PLUGINSD, "REPLAY: is replaying dimension '%s'/'%s' to '%s'", rrdset_id(st), dimension, value_str);
1025
+
1026
+ if (likely(value_str)) {
1027
+ RRDDIM *rd = rrddim_find(st, dimension);
1028
+ if(unlikely(!rd)) {
1029
+ error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_SET " to dimension with id '%s' on chart '%s' ('%s') on host '%s', which does not exist. Disabling it.",
1030
+ dimension, rrdset_name(st), rrdset_id(st), rrdhost_hostname(st->rrdhost));
1031
+ goto disable;
1032
+ }
1033
+ else {
1034
+ NETDATA_DOUBLE value = strtondd(value_str, NULL);
1035
+ SN_FLAGS flags = SN_FLAG_NONE;
1036
+
1037
+ char c;
1038
+ while((c = *flags_str++)) {
1039
+ switch(c) {
1040
+ case 'R':
1041
+ flags |= SN_FLAG_RESET;
1042
+ break;
1043
+
1044
+ case 'E':
1045
+ flags |= SN_EMPTY_SLOT;
1046
+ value = NAN;
1047
+ break;
1048
+
1049
+ default:
1050
+ error("unknown flag '%c'", c);
1051
+ break;
1052
+ }
1053
+ }
1054
+
1055
+ if(!netdata_double_isnumber(value)) {
1056
+ value = NAN;
1057
+ flags = SN_EMPTY_SLOT;
1058
+ }
1059
+
1060
+ rrddim_store_metric(rd, ((PARSER_USER_OBJECT *) user)->replay.end_time_ut, value, flags);
1061
+ rd->last_collected_time.tv_sec = ((PARSER_USER_OBJECT *) user)->replay.end_time;
1062
+ rd->last_collected_time.tv_usec = 0;
1063
+ rd->collections_counter++;
1064
+ }
1065
+ }
1066
+ return PARSER_RC_OK;
1067
+
1068
+disable:
1069
+ ((PARSER_USER_OBJECT *) user)->enabled = 0;
1070
+ return PARSER_RC_ERROR;
1071
+}
1072
+
1073
+PARSER_RC pluginsd_replay_rrddim_collection_state(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
1074
+{
1075
+ char *dimension = get_word(words, num_words, 1);
1076
+ char *last_collected_ut_str = get_word(words, num_words, 2);
1077
+ char *last_collected_value_str = get_word(words, num_words, 3);
1078
+ char *last_calculated_value_str = get_word(words, num_words, 4);
1079
+ char *last_stored_value_str = get_word(words, num_words, 5);
1080
+
1081
+ RRDSET *st = ((PARSER_USER_OBJECT *) user)->st;
1082
+ RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
1083
+
1084
+ if (unlikely(!st)) {
1085
+ error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_RRDDIM_STATE " on dimension '%s' on host '%s', without a " PLUGINSD_KEYWORD_REPLAY_BEGIN ". Disabling it.",
1086
+ dimension, rrdhost_hostname(host));
1087
+ goto disable;
1088
+ }
1089
+
1090
+ if (unlikely(!dimension || !*dimension)) {
1091
+ error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_RRDDIM_STATE " on chart '%s' of host '%s', without a dimension. Disabling it.",
1092
+ rrdset_id(st), rrdhost_hostname(host));
1093
+ goto disable;
1094
+ }
1095
+
1096
+ RRDDIM *rd = rrddim_find(st, dimension);
1097
+ if(unlikely(!rd)) {
1098
+ error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_RRDDIM_STATE " to dimension with id '%s' on chart '%s' ('%s') on host '%s', which does not exist. Disabling it.",
1099
+ dimension, rrdset_name(st), rrdset_id(st), rrdhost_hostname(st->rrdhost));
1100
+ goto disable;
1101
+ }
1102
+
1103
+ usec_t dim_last_collected_ut = (usec_t)rd->last_collected_time.tv_sec * USEC_PER_SEC + (usec_t)rd->last_collected_time.tv_usec;
1104
+ usec_t last_collected_ut = last_collected_ut_str ? str2ull(last_collected_ut_str) : 0;
1105
+ if(last_collected_ut > dim_last_collected_ut) {
1106
+ rd->last_collected_time.tv_sec = last_collected_ut / USEC_PER_SEC;
1107
+ rd->last_collected_time.tv_usec = last_collected_ut % USEC_PER_SEC;
1108
+ }
1109
+
1110
+ rd->last_collected_value = last_collected_value_str ? str2ll(last_collected_value_str, NULL) : 0;
1111
+ rd->last_calculated_value = last_calculated_value_str ? str2ndd(last_calculated_value_str, NULL) : 0;
1112
+ rd->last_stored_value = last_stored_value_str ? str2ndd(last_stored_value_str, NULL) : 0.0;
1113
+ return PARSER_RC_OK;
1114
+
1115
+disable:
1116
+ ((PARSER_USER_OBJECT *) user)->enabled = 0;
1117
+ return PARSER_RC_ERROR;
1118
+}
1119
+
1120
+PARSER_RC pluginsd_replay_rrdset_collection_state(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
1121
+{
1122
+ char *last_collected_ut_str = get_word(words, num_words, 1);
1123
+ char *last_updated_ut_str = get_word(words, num_words, 2);
1124
+ char *last_collected_total_str = get_word(words, num_words, 3);
1125
+ char *collected_total_str = get_word(words, num_words, 4);
1126
+
1127
+ RRDSET *st = ((PARSER_USER_OBJECT *) user)->st;
1128
+ RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
1129
+
1130
+ if (unlikely(!st)) {
1131
+ error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_RRDSET_STATE " on host '%s', without a " PLUGINSD_KEYWORD_REPLAY_BEGIN ". Disabling it.",
1132
+ rrdhost_hostname(host));
1133
+ goto disable;
1134
+ }
1135
+
1136
+ usec_t chart_last_collected_ut = (usec_t)st->last_collected_time.tv_sec * USEC_PER_SEC + (usec_t)st->last_collected_time.tv_usec;
1137
+ usec_t last_collected_ut = last_collected_ut_str ? str2ull(last_collected_ut_str) : 0;
1138
+ if(last_collected_ut > chart_last_collected_ut) {
1139
+ st->last_collected_time.tv_sec = last_collected_ut / USEC_PER_SEC;
1140
+ st->last_collected_time.tv_usec = last_collected_ut % USEC_PER_SEC;
1141
+ }
1142
+
1143
+ usec_t chart_last_updated_ut = (usec_t)st->last_updated.tv_sec * USEC_PER_SEC + (usec_t)st->last_updated.tv_usec;
1144
+ usec_t last_updated_ut = last_updated_ut_str ? str2ull(last_updated_ut_str) : 0;
1145
+ if(last_updated_ut > chart_last_updated_ut) {
1146
+ st->last_updated.tv_sec = last_updated_ut / USEC_PER_SEC;
1147
+ st->last_updated.tv_usec = last_updated_ut % USEC_PER_SEC;
1148
+ }
1149
+
1150
+ st->last_collected_total = last_collected_total_str ? strtoll(last_collected_total_str, NULL, 0) : 0;
1151
+ st->collected_total = collected_total_str ? strtoll(collected_total_str, NULL, 0) : 0;
1152
+
1153
+ st->counter++;
1154
+ st->counter_done++;
1155
+
1156
+ return PARSER_RC_OK;
1157
+
1158
+disable:
1159
+ ((PARSER_USER_OBJECT *) user)->enabled = 0;
1160
+ return PARSER_RC_ERROR;
1161
+}
1162
+
1163
+PARSER_RC pluginsd_replay_end(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action __maybe_unused)
1164
+{
1165
+ if (num_words < 7) {
1166
+ error("REPLAY: malformed " PLUGINSD_KEYWORD_REPLAY_END " command");
1167
+ return PARSER_RC_ERROR;
1168
+ }
1169
+
1170
+ time_t update_every_child = str2l(get_word(words, num_words, 1));
1171
+ time_t first_entry_child = str2l(get_word(words, num_words, 2));
1172
+ time_t last_entry_child = str2l(get_word(words, num_words, 3));
1173
+
1174
+ bool start_streaming = (strcmp(get_word(words, num_words, 4), "true") == 0);
1175
+ time_t first_entry_requested = str2l(get_word(words, num_words, 5));
1176
+ time_t last_entry_requested = str2l(get_word(words, num_words, 6));
1177
+
1178
+ PARSER_USER_OBJECT *user_object = user;
1179
+
1180
+ RRDSET *st = ((PARSER_USER_OBJECT *) user)->st;
1181
+ RRDHOST *host = ((PARSER_USER_OBJECT *) user)->host;
1182
+
1183
+ if (unlikely(!st)) {
1184
+ error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_END " on host '%s', without a " PLUGINSD_KEYWORD_REPLAY_BEGIN ". Disabling it.",
1185
+ rrdhost_hostname(host));
1186
+ return PARSER_RC_ERROR;
1187
+ }
1188
+
1189
+ ((PARSER_USER_OBJECT *) user)->st = NULL;
1190
+ ((PARSER_USER_OBJECT *) user)->count++;
1191
+
1192
+ st->counter++;
1193
+ st->counter_done++;
1194
+
1195
+ if (start_streaming) {
1196
+ if (st->update_every != update_every_child)
1197
+ rrdset_set_update_every(st, update_every_child);
1198
+
1199
+ rrdset_flag_set(st, RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
1200
+ rrdset_flag_clear(st, RRDSET_FLAG_SYNC_CLOCK);
1201
+ return PARSER_RC_OK;
1202
+ }
1203
+
1204
+ bool ok = replicate_chart_request(send_to_plugin, user_object->parser, host, st, first_entry_child, last_entry_child,
1205
+ first_entry_requested, last_entry_requested);
1206
+ return ok ? PARSER_RC_OK : PARSER_RC_ERROR;
1207
+}
1208
+
1209
static void pluginsd_process_thread_cleanup(void *ptr) {
1210
PARSER *parser = (PARSER *)ptr;
1211
rrd_collector_finished();
@@ -895,7 +1246,7 @@ inline size_t pluginsd_process(RRDHOST *host, struct plugind *cd, FILE *fp_plugi
1246
};
1247
1248
// fp_plugin_output = our input; fp_plugin_input = our output
898
- PARSER *parser = parser_init(host, &user, fp_plugin_output, fp_plugin_input, PARSER_INPUT_SPLIT);
1249
+ PARSER *parser = parser_init(host, &user, fp_plugin_output, fp_plugin_input, PARSER_INPUT_SPLIT, NULL);
1250
1251
rrd_collector_started();
1252
collectors/plugins.d/pluginsd_parser.h
+11
-5
@@ -5,7 +5,6 @@
5
6
#include "parser/parser.h"
7
8
-
8
typedef struct parser_user_object {
9
PARSER *parser;
10
RRDSET *st;
@@ -20,10 +19,17 @@ typedef struct parser_user_object {
19
uint8_t st_exists;
20
uint8_t host_exists;
21
void *private; // the user can set this for private use
22
+
23
+ struct {
24
+ time_t start_time;
25
+ time_t end_time;
26
+
27
+ usec_t start_time_ut;
28
+ usec_t end_time_ut;
29
+ } replay;
30
} PARSER_USER_OBJECT;
31
25
-PARSER_RC pluginsd_function(char **words, void *user, PLUGINSD_ACTION *plugins_action);
26
-PARSER_RC pluginsd_function_result_begin(char **words, void *user, PLUGINSD_ACTION *plugins_action);
32
+PARSER_RC pluginsd_function(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
33
+PARSER_RC pluginsd_function_result_begin(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
34
void inflight_functions_init(PARSER *parser);
28
-
29
-#endif //NETDATA_PLUGINSD_PARSER_H
\ No newline at end of file
35
+#endif //NETDATA_PLUGINSD_PARSER_H
collectors/statsd.plugin/statsd.c
+9
-9
@@ -1469,23 +1469,23 @@ static int statsd_readfile(const char *filename, STATSD_APP *app, STATSD_APP_CHA
1469
}
1470
else if (!strcmp(name, "dimension")) {
1471
// metric [name [type [multiplier [divisor]]]]
1472
- char *words[10];
1473
- pluginsd_split_words(value, words, 10, NULL, NULL, 0);
1472
+ char *words[10] = { NULL };
1473
+ size_t num_words = pluginsd_split_words(value, words, 10, NULL, NULL, 0);
1474
1475
int pattern = 0;
1476
size_t i = 0;
1477
- char *metric_name = words[i++];
1477
+ char *metric_name = get_word(words, num_words, i++);
1478
1479
if(strcmp(metric_name, "pattern") == 0) {
1480
- metric_name = words[i++];
1480
+ metric_name = get_word(words, num_words, i++);
1481
pattern = 1;
1482
}
1483
1484
- char *dim_name = words[i++];
1485
- char *type = words[i++];
1486
- char *multiplier = words[i++];
1487
- char *divisor = words[i++];
1488
- char *opts = words[i++];
1484
+ char *dim_name = get_word(words, num_words, i++);
1485
+ char *type = get_word(words, num_words, i++);
1486
+ char *multiplier = get_word(words, num_words, i++);
1487
+ char *divisor = get_word(words, num_words, i++);
1488
+ char *opts = get_word(words, num_words, i++);
1489
1490
RRDDIM_FLAGS flags = RRDDIM_FLAG_NONE;
1491
RRDDIM_OPTIONS options = RRDDIM_OPTION_NONE;
collectors/tc.plugin/plugin_tc.c
-2
@@ -805,8 +805,6 @@ static inline struct tc_class *tc_class_add(struct tc_device *n, char *id, bool
805
// tc_device_index_del(d);
806
//}
807
808
-#define PLUGINSD_MAX_WORDS 20
809
-
808
static inline int tc_space(char c) {
809
switch(c) {
810
case ' ':
daemon/static_threads_macos.c
+6
-4
@@ -12,18 +12,20 @@ const struct netdata_static_thread static_threads_macos[] = {
12
.enabled = 1,
13
.thread = NULL,
14
.init_routine = NULL,
15
- .start_routine = macos_main
15
+ .start_routine = macos_main,
16
+ .env_name = NULL,
17
+ .global_variable = NULL,
18
},
19
18
- {NULL, NULL, NULL, 0, NULL, NULL, NULL}
20
+ {NULL, NULL, NULL, 0, NULL, NULL, NULL, NULL, NULL}
21
};
22
23
const struct netdata_static_thread static_threads_freebsd[] = {
22
- {NULL, NULL, NULL, 0, NULL, NULL, NULL}
24
+ {NULL, NULL, NULL, 0, NULL, NULL, NULL, NULL, NULL}
25
};
26
27
const struct netdata_static_thread static_threads_linux[] = {
26
- {NULL, NULL, NULL, 0, NULL, NULL, NULL}
28
+ {NULL, NULL, NULL, 0, NULL, NULL, NULL, NULL, NULL}
29
};
30
31
struct netdata_static_thread *static_threads_get() {
daemon/unit_test.c
+3
@@ -1739,6 +1739,9 @@ static RRDHOST *dbengine_rrdhost_find_or_create(char *name)
1739
, default_rrdpush_destination
1740
, default_rrdpush_api_key
1741
, default_rrdpush_send_charts_matching
1742
+ , default_rrdpush_enable_replication
1743
+ , default_rrdpush_seconds_to_replicate
1744
+ , default_rrdpush_replication_step
1745
, NULL
1746
, 0
1747
);
database/rrd.h
+18
-3
@@ -523,13 +523,15 @@ typedef enum rrdset_flags {
523
// least rrdset_free_obsolete_time seconds ago.
524
RRDSET_FLAG_ARCHIVED = (1 << 15),
525
RRDSET_FLAG_METADATA_UPDATE = (1 << 16), // Mark that metadata needs to be stored
526
- RRDSET_FLAG_PENDING_FOREACH_ALARMS = (1 << 17), // contains dims with uninitialized foreach alarms
526
RRDSET_FLAG_ANOMALY_DETECTION = (1 << 18), // flag to identify anomaly detection charts.
527
RRDSET_FLAG_INDEXED_ID = (1 << 19), // the rrdset is indexed by its id
528
RRDSET_FLAG_INDEXED_NAME = (1 << 20), // the rrdset is indexed by its name
529
530
RRDSET_FLAG_ANOMALY_RATE_CHART = (1 << 21), // the rrdset is for storing anomaly rates for all dimensions
531
RRDSET_FLAG_PENDING_HEALTH_INITIALIZATION = (1 << 22),
532
+
533
+ RRDSET_FLAG_SENDER_REPLICATION_FINISHED = (1 << 23), // the sending side has completed replication
534
+ RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED = (1 << 24), // the receiving side has completed replication
535
} RRDSET_FLAGS;
536
537
#define rrdset_flag_check(st, flag) (__atomic_load_n(&((st)->flags), __ATOMIC_SEQ_CST) & (flag))
@@ -935,6 +937,10 @@ struct rrdhost {
937
struct rrdpush_destinations *destination; // the current destination from the above list
938
SIMPLE_PATTERN *rrdpush_send_charts_matching; // pattern to match the charts to be sent
939
940
+ bool rrdpush_enable_replication; // enable replication
941
+ time_t rrdpush_seconds_to_replicate; // max time we want to replicate from the child
942
+ time_t rrdpush_replication_step; // seconds per replication step
943
+
944
// the following are state information for the threading
945
// streaming metrics from this netdata to an upstream netdata
946
struct sender_state *sender;
@@ -1098,6 +1104,9 @@ RRDHOST *rrdhost_find_or_create(
1104
, char *rrdpush_destination
1105
, char *rrdpush_api_key
1106
, char *rrdpush_send_charts_matching
1107
+ , bool rrdpush_enable_replication
1108
+ , time_t rrdpush_seconds_to_replicate
1109
+ , time_t rrdpush_replication_step
1110
, struct rrdhost_system_info *system_info
1111
, bool is_archived
1112
);
@@ -1121,6 +1130,9 @@ void rrdhost_update(RRDHOST *host
1130
, char *rrdpush_destination
1131
, char *rrdpush_api_key
1132
, char *rrdpush_send_charts_matching
1133
+ , bool rrdpush_enable_replication
1134
+ , time_t rrdpush_seconds_to_replicate
1135
+ , time_t rrdpush_replication_step
1136
, struct rrdhost_system_info *system_info
1137
);
1138
@@ -1190,6 +1202,8 @@ int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, time_t now
1202
1203
void rrdset_update_heterogeneous_flag(RRDSET *st);
1204
1205
+time_t rrdset_set_update_every(RRDSET *st, time_t update_every);
1206
+
1207
RRDSET *rrdset_find(RRDHOST *host, const char *id);
1208
#define rrdset_find_localhost(id) rrdset_find(localhost, id)
1209
/* This will not return charts that are archived */
@@ -1310,8 +1324,9 @@ RRDHOST *rrdhost_create(
1324
const char *hostname, const char *registry_hostname, const char *guid, const char *os, const char *timezone,
1325
const char *abbrev_timezone, int32_t utc_offset,const char *tags, const char *program_name, const char *program_version,
1326
int update_every, long entries, RRD_MEMORY_MODE memory_mode, unsigned int health_enabled, unsigned int rrdpush_enabled,
1313
- char *rrdpush_destination, char *rrdpush_api_key, char *rrdpush_send_charts_matching, struct rrdhost_system_info *system_info,
1314
- int is_localhost, bool is_archived);
1327
+ char *rrdpush_destination, char *rrdpush_api_key, char *rrdpush_send_charts_matching,
1328
+ bool rrdpush_enable_replication, time_t rrdpush_seconds_to_replicate, time_t rrdpush_replication_step,
1329
+ struct rrdhost_system_info *system_info, int is_localhost, bool is_archived);
1330
1331
#endif /* NETDATA_RRD_INTERNALS */
1332
database/rrdcontext.c
+2
-2
@@ -2417,7 +2417,7 @@ static void query_target_add_metric(QUERY_TARGET_LOCALS *qtl, RRDMETRIC_ACQUIRED
2417
2418
if(!common_first_time_t)
2419
common_first_time_t = tier_retention[tier].db_first_time_t;
2420
- else
2420
+ else if(tier_retention[tier].db_first_time_t)
2421
common_first_time_t = MIN(common_first_time_t, tier_retention[tier].db_first_time_t);
2422
2423
if(!common_last_time_t)
@@ -2427,7 +2427,7 @@ static void query_target_add_metric(QUERY_TARGET_LOCALS *qtl, RRDMETRIC_ACQUIRED
2427
2428
if(!common_update_every)
2429
common_update_every = tier_retention[tier].db_update_every;
2430
- else
2430
+ else if(tier_retention[tier].db_update_every)
2431
common_update_every = MIN(common_update_every, tier_retention[tier].db_update_every);
2432
2433
tiers_added++;
database/rrdhost.c
+45
-6
@@ -248,6 +248,9 @@ RRDHOST *rrdhost_create(const char *hostname,
248
char *rrdpush_destination,
249
char *rrdpush_api_key,
250
char *rrdpush_send_charts_matching,
251
+ bool rrdpush_enable_replication,
252
+ time_t rrdpush_seconds_to_replicate,
253
+ time_t rrdpush_replication_step,
254
struct rrdhost_system_info *system_info,
255
int is_localhost,
256
bool archived
@@ -287,6 +290,24 @@ int is_legacy = 1;
290
host, rrdpush_enabled, rrdpush_destination, rrdpush_api_key, rrdpush_send_charts_matching);
291
}
292
293
+ host->rrdpush_enable_replication = rrdpush_enable_replication;
294
+ host->rrdpush_seconds_to_replicate = rrdpush_seconds_to_replicate;
295
+ host->rrdpush_replication_step = rrdpush_replication_step;
296
+
297
+ switch(memory_mode) {
298
+ default:
299
+ case RRD_MEMORY_MODE_ALLOC:
300
+ case RRD_MEMORY_MODE_MAP:
301
+ case RRD_MEMORY_MODE_SAVE:
302
+ case RRD_MEMORY_MODE_RAM:
303
+ if(host->rrdpush_seconds_to_replicate > host->rrd_history_entries * host->rrd_update_every)
304
+ host->rrdpush_seconds_to_replicate = host->rrd_history_entries * host->rrd_update_every;
305
+ break;
306
+
307
+ case RRD_MEMORY_MODE_DBENGINE:
308
+ break;
309
+ }
310
+
311
netdata_rwlock_init(&host->rrdhost_rwlock);
312
netdata_mutex_init(&host->aclk_state_lock);
313
netdata_mutex_init(&host->receiver_lock);
@@ -520,13 +541,15 @@ void rrdhost_update(RRDHOST *host
541
, char *rrdpush_destination
542
, char *rrdpush_api_key
543
, char *rrdpush_send_charts_matching
544
+ , bool rrdpush_enable_replication
545
+ , time_t rrdpush_seconds_to_replicate
546
+ , time_t rrdpush_replication_step
547
, struct rrdhost_system_info *system_info
548
)
549
{
550
UNUSED(guid);
551
552
host->health_enabled = (mode == RRD_MEMORY_MODE_NONE) ? 0 : health_enabled;
529
- //host->stream_version = STREAMING_PROTOCOL_CURRENT_VERSION; Unused?
553
554
rrdhost_system_info_free(host->system_info);
555
host->system_info = system_info;
@@ -560,12 +583,12 @@ void rrdhost_update(RRDHOST *host
583
if(host->rrd_update_every != update_every)
584
error("Host '%s' has an update frequency of %d seconds, but the wanted one is %d seconds. Restart netdata here to apply the new settings.", rrdhost_hostname(host), host->rrd_update_every, update_every);
585
563
- if(host->rrd_history_entries < history)
564
- error("Host '%s' has history of %ld entries, but the wanted one is %ld entries. Restart netdata here to apply the new settings.", rrdhost_hostname(host), host->rrd_history_entries, history);
565
-
586
if(host->rrd_memory_mode != mode)
587
error("Host '%s' has memory mode '%s', but the wanted one is '%s'. Restart netdata here to apply the new settings.", rrdhost_hostname(host), rrd_memory_mode_name(host->rrd_memory_mode), rrd_memory_mode_name(mode));
588
589
+ else if(host->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE && host->rrd_history_entries < history)
590
+ error("Host '%s' has history of %ld entries, but the wanted one is %ld entries. Restart netdata here to apply the new settings.", rrdhost_hostname(host), host->rrd_history_entries, history);
591
+
592
// update host tags
593
rrdhost_init_tags(host, tags);
594
@@ -593,6 +616,11 @@ void rrdhost_update(RRDHOST *host
616
rrdcalctemplate_index_init(host);
617
rrdcalc_rrdhost_index_init(host);
618
619
+ host->rrdpush_enable_replication = rrdpush_enable_replication;
620
+ host->rrdpush_seconds_to_replicate = rrdpush_seconds_to_replicate;
621
+ host->rrdpush_replication_step = rrdpush_replication_step;
622
+
623
+
624
rrd_hosts_available++;
625
ml_new_host(host);
626
rrdhost_load_rrdcontext_data(host);
@@ -601,8 +629,6 @@ void rrdhost_update(RRDHOST *host
629
630
if (health_enabled)
631
health_thread_spawn(host);
604
-
605
- return;
632
}
633
634
RRDHOST *rrdhost_find_or_create(
@@ -624,6 +650,9 @@ RRDHOST *rrdhost_find_or_create(
650
, char *rrdpush_destination
651
, char *rrdpush_api_key
652
, char *rrdpush_send_charts_matching
653
+ , bool rrdpush_enable_replication
654
+ , time_t rrdpush_seconds_to_replicate
655
+ , time_t rrdpush_replication_step
656
, struct rrdhost_system_info *system_info
657
, bool archived
658
) {
@@ -658,6 +687,9 @@ RRDHOST *rrdhost_find_or_create(
687
, rrdpush_destination
688
, rrdpush_api_key
689
, rrdpush_send_charts_matching
690
+ , rrdpush_enable_replication
691
+ , rrdpush_seconds_to_replicate
692
+ , rrdpush_replication_step
693
, system_info
694
, 0
695
, archived
@@ -683,6 +715,9 @@ RRDHOST *rrdhost_find_or_create(
715
, rrdpush_destination
716
, rrdpush_api_key
717
, rrdpush_send_charts_matching
718
+ , rrdpush_enable_replication
719
+ , rrdpush_seconds_to_replicate
720
+ , rrdpush_replication_step
721
, system_info);
722
}
723
if (host) {
@@ -894,6 +929,9 @@ unittest:
929
, default_rrdpush_destination
930
, default_rrdpush_api_key
931
, default_rrdpush_send_charts_matching
932
+ , default_rrdpush_enable_replication
933
+ , default_rrdpush_seconds_to_replicate
934
+ , default_rrdpush_replication_step
935
, system_info
936
, 1
937
, 0
@@ -1003,6 +1041,7 @@ void stop_streaming_sender(RRDHOST *host)
1041
if (host->sender->compressor)
1042
host->sender->compressor->destroy(&host->sender->compressor);
1043
#endif
1044
+ dictionary_destroy(host->sender->replication_requests);
1045
freez(host->sender);
1046
host->sender = NULL;
1047
}
database/rrdset.c
+99
-108
@@ -141,6 +141,10 @@ static void rrdset_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, v
141
st->rrdhost = host;
142
143
st->flags = RRDSET_FLAG_SYNC_CLOCK | RRDSET_FLAG_INDEXED_ID;
144
+
145
+ if(host == localhost || !host->receiver || !stream_has_capability(host->receiver, STREAM_CAP_REPLICATION))
146
+ st->flags |= RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED;
147
+
148
if(unlikely(st->id == anomaly_rates_chart))
149
st->flags |= RRDSET_FLAG_ANOMALY_RATE_CHART;
150
@@ -285,18 +289,7 @@ static bool rrdset_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused,
289
}
290
291
if (unlikely(st->update_every != ctr->update_every)) {
288
- st->update_every = ctr->update_every;
289
-
290
- // switch update every to the storage engine
291
- RRDDIM *rd;
292
- rrddim_foreach_read(rd, st) {
293
- for (size_t tier = 0; tier < storage_tiers; tier++) {
294
- if (rd->tiers[tier] && rd->tiers[tier]->db_collection_handle)
295
- rd->tiers[tier]->collect_ops->change_collection_frequency(rd->tiers[tier]->db_collection_handle, (int)(st->rrdhost->db[tier].tier_grouping * st->update_every));
296
- }
297
- }
298
- rrddim_foreach_done(rd);
299
-
292
+ rrdset_set_update_every(st, ctr->update_every);
293
ctr->react_action |= RRDSET_REACT_UPDATED;
294
}
295
@@ -876,7 +869,13 @@ void rrdset_timed_next(RRDSET *st, struct timeval now, usec_t duration_since_las
869
// oops! the database is in the future
870
#ifdef NETDATA_INTERNAL_CHECKS
871
info("RRD database for chart '%s' on host '%s' is %0.5" NETDATA_DOUBLE_MODIFIER
879
- " secs in the future (counter #%zu, update #%zu). Adjusting it to current time.", rrdset_id(st), rrdhost_hostname(st->rrdhost), (NETDATA_DOUBLE)-since_last_usec / USEC_PER_SEC, st->counter, st->counter_done);
872
+ " secs in the future (counter #%zu, update #%zu). Adjusting it to current time."
873
+ , rrdset_id(st)
874
+ , rrdhost_hostname(st->rrdhost)
875
+ , (NETDATA_DOUBLE)-since_last_usec / USEC_PER_SEC
876
+ , st->counter
877
+ , st->counter_done
878
+ );
879
#endif
880
881
st->last_collected_time.tv_sec = now.tv_sec - st->update_every;
@@ -930,14 +929,18 @@ void rrdset_timed_next(RRDSET *st, struct timeval now, usec_t duration_since_las
929
#endif
930
}
931
933
- #ifdef NETDATA_INTERNAL_CHECKS
932
debug(D_RRD_CALLS, "rrdset_timed_next() for chart %s with duration since last update %llu usec", rrdset_name(st), duration_since_last_update);
933
rrdset_debug(st, "NEXT: %llu microseconds", duration_since_last_update);
934
937
- if(discarded && discarded != duration_since_last_update)
938
- info("host '%s', chart '%s': discarded data collection time of %llu usec, replaced with %llu usec, reason: '%s'", rrdhost_hostname(st->rrdhost), rrdset_id(st), discarded, duration_since_last_update, discard_reason?discard_reason:"UNDEFINED");
939
-
940
- #endif
935
+ internal_error(discarded && discarded != duration_since_last_update,
936
+ "host '%s', chart '%s': discarded data collection time of %llu usec, "
937
+ "replaced with %llu usec, reason: '%s'"
938
+ , rrdhost_hostname(st->rrdhost)
939
+ , rrdset_id(st)
940
+ , discarded
941
+ , duration_since_last_update
942
+ , discard_reason?discard_reason:"UNDEFINED"
943
+ );
944
945
st->usec_since_last_update = duration_since_last_update;
946
}
@@ -968,9 +971,7 @@ static inline usec_t rrdset_init_last_collected_time(RRDSET *st, struct timeval
971
972
usec_t last_collect_ut = st->last_collected_time.tv_sec * USEC_PER_SEC + st->last_collected_time.tv_usec;
973
971
- #ifdef NETDATA_INTERNAL_CHECKS
974
rrdset_debug(st, "initialized last collected time to %0.3" NETDATA_DOUBLE_MODIFIER, (NETDATA_DOUBLE)last_collect_ut / USEC_PER_SEC);
973
- #endif
975
976
return last_collect_ut;
977
}
@@ -981,9 +982,7 @@ static inline usec_t rrdset_update_last_collected_time(RRDSET *st) {
982
st->last_collected_time.tv_sec = (time_t) (ut / USEC_PER_SEC);
983
st->last_collected_time.tv_usec = (suseconds_t) (ut % USEC_PER_SEC);
984
984
- #ifdef NETDATA_INTERNAL_CHECKS
985
rrdset_debug(st, "updated last collected time to %0.3" NETDATA_DOUBLE_MODIFIER, (NETDATA_DOUBLE)last_collect_ut / USEC_PER_SEC);
986
- #endif
986
987
return last_collect_ut;
988
}
@@ -1000,9 +999,7 @@ static inline usec_t rrdset_init_last_updated_time(RRDSET *st) {
999
1000
usec_t last_updated_ut = st->last_updated.tv_sec * USEC_PER_SEC + st->last_updated.tv_usec;
1001
1003
- #ifdef NETDATA_INTERNAL_CHECKS
1002
rrdset_debug(st, "initialized last updated time to %0.3" NETDATA_DOUBLE_MODIFIER, (NETDATA_DOUBLE)last_updated_ut / USEC_PER_SEC);
1005
- #endif
1003
1004
return last_updated_ut;
1005
}
@@ -1165,17 +1162,24 @@ static inline size_t rrdset_done_interpolate(
1162
1163
for( ; next_store_ut <= now_collect_ut ; last_collect_ut = next_store_ut, next_store_ut += update_every_ut, iterations-- ) {
1164
1168
- #ifdef NETDATA_INTERNAL_CHECKS
1169
- if(iterations < 0) { error("INTERNAL CHECK: %s: iterations calculation wrapped! first_ut = %llu, last_stored_ut = %llu, next_store_ut = %llu, now_collect_ut = %llu", rrdset_name(st), first_ut, last_stored_ut, next_store_ut, now_collect_ut); }
1165
+ internal_error(iterations < 0,
1166
+ "RRDSET: '%s': iterations calculation wrapped! "
1167
+ "first_ut = %llu, last_stored_ut = %llu, next_store_ut = %llu, now_collect_ut = %llu"
1168
+ , rrdset_id(st)
1169
+ , first_ut
1170
+ , last_stored_ut
1171
+ , next_store_ut
1172
+ , now_collect_ut
1173
+ );
1174
+
1175
rrdset_debug(st, "last_stored_ut = %0.3" NETDATA_DOUBLE_MODIFIER " (last updated time)", (NETDATA_DOUBLE)last_stored_ut/USEC_PER_SEC);
1176
rrdset_debug(st, "next_store_ut = %0.3" NETDATA_DOUBLE_MODIFIER " (next interpolation point)", (NETDATA_DOUBLE)next_store_ut/USEC_PER_SEC);
1172
- #endif
1177
1178
last_ut = next_store_ut;
1179
1180
struct rda_item *rda;
1181
size_t dim_id;
1178
- for(dim_id = 0, rda = rda_base, rd = rda->rd ; dim_id < rda_slots ; ++dim_id, ++rda) {
1182
+ for(dim_id = 0, rda = rda_base ; dim_id < rda_slots ; ++dim_id, ++rda) {
1183
rd = rda->rd;
1184
if(unlikely(!rd)) continue;
1185
@@ -1189,7 +1193,6 @@ static inline size_t rrdset_done_interpolate(
1193
/ (NETDATA_DOUBLE)(now_collect_ut - last_collect_ut)
1194
);
1195
1192
- #ifdef NETDATA_INTERNAL_CHECKS
1196
rrdset_debug(st, "%s: CALC2 INC " NETDATA_DOUBLE_FORMAT " = "
1197
NETDATA_DOUBLE_FORMAT
1198
" * (%llu - %llu)"
@@ -1200,7 +1203,6 @@ static inline size_t rrdset_done_interpolate(
1203
, next_store_ut, last_collect_ut
1204
, now_collect_ut, last_collect_ut
1205
);
1203
- #endif
1206
1207
rd->calculated_value -= new_value;
1208
new_value += rd->last_calculated_value;
@@ -1209,12 +1211,10 @@ static inline size_t rrdset_done_interpolate(
1211
1212
if(unlikely(next_store_ut - last_stored_ut < update_every_ut)) {
1213
1212
- #ifdef NETDATA_INTERNAL_CHECKS
1214
rrdset_debug(st, "%s: COLLECTION POINT IS SHORT " NETDATA_DOUBLE_FORMAT " - EXTRAPOLATING",
1215
rrddim_name(rd)
1216
, (NETDATA_DOUBLE)(next_store_ut - last_stored_ut)
1217
);
1217
- #endif
1218
1219
new_value = new_value * (NETDATA_DOUBLE)(st->update_every * USEC_PER_SEC) / (NETDATA_DOUBLE)(next_store_ut - last_stored_ut);
1220
}
@@ -1243,7 +1243,6 @@ static inline size_t rrdset_done_interpolate(
1243
+ rd->last_calculated_value
1244
);
1245
1246
- #ifdef NETDATA_INTERNAL_CHECKS
1246
rrdset_debug(st, "%s: CALC2 DEF " NETDATA_DOUBLE_FORMAT " = ((("
1247
"(" NETDATA_DOUBLE_FORMAT " - " NETDATA_DOUBLE_FORMAT ")"
1248
" * %llu"
@@ -1253,7 +1252,6 @@ static inline size_t rrdset_done_interpolate(
1252
, (next_store_ut - first_ut)
1253
, (now_collect_ut - first_ut), rd->last_calculated_value
1254
);
1256
- #endif
1255
}
1256
break;
1257
}
@@ -1278,9 +1276,7 @@ static inline size_t rrdset_done_interpolate(
1276
else {
1277
(void) ml_is_anomalous(rd, 0, false);
1278
1281
- #ifdef NETDATA_INTERNAL_CHECKS
1279
rrdset_debug(st, "%s: STORE[%ld] = NON EXISTING ", rrddim_name(rd), current_entry);
1283
- #endif
1280
1281
rrddim_store_metric(rd, next_store_ut, NAN, SN_FLAG_NONE);
1282
rd->last_stored_value = NAN;
@@ -1328,9 +1324,7 @@ static inline void rrdset_done_fill_the_gap(RRDSET *st) {
1324
rd->db[current_entry] = pack_storage_number(NAN, SN_FLAG_NONE);
1325
current_entry = ((current_entry + 1) >= entries) ? 0 : current_entry + 1;
1326
1331
- #ifdef NETDATA_INTERNAL_CHECKS
1327
rrdset_debug(st, "%s: STORE[%ld] = NON EXISTING (FILLED THE GAP)", rrddim_name(rd), current_entry);
1333
- #endif
1328
}
1329
}
1330
rrddim_foreach_done(rd);
@@ -1356,7 +1350,7 @@ void rrdset_done(RRDSET *st) {
1350
void rrdset_timed_done(RRDSET *st, struct timeval now) {
1351
if(unlikely(netdata_exit)) return;
1352
1359
- debug(D_RRD_CALLS, "rrdset_done() for chart %s", rrdset_name(st));
1353
+ debug(D_RRD_CALLS, "rrdset_done() for chart '%s'", rrdset_name(st));
1354
1355
RRDDIM *rd;
1356
@@ -1381,17 +1375,15 @@ void rrdset_timed_done(RRDSET *st, struct timeval now) {
1375
// check if the chart has a long time to be updated
1376
if(unlikely(st->usec_since_last_update > st->entries * update_every_ut &&
1377
st->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE && st->rrd_memory_mode != RRD_MEMORY_MODE_NONE)) {
1384
- info("host '%s', chart %s: took too long to be updated (counter #%zu, update #%zu, %0.3" NETDATA_DOUBLE_MODIFIER
1385
- " secs). Resetting it.", rrdhost_hostname(st->rrdhost), rrdset_name(st), st->counter, st->counter_done, (NETDATA_DOUBLE)st->usec_since_last_update / USEC_PER_SEC);
1378
+ info("host '%s', chart '%s': took too long to be updated (counter #%zu, update #%zu, %0.3" NETDATA_DOUBLE_MODIFIER
1379
+ " secs). Resetting it.", rrdhost_hostname(st->rrdhost), rrdset_id(st), st->counter, st->counter_done, (NETDATA_DOUBLE)st->usec_since_last_update / USEC_PER_SEC);
1380
rrdset_reset(st);
1381
st->usec_since_last_update = update_every_ut;
1382
store_this_entry = 0;
1383
first_entry = 1;
1384
}
1385
1392
- #ifdef NETDATA_INTERNAL_CHECKS
1386
rrdset_debug(st, "microseconds since last update: %llu", st->usec_since_last_update);
1394
- #endif
1387
1388
// set last_collected_time
1389
if(unlikely(!st->last_collected_time.tv_sec)) {
@@ -1428,9 +1420,9 @@ void rrdset_timed_done(RRDSET *st, struct timeval now) {
1420
if(unlikely(dt_usec(&st->last_collected_time, &st->last_updated) > st->entries * update_every_ut &&
1421
st->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE)) {
1422
info(
1431
- "%s: too old data (last updated at %"PRId64".%"PRId64", last collected at %"PRId64".%"PRId64"). "
1423
+ "'%s': too old data (last updated at %"PRId64".%"PRId64", last collected at %"PRId64".%"PRId64"). "
1424
"Resetting it. Will not store the next entry.",
1433
- rrdset_name(st),
1425
+ rrdset_id(st),
1426
(int64_t)st->last_updated.tv_sec,
1427
(int64_t)st->last_updated.tv_usec,
1428
(int64_t)st->last_collected_time.tv_sec,
@@ -1450,9 +1442,9 @@ void rrdset_timed_done(RRDSET *st, struct timeval now) {
1442
if(unlikely(st->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE &&
1443
dt_usec(&st->last_collected_time, &st->last_updated) > (RRDENG_BLOCK_SIZE / sizeof(storage_number)) * update_every_ut)) {
1444
info(
1453
- "%s: too old data (last updated at %" PRId64 ".%" PRId64 ", last collected at %" PRId64 ".%" PRId64 "). "
1445
+ "'%s': too old data (last updated at %" PRId64 ".%" PRId64 ", last collected at %" PRId64 ".%" PRId64 "). "
1446
"Resetting it. Will not store the next entry.",
1455
- rrdset_name(st),
1447
+ rrdset_id(st),
1448
(int64_t)st->last_updated.tv_sec,
1449
(int64_t)st->last_updated.tv_usec,
1450
(int64_t)st->last_collected_time.tv_sec,
@@ -1497,16 +1489,12 @@ void rrdset_timed_done(RRDSET *st, struct timeval now) {
1489
store_this_entry = 1;
1490
last_collect_ut = next_store_ut - update_every_ut;
1491
1500
- #ifdef NETDATA_INTERNAL_CHECKS
1492
rrdset_debug(st, "Fixed first entry.");
1502
- #endif
1493
}
1494
else {
1495
store_this_entry = 0;
1496
1507
- #ifdef NETDATA_INTERNAL_CHECKS
1497
rrdset_debug(st, "Will not store the next entry.");
1509
- #endif
1498
}
1499
}
1500
@@ -1555,19 +1543,17 @@ after_first_database_work:
1543
if (unlikely(st->rrd_memory_mode == RRD_MEMORY_MODE_NONE))
1544
goto after_second_database_work;
1545
1558
- #ifdef NETDATA_INTERNAL_CHECKS
1546
rrdset_debug(st, "last_collect_ut = %0.3" NETDATA_DOUBLE_MODIFIER " (last collection time)", (NETDATA_DOUBLE)last_collect_ut/USEC_PER_SEC);
1547
rrdset_debug(st, "now_collect_ut = %0.3" NETDATA_DOUBLE_MODIFIER " (current collection time)", (NETDATA_DOUBLE)now_collect_ut/USEC_PER_SEC);
1548
rrdset_debug(st, "last_stored_ut = %0.3" NETDATA_DOUBLE_MODIFIER " (last updated time)", (NETDATA_DOUBLE)last_stored_ut/USEC_PER_SEC);
1549
rrdset_debug(st, "next_store_ut = %0.3" NETDATA_DOUBLE_MODIFIER " (next interpolation point)", (NETDATA_DOUBLE)next_store_ut/USEC_PER_SEC);
1563
- #endif
1550
1551
uint32_t has_reset_value = 0;
1552
1553
// process all dimensions to calculate their values
1554
// based on the collected figures only
1555
// at this stage we do not interpolate anything
1570
- for(dim_id = 0, rda = rda_base, rd = rda->rd ; dim_id < rda_slots ; ++dim_id, ++rda) {
1556
+ for(dim_id = 0, rda = rda_base ; dim_id < rda_slots ; ++dim_id, ++rda) {
1557
rd = rda->rd;
1558
if(unlikely(!rd)) continue;
1559
@@ -1576,7 +1562,6 @@ after_first_database_work:
1562
continue;
1563
}
1564
1579
- #ifdef NETDATA_INTERNAL_CHECKS
1565
rrdset_debug(st, "%s: START "
1566
" last_collected_value = " COLLECTED_NUMBER_FORMAT
1567
" collected_value = " COLLECTED_NUMBER_FORMAT
@@ -1588,7 +1573,6 @@ after_first_database_work:
1573
, rd->last_calculated_value
1574
, rd->calculated_value
1575
);
1591
- #endif
1576
1577
switch(rd->algorithm) {
1578
case RRD_ALGORITHM_ABSOLUTE:
@@ -1596,18 +1580,16 @@ after_first_database_work:
1580
* (NETDATA_DOUBLE)rd->multiplier
1581
/ (NETDATA_DOUBLE)rd->divisor;
1582
1599
- #ifdef NETDATA_INTERNAL_CHECKS
1583
rrdset_debug(st, "%s: CALC ABS/ABS-NO-IN " NETDATA_DOUBLE_FORMAT " = "
1584
COLLECTED_NUMBER_FORMAT
1585
" * " NETDATA_DOUBLE_FORMAT
1603
- " / " NETDATA_DOUBLE_FORMAT, rrddim_name(rd)
1586
+ " / " NETDATA_DOUBLE_FORMAT
1587
+ , rrddim_name(rd)
1588
, rd->calculated_value
1589
, rd->collected_value
1590
, (NETDATA_DOUBLE)rd->multiplier
1591
, (NETDATA_DOUBLE)rd->divisor
1592
);
1609
- #endif
1610
-
1593
break;
1594
1595
case RRD_ALGORITHM_PCENT_OVER_ROW_TOTAL:
@@ -1621,7 +1603,6 @@ after_first_database_work:
1603
* (NETDATA_DOUBLE)rd->collected_value
1604
/ (NETDATA_DOUBLE)st->collected_total;
1605
1624
- #ifdef NETDATA_INTERNAL_CHECKS
1606
rrdset_debug(st, "%s: CALC PCENT-ROW " NETDATA_DOUBLE_FORMAT " = 100"
1607
" * " COLLECTED_NUMBER_FORMAT
1608
" / " COLLECTED_NUMBER_FORMAT
@@ -1630,8 +1611,6 @@ after_first_database_work:
1611
, rd->collected_value
1612
, st->collected_total
1613
);
1633
- #endif
1634
-
1614
break;
1615
1616
case RRD_ALGORITHM_INCREMENTAL:
@@ -1645,8 +1624,9 @@ after_first_database_work:
1624
// It is imperative to set the comparison to uint64_t since type collected_number is signed and
1625
// produces wrong results as far as incremental counters are concerned.
1626
if(unlikely((uint64_t)rd->last_collected_value > (uint64_t)rd->collected_value)) {
1648
- debug(D_RRD_STATS, "%s.%s: RESET or OVERFLOW. Last collected value = " COLLECTED_NUMBER_FORMAT ", current = " COLLECTED_NUMBER_FORMAT
1649
- , rrdset_name(st), rrddim_name(rd)
1627
+ debug(D_RRD_STATS, "'%s' / '%s': RESET or OVERFLOW. Last collected value = " COLLECTED_NUMBER_FORMAT ", current = " COLLECTED_NUMBER_FORMAT
1628
+ , rrdset_id(st)
1629
+ , rrddim_name(rd)
1630
, rd->last_collected_value
1631
, rd->collected_value);
1632
@@ -1689,19 +1669,17 @@ after_first_database_work:
1669
/ (NETDATA_DOUBLE) rd->divisor;
1670
}
1671
1692
- #ifdef NETDATA_INTERNAL_CHECKS
1672
rrdset_debug(st, "%s: CALC INC PRE " NETDATA_DOUBLE_FORMAT " = ("
1673
COLLECTED_NUMBER_FORMAT " - " COLLECTED_NUMBER_FORMAT
1674
")"
1675
" * " NETDATA_DOUBLE_FORMAT
1697
- " / " NETDATA_DOUBLE_FORMAT, rrddim_name(rd)
1676
+ " / " NETDATA_DOUBLE_FORMAT
1677
+ , rrddim_name(rd)
1678
, rd->calculated_value
1679
, rd->collected_value, rd->last_collected_value
1680
, (NETDATA_DOUBLE)rd->multiplier
1681
, (NETDATA_DOUBLE)rd->divisor
1682
);
1703
- #endif
1704
-
1683
break;
1684
1685
case RRD_ALGORITHM_PCENT_OVER_DIFF_TOTAL:
@@ -1713,8 +1691,9 @@ after_first_database_work:
1691
// if the new is smaller than the old (an overflow, or reset), set the old equal to the new
1692
// to reset the calculation (it will give zero as the calculation for this second)
1693
if(unlikely(rd->last_collected_value > rd->collected_value)) {
1716
- debug(D_RRD_STATS, "%s.%s: RESET or OVERFLOW. Last collected value = " COLLECTED_NUMBER_FORMAT ", current = " COLLECTED_NUMBER_FORMAT
1717
- , rrdset_name(st), rrddim_name(rd)
1694
+ debug(D_RRD_STATS, "'%s' / '%s': RESET or OVERFLOW. Last collected value = " COLLECTED_NUMBER_FORMAT ", current = " COLLECTED_NUMBER_FORMAT
1695
+ , rrdset_id(st)
1696
+ , rrddim_name(rd)
1697
, rd->last_collected_value
1698
, rd->collected_value
1699
);
@@ -1735,7 +1714,6 @@ after_first_database_work:
1714
* (NETDATA_DOUBLE)(rd->collected_value - rd->last_collected_value)
1715
/ (NETDATA_DOUBLE)(st->collected_total - st->last_collected_total);
1716
1738
- #ifdef NETDATA_INTERNAL_CHECKS
1717
rrdset_debug(st, "%s: CALC PCENT-DIFF " NETDATA_DOUBLE_FORMAT " = 100"
1718
" * (" COLLECTED_NUMBER_FORMAT " - " COLLECTED_NUMBER_FORMAT ")"
1719
" / (" COLLECTED_NUMBER_FORMAT " - " COLLECTED_NUMBER_FORMAT ")"
@@ -1744,8 +1722,6 @@ after_first_database_work:
1722
, rd->collected_value, rd->last_collected_value
1723
, st->collected_total, st->last_collected_total
1724
);
1747
- #endif
1748
-
1725
break;
1726
1727
default:
@@ -1753,29 +1729,24 @@ after_first_database_work:
1729
// it gets noticed when we add new types
1730
rd->calculated_value = 0;
1731
1756
- #ifdef NETDATA_INTERNAL_CHECKS
1732
rrdset_debug(st, "%s: CALC " NETDATA_DOUBLE_FORMAT " = 0"
1733
, rrddim_name(rd)
1734
, rd->calculated_value
1735
);
1761
- #endif
1762
-
1736
break;
1737
}
1738
1766
- #ifdef NETDATA_INTERNAL_CHECKS
1739
rrdset_debug(st, "%s: PHASE2 "
1740
" last_collected_value = " COLLECTED_NUMBER_FORMAT
1741
" collected_value = " COLLECTED_NUMBER_FORMAT
1742
" last_calculated_value = " NETDATA_DOUBLE_FORMAT
1771
- " calculated_value = " NETDATA_DOUBLE_FORMAT, rrddim_name(rd)
1772
- , rd->last_collected_value
1773
- , rd->collected_value
1774
- , rd->last_calculated_value
1775
- , rd->calculated_value
1743
+ " calculated_value = " NETDATA_DOUBLE_FORMAT
1744
+ , rrddim_name(rd)
1745
+ , rd->last_collected_value
1746
+ , rd->collected_value
1747
+ , rd->last_calculated_value
1748
+ , rd->calculated_value
1749
);
1777
- #endif
1778
-
1750
}
1751
1752
// at this point we have all the calculated values ready
@@ -1805,43 +1776,41 @@ after_first_database_work:
1776
after_second_database_work:
1777
st->last_collected_total = st->collected_total;
1778
1808
- for(dim_id = 0, rda = rda_base, rd = rda->rd ; dim_id < rda_slots ; ++dim_id, ++rda) {
1779
+ for(dim_id = 0, rda = rda_base ; dim_id < rda_slots ; ++dim_id, ++rda) {
1780
rd = rda->rd;
1781
if(unlikely(!rd)) continue;
1782
1783
if(unlikely(!rd->updated))
1784
continue;
1785
1815
- #ifdef NETDATA_INTERNAL_CHECKS
1786
rrdset_debug(st, "%s: setting last_collected_value (old: " COLLECTED_NUMBER_FORMAT ") to last_collected_value (new: " COLLECTED_NUMBER_FORMAT ")", rrddim_name(rd), rd->last_collected_value, rd->collected_value);
1817
- #endif
1787
1788
rd->last_collected_value = rd->collected_value;
1789
1790
switch(rd->algorithm) {
1791
case RRD_ALGORITHM_INCREMENTAL:
1792
if(unlikely(!first_entry)) {
1824
- #ifdef NETDATA_INTERNAL_CHECKS
1825
- rrdset_debug(st, "%s: setting last_calculated_value (old: " NETDATA_DOUBLE_FORMAT
1826
- ") to last_calculated_value (new: " NETDATA_DOUBLE_FORMAT ")", rrddim_name(rd), rd->last_calculated_value + rd->calculated_value, rd->calculated_value);
1827
- #endif
1793
+ rrdset_debug(st, "%s: setting last_calculated_value (old: " NETDATA_DOUBLE_FORMAT ") to "
1794
+ "last_calculated_value (new: " NETDATA_DOUBLE_FORMAT ")"
1795
+ , rrddim_name(rd)
1796
+ , rd->last_calculated_value + rd->calculated_value
1797
+ , rd->calculated_value);
1798
1799
rd->last_calculated_value += rd->calculated_value;
1800
}
1801
else {
1832
- #ifdef NETDATA_INTERNAL_CHECKS
1802
rrdset_debug(st, "THIS IS THE FIRST POINT");
1834
- #endif
1803
}
1804
break;
1805
1806
case RRD_ALGORITHM_ABSOLUTE:
1807
case RRD_ALGORITHM_PCENT_OVER_ROW_TOTAL:
1808
case RRD_ALGORITHM_PCENT_OVER_DIFF_TOTAL:
1841
- #ifdef NETDATA_INTERNAL_CHECKS
1842
- rrdset_debug(st, "%s: setting last_calculated_value (old: " NETDATA_DOUBLE_FORMAT
1843
- ") to last_calculated_value (new: " NETDATA_DOUBLE_FORMAT ")", rrddim_name(rd), rd->last_calculated_value, rd->calculated_value);
1844
- #endif
1809
+ rrdset_debug(st, "%s: setting last_calculated_value (old: " NETDATA_DOUBLE_FORMAT ") to "
1810
+ "last_calculated_value (new: " NETDATA_DOUBLE_FORMAT ")"
1811
+ , rrddim_name(rd)
1812
+ , rd->last_calculated_value
1813
+ , rd->calculated_value);
1814
1815
rd->last_calculated_value = rd->calculated_value;
1816
break;
@@ -1851,19 +1820,17 @@ after_second_database_work:
1820
rd->collected_value = 0;
1821
rd->updated = 0;
1822
1854
- #ifdef NETDATA_INTERNAL_CHECKS
1823
rrdset_debug(st, "%s: END "
1824
" last_collected_value = " COLLECTED_NUMBER_FORMAT
1825
" collected_value = " COLLECTED_NUMBER_FORMAT
1826
" last_calculated_value = " NETDATA_DOUBLE_FORMAT
1859
- " calculated_value = " NETDATA_DOUBLE_FORMAT, rrddim_name(rd)
1860
- , rd->last_collected_value
1861
- , rd->collected_value
1862
- , rd->last_calculated_value
1863
- , rd->calculated_value
1827
+ " calculated_value = " NETDATA_DOUBLE_FORMAT
1828
+ , rrddim_name(rd)
1829
+ , rd->last_collected_value
1830
+ , rd->collected_value
1831
+ , rd->last_calculated_value
1832
+ , rd->calculated_value
1833
);
1865
- #endif
1866
-
1834
}
1835
1836
// ALL DONE ABOUT THE DATA UPDATE
@@ -1874,14 +1841,14 @@ after_second_database_work:
1841
1842
rrdset_memory_file_update(st);
1843
1877
- for(dim_id = 0, rda = rda_base, rd = rda->rd ; dim_id < rda_slots ; ++dim_id, ++rda) {
1844
+ for(dim_id = 0, rda = rda_base; dim_id < rda_slots ; ++dim_id, ++rda) {
1845
rd = rda->rd;
1846
if(unlikely(!rd)) continue;
1847
rrddim_memory_file_update(rd);
1848
}
1849
}
1850
1884
- for(dim_id = 0, rda = rda_base, rd = rda->rd ; dim_id < rda_slots ; ++dim_id, ++rda) {
1851
+ for(dim_id = 0, rda = rda_base; dim_id < rda_slots ; ++dim_id, ++rda) {
1852
rd = rda->rd;
1853
if(unlikely(!rd)) continue;
1854
@@ -1895,6 +1862,30 @@ after_second_database_work:
1862
netdata_thread_enable_cancelability();
1863
}
1864
1865
+time_t rrdset_set_update_every(RRDSET *st, time_t update_every) {
1866
+
1867
+ internal_error(true, "RRDSET '%s' switching update every from %d to %d",
1868
+ rrdset_id(st), (int)st->update_every, (int)update_every);
1869
+
1870
+ time_t prev_update_every = st->update_every;
1871
+ st->update_every = update_every;
1872
+
1873
+ // switch update every to the storage engine
1874
+ RRDDIM *rd;
1875
+ rrddim_foreach_read(rd, st) {
1876
+ for (size_t tier = 0; tier < storage_tiers; tier++) {
1877
+ if (rd->tiers[tier] && rd->tiers[tier]->db_collection_handle)
1878
+ rd->tiers[tier]->collect_ops->change_collection_frequency(rd->tiers[tier]->db_collection_handle, (int)(st->rrdhost->db[tier].tier_grouping * st->update_every));
1879
+ }
1880
+
1881
+ assert(rd->update_every == prev_update_every &&
1882
+ "chart's update every differs from the update every of its dimensions");
1883
+ rd->update_every = st->update_every;
1884
+ }
1885
+ rrddim_foreach_done(rd);
1886
+
1887
+ return prev_update_every;
1888
+}
1889
1890
// ----------------------------------------------------------------------------
1891
// compatibility layer for RRDSET files v019
database/sqlite/sqlite_aclk.c
+3
@@ -251,6 +251,9 @@ static int create_host_callback(void *data, int argc, char **argv, char **column
251
, NULL //destination
252
, NULL // api key
253
, NULL // send charts matching
254
+ , false // rrdpush_enable_replication
255
+ , 0 // rrdpush_seconds_to_replicate
256
+ , 0 // rrdpush_replication_step
257
, system_info
258
, 1
259
);
libnetdata/libnetdata.c
+6
-5
@@ -1816,10 +1816,11 @@ inline int config_isspace(char c)
1816
}
1817
1818
// split a text into words, respecting quotes
1819
-inline int quoted_strings_splitter(char *str, char **words, int max_words, int (*custom_isspace)(char), char *recover_input, char **recover_location, int max_recover)
1819
+inline size_t quoted_strings_splitter(char *str, char **words, size_t max_words, int (*custom_isspace)(char), char *recover_input, char **recover_location, int max_recover)
1820
{
1821
char *s = str, quote = 0;
1822
- int i = 0, rec = 0;
1822
+ size_t i = 0;
1823
+ int rec = 0;
1824
char *recover = recover_input;
1825
1826
// skip all white space
@@ -1891,13 +1892,13 @@ inline int quoted_strings_splitter(char *str, char **words, int max_words, int (
1892
s++;
1893
}
1894
1894
- // terminate the words
1895
- memset(&words[i], 0, (max_words - i) * sizeof (char *));
1895
+ if (i < max_words)
1896
+ words[i] = NULL;
1897
1898
return i;
1899
}
1900
1900
-inline int pluginsd_split_words(char *str, char **words, int max_words, char *recover_input, char **recover_location, int max_recover)
1901
+inline size_t pluginsd_split_words(char *str, char **words, size_t max_words, char *recover_input, char **recover_location, int max_recover)
1902
{
1903
return quoted_strings_splitter(str, words, max_words, pluginsd_space, recover_input, recover_location, max_recover);
1904
}
libnetdata/libnetdata.h
+10
-2
@@ -405,8 +405,16 @@ void bitmap256_set_bit(BITMAP256 *ptr, uint8_t idx, bool value);
405
#define PLUGINSD_LINE_MAX (COMPRESSION_MAX_MSG_SIZE - 1024)
406
int config_isspace(char c);
407
int pluginsd_space(char c);
408
-int quoted_strings_splitter(char *str, char **words, int max_words, int (*custom_isspace)(char), char *recover_input, char **recover_location, int max_recover);
409
-int pluginsd_split_words(char *str, char **words, int max_words, char *recover_string, char **recover_location, int max_recover);
408
+
409
+size_t quoted_strings_splitter(char *str, char **words, size_t max_words, int (*custom_isspace)(char), char *recover_input, char **recover_location, int max_recover);
410
+size_t pluginsd_split_words(char *str, char **words, size_t max_words, char *recover_string, char **recover_location, int max_recover);
411
+
412
+static inline char *get_word(char **words, size_t num_words, size_t index) {
413
+ if (index >= num_words)
414
+ return NULL;
415
+
416
+ return words[index];
417
+}
418
419
bool run_command_and_copy_output_to_stdout(const char *command, int max_line_length);
420
libnetdata/log/log.c
+12
-15
@@ -460,16 +460,13 @@ void syslog_init() {
460
}
461
}
462
463
-#define LOG_DATE_LENGTH 26
464
-
465
-static inline void log_date(char *buffer, size_t len) {
463
+void log_date(char *buffer, size_t len, time_t now) {
464
if(unlikely(!buffer || !len))
465
return;
466
469
- time_t t;
467
+ time_t t = now;
468
struct tm *tmp, tmbuf;
469
472
- t = now_realtime_sec();
470
tmp = localtime_r(&t, &tmbuf);
471
472
if (tmp == NULL) {
@@ -639,7 +636,7 @@ int error_log_limit(int reset) {
636
if(reset) {
637
if(prevented) {
638
char date[LOG_DATE_LENGTH];
642
- log_date(date, LOG_DATE_LENGTH);
639
+ log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
640
fprintf(
641
stderr,
642
"%s: %s LOG FLOOD PROTECTION reset for process '%s' "
@@ -662,7 +659,7 @@ int error_log_limit(int reset) {
659
if(now - start > error_log_throttle_period) {
660
if(prevented) {
661
char date[LOG_DATE_LENGTH];
665
- log_date(date, LOG_DATE_LENGTH);
662
+ log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
663
fprintf(
664
stderr,
665
"%s: %s LOG FLOOD PROTECTION resuming logging from process '%s' "
@@ -686,7 +683,7 @@ int error_log_limit(int reset) {
683
if(counter > error_log_errors_per_period) {
684
if(!prevented) {
685
char date[LOG_DATE_LENGTH];
689
- log_date(date, LOG_DATE_LENGTH);
686
+ log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
687
fprintf(
688
stderr,
689
"%s: %s LOG FLOOD PROTECTION too many logs (%lu logs in %"PRId64" seconds, threshold is set to %lu logs "
@@ -741,7 +738,7 @@ void debug_int( const char *file, const char *function, const unsigned long line
738
va_list args;
739
740
char date[LOG_DATE_LENGTH];
744
- log_date(date, LOG_DATE_LENGTH);
741
+ log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
742
743
va_start( args, fmt );
744
printf("%s: %s DEBUG : %s : (%04lu@%-20.20s:%-15.15s): ", date, program_name, netdata_thread_tag(), line, file, function);
@@ -780,7 +777,7 @@ void info_int( const char *file __maybe_unused, const char *function __maybe_unu
777
}
778
779
char date[LOG_DATE_LENGTH];
783
- log_date(date, LOG_DATE_LENGTH);
780
+ log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
781
782
va_start( args, fmt );
783
#ifdef NETDATA_INTERNAL_CHECKS
@@ -842,7 +839,7 @@ void error_int( const char *prefix, const char *file __maybe_unused, const char
839
}
840
841
char date[LOG_DATE_LENGTH];
845
- log_date(date, LOG_DATE_LENGTH);
842
+ log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
843
844
va_start( args, fmt );
845
#ifdef NETDATA_INTERNAL_CHECKS
@@ -905,7 +902,7 @@ void fatal_int( const char *file, const char *function, const unsigned long line
902
}
903
904
char date[LOG_DATE_LENGTH];
908
- log_date(date, LOG_DATE_LENGTH);
905
+ log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
906
907
log_lock();
908
@@ -960,7 +957,7 @@ void log_access( const char *fmt, ... ) {
957
netdata_mutex_lock(&access_mutex);
958
959
char date[LOG_DATE_LENGTH];
963
- log_date(date, LOG_DATE_LENGTH);
960
+ log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
961
fprintf(stdaccess, "%s: ", date);
962
963
va_start( args, fmt );
@@ -992,7 +989,7 @@ void log_health( const char *fmt, ... ) {
989
netdata_mutex_lock(&health_mutex);
990
991
char date[LOG_DATE_LENGTH];
995
- log_date(date, LOG_DATE_LENGTH);
992
+ log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
993
fprintf(stdhealth, "%s: ", date);
994
995
va_start( args, fmt );
@@ -1012,7 +1009,7 @@ void log_aclk_message_bin( const char *data, const size_t data_len, int tx, cons
1009
netdata_mutex_lock(&aclklog_mutex);
1010
1011
char date[LOG_DATE_LENGTH];
1015
- log_date(date, LOG_DATE_LENGTH);
1012
+ log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
1013
fprintf(aclklog, "%s: %s Msg:\"%s\", MQTT-topic:\"%s\": ", date, tx ? "OUTGOING" : "INCOMING", message_name, mqtt_topic);
1014
1015
fwrite(data, data_len, 1, aclklog);
libnetdata/log/log.h
+5
-1
@@ -45,7 +45,8 @@ extern "C" {
45
#define D_ACLK 0x0000000200000000
46
#define D_METADATALOG 0x0000000400000000
47
#define D_ACLK_SYNC 0x0000000800000000
48
-#define D_META_SYNC 0x0000002000000000
48
+#define D_META_SYNC 0x0000001000000000
49
+#define D_REPLICATION 0x0000002000000000
50
#define D_SYSTEM 0x8000000000000000
51
52
extern int web_server_is_multithreaded;
@@ -85,6 +86,9 @@ int error_log_limit(int reset);
86
void open_all_log_files();
87
void reopen_all_log_files();
88
89
+#define LOG_DATE_LENGTH 26
90
+void log_date(char *buffer, size_t len, time_t now);
91
+
92
static inline void debug_dummy(void) {}
93
94
void error_log_limit_reset(void);
libnetdata/socket/security.h
+1
-1
@@ -37,7 +37,7 @@
37
#include <openssl/decoder.h>
38
#endif
39
40
-struct netdata_ssl{
40
+struct netdata_ssl {
41
SSL *conn; //SSL connection
42
uint32_t flags; //The flags for SSL connection
43
};
libnetdata/socket/socket.c
+5
-1
@@ -1000,9 +1000,13 @@ ssize_t send_timeout(int sockfd, void *buf, size_t len, int flags, int timeout)
1000
1001
#ifdef ENABLE_HTTPS
1002
if(ssl->conn) {
1003
- if (!ssl->flags) {
1003
+ if (ssl->flags == NETDATA_SSL_HANDSHAKE_COMPLETE) {
1004
return SSL_write(ssl->conn, buf, len);
1005
}
1006
+ else {
1007
+ error("cannot write to SSL connection - connection is not ready.");
1008
+ return -1;
1009
+ }
1010
}
1011
#endif
1012
return send(sockfd, buf, len, flags);
parser/parser.c
+19
-8
@@ -29,7 +29,7 @@ inline int find_first_keyword(const char *str, char *keyword, int max_size, int
29
*
30
*/
31
32
-PARSER *parser_init(RRDHOST *host, void *user, void *input, void *output, PARSER_INPUT_TYPE flags)
32
+PARSER *parser_init(RRDHOST *host, void *user, void *input, void *output, PARSER_INPUT_TYPE flags, void *ssl __maybe_unused)
33
{
34
PARSER *parser;
35
@@ -38,6 +38,9 @@ PARSER *parser_init(RRDHOST *host, void *user, void *input, void *output, PARSER
38
parser->user = user;
39
parser->input = input;
40
parser->output = output;
41
+#ifdef ENABLE_HTTPS
42
+ parser->ssl_output = ssl;
43
+#endif
44
parser->flags = flags;
45
parser->host = host;
46
parser->worker_job_next_id = WORKER_PARSER_FIRST_JOB;
@@ -51,6 +54,7 @@ PARSER *parser_init(RRDHOST *host, void *user, void *input, void *output, PARSER
54
if (unlikely(!(flags & PARSER_NO_PARSE_INIT))) {
55
parser_add_keyword(parser, PLUGINSD_KEYWORD_FLUSH, pluginsd_flush);
56
parser_add_keyword(parser, PLUGINSD_KEYWORD_CHART, pluginsd_chart);
57
+ parser_add_keyword(parser, PLUGINSD_KEYWORD_CHART_DEFINITION_END, pluginsd_chart_definition_end);
58
parser_add_keyword(parser, PLUGINSD_KEYWORD_DIMENSION, pluginsd_dimension);
59
parser_add_keyword(parser, PLUGINSD_KEYWORD_DISABLE, pluginsd_disable);
60
parser_add_keyword(parser, PLUGINSD_KEYWORD_VARIABLE, pluginsd_variable);
@@ -61,9 +65,15 @@ PARSER *parser_init(RRDHOST *host, void *user, void *input, void *output, PARSER
65
parser_add_keyword(parser, PLUGINSD_KEYWORD_CLABEL, pluginsd_clabel);
66
parser_add_keyword(parser, PLUGINSD_KEYWORD_BEGIN, pluginsd_begin);
67
parser_add_keyword(parser, PLUGINSD_KEYWORD_SET, pluginsd_set);
64
- parser_add_keyword(parser, PLUGINSD_KEYWORD_FUNCTION, pluginsd_function);
68
+
69
+ parser_add_keyword(parser, PLUGINSD_KEYWORD_FUNCTION, pluginsd_function);
70
parser_add_keyword(parser, PLUGINSD_KEYWORD_FUNCTION_RESULT_BEGIN, pluginsd_function_result_begin);
66
- //parser_add_keyword(parser, PLUGINSD_KEYWORD_GAPS_REQUEST, pluginsd_gaps_request);
71
+
72
+ parser_add_keyword(parser, PLUGINSD_KEYWORD_REPLAY_BEGIN, pluginsd_replay_rrdset_begin);
73
+ parser_add_keyword(parser, PLUGINSD_KEYWORD_REPLAY_SET, pluginsd_replay_set);
74
+ parser_add_keyword(parser, PLUGINSD_KEYWORD_REPLAY_RRDDIM_STATE, pluginsd_replay_rrddim_collection_state);
75
+ parser_add_keyword(parser, PLUGINSD_KEYWORD_REPLAY_RRDSET_STATE, pluginsd_replay_rrdset_collection_state);
76
+ parser_add_keyword(parser, PLUGINSD_KEYWORD_REPLAY_END, pluginsd_replay_end);
77
}
78
79
return parser;
@@ -246,7 +256,7 @@ int parser_next(PARSER *parser)
256
inline int parser_action(PARSER *parser, char *input)
257
{
258
PARSER_RC rc = PARSER_RC_OK;
249
- char *words[PLUGINSD_MAX_WORDS] = { NULL };
259
+ char *words[PLUGINSD_MAX_WORDS];
260
char command[PLUGINSD_LINE_MAX + 1];
261
keyword_function action_function;
262
keyword_function *action_function_list = NULL;
@@ -303,10 +313,11 @@ inline int parser_action(PARSER *parser, char *input)
313
if (unlikely(!find_first_keyword(input, command, PLUGINSD_LINE_MAX, pluginsd_space)))
314
return 0;
315
316
+ size_t num_words = 0;
317
if ((parser->flags & PARSER_INPUT_KEEP_ORIGINAL) == PARSER_INPUT_KEEP_ORIGINAL)
307
- pluginsd_split_words(input, words, PLUGINSD_MAX_WORDS, parser->recover_input, parser->recover_location, PARSER_MAX_RECOVER_KEYWORDS);
318
+ num_words = pluginsd_split_words(input, words, PLUGINSD_MAX_WORDS, parser->recover_input, parser->recover_location, PARSER_MAX_RECOVER_KEYWORDS);
319
else
309
- pluginsd_split_words(input, words, PLUGINSD_MAX_WORDS, NULL, NULL, 0);
320
+ num_words = pluginsd_split_words(input, words, PLUGINSD_MAX_WORDS, NULL, NULL, 0);
321
322
uint32_t command_hash = simple_hash(command);
323
@@ -323,7 +334,7 @@ inline int parser_action(PARSER *parser, char *input)
334
335
if (unlikely(!action_function_list)) {
336
if (unlikely(parser->unknown_function))
326
- rc = parser->unknown_function(words, parser->user, NULL);
337
+ rc = parser->unknown_function(words, num_words, parser->user, NULL);
338
else
339
rc = PARSER_RC_ERROR;
340
@@ -332,7 +343,7 @@ inline int parser_action(PARSER *parser, char *input)
343
else {
344
worker_is_busy(worker_job_id);
345
while ((action_function = *action_function_list) != NULL) {
335
- rc = action_function(words, parser->user, parser->plugins_action);
346
+ rc = action_function(words, num_words, parser->user, parser->plugins_action);
347
if (unlikely(rc == PARSER_RC_ERROR || rc == PARSER_RC_STOP)) {
348
internal_error(true, "action_function() failed with rc = %u", rc);
349
break;
parser/parser.h
+27
-17
@@ -35,7 +35,7 @@ typedef enum parser_input_type {
35
36
#define PARSER_INPUT_FULL (PARSER_INPUT_SPLIT|PARSER_INPUT_ORIGINAL)
37
38
-typedef PARSER_RC (*keyword_function)(char **, void *, PLUGINSD_ACTION *plugins_action);
38
+typedef PARSER_RC (*keyword_function)(char **words, size_t num_words, void *user_data, PLUGINSD_ACTION *plugins_action);
39
40
typedef struct parser_keyword {
41
size_t worker_job_id;
@@ -57,6 +57,9 @@ typedef struct parser {
57
RRDHOST *host;
58
void *input; // Input source e.g. stream
59
void *output; // Stream to send commands to plugin
60
+#ifdef ENABLE_HTTPS
61
+ struct netdata_ssl *ssl_output;
62
+#endif
63
PARSER_DATA *data; // extra input
64
PARSER_KEYWORD *keyword; // List of parse keywords and functions
65
PLUGINSD_ACTION *plugins_action;
@@ -91,7 +94,7 @@ typedef struct parser {
94
95
int find_first_keyword(const char *str, char *keyword, int max_size, int (*custom_isspace)(char));
96
94
-PARSER *parser_init(RRDHOST *host, void *user, void *input, void *output, PARSER_INPUT_TYPE flags);
97
+PARSER *parser_init(RRDHOST *host, void *user, void *input, void *output, PARSER_INPUT_TYPE flags, void *ssl);
98
int parser_add_keyword(PARSER *working_parser, char *keyword, keyword_function func);
99
int parser_next(PARSER *working_parser);
100
int parser_action(PARSER *working_parser, char *input);
@@ -101,20 +104,27 @@ int parser_recover_input(PARSER *working_parser);
104
105
size_t pluginsd_process(RRDHOST *host, struct plugind *cd, FILE *fp_plugin_input, FILE *fp_plugin_output, int trust_durations);
106
104
-PARSER_RC pluginsd_set(char **words, void *user, PLUGINSD_ACTION *plugins_action);
105
-PARSER_RC pluginsd_begin(char **words, void *user, PLUGINSD_ACTION *plugins_action);
106
-PARSER_RC pluginsd_end(char **words, void *user, PLUGINSD_ACTION *plugins_action);
107
-PARSER_RC pluginsd_chart(char **words, void *user, PLUGINSD_ACTION *plugins_action);
108
-PARSER_RC pluginsd_dimension(char **words, void *user, PLUGINSD_ACTION *plugins_action);
109
-PARSER_RC pluginsd_variable(char **words, void *user, PLUGINSD_ACTION *plugins_action);
110
-PARSER_RC pluginsd_flush(char **words, void *user, PLUGINSD_ACTION *plugins_action);
111
-PARSER_RC pluginsd_disable(char **words, void *user, PLUGINSD_ACTION *plugins_action);
112
-PARSER_RC pluginsd_label(char **words, void *user, PLUGINSD_ACTION *plugins_action);
113
-PARSER_RC pluginsd_overwrite(char **words, void *user, PLUGINSD_ACTION *plugins_action);
114
-PARSER_RC pluginsd_guid(char **words, void *user, PLUGINSD_ACTION *plugins_action);
115
-PARSER_RC pluginsd_context(char **words, void *user, PLUGINSD_ACTION *plugins_action);
116
-PARSER_RC pluginsd_tombstone(char **words, void *user, PLUGINSD_ACTION *plugins_action);
117
-PARSER_RC pluginsd_clabel_commit(char **words, void *user, PLUGINSD_ACTION *plugins_action);
118
-PARSER_RC pluginsd_clabel(char **words, void *user, PLUGINSD_ACTION *plugins_action);
107
+PARSER_RC pluginsd_set(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
108
+PARSER_RC pluginsd_begin(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
109
+PARSER_RC pluginsd_end(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
110
+PARSER_RC pluginsd_chart(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
111
+PARSER_RC pluginsd_chart_definition_end(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
112
+PARSER_RC pluginsd_dimension(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
113
+PARSER_RC pluginsd_variable(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
114
+PARSER_RC pluginsd_flush(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
115
+PARSER_RC pluginsd_disable(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
116
+PARSER_RC pluginsd_label(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
117
+PARSER_RC pluginsd_overwrite(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
118
+PARSER_RC pluginsd_guid(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
119
+PARSER_RC pluginsd_context(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
120
+PARSER_RC pluginsd_tombstone(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
121
+PARSER_RC pluginsd_clabel_commit(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
122
+PARSER_RC pluginsd_clabel(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
123
+
124
+PARSER_RC pluginsd_replay_rrdset_begin(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
125
+PARSER_RC pluginsd_replay_rrddim_collection_state(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
126
+PARSER_RC pluginsd_replay_rrdset_collection_state(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
127
+PARSER_RC pluginsd_replay_set(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
128
+PARSER_RC pluginsd_replay_end(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action);
129
130
#endif
streaming/receiver.c
+54
-82
@@ -65,99 +65,43 @@ static void rrdpush_receiver_thread_cleanup(void *ptr) {
65
66
#include "collectors/plugins.d/pluginsd_parser.h"
67
68
-PARSER_RC streaming_timestamp(char **words, void *user, PLUGINSD_ACTION *plugins_action)
68
+PARSER_RC streaming_claimed_id(char **words, size_t num_words, void *user, PLUGINSD_ACTION *plugins_action)
69
{
70
UNUSED(plugins_action);
71
- char *remote_time_txt = words[1];
72
- time_t remote_time = 0;
73
- RRDHOST *host = ((PARSER_USER_OBJECT *)user)->host;
74
- struct plugind *cd = ((PARSER_USER_OBJECT *)user)->cd;
75
- if (!(cd->capabilities & STREAM_CAP_GAP_FILLING)) {
76
- error("STREAM %s from %s: Child negotiated version %u but sent TIMESTAMP!", rrdhost_hostname(host), cd->cmd, cd->capabilities);
77
- return PARSER_RC_OK; // Ignore error and continue stream
78
- }
79
- if (remote_time_txt && *remote_time_txt) {
80
- remote_time = str2ull(remote_time_txt);
81
- time_t now = now_realtime_sec(), prev = rrdhost_last_entry_t(host);
82
- time_t gap = 0;
83
- if (prev == 0)
84
- info(
85
- "STREAM %s from %s: Initial connection (no gap to check), "
86
- "remote=%"PRId64" local=%"PRId64" slew=%"PRId64"",
87
- rrdhost_hostname(host),
88
- cd->cmd,
89
- (int64_t)remote_time,
90
- (int64_t)now,
91
- (int64_t)now - remote_time);
92
- else {
93
- gap = now - prev;
94
- info(
95
- "STREAM %s from %s: Checking for gaps... "
96
- "remote=%"PRId64" local=%"PRId64"..%"PRId64" slew=%"PRId64" %"PRId64"-sec gap",
97
- rrdhost_hostname(host),
98
- cd->cmd,
99
- (int64_t)remote_time,
100
- (int64_t)prev,
101
- (int64_t)now,
102
- (int64_t)(remote_time - now),
103
- (int64_t)gap);
104
- }
105
- char message[128];
106
- sprintf(
107
- message,
108
- "REPLICATE %"PRId64" %"PRId64"\n",
109
- (int64_t)(remote_time - gap),
110
- (int64_t)remote_time);
111
- int ret;
112
-#ifdef ENABLE_HTTPS
113
- SSL *conn = host->receiver->ssl.conn ;
114
- if(conn && !host->receiver->ssl.flags) {
115
- ret = SSL_write(conn, message, strlen(message));
116
- } else {
117
- ret = send(host->receiver->fd, message, strlen(message), MSG_DONTWAIT);
118
- }
119
-#else
120
- ret = send(host->receiver->fd, message, strlen(message), MSG_DONTWAIT);
121
-#endif
122
- if (ret != (int)strlen(message))
123
- error("Failed to send initial timestamp - gaps may appear in charts");
124
- return PARSER_RC_OK;
125
- }
126
- return PARSER_RC_ERROR;
127
-}
71
129
-PARSER_RC streaming_claimed_id(char **words, void *user, PLUGINSD_ACTION *plugins_action)
130
-{
131
- UNUSED(plugins_action);
72
+ const char *host_uuid_str = get_word(words, num_words, 1);
73
+ const char *claim_id_str = get_word(words, num_words, 2);
74
133
- uuid_t uuid;
134
- RRDHOST *host = ((PARSER_USER_OBJECT *)user)->host;
135
-
136
- if (!words[1] || !words[2]) {
137
- error("Command CLAIMED_ID came malformed, uuid = '%s', claim_id = '%s'", words[1]?words[1]:"[unset]", words[2]?words[2]:"[unset]");
75
+ if (!host_uuid_str || !claim_id_str) {
76
+ error("Command CLAIMED_ID came malformed, uuid = '%s', claim_id = '%s'",
77
+ host_uuid_str ? host_uuid_str : "[unset]",
78
+ claim_id_str ? claim_id_str : "[unset]");
79
return PARSER_RC_ERROR;
80
}
81
82
+ uuid_t uuid;
83
+ RRDHOST *host = ((PARSER_USER_OBJECT *)user)->host;
84
+
85
// We don't need the parsed UUID
86
// just do it to check the format
143
- if(uuid_parse(words[1], uuid)) {
144
- error("1st parameter (host GUID) to CLAIMED_ID command is not valid GUID. Received: \"%s\".", words[1]);
87
+ if(uuid_parse(host_uuid_str, uuid)) {
88
+ error("1st parameter (host GUID) to CLAIMED_ID command is not valid GUID. Received: \"%s\".", host_uuid_str);
89
return PARSER_RC_ERROR;
90
}
147
- if(uuid_parse(words[2], uuid) && strcmp(words[2], "NULL")) {
148
- error("2nd parameter (Claim ID) to CLAIMED_ID command is not valid GUID. Received: \"%s\".", words[2]);
91
+ if(uuid_parse(claim_id_str, uuid) && strcmp(claim_id_str, "NULL")) {
92
+ error("2nd parameter (Claim ID) to CLAIMED_ID command is not valid GUID. Received: \"%s\".", claim_id_str);
93
return PARSER_RC_ERROR;
94
}
95
152
- if(strcmp(words[1], host->machine_guid)) {
153
- error("Claim ID is for host \"%s\" but it came over connection for \"%s\"", words[1], host->machine_guid);
96
+ if(strcmp(host_uuid_str, host->machine_guid)) {
97
+ error("Claim ID is for host \"%s\" but it came over connection for \"%s\"", host_uuid_str, host->machine_guid);
98
return PARSER_RC_OK; //the message is OK problem must be somewhere else
99
}
100
101
rrdhost_aclk_state_lock(host);
102
if (host->aclk_state.claimed_id)
103
freez(host->aclk_state.claimed_id);
160
- host->aclk_state.claimed_id = strcmp(words[2], "NULL") ? strdupz(words[2]) : NULL;
104
+ host->aclk_state.claimed_id = strcmp(claim_id_str, "NULL") ? strdupz(claim_id_str) : NULL;
105
106
metaqueue_store_claim_id(&host->host_uuid, host->aclk_state.claimed_id ? &uuid : NULL);
107
@@ -400,7 +344,7 @@ static void streaming_parser_thread_cleanup(void *ptr) {
344
parser_destroy(parser);
345
}
346
403
-size_t streaming_parser(struct receiver_state *rpt, struct plugind *cd, FILE *fp_in, FILE *fp_out) {
347
+static size_t streaming_parser(struct receiver_state *rpt, struct plugind *cd, FILE *fp_in, FILE *fp_out, void *ssl) {
348
size_t result;
349
350
PARSER_USER_OBJECT user = {
@@ -411,7 +355,7 @@ size_t streaming_parser(struct receiver_state *rpt, struct plugind *cd, FILE *fp
355
.trust_durations = 1
356
};
357
414
- PARSER *parser = parser_init(rpt->host, &user, fp_in, fp_out, PARSER_INPUT_SPLIT);
358
+ PARSER *parser = parser_init(rpt->host, &user, fp_in, fp_out, PARSER_INPUT_SPLIT, ssl);
359
360
rrd_collector_started();
361
@@ -419,7 +363,6 @@ size_t streaming_parser(struct receiver_state *rpt, struct plugind *cd, FILE *fp
363
// so, parser needs to be allocated before pushing it
364
netdata_thread_cleanup_push(streaming_parser_thread_cleanup, parser);
365
422
- parser_add_keyword(parser, "TIMESTAMP", streaming_timestamp);
366
parser_add_keyword(parser, "CLAIMED_ID", streaming_claimed_id);
367
368
user.parser = parser;
@@ -472,6 +415,9 @@ static int rrdpush_receive(struct receiver_state *rpt)
415
char *rrdpush_destination = default_rrdpush_destination;
416
char *rrdpush_api_key = default_rrdpush_api_key;
417
char *rrdpush_send_charts_matching = default_rrdpush_send_charts_matching;
418
+ bool rrdpush_enable_replication = default_rrdpush_enable_replication;
419
+ time_t rrdpush_seconds_to_replicate = default_rrdpush_seconds_to_replicate;
420
+ time_t rrdpush_replication_step = default_rrdpush_replication_step;
421
time_t alarms_delay = 60;
422
423
rpt->update_every = (int)appconfig_get_number(&stream_config, rpt->machine_guid, "update every", rpt->update_every);
@@ -507,6 +453,15 @@ static int rrdpush_receive(struct receiver_state *rpt)
453
rrdpush_send_charts_matching = appconfig_get(&stream_config, rpt->key, "default proxy send charts matching", rrdpush_send_charts_matching);
454
rrdpush_send_charts_matching = appconfig_get(&stream_config, rpt->machine_guid, "proxy send charts matching", rrdpush_send_charts_matching);
455
456
+ rrdpush_enable_replication = appconfig_get_boolean(&stream_config, rpt->key, "enable replication", rrdpush_enable_replication);
457
+ rrdpush_enable_replication = appconfig_get_boolean(&stream_config, rpt->machine_guid, "enable replication", rrdpush_enable_replication);
458
+
459
+ rrdpush_seconds_to_replicate = appconfig_get_number(&stream_config, rpt->key, "seconds to replicate", rrdpush_seconds_to_replicate);
460
+ rrdpush_seconds_to_replicate = appconfig_get_number(&stream_config, rpt->machine_guid, "seconds to replicate", rrdpush_seconds_to_replicate);
461
+
462
+ rrdpush_replication_step = appconfig_get_number(&stream_config, rpt->key, "seconds per replication step", rrdpush_replication_step);
463
+ rrdpush_replication_step = appconfig_get_number(&stream_config, rpt->machine_guid, "seconds per replication step", rrdpush_replication_step);
464
+
465
#ifdef ENABLE_COMPRESSION
466
unsigned int rrdpush_compression = default_compression_enabled;
467
rrdpush_compression = appconfig_get_boolean(&stream_config, rpt->key, "enable compression", rrdpush_compression);
@@ -556,6 +511,9 @@ static int rrdpush_receive(struct receiver_state *rpt)
511
, rrdpush_destination
512
, rrdpush_api_key
513
, rrdpush_send_charts_matching
514
+ , rrdpush_enable_replication
515
+ , rrdpush_seconds_to_replicate
516
+ , rrdpush_replication_step
517
, rpt->system_info
518
, 0
519
);
@@ -601,6 +559,9 @@ static int rrdpush_receive(struct receiver_state *rpt)
559
rrdpush_destination,
560
rrdpush_api_key,
561
rrdpush_send_charts_matching,
562
+ rrdpush_enable_replication,
563
+ rrdpush_seconds_to_replicate,
564
+ rrdpush_replication_step,
565
rpt->system_info);
566
rrd_unlock();
567
}
@@ -748,8 +709,11 @@ static int rrdpush_receive(struct receiver_state *rpt)
709
rrdhost_unlock(rpt->host);
710
711
// call the plugins.d processor to receive the metrics
751
- info("STREAM %s [receive from [%s]:%s]: receiving metrics...", rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port);
752
- log_stream_connection(rpt->client_ip, rpt->client_port, rpt->key, rpt->host->machine_guid, rrdhost_hostname(rpt->host), "CONNECTED");
712
+ info("STREAM %s [receive from [%s]:%s]: receiving metrics...",
713
+ rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port);
714
+
715
+ log_stream_connection(rpt->client_ip, rpt->client_port,
716
+ rpt->key, rpt->host->machine_guid, rrdhost_hostname(rpt->host), "CONNECTED");
717
718
cd.capabilities = rpt->capabilities;
719
@@ -764,12 +728,20 @@ static int rrdpush_receive(struct receiver_state *rpt)
728
729
rrdcontext_host_child_connected(rpt->host);
730
767
- size_t count = streaming_parser(rpt, &cd, fp_in, fp_out);
731
+ size_t count = streaming_parser(rpt, &cd, fp_in, fp_out,
732
+#ifdef ENABLE_HTTPS
733
+ (rpt->ssl.conn) ? &rpt->ssl : NULL
734
+#else
735
+ NULL
736
+#endif
737
+ );
738
769
- log_stream_connection(rpt->client_ip, rpt->client_port, rpt->key, rpt->host->machine_guid, rpt->hostname,
739
+ log_stream_connection(rpt->client_ip, rpt->client_port,
740
+ rpt->key, rpt->host->machine_guid, rpt->hostname,
741
"DISCONNECTED");
771
- error("STREAM %s [receive from [%s]:%s]: disconnected (completed %zu updates).", rpt->hostname, rpt->client_ip,
772
- rpt->client_port, count);
742
+
743
+ error("STREAM %s [receive from [%s]:%s]: disconnected (completed %zu updates).",
744
+ rpt->hostname, rpt->client_ip, rpt->client_port, count);
745
746
rrdcontext_host_child_disconnected(rpt->host);
747
streaming/replication.c
new
+344
@@ -0,0 +1,344 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "replication.h"
4
+
5
+static time_t replicate_chart_timeframe(BUFFER *wb, RRDSET *st, time_t after, time_t before, bool enable_streaming) {
6
+ size_t dimensions = rrdset_number_of_dimensions(st);
7
+
8
+ struct storage_engine_query_ops *ops = &st->rrdhost->db[0].eng->api.query_ops;
9
+
10
+ struct {
11
+ DICTIONARY *dict;
12
+ const DICTIONARY_ITEM *rda;
13
+ RRDDIM *rd;
14
+ struct storage_engine_query_handle handle;
15
+ STORAGE_POINT sp;
16
+ } data[dimensions];
17
+
18
+ memset(data, 0, sizeof(data));
19
+
20
+ if(enable_streaming && st->last_updated.tv_sec > before) {
21
+ internal_error(true, "REPLAY: '%s' overwriting replication before from %llu to %llu",
22
+ rrdset_id(st),
23
+ (unsigned long long)before,
24
+ (unsigned long long)st->last_updated.tv_sec
25
+ );
26
+ before = st->last_updated.tv_sec;
27
+ }
28
+
29
+ // prepare our array of dimensions
30
+ {
31
+ RRDDIM *rd;
32
+ rrddim_foreach_read(rd, st) {
33
+ if (rd_dfe.counter >= dimensions)
34
+ break;
35
+
36
+ data[rd_dfe.counter].dict = rd_dfe.dict;
37
+ data[rd_dfe.counter].rda = dictionary_acquired_item_dup(rd_dfe.dict, rd_dfe.item);
38
+ data[rd_dfe.counter].rd = rd;
39
+
40
+ ops->init(rd->tiers[0]->db_metric_handle, &data[rd_dfe.counter].handle, after, before);
41
+ }
42
+ rrddim_foreach_done(rd);
43
+ }
44
+
45
+ time_t now = after, actual_after = 0, actual_before = 0;
46
+ while(now <= before) {
47
+ time_t min_start_time = 0, min_end_time = 0;
48
+ for (size_t i = 0; i < dimensions && data[i].rd; i++) {
49
+ // fetch the first valid point for the dimension
50
+ int max_skip = 100;
51
+ while(data[i].sp.end_time < now && !ops->is_finished(&data[i].handle) && max_skip-- > 0)
52
+ data[i].sp = ops->next_metric(&data[i].handle);
53
+
54
+ if(max_skip <= 0)
55
+ error("REPLAY: host '%s', chart '%s', dimension '%s': db does not advance the query beyond time %llu",
56
+ rrdhost_hostname(st->rrdhost), rrdset_id(st), rrddim_id(data[i].rd), (unsigned long long)now);
57
+
58
+ if(data[i].sp.end_time < now)
59
+ continue;
60
+
61
+ if(!min_start_time) {
62
+ min_start_time = data[i].sp.start_time;
63
+ min_end_time = data[i].sp.end_time;
64
+ }
65
+ else {
66
+ min_start_time = MIN(min_start_time, data[i].sp.start_time);
67
+ min_end_time = MIN(min_end_time, data[i].sp.end_time);
68
+ }
69
+ }
70
+
71
+ if(min_end_time < now) {
72
+ internal_error(true,
73
+ "REPLAY: host '%s', chart '%s': no data on any dimension beyond time %llu",
74
+ rrdhost_hostname(st->rrdhost), rrdset_id(st), (unsigned long long)now);
75
+ break;
76
+ }
77
+
78
+ if(min_end_time <= min_start_time)
79
+ min_start_time = min_end_time - st->update_every;
80
+
81
+ if(!actual_after) {
82
+ actual_after = min_end_time;
83
+ actual_before = min_end_time;
84
+ }
85
+ else
86
+ actual_before = min_end_time;
87
+
88
+ buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_BEGIN " '' %llu %llu\n"
89
+ , (unsigned long long)min_start_time
90
+ , (unsigned long long)min_end_time);
91
+
92
+ // output the replay values for this time
93
+ for (size_t i = 0; i < dimensions && data[i].rd; i++) {
94
+ if(data[i].sp.start_time <= min_end_time && data[i].sp.end_time >= min_end_time)
95
+ buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_SET " \"%s\" " NETDATA_DOUBLE_FORMAT_AUTO " \"%s\"\n",
96
+ rrddim_id(data[i].rd), data[i].sp.sum, data[i].sp.flags & SN_FLAG_RESET ? "R" : "");
97
+ else
98
+ buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_SET " \"%s\" NAN \"E\"\n",
99
+ rrddim_id(data[i].rd));
100
+ }
101
+
102
+ now = min_end_time + 1;
103
+ }
104
+
105
+#ifdef NETDATA_INTERNAL_CHECKS
106
+ if(actual_after) {
107
+ char actual_after_buf[LOG_DATE_LENGTH + 1], actual_before_buf[LOG_DATE_LENGTH + 1];
108
+ log_date(actual_after_buf, LOG_DATE_LENGTH, actual_after);
109
+ log_date(actual_before_buf, LOG_DATE_LENGTH, actual_before);
110
+ internal_error(true,
111
+ "REPLAY: host '%s', chart '%s': sending data %llu [%s] to %llu [%s] (requested %llu [delta %lld] to %llu [delta %lld])",
112
+ rrdhost_hostname(st->rrdhost), rrdset_id(st),
113
+ (unsigned long long)actual_after, actual_after_buf, (unsigned long long)actual_before, actual_before_buf,
114
+ (unsigned long long)after, (long long)(actual_after - after), (unsigned long long)before, (long long)(actual_before - before));
115
+ }
116
+ else
117
+ internal_error(true,
118
+ "REPLAY: host '%s', chart '%s': nothing to send (requested %llu to %llu)",
119
+ rrdhost_hostname(st->rrdhost), rrdset_id(st),
120
+ (unsigned long long)after, (unsigned long long)before);
121
+#endif
122
+
123
+ // release all the dictionary items acquired
124
+ // finalize the queries
125
+ for(size_t i = 0; i < dimensions && data[i].rda ;i++) {
126
+ ops->finalize(&data[i].handle);
127
+ dictionary_acquired_item_release(data[i].dict, data[i].rda);
128
+ }
129
+
130
+ return before;
131
+}
132
+
133
+static void replicate_chart_collection_state(BUFFER *wb, RRDSET *st) {
134
+ RRDDIM *rd;
135
+ rrddim_foreach_read(rd, st) {
136
+ buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_RRDDIM_STATE " \"%s\" %llu %lld " NETDATA_DOUBLE_FORMAT_AUTO " " NETDATA_DOUBLE_FORMAT_AUTO "\n",
137
+ rrddim_id(rd),
138
+ (usec_t)rd->last_collected_time.tv_sec * USEC_PER_SEC + (usec_t)rd->last_collected_time.tv_usec,
139
+ rd->last_collected_value,
140
+ rd->last_calculated_value,
141
+ rd->last_stored_value
142
+ );
143
+ }
144
+ rrddim_foreach_done(rd);
145
+
146
+ buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_RRDSET_STATE " %llu %llu " TOTAL_NUMBER_FORMAT " " TOTAL_NUMBER_FORMAT "\n",
147
+ (usec_t)st->last_collected_time.tv_sec * USEC_PER_SEC + (usec_t)st->last_collected_time.tv_usec,
148
+ (usec_t)st->last_updated.tv_sec * USEC_PER_SEC + (usec_t)st->last_updated.tv_usec,
149
+ st->last_collected_total,
150
+ st->collected_total
151
+ );
152
+}
153
+
154
+bool replicate_chart_response(RRDHOST *host, RRDSET *st, bool start_streaming, time_t after, time_t before) {
155
+ time_t query_after = after;
156
+ time_t query_before = before;
157
+ time_t now = now_realtime_sec();
158
+
159
+ // find the first entry we have
160
+ time_t first_entry_local = rrdset_first_entry_t(st);
161
+ if(first_entry_local > now) {
162
+ internal_error(true,
163
+ "RRDSET: '%s' first time %llu is in the future (now is %llu)",
164
+ rrdset_id(st), (unsigned long long)first_entry_local, (unsigned long long)now);
165
+ first_entry_local = now;
166
+ }
167
+
168
+ if (query_after < first_entry_local)
169
+ query_after = first_entry_local;
170
+
171
+ // find the latest entry we have
172
+ time_t last_entry_local = st->last_updated.tv_sec;
173
+ if(last_entry_local > now) {
174
+ internal_error(true,
175
+ "RRDSET: '%s' last updated time %llu is in the future (now is %llu)",
176
+ rrdset_id(st), (unsigned long long)last_entry_local, (unsigned long long)now);
177
+ last_entry_local = now;
178
+ }
179
+
180
+ if (query_before > last_entry_local)
181
+ query_before = last_entry_local;
182
+
183
+ // if the parent asked us to start streaming, then fill the rest with the data that we have
184
+ if (start_streaming)
185
+ query_before = last_entry_local;
186
+
187
+ if (query_after > query_before) {
188
+ time_t tmp = query_before;
189
+ query_before = query_after;
190
+ query_after = tmp;
191
+ }
192
+
193
+ bool enable_streaming = (start_streaming || query_before == last_entry_local || !after || !before) ? true : false;
194
+
195
+ // we might want to optimize this by filling a temporary buffer
196
+ // and copying the result to the host's buffer in order to avoid
197
+ // holding the host's buffer lock for too long
198
+ BUFFER *wb = sender_start(host->sender);
199
+ {
200
+ // pass the original after/before so that the parent knows about
201
+ // which time range we responded
202
+ buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_BEGIN " \"%s\"\n", rrdset_id(st));
203
+
204
+ if(after != 0 && before != 0)
205
+ before = replicate_chart_timeframe(wb, st, query_after, query_before, enable_streaming);
206
+ else {
207
+ after = 0;
208
+ before = 0;
209
+ enable_streaming = true;
210
+ }
211
+
212
+ if(enable_streaming)
213
+ replicate_chart_collection_state(wb, st);
214
+
215
+ // end with first/last entries we have, and the first start time and
216
+ // last end time of the data we sent
217
+ buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_END " %d %llu %llu %s %llu %llu\n",
218
+ (int)st->update_every, (unsigned long long)first_entry_local, (unsigned long long)last_entry_local,
219
+ enable_streaming ? "true" : "false", (unsigned long long)after, (unsigned long long)before);
220
+ }
221
+ sender_commit(host->sender, wb);
222
+
223
+ return enable_streaming;
224
+}
225
+
226
+static bool send_replay_chart_cmd(send_command callback, void *callback_data, RRDSET *st, bool start_streaming, time_t after, time_t before) {
227
+
228
+#ifdef NETDATA_INTERNAL_CHECKS
229
+ if(after && before) {
230
+ char after_buf[LOG_DATE_LENGTH + 1], before_buf[LOG_DATE_LENGTH + 1];
231
+ log_date(after_buf, LOG_DATE_LENGTH, after);
232
+ log_date(before_buf, LOG_DATE_LENGTH, before);
233
+ internal_error(true,
234
+ "REPLAY: host '%s', chart '%s': sending replication request %llu [%s] to %llu [%s], start streaming: %s",
235
+ rrdhost_hostname(st->rrdhost), rrdset_id(st),
236
+ (unsigned long long)after, after_buf, (unsigned long long)before, before_buf,
237
+ start_streaming?"true":"false");
238
+ }
239
+ else {
240
+ internal_error(true,
241
+ "REPLAY: host '%s', chart '%s': sending empty replication request, start streaming: %s",
242
+ rrdhost_hostname(st->rrdhost), rrdset_id(st),
243
+ start_streaming?"true":"false");
244
+ }
245
+#endif
246
+
247
+ debug(D_REPLICATION, PLUGINSD_KEYWORD_REPLAY_CHART " \"%s\" \"%s\" %llu %llu\n",
248
+ rrdset_id(st), start_streaming ? "true" : "false", (unsigned long long)after, (unsigned long long)before);
249
+
250
+ char buffer[2048 + 1];
251
+ snprintfz(buffer, 2048, PLUGINSD_KEYWORD_REPLAY_CHART " \"%s\" \"%s\" %llu %llu\n",
252
+ rrdset_id(st), start_streaming ? "true" : "false",
253
+ (unsigned long long)after, (unsigned long long)before);
254
+
255
+ int ret = callback(buffer, callback_data);
256
+ if (ret < 0) {
257
+ error("failed to send replay request to child (ret=%d)", ret);
258
+ return false;
259
+ }
260
+
261
+ return true;
262
+}
263
+
264
+bool replicate_chart_request(send_command callback, void *callback_data, RRDHOST *host, RRDSET *st,
265
+ time_t first_entry_child, time_t last_entry_child,
266
+ time_t prev_first_entry_wanted, time_t prev_last_entry_wanted)
267
+{
268
+ time_t now = now_realtime_sec();
269
+
270
+ // if replication is disabled, send an empty replication request
271
+ // asking no data
272
+ if (!host->rrdpush_enable_replication) {
273
+ internal_error(true,
274
+ "REPLAY: host '%s', chart '%s': sending empty replication request because replication is disabled",
275
+ rrdhost_hostname(host), rrdset_id(st));
276
+
277
+ return send_replay_chart_cmd(callback, callback_data, st, true, 0, 0);
278
+ }
279
+
280
+ // Child has no stored data
281
+ if (!last_entry_child) {
282
+ error("REPLAY: host '%s', chart '%s': sending empty replication request because child has no stored data",
283
+ rrdhost_hostname(host), rrdset_id(st));
284
+
285
+ return send_replay_chart_cmd(callback, callback_data, st, true, 0, 0);
286
+ }
287
+
288
+ // Nothing to get if the chart has not dimensions
289
+ if (!rrdset_number_of_dimensions(st)) {
290
+ error("REPLAY: host '%s', chart '%s': sending empty replication request because chart has no dimensions",
291
+ rrdhost_hostname(host), rrdset_id(st));
292
+
293
+ return send_replay_chart_cmd(callback, callback_data, st, true, 0, 0);
294
+ }
295
+
296
+ // if the child's first/last entries are nonsensical, resume streaming
297
+ // without asking for any data
298
+ if (first_entry_child <= 0) {
299
+ error("REPLAY: host '%s', chart '%s': sending empty replication because first entry of the child is invalid (%llu)",
300
+ rrdhost_hostname(host), rrdset_id(st), (unsigned long long)first_entry_child);
301
+
302
+ return send_replay_chart_cmd(callback, callback_data, st, true, 0, 0);
303
+ }
304
+
305
+ if (first_entry_child > last_entry_child) {
306
+ error("REPLAY: host '%s', chart '%s': sending empty replication because child timings are invalid (first entry %llu > last entry %llu)",
307
+ rrdhost_hostname(host), rrdset_id(st), (unsigned long long)first_entry_child, (unsigned long long)last_entry_child);
308
+
309
+ return send_replay_chart_cmd(callback, callback_data, st, true, 0, 0);
310
+ }
311
+
312
+ time_t last_entry_local = rrdset_last_entry_t(st);
313
+ if(last_entry_local > now) {
314
+ internal_error(true,
315
+ "REPLAY: host '%s', chart '%s': local last entry time %llu is in the future (now is %llu). Adjusting it.",
316
+ rrdhost_hostname(host), rrdset_id(st), (unsigned long long)last_entry_local, (unsigned long long)now);
317
+ last_entry_local = now;
318
+ }
319
+
320
+ // should never happen but it if does, start streaming without asking
321
+ // for any data
322
+ if (last_entry_local > last_entry_child) {
323
+ error("REPLAY: host '%s', chart '%s': sending empty replication request because our last entry (%llu) in later than the child one (%llu)",
324
+ rrdhost_hostname(host), rrdset_id(st), (unsigned long long)last_entry_local, (unsigned long long)last_entry_child);
325
+
326
+ return send_replay_chart_cmd(callback, callback_data, st, true, 0, 0);
327
+ }
328
+
329
+ time_t first_entry_wanted;
330
+ if (prev_first_entry_wanted && prev_last_entry_wanted) {
331
+ first_entry_wanted = prev_last_entry_wanted;
332
+ if ((now - first_entry_wanted) > host->rrdpush_seconds_to_replicate)
333
+ first_entry_wanted = now - host->rrdpush_seconds_to_replicate;
334
+ }
335
+ else
336
+ first_entry_wanted = MAX(last_entry_local, first_entry_child);
337
+
338
+ time_t last_entry_wanted = first_entry_wanted + host->rrdpush_replication_step;
339
+ last_entry_wanted = MIN(last_entry_wanted, last_entry_child);
340
+
341
+ bool start_streaming = (last_entry_wanted == last_entry_child);
342
+
343
+ return send_replay_chart_cmd(callback, callback_data, st, start_streaming, first_entry_wanted, last_entry_wanted);
344
+}
streaming/replication.h
new
+17
@@ -0,0 +1,17 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef REPLICATION_H
4
+#define REPLICATION_H
5
+
6
+#include "daemon/common.h"
7
+
8
+bool replicate_chart_response(RRDHOST *rh, RRDSET *rs, bool start_streaming, time_t after, time_t before);
9
+
10
+typedef int (*send_command)(const char *txt, void *data);
11
+
12
+bool replicate_chart_request(send_command callback, void *callback_data,
13
+ RRDHOST *rh, RRDSET *rs,
14
+ time_t first_entry_child, time_t last_entry_child,
15
+ time_t response_first_start_time, time_t response_last_end_time);
16
+
17
+#endif /* REPLICATION_H */
streaming/rrdpush.c
+37
-51
@@ -46,6 +46,9 @@ unsigned int default_compression_enabled = 1;
46
char *default_rrdpush_destination = NULL;
47
char *default_rrdpush_api_key = NULL;
48
char *default_rrdpush_send_charts_matching = NULL;
49
+bool default_rrdpush_enable_replication = true;
50
+time_t default_rrdpush_seconds_to_replicate = 86400;
51
+time_t default_rrdpush_replication_step = 600;
52
#ifdef ENABLE_HTTPS
53
int netdata_use_ssl_on_stream = NETDATA_SSL_OPTIONAL;
54
char *netdata_ssl_ca_path = NULL;
@@ -100,6 +103,11 @@ int rrdpush_init() {
103
default_rrdpush_destination = appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "destination", "");
104
default_rrdpush_api_key = appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "api key", "");
105
default_rrdpush_send_charts_matching = appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "send charts matching", "*");
106
+
107
+ default_rrdpush_enable_replication = config_get_boolean(CONFIG_SECTION_DB, "enable replication", default_rrdpush_enable_replication);
108
+ default_rrdpush_seconds_to_replicate = config_get_number(CONFIG_SECTION_DB, "seconds to replicate", default_rrdpush_seconds_to_replicate);
109
+ default_rrdpush_replication_step = config_get_number(CONFIG_SECTION_DB, "seconds per replication step", default_rrdpush_replication_step);
110
+
111
rrdhost_free_orphan_time = config_get_number(CONFIG_SECTION_DB, "cleanup orphan hosts after secs", rrdhost_free_orphan_time);
112
113
#ifdef ENABLE_COMPRESSION
@@ -153,16 +161,20 @@ int rrdpush_init() {
161
// this is for the first iterations of each chart
162
unsigned int remote_clock_resync_iterations = 60;
163
156
-
164
static inline bool should_send_chart_matching(RRDSET *st) {
165
// get all the flags we need to check, with one atomic operation
166
RRDSET_FLAGS flags = rrdset_flag_check(st,
160
- RRDSET_FLAG_UPSTREAM_SEND
161
- |RRDSET_FLAG_UPSTREAM_IGNORE
162
- |RRDSET_FLAG_ANOMALY_RATE_CHART
163
- |RRDSET_FLAG_ANOMALY_DETECTION);
167
+ RRDSET_FLAG_UPSTREAM_SEND
168
+ | RRDSET_FLAG_UPSTREAM_IGNORE
169
+ | RRDSET_FLAG_ANOMALY_RATE_CHART
170
+ | RRDSET_FLAG_ANOMALY_DETECTION
171
+ | RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED
172
+ );
173
+
174
+ if(!(flags & RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED))
175
+ return false;
176
165
- if(unlikely(!flags)) {
177
+ if(unlikely(!(flags & (RRDSET_FLAG_UPSTREAM_SEND | RRDSET_FLAG_UPSTREAM_IGNORE)))) {
178
RRDHOST *host = st->rrdhost;
179
180
// Do not stream anomaly rates charts.
@@ -295,20 +307,25 @@ static inline void rrdpush_send_chart_definition(BUFFER *wb, RRDSET *st) {
307
// send the chart local custom variables
308
rrdsetvar_print_to_streaming_custom_chart_variables(st, wb);
309
310
+ if (stream_has_capability(host->sender, STREAM_CAP_REPLICATION)) {
311
+ time_t first_entry_local = rrdset_first_entry_t(st);
312
+ time_t last_entry_local = st->last_updated.tv_sec;
313
+ buffer_sprintf(wb, "CHART_DEFINITION_END %ld %ld\n", first_entry_local, last_entry_local);
314
+ }
315
+
316
st->upstream_resync_time = st->last_collected_time.tv_sec + (remote_clock_resync_iterations * st->update_every);
317
}
318
319
// sends the current chart dimensions
302
-static inline void rrdpush_send_chart_metrics(BUFFER *wb, RRDSET *st, struct sender_state *s) {
320
+void rrdpush_send_chart_metrics(BUFFER *wb, RRDSET *st, struct sender_state *s) {
321
buffer_fast_strcat(wb, "BEGIN \"", 7);
322
buffer_fast_strcat(wb, rrdset_id(st), string_strlen(st->id));
323
buffer_fast_strcat(wb, "\" ", 2);
306
- buffer_print_llu(wb, (st->last_collected_time.tv_sec > st->upstream_resync_time)?st->usec_since_last_update:0);
324
308
- if (stream_has_capability(s, STREAM_CAP_GAP_FILLING)) {
309
- buffer_fast_strcat(wb, " ", 1);
310
- buffer_print_ll(wb, st->last_collected_time.tv_sec);
311
- }
325
+ if(stream_has_capability(s, STREAM_CAP_REPLICATION) || st->last_collected_time.tv_sec > st->upstream_resync_time)
326
+ buffer_print_llu(wb, st->usec_since_last_update);
327
+ else
328
+ buffer_fast_strcat(wb, "0", 1);
329
330
buffer_fast_strcat(wb, "\n", 1);
331
@@ -350,62 +367,30 @@ bool rrdset_push_chart_definition_now(RRDSET *st) {
367
return true;
368
}
369
353
-bool rrdpush_incremental_transmission_of_chart_definitions(RRDHOST *host, DICTFE *dictfe, bool restart, bool stop) {
354
- if(stop || restart)
355
- dictionary_foreach_done(dictfe);
356
-
357
- if(stop)
358
- return false;
359
-
360
- RRDSET *st = NULL;
361
-
362
- if(unlikely(!dictfe->dict)) {
363
- st = dictionary_foreach_start_rw(dictfe, host->rrdset_root_index, DICTIONARY_LOCK_REENTRANT);
364
- }
365
- else
366
- st = dictionary_foreach_next(dictfe);
367
-
368
- do {
369
- while(st && !need_to_send_chart_definition(st))
370
- st = dictionary_foreach_next(dictfe);
371
-
372
- if(st && rrdset_push_chart_definition_now(st))
373
- break;
374
-
375
- } while((st = dictionary_foreach_next(dictfe)));
376
-
377
- if (!st) {
378
- dictionary_foreach_done(dictfe);
379
- return false;
380
- }
381
-
382
- return true;
383
-}
384
-
370
void rrdset_done_push(RRDSET *st) {
371
RRDHOST *host = st->rrdhost;
372
373
// fetch the flags we need to check with one atomic operation
389
- RRDHOST_FLAGS flags = rrdhost_flag_check(host,
374
+ RRDHOST_FLAGS host_flags = rrdhost_flag_check(host,
375
RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS
376
| RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS
377
| RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN
378
);
379
380
// check if we are not connected
396
- if(unlikely(!(flags & RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS))) {
381
+ if(unlikely(!(host_flags & RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS))) {
382
398
- if(unlikely(!(flags & RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN)))
383
+ if(unlikely(!(host_flags & RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN)))
384
rrdpush_sender_thread_spawn(host);
385
401
- if(unlikely(!(flags & RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS))) {
386
+ if(unlikely(!(host_flags & RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS))) {
387
rrdhost_flag_set(host, RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS);
388
error("STREAM %s [send]: not ready - collected metrics are not sent to parent.", rrdhost_hostname(host));
389
}
390
391
return;
392
}
408
- else if(unlikely(flags & RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS)) {
393
+ else if(unlikely(host_flags & RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS)) {
394
info("STREAM %s [send]: sending metrics to parent...", rrdhost_hostname(host));
395
rrdhost_flag_clear(host, RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS);
396
}
@@ -418,7 +403,8 @@ void rrdset_done_push(RRDSET *st) {
403
if(unlikely(need_to_send_chart_definition(st)))
404
rrdpush_send_chart_definition(wb, st);
405
421
- rrdpush_send_chart_metrics(wb, st, host->sender);
406
+ if (rrdset_flag_check(st, RRDSET_FLAG_SENDER_REPLICATION_FINISHED))
407
+ rrdpush_send_chart_metrics(wb, st, host->sender);
408
409
sender_commit(host->sender, wb);
410
}
@@ -949,7 +935,7 @@ static void stream_capabilities_to_string(BUFFER *wb, STREAM_CAPABILITIES caps)
935
if(caps & STREAM_CAP_CLABELS) buffer_strcat(wb, "CLABELS ");
936
if(caps & STREAM_CAP_COMPRESSION) buffer_strcat(wb, "COMPRESSION ");
937
if(caps & STREAM_CAP_FUNCTIONS) buffer_strcat(wb, "FUNCTIONS ");
952
- if(caps & STREAM_CAP_GAP_FILLING) buffer_strcat(wb, "GAP_FILLING ");
938
+ if(caps & STREAM_CAP_REPLICATION) buffer_strcat(wb, "REPLICATION");
939
}
940
941
void log_receiver_capabilities(struct receiver_state *rpt) {
streaming/rrdpush.h
+14
-4
@@ -7,6 +7,7 @@
7
#include "libnetdata/libnetdata.h"
8
#include "web/server/web_client.h"
9
#include "daemon/common.h"
10
+#include "replication.h"
11
12
#define CONNECTED_TO_SIZE 100
13
@@ -34,7 +35,7 @@ typedef enum {
35
STREAM_CAP_CLABELS = (1 << 9), // chart labels supported
36
STREAM_CAP_COMPRESSION = (1 << 10), // lz4 compression supported
37
STREAM_CAP_FUNCTIONS = (1 << 11), // plugin functions supported
37
- STREAM_CAP_GAP_FILLING = (1 << 12), // gap filling supported
38
+ STREAM_CAP_REPLICATION = (1 << 12), // replication supported
39
40
// this must be signed int, so don't use the last bit
41
// needed for negotiating errors between parent and child
@@ -46,7 +47,10 @@ typedef enum {
47
#define STREAM_HAS_COMPRESSION 0
48
#endif //ENABLE_COMPRESSION
49
49
-#define STREAM_OUR_CAPABILITIES (STREAM_CAP_V1 | STREAM_CAP_V2 | STREAM_CAP_VN | STREAM_CAP_VCAPS | STREAM_CAP_HLABELS | STREAM_CAP_CLAIM | STREAM_CAP_CLABELS | STREAM_HAS_COMPRESSION | STREAM_CAP_FUNCTIONS)
50
+#define STREAM_OUR_CAPABILITIES ( \
51
+ STREAM_CAP_V1 | STREAM_CAP_V2 | STREAM_CAP_VN | STREAM_CAP_VCAPS | \
52
+ STREAM_CAP_HLABELS | STREAM_CAP_CLAIM | STREAM_CAP_CLABELS | \
53
+ STREAM_HAS_COMPRESSION | STREAM_CAP_FUNCTIONS | STREAM_CAP_REPLICATION )
54
55
#define stream_has_capability(rpt, capability) ((rpt) && ((rpt)->capabilities & (capability)))
56
@@ -164,6 +168,8 @@ struct sender_state {
168
#ifdef ENABLE_HTTPS
169
struct netdata_ssl ssl; // Structure used to encrypt the connection
170
#endif
171
+
172
+ DICTIONARY *replication_requests;
173
};
174
175
struct receiver_state {
@@ -218,12 +224,15 @@ extern unsigned int default_compression_enabled;
224
extern char *default_rrdpush_destination;
225
extern char *default_rrdpush_api_key;
226
extern char *default_rrdpush_send_charts_matching;
227
+extern bool default_rrdpush_enable_replication;
228
+extern time_t default_rrdpush_seconds_to_replicate;
229
+extern time_t default_rrdpush_replication_step;
230
extern unsigned int remote_clock_resync_iterations;
231
232
void rrdpush_destinations_init(RRDHOST *host);
233
void rrdpush_destinations_free(RRDHOST *host);
234
226
-void sender_init(RRDHOST *parent);
235
+void sender_init(RRDHOST *host);
236
BUFFER *sender_start(struct sender_state *s);
237
void sender_commit(struct sender_state *s, BUFFER *wb);
238
void sender_cancel(struct sender_state *s);
@@ -232,7 +241,6 @@ bool rrdpush_receiver_needs_dbengine();
241
int configured_as_parent();
242
void rrdset_done_push(RRDSET *st);
243
bool rrdset_push_chart_definition_now(RRDSET *st);
235
-bool rrdpush_incremental_transmission_of_chart_definitions(RRDHOST *host, DICTFE *dictfe, bool restart, bool stop);
244
void *rrdpush_sender_thread(void *ptr);
245
void rrdpush_send_host_labels(RRDHOST *host);
246
void rrdpush_claimed_id(RRDHOST *host);
@@ -264,4 +272,6 @@ void log_sender_capabilities(struct sender_state *s);
272
STREAM_CAPABILITIES convert_stream_version_to_capabilities(int32_t version);
273
int32_t stream_capabilities_to_vn(uint32_t caps);
274
275
+void rrdpush_send_chart_metrics(BUFFER *wb, RRDSET *st, struct sender_state *s);
276
+
277
#endif //NETDATA_RRDPUSH_H
streaming/sender.c
+163
-73
@@ -31,6 +31,12 @@ extern int netdata_use_ssl_on_stream;
31
extern char *netdata_ssl_ca_path;
32
extern char *netdata_ssl_ca_file;
33
34
+struct replication_request {
35
+ bool start_streaming;
36
+ time_t after;
37
+ time_t before;
38
+};
39
+
40
static __thread BUFFER *sender_thread_buffer = NULL;
41
static __thread bool sender_thread_buffer_used = false;
42
@@ -94,6 +100,13 @@ void sender_commit(struct sender_state *s, BUFFER *wb) {
100
101
netdata_mutex_lock(&s->mutex);
102
103
+ if(unlikely(s->host->sender->buffer->max_size < (buffer_strlen(wb) + 1) * 2)) {
104
+ error("STREAM %s [send to %s]: max buffer size of %zu is too small for data of size %zu. Increasing the max buffer size to twice the max data size.",
105
+ rrdhost_hostname(s->host), s->connected_to, s->host->sender->buffer->max_size, buffer_strlen(wb) + 1);
106
+
107
+ s->host->sender->buffer->max_size = (buffer_strlen(wb) + 1) * 2;
108
+ }
109
+
110
#ifdef ENABLE_COMPRESSION
111
if (s->flags & SENDER_FLAG_COMPRESSION && s->compressor) {
112
while(src_len) {
@@ -213,10 +226,24 @@ static void rrdpush_sender_thread_send_custom_host_variables(RRDHOST *host) {
226
// resets all the chart, so that their definitions
227
// will be resent to the central netdata
228
static void rrdpush_sender_thread_reset_all_charts(RRDHOST *host) {
229
+ error("Clearing stream_collected_metrics flag in charts of host %s", rrdhost_hostname(host));
230
+
231
+ bool receive_has_replication = host != localhost && host->receiver && stream_has_capability(host->receiver, STREAM_CAP_REPLICATION);
232
+ bool send_has_replication = host->sender && stream_has_capability(host->sender, STREAM_CAP_REPLICATION);
233
+
234
RRDSET *st;
235
rrdset_foreach_read(st, host) {
236
rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
237
238
+ if(!receive_has_replication)
239
+ rrdset_flag_set(st, RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
240
+
241
+ if(send_has_replication)
242
+ // it will be enabled once replication is done on the sending side
243
+ rrdset_flag_clear(st, RRDSET_FLAG_SENDER_REPLICATION_FINISHED);
244
+ else
245
+ rrdset_flag_set(st, RRDSET_FLAG_SENDER_REPLICATION_FINISHED);
246
+
247
st->upstream_resync_time = 0;
248
249
RRDDIM *rd;
@@ -234,6 +261,7 @@ static inline void rrdpush_sender_thread_data_flush(RRDHOST *host) {
261
262
rrdpush_sender_thread_reset_all_charts(host);
263
rrdpush_sender_thread_send_custom_host_variables(host);
264
+ dictionary_flush(host->sender->replication_requests);
265
}
266
267
void rrdpush_encode_variable(stream_encoded_t *se, RRDHOST *host)
@@ -848,17 +876,19 @@ void execute_commands(struct sender_state *s) {
876
internal_error(true, "STREAM %s [send to %s] received command over connection: %s", rrdhost_hostname(s->host), s->connected_to, start);
877
878
char *words[PLUGINSD_MAX_WORDS] = { NULL };
851
- pluginsd_split_words(start, words, PLUGINSD_MAX_WORDS, NULL, NULL, 0);
879
+ size_t num_words = pluginsd_split_words(start, words, PLUGINSD_MAX_WORDS, NULL, NULL, 0);
880
853
- if(words[0] && strcmp(words[0], PLUGINSD_KEYWORD_FUNCTION) == 0) {
854
- char *transaction = words[1];
855
- char *timeout_s = words[2];
856
- char *function = words[3];
881
+ const char *keyword = get_word(words, num_words, 0);
882
+
883
+ if(keyword && strcmp(keyword, PLUGINSD_KEYWORD_FUNCTION) == 0) {
884
+ char *transaction = get_word(words, num_words, 1);
885
+ char *timeout_s = get_word(words, num_words, 2);
886
+ char *function = get_word(words, num_words, 3);
887
888
if(!transaction || !*transaction || !timeout_s || !*timeout_s || !function || !*function) {
889
error("STREAM %s [send to %s] %s execution command is incomplete (transaction = '%s', timeout = '%s', function = '%s'). Ignoring it.",
890
rrdhost_hostname(s->host), s->connected_to,
861
- words[0],
891
+ keyword,
892
transaction?transaction:"(unset)",
893
timeout_s?timeout_s:"(unset)",
894
function?function:"(unset)");
@@ -879,9 +909,32 @@ void execute_commands(struct sender_state *s) {
909
stream_execute_function_callback(wb, code, tmp);
910
}
911
}
882
- }
883
- else
912
+ } else if (keyword && strcmp(keyword, PLUGINSD_KEYWORD_REPLAY_CHART) == 0) {
913
+ const char *chart_id = get_word(words, num_words, 1);
914
+ const char *start_streaming = get_word(words, num_words, 2);
915
+ const char *after = get_word(words, num_words, 3);
916
+ const char *before = get_word(words, num_words, 4);
917
+
918
+ if (!chart_id || !start_streaming || !after || !before) {
919
+ error("STREAM %s [send to %s] %s command is incomplete"
920
+ " (chart=%s, start_streaming=%s, after=%s, before=%s)",
921
+ rrdhost_hostname(s->host), s->connected_to,
922
+ keyword,
923
+ chart_id ? chart_id : "(unset)",
924
+ start_streaming ? start_streaming : "(unset)",
925
+ after ? after : "(unset)",
926
+ before ? before : "(unset)");
927
+ } else {
928
+ struct replication_request tmp = {
929
+ .start_streaming = !strcmp(start_streaming, "true"),
930
+ .after = strtoll(after, NULL, 0),
931
+ .before = strtoll(before, NULL, 0),
932
+ };
933
+ dictionary_set(s->replication_requests, chart_id, &tmp, sizeof(struct replication_request));
934
+ }
935
+ } else {
936
error("STREAM %s [send to %s] received unknown command over connection: %s", rrdhost_hostname(s->host), s->connected_to, words[0]?words[0]:"(unset)");
937
+ }
938
939
start = newline + 1;
940
}
@@ -898,44 +951,9 @@ void execute_commands(struct sender_state *s) {
951
struct rrdpush_sender_thread_data {
952
struct sender_state *sender_state;
953
RRDHOST *host;
901
- DICTFE dictfe;
902
- enum {
903
- SENDING_DEFINITIONS_RESTART,
904
- SENDING_DEFINITIONS_CONTINUE,
905
- SENDING_DEFINITIONS_DONE,
906
- } sending_definitions_status;
954
char *pipe_buffer;
955
};
956
910
-static size_t cbuffer_available_bytes_with_lock(struct rrdpush_sender_thread_data *thread_data) {
911
- netdata_mutex_lock(&thread_data->sender_state->mutex);
912
- size_t outstanding = cbuffer_available_size_unsafe(thread_data->sender_state->host->sender->buffer);
913
- netdata_mutex_unlock(&thread_data->sender_state->mutex);
914
- return outstanding;
915
-}
916
-
917
-static void rrdpush_queue_incremental_definitions(struct rrdpush_sender_thread_data *thread_data) {
918
-
919
- while(rrdhost_flag_check(thread_data->host, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED)
920
- && thread_data->sending_definitions_status != SENDING_DEFINITIONS_DONE
921
- && cbuffer_available_bytes_with_lock(thread_data) > (thread_data->sender_state->buffer->max_size / 2)) {
922
-
923
- if(thread_data->sending_definitions_status == SENDING_DEFINITIONS_RESTART)
924
- info("STREAM %s [send to %s]: sending metric definitions...", rrdhost_hostname(thread_data->host), thread_data->sender_state->connected_to);
925
-
926
- bool more_defs_available = rrdpush_incremental_transmission_of_chart_definitions(
927
- thread_data->sender_state->host, &thread_data->dictfe,
928
- thread_data->sending_definitions_status == SENDING_DEFINITIONS_RESTART, false);
929
-
930
- if (unlikely(!more_defs_available)) {
931
- thread_data->sending_definitions_status = SENDING_DEFINITIONS_DONE;
932
- info("STREAM %s [send to %s]: sending metric definitions finished.", rrdhost_hostname(thread_data->host), thread_data->sender_state->connected_to);
933
- }
934
- else
935
- thread_data->sending_definitions_status = SENDING_DEFINITIONS_CONTINUE;
936
- }
937
-}
938
-
957
static bool rrdpush_sender_pipe_close(RRDHOST *host, int *pipe_fds, bool reopen) {
958
static netdata_mutex_t mutex = NETDATA_MUTEX_INITIALIZER;
959
@@ -997,8 +1015,6 @@ static void rrdpush_sender_thread_cleanup_callback(void *ptr) {
1015
1016
RRDHOST *host = data->host;
1017
1000
- rrdpush_incremental_transmission_of_chart_definitions(host, &data->dictfe, false, true);
1001
-
1018
netdata_mutex_lock(&host->sender->mutex);
1019
1020
info("STREAM %s [send]: sending thread cleans up...", rrdhost_hostname(host));
@@ -1021,28 +1037,114 @@ static void rrdpush_sender_thread_cleanup_callback(void *ptr) {
1037
freez(data);
1038
}
1039
1024
-void sender_init(RRDHOST *parent)
1040
+static void replication_request_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value __maybe_unused, void *sender_state __maybe_unused) {
1041
+ ;
1042
+}
1043
+static bool replication_request_conflict_callback(const DICTIONARY_ITEM *item, void *old_value, void *new_value, void *sender_state) {
1044
+ struct sender_state *s = sender_state;
1045
+ struct replication_request *rr = old_value;
1046
+ struct replication_request *rr_new = new_value;
1047
+
1048
+ error("STREAM %s [send to %s]: duplicate replication command received for chart '%s' (existing from %llu to %llu [%s], new from %llu to %llu [%s])",
1049
+ rrdhost_hostname(s->host), s->connected_to, dictionary_acquired_item_name(item),
1050
+ (unsigned long long)rr->after, (unsigned long long)rr->before, rr->start_streaming?"true":"false",
1051
+ (unsigned long long)rr_new->after, (unsigned long long)rr_new->before, rr_new->start_streaming?"true":"false");
1052
+
1053
+ bool updated = false;
1054
+
1055
+ if(rr_new->after < rr->after) {
1056
+ rr->after = rr_new->after;
1057
+ updated = true;
1058
+ }
1059
+
1060
+ if(rr_new->before > rr->before) {
1061
+ rr->before = rr_new->before;
1062
+ updated = true;
1063
+ }
1064
+
1065
+ if(rr_new->start_streaming != rr->start_streaming) {
1066
+ rr->start_streaming = true;
1067
+ updated = true;
1068
+ }
1069
+
1070
+ return updated;
1071
+}
1072
+static void replication_request_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value __maybe_unused, void *sender_state __maybe_unused) {
1073
+ ;
1074
+}
1075
+
1076
+void sender_init(RRDHOST *host)
1077
{
1026
- if (parent->sender)
1078
+ if (host->sender)
1079
return;
1080
1029
- parent->sender = callocz(1, sizeof(*parent->sender));
1030
- parent->sender->host = parent;
1031
- parent->sender->buffer = cbuffer_new(1024, 1024*1024);
1032
- parent->sender->capabilities = STREAM_OUR_CAPABILITIES;
1081
+ host->sender = callocz(1, sizeof(*host->sender));
1082
+ host->sender->host = host;
1083
+ host->sender->buffer = cbuffer_new(1024, 1024 * 1024);
1084
+ host->sender->capabilities = STREAM_OUR_CAPABILITIES;
1085
1034
- parent->sender->rrdpush_sender_pipe[PIPE_READ] = -1;
1035
- parent->sender->rrdpush_sender_pipe[PIPE_WRITE] = -1;
1036
- parent->sender->rrdpush_sender_socket = -1;
1086
+ host->sender->rrdpush_sender_pipe[PIPE_READ] = -1;
1087
+ host->sender->rrdpush_sender_pipe[PIPE_WRITE] = -1;
1088
+ host->sender->rrdpush_sender_socket = -1;
1089
1090
#ifdef ENABLE_COMPRESSION
1091
if(default_compression_enabled) {
1040
- parent->sender->flags |= SENDER_FLAG_COMPRESSION;
1041
- parent->sender->compressor = create_compressor();
1092
+ host->sender->flags |= SENDER_FLAG_COMPRESSION;
1093
+ host->sender->compressor = create_compressor();
1094
}
1095
#endif
1096
1045
- netdata_mutex_init(&parent->sender->mutex);
1097
+ netdata_mutex_init(&host->sender->mutex);
1098
+
1099
+ host->sender->replication_requests = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
1100
+ dictionary_register_insert_callback(host->sender->replication_requests, replication_request_insert_callback, host->sender);
1101
+ dictionary_register_conflict_callback(host->sender->replication_requests, replication_request_conflict_callback, host->sender);
1102
+ dictionary_register_delete_callback(host->sender->replication_requests, replication_request_delete_callback, host->sender);
1103
+}
1104
+
1105
+static size_t sender_buffer_used_percent(struct sender_state *s) {
1106
+ netdata_mutex_lock(&s->mutex);
1107
+ size_t available = cbuffer_available_size_unsafe(s->host->sender->buffer);
1108
+ netdata_mutex_unlock(&s->mutex);
1109
+
1110
+ return (s->host->sender->buffer->max_size - available) * 100 / s->host->sender->buffer->max_size;
1111
+}
1112
+
1113
+static void process_replication_requests(struct sender_state *s) {
1114
+ if(dictionary_entries(s->replication_requests) == 0)
1115
+ return;
1116
+
1117
+ struct replication_request *rr;
1118
+ dfe_start_write(s->replication_requests, rr) {
1119
+ size_t used_percent = sender_buffer_used_percent(s);
1120
+ if(used_percent > 50) break;
1121
+
1122
+ // delete it from the dictionary
1123
+ // the current item is referenced - it will not go away until the next iteration of the dfe loop
1124
+ dictionary_del(s->replication_requests, rr_dfe.name);
1125
+
1126
+ // find the chart
1127
+ RRDSET *st = rrdset_find(s->host, rr_dfe.name);
1128
+ if(unlikely(!st)) {
1129
+ internal_error(true,
1130
+ "STREAM %s [send to %s]: cannot find chart '%s' to satisfy pending replication command."
1131
+ , rrdhost_hostname(s->host), s->connected_to, rr_dfe.name);
1132
+ continue;
1133
+ }
1134
+
1135
+ // send the replication data
1136
+ bool start_streaming = replicate_chart_response(st->rrdhost, st,
1137
+ rr->start_streaming, rr->after, rr->before);
1138
+
1139
+ // enable normal streaming if we have to
1140
+ if (start_streaming) {
1141
+ debug(D_REPLICATION, "Enabling metric streaming for chart %s.%s",
1142
+ rrdhost_hostname(s->host), rrdset_id(st));
1143
+
1144
+ rrdset_flag_set(st, RRDSET_FLAG_SENDER_REPLICATION_FINISHED);
1145
+ }
1146
+ }
1147
+ dfe_done(rr);
1148
}
1149
1150
void *rrdpush_sender_thread(void *ptr) {
@@ -1127,7 +1229,6 @@ void *rrdpush_sender_thread(void *ptr) {
1229
thread_data->pipe_buffer = mallocz(pipe_buffer_size);
1230
thread_data->sender_state = s;
1231
thread_data->host = s->host;
1130
- thread_data->sending_definitions_status = SENDING_DEFINITIONS_RESTART;
1232
1233
// reset our cleanup flags
1234
rrdhost_flag_clear(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_JOIN);
@@ -1141,7 +1242,6 @@ void *rrdpush_sender_thread(void *ptr) {
1242
// The connection attempt blocks (after which we use the socket in nonblocking)
1243
if(unlikely(s->rrdpush_sender_socket == -1)) {
1244
worker_is_busy(WORKER_SENDER_JOB_CONNECT);
1144
- thread_data->sending_definitions_status = SENDING_DEFINITIONS_RESTART;
1245
rrdhost_flag_clear(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS);
1246
s->flags &= ~SENDER_FLAG_OVERFLOW;
1247
s->read_len = 0;
@@ -1151,13 +1251,6 @@ void *rrdpush_sender_thread(void *ptr) {
1251
if(unlikely(!attempt_to_connect(s)))
1252
continue;
1253
1154
- if (stream_has_capability(s, STREAM_CAP_GAP_FILLING)) {
1155
- time_t now = now_realtime_sec();
1156
- BUFFER *wb = sender_start(s);
1157
- buffer_sprintf(wb, "TIMESTAMP %"PRId64"", (int64_t)now);
1158
- sender_commit(s, wb);
1159
- }
1160
-
1254
rrdpush_claimed_id(s->host);
1255
rrdpush_send_host_labels(s->host);
1256
@@ -1178,9 +1271,6 @@ void *rrdpush_sender_thread(void *ptr) {
1271
continue;
1272
}
1273
1181
- if(unlikely(thread_data->sending_definitions_status != SENDING_DEFINITIONS_DONE))
1182
- rrdpush_queue_incremental_definitions(thread_data);
1183
-
1274
netdata_mutex_lock(&s->mutex);
1275
size_t outstanding = cbuffer_next_unsafe(s->host->sender->buffer, NULL);
1276
size_t available = cbuffer_available_size_unsafe(s->host->sender->buffer);
@@ -1191,10 +1281,8 @@ void *rrdpush_sender_thread(void *ptr) {
1281
if(outstanding)
1282
s->send_attempts++;
1283
else {
1194
- if(unlikely(thread_data->sending_definitions_status == SENDING_DEFINITIONS_DONE
1195
- && rrdhost_flag_check(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED)
1196
- && !rrdhost_flag_check(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS)
1197
- )) {
1284
+ if(unlikely(rrdhost_flag_check(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED) &&
1285
+ !rrdhost_flag_check(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS))) {
1286
// let the data collection threads know we are ready to push metrics
1287
rrdhost_flag_set(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS);
1288
info("STREAM %s [send to %s]: enabling metrics streaming...", rrdhost_hostname(s->host), s->connected_to);
@@ -1287,6 +1375,8 @@ void *rrdpush_sender_thread(void *ptr) {
1375
execute_commands(s);
1376
}
1377
1378
+ process_replication_requests(s);
1379
+
1380
if(unlikely(fds[Collector].revents & (POLLERR|POLLHUP|POLLNVAL))) {
1381
char *error = NULL;
1382
streaming/stream.conf
+32
-21
@@ -33,24 +33,19 @@
33
destination =
34
35
# Skip Certificate verification?
36
- #
36
# The netdata child is configurated to avoid invalid SSL/TLS certificate,
37
# so certificates that are self-signed or expired will stop the streaming.
38
# Case the server certificate is not valid, you can enable the use of
39
# 'bad' certificates setting the next option as 'yes'.
41
- #
40
#ssl skip certificate verification = yes
41
42
# Certificate Authority Path
45
- #
43
# OpenSSL has a default directory where the known certificates are stored,
44
# case it is necessary it is possible to change this rule using the variable
45
# "CApath"
49
- #
46
#CApath = /etc/ssl/certs/
47
48
# Certificate Authority file
53
- #
49
# When the Netdata parent has certificate, that is not recognized as valid,
50
# we can add this certificate in the list of known certificates in CApath
51
# and give for Netdata as argument.
@@ -61,8 +56,7 @@
56
api key =
57
58
# Stream Compression
64
- #
65
- # The netdata child is configurated to enable stream compression by default.
59
+ # The default is enabled
60
# You can control stream compression in this agent with options: yes | no
61
#enable compression = yes
62
@@ -91,6 +85,7 @@
85
reconnect delay seconds = 5
86
87
# Sync the clock of the charts for that many iterations, when starting.
88
+ # It is ignored when replication is enabled
89
initial clock resync iterations = 60
90
91
# -----------------------------------------------------------------------------
@@ -127,9 +122,8 @@
122
123
# The default history in entries, for all hosts using this API key.
124
# You can also set it per host below.
130
- # If you don't set it here, the history size of the central netdata
131
- # will be used.
132
- default history = 3600
125
+ # For the default db mode (dbengine), this is ignored.
126
+ #default history = 3600
127
128
# The default memory mode to be used for all hosts using this API key.
129
# You can also set it per host below.
@@ -140,7 +134,7 @@
134
# ram keep it in RAM, don't touch the disk
135
# none no database at all (use this on headless proxies)
136
# dbengine like a traditional database
143
- default memory mode = ram
137
+ #default memory mode = dbengine
138
139
# Shall we enable health monitoring for the hosts using this API key?
140
# 3 possible values:
@@ -150,7 +144,7 @@
144
# ensure that the netdata process on the child is gracefully stopped, to prevent invalid last_collected alarms
145
# You can also set it per host, below.
146
# The default is taken from [health].enabled of netdata.conf
153
- health enabled by default = auto
147
+ #health enabled by default = auto
148
149
# postpone alarms for a short period after the sender is connected
150
default postpone alarms on connect seconds = 60
@@ -163,11 +157,19 @@
157
#default proxy send charts matching = *
158
159
# Stream Compression
166
- #
167
- # The stream with the child can be configurated to enable stream compression.
160
+ # By default it is enabled.
161
# You can control stream compression in this parent agent stream with options: yes | no
162
#enable compression = yes
163
164
+ # Replication
165
+ # Enable replication for all hosts using this api key. Default: enabled
166
+ #enable replication = yes
167
+
168
+ # How many seconds to replicate from each child. Default: a day
169
+ #seconds to replicate = 86400
170
+
171
+ # The duration we want to replicate per each step.
172
+ #replication_step = 600
173
174
# -----------------------------------------------------------------------------
175
# 3. PER SENDING HOST SETTINGS, ON PARENT NETDATA
@@ -197,14 +199,15 @@
199
# and at stream.conf [API_KEY].allow from
200
allow from = *
201
200
- # The number of entries in the database
201
- history = 3600
202
+ # The number of entries in the database.
203
+ # This is ignored for db mode dbengine.
204
+ #history = 3600
205
206
# The memory mode of the database: save | map | ram | none | dbengine
204
- memory mode = save
207
+ #memory mode = dbengine
208
209
# Health / alarms control: yes | no | auto
207
- health enabled = yes
210
+ #health enabled = auto
211
212
# postpone alarms when the sender connects
213
postpone alarms on connect seconds = 60
@@ -217,8 +220,16 @@
220
#proxy send charts matching = *
221
222
# Stream Compression
220
- #
221
- # The stream with the child can be configurated to enable stream compression.
223
+ # By default, enabled.
224
# You can control stream compression in this parent agent stream with options: yes | no
225
#enable compression = yes
224
-
\ No newline at end of file
226
+
227
+ # Replication
228
+ # Enable replication for all hosts using this api key.
229
+ #enable replication = yes
230
+
231
+ # How many seconds to replicate from each child.
232
+ #seconds to replicate = 86400
233
+
234
+ # The duration we want to replicate per each step.
235
+ #replication_step = 600