@cryptotaxi247 / netdata-1 / commits / 3e508c8f9

New logging layer (#16357)

* cleanup of logging - wip * first working iteration * add errno annotator * replace old logging functions with netdata_logger() * cleanup * update error_limit * fix remanining error_limit references * work on fatal() * started working on structured logs * full cleanup * default logging to files; fix all plugins initialization * fix formatting of numbers * cleanup and reorg * fix coverity issues * cleanup obsolete code * fix formatting of numbers * fix log rotation * fix for older systems * add detection of systemd journal via stderr * finished on access.log * remove left-over transport * do not add empty fields to the logs * journal get compact uuids; X-Transaction-ID header is added in web responses * allow compiling on systems without memfd sealing * added libnetdata/uuid directory * move datetime formatters to libnetdata * add missing files * link the makefiles in libnetdata * added uuid_parse_flexi() to parse UUIDs with and without hyphens; the web server now read X-Transaction-ID and uses it for functions and web responses * added stream receiver, sender, proc plugin and pluginsd log stack * iso8601 advanced usage; line_splitter module in libnetdata; code cleanup * add message ids to streaming inbound and outbound connections * cleanup line_splitter between lines to avoid logging garbage; when killing children, kill them with SIGABRT if internal checks is enabled * send SIGABRT to external plugins only if we are not shutting down * fix cross cleanup in pluginsd parser * fatal when there is a stack error in logs * compile netdata with -fexceptions * do not kill external plugins with SIGABRT * metasync info logs to debug level * added severity to logs * added json output; added options per log output; added documentation; fixed issues mentioned * allow memfd only on linux * moved journal low level functions to journal.c/h * move health logs to daemon.log with proper priorities * fixed a couple of bugs; health log in journal * updated docs * systemd-cat-native command to push structured logs to journal from the command line * fix makefiles * restored NETDATA_LOG_SEVERITY_LEVEL * fix makefiles * systemd-cat-native can also work as the logger of Netdata scripts * do not require a socket to systemd-journal to log-as-netdata * alarm notify logs in native format * properly compare log ids * fatals log alerts; alarm-notify.sh working * fix overflow warning * alarm-notify.sh now logs the request (command line) * anotate external plugins logs with the function cmd they run * added context, component and type to alarm-notify.sh; shell sanitization removes control character and characters that may be expanded by bash * reformatted alarm-notify logs * unify cgroup-network-helper.sh * added quotes around params * charts.d.plugin switched logging to journal native * quotes for logfmt * unify the status codes of streaming receivers and senders * alarm-notify: dont log anything, if there is nothing to do * all external plugins log to stderr when running outside netdata; alarm-notify now shows an error when notifications menthod are needed but are not available * migrate cgroup-name.sh to new logging * systemd-cat-native now supports messages with newlines * socket.c logs use priority * cleanup log field types * inherit the systemd set INVOCATION_ID if found * allow systemd-cat-native to send messages to a systemd-journal-remote URL * log2journal command that can convert structured logs to journal export format * various fixes and documentation of log2journal * updated log2journal docs * updated log2journal docs * updated documentation of fields * allow compiling without libcurl * do not use socket as format string * added version information to newly added tools * updated documentation and help messages * fix the namespace socket path * print errno with error * do not timeout * updated docs * updated docs * updated docs * log2journal updated docs and params * when talking to a remote journal, systemd-cat-native batches the messages * enable lz4 compression for systemd-cat-native when sending messages to a systemd-journal-remote * Revert "enable lz4 compression for systemd-cat-native when sending messages to a systemd-journal-remote" This reverts commit b079d53c11f6687cd64d804fdd7b24c0492bf245. * note about uncompressed traffic * log2journal: code reorg and cleanup to make modular * finished rewriting log2journal * more comments * rewriting rules support * increased limits * updated docs * updated docs * fix old log call * use journal only when stderr is connected to journal * update netdata.spec for libcurl, libpcre2 and log2journal * pcre2-devel * do not require pcre2 in centos < 8, amazonlinux < 2023, open suse * log2journal only on systems pcre2 is available * ignore log2journal in .gitignore * avoid log2journal on centos 7, amazonlinux 2 and opensuse * add pcre2-8 to static build * undo last commit * Bundle to static Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> * Add build deps for deb packages Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> * Add dependencies; build from source Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> * Test build for amazon linux and centos expect to fail for suse Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> * fix minor oversight Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> * Reorg code * Add the install from source (deps) as a TODO * Not enable the build on suse ecosystem Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> --------- Signed-off-by: Tasos Katsoulas <tasos@netdata.cloud> Co-authored-by: Tasos Katsoulas <tasos@netdata.cloud>

Costa Tsaousis committed Nov 22, 2023 at 08:27 UTC 3e508c8f95ab0bdf8b6d74501437210d7b8d2919
120 files changed +8518 -2904
.gitignore
+2
@@ -41,6 +41,8 @@ sha256sums.txt
41 # netdata binaries
42 netdata
43 netdatacli
44 +systemd-cat-native
45 +log2journal
46 !netdata/
47 upload/
48 artifacts/
Makefile.am
+39
@@ -128,6 +128,7 @@ AM_CFLAGS = \
128 $(OPTIONAL_CUPS_CFLAGS) \
129 $(OPTIONAL_XENSTAT_CFLAGS) \
130 $(OPTIONAL_BPF_CFLAGS) \
131 + $(OPTIONAL_SYSTEMD_CFLAGS) \
132 $(OPTIONAL_GTEST_CFLAGS) \
133 $(NULL)
134
@@ -145,12 +146,18 @@ LIBNETDATA_FILES = \
146 libnetdata/avl/avl.h \
147 libnetdata/buffer/buffer.c \
148 libnetdata/buffer/buffer.h \
149 + libnetdata/buffered_reader/buffered_reader.c \
150 + libnetdata/buffered_reader/buffered_reader.h \
151 libnetdata/circular_buffer/circular_buffer.c \
152 libnetdata/circular_buffer/circular_buffer.h \
153 libnetdata/clocks/clocks.c \
154 libnetdata/clocks/clocks.h \
155 libnetdata/completion/completion.c \
156 libnetdata/completion/completion.h \
157 + libnetdata/datetime/iso8601.c \
158 + libnetdata/datetime/iso8601.h \
159 + libnetdata/datetime/rfc7231.c \
160 + libnetdata/datetime/rfc7231.h \
161 libnetdata/dictionary/dictionary.c \
162 libnetdata/dictionary/dictionary.h \
163 libnetdata/eval/eval.c \
@@ -167,8 +174,12 @@ LIBNETDATA_FILES = \
174 libnetdata/libnetdata.c \
175 libnetdata/libnetdata.h \
176 libnetdata/required_dummies.h \
177 + libnetdata/line_splitter/line_splitter.c \
178 + libnetdata/line_splitter/line_splitter.h \
179 libnetdata/locks/locks.c \
180 libnetdata/locks/locks.h \
181 + libnetdata/log/journal.c \
182 + libnetdata/log/journal.h \
183 libnetdata/log/log.c \
184 libnetdata/log/log.h \
185 libnetdata/onewayalloc/onewayalloc.c \
@@ -195,6 +206,8 @@ LIBNETDATA_FILES = \
206 libnetdata/threads/threads.h \
207 libnetdata/url/url.c \
208 libnetdata/url/url.h \
209 + libnetdata/uuid/uuid.c \
210 + libnetdata/uuid/uuid.h \
211 libnetdata/json/json.c \
212 libnetdata/json/json.h \
213 libnetdata/json/jsmn.c \
@@ -323,6 +336,16 @@ SYSTEMD_JOURNAL_PLUGIN_FILES = \
336 $(LIBNETDATA_FILES) \
337 $(NULL)
338
339 +SYSTEMD_CAT_NATIVE_FILES = \
340 + libnetdata/log/systemd-cat-native.c \
341 + libnetdata/log/systemd-cat-native.h \
342 + $(LIBNETDATA_FILES) \
343 + $(NULL)
344 +
345 +LOG2JOURNAL_FILES = \
346 + libnetdata/log/log2journal.c \
347 + $(NULL)
348 +
349 CUPS_PLUGIN_FILES = \
350 collectors/cups.plugin/cups_plugin.c \
351 $(LIBNETDATA_FILES) \
@@ -1179,6 +1202,7 @@ NETDATA_COMMON_LIBS = \
1202 $(OPTIONAL_MQTT_LIBS) \
1203 $(OPTIONAL_UV_LIBS) \
1204 $(OPTIONAL_LZ4_LIBS) \
1205 + $(OPTIONAL_CURL_LIBS) \
1206 $(OPTIONAL_ZSTD_LIBS) \
1207 $(OPTIONAL_BROTLIENC_LIBS) \
1208 $(OPTIONAL_BROTLIDEC_LIBS) \
@@ -1190,6 +1214,7 @@ NETDATA_COMMON_LIBS = \
1214 $(OPTIONAL_YAML_LIBS) \
1215 $(OPTIONAL_ATOMIC_LIBS) \
1216 $(OPTIONAL_DL_LIBS) \
1217 + $(OPTIONAL_SYSTEMD_LIBS) \
1218 $(OPTIONAL_GTEST_LIBS) \
1219 $(NULL)
1220
@@ -1290,6 +1315,14 @@ if ENABLE_PLUGIN_FREEIPMI
1315 $(NULL)
1316 endif
1317
1318 +if ENABLE_LOG2JOURNAL
1319 + sbin_PROGRAMS += log2journal
1320 + log2journal_SOURCES = $(LOG2JOURNAL_FILES)
1321 + log2journal_LDADD = \
1322 + $(OPTIONAL_PCRE2_LIBS) \
1323 + $(NULL)
1324 +endif
1325 +
1326 if ENABLE_PLUGIN_SYSTEMD_JOURNAL
1327 plugins_PROGRAMS += systemd-journal.plugin
1328 systemd_journal_plugin_SOURCES = $(SYSTEMD_JOURNAL_PLUGIN_FILES)
@@ -1299,6 +1332,12 @@ if ENABLE_PLUGIN_SYSTEMD_JOURNAL
1332 $(NULL)
1333 endif
1334
1335 +sbin_PROGRAMS += systemd-cat-native
1336 +systemd_cat_native_SOURCES = $(SYSTEMD_CAT_NATIVE_FILES)
1337 +systemd_cat_native_LDADD = \
1338 + $(NETDATA_COMMON_LIBS) \
1339 + $(NULL)
1340 +
1341 if ENABLE_PLUGIN_EBPF
1342 plugins_PROGRAMS += ebpf.plugin
1343 ebpf_plugin_SOURCES = $(EBPF_PLUGIN_FILES)
aclk/aclk.c
+62 -28
@@ -154,7 +154,9 @@ biofailed:
154
155 static int wait_till_cloud_enabled()
156 {
157 - netdata_log_info("Waiting for Cloud to be enabled");
157 + nd_log(NDLS_DAEMON, NDLP_INFO,
158 + "Waiting for Cloud to be enabled");
159 +
160 while (!netdata_cloud_enabled) {
161 sleep_usec(USEC_PER_SEC * 1);
162 if (!service_running(SERVICE_ACLK))
@@ -236,14 +238,19 @@ void aclk_mqtt_wss_log_cb(mqtt_wss_log_type_t log_type, const char* str)
238 case MQTT_WSS_LOG_WARN:
239 error_report("%s", str);
240 return;
241 +
242 case MQTT_WSS_LOG_INFO:
240 - netdata_log_info("%s", str);
243 + nd_log(NDLS_DAEMON, NDLP_INFO,
244 + "%s",
245 + str);
246 return;
247 +
248 case MQTT_WSS_LOG_DEBUG:
243 - netdata_log_debug(D_ACLK, "%s", str);
249 return;
250 +
251 default:
246 - netdata_log_error("Unknown log type from mqtt_wss");
252 + nd_log(NDLS_DAEMON, NDLP_ERR,
253 + "Unknown log type from mqtt_wss");
254 }
255 }
256
@@ -297,7 +304,9 @@ static void puback_callback(uint16_t packet_id)
304 #endif
305
306 if (aclk_shared_state.mqtt_shutdown_msg_id == (int)packet_id) {
300 - netdata_log_info("Shutdown message has been acknowledged by the cloud. Exiting gracefully");
307 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
308 + "Shutdown message has been acknowledged by the cloud. Exiting gracefully");
309 +
310 aclk_shared_state.mqtt_shutdown_msg_rcvd = 1;
311 }
312 }
@@ -335,9 +344,11 @@ static int handle_connection(mqtt_wss_client client)
344 }
345
346 if (disconnect_req || aclk_kill_link) {
338 - netdata_log_info("Going to restart connection due to disconnect_req=%s (cloud req), aclk_kill_link=%s (reclaim)",
339 - disconnect_req ? "true" : "false",
340 - aclk_kill_link ? "true" : "false");
347 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
348 + "Going to restart connection due to disconnect_req=%s (cloud req), aclk_kill_link=%s (reclaim)",
349 + disconnect_req ? "true" : "false",
350 + aclk_kill_link ? "true" : "false");
351 +
352 disconnect_req = 0;
353 aclk_kill_link = 0;
354 aclk_graceful_disconnect(client);
@@ -390,7 +401,9 @@ static inline void mqtt_connected_actions(mqtt_wss_client client)
401
402 void aclk_graceful_disconnect(mqtt_wss_client client)
403 {
393 - netdata_log_info("Preparing to gracefully shutdown ACLK connection");
404 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
405 + "Preparing to gracefully shutdown ACLK connection");
406 +
407 aclk_queue_lock();
408 aclk_queue_flush();
409
@@ -403,17 +416,22 @@ void aclk_graceful_disconnect(mqtt_wss_client client)
416 break;
417 }
418 if (aclk_shared_state.mqtt_shutdown_msg_rcvd) {
406 - netdata_log_info("MQTT App Layer `disconnect` message sent successfully");
419 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
420 + "MQTT App Layer `disconnect` message sent successfully");
421 break;
422 }
423 }
410 - netdata_log_info("ACLK link is down");
411 - netdata_log_access("ACLK DISCONNECTED");
424 +
425 + nd_log(NDLS_DAEMON, NDLP_WARNING, "ACLK link is down");
426 + nd_log(NDLS_ACCESS, NDLP_WARNING, "ACLK DISCONNECTED");
427 +
428 aclk_stats_upd_online(0);
429 last_disconnect_time = now_realtime_sec();
430 aclk_connected = 0;
431
416 - netdata_log_info("Attempting to gracefully shutdown the MQTT/WSS connection");
432 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
433 + "Attempting to gracefully shutdown the MQTT/WSS connection");
434 +
435 mqtt_wss_disconnect(client, 1000);
436 }
437
@@ -455,7 +473,9 @@ static int aclk_block_till_recon_allowed() {
473 next_connection_attempt = now_realtime_sec() + (recon_delay / MSEC_PER_SEC);
474 last_backoff_value = (float)recon_delay / MSEC_PER_SEC;
475
458 - netdata_log_info("Wait before attempting to reconnect in %.3f seconds", recon_delay / (float)MSEC_PER_SEC);
476 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
477 + "Wait before attempting to reconnect in %.3f seconds", recon_delay / (float)MSEC_PER_SEC);
478 +
479 // we want to wake up from time to time to check netdata_exit
480 while (recon_delay)
481 {
@@ -593,7 +613,9 @@ static int aclk_attempt_to_connect(mqtt_wss_client client)
613 return 1;
614 }
615
596 - netdata_log_info("Attempting connection now");
616 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
617 + "Attempting connection now");
618 +
619 memset(&base_url, 0, sizeof(url_t));
620 if (url_parse(aclk_cloud_base_url, &base_url)) {
621 aclk_status = ACLK_STATUS_INVALID_CLOUD_URL;
@@ -680,7 +702,9 @@ static int aclk_attempt_to_connect(mqtt_wss_client client)
702 error_report("Can't use encoding=proto without at least \"proto\" capability.");
703 continue;
704 }
683 - netdata_log_info("New ACLK protobuf protocol negotiated successfully (/env response).");
705 +
706 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
707 + "New ACLK protobuf protocol negotiated successfully (/env response).");
708
709 memset(&auth_url, 0, sizeof(url_t));
710 if (url_parse(aclk_env->auth_endpoint, &auth_url)) {
@@ -750,9 +774,9 @@ static int aclk_attempt_to_connect(mqtt_wss_client client)
774
775 if (!ret) {
776 last_conn_time_mqtt = now_realtime_sec();
753 - netdata_log_info("ACLK connection successfully established");
777 + nd_log(NDLS_DAEMON, NDLP_INFO, "ACLK connection successfully established");
778 aclk_status = ACLK_STATUS_CONNECTED;
755 - netdata_log_access("ACLK CONNECTED");
779 + nd_log(NDLS_ACCESS, NDLP_INFO, "ACLK CONNECTED");
780 mqtt_connected_actions(client);
781 return 0;
782 }
@@ -798,7 +822,9 @@ void *aclk_main(void *ptr)
822 netdata_thread_disable_cancelability();
823
824 #if defined( DISABLE_CLOUD ) || !defined( ENABLE_ACLK )
801 - netdata_log_info("Killing ACLK thread -> cloud functionality has been disabled");
825 + nd_log(NDLS_DAEMON, NDLP_INFO,
826 + "Killing ACLK thread -> cloud functionality has been disabled");
827 +
828 static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
829 return NULL;
830 #endif
@@ -857,7 +883,7 @@ void *aclk_main(void *ptr)
883 aclk_stats_upd_online(0);
884 last_disconnect_time = now_realtime_sec();
885 aclk_connected = 0;
860 - netdata_log_access("ACLK DISCONNECTED");
886 + nd_log(NDLS_ACCESS, NDLP_WARNING, "ACLK DISCONNECTED");
887 }
888 } while (service_running(SERVICE_ACLK));
889
@@ -924,7 +950,9 @@ void aclk_host_state_update(RRDHOST *host, int cmd)
950 rrdhost_aclk_state_unlock(localhost);
951 create_query->data.bin_payload.topic = ACLK_TOPICID_CREATE_NODE;
952 create_query->data.bin_payload.msg_name = "CreateNodeInstance";
927 - netdata_log_info("Registering host=%s, hops=%u", host->machine_guid, host->system_info->hops);
953 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
954 + "Registering host=%s, hops=%u", host->machine_guid, host->system_info->hops);
955 +
956 aclk_queue_query(create_query);
957 return;
958 }
@@ -947,8 +975,10 @@ void aclk_host_state_update(RRDHOST *host, int cmd)
975 query->data.bin_payload.payload = generate_node_instance_connection(&query->data.bin_payload.size, &node_state_update);
976 rrdhost_aclk_state_unlock(localhost);
977
950 - netdata_log_info("Queuing status update for node=%s, live=%d, hops=%u",(char*)node_state_update.node_id, cmd,
951 - host->system_info->hops);
978 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
979 + "Queuing status update for node=%s, live=%d, hops=%u",
980 + (char*)node_state_update.node_id, cmd, host->system_info->hops);
981 +
982 freez((void*)node_state_update.node_id);
983 query->data.bin_payload.msg_name = "UpdateNodeInstanceConnection";
984 query->data.bin_payload.topic = ACLK_TOPICID_NODE_CONN;
@@ -990,9 +1020,10 @@ void aclk_send_node_instances()
1020 node_state_update.claim_id = localhost->aclk_state.claimed_id;
1021 query->data.bin_payload.payload = generate_node_instance_connection(&query->data.bin_payload.size, &node_state_update);
1022 rrdhost_aclk_state_unlock(localhost);
993 - netdata_log_info("Queuing status update for node=%s, live=%d, hops=%d",(char*)node_state_update.node_id,
994 - list->live,
995 - list->hops);
1023 +
1024 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1025 + "Queuing status update for node=%s, live=%d, hops=%d",
1026 + (char*)node_state_update.node_id, list->live, list->hops);
1027
1028 freez((void*)node_state_update.capabilities);
1029 freez((void*)node_state_update.node_id);
@@ -1014,8 +1045,11 @@ void aclk_send_node_instances()
1045 node_instance_creation.claim_id = localhost->aclk_state.claimed_id,
1046 create_query->data.bin_payload.payload = generate_node_instance_creation(&create_query->data.bin_payload.size, &node_instance_creation);
1047 rrdhost_aclk_state_unlock(localhost);
1017 - netdata_log_info("Queuing registration for host=%s, hops=%d",(char*)node_instance_creation.machine_guid,
1018 - list->hops);
1048 +
1049 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1050 + "Queuing registration for host=%s, hops=%d",
1051 + (char*)node_instance_creation.machine_guid, list->hops);
1052 +
1053 freez((void *)node_instance_creation.machine_guid);
1054 aclk_queue_query(create_query);
1055 }
aclk/aclk_query.c
+8 -19
@@ -90,6 +90,12 @@ static bool aclk_web_client_interrupt_cb(struct web_client *w __maybe_unused, vo
90 }
91
92 static int http_api_v2(struct aclk_query_thread *query_thr, aclk_query_t query) {
93 + ND_LOG_STACK lgs[] = {
94 + ND_LOG_FIELD_TXT(NDF_SRC_TRANSPORT, "aclk"),
95 + ND_LOG_FIELD_END(),
96 + };
97 + ND_LOG_STACK_PUSH(lgs);
98 +
99 int retval = 0;
100 BUFFER *local_buffer = NULL;
101 size_t size = 0;
@@ -110,7 +116,7 @@ static int http_api_v2(struct aclk_query_thread *query_thr, aclk_query_t query)
116 usec_t t;
117 web_client_timeout_checkpoint_set(w, query->timeout);
118 if(web_client_timeout_checkpoint_and_check(w, &t)) {
113 - netdata_log_access("QUERY CANCELED: QUEUE TIME EXCEEDED %llu ms (LIMIT %d ms)", t / USEC_PER_MS, query->timeout);
119 + nd_log(NDLS_ACCESS, NDLP_ERR, "QUERY CANCELED: QUEUE TIME EXCEEDED %llu ms (LIMIT %d ms)", t / USEC_PER_MS, query->timeout);
120 retval = 1;
121 w->response.code = HTTP_RESP_SERVICE_UNAVAILABLE;
122 aclk_http_msg_v2_err(query_thr->client, query->callback_topic, query->msg_id, w->response.code, CLOUD_EC_SND_TIMEOUT, CLOUD_EMSG_SND_TIMEOUT, NULL, 0);
@@ -217,25 +223,8 @@ static int http_api_v2(struct aclk_query_thread *query_thr, aclk_query_t query)
223 // send msg.
224 w->response.code = aclk_http_msg_v2(query_thr->client, query->callback_topic, query->msg_id, t, query->created, w->response.code, local_buffer->buffer, local_buffer->len);
225
220 - struct timeval tv;
221 -
226 cleanup:
223 - now_monotonic_high_precision_timeval(&tv);
224 - netdata_log_access("%llu: %d '[ACLK]:%d' '%s' (sent/all = %zu/%zu bytes %0.0f%%, prep/sent/total = %0.2f/%0.2f/%0.2f ms) %d '%s'",
225 - w->id
226 - , gettid()
227 - , query_thr->idx
228 - , "DATA"
229 - , sent
230 - , size
231 - , size > sent ? -(((size - sent) / (double)size) * 100.0) : ((size > 0) ? (((sent - size ) / (double)size) * 100.0) : 0.0)
232 - , dt_usec(&w->timings.tv_ready, &w->timings.tv_in) / 1000.0
233 - , dt_usec(&tv, &w->timings.tv_ready) / 1000.0
234 - , dt_usec(&tv, &w->timings.tv_in) / 1000.0
235 - , w->response.code
236 - , strip_control_characters((char *)buffer_tostring(w->url_as_received))
237 - );
238 -
227 + web_client_log_completed_request(w, false);
228 web_client_release_to_cache(w);
229
230 pending_req_list_rm(query->msg_id);
aclk/aclk_rx_msgs.c
+1 -1
@@ -455,7 +455,7 @@ int cancel_pending_req(const char *msg, size_t msg_len)
455 return 1;
456 }
457
458 - netdata_log_access("ACLK CancelPendingRequest REQ: %s, cloud trace-id: %s", cmd.request_id, cmd.trace_id);
458 + nd_log(NDLS_ACCESS, NDLP_NOTICE, "ACLK CancelPendingRequest REQ: %s, cloud trace-id: %s", cmd.request_id, cmd.trace_id);
459
460 if (mark_pending_req_cancelled(cmd.request_id))
461 error_report("CancelPending Request for %s failed. No such pending request.", cmd.request_id);
claim/claim.c
+2 -2
@@ -323,11 +323,11 @@ static bool check_claim_param(const char *s) {
323 }
324
325 void claim_reload_all(void) {
326 - error_log_limit_unlimited();
326 + nd_log_limits_unlimited();
327 load_claiming_state();
328 registry_update_cloud_base_url();
329 rrdpush_send_claimed_id(localhost);
330 - error_log_limit_reset();
330 + nd_log_limits_reset();
331 }
332
333 int api_v2_claim(struct web_client *w, char *url) {
cli/cli.c
+5 -12
@@ -3,25 +3,18 @@
3 #include "cli.h"
4 #include "daemon/pipename.h"
5
6 -void error_int(int is_collector __maybe_unused, const char *prefix __maybe_unused, const char *file __maybe_unused, const char *function __maybe_unused, const unsigned long line __maybe_unused, const char *fmt, ... ) {
7 - FILE *fp = stderr;
8 -
6 +void netdata_logger(ND_LOG_SOURCES source, ND_LOG_FIELD_PRIORITY priority, const char *file, const char *function, unsigned long line, const char *fmt, ... ) {
7 va_list args;
10 - va_start( args, fmt );
11 - vfprintf(fp, fmt, args );
12 - va_end( args );
8 + va_start(args, fmt);
9 + vfprintf(stderr, fmt, args );
10 + va_end(args);
11 }
12
13 #ifdef NETDATA_INTERNAL_CHECKS
14
15 uint64_t debug_flags;
16
19 -void debug_int( const char *file __maybe_unused , const char *function __maybe_unused , const unsigned long line __maybe_unused, const char *fmt __maybe_unused, ... )
20 -{
21 -
22 -}
23 -
24 -void fatal_int( const char *file __maybe_unused, const char *function __maybe_unused, const unsigned long line __maybe_unused, const char *fmt __maybe_unused, ... )
17 +void netdata_logger_fatal( const char *file __maybe_unused, const char *function __maybe_unused, const unsigned long line __maybe_unused, const char *fmt __maybe_unused, ... )
18 {
19 abort();
20 };
collectors/apps.plugin/apps_plugin.c
+1 -15
@@ -5234,25 +5234,11 @@ static void function_processes(const char *transaction, char *function __maybe_u
5234 static bool apps_plugin_exit = false;
5235
5236 int main(int argc, char **argv) {
5237 - // debug_flags = D_PROCFILE;
5238 - stderror = stderr;
5239 -
5237 clocks_init();
5238 + nd_log_initialize_for_external_plugins("apps.plugin");
5239
5240 pagesize = (size_t)sysconf(_SC_PAGESIZE);
5241
5244 - // set the name for logging
5245 - program_name = "apps.plugin";
5246 -
5247 - // disable syslog for apps.plugin
5248 - error_log_syslog = 0;
5249 -
5250 - // set errors flood protection to 100 logs per hour
5251 - error_log_errors_per_period = 100;
5252 - error_log_throttle_period = 3600;
5253 -
5254 - log_set_global_severity_for_external_plugins();
5255 -
5242 bool send_resource_usage = true;
5243 {
5244 const char *s = getenv("NETDATA_INTERNALS_MONITORING");
collectors/cgroups.plugin/cgroup-name.sh
+76 -27
@@ -12,57 +12,106 @@
12 export PATH="${PATH}:/sbin:/usr/sbin:/usr/local/sbin"
13 export LC_ALL=C
14
15 +cmd_line="'${0}' $(printf "'%s' " "${@}")"
16 +
17 # -----------------------------------------------------------------------------
18 +# logging
19
20 PROGRAM_NAME="$(basename "${0}")"
21
19 -LOG_LEVEL_ERR=1
20 -LOG_LEVEL_WARN=2
21 -LOG_LEVEL_INFO=3
22 -LOG_LEVEL="$LOG_LEVEL_INFO"
23 -
24 -set_log_severity_level() {
25 - case ${NETDATA_LOG_SEVERITY_LEVEL,,} in
26 - "info") LOG_LEVEL="$LOG_LEVEL_INFO";;
27 - "warn" | "warning") LOG_LEVEL="$LOG_LEVEL_WARN";;
28 - "err" | "error") LOG_LEVEL="$LOG_LEVEL_ERR";;
22 +# these should be the same with syslog() priorities
23 +NDLP_EMERG=0 # system is unusable
24 +NDLP_ALERT=1 # action must be taken immediately
25 +NDLP_CRIT=2 # critical conditions
26 +NDLP_ERR=3 # error conditions
27 +NDLP_WARN=4 # warning conditions
28 +NDLP_NOTICE=5 # normal but significant condition
29 +NDLP_INFO=6 # informational
30 +NDLP_DEBUG=7 # debug-level messages
31 +
32 +# the max (numerically) log level we will log
33 +LOG_LEVEL=$NDLP_INFO
34 +
35 +set_log_min_priority() {
36 + case "${NETDATA_LOG_PRIORITY_LEVEL,,}" in
37 + "emerg" | "emergency")
38 + LOG_LEVEL=$NDLP_EMERG
39 + ;;
40 +
41 + "alert")
42 + LOG_LEVEL=$NDLP_ALERT
43 + ;;
44 +
45 + "crit" | "critical")
46 + LOG_LEVEL=$NDLP_CRIT
47 + ;;
48 +
49 + "err" | "error")
50 + LOG_LEVEL=$NDLP_ERR
51 + ;;
52 +
53 + "warn" | "warning")
54 + LOG_LEVEL=$NDLP_WARN
55 + ;;
56 +
57 + "notice")
58 + LOG_LEVEL=$NDLP_NOTICE
59 + ;;
60 +
61 + "info")
62 + LOG_LEVEL=$NDLP_INFO
63 + ;;
64 +
65 + "debug")
66 + LOG_LEVEL=$NDLP_DEBUG
67 + ;;
68 esac
69 }
70
32 -set_log_severity_level
33 -
34 -logdate() {
35 - date "+%Y-%m-%d %H:%M:%S"
36 -}
71 +set_log_min_priority
72
73 log() {
39 - local status="${1}"
40 - shift
41 -
42 - echo >&2 "$(logdate): ${PROGRAM_NAME}: ${status}: ${*}"
43 -
74 + local level="${1}"
75 + shift 1
76 +
77 + [[ -n "$level" && -n "$LOG_LEVEL" && "$level" -gt "$LOG_LEVEL" ]] && return
78 +
79 + systemd-cat-native --log-as-netdata --newline="{NEWLINE}" <<EOFLOG
80 +INVOCATION_ID=${NETDATA_INVOCATION_ID}
81 +SYSLOG_IDENTIFIER=${PROGRAM_NAME}
82 +PRIORITY=${level}
83 +THREAD_TAG="cgroup-name"
84 +ND_LOG_SOURCE=collector
85 +ND_REQUEST=${cmd_line}
86 +MESSAGE=${*//[$'\r\n']/{NEWLINE}}
87 +
88 +EOFLOG
89 + # AN EMPTY LINE IS NEEDED ABOVE
90 }
91
92 info() {
47 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_INFO" -gt "$LOG_LEVEL" ]] && return
48 - log INFO "${@}"
93 + log "$NDLP_INFO" "${@}"
94 }
95
96 warning() {
52 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_WARN" -gt "$LOG_LEVEL" ]] && return
53 - log WARNING "${@}"
97 + log "$NDLP_WARN" "${@}"
98 }
99
100 error() {
57 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_ERR" -gt "$LOG_LEVEL" ]] && return
58 - log ERROR "${@}"
101 + log "$NDLP_ERR" "${@}"
102 }
103
104 fatal() {
62 - log FATAL "${@}"
105 + log "$NDLP_ALERT" "${@}"
106 exit 1
107 }
108
109 +debug() {
110 + log "$NDLP_DEBUG" "${@}"
111 +}
112 +
113 +# -----------------------------------------------------------------------------
114 +
115 function parse_docker_like_inspect_output() {
116 local output="${1}"
117 eval "$(grep -E "^(NOMAD_NAMESPACE|NOMAD_JOB_NAME|NOMAD_TASK_NAME|NOMAD_SHORT_ALLOC_ID|CONT_NAME|IMAGE_NAME)=" <<<"$output")"
collectors/cgroups.plugin/cgroup-network-helper.sh
+88 -33
@@ -29,65 +29,117 @@
29
30 export LC_ALL=C
31
32 -PROGRAM_NAME="$(basename "${0}")"
32 +cmd_line="'${0}' $(printf "'%s' " "${@}")"
33 +
34 +# -----------------------------------------------------------------------------
35 +# logging
36
34 -LOG_LEVEL_ERR=1
35 -LOG_LEVEL_WARN=2
36 -LOG_LEVEL_INFO=3
37 -LOG_LEVEL="$LOG_LEVEL_INFO"
37 +PROGRAM_NAME="$(basename "${0}")"
38
39 -set_log_severity_level() {
40 - case ${NETDATA_LOG_SEVERITY_LEVEL,,} in
41 - "info") LOG_LEVEL="$LOG_LEVEL_INFO";;
42 - "warn" | "warning") LOG_LEVEL="$LOG_LEVEL_WARN";;
43 - "err" | "error") LOG_LEVEL="$LOG_LEVEL_ERR";;
39 +# these should be the same with syslog() priorities
40 +NDLP_EMERG=0 # system is unusable
41 +NDLP_ALERT=1 # action must be taken immediately
42 +NDLP_CRIT=2 # critical conditions
43 +NDLP_ERR=3 # error conditions
44 +NDLP_WARN=4 # warning conditions
45 +NDLP_NOTICE=5 # normal but significant condition
46 +NDLP_INFO=6 # informational
47 +NDLP_DEBUG=7 # debug-level messages
48 +
49 +# the max (numerically) log level we will log
50 +LOG_LEVEL=$NDLP_INFO
51 +
52 +set_log_min_priority() {
53 + case "${NETDATA_LOG_PRIORITY_LEVEL,,}" in
54 + "emerg" | "emergency")
55 + LOG_LEVEL=$NDLP_EMERG
56 + ;;
57 +
58 + "alert")
59 + LOG_LEVEL=$NDLP_ALERT
60 + ;;
61 +
62 + "crit" | "critical")
63 + LOG_LEVEL=$NDLP_CRIT
64 + ;;
65 +
66 + "err" | "error")
67 + LOG_LEVEL=$NDLP_ERR
68 + ;;
69 +
70 + "warn" | "warning")
71 + LOG_LEVEL=$NDLP_WARN
72 + ;;
73 +
74 + "notice")
75 + LOG_LEVEL=$NDLP_NOTICE
76 + ;;
77 +
78 + "info")
79 + LOG_LEVEL=$NDLP_INFO
80 + ;;
81 +
82 + "debug")
83 + LOG_LEVEL=$NDLP_DEBUG
84 + ;;
85 esac
86 }
87
47 -set_log_severity_level
48 -
49 -logdate() {
50 - date "+%Y-%m-%d %H:%M:%S"
51 -}
88 +set_log_min_priority
89
90 log() {
54 - local status="${1}"
55 - shift
56 -
57 - echo >&2 "$(logdate): ${PROGRAM_NAME}: ${status}: ${*}"
58 -
91 + local level="${1}"
92 + shift 1
93 +
94 + [[ -n "$level" && -n "$LOG_LEVEL" && "$level" -gt "$LOG_LEVEL" ]] && return
95 +
96 + systemd-cat-native --log-as-netdata --newline="{NEWLINE}" <<EOFLOG
97 +INVOCATION_ID=${NETDATA_INVOCATION_ID}
98 +SYSLOG_IDENTIFIER=${PROGRAM_NAME}
99 +PRIORITY=${level}
100 +THREAD_TAG="cgroup-network-helper.sh"
101 +ND_LOG_SOURCE=collector
102 +ND_REQUEST=${cmd_line}
103 +MESSAGE=${*//[$'\r\n']/{NEWLINE}}
104 +
105 +EOFLOG
106 + # AN EMPTY LINE IS NEEDED ABOVE
107 }
108
109 info() {
62 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_INFO" -gt "$LOG_LEVEL" ]] && return
63 - log INFO "${@}"
110 + log "$NDLP_INFO" "${@}"
111 }
112
113 warning() {
67 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_WARN" -gt "$LOG_LEVEL" ]] && return
68 - log WARNING "${@}"
114 + log "$NDLP_WARN" "${@}"
115 }
116
117 error() {
72 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_ERR" -gt "$LOG_LEVEL" ]] && return
73 - log ERROR "${@}"
118 + log "$NDLP_ERR" "${@}"
119 }
120
121 fatal() {
77 - log FATAL "${@}"
78 - exit 1
122 + log "$NDLP_ALERT" "${@}"
123 + exit 1
124 }
125
81 -debug=${NETDATA_CGROUP_NETWORK_HELPER_DEBUG=0}
126 debug() {
83 - [ "${debug}" = "1" ] && log DEBUG "${@}"
127 + log "$NDLP_DEBUG" "${@}"
128 }
129
130 +debug=0
131 +if [ "${NETDATA_CGROUP_NETWORK_HELPER_DEBUG-0}" = "1" ]; then
132 + debug=1
133 + LOG_LEVEL=$NDLP_DEBUG
134 +fi
135 +
136 # -----------------------------------------------------------------------------
137 # check for BASH v4+ (required for associative arrays)
138
89 -[ $(( BASH_VERSINFO[0] )) -lt 4 ] && \
90 - fatal "BASH version 4 or later is required (this is ${BASH_VERSION})."
139 +if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
140 + echo >&2 "BASH version 4 or later is required (this is ${BASH_VERSION})."
141 + exit 1
142 +fi
143
144 # -----------------------------------------------------------------------------
145 # parse the arguments
@@ -99,7 +151,10 @@ do
151 case "${1}" in
152 --cgroup) cgroup="${2}"; shift 1;;
153 --pid|-p) pid="${2}"; shift 1;;
102 - --debug|debug) debug=1;;
154 + --debug|debug)
155 + debug=1
156 + LOG_LEVEL=$NDLP_DEBUG
157 + ;;
158 *) fatal "Cannot understand argument '${1}'";;
159 esac
160
collectors/cgroups.plugin/cgroup-network.c
+2 -5
@@ -649,12 +649,11 @@ void usage(void) {
649 }
650
651 int main(int argc, char **argv) {
652 - stderror = stderr;
652 pid_t pid = 0;
653
655 - program_name = argv[0];
654 program_version = VERSION;
657 - error_log_syslog = 0;
655 + clocks_init();
656 + nd_log_initialize_for_external_plugins("cgroup-network");
657
658 // since cgroup-network runs as root, prevent it from opening symbolic links
659 procfile_open_flags = O_RDONLY|O_NOFOLLOW;
@@ -687,8 +686,6 @@ int main(int argc, char **argv) {
686
687 if(argc != 3)
688 usage();
690 -
691 - log_set_global_severity_for_external_plugins();
689
690 int arg = 1;
691 int helper = 1;
collectors/charts.d.plugin/charts.d.plugin.in
+103 -46
@@ -16,24 +16,111 @@
16 export PATH="${PATH}:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin"
17
18 PROGRAM_FILE="$0"
19 -PROGRAM_NAME="$(basename $0)"
20 -PROGRAM_NAME="${PROGRAM_NAME/.plugin/}"
19 MODULE_NAME="main"
20
23 -LOG_LEVEL_ERR=1
24 -LOG_LEVEL_WARN=2
25 -LOG_LEVEL_INFO=3
26 -LOG_LEVEL="$LOG_LEVEL_INFO"
27 -
28 -set_log_severity_level() {
29 - case ${NETDATA_LOG_SEVERITY_LEVEL,,} in
30 - "info") LOG_LEVEL="$LOG_LEVEL_INFO";;
31 - "warn" | "warning") LOG_LEVEL="$LOG_LEVEL_WARN";;
32 - "err" | "error") LOG_LEVEL="$LOG_LEVEL_ERR";;
21 +# -----------------------------------------------------------------------------
22 +# logging
23 +
24 +PROGRAM_NAME="$(basename "${0}")"
25 +
26 +# these should be the same with syslog() priorities
27 +NDLP_EMERG=0 # system is unusable
28 +NDLP_ALERT=1 # action must be taken immediately
29 +NDLP_CRIT=2 # critical conditions
30 +NDLP_ERR=3 # error conditions
31 +NDLP_WARN=4 # warning conditions
32 +NDLP_NOTICE=5 # normal but significant condition
33 +NDLP_INFO=6 # informational
34 +NDLP_DEBUG=7 # debug-level messages
35 +
36 +# the max (numerically) log level we will log
37 +LOG_LEVEL=$NDLP_INFO
38 +
39 +set_log_min_priority() {
40 + case "${NETDATA_LOG_PRIORITY_LEVEL,,}" in
41 + "emerg" | "emergency")
42 + LOG_LEVEL=$NDLP_EMERG
43 + ;;
44 +
45 + "alert")
46 + LOG_LEVEL=$NDLP_ALERT
47 + ;;
48 +
49 + "crit" | "critical")
50 + LOG_LEVEL=$NDLP_CRIT
51 + ;;
52 +
53 + "err" | "error")
54 + LOG_LEVEL=$NDLP_ERR
55 + ;;
56 +
57 + "warn" | "warning")
58 + LOG_LEVEL=$NDLP_WARN
59 + ;;
60 +
61 + "notice")
62 + LOG_LEVEL=$NDLP_NOTICE
63 + ;;
64 +
65 + "info")
66 + LOG_LEVEL=$NDLP_INFO
67 + ;;
68 +
69 + "debug")
70 + LOG_LEVEL=$NDLP_DEBUG
71 + ;;
72 esac
73 }
74
36 -set_log_severity_level
75 +set_log_min_priority
76 +
77 +log() {
78 + local level="${1}"
79 + shift 1
80 +
81 + [[ -n "$level" && -n "$LOG_LEVEL" && "$level" -gt "$LOG_LEVEL" ]] && return
82 +
83 + systemd-cat-native --log-as-netdata <<EOFLOG
84 +INVOCATION_ID=${NETDATA_INVOCATION_ID}
85 +SYSLOG_IDENTIFIER=${PROGRAM_NAME}
86 +PRIORITY=${level}
87 +THREAD_TAG="charts.d.plugin"
88 +ND_LOG_SOURCE=collector
89 +MESSAGE=${MODULE_NAME}: ${*//[$'\r\n']}
90 +
91 +EOFLOG
92 + # AN EMPTY LINE IS NEEDED ABOVE
93 +}
94 +
95 +info() {
96 + log "$NDLP_INFO" "${@}"
97 +}
98 +
99 +warning() {
100 + log "$NDLP_WARN" "${@}"
101 +}
102 +
103 +error() {
104 + log "$NDLP_ERR" "${@}"
105 +}
106 +
107 +fatal() {
108 + log "$NDLP_ALERT" "${@}"
109 + echo "DISABLE"
110 + exit 1
111 +}
112 +
113 +debug() {
114 + [ "$debug" = "1" ] && log "$NDLP_DEBUG" "${@}"
115 +}
116 +
117 +# -----------------------------------------------------------------------------
118 +# check for BASH v4+ (required for associative arrays)
119 +
120 +if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
121 + echo >&2 "BASH version 4 or later is required (this is ${BASH_VERSION})."
122 + exit 1
123 +fi
124
125 # -----------------------------------------------------------------------------
126 # create temp dir
@@ -62,39 +149,6 @@ logdate() {
149 date "+%Y-%m-%d %H:%M:%S"
150 }
151
65 -log() {
66 - local status="${1}"
67 - shift
68 -
69 - echo >&2 "$(logdate): ${PROGRAM_NAME}: ${status}: ${MODULE_NAME}: ${*}"
70 -
71 -}
72 -
73 -info() {
74 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_INFO" -gt "$LOG_LEVEL" ]] && return
75 - log INFO "${@}"
76 -}
77 -
78 -warning() {
79 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_WARN" -gt "$LOG_LEVEL" ]] && return
80 - log WARNING "${@}"
81 -}
82 -
83 -error() {
84 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_ERR" -gt "$LOG_LEVEL" ]] && return
85 - log ERROR "${@}"
86 -}
87 -
88 -fatal() {
89 - log FATAL "${@}"
90 - echo "DISABLE"
91 - exit 1
92 -}
93 -
94 -debug() {
95 - [ $debug -eq 1 ] && log DEBUG "${@}"
96 -}
97 -
152 # -----------------------------------------------------------------------------
153 # check a few commands
154
@@ -194,12 +248,14 @@ while [ ! -z "$1" ]; do
248
249 if [ "$1" = "debug" -o "$1" = "all" ]; then
250 debug=1
251 + LOG_LEVEL=$NDLP_DEBUG
252 shift
253 continue
254 fi
255
256 if [ -f "$chartsd/$1.chart.sh" ]; then
257 debug=1
258 + LOG_LEVEL=$NDLP_DEBUG
259 chart_only="$(echo $1.chart.sh | sed "s/\.chart\.sh$//g")"
260 shift
261 continue
@@ -207,6 +263,7 @@ while [ ! -z "$1" ]; do
263
264 if [ -f "$chartsd/$1" ]; then
265 debug=1
266 + LOG_LEVEL=$NDLP_DEBUG
267 chart_only="$(echo $1 | sed "s/\.chart\.sh$//g")"
268 shift
269 continue
collectors/cups.plugin/cups_plugin.c
+1 -15
@@ -226,22 +226,8 @@ void reset_metrics() {
226 }
227
228 int main(int argc, char **argv) {
229 - stderror = stderr;
229 clocks_init();
231 -
232 - // ------------------------------------------------------------------------
233 - // initialization of netdata plugin
234 -
235 - program_name = "cups.plugin";
236 -
237 - // disable syslog
238 - error_log_syslog = 0;
239 -
240 - // set errors flood protection to 100 logs per hour
241 - error_log_errors_per_period = 100;
242 - error_log_throttle_period = 3600;
243 -
244 - log_set_global_severity_for_external_plugins();
230 + nd_log_initialize_for_external_plugins("cups.plugin");
231
232 parse_command_line(argc, argv);
233
collectors/debugfs.plugin/debugfs_plugin.c
+2 -10
@@ -159,16 +159,8 @@ static void debugfs_parse_args(int argc, char **argv)
159
160 int main(int argc, char **argv)
161 {
162 - // debug_flags = D_PROCFILE;
163 - stderror = stderr;
164 -
165 - // set the name for logging
166 - program_name = "debugfs.plugin";
167 -
168 - // disable syslog for debugfs.plugin
169 - error_log_syslog = 0;
170 -
171 - log_set_global_severity_for_external_plugins();
162 + clocks_init();
163 + nd_log_initialize_for_external_plugins("debugfs.plugin");
164
165 netdata_configured_host_prefix = getenv("NETDATA_HOST_PREFIX");
166 if (verify_netdata_host_prefix() == -1)
collectors/ebpf.plugin/ebpf.c
+2 -14
@@ -4024,11 +4024,9 @@ static void ebpf_manage_pid(pid_t pid)
4024 */
4025 int main(int argc, char **argv)
4026 {
4027 - stderror = stderr;
4028 -
4029 - log_set_global_severity_for_external_plugins();
4030 -
4027 clocks_init();
4028 + nd_log_initialize_for_external_plugins("ebpf.plugin");
4029 +
4030 main_thread_id = gettid();
4031
4032 set_global_variables();
@@ -4038,16 +4036,6 @@ int main(int argc, char **argv)
4036 if (ebpf_check_conditions())
4037 return 2;
4038
4041 - // set name
4042 - program_name = "ebpf.plugin";
4043 -
4044 - // disable syslog
4045 - error_log_syslog = 0;
4046 -
4047 - // set errors flood protection to 100 logs per hour
4048 - error_log_errors_per_period = 100;
4049 - error_log_throttle_period = 3600;
4050 -
4039 if (ebpf_adjust_memory_limit())
4040 return 3;
4041
collectors/freeipmi.plugin/freeipmi_plugin.c
+4 -20
@@ -1622,29 +1622,13 @@ static void plugin_exit(int code) {
1622 }
1623
1624 int main (int argc, char **argv) {
1625 - bool netdata_do_sel = IPMI_ENABLE_SEL_BY_DEFAULT;
1626 -
1627 - stderror = stderr;
1625 clocks_init();
1626 + nd_log_initialize_for_external_plugins("freeipmi.plugin");
1627 + netdata_threads_init_for_external_plugins(0); // set the default threads stack size here
1628
1630 - bool debug = false;
1631 -
1632 - // ------------------------------------------------------------------------
1633 - // initialization of netdata plugin
1634 -
1635 - program_name = "freeipmi.plugin";
1636 -
1637 - // disable syslog
1638 - error_log_syslog = 0;
1639 -
1640 - // set errors flood protection to 100 logs per hour
1641 - error_log_errors_per_period = 100;
1642 - error_log_throttle_period = 3600;
1643 -
1644 - log_set_global_severity_for_external_plugins();
1629 + bool netdata_do_sel = IPMI_ENABLE_SEL_BY_DEFAULT;
1630
1646 - // initialize the threads
1647 - netdata_threads_init_for_external_plugins(0); // set the default threads stack size here
1631 + bool debug = false;
1632
1633 // ------------------------------------------------------------------------
1634 // parse command line parameters
collectors/nfacct.plugin/plugin_nfacct.c
+1 -15
@@ -747,22 +747,8 @@ void nfacct_signals()
747 }
748
749 int main(int argc, char **argv) {
750 - stderror = stderr;
750 clocks_init();
752 -
753 - // ------------------------------------------------------------------------
754 - // initialization of netdata plugin
755 -
756 - program_name = "nfacct.plugin";
757 -
758 - // disable syslog
759 - error_log_syslog = 0;
760 -
761 - // set errors flood protection to 100 logs per hour
762 - error_log_errors_per_period = 100;
763 - error_log_throttle_period = 3600;
764 -
765 - log_set_global_severity_for_external_plugins();
751 + nd_log_initialize_for_external_plugins("nfacct.plugin");
752
753 // ------------------------------------------------------------------------
754 // parse command line parameters
collectors/perf.plugin/perf_plugin.c
+1 -15
@@ -1283,22 +1283,8 @@ void parse_command_line(int argc, char **argv) {
1283 }
1284
1285 int main(int argc, char **argv) {
1286 - stderror = stderr;
1286 clocks_init();
1288 -
1289 - // ------------------------------------------------------------------------
1290 - // initialization of netdata plugin
1291 -
1292 - program_name = "perf.plugin";
1293 -
1294 - // disable syslog
1295 - error_log_syslog = 0;
1296 -
1297 - // set errors flood protection to 100 logs per hour
1298 - error_log_errors_per_period = 100;
1299 - error_log_throttle_period = 3600;
1300 -
1301 - log_set_global_severity_for_external_plugins();
1287 + nd_log_initialize_for_external_plugins("perf.plugin");
1288
1289 parse_command_line(argc, argv);
1290
collectors/plugins.d/plugins_d.c
+49 -27
@@ -47,8 +47,7 @@ static inline bool plugin_is_running(struct plugind *cd) {
47 return ret;
48 }
49
50 -static void pluginsd_worker_thread_cleanup(void *arg)
51 -{
50 +static void pluginsd_worker_thread_cleanup(void *arg) {
51 struct plugind *cd = (struct plugind *)arg;
52
53 worker_unregister();
@@ -143,41 +142,64 @@ static void *pluginsd_worker_thread(void *arg) {
142
143 netdata_thread_cleanup_push(pluginsd_worker_thread_cleanup, arg);
144
146 - struct plugind *cd = (struct plugind *)arg;
147 - plugin_set_running(cd);
145 + {
146 + struct plugind *cd = (struct plugind *) arg;
147 + plugin_set_running(cd);
148
149 - size_t count = 0;
149 + size_t count = 0;
150
151 - while (service_running(SERVICE_COLLECTORS)) {
152 - FILE *fp_child_input = NULL;
153 - FILE *fp_child_output = netdata_popen(cd->cmd, &cd->unsafe.pid, &fp_child_input);
151 + while(service_running(SERVICE_COLLECTORS)) {
152 + FILE *fp_child_input = NULL;
153 + FILE *fp_child_output = netdata_popen(cd->cmd, &cd->unsafe.pid, &fp_child_input);
154
155 - if (unlikely(!fp_child_input || !fp_child_output)) {
156 - netdata_log_error("PLUGINSD: 'host:%s', cannot popen(\"%s\", \"r\").", rrdhost_hostname(cd->host), cd->cmd);
157 - break;
158 - }
155 + if(unlikely(!fp_child_input || !fp_child_output)) {
156 + netdata_log_error("PLUGINSD: 'host:%s', cannot popen(\"%s\", \"r\").",
157 + rrdhost_hostname(cd->host), cd->cmd);
158 + break;
159 + }
160
160 - netdata_log_info("PLUGINSD: 'host:%s' connected to '%s' running on pid %d",
161 - rrdhost_hostname(cd->host), cd->fullfilename, cd->unsafe.pid);
161 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
162 + "PLUGINSD: 'host:%s' connected to '%s' running on pid %d",
163 + rrdhost_hostname(cd->host),
164 + cd->fullfilename, cd->unsafe.pid);
165
163 - count = pluginsd_process(cd->host, cd, fp_child_input, fp_child_output, 0);
166 + const char *plugin = strrchr(cd->fullfilename, '/');
167 + if(plugin)
168 + plugin++;
169 + else
170 + plugin = cd->fullfilename;
171
165 - netdata_log_info("PLUGINSD: 'host:%s', '%s' (pid %d) disconnected after %zu successful data collections (ENDs).",
166 - rrdhost_hostname(cd->host), cd->fullfilename, cd->unsafe.pid, count);
172 + char module[100];
173 + snprintfz(module, sizeof(module), "plugins.d[%s]", plugin);
174 + ND_LOG_STACK lgs[] = {
175 + ND_LOG_FIELD_TXT(NDF_MODULE, module),
176 + ND_LOG_FIELD_TXT(NDF_NIDL_NODE, rrdhost_hostname(cd->host)),
177 + ND_LOG_FIELD_TXT(NDF_SRC_TRANSPORT, "pluginsd"),
178 + ND_LOG_FIELD_END(),
179 + };
180 + ND_LOG_STACK_PUSH(lgs);
181
168 - killpid(cd->unsafe.pid);
182 + count = pluginsd_process(cd->host, cd, fp_child_input, fp_child_output, 0);
183
170 - int worker_ret_code = netdata_pclose(fp_child_input, fp_child_output, cd->unsafe.pid);
184 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
185 + "PLUGINSD: 'host:%s', '%s' (pid %d) disconnected after %zu successful data collections (ENDs).",
186 + rrdhost_hostname(cd->host), cd->fullfilename, cd->unsafe.pid, count);
187
172 - if (likely(worker_ret_code == 0))
173 - pluginsd_worker_thread_handle_success(cd);
174 - else
175 - pluginsd_worker_thread_handle_error(cd, worker_ret_code);
188 + killpid(cd->unsafe.pid);
189
177 - cd->unsafe.pid = 0;
178 - if (unlikely(!plugin_is_enabled(cd)))
179 - break;
180 - }
190 + int worker_ret_code = netdata_pclose(fp_child_input, fp_child_output, cd->unsafe.pid);
191 +
192 + if(likely(worker_ret_code == 0))
193 + pluginsd_worker_thread_handle_success(cd);
194 + else
195 + pluginsd_worker_thread_handle_error(cd, worker_ret_code);
196 +
197 + cd->unsafe.pid = 0;
198 +
199 + if(unlikely(!plugin_is_enabled(cd)))
200 + break;
201 + }
202 + }
203
204 netdata_thread_cleanup_pop(1);
205 return NULL;
collectors/plugins.d/plugins_d.h
-2
@@ -21,8 +21,6 @@
21 #define PLUGINSD_KEYWORD_REPORT_JOB_STATUS "REPORT_JOB_STATUS"
22 #define PLUGINSD_KEYWORD_DELETE_JOB "DELETE_JOB"
23
24 -#define PLUGINSD_MAX_WORDS 30
25 -
24 #define PLUGINSD_MAX_DIRECTORIES 20
25 extern char *plugin_directories[PLUGINSD_MAX_DIRECTORIES];
26
collectors/plugins.d/pluginsd_parser.c
+88 -92
@@ -153,11 +153,12 @@ static inline bool pluginsd_set_scope_chart(PARSER *parser, RRDSET *st, const ch
153
154 if(unlikely(old_collector_tid)) {
155 if(old_collector_tid != my_collector_tid) {
156 - error_limit_static_global_var(erl, 1, 0);
157 - error_limit(&erl, "PLUGINSD: keyword %s: 'host:%s/chart:%s' is collected twice (my tid %d, other collector tid %d)",
158 - keyword ? keyword : "UNKNOWN",
159 - rrdhost_hostname(st->rrdhost), rrdset_id(st),
160 - my_collector_tid, old_collector_tid);
156 + nd_log_limit_static_global_var(erl, 1, 0);
157 + nd_log_limit(&erl, NDLS_COLLECTORS, NDLP_WARNING,
158 + "PLUGINSD: keyword %s: 'host:%s/chart:%s' is collected twice (my tid %d, other collector tid %d)",
159 + keyword ? keyword : "UNKNOWN",
160 + rrdhost_hostname(st->rrdhost), rrdset_id(st),
161 + my_collector_tid, old_collector_tid);
162
163 return false;
164 }
@@ -389,8 +390,9 @@ static inline PARSER_RC PLUGINSD_DISABLE_PLUGIN(PARSER *parser, const char *keyw
390 parser->user.enabled = 0;
391
392 if(keyword && msg) {
392 - error_limit_static_global_var(erl, 1, 0);
393 - error_limit(&erl, "PLUGINSD: keyword %s: %s", keyword, msg);
393 + nd_log_limit_static_global_var(erl, 1, 0);
394 + nd_log_limit(&erl, NDLS_COLLECTORS, NDLP_INFO,
395 + "PLUGINSD: keyword %s: %s", keyword, msg);
396 }
397
398 return PARSER_RC_ERROR;
@@ -1109,7 +1111,8 @@ void pluginsd_function_cancel(void *data) {
1111 dfe_done(t);
1112
1113 if(sent <= 0)
1112 - netdata_log_error("PLUGINSD: FUNCTION_CANCEL request didn't match any pending function requests in pluginsd.d.");
1114 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
1115 + "PLUGINSD: FUNCTION_CANCEL request didn't match any pending function requests in pluginsd.d.");
1116 }
1117
1118 // this is the function that is called from
@@ -1626,9 +1629,10 @@ static inline PARSER_RC pluginsd_replay_set(char **words, size_t num_words, PARS
1629 if(!st) return PLUGINSD_DISABLE_PLUGIN(parser, NULL, NULL);
1630
1631 if(!parser->user.replay.rset_enabled) {
1629 - error_limit_static_thread_var(erl, 1, 0);
1630 - error_limit(&erl, "PLUGINSD: 'host:%s/chart:%s' got a %s but it is disabled by %s errors",
1631 - rrdhost_hostname(host), rrdset_id(st), PLUGINSD_KEYWORD_REPLAY_SET, PLUGINSD_KEYWORD_REPLAY_BEGIN);
1632 + nd_log_limit_static_thread_var(erl, 1, 0);
1633 + nd_log_limit(&erl, NDLS_COLLECTORS, NDLP_ERR,
1634 + "PLUGINSD: 'host:%s/chart:%s' got a %s but it is disabled by %s errors",
1635 + rrdhost_hostname(host), rrdset_id(st), PLUGINSD_KEYWORD_REPLAY_SET, PLUGINSD_KEYWORD_REPLAY_BEGIN);
1636
1637 // we have to return OK here
1638 return PARSER_RC_OK;
@@ -1675,8 +1679,10 @@ static inline PARSER_RC pluginsd_replay_set(char **words, size_t num_words, PARS
1679 rd->collector.counter++;
1680 }
1681 else {
1678 - error_limit_static_global_var(erl, 1, 0);
1679 - error_limit(&erl, "PLUGINSD: 'host:%s/chart:%s/dim:%s' has the ARCHIVED flag set, but it is replicated. Ignoring data.",
1682 + nd_log_limit_static_global_var(erl, 1, 0);
1683 + nd_log_limit(&erl, NDLS_COLLECTORS, NDLP_WARNING,
1684 + "PLUGINSD: 'host:%s/chart:%s/dim:%s' has the ARCHIVED flag set, but it is replicated. "
1685 + "Ignoring data.",
1686 rrdhost_hostname(st->rrdhost), rrdset_id(st), rrddim_name(rd));
1687 }
1688 }
@@ -2832,61 +2838,6 @@ static inline PARSER_RC streaming_claimed_id(char **words, size_t num_words, PAR
2838
2839 // ----------------------------------------------------------------------------
2840
2835 -static inline bool buffered_reader_read(struct buffered_reader *reader, int fd) {
2836 -#ifdef NETDATA_INTERNAL_CHECKS
2837 - if(reader->read_buffer[reader->read_len] != '\0')
2838 - fatal("%s(): read_buffer does not start with zero", __FUNCTION__ );
2839 -#endif
2840 -
2841 - ssize_t bytes_read = read(fd, reader->read_buffer + reader->read_len, sizeof(reader->read_buffer) - reader->read_len - 1);
2842 - if(unlikely(bytes_read <= 0))
2843 - return false;
2844 -
2845 - reader->read_len += bytes_read;
2846 - reader->read_buffer[reader->read_len] = '\0';
2847 -
2848 - return true;
2849 -}
2850 -
2851 -static inline bool buffered_reader_read_timeout(struct buffered_reader *reader, int fd, int timeout_ms) {
2852 - errno = 0;
2853 - struct pollfd fds[1];
2854 -
2855 - fds[0].fd = fd;
2856 - fds[0].events = POLLIN;
2857 -
2858 - int ret = poll(fds, 1, timeout_ms);
2859 -
2860 - if (ret > 0) {
2861 - /* There is data to read */
2862 - if (fds[0].revents & POLLIN)
2863 - return buffered_reader_read(reader, fd);
2864 -
2865 - else if(fds[0].revents & POLLERR) {
2866 - netdata_log_error("PARSER: read failed: POLLERR.");
2867 - return false;
2868 - }
2869 - else if(fds[0].revents & POLLHUP) {
2870 - netdata_log_error("PARSER: read failed: POLLHUP.");
2871 - return false;
2872 - }
2873 - else if(fds[0].revents & POLLNVAL) {
2874 - netdata_log_error("PARSER: read failed: POLLNVAL.");
2875 - return false;
2876 - }
2877 -
2878 - netdata_log_error("PARSER: poll() returned positive number, but POLLIN|POLLERR|POLLHUP|POLLNVAL are not set.");
2879 - return false;
2880 - }
2881 - else if (ret == 0) {
2882 - netdata_log_error("PARSER: timeout while waiting for data.");
2883 - return false;
2884 - }
2885 -
2886 - netdata_log_error("PARSER: poll() failed with code %d.", ret);
2887 - return false;
2888 -}
2889 -
2841 void pluginsd_process_thread_cleanup(void *ptr) {
2842 PARSER *parser = (PARSER *)ptr;
2843
@@ -2905,6 +2856,33 @@ void pluginsd_process_thread_cleanup(void *ptr) {
2856 parser_destroy(parser);
2857 }
2858
2859 +bool parser_reconstruct_node(BUFFER *wb, void *ptr) {
2860 + PARSER *parser = ptr;
2861 + if(!parser || !parser->user.host)
2862 + return false;
2863 +
2864 + buffer_strcat(wb, rrdhost_hostname(parser->user.host));
2865 + return true;
2866 +}
2867 +
2868 +bool parser_reconstruct_instance(BUFFER *wb, void *ptr) {
2869 + PARSER *parser = ptr;
2870 + if(!parser || !parser->user.st)
2871 + return false;
2872 +
2873 + buffer_strcat(wb, rrdset_name(parser->user.st));
2874 + return true;
2875 +}
2876 +
2877 +bool parser_reconstruct_context(BUFFER *wb, void *ptr) {
2878 + PARSER *parser = ptr;
2879 + if(!parser || !parser->user.st)
2880 + return false;
2881 +
2882 + buffer_strcat(wb, string2str(parser->user.st->context));
2883 + return true;
2884 +}
2885 +
2886 inline size_t pluginsd_process(RRDHOST *host, struct plugind *cd, FILE *fp_plugin_input, FILE *fp_plugin_output, int trust_durations)
2887 {
2888 int enabled = cd->unsafe.enabled;
@@ -2952,33 +2930,51 @@ inline size_t pluginsd_process(RRDHOST *host, struct plugind *cd, FILE *fp_plugi
2930 // so, parser needs to be allocated before pushing it
2931 netdata_thread_cleanup_push(pluginsd_process_thread_cleanup, parser);
2932
2955 - buffered_reader_init(&parser->reader);
2956 - BUFFER *buffer = buffer_create(sizeof(parser->reader.read_buffer) + 2, NULL);
2957 - while(likely(service_running(SERVICE_COLLECTORS))) {
2958 - if (unlikely(!buffered_reader_next_line(&parser->reader, buffer))) {
2959 - if(unlikely(!buffered_reader_read_timeout(&parser->reader, fileno((FILE *)parser->fp_input), 2 * 60 * MSEC_PER_SEC)))
2960 - break;
2961 -
2962 - continue;
2963 - }
2964 -
2965 - if(unlikely(parser_action(parser, buffer->buffer)))
2966 - break;
2933 + {
2934 + ND_LOG_STACK lgs[] = {
2935 + ND_LOG_FIELD_CB(NDF_REQUEST, line_splitter_reconstruct_line, &parser->line),
2936 + ND_LOG_FIELD_CB(NDF_NIDL_NODE, parser_reconstruct_node, parser),
2937 + ND_LOG_FIELD_CB(NDF_NIDL_INSTANCE, parser_reconstruct_instance, parser),
2938 + ND_LOG_FIELD_CB(NDF_NIDL_CONTEXT, parser_reconstruct_context, parser),
2939 + ND_LOG_FIELD_END(),
2940 + };
2941 + ND_LOG_STACK_PUSH(lgs);
2942 +
2943 + buffered_reader_init(&parser->reader);
2944 + BUFFER *buffer = buffer_create(sizeof(parser->reader.read_buffer) + 2, NULL);
2945 + while(likely(service_running(SERVICE_COLLECTORS))) {
2946 +
2947 + if(unlikely(!buffered_reader_next_line(&parser->reader, buffer))) {
2948 + buffered_reader_ret_t ret = buffered_reader_read_timeout(
2949 + &parser->reader,
2950 + fileno((FILE *) parser->fp_input),
2951 + 2 * 60 * MSEC_PER_SEC, true
2952 + );
2953 +
2954 + if(unlikely(ret != BUFFERED_READER_READ_OK))
2955 + break;
2956 +
2957 + continue;
2958 + }
2959 +
2960 + if(unlikely(parser_action(parser, buffer->buffer)))
2961 + break;
2962
2968 - buffer->len = 0;
2969 - buffer->buffer[0] = '\0';
2970 - }
2971 - buffer_free(buffer);
2963 + buffer->len = 0;
2964 + buffer->buffer[0] = '\0';
2965 + }
2966 + buffer_free(buffer);
2967
2973 - cd->unsafe.enabled = parser->user.enabled;
2974 - count = parser->user.data_collections_count;
2968 + cd->unsafe.enabled = parser->user.enabled;
2969 + count = parser->user.data_collections_count;
2970
2976 - if (likely(count)) {
2977 - cd->successful_collections += count;
2978 - cd->serial_failures = 0;
2979 - }
2980 - else
2981 - cd->serial_failures++;
2971 + if(likely(count)) {
2972 + cd->successful_collections += count;
2973 + cd->serial_failures = 0;
2974 + }
2975 + else
2976 + cd->serial_failures++;
2977 + }
2978
2979 // free parser with the pop function
2980 netdata_thread_cleanup_pop(1);
collectors/plugins.d/pluginsd_parser.h
+21 -24
@@ -97,7 +97,6 @@ typedef struct parser {
97 PARSER_REPERTOIRE repertoire;
98 uint32_t flags;
99 int fd; // Socket
100 - size_t line;
100 FILE *fp_input; // Input source e.g. stream
101 FILE *fp_output; // Stream to send commands to plugin
102
@@ -111,6 +110,8 @@ typedef struct parser {
110 PARSER_USER_OBJECT user; // User defined structure to hold extra state between calls
111
112 struct buffered_reader reader;
113 + struct line_splitter line;
114 + PARSER_KEYWORD *keyword;
115
116 struct {
117 const char *end_keyword;
@@ -162,13 +163,17 @@ static inline PARSER_KEYWORD *parser_find_keyword(PARSER *parser, const char *co
163 return NULL;
164 }
165
166 +bool parser_reconstruct_node(BUFFER *wb, void *ptr);
167 +bool parser_reconstruct_instance(BUFFER *wb, void *ptr);
168 +bool parser_reconstruct_context(BUFFER *wb, void *ptr);
169 +
170 static inline int parser_action(PARSER *parser, char *input) {
171 #ifdef NETDATA_LOG_STREAM_RECEIVE
172 static __thread char line[PLUGINSD_LINE_MAX + 1];
173 strncpyz(line, input, sizeof(line) - 1);
174 #endif
175
171 - parser->line++;
176 + parser->line.count++;
177
178 if(unlikely(parser->flags & PARSER_DEFER_UNTIL_KEYWORD)) {
179 char command[100 + 1];
@@ -200,24 +205,25 @@ static inline int parser_action(PARSER *parser, char *input) {
205 return 0;
206 }
207
203 - static __thread char *words[PLUGINSD_MAX_WORDS];
204 - size_t num_words = quoted_strings_splitter_pluginsd(input, words, PLUGINSD_MAX_WORDS);
205 - const char *command = get_word(words, num_words, 0);
208 + parser->line.num_words = quoted_strings_splitter_pluginsd(input, parser->line.words, PLUGINSD_MAX_WORDS);
209 + const char *command = get_word(parser->line.words, parser->line.num_words, 0);
210
207 - if(unlikely(!command))
211 + if(unlikely(!command)) {
212 + line_splitter_reset(&parser->line);
213 return 0;
214 + }
215
216 PARSER_RC rc;
211 - PARSER_KEYWORD *t = parser_find_keyword(parser, command);
212 - if(likely(t)) {
213 - worker_is_busy(t->worker_job_id);
217 + parser->keyword = parser_find_keyword(parser, command);
218 + if(likely(parser->keyword)) {
219 + worker_is_busy(parser->keyword->worker_job_id);
220
221 #ifdef NETDATA_LOG_STREAM_RECEIVE
216 - if(parser->user.stream_log_fp && t->repertoire & parser->user.stream_log_repertoire)
222 + if(parser->user.stream_log_fp && parser->keyword->repertoire & parser->user.stream_log_repertoire)
223 fprintf(parser->user.stream_log_fp, "%s", line);
224 #endif
225
220 - rc = parser_execute(parser, t, words, num_words);
226 + rc = parser_execute(parser, parser->keyword, parser->line.words, parser->line.num_words);
227 // rc = (*t->func)(words, num_words, parser);
228 worker_is_idle();
229 }
@@ -225,22 +231,13 @@ static inline int parser_action(PARSER *parser, char *input) {
231 rc = PARSER_RC_ERROR;
232
233 if(rc == PARSER_RC_ERROR) {
228 - BUFFER *wb = buffer_create(PLUGINSD_LINE_MAX, NULL);
229 - for(size_t i = 0; i < num_words ;i++) {
230 - if(i) buffer_fast_strcat(wb, " ", 1);
231 -
232 - buffer_fast_strcat(wb, "\"", 1);
233 - const char *s = get_word(words, num_words, i);
234 - buffer_strcat(wb, s?s:"");
235 - buffer_fast_strcat(wb, "\"", 1);
236 - }
237 -
234 + CLEAN_BUFFER *wb = buffer_create(PLUGINSD_LINE_MAX, NULL);
235 + line_splitter_reconstruct_line(wb, &parser->line);
236 netdata_log_error("PLUGINSD: parser_action('%s') failed on line %zu: { %s } (quotes added to show parsing)",
239 - command, parser->line, buffer_tostring(wb));
240 -
241 - buffer_free(wb);
237 + command, parser->line.count, buffer_tostring(wb));
238 }
239
240 + line_splitter_reset(&parser->line);
241 return (rc == PARSER_RC_ERROR || rc == PARSER_RC_STOP);
242 }
243
collectors/proc.plugin/plugin_proc.c
+45 -29
@@ -138,6 +138,12 @@ static bool is_lxcfs_proc_mounted() {
138 return false;
139 }
140
141 +static bool log_proc_module(BUFFER *wb, void *data) {
142 + struct proc_module *pm = data;
143 + buffer_sprintf(wb, "proc.plugin[%s]", pm->name);
144 + return true;
145 +}
146 +
147 void *proc_main(void *ptr)
148 {
149 worker_register("PROC");
@@ -153,46 +159,56 @@ void *proc_main(void *ptr)
159
160 netdata_thread_cleanup_push(proc_main_cleanup, ptr);
161
156 - config_get_boolean("plugin:proc", "/proc/pagetypeinfo", CONFIG_BOOLEAN_NO);
162 + {
163 + config_get_boolean("plugin:proc", "/proc/pagetypeinfo", CONFIG_BOOLEAN_NO);
164
158 - // check the enabled status for each module
159 - int i;
160 - for (i = 0; proc_modules[i].name; i++) {
161 - struct proc_module *pm = &proc_modules[i];
165 + // check the enabled status for each module
166 + int i;
167 + for(i = 0; proc_modules[i].name; i++) {
168 + struct proc_module *pm = &proc_modules[i];
169
163 - pm->enabled = config_get_boolean("plugin:proc", pm->name, CONFIG_BOOLEAN_YES);
164 - pm->rd = NULL;
170 + pm->enabled = config_get_boolean("plugin:proc", pm->name, CONFIG_BOOLEAN_YES);
171 + pm->rd = NULL;
172
166 - worker_register_job_name(i, proc_modules[i].dim);
167 - }
173 + worker_register_job_name(i, proc_modules[i].dim);
174 + }
175
169 - usec_t step = localhost->rrd_update_every * USEC_PER_SEC;
170 - heartbeat_t hb;
171 - heartbeat_init(&hb);
176 + usec_t step = localhost->rrd_update_every * USEC_PER_SEC;
177 + heartbeat_t hb;
178 + heartbeat_init(&hb);
179
173 - inside_lxc_container = is_lxcfs_proc_mounted();
180 + inside_lxc_container = is_lxcfs_proc_mounted();
181
175 - while (service_running(SERVICE_COLLECTORS)) {
176 - worker_is_idle();
177 - usec_t hb_dt = heartbeat_next(&hb, step);
182 +#define LGS_MODULE_ID 0
183
179 - if (unlikely(!service_running(SERVICE_COLLECTORS)))
180 - break;
184 + ND_LOG_STACK lgs[] = {
185 + [LGS_MODULE_ID] = ND_LOG_FIELD_TXT(NDF_MODULE, "proc.plugin"),
186 + ND_LOG_FIELD_END(),
187 + };
188 + ND_LOG_STACK_PUSH(lgs);
189
182 - for (i = 0; proc_modules[i].name; i++) {
183 - if (unlikely(!service_running(SERVICE_COLLECTORS)))
184 - break;
190 + while(service_running(SERVICE_COLLECTORS)) {
191 + worker_is_idle();
192 + usec_t hb_dt = heartbeat_next(&hb, step);
193
186 - struct proc_module *pm = &proc_modules[i];
187 - if (unlikely(!pm->enabled))
188 - continue;
194 + if(unlikely(!service_running(SERVICE_COLLECTORS)))
195 + break;
196
190 - netdata_log_debug(D_PROCNETDEV_LOOP, "PROC calling %s.", pm->name);
197 + for(i = 0; proc_modules[i].name; i++) {
198 + if(unlikely(!service_running(SERVICE_COLLECTORS)))
199 + break;
200
192 - worker_is_busy(i);
193 - pm->enabled = !pm->func(localhost->rrd_update_every, hb_dt);
194 - }
195 - }
201 + struct proc_module *pm = &proc_modules[i];
202 + if(unlikely(!pm->enabled))
203 + continue;
204 +
205 + worker_is_busy(i);
206 + lgs[LGS_MODULE_ID] = ND_LOG_FIELD_CB(NDF_MODULE, log_proc_module, pm);
207 + pm->enabled = !pm->func(localhost->rrd_update_every, hb_dt);
208 + lgs[LGS_MODULE_ID] = ND_LOG_FIELD_TXT(NDF_MODULE, "proc.plugin");
209 + }
210 + }
211 + }
212
213 netdata_thread_cleanup_pop(1);
214 return NULL;
collectors/slabinfo.plugin/slabinfo.c
+1 -4
@@ -336,14 +336,11 @@ void usage(void) {
336 }
337
338 int main(int argc, char **argv) {
339 - stderror = stderr;
339 clocks_init();
340 + nd_log_initialize_for_external_plugins("slabinfo.plugin");
341
342 program_name = argv[0];
343 program_version = "0.1";
344 - error_log_syslog = 0;
345 -
346 - log_set_global_severity_for_external_plugins();
344
345 int update_every = 1, i, n, freq = 0;
346
collectors/statsd.plugin/statsd.c
+1 -1
@@ -2326,7 +2326,7 @@ static inline void statsd_flush_index_metrics(STATSD_INDEX *index, void (*flush_
2326 if(unlikely(is_metric_checked(m))) break;
2327
2328 if(unlikely(!(m->options & STATSD_METRIC_OPTION_CHECKED_IN_APPS))) {
2329 - netdata_log_access("NEW STATSD METRIC '%s': '%s'", statsd_metric_type_string(m->type), m->name);
2329 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "NEW STATSD METRIC '%s': '%s'", statsd_metric_type_string(m->type), m->name);
2330 check_if_metric_is_for_app(index, m);
2331 m->options |= STATSD_METRIC_OPTION_CHECKED_IN_APPS;
2332 }
collectors/systemd-journal.plugin/systemd-journal-annotations.c
+49 -25
@@ -408,35 +408,50 @@ void netdata_systemd_journal_transform_boot_id(FACETS *facets __maybe_unused, BU
408 };
409
410 sd_journal *j = NULL;
411 - if(sd_journal_open_files(&j, files, ND_SD_JOURNAL_OPEN_FLAGS) < 0 || !j) {
412 - internal_error(true, "JOURNAL: cannot open file '%s' to get boot_id", jf_dfe.name);
411 + int r = sd_journal_open_files(&j, files, ND_SD_JOURNAL_OPEN_FLAGS);
412 + if(r < 0 || !j) {
413 + internal_error(true, "JOURNAL: while looking for the first timestamp of boot_id '%s', "
414 + "sd_journal_open_files('%s') returned %d",
415 + boot_id, jf_dfe.name, r);
416 continue;
417 }
418
419 char m[100];
420 size_t len = snprintfz(m, sizeof(m), "_BOOT_ID=%s", boot_id);
421
419 - if(sd_journal_add_match(j, m, len) < 0) {
420 - internal_error(true, "JOURNAL: cannot add match '%s' to file '%s'", m, jf_dfe.name);
422 + r = sd_journal_add_match(j, m, len);
423 + if(r < 0) {
424 + internal_error(true, "JOURNAL: while looking for the first timestamp of boot_id '%s', "
425 + "sd_journal_add_match('%s') on file '%s' returned %d",
426 + boot_id, m, jf_dfe.name, r);
427 sd_journal_close(j);
428 continue;
429 }
430
425 - if(sd_journal_seek_head(j) < 0) {
426 - internal_error(true, "JOURNAL: cannot seek head to file '%s'", jf_dfe.name);
431 + r = sd_journal_seek_head(j);
432 + if(r < 0) {
433 + internal_error(true, "JOURNAL: while looking for the first timestamp of boot_id '%s', "
434 + "sd_journal_seek_head() on file '%s' returned %d",
435 + boot_id, jf_dfe.name, r);
436 sd_journal_close(j);
437 continue;
438 }
439
431 - if(sd_journal_next(j) < 0) {
432 - internal_error(true, "JOURNAL: cannot get next of file '%s'", jf_dfe.name);
440 + r = sd_journal_next(j);
441 + if(r < 0) {
442 + internal_error(true, "JOURNAL: while looking for the first timestamp of boot_id '%s', "
443 + "sd_journal_next() on file '%s' returned %d",
444 + boot_id, jf_dfe.name, r);
445 sd_journal_close(j);
446 continue;
447 }
448
449 usec_t t_ut = 0;
438 - if(sd_journal_get_realtime_usec(j, &t_ut) < 0 || !t_ut) {
439 - internal_error(true, "JOURNAL: cannot get realtime_usec of file '%s'", jf_dfe.name);
450 + r = sd_journal_get_realtime_usec(j, &t_ut);
451 + if(r < 0 || !t_ut) {
452 + internal_error(r != -EADDRNOTAVAIL, "JOURNAL: while looking for the first timestamp of boot_id '%s', "
453 + "sd_journal_get_realtime_usec() on file '%s' returned %d",
454 + boot_id, jf_dfe.name, r);
455 sd_journal_close(j);
456 continue;
457 }
@@ -454,25 +469,21 @@ void netdata_systemd_journal_transform_boot_id(FACETS *facets __maybe_unused, BU
469 ut = *p_ut;
470
471 if(ut != UINT64_MAX) {
457 - time_t timestamp_sec = (time_t)(ut / USEC_PER_SEC);
458 - struct tm tm;
459 - char buffer[30];
460 -
461 - gmtime_r(&timestamp_sec, &tm);
462 - strftime(buffer, sizeof(buffer), "%Y-%m-%d %H:%M:%S", &tm);
472 + char buffer[ISO8601_MAX_LENGTH];
473 + iso8601_datetime_ut(buffer, sizeof(buffer), ut, ISO8601_UTC);
474
475 switch(scope) {
476 default:
477 case FACETS_TRANSFORM_DATA:
478 case FACETS_TRANSFORM_VALUE:
468 - buffer_sprintf(wb, " (%s UTC) ", buffer);
479 + buffer_sprintf(wb, " (%s) ", buffer);
480 break;
481
482 case FACETS_TRANSFORM_FACET:
483 case FACETS_TRANSFORM_FACET_SORT:
484 case FACETS_TRANSFORM_HISTOGRAM:
485 buffer_flush(wb);
475 - buffer_sprintf(wb, "%s UTC", buffer);
486 + buffer_sprintf(wb, "%s", buffer);
487 break;
488 }
489 }
@@ -537,13 +548,9 @@ void netdata_systemd_journal_transform_timestamp_usec(FACETS *facets __maybe_unu
548 if(*v && isdigit(*v)) {
549 uint64_t ut = str2ull(buffer_tostring(wb), NULL);
550 if(ut) {
540 - time_t timestamp_sec = (time_t)(ut / USEC_PER_SEC);
541 - struct tm tm;
542 - char buffer[30];
543 -
544 - gmtime_r(&timestamp_sec, &tm);
545 - strftime(buffer, sizeof(buffer), "%Y-%m-%d %H:%M:%S", &tm);
546 - buffer_sprintf(wb, " (%s.%06llu UTC)", buffer, ut % USEC_PER_SEC);
551 + char buffer[ISO8601_MAX_LENGTH];
552 + iso8601_datetime_ut(buffer, sizeof(buffer), ut, ISO8601_UTC | ISO8601_MICROSECONDS);
553 + buffer_sprintf(wb, " (%s)", buffer);
554 }
555 }
556 }
@@ -703,6 +710,23 @@ void netdata_systemd_journal_message_ids_init(void) {
710 // gnome-shell
711 // https://gitlab.gnome.org/GNOME/gnome-shell/-/blob/main/js/ui/main.js#L56
712 i.msg = "Gnome shell started";dictionary_set(known_journal_messages_ids, "f3ea493c22934e26811cd62abe8e203a", &i, sizeof(i));
713 +
714 + // flathub
715 + // https://docs.flatpak.org/de/latest/flatpak-command-reference.html
716 + i.msg = "Flatpak cache"; dictionary_set(known_journal_messages_ids, "c7b39b1e006b464599465e105b361485", &i, sizeof(i));
717 +
718 + // ???
719 + i.msg = "Flathub pulls"; dictionary_set(known_journal_messages_ids, "75ba3deb0af041a9a46272ff85d9e73e", &i, sizeof(i));
720 + i.msg = "Flathub pull errors"; dictionary_set(known_journal_messages_ids, "f02bce89a54e4efab3a94a797d26204a", &i, sizeof(i));
721 +
722 + // ??
723 + i.msg = "Boltd starting"; dictionary_set(known_journal_messages_ids, "dd11929c788e48bdbb6276fb5f26b08a", &i, sizeof(i));
724 +
725 + // Netdata
726 + i.msg = "Netdata connection from child"; dictionary_set(known_journal_messages_ids, "ed4cdb8f1beb4ad3b57cb3cae2d162fa", &i, sizeof(i));
727 + i.msg = "Netdata connection to parent"; dictionary_set(known_journal_messages_ids, "6e2e3839067648968b646045dbf28d66", &i, sizeof(i));
728 + i.msg = "Netdata alert transition"; dictionary_set(known_journal_messages_ids, "9ce0cb58ab8b44df82c4bf1ad9ee22de", &i, sizeof(i));
729 + i.msg = "Netdata alert notification"; dictionary_set(known_journal_messages_ids, "6db0018e83e34320ae2a659d78019fb7", &i, sizeof(i));
730 }
731
732 void netdata_systemd_journal_transform_message_id(FACETS *facets __maybe_unused, BUFFER *wb, FACETS_TRANSFORMATION_SCOPE scope __maybe_unused, void *data __maybe_unused) {
collectors/systemd-journal.plugin/systemd-journal-files.c
+1 -1
@@ -333,8 +333,8 @@ static void files_registry_delete_cb(const DICTIONARY_ITEM *item, void *value, v
333 struct journal_file *jf = value; (void)jf;
334 const char *filename = dictionary_acquired_item_name(item); (void)filename;
335
336 - string_freez(jf->source);
336 internal_error(true, "removed journal file '%s'", filename);
337 + string_freez(jf->source);
338 }
339
340 void journal_directory_scan(const char *dirname, int depth, usec_t last_scan_ut) {
collectors/systemd-journal.plugin/systemd-journal.c
+12
@@ -165,6 +165,18 @@
165 "|IMAGE_NAME" /* undocumented */ \
166 /* "|CONTAINER_PARTIAL_MESSAGE" */ \
167 \
168 + \
169 + /* --- NETDATA --- */ \
170 + \
171 + "|ND_NIDL_NODE" \
172 + "|ND_NIDL_CONTEXT" \
173 + "|ND_LOG_SOURCE" \
174 + /*"|ND_MODULE" */ \
175 + "|ND_ALERT_NAME" \
176 + "|ND_ALERT_CLASS" \
177 + "|ND_ALERT_COMPONENT" \
178 + "|ND_ALERT_TYPE" \
179 + \
180 ""
181
182 // ----------------------------------------------------------------------------
collectors/systemd-journal.plugin/systemd-main.c
+1 -12
@@ -9,19 +9,8 @@ netdata_mutex_t stdout_mutex = NETDATA_MUTEX_INITIALIZER;
9 static bool plugin_should_exit = false;
10
11 int main(int argc __maybe_unused, char **argv __maybe_unused) {
12 - stderror = stderr;
12 clocks_init();
14 -
15 - program_name = "systemd-journal.plugin";
16 -
17 - // disable syslog
18 - error_log_syslog = 0;
19 -
20 - // set errors flood protection to 100 logs per hour
21 - error_log_errors_per_period = 100;
22 - error_log_throttle_period = 3600;
23 -
24 - log_set_global_severity_for_external_plugins();
13 + nd_log_initialize_for_external_plugins("systemd-journal.plugin");
14
15 netdata_configured_host_prefix = getenv("NETDATA_HOST_PREFIX");
16 if(verify_netdata_host_prefix() == -1) exit(1);
collectors/systemd-journal.plugin/systemd-units.c
-2
@@ -676,8 +676,6 @@ static void update_freezer_state(UnitInfo *u, UnitAttribute *ua) {
676 // ----------------------------------------------------------------------------
677 // common helpers
678
679 -#define _cleanup_(x) __attribute__((__cleanup__(x)))
680 -
679 static void log_dbus_error(int r, const char *msg) {
680 netdata_log_error("SYSTEMD_UNITS: %s failed with error %d (%s)", msg, r, strerror(-r));
681 }
collectors/xenstat.plugin/xenstat_plugin.c
+1 -9
@@ -920,7 +920,6 @@ static void xenstat_send_domain_metrics() {
920 }
921
922 int main(int argc, char **argv) {
923 - stderror = stderr;
923 clocks_init();
924
925 // ------------------------------------------------------------------------
@@ -928,14 +927,7 @@ int main(int argc, char **argv) {
927
928 program_name = "xenstat.plugin";
929
931 - // disable syslog
932 - error_log_syslog = 0;
933 -
934 - // set errors flood protection to 100 logs per hour
935 - error_log_errors_per_period = 100;
936 - error_log_throttle_period = 3600;
937 -
938 - log_set_global_severity_for_external_plugins();
930 + nd_log_initialize_for_external_plugins();
931
932 // ------------------------------------------------------------------------
933 // parse command line parameters
configure.ac
+50 -12
@@ -54,6 +54,8 @@ else
54 AC_CHECK_TOOL([AR], [ar])
55 fi
56
57 +CFLAGS="$CFLAGS -fexceptions"
58 +
59 # -----------------------------------------------------------------------------
60 # configurable options
61
@@ -571,6 +573,48 @@ AC_CHECK_LIB(
573 [LZ4_LIBS="-llz4"]
574 )
575
576 +# -----------------------------------------------------------------------------
577 +# libcurl
578 +
579 +PKG_CHECK_MODULES(
580 + [LIBCURL],
581 + [libcurl],
582 + [AC_CHECK_LIB(
583 + [curl],
584 + [curl_easy_init],
585 + [have_libcurl=yes],
586 + [have_libcurl=no]
587 + )],
588 + [have_libcurl=no]
589 +)
590 +
591 +if test "x$have_libcurl" = "xyes"; then
592 + AC_DEFINE([HAVE_CURL], [1], [libcurl usability])
593 + OPTIONAL_CURL_LIBS="-lcurl"
594 +fi
595 +
596 +# -----------------------------------------------------------------------------
597 +# PCRE2
598 +
599 +PKG_CHECK_MODULES(
600 + [LIBPCRE2],
601 + [libpcre2-8],
602 + [AC_CHECK_LIB(
603 + [pcre2-8],
604 + [pcre2_compile_8],
605 + [have_libpcre2=yes],
606 + [have_libpcre2=no]
607 + )],
608 + [have_libpcre2=no]
609 +)
610 +
611 +if test "x$have_libpcre2" = "xyes"; then
612 + AC_DEFINE([HAVE_PCRE2], [1], [PCRE2 usability])
613 + OPTIONAL_PCRE2_LIBS="-lpcre2-8"
614 +fi
615 +
616 +AM_CONDITIONAL([ENABLE_LOG2JOURNAL], [test "${have_libpcre2}" = "yes"])
617 +
618 # -----------------------------------------------------------------------------
619 # zstd
620
@@ -1590,18 +1634,6 @@ PKG_CHECK_MODULES(
1634 [have_libssl=no]
1635 )
1636
1593 -PKG_CHECK_MODULES(
1594 - [LIBCURL],
1595 - [libcurl],
1596 - [AC_CHECK_LIB(
1597 - [curl],
1598 - [curl_easy_init],
1599 - [have_libcurl=yes],
1600 - [have_libcurl=no]
1601 - )],
1602 - [have_libcurl=no]
1603 -)
1604 -
1637 PKG_CHECK_MODULES(
1638 [AWS_CPP_SDK_CORE],
1639 [aws-cpp-sdk-core],
@@ -1946,6 +1978,8 @@ AC_SUBST([OPTIONAL_UV_LIBS])
1978 AC_SUBST([OPTIONAL_LZ4_LIBS])
1979 AC_SUBST([OPTIONAL_BROTLIENC_LIBS])
1980 AC_SUBST([OPTIONAL_BROTLIDEC_LIBS])
1981 +AC_SUBST([OPTIONAL_CURL_LIBS])
1982 +AC_SUBST([OPTIONAL_PCRE2_LIBS])
1983 AC_SUBST([OPTIONAL_ZSTD_LIBS])
1984 AC_SUBST([OPTIONAL_SSL_LIBS])
1985 AC_SUBST([OPTIONAL_JSONC_LIBS])
@@ -2073,15 +2107,18 @@ AC_CONFIG_FILES([
2107 libnetdata/aral/Makefile
2108 libnetdata/avl/Makefile
2109 libnetdata/buffer/Makefile
2110 + libnetdata/buffered_reader/Makefile
2111 libnetdata/clocks/Makefile
2112 libnetdata/completion/Makefile
2113 libnetdata/config/Makefile
2114 + libnetdata/datetime/Makefile
2115 libnetdata/dictionary/Makefile
2116 libnetdata/ebpf/Makefile
2117 libnetdata/eval/Makefile
2118 libnetdata/facets/Makefile
2119 libnetdata/functions_evloop/Makefile
2120 libnetdata/july/Makefile
2121 + libnetdata/line_splitter/Makefile
2122 libnetdata/locks/Makefile
2123 libnetdata/log/Makefile
2124 libnetdata/onewayalloc/Makefile
@@ -2095,6 +2132,7 @@ AC_CONFIG_FILES([
2132 libnetdata/storage_number/tests/Makefile
2133 libnetdata/threads/Makefile
2134 libnetdata/url/Makefile
2135 + libnetdata/uuid/Makefile
2136 libnetdata/json/Makefile
2137 libnetdata/health/Makefile
2138 libnetdata/worker_utilization/Makefile
contrib/debian/control
+2
@@ -4,6 +4,7 @@ Build-Depends: debhelper (>= 9.20160709),
4 dpkg-dev (>= 1.13.19),
5 zlib1g-dev,
6 uuid-dev,
7 + libcurl4-openssl-dev,
8 libelf-dev,
9 libuv1-dev,
10 liblz4-dev,
@@ -15,6 +16,7 @@ Build-Depends: debhelper (>= 9.20160709),
16 libipmimonitoring-dev,
17 libnetfilter-acct-dev,
18 libsnappy-dev,
19 + libpcre2-8-0,
20 libprotobuf-dev,
21 libprotoc-dev,
22 libsystemd-dev,
daemon/commands.c
+8 -9
@@ -142,10 +142,10 @@ static cmd_status_t cmd_reload_health_execute(char *args, char **message)
142 (void)args;
143 (void)message;
144
145 - error_log_limit_unlimited();
145 + nd_log_limits_unlimited();
146 netdata_log_info("COMMAND: Reloading HEALTH configuration.");
147 health_reload();
148 - error_log_limit_reset();
148 + nd_log_limits_reset();
149
150 return CMD_STATUS_SUCCESS;
151 }
@@ -155,11 +155,11 @@ static cmd_status_t cmd_save_database_execute(char *args, char **message)
155 (void)args;
156 (void)message;
157
158 - error_log_limit_unlimited();
158 + nd_log_limits_unlimited();
159 netdata_log_info("COMMAND: Saving databases.");
160 rrdhost_save_all();
161 netdata_log_info("COMMAND: Databases saved.");
162 - error_log_limit_reset();
162 + nd_log_limits_reset();
163
164 return CMD_STATUS_SUCCESS;
165 }
@@ -169,10 +169,9 @@ static cmd_status_t cmd_reopen_logs_execute(char *args, char **message)
169 (void)args;
170 (void)message;
171
172 - error_log_limit_unlimited();
173 - netdata_log_info("COMMAND: Reopening all log files.");
174 - reopen_all_log_files();
175 - error_log_limit_reset();
172 + nd_log_limits_unlimited();
173 + nd_log_reopen_log_files();
174 + nd_log_limits_reset();
175
176 return CMD_STATUS_SUCCESS;
177 }
@@ -182,7 +181,7 @@ static cmd_status_t cmd_exit_execute(char *args, char **message)
181 (void)args;
182 (void)message;
183
185 - error_log_limit_unlimited();
184 + nd_log_limits_unlimited();
185 netdata_log_info("COMMAND: Cleaning up to exit.");
186 netdata_cleanup_and_exit(0);
187 exit(0);
daemon/daemon.c
+1 -17
@@ -31,22 +31,6 @@ void get_netdata_execution_path(void) {
31 dirname(netdata_exe_path);
32 }
33
34 -static void chown_open_file(int fd, uid_t uid, gid_t gid) {
35 - if(fd == -1) return;
36 -
37 - struct stat buf;
38 -
39 - if(fstat(fd, &buf) == -1) {
40 - netdata_log_error("Cannot fstat() fd %d", fd);
41 - return;
42 - }
43 -
44 - if((buf.st_uid != uid || buf.st_gid != gid) && S_ISREG(buf.st_mode)) {
45 - if(fchown(fd, uid, gid) == -1)
46 - netdata_log_error("Cannot fchown() fd %d.", fd);
47 - }
48 -}
49 -
34 static void fix_directory_file_permissions(const char *dirname, uid_t uid, gid_t gid, bool recursive)
35 {
36 char filename[FILENAME_MAX + 1];
@@ -150,9 +134,9 @@ int become_user(const char *username, int pid_fd) {
134 }
135 }
136
137 + nd_log_chown_log_files(uid, gid);
138 chown_open_file(STDOUT_FILENO, uid, gid);
139 chown_open_file(STDERR_FILENO, uid, gid);
155 - chown_open_file(stdaccess_fd, uid, gid);
140 chown_open_file(pid_fd, uid, gid);
141
142 if(supplementary_groups && ngroups > 0) {
daemon/main.c
+63 -35
@@ -315,7 +315,7 @@ void netdata_cleanup_and_exit(int ret) {
315 const char *prev_msg = NULL;
316 bool timeout = false;
317
318 - error_log_limit_unlimited();
318 + nd_log_limits_unlimited();
319 netdata_log_info("NETDATA SHUTDOWN: initializing shutdown with code %d...", ret);
320
321 send_statistics("EXIT", ret?"ERROR":"OK","-");
@@ -449,8 +449,9 @@ void netdata_cleanup_and_exit(int ret) {
449 running += rrdeng_collectors_running(multidb_ctx[tier]);
450
451 if(running) {
452 - error_limit_static_thread_var(erl, 1, 100 * USEC_PER_MS);
453 - error_limit(&erl, "waiting for %zu collectors to finish", running);
452 + nd_log_limit_static_thread_var(erl, 1, 100 * USEC_PER_MS);
453 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_NOTICE,
454 + "waiting for %zu collectors to finish", running);
455 // sleep_usec(100 * USEC_PER_MS);
456 cleanup_destroyed_dictionaries();
457 }
@@ -618,8 +619,14 @@ int killpid(pid_t pid) {
619 int ret;
620 netdata_log_debug(D_EXIT, "Request to kill pid %d", pid);
621
622 + int signal = SIGTERM;
623 +//#ifdef NETDATA_INTERNAL_CHECKS
624 +// if(service_running(SERVICE_COLLECTORS))
625 +// signal = SIGABRT;
626 +//#endif
627 +
628 errno = 0;
622 - ret = kill(pid, SIGTERM);
629 + ret = kill(pid, signal);
630 if (ret == -1) {
631 switch(errno) {
632 case ESRCH:
@@ -666,7 +673,7 @@ static void set_nofile_limit(struct rlimit *rl) {
673 }
674
675 void cancel_main_threads() {
669 - error_log_limit_unlimited();
676 + nd_log_limits_unlimited();
677
678 int i, found = 0;
679 usec_t max = 5 * USEC_PER_SEC, step = 100000;
@@ -756,7 +763,7 @@ int help(int exitcode) {
763 " | '-' '-' '-' '-' real-time performance monitoring, done right! \n"
764 " +----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+--->\n"
765 "\n"
759 - " Copyright (C) 2016-2022, Netdata, Inc. <info@netdata.cloud>\n"
766 + " Copyright (C) 2016-2023, Netdata, Inc. <info@netdata.cloud>\n"
767 " Released under GNU General Public License v3 or later.\n"
768 " All rights reserved.\n"
769 "\n"
@@ -845,44 +852,49 @@ static void security_init(){
852 #endif
853
854 static void log_init(void) {
855 + nd_log_set_facility(config_get(CONFIG_SECTION_LOGS, "facility", "daemon"));
856 +
857 + time_t period = ND_LOG_DEFAULT_THROTTLE_PERIOD;
858 + size_t logs = ND_LOG_DEFAULT_THROTTLE_LOGS;
859 + period = config_get_number(CONFIG_SECTION_LOGS, "logs flood protection period", period);
860 + logs = (unsigned long)config_get_number(CONFIG_SECTION_LOGS, "logs to trigger flood protection", (long long int)logs);
861 + nd_log_set_flood_protection(logs, period);
862 +
863 + nd_log_set_priority_level(config_get(CONFIG_SECTION_LOGS, "level", NDLP_INFO_STR));
864 +
865 char filename[FILENAME_MAX + 1];
866 snprintfz(filename, FILENAME_MAX, "%s/debug.log", netdata_configured_log_dir);
850 - stdout_filename = config_get(CONFIG_SECTION_LOGS, "debug", filename);
867 + nd_log_set_user_settings(NDLS_DEBUG, config_get(CONFIG_SECTION_LOGS, "debug", filename));
868
852 - snprintfz(filename, FILENAME_MAX, "%s/error.log", netdata_configured_log_dir);
853 - stderr_filename = config_get(CONFIG_SECTION_LOGS, "error", filename);
869 + bool with_journal = is_stderr_connected_to_journal() /* || nd_log_journal_socket_available() */;
870 + if(with_journal)
871 + snprintfz(filename, FILENAME_MAX, "journal");
872 + else
873 + snprintfz(filename, FILENAME_MAX, "%s/daemon.log", netdata_configured_log_dir);
874 + nd_log_set_user_settings(NDLS_DAEMON, config_get(CONFIG_SECTION_LOGS, "daemon", filename));
875
855 - snprintfz(filename, FILENAME_MAX, "%s/collector.log", netdata_configured_log_dir);
856 - stdcollector_filename = config_get(CONFIG_SECTION_LOGS, "collector", filename);
876 + if(with_journal)
877 + snprintfz(filename, FILENAME_MAX, "journal");
878 + else
879 + snprintfz(filename, FILENAME_MAX, "%s/collector.log", netdata_configured_log_dir);
880 + nd_log_set_user_settings(NDLS_COLLECTORS, config_get(CONFIG_SECTION_LOGS, "collector", filename));
881
882 snprintfz(filename, FILENAME_MAX, "%s/access.log", netdata_configured_log_dir);
859 - stdaccess_filename = config_get(CONFIG_SECTION_LOGS, "access", filename);
883 + nd_log_set_user_settings(NDLS_ACCESS, config_get(CONFIG_SECTION_LOGS, "access", filename));
884
861 - snprintfz(filename, FILENAME_MAX, "%s/health.log", netdata_configured_log_dir);
862 - stdhealth_filename = config_get(CONFIG_SECTION_LOGS, "health", filename);
885 + if(with_journal)
886 + snprintfz(filename, FILENAME_MAX, "journal");
887 + else
888 + snprintfz(filename, FILENAME_MAX, "%s/health.log", netdata_configured_log_dir);
889 + nd_log_set_user_settings(NDLS_HEALTH, config_get(CONFIG_SECTION_LOGS, "health", filename));
890
891 #ifdef ENABLE_ACLK
892 aclklog_enabled = config_get_boolean(CONFIG_SECTION_CLOUD, "conversation log", CONFIG_BOOLEAN_NO);
893 if (aclklog_enabled) {
894 snprintfz(filename, FILENAME_MAX, "%s/aclk.log", netdata_configured_log_dir);
868 - aclklog_filename = config_get(CONFIG_SECTION_CLOUD, "conversation log file", filename);
895 + nd_log_set_user_settings(NDLS_ACLK, config_get(CONFIG_SECTION_CLOUD, "conversation log file", filename));
896 }
897 #endif
871 -
872 - char deffacility[8];
873 - snprintfz(deffacility,7,"%s","daemon");
874 - facility_log = config_get(CONFIG_SECTION_LOGS, "facility", deffacility);
875 -
876 - error_log_throttle_period = config_get_number(CONFIG_SECTION_LOGS, "errors flood protection period", error_log_throttle_period);
877 - error_log_errors_per_period = (unsigned long)config_get_number(CONFIG_SECTION_LOGS, "errors to trigger flood protection", (long long int)error_log_errors_per_period);
878 - error_log_errors_per_period_backup = error_log_errors_per_period;
879 -
880 - setenv("NETDATA_ERRORS_THROTTLE_PERIOD", config_get(CONFIG_SECTION_LOGS, "errors flood protection period" , ""), 1);
881 - setenv("NETDATA_ERRORS_PER_PERIOD", config_get(CONFIG_SECTION_LOGS, "errors to trigger flood protection", ""), 1);
882 -
883 - char *selected_level = config_get(CONFIG_SECTION_LOGS, "severity level", NETDATA_LOG_LEVEL_INFO_STR);
884 - global_log_severity_level = log_severity_string_to_severity_level(selected_level);
885 - setenv("NETDATA_LOG_SEVERITY_LEVEL", selected_level , 1);
898 }
899
900 char *initialize_lock_directory_path(char *prefix)
@@ -1054,6 +1066,17 @@ static void backwards_compatible_config() {
1066 config_move(CONFIG_SECTION_GLOBAL, "enable zero metrics",
1067 CONFIG_SECTION_DB, "enable zero metrics");
1068
1069 + config_move(CONFIG_SECTION_LOGS, "error",
1070 + CONFIG_SECTION_LOGS, "daemon");
1071 +
1072 + config_move(CONFIG_SECTION_LOGS, "severity level",
1073 + CONFIG_SECTION_LOGS, "level");
1074 +
1075 + config_move(CONFIG_SECTION_LOGS, "errors to trigger flood protection",
1076 + CONFIG_SECTION_LOGS, "logs to trigger flood protection");
1077 +
1078 + config_move(CONFIG_SECTION_LOGS, "errors flood protection period",
1079 + CONFIG_SECTION_LOGS, "logs flood protection period");
1080 }
1081
1082 static int get_hostname(char *buf, size_t buf_size) {
@@ -1354,6 +1377,7 @@ int pluginsd_parser_unittest(void);
1377 void replication_initialize(void);
1378 void bearer_tokens_init(void);
1379 int unittest_rrdpush_compressions(void);
1380 +int uuid_unittest(void);
1381
1382 int main(int argc, char **argv) {
1383 // initialize the system clocks
@@ -1363,8 +1387,6 @@ int main(int argc, char **argv) {
1387 usec_t started_ut = now_monotonic_usec();
1388 usec_t last_ut = started_ut;
1389 const char *prev_msg = NULL;
1366 - // Initialize stderror avoiding coredump when netdata_log_info() or netdata_log_error() is called
1367 - stderror = stderr;
1390
1391 int i;
1392 int config_loaded = 0;
@@ -1516,6 +1538,8 @@ int main(int argc, char **argv) {
1538 return 1;
1539 if (ctx_unittest())
1540 return 1;
1541 + if (uuid_unittest())
1542 + return 1;
1543 fprintf(stderr, "\n\nALL TESTS PASSED\n\n");
1544 return 0;
1545 }
@@ -1542,6 +1566,10 @@ int main(int argc, char **argv) {
1566 unittest_running = true;
1567 return buffer_unittest();
1568 }
1569 + else if(strcmp(optarg, "uuidtest") == 0) {
1570 + unittest_running = true;
1571 + return uuid_unittest();
1572 + }
1573 #ifdef ENABLE_DBENGINE
1574 else if(strcmp(optarg, "mctest") == 0) {
1575 unittest_running = true;
@@ -1919,10 +1947,10 @@ int main(int argc, char **argv) {
1947 // get log filenames and settings
1948
1949 log_init();
1922 - error_log_limit_unlimited();
1950 + nd_log_limits_unlimited();
1951
1952 // initialize the log files
1925 - open_all_log_files();
1953 + nd_log_initialize();
1954 netdata_log_info("Netdata agent version \""VERSION"\" is starting");
1955
1956 ieee754_doubles = is_system_ieee754_double();
@@ -2103,7 +2131,7 @@ int main(int argc, char **argv) {
2131 // ------------------------------------------------------------------------
2132 // enable log flood protection
2133
2106 - error_log_limit_reset();
2134 + nd_log_limits_reset();
2135
2136 // Load host labels
2137 delta_startup_time("collect host labels");
daemon/signals.c
+7 -7
@@ -203,28 +203,28 @@ void signals_handle(void) {
203
204 switch (signals_waiting[i].action) {
205 case NETDATA_SIGNAL_RELOAD_HEALTH:
206 - error_log_limit_unlimited();
206 + nd_log_limits_unlimited();
207 netdata_log_info("SIGNAL: Received %s. Reloading HEALTH configuration...", name);
208 - error_log_limit_reset();
208 + nd_log_limits_reset();
209 execute_command(CMD_RELOAD_HEALTH, NULL, NULL);
210 break;
211
212 case NETDATA_SIGNAL_SAVE_DATABASE:
213 - error_log_limit_unlimited();
213 + nd_log_limits_unlimited();
214 netdata_log_info("SIGNAL: Received %s. Saving databases...", name);
215 - error_log_limit_reset();
215 + nd_log_limits_reset();
216 execute_command(CMD_SAVE_DATABASE, NULL, NULL);
217 break;
218
219 case NETDATA_SIGNAL_REOPEN_LOGS:
220 - error_log_limit_unlimited();
220 + nd_log_limits_unlimited();
221 netdata_log_info("SIGNAL: Received %s. Reopening all log files...", name);
222 - error_log_limit_reset();
222 + nd_log_limits_reset();
223 execute_command(CMD_REOPEN_LOGS, NULL, NULL);
224 break;
225
226 case NETDATA_SIGNAL_EXIT_CLEANLY:
227 - error_log_limit_unlimited();
227 + nd_log_limits_unlimited();
228 netdata_log_info("SIGNAL: Received %s. Cleaning up to exit...", name);
229 commands_exit();
230 netdata_cleanup_and_exit(0);
daemon/unit_test.c
+3 -3
@@ -2118,7 +2118,7 @@ int test_dbengine(void)
2118 RRDDIM *rd[CHARTS][DIMS];
2119 time_t time_start[REGIONS], time_end[REGIONS];
2120
2121 - error_log_limit_unlimited();
2121 + nd_log_limits_unlimited();
2122 fprintf(stderr, "\nRunning DB-engine test\n");
2123
2124 default_rrd_memory_mode = RRD_MEMORY_MODE_DBENGINE;
@@ -2347,7 +2347,7 @@ void generate_dbengine_dataset(unsigned history_seconds)
2347 (1024 * 1024);
2348 default_rrdeng_disk_quota_mb -= default_rrdeng_disk_quota_mb * EXPECTED_COMPRESSION_RATIO / 100;
2349
2350 - error_log_limit_unlimited();
2350 + nd_log_limits_unlimited();
2351 fprintf(stderr, "Initializing localhost with hostname 'dbengine-dataset'");
2352
2353 host = dbengine_rrdhost_find_or_create("dbengine-dataset");
@@ -2522,7 +2522,7 @@ void dbengine_stress_test(unsigned TEST_DURATION_SEC, unsigned DSET_CHARTS, unsi
2522 unsigned i, j;
2523 time_t time_start, test_duration;
2524
2525 - error_log_limit_unlimited();
2525 + nd_log_limits_unlimited();
2526
2527 if (!TEST_DURATION_SEC)
2528 TEST_DURATION_SEC = 10;
database/contexts/rrdcontext.c
+43 -22
@@ -224,26 +224,31 @@ void rrdcontext_hub_checkpoint_command(void *ptr) {
224 struct ctxs_checkpoint *cmd = ptr;
225
226 if(!rrdhost_check_our_claim_id(cmd->claim_id)) {
227 - netdata_log_error("RRDCONTEXT: received checkpoint command for claim_id '%s', node id '%s', but this is not our claim id. Ours '%s', received '%s'. Ignoring command.",
228 - cmd->claim_id, cmd->node_id,
229 - localhost->aclk_state.claimed_id?localhost->aclk_state.claimed_id:"NOT SET",
230 - cmd->claim_id);
227 + nd_log(NDLS_DAEMON, NDLP_WARNING,
228 + "RRDCONTEXT: received checkpoint command for claim_id '%s', node id '%s', "
229 + "but this is not our claim id. Ours '%s', received '%s'. Ignoring command.",
230 + cmd->claim_id, cmd->node_id,
231 + localhost->aclk_state.claimed_id?localhost->aclk_state.claimed_id:"NOT SET",
232 + cmd->claim_id);
233
234 return;
235 }
236
237 RRDHOST *host = rrdhost_find_by_node_id(cmd->node_id);
238 if(!host) {
237 - netdata_log_error("RRDCONTEXT: received checkpoint command for claim id '%s', node id '%s', but there is no node with such node id here. Ignoring command.",
238 - cmd->claim_id,
239 - cmd->node_id);
239 + nd_log(NDLS_DAEMON, NDLP_WARNING,
240 + "RRDCONTEXT: received checkpoint command for claim id '%s', node id '%s', "
241 + "but there is no node with such node id here. Ignoring command.",
242 + cmd->claim_id, cmd->node_id);
243
244 return;
245 }
246
247 if(rrdhost_flag_check(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)) {
245 - netdata_log_info("RRDCONTEXT: received checkpoint command for claim id '%s', node id '%s', while node '%s' has an active context streaming.",
246 - cmd->claim_id, cmd->node_id, rrdhost_hostname(host));
248 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
249 + "RRDCONTEXT: received checkpoint command for claim id '%s', node id '%s', "
250 + "while node '%s' has an active context streaming.",
251 + cmd->claim_id, cmd->node_id, rrdhost_hostname(host));
252
253 // disable it temporarily, so that our worker will not attempt to send messages in parallel
254 rrdhost_flag_clear(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS);
@@ -252,8 +257,10 @@ void rrdcontext_hub_checkpoint_command(void *ptr) {
257 uint64_t our_version_hash = rrdcontext_version_hash(host);
258
259 if(cmd->version_hash != our_version_hash) {
255 - netdata_log_error("RRDCONTEXT: received version hash %"PRIu64" for host '%s', does not match our version hash %"PRIu64". Sending snapshot of all contexts.",
256 - cmd->version_hash, rrdhost_hostname(host), our_version_hash);
260 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
261 + "RRDCONTEXT: received version hash %"PRIu64" for host '%s', does not match our version hash %"PRIu64". "
262 + "Sending snapshot of all contexts.",
263 + cmd->version_hash, rrdhost_hostname(host), our_version_hash);
264
265 #ifdef ENABLE_ACLK
266 // prepare the snapshot
@@ -275,41 +282,55 @@ void rrdcontext_hub_checkpoint_command(void *ptr) {
282 #endif
283 }
284
278 - internal_error(true, "RRDCONTEXT: host '%s' enabling streaming of contexts", rrdhost_hostname(host));
285 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
286 + "RRDCONTEXT: host '%s' enabling streaming of contexts",
287 + rrdhost_hostname(host));
288 +
289 rrdhost_flag_set(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS);
290 char node_str[UUID_STR_LEN];
291 uuid_unparse_lower(*host->node_id, node_str);
282 - netdata_log_access("ACLK REQ [%s (%s)]: STREAM CONTEXTS ENABLED", node_str, rrdhost_hostname(host));
292 + nd_log(NDLS_ACCESS, NDLP_DEBUG,
293 + "ACLK REQ [%s (%s)]: STREAM CONTEXTS ENABLED",
294 + node_str, rrdhost_hostname(host));
295 }
296
297 void rrdcontext_hub_stop_streaming_command(void *ptr) {
298 struct stop_streaming_ctxs *cmd = ptr;
299
300 if(!rrdhost_check_our_claim_id(cmd->claim_id)) {
289 - netdata_log_error("RRDCONTEXT: received stop streaming command for claim_id '%s', node id '%s', but this is not our claim id. Ours '%s', received '%s'. Ignoring command.",
290 - cmd->claim_id, cmd->node_id,
291 - localhost->aclk_state.claimed_id?localhost->aclk_state.claimed_id:"NOT SET",
292 - cmd->claim_id);
301 + nd_log(NDLS_DAEMON, NDLP_WARNING,
302 + "RRDCONTEXT: received stop streaming command for claim_id '%s', node id '%s', "
303 + "but this is not our claim id. Ours '%s', received '%s'. Ignoring command.",
304 + cmd->claim_id, cmd->node_id,
305 + localhost->aclk_state.claimed_id?localhost->aclk_state.claimed_id:"NOT SET",
306 + cmd->claim_id);
307
308 return;
309 }
310
311 RRDHOST *host = rrdhost_find_by_node_id(cmd->node_id);
312 if(!host) {
299 - netdata_log_error("RRDCONTEXT: received stop streaming command for claim id '%s', node id '%s', but there is no node with such node id here. Ignoring command.",
300 - cmd->claim_id, cmd->node_id);
313 + nd_log(NDLS_DAEMON, NDLP_WARNING,
314 + "RRDCONTEXT: received stop streaming command for claim id '%s', node id '%s', "
315 + "but there is no node with such node id here. Ignoring command.",
316 + cmd->claim_id, cmd->node_id);
317
318 return;
319 }
320
321 if(!rrdhost_flag_check(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS)) {
306 - netdata_log_error("RRDCONTEXT: received stop streaming command for claim id '%s', node id '%s', but node '%s' does not have active context streaming. Ignoring command.",
307 - cmd->claim_id, cmd->node_id, rrdhost_hostname(host));
322 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
323 + "RRDCONTEXT: received stop streaming command for claim id '%s', node id '%s', "
324 + "but node '%s' does not have active context streaming. Ignoring command.",
325 + cmd->claim_id, cmd->node_id, rrdhost_hostname(host));
326
327 return;
328 }
329
312 - internal_error(true, "RRDCONTEXT: host '%s' disabling streaming of contexts", rrdhost_hostname(host));
330 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
331 + "RRDCONTEXT: host '%s' disabling streaming of contexts",
332 + rrdhost_hostname(host));
333 +
334 rrdhost_flag_clear(host, RRDHOST_FLAG_ACLK_STREAM_CONTEXTS);
335 }
336
database/engine/cache.c
+4 -3
@@ -1171,9 +1171,10 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1171 if(all_of_them && !filter) {
1172 pgc_ll_lock(cache, &cache->clean);
1173 if(cache->clean.stats->entries) {
1174 - error_limit_static_global_var(erl, 1, 0);
1175 - error_limit(&erl, "DBENGINE CACHE: cannot free all clean pages, %zu are still in the clean queue",
1176 - cache->clean.stats->entries);
1174 + nd_log_limit_static_global_var(erl, 1, 0);
1175 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_NOTICE,
1176 + "DBENGINE CACHE: cannot free all clean pages, %zu are still in the clean queue",
1177 + cache->clean.stats->entries);
1178 }
1179 pgc_ll_unlock(cache, &cache->clean);
1180 }
database/engine/datafile.c
+8 -3
@@ -479,14 +479,18 @@ int create_new_datafile_pair(struct rrdengine_instance *ctx, bool having_lock)
479 int ret;
480 char path[RRDENG_PATH_MAX];
481
482 - netdata_log_info("DBENGINE: creating new data and journal files in path %s", ctx->config.dbfiles_path);
482 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
483 + "DBENGINE: creating new data and journal files in path %s",
484 + ctx->config.dbfiles_path);
485 +
486 datafile = datafile_alloc_and_init(ctx, 1, fileno);
487 ret = create_data_file(datafile);
488 if(ret)
489 goto error_after_datafile;
490
491 generate_datafilepath(datafile, path, sizeof(path));
489 - netdata_log_info("DBENGINE: created data file \"%s\".", path);
492 + nd_log(NDLS_DAEMON, NDLP_INFO,
493 + "DBENGINE: created data file \"%s\".", path);
494
495 journalfile = journalfile_alloc_and_init(datafile);
496 ret = journalfile_create(journalfile, datafile);
@@ -494,7 +498,8 @@ int create_new_datafile_pair(struct rrdengine_instance *ctx, bool having_lock)
498 goto error_after_journalfile;
499
500 journalfile_v1_generate_path(datafile, path, sizeof(path));
497 - netdata_log_info("DBENGINE: created journal file \"%s\".", path);
501 + nd_log(NDLS_DAEMON, NDLP_INFO,
502 + "DBENGINE: created journal file \"%s\".", path);
503
504 ctx_current_disk_space_increase(ctx, datafile->pos + journalfile->unsafe.pos);
505 datafile_list_insert(ctx, datafile, having_lock);
database/engine/metric.c
+15 -12
@@ -592,27 +592,30 @@ inline void mrg_update_metric_retention_and_granularity_by_uuid(
592 time_t update_every_s, time_t now_s)
593 {
594 if(unlikely(last_time_s > now_s)) {
595 - error_limit_static_global_var(erl, 1, 0);
596 - error_limit(&erl, "DBENGINE JV2: wrong last time on-disk (%ld - %ld, now %ld), "
597 - "fixing last time to now",
598 - first_time_s, last_time_s, now_s);
595 + nd_log_limit_static_global_var(erl, 1, 0);
596 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_WARNING,
597 + "DBENGINE JV2: wrong last time on-disk (%ld - %ld, now %ld), "
598 + "fixing last time to now",
599 + first_time_s, last_time_s, now_s);
600 last_time_s = now_s;
601 }
602
603 if (unlikely(first_time_s > last_time_s)) {
603 - error_limit_static_global_var(erl, 1, 0);
604 - error_limit(&erl, "DBENGINE JV2: wrong first time on-disk (%ld - %ld, now %ld), "
605 - "fixing first time to last time",
606 - first_time_s, last_time_s, now_s);
604 + nd_log_limit_static_global_var(erl, 1, 0);
605 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_WARNING,
606 + "DBENGINE JV2: wrong first time on-disk (%ld - %ld, now %ld), "
607 + "fixing first time to last time",
608 + first_time_s, last_time_s, now_s);
609
610 first_time_s = last_time_s;
611 }
612
613 if (unlikely(first_time_s == 0 || last_time_s == 0)) {
612 - error_limit_static_global_var(erl, 1, 0);
613 - error_limit(&erl, "DBENGINE JV2: zero on-disk timestamps (%ld - %ld, now %ld), "
614 - "using them as-is",
615 - first_time_s, last_time_s, now_s);
614 + nd_log_limit_static_global_var(erl, 1, 0);
615 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_WARNING,
616 + "DBENGINE JV2: zero on-disk timestamps (%ld - %ld, now %ld), "
617 + "using them as-is",
618 + first_time_s, last_time_s, now_s);
619 }
620
621 bool added = false;
database/engine/pdc.c
+5 -5
@@ -772,7 +772,7 @@ VALIDATED_PAGE_DESCRIPTOR validate_page(
772
773 if(unlikely(!vd.is_valid || updated)) {
774 #ifndef NETDATA_INTERNAL_CHECKS
775 - error_limit_static_global_var(erl, 1, 0);
775 + nd_log_limit_static_global_var(erl, 1, 0);
776 #endif
777 char uuid_str[UUID_STR_LEN + 1];
778 uuid_unparse(*uuid, uuid_str);
@@ -788,7 +788,7 @@ VALIDATED_PAGE_DESCRIPTOR validate_page(
788 #ifdef NETDATA_INTERNAL_CHECKS
789 internal_error(true,
790 #else
791 - error_limit(&erl,
791 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR,
792 #endif
793 "DBENGINE: metric '%s' %s invalid page of type %u "
794 "from %ld to %ld (now %ld), update every %ld, page length %zu, entries %zu (flags: %s)",
@@ -808,7 +808,7 @@ VALIDATED_PAGE_DESCRIPTOR validate_page(
808 #ifdef NETDATA_INTERNAL_CHECKS
809 internal_error(true,
810 #else
811 - error_limit(&erl,
811 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR,
812 #endif
813 "DBENGINE: metric '%s' %s page of type %u "
814 "from %ld to %ld (now %ld), update every %ld, page length %zu, entries %zu (flags: %s), "
@@ -915,8 +915,8 @@ static void epdl_extent_loading_error_log(struct rrdengine_instance *ctx, EPDL *
915 if(end_time_s)
916 log_date(end_time_str, LOG_DATE_LENGTH, end_time_s);
917
918 - error_limit_static_global_var(erl, 1, 0);
919 - error_limit(&erl,
918 + nd_log_limit_static_global_var(erl, 1, 0);
919 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR,
920 "DBENGINE: error while reading extent from datafile %u of tier %d, at offset %" PRIu64 " (%u bytes) "
921 "%s from %ld (%s) to %ld (%s) %s%s: "
922 "%s",
database/engine/rrdengine.c
+13 -3
@@ -1478,12 +1478,19 @@ static void *journal_v2_indexing_tp_worker(struct rrdengine_instance *ctx __mayb
1478 spinlock_unlock(&datafile->writers.spinlock);
1479
1480 if(!available) {
1481 - netdata_log_info("DBENGINE: journal file %u needs to be indexed, but it has writers working on it - skipping it for now", datafile->fileno);
1481 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
1482 + "DBENGINE: journal file %u needs to be indexed, but it has writers working on it - "
1483 + "skipping it for now",
1484 + datafile->fileno);
1485 +
1486 datafile = datafile->next;
1487 continue;
1488 }
1489
1486 - netdata_log_info("DBENGINE: journal file %u is ready to be indexed", datafile->fileno);
1490 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1491 + "DBENGINE: journal file %u is ready to be indexed",
1492 + datafile->fileno);
1493 +
1494 pgc_open_cache_to_journal_v2(open_cache, (Word_t) ctx, (int) datafile->fileno, ctx->config.page_type,
1495 journalfile_migrate_to_v2_callback, (void *) datafile->journalfile);
1496
@@ -1496,7 +1503,10 @@ static void *journal_v2_indexing_tp_worker(struct rrdengine_instance *ctx __mayb
1503 }
1504
1505 errno = 0;
1499 - internal_error(count, "DBENGINE: journal indexing done; %u files processed", count);
1506 + if(count)
1507 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1508 + "DBENGINE: journal indexing done; %u files processed",
1509 + count);
1510
1511 worker_is_idle();
1512
database/engine/rrdengineapi.c
+9 -9
@@ -361,12 +361,12 @@ static void rrdeng_store_metric_create_new_page(struct rrdeng_collect_handle *ha
361 #ifdef NETDATA_INTERNAL_CHECKS
362 internal_error(true,
363 #else
364 - error_limit_static_global_var(erl, 1, 0);
365 - error_limit(&erl,
366 - #endif
367 - "DBENGINE: metric '%s' new page from %ld to %ld, update every %ld, has a conflict in main cache "
368 - "with existing %s%s page from %ld to %ld, update every %ld - "
369 - "is it collected more than once?",
364 + nd_log_limit_static_global_var(erl, 1, 0);
365 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_WARNING,
366 +#endif
367 + "DBENGINE: metric '%s' new page from %ld to %ld, update every %ld, has a conflict in main cache "
368 + "with existing %s%s page from %ld to %ld, update every %ld - "
369 + "is it collected more than once?",
370 uuid,
371 page_entry.start_time_s, page_entry.end_time_s, (time_t)page_entry.update_every_s,
372 pgc_is_page_hot(pgc_page) ? "hot" : "not-hot",
@@ -521,8 +521,8 @@ static void store_metric_next_error_log(struct rrdeng_collect_handle *handle __m
521 collect_page_flags_to_buffer(wb, handle->page_flags);
522 }
523
524 - error_limit_static_global_var(erl, 1, 0);
525 - error_limit(&erl,
524 + nd_log_limit_static_global_var(erl, 1, 0);
525 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_NOTICE,
526 "DBENGINE: metric '%s' collected point at %ld, %s last collection at %ld, "
527 "update every %ld, %s page from %ld to %ld, position %u (of %u), flags: %s",
528 uuid,
@@ -535,7 +535,7 @@ static void store_metric_next_error_log(struct rrdeng_collect_handle *handle __m
535 (time_t)(handle->page_end_time_ut / USEC_PER_SEC),
536 handle->page_position, handle->page_entries_max,
537 wb ? buffer_tostring(wb) : ""
538 - );
538 + );
539
540 buffer_free(wb);
541 #else
database/rrd.h
+1 -1
@@ -1023,7 +1023,7 @@ typedef enum __attribute__ ((__packed__)) rrdhost_flags {
1023
1024 #ifdef NETDATA_INTERNAL_CHECKS
1025 #define rrdset_debug(st, fmt, args...) do { if(unlikely(debug_flags & D_RRD_STATS && rrdset_flag_check(st, RRDSET_FLAG_DEBUG))) \
1026 - debug_int(__FILE__, __FUNCTION__, __LINE__, "%s: " fmt, rrdset_name(st), ##args); } while(0)
1026 + netdata_logger(NDLS_DEBUG, NDLP_DEBUG, __FILE__, __FUNCTION__, __LINE__, "%s: " fmt, rrdset_name(st), ##args); } while(0)
1027 #else
1028 #define rrdset_debug(st, fmt, args...) debug_dummy()
1029 #endif
database/rrdcalc.c
+4 -4
@@ -809,10 +809,10 @@ void rrdcalc_delete_alerts_not_matching_host_labels_from_this_host(RRDHOST *host
809 continue;
810
811 if(!rrdlabels_match_simple_pattern_parsed(host->rrdlabels, rc->host_labels_pattern, '=', NULL)) {
812 - netdata_log_health("Health configuration for alarm '%s' cannot be applied, because the host %s does not have the label(s) '%s'",
813 - rrdcalc_name(rc),
814 - rrdhost_hostname(host),
815 - rrdcalc_host_labels(rc));
812 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
813 + "Health configuration for alarm '%s' cannot be applied, "
814 + "because the host %s does not have the label(s) '%s'",
815 + rrdcalc_name(rc), rrdhost_hostname(host), rrdcalc_host_labels(rc));
816
817 rrdcalc_unlink_and_delete(host, rc, false);
818 }
database/rrdfunctions.c
+2 -2
@@ -1007,11 +1007,11 @@ int rrd_function_run(RRDHOST *host, BUFFER *result_wb, int timeout, const char *
1007 // the function can only be executed in async mode
1008 // put the function into the inflight requests
1009
1010 - char uuid_str[UUID_STR_LEN];
1010 + char uuid_str[UUID_COMPACT_STR_LEN];
1011 if(!transaction) {
1012 uuid_t uuid;
1013 uuid_generate_random(uuid);
1014 - uuid_unparse_lower(uuid, uuid_str);
1014 + uuid_unparse_lower_compact(uuid, uuid_str);
1015 transaction = uuid_str;
1016 }
1017
database/rrdhost.c
+167 -102
@@ -80,8 +80,6 @@ static inline void rrdhost_init() {
80 }
81
82 RRDHOST_ACQUIRED *rrdhost_find_and_acquire(const char *machine_guid) {
83 - netdata_log_debug(D_RRD_CALLS, "rrdhost_find_and_acquire() host %s", machine_guid);
84 -
83 return (RRDHOST_ACQUIRED *)dictionary_get_and_acquire_item(rrdhost_root_index, machine_guid);
84 }
85
@@ -116,8 +114,9 @@ static inline RRDHOST *rrdhost_index_add_by_guid(RRDHOST *host) {
114 rrdhost_option_set(host, RRDHOST_OPTION_INDEXED_MACHINE_GUID);
115 else {
116 rrdhost_option_clear(host, RRDHOST_OPTION_INDEXED_MACHINE_GUID);
119 - netdata_log_error("RRDHOST: %s() host with machine guid '%s' is already indexed",
120 - __FUNCTION__, host->machine_guid);
117 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
118 + "RRDHOST: host with machine guid '%s' is already indexed. Not adding it again.",
119 + host->machine_guid);
120 }
121
122 return host;
@@ -126,8 +125,9 @@ static inline RRDHOST *rrdhost_index_add_by_guid(RRDHOST *host) {
125 static void rrdhost_index_del_by_guid(RRDHOST *host) {
126 if(rrdhost_option_check(host, RRDHOST_OPTION_INDEXED_MACHINE_GUID)) {
127 if(!dictionary_del(rrdhost_root_index, host->machine_guid))
129 - netdata_log_error("RRDHOST: %s() failed to delete machine guid '%s' from index",
130 - __FUNCTION__, host->machine_guid);
128 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
129 + "RRDHOST: failed to delete machine guid '%s' from index",
130 + host->machine_guid);
131
132 rrdhost_option_clear(host, RRDHOST_OPTION_INDEXED_MACHINE_GUID);
133 }
@@ -148,8 +148,9 @@ static inline void rrdhost_index_del_hostname(RRDHOST *host) {
148
149 if(rrdhost_option_check(host, RRDHOST_OPTION_INDEXED_HOSTNAME)) {
150 if(!dictionary_del(rrdhost_root_index_hostname, rrdhost_hostname(host)))
151 - netdata_log_error("RRDHOST: %s() failed to delete hostname '%s' from index",
152 - __FUNCTION__, rrdhost_hostname(host));
151 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
152 + "RRDHOST: failed to delete hostname '%s' from index",
153 + rrdhost_hostname(host));
154
155 rrdhost_option_clear(host, RRDHOST_OPTION_INDEXED_HOSTNAME);
156 }
@@ -303,11 +304,11 @@ static RRDHOST *rrdhost_create(
304 int is_localhost,
305 bool archived
306 ) {
306 - netdata_log_debug(D_RRDHOST, "Host '%s': adding with guid '%s'", hostname, guid);
307 -
307 if(memory_mode == RRD_MEMORY_MODE_DBENGINE && !dbengine_enabled) {
309 - netdata_log_error("memory mode 'dbengine' is not enabled, but host '%s' is configured for it. Falling back to 'alloc'",
310 - hostname);
308 + nd_log(NDLS_DAEMON, NDLP_ERR,
309 + "memory mode 'dbengine' is not enabled, but host '%s' is configured for it. Falling back to 'alloc'",
310 + hostname);
311 +
312 memory_mode = RRD_MEMORY_MODE_ALLOC;
313 }
314
@@ -392,7 +393,9 @@ int is_legacy = 1;
393 (host->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE && is_legacy))) {
394 int r = mkdir(host->cache_dir, 0775);
395 if(r != 0 && errno != EEXIST)
395 - netdata_log_error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), host->cache_dir);
396 + nd_log(NDLS_DAEMON, NDLP_CRIT,
397 + "Host '%s': cannot create directory '%s'",
398 + rrdhost_hostname(host), host->cache_dir);
399 }
400 }
401
@@ -418,7 +421,9 @@ int is_legacy = 1;
421 ret = mkdir(dbenginepath, 0775);
422
423 if (ret != 0 && errno != EEXIST)
421 - netdata_log_error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), dbenginepath);
424 + nd_log(NDLS_DAEMON, NDLP_CRIT,
425 + "Host '%s': cannot create directory '%s'",
426 + rrdhost_hostname(host), dbenginepath);
427 else
428 ret = 0; // succeed
429
@@ -459,8 +464,9 @@ int is_legacy = 1;
464 }
465
466 if (ret) { // check legacy or multihost initialization success
462 - netdata_log_error("Host '%s': cannot initialize host with machine guid '%s'. Failed to initialize DB engine at '%s'.",
463 - rrdhost_hostname(host), host->machine_guid, host->cache_dir);
467 + nd_log(NDLS_DAEMON, NDLP_CRIT,
468 + "Host '%s': cannot initialize host with machine guid '%s'. Failed to initialize DB engine at '%s'.",
469 + rrdhost_hostname(host), host->machine_guid, host->cache_dir);
470
471 rrd_wrlock();
472 rrdhost_free___while_having_rrd_wrlock(host, true);
@@ -508,10 +514,13 @@ int is_legacy = 1;
514
515 RRDHOST *t = rrdhost_index_add_by_guid(host);
516 if(t != host) {
511 - netdata_log_error("Host '%s': cannot add host with machine guid '%s' to index. It already exists as host '%s' with machine guid '%s'.",
512 - rrdhost_hostname(host), host->machine_guid, rrdhost_hostname(t), t->machine_guid);
517 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
518 + "Host '%s': cannot add host with machine guid '%s' to index. It already exists as host '%s' with machine guid '%s'.",
519 + rrdhost_hostname(host), host->machine_guid, rrdhost_hostname(t), t->machine_guid);
520 +
521 if (!is_localhost)
522 rrdhost_free___while_having_rrd_wrlock(host, true);
523 +
524 rrd_unlock();
525 return NULL;
526 }
@@ -527,21 +536,22 @@ int is_legacy = 1;
536
537 // ------------------------------------------------------------------------
538
530 - netdata_log_info("Host '%s' (at registry as '%s') with guid '%s' initialized"
531 - ", os '%s'"
532 - ", timezone '%s'"
533 - ", tags '%s'"
534 - ", program_name '%s'"
535 - ", program_version '%s'"
536 - ", update every %d"
537 - ", memory mode %s"
538 - ", history entries %d"
539 - ", streaming %s"
540 - " (to '%s' with api key '%s')"
541 - ", health %s"
542 - ", cache_dir '%s'"
543 - ", alarms default handler '%s'"
544 - ", alarms default recipient '%s'"
539 + nd_log(NDLS_DAEMON, NDLP_INFO,
540 + "Host '%s' (at registry as '%s') with guid '%s' initialized"
541 + ", os '%s'"
542 + ", timezone '%s'"
543 + ", tags '%s'"
544 + ", program_name '%s'"
545 + ", program_version '%s'"
546 + ", update every %d"
547 + ", memory mode %s"
548 + ", history entries %d"
549 + ", streaming %s"
550 + " (to '%s' with api key '%s')"
551 + ", health %s"
552 + ", cache_dir '%s'"
553 + ", alarms default handler '%s'"
554 + ", alarms default recipient '%s'"
555 , rrdhost_hostname(host)
556 , rrdhost_registry_hostname(host)
557 , host->machine_guid
@@ -621,44 +631,56 @@ static void rrdhost_update(RRDHOST *host
631 host->registry_hostname = string_strdupz((registry_hostname && *registry_hostname)?registry_hostname:hostname);
632
633 if(strcmp(rrdhost_hostname(host), hostname) != 0) {
624 - netdata_log_info("Host '%s' has been renamed to '%s'. If this is not intentional it may mean multiple hosts are using the same machine_guid.", rrdhost_hostname(host), hostname);
634 + nd_log(NDLS_DAEMON, NDLP_WARNING,
635 + "Host '%s' has been renamed to '%s'. If this is not intentional it may mean multiple hosts are using the same machine_guid.",
636 + rrdhost_hostname(host), hostname);
637 +
638 rrdhost_init_hostname(host, hostname, true);
639 } else {
640 rrdhost_index_add_hostname(host);
641 }
642
643 if(strcmp(rrdhost_program_name(host), program_name) != 0) {
631 - netdata_log_info("Host '%s' switched program name from '%s' to '%s'", rrdhost_hostname(host), rrdhost_program_name(host), program_name);
644 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
645 + "Host '%s' switched program name from '%s' to '%s'",
646 + rrdhost_hostname(host), rrdhost_program_name(host), program_name);
647 +
648 STRING *t = host->program_name;
649 host->program_name = string_strdupz(program_name);
650 string_freez(t);
651 }
652
653 if(strcmp(rrdhost_program_version(host), program_version) != 0) {
638 - netdata_log_info("Host '%s' switched program version from '%s' to '%s'", rrdhost_hostname(host), rrdhost_program_version(host), program_version);
654 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
655 + "Host '%s' switched program version from '%s' to '%s'",
656 + rrdhost_hostname(host), rrdhost_program_version(host), program_version);
657 +
658 STRING *t = host->program_version;
659 host->program_version = string_strdupz(program_version);
660 string_freez(t);
661 }
662
663 if(host->rrd_update_every != update_every)
645 - netdata_log_error("Host '%s' has an update frequency of %d seconds, but the wanted one is %d seconds. "
646 - "Restart netdata here to apply the new settings.",
647 - rrdhost_hostname(host), host->rrd_update_every, update_every);
664 + nd_log(NDLS_DAEMON, NDLP_WARNING,
665 + "Host '%s' has an update frequency of %d seconds, but the wanted one is %d seconds. "
666 + "Restart netdata here to apply the new settings.",
667 + rrdhost_hostname(host), host->rrd_update_every, update_every);
668
669 if(host->rrd_memory_mode != mode)
650 - netdata_log_error("Host '%s' has memory mode '%s', but the wanted one is '%s'. "
651 - "Restart netdata here to apply the new settings.",
652 - rrdhost_hostname(host),
653 - rrd_memory_mode_name(host->rrd_memory_mode),
654 - rrd_memory_mode_name(mode));
670 + nd_log(NDLS_DAEMON, NDLP_WARNING,
671 + "Host '%s' has memory mode '%s', but the wanted one is '%s'. "
672 + "Restart netdata here to apply the new settings.",
673 + rrdhost_hostname(host),
674 + rrd_memory_mode_name(host->rrd_memory_mode),
675 + rrd_memory_mode_name(mode));
676
677 else if(host->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE && host->rrd_history_entries < history)
657 - netdata_log_error("Host '%s' has history of %d entries, but the wanted one is %ld entries. "
658 - "Restart netdata here to apply the new settings.",
659 - rrdhost_hostname(host),
660 - host->rrd_history_entries,
661 - history);
678 + nd_log(NDLS_DAEMON, NDLP_WARNING,
679 + "Host '%s' has history of %d entries, but the wanted one is %ld entries. "
680 + "Restart netdata here to apply the new settings.",
681 + rrdhost_hostname(host),
682 + host->rrd_history_entries,
683 + history);
684
685 // update host tags
686 rrdhost_init_tags(host, tags);
@@ -700,7 +722,9 @@ static void rrdhost_update(RRDHOST *host
722 ml_host_new(host);
723
724 rrdhost_load_rrdcontext_data(host);
703 - netdata_log_info("Host %s is not in archived mode anymore", rrdhost_hostname(host));
725 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
726 + "Host %s is not in archived mode anymore",
727 + rrdhost_hostname(host));
728 }
729
730 spinlock_unlock(&host->rrdhost_update_lock);
@@ -731,8 +755,6 @@ RRDHOST *rrdhost_find_or_create(
755 , struct rrdhost_system_info *system_info
756 , bool archived
757 ) {
734 - netdata_log_debug(D_RRDHOST, "Searching for host '%s' with guid '%s'", hostname, guid);
735 -
758 RRDHOST *host = rrdhost_find_by_guid(guid);
759 if (unlikely(host && host->rrd_memory_mode != mode && rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED))) {
760
@@ -740,10 +762,11 @@ RRDHOST *rrdhost_find_or_create(
762 return host;
763
764 /* If a legacy memory mode instantiates all dbengine state must be discarded to avoid inconsistencies */
743 - netdata_log_error("Archived host '%s' has memory mode '%s', but the wanted one is '%s'. Discarding archived state.",
744 - rrdhost_hostname(host),
745 - rrd_memory_mode_name(host->rrd_memory_mode),
746 - rrd_memory_mode_name(mode));
765 + nd_log(NDLS_DAEMON, NDLP_INFO,
766 + "Archived host '%s' has memory mode '%s', but the wanted one is '%s'. Discarding archived state.",
767 + rrdhost_hostname(host),
768 + rrd_memory_mode_name(host->rrd_memory_mode),
769 + rrd_memory_mode_name(mode));
770
771 rrd_wrlock();
772 rrdhost_free___while_having_rrd_wrlock(host, true);
@@ -851,18 +874,26 @@ void dbengine_init(char *hostname) {
874 if (read_num > 0 && read_num <= MAX_PAGES_PER_EXTENT)
875 rrdeng_pages_per_extent = read_num;
876 else {
854 - netdata_log_error("Invalid dbengine pages per extent %u given. Using %u.", read_num, rrdeng_pages_per_extent);
877 + nd_log(NDLS_DAEMON, NDLP_WARNING,
878 + "Invalid dbengine pages per extent %u given. Using %u.",
879 + read_num, rrdeng_pages_per_extent);
880 +
881 config_set_number(CONFIG_SECTION_DB, "dbengine pages per extent", rrdeng_pages_per_extent);
882 }
883
884 storage_tiers = config_get_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
885 if(storage_tiers < 1) {
860 - netdata_log_error("At least 1 storage tier is required. Assuming 1.");
886 + nd_log(NDLS_DAEMON, NDLP_WARNING,
887 + "At least 1 storage tier is required. Assuming 1.");
888 +
889 storage_tiers = 1;
890 config_set_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
891 }
892 if(storage_tiers > RRD_STORAGE_TIERS) {
865 - netdata_log_error("Up to %d storage tier are supported. Assuming %d.", RRD_STORAGE_TIERS, RRD_STORAGE_TIERS);
893 + nd_log(NDLS_DAEMON, NDLP_WARNING,
894 + "Up to %d storage tier are supported. Assuming %d.",
895 + RRD_STORAGE_TIERS, RRD_STORAGE_TIERS);
896 +
897 storage_tiers = RRD_STORAGE_TIERS;
898 config_set_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
899 }
@@ -884,7 +915,9 @@ void dbengine_init(char *hostname) {
915
916 int ret = mkdir(dbenginepath, 0775);
917 if (ret != 0 && errno != EEXIST) {
887 - netdata_log_error("DBENGINE on '%s': cannot create directory '%s'", hostname, dbenginepath);
918 + nd_log(NDLS_DAEMON, NDLP_CRIT,
919 + "DBENGINE on '%s': cannot create directory '%s'",
920 + hostname, dbenginepath);
921 break;
922 }
923
@@ -904,9 +937,9 @@ void dbengine_init(char *hostname) {
937 if(grouping_iterations < 2) {
938 grouping_iterations = 2;
939 config_set_number(CONFIG_SECTION_DB, dbengineconfig, grouping_iterations);
907 - netdata_log_error("DBENGINE on '%s': 'dbegnine tier %zu update every iterations' cannot be less than 2. Assuming 2.",
908 - hostname,
909 - tier);
940 + nd_log(NDLS_DAEMON, NDLP_WARNING,
941 + "DBENGINE on '%s': 'dbegnine tier %zu update every iterations' cannot be less than 2. Assuming 2.",
942 + hostname, tier);
943 }
944
945 snprintfz(dbengineconfig, 200, "dbengine tier %zu backfill", tier);
@@ -915,7 +948,10 @@ void dbengine_init(char *hostname) {
948 else if(strcmp(bf, "full") == 0) backfill = RRD_BACKFILL_FULL;
949 else if(strcmp(bf, "none") == 0) backfill = RRD_BACKFILL_NONE;
950 else {
918 - netdata_log_error("DBENGINE: unknown backfill value '%s', assuming 'new'", bf);
951 + nd_log(NDLS_DAEMON, NDLP_WARNING,
952 + "DBENGINE: unknown backfill value '%s', assuming 'new'",
953 + bf);
954 +
955 config_set(CONFIG_SECTION_DB, dbengineconfig, "new");
956 backfill = RRD_BACKFILL_NEW;
957 }
@@ -926,10 +962,10 @@ void dbengine_init(char *hostname) {
962
963 if(tier > 0 && get_tier_grouping(tier) > 65535) {
964 storage_tiers_grouping_iterations[tier] = 1;
929 - netdata_log_error("DBENGINE on '%s': dbengine tier %zu gives aggregation of more than 65535 points of tier 0. Disabling tiers above %zu",
930 - hostname,
931 - tier,
932 - tier);
965 + nd_log(NDLS_DAEMON, NDLP_WARNING,
966 + "DBENGINE on '%s': dbengine tier %zu gives aggregation of more than 65535 points of tier 0. "
967 + "Disabling tiers above %zu",
968 + hostname, tier, tier);
969 break;
970 }
971
@@ -957,21 +993,19 @@ void dbengine_init(char *hostname) {
993 netdata_thread_join(tiers_init[tier].thread, &ptr);
994
995 if(tiers_init[tier].ret != 0) {
960 - netdata_log_error("DBENGINE on '%s': Failed to initialize multi-host database tier %zu on path '%s'",
961 - hostname,
962 - tiers_init[tier].tier,
963 - tiers_init[tier].path);
996 + nd_log(NDLS_DAEMON, NDLP_ERR,
997 + "DBENGINE on '%s': Failed to initialize multi-host database tier %zu on path '%s'",
998 + hostname, tiers_init[tier].tier, tiers_init[tier].path);
999 }
1000 else if(created_tiers == tier)
1001 created_tiers++;
1002 }
1003
1004 if(created_tiers && created_tiers < storage_tiers) {
970 - netdata_log_error("DBENGINE on '%s': Managed to create %zu tiers instead of %zu. Continuing with %zu available.",
971 - hostname,
972 - created_tiers,
973 - storage_tiers,
974 - created_tiers);
1005 + nd_log(NDLS_DAEMON, NDLP_WARNING,
1006 + "DBENGINE on '%s': Managed to create %zu tiers instead of %zu. Continuing with %zu available.",
1007 + hostname, created_tiers, storage_tiers, created_tiers);
1008 +
1009 storage_tiers = created_tiers;
1010 }
1011 else if(!created_tiers)
@@ -984,7 +1018,10 @@ void dbengine_init(char *hostname) {
1018 #else
1019 storage_tiers = config_get_number(CONFIG_SECTION_DB, "storage tiers", 1);
1020 if(storage_tiers != 1) {
987 - netdata_log_error("DBENGINE is not available on '%s', so only 1 database tier can be supported.", hostname);
1021 + nd_log(NDLS_DAEMON, NDLP_WARNING,
1022 + "DBENGINE is not available on '%s', so only 1 database tier can be supported.",
1023 + hostname);
1024 +
1025 storage_tiers = 1;
1026 config_set_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
1027 }
@@ -1000,7 +1037,9 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info, bool unitt
1037 set_late_global_environment(system_info);
1038 fatal("Failed to initialize SQLite");
1039 }
1003 - netdata_log_info("Skipping SQLITE metadata initialization since memory mode is not dbengine");
1040 +
1041 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1042 + "Skipping SQLITE metadata initialization since memory mode is not dbengine");
1043 }
1044
1045 if (unlikely(sql_init_context_database(system_info ? 0 : 1))) {
@@ -1015,23 +1054,28 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info, bool unitt
1054 rrdpush_init();
1055
1056 if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE || rrdpush_receiver_needs_dbengine()) {
1018 - netdata_log_info("DBENGINE: Initializing ...");
1057 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1058 + "DBENGINE: Initializing ...");
1059 +
1060 dbengine_init(hostname);
1061 }
1021 - else {
1022 - netdata_log_info("DBENGINE: Not initializing ...");
1062 + else
1063 storage_tiers = 1;
1024 - }
1064
1065 if (!dbengine_enabled) {
1066 if (storage_tiers > 1) {
1028 - netdata_log_error("dbengine is not enabled, but %zu tiers have been requested. Resetting tiers to 1",
1029 - storage_tiers);
1067 + nd_log(NDLS_DAEMON, NDLP_WARNING,
1068 + "dbengine is not enabled, but %zu tiers have been requested. Resetting tiers to 1",
1069 + storage_tiers);
1070 +
1071 storage_tiers = 1;
1072 }
1073
1074 if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
1034 - netdata_log_error("dbengine is not enabled, but it has been given as the default db mode. Resetting db mode to alloc");
1075 + nd_log(NDLS_DAEMON, NDLP_WARNING,
1076 + "dbengine is not enabled, but it has been given as the default db mode. "
1077 + "Resetting db mode to alloc");
1078 +
1079 default_rrd_memory_mode = RRD_MEMORY_MODE_ALLOC;
1080 }
1081 }
@@ -1040,7 +1084,6 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info, bool unitt
1084 if(!unittest)
1085 metadata_sync_init();
1086
1043 - netdata_log_debug(D_RRDHOST, "Initializing localhost with hostname '%s'", hostname);
1087 localhost = rrdhost_create(
1088 hostname
1089 , registry_get_this_machine_hostname()
@@ -1177,7 +1220,9 @@ void rrdhost_free___while_having_rrd_wrlock(RRDHOST *host, bool force) {
1220 if(!host) return;
1221
1222 if (netdata_exit || force) {
1180 - netdata_log_info("RRD: 'host:%s' freeing memory...", rrdhost_hostname(host));
1223 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1224 + "RRD: 'host:%s' freeing memory...",
1225 + rrdhost_hostname(host));
1226
1227 // ------------------------------------------------------------------------
1228 // first remove it from the indexes, so that it will not be discoverable
@@ -1243,7 +1288,10 @@ void rrdhost_free___while_having_rrd_wrlock(RRDHOST *host, bool force) {
1288 #endif
1289
1290 if (!netdata_exit && !force) {
1246 - netdata_log_info("RRD: 'host:%s' is now in archive mode...", rrdhost_hostname(host));
1291 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1292 + "RRD: 'host:%s' is now in archive mode...",
1293 + rrdhost_hostname(host));
1294 +
1295 rrdhost_flag_set(host, RRDHOST_FLAG_ARCHIVED | RRDHOST_FLAG_ORPHAN);
1296 return;
1297 }
@@ -1313,7 +1361,9 @@ void rrd_finalize_collection_for_all_hosts(void) {
1361 void rrdhost_save_charts(RRDHOST *host) {
1362 if(!host) return;
1363
1316 - netdata_log_info("RRD: 'host:%s' saving / closing database...", rrdhost_hostname(host));
1364 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1365 + "RRD: 'host:%s' saving / closing database...",
1366 + rrdhost_hostname(host));
1367
1368 RRDSET *st;
1369
@@ -1442,7 +1492,9 @@ static void rrdhost_load_config_labels(void) {
1492 int status = config_load(NULL, 1, CONFIG_SECTION_HOST_LABEL);
1493 if(!status) {
1494 char *filename = CONFIG_DIR "/" CONFIG_FILENAME;
1445 - netdata_log_error("RRDLABEL: Cannot reload the configuration file '%s', using labels in memory", filename);
1495 + nd_log(NDLS_DAEMON, NDLP_WARNING,
1496 + "RRDLABEL: Cannot reload the configuration file '%s', using labels in memory",
1497 + filename);
1498 }
1499
1500 struct section *co = appconfig_get_section(&netdata_config, CONFIG_SECTION_HOST_LABEL);
@@ -1462,12 +1514,13 @@ static void rrdhost_load_kubernetes_labels(void) {
1514 sprintf(label_script, "%s/%s", netdata_configured_primary_plugins_dir, "get-kubernetes-labels.sh");
1515
1516 if (unlikely(access(label_script, R_OK) != 0)) {
1465 - netdata_log_error("Kubernetes pod label fetching script %s not found.",label_script);
1517 + nd_log(NDLS_DAEMON, NDLP_ERR,
1518 + "Kubernetes pod label fetching script %s not found.",
1519 + label_script);
1520 +
1521 return;
1522 }
1523
1469 - netdata_log_debug(D_RRDHOST, "Attempting to fetch external labels via %s", label_script);
1470 -
1524 pid_t pid;
1525 FILE *fp_child_input;
1526 FILE *fp_child_output = netdata_popen(label_script, &pid, &fp_child_input);
@@ -1481,7 +1534,9 @@ static void rrdhost_load_kubernetes_labels(void) {
1534 // Here we'll inform with an ERROR that the script failed, show whatever (if anything) was added to the list of labels, free the memory and set the return to null
1535 int rc = netdata_pclose(fp_child_input, fp_child_output, pid);
1536 if(rc)
1484 - netdata_log_error("%s exited abnormally. Failed to get kubernetes labels.", label_script);
1537 + nd_log(NDLS_DAEMON, NDLP_ERR,
1538 + "%s exited abnormally. Failed to get kubernetes labels.",
1539 + label_script);
1540 }
1541
1542 void reload_host_labels(void) {
@@ -1501,7 +1556,9 @@ void reload_host_labels(void) {
1556 }
1557
1558 void rrdhost_finalize_collection(RRDHOST *host) {
1504 - netdata_log_info("RRD: 'host:%s' stopping data collection...", rrdhost_hostname(host));
1559 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1560 + "RRD: 'host:%s' stopping data collection...",
1561 + rrdhost_hostname(host));
1562
1563 RRDSET *st;
1564 rrdset_foreach_read(st, host)
@@ -1515,7 +1572,9 @@ void rrdhost_finalize_collection(RRDHOST *host) {
1572 void rrdhost_delete_charts(RRDHOST *host) {
1573 if(!host) return;
1574
1518 - netdata_log_info("RRD: 'host:%s' deleting disk files...", rrdhost_hostname(host));
1575 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1576 + "RRD: 'host:%s' deleting disk files...",
1577 + rrdhost_hostname(host));
1578
1579 RRDSET *st;
1580
@@ -1523,8 +1582,8 @@ void rrdhost_delete_charts(RRDHOST *host) {
1582 // we get a write lock
1583 // to ensure only one thread is saving the database
1584 rrdset_foreach_write(st, host){
1526 - rrdset_delete_files(st);
1527 - }
1585 + rrdset_delete_files(st);
1586 + }
1587 rrdset_foreach_done(st);
1588 }
1589
@@ -1537,7 +1596,9 @@ void rrdhost_delete_charts(RRDHOST *host) {
1596 void rrdhost_cleanup_charts(RRDHOST *host) {
1597 if(!host) return;
1598
1540 - netdata_log_info("RRD: 'host:%s' cleaning up disk files...", rrdhost_hostname(host));
1599 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1600 + "RRD: 'host:%s' cleaning up disk files...",
1601 + rrdhost_hostname(host));
1602
1603 RRDSET *st;
1604 uint32_t rrdhost_delete_obsolete_charts = rrdhost_option_check(host, RRDHOST_OPTION_DELETE_OBSOLETE_CHARTS);
@@ -1564,7 +1625,9 @@ void rrdhost_cleanup_charts(RRDHOST *host) {
1625 // RRDHOST - save all hosts to disk
1626
1627 void rrdhost_save_all(void) {
1567 - netdata_log_info("RRD: saving databases [%zu hosts(s)]...", rrdhost_hosts_available());
1628 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1629 + "RRD: saving databases [%zu hosts(s)]...",
1630 + rrdhost_hosts_available());
1631
1632 rrd_rdlock();
1633
@@ -1579,7 +1642,9 @@ void rrdhost_save_all(void) {
1642 // RRDHOST - save or delete all hosts from disk
1643
1644 void rrdhost_cleanup_all(void) {
1582 - netdata_log_info("RRD: cleaning up database [%zu hosts(s)]...", rrdhost_hosts_available());
1645 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1646 + "RRD: cleaning up database [%zu hosts(s)]...",
1647 + rrdhost_hosts_available());
1648
1649 rrd_rdlock();
1650
database/rrdset.c
+9
@@ -1323,6 +1323,14 @@ void rrddim_store_metric_with_trace(RRDDIM *rd, usec_t point_end_time_ut, NETDAT
1323 #else // !NETDATA_LOG_COLLECTION_ERRORS
1324 void rrddim_store_metric(RRDDIM *rd, usec_t point_end_time_ut, NETDATA_DOUBLE n, SN_FLAGS flags) {
1325 #endif // !NETDATA_LOG_COLLECTION_ERRORS
1326 +
1327 + static __thread struct log_stack_entry lgs[] = {
1328 + [0] = ND_LOG_FIELD_STR(NDF_NIDL_DIMENSION, NULL),
1329 + [1] = ND_LOG_FIELD_END(),
1330 + };
1331 + lgs[0].str = rd->id;
1332 + log_stack_push(lgs);
1333 +
1334 #ifdef NETDATA_LOG_COLLECTION_ERRORS
1335 rd->rrddim_store_metric_count++;
1336
@@ -1384,6 +1392,7 @@ void rrddim_store_metric(RRDDIM *rd, usec_t point_end_time_ut, NETDATA_DOUBLE n,
1392 }
1393
1394 rrdcontext_collected_rrddim(rd);
1395 + log_stack_pop(&lgs);
1396 }
1397
1398 void store_metric_collection_completed() {
database/sqlite/sqlite_aclk_alert.c
+28 -30
@@ -267,7 +267,7 @@ void aclk_push_alert_event(struct aclk_sync_cfg_t *wc)
267 int rc;
268
269 if (unlikely(!wc->alert_updates)) {
270 - netdata_log_access(
270 + nd_log(NDLS_ACCESS, NDLP_NOTICE,
271 "ACLK STA [%s (%s)]: Ignoring alert push event, updates have been turned off for this node.",
272 wc->node_id,
273 wc->host ? rrdhost_hostname(wc->host) : "N/A");
@@ -424,7 +424,7 @@ void aclk_push_alert_event(struct aclk_sync_cfg_t *wc)
424
425 } else {
426 if (wc->alerts_log_first_sequence_id)
427 - netdata_log_access(
427 + nd_log(NDLS_ACCESS, NDLP_DEBUG,
428 "ACLK RES [%s (%s)]: ALERTS SENT from %" PRIu64 " to %" PRIu64 "",
429 wc->node_id,
430 wc->host ? rrdhost_hostname(wc->host) : "N/A",
@@ -523,8 +523,11 @@ void aclk_send_alarm_configuration(char *config_hash)
523 if (unlikely(!wc))
524 return;
525
526 - netdata_log_access(
527 - "ACLK REQ [%s (%s)]: Request to send alert config %s.", wc->node_id, rrdhost_hostname(wc->host), config_hash);
526 + nd_log(NDLS_ACCESS, NDLP_DEBUG,
527 + "ACLK REQ [%s (%s)]: Request to send alert config %s.",
528 + wc->node_id,
529 + wc->host ? rrdhost_hostname(wc->host) : "N/A",
530 + config_hash);
531
532 aclk_push_alert_config(wc->node_id, config_hash);
533 }
@@ -634,13 +637,13 @@ int aclk_push_alert_config_event(char *node_id __maybe_unused, char *config_hash
637 }
638
639 if (likely(p_alarm_config.cfg_hash)) {
637 - netdata_log_access("ACLK RES [%s (%s)]: Sent alert config %s.", wc->node_id, wc->host ? rrdhost_hostname(wc->host) : "N/A", config_hash);
640 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "ACLK RES [%s (%s)]: Sent alert config %s.", wc->node_id, wc->host ? rrdhost_hostname(wc->host) : "N/A", config_hash);
641 aclk_send_provide_alarm_cfg(&p_alarm_config);
642 freez(p_alarm_config.cfg_hash);
643 destroy_aclk_alarm_configuration(&alarm_config);
644 }
645 else
643 - netdata_log_access("ACLK STA [%s (%s)]: Alert config for %s not found.", wc->node_id, wc->host ? rrdhost_hostname(wc->host) : "N/A", config_hash);
646 + nd_log(NDLS_ACCESS, NDLP_WARNING, "ACLK STA [%s (%s)]: Alert config for %s not found.", wc->node_id, wc->host ? rrdhost_hostname(wc->host) : "N/A", config_hash);
647
648 bind_fail:
649 rc = sqlite3_finalize(res);
@@ -669,20 +672,15 @@ void aclk_start_alert_streaming(char *node_id, bool resets)
672 return;
673
674 if (unlikely(!host->health.health_enabled)) {
672 - netdata_log_access(
673 - "ACLK STA [%s (N/A)]: Ignoring request to stream alert state changes, health is disabled.", node_id);
675 + nd_log(NDLS_ACCESS, NDLP_NOTICE, "ACLK STA [%s (N/A)]: Ignoring request to stream alert state changes, health is disabled.", node_id);
676 return;
677 }
678
679 if (resets) {
678 - netdata_log_access(
679 - "ACLK REQ [%s (%s)]: STREAM ALERTS ENABLED (RESET REQUESTED)",
680 - node_id,
681 - wc->host ? rrdhost_hostname(wc->host) : "N/A");
680 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "ACLK REQ [%s (%s)]: STREAM ALERTS ENABLED (RESET REQUESTED)", node_id, wc->host ? rrdhost_hostname(wc->host) : "N/A");
681 sql_queue_existing_alerts_to_aclk(host);
682 } else
684 - netdata_log_access(
685 - "ACLK REQ [%s (%s)]: STREAM ALERTS ENABLED", node_id, wc->host ? rrdhost_hostname(wc->host) : "N/A");
683 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "ACLK REQ [%s (%s)]: STREAM ALERTS ENABLED", node_id, wc->host ? rrdhost_hostname(wc->host) : "N/A");
684
685 wc->alert_updates = 1;
686 wc->alert_queue_removed = SEND_REMOVED_AFTER_HEALTH_LOOPS;
@@ -725,7 +723,7 @@ void sql_process_queue_removed_alerts_to_aclk(char *node_id)
723
724 rc = execute_insert(res);
725 if (likely(rc == SQLITE_DONE)) {
728 - netdata_log_access("ACLK STA [%s (%s)]: QUEUED REMOVED ALERTS", wc->node_id, rrdhost_hostname(wc->host));
726 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "ACLK STA [%s (%s)]: QUEUED REMOVED ALERTS", wc->node_id, rrdhost_hostname(wc->host));
727 rrdhost_flag_set(wc->host, RRDHOST_FLAG_ACLK_STREAM_ALERTS);
728 wc->alert_queue_removed = 0;
729 }
@@ -758,15 +756,15 @@ void aclk_process_send_alarm_snapshot(char *node_id, char *claim_id __maybe_unus
756
757 RRDHOST *host = find_host_by_node_id(node_id);
758 if (unlikely(!host || !(wc = host->aclk_config))) {
761 - netdata_log_access("ACLK STA [%s (N/A)]: ACLK node id does not exist", node_id);
759 + nd_log(NDLS_ACCESS, NDLP_WARNING, "ACLK STA [%s (N/A)]: ACLK node id does not exist", node_id);
760 return;
761 }
762
765 - netdata_log_access(
766 - "IN [%s (%s)]: Request to send alerts snapshot, snapshot_uuid %s",
767 - node_id,
768 - wc->host ? rrdhost_hostname(wc->host) : "N/A",
769 - snapshot_uuid);
763 + nd_log(NDLS_ACCESS, NDLP_DEBUG,
764 + "IN [%s (%s)]: Request to send alerts snapshot, snapshot_uuid %s",
765 + node_id,
766 + wc->host ? rrdhost_hostname(wc->host) : "N/A",
767 + snapshot_uuid);
768
769 if (wc->alerts_snapshot_uuid && !strcmp(wc->alerts_snapshot_uuid,snapshot_uuid))
770 return;
@@ -855,7 +853,7 @@ void aclk_push_alert_snapshot_event(char *node_id __maybe_unused)
853 RRDHOST *host = find_host_by_node_id(node_id);
854
855 if (unlikely(!host)) {
858 - netdata_log_access("AC [%s (N/A)]: Node id not found", node_id);
856 + nd_log(NDLS_ACCESS, NDLP_WARNING, "AC [%s (N/A)]: Node id not found", node_id);
857 freez(node_id);
858 return;
859 }
@@ -865,7 +863,7 @@ void aclk_push_alert_snapshot_event(char *node_id __maybe_unused)
863
864 // we perhaps we don't need this for snapshots
865 if (unlikely(!wc->alert_updates)) {
868 - netdata_log_access(
866 + nd_log(NDLS_ACCESS, NDLP_NOTICE,
867 "ACLK STA [%s (%s)]: Ignoring alert snapshot event, updates have been turned off for this node.",
868 wc->node_id,
869 wc->host ? rrdhost_hostname(wc->host) : "N/A");
@@ -879,7 +877,7 @@ void aclk_push_alert_snapshot_event(char *node_id __maybe_unused)
877 if (unlikely(!claim_id))
878 return;
879
882 - netdata_log_access("ACLK REQ [%s (%s)]: Sending alerts snapshot, snapshot_uuid %s", wc->node_id, rrdhost_hostname(wc->host), wc->alerts_snapshot_uuid);
880 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "ACLK REQ [%s (%s)]: Sending alerts snapshot, snapshot_uuid %s", wc->node_id, rrdhost_hostname(wc->host), wc->alerts_snapshot_uuid);
881
882 uint32_t cnt = 0;
883
@@ -1057,9 +1055,9 @@ void aclk_send_alarm_checkpoint(char *node_id, char *claim_id __maybe_unused)
1055 RRDHOST *host = find_host_by_node_id(node_id);
1056
1057 if (unlikely(!host || !(wc = host->aclk_config)))
1060 - netdata_log_access("ACLK REQ [%s (N/A)]: ALERTS CHECKPOINT REQUEST RECEIVED FOR INVALID NODE", node_id);
1058 + nd_log(NDLS_ACCESS, NDLP_WARNING, "ACLK REQ [%s (N/A)]: ALERTS CHECKPOINT REQUEST RECEIVED FOR INVALID NODE", node_id);
1059 else {
1062 - netdata_log_access("ACLK REQ [%s (%s)]: ALERTS CHECKPOINT REQUEST RECEIVED", node_id, rrdhost_hostname(host));
1060 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "ACLK REQ [%s (%s)]: ALERTS CHECKPOINT REQUEST RECEIVED", node_id, rrdhost_hostname(host));
1061 wc->alert_checkpoint_req = SEND_CHECKPOINT_AFTER_HEALTH_LOOPS;
1062 }
1063 }
@@ -1087,14 +1085,14 @@ void aclk_push_alarm_checkpoint(RRDHOST *host __maybe_unused)
1085 #ifdef ENABLE_ACLK
1086 struct aclk_sync_cfg_t *wc = host->aclk_config;
1087 if (unlikely(!wc)) {
1090 - netdata_log_access("ACLK REQ [%s (N/A)]: ALERTS CHECKPOINT REQUEST RECEIVED FOR INVALID NODE", rrdhost_hostname(host));
1088 + nd_log(NDLS_ACCESS, NDLP_WARNING, "ACLK REQ [%s (N/A)]: ALERTS CHECKPOINT REQUEST RECEIVED FOR INVALID NODE", rrdhost_hostname(host));
1089 return;
1090 }
1091
1092 if (rrdhost_flag_check(host, RRDHOST_FLAG_ACLK_STREAM_ALERTS)) {
1093 //postpone checkpoint send
1094 wc->alert_checkpoint_req += 3;
1097 - netdata_log_access("ACLK REQ [%s (N/A)]: ALERTS CHECKPOINT POSTPONED", rrdhost_hostname(host));
1095 + nd_log(NDLS_ACCESS, NDLP_NOTICE, "ACLK REQ [%s (N/A)]: ALERTS CHECKPOINT POSTPONED", rrdhost_hostname(host));
1096 return;
1097 }
1098
@@ -1157,9 +1155,9 @@ void aclk_push_alarm_checkpoint(RRDHOST *host __maybe_unused)
1155
1156 aclk_send_provide_alarm_checkpoint(&alarm_checkpoint);
1157 freez(claim_id);
1160 - netdata_log_access("ACLK RES [%s (%s)]: ALERTS CHECKPOINT SENT", wc->node_id, rrdhost_hostname(host));
1158 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "ACLK RES [%s (%s)]: ALERTS CHECKPOINT SENT", wc->node_id, rrdhost_hostname(host));
1159 } else
1162 - netdata_log_access("ACLK RES [%s (%s)]: FAILED TO CREATE ALERTS CHECKPOINT HASH", wc->node_id, rrdhost_hostname(host));
1160 + nd_log(NDLS_ACCESS, NDLP_ERR, "ACLK RES [%s (%s)]: FAILED TO CREATE ALERTS CHECKPOINT HASH", wc->node_id, rrdhost_hostname(host));
1161
1162 wc->alert_checkpoint_req = 0;
1163 buffer_free(alarms_to_hash);
database/sqlite/sqlite_aclk_node.c
+5 -4
@@ -43,7 +43,7 @@ static void build_node_collectors(RRDHOST *host)
43 dictionary_destroy(dict);
44 freez(upd_node_collectors.claim_id);
45
46 - netdata_log_access("ACLK RES [%s (%s)]: NODE COLLECTORS SENT", wc->node_id, rrdhost_hostname(host));
46 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "ACLK RES [%s (%s)]: NODE COLLECTORS SENT", wc->node_id, rrdhost_hostname(host));
47 }
48
49 static void build_node_info(RRDHOST *host)
@@ -103,7 +103,7 @@ static void build_node_info(RRDHOST *host)
103 node_info.data.host_labels_ptr = host->rrdlabels;
104
105 aclk_update_node_info(&node_info);
106 - netdata_log_access("ACLK RES [%s (%s)]: NODE INFO SENT for guid [%s] (%s)", wc->node_id, rrdhost_hostname(wc->host), host->machine_guid, wc->host == localhost ? "parent" : "child");
106 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "ACLK RES [%s (%s)]: NODE INFO SENT for guid [%s] (%s)", wc->node_id, rrdhost_hostname(wc->host), host->machine_guid, wc->host == localhost ? "parent" : "child");
107
108 rrd_unlock();
109 freez(node_info.claim_id);
@@ -169,8 +169,9 @@ void aclk_check_node_info_and_collectors(void)
169 dfe_done(host);
170
171 if (context_loading || replicating) {
172 - error_limit_static_thread_var(erl, 10, 100 * USEC_PER_MS);
173 - error_limit(&erl, "%zu nodes loading contexts, %zu replicating data", context_loading, replicating);
172 + nd_log_limit_static_thread_var(erl, 10, 100 * USEC_PER_MS);
173 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_INFO,
174 + "%zu nodes loading contexts, %zu replicating data", context_loading, replicating);
175 }
176 }
177
database/sqlite/sqlite_health.c
+3 -1
@@ -914,7 +914,9 @@ void sql_health_alarm_log_load(RRDHOST *host)
914 if (unlikely(!host->health_log.next_alarm_id || host->health_log.next_alarm_id <= host->health_max_alarm_id))
915 host->health_log.next_alarm_id = host->health_max_alarm_id + 1;
916
917 - netdata_log_health("[%s]: Table health_log, loaded %zd alarm entries, errors in %zd entries.", rrdhost_hostname(host), loaded, errored);
917 + nd_log(NDLS_DAEMON, errored ? NDLP_WARNING : NDLP_DEBUG,
918 + "[%s]: Table health_log, loaded %zd alarm entries, errors in %zd entries.",
919 + rrdhost_hostname(host), loaded, errored);
920
921 ret = sqlite3_finalize(res);
922 if (unlikely(ret != SQLITE_OK))
database/sqlite/sqlite_metadata.c
+14 -14
@@ -1144,7 +1144,7 @@ void vacuum_database(sqlite3 *database, const char *db_alias, int threshold, int
1144 if (free_pages > (total_pages * threshold / 100)) {
1145
1146 int do_free_pages = (int) (free_pages * vacuum_pc / 100);
1147 - netdata_log_info("%s: Freeing %d database pages", db_alias, do_free_pages);
1147 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "%s: Freeing %d database pages", db_alias, do_free_pages);
1148
1149 char sql[128];
1150 snprintfz(sql, 127, "PRAGMA incremental_vacuum(%d)", do_free_pages);
@@ -1258,7 +1258,7 @@ static void start_all_host_load_context(uv_work_t *req __maybe_unused)
1258 RRDHOST *host;
1259
1260 size_t max_threads = MIN(get_netdata_cpus() / 2, 6);
1261 - netdata_log_info("METADATA: Using %zu threads for context loading", max_threads);
1261 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "METADATA: Using %zu threads for context loading", max_threads);
1262 struct host_context_load_thread *hclt = callocz(max_threads, sizeof(*hclt));
1263
1264 size_t thread_index;
@@ -1291,7 +1291,7 @@ static void start_all_host_load_context(uv_work_t *req __maybe_unused)
1291 cleanup_finished_threads(hclt, max_threads, true);
1292 freez(hclt);
1293 usec_t ended_ut = now_monotonic_usec(); (void)ended_ut;
1294 - netdata_log_info("METADATA: host contexts loaded in %0.2f ms", (double)(ended_ut - started_ut) / USEC_PER_MS);
1294 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "METADATA: host contexts loaded in %0.2f ms", (double)(ended_ut - started_ut) / USEC_PER_MS);
1295
1296 worker_is_idle();
1297 }
@@ -1556,7 +1556,7 @@ static void metadata_event_loop(void *arg)
1556 wc->timer_req.data = wc;
1557 fatal_assert(0 == uv_timer_start(&wc->timer_req, timer_cb, TIMER_INITIAL_PERIOD_MS, TIMER_REPEAT_PERIOD_MS));
1558
1559 - netdata_log_info("Starting metadata sync thread");
1559 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "Starting metadata sync thread");
1560
1561 struct metadata_cmd cmd;
1562 memset(&cmd, 0, sizeof(cmd));
@@ -1684,7 +1684,7 @@ static void metadata_event_loop(void *arg)
1684 freez(loop);
1685 worker_unregister();
1686
1687 - netdata_log_info("Shutting down event loop");
1687 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "Shutting down event loop");
1688 completion_mark_complete(&wc->start_stop_complete);
1689 if (wc->scan_complete) {
1690 completion_destroy(wc->scan_complete);
@@ -1710,15 +1710,15 @@ void metadata_sync_shutdown(void)
1710
1711 struct metadata_cmd cmd;
1712 memset(&cmd, 0, sizeof(cmd));
1713 - netdata_log_info("METADATA: Sending a shutdown command");
1713 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "METADATA: Sending a shutdown command");
1714 cmd.opcode = METADATA_SYNC_SHUTDOWN;
1715 metadata_enq_cmd(&metasync_worker, &cmd);
1716
1717 /* wait for metadata thread to shut down */
1718 - netdata_log_info("METADATA: Waiting for shutdown ACK");
1718 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "METADATA: Waiting for shutdown ACK");
1719 completion_wait_for(&metasync_worker.start_stop_complete);
1720 completion_destroy(&metasync_worker.start_stop_complete);
1721 - netdata_log_info("METADATA: Shutdown complete");
1721 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "METADATA: Shutdown complete");
1722 }
1723
1724 void metadata_sync_shutdown_prepare(void)
@@ -1735,20 +1735,20 @@ void metadata_sync_shutdown_prepare(void)
1735 completion_init(compl);
1736 __atomic_store_n(&wc->scan_complete, compl, __ATOMIC_RELAXED);
1737
1738 - netdata_log_info("METADATA: Sending a scan host command");
1738 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "METADATA: Sending a scan host command");
1739 uint32_t max_wait_iterations = 2000;
1740 while (unlikely(metadata_flag_check(&metasync_worker, METADATA_FLAG_PROCESSING)) && max_wait_iterations--) {
1741 if (max_wait_iterations == 1999)
1742 - netdata_log_info("METADATA: Current worker is running; waiting to finish");
1742 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "METADATA: Current worker is running; waiting to finish");
1743 sleep_usec(1000);
1744 }
1745
1746 cmd.opcode = METADATA_SCAN_HOSTS;
1747 metadata_enq_cmd(&metasync_worker, &cmd);
1748
1749 - netdata_log_info("METADATA: Waiting for host scan completion");
1749 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "METADATA: Waiting for host scan completion");
1750 completion_wait_for(wc->scan_complete);
1751 - netdata_log_info("METADATA: Host scan complete; can continue with shutdown");
1751 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "METADATA: Host scan complete; can continue with shutdown");
1752 }
1753
1754 // -------------------------------------------------------------
@@ -1766,7 +1766,7 @@ void metadata_sync_init(void)
1766 completion_wait_for(&wc->start_stop_complete);
1767 completion_destroy(&wc->start_stop_complete);
1768
1769 - netdata_log_info("SQLite metadata sync initialization complete");
1769 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "SQLite metadata sync initialization complete");
1770 }
1771
1772
@@ -1825,7 +1825,7 @@ void metadata_queue_load_host_context(RRDHOST *host)
1825 if (unlikely(!metasync_worker.loop))
1826 return;
1827 queue_metadata_cmd(METADATA_LOAD_HOST_CONTEXT, host, NULL);
1828 - netdata_log_info("Queued command to load host contexts");
1828 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "Queued command to load host contexts");
1829 }
1830
1831 //
exporting/aws_kinesis/aws_kinesis.c
+2 -2
@@ -153,8 +153,8 @@ void aws_kinesis_connector_worker(void *instance_p)
153 char error_message[ERROR_LINE_MAX + 1] = "";
154
155 netdata_log_debug(D_EXPORTING,
156 - "EXPORTING: kinesis_put_record(): dest = %s, id = %s, key = %s, stream = %s, partition_key = %s, \ "
157 - " buffer = %zu, record = %zu",
156 + "EXPORTING: kinesis_put_record(): dest = %s, id = %s, key = %s, stream = %s, partition_key = %s, "
157 + "buffer = %zu, record = %zu",
158 instance->config.destination,
159 connector_specific_config->auth_key_id,
160 connector_specific_config->secure_key,
health/health.c
+83 -25
@@ -82,10 +82,13 @@ static bool prepare_command(BUFFER *wb,
82 const char *edit_command,
83 const char *machine_guid,
84 uuid_t *transition_id,
85 - const char *summary
85 + const char *summary,
86 + const char *context,
87 + const char *component,
88 + const char *type
89 ) {
90 char buf[8192];
88 - size_t n = 8192 - 1;
91 + size_t n = sizeof(buf) - 1;
92
93 buffer_strcat(wb, "exec");
94
@@ -195,6 +198,18 @@ static bool prepare_command(BUFFER *wb,
198 return false;
199 buffer_sprintf(wb, " '%s'", buf);
200
201 + if (!sanitize_command_argument_string(buf, context, n))
202 + return false;
203 + buffer_sprintf(wb, " '%s'", buf);
204 +
205 + if (!sanitize_command_argument_string(buf, component, n))
206 + return false;
207 + buffer_sprintf(wb, " '%s'", buf);
208 +
209 + if (!sanitize_command_argument_string(buf, type, n))
210 + return false;
211 + buffer_sprintf(wb, " '%s'", buf);
212 +
213 return true;
214 }
215
@@ -342,7 +357,9 @@ static void health_reload_host(RRDHOST *host) {
357 if(unlikely(!host->health.health_enabled) && !rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH))
358 return;
359
345 - netdata_log_health("[%s]: Reloading health.", rrdhost_hostname(host));
360 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
361 + "[%s]: Reloading health.",
362 + rrdhost_hostname(host));
363
364 char *user_path = health_user_config_dir();
365 char *stock_path = health_stock_config_dir();
@@ -436,8 +453,10 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
453
454 if(unlikely(ae->new_status <= RRDCALC_STATUS_CLEAR && (ae->flags & HEALTH_ENTRY_FLAG_NO_CLEAR_NOTIFICATION))) {
455 // do not send notifications for disabled statuses
439 - netdata_log_debug(D_HEALTH, "Health not sending notification for alarm '%s.%s' status %s (it has no-clear-notification enabled)", ae_chart_id(ae), ae_name(ae), rrdcalc_status2string(ae->new_status));
440 - netdata_log_health("[%s]: Health not sending notification for alarm '%s.%s' status %s (it has no-clear-notification enabled)", rrdhost_hostname(host), ae_chart_id(ae), ae_name(ae), rrdcalc_status2string(ae->new_status));
456 +
457 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
458 + "[%s]: Health not sending notification for alarm '%s.%s' status %s (it has no-clear-notification enabled)",
459 + rrdhost_hostname(host), ae_chart_id(ae), ae_name(ae), rrdcalc_status2string(ae->new_status));
460
461 // mark it as run, so that we will send the same alarm if it happens again
462 goto done;
@@ -454,10 +473,10 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
473 // we have executed this alarm notification in the past
474 if(last_executed_status == ae->new_status && !(ae->flags & HEALTH_ENTRY_FLAG_IS_REPEATING)) {
475 // don't send the notification for the same status again
457 - netdata_log_debug(D_HEALTH, "Health not sending again notification for alarm '%s.%s' status %s", ae_chart_id(ae), ae_name(ae)
458 - , rrdcalc_status2string(ae->new_status));
459 - netdata_log_health("[%s]: Health not sending again notification for alarm '%s.%s' status %s", rrdhost_hostname(host), ae_chart_id(ae), ae_name(ae)
460 - , rrdcalc_status2string(ae->new_status));
476 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
477 + "[%s]: Health not sending again notification for alarm '%s.%s' status %s",
478 + rrdhost_hostname(host), ae_chart_id(ae), ae_name(ae),
479 + rrdcalc_status2string(ae->new_status));
480 goto done;
481 }
482 }
@@ -476,11 +495,16 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
495
496 // Check if alarm notifications are silenced
497 if (ae->flags & HEALTH_ENTRY_FLAG_SILENCED) {
479 - netdata_log_health("[%s]: Health not sending notification for alarm '%s.%s' status %s (command API has disabled notifications)", rrdhost_hostname(host), ae_chart_id(ae), ae_name(ae), rrdcalc_status2string(ae->new_status));
498 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
499 + "[%s]: Health not sending notification for alarm '%s.%s' status %s "
500 + "(command API has disabled notifications)",
501 + rrdhost_hostname(host), ae_chart_id(ae), ae_name(ae), rrdcalc_status2string(ae->new_status));
502 goto done;
503 }
504
483 - netdata_log_health("[%s]: Sending notification for alarm '%s.%s' status %s.", rrdhost_hostname(host), ae_chart_id(ae), ae_name(ae), rrdcalc_status2string(ae->new_status));
505 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
506 + "[%s]: Sending notification for alarm '%s.%s' status %s.",
507 + rrdhost_hostname(host), ae_chart_id(ae), ae_name(ae), rrdcalc_status2string(ae->new_status));
508
509 const char *exec = (ae->exec) ? ae_exec(ae) : string2str(host->health.health_default_exec);
510 const char *recipient = (ae->recipient) ? ae_recipient(ae) : string2str(host->health.health_default_recipient);
@@ -581,7 +605,11 @@ static inline void health_alarm_execute(RRDHOST *host, ALARM_ENTRY *ae) {
605 edit_command,
606 host->machine_guid,
607 &ae->transition_id,
584 - host->health.use_summary_for_notifications && ae->summary?ae_summary(ae):ae_name(ae));
608 + host->health.use_summary_for_notifications && ae->summary?ae_summary(ae):ae_name(ae),
609 + string2str(ae->chart_context),
610 + string2str(ae->component),
611 + string2str(ae->type)
612 + );
613
614 const char *command_to_run = buffer_tostring(wb);
615 if (ok) {
@@ -778,7 +806,8 @@ static void health_main_cleanup(void *ptr) {
806 netdata_log_info("cleaning up...");
807 static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
808
781 - netdata_log_health("Health thread ended.");
809 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
810 + "Health thread ended.");
811 }
812
813 static void initialize_health(RRDHOST *host)
@@ -790,7 +819,9 @@ static void initialize_health(RRDHOST *host)
819
820 rrdhost_flag_set(host, RRDHOST_FLAG_INITIALIZED_HEALTH);
821
793 - netdata_log_health("[%s]: Initializing health.", rrdhost_hostname(host));
822 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
823 + "[%s]: Initializing health.",
824 + rrdhost_hostname(host));
825
826 host->health.health_default_warn_repeat_every = config_get_duration(CONFIG_SECTION_HEALTH, "default repeat warning", "never");
827 host->health.health_default_crit_repeat_every = config_get_duration(CONFIG_SECTION_HEALTH, "default repeat critical", "never");
@@ -803,7 +834,11 @@ static void initialize_health(RRDHOST *host)
834
835 long n = config_get_number(CONFIG_SECTION_HEALTH, "in memory max health log entries", host->health_log.max);
836 if(n < 10) {
806 - netdata_log_health("Host '%s': health configuration has invalid max log entries %ld. Using default %u", rrdhost_hostname(host), n, host->health_log.max);
837 + nd_log(NDLS_DAEMON, NDLP_WARNING,
838 + "Host '%s': health configuration has invalid max log entries %ld. "
839 + "Using default %u",
840 + rrdhost_hostname(host), n, host->health_log.max);
841 +
842 config_set_number(CONFIG_SECTION_HEALTH, "in memory max health log entries", (long)host->health_log.max);
843 }
844 else
@@ -811,7 +846,11 @@ static void initialize_health(RRDHOST *host)
846
847 uint32_t m = config_get_number(CONFIG_SECTION_HEALTH, "health log history", HEALTH_LOG_DEFAULT_HISTORY);
848 if (m < HEALTH_LOG_MINIMUM_HISTORY) {
814 - netdata_log_health("Host '%s': health configuration has invalid health log history %u. Using minimum %d", rrdhost_hostname(host), m, HEALTH_LOG_MINIMUM_HISTORY);
849 + nd_log(NDLS_DAEMON, NDLP_WARNING,
850 + "Host '%s': health configuration has invalid health log history %u. "
851 + "Using minimum %d",
852 + rrdhost_hostname(host), m, HEALTH_LOG_MINIMUM_HISTORY);
853 +
854 config_set_number(CONFIG_SECTION_HEALTH, "health log history", HEALTH_LOG_MINIMUM_HISTORY);
855 m = HEALTH_LOG_MINIMUM_HISTORY;
856 }
@@ -823,7 +862,9 @@ static void initialize_health(RRDHOST *host)
862 } else
863 host->health_log.health_log_history = m;
864
826 - netdata_log_health("[%s]: Health log history is set to %u seconds (%u days)", rrdhost_hostname(host), host->health_log.health_log_history, host->health_log.health_log_history / 86400);
865 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
866 + "[%s]: Health log history is set to %u seconds (%u days)",
867 + rrdhost_hostname(host), host->health_log.health_log_history, host->health_log.health_log_history / 86400);
868
869 conf_enabled_alarms = simple_pattern_create(config_get(CONFIG_SECTION_HEALTH, "enabled alarms", "*"), NULL,
870 SIMPLE_PATTERN_EXACT, true);
@@ -1049,7 +1090,7 @@ void *health_main(void *ptr) {
1090 if (unlikely(check_if_resumed_from_suspension())) {
1091 apply_hibernation_delay = 1;
1092
1052 - netdata_log_health(
1093 + nd_log(NDLS_DAEMON, NDLP_NOTICE,
1094 "Postponing alarm checks for %"PRId64" seconds, "
1095 "because it seems that the system was just resumed from suspension.",
1096 (int64_t)hibernation_delay);
@@ -1058,8 +1099,9 @@ void *health_main(void *ptr) {
1099 if (unlikely(silencers->all_alarms && silencers->stype == STYPE_DISABLE_ALARMS)) {
1100 static int logged=0;
1101 if (!logged) {
1061 - netdata_log_health("Skipping health checks, because all alarms are disabled via a %s command.",
1062 - HEALTH_CMDAPI_CMD_DISABLEALL);
1102 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1103 + "Skipping health checks, because all alarms are disabled via a %s command.",
1104 + HEALTH_CMDAPI_CMD_DISABLEALL);
1105 logged = 1;
1106 }
1107 }
@@ -1081,7 +1123,7 @@ void *health_main(void *ptr) {
1123 rrdcalc_delete_alerts_not_matching_host_labels_from_this_host(host);
1124
1125 if (unlikely(apply_hibernation_delay)) {
1084 - netdata_log_health(
1126 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1127 "[%s]: Postponing health checks for %"PRId64" seconds.",
1128 rrdhost_hostname(host),
1129 (int64_t)hibernation_delay);
@@ -1094,20 +1136,30 @@ void *health_main(void *ptr) {
1136 continue;
1137 }
1138
1097 - netdata_log_health("[%s]: Resuming health checks after delay.", rrdhost_hostname(host));
1139 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1140 + "[%s]: Resuming health checks after delay.",
1141 + rrdhost_hostname(host));
1142 +
1143 host->health.health_delay_up_to = 0;
1144 }
1145
1146 // wait until cleanup of obsolete charts on children is complete
1147 if (host != localhost) {
1148 if (unlikely(host->trigger_chart_obsoletion_check == 1)) {
1104 - netdata_log_health("[%s]: Waiting for chart obsoletion check.", rrdhost_hostname(host));
1149 +
1150 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1151 + "[%s]: Waiting for chart obsoletion check.",
1152 + rrdhost_hostname(host));
1153 +
1154 continue;
1155 }
1156 }
1157
1158 if (!health_running_logged) {
1110 - netdata_log_health("[%s]: Health is running.", rrdhost_hostname(host));
1159 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1160 + "[%s]: Health is running.",
1161 + rrdhost_hostname(host));
1162 +
1163 health_running_logged = true;
1164 }
1165
@@ -1161,6 +1213,7 @@ void *health_main(void *ptr) {
1213 rrdcalc_isrepeating(rc)?HEALTH_ENTRY_FLAG_IS_REPEATING:0);
1214
1215 if (ae) {
1216 + health_log_alert(host, ae);
1217 health_alarm_log_add_entry(host, ae);
1218 rc->old_status = rc->status;
1219 rc->status = RRDCALC_STATUS_REMOVED;
@@ -1432,9 +1485,13 @@ void *health_main(void *ptr) {
1485 )
1486 );
1487
1488 + health_log_alert(host, ae);
1489 health_alarm_log_add_entry(host, ae);
1490
1437 - netdata_log_health("[%s]: Alert event for [%s.%s], value [%s], status [%s].", rrdhost_hostname(host), ae_chart_id(ae), ae_name(ae), ae_new_value_string(ae), rrdcalc_status2string(ae->new_status));
1491 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1492 + "[%s]: Alert event for [%s.%s], value [%s], status [%s].",
1493 + rrdhost_hostname(host), ae_chart_id(ae), ae_name(ae), ae_new_value_string(ae),
1494 + rrdcalc_status2string(ae->new_status));
1495
1496 rc->last_status_change_value = rc->value;
1497 rc->last_status_change = now;
@@ -1519,6 +1576,7 @@ void *health_main(void *ptr) {
1576 )
1577 );
1578
1579 + health_log_alert(host, ae);
1580 ae->last_repeat = rc->last_repeat;
1581 if (!(rc->run_flags & RRDCALC_FLAG_RUN_ONCE) && rc->status == RRDCALC_STATUS_CLEAR) {
1582 ae->flags |= HEALTH_ENTRY_RUN_ONCE;
health/health.h
+3
@@ -105,4 +105,7 @@ void sql_refresh_hashes(void);
105 void health_add_host_labels(void);
106 void health_string2json(BUFFER *wb, const char *prefix, const char *label, const char *value, const char *suffix);
107
108 +void health_log_alert_transition_with_trace(RRDHOST *host, ALARM_ENTRY *ae, int line, const char *file, const char *function);
109 +#define health_log_alert(host, ae) health_log_alert_transition_with_trace(host, ae, __LINE__, __FILE__, __FUNCTION__)
110 +
111 #endif //NETDATA_HEALTH_H
health/health_config.c
+9 -2
@@ -1368,7 +1368,10 @@ void health_readdir(RRDHOST *host, const char *user_path, const char *stock_path
1368 CONFIG_BOOLEAN_YES);
1369
1370 if (!stock_enabled) {
1371 - netdata_log_health("[%s]: Netdata will not load stock alarms.", rrdhost_hostname(host));
1371 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1372 + "[%s]: Netdata will not load stock alarms.",
1373 + rrdhost_hostname(host));
1374 +
1375 stock_path = user_path;
1376 }
1377
@@ -1376,6 +1379,10 @@ void health_readdir(RRDHOST *host, const char *user_path, const char *stock_path
1379 health_rrdvars = health_rrdvariables_create();
1380
1381 recursive_config_double_dir_load(user_path, stock_path, subpath, health_readfile, (void *) host, 0);
1379 - netdata_log_health("[%s]: Read health configuration.", rrdhost_hostname(host));
1382 +
1383 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1384 + "[%s]: Read health configuration.",
1385 + rrdhost_hostname(host));
1386 +
1387 sql_store_hashes = 0;
1388 }
health/health_log.c
+73
@@ -8,6 +8,79 @@ inline void health_alarm_log_save(RRDHOST *host, ALARM_ENTRY *ae) {
8 sql_health_alarm_log_save(host, ae);
9 }
10
11 +
12 +void health_log_alert_transition_with_trace(RRDHOST *host, ALARM_ENTRY *ae, int line, const char *file, const char *function) {
13 + ND_LOG_STACK lgs[] = {
14 + ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &health_alert_transition_msgid),
15 + ND_LOG_FIELD_STR(NDF_NIDL_NODE, host->hostname),
16 + ND_LOG_FIELD_STR(NDF_NIDL_INSTANCE, ae->chart_name),
17 + ND_LOG_FIELD_STR(NDF_NIDL_CONTEXT, ae->chart_context),
18 + ND_LOG_FIELD_U64(NDF_ALERT_ID, ae->alarm_id),
19 + ND_LOG_FIELD_U64(NDF_ALERT_UNIQUE_ID, ae->unique_id),
20 + ND_LOG_FIELD_U64(NDF_ALERT_EVENT_ID, ae->alarm_event_id),
21 + ND_LOG_FIELD_UUID(NDF_ALERT_CONFIG_HASH, &ae->config_hash_id),
22 + ND_LOG_FIELD_UUID(NDF_ALERT_TRANSITION_ID, &ae->transition_id),
23 + ND_LOG_FIELD_STR(NDF_ALERT_NAME, ae->name),
24 + ND_LOG_FIELD_STR(NDF_ALERT_CLASS, ae->classification),
25 + ND_LOG_FIELD_STR(NDF_ALERT_COMPONENT, ae->component),
26 + ND_LOG_FIELD_STR(NDF_ALERT_TYPE, ae->type),
27 + ND_LOG_FIELD_STR(NDF_ALERT_EXEC, ae->exec),
28 + ND_LOG_FIELD_STR(NDF_ALERT_RECIPIENT, ae->recipient),
29 + ND_LOG_FIELD_STR(NDF_ALERT_SOURCE, ae->exec),
30 + ND_LOG_FIELD_STR(NDF_ALERT_UNITS, ae->units),
31 + ND_LOG_FIELD_STR(NDF_ALERT_SUMMARY, ae->summary),
32 + ND_LOG_FIELD_STR(NDF_ALERT_INFO, ae->info),
33 + ND_LOG_FIELD_DBL(NDF_ALERT_VALUE, ae->new_value),
34 + ND_LOG_FIELD_DBL(NDF_ALERT_VALUE_OLD, ae->old_value),
35 + ND_LOG_FIELD_TXT(NDF_ALERT_STATUS, rrdcalc_status2string(ae->new_status)),
36 + ND_LOG_FIELD_TXT(NDF_ALERT_STATUS_OLD, rrdcalc_status2string(ae->old_status)),
37 + ND_LOG_FIELD_I64(NDF_ALERT_DURATION, ae->duration),
38 + ND_LOG_FIELD_I64(NDF_RESPONSE_CODE, ae->exec_code),
39 + ND_LOG_FIELD_U64(NDF_ALERT_NOTIFICATION_REALTIME_USEC, ae->delay_up_to_timestamp * USEC_PER_SEC),
40 + ND_LOG_FIELD_END(),
41 + };
42 + ND_LOG_STACK_PUSH(lgs);
43 +
44 + errno = 0;
45 +
46 + ND_LOG_FIELD_PRIORITY priority = NDLP_INFO;
47 +
48 + switch(ae->new_status) {
49 + case RRDCALC_STATUS_UNDEFINED:
50 + if(ae->old_status >= RRDCALC_STATUS_CLEAR)
51 + priority = NDLP_NOTICE;
52 + else
53 + priority = NDLP_DEBUG;
54 + break;
55 +
56 + default:
57 + case RRDCALC_STATUS_UNINITIALIZED:
58 + case RRDCALC_STATUS_REMOVED:
59 + priority = NDLP_DEBUG;
60 + break;
61 +
62 + case RRDCALC_STATUS_CLEAR:
63 + priority = NDLP_INFO;
64 + break;
65 +
66 + case RRDCALC_STATUS_WARNING:
67 + if(ae->old_status < RRDCALC_STATUS_WARNING)
68 + priority = NDLP_WARNING;
69 + break;
70 +
71 + case RRDCALC_STATUS_CRITICAL:
72 + if(ae->old_status < RRDCALC_STATUS_CRITICAL)
73 + priority = NDLP_CRIT;
74 + break;
75 + }
76 +
77 + netdata_logger(NDLS_HEALTH, priority, file, function, line,
78 + "ALERT '%s' of instance '%s' on node '%s', transitioned from %s to %s",
79 + string2str(ae->name), string2str(ae->chart), string2str(host->hostname),
80 + rrdcalc_status2string(ae->old_status), rrdcalc_status2string(ae->new_status)
81 + );
82 +}
83 +
84 // ----------------------------------------------------------------------------
85 // health alarm log management
86
health/notifications/alarm-notify.sh.in
+299 -190
@@ -42,6 +42,8 @@
42 # -----------------------------------------------------------------------------
43 # testing notifications
44
45 +cmd_line="'${0}' $(printf "'%s' " "${@}")"
46 +
47 if { [ "${1}" = "test" ] || [ "${2}" = "test" ]; } && [ "${#}" -le 2 ]; then
48 if [ "${2}" = "test" ]; then
49 recipient="${1}"
@@ -78,61 +80,139 @@ export PATH="${PATH}:/sbin:/usr/sbin:/usr/local/sbin"
80 export LC_ALL=C
81
82 # -----------------------------------------------------------------------------
83 +# logging
84
85 PROGRAM_NAME="$(basename "${0}")"
86
84 -LOG_LEVEL_ERR=1
85 -LOG_LEVEL_WARN=2
86 -LOG_LEVEL_INFO=3
87 -LOG_LEVEL="$LOG_LEVEL_INFO"
87 +# these should be the same with syslog() priorities
88 +NDLP_EMERG=0 # system is unusable
89 +NDLP_ALERT=1 # action must be taken immediately
90 +NDLP_CRIT=2 # critical conditions
91 +NDLP_ERR=3 # error conditions
92 +NDLP_WARN=4 # warning conditions
93 +NDLP_NOTICE=5 # normal but significant condition
94 +NDLP_INFO=6 # informational
95 +NDLP_DEBUG=7 # debug-level messages
96 +
97 +# the max (numerically) log level we will log
98 +LOG_LEVEL=$NDLP_INFO
99 +
100 +set_log_min_priority() {
101 + case "${NETDATA_LOG_PRIORITY_LEVEL,,}" in
102 + "emerg" | "emergency")
103 + LOG_LEVEL=$NDLP_EMERG
104 + ;;
105
89 -set_log_severity_level() {
90 - case ${NETDATA_LOG_SEVERITY_LEVEL,,} in
91 - "info") LOG_LEVEL="$LOG_LEVEL_INFO";;
92 - "warn" | "warning") LOG_LEVEL="$LOG_LEVEL_WARN";;
93 - "err" | "error") LOG_LEVEL="$LOG_LEVEL_ERR";;
94 - esac
95 -}
106 + "alert")
107 + LOG_LEVEL=$NDLP_ALERT
108 + ;;
109
97 -set_log_severity_level
110 + "crit" | "critical")
111 + LOG_LEVEL=$NDLP_CRIT
112 + ;;
113
99 -logdate() {
100 - date "+%Y-%m-%d %H:%M:%S"
101 -}
114 + "err" | "error")
115 + LOG_LEVEL=$NDLP_ERR
116 + ;;
117
103 -log() {
104 - local status="${1}"
105 - shift
118 + "warn" | "warning")
119 + LOG_LEVEL=$NDLP_WARN
120 + ;;
121
107 - echo >&2 "$(logdate): ${PROGRAM_NAME}: ${status}: ${*}"
122 + "notice")
123 + LOG_LEVEL=$NDLP_NOTICE
124 + ;;
125
126 + "info")
127 + LOG_LEVEL=$NDLP_INFO
128 + ;;
129 +
130 + "debug")
131 + LOG_LEVEL=$NDLP_DEBUG
132 + ;;
133 + esac
134 +}
135 +
136 +set_log_min_priority
137 +
138 +log() {
139 + local level="${1}"
140 + shift 1
141 +
142 + [[ -n "$level" && -n "$LOG_LEVEL" && "$level" -gt "$LOG_LEVEL" ]] && return
143 +
144 + systemd-cat-native --log-as-netdata --newline="{NEWLINE}" <<EOFLOG
145 +INVOCATION_ID=${NETDATA_INVOCATION_ID}
146 +SYSLOG_IDENTIFIER=${PROGRAM_NAME}
147 +PRIORITY=${level}
148 +THREAD_TAG="alarm-notify"
149 +ND_LOG_SOURCE=health
150 +ND_NIDL_NODE=${host}
151 +ND_NIDL_INSTANCE=${chart}
152 +ND_NIDL_CONTEXT=${context}
153 +ND_ALERT_NAME=${name}
154 +ND_ALERT_ID=${alarm_id}
155 +ND_ALERT_UNIQUE_ID=${unique_id}
156 +ND_ALERT_EVENT_ID=${alarm_event_id}
157 +ND_ALERT_TRANSITION_ID=${transition_id//-/}
158 +ND_ALERT_CLASS=${classification}
159 +ND_ALERT_COMPONENT=${component}
160 +ND_ALERT_TYPE=${type}
161 +ND_ALERT_RECIPIENT=${roles}
162 +ND_ALERT_VALUE=${value}
163 +ND_ALERT_VALUE_OLD=${old_value}
164 +ND_ALERT_STATUS=${status}
165 +ND_ALERT_STATUS_OLD=${old_status}
166 +ND_ALERT_UNITS=${units}
167 +ND_ALERT_SUMMARY=${summary}
168 +ND_ALERT_INFO=${info}
169 +ND_ALERT_DURATION=${duration}
170 +ND_REQUEST=${cmd_line}
171 +MESSAGE_ID=6db0018e83e34320ae2a659d78019fb7
172 +MESSAGE=[ALERT NOTIFICATION]: ${*//[$'\r\n']/{NEWLINE}}
173 +
174 +EOFLOG
175 + # AN EMPTY LINE IS NEEDED ABOVE
176 }
177
178 info() {
112 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_INFO" -gt "$LOG_LEVEL" ]] && return
113 - log INFO "${@}"
179 + log "$NDLP_INFO" "${@}"
180 }
181
182 warning() {
117 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_WARN" -gt "$LOG_LEVEL" ]] && return
118 - log WARNING "${@}"
183 + log "$NDLP_WARN" "${@}"
184 }
185
186 error() {
122 - [[ -n "$LOG_LEVEL" && "$LOG_LEVEL_ERR" -gt "$LOG_LEVEL" ]] && return
123 - log ERROR "${@}"
187 + log "$NDLP_ERR" "${@}"
188 }
189
190 fatal() {
127 - log FATAL "${@}"
191 + log "$NDLP_ALERT" "${@}"
192 exit 1
193 }
194
131 -debug=${NETDATA_ALARM_NOTIFY_DEBUG-0}
195 debug() {
133 - [ "${debug}" = "1" ] && log DEBUG "${@}"
196 + log "$NDLP_DEBUG" "${@}"
197 }
198
199 +debug=0
200 +if [ "${NETDATA_ALARM_NOTIFY_DEBUG-0}" = "1" ]; then
201 + debug=1
202 + LOG_LEVEL=$NDLP_DEBUG
203 +fi
204 +
205 +# -----------------------------------------------------------------------------
206 +# check for BASH v4+ (required for associative arrays)
207 +
208 +if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
209 + echo >&2 "BASH version 4 or later is required (this is ${BASH_VERSION})."
210 + exit 1
211 +fi
212 +
213 +
214 +# -----------------------------------------------------------------------------
215 +
216 docurl() {
217 if [ -z "${curl}" ]; then
218 error "${curl} is unset."
@@ -199,16 +279,9 @@ ntfy
279 # this is to be overwritten by the config file
280
281 custom_sender() {
202 - info "not sending custom notification for ${status} of '${host}.${chart}.${name}'"
282 + info "custom notification mechanism is not configured; not sending ${notification_description}"
283 }
284
205 -# -----------------------------------------------------------------------------
206 -
207 -# check for BASH v4+ (required for associative arrays)
208 -if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
209 - fatal "BASH version 4 or later is required (this is ${BASH_VERSION})."
210 -fi
211 -
285 # -----------------------------------------------------------------------------
286 # defaults to allow running this script by hand
287
@@ -228,8 +301,8 @@ if [[ ${1} = "unittest" ]]; then
301 status="${4}" # the current status : REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL
302 old_status="${5}" # the previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL
303 elif [[ ${1} = "dump_methods" ]]; then
231 - dump_methods=1
232 - status="WARNING"
304 + dump_methods=1
305 + status="WARNING"
306 else
307 roles="${1}" # the roles that should be notified for this event
308 args_host="${2}" # the host generated this event
@@ -263,6 +336,9 @@ else
336 child_machine_guid="${28}" # the machine_guid of the child
337 transition_id="${29}" # the transition_id of the alert
338 summary="${30}" # the summary text field of the alert
339 + context="${31}" # the context of the chart
340 + component="${32}"
341 + type="${33}"
342 fi
343
344 # -----------------------------------------------------------------------------
@@ -276,18 +352,20 @@ else
352 host="${args_host}"
353 fi
354
355 +notification_description="notification to '${roles}' for transition from ${old_status} to ${status}, of alert '${name}' = '${value_string}', of instance '${chart}', context '${context}' on host '${host}'"
356 +
357 # -----------------------------------------------------------------------------
358 # screen statuses we don't need to send a notification
359
360 # don't do anything if this is not WARNING, CRITICAL or CLEAR
361 if [ "${status}" != "WARNING" ] && [ "${status}" != "CRITICAL" ] && [ "${status}" != "CLEAR" ]; then
284 - info "not sending notification for ${status} of '${host}.${chart}.${name}'"
362 + debug "not sending ${notification_description}"
363 exit 1
364 fi
365
366 # don't do anything if this is CLEAR, but it was not WARNING or CRITICAL
367 if [ "${clear_alarm_always}" != "YES" ] && [ "${old_status}" != "WARNING" ] && [ "${old_status}" != "CRITICAL" ] && [ "${status}" = "CLEAR" ]; then
290 - info "not sending notification for ${status} of '${host}.${chart}.${name}' (last status was ${old_status})"
368 + debug "not sending ${notification_description}"
369 exit 1
370 fi
371
@@ -434,7 +512,7 @@ else
512 debug "Loading config file '${CONFIG}'..."
513 source "${CONFIG}" || error "Failed to load config file '${CONFIG}'."
514 else
437 - warning "Cannot find file '${CONFIG}'."
515 + debug "Cannot find file '${CONFIG}'."
516 fi
517 done
518 fi
@@ -598,7 +676,16 @@ filter_recipient_by_criticality() {
676 }
677
678 # -----------------------------------------------------------------------------
601 -# verify the delivery methods supported
679 +# check the configured targets
680 +
681 +# check email
682 +if [ "${SEND_EMAIL}" = "AUTO" ]; then
683 + if command -v curl >/dev/null 2>&1; then
684 + SEND_EMAIL="YES"
685 + else
686 + SEND_EMAIL="NO"
687 + fi
688 +fi
689
690 # check slack
691 [ -z "${SLACK_WEBHOOK_URL}" ] && SEND_SLACK="NO"
@@ -677,112 +764,121 @@ filter_recipient_by_criticality() {
764 # check custom
765 [ -z "${DEFAULT_RECIPIENT_CUSTOM}" ] && SEND_CUSTOM="NO"
766
680 -if [ "${SEND_PUSHOVER}" = "YES" ] ||
681 - [ "${SEND_SLACK}" = "YES" ] ||
682 - [ "${SEND_ROCKETCHAT}" = "YES" ] ||
683 - [ "${SEND_ALERTA}" = "YES" ] ||
684 - [ "${SEND_PD}" = "YES" ] ||
685 - [ "${SEND_FLOCK}" = "YES" ] ||
686 - [ "${SEND_DISCORD}" = "YES" ] ||
687 - [ "${SEND_HIPCHAT}" = "YES" ] ||
688 - [ "${SEND_TWILIO}" = "YES" ] ||
689 - [ "${SEND_MESSAGEBIRD}" = "YES" ] ||
690 - [ "${SEND_KAVENEGAR}" = "YES" ] ||
691 - [ "${SEND_TELEGRAM}" = "YES" ] ||
692 - [ "${SEND_PUSHBULLET}" = "YES" ] ||
693 - [ "${SEND_KAFKA}" = "YES" ] ||
694 - [ "${SEND_FLEEP}" = "YES" ] ||
695 - [ "${SEND_PROWL}" = "YES" ] ||
696 - [ "${SEND_MATRIX}" = "YES" ] ||
697 - [ "${SEND_CUSTOM}" = "YES" ] ||
698 - [ "${SEND_MSTEAMS}" = "YES" ] ||
699 - [ "${SEND_DYNATRACE}" = "YES" ] ||
700 - [ "${SEND_OPSGENIE}" = "YES" ] ||
701 - [ "${SEND_GOTIFY}" = "YES" ] ||
702 - [ "${SEND_NTFY}" = "YES" ]; then
703 - # if we need curl, check for the curl command
704 - if [ -z "${curl}" ]; then
705 - curl="$(command -v curl 2>/dev/null)"
706 - fi
707 - if [ -z "${curl}" ]; then
708 - error "Cannot find curl command in the system path. Disabling all curl based notifications."
709 - SEND_PUSHOVER="NO"
710 - SEND_PUSHBULLET="NO"
711 - SEND_TELEGRAM="NO"
712 - SEND_SLACK="NO"
713 - SEND_MSTEAMS="NO"
714 - SEND_ROCKETCHAT="NO"
715 - SEND_ALERTA="NO"
716 - SEND_PD="NO"
717 - SEND_FLOCK="NO"
718 - SEND_DISCORD="NO"
719 - SEND_TWILIO="NO"
720 - SEND_HIPCHAT="NO"
721 - SEND_MESSAGEBIRD="NO"
722 - SEND_KAVENEGAR="NO"
723 - SEND_KAFKA="NO"
724 - SEND_FLEEP="NO"
725 - SEND_PROWL="NO"
726 - SEND_MATRIX="NO"
727 - SEND_CUSTOM="NO"
728 - SEND_DYNATRACE="NO"
729 - SEND_OPSGENIE="NO"
730 - SEND_GOTIFY="NO"
731 - SEND_NTFY="NO"
732 - fi
733 -fi
767 +# -----------------------------------------------------------------------------
768 +# check the availability of targets
769
735 -if [ "${SEND_SMS}" = "YES" ]; then
736 - if [ -z "${sendsms}" ]; then
737 - sendsms="$(command -v sendsms 2>/dev/null)"
770 +check_supported_targets() {
771 + local log=${1}
772 + shift
773 +
774 + if [ "${SEND_PUSHOVER}" = "YES" ] ||
775 + [ "${SEND_SLACK}" = "YES" ] ||
776 + [ "${SEND_ROCKETCHAT}" = "YES" ] ||
777 + [ "${SEND_ALERTA}" = "YES" ] ||
778 + [ "${SEND_PD}" = "YES" ] ||
779 + [ "${SEND_FLOCK}" = "YES" ] ||
780 + [ "${SEND_DISCORD}" = "YES" ] ||
781 + [ "${SEND_HIPCHAT}" = "YES" ] ||
782 + [ "${SEND_TWILIO}" = "YES" ] ||
783 + [ "${SEND_MESSAGEBIRD}" = "YES" ] ||
784 + [ "${SEND_KAVENEGAR}" = "YES" ] ||
785 + [ "${SEND_TELEGRAM}" = "YES" ] ||
786 + [ "${SEND_PUSHBULLET}" = "YES" ] ||
787 + [ "${SEND_KAFKA}" = "YES" ] ||
788 + [ "${SEND_FLEEP}" = "YES" ] ||
789 + [ "${SEND_PROWL}" = "YES" ] ||
790 + [ "${SEND_MATRIX}" = "YES" ] ||
791 + [ "${SEND_CUSTOM}" = "YES" ] ||
792 + [ "${SEND_MSTEAMS}" = "YES" ] ||
793 + [ "${SEND_DYNATRACE}" = "YES" ] ||
794 + [ "${SEND_OPSGENIE}" = "YES" ] ||
795 + [ "${SEND_GOTIFY}" = "YES" ] ||
796 + [ "${SEND_NTFY}" = "YES" ]; then
797 + # if we need curl, check for the curl command
798 + if [ -z "${curl}" ]; then
799 + curl="$(command -v curl 2>/dev/null)"
800 + fi
801 + if [ -z "${curl}" ]; then
802 + $log "Cannot find curl command in the system path. Disabling all curl based notifications."
803 + SEND_PUSHOVER="NO"
804 + SEND_PUSHBULLET="NO"
805 + SEND_TELEGRAM="NO"
806 + SEND_SLACK="NO"
807 + SEND_MSTEAMS="NO"
808 + SEND_ROCKETCHAT="NO"
809 + SEND_ALERTA="NO"
810 + SEND_PD="NO"
811 + SEND_FLOCK="NO"
812 + SEND_DISCORD="NO"
813 + SEND_TWILIO="NO"
814 + SEND_HIPCHAT="NO"
815 + SEND_MESSAGEBIRD="NO"
816 + SEND_KAVENEGAR="NO"
817 + SEND_KAFKA="NO"
818 + SEND_FLEEP="NO"
819 + SEND_PROWL="NO"
820 + SEND_MATRIX="NO"
821 + SEND_CUSTOM="NO"
822 + SEND_DYNATRACE="NO"
823 + SEND_OPSGENIE="NO"
824 + SEND_GOTIFY="NO"
825 + SEND_NTFY="NO"
826 + fi
827 fi
739 - if [ -z "${sendsms}" ]; then
740 - SEND_SMS="NO"
828 +
829 + if [ "${SEND_SMS}" = "YES" ]; then
830 + if [ -z "${sendsms}" ]; then
831 + sendsms="$(command -v sendsms 2>/dev/null)"
832 + fi
833 + if [ -z "${sendsms}" ]; then
834 + SEND_SMS="NO"
835 + fi
836 fi
742 -fi
743 -# if we need sendmail, check for the sendmail command
744 -if [ "${SEND_EMAIL}" = "YES" ] && [ -z "${sendmail}" ]; then
745 - sendmail="$(command -v sendmail 2>/dev/null)"
746 - if [ -z "${sendmail}" ]; then
747 - debug "Cannot find sendmail command in the system path. Disabling email notifications."
748 - SEND_EMAIL="NO"
837 + # if we need sendmail, check for the sendmail command
838 + if [ "${SEND_EMAIL}" = "YES" ] && [ -z "${sendmail}" ]; then
839 + sendmail="$(command -v sendmail 2>/dev/null)"
840 + if [ -z "${sendmail}" ]; then
841 + $log "Cannot find sendmail command in the system path. Disabling email notifications."
842 + SEND_EMAIL="NO"
843 + fi
844 fi
750 -fi
845
752 -# if we need logger, check for the logger command
753 -if [ "${SEND_SYSLOG}" = "YES" ] && [ -z "${logger}" ]; then
754 - logger="$(command -v logger 2>/dev/null)"
755 - if [ -z "${logger}" ]; then
756 - debug "Cannot find logger command in the system path. Disabling syslog notifications."
757 - SEND_SYSLOG="NO"
846 + # if we need logger, check for the logger command
847 + if [ "${SEND_SYSLOG}" = "YES" ] && [ -z "${logger}" ]; then
848 + logger="$(command -v logger 2>/dev/null)"
849 + if [ -z "${logger}" ]; then
850 + $log "Cannot find logger command in the system path. Disabling syslog notifications."
851 + SEND_SYSLOG="NO"
852 + fi
853 fi
759 -fi
854
761 -# if we need aws, check for the aws command
762 -if [ "${SEND_AWSSNS}" = "YES" ] && [ -z "${aws}" ]; then
763 - aws="$(command -v aws 2>/dev/null)"
764 - if [ -z "${aws}" ]; then
765 - debug "Cannot find aws command in the system path. Disabling Amazon SNS notifications."
766 - SEND_AWSSNS="NO"
855 + # if we need aws, check for the aws command
856 + if [ "${SEND_AWSSNS}" = "YES" ] && [ -z "${aws}" ]; then
857 + aws="$(command -v aws 2>/dev/null)"
858 + if [ -z "${aws}" ]; then
859 + $log "Cannot find aws command in the system path. Disabling Amazon SNS notifications."
860 + SEND_AWSSNS="NO"
861 + fi
862 fi
768 -fi
863
770 -# if we need nc, check for the nc command
771 -if [ "${SEND_IRC}" = "YES" ] && [ -z "${nc}" ]; then
772 - nc="$(command -v nc 2>/dev/null)"
773 - if [ -z "${nc}" ]; then
774 - debug "Cannot find nc command in the system path. Disabling IRC notifications."
775 - SEND_IRC="NO"
864 + # if we need nc, check for the nc command
865 + if [ "${SEND_IRC}" = "YES" ] && [ -z "${nc}" ]; then
866 + nc="$(command -v nc 2>/dev/null)"
867 + if [ -z "${nc}" ]; then
868 + $log "Cannot find nc command in the system path. Disabling IRC notifications."
869 + SEND_IRC="NO"
870 + fi
871 fi
777 -fi
872 +}
873
874 if [ ${dump_methods} ]; then
875 + check_supported_targets debug
876 for name in "${!SEND_@}"; do
877 if [ "${!name}" = "YES" ]; then
878 echo "$name"
879 fi
880 done
785 - exit
881 + exit 0
882 fi
883
884 # -----------------------------------------------------------------------------
@@ -790,6 +886,7 @@ fi
886
887 # netdata may call us with multiple roles, and roles may have multiple but
888 # overlapping recipients - so, here we find the unique recipients.
889 +have_to_send_something="NO"
890 for method_name in ${method_names}; do
891 send_var="SEND_${method_name^^}"
892 if [ "${!send_var}" = "NO" ]; then
@@ -819,7 +916,11 @@ for method_name in ${method_names}; do
916 to_var="to_${method_name}"
917 declare to_${method_name}="${!arr_var[*]}"
918
822 - [ -z "${!to_var}" ] && declare ${send_var}="NO"
919 + if [ -z "${!to_var}" ]; then
920 + declare ${send_var}="NO"
921 + else
922 + have_to_send_something="YES"
923 + fi
924 done
925
926 # -----------------------------------------------------------------------------
@@ -884,10 +985,18 @@ for method in "${SEND_EMAIL}" \
985 break
986 fi
987 done
988 +
989 if [ "$proceed" -eq 0 ]; then
888 - fatal "All notification methods are disabled. Not sending notification for host '${host}', chart '${chart}' to '${roles}' for '${name}' = '${value}' for status '${status}'."
990 + if [ "${have_to_send_something}" = "NO" ]; then
991 + debug "All notification methods are disabled; not sending ${notification_description}."
992 + exit 0
993 + else
994 + fatal "All notification methods are disabled; not sending ${notification_description}."
995 + fi
996 fi
997
998 +check_supported_targets error
999 +
1000 # -----------------------------------------------------------------------------
1001 # get the date the alarm happened
1002
@@ -1023,10 +1132,10 @@ send_email() {
1132 ret=$?
1133
1134 if [ ${ret} -eq 0 ]; then
1026 - info "sent email notification for: ${host} ${chart}.${name} is ${status} to '${to_email}'"
1135 + info "sent email to '${to_email}' for ${notification_description}"
1136 return 0
1137 else
1029 - error "failed to send email notification for: ${host} ${chart}.${name} is ${status} to '${to_email}' with error code ${ret} (${cmd_output})."
1138 + error "failed to send email to '${to_email}' for ${notification_description}, with error code ${ret} (${cmd_output})."
1139 return 1
1140 fi
1141 fi
@@ -1065,10 +1174,10 @@ send_pushover() {
1174 https://api.pushover.net/1/messages.json)
1175
1176 if [ "${httpcode}" = "200" ]; then
1068 - info "sent pushover notification for: ${host} ${chart}.${name} is ${status} to '${user}'"
1177 + info "sent pushover notification to '${user}' for ${notification_description}"
1178 sent=$((sent + 1))
1179 else
1071 - error "failed to send pushover notification for: ${host} ${chart}.${name} is ${status} to '${user}' with HTTP response status code ${httpcode}."
1180 + error "failed to send pushover notification to '${user}' for ${notification_description}, with HTTP response status code ${httpcode}."
1181 fi
1182 done
1183
@@ -1112,10 +1221,10 @@ EOF
1221 ) "https://api.pushbullet.com/v2/pushes" -X POST)
1222
1223 if [ "${httpcode}" = "200" ]; then
1115 - info "sent pushbullet notification for: ${host} ${chart}.${name} is ${status} to '${userOrChannelTag}'"
1224 + info "sent pushbullet notification to '${userOrChannelTag}' for ${notification_description}"
1225 sent=$((sent + 1))
1226 else
1118 - error "failed to send pushbullet notification for: ${host} ${chart}.${name} is ${status} to '${userOrChannelTag}' with HTTP response status code ${httpcode}."
1227 + error "failed to send pushbullet notification to '${userOrChannelTag}' for ${notification_description}, with HTTP response status code ${httpcode}."
1228 fi
1229 done
1230
@@ -1136,10 +1245,10 @@ send_kafka() {
1245 "${KAFKA_URL}")
1246
1247 if [ "${httpcode}" = "204" ]; then
1139 - info "sent kafka data for: ${host} ${chart}.${name} is ${status} and ip '${KAFKA_SENDER_IP}'"
1248 + info "sent kafka data to '${KAFKA_SENDER_IP}' for ${notification_description}"
1249 sent=$((sent + 1))
1250 else
1142 - error "failed to send kafka data for: ${host} ${chart}.${name} is ${status} and ip '${KAFKA_SENDER_IP}' with HTTP response status code ${httpcode}."
1251 + error "failed to send kafka data to '${KAFKA_SENDER_IP}' for ${notification_description}, with HTTP response status code ${httpcode}."
1252 fi
1253
1254 [ ${sent} -gt 0 ] && return 0
@@ -1237,10 +1346,10 @@ EOF
1346 fi
1347 httpcode=$(docurl -X POST --data "${payload}" ${url})
1348 if [ "${httpcode}" = "${response_code}" ]; then
1240 - info "sent pagerduty notification for: ${host} ${chart}.${name} is ${status}'"
1349 + info "sent pagerduty event for ${notification_description}"
1350 sent=$((sent + 1))
1351 else
1243 - error "failed to send pagerduty notification for: ${host} ${chart}.${name} is ${status}, with HTTP response status code ${httpcode}."
1352 + error "failed to send pagerduty event for ${notification_description}, with HTTP response status code ${httpcode}."
1353 fi
1354 done
1355
@@ -1266,10 +1375,10 @@ send_twilio() {
1375 "https://api.twilio.com/2010-04-01/Accounts/${accountsid}/Messages.json")
1376
1377 if [ "${httpcode}" = "201" ]; then
1269 - info "sent Twilio SMS for: ${host} ${chart}.${name} is ${status} to '${user}'"
1378 + info "sent Twilio SMS to '${user}' for ${notification_description}"
1379 sent=$((sent + 1))
1380 else
1272 - error "failed to send Twilio SMS for: ${host} ${chart}.${name} is ${status} to '${user}' with HTTP response status code ${httpcode}."
1381 + error "failed to send Twilio SMS to '${user}' for ${notification_description}, with HTTP response status code ${httpcode}."
1382 fi
1383 done
1384
@@ -1315,10 +1424,10 @@ send_hipchat() {
1424 "https://${HIPCHAT_SERVER}/v2/room/${room}/notification")
1425
1426 if [ "${httpcode}" = "204" ]; then
1318 - info "sent HipChat notification for: ${host} ${chart}.${name} is ${status} to '${room}'"
1427 + info "sent HipChat notification to '${room}' for ${notification_description}"
1428 sent=$((sent + 1))
1429 else
1321 - error "failed to send HipChat notification for: ${host} ${chart}.${name} is ${status} to '${room}' with HTTP response status code ${httpcode}."
1430 + error "failed to send HipChat notification to '${room}' for ${notification_description}, with HTTP response status code ${httpcode}."
1431 fi
1432 done
1433
@@ -1345,10 +1454,10 @@ send_messagebird() {
1454 "https://rest.messagebird.com/messages")
1455
1456 if [ "${httpcode}" = "201" ]; then
1348 - info "sent Messagebird SMS for: ${host} ${chart}.${name} is ${status} to '${user}'"
1457 + info "sent Messagebird SMS to '${user}' for ${notification_description}"
1458 sent=$((sent + 1))
1459 else
1351 - error "failed to send Messagebird SMS for: ${host} ${chart}.${name} is ${status} to '${user}' with HTTP response status code ${httpcode}."
1460 + error "failed to send Messagebird SMS to '${user}' for ${notification_description}, with HTTP response status code ${httpcode}."
1461 fi
1462 done
1463
@@ -1372,10 +1481,10 @@ send_kavenegar() {
1481 --data-urlencode "message=${title} ${message}")
1482
1483 if [ "${httpcode}" = "200" ]; then
1375 - info "sent Kavenegar SMS for: ${host} ${chart}.${name} is ${status} to '${user}'"
1484 + info "sent Kavenegar SMS to '${user}' for ${notification_description}"
1485 sent=$((sent + 1))
1486 else
1378 - error "failed to send Kavenegar SMS for: ${host} ${chart}.${name} is ${status} to '${user}' with HTTP response status code ${httpcode}."
1487 + error "failed to send Kavenegar SMS to '${user}' for ${notification_description}, with HTTP response status code ${httpcode}."
1488 fi
1489 done
1490
@@ -1416,21 +1525,21 @@ send_telegram() {
1525 notify_telegram=0
1526
1527 if [ "${httpcode}" = "200" ]; then
1419 - info "sent telegram notification for: ${host} ${chart}.${name} is ${status} to '${chatid}'"
1528 + info "sent telegram notification to '${chatid}' for ${notification_description}"
1529 sent=$((sent + 1))
1530 elif [ "${httpcode}" = "401" ]; then
1422 - error "failed to send telegram notification for: ${host} ${chart}.${name} is ${status} to '${chatid}': Wrong bot token."
1531 + error "failed to send telegram notification to '${chatid}' for ${notification_description}, wrong bot token."
1532 elif [ "${httpcode}" = "429" ]; then
1533 if [ "$notify_retries" -gt 0 ]; then
1425 - error "failed to send telegram notification for: ${host} ${chart}.${name} is ${status} to '${chatid}': rate limit exceeded, retrying after 1s."
1534 + error "failed to send telegram notification to '${chatid}' for ${notification_description}, rate limit exceeded, retrying after 1s."
1535 notify_retries=$((notify_retries - 1))
1536 notify_telegram=1
1537 sleep 1
1538 else
1430 - error "failed to send telegram notification for: ${host} ${chart}.${name} is ${status} to '${chatid}': rate limit exceeded."
1539 + error "failed to send telegram notification to '${chatid}' for ${notification_description}, rate limit exceeded."
1540 fi
1541 else
1433 - error "failed to send telegram notification for: ${host} ${chart}.${name} is ${status} to '${chatid}' with HTTP response status code ${httpcode}."
1542 + error "failed to send telegram notification to '${chatid}' for ${notification_description}, with HTTP response status code ${httpcode}."
1543 fi
1544 done
1545 done
@@ -1487,10 +1596,10 @@ EOF
1596 httpcode=$(docurl -H "Content-Type: application/json" -d "${payload}" "${cur_webhook}")
1597
1598 if [ "${httpcode}" = "200" ]; then
1490 - info "sent Microsoft team notification for: ${host} ${chart}.${name} is ${status} to '${cur_webhook}'"
1599 + info "sent Microsoft team notification to '${cur_webhook}' for ${notification_description}"
1600 sent=$((sent + 1))
1601 else
1493 - error "failed to send Microsoft team notification for: ${host} ${chart}.${name} is ${status} to '${cur_webhook}', with HTTP response status code ${httpcode}."
1602 + error "failed to send Microsoft team to '${cur_webhook}' for ${notification_description}, with HTTP response status code ${httpcode}."
1603 fi
1604 done
1605
@@ -1558,10 +1667,10 @@ EOF
1667
1668 httpcode=$(docurl -X POST --data-urlencode "payload=${payload}" "${webhook}")
1669 if [ "${httpcode}" = "200" ]; then
1561 - info "sent slack notification for: ${host} ${chart}.${name} is ${status} ${chstr}"
1670 + info "sent slack notification ${chstr} for ${notification_description}"
1671 sent=$((sent + 1))
1672 else
1564 - error "failed to send slack notification for: ${host} ${chart}.${name} is ${status} ${chstr}, with HTTP response status code ${httpcode}."
1673 + error "failed to send slack notification ${chstr} for ${notification_description}, with HTTP response status code ${httpcode}."
1674 fi
1675 done
1676
@@ -1616,10 +1725,10 @@ EOF
1725
1726 httpcode=$(docurl -X POST --data-urlencode "payload=${payload}" "${webhook}")
1727 if [ "${httpcode}" = "200" ]; then
1619 - info "sent rocketchat notification for: ${host} ${chart}.${name} is ${status} to '${channel}'"
1728 + info "sent rocketchat notification to '${channel}' for ${notification_description}"
1729 sent=$((sent + 1))
1730 else
1622 - error "failed to send rocketchat notification for: ${host} ${chart}.${name} is ${status} to '${channel}', with HTTP response status code ${httpcode}."
1731 + error "failed to send rocketchat notification to '${channel}' for ${notification_description}, with HTTP response status code ${httpcode}."
1732 fi
1733 done
1734
@@ -1685,12 +1794,12 @@ EOF
1794 httpcode=$(docurl -X POST "${webhook}/alert" -H "Content-Type: application/json" -H "Authorization: $auth" --data "${payload}")
1795
1796 if [ "${httpcode}" = "200" ] || [ "${httpcode}" = "201" ]; then
1688 - info "sent alerta notification for: ${host} ${chart}.${name} is ${status} to '${channel}'"
1797 + info "sent alerta notification to '${channel}' for ${notification_description}"
1798 sent=$((sent + 1))
1799 elif [ "${httpcode}" = "202" ]; then
1691 - info "suppressed alerta notification for: ${host} ${chart}.${name} is ${status} to '${channel}'"
1800 + info "suppressed alerta notification to '${channel}' for ${notification_description}"
1801 else
1693 - error "failed to send alerta notification for: ${host} ${chart}.${name} is ${status} to '${channel}', with HTTP response status code ${httpcode}."
1802 + error "failed to send alerta notification to '${channel}' for ${notification_description}, with HTTP response status code ${httpcode}."
1803 fi
1804 done
1805
@@ -1740,10 +1849,10 @@ send_flock() {
1849 ]
1850 }")
1851 if [ "${httpcode}" = "200" ]; then
1743 - info "sent flock notification for: ${host} ${chart}.${name} is ${status} to '${channel}'"
1852 + info "sent flock notification to '${channel}' for ${notification_description}"
1853 sent=$((sent + 1))
1854 else
1746 - error "failed to send flock notification for: ${host} ${chart}.${name} is ${status} to '${channel}', with HTTP response status code ${httpcode}."
1855 + error "failed to send flock notification to '${channel}' for ${notification_description}, with HTTP response status code ${httpcode}."
1856 fi
1857 done
1858
@@ -1801,10 +1910,10 @@ EOF
1910
1911 httpcode=$(docurl -X POST --data-urlencode "payload=${payload}" "${webhook}")
1912 if [ "${httpcode}" = "200" ]; then
1804 - info "sent discord notification for: ${host} ${chart}.${name} is ${status} to '${channel}'"
1913 + info "sent discord notification to '${channel}' for ${notification_description}"
1914 sent=$((sent + 1))
1915 else
1807 - error "failed to send discord notification for: ${host} ${chart}.${name} is ${status} to '${channel}', with HTTP response status code ${httpcode}."
1916 + error "failed to send discord notification to '${channel}' for ${notification_description}, with HTTP response status code ${httpcode}."
1917 fi
1918 done
1919
@@ -1830,10 +1939,10 @@ send_fleep() {
1939 httpcode=$(docurl -X POST --data "${data}" "https://fleep.io/hook/${hook}")
1940
1941 if [ "${httpcode}" = "200" ]; then
1833 - info "sent fleep data for: ${host} ${chart}.${name} is ${status} and user '${FLEEP_SENDER}'"
1942 + info "sent fleep data to user '${FLEEP_SENDER}' for ${notification_description}"
1943 sent=$((sent + 1))
1944 else
1836 - error "failed to send fleep data for: ${host} ${chart}.${name} is ${status} and user '${FLEEP_SENDER}' with HTTP response status code ${httpcode}."
1945 + error "failed to send fleep data to user '${FLEEP_SENDER}' for ${notification_description}, with HTTP response status code ${httpcode}."
1946 fi
1947 done
1948
@@ -1875,10 +1984,10 @@ send_prowl() {
1984 httpcode=$(docurl -X POST --data "${data}" "https://api.prowlapp.com/publicapi/add")
1985
1986 if [ "${httpcode}" = "200" ]; then
1878 - info "sent prowl data for: ${host} ${chart}.${name} is ${status}"
1987 + info "sent prowl event for ${notification_description}"
1988 sent=1
1989 else
1881 - error "failed to send prowl data for: ${host} ${chart}.${name} is ${status} with with error code ${httpcode}."
1990 + error "failed to send prowl event for ${notification_description}, with HTTP response status code ${httpcode}."
1991 fi
1992
1993 [ ${sent} -gt 0 ] && return 0
@@ -1914,10 +2023,10 @@ send_irc() {
2023 done
2024
2025 if [ "${error}" -eq 0 ]; then
1917 - info "sent irc notification for: ${host} ${chart}.${name} is ${status} to '${CHANNEL}'"
2026 + info "sent irc notification to '${CHANNEL}' for ${notification_description}"
2027 sent=$((sent + 1))
2028 else
1920 - error "failed to send irc notification for: ${host} ${chart}.${name} is ${status} to '${CHANNEL}', with error code ${code}."
2029 + error "failed to send irc notification to '${CHANNEL}' for ${notification_description}, with error code ${code}."
2030 fi
2031 done
2032 fi
@@ -1942,10 +2051,10 @@ send_awssns() {
2051 # Extract the region from the target ARN. We need to explicitly specify the region so that it matches up correctly.
2052 region="$(echo ${target} | cut -f 4 -d ':')"
2053 if ${aws} sns publish --region "${region}" --subject "${host} ${status_message} - ${name//_/ } - ${chart}" --message "${message}" --target-arn ${target} &>/dev/null; then
1945 - info "sent Amazon SNS notification for: ${host} ${chart}.${name} is ${status} to '${target}'"
2054 + info "sent Amazon SNS notification to '${target}' for ${notification_description}"
2055 sent=$((sent + 1))
2056 else
1948 - error "failed to send Amazon SNS notification for: ${host} ${chart}.${name} is ${status} to '${target}'"
2057 + error "failed to send Amazon SNS notification to '${target}' for ${notification_description}"
2058 fi
2059 done
2060
@@ -1987,10 +2096,10 @@ EOF
2096
2097 httpcode=$(docurl -X POST --data "${payload}" "${webhook}")
2098 if [ "${httpcode}" == "200" ]; then
1990 - info "sent Matrix notification for: ${host} ${chart}.${name} is ${status} to '${room}'"
2099 + info "sent Matrix notification to '${room}' for ${notification_description}"
2100 sent=$((sent + 1))
2101 else
1993 - error "failed to send Matrix notification for: ${host} ${chart}.${name} is ${status} to '${room}', with HTTP response status code ${httpcode}."
2102 + error "failed to send Matrix notification to '${room}' for ${notification_description}, with HTTP response status code ${httpcode}."
2103 fi
2104 done
2105
@@ -2089,10 +2198,10 @@ send_sms() {
2198 errmessage=$($sendsms $phone "$msg" 2>&1)
2199 errcode=$?
2200 if [ ${errcode} -eq 0 ]; then
2092 - info "sent smstools3 SMS for: ${host} ${chart}.${name} is ${status} to '${user}'"
2201 + info "sent smstools3 SMS to '${user}' for ${notification_description}"
2202 sent=$((sent + 1))
2203 else
2095 - error "failed to send smstools3 SMS for: ${host} ${chart}.${name} is ${status} to '${user}' with error code ${errcode}: ${errmessage}."
2204 + error "failed to send smstools3 SMS to '${user}' for ${notification_description}, with error code ${errcode}: ${errmessage}."
2205 fi
2206 done
2207
@@ -2139,14 +2248,14 @@ EOF
2248
2249 if [ ${ret} -eq 0 ]; then
2250 if [ "${httpcode}" = "200" ]; then
2142 - info "sent ${DYNATRACE_EVENT} to ${DYNATRACE_SERVER}"
2251 + info "sent Dynatrace event '${DYNATRACE_EVENT}' to '${DYNATRACE_SERVER}' for ${notification_description}"
2252 return 0
2253 else
2145 - warning "Dynatrace ${DYNATRACE_SERVER} responded ${httpcode} notification for: ${host} ${chart}.${name} is ${status} was not sent!"
2254 + warning "failed to send Dynatrace event to '${DYNATRACE_SERVER}' for ${notification_description}, with HTTP response status code ${httpcode}"
2255 return 1
2256 fi
2257 else
2149 - error "failed to sent ${DYNATRACE_EVENT} notification for: ${host} ${chart}.${name} is ${status} to ${DYNATRACE_SERVER} with error code ${ret}."
2258 + error "failed to sent Dynatrace '${DYNATRACE_EVENT}' to '${DYNATRACE_SERVER}' for ${notification_description}, with code ${ret}."
2259 return 1
2260 fi
2261 }
@@ -2204,9 +2313,9 @@ EOF
2313 httpcode=$(docurl -X POST -H "Content-Type: application/json" -d "${payload}" "${OPSGENIE_API_URL}/v1/json/integrations/webhooks/netdata?apiKey=${OPSGENIE_API_KEY}")
2314 # https://docs.opsgenie.com/docs/alert-api#create-alert
2315 if [ "${httpcode}" = "200" ]; then
2207 - info "sent opsgenie notification for: ${host} ${chart}.${name} is ${status}"
2316 + info "sent opsgenie event for ${notification_description}"
2317 else
2209 - error "failed to send opsgenie notification for: ${host} ${chart}.${name} is ${status}, with HTTP error code ${httpcode}."
2318 + error "failed to send opsgenie event for ${notification_description}, with HTTP response status code ${httpcode}."
2319 return 1
2320 fi
2321
@@ -2243,9 +2352,9 @@ EOF
2352
2353 httpcode=$(docurl -X POST -H "Content-Type: application/json" -d "${payload}" "${GOTIFY_APP_URL}/message?token=${GOTIFY_APP_TOKEN}")
2354 if [ "${httpcode}" = "200" ]; then
2246 - info "sent gotify notification for: ${host} ${chart}.${name} is ${status}"
2355 + info "sent gotify event for ${notification_description}"
2356 else
2248 - error "failed to send gotify notification for: ${host} ${chart}.${name} is ${status}, with HTTP error code ${httpcode}."
2357 + error "failed to send gotify event for ${notification_description}, with HTTP response status code ${httpcode}."
2358 return 1
2359 fi
2360
@@ -2298,10 +2407,10 @@ send_ntfy() {
2407 -d "${msg}" \
2408 ${recipient})
2409 if [ "${httpcode}" == "200" ]; then
2301 - info "sent ntfy notification for: ${host} ${chart}.${name} is ${status} to '${recipient}'"
2410 + info "sent ntfy notification to '${recipient}' for ${notification_description}"
2411 sent=$((sent + 1))
2412 else
2304 - error "failed to send ntfy notification for: ${host} ${chart}.${name} is ${status} to '${recipient}', with HTTP response status code ${httpcode}."
2413 + error "failed to send ntfy notification to '${recipient}' for ${notification_description}, with HTTP response status code ${httpcode}."
2414 fi
2415 done
2416
health/notifications/health_alarm_notify.conf
+2 -2
@@ -211,8 +211,8 @@ sendsms=""
211 # EMAIL_SENDER="\"User Name\" <user@domain>"
212 EMAIL_SENDER=""
213
214 -# enable/disable sending emails
215 -SEND_EMAIL="YES"
214 +# enable/disable sending emails, set this YES, or NO, AUTO to enable/disable based on sendmail availability
215 +SEND_EMAIL="AUTO"
216
217 # if a role recipient is not configured, an email will be send to:
218 DEFAULT_RECIPIENT_EMAIL="root"
libnetdata/Makefile.am
+4
@@ -8,9 +8,11 @@ SUBDIRS = \
8 aral \
9 avl \
10 buffer \
11 + buffered_reader \
12 clocks \
13 completion \
14 config \
15 + datetime \
16 dictionary \
17 ebpf \
18 eval \
@@ -19,6 +21,7 @@ SUBDIRS = \
21 json \
22 july \
23 health \
24 + line_splitter \
25 locks \
26 log \
27 onewayalloc \
@@ -31,6 +34,7 @@ SUBDIRS = \
34 string \
35 threads \
36 url \
37 + uuid \
38 worker_utilization \
39 tests \
40 $(NULL)
libnetdata/buffer/buffer.c
+17 -30
@@ -81,6 +81,7 @@ void buffer_snprintf(BUFFER *wb, size_t len, const char *fmt, ...)
81
82 va_list args;
83 va_start(args, fmt);
84 + // vsnprintfz() returns the number of bytes actually written - after possible truncation
85 wb->len += vsnprintfz(&wb->buffer[wb->len], len, fmt, args);
86 va_end(args);
87
@@ -89,53 +90,39 @@ void buffer_snprintf(BUFFER *wb, size_t len, const char *fmt, ...)
90 // the buffer is \0 terminated by vsnprintfz
91 }
92
92 -void buffer_vsprintf(BUFFER *wb, const char *fmt, va_list args)
93 -{
93 +inline void buffer_vsprintf(BUFFER *wb, const char *fmt, va_list args) {
94 if(unlikely(!fmt || !*fmt)) return;
95
96 - size_t wrote = 0, need = 2, space_remaining = 0;
96 + size_t full_size_bytes = 0, need = 2, space_remaining = 0;
97
98 do {
99 - need += space_remaining * 2;
99 + need += full_size_bytes + 2;
100
101 - netdata_log_debug(D_WEB_BUFFER, "web_buffer_sprintf(): increasing web_buffer at position %zu, size = %zu, by %zu bytes (wrote = %zu)\n", wb->len, wb->size, need, wrote);
101 buffer_need_bytes(wb, need);
102
103 space_remaining = wb->size - wb->len - 1;
104
106 - wrote = (size_t) vsnprintfz(&wb->buffer[wb->len], space_remaining, fmt, args);
105 + // Use the copy of va_list for vsnprintf
106 + va_list args_copy;
107 + va_copy(args_copy, args);
108 + // vsnprintf() returns the number of bytes required, even if bigger than the buffer provided
109 + full_size_bytes = (size_t) vsnprintf(&wb->buffer[wb->len], space_remaining, fmt, args_copy);
110 + va_end(args_copy);
111
108 - } while(wrote >= space_remaining);
112 + } while(full_size_bytes >= space_remaining);
113
110 - wb->len += wrote;
114 + wb->len += full_size_bytes;
115
112 - // the buffer is \0 terminated by vsnprintf
116 + wb->buffer[wb->len] = '\0';
117 + buffer_overflow_check(wb);
118 }
119
120 void buffer_sprintf(BUFFER *wb, const char *fmt, ...)
121 {
117 - if(unlikely(!fmt || !*fmt)) return;
118 -
122 va_list args;
120 - size_t wrote = 0, need = 2, space_remaining = 0;
121 -
122 - do {
123 - need += space_remaining * 2;
124 -
125 - netdata_log_debug(D_WEB_BUFFER, "web_buffer_sprintf(): increasing web_buffer at position %zu, size = %zu, by %zu bytes (wrote = %zu)\n", wb->len, wb->size, need, wrote);
126 - buffer_need_bytes(wb, need);
127 -
128 - space_remaining = wb->size - wb->len - 1;
129 -
130 - va_start(args, fmt);
131 - wrote = (size_t) vsnprintfz(&wb->buffer[wb->len], space_remaining, fmt, args);
132 - va_end(args);
133 -
134 - } while(wrote >= space_remaining);
135 -
136 - wb->len += wrote;
137 -
138 - // the buffer is \0 terminated by vsnprintf
123 + va_start(args, fmt);
124 + buffer_vsprintf(wb, fmt, args);
125 + va_end(args);
126 }
127
128 // generate a javascript date, the fastest possible way...
libnetdata/buffer/buffer.h
+26
@@ -94,6 +94,8 @@ typedef struct web_buffer {
94 } json;
95 } BUFFER;
96
97 +#define CLEAN_BUFFER _cleanup_(buffer_freep) BUFFER
98 +
99 #define buffer_cacheable(wb) do { (wb)->options |= WB_CONTENT_CACHEABLE; if((wb)->options & WB_CONTENT_NO_CACHEABLE) (wb)->options &= ~WB_CONTENT_NO_CACHEABLE; } while(0)
100 #define buffer_no_cacheable(wb) do { (wb)->options |= WB_CONTENT_NO_CACHEABLE; if((wb)->options & WB_CONTENT_CACHEABLE) (wb)->options &= ~WB_CONTENT_CACHEABLE; (wb)->expires = 0; } while(0)
101
@@ -135,6 +137,10 @@ BUFFER *buffer_create(size_t size, size_t *statistics);
137 void buffer_free(BUFFER *b);
138 void buffer_increase(BUFFER *b, size_t free_size_required);
139
140 +static inline void buffer_freep(BUFFER **bp) {
141 + if(bp) buffer_free(*bp);
142 +}
143 +
144 void buffer_snprintf(BUFFER *wb, size_t len, const char *fmt, ...) PRINTFLIKE(3, 4);
145 void buffer_vsprintf(BUFFER *wb, const char *fmt, va_list args);
146 void buffer_sprintf(BUFFER *wb, const char *fmt, ...) PRINTFLIKE(2,3);
@@ -210,6 +216,13 @@ static inline void buffer_fast_rawcat(BUFFER *wb, const char *txt, size_t len) {
216 buffer_overflow_check(wb);
217 }
218
219 +static inline void buffer_putc(BUFFER *wb, char c) {
220 + buffer_need_bytes(wb, 2);
221 + wb->buffer[wb->len++] = c;
222 + wb->buffer[wb->len] = '\0';
223 + buffer_overflow_check(wb);
224 +}
225 +
226 static inline void buffer_fast_strcat(BUFFER *wb, const char *txt, size_t len) {
227 if(unlikely(!txt || !*txt || !len)) return;
228
@@ -283,6 +296,19 @@ static inline void buffer_strncat(BUFFER *wb, const char *txt, size_t len) {
296 buffer_overflow_check(wb);
297 }
298
299 +static inline void buffer_memcat(BUFFER *wb, const void *mem, size_t bytes) {
300 + if(unlikely(!mem)) return;
301 +
302 + buffer_need_bytes(wb, bytes + 1);
303 +
304 + memcpy(&wb->buffer[wb->len], mem, bytes);
305 +
306 + wb->len += bytes;
307 + wb->buffer[wb->len] = '\0';
308 +
309 + buffer_overflow_check(wb);
310 +}
311 +
312 static inline void buffer_json_strcat(BUFFER *wb, const char *txt) {
313 if(unlikely(!txt || !*txt)) return;
314
libnetdata/buffered_reader/Makefile.am new
+8
@@ -0,0 +1,8 @@
1 +# SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +AUTOMAKE_OPTIONS = subdir-objects
4 +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
5 +
6 +dist_noinst_DATA = \
7 + README.md \
8 + $(NULL)
libnetdata/buffered_reader/README.md
libnetdata/buffered_reader/buffered_reader.c new
+3
@@ -0,0 +1,3 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
libnetdata/buffered_reader/buffered_reader.h new
+146
@@ -0,0 +1,146 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
4 +
5 +#ifndef NETDATA_BUFFERED_READER_H
6 +#define NETDATA_BUFFERED_READER_H
7 +
8 +struct buffered_reader {
9 + ssize_t read_len;
10 + ssize_t pos;
11 + char read_buffer[PLUGINSD_LINE_MAX + 1];
12 +};
13 +
14 +static inline void buffered_reader_init(struct buffered_reader *reader) {
15 + reader->read_buffer[0] = '\0';
16 + reader->read_len = 0;
17 + reader->pos = 0;
18 +}
19 +
20 +typedef enum {
21 + BUFFERED_READER_READ_OK = 0,
22 + BUFFERED_READER_READ_FAILED = -1,
23 + BUFFERED_READER_READ_BUFFER_FULL = -2,
24 + BUFFERED_READER_READ_POLLERR = -3,
25 + BUFFERED_READER_READ_POLLHUP = -4,
26 + BUFFERED_READER_READ_POLLNVAL = -5,
27 + BUFFERED_READER_READ_POLL_UNKNOWN = -6,
28 + BUFFERED_READER_READ_POLL_TIMEOUT = -7,
29 + BUFFERED_READER_READ_POLL_FAILED = -8,
30 +} buffered_reader_ret_t;
31 +
32 +
33 +static inline buffered_reader_ret_t buffered_reader_read(struct buffered_reader *reader, int fd) {
34 +#ifdef NETDATA_INTERNAL_CHECKS
35 + if(reader->read_buffer[reader->read_len] != '\0')
36 + fatal("read_buffer does not start with zero");
37 +#endif
38 +
39 + char *read_at = reader->read_buffer + reader->read_len;
40 + ssize_t remaining = sizeof(reader->read_buffer) - reader->read_len - 1;
41 +
42 + if(unlikely(remaining <= 0))
43 + return BUFFERED_READER_READ_BUFFER_FULL;
44 +
45 + ssize_t bytes_read = read(fd, read_at, remaining);
46 + if(unlikely(bytes_read <= 0))
47 + return BUFFERED_READER_READ_FAILED;
48 +
49 + reader->read_len += bytes_read;
50 + reader->read_buffer[reader->read_len] = '\0';
51 +
52 + return BUFFERED_READER_READ_OK;
53 +}
54 +
55 +static inline buffered_reader_ret_t buffered_reader_read_timeout(struct buffered_reader *reader, int fd, int timeout_ms, bool log_error) {
56 + errno = 0;
57 + struct pollfd fds[1];
58 +
59 + fds[0].fd = fd;
60 + fds[0].events = POLLIN;
61 +
62 + int ret = poll(fds, 1, timeout_ms);
63 +
64 + if (ret > 0) {
65 + /* There is data to read */
66 + if (fds[0].revents & POLLIN)
67 + return buffered_reader_read(reader, fd);
68 +
69 + else if(fds[0].revents & POLLERR) {
70 + if(log_error)
71 + netdata_log_error("PARSER: read failed: POLLERR.");
72 + return BUFFERED_READER_READ_POLLERR;
73 + }
74 + else if(fds[0].revents & POLLHUP) {
75 + if(log_error)
76 + netdata_log_error("PARSER: read failed: POLLHUP.");
77 + return BUFFERED_READER_READ_POLLHUP;
78 + }
79 + else if(fds[0].revents & POLLNVAL) {
80 + if(log_error)
81 + netdata_log_error("PARSER: read failed: POLLNVAL.");
82 + return BUFFERED_READER_READ_POLLNVAL;
83 + }
84 +
85 + if(log_error)
86 + netdata_log_error("PARSER: poll() returned positive number, but POLLIN|POLLERR|POLLHUP|POLLNVAL are not set.");
87 + return BUFFERED_READER_READ_POLL_UNKNOWN;
88 + }
89 + else if (ret == 0) {
90 + if(log_error)
91 + netdata_log_error("PARSER: timeout while waiting for data.");
92 + return BUFFERED_READER_READ_POLL_TIMEOUT;
93 + }
94 +
95 + if(log_error)
96 + netdata_log_error("PARSER: poll() failed with code %d.", ret);
97 + return BUFFERED_READER_READ_POLL_FAILED;
98 +}
99 +
100 +/* Produce a full line if one exists, statefully return where we start next time.
101 + * When we hit the end of the buffer with a partial line move it to the beginning for the next fill.
102 + */
103 +static inline bool buffered_reader_next_line(struct buffered_reader *reader, BUFFER *dst) {
104 + buffer_need_bytes(dst, reader->read_len - reader->pos + 2);
105 +
106 + size_t start = reader->pos;
107 +
108 + char *ss = &reader->read_buffer[start];
109 + char *se = &reader->read_buffer[reader->read_len];
110 + char *ds = &dst->buffer[dst->len];
111 + char *de = &ds[dst->size - dst->len - 2];
112 +
113 + if(ss >= se) {
114 + *ds = '\0';
115 + reader->pos = 0;
116 + reader->read_len = 0;
117 + reader->read_buffer[reader->read_len] = '\0';
118 + return false;
119 + }
120 +
121 + // copy all bytes to buffer
122 + while(ss < se && ds < de && *ss != '\n') {
123 + *ds++ = *ss++;
124 + dst->len++;
125 + }
126 +
127 + // if we have a newline, return the buffer
128 + if(ss < se && ds < de && *ss == '\n') {
129 + // newline found in the r->read_buffer
130 +
131 + *ds++ = *ss++; // copy the newline too
132 + dst->len++;
133 +
134 + *ds = '\0';
135 +
136 + reader->pos = ss - reader->read_buffer;
137 + return true;
138 + }
139 +
140 + reader->pos = 0;
141 + reader->read_len = 0;
142 + reader->read_buffer[reader->read_len] = '\0';
143 + return false;
144 +}
145 +
146 +#endif //NETDATA_BUFFERED_READER_H
libnetdata/clocks/clocks.c
+14 -6
@@ -298,8 +298,10 @@ usec_t heartbeat_next(heartbeat_t *hb, usec_t tick) {
298 // TODO: The heartbeat tick should be specified at the heartbeat_init() function
299 usec_t tmp = (now_realtime_usec() * clock_realtime_resolution) % (tick / 2);
300
301 - error_limit_static_global_var(erl, 10, 0);
302 - error_limit(&erl, "heartbeat randomness of %"PRIu64" is too big for a tick of %"PRIu64" - setting it to %"PRIu64"", hb->randomness, tick, tmp);
301 + nd_log_limit_static_global_var(erl, 10, 0);
302 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_NOTICE,
303 + "heartbeat randomness of %"PRIu64" is too big for a tick of %"PRIu64" - setting it to %"PRIu64"",
304 + hb->randomness, tick, tmp);
305 hb->randomness = tmp;
306 }
307
@@ -325,13 +327,19 @@ usec_t heartbeat_next(heartbeat_t *hb, usec_t tick) {
327
328 if(unlikely(now < next)) {
329 errno = 0;
328 - error_limit_static_global_var(erl, 10, 0);
329 - error_limit(&erl, "heartbeat clock: woke up %"PRIu64" microseconds earlier than expected (can be due to the CLOCK_REALTIME set to the past).", next - now);
330 + nd_log_limit_static_global_var(erl, 10, 0);
331 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_NOTICE,
332 + "heartbeat clock: woke up %"PRIu64" microseconds earlier than expected "
333 + "(can be due to the CLOCK_REALTIME set to the past).",
334 + next - now);
335 }
336 else if(unlikely(now - next > tick / 2)) {
337 errno = 0;
333 - error_limit_static_global_var(erl, 10, 0);
334 - error_limit(&erl, "heartbeat clock: woke up %"PRIu64" microseconds later than expected (can be due to system load or the CLOCK_REALTIME set to the future).", now - next);
338 + nd_log_limit_static_global_var(erl, 10, 0);
339 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_NOTICE,
340 + "heartbeat clock: woke up %"PRIu64" microseconds later than expected "
341 + "(can be due to system load or the CLOCK_REALTIME set to the future).",
342 + now - next);
343 }
344
345 if(unlikely(!hb->realtime)) {
libnetdata/datetime/Makefile.am new
+8
@@ -0,0 +1,8 @@
1 +# SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +AUTOMAKE_OPTIONS = subdir-objects
4 +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
5 +
6 +dist_noinst_DATA = \
7 + README.md \
8 + $(NULL)
libnetdata/datetime/README.md new
+11
@@ -0,0 +1,11 @@
1 +<!--
2 +title: "Datetime"
3 +custom_edit_url: https://github.com/netdata/netdata/edit/master/libnetdata/datetime/README.md
4 +sidebar_label: "Datetime"
5 +learn_topic_type: "Tasks"
6 +learn_rel_path: "Developers/libnetdata"
7 +-->
8 +
9 +# Datetime
10 +
11 +Formatting dates and timestamps.
libnetdata/datetime/iso8601.c new
+81
@@ -0,0 +1,81 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
4 +
5 +size_t iso8601_datetime_ut(char *buffer, size_t len, usec_t now_ut, ISO8601_OPTIONS options) {
6 + if(unlikely(!buffer || len == 0))
7 + return 0;
8 +
9 + time_t t = (time_t)(now_ut / USEC_PER_SEC);
10 + struct tm *tmp, tmbuf;
11 +
12 + if(options & ISO8601_UTC)
13 + // Use gmtime_r for UTC time conversion.
14 + tmp = gmtime_r(&t, &tmbuf);
15 + else
16 + // Use localtime_r for local time conversion.
17 + tmp = localtime_r(&t, &tmbuf);
18 +
19 + if (unlikely(!tmp)) {
20 + buffer[0] = '\0';
21 + return 0;
22 + }
23 +
24 + // Format the date and time according to the ISO 8601 format.
25 + size_t used_length = strftime(buffer, len, "%Y-%m-%dT%H:%M:%S", tmp);
26 + if (unlikely(used_length == 0)) {
27 + buffer[0] = '\0';
28 + return 0;
29 + }
30 +
31 + if(options & ISO8601_MILLISECONDS) {
32 + // Calculate the remaining microseconds
33 + int milliseconds = (int) ((now_ut % USEC_PER_SEC) / USEC_PER_MS);
34 + if(milliseconds && len - used_length > 4)
35 + used_length += snprintfz(buffer + used_length, len - used_length, ".%03d", milliseconds);
36 + }
37 + else if(options & ISO8601_MICROSECONDS) {
38 + // Calculate the remaining microseconds
39 + int microseconds = (int) (now_ut % USEC_PER_SEC);
40 + if(microseconds && len - used_length > 7)
41 + used_length += snprintfz(buffer + used_length, len - used_length, ".%06d", microseconds);
42 + }
43 +
44 + if(options & ISO8601_UTC) {
45 + if(used_length + 1 < len) {
46 + buffer[used_length++] = 'Z';
47 + buffer[used_length] = '\0'; // null-terminate the string.
48 + }
49 + }
50 + else {
51 + // Calculate the timezone offset in hours and minutes from UTC.
52 + long offset = tmbuf.tm_gmtoff;
53 + int hours = (int) (offset / 3600); // Convert offset seconds to hours.
54 + int minutes = (int) ((offset % 3600) / 60); // Convert remainder to minutes (keep the sign for minutes).
55 +
56 + // Check if timezone is UTC.
57 + if(hours == 0 && minutes == 0) {
58 + // For UTC, append 'Z' to the timestamp.
59 + if(used_length + 1 < len) {
60 + buffer[used_length++] = 'Z';
61 + buffer[used_length] = '\0'; // null-terminate the string.
62 + }
63 + }
64 + else {
65 + // For non-UTC, format the timezone offset. Omit minutes if they are zero.
66 + if(minutes == 0) {
67 + // Check enough space is available for the timezone offset string.
68 + if(used_length + 3 < len) // "+hh\0"
69 + used_length += snprintfz(buffer + used_length, len - used_length, "%+03d", hours);
70 + }
71 + else {
72 + // Check enough space is available for the timezone offset string.
73 + if(used_length + 6 < len) // "+hh:mm\0"
74 + used_length += snprintfz(buffer + used_length, len - used_length,
75 + "%+03d:%02d", hours, abs(minutes));
76 + }
77 + }
78 + }
79 +
80 + return used_length;
81 +}
libnetdata/datetime/iso8601.h new
+18
@@ -0,0 +1,18 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
4 +
5 +#ifndef NETDATA_ISO8601_H
6 +#define NETDATA_ISO8601_H
7 +
8 +typedef enum __attribute__((__packed__)) {
9 + ISO8601_UTC = (1 << 0),
10 + ISO8601_LOCAL_TIMEZONE = (1 << 1),
11 + ISO8601_MILLISECONDS = (1 << 2),
12 + ISO8601_MICROSECONDS = (1 << 3),
13 +} ISO8601_OPTIONS;
14 +
15 +#define ISO8601_MAX_LENGTH 64
16 +size_t iso8601_datetime_ut(char *buffer, size_t len, usec_t now_ut, ISO8601_OPTIONS options);
17 +
18 +#endif //NETDATA_ISO8601_H
libnetdata/datetime/rfc7231.c new
+29
@@ -0,0 +1,29 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
4 +
5 +inline size_t rfc7231_datetime(char *buffer, size_t len, time_t now_t) {
6 + if (unlikely(!buffer || !len))
7 + return 0;
8 +
9 + struct tm *tmp, tmbuf;
10 +
11 + // Use gmtime_r for UTC time conversion.
12 + tmp = gmtime_r(&now_t, &tmbuf);
13 +
14 + if (unlikely(!tmp)) {
15 + buffer[0] = '\0';
16 + return 0;
17 + }
18 +
19 + // Format the date and time according to the RFC 7231 format.
20 + size_t ret = strftime(buffer, len, "%a, %d %b %Y %H:%M:%S GMT", tmp);
21 + if (unlikely(ret == 0))
22 + buffer[0] = '\0';
23 +
24 + return ret;
25 +}
26 +
27 +size_t rfc7231_datetime_ut(char *buffer, size_t len, usec_t now_ut) {
28 + return rfc7231_datetime(buffer, len, (time_t) (now_ut / USEC_PER_SEC));
29 +}
libnetdata/datetime/rfc7231.h new
+12
@@ -0,0 +1,12 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
4 +
5 +#ifndef NETDATA_RFC7231_H
6 +#define NETDATA_RFC7231_H
7 +
8 +#define RFC7231_MAX_LENGTH 30
9 +size_t rfc7231_datetime(char *buffer, size_t len, time_t now_t);
10 +size_t rfc7231_datetime_ut(char *buffer, size_t len, usec_t now_ut);
11 +
12 +#endif //NETDATA_RFC7231_H
libnetdata/functions_evloop/functions_evloop.c
+6
@@ -64,6 +64,12 @@ static void *rrd_functions_worker_globals_worker_main(void *arg) {
64 pthread_mutex_unlock(&wg->worker_mutex);
65
66 if(acquired) {
67 + ND_LOG_STACK lgs[] = {
68 + ND_LOG_FIELD_TXT(NDF_REQUEST, j->cmd),
69 + ND_LOG_FIELD_END(),
70 + };
71 + ND_LOG_STACK_PUSH(lgs);
72 +
73 last_acquired = true;
74 j = dictionary_acquired_item_value(acquired);
75 j->cb(j->transaction, j->cmd, j->timeout, &j->cancelled);
libnetdata/inlined.h
+20 -16
@@ -426,13 +426,24 @@ static inline void sanitize_json_string(char *dst, const char *src, size_t dst_s
426 }
427
428 static inline bool sanitize_command_argument_string(char *dst, const char *src, size_t dst_size) {
429 + if(dst_size)
430 + *dst = '\0';
431 +
432 // skip leading dashes
430 - while (src[0] == '-')
433 + while (*src == '-')
434 src++;
435
433 - // escape single quotes
434 - while (src[0] != '\0') {
435 - if (src[0] == '\'') {
436 + while (*src != '\0') {
437 + if (dst_size < 1)
438 + return false;
439 +
440 + if (iscntrl(*src) || *src == '$') {
441 + // remove control characters and characters that are expanded by bash
442 + *dst++ = '_';
443 + dst_size--;
444 + }
445 + else if (*src == '\'' || *src == '`') {
446 + // escape single quotes
447 if (dst_size < 4)
448 return false;
449
@@ -440,14 +451,10 @@ static inline bool sanitize_command_argument_string(char *dst, const char *src,
451
452 dst += 4;
453 dst_size -= 4;
443 - } else {
444 - if (dst_size < 1)
445 - return false;
446 -
447 - dst[0] = src[0];
448 -
449 - dst += 1;
450 - dst_size -= 1;
454 + }
455 + else {
456 + *dst++ = *src;
457 + dst_size--;
458 }
459
460 src++;
@@ -456,6 +463,7 @@ static inline bool sanitize_command_argument_string(char *dst, const char *src,
463 // make sure we have space to terminate the string
464 if (dst_size == 0)
465 return false;
466 +
467 *dst = '\0';
468
469 return true;
@@ -531,10 +539,6 @@ static inline int read_single_base64_or_hex_number_file(const char *filename, un
539 }
540 }
541
534 -static inline int uuid_memcmp(const uuid_t *uu1, const uuid_t *uu2) {
535 - return memcmp(uu1, uu2, sizeof(uuid_t));
536 -}
537 -
542 static inline char *strsep_skip_consecutive_separators(char **ptr, char *s) {
543 char *p = (char *)"";
544 while (p && !p[0] && *ptr) p = strsep(ptr, s);
libnetdata/libnetdata.c
+3 -48
@@ -1269,17 +1269,19 @@ char *fgets_trim_len(char *buf, size_t buf_size, FILE *fp, size_t *len) {
1269 return s;
1270 }
1271
1272 +// vsnprintfz() returns the number of bytes actually written - after possible truncation
1273 int vsnprintfz(char *dst, size_t n, const char *fmt, va_list args) {
1274 if(unlikely(!n)) return 0;
1275
1276 int size = vsnprintf(dst, n, fmt, args);
1277 dst[n - 1] = '\0';
1278
1278 - if (unlikely((size_t) size > n)) size = (int)n;
1279 + if (unlikely((size_t) size >= n)) size = (int)(n - 1);
1280
1281 return size;
1282 }
1283
1284 +// snprintfz() returns the number of bytes actually written - after possible truncation
1285 int snprintfz(char *dst, size_t n, const char *fmt, ...) {
1286 va_list args;
1287
@@ -1694,53 +1696,6 @@ char *find_and_replace(const char *src, const char *find, const char *replace, c
1696 return value;
1697 }
1698
1697 -inline int pluginsd_isspace(char c) {
1698 - switch(c) {
1699 - case ' ':
1700 - case '\t':
1701 - case '\r':
1702 - case '\n':
1703 - case '=':
1704 - return 1;
1705 -
1706 - default:
1707 - return 0;
1708 - }
1709 -}
1710 -
1711 -inline int config_isspace(char c) {
1712 - switch (c) {
1713 - case ' ':
1714 - case '\t':
1715 - case '\r':
1716 - case '\n':
1717 - case ',':
1718 - return 1;
1719 -
1720 - default:
1721 - return 0;
1722 - }
1723 -}
1724 -
1725 -inline int group_by_label_isspace(char c) {
1726 - if(c == ',' || c == '|')
1727 - return 1;
1728 -
1729 - return 0;
1730 -}
1731 -
1732 -bool isspace_map_pluginsd[256] = {};
1733 -bool isspace_map_config[256] = {};
1734 -bool isspace_map_group_by_label[256] = {};
1735 -
1736 -__attribute__((constructor)) void initialize_is_space_arrays(void) {
1737 - for(int c = 0; c < 256 ; c++) {
1738 - isspace_map_pluginsd[c] = pluginsd_isspace((char) c);
1739 - isspace_map_config[c] = config_isspace((char) c);
1740 - isspace_map_group_by_label[c] = group_by_label_isspace((char) c);
1741 - }
1742 -}
1743 -
1699 bool run_command_and_copy_output_to_stdout(const char *command, int max_line_length) {
1700 pid_t pid;
1701 FILE *fp = netdata_popen(command, &pid, NULL);
libnetdata/libnetdata.h
+9 -97
@@ -201,6 +201,8 @@ extern "C" {
201 // ----------------------------------------------------------------------------
202 // netdata common definitions
203
204 +#define _cleanup_(x) __attribute__((__cleanup__(x)))
205 +
206 #ifdef __GNUC__
207 #define GCC_VERSION (__GNUC__ * 10000 + __GNUC_MINOR__ * 100 + __GNUC_PATCHLEVEL__)
208 #endif // __GNUC__
@@ -685,103 +687,6 @@ static inline BITMAPX *bitmapX_create(uint32_t bits) {
687 #define COMPRESSION_MAX_OVERHEAD 128
688 #define COMPRESSION_MAX_MSG_SIZE (COMPRESSION_MAX_CHUNK - COMPRESSION_MAX_OVERHEAD - 1)
689 #define PLUGINSD_LINE_MAX (COMPRESSION_MAX_MSG_SIZE - 768)
688 -int pluginsd_isspace(char c);
689 -int config_isspace(char c);
690 -int group_by_label_isspace(char c);
691 -
692 -extern bool isspace_map_pluginsd[256];
693 -extern bool isspace_map_config[256];
694 -extern bool isspace_map_group_by_label[256];
695 -
696 -static inline size_t quoted_strings_splitter(char *str, char **words, size_t max_words, bool *isspace_map) {
697 - char *s = str, quote = 0;
698 - size_t i = 0;
699 -
700 - // skip all white space
701 - while (unlikely(isspace_map[(uint8_t)*s]))
702 - s++;
703 -
704 - if(unlikely(!*s)) {
705 - words[i] = NULL;
706 - return 0;
707 - }
708 -
709 - // check for quote
710 - if (unlikely(*s == '\'' || *s == '"')) {
711 - quote = *s; // remember the quote
712 - s++; // skip the quote
713 - }
714 -
715 - // store the first word
716 - words[i++] = s;
717 -
718 - // while we have something
719 - while (likely(*s)) {
720 - // if it is an escape
721 - if (unlikely(*s == '\\' && s[1])) {
722 - s += 2;
723 - continue;
724 - }
725 -
726 - // if it is a quote
727 - else if (unlikely(*s == quote)) {
728 - quote = 0;
729 - *s = ' ';
730 - continue;
731 - }
732 -
733 - // if it is a space
734 - else if (unlikely(quote == 0 && isspace_map[(uint8_t)*s])) {
735 - // terminate the word
736 - *s++ = '\0';
737 -
738 - // skip all white space
739 - while (likely(isspace_map[(uint8_t)*s]))
740 - s++;
741 -
742 - // check for a quote
743 - if (unlikely(*s == '\'' || *s == '"')) {
744 - quote = *s; // remember the quote
745 - s++; // skip the quote
746 - }
747 -
748 - // if we reached the end, stop
749 - if (unlikely(!*s))
750 - break;
751 -
752 - // store the next word
753 - if (likely(i < max_words))
754 - words[i++] = s;
755 - else
756 - break;
757 - }
758 -
759 - // anything else
760 - else
761 - s++;
762 - }
763 -
764 - if (likely(i < max_words))
765 - words[i] = NULL;
766 -
767 - return i;
768 -}
769 -
770 -#define quoted_strings_splitter_query_group_by_label(str, words, max_words) \
771 - quoted_strings_splitter(str, words, max_words, isspace_map_group_by_label)
772 -
773 -#define quoted_strings_splitter_config(str, words, max_words) \
774 - quoted_strings_splitter(str, words, max_words, isspace_map_config)
775 -
776 -#define quoted_strings_splitter_pluginsd(str, words, max_words) \
777 - quoted_strings_splitter(str, words, max_words, isspace_map_pluginsd)
778 -
779 -static inline char *get_word(char **words, size_t num_words, size_t index) {
780 - if (unlikely(index >= num_words))
781 - return NULL;
782 -
783 - return words[index];
784 -}
690
691 bool run_command_and_copy_output_to_stdout(const char *command, int max_line_length);
692
@@ -803,6 +708,8 @@ extern char *netdata_configured_host_prefix;
708 #define XXH_INLINE_ALL
709 #include "xxhash.h"
710
711 +#include "uuid/uuid.h"
712 +
713 #include "libjudy/src/Judy.h"
714 #include "july/july.h"
715 #include "os.h"
@@ -812,7 +719,10 @@ extern char *netdata_configured_host_prefix;
719 #include "circular_buffer/circular_buffer.h"
720 #include "avl/avl.h"
721 #include "inlined.h"
722 +#include "line_splitter/line_splitter.h"
723 #include "clocks/clocks.h"
724 +#include "datetime/iso8601.h"
725 +#include "datetime/rfc7231.h"
726 #include "completion/completion.h"
727 #include "popen/popen.h"
728 #include "simple_pattern/simple_pattern.h"
@@ -821,7 +731,9 @@ extern char *netdata_configured_host_prefix;
731 #endif
732 #include "socket/socket.h"
733 #include "config/appconfig.h"
734 +#include "log/journal.h"
735 #include "log/log.h"
736 +#include "buffered_reader/buffered_reader.h"
737 #include "procfile/procfile.h"
738 #include "string/string.h"
739 #include "dictionary/dictionary.h"
libnetdata/line_splitter/Makefile.am new
+8
@@ -0,0 +1,8 @@
1 +# SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +AUTOMAKE_OPTIONS = subdir-objects
4 +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
5 +
6 +dist_noinst_DATA = \
7 + README.md \
8 + $(NULL)
libnetdata/line_splitter/README.md new
+14
@@ -0,0 +1,14 @@
1 +<!--
2 +title: "Log"
3 +custom_edit_url: https://github.com/netdata/netdata/edit/master/libnetdata/log/README.md
4 +sidebar_label: "Log"
5 +learn_status: "Published"
6 +learn_topic_type: "Tasks"
7 +learn_rel_path: "Developers/libnetdata"
8 +-->
9 +
10 +# Log
11 +
12 +The netdata log library supports debug, info, error and fatal error logging.
13 +By default we have an access log, an error log and a collectors log.
14 +
libnetdata/line_splitter/line_splitter.c new
+69
@@ -0,0 +1,69 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
4 +
5 +
6 +bool line_splitter_reconstruct_line(BUFFER *wb, void *ptr) {
7 + struct line_splitter *spl = ptr;
8 + if(!spl) return false;
9 +
10 + size_t added = 0;
11 + for(size_t i = 0; i < spl->num_words ;i++) {
12 + if(i) buffer_fast_strcat(wb, " ", 1);
13 +
14 + buffer_fast_strcat(wb, "'", 1);
15 + const char *s = get_word(spl->words, spl->num_words, i);
16 + buffer_strcat(wb, s?s:"");
17 + buffer_fast_strcat(wb, "'", 1);
18 + added++;
19 + }
20 +
21 + return added > 0;
22 +}
23 +
24 +inline int pluginsd_isspace(char c) {
25 + switch(c) {
26 + case ' ':
27 + case '\t':
28 + case '\r':
29 + case '\n':
30 + case '=':
31 + return 1;
32 +
33 + default:
34 + return 0;
35 + }
36 +}
37 +
38 +inline int config_isspace(char c) {
39 + switch (c) {
40 + case ' ':
41 + case '\t':
42 + case '\r':
43 + case '\n':
44 + case ',':
45 + return 1;
46 +
47 + default:
48 + return 0;
49 + }
50 +}
51 +
52 +inline int group_by_label_isspace(char c) {
53 + if(c == ',' || c == '|')
54 + return 1;
55 +
56 + return 0;
57 +}
58 +
59 +bool isspace_map_pluginsd[256] = {};
60 +bool isspace_map_config[256] = {};
61 +bool isspace_map_group_by_label[256] = {};
62 +
63 +__attribute__((constructor)) void initialize_is_space_arrays(void) {
64 + for(int c = 0; c < 256 ; c++) {
65 + isspace_map_pluginsd[c] = pluginsd_isspace((char) c);
66 + isspace_map_config[c] = config_isspace((char) c);
67 + isspace_map_group_by_label[c] = group_by_label_isspace((char) c);
68 + }
69 +}
libnetdata/line_splitter/line_splitter.h new
+120
@@ -0,0 +1,120 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
4 +
5 +#ifndef NETDATA_LINE_SPLITTER_H
6 +#define NETDATA_LINE_SPLITTER_H
7 +
8 +#define PLUGINSD_MAX_WORDS 30
9 +
10 +struct line_splitter {
11 + size_t count; // counts number of lines
12 + char *words[PLUGINSD_MAX_WORDS]; // an array of pointers for the words in this line
13 + size_t num_words; // the number of pointers used in this line
14 +};
15 +
16 +bool line_splitter_reconstruct_line(BUFFER *wb, void *ptr);
17 +
18 +static inline void line_splitter_reset(struct line_splitter *line) {
19 + line->num_words = 0;
20 +}
21 +
22 +int pluginsd_isspace(char c);
23 +int config_isspace(char c);
24 +int group_by_label_isspace(char c);
25 +
26 +extern bool isspace_map_pluginsd[256];
27 +extern bool isspace_map_config[256];
28 +extern bool isspace_map_group_by_label[256];
29 +
30 +static inline size_t quoted_strings_splitter(char *str, char **words, size_t max_words, bool *isspace_map) {
31 + char *s = str, quote = 0;
32 + size_t i = 0;
33 +
34 + // skip all white space
35 + while (unlikely(isspace_map[(uint8_t)*s]))
36 + s++;
37 +
38 + if(unlikely(!*s)) {
39 + words[i] = NULL;
40 + return 0;
41 + }
42 +
43 + // check for quote
44 + if (unlikely(*s == '\'' || *s == '"')) {
45 + quote = *s; // remember the quote
46 + s++; // skip the quote
47 + }
48 +
49 + // store the first word
50 + words[i++] = s;
51 +
52 + // while we have something
53 + while (likely(*s)) {
54 + // if it is an escape
55 + if (unlikely(*s == '\\' && s[1])) {
56 + s += 2;
57 + continue;
58 + }
59 +
60 + // if it is a quote
61 + else if (unlikely(*s == quote)) {
62 + quote = 0;
63 + *s = ' ';
64 + continue;
65 + }
66 +
67 + // if it is a space
68 + else if (unlikely(quote == 0 && isspace_map[(uint8_t)*s])) {
69 + // terminate the word
70 + *s++ = '\0';
71 +
72 + // skip all white space
73 + while (likely(isspace_map[(uint8_t)*s]))
74 + s++;
75 +
76 + // check for a quote
77 + if (unlikely(*s == '\'' || *s == '"')) {
78 + quote = *s; // remember the quote
79 + s++; // skip the quote
80 + }
81 +
82 + // if we reached the end, stop
83 + if (unlikely(!*s))
84 + break;
85 +
86 + // store the next word
87 + if (likely(i < max_words))
88 + words[i++] = s;
89 + else
90 + break;
91 + }
92 +
93 + // anything else
94 + else
95 + s++;
96 + }
97 +
98 + if (likely(i < max_words))
99 + words[i] = NULL;
100 +
101 + return i;
102 +}
103 +
104 +#define quoted_strings_splitter_query_group_by_label(str, words, max_words) \
105 + quoted_strings_splitter(str, words, max_words, isspace_map_group_by_label)
106 +
107 +#define quoted_strings_splitter_config(str, words, max_words) \
108 + quoted_strings_splitter(str, words, max_words, isspace_map_config)
109 +
110 +#define quoted_strings_splitter_pluginsd(str, words, max_words) \
111 + quoted_strings_splitter(str, words, max_words, isspace_map_pluginsd)
112 +
113 +static inline char *get_word(char **words, size_t num_words, size_t index) {
114 + if (unlikely(index >= num_words))
115 + return NULL;
116 +
117 + return words[index];
118 +}
119 +
120 +#endif //NETDATA_LINE_SPLITTER_H
libnetdata/log/Makefile.am
+1
@@ -5,4 +5,5 @@ MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
5
6 dist_noinst_DATA = \
7 README.md \
8 + log2journal.md \
9 $(NULL)
libnetdata/log/README.md
+191 -3
@@ -7,8 +7,196 @@ learn_topic_type: "Tasks"
7 learn_rel_path: "Developers/libnetdata"
8 -->
9
10 -# Log
10 +# Netdata Logging
11
12 -The netdata log library supports debug, info, error and fatal error logging.
13 -By default we have an access log, an error log and a collectors log.
12 +This document describes how Netdata generates its own logs, not how Netdata manages and queries logs databases.
13 +
14 +## Log sources
15 +
16 +Netdata supports the following log sources:
17 +
18 +1. **daemon**, logs generated by Netdata daemon.
19 +2. **collector**, logs generated by Netdata collectors, including internal and external ones.
20 +3. **access**, API requests received by Netdata
21 +4. **health**, logs all alert transitions
22 +
23 +## Log outputs
24 +
25 +For each log source, Netdata supports the following output methods:
26 +
27 +- **off**, to disable this log source
28 +- **journal**, to send the logs to systemd-journal.
29 +- **syslog**, to send the logs to syslog.
30 +- **system**, to send the output to `stderr` or `stdout` depending on the log source.
31 +- **stdout**, to write the logs to Netdata's `stdout`.
32 +- **stderr**, to write the logs to Netdata's `stderr`.
33 +- **filename**, to send the logs to a file.
34 +
35 +For `daemon` and `collector` the default is `journal` when systemd-journal is available.
36 +To decide if systemd-journal is available, Netdata checks:
37 +
38 +1. `stderr` is connected to systemd-journald
39 +2. `/run/systemd/journal/socket` exists
40 +3. `/host/run/systemd/journal/socket` exists (`/host` is configurable in containers)
41 +
42 +If any of the above is detected, Netdata will select `journal` for `daemon` and `collector` sources.
43 +
44 +All other sources default to a file.
45 +
46 +## Log formats
47 +
48 +Netdata supports the follow formats for its logs:
49 +
50 +- **journal**, this is automatically selected when logging to systemd-journal.
51 +- **logfmt**, this is the default when logging to any output other than `journal`. In this format, Netdata annotates the fields to make them human readable.
52 +- **json**, to write logs lines in json format. The output is machine readable, similar to `journal`.
53 +
54 +## Log levels
55 +
56 +Each time Netdata logs, it assigns a priority to the log. It can be one of this (in order of importance):
57 +
58 +- **emergency**, a fatal condition; most likely Netdata will exit immediately after,
59 +- **alert**, a very important issue that may affect how Netdata operates,
60 +- **critical**, a very important issue the user should know which, Netdata thinks it can survive,
61 +- **error**, an error condition indicating that Netdata is trying to do something but it fails,
62 +- **warning**, something that may or may not affect the operation of Netdata, but the outcome cannot be determined at the time Netdata logs,
63 +- **notice**, something that does not affect the operation of Netdata, but the user should notice,
64 +- **info**, the default log level about information the user should know,
65 +- **debug**, these are more verbose logs that can be ignored,
66 +
67 +## Logs Configuration
68 +
69 +In `netdata.conf`, there are the following settings:
70 +
71 +```
72 +[logs]
73 + # logs to trigger flood protection = 600
74 + # logs flood protection period = 3600
75 + # facility = daemon
76 + # level = info
77 + # daemon = journal
78 + # collector = journal
79 + # access = /var/log/netdata/access.log
80 + # health = /var/log/netdata/health.log
81 +```
82 +
83 +- `logs to trigger flood protection` and `logs flood protection period` enable logs flood protection for `daemon` and `collector` sources. It can also be configured per log source.
84 +- `facility` is used only when Netdata logs to syslog.
85 +- `level` defines the minimum [log level](#log-levels) of logs that will be logged. This setting is applied only to `daemon` and `collector` sources. It can also be configured per source.
86 +
87 +### Configuring log sources
88 +
89 +Each for the sources (`daemon`, `collector`, `access`, `health`), accepts the following:
90 +
91 +```
92 +source = {FORMAT},level={LEVEL},protection={LOG}/{PERIOD}@{OUTPUT}
93 +```
94 +
95 +Where:
96 +
97 +- `{FORMAT}`, is one of the [log formats](#log-formats),
98 +- `{LEVEL}`, is the minimum [log level](#log-levels) to be logged,
99 +- `{LOGS}` is the number of `logs to trigger flood protection` configured per output,
100 +- `{PERIOD}` is the equivalent of `logs flood protection period` configured per output,
101 +- `{OUTPUT}` is one of the `[log outputs](#log-outputs),
102 +
103 +All parameters can be omitted, except `{OUTPUT}`. If `{OUTPUT}` is the only given parameter, `@` can be omitted.
104 +
105 +### Logs rotation
106 +
107 +Netdata comes with `logrotate` configuration to rotate its log files periodically.
108 +
109 +The default is usually found in `/etc/logrotate.d/netdata`.
110 +
111 +Sending a `SIGHUP` to Netdata, will instruct it to re-open all its log files.
112 +
113 +## Log Fields
114 +
115 +Netdata exposes the following fields to its logs:
116 +
117 +| journal | logfmt | json | Description |
118 +|:--------------------------------------:|:------------------------------:|:------------------------------:|:---------------------------------------------------------------------------------------------------------:|
119 +| `_SOURCE_REALTIME_TIMESTAMP` | `time` | `time` | the timestamp of the event |
120 +| `SYSLOG_IDENTIFIER` | `comm` | `comm` | the program logging the event |
121 +| `ND_LOG_SOURCE` | `source` | `source` | one of the [log sources](#log-sources) |
122 +| `PRIORITY`<br/>numeric | `level`<br/>text | `level`<br/>numeric | one of the [log levels](#log-levels) |
123 +| `ERRNO` | `errno` | `errno` | the numeric value of `errno` |
124 +| `INVOCATION_ID` | - | - | a unique UUID of the Netdata session, reset on every Netdata restart, inherited by systemd when available |
125 +| `CODE_LINE` | - | - | the line number of of the source code logging this event |
126 +| `CODE_FILE` | - | - | the filename of the source code logging this event |
127 +| `CODE_FUNCTION` | - | - | the function name of the source code logging this event |
128 +| `TID` | `tid` | `tid` | the thread id of the thread logging this event |
129 +| `THREAD_TAG` | `thread` | `thread` | the name of the thread logging this event |
130 +| `MESSAGE_ID` | `msg_id` | `msg_id` | see [message IDs](#message-ids) |
131 +| `ND_MODULE` | `module` | `module` | the Netdata module logging this event |
132 +| `ND_NIDL_NODE` | `node` | `node` | the hostname of the node the event is related to |
133 +| `ND_NIDL_INSTANCE` | `instance` | `instance` | the instance of the node the event is related to |
134 +| `ND_NIDL_CONTEXT` | `context` | `context` | the context the event is related to (this is usually the chart name, as shown on netdata dashboards |
135 +| `ND_NIDL_DIMENSION` | `dimension` | `dimension` | the dimension the event is related to |
136 +| `ND_SRC_TRANSPORT` | `src_transport` | `src_transport` | when the event happened during a request, this is the request transport |
137 +| `ND_SRC_IP` | `src_ip` | `src_ip` | when the event happened during an inbound request, this is the IP the request came from |
138 +| `ND_SRC_PORT` | `src_port` | `src_port` | when the event happened during an inbound request, this is the port the request came from |
139 +| `ND_SRC_CAPABILITIES` | `src_capabilities` | `src_capabilities` | when the request came from a child, this is the communication capabilities of the child |
140 +| `ND_DST_TRANSPORT` | `dst_transport` | `dst_transport` | when the event happened during an outbound request, this is the outbound request transport |
141 +| `ND_DST_IP` | `dst_ip` | `dst_ip` | when the event happened during an outbound request, this is the IP the request destination |
142 +| `ND_DST_PORT` | `dst_port` | `dst_port` | when the event happened during an outbound request, this is the port the request destination |
143 +| `ND_DST_CAPABILITIES` | `dst_capabilities` | `dst_capabilities` | when the request goes to a parent, this is the communication capabilities of the parent |
144 +| `ND_REQUEST_METHOD` | `req_method` | `req_method` | when the event happened during an inbound request, this is the method the request was received |
145 +| `ND_RESPONSE_CODE` | `code` | `code` | when responding to a request, this this the response code |
146 +| `ND_CONNECTION_ID` | `conn` | `conn` | when there is a connection id for an inbound connection, this is the connection id |
147 +| `ND_TRANSACTION_ID` | `transaction` | `transaction` | the transaction id (UUID) of all API requests |
148 +| `ND_RESPONSE_SENT_BYTES` | `sent_bytes` | `sent_bytes` | the bytes we sent to API responses |
149 +| `ND_RESPONSE_SIZE_BYTES` | `size_bytes` | `size_bytes` | the uncompressed bytes of the API responses |
150 +| `ND_RESPONSE_PREP_TIME_USEC` | `prep_ut` | `prep_ut` | the time needed to prepare a response |
151 +| `ND_RESPONSE_SENT_TIME_USEC` | `sent_ut` | `sent_ut` | the time needed to send a response |
152 +| `ND_RESPONSE_TOTAL_TIME_USEC` | `total_ut` | `total_ut` | the total time needed to complete a response |
153 +| `ND_ALERT_ID` | `alert_id` | `alert_id` | the alert id this event is related to |
154 +| `ND_ALERT_EVENT_ID` | `alert_event_id` | `alert_event_id` | a sequential number of the alert transition (per host) |
155 +| `ND_ALERT_UNIQUE_ID` | `alert_unique_id` | `alert_unique_id` | a sequential number of the alert transition (per alert) |
156 +| `ND_ALERT_TRANSITION_ID` | `alert_transition_id` | `alert_transition_id` | the unique UUID of this alert transition |
157 +| `ND_ALERT_CONFIG` | `alert_config` | `alert_config` | the alert configuration hash (UUID) |
158 +| `ND_ALERT_NAME` | `alert` | `alert` | the alert name |
159 +| `ND_ALERT_CLASS` | `alert_class` | `alert_class` | the alert classification |
160 +| `ND_ALERT_COMPONENT` | `alert_component` | `alert_component` | the alert component |
161 +| `ND_ALERT_TYPE` | `alert_type` | `alert_type` | the alert type |
162 +| `ND_ALERT_EXEC` | `alert_exec` | `alert_exec` | the alert notification program |
163 +| `ND_ALERT_RECIPIENT` | `alert_recipient` | `alert_recipient` | the alert recipient(s) |
164 +| `ND_ALERT_VALUE` | `alert_value` | `alert_value` | the current alert value |
165 +| `ND_ALERT_VALUE_OLD` | `alert_value_old` | `alert_value_old` | the previous alert value |
166 +| `ND_ALERT_STATUS` | `alert_status` | `alert_status` | the current alert status |
167 +| `ND_ALERT_STATUS_OLD` | `alert_value_old` | `alert_value_old` | the previous alert value |
168 +| `ND_ALERT_UNITS` | `alert_units` | `alert_units` | the units of the alert |
169 +| `ND_ALERT_SUMMARY` | `alert_summary` | `alert_summary` | the summary text of the alert |
170 +| `ND_ALERT_INFO` | `alert_info` | `alert_info` | the info text of the alert |
171 +| `ND_ALERT_DURATION` | `alert_duration` | `alert_duration` | the duration the alert was in its previous state |
172 +| `ND_ALERT_NOTIFICATION_TIMESTAMP_USEC` | `alert_notification_timestamp` | `alert_notification_timestamp` | the timestamp the notification delivery is scheduled |
173 +| `ND_REQUEST` | `request` | `request` | the full request during which the event happened |
174 +| `MESSAGE` | `msg` | `msg` | the event message |
175 +
176 +
177 +### Message IDs
178 +
179 +Netdata assigns specific message IDs to certain events:
180 +
181 +- `ed4cdb8f1beb4ad3b57cb3cae2d162fa` when a Netdata child connects to this Netdata
182 +- `6e2e3839067648968b646045dbf28d66` when this Netdata connects to a Netdata parent
183 +- `9ce0cb58ab8b44df82c4bf1ad9ee22de` when alerts change state
184 +- `6db0018e83e34320ae2a659d78019fb7` when notifications are sent
185 +
186 +You can view these events using the Netdata systemd-journal.plugin at the `MESSAGE_ID` filter,
187 +or using `journalctl` like this:
188 +
189 +```bash
190 +# query children connection
191 +journalctl MESSAGE_ID=ed4cdb8f1beb4ad3b57cb3cae2d162fa
192 +
193 +# query parent connection
194 +journalctl MESSAGE_ID=6e2e3839067648968b646045dbf28d66
195 +
196 +# query alert transitions
197 +journalctl MESSAGE_ID=9ce0cb58ab8b44df82c4bf1ad9ee22de
198 +
199 +# query alert notifications
200 +journalctl MESSAGE_ID=6db0018e83e34320ae2a659d78019fb7
201 +```
202
libnetdata/log/journal.c new
+138
@@ -0,0 +1,138 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "journal.h"
4 +
5 +bool is_path_unix_socket(const char *path) {
6 + if(!path || !*path)
7 + return false;
8 +
9 + struct stat statbuf;
10 +
11 + // Check if the path is valid
12 + if (!path || !*path)
13 + return false;
14 +
15 + // Use stat to check if the file exists and is a socket
16 + if (stat(path, &statbuf) == -1)
17 + // The file does not exist or cannot be accessed
18 + return false;
19 +
20 + // Check if the file is a socket
21 + if (S_ISSOCK(statbuf.st_mode))
22 + return true;
23 +
24 + return false;
25 +}
26 +
27 +bool is_stderr_connected_to_journal(void) {
28 + const char *journal_stream = getenv("JOURNAL_STREAM");
29 + if (!journal_stream)
30 + return false; // JOURNAL_STREAM is not set
31 +
32 + struct stat stderr_stat;
33 + if (fstat(STDERR_FILENO, &stderr_stat) < 0)
34 + return false; // Error in getting stderr info
35 +
36 + // Parse device and inode from JOURNAL_STREAM
37 + char *endptr;
38 + long journal_dev = strtol(journal_stream, &endptr, 10);
39 + if (*endptr != ':')
40 + return false; // Format error in JOURNAL_STREAM
41 +
42 + long journal_ino = strtol(endptr + 1, NULL, 10);
43 +
44 + return (stderr_stat.st_dev == (dev_t)journal_dev) && (stderr_stat.st_ino == (ino_t)journal_ino);
45 +}
46 +
47 +int journal_direct_fd(const char *path) {
48 + if(!path || !*path)
49 + path = JOURNAL_DIRECT_SOCKET;
50 +
51 + if(!is_path_unix_socket(path))
52 + return -1;
53 +
54 + int fd = socket(AF_UNIX, SOCK_DGRAM, 0);
55 + if (fd < 0) return -1;
56 +
57 + struct sockaddr_un addr;
58 + memset(&addr, 0, sizeof(struct sockaddr_un));
59 + addr.sun_family = AF_UNIX;
60 + strncpy(addr.sun_path, path, sizeof(addr.sun_path) - 1);
61 +
62 + // Connect the socket (optional, but can simplify send operations)
63 + if (connect(fd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
64 + close(fd);
65 + return -1;
66 + }
67 +
68 + return fd;
69 +}
70 +
71 +static inline bool journal_send_with_memfd(int fd, const char *msg, size_t msg_len) {
72 +#if defined(__NR_memfd_create) && defined(MFD_ALLOW_SEALING) && defined(F_ADD_SEALS) && defined(F_SEAL_SHRINK) && defined(F_SEAL_GROW) && defined(F_SEAL_WRITE)
73 + // Create a memory file descriptor
74 + int memfd = (int)syscall(__NR_memfd_create, "journald", MFD_ALLOW_SEALING);
75 + if (memfd < 0) return false;
76 +
77 + // Write data to the memfd
78 + if (write(memfd, msg, msg_len) != (ssize_t)msg_len) {
79 + close(memfd);
80 + return false;
81 + }
82 +
83 + // Seal the memfd to make it immutable
84 + if (fcntl(memfd, F_ADD_SEALS, F_SEAL_SHRINK | F_SEAL_GROW | F_SEAL_WRITE) < 0) {
85 + close(memfd);
86 + return false;
87 + }
88 +
89 + struct iovec iov = {0};
90 + struct msghdr msghdr = {0};
91 + struct cmsghdr *cmsghdr;
92 + char cmsgbuf[CMSG_SPACE(sizeof(int))];
93 +
94 + msghdr.msg_iov = &iov;
95 + msghdr.msg_iovlen = 1;
96 + msghdr.msg_control = cmsgbuf;
97 + msghdr.msg_controllen = sizeof(cmsgbuf);
98 +
99 + cmsghdr = CMSG_FIRSTHDR(&msghdr);
100 + cmsghdr->cmsg_level = SOL_SOCKET;
101 + cmsghdr->cmsg_type = SCM_RIGHTS;
102 + cmsghdr->cmsg_len = CMSG_LEN(sizeof(int));
103 + memcpy(CMSG_DATA(cmsghdr), &memfd, sizeof(int));
104 +
105 + ssize_t r = sendmsg(fd, &msghdr, 0);
106 +
107 + close(memfd);
108 + return r >= 0;
109 +#else
110 + return false;
111 +#endif
112 +}
113 +
114 +bool journal_direct_send(int fd, const char *msg, size_t msg_len) {
115 + // Send the datagram
116 + if (send(fd, msg, msg_len, 0) < 0) {
117 + if(errno != EMSGSIZE)
118 + return false;
119 +
120 + // datagram is too large, fallback to memfd
121 + if(!journal_send_with_memfd(fd, msg, msg_len))
122 + return false;
123 + }
124 +
125 + return true;
126 +}
127 +
128 +void journal_construct_path(char *dst, size_t dst_len, const char *host_prefix, const char *namespace_str) {
129 + if(!host_prefix)
130 + host_prefix = "";
131 +
132 + if(namespace_str)
133 + snprintfz(dst, dst_len, "%s/run/systemd/journal.%s/socket",
134 + host_prefix, namespace_str);
135 + else
136 + snprintfz(dst, dst_len, "%s" JOURNAL_DIRECT_SOCKET,
137 + host_prefix);
138 +}
libnetdata/log/journal.h new
+18
@@ -0,0 +1,18 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
4 +
5 +#ifndef NETDATA_LOG_JOURNAL_H
6 +#define NETDATA_LOG_JOURNAL_H
7 +
8 +#define JOURNAL_DIRECT_SOCKET "/run/systemd/journal/socket"
9 +
10 +void journal_construct_path(char *dst, size_t dst_len, const char *host_prefix, const char *namespace_str);
11 +
12 +int journal_direct_fd(const char *path);
13 +bool journal_direct_send(int fd, const char *msg, size_t msg_len);
14 +
15 +bool is_path_unix_socket(const char *path);
16 +bool is_stderr_connected_to_journal(void);
17 +
18 +#endif //NETDATA_LOG_JOURNAL_H
libnetdata/log/log.c
+2097 -911
@@ -1,1038 +1,2233 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 -#include <daemon/main.h>
3 +#define SD_JOURNAL_SUPPRESS_LOCATION
4 +
5 #include "../libnetdata.h"
6 +#include <daemon/main.h>
7 +
8 +#ifdef __FreeBSD__
9 +#include <sys/endian.h>
10 +#endif
11 +
12 +#ifdef __APPLE__
13 +#include <machine/endian.h>
14 +#endif
15
16 #ifdef HAVE_BACKTRACE
17 #include <execinfo.h>
18 #endif
19
10 -int web_server_is_multithreaded = 1;
20 +#ifdef HAVE_SYSTEMD
21 +#include <systemd/sd-journal.h>
22 +#endif
23 +
24 +#include <syslog.h>
25
26 const char *program_name = "";
27 +
28 uint64_t debug_flags = 0;
29
15 -int access_log_syslog = 0;
16 -int error_log_syslog = 0;
17 -int collector_log_syslog = 0;
18 -int output_log_syslog = 0; // debug log
19 -int health_log_syslog = 0;
30 +#ifdef ENABLE_ACLK
31 +int aclklog_enabled = 0;
32 +#endif
33 +
34 +// ----------------------------------------------------------------------------
35 +
36 +struct nd_log_source;
37 +static bool nd_log_limit_reached(struct nd_log_source *source);
38
21 -int stdaccess_fd = -1;
22 -FILE *stdaccess = NULL;
39 +// ----------------------------------------------------------------------------
40 +// logging method
41 +
42 +typedef enum __attribute__((__packed__)) {
43 + NDLO_DISABLED = 0,
44 + NDLO_DEVNULL,
45 + NDLO_DEFAULT,
46 + NDLO_JOURNAL,
47 + NDLO_SYSLOG,
48 + NDLO_STDOUT,
49 + NDLO_STDERR,
50 + NDLO_FILE,
51 +} ND_LOG_OUTPUT;
52 +
53 +
54 +static struct {
55 + ND_LOG_OUTPUT output;
56 + const char *name;
57 +} nd_log_outputs[] = {
58 + { .output = NDLO_DISABLED, .name = "none" },
59 + { .output = NDLO_DEVNULL, .name = "/dev/null" },
60 + { .output = NDLO_DEFAULT, .name = "default" },
61 + { .output = NDLO_JOURNAL, .name = "journal" },
62 + { .output = NDLO_SYSLOG, .name = "syslog" },
63 + { .output = NDLO_STDOUT, .name = "stdout" },
64 + { .output = NDLO_STDERR, .name = "stderr" },
65 + { .output = NDLO_FILE, .name = "file" },
66 +};
67 +
68 +static ND_LOG_OUTPUT nd_log_output2id(const char *output) {
69 + if(!output || !*output)
70 + return NDLO_DEFAULT;
71 +
72 + size_t entries = sizeof(nd_log_outputs) / sizeof(nd_log_outputs[0]);
73 + for(size_t i = 0; i < entries ;i++) {
74 + if(strcmp(nd_log_outputs[i].name, output) == 0)
75 + return nd_log_outputs[i].output;
76 + }
77
24 -int stdhealth_fd = -1;
25 -FILE *stdhealth = NULL;
78 + return NDLO_FILE;
79 +}
80
27 -int stdcollector_fd = -1;
28 -FILE *stderror = NULL;
81 +static const char *nd_log_id2output(ND_LOG_OUTPUT output) {
82 + size_t entries = sizeof(nd_log_outputs) / sizeof(nd_log_outputs[0]);
83 + for(size_t i = 0; i < entries ;i++) {
84 + if(output == nd_log_outputs[i].output)
85 + return nd_log_outputs[i].name;
86 + }
87
30 -const char *stdaccess_filename = NULL;
31 -const char *stderr_filename = NULL;
32 -const char *stdout_filename = NULL;
33 -const char *facility_log = NULL;
34 -const char *stdhealth_filename = NULL;
35 -const char *stdcollector_filename = NULL;
88 + return "unknown";
89 +}
90
37 -netdata_log_level_t global_log_severity_level = NETDATA_LOG_LEVEL_INFO;
91 +// ----------------------------------------------------------------------------
92 +// workaround strerror_r()
93
39 -#ifdef ENABLE_ACLK
40 -const char *aclklog_filename = NULL;
41 -int aclklog_fd = -1;
42 -FILE *aclklog = NULL;
43 -int aclklog_syslog = 1;
44 -int aclklog_enabled = 0;
94 +#if defined(STRERROR_R_CHAR_P)
95 +// GLIBC version of strerror_r
96 +static const char *strerror_result(const char *a, const char *b) { (void)b; return a; }
97 +#elif defined(HAVE_STRERROR_R)
98 +// POSIX version of strerror_r
99 +static const char *strerror_result(int a, const char *b) { (void)a; return b; }
100 +#elif defined(HAVE_C__GENERIC)
101 +
102 +// what a trick!
103 +// http://stackoverflow.com/questions/479207/function-overloading-in-c
104 +static const char *strerror_result_int(int a, const char *b) { (void)a; return b; }
105 +static const char *strerror_result_string(const char *a, const char *b) { (void)b; return a; }
106 +
107 +#define strerror_result(a, b) _Generic((a), \
108 + int: strerror_result_int, \
109 + char *: strerror_result_string \
110 + )(a, b)
111 +
112 +#else
113 +#error "cannot detect the format of function strerror_r()"
114 #endif
115
116 +static const char *errno2str(int errnum, char *buf, size_t size) {
117 + return strerror_result(strerror_r(errnum, buf, size), buf);
118 +}
119 +
120 // ----------------------------------------------------------------------------
48 -// Log facility(https://tools.ietf.org/html/rfc5424)
121 +// facilities
122 //
50 -// The facilities accepted in the Netdata are in according with the following
51 -// header files for their respective operating system:
52 -// sys/syslog.h (Linux )
123 +// sys/syslog.h (Linux)
124 // sys/sys/syslog.h (FreeBSD)
125 // bsd/sys/syslog.h (darwin-xnu)
126
56 -#define LOG_AUTH_KEY "auth"
57 -#define LOG_AUTHPRIV_KEY "authpriv"
58 -#ifdef __FreeBSD__
59 -# define LOG_CONSOLE_KEY "console"
60 -#endif
61 -#define LOG_CRON_KEY "cron"
62 -#define LOG_DAEMON_KEY "daemon"
63 -#define LOG_FTP_KEY "ftp"
64 -#ifdef __APPLE__
65 -# define LOG_INSTALL_KEY "install"
66 -#endif
67 -#define LOG_KERN_KEY "kern"
68 -#define LOG_LPR_KEY "lpr"
69 -#define LOG_MAIL_KEY "mail"
70 -//#define LOG_INTERNAL_MARK_KEY "mark"
71 -#ifdef __APPLE__
72 -# define LOG_NETINFO_KEY "netinfo"
73 -# define LOG_RAS_KEY "ras"
74 -# define LOG_REMOTEAUTH_KEY "remoteauth"
75 -#endif
76 -#define LOG_NEWS_KEY "news"
77 -#ifdef __FreeBSD__
78 -# define LOG_NTP_KEY "ntp"
79 -#endif
80 -#define LOG_SECURITY_KEY "security"
81 -#define LOG_SYSLOG_KEY "syslog"
82 -#define LOG_USER_KEY "user"
83 -#define LOG_UUCP_KEY "uucp"
84 -#ifdef __APPLE__
85 -# define LOG_LAUNCHD_KEY "launchd"
86 -#endif
87 -#define LOG_LOCAL0_KEY "local0"
88 -#define LOG_LOCAL1_KEY "local1"
89 -#define LOG_LOCAL2_KEY "local2"
90 -#define LOG_LOCAL3_KEY "local3"
91 -#define LOG_LOCAL4_KEY "local4"
92 -#define LOG_LOCAL5_KEY "local5"
93 -#define LOG_LOCAL6_KEY "local6"
94 -#define LOG_LOCAL7_KEY "local7"
95 -
96 -static int log_facility_id(const char *facility_name)
97 -{
98 - static int
99 - hash_auth = 0,
100 - hash_authpriv = 0,
101 -#ifdef __FreeBSD__
102 - hash_console = 0,
103 -#endif
104 - hash_cron = 0,
105 - hash_daemon = 0,
106 - hash_ftp = 0,
107 -#ifdef __APPLE__
108 - hash_install = 0,
109 -#endif
110 - hash_kern = 0,
111 - hash_lpr = 0,
112 - hash_mail = 0,
113 -// hash_mark = 0,
114 -#ifdef __APPLE__
115 - hash_netinfo = 0,
116 - hash_ras = 0,
117 - hash_remoteauth = 0,
118 -#endif
119 - hash_news = 0,
120 -#ifdef __FreeBSD__
121 - hash_ntp = 0,
122 -#endif
123 - hash_security = 0,
124 - hash_syslog = 0,
125 - hash_user = 0,
126 - hash_uucp = 0,
127 -#ifdef __APPLE__
128 - hash_launchd = 0,
129 -#endif
130 - hash_local0 = 0,
131 - hash_local1 = 0,
132 - hash_local2 = 0,
133 - hash_local3 = 0,
134 - hash_local4 = 0,
135 - hash_local5 = 0,
136 - hash_local6 = 0,
137 - hash_local7 = 0;
138 -
139 - if(unlikely(!hash_auth))
140 - {
141 - hash_auth = simple_hash(LOG_AUTH_KEY);
142 - hash_authpriv = simple_hash(LOG_AUTHPRIV_KEY);
143 -#ifdef __FreeBSD__
144 - hash_console = simple_hash(LOG_CONSOLE_KEY);
145 -#endif
146 - hash_cron = simple_hash(LOG_CRON_KEY);
147 - hash_daemon = simple_hash(LOG_DAEMON_KEY);
148 - hash_ftp = simple_hash(LOG_FTP_KEY);
149 -#ifdef __APPLE__
150 - hash_install = simple_hash(LOG_INSTALL_KEY);
151 -#endif
152 - hash_kern = simple_hash(LOG_KERN_KEY);
153 - hash_lpr = simple_hash(LOG_LPR_KEY);
154 - hash_mail = simple_hash(LOG_MAIL_KEY);
155 -// hash_mark = simple_uhash();
156 -#ifdef __APPLE__
157 - hash_netinfo = simple_hash(LOG_NETINFO_KEY);
158 - hash_ras = simple_hash(LOG_RAS_KEY);
159 - hash_remoteauth = simple_hash(LOG_REMOTEAUTH_KEY);
160 -#endif
161 - hash_news = simple_hash(LOG_NEWS_KEY);
127 +static struct {
128 + int facility;
129 + const char *name;
130 +} nd_log_facilities[] = {
131 + { LOG_AUTH, "auth" },
132 + { LOG_AUTHPRIV, "authpriv" },
133 + { LOG_CRON, "cron" },
134 + { LOG_DAEMON, "daemon" },
135 + { LOG_FTP, "ftp" },
136 + { LOG_KERN, "kern" },
137 + { LOG_LPR, "lpr" },
138 + { LOG_MAIL, "mail" },
139 + { LOG_NEWS, "news" },
140 + { LOG_SYSLOG, "syslog" },
141 + { LOG_USER, "user" },
142 + { LOG_UUCP, "uucp" },
143 + { LOG_LOCAL0, "local0" },
144 + { LOG_LOCAL1, "local1" },
145 + { LOG_LOCAL2, "local2" },
146 + { LOG_LOCAL3, "local3" },
147 + { LOG_LOCAL4, "local4" },
148 + { LOG_LOCAL5, "local5" },
149 + { LOG_LOCAL6, "local6" },
150 + { LOG_LOCAL7, "local7" },
151 +
152 #ifdef __FreeBSD__
163 - hash_ntp = simple_hash(LOG_NTP_KEY);
153 + { LOG_CONSOLE, "console" },
154 + { LOG_NTP, "ntp" },
155 +
156 + // FreeBSD does not consider 'security' as deprecated.
157 + { LOG_SECURITY, "security" },
158 +#else
159 + // For all other O/S 'security' is mapped to 'auth'.
160 + { LOG_AUTH, "security" },
161 #endif
165 - hash_security = simple_hash(LOG_SECURITY_KEY);
166 - hash_syslog = simple_hash(LOG_SYSLOG_KEY);
167 - hash_user = simple_hash(LOG_USER_KEY);
168 - hash_uucp = simple_hash(LOG_UUCP_KEY);
162 +
163 #ifdef __APPLE__
170 - hash_launchd = simple_hash(LOG_LAUNCHD_KEY);
164 + { LOG_INSTALL, "install" },
165 + { LOG_NETINFO, "netinfo" },
166 + { LOG_RAS, "ras" },
167 + { LOG_REMOTEAUTH, "remoteauth" },
168 + { LOG_LAUNCHD, "launchd" },
169 +
170 #endif
172 - hash_local0 = simple_hash(LOG_LOCAL0_KEY);
173 - hash_local1 = simple_hash(LOG_LOCAL1_KEY);
174 - hash_local2 = simple_hash(LOG_LOCAL2_KEY);
175 - hash_local3 = simple_hash(LOG_LOCAL3_KEY);
176 - hash_local4 = simple_hash(LOG_LOCAL4_KEY);
177 - hash_local5 = simple_hash(LOG_LOCAL5_KEY);
178 - hash_local6 = simple_hash(LOG_LOCAL6_KEY);
179 - hash_local7 = simple_hash(LOG_LOCAL7_KEY);
171 +};
172 +
173 +static int nd_log_facility2id(const char *facility) {
174 + size_t entries = sizeof(nd_log_facilities) / sizeof(nd_log_facilities[0]);
175 + for(size_t i = 0; i < entries ;i++) {
176 + if(strcmp(nd_log_facilities[i].name, facility) == 0)
177 + return nd_log_facilities[i].facility;
178 }
179
182 - int hash = simple_hash(facility_name);
183 - if ( hash == hash_auth )
184 - {
185 - return LOG_AUTH;
180 + return LOG_DAEMON;
181 +}
182 +
183 +static const char *nd_log_id2facility(int facility) {
184 + size_t entries = sizeof(nd_log_facilities) / sizeof(nd_log_facilities[0]);
185 + for(size_t i = 0; i < entries ;i++) {
186 + if(nd_log_facilities[i].facility == facility)
187 + return nd_log_facilities[i].name;
188 }
187 - else if ( hash == hash_authpriv )
188 - {
189 - return LOG_AUTHPRIV;
189 +
190 + return "daemon";
191 +}
192 +
193 +// ----------------------------------------------------------------------------
194 +// priorities
195 +
196 +static struct {
197 + ND_LOG_FIELD_PRIORITY priority;
198 + const char *name;
199 +} nd_log_priorities[] = {
200 + { .priority = NDLP_EMERG, .name = "emergency" },
201 + { .priority = NDLP_EMERG, .name = "emerg" },
202 + { .priority = NDLP_ALERT, .name = "alert" },
203 + { .priority = NDLP_CRIT, .name = "critical" },
204 + { .priority = NDLP_CRIT, .name = "crit" },
205 + { .priority = NDLP_ERR, .name = "error" },
206 + { .priority = NDLP_ERR, .name = "err" },
207 + { .priority = NDLP_WARNING, .name = "warning" },
208 + { .priority = NDLP_WARNING, .name = "warn" },
209 + { .priority = NDLP_NOTICE, .name = "notice" },
210 + { .priority = NDLP_INFO, .name = NDLP_INFO_STR },
211 + { .priority = NDLP_DEBUG, .name = "debug" },
212 +};
213 +
214 +int nd_log_priority2id(const char *priority) {
215 + size_t entries = sizeof(nd_log_priorities) / sizeof(nd_log_priorities[0]);
216 + for(size_t i = 0; i < entries ;i++) {
217 + if(strcmp(nd_log_priorities[i].name, priority) == 0)
218 + return nd_log_priorities[i].priority;
219 }
191 -#ifdef __FreeBSD__
192 - else if ( hash == hash_console )
193 - {
194 - return LOG_CONSOLE;
220 +
221 + return NDLP_INFO;
222 +}
223 +
224 +static const char *nd_log_id2priority(ND_LOG_FIELD_PRIORITY priority) {
225 + size_t entries = sizeof(nd_log_priorities) / sizeof(nd_log_priorities[0]);
226 + for(size_t i = 0; i < entries ;i++) {
227 + if(priority == nd_log_priorities[i].priority)
228 + return nd_log_priorities[i].name;
229 }
196 -#endif
197 - else if ( hash == hash_cron )
198 - {
199 - return LOG_CRON;
230 +
231 + return NDLP_INFO_STR;
232 +}
233 +
234 +// ----------------------------------------------------------------------------
235 +// log sources
236 +
237 +const char *log_sources_str[] = {
238 + [NDLS_UNSET] = "UNSET",
239 + [NDLS_ACCESS] = "access",
240 + [NDLS_ACLK] = "aclk",
241 + [NDLS_COLLECTORS] = "collector",
242 + [NDLS_DAEMON] = "daemon",
243 + [NDLS_HEALTH] = "health",
244 + [NDLS_DEBUG] = "debug",
245 +};
246 +
247 +static const char *nd_log_source2str(ND_LOG_SOURCES source) {
248 + size_t entries = sizeof(log_sources_str) / sizeof(log_sources_str[0]);
249 + if(source < entries)
250 + return log_sources_str[source];
251 +
252 + return "UNKNOWN";
253 +}
254 +
255 +// ----------------------------------------------------------------------------
256 +// log output formats
257 +
258 +typedef enum __attribute__((__packed__)) {
259 + NDLOF_JOURNAL,
260 + NDLOF_LOGFMT,
261 + NDLOF_JSON,
262 +} ND_LOG_OUTPUT_FORMAT;
263 +
264 +static struct {
265 + ND_LOG_OUTPUT_FORMAT format;
266 + const char *name;
267 +} nd_log_formats[] = {
268 + { .format = NDLOF_JOURNAL, .name = "journal" },
269 + { .format = NDLOF_LOGFMT, .name = "logfmt" },
270 + { .format = NDLOF_JSON, .name = "json" },
271 +};
272 +
273 +static ND_LOG_OUTPUT_FORMAT nd_log_format2id(const char *format) {
274 + if(!format || !*format)
275 + return NDLOF_LOGFMT;
276 +
277 + size_t entries = sizeof(nd_log_formats) / sizeof(nd_log_formats[0]);
278 + for(size_t i = 0; i < entries ;i++) {
279 + if(strcmp(nd_log_formats[i].name, format) == 0)
280 + return nd_log_formats[i].format;
281 }
201 - else if ( hash == hash_daemon )
202 - {
203 - return LOG_DAEMON;
282 +
283 + return NDLOF_LOGFMT;
284 +}
285 +
286 +static const char *nd_log_id2format(ND_LOG_OUTPUT_FORMAT format) {
287 + size_t entries = sizeof(nd_log_formats) / sizeof(nd_log_formats[0]);
288 + for(size_t i = 0; i < entries ;i++) {
289 + if(format == nd_log_formats[i].format)
290 + return nd_log_formats[i].name;
291 }
205 - else if ( hash == hash_ftp )
206 - {
207 - return LOG_FTP;
292 +
293 + return "logfmt";
294 +}
295 +
296 +// ----------------------------------------------------------------------------
297 +// format dates
298 +
299 +void log_date(char *buffer, size_t len, time_t now) {
300 + if(unlikely(!buffer || !len))
301 + return;
302 +
303 + time_t t = now;
304 + struct tm *tmp, tmbuf;
305 +
306 + tmp = localtime_r(&t, &tmbuf);
307 +
308 + if (unlikely(!tmp)) {
309 + buffer[0] = '\0';
310 + return;
311 }
209 -#ifdef __APPLE__
210 - else if ( hash == hash_install )
211 - {
212 - return LOG_INSTALL;
312 +
313 + if (unlikely(strftime(buffer, len, "%Y-%m-%d %H:%M:%S", tmp) == 0))
314 + buffer[0] = '\0';
315 +
316 + buffer[len - 1] = '\0';
317 +}
318 +
319 +// ----------------------------------------------------------------------------
320 +
321 +struct nd_log_limit {
322 + usec_t started_monotonic_ut;
323 + uint32_t counter;
324 + uint32_t prevented;
325 +
326 + uint32_t throttle_period;
327 + uint32_t logs_per_period;
328 + uint32_t logs_per_period_backup;
329 +};
330 +
331 +#define ND_LOG_LIMITS_DEFAULT (struct nd_log_limit){ .logs_per_period = ND_LOG_DEFAULT_THROTTLE_LOGS, .logs_per_period_backup = ND_LOG_DEFAULT_THROTTLE_LOGS, .throttle_period = ND_LOG_DEFAULT_THROTTLE_PERIOD, }
332 +#define ND_LOG_LIMITS_UNLIMITED (struct nd_log_limit){ .logs_per_period = 0, .logs_per_period_backup = 0, .throttle_period = 0, }
333 +
334 +struct nd_log_source {
335 + SPINLOCK spinlock;
336 + ND_LOG_OUTPUT method;
337 + ND_LOG_OUTPUT_FORMAT format;
338 + const char *filename;
339 + int fd;
340 + FILE *fp;
341 +
342 + ND_LOG_FIELD_PRIORITY min_priority;
343 + const char *pending_msg;
344 + struct nd_log_limit limits;
345 +};
346 +
347 +static __thread ND_LOG_SOURCES overwrite_thread_source = 0;
348 +
349 +void nd_log_set_thread_source(ND_LOG_SOURCES source) {
350 + overwrite_thread_source = source;
351 +}
352 +
353 +static struct {
354 + uuid_t invocation_id;
355 +
356 + ND_LOG_SOURCES overwrite_process_source;
357 +
358 + struct nd_log_source sources[_NDLS_MAX];
359 +
360 + struct {
361 + bool initialized;
362 + } journal;
363 +
364 + struct {
365 + bool initialized;
366 + int fd;
367 + } journal_direct;
368 +
369 + struct {
370 + bool initialized;
371 + int facility;
372 + } syslog;
373 +
374 + struct {
375 + SPINLOCK spinlock;
376 + bool initialized;
377 + } std_output;
378 +
379 + struct {
380 + SPINLOCK spinlock;
381 + bool initialized;
382 + } std_error;
383 +
384 +} nd_log = {
385 + .overwrite_process_source = 0,
386 + .journal = {
387 + .initialized = false,
388 + },
389 + .journal_direct = {
390 + .initialized = false,
391 + .fd = -1,
392 + },
393 + .syslog = {
394 + .initialized = false,
395 + .facility = LOG_DAEMON,
396 + },
397 + .std_output = {
398 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
399 + .initialized = false,
400 + },
401 + .std_error = {
402 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
403 + .initialized = false,
404 + },
405 + .sources = {
406 + [NDLS_UNSET] = {
407 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
408 + .method = NDLO_DISABLED,
409 + .format = NDLOF_JOURNAL,
410 + .filename = NULL,
411 + .fd = -1,
412 + .fp = NULL,
413 + .min_priority = NDLP_EMERG,
414 + .limits = ND_LOG_LIMITS_UNLIMITED,
415 + },
416 + [NDLS_ACCESS] = {
417 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
418 + .method = NDLO_DEFAULT,
419 + .format = NDLOF_LOGFMT,
420 + .filename = LOG_DIR "/access.log",
421 + .fd = -1,
422 + .fp = NULL,
423 + .min_priority = NDLP_DEBUG,
424 + .limits = ND_LOG_LIMITS_UNLIMITED,
425 + },
426 + [NDLS_ACLK] = {
427 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
428 + .method = NDLO_FILE,
429 + .format = NDLOF_LOGFMT,
430 + .filename = LOG_DIR "/aclk.log",
431 + .fd = -1,
432 + .fp = NULL,
433 + .min_priority = NDLP_DEBUG,
434 + .limits = ND_LOG_LIMITS_UNLIMITED,
435 + },
436 + [NDLS_COLLECTORS] = {
437 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
438 + .method = NDLO_DEFAULT,
439 + .format = NDLOF_LOGFMT,
440 + .filename = LOG_DIR "/collectors.log",
441 + .fd = STDERR_FILENO,
442 + .fp = NULL,
443 + .min_priority = NDLP_INFO,
444 + .limits = ND_LOG_LIMITS_DEFAULT,
445 + },
446 + [NDLS_DEBUG] = {
447 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
448 + .method = NDLO_DISABLED,
449 + .format = NDLOF_LOGFMT,
450 + .filename = LOG_DIR "/debug.log",
451 + .fd = STDOUT_FILENO,
452 + .fp = NULL,
453 + .min_priority = NDLP_DEBUG,
454 + .limits = ND_LOG_LIMITS_UNLIMITED,
455 + },
456 + [NDLS_DAEMON] = {
457 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
458 + .method = NDLO_DEFAULT,
459 + .filename = LOG_DIR "/daemon.log",
460 + .format = NDLOF_LOGFMT,
461 + .fd = -1,
462 + .fp = NULL,
463 + .min_priority = NDLP_INFO,
464 + .limits = ND_LOG_LIMITS_DEFAULT,
465 + },
466 + [NDLS_HEALTH] = {
467 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
468 + .method = NDLO_DEFAULT,
469 + .format = NDLOF_LOGFMT,
470 + .filename = LOG_DIR "/health.log",
471 + .fd = -1,
472 + .fp = NULL,
473 + .min_priority = NDLP_DEBUG,
474 + .limits = ND_LOG_LIMITS_UNLIMITED,
475 + },
476 + },
477 +};
478 +
479 +__attribute__((constructor)) void initialize_invocation_id(void) {
480 + // check for a NETDATA_INVOCATION_ID
481 + if(uuid_parse_flexi(getenv("NETDATA_INVOCATION_ID"), nd_log.invocation_id) != 0) {
482 + // not found, check for systemd set INVOCATION_ID
483 + if(uuid_parse_flexi(getenv("INVOCATION_ID"), nd_log.invocation_id) != 0) {
484 + // not found, generate a new one
485 + uuid_generate_random(nd_log.invocation_id);
486 + }
487 }
214 -#endif
215 - else if ( hash == hash_kern )
216 - {
217 - return LOG_KERN;
488 +
489 + char uuid[UUID_COMPACT_STR_LEN];
490 + uuid_unparse_lower_compact(nd_log.invocation_id, uuid);
491 + setenv("NETDATA_INVOCATION_ID", uuid, 1);
492 +}
493 +
494 +void nd_log_set_user_settings(ND_LOG_SOURCES source, const char *setting) {
495 + char buf[FILENAME_MAX + 100];
496 + if(setting && *setting)
497 + strncpyz(buf, setting, sizeof(buf) - 1);
498 + else
499 + buf[0] = '\0';
500 +
501 + struct nd_log_source *ls = &nd_log.sources[source];
502 + char *output = strrchr(buf, '@');
503 +
504 + if(!output)
505 + // all of it is the output
506 + output = buf;
507 + else {
508 + // we found an '@', the next char is the output
509 + *output = '\0';
510 + output++;
511 +
512 + // parse the other params
513 + char *remaining = buf;
514 + while(remaining) {
515 + char *value = strsep_skip_consecutive_separators(&remaining, ",");
516 + if (!value || !*value) continue;
517 +
518 + char *name = strsep_skip_consecutive_separators(&value, "=");
519 + if (!name || !*name) continue;
520 +
521 + if(strcmp(name, "logfmt") == 0)
522 + ls->format = NDLOF_LOGFMT;
523 + else if(strcmp(name, "json") == 0)
524 + ls->format = NDLOF_JSON;
525 + else if(strcmp(name, "journal") == 0)
526 + ls->format = NDLOF_JOURNAL;
527 + else if(strcmp(name, "level") == 0 && value && *value)
528 + ls->min_priority = nd_log_priority2id(value);
529 + else if(strcmp(name, "protection") == 0 && value && *value) {
530 + if(strcmp(value, "off") == 0 || strcmp(value, "none") == 0) {
531 + ls->limits = ND_LOG_LIMITS_UNLIMITED;
532 + ls->limits.counter = 0;
533 + ls->limits.prevented = 0;
534 + }
535 + else {
536 + ls->limits = ND_LOG_LIMITS_DEFAULT;
537 +
538 + char *slash = strchr(value, '/');
539 + if(slash) {
540 + *slash = '\0';
541 + slash++;
542 + ls->limits.logs_per_period = ls->limits.logs_per_period_backup = str2u(value);
543 + ls->limits.throttle_period = str2u(slash);
544 + }
545 + else {
546 + ls->limits.logs_per_period = ls->limits.logs_per_period_backup = str2u(value);
547 + ls->limits.throttle_period = ND_LOG_DEFAULT_THROTTLE_PERIOD;
548 + }
549 + }
550 + }
551 + else
552 + nd_log(NDLS_DAEMON, NDLP_ERR, "Error while parsing configuration of log source '%s'. "
553 + "In config '%s', '%s' is not understood.",
554 + nd_log_source2str(source), setting, name);
555 + }
556 }
219 - else if ( hash == hash_lpr )
220 - {
221 - return LOG_LPR;
557 +
558 + if(!output || !*output || strcmp(output, "none") == 0 || strcmp(output, "off") == 0) {
559 + ls->method = NDLO_DISABLED;
560 + ls->filename = "/dev/null";
561 }
223 - else if ( hash == hash_mail )
224 - {
225 - return LOG_MAIL;
562 + else if(strcmp(output, "journal") == 0) {
563 + ls->method = NDLO_JOURNAL;
564 + ls->filename = NULL;
565 }
227 - /*
228 - else if ( hash == hash_mark )
229 - {
230 - //this is internal for all OS
231 - return INTERNAL_MARK;
566 + else if(strcmp(output, "syslog") == 0) {
567 + ls->method = NDLO_SYSLOG;
568 + ls->filename = NULL;
569 }
233 - */
234 -#ifdef __APPLE__
235 - else if ( hash == hash_netinfo )
236 - {
237 - return LOG_NETINFO;
570 + else if(strcmp(output, "/dev/null") == 0) {
571 + ls->method = NDLO_DEVNULL;
572 + ls->filename = "/dev/null";
573 }
239 - else if ( hash == hash_ras )
240 - {
241 - return LOG_RAS;
574 + else if(strcmp(output, "system") == 0) {
575 + if(ls->fd == STDERR_FILENO) {
576 + ls->method = NDLO_STDERR;
577 + ls->filename = NULL;
578 + ls->fd = STDERR_FILENO;
579 + }
580 + else {
581 + ls->method = NDLO_STDOUT;
582 + ls->filename = NULL;
583 + ls->fd = STDOUT_FILENO;
584 + }
585 }
243 - else if ( hash == hash_remoteauth )
244 - {
245 - return LOG_REMOTEAUTH;
586 + else if(strcmp(output, "stderr") == 0) {
587 + ls->method = NDLO_STDERR;
588 + ls->filename = NULL;
589 + ls->fd = STDERR_FILENO;
590 }
247 -#endif
248 - else if ( hash == hash_news )
249 - {
250 - return LOG_NEWS;
591 + else if(strcmp(output, "stdout") == 0) {
592 + ls->method = NDLO_STDOUT;
593 + ls->filename = NULL;
594 + ls->fd = STDOUT_FILENO;
595 }
252 -#ifdef __FreeBSD__
253 - else if ( hash == hash_ntp )
254 - {
255 - return LOG_NTP;
596 + else {
597 + ls->method = NDLO_FILE;
598 + ls->filename = strdupz(output);
599 }
600 +
601 +#if defined(NETDATA_INTERNAL_CHECKS) || defined(NETDATA_DEV_MODE)
602 + ls->min_priority = NDLP_DEBUG;
603 #endif
258 - else if ( hash == hash_security )
259 - {
260 - //FreeBSD is the unique that does not consider
261 - //this facility deprecated. We are keeping
262 - //it for other OS while they are kept in their headers.
263 -#ifdef __FreeBSD__
264 - return LOG_SECURITY;
604 +
605 + if(source == NDLS_COLLECTORS) {
606 + // set the method for the collector processes we will spawn
607 +
608 + ND_LOG_OUTPUT method;
609 + ND_LOG_OUTPUT_FORMAT format = ls->format;
610 + ND_LOG_FIELD_PRIORITY priority = ls->min_priority;
611 +
612 + if(ls->method == NDLO_SYSLOG || ls->method == NDLO_JOURNAL)
613 + method = ls->method;
614 + else
615 + method = NDLO_STDERR;
616 +
617 + setenv("NETDATA_LOG_METHOD", nd_log_id2output(method), 1);
618 + setenv("NETDATA_LOG_FORMAT", nd_log_id2format(format), 1);
619 + setenv("NETDATA_LOG_SEVERITY_LEVEL", nd_log_id2priority(priority), 1);
620 + setenv("NETDATA_LOG_PRIORITY_LEVEL", nd_log_id2priority(priority), 1);
621 + }
622 +}
623 +
624 +void nd_log_set_priority_level(const char *setting) {
625 + if(!setting || !*setting)
626 + setting = "info";
627 +
628 + ND_LOG_FIELD_PRIORITY priority = nd_log_priority2id(setting);
629 +
630 +#if defined(NETDATA_INTERNAL_CHECKS) || defined(NETDATA_DEV_MODE)
631 + priority = NDLP_DEBUG;
632 +#endif
633 +
634 + nd_log.sources[NDLS_DAEMON].min_priority = priority;
635 + nd_log.sources[NDLS_COLLECTORS].min_priority = priority;
636 +
637 + // backwards compatibility
638 + setenv("NETDATA_LOG_SEVERITY_LEVEL", nd_log_id2priority(priority), 1);
639 +
640 + // the right one
641 + setenv("NETDATA_LOG_PRIORITY_LEVEL", nd_log_id2priority(priority), 1);
642 +}
643 +
644 +void nd_log_set_facility(const char *facility) {
645 + if(!facility || !*facility)
646 + facility = "daemon";
647 +
648 + nd_log.syslog.facility = nd_log_facility2id(facility);
649 + setenv("NETDATA_SYSLOG_FACILITY", nd_log_id2facility(nd_log.syslog.facility), 1);
650 +}
651 +
652 +void nd_log_set_flood_protection(size_t logs, time_t period) {
653 + nd_log.sources[NDLS_DAEMON].limits.logs_per_period =
654 + nd_log.sources[NDLS_DAEMON].limits.logs_per_period_backup;
655 + nd_log.sources[NDLS_COLLECTORS].limits.logs_per_period =
656 + nd_log.sources[NDLS_COLLECTORS].limits.logs_per_period_backup = logs;
657 +
658 + nd_log.sources[NDLS_DAEMON].limits.throttle_period =
659 + nd_log.sources[NDLS_COLLECTORS].limits.throttle_period = period;
660 +
661 + char buf[100];
662 + snprintfz(buf, sizeof(buf), "%" PRIu64, (uint64_t )period);
663 + setenv("NETDATA_ERRORS_THROTTLE_PERIOD", buf, 1);
664 + snprintfz(buf, sizeof(buf), "%" PRIu64, (uint64_t )logs);
665 + setenv("NETDATA_ERRORS_PER_PERIOD", buf, 1);
666 +}
667 +
668 +static bool nd_log_journal_systemd_init(void) {
669 +#ifdef HAVE_SYSTEMD
670 + nd_log.journal.initialized = true;
671 #else
266 - return LOG_AUTH;
672 + nd_log.journal.initialized = false;
673 #endif
674 +
675 + return nd_log.journal.initialized;
676 +}
677 +
678 +static bool nd_log_journal_direct_init(const char *path) {
679 + if(nd_log.journal_direct.initialized)
680 + return true;
681 +
682 + char filename[FILENAME_MAX + 1];
683 + if(!is_path_unix_socket(path)) {
684 + journal_construct_path(filename, sizeof(filename), netdata_configured_host_prefix, "netdata");
685 + if (!is_path_unix_socket(filename)) {
686 + journal_construct_path(filename, sizeof(filename), netdata_configured_host_prefix, NULL);
687 + if (!is_path_unix_socket(filename)) {
688 + journal_construct_path(filename, sizeof(filename), NULL, "netdata");
689 + if (!is_path_unix_socket(filename)) {
690 + journal_construct_path(filename, sizeof(filename), NULL, NULL);
691 + if (!is_path_unix_socket(filename))
692 + return false;
693 + }
694 + }
695 + }
696 }
269 - else if ( hash == hash_syslog )
270 - {
271 - return LOG_SYSLOG;
272 - }
273 - else if ( hash == hash_user )
274 - {
275 - return LOG_USER;
276 - }
277 - else if ( hash == hash_uucp )
278 - {
279 - return LOG_UUCP;
280 - }
281 - else if ( hash == hash_local0 )
282 - {
283 - return LOG_LOCAL0;
284 - }
285 - else if ( hash == hash_local1 )
286 - {
287 - return LOG_LOCAL1;
288 - }
289 - else if ( hash == hash_local2 )
290 - {
291 - return LOG_LOCAL2;
697 + else
698 + snprintfz(filename, sizeof(filename), "%s", path);
699 +
700 + int fd = journal_direct_fd(filename);
701 + if(fd < 0)
702 + return false;
703 +
704 + nd_log.journal_direct.fd = fd;
705 + nd_log.journal_direct.initialized = true;
706 +
707 + if(nd_log.sources[NDLS_COLLECTORS].method == NDLO_JOURNAL)
708 + setenv("NETDATA_SYSTEMD_JOURNAL_PATH", filename, 1);
709 +
710 + return true;
711 +}
712 +
713 +static void nd_log_syslog_init() {
714 + if(nd_log.syslog.initialized)
715 + return;
716 +
717 + openlog(program_name, LOG_PID, nd_log.syslog.facility);
718 + nd_log.syslog.initialized = true;
719 +}
720 +
721 +void nd_log_initialize_for_external_plugins(const char *name) {
722 + // if we don't run under Netdata, log to stderr,
723 + // otherwise, use the logging method Netdata wants us to use.
724 + setenv("NETDATA_LOG_METHOD", "stderr", 0);
725 + setenv("NETDATA_LOG_FORMAT", "logfmt", 0);
726 +
727 + nd_log.overwrite_process_source = NDLS_COLLECTORS;
728 + program_name = name;
729 +
730 + for(size_t i = 0; i < _NDLS_MAX ;i++) {
731 + nd_log.sources[i].method = STDERR_FILENO;
732 + nd_log.sources[i].fd = -1;
733 + nd_log.sources[i].fp = NULL;
734 }
293 - else if ( hash == hash_local3 )
294 - {
295 - return LOG_LOCAL3;
735 +
736 + nd_log_set_priority_level(getenv("NETDATA_LOG_PRIORITY_LEVEL"));
737 + nd_log_set_facility(getenv("NETDATA_SYSLOG_FACILITY"));
738 +
739 + time_t period = 1200;
740 + size_t logs = 200;
741 + const char *s = getenv("NETDATA_ERRORS_THROTTLE_PERIOD");
742 + if(s && *s >= '0' && *s <= '9') {
743 + period = str2l(s);
744 + if(period < 0) period = 0;
745 }
297 - else if ( hash == hash_local4 )
298 - {
299 - return LOG_LOCAL4;
746 +
747 + s = getenv("NETDATA_ERRORS_PER_PERIOD");
748 + if(s && *s >= '0' && *s <= '9')
749 + logs = str2u(s);
750 +
751 + nd_log_set_flood_protection(logs, period);
752 +
753 + if(!netdata_configured_host_prefix) {
754 + s = getenv("NETDATA_HOST_PREFIX");
755 + if(s && *s)
756 + netdata_configured_host_prefix = (char *)s;
757 }
301 - else if ( hash == hash_local5 )
302 - {
303 - return LOG_LOCAL5;
758 +
759 + ND_LOG_OUTPUT method = nd_log_output2id(getenv("NETDATA_LOG_METHOD"));
760 + ND_LOG_OUTPUT_FORMAT format = nd_log_format2id(getenv("NETDATA_LOG_FORMAT"));
761 +
762 + if(method != NDLO_JOURNAL && method != NDLO_SYSLOG && method != NDLO_STDERR) {
763 + if(is_stderr_connected_to_journal()) {
764 + nd_log(NDLS_COLLECTORS, NDLP_WARNING, "NETDATA_LOG_METHOD is not set. Using journal.");
765 + method = NDLO_JOURNAL;
766 + }
767 + else {
768 + nd_log(NDLS_COLLECTORS, NDLP_WARNING, "NETDATA_LOG_METHOD is not set. Using stderr.");
769 + method = NDLO_STDERR;
770 + }
771 }
305 - else if ( hash == hash_local6 )
306 - {
307 - return LOG_LOCAL6;
772 +
773 + switch(method) {
774 + case NDLO_JOURNAL:
775 + if(!nd_log_journal_direct_init(getenv("NETDATA_SYSTEMD_JOURNAL_PATH")) ||
776 + !nd_log_journal_direct_init(NULL) || !nd_log_journal_systemd_init()) {
777 + nd_log(NDLS_COLLECTORS, NDLP_WARNING, "Failed to initialize journal. Using stderr.");
778 + method = NDLO_STDERR;
779 + }
780 + break;
781 +
782 + case NDLO_SYSLOG:
783 + nd_log_syslog_init();
784 + break;
785 +
786 + default:
787 + method = NDLO_STDERR;
788 + break;
789 }
309 - else if ( hash == hash_local7 )
310 - {
311 - return LOG_LOCAL7;
790 +
791 + for(size_t i = 0; i < _NDLS_MAX ;i++) {
792 + nd_log.sources[i].method = method;
793 + nd_log.sources[i].format = format;
794 + nd_log.sources[i].fd = -1;
795 + nd_log.sources[i].fp = NULL;
796 }
313 -#ifdef __APPLE__
314 - else if ( hash == hash_launchd )
315 - {
316 - return LOG_LAUNCHD;
797 +
798 +// nd_log(NDLS_COLLECTORS, NDLP_NOTICE, "FINAL_LOG_METHOD: %s", nd_log_id2method(method));
799 +}
800 +
801 +static bool nd_log_replace_existing_fd(struct nd_log_source *e, int new_fd) {
802 + if(new_fd == -1 || e->fd == -1 ||
803 + (e->fd == STDOUT_FILENO && nd_log.std_output.initialized) ||
804 + (e->fd == STDERR_FILENO && nd_log.std_error.initialized))
805 + return false;
806 +
807 + if(new_fd != e->fd) {
808 + int t = dup2(new_fd, e->fd);
809 +
810 + bool ret = true;
811 + if (t == -1) {
812 + netdata_log_error("Cannot dup2() new fd %d to old fd %d for '%s'", new_fd, e->fd, e->filename);
813 + ret = false;
814 + }
815 + else
816 + close(new_fd);
817 +
818 + if(e->fd == STDOUT_FILENO)
819 + nd_log.std_output.initialized = true;
820 + else if(e->fd == STDERR_FILENO)
821 + nd_log.std_error.initialized = true;
822 +
823 + return ret;
824 }
318 -#endif
825
320 - return LOG_DAEMON;
826 + return false;
827 }
828
323 -//we do not need to use this now, but I already created this function to be
324 -//used case necessary.
325 -/*
326 -char *log_facility_name(int code)
327 -{
328 - char *defvalue = { "daemon" };
329 - switch(code)
330 - {
331 - case LOG_AUTH:
332 - {
333 - return "auth";
334 - }
335 - case LOG_AUTHPRIV:
336 - {
337 - return "authpriv";
338 - }
339 -#ifdef __FreeBSD__
340 - case LOG_CONSOLE:
341 - {
342 - return "console";
343 - }
344 -#endif
345 - case LOG_CRON:
346 - {
347 - return "cron";
348 - }
349 - case LOG_DAEMON:
350 - {
351 - return defvalue;
352 - }
353 - case LOG_FTP:
354 - {
355 - return "ftp";
356 - }
357 -#ifdef __APPLE__
358 - case LOG_INSTALL:
359 - {
360 - return "install";
361 - }
362 -#endif
363 - case LOG_KERN:
364 - {
365 - return "kern";
366 - }
367 - case LOG_LPR:
368 - {
369 - return "lpr";
370 - }
371 - case LOG_MAIL:
372 - {
373 - return "mail";
374 - }
375 -#ifdef __APPLE__
376 - case LOG_NETINFO:
377 - {
378 - return "netinfo" ;
379 - }
380 - case LOG_RAS:
381 - {
382 - return "ras";
383 - }
384 - case LOG_REMOTEAUTH:
385 - {
386 - return "remoteauth";
387 - }
388 -#endif
389 - case LOG_NEWS:
390 - {
391 - return "news";
392 - }
393 -#ifdef __FreeBSD__
394 - case LOG_NTP:
395 - {
396 - return "ntp" ;
397 - }
398 - case LOG_SECURITY:
399 - {
400 - return "security";
401 - }
402 -#endif
403 - case LOG_SYSLOG:
404 - {
405 - return "syslog";
406 - }
407 - case LOG_USER:
408 - {
409 - return "user";
410 - }
411 - case LOG_UUCP:
412 - {
413 - return "uucp";
414 - }
415 - case LOG_LOCAL0:
416 - {
417 - return "local0";
418 - }
419 - case LOG_LOCAL1:
420 - {
421 - return "local1";
422 - }
423 - case LOG_LOCAL2:
424 - {
425 - return "local2";
426 - }
427 - case LOG_LOCAL3:
428 - {
429 - return "local3";
430 - }
431 - case LOG_LOCAL4:
432 - {
433 - return "local4" ;
434 - }
435 - case LOG_LOCAL5:
436 - {
437 - return "local5";
829 +static void nd_log_open(struct nd_log_source *e, ND_LOG_SOURCES source) {
830 + if(e->method == NDLO_DEFAULT)
831 + nd_log_set_user_settings(source, e->filename);
832 +
833 + if((e->method == NDLO_FILE && !e->filename) ||
834 + (e->method == NDLO_DEVNULL && e->fd == -1))
835 + e->method = NDLO_DISABLED;
836 +
837 + if(e->fp)
838 + fflush(e->fp);
839 +
840 + switch(e->method) {
841 + case NDLO_SYSLOG:
842 + nd_log_syslog_init();
843 + break;
844 +
845 + case NDLO_JOURNAL:
846 + nd_log_journal_direct_init(NULL);
847 + nd_log_journal_systemd_init();
848 + break;
849 +
850 + case NDLO_STDOUT:
851 + e->fp = stdout;
852 + e->fd = STDOUT_FILENO;
853 + break;
854 +
855 + default:
856 + case NDLO_DEFAULT:
857 + case NDLO_STDERR:
858 + e->method = NDLO_STDERR;
859 + e->fp = stderr;
860 + e->fd = STDERR_FILENO;
861 + break;
862 +
863 + case NDLO_DEVNULL:
864 + case NDLO_FILE: {
865 + int fd = open(e->filename, O_WRONLY | O_APPEND | O_CREAT, 0664);
866 + if(fd == -1) {
867 + if(e->fd != STDOUT_FILENO && e->fd != STDERR_FILENO) {
868 + e->fd = STDERR_FILENO;
869 + e->method = NDLO_STDERR;
870 + netdata_log_error("Cannot open log file '%s'. Falling back to stderr.", e->filename);
871 + }
872 + else
873 + netdata_log_error("Cannot open log file '%s'. Leaving fd %d as-is.", e->filename, e->fd);
874 }
439 - case LOG_LOCAL6:
440 - {
441 - return "local6";
875 + else {
876 + if (!nd_log_replace_existing_fd(e, fd)) {
877 + if(e->fd == STDOUT_FILENO || e->fd == STDERR_FILENO) {
878 + if(e->fd == STDOUT_FILENO)
879 + e->method = NDLO_STDOUT;
880 + else if(e->fd == STDERR_FILENO)
881 + e->method = NDLO_STDERR;
882 +
883 + // we have dup2() fd, so we can close the one we opened
884 + if(fd != STDOUT_FILENO && fd != STDERR_FILENO)
885 + close(fd);
886 + }
887 + else
888 + e->fd = fd;
889 + }
890 }
443 - case LOG_LOCAL7:
444 - {
445 - return "local7" ;
891 +
892 + // at this point we have e->fd set properly
893 +
894 + if(e->fd == STDOUT_FILENO)
895 + e->fp = stdout;
896 + else if(e->fd == STDERR_FILENO)
897 + e->fp = stderr;
898 +
899 + if(!e->fp) {
900 + e->fp = fdopen(e->fd, "a");
901 + if (!e->fp) {
902 + netdata_log_error("Cannot fdopen() fd %d ('%s')", e->fd, e->filename);
903 +
904 + if(e->fd != STDOUT_FILENO && e->fd != STDERR_FILENO)
905 + close(e->fd);
906 +
907 + e->fp = stderr;
908 + e->fd = STDERR_FILENO;
909 + }
910 }
447 -#ifdef __APPLE__
448 - case LOG_LAUNCHD:
449 - {
450 - return "launchd";
911 + else {
912 + if (setvbuf(e->fp, NULL, _IOLBF, 0) != 0)
913 + netdata_log_error("Cannot set line buffering on fd %d ('%s')", e->fd, e->filename);
914 }
452 -#endif
915 + }
916 + break;
917 }
918 +}
919 +
920 +static void nd_log_stdin_init(int fd, const char *filename) {
921 + int f = open(filename, O_WRONLY | O_APPEND | O_CREAT, 0664);
922 + if(f == -1)
923 + return;
924 +
925 + if(f != fd) {
926 + dup2(f, fd);
927 + close(f);
928 + }
929 +}
930 +
931 +void nd_log_initialize(void) {
932 + nd_log_stdin_init(STDIN_FILENO, "/dev/null");
933
455 - return defvalue;
934 + for(size_t i = 0 ; i < _NDLS_MAX ; i++)
935 + nd_log_open(&nd_log.sources[i], i);
936 }
457 -*/
937
459 -// ----------------------------------------------------------------------------
938 +void nd_log_reopen_log_files(void) {
939 + netdata_log_info("Reopening all log files.");
940
461 -void syslog_init() {
462 - static int i = 0;
941 + nd_log.std_output.initialized = false;
942 + nd_log.std_error.initialized = false;
943 + nd_log_initialize();
944
464 - if(!i) {
465 - openlog(program_name, LOG_PID,log_facility_id(facility_log));
466 - i = 1;
467 - }
945 + netdata_log_info("Log files re-opened.");
946 }
947
470 -void log_date(char *buffer, size_t len, time_t now) {
471 - if(unlikely(!buffer || !len))
472 - return;
473 -
474 - time_t t = now;
475 - struct tm *tmp, tmbuf;
948 +void chown_open_file(int fd, uid_t uid, gid_t gid) {
949 + if(fd == -1) return;
950
477 - tmp = localtime_r(&t, &tmbuf);
951 + struct stat buf;
952
479 - if (tmp == NULL) {
480 - buffer[0] = '\0';
953 + if(fstat(fd, &buf) == -1) {
954 + netdata_log_error("Cannot fstat() fd %d", fd);
955 return;
956 }
957
484 - if (unlikely(strftime(buffer, len, "%Y-%m-%d %H:%M:%S", tmp) == 0))
485 - buffer[0] = '\0';
486 -
487 - buffer[len - 1] = '\0';
958 + if((buf.st_uid != uid || buf.st_gid != gid) && S_ISREG(buf.st_mode)) {
959 + if(fchown(fd, uid, gid) == -1)
960 + netdata_log_error("Cannot fchown() fd %d.", fd);
961 + }
962 }
963
490 -static netdata_mutex_t log_mutex = NETDATA_MUTEX_INITIALIZER;
491 -static inline void log_lock() {
492 - netdata_mutex_lock(&log_mutex);
493 -}
494 -static inline void log_unlock() {
495 - netdata_mutex_unlock(&log_mutex);
964 +void nd_log_chown_log_files(uid_t uid, gid_t gid) {
965 + for(size_t i = 0 ; i < _NDLS_MAX ; i++) {
966 + if(nd_log.sources[i].fd != -1 && nd_log.sources[i].fd != STDIN_FILENO)
967 + chown_open_file(nd_log.sources[i].fd, uid, gid);
968 + }
969 }
970
498 -static FILE *open_log_file(int fd, FILE *fp, const char *filename, int *enabled_syslog, int is_stdaccess, int *fd_ptr) {
499 - int f, devnull = 0;
971 +// ----------------------------------------------------------------------------
972 +// annotators
973 +struct log_field;
974 +static void errno_annotator(BUFFER *wb, const char *key, struct log_field *lf);
975 +static void priority_annotator(BUFFER *wb, const char *key, struct log_field *lf);
976 +static void timestamp_usec_annotator(BUFFER *wb, const char *key, struct log_field *lf);
977 +
978 +// ----------------------------------------------------------------------------
979
501 - if(!filename || !*filename || !strcmp(filename, "none") || !strcmp(filename, "/dev/null")) {
502 - filename = "/dev/null";
503 - devnull = 1;
980 +typedef void (*annotator_t)(BUFFER *wb, const char *key, struct log_field *lf);
981 +
982 +struct log_field {
983 + const char *journal;
984 + const char *logfmt;
985 + annotator_t logfmt_annotator;
986 + struct log_stack_entry entry;
987 +};
988 +
989 +#define THREAD_LOG_STACK_MAX 50
990 +
991 +static __thread struct log_stack_entry *thread_log_stack_base[THREAD_LOG_STACK_MAX];
992 +static __thread size_t thread_log_stack_next = 0;
993 +
994 +static __thread struct log_field thread_log_fields[_NDF_MAX] = {
995 + // THE ORDER DEFINES THE ORDER FIELDS WILL APPEAR IN logfmt
996 +
997 + [NDF_STOP] = { // processing will not stop on this - so it is ok to be first
998 + .journal = NULL,
999 + .logfmt = NULL,
1000 + .logfmt_annotator = NULL,
1001 + },
1002 + [NDF_TIMESTAMP_REALTIME_USEC] = {
1003 + .journal = NULL,
1004 + .logfmt = "time",
1005 + .logfmt_annotator = timestamp_usec_annotator,
1006 + },
1007 + [NDF_SYSLOG_IDENTIFIER] = {
1008 + .journal = "SYSLOG_IDENTIFIER", // standard journald field
1009 + .logfmt = "comm",
1010 + },
1011 + [NDF_LOG_SOURCE] = {
1012 + .journal = "ND_LOG_SOURCE",
1013 + .logfmt = "source",
1014 + },
1015 + [NDF_PRIORITY] = {
1016 + .journal = "PRIORITY", // standard journald field
1017 + .logfmt = "level",
1018 + .logfmt_annotator = priority_annotator,
1019 + },
1020 + [NDF_ERRNO] = {
1021 + .journal = "ERRNO", // standard journald field
1022 + .logfmt = "errno",
1023 + .logfmt_annotator = errno_annotator,
1024 + },
1025 + [NDF_INVOCATION_ID] = {
1026 + .journal = "INVOCATION_ID", // standard journald field
1027 + .logfmt = NULL,
1028 + },
1029 + [NDF_LINE] = {
1030 + .journal = "CODE_LINE", // standard journald field
1031 + .logfmt = NULL,
1032 + },
1033 + [NDF_FILE] = {
1034 + .journal = "CODE_FILE", // standard journald field
1035 + .logfmt = NULL,
1036 + },
1037 + [NDF_FUNC] = {
1038 + .journal = "CODE_FUNC", // standard journald field
1039 + .logfmt = NULL,
1040 + },
1041 + [NDF_TID] = {
1042 + .journal = "TID", // standard journald field
1043 + .logfmt = "tid",
1044 + },
1045 + [NDF_THREAD_TAG] = {
1046 + .journal = "THREAD_TAG",
1047 + .logfmt = "thread",
1048 + },
1049 + [NDF_MESSAGE_ID] = {
1050 + .journal = "MESSAGE_ID",
1051 + .logfmt = "msg_id",
1052 + },
1053 + [NDF_MODULE] = {
1054 + .journal = "ND_MODULE",
1055 + .logfmt = "module",
1056 + },
1057 + [NDF_NIDL_NODE] = {
1058 + .journal = "ND_NIDL_NODE",
1059 + .logfmt = "node",
1060 + },
1061 + [NDF_NIDL_INSTANCE] = {
1062 + .journal = "ND_NIDL_INSTANCE",
1063 + .logfmt = "instance",
1064 + },
1065 + [NDF_NIDL_CONTEXT] = {
1066 + .journal = "ND_NIDL_CONTEXT",
1067 + .logfmt = "context",
1068 + },
1069 + [NDF_NIDL_DIMENSION] = {
1070 + .journal = "ND_NIDL_DIMENSION",
1071 + .logfmt = "dimension",
1072 + },
1073 + [NDF_SRC_TRANSPORT] = {
1074 + .journal = "ND_SRC_TRANSPORT",
1075 + .logfmt = "src_transport",
1076 + },
1077 + [NDF_SRC_IP] = {
1078 + .journal = "ND_SRC_IP",
1079 + .logfmt = "src_ip",
1080 + },
1081 + [NDF_SRC_PORT] = {
1082 + .journal = "ND_SRC_PORT",
1083 + .logfmt = "src_port",
1084 + },
1085 + [NDF_SRC_CAPABILITIES] = {
1086 + .journal = "ND_SRC_CAPABILITIES",
1087 + .logfmt = "src_capabilities",
1088 + },
1089 + [NDF_DST_TRANSPORT] = {
1090 + .journal = "ND_DST_TRANSPORT",
1091 + .logfmt = "dst_transport",
1092 + },
1093 + [NDF_DST_IP] = {
1094 + .journal = "ND_DST_IP",
1095 + .logfmt = "dst_ip",
1096 + },
1097 + [NDF_DST_PORT] = {
1098 + .journal = "ND_DST_PORT",
1099 + .logfmt = "dst_port",
1100 + },
1101 + [NDF_DST_CAPABILITIES] = {
1102 + .journal = "ND_DST_CAPABILITIES",
1103 + .logfmt = "dst_capabilities",
1104 + },
1105 + [NDF_REQUEST_METHOD] = {
1106 + .journal = "ND_REQUEST_METHOD",
1107 + .logfmt = "req_method",
1108 + },
1109 + [NDF_RESPONSE_CODE] = {
1110 + .journal = "ND_RESPONSE_CODE",
1111 + .logfmt = "code",
1112 + },
1113 + [NDF_CONNECTION_ID] = {
1114 + .journal = "ND_CONNECTION_ID",
1115 + .logfmt = "conn",
1116 + },
1117 + [NDF_TRANSACTION_ID] = {
1118 + .journal = "ND_TRANSACTION_ID",
1119 + .logfmt = "transaction",
1120 + },
1121 + [NDF_RESPONSE_SENT_BYTES] = {
1122 + .journal = "ND_RESPONSE_SENT_BYTES",
1123 + .logfmt = "sent_bytes",
1124 + },
1125 + [NDF_RESPONSE_SIZE_BYTES] = {
1126 + .journal = "ND_RESPONSE_SIZE_BYTES",
1127 + .logfmt = "size_bytes",
1128 + },
1129 + [NDF_RESPONSE_PREPARATION_TIME_USEC] = {
1130 + .journal = "ND_RESPONSE_PREP_TIME_USEC",
1131 + .logfmt = "prep_ut",
1132 + },
1133 + [NDF_RESPONSE_SENT_TIME_USEC] = {
1134 + .journal = "ND_RESPONSE_SENT_TIME_USEC",
1135 + .logfmt = "sent_ut",
1136 + },
1137 + [NDF_RESPONSE_TOTAL_TIME_USEC] = {
1138 + .journal = "ND_RESPONSE_TOTAL_TIME_USEC",
1139 + .logfmt = "total_ut",
1140 + },
1141 + [NDF_ALERT_ID] = {
1142 + .journal = "ND_ALERT_ID",
1143 + .logfmt = "alert_id",
1144 + },
1145 + [NDF_ALERT_UNIQUE_ID] = {
1146 + .journal = "ND_ALERT_UNIQUE_ID",
1147 + .logfmt = "alert_unique_id",
1148 + },
1149 + [NDF_ALERT_TRANSITION_ID] = {
1150 + .journal = "ND_ALERT_TRANSITION_ID",
1151 + .logfmt = "alert_transition_id",
1152 + },
1153 + [NDF_ALERT_EVENT_ID] = {
1154 + .journal = "ND_ALERT_EVENT_ID",
1155 + .logfmt = "alert_event_id",
1156 + },
1157 + [NDF_ALERT_CONFIG_HASH] = {
1158 + .journal = "ND_ALERT_CONFIG",
1159 + .logfmt = "alert_config",
1160 + },
1161 + [NDF_ALERT_NAME] = {
1162 + .journal = "ND_ALERT_NAME",
1163 + .logfmt = "alert",
1164 + },
1165 + [NDF_ALERT_CLASS] = {
1166 + .journal = "ND_ALERT_CLASS",
1167 + .logfmt = "alert_class",
1168 + },
1169 + [NDF_ALERT_COMPONENT] = {
1170 + .journal = "ND_ALERT_COMPONENT",
1171 + .logfmt = "alert_component",
1172 + },
1173 + [NDF_ALERT_TYPE] = {
1174 + .journal = "ND_ALERT_TYPE",
1175 + .logfmt = "alert_type",
1176 + },
1177 + [NDF_ALERT_EXEC] = {
1178 + .journal = "ND_ALERT_EXEC",
1179 + .logfmt = "alert_exec",
1180 + },
1181 + [NDF_ALERT_RECIPIENT] = {
1182 + .journal = "ND_ALERT_RECIPIENT",
1183 + .logfmt = "alert_recipient",
1184 + },
1185 + [NDF_ALERT_VALUE] = {
1186 + .journal = "ND_ALERT_VALUE",
1187 + .logfmt = "alert_value",
1188 + },
1189 + [NDF_ALERT_VALUE_OLD] = {
1190 + .journal = "ND_ALERT_VALUE_OLD",
1191 + .logfmt = "alert_value_old",
1192 + },
1193 + [NDF_ALERT_STATUS] = {
1194 + .journal = "ND_ALERT_STATUS",
1195 + .logfmt = "alert_status",
1196 + },
1197 + [NDF_ALERT_STATUS_OLD] = {
1198 + .journal = "ND_ALERT_STATUS_OLD",
1199 + .logfmt = "alert_value_old",
1200 + },
1201 + [NDF_ALERT_UNITS] = {
1202 + .journal = "ND_ALERT_UNITS",
1203 + .logfmt = "alert_units",
1204 + },
1205 + [NDF_ALERT_SUMMARY] = {
1206 + .journal = "ND_ALERT_SUMMARY",
1207 + .logfmt = "alert_summary",
1208 + },
1209 + [NDF_ALERT_INFO] = {
1210 + .journal = "ND_ALERT_INFO",
1211 + .logfmt = "alert_info",
1212 + },
1213 + [NDF_ALERT_DURATION] = {
1214 + .journal = "ND_ALERT_DURATION",
1215 + .logfmt = "alert_duration",
1216 + },
1217 + [NDF_ALERT_NOTIFICATION_REALTIME_USEC] = {
1218 + .journal = "ND_ALERT_NOTIFICATION_TIMESTAMP_USEC",
1219 + .logfmt = "alert_notification_timestamp",
1220 + .logfmt_annotator = timestamp_usec_annotator,
1221 + },
1222 +
1223 + // put new items here
1224 + // leave the request URL and the message last
1225 +
1226 + [NDF_REQUEST] = {
1227 + .journal = "ND_REQUEST",
1228 + .logfmt = "request",
1229 + },
1230 + [NDF_MESSAGE] = {
1231 + .journal = "MESSAGE",
1232 + .logfmt = "msg",
1233 + },
1234 +};
1235 +
1236 +#define THREAD_FIELDS_MAX (sizeof(thread_log_fields) / sizeof(thread_log_fields[0]))
1237 +
1238 +ND_LOG_FIELD_ID nd_log_field_id_by_name(const char *field, size_t len) {
1239 + for(size_t i = 0; i < THREAD_FIELDS_MAX ;i++) {
1240 + if(thread_log_fields[i].journal && strlen(thread_log_fields[i].journal) == len && strncmp(field, thread_log_fields[i].journal, len) == 0)
1241 + return i;
1242 }
1243
506 - if(!strcmp(filename, "syslog")) {
507 - filename = "/dev/null";
508 - devnull = 1;
1244 + return NDF_STOP;
1245 +}
1246
510 - syslog_init();
511 - if(enabled_syslog) *enabled_syslog = 1;
512 - }
513 - else if(enabled_syslog) *enabled_syslog = 0;
514 -
515 - // don't do anything if the user is willing
516 - // to have the standard one
517 - if(!strcmp(filename, "system")) {
518 - if(fd != -1 && !is_stdaccess) {
519 - if(fd_ptr) *fd_ptr = fd;
520 - return fp;
521 - }
1247 +void log_stack_pop(void *ptr) {
1248 + if(!ptr) return;
1249 +
1250 + struct log_stack_entry *lgs = *(struct log_stack_entry (*)[])ptr;
1251
523 - filename = "stderr";
1252 + if(unlikely(!thread_log_stack_next || lgs != thread_log_stack_base[thread_log_stack_next - 1])) {
1253 + fatal("You cannot pop in the middle of the stack, or an item not in the stack");
1254 + return;
1255 }
1256
526 - if(!strcmp(filename, "stdout"))
527 - f = STDOUT_FILENO;
1257 + thread_log_stack_next--;
1258 +}
1259
529 - else if(!strcmp(filename, "stderr"))
530 - f = STDERR_FILENO;
1260 +void log_stack_push(struct log_stack_entry *lgs) {
1261 + if(!lgs || thread_log_stack_next >= THREAD_LOG_STACK_MAX) return;
1262 + thread_log_stack_base[thread_log_stack_next++] = lgs;
1263 +}
1264
532 - else {
533 - f = open(filename, O_WRONLY | O_APPEND | O_CREAT, 0664);
534 - if(f == -1) {
535 - netdata_log_error("Cannot open file '%s'. Leaving %d to its default.", filename, fd);
536 - if(fd_ptr) *fd_ptr = fd;
537 - return fp;
1265 +// ----------------------------------------------------------------------------
1266 +// json formatter
1267 +
1268 +static void nd_logger_json(BUFFER *wb, struct log_field *fields, size_t fields_max) {
1269 +
1270 + // --- FIELD_PARSER_VERSIONS ---
1271 + //
1272 + // IMPORTANT:
1273 + // THERE ARE 6 VERSIONS OF THIS CODE
1274 + //
1275 + // 1. journal (direct socket API),
1276 + // 2. journal (libsystemd API),
1277 + // 3. logfmt,
1278 + // 4. json,
1279 + // 5. convert to uint64
1280 + // 6. convert to int64
1281 + //
1282 + // UPDATE ALL OF THEM FOR NEW FEATURES OR FIXES
1283 +
1284 + buffer_json_initialize(wb, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY);
1285 + CLEAN_BUFFER *tmp = NULL;
1286 +
1287 + for (size_t i = 0; i < fields_max; i++) {
1288 + if (!fields[i].entry.set || !fields[i].logfmt)
1289 + continue;
1290 +
1291 + const char *key = fields[i].logfmt;
1292 +
1293 + const char *s = NULL;
1294 + switch(fields[i].entry.type) {
1295 + case NDFT_TXT:
1296 + s = fields[i].entry.txt;
1297 + break;
1298 + case NDFT_STR:
1299 + s = string2str(fields[i].entry.str);
1300 + break;
1301 + case NDFT_BFR:
1302 + s = buffer_tostring(fields[i].entry.bfr);
1303 + break;
1304 + case NDFT_U64:
1305 + buffer_json_member_add_uint64(wb, key, fields[i].entry.u64);
1306 + break;
1307 + case NDFT_I64:
1308 + buffer_json_member_add_int64(wb, key, fields[i].entry.i64);
1309 + break;
1310 + case NDFT_DBL:
1311 + buffer_json_member_add_double(wb, key, fields[i].entry.dbl);
1312 + break;
1313 + case NDFT_UUID:{
1314 + char u[UUID_COMPACT_STR_LEN];
1315 + uuid_unparse_lower_compact(*fields[i].entry.uuid, u);
1316 + buffer_json_member_add_string(wb, key, u);
1317 + }
1318 + break;
1319 + case NDFT_CALLBACK: {
1320 + if(!tmp)
1321 + tmp = buffer_create(1024, NULL);
1322 + else
1323 + buffer_flush(tmp);
1324 + if(fields[i].entry.cb.formatter(tmp, fields[i].entry.cb.formatter_data))
1325 + s = buffer_tostring(tmp);
1326 + else
1327 + s = NULL;
1328 + }
1329 + break;
1330 + default:
1331 + s = "UNHANDLED";
1332 + break;
1333 }
1334 +
1335 + if(s && *s)
1336 + buffer_json_member_add_string(wb, key, s);
1337 }
1338
541 - // if there is a level-2 file pointer
542 - // flush it before switching the level-1 fds
543 - if(fp)
544 - fflush(fp);
1339 + buffer_json_finalize(wb);
1340 +}
1341
546 - if(devnull && is_stdaccess) {
547 - fd = -1;
548 - fp = NULL;
1342 +// ----------------------------------------------------------------------------
1343 +// logfmt formatter
1344 +
1345 +
1346 +static int64_t log_field_to_int64(struct log_field *lf) {
1347 +
1348 + // --- FIELD_PARSER_VERSIONS ---
1349 + //
1350 + // IMPORTANT:
1351 + // THERE ARE 6 VERSIONS OF THIS CODE
1352 + //
1353 + // 1. journal (direct socket API),
1354 + // 2. journal (libsystemd API),
1355 + // 3. logfmt,
1356 + // 4. json,
1357 + // 5. convert to uint64
1358 + // 6. convert to int64
1359 + //
1360 + // UPDATE ALL OF THEM FOR NEW FEATURES OR FIXES
1361 +
1362 + CLEAN_BUFFER *tmp = NULL;
1363 + const char *s = NULL;
1364 +
1365 + switch(lf->entry.type) {
1366 + case NDFT_UUID:
1367 + case NDFT_UNSET:
1368 + return 0;
1369 +
1370 + case NDFT_TXT:
1371 + s = lf->entry.txt;
1372 + break;
1373 +
1374 + case NDFT_STR:
1375 + s = string2str(lf->entry.str);
1376 + break;
1377 +
1378 + case NDFT_BFR:
1379 + s = buffer_tostring(lf->entry.bfr);
1380 + break;
1381 +
1382 + case NDFT_CALLBACK:
1383 + if(!tmp)
1384 + tmp = buffer_create(0, NULL);
1385 + else
1386 + buffer_flush(tmp);
1387 +
1388 + if(lf->entry.cb.formatter(tmp, lf->entry.cb.formatter_data))
1389 + s = buffer_tostring(tmp);
1390 + else
1391 + s = NULL;
1392 + break;
1393 +
1394 + case NDFT_U64:
1395 + return lf->entry.u64;
1396 +
1397 + case NDFT_I64:
1398 + return lf->entry.i64;
1399 +
1400 + case NDFT_DBL:
1401 + return lf->entry.dbl;
1402 }
1403
551 - if(fd != f && fd != -1) {
552 - // it automatically closes
553 - int t = dup2(f, fd);
554 - if (t == -1) {
555 - netdata_log_error("Cannot dup2() new fd %d to old fd %d for '%s'", f, fd, filename);
556 - close(f);
557 - if(fd_ptr) *fd_ptr = fd;
558 - return fp;
559 - }
560 - // netdata_log_info("dup2() new fd %d to old fd %d for '%s'", f, fd, filename);
561 - close(f);
562 - }
563 - else fd = f;
1404 + if(s && *s)
1405 + return str2ll(s, NULL);
1406
565 - if(!fp) {
566 - fp = fdopen(fd, "a");
567 - if (!fp)
568 - netdata_log_error("Cannot fdopen() fd %d ('%s')", fd, filename);
569 - else {
570 - if (setvbuf(fp, NULL, _IOLBF, 0) != 0)
571 - netdata_log_error("Cannot set line buffering on fd %d ('%s')", fd, filename);
572 - }
1407 + return 0;
1408 +}
1409 +
1410 +static uint64_t log_field_to_uint64(struct log_field *lf) {
1411 +
1412 + // --- FIELD_PARSER_VERSIONS ---
1413 + //
1414 + // IMPORTANT:
1415 + // THERE ARE 6 VERSIONS OF THIS CODE
1416 + //
1417 + // 1. journal (direct socket API),
1418 + // 2. journal (libsystemd API),
1419 + // 3. logfmt,
1420 + // 4. json,
1421 + // 5. convert to uint64
1422 + // 6. convert to int64
1423 + //
1424 + // UPDATE ALL OF THEM FOR NEW FEATURES OR FIXES
1425 +
1426 + CLEAN_BUFFER *tmp = NULL;
1427 + const char *s = NULL;
1428 +
1429 + switch(lf->entry.type) {
1430 + case NDFT_UUID:
1431 + case NDFT_UNSET:
1432 + return 0;
1433 +
1434 + case NDFT_TXT:
1435 + s = lf->entry.txt;
1436 + break;
1437 +
1438 + case NDFT_STR:
1439 + s = string2str(lf->entry.str);
1440 + break;
1441 +
1442 + case NDFT_BFR:
1443 + s = buffer_tostring(lf->entry.bfr);
1444 + break;
1445 +
1446 + case NDFT_CALLBACK:
1447 + if(!tmp)
1448 + tmp = buffer_create(0, NULL);
1449 + else
1450 + buffer_flush(tmp);
1451 +
1452 + if(lf->entry.cb.formatter(tmp, lf->entry.cb.formatter_data))
1453 + s = buffer_tostring(tmp);
1454 + else
1455 + s = NULL;
1456 + break;
1457 +
1458 + case NDFT_U64:
1459 + return lf->entry.u64;
1460 +
1461 + case NDFT_I64:
1462 + return lf->entry.i64;
1463 +
1464 + case NDFT_DBL:
1465 + return lf->entry.dbl;
1466 }
1467
575 - if(fd_ptr) *fd_ptr = fd;
576 - return fp;
577 -}
1468 + if(s && *s)
1469 + return str2uint64_t(s, NULL);
1470
579 -void reopen_all_log_files() {
580 - if(stdout_filename)
581 - open_log_file(STDOUT_FILENO, stdout, stdout_filename, &output_log_syslog, 0, NULL);
1471 + return 0;
1472 +}
1473
583 - if(stdcollector_filename)
584 - open_log_file(STDERR_FILENO, stderr, stdcollector_filename, &collector_log_syslog, 0, NULL);
1474 +static void timestamp_usec_annotator(BUFFER *wb, const char *key, struct log_field *lf) {
1475 + usec_t ut = log_field_to_uint64(lf);
1476
586 - if(stderr_filename) {
587 - // Netdata starts using stderr and if it has success to open file it redirects
588 - FILE *fp = open_log_file(stdcollector_fd, stderror, stderr_filename,
589 - &error_log_syslog, 1, &stdcollector_fd);
590 - if (fp)
591 - stderror = fp;
592 - }
1477 + if(!ut)
1478 + return;
1479
594 -#ifdef ENABLE_ACLK
595 - if (aclklog_enabled)
596 - aclklog = open_log_file(aclklog_fd, aclklog, aclklog_filename, NULL, 0, &aclklog_fd);
597 -#endif
1480 + char datetime[ISO8601_MAX_LENGTH];
1481 + iso8601_datetime_ut(datetime, sizeof(datetime), ut, ISO8601_LOCAL_TIMEZONE | ISO8601_MILLISECONDS);
1482
599 - if(stdaccess_filename)
600 - stdaccess = open_log_file(stdaccess_fd, stdaccess, stdaccess_filename, &access_log_syslog, 1, &stdaccess_fd);
1483 + if(buffer_strlen(wb))
1484 + buffer_fast_strcat(wb, " ", 1);
1485
602 - if(stdhealth_filename)
603 - stdhealth = open_log_file(stdhealth_fd, stdhealth, stdhealth_filename, &health_log_syslog, 1, &stdhealth_fd);
1486 + buffer_strcat(wb, key);
1487 + buffer_fast_strcat(wb, "=", 1);
1488 + buffer_json_strcat(wb, datetime);
1489 }
1490
606 -void open_all_log_files() {
607 - // disable stdin
608 - open_log_file(STDIN_FILENO, stdin, "/dev/null", NULL, 0, NULL);
1491 +static void errno_annotator(BUFFER *wb, const char *key, struct log_field *lf) {
1492 + int64_t errnum = log_field_to_int64(lf);
1493
610 - open_log_file(STDOUT_FILENO, stdout, stdout_filename, &output_log_syslog, 0, NULL);
611 - open_log_file(STDERR_FILENO, stderr, stdcollector_filename, &collector_log_syslog, 0, NULL);
1494 + if(errnum == 0)
1495 + return;
1496
613 - // Netdata starts using stderr and if it has success to open file it redirects
614 - FILE *fp = open_log_file(stdcollector_fd, NULL, stderr_filename, &error_log_syslog, 1, &stdcollector_fd);
615 - if (fp)
616 - stderror = fp;
1497 + char buf[1024];
1498 + const char *s = errno2str(errnum, buf, sizeof(buf));
1499
618 -#ifdef ENABLE_ACLK
619 - if(aclklog_enabled)
620 - aclklog = open_log_file(aclklog_fd, aclklog, aclklog_filename, NULL, 0, &aclklog_fd);
621 -#endif
1500 + if(buffer_strlen(wb))
1501 + buffer_fast_strcat(wb, " ", 1);
1502 +
1503 + buffer_strcat(wb, key);
1504 + buffer_fast_strcat(wb, "=\"", 2);
1505 + buffer_print_int64(wb, errnum);
1506 + buffer_fast_strcat(wb, ", ", 2);
1507 + buffer_json_strcat(wb, s);
1508 + buffer_fast_strcat(wb, "\"", 1);
1509 +}
1510 +
1511 +static void priority_annotator(BUFFER *wb, const char *key, struct log_field *lf) {
1512 + uint64_t pri = log_field_to_uint64(lf);
1513
623 - stdaccess = open_log_file(stdaccess_fd, stdaccess, stdaccess_filename, &access_log_syslog, 1, &stdaccess_fd);
1514 + if(buffer_strlen(wb))
1515 + buffer_fast_strcat(wb, " ", 1);
1516
625 - stdhealth = open_log_file(stdhealth_fd, stdhealth, stdhealth_filename, &health_log_syslog, 1, &stdhealth_fd);
1517 + buffer_strcat(wb, key);
1518 + buffer_fast_strcat(wb, "=", 1);
1519 + buffer_strcat(wb, nd_log_id2priority(pri));
1520 }
1521
628 -// ----------------------------------------------------------------------------
629 -// error log throttling
1522 +static bool needs_quotes_for_logfmt(const char *s) {
1523 + static bool safe_for_logfmt[256] = {
1524 + [' '] = true, ['!'] = true, ['"'] = false, ['#'] = true, ['$'] = true, ['%'] = true, ['&'] = true,
1525 + ['\''] = true, ['('] = true, [')'] = true, ['*'] = true, ['+'] = true, [','] = true, ['-'] = true,
1526 + ['.'] = true, ['/'] = true, ['0'] = true, ['1'] = true, ['2'] = true, ['3'] = true, ['4'] = true,
1527 + ['5'] = true, ['6'] = true, ['7'] = true, ['8'] = true, ['9'] = true, [':'] = true, [';'] = true,
1528 + ['<'] = true, ['='] = true, ['>'] = true, ['?'] = true, ['@'] = true, ['A'] = true, ['B'] = true,
1529 + ['C'] = true, ['D'] = true, ['E'] = true, ['F'] = true, ['G'] = true, ['H'] = true, ['I'] = true,
1530 + ['J'] = true, ['K'] = true, ['L'] = true, ['M'] = true, ['N'] = true, ['O'] = true, ['P'] = true,
1531 + ['Q'] = true, ['R'] = true, ['S'] = true, ['T'] = true, ['U'] = true, ['V'] = true, ['W'] = true,
1532 + ['X'] = true, ['Y'] = true, ['Z'] = true, ['['] = true, ['\\'] = false, [']'] = true, ['^'] = true,
1533 + ['_'] = true, ['`'] = true, ['a'] = true, ['b'] = true, ['c'] = true, ['d'] = true, ['e'] = true,
1534 + ['f'] = true, ['g'] = true, ['h'] = true, ['i'] = true, ['j'] = true, ['k'] = true, ['l'] = true,
1535 + ['m'] = true, ['n'] = true, ['o'] = true, ['p'] = true, ['q'] = true, ['r'] = true, ['s'] = true,
1536 + ['t'] = true, ['u'] = true, ['v'] = true, ['w'] = true, ['x'] = true, ['y'] = true, ['z'] = true,
1537 + ['{'] = true, ['|'] = true, ['}'] = true, ['~'] = true, [0x7f] = true,
1538 + };
1539 +
1540 + if(!*s)
1541 + return true;
1542 +
1543 + while(*s) {
1544 + if(*s == '=' || isspace(*s) || !safe_for_logfmt[(uint8_t)*s])
1545 + return true;
1546 +
1547 + s++;
1548 + }
1549
631 -time_t error_log_throttle_period = 1200;
632 -unsigned long error_log_errors_per_period = 200;
633 -unsigned long error_log_errors_per_period_backup = 0;
1550 + return false;
1551 +}
1552
635 -int error_log_limit(int reset) {
636 - static time_t start = 0;
637 - static unsigned long counter = 0, prevented = 0;
1553 +static void string_to_logfmt(BUFFER *wb, const char *s) {
1554 + bool spaces = needs_quotes_for_logfmt(s);
1555
639 - FILE *fp = stderror ? stderror : stderr;
1556 + if(spaces)
1557 + buffer_fast_strcat(wb, "\"", 1);
1558
641 - // fprintf(fp, "FLOOD: counter=%lu, allowed=%lu, backup=%lu, period=%llu\n", counter, error_log_errors_per_period, error_log_errors_per_period_backup, (unsigned long long)error_log_throttle_period);
1559 + buffer_json_strcat(wb, s);
1560
643 - // do not throttle if the period is 0
644 - if(error_log_throttle_period == 0)
645 - return 0;
1561 + if(spaces)
1562 + buffer_fast_strcat(wb, "\"", 1);
1563 +}
1564
647 - // prevent all logs if the errors per period is 0
648 - if(error_log_errors_per_period == 0)
649 -#ifdef NETDATA_INTERNAL_CHECKS
650 - return 0;
651 -#else
652 - return 1;
653 -#endif
1565 +static void nd_logger_logfmt(BUFFER *wb, struct log_field *fields, size_t fields_max) {
1566
655 - time_t now = now_monotonic_sec();
656 - if(!start) start = now;
657 -
658 - if(reset) {
659 - if(prevented) {
660 - char date[LOG_DATE_LENGTH];
661 - log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
662 - fprintf(
663 - fp,
664 - "%s: %s LOG FLOOD PROTECTION reset for process '%s' "
665 - "(prevented %lu logs in the last %"PRId64" seconds).\n",
666 - date,
667 - program_name,
668 - program_name,
669 - prevented,
670 - (int64_t)(now - start));
671 - }
1567 + // --- FIELD_PARSER_VERSIONS ---
1568 + //
1569 + // IMPORTANT:
1570 + // THERE ARE 6 VERSIONS OF THIS CODE
1571 + //
1572 + // 1. journal (direct socket API),
1573 + // 2. journal (libsystemd API),
1574 + // 3. logfmt,
1575 + // 4. json,
1576 + // 5. convert to uint64
1577 + // 6. convert to int64
1578 + //
1579 + // UPDATE ALL OF THEM FOR NEW FEATURES OR FIXES
1580
673 - start = now;
674 - counter = 0;
675 - prevented = 0;
676 - }
1581 + CLEAN_BUFFER *tmp = NULL;
1582 +
1583 + for (size_t i = 0; i < fields_max; i++) {
1584 + if (!fields[i].entry.set || !fields[i].logfmt)
1585 + continue;
1586
678 - // detect if we log too much
679 - counter++;
680 -
681 - if(now - start > error_log_throttle_period) {
682 - if(prevented) {
683 - char date[LOG_DATE_LENGTH];
684 - log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
685 - fprintf(
686 - fp,
687 - "%s: %s LOG FLOOD PROTECTION resuming logging from process '%s' "
688 - "(prevented %lu logs in the last %"PRId64" seconds).\n",
689 - date,
690 - program_name,
691 - program_name,
692 - prevented,
693 - (int64_t)error_log_throttle_period);
1587 + const char *key = fields[i].logfmt;
1588 +
1589 + if(fields[i].logfmt_annotator)
1590 + fields[i].logfmt_annotator(wb, key, &fields[i]);
1591 + else {
1592 + if(buffer_strlen(wb))
1593 + buffer_fast_strcat(wb, " ", 1);
1594 +
1595 + switch(fields[i].entry.type) {
1596 + case NDFT_TXT:
1597 + if(*fields[i].entry.txt) {
1598 + buffer_strcat(wb, key);
1599 + buffer_fast_strcat(wb, "=", 1);
1600 + string_to_logfmt(wb, fields[i].entry.txt);
1601 + }
1602 + break;
1603 + case NDFT_STR:
1604 + buffer_strcat(wb, key);
1605 + buffer_fast_strcat(wb, "=", 1);
1606 + string_to_logfmt(wb, string2str(fields[i].entry.str));
1607 + break;
1608 + case NDFT_BFR:
1609 + if(buffer_strlen(fields[i].entry.bfr)) {
1610 + buffer_strcat(wb, key);
1611 + buffer_fast_strcat(wb, "=", 1);
1612 + string_to_logfmt(wb, buffer_tostring(fields[i].entry.bfr));
1613 + }
1614 + break;
1615 + case NDFT_U64:
1616 + buffer_strcat(wb, key);
1617 + buffer_fast_strcat(wb, "=", 1);
1618 + buffer_print_uint64(wb, fields[i].entry.u64);
1619 + break;
1620 + case NDFT_I64:
1621 + buffer_strcat(wb, key);
1622 + buffer_fast_strcat(wb, "=", 1);
1623 + buffer_print_int64(wb, fields[i].entry.i64);
1624 + break;
1625 + case NDFT_DBL:
1626 + buffer_strcat(wb, key);
1627 + buffer_fast_strcat(wb, "=", 1);
1628 + buffer_print_netdata_double(wb, fields[i].entry.dbl);
1629 + break;
1630 + case NDFT_UUID: {
1631 + char u[UUID_COMPACT_STR_LEN];
1632 + uuid_unparse_lower_compact(*fields[i].entry.uuid, u);
1633 + buffer_strcat(wb, key);
1634 + buffer_fast_strcat(wb, "=", 1);
1635 + buffer_fast_strcat(wb, u, sizeof(u) - 1);
1636 + }
1637 + break;
1638 + case NDFT_CALLBACK: {
1639 + if(!tmp)
1640 + tmp = buffer_create(1024, NULL);
1641 + else
1642 + buffer_flush(tmp);
1643 + if(fields[i].entry.cb.formatter(tmp, fields[i].entry.cb.formatter_data)) {
1644 + buffer_strcat(wb, key);
1645 + buffer_fast_strcat(wb, "=", 1);
1646 + string_to_logfmt(wb, buffer_tostring(tmp));
1647 + }
1648 + }
1649 + break;
1650 + default:
1651 + buffer_strcat(wb, "UNHANDLED");
1652 + break;
1653 + }
1654 }
1655 + }
1656 +}
1657
696 - // restart the period accounting
697 - start = now;
698 - counter = 1;
699 - prevented = 0;
1658 +// ----------------------------------------------------------------------------
1659 +// journal logger
1660
701 - // log this error
702 - return 0;
1661 +bool nd_log_journal_socket_available(void) {
1662 + if(netdata_configured_host_prefix && *netdata_configured_host_prefix) {
1663 + char filename[FILENAME_MAX + 1];
1664 +
1665 + snprintfz(filename, sizeof(filename), "%s%s",
1666 + netdata_configured_host_prefix, "/run/systemd/journal/socket");
1667 +
1668 + if(is_path_unix_socket(filename))
1669 + return true;
1670 }
1671
705 - if(counter > error_log_errors_per_period) {
706 - if(!prevented) {
707 - char date[LOG_DATE_LENGTH];
708 - log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
709 - fprintf(
710 - fp,
711 - "%s: %s LOG FLOOD PROTECTION too many logs (%lu logs in %"PRId64" seconds, threshold is set to %lu logs "
712 - "in %"PRId64" seconds). Preventing more logs from process '%s' for %"PRId64" seconds.\n",
713 - date,
714 - program_name,
715 - counter,
716 - (int64_t)(now - start),
717 - error_log_errors_per_period,
718 - (int64_t)error_log_throttle_period,
719 - program_name,
720 - (int64_t)(start + error_log_throttle_period - now));
1672 + return is_path_unix_socket("/run/systemd/journal/socket");
1673 +}
1674 +
1675 +static bool nd_logger_journal_libsystemd(struct log_field *fields, size_t fields_max) {
1676 +#ifdef HAVE_SYSTEMD
1677 +
1678 + // --- FIELD_PARSER_VERSIONS ---
1679 + //
1680 + // IMPORTANT:
1681 + // THERE ARE 6 VERSIONS OF THIS CODE
1682 + //
1683 + // 1. journal (direct socket API),
1684 + // 2. journal (libsystemd API),
1685 + // 3. logfmt,
1686 + // 4. json,
1687 + // 5. convert to uint64
1688 + // 6. convert to int64
1689 + //
1690 + // UPDATE ALL OF THEM FOR NEW FEATURES OR FIXES
1691 +
1692 + struct iovec iov[fields_max];
1693 + int iov_count = 0;
1694 +
1695 + memset(iov, 0, sizeof(iov));
1696 +
1697 + CLEAN_BUFFER *tmp = NULL;
1698 +
1699 + for (size_t i = 0; i < fields_max; i++) {
1700 + if (!fields[i].entry.set || !fields[i].journal)
1701 + continue;
1702 +
1703 + const char *key = fields[i].journal;
1704 + char *value = NULL;
1705 + switch (fields[i].entry.type) {
1706 + case NDFT_TXT:
1707 + if(*fields[i].entry.txt)
1708 + asprintf(&value, "%s=%s", key, fields[i].entry.txt);
1709 + break;
1710 + case NDFT_STR:
1711 + asprintf(&value, "%s=%s", key, string2str(fields[i].entry.str));
1712 + break;
1713 + case NDFT_BFR:
1714 + if(buffer_strlen(fields[i].entry.bfr))
1715 + asprintf(&value, "%s=%s", key, buffer_tostring(fields[i].entry.bfr));
1716 + break;
1717 + case NDFT_U64:
1718 + asprintf(&value, "%s=%" PRIu64, key, fields[i].entry.u64);
1719 + break;
1720 + case NDFT_I64:
1721 + asprintf(&value, "%s=%" PRId64, key, fields[i].entry.i64);
1722 + break;
1723 + case NDFT_DBL:
1724 + asprintf(&value, "%s=%f", key, fields[i].entry.dbl);
1725 + break;
1726 + case NDFT_UUID: {
1727 + char u[UUID_COMPACT_STR_LEN];
1728 + uuid_unparse_lower_compact(*fields[i].entry.uuid, u);
1729 + asprintf(&value, "%s=%s", key, u);
1730 + }
1731 + break;
1732 + case NDFT_CALLBACK: {
1733 + if(!tmp)
1734 + tmp = buffer_create(1024, NULL);
1735 + else
1736 + buffer_flush(tmp);
1737 + if(fields[i].entry.cb.formatter(tmp, fields[i].entry.cb.formatter_data))
1738 + asprintf(&value, "%s=%s", key, buffer_tostring(tmp));
1739 + }
1740 + break;
1741 + default:
1742 + asprintf(&value, "%s=%s", key, "UNHANDLED");
1743 + break;
1744 + }
1745 +
1746 + if (value) {
1747 + iov[iov_count].iov_base = value;
1748 + iov[iov_count].iov_len = strlen(value);
1749 + iov_count++;
1750 }
1751 + }
1752
723 - prevented++;
1753 + int r = sd_journal_sendv(iov, iov_count);
1754
725 - // prevent logging this error
726 -#ifdef NETDATA_INTERNAL_CHECKS
727 - return 0;
728 -#else
729 - return 1;
730 -#endif
1755 + // Clean up allocated memory
1756 + for (int i = 0; i < iov_count; i++) {
1757 + if (iov[i].iov_base != NULL) {
1758 + free(iov[i].iov_base);
1759 + }
1760 }
1761
733 - return 0;
1762 + return r == 0;
1763 +#else
1764 + return false;
1765 +#endif
1766 }
1767
736 -void error_log_limit_reset(void) {
737 - log_lock();
1768 +static bool nd_logger_journal_direct(struct log_field *fields, size_t fields_max) {
1769 + if(!nd_log.journal_direct.initialized)
1770 + return false;
1771 +
1772 + // --- FIELD_PARSER_VERSIONS ---
1773 + //
1774 + // IMPORTANT:
1775 + // THERE ARE 6 VERSIONS OF THIS CODE
1776 + //
1777 + // 1. journal (direct socket API),
1778 + // 2. journal (libsystemd API),
1779 + // 3. logfmt,
1780 + // 4. json,
1781 + // 5. convert to uint64
1782 + // 6. convert to int64
1783 + //
1784 + // UPDATE ALL OF THEM FOR NEW FEATURES OR FIXES
1785 +
1786 + CLEAN_BUFFER *wb = buffer_create(4096, NULL);
1787 + CLEAN_BUFFER *tmp = NULL;
1788 +
1789 + for (size_t i = 0; i < fields_max; i++) {
1790 + if (!fields[i].entry.set || !fields[i].journal)
1791 + continue;
1792 +
1793 + const char *key = fields[i].journal;
1794 +
1795 + const char *s = NULL;
1796 + switch(fields[i].entry.type) {
1797 + case NDFT_TXT:
1798 + s = fields[i].entry.txt;
1799 + break;
1800 + case NDFT_STR:
1801 + s = string2str(fields[i].entry.str);
1802 + break;
1803 + case NDFT_BFR:
1804 + s = buffer_tostring(fields[i].entry.bfr);
1805 + break;
1806 + case NDFT_U64:
1807 + buffer_strcat(wb, key);
1808 + buffer_putc(wb, '=');
1809 + buffer_print_uint64(wb, fields[i].entry.u64);
1810 + buffer_putc(wb, '\n');
1811 + break;
1812 + case NDFT_I64:
1813 + buffer_strcat(wb, key);
1814 + buffer_putc(wb, '=');
1815 + buffer_print_int64(wb, fields[i].entry.i64);
1816 + buffer_putc(wb, '\n');
1817 + break;
1818 + case NDFT_DBL:
1819 + buffer_strcat(wb, key);
1820 + buffer_putc(wb, '=');
1821 + buffer_print_netdata_double(wb, fields[i].entry.dbl);
1822 + buffer_putc(wb, '\n');
1823 + break;
1824 + case NDFT_UUID:{
1825 + char u[UUID_COMPACT_STR_LEN];
1826 + uuid_unparse_lower_compact(*fields[i].entry.uuid, u);
1827 + buffer_strcat(wb, key);
1828 + buffer_putc(wb, '=');
1829 + buffer_fast_strcat(wb, u, sizeof(u) - 1);
1830 + buffer_putc(wb, '\n');
1831 + }
1832 + break;
1833 + case NDFT_CALLBACK: {
1834 + if(!tmp)
1835 + tmp = buffer_create(1024, NULL);
1836 + else
1837 + buffer_flush(tmp);
1838 + if(fields[i].entry.cb.formatter(tmp, fields[i].entry.cb.formatter_data))
1839 + s = buffer_tostring(tmp);
1840 + else
1841 + s = NULL;
1842 + }
1843 + break;
1844 + default:
1845 + s = "UNHANDLED";
1846 + break;
1847 + }
1848
739 - error_log_errors_per_period = error_log_errors_per_period_backup;
740 - error_log_limit(1);
1849 + if(s && *s) {
1850 + buffer_strcat(wb, key);
1851 + if(!strchr(s, '\n')) {
1852 + buffer_putc(wb, '=');
1853 + buffer_strcat(wb, s);
1854 + buffer_putc(wb, '\n');
1855 + }
1856 + else {
1857 + buffer_putc(wb, '\n');
1858 + size_t size = strlen(s);
1859 + uint64_t le_size = htole64(size);
1860 + buffer_memcat(wb, &le_size, sizeof(le_size));
1861 + buffer_memcat(wb, s, size);
1862 + buffer_putc(wb, '\n');
1863 + }
1864 + }
1865 + }
1866
742 - log_unlock();
1867 + return journal_direct_send(nd_log.journal_direct.fd, buffer_tostring(wb), buffer_strlen(wb));
1868 }
1869
745 -void error_log_limit_unlimited(void) {
746 - log_lock();
1870 +// ----------------------------------------------------------------------------
1871 +// syslog logger - uses logfmt
1872
748 - error_log_errors_per_period = error_log_errors_per_period_backup;
749 - error_log_limit(1);
1873 +static bool nd_logger_syslog(int priority, ND_LOG_OUTPUT_FORMAT format, struct log_field *fields, size_t fields_max) {
1874 + CLEAN_BUFFER *wb = buffer_create(1024, NULL);
1875
751 - error_log_errors_per_period = ((error_log_errors_per_period_backup * 10) < 10000) ? 10000 : (error_log_errors_per_period_backup * 10);
1876 + nd_logger_logfmt(wb, fields, fields_max);
1877 + syslog(priority, "%s", buffer_tostring(wb));
1878
753 - log_unlock();
1879 + return true;
1880 }
1881
1882 // ----------------------------------------------------------------------------
757 -// debug log
758 -
759 -void debug_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) {
760 - va_list args;
1883 +// file logger - uses logfmt
1884
762 - char date[LOG_DATE_LENGTH];
763 - log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
1885 +static bool nd_logger_file(FILE *fp, ND_LOG_OUTPUT_FORMAT format, struct log_field *fields, size_t fields_max) {
1886 + BUFFER *wb = buffer_create(1024, NULL);
1887
765 - va_start( args, fmt );
766 - printf("%s: %s DEBUG : %s : (%04lu@%-20.20s:%-15.15s): ", date, program_name, netdata_thread_tag(), line, file, function);
767 - vprintf(fmt, args);
768 - va_end( args );
769 - putchar('\n');
1888 + if(format == NDLOF_JSON)
1889 + nd_logger_json(wb, fields, fields_max);
1890 + else
1891 + nd_logger_logfmt(wb, fields, fields_max);
1892
771 - if(output_log_syslog) {
772 - va_start( args, fmt );
773 - vsyslog(LOG_ERR, fmt, args );
774 - va_end( args );
775 - }
1893 + int r = fprintf(fp, "%s\n", buffer_tostring(wb));
1894 + fflush(fp);
1895
777 - fflush(stdout);
1896 + buffer_free(wb);
1897 + return r > 0;
1898 }
1899
1900 // ----------------------------------------------------------------------------
781 -// info log
1901 +// logger router
1902 +
1903 +static ND_LOG_OUTPUT nd_logger_select_output(ND_LOG_SOURCES source, FILE **fpp, SPINLOCK **spinlock) {
1904 + *spinlock = NULL;
1905 + ND_LOG_OUTPUT output = nd_log.sources[source].method;
1906 +
1907 + switch(output) {
1908 + case NDLO_JOURNAL:
1909 + if(unlikely(!nd_log.journal_direct.initialized && !nd_log.journal.initialized)) {
1910 + output = NDLO_FILE;
1911 + *fpp = stderr;
1912 + *spinlock = &nd_log.std_error.spinlock;
1913 + }
1914 + else {
1915 + *fpp = NULL;
1916 + *spinlock = NULL;
1917 + }
1918 + break;
1919
783 -void info_int( int is_collector, const char *file __maybe_unused, const char *function __maybe_unused, const unsigned long line __maybe_unused, const char *fmt, ... )
784 -{
785 -#if !defined(NETDATA_INTERNAL_CHECKS) && !defined(NETDATA_DEV_MODE)
786 - if (NETDATA_LOG_LEVEL_INFO > global_log_severity_level)
787 - return;
788 -#endif
1920 + case NDLO_SYSLOG:
1921 + if(unlikely(!nd_log.syslog.initialized)) {
1922 + output = NDLO_FILE;
1923 + *spinlock = &nd_log.std_error.spinlock;
1924 + *fpp = stderr;
1925 + }
1926 + else {
1927 + *spinlock = NULL;
1928 + *fpp = NULL;
1929 + }
1930 + break;
1931
790 - va_list args;
791 - FILE *fp = (is_collector || !stderror) ? stderr : stderror;
1932 + case NDLO_FILE:
1933 + if(!nd_log.sources[source].fp) {
1934 + *fpp = stderr;
1935 + *spinlock = &nd_log.std_error.spinlock;
1936 + }
1937 + else {
1938 + *fpp = nd_log.sources[source].fp;
1939 + *spinlock = &nd_log.sources[source].spinlock;
1940 + }
1941 + break;
1942
793 - log_lock();
1943 + case NDLO_STDOUT:
1944 + output = NDLO_FILE;
1945 + *fpp = stdout;
1946 + *spinlock = &nd_log.std_output.spinlock;
1947 + break;
1948
795 - // prevent logging too much
796 - if (error_log_limit(0)) {
797 - log_unlock();
798 - return;
1949 + default:
1950 + case NDLO_DEFAULT:
1951 + case NDLO_STDERR:
1952 + output = NDLO_FILE;
1953 + *fpp = stderr;
1954 + *spinlock = &nd_log.std_error.spinlock;
1955 + break;
1956 +
1957 + case NDLO_DISABLED:
1958 + case NDLO_DEVNULL:
1959 + output = NDLO_DISABLED;
1960 + *fpp = NULL;
1961 + *spinlock = NULL;
1962 + break;
1963 }
1964
801 - if(collector_log_syslog) {
802 - va_start( args, fmt );
803 - vsyslog(LOG_INFO, fmt, args );
804 - va_end( args );
1965 + return output;
1966 +}
1967 +
1968 +// ----------------------------------------------------------------------------
1969 +// high level logger
1970 +
1971 +static void nd_logger_log_fields(SPINLOCK *spinlock, FILE *fp, bool limit, ND_LOG_FIELD_PRIORITY priority,
1972 + ND_LOG_OUTPUT output, struct nd_log_source *source,
1973 + struct log_field *fields, size_t fields_max) {
1974 + if(spinlock)
1975 + spinlock_lock(spinlock);
1976 +
1977 + // check the limits
1978 + if(limit && nd_log_limit_reached(source))
1979 + goto cleanup;
1980 +
1981 + if(output == NDLO_JOURNAL) {
1982 + if(!nd_logger_journal_direct(fields, fields_max) && !nd_logger_journal_libsystemd(fields, fields_max)) {
1983 + // we can't log to journal, let's log to stderr
1984 + if(spinlock)
1985 + spinlock_unlock(spinlock);
1986 +
1987 + output = NDLO_FILE;
1988 + spinlock = &nd_log.std_error.spinlock;
1989 + fp = stderr;
1990 +
1991 + if(spinlock)
1992 + spinlock_lock(spinlock);
1993 + }
1994 }
1995
807 - char date[LOG_DATE_LENGTH];
808 - log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
1996 + if(output == NDLO_SYSLOG)
1997 + nd_logger_syslog(priority, source->format, fields, fields_max);
1998
810 - va_start( args, fmt );
811 -#ifdef NETDATA_INTERNAL_CHECKS
812 - fprintf(fp, "%s: %s INFO : %s : (%04lu@%-20.20s:%-15.15s): ",
813 - date, program_name, netdata_thread_tag(), line, file, function);
814 -#else
815 - fprintf(fp, "%s: %s INFO : %s : ", date, program_name, netdata_thread_tag());
816 -#endif
817 - vfprintf(fp, fmt, args );
818 - va_end( args );
1999 + if(output == NDLO_FILE)
2000 + nd_logger_file(fp, source->format, fields, fields_max);
2001
820 - fputc('\n', fp);
2002
822 - log_unlock();
2003 +cleanup:
2004 + if(spinlock)
2005 + spinlock_unlock(spinlock);
2006 }
2007
825 -// ----------------------------------------------------------------------------
826 -// error log
2008 +static void nd_logger_unset_all_thread_fields(void) {
2009 + size_t fields_max = THREAD_FIELDS_MAX;
2010 + for(size_t i = 0; i < fields_max ; i++)
2011 + thread_log_fields[i].entry.set = false;
2012 +}
2013
828 -#if defined(STRERROR_R_CHAR_P)
829 -// GLIBC version of strerror_r
830 -static const char *strerror_result(const char *a, const char *b) { (void)b; return a; }
831 -#elif defined(HAVE_STRERROR_R)
832 -// POSIX version of strerror_r
833 -static const char *strerror_result(int a, const char *b) { (void)a; return b; }
834 -#elif defined(HAVE_C__GENERIC)
2014 +static void nd_logger_merge_log_stack_to_thread_fields(void) {
2015 + for(size_t c = 0; c < thread_log_stack_next ;c++) {
2016 + struct log_stack_entry *lgs = thread_log_stack_base[c];
2017
836 -// what a trick!
837 -// http://stackoverflow.com/questions/479207/function-overloading-in-c
838 -static const char *strerror_result_int(int a, const char *b) { (void)a; return b; }
839 -static const char *strerror_result_string(const char *a, const char *b) { (void)b; return a; }
2018 + for(size_t i = 0; lgs[i].id != NDF_STOP ; i++) {
2019 + if(lgs[i].id >= _NDF_MAX || !lgs[i].set)
2020 + continue;
2021
841 -#define strerror_result(a, b) _Generic((a), \
842 - int: strerror_result_int, \
843 - char *: strerror_result_string \
844 - )(a, b)
2022 + struct log_stack_entry *e = &lgs[i];
2023 + ND_LOG_STACK_FIELD_TYPE type = lgs[i].type;
2024
846 -#else
847 -#error "cannot detect the format of function strerror_r()"
848 -#endif
2025 + // do not add empty / unset fields
2026 + if((type == NDFT_TXT && (!e->txt || !*e->txt)) ||
2027 + (type == NDFT_BFR && (!e->bfr || !buffer_strlen(e->bfr))) ||
2028 + (type == NDFT_STR && !e->str) ||
2029 + (type == NDFT_UUID && !e->uuid) ||
2030 + (type == NDFT_CALLBACK && !e->cb.formatter) ||
2031 + type == NDFT_UNSET)
2032 + continue;
2033
850 -void error_limit_int(ERROR_LIMIT *erl, const char *prefix, const char *file __maybe_unused, const char *function __maybe_unused, const unsigned long line __maybe_unused, const char *fmt, ... ) {
851 - FILE *fp = stderror ? stderror : stderr;
2034 + thread_log_fields[lgs[i].id].entry = *e;
2035 + }
2036 + }
2037 +}
2038
853 - if(erl->sleep_ut)
854 - sleep_usec(erl->sleep_ut);
2039 +static void nd_logger(const char *file, const char *function, const unsigned long line,
2040 + ND_LOG_SOURCES source, ND_LOG_FIELD_PRIORITY priority, bool limit, int saved_errno,
2041 + const char *fmt, va_list ap) {
2042
856 - // save a copy of errno - just in case this function generates a new error
857 - int __errno = errno;
2043 + SPINLOCK *spinlock;
2044 + FILE *fp;
2045 + ND_LOG_OUTPUT output = nd_logger_select_output(source, &fp, &spinlock);
2046 + if(output != NDLO_FILE && output != NDLO_JOURNAL && output != NDLO_SYSLOG)
2047 + return;
2048
859 - va_list args;
2049 + // mark all fields as unset
2050 + nd_logger_unset_all_thread_fields();
2051
861 - log_lock();
2052 + // flatten the log stack into the fields
2053 + nd_logger_merge_log_stack_to_thread_fields();
2054
863 - erl->count++;
864 - time_t now = now_boottime_sec();
865 - if(now - erl->last_logged < erl->log_every) {
866 - log_unlock();
867 - return;
868 - }
2055 + // set the common fields that are automatically set by the logging subsystem
2056
870 - // prevent logging too much
871 - if (error_log_limit(0)) {
872 - log_unlock();
873 - return;
2057 + if(likely(!thread_log_fields[NDF_INVOCATION_ID].entry.set))
2058 + thread_log_fields[NDF_INVOCATION_ID].entry = ND_LOG_FIELD_UUID(NDF_INVOCATION_ID, &nd_log.invocation_id);
2059 +
2060 + if(likely(!thread_log_fields[NDF_LOG_SOURCE].entry.set))
2061 + thread_log_fields[NDF_LOG_SOURCE].entry = ND_LOG_FIELD_TXT(NDF_LOG_SOURCE, nd_log_source2str(source));
2062 +
2063 + if(likely(!thread_log_fields[NDF_SYSLOG_IDENTIFIER].entry.set))
2064 + thread_log_fields[NDF_SYSLOG_IDENTIFIER].entry = ND_LOG_FIELD_TXT(NDF_SYSLOG_IDENTIFIER, program_name);
2065 +
2066 + if(likely(!thread_log_fields[NDF_LINE].entry.set)) {
2067 + thread_log_fields[NDF_LINE].entry = ND_LOG_FIELD_U64(NDF_LINE, line);
2068 + thread_log_fields[NDF_FILE].entry = ND_LOG_FIELD_TXT(NDF_FILE, file);
2069 + thread_log_fields[NDF_FUNC].entry = ND_LOG_FIELD_TXT(NDF_FUNC, function);
2070 }
2071
876 - if(collector_log_syslog) {
877 - va_start( args, fmt );
878 - vsyslog(LOG_ERR, fmt, args );
879 - va_end( args );
2072 + if(likely(!thread_log_fields[NDF_PRIORITY].entry.set)) {
2073 + thread_log_fields[NDF_PRIORITY].entry = ND_LOG_FIELD_U64(NDF_PRIORITY, priority);
2074 }
2075
882 - char date[LOG_DATE_LENGTH];
883 - log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
2076 + if(likely(!thread_log_fields[NDF_TID].entry.set))
2077 + thread_log_fields[NDF_TID].entry = ND_LOG_FIELD_U64(NDF_TID, gettid());
2078 +
2079 + if(likely(!thread_log_fields[NDF_THREAD_TAG].entry.set)) {
2080 + char os_threadname[NETDATA_THREAD_NAME_MAX + 1];
2081 + const char *thread_tag = netdata_thread_tag();
2082 + if(!netdata_thread_tag_exists()) {
2083 + if (!netdata_thread_tag_exists()) {
2084 + os_thread_get_current_name_np(os_threadname);
2085 + if ('\0' != os_threadname[0])
2086 + /* If it is not an empty string replace "MAIN" thread_tag */
2087 + thread_tag = os_threadname;
2088 + }
2089 + }
2090 + thread_log_fields[NDF_THREAD_TAG].entry = ND_LOG_FIELD_TXT(NDF_THREAD_TAG, thread_tag);
2091
885 - va_start( args, fmt );
886 -#ifdef NETDATA_INTERNAL_CHECKS
887 - fprintf(fp, "%s: %s %-5.5s : %s : (%04lu@%-20.20s:%-15.15s): ",
888 - date, program_name, prefix, netdata_thread_tag(), line, file, function);
889 -#else
890 - fprintf(fp, "%s: %s %-5.5s : %s : ", date, program_name, prefix, netdata_thread_tag());
891 -#endif
892 - vfprintf(fp, fmt, args );
893 - va_end( args );
2092 + // TODO: fix the ND_MODULE in logging by setting proper module name in threads
2093 +// if(!thread_log_fields[NDF_MODULE].entry.set)
2094 +// thread_log_fields[NDF_MODULE].entry = ND_LOG_FIELD_CB(NDF_MODULE, thread_tag_to_module, (void *)thread_tag);
2095 + }
2096
895 - if(erl->count > 1)
896 - fprintf(fp, " (similar messages repeated %zu times in the last %llu secs)",
897 - erl->count, (unsigned long long)(erl->last_logged ? now - erl->last_logged : 0));
2097 + if(likely(!thread_log_fields[NDF_TIMESTAMP_REALTIME_USEC].entry.set))
2098 + thread_log_fields[NDF_TIMESTAMP_REALTIME_USEC].entry = ND_LOG_FIELD_U64(NDF_TIMESTAMP_REALTIME_USEC, now_realtime_usec());
2099
899 - if(erl->sleep_ut)
900 - fprintf(fp, " (sleeping for %"PRIu64" microseconds every time this happens)", erl->sleep_ut);
2100 + if(saved_errno != 0 && !thread_log_fields[NDF_ERRNO].entry.set)
2101 + thread_log_fields[NDF_ERRNO].entry = ND_LOG_FIELD_I64(NDF_ERRNO, saved_errno);
2102
902 - if(__errno) {
903 - char buf[1024];
904 - fprintf(fp,
905 - " (errno %d, %s)\n", __errno, strerror_result(strerror_r(__errno, buf, 1023), buf));
906 - errno = 0;
2103 + CLEAN_BUFFER *wb = NULL;
2104 + if(fmt && !thread_log_fields[NDF_MESSAGE].entry.set) {
2105 + wb = buffer_create(1024, NULL);
2106 + buffer_vsprintf(wb, fmt, ap);
2107 + thread_log_fields[NDF_MESSAGE].entry = ND_LOG_FIELD_TXT(NDF_MESSAGE, buffer_tostring(wb));
2108 }
908 - else
909 - fputc('\n', fp);
910 -
911 - erl->last_logged = now;
912 - erl->count = 0;
2109
914 - log_unlock();
915 -}
2110 + nd_logger_log_fields(spinlock, fp, limit, priority, output, &nd_log.sources[source],
2111 + thread_log_fields, THREAD_FIELDS_MAX);
2112
917 -void error_int(int is_collector, const char *prefix, const char *file __maybe_unused, const char *function __maybe_unused, const unsigned long line __maybe_unused, const char *fmt, ... ) {
918 -#if !defined(NETDATA_INTERNAL_CHECKS) && !defined(NETDATA_DEV_MODE)
919 - if (NETDATA_LOG_LEVEL_ERROR > global_log_severity_level)
920 - return;
921 -#endif
2113 + if(nd_log.sources[source].pending_msg) {
2114 + // log a pending message
2115
923 - // save a copy of errno - just in case this function generates a new error
924 - int __errno = errno;
925 - FILE *fp = (is_collector || !stderror) ? stderr : stderror;
2116 + nd_logger_unset_all_thread_fields();
2117
927 - va_list args;
2118 + thread_log_fields[NDF_TIMESTAMP_REALTIME_USEC].entry = (struct log_stack_entry){
2119 + .set = true,
2120 + .type = NDFT_U64,
2121 + .u64 = now_realtime_usec(),
2122 + };
2123
929 - log_lock();
2124 + thread_log_fields[NDF_LOG_SOURCE].entry = (struct log_stack_entry){
2125 + .set = true,
2126 + .type = NDFT_TXT,
2127 + .txt = nd_log_source2str(source),
2128 + };
2129
931 - // prevent logging too much
932 - if (error_log_limit(0)) {
933 - log_unlock();
934 - return;
935 - }
2130 + thread_log_fields[NDF_SYSLOG_IDENTIFIER].entry = (struct log_stack_entry){
2131 + .set = true,
2132 + .type = NDFT_TXT,
2133 + .txt = program_name,
2134 + };
2135
937 - if(collector_log_syslog) {
938 - va_start( args, fmt );
939 - vsyslog(LOG_ERR, fmt, args );
940 - va_end( args );
941 - }
2136 + thread_log_fields[NDF_MESSAGE].entry = (struct log_stack_entry){
2137 + .set = true,
2138 + .type = NDFT_TXT,
2139 + .txt = nd_log.sources[source].pending_msg,
2140 + };
2141
943 - char date[LOG_DATE_LENGTH];
944 - log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
2142 + nd_logger_log_fields(spinlock, fp, false, priority, output,
2143 + &nd_log.sources[source],
2144 + thread_log_fields, THREAD_FIELDS_MAX);
2145
946 - va_start( args, fmt );
947 -#ifdef NETDATA_INTERNAL_CHECKS
948 - fprintf(fp, "%s: %s %-5.5s : %s : (%04lu@%-20.20s:%-15.15s): ",
949 - date, program_name, prefix, netdata_thread_tag(), line, file, function);
950 -#else
951 - fprintf(fp, "%s: %s %-5.5s : %s : ", date, program_name, prefix, netdata_thread_tag());
952 -#endif
953 - vfprintf(fp, fmt, args );
954 - va_end( args );
955 -
956 - if(__errno) {
957 - char buf[1024];
958 - fprintf(fp,
959 - " (errno %d, %s)\n", __errno, strerror_result(strerror_r(__errno, buf, 1023), buf));
960 - errno = 0;
2146 + freez((void *)nd_log.sources[source].pending_msg);
2147 + nd_log.sources[source].pending_msg = NULL;
2148 }
962 - else
963 - fputc('\n', fp);
2149
965 - log_unlock();
2150 + errno = 0;
2151 }
2152
968 -#ifdef NETDATA_INTERNAL_CHECKS
969 -static void crash_netdata(void) {
970 - // make Netdata core dump
971 - abort();
972 -}
973 -#endif
2153 +static ND_LOG_SOURCES nd_log_validate_source(ND_LOG_SOURCES source) {
2154 + if(source >= _NDLS_MAX)
2155 + source = NDLS_DAEMON;
2156
975 -#ifdef HAVE_BACKTRACE
976 -#define BT_BUF_SIZE 100
977 -static void print_call_stack(void) {
978 - FILE *fp = (!stderror) ? stderr : stderror;
2157 + if(overwrite_thread_source)
2158 + source = overwrite_thread_source;
2159
980 - int nptrs;
981 - void *buffer[BT_BUF_SIZE];
2160 + if(nd_log.overwrite_process_source)
2161 + source = nd_log.overwrite_process_source;
2162
983 - nptrs = backtrace(buffer, BT_BUF_SIZE);
984 - if(nptrs)
985 - backtrace_symbols_fd(buffer, nptrs, fileno(fp));
2163 + return source;
2164 }
987 -#endif
2165
989 -void fatal_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) {
990 - FILE *fp = stderror ? stderror : stderr;
2166 +// ----------------------------------------------------------------------------
2167 +// public API for loggers
2168 +
2169 +void netdata_logger(ND_LOG_SOURCES source, ND_LOG_FIELD_PRIORITY priority, const char *file, const char *function, unsigned long line, const char *fmt, ... ) {
2170 + int saved_errno = errno;
2171 + source = nd_log_validate_source(source);
2172 +
2173 + if((source == NDLS_DAEMON || source == NDLS_COLLECTORS) && priority > nd_log.sources[source].min_priority)
2174 + return;
2175
992 - // save a copy of errno - just in case this function generates a new error
993 - int __errno = errno;
2176 va_list args;
995 - const char *thread_tag;
996 - char os_threadname[NETDATA_THREAD_NAME_MAX + 1];
2177 + va_start(args, fmt);
2178 + nd_logger(file, function, line, source, priority,
2179 + source == NDLS_DAEMON || source == NDLS_COLLECTORS,
2180 + saved_errno, fmt, args);
2181 + va_end(args);
2182 +}
2183
998 - if(collector_log_syslog) {
999 - va_start( args, fmt );
1000 - vsyslog(LOG_CRIT, fmt, args );
1001 - va_end( args );
1002 - }
2184 +void netdata_logger_with_limit(ERROR_LIMIT *erl, ND_LOG_SOURCES source, ND_LOG_FIELD_PRIORITY priority, const char *file __maybe_unused, const char *function __maybe_unused, const unsigned long line __maybe_unused, const char *fmt, ... ) {
2185 + int saved_errno = errno;
2186 + source = nd_log_validate_source(source);
2187
1004 - thread_tag = netdata_thread_tag();
1005 - if (!netdata_thread_tag_exists()) {
1006 - os_thread_get_current_name_np(os_threadname);
1007 - if ('\0' != os_threadname[0]) { /* If it is not an empty string replace "MAIN" thread_tag */
1008 - thread_tag = os_threadname;
1009 - }
2188 + if(erl->sleep_ut)
2189 + sleep_usec(erl->sleep_ut);
2190 +
2191 + spinlock_lock(&erl->spinlock);
2192 +
2193 + erl->count++;
2194 + time_t now = now_boottime_sec();
2195 + if(now - erl->last_logged < erl->log_every) {
2196 + spinlock_unlock(&erl->spinlock);
2197 + return;
2198 }
2199
1012 - char date[LOG_DATE_LENGTH];
1013 - log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
2200 + spinlock_unlock(&erl->spinlock);
2201
1015 - log_lock();
2202 + va_list args;
2203 + va_start(args, fmt);
2204 + nd_logger(file, function, line, source, priority,
2205 + source == NDLS_DAEMON || source == NDLS_COLLECTORS,
2206 + saved_errno, fmt, args);
2207 + va_end(args);
2208 +}
2209
1017 - va_start( args, fmt );
1018 -#ifdef NETDATA_INTERNAL_CHECKS
1019 - fprintf(fp,
1020 - "%s: %s FATAL : %s : (%04lu@%-20.20s:%-15.15s): ", date, program_name, thread_tag, line, file, function);
1021 -#else
1022 - fprintf(fp, "%s: %s FATAL : %s : ", date, program_name, thread_tag);
1023 -#endif
1024 - vfprintf(fp, fmt, args );
1025 - va_end( args );
2210 +void netdata_logger_fatal( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) {
2211 + int saved_errno = errno;
2212 + ND_LOG_SOURCES source = NDLS_DAEMON;
2213 + source = nd_log_validate_source(source);
2214
1027 - perror(" # ");
1028 - fputc('\n', fp);
2215 + va_list args;
2216 + va_start(args, fmt);
2217 + nd_logger(file, function, line, source, NDLP_ALERT, true, saved_errno, fmt, args);
2218 + va_end(args);
2219
1030 - log_unlock();
2220 + char date[LOG_DATE_LENGTH];
2221 + log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
2222
2223 char action_data[70+1];
1033 - snprintfz(action_data, 70, "%04lu@%-10.10s:%-15.15s/%d", line, file, function, __errno);
2224 + snprintfz(action_data, 70, "%04lu@%-10.10s:%-15.15s/%d", line, file, function, saved_errno);
2225 char action_result[60+1];
2226
2227 + const char *thread_tag = thread_log_fields[NDF_THREAD_TAG].entry.txt;
2228 + if(!thread_tag)
2229 + thread_tag = "UNKNOWN";
2230 +
2231 const char *tag_to_send = thread_tag;
2232
2233 // anonymize thread names
@@ -1045,129 +2240,120 @@ void fatal_int( const char *file, const char *function, const unsigned long line
2240 send_statistics("FATAL", action_result, action_data);
2241
2242 #ifdef HAVE_BACKTRACE
1048 - print_call_stack();
2243 + int fd = nd_log.sources[NDLS_DAEMON].fd;
2244 + if(fd == -1)
2245 + fd = STDERR_FILENO;
2246 +
2247 + int nptrs;
2248 + void *buffer[10000];
2249 +
2250 + nptrs = backtrace(buffer, sizeof(buffer));
2251 + if(nptrs)
2252 + backtrace_symbols_fd(buffer, nptrs, fd);
2253 #endif
2254
2255 #ifdef NETDATA_INTERNAL_CHECKS
1052 - crash_netdata();
2256 + abort();
2257 #endif
2258
2259 netdata_cleanup_and_exit(1);
2260 }
2261
2262 // ----------------------------------------------------------------------------
1059 -// access log
2263 +// log limits
2264
1061 -void netdata_log_access( const char *fmt, ... ) {
1062 - va_list args;
1063 -
1064 - if(access_log_syslog) {
1065 - va_start( args, fmt );
1066 - vsyslog(LOG_INFO, fmt, args );
1067 - va_end( args );
1068 - }
1069 -
1070 - if(stdaccess) {
1071 - static netdata_mutex_t access_mutex = NETDATA_MUTEX_INITIALIZER;
2265 +void nd_log_limits_reset(void) {
2266 + usec_t now_ut = now_monotonic_usec();
2267
1073 - if(web_server_is_multithreaded)
1074 - netdata_mutex_lock(&access_mutex);
2268 + spinlock_lock(&nd_log.std_output.spinlock);
2269 + spinlock_lock(&nd_log.std_error.spinlock);
2270
1076 - char date[LOG_DATE_LENGTH];
1077 - log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
1078 - fprintf(stdaccess, "%s: ", date);
1079 -
1080 - va_start( args, fmt );
1081 - vfprintf( stdaccess, fmt, args );
1082 - va_end( args );
1083 - fputc('\n', stdaccess);
1084 -
1085 - if(web_server_is_multithreaded)
1086 - netdata_mutex_unlock(&access_mutex);
2271 + for(size_t i = 0; i < _NDLS_MAX ;i++) {
2272 + spinlock_lock(&nd_log.sources[i].spinlock);
2273 + nd_log.sources[i].limits.prevented = 0;
2274 + nd_log.sources[i].limits.counter = 0;
2275 + nd_log.sources[i].limits.started_monotonic_ut = now_ut;
2276 + nd_log.sources[i].limits.logs_per_period = nd_log.sources[i].limits.logs_per_period_backup;
2277 + spinlock_unlock(&nd_log.sources[i].spinlock);
2278 }
1088 -}
1089 -
1090 -// ----------------------------------------------------------------------------
1091 -// health log
2279
1093 -void netdata_log_health( const char *fmt, ... ) {
1094 - va_list args;
2280 + spinlock_unlock(&nd_log.std_output.spinlock);
2281 + spinlock_unlock(&nd_log.std_error.spinlock);
2282 +}
2283
1096 - if(health_log_syslog) {
1097 - va_start( args, fmt );
1098 - vsyslog(LOG_INFO, fmt, args );
1099 - va_end( args );
2284 +void nd_log_limits_unlimited(void) {
2285 + nd_log_limits_reset();
2286 + for(size_t i = 0; i < _NDLS_MAX ;i++) {
2287 + nd_log.sources[i].limits.logs_per_period = 0;
2288 }
2289 +}
2290
1102 - if(stdhealth) {
1103 - static netdata_mutex_t health_mutex = NETDATA_MUTEX_INITIALIZER;
1104 -
1105 - if(web_server_is_multithreaded)
1106 - netdata_mutex_lock(&health_mutex);
2291 +static bool nd_log_limit_reached(struct nd_log_source *source) {
2292 + if(source->limits.throttle_period == 0 || source->limits.logs_per_period == 0)
2293 + return false;
2294
1108 - char date[LOG_DATE_LENGTH];
1109 - log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
1110 - fprintf(stdhealth, "%s: ", date);
2295 + usec_t now_ut = now_monotonic_usec();
2296 + if(!source->limits.started_monotonic_ut)
2297 + source->limits.started_monotonic_ut = now_ut;
2298
1112 - va_start( args, fmt );
1113 - vfprintf( stdhealth, fmt, args );
1114 - va_end( args );
1115 - fputc('\n', stdhealth);
2299 + source->limits.counter++;
2300
1117 - if(web_server_is_multithreaded)
1118 - netdata_mutex_unlock(&health_mutex);
1119 - }
1120 -}
2301 + if(now_ut - source->limits.started_monotonic_ut > (usec_t)source->limits.throttle_period) {
2302 + if(source->limits.prevented) {
2303 + BUFFER *wb = buffer_create(1024, NULL);
2304 + buffer_sprintf(wb,
2305 + "LOG FLOOD PROTECTION: resuming logging "
2306 + "(prevented %"PRIu32" logs in the last %"PRIu32" seconds).",
2307 + source->limits.prevented,
2308 + source->limits.throttle_period);
2309
1122 -#ifdef ENABLE_ACLK
1123 -void log_aclk_message_bin( const char *data, const size_t data_len, int tx, const char *mqtt_topic, const char *message_name) {
1124 - if (aclklog) {
1125 - static netdata_mutex_t aclklog_mutex = NETDATA_MUTEX_INITIALIZER;
1126 - netdata_mutex_lock(&aclklog_mutex);
2310 + if(source->pending_msg)
2311 + freez((void *)source->pending_msg);
2312
1128 - char date[LOG_DATE_LENGTH];
1129 - log_date(date, LOG_DATE_LENGTH, now_realtime_sec());
1130 - fprintf(aclklog, "%s: %s Msg:\"%s\", MQTT-topic:\"%s\": ", date, tx ? "OUTGOING" : "INCOMING", message_name, mqtt_topic);
2313 + source->pending_msg = strdupz(buffer_tostring(wb));
2314
1132 - fwrite(data, data_len, 1, aclklog);
2315 + buffer_free(wb);
2316 + }
2317
1134 - fputc('\n', aclklog);
2318 + // restart the period accounting
2319 + source->limits.started_monotonic_ut = now_ut;
2320 + source->limits.counter = 1;
2321 + source->limits.prevented = 0;
2322
1136 - netdata_mutex_unlock(&aclklog_mutex);
2323 + // log this error
2324 + return false;
2325 }
1138 -}
1139 -#endif
1140 -
1141 -void log_set_global_severity_level(netdata_log_level_t value)
1142 -{
1143 - global_log_severity_level = value;
1144 -}
2326
1146 -netdata_log_level_t log_severity_string_to_severity_level(char *level)
1147 -{
1148 - if (!strcmp(level, NETDATA_LOG_LEVEL_INFO_STR))
1149 - return NETDATA_LOG_LEVEL_INFO;
1150 - if (!strcmp(level, NETDATA_LOG_LEVEL_ERROR_STR) || !strcmp(level, NETDATA_LOG_LEVEL_ERROR_SHORT_STR))
1151 - return NETDATA_LOG_LEVEL_ERROR;
2327 + if(source->limits.counter > source->limits.logs_per_period) {
2328 + if(!source->limits.prevented) {
2329 + BUFFER *wb = buffer_create(1024, NULL);
2330 + buffer_sprintf(wb,
2331 + "LOG FLOOD PROTECTION: too many logs (%"PRIu32" logs in %"PRId64" seconds, threshold is set to %"PRIu32" logs "
2332 + "in %"PRIu32" seconds). Preventing more logs from process '%s' for %"PRId64" seconds.",
2333 + source->limits.counter,
2334 + (int64_t)((now_ut - source->limits.started_monotonic_ut) / USEC_PER_SEC),
2335 + source->limits.logs_per_period,
2336 + source->limits.throttle_period,
2337 + program_name,
2338 + (int64_t)((source->limits.started_monotonic_ut + (source->limits.throttle_period * USEC_PER_SEC) - now_ut)) / USEC_PER_SEC);
2339 +
2340 + if(source->pending_msg)
2341 + freez((void *)source->pending_msg);
2342 +
2343 + source->pending_msg = strdupz(buffer_tostring(wb));
2344 +
2345 + buffer_free(wb);
2346 + }
2347
1153 - return NETDATA_LOG_LEVEL_INFO;
1154 -}
2348 + source->limits.prevented++;
2349
1156 -char *log_severity_level_to_severity_string(netdata_log_level_t level)
1157 -{
1158 - switch (level) {
1159 - case NETDATA_LOG_LEVEL_ERROR:
1160 - return NETDATA_LOG_LEVEL_ERROR_STR;
1161 - case NETDATA_LOG_LEVEL_INFO:
1162 - default:
1163 - return NETDATA_LOG_LEVEL_INFO_STR;
2350 + // prevent logging this error
2351 +#ifdef NETDATA_INTERNAL_CHECKS
2352 + return false;
2353 +#else
2354 + return true;
2355 +#endif
2356 }
1165 -}
2357
1167 -void log_set_global_severity_for_external_plugins() {
1168 - char *s = getenv("NETDATA_LOG_SEVERITY_LEVEL");
1169 - if (!s)
1170 - return;
1171 - netdata_log_level_t level = log_severity_string_to_severity_level(s);
1172 - log_set_global_severity_level(level);
2358 + return false;
2359 }
libnetdata/log/log.h
+218 -82
@@ -9,6 +9,181 @@ extern "C" {
9
10 #include "../libnetdata.h"
11
12 +#define ND_LOG_DEFAULT_THROTTLE_LOGS 1200
13 +#define ND_LOG_DEFAULT_THROTTLE_PERIOD 3600
14 +
15 +typedef enum __attribute__((__packed__)) {
16 + NDLS_UNSET = 0, // internal use only
17 + NDLS_ACCESS, // access.log
18 + NDLS_ACLK, // aclk.log
19 + NDLS_COLLECTORS, // collectors.log
20 + NDLS_DAEMON, // error.log
21 + NDLS_HEALTH, // health.log
22 + NDLS_DEBUG, // debug.log
23 +
24 + // terminator
25 + _NDLS_MAX,
26 +} ND_LOG_SOURCES;
27 +
28 +typedef enum __attribute__((__packed__)) {
29 + NDLP_EMERG = LOG_EMERG,
30 + NDLP_ALERT = LOG_ALERT,
31 + NDLP_CRIT = LOG_CRIT,
32 + NDLP_ERR = LOG_ERR,
33 + NDLP_WARNING = LOG_WARNING,
34 + NDLP_NOTICE = LOG_NOTICE,
35 + NDLP_INFO = LOG_INFO,
36 + NDLP_DEBUG = LOG_DEBUG,
37 +} ND_LOG_FIELD_PRIORITY;
38 +
39 +typedef enum __attribute__((__packed__)) {
40 + // KEEP THESE IN THE SAME ORDER AS in thread_log_fields (log.c)
41 + // so that it easy to audit for missing fields
42 +
43 + NDF_STOP = 0,
44 + NDF_TIMESTAMP_REALTIME_USEC, // the timestamp of the log message - added automatically
45 + NDF_SYSLOG_IDENTIFIER, // the syslog identifier of the application - added automatically
46 + NDF_LOG_SOURCE, // DAEMON, COLLECTORS, HEALTH, ACCESS, ACLK - set at the log call
47 + NDF_PRIORITY, // the syslog priority (severity) - set at the log call
48 + NDF_ERRNO, // the ERRNO at the time of the log call - added automatically
49 + NDF_INVOCATION_ID, // the INVOCATION_ID of Netdata - added automatically
50 + NDF_LINE, // the source code file line number - added automatically
51 + NDF_FILE, // the source code filename - added automatically
52 + NDF_FUNC, // the source code function - added automatically
53 + NDF_TID, // the thread ID of the thread logging - added automatically
54 + NDF_THREAD_TAG, // the thread tag of the thread logging - added automatically
55 + NDF_MESSAGE_ID, // for specific events
56 + NDF_MODULE, // for internal plugin module, all other get the NDF_THREAD_TAG
57 +
58 + NDF_NIDL_NODE, // the node / rrdhost currently being worked
59 + NDF_NIDL_INSTANCE, // the instance / rrdset currently being worked
60 + NDF_NIDL_CONTEXT, // the context of the instance currently being worked
61 + NDF_NIDL_DIMENSION, // the dimension / rrddim currently being worked
62 +
63 + // web server, aclk and stream receiver
64 + NDF_SRC_TRANSPORT, // the transport we received the request, one of: http, https, pluginsd
65 +
66 + // web server and stream receiver
67 + NDF_SRC_IP, // the streaming / web server source IP
68 + NDF_SRC_PORT, // the streaming / web server source Port
69 + NDF_SRC_CAPABILITIES, // the stream receiver capabilities
70 +
71 + // stream sender (established links)
72 + NDF_DST_TRANSPORT, // the transport we send the request, one of: http, https
73 + NDF_DST_IP, // the destination streaming IP
74 + NDF_DST_PORT, // the destination streaming Port
75 + NDF_DST_CAPABILITIES, // the destination streaming capabilities
76 +
77 + // web server, aclk and stream receiver
78 + NDF_REQUEST_METHOD, // for http like requests, the http request method
79 + NDF_RESPONSE_CODE, // for http like requests, the http response code, otherwise a status string
80 +
81 + // web server (all), aclk (queries)
82 + NDF_CONNECTION_ID, // the web server connection ID
83 + NDF_TRANSACTION_ID, // the web server and API transaction ID
84 + NDF_RESPONSE_SENT_BYTES, // for http like requests, the response bytes
85 + NDF_RESPONSE_SIZE_BYTES, // for http like requests, the uncompressed response size
86 + NDF_RESPONSE_PREPARATION_TIME_USEC, // for http like requests, the preparation time
87 + NDF_RESPONSE_SENT_TIME_USEC, // for http like requests, the time to send the response back
88 + NDF_RESPONSE_TOTAL_TIME_USEC, // for http like requests, the total time to complete the response
89 +
90 + // health alerts
91 + NDF_ALERT_ID,
92 + NDF_ALERT_UNIQUE_ID,
93 + NDF_ALERT_EVENT_ID,
94 + NDF_ALERT_TRANSITION_ID,
95 + NDF_ALERT_CONFIG_HASH,
96 + NDF_ALERT_NAME,
97 + NDF_ALERT_CLASS,
98 + NDF_ALERT_COMPONENT,
99 + NDF_ALERT_TYPE,
100 + NDF_ALERT_EXEC,
101 + NDF_ALERT_RECIPIENT,
102 + NDF_ALERT_DURATION,
103 + NDF_ALERT_VALUE,
104 + NDF_ALERT_VALUE_OLD,
105 + NDF_ALERT_STATUS,
106 + NDF_ALERT_STATUS_OLD,
107 + NDF_ALERT_SOURCE,
108 + NDF_ALERT_UNITS,
109 + NDF_ALERT_SUMMARY,
110 + NDF_ALERT_INFO,
111 + NDF_ALERT_NOTIFICATION_REALTIME_USEC,
112 + // NDF_ALERT_FLAGS,
113 +
114 + // put new items here
115 + // leave the request URL and the message last
116 +
117 + NDF_REQUEST, // the request we are currently working on
118 + NDF_MESSAGE, // the log message, if any
119 +
120 + // terminator
121 + _NDF_MAX,
122 +} ND_LOG_FIELD_ID;
123 +
124 +typedef enum __attribute__((__packed__)) {
125 + NDFT_UNSET = 0,
126 + NDFT_TXT,
127 + NDFT_STR,
128 + NDFT_BFR,
129 + NDFT_U64,
130 + NDFT_I64,
131 + NDFT_DBL,
132 + NDFT_UUID,
133 + NDFT_CALLBACK,
134 +} ND_LOG_STACK_FIELD_TYPE;
135 +
136 +void nd_log_set_user_settings(ND_LOG_SOURCES source, const char *setting);
137 +void nd_log_set_facility(const char *facility);
138 +void nd_log_set_priority_level(const char *setting);
139 +void nd_log_initialize(void);
140 +void nd_log_reopen_log_files(void);
141 +void chown_open_file(int fd, uid_t uid, gid_t gid);
142 +void nd_log_chown_log_files(uid_t uid, gid_t gid);
143 +void nd_log_set_flood_protection(size_t logs, time_t period);
144 +void nd_log_initialize_for_external_plugins(const char *name);
145 +void nd_log_set_thread_source(ND_LOG_SOURCES source);
146 +bool nd_log_journal_socket_available(void);
147 +ND_LOG_FIELD_ID nd_log_field_id_by_name(const char *field, size_t len);
148 +int nd_log_priority2id(const char *priority);
149 +
150 +typedef bool (*log_formatter_callback_t)(BUFFER *wb, void *data);
151 +
152 +struct log_stack_entry {
153 + ND_LOG_FIELD_ID id;
154 + ND_LOG_STACK_FIELD_TYPE type;
155 + bool set;
156 + union {
157 + const char *txt;
158 + struct netdata_string *str;
159 + BUFFER *bfr;
160 + uint64_t u64;
161 + int64_t i64;
162 + double dbl;
163 + const uuid_t *uuid;
164 + struct {
165 + log_formatter_callback_t formatter;
166 + void *formatter_data;
167 + } cb;
168 + };
169 +};
170 +
171 +#define ND_LOG_STACK _cleanup_(log_stack_pop) struct log_stack_entry
172 +#define ND_LOG_STACK_PUSH(lgs) log_stack_push(lgs)
173 +
174 +#define ND_LOG_FIELD_TXT(field, value) (struct log_stack_entry){ .id = (field), .type = NDFT_TXT, .txt = (value), .set = true, }
175 +#define ND_LOG_FIELD_STR(field, value) (struct log_stack_entry){ .id = (field), .type = NDFT_STR, .str = (value), .set = true, }
176 +#define ND_LOG_FIELD_BFR(field, value) (struct log_stack_entry){ .id = (field), .type = NDFT_BFR, .bfr = (value), .set = true, }
177 +#define ND_LOG_FIELD_U64(field, value) (struct log_stack_entry){ .id = (field), .type = NDFT_U64, .u64 = (value), .set = true, }
178 +#define ND_LOG_FIELD_I64(field, value) (struct log_stack_entry){ .id = (field), .type = NDFT_I64, .i64 = (value), .set = true, }
179 +#define ND_LOG_FIELD_DBL(field, value) (struct log_stack_entry){ .id = (field), .type = NDFT_DBL, .dbl = (value), .set = true, }
180 +#define ND_LOG_FIELD_CB(field, func, data) (struct log_stack_entry){ .id = (field), .type = NDFT_CALLBACK, .cb = { .formatter = (func), .formatter_data = (data) }, .set = true, }
181 +#define ND_LOG_FIELD_UUID(field, value) (struct log_stack_entry){ .id = (field), .type = NDFT_UUID, .uuid = (value), .set = true, }
182 +#define ND_LOG_FIELD_END() (struct log_stack_entry){ .id = NDF_STOP, .type = NDFT_UNSET, .set = false, }
183 +
184 +void log_stack_pop(void *ptr);
185 +void log_stack_push(struct log_stack_entry *lgs);
186 +
187 #define D_WEB_BUFFER 0x0000000000000001
188 #define D_WEB_CLIENT 0x0000000000000002
189 #define D_LISTENER 0x0000000000000004
@@ -46,114 +221,75 @@ extern "C" {
221 #define D_REPLICATION 0x0000002000000000
222 #define D_SYSTEM 0x8000000000000000
223
49 -extern int web_server_is_multithreaded;
50 -
224 extern uint64_t debug_flags;
225
226 extern const char *program_name;
227
55 -extern int stdaccess_fd;
56 -extern FILE *stdaccess;
228 +#ifdef ENABLE_ACLK
229 +extern int aclklog_enabled;
230 +#endif
231
58 -extern int stdhealth_fd;
59 -extern FILE *stdhealth;
232 +#define LOG_DATE_LENGTH 26
233 +void log_date(char *buffer, size_t len, time_t now);
234
61 -extern int stdcollector_fd;
62 -extern FILE *stderror;
235 +static inline void debug_dummy(void) {}
236
64 -extern const char *stdaccess_filename;
65 -extern const char *stderr_filename;
66 -extern const char *stdout_filename;
67 -extern const char *stdhealth_filename;
68 -extern const char *stdcollector_filename;
69 -extern const char *facility_log;
237 +void nd_log_limits_reset(void);
238 +void nd_log_limits_unlimited(void);
239
71 -#ifdef ENABLE_ACLK
72 -extern const char *aclklog_filename;
73 -extern int aclklog_fd;
74 -extern FILE *aclklog;
75 -extern int aclklog_enabled;
240 +#define NDLP_INFO_STR "info"
241 +
242 +#ifdef NETDATA_INTERNAL_CHECKS
243 +#define netdata_log_debug(type, args...) do { if(unlikely(debug_flags & type)) netdata_logger(NDLS_DEBUG, NDLP_DEBUG, __FILE__, __FUNCTION__, __LINE__, ##args); } while(0)
244 +#define internal_error(condition, args...) do { if(unlikely(condition)) netdata_logger(NDLS_DAEMON, NDLP_DEBUG, __FILE__, __FUNCTION__, __LINE__, ##args); } while(0)
245 +#define internal_fatal(condition, args...) do { if(unlikely(condition)) netdata_logger_fatal(__FILE__, __FUNCTION__, __LINE__, ##args); } while(0)
246 +#else
247 +#define netdata_log_debug(type, args...) debug_dummy()
248 +#define internal_error(args...) debug_dummy()
249 +#define internal_fatal(args...) debug_dummy()
250 #endif
251
78 -extern int access_log_syslog;
79 -extern int error_log_syslog;
80 -extern int output_log_syslog;
81 -extern int health_log_syslog;
252 +#define fatal(args...) netdata_logger_fatal(__FILE__, __FUNCTION__, __LINE__, ##args)
253 +#define fatal_assert(expr) ((expr) ? (void)(0) : netdata_logger_fatal(__FILE__, __FUNCTION__, __LINE__, "Assertion `%s' failed", #expr))
254
83 -extern time_t error_log_throttle_period;
84 -extern unsigned long error_log_errors_per_period, error_log_errors_per_period_backup;
85 -int error_log_limit(int reset);
255 +// ----------------------------------------------------------------------------
256 +// normal logging
257
87 -void open_all_log_files();
88 -void reopen_all_log_files();
258 +void netdata_logger(ND_LOG_SOURCES source, ND_LOG_FIELD_PRIORITY priority, const char *file, const char *function, unsigned long line, const char *fmt, ... ) PRINTFLIKE(6, 7);
259 +#define nd_log(NDLS, NDLP, args...) netdata_logger(NDLS, NDLP, __FILE__, __FUNCTION__, __LINE__, ##args)
260 +#define nd_log_daemon(NDLP, args...) netdata_logger(NDLS_DAEMON, NDLP, __FILE__, __FUNCTION__, __LINE__, ##args)
261 +#define nd_log_collector(NDLP, args...) netdata_logger(NDLS_COLLECTORS, NDLP, __FILE__, __FUNCTION__, __LINE__, ##args)
262
90 -#define LOG_DATE_LENGTH 26
91 -void log_date(char *buffer, size_t len, time_t now);
263 +#define netdata_log_info(args...) netdata_logger(NDLS_DAEMON, NDLP_INFO, __FILE__, __FUNCTION__, __LINE__, ##args)
264 +#define netdata_log_error(args...) netdata_logger(NDLS_DAEMON, NDLP_ERR, __FILE__, __FUNCTION__, __LINE__, ##args)
265 +#define collector_info(args...) netdata_logger(NDLS_COLLECTORS, NDLP_INFO, __FILE__, __FUNCTION__, __LINE__, ##args)
266 +#define collector_error(args...) netdata_logger(NDLS_COLLECTORS, NDLP_ERR, __FILE__, __FUNCTION__, __LINE__, ##args)
267
93 -static inline void debug_dummy(void) {}
268 +#define log_aclk_message_bin(__data, __data_len, __tx, __mqtt_topic, __message_name) \
269 + nd_log(NDLS_ACLK, NDLP_INFO, \
270 + "direction:%s message:'%s' topic:'%s' json:'%.*s'", \
271 + (__tx) ? "OUTGOING" : "INCOMING", __message_name, __mqtt_topic, (int)(__data_len), __data)
272
95 -void error_log_limit_reset(void);
96 -void error_log_limit_unlimited(void);
273 +// ----------------------------------------------------------------------------
274 +// logging with limits
275
276 typedef struct error_with_limit {
277 + SPINLOCK spinlock;
278 time_t log_every;
279 size_t count;
280 time_t last_logged;
281 usec_t sleep_ut;
282 } ERROR_LIMIT;
283
105 -typedef enum netdata_log_level {
106 - NETDATA_LOG_LEVEL_ERROR,
107 - NETDATA_LOG_LEVEL_INFO,
108 -
109 - NETDATA_LOG_LEVEL_END
110 -} netdata_log_level_t;
284 +#define nd_log_limit_static_global_var(var, log_every_secs, sleep_usecs) static ERROR_LIMIT var = { .last_logged = 0, .count = 0, .log_every = (log_every_secs), .sleep_ut = (sleep_usecs) }
285 +#define nd_log_limit_static_thread_var(var, log_every_secs, sleep_usecs) static __thread ERROR_LIMIT var = { .last_logged = 0, .count = 0, .log_every = (log_every_secs), .sleep_ut = (sleep_usecs) }
286 +void netdata_logger_with_limit(ERROR_LIMIT *erl, ND_LOG_SOURCES source, ND_LOG_FIELD_PRIORITY priority, const char *file, const char *function, unsigned long line, const char *fmt, ... ) PRINTFLIKE(7, 8);;
287 +#define nd_log_limit(erl, NDLS, NDLP, args...) netdata_logger_with_limit(erl, NDLS, NDLP, __FILE__, __FUNCTION__, __LINE__, ##args)
288
112 -#define NETDATA_LOG_LEVEL_INFO_STR "info"
113 -#define NETDATA_LOG_LEVEL_ERROR_STR "error"
114 -#define NETDATA_LOG_LEVEL_ERROR_SHORT_STR "err"
115 -
116 -extern netdata_log_level_t global_log_severity_level;
117 -netdata_log_level_t log_severity_string_to_severity_level(char *level);
118 -char *log_severity_level_to_severity_string(netdata_log_level_t level);
119 -void log_set_global_severity_level(netdata_log_level_t value);
120 -void log_set_global_severity_for_external_plugins();
121 -
122 -#define error_limit_static_global_var(var, log_every_secs, sleep_usecs) static ERROR_LIMIT var = { .last_logged = 0, .count = 0, .log_every = (log_every_secs), .sleep_ut = (sleep_usecs) }
123 -#define error_limit_static_thread_var(var, log_every_secs, sleep_usecs) static __thread ERROR_LIMIT var = { .last_logged = 0, .count = 0, .log_every = (log_every_secs), .sleep_ut = (sleep_usecs) }
124 -
125 -#ifdef NETDATA_INTERNAL_CHECKS
126 -#define netdata_log_debug(type, args...) do { if(unlikely(debug_flags & type)) debug_int(__FILE__, __FUNCTION__, __LINE__, ##args); } while(0)
127 -#define internal_error(condition, args...) do { if(unlikely(condition)) error_int(0, "IERR", __FILE__, __FUNCTION__, __LINE__, ##args); } while(0)
128 -#define internal_fatal(condition, args...) do { if(unlikely(condition)) fatal_int(__FILE__, __FUNCTION__, __LINE__, ##args); } while(0)
129 -#else
130 -#define netdata_log_debug(type, args...) debug_dummy()
131 -#define internal_error(args...) debug_dummy()
132 -#define internal_fatal(args...) debug_dummy()
133 -#endif
134 -
135 -#define netdata_log_info(args...) info_int(0, __FILE__, __FUNCTION__, __LINE__, ##args)
136 -#define collector_info(args...) info_int(1, __FILE__, __FUNCTION__, __LINE__, ##args)
137 -#define infoerr(args...) error_int(0, "INFO", __FILE__, __FUNCTION__, __LINE__, ##args)
138 -#define netdata_log_error(args...) error_int(0, "ERROR", __FILE__, __FUNCTION__, __LINE__, ##args)
139 -#define collector_infoerr(args...) error_int(1, "INFO", __FILE__, __FUNCTION__, __LINE__, ##args)
140 -#define collector_error(args...) error_int(1, "ERROR", __FILE__, __FUNCTION__, __LINE__, ##args)
141 -#define error_limit(erl, args...) error_limit_int(erl, "ERROR", __FILE__, __FUNCTION__, __LINE__, ##args)
142 -#define fatal(args...) fatal_int(__FILE__, __FUNCTION__, __LINE__, ##args)
143 -#define fatal_assert(expr) ((expr) ? (void)(0) : fatal_int(__FILE__, __FUNCTION__, __LINE__, "Assertion `%s' failed", #expr))
289 +// ----------------------------------------------------------------------------
290
291 void send_statistics(const char *action, const char *action_result, const char *action_data);
146 -void debug_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) PRINTFLIKE(4, 5);
147 -void info_int( int is_collector, const char *file, const char *function, const unsigned long line, const char *fmt, ... ) PRINTFLIKE(5, 6);
148 -void error_int( int is_collector, const char *prefix, const char *file, const char *function, const unsigned long line, const char *fmt, ... ) PRINTFLIKE(6, 7);
149 -void error_limit_int(ERROR_LIMIT *erl, const char *prefix, const char *file __maybe_unused, const char *function __maybe_unused, unsigned long line __maybe_unused, const char *fmt, ... ) PRINTFLIKE(6, 7);;
150 -void fatal_int( const char *file, const char *function, const unsigned long line, const char *fmt, ... ) NORETURN PRINTFLIKE(4, 5);
151 -void netdata_log_access( const char *fmt, ... ) PRINTFLIKE(1, 2);
152 -void netdata_log_health( const char *fmt, ... ) PRINTFLIKE(1, 2);
153 -
154 -#ifdef ENABLE_ACLK
155 -void log_aclk_message_bin( const char *data, const size_t data_len, int tx, const char *mqtt_topic, const char *message_name);
156 -#endif
292 +void netdata_logger_fatal( const char *file, const char *function, unsigned long line, const char *fmt, ... ) NORETURN PRINTFLIKE(4, 5);
293
294 # ifdef __cplusplus
295 }
libnetdata/log/log2journal.c new
+1015
@@ -0,0 +1,1015 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +// only for PACKAGE_VERSION
4 +#include "config.h"
5 +
6 +#include <stdio.h>
7 +#include <stdlib.h>
8 +#include <string.h>
9 +#include <stdbool.h>
10 +#include <string.h>
11 +#include <ctype.h>
12 +#include <stdarg.h>
13 +
14 +#define XXH_INLINE_ALL
15 +#include "../xxhash.h"
16 +
17 +#define PCRE2_CODE_UNIT_WIDTH 8
18 +#include <pcre2.h>
19 +
20 +#define MAX_OUTPUT_KEYS 1024
21 +#define OVECCOUNT (MAX_OUTPUT_KEYS * 3) // should be a multiple of 3
22 +#define MAX_LINE_LENGTH (1024 * 1024)
23 +#define MAX_KEY_DUPS (MAX_OUTPUT_KEYS / 2)
24 +#define MAX_INJECTIONS (MAX_OUTPUT_KEYS / 2)
25 +#define MAX_REWRITES (MAX_OUTPUT_KEYS / 2)
26 +#define MAX_KEY_DUPS_KEYS 20
27 +
28 +#define MAX_KEY_LEN 64 // according to systemd-journald
29 +#define MAX_VALUE_LEN (48 * 1024) // according to systemd-journald
30 +
31 +void display_help(const char *name) {
32 + printf("\n");
33 + printf("Netdata log2journal " PACKAGE_VERSION "\n");
34 + printf("\n");
35 + printf("Convert structured log input to systemd Journal Export Format.\n");
36 + printf("\n");
37 + printf("Using PCRE2 patterns, extract the fields from structured logs on the standard\n");
38 + printf("input, and generate output according to systemd Journal Export Format.\n");
39 + printf("\n");
40 + printf("Usage: %s [OPTIONS] PATTERN\n", name);
41 + printf("\n");
42 + printf("Options:\n");
43 + printf("\n");
44 + printf(" --filename-key=KEY\n");
45 + printf(" Add a field with KEY as the key and the current filename as value.\n");
46 + printf(" Automatically detects filenames when piped after 'tail -F',\n");
47 + printf(" and tail matches multiple filenames.\n");
48 + printf(" To inject the filename when tailing a single file, use --inject.\n");
49 + printf("\n");
50 + printf(" --unmatched-key=KEY\n");
51 + printf(" Include unmatched log entries in the output with KEY as the field name.\n");
52 + printf(" Use this to include unmatched entries to the output stream.\n");
53 + printf(" Usually it should be set to --unmatched-key=MESSAGE so that the\n");
54 + printf(" unmatched entry will appear as the log message in the journals.\n");
55 + printf(" Use --inject-unmatched to inject additional fields to unmatched lines.\n");
56 + printf("\n");
57 + printf(" --duplicate=TARGET=KEY1[,KEY2[,KEY3[,...]]\n");
58 + printf(" Create a new key called TARGET, duplicating the values of the keys\n");
59 + printf(" given. Useful for further processing. When multiple keys are given,\n");
60 + printf(" their values are separated by comma.\n");
61 + printf(" Up to %d duplications can be given on the command line, and up to\n", MAX_KEY_DUPS);
62 + printf(" %d keys per duplication command are allowed.\n", MAX_KEY_DUPS_KEYS);
63 + printf("\n");
64 + printf(" --inject=LINE\n");
65 + printf(" Inject constant fields to the output (both matched and unmatched logs).\n");
66 + printf(" --inject entries are added to unmatched lines too, when their key is\n");
67 + printf(" not used in --inject-unmatched (--inject-unmatched override --inject).\n");
68 + printf(" Up to %d fields can be injected.\n", MAX_INJECTIONS);
69 + printf("\n");
70 + printf(" --inject-unmatched=LINE\n");
71 + printf(" Inject lines into the output for each unmatched log entry.\n");
72 + printf(" Usually, --inject-unmatched=PRIORITY=3 is needed to mark the unmatched\n");
73 + printf(" lines as errors, so that they can easily be spotted in the journals.\n");
74 + printf(" Up to %d such lines can be injected.\n", MAX_INJECTIONS);
75 + printf("\n");
76 + printf(" --rewrite=KEY=/SearchPattern/ReplacePattern\n");
77 + printf(" Apply a rewrite rule to the values of a specific key.\n");
78 + printf(" The first character after KEY= is the separator, which should also\n");
79 + printf(" be used between the search pattern and the replacement pattern.\n");
80 + printf(" The search pattern is a PCRE2 regular expression, and the replacement\n");
81 + printf(" pattern supports literals and named capture groups from the search pattern.\n");
82 + printf(" Example:\n");
83 + printf(" --rewrite=DATE=/^(?<year>\\d{4})-(?<month>\\d{2})-(?<day>\\d{2})$/\n");
84 + printf(" ${day}/${month}/${year}\n");
85 + printf(" This will rewrite dates in the format YYYY-MM-DD to DD/MM/YYYY.\n");
86 + printf("\n");
87 + printf(" Only one rewrite rule is applied per key; the sequence of rewrites stops\n");
88 + printf(" for the key once a rule matches it. This allows providing a sequence of\n");
89 + printf(" independent rewriting rules for the same key, matching the different values\n");
90 + printf(" the key may get, and also provide a catch-all rewrite rule at the end of the\n");
91 + printf(" sequence for setting the key value if no other rule matched it.\n");
92 + printf("\n");
93 + printf(" The combination of duplicating keys with the values of multiple other keys\n");
94 + printf(" combined with multiple rewrite rules, allows creating complex rules for\n");
95 + printf(" rewriting key values.\n");
96 + printf("\n");
97 + printf(" Up to %d rewriting rules are allowed.\n", MAX_REWRITES);
98 + printf("\n");
99 + printf(" -h, --help\n");
100 + printf(" Display this help and exit.\n");
101 + printf("\n");
102 + printf(" PATTERN\n");
103 + printf(" PATTERN should be a valid PCRE2 regular expression.\n");
104 + printf(" RE2 regular expressions (like the ones usually used in Go applications),\n");
105 + printf(" are usually valid PCRE2 patterns too.\n");
106 + printf(" Regular expressions without named groups are ignored.\n");
107 + printf("\n");
108 + printf("The maximum line length accepted is %d characters.\n", MAX_LINE_LENGTH);
109 + printf("The maximum number of fields in the PCRE2 pattern is %d.\n", OVECCOUNT / 3);
110 + printf("\n");
111 + printf("PIPELINE AND SEQUENCE OF PROCESSING\n");
112 + printf("\n");
113 + printf("This is a simple diagram of the pipeline taking place:\n");
114 + printf("\n");
115 + printf(" +---------------------------------------------------+\n");
116 + printf(" | INPUT |\n");
117 + printf(" +---------------------------------------------------+\n");
118 + printf(" v v\n");
119 + printf(" +---------------------------------+ |\n");
120 + printf(" | EXTRACT FIELDS AND VALUES | |\n");
121 + printf(" +---------------------------------+ |\n");
122 + printf(" v v |\n");
123 + printf(" +---------------+ | |\n");
124 + printf(" | DUPLICATE | | |\n");
125 + printf(" | create fields | | |\n");
126 + printf(" | with values | | |\n");
127 + printf(" +---------------+ | |\n");
128 + printf(" v v v\n");
129 + printf(" +---------------------------------+ +--------------+\n");
130 + printf(" | REWRITE PIPELINES | | INJECT |\n");
131 + printf(" | altering the values | | constants |\n");
132 + printf(" +---------------------------------+ +--------------+\n");
133 + printf(" v v\n");
134 + printf(" +---------------------------------------------------+\n");
135 + printf(" | OUTPUT |\n");
136 + printf(" +---------------------------------------------------+\n");
137 + printf("\n");
138 + printf("JOURNAL FIELDS RULES (enforced by systemd-journald)\n");
139 + printf("\n");
140 + printf(" - field names can be up to 64 characters\n");
141 + printf(" - the only allowed field characters are A-Z, 0-9 and underscore\n");
142 + printf(" - the first character of fields cannot be a digit\n");
143 + printf(" - protected journal fields start with underscore:\n");
144 + printf(" * they are accepted by systemd-journal-remote\n");
145 + printf(" * they are NOT accepted by a local systemd-journald\n");
146 + printf("\n");
147 + printf(" For best results, always include these fields:\n");
148 + printf("\n");
149 + printf(" MESSAGE=TEXT\n");
150 + printf(" The MESSAGE is the body of the log entry.\n");
151 + printf(" This field is what we usually see in our logs.\n");
152 + printf("\n");
153 + printf(" PRIORITY=NUMBER\n");
154 + printf(" PRIORITY sets the severity of the log entry.\n");
155 + printf(" 0=emerg, 1=alert, 2=crit, 3=err, 4=warn, 5=notice, 6=info, 7=debug\n");
156 + printf(" - Emergency events (0) are usually broadcast to all terminals.\n");
157 + printf(" - Emergency, alert, critical, and error (0-3) are usually colored red.\n");
158 + printf(" - Warning (4) entries are usually colored yellow.\n");
159 + printf(" - Notice (5) entries are usually bold or have a brighter white color.\n");
160 + printf(" - Info (6) entries are the default.\n");
161 + printf(" - Debug (7) entries are usually grayed or dimmed.\n");
162 + printf("\n");
163 + printf(" SYSLOG_IDENTIFIER=NAME\n");
164 + printf(" SYSLOG_IDENTIFIER sets the name of application.\n");
165 + printf(" Use something descriptive, like: SYSLOG_IDENTIFIER=nginx-logs\n");
166 + printf("\n");
167 + printf("You can find the most common fields at 'man systemd.journal-fields'.\n");
168 + printf("\n");
169 +}
170 +
171 +// ----------------------------------------------------------------------------
172 +
173 +// enable the compiler to check for printf like errors on our log2stderr() function
174 +static void log2stderr(const char *format, ...) __attribute__ ((format(__printf__, 1, 2)));
175 +static void log2stderr(const char *format, ...) {
176 + va_list args;
177 + va_start(args, format);
178 + vfprintf(stderr, format, args);
179 + va_end(args);
180 + fprintf(stderr, "\n");
181 +}
182 +
183 +// ----------------------------------------------------------------------------
184 +
185 +size_t copy_to_buffer(char *dst, size_t dst_size, const char *src, size_t src_len) {
186 + if(dst_size < 2) {
187 + if(dst_size == 1)
188 + *dst = '\0';
189 +
190 + return 0;
191 + }
192 +
193 + if(src_len <= dst_size - 1) {
194 + memcpy(dst, src, src_len);
195 + dst[src_len] = '\0';
196 + return src_len;
197 + }
198 + else {
199 + memcpy(dst, src, dst_size - 1);
200 + dst[dst_size - 1] = '\0';
201 + return dst_size - 1;
202 + }
203 +}
204 +
205 +// ----------------------------------------------------------------------------
206 +
207 +typedef struct txt {
208 + char *s;
209 + size_t size;
210 +} TEXT;
211 +
212 +static void txt_replace(TEXT *txt, const char *s, size_t len) {
213 + if(len + 1 <= txt->size) {
214 + // the existing value allocation, fits our value
215 +
216 + memcpy(txt->s, s, len);
217 + txt->s[len] = '\0';
218 + }
219 + else {
220 + // no existing value allocation, or too small for our value
221 +
222 + if(txt->s)
223 + free(txt->s);
224 +
225 + txt->s = strndup(s, len);
226 + txt->size = len + 1;
227 + }
228 +}
229 +
230 +// ----------------------------------------------------------------------------
231 +
232 +typedef struct key_value {
233 + char key[MAX_KEY_LEN + 1];
234 + TEXT value;
235 + bool on_unmatched;
236 +} KEY_VALUE;
237 +
238 +void key_value_replace(KEY_VALUE *kv, const char *key, size_t key_len, const char *value, size_t value_len) {
239 + copy_to_buffer(kv->key, sizeof(kv->key), key, key_len);
240 + txt_replace(&kv->value, value, value_len);
241 +}
242 +
243 +// ----------------------------------------------------------------------------
244 +
245 +struct key_dup {
246 + XXH64_hash_t hash;
247 + char *target;
248 + char *keys[MAX_KEY_DUPS_KEYS];
249 + TEXT values[MAX_KEY_DUPS_KEYS];
250 + size_t used;
251 + bool exposed;
252 +};
253 +
254 +struct replacement_node {
255 + bool is_variable;
256 + const char *s;
257 + size_t len;
258 + struct replacement_node *next;
259 +};
260 +
261 +struct key_rewrite {
262 + XXH64_hash_t hash;
263 + char *key;
264 + char *search_pattern;
265 + char *replace_pattern;
266 + pcre2_code *re;
267 + pcre2_match_data *match_data;
268 + struct replacement_node *nodes;
269 +};
270 +
271 +struct log_job {
272 + const char *pattern;
273 +
274 + struct {
275 + const char *key;
276 + char current[FILENAME_MAX + 1];
277 + bool last_line_was_empty;
278 + } filename;
279 +
280 + struct {
281 + KEY_VALUE keys[MAX_INJECTIONS];
282 + size_t used;
283 + } injections;
284 +
285 + struct {
286 + const char *key;
287 + struct {
288 + KEY_VALUE keys[MAX_INJECTIONS];
289 + size_t used;
290 + } injections;
291 + } unmatched;
292 +
293 + struct {
294 + struct key_dup array[MAX_KEY_DUPS];
295 + size_t used;
296 + } dups;
297 +
298 + struct {
299 + struct key_rewrite array[MAX_REWRITES];
300 + size_t used;
301 + } rewrites;
302 +};
303 +
304 +void jb_cleanup(struct log_job *jb) {
305 + for(size_t i = 0; i < jb->injections.used ;i++) {
306 + if(jb->injections.keys[i].value.s)
307 + free(jb->injections.keys[i].value.s);
308 + }
309 +
310 + for(size_t i = 0; i < jb->unmatched.injections.used ;i++) {
311 + if(jb->unmatched.injections.keys[i].value.s)
312 + free(jb->unmatched.injections.keys[i].value.s);
313 + }
314 +
315 + for(size_t i = 0; i < jb->dups.used ;i++) {
316 + struct key_dup *kd = &jb->dups.array[i];
317 +
318 + if(kd->target)
319 + free(kd->target);
320 +
321 + for(size_t j = 0; j < kd->used ; j++) {
322 + if (kd->keys[j])
323 + free(kd->keys[j]);
324 +
325 + if (kd->values[j].s)
326 + free(kd->values[j].s);
327 + }
328 + }
329 +
330 + for(size_t i = 0; i < jb->rewrites.used; i++) {
331 + struct key_rewrite *rw = &jb->rewrites.array[i];
332 +
333 + if (rw->key)
334 + free(rw->key);
335 +
336 + if (rw->search_pattern)
337 + free(rw->search_pattern);
338 +
339 + if (rw->replace_pattern)
340 + free(rw->replace_pattern);
341 +
342 + if(rw->match_data)
343 + pcre2_match_data_free(rw->match_data);
344 +
345 + if (rw->re)
346 + pcre2_code_free(rw->re);
347 +
348 + // Cleanup for replacement nodes linked list
349 + struct replacement_node *current = rw->nodes;
350 + while (current != NULL) {
351 + struct replacement_node *next = current->next;
352 +
353 + if (current->s)
354 + free((void *)current->s);
355 +
356 + free(current);
357 + current = next;
358 + }
359 + }
360 +
361 + memset(jb, 0, sizeof(*jb));
362 +}
363 +
364 +// ----------------------------------------------------------------------------
365 +// PCRE2
366 +
367 +static pcre2_code *jb_compile_pcre2_pattern(const char *pattern) {
368 + int error_number;
369 + PCRE2_SIZE error_offset;
370 + PCRE2_SPTR pattern_ptr = (PCRE2_SPTR)pattern;
371 +
372 + pcre2_code *re = pcre2_compile(pattern_ptr, PCRE2_ZERO_TERMINATED, 0, &error_number, &error_offset, NULL);
373 + if (re == NULL) {
374 + PCRE2_UCHAR errbuf[1024];
375 + pcre2_get_error_message(error_number, errbuf, sizeof(errbuf));
376 + log2stderr("PCRE2 compilation failed at offset %d: %s", (int)error_offset, errbuf);
377 + log2stderr("Check for common regex syntax errors or unsupported PCRE2 patterns.");
378 + return NULL;
379 + }
380 +
381 + return re;
382 +}
383 +
384 +static inline bool jb_pcre2_match(pcre2_code *re, pcre2_match_data *match_data, char *line, size_t len) {
385 + int rc = pcre2_match(re, (PCRE2_SPTR)line, len, 0, 0, match_data, NULL);
386 + if(rc < 0) {
387 + PCRE2_UCHAR errbuf[1024];
388 + pcre2_get_error_message(rc, errbuf, sizeof(errbuf));
389 + log2stderr("PCRE2 error %d: %s on: %s", rc, errbuf, line);
390 + return false;
391 + }
392 +
393 + return true;
394 +}
395 +
396 +// ----------------------------------------------------------------------------
397 +
398 +static char *rewrite_value(struct log_job *jb, const char *key, XXH64_hash_t hash, const char *value, size_t value_len) {
399 + static __thread char rewritten_value[MAX_VALUE_LEN + 1];
400 +
401 + for (size_t i = 0; i < jb->rewrites.used; i++) {
402 + struct key_rewrite *rw = &jb->rewrites.array[i];
403 +
404 + if (rw->hash == hash && strcmp(rw->key, key) == 0) {
405 + if (!jb_pcre2_match(rw->re, rw->match_data, (char *)value, value_len)) {
406 + continue; // No match found, skip to next rewrite rule
407 + }
408 +
409 + PCRE2_SIZE *ovector = pcre2_get_ovector_pointer(rw->match_data);
410 +
411 + char *buffer = rewritten_value;
412 + size_t buffer_remaining = sizeof(rewritten_value);
413 +
414 + // Iterate through the linked list of replacement nodes
415 + for (struct replacement_node *node = rw->nodes; node != NULL; node = node->next) {
416 + if (node->is_variable) {
417 + uint32_t groupnumber = pcre2_substring_number_from_name(rw->re, (PCRE2_SPTR)node->s);
418 + PCRE2_SIZE start_offset = ovector[2 * groupnumber];
419 + PCRE2_SIZE end_offset = ovector[2 * groupnumber + 1];
420 + PCRE2_SIZE length = end_offset - start_offset;
421 +
422 + size_t copied = copy_to_buffer(buffer, buffer_remaining, value + start_offset, length);
423 + buffer += copied;
424 + buffer_remaining -= copied;
425 + }
426 + else {
427 + size_t len = node->len;
428 + size_t copied = copy_to_buffer(buffer, buffer_remaining, node->s, len);
429 + buffer += copied;
430 + buffer_remaining -= copied;
431 + }
432 + }
433 +
434 + return rewritten_value;
435 + }
436 + }
437 +
438 + return NULL;
439 +}
440 +
441 +// ----------------------------------------------------------------------------
442 +
443 +static inline void send_key_value_error(const char *key, const char *format, ...) __attribute__ ((format(__printf__, 2, 3)));
444 +static inline void send_key_value_error(const char *key, const char *format, ...) {
445 + printf("%s=", key);
446 + va_list args;
447 + va_start(args, format);
448 + vprintf(format, args);
449 + va_end(args);
450 + printf("\n");
451 +}
452 +
453 +static inline void send_key_value_and_rewrite(struct log_job *jb, const char *key, XXH64_hash_t hash, const char *value, size_t len) {
454 + char *rewritten = rewrite_value(jb, key, hash, value, len);
455 + if(!rewritten)
456 + printf("%s=%.*s\n", key, (int)len, value);
457 + else
458 + printf("%s=%s\n", key, rewritten);
459 +}
460 +
461 +static inline void send_key_value_constant(struct log_job *jb, const char *key, const char *value) {
462 + printf("%s=%s\n", key, value);
463 +}
464 +
465 +// ----------------------------------------------------------------------------
466 +// command line params
467 +
468 +struct replacement_node *add_replacement_node(struct replacement_node **head, bool is_variable, const char *text) {
469 + struct replacement_node *new_node = malloc(sizeof(struct replacement_node));
470 + if (!new_node)
471 + return NULL;
472 +
473 + new_node->is_variable = is_variable;
474 + new_node->s = text;
475 + new_node->len = strlen(text);
476 + new_node->next = NULL;
477 +
478 + if (*head == NULL)
479 + *head = new_node;
480 +
481 + else {
482 + struct replacement_node *current = *head;
483 +
484 + // append it
485 + while (current->next != NULL)
486 + current = current->next;
487 +
488 + current->next = new_node;
489 + }
490 +
491 + return new_node;
492 +}
493 +
494 +bool parse_replacement_pattern(struct key_rewrite *rw) {
495 + const char *current = rw->replace_pattern;
496 +
497 + while (*current != '\0') {
498 + if (*current == '$' && *(current + 1) == '{') {
499 + // Start of a variable
500 + const char *end = strchr(current, '}');
501 + if (!end) {
502 + log2stderr("Error: Missing closing brace in replacement pattern: %s", rw->replace_pattern);
503 + return false;
504 + }
505 +
506 + size_t name_length = end - current - 2; // Length of the variable name
507 + char *variable_name = strndup(current + 2, name_length);
508 + if (!variable_name) {
509 + log2stderr("Error: Memory allocation failed for variable name.");
510 + return false;
511 + }
512 +
513 + struct replacement_node *node = add_replacement_node(&(rw->nodes), true, variable_name);
514 + if (!node) {
515 + free(variable_name);
516 + log2stderr("Error: Failed to add replacement node for variable.");
517 + return false;
518 + }
519 +
520 + current = end + 1; // Move past the variable
521 + }
522 + else {
523 + // Start of literal text
524 + const char *start = current;
525 + while (*current != '\0' && !(*current == '$' && *(current + 1) == '{')) {
526 + current++;
527 + }
528 +
529 + size_t text_length = current - start;
530 + char *text = strndup(start, text_length);
531 + if (!text) {
532 + log2stderr("Error: Memory allocation failed for literal text.");
533 + return false;
534 + }
535 +
536 + struct replacement_node *node = add_replacement_node(&(rw->nodes), false, text);
537 + if (!node) {
538 + free(text);
539 + log2stderr("Error: Failed to add replacement node for text.");
540 + return false;
541 + }
542 + }
543 + }
544 +
545 + return true;
546 +}
547 +
548 +static bool is_symbol(char c) {
549 + return !isalpha(c) && !isdigit(c) && !iscntrl(c);
550 +}
551 +
552 +static bool parse_rewrite(struct log_job *jb, const char *param) {
553 + // Search for '=' in param
554 + const char *equal_sign = strchr(param, '=');
555 + if (!equal_sign || equal_sign == param) {
556 + log2stderr("Error: Invalid rewrite format, '=' not found or at the start in %s", param);
557 + return false;
558 + }
559 +
560 + // Get the next character as the separator
561 + char separator = *(equal_sign + 1);
562 + if (!separator || !is_symbol(separator)) {
563 + log2stderr("Error: rewrite separator not found after '=', or is not one of /\\|-# in: %s", param);
564 + return false;
565 + }
566 +
567 + // Find the next occurrence of the separator
568 + const char *second_separator = strchr(equal_sign + 2, separator);
569 + if (!second_separator) {
570 + log2stderr("Error: rewrite second separator not found in: %s", param);
571 + return false;
572 + }
573 +
574 + // Check if the search pattern is empty
575 + if (equal_sign + 1 == second_separator) {
576 + log2stderr("Error: rewrite search pattern is empty in: %s", param);
577 + return false;
578 + }
579 +
580 + // Check if the replacement pattern is empty
581 + if (*(second_separator + 1) == '\0') {
582 + log2stderr("Error: rewrite replacement pattern is empty in: %s", param);
583 + return false;
584 + }
585 +
586 + // Reserve a slot in rewrites
587 + if (jb->rewrites.used >= MAX_REWRITES) {
588 + log2stderr("Error: Exceeded maximum number of rewrite rules, while processing: %s", param);
589 + return false;
590 + }
591 +
592 + // Extract key, search pattern, and replacement pattern
593 + char *key = strndup(param, equal_sign - param);
594 + char *search_pattern = strndup(equal_sign + 2, second_separator - (equal_sign + 2));
595 + char *replace_pattern = strdup(second_separator + 1);
596 +
597 + // Create the PCRE2 pattern
598 + pcre2_code *re = jb_compile_pcre2_pattern(search_pattern);
599 + if (!re) {
600 + free(key);
601 + free(search_pattern);
602 + free(replace_pattern);
603 + return false;
604 + }
605 +
606 + struct key_rewrite *rw = &jb->rewrites.array[jb->rewrites.used++];
607 + rw->hash = XXH3_64bits(key, strlen(key));
608 + rw->key = key;
609 + rw->search_pattern = search_pattern;
610 + rw->replace_pattern = replace_pattern;
611 + rw->re = re;
612 + rw->match_data = pcre2_match_data_create_from_pattern(rw->re, NULL);
613 +
614 + // Parse the replacement pattern and create the linked list
615 + if (!parse_replacement_pattern(rw)) {
616 + pcre2_match_data_free(rw->match_data);
617 + pcre2_code_free(rw->re);
618 + free(rw->key);
619 + free(rw->search_pattern);
620 + free(rw->replace_pattern);
621 + jb->rewrites.used--;
622 + return false;
623 + }
624 +
625 + return true;
626 +}
627 +
628 +bool parse_parameters(struct log_job *jb, int argc, char **argv) {
629 + for (int i = 1; i < argc; i++) {
630 + char *arg = argv[i];
631 + if (strcmp(arg, "--help") == 0 || strcmp(arg, "-h") == 0) {
632 + display_help(argv[0]);
633 + exit(0);
634 + }
635 + else if (strncmp(arg, "--filename-key=", 15) == 0)
636 + jb->filename.key = arg + 15;
637 + else if (strncmp(arg, "--unmatched-key=", 16) == 0)
638 + jb->unmatched.key = arg + 16;
639 + else if(strncmp(arg, "--duplicate=", 12) == 0) {
640 + const char *first_key = arg + 12;
641 + const char *comma = strchr(first_key, '=');
642 + if(!comma) {
643 + log2stderr("Error: --duplicate=TARGET=KEY1,... is missing the equal sign.");
644 + return false;
645 + }
646 + const char *next_key = comma + 1;
647 +
648 + if (jb->dups.used >= MAX_KEY_DUPS) {
649 + log2stderr("Error: too many duplications. You can duplicate up to %d keys.", MAX_KEY_DUPS);
650 + return false;
651 + }
652 +
653 + size_t first_key_len = comma - first_key;
654 + struct key_dup *kd = &jb->dups.array[jb->dups.used++];
655 + kd->target = strndup(first_key, first_key_len);
656 + kd->hash = XXH3_64bits(first_key, first_key_len);
657 + kd->used = 0;
658 +
659 + while(next_key) {
660 + if(kd->used >= MAX_KEY_DUPS_KEYS) {
661 + log2stderr("Error: too many keys in duplication of target '%s'.", kd->target);
662 + return false;
663 + }
664 +
665 + first_key = next_key;
666 + comma = strchr(first_key, ',');
667 +
668 + if(comma) {
669 + first_key_len = comma - first_key;
670 + kd->keys[kd->used++] = strndup(first_key, first_key_len);
671 + next_key = comma + 1;
672 + }
673 + else {
674 + kd->keys[kd->used++] = strdup(first_key);
675 + next_key = NULL;
676 + }
677 + }
678 + }
679 + else if(strncmp(arg, "--inject=", 9) == 0) {
680 + if(jb->injections.used >= MAX_INJECTIONS) {
681 + log2stderr("Error: too many injections. You can inject up to %d lines.", MAX_INJECTIONS);
682 + return false;
683 + }
684 +
685 + const char *key = arg + 9;
686 + const char *equal = strchr(key, '=');
687 + if(!equal) {
688 + log2stderr("Error: injection '%s' does not have an equal sign.", key);
689 + return false;
690 + }
691 +
692 + key_value_replace(&jb->injections.keys[jb->injections.used++],
693 + key, equal - key,
694 + equal + 1, strlen(equal + 1));
695 + }
696 + else if(strncmp(arg, "--inject-unmatched=", 19) == 0) {
697 + if(jb->unmatched.injections.used >= MAX_INJECTIONS) {
698 + log2stderr("Error: too many unmatched injections. You can inject up to %d lines.", MAX_INJECTIONS);
699 + return false;
700 + }
701 + const char *key = arg + 19;
702 + const char *equal = strchr(key, '=');
703 + if(!equal) {
704 + log2stderr("Error: unmatched injection '%s' does not have an equal sign.", key);
705 + return false;
706 + }
707 +
708 + key_value_replace(&jb->unmatched.injections.keys[jb->unmatched.injections.used++],
709 + key, equal - key,
710 + equal + 1, strlen(equal + 1));
711 + }
712 + else if(strncmp(arg, "--rewrite=", 10) == 0) {
713 + if(!parse_rewrite(jb, arg + 10))
714 + return false;
715 + }
716 + else {
717 + // Assume it's the pattern if not recognized as a parameter
718 + if (!jb->pattern) {
719 + jb->pattern = arg;
720 + } else {
721 + log2stderr("Error: Multiple patterns detected. Specify only one pattern.");
722 + return false;
723 + }
724 + }
725 + }
726 +
727 + // Check if a pattern is set and exactly one pattern is specified
728 + if (!jb->pattern) {
729 + log2stderr("Error: Pattern not specified.");
730 + display_help(argv[0]);
731 + return false;
732 + }
733 +
734 + return true;
735 +}
736 +
737 +// ----------------------------------------------------------------------------
738 +// injection of constant fields
739 +
740 +static void jb_select_which_injections_should_be_injected_on_unmatched(struct log_job *jb) {
741 + // mark all injections to be added to unmatched logs
742 + for(size_t i = 0; i < jb->injections.used ; i++)
743 + jb->injections.keys[i].on_unmatched = true;
744 +
745 + if(jb->injections.used && jb->unmatched.injections.used) {
746 + // we have both injections and injections on unmatched
747 +
748 + // we find all the injections that are also configured as injections on unmatched,
749 + // and we disable them, so that the output will not have the same key twice
750 +
751 + for(size_t i = 0; i < jb->injections.used ;i++) {
752 + for(size_t u = 0; u < jb->unmatched.injections.used ; u++) {
753 + if(strcmp(jb->injections.keys[i].key, jb->unmatched.injections.keys[u].key) == 0)
754 + jb->injections.keys[i].on_unmatched = false;
755 + }
756 + }
757 + }
758 +}
759 +
760 +
761 +static inline void jb_finalize_injections(struct log_job *jb, bool line_is_matched) {
762 + for (size_t j = 0; j < jb->injections.used; j++) {
763 + if(!line_is_matched && !jb->injections.keys[j].on_unmatched)
764 + continue;
765 +
766 + send_key_value_constant(jb, jb->injections.keys[j].key, jb->injections.keys[j].value.s);
767 + }
768 +}
769 +
770 +static inline void jb_reset_injections(struct log_job *jb) {
771 + for(size_t d = 0; d < jb->dups.used ; d++) {
772 + struct key_dup *kd = &jb->dups.array[d];
773 + kd->exposed = false;
774 +
775 + for(size_t g = 0; g < kd->used ; g++) {
776 + if(kd->values[g].s)
777 + kd->values[g].s[0] = '\0';
778 + }
779 + }
780 +}
781 +
782 +// ----------------------------------------------------------------------------
783 +// duplications
784 +
785 +static inline void jb_send_duplications_for_key(struct log_job *jb, const char *key, XXH64_hash_t hash, const char *value, size_t value_len) {
786 + // IMPORTANT:
787 + // The 'value' may not be NULL terminated and have more data that the value we need
788 +
789 + for (size_t d = 0; d < jb->dups.used; d++) {
790 + struct key_dup *kd = &jb->dups.array[d];
791 +
792 + if(kd->exposed || kd->used == 0)
793 + continue;
794 +
795 + if(kd->used == 1) {
796 + // just one key to be duplicated
797 + if(strcmp(kd->keys[0], key) == 0) {
798 + send_key_value_and_rewrite(jb, kd->target, kd->hash, value, value_len);
799 + kd->exposed = true;
800 + }
801 + }
802 + else {
803 + // multiple keys to be duplicated
804 + for(size_t g = 0; g < kd->used ; g++) {
805 + if(strcmp(kd->keys[g], key) == 0)
806 + txt_replace(&kd->values[g], value, value_len);
807 + }
808 + }
809 + }
810 +}
811 +
812 +static inline void jb_send_remaining_duplications(struct log_job *jb) {
813 + static __thread char buffer[MAX_VALUE_LEN + 1];
814 +
815 + // IMPORTANT:
816 + // all duplications are exposed, even the ones we haven't found their keys in the source,
817 + // so that the output always has the same fields for matched entries.
818 +
819 + for(size_t d = 0; d < jb->dups.used ; d++) {
820 + struct key_dup *kd = &jb->dups.array[d];
821 +
822 + if(kd->exposed || kd->used == 0)
823 + continue;
824 +
825 + buffer[0] = '\0';
826 + size_t remaining = sizeof(buffer);
827 + char *s = buffer;
828 +
829 + for(size_t g = 0; g < kd->used ; g++) {
830 + if(remaining < 2) {
831 + log2stderr("Warning: duplicated key '%s' cannot fit the values.", kd->target);
832 + break;
833 + }
834 +
835 + if(g > 0) {
836 + *s++ = ',';
837 + *s = '\0';
838 + remaining--;
839 + }
840 +
841 + char *value = (kd->values[g].s && kd->values[g].s[0]) ? kd->values[g].s : "[unavailable]";
842 + size_t len = strlen(value);
843 + size_t copied = copy_to_buffer(s, remaining, value, len);
844 + remaining -= copied;
845 + s += copied;
846 +
847 + if(copied != len) {
848 + log2stderr("Warning: duplicated key '%s' will have truncated value", jb->dups.array[d].target);
849 + break;
850 + }
851 + }
852 + send_key_value_and_rewrite(jb, kd->target, kd->hash, buffer, s - buffer);
853 + }
854 +}
855 +
856 +// ----------------------------------------------------------------------------
857 +// filename injection
858 +
859 +static inline void jb_inject_filename(struct log_job *jb) {
860 + if (jb->filename.key && jb->filename.current[0])
861 + send_key_value_constant(jb, jb->filename.key, jb->filename.current);
862 +}
863 +
864 +static inline bool jb_switched_filename(struct log_job *jb, const char *line, size_t len) {
865 + // IMPORTANT:
866 + // Return TRUE when the caller should skip this line (because it is ours).
867 + // Unfortunately, we have to consume empty lines too.
868 +
869 + // IMPORTANT:
870 + // filename may not be NULL terminated and have more data than the filename.
871 +
872 + if (!len) {
873 + jb->filename.last_line_was_empty = true;
874 + return true;
875 + }
876 +
877 + // Check if it's a log file change line
878 + if (jb->filename.last_line_was_empty && line[0] == '=' && strncmp(line, "==> ", 4) == 0) {
879 + const char *start = line + 4;
880 + const char *end = strstr(line, " <==");
881 + while (*start == ' ') start++;
882 + if (*start != '\n' && *start != '\0' && end) {
883 + copy_to_buffer(jb->filename.current, sizeof(jb->filename.current),
884 + start, end - start);
885 + return true;
886 + }
887 + }
888 +
889 + jb->filename.last_line_was_empty = false;
890 + return false;
891 +}
892 +
893 +// ----------------------------------------------------------------------------
894 +// input reading
895 +
896 +static char *get_next_line(struct log_job *jb, char *buffer, size_t size, size_t *line_length) {
897 + if(!fgets(buffer, (int)size, stdin)) {
898 + *line_length = 0;
899 + return NULL;
900 + }
901 +
902 + char *line = buffer;
903 + size_t len = strlen(line);
904 +
905 + // remove trailing newlines and spaces
906 + while(len > 1 && (line[len - 1] == '\n' || isspace(line[len - 1])))
907 + line[--len] = '\0';
908 +
909 + // skip leading spaces
910 + while(isspace(*line)) {
911 + line++;
912 + len--;
913 + }
914 +
915 + *line_length = len;
916 + return line;
917 +}
918 +
919 +// ----------------------------------------------------------------------------
920 +
921 +static inline void jb_traverse_pcre2_named_groups_and_send_keys(struct log_job *jb, pcre2_code *re, pcre2_match_data *match_data, char *line) {
922 + PCRE2_SIZE *ovector = pcre2_get_ovector_pointer(match_data);
923 + uint32_t namecount;
924 + pcre2_pattern_info(re, PCRE2_INFO_NAMECOUNT, &namecount);
925 +
926 + if (namecount > 0) {
927 + PCRE2_SPTR name_table;
928 + pcre2_pattern_info(re, PCRE2_INFO_NAMETABLE, &name_table);
929 + uint32_t name_entry_size;
930 + pcre2_pattern_info(re, PCRE2_INFO_NAMEENTRYSIZE, &name_entry_size);
931 +
932 + const unsigned char *tabptr = name_table;
933 + for (uint32_t i = 0; i < namecount; i++) {
934 + int n = (tabptr[0] << 8) | tabptr[1];
935 + const char *group_name = (const char *)(tabptr + 2);
936 +
937 + PCRE2_SIZE start_offset = ovector[2 * n];
938 + PCRE2_SIZE end_offset = ovector[2 * n + 1];
939 + PCRE2_SIZE group_length = end_offset - start_offset;
940 +
941 + XXH64_hash_t hash = XXH3_64bits(group_name, strlen(group_name));
942 +
943 + send_key_value_and_rewrite(jb, group_name, hash, line + start_offset, group_length);
944 +
945 + // process the duplications
946 + jb_send_duplications_for_key(jb, group_name, hash, line + start_offset, group_length);
947 +
948 + tabptr += name_entry_size;
949 + }
950 +
951 + // print all non-exposed duplications
952 + jb_send_remaining_duplications(jb);
953 + }
954 +}
955 +
956 +// ----------------------------------------------------------------------------
957 +
958 +struct log_job log_job = { 0 };
959 +int main(int argc, char *argv[]) {
960 + struct log_job *jb = &log_job;
961 +
962 + if(!parse_parameters(jb, argc, argv))
963 + exit(1);
964 +
965 + jb_select_which_injections_should_be_injected_on_unmatched(jb);
966 +
967 + pcre2_code *re = jb_compile_pcre2_pattern(jb->pattern);
968 + pcre2_match_data *match_data = pcre2_match_data_create_from_pattern(re, NULL);
969 +
970 + char buffer[MAX_LINE_LENGTH];
971 + char *line;
972 + size_t len;
973 +
974 + while ((line = get_next_line(jb, buffer, sizeof(buffer), &len))) {
975 + if(jb_switched_filename(jb, line, len))
976 + continue;
977 +
978 + jb_reset_injections(jb);
979 +
980 + bool line_is_matched;
981 + if(!jb_pcre2_match(re, match_data, line, len)) {
982 + line_is_matched = false;
983 +
984 + if (jb->unmatched.key) {
985 + // we are sending errors to Journal
986 + send_key_value_error(jb->unmatched.key, "PCRE2 error on: %s", line);
987 +
988 + for (size_t j = 0; j < jb->unmatched.injections.used; j++)
989 + send_key_value_constant(jb, jb->unmatched.injections.keys[j].key,
990 + jb->unmatched.injections.keys[j].value.s);
991 + }
992 + else {
993 + // we are just logging errors to stderr
994 + continue;
995 + }
996 + }
997 + else {
998 + line_is_matched = true;
999 + jb_traverse_pcre2_named_groups_and_send_keys(jb, re, match_data, line);
1000 + }
1001 +
1002 + jb_inject_filename(jb);
1003 + jb_finalize_injections(jb, line_is_matched);
1004 +
1005 + printf("\n");
1006 + fflush(stdout);
1007 + }
1008 +
1009 + // Release memory used for the compiled regular expression and match data
1010 + pcre2_match_data_free(match_data);
1011 + pcre2_code_free(re);
1012 + jb_cleanup(jb);
1013 +
1014 + return 0;
1015 +}
libnetdata/log/log2journal.md new
+518
@@ -0,0 +1,518 @@
1 +# log2journal
2 +
3 +`log2journal` and `systemd-cat-native` can be used to convert a structured log file, such as the ones generated by web servers, into `systemd-journal` entries.
4 +
5 +By combining these tools, together with the usual UNIX shell tools you can create advanced log processing pipelines sending any kind of structured text logs to systemd-journald. This is a simple, but powerful and efficient way to handle log processing.
6 +
7 +The process involves the usual piping of shell commands, to get and process the log files in realtime.
8 +
9 +The overall process looks like this:
10 +
11 +```bash
12 +tail -F /var/log/nginx/*.log |\ # outputs log lines
13 + log2journal 'PATTERN' |\ # outputs Journal Export Format
14 + sed -u -e SEARCH-REPLACE-RULES |\ # optional rewriting rules
15 + systemd-cat-native # send to local/remote journald
16 +```
17 +
18 +Let's see the steps:
19 +
20 +1. `tail -F /var/log/nginx/*.log`<br/>this command will tail all `*.log` files in `/var/log/nginx/`. We use `-F` instead of `-f` to ensure that files will still be tailed after log rotation.
21 +2. `log2joural` is a Netdata program. It reads log entries and extracts fields, according to the PCRE2 pattern it accepts. It can also apply some basic operations on the fields, like injecting new fields or duplicating existing ones. The output of `log2journal` is in Systemd Journal Export Format, and it looks like this:
22 + ```bash
23 + KEY1=VALUE1 # << start of the first log line
24 + KEY2=VALUE2
25 + # << log lines separator
26 + KEY1=VALUE1 # << start of the second log line
27 + KEY2=VALUE2
28 + ```
29 +3. `sed` is an optional step and is an example. Any kind of processing can be applied at this stage, in case we want to alter the fields in some way. For example, we may want to set the PRIORITY field of systemd-journal to make Netdata dashboards and `journalctl` color the internal server errors. Or we may want to anonymize the logs, to remove sensitive information from them. Or even we may want to remove the variable parts of the requests, to make them uniform. We will see below how such processing can be done.
30 +4. `systemd-cat-native` is a Netdata program. I can send the logs to a local `systemd-journald` (journal namespaces supported), or to a remote `systemd-journal-remote`.
31 +
32 +## Real-life example
33 +
34 +We have an nginx server logging in this format:
35 +
36 +```bash
37 + log_format access '$remote_addr - $remote_user [$time_local] '
38 + '"$request" $status $body_bytes_sent '
39 + '$request_length $request_time '
40 + '"$http_referer" "$http_user_agent"';
41 +```
42 +
43 +First, let's find the right pattern for `log2journal`. We ask ChatGPT:
44 +
45 +```
46 +My nginx log uses this log format:
47 +
48 +log_format access '$remote_addr - $remote_user [$time_local] '
49 + '"$request" $status $body_bytes_sent '
50 + '$request_length $request_time '
51 + '"$http_referer" "$http_user_agent"';
52 +
53 +I want to use `log2joural` to convert this log for systemd-journal.
54 +`log2journal` accepts a PCRE2 regular expression, using the named groups
55 +in the pattern as the journal fields to extract from the logs.
56 +
57 +Prefix all PCRE2 group names with `NGINX_` and use capital characters only.
58 +
59 +For the $request, use the field `MESSAGE` (without NGINX_ prefix), so that
60 +it will appear in systemd journals as the message of the log.
61 +
62 +Please give me the PCRE2 pattern.
63 +```
64 +
65 +ChatGPT replies with this:
66 +
67 +```regexp
68 +^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>[^"]+)" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"
69 +```
70 +
71 +Let's test it with a sample line (instead of `tail`):
72 +
73 +```bash
74 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>[^"]+)" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"'
75 +MESSAGE=GET /index.html HTTP/1.1
76 +NGINX_BODY_BYTES_SENT=4172
77 +NGINX_HTTP_REFERER=-
78 +NGINX_HTTP_USER_AGENT=Go-http-client/1.1
79 +NGINX_REMOTE_ADDR=1.2.3.4
80 +NGINX_REMOTE_USER=-
81 +NGINX_REQUEST_LENGTH=104
82 +NGINX_REQUEST_TIME=0.001
83 +NGINX_STATUS=200
84 +NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
85 +
86 +```
87 +
88 +As you can see, it extracted all the fields.
89 +
90 +The `MESSAGE` however, has 3 fields by itself: the method, the URL and the procotol version. Let's ask ChatGPT to extract these too:
91 +
92 +```
93 +I see that the MESSAGE has 3 key items in it. The request method (GET, POST,
94 +etc), the URL and HTTP protocol version.
95 +
96 +I want to keep the MESSAGE as it is, with all the information in it, but also
97 +extract the 3 items from it as separate fields.
98 +
99 +Can this be done?
100 +```
101 +
102 +ChatGPT responded with this:
103 +
104 +```regexp
105 +^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"
106 +```
107 +
108 +Let's test this too:
109 +
110 +```bash
111 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"'
112 +MESSAGE=GET /index.html HTTP/1.1 # <<<<<<<<< MESSAGE
113 +NGINX_BODY_BYTES_SENT=4172
114 +NGINX_HTTP_REFERER=-
115 +NGINX_HTTP_USER_AGENT=Go-http-client/1.1
116 +NGINX_HTTP_VERSION=1.1 # <<<<<<<<< VERSION
117 +NGINX_METHOD=GET # <<<<<<<<< METHOD
118 +NGINX_REMOTE_ADDR=1.2.3.4
119 +NGINX_REMOTE_USER=-
120 +NGINX_REQUEST_LENGTH=104
121 +NGINX_REQUEST_TIME=0.001
122 +NGINX_STATUS=200
123 +NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
124 +NGINX_URL=/index.html # <<<<<<<<< URL
125 +
126 +```
127 +
128 +Ideally, we would want the 5xx errors to be red in our `journalctl` output. To achieve that we need to add a PRIORITY field to set the log level. Log priorities are numeric and follow the `syslog` priorities. Checking `/usr/include/sys/syslog.h` we can see these:
129 +
130 +```c
131 +#define LOG_EMERG 0 /* system is unusable */
132 +#define LOG_ALERT 1 /* action must be taken immediately */
133 +#define LOG_CRIT 2 /* critical conditions */
134 +#define LOG_ERR 3 /* error conditions */
135 +#define LOG_WARNING 4 /* warning conditions */
136 +#define LOG_NOTICE 5 /* normal but significant condition */
137 +#define LOG_INFO 6 /* informational */
138 +#define LOG_DEBUG 7 /* debug-level messages */
139 +```
140 +
141 +Avoid setting priority to 0 (`LOG_EMERG`), because these will be on your terminal (the journal uses `wall` to let you know of such events). A good priority for errors is 3 (red in `journalctl`), or 4 (yellow in `journalctl`).
142 +
143 +To set the PRIORITY field in the output, we can use `NGINX_STATUS` fields. We need a copy of it, which we will alter later.
144 +
145 +We can instruct `log2journal` to duplicate `NGINX_STATUS`, like this: `log2journal --duplicate=STATUS2PRIORITY=NGINX_STATUS`. Let's try it:
146 +
147 +```bash
148 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --duplicate=STATUS2PRIORITY=NGINX_STATUS
149 +MESSAGE=GET /index.html HTTP/1.1
150 +NGINX_BODY_BYTES_SENT=4172
151 +NGINX_HTTP_REFERER=-
152 +NGINX_HTTP_USER_AGENT=Go-http-client/1.1
153 +NGINX_HTTP_VERSION=1.1
154 +NGINX_METHOD=GET
155 +NGINX_REMOTE_ADDR=1.2.3.4
156 +NGINX_REMOTE_USER=-
157 +NGINX_REQUEST_LENGTH=104
158 +NGINX_REQUEST_TIME=0.001
159 +NGINX_STATUS=200
160 +STATUS2PRIORITY=200 # <<<<<<<<< STATUS2PRIORITY IS HERE
161 +NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
162 +NGINX_URL=/index.html
163 +
164 +```
165 +
166 +Now that we have the `STATUS2PRIORITY` field equal to the `NGINX_STATUS`, we can use a `sed` command to change it to the `PRIORITY` field we want. The `sed` command could be:
167 +
168 +```bash
169 +sed -u -e 's|STATUS2PRIORITY=5.*|PRIORITY=3|' -e 's|STATUS2PRIORITY=.*|PRIORITY=6|'
170 +```
171 +
172 +We use `-u` for unbuffered communication.
173 +
174 +This command first changes all 5xx `STATUS2PRIORITY` fields to `PRIORITY=3` (error) and then changes all the rest to `PRIORITY=6` (info). Let's see the whole of it:
175 +
176 +```bash
177 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --duplicate=STATUS2PRIORITY=NGINX_STATUS | sed -u -e 's|STATUS2PRIORITY=5.*|PRIORITY=3|' -e 's|STATUS2PRIORITY=.*|PRIORITY=6|'
178 +MESSAGE=GET /index.html HTTP/1.1
179 +NGINX_BODY_BYTES_SENT=4172
180 +NGINX_HTTP_REFERER=-
181 +NGINX_HTTP_USER_AGENT=Go-http-client/1.1
182 +NGINX_HTTP_VERSION=1.1
183 +NGINX_METHOD=GET
184 +NGINX_REMOTE_ADDR=1.2.3.4
185 +NGINX_REMOTE_USER=-
186 +NGINX_REQUEST_LENGTH=104
187 +NGINX_REQUEST_TIME=0.001
188 +NGINX_STATUS=200
189 +PRIORITY=6 # <<<<<<<<< PRIORITY IS HERE
190 +NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
191 +NGINX_URL=/index.html
192 +
193 +```
194 +
195 +Similarly, we could duplicate `NGINX_URL` to `NGINX_ENDPOINT` and then process it with sed to remove any query string, or replace IDs in the URL path with constant names, thus giving us uniform endpoints independently of the parameters.
196 +
197 +To complete the example, we can also inject a `SYSLOG_IDENTIFIER` with `log2journal`, using `--inject=SYSLOG_IDENTIFIER=nginx`, like this:
198 +
199 +```bash
200 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --duplicate=STATUS2PRIORITY=NGINX_STATUS --inject=SYSLOG_IDENTIFIER=nginx | sed -u -e 's|STATUS2PRIORITY=5.*|PRIORITY=3|' -e 's|STATUS2PRIORITY=.*|PRIORITY=6|'
201 +MESSAGE=GET /index.html HTTP/1.1
202 +NGINX_BODY_BYTES_SENT=4172
203 +NGINX_HTTP_REFERER=-
204 +NGINX_HTTP_USER_AGENT=Go-http-client/1.1
205 +NGINX_HTTP_VERSION=1.1
206 +NGINX_METHOD=GET
207 +NGINX_REMOTE_ADDR=1.2.3.4
208 +NGINX_REMOTE_USER=-
209 +NGINX_REQUEST_LENGTH=104
210 +NGINX_REQUEST_TIME=0.001
211 +NGINX_STATUS=200
212 +PRIORITY=6
213 +NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000
214 +NGINX_URL=/index.html
215 +SYSLOG_IDENTIFIER=nginx # <<<<<<<<< THIS HAS BEEN ADDED
216 +
217 +```
218 +
219 +Now the message is ready to be sent to a systemd-journal. For this we use `systemd-cat-native`. This command can send such messages to a journal running on the localhost, a local journal namespace, or a `systemd-journal-remote` running on another server. By just appending `| systemd-cat-native` to the command, the message will be sent to the local journal.
220 +
221 +
222 +```bash
223 +# echo '1.2.3.4 - - [19/Nov/2023:00:24:43 +0000] "GET /index.html HTTP/1.1" 200 4172 104 0.001 "-" "Go-http-client/1.1"' | log2journal '^(?<NGINX_REMOTE_ADDR>[^ ]+) - (?<NGINX_REMOTE_USER>[^ ]+) \[(?<NGINX_TIME_LOCAL>[^\]]+)\] "(?<MESSAGE>(?<NGINX_METHOD>[A-Z]+) (?<NGINX_URL>[^ ]+) HTTP/(?<NGINX_HTTP_VERSION>[^"]+))" (?<NGINX_STATUS>\d+) (?<NGINX_BODY_BYTES_SENT>\d+) (?<NGINX_REQUEST_LENGTH>\d+) (?<NGINX_REQUEST_TIME>[\d.]+) "(?<NGINX_HTTP_REFERER>[^"]*)" "(?<NGINX_HTTP_USER_AGENT>[^"]*)"' --duplicate=STATUS2PRIORITY=NGINX_STATUS --inject=SYSLOG_IDENTIFIER=nginx | sed -u -e 's|STATUS2PRIORITY=5.*|PRIORITY=3|' -e 's|STATUS2PRIORITY=.*|PRIORITY=6|' | systemd-cat-native
224 +# no output
225 +
226 +# let's find the message
227 +# journalctl -o verbose SYSLOG_IDENTIFIER=nginx
228 +Sun 2023-11-19 04:34:06.583912 EET [s=1eb59e7934984104ab3b61f5d9648057;i=115b6d4;b=7282d89d2e6e4299969a6030302ff3e4;m=69b419673;t=60a783417ac72;x=2cec5dde8bf01ee7]
229 + PRIORITY=6
230 + _UID=0
231 + _GID=0
232 + _BOOT_ID=7282d89d2e6e4299969a6030302ff3e4
233 + _MACHINE_ID=6b72c55db4f9411dbbb80b70537bf3a8
234 + _HOSTNAME=costa-xps9500
235 + _RUNTIME_SCOPE=system
236 + _TRANSPORT=journal
237 + _CAP_EFFECTIVE=1ffffffffff
238 + _AUDIT_LOGINUID=1000
239 + _AUDIT_SESSION=1
240 + _SYSTEMD_CGROUP=/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-59780d3d-a3ff-4a82-a6fe-8d17d2261106.scope
241 + _SYSTEMD_OWNER_UID=1000
242 + _SYSTEMD_UNIT=user@1000.service
243 + _SYSTEMD_USER_UNIT=vte-spawn-59780d3d-a3ff-4a82-a6fe-8d17d2261106.scope
244 + _SYSTEMD_SLICE=user-1000.slice
245 + _SYSTEMD_USER_SLICE=app-org.gnome.Terminal.slice
246 + _SYSTEMD_INVOCATION_ID=6195d8c4c6654481ac9a30e9a8622ba1
247 + _COMM=systemd-cat-nat
248 + MESSAGE=GET /index.html HTTP/1.1 # <<<<<<<<< CHECK
249 + NGINX_BODY_BYTES_SENT=4172 # <<<<<<<<< CHECK
250 + NGINX_HTTP_REFERER=- # <<<<<<<<< CHECK
251 + NGINX_HTTP_USER_AGENT=Go-http-client/1.1 # <<<<<<<<< CHECK
252 + NGINX_HTTP_VERSION=1.1 # <<<<<<<<< CHECK
253 + NGINX_METHOD=GET # <<<<<<<<< CHECK
254 + NGINX_REMOTE_ADDR=1.2.3.4 # <<<<<<<<< CHECK
255 + NGINX_REMOTE_USER=- # <<<<<<<<< CHECK
256 + NGINX_REQUEST_LENGTH=104 # <<<<<<<<< CHECK
257 + NGINX_REQUEST_TIME=0.001 # <<<<<<<<< CHECK
258 + NGINX_STATUS=200 # <<<<<<<<< CHECK
259 + NGINX_TIME_LOCAL=19/Nov/2023:00:24:43 +0000 # <<<<<<<<< CHECK
260 + NGINX_URL=/index.html # <<<<<<<<< CHECK
261 + SYSLOG_IDENTIFIER=nginx # <<<<<<<<< CHECK
262 + _PID=354312
263 + _SOURCE_REALTIME_TIMESTAMP=1700361246583912
264 +
265 +```
266 +
267 +So, the log line, with all its fields parsed, ended up in systemd-journal.
268 +
269 +The complete example, would look like the following script.
270 +Running this script with parameter `test` will produce output on the terminal for you to inspect.
271 +Unmatched log entries are added to the journal with PRIORITY=1 (`ERR_ALERT`), so that you can spot them.
272 +
273 +We also used the `--filename-key` of `log2journal`, which parses the filename when `tail` switches output
274 +between files, and adds the field `NGINX_LOG_FILE` with the filename each log line comes from.
275 +
276 +Finally, the script also adds the field `NGINX_STATUS_FAMILY` taking values `2xx`, `3xx`, etc, so that
277 +it is easy to find all the logs of a specific status family.
278 +
279 +```bash
280 +#!/usr/bin/env bash
281 +
282 +test=0
283 +last=0
284 +send_or_show='./systemd-cat-native'
285 +[ "${1}" = "test" ] && test=1 && last=100 && send_or_show=cat
286 +
287 +pattern='(?x) # Enable PCRE2 extended mode
288 +^
289 +(?<NGINX_REMOTE_ADDR>[^ ]+) \s - \s # NGINX_REMOTE_ADDR
290 +(?<NGINX_REMOTE_USER>[^ ]+) \s # NGINX_REMOTE_USER
291 +\[
292 + (?<NGINX_TIME_LOCAL>[^\]]+) # NGINX_TIME_LOCAL
293 +\]
294 +\s+ "
295 +(?<MESSAGE> # MESSAGE
296 + (?<NGINX_METHOD>[A-Z]+) \s+ # NGINX_METHOD
297 + (?<NGINX_URL>[^ ]+) \s+ # NGINX_URL
298 + HTTP/(?<NGINX_HTTP_VERSION>[^"]+) # NGINX_HTTP_VERSION
299 +)
300 +" \s+
301 +(?<NGINX_STATUS>\d+) \s+ # NGINX_STATUS
302 +(?<NGINX_BODY_BYTES_SENT>\d+) \s+ # NGINX_BODY_BYTES_SENT
303 +"(?<NGINX_HTTP_REFERER>[^"]*)" \s+ # NGINX_HTTP_REFERER
304 +"(?<NGINX_HTTP_USER_AGENT>[^"]*)" # NGINX_HTTP_USER_AGENT
305 +'
306 +
307 +tail -n $last -F /var/log/nginx/*access.log |\
308 + log2journal "${pattern}" \
309 + --filename-key=NGINX_LOG_FILE \
310 + --duplicate=STATUS2PRIORITY=NGINX_STATUS \
311 + --duplicate=STATUS_FAMILY=NGINX_STATUS \
312 + --inject=SYSLOG_IDENTIFIER=nginx \
313 + --unmatched-key=MESSAGE \
314 + --inject-unmatched=PRIORITY=1 \
315 + | sed -u \
316 + -e 's|^STATUS2PRIORITY=5.*$|PRIORITY=3|' \
317 + -e 's|^STATUS2PRIORITY=.*$|PRIORITY=6|' \
318 + -e 's|^STATUS_FAMILY=\([0-9]\).*$|NGINX_STATUS_FAMILY=\1xx|' \
319 + -e 's|^STATUS_FAMILY=.*$|NGINX_STATUS_FAMILY=UNKNOWN|' \
320 + | $send_or_show
321 +```
322 +
323 +
324 +## `log2journal` options
325 +
326 +```
327 +
328 +Netdata log2journal v1.43.0-337-g116dc1bc3
329 +
330 +Convert structured log input to systemd Journal Export Format.
331 +
332 +Using PCRE2 patterns, extract the fields from structured logs on the standard
333 +input, and generate output according to systemd Journal Export Format
334 +
335 +Usage: ./log2journal [OPTIONS] PATTERN
336 +
337 +Options:
338 +
339 + --filename-key=KEY
340 + Add a field with KEY as the key and the current filename as value.
341 + Automatically detects filenames when piped after 'tail -F',
342 + and tail matches multiple filenames.
343 + To inject the filename when tailing a single file, use --inject.
344 +
345 + --unmatched-key=KEY
346 + Include unmatched log entries in the output with KEY as the field name.
347 + Use this to include unmatched entries to the output stream.
348 + Usually it should be set to --unmatched-key=MESSAGE so that the
349 + unmatched entry will appear as the log message in the journals.
350 + Use --inject-unmatched to inject additional fields to unmatched lines.
351 +
352 + --duplicate=TARGET=KEY1[,KEY2[,KEY3[,...]]
353 + Create a new key called TARGET, duplicating the values of the keys
354 + given. Useful for further processing. When multiple keys are given,
355 + their values are separated by comma.
356 + Up to 2048 duplications can be given on the command line, and up to
357 + 10 keys per duplication command are allowed.
358 +
359 + --inject=LINE
360 + Inject constant fields to the output (both matched and unmatched logs).
361 + --inject entries are added to unmatched lines too, when their key is
362 + not used in --inject-unmatched (--inject-unmatched override --inject).
363 + Up to 2048 fields can be injected.
364 +
365 + --inject-unmatched=LINE
366 + Inject lines into the output for each unmatched log entry.
367 + Usually, --inject-unmatched=PRIORITY=3 is needed to mark the unmatched
368 + lines as errors, so that they can easily be spotted in the journals.
369 + Up to 2048 such lines can be injected.
370 +
371 + -h, --help
372 + Display this help and exit.
373 +
374 + PATTERN
375 + PATTERN should be a valid PCRE2 regular expression.
376 + RE2 regular expressions (like the ones usually used in Go applications),
377 + are usually valid PCRE2 patterns too.
378 + Regular expressions without named groups are ignored.
379 +
380 +The maximum line length accepted is 1048576 characters.
381 +The maximum number of fields in the PCRE2 pattern is 8192.
382 +
383 +JOURNAL FIELDS RULES (enforced by systemd-journald)
384 +
385 + - field names can be up to 64 characters
386 + - the only allowed field characters are A-Z, 0-9 and underscore
387 + - the first character of fields cannot be a digit
388 + - protected journal fields start with underscore:
389 + * they are accepted by systemd-journal-remote
390 + * they are NOT accepted by a local systemd-journald
391 +
392 + For best results, always include these fields:
393 +
394 + MESSAGE=TEXT
395 + The MESSAGE is the body of the log entry.
396 + This field is what we usually see in our logs.
397 +
398 + PRIORITY=NUMBER
399 + PRIORITY sets the severity of the log entry.
400 + 0=emerg, 1=alert, 2=crit, 3=err, 4=warn, 5=notice, 6=info, 7=debug
401 + - Emergency events (0) are usually broadcast to all terminals.
402 + - Emergency, alert, critical, and error (0-3) are usually colored red.
403 + - Warning (4) entries are usually colored yellow.
404 + - Notice (5) entries are usually bold or have a brighter white color.
405 + - Info (6) entries are the default.
406 + - Debug (7) entries are usually grayed or dimmed.
407 +
408 + SYSLOG_IDENTIFIER=NAME
409 + SYSLOG_IDENTIFIER sets the name of application.
410 + Use something descriptive, like: SYSLOG_IDENTIFIER=nginx-logs
411 +
412 +You can find the most common fields at 'man systemd.journal-fields'.
413 +
414 +```
415 +
416 +## `systemd-cat-native` options
417 +
418 +```
419 +
420 +Netdata systemd-cat-native v1.43.0-319-g4ada93a6e
421 +
422 +This program reads from its standard input, lines in the format:
423 +
424 +KEY1=VALUE1\n
425 +KEY2=VALUE2\n
426 +KEYN=VALUEN\n
427 +\n
428 +
429 +and sends them to systemd-journal.
430 +
431 + - Binary journal fields are not accepted at its input
432 + - Binary journal fields can be generated after newline processing
433 + - Messages have to be separated by an empty line
434 + - Keys starting with underscore are not accepted (by journald)
435 + - Other rules imposed by systemd-journald are imposed (by journald)
436 +
437 +Usage:
438 +
439 + ./systemd-cat-native
440 + [--newline=STRING]
441 + [--log-as-netdata|-N]
442 + [--namespace=NAMESPACE] [--socket=PATH]
443 + [--url=URL [--key=FILENAME] [--cert=FILENAME] [--trust=FILENAME|all]]
444 +
445 +The program has the following modes of logging:
446 +
447 + * Log to a local systemd-journald or stderr
448 +
449 + This is the default mode. If systemd-journald is available, logs will be
450 + sent to systemd, otherwise logs will be printed on stderr, using logfmt
451 + formatting. Options --socket and --namespace are available to configure
452 + the journal destination:
453 +
454 + --socket=PATH
455 + The path of a systemd-journald UNIX socket.
456 + The program will use the default systemd-journald socket when this
457 + option is not used.
458 +
459 + --namespace=NAMESPACE
460 + The name of a configured and running systemd-journald namespace.
461 + The program will produce the socket path based on its internal
462 + defaults, to send the messages to the systemd journal namespace.
463 +
464 + * Log as Netdata, enabled with --log-as-netdata or -N
465 +
466 + In this mode the program uses environment variables set by Netdata for
467 + the log destination. Only log fields defined by Netdata are accepted.
468 + If the environment variables expected by Netdata are not found, it
469 + falls back to stderr logging in logfmt format.
470 +
471 + * Log to a systemd-journal-remote TCP socket, enabled with --url=URL
472 +
473 + In this mode, the program will directly sent logs to a remote systemd
474 + journal (systemd-journal-remote expected at the destination)
475 + This mode is available even when the local system does not support
476 + systemd, or even it is not Linux, allowing a remote Linux systemd
477 + journald to become the logs database of the local system.
478 +
479 + --url=URL
480 + The destination systemd-journal-remote address and port, similarly
481 + to what /etc/systemd/journal-upload.conf accepts.
482 + Usually it is in the form: https://ip.address:19532
483 + Both http and https URLs are accepted. When using https, the
484 + following additional options are accepted:
485 +
486 + --key=FILENAME
487 + The filename of the private key of the server.
488 + The default is: /etc/ssl/private/journal-upload.pem
489 +
490 + --cert=FILENAME
491 + The filename of the public key of the server.
492 + The default is: /etc/ssl/certs/journal-upload.pem
493 +
494 + --trust=FILENAME | all
495 + The filename of the trusted CA public key.
496 + The default is: /etc/ssl/ca/trusted.pem
497 + The keyword 'all' can be used to trust all CAs.
498 +
499 + NEWLINES PROCESSING
500 + systemd-journal logs entries may have newlines in them. However the
501 + Journal Export Format uses binary formatted data to achieve this,
502 + making it hard for text processing.
503 +
504 + To overcome this limitation, this program allows single-line text
505 + formatted values at its input, to be binary formatted multi-line Journal
506 + Export Format at its output.
507 +
508 + To achieve that it allows replacing a given string to a newline.
509 + The parameter --newline=STRING allows setting the string to be replaced
510 + with newlines.
511 +
512 + For example by setting --newline='{NEWLINE}', the program will replace
513 + all occurrences of {NEWLINE} with the newline character, within each
514 + VALUE of the KEY=VALUE lines. Once this this done, the program will
515 + switch the field to the binary Journal Export Format before sending the
516 + log event to systemd-journal.
517 +
518 +```
libnetdata/log/systemd-cat-native.c new
+781
@@ -0,0 +1,781 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "systemd-cat-native.h"
4 +#include "../required_dummies.h"
5 +
6 +#ifdef __FreeBSD__
7 +#include <sys/endian.h>
8 +#endif
9 +
10 +#ifdef __APPLE__
11 +#include <machine/endian.h>
12 +#endif
13 +
14 +static void log_message_to_stderr(BUFFER *msg) {
15 + CLEAN_BUFFER *tmp = buffer_create(0, NULL);
16 +
17 + for(size_t i = 0; i < msg->len ;i++) {
18 + if(isprint(msg->buffer[i]))
19 + buffer_putc(tmp, msg->buffer[i]);
20 + else {
21 + buffer_putc(tmp, '[');
22 + buffer_print_uint64_hex(tmp, msg->buffer[i]);
23 + buffer_putc(tmp, ']');
24 + }
25 + }
26 +
27 + fprintf(stderr, "SENDING: %s\n", buffer_tostring(tmp));
28 +}
29 +
30 +static inline buffered_reader_ret_t get_next_line(struct buffered_reader *reader, BUFFER *line, int timeout_ms) {
31 + while(true) {
32 + if(unlikely(!buffered_reader_next_line(reader, line))) {
33 + buffered_reader_ret_t ret = buffered_reader_read_timeout(reader, STDIN_FILENO, timeout_ms, false);
34 + if(unlikely(ret != BUFFERED_READER_READ_OK))
35 + return ret;
36 +
37 + continue;
38 + }
39 + else {
40 + // make sure the buffer is NULL terminated
41 + line->buffer[line->len] = '\0';
42 +
43 + // remove the trailing newlines
44 + while(line->len && line->buffer[line->len - 1] == '\n')
45 + line->buffer[--line->len] = '\0';
46 +
47 + return BUFFERED_READER_READ_OK;
48 + }
49 + }
50 +}
51 +
52 +static inline size_t copy_replacing_newlines(char *dst, size_t dst_len, const char *src, size_t src_len, const char *newline) {
53 + if (!dst || !src) return 0;
54 +
55 + const char *current_src = src;
56 + const char *src_end = src + src_len; // Pointer to the end of src
57 + char *current_dst = dst;
58 + size_t remaining_dst_len = dst_len;
59 + size_t newline_len = newline && *newline ? strlen(newline) : 0;
60 +
61 + size_t bytes_copied = 0; // To track the number of bytes copied
62 +
63 + while (remaining_dst_len > 1 && current_src < src_end) {
64 + if (newline_len > 0) {
65 + const char *found = strstr(current_src, newline);
66 + if (found && found < src_end) {
67 + size_t copy_len = found - current_src;
68 + if (copy_len >= remaining_dst_len) copy_len = remaining_dst_len - 1;
69 +
70 + memcpy(current_dst, current_src, copy_len);
71 + current_dst += copy_len;
72 + *current_dst++ = '\n';
73 + remaining_dst_len -= (copy_len + 1);
74 + bytes_copied += copy_len + 1; // +1 for the newline character
75 + current_src = found + newline_len;
76 + continue;
77 + }
78 + }
79 +
80 + // Copy the remaining part of src to dst
81 + size_t copy_len = src_end - current_src;
82 + if (copy_len >= remaining_dst_len) copy_len = remaining_dst_len - 1;
83 +
84 + memcpy(current_dst, current_src, copy_len);
85 + current_dst += copy_len;
86 + remaining_dst_len -= copy_len;
87 + bytes_copied += copy_len;
88 + break;
89 + }
90 +
91 + // Ensure the string is null-terminated
92 + *current_dst = '\0';
93 +
94 + return bytes_copied;
95 +}
96 +
97 +static inline void buffer_memcat_replacing_newlines(BUFFER *wb, const char *src, size_t src_len, const char *newline) {
98 + if(!src) return;
99 +
100 + const char *equal;
101 + if(!newline || !*newline || !strstr(src, newline) || !(equal = strchr(src, '='))) {
102 + buffer_memcat(wb, src, src_len);
103 + buffer_putc(wb, '\n');
104 + return;
105 + }
106 +
107 + size_t key_len = equal - src;
108 + buffer_memcat(wb, src, key_len);
109 + buffer_putc(wb, '\n');
110 +
111 + char *length_ptr = &wb->buffer[wb->len];
112 + uint64_t le_size = 0;
113 + buffer_memcat(wb, &le_size, sizeof(le_size));
114 +
115 + const char *value = ++equal;
116 + size_t value_len = src_len - key_len - 1;
117 + buffer_need_bytes(wb, value_len + 1);
118 + size_t size = copy_replacing_newlines(&wb->buffer[wb->len], value_len + 1, value, value_len, newline);
119 + wb->len += size;
120 + buffer_putc(wb, '\n');
121 +
122 + le_size = htole64(size);
123 + memcpy(length_ptr, &le_size, sizeof(le_size));
124 +}
125 +
126 +// ----------------------------------------------------------------------------
127 +// log to a systemd-journal-remote
128 +
129 +#ifdef HAVE_CURL
130 +#include <curl/curl.h>
131 +
132 +#ifndef HOST_NAME_MAX
133 +#define HOST_NAME_MAX 256
134 +#endif
135 +
136 +char global_hostname[HOST_NAME_MAX] = "";
137 +char global_boot_id[UUID_COMPACT_STR_LEN] = "";
138 +#define BOOT_ID_PATH "/proc/sys/kernel/random/boot_id"
139 +
140 +#define DEFAULT_PRIVATE_KEY "/etc/ssl/private/journal-upload.pem"
141 +#define DEFAULT_PUBLIC_KEY "/etc/ssl/certs/journal-upload.pem"
142 +#define DEFAULT_CA_CERT "/etc/ssl/ca/trusted.pem"
143 +
144 +struct upload_data {
145 + char *data;
146 + size_t length;
147 +};
148 +
149 +static size_t systemd_journal_remote_read_callback(void *ptr, size_t size, size_t nmemb, void *userp) {
150 + struct upload_data *upload = (struct upload_data *)userp;
151 + size_t buffer_size = size * nmemb;
152 +
153 + if (upload->length) {
154 + size_t copy_size = upload->length < buffer_size ? upload->length : buffer_size;
155 + memcpy(ptr, upload->data, copy_size);
156 + upload->data += copy_size;
157 + upload->length -= copy_size;
158 + return copy_size;
159 + }
160 +
161 + return 0;
162 +}
163 +
164 +CURL* initialize_connection_to_systemd_journal_remote(const char* url, const char* private_key, const char* public_key, const char* ca_cert, struct curl_slist **headers) {
165 + CURL *curl = curl_easy_init();
166 + if (!curl) {
167 + fprintf(stderr, "Failed to initialize curl\n");
168 + return NULL;
169 + }
170 +
171 + *headers = curl_slist_append(*headers, "Content-Type: application/vnd.fdo.journal");
172 + *headers = curl_slist_append(*headers, "Transfer-Encoding: chunked");
173 + curl_easy_setopt(curl, CURLOPT_HTTPHEADER, *headers);
174 + curl_easy_setopt(curl, CURLOPT_URL, url);
175 + curl_easy_setopt(curl, CURLOPT_POST, 1L);
176 + curl_easy_setopt(curl, CURLOPT_READFUNCTION, systemd_journal_remote_read_callback);
177 +
178 + if (strncmp(url, "https://", 8) == 0) {
179 + if (private_key) curl_easy_setopt(curl, CURLOPT_SSLKEY, private_key);
180 + if (public_key) curl_easy_setopt(curl, CURLOPT_SSLCERT, public_key);
181 +
182 + if (strcmp(ca_cert, "all") != 0) {
183 + curl_easy_setopt(curl, CURLOPT_CAINFO, ca_cert);
184 + } else {
185 + curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
186 + }
187 + }
188 + // curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); // Remove for less verbose output
189 +
190 + return curl;
191 +}
192 +
193 +static void journal_remote_complete_event(BUFFER *msg, usec_t *monotonic_ut) {
194 + usec_t ut = now_monotonic_usec();
195 +
196 + if(monotonic_ut)
197 + *monotonic_ut = ut;
198 +
199 + buffer_sprintf(msg,
200 + ""
201 + "__REALTIME_TIMESTAMP=%llu\n"
202 + "__MONOTONIC_TIMESTAMP=%llu\n"
203 + "_BOOT_ID=%s\n"
204 + "_HOSTNAME=%s\n"
205 + "\n"
206 + , now_realtime_usec()
207 + , ut
208 + , global_boot_id
209 + , global_hostname
210 + );
211 +}
212 +
213 +static CURLcode journal_remote_send_buffer(CURL* curl, BUFFER *msg) {
214 +
215 + // log_message_to_stderr(msg);
216 +
217 + struct upload_data upload = {0};
218 +
219 + if (!curl || !buffer_strlen(msg))
220 + return CURLE_FAILED_INIT;
221 +
222 + upload.data = (char *) buffer_tostring(msg);
223 + upload.length = buffer_strlen(msg);
224 +
225 + curl_easy_setopt(curl, CURLOPT_READDATA, &upload);
226 + curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, (curl_off_t)upload.length);
227 +
228 + return curl_easy_perform(curl);
229 +}
230 +
231 +typedef enum {
232 + LOG_TO_JOURNAL_REMOTE_BAD_PARAMS = -1,
233 + LOG_TO_JOURNAL_REMOTE_CANNOT_INITIALIZE = -2,
234 + LOG_TO_JOURNAL_REMOTE_CANNOT_SEND = -3,
235 + LOG_TO_JOURNAL_REMOTE_CANNOT_READ = -4,
236 +} log_to_journal_remote_ret_t;
237 +
238 +static log_to_journal_remote_ret_t log_input_to_journal_remote(const char *url, const char *key, const char *cert, const char *trust, const char *newline, int timeout_ms) {
239 + if(!url || !*url) {
240 + fprintf(stderr, "No URL is given.\n");
241 + return LOG_TO_JOURNAL_REMOTE_BAD_PARAMS;
242 + }
243 +
244 + if(timeout_ms < 10)
245 + timeout_ms = 10;
246 +
247 + global_boot_id[0] = '\0';
248 + char boot_id[1024];
249 + if(read_file(BOOT_ID_PATH, boot_id, sizeof(boot_id)) == 0) {
250 + uuid_t uuid;
251 + if(uuid_parse_flexi(boot_id, uuid) == 0)
252 + uuid_unparse_lower_compact(uuid, global_boot_id);
253 + else
254 + fprintf(stderr, "WARNING: cannot parse the UUID found in '%s'.\n", BOOT_ID_PATH);
255 + }
256 +
257 + if(global_boot_id[0] == '\0') {
258 + fprintf(stderr, "WARNING: cannot read '%s'. Will generate a random _BOOT_ID.\n", BOOT_ID_PATH);
259 + uuid_t uuid;
260 + uuid_generate_random(uuid);
261 + uuid_unparse_lower_compact(uuid, global_boot_id);
262 + }
263 +
264 + if(global_hostname[0] == '\0') {
265 + if(gethostname(global_hostname, sizeof(global_hostname)) != 0) {
266 + fprintf(stderr, "WARNING: cannot get system's hostname. Will use internal default.\n");
267 + snprintfz(global_hostname, sizeof(global_hostname), "systemd-cat-native-unknown-hostname");
268 + }
269 + }
270 +
271 + if(!key)
272 + key = DEFAULT_PRIVATE_KEY;
273 +
274 + if(!cert)
275 + cert = DEFAULT_PUBLIC_KEY;
276 +
277 + if(!trust)
278 + trust = DEFAULT_CA_CERT;
279 +
280 + char full_url[4096];
281 + snprintfz(full_url, sizeof(full_url), "%s/upload", url);
282 +
283 + CURL *curl;
284 + CURLcode res = CURLE_OK;
285 + struct curl_slist *headers = NULL;
286 +
287 + curl_global_init(CURL_GLOBAL_ALL);
288 + curl = initialize_connection_to_systemd_journal_remote(full_url, key, cert, trust, &headers);
289 +
290 + if(!curl)
291 + return LOG_TO_JOURNAL_REMOTE_CANNOT_INITIALIZE;
292 +
293 + struct buffered_reader reader;
294 + buffered_reader_init(&reader);
295 + CLEAN_BUFFER *line = buffer_create(sizeof(reader.read_buffer), NULL);
296 + CLEAN_BUFFER *msg = buffer_create(sizeof(reader.read_buffer), NULL);
297 +
298 + size_t msg_full_events = 0;
299 + size_t msg_partial_fields = 0;
300 + usec_t msg_started_ut = 0;
301 + size_t failures = 0;
302 + size_t messages_logged = 0;
303 +
304 + log_to_journal_remote_ret_t ret = 0;
305 +
306 + while(true) {
307 + buffered_reader_ret_t rc = get_next_line(&reader, line, timeout_ms);
308 + if(rc == BUFFERED_READER_READ_POLL_TIMEOUT) {
309 + if(msg_full_events && !msg_partial_fields) {
310 + res = journal_remote_send_buffer(curl, msg);
311 + if(res != CURLE_OK) {
312 + fprintf(stderr, "journal_remote_send_buffer() failed: %s\n", curl_easy_strerror(res));
313 + failures++;
314 + ret = LOG_TO_JOURNAL_REMOTE_CANNOT_SEND;
315 + goto cleanup;
316 + }
317 + else
318 + messages_logged++;
319 +
320 + msg_full_events = 0;
321 + buffer_flush(msg);
322 + }
323 + }
324 + else if(rc == BUFFERED_READER_READ_OK) {
325 + if(!line->len) {
326 + // an empty line - we are done for this message
327 + if(msg_partial_fields) {
328 + msg_partial_fields = 0;
329 +
330 + usec_t ut;
331 + journal_remote_complete_event(msg, &ut);
332 + if(!msg_full_events)
333 + msg_started_ut = ut;
334 +
335 + msg_full_events++;
336 +
337 + if(ut - msg_started_ut >= USEC_PER_SEC / 2) {
338 + res = journal_remote_send_buffer(curl, msg);
339 + if(res != CURLE_OK) {
340 + fprintf(stderr, "journal_remote_send_buffer() failed: %s\n", curl_easy_strerror(res));
341 + failures++;
342 + ret = LOG_TO_JOURNAL_REMOTE_CANNOT_SEND;
343 + goto cleanup;
344 + }
345 + else
346 + messages_logged++;
347 +
348 + msg_full_events = 0;
349 + buffer_flush(msg);
350 + }
351 + }
352 + }
353 + else {
354 + buffer_memcat_replacing_newlines(msg, line->buffer, line->len, newline);
355 + msg_partial_fields++;
356 + }
357 +
358 + buffer_flush(line);
359 + }
360 + else {
361 + fprintf(stderr, "cannot read input data, failed with code %d\n", rc);
362 + ret = LOG_TO_JOURNAL_REMOTE_CANNOT_READ;
363 + break;
364 + }
365 + }
366 +
367 + if (msg_full_events || msg_partial_fields) {
368 + if(msg_partial_fields) {
369 + msg_partial_fields = 0;
370 + msg_full_events++;
371 + journal_remote_complete_event(msg, NULL);
372 + }
373 +
374 + if(msg_full_events) {
375 + res = journal_remote_send_buffer(curl, msg);
376 + if(res != CURLE_OK) {
377 + fprintf(stderr, "journal_remote_send_buffer() failed: %s\n", curl_easy_strerror(res));
378 + failures++;
379 + }
380 + else
381 + messages_logged++;
382 +
383 + msg_full_events = 0;
384 + buffer_flush(msg);
385 + }
386 + }
387 +
388 +cleanup:
389 + curl_easy_cleanup(curl);
390 + curl_slist_free_all(headers);
391 + curl_global_cleanup();
392 +
393 + return ret;
394 +}
395 +
396 +#endif
397 +
398 +static int help(void) {
399 + fprintf(stderr,
400 + "\n"
401 + "Netdata systemd-cat-native " PACKAGE_VERSION "\n"
402 + "\n"
403 + "This program reads from its standard input, lines in the format:\n"
404 + "\n"
405 + "KEY1=VALUE1\\n\n"
406 + "KEY2=VALUE2\\n\n"
407 + "KEYN=VALUEN\\n\n"
408 + "\\n\n"
409 + "\n"
410 + "and sends them to systemd-journal.\n"
411 + "\n"
412 + " - Binary journal fields are not accepted at its input\n"
413 + " - Binary journal fields can be generated after newline processing\n"
414 + " - Messages have to be separated by an empty line\n"
415 + " - Keys starting with underscore are not accepted (by journald)\n"
416 + " - Other rules imposed by systemd-journald are imposed (by journald)\n"
417 + "\n"
418 + "Usage:\n"
419 + "\n"
420 + " %s\n"
421 + " [--newline=STRING]\n"
422 + " [--log-as-netdata|-N]\n"
423 + " [--namespace=NAMESPACE] [--socket=PATH]\n"
424 +#ifdef HAVE_CURL
425 + " [--url=URL [--key=FILENAME] [--cert=FILENAME] [--trust=FILENAME|all]]\n"
426 +#endif
427 + "\n"
428 + "The program has the following modes of logging:\n"
429 + "\n"
430 + " * Log to a local systemd-journald or stderr\n"
431 + "\n"
432 + " This is the default mode. If systemd-journald is available, logs will be\n"
433 + " sent to systemd, otherwise logs will be printed on stderr, using logfmt\n"
434 + " formatting. Options --socket and --namespace are available to configure\n"
435 + " the journal destination:\n"
436 + "\n"
437 + " --socket=PATH\n"
438 + " The path of a systemd-journald UNIX socket.\n"
439 + " The program will use the default systemd-journald socket when this\n"
440 + " option is not used.\n"
441 + "\n"
442 + " --namespace=NAMESPACE\n"
443 + " The name of a configured and running systemd-journald namespace.\n"
444 + " The program will produce the socket path based on its internal\n"
445 + " defaults, to send the messages to the systemd journal namespace.\n"
446 + "\n"
447 + " * Log as Netdata, enabled with --log-as-netdata or -N\n"
448 + "\n"
449 + " In this mode the program uses environment variables set by Netdata for\n"
450 + " the log destination. Only log fields defined by Netdata are accepted.\n"
451 + " If the environment variables expected by Netdata are not found, it\n"
452 + " falls back to stderr logging in logfmt format.\n"
453 +#ifdef HAVE_CURL
454 + "\n"
455 + " * Log to a systemd-journal-remote TCP socket, enabled with --url=URL\n"
456 + "\n"
457 + " In this mode, the program will directly sent logs to a remote systemd\n"
458 + " journal (systemd-journal-remote expected at the destination)\n"
459 + " This mode is available even when the local system does not support\n"
460 + " systemd, or even it is not Linux, allowing a remote Linux systemd\n"
461 + " journald to become the logs database of the local system.\n"
462 + "\n"
463 + " Unfortunately systemd-journal-remote does not accept compressed\n"
464 + " data over the network, so the stream will be uncompressed.\n"
465 + "\n"
466 + " --url=URL\n"
467 + " The destination systemd-journal-remote address and port, similarly\n"
468 + " to what /etc/systemd/journal-upload.conf accepts.\n"
469 + " Usually it is in the form: https://ip.address:19532\n"
470 + " Both http and https URLs are accepted. When using https, the\n"
471 + " following additional options are accepted:\n"
472 + "\n"
473 + " --key=FILENAME\n"
474 + " The filename of the private key of the server.\n"
475 + " The default is: " DEFAULT_PRIVATE_KEY "\n"
476 + "\n"
477 + " --cert=FILENAME\n"
478 + " The filename of the public key of the server.\n"
479 + " The default is: " DEFAULT_PUBLIC_KEY "\n"
480 + "\n"
481 + " --trust=FILENAME | all\n"
482 + " The filename of the trusted CA public key.\n"
483 + " The default is: " DEFAULT_CA_CERT "\n"
484 + " The keyword 'all' can be used to trust all CAs.\n"
485 + "\n"
486 + " --keep-trying\n"
487 + " Keep trying to send the message, if the remote journal is not there.\n"
488 +#endif
489 + "\n"
490 + " NEWLINES PROCESSING\n"
491 + " systemd-journal logs entries may have newlines in them. However the\n"
492 + " Journal Export Format uses binary formatted data to achieve this,\n"
493 + " making it hard for text processing.\n"
494 + "\n"
495 + " To overcome this limitation, this program allows single-line text\n"
496 + " formatted values at its input, to be binary formatted multi-line Journal\n"
497 + " Export Format at its output.\n"
498 + "\n"
499 + " To achieve that it allows replacing a given string to a newline.\n"
500 + " The parameter --newline=STRING allows setting the string to be replaced\n"
501 + " with newlines.\n"
502 + "\n"
503 + " For example by setting --newline='{NEWLINE}', the program will replace\n"
504 + " all occurrences of {NEWLINE} with the newline character, within each\n"
505 + " VALUE of the KEY=VALUE lines. Once this this done, the program will\n"
506 + " switch the field to the binary Journal Export Format before sending the\n"
507 + " log event to systemd-journal.\n"
508 + "\n",
509 + program_name);
510 +
511 + return 1;
512 +}
513 +
514 +// ----------------------------------------------------------------------------
515 +// log as Netdata
516 +
517 +static void lgs_reset(struct log_stack_entry *lgs) {
518 + for(size_t i = 1; i < _NDF_MAX ;i++) {
519 + if(lgs[i].type == NDFT_TXT && lgs[i].set && lgs[i].txt)
520 + freez((void *)lgs[i].txt);
521 +
522 + lgs[i] = ND_LOG_FIELD_TXT(i, NULL);
523 + }
524 +
525 + lgs[0] = ND_LOG_FIELD_TXT(NDF_MESSAGE, NULL);
526 + lgs[_NDF_MAX] = ND_LOG_FIELD_END();
527 +}
528 +
529 +static const char *strdupz_replacing_newlines(const char *src, const char *newline) {
530 + if(!src) src = "";
531 +
532 + size_t src_len = strlen(src);
533 + char *buffer = mallocz(src_len + 1);
534 + copy_replacing_newlines(buffer, src_len + 1, src, src_len, newline);
535 + return buffer;
536 +}
537 +
538 +static int log_input_as_netdata(const char *newline, int timeout_ms) {
539 + struct buffered_reader reader;
540 + buffered_reader_init(&reader);
541 + CLEAN_BUFFER *line = buffer_create(sizeof(reader.read_buffer), NULL);
542 +
543 + ND_LOG_STACK lgs[_NDF_MAX + 1] = { 0 };
544 + ND_LOG_STACK_PUSH(lgs);
545 + lgs_reset(lgs);
546 +
547 + size_t fields_added = 0;
548 + size_t messages_logged = 0;
549 + ND_LOG_FIELD_PRIORITY priority = NDLP_INFO;
550 +
551 + while(get_next_line(&reader, line, timeout_ms) == BUFFERED_READER_READ_OK) {
552 + if(!line->len) {
553 + // an empty line - we are done for this message
554 +
555 + nd_log(NDLS_HEALTH, priority,
556 + "added %d fields", // if the user supplied a MESSAGE, this will be ignored
557 + fields_added);
558 +
559 + lgs_reset(lgs);
560 + fields_added = 0;
561 + messages_logged++;
562 + }
563 + else {
564 + char *equal = strchr(line->buffer, '=');
565 + if(equal) {
566 + const char *field = line->buffer;
567 + size_t field_len = equal - line->buffer;
568 + ND_LOG_FIELD_ID id = nd_log_field_id_by_name(field, field_len);
569 + if(id != NDF_STOP) {
570 + const char *value = ++equal;
571 +
572 + if(lgs[id].txt)
573 + freez((void *) lgs[id].txt);
574 +
575 + lgs[id].txt = strdupz_replacing_newlines(value, newline);
576 + lgs[id].set = true;
577 +
578 + fields_added++;
579 +
580 + if(id == NDF_PRIORITY)
581 + priority = nd_log_priority2id(value);
582 + }
583 + else {
584 + struct log_stack_entry backup = lgs[NDF_MESSAGE];
585 + lgs[NDF_MESSAGE] = ND_LOG_FIELD_TXT(NDF_MESSAGE, NULL);
586 +
587 + nd_log(NDLS_COLLECTORS, NDLP_ERR,
588 + "Field '%.*s' is not a Netdata field. Ignoring it.",
589 + field_len, field);
590 +
591 + lgs[NDF_MESSAGE] = backup;
592 + }
593 + }
594 + else {
595 + struct log_stack_entry backup = lgs[NDF_MESSAGE];
596 + lgs[NDF_MESSAGE] = ND_LOG_FIELD_TXT(NDF_MESSAGE, NULL);
597 +
598 + nd_log(NDLS_COLLECTORS, NDLP_ERR,
599 + "Line does not contain an = sign; ignoring it: %s",
600 + line->buffer);
601 +
602 + lgs[NDF_MESSAGE] = backup;
603 + }
604 + }
605 +
606 + buffer_flush(line);
607 + }
608 +
609 + if(fields_added) {
610 + nd_log(NDLS_HEALTH, priority, "added %d fields", fields_added);
611 + messages_logged++;
612 + }
613 +
614 + return messages_logged ? 0 : 1;
615 +}
616 +
617 +// ----------------------------------------------------------------------------
618 +// log to a local systemd-journald
619 +
620 +static bool journal_local_send_buffer(int fd, BUFFER *msg) {
621 + // log_message_to_stderr(msg);
622 +
623 + bool ret = journal_direct_send(fd, msg->buffer, msg->len);
624 + if (!ret)
625 + fprintf(stderr, "Cannot send message to systemd journal.\n");
626 +
627 + return ret;
628 +}
629 +
630 +static int log_input_to_journal(const char *socket, const char *namespace, const char *newline, int timeout_ms) {
631 + char path[FILENAME_MAX + 1];
632 + int fd = -1;
633 +
634 + if(socket)
635 + snprintfz(path, sizeof(path), "%s", socket);
636 + else
637 + journal_construct_path(path, sizeof(path), NULL, namespace);
638 +
639 + fd = journal_direct_fd(path);
640 + if (fd == -1) {
641 + fprintf(stderr, "Cannot open '%s' as a UNIX socket (errno = %d)\n",
642 + path, errno);
643 + return 1;
644 + }
645 +
646 + struct buffered_reader reader;
647 + buffered_reader_init(&reader);
648 + CLEAN_BUFFER *line = buffer_create(sizeof(reader.read_buffer), NULL);
649 + CLEAN_BUFFER *msg = buffer_create(sizeof(reader.read_buffer), NULL);
650 +
651 + size_t messages_logged = 0;
652 + size_t failed_messages = 0;
653 +
654 + while(get_next_line(&reader, line, timeout_ms) == BUFFERED_READER_READ_OK) {
655 + if (!line->len) {
656 + // an empty line - we are done for this message
657 + if (msg->len) {
658 + if(journal_local_send_buffer(fd, msg))
659 + messages_logged++;
660 + else {
661 + failed_messages++;
662 + goto cleanup;
663 + }
664 + }
665 +
666 + buffer_flush(msg);
667 + }
668 + else
669 + buffer_memcat_replacing_newlines(msg, line->buffer, line->len, newline);
670 +
671 + buffer_flush(line);
672 + }
673 +
674 + if (msg && msg->len) {
675 + if(journal_local_send_buffer(fd, msg))
676 + messages_logged++;
677 + else
678 + failed_messages++;
679 + }
680 +
681 +cleanup:
682 + return !failed_messages && messages_logged ? 0 : 1;
683 +}
684 +
685 +int main(int argc, char *argv[]) {
686 + clocks_init();
687 + nd_log_initialize_for_external_plugins(argv[0]);
688 +
689 + int timeout_ms = -1; // wait forever
690 + bool log_as_netdata = false;
691 + const char *newline = NULL;
692 + const char *namespace = NULL;
693 + const char *socket = getenv("NETDATA_SYSTEMD_JOURNAL_PATH");
694 +#ifdef HAVE_CURL
695 + const char *url = NULL;
696 + const char *key = NULL;
697 + const char *cert = NULL;
698 + const char *trust = NULL;
699 + bool keep_trying = false;
700 +#endif
701 +
702 + for(int i = 1; i < argc ;i++) {
703 + const char *k = argv[i];
704 +
705 + if(strcmp(k, "--help") == 0 || strcmp(k, "-h") == 0)
706 + return help();
707 +
708 + else if(strcmp(k, "--log-as-netdata") == 0 || strcmp(k, "-N") == 0)
709 + log_as_netdata = true;
710 +
711 + else if(strncmp(k, "--namespace=", 12) == 0)
712 + namespace = &k[12];
713 +
714 + else if(strncmp(k, "--socket=", 9) == 0)
715 + socket = &k[9];
716 +
717 + else if(strncmp(k, "--newline=", 10) == 0)
718 + newline = &k[10];
719 +
720 +#ifdef HAVE_CURL
721 + else if (strncmp(k, "--url=", 6) == 0)
722 + url = &k[6];
723 +
724 + else if (strncmp(k, "--key=", 6) == 0)
725 + key = &k[6];
726 +
727 + else if (strncmp(k, "--cert=", 7) == 0)
728 + cert = &k[7];
729 +
730 + else if (strncmp(k, "--trust=", 8) == 0)
731 + trust = &k[8];
732 +
733 + else if (strcmp(k, "--keep-trying") == 0)
734 + keep_trying = true;
735 +#endif
736 + else {
737 + fprintf(stderr, "Unknown parameter '%s'\n", k);
738 + return 1;
739 + }
740 + }
741 +
742 +#ifdef HAVE_CURL
743 + if(log_as_netdata && url) {
744 + fprintf(stderr, "Cannot log to a systemd-journal-remote URL as Netdata. "
745 + "Please either give --url or --log-as-netdata, not both.\n");
746 + return 1;
747 + }
748 +
749 + if(socket && url) {
750 + fprintf(stderr, "Cannot log to a systemd-journal-remote URL using a UNIX socket. "
751 + "Please either give --url or --socket, not both.\n");
752 + return 1;
753 + }
754 +
755 + if(url && namespace) {
756 + fprintf(stderr, "Cannot log to a systemd-journal-remote URL using a namespace. "
757 + "Please either give --url or --namespace, not both.\n");
758 + return 1;
759 + }
760 +#endif
761 +
762 + if(log_as_netdata && namespace) {
763 + fprintf(stderr, "Cannot log as netdata using a namespace. "
764 + "Please either give --log-as-netdata or --namespace, not both.\n");
765 + return 1;
766 + }
767 +
768 + if(log_as_netdata)
769 + return log_input_as_netdata(newline, timeout_ms);
770 +
771 +#ifdef HAVE_CURL
772 + if(url) {
773 + log_to_journal_remote_ret_t rc;
774 + do {
775 + rc = log_input_to_journal_remote(url, key, cert, trust, newline, timeout_ms);
776 + } while(keep_trying && rc == LOG_TO_JOURNAL_REMOTE_CANNOT_SEND);
777 + }
778 +#endif
779 +
780 + return log_input_to_journal(socket, namespace, newline, timeout_ms);
781 +}
libnetdata/log/systemd-cat-native.h new
+8
@@ -0,0 +1,8 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
4 +
5 +#ifndef NETDATA_SYSTEMD_CAT_NATIVE_H
6 +#define NETDATA_SYSTEMD_CAT_NATIVE_H
7 +
8 +#endif //NETDATA_SYSTEMD_CAT_NATIVE_H
libnetdata/socket/security.c
+24 -17
@@ -24,7 +24,7 @@ static SOCKET_PEERS netdata_ssl_peers(NETDATA_SSL *ssl) {
24 }
25
26 static void netdata_ssl_log_error_queue(const char *call, NETDATA_SSL *ssl, unsigned long err) {
27 - error_limit_static_thread_var(erl, 1, 0);
27 + nd_log_limit_static_thread_var(erl, 1, 0);
28
29 if(err == SSL_ERROR_NONE)
30 err = ERR_get_error();
@@ -103,8 +103,9 @@ static void netdata_ssl_log_error_queue(const char *call, NETDATA_SSL *ssl, unsi
103 ERR_error_string_n(err, str, 1024);
104 str[1024] = '\0';
105 SOCKET_PEERS peers = netdata_ssl_peers(ssl);
106 - error_limit(&erl, "SSL: %s() on socket local [[%s]:%d] <-> remote [[%s]:%d], returned error %lu (%s): %s",
107 - call, peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, err, code, str);
106 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR,
107 + "SSL: %s() on socket local [[%s]:%d] <-> remote [[%s]:%d], returned error %lu (%s): %s",
108 + call, peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, err, code, str);
109
110 } while((err = ERR_get_error()));
111 }
@@ -179,7 +180,7 @@ void netdata_ssl_close(NETDATA_SSL *ssl) {
180 }
181
182 static inline bool is_handshake_complete(NETDATA_SSL *ssl, const char *op) {
182 - error_limit_static_thread_var(erl, 1, 0);
183 + nd_log_limit_static_thread_var(erl, 1, 0);
184
185 if(unlikely(!ssl->conn)) {
186 internal_error(true, "SSL: trying to %s on a NULL connection", op);
@@ -189,22 +190,25 @@ static inline bool is_handshake_complete(NETDATA_SSL *ssl, const char *op) {
190 switch(ssl->state) {
191 case NETDATA_SSL_STATE_NOT_SSL: {
192 SOCKET_PEERS peers = netdata_ssl_peers(ssl);
192 - error_limit(&erl, "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on non-SSL connection",
193 - peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
193 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_WARNING,
194 + "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on non-SSL connection",
195 + peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
196 return false;
197 }
198
199 case NETDATA_SSL_STATE_INIT: {
200 SOCKET_PEERS peers = netdata_ssl_peers(ssl);
199 - error_limit(&erl, "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on an incomplete connection",
200 - peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
201 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_WARNING,
202 + "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on an incomplete connection",
203 + peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
204 return false;
205 }
206
207 case NETDATA_SSL_STATE_FAILED: {
208 SOCKET_PEERS peers = netdata_ssl_peers(ssl);
206 - error_limit(&erl, "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on a failed connection",
207 - peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
209 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_WARNING,
210 + "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on a failed connection",
211 + peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
212 return false;
213 }
214
@@ -296,7 +300,7 @@ ssize_t netdata_ssl_write(NETDATA_SSL *ssl, const void *buf, size_t num) {
300 }
301
302 static inline bool is_handshake_initialized(NETDATA_SSL *ssl, const char *op) {
299 - error_limit_static_thread_var(erl, 1, 0);
303 + nd_log_limit_static_thread_var(erl, 1, 0);
304
305 if(unlikely(!ssl->conn)) {
306 internal_error(true, "SSL: trying to %s on a NULL connection", op);
@@ -306,8 +310,9 @@ static inline bool is_handshake_initialized(NETDATA_SSL *ssl, const char *op) {
310 switch(ssl->state) {
311 case NETDATA_SSL_STATE_NOT_SSL: {
312 SOCKET_PEERS peers = netdata_ssl_peers(ssl);
309 - error_limit(&erl, "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on non-SSL connection",
310 - peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
313 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_WARNING,
314 + "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on non-SSL connection",
315 + peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
316 return false;
317 }
318
@@ -317,15 +322,17 @@ static inline bool is_handshake_initialized(NETDATA_SSL *ssl, const char *op) {
322
323 case NETDATA_SSL_STATE_FAILED: {
324 SOCKET_PEERS peers = netdata_ssl_peers(ssl);
320 - error_limit(&erl, "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on a failed connection",
321 - peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
325 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_WARNING,
326 + "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on a failed connection",
327 + peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
328 return false;
329 }
330
331 case NETDATA_SSL_STATE_COMPLETE: {
332 SOCKET_PEERS peers = netdata_ssl_peers(ssl);
327 - error_limit(&erl, "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on an complete connection",
328 - peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
333 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_WARNING,
334 + "SSL: on socket local [[%s]:%d] <-> remote [[%s]:%d], attempt to %s on an complete connection",
335 + peers.local.ip, peers.local.port, peers.peer.ip, peers.peer.port, op);
336 return false;
337 }
338 }
libnetdata/socket/socket.c
+314 -210
@@ -158,7 +158,9 @@ int sock_setnonblock(int fd) {
158
159 int ret = fcntl(fd, F_SETFL, flags);
160 if(ret < 0)
161 - netdata_log_error("Failed to set O_NONBLOCK on socket %d", fd);
161 + nd_log(NDLS_DAEMON, NDLP_ERR,
162 + "Failed to set O_NONBLOCK on socket %d",
163 + fd);
164
165 return ret;
166 }
@@ -171,7 +173,9 @@ int sock_delnonblock(int fd) {
173
174 int ret = fcntl(fd, F_SETFL, flags);
175 if(ret < 0)
174 - netdata_log_error("Failed to remove O_NONBLOCK on socket %d", fd);
176 + nd_log(NDLS_DAEMON, NDLP_ERR,
177 + "Failed to remove O_NONBLOCK on socket %d",
178 + fd);
179
180 return ret;
181 }
@@ -180,7 +184,9 @@ int sock_setreuse(int fd, int reuse) {
184 int ret = setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse));
185
186 if(ret == -1)
183 - netdata_log_error("Failed to set SO_REUSEADDR on socket %d", fd);
187 + nd_log(NDLS_DAEMON, NDLP_ERR,
188 + "Failed to set SO_REUSEADDR on socket %d",
189 + fd);
190
191 return ret;
192 }
@@ -191,7 +197,9 @@ int sock_setreuse_port(int fd, int reuse) {
197 #ifdef SO_REUSEPORT
198 ret = setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &reuse, sizeof(reuse));
199 if(ret == -1 && errno != ENOPROTOOPT)
194 - netdata_log_error("failed to set SO_REUSEPORT on socket %d", fd);
200 + nd_log(NDLS_DAEMON, NDLP_ERR,
201 + "failed to set SO_REUSEPORT on socket %d",
202 + fd);
203 #else
204 ret = -1;
205 #endif
@@ -205,7 +213,9 @@ int sock_enlarge_in(int fd) {
213 ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &bs, sizeof(bs));
214
215 if(ret == -1)
208 - netdata_log_error("Failed to set SO_RCVBUF on socket %d", fd);
216 + nd_log(NDLS_DAEMON, NDLP_ERR,
217 + "Failed to set SO_RCVBUF on socket %d",
218 + fd);
219
220 return ret;
221 }
@@ -215,7 +225,9 @@ int sock_enlarge_out(int fd) {
225 ret = setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &bs, sizeof(bs));
226
227 if(ret == -1)
218 - netdata_log_error("Failed to set SO_SNDBUF on socket %d", fd);
228 + nd_log(NDLS_DAEMON, NDLP_ERR,
229 + "Failed to set SO_SNDBUF on socket %d",
230 + fd);
231
232 return ret;
233 }
@@ -250,11 +262,12 @@ char *strdup_client_description(int family, const char *protocol, const char *ip
262 int create_listen_socket_unix(const char *path, int listen_backlog) {
263 int sock;
264
253 - netdata_log_debug(D_LISTENER, "LISTENER: UNIX creating new listening socket on path '%s'", path);
254 -
265 sock = socket(AF_UNIX, SOCK_STREAM, 0);
266 if(sock < 0) {
257 - netdata_log_error("LISTENER: UNIX socket() on path '%s' failed.", path);
267 + nd_log(NDLS_DAEMON, NDLP_ERR,
268 + "LISTENER: UNIX socket() on path '%s' failed.",
269 + path);
270 +
271 return -1;
272 }
273
@@ -268,37 +281,47 @@ int create_listen_socket_unix(const char *path, int listen_backlog) {
281
282 errno = 0;
283 if (unlink(path) == -1 && errno != ENOENT)
271 - netdata_log_error("LISTENER: failed to remove existing (probably obsolete or left-over) file on UNIX socket path '%s'.", path);
284 + nd_log(NDLS_DAEMON, NDLP_ERR,
285 + "LISTENER: failed to remove existing (probably obsolete or left-over) file on UNIX socket path '%s'.",
286 + path);
287
288 if(bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) {
289 close(sock);
275 - netdata_log_error("LISTENER: UNIX bind() on path '%s' failed.", path);
290 + nd_log(NDLS_DAEMON, NDLP_ERR,
291 + "LISTENER: UNIX bind() on path '%s' failed.",
292 + path);
293 +
294 return -1;
295 }
296
297 // we have to chmod this to 0777 so that the client will be able
298 // to read from and write to this socket.
299 if(chmod(path, 0777) == -1)
282 - netdata_log_error("LISTENER: failed to chmod() socket file '%s'.", path);
300 + nd_log(NDLS_DAEMON, NDLP_ERR,
301 + "LISTENER: failed to chmod() socket file '%s'.",
302 + path);
303
304 if(listen(sock, listen_backlog) < 0) {
305 close(sock);
286 - netdata_log_error("LISTENER: UNIX listen() on path '%s' failed.", path);
306 + nd_log(NDLS_DAEMON, NDLP_ERR,
307 + "LISTENER: UNIX listen() on path '%s' failed.",
308 + path);
309 +
310 return -1;
311 }
312
290 - netdata_log_debug(D_LISTENER, "LISTENER: Listening on UNIX path '%s'", path);
313 return sock;
314 }
315
316 int create_listen_socket4(int socktype, const char *ip, uint16_t port, int listen_backlog) {
317 int sock;
318
297 - netdata_log_debug(D_LISTENER, "LISTENER: IPv4 creating new listening socket on ip '%s' port %d, socktype %d", ip, port, socktype);
298 -
319 sock = socket(AF_INET, socktype, 0);
320 if(sock < 0) {
301 - netdata_log_error("LISTENER: IPv4 socket() on ip '%s' port %d, socktype %d failed.", ip, port, socktype);
321 + nd_log(NDLS_DAEMON, NDLP_ERR,
322 + "LISTENER: IPv4 socket() on ip '%s' port %d, socktype %d failed.",
323 + ip, port, socktype);
324 +
325 return -1;
326 }
327
@@ -314,24 +337,36 @@ int create_listen_socket4(int socktype, const char *ip, uint16_t port, int liste
337
338 int ret = inet_pton(AF_INET, ip, (void *)&name.sin_addr.s_addr);
339 if(ret != 1) {
317 - netdata_log_error("LISTENER: Failed to convert IP '%s' to a valid IPv4 address.", ip);
340 + nd_log(NDLS_DAEMON, NDLP_ERR,
341 + "LISTENER: Failed to convert IP '%s' to a valid IPv4 address.",
342 + ip);
343 +
344 close(sock);
345 return -1;
346 }
347
348 if(bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) {
349 close(sock);
324 - netdata_log_error("LISTENER: IPv4 bind() on ip '%s' port %d, socktype %d failed.", ip, port, socktype);
350 + nd_log(NDLS_DAEMON, NDLP_ERR,
351 + "LISTENER: IPv4 bind() on ip '%s' port %d, socktype %d failed.",
352 + ip, port, socktype);
353 +
354 return -1;
355 }
356
357 if(socktype == SOCK_STREAM && listen(sock, listen_backlog) < 0) {
358 close(sock);
330 - netdata_log_error("LISTENER: IPv4 listen() on ip '%s' port %d, socktype %d failed.", ip, port, socktype);
359 + nd_log(NDLS_DAEMON, NDLP_ERR,
360 + "LISTENER: IPv4 listen() on ip '%s' port %d, socktype %d failed.",
361 + ip, port, socktype);
362 +
363 return -1;
364 }
365
334 - netdata_log_debug(D_LISTENER, "LISTENER: Listening on IPv4 ip '%s' port %d, socktype %d", ip, port, socktype);
366 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
367 + "LISTENER: Listening on IPv4 ip '%s' port %d, socktype %d",
368 + ip, port, socktype);
369 +
370 return sock;
371 }
372
@@ -339,11 +374,12 @@ int create_listen_socket6(int socktype, uint32_t scope_id, const char *ip, int p
374 int sock;
375 int ipv6only = 1;
376
342 - netdata_log_debug(D_LISTENER, "LISTENER: IPv6 creating new listening socket on ip '%s' port %d, socktype %d", ip, port, socktype);
343 -
377 sock = socket(AF_INET6, socktype, 0);
378 if (sock < 0) {
346 - netdata_log_error("LISTENER: IPv6 socket() on ip '%s' port %d, socktype %d, failed.", ip, port, socktype);
379 + nd_log(NDLS_DAEMON, NDLP_ERR,
380 + "LISTENER: IPv6 socket() on ip '%s' port %d, socktype %d, failed.",
381 + ip, port, socktype);
382 +
383 return -1;
384 }
385
@@ -354,7 +390,9 @@ int create_listen_socket6(int socktype, uint32_t scope_id, const char *ip, int p
390
391 /* IPv6 only */
392 if(setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, (void*)&ipv6only, sizeof(ipv6only)) != 0)
357 - netdata_log_error("LISTENER: Cannot set IPV6_V6ONLY on ip '%s' port %d, socktype %d.", ip, port, socktype);
393 + nd_log(NDLS_DAEMON, NDLP_ERR,
394 + "LISTENER: Cannot set IPV6_V6ONLY on ip '%s' port %d, socktype %d.",
395 + ip, port, socktype);
396
397 struct sockaddr_in6 name;
398 memset(&name, 0, sizeof(struct sockaddr_in6));
@@ -364,7 +402,10 @@ int create_listen_socket6(int socktype, uint32_t scope_id, const char *ip, int p
402
403 int ret = inet_pton(AF_INET6, ip, (void *)&name.sin6_addr.s6_addr);
404 if(ret != 1) {
367 - netdata_log_error("LISTENER: Failed to convert IP '%s' to a valid IPv6 address.", ip);
405 + nd_log(NDLS_DAEMON, NDLP_ERR,
406 + "LISTENER: Failed to convert IP '%s' to a valid IPv6 address.",
407 + ip);
408 +
409 close(sock);
410 return -1;
411 }
@@ -373,23 +414,35 @@ int create_listen_socket6(int socktype, uint32_t scope_id, const char *ip, int p
414
415 if (bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) {
416 close(sock);
376 - netdata_log_error("LISTENER: IPv6 bind() on ip '%s' port %d, socktype %d failed.", ip, port, socktype);
417 + nd_log(NDLS_DAEMON, NDLP_ERR,
418 + "LISTENER: IPv6 bind() on ip '%s' port %d, socktype %d failed.",
419 + ip, port, socktype);
420 +
421 return -1;
422 }
423
424 if (socktype == SOCK_STREAM && listen(sock, listen_backlog) < 0) {
425 close(sock);
382 - netdata_log_error("LISTENER: IPv6 listen() on ip '%s' port %d, socktype %d failed.", ip, port, socktype);
426 + nd_log(NDLS_DAEMON, NDLP_ERR,
427 + "LISTENER: IPv6 listen() on ip '%s' port %d, socktype %d failed.",
428 + ip, port, socktype);
429 +
430 return -1;
431 }
432
386 - netdata_log_debug(D_LISTENER, "LISTENER: Listening on IPv6 ip '%s' port %d, socktype %d", ip, port, socktype);
433 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
434 + "LISTENER: Listening on IPv6 ip '%s' port %d, socktype %d",
435 + ip, port, socktype);
436 +
437 return sock;
438 }
439
440 static inline int listen_sockets_add(LISTEN_SOCKETS *sockets, int fd, int family, int socktype, const char *protocol, const char *ip, uint16_t port, int acl_flags) {
441 if(sockets->opened >= MAX_LISTEN_FDS) {
392 - netdata_log_error("LISTENER: Too many listening sockets. Failed to add listening %s socket at ip '%s' port %d, protocol %s, socktype %d", protocol, ip, port, protocol, socktype);
442 + nd_log(NDLS_DAEMON, NDLP_ERR,
443 + "LISTENER: Too many listening sockets. Failed to add listening %s socket at ip '%s' port %d, protocol %s, socktype %d",
444 + protocol, ip, port, protocol, socktype);
445 +
446 close(fd);
447 return -1;
448 }
@@ -519,7 +572,10 @@ static inline int bind_to_this(LISTEN_SOCKETS *sockets, const char *definition,
572 protocol_str = "unix";
573 int fd = create_listen_socket_unix(path, listen_backlog);
574 if (fd == -1) {
522 - netdata_log_error("LISTENER: Cannot create unix socket '%s'", path);
575 + nd_log(NDLS_DAEMON, NDLP_ERR,
576 + "LISTENER: Cannot create unix socket '%s'",
577 + path);
578 +
579 sockets->failed++;
580 } else {
581 acl_flags = WEB_CLIENT_ACL_DASHBOARD | WEB_CLIENT_ACL_REGISTRY | WEB_CLIENT_ACL_BADGE | WEB_CLIENT_ACL_MGMT | WEB_CLIENT_ACL_NETDATACONF | WEB_CLIENT_ACL_STREAMING | WEB_CLIENT_ACL_SSL_DEFAULT;
@@ -585,7 +641,10 @@ static inline int bind_to_this(LISTEN_SOCKETS *sockets, const char *definition,
641 if(*interface) {
642 scope_id = if_nametoindex(interface);
643 if(!scope_id)
588 - netdata_log_error("LISTENER: Cannot find a network interface named '%s'. Continuing with limiting the network interface", interface);
644 + nd_log(NDLS_DAEMON, NDLP_ERR,
645 + "LISTENER: Cannot find a network interface named '%s'. "
646 + "Continuing with limiting the network interface",
647 + interface);
648 }
649
650 if(!*ip || *ip == '*' || !strcmp(ip, "any") || !strcmp(ip, "all"))
@@ -605,7 +664,10 @@ static inline int bind_to_this(LISTEN_SOCKETS *sockets, const char *definition,
664
665 int r = getaddrinfo(ip, port, &hints, &result);
666 if (r != 0) {
608 - netdata_log_error("LISTENER: getaddrinfo('%s', '%s'): %s\n", ip, port, gai_strerror(r));
667 + nd_log(NDLS_DAEMON, NDLP_ERR,
668 + "LISTENER: getaddrinfo('%s', '%s'): %s\n",
669 + ip, port, gai_strerror(r));
670 +
671 return -1;
672 }
673
@@ -622,7 +684,6 @@ static inline int bind_to_this(LISTEN_SOCKETS *sockets, const char *definition,
684 struct sockaddr_in *sin = (struct sockaddr_in *) rp->ai_addr;
685 inet_ntop(AF_INET, &sin->sin_addr, rip, INET_ADDRSTRLEN);
686 rport = ntohs(sin->sin_port);
625 - // netdata_log_info("Attempting to listen on IPv4 '%s' ('%s'), port %d ('%s'), socktype %d", rip, ip, rport, port, socktype);
687 fd = create_listen_socket4(socktype, rip, rport, listen_backlog);
688 break;
689 }
@@ -631,18 +692,23 @@ static inline int bind_to_this(LISTEN_SOCKETS *sockets, const char *definition,
692 struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *) rp->ai_addr;
693 inet_ntop(AF_INET6, &sin6->sin6_addr, rip, INET6_ADDRSTRLEN);
694 rport = ntohs(sin6->sin6_port);
634 - // netdata_log_info("Attempting to listen on IPv6 '%s' ('%s'), port %d ('%s'), socktype %d", rip, ip, rport, port, socktype);
695 fd = create_listen_socket6(socktype, scope_id, rip, rport, listen_backlog);
696 break;
697 }
698
699 default:
640 - netdata_log_debug(D_LISTENER, "LISTENER: Unknown socket family %d", family);
700 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
701 + "LISTENER: Unknown socket family %d",
702 + family);
703 +
704 break;
705 }
706
707 if (fd == -1) {
645 - netdata_log_error("LISTENER: Cannot bind to ip '%s', port %d", rip, rport);
708 + nd_log(NDLS_DAEMON, NDLP_ERR,
709 + "LISTENER: Cannot bind to ip '%s', port %d",
710 + rip, rport);
711 +
712 sockets->failed++;
713 }
714 else {
@@ -664,13 +730,14 @@ int listen_sockets_setup(LISTEN_SOCKETS *sockets) {
730 long long int old_port = sockets->default_port;
731 long long int new_port = appconfig_get_number(sockets->config, sockets->config_section, "default port", sockets->default_port);
732 if(new_port < 1 || new_port > 65535) {
667 - netdata_log_error("LISTENER: Invalid listen port %lld given. Defaulting to %lld.", new_port, old_port);
733 + nd_log(NDLS_DAEMON, NDLP_ERR,
734 + "LISTENER: Invalid listen port %lld given. Defaulting to %lld.",
735 + new_port, old_port);
736 +
737 sockets->default_port = (uint16_t) appconfig_set_number(sockets->config, sockets->config_section, "default port", old_port);
738 }
739 else sockets->default_port = (uint16_t)new_port;
740
672 - netdata_log_debug(D_OPTIONS, "LISTENER: Default listen port set to %d.", sockets->default_port);
673 -
741 char *s = appconfig_get(sockets->config, sockets->config_section, "bind to", sockets->default_bind_to);
742 while(*s) {
743 char *e = s;
@@ -694,7 +761,9 @@ int listen_sockets_setup(LISTEN_SOCKETS *sockets) {
761 if(sockets->failed) {
762 size_t i;
763 for(i = 0; i < sockets->opened ;i++)
697 - netdata_log_info("LISTENER: Listen socket %s opened successfully.", sockets->fds_names[i]);
764 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
765 + "LISTENER: Listen socket %s opened successfully.",
766 + sockets->fds_names[i]);
767 }
768
769 return (int)sockets->opened;
@@ -711,13 +780,18 @@ int listen_sockets_setup(LISTEN_SOCKETS *sockets) {
780 static inline int connect_to_unix(const char *path, struct timeval *timeout) {
781 int fd = socket(AF_UNIX, SOCK_STREAM, 0);
782 if(fd == -1) {
714 - netdata_log_error("Failed to create UNIX socket() for '%s'", path);
783 + nd_log(NDLS_DAEMON, NDLP_ERR,
784 + "Failed to create UNIX socket() for '%s'",
785 + path);
786 +
787 return -1;
788 }
789
790 if(timeout) {
791 if(setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, (char *) timeout, sizeof(struct timeval)) < 0)
720 - netdata_log_error("Failed to set timeout on UNIX socket '%s'", path);
792 + nd_log(NDLS_DAEMON, NDLP_ERR,
793 + "Failed to set timeout on UNIX socket '%s'",
794 + path);
795 }
796
797 struct sockaddr_un addr;
@@ -726,12 +800,17 @@ static inline int connect_to_unix(const char *path, struct timeval *timeout) {
800 strncpy(addr.sun_path, path, sizeof(addr.sun_path)-1);
801
802 if (connect(fd, (struct sockaddr*)&addr, sizeof(addr)) == -1) {
729 - netdata_log_error("Cannot connect to UNIX socket on path '%s'.", path);
803 + nd_log(NDLS_DAEMON, NDLP_ERR,
804 + "Cannot connect to UNIX socket on path '%s'.",
805 + path);
806 +
807 close(fd);
808 return -1;
809 }
810
734 - netdata_log_debug(D_CONNECT_TO, "Connected to UNIX socket on path '%s'.", path);
811 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
812 + "Connected to UNIX socket on path '%s'.",
813 + path);
814
815 return fd;
816 }
@@ -757,10 +836,24 @@ int connect_to_this_ip46(int protocol, int socktype, const char *host, uint32_t
836
837 int ai_err = getaddrinfo(host, service, &hints, &ai_head);
838 if (ai_err != 0) {
760 - netdata_log_error("Cannot resolve host '%s', port '%s': %s", host, service, gai_strerror(ai_err));
839 +
840 + nd_log(NDLS_DAEMON, NDLP_ERR,
841 + "Cannot resolve host '%s', port '%s': %s",
842 + host, service, gai_strerror(ai_err));
843 +
844 return -1;
845 }
846
847 + char hostBfr[NI_MAXHOST + 1];
848 + char servBfr[NI_MAXSERV + 1];
849 +
850 + ND_LOG_STACK lgs[] = {
851 + ND_LOG_FIELD_TXT(NDF_DST_IP, hostBfr),
852 + ND_LOG_FIELD_TXT(NDF_DST_PORT, servBfr),
853 + ND_LOG_FIELD_END(),
854 + };
855 + ND_LOG_STACK_PUSH(lgs);
856 +
857 int fd = -1;
858 for (ai = ai_head; ai != NULL && fd == -1; ai = ai->ai_next) {
859
@@ -771,9 +864,6 @@ int connect_to_this_ip46(int protocol, int socktype, const char *host, uint32_t
864 }
865 }
866
774 - char hostBfr[NI_MAXHOST + 1];
775 - char servBfr[NI_MAXSERV + 1];
776 -
867 getnameinfo(ai->ai_addr,
868 ai->ai_addrlen,
869 hostBfr,
@@ -782,54 +872,21 @@ int connect_to_this_ip46(int protocol, int socktype, const char *host, uint32_t
872 sizeof(servBfr),
873 NI_NUMERICHOST | NI_NUMERICSERV);
874
785 - netdata_log_debug(D_CONNECT_TO, "Address info: host = '%s', service = '%s', ai_flags = 0x%02X, ai_family = %d (PF_INET = %d, PF_INET6 = %d), ai_socktype = %d (SOCK_STREAM = %d, SOCK_DGRAM = %d), ai_protocol = %d (IPPROTO_TCP = %d, IPPROTO_UDP = %d), ai_addrlen = %lu (sockaddr_in = %lu, sockaddr_in6 = %lu)",
786 - hostBfr,
787 - servBfr,
788 - (unsigned int)ai->ai_flags,
789 - ai->ai_family,
790 - PF_INET,
791 - PF_INET6,
792 - ai->ai_socktype,
793 - SOCK_STREAM,
794 - SOCK_DGRAM,
795 - ai->ai_protocol,
796 - IPPROTO_TCP,
797 - IPPROTO_UDP,
798 - (unsigned long)ai->ai_addrlen,
799 - (unsigned long)sizeof(struct sockaddr_in),
800 - (unsigned long)sizeof(struct sockaddr_in6));
801 -
875 switch (ai->ai_addr->sa_family) {
876 case PF_INET: {
877 struct sockaddr_in *pSadrIn = (struct sockaddr_in *)ai->ai_addr;
878 (void)pSadrIn;
806 -
807 - netdata_log_debug(D_CONNECT_TO, "ai_addr = sin_family: %d (AF_INET = %d, AF_INET6 = %d), sin_addr: '%s', sin_port: '%s'",
808 - pSadrIn->sin_family,
809 - AF_INET,
810 - AF_INET6,
811 - hostBfr,
812 - servBfr);
879 break;
880 }
881
882 case PF_INET6: {
883 struct sockaddr_in6 *pSadrIn6 = (struct sockaddr_in6 *) ai->ai_addr;
884 (void)pSadrIn6;
819 -
820 - netdata_log_debug(D_CONNECT_TO,"ai_addr = sin6_family: %d (AF_INET = %d, AF_INET6 = %d), sin6_addr: '%s', sin6_port: '%s', sin6_flowinfo: %u, sin6_scope_id: %u",
821 - pSadrIn6->sin6_family,
822 - AF_INET,
823 - AF_INET6,
824 - hostBfr,
825 - servBfr,
826 - pSadrIn6->sin6_flowinfo,
827 - pSadrIn6->sin6_scope_id);
885 break;
886 }
887
888 default: {
832 - netdata_log_debug(D_CONNECT_TO, "Unknown protocol family %d.", ai->ai_family);
889 + // Unknown protocol family
890 continue;
891 }
892 }
@@ -838,13 +895,17 @@ int connect_to_this_ip46(int protocol, int socktype, const char *host, uint32_t
895 if(fd != -1) {
896 if(timeout) {
897 if(setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, (char *) timeout, sizeof(struct timeval)) < 0)
841 - netdata_log_error("Failed to set timeout on the socket to ip '%s' port '%s'", hostBfr, servBfr);
898 + nd_log(NDLS_DAEMON, NDLP_ERR,
899 + "Failed to set timeout on the socket to ip '%s' port '%s'",
900 + hostBfr, servBfr);
901 }
902
903 errno = 0;
904 if(connect(fd, ai->ai_addr, ai->ai_addrlen) < 0) {
905 if(errno == EALREADY || errno == EINPROGRESS) {
847 - internal_error(true, "Waiting for connection to ip %s port %s to be established", hostBfr, servBfr);
906 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
907 + "Waiting for connection to ip %s port %s to be established",
908 + hostBfr, servBfr);
909
910 // Convert 'struct timeval' to milliseconds for poll():
911 int timeout_milliseconds = timeout->tv_sec * 1000 + timeout->tv_usec / 1000;
@@ -858,11 +919,19 @@ int connect_to_this_ip46(int protocol, int socktype, const char *host, uint32_t
919 // poll() completed normally. We can check the revents to see what happened
920 if (fds[0].revents & POLLOUT) {
921 // connect() completed successfully, socket is writable.
861 - netdata_log_info("connect() to ip %s port %s completed successfully", hostBfr, servBfr);
922 +
923 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
924 + "connect() to ip %s port %s completed successfully",
925 + hostBfr, servBfr);
926 +
927 }
928 else {
929 // This means that the socket is in error. We will close it and set fd to -1
865 - netdata_log_error("Failed to connect to '%s', port '%s'.", hostBfr, servBfr);
930 +
931 + nd_log(NDLS_DAEMON, NDLP_ERR,
932 + "Failed to connect to '%s', port '%s'.",
933 + hostBfr, servBfr);
934 +
935 close(fd);
936 fd = -1;
937 }
@@ -870,27 +939,38 @@ int connect_to_this_ip46(int protocol, int socktype, const char *host, uint32_t
939 else if (ret == 0) {
940 // poll() timed out, the connection is not established within the specified timeout.
941 errno = 0;
873 - netdata_log_error("Timed out while connecting to '%s', port '%s'.", hostBfr, servBfr);
942 +
943 + nd_log(NDLS_DAEMON, NDLP_ERR,
944 + "Timed out while connecting to '%s', port '%s'.",
945 + hostBfr, servBfr);
946 +
947 close(fd);
948 fd = -1;
949 }
877 - else {
950 + else { // ret < 0
951 // poll() returned an error.
879 - netdata_log_error("Failed to connect to '%s', port '%s'. poll() returned %d", hostBfr, servBfr, ret);
952 + nd_log(NDLS_DAEMON, NDLP_ERR,
953 + "Failed to connect to '%s', port '%s'. poll() returned %d",
954 + hostBfr, servBfr, ret);
955 +
956 close(fd);
957 fd = -1;
958 }
959 }
960 else {
885 - netdata_log_error("Failed to connect to '%s', port '%s'", hostBfr, servBfr);
961 + nd_log(NDLS_DAEMON, NDLP_ERR,
962 + "Failed to connect to '%s', port '%s'",
963 + hostBfr, servBfr);
964 +
965 close(fd);
966 fd = -1;
967 }
968 }
890 -
891 - if(fd != -1)
892 - netdata_log_debug(D_CONNECT_TO, "Connected to '%s' on port '%s'.", hostBfr, servBfr);
969 }
970 + else
971 + nd_log(NDLS_DAEMON, NDLP_ERR,
972 + "Failed to socket() to '%s', port '%s'",
973 + hostBfr, servBfr);
974 }
975
976 freeaddrinfo(ai_head);
@@ -965,17 +1045,20 @@ int connect_to_this(const char *definition, int default_port, struct timeval *ti
1045 service = e;
1046 }
1047
968 - netdata_log_debug(D_CONNECT_TO, "Attempting connection to host = '%s', service = '%s', interface = '%s', protocol = %d (tcp = %d, udp = %d)", host, service, interface, protocol, IPPROTO_TCP, IPPROTO_UDP);
969 -
1048 if(!*host) {
971 - netdata_log_error("Definition '%s' does not specify a host.", definition);
1049 + nd_log(NDLS_DAEMON, NDLP_ERR,
1050 + "Definition '%s' does not specify a host.",
1051 + definition);
1052 +
1053 return -1;
1054 }
1055
1056 if(*interface) {
1057 scope_id = if_nametoindex(interface);
1058 if(!scope_id)
978 - netdata_log_error("Cannot find a network interface named '%s'. Continuing with limiting the network interface", interface);
1059 + nd_log(NDLS_DAEMON, NDLP_ERR,
1060 + "Cannot find a network interface named '%s'. Continuing with limiting the network interface",
1061 + interface);
1062 }
1063
1064 if(!*service)
@@ -1117,7 +1200,6 @@ ssize_t recv_timeout(int sockfd, void *buf, size_t len, int flags, int timeout)
1200 }
1201 #endif
1202
1120 - internal_error(true, "%s(): calling recv()", __FUNCTION__ );
1203 return recv(sockfd, buf, len, flags);
1204 }
1205
@@ -1160,7 +1242,9 @@ ssize_t send_timeout(int sockfd, void *buf, size_t len, int flags, int timeout)
1242 return netdata_ssl_write(ssl, buf, len);
1243 }
1244 else {
1163 - netdata_log_error("cannot write to SSL connection - connection is not ready.");
1245 + nd_log(NDLS_DAEMON, NDLP_ERR,
1246 + "cannot write to SSL connection - connection is not ready.");
1247 +
1248 return -1;
1249 }
1250 }
@@ -1224,7 +1308,6 @@ int accept4(int sock, struct sockaddr *addr, socklen_t *addrlen, int flags) {
1308 int connection_allowed(int fd, char *client_ip, char *client_host, size_t hostsize, SIMPLE_PATTERN *access_list,
1309 const char *patname, int allow_dns)
1310 {
1227 - netdata_log_debug(D_LISTENER,"checking %s... (allow_dns=%d)", patname, allow_dns);
1311 if (!access_list)
1312 return 1;
1313 if (simple_pattern_matches(access_list, client_ip))
@@ -1239,16 +1322,21 @@ int connection_allowed(int fd, char *client_ip, char *client_host, size_t hostsi
1322 if (err != 0 ||
1323 (err = getnameinfo((struct sockaddr *)&sadr, addrlen, client_host, (socklen_t)hostsize,
1324 NULL, 0, NI_NAMEREQD)) != 0) {
1242 - netdata_log_error("Incoming %s on '%s' does not match a numeric pattern, and host could not be resolved (err=%s)",
1243 - patname, client_ip, gai_strerror(err));
1325 +
1326 + nd_log(NDLS_DAEMON, NDLP_ERR,
1327 + "Incoming %s on '%s' does not match a numeric pattern, and host could not be resolved (err=%s)",
1328 + patname, client_ip, gai_strerror(err));
1329 +
1330 if (hostsize >= 8)
1331 strcpy(client_host,"UNKNOWN");
1332 return 0;
1333 }
1334 struct addrinfo *addr_infos = NULL;
1335 if (getaddrinfo(client_host, NULL, NULL, &addr_infos) !=0 ) {
1250 - netdata_log_error("LISTENER: cannot validate hostname '%s' from '%s' by resolving it",
1251 - client_host, client_ip);
1336 + nd_log(NDLS_DAEMON, NDLP_ERR,
1337 + "LISTENER: cannot validate hostname '%s' from '%s' by resolving it",
1338 + client_host, client_ip);
1339 +
1340 if (hostsize >= 8)
1341 strcpy(client_host,"UNKNOWN");
1342 return 0;
@@ -1266,8 +1354,6 @@ int connection_allowed(int fd, char *client_ip, char *client_host, size_t hostsi
1354 inet_ntop(AF_INET6, &((struct sockaddr_in6*)(scan->ai_addr))->sin6_addr, address, INET6_ADDRSTRLEN);
1355 break;
1356 }
1269 - netdata_log_debug(D_LISTENER, "Incoming ip %s rev-resolved onto %s, validating against forward-resolution %s",
1270 - client_ip, client_host, address);
1357 if (!strcmp(client_ip, address)) {
1358 validated = 1;
1359 break;
@@ -1275,18 +1361,19 @@ int connection_allowed(int fd, char *client_ip, char *client_host, size_t hostsi
1361 scan = scan->ai_next;
1362 }
1363 if (!validated) {
1278 - netdata_log_error("LISTENER: Cannot validate '%s' as ip of '%s', not listed in DNS", client_ip, client_host);
1364 + nd_log(NDLS_DAEMON, NDLP_ERR,
1365 + "LISTENER: Cannot validate '%s' as ip of '%s', not listed in DNS",
1366 + client_ip, client_host);
1367 +
1368 if (hostsize >= 8)
1369 strcpy(client_host,"UNKNOWN");
1370 }
1371 if (addr_infos!=NULL)
1372 freeaddrinfo(addr_infos);
1373 }
1285 - if (!simple_pattern_matches(access_list, client_host)) {
1286 - netdata_log_debug(D_LISTENER, "Incoming connection on '%s' (%s) does not match allowed pattern for %s",
1287 - client_ip, client_host, patname);
1374 + if (!simple_pattern_matches(access_list, client_host))
1375 return 0;
1289 - }
1376 +
1377 return 1;
1378 }
1379
@@ -1301,7 +1388,10 @@ int accept_socket(int fd, int flags, char *client_ip, size_t ipsize, char *clien
1388 if (likely(nfd >= 0)) {
1389 if (getnameinfo((struct sockaddr *)&sadr, addrlen, client_ip, (socklen_t)ipsize,
1390 client_port, (socklen_t)portsize, NI_NUMERICHOST | NI_NUMERICSERV) != 0) {
1304 - netdata_log_error("LISTENER: cannot getnameinfo() on received client connection.");
1391 +
1392 + nd_log(NDLS_DAEMON, NDLP_ERR,
1393 + "LISTENER: cannot getnameinfo() on received client connection.");
1394 +
1395 strncpyz(client_ip, "UNKNOWN", ipsize);
1396 strncpyz(client_port, "UNKNOWN", portsize);
1397 }
@@ -1319,31 +1409,34 @@ int accept_socket(int fd, int flags, char *client_ip, size_t ipsize, char *clien
1409
1410 switch (((struct sockaddr *)&sadr)->sa_family) {
1411 case AF_UNIX:
1322 - netdata_log_debug(D_LISTENER, "New UNIX domain web client from %s on socket %d.", client_ip, fd);
1412 + // netdata_log_debug(D_LISTENER, "New UNIX domain web client from %s on socket %d.", client_ip, fd);
1413 // set the port - certain versions of libc return garbage on unix sockets
1414 strncpyz(client_port, "UNIX", portsize);
1415 break;
1416
1417 case AF_INET:
1328 - netdata_log_debug(D_LISTENER, "New IPv4 web client from %s port %s on socket %d.", client_ip, client_port, fd);
1418 + // netdata_log_debug(D_LISTENER, "New IPv4 web client from %s port %s on socket %d.", client_ip, client_port, fd);
1419 break;
1420
1421 case AF_INET6:
1422 if (strncmp(client_ip, "::ffff:", 7) == 0) {
1423 memmove(client_ip, &client_ip[7], strlen(&client_ip[7]) + 1);
1334 - netdata_log_debug(D_LISTENER, "New IPv4 web client from %s port %s on socket %d.", client_ip, client_port, fd);
1424 + // netdata_log_debug(D_LISTENER, "New IPv4 web client from %s port %s on socket %d.", client_ip, client_port, fd);
1425 }
1336 - else
1337 - netdata_log_debug(D_LISTENER, "New IPv6 web client from %s port %s on socket %d.", client_ip, client_port, fd);
1426 + // else
1427 + // netdata_log_debug(D_LISTENER, "New IPv6 web client from %s port %s on socket %d.", client_ip, client_port, fd);
1428 break;
1429
1430 default:
1341 - netdata_log_debug(D_LISTENER, "New UNKNOWN web client from %s port %s on socket %d.", client_ip, client_port, fd);
1431 + // netdata_log_debug(D_LISTENER, "New UNKNOWN web client from %s port %s on socket %d.", client_ip, client_port, fd);
1432 break;
1433 }
1434 if (!connection_allowed(nfd, client_ip, client_host, hostsize, access_list, "connection", allow_dns)) {
1435 errno = 0;
1346 - netdata_log_error("Permission denied for client '%s', port '%s'", client_ip, client_port);
1436 + nd_log(NDLS_DAEMON, NDLP_WARNING,
1437 + "Permission denied for client '%s', port '%s'",
1438 + client_ip, client_port);
1439 +
1440 close(nfd);
1441 nfd = -1;
1442 errno = EPERM;
@@ -1351,7 +1444,9 @@ int accept_socket(int fd, int flags, char *client_ip, size_t ipsize, char *clien
1444 }
1445 #ifdef HAVE_ACCEPT4
1446 else if (errno == ENOSYS)
1354 - netdata_log_error("netdata has been compiled with the assumption that the system has the accept4() call, but it is not here. Recompile netdata like this: ./configure --disable-accept4 ...");
1447 + nd_log(NDLS_DAEMON, NDLP_ERR,
1448 + "Netdata has been compiled with the assumption that the system has the accept4() call, but it is not here. "
1449 + "Recompile netdata like this: ./configure --disable-accept4 ...");
1450 #endif
1451
1452 return nfd;
@@ -1379,19 +1474,16 @@ inline POLLINFO *poll_add_fd(POLLJOB *p
1474 , int (*snd_callback)(POLLINFO * /*pi*/, short int * /*events*/)
1475 , void *data
1476 ) {
1382 - netdata_log_debug(D_POLLFD, "POLLFD: ADD: request to add fd %d, slots = %zu, used = %zu, min = %zu, max = %zu, next free = %zd", fd, p->slots, p->used, p->min, p->max, p->first_free?(ssize_t)p->first_free->slot:(ssize_t)-1);
1383 -
1477 if(unlikely(fd < 0)) return NULL;
1478
1479 //if(p->limit && p->used >= p->limit) {
1387 - // netdata_log_info("Max sockets limit reached (%zu sockets), dropping connection", p->used);
1480 + // nd_log(NDLS_DAEMON, NDLP_WARNING, "Max sockets limit reached (%zu sockets), dropping connection", p->used);
1481 // close(fd);
1482 // return NULL;
1483 //}
1484
1485 if(unlikely(!p->first_free)) {
1486 size_t new_slots = p->slots + POLL_FDS_INCREASE_STEP;
1394 - netdata_log_debug(D_POLLFD, "POLLFD: ADD: increasing size (current = %zu, new = %zu, used = %zu, min = %zu, max = %zu)", p->slots, new_slots, p->used, p->min, p->max);
1487
1488 p->fds = reallocz(p->fds, sizeof(struct pollfd) * new_slots);
1489 p->inf = reallocz(p->inf, sizeof(POLLINFO) * new_slots);
@@ -1399,7 +1491,6 @@ inline POLLINFO *poll_add_fd(POLLJOB *p
1491 // reset all the newly added slots
1492 ssize_t i;
1493 for(i = new_slots - 1; i >= (ssize_t)p->slots ; i--) {
1402 - netdata_log_debug(D_POLLFD, "POLLFD: ADD: resetting new slot %zd", i);
1494 p->fds[i].fd = -1;
1495 p->fds[i].events = 0;
1496 p->fds[i].revents = 0;
@@ -1430,8 +1521,6 @@ inline POLLINFO *poll_add_fd(POLLJOB *p
1521 POLLINFO *pi = p->first_free;
1522 p->first_free = p->first_free->next;
1523
1433 - netdata_log_debug(D_POLLFD, "POLLFD: ADD: selected slot %zu, next free is %zd", pi->slot, p->first_free?(ssize_t)p->first_free->slot:(ssize_t)-1);
1434 -
1524 struct pollfd *pf = &p->fds[pi->slot];
1525 pf->fd = fd;
1526 pf->events = POLLIN;
@@ -1472,8 +1561,6 @@ inline POLLINFO *poll_add_fd(POLLJOB *p
1561 }
1562 netdata_thread_enable_cancelability();
1563
1475 - netdata_log_debug(D_POLLFD, "POLLFD: ADD: completed, slots = %zu, used = %zu, min = %zu, max = %zu, next free = %zd", p->slots, p->used, p->min, p->max, p->first_free?(ssize_t)p->first_free->slot:(ssize_t)-1);
1476 -
1564 return pi;
1565 }
1566
@@ -1481,7 +1568,6 @@ inline void poll_close_fd(POLLINFO *pi) {
1568 POLLJOB *p = pi->p;
1569
1570 struct pollfd *pf = &p->fds[pi->slot];
1484 - netdata_log_debug(D_POLLFD, "POLLFD: DEL: request to clear slot %zu (fd %d), old next free was %zd", pi->slot, pf->fd, p->first_free?(ssize_t)p->first_free->slot:(ssize_t)-1);
1571
1572 if(unlikely(pf->fd == -1)) return;
1573
@@ -1492,7 +1578,9 @@ inline void poll_close_fd(POLLINFO *pi) {
1578
1579 if(likely(!(pi->flags & POLLINFO_FLAG_DONT_CLOSE))) {
1580 if(close(pf->fd) == -1)
1495 - netdata_log_error("Failed to close() poll_events() socket %d", pf->fd);
1581 + nd_log(NDLS_DAEMON, NDLP_ERR,
1582 + "Failed to close() poll_events() socket %d",
1583 + pf->fd);
1584 }
1585 }
1586
@@ -1533,8 +1621,6 @@ inline void poll_close_fd(POLLINFO *pi) {
1621 }
1622 }
1623 netdata_thread_enable_cancelability();
1536 -
1537 - netdata_log_debug(D_POLLFD, "POLLFD: DEL: completed, slots = %zu, used = %zu, min = %zu, max = %zu, next free = %zd", p->slots, p->used, p->min, p->max, p->first_free?(ssize_t)p->first_free->slot:(ssize_t)-1);
1624 }
1625
1626 void *poll_default_add_callback(POLLINFO *pi, short int *events, void *data) {
@@ -1542,14 +1628,13 @@ void *poll_default_add_callback(POLLINFO *pi, short int *events, void *data) {
1628 (void)events;
1629 (void)data;
1630
1545 - // netdata_log_error("POLLFD: internal error: poll_default_add_callback() called");
1546 -
1631 return NULL;
1632 }
1633
1634 void poll_default_del_callback(POLLINFO *pi) {
1635 if(pi->data)
1552 - netdata_log_error("POLLFD: internal error: del_callback_default() called with data pointer - possible memory leak");
1636 + nd_log(NDLS_DAEMON, NDLP_ERR,
1637 + "POLLFD: internal error: del_callback_default() called with data pointer - possible memory leak");
1638 }
1639
1640 int poll_default_rcv_callback(POLLINFO *pi, short int *events) {
@@ -1563,12 +1648,17 @@ int poll_default_rcv_callback(POLLINFO *pi, short int *events) {
1648 if (rc < 0) {
1649 // read failed
1650 if (errno != EWOULDBLOCK && errno != EAGAIN) {
1566 - netdata_log_error("POLLFD: poll_default_rcv_callback(): recv() failed with %zd.", rc);
1651 + nd_log(NDLS_DAEMON, NDLP_ERR,
1652 + "POLLFD: poll_default_rcv_callback(): recv() failed with %zd.",
1653 + rc);
1654 +
1655 return -1;
1656 }
1657 } else if (rc) {
1658 // data received
1571 - netdata_log_info("POLLFD: internal error: poll_default_rcv_callback() is discarding %zd bytes received on socket %d", rc, pi->fd);
1659 + nd_log(NDLS_DAEMON, NDLP_WARNING,
1660 + "POLLFD: internal error: poll_default_rcv_callback() is discarding %zd bytes received on socket %d",
1661 + rc, pi->fd);
1662 }
1663 } while (rc != -1);
1664
@@ -1578,7 +1668,10 @@ int poll_default_rcv_callback(POLLINFO *pi, short int *events) {
1668 int poll_default_snd_callback(POLLINFO *pi, short int *events) {
1669 *events &= ~POLLOUT;
1670
1581 - netdata_log_info("POLLFD: internal error: poll_default_snd_callback(): nothing to send on socket %d", pi->fd);
1671 + nd_log(NDLS_DAEMON, NDLP_WARNING,
1672 + "POLLFD: internal error: poll_default_snd_callback(): nothing to send on socket %d",
1673 + pi->fd);
1674 +
1675 return 0;
1676 }
1677
@@ -1600,17 +1693,25 @@ static void poll_events_cleanup(void *data) {
1693 }
1694
1695 static int poll_process_error(POLLINFO *pi, struct pollfd *pf, short int revents) {
1603 - netdata_log_error("POLLFD: LISTENER: received %s %s %s on socket at slot %zu (fd %d) client '%s' port '%s' expecting %s %s %s, having %s %s %s"
1604 - , revents & POLLERR ? "POLLERR" : ""
1605 - , revents & POLLHUP ? "POLLHUP" : ""
1606 - , revents & POLLNVAL ? "POLLNVAL" : ""
1607 - , pi->slot
1608 - , pi->fd
1609 - , pi->client_ip ? pi->client_ip : "<undefined-ip>"
1610 - , pi->client_port ? pi->client_port : "<undefined-port>"
1611 - , pf->events & POLLIN ? "POLLIN" : "", pf->events & POLLOUT ? "POLLOUT" : "", pf->events & POLLPRI ? "POLLPRI" : ""
1612 - , revents & POLLIN ? "POLLIN" : "", revents & POLLOUT ? "POLLOUT" : "", revents & POLLPRI ? "POLLPRI" : ""
1613 - );
1696 + ND_LOG_STACK lgs[] = {
1697 + ND_LOG_FIELD_TXT(NDF_SRC_IP, pi->client_ip),
1698 + ND_LOG_FIELD_TXT(NDF_SRC_PORT, pi->client_port),
1699 + ND_LOG_FIELD_END(),
1700 + };
1701 + ND_LOG_STACK_PUSH(lgs);
1702 +
1703 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1704 + "POLLFD: LISTENER: received %s %s %s on socket at slot %zu (fd %d) client '%s' port '%s' expecting %s %s %s, having %s %s %s"
1705 + , revents & POLLERR ? "POLLERR" : ""
1706 + , revents & POLLHUP ? "POLLHUP" : ""
1707 + , revents & POLLNVAL ? "POLLNVAL" : ""
1708 + , pi->slot
1709 + , pi->fd
1710 + , pi->client_ip ? pi->client_ip : "<undefined-ip>"
1711 + , pi->client_port ? pi->client_port : "<undefined-port>"
1712 + , pf->events & POLLIN ? "POLLIN" : "", pf->events & POLLOUT ? "POLLOUT" : "", pf->events & POLLPRI ? "POLLPRI" : ""
1713 + , revents & POLLIN ? "POLLIN" : "", revents & POLLOUT ? "POLLOUT" : "", revents & POLLPRI ? "POLLPRI" : ""
1714 + );
1715
1716 pf->events = 0;
1717 poll_close_fd(pi);
@@ -1621,8 +1722,6 @@ static inline int poll_process_send(POLLJOB *p, POLLINFO *pi, struct pollfd *pf,
1722 pi->last_sent_t = now;
1723 pi->send_count++;
1724
1624 - netdata_log_debug(D_POLLFD, "POLLFD: LISTENER: sending data to socket on slot %zu (fd %d)", pi->slot, pf->fd);
1625 -
1725 pf->events = 0;
1726
1727 // remember the slot, in case we need to close it later
@@ -1642,8 +1741,6 @@ static inline int poll_process_tcp_read(POLLJOB *p, POLLINFO *pi, struct pollfd
1741 pi->last_received_t = now;
1742 pi->recv_count++;
1743
1645 - netdata_log_debug(D_POLLFD, "POLLFD: LISTENER: reading data from TCP client slot %zu (fd %d)", pi->slot, pf->fd);
1646 -
1744 pf->events = 0;
1745
1746 // remember the slot, in case we need to close it later
@@ -1663,8 +1760,6 @@ static inline int poll_process_udp_read(POLLINFO *pi, struct pollfd *pf, time_t
1760 pi->last_received_t = now;
1761 pi->recv_count++;
1762
1666 - netdata_log_debug(D_POLLFD, "POLLFD: LISTENER: reading data from UDP slot %zu (fd %d)", pi->slot, pf->fd);
1667 -
1763 // TODO: access_list is not applied to UDP
1764 // but checking the access list on every UDP packet will destroy
1765 // performance, especially for statsd.
@@ -1683,14 +1778,10 @@ static int poll_process_new_tcp_connection(POLLJOB *p, POLLINFO *pi, struct poll
1778 pi->last_received_t = now;
1779 pi->recv_count++;
1780
1686 - netdata_log_debug(D_POLLFD, "POLLFD: LISTENER: accepting connections from slot %zu (fd %d)", pi->slot, pf->fd);
1687 -
1781 char client_ip[INET6_ADDRSTRLEN] = "";
1782 char client_port[NI_MAXSERV] = "";
1783 char client_host[NI_MAXHOST] = "";
1784
1692 - netdata_log_debug(D_POLLFD, "POLLFD: LISTENER: calling accept4() slot %zu (fd %d)", pi->slot, pf->fd);
1693 -
1785 int nfd = accept_socket(
1786 pf->fd,SOCK_NONBLOCK,
1787 client_ip, INET6_ADDRSTRLEN, client_port,NI_MAXSERV, client_host, NI_MAXHOST,
@@ -1700,15 +1791,15 @@ static int poll_process_new_tcp_connection(POLLJOB *p, POLLINFO *pi, struct poll
1791 if (unlikely(nfd < 0)) {
1792 // accept failed
1793
1703 - netdata_log_debug(D_POLLFD, "POLLFD: LISTENER: accept4() slot %zu (fd %d) failed.", pi->slot, pf->fd);
1704 -
1794 if(unlikely(errno == EMFILE)) {
1706 - error_limit_static_global_var(erl, 10, 1000);
1707 - error_limit(&erl, "POLLFD: LISTENER: too many open files - used by this thread %zu, max for this thread %zu",
1708 - p->used, p->limit);
1795 + nd_log_limit_static_global_var(erl, 10, 1000);
1796 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR,
1797 + "POLLFD: LISTENER: too many open files - used by this thread %zu, max for this thread %zu",
1798 + p->used, p->limit);
1799 }
1800 else if(unlikely(errno != EWOULDBLOCK && errno != EAGAIN))
1711 - netdata_log_error("POLLFD: LISTENER: accept() failed.");
1801 + nd_log(NDLS_DAEMON, NDLP_ERR,
1802 + "POLLFD: LISTENER: accept() failed.");
1803
1804 }
1805 else {
@@ -1755,7 +1846,8 @@ void poll_events(LISTEN_SOCKETS *sockets
1846 , size_t max_tcp_sockets
1847 ) {
1848 if(!sockets || !sockets->opened) {
1758 - netdata_log_error("POLLFD: internal error: no listening sockets are opened");
1849 + nd_log(NDLS_DAEMON, NDLP_ERR,
1850 + "POLLFD: internal error: no listening sockets are opened");
1851 return;
1852 }
1853
@@ -1808,7 +1900,9 @@ void poll_events(LISTEN_SOCKETS *sockets
1900 );
1901
1902 pi->data = data;
1811 - netdata_log_info("POLLFD: LISTENER: listening on '%s'", (sockets->fds_names[i])?sockets->fds_names[i]:"UNKNOWN");
1903 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1904 + "POLLFD: LISTENER: listening on '%s'",
1905 + (sockets->fds_names[i])?sockets->fds_names[i]:"UNKNOWN");
1906 }
1907
1908 int listen_sockets_active = 1;
@@ -1832,7 +1926,6 @@ void poll_events(LISTEN_SOCKETS *sockets
1926 now_usec = now_boottime_usec();
1927
1928 if(unlikely(timer_usec && now_usec >= next_timer_usec)) {
1835 - netdata_log_debug(D_POLLFD, "Calling timer callback after %zu usec", (size_t)(now_usec - last_timer_usec));
1929 last_timer_usec = now_usec;
1930 p.tmr_callback(p.timer_data);
1931 now_usec = now_boottime_usec();
@@ -1849,7 +1942,11 @@ void poll_events(LISTEN_SOCKETS *sockets
1942 // enable or disable the TCP listening sockets, based on the current number of sockets used and the limit set
1943 if((listen_sockets_active && (p.limit && p.used >= p.limit)) || (!listen_sockets_active && (!p.limit || p.used < p.limit))) {
1944 listen_sockets_active = !listen_sockets_active;
1852 - netdata_log_info("%s listening sockets (used TCP sockets %zu, max allowed for this worker %zu)", (listen_sockets_active)?"ENABLING":"DISABLING", p.used, p.limit);
1945 +
1946 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1947 + "%s listening sockets (used TCP sockets %zu, max allowed for this worker %zu)",
1948 + (listen_sockets_active)?"ENABLING":"DISABLING", p.used, p.limit);
1949 +
1950 for (i = 0; i <= p.max; i++) {
1951 if(p.inf[i].flags & POLLINFO_FLAG_SERVER_SOCKET && p.inf[i].socktype == SOCK_STREAM) {
1952 p.fds[i].events = (short int) ((listen_sockets_active) ? POLLIN : 0);
@@ -1857,16 +1954,19 @@ void poll_events(LISTEN_SOCKETS *sockets
1954 }
1955 }
1956
1860 - netdata_log_debug(D_POLLFD, "POLLFD: LISTENER: Waiting on %zu sockets for %zu ms...", p.max + 1, (size_t)timeout_ms);
1957 retval = poll(p.fds, p.max + 1, timeout_ms);
1958 time_t now = now_boottime_sec();
1959
1960 if(unlikely(retval == -1)) {
1865 - netdata_log_error("POLLFD: LISTENER: poll() failed while waiting on %zu sockets.", p.max + 1);
1961 + nd_log(NDLS_DAEMON, NDLP_ERR,
1962 + "POLLFD: LISTENER: poll() failed while waiting on %zu sockets.",
1963 + p.max + 1);
1964 +
1965 break;
1966 }
1967 else if(unlikely(!retval)) {
1869 - netdata_log_debug(D_POLLFD, "POLLFD: LISTENER: poll() timeout.");
1968 + // timeout
1969 + ;
1970 }
1971 else {
1972 POLLINFO *pi;
@@ -1920,31 +2020,34 @@ void poll_events(LISTEN_SOCKETS *sockets
2020 conns[conns_max++] = i;
2021 }
2022 else
1923 - netdata_log_error("POLLFD: LISTENER: server slot %zu (fd %d) connection from %s port %s using unhandled socket type %d."
1924 - , i
1925 - , pi->fd
1926 - , pi->client_ip ? pi->client_ip : "<undefined-ip>"
1927 - , pi->client_port ? pi->client_port : "<undefined-port>"
1928 - , pi->socktype
1929 - );
2023 + nd_log(NDLS_DAEMON, NDLP_ERR,
2024 + "POLLFD: LISTENER: server slot %zu (fd %d) connection from %s port %s using unhandled socket type %d."
2025 + , i
2026 + , pi->fd
2027 + , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2028 + , pi->client_port ? pi->client_port : "<undefined-port>"
2029 + , pi->socktype
2030 + );
2031 }
2032 else
1932 - netdata_log_error("POLLFD: LISTENER: client slot %zu (fd %d) data from %s port %s using flags %08X is neither client nor server."
1933 - , i
1934 - , pi->fd
1935 - , pi->client_ip ? pi->client_ip : "<undefined-ip>"
1936 - , pi->client_port ? pi->client_port : "<undefined-port>"
1937 - , pi->flags
1938 - );
2033 + nd_log(NDLS_DAEMON, NDLP_ERR,
2034 + "POLLFD: LISTENER: client slot %zu (fd %d) data from %s port %s using flags %08X is neither client nor server."
2035 + , i
2036 + , pi->fd
2037 + , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2038 + , pi->client_port ? pi->client_port : "<undefined-port>"
2039 + , pi->flags
2040 + );
2041 }
2042 else
1941 - netdata_log_error("POLLFD: LISTENER: socket slot %zu (fd %d) client %s port %s unhandled event id %d."
1942 - , i
1943 - , pi->fd
1944 - , pi->client_ip ? pi->client_ip : "<undefined-ip>"
1945 - , pi->client_port ? pi->client_port : "<undefined-port>"
1946 - , revents
1947 - );
2043 + nd_log(NDLS_DAEMON, NDLP_ERR,
2044 + "POLLFD: LISTENER: socket slot %zu (fd %d) client %s port %s unhandled event id %d."
2045 + , i
2046 + , pi->fd
2047 + , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2048 + , pi->client_port ? pi->client_port : "<undefined-port>"
2049 + , revents
2050 + );
2051 }
2052
2053 // process sends
@@ -1997,23 +2100,25 @@ void poll_events(LISTEN_SOCKETS *sockets
2100
2101 if(likely(pi->flags & POLLINFO_FLAG_CLIENT_SOCKET)) {
2102 if (unlikely(pi->send_count == 0 && p.complete_request_timeout > 0 && (now - pi->connected_t) >= p.complete_request_timeout)) {
2000 - netdata_log_info("POLLFD: LISTENER: client slot %zu (fd %d) from %s port %s has not sent a complete request in %zu seconds - closing it. "
2001 - , i
2002 - , pi->fd
2003 - , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2004 - , pi->client_port ? pi->client_port : "<undefined-port>"
2005 - , (size_t) p.complete_request_timeout
2006 - );
2103 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
2104 + "POLLFD: LISTENER: client slot %zu (fd %d) from %s port %s has not sent a complete request in %zu seconds - closing it. "
2105 + , i
2106 + , pi->fd
2107 + , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2108 + , pi->client_port ? pi->client_port : "<undefined-port>"
2109 + , (size_t) p.complete_request_timeout
2110 + );
2111 poll_close_fd(pi);
2112 }
2113 else if(unlikely(pi->recv_count && p.idle_timeout > 0 && now - ((pi->last_received_t > pi->last_sent_t) ? pi->last_received_t : pi->last_sent_t) >= p.idle_timeout )) {
2010 - netdata_log_info("POLLFD: LISTENER: client slot %zu (fd %d) from %s port %s is idle for more than %zu seconds - closing it. "
2011 - , i
2012 - , pi->fd
2013 - , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2014 - , pi->client_port ? pi->client_port : "<undefined-port>"
2015 - , (size_t) p.idle_timeout
2016 - );
2114 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
2115 + "POLLFD: LISTENER: client slot %zu (fd %d) from %s port %s is idle for more than %zu seconds - closing it. "
2116 + , i
2117 + , pi->fd
2118 + , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2119 + , pi->client_port ? pi->client_port : "<undefined-port>"
2120 + , (size_t) p.idle_timeout
2121 + );
2122 poll_close_fd(pi);
2123 }
2124 }
@@ -2022,5 +2127,4 @@ void poll_events(LISTEN_SOCKETS *sockets
2127 }
2128
2129 netdata_thread_cleanup_pop(1);
2025 - netdata_log_debug(D_POLLFD, "POLLFD: LISTENER: cleanup completed");
2130 }
libnetdata/threads/threads.c
+17 -19
@@ -134,8 +134,6 @@ size_t netdata_threads_init(void) {
134 i = pthread_attr_getstacksize(netdata_threads_attr, &stacksize);
135 if(i != 0)
136 fatal("pthread_attr_getstacksize() failed with code %d.", i);
137 - else
138 - netdata_log_debug(D_OPTIONS, "initial pthread stack size is %zu bytes", stacksize);
137
138 return stacksize;
139 }
@@ -152,12 +150,12 @@ void netdata_threads_init_after_fork(size_t stacksize) {
150 if(netdata_threads_attr && stacksize > (size_t)PTHREAD_STACK_MIN) {
151 i = pthread_attr_setstacksize(netdata_threads_attr, stacksize);
152 if(i != 0)
155 - netdata_log_error("pthread_attr_setstacksize() to %zu bytes, failed with code %d.", stacksize, i);
153 + nd_log(NDLS_DAEMON, NDLP_WARNING, "pthread_attr_setstacksize() to %zu bytes, failed with code %d.", stacksize, i);
154 else
157 - netdata_log_info("Set threads stack size to %zu bytes", stacksize);
155 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "Set threads stack size to %zu bytes", stacksize);
156 }
157 else
160 - netdata_log_error("Invalid pthread stacksize %zu", stacksize);
158 + nd_log(NDLS_DAEMON, NDLP_WARNING, "Invalid pthread stacksize %zu", stacksize);
159 }
160
161 // ----------------------------------------------------------------------------
@@ -183,12 +181,12 @@ void rrd_collector_finished(void);
181 static void thread_cleanup(void *ptr) {
182 if(netdata_thread != ptr) {
183 NETDATA_THREAD *info = (NETDATA_THREAD *)ptr;
186 - netdata_log_error("THREADS: internal error - thread local variable does not match the one passed to this function. Expected thread '%s', passed thread '%s'", netdata_thread->tag, info->tag);
184 + nd_log(NDLS_DAEMON, NDLP_ERR, "THREADS: internal error - thread local variable does not match the one passed to this function. Expected thread '%s', passed thread '%s'", netdata_thread->tag, info->tag);
185 }
186 spinlock_lock(&netdata_thread->detach_lock);
187
188 if(!(netdata_thread->options & NETDATA_THREAD_OPTION_DONT_LOG_CLEANUP))
191 - netdata_log_info("thread with task id %d finished", gettid());
189 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "thread with task id %d finished", gettid());
190
191 rrd_collector_finished();
192 sender_thread_buffer_free();
@@ -222,9 +220,9 @@ static void thread_set_name_np(NETDATA_THREAD *nt) {
220 #endif
221
222 if (ret != 0)
225 - netdata_log_error("cannot set pthread name of %d to %s. ErrCode: %d", gettid(), threadname, ret);
223 + nd_log(NDLS_DAEMON, NDLP_WARNING, "cannot set pthread name of %d to %s. ErrCode: %d", gettid(), threadname, ret);
224 else
227 - netdata_log_info("set name of thread %d to %s", gettid(), threadname);
225 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "set name of thread %d to %s", gettid(), threadname);
226
227 }
228 }
@@ -247,7 +245,7 @@ void uv_thread_set_name_np(uv_thread_t ut, const char* name) {
245 thread_name_get(true);
246
247 if (ret)
250 - netdata_log_info("cannot set libuv thread name to %s. Err: %d", threadname, ret);
248 + nd_log(NDLS_DAEMON, NDLP_NOTICE, "cannot set libuv thread name to %s. Err: %d", threadname, ret);
249 }
250
251 void os_thread_get_current_name_np(char threadname[NETDATA_THREAD_NAME_MAX + 1])
@@ -264,13 +262,13 @@ static void *netdata_thread_init(void *ptr) {
262 netdata_thread = (NETDATA_THREAD *)ptr;
263
264 if(!(netdata_thread->options & NETDATA_THREAD_OPTION_DONT_LOG_STARTUP))
267 - netdata_log_info("thread created with task id %d", gettid());
265 + nd_log(NDLS_DAEMON, NDLP_DEBUG, "thread created with task id %d", gettid());
266
267 if(pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL) != 0)
270 - netdata_log_error("cannot set pthread cancel type to DEFERRED.");
268 + nd_log(NDLS_DAEMON, NDLP_WARNING, "cannot set pthread cancel type to DEFERRED.");
269
270 if(pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL) != 0)
273 - netdata_log_error("cannot set pthread cancel state to ENABLE.");
271 + nd_log(NDLS_DAEMON, NDLP_WARNING, "cannot set pthread cancel state to ENABLE.");
272
273 thread_set_name_np(ptr);
274
@@ -294,13 +292,13 @@ int netdata_thread_create(netdata_thread_t *thread, const char *tag, NETDATA_THR
292
293 int ret = pthread_create(thread, netdata_threads_attr, netdata_thread_init, info);
294 if(ret != 0)
297 - netdata_log_error("failed to create new thread for %s. pthread_create() failed with code %d", tag, ret);
295 + nd_log(NDLS_DAEMON, NDLP_ERR, "failed to create new thread for %s. pthread_create() failed with code %d", tag, ret);
296
297 else {
298 if (!(options & NETDATA_THREAD_OPTION_JOINABLE)) {
299 int ret2 = pthread_detach(*thread);
300 if (ret2 != 0)
303 - netdata_log_error("cannot request detach of newly created %s thread. pthread_detach() failed with code %d", tag, ret2);
301 + nd_log(NDLS_DAEMON, NDLP_WARNING, "cannot request detach of newly created %s thread. pthread_detach() failed with code %d", tag, ret2);
302 }
303 }
304
@@ -318,9 +316,9 @@ int netdata_thread_cancel(netdata_thread_t thread) {
316 int ret = pthread_cancel(thread);
317 if(ret != 0)
318 #ifdef NETDATA_INTERNAL_CHECKS
321 - netdata_log_error("cannot cancel thread. pthread_cancel() failed with code %d at %d@%s, function %s()", ret, line, file, function);
319 + nd_log(NDLS_DAEMON, NDLP_WARNING, "cannot cancel thread. pthread_cancel() failed with code %d at %d@%s, function %s()", ret, line, file, function);
320 #else
323 - netdata_log_error("cannot cancel thread. pthread_cancel() failed with code %d.", ret);
321 + nd_log(NDLS_DAEMON, NDLP_WARNING, "cannot cancel thread. pthread_cancel() failed with code %d.", ret);
322 #endif
323
324 return ret;
@@ -332,7 +330,7 @@ int netdata_thread_cancel(netdata_thread_t thread) {
330 int netdata_thread_join(netdata_thread_t thread, void **retval) {
331 int ret = pthread_join(thread, retval);
332 if(ret != 0)
335 - netdata_log_error("cannot join thread. pthread_join() failed with code %d.", ret);
333 + nd_log(NDLS_DAEMON, NDLP_WARNING, "cannot join thread. pthread_join() failed with code %d.", ret);
334
335 return ret;
336 }
@@ -340,7 +338,7 @@ int netdata_thread_join(netdata_thread_t thread, void **retval) {
338 int netdata_thread_detach(pthread_t thread) {
339 int ret = pthread_detach(thread);
340 if(ret != 0)
343 - netdata_log_error("cannot detach thread. pthread_detach() failed with code %d.", ret);
341 + nd_log(NDLS_DAEMON, NDLP_WARNING, "cannot detach thread. pthread_detach() failed with code %d.", ret);
342
343 return ret;
344 }
libnetdata/uuid/Makefile.am new
+8
@@ -0,0 +1,8 @@
1 +# SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +AUTOMAKE_OPTIONS = subdir-objects
4 +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
5 +
6 +dist_noinst_DATA = \
7 + README.md \
8 + $(NULL)
libnetdata/uuid/README.md new
+13
@@ -0,0 +1,13 @@
1 +<!--
2 +title: "UUID"
3 +custom_edit_url: https://github.com/netdata/netdata/edit/master/libnetdata/uuid/README.md
4 +sidebar_label: "UUID"
5 +learn_topic_type: "Tasks"
6 +learn_rel_path: "Developers/libnetdata"
7 +-->
8 +
9 +# UUID
10 +
11 +Netdata uses libuuid for managing UUIDs.
12 +
13 +In this folder are a few custom helpers.
\ No newline at end of file
libnetdata/uuid/uuid.c new
+179
@@ -0,0 +1,179 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "../libnetdata.h"
4 +
5 +void uuid_unparse_lower_compact(const uuid_t uuid, char *out) {
6 + static const char *hex_chars = "0123456789abcdef";
7 + for (int i = 0; i < 16; i++) {
8 + out[i * 2] = hex_chars[(uuid[i] >> 4) & 0x0F];
9 + out[i * 2 + 1] = hex_chars[uuid[i] & 0x0F];
10 + }
11 + out[32] = '\0'; // Null-terminate the string
12 +}
13 +
14 +inline int uuid_parse_compact(const char *in, uuid_t uuid) {
15 + if (strlen(in) != 32)
16 + return -1; // Invalid input length
17 +
18 + for (int i = 0; i < 16; i++) {
19 + int high = hex_char_to_int(in[i * 2]);
20 + int low = hex_char_to_int(in[i * 2 + 1]);
21 +
22 + if (high < 0 || low < 0)
23 + return -1; // Invalid hexadecimal character
24 +
25 + uuid[i] = (high << 4) | low;
26 + }
27 +
28 + return 0; // Success
29 +}
30 +
31 +int uuid_parse_flexi(const char *in, uuid_t uu) {
32 + if(!in || !*in)
33 + return -1;
34 +
35 + size_t hexCharCount = 0;
36 + size_t hyphenCount = 0;
37 + const char *s = in;
38 + int byteIndex = 0;
39 + uuid_t uuid; // work on a temporary place, to not corrupt the previous value of uu if we fail
40 +
41 + while (*s && byteIndex < 16) {
42 + if (*s == '-') {
43 + s++;
44 + hyphenCount++;
45 +
46 + if (unlikely(hyphenCount > 4))
47 + // Too many hyphens
48 + return -2;
49 + }
50 +
51 + if (likely(isxdigit(*s))) {
52 + int high = hex_char_to_int(*s++);
53 + hexCharCount++;
54 +
55 + if (likely(isxdigit(*s))) {
56 + int low = hex_char_to_int(*s++);
57 + hexCharCount++;
58 +
59 + uuid[byteIndex++] = (high << 4) | low;
60 + }
61 + else
62 + // Not a valid UUID (expected a pair of hex digits)
63 + return -3;
64 + }
65 + else
66 + // Not a valid UUID
67 + return -4;
68 + }
69 +
70 + if (unlikely(byteIndex < 16))
71 + // Not enough data to form a UUID
72 + return -5;
73 +
74 + if (unlikely(hexCharCount != 32))
75 + // wrong number of hex digits
76 + return -6;
77 +
78 + if(unlikely(hyphenCount != 0 && hyphenCount != 4))
79 + // wrong number of hyphens
80 + return -7;
81 +
82 + // copy the final value
83 + memcpy(uu, uuid, sizeof(uuid_t));
84 +
85 + return 0;
86 +}
87 +
88 +
89 +// ----------------------------------------------------------------------------
90 +// unit test
91 +
92 +static inline void remove_hyphens(const char *uuid_with_hyphens, char *uuid_without_hyphens) {
93 + while (*uuid_with_hyphens) {
94 + if (*uuid_with_hyphens != '-') {
95 + *uuid_without_hyphens++ = *uuid_with_hyphens;
96 + }
97 + uuid_with_hyphens++;
98 + }
99 + *uuid_without_hyphens = '\0';
100 +}
101 +
102 +int uuid_unittest(void) {
103 + const int num_tests = 100000;
104 + int failed_tests = 0;
105 +
106 + int i;
107 + for (i = 0; i < num_tests; i++) {
108 + uuid_t original_uuid, parsed_uuid;
109 + char uuid_str_with_hyphens[UUID_STR_LEN], uuid_str_without_hyphens[UUID_COMPACT_STR_LEN];
110 +
111 + // Generate a random UUID
112 + switch(i % 2) {
113 + case 0:
114 + uuid_generate(original_uuid);
115 + break;
116 +
117 + case 1:
118 + uuid_generate_random(original_uuid);
119 + break;
120 + }
121 +
122 + // Unparse it with hyphens
123 + bool lower = false;
124 + switch(i % 3) {
125 + case 0:
126 + uuid_unparse_lower(original_uuid, uuid_str_with_hyphens);
127 + lower = true;
128 + break;
129 +
130 + case 1:
131 + uuid_unparse(original_uuid, uuid_str_with_hyphens);
132 + break;
133 +
134 + case 2:
135 + uuid_unparse_upper(original_uuid, uuid_str_with_hyphens);
136 + break;
137 + }
138 +
139 + // Remove the hyphens
140 + remove_hyphens(uuid_str_with_hyphens, uuid_str_without_hyphens);
141 +
142 + if(lower) {
143 + char test[UUID_COMPACT_STR_LEN];
144 + uuid_unparse_lower_compact(original_uuid, test);
145 + if(strcmp(test, uuid_str_without_hyphens) != 0) {
146 + printf("uuid_unparse_lower_compact() failed, expected '%s', got '%s'\n",
147 + uuid_str_without_hyphens, test);
148 + failed_tests++;
149 + }
150 + }
151 +
152 + // Parse the UUID string with hyphens
153 + int parse_result = uuid_parse_flexi(uuid_str_with_hyphens, parsed_uuid);
154 + if (parse_result != 0) {
155 + printf("uuid_parse_flexi() returned -1 (parsing error) for UUID with hyphens: %s\n", uuid_str_with_hyphens);
156 + failed_tests++;
157 + } else if (uuid_compare(original_uuid, parsed_uuid) != 0) {
158 + printf("uuid_parse_flexi() parsed value mismatch for UUID with hyphens: %s\n", uuid_str_with_hyphens);
159 + failed_tests++;
160 + }
161 +
162 + // Parse the UUID string without hyphens
163 + parse_result = uuid_parse_flexi(uuid_str_without_hyphens, parsed_uuid);
164 + if (parse_result != 0) {
165 + printf("uuid_parse_flexi() returned -1 (parsing error) for UUID without hyphens: %s\n", uuid_str_without_hyphens);
166 + failed_tests++;
167 + }
168 + else if(uuid_compare(original_uuid, parsed_uuid) != 0) {
169 + printf("uuid_parse_flexi() parsed value mismatch for UUID without hyphens: %s\n", uuid_str_without_hyphens);
170 + failed_tests++;
171 + }
172 +
173 + if(failed_tests)
174 + break;
175 + }
176 +
177 + printf("UUID: failed %d out of %d tests.\n", failed_tests, i);
178 + return failed_tests;
179 +}
libnetdata/uuid/uuid.h new
+29
@@ -0,0 +1,29 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_UUID_H
4 +#define NETDATA_UUID_H
5 +
6 +UUID_DEFINE(streaming_from_child_msgid, 0xed,0x4c,0xdb, 0x8f, 0x1b, 0xeb, 0x4a, 0xd3, 0xb5, 0x7c, 0xb3, 0xca, 0xe2, 0xd1, 0x62, 0xfa);
7 +UUID_DEFINE(streaming_to_parent_msgid, 0x6e, 0x2e, 0x38, 0x39, 0x06, 0x76, 0x48, 0x96, 0x8b, 0x64, 0x60, 0x45, 0xdb, 0xf2, 0x8d, 0x66);
8 +UUID_DEFINE(health_alert_transition_msgid, 0x9c, 0xe0, 0xcb, 0x58, 0xab, 0x8b, 0x44, 0xdf, 0x82, 0xc4, 0xbf, 0x1a, 0xd9, 0xee, 0x22, 0xde);
9 +
10 +// this is also defined in alarm-notify.sh.in
11 +UUID_DEFINE(health_alert_notification_msgid, 0x6d, 0xb0, 0x01, 0x8e, 0x83, 0xe3, 0x43, 0x20, 0xae, 0x2a, 0x65, 0x9d, 0x78, 0x01, 0x9f, 0xb7);
12 +
13 +#define UUID_COMPACT_STR_LEN 33
14 +void uuid_unparse_lower_compact(const uuid_t uuid, char *out);
15 +int uuid_parse_compact(const char *in, uuid_t uuid);
16 +int uuid_parse_flexi(const char *in, uuid_t uuid);
17 +
18 +static inline int uuid_memcmp(const uuid_t *uu1, const uuid_t *uu2) {
19 + return memcmp(uu1, uu2, sizeof(uuid_t));
20 +}
21 +
22 +static inline int hex_char_to_int(char c) {
23 + if (c >= '0' && c <= '9') return c - '0';
24 + if (c >= 'a' && c <= 'f') return c - 'a' + 10;
25 + if (c >= 'A' && c <= 'F') return c - 'A' + 10;
26 + return -1; // Invalid hexadecimal character
27 +}
28 +
29 +#endif //NETDATA_UUID_H
netdata.spec.in
+12
@@ -104,6 +104,12 @@ BuildRequires: zlib-devel
104 BuildRequires: libuuid-devel
105 BuildRequires: libuv-devel >= 1
106 BuildRequires: openssl-devel
107 +BuildRequires: libcurl-devel
108 +%if 0%{?suse_version}
109 +# log2journal is not available on these systems
110 +%else
111 +BuildRequires: pcre2-devel
112 +%endif
113 %if 0%{?suse_version}
114 BuildRequires: protobuf-devel
115 BuildRequires: libprotobuf-c-devel
@@ -550,6 +556,12 @@ rm -rf "${RPM_BUILD_ROOT}"
556 %{_libdir}/%{name}
557 %{_sbindir}/%{name}
558 %{_sbindir}/netdatacli
559 +%if 0%{?suse_version}
560 +# log2journal is not available on these systems
561 +%else
562 +%{_sbindir}/log2journal
563 +%endif
564 +%{_sbindir}/systemd-cat-native
565 %{_sbindir}/%{name}-claim.sh
566
567 %{_unitdir}/%{name}.service
packaging/makeself/jobs/50-curl.install.sh
+2 -1
@@ -36,7 +36,7 @@ if [ "${CACHE_HIT:-0}" -eq 0 ]; then
36 run autoreconf -fi
37
38 run ./configure \
39 - --prefix="${NETDATA_INSTALL_PATH}" \
39 + --prefix="/curl-local" \
40 --enable-optimize \
41 --disable-shared \
42 --enable-static \
@@ -69,6 +69,7 @@ run make install
69
70 store_cache curl "${NETDATA_MAKESELF_PATH}/tmp/curl"
71
72 +cp /curl-local/bin/curl "${NETDATA_INSTALL_PATH}"/bin/curl
73 if [ "${NETDATA_BUILD_WITH_DEBUG}" -eq 0 ]; then
74 run strip "${NETDATA_INSTALL_PATH}"/bin/curl
75 fi
packaging/makeself/jobs/70-netdata-git.install.sh
+4 -4
@@ -7,12 +7,12 @@
7 cd "${NETDATA_SOURCE_PATH}" || exit 1
8
9 if [ "${NETDATA_BUILD_WITH_DEBUG}" -eq 0 ]; then
10 - export CFLAGS="-ffunction-sections -fdata-sections -static -O2 -funroll-loops -I/openssl-static/include -I/libnetfilter-acct-static/include/libnetfilter_acct -I/usr/include/libmnl -pipe"
10 + export CFLAGS="-ffunction-sections -fdata-sections -static -O2 -funroll-loops -I/openssl-static/include -I/libnetfilter-acct-static/include/libnetfilter_acct -I/curl-local/include/curl -I/usr/include/libmnl -pipe"
11 else
12 - export CFLAGS="-static -O1 -pipe -ggdb -Wall -Wextra -Wformat-signedness -DNETDATA_INTERNAL_CHECKS=1 -I/openssl-static/include -I/libnetfilter-acct-static/include/libnetfilter_acct -I/usr/include/libmnl"
12 + export CFLAGS="-static -O1 -pipe -ggdb -Wall -Wextra -Wformat-signedness -DNETDATA_INTERNAL_CHECKS=1 -I/openssl-static/include -I/libnetfilter-acct-static/include/libnetfilter_acct -I/curl-local/include/curl -I/usr/include/libmnl"
13 fi
14
15 -export LDFLAGS="-Wl,--gc-sections -static -L/openssl-static/lib64 -L/libnetfilter-acct-static/lib -lnetfilter_acct -L/usr/lib -lmnl -L/usr/lib -lzstd"
15 +export LDFLAGS="-Wl,--gc-sections -static -L/openssl-static/lib64 -L/libnetfilter-acct-static/lib -lnetfilter_acct -L/usr/lib -lmnl -L/usr/lib -lzstd -L/curl-local/lib"
16
17 # We export this to 'yes', installer sets this to .environment.
18 # The updater consumes this one, so that it can tell whether it should update a static install or a non-static one
@@ -21,7 +21,7 @@ export IS_NETDATA_STATIC_BINARY="yes"
21 # Set eBPF LIBC to "static" to bundle the `-static` variant of the kernel-collector
22 export EBPF_LIBC="static"
23 export PKG_CONFIG="pkg-config --static"
24 -export PKG_CONFIG_PATH="/openssl-static/lib64/pkgconfig:/libnetfilter-acct-static/lib/pkgconfig:/usr/lib/pkgconfig"
24 +export PKG_CONFIG_PATH="/openssl-static/lib64/pkgconfig:/libnetfilter-acct-static/lib/pkgconfig:/usr/lib/pkgconfig:/curl-local/lib/pkgconfig"
25
26 # Set correct CMake flags for building against non-System OpenSSL
27 # See: https://github.com/warmcat/libwebsockets/blob/master/READMEs/README.build.md
registry/registry.c
+8 -10
@@ -27,21 +27,19 @@ static inline void registry_unlock(void) {
27 // COOKIES
28
29 static void registry_set_cookie(struct web_client *w, const char *guid) {
30 - char e_date[100];
31 - time_t et = now_realtime_sec() + registry.persons_expiration;
32 - struct tm e_tm_buf, *etm = gmtime_r(&et, &e_tm_buf);
33 - strftime(e_date, sizeof(e_date), "%a, %d %b %Y %H:%M:%S %Z", etm);
30 + char rfc7231_expires[RFC7231_MAX_LENGTH];
31 + rfc7231_datetime(rfc7231_expires, sizeof(rfc7231_expires), now_realtime_sec() + registry.persons_expiration);
32
35 - buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s\r\n", guid, e_date);
36 - buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; SameSite=Strict; Expires=%s\r\n", guid, e_date);
33 + buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s\r\n", guid, rfc7231_expires);
34 + buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; SameSite=Strict; Expires=%s\r\n", guid, rfc7231_expires);
35 if(registry.enable_cookies_samesite_secure)
38 - buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s; SameSite=None; Secure\r\n", guid, e_date);
36 + buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s; SameSite=None; Secure\r\n", guid, rfc7231_expires);
37
38 if(registry.registry_domain && *registry.registry_domain) {
41 - buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s; Domain=%s\r\n", guid, e_date, registry.registry_domain);
42 - buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s; Domain=%s; SameSite=Strict\r\n", guid, e_date, registry.registry_domain);
39 + buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s; Domain=%s\r\n", guid, rfc7231_expires, registry.registry_domain);
40 + buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s; Domain=%s; SameSite=Strict\r\n", guid, rfc7231_expires, registry.registry_domain);
41 if(registry.enable_cookies_samesite_secure)
44 - buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s; Domain=%s; SameSite=None; Secure\r\n", guid, e_date, registry.registry_domain);
42 + buffer_sprintf(w->response.header, "Set-Cookie: " NETDATA_REGISTRY_COOKIE_NAME "=%s; Expires=%s; Domain=%s; SameSite=None; Secure\r\n", guid, rfc7231_expires, registry.registry_domain);
43 }
44
45 w->response.has_cookies = true;
registry/registry_db.c
+5 -5
@@ -112,7 +112,7 @@ int registry_db_save(void) {
112 if(unlikely(!registry_db_should_be_saved()))
113 return -2;
114
115 - error_log_limit_unlimited();
115 + nd_log_limits_unlimited();
116
117 char tmp_filename[FILENAME_MAX + 1];
118 char old_filename[FILENAME_MAX + 1];
@@ -124,7 +124,7 @@ int registry_db_save(void) {
124 FILE *fp = fopen(tmp_filename, "w");
125 if(!fp) {
126 netdata_log_error("REGISTRY: Cannot create file: %s", tmp_filename);
127 - error_log_limit_reset();
127 + nd_log_limits_reset();
128 return -1;
129 }
130
@@ -135,7 +135,7 @@ int registry_db_save(void) {
135 if(bytes1 < 0) {
136 netdata_log_error("REGISTRY: Cannot save registry machines - return value %d", bytes1);
137 fclose(fp);
138 - error_log_limit_reset();
138 + nd_log_limits_reset();
139 return bytes1;
140 }
141 netdata_log_debug(D_REGISTRY, "REGISTRY: saving machines took %d bytes", bytes1);
@@ -145,7 +145,7 @@ int registry_db_save(void) {
145 if(bytes2 < 0) {
146 netdata_log_error("REGISTRY: Cannot save registry persons - return value %d", bytes2);
147 fclose(fp);
148 - error_log_limit_reset();
148 + nd_log_limits_reset();
149 return bytes2;
150 }
151 netdata_log_debug(D_REGISTRY, "REGISTRY: saving persons took %d bytes", bytes2);
@@ -204,7 +204,7 @@ int registry_db_save(void) {
204 }
205
206 // continue operations
207 - error_log_limit_reset();
207 + nd_log_limits_reset();
208
209 return -1;
210 }
streaming/receiver.c
+145 -112
@@ -252,52 +252,6 @@ static inline bool receiver_read_compressed(struct receiver_state *r, STREAM_HAN
252 return true;
253 }
254
255 -/* Produce a full line if one exists, statefully return where we start next time.
256 - * When we hit the end of the buffer with a partial line move it to the beginning for the next fill.
257 - */
258 -inline bool buffered_reader_next_line(struct buffered_reader *reader, BUFFER *dst) {
259 - buffer_need_bytes(dst, reader->read_len - reader->pos + 2);
260 -
261 - size_t start = reader->pos;
262 -
263 - char *ss = &reader->read_buffer[start];
264 - char *se = &reader->read_buffer[reader->read_len];
265 - char *ds = &dst->buffer[dst->len];
266 - char *de = &ds[dst->size - dst->len - 2];
267 -
268 - if(ss >= se) {
269 - *ds = '\0';
270 - reader->pos = 0;
271 - reader->read_len = 0;
272 - reader->read_buffer[reader->read_len] = '\0';
273 - return false;
274 - }
275 -
276 - // copy all bytes to buffer
277 - while(ss < se && ds < de && *ss != '\n') {
278 - *ds++ = *ss++;
279 - dst->len++;
280 - }
281 -
282 - // if we have a newline, return the buffer
283 - if(ss < se && ds < de && *ss == '\n') {
284 - // newline found in the r->read_buffer
285 -
286 - *ds++ = *ss++; // copy the newline too
287 - dst->len++;
288 -
289 - *ds = '\0';
290 -
291 - reader->pos = ss - reader->read_buffer;
292 - return true;
293 - }
294 -
295 - reader->pos = 0;
296 - reader->read_len = 0;
297 - reader->read_buffer[reader->read_len] = '\0';
298 - return false;
299 -}
300 -
255 bool plugin_is_enabled(struct plugind *cd);
256
257 static void receiver_set_exit_reason(struct receiver_state *rpt, STREAM_HANDSHAKE reason, bool force) {
@@ -356,45 +310,59 @@ static size_t streaming_parser(struct receiver_state *rpt, struct plugind *cd, i
310 // so, parser needs to be allocated before pushing it
311 netdata_thread_cleanup_push(pluginsd_process_thread_cleanup, parser);
312
359 - bool compressed_connection = rrdpush_decompression_initialize(rpt);
313 + {
314 + bool compressed_connection = rrdpush_decompression_initialize(rpt);
315
361 - buffered_reader_init(&rpt->reader);
316 + buffered_reader_init(&rpt->reader);
317
318 #ifdef NETDATA_LOG_STREAM_RECEIVE
364 - {
365 - char filename[FILENAME_MAX + 1];
366 - snprintfz(filename, FILENAME_MAX, "/tmp/stream-receiver-%s.txt", rpt->host ? rrdhost_hostname(rpt->host) : "unknown");
367 - parser->user.stream_log_fp = fopen(filename, "w");
368 - parser->user.stream_log_repertoire = PARSER_REP_METADATA;
369 - }
319 + {
320 + char filename[FILENAME_MAX + 1];
321 + snprintfz(filename, FILENAME_MAX, "/tmp/stream-receiver-%s.txt", rpt->host ? rrdhost_hostname(
322 + rpt->host) : "unknown"
323 + );
324 + parser->user.stream_log_fp = fopen(filename, "w");
325 + parser->user.stream_log_repertoire = PARSER_REP_METADATA;
326 + }
327 #endif
328
372 - BUFFER *buffer = buffer_create(sizeof(rpt->reader.read_buffer), NULL);
373 - while(!receiver_should_stop(rpt)) {
329 + CLEAN_BUFFER *buffer = buffer_create(sizeof(rpt->reader.read_buffer), NULL);
330
375 - if(!buffered_reader_next_line(&rpt->reader, buffer)) {
376 - STREAM_HANDSHAKE reason = STREAM_HANDSHAKE_DISCONNECT_UNKNOWN_SOCKET_READ_ERROR;
331 + ND_LOG_STACK lgs[] = {
332 + ND_LOG_FIELD_CB(NDF_REQUEST, line_splitter_reconstruct_line, &parser->line),
333 + ND_LOG_FIELD_CB(NDF_NIDL_NODE, parser_reconstruct_node, parser),
334 + ND_LOG_FIELD_CB(NDF_NIDL_INSTANCE, parser_reconstruct_instance, parser),
335 + ND_LOG_FIELD_CB(NDF_NIDL_CONTEXT, parser_reconstruct_context, parser),
336 + ND_LOG_FIELD_END(),
337 + };
338 + ND_LOG_STACK_PUSH(lgs);
339
378 - bool have_new_data = compressed_connection ? receiver_read_compressed(rpt, &reason) : receiver_read_uncompressed(rpt, &reason);
340 + while(!receiver_should_stop(rpt)) {
341
380 - if(unlikely(!have_new_data)) {
381 - receiver_set_exit_reason(rpt, reason, false);
382 - break;
383 - }
342 + if(!buffered_reader_next_line(&rpt->reader, buffer)) {
343 + STREAM_HANDSHAKE reason = STREAM_HANDSHAKE_DISCONNECT_UNKNOWN_SOCKET_READ_ERROR;
344
385 - continue;
386 - }
345 + bool have_new_data = compressed_connection ? receiver_read_compressed(rpt, &reason)
346 + : receiver_read_uncompressed(rpt, &reason);
347
388 - if (unlikely(parser_action(parser, buffer->buffer))) {
389 - receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_PARSER_FAILED, false);
390 - break;
391 - }
348 + if(unlikely(!have_new_data)) {
349 + receiver_set_exit_reason(rpt, reason, false);
350 + break;
351 + }
352
393 - buffer->len = 0;
394 - buffer->buffer[0] = '\0';
395 - }
396 - buffer_free(buffer);
397 - result = parser->user.data_collections_count;
353 + continue;
354 + }
355 +
356 + if(unlikely(parser_action(parser, buffer->buffer))) {
357 + receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_PARSER_FAILED, false);
358 + break;
359 + }
360 +
361 + buffer->len = 0;
362 + buffer->buffer[0] = '\0';
363 + }
364 + result = parser->user.data_collections_count;
365 + }
366
367 // free parser with the pop function
368 netdata_thread_cleanup_pop(1);
@@ -435,10 +403,10 @@ static bool rrdhost_set_receiver(RRDHOST *host, struct receiver_state *rpt) {
403 if (rpt->config.health_enabled != CONFIG_BOOLEAN_NO) {
404 if (rpt->config.alarms_delay > 0) {
405 host->health.health_delay_up_to = now_realtime_sec() + rpt->config.alarms_delay;
438 - netdata_log_health(
439 - "[%s]: Postponing health checks for %" PRId64 " seconds, because it was just connected.",
440 - rrdhost_hostname(host),
441 - (int64_t) rpt->config.alarms_delay);
406 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
407 + "[%s]: Postponing health checks for %" PRId64 " seconds, because it was just connected.",
408 + rrdhost_hostname(host),
409 + (int64_t) rpt->config.alarms_delay);
410 }
411 }
412
@@ -556,26 +524,31 @@ static void rrdpush_send_error_on_taken_over_connection(struct receiver_state *r
524 5);
525 }
526
559 -void rrdpush_receive_log_status(struct receiver_state *rpt, const char *msg, const char *status) {
560 -
561 - log_stream_connection(rpt->client_ip, rpt->client_port,
562 - (rpt->key && *rpt->key)? rpt->key : "-",
563 - (rpt->machine_guid && *rpt->machine_guid) ? rpt->machine_guid : "-",
564 - (rpt->hostname && *rpt->hostname) ? rpt->hostname : "-",
565 - status);
566 -
567 - netdata_log_info("STREAM '%s' [receive from [%s]:%s]: "
568 - "%s. "
569 - "STATUS: %s%s%s%s"
570 - , rpt->hostname
571 - , rpt->client_ip, rpt->client_port
572 - , msg
573 - , status
574 - , rpt->exit.reason != STREAM_HANDSHAKE_NEVER?" (":""
575 - , stream_handshake_error_to_string(rpt->exit.reason)
576 - , rpt->exit.reason != STREAM_HANDSHAKE_NEVER?")":""
577 - );
578 -
527 +void rrdpush_receive_log_status(struct receiver_state *rpt, const char *msg, const char *status, ND_LOG_FIELD_PRIORITY priority) {
528 + // this function may be called BEFORE we spawn the receiver thread
529 + // so, we need to add the fields again (it does not harm)
530 + ND_LOG_STACK lgs[] = {
531 + ND_LOG_FIELD_TXT(NDF_SRC_IP, rpt->client_ip),
532 + ND_LOG_FIELD_TXT(NDF_SRC_PORT, rpt->client_port),
533 + ND_LOG_FIELD_TXT(NDF_NIDL_NODE, (rpt->hostname && *rpt->hostname) ? rpt->hostname : ""),
534 + ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, status),
535 + ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_from_child_msgid),
536 + ND_LOG_FIELD_END(),
537 + };
538 + ND_LOG_STACK_PUSH(lgs);
539 +
540 + nd_log(NDLS_ACCESS, priority, "api_key:'%s' machine_guid:'%s' msg:'%s'"
541 + , (rpt->key && *rpt->key)? rpt->key : ""
542 + , (rpt->machine_guid && *rpt->machine_guid) ? rpt->machine_guid : ""
543 + , msg);
544 +
545 + nd_log(NDLS_DAEMON, priority, "STREAM_RECEIVER for '%s': %s %s%s%s"
546 + , (rpt->hostname && *rpt->hostname) ? rpt->hostname : ""
547 + , msg
548 + , rpt->exit.reason != STREAM_HANDSHAKE_NEVER?" (":""
549 + , stream_handshake_error_to_string(rpt->exit.reason)
550 + , rpt->exit.reason != STREAM_HANDSHAKE_NEVER?")":""
551 + );
552 }
553
554 static void rrdpush_receive(struct receiver_state *rpt)
@@ -688,13 +661,19 @@ static void rrdpush_receive(struct receiver_state *rpt)
661 );
662
663 if(!host) {
691 - rrdpush_receive_log_status(rpt, "failed to find/create host structure", "INTERNAL ERROR DROPPING CONNECTION");
664 + rrdpush_receive_log_status(
665 + rpt,"failed to find/create host structure, rejecting connection",
666 + RRDPUSH_STATUS_INTERNAL_SERVER_ERROR, NDLP_ERR);
667 +
668 rrdpush_send_error_on_taken_over_connection(rpt, START_STREAMING_ERROR_INTERNAL_ERROR);
669 goto cleanup;
670 }
671
672 if (unlikely(rrdhost_flag_check(host, RRDHOST_FLAG_PENDING_CONTEXT_LOAD))) {
697 - rrdpush_receive_log_status(rpt, "host is initializing", "INITIALIZATION IN PROGRESS RETRY LATER");
673 + rrdpush_receive_log_status(
674 + rpt, "host is initializing, retry later",
675 + RRDPUSH_STATUS_INITIALIZATION_IN_PROGRESS, NDLP_NOTICE);
676 +
677 rrdpush_send_error_on_taken_over_connection(rpt, START_STREAMING_ERROR_INITIALIZATION);
678 goto cleanup;
679 }
@@ -703,7 +682,10 @@ static void rrdpush_receive(struct receiver_state *rpt)
682 rpt->system_info = NULL;
683
684 if(!rrdhost_set_receiver(host, rpt)) {
706 - rrdpush_receive_log_status(rpt, "host is already served by another receiver", "DUPLICATE RECEIVER DROPPING CONNECTION");
685 + rrdpush_receive_log_status(
686 + rpt, "host is already served by another receiver",
687 + RRDPUSH_STATUS_DUPLICATE_RECEIVER, NDLP_INFO);
688 +
689 rrdpush_send_error_on_taken_over_connection(rpt, START_STREAMING_ERROR_ALREADY_STREAMING);
690 goto cleanup;
691 }
@@ -784,7 +766,9 @@ static void rrdpush_receive(struct receiver_state *rpt)
766
767 if(bytes_sent != (ssize_t)strlen(initial_response)) {
768 internal_error(true, "Cannot send response, got %zd bytes, expecting %zu bytes", bytes_sent, strlen(initial_response));
787 - rrdpush_receive_log_status(rpt, "cannot reply back", "CANT REPLY DROPPING CONNECTION");
769 + rrdpush_receive_log_status(
770 + rpt, "cannot reply back, dropping connection",
771 + RRDPUSH_STATUS_CANT_REPLY, NDLP_ERR);
772 goto cleanup;
773 }
774 #ifdef ENABLE_H2O
@@ -815,7 +799,9 @@ static void rrdpush_receive(struct receiver_state *rpt)
799 , rpt->fd);
800 }
801
818 - rrdpush_receive_log_status(rpt, "ready to receive data", "CONNECTED");
802 + rrdpush_receive_log_status(
803 + rpt, "connected and ready to receive data",
804 + RRDPUSH_STATUS_CONNECTED, NDLP_INFO);
805
806 #ifdef ENABLE_ACLK
807 // in case we have cloud connection we inform cloud
@@ -842,7 +828,9 @@ static void rrdpush_receive(struct receiver_state *rpt)
828 {
829 char msg[100 + 1];
830 snprintfz(msg, 100, "disconnected (completed %zu updates)", count);
845 - rrdpush_receive_log_status(rpt, msg, "DISCONNECTED");
831 + rrdpush_receive_log_status(
832 + rpt, msg,
833 + RRDPUSH_STATUS_DISCONNECTED, NDLP_WARNING);
834 }
835
836 #ifdef ENABLE_ACLK
@@ -873,19 +861,64 @@ static void rrdpush_receiver_thread_cleanup(void *ptr) {
861 rrdhost_set_is_parent_label();
862 }
863
864 +static bool stream_receiver_log_capabilities(BUFFER *wb, void *ptr) {
865 + struct receiver_state *rpt = ptr;
866 + if(!rpt)
867 + return false;
868 +
869 + stream_capabilities_to_string(wb, rpt->capabilities);
870 + return true;
871 +}
872 +
873 +static bool stream_receiver_log_transport(BUFFER *wb, void *ptr) {
874 + struct receiver_state *rpt = ptr;
875 + if(!rpt)
876 + return false;
877 +
878 +#ifdef ENABLE_HTTPS
879 + buffer_strcat(wb, SSL_connection(&rpt->ssl) ? "https" : "http");
880 +#else
881 + buffer_strcat(wb, "http");
882 +#endif
883 + return true;
884 +}
885 +
886 void *rrdpush_receiver_thread(void *ptr) {
887 netdata_thread_cleanup_push(rrdpush_receiver_thread_cleanup, ptr);
888
879 - worker_register("STREAMRCV");
880 - worker_register_job_custom_metric(WORKER_RECEIVER_JOB_BYTES_READ, "received bytes", "bytes/s", WORKER_METRIC_INCREMENT);
881 - worker_register_job_custom_metric(WORKER_RECEIVER_JOB_BYTES_UNCOMPRESSED, "uncompressed bytes", "bytes/s", WORKER_METRIC_INCREMENT);
882 - worker_register_job_custom_metric(WORKER_RECEIVER_JOB_REPLICATION_COMPLETION, "replication completion", "%", WORKER_METRIC_ABSOLUTE);
889 + {
890 + worker_register("STREAMRCV");
891
884 - struct receiver_state *rpt = (struct receiver_state *)ptr;
885 - rpt->tid = gettid();
886 - netdata_log_info("STREAM %s [%s]:%s: receive thread created (task id %d)", rpt->hostname, rpt->client_ip, rpt->client_port, rpt->tid);
892 + worker_register_job_custom_metric(WORKER_RECEIVER_JOB_BYTES_READ,
893 + "received bytes", "bytes/s",
894 + WORKER_METRIC_INCREMENT);
895
888 - rrdpush_receive(rpt);
896 + worker_register_job_custom_metric(WORKER_RECEIVER_JOB_BYTES_UNCOMPRESSED,
897 + "uncompressed bytes", "bytes/s",
898 + WORKER_METRIC_INCREMENT);
899 +
900 + worker_register_job_custom_metric(WORKER_RECEIVER_JOB_REPLICATION_COMPLETION,
901 + "replication completion", "%",
902 + WORKER_METRIC_ABSOLUTE);
903 +
904 + struct receiver_state *rpt = (struct receiver_state *) ptr;
905 + rpt->tid = gettid();
906 +
907 + ND_LOG_STACK lgs[] = {
908 + ND_LOG_FIELD_TXT(NDF_SRC_IP, rpt->client_ip),
909 + ND_LOG_FIELD_TXT(NDF_SRC_PORT, rpt->client_port),
910 + ND_LOG_FIELD_TXT(NDF_NIDL_NODE, rpt->hostname),
911 + ND_LOG_FIELD_CB(NDF_SRC_TRANSPORT, stream_receiver_log_transport, rpt),
912 + ND_LOG_FIELD_CB(NDF_SRC_CAPABILITIES, stream_receiver_log_capabilities, rpt),
913 + ND_LOG_FIELD_END(),
914 + };
915 + ND_LOG_STACK_PUSH(lgs);
916 +
917 + netdata_log_info("STREAM %s [%s]:%s: receive thread started", rpt->hostname, rpt->client_ip
918 + , rpt->client_port);
919 +
920 + rrdpush_receive(rpt);
921 + }
922
923 netdata_thread_cleanup_pop(1);
924 return NULL;
streaming/replication.c
+13 -12
@@ -352,8 +352,8 @@ static bool replication_query_execute(BUFFER *wb, struct replication_query *q, s
352 if(max_skip <= 0) {
353 d->skip = true;
354
355 - error_limit_static_global_var(erl, 1, 0);
356 - error_limit(&erl,
355 + nd_log_limit_static_global_var(erl, 1, 0);
356 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR,
357 "STREAM_SENDER REPLAY ERROR: 'host:%s/chart:%s/dim:%s': db does not advance the query "
358 "beyond time %llu (tried 1000 times to get the next point and always got back a point in the past)",
359 rrdhost_hostname(q->st->rrdhost), rrdset_id(q->st), rrddim_id(d->rd),
@@ -402,14 +402,15 @@ static bool replication_query_execute(BUFFER *wb, struct replication_query *q, s
402 fix_min_start_time = min_end_time - min_update_every;
403
404 #ifdef NETDATA_INTERNAL_CHECKS
405 - error_limit_static_global_var(erl, 1, 0);
406 - error_limit(&erl, "REPLAY WARNING: 'host:%s/chart:%s' "
407 - "misaligned dimensions, "
408 - "update every (min: %ld, max: %ld), "
409 - "start time (min: %ld, max: %ld), "
410 - "end time (min %ld, max %ld), "
411 - "now %ld, last end time sent %ld, "
412 - "min start time is fixed to %ld",
405 + nd_log_limit_static_global_var(erl, 1, 0);
406 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_WARNING,
407 + "REPLAY WARNING: 'host:%s/chart:%s' "
408 + "misaligned dimensions, "
409 + "update every (min: %ld, max: %ld), "
410 + "start time (min: %ld, max: %ld), "
411 + "end time (min %ld, max %ld), "
412 + "now %ld, last end time sent %ld, "
413 + "min start time is fixed to %ld",
414 rrdhost_hostname(q->st->rrdhost), rrdset_id(q->st),
415 min_update_every, max_update_every,
416 min_start_time, max_start_time,
@@ -757,8 +758,8 @@ static void replicate_log_request(struct replication_request_details *r, const c
758 #ifdef NETDATA_INTERNAL_CHECKS
759 internal_error(true,
760 #else
760 - error_limit_static_global_var(erl, 1, 0);
761 - error_limit(&erl,
761 + nd_log_limit_static_global_var(erl, 1, 0);
762 + nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR,
763 #endif
764 "REPLAY ERROR: 'host:%s/chart:%s' child sent: "
765 "db from %ld to %ld%s, wall clock time %ld, "
streaming/rrdpush.c
+57 -83
@@ -57,12 +57,12 @@ static void load_stream_conf() {
57 errno = 0;
58 char *filename = strdupz_path_subpath(netdata_configured_user_config_dir, "stream.conf");
59 if(!appconfig_load(&stream_config, filename, 0, NULL)) {
60 - netdata_log_info("CONFIG: cannot load user config '%s'. Will try stock config.", filename);
60 + nd_log_daemon(NDLP_NOTICE, "CONFIG: cannot load user config '%s'. Will try stock config.", filename);
61 freez(filename);
62
63 filename = strdupz_path_subpath(netdata_configured_stock_config_dir, "stream.conf");
64 if(!appconfig_load(&stream_config, filename, 0, NULL))
65 - netdata_log_info("CONFIG: cannot load stock config '%s'. Running with internal defaults.", filename);
65 + nd_log_daemon(NDLP_NOTICE, "CONFIG: cannot load stock config '%s'. Running with internal defaults.", filename);
66 }
67 freez(filename);
68 }
@@ -128,7 +128,7 @@ int rrdpush_init() {
128 rrdpush_compression_levels[COMPRESSION_ALGORITHM_GZIP]);
129
130 if(default_rrdpush_enabled && (!default_rrdpush_destination || !*default_rrdpush_destination || !default_rrdpush_api_key || !*default_rrdpush_api_key)) {
131 - netdata_log_error("STREAM [send]: cannot enable sending thread - information is missing.");
131 + nd_log_daemon(NDLP_WARNING, "STREAM [send]: cannot enable sending thread - information is missing.");
132 default_rrdpush_enabled = 0;
133 }
134
@@ -136,7 +136,7 @@ int rrdpush_init() {
136 netdata_ssl_validate_certificate_sender = !appconfig_get_boolean(&stream_config, CONFIG_SECTION_STREAM, "ssl skip certificate verification", !netdata_ssl_validate_certificate);
137
138 if(!netdata_ssl_validate_certificate_sender)
139 - netdata_log_info("SSL: streaming senders will skip SSL certificates verification.");
139 + nd_log_daemon(NDLP_NOTICE, "SSL: streaming senders will skip SSL certificates verification.");
140
141 netdata_ssl_ca_path = appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "CApath", NULL);
142 netdata_ssl_ca_file = appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "CAfile", NULL);
@@ -542,13 +542,13 @@ RRDSET_STREAM_BUFFER rrdset_push_metric_initialize(RRDSET *st, time_t wall_clock
542
543 if(unlikely(!(host_flags & RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS))) {
544 rrdhost_flag_set(host, RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS);
545 - netdata_log_error("STREAM %s [send]: not ready - collected metrics are not sent to parent.", rrdhost_hostname(host));
545 + nd_log_daemon(NDLP_NOTICE, "STREAM %s [send]: not ready - collected metrics are not sent to parent.", rrdhost_hostname(host));
546 }
547
548 return (RRDSET_STREAM_BUFFER) { .wb = NULL, };
549 }
550 else if(unlikely(host_flags & RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS)) {
551 - netdata_log_info("STREAM %s [send]: sending metrics to parent...", rrdhost_hostname(host));
551 + nd_log_daemon(NDLP_INFO, "STREAM %s [send]: sending metrics to parent...", rrdhost_hostname(host));
552 rrdhost_flag_clear(host, RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS);
553 }
554
@@ -741,11 +741,9 @@ int connect_to_one_of_destinations(
741 if(d->postpone_reconnection_until > now)
742 continue;
743
744 - internal_error(true,
744 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
745 "STREAM %s: connecting to '%s' (default port: %d)...",
746 - rrdhost_hostname(host),
747 - string2str(d->destination),
748 - default_port);
746 + rrdhost_hostname(host), string2str(d->destination), default_port);
747
748 if (reconnects_counter)
749 *reconnects_counter += 1;
@@ -798,7 +796,7 @@ bool destinations_init_add_one(char *entry, void *data) {
796 DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(t->list, d, prev, next);
797
798 t->count++;
801 - netdata_log_info("STREAM: added streaming destination No %d: '%s' to host '%s'", t->count, string2str(d->destination), rrdhost_hostname(t->host));
799 + nd_log_daemon(NDLP_INFO, "STREAM: added streaming destination No %d: '%s' to host '%s'", t->count, string2str(d->destination), rrdhost_hostname(t->host));
800
801 return false; // we return false, so that we will get all defined destinations
802 }
@@ -867,11 +865,6 @@ void rrdpush_sender_thread_stop(RRDHOST *host, STREAM_HANDSHAKE reason, bool wai
865 // ----------------------------------------------------------------------------
866 // rrdpush receiver thread
867
870 -void log_stream_connection(const char *client_ip, const char *client_port, const char *api_key, const char *machine_guid, const char *host, const char *msg) {
871 - netdata_log_access("STREAM: %d '[%s]:%s' '%s' host '%s' api key '%s' machine guid '%s'", gettid(), client_ip, client_port, msg, host, api_key, machine_guid);
872 -}
873 -
874 -
868 static void rrdpush_sender_thread_spawn(RRDHOST *host) {
869 sender_lock(host->sender);
870
@@ -880,7 +873,7 @@ static void rrdpush_sender_thread_spawn(RRDHOST *host) {
873 snprintfz(tag, NETDATA_THREAD_TAG_MAX, THREAD_TAG_STREAM_SENDER "[%s]", rrdhost_hostname(host));
874
875 if(netdata_thread_create(&host->rrdpush_sender_thread, tag, NETDATA_THREAD_OPTION_DEFAULT, rrdpush_sender_thread, (void *) host->sender))
883 - netdata_log_error("STREAM %s [send]: failed to create new thread for client.", rrdhost_hostname(host));
876 + nd_log_daemon(NDLP_ERR, "STREAM %s [send]: failed to create new thread for client.", rrdhost_hostname(host));
877 else
878 rrdhost_flag_set(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN);
879 }
@@ -1040,7 +1033,7 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1033 rpt->capabilities = convert_stream_version_to_capabilities(1, NULL, false);
1034
1035 if (unlikely(rrdhost_set_system_info_variable(rpt->system_info, name, value))) {
1043 - netdata_log_info("STREAM '%s' [receive from [%s]:%s]: "
1036 + nd_log_daemon(NDLP_NOTICE, "STREAM '%s' [receive from [%s]:%s]: "
1037 "request has parameter '%s' = '%s', which is not used."
1038 , (rpt->hostname && *rpt->hostname) ? rpt->hostname : "-"
1039 , rpt->client_ip, rpt->client_port
@@ -1069,9 +1062,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1062
1063 if(!rpt->key || !*rpt->key) {
1064 rrdpush_receive_log_status(
1072 - rpt,
1073 - "request without an API key",
1074 - "NO API KEY PERMISSION DENIED");
1065 + rpt, "request without an API key, rejecting connection",
1066 + RRDPUSH_STATUS_NO_API_KEY, NDLP_WARNING);
1067
1068 receiver_state_free(rpt);
1069 return rrdpush_receiver_permission_denied(w);
@@ -1079,9 +1071,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1071
1072 if(!rpt->hostname || !*rpt->hostname) {
1073 rrdpush_receive_log_status(
1082 - rpt,
1083 - "request without a hostname",
1084 - "NO HOSTNAME PERMISSION DENIED");
1074 + rpt, "request without a hostname, rejecting connection",
1075 + RRDPUSH_STATUS_NO_HOSTNAME, NDLP_WARNING);
1076
1077 receiver_state_free(rpt);
1078 return rrdpush_receiver_permission_denied(w);
@@ -1092,9 +1083,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1083
1084 if(!rpt->machine_guid || !*rpt->machine_guid) {
1085 rrdpush_receive_log_status(
1095 - rpt,
1096 - "request without a machine GUID",
1097 - "NO MACHINE GUID PERMISSION DENIED");
1086 + rpt, "request without a machine GUID, rejecting connection",
1087 + RRDPUSH_STATUS_NO_MACHINE_GUID, NDLP_WARNING);
1088
1089 receiver_state_free(rpt);
1090 return rrdpush_receiver_permission_denied(w);
@@ -1105,9 +1095,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1095
1096 if (regenerate_guid(rpt->key, buf) == -1) {
1097 rrdpush_receive_log_status(
1108 - rpt,
1109 - "API key is not a valid UUID (use the command uuidgen to generate one)",
1110 - "INVALID API KEY PERMISSION DENIED");
1098 + rpt, "API key is not a valid UUID (use the command uuidgen to generate one)",
1099 + RRDPUSH_STATUS_INVALID_API_KEY, NDLP_WARNING);
1100
1101 receiver_state_free(rpt);
1102 return rrdpush_receiver_permission_denied(w);
@@ -1115,9 +1104,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1104
1105 if (regenerate_guid(rpt->machine_guid, buf) == -1) {
1106 rrdpush_receive_log_status(
1118 - rpt,
1119 - "machine GUID is not a valid UUID",
1120 - "INVALID MACHINE GUID PERMISSION DENIED");
1107 + rpt, "machine GUID is not a valid UUID",
1108 + RRDPUSH_STATUS_INVALID_MACHINE_GUID, NDLP_WARNING);
1109
1110 receiver_state_free(rpt);
1111 return rrdpush_receiver_permission_denied(w);
@@ -1128,9 +1116,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1116 if(!api_key_type || !*api_key_type) api_key_type = "unknown";
1117 if(strcmp(api_key_type, "api") != 0) {
1118 rrdpush_receive_log_status(
1131 - rpt,
1132 - "API key is a machine GUID",
1133 - "INVALID API KEY PERMISSION DENIED");
1119 + rpt, "API key is a machine GUID",
1120 + RRDPUSH_STATUS_INVALID_API_KEY, NDLP_WARNING);
1121
1122 receiver_state_free(rpt);
1123 return rrdpush_receiver_permission_denied(w);
@@ -1138,9 +1125,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1125
1126 if(!appconfig_get_boolean(&stream_config, rpt->key, "enabled", 0)) {
1127 rrdpush_receive_log_status(
1141 - rpt,
1142 - "API key is not enabled",
1143 - "API KEY DISABLED PERMISSION DENIED");
1128 + rpt, "API key is not enabled",
1129 + RRDPUSH_STATUS_API_KEY_DISABLED, NDLP_WARNING);
1130
1131 receiver_state_free(rpt);
1132 return rrdpush_receiver_permission_denied(w);
@@ -1156,9 +1142,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1142 simple_pattern_free(key_allow_from);
1143
1144 rrdpush_receive_log_status(
1159 - rpt,
1160 - "API key is not allowed from this IP",
1161 - "NOT ALLOWED IP PERMISSION DENIED");
1145 + rpt, "API key is not allowed from this IP",
1146 + RRDPUSH_STATUS_NOT_ALLOWED_IP, NDLP_WARNING);
1147
1148 receiver_state_free(rpt);
1149 return rrdpush_receiver_permission_denied(w);
@@ -1174,9 +1159,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1159
1160 if (strcmp(machine_guid_type, "machine") != 0) {
1161 rrdpush_receive_log_status(
1177 - rpt,
1178 - "machine GUID is an API key",
1179 - "INVALID MACHINE GUID PERMISSION DENIED");
1162 + rpt, "machine GUID is an API key",
1163 + RRDPUSH_STATUS_INVALID_MACHINE_GUID, NDLP_WARNING);
1164
1165 receiver_state_free(rpt);
1166 return rrdpush_receiver_permission_denied(w);
@@ -1185,9 +1169,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1169
1170 if(!appconfig_get_boolean(&stream_config, rpt->machine_guid, "enabled", 1)) {
1171 rrdpush_receive_log_status(
1188 - rpt,
1189 - "machine GUID is not enabled",
1190 - "MACHINE GUID DISABLED PERMISSION DENIED");
1172 + rpt, "machine GUID is not enabled",
1173 + RRDPUSH_STATUS_MACHINE_GUID_DISABLED, NDLP_WARNING);
1174
1175 receiver_state_free(rpt);
1176 return rrdpush_receiver_permission_denied(w);
@@ -1203,9 +1186,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1186 simple_pattern_free(machine_allow_from);
1187
1188 rrdpush_receive_log_status(
1206 - rpt,
1207 - "machine GUID is not allowed from this IP",
1208 - "NOT ALLOWED IP PERMISSION DENIED");
1189 + rpt, "machine GUID is not allowed from this IP",
1190 + RRDPUSH_STATUS_NOT_ALLOWED_IP, NDLP_WARNING);
1191
1192 receiver_state_free(rpt);
1193 return rrdpush_receiver_permission_denied(w);
@@ -1220,9 +1202,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1202 rrdpush_receiver_takeover_web_connection(w, rpt);
1203
1204 rrdpush_receive_log_status(
1223 - rpt,
1224 - "machine GUID is my own",
1225 - "LOCALHOST PERMISSION DENIED");
1205 + rpt, "machine GUID is my own",
1206 + RRDPUSH_STATUS_LOCALHOST, NDLP_DEBUG);
1207
1208 char initial_response[HTTP_HEADER_SIZE + 1];
1209 snprintfz(initial_response, HTTP_HEADER_SIZE, "%s", START_STREAMING_ERROR_SAME_LOCALHOST);
@@ -1233,11 +1214,11 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1214 #endif
1215 rpt->fd, initial_response, strlen(initial_response), 0, 60) != (ssize_t)strlen(initial_response)) {
1216
1236 - netdata_log_error("STREAM '%s' [receive from [%s]:%s]: "
1237 - "failed to reply."
1238 - , rpt->hostname
1239 - , rpt->client_ip, rpt->client_port
1240 - );
1217 + nd_log_daemon(NDLP_ERR, "STREAM '%s' [receive from [%s]:%s]: "
1218 + "failed to reply."
1219 + , rpt->hostname
1220 + , rpt->client_ip, rpt->client_port
1221 + );
1222 }
1223
1224 receiver_state_free(rpt);
@@ -1263,9 +1244,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1244 (long)(web_client_streaming_rate_t - (now - last_stream_accepted_t)));
1245
1246 rrdpush_receive_log_status(
1266 - rpt,
1267 - msg,
1268 - "RATE LIMIT TRY LATER");
1247 + rpt, msg,
1248 + RRDPUSH_STATUS_RATE_LIMIT, NDLP_NOTICE);
1249
1250 receiver_state_free(rpt);
1251 return rrdpush_receiver_too_busy_now(w);
@@ -1313,29 +1293,26 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1293 // we can proceed with this connection
1294 receiver_stale = false;
1295
1316 - netdata_log_info("STREAM '%s' [receive from [%s]:%s]: "
1317 - "stopped previous stale receiver to accept this one."
1318 - , rpt->hostname
1319 - , rpt->client_ip, rpt->client_port
1320 - );
1296 + nd_log_daemon(NDLP_NOTICE, "STREAM '%s' [receive from [%s]:%s]: "
1297 + "stopped previous stale receiver to accept this one."
1298 + , rpt->hostname
1299 + , rpt->client_ip, rpt->client_port
1300 + );
1301 }
1302
1303 if (receiver_working || receiver_stale) {
1304 // another receiver is already connected
1305 // try again later
1306
1327 -#ifdef NETDATA_INTERNAL_CHECKS
1307 char msg[200 + 1];
1308 snprintfz(msg, 200,
1309 "multiple connections for same host, "
1331 - "old connection was used %ld secs ago%s",
1310 + "old connection was last used %ld secs ago%s",
1311 age, receiver_stale ? " (signaled old receiver to stop)" : " (new connection not accepted)");
1312
1313 rrdpush_receive_log_status(
1335 - rpt,
1336 - msg,
1337 - "ALREADY CONNECTED");
1338 -#endif
1314 + rpt, msg,
1315 + RRDPUSH_STATUS_ALREADY_CONNECTED, NDLP_DEBUG);
1316
1317 // Have not set WEB_CLIENT_FLAG_DONT_CLOSE_SOCKET - caller should clean up
1318 buffer_flush(w->response.data);
@@ -1345,8 +1322,6 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1322 }
1323 }
1324
1348 - netdata_log_debug(D_SYSTEM, "starting STREAM receive thread.");
1349 -
1325 rrdpush_receiver_takeover_web_connection(w, rpt);
1326
1327 char tag[NETDATA_THREAD_TAG_MAX + 1];
@@ -1355,9 +1330,8 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
1330
1331 if(netdata_thread_create(&rpt->thread, tag, NETDATA_THREAD_OPTION_DEFAULT, rrdpush_receiver_thread, (void *)rpt)) {
1332 rrdpush_receive_log_status(
1358 - rpt,
1359 - "can't create receiver thread",
1360 - "INTERNAL SERVER ERROR");
1333 + rpt, "can't create receiver thread",
1334 + RRDPUSH_STATUS_INTERNAL_SERVER_ERROR, NDLP_ERR);
1335
1336 buffer_flush(w->response.data);
1337 buffer_strcat(w->response.data, "Can't handle this request");
@@ -1452,7 +1426,7 @@ static struct {
1426 {0 , NULL },
1427 };
1428
1455 -static void stream_capabilities_to_string(BUFFER *wb, STREAM_CAPABILITIES caps) {
1429 +void stream_capabilities_to_string(BUFFER *wb, STREAM_CAPABILITIES caps) {
1430 for(size_t i = 0; capability_names[i].str ; i++) {
1431 if(caps & capability_names[i].cap) {
1432 buffer_strcat(wb, capability_names[i].str);
@@ -1479,8 +1453,8 @@ void log_receiver_capabilities(struct receiver_state *rpt) {
1453 BUFFER *wb = buffer_create(100, NULL);
1454 stream_capabilities_to_string(wb, rpt->capabilities);
1455
1482 - netdata_log_info("STREAM %s [receive from [%s]:%s]: established link with negotiated capabilities: %s",
1483 - rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port, buffer_tostring(wb));
1456 + nd_log_daemon(NDLP_INFO, "STREAM %s [receive from [%s]:%s]: established link with negotiated capabilities: %s",
1457 + rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port, buffer_tostring(wb));
1458
1459 buffer_free(wb);
1460 }
@@ -1489,8 +1463,8 @@ void log_sender_capabilities(struct sender_state *s) {
1463 BUFFER *wb = buffer_create(100, NULL);
1464 stream_capabilities_to_string(wb, s->capabilities);
1465
1492 - netdata_log_info("STREAM %s [send to %s]: established link with negotiated capabilities: %s",
1493 - rrdhost_hostname(s->host), s->connected_to, buffer_tostring(wb));
1466 + nd_log_daemon(NDLP_INFO, "STREAM %s [send to %s]: established link with negotiated capabilities: %s",
1467 + rrdhost_hostname(s->host), s->connected_to, buffer_tostring(wb));
1468
1469 buffer_free(wb);
1470 }
streaming/rrdpush.h
+28 -15
@@ -106,6 +106,31 @@ static inline bool stream_has_more_than_one_capability_of(STREAM_CAPABILITIES ca
106 #define START_STREAMING_ERROR_INTERNAL_ERROR "The server encountered an internal error. Try later."
107 #define START_STREAMING_ERROR_INITIALIZATION "The server is initializing. Try later."
108
109 +#define RRDPUSH_STATUS_CONNECTED "CONNECTED"
110 +#define RRDPUSH_STATUS_ALREADY_CONNECTED "ALREADY CONNECTED"
111 +#define RRDPUSH_STATUS_DISCONNECTED "DISCONNECTED"
112 +#define RRDPUSH_STATUS_RATE_LIMIT "RATE LIMIT TRY LATER"
113 +#define RRDPUSH_STATUS_INITIALIZATION_IN_PROGRESS "INITIALIZATION IN PROGRESS RETRY LATER"
114 +#define RRDPUSH_STATUS_INTERNAL_SERVER_ERROR "INTERNAL SERVER ERROR DROPPING CONNECTION"
115 +#define RRDPUSH_STATUS_DUPLICATE_RECEIVER "DUPLICATE RECEIVER DROPPING CONNECTION"
116 +#define RRDPUSH_STATUS_CANT_REPLY "CANT REPLY DROPPING CONNECTION"
117 +#define RRDPUSH_STATUS_NO_HOSTNAME "NO HOSTNAME PERMISSION DENIED"
118 +#define RRDPUSH_STATUS_NO_API_KEY "NO API KEY PERMISSION DENIED"
119 +#define RRDPUSH_STATUS_INVALID_API_KEY "INVALID API KEY PERMISSION DENIED"
120 +#define RRDPUSH_STATUS_NO_MACHINE_GUID "NO MACHINE GUID PERMISSION DENIED"
121 +#define RRDPUSH_STATUS_MACHINE_GUID_DISABLED "MACHINE GUID DISABLED PERMISSION DENIED"
122 +#define RRDPUSH_STATUS_INVALID_MACHINE_GUID "INVALID MACHINE GUID PERMISSION DENIED"
123 +#define RRDPUSH_STATUS_API_KEY_DISABLED "API KEY DISABLED PERMISSION DENIED"
124 +#define RRDPUSH_STATUS_NOT_ALLOWED_IP "NOT ALLOWED IP PERMISSION DENIED"
125 +#define RRDPUSH_STATUS_LOCALHOST "LOCALHOST PERMISSION DENIED"
126 +#define RRDPUSH_STATUS_PERMISSION_DENIED "PERMISSION DENIED"
127 +#define RRDPUSH_STATUS_BAD_HANDSHAKE "BAD HANDSHAKE"
128 +#define RRDPUSH_STATUS_TIMEOUT "TIMEOUT"
129 +#define RRDPUSH_STATUS_CANT_UPGRADE_CONNECTION "CANT UPGRADE CONNECTION"
130 +#define RRDPUSH_STATUS_SSL_ERROR "SSL ERROR"
131 +#define RRDPUSH_STATUS_INVALID_SSL_CERTIFICATE "INVALID SSL CERTIFICATE"
132 +#define RRDPUSH_STATUS_CANT_ESTABLISH_SSL_CONNECTION "CANT ESTABLISH SSL CONNECTION"
133 +
134 typedef enum {
135 STREAM_HANDSHAKE_OK_V3 = 3, // v3+
136 STREAM_HANDSHAKE_OK_V2 = 2, // v2
@@ -214,6 +239,7 @@ struct sender_state {
239
240 uint16_t hops;
241
242 + struct line_splitter line;
243 struct compressor_state compressor;
244
245 #ifdef NETDATA_LOG_STREAM_SENDER
@@ -306,19 +332,6 @@ typedef struct stream_node_instance {
332 } STREAM_NODE_INSTANCE;
333 */
334
309 -struct buffered_reader {
310 - ssize_t read_len;
311 - ssize_t pos;
312 - char read_buffer[PLUGINSD_LINE_MAX + 1];
313 -};
314 -
315 -bool buffered_reader_next_line(struct buffered_reader *reader, BUFFER *dst);
316 -static inline void buffered_reader_init(struct buffered_reader *reader) {
317 - reader->read_buffer[0] = '\0';
318 - reader->read_len = 0;
319 - reader->pos = 0;
320 -}
321 -
335 struct receiver_state {
336 RRDHOST *host;
337 pid_t tid;
@@ -452,7 +465,6 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_stri
465 void rrdpush_sender_thread_stop(RRDHOST *host, STREAM_HANDSHAKE reason, bool wait);
466
467 void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, const RRDVAR_ACQUIRED *rva);
455 -void log_stream_connection(const char *client_ip, const char *client_port, const char *api_key, const char *machine_guid, const char *host, const char *msg);
468 int connect_to_one_of_destinations(
469 RRDHOST *host,
470 int default_port,
@@ -467,11 +479,12 @@ void rrdpush_signal_sender_to_wake_up(struct sender_state *s);
479 void rrdpush_reset_destinations_postpone_time(RRDHOST *host);
480 const char *stream_handshake_error_to_string(STREAM_HANDSHAKE handshake_error);
481 void stream_capabilities_to_json_array(BUFFER *wb, STREAM_CAPABILITIES caps, const char *key);
470 -void rrdpush_receive_log_status(struct receiver_state *rpt, const char *msg, const char *status);
482 +void rrdpush_receive_log_status(struct receiver_state *rpt, const char *msg, const char *status, ND_LOG_FIELD_PRIORITY priority);
483 void log_receiver_capabilities(struct receiver_state *rpt);
484 void log_sender_capabilities(struct sender_state *s);
485 STREAM_CAPABILITIES convert_stream_version_to_capabilities(int32_t version, RRDHOST *host, bool sender);
486 int32_t stream_capabilities_to_vn(uint32_t caps);
487 +void stream_capabilities_to_string(BUFFER *wb, STREAM_CAPABILITIES caps);
488
489 void receiver_state_free(struct receiver_state *rpt);
490 bool stop_streaming_receiver(RRDHOST *host, STREAM_HANDSHAKE reason);
streaming/sender.c
+247 -91
@@ -4,32 +4,33 @@
4 #include "common.h"
5 #include "aclk/https_client.h"
6
7 -#define WORKER_SENDER_JOB_CONNECT 0
8 -#define WORKER_SENDER_JOB_PIPE_READ 1
9 -#define WORKER_SENDER_JOB_SOCKET_RECEIVE 2
10 -#define WORKER_SENDER_JOB_EXECUTE 3
11 -#define WORKER_SENDER_JOB_SOCKET_SEND 4
12 -#define WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE 5
13 -#define WORKER_SENDER_JOB_DISCONNECT_OVERFLOW 6
14 -#define WORKER_SENDER_JOB_DISCONNECT_TIMEOUT 7
15 -#define WORKER_SENDER_JOB_DISCONNECT_POLL_ERROR 8
16 -#define WORKER_SENDER_JOB_DISCONNECT_SOCKET_ERROR 9
17 -#define WORKER_SENDER_JOB_DISCONNECT_SSL_ERROR 10
18 -#define WORKER_SENDER_JOB_DISCONNECT_PARENT_CLOSED 11
19 -#define WORKER_SENDER_JOB_DISCONNECT_RECEIVE_ERROR 12
20 -#define WORKER_SENDER_JOB_DISCONNECT_SEND_ERROR 13
21 -#define WORKER_SENDER_JOB_DISCONNECT_NO_COMPRESSION 14
22 -#define WORKER_SENDER_JOB_BUFFER_RATIO 15
23 -#define WORKER_SENDER_JOB_BYTES_RECEIVED 16
24 -#define WORKER_SENDER_JOB_BYTES_SENT 17
25 -#define WORKER_SENDER_JOB_BYTES_COMPRESSED 18
26 -#define WORKER_SENDER_JOB_BYTES_UNCOMPRESSED 19
27 -#define WORKER_SENDER_JOB_BYTES_COMPRESSION_RATIO 20
28 -#define WORKER_SENDER_JOB_REPLAY_REQUEST 21
29 -#define WORKER_SENDER_JOB_FUNCTION_REQUEST 22
30 -#define WORKER_SENDER_JOB_REPLAY_DICT_SIZE 23
31 -
32 -#if WORKER_UTILIZATION_MAX_JOB_TYPES < 21
7 +#define WORKER_SENDER_JOB_CONNECT 0
8 +#define WORKER_SENDER_JOB_PIPE_READ 1
9 +#define WORKER_SENDER_JOB_SOCKET_RECEIVE 2
10 +#define WORKER_SENDER_JOB_EXECUTE 3
11 +#define WORKER_SENDER_JOB_SOCKET_SEND 4
12 +#define WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE 5
13 +#define WORKER_SENDER_JOB_DISCONNECT_OVERFLOW 6
14 +#define WORKER_SENDER_JOB_DISCONNECT_TIMEOUT 7
15 +#define WORKER_SENDER_JOB_DISCONNECT_POLL_ERROR 8
16 +#define WORKER_SENDER_JOB_DISCONNECT_SOCKET_ERROR 9
17 +#define WORKER_SENDER_JOB_DISCONNECT_SSL_ERROR 10
18 +#define WORKER_SENDER_JOB_DISCONNECT_PARENT_CLOSED 11
19 +#define WORKER_SENDER_JOB_DISCONNECT_RECEIVE_ERROR 12
20 +#define WORKER_SENDER_JOB_DISCONNECT_SEND_ERROR 13
21 +#define WORKER_SENDER_JOB_DISCONNECT_NO_COMPRESSION 14
22 +#define WORKER_SENDER_JOB_BUFFER_RATIO 15
23 +#define WORKER_SENDER_JOB_BYTES_RECEIVED 16
24 +#define WORKER_SENDER_JOB_BYTES_SENT 17
25 +#define WORKER_SENDER_JOB_BYTES_COMPRESSED 18
26 +#define WORKER_SENDER_JOB_BYTES_UNCOMPRESSED 19
27 +#define WORKER_SENDER_JOB_BYTES_COMPRESSION_RATIO 20
28 +#define WORKER_SENDER_JOB_REPLAY_REQUEST 21
29 +#define WORKER_SENDER_JOB_FUNCTION_REQUEST 22
30 +#define WORKER_SENDER_JOB_REPLAY_DICT_SIZE 23
31 +#define WORKER_SENDER_JOB_DISCONNECT_CANT_UPGRADE_CONNECTION 24
32 +
33 +#if WORKER_UTILIZATION_MAX_JOB_TYPES < 25
34 #error WORKER_UTILIZATION_MAX_JOB_TYPES has to be at least 21
35 #endif
36
@@ -353,8 +354,7 @@ static inline void rrdpush_sender_thread_close_socket(RRDHOST *host) {
354 rrdpush_sender_charts_and_replication_reset(host);
355 }
356
356 -void rrdpush_encode_variable(stream_encoded_t *se, RRDHOST *host)
357 -{
357 +void rrdpush_encode_variable(stream_encoded_t *se, RRDHOST *host) {
358 se->os_name = (host->system_info->host_os_name)?url_encode(host->system_info->host_os_name):strdupz("");
359 se->os_id = (host->system_info->host_os_id)?url_encode(host->system_info->host_os_id):strdupz("");
360 se->os_version = (host->system_info->host_os_version)?url_encode(host->system_info->host_os_version):strdupz("");
@@ -362,128 +362,155 @@ void rrdpush_encode_variable(stream_encoded_t *se, RRDHOST *host)
362 se->kernel_version = (host->system_info->kernel_version)?url_encode(host->system_info->kernel_version):strdupz("");
363 }
364
365 -void rrdpush_clean_encoded(stream_encoded_t *se)
366 -{
367 - if (se->os_name)
365 +void rrdpush_clean_encoded(stream_encoded_t *se) {
366 + if (se->os_name) {
367 freez(se->os_name);
368 + se->os_name = NULL;
369 + }
370
370 - if (se->os_id)
371 + if (se->os_id) {
372 freez(se->os_id);
373 + se->os_id = NULL;
374 + }
375
373 - if (se->os_version)
376 + if (se->os_version) {
377 freez(se->os_version);
378 + se->os_version = NULL;
379 + }
380
376 - if (se->kernel_name)
381 + if (se->kernel_name) {
382 freez(se->kernel_name);
383 + se->kernel_name = NULL;
384 + }
385
379 - if (se->kernel_version)
386 + if (se->kernel_version) {
387 freez(se->kernel_version);
388 + se->kernel_version = NULL;
389 + }
390 }
391
392 struct {
393 const char *response;
394 + const char *status;
395 size_t length;
396 int32_t version;
397 bool dynamic;
398 const char *error;
399 int worker_job_id;
400 int postpone_reconnect_seconds;
391 - bool prevent_log;
401 + ND_LOG_FIELD_PRIORITY priority;
402 } stream_responses[] = {
403 {
404 .response = START_STREAMING_PROMPT_VN,
405 .length = sizeof(START_STREAMING_PROMPT_VN) - 1,
406 + .status = RRDPUSH_STATUS_CONNECTED,
407 .version = STREAM_HANDSHAKE_OK_V3, // and above
408 .dynamic = true, // dynamic = we will parse the version / capabilities
409 .error = NULL,
410 .worker_job_id = 0,
411 .postpone_reconnect_seconds = 0,
412 + .priority = NDLP_INFO,
413 },
414 {
415 .response = START_STREAMING_PROMPT_V2,
416 .length = sizeof(START_STREAMING_PROMPT_V2) - 1,
417 + .status = RRDPUSH_STATUS_CONNECTED,
418 .version = STREAM_HANDSHAKE_OK_V2,
419 .dynamic = false,
420 .error = NULL,
421 .worker_job_id = 0,
422 .postpone_reconnect_seconds = 0,
423 + .priority = NDLP_INFO,
424 },
425 {
426 .response = START_STREAMING_PROMPT_V1,
427 .length = sizeof(START_STREAMING_PROMPT_V1) - 1,
428 + .status = RRDPUSH_STATUS_CONNECTED,
429 .version = STREAM_HANDSHAKE_OK_V1,
430 .dynamic = false,
431 .error = NULL,
432 .worker_job_id = 0,
433 .postpone_reconnect_seconds = 0,
434 + .priority = NDLP_INFO,
435 },
436 {
437 .response = START_STREAMING_ERROR_SAME_LOCALHOST,
438 .length = sizeof(START_STREAMING_ERROR_SAME_LOCALHOST) - 1,
439 + .status = RRDPUSH_STATUS_LOCALHOST,
440 .version = STREAM_HANDSHAKE_ERROR_LOCALHOST,
441 .dynamic = false,
442 .error = "remote server rejected this stream, the host we are trying to stream is its localhost",
443 .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
444 .postpone_reconnect_seconds = 60 * 60, // the IP may change, try it every hour
428 - .prevent_log = true,
445 + .priority = NDLP_DEBUG,
446 },
447 {
448 .response = START_STREAMING_ERROR_ALREADY_STREAMING,
449 .length = sizeof(START_STREAMING_ERROR_ALREADY_STREAMING) - 1,
450 + .status = RRDPUSH_STATUS_ALREADY_CONNECTED,
451 .version = STREAM_HANDSHAKE_ERROR_ALREADY_CONNECTED,
452 .dynamic = false,
453 .error = "remote server rejected this stream, the host we are trying to stream is already streamed to it",
454 .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
455 .postpone_reconnect_seconds = 2 * 60, // 2 minutes
438 - .prevent_log = true,
456 + .priority = NDLP_DEBUG,
457 },
458 {
459 .response = START_STREAMING_ERROR_NOT_PERMITTED,
460 .length = sizeof(START_STREAMING_ERROR_NOT_PERMITTED) - 1,
461 + .status = RRDPUSH_STATUS_PERMISSION_DENIED,
462 .version = STREAM_HANDSHAKE_ERROR_DENIED,
463 .dynamic = false,
464 .error = "remote server denied access, probably we don't have the right API key?",
465 .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
466 .postpone_reconnect_seconds = 1 * 60, // 1 minute
467 + .priority = NDLP_ERR,
468 },
469 {
470 .response = START_STREAMING_ERROR_BUSY_TRY_LATER,
471 .length = sizeof(START_STREAMING_ERROR_BUSY_TRY_LATER) - 1,
472 + .status = RRDPUSH_STATUS_RATE_LIMIT,
473 .version = STREAM_HANDSHAKE_BUSY_TRY_LATER,
474 .dynamic = false,
475 .error = "remote server is currently busy, we should try later",
476 .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
477 .postpone_reconnect_seconds = 2 * 60, // 2 minutes
478 + .priority = NDLP_NOTICE,
479 },
480 {
481 .response = START_STREAMING_ERROR_INTERNAL_ERROR,
482 .length = sizeof(START_STREAMING_ERROR_INTERNAL_ERROR) - 1,
483 + .status = RRDPUSH_STATUS_INTERNAL_SERVER_ERROR,
484 .version = STREAM_HANDSHAKE_INTERNAL_ERROR,
485 .dynamic = false,
486 .error = "remote server is encountered an internal error, we should try later",
487 .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
488 .postpone_reconnect_seconds = 5 * 60, // 5 minutes
489 + .priority = NDLP_CRIT,
490 },
491 {
492 .response = START_STREAMING_ERROR_INITIALIZATION,
493 .length = sizeof(START_STREAMING_ERROR_INITIALIZATION) - 1,
494 + .status = RRDPUSH_STATUS_INITIALIZATION_IN_PROGRESS,
495 .version = STREAM_HANDSHAKE_INITIALIZATION,
496 .dynamic = false,
497 .error = "remote server is initializing, we should try later",
498 .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
499 .postpone_reconnect_seconds = 2 * 60, // 2 minute
500 + .priority = NDLP_NOTICE,
501 },
502
503 // terminator
504 {
505 .response = NULL,
506 .length = 0,
507 + .status = RRDPUSH_STATUS_BAD_HANDSHAKE,
508 .version = STREAM_HANDSHAKE_ERROR_BAD_HANDSHAKE,
509 .dynamic = false,
510 .error = "remote node response is not understood, is it Netdata?",
511 .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
512 .postpone_reconnect_seconds = 1 * 60, // 1 minute
486 - .prevent_log = false,
513 + .priority = NDLP_ERR,
514 }
515 };
516
@@ -513,8 +540,9 @@ static inline bool rrdpush_sender_validate_response(RRDHOST *host, struct sender
540 return true;
541 }
542
516 - bool prevent_log = stream_responses[i].prevent_log;
543 + ND_LOG_FIELD_PRIORITY priority = stream_responses[i].priority;
544 const char *error = stream_responses[i].error;
545 + const char *status = stream_responses[i].status;
546 int worker_job_id = stream_responses[i].worker_job_id;
547 int delay = stream_responses[i].postpone_reconnect_seconds;
548
@@ -523,15 +551,18 @@ static inline bool rrdpush_sender_validate_response(RRDHOST *host, struct sender
551 host->destination->reason = version;
552 host->destination->postpone_reconnection_until = now_realtime_sec() + delay;
553
526 - char buf[LOG_DATE_LENGTH];
527 - log_date(buf, LOG_DATE_LENGTH, host->destination->postpone_reconnection_until);
554 + ND_LOG_STACK lgs[] = {
555 + ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, status),
556 + ND_LOG_FIELD_END(),
557 + };
558 + ND_LOG_STACK_PUSH(lgs);
559 +
560 + char buf[ISO8601_MAX_LENGTH];
561 + iso8601_datetime_ut(buf, sizeof(buf), host->destination->postpone_reconnection_until * USEC_PER_SEC, 0);
562
529 - if(prevent_log)
530 - internal_error(true, "STREAM %s [send to %s]: %s - will retry in %ld secs, at %s",
531 - rrdhost_hostname(host), s->connected_to, error, delay, buf);
532 - else
533 - netdata_log_error("STREAM %s [send to %s]: %s - will retry in %d secs, at %s",
534 - rrdhost_hostname(host), s->connected_to, error, delay, buf);
563 + nd_log(NDLS_DAEMON, priority,
564 + "STREAM %s [send to %s]: %s - will retry in %d secs, at %s",
565 + rrdhost_hostname(host), s->connected_to, error, delay, buf);
566
567 return false;
568 }
@@ -557,6 +588,12 @@ static bool rrdpush_sender_connect_ssl(struct sender_state *s __maybe_unused) {
588 if(!netdata_ssl_connect(&host->sender->ssl)) {
589 // couldn't connect
590
591 + ND_LOG_STACK lgs[] = {
592 + ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_SSL_ERROR),
593 + ND_LOG_FIELD_END(),
594 + };
595 + ND_LOG_STACK_PUSH(lgs);
596 +
597 worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_SSL_ERROR);
598 rrdpush_sender_thread_close_socket(host);
599 host->destination->reason = STREAM_HANDSHAKE_ERROR_SSL_ERROR;
@@ -568,6 +605,12 @@ static bool rrdpush_sender_connect_ssl(struct sender_state *s __maybe_unused) {
605 security_test_certificate(host->sender->ssl.conn)) {
606 // certificate is not valid
607
608 + ND_LOG_STACK lgs[] = {
609 + ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_INVALID_SSL_CERTIFICATE),
610 + ND_LOG_FIELD_END(),
611 + };
612 + ND_LOG_STACK_PUSH(lgs);
613 +
614 worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_SSL_ERROR);
615 netdata_log_error("SSL: closing the stream connection, because the server SSL certificate is not valid.");
616 rrdpush_sender_thread_close_socket(host);
@@ -579,6 +622,12 @@ static bool rrdpush_sender_connect_ssl(struct sender_state *s __maybe_unused) {
622 return true;
623 }
624
625 + ND_LOG_STACK lgs[] = {
626 + ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_CANT_ESTABLISH_SSL_CONNECTION),
627 + ND_LOG_FIELD_END(),
628 + };
629 + ND_LOG_STACK_PUSH(lgs);
630 +
631 netdata_log_error("SSL: failed to establish connection.");
632 return false;
633
@@ -826,6 +875,13 @@ static bool rrdpush_sender_thread_connect_to_parent(RRDHOST *host, int default_p
875 return false;
876
877 if (s->parent_using_h2o && rrdpush_http_upgrade_prelude(host, s)) {
878 + ND_LOG_STACK lgs[] = {
879 + ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_CANT_UPGRADE_CONNECTION),
880 + ND_LOG_FIELD_END(),
881 + };
882 + ND_LOG_STACK_PUSH(lgs);
883 +
884 + worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_CANT_UPGRADE_CONNECTION);
885 rrdpush_sender_thread_close_socket(host);
886 host->destination->reason = STREAM_HANDSHAKE_ERROR_HTTP_UPGRADE;
887 host->destination->postpone_reconnection_until = now_realtime_sec() + 1 * 60;
@@ -845,9 +901,19 @@ static bool rrdpush_sender_thread_connect_to_parent(RRDHOST *host, int default_p
901 timeout);
902
903 if(bytes <= 0) { // timeout is 0
904 + ND_LOG_STACK lgs[] = {
905 + ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_TIMEOUT),
906 + ND_LOG_FIELD_END(),
907 + };
908 + ND_LOG_STACK_PUSH(lgs);
909 +
910 worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_TIMEOUT);
911 rrdpush_sender_thread_close_socket(host);
850 - netdata_log_error("STREAM %s [send to %s]: failed to send HTTP header to remote netdata.", rrdhost_hostname(host), s->connected_to);
912 +
913 + nd_log(NDLS_DAEMON, NDLP_ERR,
914 + "STREAM %s [send to %s]: failed to send HTTP header to remote netdata.",
915 + rrdhost_hostname(host), s->connected_to);
916 +
917 host->destination->reason = STREAM_HANDSHAKE_ERROR_SEND_TIMEOUT;
918 host->destination->postpone_reconnection_until = now_realtime_sec() + 1 * 60;
919 return false;
@@ -864,22 +930,35 @@ static bool rrdpush_sender_thread_connect_to_parent(RRDHOST *host, int default_p
930 timeout);
931
932 if(bytes <= 0) { // timeout is 0
933 + ND_LOG_STACK lgs[] = {
934 + ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_TIMEOUT),
935 + ND_LOG_FIELD_END(),
936 + };
937 + ND_LOG_STACK_PUSH(lgs);
938 +
939 worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_TIMEOUT);
940 rrdpush_sender_thread_close_socket(host);
869 - netdata_log_error("STREAM %s [send to %s]: remote netdata does not respond.", rrdhost_hostname(host), s->connected_to);
941 +
942 + nd_log(NDLS_DAEMON, NDLP_ERR,
943 + "STREAM %s [send to %s]: remote netdata does not respond.",
944 + rrdhost_hostname(host), s->connected_to);
945 +
946 host->destination->reason = STREAM_HANDSHAKE_ERROR_RECEIVE_TIMEOUT;
947 host->destination->postpone_reconnection_until = now_realtime_sec() + 30;
948 return false;
949 }
950
951 if(sock_setnonblock(s->rrdpush_sender_socket) < 0)
876 - netdata_log_error("STREAM %s [send to %s]: cannot set non-blocking mode for socket.", rrdhost_hostname(host), s->connected_to);
952 + nd_log(NDLS_DAEMON, NDLP_WARNING,
953 + "STREAM %s [send to %s]: cannot set non-blocking mode for socket.",
954 + rrdhost_hostname(host), s->connected_to);
955
956 if(sock_enlarge_out(s->rrdpush_sender_socket) < 0)
879 - netdata_log_error("STREAM %s [send to %s]: cannot enlarge the socket buffer.", rrdhost_hostname(host), s->connected_to);
957 + nd_log(NDLS_DAEMON, NDLP_WARNING,
958 + "STREAM %s [send to %s]: cannot enlarge the socket buffer.",
959 + rrdhost_hostname(host), s->connected_to);
960
961 http[bytes] = '\0';
882 - netdata_log_debug(D_STREAM, "Response to sender from far end: %s", http);
962 if(!rrdpush_sender_validate_response(host, s, http, bytes))
963 return false;
964
@@ -887,13 +966,26 @@ static bool rrdpush_sender_thread_connect_to_parent(RRDHOST *host, int default_p
966
967 log_sender_capabilities(s);
968
890 - netdata_log_debug(D_STREAM, "STREAM: Connected on fd %d...", s->rrdpush_sender_socket);
969 + ND_LOG_STACK lgs[] = {
970 + ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_CONNECTED),
971 + ND_LOG_FIELD_END(),
972 + };
973 + ND_LOG_STACK_PUSH(lgs);
974 +
975 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
976 + "STREAM %s: connected to %s...",
977 + rrdhost_hostname(host), s->connected_to);
978
979 return true;
980 }
981
895 -static bool attempt_to_connect(struct sender_state *state)
896 -{
982 +static bool attempt_to_connect(struct sender_state *state) {
983 + ND_LOG_STACK lgs[] = {
984 + ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_to_parent_msgid),
985 + ND_LOG_FIELD_END(),
986 + };
987 + ND_LOG_STACK_PUSH(lgs);
988 +
989 state->send_attempts = 0;
990
991 // reset the bytes we have sent for this session
@@ -1062,6 +1154,12 @@ void stream_execute_function_callback(BUFFER *func_wb, int code, void *data) {
1154 void execute_commands(struct sender_state *s) {
1155 worker_is_busy(WORKER_SENDER_JOB_EXECUTE);
1156
1157 + ND_LOG_STACK lgs[] = {
1158 + ND_LOG_FIELD_CB(NDF_REQUEST, line_splitter_reconstruct_line, &s->line),
1159 + ND_LOG_FIELD_END(),
1160 + };
1161 + ND_LOG_STACK_PUSH(lgs);
1162 +
1163 char *start = s->read_buffer, *end = &s->read_buffer[s->read_len], *newline;
1164 *end = 0;
1165 while( start < end && (newline = strchr(start, '\n')) ) {
@@ -1075,27 +1173,22 @@ void execute_commands(struct sender_state *s) {
1173 continue;
1174 }
1175
1078 - netdata_log_access("STREAM: %d from '%s' for host '%s': %s",
1079 - gettid(), s->connected_to, rrdhost_hostname(s->host), start);
1176 + s->line.count++;
1177 + s->line.num_words = quoted_strings_splitter_pluginsd(start, s->line.words, PLUGINSD_MAX_WORDS);
1178 + const char *command = get_word(s->line.words, s->line.num_words, 0);
1179
1081 - // internal_error(true, "STREAM %s [send to %s] received command over connection: %s", rrdhost_hostname(s->host), s->connected_to, start);
1082 -
1083 - char *words[PLUGINSD_MAX_WORDS] = { NULL };
1084 - size_t num_words = quoted_strings_splitter_pluginsd(start, words, PLUGINSD_MAX_WORDS);
1085 -
1086 - const char *keyword = get_word(words, num_words, 0);
1087 -
1088 - if(keyword && (strcmp(keyword, PLUGINSD_KEYWORD_FUNCTION) == 0 || strcmp(keyword, PLUGINSD_KEYWORD_FUNCTION_PAYLOAD_END) == 0)) {
1180 + if(command && (strcmp(command, PLUGINSD_KEYWORD_FUNCTION) == 0 || strcmp(command, PLUGINSD_KEYWORD_FUNCTION_PAYLOAD_END) == 0)) {
1181 worker_is_busy(WORKER_SENDER_JOB_FUNCTION_REQUEST);
1182 + nd_log(NDLS_ACCESS, NDLP_INFO, NULL);
1183
1091 - char *transaction = s->receiving_function_payload ? s->function_payload.txid : get_word(words, num_words, 1);
1092 - char *timeout_s = s->receiving_function_payload ? s->function_payload.timeout : get_word(words, num_words, 2);
1093 - char *function = s->receiving_function_payload ? s->function_payload.fn_name : get_word(words, num_words, 3);
1184 + char *transaction = s->receiving_function_payload ? s->function_payload.txid : get_word(s->line.words, s->line.num_words, 1);
1185 + char *timeout_s = s->receiving_function_payload ? s->function_payload.timeout : get_word(s->line.words, s->line.num_words, 2);
1186 + char *function = s->receiving_function_payload ? s->function_payload.fn_name : get_word(s->line.words, s->line.num_words, 3);
1187
1188 if(!transaction || !*transaction || !timeout_s || !*timeout_s || !function || !*function) {
1189 netdata_log_error("STREAM %s [send to %s] %s execution command is incomplete (transaction = '%s', timeout = '%s', function = '%s'). Ignoring it.",
1190 rrdhost_hostname(s->host), s->connected_to,
1098 - keyword,
1191 + command,
1192 transaction?transaction:"(unset)",
1193 timeout_s?timeout_s:"(unset)",
1194 function?function:"(unset)");
@@ -1133,9 +1226,12 @@ void execute_commands(struct sender_state *s) {
1226 memset(&s->function_payload, 0, sizeof(struct function_payload_state));
1227 }
1228 }
1136 - else if (keyword && strcmp(keyword, PLUGINSD_KEYWORD_FUNCTION_PAYLOAD) == 0) {
1229 + else if (command && strcmp(command, PLUGINSD_KEYWORD_FUNCTION_PAYLOAD) == 0) {
1230 + nd_log(NDLS_ACCESS, NDLP_INFO, NULL);
1231 +
1232 if (s->receiving_function_payload) {
1138 - netdata_log_error("STREAM %s [send to %s] received %s command while already receiving function payload", rrdhost_hostname(s->host), s->connected_to, keyword);
1233 + netdata_log_error("STREAM %s [send to %s] received %s command while already receiving function payload",
1234 + rrdhost_hostname(s->host), s->connected_to, command);
1235 s->receiving_function_payload = false;
1236 buffer_free(s->function_payload.payload);
1237 s->function_payload.payload = NULL;
@@ -1143,14 +1239,14 @@ void execute_commands(struct sender_state *s) {
1239 // TODO send error response
1240 }
1241
1146 - char *transaction = get_word(words, num_words, 1);
1147 - char *timeout_s = get_word(words, num_words, 2);
1148 - char *function = get_word(words, num_words, 3);
1242 + char *transaction = get_word(s->line.words, s->line.num_words, 1);
1243 + char *timeout_s = get_word(s->line.words, s->line.num_words, 2);
1244 + char *function = get_word(s->line.words, s->line.num_words, 3);
1245
1246 if(!transaction || !*transaction || !timeout_s || !*timeout_s || !function || !*function) {
1247 netdata_log_error("STREAM %s [send to %s] %s execution command is incomplete (transaction = '%s', timeout = '%s', function = '%s'). Ignoring it.",
1248 rrdhost_hostname(s->host), s->connected_to,
1153 - keyword,
1249 + command,
1250 transaction?transaction:"(unset)",
1251 timeout_s?timeout_s:"(unset)",
1252 function?function:"(unset)");
@@ -1159,30 +1255,32 @@ void execute_commands(struct sender_state *s) {
1255 s->receiving_function_payload = true;
1256 s->function_payload.payload = buffer_create(4096, &netdata_buffers_statistics.buffers_functions);
1257
1162 - s->function_payload.txid = strdupz(get_word(words, num_words, 1));
1163 - s->function_payload.timeout = strdupz(get_word(words, num_words, 2));
1164 - s->function_payload.fn_name = strdupz(get_word(words, num_words, 3));
1258 + s->function_payload.txid = strdupz(get_word(s->line.words, s->line.num_words, 1));
1259 + s->function_payload.timeout = strdupz(get_word(s->line.words, s->line.num_words, 2));
1260 + s->function_payload.fn_name = strdupz(get_word(s->line.words, s->line.num_words, 3));
1261 }
1166 - else if(keyword && strcmp(keyword, PLUGINSD_KEYWORD_FUNCTION_CANCEL) == 0) {
1262 + else if(command && strcmp(command, PLUGINSD_KEYWORD_FUNCTION_CANCEL) == 0) {
1263 worker_is_busy(WORKER_SENDER_JOB_FUNCTION_REQUEST);
1264 + nd_log(NDLS_ACCESS, NDLP_DEBUG, NULL);
1265
1169 - char *transaction = get_word(words, num_words, 1);
1266 + char *transaction = get_word(s->line.words, s->line.num_words, 1);
1267 if(transaction && *transaction)
1268 rrd_function_cancel(transaction);
1269 }
1173 - else if (keyword && strcmp(keyword, PLUGINSD_KEYWORD_REPLAY_CHART) == 0) {
1270 + else if (command && strcmp(command, PLUGINSD_KEYWORD_REPLAY_CHART) == 0) {
1271 worker_is_busy(WORKER_SENDER_JOB_REPLAY_REQUEST);
1272 + nd_log(NDLS_ACCESS, NDLP_DEBUG, NULL);
1273
1176 - const char *chart_id = get_word(words, num_words, 1);
1177 - const char *start_streaming = get_word(words, num_words, 2);
1178 - const char *after = get_word(words, num_words, 3);
1179 - const char *before = get_word(words, num_words, 4);
1274 + const char *chart_id = get_word(s->line.words, s->line.num_words, 1);
1275 + const char *start_streaming = get_word(s->line.words, s->line.num_words, 2);
1276 + const char *after = get_word(s->line.words, s->line.num_words, 3);
1277 + const char *before = get_word(s->line.words, s->line.num_words, 4);
1278
1279 if (!chart_id || !start_streaming || !after || !before) {
1280 netdata_log_error("STREAM %s [send to %s] %s command is incomplete"
1281 " (chart=%s, start_streaming=%s, after=%s, before=%s)",
1282 rrdhost_hostname(s->host), s->connected_to,
1185 - keyword,
1283 + command,
1284 chart_id ? chart_id : "(unset)",
1285 start_streaming ? start_streaming : "(unset)",
1286 after ? after : "(unset)",
@@ -1197,12 +1295,14 @@ void execute_commands(struct sender_state *s) {
1295 }
1296 }
1297 else {
1200 - netdata_log_error("STREAM %s [send to %s] received unknown command over connection: %s", rrdhost_hostname(s->host), s->connected_to, words[0]?words[0]:"(unset)");
1298 + netdata_log_error("STREAM %s [send to %s] received unknown command over connection: %s", rrdhost_hostname(s->host), s->connected_to, s->line.words[0]?s->line.words[0]:"(unset)");
1299 }
1300
1301 + line_splitter_reset(&s->line);
1302 worker_is_busy(WORKER_SENDER_JOB_EXECUTE);
1303 start = newline + 1;
1304 }
1305 +
1306 if (start < end) {
1307 memmove(s->read_buffer, start, end-start);
1308 s->read_len = end - start;
@@ -1397,7 +1497,61 @@ void rrdpush_initialize_ssl_ctx(RRDHOST *host __maybe_unused) {
1497 #endif
1498 }
1499
1500 +static bool stream_sender_log_capabilities(BUFFER *wb, void *ptr) {
1501 + struct sender_state *state = ptr;
1502 + if(!state)
1503 + return false;
1504 +
1505 + stream_capabilities_to_string(wb, state->capabilities);
1506 + return true;
1507 +}
1508 +
1509 +static bool stream_sender_log_transport(BUFFER *wb, void *ptr) {
1510 + struct sender_state *state = ptr;
1511 + if(!state)
1512 + return false;
1513 +
1514 +#ifdef ENABLE_HTTPS
1515 + buffer_strcat(wb, SSL_connection(&state->ssl) ? "https" : "http");
1516 +#else
1517 + buffer_strcat(wb, "http");
1518 +#endif
1519 + return true;
1520 +}
1521 +
1522 +static bool stream_sender_log_dst_ip(BUFFER *wb, void *ptr) {
1523 + struct sender_state *state = ptr;
1524 + if(!state || state->rrdpush_sender_socket == -1)
1525 + return false;
1526 +
1527 + SOCKET_PEERS peers = socket_peers(state->rrdpush_sender_socket);
1528 + buffer_strcat(wb, peers.peer.ip);
1529 + return true;
1530 +}
1531 +
1532 +static bool stream_sender_log_dst_port(BUFFER *wb, void *ptr) {
1533 + struct sender_state *state = ptr;
1534 + if(!state || state->rrdpush_sender_socket == -1)
1535 + return false;
1536 +
1537 + SOCKET_PEERS peers = socket_peers(state->rrdpush_sender_socket);
1538 + buffer_print_uint64(wb, peers.peer.port);
1539 + return true;
1540 +}
1541 +
1542 void *rrdpush_sender_thread(void *ptr) {
1543 + struct sender_state *s = ptr;
1544 +
1545 + ND_LOG_STACK lgs[] = {
1546 + ND_LOG_FIELD_STR(NDF_NIDL_NODE, s->host->hostname),
1547 + ND_LOG_FIELD_CB(NDF_DST_IP, stream_sender_log_dst_ip, s),
1548 + ND_LOG_FIELD_CB(NDF_DST_PORT, stream_sender_log_dst_port, s),
1549 + ND_LOG_FIELD_CB(NDF_DST_TRANSPORT, stream_sender_log_transport, s),
1550 + ND_LOG_FIELD_CB(NDF_SRC_CAPABILITIES, stream_sender_log_capabilities, s),
1551 + ND_LOG_FIELD_END(),
1552 + };
1553 + ND_LOG_STACK_PUSH(lgs);
1554 +
1555 worker_register("STREAMSND");
1556 worker_register_job_name(WORKER_SENDER_JOB_CONNECT, "connect");
1557 worker_register_job_name(WORKER_SENDER_JOB_PIPE_READ, "pipe read");
@@ -1416,6 +1570,7 @@ void *rrdpush_sender_thread(void *ptr) {
1570 worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_SEND_ERROR, "disconnect send error");
1571 worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_NO_COMPRESSION, "disconnect no compression");
1572 worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE, "disconnect bad handshake");
1573 + worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_CANT_UPGRADE_CONNECTION, "disconnect cant upgrade");
1574
1575 worker_register_job_name(WORKER_SENDER_JOB_REPLAY_REQUEST, "replay request");
1576 worker_register_job_name(WORKER_SENDER_JOB_FUNCTION_REQUEST, "function");
@@ -1428,8 +1583,6 @@ void *rrdpush_sender_thread(void *ptr) {
1583 worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_COMPRESSION_RATIO, "cumulative compression savings ratio", "%", WORKER_METRIC_ABSOLUTE);
1584 worker_register_job_custom_metric(WORKER_SENDER_JOB_REPLAY_DICT_SIZE, "replication dict entries", "entries", WORKER_METRIC_ABSOLUTE);
1585
1431 - struct sender_state *s = ptr;
1432 -
1586 if(!rrdhost_has_rrdpush_sender_enabled(s->host) || !s->host->rrdpush_send_destination ||
1587 !*s->host->rrdpush_send_destination || !s->host->rrdpush_send_api_key ||
1588 !*s->host->rrdpush_send_api_key) {
@@ -1523,7 +1676,10 @@ void *rrdpush_sender_thread(void *ptr) {
1676 s->replication.oldest_request_after_t = 0;
1677
1678 rrdhost_flag_set(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS);
1526 - netdata_log_info("STREAM %s [send to %s]: enabling metrics streaming...", rrdhost_hostname(s->host), s->connected_to);
1679 +
1680 + nd_log(NDLS_DAEMON, NDLP_DEBUG,
1681 + "STREAM %s [send to %s]: enabling metrics streaming...",
1682 + rrdhost_hostname(s->host), s->connected_to);
1683
1684 continue;
1685 }
web/api/queries/query.c
+2 -2
@@ -3617,12 +3617,12 @@ RRDR *rrd2rrdr(ONEWAYALLOC *owa, QUERY_TARGET *qt) {
3617 bool cancel = false;
3618 if (qt->request.interrupt_callback && qt->request.interrupt_callback(qt->request.interrupt_callback_data)) {
3619 cancel = true;
3620 - netdata_log_access("QUERY INTERRUPTED");
3620 + nd_log(NDLS_ACCESS, NDLP_NOTICE, "QUERY INTERRUPTED");
3621 }
3622
3623 if (qt->request.timeout_ms && ((NETDATA_DOUBLE)(now_ut - qt->timings.received_ut) / 1000.0) > (NETDATA_DOUBLE)qt->request.timeout_ms) {
3624 cancel = true;
3625 - netdata_log_access("QUERY CANCELED RUNTIME EXCEEDED %0.2f ms (LIMIT %lld ms)",
3625 + nd_log(NDLS_ACCESS, NDLP_WARNING, "QUERY CANCELED RUNTIME EXCEEDED %0.2f ms (LIMIT %lld ms)",
3626 (NETDATA_DOUBLE)(now_ut - qt->timings.received_ut) / 1000.0, (long long)qt->request.timeout_ms);
3627 }
3628
web/api/web_api_v1.c
+4 -1
@@ -1410,7 +1410,10 @@ int web_client_api_request_v1_function(RRDHOST *host, struct web_client *w, char
1410 wb->content_type = CT_APPLICATION_JSON;
1411 buffer_no_cacheable(wb);
1412
1413 - return rrd_function_run(host, wb, timeout, function, true, NULL,
1413 + char transaction[UUID_COMPACT_STR_LEN];
1414 + uuid_unparse_lower_compact(w->transaction, transaction);
1415 +
1416 + return rrd_function_run(host, wb, timeout, function, true, transaction,
1417 NULL, NULL,
1418 web_client_interrupt_callback, w, NULL);
1419 }
web/rtc/webrtc.c
+14 -23
@@ -279,7 +279,11 @@ static size_t webrtc_send_in_chunks(WEBRTC_DC *chan, const char *data, size_t si
279 }
280
281 static void webrtc_execute_api_request(WEBRTC_DC *chan, const char *request, size_t size __maybe_unused, bool binary __maybe_unused) {
282 - struct timeval tv;
282 + ND_LOG_STACK lgs[] = {
283 + ND_LOG_FIELD_TXT(NDF_SRC_TRANSPORT, "webrtc"),
284 + ND_LOG_FIELD_END(),
285 + };
286 + ND_LOG_STACK_PUSH(lgs);
287
288 internal_error(true, "WEBRTC[%d],DC[%d]: got request '%s' of size %zu and type %s.",
289 chan->conn->pc, chan->dc, request, size, binary?"binary":"text");
@@ -304,6 +308,7 @@ static void webrtc_execute_api_request(WEBRTC_DC *chan, const char *request, siz
308 web_client_timeout_checkpoint_set(w, 0);
309 web_client_decode_path_and_query_string(w, path);
310 path = (char *)buffer_tostring(w->url_path_decoded);
311 +
312 w->response.code = (short)web_client_api_request_with_node_selection(localhost, w, path);
313 web_client_timeout_checkpoint_response_ready(w, NULL);
314
@@ -346,21 +351,7 @@ static void webrtc_execute_api_request(WEBRTC_DC *chan, const char *request, siz
351 w->statistics.sent_bytes = sent_bytes;
352
353 cleanup:
349 - now_monotonic_high_precision_timeval(&tv);
350 - netdata_log_access("%llu: %d '[RTC]:%d:%d' '%s' (sent/all = %zu/%zu bytes %0.0f%%, prep/sent/total = %0.2f/%0.2f/%0.2f ms) %d '%s'",
351 - w->id
352 - , gettid()
353 - , chan->conn->pc, chan->dc
354 - , "DATA"
355 - , sent_bytes
356 - , response_size
357 - , response_size > sent_bytes ? -(((double)(response_size - sent_bytes) / (double)response_size) * 100.0) : ((response_size > 0) ? (((sent_bytes - response_size) / (double)response_size) * 100.0) : 0.0)
358 - , dt_usec(&w->timings.tv_ready, &w->timings.tv_in) / 1000.0
359 - , dt_usec(&tv, &w->timings.tv_ready) / 1000.0
360 - , dt_usec(&tv, &w->timings.tv_in) / 1000.0
361 - , w->response.code
362 - , strip_control_characters((char *)buffer_tostring(w->url_as_received))
363 - );
354 + web_client_log_completed_request(w, false);
355 web_client_release_to_cache(w);
356 }
357
@@ -373,7 +364,7 @@ static void myOpenCallback(int id __maybe_unused, void *user_ptr) {
364 WEBRTC_DC *chan = user_ptr;
365 internal_fatal(chan->dc != id, "WEBRTC[%d],DC[%d]: dc mismatch, expected %d, got %d", chan->conn->pc, chan->dc, chan->dc, id);
366
376 - netdata_log_access("WEBRTC[%d],DC[%d]: %d DATA CHANNEL '%s' OPEN", chan->conn->pc, chan->dc, gettid(), chan->label);
367 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "WEBRTC[%d],DC[%d]: %d DATA CHANNEL '%s' OPEN", chan->conn->pc, chan->dc, gettid(), chan->label);
368 internal_error(true, "WEBRTC[%d],DC[%d]: data channel opened.", chan->conn->pc, chan->dc);
369 chan->open = true;
370 }
@@ -391,7 +382,7 @@ static void myClosedCallback(int id __maybe_unused, void *user_ptr) {
382 DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(chan->conn->channels.head, chan, link.prev, link.next);
383 spinlock_unlock(&chan->conn->channels.spinlock);
384
394 - netdata_log_access("WEBRTC[%d],DC[%d]: %d DATA CHANNEL '%s' CLOSED", chan->conn->pc, chan->dc, gettid(), chan->label);
385 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "WEBRTC[%d],DC[%d]: %d DATA CHANNEL '%s' CLOSED", chan->conn->pc, chan->dc, gettid(), chan->label);
386
387 freez(chan->label);
388 freez(chan);
@@ -573,27 +564,27 @@ static void myStateChangeCallback(int pc __maybe_unused, rtcState state, void *u
564 break;
565
566 case RTC_CONNECTING:
576 - netdata_log_access("WEBRTC[%d]: %d CONNECTING", conn->pc, gettid());
567 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "WEBRTC[%d]: %d CONNECTING", conn->pc, gettid());
568 internal_error(true, "WEBRTC[%d]: connecting...", conn->pc);
569 break;
570
571 case RTC_CONNECTED:
581 - netdata_log_access("WEBRTC[%d]: %d CONNECTED", conn->pc, gettid());
572 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "WEBRTC[%d]: %d CONNECTED", conn->pc, gettid());
573 internal_error(true, "WEBRTC[%d]: connected!", conn->pc);
574 break;
575
576 case RTC_DISCONNECTED:
586 - netdata_log_access("WEBRTC[%d]: %d DISCONNECTED", conn->pc, gettid());
577 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "WEBRTC[%d]: %d DISCONNECTED", conn->pc, gettid());
578 internal_error(true, "WEBRTC[%d]: disconnected.", conn->pc);
579 break;
580
581 case RTC_FAILED:
591 - netdata_log_access("WEBRTC[%d]: %d CONNECTION FAILED", conn->pc, gettid());
582 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "WEBRTC[%d]: %d CONNECTION FAILED", conn->pc, gettid());
583 internal_error(true, "WEBRTC[%d]: failed.", conn->pc);
584 break;
585
586 case RTC_CLOSED:
596 - netdata_log_access("WEBRTC[%d]: %d CONNECTION CLOSED", conn->pc, gettid());
587 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "WEBRTC[%d]: %d CONNECTION CLOSED", conn->pc, gettid());
588 internal_error(true, "WEBRTC[%d]: closed.", conn->pc);
589 spinlock_lock(&webrtc_base.unsafe.spinlock);
590 webrtc_destroy_connection_unsafe(conn);
web/server/h2o/http_server.c
+2 -2
@@ -272,7 +272,7 @@ static int netdata_uberhandler(h2o_handler_t *self, h2o_req_t *req)
272 if (host != NULL)
273 uuid_unparse_lower(host->host_uuid, host_uuid_str);
274
275 - netdata_log_access("HTTPD OK method: " PRINTF_H2O_IOVEC_FMT
275 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "HTTPD OK method: " PRINTF_H2O_IOVEC_FMT
276 ", path: " PRINTF_H2O_IOVEC_FMT
277 ", as host: %s"
278 ", response: %d",
@@ -281,7 +281,7 @@ static int netdata_uberhandler(h2o_handler_t *self, h2o_req_t *req)
281 host == NULL ? "unknown" : (localhost ? "localhost" : host_uuid_str),
282 req->res.status);
283 } else {
284 - netdata_log_access("HTTPD %d"
284 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "HTTPD %d"
285 " method: " PRINTF_H2O_IOVEC_FMT
286 ", path: " PRINTF_H2O_IOVEC_FMT
287 ", forwarding to file handler as path: " PRINTF_H2O_IOVEC_FMT,
web/server/h2o/streaming.c
+3 -2
@@ -294,8 +294,9 @@ void stream_process(h2o_stream_conn_t *conn, int initial)
294 socklen_t len = h2o_socket_getpeername(conn->sock, &client);
295 char peername[NI_MAXHOST];
296 size_t peername_len = h2o_socket_getnumerichost(&client, len, peername);
297 - memcpy(w.client_ip, peername, peername_len);
298 - w.client_ip[peername_len] = 0;
297 + size_t cpy_len = sizeof(w.client_ip) < peername_len ? sizeof(w.client_ip) : peername_len;
298 + memcpy(w.client_ip, peername, cpy_len);
299 + w.client_ip[cpy_len - 1] = 0;
300 w.user_agent = conn->user_agent;
301
302 rc = rrdpush_receiver_thread_spawn(&w, conn->url, conn);
web/server/static/static-threaded.c
+1 -3
@@ -294,7 +294,7 @@ static int web_server_rcv_callback(POLLINFO *pi, short int *events) {
294 netdata_log_debug(D_WEB_CLIENT, "%llu: processing received data on fd %d.", w->id, fd);
295 worker_is_idle();
296 worker_is_busy(WORKER_JOB_PROCESS);
297 - web_client_process_request(w);
297 + web_client_process_request_from_web_server(w);
298
299 if (unlikely(w->mode == WEB_CLIENT_MODE_STREAM)) {
300 web_client_send(w);
@@ -538,8 +538,6 @@ void *socket_listen_main_static_threaded(void *ptr) {
538 static_workers_private_data = callocz((size_t)static_threaded_workers_count,
539 sizeof(struct web_server_static_threaded_worker));
540
541 - web_server_is_multithreaded = (static_threaded_workers_count > 1);
542 -
541 int i;
542 for (i = 1; i < static_threaded_workers_count; i++) {
543 static_workers_private_data[i].id = i;
web/server/web_client.c
+153 -73
@@ -163,77 +163,96 @@ static void web_client_reset_allocations(struct web_client *w, bool free_all) {
163 web_client_reset_path_flags(w);
164 }
165
166 -void web_client_request_done(struct web_client *w) {
167 - web_client_uncork_socket(w);
166 +const char *get_request_method(struct web_client *w) {
167 + switch(w->mode) {
168 + case WEB_CLIENT_MODE_FILECOPY:
169 + return "FILECOPY";
170
169 - netdata_log_debug(D_WEB_CLIENT, "%llu: Resetting client.", w->id);
171 + case WEB_CLIENT_MODE_OPTIONS:
172 + return "OPTIONS";
173 +
174 + case WEB_CLIENT_MODE_STREAM:
175 + return "STREAM";
176 +
177 + case WEB_CLIENT_MODE_POST:
178 + return "POST";
179 +
180 + case WEB_CLIENT_MODE_PUT:
181 + return "PUT";
182 +
183 + case WEB_CLIENT_MODE_GET:
184 + return "GET";
185
171 - if(likely(buffer_strlen(w->url_as_received))) {
172 - struct timeval tv;
173 - now_monotonic_high_precision_timeval(&tv);
186 + case WEB_CLIENT_MODE_DELETE:
187 + return "DELETE";
188
175 - size_t size = (w->mode == WEB_CLIENT_MODE_FILECOPY)?w->response.rlen:w->response.data->len;
176 - size_t sent = size;
177 - if(likely(w->response.zoutput)) sent = (size_t)w->response.zstream.total_out;
189 + default:
190 + return "UNKNOWN";
191 + }
192 +}
193
179 - // --------------------------------------------------------------------
180 - // global statistics
194 +void web_client_log_completed_request(struct web_client *w, bool update_web_stats) {
195 + struct timeval tv;
196 + now_monotonic_high_precision_timeval(&tv);
197
198 + size_t size = (w->mode == WEB_CLIENT_MODE_FILECOPY)?w->response.rlen:w->response.data->len;
199 + size_t sent = size;
200 + if(likely(w->response.zoutput)) sent = (size_t)w->response.zstream.total_out;
201 +
202 + if(update_web_stats)
203 global_statistics_web_request_completed(dt_usec(&tv, &w->timings.tv_in),
204 w->statistics.received_bytes,
205 w->statistics.sent_bytes,
206 size,
207 sent);
208
188 - w->statistics.received_bytes = 0;
189 - w->statistics.sent_bytes = 0;
190 -
191 -
192 - // --------------------------------------------------------------------
193 -
194 - const char *mode;
195 - switch(w->mode) {
196 - case WEB_CLIENT_MODE_FILECOPY:
197 - mode = "FILECOPY";
198 - break;
199 -
200 - case WEB_CLIENT_MODE_OPTIONS:
201 - mode = "OPTIONS";
202 - break;
209 + usec_t prep_ut = w->timings.tv_ready.tv_sec ? dt_usec(&w->timings.tv_ready, &w->timings.tv_in) : 0;
210 + usec_t sent_ut = w->timings.tv_ready.tv_sec ? dt_usec(&tv, &w->timings.tv_ready) : 0;
211 + usec_t total_ut = dt_usec(&tv, &w->timings.tv_in);
212 + strip_control_characters((char *)buffer_tostring(w->url_as_received));
213 +
214 + ND_LOG_STACK lgs[] = {
215 + ND_LOG_FIELD_U64(NDF_CONNECTION_ID, w->id),
216 + ND_LOG_FIELD_UUID(NDF_TRANSACTION_ID, &w->transaction),
217 + ND_LOG_FIELD_TXT(NDF_NIDL_NODE, w->client_host),
218 + ND_LOG_FIELD_TXT(NDF_REQUEST_METHOD, get_request_method(w)),
219 + ND_LOG_FIELD_BFR(NDF_REQUEST, w->url_as_received),
220 + ND_LOG_FIELD_U64(NDF_RESPONSE_CODE, w->response.code),
221 + ND_LOG_FIELD_U64(NDF_RESPONSE_SENT_BYTES, sent),
222 + ND_LOG_FIELD_U64(NDF_RESPONSE_SIZE_BYTES, size),
223 + ND_LOG_FIELD_U64(NDF_RESPONSE_PREPARATION_TIME_USEC, prep_ut),
224 + ND_LOG_FIELD_U64(NDF_RESPONSE_SENT_TIME_USEC, sent_ut),
225 + ND_LOG_FIELD_U64(NDF_RESPONSE_TOTAL_TIME_USEC, total_ut),
226 + ND_LOG_FIELD_END(),
227 + };
228 + ND_LOG_STACK_PUSH(lgs);
229 +
230 + ND_LOG_FIELD_PRIORITY prio = NDLP_INFO;
231 + if(w->response.code >= 500)
232 + prio = NDLP_EMERG;
233 + else if(w->response.code >= 400)
234 + prio = NDLP_WARNING;
235 + else if(w->response.code >= 300)
236 + prio = NDLP_NOTICE;
237 +
238 + // access log
239 + nd_log(NDLS_ACCESS, prio, NULL);
240 +}
241
204 - case WEB_CLIENT_MODE_STREAM:
205 - mode = "STREAM";
206 - break;
242 +void web_client_request_done(struct web_client *w) {
243 + ND_LOG_STACK lgs[] = {
244 + ND_LOG_FIELD_TXT(NDF_SRC_IP, w->client_ip),
245 + ND_LOG_FIELD_TXT(NDF_SRC_PORT, w->client_port),
246 + ND_LOG_FIELD_END(),
247 + };
248 + ND_LOG_STACK_PUSH(lgs);
249
208 - case WEB_CLIENT_MODE_POST:
209 - case WEB_CLIENT_MODE_PUT:
210 - case WEB_CLIENT_MODE_GET:
211 - case WEB_CLIENT_MODE_DELETE:
212 - mode = "DATA";
213 - break;
250 + web_client_uncork_socket(w);
251
215 - default:
216 - mode = "UNKNOWN";
217 - break;
218 - }
252 + netdata_log_debug(D_WEB_CLIENT, "%llu: Resetting client.", w->id);
253
220 - // access log
221 - netdata_log_access("%llu: %d '[%s]:%s' '%s' (sent/all = %zu/%zu bytes %0.0f%%, prep/sent/total = %0.2f/%0.2f/%0.2f ms) %d '%s'",
222 - w->id
223 - , gettid()
224 - , w->client_ip
225 - , w->client_port
226 - , mode
227 - , sent
228 - , size
229 - , -((size > 0) ? ((double)(size - sent) / (double) size * 100.0) : 0.0)
230 - , (double)dt_usec(&w->timings.tv_ready, &w->timings.tv_in) / 1000.0
231 - , (double)dt_usec(&tv, &w->timings.tv_ready) / 1000.0
232 - , (double)dt_usec(&tv, &w->timings.tv_in) / 1000.0
233 - , w->response.code
234 - , strip_control_characters((char *)buffer_tostring(w->url_as_received))
235 - );
236 - }
254 + if(likely(buffer_strlen(w->url_as_received)))
255 + web_client_log_completed_request(w, true);
256
257 if(unlikely(w->mode == WEB_CLIENT_MODE_FILECOPY)) {
258 if(w->ifd != w->ofd) {
@@ -268,6 +287,9 @@ void web_client_request_done(struct web_client *w) {
287 w->response.sent = 0;
288 w->response.code = 0;
289 w->response.zoutput = false;
290 +
291 + w->statistics.received_bytes = 0;
292 + w->statistics.sent_bytes = 0;
293 }
294
295 static struct {
@@ -710,12 +732,22 @@ static inline int UNUSED_FUNCTION(check_host_and_mgmt_acl_and_call)(RRDHOST *hos
732 return check_host_and_call(host, w, url, func);
733 }
734
713 -int web_client_api_request(RRDHOST *host, struct web_client *w, char *url_path_fragment)
714 -{
735 +int web_client_api_request(RRDHOST *host, struct web_client *w, char *url_path_fragment) {
736 + ND_LOG_STACK lgs[] = {
737 + ND_LOG_FIELD_TXT(NDF_SRC_IP, w->client_ip),
738 + ND_LOG_FIELD_TXT(NDF_SRC_PORT, w->client_port),
739 + ND_LOG_FIELD_TXT(NDF_NIDL_NODE, w->client_host),
740 + ND_LOG_FIELD_TXT(NDF_REQUEST_METHOD, get_request_method(w)),
741 + ND_LOG_FIELD_BFR(NDF_REQUEST, w->url_as_received),
742 + ND_LOG_FIELD_U64(NDF_CONNECTION_ID, w->id),
743 + ND_LOG_FIELD_UUID(NDF_TRANSACTION_ID, &w->transaction),
744 + ND_LOG_FIELD_END(),
745 + };
746 + ND_LOG_STACK_PUSH(lgs);
747 +
748 // get the api version
749 char *tok = strsep_skip_consecutive_separators(&url_path_fragment, "/");
750 if(tok && *tok) {
718 - netdata_log_debug(D_WEB_CLIENT, "%llu: Searching for API version '%s'.", w->id, tok);
751 if(strcmp(tok, "v2") == 0)
752 return web_client_api_request_v2(host, w, url_path_fragment);
753 else if(strcmp(tok, "v1") == 0)
@@ -978,7 +1010,7 @@ const char *web_response_code_to_string(int code) {
1010
1011 static inline char *http_header_parse(struct web_client *w, char *s, int parse_useragent) {
1012 static uint32_t hash_origin = 0, hash_connection = 0, hash_donottrack = 0, hash_useragent = 0,
981 - hash_authorization = 0, hash_host = 0, hash_forwarded_host = 0;
1013 + hash_authorization = 0, hash_host = 0, hash_forwarded_host = 0, hash_transaction_id = 0;
1014 static uint32_t hash_accept_encoding = 0;
1015
1016 if(unlikely(!hash_origin)) {
@@ -990,6 +1022,7 @@ static inline char *http_header_parse(struct web_client *w, char *s, int parse_u
1022 hash_authorization = simple_uhash("X-Auth-Token");
1023 hash_host = simple_uhash("Host");
1024 hash_forwarded_host = simple_uhash("X-Forwarded-Host");
1025 + hash_transaction_id = simple_uhash("X-Transaction-ID");
1026 }
1027
1028 char *e = s;
@@ -1057,6 +1090,11 @@ static inline char *http_header_parse(struct web_client *w, char *s, int parse_u
1090 strncpyz(buffer, v, ((size_t)(ve - v) < sizeof(buffer) - 1 ? (size_t)(ve - v) : sizeof(buffer) - 1));
1091 w->forwarded_host = strdupz(buffer);
1092 }
1093 + else if(hash == hash_transaction_id && !strcasecmp(s, "X-Transaction-ID")) {
1094 + char buffer[UUID_STR_LEN * 2];
1095 + strncpyz(buffer, v, ((size_t)(ve - v) < sizeof(buffer) - 1 ? (size_t)(ve - v) : sizeof(buffer) - 1));
1096 + uuid_parse_flexi(buffer, w->transaction); // will not alter w->transaction if it fails
1097 + }
1098
1099 *e = ':';
1100 *ve = '\r';
@@ -1305,16 +1343,9 @@ void web_client_build_http_header(struct web_client *w) {
1343 const char *code_msg = web_response_code_to_string(w->response.code);
1344
1345 // prepare the last modified and expiration dates
1308 - char date[32], edate[32];
1309 - {
1310 - struct tm tmbuf, *tm;
1311 -
1312 - tm = gmtime_r(&w->response.data->date, &tmbuf);
1313 - strftime(date, sizeof(date), "%a, %d %b %Y %H:%M:%S %Z", tm);
1314 -
1315 - tm = gmtime_r(&w->response.data->expires, &tmbuf);
1316 - strftime(edate, sizeof(edate), "%a, %d %b %Y %H:%M:%S %Z", tm);
1317 - }
1346 + char rfc7231_date[RFC7231_MAX_LENGTH], rfc7231_expires[RFC7231_MAX_LENGTH];
1347 + rfc7231_datetime(rfc7231_date, sizeof(rfc7231_date), w->response.data->date);
1348 + rfc7231_datetime(rfc7231_expires, sizeof(rfc7231_expires), w->response.data->expires);
1349
1350 if (w->response.code == HTTP_RESP_HTTPS_UPGRADE) {
1351 buffer_sprintf(w->response.header_output,
@@ -1340,7 +1371,7 @@ void web_client_build_http_header(struct web_client *w) {
1371 VERSION,
1372 w->origin ? w->origin : "*",
1373 content_type_string,
1343 - date);
1374 + rfc7231_date);
1375 }
1376
1377 if(unlikely(web_x_frame_options))
@@ -1374,7 +1405,7 @@ void web_client_build_http_header(struct web_client *w) {
1405 "Cache-Control: %s\r\n"
1406 "Expires: %s\r\n",
1407 (w->response.data->options & WB_CONTENT_NO_CACHEABLE)?"no-cache, no-store, must-revalidate\r\nPragma: no-cache":"public",
1377 - edate);
1408 + rfc7231_expires);
1409 }
1410
1411 // copy a possibly available custom header
@@ -1398,6 +1429,11 @@ void web_client_build_http_header(struct web_client *w) {
1429 }
1430 }
1431
1432 + char uuid[UUID_COMPACT_STR_LEN];
1433 + uuid_unparse_lower_compact(w->transaction, uuid);
1434 + buffer_sprintf(w->response.header_output,
1435 + "X-Transaction-ID: %s\r\n", uuid);
1436 +
1437 // end of HTTP header
1438 buffer_strcat(w->response.header_output, "\r\n");
1439 }
@@ -1541,6 +1577,20 @@ static inline int web_client_switch_host(RRDHOST *host, struct web_client *w, ch
1577 }
1578
1579 int web_client_api_request_with_node_selection(RRDHOST *host, struct web_client *w, char *decoded_url_path) {
1580 + // entry point for all API requests
1581 +
1582 + ND_LOG_STACK lgs[] = {
1583 + ND_LOG_FIELD_TXT(NDF_REQUEST_METHOD, get_request_method(w)),
1584 + ND_LOG_FIELD_BFR(NDF_REQUEST, w->url_as_received),
1585 + ND_LOG_FIELD_U64(NDF_CONNECTION_ID, w->id),
1586 + ND_LOG_FIELD_UUID(NDF_TRANSACTION_ID, &w->transaction),
1587 + ND_LOG_FIELD_END(),
1588 + };
1589 + ND_LOG_STACK_PUSH(lgs);
1590 +
1591 + // give a new transaction id to the request
1592 + uuid_generate_random(w->transaction);
1593 +
1594 static uint32_t
1595 hash_api = 0,
1596 hash_host = 0,
@@ -1729,7 +1779,37 @@ static inline int web_client_process_url(RRDHOST *host, struct web_client *w, ch
1779 return mysendfile(w, filename);
1780 }
1781
1732 -void web_client_process_request(struct web_client *w) {
1782 +static bool web_server_log_transport(BUFFER *wb, void *ptr) {
1783 + struct web_client *w = ptr;
1784 + if(!w)
1785 + return false;
1786 +
1787 +#ifdef ENABLE_HTTPS
1788 + buffer_strcat(wb, SSL_connection(&w->ssl) ? "https" : "http");
1789 +#else
1790 + buffer_strcat(wb, "http");
1791 +#endif
1792 + return true;
1793 +}
1794 +
1795 +void web_client_process_request_from_web_server(struct web_client *w) {
1796 + // entry point for web server requests
1797 +
1798 + ND_LOG_STACK lgs[] = {
1799 + ND_LOG_FIELD_CB(NDF_SRC_TRANSPORT, web_server_log_transport, w),
1800 + ND_LOG_FIELD_TXT(NDF_SRC_IP, w->client_ip),
1801 + ND_LOG_FIELD_TXT(NDF_SRC_PORT, w->client_port),
1802 + ND_LOG_FIELD_TXT(NDF_NIDL_NODE, w->client_host),
1803 + ND_LOG_FIELD_TXT(NDF_REQUEST_METHOD, get_request_method(w)),
1804 + ND_LOG_FIELD_BFR(NDF_REQUEST, w->url_as_received),
1805 + ND_LOG_FIELD_U64(NDF_CONNECTION_ID, w->id),
1806 + ND_LOG_FIELD_UUID(NDF_TRANSACTION_ID, &w->transaction),
1807 + ND_LOG_FIELD_END(),
1808 + };
1809 + ND_LOG_STACK_PUSH(lgs);
1810 +
1811 + // give a new transaction id to the request
1812 + uuid_generate_random(w->transaction);
1813
1814 // start timing us
1815 web_client_timeout_checkpoint_init(w);
web/server/web_client.h
+4 -1
@@ -136,6 +136,8 @@ struct web_client {
136 unsigned long long id;
137 size_t use_count;
138
139 + uuid_t transaction;
140 +
141 WEB_CLIENT_FLAGS flags; // status flags for the client
142 WEB_CLIENT_MODE mode; // the operational mode of the client
143 WEB_CLIENT_ACL acl; // the access list of the client
@@ -207,7 +209,7 @@ ssize_t web_client_send(struct web_client *w);
209 ssize_t web_client_receive(struct web_client *w);
210 ssize_t web_client_read_file(struct web_client *w);
211
210 -void web_client_process_request(struct web_client *w);
212 +void web_client_process_request_from_web_server(struct web_client *w);
213 void web_client_request_done(struct web_client *w);
214
215 void buffer_data_options2string(BUFFER *wb, uint32_t options);
@@ -234,5 +236,6 @@ void web_client_timeout_checkpoint_set(struct web_client *w, int timeout_ms);
236 usec_t web_client_timeout_checkpoint(struct web_client *w);
237 bool web_client_timeout_checkpoint_and_check(struct web_client *w, usec_t *usec_since_last_checkpoint);
238 usec_t web_client_timeout_checkpoint_response_ready(struct web_client *w, usec_t *usec_since_last_checkpoint);
239 +void web_client_log_completed_request(struct web_client *w, bool update_web_stats);
240
241 #endif
web/server/web_client_cache.c
+1
@@ -115,6 +115,7 @@ struct web_client *web_client_get_from_cache(void) {
115 // initialize it
116 w->use_count++;
117 w->mode = WEB_CLIENT_MODE_GET;
118 + memset(w->transaction, 0, sizeof(w->transaction));
119
120 return w;
121 }
web/server/web_server.c
+14 -1
@@ -130,5 +130,18 @@ void web_client_update_acl_matches(struct web_client *w) {
130 // --------------------------------------------------------------------------------------
131
132 void web_server_log_connection(struct web_client *w, const char *msg) {
133 - netdata_log_access("%llu: %d '[%s]:%s' '%s'", w->id, gettid(), w->client_ip, w->client_port, msg);
133 + ND_LOG_STACK lgs[] = {
134 + ND_LOG_FIELD_U64(NDF_CONNECTION_ID, w->id),
135 +#ifdef ENABLE_HTTPS
136 + ND_LOG_FIELD_TXT(NDF_SRC_TRANSPORT, SSL_connection(&w->ssl) ? "https" : "http"),
137 +#else
138 + ND_LOG_FIELD_TXT(NDF_SRC_TRANSPORT, "http"),
139 +#endif
140 + ND_LOG_FIELD_TXT(NDF_SRC_IP, w->client_ip),
141 + ND_LOG_FIELD_TXT(NDF_SRC_PORT, w->client_port),
142 + ND_LOG_FIELD_END(),
143 + };
144 + ND_LOG_STACK_PUSH(lgs);
145 +
146 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "[%s]:%s %s", w->client_ip, w->client_port, msg);
147 }